Testee protocol

A testee is a small program that wraps one Stellar SDK at one exact version and answers requests from the conformance runner (shapekit conformance run). The runner sends corpus data through the SDK's own decoder, encoder, strkey codec, ordering functions and native projection, and compares what comes back with the network's answer (the corpus manifest) and with the other SDKs.

Protocol version: 1. The machine-readable form is schema/testee-protocol.schema.json. Anything this page and the schema disagree on is a bug in one of them.

TransportCopy Anchor Link

• The runner starts the testee as a child process and talks to it over stdin and stdout.
• Every message is one JSON object encoded as UTF-8 on one line, terminated by \n. No pretty printing, no blank lines, no byte order mark.
• The runner sends one request and waits for exactly one response before it sends the next. Requests and responses are matched by id as well; a response whose id differs from the outstanding request is a protocol violation.
• stdout carries responses only. Logging, warnings and stack traces go to stderr, which the runner keeps and shows next to any error result.
• A line can be large (a full LedgerCloseMeta in base64). Testees must accept request lines of at least 64 MiB.
• A response line may be at most 256 MiB (268435456 bytes, not counting the \n). The runner stops reading a longer line at that size, records a protocol violation (error) and restarts the testee, so a testee that prints without end cannot grow the runner without bound. A response that is not UTF-8 is a protocol violation too.
• When stdin reaches end of file the testee exits with status 0.
• The runner enforces a per-request timeout and an address-space or data-segment limit on the testee process (see "Limits the runner applies"). The timeout covers writing the request as well as reading the response: a testee that stops reading stdin cannot block the runner on a request larger than the pipe buffer. A testee that times out, exits or is killed is restarted, said hello to again, and the request that was in flight is recorded as error.

MessagesCopy Anchor Link

RequestCopy Anchor Link

{"id": "17", "op": "decode_encode", "type": "TransactionEnvelope", "xdr": "AAAAAg...", "depth_limit": 1000, "len_limit": 4096}

• id: string chosen by the runner, unique within one process lifetime. Echo it back. A line that is not a JSON object with a string id is answered with "id": "" and a bad-request error.
• op: one of the operation names below.
• The other members depend on op. Unknown members must be ignored, so that later protocol versions can add optional members.

ResponseCopy Anchor Link

Exactly one of three forms:

{"id": "17", "status": "ok", "result": {"xdr": "AAAAAg..."}}
{"id": "17", "status": "error", "error": {"class": "rejected", "message": "xdr value invalid"}}
{"id": "17", "status": "unsupported", "reason": "go-stellar-sdk has no scval map sort"}

• status: "ok": the SDK produced a value. result holds the op's result members.
• status: "error": the SDK, or the testee, did not produce a value. error.class says why, error.message is the SDK's own message (free text, never compared).
• status: "unsupported": the SDK version under test has no API for this operation, or for this particular input of it (for example an XDR type the SDK does not know). reason is required and should name what is missing. The matrix shows it as not applicable and it is never counted as a pass.

Error classesCopy Anchor Link

A testee that cannot tell rejected from panic (because the SDK uses one exception type for both) reports rejected. Catch panics around every SDK call so one input cannot take the process down; a process that dies anyway is recorded as error for that request.

Value encodingsCopy Anchor Link

These are fixed for every op so that the runner compares like with like.

helloCopy Anchor Link

Sent first, after every (re)start.

Request: {"id": "0", "op": "hello"}

Result members:

OperationsCopy Anchor Link

decode_encodeCopy Anchor Link

Decode bytes as one value of a type and encode the value again.

Request: type (string), xdr (base64), depth_limit (integer), len_limit (integer).

• Decode must consume the whole input. Trailing bytes are a rejection.
• depth_limit and len_limit are the runner's decode limits (Limits { depth, len } in rs-stellar-xdr). A testee passes them to the SDK when the SDK's API accepts limits and reports decode_limits: "honoured". Otherwise it decodes with the SDK's defaults and reports "ignored"; the matrix then notes that depth and length entries ran under SDK defaults.
• Re-encode with the SDK's encoder and the same limits.

Result: xdr (base64 of the re-encoded bytes).

decode_jsonCopy Anchor Link

Decode bytes as one value of a type and render it as SEP-51 XDR-JSON.

Request: same members as decode_encode.

Result: json (the SEP-51 value as a JSON value, not a string).

A testee implements this only with the SDK's own decoded objects: either an SDK-native SEP-51 renderer, or a projection the testee writes from the SDK's decoded value (js-xdr type introspection, Go reflection). A second XDR decoder inside the testee defeats the check.

encode_jsonCopy Anchor Link

Read a SEP-51 XDR-JSON value of a type and encode it as XDR.

Request: type (string), json (JSON value).

Result: xdr (base64).

This op reads JSON where every other op reads bytes. It is needed because the rs-stellar-xdr advisory GHSA-3xhm-p452-9wjh is a JSON-to-XDR flaw: 28.0.0 accepts a JSON ScSpecFunctionV0 with two outputs (the XDR allows at most one) and encodes bytes that no decoder accepts. No decode-side op can see it. SDKs without a SEP-51 reader answer unsupported.

strkey_decodeCopy Anchor Link

Request: text (string, passed to the SDK unchanged, including any whitespace or case oddities), or instead text_base64 (standard base64 of the input bytes) when the input is not UTF-8 and so has no JSON string form. Exactly one of the two is present.

A testee passes the text_base64 bytes to the SDK as they are when the SDK's decoder can take them (a Go string, a Rust &[u8]). An SDK whose decoder takes only Unicode text cannot be given bytes that are not UTF-8, so its testee answers unsupported for them rather than picking a lossy mapping to text.

Result: kind (strkey kind) and payload (base64 of the raw payload, defined below).

If the SDK has one decoder per kind rather than a generic one, the testee picks the decoder from the first character and reports rejected when that decoder refuses the text, and when no decoder exists for that first character.

strkey_encodeCopy Anchor Link

Request: kind (strkey kind), payload (base64 raw payload).

Result: text.

The testee calls the SDK's encoder for that kind with the payload in whatever structured form the SDK's encoder takes, and must not repair the payload first. An SDK encoder that refuses the payload is rejected.

Strkey kinds and raw payloadsCopy Anchor Link

The raw payload is the byte string between the version byte and the checksum, as SEP-23 defines it. A testee whose SDK returns structured values rebuilds this layout from them.

asset_orderCopy Anchor Link

Request: a, b (each base64 XDR of an Asset).

The testee decodes both with the SDK's XDR decoder, turns them into the SDK's asset objects where the comparison lives on those objects (JS Asset.fromOperation), and calls the SDK's comparison (Asset.compare in JS, xdr.Asset.LessThan in Go, Ord in Rust).

Result: order (-1, 0 or 1). An SDK that only offers "less than" answers with lt(a, b) ? -1 : lt(b, a) ? 1 : 0.

pool_idCopy Anchor Link

Request: a, b (base64 XDR Asset), fee (integer, basis points; 30 is the only fee core accepts).

The assets are passed to the SDK in the given order. An SDK that refuses an unordered or equal pair answers rejected. An SDK with XDR types but no pool ID function (Rust stellar-xdr) computes SHA-256 of the LiquidityPoolParameters XDR written by the SDK's own encoder, and refuses pairs that the SDK's own Asset ordering does not put strictly in order, as core does.

Result: pool_id (lowercase hex).

map_sortCopy Anchor Link

Request: entries, an array of {"key": <base64 ScVal>, "val": <base64 ScVal>} in input order.

The testee calls the SDK's sorted-map constructor (nativeToScVal/scvSortedMap in JS, ScMap::sorted_from in Rust).

Result: xdr (base64 of the resulting ScVal, arm SCV_MAP).

map_validateCopy Anchor Link

Request: xdr (base64 of an ScVal whose arm is SCV_MAP with a present map).

Result: valid (boolean): whether the SDK's own map validation accepts the map. An SDK validation that refuses the map is "valid": false, not an error; rejected is only for bytes the SDK cannot decode.

id_parseCopy Anchor Link

Request: kind ("muxed" or "offer"), input (a JSON string or a JSON number, written by the runner exactly as the corpus vector has it, which includes numbers above 2^53 and negative numbers).

The testee hands input to the SDK API that takes a textual or numeric ID (JS MuxedAccount and Operation.manageSellOffer; in Rust the only textual path is the XDR-JSON reader of the id and offer_id fields), in the JSON type it arrived as. A testee must read numbers without going through a binary floating point type unless that is what the SDK itself does with them. If the SDK has no API for the input's JSON type, answer unsupported.

Result: id, the 64-bit value the SDK would encode, as a decimal string (unsigned for muxed, signed for offer).

to_nativeCopy Anchor Link

Request: xdr (base64 of an ScVal).

The testee calls the SDK's native projection (JS scValToNative) and describes the result losslessly as a native value:

Result: native.

How the runner uses the opsCopy Anchor Link

Per manifest entry kind (crates/shapekit-manifest/src/model.rs). Vector cases (corpus/vectors/<kind>/<id>.json) carry their op name in op, their inputs, and expected; the runner turns each into the request below. A case it cannot map is an error (a runner bug), never skipped.

Where the expected XDR-JSON contains stellar-xdr's <INVALID: marker (it writes one for a value with no strkey form, such as a signed-payload signer with an empty payload), SEP-51 has no canonical rendering of the value. decode_json then passes with any JSON the SDK renders (it still fails if the SDK refuses), and encode_json is not sent, since there is no canonical JSON to encode from. decode_encode checks the byte round trip as usual.

Decode requests carry depth_limit 1000 (DECODE_DEPTH, the depth core's Rust parser applies to envelopes from protocol 28) and len_limit equal to the input length, the limits the corpus expectations were computed under.

Every xdr variant (testnet, mainnet, any) is checked separately. An entry's result is the worst of its checks, in this order: error, diverges-from-network, diverges-from-peers, unsupported, reference, pass (a reference check counts as a pass for the entry's result, see below). A panic always counts as diverges-from-network.

Because an entry is as bad as its worst check, one op an SDK lacks (Go has no encode_json) makes the whole entry unsupported. The matrix therefore also counts every check on its own, per testee (check_summary) and per testee and op (op_summary), and the run prints both the entry and the check totals.

Implementation-defined checks are decided by one vote per SDK: the SDK's current version in versions.toml votes when the run holds it, otherwise its last version in the run (versions.toml order, or the order of --version). An SDK whose voting version does not answer casts no vote. The most common vote is the majority; every testee, voting or not, whose answer differs from it is diverges-from-peers, and with a tie no answer is the majority. With fewer than two votes, as in any run of a single SDK however many versions it holds, there is no peer to compare with and the check counts as unsupported, not as a pass.

versions.toml may give an SDK version a max_protocol. A divergence on an entry generated at a later protocol (protocol.generated_at, else protocol.valid_min) is then reported as unsupported with the original verdict in its detail: an SDK that predates a protocol is not expected to read it. A panic or a testee process that died is never downgraded this way: crashing on unknown XDR is a robustness defect, not a missing feature.

The reference testee and the cross-checkCopy Anchor Link

Every expected_json in the corpus is the XDR-JSON rendering of the stellar-xdr version the generator links (generated_with.stellar_xdr in the manifest). The Rust testee at that version renders with the same library, so its decode_json and encode_json checks agreeing with expected_json prove nothing beyond the library agreeing with itself. The matrix marks that testee as the reference for expected_json (reference_for in testees) and counts those checks as reference (listed in the cell's reference_checks), never as pass. A reference check that disagrees is still a failure: it means the library cannot read back its own rendering. In the entry summary, a passing cell with reference checks is counted under reference.

shapekit conformance crosscheck supplies the independent evidence: it sends every decode_json check with a canonical expected_json to the Go and JS testees at their current versions and reports, per entry, where either renders something else. A disagreement where two decoders render the same other value points at stellar-xdr's rendering; one where a single decoder differs while another agrees is that SDK's divergence. The command fails unless every disagreement is a single-SDK divergence already in that SDK's known-failures list.

Known failuresCopy Anchor Link

conformance/known-failures/<sdk>.txt lists one known failure per line:

<entry-id> <version or *> <result> <issue link or findings/issues/ draft> <check>[, <check>...]

The first four fields are separated by whitespace. Everything after the issue is the list of failing checks, separated by commas, each named exactly as the matrix names it (testnet/decode_json, case 3/strkey_decode). Every listed check must fail with <result>; checks of one entry that fail with different results go on separate lines. * stands for every version and is only used where every version fails the same checks. A # starts a comment only at the start of a line or after whitespace, so an issue URL keeps its #fragment.

Each check is judged on its own. A failing check (diverges-from-network, diverges-from-peers, error) listed with its result does not fail the run. A failing check that no line lists fails it, even when other checks of the same entry are listed, so a new regression next to a known failure is caught. A listed check that now passes, is reference, is unsupported, fails with another result, or is not a check of the entry fails the run too, until the line is fixed. Every line must name an entry in the manifest and a version in versions.toml. A diverges-from-peers check can only be reproduced by a run of more than one SDK; a single-SDK run does not ask for such a line to be removed.

Limits the runner appliesCopy Anchor Link

• Per-request timeout, default 10 s, set per SDK in conformance/versions.toml. It covers the write of the request and the read of the response.
• Response lines of at most 256 MiB.
• A memory limit set with setrlimit in the child before exec: RLIMIT_AS (address space) for testees whose runtime reserves little virtual memory (Rust, Go), RLIMIT_DATA for runtimes that reserve large PROT_NONE regions up front (Node's V8 cage), where an address-space cap would kill the process before it touches the data. The resource and size are set per SDK in versions.toml.

Writing a testeeCopy Anchor Link

1. Read requests line by line, answer each with exactly one line, flush after every line.
2. Answer hello with versions read from the build (Go debug.ReadBuildInfo, the installed package's package.json, Cargo's resolved dependency) and never from a constant.
3. Implement each op against the SDK's public API only. Where the SDK has no API, answer unsupported with a reason. Never emulate an SDK feature inside the testee.
4. Catch every exception or panic around SDK calls and classify it.
5. Add the testee's build and run command per SDK version to conformance/versions.toml.

The reference testees are testees/rust, testees/go and testees/js.