Adapter interface

The replay harness (shapekit harness run) feeds recorded chains to an indexer and checks what the indexer stored against the meta oracle. It never talks to the indexer directly. An adapter sits in between: a program that starts the indexer, points it at the harness's ledger delivery endpoints, reports how far it got, and reads back what it stored in the oracle's vocabulary.

Protocol version: 1. The machine-readable form is schema/adapter-protocol.schema.json, generated from crates/shapekit-harness/src/protocol.rs together with the vocabulary section at the end of this page (shapekit harness schema --write; cargo test -p shapekit-harness fails when either is stale). Anything this page and the schema disagree on is a bug in one of them.

The reference adapter is adapters/walletbackend (wallet-backend, Go).

What the harness does with an adapterCopy Anchor Link

For each chain (one recorded ledger sequence per shape family, protocol and passphrase):

1. Creates a Docker network with --internal and starts the replay server (shapekit-serve, in process) bound to that network's gateway address on a free port. The visible head is set to the bootstrap ledger (see "Bootstrap").
2. Starts the adapter program, sends describe, then start.
3. Waits until status reports last_ingested_ledger at or past the bootstrap ledger.
4. For every later ledger of the chain in order: moves the server's head to it, checks that the server answers getLedgers for it (a ledger the server cannot serve is a harness error, never a target verdict), sends advance, and polls status until the target has ingested it, the target died, or it stopped making progress.
5. Sends export for every ledger the target ingested and compares the records with the oracle files of the chain's corpus entries.
6. Sends stop, keeps the returned logs, removes the network and the work directory.

A new adapter process is started for every chain, so a target that halts in one family does not hide the others.

TransportCopy Anchor Link

• The harness starts the adapter as a child process with the command line it was given and talks to it over stdin and stdout. The working directory is the harness's.
• Every message is one JSON object encoded as UTF-8 on one line, terminated by \n. No pretty printing, no blank lines, no byte order mark.
• The harness sends one request and reads response lines until the final line for that request (status ok or error) before it sends the next. Every response line carries the request's id. A line with another id is a protocol violation and ends the chain's run.
• Only export has more than one response line: any number of record lines, then the final line.
• stdout carries responses only. Logs go to stderr; the harness saves the adapter's stderr next to the report (logs/<chain>.<network>.adapter.log).
• Members the receiver does not know must be ignored, so later protocol versions can add optional members.
• When stdin reaches end of file the adapter tears the target down as for stop and exits.
• The harness enforces a timeout per request (defaults: describe 60 s, start 30 min, since it may build images, advance 120 s, status 300 s, export 10 min, stop 5 min). --start-timeout and --status-timeout change the two that depend on the host. The status default is sized for a heavily loaded host: a wallet-backend status answer (a database query and a container inspection) took more than 60 s at a load average near 200, and a status timeout ends the chain. On a timeout, a protocol violation or an adapter exit, the harness sends stop where it still can, kills the adapter process and records the chain's remaining entries as not-evaluated with the reason.
• A request whose timeout passed is abandoned, not forgotten: when its answer lines arrive later, while the harness waits for the reply to a later request (in practice stop), they are skipped and listed in the chain's notes, up to and including the abandoned request's final line. This is the one exception to the id rule above. A line that carries an id the harness never sent, or the id of a request that was already answered, is still a protocol violation, and so is anything that arrives after the reply the harness waits for (the harness does not read it). The adapter must therefore not leave containers behind when it is killed: label them with the chain's docker_network name, which the harness removes along with every container still attached to it.

RequestCopy Anchor Link

{"id": "3", "cmd": "advance", "head": 65}

• id: string, unique within one adapter process.
• cmd: describe, start, advance, status, export or stop. The command's members sit next to cmd.

ResponseCopy Anchor Link

{"id": "3", "status": "ok", "result": {"head": 65}}
{"id": "7", "status": "record", "record": {"type": "transaction", "ledger": 65, "hash": "...", "participants": [], "changes": []}}
{"id": "4", "status": "error", "error": {"class": "target", "message": "timescaledb did not accept connections within 120s"}}

Error classesCopy Anchor Link

An error never stands for a verdict. A target that crashes or halts while ingesting is reported through status, not through an error response.

CommandsCopy Anchor Link

describeCopy Anchor Link

Request: {"id": "0", "cmd": "describe", "protocol": 1}

Result:

startCopy Anchor Link

Request members (besides id and cmd):

The adapter brings the target up and answers once the ingesting process runs. It configures the target with the passphrase and the endpoints of its delivery mode only. The target gets no other network access: containers join docker_network only, which has no route off the host. The replay server URLs point at the gateway address of that network.

Result: components, an object of free-form strings (image ids, versions of services the target needs) that the report keeps.

advanceCopy Anchor Link

Request: {"cmd": "advance", "head": N}. Ledgers up to N are now visible on the replay server. In rpc mode the target sees them through getLedgers on its own and the adapter just answers. In sep54-filesystem and sep54-s3 modes the adapter copies the batches whose last ledger is at or below N (from lake.url, or from lake.path skipping later batches) into the directory or bucket the target reads, then answers.

Result: head, equal to N.

statusCopy Anchor Link

Result:

health: running does not mean healthy. A target that retries the same ledger forever is alive and running; the harness calls it halted when it stops advancing while last_error is set (see "Verdicts").

exportCopy Anchor Link

Request: {"cmd": "export", "from": A, "to": B}. Stream one record line per stored transaction in ledgers A to B inclusive, in ledger and apply order, then {"status": "ok", "result": {"records": N}} where N counts the record lines. Stream from the database cursor; do not build the whole answer in memory.

Records are of two types:

{"type": "transaction", "ledger": 65, "hash": "6f9d...", "participants": [...], "changes": [...], "unmapped": [...]}
{"type": "unattributed", "ledger": 65, "unmapped": [...]}

transaction:

A participant's address is the canonical strkey (G, C, L or B; never M, the mux id goes in mux_ids as decimal strings). When the stored bytes do not decode, address is null and raw says why.

A change is a vocabulary change (see the generated section below) with only the fields the target stores: kind, the claimed fields, optionally scope and op_index, and

• raw: for every address field read from stored bytes, {"column", "hex", "strkey"} when the bytes decode or {"column", "hex", "error"} when they do not. In the second case the field itself is null. hex is the stored value exactly, for wallet-backend's AddressBytea the version byte followed by the key bytes. Adapters must not repair, pad or truncate the bytes before decoding: a 33-byte value that cannot hold a 40-byte muxed key is exactly what the harness is looking for.
• source: {"table", "category", "reason"}, the row the change came from.

Values use the vocabulary's encodings exactly: amounts, balances, limits, ids and sequence numbers as decimal strings ("-100"), 32-bit values as JSON integers, assets as native or CODE:ISSUER (SEP-11), pool-share trustline assets as pool-share:<L strkey of the pool id>, hashes and XDR as lowercase hex or base64 as the field's description says, signers as full strkeys including P... signed-payload signers and their payload.

unmapped rows: {"table", "category", "reason", "row", "note"}. row holds every column (byte columns as lowercase hex); note says why there is no mapping. Rows are never dropped silently: an adapter that cannot map a row reports it here, and the report lists it next to the entry so a reviewer can tell an indexer drop from an adapter gap.

stopCopy Anchor Link

Result: logs, the tail of the target's logs (at most 1 MiB). The adapter removes every container, volume and file it created, then exits with status 0.

ClaimsCopy Anchor Link

describe.claims says which oracle facts the target records:

{
  "participants": {"address_kinds": ["account", "contract"], "mux_ids": true},
  "changes": [
    {"kind": "signer-added", "fields": ["account", "signer", "weight"]},
    {"kind": "trustline-created", "fields": ["account", "asset", "limit"]},
    {"kind": "token-event", "where": {"event": ["transfer", "mint", "burn", "clawback"]},
     "fields": ["event", "contract", "from", "to", "amount"]}
  ]
}

• participants: the oracle's actual participants (never the envelope's declared ones) whose address kind is listed are expected among the exported participants. With mux_ids, their mux ids are compared too.
• each changes item claims the oracle changes of kind that satisfy where (every listed field equals one of its values; an absent field equals null), compared on fields only.

A claim is a promise. Anything inside it that the export lacks is a finding; anything outside it is out-of-scope. Claim what the target is meant to record, not what it happens to get right.

BootstrapCopy Anchor Link

describe.bootstrap says how the target starts:

• checkpoint: the target reads the archive's .well-known/stellar-history.json, loads the ledger state of that checkpoint, and starts ingesting at the checkpoint ledger. wallet-backend live mode on an empty database does this at every commit pinned by this project (internal/services/ingest_live.go: archive.GetLatestLedgerSequence(), then PopulateFromCheckpoint, then ingestion from that ledger). It refuses an archive whose latest ledger is 0. The harness therefore picks the newest checkpoint before the chain's first entry ledger as ledgers.bootstrap and serves the archive as of that checkpoint.
• ledger: the target can be told a start ledger. ledgers.bootstrap is the chain's first ledger.

Recipe chains are recorded so that the first checkpoint exists before anything else happens: after the protocol upgrade in ledger 2, shapekit gen chains closes empty ledgers through ledger 63 (core's first checkpoint at the default frequency of 64) and waits for it to be published, then applies network settings and runs the recipe from ledger 64. The archive in each chain's archive.tar holds that checkpoint's real buckets. A chain recorded before this change has no checkpoint; for checkpoint targets its entries are reported not-evaluated with that reason until the chain is regenerated.

Network accessCopy Anchor Link

Containers reach the replay server through the internal network's gateway: Docker assigns the gateway address of a user-defined bridge network to the host side of the bridge, and a container on an --internal network can reach that address but has no default route, so nothing beyond the host is reachable. The harness binds the replay server to exactly that address, so the server is not reachable from the LAN either. host.docker.internal with host-gateway is not used: it resolves to the default bridge's address (172.17.0.1), which a container on an internal network cannot route to.

With --no-docker the harness creates no network and binds the server to 127.0.0.1, for adapters whose target runs as a host process.

VerdictsCopy Anchor Link

The harness gives every corpus entry of the chain that has an oracle file one verdict. Entries without an oracle file (ledger entries, synthetic entries, vectors) are not replayed and are counted separately.

When an entry has several problems the verdict is the first in the order halted, crashed, oom, timeout, undecodable-address, wrong-value, dropped, and every problem is listed in the report. A target stops "at" ledger L when L is the first ledger it did not ingest; every entry in L gets the stop verdict, entries in later ledgers are not-evaluated.

Facts the meta cannot showCopy Anchor Link

token-event and contract-event facts come only from events, and older meta formats have no place for most events. The oracle records, per transaction, events:

A claim on an event kind is not comparable for a transaction when the coverage lacks any scope the claim can match: the scopes of the stages its where.stage lists (before-all-txs is fee processing, operation, after-tx, after-all-txs is post-apply fee), or every event scope when the claim does not restrict stage. For such a claim the harness compares neither side on that transaction: an oracle fact is not expected, an exported fact is not extra, and the entry lists the claim under its detail. The verdict then follows the comparable facts: any problem there gives dropped or mishandled as usual; otherwise the entry is out-of-scope, never pass, because a pass would say the event claims were checked. Example: on cap67.unified-events.p22-payment-transfer (protocol 22, classic events off, meta v3) wallet-backend derives a transfer and a fee from the operation; the meta shows neither, and that is not evidence of anything.

The harness refuses oracle files without events (written before the field existed); shapekit oracle run rewrites them.

Pairing for wrong-value: an expected change and an exported change of the same kind are a pair when they agree on the kind's key fields that the claim compares:

Changes are compared as multisets: two equal transfers in one transaction must both be exported.

OutputCopy Anchor Link

shapekit harness run writes into --out (default harness-out/):

• harness-report.json: run metadata (adapter, target and version, delivery mode, corpus manifest hash, harness peak resident memory per round), per-chain results (bootstrap ledger, last ingested ledger, target peak memory, stop reason) and one row per entry with verdict, reason, the problems found and any unmapped rows.
• HARNESS.md: the same as tables, one row per entry.
• logs/: adapter stderr (<chain>.<network>.adapter.log) and the target logs stop returned (<chain>.<network>.target.log) per chain; with --repeat, one logs/round-<n>/ per round.

--repeat N runs the whole selection N times in one process and fails when a later round's peak resident memory exceeds the first round's by more than --memory-slack (default 10% or 16 MiB, whichever is larger); --memory-budget fails a round over a fixed bound. Peak memory is read from /proc/self/status (VmHWM), reset between rounds through /proc/self/clear_refs.

ReproductionsCopy Anchor Link

shapekit harness reproduce --table <toml> runs pairs of target builds over named chains and checks the verdicts of named entries:

[adapter]
# the command line; {commit} is replaced by each case's commit
command = ["adapters/bin/walletbackend-adapter", "--commit", "{commit}"]

[[reproduction]]
id = "wallet-backend-729"
issue = "https://github.com/stellar/wallet-backend/pull/729"
chain = "cap40.signed-payload-signer.p28"
network = "testnet"

[[reproduction.case]]
label = "pre-fix"
commit = "2a0a84ea12523b6c92e278c0c84b443041240095"
expect = [
  { entry = "cap40.signed-payload-signer.p28-set-options-payload-1-meta", verdict = "mishandled", reason = "halted" },
]

[[reproduction.case]]
label = "fixed"
commit = "2c873353c1f0f3c9b145e25a7d009023ad104a1c"
expect = [
  { entry = "cap40.signed-payload-signer.p28-set-options-payload-1-meta", verdict = "pass" },
]

The command runs with the harness's environment, so go must be on PATH for the table in adapters/walletbackend/reproductions.toml. A case holds when every listed entry has the listed verdict (and reason, when given), and its chain ran without a harness error. The command exits non-zero when any case does not hold, and writes reproductions.json and REPRODUCTIONS.md next to each case's full report.

Writing an adapterCopy Anchor Link

1. Read describe and start above, then the vocabulary below, and decide the claims: one item per vocabulary kind the target stores, with exactly the fields it keeps.
2. Map the target's rows to vocabulary changes. When the target stores something the vocabulary has no kind for, report the rows as unmapped.
3. Decode stored addresses with a strict strkey decoder and report the bytes in raw.
4. Apply the memory limit and join only docker_network.
5. Run shapekit harness run --adapter "<your command>" --chain <id> --network testnet on a small chain and read HARNESS.md.

The schema check cargo test -p shapekit-harness validates recorded adapter traffic against the schema; add a recording of your adapter's answers there when you add an adapter.

The wallet-backend adapterCopy Anchor Link

adapters/walletbackend (Go), binary adapters/walletbackend/cmd/wb-adapter:

(cd adapters && go build -o bin/wb-adapter ./walletbackend/cmd/wb-adapter)
shapekit harness run --adapter "adapters/bin/wb-adapter --commit <full commit hash>"

--repo names a wallet-backend clone (default: <user cache dir>/shapekit/wallet-backend, cloned from github.com/stellar/wallet-backend on first use); --db-memory caps the database container (default 1 GiB).

• Build. git archive <commit> | docker buildx build --load - with the commit's own Dockerfile, image shapekit-wallet-backend:<commit>. Each FROM in that Dockerfile is replaced by its base image pinned by digest (--build-context <tag>=docker-image://<image>@<digest>, the table BaseImages in docker.go, the same digests as pins.toml); a Dockerfile naming a base with no pin is refused. The image carries a label with the set of pinned bases it was built on and is reused only when that label matches. Building needs network access; running does not.
• Services. timescale/timescaledb:2.28.2-pg17 (by digest, as in pins.toml) with the settings of wallet-backend's docker-compose.yaml, the timescaledb extension created before wallet-backend migrate up, then wallet-backend ingest (live mode) with --memory and --memory-swap set to limits.memory_bytes and --restart on-failure. All containers join docker_network only. The restart policy stands in for the deployment's restart loop: wallet-backend gives up on a ledger after five persist attempts and exits, and when restarted it retries the same ledger, so a give-up is reported as a halt (running, cursor stuck, last_error set). crashed means a Go runtime failure in the log or a container that stayed down; oom means docker reports an OOM kill.
• Configuration. NETWORK_PASSPHRASE, RPC_URL, ARCHIVE_URL, CHECKPOINT_FREQUENCY from start; LEDGER_BACKEND_TYPE always set (commits before a9bac8d2 default to the datastore); STELLAR_ENVIRONMENT for commits whose ingest command still requires it; LOG_LEVEL=INFO. sep54-s3 adds chrislusf/seaweedfs:4.46 run as in wallet-backend's integration tests (weed server -s3, no auth, bucket ledgers) and the DATASTORE_* variables; advance uploads .config.json and every batch at or below the head from lake.path.
• Bootstrap. checkpoint: an empty database makes live mode load the archive's latest checkpoint and start there.
• Status. last_ingested_ledger is ingest_store.latest_ingest_ledger. last_error is the newest log line at level error, fatal or panic, a retry warning whose message starts with Error (Error ingesting data ..., Error fetching ledger ...), or the Error: / panic: line printed on exit. Peak memory is the ingest container's cgroup memory.peak, sampled every second.
• Protocols. Up to 28 for commits on go-stellar-sdk v0.7.3, up to 30 on v0.8.0; another sdk version is an error until it is added.

Read-back joins transactions (hash, ledger), transactions_accounts (participants) and state_changes on to_id, selecting every column so commits with other columns read as they are. Token ids (asset contract ids) on trustline rows are named by recomputing the asset contract id of every asset in trustline_assets and the SAC rows of contract_tokens. Address bytes (AddressBytea, SignerKeyBytea) are decoded as stored, version byte then payload, with go-stellar-sdk's strict strkey decoder; account columns accept G and C, signer columns G, T, X and P. Transaction ids are TOIDs: the operation index is the low 12 bits minus one, operation_id 0 is the transaction's fee row.

wallet-backend's trustline flag bits (bit 0 authorized, bit 5 clawback enabled, bit 6 authorized to maintain liabilities) become core's TrustLineFlags values (1, 4, 2). Two BALANCE rows pair only when the second has the next state_change_id after the first, the reason the pair needs, and the same token and amount. wallet-backend writes the two rows of one token event next to each other (createDebitCreditPair, handleMint, handleDefaultBurnOrClawback in internal/indexer/processors/token_transfer.go) and numbers each stream's rows 1..N per operation in that order (AssignStateChangeOrdinals), so this pairs rows of one event and never joins rows of two events that happen to agree on token and amount, such as a one-sided liquidity-pool leg of a path payment followed by a transfer. State change and participant rows whose transaction has no transactions row come out as unattributed records.

Claims follow from the table: participants of kind account only (wallet-backend does not record event-emitting contracts as participants, and keeps no mux ids), every kind above on the fields it stores, token-event transfer/mint/burn/clawback on event, contract, from, to, amount, to_muxed_id, the fee event of stage before-all-txs on event, contract, from (the stored fee row is net of any refund, so its amount and the refund event are not claimed), and approve. Values wallet-backend does not keep (starting balances, threshold values, account flag values, trustline flag values after authorization changes, executables) are not claimed.

Two claims cover facts that wallet-backend at the current pin does not store in the oracle's form, and they are kept on purpose:

• clawback. wallet-backend's TokenTransferProcessor receives the token transfer processor's Clawback event (handleClawback) and writes it as a BURN row for the issuer plus a DEBIT row for the holder, the same rows a payment to the issuer produces. The oracle, following CAP-67, records a clawback event. The information reaches wallet-backend and its state_change_reason column is an open vocabulary, so the loss happens in wallet-backend's mapping, not in what the ledger or its schema can carry. A wallet that shows this history would show a clawback as a voluntary payment. Each clawback therefore counts as one dropped clawback and one extra burn, which is finding WB-W3.
• to_muxed_id on every arm. CAP-67 puts the destination's mux id or, without one, the transaction memo into a transfer's to_muxed_id, typed as u64, string or bytes. wallet-backend's setToMuxedID keeps only the u64 arm, although its to_muxed_id column is TEXT. A text, hash or return memo is lost, which is finding WB-W4.

The rule for both: a claim says what wallet-backend is expected to record of a fact it ingests and has a place for, judged against the oracle. Claiming only what it stores today would turn these losses into out-of-scope and hide them. The verdict detail and the finding's text name the cause, so a reader can see that the difference is a design choice in wallet-backend and decide whether it matters for their use.

Oracle vocabularyCopy Anchor Link

What the oracle says each transaction changed, read only from LedgerCloseMeta, in <variant>.oracle.json next to each eligible corpus entry. Conventions for every kind:

• Addresses are canonical strkeys: G accounts, C contracts, L liquidity pools, B claimable balances. A mux id never replaces the base address.
• A participant's mux_ids come only from addresses that are muxed in the meta itself (SC_ADDRESS_TYPE_MUXED_ACCOUNT or muxed contract addresses in event topics). A token event's to_muxed_id is kept on the event as typed and never becomes a participant mux id: core writes the transaction memo there too, so a u64 may be a memo id (since version 2).
• Signers keep their own strkey form: G, T (pre-auth), X (hash-x), P (signed payload, with its payload).
• Amounts, balances, limits, ids and sequence numbers are decimal strings; flags, weights, thresholds and ledger numbers are JSON integers.
• Assets are native or CODE:ISSUER; trustline assets may also be pool-share:<L strkey>, the pool id in its CAP-38 L strkey form. Pools are named by the same L strkey everywhere.
• action is created, updated, removed or restored.

The oracle file of an entry is {"oracle_vocab_version", "network_passphrase", "ledgers": [{"ledger_seq", "ledger_hash", "protocol", "close_time", "transactions", "upgrades", "evicted"}]}; each transaction has hash, apply_index, successful, result, inner_hash, fee {source, charged, balance_delta}, participants (actual, from meta and events), changes, declared (what the envelope names, kept apart and never used as the expected participants) and events (which events the meta format can carry, see "Facts the meta cannot show").

Oracle vocabulary version 2 (ORACLE_VOCAB_VERSION in crates/shapekit-oracle/src/vocab.rs). Every change also has scope (fee-processing, tx-before, operation, tx-after, post-apply-fee, upgrade) and, for operation scope, op_index (integer, from 0).

account-createdCopy Anchor Link

account-removedCopy Anchor Link

balance-changeCopy Anchor Link

A change in what an address holds of an asset: account native balance, trustline balance, claimable balance amount, pool reserve, or a Stellar Asset Contract balance entry. holder is the address whose holding changed.

trustline-createdCopy Anchor Link

trustline-removedCopy Anchor Link

trustline-flagsCopy Anchor Link

trustline-limitCopy Anchor Link

signer-addedCopy Anchor Link

signer-removedCopy Anchor Link

signer-weightCopy Anchor Link

Includes the master key, whose weight is threshold 0 of the account.

thresholdsCopy Anchor Link

account-flagsCopy Anchor Link

home-domainCopy Anchor Link

inflation-destinationCopy Anchor Link

sequenceCopy Anchor Link

data-entryCopy Anchor Link

Data entry name and values as hex, since neither has to be UTF-8.

sponsorshipCopy Anchor Link

The sponsor of a ledger entry or of one signer changed (created, revoked, transferred, or the sponsored thing went away). target names the entry or signer.

claimable-balance-createdCopy Anchor Link

claimable-balance-claimedCopy Anchor Link

Removed by a claim. The operation type tells a claim from a clawback, since the entry change itself is the same.

claimable-balance-clawed-backCopy Anchor Link

pool-sharesCopy Anchor Link

A pool-share trustline balance.

liquidity-poolCopy Anchor Link

offerCopy Anchor Link

contract-deployedCopy Anchor Link

A contract instance entry appeared. Read from meta, never from auth entries.

contract-codeCopy Anchor Link

contract-dataCopy Anchor Link

ttlCopy Anchor Link

config-settingCopy Anchor Link

tradeCopy Anchor Link

One offer or pool crossed by a manage-offer or path-payment operation, from the operation result's claim atoms. counterparty is the offer's seller or the pool.

contract-eventCopy Anchor Link

A consensus contract event that is not a token event (for example a protocol's own events, such as Blend's supply or borrow), with topics and data as base64 XDR.

token-eventCopy Anchor Link

A token event (CAP-67 and SEP-41 shapes): transfer, mint, burn, clawback, set_authorized, fee, approve.