Skip to content

perf(api): resolve tx inputs without unpacking whole prev-tx records#1595

Open
Aharonee wants to merge 1 commit into
trezor:masterfrom
pearl-research-labs:perf/tx-input-resolution
Open

perf(api): resolve tx inputs without unpacking whole prev-tx records#1595
Aharonee wants to merge 1 commit into
trezor:masterfrom
pearl-research-labs:perf/tx-input-resolution

Conversation

@Aharonee

Copy link
Copy Markdown

Summary

GetTransactionFromBchainTx — the path behind GET /api/v2/tx/<txid> and address
queries with details=txs — resolves each input's previous output by calling
db.GetTxAddresses(prevTxid), which unpacks the entire TxAddresses record
(every input and every output of the referenced transaction) just to read a single
Outputs[vout]. For transactions with many inputs that reference large previous
transactions, this allocates on the order of inputs × record size, so a single API
response can allocate hundreds of MB of transient heap. On high-fan-in addresses
(pools, treasuries, consolidation/sweep wallets) this drives severe memory
amplification.

This PR makes input resolution read only the one output it needs, taking
allocations from quadratic to linear with no change in behavior or on-disk format.

Root cause

  • The vin loop calls GetTxAddresses(prevTxid) once per input.
  • GetTxAddresses fully deserializes the record: slices + big.Ints for all inputs
    and all outputs of the previous tx — only to read one output.
  • When K inputs reference the same K-output previous tx, cost ≈ K × unpack(K) =
    O(K²) allocations. In general the amplification per input is the number of
    outputs in the previous tx it spends.

Fix

  • Add db.GetTxAddressesOutput(txid, vout): returns just Outputs[vout]. It walks the
    packed record computing the byte length of the height, inputs, and preceding outputs
    (packedTxInputLen / packedTxOutputLen) and skips them without allocating, then
    unpacks only the target output — O(1) allocations regardless of the referenced tx's
    size.
  • Use it in the vin-resolution loop. Resolved value/addresses are identical, and the
    txCache fallback for not-yet-indexed transactions is preserved.
  • Works unchanged with and without -extendedindex.

Benchmark

BenchmarkTxInputResolution (added in this PR) resolves a transaction whose inputs all
spend outputs of one large previous tx, sweeping the input fan-in. Machine:
Apple M4 Pro, darwin/arm64, Go default -benchtime.

Allocated bytes per call (B/op)

fan-in (inputs) before after reduction
8 25,888 17,120 1.5×
32 196,705 57,184 3.4×
128 2,445,679 216,418 11×
512 36,507,509 (≈34.8 MB) 851,698 (≈0.8 MB) 43×
2048 540,268,338 (≈515 MB) 3,379,106 (≈3.2 MB) 160×

Allocations per call (allocs/op)

fan-in before after reduction
8 321 193 1.7×
32 2,673 625 4.3×
128 35,121 2,353 15×
512 533,553 9,265 58×
2048 8,425,529 36,913 228×

Time per call (ns/op)

fan-in before after speedup
8 10,258 8,385 1.2×
32 57,981 30,730 1.9×
128 565,081 136,454 4.1×
512 7,896,200 884,482 8.9×
2048 121,848,819 (≈122 ms) 11,650,475 (≈12 ms) 10×

Reproducing the numbers

# After (this PR):
go test -tags unittest -run x -bench TxInputResolution -benchmem ./api/

# Before (pre-fix): restore the previous input-resolution path, run, then put the fix back.
# api/worker.go is the only production file changed, so reverting it to master recreates
# the old behavior while keeping the benchmark this PR adds.
git checkout master -- api/worker.go     # or: git checkout HEAD~1 -- api/worker.go
go test -tags unittest -run x -bench TxInputResolution -benchmem ./api/
git checkout HEAD -- api/worker.go        # restore the fix

If your environment's RocksDB build isn't linked with a compression library, append
-args -noCompression to the go test commands.

Tests

  • TestGetTxAddressesOutput (both index modes): asserts the single-output read equals
    GetTxAddresses().Outputs[vout] for a record containing a coinbase input and a mix of
    spent/unspent outputs (covers every packedTxInputLen/packedTxOutputLen branch),
    plus out-of-range vout and missing-txid cases.
  • BenchmarkTxInputResolution: input fan-in scaling (above).

Scope / risk

  • Additive and contained: one new exported method + two small non-allocating length
    helpers in db/rocksdb.go, one call site changed in api/worker.go. No on-disk format
    change; no behavioral change to API responses.

GetTransactionFromBchainTx resolved each input's previous output by calling
GetTxAddresses, which unpacks an entire TxAddresses record (all inputs and
outputs) just to read a single output. For transactions with many inputs that
reference large previous transactions this allocates on the order of
inputs x record size, driving heap usage up sharply on high-fan-in addresses.

Add db.GetTxAddressesOutput, which returns a single Outputs[vout] by skipping
over the height, inputs and preceding outputs by length without allocating
them, and use it in the vin-resolution loop. Allocation now scales with the
number of inputs rather than inputs x record size.

Adds TestGetTxAddressesOutput (both index modes) and BenchmarkTxInputResolution
covering the input fan-in scaling.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant