perf(api): resolve tx inputs without unpacking whole prev-tx records#1595
Open
Aharonee wants to merge 1 commit into
Open
perf(api): resolve tx inputs without unpacking whole prev-tx records#1595Aharonee wants to merge 1 commit into
Aharonee wants to merge 1 commit into
Conversation
GetTransactionFromBchainTx resolved each input's previous output by calling GetTxAddresses, which unpacks an entire TxAddresses record (all inputs and outputs) just to read a single output. For transactions with many inputs that reference large previous transactions this allocates on the order of inputs x record size, driving heap usage up sharply on high-fan-in addresses. Add db.GetTxAddressesOutput, which returns a single Outputs[vout] by skipping over the height, inputs and preceding outputs by length without allocating them, and use it in the vin-resolution loop. Allocation now scales with the number of inputs rather than inputs x record size. Adds TestGetTxAddressesOutput (both index modes) and BenchmarkTxInputResolution covering the input fan-in scaling.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
GetTransactionFromBchainTx— the path behindGET /api/v2/tx/<txid>and addressqueries with
details=txs— resolves each input's previous output by callingdb.GetTxAddresses(prevTxid), which unpacks the entireTxAddressesrecord(every input and every output of the referenced transaction) just to read a single
Outputs[vout]. For transactions with many inputs that reference large previoustransactions, this allocates on the order of
inputs × record size, so a single APIresponse can allocate hundreds of MB of transient heap. On high-fan-in addresses
(pools, treasuries, consolidation/sweep wallets) this drives severe memory
amplification.
This PR makes input resolution read only the one output it needs, taking
allocations from quadratic to linear with no change in behavior or on-disk format.
Root cause
GetTxAddresses(prevTxid)once per input.GetTxAddressesfully deserializes the record: slices +big.Ints for all inputsand all outputs of the previous tx — only to read one output.
Kinputs reference the sameK-output previous tx, cost ≈K × unpack(K)=O(K²) allocations. In general the amplification per input is the number of
outputs in the previous tx it spends.
Fix
db.GetTxAddressesOutput(txid, vout): returns justOutputs[vout]. It walks thepacked record computing the byte length of the height, inputs, and preceding outputs
(
packedTxInputLen/packedTxOutputLen) and skips them without allocating, thenunpacks only the target output — O(1) allocations regardless of the referenced tx's
size.
txCachefallback for not-yet-indexed transactions is preserved.-extendedindex.Benchmark
BenchmarkTxInputResolution(added in this PR) resolves a transaction whose inputs allspend outputs of one large previous tx, sweeping the input fan-in. Machine:
Apple M4 Pro,
darwin/arm64, Go default-benchtime.Allocated bytes per call (
B/op)Allocations per call (
allocs/op)Time per call (
ns/op)Reproducing the numbers
Tests
TestGetTxAddressesOutput(both index modes): asserts the single-output read equalsGetTxAddresses().Outputs[vout]for a record containing a coinbase input and a mix ofspent/unspent outputs (covers every
packedTxInputLen/packedTxOutputLenbranch),plus out-of-range
voutand missing-txid cases.BenchmarkTxInputResolution: input fan-in scaling (above).Scope / risk
helpers in
db/rocksdb.go, one call site changed inapi/worker.go. No on-disk formatchange; no behavioral change to API responses.