feat(cbor): encoding preservation with Redeemers union redesign - #191
Merged
Conversation
Introduce symbol-based encoding metadata (CBOR.encodingKey) that captures byte-level CBOR encoding details during decode and replays them during encode, achieving bit-perfect round-trip fidelity. Redesign Redeemers as a Credential-style discriminated union: RedeemerMap (CDDL map format) + RedeemerArray (legacy array format), where RedeemerMap stores a proper ReadonlyMap<[tag, index], RedeemerValue> mirroring the Conway CDDL spec exactly.
Contributor
There was a problem hiding this comment.
Pull request overview
Implements CBOR encoding-preservation to enable bit-perfect decode → encode round-trips (preventing txId/signature drift) and redesigns redeemers into a discriminated union so the wire format (map vs array) is represented in the type system.
Changes:
- Add CBOR “preserve” codec mode with per-node encoding metadata (length forms, integer widths, key order, etc.) captured on decode and replayed on encode.
- Redesign redeemers as a union:
RedeemerMap(Conway map) andRedeemerArray(legacy array), updating hashing/builders/witness-set handling accordingly. - Thread encoding metadata through
Transaction,TransactionBody, andTransactionWitnessSettransforms to keep byte-level fidelity through domain transformations.
Reviewed changes
Copilot reviewed 57 out of 57 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/evolution/src/utils/Hash.ts | Updates script-data hashing to use preserve options and new redeemers union constructors. |
| packages/evolution/src/sdk/builders/TxBuilderImpl.ts | Emits witness-set redeemers as RedeemerMap (or undefined) using the new union design. |
| packages/evolution/src/sdk/builders/SignBuilderImpl.ts | Adjusts witness-set assembly to carry redeemers as the new union type without array copying. |
| packages/evolution/src/message-signing/Header.ts | Switches CBOR decode defaults to preserve options for round-trip fidelity. |
| packages/evolution/src/message-signing/CoseSign1.ts | Uses preserve options when serializing COSE Sig_structure and decoding. |
| packages/evolution/src/message-signing/CoseSign.ts | Uses preserve options for COSE Sign / Signature encoding/decoding paths. |
| packages/evolution/src/message-signing/CoseKey.ts | Switches COSE key decoding defaults to preserve options. |
| packages/evolution/src/Withdrawals.ts | Switches CBOR encode/decode defaults to preserve options. |
| packages/evolution/src/VrfCert.ts | Switches CBOR encode/decode defaults to preserve options. |
| packages/evolution/src/VotingProcedures.ts | Switches CBOR encode/decode defaults to preserve options. |
| packages/evolution/src/Value.ts | Switches CBOR encode/decode defaults to preserve options. |
| packages/evolution/src/UnitInterval.ts | Switches CBOR schema defaults to preserve options. |
| packages/evolution/src/TxOut.ts | Switches CBOR encode/decode defaults to preserve options. |
| packages/evolution/src/TransactionWitnessSet.ts | Changes redeemers field to union type; threads CBOR encoding metadata; updates equality/hash/arbitrary generation. |
| packages/evolution/src/TransactionOutput.ts | Switches CBOR encode/decode defaults to preserve options. |
| packages/evolution/src/TransactionMetadatum.ts | Switches CBOR defaults to preserve; replaces hand-rolled equality with schema-derived equivalence. |
| packages/evolution/src/TransactionInput.ts | Switches CBOR encode/decode defaults to preserve options. |
| packages/evolution/src/TransactionBody.ts | Threads encoding metadata between domain and CBOR map representation; switches CBOR defaults to preserve. |
| packages/evolution/src/Transaction.ts | Threads encoding metadata through the outer transaction tuple; adds domain-level addVKeyWitnesses. |
| packages/evolution/src/SingleHostName.ts | Switches CBOR schema defaults to preserve options. |
| packages/evolution/src/SingleHostAddr.ts | Switches CBOR schema defaults to preserve options. |
| packages/evolution/src/ScriptRef.ts | Switches CBOR encode/decode defaults to preserve options. |
| packages/evolution/src/Script.ts | Switches CBOR encode/decode defaults to preserve options. |
| packages/evolution/src/Relay.ts | Switches CBOR schema defaults to preserve options. |
| packages/evolution/src/Redeemers.ts | Introduces RedeemerMap/RedeemerArray union, CDDL transforms for both formats, and updated CBOR helpers/arbitraries. |
| packages/evolution/src/Redeemer.ts | Switches CBOR encode/decode defaults to preserve options. |
| packages/evolution/src/ProtocolVersion.ts | Switches CBOR encode/decode defaults to preserve options. |
| packages/evolution/src/ProtocolParamUpdate.ts | Switches CBOR encode/decode defaults to preserve options. |
| packages/evolution/src/ProposalProcedures.ts | Switches CBOR encode/decode defaults to preserve options. |
| packages/evolution/src/ProposalProcedure.ts | Switches CBOR encode/decode defaults to preserve options. |
| packages/evolution/src/PoolParams.ts | Switches CBOR schema defaults to preserve options. |
| packages/evolution/src/PoolMetadata.ts | Switches CBOR encode/decode defaults to preserve options. |
| packages/evolution/src/OperationalCert.ts | Switches CBOR schema defaults to preserve options. |
| packages/evolution/src/NonnegativeInterval.ts | Switches CBOR schema defaults to preserve options. |
| packages/evolution/src/NativeScriptsOLD.ts | Switches CBOR schema defaults to preserve options. |
| packages/evolution/src/NativeScripts.ts | Switches CBOR schema defaults to preserve options. |
| packages/evolution/src/MultiHostName.ts | Switches CBOR schema defaults to preserve options. |
| packages/evolution/src/MultiAsset.ts | Switches CBOR schema defaults to preserve options. |
| packages/evolution/src/Mint.ts | Switches CBOR encode/decode defaults to preserve options. |
| packages/evolution/src/Metadata.ts | Switches CBOR encode/decode defaults to preserve options. |
| packages/evolution/src/HeaderBody.ts | Switches CBOR schema defaults to preserve options. |
| packages/evolution/src/Header.ts | Switches CBOR schema defaults to preserve options. |
| packages/evolution/src/GovernanceAction.ts | Switches CBOR encode/decode defaults to preserve options. |
| packages/evolution/src/DatumOption.ts | Switches CBOR encode/decode defaults to preserve options. |
| packages/evolution/src/Data.ts | Replaces hand-rolled equality with schema-derived equivalence. |
| packages/evolution/src/DRep.ts | Switches CBOR schema defaults to preserve options. |
| packages/evolution/src/Credential.ts | Switches CBOR schema defaults to preserve options. |
| packages/evolution/src/CostModel.ts | Switches CBOR options used for language-views encoding to preserve options. |
| packages/evolution/src/Constitution.ts | Switches CBOR schema defaults to preserve options. |
| packages/evolution/src/Certificate.ts | Switches CBOR schema defaults to preserve options. |
| packages/evolution/src/CBOR.ts | Adds preserve-mode + encoding metadata capture/replay, defaults to preserve options, and structural CBOR equivalence helper. |
| packages/evolution/src/BootstrapWitness.ts | Switches CBOR defaults to preserve options (including inner CBOR encodes in arbitrary). |
| packages/evolution/src/AuxiliaryData.ts | Switches CBOR encode/decode defaults to preserve options. |
| packages/evolution/src/Assets/index.ts | Switches CBOR encode/decode defaults to preserve options. |
| packages/evolution/src/Anchor.ts | Switches CBOR schema defaults to preserve options. |
| .specs/cbor-encoding-preservation.md | Adds a design/spec document describing encoding metadata capture/replay and threading rules. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+313
to
+337
| // Thread encoding metadata from domain object to CBOR AST with key order guard | ||
| const enc = CBOR.getEncoding(toA) | ||
| if (enc !== undefined) { | ||
| if (enc.keyOrder && enc.keyOrder.length !== record.size) { | ||
| // Key set changed — rebuild keyOrder and entries to match new map | ||
| const newKeyOrder: Array<CBOR.CBOR> = [] | ||
| const newEntries: Array<readonly [CBOR.CBOREncoding | undefined, CBOR.CBOREncoding | undefined]> = [] | ||
| for (let i = 0; i < enc.keyOrder.length; i++) { | ||
| const oldKey = enc.keyOrder[i] | ||
| if (record.has(oldKey as bigint)) { | ||
| newKeyOrder.push(oldKey) | ||
| newEntries.push(enc.entries?.[i] ?? [undefined, undefined]) | ||
| } | ||
| } | ||
| for (const [key] of record) { | ||
| if (!enc.keyOrder.some((k) => CBOR.equals(k, key))) { | ||
| newKeyOrder.push(key) | ||
| newEntries.push([undefined, undefined]) | ||
| } | ||
| } | ||
| CBOR.setEncoding(record, { ...enc, keyOrder: newKeyOrder, entries: newEntries }) | ||
| } else { | ||
| CBOR.setEncoding(record, enc) | ||
| } | ||
| } |
There was a problem hiding this comment.
The key-order rebuild guard only checks enc.keyOrder.length !== record.size. If the key set changes but the size stays the same (e.g. one key removed and another added), the old keyOrder will be reused and can reference keys that no longer exist, leading to incorrect re-encoding (or undefined lookups downstream). Consider also validating that every key in enc.keyOrder exists in record (and that record has no extra keys) before reusing the existing keyOrder/entries.
Comment on lines
+390
to
+416
| // Thread encoding metadata from domain object to CBOR AST | ||
| // with key order + entries rebuild guard for when keys change | ||
| const enc = CBOR.getEncoding(toA) | ||
| if (enc !== undefined) { | ||
| if (enc.keyOrder && enc.keyOrder.length !== record.size) { | ||
| // Key set changed — rebuild keyOrder and entries to match new map | ||
| const newKeyOrder: Array<CBOR.CBOR> = [] | ||
| const newEntries: Array<readonly [CBOR.CBOREncoding | undefined, CBOR.CBOREncoding | undefined]> = [] | ||
| // Preserve original entries for keys that still exist (in original order) | ||
| for (let i = 0; i < enc.keyOrder.length; i++) { | ||
| const oldKey = enc.keyOrder[i] | ||
| if (record.has(oldKey as bigint)) { | ||
| newKeyOrder.push(oldKey) | ||
| newEntries.push(enc.entries?.[i] ?? [undefined, undefined]) | ||
| } | ||
| } | ||
| // Append new keys (those not in original keyOrder) | ||
| for (const [key] of record) { | ||
| if (!enc.keyOrder.some((k) => CBOR.equals(k, key))) { | ||
| newKeyOrder.push(key) | ||
| newEntries.push([undefined, undefined]) | ||
| } | ||
| } | ||
| CBOR.setEncoding(record, { ...enc, keyOrder: newKeyOrder, entries: newEntries }) | ||
| } else { | ||
| CBOR.setEncoding(record, enc) | ||
| } |
There was a problem hiding this comment.
The key-order rebuild guard only checks enc.keyOrder.length !== record.size. If the key set changes but the size stays the same, the old keyOrder can be reused even though it contains keys that are no longer present, which can break preserve-mode re-encoding. Consider validating actual key-set equality (all keys present, no extras) before reusing the previous keyOrder/entries.
Comment on lines
+156
to
+166
| [Equal.symbol](that: unknown): boolean { | ||
| if (!(that instanceof RedeemerMap)) return false | ||
| if (this.value.size !== that.value.size) return false | ||
| const thisArr = this.toArray() | ||
| const thatArr = that.toArray() | ||
| return arrayEquals(thisArr, thatArr) | ||
| } | ||
|
|
||
| [Hash.symbol](): number { | ||
| return Hash.cached(this, arrayHash(this.toArray())) | ||
| } |
There was a problem hiding this comment.
RedeemerMap equality/hash are currently order-sensitive because they compare/hash via toArray() in Map iteration order. Two RedeemerMaps with identical key/value pairs but different insertion order will compare unequal and hash differently. If redeemers are intended to be a mathematical map (like MultiAsset), switch to order-insensitive comparison/hashing based on key/value content (e.g. size + for-each key lookup with structural equality).
Comment on lines
+175
to
+181
| export const makeRedeemerMap = (redeemers: ReadonlyArray<Redeemer.Redeemer>): RedeemerMap => { | ||
| const entries: Array<readonly [RedeemerKey, RedeemerValue]> = redeemers.map((r) => [ | ||
| [r.tag, r.index] as const, | ||
| new RedeemerValue({ data: r.data, exUnits: r.exUnits }) | ||
| ]) | ||
| return new RedeemerMap({ value: new Map(entries) }) | ||
| } |
There was a problem hiding this comment.
makeRedeemerMap builds a Map from an array; if the input array contains duplicate [tag, index] keys, later entries silently overwrite earlier ones. Since duplicates likely indicate an invalid redeemer set, consider detecting duplicates and failing fast (throw / ParseResult error) or documenting the overwrite behavior explicitly.
| * @since 2.0.0 | ||
| * @category constants | ||
| */ | ||
| export const encodingKey: unique symbol = Symbol.for("evolution.cbor.encoding") |
There was a problem hiding this comment.
encodingKey is typed as unique symbol but initialized via Symbol.for(...) (which is typed as symbol). This is likely to fail TypeScript assignability checks. Consider exporting it as symbol (or omit the annotation) while still using Symbol.for to ensure cross-module identity.
Suggested change
| export const encodingKey: unique symbol = Symbol.for("evolution.cbor.encoding") | |
| export const encodingKey = Symbol.for("evolution.cbor.encoding") |
Comment on lines
+846
to
+875
| /** Encode a CBOR header: major type (0-7) + value with specific ByteSize width. */ | ||
| const encodeIntHeader = (majorType: number, value: bigint, byteSize: ByteSize): Uint8Array => { | ||
| const mt = majorType << 5 | ||
| if (byteSize === 0) return new Uint8Array([mt | Number(value)]) | ||
| if (byteSize === 1) return new Uint8Array([mt | 24, Number(value)]) | ||
| if (byteSize === 2) { | ||
| const n = Number(value) | ||
| return new Uint8Array([mt | 25, (n >> 8) & 0xff, n & 0xff]) | ||
| } | ||
| if (byteSize === 4) { | ||
| const n = Number(value) | ||
| return new Uint8Array([mt | 26, (n >> 24) & 0xff, (n >> 16) & 0xff, (n >> 8) & 0xff, n & 0xff]) | ||
| } | ||
| // byteSize === 8 | ||
| const low = Number(value & 0xffffffffn) | ||
| const high = Number(value >> 32n) | ||
| return new Uint8Array([ | ||
| mt | 27, | ||
| (high >> 24) & 0xff, | ||
| (high >> 16) & 0xff, | ||
| (high >> 8) & 0xff, | ||
| high & 0xff, | ||
| (low >> 24) & 0xff, | ||
| (low >> 16) & 0xff, | ||
| (low >> 8) & 0xff, | ||
| low & 0xff | ||
| ]) | ||
| } | ||
|
|
||
| export const internalEncodeSync = (value: CBOR, options: CodecOptions = PRESERVE_OPTIONS, encoding?: CBOREncoding): Uint8Array => { |
There was a problem hiding this comment.
In preserve mode, ByteSize = 0 means “inline” encoding, which is only valid for values < 24. encodeIntHeader is used with byteSize === 0 without validating that value < 24, so mutations that increase an int/length beyond 23 (while preserving old metadata) can produce invalid CBOR (wrong major type / additional info). Add a guard to either (a) upgrade to the minimal width that fits, or (b) ignore the metadata and fall back to default encoding when the value/length no longer fits the recorded width.
Comment on lines
+1457
to
+1460
| // Use keyOrder from encoding to replay original insertion order | ||
| if (encoding?.keyOrder) { | ||
| const pairs: Array<[CBOR, CBOR]> = encoding.keyOrder.map((key) => [key, mapGetCBOR(value, key) as CBOR]) | ||
| return encodeMapEntriesSync(pairs, options, encoding) |
There was a problem hiding this comment.
encodeMapSync replays encoding.keyOrder but forces the value via mapGetCBOR(... ) as CBOR. If the map’s key set changed (even with the same size), mapGetCBOR can return undefined, which then gets encoded as CBOR undefined (or causes downstream issues) instead of falling back. Consider detecting missing keys (any lookup returns undefined) and falling back to Array.from(value.entries()) (or rebuilding keyOrder) rather than casting.
Suggested change
| // Use keyOrder from encoding to replay original insertion order | |
| if (encoding?.keyOrder) { | |
| const pairs: Array<[CBOR, CBOR]> = encoding.keyOrder.map((key) => [key, mapGetCBOR(value, key) as CBOR]) | |
| return encodeMapEntriesSync(pairs, options, encoding) | |
| // Use keyOrder from encoding to replay original insertion order when it still | |
| // matches the current map; otherwise fall back to the current entries. | |
| if (encoding?.keyOrder) { | |
| const pairs: Array<[CBOR, CBOR]> = [] | |
| let missingKey = false | |
| for (const key of encoding.keyOrder) { | |
| const mapped = mapGetCBOR(value, key) | |
| if (mapped === undefined) { | |
| missingKey = true | |
| break | |
| } | |
| pairs.push([key, mapped]) | |
| } | |
| // Only trust keyOrder if every key was found and sizes are consistent | |
| if (!missingKey && pairs.length === encoding.keyOrder.length && pairs.length === value.size) { | |
| return encodeMapEntriesSync(pairs, options, encoding) | |
| } |
… ByteSize overflow, order-insensitive RedeemerMap equality, duplicate key detection
MapCDDLSchema used Schema.Map (Encoded = array-of-tuples) instead of Schema.MapFromSelf (Encoded = JS Map), causing toCBORBytesMap to emit CBOR arrays (major type 4) instead of CBOR maps (major type 5). The witness set correctly wrapped in new Map() producing real CBOR maps, so hashScriptData and witness set bytes diverged — the node rejected transactions with script integrity hash mismatch. - Redeemers.MapCDDLSchema: Schema.Map → Schema.MapFromSelf - TransactionWitnessSet decode: remove stale Map→entries conversion - hashScriptData: accept concrete Redeemers union, dispatch on _tag - TxBuilderImpl: unique fake redeemer indices, pass RedeemerMap directly - TransactionBuilder: remove scriptDataFormat from BuildOptions
Replace .length/[0] array access with .size/.toArray() across tests to match the new RedeemerMap/RedeemerArray discriminated union.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Transaction CBOR round-trips have two failure modes: standard decode→encode re-encodes the body (changing the txId when the original CBOR was non-canonical), and Conway map-format redeemers were silently dropped on decode because only the array format was handled. Both failures mean that re-signing or augmenting a transaction can change it in ways the original signer never authorised.
CBOR.tsgains aCBORFormatdiscriminated union (8 variants) and a parallelWithFormatAPI surface —fromCBORBytesWithFormat,toCBORBytesWithFormat, and hex equivalents — that captures the full encoding tree on decode and replays it exactly on encode without touching intermediate CBOR values.Transaction.addVKeyWitnessesHex/Bytesnow splices witness sets at the byte level, keeping the body bytes untouched. Implicit "preserve mode" and itsPRESERVE_OPTIONSconstant are removed entirely — theWithFormatpath is the only route to byte-identical re-encoding.Closes #174