diff --git a/cardano-api/src/Cardano/Api/Consensus.hs b/cardano-api/src/Cardano/Api/Consensus.hs index 6353a36a84..b2e709ae50 100644 --- a/cardano-api/src/Cardano/Api/Consensus.hs +++ b/cardano-api/src/Cardano/Api/Consensus.hs @@ -43,6 +43,9 @@ module Cardano.Api.Consensus , reflBlockType , Protocol (..) , ProtocolInfoArgs (..) + , cardanoLedgerTransitionConfig + , byronProtocolParams + , byronGenesis , ProtocolClient (..) , ProtocolClientInfoArgs (..) , nodeSystemStart @@ -79,6 +82,7 @@ module Cardano.Api.Consensus , StandardCrypto , TopLevelConfig , ledgerState + , shelleyLedgerGenesis , blockHash , blockNo , blockSlot diff --git a/cardano-api/src/Cardano/Api/Consensus/Internal/Protocol.hs b/cardano-api/src/Cardano/Api/Consensus/Internal/Protocol.hs index 7c22f91c0b..2a02e63cdb 100644 --- a/cardano-api/src/Cardano/Api/Consensus/Internal/Protocol.hs +++ b/cardano-api/src/Cardano/Api/Consensus/Internal/Protocol.hs @@ -15,6 +15,9 @@ module Cardano.Api.Consensus.Internal.Protocol , reflBlockType , Protocol (..) , ProtocolInfoArgs (..) + , cardanoLedgerTransitionConfig + , byronProtocolParams + , byronGenesis , ProtocolClient (..) , ProtocolClientInfoArgs (..) , nodeSystemStart diff --git a/cardano-api/src/Cardano/Api/Consensus/Internal/Reexport.hs b/cardano-api/src/Cardano/Api/Consensus/Internal/Reexport.hs index 82acf9775d..94521666ae 100644 --- a/cardano-api/src/Cardano/Api/Consensus/Internal/Reexport.hs +++ b/cardano-api/src/Cardano/Api/Consensus/Internal/Reexport.hs @@ -23,6 +23,7 @@ module Cardano.Api.Consensus.Internal.Reexport , StandardCrypto , TopLevelConfig , ledgerState + , shelleyLedgerGenesis , blockHash , blockNo , blockSlot @@ -78,6 +79,7 @@ import Ouroboros.Consensus.Protocol.Praos.Common , PraosProtocolSupportsNodeCrypto , getOpCertCounters ) +import Ouroboros.Consensus.Shelley.Ledger.Ledger (shelleyLedgerGenesis) import Ouroboros.Consensus.Shelley.Node (ShelleyGenesisStaking (..)) import Ouroboros.Consensus.Storage.Common (BlockComponent (..)) import Ouroboros.Consensus.Util.Condense (condense) diff --git a/cardano-api/src/Cardano/Api/LedgerState.hs b/cardano-api/src/Cardano/Api/LedgerState.hs index 5aa05eaefe..18db395416 100644 --- a/cardano-api/src/Cardano/Api/LedgerState.hs +++ b/cardano-api/src/Cardano/Api/LedgerState.hs @@ -85,6 +85,9 @@ module Cardano.Api.LedgerState , ShelleyConfig (..) , GenesisHashShelley (..) , readShelleyGenesisConfig + , readShelleyGenesis + , ShelleyGenesisError (..) + , renderShelleyGenesisError , shelleyPraosNonce -- *** Alonzo Genesis Config diff --git a/cardano-rpc/cardano-rpc.cabal b/cardano-rpc/cardano-rpc.cabal index 119b5f3f5d..44a3db9cf6 100644 --- a/cardano-rpc/cardano-rpc.cabal +++ b/cardano-rpc/cardano-rpc.cabal @@ -70,6 +70,7 @@ library Cardano.Rpc.Server.Internal.UtxoRpc.Type.Byron Cardano.Rpc.Server.Internal.UtxoRpc.Type.Certificate Cardano.Rpc.Server.Internal.UtxoRpc.Type.ChainPoint + Cardano.Rpc.Server.Internal.UtxoRpc.Type.Genesis Cardano.Rpc.Server.Internal.UtxoRpc.Type.Governance Cardano.Rpc.Server.Internal.UtxoRpc.Type.PlutusData Cardano.Rpc.Server.Internal.UtxoRpc.Type.ProtocolParameters @@ -113,6 +114,7 @@ library data-default, errors, filepath, + formatting, generic-data, grapesy, grpc-spec, diff --git a/cardano-rpc/docs/node-kernel-access/01-node-access-types.md b/cardano-rpc/docs/node-kernel-access/01-node-access-types.md new file mode 100644 index 0000000000..25a9c21921 --- /dev/null +++ b/cardano-rpc/docs/node-kernel-access/01-node-access-types.md @@ -0,0 +1,160 @@ +# Piece 1: NodeKernelAccess types and cardano-rpc plumbing + +## Status: as built + +Piece 1 shipped a deliberately thin `NodeKernelAccess`, not the snapshot record described below. +As built, the module is `Cardano.Rpc.Server.NodeKernelAccess`, and the record itself lives in `Cardano.Rpc.Server.NodeKernelAccess.Type`. +The module exports `mkNodeKernelAccess`, `fetchBlock` and `grabNodeKernelAccess`; there is no `withNodeKernelAccess`, no `LedgerSnapshot` and no `nkaWithSnapshot` / `nkaSubmitTx`. +The record has three fields - `chainDb`, `systemStart` and `readEraHistory` - because direct `chainDb` access was enough to serve FetchBlock (pieces 2 and 3), so the snapshot interface was deferred. + +Everything from "## Acceptance criteria" onwards describes the originally-planned snapshot design. +That design is the target that pieces 4-7 grow into as they migrate the query, submit and eval methods off N2C; it is not current code. + +## Problem + +cardano-rpc currently threads `LocalNodeConnectInfo` through its environment and `MonadRpc` constraint. +Every RPC method grabs the connection info and opens a fresh N2C socket connection per request. +To support direct ledger state access (ADR-019), we need a new abstraction that replaces this pattern with an `IORef (Maybe NodeKernelAccess)` passed in by cardano-node at startup. + +## Why + +This piece creates the `NodeKernelAccess` abstraction and wires it through the cardano-rpc infrastructure. +After this piece, cardano-rpc compiles with the new types threaded through, but no new RPC methods exist yet (piece 2) and no node-side implementation exists yet (piece 3). +Separating the plumbing from the method rewrites and node-side implementation keeps each piece small and reviewable. + +## User value + +As a cardano-rpc developer, I want the `NodeKernelAccess` abstraction and environment wiring in place so that I can rewrite individual RPC methods to use node kernel access in subsequent pieces. + +## Acceptance criteria + +These criteria describe the originally-planned snapshot design (see "Status: as built" above), which is the target for pieces 4-7 rather than what piece 1 shipped. + +1. **AC1: NodeKernelAccess module** - A new module `Cardano.Rpc.Server.Internal.NodeKernelAccess` exists at `src/Cardano/Rpc/Server/Internal/NodeKernelAccess.hs`, exporting `NodeKernelAccess(..)`, `LedgerSnapshot(..)`, and `withNodeKernelAccess`. + `NodeKernelAccess` is a record with three fields: `nkaWithSnapshot :: forall a. (LedgerSnapshot -> IO a) -> IO a`, `nkaSubmitTx :: TxInMode -> IO (SubmitResult TxValidationErrorInCardanoMode)`, and `nkaFetchBlock :: SlotNo -> ByteString -> IO (Maybe ByteString)`. + `LedgerSnapshot` is a newtype wrapping `runQuery :: forall result. QueryInMode result -> IO result`. + The module is listed in `exposed-modules` in `cardano-rpc.cabal`. + - Test: unit - compiles and is importable from the test suite + +2. **AC2: withNodeKernelAccess unavailable behaviour** - `withNodeKernelAccess` reads the `IORef (Maybe NodeKernelAccess)`; when the value is `Nothing`, it throws a `GrpcException` with `grpcError = GrpcUnavailable` and message containing "not yet initialised". + When the value is `Just na`, it passes `na` to the callback and returns the callback's result. + - Test: unit - `H.propertyOnce`: create `IORef Nothing`, call `withNodeKernelAccess`, assert `GrpcException` with `GrpcUnavailable` is thrown; create `IORef (Just mockNodeKernelAccess)`, call `withNodeKernelAccess`, assert callback receives the value and its return value is propagated + +3. **AC3: Server.hs signature change** - `runRpcServer` signature changes from `Tracer IO TraceRpc -> (RpcConfig, NetworkMagic) -> IO ()` to `Tracer IO TraceRpc -> RpcConfig -> NetworkMagic -> IORef (Maybe NodeKernelAccess) -> IO ()`. + The module re-exports `NodeKernelAccess(..)` and `LedgerSnapshot(..)`. + `RpcEnv` construction is updated to include both `rpcNodeKernelAccess` (from the new parameter) and `rpcLocalNodeConnectInfo` (preserved temporarily). + Note: `methodsSyncRpc` is NOT registered in this piece - that happens in piece 2 when the SyncService proto and handler exist. + - Test: unit - compiles (API change verified by build) + +4. **AC4: Environment and MonadRpc wiring** - `RpcEnv` in `Env.hs` gains a new field `rpcNodeKernelAccess :: !(IORef (Maybe NodeKernelAccess))`. + A `Has (IORef (Maybe NodeKernelAccess)) RpcEnv` instance is added to `Monad.hs`. + `MonadRpc` constraint includes `Has (IORef (Maybe NodeKernelAccess)) e`. + The old `rpcLocalNodeConnectInfo` field and `Has LocalNodeConnectInfo RpcEnv` instance are kept temporarily so that method files compile unchanged. + - Test: unit - compiles; the new constraint is exercised by `withNodeKernelAccess` usage in the test from AC2 + +5. **AC5: Tracing for new trace types** - `Tracing.hs` gains a `TraceRpcSync` sum type with constructors: `TraceRpcFetchBlockSpan TraceSpanEvent` (span begin/end), `TraceRpcFetchBlockNotFound SlotNo` (block not on chain). + `TraceRpc` gains a `TraceRpcSync TraceRpcSync` constructor. + `Pretty` instances render the span events as "Started fetch block method" / "Finished fetch block method" and the not-found as "Block not found at slot ". + An `Inject TraceRpcSync TraceRpc` instance is provided. + `TraceRpcSubmitN2cConnectionError SomeException` is replaced by `TraceRpcNodeKernelAccessUnavailable` (no payload) and `TraceRpcForkerError String`. + `Pretty TraceRpcSubmit` renders them as `"Ledger access unavailable (node kernel not yet initialised)"` and `"Forker error: "` respectively. + The corresponding one-line update in `Submit.hs` (replacing `Left $ TraceRpcSubmitN2cConnectionError e` with `Left $ TraceRpcNodeKernelAccessUnavailable`) is included so that the build stays clean. + - Test: unit - `H.propertyOnce` asserting the `Pretty` output of each new constructor contains the expected substrings + +## Out of scope + +- Populating the `cardano` oneof field in `AnyChainBlock` (requires protobuf block type mapping, a separate piece of work). +- Streaming RPCs from the sync proto (`FollowTip`, `DumpHistory`). +- Rewriting existing RPC methods (Query, Submit, Eval, Node) to use `NodeKernelAccess` (pieces 4-7). +- Removing `rpcLocalNodeConnectInfo` and `Has LocalNodeConnectInfo` from `RpcEnv` / `MonadRpc` (happens when the last N2C method is rewritten in pieces 4-7). +- Removing `mkLocalNodeConnectInfo` (removed alongside `rpcLocalNodeConnectInfo`). +- Removing `nodeSocketPath` from `RpcConfig` (still needed for `nodeSocketPathToRpcSocketPath`). +- Proto definitions and codegen (piece 2). +- FetchBlock handler (piece 2). +- `mkNodeKernelAccess` in cardano-node (piece 3). +- Node startup wiring (piece 3). +- Adding new E2E tests (no runtime behaviour changes in this piece). + +## Definition of done + +- [ ] All AC tests written (compile, fail on stubs) +- [ ] Implementation complete (all tests pass via `cabal test`) +- [ ] `cabal build cardano-rpc` succeeds from `/work` with no warnings +- [ ] Nix CI checks pass +- [ ] haskell-reviewer agent finds no critical or style issues +- [ ] fourmolu clean (`scripts/devshell/prettify` run on changed files) +- [ ] No build warnings + +## Notes + +### Design decision: keep both fields temporarily + +This piece adds `rpcNodeKernelAccess :: IORef (Maybe NodeKernelAccess)` to `RpcEnv` alongside the existing `rpcLocalNodeConnectInfo :: LocalNodeConnectInfo`. +Removing `rpcLocalNodeConnectInfo` would break every existing method file (`Node.hs`, `Query.hs`, `Submit.hs`, `Eval.hs`) because they all use `nodeConnInfo <- grab` to obtain a `LocalNodeConnectInfo`. +Rewriting those method bodies is the work of pieces 4-7. + +Both fields coexist in `RpcEnv` and both `Has` instances exist in `MonadRpc`. +This means: +- Existing method files compile without any changes. +- Runtime behaviour of existing methods is unchanged (they still use N2C). +- Pieces 4-7 each rewrite one method's N2C usage; the last piece to land removes the old field and instance. + +### Files affected + +| File | Change | +|---|---| +| `src/Cardano/Rpc/Server/Internal/NodeKernelAccess.hs` | **New.** `NodeKernelAccess`, `LedgerSnapshot`, `withNodeKernelAccess`. | +| `src/Cardano/Rpc/Server/Internal/Env.hs` | Add `rpcNodeKernelAccess` field alongside existing `rpcLocalNodeConnectInfo`. | +| `src/Cardano/Rpc/Server/Internal/Monad.hs` | Add `Has (IORef (Maybe NodeKernelAccess)) RpcEnv` instance. Add constraint to `MonadRpc`. | +| `src/Cardano/Rpc/Server/Internal/Tracing.hs` | Add `TraceRpcSync` type and constructors. Replace `TraceRpcSubmitN2cConnectionError` with `TraceRpcNodeKernelAccessUnavailable` and `TraceRpcForkerError`. | +| `src/Cardano/Rpc/Server/Internal/UtxoRpc/Submit.hs` | One-line trace constructor update. | +| `src/Cardano/Rpc/Server.hs` | New signature, re-exports, updated `RpcEnv` construction. | +| `cardano-rpc.cabal` | Add `NodeKernelAccess` module to `exposed-modules`. | + +**cardano-node** (must update in lockstep to keep `-Werror` clean): + +| File | Change | +|---|---| +| `src/Cardano/Node/Tracing/Tracers/Rpc.hs` | Handle renamed `TraceRpcNodeKernelAccessUnavailable`/`TraceRpcForkerError` and new `TraceRpcSync` constructors in `forMachine`, `asMetrics`, `namespaceFor`, `severityFor`, `documentFor`, `allNamespaces`. | +| `src/Cardano/Node/Run.hs` | Create `nodeKernelAccessRef <- newIORef Nothing`, pass through `rpcServerLoop` to `runRpcServer`. Update `rpcServerLoop` signature. | + +### Gotchas for the implementer + +- **Import narrowing in `Monad.hs`**: when adding the new `Has` instance, ensure `Inject` (used by `putTrace`) is still available. + Currently it comes from `import Cardano.Api`; if imports are narrowed, import it explicitly from `Cardano.Api.Era`. + +- **`RankNTypes` extension.** Both `NodeKernelAccess` and `LedgerSnapshot` use higher-rank fields, requiring the `RankNTypes` extension in `NodeKernelAccess.hs`. + +- **`runRpcServer` keeps `NetworkMagic`.** The old `rpcLocalNodeConnectInfo` is still used by existing methods, so `mkLocalNodeConnectInfo` still needs `NetworkMagic`. + It is dropped only when `rpcLocalNodeConnectInfo` is finally removed in a later piece. + +- **`Submit.hs` trace constructor.** `Submit.hs` currently references `TraceRpcSubmitN2cConnectionError` in its `submitTx` helper. + The trace constructor rename requires a corresponding one-line update in `Submit.hs`: replace `Left $ TraceRpcSubmitN2cConnectionError e` with `Left $ TraceRpcNodeKernelAccessUnavailable` (dropping the exception payload, since the new constructor carries no payload). + +- **`SomeException` import**: `Control.Exception` is still needed in `Tracing.hs` because `TraceRpcError` and `TraceRpcFatalError` use `SomeException`. + +- **`GrpcException` import**: `withNodeKernelAccess` throws `GrpcException` from `Network.GRPC.Spec`. + `grpc-spec` is already a dependency of `cardano-rpc`. + +- **`RpcConfig.nodeSocketPath` stays**: ADR-019 explicitly notes this. + The config field remains for deriving `rpcSocketPath` via `nodeSocketPathToRpcSocketPath`. + +### Dependencies + +- **Upstream:** none (this is the first piece). +- **Downstream:** all pieces 2-8 depend on this (for the `NodeKernelAccess` record, environment wiring, and tracing). + +### Testing approach + +This piece is primarily a wiring/structural change. +Two ACs have genuine Hedgehog property tests: +- AC2 (`withNodeKernelAccess` behaviour): `H.propertyOnce` covering the `Nothing` and `Just` branches. +- AC5 (tracing pretty-print): `H.propertyOnce` asserting rendered output of the new constructors. + +AC1, AC3, AC4 are verified by successful compilation. + +## Reference docs + +- [Consensus protocol and snapshots](analysis-consensus-protocol.md) - snapshot consistency rationale +- [API signatures](prereqs-api-signatures.md) - `NodeKernelAccess` type design context +- [Implementation details](prereqs-implementation-details.md) - subtle gotchas for the interface diff --git a/cardano-rpc/docs/node-kernel-access/02-fetchblock-proto-and-handler.md b/cardano-rpc/docs/node-kernel-access/02-fetchblock-proto-and-handler.md new file mode 100644 index 0000000000..5c393d307e --- /dev/null +++ b/cardano-rpc/docs/node-kernel-access/02-fetchblock-proto-and-handler.md @@ -0,0 +1,134 @@ +# Piece 2: FetchBlock proto and handler + +## Problem + +cardano-rpc has no `FetchBlock` implementation and no direct ChainDB access path. +Before rewriting existing N2C-based RPCs, we need a clean end-to-end proof that direct node access works, using a new method with no legacy code to break. + +## Why + +This piece adds the SyncService proto definition and the `fetchBlockMethod` handler. +After this piece, the proto bindings exist, the handler compiles, but it cannot run end-to-end yet (needs piece 3 for node-side implementation). +Starting with a new RPC method de-risks the node kernel access architecture without touching any existing N2C code. + +## User value + +As a dApp developer, I want to fetch raw block bytes by slot and hash via gRPC so that I can decode blocks client-side without running a separate chain indexer. + +## Acceptance criteria + +1. **AC1: sync.proto and codegen** - A new proto file `cardano-rpc/proto/utxorpc/v1beta/sync/sync.proto` exists, containing the `SyncService` with only the `FetchBlock` RPC and its message types (`BlockRef`, `FetchBlockRequest`, `FetchBlockResponse`, `AnyChainBlock`). + Running `buf generate proto` in the nix dev shell produces proto-lens bindings under `gen/` (`Proto.Utxorpc.V1beta.Sync.Sync` and `Proto.Utxorpc.V1beta.Sync.Sync_Fields`). + Both generated modules are listed in `cardano-rpc.cabal` under `library gen`. + - Test: unit - `cabal build cardano-rpc` compiles the generated modules + +2. **AC2: Proto API wrapper for SyncService** - A new module `Cardano.Rpc.Proto.Api.UtxoRpc.Sync` exists, following the same pattern as `Query.hs` and `Submit.hs` (re-exports generated proto modules, declares `RequestMetadata`, `ResponseInitialMetadata`, `ResponseTrailingMetadata` type instances for `Protobuf SyncService`). + The module is listed in `exposed-modules` in `cardano-rpc.cabal`. + - Test: unit - compiles + +3. **AC3: fetchBlockMethod implementation** - A new module `Cardano.Rpc.Server.Internal.UtxoRpc.Sync` exists with `fetchBlockMethod :: FetchBlockRequest -> RpcHandler FetchBlockResponse`. + For each `BlockRef` in the request, the method extracts `slot` and `hash` fields. + It calls `nkaFetchBlock slotNo hashBytes` via `withNodeKernelAccess`. + If the result is `Just rawBytes`, the block is wrapped in an `AnyChainBlock` with `native_bytes` set to `rawBytes`. + If the result is `Nothing`, the block is omitted from the response (not-found blocks are skipped silently) and a `TraceRpcFetchBlockNotFound` trace is emitted with the slot number. + Only the `native_bytes` field is populated; the `cardano` oneof field is left empty. + - Test: E2E - `hprop_rpc_fetch_block` verifies a produced block can be fetched and its raw bytes are non-empty + +4. **AC4: Missing slot returns INVALID_ARGUMENT** - When a `BlockRef` has `slot == 0` (the proto default, meaning unset) but a non-empty `hash`, `fetchBlockMethod` returns a gRPC `INVALID_ARGUMENT` error with a message containing "slot is required". + A `RealPoint` cannot be constructed without both slot and hash; the method validates this before calling `nkaFetchBlock`. + - Test: unit - `H.propertyOnce`: construct a `FetchBlockRequest` with a `BlockRef` whose slot is 0 and hash is non-empty, call `fetchBlockMethod`, assert `GrpcException` with `GrpcInvalidArgument` is thrown + +5. **AC5: Empty hash returns INVALID_ARGUMENT** - When a `BlockRef` has a non-zero `slot` but an empty `hash`, `fetchBlockMethod` returns a gRPC `INVALID_ARGUMENT` error with a message containing "hash is required". + - Test: unit - `H.propertyOnce`: construct a `FetchBlockRequest` with a `BlockRef` whose slot is non-zero and hash is empty, call `fetchBlockMethod`, assert `GrpcException` with `GrpcInvalidArgument` is thrown + +6. **AC6: Server.hs registers SyncService** - `Server.hs` registers `methodsSyncRpc` alongside `methodsNodeRpc`, `methodsUtxoRpc`, and `methodsUtxoRpcSubmit`. + - Test: unit - compiles (verified by build) + +## Out of scope + +- Populating the `cardano` oneof field in `AnyChainBlock` (requires protobuf block type mapping, a separate piece of work). +- Streaming RPCs from the sync proto (`FollowTip`, `DumpHistory`). +- `mkNodeKernelAccess` in cardano-node (piece 3). +- Node startup wiring in `Run.hs` (piece 3). +- E2E test infrastructure (piece 3 provides the node-side implementation needed to run FetchBlock end-to-end). +- `field_mask` support on `FetchBlockRequest` (parse-into-proto would need the `cardano` field populated first). +- Handling of `BlockRef.height` and `BlockRef.timestamp` fields (not needed for `RealPoint` construction; reserved for future use). + +## Definition of done + +- [ ] All AC tests written (compile, fail on stubs) +- [ ] Implementation complete (all tests pass via `cabal test`) +- [ ] `cabal build cardano-rpc` succeeds from `/work` with no warnings +- [ ] Nix CI checks pass +- [ ] haskell-reviewer agent finds no critical or style issues +- [ ] fourmolu clean (`scripts/devshell/prettify` run on changed files) +- [ ] No build warnings + +## Notes + +### Design decision: skip not-found blocks (not NOT_FOUND status) + +The proto `FetchBlockResponse` has `repeated AnyChainBlock block` with no per-element status field. +A missing block can only be expressed by omission from the list or by failing the entire request with a gRPC NOT_FOUND status. +Failing the entire batch because one block is missing would be surprising and unhelpful for clients requesting multiple blocks. +Therefore, blocks not found in ChainDB are silently omitted from the response. +Clients can compare the count of returned blocks against the count of requested `BlockRef` entries to detect missing blocks. + +### Files affected + +| File | Change | +|---|---| +| `proto/utxorpc/v1beta/sync/sync.proto` | **New.** FetchBlock RPC and message types. | +| `gen/Proto/Utxorpc/V1beta/Sync/Sync.hs` | **Generated.** Proto-lens bindings (do not edit manually). | +| `gen/Proto/Utxorpc/V1beta/Sync/Sync_Fields.hs` | **Generated.** Proto-lens field accessors (do not edit manually). | +| `src/Cardano/Rpc/Proto/Api/UtxoRpc/Sync.hs` | **New.** Proto API wrapper for SyncService. | +| `src/Cardano/Rpc/Server/Internal/UtxoRpc/Sync.hs` | **New.** `fetchBlockMethod`. | +| `src/Cardano/Rpc/Server.hs` | Register `methodsSyncRpc`. | +| `cardano-rpc.cabal` | Add new modules to `exposed-modules` and `library gen`. | + +### Gotchas for the implementer + +- **Proto codegen requires nix dev shell.** `buf` is not available outside it. + Run `nix develop --command bash -c "cd cardano-rpc && buf generate proto"`. + +- **`AnyChainBlock` has both `native_bytes` and `cardano`.** For this piece, only populate `native_bytes`. + The parsed `cardano` block field is a separate, complex piece of work. + +- **`GetRawBlock` returns `Lazy.ByteString`.** The proto `native_bytes` field is strict `ByteString`. + Convert via `Data.ByteString.Lazy.toStrict`. + +- **Hash bytes conversion.** The proto hash is raw bytes (`ByteString`). + `HeaderHash (CardanoBlock StandardCrypto)` is `OneEraHash` wrapping `ShortByteString`. + Convert via `OneEraHash . SBS.toShort . BS.toStrict` (if the input is lazy) or `OneEraHash . SBS.toShort` (if strict). + No CBOR wrapping is needed; it is raw hash bytes. + +- **`RealPoint` requires both slot and hash.** If the proto `BlockRef` has `slot == 0` (proto default for unset) but a non-empty hash, or a non-zero slot but empty hash, we cannot construct a `RealPoint`. + Validate both fields and return `INVALID_ARGUMENT` if either is missing. + +### Dependencies + +- **Upstream:** piece 1 (for `NodeKernelAccess`, `withNodeKernelAccess`, environment wiring, and tracing types). +- **Downstream:** piece 3 (provides the node-side `mkNodeKernelAccess` implementation needed for E2E). + +### Testing approach + +| AC | Type | What it tests | +|---|---|---| +| AC1 | unit | Proto codegen compiles | +| AC2 | unit | Proto API wrapper compiles | +| AC3 | E2E | `fetchBlockMethod` end-to-end happy path (requires piece 3) | +| AC4 | unit | Missing slot returns `INVALID_ARGUMENT` | +| AC5 | unit | Empty hash returns `INVALID_ARGUMENT` | +| AC6 | unit | Server registration compiles | + +### Open questions + +- Should `nkaFetchBlock` also support fetching by hash alone (without slot)? + `ChainDB` has `getBlockComponent` which takes a `RealPoint` (requiring both), so hash-only lookup would need a different API (`iteratorNext` or similar). + For now, both slot and hash are required. + Hash-only lookup can be a follow-up story if needed. + +## Reference docs + +- [Architecture and current state](analysis-architecture.md) - cardano-rpc overview and spec coverage +- [Build and conventions](prereqs-build-and-conventions.md) - proto codegen instructions, nix build commands diff --git a/cardano-rpc/docs/node-kernel-access/03-mk-node-access-and-wiring.md b/cardano-rpc/docs/node-kernel-access/03-mk-node-access-and-wiring.md new file mode 100644 index 0000000000..7e0d3d18ed --- /dev/null +++ b/cardano-rpc/docs/node-kernel-access/03-mk-node-access-and-wiring.md @@ -0,0 +1,169 @@ +# Piece 3: mkNodeKernelAccess and node wiring + +## Problem + +Pieces 1 and 2 define the `NodeKernelAccess` interface in cardano-rpc and add the FetchBlock handler, but no concrete implementation exists yet. +The `IORef (Maybe NodeKernelAccess)` is always `Nothing` at runtime, so every gRPC request returns `UNAVAILABLE`. +This piece provides the real implementation by constructing `NodeKernelAccess` from `NodeKernel` internals and wiring it into the node startup sequence. + +## Why + +Without this piece, the node kernel access migration is incomplete: cardano-rpc has a working interface with no backing implementation. +This is the single piece that makes the entire N2C-to-direct-access transition functional at runtime. +After this piece, FetchBlock works end-to-end. + +## User value + +As a cardano-node operator, I want the gRPC server to query ledger state directly through the node kernel so that queries are faster and do not require a separate N2C socket connection. + +## Acceptance criteria + +1. **AC1: New module exists** - A new module `Cardano.Node.Rpc.NodeKernelAccess` exists at `cardano-node/cardano-node/src/Cardano/Node/Rpc/NodeKernelAccess.hs` and exports `mkNodeKernelAccess`. + - Test: unit - `cabal build cardano-node` compiles with the new module + +2. **AC2: mkNodeKernelAccess signature** - `mkNodeKernelAccess` has the signature `NodeKernel IO RemoteAddress LocalConnectionId (CardanoBlock StandardCrypto) -> NodeKernelAccess`. + - Test: unit - compiles with correct type; incorrect type would cause a build failure + +3. **AC3: nkaFetchBlock implementation** - `nkaFetchBlock` converts the hash bytes to `HeaderHash (CardanoBlock StandardCrypto)` (via `OneEraHash` wrapping `ShortByteString`), constructs a `RealPoint`, calls `ChainDB.getBlockComponent GetRawBlock`, and returns the result as strict `ByteString` (converted from lazy via `BS.toStrict`). + - Test: E2E - `hprop_rpc_fetch_block` verifies a produced block can be fetched + +4. **AC4: Snapshot acquisition with bracket** - `nkaWithSnapshot` acquires a `ReadOnlyForker` via `getReadOnlyForkerAtPoint chainDB VolatileTip` inside `withRegistry`, and uses `bracket` to ensure `roforkerClose` is called even when the callback throws an exception. + - Test: E2E - existing `hprop_rpc_query_pparams` exercises the snapshot path end-to-end + +5. **AC5: Forker error handling** - When `getReadOnlyForkerAtPoint` returns `Left err`, `nkaWithSnapshot` throws a descriptive error containing the stringified `GetForkerError`. + - Test: unit - compiles; this is a defensive guard whose `Left` path is not exercised by E2E tests (`VolatileTip` is always available in a running testnet) + +6. **AC6: Query round-trip** - `LedgerSnapshot.runQuery` converts a `QueryInMode` to a consensus `Query` via `toConsensusQuery`, calls `answerQuery` with the forker and `ExtLedgerCfg`, then converts the result back via `fromConsensusQueryResult`. + - Test: E2E - `hprop_rpc_query_pparams` verifies the full query round-trip returns valid protocol parameters + +7. **AC7: Transaction submission** - `nkaSubmitTx` converts a `TxInMode` to a consensus `GenTx` via `toConsensusGenTx`, submits it through `addLocalTxs` using the `MkSolo` constructor (GHC 9.10), and maps `MempoolTxAdded` to `SubmitSuccess` and `MempoolTxRejected` to `SubmitFail` (converting the error via `fromConsensusApplyTxErr`). + - Test: E2E - `hprop_rpc_transaction` submits a transaction and verifies success + +8. **AC8: IORef wiring in Run.hs** - `Run.hs` creates `nodeKernelAccessRef <- newIORef Nothing` before the RPC server `withAsync`, passes it to `rpcServerLoop`, and in `rnNodeKernelHook` calls `writeIORef nodeKernelAccessRef (Just (mkNodeKernelAccess nodeKernel))`. + A comment at the `writeIORef` site documents the single-writer/many-readers safety invariant. + `rpcServerLoop` calls `runRpcServer` with the new signature, threading through the `IORef`. + - Test: E2E - any successful gRPC request in the testnet suite proves the IORef was populated + +9. **AC9: Tracing - LogFormatting and MetaTrace** - In `Cardano.Node.Tracing.Tracers.Rpc`, the `LogFormatting` instance handles `TraceRpcSync` constructors. + The `MetaTrace TraceRpc` instance is updated: `namespaceFor` maps the new constructors, `severityFor` assigns appropriate severity, `documentFor` provides descriptions, and `allNamespaces` lists the new namespaces. + Note: the initial `TraceRpcSync` constructors and the renamed submit constructors are already added in piece 1 (which must update this file in lockstep with `Tracing.hs`). + This AC covers any additional trace handling needed for `mkNodeKernelAccess`-specific paths. + - Test: unit - compiles; `-Wincomplete-patterns` (via `-Werror`) catches missing branches + +10. **AC10: E2E test for FetchBlock** - A new E2E test `hprop_rpc_fetch_block` starts a testnet, produces at least one block, fetches it via the `FetchBlock` gRPC method using the block's slot and hash, and verifies the response contains an `AnyChainBlock` whose `native_bytes` is non-empty. + The test is added to the testnet test runner alongside the existing RPC tests. + - Test: E2E - `TASTY_PATTERN='/RPC FetchBlock/' cabal test cardano-testnet-test` + +11. **AC11: Build clean and existing tests pass** - `cabal build cardano-rpc` and `cabal build cardano-node` both succeed with no errors or warnings. + `cabal test cardano-rpc:test:cardano-rpc-test` passes with no failures. + Existing E2E tests (`hprop_rpc_query_pparams`, `hprop_rpc_transaction`, `hprop_rpc_search_utxos`) pass without modification. + - Test: E2E - full CI pass + +12. **AC12: Cabal module listing** - `cardano-node.cabal` lists `Cardano.Node.Rpc.NodeKernelAccess` in `exposed-modules`. + - Test: unit - `cabal build cardano-node` would fail if the module is missing from the listing + +## Out of scope + +- All cardano-rpc-side type and interface changes (pieces 1 and 2 define `NodeKernelAccess`, `LedgerSnapshot`, `withNodeKernelAccess`, `fetchBlockMethod`). +- Rewriting existing RPC method bodies (pieces 4-7). +- New integration test coverage beyond `hprop_rpc_fetch_block` or conformance tests (piece 8). +- Dynamic era handling in `getEraMethod` (remains hardcoded to Conway). +- Removal of `nodeSocketPath` from `RpcConfig` (kept for backward compatibility with testnet and POM configuration). +- Performance benchmarking of direct access versus N2C. +- Streaming or ChainSync support. + +## Definition of done + +- [ ] All AC tests written (compile, fail on stubs) +- [ ] Implementation complete (all tests pass) +- [ ] `cabal build cardano-node` succeeds with no warnings +- [ ] `cabal build cardano-rpc` succeeds with no warnings +- [ ] E2E tests pass: `cabal test cardano-testnet-test --test-option='-p /RPC/'` +- [ ] Nix CI checks pass +- [ ] haskell-reviewer agent finds no critical or style issues +- [ ] fourmolu clean +- [ ] No build warnings + +## Notes + +### Key imports for mkNodeKernelAccess + +- `Cardano.Api.Query.Internal.Type.QueryInMode (toConsensusQuery, fromConsensusQueryResult)` +- `Cardano.Api.Consensus.Internal.InMode (toConsensusGenTx, fromConsensusApplyTxErr)` +- `Ouroboros.Consensus.Ledger.Query (answerQuery)` +- `Ouroboros.Consensus.Storage.ChainDB (getReadOnlyForkerAtPoint)` +- `Control.ResourceRegistry (withRegistry)` +- `Ouroboros.Consensus.Mempool.API (addLocalTxs, MempoolAddTxResult (..))` + +### Files affected + +**cardano-node:** + +| File | Change | +|---|---| +| `src/Cardano/Node/Rpc/NodeKernelAccess.hs` | **New.** `mkNodeKernelAccess` from `NodeKernel`. | +| `src/Cardano/Node/Run.hs` | IORef creation, kernel hook population, `rpcServerLoop` signature update. | +| `src/Cardano/Node/Tracing/Tracers/Rpc.hs` | `LogFormatting` and `MetaTrace` instances for `TraceRpcSync`. | +| `cardano-node.cabal` | Add `Cardano.Node.Rpc.NodeKernelAccess` to `exposed-modules`. | + +**cardano-testnet:** + +| File | Change | +|---|---| +| `test/cardano-testnet-test/Cardano/Testnet/Test/Rpc/FetchBlock.hs` | **New.** `hprop_rpc_fetch_block` E2E test. | +| `test/cardano-testnet-test/cardano-testnet-test.hs` | Register new test. | + +### Gotchas for the implementer + +- **`ReadOnlyForker` must be closed even on exception;** use `bracket` with `roforkerClose` inside `withRegistry`. + +- **`getReadOnlyForkerAtPoint` can return `Left GetForkerError`** if the point is not on chain or too old; throw a descriptive error. + +- **`MkSolo` constructor.** On GHC 9.10, use `MkSolo` (not `Solo`) from `Data.Tuple` when calling `addLocalTxs`. + +- **`toConsensusQuery` returns `Some (Query (CardanoBlock c))`;** pattern match with `Some`. + +- **`fromConsensusQueryResult` for era-specific queries returns `Either EraMismatch result`;** the `runQuery` implementation must handle the `Left` case. + +- **`RealPoint` requires both slot and hash.** The `nkaFetchBlock` implementation must construct `RealPoint` from the slot and hash provided by the handler. + +- **`rpcServerLoop` currently calls `runRpcServer rpcTracer (config, networkMagic)`.** The signature change adds the `IORef` parameter and uncurries the tuple (pieces 1 and 2 handle the cardano-rpc side of this change). + +- **The `rpcServerLoop` signature change touches a function that also handles SIGHUP reconfiguration.** Care needed not to break the config reload path. + +### Risks + +| Risk | Status | Mitigation | +|------|--------|------------| +| `answerQuery` API differs from expected signature | **Verified** (2026-05-22) | Signature matches: `ExtLedgerCfg blk -> ReadOnlyForker' m blk -> Query blk result -> m result` | +| Forker lifecycle - leaking forkers on exceptions | **Addressed** | `bracket` pattern in `nkaWithSnapshot`; `withRegistry` provides additional safety net | +| `addLocalTxs` API changed in recent consensus | **Verified** (2026-05-22) | Name and signature confirmed; use `MkSolo` constructor on GHC 9.10 | +| Concurrent reads during kernel initialisation race | Low risk | `IORef` write in `rnNodeKernelHook` happens before any RPC can succeed; `withNodeKernelAccess` checks atomically | + +### Dependencies + +- **Upstream:** pieces 1 and 2 (for `NodeKernelAccess` types, environment wiring, proto definitions, and handler). +- **Downstream:** pieces 4-8 (method rewrites and integration testing). + +### Testing approach + +| AC | Type | What it tests | +|---|---|---| +| AC1 | unit | New module compiles | +| AC2 | unit | `mkNodeKernelAccess` type signature | +| AC3 | E2E | `nkaFetchBlock` via `hprop_rpc_fetch_block` | +| AC4 | E2E | Snapshot path via `hprop_rpc_query_pparams` | +| AC5 | unit | Defensive forker error guard (compiles) | +| AC6 | E2E | Full query round-trip | +| AC7 | E2E | Transaction submission via `hprop_rpc_transaction` | +| AC8 | E2E | IORef wiring (proven by any successful gRPC request) | +| AC9 | unit | Tracing instances compile | +| AC10 | E2E | `hprop_rpc_fetch_block` dedicated integration test | +| AC11 | E2E | Full regression (all existing tests pass) | +| AC12 | unit | Cabal listing (compiles) | + +## Reference docs + +- [API signatures](prereqs-api-signatures.md) - `answerQuery`, `getReadOnlyForkerAtPoint`, `toConsensusQuery` signatures +- [UTxO-HD internals](analysis-utxohd-internals.md) - forker lifecycle, backing store mechanics +- [Build and conventions](prereqs-build-and-conventions.md) - nix build commands for cardano-node diff --git a/cardano-rpc/docs/node-kernel-access/04-rewrite-query-methods.md b/cardano-rpc/docs/node-kernel-access/04-rewrite-query-methods.md new file mode 100644 index 0000000000..f54356d643 --- /dev/null +++ b/cardano-rpc/docs/node-kernel-access/04-rewrite-query-methods.md @@ -0,0 +1,86 @@ +# Piece 4: Rewrite query methods to use NodeKernelAccess + +## Problem + +The three query methods in `Query.hs` (`readParamsMethod`, `readUtxosMethod`, `searchUtxosMethod`) use the Node-to-Client IPC pattern (`executeLocalStateQueryExpr` via `LocalNodeConnectInfo`). +This creates a new socket connection per request and double-serialises data through CBOR. +These methods need to be rewritten to use the snapshot-based `NodeKernelAccess` interface for direct ledger state access. + +## Why + +Eliminating the N2C round-trip reduces latency and resource overhead for every query request. +Using a single `nkaWithSnapshot` call per request preserves the consistency guarantee (all queries see the same ledger state) while removing the socket and serialisation costs. + +## User value + +As a dApp developer querying protocol parameters or UTxOs via gRPC, I want responses served directly from the ledger state so that queries are faster and I do not pay the overhead of a Node-to-Client IPC round-trip. + +## Acceptance criteria + +1. **AC1: readParamsMethod uses NodeKernelAccess** - `readParamsMethod` acquires a `LedgerSnapshot` via `withNodeKernelAccess` and `nkaWithSnapshot`, then runs all queries (`QueryCurrentEra`, `QueryProtocolParameters`, `QuerySystemStart`, `QueryEraHistory`, `QueryChainPoint`, `QueryChainBlockNo`) against that snapshot. + No references to `executeLocalStateQueryExpr`, `determineEra`, or `VolatileTip` remain in this method. + - Test: E2E - `hprop_rpc_query_pparams` passes, validating all 44 protocol parameter fields match the ledger and the ledger tip (slot, hash, height) is correct. + +2. **AC2: readUtxosMethod uses NodeKernelAccess** - `readUtxosMethod` follows the same snapshot pattern as AC1, querying UTxOs via `QueryUTxO` inside `nkaWithSnapshot`. + The `txoRefToTxIn` helper and protobuf conversion logic remain unchanged. + - Test: E2E - `hprop_rpc_query_pparams` readUtxos assertion passes (UTxO set matches epoch state view). + +3. **AC3: searchUtxosMethod uses NodeKernelAccess** - `searchUtxosMethod` follows the same snapshot pattern, with post-query filtering and pagination logic unchanged. + - Test: E2E - `hprop_rpc_search_utxos` passes (exact address, payment credential, empty, and whole-set predicates all return correct results). + +4. **AC4: Snapshot consistency** - Each query method opens exactly one `nkaWithSnapshot` call, and all queries within that call (era detection, domain query, systemStart, eraHistory, chainPoint, blockNo) execute against the same `LedgerSnapshot`. + This preserves the consistency guarantee of the previous `executeLocalStateQueryExpr` pattern. + - Test: unit - code review; verified structurally by the single `nkaWithSnapshot` callback per method. A Haddock note on each method documents this invariant. + +5. **AC5: systemStart and eraHistory inside snapshot** - Every query method queries `QuerySystemStart` and `QueryEraHistory` inside the snapshot callback (not outside it), so that `slotToTimestamp` receives values consistent with the chain point. + - Test: unit - compile-time verification; if either query is moved outside the snapshot callback, the types enforce that the values are not available. Code review confirms placement. + +6. **AC6: Unused N2C imports removed** - The following symbols are no longer imported in `Query.hs`: `throwExceptT`, `executeLocalStateQueryExpr`, `queryProtocolParameters`, `queryChainPoint`, `queryChainBlockNo`, `queryUtxo`, `VolatileTip`. + `Cardano.Rpc.Server.Internal.NodeKernelAccess` is added as an import. + - Test: unit - `-Wall` clean build with no unused-import warnings. + +7. **AC7: Pagination unit tests unaffected** - The `paginateByTxIn` function and its six unit tests in `Test.Cardano.Rpc.Pagination` remain unchanged and pass. + - Test: unit - `cabal test cardano-rpc-test` passes with all pagination properties green. + +8. **AC8: Build clean** - `cabal build cardano-rpc` compiles with no errors or warnings. + - Test: unit - successful build under `-Wall -Werror` (or project-level warning settings). + +## Out of scope + +- Rewriting `evalTxMethod` in `Eval.hs` (separate piece). +- Rewriting `submitTxMethod` in `Submit.hs` (separate piece). +- Rewriting `getProtocolParamsJsonMethod` in `Node.hs` (separate piece). +- Changes to `Env.hs`, `Monad.hs`, or `Server.hs` (covered by piece 1: NodeKernelAccess types). +- Changes to `Tracing.hs` trace constructors (covered by piece 1). +- Updating the `hprop_rpc_query_pparams` timestamp assertion (currently hardcoded to `=== 0` with a TODO comment; this piece preserves existing behaviour). +- Creating `mkNodeKernelAccess` in cardano-node (piece 3). +- Integration test infrastructure changes (piece 8). + +## Definition of done + +- [ ] All AC tests written (compile, fail on stubs) +- [ ] Implementation complete (all tests pass via `cabal test`) +- [ ] Nix CI checks pass (`nix build 'path:/work#checks.x86_64-linux.test'` and `e2e`) +- [ ] haskell-reviewer agent finds no critical or style issues +- [ ] fourmolu clean +- [ ] No build warnings + +## Notes + +- **Piece 1 prerequisite assumption:** this story assumes piece 1 delivers the `Has (IORef (Maybe NodeKernelAccess)) RpcEnv` instance in `Monad.hs`, the `MonadRpc` constraint update, and the `RpcEnv` field change in `Env.hs`. + If piece 1 is scoped more narrowly (only `NodeKernelAccess.hs` itself), those wiring changes must be pulled into this piece or an intermediate one. +- **Integration tests require piece 3:** the E2E tests (`hprop_rpc_query_pparams`, `hprop_rpc_search_utxos`, `hprop_rpc_transaction`) spin up a real testnet node, which needs `mkNodeKernelAccess` wired in `Run.hs` (piece 3). + This piece can be validated at the build and unit-test level independently; full E2E validation happens after piece 3 lands. +- **Timestamp field:** `hprop_rpc_query_pparams` asserts `timestamp === 0` (line 102) with the comment "not possible to implement at this moment". + After this rewrite, `systemStart` and `eraHistory` are available inside the snapshot, so `slotToTimestamp` should produce real values. + However, updating the test assertion is out of scope for this piece; it should be addressed in a follow-up once the full pipeline is wired end-to-end. +- **Mechanical transformation:** all three methods follow the identical old-to-new pattern. + The only difference is the domain query (`QueryProtocolParameters` vs `QueryUTxO`). + Consider extracting a shared `withQuerySnapshot` helper if the duplication becomes unwieldy, but this is an implementation decision, not an AC. +- **File changed:** `cardano-api/cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/Query.hs` (single file). + +## Reference docs + +- [Consensus protocol and snapshots](analysis-consensus-protocol.md) - snapshot consistency and `answerQuery` dispatch +- [Implementation details](prereqs-implementation-details.md) - query inventory for ReadParams/ReadUtxos/SearchUtxos +- [UTxO-HD internals](analysis-utxohd-internals.md) - forker mechanics for UTxO queries diff --git a/cardano-rpc/docs/node-kernel-access/05-rewrite-submit-method.md b/cardano-rpc/docs/node-kernel-access/05-rewrite-submit-method.md new file mode 100644 index 0000000000..8ccef47acf --- /dev/null +++ b/cardano-rpc/docs/node-kernel-access/05-rewrite-submit-method.md @@ -0,0 +1,82 @@ +# Piece 5: Rewrite SubmitTx to use NodeKernelAccess + +## Problem + +`submitTxMethod` in `Submit.hs` uses Node-to-Client IPC for both era detection (`determineEra`) and transaction submission (`submitTxToNodeLocal`). +Each call opens a separate N2C connection, adding latency and requiring `tryAny` wrapping to handle connection-level exceptions. +This must be replaced with the `NodeKernelAccess` interface introduced in piece 1. + +## Why + +Eliminating N2C from the submit path removes a connection per request, avoids double serialisation, and brings `submitTxMethod` in line with the node kernel access architecture (ADR-019). + +## User value + +As a cardano-rpc operator, I want transaction submission to use in-process ledger access so that submit latency is lower and N2C connection failures are no longer a failure mode. + +## Acceptance criteria + +1. **AC1: Era detection via snapshot** - `submitTxMethod` determines the current era by calling `withNodeKernelAccess laRef` then `nkaWithSnapshot` and `runQuery snapshot QueryCurrentEra`, replacing the previous `determineEra nodeConnInfo` N2C call. + - Test: manual - verify by code inspection that `determineEra` is no longer called in `Submit.hs` + +2. **AC2: Submission via nkaSubmitTx** - Transaction submission calls `withNodeKernelAccess laRef $ \la -> nkaSubmitTx la (TxInMode sbe tx)`, a separate `withNodeKernelAccess` invocation from the era detection in AC1. + Era detection requires a ledger state snapshot; submission goes through the mempool and does not. + These are intentionally two separate `withNodeKernelAccess` calls to avoid holding a forker open during mempool insertion. + - Test: manual - verify by code inspection that `submitTxToNodeLocal` is no longer called and that era detection and submission use separate `withNodeKernelAccess` calls + +3. **AC3: N2C error wrapping removed** - The `tryAny` wrapping and `first TraceRpcSubmitN2cConnectionError` mapping are removed from the `submitTx` helper. + N2C connection errors are no longer possible via the `NodeKernelAccess` path, so this error category does not apply. + - Test: manual - `grep -En 'TraceRpcSubmitN2cConnectionError|tryAny' Submit.hs` returns no results + +4. **AC4: Submit result pattern matching preserved** - The `SubmitFail`/`SubmitSuccess` (or equivalent `TxSubmitFail`/`TxSubmitSuccess`) pattern matching on the submission result is preserved. + Validation errors are still wrapped as `TraceRpcSubmitTxValidationError` and the transaction ID is still extracted from the ledger transaction on success. + - Test: E2E - `hprop_rpc_transaction` verifies the full submit-then-query round trip (existing test, no changes needed) + +5. **AC5: Import housekeeping** - The following import changes are made in `Submit.hs`: + - Added: `Cardano.Rpc.Server.Internal.NodeKernelAccess` (for `withNodeKernelAccess`, `nkaWithSnapshot`, `runQuery`, `nkaSubmitTx`) + - Removed: `submitTxToNodeLocal` usage (from `Cardano.Api`) + - Removed: `determineEra` usage (from `Cardano.Api`) + - Removed: `throwExceptT` usage (from `Cardano.Rpc.Server.Internal.Error`) + - Test: unit - `cabal build cardano-rpc` compiles with no errors and no new warnings + +6. **AC6: Existing E2E test passes** - `hprop_rpc_transaction` in `cardano-testnet-test` passes without modification, confirming that observable behaviour is unchanged. + - Test: E2E - run `hprop_rpc_transaction` via the testnet test suite + +## Out of scope + +- Removal of the `TraceRpcSubmitN2cConnectionError` constructor from `Tracing.hs` (covered by piece 1) +- Updates to `cardano-node` tracer references in `Rpc.hs` (piece 3) +- Changes to any other RPC method (`Query.hs`, `Eval.hs`, `Node.hs`) +- New tracing constructors for ledger-access errors (piece 1) +- Changes to `Monad.hs`, `Env.hs`, or `Server.hs` (piece 1: NodeKernelAccess types) +- The `NodeKernelAccess` module itself (piece 1) + +## Definition of done + +- [ ] All AC verifications completed (code inspection, compilation, E2E) +- [ ] `cabal build cardano-rpc` compiles with no errors or warnings +- [ ] `hprop_rpc_transaction` E2E test passes +- [ ] Nix CI checks pass (`nix build 'path:.#checks.x86_64-linux.test'` and `e2e`) +- [ ] haskell-reviewer agent finds no critical or style issues +- [ ] fourmolu clean +- [ ] No build warnings + +## Notes + +- **Hard compile-time dependency on piece 1:** This piece imports `Cardano.Rpc.Server.Internal.NodeKernelAccess` and calls `withNodeKernelAccess`, `nkaWithSnapshot`, `runQuery`, and `nkaSubmitTx`. + None of these exist until piece 1 is landed. + The `Has (IORef (Maybe NodeKernelAccess)) RpcEnv` instance and the updated `MonadRpc` constraint are also delivered by piece 1. +- **TDD fit:** This is a behaviour-preserving refactor. + The existing E2E test (`hprop_rpc_transaction`) is the primary verification. + No new unit tests are introduced because testing `submitTxMethod` in isolation would require either a real node or a `NodeKernelAccess` mock, neither of which is established in this codebase yet. + Compile success and the existing E2E are the practical gates. +- **Two withNodeKernelAccess calls by design:** Era detection opens a forker (via `nkaWithSnapshot`) to query the ledger state. + Submission goes through the mempool (via `nkaSubmitTx`) and does not need a forker. + Bundling them in a single `nkaWithSnapshot` callback would hold the forker open during mempool insertion, which is unnecessary and blocks other forker consumers. +- **Current file:** `cardano-api/cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/Submit.hs` +- **E2E test file:** `cardano-node/cardano-testnet/test/cardano-testnet-test/Cardano/Testnet/Test/Rpc/Transaction.hs` + +## Reference docs + +- [Architecture and current state](analysis-architecture.md) - current N2C submit path +- [API signatures](prereqs-api-signatures.md) - `addLocalTxs` signature diff --git a/cardano-rpc/docs/node-kernel-access/06-rewrite-eval-method.md b/cardano-rpc/docs/node-kernel-access/06-rewrite-eval-method.md new file mode 100644 index 0000000000..f199cf49f7 --- /dev/null +++ b/cardano-rpc/docs/node-kernel-access/06-rewrite-eval-method.md @@ -0,0 +1,110 @@ +# Piece 6: Rewrite EvalTx to use NodeKernelAccess + +## Problem + +`evalTxMethod` in `Eval.hs` queries seven different ledger state values via Node-to-Client IPC using `executeLocalStateQueryExpr`. +This incurs a connection-per-request cost and CBOR serialisation round-trips that are unnecessary when the RPC server runs in-process with the node. + +## Why + +EvalTx is the most query-intensive RPC method. +Replacing N2C with direct `NodeKernelAccess` removes IPC overhead and double serialisation for all seven queries in a single call. + +## User value + +As a DApp developer calling `EvalTx`, I want transaction evaluation to use node kernel access so that execution unit estimates return faster and with lower resource cost. + +## Acceptance criteria + +1. **AC1: Single snapshot for all queries** - All seven queries (protocol parameters, UTxO by TxIn, system start, era history, stake delegation deposits, DRep state, stake pool parameters) execute inside a single `nkaWithSnapshot` callback, ensuring they share one `ReadOnlyForker` and see the same ledger state. + - Test: manual - Code review confirms `evalTxMethod` calls `nkaWithSnapshot` exactly once and all `runQuery` calls are inside that callback. + +2. **AC2: IORef NodeKernelAccess from environment** - `evalTxMethod` obtains `IORef (Maybe NodeKernelAccess)` via `grab` instead of `LocalNodeConnectInfo`. + - Test: unit - The module compiles against the updated `MonadRpc` constraint that provides `Has (IORef (Maybe NodeKernelAccess)) e` instead of `Has LocalNodeConnectInfo e`. + +3. **AC3: Era detection inside snapshot** - The current era is determined via `runQuery snapshot QueryCurrentEra` inside the snapshot callback, replacing the separate `determineEra` call over N2C. + - Test: unit - Compilation succeeds; `determineEra` and `throwExceptT` are no longer called. + +4. **AC4: Eon escapes the snapshot** - The `eon` existential (from `forEraInEon @Era`) is returned as part of the snapshot callback's result tuple, making it available for post-snapshot protobuf conversion and `evaluateTransaction`. + - Test: unit - Compilation succeeds; the `obtainCommonConstraints eon $ do ...` block after the snapshot uses the `eon` value returned from the callback. + +5. **AC5: No N2C imports remain** - `executeLocalStateQueryExpr`, `queryProtocolParameters`, `queryUtxo`, `querySystemStart`, `queryEraHistory`, `queryStakeDelegDeposits`, `queryDRepState`, `queryStakePoolParameters`, `VolatileTip`, and `determineEra` are no longer imported or called in `Eval.hs`. + - Test: manual - Code review and `grep` confirm none of these names appear in `Eval.hs`. + +6. **AC6: NodeKernelAccess import added** - `Eval.hs` imports `Cardano.Rpc.Server.Internal.NodeKernelAccess` (for `withNodeKernelAccess`, `LedgerSnapshot`, and `runQuery`). + - Test: unit - The module compiles with the new import; no unused-import warning. + +7. **AC7: Post-snapshot logic unchanged** - The `evaluateTransaction` call, redeemer data assembly, balance check, and protobuf response construction remain unchanged from the current implementation. + - Test: manual - Code review confirms the `obtainCommonConstraints eon $ do ...` block is identical to the pre-rewrite version. + +8. **AC8: Compiles without warnings** - `cabal build cardano-rpc` completes with no errors and no warnings (including `-Wunused-packages` and `-Wredundant-constraints`). + - Test: unit - Build succeeds cleanly. + +9. **AC9: Full test suite passes** - All tests in `cardano-rpc-test` pass after the rewrite, including the existing protobuf conversion tests (`hprop_mkProtoTxEval_success`, `hprop_mkProtoTxEval_with_errors`, `hprop_scriptWitnessIndex_to_redeemerPurpose`, `hprop_mkProtoRedeemer`, `hprop_scriptExecutionError_to_evalReport`). + - Test: unit - `cabal test cardano-rpc-test` exits successfully with no failures. + +10. **AC10: ExBudget results match baseline** - Submitting a transaction evaluation request to the rewritten method produces execution unit estimates consistent with known mainnet baselines or prior N2C results. + - Test: manual - Run a known transaction evaluation against a local testnet and verify the returned `ExBudget` values, fee, and balance check results are reasonable and match expectations. + +## Out of scope + +- Changes to `extractBalanceCheckCreds` (pure helper, unaffected by the data source change). +- Changes to `mkProtoTxEval`, `mkProtoRedeemer`, `scriptExecutionErrorToEvalReport`, or any protobuf conversion code in `Type.hs`. +- Changes to the post-snapshot evaluation block (`obtainCommonConstraints eon $ do ...` containing `evaluateTransaction`, redeemer assembly, balance checking, and response construction). +- Adding an automated integration test for `EvalTx` (this is a known gap; piece 8 validates the full method end-to-end). +- Changes to other RPC methods (`Query.hs`, `Submit.hs`, `Node.hs`); those are covered by separate pieces. +- Removing `import Cardano.Api` from `Eval.hs`; the module uses many `Cardano.Api` names for era handling and serialisation, so the blanket import stays. +- Removing `import Cardano.Rpc.Server.Internal.Error` from `Eval.hs`; `throwEither` is still used by `putTraceThrowEither`. + +## Definition of done + +- [ ] All AC tests written (compile, fail on stubs) +- [ ] Implementation complete (all tests pass via `cabal test`) +- [ ] Nix CI checks pass (`nix build 'path:.#cardano-rpc:lib:cardano-rpc'` and `nix build 'path:.#cardano-rpc:test:cardano-rpc-test'`) +- [ ] haskell-reviewer agent finds no critical or style issues +- [ ] fourmolu clean +- [ ] No build warnings + +## Notes + +### Dependency + +This piece requires piece 1 (NodeKernelAccess types) to be complete. +Without `Cardano.Rpc.Server.Internal.NodeKernelAccess` and the updated `MonadRpc` constraint (from piece 1), this code cannot compile. + +### Snapshot consistency invariant + +Snapshot consistency is the most critical invariant in this rewrite. +If protocol parameters come from block N but UTxOs come from block N+1, transaction evaluation will produce invalid results (wrong fee calculations, incorrect script execution budgets, or spurious failures). +The `nkaWithSnapshot` pattern guarantees all seven queries share one `ReadOnlyForker`, preserving the same consistency guarantee as the current `executeLocalStateQueryExpr` block. + +### Eon existential escape + +The `eon` value determined inside the snapshot callback must be returned as part of the result tuple. +It is needed by the post-snapshot `obtainCommonConstraints eon $ do ...` block for `evaluateTransaction` and protobuf conversion. +See the prerequisites document (section 4.1) for details on why this works with existential types. + +### Integration test gap + +No existing automated test exercises `evalTxMethod` end-to-end. +The unit tests in `Test.Cardano.Rpc.Eval` cover protobuf conversion helpers (`mkProtoTxEval`, `mkProtoRedeemer`, `scriptExecutionErrorToEvalReport`) but not the method itself. +Piece 8 (integration testing) is expected to close this gap. +AC10 covers a manual verification in the interim. + +### Query arguments unchanged + +The seven query arguments come from `extractBalanceCheckCreds` and `allInputs` exactly as in the current implementation. +The rewrite changes only the call surface (`runQuery snapshot (QueryInMode ...)` instead of `throwEither =<< throwEither =<< queryXxx sbe ...`), not the arguments themselves. +Specifically: `QueryUTxOByTxIn allInputs`, `apiStakeCreds` (for stake delegation deposits), `unregDRepCreds` (for DRep state), and `apiPoolIds` (for stake pool parameters). + +### Import changes summary + +- Add: `Cardano.Rpc.Server.Internal.NodeKernelAccess (withNodeKernelAccess, LedgerSnapshot (..))` +- Remove: `Cardano.Rpc.Server.Internal.Error` is kept (still used by `putTraceThrowEither`) +- The duplicate qualified import (`U5c` and `UtxoRpc` both pointing to `Cardano.Rpc.Proto.Api.UtxoRpc.Submit`) is pre-existing and not addressed by this piece. + +## Reference docs + +- [Consensus protocol and snapshots](analysis-consensus-protocol.md) - snapshot consistency (critical for 7-query EvalTx) +- [Implementation details](prereqs-implementation-details.md) - full EvalTx query inventory +- [API signatures](prereqs-api-signatures.md) - query type signatures diff --git a/cardano-rpc/docs/node-kernel-access/07-rewrite-node-methods.md b/cardano-rpc/docs/node-kernel-access/07-rewrite-node-methods.md new file mode 100644 index 0000000000..68416ce49c --- /dev/null +++ b/cardano-rpc/docs/node-kernel-access/07-rewrite-node-methods.md @@ -0,0 +1,72 @@ +# Piece 7: Rewrite Node methods to use NodeKernelAccess + +## Problem + +`getProtocolParamsJsonMethod` in `Node.hs` uses the N2C pattern (`determineEra` + `executeLocalStateQueryExpr` + `queryProtocolParameters`) which opens a new socket connection per request and double-serialises data. +This is the last remaining method in `Node.hs` that needs migrating to the snapshot-based `NodeKernelAccess` interface. + +## Why + +Replacing the N2C call path with node kernel access removes per-request socket overhead and double serialisation for this method, bringing `Node.hs` in line with the new architecture defined in ADR-019. + +## User value + +As a cardano-rpc maintainer, I want `getProtocolParamsJsonMethod` to use the snapshot-based `NodeKernelAccess` interface so that the Node service no longer depends on N2C for protocol parameter queries. + +## Acceptance criteria + +1. **AC1: `getEraMethod` unchanged** - The body of `getEraMethod` remains byte-identical to its current form (hardcoded Conway return). + Making it dynamic is out of scope until new eras are added. + - Test: manual - `git diff` shows no changes to `getEraMethod` + +2. **AC2: Snapshot-based protocol parameter query** - `getProtocolParamsJsonMethod` obtains a `NodeKernelAccess` via `grab @(IORef (Maybe NodeKernelAccess))` and queries protocol parameters inside a single `nkaWithSnapshot` callback using `runQuery`. + Both `pparams` and `eon` are returned from the snapshot block so that `obtainCommonConstraints eon $ A.encode pparams` continues to work outside it. + - Test: manual - code review confirms the snapshot pattern is used correctly + +3. **AC3: Era detection inside snapshot** - The current era is determined via `runQuery snapshot QueryCurrentEra` inside the snapshot callback, replacing the N2C `determineEra` call. + The `forEraInEon @Era` guard and `convert eon` pattern are preserved. + - Test: E2E - `hprop_rpc_query_pparams` validates the returned era-specific protocol parameters match the ledger + +4. **AC4: Import cleanup and clean compilation** - `Cardano.Rpc.Server.Internal.Error` is removed from imports (no more `throwExceptT`, `throwEither`). + `Cardano.Rpc.Server.Internal.NodeKernelAccess` is added. + `cabal build cardano-rpc` succeeds with no warnings related to `Node.hs`. + - Test: unit - `cabal build cardano-rpc` exits with code 0 and no warnings + +5. **AC5: E2E behavioural equivalence** - `hprop_rpc_query_pparams` passes, confirming that the rewritten method returns identical protocol parameters, ledger tip slot, block hash, and block number to the N2C implementation. + - Test: E2E - `hprop_rpc_query_pparams` in `cardano-testnet-test` + +## Out of scope + +- Making `getEraMethod` dynamic (future work when new eras arrive). +- Changes to `Env.hs`, `Monad.hs`, or `Server.hs` (covered by piece 1). +- Creating `NodeKernelAccess.hs` itself (piece 1 prerequisite). +- Rewriting methods in `Query.hs`, `Eval.hs`, or `Submit.hs` (pieces 4-6). +- Tracing changes in `Tracing.hs` or `Rpc.hs` (piece 1 and piece 3). +- Changes to `cardano-rpc.cabal` (piece 1 adds the new module). + +## Definition of done + +- [ ] All AC verifications mapped to existing tests or confirmed via code review +- [ ] Implementation complete (all existing tests still pass via `cabal test`) +- [ ] Nix CI checks pass +- [ ] haskell-reviewer agent finds no critical or style issues +- [ ] fourmolu clean +- [ ] No build warnings + +## Notes + +- **Dependency:** piece 1 (NodeKernelAccess types) must be merged first. + The environment and monad wiring that puts `IORef (Maybe NodeKernelAccess)` into `RpcEnv` and `MonadRpc` is delivered by piece 1 and must be in place before this piece compiles. +- **`eon` escapes the snapshot block** because it is needed at line 50 of the current code: `obtainCommonConstraints eon $ A.encode pparams`. + The implementer must return `(pparams, eon)` from the `nkaWithSnapshot` callback, not just `pparams`. +- **Error handling change:** The current code uses `throwExceptT` and nested `throwEither` chains for N2C error conversion. + The new code relies on `withNodeKernelAccess` throwing `GrpcException` with `GrpcUnavailable` if the kernel is not yet initialised, and on `runQuery` propagating exceptions directly. + This is simpler but changes the exception type from `RpcException` to `GrpcException` for the "not initialised" case. +- **Smallest piece in the plan:** This rewrites one method in one file. + It is a good candidate for a quick early win and a template for the larger Query/Eval/Submit rewrites. +- **File:** `cardano-api/cardano-rpc/src/Cardano/Rpc/Server/Internal/Node.hs` + +## Reference docs + +- [Architecture and current state](analysis-architecture.md) - current N2C query path +- [API signatures](prereqs-api-signatures.md) - query type signatures diff --git a/cardano-rpc/docs/node-kernel-access/08-integration-testing.md b/cardano-rpc/docs/node-kernel-access/08-integration-testing.md new file mode 100644 index 0000000000..e7cd23b4d1 --- /dev/null +++ b/cardano-rpc/docs/node-kernel-access/08-integration-testing.md @@ -0,0 +1,97 @@ +# Piece 8: Integration testing and validation + +## Problem + +All seven preceding pieces replace the N2C IPC path with node kernel access, but there is no explicit validation that the existing integration tests still pass, that no N2C artefacts remain in the codebase, and that new tracing constructors appear in logs. +Without this validation gate, the ADR-019 migration could regress end-to-end behaviour silently. + +## Why + +This is the final piece in the ADR-019 node kernel access migration. +It gates the merge of the whole effort by confirming that the observable behaviour of cardano-rpc is unchanged from the perspective of gRPC clients, while verifying that the internal wiring has shifted entirely to in-process ledger access. + +## User value + +As a cardano-rpc maintainer, I want to confirm that the node kernel access migration preserves all existing integration test behaviour and identify any new test coverage gaps, so that the ADR-019 work can be merged with confidence. + +## Acceptance criteria + +1. **AC1: `hprop_rpc_query_pparams` passes** - The existing E2E test that validates all 44 protocol parameters match between the gRPC response and the ledger state passes without modification. + The test connects via `Rpc.withConnection` to the gRPC socket, which is unchanged by this migration. + - Test: E2E - `TASTY_PATTERN='/RPC Query Protocol Params/' cabal test cardano-testnet-test` + +2. **AC2: `hprop_rpc_transaction` passes** - The existing E2E test that performs a full round-trip (fetch UTxOs via SearchUtxos, build transaction, submit via SubmitTx, confirm on-chain via SearchUtxos) passes without modification. + - Test: E2E - `TASTY_PATTERN='/RPC Transaction Submit/' cabal test cardano-testnet-test` + +3. **AC3: `hprop_rpc_search_utxos` passes** - The existing E2E test that exercises SearchUtxos with exact-address predicates, payment-credential predicates, non-matching predicates, and predicate-less queries passes without modification. + - Test: E2E - `TASTY_PATTERN='/RPC SearchUtxos/' cabal test cardano-testnet-test` + +4. **AC4: No RPC test file changes** - No files under `cardano-testnet/test/cardano-testnet-test/Cardano/Testnet/Test/Rpc/` are modified by the ADR-019 branch relative to the base branch. + `Testnet/Types.hs` is also unchanged; `nodeRpcSocketPath` still delegates to `nodeSocketPathToRpcSocketPath`. + - Test: manual - `git diff main -- cardano-testnet/test/cardano-testnet-test/Cardano/Testnet/Test/Rpc/ cardano-testnet/src/Testnet/Types.hs` shows zero changes + +5. **AC5: No N2C trace constructors remain** - `TraceRpcSubmitN2cConnectionError` does not appear anywhere in the cardano-rpc or cardano-node source trees. + The constructor was removed in piece 1 and replaced by `TraceRpcNodeKernelAccessUnavailable` and `TraceRpcForkerError`. + - Test: unit - `grep -r 'TraceRpcSubmitN2cConnectionError' cardano-api/cardano-rpc/src/ cardano-node/cardano-node/src/` returns no matches + +6. **AC6: No N2C connection imports in RPC methods** - The modules `Query.hs`, `Submit.hs`, `Eval.hs`, and `Node.hs` no longer import `executeLocalStateQueryExpr`, `submitTxToNodeLocal`, `determineEra`, or `VolatileTip` from `Cardano.Api`. + - Test: unit - `grep -l 'executeLocalStateQueryExpr\|submitTxToNodeLocal\|determineEra\|VolatileTip' cardano-api/cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/*.hs cardano-api/cardano-rpc/src/Cardano/Rpc/Server/Internal/Node.hs` returns no matches + +7. **AC7: No runtime N2C connection from cardano-rpc** - The absence of `executeLocalStateQueryExpr`, `submitTxToNodeLocal`, and `determineEra` imports (AC6) is the static proof that cardano-rpc no longer opens N2C connections at runtime. + As additional confirmation, a manual testnet run should show no `connectToLocalNode` traces originating from cardano-rpc in the node logs. + - Test: manual - inspect structured trace output from a testnet run and confirm no N2C connection traces are attributed to cardano-rpc + +8. **AC8: New trace constructors appear in trace configuration** - `NodeKernelAccessUnavailable` and `ForkerError` appear in the `allNamespaces` list of `MetaTrace TraceRpc` in `cardano-node/src/Cardano/Node/Tracing/Tracers/Rpc.hs`. + - Test: unit - `grep -c 'NodeKernelAccessUnavailable\|ForkerError' cardano-node/cardano-node/src/Cardano/Node/Tracing/Tracers/Rpc.hs` returns at least 2 + +9. **AC9: Coverage gaps documented** - A note is added to this story (or a follow-up ticket is filed) listing the known integration test coverage gaps: + (a) No E2E test for EvalTx (the `evalTxMethod` is exercised only by unit tests on the protobuf conversion layer, not end-to-end). + (b) No E2E test for the UNAVAILABLE gRPC response during startup before `NodeKernel` is ready. + (c) No E2E test for forker acquisition failure. + (d) No E2E test for snapshot consistency under concurrent ledger state changes. + - Test: manual - reviewer confirms the gaps are documented + +## Out of scope + +- Writing new E2E tests for EvalTx, UNAVAILABLE startup response, forker errors, or snapshot consistency (documented as gaps in AC9; to be addressed in separate stories). +- Performance benchmarking of node kernel access vs N2C (separate initiative). +- Changes to `Testnet/Types.hs`, `Testnet/Start/Types.hs`, or any RPC test file. +- Modifications to the gRPC client library (`Cardano.Rpc.Client`). +- Changes to protobuf definitions or generated code. + +## Definition of done + +- [ ] All three E2E tests pass: `hprop_rpc_query_pparams`, `hprop_rpc_transaction`, `hprop_rpc_search_utxos` +- [ ] N2C artefact checks pass (AC5, AC6) +- [ ] New trace constructors verified (AC8) +- [ ] No RPC test files modified (AC4) +- [ ] Coverage gaps documented (AC9) +- [ ] Nix CI checks pass (`nix build 'path:/work/cardano-api#checks.x86_64-linux.test'` and `e2e`) +- [ ] haskell-reviewer agent finds no critical or style issues across all ADR-019 changes +- [ ] fourmolu clean +- [ ] No build warnings + +## Notes + +- **Dependency:** all pieces 1-7 must be complete before this piece can be validated. + This is the merge gate for the entire ADR-019 node kernel access effort. +- **Three E2E tests, not two:** The implementation plan's Step 13 mentions two tests (`hprop_rpc_query_pparams` and `hprop_rpc_transaction`), but `hprop_rpc_search_utxos` also exists in `cardano-testnet/test/cardano-testnet-test/Cardano/Testnet/Test/Rpc/SearchUtxos.hs` and exercises the same gRPC client path. + All three must pass. +- **Testnet wiring is automatic:** The testnet creates a real `cardano-node` with `runtimeEnableRpc = RpcEnabled` (or `cardanoEnableRpc = RpcEnabled` in `SearchUtxos`). + The `Run.hs` wiring in piece 3 populates the `IORef (Maybe NodeKernelAccess)` via `rnNodeKernelHook`, so the RPC server transitions from UNAVAILABLE to serving automatically. +- **gRPC client path is unchanged:** Tests connect via `Rpc.withConnection def (Rpc.ServerUnix rpcSocket)`, which talks to the gRPC Unix socket. + The internal switch from N2C to node kernel access is invisible at this layer. +- **Unit tests in cardano-rpc-test:** The unit tests in `cardano-rpc/test/` (protocol parameter conversion, predicate matching, eval protobuf construction, pagination, type conversion) test the conversion layer which is unaffected by this migration. + They should continue to pass without changes. +- **File locations:** + - E2E tests: `cardano-node/cardano-testnet/test/cardano-testnet-test/Cardano/Testnet/Test/Rpc/{Query,Transaction,SearchUtxos}.hs` + - Test runner: `cardano-node/cardano-testnet/test/cardano-testnet-test/cardano-testnet-test.hs` + - Testnet types: `cardano-node/cardano-testnet/src/Testnet/Types.hs` (exports `nodeRpcSocketPath`) + - Node wiring: `cardano-node/cardano-node/src/Cardano/Node/Run.hs` + - Trace instances: `cardano-node/cardano-node/src/Cardano/Node/Tracing/Tracers/Rpc.hs` + +## Reference docs + +- [Implementation details](prereqs-implementation-details.md) - verification checklist +- [Build and conventions](prereqs-build-and-conventions.md) - test commands +- [Architecture and current state](analysis-architecture.md) - spec coverage gaps diff --git a/cardano-rpc/docs/node-kernel-access/09-follow-tip.md b/cardano-rpc/docs/node-kernel-access/09-follow-tip.md new file mode 100644 index 0000000000..9235505eed --- /dev/null +++ b/cardano-rpc/docs/node-kernel-access/09-follow-tip.md @@ -0,0 +1,261 @@ +# Piece 9: FollowTip streaming endpoint + +## Problem + +Chain-following clients (Kupo, Scrolls, Oura, custom indexers) currently need a direct N2C ChainSync connection to stream blocks from the node. +This requires local socket access, a Haskell-compatible Ouroboros implementation, and version negotiation. +There is no gRPC equivalent. + +## Why + +`FollowTip` is the gRPC equivalent of the Ouroboros ChainSync mini-protocol and the last method required by #1229 (replacing Ogmios as Kupo's chain data source). +It is the first server-streaming RPC in cardano-rpc and the first to use `ChainDB` follower capabilities. +After this piece, remote clients can follow the chain over TCP using standard gRPC libraries in any language. + +## User value + +As a chain indexer developer, I want to follow the chain tip over gRPC so that I can stream fully parsed blocks without implementing the Ouroboros ChainSync protocol or having local socket access to the node. + +## Already in place + +Do not re-plan these; they exist on the FetchBlock/ReadTip PR stack (#1247, #1258, #1259) and this piece builds on them. + +- `sync.proto` already declares `rpc FollowTip(FollowTipRequest) returns (stream FollowTipResponse)` with `repeated BlockRef intersect` in the request and the `apply`/`undo`/`reset` oneof plus `BlockRef tip` in the response; proto-lens bindings are generated. +- `Server.hs` already registers a server-streaming stub for `followTip` (`mkServerStreaming $ \_ _ -> unimplemented`). +- `NodeKernelAccess` lives in cardano-rpc (`Cardano.Rpc.Server.NodeKernelAccess` and `.Type`) with fields `chainDb`, `systemStart` and `readEraHistory` (NoFieldSelectors); cardano-node only constructs it in `Cardano.Node.Run` and hands it over through an `IORef`. +- Full block-to-proto conversion exists: `fetchBlockMethod` assembles header, parsed transactions (`txToUtxoRpcTx`, all eras including Byron via `byronBlockTxs`) and slot timestamp. +- Tip projection exists: `readTipMethod` builds a fully populated `BlockRef` (slot, hash, height, timestamp) from a ChainDB header. +- Tracing pattern: `TraceRpcSync` span constructors wrapped via `wrapInSpan`, with seven exhaustive sites in cardano-node's `Cardano.Node.Tracing.Tracers.Rpc` (`forMachine`, `asMetrics`, `namespaceFor`, `severityFor`, `documentFor`, `metricsDocFor`, `allNamespaces`). + +## Revised scope decision: parsed blocks are in scope + +The original plan streamed `native_bytes` only. +That contradicts #1229: Kupo needs full transaction data (inputs, outputs, datums, scripts, metadata) in the stream, and re-parsing CBOR client-side defeats the purpose of the gRPC interface. +Since FetchBlock already builds the fully parsed `cardano` block, `FollowTip` reuses that assembly and streams both `native_bytes` and the parsed block. + +## Phases + +Each phase is one commit, builds with zero warnings from `/work`, and passes the full cardano-rpc test suite. +Phases 1-4 are cardano-api repo commits; phase 5 is the cardano-node repo counterpart. + +### Phase 1: cardano-api re-exports for ChainDB followers + +Extend `Cardano.Api.Consensus` (via the `ChainDB.` qualified exports and `Internal/Reexport.hs`) with the follower vocabulary: + +- `ChainDB.newFollower`, the `Follower` type and its field accessors (`followerInstructionBlocking`, `followerForward`, `followerClose`), and `ChainDB.ChainType (..)`. +- `withRegistry` / `ResourceRegistry` (from the resource-registry package) so cardano-rpc does not need a direct ouroboros-consensus dependency. +- `ChainUpdate (..)` (`AddBlock` / `RollBack`) used by follower instructions. + +Changelog fragment: cardano-api, `compatible`. + +- Test: unit - compiles; no behaviour change. +- Review: export-list-only diff, mirrors the `getTipHeader` re-export commit in #1259. + +### Phase 2: follower capability in NodeKernelAccess + +In `Cardano.Rpc.Server.NodeKernelAccess`: + +- `data ChainChange = ChainApply (ByteString, BlockInMode) | ChainRollBack ChainPoint` - the apply payload is the same raw-bytes-plus-parsed-block pair `fetchBlock` returns. + Note: consensus `RollBack` carries only the rollback point, never the rolled-back blocks, which constrains what `undo` can contain (see the scope note in phase 4). +- `data ChainFollower = ChainFollower { nextChange :: IO ChainChange, findIntersect :: [ChainPoint] -> IO (Maybe ChainPoint) }`. +- `withFollower :: NodeKernelAccess -> (ChainFollower -> IO a) -> IO a` with bracket semantics: `withRegistry`, `ChainDB.newFollower` with a block component fetching raw bytes and the block (as in `fetchBlock`), `followerClose` on all exit paths. +- `nextChange` wraps `followerInstructionBlocking`, mapping `AddBlock` to `ChainApply` and `RollBack` to `ChainRollBack`; `findIntersect` wraps `followerForward`. + +No server wiring yet; the stub stays. + +- Test: unit - compiles; the E2E test in phase 5 exercises it. +- Review: self-contained module addition, no call sites change. + +### Phase 3: extract shared block and tip assembly (pure refactor) + +- Extract the response-body assembly from `fetchBlockMethod` into a shared function (raw bytes + `BlockInMode` + timestamp to `AnyChainBlock` with `native_bytes`, `cardano.header`, `cardano.body.tx`, `cardano.timestamp`). +- Extract the `BlockRef` tip projection from `readTipMethod` (header + timestamp to `BlockRef`). +- `fetchBlockMethod` and `readTipMethod` call the extracted functions; behaviour is unchanged. + +- Test: existing cardano-rpc suite passes unchanged; E2E FetchBlock/ReadTip assertions in cardano-node still pass. +- Review: pure code motion, diff should show no logic edits. + +### Phase 4: followTipMethod handler, wiring, tracing, README + +- `followTipMethod` in `Cardano.Rpc.Server.Internal.UtxoRpc.Sync`, the grapesy server-streaming handler (request plus a send callback): + 1. `grabNodeKernelAccess`, then `withFollower`. + 2. Convert `repeated BlockRef intersect` to `[ChainPoint]`; reject malformed hashes with `INVALID_ARGUMENT` (as FetchBlock does). + 3. Empty intersect: follow from the current tip (read the tip point and intersect at it) - Dolos does the same, and it protects a naive client from an accidental full-chain replay. + An intersection ref with an empty hash denotes origin; clients append it as an infallible catch-all, and full-history sync is requested this way. + Non-empty intersect with no match: fail with `NOT_FOUND` before any stream message - the ecosystem has no carry-on precedent (Dolos panics, Blink Labs errs with code Unknown, ChainSync leaves the reaction to the client), and a wholly unmatched list signals garbage refs or the wrong network. + After a successful `followerForward`, the follower's first instruction is a `RollBack` to the intersection point - it comes out as the initial `reset`, telling the client where streaming resumes. + 4. Loop: `nextChange`, assemble the response with the phase 3 helpers, attach the current tip via `getTipHeader` and the tip projection, send, repeat. + `ChainApply` becomes an `apply` action with the full parsed block; `ChainRollBack` becomes a `reset` action with the rollback point's `BlockRef`. + Scope note: the UTxO RPC spec also allows `undo` actions carrying the rolled-back blocks, but consensus followers do not provide them (`RollBack` is point-only); serving `undo` needs a per-stream buffer of recently applied blocks, which is deferred (see out of scope). + Clients handle `reset` exactly like ChainSync's `RollBackward`, which is what Kupo's sync loop already does. + 5. Cleanup on client disconnect, stream close or exception is guaranteed by the `withFollower` bracket (grapesy cancels the handler thread on disconnect). +- New `TraceRpcSync` constructor: `TraceRpcFollowTipSpan TraceSpanEvent` (the `NOT_FOUND` error path is covered by the generic request-error tracing). + No per-block apply/undo traces: they would fire for every block on the chain and the prometheus request counter plus consensus block traces already cover it. + Note: `wrapInSpan` wraps unary handlers; the streaming handler needs a streaming-shaped span wrapper (begin on stream open, end on stream close) - add `wrapInSpanStreaming` next to it. +- `Server.hs`: replace the followTip stub. +- README: FollowTip row to supported. +- Changelog fragment: cardano-rpc, `feature`. + +- Test: unit - compiles; handler logic is exercised end-to-end in phase 5. +- Review: the only phase with real logic; everything below it is already reviewed. + +### Phase 5: cardano-node tracing and E2E test (cardano-node repo) + +- `Cardano.Node.Tracing.Tracers.Rpc`: extend all seven exhaustive sites for the two new constructors; `rpc.request.SyncService.FollowTip` counter on span begin; severity Debug for the span, Info for reset. +- New `Cardano.Testnet.Test.Rpc.FollowTip` with `hprop_rpc_follow_tip`: + 1. Start a testnet with RPC enabled (as in the FetchBlock test). + 2. Open a `FollowTip` stream with a single origin ref (empty hash) in the intersect list; assert the first message is a `reset` to origin, then `apply` actions with non-empty `native_bytes`, populated `cardano.header`, and a `tip` with a 32-byte hash. + 2a. Open a stream with an empty intersect list; assert the first message is a `reset` at (or after) the CLI-queried tip. + 3. Submit a transaction over gRPC, keep reading until an `apply` block contains its hash in `cardano.body.tx`, and assert the parsed transaction fields against the submitted values. + 4. Re-open a stream with the intersect set to an already-seen block ref and assert streaming resumes after that point (no replay from origin). + 5. Assert an unknown intersect ref (valid 32-byte hash not on the chain, no origin fallback) fails the stream with `NOT_FOUND` before any message. +- Register the test in `cardano-testnet-test.hs`. + +- Test: `TASTY_PATTERN='/RPC FollowTip/' cabal test cardano-testnet-test`. +- Review: tracer arms are mechanical; the test is the substance. + +### Phase 6a: injectable followTip loop (pure refactor, enables unit tests) + +`followTipMethod` currently reaches for its collaborators directly (`ChainFollower` from `withFollower`, `fetchBlock`, `getTipHeader`, `readEraHistory`), which makes it untestable without a live ChainDB. +Extract the streaming loop into a function taking its capabilities as arguments: + +- `followTipStream :: ChainFollower -> FollowTipEnv m -> (NextElem (Proto U5c.FollowTipResponse) -> IO ()) -> [ChainPoint] -> m ()` - exact shape to taste, but the loop must receive the follower, a block-by-point fetch (`ChainPoint -> m (Maybe (ByteString, BlockInMode))`), a tip reader and a slot-to-timestamp function as inputs rather than closing over `NodeKernelAccess`. +- `followTipMethod` becomes a thin wrapper: request parsing and intersection setup (unchanged), then calls the loop with the real capabilities. +- Behaviour is byte-identical; the commit must read as code motion plus parameter threading. +- New unit tests in `cardano-rpc-test` drive the loop with a scripted `ChainFollower` (it is a plain record of `IO` actions, constructible from a mutable list of `ChainChange`s) and stub capabilities: assert the current semantics - initial reset from the scripted rollback, apply assembly, tip attachment, per-message ordering. + This is the first unit coverage for the streaming loop at all. + +- Test: `cabal test cardano-rpc-test` - new scripted-follower properties plus the existing suite. +- Review: refactor and tests only, no semantic change. + +### Phase 6b: `undo` delivery via ChainDB re-fetch (approach B) + +Rollbacks currently degrade to `reset` because consensus `RollBack` is point-only. +Serve `undo` with full blocks by remembering only the POINTS of applied blocks and re-fetching the bodies on rollback: + +- Track applied points per stream: a `Seq (SlotNo, Hash BlockHeader)` - not a `Seq ChainPoint` - newest first, threaded through the loop, appended on every `apply`, capped at a generous constant (`followTipUndoWindow`) comfortably above any real rollback depth (5000 entries is ~200 KB per stream; rollbacks beyond the security parameter cannot happen, so any cap >= k is semantically complete on mainnet - testnets with tiny k are always complete too). + The tuple representation is deliberate: an applied point always has a real slot and hash (it comes straight from a decoded block's header), so `ChainPointAtGenesis` is structurally impossible to track instead of an invariant to maintain by convention. + Alongside the tracked points, remember the stream's start point (the resolved intersection) as the window floor: a rollback landing exactly on the floor is in-window even though the floor itself is never a tracked applied point, so a rollback all the way back to stream start undoes everything tracked, newest first. + The floor only moves when a rollback falls outside the window (point 2 below), to that rollback's target, so a later rollback landing on the same point again is still served as undo instead of a second reset. +- On `ChainRollBack point`: + 1. Split the tracked points at the rollback target: everything newer than `point` is the undone suffix; `point` itself (if it is a tracked point, or the floor) and older entries are kept. + 2. If the target is not in the tracked window (deeper than the cap, a rollback below the stream's start, or the initial rollback-to-intersection at stream open when nothing is tracked yet): clear the tracking, move the window floor to the rollback target, and emit today's single `reset` - the unchanged fallback, which also preserves the first-message-reset invariant. + 3. Otherwise, for each undone point NEWEST FIRST: re-fetch via the existing `fetchBlock` path (the block lives in the VolatileDB - garbage collection only prunes below the immutable tip, so a fresh rollback is ~k blocks away from the threshold; the race is real but tiny). + On a successful fetch, emit `undo` with the full re-assembled `AnyChainBlock` (same `mkAnyChainBlock`, timestamp recomputed - deterministic, so the payload equals the original `apply`). + On a fetch miss (GC won the race), stop the undo sequence and emit a single `reset` to the rollback point - partial-undo-then-reset is coherent, since `reset` is absolute positioning. + If there is nothing to undo (the target is the floor or the newest tracked point, with no newer entries), a plain `reset` communicates the position with nothing to undo - this is also how the stream-opening rollback-to-intersection is served, since nothing is tracked yet. + 4. Every emitted message carries the current tip, as today. +- All `followTipStream`-local; no `NodeKernelAccess` or `ChainFollower` changes. +- No new trace constructors: the fetch-miss fallback is rare and visible to the client as the `reset`; considered and skipped. +- Unit tests via the phase 6a harness: scripted apply/rollback sequences asserting undo emission content and newest-first order, the exact partial-fallback behaviour (undo, undo, fetch miss -> reset), a rollback to the stream's start undoing everything newest-first, cap-overflow degrading to reset even when the fetch would have succeeded, and that the initial rollback-to-intersection still produces the opening reset. +- Update the FollowTip changelog fragment (undo delivery), the README stays as-is (FollowTip already supported), and rewrite the out-of-scope bullet below. +- E2E: still impossible on a single-node testnet; conformance testing against Dolos remains the cross-check for live rollbacks. + +- Test: `cabal test cardano-rpc-test`. +- Review: the split logic and fallback semantics are the substance; re-fetch plumbing is mechanical. + +### Ecosystem context + +Dolos is the only other UTxO RPC implementation that actually serves `undo`. +It stashes full block bodies in its own write-ahead log at apply time, bounded by `max_rollback` (~129600 slots on mainnet, the security window), and panics if a rollback ever needs to go deeper. +It also ships `undo` with the tip left unset. +dugite is reset-only: it never emits `undo`. +Every SDK examined types `undo` as a full block and accepts `reset` as a valid fallback action. +Our approach B leans on the VolatileDB instead of a per-stream payload buffer: abandoned forks stay there until garbage collection passes the immutable tip, so re-fetching by point works without any stream-side storage. +`reset` remains the graceful floor for the rare case a block is no longer there. +Every message we emit, `undo` included, carries the current tip, unlike Dolos. + +## Out of scope + +- `FieldMask` support (also ignored by the other query methods). +- Concurrent follower limits and idle timeouts (follow-up; the node already supports one follower per N2C ChainSync client, so the mechanism scales the same way). +- A per-stream ring buffer of applied blocks (approach A for serving `undo`): superseded by phase 6b's VolatileDB re-fetch (approach B), which needs no per-stream payload storage. + Rollbacks outside the tracked window, or ones whose blocks garbage collection has already pruned, still fall back to `reset`. +- Rollback E2E coverage: a single-node testnet cannot produce a rollback; the rollback mapping is covered by the type-level totality of the `ChainUpdate` translation and left to conformance testing against other implementations. +- `DumpHistory` (separate piece). +- Backpressure tuning beyond gRPC's built-in HTTP/2 flow control. + +## Definition of done + +- [ ] Phases land as separate commits, each building with zero warnings and passing tests. +- [ ] E2E test passes: `TASTY_PATTERN='/RPC FollowTip/' cabal test cardano-testnet-test`. +- [ ] Nix CI checks pass. +- [ ] fourmolu clean (`scripts/devshell/prettify` on changed files; cardano-node has no prettify script). +- [ ] Changelog fragments present with the real PR numbers (herald counts only fragments added since the fork point). +- [ ] README coverage table updated. + +## Notes + +### Key differences from Ouroboros ChainSync + +| Aspect | ChainSync | FollowTip | +|--------|-----------|-----------| +| Transport | Unix socket (N2C) | TCP/gRPC | +| Rollback | `MsgRollBackward` with the point | `undo` actions with the rolled-back blocks, re-fetched from ChainDB, falling back to a `reset` with the point's `BlockRef` outside the tracked window | +| No-intersect | `MsgIntersectNotFound` (informative, client reacts) | `NOT_FOUND` error; clients append an origin ref as a catch-all | +| Pipelining | Client-driven (`MsgRequestNextPipelined`) | Server-driven (gRPC stream backpressure) | +| Block format | CBOR | Proto (`AnyChainBlock` with `native_bytes` and parsed block) | + +### Why N2C ChainSync is insufficient + +Each gRPC `FollowTip` client would need its own N2C socket connection with a dedicated ChainSync follower. +The node creates a fresh follower per N2C connection with no way to multiplex. +In-process access via `ChainDB.newFollower` lets cardano-rpc create followers directly without socket overhead. + +### Tip reporting + +Each `FollowTipResponse` includes the current tip, read via `getTipHeader` and projected with the same function `readTipMethod` uses. +This lets clients measure sync lag (difference between their position and the tip). + +### Files affected + +**cardano-api repo:** + +| File | Phase | Change | +|---|---|---| +| `cardano-api/src/Cardano/Api/Consensus.hs` and `Consensus/Internal/Reexport.hs` | 1 | Follower re-exports. | +| `cardano-rpc/src/Cardano/Rpc/Server/NodeKernelAccess.hs` (and `.Type`) | 2 | `withFollower`, `ChainFollower`, `ChainChange`. | +| `cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/Sync.hs` | 3, 4 | Extracted assembly helpers; `followTipMethod`. | +| `cardano-rpc/src/Cardano/Rpc/Server/Internal/Tracing.hs` | 4 | `TraceRpcFollowTipSpan`, `wrapInSpanStreaming`. | +| `cardano-rpc/src/Cardano/Rpc/Server.hs` | 4 | Register `followTipMethod`. | +| `cardano-rpc/README.md`, `.changes/` | 1, 4 | Coverage row, fragments. | + +**cardano-node repo:** + +| File | Phase | Change | +|---|---|---| +| `cardano-node/src/Cardano/Node/Tracing/Tracers/Rpc.hs` | 5 | Two new constructors across all seven case sites plus counter. | +| `cardano-testnet/test/cardano-testnet-test/Cardano/Testnet/Test/Rpc/FollowTip.hs` | 5 | New E2E test. | +| `cardano-testnet/test/cardano-testnet-test/cardano-testnet-test.hs` | 5 | Register the test. | + +### Gotchas for the implementer + +- **grapesy server-streaming.** First streaming handler in the codebase; the `mkServerStreaming` handler receives the request and a send callback instead of returning a response. + `wrapInSpan` does not fit its shape - add a streaming variant. +- **`followerInstructionBlocking`, not `followerInstruction`.** The non-blocking variant would busy-wait. +- **Rollbacks are point-only.** `ChainUpdate`'s `RollBack` never carries the rolled-back blocks; `undo`-with-blocks (phase 6b) needs a re-fetch by point from ChainDB, not a per-stream buffer. + After `followerForward` succeeds, the first instruction is a `RollBack` to the intersection - expected, not an error. +- **`ChainDB.newFollower` needs a `ResourceRegistry`.** Use `withRegistry`; tie follower cleanup to the bracket, not to garbage collection. +- **GHC 9.6/9.10 MonoLocalBinds.** Local bindings whose type comes out of a constraint continuation (`anyEraTxConstraints`) and mid-do GADT dispatches need explicit type signatures; 9.12 infers them, so these break only in CI (see AGENTS.md GHC gotchas). +- **Proto is already generated.** Do not touch `gen/`; no codegen step is needed for this piece. +- **New trace constructors break cardano-node compilation** until all seven case sites in `Tracers/Rpc.hs` are extended; coordinate the cardano-api and cardano-node PRs like FetchBlock/ReadTip did. + +### Risks + +| Risk | Mitigation | +|------|------------| +| Follower leak on unclean client disconnect | Bracket in `withFollower`; grapesy cancels the handler thread on disconnect, which unwinds the bracket | +| High memory per follower with many clients | Out of scope; concurrent limits can be added later | +| Parsed-block assembly cost per streamed block | Same conversion FetchBlock already does once per request; if profiling shows pressure, a `FieldMask` fast path can skip tx parsing later | +| `followerForward` intersection semantics differ from expectation | Covered by E2E steps 4 and 5 (resume after known point, reset on unknown point) | + +### Dependencies + +- **Upstream:** the FetchBlock/ReadTip PR stack (#1247, #1258, #1259) must merge first; phase 1 also depends on the pinned consensus version exposing the follower API. +- **Downstream:** Kupo's gRPC `ChainProducer` backend (out of scope here, tracked by #1229). + +## Reference docs + +- [Architecture and current state](analysis-architecture.md) - cardano-rpc overview and spec coverage +- [Build and conventions](prereqs-build-and-conventions.md) - build instructions +- GitHub issues: #1219 (FollowTip), #1229 (Kupo/Ogmios replacement) diff --git a/cardano-rpc/docs/node-kernel-access/README.md b/cardano-rpc/docs/node-kernel-access/README.md new file mode 100644 index 0000000000..cd21c5a618 --- /dev/null +++ b/cardano-rpc/docs/node-kernel-access/README.md @@ -0,0 +1,66 @@ +# Direct Node Access for cardano-rpc + +ADR: [ADR-019](../../../../cardano-node-wiki/docs/ADR-019-node-kernel-access-for-cardano-rpc.md) + +## Overview + +Replace N2C IPC (Unix socket queries to cardano-node) with in-process access to `NodeKernel`. +This covers ledger state queries (via `answerQuery`), block retrieval (via `ChainDB`), and transaction submission (via `Mempool`). +See ADR-019 for the full rationale. + +## Reference documents + +Technical analysis (split by topic): +- [Architecture and current state](analysis-architecture.md) - cardano-rpc overview, spec coverage, data path +- [UTxO-HD internals](analysis-utxohd-internals.md) - backing store, LedgerTables, forker mechanics +- [Consensus protocol and snapshots](analysis-consensus-protocol.md) - LocalStateQuery, in-memory state, snapshot consistency + +Prerequisites (split by topic): +- [API signatures](prereqs-api-signatures.md) - verified consensus and cardano-api type signatures +- [Build and conventions](prereqs-build-and-conventions.md) - project layout, nix build, codebase gotchas +- [Implementation details](prereqs-implementation-details.md) - subtle gotchas, query inventory, verification checklist + +Cross-cutting: +- [Implementation plan](implementation-plan.md) - file summary, verification, risks + +## Deliverables + +Each piece is independently deliverable and testable. +Piece 1 creates the `NodeKernelAccess` abstraction and wires it through cardano-rpc. +Piece 2 adds the FetchBlock proto and handler. +Piece 3 implements `mkNodeKernelAccess` in cardano-node and wires it into the startup sequence, making FetchBlock work end-to-end. +Pieces 4-7 rewrite existing N2C methods to use `NodeKernelAccess`. +Piece 9 adds the FollowTip server-streaming endpoint. +Piece 8 is the final validation. + +``` +1 (NodeKernelAccess types) ─► 2 (FetchBlock proto) ─► 3 (mkNodeKernelAccess + wiring) ─► 4 (query methods) ─┐ + ├► 5 (submit method) │ + ├► 6 (eval method) ├► 8 (integration testing) + ├► 7 (node methods) │ + └► 9 (FollowTip) ─┘ +``` + +| # | Deliverable | Scope | Repo | +|---|-------------|-------|------| +| 1 | [NodeKernelAccess types and plumbing](01-node-access-types.md) | `NodeKernelAccess` record, env/monad/tracing/server wiring | cardano-rpc | +| 2 | [FetchBlock proto and handler](02-fetchblock-proto-and-handler.md) | SyncService proto, codegen, `fetchBlockMethod`, server registration | cardano-rpc | +| 3 | [mkNodeKernelAccess and node wiring](03-mk-node-access-and-wiring.md) | `mkNodeKernelAccess` from `NodeKernel`, `Run.hs` IORef wiring, E2E test | cardano-node | +| 4 | [Rewrite query methods](04-rewrite-query-methods.md) | ReadParams, ReadUtxos, SearchUtxos switch to snapshot pattern | cardano-rpc | +| 5 | [Rewrite submit method](05-rewrite-submit-method.md) | SubmitTx switches to nkaSubmitTx | cardano-rpc | +| 6 | [Rewrite eval method](06-rewrite-eval-method.md) | EvalTx (7 queries) switches to snapshot | cardano-rpc | +| 7 | [Rewrite node methods](07-rewrite-node-methods.md) | GetProtocolParamsJson switches to snapshot | cardano-rpc | +| 8 | [Integration testing](08-integration-testing.md) | Validate all tests pass, document coverage gaps | both | +| 9 | [FollowTip streaming](09-follow-tip.md) | Server-streaming chain follower via gRPC | both | + +## NodeKernel coverage + +`NodeKernel` provides access to three subsystems: + +| Subsystem | Access | RPCs covered | +|-----------|--------|-------------| +| **ChainDB** | `getBlockComponent`, `getReadOnlyForkerAtPoint`, `newFollower` | FetchBlock, DumpHistory, FollowTip, ReadTip, all ledger queries | +| **Mempool** | `addLocalTxs`, `getSnapshot` | SubmitTx, ReadMempool | +| **TopLevelConfig** | `getTopLevelConfig` | ReadGenesis, `ExtLedgerCfg` for `answerQuery` | + +Two spec RPCs (ReadTx, ReadData) require indexes the node does not build - these need an external indexer. diff --git a/cardano-rpc/docs/node-kernel-access/analysis-architecture.md b/cardano-rpc/docs/node-kernel-access/analysis-architecture.md new file mode 100644 index 0000000000..d1963b62e4 --- /dev/null +++ b/cardano-rpc/docs/node-kernel-access/analysis-architecture.md @@ -0,0 +1,167 @@ +# Architecture and Current State + +This document covers cardano-rpc's current architecture, spec coverage, and data flow. + +--- + +## cardano-rpc Implementation Overview + +**Three gRPC services** defined in proto files: + +| Service | Methods | Proto | +|---------|---------|-------| +| **Node** (custom) | `GetEra`, `GetProtocolParamsJson` | `cardano/rpc/node.proto` | +| **QueryService** (UTxO RPC) | `ReadParams`, `ReadUtxos`, `SearchUtxos` | `utxorpc/v1beta/query/query.proto` | +| **SubmitService** (UTxO RPC) | `SubmitTx`, `EvalTx` | `utxorpc/v1beta/submit/submit.proto` | + +**Architecture:** +- `RpcEnv` holds config, tracer, and node connection info - injected via `Has` typeclass + `MonadRpc` constraint. +- Server runs on a **Unix domain socket** (insecure/local IPC), default `rpc.sock` next to the node socket. +- Queries the node via standard Node-to-Client local state queries. +- Approximately 2000 lines across 16 modules plus generated proto-lens code in `gen/`. + +**Key modules:** +- `Server.hs` - entry point, registers method groups, top-level exception handler. +- `Type.hs` (~600 lines) - bidirectional conversions between cardano-api/ledger types and protobuf. +- `Predicate.hs` - UTxO pattern matching (address, asset, composite predicates) with address extraction optimisation. +- `Query.hs` - pagination (token = `TxId#OutputIndex`, default 100 items), deterministic sort by TxIn. +- `Submit.hs` - CBOR deserialisation -> era validation -> node submission. + +**Node integration** (in the worktree): +- `--grpc-enable` / `--grpc-socket-path` CLI flags. +- `EnableRpc` / `RpcSocketPath` config keys. +- Runs concurrently via `withAsync` in `Cardano.Node.Run`. +- Structured tracing with metrics (`rpc.request.QueryService.*`, etc.). + +**Integration tests** in `cardano-testnet/test/`: +- `hprop_rpc_query_pparams` - validates all 44 protocol params match ledger. +- `hprop_rpc_transaction` - full round-trip: fetch UTxOs -> build tx -> submit via RPC -> confirm on-chain. + +--- + +### ADR-018 + +**Primary document:** `ADR-018-cardano-rpc-grpc-server.md` + +**Status:** Proposed (2026-03-11) + +**Key decisions:** +- Implements the **UTxO RPC** standard (`utxorpc.org`) plus custom Cardano extensions. +- **Opt-in and experimental** - must be explicitly enabled. +- Unix socket keeps the same local-access-only security model as the existing node socket. +- Remote access via **Envoy reverse proxy** with TLS, rate limiting, and mTLS auth. + +**Current limitations acknowledged:** +- Serialisation overhead (CBOR encode/decode round-trips). +- Connection-per-request cost (Ouroboros handshake). +- Sequential mini-protocol bottleneck. + +**Planned improvements:** +- Direct ledger state access (in-memory TVar/STM reads, bypassing IPC). +- Connection pooling over persistent IPC connections. + +**Dependencies:** `grapesy`, `grpc-spec`, `proto-lens`. + +--- + +## UTxO RPC Spec Coverage + +Current cardano-rpc implements a subset of the UTxO RPC v1beta specification. +The full spec defines four services; cardano-rpc currently covers two partially. + +**QueryService** (8 spec RPCs): + +| RPC | Implemented | Notes | +|-----|-------------|-------| +| ReadParams | Yes | Protocol parameters | +| ReadUtxos | Yes | UTxO lookup by TxIn | +| SearchUtxos | Yes | UTxO search by predicate | +| ReadData | No | Datum by hash - needs PlutusData lookup | +| ReadTx | No | Transaction by hash - needs ChainDB index | +| ReadGenesis | No | Genesis config - available from TopLevelConfig | +| ReadEraSummary | No | Era history - available via QueryEraHistory | +| ReadState | No | Generic state query (e.g. GetStakePoolDistribution) | + +**SubmitService** (5 spec RPCs): + +| RPC | Implemented | Notes | +|-----|-------------|-------| +| SubmitTx | Yes | Transaction submission | +| EvalTx | Yes | Transaction evaluation (7 N2C queries) | +| WaitForTx | No | Streaming - watch for tx confirmation | +| ReadMempool | No | Mempool snapshot - Mempool.getSnapshot | +| WatchMempool | No | Streaming - mempool changes | + +**SyncService** (4 spec RPCs) and **WatchService** (1 spec RPC) are not yet in cardano-rpc's proto files. + +The node kernel access design must support all currently implemented RPCs. +Future RPCs (ReadMempool, ReadGenesis, ReadEraSummary, ReadState) will also benefit from direct access. + +--- + +## Data Path: gRPC to Ledger State + +cardano-rpc does **not** read ledger state directly. +It acts as a **Node-to-Client (N2C) IPC client** that connects to the same node it's running inside, via a Unix domain socket. + +### The connection chain + +``` +gRPC client + -> cardano-rpc server (Unix socket: rpc.sock) + -> cardano-api IPC layer (Unix socket: node.sock) + -> cardano-node's Ouroboros mini-protocol server + -> in-memory ledger state +``` + +### Step by step + +**1. Fresh connection per request** (`Env.hs:16-29`) + +`RpcEnv` holds a `LocalNodeConnectInfo` (socket path + network magic + consensus mode params). +The TODO on line 19 confirms: there's currently **one connection per RPC request** - no connection pooling yet. + +**2. Era determination** - every handler starts with: +```haskell +AnyCardanoEra era <- liftIO . throwExceptT $ determineEra nodeConnInfo +``` +This opens a N2C connection and runs the `QueryCurrentEra` mini-protocol query. + +**3. Local State Query mini-protocol** (`Query.hs:52`, `Node.hs:47`) + +The core mechanism is `executeLocalStateQueryExpr` from `cardano-api`. +This function (`IPC/Internal/Monad.hs:71-93`): +1. Opens a **new** N2C connection via `connectToLocalNodeWithVersion` (Ouroboros network layer -> `Net.connectTo` on the local Unix socket). +2. Negotiates the Node-to-Client protocol version. +3. Runs the **LocalStateQuery** mini-protocol: + - Sends `MsgAcquire` targeting `VolatileTip` (latest ledger state). + - The node acquires a **read snapshot** of ledger state at the tip. + - Sends `MsgQuery` for each query (protocol params, UTxOs, chain point, block number). + - Receives results. + - Sends `MsgRelease` then `MsgDone`. + +**4. Specific queries used:** + +| RPC Method | Ouroboros Queries | +|---|---| +| `ReadParams` | `queryProtocolParameters` + `querySystemStart` + `queryEraHistory` + `queryChainPoint` + `queryChainBlockNo` | +| `ReadUtxos` | `queryUtxo` (by TxIn set) + `querySystemStart` + `queryEraHistory` + chain point + block no | +| `SearchUtxos` | `queryUtxo` (by address set or whole) + `querySystemStart` + `queryEraHistory` + chain point + block no, then client-side predicate filtering + pagination | +| `GetProtocolParamsJson` | `queryProtocolParameters` | +| `SubmitTx` | `submitTxToNodeLocal` (LocalTxSubmission mini-protocol) | +| `EvalTx` | `queryProtocolParameters` + `queryUtxo` (by TxIn) + `querySystemStart` + `queryEraHistory` + `queryStakeDelegDeposits` + `queryDRepState` + `queryStakePoolParameters` | + +**5. Query optimisation in SearchUtxos** (`Query.hs:104-106`) + +Before querying, `extractAddressesFromPredicate` inspects the predicate tree. +If addresses can be extracted, it uses `QueryUTxOByAddress` (indexed lookup in the ledger). +Otherwise it falls back to `QueryUTxOWhole` (fetches entire UTxO set - expensive). + +### Key implications + +- **Every RPC request opens a new Ouroboros connection** - full handshake + protocol negotiation each time. +- **Queries are sequential** within a connection (the LocalStateQuery protocol is inherently sequential). +- **All data passes through CBOR serialisation** - the node serialises ledger state into CBOR over the socket, cardano-api deserialises it, then cardano-rpc re-serialises to protobuf. +- **The node acquires a consistent snapshot** at `VolatileTip` - so protocol params, UTxOs, and chain point within one `executeLocalStateQueryExpr` call are all from the same ledger state. + +This is the architecture the ADR acknowledges as having overhead, with the planned improvement being **direct ledger state access** via TVar/STM reads (bypassing IPC entirely). diff --git a/cardano-rpc/docs/node-kernel-access/analysis-consensus-protocol.md b/cardano-rpc/docs/node-kernel-access/analysis-consensus-protocol.md new file mode 100644 index 0000000000..51e83a0d98 --- /dev/null +++ b/cardano-rpc/docs/node-kernel-access/analysis-consensus-protocol.md @@ -0,0 +1,190 @@ +# Consensus Protocol and Snapshot Consistency + +This document covers the consensus query protocol, in-memory ledger state architecture, and snapshot consistency requirements. + +--- + +## LocalStateQuery Server-Side Protocol + +### Protocol Setup + +The server is initialised in `NodeToClient.hs`: +```haskell +hStateQueryServer = + localStateQueryServer (ExtLedgerCfg cfg) + . ChainDB.getReadOnlyForkerAtPoint getChainDB +``` + +### Protocol States + +``` +StIdle --> Acquire --> StAcquiring --> Acquired --> StAcquired +StAcquired --> Query --> StQuerying --> Result --> StAcquired +StAcquired --> Release --> StIdle +StAcquired --> ReAcquire --> StAcquiring +``` + +### Query Answering + +```haskell +answerQuery config forker query = case query of + BlockQuery blockQuery -> + case sing :: Sing footprint of + SQFNoTables -> + answerPureBlockQuery config blockQuery + <$> atomically (roforkerGetLedgerState forker) + SQFLookupTables -> + answerBlockQueryLookup config blockQuery forker + SQFTraverseTables -> + answerBlockQueryTraverse config blockQuery forker + GetSystemStart -> + pure $ getSystemStart config + GetChainBlockNo -> + headerStateBlockNo . headerState + <$> atomically (roforkerGetLedgerState forker) + GetChainPoint -> + headerStatePoint . headerState + <$> atomically (roforkerGetLedgerState forker) +``` + +### Acquiring: ReadOnlyForker + +When a client sends "Acquire" with a target point: +```haskell +getReadOnlyForkerAtPoint :: + ResourceRegistry m -> + Target (Point blk) -> + m (Either GetForkerError (ReadOnlyForker' m blk)) +``` + +Returns a read-only forker providing: +- Consistent view of ledger state at the requested point. +- Access to ledger tables for queries that need them. +- Automatic resource cleanup. + +### What Node Kernel Access Must Replicate + +1. Accessing the ChainDB (from NodeKernel). +2. Reading current ledger state atomically: `atomically (ChainDB.getCurrentLedger chainDB)`. +3. For specific point queries: `ChainDB.getReadOnlyForkerAtPoint chainDB reg target`. +4. Reading ledger tables if needed. +5. Converting query results to appropriate response types. + +--- + +## Ledger State In-Memory Architecture + +### Node Kernel Structure + +```haskell +data NodeKernel m addrNTN addrNTC blk = NodeKernel + { getChainDB :: ChainDB m blk + , getMempool :: Mempool m blk + , getTopLevelConfig :: TopLevelConfig blk + , ... (peer management, tracers, block forging state, etc.) + } +``` + +Wrapped in `NodeKernelData`: +```haskell +newtype NodeKernelData blk = + NodeKernelData + { unNodeKernelData :: IORef (StrictMaybe (NodeKernel IO RemoteAddress LocalConnectionId blk)) + } +``` + +### Access Points to Ledger State + +**Via ChainDB API:** +```haskell +getCurrentLedger :: STM m (ExtLedgerState blk EmptyMK) +getImmutableLedger :: STM m (ExtLedgerState blk EmptyMK) +getPastLedger :: Point blk -> STM m (Maybe (ExtLedgerState blk EmptyMK)) +getCurrentChain :: STM m (AnchoredFragment (Header blk)) +getReadOnlyForkerAtPoint :: ResourceRegistry m -> Target (Point blk) -> m (Either GetForkerError (ReadOnlyForker' m blk)) +``` + +**Via ReadOnlyForker:** +```haskell +data ReadOnlyForker m l = ReadOnlyForker + { roforkerGetLedgerState :: STM m (l EmptyMK) + , roforkerReadTables :: LedgerTables l KeysMK -> m (LedgerTables l ValuesMK) + , roforkerRangeReadTables :: RangeQueryPrevious l -> m (LedgerTables l ValuesMK, Maybe (TxIn l)) + } +``` + +### Concrete Ledger State Types (from Shelley) + +```haskell +data NewEpochState era = NewEpochState + { nesEL :: EpochNo, nesEs :: EpochState era, nesRu :: PulsingRewUpdate era + , nesPd :: PoolDistr, nesBprev :: BlocksMade, nesBcur :: BlocksMade } + +data UTxOState era = UTxOState + { _utxosUtxo :: UTxO era, _utxosDeposited :: Coin, _utxosFees :: Coin + , _utxosGovState :: GovState era, _utxosDonation :: Coin } +``` + +### Existing In-Process Access Patterns + +**LedgerMetrics Tracer** - traces metrics every N slots using `mapNodeKernelDataIO` and `nkQueryLedger`. + +**Forging Loop** - gets ledger state for a specific point using `getReadOnlyForkerAtPoint`. + +**Peer Selection** - reads immutable ledger via `getImmutableLedger`. + +--- + +## cardano-rpc Implementation Details + +### Protocol Definitions + +1. **`cardano/rpc/node.proto`** - `GetEra()`, `GetProtocolParamsJson()`, Era enum. +2. **`utxorpc/v1beta/query/query.proto`** - `ReadParams()`, `ReadUtxos()`, `SearchUtxos()` with pagination. +3. **`utxorpc/v1beta/submit/submit.proto`** - `SubmitTx()`, `EvalTx()` with CBOR-serialised transactions. +4. **`utxorpc/v1beta/cardano/cardano.proto`** - Complete Cardano data structures (Tx, TxOutput, PParams with 44 fields, certificates, scripts PlutusV1-V4, governance). + +### Key Architectural Patterns + +- **Dependency Injection**: `RpcEnv` + `Has` typeclass + `MonadRpc` constraint + `RIO` monad. +- **Error Handling**: `RpcException` GADT, `throwEither`/`throwExceptT` helpers, top-level handler. +- **Tracing**: Span-based with random IDs, metrics emission, integration with cardano-node tracers. +- **Query Optimisation**: Address extraction from predicates for indexed lookup (`QueryUTxOByAddress`). +- **Pagination**: Token = `TxId#OutputIndex`, deterministic sort by TxIn, default 100 items. +- **Number Representation**: BigInt follows RFC 8949 CBOR encoding (int64, big_u_int, big_n_int). + +### Integration Tests + +1. `hprop_rpc_query_pparams` - validates all 44 protocol params match ledger. +2. `hprop_rpc_transaction` - full round-trip: fetch UTxOs -> build tx -> submit -> confirm on-chain. + +--- + +## Snapshot Consistency + +The current N2C code runs all queries within one `executeLocalStateQueryExpr` call, which acquires a single ledger snapshot at `VolatileTip`. +Protocol parameters, UTxO results, chain tip, and block number all come from the same ledger state. + +This is critical for correctness: +- `EvalTx` queries 7 different things (pparams, UTxOs, system start, era history, stake delegations, DRep state, stake pool params) that must be mutually consistent for transaction evaluation to be valid. +- Query methods use chain tip alongside query results; a mismatch would return UTxOs from one block with a chain tip from another. + +A naive `NodeKernelAccess` design with individual callbacks per query type would break this guarantee. +Each callback would acquire its own `ReadOnlyForker` against a potentially different chain tip. + +The solution is a snapshot-based design: + +```haskell +data NodeKernelAccess = NodeKernelAccess + { nkaWithSnapshot :: forall a. (LedgerSnapshot -> IO a) -> IO a + , nkaSubmitTx :: TxInMode -> IO (SubmitResult TxValidationErrorInCardanoMode) + } + +newtype LedgerSnapshot = LedgerSnapshot + { runQuery :: forall result. QueryInMode result -> IO result } +``` + +`nkaWithSnapshot` acquires a single `ReadOnlyForker` and wraps it. +All queries within one `nkaWithSnapshot` call share the same forker and therefore the same ledger state. + +Note: this snapshot design is forward-looking - it is the target interface for pieces 4-7, not current code (as built, `NodeKernelAccess` is the thinner `chainDb` / `systemStart` / `readEraHistory` record; see `01-node-access-types.md`). diff --git a/cardano-rpc/docs/node-kernel-access/analysis-utxohd-internals.md b/cardano-rpc/docs/node-kernel-access/analysis-utxohd-internals.md new file mode 100644 index 0000000000..57f07486a3 --- /dev/null +++ b/cardano-rpc/docs/node-kernel-access/analysis-utxohd-internals.md @@ -0,0 +1,287 @@ +# UTxO-HD Internals + +This document covers the UTxO-HD backing store architecture and its implications for node kernel access. + +--- + +## UTxO-HD Architecture Impact + +UTxO-HD fundamentally changes what "direct access" means for UTxO data. + +The key insight is that **the UTxO set is no longer in the in-memory ledger state**. +When you call: + +```haskell +atomically (ChainDB.getCurrentLedger chainDB) + :: STM IO (ExtLedgerState blk EmptyMK) +``` + +That `EmptyMK` is literal - the `shelleyLedgerTables` field is `EmptyMK`, meaning **no UTxO data**. +The UTxO lives in a backing store (either an in-memory `TVar` map or an on-disk LSM tree), accessed only through `ReadOnlyForker`. + +### The three tiers of data access + +**Tier 1: Pure ledger state (EmptyMK) - free, STM** +- Protocol parameters (`getPParams`), epoch number, stake distribution, chain point, block number. +- These live in `NewEpochState` which is fully in-memory regardless of UTxO-HD. +- Access: `atomically (getCurrentLedger chainDB)` -> pure extraction. +- No forker needed, no IO, no disk. + +**Tier 2: Point lookups (KeysMK -> ValuesMK) - cheap, needs forker** +- `GetUTxOByTxIn`: look up specific TxIns. +- Access: `roforkerReadTables forker (LedgerTables (KeysMK txinSet))`. +- O(m log n) - goes to backing store for the specific keys. +- On V2/LSM: disk reads for specific keys. +- On V1/InMemory: map lookups in TVar. + +**Tier 3: Table traversals (RangeRead) - expensive, needs forker + iteration** +- `GetUTxOByAddress`, `GetUTxOWhole`: full or filtered scan. +- Access: loop calling `roforkerRangeReadTables` with `QueryBatchSize` (default 100k entries per batch). +- O(n) - must scan entire backing store, applying a predicate filter per batch. +- On V2/LSM: sequential range reads through LSM tree levels, deserialising `TxOutBytes` back to `Core.TxOut era`. +- The existing consensus code in `answerShelleyTraversingQueries` does exactly this loop. + +### What this means for the direct access design + +**For Tier 1 (ReadParams pparams):** The win is massive and simple. +Replace the entire N2C round-trip with a single STM read. +No forker, no table access, no disk. +This is the easy win. + +**For Tier 2 (ReadUtxos by TxIn):** The win is real but more nuanced. +You skip the N2C + CBOR overhead, but you still need: +1. Acquire a `ReadOnlyForker` via `ChainDB.getReadOnlyForkerAtPoint` (allocates resources). +2. Call `roforkerReadTables` (may hit disk on LSM backend). +3. Close the forker. + +The backing store IO is the same cost either way - you just eliminate the IPC + serialisation wrapper. +For small lookups this is a big relative improvement. + +**For Tier 3 (SearchUtxos by address / whole UTxO):** The most interesting case. +Currently: +- N2C server does the full range-read loop internally, accumulates the entire result, CBOR-serialises the whole `UTxO era`, sends it over the socket. +- cardano-api deserialises the whole thing. +- cardano-rpc converts to protobuf. + +With direct access: +- cardano-rpc can call `roforkerRangeReadTables` directly in a loop. +- Can apply predicate filtering **per batch** (already happens on the consensus side, now happens in-process). +- Can implement **true streaming pagination** - instead of reading the entire filtered result and then slicing, stop after collecting enough items for one page. +- Converts ledger types -> protobuf directly per batch, no intermediate CBOR or cardano-api types. + +This is where the design gets the most interesting improvement: **pagination can be pushed down to the backing store level**. +Currently `SearchUtxos` fetches the entire filtered UTxO set, sorts it, and slices. +With direct range reads, you can stop early after filling a page. + +**For tx submission (Mempool.addTx):** The tx needs to be a `GenTx blk` (consensus type), not the cardano-api `TxInMode` wrapper. +The current path is: CBOR bytes -> `deserialiseFromCBOR` -> cardano-api `Tx era` -> `toConsensusGenTx` -> N2C LocalTxSubmission -> mempool. +Direct path: CBOR bytes -> `deserialiseFromCBOR` -> `GenTx blk` -> `Mempool.addTx`. +Skips the entire protocol layer. + +### The HFC complication + +`ChainDB` and `NodeKernel` are parameterised by `CardanoBlock StandardCrypto`, which is a `HardForkBlock` across all eras. +The `LedgerTables` use `CanonicalTxIn` and `HardForkTxOut` - canonical representations that abstract across eras. +To get era-specific `Core.TxOut era`, you need `ejectLedgerTables` from the HFC layer. + +The existing `answerShelleyLookupQueries` and `answerShelleyTraversingQueries` already handle this via injection/ejection functions passed as parameters. +cardano-rpc's direct access layer would need to replicate this pattern - or better, reuse the existing `answerQuery` function from consensus with a directly-obtained forker rather than one obtained through the protocol. + +### Depth of integration + +There's a spectrum of how deep cardano-rpc reaches into consensus: + +1. **Shallow**: Reuse `answerQuery` from consensus - get a forker, call `answerQuery cfg forker query`, get back the same types the N2C server would produce. + Eliminates IPC + CBOR serialisation but still uses the consensus query dispatch. + Simplest, lowest risk. + +2. **Medium**: Call `roforkerReadTables` / `roforkerRangeReadTables` directly, handle HFC ejection, convert ledger types -> protobuf. + More code but enables streaming pagination optimisation. + +3. **Deep**: Bypass forkers for Tier 1 queries entirely (just STM reads), use forkers only for Tier 2/3. + Mix-and-match per query type. + +Option 3 is what was outlined in the earlier design. +The risk is coupling to consensus internals. +But Tier 1 queries via STM are so simple and stable that the coupling is minimal, and the existing `nkQueryLedger` helper in `Queries.hs` already does exactly this for metrics. + +--- + +## LedgerTables Type Family System + +### Core Type Definition + +**File:** `ouroboros-consensus/Ouroboros/Consensus/Ledger/Tables/Basics.hs` + +```haskell +type LedgerTables :: LedgerStateKind -> MapKind -> Type +newtype LedgerTables l mk = LedgerTables + { getLedgerTables :: mk (TxIn l) (TxOut l) + } +``` + +Where `MapKind` is `Type -> Type -> Type` (key and value parameters). + +### MapKind Definitions + +**File:** `ouroboros-consensus/Ouroboros/Consensus/Ledger/Tables/MapKind.hs` + +- **EmptyMK**: `data EmptyMK k v = EmptyMK` - no UTxO data present. +- **KeysMK**: `newtype KeysMK k v = KeysMK (Set k)` - only key set. +- **ValuesMK**: `newtype ValuesMK k v = ValuesMK {getValuesMK :: Map k v}` - full key-value map. +- **DiffMK**: `newtype DiffMK k v = DiffMK {getDiffMK :: Diff k v}` - changes (Insert v | Delete). +- **TrackingMK**: `data TrackingMK k v = TrackingMK !(Map k v) !(Diff k v)` - values + accumulated diffs. +- **SeqDiffMK**: finger tree of diffs. + +### What EmptyMK Means Concretely + +When you have `LedgerState era EmptyMK`: +- The `EmptyMK` represents that **the UTxO map data is literally not present** in this ledger state. +- The Consensus layer stores actual UTxO data separately on disk (in the backing store). +- The UTxO can be "stowed" (extracted from the ledger) or "unstowed" (injected back into the ledger). + +### Shelley-Specific Instance + +```haskell +type instance TxIn (LedgerState (ShelleyBlock proto era)) = BigEndianTxIn +type instance TxOut (LedgerState (ShelleyBlock proto era)) = Core.TxOut era +``` + +`BigEndianTxIn` is a wrapper that ensures proper byte ordering for serialisation. + +### Cardano HFC Handling + +```haskell +type instance TxIn (LedgerState (HardForkBlock xs)) = CanonicalTxIn xs +type instance TxOut (LedgerState (HardForkBlock xs)) = HardForkTxOut xs +``` + +Key functions: `injectLedgerTables` and `ejectLedgerTables` transform between era-specific and canonical representations via `bimapLedgerTables`. + +### How cardano-api Gets UTxO from Queries + +The flow: +1. cardano-api calls `queryUtxo` with a filter. +2. Converts to `toConsensusQuery` -> `GetUTxOWhole` / `GetUTxOByAddress` / `GetUTxOByTxIn`. +3. Consensus queries in-memory LedgerState (with `ValuesMK`). +4. Result comes as `Shelley.UTxO (ShelleyLedgerEra era)`. +5. Converts via `fromLedgerUTxO` to cardano-api `UTxO era`. + +--- + +## UTxO-HD Architecture Report + +### LedgerDB and Backing Store + +The LedgerDB is responsible for: +- Maintaining in-memory ledger state at the tip. +- Maintaining past k in-memory ledger states (supports rollback). +- Providing LedgerTables at any of the last k states. +- Storing snapshots on disk. +- Flushing LedgerTable differences to backing store. + +### Backing Store Abstraction + +```haskell +data BackingStore m keys key values diff = BackingStore + { bsClose :: m () + , bsCopy :: SerializeTablesHint values -> FS.FsPath -> m () + , bsValueHandle :: m (BackingStoreValueHandle m keys key values) + , bsWrite :: SlotNo -> WriteHint diff -> diff -> m () + , bsSnapshotBackend :: SnapshotBackend + } +``` + +### Two Backend Implementations + +**V1 Backend - InMemory:** +- Uses a `TVar` holding full map. +- All data stays in memory. +- Used for testing or small deployments. + +**V2 Backend - LSM (Log-Structured Merge):** +- Uses LSM trees from the `lsm-tree` library. +- Keys and values serialised to bytes. +- Uses indexed packing (`IndexedMemPack`) for efficient serialisation. + +### The Forker + +```haskell +data Forker m l = Forker + { forkerReadTables :: LedgerTables l KeysMK -> m (LedgerTables l ValuesMK) + , forkerRangeReadTables :: RangeQueryPrevious l -> m (LedgerTables l ValuesMK, Maybe (TxIn l)) + , forkerGetLedgerState :: STM m (l EmptyMK) -- NO UTxO + , forkerReadStatistics :: m Statistics + , forkerPush :: l DiffMK -> m () + , forkerCommit :: STM m () + } +``` + +**Critical insight:** `forkerGetLedgerState` returns `l EmptyMK` - **the UTxO data is NOT included**. +You must explicitly call `forkerReadTables`. + +### Range Queries and BatchSize + +`QueryBatchSize` defaults to 100,000 entries. +`roforkerRangeReadTables` returns up to that many entries plus one additional key for pagination. +Client iterates until `Nothing` (end of table). + +### ReadOnlyForker vs getCurrentLedger + +| Aspect | getCurrentLedger | getReadOnlyForker | +|--------|------------------|------------------| +| Returns | `l EmptyMK` (STM) | `Forker m l` (IO) | +| UTxO access | NO | YES via `roforkerReadTables` | +| Speed | Fast (STM, in-memory) | Slower (allocates, I/O) | +| Use case | Ledger state queries, consensus | Query answering requiring UTxO | +| Resource management | None | Must close forker | + +### Three Footprint Categories for Shelley Queries + +**QFNoTables:** `GetLedgerTip`, `GetEpochNo`, `GetCurrentPParams`, etc. - uses `answerPureBlockQuery`, just in-memory ledger state. + +**QFLookupTables:** `GetUTxOByTxIn` - uses `answerShelleyLookupQueries`, O(m * log n) for m inputs. + +**QFTraverseTables:** `GetUTxOByAddress`, `GetUTxOWhole` - uses `answerShelleyTraversingQueries`, O(n) full scan with batched range reads. + +### UTxO-HD Pipeline Diagram + +``` +Client Query (e.g., GetUTxOByAddress) + | + v +answerShelleyTraversingQueries + | + v +getReadOnlyForker(cfg, pt) -> Forker + | + v +roforkerRangeReadTables (QueryBatchSize = 100k) + | + v +BackingStore.bsvhRangeRead + |- V1: reads from TVar (InMemory) + |- V2: reads from LSM tree + | + v +Apply DbChangelog diffs (forward apply) [V1 only] + | + v +Filter by predicate (e.g., address match) + | + v +Accumulate into result (loop until end) + | + v +Return UTxO era to client +``` + +### Why MapKind Matters + +The MapKind parameter elegantly separates concerns: +- **`EmptyMK`** = "I have the ledger rules state but no UTxO". +- **`ValuesMK`** = "I have loaded UTxO from disk". +- **`KeysMK`** = "Here are the keys I want to read". +- **`DiffMK`** = "Here are the changes since last checkpoint". + +This allows the type system to enforce that you cannot accidentally use UTxO when you have not loaded it. diff --git a/cardano-rpc/docs/node-kernel-access/implementation-plan.md b/cardano-rpc/docs/node-kernel-access/implementation-plan.md new file mode 100644 index 0000000000..cdf7d046ba --- /dev/null +++ b/cardano-rpc/docs/node-kernel-access/implementation-plan.md @@ -0,0 +1,141 @@ +# Node Kernel Access for cardano-rpc: Implementation Plan + +Related ADR: [ADR-019](./ADR-019-node-kernel-access-for-cardano-rpc.md) + +--- + +## Overview + +Changes span two submodules: **cardano-api** (contains cardano-rpc) and **cardano-node**. +Work the cardano-rpc side first, then the cardano-node side. + +Detailed step-by-step implementation is in the numbered story files: + +- [01-node-access-types.md](01-node-access-types.md) - `NodeKernelAccess` types, `Env`, `Monad`, `Tracing`, `Server`, and cabal wiring +- [02-fetchblock-proto-and-handler.md](02-fetchblock-proto-and-handler.md) - SyncService proto, codegen, `fetchBlockMethod`, server registration +- [03-mk-node-access-and-wiring.md](03-mk-node-access-and-wiring.md) - `mkNodeKernelAccess` in cardano-node, `Run.hs` wiring, E2E test, tracing +- [04-rewrite-query-methods.md](04-rewrite-query-methods.md) - Rewrite `Query.hs` to use snapshot pattern +- [05-rewrite-submit-method.md](05-rewrite-submit-method.md) - Rewrite `Submit.hs` to use `NodeKernelAccess` +- [06-rewrite-eval-method.md](06-rewrite-eval-method.md) - Rewrite `Eval.hs` to use snapshot pattern +- [07-rewrite-node-methods.md](07-rewrite-node-methods.md) - Rewrite `Node.hs` to use snapshot pattern +- [08-integration-testing.md](08-integration-testing.md) - Integration testing and validation + +--- + +## Current state analysis + +### Key code paths being replaced + +All query/submit methods currently follow the same N2C pattern: + +```haskell +-- 1. Grab the LocalNodeConnectInfo from the reader env +nodeConnInfo <- grab + +-- 2. Determine the current era via N2C +AnyCardanoEra era <- liftIO . throwExceptT $ determineEra nodeConnInfo + +-- 3. Execute a local state query expression over N2C +(result, ...) <- liftIO . (throwEither =<<) $ + executeLocalStateQueryExpr nodeConnInfo VolatileTip $ do + result <- throwEither =<< throwEither =<< queryXxx sbe ... + chainPoint <- throwEither =<< queryChainPoint + blockNo <- throwEither =<< queryChainBlockNo + pure (result, chainPoint, blockNo) +``` + +Every query method also runs `querySystemStart` and `queryEraHistory` inside the same `executeLocalStateQueryExpr` call for slot-to-timestamp conversion. + +This pattern appears in: +- `Query.hs`: `readParamsMethod`, `readUtxosMethod`, `searchUtxosMethod` +- `Eval.hs`: `evalTxMethod` (queries 7 things: protocol parameters, UTxO by TxIn set, system start, era history, stake deleg deposits, DRep state, stake pool parameters) +- `Submit.hs`: `submitTxMethod` (era detection + `submitTxToNodeLocal`) +- `Node.hs`: `getProtocolParamsJsonMethod` (era detection + query) + +### Current environment wiring + +``` +RpcEnv (Env.hs) + |- config :: RpcConfig + |- tracer :: Tracer m TraceRpc + +- rpcLocalNodeConnectInfo :: LocalNodeConnectInfo <- REPLACE with IORef + +Monad.hs: + instance Has LocalNodeConnectInfo RpcEnv <- REPLACE + type MonadRpc: Has LocalNodeConnectInfo e <- REPLACE + +Server.hs: + runRpcServer :: Tracer IO TraceRpc -> (RpcConfig, NetworkMagic) -> IO () + calls mkLocalNodeConnectInfo to build the env <- REPLACE +``` + +### Current tracing (Tracing.hs + Rpc.hs) + +``` +TraceRpcSubmit + |- TraceRpcSubmitN2cConnectionError SomeException <- REMOVE + |- TraceRpcSubmitTxDecodingError DecoderError (keep) + |- TraceRpcSubmitTxValidationError ... (keep) + +- TraceRpcSubmitSpan TraceSpanEvent (keep) +``` + +The N2C error is wrapped in `Submit.hs` via `tryAny` + `first TraceRpcSubmitN2cConnectionError`. + +--- + +## File summary + +### New files (as built) + +| File | Purpose | +|------|---------| +| `cardano-api/cardano-rpc/src/Cardano/Rpc/Server/NodeKernelAccess.hs` | `mkNodeKernelAccess`, `fetchBlock` and `grabNodeKernelAccess`; the `NodeKernelAccess` record itself lives in `Cardano/Rpc/Server/NodeKernelAccess/Type.hs`. No `withNodeKernelAccess` was built. | +| `cardano-node/cardano-node/src/Cardano/Node/Run.hs` (modified, not new) | `mkNodeKernelAccess` is called inline here; no dedicated cardano-node module was created. | + +### Modified files (12) + +| File | Change | Story | +|------|--------|-------| +| `cardano-api/cardano-rpc/src/Cardano/Rpc/Server.hs` | New signature, re-export `NodeKernelAccess` + `LedgerSnapshot`, IORef wiring, register `methodsSyncRpc` | [01](01-node-access-types.md), [02](02-fetchblock-proto-and-handler.md) | +| `cardano-api/cardano-rpc/src/Cardano/Rpc/Server/Internal/Env.hs` | `RpcEnv` holds `IORef`, drop `LocalNodeConnectInfo` | [01](01-node-access-types.md) | +| `cardano-api/cardano-rpc/src/Cardano/Rpc/Server/Internal/Monad.hs` | `Has` instance + `MonadRpc` update | [01](01-node-access-types.md) | +| `cardano-api/cardano-rpc/src/Cardano/Rpc/Server/Internal/Tracing.hs` | Replace N2C trace with ledger-access traces, add `TraceRpcSync` | [01](01-node-access-types.md) | +| `cardano-api/cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/Query.hs` | Rewrite 3 query methods to use snapshot pattern | [04](04-rewrite-query-methods.md) | +| `cardano-api/cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/Eval.hs` | Rewrite `evalTxMethod` to use snapshot pattern (7 queries) | [06](06-rewrite-eval-method.md) | +| `cardano-api/cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/Submit.hs` | Rewrite submit with snapshot for era detection | [05](05-rewrite-submit-method.md) | +| `cardano-api/cardano-rpc/src/Cardano/Rpc/Server/Internal/Node.hs` | Rewrite `getProtocolParamsJsonMethod` to use snapshot pattern | [07](07-rewrite-node-methods.md) | +| `cardano-api/cardano-rpc/cardano-rpc.cabal` | Add `NodeKernelAccess` module, proto gen modules | [01](01-node-access-types.md), [02](02-fetchblock-proto-and-handler.md) | +| `cardano-node/cardano-node/src/Cardano/Node/Run.hs` | IORef + writeIORef in kernel hook | [03](03-mk-node-access-and-wiring.md) | +| `cardano-node/cardano-node/src/Cardano/Node/Tracing/Tracers/Rpc.hs` | Update trace instances | [03](03-mk-node-access-and-wiring.md) | +| `cardano-node/cardano-node/cardano-node.cabal` | Add new module | [03](03-mk-node-access-and-wiring.md) | + +### Unchanged + +| File | Why | +|------|-----| +| `cardano-rpc/src/Cardano/Rpc/Server/Config.hs` | `nodeSocketPath` still needed for socket path derivation | +| `cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/Type.*` | The N2C-to-node-kernel migration does not touch the type conversions (now split across `Type.hs` and the `Type/*.hs` submodules). | +| `cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/Predicate.hs` | Filtering untouched | +| Testnet test files | gRPC client API unchanged | + +--- + +## Verification + +1. `nix build .#cardano-rpc` in cardano-api submodule +2. `nix build .#cardano-rpc:test:cardano-rpc-test` - unit tests +3. `nix build .#cardano-node` in cardano-node submodule +4. Integration tests via testnet (if available) + +--- + +## Risks and mitigations + +| Risk | Status | Mitigation | +|------|--------|------------| +| `answerQuery` API differs from expected signature | **Verified** (2026-05-22) | Signature matches: `ExtLedgerCfg blk -> ReadOnlyForker' m blk -> Query blk result -> m result` | +| Forker lifecycle - leaking forkers on exceptions | **Addressed** | `bracket` pattern in `nkaWithSnapshot`; `withRegistry` provides additional safety net | +| `QueryCurrentEra` not available via `toConsensusQuery` | **Verified: no risk** | `toConsensusQuery` handles `QueryCurrentEra` by wrapping as `BlockQuery (QueryHardFork GetCurrentEra)` | +| `addLocalTxs` API changed in recent consensus | **Verified** (2026-05-22) | Name and signature confirmed; use `MkSolo` constructor on GHC 9.10 | +| Concurrent reads during kernel initialisation race | Low risk | `IORef` write in `rnNodeKernelHook` happens before any RPC can succeed; `withNodeKernelAccess` checks atomically | +| `Target` / `withRegistry` import paths wrong | **Fixed** | `Target` from `Ouroboros.Network.Protocol.LocalStateQuery.Type`; `withRegistry` from `Control.ResourceRegistry` | diff --git a/cardano-rpc/docs/node-kernel-access/plan-eon-ledger-constraints.md b/cardano-rpc/docs/node-kernel-access/plan-eon-ledger-constraints.md new file mode 100644 index 0000000000..ac36fcb657 --- /dev/null +++ b/cardano-rpc/docs/node-kernel-access/plan-eon-ledger-constraints.md @@ -0,0 +1,81 @@ +# Plan: replace fine-grained eons with ledger-class constraints + +## Goal + +Reviewer guidance: cardano-rpc should only use the experimental `Era era` or `ShelleyBasedEra era`. +Any other eon (`AllegraEraOnwards`, `MaryEraOnwards`, `AlonzoEraOnwards`, `BabbageEraOnwards`, `ConwayEraOnwards`, `ShelleyToBabbageEra`) indicates a cardano-api function that needs converting. +This plan removes those eons from cardano-rpc: experimental `Era` where the code is Conway-onwards only, ledger-class constraints (`AnyEra*` from cardano-ledger-api) where the code must serve every Shelley-based era. + +## Verified foundations + +- The pinned `cardano-ledger-api 1.13.0.0` already ships the `AnyEra*` class family (verified in the installed interface files: `AnyEraTxBody`, `vldtTxBodyG`, `mintTxBodyG` present); no ledger bump is required. +- The ledger class chain is Dijkstra-total: `MaryEraTxBody`, `AlonzoEraTxBody`/`AlonzoEraTx`/`AlonzoEraTxWits`, `BabbageEraTxBody` and `ConwayEraTxBody` all have Dijkstra instances; only `ShelleyEraTxCert` excludes Dijkstra, by design (`AtMostEra "Conway"` superclass). +- `instance AnyEraTxBody DijkstraEra` and `instance AnyEraTxCert DijkstraEra` are empty instance bodies: era-gated getters degrade to `Nothing` instead of crashing, unlike cardano-api's eon bundles with their `error "TODO Dijkstra"` stubs. +- The experimental `Era` is a deliberate Conway+Dijkstra sliding window and cannot serve the historical arm; `sbeToEra` is the partial bridge. +- The current eon usage in cardano-rpc is 12 dispatch sites and 5 pure constraint-obtainment sites across Tx.hs, Certificate.hs, ProtocolParameters.hs and two test modules. + +## Phase 0 - spikes (no production changes) + +1. Check `AnyEraTxCert` variant coverage at the 1.13 pin against the 19 certificate oneof arms `Certificate.hs` maps; the phase 4 shape depends on this. +2. Pin down the `TopTx`/`TxLevel` axis: the collateral getters and `isValidTxL` are typed over `TxBody TopTx era`/`Tx TopTx era`; determine how cardano-api's `ShelleyTx` payload lines up with that parameter at the current pin. +3. Name-by-name check that every getter the nine Tx.hs gates need exists at 1.13 (mint, reference inputs, collateral inputs, collateral return, total collateral, is-valid, datums, redeemers, proposals); the vendored 1.14 checkout is not proof for the pin. + +Gate: written findings; go/no-go for the phase 3 and phase 4 shapes. +Fallback if 1.13 lacks needed getters: bump the CHaP index for a newer cardano-ledger-api (separate decision, `/bump-indices`). + +## Phase 1 - upstream cardano-api: extend `EraCommonConstraints` + +- Add `L.ConwayEraPParams (LedgerEra era)` to `EraCommonConstraints` in `Cardano.Api.Experimental.Era`. +- Drop the `conwayEraOnwardsConstraints (convert era)` detour in `utxoRpcPParamsToProtocolParams` (`Type/ProtocolParameters.hs`); `obtainCommonConstraints era` alone then suffices. +- Changelog fragment for cardano-api (compatible: the bundle provides more, callers unaffected). + +Gate: `cabal build cardano-rpc && cabal test cardano-rpc-test` from `/work`, all 71 tests, zero warnings. +Independent of every other phase; can go first as its own PR. + +## Phase 2 - upstream cardano-api: convert `toScriptIndex` - REJECTED (attempted 2026-07-15, reverted) + +- The attempt worked inside this repo (build and all 71 tests green) but broke cardano-node: `Cardano.Node.Tracing.Era.Shelley` calls `Api.toScriptIndex alonzoOnwards` at two sites, and other out-of-repo consumers likely exist. +- Converting `toScriptIndex` therefore needs a coordinated cross-repo migration, which is out of scope for this PR; the signature stays `AlonzoEraOnwards era -> ...`. +- Consequence for phase 3: the redeemer-indexing site in `Type/Tx.hs` keeps one local eon derivation for the `toScriptIndex` argument (`Eval.hs` already derives it totally from the experimental `Era` via `convert`, which satisfies the reviewer's rule). +- If upstream ever coordinates the change, the working shape is recorded here: `ShelleyBasedEra era` parameter with an `L.AlonzoEraScript (ShelleyLedgerEra era)` constraint, purposes projected through the `L.AlonzoEraScript`/`L.ConwayEraScript` class methods (`toSpendingPurpose` etc.), Conway-onwards purposes matched at the concrete constructors; downstream eon-holding callers pass `convert `. + +## Phase 3 - Tx.hs: replace the nine eon gates with `AnyEra*` getters + +- Add one local constraint boundary in cardano-rpc: a function matching the seven `ShelleyBasedEra` constructors (the allowed GADT) and providing the `AnyEraTx`/`AnyEraTxBody`/`AnyEraTxWits` instances for `ShelleyLedgerEra era`; at concrete constructors all instances resolve with no eon machinery, and the Dijkstra arm compiles because the ledger instances exist. +- This removes the `shelleyBasedEraConstraints` crash from the transaction path: Dijkstra blocks convert instead of erroring. +- Replace each `forShelleyBasedEraInEon` gate with the corresponding getter, mapping `Nothing` to the current proto default (`[]`, `mempty`, `defMessage`, `IsValid True`): mint, reference inputs, collateral inputs, collateral return + total collateral, is-valid, datums, redeemers, proposals, validity interval. +- The Shelley `ttlTxBodyL` special case stays as a concrete `ShelleyBasedEraShelley` arm if `vldtTxBodyG`'s total getter does not subsume it (phase 0 answers this). +- Delete the proposals tripwire comment: with a `ConwayEraTxBody`-backed getter the Dijkstra arm is live code, not a dead arm. +- Keep the `ShelleyBasedEra era` parameter: the output and address sub-conversions (`fromShelleyTxOut`, `serialiseAddress`) still need it, and it feeds the phase 2 `toScriptIndex`. +- Tests: the six per-era totality properties and the Conway projection property are the oracle and must stay green unchanged; keep the four `forShelleyBasedEraMaybeEon` feature oracles in `FetchBlockTx.hs` (asking whether an era supports a feature is legitimate regardless of how production answers it). + +Gate: build + all 71 tests, zero warnings; wire behaviour byte-identical for Shelley..Conway. +Note for the changelog: Dijkstra behaviour changes from runtime crash to working conversion. + +## Phase 4 - Certificate.hs - DONE (getter approach rejected, concrete dispatch implemented) + +- The phase 0.1 coverage claim did not survive a source check: `AnyEraTxCert` cannot read pre-Conway stake delegations (`anyEraToDelegTxCert = const Nothing` for Shelley..Babbage and no legacy `DelegStakeTxCert` getter exists), and a getter-based encoder needs an error fallback for unmatched certificates, reintroducing hidden partiality. +- Implemented instead as an exploded seven-constructor `ShelleyBasedEra` match onto the three constructor-exhaustive family matchers (`shelleyTxCertToUtxoRpcCertificate`, `conwayTxCertToUtxoRpcCertificate`, `dijkstraTxCertToUtxoRpcCertificate`); the compiler proves totality per era and `caseShelleyToBabbageOrConwayEraOnwards` is no longer used. +- The Dijkstra certificate conversion is live again (delegation deposits bridge losslessly through `dijkstraToConwayDelegCert`; `cardano-ledger-dijkstra` is a direct dependency once more), so the whole Dijkstra transaction path now converts. + +Gate passed: build + all 71 tests, zero warnings; the certificate-arm routing assertions in the Conway projection property unchanged. + +## Phase 5 - upstream cardano-api conversions - PARTIALLY DONE + +- DONE: `toScriptIndexAlonzo`, `toScriptIndexConway` and `toScriptIndexDijkstra` are exported from `Cardano.Api.Tx.Internal.Body` (additive, signatures generalised over the ledger era); cardano-rpc's local duplicates are deleted and the redeemer dispatch uses the exports. +- DEFERRED: relaxing `queryDRepState` (and siblings) and `queryStakeDelegDeposits` to `ShelleyBasedEra era` breaks out-of-repo callers (cardano-cli `EraBased/Query/Run.hs`, cardano-testnet `Components/Query.hs` pass eon witnesses), the same cross-repo coordination class as the rejected phase 2. `Eval.hs` already derives these eons totally from the experimental `Era` via `convert`, which satisfies the reviewer's rule. +- VERIFIED for the future relaxation: the consensus version-gating table (`Ouroboros.Consensus.Shelley.Ledger.Query`) marks `GetStakeDelegDeposits` as `const True` - valid at every protocol version - so cardano-api's `BabbageEraOnwards` restriction is API decoration only; `GetDRepState` and friends are `(>= v8)` version-guarded at runtime. +- DEFERRED: converting `toLedgerValue` away from `MaryEraOnwards` breaks cardano-cli callers; `Predicate.hs`'s test-only witness already derives totally via `convert` from the experimental `Era`. + +## Phase 6 - cleanup - DONE + +- AGENTS.md documents the exploded `ShelleyBasedEra` dispatch as the accepted pattern, the `AnyEra*` getters, `anyEraTxConstraints`, and the `AnyEraTxCert` legacy-delegation gap. +- No unused eon imports remain (all builds pass with zero warnings and `-Wunused-imports`). +- Final verification passed: `cabal build cardano-rpc && cabal test cardano-rpc-test && cabal build cardano-testnet-test` from `/work`, zero warnings, all 71 tests green. + +## Risks and open questions + +- The `TopTx` axis (phase 0.2) is the main mechanical unknown; four getters depend on it. +- The 1.13 pin may lack getters the vendored 1.14 has (phase 0.3); the fallback is a CHaP bump. +- Dijkstra tx conversion changes from crash to graceful output; confirm that is wanted now rather than after upstream Dijkstra wiring completes. +- If phase 4 lands on the fallback, `caseShelleyToBabbageOrConwayEraOnwards` remains in exactly one place; confirm with the reviewer that the certificate dispatch is an accepted exception to the eon rule. diff --git a/cardano-rpc/docs/node-kernel-access/plan-readgenesis-phases.md b/cardano-rpc/docs/node-kernel-access/plan-readgenesis-phases.md new file mode 100644 index 0000000000..1db6aaf336 --- /dev/null +++ b/cardano-rpc/docs/node-kernel-access/plan-readgenesis-phases.md @@ -0,0 +1,247 @@ +# ReadGenesis implementation phases + +Implements the ReadGenesis QueryService method. +Each phase compiles independently and can be reviewed as a separate commit. + +Genesis data is static - it never changes after node startup. +All data is extracted once in `mkNodeKernelAccess` and stored as a pure field on `NodeKernelAccess`. +It all comes from one source, and it is not a disk read: + +- **`ProtocolInfoArgs`** (threaded from the boot-time `SomeConsensusProtocol`): the full, uncompacted `TransitionConfig LatestKnownEra` - Shelley, Alonzo, Conway and Dijkstra genesis with `sgInitialFunds` and `sgStaking` intact - plus the Shelley genesis hash computed at boot, plus the Byron `Cardano.Chain.Genesis.Config` (genesis data and hash), read directly off `CardanoProtocolParams`'s `byronProtocolParams` field with no hard-fork navigation needed. + +## Related documents + +- [story-read-genesis.md](../story-read-genesis.md) - the user story: acceptance criteria, out-of-scope list, open questions. +- [README.md](README.md) - node-kernel-access overview; capability table mapping `TopLevelConfig` to ReadGenesis. +- [analysis-architecture.md](analysis-architecture.md) - why ReadGenesis benefits from direct node access rather than N2C IPC. +- [03-mk-node-access-and-wiring.md](03-mk-node-access-and-wiring.md) - how `NodeKernelAccess` is populated from the node kernel hook. + Partially stale: the implemented `mkNodeKernelAccess` lives in cardano-rpc's `NodeKernelAccess.hs` and additionally takes a `TopLevelConfig`. +- ADR-019 "Node kernel access for cardano-rpc" (cardano-node-wiki, `docs/ADR-019-node-kernel-access-for-cardano-rpc.md`) - the architecture decision this work builds on; its capability table backs ReadGenesis with `TopLevelConfig`. +- [IntersectMBO/cardano-api#1217](https://github.com/IntersectMBO/cardano-api/issues/1217) - the GitHub issue tracking this method. + The issue's proposed data source (`queryLedgerConfig` over N2C IPC) predates node kernel access; this plan supersedes it. +- [UTxO RPC query spec](https://utxorpc.org/query/spec/) - `ReadGenesisRequest`/`ReadGenesisResponse` shape. +- [CIP-34](https://cips.cardano.org/cip/CIP-34) - Cardano's CAIP-2 chain identifier. + +## Phase 1: Proto service method + stub handler + registration + tracing + +Atomic: proto change requires matching handler registration. + +- Add `rpc ReadGenesis(ReadGenesisRequest) returns (ReadGenesisResponse)` to the `QueryService` block in `proto/utxorpc/v1beta/query/query.proto` +- Regenerate: `nix develop --command bash -c "cd cardano-rpc && buf generate proto"` +- Add `TraceRpcQueryReadGenesisSpan TraceSpanEvent` constructor to `TraceRpcQuery` in `Tracing.hs` +- Add stub `readGenesisMethod` in `Query.hs` returning `GrpcUnimplemented` +- Register in `methodsUtxoRpc` in `Server.hs` - position must match `ServiceMethods` order (verify against generated code) +- Update cardano-node `Tracers/Rpc.hs` with new trace constructor + +Build: `cabal build cardano-rpc && cabal build cardano-node` + +## Phase 2: Genesis threading + NKA genesis bundle + envelope fields + +Adds the static genesis data to `NodeKernelAccessF` and populates the response envelope. + +Node-side wiring (cardano-node): + +- Genesis file paths still live in `NodeConfiguration` (`ncProtocolConfig` holds `npcByronGenesisFile`, `npcShelleyGenesisFile`, `npcAlonzoGenesisFile`, `npcConwayGenesisFile`, `Cardano.Node.Types`, `GenesisFile` newtypes over `FilePath`), but cardano-rpc no longer sees any of them directly. + The Shelley genesis is still read once, at boot, by `mkSomeConsensusProtocolCardano` (`Cardano.Node.Protocol.Cardano`) via `Shelley.readGenesis npcShelleyGenesisFile npcShelleyGenesisFileHash` - that call was always there for consensus's own needs, and cardano-rpc now reuses its result instead of reading the file again. +- `mkConsensusProtocol` `coerce`s the hash it already has (node's `GenesisHash` and cardano-api's `GenesisHashShelley` are both newtypes over the same Blake2b-256 hash) into a `GenesisHashShelley`, and returns it alongside `SomeConsensusProtocol` as a tuple, rather than as a field on `SomeConsensusProtocol` itself. + The exported `SomeConsensusProtocol` constructor keeps its upstream (`ouroboros-consensus`) shape unchanged; it carries no extra genesis-hash field. +- `nc :: NodeConfiguration` is in scope at both wiring sites in `handleSimpleNode` (`cardano-node/src/Cardano/Node/Run.hs`): the `rpcServerLoop` launch and the `rnNodeKernelHook` lambda that calls `mkNodeKernelAccess`. +- Pass the `GenesisHashShelley` returned alongside `SomeConsensusProtocol` and the `ProtocolInfoArgs` already in scope into `mkNodeKernelAccess`, instead of a `FilePath`. + `ProtocolInfoArgs`'s Cardano-block instance (`ProtocolInfoArgsCardano`, `Cardano.Api.Consensus.Internal.Protocol`) wraps `CardanoProtocolParams`, whose `cardanoLedgerTransitionConfig :: TransitionConfig LatestKnownEra` field is the already-parsed Shelley-onwards transition config; `cardanoLedgerTransitionConfig` is re-exported from `Cardano.Api.Consensus`. +- The node kernel access warnings are RPC traces, not startup traces. + `StartupTrace` (`Cardano.Node.Startup`) therefore loses its `RpcUnsupportedBlockType` constructor, along with every clause for it in `Cardano.Node.Tracing.Tracers.Startup` (`forMachine`, `namespaceFor`, `severityFor`, `allNamespaces`, `ppStartupInfoTrace`). +- Handle the new `TraceRpcNodeKernelAccess` constructor in `Cardano.Node.Tracing.Tracers.Rpc` instead, in the hand-maintained clause lists that need it: `forMachine`, `namespaceFor`, `severityFor`, `documentFor` and `allNamespaces`. + The only namespace is `RPC.NodeKernelAccess.UnsupportedBlockType`, at severity `Warning`; there is no `ShelleyGenesisUnavailable` namespace, because there is no disk read left to fail. + It is not a metric, so `asMetrics` and `metricsDocFor` keep their catch-all defaults. +- Run.hs then passes `rpcTracer tracers` straight into `mkNodeKernelAccess` - it is already a `Tracer IO TraceRpc`, so no contramap is needed. + +cardano-rpc side: + +- Add `GenesisBundle` record to `NodeKernelAccess/Type.hs` (the module has `NoFieldSelectors`, so fields are unprefixed and read by punning). + It reuses established types rather than exploding them into primitives: + - `byronConfig :: !Byron.Config` - `Cardano.Chain.Genesis.Config` already bundles the genesis data with the hash the Byron ledger computed when it parsed the file, so it is kept whole. + - `shelleyGenesisHash :: !GenesisHashShelley` - the boot-time hash returned alongside `SomeConsensusProtocol` from `mkConsensusProtocol`. + `GenesisBundle` is only ever constructed for the Cardano protocol, so the hash is always available here; there is no missing-hash case to represent. + - `transitionConfig :: !(TransitionConfig LatestKnownEra)` - the Shelley-onwards genesis configuration, the same representation `Cardano.Api.LedgerState.GenesisConfig` uses. +- Add pure field `genesisConfig :: GenesisBundle` to `NodeKernelAccess` +- `mkNodeKernelAccess` takes the plain `GenesisHashShelley` and the `ProtocolInfoArgs blk` instead of a `FilePath`, and passes them straight to `readGenesisBundle` (which needs nothing else - not even the `TopLevelConfig` that `mkNodeKernelAccess` itself still keeps, for system start, era history and the security parameter). +- `readGenesisBundle` is a pure function - no disk read, no `IO`, no `ExceptT`, and no hard-fork navigation. + Pattern-matching `ProtocolInfoArgsCardano` on its parameter unwraps `CardanoProtocolParams` directly; its `byronProtocolParams` field is a `ProtocolParamsByron` carrying the Byron `Cardano.Chain.Genesis.Config` (`byronGenesis`) as-is, so there is no `HardForkLedgerConfig` walk needed to reach it. + The Byron `LedgerConfig` is the same `Cardano.Chain.Genesis.Config` that `configBlock` carries, so the block config needs no separate walk. +- The `transitionConfig` field is no longer assembled from per-era pieces. + It comes straight from `Consensus.cardanoLedgerTransitionConfig` applied to the `CardanoProtocolParams` unwrapped out of the passed-in `ProtocolInfoArgsCardano` - the same value cardano-node parsed and built at boot to construct `TopLevelConfig` in the first place. + This removes the `shelleyLedgerTranslationContext` extraction for Alonzo/Conway/Dijkstra and the `Ledger.mkLatestTransitionConfig` reconstruction that this plan originally called for; Phases 3, 5 and 6 still project the per-era genesis back out the same way (`tcShelleyGenesisL` reaches the Shelley genesis from any era, Alonzo and Conway need `tcPreviousEraConfigL` chained to that era followed by `tcTranslationContextL`). +- The disk read is gone entirely: no `readShelleyGenesis` call, no `ShelleyConfig`/`ShelleyGenesisError` handling, no compacted-genesis fallback, and no `TraceRpcShelleyGenesisUnavailable` trace. + The hash is always available at boot for the Cardano protocol (it is computed once, by consensus's own `Shelley.readGenesis` call, and threaded through rather than recomputed), so there is nothing left that can fail here. +- `Cardano.Api.LedgerState`'s export list does not need `readShelleyGenesis`, `ShelleyGenesisError (..)` or `renderShelleyGenesisError` added. + `ShelleyConfig (..)` and `GenesisHashShelley (..)` were already exported, and only the latter is still needed. +- The tracer argument of `mkNodeKernelAccess` is `Tracer m TraceRpc` instead of `Tracer m Text`, and the one remaining warning lives in a `TraceRpcNodeKernelAccess` sub-sum in `Cardano.Rpc.Server.Internal.Tracing` alongside `TraceRpcQuery`/`TraceRpcSubmit`/`TraceRpcSync`, re-exported from `Cardano.Rpc.Server`. +- Replace stub handler: populate `genesis` (the Shelley genesis hash bytes) and `caip2` (see "Resolved" below). + The network magic comes from `transitionConfig ^. tcShelleyGenesisL` at the call site, so `GenesisBundle` does not carry it separately. +- `cardano` oneof remains empty (valid proto default) +- Add consensus re-exports to `Cardano.Api.Consensus` as needed. + `TopLevelConfig`, `configBlock`, `configLedger`, `ProtocolInfoArgs (..)` and `cardanoLedgerTransitionConfig` are already there; reaching the Byron genesis needs only `byronProtocolParams` and `byronGenesis`, both minimal accessor re-exports - `Cardano.Api.Consensus.Internal.Protocol` already has them unqualified in scope via its existing `Ouroboros.Consensus.Cardano`/`Ouroboros.Consensus.Cardano.Node` imports, so this is an export-list addition only, no new import. + No sop-core dependency is needed at all: with no hard-fork walk, cardano-rpc has no use for `NP`, so there is no `strict-sop-core` dependency, no `Data.SOP.Strict (NP (..))` import, and no `hiding ((:*))` needed on the `Network.GRPC.Spec` import (nothing left to collide with). + +Build: `cabal build cardano-rpc && cabal build cardano-node` + +Alternative considered, and now adopted: capture the Shelley genesis hash and transition config during protocol setup and thread the values instead of a path. +This was originally rejected as changing protocol-setup plumbing shared by many call sites, while path threading looked local to the RPC wiring; the final design threads them anyway, alongside `SomeConsensusProtocol` and via `ProtocolInfoArgs`, because it removes an entire read-and-fallback surface from cardano-rpc rather than merely relocating a file path. + +### Resolved + +Both open questions were settled by comparing against Dolos, the reference UTxO RPC implementation. + +- **Which genesis hash?** The Shelley one: the Blake2b-256 hash of the raw Shelley genesis JSON file bytes, computed once by `Cardano.Node.Protocol.Shelley.readGenesis` at node startup and threaded into cardano-rpc as a plain `GenesisHashShelley` returned alongside `SomeConsensusProtocol` from `mkConsensusProtocol`, rather than recomputed from a second read. + Not the Byron hash, even though Byron's is the CIP-34 chain identifier. +- **CAIP-2 format.** Dolos does not derive it from CIP-34; it keys the identifier on the Shelley network magic. + 764824073 gives `cardano:mainnet`, 1 gives `cardano:preprod`, 2 gives `cardano:preview`, and any other magic gives `cardano:`. + cardano-rpc follows the same scheme so that clients see identical chain identifiers from either server. + +## Phases 3-6: per-era `cardano` field mapping (as built) + +Phases 3-6 are implemented together in one new module, +`Cardano.Rpc.Server.Internal.UtxoRpc.Type.Genesis`, re-exported from the +`Cardano.Rpc.Server.Internal.UtxoRpc.Type` umbrella and wired into +`readGenesisMethod` via `& U5c.cardano .~ genesisBundleToProto genesisBundle` +(setting the `cardano` lens auto-wraps into the oneof). + +### Composition design + +The proto `Genesis` message has a single `cost_models` field fed by two eras +(PlutusV1 from Alonzo, PlutusV3 from Conway), and proto-lens has no field-merge. +Each era is therefore a pure updater over one shared accumulator - +`byronGenesisToProto`, `shelleyGenesisToProto`, `alonzoGenesisToProto`, +`conwayGenesisToProto`, each `X -> Proto U5c.Genesis -> Proto U5c.Genesis`. +`genesisBundleToProto` threads a single `defMessage` through all four, so Alonzo +sets `costModels . maybe'plutusV1` and Conway sets `costModels . maybe'plutusV3` +on the same message and both survive. +The per-era genesis values are projected out of the `GenesisBundle` with no +hard-fork navigation: Byron via `configGenesisData byronConfig`, Shelley via +`transitionConfig ^. tcShelleyGenesisL`, Conway via +`tcPreviousEraConfigL . tcTranslationContextL`, and Alonzo via three +`tcPreviousEraConfigL` hops (Dijkstra to Conway to Babbage to Alonzo) followed by +`tcTranslationContextL`. + +### Phase 3: Shelley genesis (proto fields 10-23) + +`shelleyGenesisToProto` maps `sgActiveSlotsCoeff`, `sgEpochLength`, `sgGenDelegs`, +`sgInitialFunds`, `sgMaxKESEvolutions`, `sgMaxLovelaceSupply`, `sgNetworkId`, +`sgNetworkMagic`, `sgProtocolParams`, `sgSecurityParam`, `sgSlotLength`, +`sgSlotsPerKESPeriod`, `sgSystemStart` and `sgUpdateQuorum`. +`sgNetworkId` renders as the text `Mainnet`/`Testnet`; `sgSystemStart` as +ISO-8601 via `iso8601Show`; `sgSlotLength` as milliseconds +(`round (1000 * fromNominalDiffTimeMicro sgSlotLength)`, so mainnet's 1 second is +1000); `sgGenDelegs` and `sgInitialFunds` keys as lowercase base16 of the key +hash and of `serialiseAddr` respectively. +`sgInitialFunds` (field 13) is populated because the Shelley genesis is projected +from the uncompacted boot-time transition config - the compacted in-memory copy +would yield an empty map; on mainnet and the published testnets it is genuinely +empty, custom networks (e.g. cardano-testnet) populate it. +The proto `PParams` sub-message is built by hand from the era-generic +`EraPParams` lenses on `PParams ShelleyEra` (`ppTxFeePerByteL`, `ppTxFeeFixedL`, +`ppMaxBBSizeL`, `ppMaxTxSizeL`, `ppMaxBHSizeL`, `ppKeyDepositL`, `ppPoolDepositL`, +`ppEMaxL`, `ppNOptL`, `ppA0L`, `ppRhoL`, `ppTauL`, `ppMinPoolCostL`, +`ppProtocolVersionL`), because `protocolParamsToUtxoRpcPParams` requires +`ConwayEraPParams`. +`sgStaking` has no proto counterpart, so it is not mapped. + +### Phase 4: Byron genesis (proto fields 1-9) + +`byronGenesisToProto` maps `gdAvvmDistr`, `gdProtocolParameters` (to +`protocolConsts` and `blockVersionData`), `gdStartTime`, `gdGenesisKeyHashes` +(proto `bootStakeholders`), `gdHeavyDelegation` and `gdNonAvvmBalances`. +`gdStartTime` is Unix seconds (`round . utcTimeToPOSIXSeconds`). +All hashes and keys reuse the Byron canonical-JSON formatters so the wire values +match the on-disk genesis: key hashes via `sformat hashHexF . unKeyHash`, +delegation certificate signatures via `fullSignatureHexF`, issuer/delegate +verification keys via `fullVerificationKeyF` (standard base64), non-AVVM +addresses via `addressF` (base58), and AVVM redeem keys via +`redeemVKB64UrlF . fromCompactRedeemVerificationKey` (base64url). +Byron stores only a set of genesis key hashes, so each `bootStakeholders` weight +is synthesised as 1, matching the genesis JSON. +`blockVersionData`'s size and threshold fields are stringified numbers; +LovelacePortion values render as their raw Word64 numerator (recovered via +`lovelacePortionToRational` because `unLovelacePortion` is not exported), and the +`txFeePolicy` summand/multiplier reproduce the Byron JSON's 1e9 scaling. + +### Phase 5: Alonzo genesis (proto fields 24-31) + +`alonzoGenesisToProto` maps `agCoinsPerUTxOWord`, `agPrices` (proto +`executionPrices`), `agMaxTxExUnits`, `agMaxBlockExUnits`, `agMaxValSize`, +`agCollateralPercentage`, `agMaxCollateralInputs` and the PlutusV1 cost model from +`agPlutusV1CostModel` (set on `costModels . maybe'plutusV1`). +`agExtraConfig` is not read; PlutusV2 and PlutusV4 are never in genesis. + +### Phase 6: Conway genesis (proto fields 32-42) + +`conwayGenesisToProto` maps `cgUpgradePParams` (committee min size and max term +length, gov action lifetime and deposit, DRep deposit and activity, min fee ref +script cost per byte, and the pool/DRep voting thresholds mapped to their named +proto fields), `cgConstitution` (anchor plus guardrails script hash) and +`cgCommittee` (threshold plus members), and the PlutusV3 cost model from +`ucppPlutusV3CostModel` (set on `costModels . maybe'plutusV3`, composing with +Alonzo's PlutusV1 on the shared accumulator). + +Build: `cabal build cardano-rpc` + +## Phase 7: E2E test + +- Create `cardano-testnet/test/cardano-testnet-test/Cardano/Testnet/Test/Rpc/Genesis.hs` + - `hprop_rpc_read_genesis`: start testnet with `RpcEnabled`, call ReadGenesis via gRPC + - Assert: `genesis` is 32 bytes, `caip2` is non-empty, `cardano` is set + - Assert Shelley fields: `epochLength > 0`, `networkMagic` matches testnet magic, `systemStart` non-empty + - Assert `initialFunds` is non-empty: cardano-testnet funds its wallets there, so this proves end to end that the uncompacted boot-time genesis reached the RPC response (the compacted in-memory genesis would yield an empty map) + - Assert Byron fields: `protocolConsts` present, `startTime > 0` + - Assert Alonzo fields: `executionPrices` present, `maxTxExUnits` non-zero +- Register in `cardano-testnet-test.hs` test runner + +Build: `TASTY_PATTERN='/RPC ReadGenesis/' cabal test cardano-testnet-test` + +## Phase dependency graph + +``` +Phase 1 (proto + stub) + | + v +Phase 2 (NKA bundle + envelope) + | + +---> Phase 3 (Shelley) --+ + | | + +---> Phase 4 (Byron) --+--> Phase 7 (E2E test) + | | + +---> Phase 5 (Alonzo) --+ + | | + +---> Phase 6 (Conway) --+ (optional) +``` + +Phases 3-6 are independent of each other. +Phase 7 depends on at least phases 3-5 for meaningful assertions. + +## Design notes + +- **No cardano-api boundary violation.** + All genesis types are reachable from cardano-api or from ledger packages cardano-rpc already depends on, so `GenesisBundle` needs no consensus import of its own. + In practice the fields are typed with the ledger types directly (`Cardano.Ledger.Shelley.Genesis`, `Cardano.Ledger.Alonzo.Genesis`, `Cardano.Ledger.Conway.Genesis`, `Cardano.Chain.Genesis`), matching how the rest of cardano-rpc imports ledger types; `Cardano.Api.Genesis` names Alonzo and Conway genesis only through the `AlonzoGenesisConfig`/`ConwayGenesisConfig` aliases. +- **Reaching the Byron genesis needs no hard-fork navigation**, but the call to `readGenesisBundle` still requires `blk ~ CardanoBlock StandardCrypto`, because its signature is monomorphic in `Consensus.ProtocolInfoArgs (CardanoBlock StandardCrypto)`. + Matching the `CardanoBlockType` constructor of cardano-api's `BlockType` GADT (as `mkNodeKernelAccess` already does) brings that equality into scope; there is no `withBlockTypeConstraints` helper, and none is needed. +- **Why the disk read is no longer needed.** + `TopLevelConfig` only retains a compacted per-era `ShelleyLedgerConfig` (`Ouroboros.Consensus.Shelley.Ledger.Config`'s `CompactGenesis`): `compactGenesis` sets `sgInitialFunds = mempty` and `sgStaking = emptyGenesisStaking`, which is why this plan originally called for a second file read to recover them. + `ProtocolInfoArgs`, threaded in from `mkNodeKernelAccess`'s caller, is the value cardano-node built before that compaction happens: `cardanoLedgerTransitionConfig` on the `CardanoProtocolParams` it carries is the full, uncompacted `TransitionConfig LatestKnownEra`, with `sgInitialFunds` and `sgStaking` intact. + Reading the transition config off `ProtocolInfoArgs` instead of `TopLevelConfig` therefore sidesteps the compaction entirely, so the second disk read (and its fallback) can be deleted rather than merely deferred. +- **Mainnet data shape.** + `mainnet-shelley-genesis.json` has `"initialFunds": {}` and no `"staking"` key at all; the testnet template is the same. + An empty `initial_funds` in a mainnet response is therefore correct data, not a bug. + +## Known limitations + +- `sgStaking` is available via the transition config threaded from `ProtocolInfoArgs`, but the proto `Genesis` message has no staking field to carry it (spec gap; raise upstream if a client needs it). +- The proto `Genesis` message has no Dijkstra-era fields (it stops at Conway, field 42); `npcDijkstraGenesisFile` is not read. +- Alonzo/Conway genesis hashes are still not derivable without also reading those files; not needed while the `genesis` field carries the Shelley hash. +- The Byron `ftsSeed` (field 3) and `vssCerts` (field 9) proto fields stay at their default: `GenesisData` at the pinned cardano-ledger-byron carries no such data. +- The `protocolConsts` `vssMaxTtl`/`vssMinTtl` stay at 0: there is no Byron ledger source for them. +- Byron `bootStakeholders` weights are all 1: the ledger stores only a set of genesis key hashes, so the per-stakeholder weight is synthesised. +- No PlutusV2 (or PlutusV4) cost model is emitted: neither appears in any era's genesis. +- The Shelley `sgProtocolParams` fields `extraEntropy`, `d` (decentralisation) and `minUTxOValue` are dropped: the proto `PParams` message has no counterpart for them. +- A proto committee member key does not distinguish a key hash from a script hash: both `KeyHashObj` and `ScriptHashObj` credentials render as bare hex. +- The Byron AVVM key encoding reuses the ledger's own `redeemVKB64UrlF` formatter (padded base64url) so it matches the on-disk genesis and Dolos; this was verified against the ledger's canonical-JSON `ToObjectKey` instance rather than by round-tripping a live genesis. diff --git a/cardano-rpc/docs/node-kernel-access/prereqs-api-signatures.md b/cardano-rpc/docs/node-kernel-access/prereqs-api-signatures.md new file mode 100644 index 0000000000..506d6125e0 --- /dev/null +++ b/cardano-rpc/docs/node-kernel-access/prereqs-api-signatures.md @@ -0,0 +1,163 @@ +# API Signatures and Type References + +Verified consensus and cardano-api type signatures needed for the node kernel access implementation. + +--- + +## 1. Consensus API Signatures (Verified) + +All signatures below were verified against the checked-out source as of 2026-05-22. + +### 1.1 `answerQuery` + +**Module:** `Ouroboros.Consensus.Ledger.Query` +**File:** `ouroboros-consensus/ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Ledger/Query.hs` + +```haskell +answerQuery + :: forall blk m result. + (BlockSupportsLedgerQuery blk, ConfigSupportsNode blk, HasAnnTip blk, MonadSTM m) + => ExtLedgerCfg blk + -> ReadOnlyForker' m blk + -> Query blk result + -> m result +``` + +- Takes `ExtLedgerCfg blk` (construct via `ExtLedgerCfg (getTopLevelConfig nk)`) +- Takes `ReadOnlyForker' m blk` (the primed alias: `ReadOnlyForker m (ExtLedgerState blk) blk`) +- `Query blk result` is a GADT with constructors: `BlockQuery`, `GetSystemStart`, `GetChainBlockNo`, `GetChainPoint`, `DebugLedgerConfig` +- Handles all three footprints internally via `sing :: Sing footprint` dispatch: `SQFNoTables` -> `answerPureBlockQuery`, `SQFLookupTables` -> `answerBlockQueryLookup`, `SQFTraverseTables` -> `answerBlockQueryTraverse` + +**Footprint dispatch in detail:** +- `SQFNoTables` - pure ledger state queries (pparams, epoch, etc.) - calls `roforkerGetLedgerState` via STM. + No table access, no IO beyond the STM read. +- `SQFLookupTables` - point lookups (UTxO by TxIn) - calls `roforkerReadTables`. + O(m log n) for m keys in a store of n entries. +- `SQFTraverseTables` - table traversals (UTxO by address, whole UTxO) - calls `roforkerRangeReadTables`. + O(n) full scan with batched range reads (default batch size 100k entries). + +`answerQuery` returns plain Haskell values - no serialisation. + +The `GetSystemStart` and `GetChainBlockNo`/`GetChainPoint` queries are handled directly at +the top level (not via `BlockQuery`). +`GetSystemStart` extracts the value from `ExtLedgerCfg` (the config, not the state). +`GetChainBlockNo` and `GetChainPoint` read from `headerState` via STM. + +### 1.2 `getReadOnlyForkerAtPoint` + +**Module:** `Ouroboros.Consensus.Storage.ChainDB.API` (API), `Ouroboros.Consensus.Storage.ChainDB.Impl.Query` (implementation) + +```haskell +getReadOnlyForkerAtPoint + :: IOLike m + => ChainDbEnv m blk + -> ResourceRegistry m + -> Target (Point blk) + -> m (Either GetForkerError (ReadOnlyForker' m blk)) +``` + +- **`Target`** is from `Ouroboros.Network.Protocol.LocalStateQuery.Type` (NOT `Ouroboros.Consensus.Block`) + ```haskell + data Target point = VolatileTip | SpecificPoint point | ImmutableTip + ``` +- **`GetForkerError`** is from `Ouroboros.Consensus.Storage.LedgerDB.Forker`: + ```haskell + data GetForkerError = PointNotOnChain | PointTooOld !(Maybe ExceededRollback) + ``` +- Returns `ReadOnlyForker' m blk` (the primed alias) +- **`roforkerClose`** is a record field of `ReadOnlyForker`, type `!(m ())` + +### 1.3 `toConsensusQuery` / `fromConsensusQueryResult` + +**Module:** `Cardano.Api.Query.Internal.Type.QueryInMode` (re-exported via `Cardano.Api.Query`) + +```haskell +toConsensusQuery + :: (HasCallStack, Consensus.CardanoBlock StandardCrypto ~ block) + => QueryInMode result + -> Some (Consensus.Query block) + +fromConsensusQueryResult + :: (HasCallStack, Consensus.CardanoBlock StandardCrypto ~ block) + => QueryInMode result + -> Consensus.Query block result' + -> result' + -> result +``` + +- Both exported. +- `toConsensusQuery` returns `Some (Consensus.Query block)` (existential wrapper). +- `fromConsensusQueryResult` needs the original `QueryInMode` for type-level dispatch. +- **`QueryCurrentEra` routes through `toConsensusQuery` correctly.** + It wraps internally as `BlockQuery (QueryHardFork GetCurrentEra)`. + No special handling needed. + +### 1.4 `toConsensusGenTx` / `fromConsensusApplyTxErr` + +**Module:** `Cardano.Api.Consensus.Internal.InMode` (re-exported via `Cardano.Api.Consensus`) + +```haskell +toConsensusGenTx + :: Consensus.CardanoBlock StandardCrypto ~ block + => TxInMode -> Consensus.GenTx block + +fromConsensusApplyTxErr + :: Consensus.CardanoBlock StandardCrypto ~ block + => Consensus.ApplyTxErr block -> TxValidationErrorInCardanoMode +``` + +- Both exported from `Cardano.Api.Consensus`. +- `SubmitResult` is from `Ouroboros.Network.Protocol.LocalTxSubmission.Type`, re-exported via `Cardano.Api.Network`. + +### 1.5 `addLocalTxs` (Mempool API) + +**Module:** `Ouroboros.Consensus.Mempool.API` + +```haskell +addLocalTxs + :: forall m blk t. (MonadSTM m, Traversable t) + => Mempool m blk -> t (GenTx blk) -> m (t (MempoolAddTxResult blk)) +``` + +- Name is `addLocalTxs` (wrapper around lower-level `addTx` field with `AddTxForLocalClient`). +- On GHC 9.10, use **`MkSolo`** (not `Solo`) as the constructor: + ```haskell + import Data.Tuple (Solo (..)) + MkSolo addTxRes <- addLocalTxs mempool (MkSolo genTx) + ``` +- Result type: + ```haskell + data MempoolAddTxResult blk + = MempoolTxAdded !(Validated (GenTx blk)) + | MempoolTxRejected !(GenTx blk) !(ApplyTxErr blk) + ``` +- Error type in rejection is `ApplyTxErr blk` (a type family; for Shelley blocks: `SL.ApplyTxError era`). + +### 1.6 `withRegistry` + +**Module:** `Control.ResourceRegistry` (NOT `Ouroboros.Consensus.Util.ResourceRegistry`) + +```haskell +withRegistry :: (ResourceRegistry m -> m a) -> m a +``` + +- Registry cleanup automatically closes registered forkers when the registry scope exits. +- This means `bracket` around `roforkerClose` is still good practice for explicit cleanup, but `withRegistry` provides a safety net if an exception bypasses the explicit close. + +### 1.7 `NodeKernel` type parameters + +**Module:** `Ouroboros.Consensus.NodeKernel` (in ouroboros-consensus-diffusion) + +```haskell +data NodeKernel m addrNTN addrNTC blk = NodeKernel + { getChainDB :: ChainDB m blk + , getMempool :: Mempool m blk + , getTopLevelConfig :: TopLevelConfig blk + , getFetchClientRegistry :: FetchClientRegistry (ConnectionId addrNTN) (HeaderWithTime blk) blk m + , ... + } +``` + +- In cardano-node (`Run.hs` line 404): `NodeKernel IO RemoteAddress LocalConnectionId blk` +- `getChainDB`, `getMempool`, `getTopLevelConfig` are all **record fields** (not standalone functions). +- `rnNodeKernelHook` receives the kernel in the callback: `\registry nodeKernel -> do ...` diff --git a/cardano-rpc/docs/node-kernel-access/prereqs-build-and-conventions.md b/cardano-rpc/docs/node-kernel-access/prereqs-build-and-conventions.md new file mode 100644 index 0000000000..855a1e5de8 --- /dev/null +++ b/cardano-rpc/docs/node-kernel-access/prereqs-build-and-conventions.md @@ -0,0 +1,265 @@ +# Build Instructions and Conventions + +Project layout, build commands, and codebase conventions for the node kernel access work. + +--- + +## 0. Project Layout + +The working directory is `/work/`. +It contains git submodules, each a separate Haskell project with its own flake/cabal setup. +Only the submodules relevant to this task are listed here. + +### Submodules relevant to this task + +``` +/work/ +├── cardano-api/ ← cardano-api repo (contains cardano-rpc) +│ ├── cardano-api/ ← the cardano-api library package +│ │ └── src/Cardano/Api/ ← cardano-api source (Query, Network.IPC, Era, Consensus, etc.) +│ └── cardano-rpc/ ← the cardano-rpc package (THIS IS THE MAIN TARGET) +│ ├── src/Cardano/Rpc/ +│ │ ├── Client.hs ← gRPC client (not modified) +│ │ ├── Proto/Api/ ← proto-lens API wrappers (not modified) +│ │ └── Server/ +│ │ ├── Config.hs ← RpcConfig (kept unchanged) +│ │ ├── Server.hs ← runRpcServer entry point (MODIFIED) +│ │ └── Internal/ +│ │ ├── Env.hs ← RpcEnv record (MODIFIED) +│ │ ├── Monad.hs ← Has typeclass, MonadRpc (MODIFIED) +│ │ ├── Error.hs ← throwEither, throwExceptT (kept) +│ │ ├── Node.hs ← getEra, getProtocolParamsJson (MODIFIED) +│ │ ├── Tracing.hs ← TraceRpc types (MODIFIED) +│ │ ├── NodeKernelAccess.hs ← NEW FILE +│ │ ├── Orphans.hs ← (not modified) +│ │ └── UtxoRpc/ +│ │ ├── Query.hs ← readParams, readUtxos, searchUtxos (MODIFIED) +│ │ ├── Submit.hs ← submitTx (MODIFIED) +│ │ ├── Type.hs ← protobuf conversion (not modified) +│ │ └── Predicate.hs ← UTxO filtering (not modified) +│ ├── gen/ ← generated proto-lens code (NEVER edit by hand) +│ ├── proto/ ← .proto definitions +│ ├── test/ ← unit tests +│ └── cardano-rpc.cabal ← (MODIFIED - add NodeKernelAccess module) +│ +├── cardano-node/ ← cardano-node repo +│ ├── cardano-node/ ← the cardano-node package +│ │ ├── src/Cardano/Node/ +│ │ │ ├── Run.hs ← node startup, withAsync runRpcServer (MODIFIED) +│ │ │ ├── Queries.hs ← existing nkQueryLedger pattern (reference only) +│ │ │ ├── Rpc/ +│ │ │ │ └── NodeKernelAccess.hs ← NEW FILE - mkNodeKernelAccess from NodeKernel +│ │ │ └── Tracing/Tracers/ +│ │ │ └── Rpc.hs ← LogFormatting/MetaTrace instances (MODIFIED) +│ │ └── cardano-node.cabal ← (MODIFIED - add new module) +│ ├── cardano-testnet/ ← integration test infrastructure +│ │ └── test/Cardano/Testnet/Test/Rpc/ ← gRPC integration tests (not modified) +│ └── @worktree/add-grpc-interface/ ← worktree for the gRPC feature branch +│ +├── ouroboros-consensus/ ← consensus repo (READ ONLY - verify APIs here) +│ ├── ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/ +│ │ ├── Ledger/Query.hs ← answerQuery function +│ │ ├── Ledger/Tables/ ← MapKind, LedgerTables definitions +│ │ ├── Storage/ChainDB/API.hs ← getReadOnlyForkerAtPoint, getCurrentLedger +│ │ ├── Storage/LedgerDB/ ← Forker, BackingStore +│ │ ├── Mempool/API.hs ← addLocalTxs +│ │ └── MiniProtocol/LocalStateQuery/Server.hs ← existing N2C query server (reference) +│ ├── ouroboros-consensus-diffusion/src/.../ +│ │ ├── NodeKernel.hs ← NodeKernel type definition +│ │ └── Network/NodeToClient.hs ← existing N2C wiring (reference) +│ └── ouroboros-consensus-cardano/src/shelley/.../ +│ └── Shelley/Ledger/Query.hs ← Shelley-specific query answering +│ +├── cardano-ledger/ ← ledger repo (READ ONLY) +│ └── eras/shelley/impl/src/Cardano/Ledger/Shelley/LedgerState.hs +│ +├── cardano-node-wiki/ ← documentation (THIS DOCUMENTATION SET) +│ └── docs/ +│ ├── ADR-018-cardano-rpc-grpc-server.md +│ ├── ADR-019-node-kernel-access-for-cardano-rpc.md +│ ├── implementation-plan.md +│ ├── analysis-architecture.md +│ ├── analysis-utxohd-internals.md +│ ├── analysis-consensus-protocol.md +│ ├── prereqs-api-signatures.md +│ ├── prereqs-build-and-conventions.md ← THIS FILE +│ └── prereqs-implementation-details.md +│ +└── ouroboros-network/ ← network repo (READ ONLY - RemoteAddress, etc.) +``` + +### Key relationships + +- **cardano-rpc** depends on **cardano-api** (types, re-exports) but NOT on consensus +- **cardano-node** depends on both **cardano-rpc** and **ouroboros-consensus** +- The `NodeKernelAccess` record lives in **cardano-rpc** (cardano-api types only) +- The `mkNodeKernelAccess` implementation lives in **cardano-node** (knows consensus types) +- This is dependency inversion: cardano-rpc defines the interface, cardano-node implements it + +### NodeKernelAccess type (snapshot-based design) + +Note: this snapshot design is forward-looking - it is the target interface for pieces 4-7, not current code (as built, `NodeKernelAccess` is the thinner `chainDb` / `systemStart` / `readEraHistory` record; see `01-node-access-types.md`). + +```haskell +data NodeKernelAccess = NodeKernelAccess + { nkaWithSnapshot :: forall a. (LedgerSnapshot -> IO a) -> IO a + , nkaSubmitTx :: TxInMode -> IO (SubmitResult TxValidationErrorInCardanoMode) + } + +newtype LedgerSnapshot = LedgerSnapshot + { runQuery :: forall result. QueryInMode result -> IO result } +``` + +`nkaWithSnapshot` acquires a single `ReadOnlyForker` and provides a `LedgerSnapshot` that +runs any `QueryInMode` against that forker. +All queries within one `nkaWithSnapshot` call see the same ledger state. +`nkaSubmitTx` goes directly to the mempool and does not need a snapshot. + +### Path convention in the implementation plan + +The implementation plan uses paths relative to `/work/`: +- `cardano-api/cardano-rpc/src/...` means `/work/cardano-api/cardano-rpc/src/...` +- `cardano-node/cardano-node/src/...` means `/work/cardano-node/cardano-node/src/...` + +--- + +## 2. Nix Build Instructions + +### The git submodule problem + +`/work` contains git submodules. +Each submodule's `.git` file points to a parent modules directory that doesn't exist in this environment. +This breaks `nix build .#...` because nix's git fetcher fails. + +### Workaround: use `path:` instead of `.` + +```bash +# In cardano-api submodule: +cd /work/cardano-api +nix build 'path:/work/cardano-api#cardano-rpc:lib:cardano-rpc' \ + --allow-import-from-derivation \ + --accept-flake-config + +# In cardano-node submodule: +cd /work/cardano-node +nix build 'path:/work/cardano-node#cardano-node:lib:cardano-node' \ + --allow-import-from-derivation \ + --accept-flake-config +``` + +### Required flags (always) + +- `--allow-import-from-derivation` - haskell.nix needs IFD to enumerate packages +- `--accept-flake-config` - to trust the flake's nixConfig settings + +### Package names (haskell.nix component style) + +| Package | Nix attribute | +|---------|---------------| +| cardano-rpc library | `cardano-rpc:lib:cardano-rpc` | +| cardano-rpc gen library | `cardano-rpc:lib:gen` | +| cardano-rpc tests | `cardano-rpc:test:cardano-rpc-test` | +| cardano-api library | `cardano-api:lib:cardano-api` | +| cardano-node library | `cardano-node:lib:cardano-node` | + +### Listing all packages + +```bash +nix eval 'path:/work/cardano-api#packages.x86_64-linux' --apply 'builtins.attrNames' \ + --allow-import-from-derivation --accept-flake-config +``` + +### Running code generation (proto-lens) + +Never manually edit files in `gen/`. +Use: +```bash +nix develop --command bash -c "cd cardano-rpc && buf generate proto" +``` + +--- + +## 3. Codebase Conventions and Gotchas + +### RIO import hiding + +RIO re-exports most of Prelude but **hides some common functions**: +- `sortBy` - import from `Data.List` +- `on` - import from `Data.Function` or `Data.Ord` +- `toList` - import from `GHC.IsList` (not `Data.Foldable`) + +Always check what RIO re-exports before assuming a function is in scope. + +### Proto wrapper: `Proto msg` + +`Proto` is a grapesy newtype wrapper. +Rules: +- Handler signatures use `Proto` in parameters and return types +- Internal functions use **plain proto-lens types** (unwrapped) +- Use `getProto` / `fmap getProto` only at the RPC handler boundary +- Never wrap intermediate values in `Proto` + +### `Inject` typeclass + +After removing `import Cardano.Api` from `Monad.hs`, the `Inject` typeclass (used by +`putTrace`) must be imported explicitly: +```haskell +import Cardano.Api.Era (Inject (..)) +``` + +### hlint preferences + +- Prefer backtick-infix sections over lambdas: `` (`f` y) `` not `\x -> f x y` +- Don't mix styles in the same function +- hlint will catch these + +### GHC version + +The project uses GHC 9.10 (from the `inotifywait.sh` commit: "bump ghc to 9.10"). + +### `-Wunused-packages` is enabled + +The cabal file has `-Wunused-packages`. +If you remove an import that was the only use of a dependency, the build will fail. +Conversely, don't add deps without checking if they're already transitively available. + +### `-Wredundant-constraints` is enabled + +Don't add typeclass constraints that aren't actually used. +The `IsEra` constraint mistake in the past was caught by this. + +--- + +## 6. Directory layout and worktrees + +### Main checkout paths + +The cardano-node files are available in the main checkout: +``` +/work/cardano-node/cardano-node/src/Cardano/Node/Run.hs +/work/cardano-node/cardano-node/src/Cardano/Node/Tracing/Tracers/Rpc.hs +/work/cardano-node/cardano-node/cardano-node.cabal +``` + +The cardano-rpc files are in the cardano-api submodule: +``` +/work/cardano-api/cardano-rpc/src/Cardano/Rpc/Server/... +/work/cardano-api/cardano-rpc/cardano-rpc.cabal +``` + +### Worktree for the gRPC feature branch + +A worktree exists at `/work/cardano-node/@worktree/add-grpc-interface/`. +This is the branch where the gRPC feature was originally developed. +The analysis file references files in this worktree path - they may differ from the main checkout if the branch has been merged or rebased. + +**When implementing:** Check whether to work in the main checkout or the worktree, depending on the current git branch state. +Use `git branch` and `git log` to determine which is current. + +### Worktree rules (from AGENTS.md) + +Worktrees ALWAYS reside in each subproject's `@worktree/` directory. +After creating a git worktree inside a submodule, update its `.git` configuration to use relative paths. +`git worktree add` writes absolute paths in submodules in two places that must both be fixed: +1. `/.git` - the `gitdir:` line pointing to the worktree metadata +2. `.git/modules//worktrees//gitdir` - the back-pointer to the worktree diff --git a/cardano-rpc/docs/node-kernel-access/prereqs-implementation-details.md b/cardano-rpc/docs/node-kernel-access/prereqs-implementation-details.md new file mode 100644 index 0000000000..9d6d467781 --- /dev/null +++ b/cardano-rpc/docs/node-kernel-access/prereqs-implementation-details.md @@ -0,0 +1,321 @@ +# Implementation Details and Verification + +Subtle implementation gotchas, per-method query inventory, and verification checklist. + +--- + +## 4. Subtle Implementation Details + +### Snapshot Consistency (IMPORTANT) + +The current N2C code runs all queries within one `executeLocalStateQueryExpr` call, acquiring a single ledger snapshot. +Protocol parameters, UTxO results, chain tip, and block number all come from the same ledger state. + +The `NodeKernelAccess` design must preserve this: use `nkaWithSnapshot` to group all queries for a single RPC handler under one `ReadOnlyForker`. +Do NOT call `nkaWithSnapshot` multiple times within a single handler - that would break consistency. + +This is especially critical for `EvalTx`, which queries 7 different things that must be mutually consistent: +1. `queryProtocolParameters` - needed for tx evaluation +2. `queryUtxo` (by TxIn set) - inputs being spent +3. `querySystemStart` - for slot-to-time conversion +4. `queryEraHistory` - for slot-to-time conversion +5. `queryStakeDelegDeposits` - for deposit tracking +6. `queryDRepState` - for governance-related evaluation +7. `queryStakePoolParameters` - for delegation-related evaluation + +If these came from different ledger states, tx evaluation could produce wrong results or spurious failures. + +### 4.1 The `eon` existential must escape the snapshot callback + +The `eon` value (from `forEraInEon @Era era ...`) is needed after the `nkaWithSnapshot` +callback for protobuf conversion. +The callback must return it as part of its result tuple: + +```haskell +-- CORRECT: eon escapes the callback +(pparams, chainPoint, blockNo, eon) <- liftIO $ withNodeKernelAccess laRef $ \la -> do + nkaWithSnapshot la $ \snapshot -> do + AnyCardanoEra era <- runQuery snapshot QueryCurrentEra + eon <- forEraInEon @Era era (error "Minimum Conway era required") pure + ... + pure (pparams, chainPoint, blockNo, eon) + +-- WRONG: eon would be scoped inside the callback only +liftIO $ withNodeKernelAccess laRef $ \la -> do + nkaWithSnapshot la $ \snapshot -> do + ... + -- can't use eon out here for protobuf conversion +``` + +This works because `eon` is an existential that's pattern-matched later. +The tuple captures the existential witness. + +### 4.2 `RankNTypes` in `NodeKernelAccess` and `LedgerSnapshot` + +Both `NodeKernelAccess` and `LedgerSnapshot` use higher-rank quantification: +```haskell +nkaWithSnapshot :: forall a. (LedgerSnapshot -> IO a) -> IO a +runQuery :: forall result. QueryInMode result -> IO result +``` + +This requires `{-# LANGUAGE RankNTypes #-}` on `NodeKernelAccess.hs`. +The `forall a.` on `nkaWithSnapshot` ensures the snapshot cannot escape the callback scope. +The `forall result.` on `runQuery` lets callers pass any `QueryInMode` variant and get the corresponding result type back. + +### 4.3 `throwEither` / `throwExceptT` still needed in Submit.hs + +After the rewrite, `Query.hs` and `Node.hs` no longer need `throwExceptT` or the double `throwEither` pattern (because `NodeKernelAccess` callbacks throw directly on error). +But `Submit.hs` still uses `throwEither` via `putTraceThrowEither` for tx validation errors. +Don't remove the `Error` module import from `Submit.hs`. + +### 4.4 `SomeException` import in Tracing.hs + +After removing `TraceRpcSubmitN2cConnectionError SomeException`, the `Control.Exception` import for `SomeException` is **still needed** because `TraceRpc` (defined in the same module) uses `SomeException` in `TraceRpcError` and `TraceRpcFatalError`. + +### 4.5 The `SearchUtxos` tracing gap (verified: no bug) + +`TraceRpcQuerySearchUtxosSpan` is already fully wired in `forMachine`, `asMetrics`, and the `MetaTrace` instance in `cardano-node/src/Cardano/Node/Tracing/Tracers/Rpc.hs`. +The originally reported pre-existing bug does not exist in the current codebase. +Step 11 of the implementation plan only needs to swap the N2C trace constructors. + +### 4.6 `Net.Tx.SubmitFail` / `Net.Tx.SubmitSuccess` still needed + +After the rewrite, `Submit.hs` still pattern-matches on `SubmitFail` / `SubmitSuccess` from `Cardano.Api.Network.IPC`. +Don't remove that qualified import. + +### 4.7 Config.hs backward compatibility + +The plan **keeps** `nodeSocketPath` in `RpcConfig`. +The earlier design iteration wanted to remove it, but analysis showed it would break: +- `cardano-node/cardano-testnet/src/Testnet/Types.hs` (line 155: `nodeRpcSocketPath`) +- `cardano-node/cardano-node/src/Cardano/Node/Configuration/POM.hs` +- `makeRpcConfig` uses it to derive the default `rpcSocketPath` + +The field just isn't used at runtime for N2C connections anymore. + +### 4.8 Submit.hs uses both `nkaWithSnapshot` and `nkaSubmitTx` + +The submit method needs two separate interactions with `NodeKernelAccess`: +1. First, `nkaWithSnapshot` to query the current era (via `runQuery QueryCurrentEra`) for tx deserialisation. +2. Then `nkaSubmitTx` (after the tx is deserialised and validated) to submit the transaction. + +Era determination now happens inside the snapshot callback via `runQuery` rather than a dedicated `laDetermineEra` field. +Tx deserialisation is pure/monadic and happens between the two calls. +Do not fold both into a single `nkaWithSnapshot` - `nkaSubmitTx` is a separate operation that goes to the mempool, not the ledger snapshot. + +### 4.9 Era mismatch in `runQuery` result unwrapping + +`fromConsensusQueryResult` for era-specific queries (pparams, utxo) returns `Either EraMismatch result`. +The `runQuery` implementation inside `mkNodeKernelAccess` must unwrap this: + +```haskell +-- Inside the runQuery implementation for era-specific queries: +raw <- answerQuery cfg forker consensusQuery +case fromConsensusQueryResult qim consensusQuery raw of + Left eraMismatch -> throwIO (userError $ "Era mismatch: " <> show eraMismatch) + Right result -> pure result +``` + +With the snapshot-based design, all queries within one `nkaWithSnapshot` call share the same `ReadOnlyForker`, so era mismatches between queries inside a single snapshot are impossible. +The only cross-snapshot era race is in `Submit.hs` (see 4.8): era detection uses one `nkaWithSnapshot` call, then `nkaSubmitTx` is a separate call - an era transition between them is theoretically possible at hard fork boundaries but extremely rare. +Throwing is acceptable since the client can retry. + +### 4.10 `QueryCurrentEra` routes through `toConsensusQuery` (verified) + +`toConsensusQuery` handles `QueryCurrentEra` by wrapping it as `BlockQuery (QueryHardFork GetCurrentEra)` internally. +No special handling is needed - just call `runQuery snapshot QueryCurrentEra` inside the `nkaWithSnapshot` callback. + +### 4.11 `runQuery` helper must use `bracket` for forker lifecycle (fixed in plan) + +The implementation plan's `runQuery` snippet now uses `bracket` for exception-safe forker cleanup. +`withRegistry` also provides a safety net (auto-closes registered resources when its scope exits), but explicit `bracket` around `roforkerClose` is still preferred for deterministic cleanup. + +### 4.12 IORef thread safety during startup + +The `IORef (Maybe NodeKernelAccess)` has one writer (`rnNodeKernelHook`) and multiple readers (gRPC handler threads). +This is safe because: +- GHC's `IORef` guarantees atomicity for single-word writes +- The write is `Nothing -> Just la` (a single pointer write) +- Readers see either `Nothing` (return UNAVAILABLE) or `Just la` (proceed) +- No read-modify-write cycle exists + +There is NO race condition here. +But if someone later adds a second write site, this assumption breaks. +Document it in a comment in `Run.hs`. + +### 4.13 `Solo` constructor name on GHC 9.10 (verified) + +GHC 9.10 uses `MkSolo` as the constructor. +Import from `Data.Tuple` (not `GHC.Tuple`): +```haskell +import Data.Tuple (Solo (..)) + +MkSolo addTxRes <- addLocalTxs mempool (MkSolo genTx) +``` + +This matches the pattern used in ouroboros-consensus's own `LocalTxSubmission/Server.hs`. + +### 4.14 `getReadOnlyForkerAtPoint` takes `ResourceRegistry` from where? (verified) + +Each call to `getReadOnlyForkerAtPoint` needs a `ResourceRegistry`. +Use `withRegistry` from `Control.ResourceRegistry` to create a fresh one per query. +This matches the pattern used by the existing N2C LocalStateQuery server in `ouroboros-consensus-diffusion/Network/NodeToClient.hs`. + +Registry cleanup does automatically close registered forkers when the registry scope exits. +Using `bracket` with `roforkerClose` inside `withRegistry` is still good practice for explicit cleanup, but the registry provides a safety net if an exception bypasses it. + +### 4.15 The `asType` in `deserialiseTx` comes from where? + +In `Submit.hs`, the existing code uses `asType` (line 52): +```haskell +deserialiseTx sbe = shelleyBasedEraConstraints sbe $ deserialiseFromCBOR asType +``` + +After the rewrite, this code is unchanged but `asType` comes from `Cardano.Api` (re-exported by the blanket `import Cardano.Api`). +If you change the `Cardano.Api` import to be more specific, make sure `AsType` / `asType` is still in scope. +It's from `Cardano.Api.Serialise.SerialiseUsing` or `Cardano.Api.Serialise.Cbor`. + +### 4.16 `NoFieldSelectors` on `Env.hs` + +`Env.hs` has `{-# LANGUAGE NoFieldSelectors #-}`. +This means `RpcEnv` fields like `rpcNodeKernelAccess` are NOT available as accessor functions. +The `Has` instance in `Monad.hs` must use `NamedFieldPuns`: + +```haskell +-- This works (NamedFieldPuns): +instance Has (IORef (Maybe NodeKernelAccess)) RpcEnv where + obtain RpcEnv{rpcNodeKernelAccess} = rpcNodeKernelAccess + +-- This does NOT work (NoFieldSelectors blocks it): +instance Has (IORef (Maybe NodeKernelAccess)) RpcEnv where + obtain = rpcNodeKernelAccess -- ERROR: not a function +``` + +The existing `Has LocalNodeConnectInfo RpcEnv` instance already uses `NamedFieldPuns`, so just follow the same pattern. + +### 4.17 Removing `import Cardano.Api` is high-risk + +Three files (Env.hs, Monad.hs, Server.hs) currently have blanket `import Cardano.Api` which re-exports hundreds of names. +The plan replaces these with specific imports. +This is the most likely source of compilation errors because it's easy to miss a name that was silently in scope. + +**Strategy:** For each file where `import Cardano.Api` is removed: +1. Remove the import +2. Try to build +3. Add specific imports for each "not in scope" error +4. Repeat until clean + +Files that **keep** `import Cardano.Api` (because they use many names from it): +- `Query.hs` - uses `AnyCardanoEra`, `forEraInEon`, `Era`, `convert`, `ShelleyBasedEra`, `UTxO`, `TxIn`, `TxIx`, `TxOut`, `CtxUTxO`, `QueryUTxOFilter`, `ChainPoint`, `BlockNo`, `WithOrigin`, `serialiseToRawBytesHexText`, `serialiseToRawBytes`, `deserialiseFromRawBytes`, `AsTxId`, `IsEra`, `obtainCommonConstraints`, `fromList`, etc. +- `Submit.hs` - uses `AnyCardanoEra`, `forEraInEon`, `ShelleyBasedEra`, `Tx`, `TxId`, `TxInMode`, `shelleyBasedEraConstraints`, `deserialiseFromCBOR`, `getTxId`, `getTxBody`, `serialiseToRawBytes`, etc. +- `Node.hs` - uses `AnyCardanoEra`, `forEraInEon`, `Era`, `convert`, `obtainCommonConstraints`, etc. + +For these files, **keep `import Cardano.Api`** and just add the `NodeKernelAccess` import alongside it. + +### 4.18 `SubmitResult` and `TxValidationErrorInCardanoMode` imports + +`SubmitResult` and `TxValidationErrorInCardanoMode` need explicit imports - they are NOT re-exported from the top-level `Cardano.Api` module. +Import both from `Cardano.Api.Network.IPC` (or alternatively, `SubmitResult` is available via `Cardano.Api.Network` and originates from `Ouroboros.Network.Protocol.LocalTxSubmission.Type`). + +This matters for `NodeKernelAccess.hs` which uses both in its type signature. + +### 4.19 No new dependencies required + +`cardano-ledger-core` and `grpc-spec` are already listed in the `cardano-rpc.cabal` dependencies. +The `NodeKernelAccess` interface uses only cardano-api types, so no consensus dependencies are needed in cardano-rpc. +The `mkNodeKernelAccess` implementation in cardano-node already has all necessary consensus dependencies. + +### 4.20 `withNodeKernelAccess` throws gRPC `UNAVAILABLE` during startup + +The chosen variant is `withNodeAccessOrUnavailable`, which reads the `IORef (Maybe NodeKernelAccess)` and throws a gRPC `UNAVAILABLE` status code if the value is `Nothing`. +This cleanly handles the startup window before `rnNodeKernelHook` writes the `Just` value. +The gRPC client can retry on `UNAVAILABLE`, which is the standard practice for transient unavailability. + +--- + +## 5. RPC Method Query Inventory + +This section lists the exact queries each RPC method executes within a single `executeLocalStateQueryExpr` call (i.e. a single ledger snapshot). +All queries within one call are consistent - they see the same ledger state. + +### ReadParams (in `Query.hs`) + +1. `queryProtocolParameters` - protocol parameters for the current era +2. `queryChainPoint` - chain tip +3. `queryChainBlockNo` - block number at tip +4. `querySystemStart` - for slot-to-timestamp conversion (via `slotToTimestamp`) +5. `queryEraHistory` - for slot-to-timestamp conversion (via `slotToTimestamp`) + +### ReadUtxos (in `Query.hs`) + +1. `queryUtxo` (by TxIn set or whole) - the requested UTxOs +2. `queryChainPoint` - chain tip +3. `queryChainBlockNo` - block number at tip +4. `querySystemStart` - for slot-to-timestamp conversion (via `slotToTimestamp`) +5. `queryEraHistory` - for slot-to-timestamp conversion (via `slotToTimestamp`) + +### SearchUtxos (in `Query.hs`) + +1. `queryUtxo` (by address set or whole, then client-side predicate filtering + pagination) +2. `queryChainPoint` - chain tip +3. `queryChainBlockNo` - block number at tip +4. `querySystemStart` - for slot-to-timestamp conversion (via `slotToTimestamp`) +5. `queryEraHistory` - for slot-to-timestamp conversion (via `slotToTimestamp`) + +### EvalTx (in `Eval.hs`) + +This is the most query-intensive method, requiring 7 queries within one snapshot: + +1. `queryProtocolParameters` - needed for tx evaluation +2. `queryUtxo` (by TxIn set) - inputs being spent +3. `querySystemStart` - for slot-to-time conversion +4. `queryEraHistory` - for slot-to-time conversion +5. `queryStakeDelegDeposits` - for deposit tracking +6. `queryDRepState` - for governance-related evaluation +7. `queryStakePoolParameters` - for delegation-related evaluation + +### GetProtocolParamsJson (in `Node.hs`) + +1. `queryProtocolParameters` - protocol parameters as JSON + +### SubmitTx (in `Submit.hs`) + +Uses `submitTxToNodeLocal` (LocalTxSubmission mini-protocol), not LocalStateQuery. +In the direct access design, this maps to `nkaSubmitTx` (mempool `addLocalTxs`). + +### Slot-to-timestamp conversion note + +`ReadParams`, `ReadUtxos`, and `SearchUtxos` all query `querySystemStart` and `queryEraHistory` for slot-to-timestamp conversion via the `slotToTimestamp` helper. +`EvalTx` also queries both for slot-to-time conversion. +The original plan did not account for these - they must be included in the `LedgerSnapshot` queries. +Both are handled by `answerQuery`: `GetSystemStart` is extracted from config, and `queryEraHistory` maps to `BlockQuery (QueryHardFork GetInterpreter)`. + +--- + +## 7. Verification Checklist + +After implementing all steps: + +- [ ] `nix build 'path:/work/cardano-api#cardano-rpc:lib:cardano-rpc' --allow-import-from-derivation --accept-flake-config` +- [ ] `nix build 'path:/work/cardano-api#cardano-rpc:test:cardano-rpc-test' --allow-import-from-derivation --accept-flake-config` +- [ ] `nix build 'path:/work/cardano-node#cardano-node:lib:cardano-node' --allow-import-from-derivation --accept-flake-config` (adjust path for worktree) +- [ ] No `-Wunused-packages` warnings +- [ ] No `-Wredundant-constraints` warnings +- [ ] hlint passes (backtick-infix, no mixed styles) +- [ ] Integration tests pass (testnet gRPC tests exercise the full path) +- [ ] Startup window test: gRPC query before kernel init returns `UNAVAILABLE` + +--- + +## 8. Files in this documentation set + +| File | Purpose | +|------|---------| +| `ADR-019-node-kernel-access-for-cardano-rpc.md` | Architecture Decision Record | +| `implementation-plan.md` | Step-by-step implementation plan with before/after code | +| `analysis-architecture.md` | Architecture and current state | +| `analysis-utxohd-internals.md` | UTxO-HD internals | +| `analysis-consensus-protocol.md` | Consensus protocol and snapshot consistency | +| `prereqs-api-signatures.md` | API signatures and type references | +| `prereqs-build-and-conventions.md` | Build instructions and conventions | +| `prereqs-implementation-details.md` | **This file** - implementation details, query inventory, and verification checklist | diff --git a/cardano-rpc/docs/story-read-genesis.md b/cardano-rpc/docs/story-read-genesis.md new file mode 100644 index 0000000000..a4932bdc97 --- /dev/null +++ b/cardano-rpc/docs/story-read-genesis.md @@ -0,0 +1,121 @@ +# Implement ReadGenesis UTxO RPC method + +## Problem + +The UTxO RPC spec defines a `ReadGenesis` method on `QueryService` that returns genesis configuration data (genesis hash, CAIP-2 network identifier, and structured genesis parameters across all eras). +cardano-rpc's local `query.proto` already has the `ReadGenesisRequest` and `ReadGenesisResponse` message types, but the `rpc ReadGenesis` line is missing from the `QueryService` block and no handler exists. + +Phased implementation plan: [node-kernel-access/plan-readgenesis-phases.md](node-kernel-access/plan-readgenesis-phases.md). + +## Why + +Genesis configuration is static, well-defined data that tooling and wallets need to discover network parameters at startup. +Without `ReadGenesis`, clients must parse genesis JSON files out-of-band, duplicating logic that the node already has. + +## User value + +As a UTxO RPC client developer, I want to call `ReadGenesis` on the query service so that I can programmatically obtain the network's genesis configuration (hash, CAIP-2 ID, and era-specific parameters) without parsing raw genesis files. + +## Acceptance criteria + +1. **AC1: Proto service registration** - The `rpc ReadGenesis(ReadGenesisRequest) returns (ReadGenesisResponse)` line is added to the `QueryService` block in `proto/utxorpc/v1beta/query/query.proto`. + Generated code is regenerated via `buf generate` in the nix devshell. + The handler is registered in `methodsUtxoRpc` in `Server.hs` in the correct positional slot matching `ServiceMethods` order. + - Test: unit - compiles; the new method slot is exercised by the gRPC method table construction + +2. **AC2: NodeKernelAccessF genesis field** - `NodeKernelAccessF` gains a new pure field (e.g. `nkaGenesisConfig :: GenesisBundle`) that holds the genesis configuration data as Haskell-native types (not proto types). + Being a pure (non-`m`) field reflects the fact that genesis data is static and never changes after node startup. + `GenesisBundle` carries the Byron and Shelley genesis hashes, network magic, and per-era genesis configs (Shelley, Alonzo, Conway) needed to populate the response. + `mkNodeKernelAccess` extracts Byron, Alonzo and Conway data from `TopLevelConfig` via `getTopLevelConfig`, and reads the full Shelley genesis from disk (path threaded from `NodeConfiguration`) to recover `sgInitialFunds`/`sgStaking`, which `CompactGenesis` erases in memory, plus the Shelley genesis hash. + See the plan doc for the wiring details and fallback behaviour. + - Test: unit - compiles; the new field is present on the record + +3. **AC3: Genesis-to-proto conversion** - A pure conversion function (e.g. `genesisToUtxoRpcGenesis`) in the `Type` module converts the Haskell genesis config values into the proto-lens `Genesis` message. + The function maps fields from all four eras: Byron (avvm_distr, block_version_data, start_time, protocol_consts, etc.), Shelley (active_slots_coeff, epoch_length, network_magic, protocol_params, system_start, etc.), Alonzo (lovelace_per_utxo_word, execution_prices, max_tx_ex_units, etc.), and Conway (committee, constitution, drep fields, voting thresholds, etc.). + - Test: unit - `H.propertyOnce` golden test: feed known testnet genesis fixture values, assert specific proto fields are populated correctly (e.g. network_magic, system_start, epoch_length, active_slots_coeff). + This is the primary unit-testable surface for the feature. + +4. **AC4: Handler assembly** - A `readGenesisMethod` handler in `Cardano.Rpc.Server.Internal.UtxoRpc.Query` (or a new module if warranted) grabs the `NodeKernelAccessF` via `grabNodeKernelAccess`, calls the genesis callback, runs the conversion function, and assembles the `ReadGenesisResponse`. + The response populates all three top-level fields: `genesis` (hash bytes), `caip2` (CIP-34 network identifier), and `cardano` (structured `Genesis` message). + - Test: E2E - calling `ReadGenesis` on a running node returns a response with all three top-level fields populated and non-empty + +5. **AC5: Tracing** - `TraceRpcQuery` gains a `TraceRpcQueryGenesisSpan TraceSpanEvent` constructor. + `Pretty` renders it as "Started ReadGenesis method" / "Finished ReadGenesis method". + The handler is wrapped in `wrapInSpan TraceRpcQueryGenesisSpan`. + - Test: unit - `H.propertyOnce` asserting the `Pretty` output of the new constructor contains the expected substrings + +6. **AC6: Node kernel unavailable** - When the node kernel is not yet initialised (`IORef` contains `Nothing`), calling `ReadGenesis` returns gRPC `UNAVAILABLE` with a message containing "not yet initialised". + This is consistent with the existing `FetchBlock` behaviour via `grabNodeKernelAccess`. + - Test: unit - `H.propertyOnce`: create `IORef Nothing`, call through the handler path, assert `GrpcException` with `GrpcUnavailable` is thrown (same pattern as the `NodeKernelAccess` unavailability unit test in piece 1) + +## Out of scope + +- **Field mask support**: `field_mask` in `ReadGenesisRequest` is ignored, consistent with `ReadParams`, `SearchUtxos`, and all other existing handlers. + Field mask normalisation and filtering is a cross-cutting concern for a separate story. +- **Byron genesis file fields requiring file I/O**: some Byron genesis fields (e.g. `vss_certs`, `heavy_delegation` maps with complex nested structures) may be deferred to a follow-up if the ledger types do not expose them directly through `TopLevelConfig`. +- **Inverse proto-to-genesis conversion**: unlike protocol parameters, `ReadGenesis` is read-only with no need for round-tripping. + No `utxoRpcGenesisToGenesis` function is needed. +- **ReadEraSummary**: this is a separate `QueryService` method defined in the upstream spec; it warrants its own story. +- **Minimum era guard**: unlike `ReadParams` which requires Conway, `ReadGenesis` returns static configuration from all eras and does not need an era check. + +## Definition of done + +- [ ] All AC tests written (compile, fail on stubs) +- [ ] Implementation complete (all tests pass via `cabal test`) +- [ ] Nix CI checks pass (`nix build 'path:.#checks.x86_64-linux.test'` and `e2e`) +- [ ] haskell-reviewer agent finds no critical or style issues +- [ ] fourmolu clean +- [ ] No build warnings + +## Notes + +### Open questions + +- **Which genesis hash for the `genesis` bytes field?** + Cardano has four genesis files (Byron, Shelley, Alonzo, Conway), each with its own hash. + The proto spec says "genesis hash for the chain" (singular). + Both candidates are available: the Byron hash from `TopLevelConfig`, the Shelley hash from hashing the genesis file bytes (see the plan doc). + Need to check what Dolos and other UTxO RPC implementations return here. + Most likely the Shelley genesis hash, as it is the one that identifies the network, but this must be confirmed against the spec or reference implementations before implementation. + The `utxorpc-compare` skill can verify what other servers return. + +- **CAIP-2 format and derivation.** + CIP-34 defines Cardano's CAIP-2 identifier. + The exact format and how to derive it from genesis data (genesis hash prefix? network magic?) needs confirmation from CIP-34 and cross-referencing with Dolos output. + +### Implementation risks + +- **Extracting per-era genesis from `TopLevelConfig`/`HardForkBlock`.** + The `mkNodeKernelAccess` function currently only uses `getChainDB` from the `NodeKernel`. + Accessing genesis configs likely requires `getTopLevelConfig` and then navigating the `HardForkLedgerConfig` to extract per-era genesis. + This may need consensus-specific accessors or new cardano-api surface. + Worth spiking early. + +- **Proto field coverage.** + The `Genesis` proto message has ~42 fields across four eras. + Some map directly to ledger types (epoch_length, network_magic); others (avvm_distr, vss_certs) may require non-trivial extraction from Byron genesis. + The golden test in AC3 should cover the high-value fields first, with explicit TODOs for any deferred fields. + +### Design decisions + +- **Genesis data is static.** + Unlike protocol parameters or UTxO state, genesis configs never change after node startup. + This means the handler can cache the converted proto message on first call, avoiding repeated conversion. + However, caching is an optimisation and not required for correctness. + +- **Follows FetchBlock pattern, with a key difference.** + FetchBlock uses an `m`-wrapped callback because it queries ChainDB per request. + ReadGenesis uses a pure field because genesis data is static. + The conversion layer (AC3) maps to proto, handler (AC4) assembles the response. + This keeps the node-kernel boundary clean and the conversion testable in isolation. + +### E2E harness + +- The E2E test in AC4 depends on having a running cardano-node with the gRPC server enabled. + The exact E2E harness location needs confirming (likely `cardano-node-tests` or nix-level integration tests). + +### Dependencies + +- Upstream proto: the `ReadGenesis` service line must be added to the local `query.proto` and code regenerated (AC1). +- cardano-node wiring: `mkNodeKernelAccess` must be updated to build the new `GenesisBundle` field and gains a Shelley genesis `FilePath` parameter. + The node-side call site (`rnNodeKernelHook` in `handleSimpleNode`, `cardano-node/src/Cardano/Node/Run.hs`) has `NodeConfiguration` in scope, so the path is extracted there from `ncProtocolConfig`'s `npcShelleyGenesisFile`. diff --git a/cardano-rpc/gen/Proto/Utxorpc/V1beta/Query/Query.hs b/cardano-rpc/gen/Proto/Utxorpc/V1beta/Query/Query.hs index 98174f8b9f..8ba43cf1b0 100644 --- a/cardano-rpc/gen/Proto/Utxorpc/V1beta/Query/Query.hs +++ b/cardano-rpc/gen/Proto/Utxorpc/V1beta/Query/Query.hs @@ -4618,7 +4618,8 @@ data QueryService = QueryService {} instance Data.ProtoLens.Service.Types.Service QueryService where type ServiceName QueryService = "QueryService" type ServicePackage QueryService = "utxorpc.v1beta.query" - type ServiceMethods QueryService = '["readParams", + type ServiceMethods QueryService = '["readGenesis", + "readParams", "readUtxos", "searchUtxos"] packedServiceDescriptor _ @@ -4627,7 +4628,8 @@ instance Data.ProtoLens.Service.Types.Service QueryService where \\n\ \ReadParams\DC2'.utxorpc.v1beta.query.ReadParamsRequest\SUB(.utxorpc.v1beta.query.ReadParamsResponse\DC2\\\n\ \\tReadUtxos\DC2&.utxorpc.v1beta.query.ReadUtxosRequest\SUB'.utxorpc.v1beta.query.ReadUtxosResponse\DC2b\n\ - \\vSearchUtxos\DC2(.utxorpc.v1beta.query.SearchUtxosRequest\SUB).utxorpc.v1beta.query.SearchUtxosResponse" + \\vSearchUtxos\DC2(.utxorpc.v1beta.query.SearchUtxosRequest\SUB).utxorpc.v1beta.query.SearchUtxosResponse\DC2b\n\ + \\vReadGenesis\DC2(.utxorpc.v1beta.query.ReadGenesisRequest\SUB).utxorpc.v1beta.query.ReadGenesisResponse" instance Data.ProtoLens.Service.Types.HasMethodImpl QueryService "readParams" where type MethodName QueryService "readParams" = "ReadParams" type MethodInput QueryService "readParams" = ReadParamsRequest @@ -4643,6 +4645,11 @@ instance Data.ProtoLens.Service.Types.HasMethodImpl QueryService "searchUtxos" w type MethodInput QueryService "searchUtxos" = SearchUtxosRequest type MethodOutput QueryService "searchUtxos" = SearchUtxosResponse type MethodStreamingType QueryService "searchUtxos" = 'Data.ProtoLens.Service.Types.NonStreaming +instance Data.ProtoLens.Service.Types.HasMethodImpl QueryService "readGenesis" where + type MethodName QueryService "readGenesis" = "ReadGenesis" + type MethodInput QueryService "readGenesis" = ReadGenesisRequest + type MethodOutput QueryService "readGenesis" = ReadGenesisResponse + type MethodStreamingType QueryService "readGenesis" = 'Data.ProtoLens.Service.Types.NonStreaming packedFileDescriptor :: Data.ByteString.ByteString packedFileDescriptor = "\n\ @@ -4752,15 +4759,16 @@ packedFileDescriptor \\SOReadTxResponse\DC20\n\ \\STXtx\CAN\SOH \SOH(\v2 .utxorpc.v1beta.query.AnyChainTxR\STXtx\DC2?\n\ \\n\ - \ledger_tip\CAN\STX \SOH(\v2 .utxorpc.v1beta.query.ChainPointR\tledgerTip2\177\STX\n\ + \ledger_tip\CAN\STX \SOH(\v2 .utxorpc.v1beta.query.ChainPointR\tledgerTip2\149\ETX\n\ \\fQueryService\DC2_\n\ \\n\ \ReadParams\DC2'.utxorpc.v1beta.query.ReadParamsRequest\SUB(.utxorpc.v1beta.query.ReadParamsResponse\DC2\\\n\ \\tReadUtxos\DC2&.utxorpc.v1beta.query.ReadUtxosRequest\SUB'.utxorpc.v1beta.query.ReadUtxosResponse\DC2b\n\ - \\vSearchUtxos\DC2(.utxorpc.v1beta.query.SearchUtxosRequest\SUB).utxorpc.v1beta.query.SearchUtxosResponseB\152\SOH\n\ + \\vSearchUtxos\DC2(.utxorpc.v1beta.query.SearchUtxosRequest\SUB).utxorpc.v1beta.query.SearchUtxosResponse\DC2b\n\ + \\vReadGenesis\DC2(.utxorpc.v1beta.query.ReadGenesisRequest\SUB).utxorpc.v1beta.query.ReadGenesisResponseB\152\SOH\n\ \\CANcom.utxorpc.v1beta.queryB\n\ - \QueryProtoP\SOH\162\STX\ETXUVQ\170\STX\DC4Utxorpc.V1beta.Query\202\STX\DC4Utxorpc\\V1beta\\Query\226\STX Utxorpc\\V1beta\\Query\\GPBMetadata\234\STX\SYNUtxorpc::V1beta::QueryJ\144;\n\ - \\a\DC2\ENQ\STX\NUL\177\SOH\SOH\n\ + \QueryProtoP\SOH\162\STX\ETXUVQ\170\STX\DC4Utxorpc.V1beta.Query\202\STX\DC4Utxorpc\\V1beta\\Query\226\STX Utxorpc\\V1beta\\Query\\GPBMetadata\234\STX\SYNUtxorpc::V1beta::QueryJ\236;\n\ + \\a\DC2\ENQ\STX\NUL\178\SOH\SOH\n\ \9\n\ \\SOH\f\DC2\ETX\STX\NUL\DC22// A consistent view of the state of the ledger\n\ \\n\ @@ -5451,7 +5459,7 @@ packedFileDescriptor \\r\n\ \\ENQ\EOT\SYN\STX\SOH\ETX\DC2\EOT\165\SOH\SUB\ESC\n\ \G\n\ - \\STX\ACK\NUL\DC2\ACK\169\SOH\NUL\177\SOH\SOH\SUB9 Service definition for querying the state of the chain.\n\ + \\STX\ACK\NUL\DC2\ACK\169\SOH\NUL\178\SOH\SOH\SUB9 Service definition for querying the state of the chain.\n\ \\n\ \\v\n\ \\ETX\ACK\NUL\SOH\DC2\EOT\169\SOH\b\DC4\n\ @@ -5481,4 +5489,13 @@ packedFileDescriptor \\r\n\ \\ENQ\ACK\NUL\STX\STX\STX\DC2\EOT\172\SOH\DC2$\n\ \\r\n\ - \\ENQ\ACK\NUL\STX\STX\ETX\DC2\EOT\172\SOH/Bb\ACKproto3" \ No newline at end of file + \\ENQ\ACK\NUL\STX\STX\ETX\DC2\EOT\172\SOH/B\n\ + \-\n\ + \\EOT\ACK\NUL\STX\ETX\DC2\EOT\173\SOH\STXD\"\US Get the chain genesis config.\n\ + \\n\ + \\r\n\ + \\ENQ\ACK\NUL\STX\ETX\SOH\DC2\EOT\173\SOH\ACK\DC1\n\ + \\r\n\ + \\ENQ\ACK\NUL\STX\ETX\STX\DC2\EOT\173\SOH\DC2$\n\ + \\r\n\ + \\ENQ\ACK\NUL\STX\ETX\ETX\DC2\EOT\173\SOH/Bb\ACKproto3" \ No newline at end of file diff --git a/cardano-rpc/proto/utxorpc/v1beta/query/query.proto b/cardano-rpc/proto/utxorpc/v1beta/query/query.proto index 8040718ad8..d69b8a0b02 100644 --- a/cardano-rpc/proto/utxorpc/v1beta/query/query.proto +++ b/cardano-rpc/proto/utxorpc/v1beta/query/query.proto @@ -171,6 +171,7 @@ service QueryService { rpc ReadParams(ReadParamsRequest) returns (ReadParamsResponse); // Get overall chain state. rpc ReadUtxos(ReadUtxosRequest) returns (ReadUtxosResponse); // Read specific UTxOs by reference. rpc SearchUtxos(SearchUtxosRequest) returns (SearchUtxosResponse); // Search for UTxO based on a pattern. + rpc ReadGenesis(ReadGenesisRequest) returns (ReadGenesisResponse); // Get the chain genesis config. // TODO: decide if we want to expand the scope // rpc DumpUtxos(ReadUtxosRequest) returns (stream ReadUtxosResponse); // Dump all available utxos diff --git a/cardano-rpc/src/Cardano/Rpc/Server.hs b/cardano-rpc/src/Cardano/Rpc/Server.hs index cf788f6dad..5d14019541 100644 --- a/cardano-rpc/src/Cardano/Rpc/Server.hs +++ b/cardano-rpc/src/Cardano/Rpc/Server.hs @@ -18,6 +18,7 @@ module Cardano.Rpc.Server , TraceRpcSubmit (..) , TraceRpcQuery (..) , TraceRpcSync (..) + , TraceRpcNodeKernelAccess (..) , TraceSpanEvent (..) ) where @@ -37,7 +38,10 @@ import Cardano.Rpc.Server.Internal.UtxoRpc.Eval import Cardano.Rpc.Server.Internal.UtxoRpc.Query import Cardano.Rpc.Server.Internal.UtxoRpc.Submit import Cardano.Rpc.Server.Internal.UtxoRpc.Sync -import Cardano.Rpc.Server.NodeKernelAccess (NodeKernelAccess, mkNodeKernelAccess) +import Cardano.Rpc.Server.NodeKernelAccess + ( NodeKernelAccess + , mkNodeKernelAccess + ) import RIO @@ -62,7 +66,8 @@ methodsUtxoRpc :: MonadRpc e m => Methods m (ProtobufMethodsOf UtxoRpc.QueryService) methodsUtxoRpc = - Method (mkNonStreaming $ wrapInSpan TraceRpcQueryParamsSpan . readParamsMethod) + Method (mkNonStreaming $ wrapInSpan TraceRpcQueryReadGenesisSpan . readGenesisMethod) + . Method (mkNonStreaming $ wrapInSpan TraceRpcQueryParamsSpan . readParamsMethod) . Method (mkNonStreaming $ wrapInSpan TraceRpcQueryReadUtxosSpan . readUtxosMethod) . Method (mkNonStreaming $ wrapInSpan TraceRpcQuerySearchUtxosSpan . searchUtxosMethod) $ NoMoreMethods diff --git a/cardano-rpc/src/Cardano/Rpc/Server/Internal/Tracing.hs b/cardano-rpc/src/Cardano/Rpc/Server/Internal/Tracing.hs index 917760bb8b..1aae342493 100644 --- a/cardano-rpc/src/Cardano/Rpc/Server/Internal/Tracing.hs +++ b/cardano-rpc/src/Cardano/Rpc/Server/Internal/Tracing.hs @@ -21,6 +21,7 @@ data TraceRpc = TraceRpcQuery TraceRpcQuery | TraceRpcSubmit TraceRpcSubmit | TraceRpcSync TraceRpcSync + | TraceRpcNodeKernelAccess TraceRpcNodeKernelAccess | TraceRpcError SomeException | TraceRpcFatalError SomeException @@ -32,6 +33,8 @@ data TraceRpcQuery TraceRpcQueryReadUtxosSpan TraceSpanEvent | -- | Span trace marking SearchUtxos query TraceRpcQuerySearchUtxosSpan TraceSpanEvent + | -- | Span trace marking ReadGenesis query + TraceRpcQueryReadGenesisSpan TraceSpanEvent deriving Show instance Pretty TraceRpc where @@ -39,6 +42,7 @@ instance Pretty TraceRpc where TraceRpcQuery t -> pretty t TraceRpcSubmit t -> pretty t TraceRpcSync t -> pretty t + TraceRpcNodeKernelAccess t -> pretty t TraceRpcError e -> "Exception when processing RPC request:\n" <> prettyException e TraceRpcFatalError e -> "RPC server fatal error: " <> prettyException e @@ -61,6 +65,8 @@ instance Pretty TraceRpcQuery where TraceRpcQueryReadUtxosSpan (SpanEnd _) -> "Finished query read UTXO method" TraceRpcQuerySearchUtxosSpan (SpanBegin _) -> "Started query search UTXO method" TraceRpcQuerySearchUtxosSpan (SpanEnd _) -> "Finished query search UTXO method" + TraceRpcQueryReadGenesisSpan (SpanBegin _) -> "Started query read genesis method" + TraceRpcQueryReadGenesisSpan (SpanEnd _) -> "Finished query read genesis method" instance Error TraceRpcQuery where prettyError = pretty @@ -123,6 +129,24 @@ instance Pretty TraceRpcSync where instance Error TraceRpcSync where prettyError = pretty +-- | Traces emitted while setting up in-process node kernel access. +-- +-- None of these are fatal: the node always starts, and RPC either serves +-- reduced data or reports @UNAVAILABLE@. +newtype TraceRpcNodeKernelAccess + = -- | Node kernel access is not supported for the running block type, + -- given here by name. + TraceRpcUnsupportedBlockType Text + deriving Show + +instance Pretty TraceRpcNodeKernelAccess where + pretty = \case + TraceRpcUnsupportedBlockType blockType -> + "Node kernel access is not supported for block type: " <> pretty blockType + +instance Error TraceRpcNodeKernelAccess where + prettyError = pretty + instance Inject TraceRpcSubmit TraceRpc where inject = TraceRpcSubmit @@ -131,3 +155,6 @@ instance Inject TraceRpcQuery TraceRpc where instance Inject TraceRpcSync TraceRpc where inject = TraceRpcSync + +instance Inject TraceRpcNodeKernelAccess TraceRpc where + inject = TraceRpcNodeKernelAccess diff --git a/cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/Query.hs b/cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/Query.hs index e6d0cc2749..3baff121bf 100644 --- a/cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/Query.hs +++ b/cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/Query.hs @@ -4,6 +4,7 @@ {-# LANGUAGE FlexibleContexts #-} {-# LANGUAGE GADTs #-} {-# LANGUAGE LambdaCase #-} +{-# LANGUAGE NamedFieldPuns #-} {-# LANGUAGE NumericUnderscores #-} {-# LANGUAGE QuantifiedConstraints #-} {-# LANGUAGE RankNTypes #-} @@ -14,6 +15,7 @@ module Cardano.Rpc.Server.Internal.UtxoRpc.Query ( readParamsMethod , readUtxosMethod , searchUtxosMethod + , readGenesisMethod , paginateByTxIn ) where @@ -28,6 +30,11 @@ import Cardano.Rpc.Server.Internal.Monad import Cardano.Rpc.Server.Internal.Orphans () import Cardano.Rpc.Server.Internal.UtxoRpc.Predicate import Cardano.Rpc.Server.Internal.UtxoRpc.Type +import Cardano.Rpc.Server.NodeKernelAccess + +import Cardano.Crypto.Hash.Class qualified as Crypto (hashToBytes) +import Cardano.Ledger.Api.Transition qualified as L (tcShelleyGenesisL) +import Cardano.Ledger.Shelley.Genesis qualified as L (sgNetworkMagic) import RIO hiding (toList) @@ -159,6 +166,37 @@ searchUtxosMethod req = do & U5c.items .~ map (uncurry txInTxOutToAnyUtxoData) page & U5c.maybe'nextToken .~ nextTok +-- | Handle the @ReadGenesis@ RPC method. +-- Returns the chain's identity - the Shelley genesis hash and the CAIP-2 chain +-- identifier - together with the @cardano@ config, the Byron, Shelley, Alonzo +-- and Conway genesis parameters mapped by 'genesisBundleToProto'. +readGenesisMethod + :: MonadRpc e m + => Proto UtxoRpc.ReadGenesisRequest + -> m (Proto UtxoRpc.ReadGenesisResponse) +readGenesisMethod _req = do + -- TODO: field masks are ignored for now (same as readParamsMethod) + NodeKernelAccess{genesisConfig = genesisBundle@GenesisBundle{shelleyGenesisHash, transitionConfig}} <- + grabNodeKernelAccess + let networkMagic = L.sgNetworkMagic $ transitionConfig ^. L.tcShelleyGenesisL + pure $ + defMessage + & U5c.genesis .~ Crypto.hashToBytes (unGenesisHashShelley shelleyGenesisHash) + & U5c.caip2 .~ networkMagicToCaip2 networkMagic + & U5c.cardano .~ genesisBundleToProto genesisBundle + +-- | The CAIP-2 chain identifier for a Cardano network, keyed on the Shelley +-- network magic. +-- This follows Dolos, the reference UTxO RPC implementation: the three +-- well-known networks get their conventional names, and any other network is +-- identified by its magic. +networkMagicToCaip2 :: Word32 -> Text +networkMagicToCaip2 = \case + 764824073 -> "cardano:mainnet" + 1 -> "cardano:preprod" + 2 -> "cardano:preview" + magic -> "cardano:" <> tshow magic + -- | Paginate a list of UTxO entries using cursor-based pagination. -- Items are sorted by 'TxIn'\'s 'Ord' instance (lexicographic on 'TxId', then numeric on 'TxIx'). -- The start token is the 'renderTxIn' of the last item on the previous page; diff --git a/cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/Type.hs b/cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/Type.hs index 10db62a0cf..4657fcdf71 100644 --- a/cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/Type.hs +++ b/cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/Type.hs @@ -4,6 +4,7 @@ -- working unchanged. module Cardano.Rpc.Server.Internal.UtxoRpc.Type ( utxoRpcPParamsToProtocolParams + , genesisBundleToProto , utxoToUtxoRpcAnyUtxoData , txInTxOutToAnyUtxoData , anyUtxoDataUtxoRpcToUtxo @@ -30,6 +31,7 @@ where import Cardano.Rpc.Server.Internal.UtxoRpc.Type.BigInt import Cardano.Rpc.Server.Internal.UtxoRpc.Type.ChainPoint +import Cardano.Rpc.Server.Internal.UtxoRpc.Type.Genesis import Cardano.Rpc.Server.Internal.UtxoRpc.Type.PlutusData import Cardano.Rpc.Server.Internal.UtxoRpc.Type.ProtocolParameters import Cardano.Rpc.Server.Internal.UtxoRpc.Type.Rational diff --git a/cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/Type/Certificate.hs b/cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/Type/Certificate.hs index 66cc192426..b54993b12f 100644 --- a/cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/Type/Certificate.hs +++ b/cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/Type/Certificate.hs @@ -7,6 +7,7 @@ module Cardano.Rpc.Server.Internal.UtxoRpc.Type.Certificate , credentialToUtxoRpcStakeCredential , anchorToUtxoRpcAnchor , scriptHashToBytes + , keyHashToBytes ) where diff --git a/cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/Type/Genesis.hs b/cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/Type/Genesis.hs new file mode 100644 index 0000000000..a20ec7d260 --- /dev/null +++ b/cardano-rpc/src/Cardano/Rpc/Server/Internal/UtxoRpc/Type/Genesis.hs @@ -0,0 +1,412 @@ +{-# LANGUAGE LambdaCase #-} +{-# LANGUAGE NamedFieldPuns #-} +{-# LANGUAGE NumericUnderscores #-} + +-- | Conversion of the network's per-era genesis configuration to the UTxO RPC +-- 'U5c.Genesis' message. +-- +-- The @Genesis@ message has one field per Byron, Shelley, Alonzo and Conway +-- genesis parameter. Each era is mapped by an updater over one shared +-- accumulator ('byronGenesisToProto', 'shelleyGenesisToProto', +-- 'alonzoGenesisToProto', 'conwayGenesisToProto'); 'genesisBundleToProto' +-- threads a single 'defMessage' through all four. The shared accumulator lets +-- Alonzo and Conway both contribute to the single @cost_models@ field +-- (PlutusV1 from Alonzo, PlutusV3 from Conway) without any message merge. +module Cardano.Rpc.Server.Internal.UtxoRpc.Type.Genesis + ( genesisBundleToProto + ) +where + +-- '?~' is not exported for the proto-lens / grapesy lens types used here, so the +-- @cost_models@ setters use the idiomatic proto-lens @.~ Just@ form instead. +{- HLINT ignore "Use ?~" -} + +import Cardano.Api.Era (Inject (..)) +import Cardano.Api.Ledger qualified as L +import Cardano.Rpc.Proto.Api.UtxoRpc.Query qualified as U5c +import Cardano.Rpc.Server.Internal.Orphans () +import Cardano.Rpc.Server.Internal.UtxoRpc.Type.Certificate + ( anchorToUtxoRpcAnchor + , keyHashToBytes + , scriptHashToBytes + ) +import Cardano.Rpc.Server.NodeKernelAccess.Type (GenesisBundle (..)) + +import Cardano.Chain.Common qualified as Byron + ( KeyHash + , LovelacePortion + , TxFeePolicy (..) + , TxSizeLinear (..) + , addressF + , lovelacePortionToRational + , lovelaceToInteger + , unBlockCount + , unKeyHash + ) +import Cardano.Chain.Delegation qualified as Byron + ( Certificate + , delegateVK + , epoch + , issuerVK + , signature + ) +import Cardano.Chain.Genesis qualified as Byron + ( GenesisAvvmBalances (..) + , GenesisData (..) + , GenesisDelegation (..) + , GenesisKeyHashes (..) + , GenesisNonAvvmBalances (..) + , configGenesisData + ) +import Cardano.Chain.Slotting qualified as Byron (getEpochNumber, unSlotNumber) +import Cardano.Chain.Update qualified as Byron (ProtocolParameters (..), SoftforkRule (..)) +import Cardano.Crypto qualified as Byron + ( fromCompactRedeemVerificationKey + , fullSignatureHexF + , fullVerificationKeyF + , hashHexF + , redeemVKB64UrlF + , unProtocolMagicId + ) +import Cardano.Ledger.Address qualified as L +import Cardano.Ledger.Alonzo.Genesis qualified as L +import Cardano.Ledger.Api qualified as L +import Cardano.Ledger.Api.Transition qualified as L +import Cardano.Ledger.BaseTypes qualified as L +import Cardano.Ledger.Conway.PParams qualified as L +import Cardano.Ledger.Hashes qualified as L +import Cardano.Ledger.Shelley.Genesis qualified as L + +import RIO + +import Data.ByteString.Base16 qualified as Base16 +import Data.Map.Strict qualified as Map +import Data.ProtoLens (defMessage) +import Data.Text qualified as Text (pack) +import Data.Text.Encoding qualified as Text (decodeUtf8) +import Data.Time.Clock.POSIX (utcTimeToPOSIXSeconds) +import Data.Time.Format.ISO8601 (iso8601Show) +import Formatting (sformat) +import GHC.Exts qualified as Exts (toList) +import Network.GRPC.Spec + +-- | Convert the network's genesis bundle to the UTxO RPC 'U5c.Genesis' +-- message, populating the Byron, Shelley, Alonzo and Conway fields. +genesisBundleToProto :: GenesisBundle -> Proto U5c.Genesis +genesisBundleToProto GenesisBundle{byronConfig, transitionConfig} = + byronGenesisToProto byronGenesis + . shelleyGenesisToProto shelleyGenesis + . alonzoGenesisToProto alonzoGenesis + . conwayGenesisToProto conwayGenesis + $ defMessage + where + byronGenesis = Byron.configGenesisData byronConfig + shelleyGenesis = transitionConfig ^. L.tcShelleyGenesisL + -- LatestKnownEra is Dijkstra; its previous era is Conway, whose translation + -- context is the Conway genesis. + conwayGenesis = transitionConfig ^. L.tcPreviousEraConfigL . L.tcTranslationContextL + -- Dijkstra -> Conway -> Babbage -> Alonzo config, whose translation context + -- is the Alonzo genesis. + alonzoGenesis = + transitionConfig + ^. L.tcPreviousEraConfigL + . L.tcPreviousEraConfigL + . L.tcPreviousEraConfigL + . L.tcTranslationContextL + +-------------------------------------------------------------------------------- +-- Byron +-------------------------------------------------------------------------------- + +byronGenesisToProto :: Byron.GenesisData -> Proto U5c.Genesis -> Proto U5c.Genesis +byronGenesisToProto genesisData message = + message + & U5c.avvmDistr .~ avvmDistr + & U5c.blockVersionData .~ blockVersionData (Byron.gdProtocolParameters genesisData) + & U5c.protocolConsts .~ protocolConsts + & U5c.startTime .~ startTime + & U5c.bootStakeholders .~ bootStakeholders + & U5c.heavyDelegation .~ heavyDelegation + & U5c.nonAvvmBalances .~ nonAvvmBalances + where + -- Unix seconds, not milliseconds. + startTime :: Word64 + startTime = round (utcTimeToPOSIXSeconds (Byron.gdStartTime genesisData)) + + protocolConsts :: Proto U5c.ProtocolConsts + protocolConsts = + defMessage + & U5c.k .~ fromIntegral (Byron.unBlockCount (Byron.gdK genesisData)) + & U5c.protocolMagic .~ Byron.unProtocolMagicId (Byron.gdProtocolMagicId genesisData) + + -- vssMaxTtl and vssMinTtl have no Byron ledger source and stay at their + -- proto default of 0. + + -- Byron stores only a set of genesis key hashes; the genesis JSON synthesises + -- weight 1 for each, which is matched here. + bootStakeholders :: Map Text Word64 + bootStakeholders = + Map.fromList + [ (byronKeyHashHex keyHash, 1) + | keyHash <- toList (Byron.unGenesisKeyHashes (Byron.gdGenesisKeyHashes genesisData)) + ] + + heavyDelegation :: Map Text (Proto U5c.HeavyDelegation) + heavyDelegation = + Map.fromList + [ (byronKeyHashHex keyHash, heavyDelegationCert cert) + | (keyHash, cert) <- + Map.toList (Byron.unGenesisDelegation (Byron.gdHeavyDelegation genesisData)) + ] + + nonAvvmBalances :: Map Text Text + nonAvvmBalances = + Map.fromList + [ (sformat Byron.addressF address, tshow (Byron.lovelaceToInteger lovelace)) + | (address, lovelace) <- + Map.toList (Byron.unGenesisNonAvvmBalances (Byron.gdNonAvvmBalances genesisData)) + ] + + avvmDistr :: Map Text Text + avvmDistr = + Map.fromList + [ ( sformat Byron.redeemVKB64UrlF (Byron.fromCompactRedeemVerificationKey redeemKey) + , tshow (Byron.lovelaceToInteger lovelace) + ) + | (redeemKey, lovelace) <- + Map.toList (Byron.unGenesisAvvmBalances (Byron.gdAvvmDistr genesisData)) + ] + +-- | Byron key hashes render as lowercase base16, matching the genesis JSON. +byronKeyHashHex :: Byron.KeyHash -> Text +byronKeyHashHex = sformat Byron.hashHexF . Byron.unKeyHash + +heavyDelegationCert :: Byron.Certificate -> Proto U5c.HeavyDelegation +heavyDelegationCert cert = + defMessage + & U5c.cert .~ sformat Byron.fullSignatureHexF (Byron.signature cert) + & U5c.delegatePk .~ sformat Byron.fullVerificationKeyF (Byron.delegateVK cert) + & U5c.issuerPk .~ sformat Byron.fullVerificationKeyF (Byron.issuerVK cert) + & U5c.omega .~ fromIntegral (Byron.getEpochNumber (Byron.epoch cert)) + +blockVersionData :: Byron.ProtocolParameters -> Proto U5c.BlockVersionData +blockVersionData pp = + defMessage + & U5c.scriptVersion .~ fromIntegral (Byron.ppScriptVersion pp) + & U5c.slotDuration .~ tshow (Byron.ppSlotDuration pp) + & U5c.maxBlockSize .~ tshow (Byron.ppMaxBlockSize pp) + & U5c.maxHeaderSize .~ tshow (Byron.ppMaxHeaderSize pp) + & U5c.maxTxSize .~ tshow (Byron.ppMaxTxSize pp) + & U5c.maxProposalSize .~ tshow (Byron.ppMaxProposalSize pp) + & U5c.mpcThd .~ tshow (lovelacePortionWord (Byron.ppMpcThd pp)) + & U5c.heavyDelThd .~ tshow (lovelacePortionWord (Byron.ppHeavyDelThd pp)) + & U5c.updateVoteThd .~ tshow (lovelacePortionWord (Byron.ppUpdateVoteThd pp)) + & U5c.updateProposalThd .~ tshow (lovelacePortionWord (Byron.ppUpdateProposalThd pp)) + & U5c.updateImplicit .~ tshow (Byron.unSlotNumber (Byron.ppUpdateProposalTTL pp)) + & U5c.unlockStakeEpoch .~ tshow (Byron.getEpochNumber (Byron.ppUnlockStakeEpoch pp)) + & U5c.softforkRule .~ softforkRule (Byron.ppSoftforkRule pp) + & U5c.txFeePolicy .~ txFeePolicy (Byron.ppTxFeePolicy pp) + +softforkRule :: Byron.SoftforkRule -> Proto U5c.SoftforkRule +softforkRule rule = + defMessage + & U5c.initThd .~ tshow (lovelacePortionWord (Byron.srInitThd rule)) + & U5c.minThd .~ tshow (lovelacePortionWord (Byron.srMinThd rule)) + & U5c.thdDecrement .~ tshow (lovelacePortionWord (Byron.srThdDecrement rule)) + +-- | The raw Word64 numerator the Byron genesis JSON emits for a +-- 'Byron.LovelacePortion' (over a fixed 1e15 denominator). The @unLovelacePortion@ +-- accessor is not exported, so the value is recovered from +-- 'Byron.lovelacePortionToRational'. +lovelacePortionWord :: Byron.LovelacePortion -> Word64 +lovelacePortionWord = round . (* 1_000_000_000_000_000) . Byron.lovelacePortionToRational + +txFeePolicy :: Byron.TxFeePolicy -> Proto U5c.TxFeePolicy +txFeePolicy = \case + Byron.TxFeePolicyTxSizeLinear (Byron.TxSizeLinear constant multiplier) -> + defMessage + -- The Byron genesis JSON scales both coefficients by 1e9. + & U5c.summand .~ tshow (1_000_000_000 * Byron.lovelaceToInteger constant) + & U5c.multiplier .~ tshow (floor (1_000_000_000 * multiplier) :: Integer) + +-------------------------------------------------------------------------------- +-- Shelley +-------------------------------------------------------------------------------- + +shelleyGenesisToProto :: L.ShelleyGenesis -> Proto U5c.Genesis -> Proto U5c.Genesis +shelleyGenesisToProto genesis message = + message + & U5c.activeSlotsCoeff .~ inject (L.unboundRational (L.sgActiveSlotsCoeff genesis)) + & U5c.epochLength .~ fromIntegral (L.unEpochSize (L.sgEpochLength genesis)) + & U5c.maxKesEvolutions .~ fromIntegral (L.sgMaxKESEvolutions genesis) + & U5c.slotsPerKesPeriod .~ fromIntegral (L.sgSlotsPerKESPeriod genesis) + & U5c.updateQuorum .~ fromIntegral (L.sgUpdateQuorum genesis) + & U5c.securityParam .~ fromIntegral (L.unNonZero (L.sgSecurityParam genesis)) + & U5c.maxLovelaceSupply .~ inject (fromIntegral (L.sgMaxLovelaceSupply genesis) :: Integer) + & U5c.networkMagic .~ L.sgNetworkMagic genesis + & U5c.networkId .~ networkIdText (L.sgNetworkId genesis) + -- Slot length in milliseconds; mainnet's 1 second yields 1000. + & U5c.slotLength .~ round (1000 * L.fromNominalDiffTimeMicro (L.sgSlotLength genesis)) + & U5c.systemStart .~ Text.pack (iso8601Show (L.sgSystemStart genesis)) + & U5c.genDelegs .~ genDelegs + & U5c.initialFunds .~ initialFunds + & U5c.protocolParams .~ protocolParams + where + networkIdText :: L.Network -> Text + networkIdText = \case + L.Mainnet -> "Mainnet" + L.Testnet -> "Testnet" + + genDelegs :: Map Text (Proto U5c.GenDelegs) + genDelegs = + Map.fromList + [ ( hexText (keyHashToBytes keyHash) + , defMessage + & U5c.delegate .~ hexText (keyHashToBytes (L.genDelegKeyHash pair)) + & U5c.vrf .~ hexText (L.hashToBytes (L.unVRFVerKeyHash (L.genDelegVrfHash pair))) + ) + | (keyHash, pair) <- Map.toList (L.sgGenDelegs genesis) + ] + + initialFunds :: Map Text (Proto U5c.BigInt) + initialFunds = + Map.fromList + [ (hexText (L.serialiseAddr address), inject coin) + | (address, coin) <- Exts.toList (L.sgInitialFunds genesis) + ] + + -- Built manually: 'protocolParamsToUtxoRpcPParams' requires 'ConwayEraPParams'. + -- The Shelley genesis holds a 'PParams ShelleyEra'; the era-generic + -- 'EraPParams' lenses read every field present in the proto message. + -- sppExtraEntropy, sppD and sppMinUTxOValue have no proto counterpart and are + -- dropped. + protocolParams :: Proto U5c.PParams + protocolParams = + defMessage + & U5c.minFeeCoefficient .~ inject (L.fromCompact (L.unCoinPerByte (pp ^. L.ppTxFeePerByteL))) + & U5c.minFeeConstant .~ inject (pp ^. L.ppTxFeeFixedL) + & U5c.maxBlockBodySize .~ fromIntegral (pp ^. L.ppMaxBBSizeL) + & U5c.maxTxSize .~ fromIntegral (pp ^. L.ppMaxTxSizeL) + & U5c.maxBlockHeaderSize .~ fromIntegral (pp ^. L.ppMaxBHSizeL) + & U5c.stakeKeyDeposit .~ inject (pp ^. L.ppKeyDepositL) + & U5c.poolDeposit .~ inject (pp ^. L.ppPoolDepositL) + & U5c.poolRetirementEpochBound .~ fromIntegral (L.unEpochInterval (pp ^. L.ppEMaxL)) + & U5c.desiredNumberOfPools .~ fromIntegral (pp ^. L.ppNOptL) + & U5c.poolInfluence .~ inject (L.unboundRational (pp ^. L.ppA0L)) + & U5c.monetaryExpansion .~ inject (L.unboundRational (pp ^. L.ppRhoL)) + & U5c.treasuryExpansion .~ inject (L.unboundRational (pp ^. L.ppTauL)) + & U5c.minPoolCost .~ inject (pp ^. L.ppMinPoolCostL) + & U5c.protocolVersion .~ inject (pp ^. L.ppProtocolVersionL) + where + pp = L.sgProtocolParams genesis + +-------------------------------------------------------------------------------- +-- Alonzo +-------------------------------------------------------------------------------- + +alonzoGenesisToProto :: L.AlonzoGenesis -> Proto U5c.Genesis -> Proto U5c.Genesis +alonzoGenesisToProto genesis message = + message + & U5c.lovelacePerUtxoWord .~ inject (L.unCoinPerWord (L.agCoinsPerUTxOWord genesis)) + & U5c.executionPrices + .~ ( defMessage + & U5c.steps .~ inject (L.unboundRational (L.prSteps prices)) + & U5c.memory .~ inject (L.unboundRational (L.prMem prices)) + ) + & U5c.maxTxExUnits .~ inject (L.agMaxTxExUnits genesis) + & U5c.maxBlockExUnits .~ inject (L.agMaxBlockExUnits genesis) + & U5c.maxValueSize .~ L.agMaxValSize genesis + & U5c.collateralPercentage .~ fromIntegral (L.agCollateralPercentage genesis) + & U5c.maxCollateralInputs .~ fromIntegral (L.agMaxCollateralInputs genesis) + -- Only PlutusV1 comes from the Alonzo genesis; PlutusV3 is set by Conway on + -- the shared accumulator, PlutusV2 and PlutusV4 never appear in genesis. + & U5c.costModels . U5c.maybe'plutusV1 + .~ Just (defMessage & U5c.values .~ L.getCostModelParams (L.agPlutusV1CostModel genesis)) + where + prices = L.agPrices genesis + +-------------------------------------------------------------------------------- +-- Conway +-------------------------------------------------------------------------------- + +conwayGenesisToProto :: L.ConwayGenesis -> Proto U5c.Genesis -> Proto U5c.Genesis +conwayGenesisToProto genesis message = + message + & U5c.committeeMinSize .~ fromIntegral (L.ucppCommitteeMinSize upgrade) + & U5c.committeeMaxTermLength + .~ fromIntegral (L.unEpochInterval (L.ucppCommitteeMaxTermLength upgrade)) + & U5c.govActionLifetime .~ fromIntegral (L.unEpochInterval (L.ucppGovActionLifetime upgrade)) + & U5c.drepActivity .~ fromIntegral (L.unEpochInterval (L.ucppDRepActivity upgrade)) + & U5c.govActionDeposit .~ inject (L.ucppGovActionDeposit upgrade) + & U5c.drepDeposit .~ inject (L.ucppDRepDeposit upgrade) + & U5c.minFeeRefScriptCostPerByte + .~ inject (L.unboundRational (L.ucppMinFeeRefScriptCostPerByte upgrade)) + & U5c.poolVotingThresholds .~ poolVotingThresholds (L.ucppPoolVotingThresholds upgrade) + & U5c.drepVotingThresholds .~ drepVotingThresholds (L.ucppDRepVotingThresholds upgrade) + & U5c.constitution .~ constitution + & U5c.committee .~ committee + & U5c.costModels . U5c.maybe'plutusV3 + .~ Just (defMessage & U5c.values .~ L.getCostModelParams (L.ucppPlutusV3CostModel upgrade)) + where + upgrade = L.cgUpgradePParams genesis + + poolVotingThresholds :: L.PoolVotingThresholds -> Proto U5c.PoolVotingThresholds + poolVotingThresholds thresholds = + defMessage + & U5c.motionNoConfidence .~ inject (L.unboundRational (thresholds ^. L.pvtMotionNoConfidenceL)) + & U5c.committeeNormal .~ inject (L.unboundRational (thresholds ^. L.pvtCommitteeNormalL)) + & U5c.committeeNoConfidence + .~ inject (L.unboundRational (thresholds ^. L.pvtCommitteeNoConfidenceL)) + & U5c.hardForkInitiation .~ inject (L.unboundRational (thresholds ^. L.pvtHardForkInitiationL)) + & U5c.ppSecurityGroup .~ inject (L.unboundRational (thresholds ^. L.pvtPPSecurityGroupL)) + + drepVotingThresholds :: L.DRepVotingThresholds -> Proto U5c.DRepVotingThresholds + drepVotingThresholds thresholds = + defMessage + & U5c.motionNoConfidence .~ inject (L.unboundRational (thresholds ^. L.dvtMotionNoConfidenceL)) + & U5c.committeeNormal .~ inject (L.unboundRational (thresholds ^. L.dvtCommitteeNormalL)) + & U5c.committeeNoConfidence + .~ inject (L.unboundRational (thresholds ^. L.dvtCommitteeNoConfidenceL)) + & U5c.updateToConstitution + .~ inject (L.unboundRational (thresholds ^. L.dvtUpdateToConstitutionL)) + & U5c.hardForkInitiation .~ inject (L.unboundRational (thresholds ^. L.dvtHardForkInitiationL)) + & U5c.ppNetworkGroup .~ inject (L.unboundRational (thresholds ^. L.dvtPPNetworkGroupL)) + & U5c.ppEconomicGroup .~ inject (L.unboundRational (thresholds ^. L.dvtPPEconomicGroupL)) + & U5c.ppTechnicalGroup .~ inject (L.unboundRational (thresholds ^. L.dvtPPTechnicalGroupL)) + & U5c.ppGovGroup .~ inject (L.unboundRational (thresholds ^. L.dvtPPGovGroupL)) + & U5c.treasuryWithdrawal .~ inject (L.unboundRational (thresholds ^. L.dvtTreasuryWithdrawalL)) + + constitution :: Proto U5c.Constitution + constitution = + defMessage + & U5c.anchor .~ anchorToUtxoRpcAnchor (L.constitutionAnchor c) + & U5c.hash .~ L.strictMaybe mempty scriptHashToBytes (L.constitutionGuardrailsScriptHash c) + where + c = L.cgConstitution genesis + + committee :: Proto U5c.Committee + committee = + defMessage + & U5c.threshold .~ inject (L.unboundRational (L.committeeThreshold c)) + & U5c.members + .~ Map.fromList + -- The proto committee member key does not distinguish a key hash from + -- a script hash; both render as bare hex. + [ (credentialHexText credential, fromIntegral (L.unEpochNo epochNo)) + | (credential, epochNo) <- Map.toList (L.committeeMembers c) + ] + where + c = L.cgCommittee genesis + +-------------------------------------------------------------------------------- +-- Helpers +-------------------------------------------------------------------------------- + +-- | Lowercase base16 rendering of raw bytes, matching genesis JSON hash keys. +hexText :: ByteString -> Text +hexText = Text.decodeUtf8 . Base16.encode + +credentialHexText :: L.Credential kr -> Text +credentialHexText = \case + L.KeyHashObj keyHash -> hexText (keyHashToBytes keyHash) + L.ScriptHashObj scriptHash -> hexText (scriptHashToBytes scriptHash) diff --git a/cardano-rpc/src/Cardano/Rpc/Server/NodeKernelAccess.hs b/cardano-rpc/src/Cardano/Rpc/Server/NodeKernelAccess.hs index 55c3edf143..c83484de2f 100644 --- a/cardano-rpc/src/Cardano/Rpc/Server/NodeKernelAccess.hs +++ b/cardano-rpc/src/Cardano/Rpc/Server/NodeKernelAccess.hs @@ -7,6 +7,7 @@ module Cardano.Rpc.Server.NodeKernelAccess ( NodeKernelAccess (..) + , GenesisBundle (..) , mkNodeKernelAccess , fetchBlock , grabNodeKernelAccess @@ -19,6 +20,7 @@ where import Cardano.Api import Cardano.Api.Consensus qualified as Consensus import Cardano.Rpc.Server.Internal.Monad (MonadRpc, grab) +import Cardano.Rpc.Server.Internal.Tracing import Cardano.Rpc.Server.NodeKernelAccess.Type import RIO (MonadUnliftIO, atomically, bracket, throwIO, withRunInIO) @@ -34,8 +36,13 @@ import Network.GRPC.Spec -- Returns 'Nothing' and traces the block type for non-Cardano block types. mkNodeKernelAccess :: Monad m - => Tracer m Text - -- ^ Tracer for unsupported block type warnings + => Tracer m TraceRpc + -- ^ Tracer for RPC events + -> GenesisHashShelley + -- ^ Boot-time Shelley genesis hash + -> Consensus.ProtocolInfoArgs blk + -- ^ Protocol info arguments (carrying the parsed genesis and transition + -- config) -> Consensus.BlockType blk -- ^ Block type witness -> Consensus.TopLevelConfig blk @@ -44,14 +51,15 @@ mkNodeKernelAccess -> Consensus.NodeKernel IO addrNTN addrNTC blk -- ^ Consensus node kernel -> m (Maybe NodeKernelAccess) -mkNodeKernelAccess tracer blockType topLevelConfig kernel = case blockType of +mkNodeKernelAccess tracer shelleyGenesisHash protocolInfoArgs blockType topLevelConfig kernel = case blockType of Consensus.CardanoBlockType -> - pure $ Just NodeKernelAccess{chainDb, systemStart, readEraHistory, securityParam} + pure $ Just NodeKernelAccess{chainDb, systemStart, readEraHistory, securityParam, genesisConfig} where chainDb = Consensus.getChainDB kernel ledgerConfig = Consensus.configLedger topLevelConfig systemStart = Consensus.nodeSystemStart topLevelConfig securityParam = Consensus.configSecurityParam topLevelConfig + genesisConfig = readGenesisBundle shelleyGenesisHash protocolInfoArgs -- Read the current ledger state (cheap STM TVar read) and recompute -- the era summary on every call - O(number_of_eras). -- This is the same approach consensus uses for GetInterpreter queries @@ -64,9 +72,31 @@ mkNodeKernelAccess tracer blockType topLevelConfig kernel = case blockType of Consensus.hardForkSummary ledgerConfig (Consensus.ledgerState extLedger) _ -> do -- unsupported block type - traceWith tracer $ pack (show blockType) + traceWith tracer . inject . TraceRpcUnsupportedBlockType . pack $ show blockType pure Nothing +-- | Gather the network's genesis configuration from the node's boot-time +-- 'Consensus.ProtocolInfoArgs'. +-- +-- No hard-fork navigation is needed: the transition config already carries +-- the full per-era genesis and translation contexts parsed at node startup, +-- and 'Consensus.CardanoProtocolParams' keeps its own copy of the Byron +-- genesis. +-- The Shelley genesis hash is passed in separately, because +-- 'Consensus.ProtocolInfoArgs' does not carry it. +-- This function only runs for the Cardano protocol, so the hash is always +-- available here. +readGenesisBundle + :: GenesisHashShelley + -> Consensus.ProtocolInfoArgs (Consensus.CardanoBlock Consensus.StandardCrypto) + -> GenesisBundle +readGenesisBundle shelleyGenesisHash (Consensus.ProtocolInfoArgsCardano cardanoProtocolParams) = + GenesisBundle + { byronConfig = Consensus.byronGenesis $ Consensus.byronProtocolParams cardanoProtocolParams + , shelleyGenesisHash + , transitionConfig = Consensus.cardanoLedgerTransitionConfig cardanoProtocolParams + } + -- | Grab the current 'NodeKernelAccess' from the environment, or throw -- gRPC UNAVAILABLE if the node kernel has not yet initialised. grabNodeKernelAccess diff --git a/cardano-rpc/src/Cardano/Rpc/Server/NodeKernelAccess/Type.hs b/cardano-rpc/src/Cardano/Rpc/Server/NodeKernelAccess/Type.hs index 6252cde91d..7a4f6299c8 100644 --- a/cardano-rpc/src/Cardano/Rpc/Server/NodeKernelAccess/Type.hs +++ b/cardano-rpc/src/Cardano/Rpc/Server/NodeKernelAccess/Type.hs @@ -3,12 +3,17 @@ module Cardano.Rpc.Server.NodeKernelAccess.Type ( NodeKernelAccess (..) + , GenesisBundle (..) ) where -import Cardano.Api (EraHistory, SystemStart) +import Cardano.Api (EraHistory, GenesisHashShelley, SystemStart) import Cardano.Api.Consensus qualified as Consensus +import Cardano.Chain.Genesis qualified as Byron (Config) +import Cardano.Ledger.Api.Era qualified as L (LatestKnownEra) +import Cardano.Ledger.Api.Transition qualified as L (TransitionConfig) + import Control.Monad.IO.Class (MonadIO) -- | In-process access to the node kernel. @@ -28,4 +33,38 @@ data NodeKernelAccess = NodeKernelAccess , securityParam :: Consensus.SecurityParam -- ^ The protocol security parameter /k/: consensus never rolls back more -- than /k/ blocks. + , genesisConfig :: GenesisBundle + -- ^ The network's genesis configuration. + -- Genesis data never changes after startup, so it is read once and stored + -- as a pure value. + } + +-- | The per-era genesis configuration of the network the node is running on. +-- +-- Gathered once, when the node kernel hook fires. The Byron genesis and the +-- Shelley-onwards transition config are both read straight off +-- 'Consensus.CardanoProtocolParams', part of cardano-node's boot-time +-- 'Consensus.ProtocolInfoArgs'. No hard-fork navigation is needed. +-- +-- The Shelley genesis hash is the exception: 'Consensus.ProtocolInfoArgs' +-- does not carry it, so it is threaded in separately from cardano-node's own +-- boot-time genesis parsing (see +-- 'Cardano.Rpc.Server.NodeKernelAccess.mkNodeKernelAccess'). +data GenesisBundle = GenesisBundle + { byronConfig :: !Byron.Config + -- ^ The Byron genesis configuration, which bundles the genesis data with + -- the hash the Byron ledger computed when it parsed the file. + -- Read directly off 'Consensus.CardanoProtocolParams' via its + -- 'Consensus.byronProtocolParams' field. + , shelleyGenesisHash :: !GenesisHashShelley + -- ^ Blake2b-256 hash of the raw Shelley genesis file bytes. + -- Computed once by cardano-node when it parses the file at boot time, and + -- threaded in from there. + , transitionConfig :: !(L.TransitionConfig L.LatestKnownEra) + -- ^ The Shelley-onwards genesis configuration, in the same representation + -- 'Cardano.Api.LedgerState.GenesisConfig' uses. + -- Taken straight from cardano-node's boot-time 'Consensus.ProtocolInfoArgs'. + -- It retains the full parsed Shelley genesis, including @sgInitialFunds@ + -- and @sgStaking@; consensus keeps only a compacted copy with those fields + -- erased. } diff --git a/experimental-api-migration-guide.md b/experimental-api-migration-guide.md new file mode 100644 index 0000000000..758e16bfbe --- /dev/null +++ b/experimental-api-migration-guide.md @@ -0,0 +1,411 @@ +# Migrating from legacy cardano-api to the experimental API + +This guide explains how to migrate code from the legacy `Cardano.Api` transaction construction API to the experimental API (`Cardano.Api.Experimental`). + +## Why migrate? + +The legacy API wraps ledger types in cardano-api-specific wrappers (`TxFee era`, `TxInsCollateral era`, `TxMetadataInEra era`, etc.) that add indirection without value. +The experimental API uses ledger types directly, supports only current and upcoming eras (Conway, Dijkstra), and is the intended replacement per ADR-004 and ADR-009. + +Symbols deprecated in favour of the experimental API include `createTransactionBody`, `signShelleyTransaction`, `getTxBody`, `TxBody`, `ShelleyTxBody`, `defaultTxBodyContent` (old), and `BalancedTxBody`. + +## Import convention + +```haskell +import Cardano.Api -- stable API (types, eras, addresses, etc.) +import Cardano.Api.Experimental -- era types, makeUnsignedTx, signTx, etc. +import qualified Cardano.Api.Experimental.Tx as Exp -- TxBodyContent, setters, TxOut, fee functions +``` + +The old and new APIs export identically-named symbols (`defaultTxBodyContent`, `setTxIns`, `setTxOuts`, etc.). +When both are in scope, qualify the experimental ones via `Exp.`. + +## Era types + +| Legacy | Experimental | Notes | +|--------|-------------|-------| +| `ShelleyBasedEra era` | `Era era` | GADT with only `ConwayEra` and `DijkstraEra` constructors | +| `CardanoEra era` | (none) | Not needed; experimental API only covers supported eras | +| `IsShelleyBasedEra era` | `IsEra era` | `IsEra` only provides `useEra :: Era era` | +| `shelleyBasedEraConstraints sbe` | `obtainCommonConstraints era` | Brings `EraCommonConstraints era` into scope | + +### Bridging eras + +Convert `ShelleyBasedEra era` to `Era era` using `sbeToEra`: + +```haskell +sbeToEra :: MonadError (DeprecatedEra era) m => ShelleyBasedEra era -> m (Era era) +``` + +Pre-Conway eras return `Left (DeprecatedEra sbe)` when `m ~ Either (DeprecatedEra era)`. +Pattern-match on the result and handle the error for unsupported eras. + +## TxBodyContent + +### Old (deprecated) + +```haskell +data TxBodyContent build era = TxBodyContent + { txIns :: [(TxIn, BuildTxWith build (Witness WitCtxTxIn era))] + , txInsCollateral :: TxInsCollateral era + , txOuts :: [TxOut CtxTx era] + , txFee :: TxFee era + , txValidityUpperBound :: TxValidityUpperBound era + , txMetadata :: TxMetadataInEra era + , txProtocolParams :: BuildTxWith build (Maybe (LedgerProtocolParameters era)) + , ... + } + +defaultTxBodyContent :: ShelleyBasedEra era -> TxBodyContent BuildTx era +``` + +### New (experimental) + +```haskell +data TxBodyContent era = TxBodyContent + { txIns :: [(TxIn, AnyWitness era)] + , txInsCollateral :: [TxIn] + , txOuts :: [TxOut era] -- wraps L.TxOut era + , txFee :: L.Coin + , txValidityUpperBound :: Maybe L.SlotNo + , txMetadata :: TxMetadata + , txProtocolParams :: Maybe (L.PParams era) + , ... + } + +defaultTxBodyContent :: TxBodyContent era -- no arguments needed +``` + +### Field-by-field mapping + +| Old type | New type | Conversion | +|----------|----------|------------| +| `(TxIn, BuildTxWith BuildTx (Witness WitCtxTxIn era))` | `(TxIn, AnyWitness era)` | Key witnesses: use `AnyKeyWitnessPlaceholder`; script witnesses: see witness section | +| `TxInsCollateral era` | `[TxIn]` | Extract the list of `TxIn`; `TxInsCollateralNone` becomes `[]` | +| `TxOut CtxTx era` | `Exp.TxOut era` | `Exp.TxOut (toShelleyTxOutAny sbe oldTxOut)` | +| `TxFee era` | `L.Coin` | `TxFeeExplicit _ coin` -> `coin` | +| `TxValidityUpperBound era` | `Maybe L.SlotNo` | `TxValidityUpperBound _ mSlot` -> `mSlot`; `Nothing` = no upper bound | +| `TxMetadataInEra era` | `TxMetadata` | `TxMetadataNone` -> `mempty`; `TxMetadataInEra _ m` -> `m` | +| `BuildTxWith BuildTx (Maybe (LedgerProtocolParameters era))` | `Maybe (L.PParams era)` | `unLedgerProtocolParameters` to unwrap; `setTxProtocolParams` wraps with `Just` | +| `TxInsCollateral era` | `[TxIn]` | Drop the eon witness | + +### Using setters + +```haskell +-- Old +defaultTxBodyContent sbe + & setTxIns [(txIn, BuildTxWith $ KeyWitness KeyWitnessForSpending)] + & setTxOuts [txOut] + & setTxFee (TxFeeExplicit sbe coin) + & setTxInsCollateral (TxInsCollateral eon [collateralTxIn]) + +-- New +Exp.defaultTxBodyContent + & Exp.setTxIns [(txIn, AnyKeyWitnessPlaceholder)] + & Exp.setTxOuts [Exp.TxOut $ toShelleyTxOutAny sbe txOut] + & Exp.setTxFee coin + & Exp.setTxInsCollateral [collateralTxIn] +``` + +## Transaction construction + +### Old flow (deprecated) + +```haskell +txBody <- createTransactionBody sbe txBodyContent +let tx = signShelleyTransaction sbe txBody [WitnessPaymentKey signingKey] +``` + +Or manual construction: +```haskell +let ledgerTxBody = mkCommonTxBody sbe txInputs txOuts fee ... + rawBody = ledgerTxBody ^. txBodyL + unsignedLedgerTx = Ledger.mkBasicTx rawBody + txHash = Ledger.extractHash $ Ledger.hashAnnotated rawBody + witVKey = WitVKey (getShelleyKeyWitnessVerificationKey sk) (makeShelleySignature txHash sk) + signedLedgerTx = unsignedLedgerTx & Ledger.witsTxL .~ ... + tx = ShelleyTx sbe signedLedgerTx +``` + +### New flow (experimental) + +```haskell +case sbeToEra sbe of + Left deprecated -> Left $ "unsupported era: " ++ show deprecated + Right era -> obtainCommonConstraints era $ do + let txBodyContent = Exp.defaultTxBodyContent + & Exp.setTxIns [(txIn, AnyKeyWitnessPlaceholder)] + & Exp.setTxOuts [Exp.TxOut $ toShelleyTxOutAny sbe txOut] + & Exp.setTxFee fee + unsignedTx <- first show $ makeUnsignedTx era txBodyContent + let witVKey = makeKeyWitness era unsignedTx (WitnessPaymentKey signingKey) + case signTx era [] [witVKey] unsignedTx of + SignedTx signedLedgerTx -> Right $ ShelleyTx sbe signedLedgerTx +``` + +Use monadic `Either` (via `first` from `Data.Bifunctor`) to flatten the `makeUnsignedTx` error case instead of nesting `case` expressions. +`signTx` has a single constructor `SignedTx` so it always succeeds - a simple `case` suffices. + +### Key API functions + +```haskell +makeUnsignedTx :: Era era -> TxBodyContent (LedgerEra era) -> Either MakeUnsignedTxError (UnsignedTx (LedgerEra era)) +makeKeyWitness :: HasCallStack => Era era -> UnsignedTx (LedgerEra era) -> ShelleyWitnessSigningKey -> L.WitVKey L.Witness +signTx :: Era era -> [L.BootstrapWitness] -> [L.WitVKey L.Witness] -> UnsignedTx (LedgerEra era) -> SignedTx era +``` + +## Converting back to `Tx era` + +The rest of the pipeline may still consume `Tx era` (the old type). +Convert from `SignedTx era` back to `Tx era` by unwrapping: + +```haskell +case signTx era [] witVKeys unsignedTx of + SignedTx signedLedgerTx -> ShelleyTx sbe signedLedgerTx +``` + +Use `case` rather than a let-binding pattern for `SignedTx` to help GHC unify `LedgerEra era` with `ShelleyLedgerEra era` under `obtainCommonConstraints`. + +## Witness types + +| Old | New | Notes | +|-----|-----|-------| +| `KeyWitness KeyWitnessForSpending` | `AnyKeyWitnessPlaceholder` | Placeholder during body construction; actual key witnesses added at signing | +| `ScriptWitness ScriptWitnessForSpending (PlutusScriptWitness ...)` | `AnyPlutusScriptWitness (...)` | See ADR-010 | +| `Witness WitCtxTxIn era` | `AnyWitness era` | Unified witness type | + +For incremental migration, use `legacyWitnessConversion` (exported from `Cardano.Api.Experimental`) to convert old-style witnesses. + +## Plutus script witnesses + +The experimental API replaces `ScriptInAnyLang` with `AnyPlutusScript era`, which wraps `PlutusScriptInEra lang era` existentially. +`PlutusScriptInEra` uses `PlutusRunnable` internally - scripts are validated at deserialisation time against the era's protocol version, not deferred to submission. + +### Decoding a Plutus script from `ScriptInAnyLang` + +When migrating code that receives a `ScriptInAnyLang` (e.g. from file deserialisation), decode it into `AnyPlutusScript` using `decodePlutusRunnable` from `Cardano.Ledger.Plutus.Language`: + +```haskell +import qualified Cardano.Ledger.Plutus.Language as L +import Cardano.Api.Experimental (eraProtVerHigh, toPlutusSLanguage, obtainCommonConstraints) +import Cardano.Api.Experimental.Plutus (AnyPlutusScript (..), plutusScriptInEraSLanguage) +import qualified Cardano.Api.Experimental as Exp (PlutusScriptInEra (..)) + +anyPlutusScript <- obtainCommonConstraints era $ + case script of + ScriptInAnyLang _lang (PlutusScript version (PlutusScriptSerialised sbs)) -> do + let slang = toPlutusSLanguage version + decode :: forall l. (L.PlutusLanguage l, Typeable l) + => L.SLanguage l -> IO (AnyPlutusScript (ShelleyLedgerEra ConwayEra)) + decode _ = case L.decodePlutusRunnable @l (eraProtVerHigh era) (L.Plutus (L.PlutusBinary sbs)) of + Left err -> throwIO $ userError $ "script decode failed: " ++ show err + Right runnable -> pure $ AnyPlutusScript (Exp.PlutusScriptInEra runnable) + obtainLangConstraints slang $ decode slang + _ -> throwIO $ userError "expected a Plutus script" +``` + +Key points: +- `toPlutusSLanguage` converts `PlutusScriptVersion` to `L.SLanguage lang`. +- `obtainLangConstraints` brings `PlutusLanguage lang` and `Typeable lang` into scope from `SLanguage lang`. + Exported from `Cardano.Api.Experimental`. +- The local `decode` helper with `forall l` + `@l` type application is needed to connect the existential language type variable through `decodePlutusRunnable` to the `AnyPlutusScript` constructor. + Without it, GHC cannot resolve the ambiguous `lang` type variable. + +### Building a Plutus script witness + +The old API used `PlutusScriptWitness` with `ScriptDatumForTxIn`: + +```haskell +-- Old +ScriptWitness ScriptWitnessForSpending + $ PlutusScriptWitness scriptLang version (PScript script') + (ScriptDatumForTxIn $ Just datum) redeemer budget +``` + +The new API uses `Exp.PlutusScriptWitness` with `PlutusScriptDatum`, then wraps it in `AnyPlutusScriptWitness`: + +```haskell +-- New +let slang = plutusScriptInEraSLanguage ps + datum = SpendingScriptDatum dummyDatum -- V1/V2: bare datum; V3/V4: wrap with Just + witness = Exp.PlutusScriptWitness slang (Exp.PScript ps) datum redeemer budget +in AnyPlutusScriptWitness + (AnyPlutusSpendingScriptWitness (createPlutusSpendingScriptWitness slang witness)) +``` + +### `PlutusScriptDatum` and CIP-69 + +The `PlutusScriptDatumF` type family changes the datum type depending on the Plutus version: +- V1/V2: `SpendingScriptDatum :: HashableScriptData -> PlutusScriptDatum lang SpendingScript` +- V3/V4 (CIP-69): `SpendingScriptDatum :: Maybe HashableScriptData -> PlutusScriptDatum lang SpendingScript` + +Construct `SpendingScriptDatum` directly, adjusting for the Plutus version: + +```haskell +-- V1/V2 +SpendingScriptDatum datum + +-- V3/V4 (CIP-69) +SpendingScriptDatum (Just datum) +``` + +The `PlutusScriptDatumF` type family resolves `SpendingScriptDatum` differently per version, so GHC enforces the correct wrapping at compile time. + +## Protocol parameters + +### Unwrapping `LedgerProtocolParameters` + +The experimental API uses `L.PParams era` directly, not wrapped in `BuildTxWith`: + +```haskell +-- Old +& setTxProtocolParams (BuildTxWith (Just ledgerParameters)) + +-- New +& Exp.setTxProtocolParams (unLedgerProtocolParameters ledgerParameters) +``` + +Or destructure in the function pattern: + +```haskell +genTx era (LedgerProtocolParameters pparams) ... = + ... + & Exp.setTxProtocolParams pparams +``` + +### When protocol parameters are needed + +Protocol parameters are required when the transaction includes Plutus scripts (for the script integrity hash). +For key-only transactions, `setTxProtocolParams` is functionally inert but should still be wired in for correctness and future-proofing. + +## Fee estimation + +| Old | New | +|-----|-----| +| `evaluateTransactionFee sbe pp txBody keyWitCount byronWitCount refScriptSize` | `Exp.evaluateTransactionFee pp unsignedTx keyWitCount byronWitCount refScriptSize` | + +The experimental version takes `UnsignedTx` directly (not `TxBody`), so `getTxBody` is no longer needed. + +For full auto-balancing, use `makeTransactionBodyAutoBalance` from `Cardano.Api.Experimental.Tx`. + +## Bridge functions for incremental migration + +| Function | Purpose | +|----------|---------| +| `sbeToEra :: MonadError (DeprecatedEra era) m => ShelleyBasedEra era -> m (Era era)` | Convert era witness | +| `convertTxBodyToUnsignedTx :: ShelleyBasedEra era -> TxBody era -> UnsignedTx (LedgerEra era)` | Convert old TxBody to UnsignedTx | +| `legacyWitnessConversion` | Convert old-style witnesses to new `AnyWitness` | +| `toShelleyTxOutAny :: ShelleyBasedEra era -> TxOut ctx era -> L.TxOut (ShelleyLedgerEra era)` | Convert old TxOut to ledger TxOut (then wrap in `Exp.TxOut`) | + +## Deprecated symbols reference + +| Symbol | Replacement | +|--------|-------------| +| `createTransactionBody` | `makeUnsignedTx` from `Cardano.Api.Experimental` | +| `signShelleyTransaction` | `makeKeyWitness` + `signTx` | +| `getTxBody` | Use `UnsignedTx` directly | +| `TxBody` / `ShelleyTxBody` | `UnsignedTx` from `Cardano.Api.Experimental` | +| `defaultTxBodyContent sbe` (old) | `Exp.defaultTxBodyContent` (no args) | +| `BalancedTxBody` | `makeTransactionBodyAutoBalance` returns `(UnsignedTx, TxBodyContent)` | +| `getTxBodyContent` | Access `TxBodyContent` fields directly | +| `ProtocolParameters` (Cardano.Api) | `L.PParams era` from ledger | + +## Error handling + +### `makeUnsignedTx` errors + +Use `Either` to handle `makeUnsignedTx` failures instead of `fromRight (error ...)`. +Always include the error in the message: + +```haskell +-- Bad: loses the error +unsignedTx = fromRight (error "failed to create tx") $ makeUnsignedTx era txBodyContent + +-- Good: preserves the error +unsignedTx = either (\err -> error $ "failed to create tx: " ++ show err) id $ makeUnsignedTx era txBodyContent + +-- Better: in IO, use throwIO +unsignedTx <- either (\err -> throwIO $ userError $ "failed: " ++ show err) pure + $ makeUnsignedTx era txBodyContent +``` + +### `evaluate` vs `catch` pattern + +The old API used `evaluate` to force a pure expression and catch exceptions: + +```haskell +-- Old +evaluate $ summary { projectedTxSize = Just $ txSizeInBytes dummyTx } + `catch` \(SomeException e) -> ... +``` + +The new API is monadic (because `makeUnsignedTx` returns `Either`), so use a `do` block wrapped in `catch`: + +```haskell +-- New +obtainCommonConstraints era (do + unsignedTx <- either (\err -> throwIO ...) pure $ makeUnsignedTx era txBodyContent + ... + pure summary { projectedTxSize = ... } + ) `catch` \(SomeException e) -> ... +``` + +## Gotchas + +- **`makeUnsignedTx DijkstraEra` is `error "TODO Dijkstra"`.** + Only ConwayEra is implemented in the experimental API today. + Your code will compile for Dijkstra but crash at runtime until cardano-api fills in the implementation. + +- **`obtainCommonConstraints` is needed for `LedgerEra era ~ ShelleyLedgerEra era`.** + Without it, GHC cannot unify the type families, and `ShelleyTx sbe signedLedgerTx` will not type-check. + +- **Use `case` not let-bindings for `SignedTx` pattern matching.** + A let-binding `let SignedTx x = signTx ...` creates a rigid type variable that GHC cannot unify. + Use `case signTx era [] wits unsignedTx of SignedTx ledgerTx -> ...` instead. + +- **`MakeUnsignedTxMissingProtocolParams` error.** + If your transaction includes Plutus scripts, you must set `Exp.setTxProtocolParams pparams`. + Without protocol parameters, `makeUnsignedTx` cannot compute the script integrity hash. + +- **Always use `useEra @era` with a type application, not bare `useEra`.** + Without `@era`, GHC 9.6 infers an ambiguous type variable `era0` disconnected from the `era` in the type signature. + GHC 9.12 is more lenient and may accept bare `useEra`, but CI runs GHC 9.6. + The function must have `ScopedTypeVariables` and a `forall era.` binding in the type signature. + +- **Ambiguous type variables with existential `AnyPlutusScript` decoding.** + `AnyPlutusScript` hides the `lang` type variable. + When calling `decodePlutusRunnable @l`, the type must be connected via a local helper with explicit `forall l` and `@l` type application. + See the decoding example above. + +- **Name collisions between old and new APIs.** + Both export `defaultTxBodyContent`, `setTxIns`, `setTxOuts`, `setTxFee`, `TxOut`, etc. + Import the experimental module qualified (`as Exp`) and hide conflicting names from `Cardano.Api` if needed. + +- **`eraProtVerHigh` name collision.** + Both `Cardano.Api` and `Cardano.Api.Experimental` export `eraProtVerHigh`. + Hide it from the old API: `import Cardano.Api hiding (eraProtVerHigh)`. + +- **`TxMetadata` vs `TxMetadataInEra`.** + The new API uses `TxMetadata` directly. + Convert: `TxMetadataNone` -> `mempty`, `TxMetadataInEra _ m` -> `m`. + +- **`Exp.TxOut` wraps a ledger `L.TxOut era`, not the old `TxOut CtxTx era`.** + Use `toShelleyTxOutAny sbe` to convert from old to ledger, then wrap in `Exp.TxOut`. + +- **Use `Exp.TxOut (LedgerEra era)` in function signatures, not `Exp.TxOut era`.** + `makeUnsignedTx` expects `TxBodyContent (LedgerEra era)`, so `Exp.setTxOuts` must receive `[Exp.TxOut (LedgerEra era)]`. + Passing `[Exp.TxOut era]` forces `TxBodyContent era` which does not unify with `TxBodyContent (LedgerEra era)` - GHC cannot deduce `era ~ LedgerEra era` even inside `obtainCommonConstraints`. + The constraint `ShelleyLedgerEra era ~ LedgerEra era` (from `EraCommonConstraints`) makes `toShelleyTxOutAny` output unify correctly with `LedgerEra era`. + +- **`toShelleyTxOutAny` needs `IsShelleyBasedEra` - scope it inside `obtainCommonConstraints`.** + `IsEra` does not imply `IsShelleyBasedEra`, so `shelleyBasedEra` and `toShelleyTxOutAny` are not available at the function signature level. + Place the conversion inline in the body (inside `obtainCommonConstraints`), not in a `where` clause outside it. + +- **Old `PlutusScript lang` to `PlutusScriptInEra lang era` conversion.** + Use `deserialisePlutusScriptInEra sLang (serialiseToCBOR oldScript)`. + This round-trips through CBOR but is the only stable conversion path when `PlutusRunnable` is not directly accessible from the old `PlutusScript`. + +- **`cardano-ledger-core` dependency may be avoidable.** + `obtainLangConstraints` and `decodeAnyPlutusScript` are now exported from `Cardano.Api.Experimental`. + `decodeAnyPlutusScript :: L.Era era => ByteString -> AnyPlutusScriptLanguage -> Either CBOR.DecoderError (AnyPlutusScript era)` provides a simpler alternative to the manual `decodePlutusRunnable` approach shown above. + Direct use of `Cardano.Ledger.Plutus.Language` is only needed for lower-level control.