[runtime, evm]: Verify BEEFY finality with an aggregate public key proof and a header digest

[dharjeezy]

Hyperbridge computes a Poseidon2 commitment over the relay chain's next BEEFY BLS keys and publishes it in an APKC header digest, which a client reads straight out of a header it has already authenticated through the parachain heads root. Signers are then proven with a gnark APK proof and a single pairing instead of a merkle path per signer, so nothing in the calldata or the verification grows with the size of the validator set. The earlier per signer merkle client is removed along with its fixtures and generated bindings, so the APK route is the only BLS path here.

Why the design changed

An earlier revision of this branch committed the BLS keys as one extra leaf of the BEEFY authority set tree. That was rejected on review, and it also kept the per signer cost that motivated moving to BLS in the first place, since every signer still travelled with a merkle path.

Committing the set in a Hyperbridge header digest asks nothing of the relay chain and nothing of other bridges. A client that verifies a Hyperbridge header at all already has everything it needs to trust a digest attached to it, so the commitment arrives with no extra proof and no extra gas.

How the commitment reaches a client

pallet-beefy-apk-digest reads Beefy::NextAuthorities out of the relay state proof that every parachain block already carries, takes the G1 half of each paired key, and absorbs 64 slots per block into a running Poseidon2 state. A full 1024 slot set is roughly 420ms of wasm, which is why it is chunked rather than done in one block. On the block the last slot lands it deposits DigestItem::Consensus(*b"APKC", (set_id, commitment)).

It commits the next set rather than the current one, and that is what makes the scheme work. A client verifying under set N learns the commitment for set N+1 and is ready before the rotation happens. Committing the current set would be circular, since you would need set N's commitment to verify the header carrying set N's commitment. A set changing part way through a chunked absorption restarts it, so a commitment can never be a mixture of two sets.

BlsApkBeefy picks the commitment up in verifyParachainHeaderProof, once those headers are proven, and stores it against the matching authority set. An update for a set whose commitment was never learned reverts with MissingApkCommitment rather than checking a proof against zero.

Following a live chain

Verifying one update is not the same as following a chain, and the ordering turns out to be forced rather than a matter of taste. The justification that rotates into a set is signed by that same set, so a client can never rotate into a set whose commitment it does not already hold. The commitment only arrives through a digest in a parachain header, and the header a rotation proof carries sits on the session boundary, where the relay has only just queued the incoming set, so it still names the set the client already knows.

A session therefore needs two proofs. The prover first proves finality inside the session it still trusts, which hands the client a header naming the incoming set, and the rotation goes through on a later tick. Choosing that block by position does not work: the digest naming a set only appears once the relay has queued it and the parachain has published it, so the prover looks for the earliest justification whose header actually carries the digest it needs. Taking the earliest rather than the last also leaves the rest of the session provable, which is what a proof for new messages needs.

The pallet had to learn the same thing. It used to turn away any proof that neither rotated nor finalized a new parachain head, on the grounds that it achieved nothing, which is exactly what a proof carrying a commitment looks like from the outside. Learning a commitment now counts as work alongside rotating.

Proving

The circuit takes about four minutes to compile and generate its keys, and a prover that starts fresh for each proof pays that every time, which is most of the wall clock. apk-beefy can now keep one prover process alive and speak to it over its stdin and stdout, so the setup happens once at startup. A proof against a live chain takes 1m49s that way against roughly seven minutes before. The one shot behaviour is still available for a prover with no serve mode.

What is here

1. pallet-beefy-apk-digest, with the rotation decision in a pure next_progress so the restart case is testable without a mock chain.
2. apk-commitment, a no_std Poseidon2 commitment matching the circuit's vectors, absorbable in chunks.
3. BlsApkBeefy.sol, a complete IConsensusV2 client: authority selection, threshold, APK proof, MMR leaf inclusion, parachain headers, digest harvesting and stale proof no-op.
4. beefy-verifier's apk module, the same client for a substrate runtime, wired through pallet-beefy-consensus-proofs with the circuit's verifying key in storage rather than baked into the runtime.
5. BlsHashToCurve.sol and BlsAggregate.sol, checked against vectors from the Rust code path the chain actually signs with.
6. Prover and relayer support for the apk variant, and PairedAuthority in beefy-verifier-primitives so the pallet and the prover read the 177 byte paired key the same way.

Validation

surface	result
Solidity	15 tests, live relay proof at 617k gas inside verify() with a real parachain header
Rust	verifier, primitives, prover and both pallets green, including a rotation interrupting an absorption and a set republished under a new id
weights	verify_apk benchmarked at 41.21ms, charged separately from the rest of a submission
live chain	all three proof types settled against a westend local relay signing with paired keys: rotation, the proof that teaches a commitment, and a proof accepted for its messages alone
cross check	the circuit's commitment, apk-commitment's, and the value the pallet published on chain all agree, and the fixture assembly asserts it

Forward chaining is the part worth calling out, since it is what the whole design rests on. Seeded with a set it had never seen, the client learned that set's commitment from a digest inside a parachain header it had verified for itself, and then accepted a rotation into it that it would have refused a step earlier.

The messaging case is pinned by construction rather than by reading the pallet's internals. tesseract/consensus/beefy/tests/apk_messaging.rs picks a justification from the set the client already trusts, which rules out rotation, and checks afterwards that no commitment changed, which rules out teaching. A proof accepted under those conditions can only have been accepted for the messages it carried.

The APK fixture is captured from a running relay rather than synthesised. Building it is a three step pipeline because the SNARK prover needs cgo and an 800MB SRS that should not enter this workspace: collect the live inputs, prove out of tree, then assemble.

[seunlanlege]

don't think we need this

[seunlanlege]

we want to precisely extract the next set from hyperbridge header, not just any parachain

[seunlanlege]

better to do this when verifying the hyperbridge header, also the digest should contain the authority set length

[seunlanlege]

Given we want to continue to support multi-proofs, through the ConsensusRouter, I think AuthoritySet might need to change to

struct AuthoritySet {
    uint256 id;
    uint256 len;
    uint256 blsPoseidonHash;
    bytes32 ecdsaMerkleRoot;
}

The ConsensusState can remain unchanged. This ensures EcdsaBeefy & BlsBeefy both continue to work on mainnet. Each client has to advance the authority set for both clients at the same time.

[seunlanlege]

needs authority set len

[seunlanlege]

why not just start with the identity point?

[seunlanlege]

not a fan of this pattern, mutations should be done explicitly, at call site

[seunlanlege]

unnecessary

[seunlanlege]

i feel as though, this is also unneccessary. Simply hash if the current set id has changed

[seunlanlege]

i don't know if we need to do this? whats wrong with using the ProverContext from gnark-apk-proofs?

[seunlanlege]

the gnark prover already does this no?