Hyperclock cache

AGENTS.md

@@ -0,0 +1,153 @@
+# AGENTS.md - Waterfalls Developer Guide
+
+Waterfalls is a Rust project providing blockchain data to Liquid and Bitcoin light-client wallets.
+
+## Development Environment
+
+Use Nix (defined in `flake.nix`): `nix develop` or `direnv allow`.
+Provides: Rust toolchain (via rust-overlay), RocksDB, OpenSSL, bitcoind, elementsd, libclang.
+
+When the nix env is not already active (e.g. sandbox), prefer `direnv exec . <command>`
+(e.g. `direnv exec . cargo check`) over `nix develop --command <command>` — it uses the
+cached nix-direnv environment and avoids flake re-evaluation overhead.
+
+## Build & Check Commands
+
+```bash
+cargo build                              # Debug build
+cargo build --release                    # Release build
+cargo check                              # Fast type-check
+cargo check --no-default-features        # Check without default features
+cargo check --no-default-features --features test_env
+cargo check --benches
+cargo check --tests
+```
+
+## Test Commands
+
+```bash
+cargo test                               # Run all tests (uses default features: test_env, db)
+cargo test test_name                     # Run a single test by name
+cargo test test_name -- --exact          # Run exactly one test (no substring match)
+cargo test -- --nocapture                # Show stdout/stderr
+cargo test -- --ignored                  # Run ignored tests (require internet)
+cargo test --test integration            # Run only integration tests
+cargo bench --features bench_test        # Run benchmarks (criterion)
+```
+
+### Feature Flags
+
+- `test_env` (default) — enables `bitcoind` dep for tests requiring local nodes
+- `db` (default) — enables RocksDB storage backend
+- `synced_node` — tests requiring a locally running synced node
+- `bench_test` — long-running benchmark tests
+- `examine_logs` — log inspection tests
+
+### Required Environment Variables for Tests
+
+```bash
+export BITCOIND_EXEC=/path/to/bitcoind   # Provided by nix develop
+export ELEMENTSD_EXEC=/path/to/elementsd # Provided by nix develop
+export RUST_LOG=debug                    # Optional: enable debug logging
+```
+
+## Formatting & Linting
+
+```bash
+cargo fmt                                # Format code (default rustfmt settings)
+cargo clippy                             # Lint
+cargo clippy -- -D warnings              # Lint, fail on warnings (CI enforced)
+```
+
+No `rustfmt.toml` or `clippy.toml` — default settings are used.
+
+## Code Style Guidelines
+
+### Imports
+
+Grouped in this order (separated by blank lines when practical):
+1. `std::` — standard library
+2. External crates — `anyhow`, `elements`, `hyper`, `serde`, `tokio`, etc.
+3. `crate::` / `super::` — internal modules
+
+Use absolute paths: `use waterfalls::be::Address` (in tests/benches), `use crate::be::Address` (within the crate).
+
+### Error Handling
+
+- **Application-level**: `anyhow::Result` for fallible operations in fetch, threads, startup.
+- **Server routes**: custom `Error` enum in `src/server/mod.rs` mapped to HTTP status codes.
+  - `CannotDecrypt` → 422, `BodyTooLarge` → 413, `BodyReadTimeout` → 408, input errors → 400, others → 500.
+- **Fetch layer**: custom `Error` enum in `src/fetch.rs` (`TxNotFound`, `BlockNotFound`, etc.).
+- **Unrecoverable**: `error_panic!` macro (defined in `src/lib.rs`) — logs via `log::error!` then panics.
+- Log errors with `log::error!` before returning them.
+
+### Logging
+
+- Use the `log` crate: `log::info!`, `log::warn!`, `log::error!`, `log::debug!`
+- Initialized via `env_logger` in `main.rs` (default filter: `info`)
+- For systemd integration: set `RUST_LOG_STYLE=SYSTEMD`
+
+### Async Code
+
+- Runtime: `tokio` with `rt-multi-thread`
+- Use `tokio::select!` for concurrent operations (e.g., signal handling)
+- Use `#[tokio::test]` for async tests
+
+### Serialization
+
+- JSON: `serde` / `serde_json` with `Serialize`/`Deserialize` derives
+- CBOR: `minicbor` with `Encode`/`Decode` derives and custom `with` helpers in `src/cbor.rs`
+- Field annotations: `#[cbor(n(X))]` for CBOR field indices, `#[serde(skip_serializing_if = ...)]`
+
+### Testing Patterns
+
+- `#[tokio::test]` for async tests
+- `#[cfg(feature = "test_env")]` gates tests needing bitcoind/elementsd
+- `#[cfg(all(feature = "test_env", feature = "db"))]` for DB-backed integration tests
+- `#[ignore = "requires internet"]` for tests hitting remote endpoints
+- `env_logger::try_init()` at test start (ignore the error if already initialized)
+- Test infrastructure in `src/test_env.rs`: `TestEnv`, `WaterfallClient`, `launch()`, `launch_with_node()`
+- Integration tests in `tests/integration.rs` use `launch_memory()` / `test_env::launch()` to spin up node + server
+
+## Project Structure
+
+```
+src/
+├── lib.rs              # Library root: types, error_panic! macro, prometheus metrics
+├── main.rs             # Binary entry: clap parsing, logging, signal handling
+├── fetch.rs            # Blockchain data fetching (esplora / local node REST)
+├── cbor.rs             # CBOR encoding helpers for block hashes
+├── test_env.rs         # Test utilities (TestEnv, WaterfallClient)
+├── be/                 # Backend types (Address, Block, BlockHeader, Descriptor, Tx, Txid)
+├── server/             # HTTP server: Arguments (clap), Network, Error enum, routing,
+│                       #   state, mempool, signing, encryption, derivation_cache, preload
+├── store/              # Store trait + AnyStore, memory.rs, db.rs (RocksDB, behind `db` feature)
+└── threads/            # Background tasks: block indexing, mempool sync
+build.rs                # Injects GIT_COMMIT_HASH at build time
+tests/integration.rs    # Integration tests
+benches/benches.rs      # Criterion benchmarks
+```
+
+## CI (`.github/workflows/rust.yml`)
+
+Runs on push/PR to `master`:
+- **tests**: downloads bitcoind 28.0 & elementsd 23.2.4, runs `cargo test` and `cargo test -- --ignored`
+- **checks**: `cargo check` with various feature combinations
+- **nix**: `nix build .` with cachix
+
+## Cursor Rules
+
+From `.cursor/rules/my-custom-rule.mdc` (always applied):
+
+1. The developer uses a Nix environment from `flake.nix` — use it when proposing commands
+2. Add new structs at the end of files, or just before `#[cfg(test)] mod tests` if present
+3. Never add new dependencies unless explicitly asked
+
+## Common Tasks
+
+```bash
+cargo run -- --network liquid --use-esplora   # Run server against esplora
+cargo test --features "test_env db"           # Run tests with DB backend
+nix build .#dockerImage && docker load < result  # Build Docker image
+cargo bench --features bench_test             # Run benchmarks
+```

README.md

@@ -1,6 +1,6 @@
 # Waterfalls
 
-Waterfalls is a new scanning mechanism for web light-clients wallets that leverages a new server endpoint. It works for bitcoin and liquid. See [API.md](API.md) for complete API documentation.
+Waterfalls is a new scanning mechanism for web light-clients wallets that leverages a new server endpoint. It works for bitcoin and liquid. See [API.md](docs/API.md) for complete API documentation.
 
 It also has an UTXO-only mode to allow wallets to know their balance and being able to construct transaction in a faster way, at the expense of not knowing entire transaction history.
 

benches/benches.rs

@@ -10,7 +10,9 @@ use elements::secp256k1_zkp::rand::{thread_rng, RngCore};
 use elements::{OutPoint, Txid};
 use elements_miniscript::Descriptor;
 use elements_miniscript::DescriptorPublicKey;
-use rocksdb::{Options, WriteBatch, DB};
+use rocksdb::{
+    BlockBasedOptions, Cache, ColumnFamilyDescriptor, DBCompressionType, Options, WriteBatch, DB,
+};
 use tempfile;
 use waterfalls::WaterfallResponse;
 
@@ -21,7 +23,8 @@ criterion_group!(
     sign_verify,
     writebatch_sorting,
     hasher,
-    txid_from_hex
+    txid_from_hex,
+    block_cache
 );
 criterion_main!(benches);
 
@@ -369,3 +372,174 @@ pub fn txid_from_hex(c: &mut Criterion) {
             },
         );
 }
+
+fn open_cache_bench_db(dir: &std::path::Path, cache: &Cache) -> DB {
+    let cfs = ["utxo", "history"]
+        .iter()
+        .map(|&name| {
+            let mut cf_opts = Options::default();
+            cf_opts.set_compression_type(DBCompressionType::None);
+
+            let mut block_opts = BlockBasedOptions::default();
+            block_opts.set_block_cache(cache);
+            block_opts.set_block_size(16 * 1024);
+            block_opts.set_cache_index_and_filter_blocks(true);
+            block_opts.set_pin_l0_filter_and_index_blocks_in_cache(true);
+            if name == "history" {
+                block_opts.set_bloom_filter(10.0, true);
+            }
+            cf_opts.set_block_based_table_factory(&block_opts);
+
+            ColumnFamilyDescriptor::new(name, cf_opts)
+        })
+        .collect::<Vec<_>>();
+
+    let mut db_opts = Options::default();
+    db_opts.create_if_missing(true);
+    db_opts.create_missing_column_families(true);
+    DB::open_cf_descriptors(&db_opts, dir, cfs).unwrap()
+}
+
+fn populate_cache_bench_db(db: &DB, num_utxo_keys: u64, num_history_keys: u64) {
+    let cf = db.cf_handle("utxo").unwrap();
+    for start in (0..num_utxo_keys).step_by(10_000) {
+        let mut batch = WriteBatch::default();
+        for i in start..(start + 10_000).min(num_utxo_keys) {
+            let mut key = [0u8; 36];
+            key[..8].copy_from_slice(&i.to_be_bytes());
+            batch.put_cf(&cf, key, i.to_be_bytes());
+        }
+        db.write(batch).unwrap();
+    }
+
+    let cf = db.cf_handle("history").unwrap();
+    for start in (0..num_history_keys).step_by(10_000) {
+        let mut batch = WriteBatch::default();
+        for i in start..(start + 10_000).min(num_history_keys) {
+            batch.put_cf(&cf, i.to_be_bytes(), vec![0xABu8; 100 + (i as usize % 50)]);
+        }
+        db.write(batch).unwrap();
+    }
+
+    for name in ["utxo", "history"] {
+        let cf = db.cf_handle(name).unwrap();
+        db.flush_cf(&cf).unwrap();
+        db.compact_range_cf(&cf, None::<&[u8]>, None::<&[u8]>);
+    }
+}
+
+/// Compares LRUCache vs HyperClockCache for RocksDB block cache using
+/// `multi_get_cf` (the actual production access pattern) under varying
+/// concurrency levels.
+///
+/// Two scenarios model production data-to-cache ratios:
+/// - **liquid**: ~21 MB data, 2 MB cache (~9%) — matches Liquid mainnet (~6%)
+/// - **bitcoin**: ~76 MB data, 2 MB cache (~2.6%) — matches Bitcoin UTXO (~3.6%)
+///
+/// Each scenario tests 1, 4, and 16 concurrent reader threads.
+pub fn block_cache(c: &mut Criterion) {
+    const LOOKUPS_PER_ITER: usize = 100;
+
+    struct Scenario {
+        name: &'static str,
+        cache_size: usize,
+        num_utxo_keys: u64,
+        num_history_keys: u64,
+    }
+
+    let scenarios = [
+        Scenario {
+            name: "liquid",
+            cache_size: 2 * 1024 * 1024,
+            num_utxo_keys: 200_000,
+            num_history_keys: 100_000,
+        },
+        Scenario {
+            name: "bitcoin",
+            cache_size: 2 * 1024 * 1024,
+            num_utxo_keys: 300_000,
+            num_history_keys: 500_000,
+        },
+    ];
+
+    let thread_counts: &[usize] = &[1, 4, 16];
+
+    for scenario in &scenarios {
+        let mut rng = thread_rng();
+        let history_keys: Vec<[u8; 8]> = (0..LOOKUPS_PER_ITER)
+            .map(|_| (rng.next_u64() % (scenario.num_history_keys * 2)).to_be_bytes())
+            .collect();
+
+        let lru_dir = tempfile::TempDir::new().unwrap();
+        let lru_cache = Cache::new_lru_cache(scenario.cache_size);
+        let lru_db = open_cache_bench_db(lru_dir.path(), &lru_cache);
+        populate_cache_bench_db(&lru_db, scenario.num_utxo_keys, scenario.num_history_keys);
+
+        let hcc_dir = tempfile::TempDir::new().unwrap();
+        let hcc_cache = Cache::new_hyper_clock_cache(scenario.cache_size, 0);
+        let hcc_db = open_cache_bench_db(hcc_dir.path(), &hcc_cache);
+        populate_cache_bench_db(&hcc_db, scenario.num_utxo_keys, scenario.num_history_keys);
+
+        let mut group = c.benchmark_group(format!("block_cache/{}", scenario.name));
+
+        for &num_threads in thread_counts {
+            let label = format!("{}t", num_threads);
+
+            group.bench_function(format!("lru/{label}"), |b| {
+                if num_threads == 1 {
+                    let cf = lru_db.cf_handle("history").unwrap();
+                    b.iter(|| {
+                        let keys: Vec<_> = history_keys.iter().map(|k| (&cf, *k)).collect();
+                        black_box(lru_db.multi_get_cf(keys));
+                    });
+                } else {
+                    b.iter_custom(|iters| {
+                        let start = std::time::Instant::now();
+                        std::thread::scope(|s| {
+                            for _ in 0..num_threads {
+                                s.spawn(|| {
+                                    let cf = lru_db.cf_handle("history").unwrap();
+                                    for _ in 0..iters {
+                                        let keys: Vec<_> =
+                                            history_keys.iter().map(|k| (&cf, *k)).collect();
+                                        black_box(lru_db.multi_get_cf(keys));
+                                    }
+                                });
+                            }
+                        });
+                        start.elapsed()
+                    });
+                }
+            });
+
+            group.bench_function(format!("hyperclock/{label}"), |b| {
+                if num_threads == 1 {
+                    let cf = hcc_db.cf_handle("history").unwrap();
+                    b.iter(|| {
+                        let keys: Vec<_> = history_keys.iter().map(|k| (&cf, *k)).collect();
+                        black_box(hcc_db.multi_get_cf(keys));
+                    });
+                } else {
+                    b.iter_custom(|iters| {
+                        let start = std::time::Instant::now();
+                        std::thread::scope(|s| {
+                            for _ in 0..num_threads {
+                                s.spawn(|| {
+                                    let cf = hcc_db.cf_handle("history").unwrap();
+                                    for _ in 0..iters {
+                                        let keys: Vec<_> =
+                                            history_keys.iter().map(|k| (&cf, *k)).collect();
+                                        black_box(hcc_db.multi_get_cf(keys));
+                                    }
+                                });
+                            }
+                        });
+                        start.elapsed()
+                    });
+                }
+            });
+        }
+
+        group.finish();
+    }
+}