Skip to content

tidesdb engine correctness, statement atomicity, and planner-accuracy#154

Merged
guycipher merged 2 commits into
masterfrom
459
Jul 7, 2026
Merged

tidesdb engine correctness, statement atomicity, and planner-accuracy#154
guycipher merged 2 commits into
masterfrom
459

Conversation

@guycipher

Copy link
Copy Markdown
Member
  • update_row reuses the cached per-index dup-check iterator instead of building and tearing down an o(num_sstables) merge heap on every row, the same caching write_row already does.matters for a bulk update that shifts a unique-indexed column

  • restore statement-level atomicity inside begin...commit.a per-statement savepoint is armed in external_lock and, on statement error, we roll back to it and revert the plugin-side state to the same boundary -- txn_key_state through an undo journal, fts meta through a snapshot, and the statement's row locks to a marker -- then bump the txn generation so cached iterators rebuild.the old path did a full transaction rollback and shifted the snapshot, losing every earlier statement's work.its cited savepoint deep-copy cost was wrong, tidesdb savepoints are o(1).falls back to full rollback when a bulk mid-commit or a user rollback-to-savepoint voids it

  • skip compression on an encrypted table's data column family.rows are encrypted before they reach the library and ciphertext does not compress, so running lz4 over it on every flush and compaction spent cpu for nothing. index cfs hold unencrypted keys and keep their compression

  • advertise ha_keyread_only per key part.the covering bitmap is built by index_flags(idx, part), so a part is keyread-only only when its field can be rebuilt from its sort key.undecodable types (varchar, decimal, float, multi-byte char) no longer get priced as a row-fetch-free covering scan that then pays a hidden pk lookup per row.reading only the decodable columns of a mixed index still covers

  • seek to the upper bound for where seccol > x on a secondary index instead of stepping over every duplicate-prefix entry, with a single step past the one row whose pk encodes to all-0xff

  • probe pk uniqueness with tidesdb_txn_contains on the write txn and drop the per-row dedicated read-committed txn and its commit+reset churn.the non-tracking check keeps the probe out of the reservation base, and the insert's own reservation still enforces uniqueness at commit

  • take the row count from tidesdb_cf_estimate_cardinality, which dedupes the mvcc versions of a key within an sstable that total_keys counts separately and includes the shared unified memtable.fixes table_rows and index cardinality reading a floored minimum for unflushed tables

  • drop the misleading tdb_max_txn_ops line from the bulk-commit batching comments, the real bound is txn memory not the op cap

  • new mtr tests for statement atomicity, encrypted-cf compression, keyread decodability, secondary greater-than range, and row cardinality.re-record index_stats, info_schema, mrr, online_ddl and per_index_btree whose baselines had captured the old floored stats

* update_row reuses the cached per-index dup-check iterator instead of
building and tearing down an o(num_sstables) merge heap on every row,
the same caching write_row already does.matters for a bulk update that
shifts a unique-indexed column

* restore statement-level atomicity inside begin...commit.a per-statement
savepoint is armed in external_lock and, on statement error, we roll back
to it and revert the plugin-side state to the same boundary -- txn_key_state
through an undo journal, fts meta through a snapshot, and the statement's
row locks to a marker -- then bump the txn generation so cached iterators
rebuild.the old path did a full transaction rollback and shifted the
snapshot, losing every earlier statement's work.its cited savepoint
deep-copy cost was wrong, tidesdb savepoints are o(1).falls back to full
rollback when a bulk mid-commit or a user rollback-to-savepoint voids it

* skip compression on an encrypted table's data column family.rows are
encrypted before they reach the library and ciphertext does not compress,
so running lz4 over it on every flush and compaction spent cpu for nothing.
index cfs hold unencrypted keys and keep their compression

* advertise ha_keyread_only per key part.the covering bitmap is built by
index_flags(idx, part), so a part is keyread-only only when its field can be
rebuilt from its sort key.undecodable types (varchar, decimal, float,
multi-byte char) no longer get priced as a row-fetch-free covering scan that
then pays a hidden pk lookup per row.reading only the decodable columns of
a mixed index still covers

* seek to the upper bound for where seccol > x on a secondary index instead of
stepping over every duplicate-prefix entry, with a single step past the one
row whose pk encodes to all-0xff

* probe pk uniqueness with tidesdb_txn_contains on the write txn and drop the
per-row dedicated read-committed txn and its commit+reset churn.the
non-tracking check keeps the probe out of the reservation base, and the
insert's own reservation still enforces uniqueness at commit

* take the row count from tidesdb_cf_estimate_cardinality, which dedupes the
mvcc versions of a key within an sstable that total_keys counts separately
and includes the shared unified memtable.fixes table_rows and index
cardinality reading a floored minimum for unflushed tables

* drop the misleading tdb_max_txn_ops line from the bulk-commit batching
comments, the real bound is txn memory not the op cap

* new mtr tests for statement atomicity, encrypted-cf compression, keyread
decodability, secondary greater-than range, and row cardinality.re-record
index_stats, info_schema, mrr, online_ddl and per_index_btree whose
baselines had captured the old floored stats
@guycipher guycipher self-assigned this Jul 7, 2026
@guycipher guycipher merged commit 84c5cdf into master Jul 7, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant