Problem
Documents are keyed by entity IRI and imported with action: 'upsert', which replaces the whole document. When two selected datasets both describe one entity – a shared Term, a Person cited by two collections, a Publisher behind two datasets – the second write replaces the first. The document ends up holding one dataset's view, not what the datasets jointly say: if A gives a Person an English label and B a Dutch one, whichever ran last wins and the other is lost.
Why the sources array is not enough
#697 proposes making the provenance stamp multi-valued. That fixes attribution and makes deletion safe, but leaves content last-writer-wins – and creates a subtler inconsistency, because a document can then list two datasets while carrying only one's values.
Merging content is a different change, because a merge you cannot reverse is worse than no merge. Once A and B are merged into one document, removing A means knowing which values were A's. Merge-on-write throws that away, so a departing dataset would force re-indexing every dataset sharing an entity with it – unbounded, and not something a nightly run can do.
Typesense has no primitive for this: no atomic array operation, and emplace/partial update merges the fields you send rather than combining two writers' values.
Shape
Keep the sources separate and derive the merge:
- one contribution per (dataset, entity), keyed
<dataset>|<entity>, holding that dataset's projected statement;
- the served document materialized from an entity's contributions.
Attribution is then the key of the row a value came from – no per-field provenance sidecar on the served document, which would bloat the indexed surface without making subtraction correct. Departure is exact and bounded by the departing dataset: enumerate its contributions, collect their entity IRIs, delete, re-materialize just those entities (delete the served document when none remain).
The sources array from #697 survives with a different job – it is derived rather than maintained, and tells the writer when the expensive path is needed: an entity whose set is [A] while A is writing can be overwritten straight through, which is the overwhelming majority.
Open decision
The merge rule per field, derivable from the declaration but needing to be stated: array unions (dedup as applyFacet already does); text merges per language, the case where two datasets genuinely complement each other; a single-valued scalar needs a tie-break, and that is where a merged document can silently disagree with both its sources.
Status
Deferred. No deployment currently has different datasets declaring triples about shared resources, so nothing depends on this yet. #697 is postponed with it – its failure mode only occurs for shared entities.
Filed so the analysis is not lost: whoever picks up #697 should know it is one layer of a larger problem, not the whole of it.
Related
Problem
Documents are keyed by entity IRI and imported with
action: 'upsert', which replaces the whole document. When two selected datasets both describe one entity – a sharedTerm, aPersoncited by two collections, aPublisherbehind two datasets – the second write replaces the first. The document ends up holding one dataset's view, not what the datasets jointly say: if A gives a Person an English label and B a Dutch one, whichever ran last wins and the other is lost.Why the sources array is not enough
#697 proposes making the provenance stamp multi-valued. That fixes attribution and makes deletion safe, but leaves content last-writer-wins – and creates a subtler inconsistency, because a document can then list two datasets while carrying only one's values.
Merging content is a different change, because a merge you cannot reverse is worse than no merge. Once A and B are merged into one document, removing A means knowing which values were A's. Merge-on-write throws that away, so a departing dataset would force re-indexing every dataset sharing an entity with it – unbounded, and not something a nightly run can do.
Typesense has no primitive for this: no atomic array operation, and
emplace/partial update merges the fields you send rather than combining two writers' values.Shape
Keep the sources separate and derive the merge:
<dataset>|<entity>, holding that dataset's projected statement;Attribution is then the key of the row a value came from – no per-field provenance sidecar on the served document, which would bloat the indexed surface without making subtraction correct. Departure is exact and bounded by the departing dataset: enumerate its contributions, collect their entity IRIs, delete, re-materialize just those entities (delete the served document when none remain).
The sources array from #697 survives with a different job – it is derived rather than maintained, and tells the writer when the expensive path is needed: an entity whose set is
[A]while A is writing can be overwritten straight through, which is the overwhelming majority.Open decision
The merge rule per field, derivable from the declaration but needing to be stated:
arrayunions (dedup asapplyFacetalready does);textmerges per language, the case where two datasets genuinely complement each other; a single-valued scalar needs a tie-break, and that is where a merged document can silently disagree with both its sources.Status
Deferred. No deployment currently has different datasets declaring triples about shared resources, so nothing depends on this yet. #697 is postponed with it – its failure mode only occurs for shared entities.
Filed so the analysis is not lost: whoever picks up #697 should know it is one layer of a larger problem, not the whole of it.
Related