Request
Right now, every format ingested into clp-s ends up with its own ingest_X function in JsonParser, which is responsible for parsing records and inserting them into an archive. Unfortunately, these ingest_X functions end up being responsible as well for file-level metadata, archive splitting, and generic archive-level features like recording the log_event_idx for each record. As such, there's a large amount of code replicated into each ingest_X function that implements core archive functionality which needs to be very carefully kept in sync.
This makes it really tedious and error-prone to add new archive-level functionality to the format. Ideally, we can refactor ingestion to avoid duplicating all of this archive-level logic in the ingestion logic for each input type.
Possible implementation
Possibly, we could offer an interface where we have a templated function which accepts a few functions with well-defined interfaces for an input type. That function would call the functions from the user to ingest records from the input one at a time, and would itself be responsible for most details of maintaining file-level metadata (with some cooperation from functions it was passed), split archives when appropriate, and support any other generic archive features which are independent of input.
We do need to be careful to maintain ingestion performance through such a refactor, but it should be feasible.
Request
Right now, every format ingested into clp-s ends up with its own
ingest_Xfunction inJsonParser, which is responsible for parsing records and inserting them into an archive. Unfortunately, theseingest_Xfunctions end up being responsible as well for file-level metadata, archive splitting, and generic archive-level features like recording thelog_event_idxfor each record. As such, there's a large amount of code replicated into eachingest_Xfunction that implements core archive functionality which needs to be very carefully kept in sync.This makes it really tedious and error-prone to add new archive-level functionality to the format. Ideally, we can refactor ingestion to avoid duplicating all of this archive-level logic in the ingestion logic for each input type.
Possible implementation
Possibly, we could offer an interface where we have a templated function which accepts a few functions with well-defined interfaces for an input type. That function would call the functions from the user to ingest records from the input one at a time, and would itself be responsible for most details of maintaining file-level metadata (with some cooperation from functions it was passed), split archives when appropriate, and support any other generic archive features which are independent of input.
We do need to be careful to maintain ingestion performance through such a refactor, but it should be feasible.