Pretraining Data Preparation#59

Merged

AmitMY merged 22 commits into

ilkerkesen:data-preparation

Feb 17, 2026

Commits on Feb 11, 2026

implement pretraining data preparation script, which creates a subset of the specified data resource.
ilkerkesen
committed
adapt welt pretraining implementation to work with the new data format
ilkerkesen
committed
document how to run the data preparation script
ilkerkesen
committed
make shard limit naming more flexible (100_000 -> 100_000_000)
ilkerkesen
committed
move the module import to top-level
ilkerkesen
committed
fix bugs listed in the PR review
ilkerkesen
committed
fix the issues raised in the PR review
ilkerkesen
committed
fix lint errors
ilkerkesen
committed
implement data loading for the prepared data within the package, so we could import it and re-use when needed.
ilkerkesen
committed
create train / validation splits at preparation time
ilkerkesen
committed
separate train and validation splits at data preparation phase
ilkerkesen
committed
make [(train/validation)]_split_units args required for data preparation phase.
ilkerkesen
committed
implement extract_text procedure to prevent duplicated text_template processing as suggested by Claude
ilkerkesen
committed
make ShardWriter a context manager, initiated with /with/ statements for secure handling
ilkerkesen
committed
do not count whitespace chars while keeping statistics
ilkerkesen
committed
refactor data preparation tests
ilkerkesen
committed
rely on the words segmenter to count number of characters
ilkerkesen
committed