Understanding BPE vs WordPiece vs Unigram by actually running them.
A hands-on notebook that shows you how different tokenizers work and when to use each one.
You'll see:
- How BPE builds vocabulary by merging common pairs
- How WordPiece scans for the longest match
- How Unigram picks the most probable split
- Which one works best for your data
No theory overload. Just run it and see what happens.
# Get the code
git clone https://github.com/aneja5/tokenizer-tour.git
cd tokenizer-tour
# Set up environment
python -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
pip install -r requirements.txt
# Download datasets (takes 2-3 min)
python prepare_data.py
# Open it
jupyter notebook tokenizer_tour.ipynbThe notebook walks through real examples:
- "unhappiness" - See how BPE discovers morphology
- Turkish words - See why English tokenizers struggle
- Emoji - See what breaks each approach
- Code identifiers - See which preserves structure
- Your own text - Test with your actual data
Tokenizers aren't just technical details. They affect:
- API costs (more tokens = more money)
- Context limits (more tokens = less text fits)
- Model performance (bad tokenization = worse results)
Pick the wrong one for your data and you'll pay for it.
After running the notebook, you'll know:
- English text? → WordPiece (most efficient)
- Multilingual? → Unigram (built for it)
- Code/emoji/mixed? → BPE (never fails)
But don't take my word for it - test on your data.
tokenizer-tour/
├── README.md ← You are here
├── requirements.txt ← Just transformers, jupyter, pandas
├── prepare_data.py ← Downloads test datasets
└── tokenizer_tour.ipynb ← The main notebook
Datasets and outputs aren't tracked (too big, regenerate locally).
- Python 3.8+
- ~15MB for datasets
- Internet connection for downloads
Open an issue or just read the notebook - the code explains itself.
MIT - do whatever you want with it.