runtimelab▮
Sign in

~/runtimelab $ cat ./bpe-tokenizer/README

Build your own BPE Tokenizer

Build a byte-pair-encoding tokenizer from scratch — training, encoding, decoding, special tokens, and the byte-level edge cases nobody warns you about. The GPT-2/GPT-4 tokenizer, cross-checked against tiktoken.

stages
14
estimated
6–8 h
level
Beginner
track
AI Foundations
languages
Python · Go
  • UTF-8 byte merging
  • vocabulary training
  • special tokens
  • encode/decode
  • byte offsets
  • compression trade-offs
Sign in to startBrowse all challenges

The stage list opens once you sign in — it is the challenge, not the pitch. Signing in is free, and the first stages of every challenge are too.