Tokenization

Understands that language models process tokens, not raw characters; knows vocabulary size matters.

Uses HuggingFace tokenizers; understands BPE algorithm; handles special tokens, padding, truncation.

Trains custom tokenizers; understands vocabulary coverage, fertility, multilingual tokenisation.

Analyses tokenisation's impact on arithmetic reasoning, code, and multilingual models; implements byte-level BPE.

Contributes to tokenisation-free models, character-level architectures, or byte-level language models.