Pretraining

Knows pretraining creates a general-purpose base model from large corpora before task-specific finetuning.

Understands CLM, MLM, and contrastive objectives; knows data mixture matters; familiar with GPT/BERT/T5 paradigms.

Can run a small pretraining experiment; understands data deduplication, tokeniser coverage, and curriculum learning basics.

Understands scaling laws; designs data mixtures; analyses emergent abilities and training dynamics.

Trains foundation models; contributes to data selection, training efficiency, or continual pretraining research.