Transformer architecture

Knows transformers power GPT/BERT; understands self-attention replaces recurrence.

Implements a minimal GPT-2 from scratch; understands encoder, decoder, feed-forward layers, residual connections.

Fine-tunes BERT/GPT on downstream tasks using HuggingFace; understands layer norm placement, KV cache, context length.

Analyses architectural variants (PaLM, LLaMA, Mistral); implements grouped-query attention, sliding window attention; understands emergent abilities.

Contributes to novel transformer architectures, mechanistic interpretability, or scaling laws research.