Home / Skills tracks / Inside a large language model
At schoolPost 16 and adults

Inside a large language model

Tokens, embeddings, attention and sampling, built up in code until hallucination and context limits stop being mysterious.

4 units, about 4 hours. Free.

1

Tokens: the units a model actually sees

Explain tokenisation and predict why character level tasks and context limits behave oddly.

2

Embeddings: meaning as coordinates

Explain what an embedding vector is and compute similarity between meanings in code.

3

Attention: choosing what matters

Compute an attention weighting in code and explain why context changes a token meaning.

4

Sampling, temperature and hallucination

Control generation with temperature and top-k, and explain hallucination mechanically.