Topic
AI & Language Models
How language models represent text, attend to context, predict tokens, and run locally or in the cloud.
Start Here
- Tokens and EmbeddingsLLMs do not normally read text one character or one word at a time. A tokenizer converts text into integer token IDs representing chunks such as whole short words, word pieces, punctuation, or bytelevel patterns.
Go Deeper
- Transformer AttentionAttention lets each token build a new representation by selectively combining information from other tokens in the context.
- What an LLM Actually PredictsAn autoregressive large language model is trained around an almost embarrassingly simple objective: given previous tokens, assign probabilities to the next token.
- Why Local vs Cloud AI Models DifferRunning an AI model locally versus through a cloud service changes where the weights and computation live, which in turn changes privacy, latency, hardware limits, cost structure, and model choice.