representations

Computing & AI

AI & Language Models

• Tokens and EmbeddingsLLMs do not normally read text one character or one word at a time. A tokenizer converts text into integer token IDs representing chunks such as whole short words, word pieces, punctuation, or bytelevel patterns.