Quick Answer: What Is a Large Language Model?
A Large Language Model (LLM) is an AI system trained to predict the next word in a sequence. That is the entire foundation. Everything GPT, Claude, Gemini, and Llama do generating text, answering questions, writing code, summarizing documents comes from repeating this simple prediction loop millions of times per second. The model does not think, reason, or understand in the human sense. It is an extremely sophisticated pattern-matching engine that has learned statistical relationships between words from training on trillions of text examples. Understanding this foundation explains both what LLMs can do and why they sometimes fail.
Tokens: The Building Blocks
Before an LLM can process text, the text must be converted into tokens. A token is a chunk of text that the model treats as a unit. On average, one token equals about 0.75 words. The word apple is one token. The word understanding is three tokens: under, stand, and ing. The word ChatGPT is one token because the model has seen it frequently enough to treat it as a single unit.
Tokenization is why AI pricing uses tokens instead of words or characters. When you use ChatGPT or Claude, you are paying for the number of tokens processed, not the number of words. A typical page of text contains about 250 tokens. A long article might be 1,000 tokens. The model context window, typically 8,000 to 200,000 tokens, determines how much text the model can consider at once. For a practical introduction to working with these models, see our prompt engineering guide.
The Three Training Phases
Creating a useful LLM requires three distinct training phases, each with a different goal and cost structure.
Phase 1: Pretraining. The model consumes trillions of tokens from the internet: web pages, books, academic papers, code repositories, and more. Its task is simple: predict the next token. The model sees a sequence of tokens and tries to guess what comes next. When it guesses wrong, it adjusts its internal parameters to improve. This process repeats billions of times over weeks or months using thousands of specialized GPUs. The result is a base model that is fluent in language but not particularly helpful. It knows how text works but does not follow instructions or answer questions usefully.
Phase 2: Instruction Tuning. The base model is fine-tuned on a curated dataset of questions and desired answers. The model learns to follow instructions rather than simply continuing text. This phase uses far less data than pretraining but requires high-quality human-generated examples. The difference between a base model and an instruct model is visible: the base model might continue your sentence, while the instruct model answers your question.
Phase 3: Reinforcement Learning from Human Feedback. Human raters compare multiple model responses and rank them by quality. The model learns to prefer responses that humans find helpful, accurate, and appropriate. This phase is what makes models safe and aligned with human preferences. It is also the most controversial phase, because what humans consider good varies across cultures and contexts.
How Transformers Work (Simplified)
The Transformer architecture, introduced in the 2017 paper Attention Is All You Need by Google researchers, is the engine behind every modern LLM. For an official introduction, see Google ML Crash Course. The key innovation is the attention mechanism. When predicting the next token, the model does not just look at the last few tokens. It can look at every previous token in the context window and decide which ones are most relevant to predicting what comes next.
Think of attention as the model ability to search through everything it has seen so far and highlight what matters. In the sentence The cat sat on the mat because it was tired, the word it refers to the cat, not the mat. The attention mechanism learns to make this connection even though the words are far apart. This ability to handle long-range context is what makes Transformers so powerful. Each token can attend to every other token, creating a rich web of relationships that captures meaning far better than previous architectures.
What LLMs Can and Cannot Do
| LLMs Can Do Well | LLMs Cannot Do |
|---|---|
| Summarize and paraphrase text | Think or reason in the human sense |
| Generate creative content based on patterns | Remember anything between conversations |
| Translate between languages | Distinguish truth from fiction reliably |
| Write and debug code | Understand concepts beyond statistical patterns |
| Answer questions based on provided context | Handle data beyond the context window |
| Follow complex instructions | Admit uncertainty without specific training |
Why Do LLMs Hallucinate?
Hallucination is not a bug. It is a feature of how LLMs work. An LLM is trained to predict plausible text, not true text. Most of the time, plausible text is also true, because the training data mostly contains true statements. But when the model does not know the answer, it does not say I do not know. It says the most plausible text based on its training, which may be confidently wrong.
Hallucinations are more common when the model is asked about obscure topics that appear infrequently in training data, recent events that happened after training, specific numerical facts that require exact recall, or topics where training data contains conflicting information. Techniques like RAG (Retrieval-Augmented Generation), grounding, and chain-of-thought prompting reduce but do not eliminate hallucinations. For more on working with AI effectively despite these limitations, read our guide to using AI at work.
Why Training Costs So Much
Training a frontier LLM costs tens to hundreds of millions of dollars. The cost comes from the massive computational requirements. Training requires thousands of GPUs running 24/7 for weeks or months. A single training run for a model like GPT-5 or Claude 4 might consume 50-100 gigawatt-hours of electricity, equivalent to the annual energy consumption of several thousand homes. The GPU hardware itself costs tens of millions. Data center infrastructure, cooling, networking, and storage add additional layers of expense.
Inference running the model after training is also expensive, which is why API pricing exists. Every time you send a query to ChatGPT or Claude, it requires significant computation. The model billions of parameters must be loaded into memory and processed for each token generated. This is why companies spend billions on AI inference infrastructure and why model efficiency is a major research focus. For more on AI economics, see our analysis of AI training costs.
How Different Models Compare
All modern LLMs use the same Transformer architecture, but they differ in scale, training data, and alignment approach. GPT models from OpenAI emphasize broad capability and consumer accessibility. Claude models from Anthropic prioritize safety and constitutional AI principles. Gemini models from Google leverage deep integration with Google services and multimodal capabilities. Llama models from Meta are open-weight, allowing customization and self-hosting. Grok from xAI emphasizes real-time information and less restrictive content policies. The choice between models depends on your specific use case, budget, and requirements for safety, customization, and integration.
Frequently Asked Questions
Does an LLM actually think or understand?
No. An LLM predicts the next token based on statistical patterns in training data. It does not think, reason, or understand in the human sense. It is an extremely sophisticated pattern-matching engine.
Why do LLMs hallucinate?
LLMs are trained to produce plausible text, not true text. When they do not know an answer, they generate the most plausible continuation rather than admitting uncertainty.
What is a token and why does pricing use it?
A token is a chunk of text (about 0.75 words) that the model processes as a unit. Pricing uses tokens because the model computational cost scales with the number of tokens processed.
How much does it cost to train an LLM?
Training a frontier model costs tens to hundreds of millions of dollars, driven by GPU hardware, energy consumption, and data center infrastructure requirements.
What is the difference between GPT, Claude, and Gemini?
All use the same Transformer architecture but differ in scale, training data, safety approach, and specific capabilities. The best choice depends on your use case.
