How Is Artificial Intelligence Made? The Five Stages Behind a Model
Artificial intelligence is made in five stages: collect and filter a training corpus, pretrain a base model on it, post-train that base into something that follows instructions, evaluate and red-team the result, then serve it behind an API. Stage two buys the knowledge. Stage three makes the thing you actually talk to. Almost every public misconception about how AI works lives in the gap between those two.
TL;DR
- Five stages: data, pretraining, post-training, evals, serving. The model you use is the output of stage three, not stage two.
- Data is subtractive. Hugging Face's FineWeb paper built 15 trillion tokens from 96 Common Crawl snapshots, then its FineWeb-Edu quality filter kept just 1.3 trillion of them, under 9 percent.
- Pretraining is the expensive part. Epoch AI measured final-run costs growing 2.4x per year since 2016, which is roughly tenfold every 2.6 years, and projected billion-dollar runs by 2027.
- Post-training is where behavior comes from. OpenAI's InstructGPT paper found humans preferred a 1.3B post-trained model over 175B GPT-3, a model 100x its size.
- Evals are the only stage you should copy. Everything else you rent.
Stage 1: the data is a subtraction problem
People imagine data collection as gathering. It is mostly discarding.
The clearest public example is FineWeb. The paper, published in June 2024, documents a 15 trillion token dataset pulled from 96 Common Crawl snapshots. That 15 trillion is already the survivor of language filtering, quality heuristics, and deduplication across the raw crawl. Then the team applied an education-quality classifier to produce FineWeb-Edu, and that subset is 1.3 trillion tokens. Same source material, and the quality filter kept under 9 percent of it.
The finding that mattered was not the size. It was that the smaller, harder-filtered set trained better models. More tokens is not better. Better tokens is better.
Beyond web crawls, a modern corpus includes licensed text, code repositories, and an increasing share of synthetic data written by other models. That last category is not a shortcut, it is a response to a ceiling. The Epoch AI data paper, revised June 4, 2024, projects that models will be trained on datasets roughly equal to the entire stock of available public human text somewhere between 2026 and 2032, sooner if labs keep overtraining. There is a finite amount of writing on the internet, and the industry can see the edge of it.
Stage 2: pretraining buys knowledge, not behavior
Pretraining is one objective repeated across trillions of tokens: predict the next token. That is the whole thing. Grammar, facts, code syntax, translation, and reasoning patterns all fall out of doing that at sufficient scale.
This is where the compute goes. The 2026 Stanford AI Index reports that global AI compute capacity has grown 3.3x per year since 2022 to reach 17.1 million H100-equivalents, with AI data center power capacity at 29.6 gigawatts. The same chapter notes something more useful for calibration: parameter counts have stayed near 1 trillion for three years. Models are not getting bigger. The compute is going into training them longer, on better data, and into the post-training stage below.
What comes out of pretraining is a base model, and a base model is close to useless as a product. Ask one a question and it will often produce more questions, because it learned that questions appear in lists of questions. It has the knowledge and none of the manners.
Stage 3: post-training makes the model you actually use
This is the stage that turns a text predictor into an assistant, and it is the least understood part of how artificial intelligence is made.
The original recipe is in OpenAI's InstructGPT paper from March 2022, and it has three steps. Fine-tune the base model on human-written demonstrations of good answers. Collect human rankings of model outputs and train a separate reward model to predict those rankings. Then optimize the first model against that reward model with reinforcement learning. The headline result is still the best argument for why this stage matters: humans preferred outputs from the 1.3 billion parameter InstructGPT over outputs from 175 billion parameter GPT-3, a model 100 times larger. Post-training bought more perceived quality than a 100x increase in scale.
The obvious problem is that human rankings are slow and expensive. Two approaches have grown out of that constraint.
Anthropic's Constitutional AI paper, published December 2022, replaces most of the human labeling with a written list of principles. The model critiques and revises its own responses against those principles, then a second model uses the same principles to rank outputs and train the reward model. The human input becomes the constitution rather than the labels, which is a far smaller and far more auditable artifact.
The newer direction drops the learned reward model entirely for tasks where correctness is checkable. DeepSeek's R1 paper, published January 2025, showed that R1-Zero developed reasoning through pure reinforcement learning with no human-labeled reasoning traces at all, and that self-reflection, verification, and strategy switching emerged from the optimization rather than being demonstrated. When a math answer or a unit test can be scored automatically, you do not need a human or a model in the loop. You need a grader.
Stage 4: evals decide whether it ships
A model is not released because it feels better. It is released because it beat the previous version on a battery of fixed test sets covering reasoning, coding, factual accuracy, refusal behavior, and safety, and did not regress on the ones that matter most.
Two things make this harder than it looks. Public benchmarks leak into training data over time, so a rising score can mean contamination rather than capability, which is why labs maintain private held-out sets. And red-teaming, where people actively try to make the model misbehave, produces findings that no automatic score catches.
This is the one stage worth copying wholesale. You will never pretrain anything, but you can absolutely build a set of thirty real inputs from your own workload with known-good outputs, and run every model and prompt change against it. That is the same mechanism, scaled down, and it is covered in detail in how to test an AI agent before you ship it.
Stage 5: serving is part of the build
The finished weights are not a product yet. Serving them at a usable price means batching requests, caching repeated prefixes, quantizing weights to smaller numeric formats, and routing across a fleet. This is why the same underlying model can appear at very different prices and latencies depending on who is hosting it, and why picking a tier matters as much as picking a lab. That decision is its own piece: how to choose an AI model.
What it costs, and why almost nobody does this
Epoch AI's cost analysis, published June 3, 2024, found that the amortized hardware and energy cost of the final training run has grown 2.4x per year since 2016, and projected that the largest runs would pass a billion dollars by 2027. Run the arithmetic on that growth rate: 2.4x annually compounds to about tenfold every 2.6 years.
Their breakdown of total development cost is the part builders usually get wrong. Hardware is 47 to 67 percent. R and D staff is 29 to 49 percent. Energy is 2 to 6 percent. The electricity bill everyone argues about is the smallest line item, and salaries are nearly half.
There is also a transparency problem worth naming. The 2026 AI Index reports that training code, parameter counts, dataset sizes, and training duration are no longer disclosed for several of the most resource-intensive systems, including those from OpenAI, Anthropic, and Google. Anyone who tells you exactly how a current frontier model was built is extrapolating from the last generation that published.
What this actually changes for you
You are not going to make artificial intelligence. You are going to build on it. Knowing the pipeline still changes three decisions.
You now know which stage your problem lives in. Missing knowledge is a stage 1 and 2 problem, and you cannot fix it with prompting, which is why retrieval exists. Wrong behavior or wrong format is a stage 3 problem, and that is fixable from the outside with better instructions or a small fine-tune. Sorting those two correctly is most of the work, and the tradeoff is laid out in RAG vs fine-tuning.
You know why open weights are a different product. An open model gives you the stage 2 output and sometimes a stage 3 checkpoint, which means you can redo post-training for your domain. That capability has a real cost profile, covered in open source AI.
You know evals are yours. Nobody else can build the test set for your workload. If you take one operational habit from how frontier models are made, take that one. If the vocabulary in this post was new, the builder's introduction to AI covers the concepts underneath it, and my daily driver stack is the tooling I actually run.
The bottom line
Artificial intelligence is made by filtering a corpus down to a fraction of what was collected, spending an enormous amount of compute teaching a model to predict the next token, then spending a comparatively tiny amount teaching it how to behave. The second step buys the knowledge. The third step is why a 1.3 billion parameter model beat one 100 times its size in human preference tests. The fourth step is the only one you should rebuild yourself.
Every stage above is rented except the evals. Get the weekly build breakdowns at /newsletter, and start with the AI daily driver stack if you want the tools I use to do this work every day.
How is artificial intelligence made, in simple terms?
In five stages. First a corpus is collected and filtered, usually from web crawls plus licensed and synthetic text, and most of what is collected gets thrown away. Second, a base model is pretrained on that corpus by predicting the next token, which is where the knowledge and the language ability come from and where nearly all the compute is spent. Third, that base model is post-trained with demonstrations and preference signals until it follows instructions instead of just continuing text. Fourth, it is evaluated and red-teamed against fixed test sets to decide whether it ships. Fifth, it is served behind an API, which involves its own engineering around batching, caching, and quantization. The model you talk to is the output of stage three, not stage two.
How much does it cost to train an AI model?
For a frontier model, more than most companies are worth. Epoch AI's analysis of rising training costs, published June 3, 2024, found the amortized hardware and energy cost of the final training run has grown 2.4x per year since 2016, and projected that the largest runs would cost more than a billion dollars by 2027. At 2.4x a year, costs go up roughly tenfold every 2.6 years. Their breakdown of development cost puts hardware at 47 to 67 percent, R and D staff at 29 to 49 percent, and energy at only 2 to 6 percent, which surprises people who assume electricity dominates. Fine-tuning an existing open model is a completely different order of magnitude and can run on a single rented GPU.
What is the difference between pretraining and fine-tuning?
Pretraining builds a base model from scratch on trillions of tokens by predicting the next token. It produces something that knows an enormous amount and does nothing useful on request, because it was never taught that a question should be answered rather than continued. Fine-tuning takes that finished base model and adjusts its behavior on a much smaller, curated dataset, typically thousands to hundreds of thousands of examples rather than trillions of tokens. Pretraining costs tens or hundreds of millions of dollars and takes months of cluster time. Fine-tuning costs anywhere from a few dollars to a few thousand. If you are building on AI rather than building AI, you will only ever touch the second one, and often you will not need even that.
Where does AI training data come from, and will it run out?
Mostly from web crawls, cleaned hard. Hugging Face's FineWeb paper, published June 2024, describes a 15 trillion token dataset built from 96 Common Crawl snapshots, and their higher-quality FineWeb-Edu subset is 1.3 trillion tokens, meaning the education filter kept under 9 percent of what survived the first pass. Beyond the crawl there is licensed text, code repositories, and a growing share of synthetic data generated by other models. On running out, the Epoch AI paper by Villalobos and colleagues, revised June 4, 2024, projected that models will be trained on datasets roughly equal to the entire stock of available public human text somewhere between 2026 and 2032, and sooner if labs keep overtraining smaller models.
How do AI labs decide a model is good enough to release?
With evals, which are fixed test sets scored automatically, plus red-teaming, which is people deliberately trying to make the model behave badly. A release decision is a comparison against the previous model on a battery of benchmarks covering reasoning, coding, factual accuracy, refusal behavior, and safety, and a model that improves on nine axes but regresses on safety does not ship. The uncomfortable part is that public benchmarks leak into training data over time, so labs maintain private held-out sets. This is also the one stage of model building you should copy directly. Building a small eval set for your own use case is the highest-leverage engineering you can do on top of someone else's model.
OpusJake is Jake Schincariol's operating system for building with AI: agents, workflows, prompts, and the free resources behind them. Get the next move every week.