AI Development in 2026: Evals First, Then Features
AI development is the practice of building software whose core behavior is produced by a model instead of by code you wrote, and it inverts the normal order of work: you write the evaluation set before you write the feature. That single inversion is the discipline. What follows is the four artifacts that make it repeatable, why teams that skip them ship faster and break more, and the arithmetic that makes running evals on every change cost less than a coffee.
TL;DR
- Google Cloud's 2025 DORA report, based on nearly 5,000 technology professionals, found 90 percent use AI at work and over 80 percent report productivity gains, while AI adoption stays negatively related to software delivery stability.
- 30 percent of those same respondents place little or no trust in AI-generated code.
- Stanford HAI's 2026 AI Index reports SWE-bench Verified performance moving from 60 percent to near 100 percent in a single year, and agent success on OSWorld from 12 percent to about 66 percent. Model capability is not your bottleneck.
- The US Census Bureau measured only 17 to 20 percent of US firms using AI between December 2025 and May 2026. Deployment is nowhere near capability.
- Four artifacts close that gap: eval set, task spec, trace log, rollback switch. A full graded 40-case eval run costs about 34 cents at published Claude rates.
What AI development is, and why the order is inverted
In normal software you write a spec, implement it, and test that the implementation matches. The spec is stable and the test asserts equality. In AI development the behavior is a distribution, not a branch. Change one sentence in a system prompt, or accept a point release of the model, and outputs shift across inputs you never touched. Equality assertions are useless. There is nothing to assert against except examples.
So the spec relocates into the test suite. Your eval set becomes the only durable statement of what the system is supposed to do, and the score on that set becomes the thing you optimize. This is not a testing preference. It is the only mechanism available.
The evidence that teams skip this step is not subtle. Google Cloud's 2025 DORA report surveyed nearly 5,000 technology professionals and found 90 percent using AI at work, with more than 80 percent reporting productivity gains, and yet AI adoption remained negatively related to delivery stability. Throughput went up. Stability did not. That is the signature of code arriving faster than any gate can check it, and it is why 30 percent of those respondents said they place little or no trust in AI-generated code.
Meanwhile the models keep getting better at the part everyone worries about. Stanford HAI's 2026 AI Index records SWE-bench Verified performance climbing from 60 percent to near 100 percent in one year, and agent success on real computer tasks in OSWorld rising from 12 percent to roughly 66 percent. Capability is not the constraint. Process is.
The four artifacts every AI project needs
Four files. Written in this order, before the feature exists. If a project has all four it can be handed to another engineer. If it has none it is a prototype wearing a deploy pipeline.
Order matters because each artifact makes the next one cheap. The eval set forces you to define correct. The task spec falls out of that definition. The trace log exists to feed new cases back into the eval set. The rollback switch is what you pull when the score was wrong.
Artifact 1: the eval set you write before the feature
Thirty to fifty cases, each an input paired with an expected output or a pass condition. Half ordinary traffic, a quarter known-hard edges, a quarter malformed or adversarial input. Write the expected outputs by hand. That afternoon of work is where you discover the three requirements nobody had agreed on.
Two rules keep it useful. First, every production failure becomes a permanent new case, so the set only grows in the direction of things that actually broke. Second, the set lives in the repo next to the code, not in a spreadsheet someone owns.
Scoring is where teams stall, so keep it dumb at first. Exact match and schema validation cover more than you expect. Use a model as a grader only for the cases where correct genuinely has multiple valid phrasings, and read the second-opinion pattern before you trust the grades. For agents that call tools, the eval also needs to assert which tools got called and in what order, which is covered in more depth in how to test an AI agent.
Artifact 2: the task spec that defines failure
One page. What goes in, what comes out, what counts as a failure, and what the system does when it fails. The last two are the ones that get skipped.
Write the failure modes as a list with a defined response for each. Model returns malformed JSON: retry once, then fall back to the deterministic path. Model returns a refusal: log it and route to a human. Latency exceeds four seconds: return the cached answer. Tool call errors: surface the error to the model once, then stop.
The spec also pins the model tier, because tier is a product decision with a price attached and not an implementation detail. If you have not made that call deliberately, work through how to choose an AI model first. A classification step running on the cheapest tier and a drafting step running two tiers up is a normal, correct architecture.
Artifacts 3 and 4: the trace log and the rollback switch
The trace log records every production run: input, prompt version, model version, tool calls, output, latency, and token cost. Not sampled. All of it, with a retention window you can afford. This is the only way to answer the question you will be asked in week three, which is "why did it do that on Tuesday." The practical setup is in AI agent observability.
Attach cost per run to the trace and you get budget forecasting for free. You also find out which 5 percent of traffic is burning 40 percent of the spend, which is usually a retry loop nobody knew about. That is the cheapest cost lever available before you start tuning the bill itself.
The rollback switch is a feature flag, and the requirement is that the pre-AI path still works. Not "still exists in git." Works, in production, reachable in one deploy. Ship every AI feature behind the flag, run it in shadow against real traffic while it writes to the trace log without touching the user, then flip it on for a percentage. If the eval score was wrong about something, the flag is what buys you the hour to find out why.
The week loop, and what it costs
Here is the loop those four artifacts produce. Write or extend evals. Build. Score. Read traces. New failures become new cases. Nothing merges without a passing score.
Now the arithmetic, because the usual objection is that running evals constantly is expensive. It is not. Anthropic's published pricing puts Claude Sonnet 5 at 2 dollars per million input tokens and 10 dollars per million output tokens, and Claude Haiku 4.5 at 1 and 5.
Take a 40-case eval at roughly 1,500 input and 300 output tokens per case. Input is 60,000 tokens, which is 12 cents. Output is 12,000 tokens, which is another 12 cents. Grading those 40 outputs on Haiku 4.5 at about 2,000 input and 80 output tokens each adds roughly 10 cents. Call it 34 cents for a full graded run. Twenty runs across a week of iteration is 6 dollars and 80 cents.
Two levers cut it further. The Batch API takes 50 percent off both sides for anything that does not need an instant answer, which describes every eval run in CI. And if your system prompt is a fixed 8,000 tokens repeated across all 40 cases, that block alone is 64 cents at the base rate but 6 cents as a cache read, since cache hits cost a tenth of base input. Caching a large fixed prompt turns the dominant cost line into a rounding error.
Compare that to what it gates. Anthropic's own worked example puts 10,000 support conversations at roughly 37 dollars on Haiku 4.5. A regression that misroutes 5 percent of those tickets costs vastly more in human time than every eval run you will make in a quarter.
The gap between what models can do and what firms have deployed is still enormous. The US Census Bureau's Business Trends and Outlook Survey measured overall AI use among US businesses at between 17 and 20 percent from December 14, 2025 through May 3, 2026, reaching 37 percent among firms with 250 or more employees and 39.7 percent in the Information sector. Capability near 100 percent on coding benchmarks, deployment near 20 percent in the economy. The distance between those two numbers is almost entirely process.
The bottom line
AI development is not harder than normal software. It is differently ordered. The specification cannot live in your head or in a ticket, because the behavior is statistical, so it lives in an eval set you write first. Once that exists, the task spec, the trace log, and the rollback switch cost an afternoon each and pay for themselves the first time something regresses at 2am.
Teams shipping AI features without those four artifacts are the ones producing DORA's pattern: more throughput, less stability, and declining trust in their own output. The fix is 34 cents a run.
Start with the eval set this week. Write 30 cases for the one AI feature you already have in production, run them, and find out what your current score actually is. Grab THE CANARY METHOD for the shadow-deploy and rollback pattern, and join the newsletter for the builds and teardowns I publish every week.
What is AI development?
AI development is building software whose core behavior comes from a model rather than from branches you wrote yourself. The difference matters more than it sounds. In normal software you specify the behavior, implement it, and test that the implementation matches the spec. In AI development you cannot specify the behavior, because the behavior is a distribution over outputs that shifts when you change a prompt, a model version, a retrieval index, or a tool description. So the spec moves into the test suite. You define acceptable behavior as a set of labeled cases with expected outputs, and the eval score becomes the thing you optimize against. Everything else in the lifecycle, including how you deploy and how you roll back, follows from that one structural fact.
Why write evals before the feature in AI development?
Because without them you have no way to tell whether a change helped. Prompt edits feel productive and are almost entirely unmeasurable by eye. You tweak wording, try three inputs, decide it looks better, and ship a regression you will find in production two weeks later. An eval set written before the feature turns that guesswork into a number you can move. It also forces the harder work up front: writing 40 expected outputs means deciding what correct actually means, which is the part teams skip. In practice the eval set is cheap to run and expensive to write, and the writing is where the product thinking happens. Build it first or you are shipping on vibes.
How many eval cases do you actually need?
Start at 30 to 50 and grow it from production failures. Thirty cases is enough to catch a format regression or a broken tool call within a single run, and small enough that a person can write the expected outputs in an afternoon. Below about 20 the score jumps around too much to trust between runs. Above 100 you stop maintaining it. The important property is not size, it is composition: roughly half ordinary cases, a quarter edge cases you already know are hard, and a quarter adversarial or malformed inputs. Every incident in production becomes a new case, permanently. A set that only grows when something breaks stays honest and stays relevant to what your users actually send.
What does it cost to run evals on every change?
Almost nothing, which removes the last excuse. At Anthropic's published rates, Claude Sonnet 5 is 2 dollars per million input tokens and 10 dollars per million output tokens. A 40-case eval at roughly 1,500 input and 300 output tokens per case costs 12 cents of input and 12 cents of output, so 24 cents per run. Grading those outputs with Claude Haiku 4.5 at 1 and 5 dollars per million adds about 10 cents. That is roughly 34 cents for a full graded run. Twenty runs across a week of iteration is under 7 dollars. The Batch API halves both sides for anything that can wait, and a prompt cache read costs a tenth of the base input rate.
Why does AI make software delivery less stable?
Because it raises change volume without raising the capacity of everything downstream. Google Cloud's 2025 DORA report, drawn from nearly 5,000 technology professionals, found AI adoption positively related to delivery throughput and product performance while remaining negatively related to delivery stability. More code arrives, more pull requests open, and review, testing, and rollback capacity stay exactly where they were. The failures then surface as incidents rather than as caught bugs. The same report found 30 percent of respondents place little or no trust in AI-generated code, which is a rational response to shipping faster than you can verify. The fix is not slowing generation down. It is putting an automated gate in front of merge.
OpusJake is Jake Schincariol's operating system for building with AI: agents, workflows, prompts, and the free resources behind them. Get the next move every week.