How to Handle LLM Rate Limits Without Making Them Worse

LLM rate limits are enforced on three separate meters: requests per minute, input tokens per minute, and output tokens per minute. You get a 429 when any one of them runs out. The fix is to read the retry-after header, back off with randomized jitter, and budget by tokens rather than by request count, because payload size is what actually moves your ceiling.

TL;DR

• You are metered on requests per minute, input tokens per minute, and output tokens per minute. The tightest of the three is the one that throttles you.
• Honor retry-after when the provider sends it. When it does not, use exponential backoff with full jitter and cap at four or five attempts.
• Jitter is the load-bearing part. Fixed backoff makes every throttled worker retry in the same instant and rebuilds the burst.
• Retry 429, 500, and 529. Never retry 400, 401, 403, or 404. Retrying deterministic failures spends real budget on requests that cannot succeed.
• Move the ceiling instead of fighting it: prompt caching, the batch API, and a cheaper model tier for the easy work.

What a 429 is actually telling you

A 429 is not "you sent too many requests." It is "one of your three meters hit zero." Requests per minute, input tokens per minute, and output tokens per minute are tracked independently, and the response tells you which one ran out if you bother to read the headers.

This is where most teams misdiagnose. A dashboard shows request volume, request volume looks fine, and the throttling is blamed on the provider. Meanwhile the workload changed last week to include a 40,000 token document in every prompt, and four requests per second is now burning more input token budget per minute than the previous version burned in an hour. The request counter never moved. The ceiling did.

The meters refill continuously rather than resetting on the minute boundary. That sounds like a detail and it changes your strategy: a burst that spends the whole minute's allocation in two seconds does not wait 58 seconds for a reset, it waits for the bucket to refill at a steady rate. Smoothing your send rate genuinely buys throughput. Bursting and then sleeping does not.

Honor retry-after, then back off with jitter

When a 429 includes a retry-after header, use it. That is the provider telling you exactly how long until capacity exists. Retrying at 500ms because it feels responsive just spends another request to be told the same thing, and that request counted against your limit.

When there is no header, use exponential backoff with full jitter:

delay = random.uniform(0, min(60, base * (2 ** attempt)))

The random.uniform(0, ...) is doing the real work. Fixed exponential backoff without jitter is a well-known way to make throttling permanent: ten workers get 429'd in the same 50ms window, all compute the same two-second delay, all retry in the same instant, and rebuild the exact burst that triggered the limit. The queue oscillates instead of draining.

Cap total attempts at four or five. Past that you are not waiting out a spike, you are holding a request that has already missed the deadline the user cared about. Check the official SDK before you write any of this: most of them already retry 408, 409, 429, and 5xx with backoff, default to two retries, and expose a max_retries setting. Wrapping your own retry loop around an SDK that already retries gives you attempts multiplied, not attempts added, and a wall-clock timeout of timeout × (max_retries + 1).

Retry these, not those

The split is deterministic versus transient.

Retry: 429 rate limited, 500 internal error, 529 overloaded, and connection or read timeouts. These reflect state that changes on its own.

Do not retry: 400 malformed request, 401 bad key, 403 no permission, 404 wrong model ID. These are the same on attempt five. Retrying them is worse than useless because each attempt still counts against your request limit. A common way a small bug becomes a throttling incident: a bad model ID ships, every request 404s, the retry wrapper hammers it five times, and now legitimate traffic is competing with a retry storm of requests that were never going to succeed.

Treat 429 and 529 as different diagnoses even though the retry mechanics are identical. A 429 says you are over your allocation, and the fix is yours: slow down, batch, or raise the tier. A 529 says the provider is saturated and your allocation is fine, so the fix is patience plus a fallback. A less-loaded model tier often clears instantly when the flagship one is congested.

Budget by tokens, not by requests

Client-side rate limiting is worth building, and the mistake is building it around request count. Requests per second is the easy number to enforce and the wrong one to enforce, because it treats a 200 token classification and a 40,000 token document summary as equivalent.

Estimate the token cost before you send. Providers expose a token counting endpoint; call it, or keep a cheap local approximation, and run a token bucket that refills at your actual per-minute allocation with the ceiling set at 80 to 85 percent of the real limit. The gap is your headroom for estimation error and for the traffic you do not control.

This flips throttling from an error you handle into a queue you own. A queue that is 200 items deep is a thing you can reason about, prioritize, and shed from. A retry storm is not. And once you have a queue, the interesting decisions become available: interactive requests jump it, background jobs wait, and anything past its deadline gets dropped rather than sent.

That last one is the discipline most teams skip. If a request has been queued for 30 seconds and the user is looking at a page that gave up at 10, sending it now spends budget on a response nobody will read. Drop it, return a degraded result, and keep the capacity for the request that just arrived.

Move the ceiling instead of fighting it

Everything above is reactive. Four things raise the ceiling itself, roughly in order of leverage.

Prompt caching. Cache reads cost about a tenth of base input price, so caching a large shared system prompt buys back real input-token-per-minute headroom, not just money. The catch is that caching is a prefix match: one interpolated timestamp near the top of the prompt invalidates everything after it. Check that your cache read counter is actually nonzero before assuming you have the headroom.

The batch API. Half price, a separate processing pool, and results typically inside an hour. Anything without a human waiting on it belongs there: nightly enrichment, backfills, evaluation runs. Moving those off your live path is often the single biggest reduction in daytime throttling, and it is a change to where the work runs, not to how it works.

Model tiering. Route classification, extraction, and routing decisions to a smaller, faster model and keep the flagship budget for work where the model choice changes the answer. Each tier has its own limits, so this splits your load across pools instead of concentrating it.

Concurrency limits per workload. A background job with generous retry settings will happily starve your interactive path while its own metrics look perfectly healthy. Give each caller a concurrency cap and a priority, so the job that can wait actually waits. If you are structuring agent work into repeatable loops rather than one-off calls, the same boundaries fall out naturally. That pattern is written up in Write Loops Not Prompts.

What to measure

Log the endpoint, model, retry-after value, attempt number, elapsed time since the first attempt, and estimated token count. Then watch two derived numbers.

Throttle rate by workload tells you who is eating the budget, and it is rarely who people assume. Time-to-success after first throttle tells you whether your backoff is tuned: a median under five seconds means you are absorbing spikes correctly, and a median around thirty seconds means requests are sitting in a queue long past the point the user gave up, which is a shedding problem dressed up as a retry problem.

If you want more of these breakdowns as they go up, they go out first to the newsletter.

The bottom line

Rate limits are a capacity planning problem, not an error handling problem. The error handling is twenty lines: honor retry-after, back off with full jitter, cap at five attempts, and never retry a 4xx that is not a 429. The capacity planning is where the wins are: know which of the three meters you are actually hitting, budget by tokens instead of requests, move asynchronous work to the batch API, and cache the prefix you send a thousand times a day.

Do those and 429s stop being an incident. They become a number on a dashboard that stays under one percent, and the retry code you wrote almost never runs.