What is rate limiting and what happens when you hit it?
Rate limiting is a provider capping how many requests you may send in a window. Past the cap it refuses, usually with a 429. What happens next is a design decision: a system either queues, retries, moves to another key, falls back to another provider, or drops the work and reports success anyway.
The provider says no. What happens next is your architecture.
Why it matters to you
Almost every AI system meets this, because model providers price on it. The common response is to upgrade the plan, which works until the next ceiling and leaves the behaviour unchanged.
The structural answer is a pool with a policy. Mooney runs three Groq keys capped at 28 requests a minute each against a provider ceiling of 30, with a rolling 60 second window per key, a 60 second cooldown on any 429, 5xx or throw, round-robin to the least loaded healthy key, and a log line per hop naming the key and the reason it moved. Two Deepgram keys carry roughly 1.4 million voice turns a month, which was audited into a headroom figure of 1,500 to 2,000 daily active users rather than left as a guess.
The log line is the part people skip and the part that pays. Without it, a system that silently degrades to its slowest provider looks identical to one that is fine.
Where this turns up in something OCTYN runs
Each line carries the date it was recorded and where it came from.
- 2026-04-25
3 Groq keys at 28 requests per minute each against a ceiling of 30, with cooldown and least-loaded routing
docs/API_FALLBACK_AUDIT.md, VET repoMooney → - 2026-04-25
Two Deepgram keys audited to roughly 1.4M voice turns a month, giving 1,500 to 2,000 daily active users of headroom
docs/API_FALLBACK_AUDIT.md, VET repoMooney →
If a proposal used this word and did not explain it, ask.
Book a call →