Industry Rate Limiting & Concurrency Architecture Benchmarks (2026)
Token bucket capacity, burst factors, and HTTP 429 Retry-After thresholds across major API gateways.
| API Archetype | Sustained Limit | Burst Capacity | Storage Engine | Backoff Strategy | Official Docs |
|---|---|---|---|---|---|
| Stripe Payment APIs | 100 req/sec | 200 tokens (2×) | Distributed Redis | Exponential Jitter | Stripe Docs ↗ |
| GitHub REST API | 5,000 req/hour (~1.4 RPS) | Fixed Window | Memcached / Edge | Reset Header Unix TS | GitHub Docs ↗ |
| Cloudflare API Shield | Custom (1k - 50k RPS) | Sliding Window Leaky | Edge Anycast Memory | HTTP 429 Retry-After | Cloudflare Docs ↗ |
Token-Bucket Algorithms & Little's Law Concurrency
Rate limiting is the first line of defense for protecting backend microservices and database pools from cascading failures and traffic surges. The Token Bucket algorithm is the industry standard (employed by Redis, Envoy, NGINX, and Cloudflare) because it naturally permits controlled traffic bursts while strictly enforcing sustained throughput ceilings.
Frequently Asked Questions
How does the token bucket algorithm work?
A token bucket holds up to a maximum capacity of tokens and refills at a steady rate per second. Each API request consumes one token. If the bucket is empty, incoming requests are rejected with HTTP 429 Too Many Requests.
How do you calculate concurrent connections from RPS and latency?
Using Little's Law: Concurrent Connections = Throughput (RPS) × Average Latency (seconds). For example, 500 RPS with a 200ms latency requires 100 concurrent open connections.
Why should clients implement exponential backoff with jitter?
When hundreds of clients retry immediately upon receiving a 429 status code at the same fixed interval, they create a 'thundering herd' that repeatedly knocks over the upstream service. Adding randomized jitter spreads out retry attempts smoothly.