How We Handle 500+ Concurrent Users: A Deep Dive Into Our Base Infrastructure
Short answer
We load-tested our AI platform with k6 from 10 to 500 concurrent users. Three failures appeared — WebSocket connection exhaustion, Redis connection timeouts and LLM API rate limits — and we fixed them with connection pooling, a load balancer, backpressure queues and automatic failover between model providers.
Key takeaways
- Load-test the full user journey, including streaming and model calls, not just HTTP endpoints.
- Connection pools and backpressure are the usual fixes for failures under concurrency.
- Multi-provider LLM failover protects you from rate limits at peak load.
When we say we've tested at 500+ concurrent users, we don't mean we ran a quick Locust script and called it done. We mean we systematically broke the system, identified every failure point, and fixed them before shipping.
Here's what that actually looked like.
The Architecture
Our Base Infrastructure runs on a microservices pattern. An API gateway handles authentication and routing. Behind it sit three core services: a transcription service (Whisper/Deepgram), an AI orchestration service (model-agnostic, routes to Claude or OpenAI based on task type), and a streaming service that manages WebSocket connections and SSE streams.
State lives in PostgreSQL. Session data lives in Redis with TTLs. Inter-service communication goes through a Redis Streams message queue for async operations and direct HTTP for synchronous ones.
The Load Test
We used k6 for load testing. The test scenario: a user connects via WebSocket, sends a voice message (simulated as a binary payload), receives a streaming text response, and disconnects. We ramped from 10 to 500 concurrent virtual users over 5 minutes, held at 500 for 10 minutes, then ramped down.
Failure mode 1: WebSocket connection pool exhaustion. At around 300 concurrent users, the streaming service started refusing connections. Root cause: the default Node.js HTTP server backlog was too small. Fix: explicit connection pool configuration and a load balancer in front of the streaming service.
Failure mode 2: Redis connection timeouts. At peak load, session reads were timing out. Root cause: a single Redis connection being shared across async handlers. Fix: a proper connection pool with min/max bounds and timeout configuration.
Failure mode 3: AI API rate limits. At 500 concurrent users, we hit OpenAI's rate limits. Fix: a request queue with backpressure, exponential backoff, and automatic failover to Claude when OpenAI is throttled.
The Results
After these fixes: 500 concurrent users, p99 latency under 2.4 seconds for a full round trip (voice-in to text-out), zero connection drops, zero dropped messages. Horizontal scaling via Kubernetes adds capacity linearly when needed.
We publish these numbers because they're the only credible evidence that a system works. Architecture diagrams don't prove anything. Load test results do.
Frequently asked questions
How do you load test an LLM application?
Script the real user journey — connect, send input, receive the streamed response, disconnect — with a tool like k6, ramp concurrency gradually, and hold at peak while measuring latency percentiles, errors and dropped connections.
What breaks first when an AI app scales?
In our testing it was connection handling: WebSocket backlogs, shared database or Redis connections, and then provider rate limits on the LLM API.
How do you handle LLM API rate limits?
Queue requests with backpressure, retry with exponential backoff, and fail over to a second provider when one is throttled.
Why publish load test results?
Numbers from a real test are better evidence that a system works than architecture diagrams, and they show buyers exactly what was tested.