AI gateway benchmark: OpenRouter, Vercel, Ramp and Keln. Comparing cache hit rate, speed, reasoning and cost.
We ran 24 hours of real agent sessions on OpenRouter :nitro, Vercel, Ramp and Keln, on DeepSeek V4.1 Flash. All 4 had ZDR, max reasoning and tool calling on. Every gateway got the same turn at the same moment, 1 turn in 20 paused for 1 to 20 minutes as a real user would. Read the full method.
The run went from 00:25 UTC on 20 October to 00:44 UTC on 21 October 2026.
13,440 turns, 3,360 per gateway.
Cache hit rate: 63% to 94%, depending on the gateway
Cache hit rate over the run: Keln 94%, OpenRouter 88%, Vercel 87%, Ramp 63%. The gap is in turns that went fully cold after the first one.
That depends on how each gateway picks a provider on the second turn. Keln keeps a conversation on the provider that holds its prefix. Vercel pins a conversation to its first provider for 5 minutes. OpenRouter and Ramp pick again on every request.
| Provider, through | Keln | OpenRouter :nitro | Vercel (tps) | Ramp |
|---|---|---|---|---|
| Provider A | 94.0%1,623 turns | 90.2%1,651 turns | 93.1%919 turns | – |
| Provider B | 92.6%352 turns | 87.5%1,634 turns | 84.1%2,349 turns | 63.0%2,955 turns |
| Provider C | 93.9%863 turns | – | – | – |
| Provider D | 94.9%308 turns | –1 turn | – | – |
| Provider E | 94.4%201 turns | – | – | – |
| Provider F | – | – | – | 59.5%308 turns |
| Others | –13 turns on 2 providers | –75 turns on 5 providers | –92 turns on 3 providers | – |
Cache hit rate per provider: cached tokens as a share of prompt tokens on the turns that provider served, through each gateway. Rows with fewer than 100 turns are left blank.
Read across a row: the same provider cached differently depending on the gateway in front of it. Provider B kept 92.6% through Keln, 87.5% through OpenRouter, 84.1% through Vercel and 63.0% through Ramp.
Provider moves ranged from 14 to 500 in 3,360 turns
Ramp moved a conversation to a different provider 500 times in 3,360 turns, OpenRouter 148, Vercel 60, Keln 14. Each move is a cold prompt: tens of thousands of tokens re-read at the uncached price, and a first token seconds later.
Cache after a 10-minute break: 23% to 75%
The longer the pause, the more of the prompt a gateway had to re-read. After a break, the next turn comes from cache only if it goes to the same provider and that provider still has the earlier turns cached. Pauses under a minute made no difference. After 10 minutes Keln still served 75% of the next prompt from cache, OpenRouter 52%, Vercel 25% and Ramp 23%.
After a pause over 10 minutes, how much of the prompt was still in cache depended on the provider that answered.
Reasoning: 403 to 1,033 tokens per turn for the same request
Every turn asked for reasoning at max, the deepest setting DeepSeek V4.1 Flash has. The model is the same and the providers overlap, so the reasoning tokens should match within the noise of temperature 1.
Keln's turns averaged 856 reasoning tokens, OpenRouter's 1,033, Vercel's 403 and Ramp's 420. Vercel and Ramp delivered less than half the reasoning for the same max request. The typical turn is the same everywhere (median 40 tokens on Keln, 42 on OpenRouter, 54 on Vercel, 36 on Ramp). The gap comes from the few turns with thousands of reasoning tokens.
Turns that reasoned past 5,000 tokens: Keln 152, OpenRouter 212, Vercel 36, Ramp 43. On Provider B, the one provider all 4 gateways used, 1 turn in 12 reasoned past 5,000 tokens through OpenRouter and 1 in 15 through Keln; through Vercel or Ramp, 1 in 70. Vercel accepted max on every turn with no warning or error.
You cannot know the model did a third of the thinking. The response comes back successful, and the cost is lower in the end. The wrong answer comes later, on a hard turn. The logs for that turn show a normal request, so the gateway is never suspected. When the same provider behaves differently behind different gateways, the issue is on the gateway side.
This is also why Vercel's bill is the lowest: $9.14, against $10.98 at OpenRouter and $11.26 at Keln. At Keln's reasoning volume, Vercel's own rates come to $10.97. Vercel charged less because it ran less reasoning.
| Route | Turns | Reasoning tokens, median | Reasoning tokens, average |
|---|---|---|---|
| Keln · Provider A | 1,623 | 33 | 872 |
| Keln · Provider C | 863 | 26 | 768 |
| Keln · Provider B | 352 | 66 | 1,203 |
| Keln · Provider D | 308 | 47 | 597 |
| Keln · Provider E | 201 | 64 | 901 |
| OpenRouter · Provider A | 1,651 | 33 | 748 |
| OpenRouter · Provider B | 1,634 | 51 | 1,331 |
| Vercel · Provider B | 2,349 | 57 | 454 |
| Vercel · Provider A | 919 | 50 | 275 |
| Ramp · Provider B | 2,955 | 49 | 464 |
| Ramp · Provider F | 308 | 0 | 0 |
Reasoning tokens as each gateway reported them in usage, per route with at least 200 turns. Ramp reports reasoning in the Responses usage shape; its figures come from the captured response bodies.
All 4 gateways decoded at about 400 tokens a second at the median
At the median all 4 decoded at 390 to 430 tokens a second.
| Keln | OpenRouter :nitro | Vercel (tps) | Ramp | |
|---|---|---|---|---|
| Turns answered | 3,360 of 3,360 | 3,360 of 3,360 | 3,360 of 3,360 | 3,263 of 3,36097 refused: the prepaid wallet ran dry at 23:44 UTC |
| Cache hit rate, whole run | 93.9% | 88.5% | 86.7% | 62.6% |
| Conversations moved | 14 | 148 | 60 | 500 |
| First token, p50 / p95 | 0.45 / 2.23 s | 0.45 / 2.02 s | 0.42 / 1.76 s | 0.45 / 4.48 s |
| Decode speed, p50 | 420 tok/s | 391 tok/s | 429 tok/s | 404 tok/s |
| Reasoning tokens per turn, mean | 856 | 1,033 | 403 | 420 |
| Total bill | $11.26 | $10.98 | $9.14 | $20.20 |
The same 3,360 turns cost $9 to $20 depending on the gateway
The same 3,360 turns cost $9.14 through Vercel, $10.98 through OpenRouter, $11.26 through Keln and $20.20 through Ramp. Ramp's bill is high because of cache misses: it paid for 47.5 million uncached prompt tokens, against 8 million on Keln.
| Keln | OpenRouter :nitro | Vercel (tps) | Ramp | |
|---|---|---|---|---|
| Total bill | $11.26 | $10.98 | $9.14 ($10.97*) | $20.20 ($21.89*) |
Vercel and Ramp delivered less than half the reasoning Keln's turns carried, and billed less for it. With their reasoning set to Keln's volume, at their own rates, Vercel's run costs $10.97 instead of $9.14 and Ramp's $21.89 instead of $20.20.
* With reasoning set to Keln's volume, at their own rates.
Keln re-read 7 million fewer uncached tokens than OpenRouter, 9.5 million fewer than Vercel and 39.5 million fewer than Ramp.
Availability: all 100%
Every gateway answered every turn it was sent. Ramp's prepaid wallet ran dry at 23:44 UTC and refused its last 97 turns for funds, a wallet limit, not a gateway fault; its other numbers stand on the 23 hours it completed.
In short
Behind the same model, the 4 gateways behaved very differently. Keln kept 94% of each prompt in cache and moved a conversation between providers 14 times in 3,360 turns. OpenRouter kept 88% and moved 148 times, Vercel 87% and 60 times, Ramp 63% and 500 times, and each move meant re-reading the whole prompt at full price. Vercel and Ramp returned less than half the reasoning the request asked for, gave no warning, and charged less as a result. Decode speed was about the same everywhere. Ramp paid nearly double because of its cache misses.
We post AI gateway comparison benchmarks, with the method included. The previous one compared OpenRouter, DeepSeek's own API and Keln on DeepSeek V4 Flash for 12 hours: read it here.
Method notes
The 8 sessions come from the public armand0e Claude Code corpus on Hugging Face, replayed with native tool calls: each recorded assistant turn is sent as the next request and the recorded tool results appended, so the conversation grows as it did for the original user. 30 rounds per session, 2 at a time, a fresh salt and fresh conversation ids per pass, 14 passes, 13,440 turns in 24.3 hours; the median turn was 40,700 tokens. reasoning_effort: max in the form each API expects, max_tokens 30000, temperature 1, streaming. 5% of turns waited 1 to 20 minutes, from a seed shared across gateways. Chart time axes leave out the 64 minutes in which no gateway received a turn.
Routes: Keln's public API as sold; OpenRouter with :nitro and provider.zdr: true; Vercel AI Gateway with sort: tps inside an explicit list of its zero-data-retention providers; Ramp Router in the Responses dialect. Cache hit rate is cached prompt tokens over all prompt tokens, as reported in usage. A move is a turn whose provider differs from the previous turn of the same conversation, read from each gateway's provider header or body field. Time to first token counts the first reasoning token. Costs are each gateway's prices for this model, summed from usage. The figures marked * reprice Vercel and Ramp at their own rates with reasoning set to Keln's 856 tokens per turn.
The raw ledger and response bodies are not published. The analysis scripts are in our repository.