AI Inference Benchmark: Keln vs Fireworks, Together AI, Baseten, Modal, DeepInfra & Z.ai
We tested DeepSeek V4.1 Flash 1017 and GLM-5.3 Flash across six inference providers and Keln, using real agent sessions with tool calling and maximum reasoning enabled. Each provider ran the same conversations, in the same order, with identical pauses between turns. You can find the full methodology at the end.
Benchmark results at a glance
DeepSeek V4.1 Flash 1017
GLM-5.3 Flash
Turns answered
Most sides answered everything we sent on both models. Baseten refused about half the DeepSeek turns with rate-limit errors: these prompts are tens of thousands of tokens each, and just 2 sessions at once put us over our account's limit. On GLM-5.3 Flash the breaks kept the load lower, so Baseten refused fewer turns there.
Modal cut 1 stream on DeepSeek, while on all the other providers answers ended normally. An app calling Modal directly gets the cut stream and has to resend the turn itself, because there is no second provider to send it to.
| Turns answered | DeepSeek V4.1 Flash 1017 | GLM-5.3 Flash |
|---|---|---|
| Keln | 1,440 of 1,440 | 240 of 240 |
| Fireworks | 1,440 of 1,440 | 240 of 240 |
| Together | 1,440 of 1,440 | – |
| Baseten | 692 of 1,440748 rate limit (429) | 217 of 24023 rate limit (429) |
| Modal | 1,439 of 1,4401 stream cut by the provider | 240 of 240 |
| DeepInfra | 1,440 of 1,440 | 240 of 240 |
| Z.ai | – | 240 of 240 |
Cache hit rate
Running inference across multiple providers can hurt prompt caching when conversations move between them. On DeepSeek, Keln switched providers on just 1 of 1,440 turns, keeping its cache hit rate at 94.3%, close to the best direct provider at 95%. On GLM-5.3 Flash, Keln reached 90.5%, the highest of the 6 sides.
Fireworks had the most cold turns on DeepSeek: 165, compared with 11 on Keln. Each cold turn meant reading most of the prompt again instead of using cached tokens.
| Cache hit rate | DeepSeek V4.1 Flash 1017 | GLM-5.3 Flash |
|---|---|---|
| Keln | 94.3%11 cold turns | 90.5%11 cold turns |
| Fireworks | 85.6%165 cold turns | 87.1%19 cold turns |
| Together | 94.7%5 cold turns | – |
| Baseten | 89.9%9 cold turns | 80.4%37 cold turns |
| Modal | 93.2%31 cold turns | 78.8%36 cold turns |
| DeepInfra | 95.0%3 cold turns | 88.7%13 cold turns |
| Z.ai | – | 85.9%22 cold turns |
Based on each side's usage. A cold turn is any turn after the first with under 10% cached.
Cache after a break
We also tested how well providers retain cached prompts when a conversation goes idle. On GLM-5.3 Flash, we added breaks of 1–20 minutes to 15% of turns, using the same timing across all providers.
| After a break of | 1 to 5 min | 5 to 10 min | 10 to 15 min | 15 to 20 min | Turns with no break | Whole run |
|---|---|---|---|---|---|---|
| Keln | 77.1%6 turns | 91.7%8 turns | 78.2%11 turns | 63.0%10 turns | 94.0% | 90.5% |
| DeepInfra | 97.5%6 turns | 80.2%8 turns | 59.3%11 turns | 38.4%10 turns | 94.0% | 88.7% |
| Z.ai | 83.3%6 turns | 81.0%8 turns | 33.6%11 turns | 16.5%10 turns | 93.8% | 85.9% |
| Baseten | 69.3%6 turns | 42.6%8 turns | 35.5%11 turns | 0.0%10 turns | 91.6% | 80.4% |
| Modal | 49.1%6 turns | 19.7%8 turns | 0.0%11 turns | 0.0%10 turns | 91.9% | 78.8% |
| Fireworks | 97.8%6 turns | 81.0%8 turns | 22.3%11 turns | 0.0%10 turns | 96.3% | 87.1% |
After a 15–20 minute break, Keln still had 63.0% of the prompt cached, compared with 38.4% on DeepInfra and 16.5% on Z.ai. Baseten, Modal and Fireworks had 0%.
Decode speed
Keln had the highest median decode speed on GLM-5.3 Flash at 432 tokens/s, 75% faster than the next fastest provider, Baseten, at 247 tokens/s.
On DeepSeek, Keln and Together were nearly tied at 432 and 437 tokens/s, both well ahead of the other providers.
| Decode speed, median | DeepSeek V4.1 Flash 1017 | GLM-5.3 Flash |
|---|---|---|
| Keln | 432 tok/sfirst token 0.35 / 0.89 s | 432 tok/sfirst token 1.01 / 2.97 s |
| Fireworks | 234 tok/sfirst token 1.21 / 2.45 s | 204 tok/sfirst token 0.79 / 1.66 s |
| Together | 437 tok/sfirst token 0.22 / 0.65 s | – |
| Baseten | 208 tok/sfirst token 0.38 / 1.07 s | 247 tok/sfirst token 0.38 / 1.59 s |
| Modal | 228 tok/sfirst token 0.72 / 1.45 s | 129 tok/sfirst token 0.99 / 2.48 s |
| DeepInfra | 98 tok/sfirst token 1.05 / 3.19 s | 176 tok/sfirst token 1.16 / 5.20 s |
| Z.ai | – | 46 tok/sfirst token 4.98 / 27.11 s |
First token is p50 / p95 and counts the first reasoning token.
Reasoning at max
We set reasoning effort to maximum for every request. All providers returned broadly similar amounts of reasoning tokens, including Keln.
In our previous benchmark, two gateways returned less than half as many reasoning tokens for the same requests. We didn't see that issue here.
Modal doesn't report reasoning tokens for GLM-5.3 Flash, so those results aren't included.
| Reasoning tokens per turn, mean | DeepSeek V4.1 Flash 1017 | GLM-5.3 Flash |
|---|---|---|
| Keln | 51426 turns over 5,000 | 3565 turns over 5,000 |
| Fireworks | 45927 turns over 5,000 | 3385 turns over 5,000 |
| Together | 55035 turns over 5,000 | – |
| Baseten | 69226 turns over 5,000 | 4396 turns over 5,000 |
| Modal | 54533 turns over 5,000 | –not reported |
| DeepInfra | 47323 turns over 5,000 | 2954 turns over 5,000 |
| Z.ai | – | 4958 turns over 5,000 |
Cost
Fireworks was the most expensive on DeepSeek, largely due to its 165 cold turns. It processed 2.5 times as many uncached prompt tokens as Keln.
On GLM-5.3 Flash, Keln was among the more expensive options. Its pricing covers more than model inference: Keln manages capacity across providers, monitors performance and output quality, and handles failover and retries. You get all of that through one API, without managing provider accounts, rate limits or caching yourself.
| Cost per 1,000 turns | DeepSeek V4.1 Flash 1017 | GLM-5.3 Flash |
|---|---|---|
| Keln | $2.06$2.97 for 1,440 turns | $2.55$0.61 for 240 turns |
| Fireworks | $2.86$4.12 for 1,440 turns | $2.06$0.49 for 240 turns |
| Together | $1.92$2.76 for 1,440 turns | – |
| Baseten | $2.63$1.82 for 692 turns | $2.33$0.51 for 217 turns |
| Modal | $1.72$2.47 for 1,439 turns | $2.45$0.59 for 240 turns |
| DeepInfra | $1.09$1.57 for 1,440 turns | $1.95$0.47 for 240 turns |
| Z.ai | – | $2.15$0.52 for 240 turns |
Baseten's cost per 1,000 turns counts only the turns it answered.
What we learned
Going directly to an inference provider can be cheaper, and performance can be excellent when capacity is available. But you're limited to that provider's infrastructure and your account's rate limits.
We saw those limits in this benchmark. Baseten rejected requests due to rate limits, Modal cut off a response mid-stream, and Fireworks had to process previously cached prompts again.
Keln takes a different approach. It runs inference across multiple providers, managing capacity, performance, and reliability behind one API. In this benchmark, Keln completed every turn, matched or exceeded the fastest direct providers on decode speed, and retained more cached tokens after longer breaks.
Keln wasn't the cheapest option, but the price includes the infrastructure management, failover, and retries you'd otherwise have to handle yourself.
We post benchmarks with the method included. The previous one compared OpenRouter, Vercel, Ramp and Keln on DeepSeek V4.1 Flash for 24 hours: read it here.
How the benchmark ran
We used 8 sessions from the public armand0e Claude Code corpus on Hugging Face, replayed with native tool calls. Each recorded assistant turn was sent as a new request, with the original tool results appended, so conversations grew as they did in the original sessions.
Each run used up to 30 rounds per session, with 2 concurrent sessions per side and fresh salts and conversation IDs for each pass. The median turn contained 40,665 tokens on DeepSeek and 38,578 on GLM-5.3 Flash.
All requests used reasoning_effort: max in the format supported by each API, temperature 1, streaming enabled, and no max_tokens limit.
Pauses between turns followed the original sessions, capped at 60 seconds, with additional breaks to test caching. On DeepSeek, 5% of turns had breaks of 30 seconds to 2 minutes. On GLM-5.3 Flash, 15% had breaks of 1–20 minutes. We used the same random seed across all sides, so each received the same pauses on the same turns.
The DeepSeek benchmark ran on November 3, 2026, from 02:30 to 08:53 UTC, with 8,640 turns total (1,440 per side). The GLM-5.3 Flash run made 3 passes in 14 hours on November 4; a pass replays the 8 sessions once. This post reports the third pass, from 03:45 to 08:32 UTC, with 1,440 turns total (roughly 240 per side).
We tested Keln through its public API. For direct providers, we used Keln's own accounts and each provider's model identifiers, reasoning settings, and per-conversation cache hints. These included the documented user field or, for Baseten, the x-session-affinity header. Fireworks served DeepSeek through its standard endpoint.
Cache hit rate is the percentage of prompt tokens reported as cached in each provider's usage data. A provider switch is counted when consecutive turns in the same conversation run on different providers. Decode speed is calculated as output tokens divided by generation time for each turn.
We calculated costs using each provider's published model prices as of November 4, 2026, applied to its reported usage. Fireworks offers two DeepSeek pricing tiers; we used the standard tier corresponding to the endpoint tested. Modal shows its rates in its console, so we used the prices shown on our account.
For Keln, we used the November 4 price list: DeepSeek at $0.30 per million input tokens, $0.01 per million cached input tokens, and $1.20 per million output tokens; GLM-5.3 Flash at $0.15, $0.05, and $0.50 respectively. The DeepSeek run was billed at the previous cached-input rate of $0.0108 per million tokens, resulting in $2.09 per 1,000 turns.
Results reflect this workload, our account limits, and the conditions during the test. Performance may vary with other workloads or concurrency levels.
The raw benchmark rows and response bodies are not publicly available.