DeepSeek V4 Flash. Keln vs DeepSeek vs OpenRouter. How we caught a provider billing double.
For 12 hours we sent the same 720 reasoning puzzles, one a minute: to Keln, to OpenRouter (:nitro) and to DeepSeek's official API. One of them took almost 5 times longer to answer. The cause was one provider. Keln had already flagged it and was already routing around it.
Data from 20:05 UTC on 15 September to 08:04 UTC on 16 September 2026. Every number below comes from the run's per-request ledger. Nothing is modelled.
What we ran: DeepSeek V4 Flash on 3 routes
We sent the same 720 reasoning puzzles to 3 routes to the same model: Keln's public API, OpenRouter with the :nitro suffix (its throughput-sorted routing), and DeepSeek's official API. The 24 puzzles rotated through the run, with the same prompt and reasoning at the deepest setting the model offers. Every provider got the same requests at the same moments, so a slow one could not fall behind and handle fewer, and no one got an easier stretch of the run. The full method is at the end.
By the end of the run, Keln's median answer had taken 27 seconds and DeepSeek's 41. OpenRouter's had taken 129. That is almost 5 times slower for the same work.
A provider can be slow in 2 obvious places:
- Starting the answer
- Streaming it.
So we checked both.
Was it slow to start?
Reasoning tokens count as the first token.
At the median, all 3 were close. Keln's first token arrived in 788 ms, DeepSeek's in 766, OpenRouter's in 666. OpenRouter was the fastest of the 3 to start.
The medians are close, but OpenRouter's line looks different from the other 2. It went above 1 second in 11 of the 48 quarter-hours, and jumped by more than 300 ms from one quarter-hour to the next 12 times. That is because a different upstream provider can serve each request. Keln jumped like that twice and never went above 1 second. DeepSeek never did either.
At the 95th percentile the gap is clearer: Keln 1,102 ms, DeepSeek 1,006 ms, OpenRouter 2,898 ms. So Keln is 2.6× faster than OpenRouter at the tail, and about the same as DeepSeek.
An unpredictable time to first token feels unreliable to a customer. But it cannot add 100 seconds to an answer, so the start is not the cause.
Was it the streaming speed?
For most of the run, Keln's route decoded at 350 to 470 tokens a second. DeepSeek's official API held steady at about 265. OpenRouter's throughput-sorted pick ran at 120 to 200.
This is a real difference. Keln decodes 2.7× faster than OpenRouter (384 vs 144 tok/s) and 1.5× faster than DeepSeek's official API (264).
But a decode that is 2.7× slower cannot make an answer 4.8× slower. If all 3 start at about the same time, and one streams less than 3 times slower, the only thing left that can explain the gap is how many tokens each one streams.
We found the answer. It was the number of tokens
That is the missing 100 seconds. For the same prompts at the same effort setting, OpenRouter's route produced a median of 18,109 reasoning tokens per answer. DeepSeek's official API produced 9,936. Keln's route produced 8,669. OpenRouter thought 2.1× longer than Keln for the same answer, and the ratio held across all 12 hours without drifting. Every one of those extra tokens takes time to stream, and every one is billed.
Was something wrong with our request?
The request body is byte-identical across the 3 columns, so the prompt itself cannot be the cause. We replayed it by hand anyway: with and without the nonce, at 3 temperatures, with the effort setting in every form a provider accepts. Nothing changed. Whatever doubles the thinking happens on the provider's side.
OpenRouter reports which upstream served each request, so we split its column by provider. 432 of its 720 requests went to one provider. Call it Provider X. Provider X thinks twice as long: a median of 18,076 reasoning tokens per answer, against 9,064 on Keln's main route and 9,936 on DeepSeek's official API.
We had already flagged Provider X
Keln sells each model from many providers at one price. That only works if every provider really serves the model it claims, at the quality it claims, so we check. Every provider gets the same hidden test questions with known answers, and we compare the numbers each one produces for the same input with the other providers serving the same weights. A provider whose numbers do not match the others is flagged, and a flagged provider is taken out of routing.
Provider X was flagged before this run started. It had the lowest score on the test questions of all 22 providers serving this model, and its numbers did not match its peers. That is why Keln sent it 2 requests out of 720, and OpenRouter's throughput-ranked routing sent it 432.
We cannot say what Provider X does differently, and for routing we do not need to. It bills twice the tokens for the same answer, and those tokens are why OpenRouter's route took almost 5 times as long to finish. You wait for the answer, and you pay for every token before it.
What an answer cost
The price per token matters less than how many tokens an answer takes.
| Keln | OpenRouter :nitro | DeepSeek official | |
|---|---|---|---|
| List price per 1M tokens, input / output | $0.22 / $0.66One rate | $0.25 to $1.32Output only, depends on the route it picks | $0.15 / $0.60Off-peak$0.30 / $1.20Peak, Mon–Fri 01:00–04:00 and 06:00–10:00 UTC |
| Tokens billed per answer, median | 9,827 | 19,412 | 10,433 |
| Cost per answer, mean | $0.0075 | $0.0083 | $0.0092$0.0064 off-peak, $0.0130 peak |
| How cost is known | Reported in usage.cost | Reported in usage.cost | Tokens × DeepSeek's published rate for the hour |
Keln's answers were the cheapest of the 3, and its price is the only one that does not move: $0.22 / $0.66 per million tokens, at any hour, whichever provider serves the request. Its route finished in a median of 9,827 output tokens, so an answer cost $0.0075: 10% less than OpenRouter and 19% less than DeepSeek's official API.
OpenRouter's price depends on the route it picks. Its routes charged $0.25 to $0.26 per million output tokens for most of the run, and it still paid more per answer, $0.0083, because each answer took 19,412 tokens.
DeepSeek's price depends on the hour. Its official API is the cheapest per token off-peak and the most expensive per answer at peak: $0.0064 for the 411 off-peak requests, $0.0130 for the 303 peak ones. The price doubles on weekday mornings UTC, and 5 of the run's 12 hours fell in that window.
Reliability, including our own misses
| Keln | OpenRouter :nitro | DeepSeek official (V4.1-Flash) | |
|---|---|---|---|
| Requests | 720 | 720 | 720 |
| Answered | 71799.6% | 70397.6% | 71499.2% |
| First token, p50 / p95 | 788 / 1,102 ms | 666 / 2,898 ms | 766 / 1,006 ms |
| End to end, p50 / p95 | 27 / 85 s | 129 / 273 s | 41 / 59 s |
| Slowest answered request | 280 s | 502 s | 87 s |
| Decode speed, p50 / p95 | 384 / 471 tok/s | 144 / 318 tok/s | 264 / 292 tok/s |
| Reasoning tokens, p50 | 8,669 | 18,109 | 9,936 |
| Output tokens, p50, reasoning + answer | 9,827 | 19,412 | 10,433 |
| Cost per answered request, mean | $0.0075 | $0.0083 | $0.0092 |
| 12-hour spend | $5.39 | $5.86 | $6.60 |
Of OpenRouter's 14 empty streams, 11 came from one provider, not Provider X. Keln sent the same provider 97 requests and returned none empty: when a stream stalls, our edge reroutes it to another provider.
All of Keln's requests went to zero-data-retention providers, the same constraint we set on the OpenRouter column.
In short
We sent the same 720 requests for the same model through 3 routes. The differences were in which provider answered and how many tokens it billed. OpenRouter's routing sent 432 of its requests to a provider that used twice the reasoning tokens for the same answer and charged for all of them. Keln's checks had flagged that provider before the run, so Keln sent it 2. The route that used fewer tokens was cheaper per answer, even with a higher list price.
Against DeepSeek's official API, Keln was faster at the median, slower at the tail and cheaper per answer. The official API does not guarantee zero data retention. Every Keln request went to a provider that does.
Method notes
720 requests per column, one a minute for 12 hours, each launched on schedule whether or not the previous one had returned. Same 24 constraint-satisfaction puzzles in rotation, same nonce-prefixed prompt, temperature 1.0, max_tokens 30000, reasoning_effort: max in the form each API expects, streaming, 600-second limit per request. The columns: Keln's public API; OpenRouter with the :nitro suffix and provider.zdr set, so it only uses zero-data-retention providers, the only kind Keln routes to; and DeepSeek's official API. One caveat on that last column: DeepSeek retired the V4-Flash 0731 build on 10 September and now serves V4.1-Flash under the rolling deepseek-flash id, so it is a newer version of the model, not the same weights. We kept it as the closest thing to the maker's own number. The window runs 13:00 to 01:00 Pacific and 04:00 to 16:00 Beijing, so it covers a US afternoon and evening and most of a Chinese working day.
Time to first token counts the first reasoning token, not the first answer token; on a reasoning model, a first-answer-token measure mostly measures decode speed. Decode speed is output tokens per second after the first token. Costs for Keln and OpenRouter are what each reported in usage.cost; DeepSeek returns no cost, so its rows are tokens multiplied by its published rate for the hour the request launched. "Answered" means the request produced a final answer; a request that spent the whole 30k budget thinking counts as a failure in every column. One Keln request at 22:11 UTC counts as answered although the benchmark client lost it: AWS reclaimed the spot instance under one of our edge pods during the run, the pod drained its streams cleanly, and only the client's idle connection was lost. Production edge nodes are on-demand, so that failure mode does not apply there. We note the adjustment here rather than leave it out. We will publish the ledger, the scripts behind these charts and the per-request data with the next run.