CAISI (US official AI evaluation and standards agency) published a report with a blunt core conclusion: DeepSeek V4Pro is equivalent to GPT-5 released last August, about 8 months behind US frontier models.
Parameters aren't worse. Benchmarks aren't worse. So where does the gap come from?
The report's answer is clear: real-world practice.
The benchmark vs. reality gap
CAISI's logic isn't hard to understand. Benchmarks are standardized — questions and scoring criteria are public. DeepSeek V4Pro's scores on MMLU, GSM8K, and SWE-bench can indeed go head-to-head with GPT-5.
But benchmarks aren't real-world practice. Real-world scenarios have dimensions that benchmark tests don't capture:
Tool call stability. In actual Agent workflows, models need to continuously call multiple APIs, handle errors, retry, and fall back. Benchmarks usually test single-round call accuracy, not long-chain stability.
Context utilization. Giving a model a 128K context window and getting it to effectively extract key information within 128K context are two different things. CAISI found that in real document processing tasks, DeepSeek V4Pro's long-context information retrieval efficiency is lower than GPT-5 of the same period.
Multi-turn conversation consistency. In complex conversations over 20 rounds, DeepSeek V4Pro is more prone to contradictions or forgetting early information.
These gaps don't show up in benchmarks but are obvious in actual use.
Where does the "8 months" number come from
CAISI didn't provide a precise formula. But from the description, its benchmarking method maps DeepSeek V4Pro's capabilities onto the US model timeline — meaning DeepSeek V4Pro's current comprehensive ability roughly matches GPT-5's level at its August 2025 release.
This benchmarking has several assumptions:
- US model capabilities progress at a predictable pace
- There's a stable mapping between benchmark and practical capabilities
- The 8-month gap is a comprehensive capability gap, not a single benchmark gap
These assumptions are debatable. But as a government agency evaluation framework, it at least provides a discussable baseline.
Is this judgment fair
Honestly, there are biased parts and reasonable parts.
The reasonable part: the practical gap really exists. DeepSeek's advantage is mainly cost — API prices are a fraction of US models. But if actual usability is worse, cheapness isn't as meaningful.
The biased part: CAISI's evaluation framework naturally favors the US model ecosystem. Evaluation task design, tool call interface definitions, even prompt language style, are all based on US model interaction conventions. A different evaluation framework might yield different results.
Additionally, "8 months" is an instantaneous snapshot. DeepSeek iterates fast — if V4Pro continues optimizing tool calling and long context capabilities in coming months, this gap may be shrinking.
Community reaction
Chinese community reactions are divided. Some think CAISI's conclusion is objective — benchmarks indeed don't represent everything, and practical gaps need to be faced. Others think this is "American institutions scoring American models," with limited credibility.
English community generally considers the report confirms their intuition: DeepSeek has high cost-performance but still needs to catch up on production environment stability.
My take
The report's biggest value isn't the "8 months" number itself, but that it points out a problem many overlook: the gap between benchmarks and real-world usage is widening.
As Agent workflows grow increasingly complex, single benchmark scores explain less and less. Models need to pass on tool calling, long context, multi-turn consistency, error recovery, and other dimensions simultaneously to be truly usable in production.
If DeepSeek wants to compete with US frontier models in production environments, its next optimization target isn't benchmarks — it's these "benchmarks can't measure but users can feel" capabilities.
Main sources: