We compared FM-Agent and Claude Ultrareview on their ability to detect correctness bugs in real code. Ultrareview found 6 buggy functions; FM-Agent with Chinese models found 24-33, while FM-Agent with Claude Opus 5 found 48.
Claude Ultrareview is Anthropic's code-review feature for reviewing a pull request or an entire code module. From Claude Code, the ultrareview command uploads the code and starts multiple review agents in parallel. Their findings are consolidated and prioritized in a final report.
Test Setup
We used Claude's C Compiler, a roughly 150,000-line compiler generated from scratch by Claude in February 2026. Although FM-Agent supports codebases above 100,000 lines, Ultrareview currently has an 8,000-line limit. We therefore tested both tools on the compiler's 6,455-line Rust Parser module.
FM-Agent was evaluated with DeepSeek V4 Flash, GLM 5.2, Kimi K3, and Claude Opus 5. Using manual inspection, FM-Agent, coding agents, and differential testing together, we confirmed that the Parser contains 76 buggy functions. Recall is calculated as the number of buggy functions found divided by 76.
Results
Claude Ultrareview found 6 buggy functions, for 7.9% recall. FM-Agent found 24 with GLM 5.2, 27 with Kimi K3, and 33 with DeepSeek V4 Flash. With Claude Opus 5, FM-Agent found 48 buggy functions and reached 63.2% recall.
Why FM-Agent Finds More Bugs
Every tested FM-Agent configuration, including those using Chinese models, detected more buggy functions than Claude Ultrareview. There are both technical and engineering reasons behind the difference.
Technically, FM-Agent does not simply ask a model to read code freely and provide review comments. It deeply integrates formal-verification ideas with LLMs, including correctness-specification generation and explicit correctness reasoning. On the engineering side, FM-Agent has been continuously improved through five biweekly releases. Enhancements such as CodeGraph-based function call graphs and custom domain knowledge help it understand intent and analyze complex codebases more precisely.
Conclusion
In this experiment, FM-Agent with both Chinese models and Claude Opus 5 found substantially more buggy functions than Claude Ultrareview. The result demonstrates the potential of combining formal-verification techniques deeply with large language models for practical bug detection.