We asked FM-Agent to verify its own correctness. The experiment found 19 real bugs, which were fixed, and revealed why the same code can receive very different judgments from different models.
The Recursion Trap
A validator-generated counterexample can launch FM-Agent again. During the first run, incremental validation reused the current repository as a testcase, causing a nested run to fall back to full validation and clear intermediate results. A self-validation guard was added to prevent this recursion.
Real Bugs Found
Manual review confirmed 19 bugs across configuration, environment checks, incremental updates, call graphs, and the Reasoner. Examples include a generation failure hidden behind the same None value as “no update”; partial .info.json results overwriting existing call-graph information; and braces inside multiline comments being counted while parsing C, C++, and Java functions.
Where False Positives Come From
False positives mainly arise from inaccurate generated specifications or validator inputs that cannot occur in the real call path. A model may turn an implementation preference into a mandatory precondition, or a mock may pass an internal function data that upstream validation would always reject.
Different Models, Different Results
On the same 457-function baseline, DeepSeek V4 Pro reported 312 mismatches while Qwen 3.8 Max reported 127. The models jointly reported 108 functions, but only 16 functions fully overlapped after classifying concrete defects. Model choice affects both the number and the type of issues discovered.
Self-validation is therefore more than a test of implementation: it tests whether specifications are grounded, counterexamples are reachable, and conclusions are reliable enough for regression and maintenance.