Back to Blog
September 27, 2026Feature

System Software Bugs and More Accurate Rust Analysis

Finding bugs in GCC, NTFS3, vLLM, and Redis, and improving Rust call graphs with rust-analyzer.

FM-Agent detected 10 bugs across GCC, the NTFS3 filesystem driver, vLLM, and Redis. As of this update, five had been confirmed by their respective projects, including a GCC bug that had already been fixed.

These large systems contain complex optimization and implementation logic, and their correctness affects many applications. The findings include two bugs in GCC, two in NTFS3, three in vLLM, and three in Redis.

Results table: GCC 2 bugs, NTFS3 2, vLLM 3, and Redis 3
Bug counts, examples, and consequences across four systems.

A GCC Optimization That Introduces Overflow

Under -O2, GCC can simplify (X + M*N) / N to X/N + M without checking whether the newly introduced quotient is representable. When X = INT_MIN and N = -1, the original expression can be fully defined, with all intermediate results representable and a nonzero divisor, while the rewritten expression evaluates the overflowing INT_MIN / -1.

On x86_64, the generated idivl instruction can trigger a SIGFPE crash. This combination of conditions escaped routine testing. The report was submitted to GCC, confirmed, and fixed.

GCC bug 127406 report with a reproducer for division overflow introduced by optimization
The GCC report and reproducing test case.
GCC commit restricting the division optimization to fix bug 127406
The upstream fix adds checks and a regression test.

Two NTFS3 Translation Bugs

Writable directories lose permissions after remounting

NTFS3 translates between POSIX permissions and NTFS attributes. When creating a child directory, it inherits the parent's read-only bit in the on-disk fa attributes. The driver sets that bit when the requested mode has no write permissions, but fails to clear it when write permissions are requested.

A writable child under a read-only parent therefore receives an incorrect on-disk attribute. Its in-memory i_mode is initially correct, masking the bug. After remounting, the persisted read-only bit removes write permissions, changing 0755 to 0555; a subsequent chmod cannot repair the issue.

Symbolic-link targets are altered

NTFS3 translates Linux and Windows path separators by replacing every / with \ when creating a symbolic link, and every \ with / when reading it. A valid target containing a literal backslash, such as a\b, is consequently read as a/b, corrupting the target path.

More Accurate Rust Call Graphs with rust-analyzer

FM-Agent previously used CodeGraph to identify Rust calls through syntax analysis and heuristic name matching. When several types define init(), calls such as self.init(), obj.field.init(), and make_list().init() require field and return-type information to resolve the correct target. Lost context can associate a call with an unrelated method, undermining specification generation and verification.

The new rust-analyzer integration uses type inference and name resolution to improve accuracy. FM-Agent reads its exported index and maps symbol references and definitions into the existing function identifiers and call-graph format. Future integrations will bring specialized analysis tools to other languages.

Thanks to Yuexuan Zhang, Qiaoyi Zheng, and Wei Huang of Shanghai Jiao Tong University for the system-software findings, and to Jiahe Mao of Nanjing University and Yucong Dong of Shanghai Jiao Tong University for the rust-analyzer integration.