What Changes When One Audit Runs Many AI Auditors?

A same-codebase benchmark of Havu, five Web3 security skills, and two GPT models

Most AI-assisted audits begin with a choice: pick one security skill, pick one model, and run it against the code.

That sounds simple, but the choice affects the result. A skill tells the model how to investigate the code. One skill may focus on state changes, another on access control, and another on Solidity-specific mistakes. The model is the engine following those instructions. Change either one and the findings can change.

Havu removes that choice from the start of the audit. It runs several skill and model paths, collects their findings, and merges reports that describe the same underlying bug. The goal is not to produce the longest possible report. It is to cover more real root causes, keep useful differences between audit approaches, and show when another broad run is mostly repeating earlier work.

We tested that idea on one Solidity codebase.

TL;DR

Havu found 32 of the benchmark's 37 root causes. All ten standalone skill runs combined found 20.

The advantage was visible even before pooling runs. The strongest standalone report contained 8 unique root causes. The weakest Havu report contained 9. Every Havu run found more unique root causes than every standalone run in this test.

Havu also reached its practical coverage ceiling quickly. It had found 31 of its final 32 root causes after Run 3, or 96.9%. Run 4 added nothing new. Run 5 added one root cause from 15 raw report entries.

The result is straightforward: combining audit perspectives improved coverage, reduced the risk of choosing the wrong skill or model, and produced a useful signal for when broad reruns should stop.

The four questions

We designed the benchmark around four questions a team would ask before paying for another scan:

1. When do additional Havu runs stop producing significant new findings?
2. Which individual security skills find the most issues over repeated runs?
3. Which individual models find the most issues over repeated runs?
4. Can a skill running alone find more than Havu's aggregated scan?

The rest of this article answers them in that order.

What we tested

Every report analyzed the same options protocol.

The standalone side used five security skills:

• SC Auditor
• Trail of Bits
• Quill
• Pashov
• Nemesis

Each skill ran once with GPT-5.5 and once with GPT-5.6. That produced ten standalone reports.

Havu then ran five times against the same target. A Havu run is different from a standalone run: it sends several audit perspectives through the code and consolidates their output into one report.

This dataset tests GPT-5.5 and GPT-5.6 directly. Havu can orchestrate a wider set of models, but those providers are outside this comparison.

How we counted findings

We used two counts throughout the article.

Raw findings are the numbered entries in a report. They measure report volume.

Root causes are the underlying bugs after equivalent reports are merged. Two findings count as one root cause when they describe the same defect and require the same fix. A single report entry can contain two root causes when it combines separate defects.

This distinction matters. A tool can produce many entries by repeating the same issue. Root-cause counts show how much distinct security coverage those entries contain.

Across the complete benchmark, the reports contained 116 raw entries and 37 root causes.

Havu covered more of the bug surface

Havu found 32 of the 37 root causes. The entire standalone suite found 20.

Figure 1. Coverage of the 37 root causes in the benchmark. Havu reached 86.5%; the pooled standalone suite reached 54.1%.

The overlap is important. Havu retained most of what the standalone tools discovered while opening a much larger area of the bug surface. It matched 15 root causes found by both approaches and added 17 that did not appear in any standalone report.

Three runs captured almost all Havu coverage

Havu's first run found 13 root causes. The second lifted the cumulative total to 22. The third lifted it to 31.

That third run crossed the 95% threshold: 31 of Havu's final 32 root causes is 96.9%.

Figure 2. Cumulative root-cause discovery. Run 3 crossed 95% of Havu's final coverage; Run 4 confirmed the plateau.

The drop in marginal discovery makes the stopping point even clearer.

Figure 3. New root causes and novelty yield by Havu run. Runs 4 and 5 produced 25 raw entries for one new root cause.

For this codebase, the answer to the first research question is: three broad Havu runs captured almost all available Havu coverage, and the fourth run confirmed that broad discovery had flattened.

That gives teams a practical workflow. Continue broad scans while the root-cause curve is climbing. Once a run adds nothing new, move the budget to a focused investigation rather than repeating the same search.

Skill choice changed the result

The five standalone skills did not behave the same way.

Pashov produced the most raw report entries, with 12 across its two model runs. SC Auditor covered the most distinct root causes, with 10. That difference is a good example of why raw volume and unique coverage need separate columns.

Figure 4. Totals across GPT-5.5 and GPT-5.6. Pashov produced the most entries; SC Auditor covered the most distinct causes.

SC Auditor has more root causes than raw entries because one report entry bundled separate defects. Pashov has fewer root causes than entries because several entries across its two runs described the same bugs.

The larger point is that no single skill dominated every measure. A user choosing one skill before the audit would be choosing which security perspective to leave out. Havu runs those perspectives together.

GPT-5.6 found more than GPT-5.5

Across all five standalone skills, GPT-5.6 produced 22 raw findings and covered 17 unique root causes. GPT-5.5 produced 18 raw findings and covered 11.

Figure 5. Standalone findings pooled by model. GPT-5.6 led on both report volume and distinct coverage in this benchmark.

The difference was not uniform. Much of GPT-5.6's raw lead came from its Pashov run. That interaction between model and skill is precisely what makes choosing a single pairing difficult: the strongest combination is not obvious before the code has been reviewed.

Havu turns that uncertainty into coverage by using more than one audit path.

Did any standalone run beat Havu?

No.

The strongest standalone report was Pashov with GPT-5.6, at 8 unique root causes. Havu's five reports contained 13, 15, 22, 9, and 14 unique root causes respectively.

Even Havu's lowest-output run found one more unique root cause than the best standalone run.

Figure 6. Like-for-like report comparison using normalized root causes. Best standalone: 8. Lowest Havu: 9.

The pooled comparison is wider still. The strongest standalone skill family covered 10 root causes across two reports. Havu covered 32 across five reports. All ten standalone reports together reached 20.

This answers the fourth research question at three levels:

• No standalone report beat a Havu report.
• No standalone skill family beat the Havu suite.
• Even all five standalone skill families combined found fewer root causes than Havu.

Havu retained the standalone signal

Aggregation is useful only if it expands coverage without throwing away what the individual tools already do well.

Havu fully covered 35 of the 40 raw standalone entries. It partially covered one more. In total, Havu recovered at least part of 36 standalone entries, or 90%.

Figure 7. Havu recovery of the standalone reports, grouped by skill.

Havu reproduced every Trail of Bits, Quill, and Nemesis entry in this benchmark. It also recovered almost all of Pashov's output. The breadth did not come from abandoning the established skills; it came from combining them.

What Havu found beyond the standalone scans

Seventeen root causes appeared in Havu and nowhere in the ten standalone reports.

They were not variations of one repeated bug. They crossed several parts of the protocol:

• Collateral accounting
• Liquidation safety
• Queue and state integrity
• Authorization boundaries
• Deployment safety

This is the core benefit of aggregation. Each audit perspective explores the system differently. Havu collects those perspectives, keeps the distinct root causes, and removes the repeated descriptions.

Why aggregation changes the audit workflow

A standalone workflow asks the user to make the most important choice before seeing the code through an auditor's eyes: which skill and which model should own the review?

The benchmark shows why that is a difficult choice. Pashov led on raw volume. SC Auditor led on unique skill coverage. GPT-5.6 led overall, but its advantage depended heavily on the skill paired with it.

Havu changes the sequence:

1. Run several audit perspectives together.
2. Consolidate reports that point to the same root cause.
3. Track how many genuinely new root causes each run contributes.
4. Stop broad reruns when the discovery curve flattens.
5. Use the combined evidence to choose focused follow-up work.

The user no longer has to guess the winning pairing. Havu makes the comparison inside the scan and returns one consolidated result.

Conclusion

On this codebase, Havu found more distinct security issues than any standalone skill, any standalone model run, and all ten standalone reports combined.

It found 32 of 37 root causes, contributed 17 root causes absent from every standalone report, and recovered 90% of the standalone report surface. It also exposed a clear stopping point: three runs reached 96.9% of Havu's final coverage, and the fourth run confirmed the plateau.

That is what Havu is built to do. It combines audit skills and models so one choice does not define the limits of the review. Then it turns the combined output into a report a team can act on.

Contact SCAS Team for Early Access