Verification
What Verified, Failed check and Not verified mean on a fix, what DebugAI can check, and why the third label exists.
Every fix DebugAI offers carries a confidence figure and one of three labels. The label says whether anything tested that figure.
| Label | What it means | Its confidence |
|---|---|---|
| ✓ Verified | A mechanical check ran on the fix and passed | As the model gave it. A pass confirms the figure and never raises it |
| ✗ Failed check | A mechanical check ran and the fix failed it. DebugAI disproved its own fix before showing it to you | Capped at 15% |
| Not verified | Nothing checked this fix | The model's own estimate, untouched |
Hover over a label in the results panel to see the reason behind it.
What DebugAI can check
The checks are deliberately narrow. Each one proves one thing and nothing more:
- Syntax errors (
SyntaxError,IndentationError). DebugAI checks that the code parses with the fix in place. A pass means the fix is valid code, not that it does what you want. - Import errors (
ImportError,ModuleNotFoundError). DebugAI checks that the import the fix names matches the files it retrieved for this debug. A pass does not mean the package is installed, because DebugAI cannot see what is installed on your machine.
Every other kind of error gets Not verified. So does a fix the checks could not finish, for example one whose code continues past its own lines and cannot be judged as a fragment.
Why Not verified has its own label
Without it, an unchecked fix at 85% and a checked fix at 85% would look the same. They are not the same claim. One is a number a model wrote; the other survived a test. Not verified keeps the two apart instead of folding the third case into the first, so you know which figures to lean on.
A fix for a runtime error, such as a TypeError or a KeyError, is Not verified, because the checks cover only syntax and import errors. That is not a warning. It means you read the diff, apply it and run your code, as you would with a fix from a colleague.
A fix that failed before
When a fix for the same error was reported as failed in this project, DebugAI caps its confidence when it offers that change again, and an agent's answer says when it was reported and as which fix. That record is part of error memory.
In the answer an agent gets
The MCP server prints one of these lines under each fix:
✓ Verified — <the reason>
✗ Failed mechanical check (confidence capped) — <the reason>
· Not verified — confidence is the model's own estimate; nothing checked this fix.The same state is in the tool's structured output as verified: true, false or null, with the reason in verification_reason. The DebugAI plugin for ChatGPT and Codex calls the three states passed, failed and not_checked.
An answer can also hand the agent a command to run after applying a fix. When the agent passed reproCommand, that command is the check, with the expected result if the agent gave one. DebugAI itself never runs it. Otherwise the answer may offer a syntax-level check command. Either way, report_outcome is how the result gets back to DebugAI.
In Proactive Scan
Proactive Scan labels findings the same way:
- verified: the claim was checked against your source.
- unverified: DebugAI could not check the claim mechanically.
- lowered: the finding's severity was lowered because the claim could not be verified.
Claims that were checked and did not hold up never reach your screen. The results panel says how many were dropped, for example 3 possible issues were checked against your source and dropped because they did not hold up.