Correction — editorial date ; actual revision : This update clarifies the boundary demonstrated by TopFlow's scanner: finding-ID membership checks do not establish ranking or commentary accuracy. Advisory descriptions are omitted from the model input, but component names remain untrusted. Human approval for outward actions is a recommended policy, not a claim that every workflow ships that gate.
Picture this: your new AI-powered security scanner finds a real vulnerability, cites a real CVE, and then on the next line cites one that doesn't exist. The prose reads exactly the same for both. Nothing in the report tells a reader which finding is real. Now imagine that report goes to your board, or worse, gets posted automatically to a pull request. The vulnerability that doesn't exist is now part of your official record, and the model that wrote it sounded just as confident as it did for the real one.
This can happen even when a language model receives accurate source data. It is a governance gap when organizations adopt AI security tooling. Teams evaluate these tools by asking whether the underlying data is accurate. They rarely ask the harder question: once accurate data goes into the model, does accurate data come out?
The assumption that quietly breaks everything
Most AI tool evaluations follow the same logic: the data source is reliable, so the tool built on top of it is reliable. A vulnerability scanner that queries a real CVE database, or a compliance tool that reads your actual policy documents, gets treated as trustworthy because its inputs are trustworthy.
That logic has a hole in it. Handing a language model verified facts and asking it to write about them doesn't make its output verified. Three things go wrong, and they go wrong independently of how good your source data is.
First, the model can simply invent things. Ask it to summarize a scan result and it may cite a CVE identifier that sounds plausible and doesn't exist, with the same fluent confidence it uses for the real ones. A developer who chases a fabricated vulnerability wastes a day. A CISO who reports one to the board burns credibility that doesn't come back easily.
Second, any third-party text the model reads becomes a vector for it to receive instructions you didn't intend to send. Advisory descriptions, PR comments, support tickets, anything written by someone outside your organization is a place an attacker can plant text that looks like data but reads like a command. If your pipeline hands that text to the model, the model can't reliably tell the difference between "this is content to summarize" and "this is an instruction to follow."
Third, checking free-text output is harder than checking a structured identifier. Automated checks can catch specific errors without establishing that an entire paragraph is accurate. You can ask a human to read every report, which doesn't scale, or you can run regex over the output looking for known bad patterns, which only catches the patterns you already anticipated.
The fix isn't a better prompt. It's a different job description for the model
The instinct is to solve this with better prompting: tell the model to only cite real CVEs, tell it to ignore embedded instructions, tell it to format its output carefully. None of that works reliably, because you're asking a system that generates statistically likely text to behave deterministically. It will follow the instruction most of the time. Most of the time isn't a security control.
The actual fix is structural, and it's a principle any CISO can apply regardless of which AI tool or vendor is involved: stop asking the model to be the author of facts, and give it a narrower job it's actually suited for, like ranking, labeling, or prioritizing a set of facts someone else already verified.
Concretely, that means the model cannot introduce new findings into the factual report. It receives a constrained, pre-verified set of items, and the only thing it's allowed to return is a structured selection from that set: an ID, an enum value, a ranking. Anything it returns that wasn't in the original set gets dropped automatically, by code, before inclusion in the report. The model still does useful work, but ID validation does not prove its ranking is sensible or its optional commentary is correct. Keep commentary separate from authoritative findings; a bounded factual path is not a guarantee about every generated word.
TopFlow's scanner demonstrates a narrow version of this boundary. Advisory descriptions are omitted from its minimized input, but finding IDs, severity labels, and component names still reach the model. Data minimization reduces exposure; it does not make the remaining text trusted. Validate the selected IDs against the source set, assemble factual findings in code, and review prioritization and commentary separately. This pattern cannot establish that the source scan found every vulnerability.
What to ask before you adopt (or build) an AI security tool
You don't need to read anyone's source code to apply this. Five questions separate a tool that's more defensible from one that's hoping for the best:
- Can the tool's output ever contain a fact, identifier, or claim that didn't come from a verified source you can name? If the answer is "the model might add color," that color is a fabrication risk.
- Does any third-party text (advisory descriptions, ticket bodies, PR comments) reach the model verbatim? If so, ask whether that text is ever omitted entirely rather than just filtered, because filtering an unknown attack surface is a losing game.
- Is the model's output validated by code against a known-good set before anyone sees it, or is a human expected to catch errors by reading carefully? Reading carefully doesn't scale, and people stop doing it after the tenth report.
- What happens when validation fails? A tool that falls back to a deterministic, model-free version of the report when the model's output looks fabricated is safer than one that either errors out or, worse, ships the fabrication anyway.
- Is there an audit trail showing what the model returned versus what was actually used? Without that record, "the model said X" is a claim you can't verify after the fact.
If a vendor can't answer these, or answers with "we use a well-crafted prompt," you're buying a tool whose safety depends on the model behaving well every single time, indefinitely, against inputs the vendor hasn't seen yet.
The combination that turns a minor gap into a major incident
There's one configuration worth flagging on its own, because it shows up constantly as AI tools mature from read-only to read-write: a system that has access to sensitive internal data, processes untrusted external text, and can take an action in the outside world without a human checking it first. Each piece is manageable alone. Together, they mean a single piece of adversarial text, embedded somewhere as ordinary-looking content, can cause the system to leak data, take an unintended action, or both, with no attacker ever touching your infrastructure directly.
Any AI security tool that's planning to move from "produces a report" to "automatically comments on the PR" or "automatically files the ticket" is walking toward this combination. For that transition, require the system to draft a response for a human to approve before anything goes out. This is a recommended approval policy, not a claim that every TopFlow workflow implements it. Build that gate before the write capability ships, not after someone asks why it wasn't there.
Make the approval decision explicit
For a pilot, ask for a decision record, not just a polished demo: the saved source findings, the model's proposed selection, and the final report after validation. Assign an owner who can explain discrepancies. A report without this evidence may be useful for exploration, but it should not become board reporting by default.
Test two different failures. A fabricated ID should be rejected. A real but irrelevant ID can pass a membership check and still mislead the reader, so it needs a separate relevance review. Record source coverage gaps too: an unsupported manifest is not evidence that the project has no vulnerabilities.
Approve outward actions separately from read-only analysis. Document who approves a ticket or pull-request comment, what evidence they see, and how an erroneous action can be withdrawn. Passing an output-validation test is not approval to add a write capability.
Take this back to your next vendor review
You don't need to become a machine learning expert to act on any of this. Just treat "the model's output" the same way you'd treat any other untrusted input, because that's what it is, even when it was built from data you trust completely. The next time someone on your team or a vendor says an AI feature is accurate because it's "grounded in real data," ask what happens between the data and the output. That gap is where the risk actually lives.
I built the URW (untrusted reasoning worker) pattern into TopFlow's own security scanner specifically to demonstrate these controls as working code rather than a slide. The technical writeup, including seven design invariants and attack-tree analysis, is on topflow.dev. The implemented scanner validates model-selected finding IDs; the broader design's human-gated outward actions are not all shipped controls. If you want help applying this thinking to a specific tool or vendor your team is evaluating, that's exactly the kind of architecture review I do. Book a consultation and bring the tool you're least sure about.