When the finding is an inference: What an AI

A finding sits in the remediation queue for eleven days. The engineer assigned to it cannot reproduce it. There is no recorded basis to dispute it: no version string, no scan artifact, no observation log. It is neither fixed nor closed. It just waits, consuming queue position. The problem isn't whether the finding is accurate or if the tool that generated it is trustworthy. The problem is the queue has no mechanism to distinguish this finding from one that could be re-run on demand, and no one wrote down what would make it go away. What intake was assuming Intake was built on two assumptions: a finding names a version on a host, and that version claim can be re-run. Those assumptions held for deterministic findings, and they held so reliably that the intake process never had to make them explicit. NIST SP 800-53 Rev. 5 control RA-5 requires monitoring and scanning for vulnerabilities at defined frequencies and when new vulnerabilities potentially affecting a system are identified — an obligation scoped to the system as inventoried (Source: NIST SP 800-53 Rev. 5 Update 1, control RA-5, https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final). The standard treats a finding as a finding. It does not distinguish between a finding the scanner matched directly and one a model inferred from behavioral signals, because when RA-5 was written, that distinction did not need to be operational. It does now. The assumption baked into intake is that reproducibility is guaranteed by the finding type. Strip that guarantee and the queue inherits a structural ambiguity it was not designed to handle. What a deterministic match carried for free Here is the move that goes unnoticed until it is absent: a version match came with a free disproof path. If the scanner said Apache 2.4.49 on host web-prod-07 , you could re-run the scan. If the version string had changed — patched, replaced, containerized — the finding disappeared. This single property is what made triage cheap. A team could dispute a finding in minutes, not days, because the observation was repeatable and the evidence was retrievable. The finding was falsifiable on demand. NIST SP 800-53 Rev. 5 control SI-2 requires organizations to identify, report, and correct system flaws — and to test fixes for effectiveness before installation (Source: NIST SP 800-53 Rev. 5 Update 1, control SI-2, https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final). The word "correct" presupposes that the team agreed the flaw exists. A finding with a free disproof path made that agreement easy: run it again, confirm the state, correct it. A finding with no disproof path breaks that chain before it starts, not because the flaw is absent but because agreement is structurally unavailable. Reproducibility was doing the work. The absence of reproducibility is not an accuracy problem. It is an intake architecture problem, and the two are not the same failure. What an inferred finding has to supply instead When there is no re-run path, the finding must carry what the re-run would have produced. This means four fields — not a confidence score. A confidence score is not a disproof path. It tells you how certain the model was, not what the model observed, not what the host state was when it observed it, and not what would make a defender disagree. A team that receives a score of 0.87 and nothing else cannot falsify the finding. It can only accept or ignore it, and neither outcome is triage. The four required fields: Observation — What signal did the inference rest on? Not the conclusion; the input. A behavioral pattern, a configuration delta, a combination of state: something a defender can look at. Host state at observation time — What did the system look like when the signal was captured? Configuration, running processes, network posture, timestamp. If the host state has changed since observation, the finding may already be invalid, and the team needs to know. Recorded basis — Why did that observation support this conclusion? This does not have to be a model explainability artifact — it has to be a statement specific enough that a defender can disagree with it. "Service X was observed without TLS on port 443 between 14:00 and 14:12 UTC" is a basis. "Anomalous behaviour detected" is not. Disproof condition — What would make this finding false? If the team cannot write down an answer, the finding cannot be closed, because there is no defined terminal state. That is the eleven-day stall, stated as a structural condition rather than a workflow failure. For the classification-basis parallel — what a generated label must carry to be actionable. | Finding Source | Is It Reproducible On Demand | What Disproves It | Required Recorded Basis | May It Enter The Queue Without That Basis | |---|---|---|---|---| | Deterministic version match (scanner confirms Apache 2.4.49 on named host) | Yes — re-run the scan against the same host | Patch applied; re-scan returns no match | CVE identifier, matched version string, host identifier, scan timestamp | No — basis is automatically present; if absent, the scan artifact is broken | | Inferred misconfiguration (model concludes TLS is absent based on observed traffic pattern) | No — traffic pattern may not recur | Direct inspection confirms TLS enforced on the relevant port | Observation window, traffic sample or log reference, specific port and service, timestamp | No — observation and host state at observation time must be recorded before intake | | Inferred combination-based exposure (model concludes two individually-acceptable configurations create an exposure together) | No — combination state may have changed | Confirm one of the two configurations has changed such that the combination no longer holds | Both configuration states at observation time, the dependency relationship between them, timestamp | No — both states and the basis for their interaction must be recorded; a score alone is insufficient | | Finding from a lying version string (service self-reports a patched version; model infers actual version from behavioural indicators) | No — the deceptive string will re-appear; the behaviour driving the inference may or may not | Behaviour no longer present after remediation; direct binary inspection confirms actual version | Specific behavioural indicators observed, what they contradict in the reported version string, timestamp | No — the contradiction between reported state and observed behaviour must be explicit in the intake record | | Finding the team cannot construct a disproof condition for | No | No condition can be written — the finding's truth value does not change based on any observable system state | The absence of a disproof condition is itself a required recorded field | Yes — but it enters the queue labelled as unfalsifiable, counted separately from findings with disproof paths | The case for inference at intake The argument so far could read as a case for excluding inferred findings. It is not. Some real exposures do not produce a version match. A misconfiguration that only becomes visible as a runtime behaviour — a service that silently degrades TLS negotiation under specific load conditions — produces no CVE match. A version string that lies about its patch level defeats the scanner's primary evidence path; behavioural inference may be the only way to surface the actual state. An exposure that arises from two configurations that are individually acceptable but collectively dangerous has no single component to match against. These are findings that no deterministic scan produces. Excluding them from intake does not make the environment safer; it makes a category of real exposure invisible to the remediation queue entirely. The argument here is about evidentiary requirements, not admissibility. Inference belongs at intake. The question is what it must carry when it arrives. The intake requirement to write down Remediation capacity is finite. A queue that cannot distinguish a version match from an inference spends the same engineer-hours on both. That is not a prioritization failure — it is a structural one. The expensive failure is not the false positive that gets investigated. It is the deterministic finding with a known disproof path that waited eleven days behind an unfalsifiable one. The single operational change intake requires: findings without a recorded disproof path should enter the queue labelled as such and counted separately from findings that carry one. The label is not a rejection. It is a signal that the team is spending time on a finding whose terminal state is undefined — and that the first action on that finding is to attempt to write down the disproof condition, not to begin remediation. What the engineer sees when a labelled finding arrives is not a confidence score. It is the four fields: the observation, the host state at observation time, the recorded basis, and the disproof condition or an explicit statement that one could not be constructed. With those fields present, the engineer can do something definable: confirm or deny the observation, check whether the host state has changed, disagree with the recorded basis, or escalate the finding for closer review. Without them, the finding sits. The queue cost is real in either direction. A labelling system that never clears unfalsifiable findings creates its own backlog. Teams that implement separate tracking for inference-sourced findings without a defined review cadence — shaped by their own policy, threat exposure, and operational capacity — can find the label becoming a permanent deferral rather than a structured escalation. The cadence is a policy decision, not a fixed interval; what drives it is how fast the environment changes relative to how long an unconfirmed exposure can wait. Sources - NIST SP 800-53 Rev. 5 (Update 1), control RA-5: https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final