Yesterday my drone early-warning demonstrator recorded four AI-generated explanations. All four were discarded by the validation gate that sits in front of them.
I looked at that number and felt good about it. A gate that rejects four out of four is a gate doing hard work. It is exactly the statistic I would put in a technical brief.
It was wrong. Not the gate. My reading of it.
What the system does
The demonstrator raises alerts. A language model is allowed to do one thing: rewrite the machine's reasoning into a sentence an operator can read at speed. It decides nothing.
Because a small model is not reliably obedient, the sentence is checked before anyone sees it. Anything unfit is thrown away and the plain wording stands. I tested that check when I built it, and it caught real problems.
So when the production counter said zero of four survived, the story wrote itself. Sporadic escalations, a small model, a strict gate. Of course some get discarded.
The counter was measuring two different things
The function that calls the model returns nothing when the gate rejects the text. It also returns nothing when the model does not answer at all. Timeout, connection refused, anything. Both cases came back as the same empty result, and I recorded both as "rejected by the gate."
That is the whole bug, and it is not really a coding mistake. It is a naming mistake. I gave one counter to two different facts.
One of those facts says the safety mechanism is working. The other says the feature is dead. They are opposites, and I had made them indistinguishable.
What was actually happening
The model was cold when an escalation arrived, and the call timed out.
Escalations are sporadic by design. The system is quiet most of the time. So nearly every escalation arrived to find the model not ready, gave up, and fell back to the plain wording.
Once the model was warm, it answered and passed the check on the first try. I confirmed that directly against the running model after the diagnosis.
The feature had not failed. It had never run.
The fix took ten minutes. Finding it took the audit trail
Three changes. Keep the model ready so a quiet hour does not guarantee the next escalation fails. Allow a slow answer more time, since it costs nothing when the fallback is already correct. And split the counter so that a rejected answer and a missing answer are counted separately.
That last one is the change that matters. The first two are configuration. The third is the difference between a number you can act on and a number that comforts you.
I would not have found any of it without something I had built the same afternoon: durable history, so every assessment is written to disk with what happened to it. That storage layer existed for a completely different reason. It was built so a host site could eventually be handed a measured false alarm rate. Four hours after it went in, it told me the headline AI feature had been inert for a day.
The second thing it caught
Once the model started answering again, one of its sentences stated a confidence level.
The check accepted it, correctly by its own rule. But look at what the sentence asserts: a measurement that nothing in the system produced.
The prompt already forbade it. The instruction is right there in the system message, and the model ignored it, which is the entire reason a gate exists instead of trust.
So the gate now rejects that phrasing rather than tolerating it. One existing test asserted the old, permissive behavior. I flipped it deliberately and wrote down why, because a test that quietly changes meaning is worse than no test.
What I would take from this
If you run a model in production behind any kind of check, the useful question is not how often the check fires. It is whether you can tell the difference between the check firing and the model never answering.
Those look identical from the outside. Both produce no output. Both leave your fallback in place. Both can be reported as a single tidy number that makes your safety mechanism look busy.
Instrument the difference. A counter that merges a working safeguard with a dead dependency is worse than no counter, because it does not stay silent. It reassures you.
The uncomfortable part is that my failure mode was the same one I write about: a confident output with nothing real behind it. It just happened to be a metric rather than a model.
A note on what this system is, because it matters in this category. DroneWatch AI is a working concept demonstrator. The threat signals are simulated. Two public feeds are live and real: ADS-B air traffic and network Remote ID of self-identifying drones. It is not an operational system, it does not detect non-cooperative drones, and no field validated performance is claimed. It is strictly defensive, detection and early warning only, with no interdiction or countermeasure function.