Everyone assumes that once AI can see the game, it can referee the game. I assumed a version of that myself. So I started testing the assumption.
The audio shortcut
I took real NBA clips and had a frontier vision model make the calls. In successive founder-produced trials, agreement with the referees' actual calls reached 74% with broadcast audio (50 clips) and 20% with audio stripped (40 clips). Those runs were not a paired experiment, so the size of the drop is not a clean effect estimate.
But the transcripts showed the model quoting announcers, and the third-party RefereeBench benchmark also found frame-only foul judgment weak, while submissions using the original video with audio scored higher—consistent with, but not a controlled replication of, my audio-shortcut finding. My ablation found an audio shortcut in my own setup; any referee demo using broadcast clips should test for the same failure. I run everything audio-off.
There is no answer key
The deeper problem is what "correct" even means. On marginal contact, even the experts split: in a study of deliberately ambiguous basketball decisions, 38 events had to be dropped because a three-expert panel could not agree on them at all. On the 283 where the panel did reach consensus, that consensus matched the call made on the floor only 38.5% of the time.
So there is no answer key sitting anywhere, waiting to be scraped. "Correct" on a marginal call is a consensus, and the consensus has to be built, clip by clip, out of expert judgment. No sensor emits a right answer. Someone has to rule.
Testing the shortcuts
My own numbers weren't safe from this either. My first foul-detection model scored 96.5% F1 on a held-out set of 358 clips: 208 fouls and 150 negatives. I didn't believe that one either. Any model can look brilliant on easy examples. So I built an adversarial benchmark: 301 real fouls hidden among 1,070 near-miss look-alikes, plays chosen because they share the surface cues a foul has but aren't one. On this benchmark, the model still separates fouls from deliberately mined look-alikes at 0.94 AUROC. The benchmark makes surface-cue shortcuts much harder; it does not prove which features the model used or how it transfers to new footage.
The frontier models fail in ways that tell you something. In a small, founder-labeled failure probe—four clean plays, six model calls per play—ungrounded models whistled a foul on 23 of 24 calls. They also reported roughly 95% confidence when wrong. That is a reason to build a larger expert evaluation, not a prevalence claim.
This is why I think refereeing is an intelligence problem, not a sports app. A game can turn on a call made in a split second, privately, with no appeal and no explanation. I want the play to decide the game, not the whistle. Seeing is not enough. Judgment remains the harder frontier.