When There Is No Ground Truth, Convergence Is the Proof

When there is no ground truth, convergence is the proof. The integrity of a hackathon's result usually rests on comparison with an external answer — the leaderboard metric, the working demo, the user the team served. But some of the most rigorous hackathons in history have produced outputs the world could not independently verify on the day of judging. The Event Horizon Telescope's M87* image was a measurement of something no one had ever seen before. The AlphaFold protein structures submitted at CASP14 were predictions made before the experimental structures were public. The Higgs boson detection at CERN in July 2012 was a signal in noise that had to be confirmed by two independent collaborations on the same beam. In each case the integrity of the result could not be established by comparison with an external answer, because there wasn't one. The integrity had to come from somewhere else.

The somewhere else is convergence. When multiple independent teams, working on the same shared dataset with methodologically distinct paths, all arrive at the same answer, the agreement itself becomes the evidence. The integrity mechanic shifts from "the answer matches the truth" to "the answers match each other across methods that should not have produced spurious agreement." This is the structural innovation that the Event Horizon Telescope's four-team blind imaging protocol embodied — and that Aaron Klein and Aaron Roodman's canonical 2005 Annual Review of Nuclear and Particle Science paper on blind analysis had been arguing for decades earlier as the standard for high-stakes physics measurements. Robert MacCoun and Saul Perlmutter, writing in Nature ten years later, pressed the same case across the empirical sciences more broadly, arguing that hiding results from analysts until the analysis is locked is the most reliable structural defense against the bias toward expected answers.

The mechanic only works when the design protects it, and four conditions matter most. The teams must be genuinely isolated; the methods used by the teams must be sufficiently distinct that spurious agreement is implausible; the parameter sweeps must be pre-registered before the data release; and the convergence event itself must be a designed dramatic moment, not an afterthought. The Event Horizon Telescope's seven-week imaging sprint in June and July 2018 met all four. The four teams paired CLEAN-based pipelines (the Americas and Global teams) against Regularized Maximum Likelihood pipelines (the East Asia and Cross Atlantic teams) precisely so that any agreement could not be attributed to shared algorithm bias. The teams committed to their imaging methods at the October 2017 EHT Image Hackathon at the Black Hole Initiative — labeled directly on screen in the Galison documentary's title cards — before being shown the real M87 data the following June. The cross-correlation reveal at the July 2018 BHI workshop, with the four teams seeing each other's images for the first time and confirming agreement above 95% pixel-to-pixel correlation, was the convergence event. These four conditions together turn the embargo into the demo day.

The pattern recurs wherever ground truth is genuinely unavailable on the day of judgment. CASP, running biennially since 1994, holds out experimental protein structures from competitors and scores their predictions against the truth only after submission — the doubly-blind protocol AlphaFold dominated at CASP14 in 2020. The Higgs boson announcement was structured as a simultaneous publication by the ATLAS and CMS collaborations precisely so that two independent detectors with two independent analysis pipelines could be seen to agree, rather than one collaboration carrying the claim alone. LIGO's gravitational wave detection in September 2015 underwent months of internal blinding before publication. The protocols differ in detail but share the underlying logic: when the universe is the only entity that knows the right answer, the integrity of the result must be argued through structural means, not asserted through authority.

The principle reaches well beyond Big Science, and recognizing that is the move the rest of this site builds on. Hackathon organizers running competitive single-track events with verifiable outputs do not need this principle; the leaderboard metric or the working demo settles integrity for them. Hackathon organizers running events whose outputs make claims about something with no external referee — scientific measurements, contested historical facts, novel design decisions, predictions of future states — do. Any hackathon whose outputs the world cannot independently verify on the day of judging is implicitly making a claim about its own integrity, and the strongest claim available is convergence across independent teams using methodologically distinct paths on a shared dataset. The format taxonomy format-taxonomy places the Blind Replication Sprint as the tenth working hackathon archetype precisely because the mechanic is structurally distinctive, not just another rubric variation. The judging principle fair-judging covers what the three conventional fair-judging architectures look like; this principle covers what fair evaluation looks like when none of those three apply. The Event Horizon Telescope case study event-horizon-telescope works the M87 example in full structural and dramaturgical depth, including the 2019 credit- attribution episode that demonstrates how convergence-based integrity can survive perfectly inside a collaboration even while the public storytelling around the result fails badly outside it.