Reading a frontier claim

Most frontier experiments never see anything, and that is the normal, publishable outcome. A search reports a curve: for each candidate mass, the largest signal rate compatible with what was seen, at a stated confidence level. Direct dark-matter searches have pushed the spin-independent WIMP–nucleon cross-section limit from about 10⁻⁴² cm² in the early 2000s to below 10⁻⁴⁷ cm² near 30 GeV — more than five orders of magnitude of parameter space closed without a single detection, and they are now approaching the irreducible background from solar and atmospheric neutrinos. Read the axes and the fine print together: that curve assumes a local halo density near 0.3 GeV cm⁻³ and a particular velocity distribution, so it excludes a model, not a particle. And "excluded at 90% CL" means the experiment would have seen more than this in 90% of repeats were the model true. It does not mean the model is 90% dead.

A p-value answers exactly one question: if only background were present, how often would a fluctuation this large or larger appear? For a one-sided Gaussian tail, 3σ gives p = 1.3 × 10⁻³, about 1 in 740, and 5σ gives 2.9 × 10⁻⁷, about 1 in 3.5 million. Particle physics calls 3σ "evidence" and reserves "observation" for 5σ, while a clinical trial is content with 0.05, and the gap is not fussiness. Three things drive it: thousands of searches run every year, so rare fluctuations are guaranteed somewhere; the prior on an exotic signal is low, so a modest p can leave a claim more likely wrong than right; and the historical record is full of 3σ effects that died. Note what the number does not cover. It is computed inside an assumed background model, and an effect the model omits shifts the central value without widening the quoted uncertainty — so no quantity of sigmas protects against it.

Do not ask how unlikely this bump is; ask how unlikely it is that the most extreme of all your searches produced one. If a scan covers N statistically independent windows, p_global ≈ 1 − (1 − p_local)^N ≈ N⋅p_local while that product stays small. Scan 100–2000 GeV with 20 GeV mass resolution and N ≈ 95: a local 3.7σ (p = 1.1 × 10⁻⁴) becomes a global p near 1.0 × 10⁻², which is 2.3σ, and 2.3σ is unremarkable. This is not hypothetical arithmetic. The 750 GeV diphoton excess of December 2015 was 3.9σ local and about 2.3σ global in ATLAS, 2.6σ local and under 1.2σ global in CMS; some five hundred theory papers followed, and the excess was gone in the next year's data. The informal version is worse — the cuts, binnings and variables you tried and did not record carry a trials factor nobody can reconstruct afterwards.

The uncountable trials factor has one honest remedy: fix everything before you look. In a blind analysis the cuts, background model and fitting procedure are frozen using simulation and sidebands, the signal region is opened once, and the result is published whatever it says. Variants add an unknown offset to the answer, or salt the data with fake signals — LIGO ran a blind hardware injection in 2010 that the collaboration analysed and wrote up before being told it was not real. Then the systematic budget: list every effect, size each one, and add independent contributions in quadrature. A budget of 1.2% energy scale, 1.0% background shape, 0.8% efficiency and 0.6% luminosity gives √(1.44 + 1.00 + 0.64 + 0.36) = 1.85%. Against a 0.40% statistical error the total is 1.90%. Quadruple the data and the total improves to 1.87% — a 1.7% gain for four times the running. That is where a measurement stops being statistics-limited, and no exposure repairs it.

Nothing in the previous three sections converts a claim into knowledge. An independent experiment with different systematics does. OPERA's neutrinos arrived 60.7 ns early with ±6.9 ns statistical and ±7.4 ns systematic uncertainty — nominally about 6σ — and the cause was a loose fibre-optic connector together with a clock drift; ICARUS on the same beamline then measured a time consistent with c. BICEP2 announced r = 0.20 in 2014 at high significance, and a joint analysis using Planck's 353 GHz dust maps reassigned the signal to polarised galactic dust, leaving r < 0.12. The successes have the same shape. The Higgs was announced in July 2012 only when ATLAS and CMS each reached about 5σ independently; GW170817 became certain when an electromagnetic counterpart arrived 1.7 s later from an entirely different kind of instrument. Two recent claims of near-ambient superconductivity collapsed within months when other groups could not reproduce them.

Seven questions, each with the measured part separated from the unmeasured one. Neutrino oscillations fix only mass-squared differences — Δm²₂₁ ≈ 7.5 × 10⁻⁵ eV² and |Δm²₃₁| ≈ 2.5 × 10⁻³ eV², so the heaviest state is at least 0.050 eV — while the ordering and the absolute scale are unknown, bounded under about 0.5 eV directly and 0.12 eV by cosmology. The baryon-to-photon ratio is measured at 6 × 10⁻¹⁰; quark-sector CP violation is orders of magnitude too small to produce it. Ωₘ ≈ 0.31 against Ω_b ≈ 0.049 leaves roughly 0.26 non-baryonic, with no candidate detected anywhere. Ω_Λ ≈ 0.69 with w = −1.03 ± 0.03 is consistent with a constant, not proof of one. H₀ disagrees at about 5σ between early- and late-universe routes. The neutron EDM bound |dₙ| < 1.8 × 10⁻²⁶ e⋅cm forces the QCD angle below 10⁻¹⁰ for no known reason. And nothing tests gravity at 10¹⁹ GeV. Write each one as open; a forecast is not a result.