Shotgate

Statistical test gates for quantum programs written in Quantinuum's Guppy and run on the Selene emulator.

A quantum program gives a different answer every run, so a test can't assert one output. It has to check the shape of thousands of outputs. The two tests people usually write for that are both bad in a way that doesn't look bad:

• "every share is within 2 points of expected" at 1,000 shots fails a correct Bell pair about a quarter of the time, purely from sampling. People learn to hit rerun.
• "no impossible outcome ever appears" is fine on an ideal emulator and fails every single noisy run, because noise makes every outcome possible.

Shotgate replaces both with a check that has a guaranteed error rate in each direction, spends only as many shots as the decision needs, and exits non-zero in CI when a program regresses.

Demo page (a recorded replay, not a live emulator): mohithgajjela.com/system-03-shotgate

Each test states what a correct program's outputs look like and reduces a batch of shots to one number: the distance from that spec, with a confidence interval around it.

from shotgate import matches, probability, forbids

matches({"00": .5, "11": .5})            # total variation distance to a distribution
probability("1", 0.3188)                 # |P(outcome) - target|
forbids(only=("0000", "1111"))           # mass on outcomes that should never happen

Every check carries two thresholds:

The gate fails once the lower end of the interval is above allow, and passes once the upper end is below detect. Between the two lines is an indifference zone: a deviation that is real but smaller than detect may go either way, and if the shot budget runs out first the verdict is INCONCLUSIVE, which CI treats as a failure with "needs more shots" rather than a guess. The guarantee, per check:

true distance <= allow   =>  P(FAIL) <= alpha     (no flaky failures)
true distance >= detect  =>  P(PASS) <= alpha     (no missed bugs)

with alpha = 0.001 by default.

Shots are drawn at 250, 500, 1,000, 2,000, 4,000, 8,000 and 16,000, and the run stops at the first look that decides. Alpha is split evenly across the seven looks (a union bound), so peeking after every batch can't inflate the error rate. An obvious bug fails after 250 shots, and every correct program in the example suite passes in under 5,000 on average.

Both are written from the definitions in shotgate/stats.py, standard library only, so each one can be checked line by line:

• Clopper-Pearson exact one-sided bounds for a single probability, via the regularised incomplete beta function (modified Lentz continued fraction). The tests pin them to closed forms: a clean run of 29 units gives an upper bound of 0.0981 at 95%, and the binomial CDF matches direct summation to 1e-10.
• Weissman's L1 inequality for whole distributions: P(||p_hat - p||_1 >= e) <= (2^k - 2) exp(-n e^2 / 2). Outcomes outside the spec's support are pooled into one bucket. That keeps the distance exact, since the spec puts zero mass there, while keeping k small.

Both are checked by simulation too: their miss rate stays under the nominal delta.

Under noise a correct program drifts away from its ideal distribution. Here, Bernstein-Vazirani drifts by 4.6 points, so a fixed 2% allowance fails it about half the time. calibrate runs each known-good program under the noise model with 200,000 separately seeded shots, takes the upper confidence bound on its drift, and writes allow and detect per test into budgets.json. It works like a snapshot file: when you change a program on purpose, you recalibrate.

The example suite has five programs with eleven planted bugs. Every program has a Z-basis check; the entangled ones also have an X-basis check, which applies a Hadamard to each qubit before measuring.

The finding that mattered most: three of the eleven bugs (a stray Z on a Bell pair, a stray S on a GHZ state, and a teleport whose Z correction never fires) produce exactly the same Z-basis statistics as the correct program. No number of shots and no statistical method on Z measurements can catch them, and they passed every Z check in every trial. One extra X-basis check catches each of them every time at the first 250-shot look. More rigor on the same measurement doesn't help. Measuring a different way does. It also cuts the other way: the teleport X check is structurally blind to a missing X correction, which only the Z check catches.

Recorded 2026-10-01 13:30 on darwin/arm64, Python 3.12.13. 300 trials per correct configuration, 100 per buggy one, 200 per power-curve point. Alpha 0.001 per check, looks at 250, 500, 1000, 2000, 4000, 8000, 16000 shots. Noise model: uniform depolarizing, 1-qubit 0.001, 2-qubit 0.005, measurement 0.003, preparation 0.001.

Shotgate columns count FAIL and INCONCLUSIVE together. The naive columns are the two tests people usually write, at a fixed 1000 shots: every outcome's share within 0.02 of expected, and no impossible outcome observed.

Share of trials in which each check failed the buggy program (average shots in brackets).

Full suite: 11 of 11 bugs caught in every trial. Z-basis checks alone: 8 of 11.

Random-bits program with qubit 0 under-rotated by a growing angle. The true distance from fair is sin(bias)/2. Here allow is 0.02 and detect is 0.08.

Trials draw consecutive, disjoint slices of one long seeded Selene shot stream per configuration. Shots are independent draws, so these are independent trials; it just avoids paying the emulator's start-up cost once per trial. python experiments/run.py re-records everything (about 15 minutes on 16 cores), and python experiments/report.py rewrites the tables above.

pip install -e ".[test]"
pytest -q                                            # 49 tests, about 90 s
python -m shotgate run                               # example suite, ideal emulator
python -m shotgate run --mutant bell:phase           # exits 1, names the failing check
python -m shotgate run --z-only --mutant bell:phase  # exits 0: the blind spot
python -m shotgate calibrate                         # writes budgets.json
python -m shotgate run --noise --budgets budgets.json --junit report.xml

Testing your own program means writing a @guppy function that reports measurements with result("m0", ...), result("m1", ...), and so on, then adding a Case with the check a correct run must satisfy. See examples/suite.py.

CI (.github/workflows/ci.yml) runs the unit and emulator tests, the suite on the ideal and noisy emulators, a step that requires a planted phase bug to fail the gate, and a quick pass of the experiments.

shotgate/stats.py      incomplete beta, Clopper-Pearson, the L1 radius
shotgate/checks.py     matches / probability / forbids
shotgate/runner.py     sequential decision rule, Selene shot source
shotgate/cli.py        run, calibrate, JUnit and JSON output
examples/programs.py   five Guppy programs, each with planted bugs
examples/suite.py      what a correct run of each must look like
experiments/           the runs behind every number above
tests/                 bounds vs closed forms and simulation, end-to-end on Selene

• Emulator only. Nothing here ran on Quantinuum hardware; that goes through Nexus with an account and credits.
• A hand-picked noise model. Uniform depolarizing noise I chose, not a model of any device. Real noise isn't uniform and drifts over a day, so budgets would need recalibrating against a real machine.
• Conservative bounds. The even alpha split and the L1 bound cost shots compared with tighter designs. I chose bounds I could verify over the tightest available.
• It can't choose your bases. The gate is only as good as the measurements a test asks for, and as the results show, that choice mattered more than the statistics.