Statistical Significance
“Significant” means unlikely under a fair wheel, not “real” and not “profitable”.
A p-value of 0.05 means a fair wheel would produce a pattern this lopsided one time in twenty. If you look at twenty wheels, one will pass by accident.
What the p-value is
The probabilitymathProbabilityChance of an outcome, from 0 to 1 or 0% to 100%. One number: 1/37 = 2.70%.Read full → of seeing deviation at least as large as the one observed, if the wheel were fair. It is a statement about data under an assumption, not about the wheel. A p-valuelabp-valueThe probability of seeing a deviation at least as large as the one observed if the wheel were fair.Read full → of 0.03 says: a fair wheel does this 3% of the time. It does not say: there is a 97% chance the wheel is biased. The second statement would need to know how common biased wheelslabBiased wheelA wheel whose physical defects make some pockets more likely than 1/37. Rare, detectable only with thousands of recorded spins.Read full → are to begin with, and in a modern casino they are very rare, which makes a single significant result far more likely to be a fluke than a find.
Multiple comparisons
Every test you run on a fair wheel has a 5% chance of passing at the 5% level. Test twenty wheels and one passes. Test one wheel in twenty time windows and one passes. Test one sample with 37 pocketequipmentPocketOne of the 37 numbered compartments on the rotor. Also: slot, cup.Read full → groupings, six sectorwheelSectorA contiguous arc of pockets on the wheel, as distinct from a group on the layout.Read full → groupings and three color groupings and something passes. The cure is to decide the test before looking at the data, run it once, and confirm on new data. The scanning that finds a “biased” wheel is usually the thing that produced the result.
Significance versus size
A huge sample can make a trivial bias significant. A pocket hitting at 1/36.5 instead of 1/37 is a real defect that would reach significance in a few million spins and would still lose money on a 35 to 1 payoutmoneyPayoutWhat a winning bet returns, quoted as “X to 1” on top of the returned stake.Read full →. What matters for a player is effect size: does any pocket hit at 1/36 or better, and by how much? The chi-square tells you about the wheel; only the hit rate tells you about the bet.
Power
The other error is missing a bias that is there. With 1,000 spins, a pocket at 1/30 (a very large bias) is detected only about half the time at the 5% level. Sample size, not cleverness, fixes this. The sample size page gives the numbers.
A practical standard
For a bias claim to be interesting: a test chosen in advance, p < 0.01 rather than 0.05, replication on an independent sample of similar size, and an estimated hit rate on the favored pockets of 1/34 or better after replication. That standard has been met a handful of times in the game’s history and never, as far as public records show, on a modern maintained wheel.
Questions
Convention from the 1920s. There is nothing special about it; 0.01 is a more sensible bar when you are scanning many wheels.
It is probably not producing uniform results. It might be a dealer effect, a recording error or a data-entry problem. Replicate.
No. It tells you the distribution is off. Sector analysis and effect size tell you where.
Fun with Math
Try these yourself first. Each solution is folded; open it when you are ready.
A reader told the Wizard of Odds about a double-zerowheelZeroThe single green pocket that no even-money, dozen or column bet covers; the source of the house edge. Also: single zero, 0.Read full → wheel on which the first dozenbetDozen1–12, 13–24 or 25–36, pays 2:1.Read full → came up 742 times in 3,000 spins. How many standard deviationsmathStandard deviationSquare root of variance; the typical distance of a result from the expected value.Read full → below the average is that, and how often would a fair wheel do it?
Show the solution
Answer About 8.1 standard deviations low. A fair wheel would do it about once in 3 × 10¹⁵ tries, which is never. The answer published with the question says 11.75; we get 8.07.
The first dozen holds 12 of the 38 pocketsequipmentPocketOne of the 37 numbered compartments on the rotor. Also: slot, cup.Read full →. In 3,000 spins it averages 3,000 × 12/38 = 947.4 hits, with a spread of √(3,000 × 12/38 × 26/38) = 25.5.
742 is 205.4 below the average: 205.4/25.5 = 8.07 spreads.
A fair wheel falls three spreads short about once in 740 tries; eight spreads short, about once in 3 × 10¹⁵ (bell curve, one side). Allowing for the reader having noticed the worst of three dozens changes nothing that matters. A result like this says the wheel, or the count, is wrong. On a single-zero wheel the same count would be 9.0 spreads low (average 973.0, spread 25.6).
Comic relief
One in twenty fair wheels is “significant”. That is what the number means.
Sources
- Pearson, K. “On the criterion that a given system of deviations from the probable…”
- Feller, W. An Introduction to Probability Theory and Its Applications, Vol. 1.
- Wizard of Odds. “Ask the Wizard”, roulette questions.