Roulette Lab · Lab · Statistics · about 3 min read

Statistical Significance

“Significant” means unlikely under a fair wheel, not “real” and not “profitable”.

Short answer

A p-value measures how extreme the chosen statistic is under a specified model. It is not the probability that the model is true, and scanning many results changes the interpretation.

convention
p < 0.05
false positive rate
5% per test
fix
replication
what it is not
p(wheel is biased)

What the p-value is

Assuming the stated null model, the p-valuelabp-valueUnder a specified null model, the probability of a test statistic at least as extreme as the observed one, using the chosen test.Read full definition → is the probabilitymathProbabilityA measure of how likely an event is, from zero to one. Percentages express the same scale from 0% to 100%.Read full definition → of a test statistic at least as extreme as the observed one. It is not the probability that the wheel is fair or that the result arose by chance. Interpretation also depends on the design, assumptions and tests attempted.

statementcorrect?
p = 0.03 describes a tail probability under the null modelyes
p = 0.03 gives a 97% probability of biasno
p = 0.03 means the pocket will stay hotno
searching many wheels requires accounting for selectionyes
an independent replication adds evidenceyes; not certainty

Multiple comparisons

At a calibrated 5% level, the false-positive probability is about 5% per test under its null model. Twenty such tests give one false positive on average, not exactly one in every run. The chance of at least one is 1 − 0.95ⁿ only for independent tests. PocketequipmentPocketA compartment on the rotor in which the ball can settle. The number of pockets depends on the wheel.Read full definition → counts and overlapping sectorswheelSectorA continuous arc of neighboring pockets around the wheel, rather than a group defined by the betting layout.Read full definition → are dependent.

At least one false positive: independent 5% tests%
1 test
5%
5 tests
22.6%
10 tests
40.1%
20 tests
64.2%
37 tests
85.0%
%
042.5085.0%
Independent-test illustration. These percentages do not directly apply to 37 pockets counted in the same sample.

Significance versus size

A large sample can detect a small departure from fairness that still does not make a bet profitable. At a true pocket probability of 1/36.5, a 35:1 bet has negative expectation. At 1/36 it breaks even. A p-value does not measure the size or practical value of an effect.

Power

Power is the chance of rejecting the null under a specified alternative. For one pocket chosen in advance, an exact one-sided 5% binomial test on 1,000 spins rejects at 37 or more hits. If the true rate is 1/30, its power is about 28%, not 50%.

A practical standard

Specify the hypothesis and sampling plan, retain the full record, report effect estimates and uncertainty, and validate a selected pattern on fresh data. A smaller p-value threshold alone does not repair data selection or recording problems.

Questions

Why use 0.05?

It is a convention, not a boundary between truth and falsehood. State the threshold before testing and account for multiple tests.

Does a tiny p-value prove physical bias?

No. Check assumptions and data quality. A model discrepancy can have several explanations.

Does significance identify a profitable pocket?

No. That requires a separate estimate of hit probability, its uncertainty and the actual payoutmoneyPayoutThe amount paid on a winning wager. Check whether the quoted figure is net profit or total return including the stake.Read full definition →.

Fun with Math

Try these yourself first. Each solution is folded; open it when you are ready.

  1. Suppose a dozenbetDozenA bet on 1–12, 13–24 or 25–36, ordinarily paying 2 to 1.Read full definition → chosen before data collection appears 742 times in 3,000 independent double-zerowheelZeroThe 0 outcome, conventionally shown in green. Ordinary color, parity, high/low, dozen and column bets do not include it.Read full definition → spins. Under the uniform model, how many standard deviationsmathStandard deviationThe square root of variance: a measure of dispersion in the same units as the result.Read full definition → below the mean is that? What is the probabilitymathProbabilityA measure of how likely an event is, from zero to one. Percentages express the same scale from 0% to 100%.Read full definition → of 742 or fewer hits?

    Show the solution

    Answer About 8.07 standard deviations below the mean. The exact binomial lower-tail probability is about 9.77 × 10⁻¹⁷. It is extremely small, but not zero.

    Under the stated model, X is binomial with n = 3,000 and p = 12/38. Its mean is np ≈ 947.37 and its standard deviation is √(np(1 − p)) ≈ 25.46.

    The standardized difference is (742 − 947.37)/25.46 ≈ −8.07. Summing the exact binomial probabilities from 0 through 742 gives P(X ≤ 742) ≈ 9.77 × 10⁻¹⁷. A normal approximation with continuity correction gives about 4.25 × 10⁻¹⁶; the relative error is large this far into the tail.

    This is the probability of the specified data event under the model, not the probability that the wheel is fair. Such a result calls for checking the observations, independence, uniformity and analysis plan. Selecting the worst-looking dozen after inspecting data, or searching many wheels or time periods, requires accounting for that selection.

Comic relief

A small p-value is not proof of bias or profit.

A small p-value is not proof of bias or profit.

Sources

  1. American Statistical Association, statement on p-values (2016) https://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf
  2. NIST, chi-square goodness-of-fit test https://www.itl.nist.gov/div898/software/dataplot/refman1/auxillar/chsqgood.htm
Default model: European, single zero; exceptions are stated in the articlemethodology ↗sources ↗