Skip to content
EM lab, home

Inference

How sure? How many? How stable?

The notebook printed point estimates. This page asks the questions a statistician would ask next: how precise those estimates are, whether the intervals around them can be trusted, how many groups the data support, and whether EM's answer depends on where it starts. Every number is precomputed from a fixed seed, listed with its sizes under seeds and sizes at the end, and the fast ones can be re-run in your browser.

question 1

How sure are the fitted parameters?

Standard errors only mean something at a maximum of the likelihood, and the notebook's 15 iterations had not reached one. So the run is finished first: same data, same random start, until the log-likelihood moves by less than 10⁻¹⁰ (170 iterations, ℓ = −415.366). The notebook's own estimates stay in the first column, unchanged. Components are ordered by mean (DR-002): component 1 is the low group, the notebook's romance lovers.

Two kinds of uncertainty sit beside each estimate. The Hessian standard error measures how sharply the log-likelihood falls away from its peak: the observed information (minus the matrix of second derivatives of ℓ at the maximum), inverted. It gives a Wald interval, θ^±1.96 SE\hat\theta \pm 1.96\,\mathrm{SE}. The parametric bootstrap simulates 1,000 new data sets of 200 ratings from the fitted mixture (seed 42), refits EM to each and reads the 2.5% and 97.5% points of the 1,000 estimates.

Fitted parameters with observed-information standard errors, Wald intervals and parametric-bootstrap intervals, components ordered by mean
parameternotebook, t = 15converged fitnotebook's maximumSE (Hessian)95% WaldSE (bootstrap)95% bootstraptrue
π₁share of the low group0.3440.2210.05440.114 to 0.327misses0.06490.110 to 0.359misses0.4
μ₁mean of the low group4.023.360.2672.84 to 3.88misses0.3102.86 to 4.114.0
μ₂mean of the high group7.367.020.1926.64 to 7.40misses0.2136.60 to 7.44misses7.5
σ₁spread of the low group1.250.860.1790.51 to 1.21misses0.1890.48 to 1.22misses1.5
σ₂spread of the high group1.271.460.1391.18 to 1.730.1491.16 to 1.761.2

misses marks an interval that does not contain the true value. π₂ = 1 − π₁ has the same standard error and intervals as π₁.

  • bootstrap estimates
  • 95% bootstrap interval
  • converged fit (notebook's maximum)
  • 95% Wald interval
  • true value
  • π₁ share of the low group

  • μ₁ mean of the low group

  • μ₂ mean of the high group

  • σ₁ spread of the low group

  • σ₂ spread of the high group

Published result: seed 42, B = 1,000 data sets of n = 200.

1,000 of 1,000 refits succeeded; the variance floor bound in 0; the two means crossed (needing relabelling, DR-002) in 0.

question 2

Do the 95% intervals cover 95% of the time?

A 95% interval should contain the true value in 95% of repeated samples. That is checkable: simulate data sets from the notebook's true mixture (60% at 7.5 with σ 1.2, 40% at 4.0 with σ 1.5, n = 200), fit each by EM from a k-means++ start, build the intervals and count. 500 data sets per scenario for Wald intervals (seed 1), once with the model exactly and once clipped to 1 to 10 like the notebook; 200 for the percentile bootstrap (B = 200 each, seed 7), paired with the Wald intervals on the same data sets.

Neither reaches 95%. Wald intervals covered between 84% and 92% of the time; percentile-bootstrap intervals between 90% and 92%. On the same data sets the bootstrap covered more often for π₁, μ₁ and σ₁ (paired 95% intervals exclude 0); for μ₂ and σ₂ the difference is within simulation noise. Part of the gain comes from width: the bootstrap intervals are wider for 4 of the 5 parameters (median widths in the table). At n = 200 with groups this close, the log-likelihood is not yet the parabola the Wald interval assumes. Clipping, perhaps surprisingly, changes little.

  • Wald, model exactly (S = 500, seed 1)
  • Wald, clipped like the notebook (S = 500, seed 1)
  • Percentile bootstrap, model exactly (S = 200, B = 200, seed 7)

Each mark is the share of intervals that contained the true value; the whiskers are Wilson 95% intervals for that share.

Coverage counts, rates with Wilson intervals and median interval widths
parameterWald, modelseed 1Wald, clippedseed 1Wald, same 200seed 7bootstrap, same 200seed 7bootstrap − Wald (paired)seed 7
π₁433/500 = 86.6%83.3% to 89.3% · width 0.352437/500 = 87.4%84.2% to 90.0% · width 0.327169/200 = 84.5%78.8% to 88.9% · width 0.350182/200 = 91.0%86.2% to 94.2% · width 0.365+6.5 pts3.0 to 10.0 · 14 vs 1 discordant
μ₁422/500 = 84.4%81.0% to 87.3% · width 1.80426/500 = 85.2%81.8% to 88.0% · width 1.64171/200 = 85.5%80.0% to 89.7% · width 1.79180/200 = 90.0%85.1% to 93.4% · width 1.85+4.5 pts0.5 to 8.5 · 13 vs 4 discordant
μ₂450/500 = 90.0%87.1% to 92.3% · width 0.92443/500 = 88.6%85.5% to 91.1% · width 0.89178/200 = 89.0%83.9% to 92.6% · width 0.85183/200 = 91.5%86.8% to 94.6% · width 0.91+2.5 pts-1.0 to 6.0 · 9 vs 4 discordant
σ₁423/500 = 84.6%81.2% to 87.5% · width 1.01436/500 = 87.2%84.0% to 89.8% · width 0.97173/200 = 86.5%81.1% to 90.6% · width 1.02183/200 = 91.5%86.8% to 94.6% · width 0.98+5.0 pts2.0 to 8.5 · 11 vs 1 discordant
σ₂456/500 = 91.2%88.4% to 93.4% · width 0.58460/500 = 92.0%89.3% to 94.1% · width 0.57178/200 = 89.0%83.9% to 92.6% · width 0.57181/200 = 90.5%85.6% to 93.8% · width 0.62+1.5 pts-1.0 to 4.0 · 5 vs 2 discordant

Widths are medians in the parameter's units. The paired column compares the two kinds of interval on the same 200 data sets: the difference in coverage in percentage points, a 95% bootstrap interval from resampling data sets (B = 4,000, seeds 100 to 104, one per parameter), and the discordant pairs (only the bootstrap interval covered, against only the Wald interval). Failed fits: 0 (model), 0 (clipped), 0 (bootstrap study); fits without a positive-definite information matrix: 0; variance floor binding: 0.

Published result: seed 1, S = 500 data sets in each scenario (Wald intervals; the bootstrap study is too slow for a browser).

question 3

How many groups do the data support?

The notebook assumed two groups. Fitting K = 1 to 4 components (variance floor σ ≥ 0.1, DR-003) and scoring each with AIC and BIC gives an uncomfortable answer: BIC prefers 3 components and AIC 4, and neither picks two. BIC's extra component is small and narrow: 4.0% of the ratings at μ = 9.96 with σ = 0.10, held at the floor. That is the pile of 7 ratings the notebook's clipping put at exactly 10.0, not another kind of viewer.

Each K gets the best of 60 ordinary starts (k-means++, Forgy and random; seed 4) and 30 pile starts, which put a narrow component on the 7 tied ratings at 10.0. The pile starts matter: ordinary starts spread their means over the bulk of the ratings with wide spreads, so none of them isolates seven identical values. For K = 3 and 4 only pile starts reached the best fit; without them the table would report lower maxima for those K.

The choice also depends on the floor. BIC picks K = 3 at σ ≥ 0.05 and 0.1, but K = 4 at σ ≥ 0.25, where 3 values of K are within 1.1 BIC points of each other. A narrow component on tied values gains likelihood as its σ shrinks, so the floor sets how much the pile is worth (DR-003).

Log-likelihood, AIC and BIC for one to four components, on all 200 ratings and without the clipped ones
KparamsℓΔAICΔBICstarts at bestσ at floorcomponents (weight at mean)ΔBIC without the 7 at 10.0
12−425.3139.112.41/1no100% at 6.21 (σ 2.03)17.0
25−413.9922.55.63/900 from pile startsno84% at 5.86 (σ 2.02) · 16% at 8.07 (σ 0.30)0.0
38−403.237.00.03/903 from pile startsyes76% at 5.51 (σ 1.80) · 20% at 8.06 (σ 0.34) · 4% at 9.96 (σ 0.10)4.1
411−396.760.02.95/905 from pile startsno22% at 3.31 (σ 0.82) · 54% at 6.35 (σ 1.17) · 20% at 8.11 (σ 0.33) · 5% at 9.92 (σ 0.13)10.9

Δ is the criterion minus the smallest in its column (0 marks the choice). “Starts at best” counts the starts that ended within 0.01 of the best log-likelihood, out of 60 ordinary starts plus 30 pile starts; that so few find it is itself a warning about multimodal likelihoods. “σ at floor” marks a best fit with a component held at σ = 0.1. Without the clipped ratings there is no pile, and BIC picks K = 2 (AIC still picks 4).

Is that unusual for the recipe?

The notebook's data are one draw. Drawing 100 fresh samples of 200 from the same recipe (seed 3; 12 starts per K, plus 6 pile starts wherever clipping piled three or more ratings onto one value) shows how the criteria behave in general. BIC picks two components in 87 of 100 samples (Wilson 79% to 92%) and in 91 of 100 when clipped; otherwise it picks one component (11 and 5 times) or three (2 and 4). AIC overfits: it picks two in only 61 and 34 of 100, and clipping makes it worse. So BIC's K = 3 on the notebook's sample is a property of this particular draw, not of the recipe.

Share of 100 simulated data sets on which BIC and AIC chose each number of components, with Wilson intervals
picked KBIC, modelBIC, clippedAIC, modelAIC, clipped
1116% to 19%52% to 11%00% to 4%00% to 4%
28779% to 92%9184% to 95%6151% to 70%3425% to 44%
321% to 7%42% to 10%2618% to 35%3425% to 44%
400% to 4%00% to 4%138% to 21%3224% to 42%

One group or two? A bootstrap likelihood-ratio test

The textbook test compares twice the gain in log-likelihood with a χ² distribution whose degrees of freedom are the extra parameters, here 5 − 2 = 3. That rests on Wilks' theorem, which needs the null hypothesis to sit inside the parameter space with every parameter identified. A single normal is a two-component mixture with

π2=0(μ2, σ2 anything)orμ1=μ2, σ1=σ2(π1 anything)\begin{aligned} &\pi_2 = 0 \\ &\qquad (\mu_2,\ \sigma_2 \text{ anything}) \\ \text{or}\quad &\mu_1 = \mu_2,\ \sigma_1 = \sigma_2 \\ &\qquad (\pi_1 \text{ anything}) \end{aligned}

so the null lies on the edge of the space (π₂ cannot go below 0) and some parameters vanish from the model there. Wilks' theorem does not apply. The parametric bootstrap side-steps it: fit one normal, simulate 500 data sets from it (seed 12), run the same two-component fit (9 starts) on each, and use those statistics as the null distribution.

  • bootstrap null (500 data sets, seed 12)
  • χ² density with 3 df
  • bootstrap 95% point: 11.49
  • χ² 95% point: 7.81
  • observed: 19.88
observed 2(ℓ₂ − ℓ₁)
19.88
ℓ₁ = −425.31, ℓ₂ = −415.37
bootstrap p-value
0.002
(1 + 0) / (500 + 1): no null statistic was as large, and p cannot be smaller with B = 500
χ²₃ p-value (not valid here)
1.8 × 10⁻⁴
tiny here too, but only because the effect is large: the reference is wrong
χ²₃ test's real false-alarm rate
16%
80 of 500 null data sets exceed 7.81 (Wilson 13.0% to 19.5%), not 5%

The bootstrap's 95% point is 11.49 against χ²₃'s 7.81. The variance floor bound in 40 of the 500 null fits (Wilson 5.9% to 10.7%); leaving those out moves the 95% point to 11.10, so the gap is not an artefact of the floor (DR-003). The 9 starts on the observed data (plus 9 pile starts) found ℓ₂ = −415.37, not the higher maximum at −413.99, so the observed statistic, if anything, understates the evidence.

question 4

Does EM's answer depend on where it starts?

200 notebook-style random starts (μ ~ U(3, 8), σ ~ U(0.5, 2), π = 0.5; seed 2025) on the notebook's 200 ratings, each run until |Δℓ| < 10⁻¹² with its whole log-likelihood trace kept. The ascent property holds in every run. The answer does depend on the start: most runs reach the maximum the notebook was heading for, and a few find the narrow-component maximum with a higher likelihood. “Keep the best of many starts” maximises the likelihood; here that means more starts make the implausible answer more likely, not less.

log-likelihood never decreased
200/200
runs; largest single drop 0
maxima reached
2
ℓ = −413.99 (10 starts); ℓ = −415.37 (190 starts)
one start reaches the best maximum
5.0%
10 of 200, Wilson 95% CI 2.7% to 9.0%

Iterations to reach each tolerance

Box: quartiles and median; whiskers: fastest and slowest of 200 random starts. The dashed tick in each row is the notebook's own start: 90 iterations to 10⁻⁴, 117 to 10⁻⁶ and 144 to 10⁻⁸ (the notebook gave it 15).

|Δℓ| < 10⁻⁴: median 110 (IQR 59 to 148) · |Δℓ| < 10⁻⁶: median 137 (IQR 86 to 175) · |Δℓ| < 10⁻⁸: median 163.5 (IQR 112 to 201)

Best of R random starts

Chance that at least one of R starts reaches the higher maximum (ℓ = −413.99), from the single-start rate and its Wilson interval.

RP(best reached)95% interval
15.0%2.7% to 9.0%
29.8%5.4% to 17.1%
522.6%13.0% to 37.5%
1040.1%24.2% to 60.9%
2064.2%42.6% to 84.7%
5092.3%75.0% to 99.1%

Published result: seed 2025, 200 notebook-style random starts on the notebook's 200 ratings.

Random start or k-means++? A paired comparison

The same 200 simulated data sets (the notebook's recipe, clipped; seed 31), each fitted once from the notebook's random start and once from k-means++ (tolerance 1e−6). Pairing by data set removes the variation between data sets from the comparison.

extra iterations for the random start
31.8
mean paired difference, 95% CI 23.8 to 39.5 (bootstrap over data sets, B = 2,000, seed 31); medians 137 vs 105
k-means++ was faster
153/200
data sets (70% to 82%); 2 ties
reached the best-known maximum
random 200/200 · k-means++ 200/200
Wilson 95% CI for each: 98.1% to 100.0% and 98.1% to 100.0%. On these data sets the starts differ in speed, not in where they end up.

reproducibility

Seeds and sizes

Every simulation on this page starts from a fixed seed, so the same code with the same seed and sizes gives the same numbers. The “Run in your browser” buttons check that for the bootstrap, the Wald coverage study and the random starts; any other seed shows how much a result moves from one simulation to the next.

The random seed and the sizes behind each simulation on this page
what the seed drivesseedsizesshown in
The notebook's 200 ratings and its random start (NumPy, in the original notebook)42n = 200Uncertainty
Parametric bootstrap of the notebook's fit42B = 1,000 data sets of n = 200Uncertainty
Wald coverage, model exactly and clipped (same seeds for both)1S = 500 data sets per scenario, n = 200Coverage
Wald and bootstrap coverage on the same data sets7S = 200 data sets, B = 200 per data setCoverage
Paired bootstrap − Wald coverage intervals (one seed per parameter)100 to 104B = 4,000 resamples of the 200 data setsCoverage
AIC and BIC for K = 1 to 4 on the notebook's ratings (also without the clipped ones, and at other variance floors)460 ordinary starts + 30 pile starts per KChoosing K
How often each criterion picks each K (model exactly and clipped)3S = 100 samples per scenario, 12 starts + 6 pile starts per KChoosing K
Bootstrap likelihood-ratio test, one group against two12B = 500 null data sets, 9 starts + 9 pile starts per fitChoosing K
Notebook-style random starts (ascent, iterations, maxima, best of R)2025200 startsConvergence
Random start against k-means++ on the same data sets31S = 200 data sets, n = 200Convergence
Paired intervals for that comparison (iterations, then reaching the best maximum)31 and 32B = 2,000 resamples of the 200 data setsConvergence