E-step · expectation
Which group does each rating probably belong to?
Using the current guess of each group, Bayes' theorem gives every rating a probability of coming from each group: its responsibility.
Expectation–Maximisation, by doing
EM, one step
at a time
Imagine knowing only how people rate films, not what kind of viewer they are, and still recovering the groups behind the ratings. That is what the EM algorithm does. This site turns a long, maths-complete explainer and its Python notebook into something you can step through, poke and break.
Watch the guided tour· three short walkthroughs
A personal project by Sunchuangyu (Rin) Huang · written 2025, revived 2026
The idea
Guess, then improve the guess. Repeat.
The explainer calls EM a smart detective. It cannot see who belongs to which group, so it alternates between two easier questions until the answers stop changing.
M-step · maximisation
Given those probabilities, what does each group look like?
Each group's share, average and spread become weighted averages, with ratings counting in proportion to how much they belong.
What the original notebook found
200 ratings, two hidden groups, 15 iterations
The notebook draws 200 “movie ratings”: 60% from sci-fi lovers around 7.5 and 40% from romance lovers around 4.0, clipped to 1 to 10. It then hides the labels and lets EM, written from scratch, find the groups from a random start. These are its printed results, reproduced by this site's TypeScript port to within 10⁻⁶.
- average rating
- 6.21
- range 1.2 to 10.0
- log-likelihood
- −416.51
- from −425.50 after iteration 1 (+8.98)
- classified correctly
- 90.5%
- of the 200 users the model was fitted to (in-sample), using γ > 0.5; 95% CI 85.6% to 93.8% (Wilson)
- iterations
- 15
- the cap it was given; not yet converged
| group | share π | mean μ | spread σ | fitted as |
|---|---|---|---|---|
| Sci-fi lovers fitted as component 2 | 0.656true 0.6 | 7.361true 7.5 | 1.273true 1.2 | component 2 |
| Romance lovers fitted as component 1 | 0.344true 0.4 | 4.022true 4.0 | 1.249true 1.5 | component 1 |
Note which component each group was fitted as: EM put the sci-fi lovers in component 2. That small detail is the first of the pitfalls.
Six ways in
Read it, run it, break it
- 01StepperThe explainer's four ratings, [2, 3, 7, 8], one E-step and M-step at a time, with the hand-worked numbers beside the exact ones.Open stepper
- 02PlaygroundThe notebook's 200 ratings, or data you make. Pick a starting guess, press play and watch the curves and the log-likelihood move.Open playground
- 03PitfallsLabel switching, stopping too early, local maxima and variance collapse: four ways a run can look fine and still be wrong.Open pitfalls
- 04InferenceStandard errors and bootstrap intervals, whether those intervals really cover 95%, how many groups the data support, and how much the answer depends on the start.Open inference
- 05MathsBayes' theorem for the E-step, calculus for the M-step, a Normal + Beta mixture and why the likelihood never goes down.Open maths
- 06MethodsData provenance, assumptions and limitations, the decision records, a model card for the fitted mixture and the AI use statement.Open methods
Honest notes
Kept visible, not quietly fixed
Rebuilding the notebook number for number turned up three things worth knowing. The originals stay as they were; the site shows what happened and why.
Some hand-worked densities are off
The explainer uses and ; both are 0.4839. The conclusions hold. See both side by side.
The labels switched
Component 1 ended up as the low-mean group, so the notebook's final cell calls an 8.5 rater a romance lover (P = 0.001 for sci-fi). Why it happens.
Fifteen iterations was not convergence
The run stopped at its cap while still improving; finishing it changes the answer. Watch it finish.
About this project
A personal explainer, rebuilt as a lab
- What
- Personal project, not coursework
- Author
- Sunchuangyu (Rin) Huang
- When
- September 2025 (explainer and notebook, last edited June 2026), October 2026 (this site)
- Data
- Synthetic ratings only; no real users
Original stack
- Markdown explainer with LaTeX maths
- Python 3 in a Jupyter notebook
- NumPy for the EM loop, SciPy for densities in plots
- pandas, Matplotlib for figures
Revived stack
- Next.js 16 (App Router), React 19, TypeScript
- Tailwind CSS v4, shadcn/ui, KaTeX rendered at build time
- Hand-drawn SVG charts; EM loops in a Web Worker
- Vitest parity tests against the notebook's own trace
Provenance
The explainer and notebook are kept unchanged in original/. A script, export_parity.py, re-executes the notebook's cells with seed 42, checks every printed line against the saved output, and exports the ratings, the random start and the per-iteration trace. The TypeScript port of EMAnalyzer is tested against that trace to 10⁻⁶, and regenerates the notebook's console output character for character. Nothing here needs a server or an account; the one optional AI feature runs in your browser with your own key, and the methods page says what it does.