The derivations, written out

Likelihood and log-likelihood

Probability distributions describe which values are likely and which are not. For a normal distribution with mean $\mu$ and standard deviation $\sigma$ the density is

If $x$ is close to $\mu$ the density is high, if it is far away it is low, and $\sigma$ controls how spread out it is. The likelihood asks how plausible the data are under given parameters. Multiplying many small densities underflows, so we add logarithms instead:

EM is a method for maximising this log-likelihood when part of the data is hidden.

The mixture set-up

Separate users into two groups (“sci-fi lovers” and “romance lovers”) without knowing who is in which.

• Observed data: $\mathbf{X} = \{x_1, x_2, \ldots, x_n\}$, the ratings.
• Latent variables: $\mathbf{Z} = \{z_{ik}\}$ with $z_{ik} = 1$ if user $i$ belongs to group $k$ and 0 otherwise; exactly one $z_{ik}$ is 1 for each user.
• Parameters: $\theta = \{\pi_1, \pi_2, \mu_1, \mu_2, \sigma_1^2, \sigma_2^2\}$, the mixing proportions, means and variances.

If we knew which group each user belonged to, the complete-data log-likelihood would be

E-step: “what's the best guess?”

The E-step replaces each unknown $z_{ik}$ by its expected value given the data and the current parameters $\theta^{(t)}$:

This is about group membership, not activity: every user is in exactly one group, and $\gamma_{ik}$ is how sure we are which. By Bayes' theorem,

The three pieces are the group's density, its prior share, and the marginal density of the rating:

Putting them together:

The numerator is the joint probability $P(x_i, z_{ik} = 1 \mid \theta^{(t)})$, the denominator the marginal, and their ratio the conditional; so $\gamma_{i1} + \gamma_{i2} = 1$.

Worked example

A user rates “The Matrix” 5 stars, with current estimates $\mu_1 = 4.5,\ \sigma_1 = 0.8,\ \pi_1 = 0.6$ (sci-fi) and $\mu_2 = 3.0,\ \sigma_2 = 1.2,\ \pi_2 = 0.4$ (romance). The explainer computes $0.6 \times f(5 \mid 4.5, 0.8^2) = 0.6 \times 0.47 = 0.28$ and $0.4 \times f(5 \mid 3.0, 1.2^2) = 0.4 \times 0.12 = 0.05$, so $\gamma_{i1} = 0.28 / (0.28 + 0.05) = 0.85$.

The hand-worked four-rating example in the explainer has the same kind of slip; the stepper replays it with both sets of numbers side by side.

M-step: “update your model”

With responsibilities in hand, each group is re-estimated by weighted averages, the weights being how strongly each user belongs to the group:

The mixing proportion is the average membership; the mean is a weighted average in which users more likely to be in group $k$ count for more; the variance is the weighted spread around the new mean. The explainer's example: users with sci-fi probabilities 0.85, 0.20 and 0.90 rated 5, 4 and 5, so

(That one checks out.) Better groups give better memberships, which give better groups: a feedback loop.

Deriving the mean update

Start from the expected complete-data log-likelihood:

Differentiate with respect to $\mu_k$:

Set it to zero and solve:

A Normal + Beta mixture

EM does not need the components to be the same family. Suppose tech-savvy users rate on a continuous 0 to 10 scale (normal) while casual users give a thumbs up or down, recorded on [0, 1] (beta):

The E-step is the same Bayes calculation with two different densities in the numerator:

The mixing proportions and the normal's $\mu$, $\sigma^2$ update exactly as before (using $\gamma_{i1}$). The beta parameters have no closed form; they solve

where $\psi(x) = \frac{d}{dx}\log\Gamma(x)$ is the digamma function. In the explainer's example, starting from $\mu = 7,\ \sigma = 1$ and $\alpha = \beta = 2$ with equal shares, a rating of 0.2 is almost certainly beta and a rating of 7.5 is certainly normal, since the beta density is zero outside [0, 1].

The algorithm and convergence

1. Initialise $\theta^{(0)}$.
2. E-step: compute $\gamma_{ik} = E[z_{ik} \mid x_i, \theta^{(t)}]$.
3. M-step: $\theta^{(t+1)} = \arg\max_{\theta} Q(\theta \mid \theta^{(t)})$.
4. Stop if $|\ell(\theta^{(t+1)}) - \ell(\theta^{(t)})| < \epsilon$; otherwise repeat.

EM guarantees the log-likelihood never decreases,

so it converges to a local maximum (or a saddle point) of the likelihood. Local is the important word: see the pitfalls page.

Revival addendum

Mixing proportions (a Lagrange multiplier)

Maximise $\sum_i \sum_k \gamma_{ik} \log \pi_k$ subject to $\pi_1 + \pi_2 = 1$:

since summing over $k$ gives $\lambda = \sum_{i,k}\gamma_{ik} = n$.

Variances

The notebook updates $\sigma_k$ as the square root of this, which is the same update. If one component's weight concentrates on a single point, the numerator goes to zero: that is the variance collapse pitfall.

Why the log-likelihood never decreases

For any distribution $q_i$ over the group of user $i$, Jensen's inequality gives a lower bound on the log-likelihood:

The E-step chooses $q_{ik} = \gamma_{ik}$, which makes the bound touch $\ell(\theta^{(t)})$; the M-step maximises the bound over $\theta$. So $\ell(\theta^{(t+1)}) \ge \text{bound}(\theta^{(t+1)}) \ge \text{bound}(\theta^{(t)}) = \ell(\theta^{(t)})$. The playground checks this on every trace it draws.