How the numbers were made, and what they cannot tell you

Context

The notebook draws its 200 ratings, the hidden groups and EM's random starting guess from NumPy's global random stream after np.random.seed(42). That stream (the legacy Mersenne Twister, NumPy's legacy_gauss normal sampler and its uniform conversion) is not available in the browser. The revived site needed the exact same data and the exact same starting guess to claim that its TypeScript port of EMAnalyzer reproduces the notebook, and it needed fresh random data for the "make your own" playground and the restart galleries.

Decision

Run the notebook's code cells verbatim in scripts/export_parity.py (same order, same seed, same class), fail unless every printed line matches the output saved in the notebook, and export the ratings, true groups, random start and per-iteration trace to web/public/data/notebook-run.json. The TypeScript port is tested against that trace. Anything generated in the browser uses a separate seeded generator (xoshiro128** seeded through splitmix32) and the interface says it is not the notebook's data.

Options considered

1. Port NumPy's generator to TypeScript (MT19937 plus the legacy normal sampler). Possible, but a large amount of code whose only purpose is to reproduce one draw, and any floating-point difference in the sampler would silently change the data.
2. Copy the numbers out of the notebook by hand. The notebook never prints all 200 ratings or the full-precision starting guess, so this is not even possible without re-running it.
3. Run Python in the browser (Pyodide). Exact, but a download of tens of megabytes for a page that only needs 24 KB of numbers.
4. Re-execute and export with a script (chosen).

Why

Provenance is the point of the revival: every number on the site should trace back to the original code. A script that refuses to export unless the notebook's own printed output is reproduced is a stronger guarantee than any reimplementation, and it keeps the browser code simple.

What happened

• The port matches the exported trace to within 10⁻⁶ in every π, μ, σ and log-likelihood of all 15 iterations, and the regenerated console output is identical, character for character, to what the notebook printed (em.parity.test.ts).
• The export records the environment it ran in: Python 3.13.9 and NumPy 2.5.3.
• The weak spot is the second random stream. The same seed gives different numbers in the playground than in NumPy, so "seed 2025" on the site and in Python describe different data sets. The site says so wherever generated data appear, but it is still a thing a reader has to keep in mind.
• The same pattern now covers the inference results: a seeded script writes a committed artefact and a test re-runs it (DR-004).

What I'd change

• Pin the export's dependencies exactly (uv lock --script) rather than with lower bounds, so a future NumPy cannot change the regenerated file.
• Re-run export_parity.py in CI and fail on any diff, instead of relying on the parity tests against a committed file.

Context

EM numbers its components arbitrarily: "component 1" is whichever bump the starting guess happened to steer towards. The notebook generated group 1 as the sci-fi lovers (mean 7.5), but its fitted component 1 ended up on the low ratings (μ₁ = 4.02). Its final cell then read component 1 as "sci-fi lover" and printed that a user who rates a film 8.5 is probably a romance lover. Its 90.5% accuracy is correct only because the accuracy line happens to encode the labels the switched way.

The upgrade made the problem bigger. A parametric bootstrap refits EM a thousand times and a coverage study refits it hundreds of times. Every refit has to name its components the same way, or the replicates of "μ₁" mix two different quantities.

Decision

• The notebook's as-run output stays on the site, unchanged and labelled as such.
• Wherever the truth is known (accuracy, the parameter tables, the playground), fitted components are matched to the true groups by mean before any comparison.
• In every inference calculation (standard errors, bootstrap, coverage, choosing K), components are ordered by mean, μ₁ < μ₂. In one dimension, ordering by mean and matching to the truth by mean give the same pairing.

Options considered

1. Fix the notebook's output. Rejected: the original stays as it ran (DR-001's principle).
2. Order by mean (chosen): the usual identifiability constraint, easy to explain ("component 1 is the low group").
3. Match each refit to a reference fit by its nearest parameters, or Stephens' relabelling algorithm. More robust when groups overlap, but harder to explain and unnecessary here.
4. Order by weight. Unstable when the weights are similar.
5. Report only label-free quantities (the mixture density, the sorted means). Honest but less useful for a reader who wants "the share of romance lovers".

Why

The notebook's groups are far apart (fitted means about 3.4 and 7.0, with standard errors near 0.2 to 0.3), so the ordering constraint never bites and its meaning is obvious to a reader.

What happened

• In the label-switching census (100 notebook-style random starts, seed 7), 46 runs ended with component 1 as the low-mean group (46%, Wilson 95% CI 36.6% to 55.7%). The switch is close to a coin toss, not bad luck.
• In the parametric bootstrap (B = 1,000, warm-started at the fit), the two means never crossed: 0 of 1,000 replicates needed relabelling.
• Under seed 0 the notebook's own accuracy line reports 10.0% for a fit that is 90.0% right once labels are matched. The matched number is the one the site uses.
• The weak spot: ordering by mean truncates the sampling distribution when the means are close. With overlapping groups (which a visitor can make in the playground) bootstrap intervals for the means would be biased. Nothing on /inference uses such data, but the rule would not be safe there.

What I'd change

• Use a relabelling algorithm (or report label-free summaries) whenever the bootstrap means come within a few standard errors of each other, and flag it on screen.
• Have the playground warn when the fitted means are too close for "component 1" to mean anything.

Context

A Gaussian mixture's likelihood has no maximum: put one component on a single rating and shrink its σ, and the log-likelihood goes to +∞. The notebook's data make this easy, because clipping at 10 piled seven ratings onto exactly 10.0. The notebook's own run never got close (its σ stayed above 0.6), and the revival already had an opt-in floor for the collapse demo plus a guard that stops a run once σ falls below 10⁻⁸.

The upgrade fits hundreds of thousands of mixtures: K = 3 and 4 for model choice, and K = 2 on data simulated from a single normal for the likelihood-ratio test. Those are exactly the fits that find spikes. Without a bound some of them degenerate, and simply discarding them would bias the test's null distribution, because the replicates where a spike wins are the ones that matter.

Decision

All inference fits (standard errors, bootstrap, coverage, choosing K, the LRT, convergence studies of simulated data) use a floor of σ ≥ 0.1. The parity code path (fit with the default varianceFloor = 0) is unchanged. Every analysis counts the fits where the floor ended up binding and the page shows those counts.

Options considered

1. No floor; drop degenerate runs. Biases the LRT null, as above.
2. Absolute floor, σ ≥ 0.1 (chosen). Interpretable in rating points and far below any real group's spread (the true σ are 1.2 and 1.5).
3. Relative floor (for example 5% of the sample SD) or scikit-learn's reg_covar = 1e-6 added to each variance. Scale-free, but 10⁻⁶ is far too small to stop a spike on seven identical values.
4. A prior on σ² (penalised or MAP EM, as in Chen and Tan's penalised likelihood), which also has theory for testing the number of components. The principled option, but a different estimator from the notebook's.
5. Model the clipping with a censored likelihood, so no component needs to absorb the pile at 10. Fixes the cause rather than the symptom.

Why

The floor is the smallest change that makes every fit well defined while leaving the fits that matter untouched, and it is easy to state and to check.

What happened

• The floor never bound for the notebook's fit, its 1,000 bootstrap refits, or the coverage-study fits.
• It decides how much the pile at 10.0 is worth when choosing K. The best three-component fit puts its third component on the clipped ratings with σ held at the floor (4.0% at μ = 9.96, ℓ = −403.23), and BIC picks K = 3 (BIC 848.85, against 851.79 for K = 4 and 854.47 for K = 2); AIC picks 4. The floor prevents degenerate spikes; it does nothing about misspecification.
• That K = 3 fit was missed at first. Review found that none of the ordinary starts (k-means++, Forgy, random) could isolate seven identical values, so the first version reported a lower K = 3 maximum (ℓ = −409.17) and said both criteria chose K = 4. The fitter now adds starts with a narrow component on any value tied three or more times, and a test pins the K = 3 maximum and BIC's choice.
• The choice moves with the floor. BIC picks K = 3 at σ ≥ 0.05 and 0.1, and K = 4 at σ ≥ 0.25, where K = 2, 3 and 4 are within 1.1 BIC points of each other. Without the seven clipped ratings the floor never binds and BIC picks K = 2.
• It did bind in the LRT: in 40 of the 500 data sets simulated under the single-normal null, the best two-component fit had a component at σ = 0.1. The null distribution therefore depends on the floor. Leaving those 40 out moves the bootstrap's 95% point only from 11.49 to 11.10, still far above the χ² value of 7.81, so the conclusion does not hinge on it.

What I'd change

• Report the LRT for several floors (0.05, 0.1, 0.25) as a sensitivity check, as the choice of K now is.
• Replace the floor with a weak inverse-gamma prior on σ², which keeps the likelihood bounded without a hard edge.
• Fit a censored mixture for ratings at 1 and 10, so the pile at 10 stops being a "component".

Context

The inference page needs about 40,000 EM fits for the bootstrap coverage study alone (200 simulated data sets, each fitted once and refitted 200 times in its bootstrap), plus 500 null data sets for the likelihood-ratio test with nine starts each, and 200 data sets for the model-selection rates. On one laptop core that takes about two minutes. The site is static and deployed from web/.

Decision

Keep every seed and size in settings.ts. A script computes all results and writes __generated__/inference.json (about 90 KB). The page imports the artefact. artefact.test.ts recomputes the MLE, standard errors, bootstrap intervals, Wald coverage, model choice and convergence study in full, and checks the slow studies through small versions stored beside them with the same seeds. The bootstrap, the coverage study and the convergence study can be re-run in a Web Worker from the page.

Options considered

1. Compute at build time in Server Components. One source of truth, but two minutes on every build and on every dev reload of the page.
2. Compute in the browser on demand. Honest and interactive, but a visitor would wait minutes for the headline numbers, and the page would have nothing to show without JavaScript.
3. Precompute with a script and test the artefact (chosen), with in-browser re-runs for the fast parts.

Why

It is the same trade as the notebook parity data (DR-001): a committed, regenerable artefact plus a test that fails when the code and the artefact disagree.

What happened

• Generation takes about 130 seconds; the artefact test takes about 10 seconds.
• Re-running the bootstrap in the browser with seed 42 reproduces the published intervals exactly, which is the simplest possible reproducibility check for a reader.
• Same seed does not always mean the same last digit. Review found that Chrome 155 returns correctly rounded Math.exp and Math.log results where Node 26 sometimes differs in the last bit, and the numerical Hessian magnifies that to about 3 × 10⁻⁷ relative in standard errors and interval widths. The coverage re-run in Chrome therefore reported "different numbers" against an up-to-date artefact. The comparison now requires counts (coverage hits, failed fits, iterations) to match exactly and other numbers to agree to 10⁻⁵ relative, and the artefact test compares Hessian-derived values at 10⁻⁶ relative instead of 10⁻⁹, so a future Node with the newer maths library does not fail it for nothing.
• The weak spot: the slow studies are verified only through their small versions. A change that altered only large-sample behaviour would slip through until someone regenerated, and a regeneration with an edited setting would not be flagged unless the settings test noticed.

What I'd change

• Regenerate the artefact in a scheduled CI job and fail on any diff.
• Spread the slow studies across worker threads so the generator finishes in seconds and could run at build time.
• Compute the Hessian with automatic differentiation or analytic second derivatives, which would make the standard errors far less sensitive to the last bit of exp and log.

Context

The lab is a teaching tool, and a plain-language reading of "what just happened in this E-step and M-step" is useful. It is also a small, realistic test of governing generative AI: the explanation is about numbers, which language models are prone to invent. There is no budget for an API key, no server, and nothing on the site may depend on AI.

Decision

• Bring your own key, browser only. The visitor pastes an Anthropic (default, Claude Haiku 4.5; Claude Sonnet 5.5 optional) or OpenAI key. It stays in sessionStorage unless they choose "remember on this device" (localStorage), and "Forget key" removes it. Calls go straight from the browser to the provider. The key never reaches this site, a log or the repository.
• Grounded input. The request contains only the iteration's numbers: parameters before and after, responsibilities (all four in the stepper; a summary of the 200 in the playground), the log-likelihood before and after, and the stopping rule.
• Structured output. The reply must match a JSON schema (provider structured-output modes), validated again with zod against exactly the same contract. Display limits (long fields shortened, at most four caveats) are applied after validation, so a reply that obeys the schema is never thrown away for its length.
• Refusals. Claude Sonnet 5.5 requests opt into Anthropic's server-side refusal fallback (fallbacks: "default", beta server-side-fallback-2026-07-01): if a safety classifier declines, Anthropic re-runs the request on the model it recommends for that category within the same call, and the audit log records the model that actually answered. Haiku 4.5 does not take the option. A refusal that still comes back is shown as a plain "declined" message.
• Transparency. Every explanation is labelled "AI-generated", shows the model, latency and token usage, and runs a grounding check that lists numbers not found in what was sent. The check covers numbers with decimals, numbers above 10 and anything in scientific notation (1e-6, 1.5 × 10⁻⁵, 10^-6), matched at the precision they are written with; whole numbers from 0 to 10 are not checked, because they are usually counts, ratings or ordinals.
• Human in the loop and audit. The visitor accepts, edits or rejects each explanation. Every call is written to an IndexedDB audit log without the key, viewable and exportable (JSON, CSV) at /ai-log. That includes failures: a refusal, a reply cut off at the token limit or one that fails validation keeps its raw reply and token usage, because the visitor paid for it.

Options considered

1. No AI. Safe, but misses a useful explanation and a governance showcase.
2. A server proxy with my key. Costs money, invites abuse and puts a secret on a server.
3. An in-browser open model (WebLLM). No key needed, but a multi-gigabyte download for a paragraph of text.
4. Bring your own key, from the browser (chosen).

On refusals: leaving the fallback out would keep the browser request a little simpler (no beta header), but a visitor on Sonnet 5.5 would then pay for a declined request and get nothing back. Anthropic's CORS preflight accepts the anthropic-beta header, so the fallback costs nothing in the browser.

Why

It keeps the site free, static and fully functional without AI, and it makes the governance controls concrete: what is sent, what comes back, who decided what, all visible to the visitor. The design is informed by the Australian Government's policy for the responsible use of AI in government, the EU AI Act's transparency principles and the NIST AI Risk Management Framework; it is not a compliance claim.

What happened

• The provider adapters, error handling, key storage, audit log and grounding check are unit-tested with the network mocked (ai.test.ts): the key only ever travels in a request header, and a key echoed in a provider error is redacted before it is stored or shown.
• No live call was made during development or CI, because the project has no key. The request shapes follow the providers' documented APIs; a real call is the first thing to check after deployment.
• The grounding check is lexical. It catches an invented number, but not a correct number used in a wrong sentence, so it is a prompt for the human reviewer, not a verdict. Review found two gaps before release, both fixed with tests: it read "3.1e-2" as accurate to ±0.05 because it ignored the exponent, and it skipped "3e-2" and "10⁻⁶" as small integers.
• Review also found that the first version's zod schema was stricter than the JSON schema sent to the provider (length and item limits the providers do not enforce), so a valid five-caveat reply was discarded after the visitor had paid for it, and that failed calls were logged without their reply or usage. Both are fixed.

What I'd change

• Build a small evaluation set (iterations with known facts: did the log-likelihood rise, did the run converge, which component owns each rating) and score explanations against it, with intervals, per model.
• Offer an in-browser model as a no-key option once small models are good enough at this.
• Check numbers semantically as well as lexically: parse each sentence's claim ("the change is below the tolerance") and test it against the snapshot.

Context

A review before release checked the AI use statement against the code and the repository, and found three places where it said less, or more, than was true:

• DR-005 and /methods said the request contains "only the iteration's numbers". The snapshot also carries the page name, a one-line description of the data set (for example the seed of browser-generated ratings), its size, the run's stopping status and fixed notes on how to read the numbers. None of it is personal, but the statement was not literally accurate.
• With Claude Sonnet 5.5, Anthropic's server-side fallback can answer a declined request with another model inside the same call. The audit log kept one model field, overwritten with the model that answered, so an auditor could not see what was requested or that a fallback happened.
• /methods said every other text on the site "was written without" AI. The commit history shows the site's code and text were developed with an AI coding assistant. The runtime claim (only the optional feature calls a model) was true; the authorship claim was not.

Decision

• What is sent. SNAPSHOT_FIELDS in web/src/lib/ai/explain-iteration.ts lists every field in plain words, /methods prints that list, and a test fails if a stepper or playground snapshot carries a field that is not on it, or the list names one that is never sent.
• Which model answered. Each audit entry stores requested_model (what the visitor's settings asked for), model (what the provider reports) and fallback (true when a fallback content block or a fallback_message entry in usage.iterations shows the fallback ran). Both new fields are in the JSON and CSV exports and on /ai-log, and the explanation's label names the fallback when it happens. Entries written before this change have only model.
• How the site was built. /methods and the README say that nothing on the site calls an AI model at runtime except the optional feature, that the code and text were developed with an AI coding assistant (Claude Code), and that every number comes from the code, the tests and the seeded scripts.

Options considered

1. Trim the snapshot to numbers only, so the old wording became true. The page name and data-set line help the model explain the right thing (four hand-worked ratings, or 200 generated ones), and dropping them would make the explanations worse to save a sentence.
2. Keep one model field and append "(fallback)" to it. Simple, but it mixes two facts into one string that the CSV export and any later analysis would have to parse.
3. Separate fields and a field list tested against the code (chosen).

Why

A transparency statement that is wrong in small ways teaches readers to discount it in big ones. Each of these claims can be checked in a minute (by reading the request in the browser's network panel, or the repository's history), so each has to be exactly true. Tying the disclosure to the code with a test keeps it true when the snapshot changes. The design remains informed by, not compliant with, the frameworks named in DR-005.

What happened

• The three issues were found in review before release, not by a visitor. No live call has been made (the project has no key), so the fallback detection is tested only against mocked responses shaped like Anthropic's documented ones.
• The log view names the requested model only when the fallback ran. A provider that reports a dated model id (OpenAI often does) is therefore not mistaken for a fallback.

What I'd change

• Show the exact JSON that will be sent before the first call, with a "send" button, rather than only describing it on /methods.
• Record the fallback's switch points (which model declined, which continued) in the audit entry, not just a flag, if a fallback ever turns up in practice.