Updated July 2026

We Built an Instrument to Prove Our Recommender Doesn't Learn

Scouly ranks small businesses for people trying to buy one. Out of the box everyone sees the same order — companies sorted by an objective 0–100 score computed from public records. The personalization engine's job is to make each user's feed theirs: learn from what they click, save, pursue, and reject, and reorder accordingly, while being able to explain every decision in plain language.

We shipped a version of that engine. It passed 33 unit tests. It survived a sixteen-agent adversarial review that found and fixed twelve real issues. Every formula was checked by hand.

Then we built the thing that could tell us whether it actually learned, and five of our load-bearing beliefs died in the first week.

This is the record of what broke. The specific bugs are interesting mostly as instances of general patterns, so I've tried to name the pattern each time.

The constraints that shaped everything

Before the failures, the box we were working in — because most recommender advice assumes the opposite of all four:

Tiny data. A real user produces 20 to 500 lifetime events. Not 20 million. Every technique had to be chosen for statistical sanity at n=50. This rules out most of what you'd reach for by default: anything with many free parameters will memorize noise at this scale.

No ML infrastructure. Everything runs in the user's browser, in TypeScript, in milliseconds. No training servers, no GPUs, no feature store, no nightly pipeline.

Explainability as a product requirement. The UI shows the reasons on screen — "ranked up because you saved 12 HVAC companies." Every ranking decision has to trace to named evidence.

Determinism. Same event log plus same calendar day must produce exactly the same feed, including its exploration. This makes every historical state replayable and every bug reproducible.

Given tiny data, we made the deliberate bet to use the simplest statistics that could possibly work: per-token Beta-Binomial evidence counting — a 300-year-old technique — rather than embeddings or a neural net. Each company is decomposed into 7–9 string tokens (vertical:hvac_plumbing, state:TX, age:20-30, loan:500k-1m, and so on), and the model keeps a running tally of evidence for and against each token, with old evidence decaying.

The reasoning held up: at 20–500 events, a per-token evidence tally with a sensible prior is the strongest model that cannot overfit badly, and it's perfectly explainable. The model's "weights" are literally "how much evidence have I seen that you like Texas."

That part was right. Almost everything else we believed was wrong.

Defining "self-learning" so it could be falsified

The phrase "self-learning" is usually marketing. We wrote down what would have to be true for the claim to be real:

  1. A signal path — user behavior becomes recorded evidence automatically.
  2. A model — that evidence updates an internal representation of taste.
  3. A behavior change — the updated model changes what the product actually does next, not just what a dashboard says.
  4. A closed loop — the changed behavior produces new user behavior, which produces new evidence.
  5. Proof — an instrument exists that could show the system is not learning, and it currently shows the opposite.

Point 5 turned out to be the most important decision in the project. Most of what follows was discovered because we built the instrument and it immediately told us things we did not want to hear.

The instrument

Two questions had to be answered before we could measure anything.

How do you evaluate a system that learns online, from a stream, when you can't run an A/B test? The stream-learning literature has a standard answer: prequential evaluation, also called test-then-train. Walk the recorded events in time order. Before the model consumes an event, ask it to predict: here are all the companies this user hasn't engaged yet — rank them; where did the user actually commit? Then reveal the event, let the model update, continue. Every event is used once for testing and once for training, in the only order that never cheats. Early predictions are legitimately bad because the model is cold, so you report the learning curve, not one average.

What are the known traps? The big one is algorithmic confounding (Chaney, Stewart & Engelhardt, RecSys 2018): if the events you evaluate on were themselves produced under the influence of the algorithm you're evaluating, the algorithm looks better than it is — it gets credit for predicting choices it caused. Their paper also showed this feedback loop homogenizes user behavior without adding real utility.

Two defenses followed. When comparing policies by replay, only use event streams logged under uniform exposure, so no candidate policy shaped the data. And when evaluating the real closed loop, don't score by prediction at all — score against ground truth. Which requires simulated users whose true preferences we wrote down ourselves and therefore know.

So we built simulated users with explicit utility functions — a written-down number for how much they truly like any company — and had them behave like people: browse, view, save what they like (with realistic noise), advance favorites through pipeline stages, remove mistakes with reasons, sometimes change their minds entirely mid-search. Because we know their true taste, we can compute exactly how good any feed is for them, with no confounding:

Two more design decisions mattered more than they look. Test points are first commitments only — the first save of a company, ranked among companies not yet engaged. Predicting that a user will advance a company they already saved is trivially easy and inflates every method equally; predicting the next commitment among hundreds of untouched candidates is the real job. And every policy variant runs with internal diagnostics — is the regression layer even training? did drift ever alarm? — because a layer that silently never fires looks identical to a layer that works, unless you count.

One methodological rule: every new mechanism must beat its own absence in the harness, or it gets demoted to documentation. Several did not survive.

Failure 1: the scorer was silently dead for exactly the users it knew best

What we believed. A code comment asserted that taste log-odds sums land roughly in [-4, 4] for a warm model. So the score squashed the taste sum with tanh(sum / 3).

What the instrument said. The engine barely beat random at predicting a simulated user's next save: 56% percentile against 50% for a coin flip, with a hit-rate-in-top-10 of 5.5%.

That could have meant the task was impossible rather than the engine being broken — so we added an oracle baseline, a ranker that cheats by reading the simulated user's true taste directly. The oracle scored 85.5% percentile and 36.4% hit@10.

The task was very possible. The engine was broken.

The diagnosis. We printed actual taste sums for a warm model across a 200-company pool. The measured range was 9.4 to 23.0 — not [-4, 4]. tanh(x/3) of anything above 9 is essentially 1.0, so 100% of the pool was pinned at the same taste value. The taste term contributed nothing to ranking once the model was warm. The personalization engine was silently inert exactly when it knew the user best, and everything degraded to the base score.

The same diagnostic surfaced a second problem: correlation between a company's taste sum and its number of tokens was 0.87. About five of every company's ~8 tokens (age band, size band, quality band, signals) are taste-neutral but activity-positive — every active user's saves carry them, so their evidence grows positive for everyone. A company with more tokens had collected more of this free positive evidence. The engine was substantially ranking companies by how many attributes they had.

The fix, in two parts: use the per-token mean affinity instead of the sum, which kills the length bias; and standardize taste values across the candidate pool at ranking time (subtract pool mean, divide by pool standard deviation), which makes ranking immune to any overall scale drift, because shared-token inflation shifts everyone equally and cancels.

After: warm prediction 56% → 73% percentile against the 85% oracle ceiling; hit@10 5.5% → 22% against a 36% ceiling. Personalization was alive.

The general lesson. A scale assumption inside a squashing function is a silent killer. Nothing crashes. Every unit test passes. The formulas are all "correct." The system just quietly stops doing its job at the moment it has the most information — and only an end-to-end instrument with a ceiling baseline can see it. Without the oracle we would have concluded the task was hard.

Failure 2: the drift detector could not detect drift

What we believed. The detector would notice a mid-search taste pivot. It watched the last 10 strong saves and counted how many the long-term model would not have recommended (scoring below half the user's historical median). Six or more disagreements out of 10 triggered an alarm.

What the instrument said. We simulated the hardest reasonable pivot — 200 events of consistent taste (HVAC in Texas), then a total flip (landscaping in Florida) — and stepped the detector through it. Disagreements: zero. Not "too few." Zero, at every checkpoint, while the user's last twelve saves were visibly landscaping and Florida in the event log.

The diagnosis. We decomposed the score of a post-pivot save under the historical model. The landscaping token contributed 1.0 (weak, correctly). Florida contributed 1.3. But its age band contributed 4.0, size band 2.9, quality band 4.0, and signals 7.2. The taste-neutral tokens — the same shared-inflation phenomenon from Failure 1 — vouched for any active company so strongly that no realistic save could ever score below the threshold. The detector had been structurally blind since the day it shipped.

Fix 1, kind-centering: score each token as its affinity minus the average affinity of its kind (all age bands, all size bands, and so on). Shared inflation lives inside each kind, so centering within kind cancels it exactly, leaving only genuine discrimination. After this change the same pivot showed 10 disagreements out of 10 at the pivot point.

But the alarm still didn't fire, because of a second gate. The design required at least two "drifting tokens," defined as tokens where the short-term and long-term views point in opposite directions. We measured affinities through the entire pivot: not a single token ever flipped sign. The reason is that our evidence is presence-only — tokens gain positive evidence when they appear in saves, and there is almost nothing to push them negative, because users don't downvote. Every affinity stays positive forever. The gate was checking for a signature that cannot exist in this kind of system.

Fix 2, the relative-gap outlier test: we printed the (short view minus long view) gap for every token during the pivot. The population of gaps sat around -1.5 — short views are systematically smaller, since they hold less evidence — but the incoming-taste tokens (landscaping, Florida, Tampa, Orlando) stood +2 to +2.5 above that median. That relative outlier is the real signature of incoming taste. New rule: a token is drifting if its gap exceeds the median gap by 2.0.

After: the detector alarmed exactly at the pivot, named exactly the incoming tokens, and stood down once the new taste was absorbed. All 33 pre-existing unit tests still passed — which is the point.

The general lesson. Sign-flip tests cannot work without negative evidence. If your system is presence-only, drift lives in relative gaps, not sign changes. More broadly: a threshold calibrated against one population silently becomes unreachable when a different population dominates the statistic.

Failure 3: the slow learner was an anchor against change

We had a ridge logistic regression layer for credit assignment — it sees all tokens of an event together, so if Dallas alone explains the saves, Texas's weight falls back toward neutral rather than being double-counted by every Dallas save.

What we believed. Blending it with the fast Beta engine as a weighted average was principled: taste = (1-g) × fast + g × slow, where g grows with data volume (about 0.8 for a warm user).

What the instrument said. Even after the first two fixes, the post-pivot feed adapted far too slowly. We probed directly: rank the full pool at different moments and measure the true (new-taste) utility of the top 20. Sixty events after the pivot, the top-20 was still worse than the pool average. The feed was still serving the abandoned taste.

Why. With g = 0.8, the fast adaptive engine kept only 20% of the vote exactly when the user had the most history. And the regression layer is trained under the slow 180-day decay, centered on the long-term profile — it re-learns a pivot over months. The mathematically elegant blend was, in dynamical terms, an anchor chained to the past.

Fix attempt 1 (failed, instructively): make the regression purely additive — taste = fast + g × (regression's learned delta from its own starting point). Adaptation was restored (post-pivot top-20 went from stuck at -0.31 to +0.06 within the run), but the stable user's prediction accuracy dropped 7 points. Cause: the old (1-g) factor had been quietly shrinking the exploration noise. Removing it let Thompson sampling's randomness through at full strength all the time.

Fix attempt 2 (landed): fold the regression into the long-term view per token, as a mean shift that preserves sampling variance:

long_view_sample = sample + g × (regression_weight − long_mean)

This moves the center of the long view toward the regression's refined estimate without touching the spread. The short-term view passes through untouched, so recent behavior can always outvote history. After: post-pivot top-20 true utility reached +0.80 against a pool average of -0.10, and the stable user's accuracy came back.

The general lesson, and the one we think is most transferable: where you place a slow learner inside a fast system matters more than how good the slow learner is. A weighted average hands the slow component veto power precisely at high data volumes. Composing them as "fast + slow correction," where the slow learner refines only the slow pathway, preserves adaptivity. We found no prior write-up phrasing it this way.

A corollary from the failed attempt: exploration noise can hide inside another component's blend factor. Removing (1-g) didn't just change the average, it un-damped Thompson sampling. Variance bookkeeping across composed randomized components is its own discipline.

Failure 4: nothing ever demoted a company you'd already passed on

What the instrument said. In the closed-loop simulation — the real product condition, where the feed shapes what the user sees — every policy's discovery-feed quality decayed over time, eventually below random. Including ours.

Why. The user picks the good companies out of the feed and saves them, which removes them from the discovery pool. What accumulates at the top is companies the user has seen and repeatedly declined — and in our signal model, a view is weak positive evidence, because they clicked into it. Nothing in the system could ever push a "seen it, passed on it" company down. Real recommender systems handle this with impression discounting; we had none, because we logged clicks but not impressions.

The fix used data we already had: a company whose first view is more than 10 days old, with no commitment since, is "seen and passed" — demote it by 0.12 × blend (scaled so a cold model demotes nothing). Ten days rules out "still thinking about it." The magnitude pushes it down a screen without burying it, because passing is weaker evidence than removing.

Stable-user feed capture improved from -24.7% to -9.7% immediately, and reached positive territory with the later layers.

The general lesson. In a closed loop, "no negative signal" is not neutral — it is a slow poison. Any finite pool plus a feed that removes what the user takes will fill the top with rejects unless something actively demotes them.

Failure 5 (a useful "no"): cross-features don't pay at this data size

The regression layer's code comment claimed it learns feature interactions like "likes HVAC and likes Texas but not HVAC-in-Texas." Reading the math carefully: it cannot. It's linear over single tokens, with no conjunction terms.

So we tested whether adding real conjunction features (vertical × state cross tokens) would pay. We built a simulated user whose taste is a pure conjunction, added the cross tokens as an engine variant, and ran the ablation.

Result: no measurable lift for the conjunction user, and nothing for anyone else. At 20–500 events there is simply no evidence budget to fund interaction features; they fragment the data. The variant stays in the harness as a standing experiment, the misleading comment was corrected, and production featurization stays single-token.

Sometimes the instrument's most valuable output is a confident "no." Without the harness this would have been a plausible-sounding roadmap item forever.

Two more things measurement changed

Drift detection is transparency, not recovery. With everything above fixed, we could finally ablate the drift machinery honestly. Removing the short timescale cost about 21 points of feed capture — the two-timescale blend is what actually recovers from pivots. Removing drift detection entirely cost about 4 points. So the drift layer's honest job description changed: it's the mechanism that lets the product say "your taste shifted — I noticed," plus insurance for very long histories. It is not the thing that makes pivots work. We rewrote the documentation to match. Claims should match ablations, not intentions.

Depletion mimics drift. Once a later layer raised personalization strength for well-predicted users, the drift detector started alarming on a stable user — 29 alarms in one 110-day run. No pivot existed. The cause was exploitation-induced depletion: a strong feed finds the user's true matches fast, the user saves them, the remaining pool holds only second-tier matches, their forced saves score below their own historical median, and the statistic fires. Exhaustion reads as change.

The distinguishing feature: a real pivot concentrates its disagreeing saves on the same few incoming tokens; depletion scatters them across unrelated tokens. So we added a coverage gate — the alarm only fires if the identified drift tokens appear in at least 60% of the disagreeing saves. The false alarm died; the true pivot was untouched.

Recorded honestly: under deep depletion a stable user's revealed preference genuinely does concentrate on second-tier taste, so an occasional bounded confirmation there is arguably correct behavior. The effect appears at 120–200 company pools under aggressive exploitation; production pools are thousands of companies, far outside that regime.

The exploration trade, measured and accepted

The no-Thompson ablation — rank on average beliefs, never sample — is consistently about 10 points better at pure prediction accuracy, and 25–50% worse at closed-loop harvest, the metric that represents actual user value.

Exploration deliberately sacrifices prediction to keep discovering; that is its entire purpose. We keep Thompson sampling on and record the trade here so nobody "optimizes" it away by staring at the prediction number alone.

Testing against people, not just distributions

The simulated users above are utility functions. To test against genuinely different kinds of person, we tried something else: six LLM agents, each briefed to invent a complete person — 600–900 words of life story, childhood, hometown, at least two formative events that directly shape their business taste, why they're buying a business — and then translate that psychology into the engine's actual feature vocabulary: a numeric weight for every token they care about, conjunction pairs, an "idiosyncrasy" level, and a decision temperature. One iron rule: every nonzero weight must trace to a named event in the life story.

The split worked because it follows a natural seam — the agents provide the psychology, the math provides the speed. (Having an LLM click through a feed one company at a time would cost hours per simulated month and add nothing measurable.)

The engine beat the unpersonalized feed for all six personalities, on both prediction and harvest. Two results matter most:

Wade is a contrarian bargain hunter whose father bought a "dying" machine shop in 1987 that outearned every flashy family bet, and who as a bank credit analyst approved a top-scored deal that defaulted. His operating theory: a high quality score measures the auction, not the business. He weights the base score negatively. For him the default product is actively harmful — his unpersonalized feed scored worse than random (33%, feed capture -90%), because everything it promoted was exactly what he avoids. The engine learned the inversion from his behavior and delivered +172% harvest. No amount of improving the base score could ever have served this user. Only learning could.

Devon is an indecisive first-timer with weak preferences and huge noise — near-unlearnable by construction. An aggressive learner would chase his noise. The engine's self-skill layer kept it humble: small personalization weight, +25% from the faint real signal, no damage.

Wade is the proof the system learns the person. Devon is the proof it knows when not to.

LLM agents turned out to be excellent at authoring evaluation personas. The weight-traces-to-story requirement produced test users of a richness we would never have hand-written — a contrarian whose distrust of scores comes from a specific 2007 loan default he approved — at the cost of a few minutes of parallel agent time. And they found real behavior differences.

The harness became a runtime organ

The last layer is the one we like most, and it fell directly out of having built the instrument.

The engine now runs a miniature prequential replay on itself every time it ranks. Take the user's last 10 first-commitments; for each, rebuild the model from only the events that preceded it, and ask "where would I have ranked this company the day before they saved it?" The average percentile is the engine's demonstrated skill for this specific user — not confidence, not evidence volume: an actual verified predictive track record.

That skill then sets how much the taste model is allowed to influence the feed. A proven model personalizes harder than the old fixed cap ever allowed (up to 0.85 of the ranking weight); an unproven or coin-flip model retreats toward the objective score (floor 0.35). The mapping is shrunk toward a prior chosen so that a brand-new model reproduces the old constant (0.55) exactly — zero behavior change until evidence argues otherwise. Do-no-harm, enforced by measurement instead of hope.

One honest bias: skill is measured on commitments the feed itself influenced, so it can flatter itself — the standard exploitation loop. It's bounded by the ceiling and floor, exploration keeps feeding it off-model evidence, and the ground-truth closed loop is the watchdog for the degenerate version.

We suspect this pattern — self-measured skill gating how aggressively a personalization system acts — generalizes well beyond this project.

The short list

  1. Shared-token inflation is the silent killer of presence-only preference systems. When users generate only positive evidence, every attribute that co-occurs with activity inflates, and any statistic that sums affinities becomes dominated by attributes carrying no taste information. It broke our scorer and our drift detector in different ways before we saw the pattern. The general fix is relative measurement: center within attribute kind, standardize within candidate pool, compare gaps to their median.
  2. Sign-flip drift tests cannot work without negative evidence.
  3. Where you place a slow learner matters more than its quality. Slow learners should refine the slow pathway, never gate the fast one.
  4. Exploration noise can hide inside another component's blend factor.
  5. Depletion mimics drift. Concentration versus scatter separates them.
  6. The evaluation harness makes a good runtime organ.
  7. A layer that silently never fires looks exactly like a layer that works. Count the firings.

The meta-lesson is the boring one, and it cost us the most to learn: every belief above survived unit tests and a sixteen-agent adversarial review. They died only under end-to-end measurement against a ceiling baseline. Reviewing formulas tells you whether the math is correct. It cannot tell you whether the system learns. Only an instrument that could have proven you wrong can do that.

Scouly is an off-market deal-sourcing tool for people buying small businesses — it ranks companies from public records (SBA loan filings, business registries, market fragmentation) and explains every point of every score. The engine described here is what makes that ranking personal. Informational tool only — not investment, legal, tax, or brokerage advice.

Reproduce the numbers in this post with npm run learn:eval and npm run learn:personas.

Scouly finds off-market businesses from public signals — see the live feed.