Tale & Theory

Books / Learning comic

The Experiment with Two Winners

Theme: Business & Management

Two victory cakes arrive at the same product meeting. Dev believes the old page has won; Lena says her new design is the clear winner. Analyst Mira discovers that their confident announcements hide different outcomes, different denominators and even different populations. The team reconstructs a real experiment from assignment records and paid orders, then explores an older campaign table whose overall rates reverse its device-level pattern. When the planned final result arrives, purchases improve but checkout errors fail a promised guardrail. Through visible arithmetic, an uncomfortable defect and a decision record anyone can follow, the colleagues learn that honest experimentation is not a contest between charts. It is a way to decide what the evidence can support, and what to test next.

Start reading →View in the interactive library
The Experiment with Two Winners cover

About this book

At a small online design-template shop, product manager Dev and designer Lena arrive at a meeting with cakes celebrating opposite winners. Dev has a report favouring the existing product page. Lena has a click count celebrating a new gallery design. Data analyst Mira does not choose the more persuasive presenter. She asks what decision the team wanted to make and retrieves the plan written before the experiment began. The first chapters establish the customer journey and the difference between a primary outcome, a diagnostic measure and a guardrail. The primary measure is a confirmed paid order within seven days among eligible assigned accounts, counting each account once. Mira insists on both parts of every fraction. Repeated clicks are not distinct people; people who never click still belong in the denominator. An early illustrative snapshot has 120 buyers among 2,000 accounts in A and 140 among 2,000 in B. The team separates a one-percentage-point difference from a roughly 16.7-percent relative increase without pretending that either description settles the experiment. Dev’s apparently contradictory spreadsheet turns out to describe earlier nonrandom campaigns. Its device mix differs sharply between versions. B has a higher purchase rate within both mobile and desktop groups, while A has the higher overall rate because it received far more high-converting desktop traffic. The story shows the raw numerators and denominators that make this Simpson-type reversal possible. Equal weighting answers a different question, but Mira is explicit that reweighting the old campaigns cannot remove unknown confounding or transform them into a randomized experiment. Engineer Jon joins the investigation. Together they verify persistent account-level assignment, reconcile payment notifications with a confirmed order ledger, and distinguish errors from purchases. An account can encounter a checkout error and later buy, so those measures are not mutually exclusive. The team waits for the planned sample and complete seven-day observation window. It also keeps operational safety monitoring separate from the temptation to stop whenever a daily line looks favourable. The final fictional sample contains 20,000 accounts per version. Purchase rates are six and seven percent, with a positive approximate uncertainty interval under the stated simple analysis. However, the proportion encountering a checkout error rises from one percent to 2.2 percent and fails the predeclared tolerance. A credible benefit does not override a failed guardrail. The team refuses to relabel that failure or search for a more flattering metric. A browser-specific pattern leads Jon to reproduce a blocked checkout button. Lena can repair the layout, but the modified page cannot inherit the old treatment’s effect estimate. The group records a decision not to launch the tested version and prepares a fresh evaluation of the repair. By the final meeting, neither colleague needs to defend a cake. Their useful achievement is a reproducible explanation of the question, evidence, uncertainty, harm and next step. All business figures and characters are fictional; the book introduces reasoning habits rather than offering universal statistical thresholds.

Published chapters

  1. 01 - Two victory cakes
  2. 02 - The fraction behind the headline
  3. 03 - The crowd changes the score
  4. 04 - Checking the machinery
  5. 05 - A lift and a warning
  6. 06 - A decision everyone can read