causalpilot

Reads an A/B test export, checks the traffic split, estimates the effect with CUPED and says whether to ship, hold or keep the test running, for analysts and engineers who have per-user test data but no experimentation platform. Give it a CSV with one row per user; it explains every call in plain words, on the command line or in the browser.

Try it in the browser (your file stays on your machine) · Report on a real e-mail experiment · Simulation benchmark

$ causalpilot simulate test.csv
Wrote 60,000 rows to test.csv (true lift +8.0%, synthetic data)
$ causalpilot analyze test.csv --group arm --control control --metric converted --metric revenue --covariate prior_orders --min-lift 0.02
test.csv: 60,000 rows, control = control
Sample ratio: treatment: 29,940, control: 30,060  (chi-square 0.24, p = 0.624)  OK

treatment vs control
  converted           8.95% -> 9.36%      CUPED +0.47 pp [+0.02 pp, +0.92 pp]  p = 0.0403  (+4.6%)
  revenue             3.217 -> 3.388      CUPED +0.1912 [-0.008097, +0.3905]  p = 0.0601  (+5.3%)
  -> SHIP: converted improves, though the size is uncertain.
     The 95% interval +0.02 pp to +0.92 pp excludes zero; the point estimate is +0.47 pp.
     The low end is below the smallest change you said is worth shipping (+0.18 pp), so a repeat with more traffic would pin the size down.

Note: These intervals assume the sample size was fixed in advance and the results were read once. If you checked repeatedly and would have stopped on a good day, rerun with --peeking, which uses intervals that stay valid under continuous monitoring.
Note: CUPED is only unbiased if the covariates were measured before assignment (prior_orders).

Run the same file with --peeking (because, honestly, you looked every morning) and the call changes:

  -> HOLD: The test cannot detect a worthwhile effect on converted at this traffic.
     The always-valid 95% interval -0.22 pp to +1.16 pp contains zero and is too wide to rule out the smallest change you said is worth shipping (+0.18 pp).
     Detecting that change with 80% power would need about 13x the current sample, which is more than the 4x limit for extending a test (--max-extend).

Why I built it

Most A/B tests I have seen analysed were a t.test and a screenshot. The p-value was the least of the problems: arms that were not the size they should be, results read every day until they looked good, and nobody writing down beforehand what lift would be worth shipping. causalpilot does the boring checks in a fixed order and refuses to call a winner when the data cannot support one.

Install

pip install git+https://github.com/appleweiping/causalpilot

Python 3.10 or newer; it depends on NumPy, pandas and Matplotlib. The browser version needs nothing: open the demo page and drop in a CSV.

Using it on your own export

The CSV needs one row per randomized user, a column naming the arm, and one or more outcome columns (0/1 or numbers). Columns measured before the test started (past orders, country, platform) can be passed as covariates.

causalpilot analyze export.csv \
  --group variant --control A \
  --metric purchased --metric revenue \
  --covariate orders_last_90d --covariate country \
  --guardrail refunded:lower \
  --min-lift 0.02 --split 50:50 \
  --out report/

• The first --metric decides the call; the others are reported alongside it.
• --guardrail metrics must not get clearly worse. Higher counts as better unless you add :lower, so write refunded:lower for refunds, cancellations, errors or anything else where a rise is bad. The browser version has a direction menu next to each guardrail.
• --min-lift is the smallest relative lift worth shipping (0.02 = 2% of the control mean; default 1%).
• --split is the planned split, control first (default: equal).
• --out writes report.md, report.json and effects.png; --json prints everything the report is built from.
• Add --peeking if you looked at the results more than once while the test ran, and --planned-n with the sample size you planned, if you had one.

Real data: the Hillstrom e-mail test

In 2008 Kevin Hillstrom published an experiment on 64,000 customers of an online retailer for the MineThatData E-Mail Analytics challenge. A third got an e-mail featuring men's merchandise, a third one featuring women's merchandise, and a third no e-mail; visits, conversions and spend were tracked for two weeks.

causalpilot hillstrom --out examples/hillstrom

downloads the CSV from the author's site (over plain HTTP, since the site's certificate did not validate for me, so the file is checked against a pinned SHA-256), uses past-year spend, recency, past purchase categories, zip-code type and channel as CUPED covariates, and treats a 5% lift in spend per customer as the smallest change worth an e-mail campaign. The full report is checked in:

vs. no e-mail	Spend per customer	95% interval	Visit rate	Call
Womens e-mail	+$0.42 (+65%)	+$0.17 to +$0.68	+4.5 pp	SHIP
Mens e-mail	+$0.77 (+118%)	+$0.48 to +$1.05	+7.6 pp	SHIP

The sample ratio check passes (21,387 / 21,306 / 21,307, p = 0.90). Two things I found worth noting:

여기까지가 README 앞부분입니다. 나머지는 원문에서 읽을 수 있습니다. README 전문 보기 ↗