BetterRankings

BetterRankings estimates how a new ranking policy would perform from logged data alone, without running a live A/B test. It then checks those estimates against a simulated online test where the true answer is known.

Live app: https://devfrancisa.github.io/BetterRankings/

It ships as two things that compute the same numbers:

• a Python package, betterrankings, with a command line tool;
• a static web app (TypeScript, no server, no tracking) that runs the same simulator and estimators in a Web Worker in your browser.

All built-in data is simulated.

The idea in brief

A ranking system shows each query a list of items, and people click some of them. Suppose the current policy shows lists with some randomness and records the probability it gave to each list it showed (the propensity). Then its logs can be reweighted to answer a different question: how many clicks would another policy have earned? This is off-policy evaluation.

The simulator knows every true click probability, so it can also run the online test exactly. That gives a known answer to judge each estimator against.

Run it locally

Python package

Requires Python 3.10 or newer.

pip install -e ".[test]"

# Write a simulated log
betterrankings simulate --queries 5000 --exploration 0.2 --seed 7 --out log.csv

# Estimate one policy (or --policy all) from a log CSV
betterrankings evaluate --log log.csv --policy relevance --clip 20 --json results.json

# Simulate, estimate every candidate and compare with the simulated online test
betterrankings compare --queries 5000 --exploration 0.2 --seed 7

# Check the estimators on real logs (see "Real data" below)
betterrankings obd --data open_bandit_dataset --campaign every

pytest

Policies: current, popularity, relevance, relevance_soft, noisy_logging, reversed. Use target_propensity to evaluate the probabilities in your CSV's target_propensity column. --clip none turns clipping off.

The package is also importable:

from betterrankings import run_comparison, evaluate_frame, read_log_csv

cmp = run_comparison(n_queries=5000, exploration=0.2, seed=7)
print(cmp.table())               # each estimator next to the true value
print(cmp.rank_agreement())      # policy pairs each estimator orders wrongly

ev = evaluate_frame(read_log_csv("log.csv"), ["relevance"], clip=20)
print(ev.policies["relevance"].estimates["doubly_robust"])

Web app

Requires Node 20.19 or newer (22.12+ for the tests).

cd web
npm install
npm run dev       # http://localhost:5173/BetterRankings/
npm test          # unit tests plus parity checks against the Python fixtures
npm run build     # static site in web/dist

The web app has five parts:

• Playground: set the logging policy's exploration, the number of logged queries and a seed, pick a candidate, and run.
• Results: each estimator's estimate and 95% interval on a shared axis with the true online value. A clip slider updates clipped IPS and doubly robust live.
• Ranking: all candidates ranked by each estimator against the true order, with the candidates worth a live test marked.
• Diagnostics: the weight distribution, effective sample size, plain-language overlap warnings, and clicks by position.
• Your log: upload a CSV and evaluate a chosen target policy. The file is parsed in the browser and never leaves your device.

Log CSV format

One row per query. Positions count from 1 at the top of the list, and lists may hold 2 to 6 items. A sample file is at web/public/sample-log.csv, and the site offers it as a download.

Column	Meaning
query_id	Any identifier
ctx_1 .. ctx_3	Query context features
item_K	Id of the item shown at position K
click_K	1 if that item was clicked, else 0
item_K_pop	Popularity feature of that item
item_K_f1 .. item_K_f3	Other item features
propensity	Probability the logging policy gave the whole ranking that was shown (greater than 0, at most 1)
target_propensity	Optional. Probability your own target policy gives the shown ranking

The shown items are treated as the candidate set: a target policy re-ranks exactly those items. The built-in policies score items with fixed weights from the simulator, so they expect 3 context features and 3 item features. To evaluate your own policy, supply target_propensity. The reward model and doubly robust estimators need the target's probability of every possible ranking, so they are not available in that mode.

The simulator

• Catalog: 200 items. Each has a popularity score and 3 features.

• Queries: each query has 3 context features and 4 candidate items drawn from the catalog. A ranking orders all 4, so there are 24 possible rankings, and every one is enumerated exactly.

• Click model: a position-based model. Item j at position k is clicked with probability theta_k * sigmoid(0.6 * pop_j + x' W v_j - 1), where theta_k = 1 / k. Attention falls with rank, and relevance depends on the query and the item.

• Logging policy: with probability 1 - epsilon, Plackett-Luce over a weaker scoring model; otherwise a uniformly random ranking. epsilon is the exploration level. The log records the exact probability of the shown ranking.

• Candidate policies:

  Policy	Behaviour
  Popularity	Greedy sort by popularity
  Relevance model	Greedy sort by a good but imperfect relevance model
  Soft relevance	Plackett-Luce over the same model
  Noisier logging	More exploration and a flatter temperature
  Reversed relevance	Deliberately worse
  Current policy	The logging policy itself, as a sanity check

This is the beginning of the README. Read the rest at the original. Read the full README ↗