Darklytics · the darkstore lens for quick commerce

One day of Blinkit, collected by hand: 21,949 listings in Bhubaneswar, and four things the data itself revealed. A data defect the anomaly detector caught on its own, stock-outs that cluster by product, prices labelled observed that were looked up on the web, and a store column that carries no information. One day cannot show how prices move, so a simulated 30-day market (the Lab) tests the rest of the engine: a leakage-safe feature layer, forecasting models held to baselines, a price simulator and optimiser, and an analyst that cites a source for every claim. Everything runs in a storefront-style web app in your browser.

Open the live site · What it does · Results · How it works · Run it locally

What is this?

Quick-commerce apps promise groceries in ten minutes, and the same product can cost a different amount on each of them. This project asks three questions a seller or a shopper would ask: who is cheapest, what is in stock, and how fast does it arrive. It answers them with a pipeline that runs from raw observations to a web app:

1. Collect and validate: 610,789 price, stock and ETA observations across 4 apps, 30 stores and 12,188 catalog products, frozen and hash-checked.
2. Engineer features without leaking the future: 109 columns, every one computed from data up to and including the moment it describes, never after.
3. Model: product matching, next price, price direction, stock-out risk, delivery time, anomalies, survival and restock models, competitor price response, assortment gaps; and, on the observed Blinkit day itself, a pack-size fixer, a category checker and an expected-price model.
4. Decide: a simulator that tests a price move before you make it, and an optimizer that finds the smallest move that reaches a target.
5. Explain: an AI analyst that answers in plain words, shows the source of every claim, and refuses questions the data cannot answer.
6. Ship: Darklytics, a storefront-style website that puts all of it one click away.

Method quality over headline numbers. The project's most important rule: one day of real Blinkit data was observed; the 30-day, four-app panel is simulated, and every number says which it is. Read how.

What you can do on the site

Three models on the observed day

Each runs on the 21,949 listings collected by hand, one row per product, with products held out by name so no near copy sits on both sides.

Model	What it answers	Held-out result
Pack-size fixer	What size is this pack, really?	Corrects the 819 rows whose stored size disagrees with the pack text; rows that looked abnormal against their peers fall from 137 to 3
Category checker	Does the name read like the shelf it is on?	91.6% accuracy from the name alone, against 59.1% for the brand's usual category; 172 products flagged for a look
Expected price	What do similar products cost?	29.7% median error, against 38.3% for the brand-and-subcategory median; its 80% band holds 82% of products

A price flag is 1.8 times as likely as an in-range product to be a misfiled one: the models check each other. Details in docs/PACK_FIXER_v1.md, docs/CATEGORY_CHECK_v1.md and docs/EXPECTED_PRICE_v1.md.

You decide, the models suggest

The Review page holds the two jobs a model cannot finish alone, across all four apps.

• Catalog fixes. The pack-size fixer and the category checker raised 255 flags. An AI reviewer (Claude) read each one and left a suggestion with its reason: 109 category moves, 52 keeps, 9 "not sure", and 85 on pack texts. Nothing changes until the owner takes a suggestion, keeps the filing or types their own value. Filter by app: Blinkit 255, Swiggy Instamart 50, Flipkart Minutes 48, BigBasket 45, since one catalog serves all four.
• Product pairs. Label pairs as the same product or not, and a live scoreboard shows how matcher v1 and v2 agree with you, with an honest 95% range. A fixed batch of 297 pairs scores the matcher; a separate batch chosen by active learning teaches it, and the two never mix, because pairs a model picks can never mark its own work.

AI suggestions are never labels: no module that trains or labels reads them, and a test keeps it that way. Does choosing pairs that way save labels? Tested where labels exist (the category task, 3 seeds): no saving below about 1,000 labels, then 90% of the full score at 3,600 labels instead of 5,000. So: score first, then teach. Details in docs/CATALOG_REVIEW_v1.md and docs/ACTIVE_LEARNING_v1.md.

SQL on every row, in your browser

The SQL console runs real SQL on all 610,789 rows of all four apps, plus what the models say about each product, with DuckDB compiled to WebAssembly. There is no server, so it works on the live site, and nothing you type leaves the page. Every row carries source, either observed or lab, so a query can always tell the Blinkit day from the simulation. Ready-made examples, a CSV download, and the first 1,000 rows of any result. See docs/SQL_CONSOLE_v1.md.

From the basket to Blinkit

Pick the app in the basket and every item gets its own button. Add on Blinkit opens Blinkit's own search for that exact product and pack in a new tab, and you tap Add there to put it in your Blinkit cart. The same works for Swiggy Instamart, Flipkart Minutes and BigBasket. No website can write into another app's cart, so the hand-off stops at the app's search, and the basket says so; the price there is that day's, which can differ from the one in the basket.

• Light and dark themes, phone, tablet and wide layouts; installable on a home screen, and it keeps working offline once loaded (a service worker, network-first so you always get the newest build).
• Hover a product and it lifts, its picture turns and the add button rises; a light follows the pointer across the mode cards; sections ease in as you scroll; headline numbers count up. All of it switches off for anyone whose system asks for reduced motion.
• Real product photos from Open Food Facts (openly licensed, CC BY-SA, credited on the Data & trust page), used only when brand and name match closely. Where none exists, a photo of the product's kind (kaju katli, marigold garlands, sweet corn) from Wikimedia Commons, each chosen by eye with no brand on it, free-licensed, credited, and labelled "Typical look" on the card. Everything else keeps a picture drawn from its name, pack and category. Photos from Google or the apps themselves are not used: they belong to the brands and the apps.
• New here? Home opens with three things to try, and every Simulate and Optimize page has a three-step guide and a "what do these words mean?" glossary.

The data, honestly

Real observations cannot be collected from these apps programmatically: there are no public APIs. The Blinkit data exists because a person collected it by hand. The other three apps, and Blinkit's history, are simulated, with the owner's informed consent, and every row says which it is:

Platform	Rows	What it is	Tag on the site
Blinkit	21,949	observed on one day, collected by hand	Observed
Blinkit	148,740	simulated 30-day panel	Lab
Swiggy Instamart	150,480	simulated 30-day panel	Lab
Flipkart Minutes	142,320	simulated 30-day panel	Lab
BigBasket	147,300	simulated 30-day panel	Lab
Total	610,789

In the data, the data_status column carries the same tag on every row, and src.load.real_only(df) keeps only the observed ones.

Product names, brands, categories, pack sizes, store names and coordinates are real throughout. On simulated rows, prices, stock, ETA, offers and badges come from SIMULATED_GENERATOR_v15.1, which is calibrated to the observed day in two places (stock-out level by category, price-ending style); docs/GENERATOR_v15_PARAMETERS.md marks every parameter as calibrated or assumed. On the website simulated numbers carry the tag Lab, observed ones Observed.

Simulated structure is planted, not discovered. When a model recovers a value the generator planted, that proves the modelling pipeline works, the way recovering Flipkart Minutes' planted −6% price gap proves the cross-platform join works. It is never a market finding, and nothing here reports one as such.

selling_price and stock_status on the 21,949 observed rows were verified against the source workbooks (last checked 2026-09-17; neither workbook ships in this repo). 27 observed prices turned out to be web-researched rather than seen in the app; they are listed in data/reference/REAL_PRICE_SOURCE_AUDIT.csv and src.load.real_observed_price_only() leaves them out.

Results

Each line says which data it rests on. Observed results describe Blinkit on 10 Jul 2026. Lab results show the pipeline recovering what the generator planted.

#	Deliverable	Data	Headline	Doc
1	Product matcher	Observed	real accuracy not measurable until humans label pairs	MATCHER_v1
1 v2	Product matcher v2	Observed + constructed edits	difference-aware features generalise to edit families never trained on (98.6% on variant swaps where similarity features reach 9.6%); abstains on 33.1% of unlabelled real pairs	MATCHER_v2
2	Next-price forecast	Lab	test MAE 9.66 vs naive 10.56 (+8.5%), beating both baselines	PRICE_FORECAST
3	Price direction	Lab	beats calibrated rules; "will drop" radar 78% precision at 5%	PRICE_DIRECTION
4	Stock-out classifier	Lab	two heads (depletion / restock) beat the Markov table; 6× lift at 1% alerts	STOCKOUT_CLASSIFIER
5	ETA prediction	Lab	beats 3 baselines; conformal intervals cover what they promise; confounding exposed	ETA_FORECAST
6	Anomaly detection	Observed + Lab	finds a real, known pack_qty defect at 70% precision@100	ANOMALY
7	Stock-out survival

The store label on the observed rows carries no detectable information. Products that share a name and pack fall in the same store in 21.0% of pairs; a label independent of the product gives 21.9%, and real per-store listing would give close to 0. Price, stock and ETA do not vary by store either, while the same test finds the store effect planted in the simulated rows. The source workbook records no way products were tied to stores, so the label's status is UNVERIFIED, and that is enforced in code (src/provenance.py): every store-level answer carries the caveat.

Matcher v2 learns from constructed edits, and knows where it was not taught. Similarity features memorise each kind of edit; features that say what differs carry over to kinds never seen in training. Where a pair differs in a way training did not cover, v2 abstains rather than guess. Real-world accuracy still waits on human labels.

A hand-logged panel can answer some questions and not others. Cut down to what a person could log, the simulated panel says the price gap needs more products, not more days; out-of-stock spells are biased in any panel shorter than three weeks; and two headline results need far more than a 128-product watchlist.

What the data cannot support. There are no customers, orders or quantities sold, so demand forecasting, price elasticity, reorder prediction and revenue are out of reach. Each has an honest substitute here: stock-out hazard instead of demand, competitive price response instead of elasticity, restock intervals instead of reorders.

How it works

flowchart TB
    raw[("data/raw<br/>610,789 rows<br/>frozen · sha256")] --> feat["Feature layer<br/>109 columns, past-only<br/>leakage check every run"]
    feat --> gate{"Data contracts<br/>21 invariants"}
    gate -- hold --> models["Models 1-10<br/>each against its baselines"]
    gate -. broken .-> stop["downstream steps<br/>BLOCKED, not stale"]
    models --> sim["Simulate & Optimize<br/>validated on the status quo"]
    sim --> analyst["AI analyst<br/>every claim sourced and tagged"]
    analyst --> site["Darklytics website<br/>one self-contained file"]
    site --> pages(["GitHub Pages<br/>live, offline-ready"])

Pipeline (scripts/run_all.py). Forty-seven steps, each a real run of its documented command. Every step names the steps it needs: when one fails, everything downstream is recorded as BLOCKED instead of fitting on stale inputs, while independent steps still run. --list, --dry-run, --step, --from, --resume and a per-step --timeout; a JSON run log and a Run Log page come out of every run.

Data contracts (src/pipeline/contracts.py). Twenty-one invariants the models rely on (composition by platform and label, provenance on every row, unrecorded fields never imputed, past-only feature bounds, a chronological split, structural NULLs exactly where a panel starts and ends, derived tables keyed one-to-one) are checked as one gate right after the data layer is built. Each contract has a test that plants a violation in a copy of the data and must catch it.

Claims ledger (scripts/check_claims.py). 37 headline claims in this README and the docs (84 numbers) are checked against the saved output each came from, so a number that drifts after a rerun fails the build.

Local server (scripts/serve_site.py). Standard library only. Gzip and ETag revalidation (a reload of the 2.7 MB page costs a 304), a content security policy and no-framing headers, a Host check against DNS rebinding, a six-digit code and per-device rate limits for phones on the same Wi-Fi, an LRU cache for the analyst's answers, /api/metrics (latency percentiles, cache hits) and a JSON 500 for any unexpected error. --public turns it into a read-only public service (any host, $PORT, per-device limits behind the host's proxy, CORS for the live site only, about 200 MB of memory): see the live Python analyst.

Tests. The full suite (count in the badge above, read from the last run) covers leakage, split and identifier guards, the planted-violation controls, server behaviour including the public mode, publishing and the fetch script's safety checks, a node run of every page and control, and the in-browser analyst re-scored on both question sets. CI runs on every push: one job rebuilds the feature layer from the frozen data and checks the contracts; another fetches the released run outputs and runs every test. A headless-browser check visits every page in light, dark, phone, tablet and wide layouts and fails on any script error, blocked resource or sideways scroll.

Run it locally

git clone https://github.com/Deepak17kb/quickcommerce-mi.git
cd quickcommerce-mi
python -m venv .venv && source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install -r requirements.txt

The quick way (minutes). Every full run is published as a release: the ML layer and every model output, as one zip. Fetch it and everything works at once:

python scripts/fetch_outputs.py          # data/ml + outputs/ from the latest release; refused if its sha256 differs
python scripts/serve_site.py             # Darklytics on http://localhost:8000, with the project's own Python analyst behind it
python -m pytest -q                      # every test

The full way (about three hours). Rebuild everything from the frozen raw data:

python scripts/run_all.py                # 47 steps in dependency order, then the tests and the website

The survival model alone takes 40 minutes; --dry-run shows the plan and --resume continues a run that stopped. Other entry points:

python -m src.pipeline.contracts                  # the 21 data contracts
python -m src.analyst "What happens if Blinkit cuts prices by 3 percent?"
python scripts/check_claims.py                    # every headline number against the output behind it
python scripts/deploy_pages.py                    # publish the built site (commit + push; the Pages workflow deploys it)
python scripts/release_outputs.py                 # publish a finished run's outputs as a release

Publishing

The live site updates itself. Every site build writes site_dist/ (the page, its service worker, icons, and the badge files above). It is tracked in git, and the Pages workflow deploys it whenever a push to main changes it, so a commit and a push are the whole deploy (python scripts/deploy_pages.py does both).

Run outputs are released, not committed. The ML layer and model outputs change with every run and would bloat the history, so each finished run is published as a GitHub Release (scripts/release_outputs.py, refused unless every step and test passed), and data/OUTPUTS_RELEASE.json pins the one this commit was built with. Products a person adds on their own machine (data/user_added/, photos from their camera) are never committed or released.

The live Python analyst

GitHub Pages serves files only, so the live site answers with the analyst engine built into the page: the same refusal rules, routing and sources as the Python analyst, scored on the project's own question sets (badge above). To put the project's own Python analyst behind it, deploy the read-only server in one click, then paste its URL into site/config.json and publish:

The blueprint (render.yaml) installs four packages, fetches the released outputs and starts serve_site.py --public. A free service sleeps when idle; the page keeps answering with its own engine while it wakes and switches to the live analyst once it does. Products you add on the live site stay in your browser: Export and Import on the Products page move them, photos included, to another browser or device.

import src.load as L

df = L.load_ml()                                  # 610,789 x 116, targets joined
train, val, test = L.splits(df, 'future_price')   # chronological, from the split column
feats = L.feature_columns(train)                  # by schema role; identifiers and targets refused
X = L.select_features(train, feats)
observed = L.real_only(df)                        # the only rows reportable as market fact

Three rules hold everywhere. Never split randomly: the split column is chronological (train 15 Sep-4 Oct, val 5-8 Oct, test 9-14 Oct). Never feed identifiers to a model: feature_columns() drops them. Never impute the structural NULLs: price_lag_1 is empty at a panel's first observation because nothing came before it.

Leakage, found and fixed. Five early features used groupby().transform() over the whole panel, so a row at time t knew prices from t+1. They were replaced by expanding *_to_date versions; the convention is that a feature at t may use data up to and including t, never later, and a data contract now enforces it.

src/load.py            data access: loading, splits, column roles, feature guard
src/pipeline/          raw build history, the ML feature layer, the data contracts
src/features/          derived past-only tables: stock clock, price grid
src/matching/          product matcher v1 and v2
src/models/            deliverables 2-10, one module each
src/simulate/          the simulator and its what-ifs
src/optimize/          price cut, restock priority and their extensions
src/analyst/           the provenance-labelled AI analyst
src/chain/             one product x store through every layer
src/audit/, planner/   what the data can support; how much more data would be enough
src/seasonal/          the festive planner
site/                  the website: shell, styles, pages, in-browser analyst, service worker
scripts/               run_all, build_site, serve_site, deploy_pages, release_outputs, fetch_outputs, checks
site_dist/             the published site (written by every build; a push deploys it)
data/raw/              the frozen observation layer (parquet, hash-checked)
data/reference/        catalog, stores, schedule, audits, ML schema
docs/                  how each layer was built and what it may claim
tests/                 the full test suite

License

Not affiliated. Darklytics is an independent student project. It is not affiliated with, endorsed by or connected to Blinkit, Swiggy Instamart, Flipkart Minutes or BigBasket; their names are used only to identify the apps compared.

The code is released under the MIT License. The data in data/ is shared for learning and research with the same permission. Product, brand and store names belong to their owners and appear only to identify what was listed. The observed Blinkit prices are one day's shelf prices (10 Jul 2026), and every simulated number is tagged Lab: neither is a statement about any company's sales, revenue or pricing strategy.

About

Built by Deepak, 3rd-year B.Tech CSE (Data Science with AI) at LPU, as a portfolio project. More work: github.com/Deepak17kb · portfolio

The full engineering notes, including every settled finding and trap, are in CLAUDE.md and docs/.