WP1 — the Wikipedia 1.0 engine

WP1 is the software behind wp1.openzim.org, the successor to the original bot of the Wikipedia 1.0 project — which, as User:WP 1.0 bot, has more all-time edits than any other account on English Wikipedia. It:

• Aggregates the quality and importance assessments of every rated English Wikipedia article, across more than 2,000 WikiProjects, into browsable quality × importance tables, with nightly updates posted back on-wiki as project tables and change logs. Log headings appear only on the log page itself, not in the table of contents of pages that transclude it.
• Lets users build selections — custom article lists defined by WikiProject, Petscan, SPARQL, combinations thereof, or plain lists — for slicing Wikipedia content.
• Turns those selections into ZIM files, via the Zimfarm, for reading offline with Kiwix.

End-user documentation lives at wp1.readthedocs.io; the API is described by openapi.yml and browsable at api.wp1.openzim.org.

Quick start (development)

Everything runs in Docker: the only hard requirement is Docker with the compose plugin. From a fresh checkout:

./scripts/util/dev-up.sh

This builds and starts the full development stack — frontend with hot reload at http://localhost:5173, API server at http://localhost:5000, plus the dev database (MariaDB), Redis, materializer workers, and MinIO (s3-compatible storage) — prints the URL of every running service, then follows the logs (Ctrl-C stops the stack). Pass -d to return after printing the URLs, and enable the local Zimfarm with COMPOSE_PROFILES=zimfarm. It wraps docker compose -f docker-compose-dev.yml up --build, which also works. No configuration is needed: the defaults in wp1/config.py already point at these services.

On the first run (and after new migrations land), migrate the dev database as described in docker/dev-db/README.md, which also covers seeding it with test Selection data. Migrations, like the rest of the host-side Python toolchain, run through Pipenv:

pip3 install pipenv       # into your global Python (3.12), not a virtualenv
pipenv install --dev

That's it. Edits to wp1-frontend/src/ hot-reload in the browser, and the source tree is volume-mounted into the API container, where Flask's debug mode auto-reloads on backend edits. Development is targeted at Linux; other platforms may not be fully supported.

Repository tour

Path	What it is
wp1/	The Python backend: wp1/logic (business logic), wp1/web (Flask API), RQ jobs, and the update engine. Runs only inside the Docker images.
wp1-frontend/	The Vue 3 + Vite + Tailwind frontend.
docker/	One subdirectory per Docker image (production and dev), including the local Zimfarm and the dev database.
db/	YoYo database migrations for the enwp10 database.
scripts/util/	Development helpers: test runner, type checker, dev-data seeder, parallel worktree stacks.
scripts/wp1/	Production deploy, rollback, and operational scripts.
docs/	The mkdocs sources for wp1.readthedocs.io.

Development

Configuration (.env)

All backend configuration lives in a single schema in wp1/config.py, read from environment variables. The committed .env.example is generated from that schema (run pipenv run python -m wp1.config after changing it; CI fails on drift) and documents every knob, its type, its default, and whether it is required in production.

For development you usually need no configuration at all: the schema defaults point at the services in docker-compose-dev.yml. To customize values, copy .env.example to .env (gitignored) and edit it; the dev containers pick it up via docker compose's env_file.

The main thing worth customizing is the WIKIDB section: the app reads the enwiki_p replica database (referred to in the code as wikidb) on Toolforge, and needs your Toolforge credentials to do so. If you are a part of the toolforge enwp10 project, you can find the credentials on toolforge in the replica.my.cnf file in the tool's home directory. This is not required for developing the frontend.

The production instance additionally requires English Wikipedia API credentials (API_USER/API_PASSWORD) for editing on-wiki tables; in development (WP1_ENV=development, the default) the jobs that edit Wikipedia are disabled.

See API security for CORS configuration, CSRF protection, authenticated requests, and ZIM notification privacy.

Wikipedia replica access (SOCKS5)

In development, wp1.db.connect("WIKIDB") always uses a SOCKS5 proxy at localhost:1080, with DNS resolution performed through the proxy. WP1 does not start that proxy. This applies only to the Wikipedia replica: WP10DB connections and production database connections are direct.

For Python running on the host:

1. Configure SSH access to login.toolforge.org using your Toolforge shell username and SSH key (for example, in ~/.ssh/config). Check access with ssh login.toolforge.org true.

2. Set WIKIDB_USER and WIKIDB_PASSWORD in your gitignored .env from the [client] section of Toolforge's ~/replica.my.cnf (or the tool account's file). These are database credentials, not your SSH or Wikipedia login. Keep WIKIDB_HOST=enwiki.analytics.db.svc.eqiad.wmflabs, WIKIDB_DB=enwiki_p, and leave WIKIDB_PORT unset (defaults to 3306).

3. Keep this command running in a separate terminal:

   ssh -N -T -D 127.0.0.1:1080 \
     -o ExitOnForwardFailure=yes \
     -o ServerAliveInterval=30 -o ServerAliveCountMax=3 \
     login.toolforge.org
   

   The loopback binding avoids exposing an unauthenticated SOCKS proxy to the network. Stop it with Ctrl-C when finished.

4. From the checkout with your .env and Python dependencies installed, verify an authenticated database query, not just an open proxy port:

   pipenv run python - <<'PY'
   import socket
   from wp1.db import connect
   
   socket.setdefaulttimeout(15)
   conn = connect("WIKIDB", connect_timeout=15, read_timeout=15)
   try:
       with conn.cursor() as cursor:
           cursor.execute("SELECT 1 AS connected, DATABASE() AS db")
           print(cursor.fetchall())
   finally:
       conn.close()
   PY
   

   Expected result: [{'connected': 1, 'db': b'enwiki_p'}].

Docker limitation: the default dev web and worker containers have separate network namespaces. Their localhost:1080 is not the host's proxy, and the Compose file does not provide a tunnel. Replica access therefore requires a SOCKS listener in each consuming container's network namespace; the host procedure above alone does not enable replica-backed Docker jobs. The proxy address is currently hardcoded, with no environment-variable override. Frontend-only development does not require the replica.

Troubleshooting:

• Connection refused at localhost:1080: the tunnel is stopped, or Python and SSH are in different network namespaces.
• SSH permission denied: check your Toolforge shell username/key, not the database password.
• MySQL error 1045 / access denied: the tunnel reached MySQL; check the replica credentials. someuser / somepass are placeholders.
• SOCKS destination/DNS errors: check the replica hostname and its reachability from Toolforge. Local DNS need not resolve it.
• WIKIDB_PORT is the destination MySQL port, not the SOCKS port. ssh -L with WIKIDB_HOST=localhost is not an alternative to SOCKS in development: the application still uses SOCKS, and that destination would be interpreted from the SSH server. Do not switch to production mode to bypass this; it also enables jobs that edit Wikipedia.

Backend tests

The Python tests need the test databases from docker-compose-test.yml; the wrapper script starts them automatically (and cleans up after interrupted runs):

./scripts/util/run_tests.sh

(or plain pipenv run pytest if the test containers are already up). No .env is needed: the pytest bootstrap (conftest.py) constructs the test configuration in code.

Frontend tests

The Cypress suite is hermetic — every API call is stubbed, so no backend or Docker services are needed, just the frontend served on port 5173:

cd wp1-frontend
pnpm dev                  # or: pnpm build --mode staging && python3 -m http.server 5173 --directory dist/
pnpm exec cypress run     # in another terminal; `cypress open` for the GUI

See wp1-frontend/README.md for more on the frontend, including running it outside Docker.

Going further

• ZIM file creation — run a complete local Zimfarm with the zimfarm and zimfarm-worker compose profiles: see docker/zimfarm/README.md.
• Parallel dev stacks — every port and container name is parameterized, so multiple checkouts/worktrees can run side by side: ./scripts/util/create_worktree.sh <branch>, documented in scripts/util/README.md.
• Manual update endpoints — in development some project endpoints are overlaid with fakes for easier frontend work: see wp1/web/dev/README.md.
• Editing the docs — the Read the Docs site rebuilds from docs/ on every push to main, and CI runs mkdocs build --strict on every PR. To preview locally: install docs/requirements.txt into a virtualenv, then run mkdocs serve from the repository root.

Production

Deploys are done with ./scripts/wp1/deploy.sh, which pushes main to the release branch (triggering the image builds on CI) and then updates the production box. Rollbacks, one-off operational scripts, SSH access to the box, and the Redis persistence rules are all documented in scripts/wp1/README.md.

Contributing

See CONTRIBUTING.md for code standards, testing requirements, and PR guidelines. Before your first commit, install the pre-commit hooks that keep the code formatted:

pipenv run pre-commit install

(Details in the comments of .pre-commit-config.yaml.)

License

GPLv2 or later, see LICENSE for more details.