QA Automation Platform
One repository of test automation work, run by one CI pipeline: browser tests, API tests, contract tests, AI-assisted test generation, test data, SQL data-integrity checks, and the tooling that keeps a suite trustworthy. It is a working example of how a team catches broken changes automatically, and it has found real bugs in the code it tests (see Results).
Technical guide →
In plain English
Software teams change their product every day, and each change can break something that used to work. This project is a working example of how a team catches those breaks automatically, before customers do.
Each folder under packages/ is a small project that either tests something (a website, an API, a database, an AI feature) or helps testers do their job (finding unreliable tests, checking bug reports, generating test data). The packages says what each one does.
The live CI dashboard: pass rate, run time and flaky jobs, rebuilt after every run on main. Shown in the viewer's light or dark theme.
At a glance
- 30 packages, about 1,640 automated tests and checks, all run by one CI workflow
- One CI job per package, so one failure never hides another
- A live CI dashboard of pass rate, run time and flaky jobs, rebuilt after every run on main
- Published test reports for every suite, with tokens redacted before publishing
- A traceability matrix: each API requirement, the test cases that check it, and their latest results
- Flaky tests tracked across the last 14 runs, with one GitHub issue per flaky test, opened and closed automatically
- Pull requests run a quick smoke set; main and the nightly run do everything
- Load testing that found a real performance bug, now fixed
- AI-written tests reviewed before they count, and an AI grader checked against a person's answers
- Every AI test linked to the record of what the AI was asked and answered, in a self-hosted Langfuse
- Mutation testing that deliberately breaks code to prove the tests would notice
- Playwright and Cypress both: a banking UI tested by Cypress in Chrome and Firefox, where a double click or a lost reply still moves money once
- A release gate that blocks a deployment, with reasons, when a rule is broken
- A daily CI health check, and an automatic one-time re-run when CI fails on main
- Uptime monitoring: a real browser checks four public sites every 30 minutes, the history sits on the dashboard, and a site that fails twice in a row opens a GitHub issue that closes itself on recovery
- Strict TypeScript in ten packages, type-checked in CI with no build step
The packages
Strongest first within each area. Counts are automated checks. The full descriptions are in the technical guide.
UI
| Package | What it does | Checks |
|---|
| multi-user-flow-patterns | Runs several users at once to catch bugs that only appear when people share a system | 9 |
| visual-regression-demo | Compares screenshots of the banking UI with baselines in light, dark and phone width, in two browsers, and proves it by planting visual bugs | 46 |
| bdd-playwright-demo | Tests written as plain-language scenarios that a product owner can read | 9 scenarios + 4 |
| a11y-aoda-audit | Audits a demo store against the WCAG 2.0 AA standard Ontario's AODA requires, before and after fixes | 16 |
API
| Package | What it does | Checks |
|---|
| qa-practice-api | A small REST API built to be tested, with 119 cases designed from a written test strategy and a passive OWASP ZAP scan in CI | 292 + 17 properties + ZAP scan |
| banking-api | A small banking API where every cent is exact: accounts in four currencies and transfers on a double-entry ledger, with no floating-point maths. Retried requests move money once, requests at the same time can't overdraw an account, and 14 SQL checks prove every transfer balances. A small banking UI on top is tested by Cypress in Chrome and Firefox | 236 + 8 properties + 69 Cypress |
| api-drift-checker | Checks a live API against its written contract; its first run found two real bugs | 17 + 39-step live check |
| api-pytest | The same designed cases as the Playwright API suite, written again in Python with pytest and requests, under the same case IDs. A fixture starts the API, so one command runs it | 40 |
| webhook-harness | Proves a webhook receiver rejects forged or replayed messages and processes each one only once | 21 |
| graphql-test-suite | Tests a GraphQL API: queries, errors, and partial results | 33 |
|
Performance
| Package | What it does | Checks |
|---|
| k6-performance | Load and spike tests with per-endpoint limits; found that each login froze every other request for about 70 ms | 5 + 3 k6 tests |
| worker-scaling-benchmark | Measures whether running tests in parallel actually makes a suite faster | 12 |
AI quality
| Package | What it does | Checks |
|---|
| ai-test-generator | Uses Claude to write browser tests from a feature description, reviews them for weak tests, and measures how many are usable as they are | 127 |
| llm-judge-check | Checks an AI grader against a person's answers before trusting it, and traces every AI test result | 81 |
| qa-for-ai-features | Tests an AI feature's response format, response time, and safety refusals | 12 |
CI tooling and helpers
| Package | What it does | Checks |
|---|
| ci-dashboard | The live web page of this repo's CI health, plus the published reports, the traceability matrix and the uptime of four public sites | 59 |
| synthetic-monitor | A real browser checks four public sites every 30 minutes and keeps 30 days of history: down after two failed runs, slow at three times a site's own median, and a GitHub issue that opens and closes itself | 56 |
| flaky-test-analyzer | Points out the tests that only pass after a retry, and keeps one GitHub issue per flaky test | 37 |
| release-gate | Reads smoke, contract and performance results, applies a policy, and blocks a deployment with clear reasons | 15 |
| mutation-testing-demo | Deliberately breaks the code to prove the tests would notice (89.2% caught) | 89 |
| pipeline-templates-demo | One reusable CI/CD pipeline written for GitHub Actions, GitLab CI, and Azure DevOps | 28 |
| matrix-build-demo | Runs the same tests across several Node versions and operating systems | 99 |
| defect-report-linter |
Learning
| Package | What it does | Checks |
|---|
| money-course | A short course, Testing money in software, built on the banking API: six objectives, a quiz for each, and an ungraded practical with six planted bugs and a model answer. Being run as a pilot | 30 |
How a change gets tested
flowchart TD
A["Someone proposes a change<br/>(a pull request)"] --> B["Quick checks, about 1.5 minutes<br/>unit, API and performance tests,<br/>plus the browser smoke tests"]
B -->|"all green"| C["Reviewed and merged"]
B -->|"something failed"| A
C --> D["Full test run on the main version<br/>and again every night"]
D --> E["Live dashboard<br/>is the suite healthy?"]
D --> F["Published test reports<br/>what exactly ran and passed"]
D --> G["Flaky-test issues<br/>tests that pass only on a retry"]
D -.->|"when releasing"| H["Release gate<br/>go or no-go for a deployment"]
The test reports page: one report per suite, from the latest full run.
The traceability matrix: each API requirement, the test cases that check it, and their latest results.
Every AI test linked to the record of what the AI was asked and answered, from a self-hosted Langfuse.
Flaky-test issues are opened and closed automatically: see the flaky-test label.
Results
Each number links to where it comes from. Figures are as of 2026-10-20.
| Result | Number | Source |
|---|
| CI jobs passing on main | 35 of 35 | CI dashboard |
| CI runs fully green on main | 100% of the last 39 | CI dashboard |
| Mutation score, shopping cart module | 89.2% (198 of 222 changes caught) | mutation-testing-demo |
| Mutation score, AI-written discount code | 93.9% of 412 changes caught by designed cases and properties together | scoreboard |
| Flaky-test tracking | Last 14 runs on main; catches a test failing 1 run in 5 about 96% of the time, 1 in 100 only 13% | flaky-test-analyzer |
| Money, race, ledger-check and UI mistakes put back on purpose and caught, banking API | 78 of 78 | banking-api |
| Visual bugs planted in the banking UI and caught by the screenshot comparison (a button a few shades off, a 4 px shift, a missing button, a clipped amount, unreadable dark mode, a wrapped column) | 6 of 6 | visual-regression-demo |
| Planted bugs in the course practical caught by the model answer, each by its own check only | 6 of 6 | money-course |
What it has found so far: real bugs, not just green ticks. A login that froze every other user for 70 milliseconds, an API that sold more stock than it had and let delivered orders go back to pending, an API anyone could sign admin tokens for with a secret printed in this repository, a webhook processed twice, a tool that rejected every rule on Windows, and test reports that would have published live access tokens. Five have full root-cause write-ups.
Two results in numbers
- A performance bug found by load testing (k6-performance): browsing slowed to 67–84 ms (p95) whenever users logged in, because password hashing blocked Node's main thread. Moving the hashing to a thread pool brought p95 down to 1 ms and logins from 72 to 40 ms. The first threshold (250 ms) had passed the bug, so the limits are now set from measurements.
- AI-written tests, reviewed before they count (ai-test-generator): 10 of 10 generated specs passed the automated review and 42 of 43 tests passed on their first run. The review caught a test with no assertion that Playwright reported as passing. The 43 tests now run as a regression suite split across 3 CI runners, with a check that no test was lost or run twice.
Words used on this page
| Term | Meaning |
|---|
| CI (continuous integration) | A service (here, GitHub Actions) that runs every test automatically whenever the code changes |
| Pull request | A proposed change, checked and reviewed before it becomes part of the main version |
| Smoke test | A short check of the most important paths, run first because it's fast |
| Regression test | A test that makes sure something that used to work still works |
| Flaky test | A test that sometimes passes and sometimes fails without any code change; it erodes trust in the whole suite |
| Shard | One slice of a test suite; slices run on separate machines at the same time |
| API | The way programs talk to each other, without a screen |
Roadmap
See ROADMAP.md: one module a day for 30 days, each through its own pull request. The plan is complete and released as v1.0.0; RETROSPECTIVE.md covers what worked, what didn't, and what v1.0 doesn't prove. Phase 2 (Days 31–60) is planned in the same file.
No employer code or data is included. Everything here targets public demo apps or local mock services. See the technical guide for the notes on live services.
