Dead Link Checker
A CLI tool to detect broken links on any website. Given a starting URL, it crawls all internal pages recursively, collects external links, checks their HTTP status codes, and outputs a CSV report.
Features
- Recursive crawling of all internal links
- External links checked but not followed
- Visited URL tracking to prevent infinite loops
- Parallel HTTP requests for performance
- CSV output:
link, referrer, http_status_code
- Markdown summary of non-200 results per scan
- Email notification via the Resend API when non-200 results are found
Installation
Requires Python 3.10+.
pip install -r requirements.txt
Usage
python src/checker.py <start_url> [options]
| Option | Default | Description |
|---|
start_url | (required) | The URL to begin crawling from (must use http or https) |
--output, -o | scans/[WEBSITE]/[TIMESTAMP]/results.csv | Path to the output CSV file |
--workers, -w | 10 | Number of parallel threads |
--timeout, -t | 10 | Per-request timeout in seconds |
--user-agent | deadlinkprobe/1.0 | User-Agent header sent with every request |
--notify-email | (omitted) | Recipient address for a post-scan email notification |
When --output is omitted, results are written to scans/[WEBSITE]/[TIMESTAMP]/ and a README.md summary is produced alongside results.csv.
Example:
python src/checker.py https://example.com --output report.csv --workers 20
Email notifications
Pass --notify-email with a recipient address to receive a summary email after each scan. Three environment variables must be set:
| Variable | Description |
|---|
RESEND_API_KEY | API key from your Resend account |
RESEND_FROM_ADDRESS | Verified sender address in Resend |
export RESEND_API_KEY=re_xxx
export [email protected]
python src/checker.py https://example.com --notify-email [email protected]
See Resend website for setting up your account.
When using npm run scan-with-email, make sure to add the environment variables:
export RESEND_API_KEY=re_xxxxx && export [email protected] && npm run scan-with-email [email protected]
Output Format
Results are written to a CSV file with the following columns:
| Column | Description |
|---|
link | The URL that was checked |
referrer | The page that contained the link |
http_status_code | HTTP response code (e.g. 200, 404, 500) |
Testing
Unit tests
python -m pytest tests/ -v
Integration test
A sample website is available to test the tool against:
https://deadlinkprobe-sample-website.netlify.app
The expected output is (sorted by referrer, then link):
link,referrer,http_status_code
https://deadlinkprobe-sample-website.netlify.app/about/,https://deadlinkprobe-sample-website.netlify.app/,404
https://deadlinkprobe-sample-website.netlify.app/blog,https://deadlinkprobe-sample-website.netlify.app/,200
https://deadlinkprobe-sample-website.netlify.app/contact,https://deadlinkprobe-sample-website.netlify.app/,200
https://iamjeremie.me/,https://deadlinkprobe-sample-website.netlify.app/contact,200
https://iamjeremie.me/doesnt-exist,https://deadlinkprobe-sample-website.netlify.app/contact,404
Notes:
/about/ is an intentional dead link (404) on the sample site.https://iamjeremie.me/ and https://iamjeremie.me/doesnt-exist are external links, checked but not crawled.- Links to
/ discovered from /contact and /blog are skipped as already visited.
Development
This project is built through the manual jli- slash-command chain in .claude/commands/
(run under Claude Code). Each command is one step — set up a worktree, write the spec,
security notes and test cases, implement, review, test, ship. See
AGENT-COMMAND-MIGRATION.md for the full chain, diagram, and
mapping.
The older orchestrator-driven multi-agent pipeline has been removed, superseded by the
jli- chain.
Pipeline artifacts
Each feature's artifacts live in one folder per issue under docs/tasks/:
docs/tasks/issue-<id>-<slug>/
├── README.md # the feature request / issue body (your input)
├── business-specifications.md # spec output
├── security-guidelines.md # security output
├── test-cases.md # plain-language test scenarios
├── technical-specifications.md # implementation notes
├── review-results.md # code-review output
└── test-results.md # test-run output
Branch naming
| Change type | Branch prefix |
|---|
| New feature | feature/ |
| Bug fix | fix/ |
| Docs only | docs/ |
License
See LICENSE.
