Jobhunter
A hosted web app + CLI tool that scrapes job listings, scores them against your CV, and tracks your applications. Free to use — sign in with Google to get started.
Architecture
[Vercel — Next.js] ←── REST + JWT ──→ [Render — FastAPI]
│
SQLAlchemy (sync)
│
[Neon — PostgreSQL]
↑
[AWS Lambda — scraper]
(EventBridge every 6h)
Jobs are scraped once into a shared pool. Each user has their own profile, match scores, and application tracking. The Lambda runs on a schedule (every 6h in prod), scrapes all sources concurrently, and writes directly to Neon.
What's working
- Google sign-in — OAuth via next-auth; instant access, no approval needed
- Job feed — personalised match scores based on your CV and preferences
- Job detail — full description, score breakdown, apply button
- AI tools — generate a tailored cover letter, rewritten CV, or LinkedIn recruiter message for any job (powered by Claude Haiku via the
jobhunter-ai private package)
- CV upload — markdown, PDF, or DOCX; skills auto-extracted via LLM and jobs re-scored
- Preferences — target titles, salary, remote preference, locations (city-level), countries to search (ISO2)
- Applications kanban — drag cards across Saved → Applied → Interview → Offer/Rejected
- Reject jobs — mark any job "not interested" (with an optional reason) to hide it from your feed; the matcher learns from it and lowers future scores for similar jobs (same company, close title match)
- Shared scraping — Lambda scrapes all sources every 6h concurrently; search terms and countries derived automatically from user preferences
- Stale job expiry — jobs not re-seen within 30 days are automatically marked inactive and hidden from the feed
- Location-filtered notifications — SNS alerts only fire for jobs in your preferred countries (fully remote jobs bypass the filter)
- CLI — full local CLI still works for power users and debugging
Scrapers
ATS platforms (multi-company)
| Scraper | Companies covered | Notes |
|---|
| Greenhouse | Anthropic, Cloudflare, Airbnb, Figma, Discord, Adyen, Ebury, Lottoland, Kambi, Rush Street Interactive, Genius Sports, Fanatics, Cabify, and ~20 more | boards-api.greenhouse.io |
| Lever | Palantir, Spotify, and more | jobs.lever.co |
| Ashby | OpenAI, Notion, Deel, ElevenLabs, Synthesia, Cursor, Perplexity, Mollie, Paddle, Plaid, and ~30 more | api.ashbyhq.com |
| Workday | Accenture, Airbus, GSK, Adobe, AstraZeneca, Maersk, BP, Unilever, Shell, Betway, Flutter Entertainment, Solera Holdings | Per-portal cap; large portals (Accenture, GSK…) scrape up to 500 jobs/run |
| SmartRecruiters | Talan, Bet365, Playtech, Evolution, Sportradar, EPAM Systems, Ciklum | api.smartrecruiters.com; detected via API probe on custom domains too (talan.com is a Nuxt/Drupal front-end over SmartRecruiters) |
| BambooHR | Various | api.bamboohr.com |
| Teamtailor | Oatly, Hedvig, Storytel, The Workshop | /jobs.json JSON Feed; Schema.org location data |
| Recruitee | Zoi, bunq, Keolis, Trusted Shops, CM.com, and 8 more | /api/offers/ JSON API; works on custom domains too |
| Workable | Rentokil Initial, 360dialog | apply.workable.com/api/v3 |
| 50skills | Carbfix | static-jobs-api.50skills.app; the front-end (jobs.50skills.com) is a bare client-rendered SPA with no server-side signal — API found by scanning the JS bundle |
| RippleHire | UST (usource) | {subdomain}.ripplehire.com/candidate; XML API found via headless-browser network capture — capped at 500 jobs/run, descriptions only fetched for new (unseen) jobs |
Job boards / aggregators
| Scraper | Source | Notes |
|---|
| Adzuna | 13+ countries | API key in SSM; countries derived from user preferences |
| The Muse | Curated tech companies | No auth required |
| Reed | UK job board | API key required |
| GibWork | Gibraltar job board (23+ companies) | gibwork.com/jobs.json; re-publishes listings hosted on SmartRecruiters, Ashby, Pinpoint, and others — country hardcoded "gi" since every listing is Gibraltar-based |
Direct company scrapers
| Scraper | Company | Notes |
|---|
| Thoughtworks | Thoughtworks | Direct careers page |
| BCG | Boston Consulting Group | |
| Coinbase | Coinbase | |
| Revolut | Revolut | |
| Coderland | Coderland | Manatal ATS proxied via Next.js; single request returns all listings |
Not active
| Scraper | Reason |
|---|
| LinkedIn | Guest endpoint blocks Lambda IPs; violates ToS |
| GitHub Jobs | Deprecated — always returns empty |
| Microsoft Careers | Returns empty |
Job matching algorithm
5-dimension scoring (100 pts total):
| Dimension | Points | Method |
|---|
| Skills | 35 | Fraction of job requirements covered by CV skills |
| Title | 25 | Word-level Jaccard + character similarity against target titles |
| Experience | 15 | Seniority match between job level and user preference |
| Location/remote | 15 | Remote preference OR location substring match (OR relationship) |
| Salary | 10 | Gradient — job salary vs user minimum |
Score maxima are served from the API (*_score_max fields on every job response) so the frontend never hardcodes the weights.
Project structure
jobhunter-agent/
├── src/
│ ├── cli.py
│ ├── models.py # SQLAlchemy models
│ ├── database.py
│ ├── job_scrapers/
│ │ ├── base_scraper.py # BaseScraper ABC
│ │ ├── registry.py # SCRAPER_MAP + DEFAULT_SOURCES
│ │ └── *.py # One file per scraper
│ ├── job_matcher.py
│ ├── job_searcher.py
│ ├── application_tracker.py
│ ├── lambda_handler.py # Lambda entry point (scrape + match phases)
│ └── user_profile.py
├── web/
│ ├── api/ # FastAPI (Render)
│ │ ├── main.py
│ │ ├── routers/
│ │ └── tests/
│ └── frontend/ # Next.js (Vercel)
│ ├── app/
│ ├── components/
│ └── lib/
├── tests/ # CLI / scraper tests
├── .github/workflows/ci.yml # CI: lint → test → deploy Lambda + Render + Vercel
├── template.yaml # SAM template (Lambda + EventBridge + SNS)
├── samconfig.toml
├── Dockerfile.lambda
├── requirements.txt
├── requirements-lambda.txt
└── pyproject.toml
CLI quick start
# Setup
python3.11 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
# Add your CV (markdown, PDF, or DOCX)
job-agent profile upload path/to/cv.md
# Scrape and match
job-agent scrape
job-agent match
# Search
job-agent jobs search --keywords "python" --min-score 30
job-agent jobs view 42
# Track applications
job-agent applications apply 42 --notes "Applied via website"
Adding a new scraper
Known ATS (Greenhouse, Lever, Ashby, Workday, SmartRecruiters, Teamtailor, Recruitee, Workable, DeJobs, 50skills)
Pass the company's careers URL — the ATS is auto-detected:
job-agent scraper add https://boards.greenhouse.io/stripe
job-agent scraper add https://accenture.wd103.myworkdayjobs.com/AccentureCareers
job-agent scraper add https://www.sixt.jobs/us # custom domain, SR detected via API probe
Or supply the ATS and config directly:
job-agent scraper add --ats smartrecruiters --config '{"company_id":"Sixt","display_name":"Sixt"}'
The config row is written to the scraper_config table and picked up on the next Lambda run — no code change or redeploy needed.
Novel ATS (platform not yet supported)
job-agent scraper add https://careers.somecompany.com
# or
job-agent scraper generate https://careers.somecompany.com
Both commands fall through to the AI generator when detection fails. The generator fetches the page, scans JS bundles for API patterns, and probes candidate endpoints; if that finds nothing, it renders the page in headless Chromium and captures the real network calls it makes (catches client-rendered React/Vue/etc. pages that never expose an API in static HTML), then falls back to scanning for an embedded job listing if there's still no API. It then calls Claude Haiku to write a draft BaseScraper subclass in src/job_scrapers/.
After writing the draft, both commands automatically run it live against the real site (fetch + parse a sample — nothing is written to the database) and print a pass/fail report: schema problems (wrong/missing fields, wrong types) and cross-record problems (e.g. several different jobs all sharing one link, a sign the per-job extraction is broken and fell back to a generic page). A clean report is a good sign, not a guarantee — review the file before registering it. After reviewing the output:
- Add the class to
src/job_scrapers/registry.py (SCRAPER_MAP + DEFAULT_SOURCES)
- Run
pytest
- Run
job-agent scrape --sources <source_name> to re-verify after any manual fixes
LOW CONFIDENCE drafts: if no live API endpoints were confirmed during generation, the file header and CLI output will say so. The API URL is Claude's best guess — verify it manually before registering.
Broken TLS cert chain: add --insecure to scraper add/scraper generate to skip certificate verification for a site with a known server-side cert misconfiguration. Only use it when you've confirmed the site itself is at fault (e.g. via openssl s_client) — it disables verification for every request the generator makes against that host.
Testing
pytest -v # All tests (CLI + web API)
pytest tests/ -v # CLI/scraper tests only
pytest web/api/tests/ -v # FastAPI router tests only
Web API tests use TestClient + in-memory SQLite — no Neon connection needed.
Deployment
CI/CD (automatic on push to master)
.github/workflows/ci.yml runs on every push:
- Python lint + tests (black, isort, flake8, mypy, bandit, pytest)
- Frontend Vitest tests
- Deploy Lambda —
sam build (QEMU arm64 on amd64 runner) + sam deploy --config-env prod
- Deploy API — Render deploy hook
- Deploy frontend — Vercel CLI
Required GitHub secrets: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, RENDER_DEPLOY_HOOK_URL, VERCEL_TOKEN, VERCEL_ORG_ID, VERCEL_PROJECT_ID.
Lambda (manual)
export DOCKER_HOST=unix:///Users/pete/.docker/run/docker.sock
sam build
sam deploy --config-env prod --no-confirm-changeset
Lambda runs in two phases, each with a full 600s budget:
- Scrape (
{}) — scrapes all sources concurrently via ThreadPoolExecutor, expires stale listings (>30 days), then invokes itself async with {"action":"match"}.
- Match (
{"action":"match"}) — computes per-user scores, sends SNS notifications for matches ≥ 70% in the user's preferred countries.
# Trigger scrape + match cycle
aws lambda invoke --function-name jobhunter-prod --region eu-west-1 \
--invocation-type Event --payload "$(echo '{}' | base64)" /dev/null
# Trigger match only
aws lambda invoke --function-name jobhunter-prod --region eu-west-1 \
--invocation-type Event --payload "$(echo '{"action":"match"}' | base64)" /dev/null
# Watch logs
aws logs tail /aws/lambda/jobhunter-prod --region eu-west-1 --follow
Web app
| Service | Config |
|---|
| Render | Root: web/api; build installs jobhunter-ai private package via GITHUB_TOKEN |
| Vercel | Root: web/frontend; pinned to lhr1 (London) in vercel.json |
| Neon | Connection string in Render env vars + SSM /jobhunter/database-url |
DB migrations
Schema changes are applied directly via Neon MCP or psql. init_db() handles new tables via create_all(), but new columns on existing tables require an explicit ALTER TABLE … ADD COLUMN IF NOT EXISTS.
User management
Sign-up is open — all new users get immediate access. The is_approved column is retained in the DB but not enforced.
Match score email notifications fire when a new job scores ≥ 70% against a user's profile (configurable via MinMatchScoreNotify in samconfig.toml — no Lambda rebuild needed).
Documentation
- CLAUDE.md — development conventions, architecture decisions, gotchas
- WORKFLOW.md — development workflow and deploy checklist
- DEPLOYMENT.md — local deployment options (systemd, Docker)
- PROJECT_PLAN.md — current state, roadmap, and future features
License
MIT
