AgentTrace

AgentTrace is an AI helper module for web scraping, shipped as one Python file. An AI agent (or you) gives it a URL or a plain-language goal; it drives a real browser, captures and verifies the network traffic of every page / click / SPA route, tells you how the site should be scraped (JSON API? hydration blob? HTML list?), extracts the data, and exports reusable API clients — politely, reproducibly, with evidence.

╔══════════════════════════════════════════════════════════════════════════════════════╗
║  pip install playwright  &&  python -m playwright install chromium                   ║
║  python agenttrace.py doctor            # is everything ready?                       ║
║  python agenttrace.py recon  <URL>      # how should this site be scraped?           ║
║  python agenttrace.py extract <URL> --paginate --output data.csv                     ║
╚══════════════════════════════════════════════════════════════════════════════════════╝

---

░▒▓ Contents

#	Section	For
0	🧭 Pilot — a whole task from plain words	start here: one command, verified result, report + script, safety model
1	🤖 AI agents: read this first	the decision flow + golden rules
2	📦 Install	Windows / macOS / Linux
3	🗺️ Task → tool map	"I want to … → use …"
4	🍳 Recipes	copy-paste solutions for common scraping jobs
5	🐍 Python API	Session and helpers
6	⌨️ CLI	every command, flags, exit codes
7	🧩 Workflow configs	no-code site configs, checkpoints
8	🔌 MCP server	use it as tools from Claude
9	📁 What a run writes	HARs, reports, evidence
10	🩺 Troubleshooting	error → fix
11	🧪 Tested	how it was tested + results
12	🧱 Inside the file	map of agenttrace.py ↔ knowledge.md
13	🇧🇩 বাংলায় সংক্ষেপে	Bengali summary

---

🧭 0. Pilot — do a whole task from plain words

You say what you want. The module works out how, does it, checks every step, and hands you the answer, a report and a reusable script. English, বাংলা or a mix. No selectors, no click sequences, no wait times.

python agenttrace.py pilot "find the price of EX5" --url https://www.phanteks.store/
python agenttrace.py pilot "daraz এ গিয়ে wireless mouse খুঁজো, কার্টে যোগ করো আর চেকআউট পেজ পর্যন্ত যাও" --url https://www.daraz.com.bd/
python agenttrace.py pilot "add it to the cart and go to the payment page" --url https://shop.example --data checkout.json
python agenttrace.py plan  "…" --url …      # how the task is understood + the stages - no browser opens
python agenttrace.py dashboard              # the same, in a local web page (127.0.0.1)
python agenttrace.py mcp --tools lite       # let an AI assistant call it as tools - the cheap way (see 0.10)

Using it from an AI model? Let it call pilot_start(goal, url, wait_s=120) once instead of browsing step by step: it gets a ~0.5 KB answer, not a conversation. Details and measured numbers in 0.10. Give the AI the guide it needs with python agenttrace.py skill --install (0.11).

0.1 What happens

 sentence ─▶ Task planner ─▶ Website observer ─▶ Action planner ─▶ Execution ─▶ Validation ─▶ Recovery ─▶ Report + script
 (EN/বাংলা)  Intent+stages   understand(): page    picks controls    Session     expected vs    overlays,     report.md/json,
             + success       type, forms, price,    by their words    + HAR       actual per     scroll,       screenshots, HAR,
             criteria        options, controls      and says why                  stage          re-observe    automation_*.py

This is the beginning of the README. Read the rest at the original. Read the full README ↗