CrashPilot Agent
The Linux crash-forensics agent behind CrashPilot. Open source, because you shouldn't have to trust a black box you run as root.
Try it in 60 seconds
No root, no account, no network once it is installed. Classify a kernel log you already have:
pip install "git+https://github.com/CrashPilotX/[email protected]#subdirectory=agent"
cat > dmesg.txt <<'EOF'
[ 8123.456789] NVRM: Xid (PCI:0000:01:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
[ 8123.456800] NVRM: GPU 0000:01:00.0: GPU has fallen off the bus.
EOF
crashpilot classify dmesg.txt
Category: gpu_fault
Severity: HIGH
Confidence: 70%
Evidence:
[ 8123.456789] NVRM: Xid (PCI:0000:01:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
[ 8123.456800] NVRM: GPU 0000:01:00.0: GPU has fallen off the bus.
The same check as a step in a GitHub Actions workflow (docs/github-action.md):
- uses: CrashPilotX/[email protected]
with:
file: artifacts/dmesg.txt
What it is: rule-based classification of crash logs (OOM kills, kernel panics, NVIDIA Xid and AMD GPU faults, thermal shutdowns, disk errors) and a Linux agent that does the same on a machine's last abnormal boot. What it isn't: a general log search or metrics store, and a log with no known crash signature comes back as unknown, not a guess. See What it detects for the exact list.
Why this repo is public
The agent needs privileged access to do its job: host PID namespace, /dev and /sys, the kernel journal, and SMART data. That is a lot to ask of a machine you care about, and "trust us" is not a good enough answer.
So the part that runs on your machine is open source and auditable. You can read exactly what it collects, what it redacts, and what it sends before you install it. The hosted dashboard and analysis backend remain a commercial product at crashpilotx.com.
What it does
A crash log tells you a process died. It doesn't tell you memory had been climbing for twenty minutes first, that a specific service pushed it over the edge, or that the same pattern happened three days ago on another box.
This agent watches every boot. When it sees an abnormal one, it reconstructs the sequence: the resource trend in the minutes before, the exact kernel and journal lines that explain it, a root cause with an honest confidence score, and what changed right before things went wrong.
What it detects
| Crash Type | Detection Method |
|---|
| Out-of-memory kill | journalctl OOM killer events |
| Kernel panic | dmesg + journalctl |
| GPU fault (NVIDIA Xid; AMD ring timeout, GPU reset, failed start) | nvidia-smi + kernel log |
| Thermal shutdown | ACPI THERMTRIP + lm-sensors |
| Machine Check Exception | mcelog / rasdaemon + dmesg |
| Sudden stop (power loss or hard lock-up) | A boot whose log ends with no shutdown sequence and no error; see below |
| Disk I/O error | dmesg + smartctl SMART data |
| Watchdog reset / soft lockup | kernel hung task detection |
| PCIe fault | Uncorrectable AER errors raise a live alert and go to the AI analysis; the offline heuristics do not classify them |
Sudden stops
A machine that loses power, or locks up hard, cannot log why. Its journal just ends. CrashPilot tells that apart from a normal shutdown or reboot by whether the end of the log shows the shutdown sequence systemd writes (the shutdown target, systemd-shutdown, and journald's last "Journal stopped"), and for a log with none it reads what surrounds the cut-off instead of stopping at "unknown":
- what started just before it: a stress test, a benchmark, a CI job, a scheduled command;
- how long the machine stayed down, which separates an automatic reset (a minute or two) from one that stayed dead until it was restarted by hand;
- whether earlier boots ended the same way. One stop can be anything; the same ending on several boots with nothing in the logs points at hardware or power, not software.
The report states what is known, not a cause: the logs cannot tell a failing power supply from a hardware lock-up. Its first steps depend on the evidence (cap the GPU's power if a GPU load had just started; check the supply's rating and the power connectors; read the BMC event log if there is one; compare how earlier boots ended). Confidence rises only with corroboration and never reaches certainty.
GPU load
Connected to a dashboard, the heartbeat reports each GPU's utilization, memory, temperature and power.
| GPU | Read through | Needs |
|---|
| NVIDIA | nvidia-smi | the NVIDIA driver's tool |
| AMD | the kernel's amdgpu driver (sysfs) | nothing extra; rocm-smi is not required |
An idle discrete AMD GPU is powered down by the driver after a few seconds. The agent never wakes a sleeping card to read it (the vendor tools do, and a wake-up every minute would change the machine's power behaviour). It reports such a card as suspended, 0% utilization, with its memory size, and shows temperature and power whenever the card is awake. Crash detection for GPU faults is separate; the table above lists what is classified.
This is the beginning of the README. Read the rest at the original. Read the full README ↗
