Christian Schultz

Platform engineer. Full-stack. Working with AI on both ends of the pipeline.

What this is for

A training system, on purpose. I train principles, technologies and everything around them, as deep and as hands-on as possible, on a true running system with real live apps: real identity, real incidents, a real public entrance, and no paying customers, yet.

• The latest stack on the weakest hardware I could put to use. The handicap is chosen: retired laptops and small desktops, to make it as hard as possible. If it runs here, it runs on any hardware; anything else is an upgrade.

• AI all over. On-prem and cloud, small local models and frontier models, simple setups and elaborate ones, at development time and at run time, each measured rather than trusted.

• Two purposes at once. Learn the maximum until the day a killer app appears, and be ready to grab that opportunity when it does. Until then everything here is learning and nothing is protected for uptime's sake; on that day the direction changes on purpose, and production becomes the main line.

• I run a Kubernetes platform at home: six bare-metal nodes built from retired laptops (the five I have audited date from 2011 to 2019), with identity, Postgres, storage, observability and a signed supply chain.

• A frontier model helps me build it. A small local model, on a Mac outside the cluster, helps me operate it, and a large cloud model grades the small one's posts once a day, with a person approving every action.

• The in-cluster software is open source or community edition. Claude, which I work with, is a paid service.

• It is a private platform with a public write-up. The write-up is the deliverable; the repositories behind it are not open.

• The rule of the whole thing: assert the property, do not trust the report.

In short, as of 2026-10-27. Running, with no organic users: the load is synthetic. Since 18 October: changes go through pull requests, git is the only way a change may reach the cluster (drift is checked daily), no plaintext Secret is left in git, the ops agent was hardened and put under measurement, and the estate was reviewed from the outside four times. Not finished: no admission control or network policies, secrets are not yet rotated or encrypted at rest in the cluster, the embedded side's nightly vulnerability upload failed every night from at least 24 September until it was fixed on 27 October, and fault injection has not fired since 13 October (its own safety gate keeps pausing it). Start with Recently, or Where to start.

Those three badges are live, and they are graded from outside the system they grade: a heartbeat for the scheduled jobs, and an off-site probe of the public entrance. They are depressed on purpose by reboots and fault injection, so they are not an availability promise. Everything else on this page is text, and I have stopped badging tool names: a badge that prints "Kubernetes" proves nothing.

Objectives and key results, Q4 2026

Set 2026-10-31, scored here in early January, misses included. Each key result is scored 0.0 to 1.0; around 0.7 is a good quarter, and 1.0 everywhere would mean the targets were too easy. Running OKRs honestly is itself one of the things being trained, so they point at work and dates that already exist rather than keeping numbers of their own.

1. Learn as much as possible, hands-on, on the running system.

• Three of the next tools learned to their own test: tracing (OpenTelemetry and Tempo), an AI evaluation framework, and a GitOps controller.
• The choice between a pull controller and push-on-merge made by 1 December, from what was measured.
• Each one written up here with its measured result.

2. Prove it on the weakest hardware: break it on purpose, watch it heal.

• The cause of the 5, 10 and 15 second request stalls named by measurement, and DNS lookup loss below 0.5 % on every machine (about 7 % on the two oldest today).
• Fault injection firing on its schedule again, for at least four consecutive weeks (its own safety gate has kept it paused since 13 October).
• One drill that powers a machine off, run end to end.

3. AI all over, measured, not trusted.

• The first person-graded sample of the operations agent's posts by 26 November, then one a month.
• The cloud reviewer's verdicts compared with the person's grades on the same posts, and the agreement published.
• Every kind of AI post recorded and checked (the morning digest too, since 30 October), with the flagged share published each month.

4. Be ready for the killer-app day.

• Two more of the seven launch gates closed or explicitly accepted: no default passwords anywhere, and real-time connections authenticated.
• Every secret that was ever in git in plaintext rotated.
• A rebuild from empty machines to running apps, from git, timed and published.

The numbers

Written by scripts/gather-stats.py in the site repository, from commands run against the cluster. What each figure counts, and the command behind it, is on the status page.

Figure	Value
Nodes (Ready)	6 (6)
CPU cores	48
Memory	99 GB
Pods running	186
Postgres clusters / instances	13 / 34
Clusters with a synchronous standby right now	8 of 13
Backup objects (completed in the last 24 hours)	348 (13)
Alert rules loaded	227
Scrape targets up	124 of 124
Commits to the platform repository	1943
Decision records	41
Test definitions (counted, not run)	1147
CI workflows	39

Measured 2026-10-27T06:10:37Z by make stats. What each figure counts, and the command behind it: the status page.

What this is, and is not

• Is: a training system, run to production standards on purpose. Real applications behind real identity, no paying customers yet, a CI/CD pipeline that deploys the apps, scheduled database backups, and a habit of checking every claim by something other than the thing making it.
• Is: preparation. When a killer app appears, the platform, the patterns and the habits should already be in place; on that day the direction changes, and production becomes the main line.
• Is not: a product with an availability commitment, or long-lived. The cluster was rebuilt on 2026-09-04, and k3s has been the only platform since the previous Docker Swarm cluster was decommissioned on 2026-06-09. "Months of uptime" would not be true.
• Is not highly available throughout. The platform's own failure-domain review, dated 2026-09-09 and not re-measured since (the whole estate was also down from 2026-10-14 to 2026-10-18), counted 7 of 13 as highly available. The ones that are not: the public entrance, the physical layer (one switch, one power feed), the image registry, the steward, and the single-instance metrics and log stores.
• Is not zero-loss everywhere. Block storage is replica-3. As of 2026-10-09, the table above shows 8 of 13 Postgres clusters with a synchronous standby. Those are the 8 three-instance clusters, configured to commit once any 1 of 2 standbys has the write. The 5 two-instance clusters are asynchronous by design.
• Is not a user base. Self-registration was switched off on 2026-09-29, per the platform's state notes (not re-checked today), so only I can create an account. The apps are deliberately modest demos that exercise every layer.

Corrections, 2026-10-27. Five statements on the site had turned stale in the pessimistic direction (pull requests, secrets in git, drift checking, the ops agent, a runner count), two on the embedded page were false throughout, and the uptime shortfall now has a cause. They are listed, with dates, on the estate page.

Retractions, 2026-10-09. Earlier versions of this profile said "no single points of failure", "RPO 0, proven" and "Docker Swarm compatible", and that the AI runs inside the cluster. None of those was true when I checked. The worst was the second: the PostgreSQL 18 cutover on 2026-09-16 silently dropped synchronous commit, so until it was restored on 2026-10-09 the application databases were asynchronous while the documentation said RPO 0. A guard now fails if a successor cluster loses that setting. Swarm was decommissioned and is not a fallback. The ops agent and its model run on a Mac outside the cluster, deliberately, because an operator that dies with the cluster is not an operator.

The parts, briefly

Platform. k3s on six nodes, Ceph replica-3 block storage, CloudNativePG, Keycloak, Traefik, Prometheus, Loki, Grafana and Alertmanager, largely managed by Ansible (a few steps, such as creating identity realms, are still by hand). Apps are Spring Boot and Flutter. It is standard Kubernetes, but I have not tested it on a managed distribution, so I do not claim it would move unchanged.

Scaling. CPU-based autoscalers are live. Scaling on request rate is not built.

Supply chain. First-party images are signed with cosign using a local key, so this is signing, not keyless Sigstore. As of 2026-10-27, 6 of the 7 first-party images running verify against the release key, and the tracker has accepted 71 SBOMs. Acceptance is not analysis. I rate it SLSA Build L2 myself; nobody has audited that. Secrets: every Secret in git has been encrypted with SOPS and age since 2026-10-25; the old values are not yet rotated and the cluster does not encrypt them at rest.

AI. Alerting is a loop with two models and a person: alerts go straight to Slack, an hourly post is composed by code, a small local model answers questions and writes a morning digest, and a cloud model grades the agent's posts once a day (drafts only, no tools; it is sent alert text, so the claim that nothing leaves the house is retired, see the loop). The ops agent answers in Slack, queries Prometheus, and proposes actions such as a node drain or a scale, which a human approves with a button. It does not act alone. Since 2026-10-26 its arguments are validated, only named approvers count, and what it says is recorded and graded (a first review found "critical" in 79 of 96 summaries, with no critical alert firing in 69 of them). An orchestrator that closes the whole delivery loop into one policy-gated workflow is not built; chaos injection is partial: my last self-rating (2026-09) counted 2 of 31 planned failure scenarios.

Embedded. A custom Yocto layer on the 6.0 LTS release (wrynose) for a Raspberry Pi 3 B+: a signed RAUC A/B image, updated over the air, with an SBOM and CVE scan. The nightly upload of that SBOM to the tracker failed every night from at least 2026-09-24 until it was fixed on 2026-10-27, and no device has reported back since 2026-09-01 (corrected on the site on 2026-10-27). A signed update and a deliberate rollback have both been proven on the board, and the same path now carries an image for a bench test rig. A Pi 3 cannot do hardware secure boot, so the updates are signed and integrity-checked, but the boot chain is not verified.

Read more

All on the site, https://schultzzznet.github.io/schultzzznet/:

• Recently: what changed, newest first, with dates.
• What I'd do differently: the mistakes, with their costs, and what four outside reviews found.
• Measurement traps: confident, wrong numbers, found by checking.
• The apps: three Flutter apps (location, hazard warnings, messaging) with screenshots, and what is not finished.
• Reliability and incidents.
• The AI ops agent and AI in development.
• The embedded image.
• Live status.

Code rather than prose: examples/ holds four scrubbed artifacts from the running system. Each exists because something looked correct and was not.

trivy-kernel-ab-split.sh	Host CVE counts that can never reach zero, and the regex that files a superseded kernel as live. Has a --self-test that demonstrates the bug.
duty-cycle-alert.yaml	Why a for: alert cannot see a signal that sawtooths across its threshold, and the two-line fix.
kured-args.yaml	A reboot daemon that logged "nothing to do" hourly for weeks while 16,002 patched-but-not-running findings piled up.
systemd-oneshot-timer.md	The RemainAfterExit trap that makes a timer fire once per boot.

Licence and security

• Code (scripts/, examples/, the Makefile and the workflow): MIT.
• Writing and images (docs/, this README): CC BY 4.0. Reuse it, with credit.
• Quotations and linked material stay with their authors. The other repositories are separate projects with their own terms.
• Found something here that should not be public, or a flaw in the scripts? See SECURITY.md.

The repositories

Only this one is public. The rest are private, so there is nothing to link.

Repository	What it is
schultzzznet	This site and its stats generator.
Platform	The cluster: Ansible, manifests, pipelines, decision records.
Yocto layer	The Raspberry Pi image and its OTA pipeline.
Mower retrofit	An RTK pattern-mowing brain for old robotic mowers.
Rover, video rover, i.MX8 Yocto, 3D printing	Smaller hardware projects.

The best engineers are not there just to code. They are there to solve problems. Marty Cagan, Empowered

[email protected] · schultzzz.net · Denmark