Shard Ecosystem

What it is

A 2-core, 3.8 GB VPS running everything I build for myself: a kanban board with its own REST API, a library that pushes books to a Kindle over WiFi, an RSVP reader, a mini-ERP, monitoring, an agent gateway, a catalog of my own Android apps. Twenty-five subdomains under shard.icu, sixteen active services, seven scheduled jobs.

This is not a weekend homelab. It is where I practice what my day job does not let me practice: when something breaks here, I am the one who fixes it, and there is nobody to escalate to.

The constraint that shapes everything

I do not have root on this machine. The camilo user is not in sudoers. The only ports open to the internet are 80 and 443 (Caddy), 22022 (SSH) and 22000 (Syncthing).

That forces a specific architecture — and it turns out to be a good one:

app (bind 127.0.0.1:PORT)
  → systemd user unit (~/.config/systemd/user/<name>.service)
  → Caddy vhost, only if it goes public
  → DNS A record at Porkbun

Every service listens on loopback only. Nothing gets exposed by accident, because nothing can be: the only thing the internet sees is Caddy. The units are user units with linger enabled, so they survive logout and start on boot — without touching a single file under /etc.

That design has one trap I learned the hard way: in Node, app.listen(PORT, cb) binds to 0.0.0.0, not to loopback. You have to write app.listen(PORT, '127.0.0.1', cb) explicitly. Verification is mandatory before exposing anything:

ss -tlnp | grep <PORT>   # must say 127.0.0.1, never 0.0.0.0

The day 26 subdomains went down

10 August 2026. The whole ecosystem stopped responding at once, except two sites.

The cause: caddy.service starts with --config /etc/caddy/Caddyfile, and that file — which needs root to edit — contained only two hosts. The other twenty-three I had been adding through Caddy’s admin API on 127.0.0.1:2019, which needs no root. It worked perfectly… and it lived in memory only.

A single systemctl reload caddy was enough to erase all twenty-three.

Diagnosis was quick; the fix was the interesting part, because the obvious solution — move everything into the Caddyfile — requires exactly the permission I do not have. So I solved it differently:

Piece	Role
caddy-config-canonical.json	Source of truth: a snapshot of the full config
caddy-restore-routes.sh	Compares the live config against a sentinel host; if missing, POST /load of the canonical
caddy-routes-restore.timer	OnBootSec=30s + every 2 minutes
caddy-snapshot-canonical.sh	Refreshes the canonical from the live config

The script is idempotent by design: if the sentinel host shard.icu answers, it exits 0 without touching anything. It acts only when it detects the config has fallen back to the bare Caddyfile. That is what makes it safe to run every two minutes.

It also waits up to 60 seconds for the admin API to come up, because on boot the timer can beat Caddy to it:

for _ in $(seq 1 60); do
  curl -sf -m 3 "$ADMIN/config/" >/dev/null 2>&1 && break
  sleep 1
done

Maximum outage window today: two minutes. Without root, and without depending on me being awake.

What it taught me

State that lives only in memory does not exist. The config worked, it answered, it served real traffic — and it vanished on a routine command. Any hot-applied configuration needs a source of truth on disk and a mechanism that reapplies it, or it is a time bomb.

A sentinel beats a full comparison. I could have diffed all 25 routes against the canonical. Checking a single host that exists only in the good config is cheaper, faster, and produces no false positives when I add a new route.

The constraint improved the design. With root I would have edited the Caddyfile, reloaded, and had no self-healing at all. Not having it forced me to build something that repairs itself.

The uncomfortable corollary

There is one step that is easy to forget and that breaks everything silently: after adding a route through the admin API, you must run the canonical snapshot. Otherwise the new route works… until the next reboot, when the timer faithfully restores the previous version and deletes it.

That is exactly the kind of detail you only learn by operating what you built.