A while back I wired a GPS module to a Raspberry Pi and turned it into a stratum-1 NTP server for the house. It worked beautifully: PPS locked, error bars in the hundreds of nanoseconds, satellites doing the timekeeping.

Then I opened my home network dashboard and the Time tab was red:

ssh: connect to host <time-server> port 22: No route to host

This is the story of what actually broke, which was not the thing I assumed — twice — and of the twenty-four days it sat broken without anyone finding out.

(Addresses below are illustrative. Assume a home LAN of 192.0.2.0/24 with a DHCP pool spanning 192.0.2.5 – 192.0.2.100.)

Wrong theory #1: the Pi is dead

“No route to host” plus no ARP reply looks like a box that’s powered off. It wasn’t. A quick sweep found the Pi alive and well at a different address — one inside the DHCP pool, holding a lease that had been renewed hours earlier.

So the Pi hadn’t died. It had moved, and DNS hadn’t.

Wrong theory #2: it’s dual-homed

Checking the router’s leases and reservations turned up something that looked like a smoking gun: there was a reservation for this host at an address above the pool, on its wired MAC, and a separate dynamic lease on a MAC one digit higher. On a Raspberry Pi the Wi-Fi MAC is typically the Ethernet MAC plus one:

aa:bb:cc:dd:ee:1a   eth0    reserved, above the pool
aa:bb:cc:dd:ee:1b   wlan0   dynamic lease, inside the pool

Two interfaces on one subnet, two valid answers to “where is this host” — classic ARP flux, and a tidy explanation for how the address drifted without anyone noticing.

It was also wrong. The Ethernet cable had been plugged in minutes earlier, while debugging. Before that, this box had been Wi-Fi only for its entire life, and the wired reservation had been sitting there dormant with nothing behind it. There was no dual-homing during the outage, because there was no second interface.

Two theories, two facts that killed them. Worth writing down, because the debugging value here was entirely in the timeline, not the topology.

The actual cause

The pool covers the low half of the subnet. Reservations, by convention on this network, live above it — and 92 of them do.

The time server was not one of them. It had an ordinary dynamic lease, inside the pool, on its Wi-Fi interface. And a while back I had added a static DNS host mapping pointing its hostname at that address.

That is the bug, and it’s worth stating plainly:

A permanent DNS record aimed at an address inside the DHCP pool, with no matching reservation, is a time bomb. The router is free to hand that address to something else. The name keeps resolving — it just resolves to the wrong host, or to nothing.

For weeks it looked fine, because DHCP leases renew and a host that stays up tends to keep its address. Nothing forced the issue.

Then something did.

The trigger

dmesg on the Pi, scrolled back far enough:

[1013882.469046] brcmf_fw_crashed: Firmware has halted or crashed
[1013882.511640] brcmf_cfg80211_get_tx_power: error (-5)
[1013883.092057] mmc1: card 0001 removed
[1013883.314334] mmc1: new ultra high speed DDR50 SDIO card at address 0001
[1013883.315903] brcmfmac: F1 signature read @0x18000000=0x15264345

The Broadcom Wi-Fi firmware halted. The SDIO card was removed and re-enumerated — note the PHY renumbering from phy0 to phy1 in the surrounding lines. The interface came back up, requested an address fresh, and got a different one from the pool. The old address went back in the pool, where it sits unallocated to this day.

Kernel ring buffer timestamps are seconds-since-boot, so converting them to a wall-clock date is worth doing:

python3 -c "
import datetime
uptime, event = 3107742, 1013882      # from /proc/uptime and the dmesg stamp
boot = datetime.datetime.now() - datetime.timedelta(seconds=uptime)
print('event:', boot + datetime.timedelta(seconds=event))"

That put the crash 24 days before I noticed. The DNS record had been pointing at an empty address that whole time.

There were plenty of warnings that this radio was unwell, too — a tight loop of brcmf_cfg80211_scan: Connecting: status (7) failures, and repeated brcmf_set_channel: set chanspec ... fail, reason -52 before the crash itself.

Why nothing told me

Here’s the part I find most instructive, and I got it wrong on the first pass.

My initial explanation was that SSH connection multiplexing had masked it — the dashboard keeps a ControlMaster connection to each host, so a live socket can outlive the correctness of what it points at. I’ve been bitten by that before, on this same network, with a stale known_hosts entry that hid behind a mux socket for six days.

But it doesn’t hold here. The mux is configured with ControlPersist=300 — the master exits five minutes after last use — and this endpoint is only polled when someone actually opens the dashboard. No socket survived 24 days. The failure today was an immediate No route to host, which is exactly what you’d expect from a fresh connection attempt, not a stale one.

The real answer is duller and more useful: the health endpoint had been reporting this failure correctly since the day of the crash, and nothing was watching it.

I had built the monitoring. /api/health returns a per-subsystem status, the Time entry had been ok: false for 24 days, and it was completely accurate the entire time. It’s just that the only way that information ever reached a human was if a human opened the page and looked at it.

A health endpoint nobody watches isn’t monitoring. It’s a status page that happens to be correct.

That’s the actual gap, and it’s a much better thing to have learned than “SSH multiplexing is tricky.”

The fix

The cable I’d plugged in during debugging turned out to be the right answer anyway: move the box to Ethernet, where a reservation already existed at an address safely above the pool. Update DNS to match. That’s it.

Then make the configuration explicit. The dashboard had been finding the time server through a default value compiled into the code — no environment variable set anywhere, so it silently fell back to a hardcoded address that had been correct when it was written. Making it an explicit setting is the whole lesson of this outage applied to configuration: an implicit default that used to be right is indistinguishable from one that still is.

One more trap: set doesn’t always replace

Applying the DNS change should have been a one-liner. On EdgeOS/Vyatta:

set system static-host-mapping host-name myhost.example.net inet 192.0.2.151

I ran it against a hostname that already had a record, expecting a replacement. Instead the host started resolving to both addresses:

$ nslookup myhost.example.net <router>
Name:   myhost.example.net
Address: 192.0.2.71
Name:   myhost.example.net
Address: 192.0.2.151

The inet node is multi-valued. set appends to it. And my own tooling hid the result, because the config parser read that node with a “give me the single value here” helper that returned the first entry and discarded the rest — so the API cheerfully reported one address while the router was serving two.

The fix is delete then set. The broader lesson is that on tree-structured config systems you have to know a node’s arity before you write to it, and “read it back to confirm” only works if your reader can represent what’s actually there. A parser that can’t express the bug can’t show you the bug.

Auditing for the rest of them

If this happened once, it has probably happened elsewhere. The check is mechanical: for every static DNS mapping, is the address inside the DHCP pool, and if so does a reservation exist for it?

inpool = lambda ip: pool_start <= ip_address(ip) <= pool_end

for record in dns_records:
    if inpool(record.ip) and record.ip not in reservations:
        print("time bomb:", record.hostname, record.ip)

Three more turned up:

  • A NAS. Unreserved dynamic lease, with a service hostname pointed at it and its address hardcoded into a stack of media containers. Same failure mode as the time server, aimed at something far more disruptive. Moving it is now a small project rather than a config edit, purely because the address got copied into a dozen places instead of a name.
  • A syslog VM. Statically configured on the VM itself, inside the pool, no reservation. Nothing stops the router leasing that address to a new device and colliding with the log collector. The pool is over half allocated, so that’s not hypothetical.
  • A dead record for a host that no longer exists.

The punchline

With the time server fixed and the dashboard green again, I ran one last check on the Pi:

chronyc clients

One client. And it was my own workstation, from a diagnostic query I’d run ten minutes earlier while testing.

Nothing else on the network has ever been syncing to it. I built a GPS-disciplined, PPS-locked, sub-microsecond stratum-1 clock, wrote a blog post about it, and never actually pointed anything at it. The 24-day outage didn’t degrade anyone’s time, because there was no one to degrade.

That’s its own kind of monitoring lesson. The correct next step isn’t just pointing hosts at it — it’s having the router hand out the NTP server via DHCP so clients get it without anyone remembering to configure them, and pointing them at the hostname rather than an address, so the next time something moves, DNS does its job.

What I’d tell past me

  1. Never point a static DNS record at an address inside the DHCP pool. If a name is permanent, the address behind it needs a reservation.
  2. Reference names, not addresses. Every hardcoded IP is a future outage with a longer fix. The NAS is a project instead of an edit for exactly this reason.
  3. A health endpoint nobody watches isn’t monitoring. Mine was right for 24 days and told no one.
  4. An implicit default that used to be correct looks exactly like one that still is. Make deployment config explicit.
  5. Convert dmesg timestamps to wall-clock time early. The entire diagnosis turned on one arithmetic step that dated the crash.
  6. Check a config node’s arity before writing to it — and make sure your tooling can represent the wrong answer, or it can’t show it to you.

The uncomfortable one is #3. Every other item is a mistake I made once and can fix. That one is a category of mistake I’ll keep making as long as the only consumer of my monitoring is me remembering to look at it.