I wanted the data on my own hardware, recorded by weewx, and I didn’t want an inexpensive IoT console on my LAN or talking to the internet at all.

The end state: a Raspberry Pi 3 (pi3-1) runs its own WPA2 access point just for the station. On that network the Pi’s DNS server says it is wunderground.com. The station uploads to the Pi, thinking it’s Weather Underground, and weewx-interceptor parses the upload and hands it to weewx. The station can’t reach the LAN or the internet, and the weewx reports are served to the LAN by nginx.

How it fits together

Vevor console --wifi (WPA2, SSID weewx-ap)--> Pi wlan0 192.168.5.1
  DHCP + DNS: dnsmasq
    wunderground.com -> 192.168.5.1
    every other name -> NXDOMAIN, nothing forwarded upstream
  HTTP to port 80 --nftables redirect--> :8090 weewx-interceptor (wu-client)
    --> weewxd --> SQLite archive
  weewx reports --> /var/www/weewx --> nginx on port 80 (eth0, LAN)
  IP forwarding off + nftables drops all forwarding to/from wlan0

The station sends a Weather Underground protocol request, a plain HTTP GET to /weatherstation/updateweatherstation.php?ID=...&tempf=...&humidity=..., and the interceptor’s wu-client mode already understands that format. Getting the station to send that request to the Pi only takes a DNS lie and a port redirect.

Hardware and host

• Raspberry Pi 3B, Raspberry Pi OS based on Debian 13 (trixie), wired to the LAN on eth0
• The Pi 3’s built-in BCM43430 wifi (brcmfmac driver, 2.4 GHz only) as the access point on wlan0
• Vevor weather station console with a BL602 wifi chip

The Pi already runs another service that uses port 8080, which is why the interceptor listens on 8090 here.

1. Base system

apt-get update && apt-get install -y git hostapd dnsmasq nginx
timedatectl set-timezone America/Los_Angeles
raspi-config nonint do_wifi_country US
rfkill unblock wifi

2. weewx and the interceptor

I installed weewx 5.5 with the git method, in a Python venv. Do this as your normal user, not root. weectl station create writes the current user and the venv’s Python path into the systemd unit it generates.

python3 -m venv ~/weewx-venv
source ~/weewx-venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install CT3 configobj Pillow ephem
git clone https://github.com/weewx/weewx ~/weewx

python3 ~/weewx/src/weectl.py station create --no-prompt \
  --driver=weewx.drivers.simulator --location="Rocklin, CA" \
  --latitude=<lat> --longitude=<lon> --altitude=<alt>,foot --units=us

python3 ~/weewx/src/weectl.py extension install --yes \
  https://github.com/matthewwall/weewx-interceptor/archive/master.zip

The simulator driver is only there so station create has something to configure. It gets replaced in the next step. Then, as root, install the service:

sh ~<user>/weewx-data/scripts/setup-daemon.sh

3. weewx.conf

The relevant changes in ~/weewx-data/weewx.conf:

[Station]
    station_type = Interceptor

[Interceptor]
    driver = user.interceptor
    device_type = wu-client
    port = 8090

[StdReport]
    HTML_ROOT = /var/www/weewx

[StdArchive]
    archive_interval = 300
    record_generation = software

Two notes:

• record_generation must be software. The interceptor has no hardware archive memory to read records from. With hardware, weewx starts up and logs loop packets, but it never saves an archive record.
• HTML_ROOT points outside the home directory, at a directory the weewx user owns. Home directories on this Pi are mode 700, so nginx can’t read the default ~/weewx-data/public_html. The smartphone and mobile sub-reports need their HTML_ROOT moved under /var/www/weewx too.

4. Patching the interceptor

Version 0.60 of the interceptor needed four small fixes for this station, all in ~/weewx-data/bin/user/interceptor.py. Keep a copy of the original, because reinstalling the extension overwrites your patches.

@@ do_POST
-            data = str(self.rfile.read(length))
+            data = _bytes_to_str(self.rfile.read(length))

@@ WUClient LABEL_MAP
             'solarradiation': 'solar_radiation',
+            'solarRadiation': 'solar_radiation',

@@ WUClient IGNORED_LABELS
+            'rainin',

@@ WUClient.Parser.parse
-                pkt['dateTime'] = self.decode_datetime(
-                    data.pop('dateutc', int(time.time() + 0.5)))
+                # Console clock is 1 h fast (no DST) and has no NTP: use the Pi clock
+                data.pop('dateutc', None)
+                pkt['dateTime'] = int(time.time() + 0.5)

What each one fixes:

1. Python 3 bytes bug. str() on a bytes object gives you the string "b'...'", not the decoded body. The parsed values then fail with could not convert string to float. The module already has a _bytes_to_str helper. It just wasn’t used here.
2. camelCase label. The Vevor sends solarRadiation where the interceptor expects solarradiation. Without the extra mapping, the log fills with “unrecognized parameter” warnings and solar radiation is never recorded.
3. Rain. The station sends both rainin (rain in the last hour) and dailyrainin. weewx works out rain per interval from the running daily total, so rainin is ignored to avoid counting the same rain twice.
4. Timestamps. The console’s clock was an hour ahead. It doesn’t handle daylight saving time, and on an isolated network it can’t reach an NTP server to correct itself. Records were being stamped an hour in the future. The Pi’s clock is correct, so the patch ignores dateutc and stamps each packet when it arrives.

5. The access point: hostapd, not NetworkManager

This part took the most time.

Raspberry Pi OS uses NetworkManager, and NetworkManager can run a wifi hotspot. An open hotspot worked fine. A WPA2 hotspot never completed a handshake with any client, not just the weather station. A kernel and firmware upgrade didn’t change that. Switching to plain hostapd got WPA2 working.

First, tell NetworkManager to leave wlan0 alone (/etc/NetworkManager/conf.d/99-weewx-ap.conf):

[keyfile]
unmanaged-devices=interface-name:wlan0

Then /etc/hostapd/hostapd.conf:

interface=wlan0
driver=nl80211
ssid=weewx-ap
country_code=US
ieee80211d=1
hw_mode=g
channel=6
ieee80211n=1
wmm_enabled=1
macaddr_acl=0
auth_algs=1
ignore_broadcast_ssid=0
wpa=2
wpa_key_mgmt=WPA-PSK
wpa_pairwise=CCMP
rsn_pairwise=CCMP
ieee80211w=0
wpa_passphrase=<your passphrase>
eapol_version=1
wpa_pairwise_update_count=10

The slow handshake

Even with hostapd and the default settings, the weather station failed to join about nine times out of ten. With debug logging on, hostapd showed the reason:

WPA: PTKSTART: Retry limit 4 reached

The station’s BL602 wifi chip is slow to answer the WPA2 4-way handshake. hostapd sends message 1, waits, resends, and gives up before the station replies. The last two lines of the config fix it:

• wpa_pairwise_update_count=10 lets hostapd resend handshake messages up to 10 times instead of 4, so a slow client has time to answer.
• eapol_version=1 makes hostapd use the older EAPOL version, which some embedded wifi stacks handle better.

After that change, the station joined on the first try every time. If you turned up hostapd’s logging to debug this (logger_syslog_level=0), turn it back down afterwards.

6. DHCP and the DNS lie

/etc/dnsmasq.d/weewx-ap.conf:

interface=wlan0
bind-interfaces
except-interface=lo
dhcp-range=192.168.5.10,192.168.5.254,1h
dhcp-option=option:router,192.168.5.1
dhcp-option=option:dns-server,192.168.5.1

# Never forward upstream. Only wunderground.com resolves, to this Pi.
no-resolv
no-poll
address=/wunderground.com/192.168.5.1
address=/#/

address=/wunderground.com/... covers the domain and all its subdomains, so rtupdate.wunderground.com and weatherstation.wunderground.com both resolve to the Pi. address=/#/ with no address answers every other name with NXDOMAIN. no-resolv means dnsmasq never asks an upstream server, so the station can’t learn a real address for anything.

I used 192.168.5.0/24 for the AP subnet because the console’s own setup hotspot uses 192.168.4.1, and I didn’t want the two to overlap.

7. Redirect and isolation firewall

A small script sets the address on wlan0 and loads an nftables table, at /usr/local/sbin/weewx-ap-net (mode 755):

#!/bin/sh
set -e
ip link set wlan0 up
ip addr replace 192.168.5.1/24 dev wlan0
nft delete table inet weewx 2>/dev/null || true
nft -f - <<'NFT'
table inet weewx {
	chain prerouting {
		type nat hook prerouting priority dstnat; policy accept;
		iifname "wlan0" tcp dport 80 counter redirect to :8090
		iifname "wlan0" tcp dport 443 counter
	}
	chain input {
		type filter hook input priority filter; policy accept;
		iifname != "wlan0" accept
		ct state established,related accept
		udp dport 67 accept
		meta l4proto { tcp, udp } th dport 53 accept
		tcp dport 8090 accept
		icmp type echo-request accept
		counter drop
	}
	chain forward {
		type filter hook forward priority filter; policy accept;
		iifname "wlan0" counter drop
		oifname "wlan0" counter drop
	}
}
NFT

• prerouting sends the station’s HTTP requests to the interceptor. The port 443 rule only counts packets, so you can see whether the station ever tries HTTPS.
• input lets wifi clients reach DHCP, DNS, the interceptor and ping, and nothing else on the Pi. SSH and nginx can’t be reached from wlan0.
• forward drops all traffic to or from wlan0. IP forwarding is also left off (no net.ipv4.ip_forward=1, no masquerade), so the isolation doesn’t depend on a single setting.

A oneshot unit runs the script at boot (/etc/systemd/system/weewx-ap-net.service):

[Unit]
Description=weewx AP: wlan0 address, port redirect and isolation firewall
Before=hostapd.service dnsmasq.service
After=NetworkManager.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/sbin/weewx-ap-net

[Install]
WantedBy=multi-user.target

hostapd and dnsmasq each get a drop-in (/etc/systemd/system/hostapd.service.d/weewx-ap.conf and the same path for dnsmasq.service.d):

[Unit]
After=weewx-ap-net.service
Wants=weewx-ap-net.service

Use Wants=, not Requires=. With Requires=, restarting weewx-ap-net to reload the firewall also restarts hostapd, and the station is kicked off the network.

systemctl daemon-reload
systemctl unmask hostapd
systemctl enable --now weewx-ap-net hostapd dnsmasq

8. Serving the reports

/etc/nginx/sites-available/weewx:

server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name _;
    root /var/www/weewx;
    index index.html;
    location / { try_files $uri $uri/ =404; }
}

rm /etc/nginx/sites-enabled/default
ln -s /etc/nginx/sites-available/weewx /etc/nginx/sites-enabled/weewx
systemctl reload nginx

LAN clients get the weewx Seasons report at http://<pi-address>/. Devices on the station’s wifi can’t load it, because port 80 on wlan0 goes to the interceptor. That’s fine, since the station is the only thing on that network.

9. Pointing the console at it

On the Vevor console’s setup page, join the weewx-ap network and enable Weather Underground with any station ID and key (1234 works, since the interceptor doesn’t check them). WeatherCloud can stay enabled. Its DNS lookups get NXDOMAIN and it quietly fails.

Checking that it works

iw dev wlan0 station dump                 # station is associated
cat /var/lib/misc/dnsmasq.leases          # and got a 192.168.5.x lease
journalctl -u hostapd | grep "handshake completed"
nft list table inet weewx                 # port-80 counter rising, drop counters at 0
journalctl -u weewx -f                    # "Added record ..." every 5 minutes
curl http://<pi-address>/                 # weewx Seasons page

In nft list table inet weewx, the port 80 redirect counter should go up with every upload. The forward drop counters should stay at zero. If they start climbing, the station is trying to reach something other than “Weather Underground”.

Gotchas, collected

• NetworkManager’s WPA2 hotspot on the Pi 3 (brcmfmac) never completed a handshake. An open hotspot worked. hostapd fixed it.
• The BL602 chip answers the handshake slowly. Set wpa_pairwise_update_count=10 and eapol_version=1, or expect “Retry limit 4 reached”.
• The interceptor’s POST handling has a Python 3 bug, which shows up as could not convert string to float.
• record_generation = hardware saves no records with the interceptor. Use software.
• The console clock is an hour fast and can’t sync on an isolated network, so use the Pi’s clock for timestamps.
• Don’t overlap the console’s setup subnet (192.168.4.x).
• sqlite3 isn’t installed by default. Query the archive with the venv’s Python sqlite3 module instead of installing the CLI.
• Run git in ~/weewx as the weewx user. As root, git refuses with a “dubious ownership” error.

The station now posts its “Weather Underground” updates to a Pi in the same house. The station has no idea, and Weather Underground never hears from it.

Then I opened my home network dashboard and the Time tab was red:

ssh: connect to host <time-server> port 22: No route to host

This is the story of what actually broke, which was not the thing I assumed — twice — and of the twenty-four days it sat broken without anyone finding out.

(Addresses below are illustrative. Assume a home LAN of 192.0.2.0/24 with a DHCP pool spanning 192.0.2.5 – 192.0.2.100.)

Wrong theory #1: the Pi is dead

“No route to host” plus no ARP reply looks like a box that’s powered off. It wasn’t. A quick sweep found the Pi alive and well at a different address — one inside the DHCP pool, holding a lease that had been renewed hours earlier.

So the Pi hadn’t died. It had moved, and DNS hadn’t.

Wrong theory #2: it’s dual-homed

Checking the router’s leases and reservations turned up something that looked like a smoking gun: there was a reservation for this host at an address above the pool, on its wired MAC, and a separate dynamic lease on a MAC one digit higher. On a Raspberry Pi the Wi-Fi MAC is typically the Ethernet MAC plus one:

aa:bb:cc:dd:ee:1a   eth0    reserved, above the pool
aa:bb:cc:dd:ee:1b   wlan0   dynamic lease, inside the pool

Two interfaces on one subnet, two valid answers to “where is this host” — classic ARP flux, and a tidy explanation for how the address drifted without anyone noticing.

It was also wrong. The Ethernet cable had been plugged in minutes earlier, while debugging. Before that, this box had been Wi-Fi only for its entire life, and the wired reservation had been sitting there dormant with nothing behind it. There was no dual-homing during the outage, because there was no second interface.

Two theories, two facts that killed them. Worth writing down, because the debugging value here was entirely in the timeline, not the topology.

The actual cause

The pool covers the low half of the subnet. Reservations, by convention on this network, live above it — and 92 of them do.

The time server was not one of them. It had an ordinary dynamic lease, inside the pool, on its Wi-Fi interface. And a while back I had added a static DNS host mapping pointing its hostname at that address.

That is the bug, and it’s worth stating plainly:

A permanent DNS record aimed at an address inside the DHCP pool, with no matching reservation, is a time bomb. The router is free to hand that address to something else. The name keeps resolving — it just resolves to the wrong host, or to nothing.

For weeks it looked fine, because DHCP leases renew and a host that stays up tends to keep its address. Nothing forced the issue.

Then something did.

The trigger

dmesg on the Pi, scrolled back far enough:

[1013882.469046] brcmf_fw_crashed: Firmware has halted or crashed
[1013882.511640] brcmf_cfg80211_get_tx_power: error (-5)
[1013883.092057] mmc1: card 0001 removed
[1013883.314334] mmc1: new ultra high speed DDR50 SDIO card at address 0001
[1013883.315903] brcmfmac: F1 signature read @0x18000000=0x15264345

The Broadcom Wi-Fi firmware halted. The SDIO card was removed and re-enumerated — note the PHY renumbering from phy0 to phy1 in the surrounding lines. The interface came back up, requested an address fresh, and got a different one from the pool. The old address went back in the pool, where it sits unallocated to this day.

Kernel ring buffer timestamps are seconds-since-boot, so converting them to a wall-clock date is worth doing:

python3 -c "
import datetime
uptime, event = 3107742, 1013882      # from /proc/uptime and the dmesg stamp
boot = datetime.datetime.now() - datetime.timedelta(seconds=uptime)
print('event:', boot + datetime.timedelta(seconds=event))"

That put the crash 24 days before I noticed. The DNS record had been pointing at an empty address that whole time.

There were plenty of warnings that this radio was unwell, too — a tight loop of brcmf_cfg80211_scan: Connecting: status (7) failures, and repeated brcmf_set_channel: set chanspec ... fail, reason -52 before the crash itself.

Why nothing told me

Here’s the part I find most instructive, and I got it wrong on the first pass.

My initial explanation was that SSH connection multiplexing had masked it — the dashboard keeps a ControlMaster connection to each host, so a live socket can outlive the correctness of what it points at. I’ve been bitten by that before, on this same network, with a stale known_hosts entry that hid behind a mux socket for six days.

But it doesn’t hold here. The mux is configured with ControlPersist=300 — the master exits five minutes after last use — and this endpoint is only polled when someone actually opens the dashboard. No socket survived 24 days. The failure today was an immediate No route to host, which is exactly what you’d expect from a fresh connection attempt, not a stale one.

The real answer is duller and more useful: the health endpoint had been reporting this failure correctly since the day of the crash, and nothing was watching it.

I had built the monitoring. /api/health returns a per-subsystem status, the Time entry had been ok: false for 24 days, and it was completely accurate the entire time. It’s just that the only way that information ever reached a human was if a human opened the page and looked at it.

A health endpoint nobody watches isn’t monitoring. It’s a status page that happens to be correct.

That’s the actual gap, and it’s a much better thing to have learned than “SSH multiplexing is tricky.”

The fix

The cable I’d plugged in during debugging turned out to be the right answer anyway: move the box to Ethernet, where a reservation already existed at an address safely above the pool. Update DNS to match. That’s it.

Then make the configuration explicit. The dashboard had been finding the time server through a default value compiled into the code — no environment variable set anywhere, so it silently fell back to a hardcoded address that had been correct when it was written. Making it an explicit setting is the whole lesson of this outage applied to configuration: an implicit default that used to be right is indistinguishable from one that still is.

One more trap: set doesn’t always replace

Applying the DNS change should have been a one-liner. On EdgeOS/Vyatta:

set system static-host-mapping host-name myhost.example.net inet 192.0.2.151

I ran it against a hostname that already had a record, expecting a replacement. Instead the host started resolving to both addresses:

$ nslookup myhost.example.net <router>
Name:   myhost.example.net
Address: 192.0.2.71
Name:   myhost.example.net
Address: 192.0.2.151

The inet node is multi-valued. set appends to it. And my own tooling hid the result, because the config parser read that node with a “give me the single value here” helper that returned the first entry and discarded the rest — so the API cheerfully reported one address while the router was serving two.

The fix is delete then set. The broader lesson is that on tree-structured config systems you have to know a node’s arity before you write to it, and “read it back to confirm” only works if your reader can represent what’s actually there. A parser that can’t express the bug can’t show you the bug.

Auditing for the rest of them

If this happened once, it has probably happened elsewhere. The check is mechanical: for every static DNS mapping, is the address inside the DHCP pool, and if so does a reservation exist for it?

inpool = lambda ip: pool_start <= ip_address(ip) <= pool_end

for record in dns_records:
    if inpool(record.ip) and record.ip not in reservations:
        print("time bomb:", record.hostname, record.ip)

Three more turned up:

• A NAS. Unreserved dynamic lease, with a service hostname pointed at it and its address hardcoded into a stack of media containers. Same failure mode as the time server, aimed at something far more disruptive. Moving it is now a small project rather than a config edit, purely because the address got copied into a dozen places instead of a name.
• A syslog VM. Statically configured on the VM itself, inside the pool, no reservation. Nothing stops the router leasing that address to a new device and colliding with the log collector. The pool is over half allocated, so that’s not hypothetical.
• A dead record for a host that no longer exists.

The punchline

With the time server fixed and the dashboard green again, I ran one last check on the Pi:

chronyc clients

One client. And it was my own workstation, from a diagnostic query I’d run ten minutes earlier while testing.

Nothing else on the network has ever been syncing to it. I built a GPS-disciplined, PPS-locked, sub-microsecond stratum-1 clock, wrote a blog post about it, and never actually pointed anything at it. The 24-day outage didn’t degrade anyone’s time, because there was no one to degrade.

That’s its own kind of monitoring lesson. The correct next step isn’t just pointing hosts at it — it’s having the router hand out the NTP server via DHCP so clients get it without anyone remembering to configure them, and pointing them at the hostname rather than an address, so the next time something moves, DNS does its job.

What I’d tell past me

1. Never point a static DNS record at an address inside the DHCP pool. If a name is permanent, the address behind it needs a reservation.
2. Reference names, not addresses. Every hardcoded IP is a future outage with a longer fix. The NAS is a project instead of an edit for exactly this reason.
3. A health endpoint nobody watches isn’t monitoring. Mine was right for 24 days and told no one.
4. An implicit default that used to be correct looks exactly like one that still is. Make deployment config explicit.
5. Convert dmesg timestamps to wall-clock time early. The entire diagnosis turned on one arithmetic step that dated the crash.
6. Check a config node’s arity before writing to it — and make sure your tooling can represent the wrong answer, or it can’t show it to you.

The uncomfortable one is #3. Every other item is a mistake I made once and can fix. That one is a category of mistake I’ll keep making as long as the only consumer of my monitoring is me remembering to look at it.