My NAS is a Raspberry Pi 5 running OpenMediaVault 7, with a Radxa Penta SATA HAT holding four drives. It serves NFS to a Proxmox host and to the box that runs Frigate, my camera NVR, which writes its recordings straight to an NFS export.

I found out it was broken sideways. I was on the Frigate host for an unrelated reason, ran df -h, and it hung. The load average was sitting at 30:

load average: 29.87, 30.29, 30.30

Frigate was stuck, not crashed

The load wasn’t CPU. It was about twenty ffmpeg processes, one per camera, all in uninterruptible sleep and all waiting on the same thing:

$ ps -eo pid,stat,etime,wchan:20,cmd | awk '$2 ~ /D/'
3286405 DN  04:31:52 rpc_wait_bit_killabl /usr/lib/ffmpeg/7.0/bin/ffmpeg ... /media/frigate/clips/previews/...
3286407 DN  04:31:52 rpc_wait_bit_killabl /usr/lib/ffmpeg/7.0/bin/ffmpeg ... /media/frigate/clips/previews/...
...

rpc_wait_bit_killable means they were blocked on an NFS call, and they had been for four and a half hours. The kernel log agreed:

nfs: server <nas> not responding, still trying

The worst part: the Frigate container still reported itself as healthy the whole time. The NFS mount is hard, so writes just block forever instead of failing, and nothing upstream noticed.

The NAS was up, NFS was not

The NAS answered ping, but port 2049 refused connections. On the NAS itself:

$ systemctl status nfs-server
○ nfs-server.service - NFS server and services
     Active: inactive (dead)

systemd[1]: Dependency failed for nfs-server.service - NFS server and services.

nfs-server depends on every exported path being mounted. OpenMediaVault exports bind mounts like /export/frigate, which point at the data drives under /srv/dev-disk-by-uuid-.... And lsblk showed only the SD card. All four data drives were gone.

The SATA controller wouldn’t start

The Penta HAT is a JMicron JMB585 on the Pi 5’s PCIe lane. The PCIe link came up fine, the controller was detected, and then the driver gave up:

ahci 0001:01:00.0: controller can't do 64bit DMA, forcing 32bit
ahci 0001:01:00.0: AHCI vers 0001.0301, 32 command slots, 6 Gbps, SATA mode
ahci 0001:01:00.0: 5/5 ports implemented (port mask 0x1f)
ahci 0001:01:00.0: failed to start port 0 (errno=-12)
ahci 0001:01:00.0: probe with driver ahci failed with error -12

-12 is ENOMEM. The controller can only address 32-bit memory, and the driver couldn’t get a DMA buffer it was able to use.

What changed

The journal made the timeline clear. The Pi had been running kernel 6.12.75 since April, with every boot showing the disks come up:

ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
...

Kernel 6.12.87 had been installed back in May but never booted, because the box had simply stayed up. This morning it rebooted into 6.12.87, and every boot since then failed with the same errno=-12. The kernel for 6.12.75 was still in /boot, so rolling back was an option. I didn’t need it.

The fix

A thread on the Radxa forum, Penta SATA HAT on RPi 5 stopped working, describes the same symptom after an apt upgrade and gives the fix: force the Pi 5 to set up that PCIe lane for 32-bit DMA. The overlay already ships with the Raspberry Pi kernel:

$ grep -A3 "^Name:.*pcie-32bit-dma-pi5" /boot/firmware/overlays/README
Name:   pcie-32bit-dma-pi5
Info:   Force PCIe config to support 32bit DMA addresses at the expense of
        having to bounce buffers (on the Pi 5).

One line at the end of /boot/firmware/config.txt, after the existing PCIe settings for the HAT:

[all]
dtparam=pciex1
dtparam=pciex1_gen=3
dtoverlay=pcie-32bit-dma-pi5

After a reboot, still on 6.12.87:

ahci 0001:01:00.0: controller can't do 64bit DMA, forcing 32bit
ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
ata4: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
ata5: SATA link up 3.0 Gbps (SStatus 123 SControl 300)

All four drives came back, the /srv and /export mounts came back, and NFS started. Nothing needed restarting on the Frigate side. Because the mount is hard, the client had been retrying the whole time, and it picked up the moment the server returned:

nfs: server <nas> OK

The stuck ffmpeg processes finished, and within a couple of minutes new segments were landing in the recordings folder again. Frigate threw away the few segments that were half-written when the NFS writes blocked (Invalid or missing video stream in segment ... Discarding.), once, and then carried on. The cost was about five hours of footage that never got written.

Gotchas, collected

  • A kernel can sit installed for months. 6.12.87 landed in May and broke nothing until the next reboot, which made “the upgrade today” look like the cause when the kernel was the real change.
  • errno=-12 here is not about RAM. The Pi had over 7 GB free. It’s about finding memory the 32-bit-only JMB585 can reach.
  • The forum thread mentions kernel 6.18. I hit it on 6.12.87 on Bookworm, so it isn’t tied to one kernel line. The same overlay fixed it.
  • The overlay has a cost. DMA goes through bounce buffers, which adds some CPU work. Behind gigabit Ethernet on a Pi, I haven’t noticed it.
  • hard NFS mounts hide outages. Frigate reported healthy for hours while none of its writes landed. Something should alert on NFS “not responding”.
  • Keep the old kernel. The previous kernel and initramfs in /boot were a ready fallback: copy them over /boot/firmware/kernel_2712.img and initramfs_2712 and hold the kernel packages.

The NAS is back on the current kernel with one extra line in config.txt, and the drives, exports and recordings are where they should be.