Three failures, and the loud one was the least interesting

A hobby project of mine started throwing 502s. It listens to air traffic control radio - a software-defined radio dongle tuned to the local tower, and a pipeline that turns what it hears into a page of transcripts. The service behind that page had been dead for two days before anything told me, and once I got in I found out the pipeline I believed was running had recorded nothing for 43 days.

The service runs in WSL on a Windows box, reached through Windows portproxy listeners. WSL was dead. Portproxy keeps accepting TCP after the thing behind it is gone, so the connection succeeded and nothing answered, which meant every check I had short of an actual request came back fine. You can't tell that apart from healthy.

Getting in to fix it was its own problem. RDP, WinRM, VNC and 22 were all filtered, and the recovery came down to physically touching the machine.

The page came back. The newest thing on it was 43 days old.

The SDR dongle feeding the pipeline had wedged on USB, and the container behind it had crash-restarted 665 times. There was a watchdog whose entire job was to catch that - and its cron entry had been returning Permission denied for a long stretch, so it hadn't actually run in months.

Fixing the permission made things worse. The watchdog checked container health by reading docker logs 2>/dev/null, and that container writes only to stderr. Suppress stderr and the watchdog sees an empty log, decides the container is hung, and restarts it. The dongle had already wedged the container on its own. The moment I chmod'd the watchdog back to life it started restarting a container that was fine, on top of the restarts that were already happening.

So there were three, and each one hid the next. The 502 was the only one loud enough to reach me and it was the least interesting.

All three of my health checks were ones a corpse could pass. What I actually needed was the timestamp on the last row written.