Eerdere incidenten

b-mirror went down for unknown reasons and could not be connected to via ssh. It also stopped producing metrics.

A reboot fixed it, but a further investigation to figure out the root cause is also underway.

C is back up now too.

As always, because at least one of a,b,c, and d was up no data has been lost and no user experience was affected.

triplepat.com went down on a push. Tailscale was down (but this wasn't detected: thing to fix #1) so the bad push was unable to bind the internal services and refused to bring anything up.

We logged the machine back into our tailnet, redeployed, and everything was fine.

No user data was lost (again: for the outage to count as real every machine must be unreachable), but we now have a new thing to monitor and alert on to prevent such outages in the future.

We rebooted the server and it was slow to come back up. As always, one server down is not a problem for us.

triplepat.com recovered.

The root cause is a failed nginx deployment due to what looks like a race condition and/or an overly-picky health check. We are auditing the health checks.

Redeploying the exact same config worked, so it's clear that this failure has something to do with either races or ephemeral machine state.