Eerdere incidenten

Doing maintenance on b ended up running it out of (initially) IOPs and (eventually) CPU and RAM. A restart fixed things.

b-mirror went down for unknown reasons and could not be connected to via ssh. It also stopped producing metrics.

A reboot fixed it, but a further investigation to figure out the root cause is also underway.

C is back up now too.

As always, because at least one of a,b,c, and d was up no data has been lost and no user experience was affected.

triplepat.com went down on a push. Tailscale was down (but this wasn't detected: thing to fix #1) so the bad push was unable to bind the internal services and refused to bring anything up.

We logged the machine back into our tailnet, redeployed, and everything was fine.

No user data was lost (again: for the outage to count as real every machine must be unreachable), but we now have a new thing to monitor and alert on to prevent such outages in the future.

We rebooted the server and it was slow to come back up. As always, one server down is not a problem for us.