Eerdere incidenten

We rebooted the server and it was slow to come back up. As always, one server down is not a problem for us.

triplepat.com recovered.

The root cause is a failed nginx deployment due to what looks like a race condition and/or an overly-picky health check. We are auditing the health checks.

Redeploying the exact same config worked, so it's clear that this failure has something to do with either races or ephemeral machine state.

The machine b.triplepat.com became unavailable and eventually rebooted due to a "power event" and corresponding Google Cloud outage in its datacenter. When it came back up, the containers did not all start successfully at boot-time.

It looks like the internal DNS for the docker-compose network didn't come-up successfully, which meant that the nginx config could not resolve the internal names for redirects, which meant that ng...

TILAA has network issues more frequently than we (or they!) would like. This was one of them. Our every-node-is-a-master-node redundancy system means that everything still managed to work fine, however.