b.triplepat.com is down
Opgelost
mrt 31 om 01:12 HDT
The machine b.triplepat.com became unavailable and eventually rebooted due to a "power event" and corresponding Google Cloud outage in its datacenter. When it came back up, the containers did not all start successfully at boot-time.
It looks like the internal DNS for the docker-compose network didn't come-up successfully, which meant that the nginx config could not resolve the internal names for redirects, which meant that nginx refused to start. (Another point for https://isitdns.com)
The power event in GCP: not our fault
The DNS stuff not working after boot: at least a little bit our fault, and certainly something we might want to work to remediate.
No user check-ins affected, because all other servers were up and running fine this whole time. No data loss. Resiliency works!
The only TODO item is to make sure our systems reboot cleanly.
Getroffen services
b.triplepat.com
Bijgewerkt
mrt 30 om 20:50 HDT
b.triplepat.com recovered.
Getroffen services
b.triplepat.com
Aangemaakt
mrt 29 om 10:57 HDT
b.triplepat.com went down.
Getroffen services
b.triplepat.com