b.triplepat.com is down
Resolved
Mar 31 at 01:12am HDT
The machine b.triplepat.com became unavailable and eventually rebooted due to a "power event" and corresponding Google Cloud outage in its datacenter. When it came back up, the containers did not all start successfully at boot-time.
It looks like the internal DNS for the docker-compose network didn't come-up successfully, which meant that the nginx config could not resolve the internal names for redirects, which meant that nginx refused to start. (Another point for https://isitdns.com)
The power event in GCP: not our fault
The DNS stuff not working after boot: at least a little bit our fault, and certainly something we might want to work to remediate.
No user check-ins affected, because all other servers were up and running fine this whole time. No data loss. Resiliency works!
The only TODO item is to make sure our systems reboot cleanly.
Affected services
b.triplepat.com
Updated
Mar 30 at 08:50pm HDT
b.triplepat.com recovered.
Affected services
b.triplepat.com
Created
Mar 29 at 10:57am HDT
b.triplepat.com went down.
Affected services
b.triplepat.com