Previous incidents
triplepat.com is down
Resolved Apr 15, 2025 at 3:37pm UTC
triplepat.com recovered.
1 previous update
triplepat.com is down
Resolved Apr 14, 2025 at 2:10pm UTC
The root cause is a failed nginx deployment due to what looks like a race condition and/or an overly-picky health check. We are auditing the health checks.
Redeploying the exact same config worked, so it's clear that this failure has something to do with either races or ephemeral machine state.
2 previous updates
b.triplepat.com is down
Resolved Mar 31, 2025 at 10:12am UTC
The machine b.triplepat.com became unavailable and eventually rebooted due to a "power event" and corresponding Google Cloud outage in its datacenter. When it came back up, the containers did not all start successfully at boot-time.
It looks like the internal DNS for the docker-compose network didn't come-up successfully, which meant that the nginx config could not resolve the internal names for redirects, which meant that ng...
2 previous updates
d.triplepat.com is down
Resolved Mar 31, 2025 at 10:14am UTC
TILAA has network issues more frequently than we (or they!) would like. This was one of them. Our every-node-is-a-master-node redundancy system means that everything still managed to work fine, however.
2 previous updates
a.triplepat.com was down after a OS upgrade + reboot
Resolved Feb 28, 2025 at 2:23pm UTC
We tried to upgrade the OS and reboot the a server yesterday to see if that was a safe and quick operation. It was not. We delayed in bringing it back online because we haven't launched and were in the middle of a major code change.
2 previous updates
d server was unreachable repeatedly
Resolved Feb 17, 2025 at 12:03pm UTC
Tilaa cloud (where the d server is hosted) had some network issues that made d.triplepat.com inaccessible. It was scheduled maintenance that ended up taking more things down than they expected.
https://status.tilaa.com/incidents/6hfj589ys0w1 and https://status.tilaa.com/incidents/4b2nb4tdw7mc are the issues on Tilaa's incident report page.
Unclear what was the root cause here, as our incident started between the two. But I suspect it ha...
a and b were briefly unavailable (multiple times!)
Resolved Feb 11, 2025 at 11:56am UTC
a and b servers were briefly unavailable, twice! Once on 7 Feb 2025 and a second time on 11 Feb 2025. The a server even had a third outage on 10 Feb 2025! This did not affect any users for two reasons:
- every server needs to go down before the check-in service becomes unavailable, and
- we're not launched.
Throughout these incidents, the servers triplepat.com, c, and [d](https://d...