Previous incidents

d.triplepat.com went down and up briefly during a deployment. It was detected by our down detector, so that means it counts as a tiny outage. As always, we run 5 master servers in parallel, so the service was still operating with 4x redundancy.

d.triplepat.com recovered.

We updated all the software on the node and that included the kernel and therefore also a reboot, which took a little longer than expected and caused this one node to have 5 minutes of downtime. 4 out of 5 servers continued just fine, and thanks to our master-master database design, we remained fully operational.

triplepat.com recovered.

A bad release and a bad deploy combined to cause the main node to go down for 25 minutes. As always, one node down (even the main node!) is no problem. The check-in service continued happily on 4 other servers in 4 different failure zones in 3 different clouds, and therefore no user data was lost.

c.triplepat.com recovered.

A brief network blip in AWS. Not a problem for us, with 4 out of 5 servers still up and accessible we remained quadruple-redundant.