Just wanted to post another update on this, because it happened a 2nd time yesterday, and for much longer (8 hours instead of 5). I was able to add an in-game message the 2nd time the site went down, by editing the web server.

We believe these two downtimes were both caused by a database table being locked by a bad query, this query used all the cpu usage and took a very long time to complete. While the database was locked up, the website stopped working entirely (since it couldn't access any data like the list of saves or whether or not you were logged in).

In addition, we think this query was the cause of the persistent 502 / 504 errors that have been happening for the last year. Occasionally, the site would go down for around 5-15 minutes with those error code, before coming back up. Triggering the specific query was enough to reproduce the database lockup, so it was definitely that one.


Simon found the bad query and it is fixed now, it won't happen again. The random 502 / 504 errors should go away, and the massively long downtimes will also not happen. We aren't sure why the short downtimes turned into extended downtimes, since the database usually recovers itself.

Also, there is no need to worry about the database or saves being corrupted. I take backups about once a month, just in case :)