02-12-21
Database server outage
VON: CHRIS
*** Ongoing story, scroll down ***
Dear customers,
we are unfortunately required to run the system on a backup database at the moment because of announce maintenance work in the main data center. This will incur additonal latency in database queries, and it is likely that it will cause problems with authentication. All authentication requests (logins) will work eventually; please bear with the server and try again if yours doesn't.
The problems will be over around 1400 UTC. (Update: see below)
We can only apologize.
Chris
UPDATE 1315 UTC: We are in contact with the provider. They are currently analyzing why the server is not coming up. Damn Corona... everyone is in Home Office. Please bear with us as we bear with them.
UPDATE 16:35 UTC: The database server is up again and resync'ing. There may be some data loss. It is possible that some transactions have not been recorded properly. If this is the case, please contact support and we'll sort you out.
UPDATE 18:01 UTC: The server was back up in the meantime, but it is unavailable again. We do have a ticket open with the provider of course and we are doing our best. Not sure whether this is true for the provider as well. Right now the whole system is running again on a backup database system and network latency will cause noticeable delays.
UPDATE Dec 5 12:15 UTC: We finally have good news. The server is physically up and running, and we are now in the process of resync'ing the databases. We should be able to switch over in a few hours, and everything should be smooth again then.
UPDATE Dec 5 15:05 UTC: Problem solved, main database syn'd and up and running again.
