Post-Mortem

What Happened?

The server hosting the public bot was incorrectly terminated due to a payment error. This has since been remedied with our hosting provider.

Length of Downtime and Timeline

The affected service was down for a total of 3 hours and 56 minutes.

Timeline

Several monitoring servers first observed the outage on April 30th at 11:22pm UTC, staff were paged 5 minutes later.

Downtime was acknowledged by Eartharoid at 12:14am UTC

Issue was fixed by Eartharoid at 12:30am UTC

Host server was brought back online by hosting provider at about 3am UTC

Host server was detected as online by monitoring at 3:18am UTC, incident was automatically resolved.

How Can This Be Improved?

First, we are now looking at resolving these payment issues proactively and before server suspension so this issue does not crop up again.

Second, we are looking at the possibility of running a second Host server with a different provider for redundancy, this would also reduce this issue cropping up again (but may take time to implement).

Finally, we should respond to and fix these issues faster, a lot of the outage time was just us responding to the issue (though, a lot was also our hosting provider being slow at reinstating the suspended server)

Closing Notes

We sincerely apologise for this issue and any consequences as a result of this issue (lost revenue, upset customers, etc). We understand your frustration and vow to do better in the future.

If your use-case requires super high availability (99.9%+) we highly recommend you upgrade to a Managed bot. These are hosted on more premium and resilient hardware, and also help support us!

The issue with the hosting provider has been resolved.

We are aware of the issue and are working on determining the root cause and a fix.

An outage was detected and declared by the monitoring system.