Post-Mortem

What Happened?

The server hosting the public bot was incorrectly terminated due to a payment error. This has since been remedied with our hosting provider.

Length of Downtime and Timeline

The affected service was down for a total of 3 hours and 56 minutes.

Timeline

Several monitoring servers first observed the outage on April 30th at 11:22pm UTC, staff were paged 5 minutes later.

Downtime was acknowledged by Eartharoid at 12:14am UTC

Issue was fixed by Earthar...

EDIT: This outage was a result of a fault in our monitoring system, the service was unaffected.

The public instance of Christmas Countdown experienced an outage for 9 minutes.

This issue was not investigated due to its short nature and low priority service (why are you excited about Christmas in May?), but we suspect it could be a small software error or monitoring fault than an actual major issue.

Discord Tickets (Public instance) recovered.

Resolved.

Post-mortem

Timeline

The crash

The public instance went offline at 00:47 UTC on Friday, March 28th. The reason for this is unknown; it was not a scheduled update. However, the reason is not particularly important as occasional crashes and automatic recoveries after a few minutes are relatively normal.
Unfortunately, despite the container automatically restarting, the service was unable to recover this time.

Token is revoked

At 06:29 UTC, Discord revokes the bot's token due to ...

Discord Tickets (Public instance) recovered.

Discord Tickets (Public instance) recovered.

img.eartharoid.me recovered.

Discord Tickets (Public instance) recovered.