Earlier today we experienced a complete outage that left us in a tough situation. Our network peering provider, PEER1, had a failure of equipment which isolated our servers, preventing any connections to the outside world. Due to the unique method of failure, the backup equipment did not kick in. As soon as we knew this was happening, we contacted PEER1 and had the issue resolved as fast as we could.
The equipment initially went down at 2:32am PST and we discovered it around 7:30am PST, with everything coming back online around 9:55am PST. Although we do not host our own email the equipment failure disrupted it resulting in us not receiving any emails from you. It was only once people started reaching out to us through social media and other channels that we learned what was going on.
Although we had fallbacks in place, the lack of communication is where we really slipped. So, thats why were sending out this email, and thats why were going to make some changes moving forward:
First, were going to establish a 24 hour emergency line for situations like this.
Second, were also investigating an externally hosted status page.
Third, were exploring additional monitoring services to let us know if anything is going wrong.
Something only tangentially related, but we are in the middle of deploying worldwide ad servers using multiple providers. Once these are launched, if a similar situation to this ever comes up again, even if adbutler.com goes down, ad serving would continue unhindered. This is currently in beta.
PEER1 is currently talking to their vendor to find out why their backup equipment didnt failover when the initial line failed. We are also talking to PEER1 to try and find out if there is a better method to avoid something like this from ever happening again.
If you have anymore questions, feel free to send them over to liam@sparklit.com.
As a whole, we wish to extend our apologies that this happened, and our thanks for your understanding. Weve got some of the best customers in the biz, and were glad to work for you.
Thanks,
SparkLIT
-- Edited by doug9270 on Wednesday 15th of June 2016 07:16:36 AM
__________________
Took the Queens Shilling and disappeared for 23 Years !!
