Jan 19, 2012, 10:55 AM
Post #9 of 26
It appears there was a server issue at the same time aswell, so two completely separate incidents were affecting site access.
Reproducing the email I just received from the hosting geeks:
As you are probably very much aware, we experienced a major outage on Wednesday January 18th, starting at 13:50 PST and ending at approximately
00:30 PST affecting all customers with servers in Vancouver.
The root cause of the outage is an explosion at 1221 West Hastings, about four blocks from our data centre. The explosion caused rebar to destroy an electrical box under the ground containing, among other things, 580 strands of fibre cable.
Unfortunately, this took our Vancouver facility completely off line (clients in London were unaffected).
This should _NOT_ have happened though. Gossamer has paid for and designed our data centre with diverse fibre paths in mind. We have fibre plans showing separate fibre going out the east and west exit of the datacentre, and taking completely separate routes. The diverse fibre connects to different upstream providers (Bell, Level3 and Peer1 currently), and we should have been able to handle this without impact to customers.
We do not yet have an answer from the fibre provider on how this could have happened. All efforts are currently focused on restoring service and ensuring connectivity to all customers. As of 00:30, a new fibre circuit has been provisioned and we are currently running off that and service has been restored to almost all customers.
Work is underway to repair the fibre, but since the damage is extensive (rebar caused heated asbestos everywhere so hydro crews in hazmat suits are cleaning the site before repair can even begin), we don't yet have an eta for returning to full service.
If you are still experiencing any problems, please do contact us at email@example.com or by phone at (604) 687-5804.
We will be updating our site at http://gt.net/ with more details as they become available.
This is the worst outage in the 10 years Gossamer has provided hosting, and I just want to apologize to all our customers. I understand the business critical nature of your sites and the extreme impact this type of outage can have. We will be doing everything in our power to make this right, and will be following up on this with much more over the next few days.
Please don't hesitate to reach out if you have questions or concerns about any of this.