My site was down for a few hours last night due to my server company having a major fuck up. Here is their excuse:
Data Center Update
Last night we experienced some issues in our data center that affected a portion of our clients. We wanted to take a moment to let you know what happened. At approximately 6:22 PM MST our primary Data Center experienced an electrical transient. An Automatic Transfer Switch on one of our legs of power failed, causing some servers to lose power. Our systems are designed with multiple, redundant sources of power that automatically transfer as needed to ensure servers stay up and running. In this instance, however, the switch itself failed. It also sent a spike back to one of our transformers, causing additional problems. Our technical teams of Data Center Technicians, System Administrators, and power experts responded immediately and worked continuously to bring all our customers back online.
Our data center utilizes many redundant systems including backup generators and 4 different UPS legs of power. These are tested every two weeks and regular maintenance is performed on all systems in order to provide the most reliable service possible. However, tonight our services did not live up to your—or our own—expectations. We are evaluating every aspect of this particular failure to eliminate the possibility of recurrence. We pride ourselves on delivering outstanding reliability and an unbeatable customer experience. Any feedback on how to improve our platform and services, as well as suggestions on how we can better communicate with you in the future, is welcome and appreciated. Thank you for your continued patience and support as we resolve this issue.
They claim to have redundant systems. I was immediately reminded of the Seinfeld episode where the car rental place knew how to take reservations. They just didn’t know how to keep the reservations. The point of having redundant systems is to avoid having a complete system crash. I hate to tell Bluehost, but their systems are not redundant. If they were, my site and thousands of others would not have gone down.
Anonymous says:
I fail to see how the Seinfeld clip has anything to do with the concept of mechanical redundancy. You must be referring to Bluehost’s failure to fulfill a contract.
A redundant system is still a system, subject to failure. A fail-safe system is a highly redundant system with many layers. It is still subject to failure.
———-
“My site was down for a few hours last night due to my server company having a major fuck up. Here is their excuse:”
——–
I think that you are being a little hard on the company. If you want hardened servers like DoD uses, well that’s going to cost you a little more.
Mechanical reality is a bitch.
Like or Dislike:
2 
2
3rd February 2013 at 3:47 pm
crazyivan says:
above
Like or Dislike:
1 
0
3rd February 2013 at 3:47 pm
Administrator says:
Actual picture of Crazy Ivan
Well-loved. Like or Dislike:
9 
0
3rd February 2013 at 3:56 pm
Mikey says:
Hey Jim,
After years in IT I’ll decode some of that for you.
“An electrical transient” is code for “we had a massive power spike from the electrical suppliers but we can’t call it that due to terms in our shitty supplier’s agreement with the power company that turned out equipment into slag before it could switch.”
“Sent a spike back to one of our transformers….” means “ten grand’s worth of gear is now scrap metal thanks to those fuckers”
“We are evaluating every aspect of this particular failure to eliminate the possibility of recurrence” — “working out how hard it is for us to generate our own power”
Well-loved. Like or Dislike:
6 
0
3rd February 2013 at 5:58 pm
Novista says:
Anony can yap about mechanical failure — nothing trumps the human element.
You would think AT&T with obligations to the government security system would have a clue. Back in 196x on xmas eve, [Redacted1] sends an open door alarm to [Redacted2]. The supervisor there sends out an untrained newbie to check the unattended microwave repeater site.
Supe tries talking newbie through some checks which only succeeds in triggering the changeover from mains to generators. Three times. The third time, after the 15 minute timeout, the contactor hangs but not to fear, the second level of redundancy is the NiCad battery system, faithfully maintained … that lasts 12 minutes. Part of our Cold War communications system was out nearly a day.
Like or Dislike:
1 
0
3rd February 2013 at 8:57 pm
AWD says:
Bluehost must be running the electricity grid for the Super Bowl
Well-loved. Like or Dislike:
9 
0
3rd February 2013 at 9:09 pm
Makati1 says:
The more complex a system the more it is open to failure. This is why the internet is going to go down someday and never come back. Why the surveillance system is more and more vulnerable as they keep adding to it. Why one power surge can take out the power for 50 million people.
Keep adding variable power inputs to the system in the form of wind generation, solar electric systems, gas fired power plants, etc. and large power draws coming online or dropping out at the same time and sooner or later something blows.
Like or Dislike:
2 
0
3rd February 2013 at 12:17 am
Kill Bill says:
Blue Host was just trying to fix the doorbell.
Like or Dislike:
2 
0
3rd February 2013 at 9:51 pm