I found this website because there used to exist in the days of old "alt.sysadmin.recovery". A newsgroup originally designated to people who ask for help along the lines of "HELP!" when they've really broken something. Since news is pretty much dead these days, I googled for a new home for hilarious sysadmin f**kups, and somehow ended up here:
Last post, 2007. I propose this place to be the new home of sysadmin disasters. It could bring hoards of traffic to your site, full of sparkling new talent just waiting for the day when they have to pour guilt and embarassment into your eternal tome.
So to get this one started, let us begin.
"RAAAAAAAARRRRR!!! ME STRONG!!!!!!!!!!"So, certain company PLC was having difficulty with their NFS machine. It had 3 TB of images on it for about 100 company-related websites, and it was struggling. Advertised as "the ultimate backup solution", this Sun XFire X4540 which was a 4U box with the capacity for 48 discs and no RAID controller just does what it says on the tin. It's a mass storage thing where you're supposed to use ZFS to manage where your mass lumps of data go. It is NOT an NFS server.
After months of campaigning, we -finally- convinced the powers that be that we needed something suited for the job and getting IO-wait load down on our websites, and the cheque was signed. We're getting some Netapps! It cost the company about £200,000. I know that isn't exactly the biggest order Netapp have ever had. I now work for a company for whom that is what they use for their dev system, but for us at the time.. well, you can buy a house for this. Netapp are a tried and tested company... what could POSSIBLY go wrong?
The manuals turned up before the kit did, and most idle guy at the time was someone whom I will call 'C'. So he started reading through the basics (none of us had used a netapp before), so when the netapps arrived, he was prime candidate to get them going and give us a bit of a brief on how to use them.
So when the kit arrives, he heads down to the datacentre, unpacks the boxes.. well, I'm not entirely sure what happened in there, but two hours later he came back. "Dead on arrival".
WHAT? Our department project manger who sat on my right went LIVID. £200,000, and they're BROKEN? I was trying to work at the time he was on the phone to the person at Netapp who said they couldn't get a guy around until next week to look at it. I had to lean to my left to concentrate. He nearly had a coronary. Eventually they arranged for someone to come in on a Saturday. This was on a Wednesday.
On Thursday, we phoned Netapp to apologise, and say that our insurance company would be covering the cost of the damage, and they needn't send anyone round, and could we have some replacement parts please?
Now, these netapps come with baseboards encased in metal boxes with these long cuboid tubes which you put these cuboid plugs down. Eight per unit, and we had 2 of them.
IN 'C's DEFENCE, the manual DID say "you have to push the plugs in quite hard". Alas, to fire one back at Netapp, there was no lip or dink or dent in the plugs to say which way up they go.
So he put them in the wrong way up, and pushed them in REALLY HARD!!
The sockets, the plastic, the silicon, the circuit board.. mashed to f**k. Utter write off.
It reminded us of that thing in Mortal Kombat where it says "FINISH HIM!!!!". In fact, for about a week, whenver 'C' arrived at his desk, we would beat our chests in homage to his immortal strength.
When a consultant from Netapp came round to talk to us, he admitted that they were "a bit of a silly design" and needed a slot to make them go the right way up, but he's the only person who ever mashed one to death because of it!
'C' goes down in history as the sysadmin who doesn't know his own strength :-)