IT manager approved downtime over lunch, but made a meal of it

Optimism is always risky, and defective hardware makes it indigestible
Who, Me?

The best part of the working day is lunchtime, but The Register tries to start Mondays in a pleasant fashion by bringing you a new installment of "Who, Me?" – the reader-contributed column in which you admit to your mistakes and detail your escapes.

This week, meet a reader we'll Regomize as "Hugh" who in the early 1990s served as IT manager for a distribution company in Canada.

"The core line of business application ran on serial terminals from an SCO server," he told The Register .

At the time, RAID was a new and expensive technology.

Hugh was therefore happy to use just one disk drive and a resilience regime that relied on nightly tape backups.

His bet paid off...

until the sole drive filled, necessitating an upgrade.

Which went smoothly as the server immediately accepted the new drive and, after several reboots, resumed operations.

Hugh was chuffed, but also a little irritated because the upgrade exposed some messy electrical cabling around the server.

He prudently wanted to clean that up even though it would again require powering down the computer.

"Since it wasn't going to take long, I was approved to do it over the lunch hour," he told Who, Me?

Hugh got the job done within 60 minutes, hit the server's power button, and anticipated swift resumption of normal service.

"It just sat there waiting," he wrote.

"I pulled the front cover off the server and the LED on the front of the drive was flashing a code.

I powered off the server, waited a bit, and tried again.

Same result."
This story comes from a time before Google, so Hugh's only way to get help was to call the disk drive manufacturer's support line.

"It was closed for lunch," he wrote.

"With much sweating, I waited the very, very long 30 minutes until the support line was open again."
He soon learned a component deep inside the drive had died, probably after he installed it during the initial upgrade.

That widget wasn't needed once the drive powered up, which explained why the server had survived several reboots.

But Hugh's wiring clean-up had killed the drive.

Hugh's company ordered a new drive, and once it arrived a flurry of activity brought the business back to life.

"Thankfully my employer was understanding that the likelihood of a failure like this was extremely rare and so would have been virtually impossible to foresee, so I kept my job," Hugh wrote.

"Then we invested in a RAID controller and a second drive."
This story has a happy ending because more than 30 years later, Hugh has moved on to work independently, but the distributor remains one of his customers!