related:
.. last night the RAID controller died in one of our web servers. Yes it was a mirrored RAID array AND backed up incremental to another machine.
No amount of cloning/backup would have helped bring this server back quickly. But having the back ups did help us rebuild it efficiently.

As computer users we all do what we can but sometimes it's all for naught.

If a drive had died; fine. Two (even more) drives dying would have been recoverable. When the drive 'controller' goes, it's no longer an easy fix.
It was the last web server we own that was not yet redundant (but scheduled to be virtualized and made redundant) and it died of a hardware failure two weeks before the maintenance.

Rebuilding a new VM from a separate hardware backup is no fun. Even when it works.

Good news is we get to move on to the DB servers that much sooner.
Bad new; 4 hours sleep between 2PM yesterday and 7:30 PM tonight.
But all sites are back up and running, SSL and all.
In about 45 minutes from now the sync from the salvaged drives will be complete and then everything will be just like yesterday (right now some sites reflect late last week) .. plus we'll have redundant fail overs.

Did I mention I'm tired?

Mirroring is good and has it's place, but if the RAID controller itself goes like it did on us last night, a good old fashioned disk image backup was the most important thing to have available.
From here forward it's redundant fail over for us; get on the virtual machine host and point it to the redundant copy. Dang near real time restore (for the win!).




I do not work here, but the benefits are still awesome
Make your sound your own!