System Recovery after Controller Failed with No Backups
Situation: On a late Friday afternoon a Law firm called with their primary data server down due to a faulty raid controller. All attempts to recover the hard drive subsystem did not produce access to a working data volume. A loaner Raid controller with drives was dispatched, installed and configured. Upon trying to merge the backup tapes it became clear that the client's employee responsible for checking the backup system was not performing this task– the client had no good backups – all tapes were blank with the exception of a 9 month old tape, which was not considered of much value to the pending court cases – the client needed a solution with data by Monday 2 P.M.
Solution: A hard drive disaster recovery firm was contacted, arrangements were made to fly the drive directly to their repair location and upon recovery of the data fly it back over the weekend. While waiting for word of the possible data recovery, our technical team stayed onsite for 55 hours straight to build a new server with all software and required settings installed\tested. On Sunday, we completed final adjustments of the new server when we got word that the data recovery process was successful and that the client’s data had been recovered. Within hours of notification we picked up the data at the local airport, brought it back to the client’s office, restored and tested the data to the client’s satisfaction.
Lessons learned: The hard drive recovery firm charged $17,000 to recover the data while the client had already invested in a good backup system. The failure of the tape backup system was due to the client's employee being negligent changing the tapes and reviewing the backup logs. The client believed they were saving money by having a non-technical person in charge of the backups. Our recommendation was to either provide better training for the person responsible for the backups or pay a network support firm to perform this important role. Additional recommendations included testing their backup tapes by restoring data on a regular basis and having a tape library to relieve the local person from having to change the tapes as often.