Server crash, backup fail, manual account restorations.

It happens to everyone in the hosting industry: hard drive failure. It happened to me on New Year’s Eve. Worst of all, I had not been watching my backups properly. We use cPanel, and its backups are amazing, but sometimes they are too amazing, because you don’t have to watch them. They are reliable in that you can just restore them and have a fully functional website back in seconds. Been there done that, and I had set cPanel to backup accounts a long time ago and just assumed they were working well. In fact, the backup drive was filling up and not all accounts on the server were being backed up properly. Needless to say, when the hard drive failed enough to cause kernel panics in FreeBSD, our only option was to reinstall on a new drive. That is when the fun started.

Once the system was reinstalled, and cPanel was installed, I had the chance to look at the backup situation. The situation was grim, and there was a lot of work to be done. Of the 70+ accounts on this server, 29 were not backed up properly via cPanel. The 40+ accounts that had cPanel backups were restored in minutes with no data loss, but the other 29 were not so lucky, and this website was one of those.

cPanel backs up roughly the following:

• Home directory
• public_html directory
• MySQL data including databases, users, tables, and the data in the tables
• Subdomains, parked domains, and addon domains (like geekpowers.com)
• Email accounts
• Ftp accounts

None of that was backed up in a restorable package, so the process was as follows:

1. Create the account manually via our billing system, which creates the cPanel account (luckily the billing system’s account had a healthy backup!).
2. Copy the /home directory and public_html directory from the old hard drive (luckily it was still usable)
3. Restore all subdomains and addon domains, pointing them to the proper directory under public_html
4. Add all databases by looking in /var/db/mysql for the username_ prefix (luckily cPanel prefixes databases with the username).
5. Copy the /oldvar/db/mysql/username_database/ contents into the matching database location in /var/db/mysql/username_database.
6. Search through the user files for the database name, and look at config files for the username & password to set as having access to the newly created databases.

Unfortunately, there was nothing to be done for email accounts and ftp accounts… that data was lost, and there was nothing for me to look at to refer to for restoring them.

I am simply amazed that the process above seemed to work. It felt so hopeless for a couple days, and I was really dreading the thought of telling customers that their data was just gone… customers that have been with me for almost 8 years. Thanks to a little ingenuity and a lot of luck, though, that did not have to happen, and we were able to restore all accounts.

Lesson learned: do not take backups for granted. I am already in talks with my datacenter about an external backup solution they offer which runs RAID 6. I don’t ever want to feel dread when a hardware failure occurs again. I want to be prepared to be back up in the shortest amount of time possible by keeping the most solid backups I can.