Paranoia is a Planning Tool
by Jerry Stern
Webmaster, PC410.com, Startupware.com
As seen in the January 2010 ASPects
Backup software is for backing up computers. Vague, isn’t it? There are lots of types of backup software, and not one of them can back up the information, business methods, and operational details of your business. The whole category is based on a false premise, that if you copy the hard drive, or the documents folder, you’re safe. False. That’s good enough for a home user, who can, and does, just copy pictures from an old computer to a new one, and then tosses the old chassis into the electronics dumper at the local recycling center. Or more likely, puts it in the basement, which serves as yet another type of archive. Business needs are altogether different, and a copy of ‘My Docs’ doesn’t do the job.
Typical Business Backups
What do most businesses do for a backup? The most common backup type is, well, nothing. Just print all the emails, buy lots of paper and file cabinets, and hope for the best.
Documents Backup
Second-most popular backup type: Copy ‘My Documents’ onto an external drive. Let’s call it a file-level backup process.
Definition: File-level backup, an uncompressed backup of documents and other data files from a computer, preferably on a device that can be read on nearly any computer. A USB or external SATA (eSata) drive is a good choice. DVDs may be good enough, if there are multiple sets, some stored off-site.
File backups work well for, well, documents. A good tech can restore documents onto a new computer in a few minutes from a file backup. Databases will require special attention. Emails are usually in a database, and if the file backup included the obscure file location, great. Here’s a typical path for Mozilla Thunderbird’s emails and address book: ‘c:\Users\Joe_PC_User\AppData\Roaming\Thunderbird\Profiles\ixcute12.default’ Part of that filename, the ‘ixcute12.default’, is total random junk, so there’s some extra work in configuring that to work on a new installation. In this case, the trick is to use the Thunderbird ‘profile manager’ to create the new data location, and move the data there manually–the data won’t be in the same place on old and new computers.
OK, so we’ll copy up the entire ‘c:\Users’ or ‘c:\Documents and Settings’ folders, right?. No, not likely. Standard Windows permissions block that, and some of the hidden data will be missed. The trick is to back up as each user both the Documents and AppData folders.
So if you rely on file-level backups, you must know where all your data is, and be sure to back it up. That’s really not good enough, as we’ll see later on.
Image Backups
The third-most popular is an image backup of the ‘C:’ drive of the file server, and maybe that goes to the boss’s workstation. Might be manual, when the secretary or system administrator gets around to it, or it might run nightly or weekly.
Definition: Image backup, a copy of the entire hard drive, in a format that can be replaced on the system after a drive failure. Could be on DVDs, or maybe an external hard drive, swapped on a schedule so that all the copies of the drive are never hooked up to power at the same time (that’s when the thunderstorm will start), or in the office at the same time (that’s the beginning of the flood). Never connect every copy of your data to your computer at the same time.
Images are terrific for recovering from hard drive meltdowns, but not very useful after the office has been burgled or burnt; restoring a drive image onto the new computer bought to replace a box removed from service after these events won’t always work. The license for the operating system isn’t transferable if it came with the box as an OEM (original equipment manufacturer) product–that’s what you get when you order a pre-built server from the big computer companies.
Worse, the operating system doesn’t like the new hardware. Yes, a good tech can make it work, maybe. Tricky, if there is RAID storage (a Redundant Array of Independent Drives) involved. Let’s keep it simple, and say there’s a pair of hard drives, mirror-copied. (That’s RAID level 1.) The computer fries beyond the point of repair during a lightning strike, so a new computer is brought in. The tech restores the image onto the new box. Boots the new box. Nothing useful happens. Why? Because the system needs a driver for the new RAID drive controller, and that isn’t in the image. The fix is to abandon the array, and just restore onto one drive without using the RAID controller, and then install the driver for the RAID controller, and then shut down, move the drive cable to the RAID card along with a blank drive to complete the mirror, configure the mirror in the RAID card BIOS, and restore the array. Yes, a good tech can do all that; shouldn’t take more than a day or so; in a recent rebuild here, 40 Gb of data and programs, on a mirrored array, took 13 hours for the card to rebuild.
But let’s be real-world here. When faced with a PC disaster, the not-always-tech-savvy boss will likely say, “I’m paying for a new box. I want that ‘Ooh, Shiny!’ stuff, uh, what is it, a new operating system?” Well, that fixes any license issues from OEM software on the old box not being transferable. So at that point, it’s just a matter of restoring the image of the hard drive to an external USB hard drive, and working from there, transferring documents to the new server, and rebuilding the server’s lists of users, machines, and user rights. Uh, no. Not from an image backup, you won’t. Documents are no problem. Databases are more work. But the server configuration isn’t stored in a useful way in an image backup.
Software Backups
If your computer crashed, could you tell your PC tech what software to re-install? And provide a stack of disks, a list of URLs, and a set of license keys for everything? Not without planning ahead. It’s the same job as buying a new computer; you have to rebuild your entire work area.
So, to manage software backups, you will need 1) a downloads folder of install programs for programs you’ve purchased online, 2) a file backup of the licenses of programs purchased online, each printed to a PDF (don’t depend on a key in an email being readable when you need it, or a printout being unburnt), 3) printouts or PDFs of site license keys and product support log-ins, and 4) a lot of luck, because the keys and software aren’t enough in a few cases. Every type of office has the program everyone loves to hate. It’s usually a specialized database. It frequently uses hardware locking for theft prevention, and it’s usually out of date.
Here’s the usual scenario for such a program in a disaster situation: A computer is stolen, fried, or drowned, along with the disks sitting in it, on it, and within 10 feet of it. The boss says ‘Rebuild’, and the tech discovers these issues: 1) The program everyone needs has to be installed on new hardware, not restored, and the drive image can’t do that. 2) There was no offsite backup of the software install program, or the license key. 3) The version of the software in use was six years old, the old version won’t run on new hardware and new operating systems, the software publisher doesn’t provide old-version downloads or support, has changed the database format three times in those six years, and the current version only imports the old data from the two most recent versions of the program.
Real-world example: A client who doesn’t like his office environment to change, ever, decided to finally move his emails from Mozilla Mail (discontinued some years back) to Microsoft Outlook. It took three conversions to move the messages, first through an external converter program to convert ‘mbx’ files to ‘eml’ files, and then a pass through Outlook Express, and then finally an import into Outlook. That’s work, around one to two hours per user, and practical as long as the conversion isn’t attempted into new hardware, which of course won’t have Outlook Express available–it’s not in Windows Vista or Windows 7.
Real-world example 2: Current versions of some accounting products can generally import data from approximately the three most-recent previous editions. Converting data from a really-old version requires installing a somewhat-newer version, converting the data, and then installing a current product and converting the data again.
In other words, take inventory: If you have old software, find out if the current version will import your data directly and easily, and keep copies of data-conversion programs along with your software backups.
Settings Backups
It should be clear now that you can’t rebuild a computer, much less your server, without knowing things that aren’t normally readable in a documents backup or an image backup. Every PC should have a PDF printer driver installed, and that should be used to create PDF printout files for all the settings that would have to be re-created manually in a disaster or a ‘missing computer’ upgrade.
Similarly, every program that stores settings in a database should have the ‘export’ or ‘backup’ options for that program used on a schedule, creating a ‘restore’ file that gets backed up with all the documents. Test the backups, before disaster hits.
Same thing applies to server settings (users, rights), address lists, favorites lists, etc.
Passwords
OK, here’s what most offices could do to backup the passwords: Photograph the PostIt Notes stuck to the monitor. No, really. (sigh)
Better: Use a password management program, and back up the password database, along with the software, regularly. Better yet, also print the entire password list to a PDF and back that up. There’s a pattern here–backup the databases as PDF printouts and as exports from each program.
How about browser-memorized web passwords? Most browsers can either export a list, or have an add-on that will allow backup and restoration of the password list. Use it to restore the list to your notebook, as a test–an untested backup isn’t trustworthy.
Special Items for Software Developers
- Backup your encryption certificates used for signing software.
- Backup your lists of upload site log-ins, and FTP passwords.
- If you self-host your web site, mirror it elsewhere.
Business Practices
Did you backup your secretary or boss yet? You should. Assume that a key person in the organization is unavailable and can’t even answer questions. It’s going to happen, hopefully temporarily, but plan ahead.
- Create a cheat sheet or a procedures manual. Include these items:
- How to process a sale, an order, a refund, any routine transaction.
- What program holds the address list, the password list, the bank account information, and who the backup person is who holds the offsite backups and master passwords.
- Who you know at your local bank.
- Where each category of report is filed, either on paper, or as a paperless report file.
- Who you buy your supplies and materials from.
- Who handles the web hosting (control panel URL, password, email accounts listing with passwords, FTP addresses), and how to use webmail on each account.
- What registrar holds the domain registration (more passwords here).
- What the automatic payments are every month, and which ones must continue in order for the business’ email, web, and other services to keep running.
- Insurance contacts and scanned contracts.
Oh, and back that up to a PDF, too, but don’t bury this one; store a paper copy off-site, along with backups. This is a start of a procedure manual. For a small shop, it might be all you need.
File Cabinets
The paperless office does exist, but mostly, we rely on dead trees far too much. Scan every document that you can’t continue the business without. Incorporation papers, tax returns, property deeds, and so on. Don’t rely on government offices to find them for you; they’ll do it, for a fee, in a few months, if you know the document numbers. Scan the important stuff, and store it off-site.
Where?
Finally, what kind of backups work? Onsite, offsite, web? First, no one backup type is adequate; I don’t care what the claims are for any one software product or service. Unless you use a service that includes real-time backup to multiple branch offices with no-downtime guarantees ($$$$$), use multiple backup types.
Web backups, like the software that sits in the background and uploads your data to a remote encrypted server, is just a documents backup with automation. Good start, but not adequate as an image backup for restoring a computer–it takes days to restore a computer backed up to a web service. Multiply that for an office.
Worse, some online backups are totally worthless for restoring a computer. When an online backup is tied to a single computer, and the computer is gone, installing the online backup account to a new computer may erase the original backup. Buyer beware–buy backup services that allow data restoration to a computer that did not originally create the backup. Sounds obvious, but it’s not an industry standard.
Better: buy an online storage account that lets you choose what to upload. It’s totally manual, but ensuring the survival of a business isn’t something that just happens automatically.
How Often?
Well, how much data can you afford to lose? If the answer is ‘one day of work’, then run nightly backups. If it’s ‘one second’, you’ll need real-time backup to multiple off-shore servers running mirror copies of your servers on multiple continents. Don’t laugh–that’s a real scenario, if you’re running an airline. In-between, the answer is usually:
1) A thorough inventory of what needs to be converted from data to documents (passwords, setup information, procedures, and so on), updated as-needed.
2) Image backups of every machine, updated after every significant or difficult software upgrade, or after each major Windows Service Pack installation.
3) Documents backups, daily or weekly for most offices, with alternate sets off-site.
Keep Everything?
And finally, a last word. Backups aren’t just for disasters. When you have multiple sets of file backups of your documents and settings and procedures, you have a time machine. Oops; I just edited a document, badly, and saved it over the original… OK, restore just that one file from last night. Oops, this file is corrupt, and I don’t know when it happened… OK, restore from three months ago.
Keep the old backups. If you’re burning DVDs, you don’t need to keep everything forever, but at the very least, keep one document set from at least every three month period, and sets from the end of each accounting year.
Overall, make the working assumption that the disaster you’re planning for is a hurricane. All the onsite hardware and backups are trash, half the staff is unreachable, and off-site backups from within a 10-kilometer radius are probably not available for a week. Take inventory early, and plan ahead.