Nothing that you do on your computer in the course of a day — or for that matter, a month — has anywhere near the significance of your backup procedure. A good backup will someday mean the difference between a minor inconvenience and a very costly disaster. It is only a matter of time before a system crash, burglary, fire, data entry error, or, heaven forbid, a ransomware infection will make your backup the only indispensable part of your system.
Consider these scenarios:
- You update the operating system (Windows) in your computer and subsequently find that you can no longer access your data. Your ledger and patient information are destroyed.
- You come into the office one day, only to find that every piece of office electronics you own is missing. On the further investigation, you find that all your tapes, data cartridges, and CD’s are gone as well, including those you had been using for your data backups.
- A power outage during a critical data update procedure corrupts several major data files.
- In an attempt to free up some disk space, you accidentally delete all the files in your SOS data directory.
- There is a fire at your office over the weekend and your computer system as well as all disks and tapes are literally melted.
- You have been making backups, but each day you use the same media for your backup. One day it is necessary to restore from your backup, but you find that the backups are not readable.
- You turn on your monitor and are faced with an announcement that all your files have been encrypted by a ransomware infection.
There and many other possibilities are out there, just waiting for the day that you can least afford the time and energy to deal with them.
What can you do to safeguard your data?
1. SOS Applications do not have a built-in backup program. Use high quality, brand name, data backup software, or the backup software that came with your tape, cartridge, CD/DVD drive, or other backup media. (Here at SOS we use Novabackup and Cloudberry Backup but there are many other possibilities.)
2. Configure your backup software so that all error checking, error correction, and/or verify options are turned ON. Also turn on the data compression feature, which is sometimes called something like “minimize space”. Enable the encryption option and if there is none, find other software. If there is an automatic compare feature, turn that on as well. We recommend that you always use the full backup option, rather that the differential or incremental options. In the event that you have to restore from a backup, having made a full-type backup will make restoring your data much easier. Save these options.
3. Next you may want to follow the instructions that came with your backup software for creating a setup or script that will permit you to backup only those files you want to save. You can, for example, have a setup that backs up only the data files in your \SOS\DATA folder, which is the folder containing the SOS database. In case of a problem when doing an update, it is a good idea to include the entire SOS folder and all its sub-folders.
4. All good backup programs include a notification feature that will tell the program to email a report to you at the end of every backup operation to let you know if any errors were encountered. Checking to be sure your backup is running correctly is critical; this option makes that part of the process quick and easy. Enable notifications and make reviewing the notification part of your daily procedure. SOS recommends that at least two members of your staff receive backup notifications.
5. Another feature offered by most backup programs is data encryption. If you will be transporting the backup out of the office, encryption is an essential safeguard to avoid a potential data breach. The loss or theft of your SOS backup would compromise the privacy of all your patients and result in serious and expensive HIPAA-related remediation, and potential civil or even criminal prosecution. Encrypting your backups, or the media on which you store them, is therefore absolutely mandatory. Make sure that you record the encryption key and that all parties who need it will be able to find it should the need arise, even years after you have gone on to greener pastures. Your backups will do you no good if you cannot decrypt them.
6. It is absolutely essential that you rotate your backup media. In short, what this means is that you never use the same cartridge, tape, or backup disk two days in a row. The bare minimum should be a rotation through three backup sets, but we strongly recommend that you purchase enough cartridges, tapes, or disks to allow you to backup each day of the week on a different tape or cartridge, plus several more for rotating off premises.
Use for backup on Monday Cartridge/tape/disk/CD/DVD #1
Use for backup on Tuesday Cartridge/tape/disk/CD/DVD #2
Use for backup on Wednesday Cartridge/tape/disk/CD/DVD #3
Use for backup on Thursday Cartridge/tape/disk/CD/DVD #4
Use for backup on first Friday of the month Cartridge/tape/disk/CD/DVD #F1
Use for backup on second Friday of the month Cartridge/tape/disk/CD/DVD #F2
(alternate the Friday backup media going forward.)
Note that there is a separate disk, cartridge or tape for each day of the week, Monday through Thursday. Friday will be our safety day, so there are five separate Friday backups. On the first Friday of the month, we make our backup and take it off premises. The following Friday you use your alternate Friday media and return the other Friday backup to the office.
The safety location is often the home of one of the people in the office, perhaps the owner of the practice or the person responsible for making the backups. Storing some backups off premises provides you with an extra margin of safety in the event of a fire, burglary, or other event that might result in the loss of the data on your hard disk, as well as any media stored near the computer. Some people use a fireproof box or safe at the office for the safety backups, but we see this as a compromise and strongly suggest storage in another location. For additional security, SOS recommends that you make periodic “archival” backups that are never over-written. Such archival backups can be made on CD-R, or DVD disks, assuming your backups will fit on a single optical disk. (Such disks are very inexpensive and take up very little space.) Archival backups should be stored in a secure off premises location, such as a bank safe-deposit box. This extra step will allow you to restore your financial data for an entire quarter or longer.
An alternative for archival backups is to make backups that are saved to an online repository. Amazon Web Services has an offering called Glacier that is perfect for archival storage of data and is ridiculously inexpensive (a fraction of a penny per gigabyte).
7. In addition to your multiple generations of local backups, you should consider using one of the many online backup services (do a web search for “online backup”), or have your IT consultant set up regular uploads of an encrypted backup to an off-site web server, preferably located far from your office. This copy will be your doomsday backup, to be used if a natural disaster or other event makes recovery of your regular backups impossible. With a backup of this sort, you can be back in business within a day or two even if your entire town were to be destroyed by a hurricane, flood, tornadoes, etc. The steps you would follow in the event of such a catastrophe should be written in your policy and procedures manual (another HIPAA requirement, by the way). Online backup services such as CrashPlan.com, Carbonite.com, iDrive.com, and MozyPro.com are inexpensive and easy to set up. In addition, most services save several versions of your backup, in case you need a copy of a backup from a couple of weeks ago and are not following the multi-generational strategy outlined above. If you go this route, be sure you select a vendor that provides for pre-transmission encryption. The only way you can be sure your data is not at risk is if you encrypt it BEFORE sending it off-site!
What we do here at SOS:
- Each night all critical data is backed up to another drive on the network, the destination folder being different each day of the week. That way we can very quickly restore data backed up over past week.
- Also, every night, using Novabackup, we backup the same data to removable external drives, one for each day of the week, and alternating on Fridays, as described above. The most recent backup goes off-site each night.
- SOS has copies of the four most recent end-of-week backups stored on Amazon’s somewhat more costly (a penny or two per gigabyte) S3 service. Each weekend the oldest weekly backup is replaced by a new one. Cloudberry Backup is used to create and transmit these backups to Amazon.
- Every month Cloudberry makes a permanent monthly archival backup on Amazon’s Glacier service, also using Cloudberry Backup.
The bottom line is that you can never have too many backups!
Note that as of October, 2013, new HIPAA rules went into effect that require you to execute a formal Business Associate Agreement (BAA) with any vendor who “creates, receives, maintains, or transmits protected health information”. That would seem to include online backup services. If you do not execute a Business Associate Agreement with this type of vendor, you might be in violation of the new HIPAA rules. At present, SOS knows of only one online backup vendor (Carbonite) that is willing to accept the level of responsibility and liability that such an agreement includes. Essentially, the vendor is on the hook if there should be a data breach. That could be a pretty darn big hook, so it is no wonder that they are not rushing to comply. Anecdotal reports indicate that MozyPro will sign a BAA if pressed, but I like that Carbonite has formal, published policies that state their status as a BA. There are probably other vendors who follow that policy as well, but we don’t have that information at present.
8. Have a disaster recovery drill on a regular basis. Start by shutting down your database and renaming your DATA folder in SOS to something else, such as DATASAVE. Now try to restore your backup, following the appropriate restore procedure for your backup software. When the restore is complete, open SOS to be sure the database is intact and contains all the data that you believe it should. If you cannot open the database, something is wrong with your backup procedure and you must correct it. If the program starts fine and you can access all your data, you know your backup procedure is working. You can now delete the DATASAVE folder.
If you follow the type of procedure outlined above, you can rest assured that your data will be safe and available should you ever have to restore from a backup.
Backing Up a Database While It Is Running
In some installations it is desirable to keep the database running 24 hours a day or to do a backup without stopping the database. You cannot backup a running SQL Anywhere database using any commercial backup software, so a special procedure must be used. (Even if you manage to make such a backup of the files while they are open, it is exceedingly unlikely that the database would run after restoring from it.)
There is a downside to running the database non-stop. There are several maintenance/clean-up procedures that run automatically when the database is started. If you never restart the database, it is possible that you might run into minor issues, such as the inability to log into the system with a particular user ID after that user disconnected from the database in an unusual fashion. See Database Tools in the Admin Module to do the cleanup manually. SOS recommends that you restart the database once each day to allow these maintenance procedures to run.
Even though you cannot backup the data files while the database is running, you can make a copy of the running database in a separate directory. You can then configure your backup software to backup the copy and to ignore the directory in which the production database files are located.
For this purpose, starting with SOS 2016 you will need user credentials for an account that has no rights other than to create a copy of the database files while they are in use. Backup-only user accounts are created on the Backup Users tab of the user account maintenance list in the Administration module. For full instructions, see Configuring Backup Users in the Administrator Guide (sosadmin.pdf, located in the SOS folder).
Create a batch file (CMD file) containing the following commands. This example assumes that the software is installed in the SOS folder on the C: drive. Make adjustments as appropriate. It is important to note that this command file must be run directly on the computer that runs the database. You cannot run it from a network workstation.
(SOS 2016 )
DBBACKUP -y -c "uid=<backup user account ID>;pwd=<backup user account password>;dsn=SOSDATA" <target directory>
Here is an example command line for SOS 2016, including an optional -y parameter to create the directory if it doesn't exist, and to overwrite existing files:
DBBACKUP -y -c "uid=BU_BILL;pwd=PASSword#1;dsn=SOSDATA" C:\SOSBU
where <target directory> is an existing directory in which you want the copy of the database to be created. This directory must be located on the computer running the database. It should NOT be a shared folder on another computer! Make sure that this target directory is included in your backup configuration so that the database copy is backed up, and set the backup to skip the \SOS\DATA directory in which the running database files are located. Most backup software will fail to make usable backups of a running database!
You can run a database validation or database copy interactively from the menus in the “DBA Utils” progam in the Admin Module. Once the program has started, go to the Tools menu. Run the validation by selecting “Check Database”. You can run an ad-hoc live database copy by choosing “Copy Database” in the Tools menu. Generally, live backups are done as a Windows scheduled task that executes one or more times each day.
It is absolutely essential that you monitor your backups closely to be sure that the current database files are being copied to your backup media. If you are using commercial backup software, be sure to turn on the “verify” or “compare” operation and inspect your backup logs or notifications every day!