The information in the Backing Up Your Data article is quite sufficient for most installations. The current document discusses advanced topics that will be of particular interest to those running the system in larger organizations with high data entry volume. The following topics include how to recover data entered since the most recent backup and how to backup up a running database. Much of this material is reprinted from the Backup and Data Recovery chapter of the SA Usage manual, with permission from Sybase, Inc. You can read this information directly in the online SA help, accessed from your Start > Programs > SOS Applications menu.
The material below explains how to use SA transaction log files to protect your data, how to make backup copies of your database and log files while the database is running, and how to recover after system and media failures.
Backing Up a Database While It Is Running
In some installations, it is desirable to keep the database running 24 hours a day. You cannot backup the running database using commercial backup software, so a special procedure must be used.
You can either:
1. Depend on Microsoft Volume Shadow Copy Services (VSS) to snapshot the database when using third party backup software after configuring your system to run the dbvss11 utility as a Windows service, or
2. (preferred) Use the included live backup utility to make a copy of the database in a separate directory, and backup that copy rather than the running production files in your backup set.
There is a downside to running the database non-stop. There are several maintenance/clean-up procedures that run automatically when the database is started. If you never restart the database, it is possible that you might run into minor issues, such as the inability to log into the system with a particular user ID after that user disconnected from the database in an unusual fashion. See Database Tools in the Admin Module to do the cleanup manually.
The recommended the following live backup procedure:
Create a CMD batch file containing the following commands:
(SOS 2015 and earlier)
C:\SOS\DBTOOLS /V
C:\SOS\DBTOOLS B=<target directory> -y
(SOS 2016 - Requires that you create a backup-only user account in SOS! See the SOSADMIN.PDF document in the \SOS folder of your system.)
c:
cd \sos\sa\bin32
DBBACKUP -c “uid=<account ID>;pwd=<account password>;dsn=SOSDATA” <target directory>
where <target directory> is an existing directory in which you want the copy of the database to be created. This directory must be located on the computer running the database. It should NOT be a shared folder on another computer! Make sure that this target directory is included in your backup configuration so that the database copy is backed up, and set the backup to skip the \SOS\DATA directory in which the running database files are located. Most backup software will fail to make usable backups of a running database!
Here is an example command line, including an optional -y parameter to create
the directory if it doesn't exist, and to overwrite existing files:
DBBACKUP -y -c "uid=BU_BILL;pwd=PASSword#1;dsn=SOSDATA" C:\SOSBU
You can run a database validation or database copy interactively from the menus in the “DBA Utils” program in the Admin Module. Once the program has started, go to the Tools menu. Run the validation by selecting “Check Database”. You can run an ad-hoc live database copy by choosing “Copy Database” in the Tools menu. Generally, live backups are done as a Windows scheduled task that executes one or more times each day.
Note that for security reasons copying the database is only possible on the computer running the database software.
It is absolutely essential that you monitor your backups closely to be sure that the current database files are being copied to your backup media. If you are using commercial backup software, be sure to turn on the “verify” or “compare” operation and inspect your backup logs every day!
If you plan to backup the running database with Backup Exec, NovaBackup, or some other backup software that uses Microsoft Volume Shadow Copy Services (VSS) to backup open files, then create and run a Windows service for the SA VSS Writer component, DBVSS11.EXE, as follows:
- Open a command window.
- On 32-bit Windows type:
CD \SOS\SA\BIN32 <enter>On 64-bit Windows type:
CD \SOS\SA\BIN64 <enter>
- On 32-bit Windows type
dbsvc -as -s Automatic -t vss -w VSSWriter “c:\SOS\SA\BIN32\dbvss11.exe” <enter>On 64-bit Windows type:
dbsvc -as -s Automatic -t vss -w VSSWriter “c:\SOS\SA\BIN64\dbvss11.exe” <enter>
Once you have created the service, you should see it listed in your Windows Services as
“SQL Anywhere – VSSWriter”
It is absolutely essential that you test your backup during a period when the database is actively in use by restoring your backup, restarting your system, and validating the restored database as described in “Validating the Database”, above.
Live Backup and Database/Log Mirroring
SQL Anywhere also has the capability of running a continuous live backup of the transaction log on a separate computer, and even doing a failover database server running on a second computer. For discussion of these topics please see the appropriate topics in SQL Anywhere Database Administration manual located on your system. Go to Start > Programs > SOS Applications > Sybase SA Manuals. Open the Database Administration manual, then find “Backup and Data Recovery” in the “Maintaining Your Database” section.
System and Media Failures
SA has features to protect your data from two categories of computer failure: system failure and media failure.
System Failure: A system failure occurs when a power failure or some other failure causes the computer or operating system to go down while there are partially completed transactions. This could occur when the computer is inappropriately turned off or rebooted, or when another application causes the operating system to crash.
Media Failure: A media failure occurs when the database file, the file system, or the device storing the database file becomes unusable.
Recovery from Failure
When failures occur, the SA recovery mechanism treats transactions properly, as atomic units of work: any incomplete transaction is rolled back and any committed transaction is preserved. This ensures that even in the event of failure, the data in your database remains in a consistent state.
Make Regular Backups
You should make regular backups of your database files so that you can recover your database in the case of a media failure. SA uses the transaction log (which you should store on a separate device from the database for maximum data safety, or continuously backup using the transaction log mirroring feature of SA) to recover information put into the database since the last full backup. For information on management of the transaction log file, please see:
http://www.sosoft.com/fod/doc126-tranlog.pdf
The SA Logs
SA uses three logs to protect your data from system and media failure. These log files exist for each database running on a database engine or server.
All these logs play a role in data recovery. Each log exists for each database running on a database engine or server. Optionally, you can maintain a mirror of the transaction log for greater protection of vital data.
The Checkpoint Log
An SA database file is composed of pages. Before a page is updated (made dirty), a copy of the original is always made. The copied pages are the checkpoint log.
Dirty pages are not written immediately to the disk. For improved performance, they are cached in memory and written to disk when the cache is full or the server has no pending requests. A checkpoint is a point at which all dirty pages are written to disk. Once all dirty pages are written to disk, the checkpoint log is deleted.
Reasons for a Checkpoint
A checkpoint can occur for several reasons:
· The database engine is shut down
· The amount of time since the last checkpoint exceeds the database option CHECKPOINT_TIME
· The estimated time to do a recovery operation exceeds the database option RECOVERY_TIME
· The database engine is idle long enough to write all dirty pages
· A transaction issues a CHECKPOINT statement
· The database engine is running without a transaction log and a transaction is committed
The priority of writing dirty pages to the disk increases as the time and the amount of work since the last checkpoint grows. This is important when the database engine does not have enough idle time to write dirty pages. The database option CHECKPOINT_TIME controls the maximum desired time between checkpoints. The database option RECOVERY_TIME controls the maximum desired time for recovery in the event of system failure. Both times are specified in minutes.
When the database server is running with multiple databases, the CHECKPOINT_TIME and RECOVERY_TIME specified by the first database started is used, unless overridden by command line switches.
For a description of the command-line options, see “The database engine” in the on-line SA manual.
The Rollback Log
As changes are made to the contents of tables, a rollback log is kept for the purpose of canceling changes. It is used to process the ROLLBACK statement for recovering from system failure. There is a separate rollback log for each transaction. When a transaction is complete, its rollback log is deleted. (Note that the term transaction in this context refers to a related set of database operations, not a transaction in the sense of a single charge or credit entry in OM.)
The Transaction Log
For a condensed and SOS-specific discussion of transaction log management, see http://www.sosoft.com/fod/doc126-tranlog.pdf on the SOS web site.
All changes to the database are stored in the transaction log in the order in which they occur. If you should suffer a data loss, you could restore your most recent backup, then use the transaction log to restore all your changes since the last backup was done. In most cases, therefore, you can restore everything up to the time of your failure or loss, not just up to the time of your backup! The only price you pay for this additional security is the disk space taken by the transaction log. In the sections below you will learn how to move the log to another disk and how to clear it from time to time (after you have made and verified a full backup).
Although use of the transaction log is optional, by default it is enabled. If you run SA with no transaction log, a checkpoint is carried out whenever a transaction is committed. The checkpoint ensures that all committed transactions are written to the disk. Checkpoints can be time consuming, so you should run with a transaction log for improved performance as well as protection against media failure and corrupted databases.
For greater protection, SA allows you to maintain two identical transaction logs in tandem. This is called transaction log mirroring.
The transaction log is not kept in the main database file. The filename of the transaction log is set when the database is installed. For additional security against media failure, the transaction log may be written to a different device than the database file. Some machines with two or more hard drive letters only have one physical disk drive with several logical drives or partitions. If you want protection against media failure, make sure that you have a machine with two separate disk drives or use a storage device on another computer on the network. Note that by default, the transaction log is put on the same device and in the same directory as the database (\SOS\DATA). This default installation, although convenient, does not protect against media failure.
Placing the transaction log on a separate device can also result in improved performance by eliminating the need for disk head movement between the transaction log and the main database file.
Moving the Transaction Log
Use the SA DBLOG utility, located in the \SOS\SA\BIN32 folder, to move the transaction log to a different drive by setting a new transaction log name. Simply copying the file from one drive to another will not work because the location of the log is stored in the database.
Make sure that the database is not running.
From the \SOS\SA\BIN32 prompt, type, in lower case:
dblog -t <new log name> <database name>
The new log name should include the drive, path, and filename of the new log file, for example, d:\soslog\sostran.log .
The database name should include the drive, path, and filename of the main database file, for example, c:\sos\data\sosdata.db.
Once you have reset the location in the database using this procedure, physically move or copy the current transaction log to the new location and, optionally, name you have specified. You will then be able to restart the database engine normally. Remember to include the transaction log, in its new location in your daily backup!
Using a Transaction Log Mirror
A transaction log mirror is an identical copy of the transaction log, maintained at the same time as the transaction log. Every time a database change is written to the transaction log, it is also written to the transaction log mirror file. By default, SA does not use a mirrored transaction log, but you can readily add a transaction log mirror to your system. The procedure is described in the next section.
A mirrored transaction log is useful for extra protection of critical data. There is a performance penalty for using a mirrored log, as each database log write operation must be carried out twice. The performance penalty will depend on the nature and volume of database traffic and on the physical configuration of the database and logs, but in most cases will not be apparent to users of the system.
A transaction log mirror should be kept on a separate physical drive from the transaction log, so that if either drive fails, the other copy of the log keeps the data safe for recovery. Both drives should be on the same computer, however! Maximum protection of your data, therefore, would require three drives (physical drives, not just partitions on the same drive). The database files (the *.DB files) would be on one drive, the transaction log (sosdata.log) on a second, and the mirrored log (sosdata.mlg) on a third.
Creating a Transaction Log Mirror
Use the SA DBLOG utility, located in the \SOS\SA\BIN32 folder, to create a transaction log mirror file on another hard drive.
Make sure that the database is not running.
From the \SOS\SA\BIN32 prompt, type, in lower case:
dblog -m <new mirror log name> <database name>
The new mirror log name should include the drive, path, and filename of the new mirror log file, for example, d:\soslog\sosdata.mlg (mlg is the default mirror extension)
The database name should include the drive, path, and filename of the main database file, for example, c:\sos\data\sosdata.db
Removing a Transaction Log Mirror
If you have been using a log mirror and no longer want to do so, use DBLOG to discontinue mirroring:
Make sure that the database is not running.
From the \SOS\SA\BIN32 prompt, type, in lower case:
dblog -r <database name>
The database name should include the drive, path, and filename of the main database file, for example, c:\sos\data\sosdata.db
Erasing Transaction Log and Mirror Files
You can erase transaction logs and mirrors using the DBERASE command-line utility. Note that this command physically deletes the file from the disk, but does not terminate the use of either. As soon as you restart the database, the files will be recreated.
To delete a mirror log file only:
Make sure that the database is not running.
From the \SOS\SA\BIN32 prompt, type, in lower case:
dberase <mirror file name>
The mirror file name should include the drive, path, and filename of the mirror file that you want to erase, for example, e:\sos\data\sosdata.mlg
To delete a transaction log file but not its mirror:
Make sure that the database is not running.
From the \SOS\SA\BIN32 prompt, type, in lower case:
dberase <transaction log file name>
The transaction log file name should include the drive, path, and filename of the transaction log file that you want to erase, for example, d:\sos\data\sosdata.log
Validating the Transaction Log and Mirror
When a database that is using a mirror starts up, the database engine or server carries out a series of checks and automatic recovery operations to confirm that the transaction log and its mirror are not corrupted, and to correct some problems if corruption is detected.
On startup, the database engine checks that the transaction log and its mirror are identical by carrying out a full comparison of the two files; if they are identical, the database starts as usual.
The comparison of log and mirror adds to database startup time when you are maintaining a log mirror.
If the database stopped because of a system failure, it is possible that some operations were written into the transaction log but not into the mirror. If the database engine finds that the transaction log and the mirror are identical up to the end of the shorter of the two files, then the remainder of the longer file is copied over into the shorter file to produce identical log and mirror. After this automatic recovery step, the database engine starts as usual.
If the check finds that the log and the mirror are different in the body of the shorter of the two, one of the two files is corrupt. In this case, the database does not start, and an error message is generated saying that the transaction log or its mirror is invalid. If this situation should occur, consult “Recovering from media failure on a mirrored transaction log” in the on-line SA Database Administration Guide.
Validating the Database
From time to time prior to running a full backup, it is a good idea to verify that the database file is not corrupt. A small portion of the database could be corrupt without you ever knowing until it was too late to recover the data.
With the database engine (single user) or server (network) running, execute the Check Database tool in the database utility that comes with your SOS software. Enter a Security Administrator id and password, then click the Admin icon. Once at the administration console window, select Database Tools > DBA Utilities > Tools > Check Database.
If desired, you can run the same procedure from the command line or from within a batch file, using the following command, from the prompt in the SOS directory:
DBTOOLS /V
You may want to include this command in a desktop shortcut. It is an excellent idea to run this routine before or after your daily backup to be sure that you are backing up an intact database.
The validation procedure can take a little while to complete on a large database. It scans every record in every table and looks up each record in each index on the table. It can be run while the database is in use, but you may notice some performance impact while it is running.
Any messages produced by the utility are collected in a file on disk (\SOS\SOSVALID.LOG) and displayed when the procedure is complete. If the database file is corrupt, you must restore the DB files (but not the LOG file) from your previous intact backup. You may then be able to apply the current transaction log in order to bring the database up to date (see “Lost Database Files” on page 28).
Backing up Your Database
On-line Backups
In some organizations, operations continue around the clock and it is not possible to stop the database to perform a traditional backup. Under these circumstances, you can use the provided Copy Database utility to make a copy of the database, while it is running, in a different directory. Once the copy is complete, the duplicated files can be included in your regular backup even though the open production database files will be skipped by your backup software. (To avoid errors in your backup logs, configure your backup to skip over the production database files or folder.)
Run Copy Database interactively from the menus in the DBA Utilities program, which is located in the Admin Module. The Admin icon appears on the login window for users who are designated as Security Administrators. On the Admin menu, select Database Tools, then DBA Utilities.
Once the program has started, go to the Tools menu. Select Copy Database to make a live copy of the running database into a different directory. Note that for security reasons copying the database is only possible on the computer running the database software.
The Copy Database utility can be run against a standalone engine or network database server. The utility will not run from a network workstation for security reasons. Using the Copy Database utility on a running database is equivalent to copying the database files when the database is not running. In other words, it provides a snapshot of a consistent database, even while it is being modified by other users.
For a full description of the SA online backup facility, see “Use the Backup utility (dbbackup) to make a server-side backup” in the online SA Database Administration Guide, page 882 (pdf page 900). It is important to understand, however, that you will not be able to use the utility directly against the secure SOSDATA database. Rather, you must run it from within the DBA Utilities shell program provided by SOS as described above or using the DBTools command line explained below.
You also can copy the database from the command line, batch file, or desktop shortcut by executing DBTOOLS.EXE B=<target directory> from the SOS directory (but only on a standalone system or directly on the database server computer). For example, assuming that the SOS folder is located on the C: drive:
C:\SOS\DBTOOLS.EXE B=C:\SOS\BACKUP
would create a copy of the currently running database in the C:\SOS\BACKUP folder. Note that the folder must exist before you execute the command. SOS recommends that the destination for the copy operation be a different drive on the database computer than that on which the production database is located. That way, if your primary drive should fail, the most recent copy of the database on the secondary drive will be readily available to use in the restoration process.
Open File Backups Using Volume Shadow Copy Service
[SOS 2010 and later] SOS recommends the Copy Database procedure detailed above for best reliability. In addition, there is a side benefit that it leaves a backup copy of your database and log on a local hard disk as a redundant backup to the one you make on removable media, internet-based backup service, or both.
[SOS 2010 and later] That said, many commercial backup programs include the ability to backup files that are in use through the Volume Shadow Copy Service available on versions of Windows starting with XP. For best results, you should create and enable the SA VSS Writer service by executing the following command from the command prompt in the \SOS\SA\BIN32 directory:
dbsvc -as -s Automatic -t vss -w SAVSSWriter “c:\sos\sa\bin32\dbvss11.exe”
It is absolutely essential that you test your backup during a period when the database is actively in use by restoring your backup, restarting your system, and validating the restored database as described in “Validating the Database”, above .
Incremental Backups of the Transaction Log
You can carry out an off-line incremental backup by making a copy of the transaction log. Alternatively, you can carry out an online incremental backup by running the SOS Copy Database utility, and subsequently backing up the SOSDATA.LOG file in the normal fashion to tape or other removable media.
You should back up the transaction log daily. This is particularly important if you have the transaction log on the same device as the database file. If you have a media failure, you could lose both files. By doing daily backups of the transaction log, you will never lose more than one day of changes.
Daily backups of the transaction log are also recommended when the transaction log tends to grow to an unmanageable size between full backups and you do not want to get a larger storage device or do more frequent full backups. In this case, you can choose to archive and delete the transaction log by selecting the appropriate options in the SOS Copy Database utility.
There is a drawback to deleting the transaction log after a daily backup. If you have media failure on the database file, there will be several transaction logs since the last full backup. Each of the transaction logs needs to be applied in sequence to bring the database up to date, as described below. It is therefore extremely important to save all transactions logs that cover database activity after the most recent verified backup of the database files themselves!
SOS strongly recommends that Transaction logs be archived onto CD-R or other permanent media that will not be overwritten and that these media be stored in a safe location, such as a fireproof data safe, or even better, off-site in a bank vault.
Backing Up, Renaming and Restarting the Transaction Log
Use of the following command will do a live backup of the transaction log into the target directory, but will also rename the backup with the form YYMMDD##.LOG (where ## is a sequential number starting with 00, in case you execute the command more than once on a single date) and restart the log. The result will be a copy of the current SOSDATA.LOG in the backup directory, but renamed based on the date, and a brand new, small SOSDATA.LOG file in the \SOS\DATA folder.
C:\SOS\DBTOOLS B=C:\BACKUPS -t -r -n
You can specify any existing directory on the server instead of “C:\BACKUPS”, but be sure to type the -t -r and -n in lower case.
If you prefer to work within a graphical interface, you can do this process from Database Tools > DBAUtils in the Administration module. Select Copy Database on the Tools menu, select the desired backup directory and check the -t, -r, and -n options as shown below.
Use the Copy Database utility to backup and restart your transaction log file.
To create the renamed backup copy in the current data directory, just type a period (.) for the Backup directory. The dated copy of the old log will be created in the same directory as the database files and new log file.
Data Recovery
System Failure
After a power failure or other system failure you should run the Windows’ scandisk utility to correct simple errors in the file system structure that might have been caused by the system failure. This step should be done before running any other software.
After a system error occurs, SA automatically recovers when you restart the database. The results of each transaction committed (written to the database) before the system error are intact. All changes by transactions that were not committed before the system failure are canceled, and therefore lost. In most cases that means you will only lose those changes that had not been completely saved prior to the failure.
The database engine automatically takes three steps to recover from a system failure:
1. Restore all pages to the most recent checkpoint, using the checkpoint log.
2. Apply any changes made between the checkpoint and the system failure. These changes are in the transaction log.
3. Rollback all uncommitted transactions, using the rollback logs. There is a separate rollback log for every connection.
Step 3 may take a long time if there are long uncommitted transactions that have already done a great deal of work before the last checkpoint.
Media Failure
Perhaps the most frequent cause of data loss is the failure of a hard drive or failure of drive controller electronics, leading to corruption of the data on the drive.
If you have backups, you can always recover all transactions that were committed before the media failure. Recovery from media failure requires you to keep the transaction log on a separate device from the database file. The information in the two files is redundant. Regular backups of the database file and the transaction log reduce the time required to recover from media failures.
The first step in recovering from a media failure is to clean up, reformat, or replace the device that failed.
The steps to take in recovery depend on whether the media failure is on the device holding your database file or on the device holding your transaction log. These discussions assume that your system stores the transaction log (or mirror) on a separate drive from the database files.
Lost Database Files
When your transaction log is still usable, but you have lost your database file, the recovery process depends on whether you keep or delete the transaction log on incremental backup.
If you have not deleted or restarted the transaction log since the last full backup, the transaction log contains everything since the last backup. Recovery involves four steps:
1. If you do not already have one, make a backup of the current transaction log immediately. The database file is gone and the only record of the changes is in the transaction log.
2. Restore the most recent full backup (the database file).
3. Change to the \SOS\SA\BIN32 directory.
4. Use the database engine with the apply transaction log (-a) switch to apply the transaction log and bring the database up to date:
dbeng11 c:\sos\data\sosdata.db -a d:\soslogs\sosdata.log
Start the database in the normal way. The database engine will come up normally and any new activity will be appended to the current transaction log.
If you have archived and deleted the transaction log since the last full backup, each transaction log since the full backup needs to be applied in sequence to bring the database up to date.
Make a backup of all transaction logs immediately. The database file is gone and the only record of the changes is in the transaction logs.
Restore the most recent full backup (the database file).
Change to the \SOS\SA\BIN32 directory.
Starting with the first transaction log after the full backup, apply each archived transaction log by starting the database engine with the apply transaction log (-a) switch. For example, if the last full backup was on Sunday and the database file is lost during the day on Thursday:
dbeng11 c:\sos\data\sosdata.db -a d:\soslogs\mon.log
dbeng11 c:\sos\data\sosdata.db -a d:\soslogs\tue.log
dbeng11 c:\sos\data\sosdata.db -a d:\soslogs\wed.log
dbeng11 c:\sos\data\sosdata.db -a d:\soslogs\sosdata.log
SA does not allow you to apply the transaction logs in the wrong order or to skip a transaction log in the sequence.
An easier approach is to put all the log files in the same directory, then use this command:
dbeng11 c:\sos\data\sosdata.db -ad d:\soslogs
where “d:\soslogs” is the directory in which you have placed all the relevant log files. The database will inspect the files and automatically apply them in the correct sequence.
After applying the transaction logs, start the database in the normal way. The database engine will come up normally and any new activity will be appended to the current transaction log.
Lost Transaction File
When your database file is still usable but you have lost your transaction log, the recovery process is as follows:
1. Make a backup of the database file immediately. The transaction log is gone and the only record of the changes is in the database file.
2. Restart the database with the -f switch:
dbeng11 c:\sos\data\sosdata.db -f
Without the -f switch, the database engine will complain about the lack of a transaction log. With the switch, the database engine will restore the database to the most recent checkpoint and then roll back any transactions that were not committed at the time of the checkpoint. A new transaction log will be created and the database will do a clean shutdown, after which you should be able to restart it normally.
Media failure on the transaction log can have more serious consequences than media failure on the database file. When you lose the transaction log, all changes since the last checkpoint are lost. By default, database checkpoints are set at 30 minute intervals. This will be a problem when you have a system failure and a media failure at the same time (such as if a power failure causes a head crash that damages the disk). Frequent checkpoints minimize the potential for lost data but also create work for the database engine writing out dirty pages.
For running high-volume or extremely critical applications, you can protect against media failure on the transaction log by mirroring the transaction log or by using a special-purpose device, such as a storage device that mirrors the transaction log automatically.
