How does Delphix stay synchronized with an ASE database?
Delphix monitors ASE for new backups or transaction logs once per minute. There are two primary ways for Delphix to retrieve ASE's backup history:
- Monitoring the source backup server log file of the ASE instance.
- Monitoring the source dump history file of the ASE instance.
Backup Server Log
The original and older method is for Delphix to parse the ASE backup server log file. When you link a dSource using this option, Delphix must parse and catalog every single backup it finds in the ASE backup server log file.
Delphix cannot rely on the file name of the backup to positively tie it to a particular source database (for example, the source database might be named "abc" but the backup file could be named "xyz_monday.dump"). Since it cannot rely on the file name, it attempts to find and catalog every single file listed in the backup server log to find out which backup was the most recent for the desired source database. This means:
- Trying to see if each file exists by searching for it 120 times in 1 minute intervals (2 hours)
- If it finds the file, it runs a "LOAD DATABASE ... WITH HEADERONLY" to find out which database it belongs to, what is the internal dump file name, creation date, how many stripes, etc.
- Delphix saves or catalogs this information in its internal database.
Searching for all of the files listed in the backup server log file and cataloging them can take hours and is very CPU intensive.
ASE Dump History
The newer method is for Delphix to read and parse the ASE dump history file. Prior to ASE version 15.7 ESD#2, ASE did not provide any structured information about backup/restore operations performed on databases (the only reference was the backup server log file). Delphix added support for parsing the ASE "dump history file" in Delphix 22.214.171.124 ( https://docs.delphix.com/docs534/release-information/release-notes-5-3-x/new-features-5-3-x ):
Support for dump history file: SAP ASE support has been extended to leverage ASEs Dump History File as an alternative ingestion mechanism to Error Log Files. Dump History File-based ingestion offers multiple efficiencies (pinpointed data, chronological and structured entries, striping history, etc.) that offer a more efficient LogSync capability. For more information please refer to Support for Dump History File.
The ASE "enable dump history" feature speeds up and simplifies the search for the most recent existing backup file. The ASE dump history file tells Delphix exactly which file belongs to which database, when it was created, how many stripes it has, and if it was compressed. Here is a snippet of the information available in this file:
2|8|bigdb|1|Dec 11 2019 10:34:30:960AM|Dec 11 2019 10:34:30:960AM|Dec 11 2019 10:34:30:960AM|/home/sybase/dump/bigdb.full.12_11_11_32|1567743|*|0|5|1| |Dec 11 2019 10:34:21:066AM|Dec 11 2019 10:34:31:080AM|31640 3|8|bigdb|1|Dec 11 2019 10:34:30:960AM|Dec 11 2019 11:49:18:983AM|Dec 11 2019 11:49:18:983AM|/home/sybase/dump/bigdb.tran.12_11_12_49|0|*|0|5|1| |Dec 11 2019 11:49:19:396AM|Dec 11 2019 11:49:19:430AM|2 3|8|bigdb|1|Dec 11 2019 11:49:18:983AM|Dec 11 2019 11:55:09:950AM|Dec 11 2019 11:55:09:950AM|/home/sybase/dump/bigdb.tran.12_11_12_55|0|*|0|5|1| |Dec 11 2019 11:55:10:760AM|Dec 11 2019 11:55:10:783AM|2 3|8|bigdb|1|Dec 11 2019 11:55:09:950AM|Dec 11 2019 12:00:01:053PM|Dec 11 2019 12:00:01:053PM|/home/sybase/dump/bigdb.tran.12_11_12_59|0|*|0|5|1| |Dec 11 2019 12:00:01:390PM|Dec 11 2019 12:00:01:426PM|2 3|8|bigdb|1|Dec 11 2019 12:00:01:053PM|Dec 11 2019 12:57:26:953PM|Dec 11 2019 12:57:26:953PM|/home/sybase/dump/bigdb.tran.12_11_13_57|0|*|0|5|1| |Dec 11 2019 12:57:27:453PM|Dec 11 2019 12:57:27:476PM|2 3|8|bigdb|1|Dec 11 2019 12:57:26:953PM|Dec 11 2019 1:07:13:056PM|Dec 11 2019 1:07:13:056PM|/home/sybase/dump/bigdb.tran.12_11_14_07|0|*|0|5|1| |Dec 11 2019 1:07:13:306PM|Dec 11 2019 1:07:13:336PM|4 3|8|bigdb|1|Dec 11 2019 1:07:13:056PM|Dec 12 2019 7:30:42:150AM|Dec 12 2019 6:46:53:676AM|/home/sybase/dump/bigdb.tran.12_12_08_30|0|*|0|5|1| |Dec 12 2019 7:30:42:753AM|Dec 12 2019 7:30:42:846AM|2 2|8|bigdb|1|Dec 20 2019 1:03:02:323PM|Dec 20 2019 1:03:02:323PM|Dec 20 2019 1:03:02:323PM|/home/sybase/dump/bigdb.full.12_20|1567743|*|0|5|1| |Dec 20 2019 1:02:51:190PM|Dec 20 2019 1:03:03:240PM|125124 3|8|bigdb|1|Dec 20 2019 1:03:02:323PM|Dec 23 2019 8:59:51:843AM|Dec 22 2019 8:31:44:923PM|/home/sybase/dump/bigdb.tran.12_23_10_00|0|*|0|5|1| |Dec 23 2019 8:59:52:110AM|Dec 23 2019 8:59:53:623AM|22652 3|8|bigdb|1|Dec 23 2019 8:59:51:843AM|Dec 23 2019 9:03:38:863AM|Dec 23 2019 9:00:49:850AM|/home/sybase/dump/bigdb.tran.12_23_10_04|0|*|0|5|1| |Dec 23 2019 9:03:39:230AM|Dec 23 2019 9:03:39:250AM|4 3|8|bigdb|1|Dec 23 2019 9:03:38:863AM|Dec 23 2019 9:42:38:063AM|Dec 23 2019 9:42:38:063AM|/home/sybase/dump/bigdb.tran.12_23_24|0|*|0|5|1| |Dec 23 2019 9:42:38:263AM|Dec 23 2019 9:42:38:290AM|2 3|8|bigdb|1|Dec 23 2019 9:42:38:063AM|Dec 23 2019 10:14:16:243AM|Dec 23 2019 10:14:16:243AM|compress::/home/sybase/dump/bigdb.tran.10_23_11_14|0|*|0|0|1| |Dec 23 2019 10:14:16:740AM|Dec 23 2019 10:14:16:763AM|2 3|8|bigdb|1|Dec 23 2019 10:14:16:243AM|Dec 23 2019 10:19:48:370AM|Dec 23 2019 10:19:48:370AM|compress::/home/sybase/dump/bigdb.tran.10_23_11_20|0|*|0|0|1| |Dec 23 2019 10:19:49:136AM|Dec 23 2019 10:19:49:160AM|4
When does the backup history come into play?
When you link a dSource or click its camera icon, you have three options to choose from:
- New Full Backup - Delphix creates a new full backup file (on Delphix storage that is mounted via NFS on the ASE staging host) and loads it. When choosing this option, Delphix does not read either the dump history file or the backup server log. It simply issues the "DUMP DATABASE" command to create a new backup and then loads this new backup into the staging database.
- Most Recent Existing Full Backup - Finds the files of the most recent existing full database dump in the "Backup Location" of the dSource and loads it.
- If the dSource is configured to use the log file of the backup server, choosing this option can delay completion of the dSource link as Delphix attempts to find and catalog every single backup listed in the backup server log file of the source database.
- If the dSource is configured to use the ASE dump history file, minimal delay will be experienced as it is much easier for Delphix to determine which backup belongs to the source database and which is the most recent.
- Specific Existing Full Backup - Specify which backup files in the Backup Location you want to load.
Whether the dSource is configured to use the most recent backup or the dump history file, Delphix does not read either file. Choosing this option is much faster because Delphix will skip directly to loading the desired backup. Delphix will then start to search for and catalog backups in the background after the linking of the source database has completed.
Which option is better, the ASE dump history file or the backup server log?
The ASE "dump history file" feature is the preferred option:
- It should be faster especially when choosing the "Most Recent Existing Full Backup" synchronization option.
- Much less CPU should be used because it does not have to search for and catalog all of the backups listed in the ASE backup server log. Rather, since the file tells Delphix which backups belong to which database, it only has to worry about the backup belonging to the specific source database.
- It is easier for Delphix to parse and understand and as such should be less prone to error.
Which of the three options ("Most Recent Existing Full Backup", "New Full Backup" or "Specific Existing Full Backup") should be used?
- When linking a dSource and choosing to use the ASE backup server log, avoid choosing the "Most Recent Existing Full Backup" option. Delphix attempts to find and catalog every single file listed in the backup server log to find out which backup was the most recent for the desired source database before loading the backup. This process can take hours.
- Linking a dSource configured to use the "dump history file" you can choose the "Most Recent Existing Full Backup" option as Delphix can quickly determine which backup was the most recent for a given source database.
The ASE "dump history file" feature does not provide any initial benefit when choosing the "New Full Backup" or "Specific Existing Full Backup" options because neither of these options initially parse the files. If Delphix is asked to use the backup server log, it will still need to parse the log, find its files, and catalog them. This can be very CPU intensive depending on the size of the backup server log.
If you have to use the ASE backup server log (because you are on a version of ASE that does not have the feature or it is not enabled in ASE), there are some ways to make this option less painful:
- Truncate the backup server log and take a new database dump before linking a new dSource. If you then choose the "Most Recent Existing Full Backup" option, it should run much quicker.
- The backup server log should be regularly maintained so it does not get too big.
- By choosing the "Specific Existing Full Backup" option, Delphix temporarily skips parsing and cataloging files from the backup server log and goes straight to trying to load the backup files specified in the dialog box. Once the dSource completes loading the backup, Delphix starts to parse and catalog all of the files in the background. Once Delphix has finished catalog all of the backups listed in the ASE backup server log file, it will continue to monitor the ASE backup server log file for new backups and catalog them as it sees them. However, since it keeps track of what line it last read, finding and cataloging new dumps does not take long.
The following articles may provide more information or related information to this article: