Skip to main content
Delphix

Special Upgrade Steps for ASE Required When Upgrading to Delphix 5.1.3.1 or later

Pre-Upgrade Steps

Prior to upgrading, Delphix quiesces all dSources and virtual databases (VDBs) by disabling them. This is done in parallel across all the sources. This process can fail on Delphix Engines running on 5.0.3.1 that have a large number of dSources and VDBs.

Troubleshooting

You would see an error during the quiesce step of the upgrade with a failure to disable a subset of the sources.

Resolution

The upgrade, when run through the GUI, re-enables all the dSources and VDBs if any of them failed to disabled as part of the pre-upgrade quiesce. There are several options:

  • Use {dx_toolkit to disable all dSources and VDBs prior to starting the upgrade. Make sure to run at most 10 disables in parallel.
    After the upgrade, use dx_toolkit to re-enable all dSources and VDBs, as they will not be re-enabled automatically. In this case, they can all be run in parallel.
  • Run the upgrade using the Delphix command line interface (CLI) and set "enableSourcesOnFailure = false" as part of the ApplyVersionParameters. This might fail a few times, but the dSources and VDBs will not be re-enabled on failure. Repeat the process until all the sources are quiesced, and the upgrade proceeds. After upgrade, all the sources will be re-enabled automatically.
  • Use the CLI to disable all of the dSources and VDBs prior to upgrading. Make sure to run at most 10 disables in parallel. Use the CLI to enable all of the dSources and VDBs after upgrading.

Post-Upgrade Steps: ASE Device Layout Changes

Prior to Delphix 5.1.3.0, Delphix created the database devices by mixing data and transaction logs on the same device and by oversizing the database devices to ensure that there was enough space for the LOAD DATABASE command to succeed. This resulted in some issues:

  • By oversizing the database devices, ASE would spend extra time clearing unused pages (zeroing them out) while executing the LOAD DATABASE command.
  • Delphix utilizes the MOUNT DATABASE command to mount staging databases and VDBs. Mixing data and transaction logs on the same database device causes ASE to do a lot of extra processing as part of the mount operation. This increases the time it takes to provision VDBs or ingest database dumps into the staging databases (since the staging database is mounted/unmounted each time a new dump is ingested).
  • Doubling the size of the VDB means the virtual to physical (V2P) feature can require up to double the space compared to the original database. While oversizing the database devices doesn’t affect the total size required for VDBs since they are on Delphix storage, this is not true if you try to do a V2P operation because the database devices are required to be as large as the VDBs as well.

Delphix 5.1.3.0 introduces a new feature whereby it attempts to mirror the source databases device layout as close as possible to avoid these issues. Special steps need to be followed in order to make existing dSources and VDBs take advantage of this feature.

NOTE: Since Delphix is attempting to mirror the source databases, if the source databases are already set up with unused pages, or have both data and logs mixed on the same devices, the performance related issues mentioned above would still be a problem. The V2P issue should no longer be an issue.

Troubleshooting

You can issue the sp_helpdb command and look for "mixed log and data" in the output to confirm whether or not the staging databases or VDBs utilize mixed data and log devices:

sp_helpdb example
1> sp_helpdb
2> go | grep mix
dxuDf674vi8snfflut_haigh 2248.0 MB neal 11 Oct 14, 2016 full 0 NULL don't recover, abort tran on log full, offline, mixed log and data
Vhaight_803 2248.0 MB neal 11 Oct 14, 2016 full 0 0 abort tran on log full, mixed log and data
master 26.0 MB sa 1 Sep 08, 2014 full 0 NULL mixed log and data
model 6.0 MB sa 3 Sep 08, 2014 full 0 NULL abort tran on log full, mixed log and data
pubs2 7.0 MB sa 8 Apr 12, 2016 full 0 NULL trunc log on chkpt, mixed log and data
sybsystemdb 12.0 MB sa 31513 Sep 08, 2014 full 0 NULL mixed log and data
sybsystemprocs 172.0 MB sa 31514 Sep 08, 2014 full 0 NULL trunc log on chkpt, mixed log and data
tempdb 106.0 MB sa 2 Oct 13, 2016 no_recovery 0 NULL select into/bulkcopy/pllsort, trunc log on chkpt, mixed log and data, allow wide dol rows

NOTE: The staging databases are typically named starting with "dx". Prior to Delphix 5.1.3.0, the staging databases are normally present only while actively ingesting a database dump. Otherwise, they get unmounted and will not be displayed. From Delphix 5.1.3.0 and later, the staging databases remain mounted but offline.

Resolution

dSources:

  • Delphix 5.1.3.0 has a new option that allows you to stop Delphix from ingesting database dumps and transaction logs by toggling a switch named "Validated Sync Mode" on the dSource configuration screen. You will need to temporarily prevent the dSource from ingesting new database dumps or transaction logs to be able to initiate a manual sync operation so that staging database gets recreated with the new device layout.

     

    1. Temporarily prevent Delphix from ingesting new database dumps or transaction logs:

      • Click the dSource on the left hand side of the screen.

      • Click on Configuration.

      • Toggle the “Validated Sync Mode” switch to disabled.

    2. Perform a manual snapshot (click the camera icon and force Delphix to ingest a "Specific Existing Full Backups" or to create a "New Full Backup"). You cannot use the "Most Recent Existing Full Backup" option because when "Validated Sync Mode" is disabled, Delphix does not catalogue the database dumps and won't be aware that a new dump may have been created. Make sure that there is a dump that is more recent than the dumps that this dSource has previously ingested, including log dumps. This will cause the device layout of the dSource's staging database to be recreated to mirror source database's device layout. This can also be performed or scripted from the command line interface:

      Manual Sync of a dSource
      $ ssh delphix_admin@delphixserver.acme.com
      Password:
      delphixserver.acme> database
      delphixserver.acme database> select db1
      delphixserver.acme database 'db1'> sync
      delphixserver.acme database 'db1' sync *> set type=ASELatestBackupSyncParameters
      delphixserver.acme database 'db1' sync *> commit
      Dispatched job JOB-3561
      DB_SYNC job started for "ASE/db1".
      Starting sync from the source database "db1".
      
    3. Re-enable ingestion of Database dumps by toggling the “Validated Sync Mode” switch back to enabled.

      NOTE: If there are a large number of dSources, consider taking a measured approach to not overwhelm the ASE instance hosting the staging server by slowly taking manual snapshots of a few dSources at a time, deleting previous snapshots or waiting until retention deletes them, before continuing as to not overload total capacity of Delphix.

       

  • A second option would be to delete and recreate dSources if there are no dependent VDBs. By using this option, disk space will not be doubled in size because Delphix will no longer need to keep track of the old device files. If Delphix replication is in use, make sure to also delete any VDBs corresponding to the old dSource and re-create them to avoid using extra space on the the replicated Delphix Engine. 

VDBs:

  • VDBs will need to be refreshed to a snapshot of the dSource (Refreshing an SAP ASE VDB) with the new device layout to get the new device layout as well.

V2P:

  • V2P from a snapshot with the new device layout will require as much space as the source database. However any V2P from an older snapshot might still require up to double the amount of storage.

Reduce Storage Consumption on the Delphix Engine:

  • After taking the manual snapshot for a dSource, Delphix will need to keep track of the old device files as well as the new. Approximately two times the disk space will be used until the older snapshots are deleted either by retention or manually.
  • For VDBs, refresh the VDB to the newer snapshot with the new device layout. Make sure to delete the older VDB snapshots either manually or via the retention policy.
  • For dSources, once no more VDBs are still being provisioned from a snapshot taken before the manual sync, delete old snapshots.
  • If Delphix replication is in use, after the above steps, make sure to do an incremental replication update, and refresh any VDBs on the replica that might still be using the old snapshots to avoid using extra space on the replicated Delphix Engine. Make sure to also delete any older snapshots as well either manually or via the retention policy.