Delphix Operations May Fail Due to ASE Error 3404 (KBA1750)
Applicable Delphix Versions
All Versions
Issue
Several different operations initiated by Delphix can fail due to ASE error 3404. ASE error 3404 can be raised when Delphix issues the ASE "MOUNT DATABASE" or "UNMOUNT DATABASE" commands causing them to fail. Typically this has been seen during ValidatedSync (when a Delphix dSource is ingesting a database dump) or when provisioning a VDB. Once a database is affected by this error other operations such as disabling or enabling may fail because they also use the MOUNT/UNMOUNT commands.
A dSource could be affected by this error while running a manual SnapSync or during the automated ValidatedSync and will present with an error similar to the following:
Failed to mount database "dxmvf14dfu8s33c5z7_accounts" on target host "blackdog.delphix.com" instance "ASE157SP120": Could not find row in sysdatabases with database id 11.
While provisioning a VDB, an error similar to the following may be presented in the GUI:
DB_PROVISION job for "ASE157SP120/Vpubs2" failed due to an error during execution: Failed to mount database "Vpubs2" on target host "blackdog.delphix.com" instance "ASE157SP120": Could not find row in sysdatabases with database id 26.
Troubleshooting
To confirm this issue, look in the Delphix debug logs for ASE error 913 followed by 3404:
[2017-10-07 23:55:47,005][INFO][ase.dbc.ASEJDBCAccessorImpl#messageHandler:1078][Worker-31513|JOB-25536|DB_SYNC(ASE_DB_CONTAINER-109)][ACTION-53420] ASE message 913 : Could not find row in sysdatabases with database id 16. [2017-10-07 23:55:47,005][INFO][ase.dbc.ASEJDBCAccessorImpl#messageHandler:1078][Worker-31513|JOB-25536|DB_SYNC(ASE_DB_CONTAINER-109)][ACTION-53420] ASE message 3404 : Rec_complete: Could not open controlling database (id 16) of controlling database in multi-db transaction.
Check the ASE error log for error 3404:
00:0000:00000:00041:2017/10/11 11:23:38.26 server Error: 3404, Severity: 21, State: 1 00:0000:00000:00041:2017/10/11 11:23:38.26 server Rec_complete: Could not open controlling database (id 16) of controlling database in multi-db transaction. 00:0000:00000:00041:2017/10/11 11:23:38.26 server dbrecdiag__populate_record: No resources to allocate DBRECDIAG_RECORD for database 27141
Resolution
SAP has confirmed that this is a bug in ASE:
812086
Description: In rare circumstances, a 3404 error, "Could not open controlling database <value> of controlling database in multi-db transaction.", may be reported when MOUNT DATABASE is executed on a database that is in load sequence and a multi-database transaction was involved earlier.
It has been fixed in the following versions of ASE:
- ASE 15.7 SP140
- ASE 16.0 SP02 PL08
- ASE 16.0 SP03 PL04
This particular bug is very problematic for Delphix because it prevents the ASE MOUNT and UNMOUNT DATABASE commands from working. Delphix heavily relies on these commands during most workflows (enable/disable/start/stop/SnapSync/provision/ValidatedSync).
ASE dSources
When this issue happens to a dSource, it can effectively become wedged. Delphix loads a database dump into the staging database and then unmounts it to take a snapshot of the file system. It is then never able to MOUNT the staging database again. Delphix created a fix to detect when a dSource has encountered ASE error 3404 and skip the database dump (or transaction log) containing the multi-database transaction in Delphix 5.2.4.0 and higher. Delphix releases prior to ASE 5.2.4.0 will require that the dSource be recreated when this issue is encountered.
ASE VDBs
If this issue happens while provisioning a VDB, the VDB will need to be force deleted from Delphix (check the "Force Delete" checkbox when deleting the VDB). Then you may need to manually DROP the database from ASE along with its devices ("sp_dropdevice").
Additional Information
Earlier versions of Delphix (prior to 5.2.5.0) require that the staging ASE instance match the source ASE instance down to the SP level. So it may be difficult to upgrade to an ASE release containing the fix because it would require upgrading the production ASE instance.
Delphix 5.2.5.0 and higher relaxes Delphix's version checking of the staging instance. ASE itself may or may not allow the "MOUNT" command to work between versions. Be sure to check that the UNMOUNT and MOUNT commands work between your desired ASE versions.
So for example, if we upgraded the ASE staging instance to ASE 15.7 SP140 but we keep the ASE instance hosting the VDBs at ASE 15.7 SP135, we would want to make sure that we can UNMOUNT a database from ASE 15.7 SP140 and then MOUNT it on ASE 15.7 SP135:
isql -Usa -Psybase -SASE157SP140 1> UNMOUNT DATABASE testdb to "/tmp/manifest" 2> GO isql -Usa -Psybase -S ASE157SP135 1> MOUNT DATABASE testdb from "/tmp/manifest" 2> go Msg 14580, Level 16, State 1: Server 'ASE157SP135', Line 1: You cannot mount the database(s) because at least one database contains functionality that is available only on the server on which it originated.
In the above example, ASE would not allow the database "testdb" to be mounted on the older release of ASE.