Restarting an ASE VDB After Incorrectly Configuring Its Device Onto the Wrong Mount Path (KBA6959)

Issue

The database devices of a VDB need to reside under the VDB mount path. If the device layout of the VDB is altered such that its devices are under the wrong path, Delphix will not be able to take snapshots or control the VDB properly. The following error may be presented when an individual attempts to disable, enable, start, or stop a VDB that is in this state:

Error: Device "bad_device" at path "/home/sybase/toolkit/564dd4a5-249f-7812-f266-65ffdf1278c1-vdb-8/datafile/bad_device" for target database "Vbroken" on environment "nealrh8.acme.com" and instance "ASE16" is not on the Delphix mount path "/home/sybase/toolkit/564dd4a5-249f-7812-f266-65ffdf1278c1-vdb-12/datafile".

Error Code: exception.db.asetarget.target.device.not_on_delphix

Suggested Action: Make sure that the device belongs to the Delphix target, or drop the device and try the operation again.

This issue commonly happens when a DBA needs to add space to a VDB. They may add a database device to local disk storage (not on Delphix storage) or they may inadvertently add the device to the mount path of another VDB (on Delphix storage but not the mount path of the correct VDB).

In the above example, the error message is saying that the database device named "bad_device" was created on a path for a VDB with "vdb-8" in the path and that it is not in the correct path with "vdb-12".

The reason the files must be on Delphix storage and under the correct path for the VDB is so that Delphix can issue the ZFS "snapshot" command. If the files are not on Delphix storage or on the wrong path, the product cannot take a ZFS snapshot of the database files.

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies

Major Release	All Sub Releases
6.0	6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1, 6.0.4.2, 6.0.5.0, 6.0.6.0
5.3	5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0
5.2	5.2.2.0, 5.2.2.1, 5.2.3.0, 5.2.4.0, 5.2.5.0, 5.2.5.1, 5.2.6.0, 5.2.6.1
5.1	5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0, 5.1.8.1, 5.1.9.0, 5.1.10.0
5.0	5.0.1.0, 5.0.1.1, 5.0.2.0, 5.0.2.1, 5.0.2.2, 5.0.2.3, 5.0.3.0, 5.0.3.1, 5.0.4.0, 5.0.4.1 ,5.0.5.0, 5.0.5.1, 5.0.5.2, 5.0.5.3, 5.0.5.4
4.3	4.3.1.0, 4.3.2.0, 4.3.2.1, 4.3.3.0, 4.3.4.0, 4.3.4.1, 4.3.5.0
4.2	4.2.0.0, 4.2.0.3, 4.2.1.0, 4.2.1.1, 4.2.2.0, 4.2.2.1, 4.2.3.0, 4.2.4.0 , 4.2.5.0, 4.2.5.1
4.1	4.1.0.0, 4.1.2.0, 4.1.3.0, 4.1.3.1, 4.1.3.2, 4.1.4.0, 4.1.5.0, 4.1.6.0

Resolution

To resolve this issue, the errant device needs to be deleted (if there is enough space in the database itself to complete the ASE "ALTER DATABASE <database_name> OFF <device_name>") or moved to the correct mount path.

Once the storage has been mounted for the VDB, the VDB needs to be brought online. If at any point in time ASE marked the storage as unavailable with ASE error 840, ASE will need to be restarted 

(SAP CR 824105 was raised to request the ability to bring the database online without restarting ASE once the NFS storage is available). For example, if the server hosting the VDB was rebooted and ASE was restarted before the NFS directories were mounted, the "ONLINE DATABASE" command will fail with ASE error 840:

isql -Usa -Psybase -SASE16 -w 220 
1> ONLINE DATABASE Vbroken
2> go
Msg 840, Level 17, State 2:
Server 'ASE16', Line 1:
Device 'AAAN$76zVxedXAz68ux9_igdb_dev5' (with physical name '/home/sybase/toolkit/564dd4a5-249f-7812-f266-65ffdf1278c1-vdb-12/datafile/dx876zVxedXAz68ux9_igdb_dev5', and virtual device number 13) has not been correctly
activated at startup time.  Please contact a user with System Administrator (SA) role.

1. Restart ASE. Watch the ASE error log for the message indicating that it has brought the database online ("Database 'Vbroken' is now online."). Alternatively, run the "sp_helpdb" command to see that the database status shows as "online".
2. Once the VDB is online, if there is enough free space in the database, the solution may be simple enough to alter the database off of the device. In the following example, we assume that there is not enough free space to alter the database off the device without first creating a new device to hold the data in the "bad" device:

-- Use the DISK INIT with skip_alloc=true clause to speedily create the new database device:
1> disk init name="good_device", physname="/home/sybase/toolkit/564dd4a5-249f-7812-f266-65ffdf1278c1-vdb-12/datafile/good_device", skip_alloc="true", size="10M"
2> go
Warning: The 'skip_alloc' option of disk init/disk resize command is turned on, and does not ensure allocation of the specified number of pages. Please make sure file system has enough space for future operations.

-- Alter the database onto the new device which resides in the correct mount path:
1> ALTER DATABASE Vbroken on good_device="10M"
2> go
Extending database by 2560 pages (10.0 megabytes) on disk good_device

-- Alter the database off of the problematic device:
1> ALTER DATABASE Vbroken off bad_device
2> go
ALTER DATABASE is starting in database 'Vbroken'. This command will process 10 allocation units.
ALTER DATABASE has finished in database 'Vbroken'. This command processed 10 allocation units of 10 requested.

3. After altering the database, confirm that all of the devices are under the correct mount path using the following query:

1> SELECT DISTINCT
2>       db.dbid, db.name, sysdev.vdevno, sysdev.phyname
3> FROM sysdatabases db
4> JOIN sysusages    sysus ON db.dbid    = sysus.dbid
5> JOIN sysdevices   sysdev ON sysdev.vdevno = sysus.vdevno
6> WHERE db.name='Vbroken'
7> go
 dbid   name                                                         vdevno     
     phyname                                                                                                                                                                                                                                                        
 ------ ------------------------------------------------------------ -----------
     ----------------------------------------------------------------------------------------------------------------------------------
      7 Vbroken                                                               13
     /home/sybase/toolkit/564dd4a5-249f-7812-f266-65ffdf1278c1-vdb-12/datafile/dx876zVxedXAz68ux9_igdb_dev5                                                                                                                                                         
      7 Vbroken                                                               14
     /home/sybase/toolkit/564dd4a5-249f-7812-f266-65ffdf1278c1-vdb-12/datafile/dxlv91zf9kQvZz937A_igdb_dev6                                                                                                                                                         
      7 Vbroken                                                               16
     /home/sybase/toolkit/564dd4a5-249f-7812-f266-65ffdf1278c1-vdb-12/datafile/good_device 

4. Finally, disable and then enable the VDB using the Delphix GUI to confirm that it is functioning as expected.
5. If the VDB is successfully disabled and enabled, delete the errant device from the file system ("rm /home/sybase/toolkit/564dd4a5-249f-7812-f266-65ffdf1278c1-vdb-8/datafile/bad_device").

---

Troubleshooting

Look at the ASE error log for error number 840 to see if ASE detected that the VDB NFS mounts were not available when it started. Error 840 shows that the given device was not available when ASE was booted, and it was unable to recover the database and has left it offline:

00:0006:00000:00002:2021/01/26 11:05:58.35 server  Device 'AAAN$76zVxedXAz68ux9_igdb_dev5' (with physical name '/home/sybase/toolkit/564dd4a5-249f-7812-f266-65ffdf1278c1-vdb-12/datafile/dx876zVxedXAz68ux9_igdb_dev5', and virtual device number 13) has not been correctly activated at startup time.  Please contact a user with System Administrator (SA) role.
00:0006:00000:00002:2021/01/26 11:05:58.35 server  Unable to proceed with the recovery of dbid <7> because of previous errors.
00:0006:00000:00002:2021/01/26 11:05:58.35 server  Continuing recovery with the next database.

 

 

---