Skip to main content
Delphix

MOUNT or UNMOUNT of ASE Database Fails Due to Being Marked Suspect (KBA1113)

  1. Last updated
  2. Save as PDF
 

 

KBA

KBA#1113

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies
Major Release All Sub Releases
6.0 6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1, 6.0.4.2, 6.0.5.0, 6.0.6.0, 6.0.6.1, 6.0.7.0, 6.0.8.0, 6.0.8.1, 6.0.9.0, 6.0.10.0, 6.0.10.1, 6.0.11.0

5.3

 5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0, 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0

5.2

5.2.2.0, 5.2.2.1, 5.2.3.0, 5.2.4.0, 5.2.5.0, 5.2.5.1, 5.2.6.0, 5.2.6.1

5.1

5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0, 5.1.8.1, 5.1.9.0, 5.1.10.0

5.0

5.0.1.0, 5.0.1.1, 5.0.2.0, 5.0.2.1, 5.0.2.2, 5.0.2.3, 5.0.3.0, 5.0.3.1, 5.0.4.0, 5.0.4.1, 5.0.5.0, 5.0.5.1, 5.0.5.2, 5.0.5.3, 5.0.5.4

Issue

ASE marks a database suspect when it thinks there is a problem that a DBA should investigate. Commands that Delphix normally issues to manage virtual databases (VDBs) and staging databases (such as MOUNT/UNMOUNT) will not work on a database that has been marked suspect.

Delphix 6.0.5.0 and higher added some additional logic to cleanup up a VDB or staging database if the ASE UNMOUNT command fails. Delphix will attempt to run the ASE DROP DATABASE command to remove the database from the ASE instance, rollback to the last good snapshot and then bring the database online.

There are still a few outstanding corner cases where Delphix may not employ all the steps required to cleanup a database that was left in a bad state (for example if the VDB is running under Delphix Self Service) and so the recovery steps in this article may be necessary in a few rare instances.

Troubleshooting

 

The attempt to start, stop, disable or enable the dSource or VDB would fail with an error similar to:

Error
Failed to mount database "Vmission" on target host "asetest.acme.com" instance "SRC_157_4K": Database 'Vmission' cannot be opened. An earlier attempt at recovery marked it 'suspect'. Check the ASE errorlog for information as to the cause.

Error Code
exception.db.asetarget.target.database.mount.failed

Suggested Action
Make sure the host is online, the user "sybase" has permissions to mount the database, and try the operation again.

Attempts to use the database will fail with ASE error 926 and the ASE sp_helpdb command will show the database status as not recovered:

1> use Vmission
2> go
Msg 926, Level 14, State 1:
Server 'SRC_157_4K', Line 1:
Database 'Vmission' cannot be opened. An earlier attempt at recovery marked it 'suspect'. Check the ASE errorlog for information as to the cause.

1> sp_helpdb
2> go | egrep "Vmission|status|-------"
name                         db_size       owner dbid  created      durability  lobcomplvl inrowlen status                                                                                    
---------------------------- ------------- ----- ----- ------------ ----------- ---------- -------- -----------------------------------------------------------------------------------------
Vmission                         2048.0 MB sa       28 Mar 20, 2017 full                 0        0 not recovered                                                                             
(return status = 0)

In the Delphix debug logs, you may see ASE error 926 raised:

[2017-04-14 07:50:19,670][DEBUG][ase.target.ASEBaseTargetDatabase#unmountDatabaseException:394][Worker-1844|JOB-1004|SOURCE_DISABLE(ASE_VIRTUAL_SOURCE-23)][ACTION-1958] Unmounting  Vmission
[2017-04-14 07:50:19,690][INFO][ase.dbc.ASEJDBCAccessorImpl#messageHandler:1045][Worker-1844|JOB-1004|SOURCE_DISABLE(ASE_VIRTUAL_SOURCE-23)][ACTION-1958] ASE message 926 : Database 'Vmission' cannot be opened. An earlier attempt at recovery marked it 'suspect'. Check the ASE errorlog for information as to the cause.

[2017-04-14 07:50:19,691][DEBUG][ase.dbc.ASEJDBCAccessorImpl#executeWithProgress:1009][Worker-1844|JOB-1004|SOURCE_DISABLE(ASE_VIRTUAL_SOURCE-23)][ACTION-1958] ERROR: 926 : Database 'Vmission' cannot be opened. An earlier attempt at recovery marked it 'suspect'. Check the ASE errorlog for information as to the cause.

Resolution

The following procedure can be used to try and clear the suspect bit and restart the database without restarting the entire ASE instance.

  1. In the Delphix GUI, select the problematic dSource or VDB. Depending on the version of Delphix, either flip the card over or select `Configuration` and toggle the enable/disable switch for the VDB or dSource to disabled. This may succeed or it may return errors that the NFS devices are in use depending on the situation.
  2. Toggle the enable/disable switch for the VDB or dSource to enabled. This should fail because the VDB has been marked suspect but it should mount the NFS devices (if they were not already mounted). Check to see that they are mounted by executing the following commands from isql:
-- Issue sp_helpdb to see the list of devices belonging to the database
sp_helpdb <database_name>
go
-- Get the physical name of one the devices
sp_helpdevice <database_device>
go

Exit isql and check to see that the NFS directory is mounted by typing following command and grepping for the VDB number (replace "vdb-22" with the value from the sp_helpdevice output):

mount | grep vdb-22 | grep datafile
/export/home/sybase/toolkit/564dfe60-40b6-aec9-26fb-3baeb4e7b23b-vdb-22/datafile on 10.43.5.162:/domain0/group-74/ase_db_container-321/ase_timeflow-322/datafile remote/read/write/setuid/devices/rstchown/fg/hard/rsize=1048576/wsize=1048576/vers=3/nointr/timeo=600/proto=tcp/sec=sys/xattr/dev=4f01d6c on Fri Apr 14 13:51:12 2017
  1. If the devices are mounted, you can clear the database's suspect bit:
sp_configure "allow updates", 1
go
use master
go
begin transaction
go
update sysdatabases
set status = status & ~256
where name="<database_name>"
go
commit transaction
go
sp_configure "allow updates", 0
go
checkpoint
go
  1. Once the status has been cleared, try using the dbcc dbreboot command to restart the database without cycling the entire ASE instance:
dbcc dbreboot ("shutdown", "<database_name>")
---------- Shutting Down Database 'Vmission' ----------
---------- Operation on Database 'Vmission' Completed Successfully ----------
dbcc dbreboot ("restart", "<database_name>")
---------- Re-starting Database 'Vmission' With Recovery ----------
Recovering database 'Vmission'.
Started estimating recovery log boundaries for database 'Vmission'.
Database 'Vmission', checkpoint=(262149, 3), first=(262149, 3), last=(262149, 6).
Completed estimating recovery log boundaries for database 'Vmission'.
Started ANALYSIS pass for database 'Vmission'.
Completed ANALYSIS pass for database 'Vmission'.
Started REDO pass for database 'Vmission'. The total number of log records to process is 4.
Redo pass of recovery has processed 1 committed and 0 aborted transactions.
Completed REDO pass for database 'Vmission'.
Recovery of database 'Vmission' will undo incomplete nested top actions.
Started filling free space info for database 'Vmission'.
Completed filling free space info for database 'Vmission'.
Started cleaning up the default data cache for database 'Vmission'.
Completed cleaning up the default data cache for database 'Vmission'.
Recovery complete.
Database 'Vmission' is now online.
---------- Operation on Database 'Vmission' Completed Successfully ----------

If there is no actual corruption, the database should come online as illustrated above. If there is actual corruption in the database, ASE will mark it suspect again during this step. If there is actual corruption, you may use Delphix to rewind the VDB to the most recent snapshot that was taken prior to the corruption event.

  1. If you were able to toggle the database to disabled but the enable failed, you will want to toggle it back to enabled if the dbcc dbreboot() command was able to successfully restart the database so that database's status is properly reflected in Delphix.

External Links

Conditional content (Pro member)
Privacy Policy Terms of Use Legal About
© 2024 Delphix Corp