Delphix Storage Migration and Oracle Cloud (OCI) (KBA7909)

Perform Storage Test

First, select the disk device to be removed and proceed with the device removal process as discussed in the Storage Migration process:

https://cd.delphix.com/docs/latest/delphix-storage-migration

Disk2:2 will be used for this exercise.  

OCIEngine> storage device
OCIEngine storage device> ls
Objects
NAME     CONFIGURED  SIZE  EXPANDABLESIZE  FRAGMENTATION
Disk2:3  true        50GB  0B              0%
Disk2:2  true        50GB  0B              0%
Disk2:1  true        50GB  0B              0%
Disk2:0  true        70GB  0B              NA

Operations
refreshCache
OCIEngine storage device> select Disk2:2
OCIEngine storage device 'Disk2:2'> remove
OCIEngine storage device 'Disk2:2' remove *> commit
    Dispatched job JOB-1
    STORAGE_DEVICE_START_REMOVAL job started for "Disk2:2".
    STORAGE_DEVICE_START_REMOVAL job for "Disk2:2" completed successfully.


OCIEngine storage device 'Disk2:2'> back
OCIEngine storage device> ls
Objects
NAME     CONFIGURED  SIZE  EXPANDABLESIZE  FRAGMENTATION
Disk2:3  true        50GB  0B              0%
Disk2:2  false       50GB  -               -
Disk2:1  true        50GB  0B              0%
Disk2:0  true        70GB  0B              NA

Once the device removal job has completed, the CONFIGURED flag is indicated as false, confirming it has been successfully removed from the storage pool.  The disk is now eligible for selection in the storage test.

In this example, the storage test is configured for READ only; this selection is optional, and WRITE could be used alternatively. The test used ultimately does not matter, but will affect the selected metric to monitor later in this process.  The initializeDevices parameter is also set to false so the disk is not initialized prior to the storage test execution.

OCIEngine> /storage test
OCIEngine storage test> create
OCIEngine storage test create *> set devices=Disk2:2
OCIEngine storage test create *> set duration=30
OCIEngine storage test create *> set tests=READ
OCIEngine storage test create *> set initalizeDevices=false
OCIEngine storage test create *> ls
Properties
    type: StorageTestParameters
    devices: Disk2:2 (*)
    duration: 30 (*)
    initializeDevices: false (*)
    initializeEntireDevice: false
    testRegion: 512GB
    tests: READ (*)

OCIEngine storage test create *> commit
    `STORAGE_TEST-1
    Dispatched job JOB-3
    STORAGE_TEST_EXECUTE job started for "system".
    Initializing storage test.
    ETA: 0:03:34.
    Starting storage benchmarking.
    Starting sequential read workload with 64 KB block size and 4 jobs.
    Starting sequential read workload with 64 KB block size and 8 jobs.

Although the duration parameter is configured to 30 minutes, the test will complete much faster than this as only READ is configured; generally this has been observed to complete in 3-5 minutes.

For the purposes of the exercise, we are executing the test described above twice to have a more distinct I/O pattern to correlate with times.

 

Note:

In the process of disk device removal, all allocated data blocks on the disk are remapped to the remaining disk devices in the storage pool, so increased I/O activity across all disks is expected prior to these storage tests. The correlation of the type of I/O as well as the timing of the tests will be critical to ensure the correct devices are removed.

 

Correlate I/O Metrics - Single Block Volume

During or after the test, the I/O metrics for the Block Volumes associated with the compute instance can be reviewed individually by navigating to Compute - Instances, and clicking through to each of the associated Block Volumes, then reviewing each Block Volume metrics page (Click the Block Volume name and scroll to bottom). 

Once the corresponding I/O pattern of significant read activity for the two ~5-minute periods are located, we can confirm this is the disk device to be removed:

Correlate I/O Metrics - Multiple Block Volumes

For an Engine with a larger number of disk devices, the individual click-through method may be cumbersome, so alternatively the metrics for multiple volumes can be displayed simultaneously by building a query in Metrics Explorer under Observability + Management - Monitoring.

Prior to this, the Attachment ID for each block device needs to be located. This can be obtained in the OCI web interface for the Delphix Engine Compute Instance by clicking the three dot menu on right side of each attached volume, and selecting Copy Attachment OCID:

Alternatively, if OCI CLI is available the volume-attachment list command can be used to capture each attached Block Volume Attachment ID and name:

oci compute volume-attachment list --instance-id <compute instance ID> | grep volumeattachment

Example:

% oci compute volume-attachment list --instance-id ocid1.instance.oc1.phx.anyhqljrvkq43pycnltv4qt4x3quby46vaxef3ewm46roujncf5vv44mh22q --auth security_token --profile test | grep volumeattachment
      "display-name": "volumeattachment20210715212453",
      "id": "ocid1.volumeattachment.oc1.phx.anyhqljrvkq43pycm4c3p33cbcazggvlp6akydhgef6lh3t3apxxclgjjalq",
...

Once all Block Volume Attachment IDs are located, the query can be built.  From Metrics Explorer, select the applicable Compartment for the Engine Compute Instance, and select the Metric namespace: "oci_blockstore".  

If the storage test was configured as described above for READ only, select the Metric name "VolumeReadOps".

Under Metric Dimensions, set the Dimension name: attachmentId and set the Dimension value to the first Block Volume Attachment OCID obtained in previous steps.  All other options can be left with default values (Interval, Statistic). Clicking Update Chart will commit this query.

Then, click Add Query button to create additional queries for each of the attached Block Volumes with the same parameters selected as above for each unique Attachment OCID.

Once all Engine disks are configured in the query, the read I/O activity for all disks attached to the Engine will be represented.  Similar to the individual Block Volume graphs, the periods of storage test execution can be located on the timeline.

In the screenshot below, all 3 disk devices from the example Engine have been added using this methodology, so 3 queries are represented on the left-hand side of the interface:

By holding the mouse cursor over the line graph on the time period of increased IOPs, the Block Volume OCID can be determined, and the volume detached.

 

 

Note:

As discussed in https://cd.delphix.com/docs/latest/deployment-for-oci, the hot-removal of disk devices is not currently supported for OCI deployments.

 

If the Delphix Engine fails to boot after this process is completed, re-attach the Block Volume and boot the Engine again, and engage Delphix Customer Support for further assistance.

 

 

 

---

Related Articles

The following articles may provide more information or related information to this article:

• Oracle Cloud Documentation - Volume Metrics
• Delphix Documentation - Storage Migration
• Delphix Documentation - Deployment for OCI