Data Control Tower Data Collection and Basic Checks (KBA9243)
KBA
KBA# 9243
Issue
This article is intended to provide instructions regarding initial data collection methods for Data Control Tower (on-premises).
This document does not apply to DCT SaaS, which is EOL as of May 2023
Applicable Delphix Versions
- Click here to view the versions of the Delphix engine to which this article applies
-
Major Release All Sub Releases 2.0 2.0.0.0
Container States
DCT deploys multiple Containers for various micro services, rather than a single monolithic process. These service states can be validated to check uptime, restarts, etc.
Docker
The Container states can be checked with docker ps or docker container ls (docker ps is an alias to docker container ls).
~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES eb8621bea375 delphix-dct-proxy:6.0.0 "/sbin/tini -- /boot…" 5 days ago Up 5 days 0.0.0.0:443->8443/tcp 60-proxy-1 fe852a37ec3a delphix-dct-app:6.0.0 "java -jar /opt/delp…" 5 days ago Up 5 days 60-gateway-1 872c71f6f6b9 delphix-dct-data-bookmarks:6.0.0 "/sbin/tini -- sh bo…" 5 days ago Up 5 days 60-data-bookmarks-1 a351dbe54dab delphix-dct-data-library:6.0.0 "/sbin/tini -- sh bo…" 5 days ago Up 5 days 60-data-library-1 5b405d5db627 delphix-dct-ui:6.0.0 "node ./index.js" 5 days ago Up 5 days 60-ui-1 131b99508cb7 delphix-dct-virtualization:6.0.0 "/sbin/tini -- sh bo…" 5 days ago Up 5 days 60-virtualization-1 ffd7622742b1 delphix-dct-jobs:6.0.0 "/sbin/tini -- sh bo…" 5 days ago Up 5 days 60-jobs-1 9f7ba417a935 delphix-dct-postgres:6.0.0 "docker-entrypoint.s…" 5 days ago Up 5 days 5432/tcp 60-database-1 334bbf9baf6a delphix-dct-graphql:6.0.0 "./tools/bootstrap.sh" 5 days ago Up 5 days 60-graphql-1
Kubernetes
The Container states can be checked with kubectl get pods. An additional namespace limiter (-n dct-services) can be used if multiple services are deployed in your Kubernetes configuration.
$ kubectl get pods NAME READY STATUS RESTARTS AGE graphql-6bc6495c6d-smjrf 1/1 Running 0 2d5h proxy-d8dfc97d7-kvsd9 1/1 Running 0 2d5h ui-b4466744d-z45n8 1/1 Running 0 2d5h virtualization-c98fb69d-gmmcz 1/1 Running 0 2d5h database-6bb6d55f46-zgbvr 1/1 Running 0 2d5h data-library-7dbf5895ff-5vj4n 1/1 Running 3 (2d5h ago) 2d5h gateway-555d6db45-hvftn 1/1 Running 0 2d5h jobs-5cd774f7bb-d72n5 1/1 Running 2 (2d5h ago) 2d5h data-bookmarks-899f457c6-jzhq8 1/1 Running 3 (2d5h ago) 2d5h job-cleaner-28013240-cpxmh 0/1 Completed 0 2m44s
Limits
Container limits may be applied as a result of standard business practices, but this can obviously restrict the resources for the application, and may cause issues (OOM, or inter-process timeouts due to insufficient CPU resources).
Container/pod memory limits are measured in bytes. This may be defined with a plain integer or fixed point number with quantity suffixes (M, G, etc), or may be in powers-of-two equivalents (Mi, Gi, etc).
Container/pod CPU limits are measured in CPU units. These units are typically millicpu (m); 200m would indicate 200 miillicpu, or .2 CPU
Docker
Docker container limits and current runtime usage can be checked in the docker stats subcommand. The --no-stream option will provide momentary output. Removing this option will continuously output statistics for the containers.
~$ docker stats --no-stream CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 613cba263f0d 501_proxy_1 0.01% 17.56MiB / 7.674GiB 0.22% 10.7MB / 10.9MB 5.85MB / 4.1kB 4 cda3c22a5bce 501_gateway_1 0.25% 522MiB / 7.674GiB 6.64% 336MB / 201MB 98.7MB / 134MB 79 d60b5a472dd0 501_virtualization_1 0.05% 146.8MiB / 7.674GiB 1.87% 63.7kB / 0B 31.2MB / 7.77MB 71 da18b7e07c97 501_jobs_1 0.04% 280.7MiB / 7.674GiB 3.57% 81MB / 76.2MB 34.8MB / 19.5MB 9 c198d0b89846 501_data-library_1 0.06% 226.8MiB / 7.674GiB 2.89% 130MB / 131MB 45.6MB / 12.4MB 205 26241420ce88 501_data-bookmarks_1 0.05% 190.2MiB / 7.674GiB 2.42% 66.9kB / 2.51kB 54.5MB / 10.8MB 93 7e5cc09e0f69 501_ui_1 0.00% 46.01MiB / 7.674GiB 0.59% 111kB / 10.3MB 45.8MB / 0B 11 1ecd346359df 501_graphql_1 0.02% 167.6MiB / 7.674GiB 2.13% 67.5kB / 31.8kB 56.4MB / 336kB 23 e96762cd9109 501_database_1 0.02% 71.87MiB / 7.674GiB 0.91% 301MB / 262MB 35.2MB / 1.39GB 22
A given container configuration can also be queried with the docker inspect command; this provides a lot of detail, so you will usually want to pare this down.
$ docker inspect 3e5fb0cc86a1
[
{
"Id": "3e5fb0cc86a1d34132ac1ae086ac27c14bc66c77ffd04d85d6ff40289061df0b",
"Created": "2023-06-22T20:09:49.079775364Z",
"Path": "java",
"Args": [
"-jar",
"/opt/delphix/app.jar"
],
"State": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 15523,
"ExitCode": 0,
"Error": "",
"StartedAt": "2023-06-22T20:09:51.313067194Z",
"FinishedAt": "0001-01-01T00:00:00Z"
...
Kubernetes
Kubernetes pod limits can be queried using the kubectl describe limits subcommand, or (similar to docker stats) by reviewing output of kubectl describe PodMetrics, which shows details of pod creation as well as current resource utilization.
Example taken from an external source:
$ kubectl describe limits mylimits --namespace=limit-example Name: mylimits Namespace: limit-example Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio ---- -------- --- --- --------------- ------------- ----------------------- Pod cpu 200m 2 - - - Pod memory 6Mi 1Gi - - - Container cpu 100m 2 200m 300m - Container memory 3Mi 1Gi 100Mi 200Mi -
The pod configuration can also be provided by the admin, or collected via kubectl get pod and output to yaml format for review.
$ kubectl get pod my-pod -o yaml
Additionally, the node details can be reviewed to determine the resources configured for the node, as well as the limits and utilization for all pods, with kubectl describe node. This command can be useful if pods are in a Pending state and no log information is provided from a DCT pod.
$ kubectl describe node
Name: kubernetes1
Roles: <none>
Labels: beta.kubernetes.io/arch=amd64
beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
kubernetes.io/hostname=kubernetes1
kubernetes.io/os=linux
microk8s.io/cluster=true
node.kubernetes.io/microk8s-controlplane=microk8s-controlplane
Annotations: node.alpha.kubernetes.io/ttl: 0
projectcalico.org/IPv4Address: 172.16.3.42/23
projectcalico.org/IPv4VXLANTunnelAddr: 10.1.230.0
volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp: Wed, 25 Jan 2023 10:31:59 +0000
Taints: <none>
Unschedulable: false
Lease:
HolderIdentity: kubernetes1
AcquireTime: <unset>
RenewTime: Fri, 05 May 2023 22:19:05 +0000
Conditions:
Type Status LastHeartbeatTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
NetworkUnavailable False Sun, 30 Apr 2023 04:10:07 +0000 Sun, 30 Apr 2023 04:10:07 +0000 CalicoIsUp Calico is running on this node
MemoryPressure False Fri, 05 May 2023 22:16:15 +0000 Wed, 25 Jan 2023 10:31:59 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available
DiskPressure False Fri, 05 May 2023 22:16:15 +0000 Wed, 25 Jan 2023 10:31:59 +0000 KubeletHasNoDiskPressure kubelet has no disk pressure
PIDPressure False Fri, 05 May 2023 22:16:15 +0000 Wed, 25 Jan 2023 10:31:59 +0000 KubeletHasSufficientPID kubelet has sufficient PID available
Ready True Fri, 05 May 2023 22:16:15 +0000 Sun, 30 Apr 2023 04:09:10 +0000 KubeletReady kubelet is posting ready status. AppArmor enabled
Addresses:
InternalIP: 172.16.3.42
Hostname: kubernetes1
Capacity:
cpu: 8
ephemeral-storage: 31811408Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 32875072Ki
pods: 110
Allocatable:
cpu: 8
ephemeral-storage: 30762832Ki
hugepages-1Gi: 0
hugepages-2Mi: 0
memory: 32772672Ki
pods: 110
System Info:
Machine ID: 2a2de0787ae44340892444a4710c62a1
System UUID: 149a3842-674b-ef70-4305-7c56b64e7bde
Boot ID: 1c51be01-37b7-4b58-a5a0-96ce14e74025
Kernel Version: 5.15.0-60-generic
OS Image: Ubuntu 22.04.1 LTS
Operating System: linux
Architecture: amd64
Container Runtime Version: containerd://1.6.15
Kubelet Version: v1.26.4
Kube-Proxy Version: v1.26.4
Non-terminated Pods: (17 in total)
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits Age
--------- ---- ------------ ---------- --------------- ------------- ---
kube-system kubernetes-dashboard-dc96f9fc-9qjlw 0 (0%) 0 (0%) 0 (0%) 0 (0%) 100d
kube-system dashboard-metrics-scraper-7bc864c59-dskkw 0 (0%) 0 (0%) 0 (0%) 0 (0%) 100d
local-path-storage local-path-provisioner-8bc8875b-jfhzm 0 (0%) 0 (0%) 0 (0%) 0 (0%) 100d
dct-services graphql-6bc6495c6d-smjrf 100m (1%) 0 (0%) 256Mi (0%) 0 (0%) 31d
dct-services proxy-d8dfc97d7-kvsd9 100m (1%) 0 (0%) 128Mi (0%) 0 (0%) 31d
dct-services ui-b4466744d-z45n8 100m (1%) 0 (0%) 128Mi (0%) 0 (0%) 31d
dct-services virtualization-c98fb69d-gmmcz 100m (1%) 0 (0%) 128Mi (0%) 0 (0%) 31d
dct-services database-6bb6d55f46-zgbvr 200m (2%) 0 (0%) 256Mi (0%) 0 (0%) 31d
dct-services data-library-7dbf5895ff-5vj4n 100m (1%) 0 (0%) 256Mi (0%) 0 (0%) 31d
dct-services gateway-555d6db45-hvftn 1 (12%) 0 (0%) 512Mi (1%) 0 (0%) 31d
dct-services jobs-5cd774f7bb-d72n5 200m (2%) 0 (0%) 256Mi (0%) 0 (0%) 31d
dct-services data-bookmarks-899f457c6-jzhq8 100m (1%) 0 (0%) 256Mi (0%) 0 (0%) 31d
ingress nginx-ingress-microk8s-controller-wdmjv 0 (0%) 0 (0%) 0 (0%) 0 (0%) 94d
kube-system coredns-6f5f9b5d74-7dt5q 100m (1%) 0 (0%) 70Mi (0%) 170Mi (0%) 100d
kube-system metrics-server-6f754f88d-qqgfj 100m (1%) 0 (0%) 200Mi (0%) 0 (0%) 100d
kube-system calico-node-bh558 250m (3%) 0 (0%) 0 (0%) 0 (0%) 5d18h
kube-system calico-kube-controllers-5dc5f99874-x6wtg 0 (0%) 0 (0%) 0 (0%) 0 (0%) 5d18h
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.)
Resource Requests Limits
-------- -------- ------
cpu 2450m (30%) 0 (0%)
memory 2446Mi (7%) 170Mi (0%)
ephemeral-storage 0 (0%) 0 (0%)
hugepages-1Gi 0 (0%) 0 (0%)
hugepages-2Mi 0 (0%) 0 (0%)
Events: <none>
Logs - Manual Collection
This section assumes local logging is implemented. Although this is the default, many installations will redirect logs to a central location, so the admin will often need to locate the container logs.
Logs for a given container can be obtained using the docker logs command. For ease of review, it may be desirable to limit the output by adding the -n <XX> argument to limit to the last XX number of lines.
The container can be identified by Container ID or by Name. Given the container ID and names listed in the previous section, consider the following example:
$ docker logs -n 10 87a9a00df173 2022-06-04 14:45:45.651 INFO 1 --- [l-2-thread-1123] .a.e.a.VirtualizationEngineApiClientImpl : Fetching system info 2022-06-04 14:45:45.744 INFO 1 --- [l-2-thread-1123] .a.e.a.VirtualizationEngineApiClientImpl : Fetching container list 2022-06-04 14:45:45.757 INFO 1 --- [l-2-thread-1123] .a.e.a.VirtualizationEngineApiClientImpl : Fetching source list 2022-06-04 14:45:45.772 INFO 1 --- [l-2-thread-1123] .a.e.a.VirtualizationEngineApiClientImpl : Fetching AppData toolkit list 2022-06-04 14:45:45.775 INFO 1 --- [l-2-thread-1123] .a.e.a.VirtualizationEngineApiClientImpl : Fetching namespace list 2022-06-04 14:45:45.777 INFO 1 --- [l-2-thread-1123] .a.e.a.VirtualizationEngineApiClientImpl : Fetching timeflows 2022-06-04 14:45:45.795 INFO 1 --- [l-2-thread-1123] .a.e.a.VirtualizationEngineApiClientImpl : Fetching JS data source list 2022-06-04 14:45:45.798 INFO 1 --- [l-2-thread-1123] .a.e.a.VirtualizationEngineApiClientImpl : Fetching JS data container list 2022-06-04 14:45:45.800 INFO 1 --- [l-2-thread-1123] .a.e.a.VirtualizationEngineApiClientImpl : Fetching JS data template list 2022-06-04 14:45:45.812 INFO 1 --- [l-2-thread-1123] .a.e.a.VirtualizationEngineApiClientImpl : Fetching source list
Additional options are available to control the output, such as --since to show logs since a given timestamp or relative time.
To collect logs from all containers, we can execute a simple for loop to collect logs from each container. An example is provided below to collect the last 48 hours of logs from each container, but obviously there are many methods to execute the same.
$ for i in `docker ps --format '{{.Names}}'`; do docker logs --since 48h $i > $i.log; done
And the resulting output generated:
$ ls -ltrh total 31M drwxrwxr-x 3 delphix delphix 4.0K Jun 1 16:36 dct -rw-rw-r-- 1 delphix delphix 0 Jun 4 15:04 dct_proxy_1.log -rw-rw-r-- 1 delphix delphix 21M Jun 4 15:04 dct_gateway_1.log -rw-rw-r-- 1 delphix delphix 5.4M Jun 4 15:04 dct_data-library_1.log -rw-rw-r-- 1 delphix delphix 0 Jun 4 15:04 dct_virtualization_1.log -rw-rw-r-- 1 delphix delphix 0 Jun 4 15:04 dct_data-bookmarks_1.log -rw-rw-r-- 1 delphix delphix 4.9M Jun 4 15:04 dct_jobs_1.log -rw-rw-r-- 1 delphix delphix 0 Jun 4 15:04 dct_database_1.log
docker-compose logs method is also available to generate logs from all containers:
$ docker-compose -f </path/to/docker-compose.yaml> logs
Scripted Data Collection via collect_bundle.sh
This section assumes local logging is implemented. Although this is default, many installations will redirect logs to a central location, so the admin will often need to locate the container logs.
In conjunction with the 8.0 release, the first data collection script has been provided to automate some of the data collection steps provided in this document by programmatically gathering logs and other data from the DCT containers.
As the script is not 'packaged' with the DCT distribution at this time, the download page can be used to get the latest version.
Example execution (Docker installation requires -d option):
$ ./collect-bundle.sh -d DCT support bundle collection started at Sat Jun 24 07:58:54 MDT 2023 Collecting logs from all DCT containers... Collecting hprof file (if one exists)... No hprof was found... Collecting Java thread dump... Collecting Java heap summary... Collecting container CPU and memory usage statistics... Generating DCT support bundle tar file... a support a support/container_stats a support/logs a support/inspectheap a support/threaddump a support/logs/80-ui-1.log a support/logs/80-graphql-1.log a support/logs/80-data-library-1.log a support/logs/80-virtualization-1.log a support/logs/80-database-1.log a support/logs/80-jobs-1.log a support/logs/80-gateway-1.log a support/logs/80-proxy-1.log a support/logs/80-data-bookmarks-1.log Deleting generated support directory... Generated support directory deleted. DCT support bundle collection completed at Sat Jun 24 07:59:00 MDT 2023
Expected content:
$ tar tvf dct-support-20230624-07-59-00.tar.gz drwxr-xr-x 0 user staff 0 Jun 24 07:58 support/ -rw-r--r-- 0 user staff 1177 Jun 24 07:59 support/container_stats drwxr-xr-x 0 user staff 0 Jun 24 07:58 support/logs/ -rw-r--r-- 0 user staff 791045 Jun 24 07:58 support/inspectheap -rw-r--r-- 0 user staff 60870 Jun 24 07:58 support/threaddump -rw-r--r-- 0 user staff 234 Jun 24 07:58 support/logs/80-ui-1.log -rw-r--r-- 0 user staff 674 Jun 24 07:58 support/logs/80-graphql-1.log -rw-r--r-- 0 user staff 39152 Jun 24 07:58 support/logs/80-data-library-1.log -rw-r--r-- 0 user staff 11000 Jun 24 07:58 support/logs/80-virtualization-1.log -rw-r--r-- 0 user staff 4286 Jun 24 07:58 support/logs/80-database-1.log -rw-r--r-- 0 user staff 42831 Jun 24 07:58 support/logs/80-jobs-1.log -rw-r--r-- 0 user staff 60919 Jun 24 07:58 support/logs/80-gateway-1.log -rw-r--r-- 0 user staff 4948 Jun 24 07:58 support/logs/80-proxy-1.log -rw-r--r-- 0 user staff 21318 Jun 24 07:58 support/logs/80-data-bookmarks-1.log
Related Articles
The following articles may provide more information or related information to this article:
- https://github.com/delphix/orbital-api-gateway
- https://docs.docker.com/storage/volumes/#backup-restore-or-migrate-data-volumes
- https://docs.docker.com/
https://kubernetes.io/docs/reference...tl/cheatsheet/