KBA1665 How to Extend the Root Filesystem on Delphix Reporting (Formerly Mission Control) VM
- Last updated
- Save as PDF
As a result of data collection from multiple Engines over time, the virtual machine (VM) where Mission Control resides may encounter disk space issues, as the root filesystem (housing the Mission Control database) approaches 100%. The following methodologies may be used to extend the root filesystem dynamically after additional disk devices are added to the VM, or if the existing physical device is expanded.
Resolution - Adding a new disk device in VMware
First, confirm the current configuration.
Please note, examples in this guide will not illustrate a VM at 100% filesystem utilization.
[root@mission-control ~]# df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/vg_missioncontrol-lv_root ext4 44G 12G 31G 28% / tmpfs tmpfs 3.9G 0 3.9G 0% /dev/shm /dev/sda1 ext4 485M 33M 427M 8% /boot [root@mission-control ~]# lvs LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert lv_root vg_missioncontrol -wi-ao---- 44.51g lv_swap vg_missioncontrol -wi-ao---- 5.00g
Once additional storage device is added to the VM, the VM can be rebooted, or a manual rescan can be executed. To rescan, we first need to determine host bus number:
[root@mission-control scsi_host]# grep mpt /sys/class/scsi_host/host?/proc_name /sys/class/scsi_host/host0/proc_name:mptspi
Whichever host# is returned, we use for the next command:
[root@mission-control scsi_host]# echo "- - -" > /sys/class/scsi_host/host0/scan Once the command completes, we should see the new device infdisk -l output
. Seen below/dev/sdb
is the new LUN.
[root@mission-control ~]# fdisk -l Disk /dev/sdb: 10.7 GB, 10737418240 bytes <<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 255 heads, 63 sectors/track, 1305 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/sda: 53.7 GB, 53687091200 bytes 255 heads, 63 sectors/track, 6527 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0006d0fd Device Boot Start End Blocks Id System /dev/sda1 * 1 64 512000 83 Linux Partition 1 does not end on cylinder boundary. /dev/sda2 64 6528 51915776 8e Linux LVM Disk /dev/mapper/vg_missioncontrol-lv_root: 47.8 GB, 47789899776 bytes 255 heads, 63 sectors/track, 5810 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/mapper/vg_missioncontrol-lv_swap: 5368 MB, 5368709120 bytes 255 heads, 63 sectors/track, 652 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000
We now use fdisk to configure. Commentary added inline, but this is all done from interactive fdisk utility. The output below will explain each of the commands entered in the preceding code block:
[root@mission-control scsi_host]# fdisk /dev/sdb Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel with disk identifier 0x023f80b1. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable. Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite) WARNING: DOS-compatible mode is deprecated. It's strongly recommended to switch off the mode (command 'c') and change display units to sectors (command 'u'). Command (m for help): n
Command n = new partition
Command action e extended p primary partition (1-4) p
Command p = primary partition
Partition number (1-4): 1
Enter 1 for first partition; only one partition will be created in this method, using the entire device.
First cylinder (1-1305, default 1): <return> Using default value 1 Last cylinder, +cylinders or +size{K,M,G} (1-1305, default 1305): <return> Using default value 1305
It is OK to use defaults for first and last cylinder, by hitting <return> at these prompts.
Command (m for help): t
Command t = change partition system ID
Selected partition 1 Hex code (type L to list codes): 8e
8e = Linux LVM (logical volume manager)
Changed system type of partition 1 to 8e (Linux LVM) Command (m for help): w
Command w = write partition table
The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks.
At this point, fdisk should exit to command prompt.
Now, create physical volume on the device. Note, a "failed to get device" error may be output, but the command will still complete successfully. This is simply an OS multi-pathing check, and some discussion around this can be found at this link.
[root@mission-control scsi_host]# pvcreate /dev/sdb1 dev_is_mpath: failed to get device for 8:17 Physical volume "/dev/sdb1" successfully created Extend the Volume Group vg_missioncontrol
[root@mission-control scsi_host]# vgextend vg_missioncontrol /dev/sdb1 Volume group "vg_missioncontrol" successfully extended Extend the logical volume lv_root
[root@mission-control scsi_host]# lvextend -l +100%FREE /dev/vg_missioncontrol/lv_root Extending logical volume lv_root to 54.50 GiB Logical volume lv_root successfully resized Finally, resize the live filesystem
[root@mission-control scsi_host]# resize2fs /dev/vg_missioncontrol/lv_root resize2fs 1.41.12 (17-May-2010) Filesystem at /dev/vg_missioncontrol/lv_root is mounted on /; on-line resizing required old desc_blocks = 3, new_desc_blocks = 4 Performing an on-line resize of /dev/vg_missioncontrol/lv_root to 14286848 (4k) blocks. The filesystem on /dev/vg_missioncontrol/lv_root is now 14286848 blocks long. Verify the changes. Observe in output below, increased Size and Avail.
[root@mission-control scsi_host]# df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/vg_missioncontrol-lv_root ext4 54G 12G 40G 23% / tmpfs tmpfs 3.9G 0 3.9G 0% /dev/shm /dev/sda1 ext4 485M 33M 427M 8% /boot
Resolution - Expanding the existing disk device
First, check/confirm the existing space:
[root@mission-control ~]# df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/vg_missioncontrol-lv_root ext4 44G 12G 31G 28% / tmpfs tmpfs 3.9G 0 3.9G 0% /dev/shm /dev/sda1 ext4 485M 33M 427M 8% /boot [root@mission-control ~]# lvs LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert lv_root vg_missioncontrol -wi-ao---- 44.51g lv_swap vg_missioncontrol -wi-ao---- 5.00g
Examine the current device size and identifier (/dev/sda in most cases for default config):
[root@mission-control ~]# fdisk -l Disk /dev/sdb: 10.7 GB, 10737418240 bytes 255 heads, 63 sectors/track, 1305 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x023f80b1 Device Boot Start End Blocks Id System /dev/sdb1 1 1305 10482381 8e Linux LVM Disk /dev/sda: 53.7 GB, 53687091200 bytes <<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 255 heads, 63 sectors/track, 6527 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0006d0fd Device Boot Start End Blocks Id System /dev/sda1 * 1 64 512000 83 Linux Partition 1 does not end on cylinder boundary. /dev/sda2 64 6528 51915776 8e Linux LVM Disk /dev/mapper/vg_missioncontrol-lv_root: 58.5 GB, 58518929408 bytes 255 heads, 63 sectors/track, 7114 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/mapper/vg_missioncontrol-lv_swap: 5368 MB, 5368709120 bytes 255 heads, 63 sectors/track, 652 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000
Second, confirm disk expanded in Vsphere GUI. Once the additional space is confirmed, we can rescan the bus to detect new space. The previous referenced command
echo "- - -" > /sys/class/scsi_host/host0/scan
will scan the bus for new devices, but not expanded space. To do THAT, we can use dmesg to confirm the bus address:
[root@mission-control ~]# dmesg | grep sda sd 0:0:0:0: [sda] 104857600 512-byte logical blocks: (53.6 GB/50.0 GiB) sd 0:0:0:0: [sda] Write Protect is off sd 0:0:0:0: [sda] Mode Sense: 61 00 00 00 sd 0:0:0:0: [sda] Cache data unavailable sd 0:0:0:0: [sda] Assuming drive cache: write through sd 0:0:0:0: [sda] Cache data unavailable sd 0:0:0:0: [sda] Assuming drive cache: write through sda: sda1 sda2 sd 0:0:0:0: [sda] Cache data unavailable sd 0:0:0:0: [sda] Assuming drive cache: write through sd 0:0:0:0: [sda] Attached SCSI disk dracut: Scanning devices sda2 for LVM logical volumes vg_missioncontrol/lv_root vg_missioncontrol/lv_swap EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: SELinux: initialized (dev sda1, type ext4), uses xattr
So here we confirm /dev/sda is 0:0:0:0. Now, we rescan this bus address:
[root@mission-control ~]# echo 1 > /sys/class/scsi_device/0\:0\:0\:0/device/rescan
We can confirm the rescan was successful using dmesg again, looking for "capacity change"...
[root@mission-control ~]# dmesg | grep sda | grep capacity sda: detected capacity change from 53687091200 to 75161927680
Or simply using fdisk again:
[root@mission-control ~]# fdisk -l Disk /dev/sdb: 10.7 GB, 10737418240 bytes 255 heads, 63 sectors/track, 1305 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x023f80b1 Device Boot Start End Blocks Id System /dev/sdb1 1 1305 10482381 8e Linux LVM Disk /dev/sda: 75.2 GB, 75161927680 bytes <<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 255 heads, 63 sectors/track, 9137 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0006d0fd Device Boot Start End Blocks Id System /dev/sda1 * 1 64 512000 83 Linux Partition 1 does not end on cylinder boundary. /dev/sda2 64 6528 51915776 8e Linux LVM Disk /dev/mapper/vg_missioncontrol-lv_root: 58.5 GB, 58518929408 bytes 255 heads, 63 sectors/track, 7114 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/mapper/vg_missioncontrol-lv_swap: 5368 MB, 5368709120 bytes 255 heads, 63 sectors/track, 652 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000
Now that the physical space is available, we need to add a new partition (it's possible to expand root partition on the fly but not recommended). fdisk interaction is provided here without commentary as was included above, but we'll substitute any values if they're not appropriate. fdisk will typically auto-populate the correct first and last cylinders based on free space available. "M" command in fdisk will provide help for the options used
[root@mission-control ~]# fdisk /dev/sda WARNING: DOS-compatible mode is deprecated. It's strongly recommended to switch off the mode (command 'c') and change display units to sectors (command 'u'). Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 3 First cylinder (6529-9137, default 6529): Using default value 6529 Last cylinder, +cylinders or +size{K,M,G} (6529-9137, default 9137): Using default value 9137 Command (m for help): t Partition number (1-4): 3 Hex code (type L to list codes): 8e Changed system type of partition 3 to 8e (Linux LVM) Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. WARNING: Re-reading the partition table failed with error 16: Device or resource busy. The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8) Syncing disks.
Then rescan for new partition using partx
[root@mission-control ~]# partx -v -a /dev/sda device /dev/sda: start 0 size 146800640 gpt: 0 slices dos: 4 slices # 1: 2048- 1026047 ( 1024000 sectors, 524 MB) # 2: 1026048-104872319 (103846272 sectors, 53169 MB) # 3: 104872320-146785904 ( 41913585 sectors, 21459 MB) # 4: 0- -1 ( 0 sectors, 0 MB) BLKPG: Device or resource busy error adding partition 1 BLKPG: Device or resource busy error adding partition 2 added partition 3
As we see above, "added partition 3"
indicates partx successfully updated the kernel with the new partition just created.
Create a physical volume on the new partition with pvcreate
[root@mission-control ~]# pvcreate /dev/sda3 dev_is_mpath: failed to get device for 8:3 Physical volume "/dev/sda3" successfully created
Extend the volume group with vgextend.
[root@mission-control ~]# vgextend vg_missioncontrol /dev/sda3 Volume group "vg_missioncontrol" successfully extended
Extend the logical volume with lvextend.
[root@mission-control ~]# lvextend -l +100%FREE /dev/vg_missioncontrol/lv_root Extending logical volume lv_root to 74.48 GiB Logical volume lv_root successfully resized
Finally, resize the live filesystem with resize2fs command.
[root@mission-control ~]# resize2fs /dev/vg_missioncontrol/lv_root resize2fs 1.41.12 (17-May-2010) Filesystem at /dev/vg_missioncontrol/lv_root is mounted on /; on-line resizing required old desc_blocks = 4, new_desc_blocks = 5 Performing an on-line resize of /dev/vg_missioncontrol/lv_root to 19525632 (4k) blocks. The filesystem on /dev/vg_missioncontrol/lv_root is now 19525632 blocks long.
We can now confirm the additional space is available in root filesystem:
[root@mission-control ~]# df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/vg_missioncontrol-lv_root ext4 74G 14G 57G 20% / tmpfs tmpfs 3.9G 0 3.9G 0% /dev/shm /dev/sda1 ext4 485M 33M 427M 8% /boot
Topic Type | |
Delphix Engine Version | |
Database Platform | |
Operating System | |
Subject Area | |
Effective Date of this Article | |
Last Modification of this Article | 11.11.2016 |
This Article Scheduled for Review | 02.09.2017 |