KBA1665 How to Extend the Root Filesystem on Delphix Reporting (Formerly Mission Control) VM
- Last updated
- Save as PDF
As a result of data collection from multiple Engines over time, the virtual machine (VM) where Mission Control resides may encounter disk space issues, as the root filesystem (housing the Mission Control database) approaches 100%. The following methodologies may be used to extend the root filesystem dynamically after additional disk devices are added to the VM, or if the existing physical device is expanded.
Resolution - Adding a new disk device in VMware
First, confirm the current configuration.
Please note, examples in this guide will not illustrate a VM at 100% filesystem utilization.
Confirm current configuration
[root@mission-control ~]# df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/vg_missioncontrol-lv_root ext4 44G 12G 31G 28% / tmpfs tmpfs 3.9G 0 3.9G 0% /dev/shm /dev/sda1 ext4 485M 33M 427M 8% /boot [root@mission-control ~]# lvs LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert lv_root vg_missioncontrol -wi-ao---- 44.51g lv_swap vg_missioncontrol -wi-ao---- 5.00g
Once additional storage device is added to the VM, the VM can be rebooted, or a manual rescan can be executed. To rescan, we first need to determine host bus number:
Find host bus #
[root@mission-control scsi_host]# grep mpt /sys/class/scsi_host/host?/proc_name /sys/class/scsi_host/host0/proc_name:mptspi
Whichever host# is returned, we use for the next command:
Rescan
[root@mission-control scsi_host]# echo "- - -" > /sys/class/scsi_host/host0/scan Once the command completes, we should see the new device infdisk -l output. Seen below/dev/sdbis the new LUN.
Checking fdisk -l output
[root@mission-control ~]# fdisk -l Disk /dev/sdb: 10.7 GB, 10737418240 bytes <<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 255 heads, 63 sectors/track, 1305 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/sda: 53.7 GB, 53687091200 bytes 255 heads, 63 sectors/track, 6527 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0006d0fd Device Boot Start End Blocks Id System /dev/sda1 * 1 64 512000 83 Linux Partition 1 does not end on cylinder boundary. /dev/sda2 64 6528 51915776 8e Linux LVM Disk /dev/mapper/vg_missioncontrol-lv_root: 47.8 GB, 47789899776 bytes 255 heads, 63 sectors/track, 5810 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/mapper/vg_missioncontrol-lv_swap: 5368 MB, 5368709120 bytes 255 heads, 63 sectors/track, 652 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000
We now use fdisk to configure. Commentary added inline, but this is all done from interactive fdisk utility. The output below will explain each of the commands entered in the preceding code block:
fdisk
[root@mission-control scsi_host]# fdisk /dev/sdb Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel Building a new DOS disklabel with disk identifier 0x023f80b1. Changes will remain in memory only, until you decide to write them. After that, of course, the previous content won't be recoverable. Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite) WARNING: DOS-compatible mode is deprecated. It's strongly recommended to switch off the mode (command 'c') and change display units to sectors (command 'u'). Command (m for help): n
Command n = new partition
fdisk (continued)
Command action e extended p primary partition (1-4) p
Command p = primary partition
fdisk (continued)
Partition number (1-4): 1
Enter 1 for first partition; only one partition will be created in this method, using the entire device.
fdisk (continued)
First cylinder (1-1305, default 1): <return>
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-1305, default 1305): <return>
Using default value 1305
It is OK to use defaults for first and last cylinder, by hitting <return> at these prompts.
fdisk (continued)
Command (m for help): t
Command t = change partition system ID
fdisk (continued)
Selected partition 1 Hex code (type L to list codes): 8e
8e = Linux LVM (logical volume manager)
fdisk (continued)
Changed system type of partition 1 to 8e (Linux LVM) Command (m for help): w
Command w = write partition table
fdisk (continued)
The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks.
At this point, fdisk should exit to command prompt.
Now, create physical volume on the device. Note, a "failed to get device" error may be output, but the command will still complete successfully. This is simply an OS multi-pathing check, and some discussion around this can be found at this link.
Create physical volume
[root@mission-control scsi_host]# pvcreate /dev/sdb1 dev_is_mpath: failed to get device for 8:17 Physical volume "/dev/sdb1" successfully created Extend the Volume Group vg_missioncontrol
Extend volume group
[root@mission-control scsi_host]# vgextend vg_missioncontrol /dev/sdb1 Volume group "vg_missioncontrol" successfully extended Extend the logical volume lv_root
Extend logical volume
[root@mission-control scsi_host]# lvextend -l +100%FREE /dev/vg_missioncontrol/lv_root Extending logical volume lv_root to 54.50 GiB Logical volume lv_root successfully resized Finally, resize the live filesystem
Resize live filesystem
[root@mission-control scsi_host]# resize2fs /dev/vg_missioncontrol/lv_root resize2fs 1.41.12 (17-May-2010) Filesystem at /dev/vg_missioncontrol/lv_root is mounted on /; on-line resizing required old desc_blocks = 3, new_desc_blocks = 4 Performing an on-line resize of /dev/vg_missioncontrol/lv_root to 14286848 (4k) blocks. The filesystem on /dev/vg_missioncontrol/lv_root is now 14286848 blocks long. Verify the changes. Observe in output below, increased Size and Avail.
[root@mission-control scsi_host]# df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/vg_missioncontrol-lv_root ext4 54G 12G 40G 23% / tmpfs tmpfs 3.9G 0 3.9G 0% /dev/shm /dev/sda1 ext4 485M 33M 427M 8% /boot
Resolution - Expanding the existing disk device
First, check/confirm the existing space:
Confirm existing configuration
[root@mission-control ~]# df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/vg_missioncontrol-lv_root ext4 44G 12G 31G 28% / tmpfs tmpfs 3.9G 0 3.9G 0% /dev/shm /dev/sda1 ext4 485M 33M 427M 8% /boot [root@mission-control ~]# lvs LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert lv_root vg_missioncontrol -wi-ao---- 44.51g lv_swap vg_missioncontrol -wi-ao---- 5.00g
Examine the current device size and identifier (/dev/sda in most cases for default config):
Check device size with fdisk -l
[root@mission-control ~]# fdisk -l Disk /dev/sdb: 10.7 GB, 10737418240 bytes 255 heads, 63 sectors/track, 1305 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x023f80b1 Device Boot Start End Blocks Id System /dev/sdb1 1 1305 10482381 8e Linux LVM Disk /dev/sda: 53.7 GB, 53687091200 bytes <<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 255 heads, 63 sectors/track, 6527 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0006d0fd Device Boot Start End Blocks Id System /dev/sda1 * 1 64 512000 83 Linux Partition 1 does not end on cylinder boundary. /dev/sda2 64 6528 51915776 8e Linux LVM Disk /dev/mapper/vg_missioncontrol-lv_root: 58.5 GB, 58518929408 bytes 255 heads, 63 sectors/track, 7114 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/mapper/vg_missioncontrol-lv_swap: 5368 MB, 5368709120 bytes 255 heads, 63 sectors/track, 652 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000
Second, confirm disk expanded in Vsphere GUI. Once the additional space is confirmed, we can rescan the bus to detect new space. The previous referenced command
echo "- - -" > /sys/class/scsi_host/host0/scan
will scan the bus for new devices, but not expanded space. To do THAT, we can use dmesg to confirm the bus address:
Confirm bus address and device identifier with dmesg
[root@mission-control ~]# dmesg | grep sda sd 0:0:0:0: [sda] 104857600 512-byte logical blocks: (53.6 GB/50.0 GiB) sd 0:0:0:0: [sda] Write Protect is off sd 0:0:0:0: [sda] Mode Sense: 61 00 00 00 sd 0:0:0:0: [sda] Cache data unavailable sd 0:0:0:0: [sda] Assuming drive cache: write through sd 0:0:0:0: [sda] Cache data unavailable sd 0:0:0:0: [sda] Assuming drive cache: write through sda: sda1 sda2 sd 0:0:0:0: [sda] Cache data unavailable sd 0:0:0:0: [sda] Assuming drive cache: write through sd 0:0:0:0: [sda] Attached SCSI disk dracut: Scanning devices sda2 for LVM logical volumes vg_missioncontrol/lv_root vg_missioncontrol/lv_swap EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: SELinux: initialized (dev sda1, type ext4), uses xattr
So here we confirm /dev/sda is 0:0:0:0. Now, we rescan this bus address:
Rescan 0:0:0:0 bus address for /dev/sda
[root@mission-control ~]# echo 1 > /sys/class/scsi_device/0\:0\:0\:0/device/rescan
We can confirm the rescan was successful using dmesg again, looking for "capacity change"...
Confirm capacity change with dmesg
[root@mission-control ~]# dmesg | grep sda | grep capacity sda: detected capacity change from 53687091200 to 75161927680
Or simply using fdisk again:
Confirm capacity change with fdisk
[root@mission-control ~]# fdisk -l Disk /dev/sdb: 10.7 GB, 10737418240 bytes 255 heads, 63 sectors/track, 1305 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x023f80b1 Device Boot Start End Blocks Id System /dev/sdb1 1 1305 10482381 8e Linux LVM Disk /dev/sda: 75.2 GB, 75161927680 bytes <<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 255 heads, 63 sectors/track, 9137 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x0006d0fd Device Boot Start End Blocks Id System /dev/sda1 * 1 64 512000 83 Linux Partition 1 does not end on cylinder boundary. /dev/sda2 64 6528 51915776 8e Linux LVM Disk /dev/mapper/vg_missioncontrol-lv_root: 58.5 GB, 58518929408 bytes 255 heads, 63 sectors/track, 7114 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Disk /dev/mapper/vg_missioncontrol-lv_swap: 5368 MB, 5368709120 bytes 255 heads, 63 sectors/track, 652 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000
Now that the physical space is available, we need to add a new partition (it's possible to expand root partition on the fly but not recommended). fdisk interaction is provided here without commentary as was included above, but we'll substitute any values if they're not appropriate. fdisk will typically auto-populate the correct first and last cylinders based on free space available. "M" command in fdisk will provide help for the options used
Create new partition with fdisk
[root@mission-control ~]# fdisk /dev/sda
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (6529-9137, default 6529):
Using default value 6529
Last cylinder, +cylinders or +size{K,M,G} (6529-9137, default 9137):
Using default value 9137
Command (m for help): t
Partition number (1-4): 3
Hex code (type L to list codes): 8e
Changed system type of partition 3 to 8e (Linux LVM)
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
Then rescan for new partition using partx
Rescan partition table with partx
[root@mission-control ~]# partx -v -a /dev/sda device /dev/sda: start 0 size 146800640 gpt: 0 slices dos: 4 slices # 1: 2048- 1026047 ( 1024000 sectors, 524 MB) # 2: 1026048-104872319 (103846272 sectors, 53169 MB) # 3: 104872320-146785904 ( 41913585 sectors, 21459 MB) # 4: 0- -1 ( 0 sectors, 0 MB) BLKPG: Device or resource busy error adding partition 1 BLKPG: Device or resource busy error adding partition 2 added partition 3
As we see above, "added partition 3" indicates partx successfully updated the kernel with the new partition just created.
Create a physical volume on the new partition with pvcreate
Create new physical volume with pvcreate
[root@mission-control ~]# pvcreate /dev/sda3 dev_is_mpath: failed to get device for 8:3 Physical volume "/dev/sda3" successfully created
Extend the volume group with vgextend.
Extend volume group with vgextend
[root@mission-control ~]# vgextend vg_missioncontrol /dev/sda3 Volume group "vg_missioncontrol" successfully extended
Extend the logical volume with lvextend.
Extend logical volume with lvextend
[root@mission-control ~]# lvextend -l +100%FREE /dev/vg_missioncontrol/lv_root Extending logical volume lv_root to 74.48 GiB Logical volume lv_root successfully resized
Finally, resize the live filesystem with resize2fs command.
Resize filesystem with resize2fs
[root@mission-control ~]# resize2fs /dev/vg_missioncontrol/lv_root resize2fs 1.41.12 (17-May-2010) Filesystem at /dev/vg_missioncontrol/lv_root is mounted on /; on-line resizing required old desc_blocks = 4, new_desc_blocks = 5 Performing an on-line resize of /dev/vg_missioncontrol/lv_root to 19525632 (4k) blocks. The filesystem on /dev/vg_missioncontrol/lv_root is now 19525632 blocks long.
We can now confirm the additional space is available in root filesystem:
Confirm file system space expanded
[root@mission-control ~]# df -Th Filesystem Type Size Used Avail Use% Mounted on /dev/mapper/vg_missioncontrol-lv_root ext4 74G 14G 57G 20% / tmpfs tmpfs 3.9G 0 3.9G 0% /dev/shm /dev/sda1 ext4 485M 33M 427M 8% /boot
| Topic Type | |
| Delphix Engine Version | |
| Database Platform | |
| Operating System | |
| Subject Area | |
| Effective Date of this Article | |
| Last Modification of this Article | 11.11.2016 |
| This Article Scheduled for Review | 02.09.2017 |