Skip to main content
Delphix

How to extend the root filesystem on Mission Control VM

Issue

As a result of data collection from multiple Engines over time, the virtual machine (VM) where Mission Control resides may encounter disk space issues, as the root filesystem (housing the Mission Control database) approaches 100%. The following methodologies may be used to extend the root filesystem dynamically after additional disk devices are added to the VM, or if the existing physical device is expanded.  

Resolution - Adding a new disk device in VMware

First, confirm the current configuration.  

Please note, examples in this guide will not illustrate a VM at 100% filesystem utilization.

Confirm current configuration
[root@mission-control ~]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/vg_missioncontrol-lv_root ext4 44G 12G 31G 28% /
tmpfs tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/sda1 ext4 485M 33M 427M 8% /boot

[root@mission-control ~]# lvs
LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert
lv_root vg_missioncontrol -wi-ao---- 44.51g 
lv_swap vg_missioncontrol -wi-ao---- 5.00g

Once additional storage device is added to the VM, the VM can be rebooted, or a manual rescan can be executed. To rescan, we first need to determine host bus number:

Find host bus #
[root@mission-control scsi_host]# grep mpt /sys/class/scsi_host/host?/proc_name
/sys/class/scsi_host/host0/proc_name:mptspi

Whichever host# is returned, we use for the next command:

Rescan
[root@mission-control scsi_host]# echo "- - -" > /sys/class/scsi_host/host0/scan
Once the command completes, we should see the new device in fdisk -l output. Seen below /dev/sdb is the new LUN.
Checking fdisk -l output
[root@mission-control ~]# fdisk -l
Disk /dev/sdb: 10.7 GB, 10737418240 bytes       <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/sda: 53.7 GB, 53687091200 bytes
255 heads, 63 sectors/track, 6527 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0006d0fd
Device Boot Start End Blocks Id System
/dev/sda1 * 1 64 512000 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 64 6528 51915776 8e Linux LVM
Disk /dev/mapper/vg_missioncontrol-lv_root: 47.8 GB, 47789899776 bytes
255 heads, 63 sectors/track, 5810 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Disk /dev/mapper/vg_missioncontrol-lv_swap: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

We now use fdisk to configure. Commentary added inline, but this is all done from interactive fdisk utility. The output below will explain each of the commands entered in the preceding code block:

fdisk
[root@mission-control scsi_host]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel with disk identifier 0x023f80b1.
Changes will remain in memory only, until you decide to write them.
After that, of course, the previous content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): n

Command n = new partition

fdisk (continued)
Command action
e extended
p primary partition (1-4)
p

Command p = primary partition

fdisk (continued)
Partition number (1-4): 1

Enter 1 for first partition; only one partition will be created in this method, using the entire device.

fdisk (continued)
First cylinder (1-1305, default 1):  <return>
Using default value 1
Last cylinder, +cylinders or +size{K,M,G} (1-1305, default 1305): <return>
Using default value 1305

It is OK to use defaults for first and last cylinder, by hitting <return> at these prompts.

fdisk (continued)
Command (m for help): t

Command t = change partition system ID 

fdisk (continued)
Selected partition 1
Hex code (type L to list codes): 8e

8e = Linux LVM (logical volume manager)

fdisk (continued)
Changed system type of partition 1 to 8e (Linux LVM)
Command (m for help): w

Command w = write partition table

fdisk (continued)
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.

At this point, fdisk should exit to command prompt.

Now, create physical volume on the device. Note, a "failed to get device" error may be output, but the command will still complete successfully. This is simply an OS multi-pathing check, and some discussion around this can be found at this link.  

Create physical volume
[root@mission-control scsi_host]# pvcreate /dev/sdb1
dev_is_mpath: failed to get device for 8:17
Physical volume "/dev/sdb1" successfully created
Extend the Volume Group vg_missioncontrol
Extend volume group
[root@mission-control scsi_host]# vgextend vg_missioncontrol /dev/sdb1
Volume group "vg_missioncontrol" successfully extended
Extend the logical volume lv_root
Extend logical volume
[root@mission-control scsi_host]# lvextend -l +100%FREE /dev/vg_missioncontrol/lv_root
Extending logical volume lv_root to 54.50 GiB
Logical volume lv_root successfully resized
Finally, resize the live filesystem
Resize live filesystem
[root@mission-control scsi_host]# resize2fs /dev/vg_missioncontrol/lv_root
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/vg_missioncontrol/lv_root is mounted on /; on-line resizing required
old desc_blocks = 3, new_desc_blocks = 4
Performing an on-line resize of /dev/vg_missioncontrol/lv_root to 14286848 (4k) blocks.
The filesystem on /dev/vg_missioncontrol/lv_root is now 14286848 blocks long.
Verify the changes. Observe in output below, increased Size and Avail.
[root@mission-control scsi_host]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/vg_missioncontrol-lv_root ext4 54G 12G 40G 23% /
tmpfs tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/sda1 ext4 485M 33M 427M 8% /boot

Resolution - Expanding the existing disk device

First, check/confirm the existing space:

Confirm existing configuration
[root@mission-control ~]# df -Th
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/vg_missioncontrol-lv_root ext4 44G 12G 31G 28% /
tmpfs tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/sda1 ext4 485M 33M 427M 8% /boot

[root@mission-control ~]# lvs
LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert
lv_root vg_missioncontrol -wi-ao---- 44.51g 
lv_swap vg_missioncontrol -wi-ao---- 5.00g

Examine the current device size and identifier (/dev/sda in most cases for default config):

Check device size with fdisk -l
[root@mission-control ~]# fdisk -l

Disk /dev/sdb: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x023f80b1

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1        1305    10482381   8e  Linux LVM

Disk /dev/sda: 53.7 GB, 53687091200 bytes       <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
255 heads, 63 sectors/track, 6527 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0006d0fd

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          64      512000   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2              64        6528    51915776   8e  Linux LVM

Disk /dev/mapper/vg_missioncontrol-lv_root: 58.5 GB, 58518929408 bytes
255 heads, 63 sectors/track, 7114 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/vg_missioncontrol-lv_swap: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Second, confirm disk expanded in Vsphere GUI. Once the additional space is confirmed, we can rescan the bus to detect new space. The previous referenced command 

echo "- - -" > /sys/class/scsi_host/host0/scan

 will scan the bus for new devices, but not expanded space. To do THAT, we can use dmesg to confirm the bus address:

Confirm bus address and device identifier with dmesg
[root@mission-control ~]# dmesg | grep sda
sd 0:0:0:0: [sda] 104857600 512-byte logical blocks: (53.6 GB/50.0 GiB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 61 00 00 00
sd 0:0:0:0: [sda] Cache data unavailable
sd 0:0:0:0: [sda] Assuming drive cache: write through
sd 0:0:0:0: [sda] Cache data unavailable
sd 0:0:0:0: [sda] Assuming drive cache: write through
 sda: sda1 sda2
sd 0:0:0:0: [sda] Cache data unavailable
sd 0:0:0:0: [sda] Assuming drive cache: write through
sd 0:0:0:0: [sda] Attached SCSI disk
dracut: Scanning devices sda2  for LVM logical volumes vg_missioncontrol/lv_root vg_missioncontrol/lv_swap 
EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: 
SELinux: initialized (dev sda1, type ext4), uses xattr

So here we confirm /dev/sda is 0:0:0:0. Now, we rescan this bus address:

Rescan 0:0:0:0 bus address for /dev/sda
[root@mission-control ~]# echo 1 > /sys/class/scsi_device/0\:0\:0\:0/device/rescan

We can confirm the rescan was successful using dmesg again, looking for "capacity change"...

Confirm capacity change with dmesg
[root@mission-control ~]# dmesg | grep sda | grep capacity
sda: detected capacity change from 53687091200 to 75161927680

Or simply using fdisk again:

Confirm capacity change with fdisk
[root@mission-control ~]# fdisk -l

Disk /dev/sdb: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x023f80b1

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1        1305    10482381   8e  Linux LVM

Disk /dev/sda: 75.2 GB, 75161927680 bytes       <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
255 heads, 63 sectors/track, 9137 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0006d0fd

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          64      512000   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2              64        6528    51915776   8e  Linux LVM

Disk /dev/mapper/vg_missioncontrol-lv_root: 58.5 GB, 58518929408 bytes
255 heads, 63 sectors/track, 7114 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000


Disk /dev/mapper/vg_missioncontrol-lv_swap: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Now that the physical space is available, we need to add a new partition (it's possible to expand root partition on the fly but not recommended). fdisk interaction is provided here without commentary as was included above, but we'll substitute any values if they're not appropriate. fdisk will typically auto-populate the correct first and last cylinders based on free space available.  "M" command in fdisk will provide help for the options used

Create new partition with fdisk
[root@mission-control ~]# fdisk /dev/sda

WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
         switch off the mode (command 'c') and change display units to
         sectors (command 'u').

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (6529-9137, default 6529): 
Using default value 6529
Last cylinder, +cylinders or +size{K,M,G} (6529-9137, default 9137): 
Using default value 9137

Command (m for help): t
Partition number (1-4): 3
Hex code (type L to list codes): 8e
Changed system type of partition 3 to 8e (Linux LVM)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.

Then rescan for new partition using partx

Rescan partition table with partx
[root@mission-control ~]# partx -v -a /dev/sda
device /dev/sda: start 0 size 146800640
gpt: 0 slices
dos: 4 slices
# 1:      2048-  1026047 (  1024000 sectors,    524 MB)
# 2:   1026048-104872319 (103846272 sectors,  53169 MB)
# 3: 104872320-146785904 ( 41913585 sectors,  21459 MB)
# 4:         0-       -1 (        0 sectors,      0 MB)
BLKPG: Device or resource busy
error adding partition 1
BLKPG: Device or resource busy
error adding partition 2
added partition 3

As we see above, "added partition 3" indicates partx successfully updated the kernel with the new partition just created.

Create a physical volume on the new partition with pvcreate

Create new physical volume with pvcreate
[root@mission-control ~]# pvcreate /dev/sda3
  dev_is_mpath: failed to get device for 8:3
  Physical volume "/dev/sda3" successfully created
Note, a "failed to get device" error may be output, but the command will still complete successfully. This is simply an OS multi-pathing check, and some discussion around this can be found at this link

Extend the volume group with vgextend.

Extend volume group with vgextend
[root@mission-control ~]# vgextend vg_missioncontrol /dev/sda3
  Volume group "vg_missioncontrol" successfully extended

Extend the logical volume with lvextend.

Extend logical volume with lvextend
[root@mission-control ~]# lvextend -l +100%FREE /dev/vg_missioncontrol/lv_root
  Extending logical volume lv_root to 74.48 GiB
  Logical volume lv_root successfully resized

Finally, resize the live filesystem with resize2fs command.

Resize filesystem with resize2fs
[root@mission-control ~]# resize2fs /dev/vg_missioncontrol/lv_root
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/vg_missioncontrol/lv_root is mounted on /; on-line resizing required
old desc_blocks = 4, new_desc_blocks = 5
Performing an on-line resize of /dev/vg_missioncontrol/lv_root to 19525632 (4k) blocks.
The filesystem on /dev/vg_missioncontrol/lv_root is now 19525632 blocks long.

We can now confirm the additional space is available in root filesystem:

Confirm file system space expanded
[root@mission-control ~]# df -Th
Filesystem                            Type   Size  Used Avail Use% Mounted on
/dev/mapper/vg_missioncontrol-lv_root ext4    74G   14G   57G  20% /
tmpfs                                 tmpfs  3.9G     0  3.9G   0% /dev/shm
/dev/sda1                             ext4   485M   33M  427M   8% /boot
Topic Type  
Delphix Engine Version  
Database Platform  
Operating System  
Subject Area  
Effective Date of this Article  
Last Modification of this Article 11.11.2016
This Article Scheduled for Review 02.09.2017
  • Was this article helpful?