Loss of Delphix Engine Network Connectivity Following Change of AWS Instance Type (KBA7366)

Issue

If network connectivity to an AWS-based Delphix engine VM instance is lost after changing the EC2 instance type, review the console log of the instance for any errors.

The instance console log, including the historical entries since the instance was powered on, can be viewed by selecting the Delphix engine instance in the AWS console and clicking Actions > Monitor and troubleshoot > Get system log.  This is shown in the screenshot below.  For newer EC2 instance types, the console is also available via the EC2 Serial Console option in the same menu, although this does not show the console history which may be needed in order to see any errors.

An error similar to the following indicates that the Delphix engine has been unable to configure the network interface.  In this situation, it is only possible to log into the engine via the instance serial console.

  [0m(0mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj(B[m(B[21A[0m(0(B[m(B[24;1H
Traceback (most recent call last):
  File "/usr/bin/delphix-startup-screen", line 394, in <module>
    curses.wrapper(installer_main)
  File "/usr/lib/python3.6/curses/__init__.py", line 94, in wrapper
    return func(stdscr, *args, **kwds)
  File "/usr/bin/delphix-startup-screen", line 384, in installer_main
    keypress = display_status(stdscr, win)
  File "/usr/bin/delphix-startup-screen", line 273, in display_status
    (hostname, ipaddr) = get_network_status()
  File "/usr/bin/delphix-startup-screen", line 225, in get_network_status
    ipaddr = output[output.index("src") + 1]
ValueError: 'src' is not in list

ip-172-16-1-81 login:

This can occur due to differences in the virtual network hardware between the two instance types, resulting in a failure to match the previous network configuration with the network interface in the new instance type.

Applicable Delphix Versions

Click here to view the versions of the Delphix engine to which this article applies

Major Release	All Sub Releases
6.0	6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1, 6.0.4.2, 6.0.5.0, 6.0.6.0, 6.0.6.1, 6.0.7.0

Resolution

To resolve this issue, connect to the EC2 Serial Console for the Delphix engine instance and log in as the user "sysadmin@SYSTEM".  The EC2 Serial Console facility is only available for EC2 instances that use Amazon's Nitro hypervisor, including the r5n family.  Older EC2 instance types do not provide interactive access to the instance console.  If the instance is running on an older instance type, you need to change to a newer instance type based on the Nitro hypervisor to complete these steps.

Details of how to connect to the EC2 Serial console are provided in this document - Accessing the Delphix Engine Console in AWS (KBA7306).
 

1. Log into the Delphix engine as the sysadmin@SYSTEM user via the instance console.

ip-172-16-1-81 login: sysadmin@SYSTEM
Password:
Welcome to Delphix 6.0.7.0

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

ip-172-16-1-81>

2. Navigate to the network section and select the primary network interface.  In this example, there is only one named ens5.

ip-172-16-1-81> network
ip-172-16-1-81 network> interface
ip-172-16-1-81 network interface> ls
Objects
NAME  MACADDRESS         MTU
ens5  0a:67:7e:62:1d:1d  1500
ip-172-16-1-81 network interface> select ens5

3. The interface properties show that it has no address configured and the state is DOWN.

ip-172-16-1-81 network interface 'ens5'> ls
Properties
    type: NetworkInterface
    name: ens5
    addresses: (empty)
    device: ens5
    macAddress: 0a:67:7e:62:1d:1d
    mtu: 1500
    mtuRange: 68-9001
    reference: NETWORK_INTERFACE-ens5
    state: DOWN

Operations
update
refresh

4. Update the interface and add a new address of type DHCP.

ip-172-16-1-81 network interface 'ens5'> update
ip-172-16-1-81 network interface 'ens5' update *> ls
Properties
    type: NetworkInterface
    name: ens5
    addresses: (empty)
    mtu: 1500
ip-172-16-1-81 network interface 'ens5' update *> edit addresses
ip-172-16-1-81 network interface 'ens5' update addresses *> add
ip-172-16-1-81 network interface 'ens5' update addresses 0 *> set addressType=DHCP
ip-172-16-1-81 network interface 'ens5' update addresses 0 *> ls
Properties
    type: InterfaceAddress (*)
    address: (unset)
    addressType: DHCP (*)
    enableSSH: true (*)

5. After committing the change, you can see that it now has an address assigned and is in the OK state.

ip-172-16-1-81 network interface 'ens5' update addresses 0 *> commit
ip-172-16-1-81 network interface 'ens5'> ls
Properties
    type: NetworkInterface
    name: ens5
    addresses:
        0:
            type: InterfaceAddress
            address: 172.16.1.81/20
            addressType: DHCP
            enableSSH: true
            sessionInUse: false
            state: OK
    device: ens5
    macAddress: 0a:67:7e:62:1d:1d
    mtu: 9001
    mtuRange: 68-9001
    reference: NETWORK_INTERFACE-ens5
    state: OK

It should now be possible to access the Delphix engine UI in your browser.  The networking changes will persist on the engine.