Adding or Refreshing a Unix/Linux Environment Fails With Error Code exception.ccc.writefile.failed or exception.ccc.connect.failed (KBA8569)
KBA
KBA# 8569
Issue
Adding or refreshing a Unix/Linux environment fails with Error Code: exception.ccc.writefile.failed or exception.ccc.connect.failed.
Please refer to the following possible examples:
- Command output net.schmizz.sshj.xfer.scp.SCPException: Received unknown response code:
- MTU issues:
- The following error is more generic and it may be caused by various issues:
Prerequisites
- Delphix engine
- Unix/Linux host environment
Applicable Delphix Versions
- Click here to view the versions of the Delphix engine to which this article applies
-
Major Release All Sub Releases 6.0 6.0.0.0, 6.0.1.0, 6.0.1.1, 6.0.2.0, 6.0.2.1, 6.0.3.0, 6.0.3.1, 6.0.4.0, 6.0.4.1, 6.0.4.2, 6.0.5.0, 6.0.6.0, 6.0.6.1, 6.0.7.0, 6.0.8.0, 6.0.8.1, 6.0.9.0, 6.0.10.0, 6.0.10.1, 6.0.11.0 5.3
5.3.0.0, 5.3.0.1, 5.3.0.2, 5.3.0.3, 5.3.1.0, 5.3.1.1, 5.3.1.2, 5.3.2.0, 5.3.3.0, 5.3.3.1, 5.3.4.0, 5.3.5.0, 5.3.6.0, 5.3.7.0, 5.3.7.1, 5.3.8.0, 5.3.8.1, 5.3.9.0 5.2
5.2.2.0, 5.2.2.1, 5.2.3.0, 5.2.4.0, 5.2.5.0, 5.2.5.1, 5.2.6.0, 5.2.6.1
5.1
5.1.0.0, 5.1.1.0, 5.1.2.0, 5.1.3.0, 5.1.4.0, 5.1.5.0, 5.1.5.1, 5.1.6.0, 5.1.7.0, 5.1.8.0, 5.1.8.1, 5.1.9.0, 5.1.10.0
5.0
5.0.1.0, 5.0.1.1, 5.0.2.0, 5.0.2.1, 5.0.2.2, 5.0.2.3, 5.0.3.0, 5.0.3.1, 5.0.4.0, 5.0.4.1, 5.0.5.0, 5.0.5.1, 5.0.5.2, 5.0.5.3, 5.0.5.4
Resolution
Please refer to the troubleshooting section to determine the possible issue and relative corrective actions.
Troubleshooting
Running scp -vvv [file] user@host:/home/user where the file is of a few MBs from another host should be able to provide more insight on the issue.
Below are a few examples of what to expect based on different possible issues:
Failed via ulimit -f (ulimit -f 10)
debug1: Sending command: scp -v -t /home/oracle3 debug2: channel 0: request exec confirm 1 debug3: send packet: type 98 debug2: channel_input_open_confirmation: channel 0: callback done debug2: channel 0: open confirm rwindow 0 rmax 32768 debug2: channel 0: rcvd adjust 2097152 debug3: receive packet: type 99 debug2: channel_input_status_confirm: type 99 id 0 debug2: exec request accepted on channel 0 Sending file modes: C0640 85540044 4229c416-f8e3-70e6-0619-2e6ccd0de36f-20150915-18-30-22.tar.gz debug2: channel 0: rcvd ext data 83 Sink: C0640 85540044 4229c416-f8e3-70e6-0619-2e6ccd0de36f-20150915-18-30-22.tar.gz debug2: channel 0: written 83 to efd 8 4229c416-f8e3-70e6-0619-2e6ccd0de36f-20150915-18-30-22.tar.gz 0% 0 0.0KB/s --:-- ETAdebug3: receive packet: type 98 debug1: client_input_channel_req: channel 0 rtype exit-signal reply 0 debug3: receive packet: type 98 debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0 debug2: channel 0: rcvd eow debug2: channel 0: chan_shutdown_read (i0 o0 sock -1 wfd 4 efd 8 [write]) debug2: channel 0: input open -> closed debug3: receive packet: type 96 debug2: channel 0: rcvd eof debug2: channel 0: output open -> drain debug2: channel 0: obuf empty debug2: channel 0: chan_shutdown_write (i3 o1 sock -1 wfd 7 efd 8 [write]) debug2: channel 0: output drain -> closed debug3: receive packet: type 97 debug2: channel 0: rcvd close debug3: channel 0: will not send data after close debug2: channel 0: almost dead debug2: channel 0: gc: notify user debug2: channel 0: gc: user detached debug2: channel 0: send close debug3: send packet: type 97 debug2: channel 0: is dead debug2: channel 0: garbage collecting debug1: channel 0: free: client-session, nchannels 1 debug3: channel 0: status: The following connections are open: #0 client-session (t4 r0 i3/1900544 o3/0 e[write]/0 fd -1/-1/8 sock -1 cc -1) debug3: send packet: type 1 debug1: fd 0 clearing O_NONBLOCK debug3: fd 1 is not O_NONBLOCK Transferred: sent 199312, received 2640 bytes, in 1.2 seconds Bytes per second: sent 165274.7, received 2189.2 debug1: Exit status -1 lost connection
Login as the OS environment user and check that the ulimit -f is set as to unlimited:
[delphix@environment ~]$ ulimit -f unlimited
If not, please contact your sysadmin to change it.
Failing because of an "echo" in .bashrc (.bashrc with interactive commands in it):
debug1: Sending command: scp -v -t /home/oracle3 debug2: channel 0: request exec confirm 1 debug3: send packet: type 98 debug2: channel_input_open_confirmation: channel 0: callback done debug2: channel 0: open confirm rwindow 0 rmax 32768 debug2: channel 0: rcvd adjust 2097152 debug3: receive packet: type 99 debug2: channel_input_status_confirm: type 99 id 0 debug2: exec request accepted on channel 0 This line will make SCP fail <---- this is output from the echo command debug2: channel 0: read<=0 rfd 4 len 0 debug2: channel 0: read failed debug2: channel 0: chan_shutdown_read (i0 o0 sock -1 wfd 4 efd 8 [write]) debug2: channel 0: input open -> drain debug2: channel 0: ibuf empty debug2: channel 0: send eof debug3: send packet: type 96 debug2: channel 0: input drain -> closed debug2: channel 0: write failed debug2: channel 0: chan_shutdown_write (i3 o0 sock -1 wfd 7 efd 8 [write]) debug2: channel 0: send eow debug3: send packet: type 98 debug2: channel 0: output open -> closed slatini:Downloads slatini$ debug3: receive packet: type 96 debug2: channel 0: rcvd eof debug3: receive packet: type 98 debug1: client_input_channel_req: channel 0 rtype exit-status reply 0 debug3: receive packet: type 97 debug2: channel 0: rcvd close debug3: channel 0: will not send data after close debug2: channel 0: almost dead debug2: channel 0: gc: notify user debug2: channel 0: gc: user detached debug2: channel 0: send close debug3: send packet: type 97 debug2: channel 0: is dead debug2: channel 0: garbage collecting debug1: channel 0: free: client-session, nchannels 1 debug3: channel 0: status: The following connections are open: #0 client-session (t4 r0 i3/0 o3/0 e[write]/0 fd -1/-1/8 sock -1 cc -1) debug3: send packet: type 1 debug1: fd 0 clearing O_NONBLOCK debug3: fd 1 is not O_NONBLOCK Transferred: sent 2440, received 2504 bytes, in 0.6 seconds Bytes per second: sent 3856.4, received 3957.6 debug1: Exit status 0
Please review your .bashrc file and remove/comment out any interactive commands or change the script to detect the shell type (external link with example: https://unix.stackexchange.com/questions/154395/running-scp-when-bashrc-of-remote-machine-includes-source-command).
MTU mismatch
It is possible that the following returns a diverse error from the one reported in the KB. In fact, the connection may hang for a long time in such cases:
debug1: Sending command: scp -v -t /home/oracle3 debug2: channel 0: request exec confirm 1 debug3: send packet: type 98 debug2: channel_input_open_confirmation: channel 0: callback done debug2: channel 0: open confirm rwindow 0 rmax 32768 debug2: channel 0: rcvd adjust 2097152 debug3: receive packet: type 99 debug2: channel_input_status_confirm: type 99 id 0 debug2: exec request accepted on channel 0 Sending file modes: C0664 103023513 jdk.tar.gz debug2: channel 0: rcvd ext data 33 Sink: C0664 103023513 jdk.tar.gz debug2: channel 0: written 33 to efd 6 4229c416-f8e3-70e6-0619-2e6ccd0de36f-20150915-18-30-22.tar.gz 0% 0 0.0KB/s --:-- ETA ebug3: send packet: type 1 packet_write_wait: Connection to 10.10.10.173 port 22: Broken pipe lost connection
Our documentation provide a few examples on how to test the MTU from your environment to the Delphix engine here: https://docs.delphix.com/docs/best-practices/best-practices-for-network-configuration
Please refer to section 2.f Jumbo frames check via ping.
Please notice that the test is not limited to Jumbo frames but also standard MTU of 1500 bytes can be checked with the same method.
MTU can be determined via "ifconfig -a" in most Unix/Linux environments. For example:
[delphix@environment ~]$ ifconfig -a ens192: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
The Delphix engine's MTU can be viewed under the Network section in the Server Setup page, by logging in as a sysadmin user.
For example:
Working example:
debug1: Sending command: scp -v -t /home/oracle3 debug2: channel 0: request exec confirm 1 debug3: send packet: type 98 debug2: channel_input_open_confirmation: channel 0: callback done debug2: channel 0: open confirm rwindow 0 rmax 32768 debug2: channel 0: rcvd adjust 2097152 debug3: receive packet: type 99 debug2: channel_input_status_confirm: type 99 id 0 debug2: exec request accepted on channel 0 Sending file modes: C0640 85540044 4229c416-f8e3-70e6-0619-2e6ccd0de36f-20150915-18-30-22.tar.gz debug2: channel 0: rcvd ext data 83 Sink: C0640 85540044 4229c416-f8e3-70e6-0619-2e6ccd0de36f-20150915-18-30-22.tar.gz debug2: channel 0: written 83 to efd 8 4229c416-f8e3-70e6-0619-2e6ccd0de36f-20150915-18-30-22.tar.gz 0% 0 0.0KB/s --:-- ETAdebug2: channel 0: rcvd adjust 98381 4229c416-f8e3-70e6-0619-2e6ccd0de36f-20150915-18-30-22.tar.gz 2% 2208KB 2.2MB/s 00:36 ETAdebug2: channel 0: rcvd adjust 131072 <snip> 4229c416-f8e3-70e6-0619-2e6ccd0de36f-20150915-18-30-22.tar.gz 100% 82MB 4.9MB/s 00:16 debug2: channel 0: read<=0 rfd 4 len 0 debug2: channel 0: read failed debug2: channel 0: chan_shutdown_read (i0 o0 sock -1 wfd 4 efd 8 [write]) debug2: channel 0: input open -> drain debug2: channel 0: ibuf empty debug2: channel 0: send eof debug3: send packet: type 96 debug2: channel 0: input drain -> closed debug3: receive packet: type 96 debug2: channel 0: rcvd eof debug2: channel 0: output open -> drain debug2: channel 0: obuf empty debug2: channel 0: chan_shutdown_write (i3 o1 sock -1 wfd 7 efd 8 [write]) debug2: channel 0: output drain -> closed debug3: receive packet: type 98 debug1: client_input_channel_req: channel 0 rtype exit-status reply 0 debug3: receive packet: type 97 debug2: channel 0: rcvd close debug3: channel 0: will not send data after close debug2: channel 0: almost dead debug2: channel 0: gc: notify user debug2: channel 0: gc: user detached debug2: channel 0: send close debug3: send packet: type 97 debug2: channel 0: is dead debug2: channel 0: garbage collecting debug1: channel 0: free: client-session, nchannels 1 debug3: channel 0: status: The following connections are open: #0 client-session (t4 r0 i3/0 o3/0 e[write]/0 fd -1/-1/8 sock -1 cc -1) debug3: send packet: type 1 debug1: fd 0 clearing O_NONBLOCK debug3: fd 1 is not O_NONBLOCK Transferred: sent 85601512, received 15880 bytes, in 17.6 seconds Bytes per second: sent 4875603.8, received 904.5 debug1: Exit status 0
Related Articles
The following articles may provide more information or related information to this article:
- Ulimit configuration: http://uw714doc.xinuos.com/en/HANDBO...aT.ulimit.html (external link)
- Bashrc with interactive commands: https://unix.stackexchange.com/quest...source-command (external link)
- MTU ping test: https://docs.delphix.com/docs/best-practices/best-practices-for-network-configuration