TrueNAS SSH Issue

[LIGISTX]

I have been facing this issue for quite a while now, long enough where I have no clue what may have caused it (years…).

 

I can not keep an active SSH client open with my truenas machine, but what is way stranger, is the web besed shell you can open from truenas’s webUI also seems to hang and stop responding.

 

When SSHed in, after 30 seconds to a few minutes, I get a SSH hang and a:

Connection to server.name.here closed. client_loop: send disconnect: Broken pipe

When in the webUI Shell, the screen just hangs and I can’t make inputs any longer.

 

I admit, earlier in my truenas/homelab career I likely copy/pasted some stuff into CLI, but I have no idea what any of that was. I was trying to get plex permissions working back on FreeNAS jails (was this called beehive maybe? It was literally a deacde ago). My truenas “works fine” and seemingly has for years and years. SMB and NFS performance always seems fine, the system itself is stable and has migrated to truenas scale years ago, etc. Things seem to work fine (except my timemachine backups really don’t like working, not sure if this is related, really not sure…).

 

To try and fix this a few months ago, I fully restarted from ground 0. I fresh installed Scale and imported my ZFS Array. I didn’t copy my config, I went through the relative pain (although, it was good to do it all again since it had been a decade…) of resetting up everything. Vlans, network adapters, users, shares, etc. Obviously the data on the array persisted, including my home directory. But this is where my linux ignorance comes in - I know enough to be dangerous these days, but I don’t understand how such an issue can persist such a nuclear option. I am in full believe my early year copy/past into CLI to try and alter permissions and stuff for jails or some other “thing I thought was smart” could have caused issues, but is this something that can persists OS wipes like this? I was “dumb” and installed oh my ZSH on my user account, but this same issue happens with root, and root is not modified in any way that I know of.

 

I had thought this could be networking related, but I can’t figure out how or what that would be. I run a pfsense network with unifi switching hardware, and no over VM/physical host has any such issues. Literally no other VM or machine has any weird SSH issue, or weird hiccup like this, and again, its only SSH and Shell via the webUI… it seems very specific to something internal to truenas.

 

I am at a total loss on how to fix this or what to even try and do to narrow in on possible issues.

 

Any help would be greatly apprecaited. @Electronics Wizardy @Eigenvektor @Lurick maybe one of yall have any ideas? I have ran out of smarts on this one...

 

Adding some more info as this may be asked:

 

/etc/ssh/ssh_config looks like:

# This is the ssh client system-wide configuration file.  See
# ssh_config(5) for more information.  This file provides defaults for
# users, and the values can be changed in per-user configuration files
# or on the command line.

# Configuration data is parsed as follows:
#  1. command line options
#  2. user-specific file
#  3. system-wide file
# Any configuration value is only changed the first time it is set.
# Thus, host-specific definitions should be at the beginning of the
# configuration file, and defaults at the end.

# Site-wide defaults for some commonly used options.  For a comprehensive
# list of available options, their meanings and defaults, please see the
# ssh_config(5) man page.

Include /etc/ssh/ssh_config.d/*.conf

Host *
#   ForwardAgent no
#   ForwardX11 no
#   ForwardX11Trusted yes
#   PasswordAuthentication yes
#   HostbasedAuthentication no
#   GSSAPIAuthentication no
#   GSSAPIDelegateCredentials no
#   GSSAPIKeyExchange no
#   GSSAPITrustDNS no
#   BatchMode no
#   CheckHostIP yes
#   AddressFamily any
#   ConnectTimeout 0
#   StrictHostKeyChecking ask
#   IdentityFile ~/.ssh/id_rsa
#   IdentityFile ~/.ssh/id_dsa
#   IdentityFile ~/.ssh/id_ecdsa
#   IdentityFile ~/.ssh/id_ed25519
#   Port 22
#   Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc
#   MACs hmac-md5,hmac-sha1,umac-64@openssh.com
#   EscapeChar ~
#   Tunnel no
#   TunnelDevice any:any
#   PermitLocalCommand no
#   VisualHostKey no
#   ProxyCommand ssh -q -W %h:%p gateway.example.com
#   RekeyLimit 1G 1h
#   UserKnownHostsFile ~/.ssh/known_hosts.d/%k
    SendEnv LANG LC_*
    HashKnownHosts yes
    GSSAPIAuthentication yes

and /etc/ssh/sshd_config:

Subsystem	sftp	internal-sftp -l ERROR -f AUTH
Protocol 2
UseDNS no
ChallengeResponseAuthentication no
VersionAddendum none
Port 22
ListenAddress 127.0.0.1
ListenAddress 10.90.5.100
ListenAddress fe80::f8f1:99ff:feb4:64d0%ens18
PermitRootLogin without-password
AllowTcpForwarding no
Compression no
PasswordAuthentication no
PubkeyAuthentication yes

# These are forced to be enabled with 2FA
UsePAM yes
PrintMotd no
SetEnv LC_ALL=C.UTF-8

# These are aux params that MUST COME LAST
# in the config. User provided "Match" blocks,
# for example, need to come AFTER the UsePam
# line. Otherwise ssh service WILL NOT START.
ClientAliveInterval = 15
ClientAliveCountMax = 3

I have .ssh folders in my other user account home dir's, but none of them have .config files.

[StDragon]

 Try changing 'ClientAliveInterval' from 15 seconds to 60.

[LIGISTX]

36 minutes ago, StDragon said:

 Try changing 'ClientAliveInterval' from 15 seconds to 60.

I don't anticipate that is the issue, since the connection "broken pipe"'s while I am actively using it. But, I just edited that to see if anything changes. Aaaand, it didn't help 

 

It will happen in the middle of displaying htop info for example. 

 

Hmm, I just went back in to change the value back to 15... and it looks like it reset itself to 15...... maybe this can't be edited? I am editing and saving as root, it seemingly saves until I restart ssh service. Strange.

 

 

[StDragon]

33 minutes ago, LIGISTX said:

I don't anticipate that is the issue, since the connection "broken pipe"'s while I am actively using it.

At this point don't bother changing the value. It's only for idle timeout or non-reponse from the SSH client.

If it's interrupting and closing the connection while you're using it, I think you've got a network issue. Just to rule out any networking shenanigans going on, uplink a patch cable directly from your NIC to the NAS and see if the SSH session breaks. It rules out any potential path of networking gear between you and the NAS.

[LIGISTX]

25 minutes ago, StDragon said:

At this point don't bother changing the value. It's only for idle timeout or non-reponse from the SSH client.

If it's interrupting and closing the connection while you're using it, I think you've got a network issue. Just to rule out any networking shenanigans going on, uplink a patch cable directly from your NIC to the NAS and see if the SSH session breaks. It rules out any potential path of networking gear between you and the NAS.

It "can't" be a physical network issue. Truenas is a VM under my proxmox host, and proxmox, along with my other multiple dozen VM's and containers don't have this issue. I should have provided that info from the start... But nothing on this same subnet has issues, so I also *don't think* its a firewall issue.

 

Also, doing a ssh -vvv provides:

 

debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
debug3: send packet: type 100
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype keepalive@openssh.com reply 1
debug3: send packet: type 100
Connection to the.server.name closed by remote host.
Connection to the.server.name closed.
debug3: send packet: type 1
client_loop: send disconnect: Broken pipe

 

[StDragon]

Same issue when running an SSH client from any of the VMs on the same host vs a physical client on the same subnet?

At this point you'll need to divide-and-conquer the scope of the probem to answer the 'where' before you can address the 'how / why'.

[LIGISTX]

13 minutes ago, StDragon said:

Same issue when running an SSH client from any of the VMs on the same host vs a physical client on the same subnet?

Confirming right now via proxmox since I have a ssh user within proxmox to pull data from truenas.

 

A little more info as well: within the webUI (which seems to be acting normally), I am able to run journalctl -u ssh -f and I see this during the same time period. I don’t actually see the connection closed for ligistx from my laptop which is at 10.70.5.11. I have proxmox SSHing in to check harddrive temps for a script I run to control fans which is why you see the proxmox ssh user info.

 

Aug 27 11:42:03 thoth sshd[134727]: Accepted publickey for ligistx from 10.70.5.11 port 60968 ssh2: key
Aug 27 11:42:03 thoth sshd[134727]: pam_unix(sshd:session): session opened for user ligistx(uid=1000) by (uid=0)
Aug 27 11:42:03 thoth sshd[134727]: pam_env(sshd:session): deprecated reading of user environment enabled
Aug 27 11:43:22 thoth sshd[134766]: Accepted publickey for proxmox_ssh from 10.90.5.50 port 49666 ssh2: key
Aug 27 11:43:22 thoth sshd[134766]: pam_unix(sshd:session): session opened for user proxmox_ssh(uid=1002) by (uid=0)
Aug 27 11:43:22 thoth sshd[134766]: pam_env(sshd:session): deprecated reading of user environment enabled
Aug 27 11:43:22 thoth sshd[134766]: pam_unix(sshd:session): session closed for user proxmox_ssh```

 

[LIGISTX]

20 minutes ago, StDragon said:

Same issue when running an SSH client from any of the VMs on the same host vs a physical client on the same subnet?

Hmm.... seemingly no issue when I log in via Proxmox with the proxmox user.

 

So, I suppose this narrows it down to either a user issue, or a network issue.... Curious.

[LIGISTX]

A packet trace from pfsense specifically against my truenas IP from before I initiate the SSH connection until after the broken pipe seems to close it shows this. I am not sure if this tells anyone anything, it doesn't tell me much...

 

12:17:21.634140 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 0
12:17:21.637008 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 0
12:17:21.639594 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 21
12:17:21.656134 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 0
12:17:21.657127 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 1448
12:17:21.657134 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 120
12:17:21.659223 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 0
12:17:21.673548 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 1208
12:17:21.695845 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 0
12:17:21.700655 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 16
12:17:21.744036 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 44
12:17:21.746912 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 0
12:17:21.747911 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 68
12:17:21.761607 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 0
12:17:21.767348 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 276
12:17:21.773088 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 0
12:17:21.793005 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 524
12:17:21.804822 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 0
12:17:21.805338 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 112
12:17:21.853562 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 0
12:17:21.856357 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 0
12:17:21.888105 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 596
12:17:21.892779 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 0
12:17:21.953320 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 0
12:17:21.954617 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 0
12:17:51.985622 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 0
12:17:51.986617 IP 10.70.5.11.61147 > 10.90.5.100.22: tcp 36
12:19:31.529415 IP 10.90.5.100.57410 > 23.186.168.127.123: UDP, length 48
12:19:36.681172 ARP, Request who-has 10.90.5.1 tell 10.90.5.100, length 46
12:19:36.681197 ARP, Reply 10.90.5.1 is-at 00:26:55:e1:78:d4, length 28

 

[StDragon]

If SSH traffic is traversing through the pfsense FW, that could be it. Here's a post for OPNsense. Not the same thing, but close to where it could be a FW optimization setting.

https://forum.opnsense.org/index.php?topic=27848.0

"try to set Firewall Optimization (Firewall -> Settings -> Advanced) to conservative ..." -zerwes

Also this

[Lurick]

6 minutes ago, StDragon said:

If SSH traffic is traversing through the pfsense FW, that could be it. Here's a post for OPNsense. Not the same thing, but close to where it could be a FW optimization setting.

https://forum.opnsense.org/index.php?topic=27848.0

"try to set Firewall Optimization (Firewall -> Settings -> Advanced) to conservative ..." -zerwes

If not this, or MTU issues

I just checked my /etc/ssh/sshd_config

Subsystem       sftp    internal-sftp -l ERROR -f AUTH
Protocol 2
UseDNS no
ChallengeResponseAuthentication no
ClientAliveCountMax 3
ClientAliveInterval 15
VersionAddendum none
Port 22
PermitRootLogin without-password
AllowTcpForwarding no
Compression no
PasswordAuthentication no
PubkeyAuthentication yes

# These are forced to be enabled with 2FA
UsePAM yes
PrintMotd no
SetEnv LC_ALL=C.UTF-8

I don't have any of the Listen addresses, you could try to take those three lines of listen addresses out and then it would listen on all interfaces. I wonder if that's causing some issue, especially with the IPv6 link local listen address.

[Lurick]

I had an issue recently with an IPSec tunnel from my opnsense box to a remote location where it wasn't setting the MTU properly until I made a change so I would get the broken pipe error and random timeouts after a minute or so. If you're going through pfsense or opnsense maybe it's worth checking the MTU values all align or maybe setting the TrueNAS MTU just a tad lower for a test and see if that helps.

[LIGISTX]

Ah, figured it out with some help from someone on the truenas forum... 

 

I had SSH bound to my management interface, but I had an interface on the 10.70 subnet my macbook is on for SMB shares to reduce load across the firewall... Replies were going out over the 10.70 subnet so the connection was timing out. I bound SSH to both managmeent and the 10.70 (only my personal devices are on this network anyways, and there is already a firewall rule to allow those devices to talk to my management network) and now when I SSH to the 10.70 interface, it all works fine. 

 

This shows my ignorance, I didn't realize binding SSH to an interface wouldn't also restrict its replies to the same interface...