Server Drops Packets Periodically

[JJ12415]

G'day everyone,

 

I’m troubleshooting an issue with my home server. The server exhibits periodic packet drops that last between 10-30 seconds. This happens every 5-10 minutes with no obvious pattern. During these drops, network connections sometimes time out or experience significant delays. I’ve investigated multiple potential causes, but I’m getting nowhere.

 

System Information:

    Server: Dell PowerEdge R620
    OS: AlmaLinux 9.5 (Kernel 5.14.0)
    CPU: Dual Intel Xeon E5-2650 v2
    Primary NIC: Intel I350 (onboard) (eno1/2/3/4)
    Secondary NIC: TP-Link TG-3468 (Realtek RTL8168 chipset) (enp66s0)
    Driver for Intel NIC: igb
    Driver for TP-Link NIC: r8169

 

Symptoms:

•     Periodic network packet drops (lasting 10-30+ seconds) every 5–10 minutes, tested in multiple different ways.
•     Connections often time out, most apparent with tasks like gaming servers, Nextcloud uploads, etc.
•     Things like SSH (eg. from laptop or phone to server) will stop displaying input or output, appearing frozen. Typing while in this state will essentially pop up once connection resumes.
•     Packet drops occur regardless of which NIC is used (both the onboard Intel I350 and the TP-Link card exhibit the same behavior).
•     /proc/interrupts shows that NIC-related IRQs are being handled by a single CPU core, even after adjusting smp_affinity (however this is somewhat beyond my knowledge/expertise, so I may misunderstand what I've read so far).

 

Steps Taken So Far:

 

1. Energy-Efficient Ethernet (EEE):

    I disabled EEE on both NICs using ethtool, as I’ve read this can cause intermittent connectivity issues on some systems.

   

ethtool --set-eee eno1 eee off

 

    Confirmed that EEE is disabled, but the issue persists.

 

2. Coalescing Settings:

    During my research, I found reference to coalescing and IRQ

   

    I attempted to adjust RX and TX interrupt coalescing using ethtool:

ethtool -C eno1 rx-usecs 50 tx-usecs 50  

    However, this failed with the error:

    netlink error: Invalid argument  

   

    It seems that coalescing is not fully supported by the NIC or driver in this setup (but again, I'm unfamiliar with coalescing/IRQ).

 

3. IRQ Affinity

    I manually adjusted the smp_affinity of the NIC’s IRQ to allow all CPU cores to handle the interrupt:

   

echo "ffffffff" | sudo tee /proc/irq/<IRQ_NUMBER>/smp_affinity  

   

    While the change reflects in the smp_affinity file, interrupts remain confined to one core as observed in /proc/interrupts.

  
   

cat /proc/interrupts
63: 0 0 0 0 0 33 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1258018 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSIX-0000:42:00.0 0-edge  enp66s0

 

   

    And the corresponding affinity mask:

   

cat /proc/irq/63/smp_affinity
0000,00200000

 

    Despite setting the affinity mask to ffffffff, interrupts still appear to be handled by a single core.

 

4. Tested Alternate NIC

    I added a TP-Link TG-3468 (Realtek RTL8168), but the same issue persists, which leads me to believe this may not be NIC-specific.

 

5. IRQ Balancing

    Checked if irqbalance was interfering with manual settings. Stopped the service to test:

   

sudo systemctl stop irqbalance

 

    No improvement after stopping irqbalance.

 

Questions:

    Interrupt Behavior: Is it normal for interrupts to remain confined to one core, even after setting a broader smp_affinity mask? Could this indicate a hardware or driver limitation?
    Packet Drops: Are there other common causes of periodic packet drops on Linux servers that I may have overlooked?
    Other Suggestions: Are there any other diagnostics or configurations I should try?

 

I’m open to any suggestions or ideas you might have. I’d obviously love to resolve this issue, as the periodic packet drops make a lot of the tasks handled by the server pointless (or at the very least, inconvenient).

 

TIA!