Jump to content

10GBase-T Peer to Peer Configuration | Solvedish

N3rdDadTV
By N3rdDadTV
in Networking
 Share
Posted
31 minutes ago, N3rdDadTV said:

I'm starting to think I may have a hardware issue, my most recent attempts have had me creating static routes using netstat through powershell with still no luck. I can get the cards to ping each other but not much more than that.

That's the thing, if they can ping each other that suggests hardware is fine and network stack is working as expected, leaving the Windows firewall as the primary suspect.  Have you tried disabling the firewall entirely?

Router:  Intel N100 (pfSense) WiFi6: Zyxel NWA210AX (1.7Gbit peak at 160Mhz)
WiFi5: Ubiquiti NanoHD OpenWRT (~500Mbit at 80Mhz) Switches: Netgear MS510TXUP, MS510TXPP, GS110EMX
ISPs: Zen Full Fibre 900 (~930Mbit down, 115Mbit up) + Three 5G (~800Mbit down, 115Mbit up)
Upgrading Laptop/Desktop CNVIo WiFi 5 cards to PCIe WiFi6e/7

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
13 hours ago, N3rdDadTV said:

I'm starting to think I may have a hardware issue, my most recent attempts have had me creating static routes using netstat through powershell with still no luck. I can get the cards to ping each other but not much more than that.

Just as a note no host file modifications, static routes or anything extra is required to get what you are wanting working. Make sure you revert anything like that as it'll more than likely cause you problems than help your situation. I have this setup style you are wanting to do.

 

From what it sounds at one point you had what is required configured correctly; 10Gb NICs on their own not overlapping subnets. That's literally all that is required on the network and IP configuration side of things.

 

Once you're back to this configuration check the Sent and Received packet counts on the NIC status

image.png.e26b10138792daec234f057f27770955.png

 

You should see both with numbers greater than zero, even without ping or SMB access working.

 

Then to rule out firewall change the default action for Inbound connections on all 3 network profile types to Allow. This is better and more reliable than disabling the firewall, at least I have found anyway. Sometimes disabling it doesn't fully disable it, silly Windows.

image.png.05d8b8f7b4fc67f669b809f9b8f21fb0.png

Not shown because my GPO doesn't allow me to change to Allow, haha oh well.

 

Then I'd install Wireshark on both computers and do some ping tests between them with a Wireshark capture running filtering only for ICMP traffic.

 

P.S. Yea saw your merch message on WAN show 🙂

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

Thank you so much for your help, WAN show wins again. I reset everything back and just assigned my 2.5 nics to 192.xx and 10G nics to 10.xx and I'm back to where I started. Before making those changes to the firewall policies I checked the activity it was substantial and I could ping the other workstation. Now it's minuscule and I can't ping the other workstation. Any other ideas? 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author
13 hours ago, Alex Atkin UK said:

That's the thing, if they can ping each other that suggests hardware is fine and network stack is working as expected, leaving the Windows firewall as the primary suspect.  Have you tried disabling the firewall entirely?

I turned off the firewall completely on all networks as one of the first steps, after seeing the comment under yours I turned them back on and allowed inbound traffic on all networks after deleting the static routes and now there's no ping and less connectivity. 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
29 minutes ago, N3rdDadTV said:

I could ping the other workstation. Now it's minuscule and I can't ping the other workstation. Any other ideas? 

So Computer 1 is able to ping Computer 2 10Gb IP address but Computer 2 is unable to ping Computer 1 10Gb IP address?

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

No, before I was able to ping each of them but no file transfer or Jperf connection. 

That was after I added the static routes. After I removed the static routes and made the changes to the firewall now there's no connection but activity is up. 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
25 minutes ago, N3rdDadTV said:

No, before I was able to ping each of them but no file transfer or Jperf connection. 

That was after I added the static routes. After I removed the static routes and made the changes to the firewall now there's no connection but activity is up. 

Could you show me the output of the 'route print' command in cmd?

 

Here is what mine looks like 

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0      10.4.10.254       10.4.10.18    281
        10.4.10.0    255.255.255.0         On-link        10.4.10.18    281
       10.4.10.18  255.255.255.255         On-link        10.4.10.18    281
      10.4.10.255  255.255.255.255         On-link        10.4.10.18    281
        127.0.0.0        255.0.0.0         On-link         127.0.0.1    331
        127.0.0.1  255.255.255.255         On-link         127.0.0.1    331
  127.255.255.255  255.255.255.255         On-link         127.0.0.1    331
      169.254.0.0      255.255.0.0         On-link   169.254.145.161    271
  169.254.145.161  255.255.255.255         On-link   169.254.145.161    271
  169.254.255.255  255.255.255.255         On-link   169.254.145.161    271
      192.168.1.0    255.255.255.0         On-link       192.168.1.3    271
      192.168.1.3  255.255.255.255         On-link       192.168.1.3    271
    192.168.1.255  255.255.255.255         On-link       192.168.1.3    271
        224.0.0.0        240.0.0.0         On-link         127.0.0.1    331
        224.0.0.0        240.0.0.0         On-link        10.4.10.18    281
        224.0.0.0        240.0.0.0         On-link       192.168.1.3    271
        224.0.0.0        240.0.0.0         On-link   169.254.145.161    271
  255.255.255.255  255.255.255.255         On-link         127.0.0.1    331
  255.255.255.255  255.255.255.255         On-link        10.4.10.18    281
  255.255.255.255  255.255.255.255         On-link       192.168.1.3    271
  255.255.255.255  255.255.255.255         On-link   169.254.145.161    271
===========================================================================
Persistent Routes:
  Network Address          Netmask  Gateway Address  Metric
          0.0.0.0          0.0.0.0      10.4.10.254  Default

 

My 10.4.10.x network is 1Gb and my 192.168.1.x network is my 10Gb.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
2 hours ago, N3rdDadTV said:

 

image.png

There is still a persistent route that has been added, the 10.10.10.1 and it's not a correct route entry either what would work.

 

2 hours ago, N3rdDadTV said:

image.png.f968eeef260cef02798054bf0e63f384.png

Same with this one however there are two persistent routes.

 

You should only have the single 0.0.0.0 default gateway route listed when doing route print.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
2 hours ago, N3rdDadTV said:

image.png.f968eeef260cef02798054bf0e63f384.png

Can we see ipconfig output too as its odd 192.168.2.0 is not showing up anywhere and where is 10.10.10.0 coming from and why is it such a large subnet?

Router:  Intel N100 (pfSense) WiFi6: Zyxel NWA210AX (1.7Gbit peak at 160Mhz)
WiFi5: Ubiquiti NanoHD OpenWRT (~500Mbit at 80Mhz) Switches: Netgear MS510TXUP, MS510TXPP, GS110EMX
ISPs: Zen Full Fibre 900 (~930Mbit down, 115Mbit up) + Three 5G (~800Mbit down, 115Mbit up)
Upgrading Laptop/Desktop CNVIo WiFi 5 cards to PCIe WiFi6e/7

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

You're using a /24 subnet on one PC and /16 subnet on the other.

They should be both /24 [255.255.255.0]

 

You're doing it for both NICs. Or am I reading this wrong?

Please share "ipconfig /all" to confirm your subnet mask.

 

 

 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
1 hour ago, PyCCo_TyPuCTo said:

You're using a /24 subnet on one PC and /16 subnet on the other.

They should be both /24 [255.255.255.0]

Nice spotting, that'll be the cause, due to one of them using 10.10.10.x rather than 10.10.x.x where the subnet mask mismatch wouldn't have mattered (unless 3rd octet wasn't a zero ofc).

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

I made the changes you all suggested and here's the config, thank you all again for your help. 

image.png.c6a699b152189ecc29a82556c9e89fc4.png

image.png.5c1e4314b860bd8d5a07ff5e435219e0.png

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
40 minutes ago, N3rdDadTV said:

 

image.png.c3d01d10d73cfeccf39114cecaf93689.png

There is still an active bad route, 10.10.0.0 gw 10.10.10.1. You'll probably have to restart the computer for it to go away.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

Back to where I started, able to ping the other card but can't route traffic through them. Connection times out on either machine when alternating client/server. Workstations don't connect through windows filesharing through these nics, only when file and print sharing is on the 2.5 nics. 

 

C:\Users\nicho>ping 10.10.10.2

Pinging 10.10.10.2 with 32 bytes of data:
Reply from 10.10.10.2: bytes=32 time<1ms TTL=128
Reply from 10.10.10.2: bytes=32 time<1ms TTL=128
Reply from 10.10.10.2: bytes=32 time<1ms TTL=128
Reply from 10.10.10.2: bytes=32 time<1ms TTL=128

Ping statistics for 10.10.10.2:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms

C:\Users\nicho>ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : NFPMain
   Primary Dns Suffix  . . . . . . . :
   Node Type . . . . . . . . . . . . : Mixed
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No

Ethernet adapter 10Gig:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : ASUS XG-C100C 10G PCI-E Network Adapter
   Physical Address. . . . . . . . . : 04-42-1A-57-D4-E9
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::6cd9:f0d:c13e:657c%13(Preferred)
   IPv4 Address. . . . . . . . . . . : 10.10.10.1(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . :
   DHCPv6 IAID . . . . . . . . . . . : 218382874
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-28-13-04-9F-F0-2F-74-D2-23-84
   DNS Servers . . . . . . . . . . . : fec0:0:0:ffff::1%1
                                       fec0:0:0:ffff::2%1
                                       fec0:0:0:ffff::3%1
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter 2.5Gig:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) Ethernet Controller (3) I225-V
   Physical Address. . . . . . . . . : D8-BB-C1-45-2B-31
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv6 Address. . . . . . . . . . . : 2603:8080:3800:603::e5e(Preferred)
   Lease Obtained. . . . . . . . . . : Monday, May 2, 2022 8:23:09 AM
   Lease Expires . . . . . . . . . . : Monday, May 9, 2022 5:10:46 AM
   IPv6 Address. . . . . . . . . . . : 2603:8080:3800:603:24af:bfc7:706b:6d40(Preferred)
   Temporary IPv6 Address. . . . . . : 2603:8080:3800:603:3d59:a10b:e4d8:ecd1(Preferred)
   Link-local IPv6 Address . . . . . : fe80::24af:bfc7:706b:6d40%20(Preferred)
   IPv4 Address. . . . . . . . . . . : 192.168.1.5(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.0.0
   Default Gateway . . . . . . . . . : fe80::9a1e:19ff:fe66:241a%20
                                       192.168.1.1
   DHCPv6 IAID . . . . . . . . . . . : 702069697
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-28-13-04-9F-F0-2F-74-D2-23-84
   DNS Servers . . . . . . . . . . . : 2001:1998:f00:2::1
                                       2001:1998:f00:1::1
                                       192.168.1.1
                                       2001:1998:f00:2::1
                                       2001:1998:f00:1::1
   NetBIOS over Tcpip. . . . . . . . : Enabled

C:\Users\nicho>route print
===========================================================================
Interface List
 13...04 42 1a 57 d4 e9 ......ASUS XG-C100C 10G PCI-E Network Adapter
 20...d8 bb c1 45 2b 31 ......Intel(R) Ethernet Controller (3) I225-V
  1...........................Software Loopback Interface 1
===========================================================================

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0      192.168.1.1      192.168.1.5    281
       10.10.10.0    255.255.255.0         On-link        10.10.10.1    271
       10.10.10.1  255.255.255.255         On-link        10.10.10.1    271
     10.10.10.255  255.255.255.255         On-link        10.10.10.1    271
        127.0.0.0        255.0.0.0         On-link         127.0.0.1    331
        127.0.0.1  255.255.255.255         On-link         127.0.0.1    331
  127.255.255.255  255.255.255.255         On-link         127.0.0.1    331
      192.168.0.0      255.255.0.0         On-link       192.168.1.5    281
      192.168.1.5  255.255.255.255         On-link       192.168.1.5    281
  192.168.255.255  255.255.255.255         On-link       192.168.1.5    281
        224.0.0.0        240.0.0.0         On-link         127.0.0.1    331
        224.0.0.0        240.0.0.0         On-link        10.10.10.1    271
        224.0.0.0        240.0.0.0         On-link       192.168.1.5    281
  255.255.255.255  255.255.255.255         On-link         127.0.0.1    331
  255.255.255.255  255.255.255.255         On-link        10.10.10.1    271
  255.255.255.255  255.255.255.255         On-link       192.168.1.5    281
===========================================================================
Persistent Routes:
  Network Address          Netmask  Gateway Address  Metric
          0.0.0.0          0.0.0.0      192.168.1.1  Default
===========================================================================

IPv6 Route Table
===========================================================================
Active Routes:
 If Metric Network Destination      Gateway
 20    281 ::/0                     fe80::9a1e:19ff:fe66:241a
  1    331 ::1/128                  On-link
 20    281 2603:8080:3800:603::/64  On-link
 20    281 2603:8080:3800:603::/64  fe80::9a1e:19ff:fe66:241a
 20    281 2603:8080:3800:603::e5e/128
                                    On-link
 20    281 2603:8080:3800:603:24af:bfc7:706b:6d40/128
                                    On-link
 20    281 2603:8080:3800:603:3d59:a10b:e4d8:ecd1/128
                                    On-link
 13    271 fe80::/64                On-link
 20    281 fe80::/64                On-link
 20    281 fe80::24af:bfc7:706b:6d40/128
                                    On-link
 13    271 fe80::6cd9:f0d:c13e:657c/128
                                    On-link
  1    331 ff00::/8                 On-link
 13    271 ff00::/8                 On-link
 20    281 ff00::/8                 On-link
===========================================================================
Persistent Routes:
  None

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author
Just now, leadeater said:

There is still an active bad route, 10.10.0.0 gw 10.10.10.1. You'll probably have to restart the computer for it to go away.

The last post was after the restart, I noticed that after I posted and on to making the previous post. 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
4 minutes ago, N3rdDadTV said:

The last post was after the restart, I noticed that after I posted and on to making the previous post. 

I'd install Wireshark on both computers and run a capture on both before you try and connect to a share then go through the process. That should give a better insight in to what is going on.

 

If both computers can ping each other then the network configuration side of things is actually fine. Either Windows isn't listening to SMB connections on the 10Gb interfaces or Windows Firewall is blocking it, something along those lines.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

This is my first experience with wireshark so I'm not sure what I'm looking at but I think this is promising?

image.thumb.png.6022b2cfa518ce2a08906f3e716bd656.png

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

After looking at this I don't feel so confident. 

image.thumb.png.2585b81495e602961bd6749f7fdab188.png

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
2 hours ago, N3rdDadTV said:

This is my first experience with wireshark so I'm not sure what I'm looking at but I think this is promising?

Filter out anything destination 10.10.10.255, however seeing so much of that type of traffic can be an indicator of an issue. There shouldn't be all that broadcast traffic going on. Either way filter it out, will likely see something more useful without it.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

image.thumb.png.29d784dca43e4bd6c67d5dd7493bc4a4.png

Here's the I/O Graph in Bits for the 10g Nic on A

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

Here's the graph with the .255 exception. 

image.thumb.png.a400b1191006526fd87cba4eec95997d.png

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Okay, I literally dug an extra NIC out and put it in my desktop to test this, because this shouldn't be this hard.

 

Steps:

  1. Install NIC in computer #1
  2. Set IPv4 address to 10.10.1.2
  3. Install NIC in computer #2
  4. Set IPv4 address to 10.10.1.3
  5. Connect #1 to #2 using Ethernet cable
  6. Run "ping 10.10.1.2" from #2; successful
  7. Run "ping 10.10.1.3" from #1; failed
    1. Open firewall for "Echo Request - ICMPv4-In"
    2. Successful
  8. Share folder on #1; access from #2 via "\\10.10.1.2" in Explorer; successful
  9. Share folder on #2; access from #1 via "\\10.10.1.3" in Explorer; successful

Are you actually sharing any folders? I assume you are, and Windows will still show an empty folder when you try to navigate to it, but I figured I should also check.

Main System (Byarlant): Ryzen 7 5800X | Asus B550-Creator ProArt | EK 240mm Basic AIO | 16GB G.Skill DDR4 3200MT/s CAS-14 | XFX Speedster SWFT 210 RX 6600 | Samsung 990 PRO 2TB / Samsung 960 PRO 512GB / 4× Crucial MX500 2TB (RAID-0) | Corsair RM750X | a 10G NIC (pending) | Inateck USB 3.0 Card | Hyte Y60 Case | Dell U3415W Monitor | Keychron K4 Brown (white backlight)

 

Laptop (Narrative): Lenovo Flex 5 81X20005US | Ryzen 5 4500U | 16GB RAM (soldered) | Vega 6 Graphics | SKHynix P31 1TB NVMe SSD | Intel AX200 Wifi (all-around awesome machine)

 

Proxmox Server (Veda): Ryzen 7 3800XT | AsRock Rack X470D4U | Corsair H80i v2 | 64GB Micron DDR4 ECC 3200MT/s | 4x 10TB WD Whites / 4x 14TB Seagate Exos / 2× Samsung PM963a 960GB SSD | Seasonic Prime Fanless 500W | Intel X540-T2 10G NIC | LSI 9207-8i HBA | Fractal Design Node 804 Case (side panels swapped to show off drives) | VMs: TrueNAS Scale; Ubuntu Server (PiHole/PiVPN/NGINX?); Windows 10 Pro; Ubuntu Server (Apache/MySQL)


Media Center/Video Capture (Jesta Cannon): Ryzen 5 1600X | ASRock B450M Pro4 R2.0 | Noctua NH-L12S | 16GB Crucial DDR4 3200MT/s CAS-22 | EVGA GTX750Ti SC | UMIS NVMe SSD 256GB / TEAMGROUP MS30 1TB | Corsair CX450M | Viewcast Osprey 260e Video Capture | Mellanox ConnectX-2 10G NIC | LG UH12NS30 BD-ROM | Silverstone Sugo SG-11 Case | Sony XR65A80K

 

Camera: Sony ɑ7II w/ Meike Grip | Sony SEL24240 | Samyang 35mm ƒ/2.8 | Sony SEL50F18F | Sony SEL2870 (kit lens) | PNY Elite Perfomance 512GB SDXC card

 

Network:

Spoiler 
                           ┌─────────────── Office/Rack ────────────────────────────────────────────────────────────────────────────┐
Google Fiber Webpass ────── UniFi Security Gateway ─── UniFi Switch 8-60W ─┬─ UniFi Switch Flex XG ═╦═ Veda (Proxmox Virtual Switch)
(500Mbps↑/500Mbps↓)                             UniFi CloudKey Gen2 (PoE) ─┴─ Veda (IPMI)           ╠═ Veda-NAS (HW Passthrough NIC)
╔═══════════════════════════════════════════════════════════════════════════════════════════════════╩═ Narrative (Asus USB 2.5G NIC)
║ ┌────── Closet ──────┐   ┌─────────────── Bedroom ──────────────────────────────────────────────────────┐
╚═ UniFi Switch Flex XG ═╤═ UniFi Switch Flex XG ═╦═ Byarlant
   (PoE)                 │                        ╠═ Narrative (Cable Matters USB-PD 2.5G Ethernet Dongle)
                         │                        ╚═ Jesta Cannon*
                         │ ┌─────────────── Media Center ──────────────────────────────────┐
Notes:                   └─ UniFi Switch 8 ─────────┬─ UniFi Access Point nanoHD (PoE)
═══ is Multi-Gigabit                                ├─ Sony Playstation 4 
─── is Gigabit                                      ├─ Pioneer VSX-S520
* = cable passed to Bedroom from Media Center       ├─ Sony XR65A80K (Google TV)
** = cable passed from Media Center to Bedroom      └─ Work Laptop** (Startech USB-PD Dock)

Retired/Other:

Spoiler

Laptop (Rozen-Zulu): Sony VAIO VPCF13WFX | Core i7-740QM | 8GB Patriot DDR3 | GT 425M | Samsung 850EVO 250GB SSD | Blu-ray Drive | Intel 7260 Wifi (lived a good life, retired with honor)

Testbed/Old Desktop (Kshatriya): Xeon X5470 @ 4.0GHz | ZALMAN CNPS9500 | Gigabyte EP45-UD3L | 8GB Nanya DDR2 400MHz | XFX HD6870 DD | OCZ Vertex 3 Max-IOPS 120GB | Corsair CX430M | HooToo USB 3.0 PCIe Card | Osprey 230 Video Capture | NZXT H230 Case

TrueNAS Server (La Vie en Rose): Xeon E3-1241v3 | Supermicro X10SLL-F | Corsair H60 | 32GB Micron DDR3L ECC 1600MHz | 1x Kingston 16GB SSD / Crucial MX500 500GB

Link to comment
Share on other sites
Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing Networking

×