Connecting two NC523SFP 10GbE Fiber issues

[Dhalion]

Hello guys,

 

I need some help setting up my 10GbE P2P

 

So I got two NC523SFP off Ebay, one is in my trueNAS Server and one in my Win 10 Desktop. They are connected with two 455883-B21.

The one in the NAS system showed up in config with no prior driver installations - no problems. For the Windows side i got the official HP Windows Server Driver, extracted the .exe and installed the drivers manually - this apparently worked as they got recognized as Network adapters, no problems here either. At least I think so lol

 

Only thing is, that they claim there's no network cable attached - however this might be because since theres no DHCP and no static IPs set?

 

I did set static IPs and Subnet and all on both sides, but this did not give me a connection or any visible change.

 

Also note that truenas has the Link State "DOWN" - do I have to UP it manually?

And is there any way how I can tell the two ports apart? Which one is ql0 and which is ql1?

 

Also also the lights on the NICs are not blinking whatsoever - is this a sign that something went wrong with the drivers or did they overheat? (Any way i can check their temperature?)

 

 

So any help is appreciated, did I do something wrong or did I miss anything? Do I have to update the NICs Firmware?

 

 

Cheers

 

 

[kasugatei]

18 hours ago, Dhalion said:

snip

Could be bad/incompatible SFP module or bad/incompatible cable. 

First easy test. What color is your optical cable? If it is yellow then you probably have single mode cable on multi mode SFP which will not work.

I personally work with Mellanox and Intel mostly, I never used Qlogic NICs, but I found this post:

https://www.robvit.com/hp-hardware/how-to-check-the-sfp-state-on-a-physical-windows-host/

 

Try that with both SFPs on your Windows machine to check the status of the SFPs.

[Dhalion]

On 2/26/2021 at 7:34 AM, Niksa said:

Could be bad/incompatible SFP module or bad/incompatible cable. 

First easy test. What color is your optical cable? If it is yellow then you probably have single mode cable on multi mode SFP which will not work.

I personally work with Mellanox and Intel mostly, I never used Qlogic NICs, but I found this post:

https://www.robvit.com/hp-hardware/how-to-check-the-sfp-state-on-a-physical-windows-host/

 

Try that with both SFPs on your Windows machine to check the status of the SFPs.

Thanks for your reply. So my fiber cable is blue..ish. It's a "3m LC UPC to LC UPC Duplex OM3 Multimode PVC (OFNR) 2.0mm" precisely.

The two Modules are "455883-B21 HPE BladeSystem c-Class kompatibel 10GBASE-SR SFP+ 850nm 300m LC MMF". According to the NICs Datasheet these are compatible.

 

I didn't get your tool working, but I set up a Second Test system running Linux and there I got the following issues anyways:

Setting up a Static IP ends with an Error "SIOCSIFFLAGS: Input/ Output Error". Same with bringing them up.

 

A quick look in syslog gives me this:

chris@chris-G31M-S2L:~$ cat /var/log/syslog | grep -i enp1
Mar  3 16:41:26 chris-G31M-S2L kernel: [    4.023634] qlcnic 0000:01:00.1 enp1s0f1: renamed from eth1
Mar  3 16:41:26 chris-G31M-S2L kernel: [    4.038809] qlcnic 0000:01:00.0 enp1s0f0: renamed from eth0
Mar  3 16:41:36 chris-G31M-S2L NetworkManager[503]: <info>  [1614786096.9760] manager: (enp1s0f0): new Ethernet device (/org/freedesktop/NetworkManager/Devices/2)
Mar  3 16:41:37 chris-G31M-S2L NetworkManager[503]: <info>  [1614786097.3126] settings: (enp1s0f0): created default wired connection 'Kabelgebundene Verbindung 1'
Mar  3 16:41:37 chris-G31M-S2L NetworkManager[503]: <info>  [1614786097.3572] device (enp1s0f0): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Mar  3 16:41:37 chris-G31M-S2L kernel: [   35.197061] qlcnic 0000:01:00.0 enp1s0f0: Rx Context[0] Created, state 0x2
Mar  3 16:41:37 chris-G31M-S2L kernel: [   35.507914] qlcnic 0000:01:00.0 enp1s0f0: Tx Context[0x8000] Created, state 0x2
Mar  3 16:41:37 chris-G31M-S2L kernel: [   35.521854] qlcnic 0000:01:00.0 enp1s0f0: Tx Context[0x8008] Created, state 0x2
Mar  3 16:41:38 chris-G31M-S2L NetworkManager[503]: <info>  [1614786098.0258] manager: (enp1s0f1): new Ethernet device (/org/freedesktop/NetworkManager/Devices/3)
Mar  3 16:41:38 chris-G31M-S2L NetworkManager[503]: <info>  [1614786098.0288] settings: (enp1s0f1): created default wired connection 'Kabelgebundene Verbindung 2'
Mar  3 16:41:38 chris-G31M-S2L NetworkManager[503]: <info>  [1614786098.0300] device (enp1s0f1): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Mar  3 16:41:38 chris-G31M-S2L kernel: [   36.070315] qlcnic 0000:01:00.1 enp1s0f1: Rx Context[1] Created, state 0x2
Mar  3 16:41:38 chris-G31M-S2L kernel: [   36.531166] qlcnic 0000:01:00.1 enp1s0f1: Tx Context[0x8001] Created, state 0x2
Mar  3 16:41:38 chris-G31M-S2L kernel: [   36.545561] qlcnic 0000:01:00.1 enp1s0f1: Tx Context[0x8009] Created, state 0x2
Mar  3 16:50:15 chris-G31M-S2L kernel: [    4.062797] qlcnic 0000:01:00.0 enp1s0f0: renamed from eth0
Mar  3 16:50:15 chris-G31M-S2L kernel: [    4.074925] qlcnic 0000:01:00.1 enp1s0f1: renamed from eth1
Mar  3 16:50:26 chris-G31M-S2L NetworkManager[501]: <info>  [1614786626.6927] manager: (enp1s0f0): new Ethernet device (/org/freedesktop/NetworkManager/Devices/2)
Mar  3 16:50:27 chris-G31M-S2L NetworkManager[501]: <info>  [1614786627.0257] settings: (enp1s0f0): created default wired connection 'Kabelgebundene Verbindung 1'
Mar  3 16:50:27 chris-G31M-S2L NetworkManager[501]: <info>  [1614786627.0778] device (enp1s0f0): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Mar  3 16:50:27 chris-G31M-S2L kernel: [   35.235760] qlcnic 0000:01:00.0 enp1s0f0: Rx Context[0] Created, state 0x2
Mar  3 16:50:27 chris-G31M-S2L kernel: [   35.546074] qlcnic 0000:01:00.0 enp1s0f0: Tx Context[0x8000] Created, state 0x2
Mar  3 16:50:27 chris-G31M-S2L kernel: [   35.561009] qlcnic 0000:01:00.0 enp1s0f0: Tx Context[0x8008] Created, state 0x2
Mar  3 16:50:28 chris-G31M-S2L NetworkManager[501]: <info>  [1614786628.0594] manager: (enp1s0f1): new Ethernet device (/org/freedesktop/NetworkManager/Devices/3)
Mar  3 16:50:28 chris-G31M-S2L NetworkManager[501]: <info>  [1614786628.0655] settings: (enp1s0f1): created default wired connection 'Kabelgebundene Verbindung 2'
Mar  3 16:50:28 chris-G31M-S2L NetworkManager[501]: <info>  [1614786628.0685] device (enp1s0f1): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Mar  3 16:50:28 chris-G31M-S2L kernel: [   36.109890] qlcnic 0000:01:00.1 enp1s0f1: Rx Context[1] Created, state 0x2
Mar  3 16:50:28 chris-G31M-S2L kernel: [   36.569959] qlcnic 0000:01:00.1 enp1s0f1: Tx Context[0x8001] Created, state 0x2
Mar  3 16:50:28 chris-G31M-S2L kernel: [   36.584914] qlcnic 0000:01:00.1 enp1s0f1: Tx Context[0x8009] Created, state 0x2
Mar  3 16:55:00 chris-G31M-S2L kernel: [  308.944100] net enp1s0f0: Device temperature 102 degrees C exceeds operating range. Immediate action needed.
Mar  3 16:55:00 chris-G31M-S2L kernel: [  308.944103] net enp1s0f1: Device temperature 102 degrees C exceeds operating range. Immediate action needed.
Mar  3 16:56:25 chris-G31M-S2L kernel: [  393.921546] net enp1s0f1: Device temperature 108 degrees C exceeds maximum allowed. Hardware has been shut down.
Mar  3 16:56:25 chris-G31M-S2L kernel: [  393.921557] net enp1s0f0: Device temperature 108 degrees C exceeds maximum allowed. Hardware has been shut down.
Mar  3 16:56:41 chris-G31M-S2L kernel: [  409.813352] qlcnic 0000:01:00.1 enp1s0f1: Failed to Delete interrupts 17
Mar  3 16:56:57 chris-G31M-S2L kernel: [  425.728261] qlcnic 0000:01:00.0 enp1s0f0: Failed to Delete interrupts 17
Mar  3 19:12:42 chris-G31M-S2L kernel: [ 8570.338685] qlcnic 0000:01:00.0 sfp0: renamed from enp1s0f0
Mar  3 19:12:42 chris-G31M-S2L NetworkManager[501]: <info>  [1614795162.4818] device (enp1s0f0): interface index 3 renamed iface from 'enp1s0f0' to 'sfp0'
Mar  3 19:12:48 chris-G31M-S2L kernel: [ 8576.330631] qlcnic 0000:01:00.1 sfp1: renamed from enp1s0f1
Mar  3 19:12:48 chris-G31M-S2L NetworkManager[501]: <info>  [1614795168.4974] device (enp1s0f1): interface index 4 renamed iface from 'enp1s0f1' to 'sfp1'
Mar  3 19:19:37 chris-G31M-S2L kernel: [   15.110471] qlcnic 0000:01:00.0 enp1s0f0: renamed from eth0
Mar  3 19:19:37 chris-G31M-S2L kernel: [   15.127121] qlcnic 0000:01:00.1 enp1s0f1: renamed from eth1
Mar  3 19:19:37 chris-G31M-S2L kernel: [   16.970928] net enp1s0f0: Device temperature 123 degrees C exceeds maximum allowed. Hardware has been shut down.
Mar  3 19:19:37 chris-G31M-S2L kernel: [   16.970931] net enp1s0f1: Device temperature 123 degrees C exceeds maximum allowed. Hardware has been shut down.
Mar  3 19:19:49 chris-G31M-S2L NetworkManager[501]: <info>  [1614795589.7690] manager: (enp1s0f0): new Ethernet device (/org/freedesktop/NetworkManager/Devices/2)
Mar  3 19:19:49 chris-G31M-S2L NetworkManager[501]: <info>  [1614795589.8251] device (enp1s0f0): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Mar  3 19:19:53 chris-G31M-S2L kernel: [   50.528754] qlcnic 0000:01:00.0 enp1s0f0: Failed to Add interrupts 17
Mar  3 19:19:55 chris-G31M-S2L NetworkManager[501]: <info>  [1614795593.9092] manager: (enp1s0f1): new Ethernet device (/org/freedesktop/NetworkManager/Devices/3)
Mar  3 19:19:55 chris-G31M-S2L NetworkManager[501]: <info>  [1614795594.0342] settings: (enp1s0f1): created default wired connection 'Kabelgebundene Verbindung 1'
Mar  3 19:19:55 chris-G31M-S2L NetworkManager[501]: <info>  [1614795594.0352] device (enp1s0f1): state change: unmanaged -> unavailable (reason 'managed', sys-iface-state: 'external')
Mar  3 19:19:58 chris-G31M-S2L kernel: [   54.740701] qlcnic 0000:01:00.1 enp1s0f1: Failed to Add interrupts 17
Mar  3 19:24:09 chris-G31M-S2L kernel: [  305.377537] qlcnic 0000:01:00.0 enp1s0f0: Failed to Add interrupts 17
Mar  3 19:24:23 chris-G31M-S2L NetworkManager[501]: <info>  [1614795862.6987] device (enp1s0f0): state change: unavailable -> disconnected (reason 'connection-assumed', sys-iface-state: 'external')
Mar  3 19:24:26 chris-G31M-S2L NetworkManager[501]: <info>  [1614795864.7477] device (enp1s0f0): Activation: starting connection 'enp1s0f0' (cb34f3ee-f85b-4c47-b09d-29ff4a1ca078)
Mar  3 19:24:45 chris-G31M-S2L NetworkManager[501]: <info>  [1614795864.7480] device (enp1s0f0): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'external')
Mar  3 19:24:45 chris-G31M-S2L NetworkManager[501]: <info>  [1614795864.7487] device (enp1s0f0): state change: prepare -> config (reason 'none', sys-iface-state: 'external')
Mar  3 19:24:46 chris-G31M-S2L NetworkManager[501]: <info>  [1614795864.7493] device (enp1s0f0): state change: config -> ip-config (reason 'none', sys-iface-state: 'external')
Mar  3 19:24:46 chris-G31M-S2L NetworkManager[501]: <warn>  [1614795864.7496] device (enp1s0f0): interface enp1s0f0 not up for IP configuration
Mar  3 19:24:46 chris-G31M-S2L NetworkManager[501]: <info>  [1614795864.7498] device (enp1s0f0): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'external')
Mar  3 19:24:53 chris-G31M-S2L NetworkManager[501]: <info>  [1614795892.3347] device (enp1s0f0): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'external')
Mar  3 19:24:53 chris-G31M-S2L NetworkManager[501]: <info>  [1614795892.3354] device (enp1s0f0): state change: secondaries -> activated (reason 'none', sys-iface-state: 'external')
Mar  3 19:24:54 chris-G31M-S2L NetworkManager[501]: <info>  [1614795892.3398] device (enp1s0f0): Activation: successful, device activated.
Mar  3 19:51:12 chris-G31M-S2L kernel: [ 1929.012413] qlcnic 0000:01:00.0 enp1s0f0: Failed to Add interrupts 17
Mar  4 18:30:59 chris-G31M-S2L kernel: [83513.700260] qlcnic 0000:01:00.0 enp1s0f0: Failed to Add interrupts 17
Mar  4 18:32:23 chris-G31M-S2L NetworkManager[501]: <info>  [1614879143.5267] device (enp1s0f1): state change: unavailable -> disconnected (reason 'connection-assumed', sys-iface-state: 'external')
Mar  4 18:32:23 chris-G31M-S2L NetworkManager[501]: <info>  [1614879143.5504] device (enp1s0f1): Activation: starting connection 'enp1s0f1' (091093a5-2bd6-440e-8137-7032c2c893be)
Mar  4 18:32:23 chris-G31M-S2L NetworkManager[501]: <info>  [1614879143.5611] device (enp1s0f1): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'external')
Mar  4 18:32:23 chris-G31M-S2L NetworkManager[501]: <info>  [1614879143.5617] device (enp1s0f1): state change: prepare -> config (reason 'none', sys-iface-state: 'external')
Mar  4 18:32:23 chris-G31M-S2L NetworkManager[501]: <info>  [1614879143.5622] device (enp1s0f1): state change: config -> ip-config (reason 'none', sys-iface-state: 'external')
Mar  4 18:32:23 chris-G31M-S2L NetworkManager[501]: <warn>  [1614879143.5625] device (enp1s0f1): interface enp1s0f1 not up for IP configuration
Mar  4 18:32:23 chris-G31M-S2L NetworkManager[501]: <info>  [1614879143.5627] device (enp1s0f1): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'external')
Mar  4 18:32:24 chris-G31M-S2L NetworkManager[501]: <info>  [1614879144.2266] device (enp1s0f1): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'external')
Mar  4 18:32:24 chris-G31M-S2L NetworkManager[501]: <info>  [1614879144.2274] device (enp1s0f1): state change: secondaries -> activated (reason 'none', sys-iface-state: 'external')
Mar  4 18:32:24 chris-G31M-S2L NetworkManager[501]: <info>  [1614879144.2372] device (enp1s0f1): Activation: successful, device activated.
Mar  4 18:32:27 chris-G31M-S2L kernel: [83601.487404] qlcnic 0000:01:00.1 enp1s0f1: Failed to Add interrupts 17
Mar  4 18:32:31 chris-G31M-S2L kernel: [83605.462844] qlcnic 0000:01:00.1 enp1s0f1: Failed to Add interrupts 17
Mar  4 18:32:44 chris-G31M-S2L kernel: [83618.933051] qlcnic 0000:01:00.1 enp1s0f1: Failed to Add interrupts 17

 

 

Looks like I got Temperature issues and some other stuff that might be related but i dont know. Maybe the NIC is fried?

 

 

[Dhalion]

Okay Update: 

 

Pointing a Fan directly at the NICs seems to solve the Temperature issue.

Both NICs now show up as "UP" in both Windows and Truenas

only problem being that they didn't connect or whatever, i can't ping each other

 

    Status and config Windows:

        

 

    Status and config Truenas:

    

[Lurick]

Windows is on 192.169.69.1

Truenas is on 192.168.69.2

 

You need to change one of those numbers to match the other so they're in the same subnet.

[Dhalion]

10 hours ago, Lurick said:

Windows is on 192.169.69.1

Truenas is on 192.168.69.2

 

You need to change one of those numbers to match the other so they're in the same subnet.

Right. Thanks, dumb mistake.

 

I corrected that, however it didn't make it work...

 

Ping from Win to Nas gives me an answer from 192.168.69.1: Host unreachable and then from 62.155.246.299: Subnet unreachable

Do I have to set some Gateway or why do i get an answer vom that 62.155.256.299?

[Lurick]

3 hours ago, Dhalion said:

Right. Thanks, dumb mistake.

 

I corrected that, however it didn't make it work...

 

Ping from Win to Nas gives me an answer from 192.168.69.1: Host unreachable and then from 62.155.246.299: Subnet unreachable

Do I have to set some Gateway or why do i get an answer vom that 62.155.256.299?

What's your normal subnet on? It sounds like it's using your default network adapter and gateway to try and get to that address instead of using the connected interface. You shouldn't have to set a gateway unless you need internet or other access to subnets beyond that.

[Dhalion]

23 minutes ago, Lurick said:

What's your normal subnet on? It sounds like it's using your default network adapter and gateway to try and get to that address instead of using the connected interface. You shouldn't have to set a gateway unless you need internet or other access to subnets beyond that.

 

My regular Internet Subnet is 192.168.178.x. I didn't specify any subnet for the 10GbE on either machine (on truenas i can't even set one, at least not in the UI)

However, even if I disable the regular Ethernet adapter, I don't get a ping. Only that now 192.168.69.1 sends the errors and not 62.155.256.299. If there's no other network other than the 10GbE it should use it right?

 

Waht is this 62.155.256.299 even? It's not in my local network

 

[Lurick]

24 minutes ago, Dhalion said:

 

My regular Internet Subnet is 192.168.178.x. I didn't specify any subnet for the 10GbE on either machine (on truenas i can't even set one, at least not in the UI)

However, even if I disable the regular Ethernet adapter, I don't get a ping. Only that now 192.168.69.1 sends the errors and not 62.155.256.299. If there's no other network other than the 10GbE it should use it right?

 

Waht is this 62.155.256.299 even? It's not in my local network

 

The 62 address is a public address, probably from your ISP's network somewhere. The fact that it sends the errors and not a reply when the other adapters are disabled makes me thing something else is wrong with the NICs themselves.

 

See if this helps:

 

[Dhalion]

1 hour ago, Lurick said:

The 62 address is a public address, probably from your ISP's network somewhere. The fact that it sends the errors and not a reply when the other adapters are disabled makes me thing something else is wrong with the NICs themselves.

 

See if this helps:

 

 

I set the connection to DHCP and set a static too - makes no visible difference, not in truenas not in windows

Which is weird bc DHCP from both sides shouldnt work right?

 

[Lurick]

4 minutes ago, Dhalion said:

 

I set the connection to DHCP and set a static too - makes no visible difference, not in truenas not in windows

Which is weird bc DHCP from both sides shouldnt work right?

 

If there isn't reachability to a DHCP server on that segment it won't work unless one side is acting as a DHCP server

[Dhalion]

Alright. After some more tests with different OSes it became obvious that the cooling of one of the NICs was not enough.

More Fans solved the Issue. Thanks for your Help though Lurick

 

Closed.