Setting up a local network.

[DROZD01]

Hello everyone! I recently bought 2 ConnectX-3 cards to improve file transfer speeds between the PC and the "server", but i don't really know how to set it up correctly.
Current setup looks like this: 


What i need is:
1) PC, Server and the Router should be in one network;
2) PC and Server should use 40GbE for transfers between each other;
3) When accessing the internet - they both should use 100Mbit link through the router;
4) Currently, both PC and Server are also connected through SoftEther VPN server hosted on a remote VPS. I wonder if i can set something up, so if some "service" is hosted on the Server - at least the requests made from the PC (in that SoftEther network) would go straight to the Server, and not to VPS and back. I thought "Local Bridge" would do that, but didn't work out as i expected.
That link through the VPN is here because i need remote access to both the PC and the Server. I also have Navidrome using that virtual network for broadcasting to remote devices.

Any advice and/or instructions on how i'd have to do all of this?

[LIGISTX]

I have no idea why you have all this vpn stuff happening…. But as far as the point to point connection, just instal the connectX’s, wire them together, and set up a subnet on both machines in the network settings. For windows, right click the adapter and go to ipv4 settings, set custom subnet that is different from all other subnets being used. Something like 10.10.10.1 (assuming you’re not using 10.10.10.x anywhere else), then do the same on the server but assign it 10.10.10.2. Then on the windows PC, mount your network share as 10.10.10.2 and it’ll go over the point to point link for file transfers. Obviously depending on what OS your server is running will dictate how you set up a static IP on that interface… but that’s how you do this. 
 

Also, iirc, they do not have connectx3 drivers for win 11, so hopefully you are on win 10. I had to purchase some newer Intel cards to get this working in my win 11 machine. I *think* x4’s are the newest that there is driver support for, and no, win 10 drivers won’t work. Plenty of info on this on Google, it’s a PITA. 

[DROZD01]

3 hours ago, LIGISTX said:

Also, iirc, they do not have connectx3 drivers for win 11, so hopefully you are on win 10. I had to purchase some newer Intel cards to get this working in my win 11 machine. I *think* x4’s are the newest that there is driver support for, and no, win 10 drivers won’t work. Plenty of info on this on Google, it’s a PITA. 

That's my post on another forum about that issue. I'm on Win 10 22H2 on PC and Windows Server 2019 on the, well, Server. Windows 10 based OS'es don't have completely working drivers for ConnectX-3 either, since Ethernet part of them is not aware of intel's new architecture. I have to resort to using IPoIB, which doesn't use the same driver.
 

3 hours ago, LIGISTX said:

I have no idea why you have all this vpn stuff happening….

Devices in the network on the pic i posted can't have a static IP address, so the only way i know to get remote access to them - is through VPN subnet.

 

3 hours ago, LIGISTX said:

For windows, right click the adapter and go to ipv4 settings, set custom subnet that is different from all other subnets being used. Something like 10.10.10.1 (assuming you’re not using 10.10.10.x anywhere else), then do the same on the server but assign it 10.10.10.2. Then on the windows PC, mount your network share as 10.10.10.2 and it’ll go over the point to point link for file transfers. Obviously depending on what OS your server is running will dictate how you set up a static IP on that interface… but that’s how you do this. 

Will try that. Though, right now the 100Mbit is on like a 192.168.1.x subnet and 40Gbe IPoIB is on 192.168.2.x, and i still see that the interface regularly goes to "initializing" and cuts the 100Mbit connection during it.

[LIGISTX]

7 hours ago, DROZD01 said:

Devices in the network on the pic i posted can't have a static IP address, so the only way i know to get remote access to them - is through VPN subnet.

You can’t set a static local IP? Why not? Or do you mean can’t set a static public IP?

 

7 hours ago, DROZD01 said:

Will try that. Though, right now the 100Mbit is on like a 192.168.1.x subnet and 40Gbe IPoIB is on 192.168.2.x, and i still see that the interface regularly goes to "initializing" and cuts the 100Mbit connection during it.

Why would it down the 100Mbit connection when you are using the other point to point interface? You must have some settings wrong, or there is physical issue with the setup. 

[DROZD01]

36 minutes ago, LIGISTX said:

You can’t set a static local IP? Why not? Or do you mean can’t set a static public IP?

Yup, my ISP does not provide static IPs. As i said - i need to be able to contact these two remotely, from completely another area.

 

37 minutes ago, LIGISTX said:

Why would it down the 100Mbit connection when you are using the other point to point interface? You must have some settings wrong, or there is physical issue with the setup. 

Well yeah, i'm quite sure some settings are wrong. That's exactly why am i here. 

[LIGISTX]

20 minutes ago, DROZD01 said:

Yup, my ISP does not provide static IPs. As i said - i need to be able to contact these two remotely, from completely another area.

 

Set up Cloudflare zero trust tunnel, or just use them for DDNS (dynamic dns). Then you could go to “mywebsiteaddress.com” and it would point to your Public IP. Noip.com does the same thing, many options exist for this. 

[DROZD01]

Alright, it was caused by errors in routing tables. This one worked fine so far: