Jump to content

Mellanox 10GbE SFP PCIe Card

will0hlep
By will0hlep
in Networking
 Share
Posted

I'm looking to upgrade the connection between my NAS and PC to 10 Gb ethernet. To keep the costs of this project down, I'm looking on the used market. However, the NICs available at good prices on the used market all seem to have SFP ports, instead of RJ45 ports, and I don't have any experiance with SFP yet.

 

To start, I heard somewhere that SFP module/transciever compatability was a minefield, with modules only be compatable with NICs made by the same company. Is this true? The NIC that had caught my eye was a Mellanox MCX311A-XCAT CX311A. Is there any way I could find a list of compatiable SPF modules/transcievers? A RJ45 module would be ideal as the cheap switches use RJ45.

 

Anyone know if such a NIC would be compatiable with TrueNAS (or would I need to do extra work to get it working)?

I might be experienced, but I'm human and I do make mistakes. Expand for common PC building advice, a short bio and a list of my components and other tech. I edit my messages after sending them alot, please refresh before posting your reply. Please try to be clear and specific, you'll get a better answer. Please remember to mark solutions once you have the information you need.

 

Common build advice: 1) Buy the cheapest (well reviewed) motherboard that has the features you need. Paying more typically only gets you features you won’t use. 2) only get as much RAM as you need, getting more won’t (typically) make your PC faster. 3) While I recommend getting an NVMe drive, you don’t need to splurge for an expensive drive with DRam cache, DRamless drives are fine for gamers. 4) paying for looks is fine, just don’t break the bank. 5) Tower coolers are usually good enough, unless you go top tier Intel or plan on OCing. 6) OCing is a dead meme, you probably shouldn’t bother. 7) "Bottlenecks" rarely matter and "Future-proofing" is a myth. 8) AIOs don't noticably improve performance past 240mm.

 

useful websites: https://www.productchart.com - helps compare monitors, https://uk.pcpartpicker.com - makes designing a PC easier.

 

He/Him

 

I'm a PhD student working in the fields of reinforcement learning and traffic control. PCs are one of my hobbies and I've built many PCs and performed upgrades on a few laptops (for myself, friends and family). My personal computers include 3 windows (10/11) machines and a TrueNAS server (and I'm looking to move to dual booting Linux Mint on my main machine in future). While I believe I have an decent amount of experience in spec’ing, building and troubleshooting computers, keep in mind I'm not an expert or a professional and I make mistakes.

 

Favourite Games of all time: World of Tanks, Runescape, Subnautica, Metroid (Fusion and Dread), Spyro: Year of the Dragon (Original and Reignited Trilogy), Crash Bash, Mario Kart Wii

 

Main PC: https://uk.pcpartpicker.com/user/will0hlep/saved/NByp3C

 

Secondary PC: https://uk.pcpartpicker.com/user/will0hlep/saved/cc9K7P

 

TrueNAS Server: https://uk.pcpartpicker.com/user/will0hlep/saved/m37w3C

 

Laptop: 13.4" ASUS GZ301ZE ROG Flow Z13, WUXGA 120Hz, i9 12900H, 16GB DDR5, 1TB NVMe SSD, 4GB RTX 3050 Ti, TB4, Win11 Home, Used with: 2*ThinkPad Universal Thunderbolt 4 Dock, Logitech G603, Logitech G502 Hero, Logitech K120, Logitech G915 TKL, Xbox Elite Wireless Controller Series 2, Logitech G PRO X Gaming-Headset (with Blue Icepop in Black), {specs to be updated: two monitors}

 

Other: LTT Screwdriver, LTT Stubby Screwdriver, IFIXIT Pro Tech Toolkit, Playstation 1 SCPH-102, Playstation 2 SCPH-30003, Gameboy Micro Silver OXY-001, Nintendo Wii U WUP-001(03), Playstation 4 CUH-1116A, Nintendo Switch OLED HEG-001, Yamaha RX-A4A Black AV Receiver, Monitor Audio Radius (4*90s, 1*200s, 2*270s, 1*380s), TP-Link TL-SG105-M2, Netgear GS308, IPhone 14 Pro Max 128GB Space Black, Secretlab TITAN Evo (Black SoftWeave Plus Fabric), 2*CyberPower BR1200ELCD-UK BRICs Series, Samsung 40" ES6800 Series 6 SMART 3D FHD LED TV, UGREEN USB 3.2 Gen 2 10Gbps M.2 NVMe SSD Enclosure, SABRENT 3.5" SATA drive docking station

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
1 hour ago, will0hlep said:

I'm looking to upgrade the connection between my NAS and PC to 10 Gb ethernet. To keep the costs of this project down, I'm looking on the used market. However, the NICs available at good prices on the used market all seem to have SFP ports, instead of RJ45 ports, and I don't have any experiance with SFP yet.

 

To start, I heard somewhere that SFP module/transciever compatability was a minefield, with modules only be compatable with NICs made by the same company. Is this true? The NIC that had caught my eye was a Mellanox MCX311A-XCAT CX311A. Is there any way I could find a list of compatiable SPF modules/transcievers? A RJ45 module would be ideal as the cheap switches use RJ45.

 

Anyone know if such a NIC would be compatiable with TrueNAS (or would I need to do extra work to get it working)?

ConnectX3's work fine in TrueNAS. I have one and its working fine. But it will not work on Win 11... which is waht drove me down the rabbit hole. It will work on Win 10 though, and youd imagine drivers on Win 10 would work - they won't, trust me, there is a lot of googling you can do, but just don't waste the time.

 

What you want to do, which is the most affordable option, is get 2 Intel X520-DA2 cards. The ConnectX's will work fine with truenas, but IMO since you are buying 2 from the start, just get two that match. I got 1 for my windows PC once I upgraded to Win 11 and the old connectX didn't work anymore, it came with the needed fiber transceiver, and I run a 10 gb link with fiber between them. 

 

Buy 2x of these: https://www.ebay.com/itm/166675754998?mkcid=16&mkevt=1&mkrid=711-127632-2357-0&ssspo=tG85ngMXR2e&sssrc=4429486&ssuid=B1xTkXm_Qfe&var=&widget_ver=artemis&media=COPY

 

Should come with 1 fiber transciever in each according to the image (the one I bought over a year ago came with one). Then you just need some fiber which is shockingly cheap. https://www.monoprice.com/product?p_id=7622

 

And that should be that... The one thing to make sure of is you will need to set up that subnet in both truenas and on the NIC in windows, share your SMB shares over that subnet in truenas, and then in windows remove all old network shares to that NAS and reset them up on the new point to point subnet. That way all traffic will actually flow over that link and not go out over your normal ethernet. Works like a charm!

Rig: i7 13700k - - Asus Z790-P Wifi - - RTX 4080 - - 4x16GB 6000MHz - - Samsung 990 Pro 2TB NVMe Boot + Main Programs - - Assorted SATA SSD's for Photo Work - - Corsair RM850x - - Sound BlasterX EA-5 - - Corsair XC8 JTC Edition - - Corsair GPU Full Cover GPU Block - - XT45 X-Flow 420 + UT60 280 rads - - EK XRES RGB PWM - - Fractal Define S2 - - Acer Predator X34 -- Logitech G502 - - Logitech G710+ - - Logitech Z5500 - - LTT Deskpad

 

Headphones/amp/dac: Schiit Lyr 3 - - Fostex TR-X00 - - Sennheiser HD 6xx

 

Homelab/ Media Server: Proxmox VE host - - 512 NVMe Samsung 980 RAID Z1 for VM's/Proxmox boot - - Xeon e5 2660 V4- - Supermicro X10SRF-i - - 128 GB ECC 2133 - - 10x4 TB WD Red RAID Z2 - - Corsair 750D - - Corsair RM650i - - Dell H310 6Gbps SAS HBA - - Intel RES2SC240 SAS Expander - - TreuNAS + many other VM’s

 

iPhone 14 Pro - 2018 MacBook Air

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

If your NAS and PC are relatively close to each other, a DAC (Direct Attached Copper) cable may be more cost-effective. (10GBASE-T copper SFPs are still expensive for some stupid reason.)

 

Remember if you go for fiber, you can't mix single-mode and multimore. The SFPs and fiber you choose all have to match the same standard.

 

Look into Intel NICs that have native copper 10GBASE-T ports, they'll cost a little more than old Mellanox cards but they'll be cheaper than Mellanox cards and 10GBASE-T SFPs.

I sold my soul for ProSupport.

Link to comment
Share on other sites
Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing Networking

×