Search the Community

Hello everyone! I recently bought 2 ConnectX-3 cards to improve file transfer speeds between the PC and the "server", but i don't really know how to set it up correctly. Current setup looks like this: What i need is: 1) PC, Server and the Router should be in one network; 2) PC and Server should use 40GbE for transfers between each other; 3) When accessing the internet - they both should use 100Mbit link through the router; 4) Currently, both PC and Server are also connected through SoftEther VPN server hosted on a remote VPS. I wonder if i can set something up, so if some "service" is hosted on the Server - at least the requests made from the PC (in that SoftEther network) would go straight to the Server, and not to VPS and back. I thought "Local Bridge" would do that, but didn't work out as i expected. That link through the VPN is here because i need remote access to both the PC and the Server. I also have Navidrome using that virtual network for broadcasting to remote devices. Any advice and/or instructions on how i'd have to do all of this?

I have a long series of questions that could be answered in a whole LinusTechTips episode: What is the purpose of Fibre Channel HBAs? Now you have generation 5 and generation 6 HBAs with speeds of 8/16/32 Gb/s transfer speeds using optical fiber for communication rather than copper wire. What is the purpose of FC today? They are marketed as for use with hard drives, but I see no INTERNAL connectors on them that would be suited for internal hard drives such as SAS or SATA connectors. So what is the purpose of them then? It looks like they are made for external communication only. But why use FC, why not use Ethernet, you have 1GbE, 10GbE, 40GbE, ... speeds for any taste and preference? Both wired and optical. So why would one use FC instead of high speed Ethernet? To be more frank, what protocols and drivers are involved with FC? Do you use TCP/IP or what is involved? When using the controller, will it just show other devices connected to it as PCI devices, or SCSI/SATA devices or what? Will it act as a hard drive controller or an Ethernet controller? What is the target hardware used for it, some special devices with FC, another computer with an FC HBA or a FC switch? If I were to build a Beowulf cluster spanning multiple computers, what would be the best way of implementing the usage of FC connections to connect the nodes in such a cluster? Are there any substantial benefits of using that instead of regular Ethernet? To tell the truth, why am I using all these four (five if we account for SATA) interfaces interchangeably? Is it expected to see these interfaces merge into one some day in the future? I guess then we should throw Thunderbolt into the mix and while we're at it why not HDMI and DP or ..... USB3?!? Even more confusing will it get when you have FibreChannel over Ethernet (FCoE)!!! Source information comes from Broadcom/LSI, Mellanox, etc... Glossary: HBA = Host Bus Adapter (used in distinction with "I/O controller" where the I/O controller mostly means the chip that handles the interface whereas an HBA is a complete solution involving the I/O chip, PCIe connector and other I/O connectors, i.e. a controller card). FC = Fibre Channel

So I wanted to bounce something off the group.... In our research lab, we're currently setting up a private micro-HPC environment due to some functionality issues with our university's central HPC cluster. We deal with very large volumes of raw genomic data, which is both extremely data-dense and resource-intensive to process. Based on our computing needs (which is highly parallelized), we have specced out 4x [2x 32-core Epyc] Thinkmate servers (256 cores total), each linked through 40Gb QSPF+ (via a top-of-rack switch) to 2x 36-drive Thinkmate storage servers. Each storage server will be filled to capacity with 12 TB drives in RAID 10, for a useable-storage total of ~216 TB each (I should note that these will contain distinct data sets, so we do not intend to cluster them). The type of data analysis we do is extremely I/O intensive, and as such RAID 10 is the obvious choice for us (also for the ability to rebuild quickly in event of drive failure is imperative). What isn't so obvious for us is how well the RAID 10 will scale in terms of read/write performance. "In theory" my understanding is that it should scale more-or-less linearly (R:N & W:N/2) (i.e. so given each enterprise drive maxes out at about ~250MB/s, we would pretty much just saturate the 40Gb QSPF+ links at a sustained read/write speed of 4.5-5GB/s). So my question is, for people who have experience building large RAID 10 arrays, how well do read/write speeds actually scale? Are my estimates way off from real-world? (and for that matter, how good is real-world 40Gbe throughput performance?) The other option we are considering is adding ~8TB NVMe SSDs to each server to act as local scratches, but that obviously would grossly increase our costs.

Hello guys I think current networking is very much limited by the commonly used 1Gbit. So I watched the video: 10x Your Network Speed.. On a Budget! I thought that 10GbE is still childsplay. So I googled about alternative technology and found that infiniband can also do normal IP networking. So I bought two Infiniband adapters with 40Gbit network speed each and a 7 Meter QSFP cable. All in all less than 100 Dollars. something like this: http://www.ebay.com/itm/QLogic-QLE7340-Single-Port-40GBp-s-QDR-Infiniband-HCA-PCI-E-x8-w-Warranty-/182288795362?hash=item2a714246e2:g:r4YAAOSwCGVX4qUG I really find the technology fascinating as normal 40GbE Adapters are VERY expensive and I never heard of infiniband before. I bought two cards and they show up with 32Gbit. File sharing speed is at about 5 Gbytes/s (not Gigabit)! I cant find anything wrong with this solution. It works, is cheap and behaves like any other NIC (because of IP over Infiniband) So normal SMB and other network stuff works as usual. So my question is if anyone here on this channel has experience with similar technology or also want faster network speed (lets put the NEED aside here) Please share you setup and experience. (This is a MHQH19B-XTR currenty used) with a dual adapter card and NIC teaming you could easily achieve 80Gbits/s.

As I regulary watch LinusTechtips on youtube I stumbled across a video called: 10x Your Network Speed.. On a Budget! I thought that is still too slow. So I googled about alternate technology and found that infiniband can also do normal IP networking. So I bought two Infiniband adapters with 40Gbit network speed each and a 7 Meter QSFP cable. All in all less than 100 Dollars. something like this: http://www.ebay.at/itm/Mellanox-ConnectX-2-VPI-Network-Adaptor-PCIe-Server-Card-/331765696568 I really find the technology fascinating as normal 40GbE Adapters are VERY expensive and I never heard of infiniband before. I bought two cards and they show up with 32Gbit. File sharing speed is at about 5 Gbytes/s (not Gigabit)! I cant find anything wrong with this solution. It works, is cheap and behaves like any other NIC (because of IP over Infiniband) So normal SMB and other network stuff works as usual. So my question is if any of you has also tried this and if someone on this channel could take a look it this to point out the possible drawbacks. (Maybe Youtube? :D)

Preface: This is not a personal build, but rather a build I'm leading for the company I work at. We're a large photography studio here down in Southern California, and we also operate one of the industry leading photography educational sites. I'm sure ya'll can guess who we are! Previous build log I've neglected a couple years ago on OTT: I'm treating this build as if it were my own, and would love to share back with the LTT community as quite a bit of knowledge was learned from Linus's videos regarding network rendering and adobe premiere. We're actually not too far from each other in the way we build our machines, except LTT does get the HW hookups, we don't hah! This will be an on-going project, and I will try and document what I can without divulging too much about our internal configuration, just you know, legal reasons. Phase 1 that will be shared in this post is where everything currently stands, this was doing over the course of a few days to move from our old location to the new location, there's still plenty more to come! Rack 1 Networking gear: 1st switch is an Edge-Core AS4610-52T running Cumulus Linux. This is a 48 port PoE+ switch with 4xSFP+ 10G, and 2x QSFP+ 20G stacking ports, unfortunately the stacking ports are not supported on Cumulus. This switch will be uplinked via 4x10G with VLAN trunks to a core 10/40g switch in Rack 2 shown later. 2nd switch is a Juniper EX3300-48T pulled from our old office, just needed additional 1G ports, as really we'll only have 8-10 devices on PoE+ ( Aruba AP's and PoE+ IPTV cameras ) 3rd switch is Cisco 3560, this switch is for management traffic on Rack 1 ( IPMI, PDU/UPS, back-end management of switches etc ) 4th switch is a Netgear XS712T that I quite hate. But kept so we don't have to add new network cards on 2 of our existing ZFS units as they have 2x10Gbase-T onboard. Below that is a pfsense router. This router will be replaced eventually with a redundant 2 node pfsense cluster running E3-1270V2's with 10G networking for LAN portion, and 2x1G switch to the EX4200-VC cluster on Rack2 for internet uplink the 4 4U servers below are Supermicro 24bay units running ZFS. Each with 128GB of RAM. One of the servers is running Dual ported 12G SAS 8TB Seagate drives with a 400GB NVMe drive as L2arc. Total raw storage ~ 504TB in raw spinny drives Rack #2 Disclaimer: I do not recommend storing liquids on top of IT equipment regardless if they are powered on or not, in hindsight I probably shouldn't have included this picture, but it's already on reddit and I took the flaming.. Lesson learned. 32x 32GB Hynix DDR4 2400 ECC REG dimms for the VM hosts + 8x16GB DDR3 1866 dimms for upgrade of old ZFS nodes ( 64GB to 128GB ) Total ram 1024GB of DDR4 + 128GB of DDR3. It's crazy how dense DRAM is now.. 8x QSFP+ SR4 40GbE optics. Initially was going to go 2x40GbE to the ZFS nodes but ended up running out of 10GbE ports. So 1x 40GbE per ZFS node ( 2x total ) and the rest of the 4x40GbE ports will be split out to 10GbE over PLR40GbE long range 4x10GbE optics to end devices. Top switch: Edge-Core AS5712 running Cumulus Linux again. 48x10GbE SFP+ and 6xQSFP+ 40GbE. Loaded with singlemode FiberStore optics ( not pictured ) Sitting above it is a 72strand MTP trunk cable going to our workstation/postproducer area. Still awaiting the fiber enclosures and LGX cassetes so we can break this out into regular LC/LC SMF patch cords to the switch Below that are two EX4200-48T's sitting in a Virtual Chassis cluster. These will be used primarily for devices that should have a redundant uplink, so I'm spanning LACP across the two "line-cards", this will also be used in LACP for our VM host nodes for Proxmox cluster/networking traffic. Ceph/Storage traffic will be bonded 10GbE for nodes that need it. Again a wild Cisco 3560 10/100 switch with 2x 1GbE LACP uplinks to the EX4200 for management traffic. Rack #2 will eventually be filled with 4x 2U 12bay Supermicro units loaded with SSD's as Ceph nodes, and then some E3 hosts for smaller VM guests. Below that you can spy a Supermicro Fattwin 4U chassis with 4x Dual Xeon V3 nodes. Each node will have 256GB of RAM, and depending on the node they will either have dual E5-2676 V3 or E5-2683 V3 CPUs. There will be minimal if not 0 local storage as all VM storage will live on the Ceph infrastructure. 4x10GbE everywhere. Random photos of our IT room 4x Eaton 9PX 6KvA UPS's, 2 per rack. Fed with 208V AC PDU's are Eaton's G3 managed EMA107-10