xl3b4n0nx

I can not believe I made top 50... RIP power bill this month. Hopefully I can hang on to top 50 for the next 2 days.

I am working on testing out some through put issues on my Dell R740xd with 2 Xeon Gold 6254 CPUs. I am using 8 Intel DC4510 8TB SSDs in a mdadm raid 0 array with xfs on top. Running parallel (8 to 12) dd commands (bs=32K count=8M) I am getting about 24GB/s reads and 6GB/s writes. The writes seem to be kind of slow and I can't figure out where the bottleneck is. Does anyone have any experience troubleshooting something like this? I know LTT posted a video about this, but my problem is a bit different and is on an Intel platform so the same techniques may not apply.

I figured the Asus router wasn't capable but I wanted to make sure. I am going to have them on the same VLAN so that won't matter.

Network upgrade (pfsense router + switches) to support VLANs and need confirmation before hardware purchases. For more details on what I am trying to do see below. Would it be possible for me to use my existing Asus RT-AC66U router as an AP for multiple VLANs? If question 1 is no, then would I be able to use this Ubiquiti AP to have a wifi SSID for each VLAN? I want to have 10GbE connnection between my main PC (add-in NIC) and my server (built in NIC) and have those be the only connections to the network as well for both. I plan to get this Netgear switch which is a 10 port (8x1Gb and 2x10Gb) managed switch. I want to have both machines on the 10Gb ports with a 1Gb uplink to the pfsense router. Would a VLAN that is setup in pfsense cause the 10Gb traffic between the PC and server to leave the switch instead of going directly to the other machine?

What I ended up having to do was force Unraid to not initialize the GPU on boot up by adding append vfio-pci.ids=10de:13bb,10de:0fbc,10de:1287,10de:0e0f,10de:0ffe,10de:0e1b into the syslinux configuration for the boot USB. What that does is force the vfio driver to grab the GPUs with those IDs. If you run 'lspci -v' you will see ID numbers similar to those for your system.

Also fired it back up for this purpose.

So @Jarsky I have found that all 3 GPUs in my system have the "kernel driver in use: nvidia" line. If I try to unbind any of them the terminal hangs and the process can;t be killed. Is there a way to prevent the auto grabbing on boot up?

I agree with you there. I have recieved much more help here than on the unraid forums. Do you know what makes linux bind GPUs? Two of mine shouldn't be in use at all, but they are bound.

I tried the unbind commands and it ended up locking up the nvidia cards. I don't understand what is grabbing them. Is there a way to determine where they are being used?

There are no adjustable settings there for me. Everything is under CBS.

I don't have it bound to a docker. It was in use in Folding at Home docker but I removed the UUID from the docker. It shouldn't be in use.

In advanced for me I have the North bridge settings. It has IOMMU and SR-IOV, both of which are enabled. I did find the Memory Interleave in AMD CBS.

Do you know where the IVRS setting might be in an AsRock BIOS? I can't find the setting.

Ohh ok. That makes sense. I was wondering if pinning CPU cores to a NUMA node was possible. I tried to assign cores based on which die they were on. I will give these a shot. Thanks!

IOMMU is on. Where is memory interleave? How would it effect this?

I am trying to setup a Ubuntu VM with a Nvidia GPU passed through. The GPU is not the primary GPU in the system so passthrough shouldn't be an issue. I am running a Threadripper 1920X on the AsRock X399 Fatal1ty Professional Gaming. The GPUs in the system are Quadro K4000 (primary), GT 730, Quadro K2000. I am trying to pass through the Quadro K2000. When I start the VM the VM and docker managers hang then the whole system hangs. A clean shutdown is not possible. The error in the system log is libvirtd tainted and then a stack trace which clearly indicates the GPU is the problem. For reference, I have other VM's running without GPUs with no issue at all. UnRAID Nvidia version 6.8.2. Diagnostics are attached. VM settings: CPU Mode: Host Passthrough Machine: Q35-4.2 USB Controller: 2.0 GPU 1: VNC GPU 2: Quadro K2000 Sound Card: Quadro audio tower-diagnostics-20200209-1621.zip

I am not stressing the CPU on all cores. I'm just letting it idle and it never goes above 3.5GHz. I have tried the system in Windows and it boosts just fine.

I have a Threadripper 1920X on an Asrock X399 Fatal1ty Professional Gaming running UnRAID 6.8.0. Everything booted up just fine and is be having as expected with the exception of the CPU frequency. The board is on BIOS vP3.50 and all settings are stock except for the memory XMP profile. When in UnRAID (kernel version 4.19.88) the CPU will only go to 3.5GHz (base frequency). I have checked /sys/devices/system/cpu/cpufreq/policy# and scaling_max_freq, cpuinfo_max_freq, and bios_limit all report 3.5GHz. cpb reports 1, so turbo is recognized and enabled. I am at a loss for what could be causing this. I’m not sure if it is motherboard or OS limiting it. Has anyone seen something like this with this board?

What OS will you be doing this on? Unraid?

I would recommend an ATX mid tower with atleast 3 5.25 inch bays. They can be converted into 4x3.5 inch bays and then usually cases like that have atleast 4 drive bays which will hold what you have now and allow for some expansion. A Full ATX tower is probably best for future expansion, but worry about that later. As for a SAS card, you don't have do have one. You can use the onboard SATA ports (usually 6) and get some pcie x1 to 4xSATA cards to add ports. That is my current setup with 9 drives (5 data, 2 parity, 2 cache). I have had 0 issues and those cards are cheap and plug n play. For 6 drives you can have 1 parity and be fine. It all comes down to the value of the data on the array and your level of paranoia regarding loss of that data. If you want to be safe(r) then go with 2 parity drives. Word of warning though, partiy is NOT a backup. Multiple drive failures can and do happen. With Unraid you only lose the data on that drive, but it is still data loss. Make sure you set up some kind of backup (online or offline) for the most critical data on the array. The parity drives work on the principle of RAID 5/6 (1 or 2 parity drives). When data is written to the array there is a parity calculation done on that set of data and the result is stored on the parity drive(s). When a disk in the array dies another can take its place and using the parity result plus the data from the other drives the data on the dead drive can be recalculated and stored on the new one. A more thorough description can be found here: https://wiki.unraid.net/Parity#How_parity_works Something you may want to consider is getting an older xeon e3 cpu with a supermicro motherboard with IPMI. Its a remote management interface that you access through the web. Through the interface you get a remote desktop view and access to the motherboards power and reset controls. It allows you control of the machine as if you were standing in front of it, but through the web. Its really nice to have. Hope this helped!

I would spring for a 3900X or the newly announced 3950X. Typically you need to allocate atleast one core for the host os. So one of you will lose a core when splitting them. This would give you 6c/5c or 8c/7c on those CPUs respectively.

So close to top 100.... 101. I got my third GPU too late in the game. Great final sprint though.

Better late than never, but I thought I would let you know I got it working. I am running it with my Quadro K4000. I needed to add `--runtime=nvidia` to the extra parameters in advanced view. I already had `all` in NVIDIA_DRIVER_CAPABILITIES and the GPU ID in the NVIDIA_VISIBLE_DEVICES. Hope this works for you.

Ohhhh. That's right. I forgot that I signed up with that name. I thought I signed up with my folding name. I just CTRL+F the wrong name. Sorry about that!

I'm with Shcok_Wave1. I also don't see my name up there and I did sign up in September. As of writing this comment I am #384 in team rankings.