A Beginners Guide to PROXMOX

[Bendy_BSD]

2 minutes ago, Windows7ge said:

A script that I think would be nice would be one that sets up most of the necessities from a clean install. One that preps the system for pass-through.

 

Is pass-though working currently? I can think of hardware or BIOS limitations but if it works right now it had to have been a Ubuntu software configuration or even kernel limitation.

That's what I'm currently doing right now.   I'm writing a script that sets up most of the necessities from a clean install (plus this will save me lots of time down the road lol)

 

currently, not sure yet.  I have yet to do it because I want to write this script first lol. If it's a hardware limitation do you think a BIOS update might do the trick?

[Windows7ge]

40 minutes ago, Bendy_BSD said:

That's what I'm currently doing right now.   I'm writing a script that sets up most of the necessities from a clean install (plus this will save me lots of time down the road lol)

 

currently, not sure yet.  I have yet to do it because I want to write this script first lol. If it's a hardware limitation do you think a BIOS update might do the trick?

My first concern before motherboard BIOS would be the CPU Stepping. With LGA2011-0 Xeons in the E5-26XX series there was a defect in (I believe) C0 or C1 (or both) versions where hardware pass-though didn't work as it should. You need C2. This is denoted as a CPU from the line marked with SR0KX.

 

CPU's marked with "SR0H8" (C1) are not capable of pass-though.

 

This could require you to buy whole new processors.

[Bendy_BSD]

10 minutes ago, Windows7ge said:

My first concern before motherboard BIOS would be the CPU Stepping. With LGA2011-0 Xeons in the E5-26XX series there was a defect in (I believe) C0 or C1 (or both) versions where hardware pass-though didn't work as it should. You need C2. This is denoted as a CPU from the line marked with SR0KX.

 

CPU's marked with "SR0H8" (C1) are not capable of pass-though.

 

This could require you to buy whole new processors.

How would I be able to determine what stepping it has?  Is there a command I can use or do I have to remove the CPUs manually?

Also!  I found the command string that will show just the vendor and device IDs:
lspci -vn | grep -PC2 'Subsystem:' -A 0 -B 0 | cut -b 13-21 | sed '/^s/d'   I finally frickin' found it!

[Windows7ge]

10 minutes ago, Bendy_BSD said:

How would I be able to determine what stepping it has?  Is there a command I can use or do I have to remove the CPUs manually?

Also!  I found the command string that will show just the vendor and device IDs:
lspci -vn | grep -PC2 'Subsystem:' -A 0 -B 0 | cut -b 13-21 | sed '/^s/d'   I finally frickin' found it!

It's engraved on the IHS. Any chance you took a clear picture of the CPU's before you installed them? To the best of my knowledge there isn't a way to find the answer through the OS unfortunately.

 

So the Internet is good for something eh?

[Bendy_BSD]

5 hours ago, Windows7ge said:

It's engraved on the IHS. Any chance you took a clear picture of the CPU's before you installed them? To the best of my knowledge there isn't a way to find the answer through the OS unfortunately.

 

So the Internet is good for something eh?

well shit...

 

do you know how to find the suffix to a specific line? for example,

 

say that I wanted to print out anything past "10de",

 

it would be (for example) 10de:0fba or 10de:1401

 

EDIT:

 

ughhhh.... so, I realized that by adding that "subsystem" thing in there, it only lists the subsystem and not the device id as well...

granted, it does show the vendor id, but not the device id.   I'm going to go ahead and figure out how I can isolate the first lines of every "sentence" to just show the ven#:dev# ids.

 

EDIT 2:

 

okay, this is the longest command but it isolates the output of:
 

lspci -vn

 

to just the ven#:dev# by using:

lspci -vn | cut -b 15-24 | sed '/^evsel/d' | sed '/in use\|Expr\|ster\|s:\|(\|,\|Ven/d' | grep : | sed '/idge\|MSI/d' | sed '1~2d'

 

I shit you not, this works beautifully. TuT

[Bendy_BSD]

7 hours ago, Windows7ge said:

It's engraved on the IHS. Any chance you took a clear picture of the CPU's before you installed them? To the best of my knowledge there isn't a way to find the answer through the OS unfortunately.

 

So the Internet is good for something eh?

Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
Address sizes:       46 bits physical, 48 bits virtual
CPU(s):              32
On-line CPU(s) list: 0-31
Thread(s) per core:  2
Core(s) per socket:  8
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               45
Model name:          Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
Stepping:            7
CPU MHz:             1524.812
CPU max MHz:         2800.0000
CPU min MHz:         1200.0000
BogoMIPS:            3999.98
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            256K
L3 cache:            20480K
NUMA node0 CPU(s):   0-7,16-23
NUMA node1 CPU(s):   8-15,24-31
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm epb pti ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts

 

 

 

Not sure if it's alright but here's the output of the lscpu command.

[Windows7ge]

@Bendy_BSD My CPUs also says Stepping: 7 so that's a good start but for verification that 5 digit code isn't in software to my knowledge.

 

Did you buy these new/used? Do you have the CPU box? (it should be marked on there) A picture of the ebay listing or wherever else you got them from?

[Bendy_BSD]

2 hours ago, Windows7ge said:

@Bendy_BSD My CPUs also says Stepping: 7 so that's a good start but for verification that 5 digit code isn't in software to my knowledge.

 

Did you buy these new/used? Do you have the CPU box? (it should be marked on there) A picture of the ebay listing or wherever else you got them from?

I think I have the CPU boxes still and yes I bought them brand new when I built my server.  also, I got PCI passthrough working perfectly. Also, what do you think of the tools I wrote to automate a basic setup for PCI passthrough?

[Windows7ge]

4 minutes ago, Bendy_BSD said:

I think I have the CPU boxes still and yes I bought them brand new when I built my server.  also, I got PCI passthrough working perfectly. Also, what do you think of the tools I wrote to automate a basic setup for PCI passthrough?

Gave it a whirl on one of my boinc servers and it works except if you don't already know the value you're looking for you don't know which entry is the GPU or other hardware device you want since none of the hardware ids are labeled.

 

It also appears to not want to register any ids that start with "10" so the objects appear with XX:XXXX instead of XXXX:XXXX. There's also a ton of repeat entries. Don't know what's causing that. Also my GPU hardware id doesn't show up.

 

What I would probably look into here is:

lspci -vnn | grep VGA
lspci -vnn | grep SCSI
etc

This might help give some information as to what ID belongs to what hardware device. I'll explore this for a little while and get back to you.

[Windows7ge]

14 hours ago, Bendy_BSD said:

lspci -vn | cut -b 15-24 | sed '/^evsel/d' | sed '/in use\|Expr\|ster\|s:\|(\|,\|Ven/d' | grep : | sed '/idge\|MSI/d' | sed '1~2d'

I came up with a shrunk version of your command that does display my GPU:

lspci -n | cut -b 15-23

So far as best as I've been able to isolate your best option would probably be:

lspci -nn | cut -b 9-

This tells you what each device is along with the Device ID. You may be able to clean it up better yourself.

[Bendy_BSD]

9 hours ago, Windows7ge said:

I came up with a shrunk version of your command that does display my GPU:

lspci -n | cut -b 15-23

So far as best as I've been able to isolate your best option would probably be:

lspci -nn | cut -b 9-

This tells you what each device is along with the Device ID. You may be able to clean it up better yourself.

darn T.T  just when I thought I was good you came up with something better and cleaner and shorter.
at least i gave it my best shot y'know?

oh nice!

[Bendy_BSD]

11 hours ago, Windows7ge said:

Gave it a whirl on one of my boinc servers and it works except if you don't already know the value you're looking for you don't know which entry is the GPU or other hardware device you want since none of the hardware ids are labeled.

 

It also appears to not want to register any ids that start with "10" so the objects appear with XX:XXXX instead of XXXX:XXXX. There's also a ton of repeat entries. Don't know what's causing that. Also my GPU hardware id doesn't show up.

 

What I would probably look into here is:

lspci -vnn | grep VGA
lspci -vnn | grep SCSI
etc

This might help give some information as to what ID belongs to what hardware device. I'll explore this for a little while and get back to you.

Gotcha.  And that's odd, the way I wrote it was that for NVIDIA and AMD GPUs it would have a prefix of 10de (NVIDIA) and 1002 (AMD) and using the asterisk after the colon to automatically fill in the rest of the IDs relating to it.  And that includes audio and video processors respectively.  Unless if there are GPUs from both companies that start differently?  If that's the case then I will have to do an overhaul on the install script.

 

Also, the script is intended for Proxmox-based machines only as I never expected this script to work on any other system than what I intended it to be used on.  However it's more surprising that it worked despite that little error. O.O

[Windows7ge]

10 hours ago, Bendy_BSD said:

darn T.T  just when I thought I was good you came up with something better and cleaner and shorter.
at least i gave it my best shot y'know?

oh nice!

If you know what you're looking for you could shrink the results further with grep.

 

9 hours ago, Bendy_BSD said:

Gotcha.  And that's odd, the way I wrote it was that for NVIDIA and AMD GPUs it would have a prefix of 10de (NVIDIA) and 1002 (AMD) and using the asterisk after the colon to automatically fill in the rest of the IDs relating to it.  And that includes audio and video processors respectively.  Unless if there are GPUs from both companies that start differently?  If that's the case then I will have to do an overhaul on the install script.

 

Also, the script is intended for Proxmox-based machines only as I never expected this script to work on any other system than what I intended it to be used on.  However it's more surprising that it worked despite that little error. O.O

I don't know if vendors have more than one ID. Maybe maybe not.

 

Proxmox is based on Debian as is the Ubuntu Server 18.04.4 LTS server I ran the command on so the fact it "worked" isn't surprising. It's entirety possible it's a very mild command compatibility issue as Proxmox does have some commands of its own that other Debian distro do not. I don't understand your entire command so I didn't want to run it on my hypervisor box. It's running some important things.

[Bendy_BSD]

43 minutes ago, Windows7ge said:

If you know what you're looking for you could shrink the results further with grep.

 

I don't know if vendors have more than one ID. Maybe maybe not.

 

Proxmox is based on Debian as is the Ubuntu Server 18.04.4 LTS server I ran the command on so the fact it "worked" isn't surprising. It's entirety possible it's a very mild command compatibility issue as Proxmox does have some commands of its own that other Debian distro do not. I don't understand your entire command so I didn't want to run it on my hypervisor box. It's running some important things.

Ah, that makes sense and I can understand that.  I don't want your system to break because of my script either so it's understandable.

 

i'm having a bit of a difficulty in getting vfio-pci to grab a few devices.  I'm trying to pass-through my server's on-board USB controller, the built-in raid controller and the network controller (for the four intel gigabit ethernet ports).

I added ehci-pci, ohci-pci, ahci, e1000e to the blacklist.conf file and I added the hex ven:dev ids to the options vfio-pci ids= list.
after the reboot it didn't append the change.  either that or it didn't want to change as it could've probably cause a failure of somesort so it denied the changes as a safety mechanism? 

 

am I doing anything wrong or is there something that i'm forgetting?

[Windows7ge]

@Bendy_BSD What are you looking to have the VM do? The performance of a standard vdisk is usually fine. You don't really need to pass a SCSI controller through to the VM unless you're doing something special like me.

 

If you pass-through the 4 built-in Ethernet ports Proxmox will have no Internet access. Chances are as the system starts Proxmox its grabbing them as it should. What does your VM need all the networking for? You can use paravirtualization and pass as many virtual adapters through as you need.

 

At least while using QEMU/KVM + virt-manager my desktop motherboard didn't seem to care having it's driver interrupted. Try just passing through one of the controllers without blocking the driver. Remember it needs to be in its own IOMMU group. If it's part of the chipset it won't work.

[Bendy_BSD]

1 minute ago, Windows7ge said:

@Bendy_BSD What are you looking to have the VM do? The performance of a standard vdisk is usually fine. You don't really need to pass a SCSI controller through to the VM unless you're doing something special like me.

 

If you pass-through the 4 built-in Ethernet ports Proxmox will have no Internet access. Chances are as the system starts Proxmox its grabbing them as it should. What does your VM need all the networking for? You can use paravirtualization and pass as many virtual adapters through as you need.

 

At least while using QEMU/KVM + virt-manager my desktop motherboard didn't seem to care having it's driver interrupted. Try just passing through one of the controllers without blocking the driver. Remember it needs to be in its own IOMMU group. If it's part of the chipset it won't work.

it's a special use case.  compartmentalization and no underutilization of hardware really.  I figure I have a $1,500+ computer so I might as well put it to good use.  the networking controller will solely be used by the pfsense vm as the firewall, the usb ports for windows 10 to enable hot-plug functionality until I can get a PCI-E usb card (money goes to the repair of my car and bills first, second is food, third is anything else or third is a new laptop and some PCI-E expansion cards lol.)

 

Also, I may have f-ed up, I blacklisted the ahci module and it prevented proxmox from booting up completely.  *facepalm*  I'm an idiot.   Do you think I can use the update initramfs command with another distro after editing the blacklist config file? or would I have to use the same thumbdrive that I used to install proxmox?

[Windows7ge]

4 minutes ago, Bendy_BSD said:

it's a special use case.  compartmentalization and no underutilization of hardware really.  I figure I have a $1,500+ computer so I might as well put it to good use.  the networking controller will solely be used by the pfsense vm as the firewall, the usb ports for windows 10 to enable hot-plug functionality until I can get a PCI-E usb card (money goes to the repair of my car and bills first, second is food, third is anything else or third is a new laptop and some PCI-E expansion cards lol.)

 

Also, I may have f-ed up, I blacklisted the ahci module and it prevented proxmox from booting up completely.  *facepalm*  I'm an idiot.   Do you think I can use the update initramfs command with another distro after editing the blacklist config file? or would I have to use the same thumbdrive that I used to install proxmox?

I do not recommend virtualizing your router. This server will go offline at times weather that be planned or not. When it does your whole house will lose internet. Pfsense is one of those appliances that really deserve to be ran bare metal.

 

Also the proxmox WebUI runs through one of the 4 interfaces. If you succeeded in unloading the e1000 driver you would have lost remote access to proxmox.

 

You may be able to recover the system through grub but if you can boot another distro, enter the drive and follow the directory to the blacklist file you can remove the entry without re-installing proxmox from scratch.

[Bendy_BSD]

4 minutes ago, Windows7ge said:

I do not recommend virtualizing your router. This server will go offline at times weather that be planned or not. When it does your whole house will lose internet. Pfsense is one of those appliances that really deserve to be ran bare metal.

 

Also the proxmox WebUI runs through one of the 4 interfaces. If you succeeded in unloading the e1000 driver you would have lost remote access to proxmox.

 

You may be able to recover the system through grub but if you can boot another distro, enter the drive and follow the directory to the blacklist file you can remove the entry without re-installing proxmox from scratch.

 

I know the risks when virtualizing my router, the router that my isp gave me is crap.  luckily i have a basic fiber connection.  not a ballin' gigabit connection but just a basic one, when i can get my hands on a SFP+ NIC card I will transfer the transceiver from the issued modem to my server that way I have an os that I can trust to handle my internet connection rather than having to rely on the router that my isp gave me.  plus more features on top of which pfsense has over my isp's issued hardware.

 

i am trying to mount the LVM volumes but the distro that i'm using spits out an error saying:

root@kali:~# mount /dev/sdb3 /mnt/PVE

mount:  /mnt/PVE:  /dev/sdb already mounted or mount point busy.

root@kali:~# umount /dev/sdb

umount:   /dev/sdb: not mounted.

root@kali:~#

 

so no idea why I can't mount the LVM volume group. : l

[Windows7ge]

So long as you know the risks. Out of curiosity what is the model number on your ISP provided router? I'm curious to know if even if you eliminate your crappy router if you might not still need a standalone modem or ONT. If you do you'll be running copper to this server not fiber.

 

It hasn't mounted in the GUI? Did you try another distro? You might be able to fix it in GRUB if it let's you get to a terminal.

[Bendy_BSD]

2 hours ago, Windows7ge said:

So long as you know the risks. Out of curiosity what is the model number on your ISP provided router? I'm curious to know if even if you eliminate your crappy router if you might not still need a standalone modem or ONT. If you do you'll be running copper to this server not fiber.

 

It hasn't mounted in the GUI? Did you try another distro? You might be able to fix it in GRUB if it let's you get to a terminal.

I would give you the model number but two things:
1.) It was part of the contract not to disclose WHAT model it was because reasons. (they gave me a dumb reason but if I wanted their internet service I had to agree to that including everything else.)

2.)  The S/N was easy to wipe off.. I guess the isp used ink instead of laser engraving.

 

Doesn't matter now, had to reinstall proxmox. it was in a bad state and I had the pci pass through tool so f*** it, may as well reinstall it.

[Windows7ge]

15 minutes ago, Bendy_BSD said:

I would give you the model number but two things:
1.) It was part of the contract not to disclose WHAT model it was because reasons. (they gave me a dumb reason but if I wanted their internet service I had to agree to that including everything else.)

2.)  The S/N was easy to wipe off.. I guess the isp used ink instead of laser engraving.

 

Doesn't matter now, had to reinstall proxmox. it was in a bad state and I had the pci pass through tool so f*** it, may as well reinstall it.

An ISP that doesn't let you disclose what hardware they use...that is astronomically stupid but alright.

 

I guess it's time for the pass-through tool to shine. Hopefully it works.

[Bendy_BSD]

8 minutes ago, Windows7ge said:

An ISP that doesn't let you disclose what hardware they use...that is astronomically stupid but alright.

 

I guess it's time for the pass-through tool to shine. Hopefully it works.

  I know man, it's stupid but a contract is a contract.  And I've had a friend who mentioned their router's S/N (they were using the same ISP)

 

It works beautifully. UuU

 

Also, I've had a problem mounting my 2TB drive in Proxmox, everytime it mounts it mounts as RO (read-only) and I tried everything, I tried the remount as root and even then I can even touch a file or write something to the drive.

The drive is in perfect condition and barely used. (out of the 2 years I owned it, I used it for about 96-384hrs. total. (1-4 months roughly)  because it's been off 90% of the time and I occasionally connect it to refresh the data (so the bits don't lose their electromagnetic field over time; a refresh).

[Windows7ge]

26 minutes ago, Bendy_BSD said:

  I know man, it's stupid but a contract is a contract.  And I've had a friend who mentioned their router's S/N (they were using the same ISP)

 

It works beautifully. UuU

 

Also, I've had a problem mounting my 2TB drive in Proxmox, everytime it mounts it mounts as RO (read-only) and I tried everything, I tried the remount as root and even then I can even touch a file or write something to the drive.

The drive is in perfect condition and barely used. (out of the 2 years I owned it, I used it for about 96-384hrs. total. (1-4 months roughly)  because it's been off 90% of the time and I occasionally connect it to refresh the data (so the bits don't lose their electromagnetic field over time; a refresh).

Lame.

 

Good.

 

Are you using ZFS? Is this a drive you want a VM to have direct access to? You can pass-through individual disks. You don't have to pass-though a hardware controller as I have.

[Bendy_BSD]

30 minutes ago, Windows7ge said:

Lame.

 

Good.

 

Are you using ZFS? Is this a drive you want a VM to have direct access to? You can pass-through individual disks. You don't have to pass-though a hardware controller as I have.

I'm not using ZFS for that drive, i'm only using NTFS on it. Though, for whatever reason I'm able to access it just fine if I boot into a live usb of kali linux.  But when booted into proxmox I can't write anything to the disk period.

Also how do you pass through a physical drive to a vm btw?

[Windows7ge]

2 minutes ago, Bendy_BSD said:

I'm not using ZFS for that drive, i'm only using NTFS on it. Though, for whatever reason I'm able to access it just fine if I boot into a live usb of kali linux.  But when booted into proxmox I can't write anything to the disk period.

Also how do you pass through a physical drive to a vm btw?

Part of the problem here is NTFS is not a well Linux supported File System. With some Googling things are telling me that PROXMOX does not support the mounting of NTFS volumes. At least not off of the default included packages. It looks like if you install this package:

ntfs-3g

It may enable you to mount a NTFS volume.

 

You can use virtio or iscsi for HDD pass-through, I haven't seen a real difference in terms of which is better but the command would be:

qm set #(VM) -scsi /dev/disk/by-id/disk-id

So if you wanted to pass a particular HDD through to VM 102:

qm set 102 -scsi /dev/disk/by-id/ata-ST3000DM001-1CH166_Z1F41BLC

If you had multiple of them:

qm set 102 -scsi /dev/disk/by-id/ata-ST3000DM001-1CH166_Z1F41BLC
qm set 102 -scsi2 /dev/disk/by-id/ata-WDC_WD2005FBYZ-01YCBB2_WD-WMC6N0H9RHJ1
qm set 102 -scsi3 /dev/disk/by-id/ata-WDC_WD2005FBYZ-01YCBB2_WD-WMC6N0H11HCD

etc. The drives would then appear in the VM properties under the Hardware tab. I've never had a problem with directly passing through HDDs but if you can spare a controller it makes things easier.