Jump to content

Proxmox GPU Passthrough | Storage Pools and Resource pools

Googs55
 Share
Posted

Hi All!

 

I would love some advice regarding my home lab setup. Below is my current hardware and some features and functions I would like my home lab data center to accomplish!

I have 3 Dell PowerEdge r710 and 1 Dell PowerEdge r410 a few of the servers have all HHD's and a few have SSD's. All except the r410 have RAID PERC controllers. One r710 has a Nvidia GT 1030. I also run a Mac OS server with an external RAID 5 hard drive using SMB for a file share. This is my main File share server and it gets the job done but it is less than ideal when I have all of these servers.

 

I have yet to transfer all the data from the Mac server to one of my PowerEdge machines because of a lack of stability in my home lab "data center" setup. I have been playing around with Proxmox and setting up my home lab "datacenter" and experimenting with all the stuff you can do with hypervisors and servers.

 

Everything that I have set up has not stuck or has been inadequate lacking a main feature or function that I'm looking for in my ideal home lab. That leads me to the functionality that I am looking for. For starters, a major requirement is running a hypervisor. Proxmox was recommended to me but I'm willing to learn something else if given a good enough reason. A major requirement is to passthrough all my GPU's to my VM's. I also need hot-swap capabilities for my storage because I can't afford to fill up all my r710's currently. I am willing to go to a software-defined storage model instead of utilizing PERC controllers. I would also like the ability to run storage pools and resource pools. In an ideal world, I also would like to tie the Mac Server into the mix using this storage as extra probably backup capacity. (every time I have tried to SMB connect it to Proxmox it fails.) Ideally, I also have some sort of replication that is occurring for redundancies.

 

Can someone guide me in the right direction to accomplish these requirements? Am I using the wrong hypervisor? Do I need to get rid of the PERC controllers? What is the best way to configure these?

 

I look forward to hearing the communities opinion and advice. Thanks in advance!

~Googs55

 

 

 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted

Proxmox should be able to do all that you want here. Ive used it a good amoutn and done all of this.

 

What do you mean by resource pools? Is there multiple users?

 

Do you want shared storage for the vms? You can setup ceph pretty easily, otherwise id just use zfs and a hba. But using the perc cards for raid and then using zfs/lvm will work fine here.

 

What issue do you get with smb? That should work fine here. Or setup a nfs share. Then you can use PBS to manage backups.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author

@Electronics Wizardy  Thanks for the reply! So I just did some more digging into resource pools. My original understanding was these could work together for one task but after reading it looks like they are used to only manage and control access to resources. Correct? 

 

Yes I want shared storage. I am reading using ceph I should not use RAID it just adds extra redundancies. Ill mess around with cephalopods today and see what I can do with hba or try zfs I've set up lvm but not sure how to set up shared storage. Do I just re flash the servers and set up zfs?

So the issue I am having with SMB is "create storage failed: error during cfs-locked 'file-storage_cfg' operation: storage 'Drobo' is not online (500)" The SMB is online. I am using CIFS is this the correct type of storage?

 

Thanks for the help!

 

~Googs
 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
4 minutes ago, Googs55 said:

hanks for the reply! So I just did some more digging into resource pools. My original understanding was these could work together for one task but after reading it looks like they are used to only manage and control access to resources. Correct? 

Yup thats right, there made for permissions, not for managing systems

 

What was your goal? Did you want to make a cluster of systems? Proxmox can do that easily

 

5 minutes ago, Googs55 said:

Yes I want shared storage. I am reading using ceph I should not use RAID it just adds extra redundancies. Ill mess around with cephalopods today and see what I can do with hba or try zfs I've set up lvm but not sure how to set up shared storage. Do I just re flash the servers and set up zfs?

Yea ceph will work fine here. You just setup ceph, it will \use the drives directly, and then then you can make vms with the ceph as the storage.

 

6 minutes ago, Googs55 said:

So the issue I am having with SMB is "create storage failed: error during cfs-locked 'file-storage_cfg' operation: storage 'Drobo' is not online (500)" The SMB is online. I am using CIFS is this the correct type of storage?

Can you mount the cifs share in the terminal directly?

 

Are you using the ip or hostname? 

 

Is the domain set correctly?

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author
2 minutes ago, Electronics Wizardy said:

Yup thats right, there made for permissions, not for managing systems

Understood. This was a misunderstanding on my part.

2 minutes ago, Electronics Wizardy said:

 

What was your goal? Did you want to make a cluster of systems? Proxmox can do that easily

Yeah I want a redundant cluster. As close to Hyper converged infrastructure as possible. I understand I am trying to fit a square box into a round hole but technically if the box is the right size it will fit. 🙂

2 minutes ago, Electronics Wizardy said:

 

Yea ceph will work fine here. You just setup ceph, it will \use the drives directly, and then then you can make vms with the ceph as the storage.

Ok How can I set up hba without using PERC? do I have to replace/remove the PERC? Or set up RAID 0? Whats the best course of action?

2 minutes ago, Electronics Wizardy said:

 

Can you mount the cifs share in the terminal directly?

I will try

2 minutes ago, Electronics Wizardy said:

 

Are you using the ip or hostname? 

Using ip

2 minutes ago, Electronics Wizardy said:

 

Is the domain set correctly?

Yes all devices on network and though vpn can access via FQD and ip.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
1 minute ago, Googs55 said:

Yeah I want a redundant cluster. As close to Hyper converged infrastructure as possible. I understand I am trying to fit a square box into a round hole but technically if the box is the right size it will fit. 🙂

Yea you can easily do that in promox with clusters. Just add all the systems to one cluster, and then setup shared storage with ceph for a hyperconerged setup.

 

2 minutes ago, Googs55 said:

Ok How can I set up hba without using PERC? do I have to replace/remove the PERC? Or set up RAID 0? Whats the best course of action?

Quote

The best way is to replace the perc with a hba, but sometimes you can flash the card to hba mode. Raid 0 mode will work if needed.

 

2 minutes ago, Googs55 said:

I will try

One other hack is if you can mount it in the terminal is to just have it auto mount using cifs in fstab, then have proxmox add it as a directory storage.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author
4 minutes ago, Electronics Wizardy said:

Yea you can easily do that in promox with clusters. Just add all the systems to one cluster, and then setup shared storage with ceph for a hyperconerged setup.

Ok sweet thanks!

4 minutes ago, Electronics Wizardy said:

 

The best way is to replace the perc with a hba, but sometimes you can flash the card to hba mode. Raid 0 mode will work if needed.

Ok so I currently have nodes set up as RAID 5. Does running ceph with RAID 5 impact speed and efficiency? or is it just over kill and thats why it is not recommended?

4 minutes ago, Electronics Wizardy said:

 

One other hack is if you can mount it in the terminal is to just have it auto mount using cifs in fstab, then have proxmox add it as a directory storage.

Ok. I have tried both SMB 2.1 and 3.0 and I still get the same error... 

 

 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
Just now, Googs55 said:

k so I currently have nodes set up as RAID 5. Does running ceph with RAID 5 impact speed and efficiency? or is it just over kill and thats why it is not recommended?

Ceph wants to have direct access to the drives. So the best setup will be to have a hba so ceph can see all the hdds, and then it will manage redundancy for you.

 

1 minute ago, Googs55 said:

Ok. I have tried both SMB 2.1 and 3.0 and I still get the same error... 

 

What error do you get when mounting in the terminal?

 

Try making a nfs share on the mac.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author
11 minutes ago, Electronics Wizardy said:

Ceph wants to have direct access to the drives. So the best setup will be to have a hba so ceph can see all the hdds, and then it will manage redundancy for you.

Ok ill restart. Setup lol

11 minutes ago, Electronics Wizardy said:

 

What error do you get when mounting in the terminal?

Same error as before.
 

11 minutes ago, Electronics Wizardy said:

 

Try making a nfs share on the mac.

Similar error for nfs share to Mac:

 

create storage failed: error during cfs-locked 'file-storage_cfg' operation: mount error: mount.nfs: access denied by server while mounting 10.0.96.105:/Users (500)

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
1 minute ago, Googs55 said:

Same error as before.

What mount command are you using? Can you show a screen shot?

 

1 minute ago, Googs55 said:

Similar error for nfs share to Mac:

 

create storage failed: error during cfs-locked 'file-storage_cfg' operation: mount error: mount.nfs: access denied by server while mounting 10.0.96.105:/Users (500)

That looks like a permissions issue, check that the mac allows this connection.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author
2 minutes ago, Electronics Wizardy said:

What mount command are you using? Can you show a screen shot?

Ok I went back and looked at command... Honestly I am not sure what "extbackup" means and that seams to be the problem. I was under the assumption that this was the type of data that can be stored on the file share. You know what happens when you assume... Command below:

 

pvesm add cifs extbackup --server 10.0.96.105 --share Drobo --username stephenguglielmo --password MyPasswordHere --smbversion 3.0

create storage failed: error during cfs-locked 'file-storage_cfg' operation: storage 'extbackup' is not online

 

2 minutes ago, Electronics Wizardy said:

 

That looks like a permissions issue, check that the mac allows this connection.

Ok. I will try to connect via nfs from another machine.

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
1 minute ago, Googs55 said:

Ok I went back and looked at command... Honestly I am not sure what "extbackup" means and that seams to be the problem. I was under the assumption that this was the type of data that can be stored on the file share. You know what happens when you assume... Command below:

 

pvesm add cifs extbackup --server 10.0.96.105 --share Drobo --username stephenguglielmo --password MyPasswordHere --smbversion 3.0

create storage failed: error during cfs-locked 'file-storage_cfg' operation: storage 'extbackup' is not online

 

Thats the name, so you can set it to whatever you want.

 

Try using the standard linux mount command to see if the system can mount it.

 

 

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
  • Author
35 minutes ago, Electronics Wizardy said:

That looks like a permissions issue, check that the mac allows this connection.

I think you may be right with standard linux mount command I am getting permission denied... 

 

Not 100% Sure if I have keyed in the correct mount point. It is an external drive does this complicate things?

Link to comment
Share on other sites
Link to post
Share on other sites
Posted
2 minutes ago, Googs55 said:

I think you may be right with standard linux mount command I am getting permission denied... 

 

Not 100% Sure if I have keyed in the correct mount point. It is an external drive does this complicate things?

The external drive on the mac shouldn't really affect things, its a network share the same was as if it was a internal drive.

Link to comment
Share on other sites
Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing Servers, NAS, and Home Lab

×