Proxmox Plex server, high I/O delay after a while of bulk adding new media on raidz1 storage

[Spixel]

Hi,

I am quite new to Proxmox and want help with some storage advise. I have 3 SSDs in raidz1 for vms root folder and 3 HDDs in raidz1 for media. I have one vm for Plex (yes I know a container in Proxmox would be better and I might change to it later) where the media is stored and mounted and one vm for downloading new media and storing it to Plex via a NFS share. This works very well for a short while but if I add a bunch of downloads (200 GB+) the the I/0 delay get very high after a while and slows the whole system down, if I pause the downloads it takes i long time before it gets normal again.

I am trying to figure out what is causing this. Could it be parity calculations for the raid that can't keep up or something else?

Was raidz a bad option for this kind of use case? I don't really "need" the redundancy it is more a nice to have if a disk fails. Raid 0 is not really an option, I don't want to redownload everything if a disk fail.

Any suggestions?

[Jarsky]

What CPU is in the server? Have you checked CPU load and processes?

[Founders]

24 minutes ago, Spixel said:

but if I add a bunch of downloads (200 GB+) the the I/0 delay get very high after a while and slows the whole system down

I think you may have answered your own question here. What CPU(s) are you using? How many threads/RAM are you allocating to each VM? If your CPU or one or both of your VM's are pinned downloading 200GB's of content you're leaving little else for your data to actually write. Are you limiting your upload bandwidth on your client? What's your up/down from your ISP? If you have a quicker connection and your downloading your content 200GB's at a time your VM's resources could be pinned pretty quick.   

[Spixel]

35 minutes ago, Founders said:

I think you may have answered your own question here. What CPU(s) are you using? How many threads/RAM are you allocating to each VM? If your CPU or one or both of your VM's are pinned downloading 200GB's of content you're leaving little else for your data to actually write. Are you limiting your upload bandwidth on your client? What's your up/down from your ISP? If you have a quicker connection and your downloading your content 200GB's at a time your VM's resources could be pinned pretty quick.   

 

50 minutes ago, Jarsky said:

What CPU is in the server? Have you checked CPU load and processes?

AMD Ryzen 5 3600.

The I/O delay gets to about 60% and CPU around 20 % so I don't thinks that is the problem.

ISP speed is 500 Mb/s and the download speed is not near that.

 

I did however notice just now that the RAM usages increased alot more or less to fuill on the plex vm that have the harddrives where the data is written to during the downloads.

Could it be queing up the writes in RAM before it can write it to the disk?
Can a cache disk for the storage disk solve this problem then?

 

[Founders]

1 hour ago, Spixel said:

Could it be queing up the writes in RAM before it can write it to the disk?

Yes, how much RAM do you have allocated to the VM? How much RAM do you have installed in the system?

 

1 hour ago, Spixel said:

Can a cache disk for the storage disk solve this problem then?

No, a cache or L2 Arch is for frequently read files. A SLOG (write cache) could help but RAM is best as a SLOG comes with write speed penalties. If the SSD used as a SLOG doesn't have power loss protection it can result in data lose. Install as much RAM as you can conformably afford before you worry about a SLOG. 

[Spixel]

9 hours ago, Founders said:

Yes, how much RAM do you have allocated to the VM? How much RAM do you have installed in the system?

 

No, a cache or L2 Arch is for frequently read files. A SLOG (write cache) could help but RAM is best as a SLOG comes with write speed penalties. If the SSD used as a SLOG doesn't have power loss protection it can result in data lose. Install as much RAM as you can conformably afford before you worry about a SLOG. 

I have 64 GB total in the server and currently have 20 GB for that vm, I have have tested to assign it 50GB  but it just uses all of it anyway so that does not really solve the problem. Data loss during the download is not really a problem I can just download the files again.

 

I will look in to if using a SSD as SLOG could help me.

How does it cause write speed penalties, I thought that part of the problem it should solve?

[Needfuldoer]

Is the NFS share hosted by the Proxmox hypervisor itself, or by the Plex VM?

[Spixel]

1 hour ago, Needfuldoer said:

Is the NFS share hosted by the Proxmox hypervisor itself, or by the Plex VM?

By the Plex VM

[Electronics Wizardy]

9 hours ago, Spixel said:

I will look in to if using a SSD as SLOG could help me.

 

Are you using sync? If not, it won't help at all. Id turn off sync here personally. Losing the last 5 sec of writes for a home server in a unexpected shutdown isn't a big deal.

 

Can you post io stats? What does iostat look like? What speeds are you seeing on the hdd?