Multi-Stream PC using FFMPEG

[Wabba]

Hello guys,

 

I work for a company that manages multiple YouTube channels and we have the need to livestream to a variety of these channels, usually for long periods of time. At the moment we are outputting around 100+ streams.

 

I'm essentially the 'expert' when it comes to this but I am currently working with what I have available. I have assigned each team an iMac which they use to encode and push the file to YouTube as a livestream using FFMPEG. The developer here has also created an easy-to-use interface so users don't need to interact with any terminals etc.

 

I now have the opportunity to centralise this process to a single machine. So I'm thinking of some kind of server which people can interface with remotely from the comfort of their own office chair. 

 

So, my questions / thoughts:

 

- What hardware would I need to stream 100 ~ 200 looping videos from a single machine? This would also need to support encoding videos people send to it at the same time.

 

- What would be the best way for multiple people to interface with the machine remotely at the same time.

(I was thinking some kind of local, web based interface where files could be organised. Not sure about interacting with FFMPEG but I imagine we will create something ourselves. Would be great to get some ideas on this)

 

I've been thinking about this all day and decided that I need to get some help / ideas, so let me know what you guys have got!

 

Thanks,

 

Tim

 

 

[AbsoluteFool]

On 10/18/2019 at 5:57 PM, Wabba said:

Hello guys,

 

I work for a company that manages multiple YouTube channels and we have the need to livestream to a variety of these channels, usually for long periods of time. At the moment we are outputting around 100+ streams.

 

I'm essentially the 'expert' when it comes to this but I am currently working with what I have available. I have assigned each team an iMac which they use to encode and push the file to YouTube as a livestream using FFMPEG. The developer here has also created an easy-to-use interface so users don't need to interact with any terminals etc.

 

I now have the opportunity to centralise this process to a single machine. So I'm thinking of some kind of server which people can interface with remotely from the comfort of their own office chair. 

 

So, my questions / thoughts:

 

- What hardware would I need to stream 100 ~ 200 looping videos from a single machine? This would also need to support encoding videos people send to it at the same time.

 

- What would be the best way for multiple people to interface with the machine remotely at the same time.

(I was thinking some kind of local, web based interface where files could be organised. Not sure about interacting with FFMPEG but I imagine we will create something ourselves. Would be great to get some ideas on this)

 

I've been thinking about this all day and decided that I need to get some help / ideas, so let me know what you guys have got!

 

Thanks,

 

Tim

 

 

FFmpeg definitely have the power for this. How ever, i'm confused by what you mean "looping" 100 ~ 200 videos"?

 

Anyways, what i can tell you from the top is my head is that FFmpeg will need fast read/writes from drives. So HDDs in RAID0 is ideal, SSDs in some sort of RAID configuration will maybe also work. I would quote atleast 64GB RAM and prefferebly two CPUs at 3Ghz +

[Wabba]

On 10/21/2019 at 11:00 AM, AbsoluteFool said:

FFmpeg definitely have the power for this. How ever, i'm confused by what you mean "looping" 100 ~ 200 videos"?

 

Anyways, what i can tell you from the top is my head is that FFmpeg will need fast read/writes from drives. So HDDs in RAID0 is ideal, SSDs in some sort of RAID configuration will maybe also work. I would quote atleast 64GB RAM and prefferebly two CPUs at 3Ghz +

Hey,

 

Thank you for the reply. So, we create a video for a single channel which we use as a livestream. If the video is 2 hours long, we set it to loop infinitely. For example, a full episode will keep playing over and over on a single channel until we tell it to stop. The idea is to do this on 100 - 200 channels at the same time (with different videos on different channels).

 

Just some more background information. We recently launched a channel and the livestream accounts for more than 10% of the channels views and watchtime.

 

At the moment, we can run around 30 of these livestreams from a single iMac. 

 

Specs:

3.6 GHz i7

16GB RAM

251GB Flash Storage

 

 

 

 

[mariushm]

I'd use a Threadripper with 12-16 cores, maybe higher depending on budget.

For the streaming part, 2-4 cores are enough ... but 12+ cores will help convert the video files and prepare them into the streaming format fast.

Makes most sense as it would allow you to install up to 128 GB of memory and cache a big part of those 100-200 videos in RAM and reduce IO.

Alternatively, you also have 2-3 M.2 slots, so you could install a bunch of 1-2 TB SSDs in the M.2 connectors and you have your read speeds. 960 GB and higher M.2 SSDs can easily do 3GB/s+ read speeds.

 

You can also add a couple mechanical drives in RAID 1 (for safety or whatever) for the bulk storage (the big video files before they're compressed down to the streaming bitrates).

 

You could install a ftp server on the computer, for example Filezilla FTP server on Windows. You could set up an account for each member on your team, and the ftp server would allow that person to upload in their own folder on those drives.

When the transfer is complete, the person can go into a web based interface, click on the file (you write the page to scan the contents of folder when user logs in on the page) and select to start the conversion process, or queue the file to be encoded.

On the server, you could have a php script or perl or some batch that constantly polls if it has something to convert (by querying a database, or reading a json file or text file the website generates) and then starts converting the files.

 

If using Windows, you could also use RealVNC to connect directly to Windows and work remotely if needed.

 

mid to high end Threadripper boards also have 10g network cards, but you have so many pci-e slots that you could add a 10g card easily... they cost around 70$ or more.