Would this be a practical idea for an application? How would i go about coding it?

[Enderman]

I'm a content creator. I use Blender. Stop using that argument, you've used it twice now.

I mean professional content creators

people who do it as a job

LMG uses premiere

MKBHD uses premiere

unboxtherapy uses premiere

hardwarecanucks uses premiere

teksyndicate uses premiere

jayztwocents uses premiere

 

do you think these people will want to switch to blender just for a bit faster rendering times?

i dont think so

[Enderman]

Edit: Do I need to mention that only a fraction of video editors actually use networked rendering?

Most render on their own machine, or have a dedicated render farm machine or rented a render farm from a company.

Even if that was the selling point, only a fraction of people would do their final render with Blender then.

yeah so in order to make something that people would actually want to use, OP would need to code some video editor like premier pro with the network rendering feature of blender combined together

 

and coding that will be extremely difficult...

unless he makes some sort of plugin that premier uses and could enable network rendering

but that would still be insanely difficult to code

 

personally i would never let someone else render a part of my videos anyway...who knows what they could do with it while its on their end

this is why blender specifically says to use the network rendering feature only with "trusted friends"

[babadoctor]

Again, read what I said. Premiere can link with After Effects and effectively enable networked rendering.

 

Citation needed.

 

There are security measures in place in regards to BURP. For example, several people render each chunk of the image and then validate it. If any one of them is different in any way, their work is invalidated and their part is not used but sent to someone else. The worst they can do is rip their chunk of the image, but they're still only getting a few pieces of the the image, not the whole image, thus not worth it. 

It makes absolute sense for Pixar to not render on a public render farm. It makes no sense for your average user not to unless they're working on a confidential project, which, if you're uploading it to youtube, you're not.

I dont think enderman understands that you can encrypt sending rendering chunks to other machines

 

its actually pretty easy

when the person renders it, it renders some sort of code that would take ages to decrypt but not render

so pretty safe

[babadoctor]

We're not talking about encryption, but do go on. Who told you this?

I heard about it on foldingforum.org

 

someone was talking about how they wanted to ask their friend to help them render a .blender file with the same software folding@home used, and someone asked about encryption, and he said that there is a plugin for blender that encrypts the file through some sort of encryption software(not sure which one) and then decrypts it through some sort of pre determined key that you give your friend

*i think it was a hash of some sort but not completely sure

 

uses some sort of really really advanced math algorithm that verifies the integrity of the render helping application kind of like VAC does, then sends it a part of the file that the already checked file is sent through, then decrypted, rendered, then sent out agin

[LtStaffel]

36 notifications about @Enderman vs @DerakDerantick.  Both of you are misunderstanding eachother. Enderman is talking more about raw video to edited video that you upload.  DerakDerantick is talking about rendering.  I'm no genius, so I could be wrong, but those are different things.  Yes DerakDerantick, Blender can do network rendering but it's reading a source file to know how to render it. There might be a video feature, I don't know. But that's what Enderman is getting at.  Enderman is talking about something like a dedicated video editer, which would be a bit different than rendering since it's adding and subtracting from the input.  Sorry for the middle-man, I couldn't resist the temptation.

[babadoctor]

Source, please.

i honestly lost the source, sorry

 

i dont really have time to fine comb 50 or so pages of posts in folding@home forums...

[trag1c]

and you think a content creator who has a powerful 5960X or whatever wants to send his data to many users so they can render it on their weak PCs and potentially see the content before it is released or maybe even modify the content to insert unwanted images?

 

yeah no. not gonna happen.

1) 5960X is a piece of **** when it comes to 3D rendering. 16 threads doesn't get you very far even LMG's 2699v3's still wouldn't be that great. A single rendering in Cycles, Maya or 3DS Max can take days for a single frame to finish. Now multiply that by 24 or 30 frames per second for a 2 minute trailer. This is why businesses like Pixar have 4 super computers spread across 4 geographic locations. Disney's Big Hero 6 took around 6 months to render with all 4 clusters  (55000+ cores total) networked into a single system.

2) Rendering =/= displaying.

3) Custom data formats are a thing. Without knowing how that data is formatted and/or compressed it is complete garbage to anyone without that information. So 99.99999% of users wouldn't have a clue on how to edit it.

 

 

As for the whole Blender, Premier thing.

 

Rendering is a very ambiguous term. Video rendering is not the same as 3D rendering. 3D rendering results in an image or image sequence which then would be brought into premier or similar for post process. 

 

Blender is a 3D modelling/sculpting package that rasterizes 3D geometry into a 2D image using the Cycles rendering engine. This works much like a video game but with much more precision and thus more computationally expensive.

 

Blender does have a basic video editor functionality but it does not match up with premier or other commercial packages. Blender is only 1 step of the process. (Often the most time consuming and computationally expensive step.) Premier would be the last step before release.  

[LtStaffel]

I'm not sure what you're getting at here, we're both talking about distributed rendering. 

Specifically, elaborate on these two sentences, please

"Enderman is talking more about raw video to edited video that you upload." and

"Enderman is talking about something like a dedicated video editer, which would be a bit different than rendering since it's adding and subtracting from the input."

Enderman liked my post and didn't reply, so I'm going to assume he's ok with me explaining it my way.  I think he means that sony vegas or premiere pro take an unedited, very long, video file.  And then they let you put in jump cuts, effects, and such, to the point where it's ready to go on YT. For example, ChannelSuperFun makes a video about a board game.  They record the whole game.  Then one of the editors pulls up premiere pro and cuts out the boring stuff, adds zooming in and out, and gun shot effects etc. Then the editor exports it to some video file, probably mp4 or something. Then they upload it to YT. The exporting is what takes time for the program, to skip the parts of the video that were cut out and to add the little pictures to each frame and add effects to the sound file etc. That process isn't really rendering in the same sense that Blender renders and image.  Because when Blender renders an image, it divides it into sections, and generates pixels on those sections, and corrects the color of those pixels.  That is a job multiple computers can work on, because the work load can be divided into those areas of pixels.  that work load can be sent to individual computers, thus the network rendering. But with a video editor, it's doing it all at once locally, there isn't a way to divide the jobs up properly.

 

Does that all make sense? (I have a horrible track record of not explaining things well )

[ALwin]

snip

 

It's a nice idea however would be difficult to do on anything other than a set of computers on a local network, for various reasons:

• Sharing resource files across the internet, especially if (for example) the rendering project involves very large media/video files.
• Computers would need to have the same programs and plugins.  You can't have one crowd sourced renderer using Premiere Pro and another using Final Cut.
• Even if every crowed sourced participant has the same version of Premiere Pro, with the same set of plugins, and the resource files are stored on a centralized cloud server, the performance will vary depending on the internet speeds of each participant and their computer.
• You can try to improve it by having each participant download a small segment of the project (including resource files) but then of course the master editor has to make sure that the project is initially segmented into workable chunks.

[Enderman]

 

this entire thread is about rendering a video after video editing it, if you didnt notice

thats the kind of thing that youtubers do, such as tech youtubers or gamers

we are not talking about rendering 3d models or advanced lighting scenes or whatever, just editing clips together and rendering it on other peoples computers

[babadoctor]

 

It's a nice idea however would be difficult to do on anything other than a set of computers on a local network, for various reasons:

• Sharing resource files across the internet, especially if (for example) the rendering project involves very large media/video files.
• Computers would need to have the same programs and plugins.  You can't have one crowd sourced renderer using Premiere Pro and another using Final Cut.
• Even if every crowed sourced participant has the same version of Premiere Pro, with the same set of plugins, and the resource files are stored on a centralized cloud server, the performance will vary depending on the internet speeds of each participant and their computer.
• You can try to improve it by having each participant download a small segment of the project (including resource files) but then of course the master editor has to make sure that the project is initially segmented into workable chunks.

 

Thats why both cilents will be using the same program that i will try to write

 

 

this entire thread is about rendering a video after video editing it, if you didnt notice

thats the kind of thing that youtubers do, such as tech youtubers or gamers

we are not talking about rendering 3d models or advanced lighting scenes or whatever, just editing clips together and rendering it on other peoples computers

3d models could be what this is about too... RENDERING ANYTHING

[trag1c]

this entire thread is about rendering a video after video editing it, if you didnt notice

thats the kind of thing that youtubers do, such as tech youtubers or gamers

we are not talking about rendering 3d models or advanced lighting scenes or whatever, just editing clips together and rendering it on other peoples computers

 

 

I want to make an application that allows you to support your favorite game developer or youtuber without having to spend money!

 

Hmm it very clearly states game developer in the op's post. So we are most definitely talking about rendering 3D models with physically accurate lighting models. Come to think about we're also talking about rendering, normal maps, specular maps, textures, cut scenes and trailers. All of which take obscene amounts of time to compute.

[ALwin]

snip

 

I don't think you would even need a program to do that.  Let me explain how I would get this done if I were working on a very large video project and it would take my workstation 10 hours to render out the final video file.

 

1. Say for example the original media files (raw footage) from my video camera was 10 video clips.  It could also be a single video clip.
2. And that I know 9 people who had the same version of Premiere Pro, Media Encoder and the same set of plugins.
3. Firstly, I would take my video file(s), do the edits, apply the effects I want, do the color grading, etc.
4. Then I would splice up the timeline sequence (containing the edited video file(s)) into 10 pieces using the cut tool.
5. I would duplicate the sequence 9 times, to get 10 identical sequences in my project explorer.
6. Open all 10 sequences in the timeline, and in each sequence delete 9 of the segments, so that Duplicate Sequence 1 has the first segment, Duplicate Sequence 2 has the second segment, etc.
7. Save the project files in a compressed folder and send them out to the 9 people I know.
8. I will render one sequence, and the other 9 will each render one of the sequences.

But here is the kicker.

If my original media files from the camera were 10 different clips, I would just send to each person the video clip that is necessary to perform the rendering I need them to do.  However if I am originally working with a single video file that I have cut into 10 pieces, either I send all of them the full length master file which could be a huge file or I have to render out a spliced out part of the master file for each of them.

 

3D modeling and design looks easier for this sort of crowd sourced rendering/render farm work because technically there are no master video footage(s).  The 3D models that need to be animated and rendered out all exist as a set of complex mathematical formulas and values.  There is no ProRes or H.264 raw footage that's 100GB+ in size.

[LtStaffel]

@Enderman and @DerakDerantick, did I correctly interpret your points?

[LukeTim]

Hang on. Are we talking about video rendering (encoding/compressing) or scene rendering (like, GPU work)?

 

I've seen different people talking about one or the other...

[babadoctor]

Hang on. Are we talking about video rendering (encoding/compressing) or scene rendering (like, GPU work)?

 

I've seen different people talking about one or the other...

Whichever one is more plausible

 

i dont mind if we talk about both, but i THINK the past 5 pages has been about scene rendering.

[trag1c]

Hang on. Are we talking about video rendering (encoding/compressing) or scene rendering (like, GPU work)?

 

I've seen different people talking about one or the other...

 

Whichever one is more plausible

 

i dont mind if we talk about both, but i THINK the past 5 pages has been about scene rendering.

I t's very mixed. Op wants to help game developers which would use scene rendering and youtubers would be video rendering.

 

I personally see scene rendering across the interwebs as a very feasible idea. I am unsure about video rendering since I have very little experience doing such tasks as well as I've never programmed an application like that before.

[babadoctor]

I t's very mixed. Op wants to help game developers which would use scene rendering and youtubers would be video rendering.

 

I personally see scene rendering across the interwebs as a very feasible idea. I am unsure about video rendering since I have very little experience doing such tasks as well as I've never programmed an application like that before.

<3 i want to create an application that makes it easy

[ALwin]

<3 i want to create an application that makes it easy

 

I don't think you can make a generic application that can be used for any kind/type of project across a wide variety of software/platforms.

 

Perhaps you should focus your scope on, for example and sake of discussion, Premiere Pro v.9 and Media Encoder users on Windows.

 

If it's for video rendering make your application focus on that aspect.  If it's for 3D model rendering make a separate application for that.  By the way, 3Ds Max already comes with tools for utilizing the power of render farms.

[Brenz]

It's a nice idea and certainly not an impossible task for an experienced software engineer but the massive limitation here is the internet.

 

Folding and mining work well because the amount of bandwidth required is tiny. Your program would require a lot of bandwidth on both ends to be able to match the likely processing power anyone making videos is likely to already have especially if they are at a point where they have enough viewers that were willing to help with rendering to even make using your app worth it.

[babadoctor]

It's a nice idea and certainly not an impossible task for an experienced software engineer but the massive limitation here is the internet.

 

Folding and mining work well because the amount of bandwidth required is tiny. Your program would require a lot of bandwidth on both ends to be able to match the likely processing power anyone making videos is likely to already have especially if they are at a point where they have enough viewers that were willing to help with rendering to even make using your app worth it.

Really?

I dont understand, all the host has to be able to handle is waiting for 10 peoples files to finish rendering, not all the same time...

[Brenz]

Really?

I dont understand, all the host has to be able to handle is waiting for 10 peoples files to finish rendering, not all the same time...

 

Before the clients can do anything the host has to split up the rendering task into small chunks and then send this unrendered data to multiple clients. This will require a lot of upload bandwidth.

 

The client then needs the bandwidth to download this data, process it and then upload it again where the host can then download it.

 

Realistically when looking at the average broadband speeds anywhere they are nowhere near enough to support this idea and as file sizes continue to rise with more and more 4k and 360 footage it's not going to get any faster any time soon

[LukeTim]

I t's very mixed. Op wants to help game developers which would use scene rendering and youtubers would be video rendering.

I personally see scene rendering across the interwebs as a very feasible idea. I am unsure about video rendering since I have very little experience doing such tasks as well as I've never programmed an application like that before.

Video would theoretically be feasible but only if you had an enormous upload bandwidth. It seems like a very sensible idea because each GOP (group of pictures) can be encoded independently of all the others, so each node in the distributed encoder could handle it's own little queue of GOPs. I believe this is how multithreaded video encoding works as it is.

Say you want to encode 4K video, and assume you want to encode using a 24 frame GOP (meaning 24 frames between key frames) that would be 8 (megapixels) by 1.5 (bytes per pixel, assuming 4:2:0 YUV encoding) by 24 (frames) which would be 288MB per GOP. A quarter of a gig. In a typical YouTube lets play you would probably have 10,000s of GOPs. Even with a business grade 1Gbps upload, for all the time it'd take uploading to your distributed encoding network you might as well do it locally.

In addition, once all the encoding is complete you would need to receive all the encoded data and put it back in order. Not an easy task, and again bandwidth constrained.

Netflix does use a similar design in their Amazon AWS based encoder, which is distributed. However, this encoder is distributed on a local/private network, so much faster, lower latency, less traffic. And they need to handle encoding to many different quality/resolution levels.

[babadoctor]

Before the clients can do anything the host has to split up the rendering task into small chunks and then send this unrendered data to multiple clients. This will require a lot of upload bandwidth.

 

The client then needs the bandwidth to download this data, process it and then upload it again where the host can then download it.

 

Realistically when looking at the average broadband speeds anywhere they are nowhere near enough to support this idea and as file sizes continue to rise with more and more 4k and 360 footage it's not going to get any faster any time soon

 

 

Video would theoretically be feasible but only if you had an enormous upload bandwidth. It seems like a very sensible idea because each GOP (group of pictures) can be encoded independently of all the others, so each node in the distributed encoder could handle it's own little queue of GOPs. I believe this is how multithreaded video encoding works as it is.

Say you want to encode 4K video, and assume you want to encode using a 24 frame GOP (meaning 24 frames between key frames) that would be 8 (megapixels) by 1.5 (bytes per pixel, assuming 4:2:0 YUV encoding) by 24 (frames) which would be 288MB per GOP. A quarter of a gig. In a typical YouTube lets play you would probably have 10,000s of GOPs. Even with a business grade 1Gbps upload, for all the time it'd take uploading to your distributed encoding network you might as well do it locally.

In addition, once all the encoding is complete you would need to receive all the encoded data and put it back in order. Not an easy task, and again bandwidth constrained.

Netflix does use a similar design in their Amazon AWS based encoder, which is distributed. However, this encoder is distributed on a local/private network, so much faster, lower latency, less traffic. And they need to handle encoding to many different quality/resolution levels.

Ooohhhh, now i understand... 

I forgot how large those files could be