video compression

[apoyusiken]

so id like to archive kick stream from a few streamers but they take a lot of space and they are uncompressed or not so much compressed i believe? do you think it is feasible to compress locally? I am looking to write some code to automatically download and compress streams from watched streamers.

[GuiltySpark_]

If you have the ability to write code, hopefully you can help us answer your question by providing a MediaInfo dump of one of these files. That'll tell us what format they're in. Likely already compressed H264.

[apoyusiken]

7 minutes ago, GuiltySpark_ said:

If you have the ability to write code, hopefully you can help us answer your question by providing a MediaInfo dump of one of these files. That'll tell us what format they're in. Likely already compressed H264.

idk how to check that on mobile but let me tell you this : a 3 and a half hour video is 12.3 gb. i think thats excessive? im sure if i uploaded it to youtube and redownloaded, it would be significantly less.

[Needfuldoer]

You can use Handbrake to re-compress the files to a tighter codec (HEVC) and/or a lower bitrate. That will result in smaller files, but they'll be more noticeably compressed (especially if you use lower-bitrate H264).

[apoyusiken]

1 minute ago, Needfuldoer said:

You can use Handbrake to re-compress the files to a tighter codec (HEVC) and/or a lower bitrate. That will result in smaller files, but they'll be more noticeably compressed (especially if you use lower-bitrate H264).

yea i can figure that out but the question is more about the feasability. I guess that would be computationally expensive?

[apoyusiken]

 

[Needfuldoer]

1 minute ago, apoyusiken said:

yea i can figure that out but the question is more about the feasability. I guess that would be computationally expensive?

Either you'll need a GPU that has hardware accelerated HEVC encoding, or it's going to take a while on your CPU.

 

CPU encoding generally creates smaller files with fewer compression artifacts, but hardware encoding gets almost as small for a fraction of the time and power consumption.

[apoyusiken]

Just now, Needfuldoer said:

Either you'll need a GPU that has hardware accelerated HEVC encoding, or it's going to take a while on your CPU.

 

CPU encoding generally creates smaller files with fewer compression artifacts, but hardware encoding gets almost as small for a fraction of the time and power consumption.

will a 2070 do it? i wanna do this for like at least 30gbs of files a day and ideally while watching a movie or something on the pc

[Needfuldoer]

5 minutes ago, apoyusiken said:

will a 2070 do it? i wanna do this for like at least 30gbs of files a day and ideally while watching a movie or something on the pc

Yes, the generation of NVENC in your 2070 can do hardware accelerated H.265 (HEVC) encoding.

 

https://en.wikipedia.org/wiki/Nvidia_NVENC#:~:text=TU116-,TU106,-TU104

[GuiltySpark_]

You may need to play with the settings and do some test encodes, its entirely possible to end up with a file larger than what you started with if you go overboard. I don't like the idea of re-compressing H264 but if you don't care about the quality, you should be able to make it work. 

[wanderingfool2]

19 minutes ago, apoyusiken said:

so id like to archive kick stream from a few streamers but they take a lot of space and they are uncompressed or not so much compressed i believe? do you think it is feasible to compress locally? I am looking to write some code to automatically download and compress streams from watched streamers.

Just going to say this first, videos are rarely ever not compressed.

 

1080p videos, 1 single frame is 1920*1080 (pixels) * 3 (colors) * 1 (8 bits/1 byte per color) = 6220800 bytes.  Which is ~5.9MiB, or if you have 30 fps that's 177 MiB/second...or 10 GiB/hour (and that's not even including audio).  It's why things are always compressed

 

 

Anyways, onto the more helpful stuff, everything is going to depend on your personal preference in regards to how much "quality" you are willing to sacrifice in regards to saving space...and how much almost "horse power" you are willing to put into it.

 

Overall I would say think of what you would be okay with on a per hour basis.  Like for myself I usually like about 1.5 GiB of quality for every hour (it's the rate at which where I sit that I won't generally notice the quality loss except in certain scenes)...but at the same time when I get 1.5 GiB files, they typically aren't using the best type of compression either.  So for example HVEC with P and B frames (vs just P frames) I could probably do 1 GiB with the same output quality.

 

The issue with that (P and B frames means it looks both forward and back, instead of just back when calculating for compression) is that it takes longer to encode (and I'm not sure you can use GPU to speed things up...except in the decode, but the encode is what would be slow).

 

So anyways, this is what I would recommend.  Download something like ffmpeg (handbrake if you want a gui), then take a few streams that have a lot of action in it (quick camera pans in a changing light environment and specifically randomized noise like confetti  or that kind of idea).  That is where compression artifacts are the worst.  Then create like 10 different test files ranging in size, and then compare. (Modifying the different bitrates)...then load up the lowest quality one, see if it's acceptable...if not keep going until you find one that you could live with...then I would go to the next one and  personally likely use that (or if it's a small enough size I might go 2 up)

 

e.g. Using h264 changing bitrates

 

1000 kbits/second ~ 450 MB/h

1500 kb/second ~ 675 MB/h

2000 kb/second ~ 900 MB/h

2500 kb/second ~ 1125MB/h

3000 kb/second ~ 1350MB/h [this is about where I personally stand]

3500 kb/second

4000 kb/second

 

So the above is what I would do, but if you can easily run the newer compression formats I would try those as well to see image quality differences.

 

Overall I would say though, if space is the most important figure out how much space you are willing to use and then try different formats to see which produces the best results (without taking a long time).

 

e.g. I use hardware encode for a bunch of stuff because I can then encode at like 2x - 10x (so 10 hours of video could take as little as 1 hour of encoding)...but if I throw my CPU at it I get like 0.5x - 2.5x kind of thing...but I get so much better quality for space...but I don't want to wait like 4 hours for a 2 hour video.

[apoyusiken]

7 minutes ago, GuiltySpark_ said:

You may need to play with the settings and do some test encodes, its entirely possible to end up with a file larger than what you started with if you go overboard. I don't like the idea of re-compressing H264 but if you don't care about the quality, you should be able to make it work. 

yea i had experienced this before. also another idea is to just upload the streams to youtube and that way i wont worry about storage at all. 

The streamers do at least 10k bitrate and say they'd like more and i really dont see the point. Am i missing something? I'm happy to be englighted.

[wanderingfool2]

22 minutes ago, Needfuldoer said:

Yes, the generation of NVENC in your 2070 can do hardware accelerated H.265 (HEVC) encoding.

 

https://en.wikipedia.org/wiki/Nvidia_NVENC#:~:text=TU116-,TU106,-TU104

My biggest gripe with NVENC is that they haven't added any backward predictions into it...if they did that I would be so happy...but I think they really tailor it for people like streamers and such not wanting to add the backwards prediction because it would be such an underutilized feature while adding complexity...but oh boy would I love if they actually supported that one day because it can make files so much smaller or quite a bit better quality

[apoyusiken]

25 minutes ago, wanderingfool2 said:

Just going to say this first, videos are rarely ever not compressed.

 

1080p videos, 1 single frame is 1920*1080 (pixels) * 3 (colors) * 1 (8 bits/1 byte per color) = 6220800 bytes.  Which is ~5.9MiB, or if you have 30 fps that's 177 MiB/second...or 10 GiB/hour (and that's not even including audio).  It's why things are always compressed

 

 

Anyways, onto the more helpful stuff, everything is going to depend on your personal preference in regards to how much "quality" you are willing to sacrifice in regards to saving space...and how much almost "horse power" you are willing to put into it.

 

Overall I would say think of what you would be okay with on a per hour basis.  Like for myself I usually like about 1.5 GiB of quality for every hour (it's the rate at which where I sit that I won't generally notice the quality loss except in certain scenes)...but at the same time when I get 1.5 GiB files, they typically aren't using the best type of compression either.  So for example HVEC with P and B frames (vs just P frames) I could probably do 1 GiB with the same output quality.

 

The issue with that (P and B frames means it looks both forward and back, instead of just back when calculating for compression) is that it takes longer to encode (and I'm not sure you can use GPU to speed things up...except in the decode, but the encode is what would be slow).

 

So anyways, this is what I would recommend.  Download something like ffmpeg (handbrake if you want a gui), then take a few streams that have a lot of action in it (quick camera pans in a changing light environment and specifically randomized noise like confetti  or that kind of idea).  That is where compression artifacts are the worst.  Then create like 10 different test files ranging in size, and then compare. (Modifying the different bitrates)...then load up the lowest quality one, see if it's acceptable...if not keep going until you find one that you could live with...then I would go to the next one and  personally likely use that (or if it's a small enough size I might go 2 up)

 

e.g. Using h264 changing bitrates

 

1000 kbits/second ~ 450 MB/h

1500 kb/second ~ 675 MB/h

2000 kb/second ~ 900 MB/h

2500 kb/second ~ 1125MB/h

3000 kb/second ~ 1350MB/h [this is about where I personally stand]

3500 kb/second

4000 kb/second

 

So the above is what I would do, but if you can easily run the newer compression formats I would try those as well to see image quality differences.

 

Overall I would say though, if space is the most important figure out how much space you are willing to use and then try different formats to see which produces the best results (without taking a long time).

 

e.g. I use hardware encode for a bunch of stuff because I can then encode at like 2x - 10x (so 10 hours of video could take as little as 1 hour of encoding)...but if I throw my CPU at it I get like 0.5x - 2.5x kind of thing...but I get so much better quality for space...but I don't want to wait like 4 hours for a 2 hour video.

since when 177*3600 is 10000?

 

how does real time compression work??

 

i think streams dont need more bitrate than movies soo..

 

also how do 1080p photos exceed 6 mbs?

[Electronics Wizardy]

1 hour ago, apoyusiken said:

idk how to check that on mobile but let me tell you this : a 3 and a half hour video is 12.3 gb. i think thats excessive? im sure if i uploaded it to youtube and redownloaded, it would be significantly less.

I'd say thats a pretty low bitrate, but I'm used to > 100GB per hours from cameras. Storage is pretty cheap these days so I'd

 

But its pretty easy to convert to something like AV1, I'd avoid using GPU if small files with usable quality is your goal.

 

27 minutes ago, apoyusiken said:

also how do 1080p photos exceed 6 mbs?

What do you mean by 1080p photos? 1080p is generally only used for video.

 

But its all up to compression amount, resolution doesn't affect filesize.

[apoyusiken]

3 minutes ago, Electronics Wizardy said:

I'd say thats a pretty low bitrate, but I'm used to > 100GB per hours from cameras. Storage is pretty cheap these days so I'd

 

But its pretty easy to convert to something like AV1, I'd avoid using GPU if small files with usable quality is your goal.

 

What do you mean by 1080p photos? 1080p is generally only used for video.

 

But its all up to compression amount, resolution doesn't affect filesize.

 

wouldnt the cpu take way too long? 

[Electronics Wizardy]

Just now, apoyusiken said:

 

wouldnt the cpu take way too long? 

Depends on the settings, but I get ~1/2 realtime compressing to AV1 on my E5 2680 v4 system, and those are like 10 year old chips now.

[apoyusiken]

2 minutes ago, Electronics Wizardy said:

Depends on the settings, but I get ~1/2 realtime compressing to AV1 on my E5 2680 v4 system, and those are like 10 year old chips now.

once i had tried 720 to 1080p upscaling and it was not feasible

[Electronics Wizardy]

Just now, apoyusiken said:

once i had tried 720 to 1080p upscaling and it was not feasible

And that related to this how? Thats a completely different task.

 

I'd say just keep the full files, 12GB is pretty tiny, and HDDs are probably cheaper than power to compress them.

[porina]

1 hour ago, apoyusiken said:

a 3 and a half hour video is 12.3 gb. i think thats excessive? im sure if i uploaded it to youtube and redownloaded, it would be significantly less.

That works out at a bitrate around 7.8 Mbps. I don't know what Kick prefers but Twitch recommend a max of 6 Mbps so it is in the right ball park. While re-coding it to a better codec can save some space, if you want bigger savings you'll probably also have to look at lower resolution or quality settings in general.

[Electronics Wizardy]

4 minutes ago, porina said:

That works out at a bitrate around 7.8 Mbps. I don't know what Kick prefers but Twitch recommend a max of 6 Mbps so it is in the right ball park. While re-coding it to a better codec can save some space, if you want bigger savings you'll probably also have to look at lower resolution or quality settings in general.

Also really depends on the content. If its someone talking in front of a camera with little motion, you can squeeze it a good amount more, if its something like a game with a lot of detail and movement, I'd really want more bit rate and you may see significant artifacts at 8mbit already.

[apoyusiken]

7 minutes ago, Electronics Wizardy said:

And that related to this how? Thats a completely different task.

 

I'd say just keep the full files, 12GB is pretty tiny, and HDDs are probably cheaper than power to compress them.

bro still it was to give you an idea of my hardware. 

 

yea 12gb is tiny but i wanna download all streams of multiple streams and i will eventually run out of space. 

[apoyusiken]

1 minute ago, Electronics Wizardy said:

Also really depends on the content. If its someone talking in front of a camera with little motion, you can squeeze it a good amount more, if its something like a game with a lot of detail and movement, I'd really want more bit rate and you may see significant artifacts at 8mbit already.

the stream consists of both but i dont care about a most games

[Electronics Wizardy]

1 minute ago, apoyusiken said:

bro still it was to give you an idea of my hardware. 

 

yea 12gb is tiny but i wanna download all streams of multiple streams and i will eventually run out of space. 

Buy a few big hdds? Again likely cheaper than the power needed to cmopress.

 

But why not run some compression tests on your own and see how it looks when compressed and how long it takes on your hardware?

[apoyusiken]

Just now, Electronics Wizardy said:

Buy a few big hdds? Again likely cheaper than the power needed to cmopress.

 

But why not run some compression tests on your own and see how it looks when compressed and how long it takes on your hardware?

currently just planning this out. as i said i also consider uploading to youtube.