youtube AV1 & HEVC RTMP Coming To YouTube Soon (Beta)

[V3ptur]

YouTube and OBS are working to develop support for AV1 and HEVC RTMP. This means that later down the line (fairly soon) livestreaming on YouTube will support H.264, H.265 and AV1 live ingest over RTMP. According to some documentations it is still in beta testing but progress seems to be getting closer to AV1 and HEVC livestreaming support. 

 

There also seems to be plans coming from YouTube to later support HDR livestreaming now that the implementation of AV1 and HEVC are in the works. 

 

AV1 has also been tested on NVENC as well as Intel encoders.

 

 

Summary

Progress for AV1 and HEVC for live ingest over RTMP is making fast progress with it going into Beta. 

 

Quotes

Quote

 This change updates the RTMP implementation in OBS to support HEVC and AV1. It additionally enables these codecs for RTMP ingestion to YouTube. Note that this YouTube feature is also in beta.

Quote

• Tested using FFmpeg as the RTMP server, and FFplay as the client.

• Tested against YouTube’s RTMP server, up to 4K 60fps.

• Tested AV1 live encoding using NVENC and Intel encoders.

My thoughts

I think having RTMP support for AV1 and HEVC is gonna change the live streaming space for the better. Having better quality optimisation for lower bandwidth streaming is gonna make live streaming more accessible for people as well as for the viewers (countries which have poor internet infrastructure). 

 

However, I am curious with how sophisticated HEVC is compared to H.264 are the latency times gonna increase already on top of what we are currently experiencing now? 

 

Is AV1 gonna allow for much closer real time latency for higher resolutions like 4K60? 

 

Overall, I think its great to see new standards be implemented within the livestreaming space and with new hardware like the RTX 4000 series it will be interesting to see how their dual AV1 chips as well as on Intel come into play. 

 

Sources

https://github.com/obsproject/obs-studio/pull/8522

https://github.com/veovera/enhanced-rtmp

 

[grg994]

1 hour ago, V3ptur said:

 HEVC is compared to H.264 are the latency times gonna increase[?] [...] Is AV1 gonna allow for much closer real time latency[?]

When people talk about "low latency" modes in video codecs they refer to the ability to support applications like video-conferencing where real-time 2-way communication is desired and ~100ms of encoding delay is problematic.

The term latency has a completely different meaning in live stream broadcasting where propagation through the content-delivery network adds an order of magnitude larger delay then what comes from the video codec. (eg. on the viewer's side we have the HLS protocol, where the (half-)target-duration which regulates how often the client polls the server for new media segments is usually 1-2 sec minimum, and that is just for the last trip that the video data takes). So AV1 can't really change that.

[SorryBella]

6 minutes ago, grg994 said:

So AV1 can't really change that.

What AV1 would change would be in efficiency really, AFAIK.

[grg994]

Well, for AV1 in live streaming I am curious if it will be able to provide significantly better quality for a couple niche scenarios. If one watches a gaming stream of an FPS game where the player moves around in an area with dense grass, the whole screen is blurry, H264 with the usual 8-9000 Mbps limit on Youtube and Twitch just cannot handle that type of scenes.

 

Generally gameplays of FPS games are just a much more difficult target for video encoding because of the nature of the camera movements. Maybe AV1 will be able to do better there? (Does anyone have some insights on this?)

[Kisai]

1 hour ago, V3ptur said:

 

Quotes

My thoughts

I think having RTMP support for AV1 and HEVC is gonna change the live streaming space for the better. Having better quality optimisation for lower bandwidth streaming is gonna make live streaming more accessible for people as well as for the viewers (countries which have poor internet infrastructure). 

 

However, I am curious with how sophisticated HEVC is compared to H.264 are the latency times gonna increase already on top of what we are currently experiencing now? 

 

Is AV1 gonna allow for much closer real time latency for higher resolutions like 4K60? 

 

Overall, I think its great to see new standards be implemented within the livestreaming space and with new hardware like the RTX 4000 series it will be interesting to see how their dual AV1 chips as well as on Intel come into play. 

 

Sources

https://github.com/obsproject/obs-studio/pull/8522

https://github.com/veovera/enhanced-rtmp

 

So more bluntly, the holding back of HEVC injest has been a bit of a crime. We're already at it's half-life, and we waste so much gawd damn bandwidth using h264.

 

And some patents have been nuked:

https://www.unifiedpatents.com/insights/2023/1/4/checking-essentiality-10057603

 

One of the advantages H265 has, that has gone practically unused because of streaming tools not supporting it, is transparency IN the video stream, and HDR support. If you can support transparency in the encode, then you can start layering video streams over each other and only do the encode once, think about how much of an improvement that makes for compositing live streams together without having to deal with chroma-keys on the compositor side.

 

But I doubt this is why Youtube is doing it now. 

 

AV1 does not support transparency, which is a step backwards. However AV1 brings the reduced bandwidth that otherwise switching to h265 would do. And when you try to composite stuff in like Davinci? Well it does work. Too bad tools like OBS don't preserve it, and you have to do some really heinous work arounds like capturing the alpha-channel transparent video via NDI, convert to ProRes, and then waste 10GB/min of disk space working on it.

 

At any rate, as soon as injest is possible with AV1, Twitch better catch up, because not having to encode twice starts looking really advantageous to picking youtube over twitch. 

 

[V3ptur]

11 hours ago, grg994 said:

Well, for AV1 in live streaming I am curious if it will be able to provide significantly better quality for a couple niche scenarios. If one watches a gaming stream of an FPS game where the player moves around in an area with dense grass, the whole screen is blurry, H264 with the usual 8-9000 Mbps limit on Youtube and Twitch just cannot handle that type of scenes.

 

Generally gameplays of FPS games are just a much more difficult target for video encoding because of the nature of the camera movements. Maybe AV1 will be able to do better there? (Does anyone have some insights on this?)

Interesting point, you do see many games like Tarkov, Apex, CoD etc. where interpretting high volumes of foliage / detail can actually cause the streams quality to degrade losing the overall quality of the stream and the games visuals. 

 

I know with AV1 in order to increase accuracy for representing shapes etc. its based off using variable block-sizes which in theory should provide better detail to objects within the stream showcasing a better representation of the stream compared to other options like H.264.

 

I also remember them talking about AV1 having warped motion compensation which should in theory be able to provide a more accurate representation of motion in streams, given how active fps games are this should significantly improve stream quality for fast paced fps games or high motion visuals. 

 

Theres a lot to AV1 and its algorithms / techniques, but it should be interesting to see when it launches. 

 

I think if YouTube gets there before Twitch it will accelerate the development process for Twitch and adoption for other platforms. I know 120fps 2K has already been tested in the past and according to some people the quality was impressive (had good feedback overall).

 

[Doobeedoo]

That's good. Can we see 120fps already? 

Also hopefully Twitch transitions soon too, certain scenes and fast motions in fps games is just horrid with current limiting bitrate, another thing they need to increase regardless. 

[WereCat]

8 hours ago, grg994 said:

Well, for AV1 in live streaming I am curious if it will be able to provide significantly better quality for a couple niche scenarios. If one watches a gaming stream of an FPS game where the player moves around in an area with dense grass, the whole screen is blurry, H264 with the usual 8-9000 Mbps limit on Youtube and Twitch just cannot handle that type of scenes.

 

Generally gameplays of FPS games are just a much more difficult target for video encoding because of the nature of the camera movements. Maybe AV1 will be able to do better there? (Does anyone have some insights on this?)

Yes it will. Twitch already did a demo of AV1 120FPS 1440p stream at 8Mbps and it looks amazing. (Not real time gameplay though since back then you could not encode AV1 so fast with any hardware).

 

https://www.twitch.tv/videos/637388605

 

edit:

 

just to add, don't expect this kind of quality from the HW accelerated AV1 encoders on the Intel, AMD and NVIDIA GPUs, it will still be miles ahead of x264 though.

[porina]

I hope I'm wrong but if you can get higher quality for a bitrate, I can see video platforms reducing bitrate for comparable quality, not increasing quality at existing bitrate.

 

Don't know how YT live streaming works, but Twitch don't offer transcoding universally. Bigger streamers (partner level) will likely have it enabled, but small streamers might not. So if you stream AV1 does that mean you might exclude viewers who can't decode it? I've already ran into a case where someone on potato internet couldn't even watch my streams.

[leadeater]

For platform providers the biggest benefit and driver for them is the reduction in bandwidth. It's really not about increasing video quality for the masses. Premium streaming partners will get access to higher quality, nobody else, at least not for long time.

[Sauron]

I kind of assumed this was already a thing, I can't imagine why it took this long... hardware H265 encoders have been commonplace for a while

[WereCat]

1 hour ago, Sauron said:

I kind of assumed this was already a thing, I can't imagine why it took this long... hardware H265 encoders have been commonplace for a while

Because H.265 / HEVC requires licencing fees while AV1 is free.

[Sauron]

7 minutes ago, WereCat said:

Because H.265 / HEVC requires licencing fees while AV1 is free.

hasn't been a problem for h264...

[WereCat]

50 minutes ago, Sauron said:

hasn't been a problem for h264...

They both use very different licences. H.265 is significantly more expensive as well.

 

 

AVC/h264

https://www.mpegla.com/wp-content/uploads/avcweb.pdf

 

HEVC/h265

https://www.mpegla.com/wp-content/uploads/HEVCweb.pdf

https://www.thebroadcastbridge.com/content/entry/11204/to-deploy-hevc-users-must-choose-what-patent-pool-to-dive-into

[Sauron]

39 minutes ago, WereCat said:

They both use very different licences. H.265 is significantly more expensive as well.

 

 

AVC/h264

https://www.mpegla.com/wp-content/uploads/avcweb.pdf

 

HEVC/h265

https://www.mpegla.com/wp-content/uploads/HEVCweb.pdf

https://www.thebroadcastbridge.com/content/entry/11204/to-deploy-hevc-users-must-choose-what-patent-pool-to-dive-into

why I hate patents, exhibit A...

[StephanTW]

good to see the biggest video providing company has started accepting AV1, this will pave the way for twitch and rumble to do the same. also better quality per kbps is always better for everyone, especially for those recording while streaming

[wanderingfool2]

22 hours ago, Sauron said:

hasn't been a problem for h264...

Aside from the massive patent cost per device (they also tried re-introducing cost for encoding based on revenue), there's also the practical elements to it.

 

The transition to h.264 was pretty significant.  H.262 (mpeg2) hadn't really been designed as an internet video streaming and compression technology was greatly increased.  In some cases the file size difference could be 80% (especially at low bitrates)...h.265 transition which is up to 50% but practically it's lower.  Then you have to factor in the amount of resources it takes to convert to h.264 vs h.265.  While there are dedicated encoders for it, it hasn't been around for nearly as long as the h.264 stuff which is now pennies on the dollar for equipment that can handle it.  Moving towards h.265 would have required investment in new equipment.

 

In general, there was much more of a need to move to h.264 as it was built more towards the web then mpeg2 was.

 

Finally you get to the decoders.  h.265 didn't have video card decoders really built into them until the 750 release, but it was on all cards by nvidias 1050.  That really does put a hamper on things, as those were only really released 6 years ago (plenty of hardware still out there that might be taxed handling h.265 streams).

[Sauron]

16 minutes ago, wanderingfool2 said:

That really does put a hamper on things, as those were only really released 6 years ago (plenty of hardware still out there that might be taxed handling h.265 streams).

Considering a pi 3 can software decode 1080p h265 quite smoothly I don't think a lot of devices would actually have issues. My 12 yo sandy x220 does it easily, too. If you have a 1440p or 4k screen, chances are your hardware is significantly better than that to begin with. I'm more convinced about patents being the obstacle and in that case I wish they'd have ignored h265 and gone straight for AV1, could have been a way to popularize the format.

[wanderingfool2]

26 minutes ago, Sauron said:

Considering a pi 3 can software decode 1080p h265 quite smoothly I don't think a lot of devices would actually have issues. My 12 yo sandy x220 does it easily, too. If you have a 1440p or 4k screen, chances are your hardware is significantly better than that to begin with. I'm more convinced about patents being the obstacle and in that case I wish they'd have ignored h265 and gone straight for AV1, could have been a way to popularize the format.

I have a few devices that struggle with h265 1080p 60fps (Nehalem).

 

Also, pi 3 depends...are you decoding 60fps video?  From what I've seen depending on the software it barely is able to pull off that.

[jasonvp]

On 3/26/2023 at 11:54 AM, porina said:

Don't know how YT live streaming works, but Twitch don't offer transcoding universally

 

That's because Twitch isn't allowed to use all of Uncle Jeff's entire infrastructure to universally transcode.  Youtube's streaming is allowed to use any part of GOOG's impressive infrastructure it needs.  Live, real time transcoding for all resolutions up to and including the one streamed.  For anyone and everyone.  And at a significantly higher input bit rate limit of 51Mbit/sec.

 

From a video perspective, Youtube streaming has been eons beyond anything Twitch has produced for quite some time.

 

 

 

[Sauron]

16 minutes ago, wanderingfool2 said:

I have a few devices that struggle with h265 1080p 60fps (Nehalem).

 

Also, pi 3 depends...are you decoding 60fps video?  From what I've seen depending on the software it barely is able to pull off that.

Admittedly not 60 fps but... it's a pi 3, the vast majority of people have much better hardware. Nehalem is what, 14 years old?

[tt2468]

New codecs are much more likely to result in higher quality streams, not lower bandwidth as one might assume. Users switching from AVC will continue streaming at the bitrate they are now (I've seen it already with my own eyes as someone working in this space), and on platforms like Twitch, you'll likely see similar bitrates being delivered to viewers.

 

 

On 3/25/2023 at 6:52 PM, V3ptur said:

However, I am curious with how sophisticated HEVC is compared to H.264 are the latency times gonna increase already on top of what we are currently experiencing now?

No, if anything they might get slightly better due to the lack of b-frame support in many encoders. More complex to encode/decode != more latency. Like GRG said too, many other factors affect latency like delivery pipeline and transcode stages.

 

On 3/25/2023 at 8:42 PM, Kisai said:

think about how much of an improvement that makes for compositing live streams together without having to deal with chroma-keys on the compositor side.

In my professional opinion, this is never going to happen on any notable scale. I think you may be confusing container format and codec, as I don't remember HEVC supporting multiple video tracks. Either way, compositing before final encode is always going to be more efficient than compositing on final decode so long as the final product (the presented stream) is the same.

 

On 3/26/2023 at 2:55 AM, Doobeedoo said:

Can we see 120fps already? 

As far as I can tell, not likely to happen on Twitch any time soon due to limitations of their infrastructure. YouTube, I have no idea.

 

 

Regarding HEVC licensing, it's my not-a-lawyer understanding that the primary limitation of delivery is that while a lot of devices do have hardware HEVC decoding, software HEVC decoding is effectively off-limits for most browsers. Some platforms like Windows do have the ability to "purchase" HEVC decode support, but it's generally not user friendly enough to be worthwhile. In some of the circles I'm in, a lot of the legal questions about what is and isn't allowed within the licensing weren't answered until very recently. The lack of HEVC adoption is 100% a result of the confusing licensing IMO.

 

 

I do also want to point out that Enhanced RTMP is effectively a stop-gap solution to there being no scalable replacement for RTMP currently available. An initiative called MoQ is being undertaken to create a protocol capable of replacing RTMP, but is considered to be more than a year away, so platforms have been looking for a way to make things like HEVC work in the short term.

[Doobeedoo]

22 minutes ago, tt2468 said:

As far as I can tell, not likely to happen on Twitch any time soon due to limitations of their infrastructure. YouTube, I have no idea.

 

 

Regarding HEVC licensing, it's my not-a-lawyer understanding that the primary limitation of delivery is that while a lot of devices do have hardware HEVC decoding, software HEVC decoding is effectively off-limits for most browsers. Some platforms like Windows do have the ability to "purchase" HEVC decode support, but it's generally not user friendly enough to be worthwhile. In some of the circles I'm in, a lot of the legal questions about what is and isn't allowed within the licensing weren't answered until very recently. The lack of HEVC adoption is 100% a result of the confusing licensing IMO.

 

 

I do also want to point out that Enhanced RTMP is effectively a stop-gap solution to there being no scalable replacement for RTMP currently available. An initiative called MoQ is being undertaken to create a protocol capable of replacing RTMP, but is considered to be more than a year away, so platforms have been looking for a way to make things like HEVC work in the short term.

Twitch already has 120fps and for a while now. I meant can we see 120fps for YT already since they slack there and AV1 for Twitch since they slack in bitrate.

[Kisai]

6 hours ago, tt2468 said:

 

 

In my professional opinion, this is never going to happen on any notable scale. I think you may be confusing container format and codec, as I don't remember HEVC supporting multiple video tracks. Either way, compositing before final encode is always going to be more efficient than compositing on final decode so long as the final product (the presented stream) is the same.

 

As far as I can tell, not likely to happen on Twitch any time soon due to limitations of their infrastructure. YouTube, I have no idea.

 

 

Nope. I can generate h265 videos with ffmpeg that I can composite in davinci without having to do ANYTHING to them. H264 I have to create a chroma key. ProRes and NDI SpeedCodec's, don't need to do anything, but Davinci won't support NDI.

 

What would be a killer application is having something like this:

Streamer A => | Composite overlaying | <= Streamer D

Streamer B => | [AmongUs, D&D, etc] | <= Streamer E

Streamer C => |                                    | <= Streamer F

Streamer D => |     A,B,C,D,E,F,G        | <= Streamer G

 

Because right now to do this, you need a chroma key and setup each 

person's stream individually within yours. So what you have a n^2 bandwidth

being consumed with more people in the stream.

 

Now if you don't have the chroma key in the way, you could have a compositor assemble the streamers onto whatever leaderboard/gameboard stream they are competing in without having them in their "own box". Vtubers have already demonstrated how well this can work, but the bandwidth cost is too expensive for a single streamer to do the compositing of, and when you have one streamer who lives 5 miles away and another who lives 2000 miles away, some crappy internet infrastructure is in the way. Now multiply that by 8.

 

What needs to happen for streaming is a way to "auto-crop" the transmission. If a streamer is only taking up the lower third of of their alpha-blended broadcast, then there is no reason to transmit the other two thirds. So the assembled scene is the same size, but the part of the video stream the streamer is not using is not transmitted since it's empty. Hell, this works even better in a CQP configuration where the it would know that it should be transmitting 1080p60, but only needs to use the lower right 600x800 pixels. 

 

But I digress, unless vtubers become more popular and there is more demand for supporting collaborative realtime entertainment, chances are alpha transparency in video will get the same treatment PNG transparency did back in the days of GIF.

 

And before you go "But NDI does this anyway", OBS does not support NDI input with audio. This is a problem with how OBS works.

 

[wanderingfool2]

4 hours ago, tt2468 said:

No, if anything they might get slightly better due to the lack of b-frame support in many encoders. More complex to encode/decode != more latency. Like GRG said too, many other factors affect latency like delivery pipeline and transcode stages.

On live streams encoders intentionally don't use b-frames if latency is a concern.  Supporting b-frames doesn't decrease latency (except maybe on stored video front in that you get better compression and already have it pre-calculated).

 

Google recommends keyframes every 2 seconds, so using b-frames would instantly add in 2 seconds of latency in that scenario.  (Since it has to wait for all the frames between the two keyframes to come in to calculate the b-frames).  It's why you would go with p-frames, as then you can start calculating and sending the p-frames immediately.