Apple Abandons JPEG In Favor of HEIF/HEVC

[LAwLz]

Why is this not all over the news? It's huge news and yet it was not mentioned in MKBHD's nor Linus' video (WANK show) about iOS 11.

 

 

TL;DR:
Apple is moving away from from a really bad, 24 year old image format.

The new format is about twice as efficient as JPEG, and supports a lot of great features.

I am worried how this will play out because of licensing issues. Will this lead to another format war?

 

 

 

Long version:

 

Don't know how many people here cares about media formats, but I care a lot.

At the WWDC a few days ago, Apple quickly mentioned that they will start using HEVC for video recording. In the same segment, they also mentioned that they will start saving pictures taken on your iPhone as HEIF files, encoded with HEVC. The move to HEVC for video is not that interesting, and hopefully Apple will move over to AV1 once that is released, but what is important and interesting is HEIF.

 

HEIF (pronounced HEEF) stands for "High Efficient File Format". It's an image container format with a ton of very great features. Apple will start saving pictures taken with iOS devices' cameras as HEVC encoded image inside HEIF files. MacOS High Sierra and iOS 11 will support it.

JPEG is old and should have been replaced years ago. Google tried with WebP but the lack of support in OSes and applications lead to it just creating headaches for users and it has mostly faded away into obscurity. With Apple's tremendous control over their iOS ecosystem, it is very possible that they will implement convertion from HEIF to JPEG/PNG in an elegant way which is transparent to the users. Once the new format has gotten traction, more and more OSes and applications might start adding native support for it and it might become a true replacement for JPEG.

 

It's important to understand that HEIF is a container, not a codec. The specifications does however include special instructions on how to encapsulate HEVC images into HEIF-complaint files. Apple has stated that they will start using HEIF as a container, and HEVC as the codec for images saved within the HEIF container. What this means is that it should be possible to save AV1 encoded images inside a HEIF file in the future.

 

To break it down:

HEVC - A codec. It specifies how to make that 1GB video into a 500MB video, without losing too much image quality.

HEIF - A container. It's like wrapping paper for your files. It allows you to put one or more files, which may have different codecs, inside a single file, and it tells your programs how to interpret these files. For example .mp4 is a container, and it makes it possible to bundle the video file and audio file into a single file which your video player understands how to play.

HEVC as an image format - Since HEVC is meant for video, it does not include any of the features needed to be used as a still image format. HEIF fixes this by including special specifications on how to encode a single image using HEVC into a HEIF-complaint file.

 

 

 

HEIF container info and benefits:

• Developed by MPEG, so it is not Apple exclusive. It's developed by the same group that created .mp4 (MPEG) and is an ISO standard.
• Extremely flexible (this can also be a drawback as we have seen with MKV, since supporting it can be complex).
• Support for multiple images in the same file (for example, if you burst shoot on your phone then all images can be saved in the same file, instead of individually). This also means that you can save an entire gallery as a single file. Wanna send 10 images from your trip to your mom? Send her a single file containing all the images, instead of 10 individual files.
• "Derived images" support, which means you can do non-destructive images editing. When a HEIF image is edited, the standard allows for the modifications to be saved as transform operations, in the metadata, instead of as a brand new image. This means that you get to keep the original image intact, as well as a modified version in the same file (with extremely little overhead). The types of native edits you can do are fairly limited (cropping, rotation, overlays and image grids), but it is possible for programs to write their own instructions. This could remove the need for special file formats such as Photoshop's psd.
• It is possible to have multiple images in the same file mix together to create a single image that is presented to the user. The most common example of this will most likely be HDR images. Right now, when you take an HDR photo you either get multiple, separate images (for example one low exposure, one with high exposure, and one that's mixed together) or the low and high exposure images are thrown out and you only get the mixed one. With HEIF, it will be possible for a HDR shot to output a single file that contains all three aforementioned versions, while only being a single file and only needing to save data for the high and low exposure shots.
• Support for several different metadata types (including Exif and XMP) and can be extended to support more.
• Other types of media such as video, audio and subtitles can be saved inside the HEIF file as well. This is very hand if we want to use it to replace animated gifs.
• Thumbnails can be embedded into the same file as the master image. I am not sure how this will work, but my guess is that it will be possible for applications to request only the thumbnail image, instead of needing to download the entire file to access the thumbnail. If this is the case, then it will be very useful for developers. For example websites would no longer need to save thumbnails and full resolution images as separate files.
• There is a field for "looping". This is useful if you want to create looping animated images (think like gifs), but you can also write a number to this field which means the animation will only loop for a certain number of times.

 

As you can probably tell by the long list, this format has many benefits over jpeg, gif and PNG which are the three most common image containers (and codecs) used today. It is clear that MPEG looked at what features are needed in modern devices and wanted to create a format that supports all of them. Not only that, but a lot of the specifications are very flexible and open, which will allow future changes or third party developers to adapt the format to suit their needs (such as Adobe embedding Photoshop transformation information inside it).

 

But like I said, HEIF is just the container. Apple also said that they are going to encode the images in the HEVC format. Why?

 

HEVC as image format benefits and features:

• Again, this is an ISO standard and is in no way limited to only Apple.
• For still images, HEVC is on average twice as efficient as JPEG. This means that your images will either be half the size they would be if they were in the JPEG format at the same quality, or you can increase the quality, or a combination of both. Samples can be found here.
• Since it is a video format, it can be animated and it will be far better than gif at it. It supports more colors and is much more efficient. There are two comparisons between HEIF/HEVC and gif in the link I posted above.
• Can be lossless (and thus, it can replace PNG too).
• It supports transparency.
• The same hardware acceleration used for video playback can also be used to decode and render images. This means that decoding the images will use less power and be faster than JPEG, gif and PNG. Even on devices which does not support hardware acceleration of HEVC, the fact that it is HEVC and based on slices allows decoding to be heavily parallelized, and more flexible. So while you might not save any power, it will still be fast to decode. It will also be possible to prioritize individual slices which may be more relevant to the user than other slices, instead of the standard "always render from top to bottom" we have today with JPEG and PNG.

 

The one thing that worries me, is that HEVC is a licensing nightmare. Compared to H.264, the per-unit fees are in some cases 10 times as high for HEVC. There is no annual fee cap anymore, as well as a 0.5% royalt on revenue generated from HEVC video services. All of those things combined meant that for Mozilla and many other companies, moving from H.264 to HEVC would increase their yearly licensing cost by about 10,000% (100 times as much). For companies like Netflix, Amazon and Google which relies even more on video services, the cost increase would have been even larger.

In fact, HEVC licensing is such a big issue that Amazon, Cisco, Google, Intel, Microsoft, Mozilla and Netflix (and many more joined later) all got together to create a new format (AV1 which I mentioned earlier) which will be open and free to use. Apple however, is not in this alliance. This move, is signaling that Apple are not interested in moving to the free and open AV1 format.

 

AV1 might be supported in HEIF in the future, but changing formats is not some small task and I doubt Apple would want to change from HEIF/HEVC to HEIF/AV1 in a year or two. Since HEVC and AV1 are also competitors, we might run into a situation where MPEG, the creators of HEIF and HEVC will make it more convenient to use HEVC over AV1 inside HEIF.

So what we might get is a format war between Apple and MPEG with HEIF/HEVC vs essentially the rest of the industry with a superior video format, but no image format alternative. Even if another standard using AV1 for images gets developed we will end up in a situation where we got two competing formats trying to solve the same issues (and HEIF/HEVC will have 1-3 year head start).

 

 

Thoughts?

Personally I think it is great that we are finally seeing a large company move away from JPEG in a graceful way. My only problem is that it is relying on HEVC which I consider essentially dead because of the AV1 codec which will hopefully be finalized in late 2017 (Q3 or Q4), with hardware support for it released in 2018.

What I want to see is HEIF extended to support AV1 so that we can get the best of both worlds. Sadly, that will probably not happen and in that case I hope that AOMedia will reinvent the wheel by creating their own image format.

[mr moose]

1 minute ago, CUDA_Cores said:

I didn't read all this, but I already know why it's not covered. It's because apply generally holds all technology to themselves and doesn't allow others to use it or pay huge liscensing fees. It's one of the biggest reasons firewire never took off in the mid 2000s when there was already USB around that was royalty free. 

 

Just because something is better doesn't mean it's widely used. 

 

ahem... betamax

What do you mean firewire didn't take of?   Firewire is essential for a lot of adat/mutlitrack audio interfaces of the time (USB was just too slow/laggy).  Not mention it was a popular replacement for parallel scsi.

[sazrocks]

10 minutes ago, CUDA_Cores said:

I didn't read all this, but I already know why it's not covered. It's because apply generally holds all technology to themselves and doesn't allow others to use it or pay huge liscensing fees.

What? Apple does not own HEIF/HEVC.

[sazrocks]

Just now, CUDA_Cores said:

OK then, I am misinformed. Then who does own it? Reading the TL:DR did not provide that information.

Read the OP...

[crystal6tak]

Read the post, got SUPER excited reading the advantage and the seeing examples you linked, then got SUPER bummed out by the licensing fees...

 

Man, I didn't know how far behind (or how much room for improvement there is) for JPEG. Hopefully AV1 comes up with a image solution before HEVC takes over.

 

EDIT:

Might be off topic, but how does VP9 fit in all this? I don't know much about it, but isn't that another royalty free video codec google is working on? People said it would take over years ago... And now they're working on AV1? So VP9 failed or something?

[PlayStation 2]

This is also what we should be doing with MP3.

[captain_to_fire]

46 minutes ago, LAwLz said:

Thoughts?

Personally I think it is great that we are finally seeing a large company move away from JPEG in a graceful way. My only problem is that it is relying on HEVC which I consider essentially dead because of the AV1 codec which will hopefully be finalized in late 2017 (Q3 or Q4), with hardware support for it released in 2018.

What I want to see is HEIF extended to support AV1 so that we can get the best of both worlds. Sadly, that will probably not happen and in that case I hope that AOMedia will reinvent the wheel by creating their own image format.

This reminds me when HTML5 is starting to get attention around 2010 and browser vendors are battling which codec to use for video: H.264 or WebM. Apple and Microsoft used H.264 whereas Google and Mozilla used WebM.

[NumLock21]

Any new type of format that can compress a file to a smaller size is a plus in my book.

Full stream if anyone is interested on watching

https://www.apple.com/apple-events/june-2017/

[mariushm]

We already had BPG (image format based on HEVC intra frame, up to 14 bit per color or lossless, one frame only, backwards compatible with jpeg color formats etc) : https://bellard.org/bpg/

 

It's just a licensing thing, how to make more money from hevc patents.

 

It's made by MPEG and naturally it's based on existing mp4 container just with a bunch of restrictions (like using only hevc intra frames , still picture "profile" that can already be used separately when compressing individual frames)

The container can pack one or multiple intra frames, can pack audio and subtitles as well inside this format .. it's a mess.

And while the container is called HEIF  where HE is from HEVC, the format is open to having other video codecs to compress stuff in the future.

 

Basically it's like TIFF all over again, and in order to decode one image you need to license the container and license hevc decoder and you need a basic mp4 parser just to extract single frames to be decoded and all this crap for what, saving 10-20% of an imager size ?

 

The real benefit for apple is that they're saving some battery power by using the hardware hevc encoder to encode one image or a burst of images as intra frames instead of using the cpu to compress... and they're already paying for hevc in the form of the chips already having this functionality inside (encoding and decoding hevc in hardware)

 

 

[Zodiark1593]

52 minutes ago, crystal6tak said:

Read the post, got SUPER excited reading the advantage and the seeing examples you linked, then got SUPER bummed out by the licensing fees...

 

Man, I didn't know how far behind (or how much room for improvement there is) for JPEG. Hopefully AV1 comes up with a image solution before HEVC takes over.

 

EDIT:

Might be off topic, but how does VP9 fit in all this? I don't know much about it, but isn't that another royalty free video codec google is working on? People said it would take over years ago... And now they're working on AV1? So VP9 failed or something?

VP9 appears to fall somewhere in between HEVC and AVC, closer to the latter. Personal encode experiences have been pretty terrible. 

 

The licensing fees are pretty terrible for HEVC, though if we start seeing traction toward AV1 for video, my thoughts on building HTPC's with a strong cpu for software decode still stands. 

[Mooshi]

Good. Jpeg is horrible and no one should be using it when making images.

[Dredgy]

3 hours ago, Dan Castellaneta said:

This is also what we should be doing with MP3.

Apple hasn't used mp3 for...a long time. They started phasing it out in like 2003 I think.

 

As for the HEVC images. Great. The iPhone is the most popular camera on the planet, so shouldn't be long at all before all the necessary platforms start to support it.

 

AV1 is not going to be popular for consumer use any time soon, if at all. For serving web video, yes, but as a capture and editing and format, not likely at all.

[PlayStation 2]

Just now, Dredgy said:

Apple hasn't used mp3 for...a long time. They started phasing it out in like 2003 I think.

 

As for the HEVC images. Great. The iPhone is the most popular camera on the planet, so shouldn't be long at all before all the necessary platforms start to support it.

 

AV1 is not going to be popular for consumer use any time soon, if at all. For serving web video, yes, but as a capture and editing and format, not likely at all.

Yeah, Apple never really pushed MP3, or has for a very long time. What I probably should've cleared up if that others should be phasing out things like JPEG and MP3.

[mariushm]

Apple used AAC because it allowed for DRM ... also a company like Apple can pay a one time fee like let's say 5 million dollars a year and not worry about licensing costs (paying per audio file sold, or paying a few pennies for every 1000 plays over 30s on iTunes ... ) . 

 

For example, that's how Cisco  makes the OpenH264 decoder available as a plugin for Firefox and other browsers - they have so many products which use this decoder (webcams, voip phones etc) that they just pay a huge one time fee every year or every few years which allows them to distribute the decoder for free in a browser like Firefox.

 

Apple is part of those licensing organizations because they hold some patents that are part of aac and/or h264 so they're basically getting a discount for having patents in those patent pools and they're paying themselves in a way when they pay the licensing costs.

 

Apple could use MP3 now if the most important thing was licensing costs - they could easily just switch from 160kbps AAC to 320kbps MP3 and not pay a few million dollars a year in licensing or whatever they pay now... but those licensing costs are nothing for a company like Apple.

 

Remember there's also Opus which is completely free and open source and has better quality than AAC and ogg vorbis and various other encoders. Most browsers already have decoders for it  and you often find it used in combination with VP9 and webm containers : https://opus-codec.org/comparison/

 

They didn't stay away from HEVC because something else was better, they most likely stayed away from it because they didn't have hardware encoders and decoders for it.  Now that the RX 4xx series has hardware encoders and decoders for HEVC and now that the new Apple "pro" stuff comes with VEGA which also has hardware encoders and decoders for HEVC , of course they're accepting HEVC. 

The Polaris as far as I know has only "hybrid" support for VP9 decoding, it's not 100% hardware decoded, and there's no hardware encoder for VP9 in the video cards. Therefore, that could be a reason why they don't support it.

AV1 is not yet completed as specification, they're still testing to figure out what algorithms bring enough quality improvement in order to justify the extra amount of time required to encode video and if they justify the extra amount of transistors and silicon space that such algorithms would introduce in hardware encoders.

It would take years to make hardware decoders for AV1 even if they start designing the chips from preliminary documents , drafts etc so there's no point discussing AV1 yet.

 

[Sniperfox47]

Ffs I wish all of these standards organizations making codecs behind massive royalty paywalls would just die. Apple's pushing for HEVC images specifically because they're a MPEG member and get something out of it.

 

I get that you're a company that needs to make money, but open standards that everyone can use are good for everyone including you, they help spread adoption which means less frustration on users part when a given device or software can't open your file because the dev couldn't afford the ridiculous licensing payout.

[mr moose]

5 hours ago, CUDA_Cores said:

Do you see Firewire widely used today?

 

No. 

 

It's been superseded. I'm not saying Firewire wasn't used back then. But aside form a few situations like you mentioned, how often do you see Firewire used today?

All the time:

 

https://www.google.com.au/search?q=firewire+audio+interface&ie=utf-8&oe=utf-8&client=firefox-b&gfe_rd=cr&ei=9Ok8WbOaBa_M8gfgnIO4BA#q=firewire+audio+interface&tbm=shop

 

Even though firewire has been superseded by other tech (which happens to all tech) it was far from your claim that it "never took off"

[Dabombinable]

6 hours ago, mr moose said:

What do you mean firewire didn't take of?   Firewire is essential for a lot of adat/mutlitrack audio interfaces of the time (USB was just too slow/laggy).  Not mention it was a popular replacement for parallel scsi.

Some people just don't understand the nightmare 11Mbps USB 1.1 used to  be. Even after the kinks got ironed out.

[mariushm]

Firewire has some problems by design , problems that can't be worked around while keeping backwards compatibility. These problems are one of the reasons you don't see them in modern operating systems.

 

See https://en.wikipedia.org/wiki/IEEE_1394#Security_issues and  https://en.wikipedia.org/wiki/DMA_attack

 

USB was never intended to be a firewire replacement or even compete with it.

It was from the start designed to be a low bandwidth thing , 4 wire (cheap) half - duplex , "simple" method of connecting peripherals like mouse, keyboard or other devices that could live with low bandwidth like paper optical scanners for example , where connection faster than parallel port which has around 100 KB/s and simpler than scsi would be nice, but it wouldn't be end of the world if you don't have speeds as high as scsi..

It was also meant to be used point to point, and work with hubs while firewire is only direct connection or at best pass through or something like that, i forget now.

 

You could probably easily make something like external pci-e x1 as replacement for firewire and get the firewire low latency and all that was good about it, without the downsides.

[mr moose]

1 hour ago, Dabombinable said:

Some people just don't understand the nightmare 11Mbps USB 1.1 used to  be. Even after the kinks got ironed out.

 

The last time I recorded was 18+ channels, Apart from being stupid to do on USB I don't think it's even possible.

[Dabombinable]

16 minutes ago, mr moose said:

 

The last time I recorded was 18+ channels, Apart from being stupid to do on USB I don't think it's even possible.

You'd probably cause the computer to lock up. The USB 1.1 ports on my Gigabyte GA-5AA do that if you try to use more than 2 devices-even those with their own power supply-at once on either port. And the less said about the now thrown out "cause anything using the 5V rail to overheat" PCI USB 2.0 card (NEC based) the better.

[mariushm]

Back then, no.

 

These days, depends on what those 18 channels record.

 

USB 1.1 full speed is 12 mbps ...  but i'm not sure how much of that bandwidth was actually possible to use by one device... because you have up to 1000 frames per second, and each frame can have 1024 bytes payload.. then theres isochronous transfers which work differently... it's complicated.

 

FLAC can do maybe 400kbps average per channel of music , around 700 kbps average for stereo audio... lossless.

For voice (ex you cut anything below let's say 50hz and anything above 15 khz) you could probably get it within 250kbps per channel

 

Nearly lossless, Opus can probably do 96 kbps per channel of music (nearly transparent) , maybe 160 kbps for stereo would be near lossless.

For voice, you could get very good results (nearly transparent) in as little as 48-64 kbps.

 

raw...

 

Yeah, if you record 16 bit 48000 Hz you basically have 96,000 bytes per second per channel.  That's 768000 bits or 768kbps. If you do 24bit that's more data... 1152 kbps.

With maybe real time compression down to around 80% using something lossless like zlib / lzo / 7z you could maybe squeeze it in 600 kbps ... for 10 mbps you could maybe get 15-16 audio channels in real time.

 

-

 

BUT there were alternatives like ETHERNET cards back then , which could reliably do 10mbps or 100 mbps without any problems, using equally cheap cables.

[porina]

The thing about "better" image formats replacing jpeg generally boils down to 3rd party app support, or specifically the lack thereof. Jpeg isn't the best, even when it was new, but it is still good enough for a lot of uses. If only Apple goes for this container/encoding, it'll probably remain niche within that ecosystem. If you want to do something smart, there are already alternative containers to do that, but they tend to be niche specific. 

 

My prediction: outside of Apple specific stuff, this will have very limited to no significant impact.

[Jito463]

Personally, I've preferred PNG for years now.  I can't remember the last time I saved anything in JPG format.

[suicidalfranco]

9 hours ago, CUDA_Cores said:

I didn't read all this, but I already know why it's not covered. It's because apply generally holds all technology to themselves and doesn't allow others to use it or pay huge liscensing fees. It's one of the biggest reasons firewire never took off in the mid 2000s when there was already USB around that was royalty free. 

 

Just because something is better doesn't mean it's widely used. 

 

ahem... betamax

you should have read the entire post then

[Valentyn]

3 hours ago, Dabombinable said:

Some people just don't understand the nightmare 11Mbps USB 1.1 used to  be. Even after the kinks got ironed out.

Aye, FireWire 400 was USB2 speeds in a USB 1.1 era, and then Firewire 800 dropped.

It was a god sent for data transfer, and audio visual work.

Hell when Apple start putting Thunderbolt 1 on their computers they still sold Firewire devices and peripherals, and the audio hardware industry is only now really phasing it out completely.

It's weird seeing Focusrite and some others move to Thunderbolt though.