[UPDATE] Wonderlust – Apple September event; new iPhones and watches

[Obioban]

1 hour ago, dalekphalm said:

I don't agree with you here. In practice, very very few people are going to use Live Photos to "capture the essence of the moment". That might be what you specifically use it for, but Apple advertises it as a way to "pick the key photo and make edits", just like I stated.

I doubt most users would even notice or care, to be honest. It sounds like you use Live Photos in a particularly niche way.

I suspect people use it for just that all the time, because the iOS generated memories often use the video portion of them.

 

I feel like Apple's marketing of them is pretty in line with what I was saying:

https://developer.apple.com/design/human-interface-guidelines/live-photos

 

Quote

Live Photos lets people capture favorite memories in a sound- and motion-rich interactive experience that adds vitality to traditional still photos.

[Dracarris]

I‘ve also never heard of Live with the purpose to chose the best frame - how should that even work given that every video format will be in severely lower quality compared to a still image?

 

Also, I‘m not sure the live video is actually stored inside the HEIC. Maybe it‘s simply linked in from an external file?

[dalekphalm]

59 minutes ago, Dracarris said:

I‘ve also never heard of Live with the purpose to chose the best frame - how should that even work given that every video format will be in severely lower quality compared to a still image?

"Severely lower quality",  maybe - but the quality is still pretty high - I've personally never noticed a major difference when I've edited a live photo and chosen a different key frame. It's a choice you get to make as a user.

59 minutes ago, Dracarris said:

Also, I‘m not sure the live video is actually stored inside the HEIC. Maybe it‘s simply linked in from an external file?

I couldn't tell you how it works from a technical perspective, but everything is in the one file, since if I share that file to another iPhone user, they can see the Live Photo in full. Same if I download a backed up version of the file.

 

According to this very old article from 2015, the iPhone 6s (where Live Photos were introduced) takes a series of still images, animates them together, and records audio separately, then stitches the whole thing together.

https://9to5mac.com/2015/09/11/how-live-photos-works/

 

If this is accurate, then there's literally no downside to this, as all the frames will be the same quality as the original key frame.

 

Given that's how it's being described, I see zero issues with Apple using the same method to create a Live Photo using JPEGXL or any other newer format.

[Obioban]

17 minutes ago, dalekphalm said:

"Severely lower quality",  maybe - but the quality is still pretty high - I've personally never noticed a major difference when I've edited a live photo and chosen a different key frame. It's a choice you get to make as a user.

I couldn't tell you how it works from a technical perspective, but everything is in the one file, since if I share that file to another iPhone user, they can see the Live Photo in full. Same if I download a backed up version of the file.

 

According to this very old article from 2015, the iPhone 6s (where Live Photos were introduced) takes a series of still images, animates them together, and records audio separately, then stitches the whole thing together.

https://9to5mac.com/2015/09/11/how-live-photos-works/

 

If this is accurate, then there's literally no downside to this, as all the frames will be the same quality as the original key frame.

 

Given that's how it's being described, I see zero issues with Apple using the same method to create a Live Photo using JPEGXL or any other newer format.

That's not accurate/how it works. There is A picture file, and a much lower resolution video.

 

If it was a series of full quality photos animated smoothly, you'd have MASSIVE files every time you took a picture.

[hishnash]

11 hours ago, LAwLz said:

It does support depth layers though. In fact, it supports 4099 channels, and a typical RGB image will only use 3. So you have 4096 channels remaining to fill with whatever data you want. Alpha channel for transparency, depth channel for depth data, maybe a thermal channel if you got a thermal camera. In that way, the format is very flexible. 

So no video side channel (I use this to store other ... non video vector data), from looking at apis I can fine it looks like the other channels however also need to be the same x,y size as each other you cant provide each with its own size/offsset/metadata.    Having the flexibility is nice as an app developer that can use it to store some of the original source data for an email within it so if users re-open the image within an app that edited it you can undo/alter old edits. 

[hishnash]

1 hour ago, dalekphalm said:

If this is accurate, then there's literally no downside to this, as all the frames will be the same quality as the original key frame.

 

Not all frames are the same quality at all, there are a few key frames that are of high quality and the rest are deltas applied to that using HEVC video encoding.  If you were to just do 60fps for 5 seconds at 24MP (10 bit) in JEPG-XL you would be looking at MASIV files as the standard does not do any frame to frame compression like a vide format does. 

[dalekphalm]

1 hour ago, Obioban said:

That's not accurate/how it works. There is A picture file, and a much lower resolution video.

 

If it was a series of full quality photos animated smoothly, you'd have MASSIVE files every time you took a picture.

Feel free to provide counter evidence to how it works.

 

My point being that in practice, I've yet to notice any serious quality degradation when choosing a new key frame.

[Obioban]

16 minutes ago, dalekphalm said:

Feel free to provide counter evidence to how it works.

 

My point being that in practice, I've yet to notice any serious quality degradation when choosing a new key frame.

On photos for Mac, select a live photo. File, export unmodified original. Choose destination, click export. What you get at that destination is a video file and an image file.

 

Original photo:

 

Changed key frame:

[dalekphalm]

1 minute ago, Obioban said:

On photos for Mac, select a photo. File, export unmodified original. Choose destination, click export. What you get at that destination is a video file and an image file.

Thank you for explaining how you got your result - that is useful information. Granted, I'm not a Mac user, so this is of limited use to me personally, but I will concede the point that it's an image file with a video file.

 

That doesn't change my stance mind you.

[LAwLz]

1 hour ago, hishnash said:

So no video side channel (I use this to store other ... non video vector data), from looking at apis I can fine it looks like the other channels however also need to be the same x,y size as each other you cant provide each with its own size/offsset/metadata.    Having the flexibility is nice as an app developer that can use it to store some of the original source data for an email within it so if users re-open the image within an app that edited it you can undo/alter old edits. 

If you just use the "video channel" to store other data then you should be able to do that in the other channels in the JPEG XL container as well.

You can also save multiple images with varying dimensions inside the same JPEG XL file if that's what you're after. You can read a bit more about it on page 6 in this whitepaper.

 

 

Apparently, I was wrong about the inter-frame compression. JPEG XL can do that by marking a frame as a "reference frame", and then use that when encoding subsequent frames. Although I doubt it's as good as the video formats.

 

 

Anyway, HEIC will have the edge when it comes to things involving video (since it is a video format after all), but for photos/pictures, JPEG XL is unmatched.

 

 

It's also worth noting that Google is currently heavily against JPEG XL, going as far as removing support for it in their software. Apple on the other hand is the company with the best support for JPEG XL, supporting it in pretty much all their software. I never though I'd experience the day where I was saying Apple were at the forefront of format support, and Google were the ones acting like a child who want everyone to like their toy and nothing else.

[Dracarris]

1 hour ago, dalekphalm said:

Feel free to provide counter evidence to how it works.

Evidence: A live photo is not 100MB+ as it would have to be if there'd be several other full-quality frames to chose from. A single still image is 3-5MB in HEIC.

[dalekphalm]

4 minutes ago, Dracarris said:

Evidence: A live photo is not 100MB+ as it would have to be if there'd be several other full-quality frames to chose from. A single still image is 3-5MB in HEIC.

To be fair, that's an explanation, not evidence, but @Obioban already provided a good explanation with photo evidence.

[hishnash]

2 hours ago, LAwLz said:

I never though I'd experience the day where I was saying Apple were at the forefront of format support, and Google were the ones acting like a child who want everyone to like their toy and nothing else.

I think this comes down to HW encoder/decoder support.  There appears to be rather good JPEG-XL support in modern apple chips and apple have been pushing hard for a HDR image standard (that JPEG-XL supports).   

 

2 hours ago, LAwLz said:

You can also save multiple images with varying dimensions inside the same JPEG XL

Im still looking for a IO library that abstracts this to a level that does not require me to implement my own.  

 

[Kisai]

4 hours ago, LAwLz said:

It's also worth noting that Google is currently heavily against JPEG XL, going as far as removing support for it in their software. Apple on the other hand is the company with the best support for JPEG XL, supporting it in pretty much all their software. I never though I'd experience the day where I was saying Apple were at the forefront of format support, and Google were the ones acting like a child who want everyone to like their toy and nothing else.

 

It's amazing how Google's solutions are always garbage and the only reason they can push them at all is by taking away the choice to not use them.

 

JPEG XL should be the de-facto replacement for JPEG. They only gave it 2 years. Did you know it took more than 5 years for PNG to replace GIF? And we still do not have a lossless animated alphachannel format?

 

Hell we don't even have a lossy alphachannel format. Not that there would ever be a desire for such as a still image. But there is for video, and the only format that supports Alpha channel is ProRes422 (ffmpeg prores_ks), which is like 100MB/sec. Not Megabits, MegaBytes. But another format that supports "alpha channel" is SpeedHQ SHQ7/SHQ9 (NDI4)

 

So what is one to do when they need to composite alpha-channels? Well, sucks to be you if you're an animator on Windows. 

 

It's kinda funny in the grand scheme of things, how all the photography and Film (live action) tools pay lip service to alpha channel needs. HEIC supports storing "metadata" for alpha, but that doesn't mean it will survive a pass through a tool that modifies the image.

 

H.265 "can" support alpha channel, in a hugely hacky way, which is why HEIC is still a hacky way to do it.

 

AVIF. Same problem as HEIC and WebP in this regard. These are not "image" formats, these a "Video formats" that require Video decoder paths, to render a still image. WebP, HEIC, AVIF should never have been "image formats", because what they really are, are containers for video data that is at the discretion of the runtime, in which everything is optional. So one browser might render it in software and be slow, and another might use a hardware decoder but doesn't support alpha.

 

So yes, JPEGXL makes sense as a still image format, because it looses the baggage of being a video format first. That makes it much more suitable for portable applications and still image software.  That means tools like Photoshop, stand-alone don't need to have video decoder licenses. Likewise all other photo and drawing tools.

 

But we mustn't forget the reason why PNG exists. There was political pushback against patented file formats. There is no reason to adopt a format that is encumbered by patents if the existing stuff we have works already. PNG replaced GIF as a still image, but we didn't get "MNG", the accompanying animated format in the browser, and it was subsequently removed by mozilla for "browser bloat" reasons, despite the fact that it was essentially just "animated PNG". So they developed APNG which is just PNG but animated like GIF (sequence of frames) instead of the more complex MNG which had some features that made it able to replace Flash.

https://bugzilla.mozilla.org/show_bug.cgi?id=18574#c72

 

Mozilla killed MNG over it being "300KB", meanwhile both Firefox and Chrome it ship with AVCODEC which supports JpegXL, a library that takes up several megabytes when compiled.

 

Killing MNG in Mozilla also killed support for JNG, which was a PNG-like container for JPEG data.

WebP, is terrible, and should never have come into existence. With the death of Flash, there is still a void left

 

Like just to point out the elephant in the room. Chrome's attempt to remove JPEGXL is like Mozilla's removal of MNG. The removal of the support, resulted in tools to make the files to cease development because nobody could then view the files. Chrome removing JPEGXL is extremely flimsy because it's part of AVCodec. They would have to make an actual choice to remove it from their fork of AVCodec.

https://chromium.googlesource.com/chromium/third_party/ffmpeg/+/refs/heads/master/configure

 

It's not like Chrome is independently linking all these libraries directly into Chrome anymore more than Firefox is.

 

But what drives support for an image support is people being able to share it, and you nerf that from the browser, and then people won't use it.

 

[LAwLz]

6 hours ago, hishnash said:

I think this comes down to HW encoder/decoder support.  There appears to be rather good JPEG-XL support in modern apple chips and apple have been pushing hard for a HDR image standard (that JPEG-XL supports).   

I don't really buy that argument for why Apple supports it and Google doesn't, because as far as I know there is no hardware acceleration going on with JPEG XL (at least not at this time), and in terms of encoding and decoding, JPEG XL is way faster than AVIF which is what Google is pushing.

 

If it was a performance thing that kept Google from implementing it, they would definitely side with JPEG XL over AVIF.

 

 

6 hours ago, hishnash said:

Im still looking for a IO library that abstracts this to a level that does not require me to implement my own.  

It's a very new format, and just like before we see that replacing an image format is a very difficult task.

If it takes off you will most likely see software implementations that do what you want them to do. But we will see if that happens.

[LAwLz]

4 hours ago, Kisai said:

JPEG XL should be the de-facto replacement for JPEG. They only gave it 2 years.

If you're talking about Google then it's even worse than that, because they didn't give it a fair chance during those two years.

They implemented it as a flag but kept it turned off by default. Then they neglected it (it took them about 2 years to fix a bug that caused a 3x performance drop because it treated static images as being animated) and then they discontinued it citing "lack of usage". No shit people won't use it if you don't even enable it by default, needing users to go into a hidden menu to enable it.

 

 

5 hours ago, Kisai said:

And we still do not have a lossless animated alphachannel format?

JPEG XL can do that, but if you want something today you can use APNG.

[hishnash]

3 hours ago, LAwLz said:

I don't really buy that argument for why Apple supports it and Google doesn't, because as far as I know there is no hardware acceleration going on with JPEG XL (at least not at this time), and in terms of encoding and decoding, JPEG XL is way faster than AVIF which is what Google is pushing.

 

I get the feeling there is something more than CPU only since if I try to encode a JPEG-XL in a widget extension (intent) I get an encoding error related to not having access to that HW units from the extension runtime, just the same as if I encode a HEIC or regular JEPG but I can encode PNGs.  

 

Typically intents cant access HW units when triggered while the app process is not running (you cant use the neural engine, gpu etc) so while their might not be any dedicated fix function for JPEG-XL apple might be using some of the existing HW units (other than pure cpu) to do some encoding otherwise I would not expect this exception to be raised. 

[LAwLz]

1 hour ago, hishnash said:

I get the feeling there is something more than CPU only since if I try to encode a JPEG-XL in a widget extension (intent) I get an encoding error related to not having access to that HW units from the extension runtime, just the same as if I encode a HEIC or regular JEPG but I can encode PNGs.  

 

Typically intents cant access HW units when triggered while the app process is not running (you cant use the neural engine, gpu etc) so while their might not be any dedicated fix function for JPEG-XL apple might be using some of the existing HW units (other than pure cpu) to do some encoding otherwise I would not expect this exception to be raised. 

It shouldn't be.

I don't know what Apple does but at least in Chrome there is no hardware acceleration for JPEG encoding or decoding. 

Chrome uses libjpeg-turbo which is purely a CPU-based encoder and decoder. It does however use SIMD instructions.

 

In any case, even if Apple does something in hardware for JPEG XL, the format is lightweight enough to do in CPU just like JPEG is. In fact, some tests shows JPEG XL being just as easy to decode as regular JPEG (when using the backward-compatible recompression option).

There is no hardware reason for why Google dropped support for it and Apple didn't. It's just that Google wants to push its own formats instead. Well, some at Google.

 

Their AVIF team released a comparison to justify their decision about a year ago and it was awful. They used severely outdated software for JPEG XL (an 18 month old version of Chrome, vs the newest for JPEG), they measured the quality using MS-SSIM which is really outdated way of measuring quality and they used a MS-SSIM tuned version of AVIF but not JPEG XL, and they even misunderstood their own charts because it shows JPEG XL being superior in most metrics, yet they concluded that it was worse.

That comparison was one of the reasons they used to remove JPEG XL support, along with stuff like "there is not enough interest from the entire ecosystem", which is funny because the only ones not interested in JPEG XL seem to be Google themselves. But things might be changing, because not too long ago someone from the Chromium team requested that they look into adding support again.