Kuo on Apple’s VR+AR HMD: late 2022 release, as powerful as an M1 Mac, untethered from iPhone/Mac

[saltycaramel]

Summary

Ming-Chi Kuo’s latest report predicts the release of Apple’s HMD in late 2022. The Quest-like visor will sport both an M1-Mac-like SoC for the OS/UI/3D/apps/games and a lower-end SoC for the visual processing (6-8 cameras will enable the real-time see-thru functionality and mixed reality). Crucially, it will work independently from the iPhone and the Mac, so you won’t need to be an existing Apple user to use it. 

 

Quotes

Quote

We predict that Apple's AR headset to be launched in 4Q22 will be equipped with two processors. The higher-end processor will have similar computing power as the M1 for Mac, whereas the lower-end processor will be in charge of sensor-related computing. 

The power management unit (PMU) design of the high-end processor is similar to that of M1 because it has the same level of computing power as M1.

….

Apple’s AR headset requires a separate processor as the computing power of the sensor is significantly higher than that of the iPhone. For example, the AR headset requires at least 6-8 optical modules to simultaneously provide continuous video see-through AR services to users. In comparison, an iPhone requires up to 3 optical modules running simultaneously and does not require continuous computing.

My thoughts

Apple’s most recent “next big things” are the Watch (2015) and the Airpods (2016). While the Airpods work ok outside of Apple’s ecosystem (while missing a couple of Apple-specific tricks), the Watch literally forces you to switch to iPhone if you wanna use it (and you should if you’re interested in smart watches at all, since it’s years beyond the wrist-wearable competition). If it’s true that the “iVisor” will be an out-of-the-box experience that doesn’t require you to be an existing iPhone or Mac user, this will be a major shift compared to previous Apple wearables introductions. 

 

As for the hardware itself, the power of an M1 Mac inside an untethered HMD? 4K micro-OLED display for each eye? 6-8 cameras + LIDARs to constantly scan the room and your hands? All packaged inside Apple’s industrial design? You have my attention.

 

I wonder if, like they iPad, this will be offered both in a WiFi-only and a WiFi+5G version. Could this be the first product to sport Apple’s rumored internally developed 5G modem? They wouldn’t risk putting it inside iPhones without using a lower volume product as a test bed before. 

 

Sources

https://www.macrumors.com/2021/11/25/kuo-apple-ar-headset-mac-level-computing/

[JayD]

It would be nice if you could optionally connect it to a Mac or Windows PC in case you wanted to run games with more horsepower. Regardless, the good thing about Apple entering this space is that you can guarantee that on day 1 there will be lots of AAA developers releasing lots of high quality games for it to get in that early adaptor $$$.

[sap]

If Apple's history of being hostile towards standard APIs is anything to go by (*cough* Metal), they're probably going to completely ignore the fact that OpenXR is now becoming the standard API for VR/AR applications, and burden developers by forcing them to use their special proprietary API to develop apps for their headset, even though OpenXR would allow them to have identical/superior performance (OpenXR is built with extensibility in mind after all). Their lack of contributions to OpenXR isn't a great sign either. This is super annoying especially for indie developers and everyday I'm getting more and more convinced that Apple has a secret deal with aliens to pointlessly waste the time of developers in order to reduce overall human productivity/morale and allow the future alien invasion to succeed.

[saltycaramel]

What they’re gonna use development-wise has been hidden in plain sight for the good part of a decade now, WWDC after WWDC

- SceneKit (2012)

- Metal (2014)

- ARkit (2017)

- RealityKit (2019)

- Object Capture (2021)

- SwiftUI (for floating widget-like 2D windows)

 

This gave Apple-focused devs years to get accustomed with what the VR/AR goggles will use. And gives Apple complete control over the vertical integration of hw+sw+battery_life. 

[RejZoR]

Many complain how Apple watch forces you to have an iPhone, but at the same time Apple Watch offers experience unlike any other watch that's not specifically connected to a phone. The way how everything works seamlessly on the watch is when you understand why they do it. I don't like it either because if I'll go with Android again, my Apple Watch 5 won't be any good, but I understand it.

[HRD]

Your title is misleading. Kuo predicted that the AR headset not VR to be released at Q42022.

[saltycaramel]

13 minutes ago, HRD said:

Your title is misleading. Kuo predicted that the AR headset not VR to be released at Q42022.

 

Fixed the title to reflect that this is about VR+AR.

 

But there’s no mistake he’s talking about Apple’s Oculus Quest style head mounted device, with fully closed design to not let any external light in, and AR see-thru performed via cameras (it will be interesting to see how they will minimize the lag). Pretty sure that can be considered more VR than AR, on a purely hardware basis. It’s not like Apple will later make a product that’s “more VR” than this. 

 

The AR-only product are the Apple Glasses later this decade and for sure they can’t be ready in 12 months with that kind of horsepower. 

 

From the article:

 

Quote

In addition to AR, Kuo says the headset will also be able to support virtual reality (VR) experiences thanks to a pair of 4K Micro OLED displays from Sony, which require the computing horsepower of an M1-like chip.

 

[Doobeedoo]

But it's Apple and it's ecosystem.

Really no excitement for VR until it advanced a lot more in terms of hardware and software, until it actually becomes ubiquitous as todays devices.

Until then, don't care... I mean there's VRChat though...

[saltycaramel]

10 minutes ago, Doobeedoo said:

But it's Apple and it's ecosystem.

Really no excitement for VR until it advanced a lot more in terms of hardware and software, until it actually becomes ubiquitous as todays devices.

Until then, don't care... I mean there's VRChat though...

 

We need a 2007_iPhone-like shock in the hw+OS+interaction department. Something that’s so beyond what we have today to mark a before and an after. Something that makes you go “wow” like the first time you used the OG iPhone multitouch capacitive touchscreen and scrolled its buttery smooth rubber band scrolling.

 

Then, both what’s outside and inside Apple’s ecosystem will thrive. That’s what happened with smartphones. 

[Spindel]

1 hour ago, saltycaramel said:

 

We need a 2007_iPhone-like shock in the hw+OS+interaction department... 

Fart apps, now in 3D

[Kisai]

9 hours ago, saltycaramel said:

 

 

As for the hardware itself, the power of an M1 Mac inside an untethered HMD? 4K micro-OLED display for each eye? 6-8 cameras + LIDARs to constantly scan the room and your hands? All packaged inside Apple’s industrial design? You have my attention.

 

I wonder if this will allow vtubers to stream with only the HMD.

 

One of the problems with AR stuff at the moment is that the iPhone is the ONLY usable face tracking solution out there that everyone can access. Vive? trash, webcam? trash, realsense? abandoned, kinect? useless. Proprietary solutions ? $20,000 and a monthly use license.  A $500 iPhone is a steal, and people complain about Android not having an equivalent, and webcam's being a poor experience.

 

 

[Commodus]

1 hour ago, Doobeedoo said:

But it's Apple and it's ecosystem.

Really no excitement for VR until it advanced a lot more in terms of hardware and software, until it actually becomes ubiquitous as todays devices.

Until then, don't care... I mean there's VRChat though...

Well, consider this: the Oculus Meta Quest 2 is built on a variant of the Snapdragon 865 from 2020. Even the A14 from that year is faster than the 865... the M1 is in another league, and the description sounds like the Apple headset's chips will represent a significant step forward from that.

 

In essence: imagine a headset that offers the power of computer-based VR in a Quest 2-like standalone design. That'd be a pretty compelling argument if Apple can rally developers, and it probably can given both the distaste for Meta and HTC's struggles.

[StDragon]

 "...with an eye toward replacing the ‌iPhone‌ within ten years."

 

I want what they're smoking. No way will society tolerate a bunch of mindless cyclops drones walking around bumping into things. It's bad enough people jaywalk with their heads lowered and fixated on screens as it is.

 

There's a saying of "looking but not seeing" when driving. Sure, you can see that red light, but if you're not concentrating on driving, then a red light doesn't mean anything until it's too late (like T-boning into another car).

[saltycaramel]

43 minutes ago, StDragon said:

 "...with an eye toward replacing the ‌iPhone‌ within ten years."

 

I want what they're smoking. No way will society tolerate a bunch of mindless cyclops drones walking around bumping into things. It's bad enough people jaywalk with their heads lowered and fixated on screens as it is.

 

There's a saying of "looking but not seeing" when driving. Sure, you can see that red light, but if you're not concentrating on driving, then a red light doesn't mean anything until it's too late (like T-boning into another car).

 

There’s a classic 2010 Futurama episode about an “eyePhone” replacing the iPhone. 

 

I think the 2022 VR+AR HMD is meant to be more of a sofa/armchair/bed/living_room/desk device. More Mac/TV/iPad than iPhone.

 

Then, a couple of years down the road, Apple is supposed to release thin-frame AR-only glasses. Those may replace the iPhone for going about places in the outside world. Or enhance the iPhone/Watch experience by superimposing a large AR virtual display over them. 

 

But the VR+AR HMD is a necessary first step towards the AR-only glasses. And it’s a thing of itself in terms of being a VR device. It’s both. It’s also a dev-kit for developers. So it’s 3 things actually. VR HMD, dev-kit for VR/AR, grandfather of the future AR-only glasses.

 

Then in the 2050-2060s we get AR smart contact lenses powered by body heat and solar energy. I wouldn’t put anything past what a society made of people born today will tolerate. 

[hishnash]

10 hours ago, sap said:

OpenXR would allow them to have identical/superior performance (OpenXR is built with extensibility in mind after all).

I would say unlikely for OpenXR to provide better performance as it is a CPU side only api and apples Metal API would let them expose this data directly to the GPU (since GPU compute shaders can schedule draw calls on thier GPUs so if your considered about input latency... and for VR/AR you are this is a big factor). 

 

[Kisai]

19 minutes ago, saltycaramel said:

 

There’s a classic 2010 Futurama episode about an “eyePhone” replacing the iPhone. 

 

I think the 2022 VR+AR HMD is meant to be more of a sofa/armchair/bed/living_room/desk device. More Mac/TV/iPad than iPhone.

 

Then, a couple of years down the road, Apple is supposed to release thin-frame AR-only glasses. Those may replace the iPhone for going about places in the outside world. Or enhance the iPhone/Watch experience by superimposing a large AR virtual display over them. 

 

But the VR+AR HMD is a necessary first step towards the AR-only glasses. And it’s a thing of itself in terms of being a VR device. It’s both. It’s also a dev-kit for developers. So it’s 3 things actually. VR HMD, dev-kit for VR/AR, grandfather of the future AR-only glasses.

 

Then in the 2050-2060s we get AR smart contact lenses powered by body heat and solar energy. I wouldn’t put anything past what a society made of people born today will tolerate. 

There is unlikely to be much, if any, more miniaturization of semiconductors. The cost for 2nm is going to be extremely expensive, and you wouldn't want to put something that generates heat against your skin, especially your eyes.

 

Some kind of new material with no thermal energy loss would need to be invented before we get to AR glasses and AR contact lenses. What we're going to end up with before that are just refinements on the existing bulky HMD.

 

Here's why VR hasn't taken off, but AR stuff largely has:

- VR HMD obstructs your vision, AR tech does not

- VR requires expensive trackers and controllers

- VR requires a large amount of space to use. 

- VR makes you sick.

 

Basically the reason you don't see very much VR stuff, is because vendors selling the HMD's are too cheap to build them in a way that people want to use them. You won't sell HMD's to people who will get sick, find the screen door effect distracting, or have no space for a VR setup.

 

If Apple can solve all of that, we're golden. But I think we're a long ways from that.

[saltycaramel]

16 minutes ago, Kisai said:

There is unlikely to be much, if any, more miniaturization of semiconductors. The cost for 2nm is going to be extremely expensive, and you wouldn't want to put something that generates heat against your skin, especially your eyes.

 

Some kind of new material with no thermal energy loss would need to be invented before we get to AR glasses and AR contact lenses. What we're going to end up with before that are just refinements on the existing bulky HMD.

 

About the lenses, it was a far fetched, half-joking comment. (interestingly enough, there are already some Apple patents mentioning smart contact lenses, but as we know patents are written to encompass as many possible use cases as possible)

 

But I wouldn’t put AR glasses in the same “thermodynamically unfeasible” far-fetched basket as AR lenses. If Facebook+Ray-Ban can build the Stories glasses today, I can’t believe Apple won’t be able to build AR glasses (with power-sipping cool-to-the-touch Apple_Watch-TDP-class silicon) 5-10 years from now. 

[Kisai]

55 minutes ago, saltycaramel said:

 

About the lenses, it was a far fetched, half-joking comment. (interestingly enough, there are already some Apple patents mentioning smart contact lenses, but as we know patents are written to encompass as many possible use cases as possible)

 

But I wouldn’t put AR glasses in the same “thermodynamically unfeasible” far-fetched basket as AR lenses. If Facebook+Ray-Ban can build the Stories glasses today, I can’t believe Apple won’t be able to build AR glasses (with power-sipping cool-to-the-touch Apple_Watch-TDP-class silicon) 5-10 years from now. 

If you watched the video on that, you'd realize that those facebook glasses are effectively unusable. At least the Apple watch has a purpose that allows it to be used without the iPhone. The FB glasses are basically just a cheap pair of cameras stuffed into glasses frames, that would normally cost that much anyway.

 

[StDragon]

29 minutes ago, Kisai said:

If you watched the video on that, you'd realize that those facebook glasses are effectively unusable. At least the Apple watch has a purpose that allows it to be used without the iPhone. The FB glasses are basically just a cheap pair of cameras stuffed into glasses frames, that would normally cost that much anyway.

 

I don't think the world will be welcoming of "glassholes".

For me, AR is a fantastic hands-free tool, more for the industrial and medical professional side of things. I could see them used by technicians in a chemical plant, or working on machinery with the tech embedded in safety glasses. Schematics and diagnostic telemetry all floating in view. I'm sure companies like Fluke will be all over AR integration with their equipment too.

But AR for the common person 24/7? No way! Too much information overload. It's bad enough with cell phones already. 

[hishnash]

24 minutes ago, StDragon said:

Too much information overload. It's bad enough with cell phones already. 

depends on the implementation, AR by FB were it is funded by ads sure but AR by apple were they see it more like the Apple Watch than the iPhone then it could be extremely minimal were they is no UI unless you make a gesture to show UI, like how the first few generations of watch requires you to raise your write for it to light up.  

 

[saltycaramel]

38 minutes ago, StDragon said:

I don't think the world will be welcoming of "glassholes".

For me, AR is a fantastic hands-free tool, more for the industrial and medical professional side of things. I could see them used by technicians in a chemical plant, or working on machinery with the tech embedded in safety glasses. Schematics and diagnostic telemetry all floating in view. I'm sure companies like Fluke will be all over AR integration with their equipment too.

But AR for the common person 24/7? No way! Too much information overload. It's bad enough with cell phones already. 

These companies are fighting to monetize every minute of our waking hours. (until they find a way to monetize dreams as well)

 

If they create a compelling experience, we will comply and rent them our eyeballs. Societal acceptance will gradually come, we are the society after all. 

 

The same things were said about smartphones and smartphone etiquette years ago. 

 

As for the pro vs consumer usage, airline pilots wouldn’t have iPads if we consumers hadn’t  subsidized their development by playing Fruit Ninja on them. Same could happen for AR headsets. Consumer tech has the advantage of having a huge scale and supply chain, compared to low volume specialized tech.

[Brooksie359]

12 hours ago, RejZoR said:

Many complain how Apple watch forces you to have an iPhone, but at the same time Apple Watch offers experience unlike any other watch that's not specifically connected to a phone. The way how everything works seamlessly on the watch is when you understand why they do it. I don't like it either because if I'll go with Android again, my Apple Watch 5 won't be any good, but I understand it.

Yeah I will have to give it to Apple that although they are super restrictive in how things work they really do provide a great experience when using multiple devices from their ecosystem. It's great marketing as it sorta forces you to use more of their products if you want that seamless experience but at the same time they do deliver on that experience so it's hard to get too mad about it.

[sap]

7 hours ago, hishnash said:

I would say unlikely for OpenXR to provide better performance as it is a CPU side only api and apples Metal API would let them expose this data directly to the GPU (since GPU compute shaders can schedule draw calls on thier GPUs so if your considered about input latency... and for VR/AR you are this is a big factor).

And that's where extensions come in. If there's a missing feature in OpenXR or Vulkan preventing a hardware vendor from making full use of their hardware, they make an extension. This is what NVIDIA, AMD, Qualcomm, Intel, ARM, Google, Huawei, Broadcom, VeriSilicon, Samsung, Facebook/Oculus, Microsoft, Varjo, Valve, Dell, Sony, HP, HTC and others are currently doing, and I don't see the need for Apple to suddenly do their own thing.

 

Vulkan and OpenXR aren't just unilateral APIs, they're shaped by many key players who are all contributing their knowledge and use cases. You can view the full list of contributors in the spec:

https://www.khronos.org/registry/vulkan/specs/1.2-extensions/html/vkspec.html#credits

https://www.khronos.org/registry/OpenXR/specs/1.0/html/xrspec.html#credits

[saltycaramel]

1 hour ago, sap said:

, and I don't see the need for Apple to suddenly do their own thing.

 

Suddenly? As I wrote in a previous post, Apple

has been paving the API-way for this since before OpenXR was even a thing. 

 

And yeah there are plenty of reasons for Apple to pursue their own path and not wait for some consortium to implement stuff. 

 

We’re talking like VR is a “solved problem” and companies just need to come together around a standard. It’s far from a solved problem, it’s a frontier and Apple is heading West.

[sap]

32 minutes ago, saltycaramel said:

Apple has been paving the API-way for this since before OpenXR was even a thing.

That's not a reason for them to break standardization that we now have. For example Oculus has also been paving their own way, but they realized it'd be best for developers if there was a standard VR/AR API rather than fragmenting things across Oculus with their own API, Apple with their own ARKit, Valve with their own SteamVR, Microsoft with their own WMR platform, etc. They've since deprecated their proprietary API and they now recommend using OpenXR for creating Oculus apps.

 

Oculus, Valve, Microsoft and others now all fully support OpenXR and you can run OpenXR applications on their headsets. There's no excuse for Apple to be special and break compatibility today, especially considering they haven't even released any headsets yet and they can still make big changes.

  

32 minutes ago, saltycaramel said:

plenty of reasons for Apple to pursue their own path and not wait for some consortium to implement stuff

This is not the way it works.

 

This page outlines the way OpenXR extensions are born: https://www.khronos.org/registry/OpenXR/specs/1.0/extprocess.html

 

In short, vendors create and maintain their own OpenXR/Vulkan runtime/driver. They maintain full rights and control over their own implementation. At any point in time, they can suddenly implement or remove an extension from their runtime without the need to wait or consult with anyone, and they can distribute the update the way they'd normally distribute driver updates.

 

Creating a new extension from scratch (like one that passes XR input directly to GPU) involves a process and is dependent on what type of extension is being made. The type of extension most applicable here is a vendor-specific extension, where only one vendor implements it. This type of extension can be developed fully in-house by the vendor. It gets registered and eventually once finished, submitted to OpenXR in a merge request.

 

Once submitted, it gets reviewed by the appropriate members/companies. This is again dependent on the type of extension. For vendor-specific extensions the bar is set really low, here's the exact wording:

• CI does not fail for any reason due to the changes within the merge request. Translation: it doesn't completely break everything

• There are no pending conflicts. Translation: it doesn't completely break everything

• There are no additions to core or KHR bitmasks or other limited-space entities. Translation: it doesn't try to change core functionality of the API

• The reviewer may also perform a check of the correctness and style of the specification and registry (xr.xml) changes.

 

There is NO waiting around for a consortium to implement anything. The closest thing is waiting for the merge request to get approved, which the bar is really low for and it can be done in probably less than 5 minutes if someone's around. Even if there's not one reviewer who can approve this basic review for weeks, there's nothing stopping Apple from just implementing the extension anyway and publishing their own docs on how to use it.