Sometimes I’ll sit down and draw strange doodles in the air that hang and hover and spin if I tap them. I sketch them in midair with a thick, black spatial stylus made by Logitech called the Muse, while wearing Apple’s Vision Pro headset.
The process feels like magic, and yet Apple often seems surprisingly indifferent to the opportunity. Pro-focused 3D creative tools that do 3D art like this don’t really live on Vision Pro. And Apple hasn’t bothered to make a spatial version of its own Pencil. This sense of half-finished frustration is exactly what makes the Vision Pro, now over two years old and in its second hardware iteration, sometimes feel dead.
But it’s not. In fact, Apple is planning more of them. Smaller, lighter, but not for at least a few years, according to Mark Gurman’s latest reports. Meanwhile, Apple’s AI-infused smart glasses may not be arriving until at least late next year… and later still for display-enabled ones that might get some Vision-like features.
But as we head toward Apple’s WWDC software conference next week, it’s the perfect time for Apple to finally unleash the Vision Pro and its many latent possibilities. Not just for that device itself, but for all the things that come after it. AI included.
Apple’s Personas use Gaussian splatting, a technology that’s still an untapped part of VisionOS’ possible future capabilities.
More power under the hood than is being tapped
I’m fascinated by the idea of spatial computing — specifically, floating screens and apps around me on demand. But what Apple has delivered so far is a limited subset of what could be. Apple’s increasingly real Persona avatars are just one facet of that latent potential.
I know the possibilities because I look at and test products like these all the time and chat with people exploring solutions that don’t exist yet. The Vision Pro has been perceived as Apple’s greatest product failure under Tim Cook — who do you know who owns one? — but it’s also widely acknowledged as the most advanced VR/AR device that exists. The M5 processor it uses, the eye tracking finesse it packs, the farther and nearer range motion sensor quality, the cameras that blend views of the world around you into pass-through video, they’re all the best.
What isn’t good is how the Vision Pro fails to implement all of this to deliver actually useful pro tools, and how it falls short in exploring the ideas Apple will have to solve in an expected range of AI-enabled wearables that don’t exist yet.
I expect Apple to make glasses, camera-enabled AirPods, and maybe even a world-aware sort of pendant or pin. But in the meantime, the Vision Pro is a very real product that already packs a lot of these possibilities, if Apple only unleashed it.
And no, endless supplies of immersive 3D video movies and sports events aren’t the answer for a $3,500 device. But what about extremely realistic 3D scans, which can be done via Gaussian splatting and displayed on Vision Pro? Or, AI that can recognize your head-worn worlds and even guide you in the middle of multiple projects?
Samsung’s Galaxy XR has been playing with Gemini Live ahead of smart glasses coming this fall. Apple can do the same.
Visual intelligence can and should happen on Vision Pro first
Camera-aware AI, or multimodal AI, is a growing space that most major AI platforms already dabble in. Meta already integrates it in smart glasses, and Google and Samsung are mixing it into mixed reality headsets that already exist, and into glasses coming by the end of this year.
Google and Samsung released the Vision Pro-like Galaxy XR headset last fall to explore ideas, like an always-on Gemini Live mode that can visually recognize the space you’re in and the apps you’re using. Now that Apple has a partnership with Google to infuse Siri with Gemini, the road seems open for similar explorations in Apple hardware, too.
I am curious and concerned about how AI intermediates with our senses in wearables, and how it can be an invasion of privacy for both the wearer and anyone in the wearer’s vicinity. But in the Vision Pro, Apple has a great way to test this, using all of the headset’s sensors and processors to explore these ideas before smart glasses, smart pins or smart AirPods get released.
There’s another small facet to AI that isn’t about agents, known as Gaussian splatting, that Vision Pro’s OS should explore far more. Unlike Apple’s stereoscopic, immersive videos shot with multiple camera lenses, Gaussian splats can create full holographic-style images and videos using AI to stitch them together. Apple’s been introducing more spatial 3D layers into Vision, Personas being the biggest part. But Vision Pro should be part of a whole 3D scanning studio system with apps Apple creates on iPhones and in a headset, as extensions of the Camera app on phones that’s already there.
Apple Watches and iPhones need to be seamless pieces to Vision Pro’s connection matrix and iPads too.
Connect all Apple’s things to Vision Pro
The Logitech Muse is like an Apple Pencil for Vision Pro that Apple never bothered to make. Similarly, Sony’s PlayStation VR 2 controllers can connect with the Vision Pro to play games in ways that aren’t otherwise possible, since Apple doesn’t make its own spatial controllers either.
Some of this is understandable, since the Vision Pro is an experimental product, and Apple often leans on companies like Logitech to explore peripheral ideas it hasn’t made yet (this happens with iPad keyboard cases, too).
And yet the Vision Pro still doesn’t connect seamlessly with other Apple products that have existed for years. While AirPods connect with it, and Macs can extend monitors or even stream Mac apps to the headset, iPhones, iPads and Watches are strangely left out, except for casting iPhone/iPad screens via AirPlay.
I want to share apps and extend monitors from iPhones and iPads near me, letting the Vision Pro work as a shared sort of computer, as it does with the Mac. There’s no reason why they can’t. iPhone-level chips can run MacOS now, as the MacBook Neo proved. Certainly, they can share screens and extend apps too, or let me juggle all these devices in magical ways with one headset that recognizes them all.
The Apple Watch is maybe the most egregious piece left out: It’s literally a wristworn control panel with motion tracking, and it could be a brilliant interface with the Vision Pro if Apple let it.
Apple’s own creative apps aren’t all in Vision native forms yet. They should be.
Open the road to pro apps, one way or the other
The Vision Pro’s biggest fault, to me, isn’t its price or size. It’s how the headset still falls behind Macs and even iPads at being a true pro computer to fit the “pro” name, even with an M5 processor under its belt now.
What do I mean by pro? I mean video editing suites, music creation tools, 3D graphics programs — anything that could and should be on VisionOS to allow for the sort of creative work that Apple prides itself on.
It’s weird to me because Vision Pros are apparently being used by filmmakers, often as on-set monitors for seeing footage or 3D models. They should be entire doorways to doing everything. A massive virtual monitor and 3D interface should enable simulation and creation that I’ve seen even in headsets like Meta’s far less powerful Quest 3.
If getting developers to make powerful pro apps is too hard a lift and Apple doesn’t want to make them (which seems to be the case after the new Creator Studio suite of apps didn’t include any Vision Pro optimizations), extending and streaming from Macs and iPads could help. VisionOS can render 3D content from Macs and even optimize it using foveated streaming, a trick that delivers high resolution only where your eyes are looking directly.
Opening a path for developers to extend Mac apps that are VisionOS-ready, or doing this for iPads and iPhones, is a necessary step. Otherwise, when it comes to getting serious work done, I’m probably taking the Vision Pro off.
