That’s fortunate, because he has now taken on the very different challenge of leading the user interface team, and that is perhaps the single greatest challenge for AR. Not to diminish the other challenges — getting the right photons into your eyes in AR is insanely hard, and making computer vision work within the power and weight budget of glasses is right up there as well — but at least they’re well-understood problems.
Whatever the standard AR user interface ultimately ends up being — and it will be years before that’s determined — it’s going to be something completely new, as clean a break from anything that’s come before as the mouse/GUI-based interface was from punch cards, printouts, and teletype machines. You’re going to have to be able to interact with your AR glasses in all the contexts you encounter in your day, so the interface will have to be multimodal. Hand gestures are good, but you’re unlikely to want to use them while you’re face to face with someone, and you won’t be able to use them at all if you’re carrying something. Voice is another good option, but it’s not a great choice in a meeting or in a noisy room. A handheld controller can be effective, but only when you have it with you (and haven’t misplaced or lost it), and only if your hands are free and it’s socially acceptable to be using it. Each mode of interaction has its strengths, but no one mode can meet all the needs, and the challenge is to design an interface that can switch seamlessly between them and decide which to use at any given moment.
Because AR glasses are enhancements for your perception, memory, and cognition, they’ll also need to anticipate what you want—and just as important, what you don’t want. As I noted earlier, if you can’t remember someone’s name, it would be great to have the glasses remind you. At the same time, if you walk into work and the glasses insist on telling you the name of every single person you see, you’ll never wear them again. AR will ultimately need to be a cloud of inference that surrounds you all day, helping you so intuitively that when you take the glasses off, it will feel like part of your brain has gone to sleep.
You might reasonably wonder why Sean is leading the user interface team, rather than a well-known UI researcher. In my experience, the key to true generational leaps is to have great problem solvers working on them, regardless of previous experience. As Thomas Kuhn observed in The Structure of Scientific Revolutions, it’s fresh faces, unattached to existing approaches, who end up trying the new, risky approaches that lead to paradigm shifts. And the truth is, VR and AR are developing so rapidly that there are no experts right now, only smart people who want to apply their skills and creativity to solving one of the hardest and most interesting multi-disciplinary problems around.
Taking new, risky approaches requires rock-solid organizational support, and for Oculus Research, the commitment comes straight from the top. Mark Zuckerberg frequently describes VR and AR together as the next computing platform, and as keys to Facebook’s 10-year strategy—check out any of his last few F8 keynotes (F8 2017, F8 2016) or many of the quarterly earnings calls. And, in fact, our AR program is the direct result of Mark’s vision — it started because Mark felt it was a long-term investment we needed to make.
Mark’s vision makes perfect sense, because both AR and VR fit seamlessly into Facebook’s mission to bring the world closer together. Even at this very early stage, social VR like Facebook Spaces shows the potential power of virtual communities. And AR will enable people to be connected more closely no matter where they are or what they’re doing.
I’ll be honest—when Mark first raised the topic of AR, I literally said that I wasn’t sure what it was useful for. That earned me a look of disbelief that was useful incentive to think a lot harder about AR’s potential. Three years later, I’m fully convinced that we’ll all be wearing AR glasses one of these years—myself included—but it was Mark’s vision that first got me thinking in that direction, and that convinced me to sign up to make AR happen.
While AR glasses have the potential to be one of the most important technologies of the twenty-first century, that won’t happen unless some very challenging practical constraints are overcome. They must be light and comfortable enough to wear all day, run off a wearable battery for many hours per charge without getting uncomfortably hot, work in full sunlight and in darkness, and have excellent visual and audio quality, both virtual and real. They must be completely socially acceptable — in fact, they need to be stylish. They need an entirely new user interface. Finally, all the rendering, display, audio, computer vision, communication, and interaction functionality needed to support virtual objects, telepresence, and perceptual/mental superpowers must come together in a system that operates within the above constraints. (See this for a broader overview of what it will take to make AR work.)
There is no combination of existing technologies that meet all those requirements today. The honest truth is that the laws of physics may make it impossible to ever build true all-day AR glasses; there’s no Moore’s Law for optics, batteries, weight, or thermal dissipation. My guess is that it is in fact possible (obviously, or I wouldn’t be trying to make it happen), and if it is possible, I think it’s highly likely that all-day AR glasses will happen within the next ten years, but it is an astonishingly difficult technical challenge on half a dozen axes, and a host of breakthroughs are going to be needed.
AR is also a largely unexplored space, so there’s no way to know in advance what the experiences are that will make AR glasses worth wearing all day. All of which means that our AR glasses effort, which spans all of the above, is an ongoing joint evolution of research, engineering, and experience prototyping; so, despite the name Oculus Research, the AR effort is in fact a mix of research, incubation, and product development.
Tackling such a huge, ambitious, multifaceted project requires close teamwork and constant communication across a large, diverse set of specialists and generalists, spanning user experience, hardware, software, optics, displays, sensing, silicon, perceptual science, computer vision, audio, user interface, operating systems, system architecture, program management, and more. It also requires fostering creativity and the ability to innovate among the various specialists and sub-projects, while still maintaining the discipline needed to get to the overall goal. That delicate balancing act perfectly suits Laura Fryer, the general manager of AR Incubation.