Hyper Vision

It is not too far away.  It is also a natural step beyond the Holodec and avoids the real space that envisages.   ..

What it does mean is that you can enter an experience and correctly realize that experience.  

I often wondered what it would feel like to ride a dolphin.  This will make just that possible without even annoying a dolphin.  Better mind to mind communication can be integrated to make even all that work seamlessly.  With this Heaven on Earth can be experienced.  In the process it can also be realized..

HYPER VISION

The world’s hottest startup isn’t located in Silicon Valley—it’s in suburban Florida. KEVIN KELLY explores what Magic Leap’s mind-bending technology tells us about the future of virtual reality.

Don’t call it a lens: Magic Leap founder Rony Abovitz displaying his company’s mysterious photonic lightfield chip.http://www.wired.com/2016/04/magic-leap-vr/ PETER YANG FOR WIRED

THERE IS SOMETHING special happening in a generic office park in an uninspiring suburb near Fort Lauderdale, Florida. Inside, amid the low gray cubicles, clustered desks, and empty swivel chairs, an impossible 8-inch robot drone from an alien planet hovers chest-high in front of a row of potted plants. It is steampunk-cute, minutely detailed. I can walk around it and examine it from any angle. I can squat to look at its ornate underside. Bending closer, I bring my face to within inches of it to inspect its tiny pipes and protruding armatures. I can see polishing swirls where the metallic surface was “milled.” When I raise a hand, it approaches and extends a glowing appendage to touch my fingertip. I reach out and move it around. I step back across the room to view it from afar. All the while it hums and slowly rotates above a desk. It looks as real as the lamps and computer monitors around it. It’s not. I’m seeing all this through a synthetic-reality headset. Intellectually, I know this drone is an elaborate simulation, but as far as my eyes are concerned it’s really there, in that ordinary office. It is a virtual object, but there is no evidence of pixels or digital artifacts in its three-dimensional fullness. If I reposition my head just so, I can get the virtual drone to line up in front of a bright office lamp and perceive that it is faintly transparent, but that hint does not impede the strong sense of it being present. This, of course, is one of the great promises of artificial reality—either you get teleported to magical places or magical things get teleported to you. And in this prototype headset, created by the much speculated about, ultrasecretive company called Magic Leap, this alien drone certainly does seem to be transported to this office in Florida—and its reality is stronger than I thought possible.

I saw other things with these magical goggles. I saw human-sized robots walk through the actual walls of the room. I could shoot them with power blasts from a prop gun I really held in my hands. I watched miniature humans wrestle each other on a real tabletop, almost like a Star Wars holographic chess game. These tiny people were obviously not real, despite their photographic realism, but they were really present—in a way that didn’t seem to reside in my eyes alone; I almost felt their presen

Virtual reality overlaid on the real world in this manner is called mixed reality, or MR. (The goggles are semitransparent, allowing you to see your actual surroundings.) It is more difficult to achieve than the classic fully immersive virtual reality, or VR, where all you see are synthetic images, and in many ways MR is the more powerful of the two technologies.

Magic Leap is not the only company creating mixed-reality technology, but right now the quality of its virtual visions exceeds all others. Because of this lead, money is pouring into this Florida office park. Google was one of the first to invest. Andreessen Horowitz, Kleiner Perkins, and others followed. In the past year, executives from most major media and tech companies have made the pilgrimage to Magic Leap’s office park to experience for themselves its futuristic synthetic reality. At the beginning of this year, the company completed what may be the largest C-round of financing in history: $793.5 million. To date, investors have funneled $1.4 billion into it.

That astounding sum is especially noteworthy because Magic Leap has not released a beta version of its product, not even to developers. Aside from potential investors and advisers, few people have been allowed to see the gear in action, and the combination of funding and mystery has fueled rampant curiosity. But to really understand what’s happening at Magic Leap, you need to also understand the tidal wave surging through the entire tech industry. All the major players—Facebook, Google, Apple, Amazon, Microsoft, Sony, Samsung—have whole groups dedicated to artificial reality, and they’re hiring more engineers daily. Facebook alone has over 400 people working on VR. Then there are some 230 other companies, such as Meta, the Void, Atheer, Lytro, and 8i, working furiously on hardware and content for this new platform. To fully appreciate Magic Leap’s gravitational pull, you really must see this emerging industry—every virtual-reality and mixed-reality headset, every VR camera technique, all the novel VR applications, beta-version VR games, every prototype VR social world.

Like I did—over the past five months.

Then you will understand just how fundamental virtual reality technology will be, and why businesses like Magic Leap have an opportunity to become some of the largest companies ever created.

Even if you’ve never tried virtual reality, you probably possess a vivid expectation of what it will be like. It’s the Matrix, a reality of such convincing verisimilitude that you can’t tell if it’s fake. It will be the Metaverse in Neal Stephenson’s rollicking 1992 novel, Snow Crash, an urban reality so enticing that some people never leave it. It will be the Oasis in the 2011 best-selling story Ready Player One, a vast planet-scale virtual reality that is the center of school and work. VR has been so fully imagined for so long, in fact, that it seems overdue.

I first put my head into virtual reality in 1989. Before even the web existed, I visited an office in Northern California whose walls were covered with neoprene surfing suits embroidered with wires, large gloves festooned with electronic components, and rows of modified swimming goggles. My host, Jaron Lanier, sporting shoulder-length blond dreadlocks, handed me a black glove and placed a set of homemade goggles secured by a web of straps onto my head. The next moment I was in an entirely different place. It was an airy, cartoony block world, not unlike the Minecraft universe. There was another avatar sharing this small world (the size of a large room) with me—Lanier.

We explored this magical artificial landscape together, which Lanier had created just hours before. Our gloved hands could pick up and move virtual objects. It was Lanier who named this new experience “virtual reality.” It felt unbelievably real. In that short visit I knew I had seen the future. The following year I organized the first public hands-on exhibit (called Cyberthon), which premiered two dozen experimental VR systems from the US military, universities, and Silicon Valley. For 24 hours in 1990, anyone who bought a ticket could try virtual reality. The quality of the VR experience at that time was primitive but still pretty good. All the key elements were there: head-mounted display, glove tracking, multiperson social immersion.

But the arrival of mass-market VR wasn’t imminent. The gear cost many scores of thousands of dollars. Over the following decades, inventors were able to improve the quality, but they were unable to lower the cost.

Twenty-five years later a most unlikely savior emerged—the smartphone! Its runaway global success drove the quality of tiny hi-res screens way up and their cost way down. Gyroscopes and motion sensors embedded in phones could be borrowed by VR displays to track head, hand, and body positions for pennies. And the processing power of a modern phone’s chip was equal to an old supercomputer, streaming movies on the tiny screen with ease. The cheap ubiquity of screens and chips allowed a teenage Palmer Luckey to gaffer-tape together his first VR headset prototypes, launching a Kickstarter campaign for the Oculus Rift in 2012. And the Rift was the starting signal that many entrepreneurs were waiting for. (Facebook bought the company for $2 billion in 2014.)

All of today’s head-mounted VR displays are built out of this cheap phone technology. Put on almost any synthetic-reality display and you enter a world born of billions of phones. Lanier, who has contributed toMicrosoft’s HoloLens MR system, estimates it would have cost more than $1 million in 1990 to achieve the results that even simple phone-inserted headsets like the Samsung Gear or Google Cardboard do today.

ONE OF THE first things I learned from my recent tour of the synthetic-reality waterfront is that virtual reality is creating the next evolution of the Internet. Today the Internet is a network of information. It contains 60 trillion web pages, remembers 4 zettabytes of data, transmits millions of emails per second, all interconnected by sextillions of transistors. Our lives and work run on this internet of information. But what we are building with artificial reality is an internet of experiences. What you share in VR or MR gear is an experience. What you encounter when you open a magic window in your living room is an experience. What you join in a mixed-reality teleconference is an experience. To a remarkable degree, all these technologically enabled experiences will rapidly intersect and inform one another.

The recurring discovery I made in each virtual world I entered was that although every one of these environments was fake, the experiences I had in them were genuine. VR does two important things: One, it generates an intense and convincing sense of what is generally called presence. Virtual landscapes, virtual objects, and virtual characters seem to be there—a perception that is not so much a visual illusion as a gut feeling. That’s magical. But the second thing it does is more important. The technology forces you to be present—in a way flatscreens do not—so that you gain authentic experiences, as authentic as in real life. People remember VR experiences not as a memory of something they saw but as something that happened to them.

Experience is the new currency in VR and MR. Technologies like Magic Leap’s will enable us to generate, transmit, quantify, refine, personalize, magnify, discover, share, reshare, and overshare experiences. This shift from the creation, transmission, and consumption of information to the creation, transmission, and consumption of experience defines this new platform. As Magic Leap founder Rony Abovitz puts it, “Ours is a journey of inner space. We are building the internet of presence and experience.”

We haven’t yet fully absorbed the enormous benefit that the internet of information has brought to the world. And yet we are about to recapitulate this accomplishment with the advent of synthetic realities. 

With a VR platform we will create a Wikipedia of experiences, potentially available to anyone, anywhere, anytime. Travel experiences—terror at the edge of an erupting volcano, wonder at a walking tour of the pyramids—once the luxury of the rich (like books in the old days), will be accessible to anyone with a VR rig. Or experiences to be shared: marching with protesters in Iran; dancing with revelers in Malawi; how about switching genders? Experiences that no humans have had: exploring Mars; living as a lobster; experiencing a close-up of your own beating heart, live.

You’ve seen a lot of this in movies and on TV or read about it in books. But you haven’t experienced it, felt it below your intellect, had it lodge in your being in a way that you can call your own. Kent Bye, founder of the podcast Voices of VR, has conducted over 400 interviews with the people creating VR and has seen almost every possible prototype of VR there is. “VR talks to our subconscious mind like no other media,” he says.

The most intense and complete sense of subconscious presence that I experienced occurred with a system called the Void, which debuted at the 2016 TED conference. The Void isn’t as advanced as Magic Leap technologically, but it integrates the best off-the-shelf parts available with custom gear to create an unforgettable experience. For several hours I watched a line of people enter the Void. Almost every person squealed with delight, screamed, laughed, and staggered away asking for more. I felt the same; I’d be happy to pay for an hour’s visit.

The Void grew out of stage magic, a theme park, and a haunted house. Every year, Ken Bretschneider, one of the three cofounders, stages a gonzo haunted house in Utah that draws 10,000 people in two days. It occurred to him that he could amplify the interactions of his house with VR. Curtis Hickman , the second cofounder, is a professional illusionist, designed tricks for big-name magicians, and is also a visual-effects producer. The third, James Jensen, started out developing special effects for film and unique experiences for theme parks. He came up with the idea of layering VR over a physical playground. The common factor among the three was their realization that VR was a new way to trick the mind into believing something imaginary is real.

Real Worlds

Virtual, mixed, augmented—that’s a lot of reality. Here’s how to keep it all straight.

—­Blanca Myers

Virtual Reality

VR places the user in another location entirely. Whether that location is ­computer-generated or captured by video, it entirely occludes the user’s natural surroundings.

Augmented Reality

In augmented reality—like Google Glass or the Yelp app’s Monocle feature on mobile devices—the visible natural world is overlaid with a layer of digital content.

Mixed Reality

In technologies like Magic Leap’s, virtual objects are integrated into—and responsive to—the natural world. A virtual ball under your desk, for example, would be blocked from view unless you bent down to look at it. In theory, MR could become VR in a dark room.

The Void takes place in a large room. You wear a 12-pound vest that carries batteries, a processor board, and 22 haptic patches that vibrate and shake you at the right moments. Your headset or goggles and earphones are connected to your vest, so you’re free to roam without a cord. Untethered, you’re released from worrying about tripping over a cable or tangling or straying too far. That relief heightens the effect of being present in the VR. Inside, you navigate an Indiana Jones-like adventure that seems to take place over a large territory. The illusion of unbounded space, or, as Hickman describes it, “a magical space bigger inside than it is outside,” is achieved by a trick called redirected walking.

As an example, whenever you turn 90 degrees in the room your VR will show you the room turning only 80 degrees. You don’t notice the difference, but the VR accumulates those small 10-degree cheats on each turn until it redirects your route away from a wall or even gets you to walk in a circle while making you think you’ve walked a mile in a straight line. Redirected touching does a similar trick. A room could contain one real block but display three virtual blocks on a shelf—blocks A, B, and C. You see your hand grab block B, but the VR system will direct your hand to touch the only real block in the room. You can replace block B and pick up block C, but in reality you’re picking up the same real block.

It’s astounding how those tiny misdirections fool your gut into believing that what you’re seeing is real. Stairs can be made to feel endless if they drop down as you walk upward. In fact, at one point in the Void a decaying floor collapses while you’re walking across it, and you see, hear, and feel—in all your body—a plunge down to the floor below. But in fact the real floor only sinks 6 inches. You can easily imagine a room 60 by 60 feet packed with a minimal set of elemental shapes, ramps, and seats, all recycled and redirected for a variety of multihour adventures.

Seeing, it turns out, is not believing. We use all our senses to gauge reality. Most of the high-end VR rigs on sale this year include dynamic binaural—that is, 3-D—audio. This is more than just stereo, which is fixed in space. To be persuasive, the apparent location of a sound needs to shift as you move your head. Deep presence includes the sensations of motion from your inner ear; if the two are out of sync with what you see, you get motion sickness. Good VR also includes touch. Jason Jerald, a professor at the Waterford Institute of Technology who wrote the book on VR (called The VR Book), claims that much of our sense of presence in VR comes from our hands. Gloves are still not consumer-ready, so hardware makers are using simple controllers with a few easily operated buttons. When you wave them, their positions are tracked, so you can manipulate virtual objects. As primitive as these stick-hands are, they double the sense of being present. Touch, vision, and sound form the essential trinity of VR.

Virtual reality hacks the brain in dozens of ways.

WHILE MAGIC LEAP has yet to achieve the immersion of the Void, it is still, by far, the most impressive on the visual front—the best at creating the illusion that virtual objects truly exist. The founder of Magic Leap, Rony Abovitz, is the perfect misfit to invent this superpower. As a kid growing up in South Florida, he was enthralled by science fiction and robots. He gravitated toward robots as a career and got a degree in biomedical engineering from the University of Miami. While still a grad student, he started a company that built robots for surgery. Before the company got off the ground, his only income was $30 a week drawing cartoons for his college newspaper. Most people find Abovitz’s cartoons more weird than funny. They are stream-of-consciousness doodles featuring alien creatures, annotated by tiny inscriptions that include secret messages to girlfriends. They do not appear to come from the mind of an engineer. As it happens, though, good virtuality takes both fantasy and physics.

Abovitz is heavyset, bespectacled, and usually smiling. He is warm and casual, at ease with himself. But he vibrates. He hums with ideas. Overflowing. One idea unleashes two more. He whips his large head around as he speaks, sweeping up more ideas. It’s hard for him to throttle their escape, to slow down how fast they issue from his brain. As in his cartoons, a discussion can leap almost anywhere. Most of his ideas seem to combine physics and biology. In his Twitter bio, Abovitz describes himself as a “friend of people, animals, and robots,” which is pretty accurate. In his conversation and his work he exhibits a rare sensitivity to both the logic of machines and the soul of biology. If you’re making robot arms that help human doctors carve into living flesh, you have to obey the laws of physics, the laws of biology, and the minds of humans. Abovitz has a knack for all three realms, and his surgery robots sold well. In 2008 his company, Mako, went public. It was sold in 2013 for $1.65 billion.

That success sparked a new idea. Could you make a virtual knee good enough to help repair a real knee? Could you augment a knee operation with an overlay of a virtual knee? Abovitz began thinking about the technology that could match virtual worlds with complex real-life surgery.

At the same time he began to create a graphic novel.

Abovitz has a deep love of science fiction, and he invented a whole world on another planet—flying whales, men in dragonfly gear, a young girl with a pet monkey-bat, and an invading army of robots. Flush with cash from his robotics company, he hired Weta Workshop, the New Zealand special-effects house co-owned by movie director Peter Jackson, to create a detailed realization of that world. The Weta team created all the props and practical effects for The Lord of the Rings, and they helped invent the culture of the Na’vi in Avatar. For Abovitz they designed his world, called Hour Blue, and filled in the details of flying whales and monkey-bats. It quickly mutated from graphic novel into virtual-reality precursor. Because what alien world would not be better experienced in immersive 3-D? Abovitz was already pioneering MR for doctors; this would be an extension of his ideas.

The company Abovitz set up to develop this immersive world was Magic Leap. Its logo would be his totem animal, the leaping whale. The hardware to create the MR would have to be invented. By this time, 2012, the Oculus Kickstarter campaign had launched, and other prototypes with similar phone-based technology were in the works. Here Abovitz deviated off the main path. Because of his work in biomedicine, he realized that VR is the most advanced technology in the world where humans are still an integral part of the hardware. To function properly, VR and MR must use biological circuits as well as silicon chips. The sense of presence you feel in these headsets is created not by the screen but by your neurology. Tricks like redirected walking operate in our brain as much as in the Nvidia processor. Abovitz saw artificial reality as a symbiont technology, part machine, part flesh. “I realized that if you give the mind and body what they want, they’ll give you back much more,” he says.

Artificial reality exploits peculiarities in our senses. It effectively hacks the human brain in dozens of ways to create what can be called a chain of persuasion. In a movie, our brains perceive real motion in a sequence of absolutely still images. In the same way, you can scan a blue whale from many angles and then render it as a 3-D volumetric image that can be displayed on a headset screen and viewed from any position. Even if we know the object isn’t real—say it’s Godzilla instead of a whale—we feel subconsciously that its presence is real.

But if even one small thing is misaligned, that discrepancy can break the gut-level illusion of presence. Something as simple as having to 

As virtual (and mixed) technology improves—and as companies start smelling profits—everyone from phone manufacturers to tech giants is getting into the game. Here’s the hardware that VR’s and MR’s biggest players are cooking up. —Chelsea Leu

In 2014, Facebook bought Oculus, the company that dreamed up the Rift headset and (literally) kick-started the 

The Taiwanese phone manufacturer teamed up with game maker Valve Software to launch a high-end headset, 

Unlike the Rift and the Vive, Sony’s PlayStation VR is designed to work not with a PC but its own game console—which more than 36 million already own. Read more

Google created Cardboard, its cheap assemble-it-yourself viewer, to bring virtual reality to the masses via their smartphones. Read more

The Gear VR straps a Galaxy smartphone (new models only) to your head to deliver games and apps—all powered by Oculus software. Read more

An open source platform for VR and MR launched in 2015 and backed by a consortium of companies like Intel and gaming outfit Razer. Read more

This startup proposes to use eye tracking to sharpen what you focus on and blur everything else, cutting down on processing power.

The company’s HoloLens is a wireless, wearable, sensor-packed computer that aggregates its data to embed holograms in the user’s environment. Read more

Initially funded via Kickstarter in 2013, the startup has created a visor that projects virtual interactive displays in the wearer’s field of vision.

Cupertino has acquired VR-related companies and patented an iPhone-compatible headset—but hasn’t announced anything yet. Read more
 

worry about tripping over a tethering cable can seed our unconsciousness with doubt. It might look like it’s there, but it won’tfeel there.

Following his hunch to exploit human biology, Abovitz set off to make an artificial-reality display in a more symbiont way. The phonelike screens used in the majority of head-mounted displays created a nagging problem: They were placed right next to your eyeballs. If the device is generating the illusion of a blue whale 100 feet away, your eyes should be focused 100 feet away. But they’re not; they’re focused on the tiny screen an inch away. Likewise, when you look at a virtual jellyfish floating 6 inches from your face, your eyes are not crossed as they would be in real life but staring straight ahead. No one is conscious of this optical mismatch, but over long use the subconscious misalignment may contribute to frequently reported discomfort and weaken the chain of persuasion. Magic Leap’s solution is an optical system that creates the illusion of depth in such a way that your eyes focus far for far things, and near for near, and will converge or diverge at the correct distances.

In trying out Magic Leap’s prototype, I found that it worked amazingly well close up, within arm’s reach, which was not true of many of the other mixed- and virtual-reality systems I used. I also found that the transition back to the real world while removing the Magic Leap’s optics was effortless, as comfortable as slipping off sunglasses, which I also did not experience in other systems. It felt natural.

Magic Leap’s competition is formidable. Microsoft is now selling development versions of its mixed-reality visor called the HoloLens. The technology is unique (so far) in that the entire contraption—processor, optics, and battery—is contained in the visor; it is truly untethered. Meta, another startup, has released an MR device that began, like Oculus, with a Kickstarter campaign. The headset is tethered to a computer, and dev kits should hit the market this fall—likely well before Magic Leap.

Abovitz realized that VR is the most advanced technology in the world where humans are still an integral part of the hardware.

All three major MR headsets rely on images that are projected edgeways onto a semitransparent material—usually glass with a coating of nanoscale ridges. The user sees the outside world through the glass, while the virtual elements are projected from a light source at the edge of the glass and then reflected into the user’s eyes by the beam-splitting nano-ridges. Magic Leap claims that its device is unique in the way it beams light into the eye, though the company declines to explain it further at this time.

However Magic Leap works, its advantage is that pixels disappear. Most screen-based, head-mounted VR displays exhibit a faint “screen door” effect that comes from a visible grid of pixels. Magic Leap’s virtual images, by contrast, are smooth and incredibly realistic. But in truth, the quality of displays in all alternative-reality gear—VR and MR alike—is improving rapidly. Month by month the resolution of all visors increases, the frame rate jumps, the dynamic range deepens, and the color space widens. Within two decades, when you look into a state-of-the-art virtual-reality display, your eye will be fooled into thinking you’re looking through a real window into a real world. It’ll be as bright and crisp as what you see out your window.

Once this small display perfects realism, it becomes the one display to rule them all. If a near-eye screen offers sufficient resolution, brightness, breadth, and color richness, it can display any number of virtual screens, of any size, inside it. While I was wearing the photonic spectacles of Magic Leap, I watched an HD movie on a virtual movie screen. It looked as bright and crisp as my 55-inch TV at home. With Microsoft’s HoloLens on, I watched a live football game on a virtual screen hovering next to a web browser window, alongside a few other virtual screens. I could fill my office with as many screens as I wanted, as big (or small) as I desired. I could click for a screen overlaid anywhere in the real world.

One of Microsoft’s ambitions for the HoloLens is to replace all the various screens in a typical office with wearable devices. The company’s demos envision workers moving virtual screens around or clicking to be teleported to a 3-D conference room with a dozen coworkers who live in different cities. I found virtual screens and virtual media within a virtual reality surprisingly natural and practical. At Magic Leap, the development team will soon abandon desktop screens altogether in favor of virtual displays. Meron Gribetz, founder of Meta, says that its new Meta 2 mixed-reality glasses will replace monitors in his company of 100 employees within a year. It’s no great leap to imagine such glasses also replacing the small screens we all keep in our pockets. In other words, this is a technology that can simultaneously upend desktop PCs, laptops, and phones. No wonder Apple, Samsung, and everyone else is paying attention. This is what disruption on a vast scale looks like.

Exclusive Footage of What It’s Like to See Through Magic Leap

Peter Jackson agrees. The director strides into a bright sunny room in his film studio outside of Wellington, New Zealand. Dressed in shorts, he looks like a hobbit who has escaped the makeup department down the street. He is short and round with a bulbous nose, his head wreathed in unruly hobbit hair. His bare feet are large and hairy. Jackson says he is less than excited with making movies these days; not the content but the process. He sees artificial reality as virgin territory for telling stories and creating new worlds. Jackson serves on an advisory panel for Magic Leap, and his company will produce content for the new gear. “This mixed reality is not an extension of 3-D movies. It’s something completely different,” he says. “Once you can create the illusion of solid objects anywhere you want, you create new entertainment opportunities.”

Jackson has been inspired by working with early prototypes of the Magic Leap glasses. “I find mixed reality much more exciting than VR,” he says. “Mixed reality doesn’t take you out of this world. Instead it adds elements to our real world. And it has great flexibility. You can add as little as you want—a single tiny figure on this tabletop talking to us—or you can replace the walls of this room with a skyscape so we’re sitting here watching clouds float by. If you have your Magic Leap glasses on, you can look up at the Empire State Building and watch it being built in the early 1930s, floor by floor, but sped up. Maybe while you are walking around the modern streets of Chicago you see gangsters driving past with tommy guns. It could be a form of education, entertainment, and tourism. In 10 years I expect that mixed-reality technology like Magic Leap will be used as much as, if not more than, smartphones.”

Jackson is sitting in a plush chair and puts his bare feet up on the coffee table. “Most science fiction films contain some form of what Magic Leap is, whether it’s moving data around with a flick of your finger or a holographic phone call or a 3-D chess game. It’s been in our consciousness for a long time. Like flying cars. But this will probably beat flying cars.”

Weta’s master skill is in making imaginary worlds believable (and thrilling) by attending to the details. Blockbuster MR and VR worlds will require the highest level of world-building. The inherent freedom of the audience to move around, to peek at the underside of things, to linger and appreciate the details, means that great effort and skill will be needed to preserve the chain of persuasion for all the things that make up that world.

Weta is working with Magic Leap to develop a small virtual world called Dr. Grordbort’s, based on sculpted ray guns. Leading this effort is Richard Taylor, who has been building worlds, often with Jackson, for nearly 30 years. Taylor has been a sculptor all his life. His love of materials—clay, stone, wood, brass, fabrics, glass—is evident throughout his workshop, which is densely crammed with hundreds of indescribably beautiful objects. The move to virtuality is a big step for him. “I was not prepared for the emotional impact of Magic Leap,” he says. “I could not have thought I would crave to be in a world with virtual artifacts and characters. But once I got over the surprise that this really works, I’ve had to rein in my ideas.” 

It will be the most social medium yet. More social than social media is today.

ARTIFICIAL REALITY WILL need world builders like Taylor and Jackson to invent the grammar of VR and MR. It took decades for the grammar of film to evolve. Cinema techniques like the establishing shot, the dissolve, and the close-up all had to be invented and then absorbed until everyone knew what they meant. None of these techniques work very well in virtual reality. It’s already clear that the language of experiences is different from what’s come before. One example: First-person point of view is the default stance for many of the videogame franchises dominating best-seller lists. Among them is Minecraft, which is played by more than 100 million people on the screens of PCs, tablets, and phones. Inside the game you see your hand or a pick. But in the virtual-reality version ofMinecraft that Microsoft is building, the experience of holding the pick and chopping the blocks is so immediate and real—even though the blocks are cartoon pixels—that the player’s own presence is greatly amplified. Their sense of being shifts inward. In tests with volunteers,Minecraft developers discovered that performing the same role in VR feels far more intimate than it does in first-person on a flatscreen. We might call this new immersive VR view the “you-person” view, because it’s the position of feeling rather than the position of observing.

Researchers found that the you-person view that VR creates is so intense that it’s emotionally taxing. People need a break after an hour. Curiously, if someone stays inside VR but pulls up a virtual flat-screen version of Minecraft and continues playing in the traditional 2-D first-person view on a virtual monitor (still wearing the VR gear), they will feel more at ease. Once rested enough by playing in first-person mode, they often switch back to the fully immersive VR.

Oculus’ Palmer Luckey is exuberant. He likes to bounce. He pumps his arms as he speaks. That body language crossed over into VR.

The degree of presence can be so strong in VR that you have to tone down the evocation of base emotions and the depiction of brute force. The usual gore and mayhem of a first-person shooter doesn’t work as well in VR. Exaggerated scenarios that are merely compelling in a flat world can be overwhelming when you’re immersed in them.

All that said, it was not the reality of artificial reality that surprised me most. It was how social it is. The best experiences I had in VR or MR involved at least one other person. More people made it better. In fact, just a few more people made it exponentially better. It’s a network effect: The joy of VR is proportional to the square of the number of people sharing it. That means VR will be the most social medium yet. More social than social media is today.

One of my first tests for the quality of virtual reality was something I call the bat-flinch test. If you stood next to someone who was holding a virtual baseball bat and they swung the bat at you, would you duck? Only if you truly believed in it. Otherwise you’d just laugh or maybe wait to see what getting hit “felt” like. You’d never wait to get hit in real life.

But a better test for VR is the poker game test. Do the avatars sitting across from you convey sufficient subtle eye contact, body language, and social presence that you can tell if they’re bluffing?

I visited an Oculus demo at Facebook’s campus, and Palmer Luckey, Oculus’ creator, joined in. We shared a virtual playground. In real life, Luckey is exuberant. He likes to bounce. He pumps his arms, not just his hands, as he speaks. That body language crossed over into VR. Even though our avatars did not map our outside visual features, Luckey’s avatar—a ghostly blue head and two ghostly hands—moved just like him. He was playfully throwing blocks at me. They passed the bat-flinch test because I was ducking. Luckey was an expert in lighting virtual firecrackers and fireworks and tossing them my way too. Their explosions were real enough that I needed to back away. His enthusiasm was contagious, so I tried to blow him up with a blaster, but I missed and knocked down a tower. While the physics of this demo, called Toybox, were remarkable—things bounced or collided with amazing verisimilitude—the toys felt real in large part because we could pass them around, share them, and collaborate on moving them. My experience was not with toys but with another person.

“Our goal is to make virtual communication even better than real-world communication,” Luckey said. “VR is the only thing that will get us there.”

The time is coming when, if someone says “let’s meet,” everyone will know that means let’s meet in VR. The default mode of VR is “together.”

AR, VR, MR: Making Sense of Magic Leap and the Future of Reality

Very soon, perhaps in five years, the bounded worlds within virtual reality will begin to be networked together into distributed virtual worlds. When you’re wearing the visor of an augmented- or mixed-reality system such as Magic Leap, HoloLens, or Meta, it maps the local environment. To make, say, a virtual teacup appear on your real table, it needs to know where your table is. The visor uses outward-facing cameras and sensors to scan your environment to create this map. Magic Leap (among others) is working on protocols that save a mapped place in the cloud so it doesn’t have to be remapped for each encounter. Your unit (or perhaps another unit in the same location) merely needs to register and update any changes in the space. This in turn will let you share virtual objects across different surroundings, even if participants are in distant places. Someone in Barcelona can drop a virtual flower into your virtual vase in Chicago. Because artificial reality is inherently social, its environments will be inherently social and networked.

That’s not to say this will be easy. Don’t let the relatively portable size of VR and MR wearables fool you. As they get smaller and lighter (and they will), the infrastructure behind them must grow larger and larger. The scale of the servers, bandwidth, processing, storage, and cleverness required to run networked virtual places at the scale of the planet for billions of people is beyond Big Data. It is Ginormous Data.

Which raises another issue. One of the underappreciated aspects of synthetic reality is that every virtual world is potentially a total surveillance state. By definition, everything inside a VR or MR world is tracked. After all, the more precisely and comprehensively your body and your behavior are tracked, the better your experience will be.

During a virtual journey, whether it lasts two minutes or two hours, the things your gaze lingers on, the places you choose to visit, how you interact with others and in what mood could all be captured in great detail to customize the experiences to your preferences and tendencies. But many other uses for this data are also obvious. This comprehensive tracking of your behavior inside these worlds could be used to sell you things, to redirect your attention, to compile a history of your interests, to persuade you subliminally, to quantify your actions for self-improvement, to personalize the next scene, and so on. If a smartphone is a surveillance device we voluntarily carry in our pocket, then VR will be a total surveillance state we voluntarily enter.

As far as I can tell, there are no VR systems that currently store the data they track or do anything with it beyond the first-order job of creating the world and your avatar. While they’re aware of this potential, they are simply too consumed with getting the virtual worlds to work to bother with exploiting the data feed. Inevitably, however, some will graduate to view this immense trove of personalized data as a commercial treasure. The familiar puzzles of its legal status, who has access to it, what government claims apply, and what can be done with it will occupy us as a society in the near future. It’s very easy to imagine a company that succeeds in dominating the VR universe quickly stockpiling intimate data on not just what you and 3 billion other people favorite but what you do on weekends, what people you pay attention to, what scares you, where you go when you’re tired, how you greet strangers, whether you’re depressed, and a thousand other details. To do that in real life would be expensive and intrusive. To do that in VR will be invisible and cheap. 

It’s fun to explore VR, but it’s more fun to create in it.

THE CREATION OF global artificial reality is an enormous project, and its adoption will start slowly. In every VR demo I tried in the past few months, I needed assistance to get the gear on and adjust the fit. Most demos required spotters to watch me. There were straps to deal with, cords to trip over, furniture to avoid. The software was glitchy. And too often, the demo required outsiders to suggest that I “turn around and look over there,” because user interfaces are still lame. “Right now VR systems, particularly the tracking, don’t work without constant technical maintenance,” says Jeremy Bailenson, who directs the Virtual Human Interaction Lab at Stanford. “I’ve been running VR for 20 years, and the bane of my existence is driver updates. VR is ready to flourish anywhere it’s worth hiring someone to maintain it.” Some of these problems are the ordinary growing pains of the prototype phase. But there are also some fundamental features missing. Chris Dixon, a partner in venture capital firm Andreessen Horowitz, who led his company’s early investment in Magic Leap, thinks VR will follow the flywheel effect: sluggish to start, its momentum slowly compounding until it’s nearly unstoppable. “What gives me hope is how good VR is right now,” he says. “Once people experience high-end VR, they’re going to want it. We’ll look back on 2020 as the VR era, but in the next five years I’m bracing for the inevitable trough of disillusionment in the hype cycle.”

As the flywheel slowly begins to turn, friction will hinder its rotation. But those friction points should also be viewed as fresh opportunities. These are problems whose solutions will enable many other innovations. Any of the following pain points might be the opening that produces the first VR billionaire:

The Dork Factor

There’s no getting around the fact that everyone looks like a dork wearing a head-mounted display. It obscures our humanity. The failure of Google Glass was in large part due to the fact that you could not pass the cool test wearing one. Remember the Segway, the stand-up personal transport? If you haven’t ridden a recent version of it, you should; they’re amazing. But even though the scooter really works, it didn’t revolutionize transportation, in part because people looked ridiculous riding it. The form factors of VR and MR have a long way to go before they become culturally invisible.

Safety

I nearly fell in a recent VR journey because I tried to jump into a pit that wasn’t really there. Oculus weirdly warns its users to “”remain seated at all times. The problem is, if you’re present—really present—in an alternative place, you’re absent from the place your body is. That’s a recipe for accidents. Mixed reality, where the room you’re actually in remains visible, can diminish the clumsiness between realms but doesn’t eliminate it. Then there is our ignorance of the long-term effects of fooling your mind and body. This is so new we don’t even know yet what questions to ask. We do know that motion sickness is real. Jeremy Bailenson found that approximately one in 30 are susceptible. But what other problems will arise after tens of thousands of hours of use?

Inadequate Interface

At this moment in its development, VR is at the same infant stage as early PCs that required a command-line input. There are no intuitive tools for easy creation. The VR industry is waiting for its Doug Engelbart to invent the equivalent of the mouse. This shortcoming is perhaps the most critical missing piece preventing a rapid takeoff. Without an interface that anyone can grasp in minutes, content can be made only by the truly dedicated.

Nearly all of the non-movie VR experiences uploaded to date were created using a computer-game engine from either Unity or Unreal (and nearly all VR so far shares a similar videogamey look too). All these first-generation experiences were created with 2-D tools—screen, windows, mouse. But VR cannot reach ubiquity until the tools for VR creation live in VR itself, until VR is bootstrapped from within VR. The first steps toward native tools were announced this spring. Both Unity and Unreal have demo’d a VR version that permits users to make VR in VR. However, to foster a smooth transition, the VR versions of both creation engines import 2-D metaphors (like menus)—the equivalent of a command line—into VR. Still missing is the breakthrough insight that takes advantage of VR’s peculiarities to deal with VR’s complexities.

I had an aha moment inside a VR app called Tilt Brush that was purchased by Google. I was using a brush to paint with light in three dimensions. My traces in the air could be thin, thick, flickering, pulsating, solid sheets, of any color. I was inside my creation, moving around with my whole body, working up a sweat. I was sketching a sculpture or sculpting a sketch or architecting a drawing or dancing up a building of light—I don’t know what to call it, but it was the most fun I’ve ever had in VR. And it’s not just for fun. Trials at Google revealed Tilt Brush could be an ideal prototyping tool. In a few minutes, even an untrained person could sketch out a design for a car or the layout of furniture in an office, and you would instantly see it.

My aha was that at its root, VR is as much a creation tool as a consumption tool. As much fun as it was to explore VR, it was more fun to make it. For a long time, no one believed amateurs would make their own videos, but that changed when you could easily film a scene by holding up a phone. VR is in line to reduce the barriers to creation even further.

Fame awaits the genius who figures out the elegant VR interface for VR creation. The tools would allow you to manipulate 3-D space with minimal gestures, voice, and gaze. You’d lift, twist, speak, and nod just so. I suspect there would be a beauty in watching a skilled creator work in VR, much like in watching a woodworker or dancer. A universal interface for working in VR would unleash the greatest expression of creativity the planet has yet seen.

Narrow Field of View

Right now the field of view in mixed-reality devices is too narrow. Of the current crop of MR spectacles, Meta 2’s field of vision is the widest, but even its coverage is inadequate. Virtual objects that are located directly in front of you, within the coverage of the screen, appear present. But when you turn your gaze away, they disappear from your peripheral vision. This breaks the chain of persuasion. Fully enclosed VR devices don’t suffer the same drawback; because you see nothing at all in your peripheral vision (only deliberate blackness), you don’t get contradictory information. Objects disappear when you turn, but the background area does too.

All mixed-reality systems labor under a second challenge that VR systems don’t: Ideally, in a mixed reality, the virtual teacup you see on your desk would be lit with the same kind of lighting, from the same direction, with the same color tone, as your real desk. To do that would require outside cameras and software that dynamically computes the lighting in the room in real time. No mixed-reality rig can do that now. The mismatch in the lighting is another weak link in the chain of persuasion. In my experience, this discrepancy tends to produce an effect I would call “artificial things really present.” You don’t confuse artificial objects with real things really present; they are artificial things really present.

Tethers

It’s hard to overstate the benefit of wearing a lightweight device that is not tethered to a fixed location. Being free to roam deepens the sense of presence, while worry about a cable tends to disrupt the spell. Screens and processors can be made much smaller, even down to a size that will fit invisibly into glasses, but batteries are the bugaboo of VR. The computational load of VR is so huge that untethered headsets will be very difficult to fuel. It’ll be a long while, if ever, before a day’s worth of battery power can be squeezed into the frames of glasses. For now they will be wired to a battery in your pocket.

THE COEVOLUTION OF science fiction and innovation is slowly being recognized as a paramount cultural force. Talk long enough to any engineer working on VR and they will eventually mention one of two books: Snow Crash or Ready Player One. Ernest Cline, the author of Ready Player One, invented the Oasis, a vast, networked virtual universe with virtual planets, where billions of people remain immersed for school, work, and play. In a delicious example of the recursive nature of science fiction’s sway, Cline’s invention of the fictional Oasis may become reality. “There’s a chance the studios will make a rudimentary virtual experience based on the fictional Oasis in the movie,” Cline says. “If it were to catch on and slowly evolve, then there is the possibility that the Oasis could become an actual real thing used by millions—as a result of me imagining it in my novel.”

Among the first people Abovitz hired at Magic Leap was Neal Stephenson, author of the other seminal VR anticipation, Snow Crash. He wanted Stephenson to be Magic Leap’s chief futurist because “he has an engineer’s mind fused with that of a great writer.” Abovitz wanted him to lead a small team developing new forms of narrative. Again, the mythmaker would be making the myths real.

The hero in Snow Crash wielded a sword in the virtual world. To woo Stephenson, four emissaries from Magic Leap showed up at Stephenson’s home with Orcrist—the “Goblin-cleaver” sword from The Hobbit trilogy. It was a reproduction of the prop handcrafted by a master swordsmith. That is, it was a false version of the real thing used in the unreal film world—a clever bit of recursiveness custom-made for mixed reality. Stephenson was intrigued. “It’s not every day that someone turns up at your house bearing a mythic sword, and so I did what anyone who has read a lot of fantasy novels would: I let them in and gave them beer,” he wrote on Magic Leap’s blog. “True to form, they invited me on a quest and asked me to sign a contract (well, an NDA actually).” Stephenson accepted the job. “We’ve maxed out what we can do with 2-D screens,” he says. “Now it’s time to unleash what is possible in 3-D, and that means redefining the medium from the ground up. We can’t do that in small steps.” He compared the challenge of VR to crossing a treacherous valley to reach new heights. He admires Abovitz because he is willing to “slog through that valley.”

It’s too early to know what virtual reality is or what it will be. Abovitz believes that synthetic reality is the ultimate human medium because it is so directly wired to our brains. “Our brain is an amazing sensory computer. Magic Leap is just the pen and paper, the typewriter, or the canvas and brush for a power that people have had brewing in them since people first appeared. The real way to the future is biology.” One thing we do know: The evolution of technology can take curious turns. Cell phones started out so bulky they needed their own luggage. It was easy to imagine them getting smaller and smaller. Which they did. But they did not merely shrink into miniature versions of themselves. As it got smaller, the mobile telephone lost its keypad, gained a hi-res color screen, started to grow in size again, and eventually stopped being used as a phone. It evolved into something different and unexpected. VR will surprise us too.

Not immediately, but within 15 years, the bulk of our work and play time will touch the virtual to some degree. Systems for delivering these shared virtual experiences will become the largest enterprises we have ever made. Fully immersive VR worlds already generate and consume gigabytes of data per experience. In the next 10 years the scale will increase from gigabytes per minute to terabytes per minute. The global technology industry—chip designers, consumer device makers, communication conglomerates, component manufacturers, content studios, software creators—will all struggle to handle the demands of this vast system as it blossoms. And only a few companies will dominate the VR networks because, as is so common in networks, success is self-reinforcing. The bigger the virtual society becomes, the more attractive it is. And the more attractive, the bigger yet it becomes. These artificial-reality winners will become the largest companies in history, dwarfing the largest companies today by any measure.

I don’t know if Magic Leap will be one of those companies. It’s not going to win the race to be first in this category, but none of the current titans were first to their markets. While Magic Leap has filed more than 150 patents, it has not yet publicly demo’d a prototype. Most important, we still don’t know enough about human perception to know what will work in virtual domains; it’ll take more VR to figure that out. We must navigate the treacherous valley before reaching new heights.

Yet something certainly has just happened. A threshold has been crossed. After a long gestation, VR is good enough to improve quickly. It’s real.