There are robot insects bursting in through the walls. Everywhere I look, the plaster is cracking, then suddenly, out they spew, their metallic claws aimed at my jugular. It sounds like the sort of techno-hallucinogenic nightmare that filmmaker David Cronenberg may concoct in one of his woozy sci-fi horror flicks. But it isn’t. This is a demo for Microsoft’s Hololens, a forthcoming “mixed reality” headset. The future is terrifying. But also sort of amazing.
I’m standing in a quiet room, upstairs from the E3 Expo in Los Angeles, with two staff from Microsoft’s Hololens development team that’s based in Redmond. They’re going to show me a couple of demos: the robot shooting game, Project X-Ray, and the one I’m really desperate to see, Minecraft Hololens – basically, the game that stole the show at Microsoft’s pre-E3 press briefing.
Announced in January, Hololens is Microsoft’s entry into the growing arena of immersive interactive technologies. It’s a self-contained, standalone headset, featuring two transparent HD holographic lenses. When you put the headset on, the device is capable of projecting computer graphics into your field of vision – so it looks like these digital objects are part of your real-world. This seamless integration of graphical elements into your normal view differentiates the technology from virtual reality headsets like Facebook’s Oculus Rift and Sony’s Project Morpheus, which immerse you in a digitally constructed world. Confused? Um, the coming world of immersive entertainment is maybe not for you.
Hololens also features an array of sensors so it can work out where you are, how you are moving, and what’s happening in the environment around you. It has a basic understanding of the objects in your field of view, too, so it recognises doors, windows, chairs, tables – elements it can use when overlaying visuals. Plus, there’s a mini-computer – or holographic processing unit – built in, so you don’t have to plug it into a computer or games console. Hololens can go anywhere with you.
The device itself looks suitably futuristic. The visor is attached to a high quality plastic band, which also contains the sensor array and the self-cooling computer processor, as well as built-in speakers with 3D spatial sound. Within this is another band, which fits over your head, with an adjuster wheel on the back to tighten it up. I slip it on carefully, and it’s surprisingly light and easy to set in position. I’ve heard some people say it’s either too tight or too loose, but for me, the lenses stay comfortably in place, directly in front of my eyes and I can move my head without it wobbling. Hololens is a work in progress and it seems some re-designing has been carried out since the original protoypes were shown in January.
They start me off with Project X-Ray. A simple menu screen is projected onto the wall in front of me, and to select the demo, I just have to lift my hand up in front of the device’s sensors, raise a finger then make a sort of clicking gesture, like pressing the button on a mouse. This selects the demo and it starts running.
Now I’m in the game, a sort of first-person sci-fi shooter. A voice tells me that there’s an invasion of robots coming and I have to prepare myself. I have an Xbox One controller, which acts as my laser gun trigger, and it fires wherever I’m looking. Suddenly, I hear a sort of digging noise and cracks start appearing in the wall in front of me, then chunks of rock burst out and I can look into what appears to be a tunnel. A tunnel behind the wall of an E3 demo room. Then robot scorpions start running out crawling onto the wall and firing at me. They scuttle about, some launch into the air around me. Their projectiles are reasonably slow, so I can dodge them, but later enemies fire waves of bullets, which I have to duck underneath to survive.
It’s surprisingly physical. As the attack heats up, robots start bursting into the room from behind me, and either side, so I have to listen out for the tell-tale scratching noises – already I’m imagining how terrifying this would be in a horror game like Resident Evil. Some robots even burrow along the interior walls so you see masonry cracks spreading out along the entire surface. It’s weird, but incredibly involving. By the end, multiple robots are firing at me, so I use a power-up on the pad’s left trigger that slows down time, allowing me to shoot multiple scorpions, earning a combo bonus. By the end, I’m almost breathless having darted about the room firing wildly for five minutes, completely immersed and almost unaware of the two Microsoft coders watching my every move.
Certainly, I’ve played basic “mixed” or “augmented” reality experiences in the past. In 2003, an early mobile phone game called Mosquito Hunt overlaid its graphics onto the camera display, so you could jerk your phone around blasting at imaginary insects. Nintendo’s 3DS console came with a range of interesting AR demos that let mini-game characters appear out of your environment. But this is different. The fact that the screen overlays your visual field, that the computer graphics are contextual and high definition, and that the tech is using the actual walls in the room as props, all heighten the experience immeasurably.
“We knew from the beginning that Hololens would need to have an understanding of the user’s real environment – we knew it would make the gameplay unique for each person, based on where they play,” explains Microsoft’s corporate vice-president Kudo Tsunoda, just before my demo. “However, as we started building more character-driven narrative experiences, we didn’t fully understand the level of emotional engagement and intensity that you can create with somebody by having the characters and the story play out like right there in their real world.
“One of the great things about gamers is that you get so attached to your favourite characters and stories. So when you start seeing those characters coming to sit down on your sofa or interacting with different parts of your house, there’s a level of immediacy and intimacy that goes beyond anything you can experience while sitting in front of a television screen.”
This becomes obvious in our second demo, the one I’d been waiting for: Minecraft. Once again, the program projects a 2D menu screen onto a wall, which lets me select a Minecraft world and load it up. Then, I’m told to look at the table behind me and ask the application to place the world right there. Suddenly, the surface of the table seems to drop down, and out of it rises an entire Minecraft landscape, jutting up into the room, looking as solid and colourful as a Lego model. I can walk around it and peer down at the sheep milling about; I can look into the windows of a house. One of the demo staff is controlling a character and I can watch him run and leap about the scenery. With a series of simple voice commands, I can ask the camera to track the character’s movement so I can look in right behind him and follow his actions more closely.
Cleverly, Hololens also comes with an array of physical commands, like a touchscreen phone. To move the map around, I simply make a pinching motion in front of my eyes and then move my hand – the Minecraft map moves with me, up and down or side to side. If I pull it up high enough, I can see the subterranean environment, the tunnels and caverns, the flowing streams of lava. I can ask for a marker to be placed where I’m looking, so interesting spots can be explored later, I can even request for a signpost to be placed in specific areas, and then “write” a message on it using a speech-to-text interface. (Though this doesn’t seem amazingly accurate right now – I want to write “there’s gold in them there hills”, but something about a “nice weekend” comes up instead.)
It’s rather beautiful, and it suggests that Hololens could become a creative tool, allowing gamers to easily modify and create their own levels. “Most people don’t have an inherent understanding of how to create in 3D – it can be very complicated on 2D screens,” says Tsunoda. “This is one of the places that Hololens and gaming come together in a very interesting way – allowing a lot more user-generated content, not only as part of the gameplay but also involving players in the creation of the game and how the gameplay evolves. I think we’re going to see more communities adding to and customising the games they play. That will be very cool.”
The main downside right now, is the restricted field of view that the graphical elements can appear in. There is a rectangle in your immediate vision, where digital content is visible – look away from that, and you lose it. Unlike virtual reality technology that completely surrounds your vision, Hololens is only active in this one central sweet spot, it’s not peripheral.
This is fine in demos like Project X-Ray where you have limited, discreet areas of action – i.e. small robots coming through your wall. But with the Minecraft demo, it was a little frustrating, viewing this vast and beautiful 3D model, but having to keep moving your head to keep it in the centre of your vision so that the whole thing didn’t disappear. Tsunoda has said that the field may be increased before release, but not by much. I think this will very much dictate the sorts of experiences we can expect, especially in games.
Other than that, three key questions remain: price, release date, battery life. Microsoft is saying that Hololens will appear “in the Windows 10” timeframe, so late summer or early autumn, then. But we’ll see. There’s no clue on the other vital elements.
But clearly this is a fascinating new technology, not just for games, but for many areas of science, education and the arts. It will be interesting to see more from rival mixed reality solution Magic Leap which is being backed by Google; it will also be intriguing to discover what happens when these augmented reality solutions go up against the virtual reality visions of Oculus, Sony and HTC.
“VR and mixed reality are complimentary technologies,” says Tsunoda. “However, the more you can allow people to interact with their real world and real people, the more that you get everyday use cases for a technology. It’s this that can take a product into the mass market.”
Minecraft on Hololens: the future of gaming is right in front of your eyes