If you’re developing for the hololens, one of the coolest features you can have is see through walls.  Of course you can’t really see ‘through’ a wall, but you can place holograms in a way that they look like they’re inside or behind the wall.  For this to work though, you need to be able to mask out the wall in the foreground.

As an example, I’ll show a breakable wall we built at the last HoloHack, where you could see the cheese inside.

The wall is made up of 4 parts..

1. Plaster
2. Bricks
3. Cheese (inside)
4. The Occluder (makes it all work)

 

 

It’s hard to see in screenshots, but the 3d depth really shows when you look on the hololens.

What’s with the big black cutout?

That’s the occluder.  It’s actually just a mesh, cut out int he shape we wanted the wall.  The material on it is using the Unlit Color shader set to black.

On HoloLens, solid black is rendered transparent.  With a mesh behind it, and the occluder on the wall, this actually makes the real world wall not visible, allowing you the ability to see ‘inside’ the wall.

I wish I could show a video of how this works, but recording on a hololens doesn’t show the occluder right (it looks correct in the headset but not the recording).

It’s also very important that you use an unlit shader (either unlit/color, or your own).  Any variance from a true black (0, 0, 0), will prevent this from working.

One of the best examples of this functionality I’ve seen is in Robo Raid.  You can see a quick video of it here.

Last weekend, I had the privledge of attending the LA HoloHack.  The HoloHack events are put on by Microsoft to expose more developers to the HoloLens, see what they build, and expand the platform.  I love hackathons, so as soon as I heard about it, I signed up, and with a team of experienced developers, we pre-planned out project and arrived excited to build something great.

Prior to this event, I’d used the HoloLens a handful of times.  I’d gone through the best demos, attended a couple HoloLens meetups, and spent some time with the emulator, but this was going to be the first full HoloLens project we’d all get to build.

What I didn’t Like

Setup

The setup process for developing on HoloLens seems like it’d be pretty straight forward.  Install Unity, and the tools on the HoloLens page.. then good to go, right?

Unfortunately, with my laptop, I had previously installed VS 2015, but did not check the UWP boxes during install.  Because of that, I couldn’t get anything to work.. the emulator wouldn’t install, I couldn’t do a hololens build..

And the only way to fix it was to re-run the VS2015 installer (and check the UWP boxes)..

It’s supposed to work with 2017, but it didn’t work for me, and the developers on my team who tried using it had constant VS crashes.

Field of View

Everyone I’ve ever talked to about the HoloLens complained about FOV..  For good reason, it’s really small, and can be really limiting.  It’s not so much the horizontal FOV either.. while that’s small, it’s managable.  The vertical FOV is only about 16deg..

Now while it’s frustrating and limiting, it doesn’t mean the device is DOA, just that you need to really heavily consider the constraints when designing.

In our case, the original game idea we had included a lot of looking up and down, which doesn’t work well at all with that vertical FOV.  This was a big hit for us, but a good learning experience for myself.  It was a reminder to always understand the limitations of the hardware and really consider them when designing.  While a game that requires lots of up and down scanning won’t work well, there are quite a few things the hololens excels at which we should have taken advantage of.

What I liked

Environmental Scanning

This was amazing..  scanning the environment is not only fast, but extremely easy to setup.  By adding a single component, you can generate a mesh for the environment that’s auto updated as more is scanned in.

Placing something on the wall just requires a quick raycast against the environment layer to get the point and normal.

Want a wireframe scan visualization of the world?  No problem, just drop on another component and it’s there automagically 🙂

The same went for finding wall and floor planes, it’s all thought of and easily accessible.

Seeing through walls

This was probably the coolest part of what we did.  If you’ve never tried a hololens, it’s hard to explain.. but if you create your app right, you can make it appear as if you’re seeing into or through a portal.  In our case, we made it look like you could break out the wall and see inside (though the inside of our wall was made of cheese).  This effect works in 3D, so you can look inside the wall from any angle and see what you’d expect..  it’s mindblowing and probably the coolest effect I’ve seen on the HoloLens.

HoloLens Toolkit

Before the event, I didn’t realize there was an entire toolkit built up for the HoloLens.  I should have known Microsoft would do it.. and I’m glad they did.  The toolkit provides way more than I can describe here, so if you’re interested, check it out on their github page – https://github.com/Microsoft/HoloToolkit-Unity

Voice Commands

Setting up voice control for the HoloLens was amazingly simple.  With a couple lines of boilerplate code, and a single event registration, you can bind any command to a method call.  This is actually a feature of UWP apps on windows 10+, but I’ve never had use for it before HoloLens.  If you need voice control, it’s definitely worth trying out Microsofts solution.

Coding for the device

Since the HoloLens is so well integrated with Unity, development was extremely easy to get started with.  Other than the specific location of your MainCamera (0, 0, 0), and the requirement for a solid black clear flag, everything else was pretty much typical Unity development.  Only input handling was a little different, but it wasn’t difficult.  Overall, developing for the device was a great experience.

Multi-player

Surprisingly, many of the entries supported multiple players sharing a world.  And from the looks of it, setting this up is actually much less difficult than you’d expect.  You can learn more about it in the HoloGrams 240 course here: https://developer.microsoft.com/en-us/windows/mixed-reality/holograms_240

Recommendations

If you’re going to jump into HoloLens development, look for opportunities where the HoloLens has a distinct advantage.

Things to watch out for

• If your game will work fine in VR, it’s probably not a good fit for AR..
• If your game requires the player to look all around and keep track of multiple things, it probably isn’t a good fit..
• Games where the player needs to be very close won’t work well… the ‘ideal’ near clip plane is around 1.5m..

Things that are great

• Overlaying onto real world people.. even putting bars over their heads with extra info can be a good starting point.
• Using the environment.. placing things on walls, having things run around, float in the world..
• Interacting with the environment..  have things bounce on the floor, pop and explode, etc.
• Audio / voice control.. it works great and it’s easy to implement (just don’t use common phrases other people in the area will randomly say)

Conclusions

While our project idea totally failed, and we didn’t come close to winning.. I learned a ton at this hackathon.  If you happen to have a HoloHack pop up near you, I definitely recommend going (Microsoft hosted events are always great)..

I’m excited for the next one, and have a few ideas that I think will work much better now that I have a stronger grasp on the devices strengths.

If you have some HoloLens ideas and want feedback, feel free to reach out, I’d love to talk about it more 🙂