I want to break down the huge and exciting undertaking of adding experimental Hand Tracking support to Vacation Simulator and my thoughts on hand tracking itself!

Thread incoming [1/13]

This project started as an R&D experiment to determine what all the showstoppers would be in bringing hand tracking support to Vacation Simulator on Quest. Not just some support, but the ability to play the whole game end to end if you wanted. [2/13]

My assumption was we would build a list of core things that could never work or needed vastly better hand tracking technology to accomplish. My hand tracking experiences to date had been pretty poor, so I felt this was a valid assumption but also an important exercise. [3/13]

What happened in practice was that it became more than a list of hand tracking roadblocks, instead a list of new and interesting challenges for us to explore. [4/13]

Over time our Roadblock List got smaller, not bigger and our resolve to work through the hardest challenges only increased. How do you teleport in an intuitive way? How do you grab a backpack? How do you throw with poor data? The list of challenges goes on and on. [5/13]

After many months we realized that we might actually be able to do this. We wouldn’t have to wait for some future world where the technology had vastly improved. Would it be perfect? Definitely not, but it would be unique, entertaining, and important. [6/13]

I’ve learned that games are a great way to move technology forward. The work we did in Job Simulator was instrumental in pushing the technology and the design of virtual interactions forward. A driver for this project was to help do a similar thing for hand tracking. [7/13]

To be clear, I really dislike the state of hand tracking to date. It generally consists of difficult gestures, uncomfortable poses, and unintuitive design. The culprit in my opinion is large hardware/software limitations combined with the lack of high quality games. [8/13]

My biggest hope with this project is that people can see some of the limitations that exist with hand tracking, but also more importantly that hand tracking doesn’t need to be just laser pointers and unintuitive pinch gestures, it can and needs to be so much more. [9/13]

Is hand tracking going to replace controllers for all VR games? Definitely not. Controllers are awesome, buttons are awesome, haptics and IMUs are awesome. But for certain games/content hand tracking will offer users a different, and in some cases, better experience. [10/13]

This project has been an absolutely huge amount of work, involving changes to almost every system in the game, but the end result is a game huge in scope, complexity, and interaction that I have played end to end (100% completion) with nothing but my hands. [11/13]

There are so many lessons, interactions, and discoveries we’ve made during the development of Vacation Simulator: Hand Tracking. We are preparing a developer blog post to shed some light on this new frontier. [12/13]

Check it out and let us know what you think. I know not everything will work for everyone, but I hope this provides a glimpse further into the future of what hand tracking can and should be. Check it out: https://www.oculus.com/experiences/quest/2393300320759737/
[13/13]