Inside this project
Overview
Microsoft Mesh was developed to allow enterprise customers to remotely collaborate together in a shared immersive space across different platforms, including PC and VR. But the VR experience wasn't always prioritized by the team and lagged behind industry trends.
As VR interaction moved rapidly toward controller-free experiences, Microsoft Mesh needed to evolve. Mesh's VR client historically relied on controllers for its entire interaction model. Meanwhile, industry leaders were shifting to hand tracking as the primary interaction paradigm.
To keep Mesh competitive in an increasingly crowded landscape — and to support more natural, intuitive collaboration — our team set out to design a fully gesture-based interaction system that was simple, comfortable, socially acceptable, and feasible across Mesh's large interaction surface.
Controller-Based System
Mesh’s controller system supported a wide range of interactions:
- Joystick locomotion
- Teleportation
- Picking up and throwing objects
- Pressing UI buttons
- Selecting and interacting with avatars
- Manipulating scene objects
Translation Challenge
We needed to redesign the interaction model itself.
Key constraints
- Gestures must be simple and easy to learn
- Must be comfortable for repetitive use
- Must be socially acceptable in enterprise settings
- Must minimize accidental activation
- Must scale to many interactions without overwhelming users
Deeply Technical, Highly Collaborative
This was deeply technical, highly collaborative work — and one of the most exciting explorations I contributed to in Mesh. I worked closely with 4 other designers, a PM, and multiple engineers throughout the project.
Lead Technical Prototyper
Served as the lead prototyper and a senior designer on the workstream and built multiple Unity prototypes using XRI and Oculus Interaction SDKs
Design & Validation
Designed multiple iterations of the interaction visuals and gesture logic, then evaluated and compared gesture models with engineering to validate feasibility
Establishing the Baseline: Point + Pinch
Before diving into locomotion, we aligned early on around one crucial part of the interaction system:
UI and object interaction would all use "point + pinch" — a raycast from the hand combined with an index–thumb pinch.
Why This Works
- Widely adopted across VR platforms
- Intuitive pointing and "clicking" with fingers
- Ergonomically simple
- Already the expected interaction pattern
Used For
- Pressing UI buttons
- Manipulating UI panels
- Selecting and interacting with objects
- Selecting avatars
- Triggering context-specific actions
The Real Challenge: How do we create an easy-to-learn gesture for locomotion and navigation without interfering with point + pinch for UI and object interactions?
Testing Emerging Industry Patterns for Locomotion
Gun Gesture
How It Worked
- Form "gun" gesture with the hand
- Aim with the hand
- Pinch with index + thumb to confirm
Testing aiming and confirmation against comfort and enterprise social expectations.
Critical Issues
Social Acceptability
Inspired by Meta's recommended patterns and tools, we first explored the "gun gesture":
Ergonomics
Looked unprofessional. Users in enterprise meetings looked like they were "pointing guns" at coworkers.
The posture strained the wrist when repeated frequently.
Hand Menu for Locomotion
Conclusion: Good industry direction, but not appropriate for enterprise VR collaboration.
Testing palm-triggered controls for teleportation and turning.
What Worked
We next explored a natural gesture already common in VR: look at your palm → open a hand menu → select teleport/turn buttons.
For Locomotion
Effective for UI interactions. We ultimately kept it for opening/closing the main system menu.
- UI button to teleport
- UI buttons to turn left/right
Required too many steps, felt indirect, and didn't support fluid interaction.
Pinch Gesture for Locomotion & Turning Driven by Palm Orientation
We didn't want to teach users an entirely new gesture and decided to stick with what works. If users aren't already targeting an interactable object or UI, the pinch gesture triggers a lightweight visual indicator that guides the user toward three possible outcomes, determined not only by hand position but also by palm orientation.
Teleporting (Forward)
Palm facing downward or forward
Natural because when users aim toward the ground or into open space to teleport, their palm is already oriented down or forward away from other interactables
Turning (Left or Right)
Palm oriented sideways
Rotating the palm matches the physical motion of turning and disables raycast avoiding conflicts with other interactables. Hovering toward arrows determines left or right
Cancellation
Palm rotated upward
Dismisses locomotion entirely. Allows users to abort the gesture.
These orientation-based states made it easy to fluidly move between teleport, turn, and cancel, and helped avoid conflicts between turning and teleportation.
Preventing Conflicts
We had to ensure pinch-to-locomote didn't interfere with pinch-to-select.
UI and object selection always takes precedence.
If a raycast is targeting an interactable item when the pinch begins, we treat it as a UI/interaction pinch — not as locomotion.
Micro-delay before activating locomotion
A very small, nearly imperceptible pause before triggering locomotion prevents accidental turns and teleports when users briefly touch thumb and index together unintentionally.
This solved for the vast majority of conflict cases. Teleportation naturally happens in open space, and turning doesn’t require the hand to aim at all.
Impact & Reflection
This workstream pushed us to rethink fundamental VR interactions for an enterprise audience. Hand tracking demanded a new level of simplicity, clarity, and social awareness — and forced us to unify locomotion, UI interaction, and object manipulation under a single, intuitive gesture model.
The resulting pattern gave Mesh a modern, predictable, and ergonomic hand-tracking foundation — one that could scale to more advanced interactions like the deferred rotation system and future gesture-based tools.
We successfully shipped a gesture system that was simple enough for first-time users, powerful enough for complex interactions, and professional enough for enterprise environments.
What Didn’t Make the Cut
As part of this workstream, we also designed and prototyped new patterns for the UI system to modernize it based on systems our team had developed over the years.
The UI would be draggable and moveable. Users could rearrange panels and snap them into place to account for different user needs and scenarios.
The design was great, but it didn't make the cut due to resource constraints.
A deferred exploration of rearrangeable and snapping panels.