XR Projects

In Augmented Reality (AR) environments sound design faces the unique challenge of seamlessly integrating virtual elements into real environments while preserving the cohesion of the user’s auditory experience. An important measure of the perceived realism of the auditory layer is plausibility. The concept of plausibility in AR audio experiences is multifaceted, incorporating both acoustic and perceptual factors. While acoustic factors originate mostly from the Virtual Acoustic Environment (VAE) design and its similarity to the real acoustic space around the user, perceptual factors stem from real auditory references originating from the environment. The core empirical study of this dissertation aims to investigate the factors posed above and at the same time validate a new methodology for plausibility evaluation in AR. During the study participants rated the plausibility of pairs of loudspeakers (real or virtual) emitting sound consecutively from different positions in the room, mimicking real-life scenarios where similar but not identical sources coexist.

Lead sound designer and sound engineer for shared audience, theatrical XR experience

A multi-user XR experience that brings to life the origin story of Frankenstein through the eyes of a young Mary Shelley. Illustrating novel new ways in which immersive content can be delivered to audiences of all sizes, this theatrical mixed reality performance transports shared audiences into the author’s world as she conjures the most famous gothic tale of all time.

Lead sound designer and sound engineer for multi-audience, cinematic VR experience

Headed by Ken Perlin, the Future Reality Lab will unveil a tale that takes participants back thousands of years, to when stories were told around a campfire, using the latest in experiential technology, featuring binaural audio, mixed reality, and a co-located untethered virtual reality experience for an unprecedented 30 viewers at a time.

• Blog post describing the project.
• VRScout: VR Film Places Entire Audience Into A Shared Immersive Experience.
• Wired: Film Festivals Move to Make Virtual Reality a Marquee Event.
• Extended virtual reality experience at Siggraph 2018.

Sound designer and sound engineer for collocated VR theatrical experience

“Holojam in Wonderland” is the world's first ever collocated theater piece for multiple actors and multiple audience members to take place entirely in shared untethered Virtual Reality. All performers and audience members are physically in the same room, able to free to walk around in that room and touch each other, yet they all see each other as avatars in a shared virtual world.

• Short clip introducing the project.
• More on the project in ACM publication.

Lead sound designer and sound engineer for VR rehabilitation project for patients with postural control dysfunction

We developed a novel VR platform with 3-dimensional sounds to help improve sensory integration and visuomotor processing for postural control and fall prevention in individuals with balance problems related to sensory deficits, such as vestibular dysfunction (disease of the inner ear). The system has scenes that simulate scenario-based environments. We can adjust the intensity of the visual and audio stimuli in the virtual scenes by controlling the user interface settings. A VR headset (HTC Vive or Oculus Rift) delivers stereo display while providing real-time position and orientation of the participants’ head. The 3D game-like scenes make participants feel immersed and gradually exposes them to situations that may induce dizziness, anxiety or imbalance in their daily-living.

Sound engineer involved in building and testing infrastructure for distributed music collaboration in VR and AR.

The NYU Holodeck is a National Science Foundation MRI Track 2 Development project (NSF award # 1626098) that will create an immersive, collaborative, virtual/ physical research environment providing unparalleled tools for research collaborations, intellectual exploration, and creative output. To prototype the transition from projection to functional AR, MR and VR simulations, our interdisciplinary team will develop an instrument and educational environment that facilitates enticing research and creates an experiential supercomputing infrastructure that empowers students, faculty, and researchers to develop new knowledge and disseminate powerful new integrated, multimodal tools and techniques.

Concerts on Holodeck

• Ozark Henry on the Holodeck: Maps to the Stars, October 2018.

Co-leader of Immersive Audio Interest Group

Led a number of music recordings in VR, experimenting with different recording techniques and mixing approaches for this new medium.

The Immersive Audio Group at NYU Steinhardt is a voluntary student organization specializing in spatial audio research projects. The focus of the group lies in the capture, analysis, synthesis, and reproduction of immersive auditory environments. This includes the capture and representation of spatial and 3D sound using binaural, transaural, multi-channel and soundfield techniques, audio display devices, virtual auditory environment simulation, games, characterization, and classification of spaces as well as augmented audio reproduction.

• 360 Video of Percussion Trio performance produced by IAG.