Producing Narration & Dialogue For Virtual Reality & 360° Video

Actor, TV host and voiceover artist Simone Kliass wearing a HTC Vive Pro VR headset at ARVORE Immersive Experiences headquarters in São Paulo, Brazil.Photo: Jason Bermingham

The buzz around immersive visual media is creating plenty of new opportunities for those who record spoken word for a living. But how do these new jobs differ from traditional ones?

I’ve written previously in SOS about overcoming the challenges of recording voiceover in various environments, ranging from the home studio to hotel rooms, cars and even a ship’s cabin. And if I’m honest, at that time I thought I’d probably taken the subject about as far as SOS readers would find useful. But the evolution of immersive media over recent years has brought some interesting changes for voice actors and dialogue recordists.

While it’s not yet completely clear what change the metaverse (a network of interactive virtual spaces being heralded by some as social technology’s latest evolutionary frontier) might bring to the world of audio production, there’s already a definite buzz around extended‑reality technology, and next‑generation VR headsets such as Meta’s new Quest Pro and Sony’s anticipated PS VR2 have piqued the curiosity of visual content makers and consumers alike. Almost inevitably, this has brought talk of conventional VR and 360° video formats back to the table in meetings between ad agencies and production houses, and for those who make a living recording dialogue there are already new VR‑related opportunities for work out there.

In this article, with the help of several industry colleagues, I’ll explore some of the technical and artistic differences when it comes to recording and producing dialogue and narration for immersive visual media. I’m based in Brazil, which boasts a burgeoning community of immersive‑content developers. The Brazilian Extended‑Reality Association (XRBR) was founded in 2019, and in the same year a São Paulo‑based company called ARVORE Immersive Experiences took home the top VR prize from the 76th Venice International Film Festival, with their short The Line. In 2020, this experience won a Primetime Emmy for Outstanding Innovation in Interactive Media. I worked as a voiceover consultant on The Line and my wife Simone Kliass, who’s a co‑founder and the current vice‑president of XRBR, recorded the narration in Portuguese.

The actors in the 360° film Eu Sou Você (2022) wore Sennheiser EW 100 G3 wireless transmitters and lavalier mics containing Sennheiser ME 102 omnidirectional capsules.Photo: Eu Sou Você

Ready, Headset, Go!

VR and 360° video are both immersive media best viewed using a headset, but there are significant differences between the two. VR is a computer‑generated digital space, in which you can actively engage with your virtual surroundings and the objects contained therein, whereas 360° video is a passively viewed spherical video image, shot by an omnidirectional camera. VR requires hardware connected to a headset with lenses, high‑resolution screens, built‑in speakers and/or a way of connecting to headphones, while a 360° video can be viewed either on a headset or a screen; really, you should only refer to a 360° video as ‘VR’ when it’s experienced using a VR or mobile‑phone headset.

The new Meta Quest Pro headset, pictured here with optional dual‑driver earbuds that offer wide dynamic range and significant ambient noise reduction.Photo: Meta

For sound engineers, discussions of VR and 360° video tend to centre on how to achieve a hyper‑realistic binaural mix and, as with most mixes these days, the most important raw ingredient is high‑quality, reverb‑free, noise‑free source material. Ambisonic microphones like the Sennheiser Ambeo VR, Rode NT‑SF1 and Zoom VRH‑8 capsule mounted on an H8 recorder may strike you as a good place to start — surround visuals need surround sound, right? But, in practice, for anything other than capturing a rich 3D soundscape, these mics are rarely the first thing you’ll pull out of your toolkit. Omnidirectional lavalier mics, for instance, work much better for recording dialogue on location. Nothing beats a directional shotgun mic with a windscreen for capturing isolated environmental sounds or Foley tracks. And a conventional capacitor or dynamic mic is still all you’ll need for narration or automated dialogue replacement (ADR) sessions in post‑production.

In his article ‘Introduction To Immersive Audio’ (SOS January 2022), Editor in Chief Sam Inglis wrote about binaural mixes and described the pros and cons of channel‑based, scene‑based, and object‑based immersive audio formats. “Channel‑ and scene‑based formats contain fully mixed audio,” he stated, “whereas an object‑based format contains the major elements of a mix plus metadata explaining how that mix should be implemented in a given playback environment.” For VR, the object‑based format seems to work best. It allows engineers to package dialogue, sound effects and music as separate audio streams, containing metadata which helps the decoder to determine where each should sit in the spatial‑audio mix.

Head‑tracking technology makes possible some ‘audio magic’ in VR but it can also be problematic, since binaurally encoded output responds differently across devices and requires engineers to adopt a mixing/mastering workflow for each headset used in an immersive experience. Fortunately, mixing for 360° video is simpler: most of the producers my wife and I work with in Brazil prefer stereo bounces, since the viewing experience is passive and stereo ensures consistent results across multiple devices. More on this later.

The Line lets you interact with a virtual diorama inspired by 1940s São Paulo.Photo: ARVORE

Estúdio JLS

Immersive audio designer Toco Cerqueira at a Dolby Atmos 9.1.6 workstation with JBL C221 monitors, 308P MkII surrounds, and Avid S6 M10 controller. Estúdio JLS have a second Dolby Atmos mixing room under construction.Photo: Bruna MachadoTo gain a better understanding of how VR and 360° video mixes differ, especially in relation to narration and dialogue, I met up with several colleagues, as well as sound engineers Daniel Sasso and Toco Sequeira, who were responsible for the final mix of The Line, mentioned in the opening paragraphs. “When working on The Line,” Daniel explained, “we received the film’s original choro soundtrack pre‑mixed. Our job was to put the soundtrack into Unity [a real‑time development platform: https://unity.com], along with the narrations and sound effects. Normally, here at JLS we mix for TV and cinema, and our primary considerations are the sound systems and acoustics of the spaces where a mix will be heard. Since VR and 360° video are hermetic experiences, we...