Over the past year, we have experimented extensively with audio description on 360-degree content in the Immersive Accessibility (ImAc) Project. The aim is finding solutions for enhancing the viewing experience of someone with sight loss watching this content.
We have broadly used the following criteria to gather feedback on the quality of experience:
As part of our testing, we used and considered different types of content and scenarios: animation – live action, fiction – documentary and on-site delivery, for example a museum walkthrough and it’s fair to say that the treatment of audio description needs to be adapted specifically to each format. So, like any other creative production, there is no one right answer. Here are some of the treatments we tried.
If you haven’t come across a 360-degree video before then imagine yourself standing at the centre of a sphere, with a screen all around you. You cannot move around within this sphere but you can twist, turn, look-up, look down or any other movement you’d like to try while you’re fixed to the centre of the sphere. This is quite unlike Virtual Reality (VR) where you have complete control over the environment. With VR, you can move around in the sphere, pick up objects, open doors and interact with objects as the story goes on. Remember, you will need to wear VR glasses regardless of whether you’re watching VR or content in 360-degree. You can read more about 360-video on this blog by video sharing platform Vimeo.
We explored the presentation styles of two elements:
A 360-degree display supports multiple assets such as characters, props, space and design etc that benefit from additional description. Nonetheless, the delivery of content is still linear and the time available for adding audio description is limited. So, the question is how to make the audio description effective enough that the viewer is immersed in the content which is one of the aims of the 360-degree format.
This is where the audio description is scripted in first person and the main character of the story becomes the describer. The style of writing and delivery is also similar to how the character sounds and speaks.
“I reversed away from him and he watched me go. I learned to drive with my knees while I played my guitar. There were four of us in the car now, Nat on guitar, Chuck on melodica, Big Sam on dashboard drums.”
This is where the describer is sitting next to the viewer or standing over their shoulder describing the scene the scene. Style of writing and delivery is casual, informal and friendly.
“She reverses the car. He watches her go. Later she steers with her knees, playing a guitar. There are three friends with her – a girl on guitar, a boy on melodica, while a bulky lad drums on the dashboard.”
This is the standard audio description style where describer objectively sets out what is happening in the scene, the characters etc.
“She puts the car into reverse and pulls away from him. He watches her go. Later she steers the car with her knees as she plays her guitar. Three friends are with her, playing guitar, melodica and one drumming on the dashboard.”
One of the trickiest aspects of describing 360-degree content is predicting what the viewer will do next – for example, look left where one can see the snow-capped peaks of the Swiss Alps or right, towards a meadow filled with daisies.
So, what happens in these situations where the viewer has complete control over what they see and when they see it?
We considered further controls which allowed people to activate descriptions when they changed direction.
In this mode, there are three audio tracks:
The original soundtrack remains untouched.
The audio description track describes the five key elements of the scene – who, what, why, where and when in order of priority for understanding the storyline as in standard audio description.
The additional audio track or the extended description is used to set the scene and describe elements such as props, characters and their costumes, lighting etc or event background knowledge which would help people with sight loss visualise the scene. The rationale here being greater familiarity with the scene is linked to greater immersion.
This track can be triggered by the viewer when they hear a "beep" that is used to signal the presence of this additional track.
In addition to the above, an audio introduction could be useful. Viewer listens to the audio introduction before watching the content which sets the scene. It refers to details that the describer cannot include in the content because of the lack time. These are quite common in the theatre world and are used to introduce to the viewers to the characters, their physical description, their costumes, the set design etc.
During our testing, we’ve discussed various factors that contribute to an immersive experience including how many voices would be considered too many in an immersive environment, use of language, intonation, delivery style and the importance of directionality and placement of the audio description in a 360-degree environment. However, it does seem that the 360-degree content opens up a whole host of alternatives on how audio description can be presented to the viewer and often a customised approach in which a combination of the scripting style and sound design would be needed to achieve the ideal experience.
Also, something to consider is the use of HMDs for people with significant sight loss who are unlikely to engage with content visually. Are they necessary? Perhaps, another device for head tracking may be more appropriate but further work is needed to investigate the viability of these alternatives.
Improving our understanding of how any form of access features i.e., audio description could be adapted for 360-degree and VR requires working with content and different formats, whether it’s sports, music, documentaries, short form fiction. So, we request content producers to please help us out and share some samples with us.
Please drop us a line on [email protected] if you’d like to discuss this further.
ImAc Project was set-up to explore the accessibility requirements of people with sight and/or hearing loss while consuming media in 360 and design the tools that will allow these users to independently access and personalise the 360-degree experience for greater enjoyment and immersion.
The ImAc Player which is one of the key deliverables offers users traditional access services such as subtitles, sign language, audio description and audio subtitles in different presentation modes that have been designed especially for the 360-format. Visit the ImAc Project website for more details.
Read more about how sound design affects the experience of 360 for viewers with sight loss.