Accessibility within AR/VR: Jonathan Avila on the AT Banter podcast

By Jonathan Avila, CPWA

I recently had the pleasure of being a guest on AT Banter: An Assistive Technology Podcast for their EPISODE 140 – Accessibility within Virtual Reality and Augmented Reality. I hope you can take some time to listen to the podcast (my part begins at 21:15), but I thought it would also be good to share some of the information here on Level Access’s blog for those who prefer to a written post.

What is VR/AR and Mixed Reality?

Virtual Reality– A computer generated simulation or 3D image or environment. Seemingly real. Such as with helmet screen, sensors, etc. The user is immersed into digital environment and cut off from the real environment. (Example: Star Wars Jedi Challenge.)

Augmented Reality– imposes computer generated view on top of the real world providing composite view. The user is not cutoff from their own reality, but computer-generated objects are placed in their real world. (Example: Pokémon Go and apps that allow you to place furniture into your physical room.)

Mixed reality– fuses the augmented and virtual space together.

Is VR/AR a practical option for those who can’t see?

The initial response I get from people when discussing the topic is that VR/AR is very visual and seems impractical for people who are blind. However, there are many possibilities beyond what people perceive.

Let’s take, for example, a technology that can map indoor spaces and communicate distance to points of interest and obstacles. The VR system knows what objects are in the environment, who is in a room, and other details such as which direction they are facing. While a sighted user would be able to see all of these things, that isn’t the only way to communicate the information. Understanding and exploring the virtual environment can be done through sounds, audio cues, tactile indications, smell, taste, temperature, proprioception, balance, acceleration, and description of what is in the visual environment. There is a convergence of AI (machine learning) and VR/AR. I think we will see more AR and AI with services like Aira used by people who are blind or visually impaired to gain access to the visual world.

While VR/AR’s core relies on visual stimulus, that does not have to be the case. For example, Microsoft demonstrated games at the Annual CSUN conference that use 3D audio for navigation. The audio could allow someone with good audio perception to navigate a virtual environment and locate items through a maze. The practical implications of AR, VR, and mixed reality stretch from travel to video games, education to healthcare, and assistive technology and beyond.

Navigation & Wayfinding

One area that holds promise for people who are blind or visually impaired is navigation and wayfinding. As you may know, getting to an address may be straightforward but most navigation apps will not lead you to the correct door to enter or help you find something or someone inside the building. While services like Aira can help with this, I envision a world where humans are not assisting but machines are identifying barriers, doors, and other information and communicating it in an alternative augmentative way that works for the particular user’s abilities. For example, people with low vision may get markings on screen to show them which way to walk. A heads-up display may show sign information in large print much in the same way that AR currently translate signs for people who don’t read a certain language.

VR is important to users with disabilities as it can allow them to reach and experience locations that might otherwise be difficult to access. It also allows users to experience that situation ahead of time before actually experiencing it. For someone who is blind or visually impaired, they can learn or plan a route or explore an area in a safe virtual environment before actually going there. For example, there was a prototype of a VR setup with a white cane that allowed the user to simulate navigation and feedback was provided through vibration of the white cane to simulate obstacles as the person was traveling in the virtual environment. In addition, the user can experience details as they move through the virtual environment—such as the locations of shops and other details—except they are provided with audio cues rather than visual ones.

This type of VR would be invaluable to a blind or low vision individual who needs to learn a route that is not part of their everyday routine, for example:

  • Emergency route to get out of their office building in case of a fire
  • Alternative route to a familiar location when road work blocks the usual way
  • Planning to visit a new city for business or vacation time

In all of these scenarios, being able to do a practice run in virtual reality would enable the individual to maintain more of their independence.

VR in the Healthcare Space

In the healthcare space, VR/AR can help people experience things in a safe environment or practice things that might otherwise cause anxiety and stress. The simulated environment can help with pain management or to relieve stress. VR can help someone to stop thinking about their pain and focus their attention on an immersive and fun experience.

VR may allow people who are blind to ride a stationary bike or run on a treadmill yet experience the sounds, smells, and sensations of riding a bike along the coast, a crowded pier, or in some exotic location. (Another example of an accessible technology that would benefit more than just those with disabilities!)

VR and Gaming

I do accessibility consulting with video game companies in both gaming accessibility and accessible communication systems. Legally speaking, much of the early access to general AR/VR will be around communication aspects and access to interface to reach these communication aspects as required through the CVAA in the United States.

Access to AR/VR is Important not only for inclusion of people with disabilities but also because it can uniquely solve some of the challenges that people with disabilities encounter. In gaming, social interaction can be critical. Being able to team up with gamers from around the world and work together to defeat the big bad guy puts everyone on the same level. Your teammates interact with you through your avatar and you can choose exactly how to represent yourself. Gaming with social chat reduces the isolation that can come with having a disability and can allow anonymity of your disability.

Accessible controllers—and the work of organizations like AbleGamers—help more people to play, making gaming a great opportunity to improve the lives of people with disabilities and create a more inclusive society.

VR for Enriching Experiences

Similar benefits are available for education and access to experiences that might otherwise be difficult to experience.

  • Inaccessible Physical Locations:Riding a mule down to the bottom of the Grand Canyon isn’t possible for some people. Some historic buildings can be difficult or impossible to access in a wheelchair. Both are things one could experience in VR.
  • Look, Don’t Touch:Most museums do not allow patrons to touch the art. But with a virtual environment, priceless art could be recreated in a medium that is tactile.
  • Personalized Entertainment:An AR display for the deaf and hard of hearing could show sign language captions or text captions synched up with the character who is talking.People who are blind could access audio description that is automatically synchronized with a movie without a special headset anywhere they go – not just at a movie theater.

Keys to inclusion in VR/AR environments

When considering VR/AR environments, there are several keys to ensuring an inclusive experience.

Accessible System Controls

All aspects of the system need to be accessible, for example:

  • powering on/off,
  • accessing the sensors and physical hardware,
  • setup/configuration,
  • navigating the menus,
  • accessing services like communication, browsers, etc.

Interaction & Input Controls

Interaction and input can include:

  • Voice control and recognition
  • Motion and gesture recognition
  • Eye and head tracking
  • Keyboard interface, controller, or sensor
  • Mobile device as input for other devices (Think Microsoft’s Xbox controller on your iPhone)

While some of these modes pose challenges for certain users, offering a variety of modes of interaction `will allow a wider set of users to choose which mode of interaction works best for them. For example, when eye tracking couldn’t be used due to a visual impairment, a head tracking or physical controller could be used for those actions. Historically, many systems focused on motion control and movement through space – but there isn’t a limitation to use these methods. For example, in Pokemon GO, typically the gamer would move their phone around until a Pokemon appeared centered on the screen, at which point, they could try to catch it. An alternative method was provided where the Pokémon was on the screen regardless of where the camera was pointed and users who could not move around their phone could still catch them all.

Flexible Customization

I call this the “one size fits one” key to accessibility. The key to inclusion is multiple modes of interaction and output that can be chosen by the end user. Control of the interface needs to allow for multiple methods of input and personalization.

For Visual Disabilities

  • Audio cues
  • Change size and color text for better contrast
  • Colorblind options
  • Ability to zoom without getting physically closer
  • Audio placement as you move through an environment (ex: Yelp Periscope)

For Deaf & Hard of Hearing

  • Ambient noise indicator
  • Text indicators
  • Closed captioning
  • Give emotions and feeling, not just text

For Vestibular Disorders

  • Motion sickness prevention.
  • Slow down movement in space
  • Central placement place
  • Input support

For Cognitive Disabilities

  • Tutorial
  • Sandbox/playground mode
  • Level difficulty
  • Intuitive menus
  • Perspective
  • Customizations
  • Contextual guidance

VR/AR has an exciting future

As an assistive technology, VR and AR have big benefits. They could be used to help some people on the autism spectrum recognize emotions. When combined with AI, they could provide facial recognition for people who are blind or visually impaired. They can overlay tutorials or instructions for people with cognitive and learning disabilities or provide cues and steps to assist people with memory challenges.

As part of the team at Level Access, I have consulted with several organizations that are focused in this space and with software vendors who make software that includes AR components.If your organization is developing VR/AR products and would like help making them accessible, contact us to get started.

What are you excited to see happen in VR/AR? Let us know in the comments.