Phase 1 Report
Phase 1 Summary
The initiative transforms conventional screen readers into experiential soundscapes using spatial audio technology. The prototype employs "layered voices and spatial audio to help users discern the text's location and context within the browser," enhancing digital accessibility. The project explores how AI-generated voices and descriptive text can create personalized sound environments, particularly benefiting blind and visually impaired users.
Phase 1 Methodology
Co-Creation Approach
The project engaged six co-creators through structured sessions at Constant vzw (Brussels-based arts organization). Three screen reader users (Chris, Bruno, and Raphael) participated in experimental design exercises and technology demonstrations, focusing on understanding varied interactions with digital content and challenges accessing complex visual information.
Prototype Development
- Initial version: Built using A-Frame framework, hosted on Glitch
- Features: Keyboard navigation, audio triggers at spatial points
- Feedback findings: Users appreciated spatial layout but needed stronger auditory boundary indicators
- Refinement: Second prototype added enhanced audio boundaries, refined sound parameters, and adjusted distance modeling
Phase 1 Results
Key Findings
1. Spatial Awareness: "Traditional screen readers often 'flatten' web experiences by reducing content to linear lists," eliminating spatial context crucial for understanding complex information like maps or images.
2. Customization Importance: Users valued control over voice characteristics, sound localization, and movement within soundscapes. The team used ElevenLabs and ChatGPT for demonstration purposes.
3. Navigation Challenges: While exploration appealed to participants, they highlighted difficulties navigating without clear auditory cues and complexity from multiple layered voices.
4. Sound Method Development: Audio layers were designed to coexist without conflict, informed by Daniel R. Montello's spatial theories, aligning sounds with different environmental scales—immediate, vista, environmental, and geographical.
Interactive Demonstrations
Prototype 1: Arrow key navigation in full-screen Chrome with headphones recommended. Try Prototype 1
Prototype 2: Enhanced version with improved controls and audio cues. Try Prototype 2
Hieronymus Bosch, The Garden of Earthly Delights, oil on oak panels, Museo del Prado, Madrid
Alt-Text Generation Demo
Garden of Earthly Delights (Hieronymus Bosch painting) example featuring:
- Four audio files demonstrating customized descriptions
- Art curator version
- Child-friendly versions (neutral and upbeat tones)
- Soundtrack integration using Imaginary Soundscapes
Garden of Earthly Delights - Art Curator Perspective
Detailed art historical description tailored for an art curator
Garden of Earthly Delights - Child-Friendly Description
Simplified description suitable for children
Garden of Earthly Delights - Child-Friendly (Upbeat Tone)
Child-friendly description with enthusiastic, upbeat tone
Garden of Earthly Delights - Child-Friendly (Upbeat Tone with Soundscape)
Child-friendly description with upbeat tone and immersive soundscape background
Phase 2 Goals
Co-Creation Objectives
- Expanded sessions with diverse visual content (charts, infographics, complex materials)
- Open-source promotion encouraging developer, sound designer, and accessibility advocate participation
- Co-creation guidelines documentation for future inclusive design projects
Technical Objectives
- Sonification of images, maps, and web elements into spatial soundscapes
- Open-source naturalistic voice customization with real-time processing
- Customizable alt-text generation system
- Real-time responsiveness to user navigation inputs
- Platform expansion beginning with image/map services (Google Maps, OpenStreetMap)
- Comprehensive accessibility tutorials and documentation
Acknowledgments
- Funding organizations: Constant, The Processing Foundation, Stimuleringsfonds
- Mentor: Luis Morales-Navarro
- Co-creators: Bruno, Chris, Raphael, Joris