An opera built around expressive control

In Paola Prestini and Brenda Shaughnessy’s opera Sensorium Ex, artificial intelligence is not used to substitute for a performer. It is used as part of an instrument through which a performer can shape a voice in real time. That distinction is central to the production, which premiered in Omaha in May 2025 and places disability, communication and technological power at the centre of its story.

Jakob Jordan, an actor with autism and apraxia who is nonspeaking, performed the role of Kitsune on two nights of the premiere run. For the work, a research team associated with New York University’s Ability Project developed a system intended to render text in a synthetic voice recognisably connected to Jordan’s own vocal characteristics. It also enabled live control over aspects such as timing, pitch and pace.

The result matters because conventional speech-generating devices can make words audible but may leave limited room for a speaker to control delivery once a phrase is submitted. In everyday human conversation, however, meaning is carried not just in vocabulary but in pauses, speed, emphasis, tone and timing. An opera is an unusually demanding environment in which to address that problem: it requires a performer to communicate character, rhythm and emotion before an audience, in coordination with other performers and music.

From assistive output to performance instrument

Sensorium Ex developed through a long collaboration among artists, disability advocates, researchers and performers. The opera’s fictional plot concerns Kitsune and his mother in a world shaped by corporate interest in artificial intelligence, but its production process was designed around real access needs. The cast includes disabled performers, and the project assembled an advisory group to bring lived experience into artistic and technical decisions.

NYU’s account of the work describes an early research phase in which AAC users discussed their experience of existing technology. A recurring concern was that device voices often felt generic or emotionally flat. This is not a superficial complaint. A voice is intertwined with identity and social participation; an inability to signal humour, irritation, enthusiasm or uncertainty can alter how a person is understood by others.

The technical system combined text-to-speech synthesis with a neural vocoder, a machine-learning component that helps generate speech audio. Researchers recorded the natural vocalisations of Jordan and fellow Kitsune performer Kader Zioueche, then used the recordings to infer vocal features that could be applied to new text. The goal was not to claim that a model had recovered a hidden, literal speaking voice. Rather, it was to create a synthetic output that better reflected each performer’s vocal identity than a standard default voice could.

Control was as consequential as vocal likeness. The team used distance-sensing hardware that could translate simple hand movements into live adjustments during performance. That made the system more akin to a playable instrument than a button that reads completed text aloud. For Jordan, this meant participating in the opera as a performer shaping phrasing, not merely as an actor accompanied by a fixed audio track.

Why the distinction matters for AAC

Augmentative and alternative communication encompasses methods ranging from gestures and communication boards to high-tech speech-generating devices. The need is substantial: the US National Institutes of Health estimates that more than five million people in the country cannot rely on spoken language as their primary means of communication some or all of the time.

Personalised speech technology is already an established area within AAC. Voice banking, for example, uses recordings to create a custom synthetic voice that can speak new phrases; message banking preserves recordings of specific words or expressions. These approaches can preserve familiarity and personal connection, especially for people who expect to lose speech through degenerative disease.

The Sensorium project addresses a related but distinct challenge. It is working with performers whose recorded speech may be atypical, sparse or difficult to use in conventional voice-banking workflows. Its design also focuses on expression after text has been entered. That is particularly relevant to people for whom speech technology can otherwise feel like a channel for transactions — requesting, answering, informing — rather than a medium for personality, relationships, performance and disagreement.

This does not mean a theatrical prototype is ready to replace established AAC tools. Jordan continued to use a more conventional text-based communication device in daily life, and the project team has said it is still developing the synthesiser for broader testing and for use on the tablets commonly used in AAC. Reliability, speed of text entry, individual access methods, cost, privacy and clinical support will all determine whether a system can become useful outside a stage setting.

Design choices are also ethical choices

AI voice systems raise legitimate concerns about consent, surveillance, ownership and impersonation. A synthetic voice can be intimate data: it may reveal health-related information and can be misused if recordings, models or outputs are not protected. The question of who trains, controls, stores and can reproduce a person’s voice is therefore not ancillary to accessibility design.

The Sensorium team has stressed local processing rather than a cloud-based service for its prototype, an approach intended to reduce third-party access to vocal data. It has also described plans to make elements of the technology available on an open-source basis. Those choices could improve transparency and experimentation, but open availability alone does not resolve the challenges of secure implementation, informed consent or long-term support.

The project’s more important principle may be participatory design. Technology for disabled people is often assessed through whether it approximates a supposedly normal function. Sensorium Ex instead frames the task as expanding agency: enabling a person to decide how they sound, when they pause and how they perform. The performers and disability community were contributors to the project’s direction, not simply subjects from whom data were extracted.

That approach should also temper the rhetoric around AI. The system is not evidence that machine learning has solved communication disability, nor that it can determine a person’s intended speech from a recording. It is a carefully developed creative tool that relies on a performer’s typed words, learned technique, collaborator support and a specific production context. Its achievement is to make those elements more expressive and visible.

Art as a testbed, not a shortcut

Opera may seem an unlikely proving ground for assistive technology, but it exposes qualities that are easy to overlook in a laboratory or a product demonstration. On stage, communication must be personal, dynamic and consequential. If an interface cannot accommodate timing, mood or artistic choice, its limits become immediately clear.

That does not make theatre a substitute for clinical research or accessible consumer design. It does, however, offer a valuable testbed. Sensorium Ex demonstrates that accessibility can shape an artwork’s form rather than being added after the main creative decisions are complete. The opera’s technology is part of its aesthetic language, while the experience of its performers challenges the narrow assumption that voice is only meaningful when it emerges through typical speech.

The broader lesson is pragmatic. AI can be most constructive when it increases a person’s control over expression instead of automating their presence away. Whether the Sensorium approach can become an everyday communication option remains an open technical and social question. Its stage success nevertheless provides a clear standard for future work: people who use communication technology should have a meaningful role in determining not only what it says, but how it lets them be heard.

Sources