An artificial intelligence has created a passable cover of a Pink Floyd song by analysing brain activity recorded while people listened to the original. The findings further our understanding of how we perceive sound and could eventually improve devices for people with speech difficulties.
Robert Knight at the University of California, Berkeley, and his colleagues studied recordings from electrodes that had been surgically implanted onto the surface of 29 people’s brains to treat epilepsy.
The participants’ brain activity was recorded while they listened to Another Brick in the Wall, Part 1 by Pink Floyd. By comparing the brain signals with the song, the researchers identified recordings from a subset of electrodes that were strongly linked to the pitch, melody, harmony and rhythm of the song.
They then trained an AI to learn links between brain activity and these musical components, excluding a 15-second segment of the song from the training data. The trained AI generated a prediction of the unseen song snippet based on the participants’ brain signals. The spectrogram – a visualisation of the audio waves – of the AI-generated clip was 43 per cent similar to the real song clip.
Here is the original song clip after some simple processing to enable a fair comparison with the AI-generated clip, which undergoes some degradation when converted from a spectrogram to audio:
And here is the clip generated by the AI:
The researchers identified an area of the brain within a region called the superior temporal gyrus that processed the rhythm of the guitar in the song. They also found that signals from the right hemisphere of the brain were more important for processing music than those from the left hemisphere, confirming results from previous studies.
By deepening our understanding of how the brain perceives music, the work could eventually help to improve devices that speak on behalf of people with speech difficulties, says Knight.
“For those with amyotrophic lateral sclerosis [a condition of the nervous system] or aphasia [a language condition], who struggle to speak, we’d like a device that really sounded like you are communicating with somebody in a human way,” he says. “Understanding how the brain represents the musical elements of speech, including tone and emotion, could make such devices sound less robotic.”
The invasive nature of the brain implants makes it unlikely that this procedure would be used for non-clinical applications, says Knight. However, other researchers have recently used AI to generate song clips from brain signals recorded using magnetic resonance imaging (MRI) scans.
If AIs can use brain signals to reconstruct music that people are imagining, not just listening to, this approach could even be used to compose music, says Ludovic Bellier at the University of California, Berkeley, a member of the study team.
As the technology progresses, AI-based recreations of songs using brain activity could raise questions around copyright infringement, depending on how similar the reconstruction is to the original music, says Jennifer Maisel at the law firm Rothwell Figg in Washington DC.
“The authorship question is really fascinating,” she says. “Would the person who records the brain activity be the author? Could the AI program itself be the author? The interesting thing is, the author may not be the person who’s listening to the song.”
Whether the person listening to the music owns the recreation could even depend on the brain regions involved, says Ceyhun Pehlivan at the law firm Linklaters in Madrid.
“Would it make any difference whether the sound originates from the non-creative part of the brain, such as the auditory cortex, instead of the frontal cortex that is responsible for creative thinking? It is likely that courts will need to assess such complex questions on a case-by-case basis,” he says.