Posted on

Nature’s conductor

Australian-born musician, composer and Fulbright Scholar Sam Nester has made a career for himself collaborating with scientists to turn natural patterns into sound.

He spoke with us about that work, which includes a residency with a U.S. National Park in Hawaii making music from earthquakes; using lie-detector-style technology to “listen” to plants; and, more recently, two projects as an artist-in-residence with New York University Abu Dhabi creating sound art from DNA and the Arabian Gulf’s coral reefs.

Nester and KUST Review started out talking about a project with a big purse and creative freedom. “That’s the dream of every artist,” Nester says.

Q:
Do you get that often?

Sometimes. I get that more and more now. Sometimes it’s, “Here’s some money, we want you to do something.”

And there’s other times where it’s like, “We want this thing and let’s work on it together,” so there’s already a basis for something.

And sometimes, like in Abu Dhabi, they say, “Here is what we do, what we research, what we study, the things that interest us, and you come in and do whatever it is you want to do. Hopefully some of that will be together.”

Q:
How did your work with NYUAD start? Were you talking to faculty to find out what they were doing or did you just come in with the concept and then figure out how to make it happen?

The genome thing (“Transposon”) is one of two big projects that we worked on. The other is to do with coral reefs in the Arabian Gulf. Right now I have a hydrophone – an underwater device with microphones – in the Arabian Gulf recording the soundscapes of the coral reefs. John Burt, a very well-known marine biologist, and I have started what is essentially the first soundscape study in the region.

IMAGE: Abjad Design

We record the sound of those coral reefs and how the acoustics of them change depending on the time of the year and the heat stress. Burt can also add that as a data point in research and look at the study of how the sound of those reefs determine health for the reef itself and everything that lives on or around the reef.

The genome thing (“Transposon”) is one of two big projects that we worked on. The other is to do with coral reefs in the Arabian Gulf. Right now I have a hydrophone – an underwater device with microphones – in the Arabian Gulf recording the soundscapes of the coral reefs. John Burt, a very well-known marine biologist, and I have started what is essentially the first soundscape study in the region.

We record the sound of those coral reefs and how the acoustics of them change depending on the time of the year and the heat stress.

In addition to that, as a sound artist and a musician, I’m listening to those things. And we’re collecting plastic from the Gulf and working with engineers at NYU to reconstitute it into discs. We’re basically making the equivalent of vinyl discs out of ocean plastic and I’m cutting into them the sounds of these dying coral reefs, essentially the last sounds of these coral reefs encoded into the material that’s choking them.

Q:
How do you see an exhibit like that working?

There will be publications and hopefully these records will have an installation coming out.

But for both of these projects, the genome one and the coral reefs project, NYU hosted their first artist-in-residence. That happened to be me. It was new for them to have an artist in residence in the sciences. I presented my work and basically went around to all the labs meeting scientists. The general project is with Stéphane Boissinot, the director of the Center of Genomics. His interest is in transposable elements.

What I wanted to do is find a way of turning the genome into sound. Boissinot’s focus is on the elements in genomics, repetitive things that also change. We started looking at that and playing around with the ideas of converting genomic information into musical parameters and then building with biomathematicians a code that would allow us to feed in any genetic information. It will churn out a MIDI file. That could be a piece of music and also a light installation.

Q:
So it’s like turning a genome into sheet music?

Yeah! Yeah. We can turn it into sheet music right now and I can go perform it. But I was also looking at turning it into sound and doing it in really scientific ways. I’m using digital software because we can have things that are played exactly for the length as they exist in the genome.

It’s also a way of saving data in sound, which we can then turn back into data. And it’s also a way for scientists to be able to look at their own work in different ways. We can start to hear patterns in things and of course there’s this idea of the art-science collaboration – looking at things that are being researched and ways artists think about things. Hopefully we come up with something that’s interesting for both disciplines.

IMAGE: Abjad Designs
Q:
You’ve done other work where you’re converting natural patterns into sound. With “Arcadia,” the interactive sound and light installation based on the biorhythms of plants, you have different site-specific installations. Is there a difference in the sound that you’re getting from plants from different ecosystems?

It is but maybe not the way we think about it. With “Arcadia” what it’s doing is registering changes in conductivity between two points in a plant. Essentially it’s a lie detector. Those changes from one plant to another plant or ecosystem plant to ecosystem plant are different because of the frequency at which those conductivity changes occur, how strong they are and how conductive on the scale it is. That’s to do with a whole bunch of things. One of them is the change of water in an individual system. It changes with patterns of day, night, time of the year or whether it’s going through photosynthesis. Those have a lot to do with health.

I was just in a desert in China in Inner Mongolia doing something with “Arcadia.” If the plant is healthy, the technology works relatively consistently. It does change from place to place but it’s not like one plant sounds like a chainsaw and another sounds like a piano.

That’s more my decision. I get to invent an instrument that the plant will “play,” but I don’t invent sheet music for it.

That was one of the big ways in which I can connect to scientists. One of the things I’ve done in the last eight years or so is collaborate with research scientists.

“Arcadia” was in no way a science project. I just wanted to make something that would make us think a little bit more about the rate at which things happen in plants. There are a whole bunch of biological processes in plants. How can I convert that into something that will give wonder or awe? There’s such an active part (of plant life) that we don’t see.

Years ago I had an installation at George Mason University of Virginia It was designed for a year in a greenhouse where scientists were researching medicinal and native plants. The scientists asked me if we can save the data. So I showed them how the information came out as data, how it came out as Musical Instrument Digital Interface (MIDI). It’s basically what John Burt is looking at with coral reefs, even though that’s acoustic recordings, the actual sounds of fish, shrimp, crabs and whatever lives on the reefs.

We’re looking at how those acoustic patterns change depending on time of day, night or season. The scientists at George Mason asked to see, to observe how that conductivity changes with other parameters they were looking at.

In the end they accepted it and decided maybe they could learn something from art. Which never happens. From there more and more things have happened like working with the U.S. National Parks and scientists with the U.S. Geological Survey to turn earthquake data into sound.

Q:
What does earthquake data sound like?

It’s really beautiful and interesting.

I was the artist-in-residence for Volcanoes National Park on the Big Island in Hawaii. I was collecting earthquake data, the same thing we’re doing with the genome project.

I say it’s the same thing, but the projects are obviously very different. The idea is the same – taking the data itself and converting three-and-a-half weeks of earthquakes into sound, into a playable score.

Earthquakes are happening all the time, not just the ones you feel. It was an idea that came from being there. I felt an earthquake, looked it up and realized there were tens of them before and hundreds happening after in the same place.

Of course, we don’t feel them day to day, but it’s fascinating because all of them are contributing to the shape and size of Hawaii and also the placement of our continents.

The project “Rifts” was amplifying something we didn’t hear into a way that you could maybe understand that there is so much hidden natural rhythm.

That’s a lot of what I look at. I amplify the hidden rhythms in nature.

IMAGE: Abjad Design

So I created this piece and when you hear the instruments that are playing and how loud they’re playing and how long they’re playing, each relating specifically to an earthquake and to the positioning we have in the data.

Q:
Is there a rhythm in nature you really want to explore next?

Yeah, there are a couple of things I’m interested in. I have a commission for a new piece for live performance in Houston, Texas, at a cistern. There’s a 15-million-gallon cistern (underground water storage tank). Other than the live performance, I’m looking at creating a light installation controlled by parameters of water data. Most of it is what I’m looking at with Antarctic ice melts.

Other natural rhythms I’m interested in are moonquakes, which I only learned about about a year ago.

I want to see if I can convert the quaking of the moon into a piece of music.

Q:
Say you’re at one of your installations. What is the best feedback you can overhear?

The feedback I like is not necessarily “I like it” or “I dislike it,” but does it make people ask more questions.

“Arcadia” is the same thing: “What is happening?” or “I wonder what my houseplant sounds like?” That’s the thing that’s amazing.

And now with the genome project it’s “What does my genetic code sound like?”

The best feedback is the audience thinking about their fears or excitement for the future. But also the way they connect those questions and thoughts that are already bouncing around. They’re all connected, of course, but there’s no narrative.

The best part is when they ask questions that are not necessarily about the art anymore.

The best feedback is when they start to take that and ask questions about other parts of life or existence. It tells me I’m on the right path.

More like this: Making science feel like stories

Posted on

Hey, grownups: Turn off the TV

Kids are often cautioned against watching too much TV, but maybe older adults should be too.

A recent study in Alzheimer’s and Dementia: Journal of the Alzheimer’s Association finds that people who reported watching TV “very often” in midlife later exhibited reduced volume in some areas of the brain. They also showed damage in the brain’s white matter associated with aging, stroke risk, cognitive decline and dementia.

“For years we’ve focused on how much people sit. Our findings suggest we should also pay attention to what they’re doing while they’re sitting,” says David Raichlen, professor of biological sciences and anthropology at the USC Dornsife College of Letters, Arts and Sciences and a senior author of the study.

The study also looked at other sedentary activities, but they didn’t have the same associations.

So it seems sitting isn’t the problem, but what you do while sitting.

More like this: Spotting early signs of Alzheimer’s

Posted on

The problem with AI and drug
discovery

AI has long been touted as the future of drug discovery. Just plug in the AI and pump out the miracle cures. Simple, right?

Not exactly and not just yet, according to researchers led by Khalifa University’s Andreas Bender.

The team took a step back to evaluate the real-world factors that complicate the process of AI-assisted drug discovery.

In their paper published in Nature Reviews Drug Discovery, Bender and his colleagues argue that AI in drug discovery needs to address some challenges better, including:

  • Researchers need to consider better how the AI will eventually be used in clinical settings when they are developing it.
  • AI needs biological and medical data that is predictive for the question the model aims to answer.
  • Researchers need to clearly define the problem the AI is supposed to solve, which then can lead to models that are properly equipped to address real-world situations.

“When I worked at AstraZeneca, people always talked about the right drug, in the right patient, dosed in the right way,” Bender tells KUSTReview.com. “We can, and need to, translate this thinking also to computational models and AI in drug discovery – which then means the right data, in the right model, validated and used in the right way.”

More like this: AI viruses join the fight against bacteria

Posted on

SPEAKING WITHOUT WORDS

Billy is a non-verbal 3-year-old with autism spectrum disorder (ASD). When Mom and Dad try to brush his teeth, he bites them. When they try to get him dressed, he kicks and punches. Billy is obviously unhappy, but what is he trying to say?

This scenario is all too familiar for many families learning to communicate with non-verbal members, but there is help: researchers like Ralf Schlosser.

Schlosser, assistant dean of research in the School of Clinical and Rehabilitation Sciences at Northeastern University, has spent his career researching augmentative and alternative communication (AAC) for those with severe speech, language and literacy difficulties. He specializes in helping children on the autism spectrum.

He believes families play an imperative role in developing communications technology.

“Sometimes families may have an advantage over clinicians or teachers in that they are likely very familiar with their child’s idiosyncratic communicative behaviors such as gestures or vocalizations and have the ability to ‘read’ these and respond appropriately.

As a result, it may sometimes feel unnecessary to use an AAC system,” he says.

IMAGE: Shutterstock

But families aren’t always together, and their methods may not work outside the home. These coping methods might, in fact, even hinder the child’s ability to communicate beyond the family circle. This is where technology can help.

Family involvement

Oliver Wendt of Purdue University emphasizes that family and user feedback are critical in AAC development. “You can no longer develop AAC technology without it,” he says.

AAC development involves co-design with direct input from users and stakeholders, usability input from families and therapists and assessments of tech used during interventions. Comments about an app’s usability, like “the interface needs to be simpler” or “it was a lot to navigate,” are crucial. “You need this type of feedback to fine-tune the app and ensure uptake within the community,” Wendt says.

From a design perspective, technology for verbally challenged individuals isn’t a one-size-fits-all solution because each person’s challenges are different.

“Technology opens the doors to interact with other individuals. Use of speech output along with interaction efficiencies is going a long way to make communication possible in schools, at work and in the community,” Schlosser tells KUST Review.

“You can no longer develop AAC technology without it (family involvement).”

– Oliver Wendt and team, 2022 paper

Another vital tool, Schlosser says: the “written gloss.” In a symbol-based AAC app or device, there are graphic symbols like an image representing the concept “drink” or “hungry” next to a corresponding word or phrase. This helps clarify the symbol’s meaning, aids in language learning and helps communication partners understand the user’s message.

He says speech-generating devices and tablets with AAC-specific apps allow individuals to talk using speech, digitized and/or synthetic. Tech also offers efficiency in word retrieval, whether singular or combined into a sentence.

One such tool is GoTalkNow.

GoTalkNow is a comprehensive, iPad-compatible AAC app based on the Project CORE research from the University of North Carolina’s Center for Literacy and Disability Studies. It includes core vocabulary — a collection of most of the words that make up typical daily communication that the user can customize by creating and adding pages. For those with literacy challenges, the app also includes symbols, the option to use images from the users’ cameras and a built-in internet-image search.

“Record video modeling sequences to pair instruction with communication. Play music to heighten interest. Choose text-to-speech or recorded audio for message content. Jump to any communication page, even from a different communication book. Add auditory cues to allow both auditory and visual scanning. Select automatic or step scanning, adjust scanning speed, and group buttons together by row or column for more efficient scanning,” the website advertises.

Skills required

Schlosser says many AAC tools require additional skill sets.

Users must depend on traditional writing and spelling or understand the graphic symbols used to represent concepts. They also need to understand the tool’s functionality and learn to use the program during interactions.

Schlosser says there are obstacles for end users. These include the facts that applications don’t always meet all of an individual’s communication needs and vocabulary selection can further overburden already busy support professionals. “Vocabulary selection in AAC for AAC applications and speech-generating devices is generally a time-consuming process, and school-based speech-language pathologists have little time given their caseloads,” Schlosser says.

But it appears that AI and machine learning can help alleviate some of these issues.

“By embracing AI-technologies such as QuickPic AAC, SLPs can leverage its capabilities to alleviate the time demands on creation of personalized materials.”

– Study by Ralf Schlosser and team, International Journal of Environmental Research and Public Health (MDPI)

A recent paper by Schlosser, Mauricio de Fontana and Christina Yu, the developers of the QuickPic AAC app that uses AI image analysis to lay out vocabulary, explores two algorithms (ChatGPT-3.5 and NLP-AAC) and found that ChatGPT-3.5 provided more relevant and accurate vocabulary.

The app analyzes an image of an activity and produces relevant words represented by picture communication symbols and organizes them in left to right categories in this order: subject, verb, attribute, object.

“This allows creating the displays on the fly or just in time, a process that was previously unimaginable. Just-in-time supports have various cognitive advantages for the learner and allow instructors to capitalize on teachable moments. Within minutes, the clinician can then customize the display as needed,” Schlosser says.

A future for wearables

Schlosser’s research has recently crossed 10,000 citations (uncommon within his field). It reflects long-term influence across research, and he is excited about what AI will bring to the table in the future. The research has only scratched the surface of its impact on AAC.

He sees many opportunities for wearable technologies and has done some small studies in this area, including tools like smart watches.

Researchers were concerned the watches’ small size might be an issue, but the individuals in the study all responded well, he says.

Another study included remote visual interventions like photos, video or text, in real-time, to assist a learner.

Schlosser also says measuring physiological responses like heart rates could also help manage anxiety and support communication.

Wendt cautions that smart watches are still too expensive for many users, but he is enthusiastic about other wearables. People on the autism spectrum with verbal challenges also frequently experience sensory impairments, so the possibilities are endless as the wearables field moves away from watches and toward tech that could be directly woven into clothing, he says.

Help from AI

And as with most industries around the globe, AI will further individual targeted tech design for AAC.

“Diagnostics and prognosis will be more effective through AI. AI can compare the communication profile of a user with data from a plethora of other users with similar conditions. This will help to determine how severely someone is involved and what the prognosis/outlook is,” Wendt says.

“Future AAC applications may be able to predict what vocabulary is needed by a user and then create that content right on the spot.”

Innovations in AR and VR in tandem with AI will only further tech development in this space, as devices become smaller and more user-friendly, they become more cost-effective. “This will allow users to learn and interact in completely new environments that allow new opportunities to learn and grow,” he tells KUST Review.

KU Autism study

With the increasing diagnoses of autism spectrum disorder around the world comes a desperation to understand its origins. While experts believe causation is multi-faceted, researchers at Khalifa University led by Hamdan Hamdan and collaborators including Fakih-IVF Fertility Center and Baylor College of Medicine are studying the condition’s genetic components.

The team has identified seven novel mutations in autistic children within the UAE population using next-generation sequencing.

“The investigation aims to uncover the roles of these novel genes as well as the complex mechanistic pathways involved in ASD using advanced techniques, such as CRISPR-Cas9 gene editing and BioID,” Hamdan says.

The CRISPR-Cas9 gene editing and BioID protein mapping individually have their challenges, but in combination could help identify how mutations contribute to autism and brain development.

The team hopes that identifying mutations specific to the Emirati population may contribute to personalized, targeted therapies and offer important findings for global ASD research highlighting the importance of population-specific studies in understanding the full spectrum of genetic variations associated with autism.

More like this: Autism checks