MIT Research News' Journal

Using math to blend musical notes seamlessly

In music, “portamento” is a term that’s been used for hundreds of years, referring to the effect of gliding a note at one pitch into a note of a lower or higher pitch. But only instruments that can continuously vary in pitch — such as the human voice, string instruments, and trombones — can pull off the effect.

Now an MIT student has invented a novel algorithm that produces a portamento effect between any two audio signals in real-time. In experiments, the algorithm seamlessly merged various audio clips, such as a piano note gliding into a human voice, and one song blending into another. His paper describing the algorithm won the “best student paper” award at the recent International Conference on Digital Audio Effects.

The algorithm relies on “optimal transport,” a geometry-based framework that determines the most efficient ways to move objects — or data points — between multiple origin and destination configurations. Formulated in the 1700s, the framework has been applied to supply chains, fluid dynamics, image alignment, 3-D modeling, computer graphics, and more.

In work that originated in a class project, Trevor Henderson, now a graduate student in computer science, applied optimal transport to interpolating audio signals — or blending one signal into another. The algorithm first breaks the audio signals into brief segments. Then, it finds the optimal way to move the pitches in each segment to pitches in the other signal, to produce the smooth glide of the portamento effect. The algorithm also includes specialized techniques to maintain the fidelity of the audio signal as it transitions.

“Optimal transport is used here to determine how to map pitches in one sound to the pitches in the other,” says Henderson, a classically trained organist who performs electronic music and has been a DJ on WMBR 88.1, MIT’s radio station. “If it’s transforming one chord into a chord with a different harmony, or with more notes, for instance, the notes will split from the first chord and find a position to seamlessly glide to in the other chord.”

According to Henderson, this is one of the first techniques to apply optimal transport to transforming audio signals. He has already used the algorithm to build equipment that seamlessly transitions between songs on his radio show. DJs could also use the equipment to transition between tracks during live performances. Other musicians might use it to blend instruments and voice on stage or in the studio.

Henderson’s co-author on the paper is Justin Solomon, an X-Consortium Career Development Assistant Professor in the Department of Electrical Engineering and Computer Science. Solomon — who also plays cello and piano — leads the Geometric Data Processing Group in the Computer Science and Artificial Intelligence Laboratory (CSAIL) and is a member of the Center for Computational Engineering.

Henderson took Solomon’s class, 6.838 (Shape Analysis), which tasks students with applying geometric tools like optimal transport to real-world applications. Student projects usually focus on 3-D shapes from virtual reality or computer graphics. So Henderson’s project came as a surprise to Solomon. “Trevor saw an abstract connection between geometry and moving frequencies around in audio signals to create a portamento effect,” Solomon says. “He was in and out of my office all semester with DJ equipment. It wasn’t what I expected to see, but it was pretty entertaining.”

For Henderson, it wasn’t too much of a stretch. “When I see a new idea, I ask, ‘Is this applicable to music?’” he says. “So, when we talked about optimal transport, I wondered what would happen if I connected it to audio spectra.”

A good way to think of optimal transport, Henderson says, is finding “a lazy way to build a sand castle.” In that analogy, the framework is used to calculate the way to move each grain of sand from its position in a shapeless pile into a corresponding position in a sand castle, using as little work as possible. In computer graphics, for instance, optimal transport can be used to transform or morph shapes by finding the optimal movement from each point on one shape into the other.

Applying this theory to audio clips involves some additional ideas from signal processing. Musical instruments produce sound through vibrations of components, depending on the instrument. Violins use strings, brass instruments use air inside hollow bodies, and humans use vocal cords. These vibrations can be captured as audio signals, where the frequency and amplitude (peak height) represent different pitches.

Conventionally, the transition between two audio signals is done with a fade, where one signal is reduced in volume while the other rises. Henderson’s algorithm, on the other hand, smoothly slides frequency segments from one clip into another, with no fading of volume.

To do so, the algorithm splits any two audio clips into windows of about 50 milliseconds. Then, it runs a Fourier transform, which turns each window into its frequency components. The frequency components within a window are lumped together into individual synthesized “notes.” Optimal transport then maps how the notes in one signal’s window will move to the notes in the other.

Then, an “interpolation parameter” takes over. That’s basically a value that determines where each note will be on the path from its starting pitch in one signal to its ending pitch in the other. Manually changing the parameter value will sweep the pitches between the two positions, producing the portamento effect. That single parameter can also be programmed into and controlled by, say, a crossfader, a slider component on a DJ’s mixing board that smoothly fades between songs. As the crossfader slides, the interpolation parameter changes to produce the effect.

Behind the scenes are two innovations that ensure a distortion-free signal. First, Henderson used a novel application of a signal-processing technique, called “frequency reassignment,” that lumps the frequency bins together to form single notes that can easily transition between signals. Second, he invented a way to synthesize new phases for each audio signal while stitching together the 50-millisecond windows, so neighboring windows don’t interfere with each other.

Next, Henderson wants to experiment with feeding the output of the effect back into its input. This, he thinks, could automatically create another classic music effect, “legato,” which is a smooth transition between distinct notes. Unlike a portamento — which plays all notes between a start and end note — a legato seamlessly transitions between two distinct notes, without capturing any notes in between.

Photovoltaic-powered sensors for the “internet of things”

By 2025, experts estimate the number of “internet of things” devices — including sensors that gather real-time data about infrastructure and the environment — could rise to 75 billion worldwide. As it stands, however, those sensors require batteries that must be replaced frequently, which can be problematic for long-term monitoring.

MIT researchers have designed photovoltaic-powered sensors that could potentially transmit data for years before they need to be replaced. To do so, they mounted thin-film perovskite cells — known for their potential low cost, flexibility, and relative ease of fabrication — as energy-harvesters on inexpensive radio-frequency identification (RFID) tags.

The cells could power the sensors in both bright sunlight and dimmer indoor conditions. Moreover, the team found the solar power actually gives the sensors a major power boost that enables greater data-transmission distances and the ability to integrate multiple sensors onto a single RFID tag.

“In the future, there could be billions of sensors all around us. With that scale, you’ll need a lot of batteries that you’ll have to recharge constantly. But what if you could self-power them using the ambient light? You could deploy them and forget them for months or years at a time,” says Sai Nithin Kantareddy, a PhD student in the MIT Auto-ID Laboratory. “This work is basically building enhanced RFID tags using energy harvesters for a range of applications.”

In a pair of papers published in the journals Advanced Functional Materials and IEEE Sensors, MIT Auto-ID Laboratory and MIT Photovoltaics Research Laboratory researchers describe using the sensors to continuously monitor indoor and outdoor temperatures over several days. The sensors transmitted data continuously at distances five times greater than traditional RFID tags — with no batteries required. Longer data-transmission ranges mean, among other things, that one reader can be used to collect data from multiple sensors simultaneously.

Depending on certain factors in their environment, such as moisture and heat, the sensors can be left inside or outside for months or, potentially, years at a time before they degrade enough to require replacement. That can be valuable for any application requiring long-term sensing, indoors and outdoors, including tracking cargo in supply chains, monitoring soil, and monitoring the energy used by equipment in buildings and homes.

Joining Kantareddy on the papers are: Department of Mechanical Engineering (MechE) postdoc Ian Mathews, researcher Shijing Sun, chemical engineering student Mariya Layurova, researcher Janak Thapa, researcher Ian Marius Peters, and Georgia Tech Professor Juan-Pablo Correa-Baena, who are all members of the Photovoltaics Research Laboratory; Rahul Bhattacharyya, a researcher in the AutoID Lab; Tonio Buonassisi, a professor in MechE; and Sanjay E. Sarma, the Fred Fort Flowers and Daniel Fort Flowers Professor of Mechanical Engineering.

Combining two low-cost technologies

In recent attempts to create self-powered sensors, other researchers have used solar cells as energy sources for internet of things (IoT) devices. But those are basically shrunken-down versions of traditional solar cells — not perovskite. The traditional cells can be efficient, long-lasting, and powerful under certain conditions “but are really infeasible for ubiquitous IoT sensors,” Kantareddy says.

Traditional solar cells, for instance, are bulky and expensive to manufacture, plus they are inflexible and cannot be made transparent, which can be useful for temperature-monitoring sensors placed on windows and car windshields. They’re also really only designed to efficiently harvest energy from powerful sunlight, not low indoor light.

Perovskite cells, on the other hand, can be printed using easy roll-to-roll manufacturing techniques for a few cents each; made thin, flexible, and transparent; and tuned to harvest energy from any kind of indoor and outdoor lighting.

The idea, then, was combining a low-cost power source with low-cost RFID tags, which are battery-free stickers used to monitor billions of products worldwide. The stickers are equipped with tiny, ultra-high-frequency antennas that each cost around three to five cents to make.

RFID tags rely on a communication technique called “backscatter,” that transmits data by reflecting modulated wireless signals off the tag and back to a reader. A wireless device called a reader — basically similar to a Wi-Fi router — pings the tag, which powers up and backscatters a unique signal containing information about the product it’s stuck to.

Traditionally, the tags harvest a little of the radio-frequency energy sent by the reader to power up a little chip inside that stores data, and uses the remaining energy to modulate the returning signal. But that amounts to only a few microwatts of power, which limits their communication range to less than a meter.

The researchers’ sensor consists of an RFID tag built on a plastic substrate. Directly connected to an integrated circuit on the tag is an array of perovskite solar cells. As with traditional systems, a reader sweeps the room, and each tag responds. But instead of using energy from the reader, it draws harvested energy from the perovskite cell to power up its circuit and send data by backscattering RF signals.

Efficiency at scale

The key innovations are in the customized cells. They’re fabricated in layers, with perovskite material sandwiched between an electrode, cathode, and special electron-transport layer materials. This achieved about 10 percent efficiency, which is fairly high for still-experimental perovskite cells. This layering structure also enabled the researchers to tune each cell for its optimal “bandgap,” which is an electron-moving property that dictates a cell’s performance in different lighting conditions. They then combined the cells into modules of four cells.

In the Advanced Functional Materials paper, the modules generated 4.3 volts of electricity under one sun illumination, which is a standard measurement for how much voltage solar cells produce under sunlight. That’s enough to power up a circuit — about 1.5 volts — and send data around 5 meters every few seconds. The modules had similar performances in indoor lighting. The IEEE Sensors paper primarily demonstrated wide‐bandgap perovskite cells for indoor applications that achieved between 18.5 percent and 21. 4 percent efficiencies under indoor fluorescent lighting, depending on how much voltage they generate. Essentially, about 45 minutes of any light source will power the sensors indoors and outdoors for about three hours.

The RFID circuit was prototyped to only monitor temperature. Next, the researchers aim to scale up and add more environmental-monitoring sensors to the mix, such as humidity, pressure, vibration, and pollution. Deployed at scale, the sensors could especially aid in long-term data-collection indoors to help build, say, algorithms that help make smart buildings more energy efficient.

“The perovskite materials we use have incredible potential as effective indoor-light harvesters. Our next step is to integrate these same technologies using printed electronics methods, potentially enabling extremely low-cost manufacturing of wireless sensors," Mathews says.