Giant Steps: an adversarial audio modem that survives noise cancellation

I built an audio modem that sends data by singing Giant Steps. Here is a minute and forty seconds of it: two full 140-byte packets, each with the head followed by a solo over its error-correction bytes.

Two full 140-byte packets, about 100 seconds total. The Lick opens each packet; the Giant Steps head begins at 1.44 seconds and the checksum/parity solo at 35.04 seconds. A 0.75-second gap separates the packets.

Download MP3

Give it past the opening phrase: the payload melody starts at 1.44 seconds, repeats the complete 16-bar head, and changes to the solo at 35.04 seconds. It sounds like a rice cooker trying to get into jazz school.

There is a reason for the voice. I wanted to move a few bytes over a voice call, where the audio pipeline is built to preserve people and remove everything else. A modem tone is a perfectly reasonable thing for a noise suppressor to delete. So I put the data in something closer to a human vowel.

The project is called sóng: Vietnamese for wave, close to sống, to live or survive, and an English word that became a design requirement. It is written in Rust. The vowels carry the bits. The pitch carries no payload data, which left a whole musical dimension available.

The default transmission now plays Giant Steps. The synchronization phrase is The Lick. The payload gets the head (the main melody), and the checksum and Reed–Solomon parity get a transcription of Coltrane’s solo until the packet runs out of bytes to protect.

The call is trying to remove your modem

JackRabbit was my attempt to get useful throughput out of an audio path. A voice call adds a different problem: the path makes decisions about which sounds deserve to reach the other end.

Gain control changes your amplitude. Band limiting removes frequencies. A lossy codec changes the spectrum. Voice activity detection can gate a signal it does not consider speech; a denoiser can suppress parts of it even while passing other audio. Surviving one of these does not imply surviving the whole chain.

Pure tones are convenient for a modem because they are easy to generate and distinguish. That does not make them convenient for a conferencing system trying to remove a persistent whine. Turning them up cannot restore a frequency that has been filtered out, and can just give the microphone something louder to clip.

I needed the information to survive inside the features the channel was already trying to preserve. Human voices have a harmonic structure, resonances that distinguish vowels, and gaps between syllables. The modem synthesizes all three.

This does not require a recording of a person, a text-to-speech engine, or words. It requires a useful imitation of the acoustics.

Your mouth is a filter

Your vocal cords provide a repeating excitation: a buzz with a fundamental frequency, F0, and harmonics at multiples of it. Your vocal tract shapes that buzz. Its resonances, called formants, emphasize different regions of the spectrum.

Change the pitch while singing “oo” and it remains “oo.” Change your mouth shape while holding a note and you can turn it into “ee.” Pitch and vowel identity can move separately. That is the separation the modem uses.

The presentation’s illustration shows two of the synthetic vowels. The vertical lines are harmonics of the pitch; the curves show the formant envelopes that shape their strengths. These are synthesis diagrams, not recordings of a human voice.

I use four vowel shapes, so each sound represents two bits:

The receiver concentrates on F2, the second formant, in the 1,000–2,500 Hz band. The project’s voice-channel models damage the lower F1 region badly enough that it cannot be the main basis for a decision. F1 still colors the sound and can help when it survives.

The simplest receiver measures the power-weighted center of the F2 band and chooses the nearest expected vowel. Those reference centers are 1,177, 1,426, 1,663, and 2,214 Hz. They differ from the nominal formants in the table because the receiver measures a band-limited collection of harmonics, not the peak of an ideal envelope. In particular, the 1,000 Hz cutoff removes the lower skirt of vowel 0 and pulls its measured center upward.

Move the measured centroid below to see the two-bit decision change. This isolates the fixed-center classifier; it does not simulate an entire call.

Classify a measured F2 centroid

Interactive instrument

Move a hypothetical measured F2 centroid through the detector’s 1,000–2,500 Hz band. The marker crosses four nearest-center decision regions and reports the vowel and two-bit symbol selected by the fixed expected centroids.

View source values

Nominal formants and measured synthesis centroids reported in the article.

Vowel	Bits	Nominal F2	Expected centroid
0	00	1074 Hz	1177 Hz
1	01	1420 Hz	1426 Hz
2	10	1663 Hz	1663 Hz
3	11	2252 Hz	2214 Hz

Nominal formants and measured synthesis centroids reported in the article.

Vowel	Bits	Nominal F2	Expected centroid
0	00	1074 Hz	1177 Hz
1	01	1420 Hz	1426 Hz
2	10	1663 Hz	1663 Hz
3	11	2252 Hz	2214 Hz

That is the basic demodulator. The rest of the receiver deals with all the ways a real recording can make that simple answer wrong.

A packet with a setlist

Four two-bit vowels make one byte. Each vowel occupies 50 milliseconds of voiced sound followed by 10 milliseconds of silence. A byte therefore takes 240 milliseconds, whether its pitch happens to be pleasing or not.

To hear the tune without the changing vowels, here is the complete head on one fixed vowel at half speed. This 31-second listening reference isolates the melody; it is not a data packet:

The complete 16-bar Giant Steps head on one fixed synthetic vowel, slowed to half the modem tempo: about 31 seconds. This is a melody reference, not an encoded packet.

Download MP3

The arrangement follows the packet structure. Parity is redundant data that lets the receiver repair damaged bytes; here, it also buys time for the outro:

The default pitch track advances on the byte grid, equivalent to a quarter-note pulse of 250 beats per minute. Longer messages cycle the head. When the payload ends, the vowel channel keeps transmitting the checksum and parity while the melody switches to the solo.

More error correction means more jazz. The packet stops when its protection bytes run out, even if Coltrane was going somewhere with that phrase.

Short packets cannot afford much of a solo. Instead, their outro uses transposed versions of The Lick. A two-byte hi ends in roughly four seconds, before it can play the head in full: enough repeating rhythm to help clock recovery, with a change of key before returning home. The final held E belongs to the end marker. The project’s damaged-header tests include a case where letting the solo continue through that marker breaks recovery; pinning the pitch preserves it.

The silences also earned their place. Filling the ten-millisecond guard with a smooth release tail caused RNNoise regressions documented in the synthesis code. A more continuous sound was a worse signal for that processing path. The encoder keeps the guard silent.

Now put it through the damage

First, one complete 140-byte packet: almost 50 seconds, including two full passes through the head and a solo outro. This is the first packet from the opening performance:

One full 140-byte packet, 49.68 seconds before MP3 padding. The Lick occupies 0–1.44 seconds, the Giant Steps payload melody 1.44–35.04 seconds, and the checksum/parity outro the remainder.

Download MP3

Now the corresponding processed version:

The same full 140-byte packet after modeled Teams-style spectral shaping at 40 dB tilt and RNNoise: about 50 seconds. This generated example was decoded back to the exact original payload.

Download MP3

Both new full-length recordings decode back to the exact original 140 bytes. The presentation’s spectrogram below illustrates the same kind of processing on its earlier, shorter example. It shows the first eight seconds before and after processing: time runs left to right, frequency bottom to top, and brighter regions contain more energy. Whole regions become weaker or disappear, but enough of the changing vowel structure remains.

“Modeled” matters here. These are repeatable software transforms from the project, not a recording of a live Teams call and not a promise about every Teams or Zoom configuration. The combined path is useful because spectral filtering can first remove features that a denoiser would otherwise preserve.

For a more obvious audible comparison, here is that same 50-second packet mixed with white noise at 20 dB SNR, followed by RNNoise’s output. Both versions also decode back to the exact original payload:

The complete 140-byte packet mixed with white noise at 20 dB SNR: about 50 seconds. Compare the background hiss with the next recording.

Download MP3

The corresponding full-length recording after RNNoise: about 50 seconds. Listen for the reduction in hiss and the complete melody that remains.

Download MP3

The little amplitude traces above the players come from the recordings. Each is normalized to its own peak; they show timing and envelope, not a common loudness scale or a decoding result. These longer recordings were generated with the modem itself and the channel transforms used by its Giant Steps example. The diagrams remain the originals from the show-and-tell deck.

Zero payload bits does not mean zero consequences

The melody is free in the bit assignment. It is not free in the acoustics.

Raising the pitch spreads the harmonics farther apart. An echo can cancel some of them. If a vowel depends too heavily on one harmonic, losing it can move the measured F2 center toward its neighbor. The encoder’s current F2 Gaussian width is 180 Hz, and the receiver has to cope with the different spectra produced as the tune changes.

The project tests chromatic melodies across B2–F♯4, including leaps and detuning. It also offers an armored payload melody using a conservative D lydian arpeggio when channel margin matters more than the tune.

On difficult recordings, the decoder retries with pilot-calibrated centers, pilot-derived channel equalization, spectral templates, and expectations for the received note. Weak symbol decisions can become erasures or second-choice candidates for error correction. A retry is another hypothesis about what was sent.

The musical structure supplies one more clue. If the length header is damaged, the receiver can compare candidate payload lengths against the known outro. Where the tune changes suggests where the payload ended. The solo has become a backup framing cue.

That clue cannot authorize delivery. The standard format protects a CRC-32 with the payload and checks it after Reed–Solomon correction. A candidate must pass the checksum; the decoder cannot simply deliver whichever guess sounds most plausible. CRC catches accidental corruption. It does not authenticate the sender or encrypt an audible message.

Forty-nine seconds for 140 bytes

Two bits per 60-millisecond slot gives 33.3 bits per second raw. Strictly, the symbol rate is 16.7 baud: one vowel every 60 milliseconds, carrying two bits. Either way, it is extremely slow. Synchronization, the length header, checksum, and parity all consume airtime.

For a maximum-size 140-byte standard message, the protected block contains 144 bytes: the payload plus four checksum bytes. The current parity rule adds 56 bytes. With the three-byte protected length header, start and end markers, and calibration pilot, that becomes 828 slots: 49.68 seconds of generated audio, or about 22.5 payload bits per second.

That is calculated encoder airtime. It excludes call buffering, capture delays, retries, and receiver computation. Short messages spend proportionally more time on framing. This is useful for a small handoff between machines, not a file transfer you would want to sit through.

There is also work left in the receiver. The repository exercises actual Opus encode/decode paths as well as noise, echo, clock mismatch, and continuous recordings, but difficult recovery can be expensive. A successful native test does not establish a browser latency budget, and a software channel does not settle what happens with a different microphone or denoiser.

Slow enough to sing, enough to bootstrap

The bargain is throughput for survival. I care about a short message getting through a phone call, bad microphones, cheap speakers, codecs and noise suppression, even when there is too little clean spectrum for a conventional modem. Different hardware and processing still change the margin.

One use I have in mind is bootstrapping a faster peer-to-peer connection. WebRTC needs peers to exchange setup information, but leaves the signaling transport to the application. A singing modem could supply that first exchange over a phone call.

Trade compact ICE credentials and candidate information, or a rendezvous token that gets both peers to the rest of the signaling. Larger negotiations need fragmentation; the session setup, including DTLS identity information, does not magically fit into one 140-byte packet. ICE tests the candidate paths, and the application can move onto a WebRTC data channel once a usable path is established, with STUN or TURN where required.

A minute spent getting connected can be worthwhile if the useful traffic then moves over the faster connection. The WebRTC handoff is a use case for this modem; the recordings above verify its acoustic messages. The code is on GitHub at deoxynet/song.rs.

Here are two full-size messages taking turns, with different payloads and a short gap between them. This generated exchange is about 100 seconds; every packet has room for the head and its own solo. I will leave them to it.

Two independently encoded 140-byte messages, joined by 0.75 seconds of silence: about 100 seconds total. Both original packets were decoded and checked against their payloads before joining.

Download MP3