How AI Music Works

The AI music pipeline starts with training from thousands of real‑world examples.

AI music generation services are trained on a vast library of existing music. We explain how the process works, and what you can do to keep your own tracks from being assimilated.

When you hear an AI‑generated song for the first time — a convincing one, not the uncanny‑valley output of a few years ago — the reaction is usually somewhere between fascination and dread. The technology is moving fast. Services like Suno and Udio can now produce broadcast‑quality tracks in under a minute, complete with vocals, instrumentation and arrangement. But how does any of this actually work? How is our own music exploited in order to make these systems work? And is there anything we can do about it?

These are questions I’ve been grappling with for the past two years, both as a musician and as a researcher. The answers turn out to be more nuanced — and more hopeful — than most of the headlines suggest.

To understand how AI music generation works, you first need to understand that these systems don’t ‘compose’ in any meaningful sense. They don’t understand harmony, melody or rhythm the way a musician does.

How AI Generates Music

To understand how AI music generation works, you first need to understand that these systems don’t ‘compose’ in any meaningful sense. They don’t understand harmony, melody or rhythm the way a musician does. Instead, they learn statistical patterns from enormous quantities of existing music, and generate new audio that follows similar patterns. The process begins long before anyone types a prompt. It starts with training.

A raw audio file is, at its most basic level, a sequence of numbers: amplitude values sampled tens of thousands of times per second. At CD quality (44.1kHz), that’s 44,100 data points for every second of mono audio. Processing this directly would be enormously expensive in computational terms, so AI systems compress the audio into more manageable representations.

The most common approach is to convert audio into a mel spectrogram — essentially a visual map of how the energy in a piece of audio is distributed across frequencies over time, weighted to approximate how humans perceive pitch. If you’ve ever seen a coloured heatmap of a song’s frequency content in a DAW or spectrum analyser, you’ve seen something similar. These spectrograms condense the raw waveform into a format that’s far more efficient for a neural network to process, while preserving the information that matters most.

More recently, systems like Meta’s EnCodec have taken this a step further. EnCodec uses a neural encoder to compress audio into discrete tokens — short numerical codes, each representing a tiny fragment of sound. Think of it as converting audio into a kind of vocabulary, where each ‘word’ represents a particular combination of timbral and tonal characteristics. The encoder breaks the audio down; a decoder reconstructs it later. The compression is remarkably efficient: a few seconds of audio can be represented by a handful of tokens.

EnCodec uses a technique called Residual Vector Quantisation, which employs multiple ‘codebooks’ that are essentially lookup tables of sound fragments. Rather than trying to capture everything in a single pass, each codebook refines the representation left behind by the previous one, progressively filling in detail. It’s not unlike how a painter might block in broad shapes first, then add finer and finer detail with each successive pass.

Two Approaches To Generation

At present, the most prominent commercial AI music generation services are Suno and Udio.Once we have a system of representing audio using tokens or spectrograms, the actual generation can happen. There are two major approaches in use today, and most commercial services use some combination of both.

Autoregressive models work much like the predictive text on your phone. Given a sequence of audio tokens, the model predicts what comes next, one token at a time. Meta’s MusicGen is the most well‑known example. It uses a transformer — the same type of architecture that powers large language models like ChatGPT — to predict the next audio token based on everything that came before it. The conditioning can come from a text description (“upbeat jazz piano with brushed drums”) or even a melodic reference. The model doesn’t understand jazz or piano; it has learned the statistical relationships between millions of audio tokens that were labelled with similar descriptions.

Diffusion models take a different approach. Instead of building audio token by token, they start with pure noise and gradually refine it into coherent audio, guided by the text prompt. At each step, the model removes a small amount of noise, nudging the output closer to something that matches the requested description. Many systems use latent diffusion, where this denoising process happens in a compressed representation of the audio rather than on the raw waveform itself, making it far more computationally efficient.

In practice, many systems combine elements of both. Suno, for example, appears to use autoregressive generation combined with diffusion‑based upscaling for high‑fidelity output. The details of these commercial architectures are proprietary, but the general principles are well established in published...