Chapter 1: DeepBach and the art of passing for Bach↗
In 2017, researchers at Sony's Computer Science Laboratory in Paris and collaborating institutions ran an experiment.
They trained a neural network called DeepBach on Bach's four-part chorales, then asked listeners to distinguish its reharmonized excerpts from the originals.
The result was striking. In an online study with 1,272 listeners, roughly half the votes identified DeepBach excerpts as Bach. The listeners were not all experts: they ranged from people who rarely heard classical music to musicians, composition students and professionals. Greater musical experience generally improved discrimination. Nor was the system composing complete works from nothing. It retained the soprano, rewrote the other voices, and presented 12-second excerpts with the same instrumental rendering. DeepBach, published at ICML 2017, offered a compelling demonstration of stylistic imitation.
It also brought a deeper question into focus:
If machine-generated music sounds enough like Bach, how far is that from being good music?
Identifying an author, recognizing a style and judging a composition are different tasks. Sounding like someone does not settle how well a piece works.
Chapter 2: Symbolic music generation, an old problem in a new form↗
First, what is symbolic music generation?
Imagine a piano. A key is pressed, a string vibrates, air moves, and a note is heard. Record that sound wave, and the result is audio.
But the event can also be described another way: “On the first beat, play middle C at medium velocity for a quarter note.” This records events and instructions rather than the waveform. MIDI mainly stores events of this kind; scores, ABC notation and MusicXML represent music at other symbolic levels. For what each representation retains and leaves out, see Musical Representation Systems.
Symbolic generation asks a machine to write these sequences: notes, chords and rhythms that can be performed as music.
The problem is surprisingly old. In the 1950s, Lejaren Hiller and Leonard Isaacson at the University of Illinois used a computer in composing the string quartet Illiac Suite. Computer composition did not begin with deep learning.
Music Transformer, released in 2018 and published at ICLR the following year, demonstrated minute-long piano generation with repetition and longer-range connections. A more specific question was becoming urgent: how could a model organize notes across time, rather than simply keep producing them?
Chapter 3: How should music become a language a machine can read?↗
This sounds simple. It is one of the field's most consequential decisions.
Suppose an alien is learning human language. One route begins with symbols that can be combined; another begins with sound and leaves the learner to find patterns. The analogy distinguishes input levels. It does not mean that symbols arrive with grammar already solved, or that audio systems never use discrete codes.
For symbolic generation, the immediate question is: where should the “words” of music begin and end?
3.1 REMI: how many positions does one note need?↗
REMI, introduced at ACM Multimedia 2020, is a representative event encoding. Bars, within-bar positions and note attributes make metrical structure more explicit in the sequence. The table summarizes information types, not a complete fixed token grammar.
| Information | Role |
|---|---|
| Bar / Position | Locate a bar and a position within it |
| Pitch | Specify note pitch |
| Velocity | Specify playing intensity |
| Duration | Specify note length |
| Tempo / Chord | Provide tempo, harmony or other contextual information |
One note may require several tokens. Polyphony, longer pieces and finer event encodings make the sequence longer still.
And that creates a problem: the sequence becomes expensive.
For ordinary dense self-attention with sequence length and feature width , the pairwise computation is usually written as:
At fixed width, doubling the length roughly quadruples this part of the work. If explicitly stored, a score matrix for 10,000 tokens contains 100 million entries per head, per layer. Implementations can avoid storing the entire matrix, but that does not make the long-sequence problem disappear.
3.2 Compound words: keep related attributes together↗
The 2021 Compound Word Transformer groups related neighboring attributes into compound words, with different output heads handling different token types. This reduces the number of steps needed to unfold individual attributes.
The paper reported five- to tenfold faster convergence in its comparisons. That is not a universal compression ratio or a guarantee of the same end-to-end speedup.
Compound words also change what must be coordinated. Attributes formerly spread across sequence positions must now work together inside a structured representation. Their dependencies remain a joint problem for the encoding and the model.
3.3 MuPT: how much music fits in 8,192 tokens?↗
The 2024 preprint MuPT takes another route: an ABC-based text representation, with SMT-ABC coordinating bar alignment across tracks.
The idea is natural. If music can be written as text, can it benefit from language-model-style sequence modeling? Using text does not, however, automatically confer all the abilities of a general language model. Music data and appropriate training are still needed.
The authors report that an 8,192-token context covers about 90% of their training data. That percentage belongs to that corpus and encoding. Change the tokenizer, polyphonic density or repertoire, and the same number of tokens need not represent the same amount of music.
Fitting a whole piece into one context is also different from producing it in one forward pass. MuPT still generates autoregressively.
Chapter 4: Which architecture?↗
4.1 Music Transformer: distance as well as position↗
Music Transformer's contribution was more than importing an NLP architecture.
Music contains repetition, variation and return. A chorus reappears at a distance; a phrase answers earlier material. Text is not devoid of these properties, but they make distant relationships especially important in music.
The Music Transformer paper uses relative-position self-attention and improves its implementation. The distances are between sequence positions. Sixty tokens apart does not mean sixty seconds apart: the same number of events can span different amounts of musical time.
That small distinction determines what information the model actually receives. Relative positions make distance easier to use; they do not label every future chorus in advance. The memory optimization concerns intermediate relative-position information, not a conversion of all dense attention to linear complexity.
4.2 Museformer: not every distant bar needs equal detail↗
Museformer, published at NeurIPS 2022, takes the question further: how should a limited attention budget be distributed across bars?
It uses fine-grained attention for selected relevant bars and coarser summaries for others. The studied patterns include offsets of one, two, four and eight bars. These are research choices informed by musical structure, not laws that every composition must obey.
The model can therefore combine local detail with more distant material in contexts exceeding 10,000 tokens. Handling a long sequence and producing convincing long-range form in every output remain separate things to check.
Chapter 5: Diffusion enters, with a different way to generate↗
Symbolic diffusion did not begin in 2023. An ISMIR 2021 paper already generated music by diffusion in the continuous latent space of a pretrained VAE.
Two distinctions matter here. Transformer is a network architecture; autoregression and diffusion describe different generation procedures. A diffusion denoiser can itself be a Transformer.
Autoregression resembles writing along a path, each step following what has already been written. Diffusion resembles revising a sketch, gradually restoring content at multiple positions. The analogy explains the update procedure. It does not mean a diffusion model knows the entire composition from the outset. An opportunity to operate globally is not a guarantee of successful global structure.
5.1 GETMusic: provide some tracks, complete the others↗
GETMusic, published at IJCAI 2025, uses GETScore to arrange note tokens by track and time. During training, source tracks remain visible while target tracks are corrupted through masking; GETDiff learns to predict the missing content.
Its flexibility is easy to grasp: supply a melody and request accompaniment, or supply some tracks and complete others. Within a predefined ensemble, one framework supports different source–target combinations.
The attraction is the interaction, rather than an impressive task count. A composer can decide part of a piece and ask the model to work around it. Such control is not exclusive to diffusion; the next chapter offers an autoregressive alternative.
5.2 Cascaded diffusion: from form to accompaniment↗
Whole-Song Hierarchical Generation, published at ICLR 2024, divides generation into four levels.
| Level | Content |
|---|---|
| Form | Whole-piece information such as phrase layout and key progression |
| Reduced lead sheet | Reduced melody and rough chords |
| Lead sheet | Complete melody and refined chords |
| Accompaniment | Piano accompaniment |
Rather than asking one model to solve every scale at once, it lets higher-level decisions constrain lower-level detail. The lower levels also use autoregressive controls. Actual methods do not always fit into two neatly opposing camps.
The paper demonstrates recognizable sectional form in its pop-music setting. It makes the whole song a more explicit modeling target without solving every musical form or making individual works exempt from evaluation.
Chapter 6: Anticipatory Music Transformer, seeing what in advance?↗
One especially interesting direction is Anticipatory Music Transformer, released in 2023 and published in TMLR in 2024. Its central idea is anticipation.
Imagine an accompanist who can already see the next passage on the score. That advance view helps prepare the harmony. The crucial ability here is reading ahead, not correctly guessing an improvisation that has not happened.
The model interleaves control information with the events to be generated. Controls already provided for a later musical time appear earlier in the context. A five-second anticipation interval lets autoregressive generation use some future constraints. It does not grant access to an unknown future melody.
The authors' interactive demonstrations include generating in segments, accepting or rejecting candidates, and controlling selected notes. These interactions divide control into smaller decisions: a person can retain some choices while asking the model to propose what comes next.
The listening test also deserves a close reading. One comparison used 20-second excerpts: a five-second prompt followed by a 15-second continuation. Against reference music, the model received 18 wins, 31 ties and 11 losses; the difference was not statistically significant (). That supports a limited comparison of those continuations. It does not establish that AI accompaniment equals human composition: failing to detect a difference does not prove equality.
Chapter 7: The evaluation crisis, when “good” is underspecified↗
Return to the opening question.
DeepBach made authorship difficult to identify. Music Transformer sought longer-range relationships. Anticipatory Music Transformer brought future controls into generation. These achievements answer different questions.
When all of them become “more musical,” what distinctions have disappeared?
7.1 First, what does a metric see?↗
Symbolic descriptions include pitch-class histograms, note density and pitch entropy. They can diagnose pitch use, event density or distributional diversity. None alone answers whether a piece moves a listener.
For notes with MIDI pitches , let fold octaves into twelve classes. A simple pitch-class distribution, without duration weighting, is:
Its entropy is:
Now consider a thought experiment. Take a passage from a Chopin nocturne. Keep every onset and duration, but permute the pitch labels. The distribution and entropy above remain unchanged, as does onset-based density. The melodic trajectory, simultaneous combinations and harmonic resolutions may be transformed beyond recognition.
Not every permutation becomes noise, and not every metric ignores order. The narrower point is that a statistic invariant to permutation cannot, by itself, judge relationships that permutation changes. Much of what matters in music lies in those relationships.
Kader and Karmaker's 2025 evaluation survey discusses gaps between automatic metrics and human judgments. CLAP and FAD concern audio or audio embeddings; applying them to symbolic models requires specifying an audio rendering. They cannot simply be combined with note statistics into one undifferentiated ruler for musical quality.
7.2 Second, human listening still requires a target↗
If metrics have blind spots, why not ask people to listen?
That is possible, but it does not make study design simple. Who listens, for how long, through which rendering, and whether the question concerns preference or style can all affect the result.
Grötschla and colleagues' 2025 study compared 12 text-to-audio models through 15,600 paired judgments from more than 2,500 participants. It used ten-second instrumental clips to accommodate the compared models' common capabilities. This was not a symbolic-music experiment, nor evidence that human studies cannot exceed ten seconds.
Short excerpts nevertheless cannot adequately test a theme's return several minutes later. A locally fluent passage need not have anywhere to go in the whole piece. Claims about long-range organization need listening tasks on the relevant timescale.
7.3 Third, disagreement is not necessarily failure↗
One listener hears a novel progression; another hears an awkward one. The same rhythm sounds lively to one person and mechanical to another.
Disagreement can contain noise, but also different aesthetic aims and experience. Reducing all of it to a mean score need not produce a clearer answer. Criteria, listener backgrounds and uncertainty should be visible, rather than mixing scores from different questions.
A correlation coefficient is not the percentage of people who judged correctly, and it cannot casually be translated into “barely better than chance.” Authorship identification, similarity judgments and preferences for generated music are different tasks.
7.4 Fourth, whose musical rules?↗
Both datasets and metrics have a scope. A scale-membership measure designed for one tonal style is not automatically a standard of excellence for every tradition. An encoding that folds pitch into twelve classes also cannot preserve pitch detail it never records.
The question is where a rule has explanatory value. If the training data, listeners and criteria cover a narrow range, the conclusion should stay within that range.
Otherwise, a system may merely score better in a familiar style while being described as understanding music better.
Chapter 8: Datasets, where everything begins↗
Several datasets appear repeatedly, but they are not interchangeable samples of music.
| Dataset | Content and scale | What a comparison must retain |
|---|---|---|
| Lakh MIDI | 176,581 deduplicated MIDI files in the full collection | The full collection differs from the audio-matched subset; report version, filtering and invalid-file handling |
| MAESTRO | About 200 hours of mainly classical piano, with roughly 3 ms audio–symbolic alignment | Precision is not stylistic coverage; official splits keep the same composition out of both training and testing |
| POP909 | Piano arrangements of 909 pop songs, with melody, secondary melody and accompaniment tracks | These are symbolic parts, not separation of every instrument from the original audio |
| JSB Chorales | A Bach-chorale benchmark used in polyphonic modeling | Quantization, voice representation and split versions affect the task; studies need not use identical settings |
Larger collections are also appearing. GigaMIDI distinguishes full pieces, loops and expressive categories. Aria-MIDI supplies large-scale piano transcriptions with filtered and deduplicated subsets. Its quality scores come from a classifier, and its metadata can be inaccurate; neither is manual certification of every file.
Scale alleviates scarcity. It does not automatically resolve duplication, stylistic coverage, transcription errors or test leakage. Before comparing two papers, it is worth asking whether “the same dataset” really means the same version and task.
Image classification has shared reference points such as ImageNet; language understanding has benchmark suites. Symbolic music has benchmarks and shared tasks too. The missing piece is a widely shared evaluation practice spanning different generation goals, not an absolute absence of evaluation.
The MIREX 2024 symbolic-generation task set a specific target: piano accompaniment for an eight-bar, 4/4, Western-pop-style lead sheet. The results page (official results poster) reports coherency, naturalness, creativity and musicality. Teams could generate 16 candidates per test item and select one for submission.
That detail changes the interpretation. The task evaluates a workflow that includes selection, rather than the average quality of one random output. Making such conditions visible is what standardization can achieve; one score need not answer every musical question.
Chapter 9: Harmonization, a new approach to an old task↗
Consider a specific task: given a melody, arrange its harmony.
DeepBach is an important example. It learns conditional note distributions using past and future context, then repeatedly resamples through pseudo-Gibbs updates. LSTMs are components of the predictor, not the sampler itself. Fixing selected voices or positions and completing the rest creates room for collaboration.
MelodyT5, published at ISMIR 2024, places harmonization within a broader score-to-score framework. ABC provides a common representation for seven melody-related tasks, including generation, harmonization and segmentation. The paper reports benefits from multitask learning for data-scarce tasks: evidence for shared representations and training, not the discovery of a universal musical grammar.
RL-Chord takes another route, training melody-to-chord generation with reinforcement learning. Its TNNLS paper appeared online in 2023 and in a 2024 issue. The authors also report adaptation for Chinese-folk harmonization with a style classifier. “Zero-shot” should not be read as absence of any style information, or transfer as proof that a universal set of harmony rules was learned.
It nevertheless leaves an interesting question: which structures learned in one kind of music transfer to another? Answering it requires specifying what transfers, rather than naming a successful result “musical grammar.”
Chapter 10: Does the machine understand music?↗
End with a question that has no simple answer.
When Anticipatory Music Transformer produces accompaniment, does it understand what it is doing?
If a dominant seventh resolves to the tonic, does the model represent that relationship? Does the representation contribute to generation? Does it experience tension and release? These are different questions. Prediction, control and listening studies can address some of them; pleasant output alone cannot settle the question of experience.
I do not know.
What is clear is that machines can now generate music that listeners find musical in some tasks. That makes the next question sharper: which kind of generation, for whom, and under what conditions of listening counts as good?
Building machines that write notes has made “writing good music” less self-evident. Evaluation must return to the work itself: how one note receives another, why a phrase needs to return, whether an ending has truly arrived.
Perhaps this will make an old question concrete again: what is the beauty of music?