Audio to MIDI: how music transcription models work

11 steps · about 5 h 40 min · Intermediate

  1. But what is the Fourier Transform? A visual introduction. (video, 3Blue1Brown, 20 min)

    Every transcription model starts by splitting sound into frequencies, so this gives you the core idea first.

  2. Tutorial Vamp Plugins Part 1: Sound und Klangfarbe (guide, HfM Weimar, 25 min)

    It links spectrograms to musical pitch and shows a constant-Q view in Sonic Visualiser.

  3. The Constant-Q Transform — A Visual Guide (interactive, Brendan Lynskey, 25 min)

    Interactive visuals show why the CQT puts musical notes on an even grid, and Basic Pitch uses it as its input.

  4. Tutorial — librosa 0.11.0 documentation (doc, librosa, 40 min)

    You compute spectrograms, onsets and chroma in Python, which makes the earlier theory concrete.

  5. Transcription - Onsets and Frames (article, Magenta, 10 min)

    A browser demo lets you hear a neural transcriber work on piano audio before you read how it was built.

  6. Onsets and Frames: Dual-Objective Piano Transcription (article, Hawthorne et al. (Google Magenta), 45 min)

    It introduces the idea of predicting note onsets separately from frames, which many later models reuse.

  7. High-resolution Piano Transcription with Pedals by Regressing Onset and Offset Times (article, Kong et al., 40 min)

    It places Onsets and Frames in the wider field and shows how later work improved timing.

  8. A Lightweight Instrument-Agnostic Model for Polyphonic Note Transcription and Multipitch Estimation (article, Bittner et al. (Spotify), arXiv, 50 min)

    This paper shows how a tiny CQT-based network of about 17K parameters can compete with much larger models.

  9. spotify/basic-pitch: A lightweight yet powerful audio-to-MIDI converter with pitch bend detection (repo, Spotify, GitHub, 30 min)

    You run a real efficient model with pip install, then read the code to match it to the paper.

  10. I Built an Audio-to-MIDI Converter That Runs on Your Laptop — Using Spotify's AI (article, DEV Community, 15 min)

    It covers practical details such as resampling to 22.05 kHz and running the model in the browser.

  11. A Lightweight Architecture for Multi-instrument Transcription with Practical Optimizations (article, arXiv, 40 min)

    Last, you see how researchers speed up and extend a lightweight model, which is the same kind of efficiency work behind this trending project.