MakeItMidi

← Back to the MP3 to MIDI converter

Audio to MIDI: What You Actually Get, and the Five Things That Break

Yes, you can turn an audio file into a MIDI file. No, what comes out is not a score you can trust. It is a rough draft that needs ears and hands on it before it is useful. The converters are good enough to save hours, and bad enough to waste hours if you trust them blind. This is what reliably goes wrong, and how people who do this every week actually keep it under control.

Two failures that come from the audio rather than the file: a 4:5:6 major triad is read as a phantom C2 two octaves too low, and a continuous vocal glide is rounded into a staircase of semitone steps.

The pipeline is always the same four steps

Every tool, free or paid, runs the same chain. Decode the audio into raw samples. Cut it into short frames, usually 20 to 50 milliseconds each. Estimate the pitch of every frame. Then glue consecutive frames into note events with a start time, a pitch, and a duration.

The hard part is step three. A waveform is a continuous squiggle; a note is a discrete instruction. You are asking the machine to invent the sheet music from the recording, and it guesses.

The install step breaks before you convert anything

If you go the code route, the environment bites first. A standard setup is:

python -m venv midi_env
source midi_env/bin/activate   # Windows: midi_env\Scripts\activate
pip install librosa soundfile pretty_midi numpy

Then import librosa fails with a confusing message. The cause is almost always that numpy got pulled to version 2 and librosa has not caught up. The fix practitioners repeat is to pin it: pip install "numpy<2". People who have been bitten twice still forget this the third time.

The dedicated converter Basic Pitch is stricter about its runtime than the docs imply. It wants Python 3.8 to 3.11. On Apple Silicon it only runs on Python 3.10 — every other version forces you into a virtual machine or container. It auto-selects a backend: CoreML on macOS, TFLite on Linux, ONNX on Windows, and TensorFlow only when you are on 3.11 or ask for it explicitly with pip install basic-pitch[tf].

The error strings you will actually see

Basic Pitch fails loud and specific. These are the ones that show up most:

  • ValueError: 🚨 {audio_path} is not a file path. — use an absolute or clean relative path, and keep Chinese or special characters out of it.
  • ValueError: 🚨 {output_dir} is not a directory. — the output folder must exist first; mkdir -p it.
  • ValueError: Input shape must be rank <= 3 — your file is stereo. Make it mono: ffmpeg -i input.mp3 -ac 1 output_mono.mp3.
  • ValueError: Target size (...) must be at least input size (...) — the clip is under half a second; pad it or merge it with silence.
  • Exception: though it looks like a saved model file with error... — the downloaded model is corrupt; re-pull with python -m basic_pitch.data.download --force, or place files by hand under basic_pitch/saved_models/icassp_2022/.

When nothing loads at all, the model was never downloaded. python -m basic_pitch.data.download fetches it; python -m basic_pitch.predict --input clip.wav --output out --log-level DEBUG shows you where it died.

Octave jumps, and chords that collapse to one note

The note detector picks one pitch per frame. Feed it a chord and it returns the loudest note and throws the rest away. That is not a bad setting; it is a single-pitch algorithm doing exactly what it was built to do. If you need the chord, you either extract just the root as a writing reference or move to a multi-pitch method, which is a different and shakier class of tool.

Octave errors are the quieter disaster. A note with weak fundamentals and strong harmonics gets read one octave off, because the detector locked onto a harmonic instead of the base. The practical patch is to narrow the search range with fmin and fmax and then run a consistency check: if a note sits about twelve semitones from its neighbors, it was probably misread. With librosa.pyin the working call looks like:

f0, voiced_flag, voiced_prob = librosa.pyin(
    y,
    fmin=librosa.note_to_hz('C2'),
    fmax=librosa.note_to_hz('C7'),
    sr=sr,
    frame_length=1024,
    hop_length=256,
)

A voiced-probability cutoff around 0.55 keeps weak notes without inviting noise, and dropping any note shorter than about 50 milliseconds kills the flicker of false notes.

Separation is the real bottleneck, not pitch

Most people think the converter listens to the song. It does not. It first slices the mix into separate tracks — vocals, piano, bass, drums — and only then analyzes each slice. That first slice is where good and bad results are decided.

In one side-by-side of seven transcription tools on the same guitar solo, three commercial tools mashed the lead and the rhythm strumming into a single "guitar" voice and pushed pitch error to roughly plus-or-minus fifteen cents. Two open-source models separated the lead but invented harmonic lines that were never played. The one tool that isolated cleanly paid for it by smoothing away every vibrato, leaving notes as stiff, center-pitched blocks.

The separation models are trained on clean studio material. Feed them a phone recording with five decibels of room noise and the quality falls off a cliff. The advertised "98% separation accuracy" is a lab number on ideal audio. The separation step also shifts time: in one test the extracted vocal track started about 23 milliseconds late against the original, which is enough to throw off any lyric-to-note alignment.

Three tools, three different keys

Run the same clip through three different transcribers — one a homegrown analyzer, one an overseas open-source project, one a big-company API — and do not be surprised when they disagree on the basics. In one such test all three outputs were opened side by side and they could not agree on the key: one said C major, one said A minor, and the third declined to assign a key at all and wrote "key ambiguous, verify by hand."

That is the honest state of the art. The output is a probabilistic guess about the score, not the score. It cannot tell that a blues bend is emotional rather than out of tune, and it cannot tell that a synth pad's harmonics are intentional rather than distortion.

Rhythm and feel get flattened

Pitch is only half of it. The rhythm layer is where personality dies. The detector finds note onsets by peak, so a hi-hat chick that lands five milliseconds before the snare gets marked as simultaneous. A free-tempo passage gets yanked onto a rigid grid. A dancehall off-beat skank gets read as a delayed note and "explained" with slurs. Swing, shuffles, rubato — all of it collapses toward the nearest straight grid unless a human pulls it back.

Harmony fares no better. The detector matches simultaneous notes to the closest chord template and ignores function. On one well-known art-rock intro it returned the right root motion but missed the maj7-sharp-11 color and the chromatic bass walk-down entirely, because it read the passing tone as a wrong note instead of a deliberate half-step line.

A workflow that survives real audio

The people who get usable results treat the machine as a fast intern, not a co-author.

Pre-process before you convert. A light denoise helps pitch detection more than any model tweak. Cut long tracks into sections — verse, chorus, bridge — because the tools lose context on long files; splitting the track into sections alone has been measured to lift accuracy by more than forty percent. Label the tricky spots by ear first.

Run at least two, preferably three, transcribers on the same clip and diff them. Where they agree, you are probably safe. Where they diverge, go back to the audio with a spectrum view and decide yourself. One practitioner keeps a library of fifty corrected "the AI was wrong here" slices and uses them to fine-tune a local separation model, reporting another twenty to thirty percent gain on their own material.

Then check in this order: key and key signature first, then the rhythmic skeleton with quantization off, then chord function, then articulation and dynamics, then layout. Delete every automatically generated articulation mark and re-add only what the performance earns.

What to expect on Monday

Set the bar at "saves most of the hand-transcription time," not "replaces transcription." A clean mono instrument with a clear melody will get you seventy percent of the way in minutes. A dense mix with vocals, distorted guitars, and reverb will get you a pile of plausible noise that needs more cleanup than starting fresh. The converter is a draft engine. The score is still yours to write.

Convert an audio file to MIDI

Related questions