AI Music Trainer

The freeform trainer reading a sung arpeggio live, captured end to end through the real getUserMedia → AudioWorklet → Rust/WASM pipeline.

Problem

Learning to sing in tune is a tight feedback loop, but most practice happens alone, with no one around to catch the small mistakes that quietly become habits. Lessons are expensive and infrequent, and most “practice apps” just play a backing track. They never actually listen to you. I wanted a tool that hears every note you sing and tells you the truth about it, instantly, in a plain browser tab.

What I built

AI Music Trainer is a web app that turns solo vocal practice into a guided session. It captures your microphone in the browser, detects your pitch in real time, and shows exactly where you’re sharp, where you’re flat, and by how many cents. None of your audio ever leaves your device.

It grew into a complete product, not a demo:

• Freeform pitch trainer. Sing anything and watch a live pitch trace, the detected note, and a cents-accurate in-tune meter that turns green the moment you land it.
• Song practice. Sing along with a scrolling piano roll of any melody while your voice is traced against the target notes and scored on pitch and timing. Matching is octave-aware, so you can sing in whatever range fits your voice.
• Import any song, at $0 infrastructure cost. Search public-domain melodies, paste ABC notation, upload a MIDI or MusicXML file, or transcribe an audio recording. Everything is parsed in the browser, so files never touch a server.
• Exercise suite. An interval ear trainer plus mic-based warm-up drills (pitch matching, interval singing, sustained notes, sirens and glides), each with its own scoring model.
• Progress dashboard. Practice streaks, an accuracy trend over time, daily practice minutes, and per-interval accuracy pulled from your ear-training history.
• Accounts and billing. Supabase auth with row-level security, and a Stripe-gated Pro tier over a free plan.

How it works

The interesting engineering is in making accurate, low-latency pitch detection run entirely client-side.

• Real-time pitch detection is Rust compiled to WebAssembly. The McLeod Pitch Method hot path lives in a Rust crate compiled to wasm32-unknown-unknown (with SIMD) and runs inside a Web Audio AudioWorklet, using a 46 ms analysis window that produces a new reading every 12 ms. It measured about 1.8× faster per detection than the pure-JavaScript path, which stays as a runtime fallback. Nothing in the real-time loop touches the network, and raw audio is never stored or uploaded.
• Importing your own songs is what keeps practice going, so I made it work entirely on the client. MIDI and MusicXML files are parsed in-browser over a shared melody core that picks the melody track, flattens chords to a skyline, and derives exact beat timing. Audio recordings are transcribed with Spotify’s open-source Basic Pitch (TensorFlow.js), then cleaned into a single monophonic line: confidence-based overlap resolution, vibrato merging, octave-jump correction, and a tempo snap with a free-time fallback for rubato. Audio imports open an interactive piano-roll editor so you can fix any stray note before saving. Because all of this runs client-side, adding songs costs nothing to serve.
• Search that actually matches. Upstream melody sources match loosely, so results are re-ranked for relevance and any candidate missing a query word is dropped. The empty state routes you to a web search plus upload instead of a dead end.
• Stack. Next.js (App Router) + TypeScript on the front end; Supabase (Postgres, auth, row-level security) for accounts and session history; Rust/WASM and the Web Audio API for the audio pipeline; TensorFlow.js for transcription; Stripe for billing. The app only ever persists session summaries, never audio.

Impact / results

• Shipped a feature-complete v1 as a solo developer: real-time trainer, song practice, four import paths, an exercise suite, a progress dashboard, auth, and billing.
• The pitch pipeline is cents-accurate end to end. I verified this by driving the full getUserMedia → AudioWorklet → WASM path with a synthetic microphone in headless Chromium and asserting exact readouts (a 220 Hz tone reads “A3, 0.0 cents”).
• Backed by 120 unit tests covering detection accuracy, WASM-vs-JS parity, the song-comparison engine, importer parsing, and the scoring math, plus a real end-to-end browser pass.
• Most importantly for practice, it catches mistakes in the moment, exactly when they’re cheapest to fix.

What I’d do next

• Deploy publicly (Vercel + Supabase + Stripe) and record a short demo for the landing hero.
• Add WASM-based vocal isolation so full-mix recordings import cleanly, not just clean solo vocals.
• Turn on the already-built, Pro-gated AI coaching layer that turns each session’s stats into plain-language practice advice.