Making elite, data-driven music feedback accessible to musicians at every level
"A cutting-edge Music teaching technology startup partnered with our team to modernize and scale an AI-powered music training platform. With real-time computer vision analysis, on-device ML, and personalized teaching via LLMs, the solution makes elite, data-driven feedback accessible to musicians at all levels."
What does the system do?
Waverley helped modernize and scale an AI-powered music learning platform that gives musicians real-time performance feedback. A smart capture device paired with a cross-platform mobile app streams practice sessions to the cloud, where computer vision and ML models analyze biomechanics and technique, delivering instant audio cues plus personalized, LLM-generated coaching reports.
Type of AI project: Computer vision · Predictive analytics / ML model · LLM application
Team location: Distributed team across Ukraine, Romania, Montenegro, Argentina, and Brazil.
AI-Powered Music Learning Platform
Our client, an early-stage music technology startup, is seeking to democratize access to data-driven learning through an AI-powered music learning platform.
The plan: leverage computer vision, biomechanics, and the latest in mobile technology to give real-time performance feedback to musicians, especially those without access to elite instruction. Supported by venture funding and preparing for presentations to major institutions, the client needed a partner to transform inherited technical assets into a robust, scalable product ready for market.
The core challenge: Modernizing an inherited platform under hardware and ML constraints
The project began with an inherited legacy codebase riddled with technical debt: circular dependencies, security gaps in the streaming pipeline, and publicly exposed training data. Improving ML accuracy required a major shift from logic-based to action-detection models, a pivot whose scope and infrastructure demands were underestimated at the start. Hardware obstacles were expected to be the ultimate roadblock in Phase 3, and coordinating distributed software, hardware, and design teams demanded significant investment in governance and process.
Legacy Codebase & Technical Debt
Inheriting a legacy codebase riddled with circular dependencies, security gaps in the streaming pipeline, and publicly exposed training data
A Major ML Architecture Pivot
Improving accuracy required shifting from logic-based to action-detection models, a pivot whose scope and infrastructure demands were underestimated at the start
Hardware Instability Risk
Battery drain, overheating, and firmware instability in the capture device were expected to be the ultimate roadblock in Phase 3
Coordinating a Distributed, Multi-Discipline Team
Navigating the complexity of coordinating distributed software, hardware, and design teams, with disparate tools and workflows, demanded significant investment in governance and process
Turning an inherited codebase into a scalable, science-backed platform
The client, alongside Waverley, reimagined the platform from the ground up: a smart capture device paired with a cross-platform mobile app (Flutter, iOS, and Android) streams practice sessions to AWS cloud infrastructure, where ML and computer vision models deliver near-real-time biomechanics analysis, instant audio feedback, and personalized LLM-generated (Anthropic Claude) coaching reports delivered via text-to-speech.
Assessment & Discovery
Took ownership of a legacy iOS app and codebase; set up the environment
Conducted architectural evaluation, ML analysis, app review, and gap assessment
Key discoveries: major gaps in streaming architecture, security risks in device connectivity (RTMP), tangled code dependencies, and unsecured training data stored in legacy Roboflow and Gemini accounts
Main deliverable: a detailed estimate for Phase 2
Data security incident remediation (parallel to Phase 1/2)
Discovered publicly accessible ML training data tied to the previous vendor's accounts on Roboflow
Downloaded and migrated the data to a private client account; revoked all public access
Fully removed legacy AWS vendor access and rotated credentials
Prototype Bring-up & Demo Delivery
Replaced the legacy iOS app with a complete Flutter rewrite, enabling cross-platform deployment (iOS and Android) and rapid publishing to TestFlight
Compressed ML models to ~11 MB TFLite format for on-device inference; integrated ML Kit
Reduced frame-by-frame streaming latency to ~200ms via FFmpeg; built a video stream emulator for pipeline testing
Reduced database query fetch times; deployed push notification backend; optimized cloud/Docker infrastructure
Integrated TTS (text-to-speech) audio feedback
Developed new UX/UI design concepts for the demo app
Created new annotation and session analysis tools; prepared labeling process and team proposals
ML analysis revealed core algorithm limitations (logic-based approach, inaccurate object detection, classification errors), prompting a strategic pivot to more robust action-detection models
Modernization (Blocked)
Scoped across several parallel tracks (infrastructure, ML enhancements, streaming, Flutter app, BA, QA, PM) to deliver a 6–7 month modernization. The phase was blocked before full execution: the hardware vendor shipped the first two test units, and inspection revealed battery drain, overheating, and connectivity instability requiring a firmware upgrade. Before the issue could be resolved, the client paused the team's involvement — a blocker on the hardware vendor's side, outside Waverley's control.
Work completed within Phase 3: GitHub migration, CI/CD foundations, Florence 2 auto-labeling integration, RTSP-to-RTMP conversion with Metal GPU texture binding (stable 15–20 min sessions), a 3D-printed rig for synchronized video + IMU data collection (near completion), WebRTC architecture investigation, RACI matrix, bi-weekly R-Y-G reporting, and change control processes.
Tech Stack at a Glance
Measurable improvements across engagement and efficiency
Phase 1 and Phase 2 were fully delivered on time and within budget.
Single Cross-Platform Codebase
Legacy iOS app replaced by a Flutter codebase (iOS + Android) published to TestFlight, reducing development overhead by an estimated 40%+
On-Device ML Inference
ML models compressed to ~11 MB TFLite — on-device inference enabled, cloud round-trips eliminated for basic detections
Faster, More Responsive Pipeline
Frame latency reduced to ~200ms; backend performance significantly improved
Security Incident Fully Remediated
Publicly accessible training data from a previous vendor was fully remediated and access secured
Improved Streaming Stability
Streaming stability improved to 15–20 continuous minutes per session during partial Phase 3 work
Governance Built From Scratch
RACI matrix, escalation procedures, and bi-weekly status reporting established to run the engagement like a venture-grade product org
Waverley delivered both contracted phases on time and within budget, converting a fragile, vendor-inherited product into a scalable commercial foundation. The engagement gave the client a defensible technical baseline, a clear modernization roadmap, and the governance infrastructure needed to operate as a venture-grade product organization — positioning the platform for scale-up.
Interesting technical decisions:
A cross-platform Flutter rewrite replaced the legacy iOS app, cutting development overhead by an estimated 40%+ while unlocking Android support.
ML models were compressed to ~11 MB TFLite, moving core inference on-device and eliminating cloud round-trips for basic detections.
A strategic pivot from logic-based to action-detection ML models, made after empirical analysis revealed classification and detection limitations in the original approach.
Let's build your AI-powered experience.
Is your platform rich in content and data that could be leveraged? We can help you build an AI solution that improves customer experience, maximizes your data's potential, and advances your company to the next level.