Adaptive Glove System for Real Time Piano Learning Assistance
Advisor: Mr. Magee
Internship Location: Synopsys. 800 N Mary Ave, Sunnyvale, CA 94085
Onsite Mentor: Amit Jalota
Learning piano independently can be daunting; mistakes often going unnoticed until they become ingrained habits. The Adaptive Glove System for Real-Time Piano Learning Assistance makes self-guided practice easier and more effective by offering instant feedback to help users build proper muscle memory. Using gloves equipped with IMU sensors, the system tracks hand and finger movements, mapping them into a virtual space to detect posture, fingering, and timing errors. A simple, visual interface delivers corrections in real time, allowing users to address mistakes as they happen. Rooted in research on motion tracking, wearable technology, and music education, this project aims to make piano learning more accessible to those without regular access to instructors. By combining cutting-edge sensor technology with user-friendly feedback, the system reduces frustration, accelerates progress, and boosts confidence. Beyond piano learning, it paves the way for broader applications in skill-building and rehabilitation, showing how wearable tech can transform education and unlock new possibilities for growth.
Project Blog
Week 0: Blog
Imagine this: you sit down at the piano, excited to learn a new piece, only to realize hours later that you’ve been practicing with the wrong technique. Now those mistakes are baked into your muscle memory, making them much harder to fix. Frustrating, right? That’s where my project, the Adaptive Glove System for Real-Time Piano Learning Assistance, comes in.
So, what’s the goal?
To make learning piano easier and more effective, especially for those without regular access to instructors. Using gloves equipped with IMU sensors, my system tracks hand and finger movements, mapping them into a virtual space. By detecting hand posture, timing, and fingering mistakes, it provides instant visual feedback so you can correct errors as they happen and develop proper muscle memory from the start.
But why me?
I’ve been playing piano for years and know firsthand how easy it is to practice incorrectly, only to have to unlearn those mistakes later. Combine that with my experience in technology and robotics, and this project feels like the perfect challenge for me. I’m drawing on research in motion tracking, wearable tech, and music education to create a system that not only teaches piano but also inspires confidence in learning.
My hope is that by the end of this journey, I’ll have built something that bridges the gap between self-guided learning and professional instruction. And who knows? This system could open the door to even bigger applications in skill development and rehabilitation.
Stay tuned for updates as I navigate this exciting adventure. There’s sure to be plenty of trial, error, and breakthroughs along the way! See you next week!
Week 1: Preparing Hardware Components and Software Framework for Motion Tracking
Hey everybody,
Welcome back to my blog! This week, I focused on finalizing the hardware components for my first prototype and started to build the software framework in Unity Hub for 3D visualization. I made good progress, but am locked behind a couple technical challenges and shipping times.
Hardware: Selecting and Ordering Components
To accurately track finger positions and movements, I need a system that combines IMU sensors for motion capture with wireless communication to transmit data to Unity in real time. After researching different options, I finalized my component selection:
IMU Sensors: The MPU-9250 (or MPU-6050, depending on availability) is my sensor of choice due to its built in 9DOF (degrees of freedom) tracking which includes a 3-axis gyroscope, accelerometer, and magnetometer. These are necessary for precise orientation tracking. However, due to the nature of the IMU sensor, it can suffer from gyroscope drift, requiring sensor fusion techniques or relocalization for correction.
Microcontroller with Bluetooth Communication: I selected the ESP32 because it provides native Bluetooth Low Energy (BLE) support, removing the need for an external Bluetooth module. The ESP32 will handle:
Reading IMU sensor data via I2C communication
Basic preprocessing to reduce data size
Transmitting data wirelessly via BLE to Unity
Managing power consumption to maintain efficiency for battery powered operation
Power System: Since the gloves must be fully wireless, I’m using 3.7V LiPo batteries with a TP4056 charging module to ensure safe charging and low power draw.
All components have been ordered, and my next step is to individually assemble and test each module before combining them into the glove system. By testing each component individually, I can identify faults early on and, if necessary, research and implement alternative solutions without disrupting the entire system.
Software: Setting Up Unity for 3D Visualization
Since my project requires real-time motion visualization, I’m leveraging Unity Hub for development and Blender for 3D modeling. This week, I focused on structuring the software framework and preparing for sensor integration once the hardware is assembled.
Refining the Unity Environment
Set up Unity with URP (Universal Render Pipeline) for optimized rendering performance.
Began designing a custom rigged hand model in Blender, ensuring that joint rotations will align correctly with incoming IMU data.
Explored Unity’s Inverse Kinematics (IK) system to determine how I can display motion between tracked points for smoother visualization.
Planning IMU Data Processing and Bluetooth Integration
Outlined a data pipeline that will receive BLE sensor input from the ESP32 and parse it within Unity.
Researched sensor fusion techniques, including Madgwick and Mahony filters, to stabilize noisy IMU readings when data becomes available.
Developed an initial approach for real-time quaternion rotation mapping, which will allow accurate representation of finger movements once the hardware is operational.
Next week, I plan to begin testing basic Unity input handling with simulated data and prepare for real-world sensor integration once the hardware components arrive.
See you next week!
Week 2: Blog: Rethinking the Build – Sensor Sizing and System Optimization
Hey guys, welcome back!
This week, I took a major step forward in defining the hardware and software structure of my adaptive glove system. While I originally planned to start assembling everything right away, I quickly realized that some of my initial assumptions, especially regarding sensor size, needed to be reevaluated. Additionally, a few of the sensors I had ordered either had not arrived or arrived in a nonfunctional state, slowing my progress down.
I began by testing the MPU-9250, expecting it to fit well at each joint. However, once I placed it on the glove, I saw an immediate issue: it was way too big. Mounting it at each joint restricted movement and placing it on the nail made it bulky enough to interfere with the piano key contact. The breakout board size was the main issue, not necessarily the chip itself. This meant I needed to find a smaller, more compact IMU that could still provide accurate motion data.
After researching alternatives, I narrowed it down to two options: the BMI160 and the BNO055. The BMI160 is a compact 6-DOF IMU with an accelerometer and gyroscope. It has a small size and low power consumption but lacks a built-in magnetometer and requires external sensor fusion. On the other hand, the BNO055 is a 9-DOF IMU with an accelerometer, gyroscope, and magnetometer, featuring built-in sensor fusion. This means the ESP32 doesn’t have to handle as much processing. Ultimately, I chose the BNO055 because its onboard sensor fusion simplifies software development and improves real-time tracking accuracy. While slightly larger than the BMI160, it’s still small enough to fit at each joint without interfering with playing.
With the IMU issue solved, I finalized the wiring and software structure. The IMUs will be placed at each joint, avoiding the fingertips to ensure clear key contact. Each BNO055 will connect to a TCA9548A I2C multiplexer, which manages multiple sensors on a single ESP32. For wireless communication, the ESP32 will transmit raw IMU data via Wi-Fi (using either ESP-NOW or TCP/IP) to the laptop for real-time processing. The software pipeline consists of the ESP32 firmware collecting and transmitting IMU data, while Unity receives the data, applies sensor fusion (using Madgwick or Mahony filters), and maps it to a 3D hand model.
Now that I have my final component list and build plan, my next step is to order the remaining components, assemble the first prototype, and begin testing real-time data streaming. I’m excited to see how this all comes together!
Hope to see you next week!
Week 3: Building the Software for IMU Data Processing
This week, I worked on building the software foundation for processing and visualizing IMU data in Unity. Since the glove will rely on motion sensors to track hand and finger movements, I needed a way to simulate how the data would behave before connecting real hardware. To do this, I set up virtual sensor points in Unity, each one representing an IMU at a different finger joint. These points will eventually control a 3D hand model, but for now, they help me test how movement data will be processed.
To make the system more realistic, I wrote a C# script that makes each sensor point move in a randomized way, mimicking real IMU behavior. This lets me see how Unity responds to motion data without needing to connect actual sensors yet. One of my biggest challenges was making the movement look natural instead of choppy or erratic. If the sensor positions updated in big jumps, the motion felt unrealistic. To fix this, I used smooth rotational changes instead of sudden updates. This will be useful when real IMU data is added because it will help prevent jittery movement caused by sensor noise.
I also had to figure out how to detect key presses using only motion data. Since the glove won’t have physical contact sensors, I needed a way to tell when a finger actually presses a key. The approach I used tracks each sensor’s starting position and measures how far it moves. If a sensor moves past a certain distance, the system registers a key press. This works because pressing a piano key usually involves a downward motion, so a noticeable vertical shift should mean the key was played. Once I start testing with real sensors, I’ll need to adjust the sensitivity to make sure it responds accurately to real finger movements.
Now that the simulation is working, the next step is getting real IMU data from the ESP32 into Unity. The ESP32 will collect motion data from multiple sensors and send it to the laptop using Wi-Fi. On the Unity side, I’ll need to write a script that receives this data in real time and applies it to the virtual sensor points, replacing the simulated movement with actual readings from the glove. One of the biggest challenges will be making sure the data is sent fast enough to keep up with real finger movements. If there’s even a slight delay, it could throw off the timing and make the system feel unresponsive. To avoid this, I’ll test different Wi-Fi communication methods, including ESP-NOW and TCP/IP, to see which one works best.
Stay tuned for more updates as I move closer to integrating the hardware and software into a single cohesive system. See you next week!
Week 4: Refining Motion Tracking and Overcoming Hardware Limitations
Welcome back to my blog!
This week, I focused on refining the software for tracking and visualizing finger movement using dummy data. Last week, I set up a basic Unity environment with placeholder points representing the IMU sensor positions. Now, I’ve been working on making the movement feel more accurate and realistic. Initially, the movement was too rigid since I was only updating the position based on raw dummy values. To fix this, I started applying smoothing algorithms, experimenting with both Madgwick and Mahony filters to stabilize orientation changes. I’m still testing which one works best for my application, but so far, Madgwick seems to provide smoother results, which is promising for fluid piano playing motion.
Another challenge I tackled was detecting key presses. The initial method, which was simply checking if a finger moved past a certain threshold, felt too imprecise. The issue is that finger motion is more nuanced than just distance traveled. To make it more reliable, I started implementing velocity-based detection as well, where I check not just the position but also how quickly a finger is accelerating toward the key. This should help differentiate intentional key presses from normal hand movement. I’ll continue tweaking the sensitivity to avoid false positives, but the early results are looking promising.
At the same time, I’ve been preparing for when I get real IMU data. I built a basic input pipeline that can receive data from an external source and update the Unity visualization in real time. Right now, it’s still using simulated data, but the structure is there so that when I connect the actual sensors, I won’t have to rewrite everything from scratch.
One of the biggest roadblocks I’m facing right now is funding. The ideal setup would require IMU sensors at every joint, but getting enough of them is expensive. Since I can’t afford all the sensors I need at the moment, I’m looking for alternative solutions. One idea is using fewer sensors and applying software-based estimations to fill in the gaps. Another approach is seeing if I can source cheaper alternatives without sacrificing accuracy. It’s a challenge, but working within constraints like this pushes me to be more creative in problem-solving.
Looking ahead, my goal is to finalize the key press detection and start integrating real sensor data into the visualization. I also want to improve how the hand model updates to make it feel more natural. Progress has been steady, and I’m excited to see how things shape up in the coming weeks.
See you guys next week!
Week 5: Back to the Hardware and Designing a Custom PCB
Welcome back to my blog!
Over the past few weeks, most of my updates have focused on software, like testing motion in Unity, simulating sensor data, and laying the groundwork for detecting key presses. But behind the scenes, I’ve also been working through a major hardware challenge that I’m finally ready to share.
This week, I designed a custom flexible PCB made specifically to fit on the fingers. The goal was to support IMU sensors in a way that keeps them small, lightweight, and positioned correctly without interfering with movement. Instead of wrapping around the finger, each PCB attaches using an ultra-thin ring-like mechanism that sits snugly between joints or on the nail. This makes it easier to place multiple sensors per finger without restricting how the hand moves while playing piano.
Due to limitations in the tools and equipment I have access to, I won’t be constructing the boards myself. I’ll be outsourcing the manufacturing to ensure that the boards are made precisely and efficiently. Still, the design process was a hands-on experience. I used KiCad to lay out the board, carefully positioning traces and components based on the IMU specifications and the space available on each finger. The final design is meant to be as compact and low-cost as possible, while still maintaining the accuracy needed for detailed motion tracking.
As I wait for the boards to arrive, I’m using this time to finalize the rest of the hardware setup and prepare for integration testing. Getting these sensors placed and functioning correctly is one of the most important steps for the entire system, so I’m looking forward to seeing how the physical build performs once everything is in place.
See you next week!
Week 6: Refining the Design
Welcome back to my blog!
Last week, I shared that I had completed the initial design of a custom flexible PCB meant to hold motion sensors on the fingers without interfering with piano movement. This week was all about refining that design, tracking down small issues, making the board more robust, and preparing it for fabrication.
After completing the first pass of the PCB layout in KiCad, I ran a series of design rule checks (DRC) to make sure everything complied with the manufacturer’s requirements. This led me to catch a few important details I hadn’t fully considered earlier. Some trace clearances were too tight, and a couple of the IMU footprints had minor alignment issues that could’ve made soldering unreliable. I also had to revisit how the flex cables would be routed and anchored, making sure the attachment mechanism wouldn’t pull or stress the board under finger movement.
Once those bugs were identified, I adjusted the layout to improve durability without increasing the board’s footprint. I widened critical traces, rechecked all via placements, and made sure the IMU sensor pads were correctly dimensioned to match the manufacturer’s specs. I also added some basic labeling on the board itself for orientation and reference during assembly and testing.
By the end of the week, I had a final version of the design that I was confident in. I packaged and exported the Gerber files, submitted them to the board manufacturer, and placed the order. Now it’s just a matter of waiting for them to arrive so I can start testing how they perform in a real setup.
While I wait, I’ll be working on prepping the software side for hardware integration. More on that soon.
See you next week!
Week 7: When the Parts Arrive… But Not As Expected
Welcome back to my blog!
After weeks of design and preparation, the first batch of custom PCBs finally arrived, which was a major milestone in the hardware side of my project. I had been anticipating this moment for a while, and unboxing the boards felt like real progress. But as I quickly found out, having the boards in hand doesn’t mean they’ll work as intended.
Right away, I noticed a few critical issues. Some of the IMU sensors weren’t communicating at all, and after double-checking connections, it became clear that a few of the traces on the board weren’t properly routed. In at least one case, a signal line that should’ve been continuous was broken entirely, likely due to a misalignment in the design that wasn’t caught during the DRC (design rule check). I also discovered that the board was slightly too rigid near the connector, making it uncomfortable to wear on curved finger joints, especially for longer practice sessions.
Additionally, some solder pads ended up being too small, making reliable sensor attachment nearly impossible. Given the small scale of the components, even a minor footprint miscalculation had major consequences. These aren’t things that can be fixed with simple rework. They’re baked into the physical design, which means I’ll need to make a revised version of the board and place another manufacturing order.
While it’s frustrating to have to reorder, this is a normal part of the hardware development process. These early-stage boards served their purpose: they exposed flaws in the design that I can now correct with confidence. I’ve already started modifying the layout: fixing broken traces, enlarging pads, adjusting the board shape, and generally tightening up the design. With these changes, I expect the next batch to be much closer to what I need.
For now, it’s a waiting game, but one that moves me closer to a fully working system.
Hope to see you guys next week!
Week 8: Blog: Waiting for parts
Welcome back to my blog!
With my replacement parts still in transit for most of the week, I focused on refining the software side of the glove system, simulating hand motion, building out functionality for interpreting finger movement, and continuing work on key press detection.
I spent a good portion of the week expanding the dummy-data-based hand simulation in Unity. Previously, I had each finger curling in isolation, but this week I added full-hand motion using procedural animation across all five fingers. I also set up a basic virtual piano in Unity using 3D cubes as keys. Each key now responds to finger positions, changing color and moving downward slightly when "pressed" based on simulated joint movement. Even though I didn’t have real sensor input yet, testing these systems with artificial values helped me tune the logic and make sure the visual feedback felt realistic.
I also built a more general structure for receiving live sensor data from the glove once it's ready. I created a TCP-based socket server in Unity to handle incoming data from the ESP32 via Wi-Fi. The idea is that once the glove is running, each sensor will send its current orientation data (probably in quaternion format), and Unity will read and apply that data to the appropriate finger joints in real time. The software now has placeholders for each joint’s incoming data and converts it to on-screen motion. This way, once the hardware is live, the transition from simulated to real-time should be seamless.
On Friday, the new batch of parts finally arrived. I haven’t fully assembled everything yet, but I began laying out the pieces and mapping out connections. My plan is to assemble and test one finger module over the weekend to verify that the IMU data can be sent correctly and interpreted in Unity. If that works, scaling up to the full hand will be the next step.
See you next week!
Week 9: Crisis & Pivot: A New Path Forward
Crisis struck this week. The custom-designed IMU sensor PCBs I had waited weeks for finally arrived… and promptly shorted the moment current touched them. I’m still not sure whether the issue lies in the manufacturing or my schematic design, but either way, the result is the same: unusable hardware and no time for replacements. With shipping lead times stretching beyond the final project deadline, I had to face the reality that my original plan wasn’t going to fly. So I pivoted, fast, to my backup.
The new system is still a glove-based tracking device, but it now captures finger motion mechanically instead of relying entirely on tiny embedded PCBs. Each finger is equipped with a badge reel-style spring-loaded mechanism connected to a potentiometer. As the finger bends, the reel rotates, and the potentiometer tracks angular displacement. For overall hand orientation, the glove includes a 9-DOF IMU. Since the custom sensors didn’t work, I had to buy a prebuilt development board for the sensor. You may be thinking, why didn’t you use this prebuilt sensor from the beginning? The answer lies in the size. This prebuilt sensor is almost 4 times the size of my custom one, leading it to not fit on the finger. Getting back to the main point, while this covers basic motion, it suffers from the well-known problem of drift: small errors in orientation build up over time, making the data less reliable.
To combat this, the glove also listens. A software module running on the ESP32 microcontroller captures ambient musical notes. These are treated as reference events: if the glove hears a known note while detecting a certain hand orientation, it can use that data point to recalibrate and realign itself, helping correct drift. The ESP32 acts as the brain of the system, collecting all potentiometer values, processing the IMU's motion data, analyzing acoustic input for pitch recognition, and merging it all into synchronized output.
This new direction combines physical tracking, inertial sensing, and audio-based error correction, and while it wasn’t my original plan, it’s a solid step forward.
Week 10: The final week
Welcome to the final week. It has been quite a journey, with both highs and lows, but it has finally come to an end.
This week was all about improvisation and iteration. I set out to connect an ESP32 microcontroller to a glove-based tracking system using a mix of potentiometers and an IMU sensor. It sounds straightforward in theory, until you hit real-world hardware.
First came the wiring chaos. With no custom PCB and limited soldering tools, I had to get creative. I tried using Dupont connectors to hook everything up, but they wouldn’t stay in place. After some trial, error, and a decent amount of hot glue, I got a semi-stable setup. Not ideal, but functional enough for prototyping. For components without pre-soldered headers, I rigged up temporary contact using male jumper wires and held them down with adhesive. It wasn’t pretty, but it got signals flowing.
Next came communication. I wrote Arduino code to read five potentiometers, then package that data and send it over WiFi via UDP to Unity. Getting the Unity side to listen correctly took some debugging, especially since the Unity receiver didn’t update inside the main thread at first. After adding the proper method calls and synchronization, I finally saw real-time data streaming in. It was a huge relief.
Along the way, I ran into every issue imaginable: analog noise on open pins, serial port lockups, a full laptop bluescreen, and even a moment where I feared I'd fried the board. Thankfully, most of these were fixed with reboots, better grounding, or wiring corrections.
If you were reading properly, you may have noticed that I mentioned an IMU sensor. This was meant to check the overall hand orientation, as all I currently measure are the notes played on the piano and which finger plays them. However, I was unable to get the IMU sensor functioning in time. When experimenting with it early in the week, I somehow managed to fry it, and there wasn't enough time to order another one with the 4 days shipping time that it had.
Despite the setbacks, I now have a functional glove that sends finger bend data wirelessly to Unity in real time, which then analyzes piano play and offers visual feedback. It’s not yet perfect, and there’s a lot left to polish, but the concept is there for a whole future in real time piano feedback.
I hope you guys enjoyed reading the blog throughout the weeks. Please contact me if you have any comments or questions. See you guys later!