AI-Powered Tabla Accompaniment for Indian Classical Music with Convolutional Neural Networks (CNNs) and Real-Time Signal Processing
Advisor: Dr. Dixit
Internship Location: Dolby Labs, SF (remote)
Onsite Mentor: Dr. Purva Gujar-Kale
Indian Classical Music relies heavily on the intricate interplay between vocalists and accompanying instruments such as the tabla. However, live accompaniment is often challenging or altogether impossible to arrange for practice and performance. This lack of consistent accompaniment limits musicians’ opportunities to perfect their skills, experiment with improvisations, and develop rhythmic fluency. Musicians today lack robust tools to address this need, and current technologies fall short in providing responsive, real-time accompaniment that authentically mirrors a human musician’s adaptive qualities. This project aims to develop an AI-powered system that provides real-time tabla accompaniment, filling an important void in music technology by offering Indian classical musicians, particularly vocalists, an AI-driven tabla accompaniment tool that closely resembles human responsiveness. I will start by gathering musical samples, proceed to train a Convolutional Neural Network (CNN) to generate rhythms overlaid on pre-recorded samples, extend this process to work in real-time, and finish with a fully functional model capable of responding dynamically to a live performer’s tempo and improvisation.
Project Blog
Week 0: Tabla Accompaniment in Indian Classical Music
Indian classical music relies heavily on improvisation, with one of its most important elements being the rhythmic interplay between the vocalist and accompanists. The tabla, a percussion instrument, plays a crucial role in responding to the singer’s phrasing, tempo, and mood. Finding a skilled tabla player for practice or performance can be a challenge, and without their support, vocalists often struggle with refining their technique, experimenting with improvisation, or establishing a solid rhythm.
This project aims to address that challenge by developing an AI-driven tabla accompaniment system that can respond to a vocalist’s performance in real-time. The goal is to use Convolutional Neural Networks (CNNs) and real-time signal processing to create software capable of analyzing a vocal input and generating tabla beats that fit rhythmically and musically.
The Vision: AI in Indian Classical Music
While AI has been applied in Western music for tasks like automated composition and transcription, it has not been explored as much in Indian classical music. Unlike Western compositions, which tend to be more structured, Indian classical music is more fluid. An accompanist must adjust to the nuances of a live performance. The question is: can AI be trained to understand these subtleties and react appropriately to a singer’s live performance?
The project will look at this question by building on work in sequence modeling and real-time signal processing. AI techniques like Recurrent Neural Networks (RNNs) have shown that it is possible to generate complex sequences based on patterns in data. Additionally, methods exist to separate percussive elements from music. Over these next 10 weeks, my goal is to create a model that can both recognize the rhythm in a vocal performance and generate appropriate tabla accompaniment.
The Plan: Concepts to Reality
Over the next ten weeks, the project will follow these key steps:
Week 1: Gather vocal and tabla samples from a variety of ragas and taals.
Week 2: Process the audio samples to extract important features like pitch, tempo, and rhythm.
Week 3: Develop the AI model using CNNs and sequence models such as LSTMs and Transformers.
Weeks 4-5: Train and refine the model to better recognize rhythms and generate appropriate responses.
Weeks 6-7: Use reinforcement learning to improve adaptability and start developing real-time processing, if time allows.
Weeks 8-10: Conduct testing, refine the system, and prepare for final presentations.
A New Tool for Musicians
This project has the potential to change how Indian classical musicians practice and perform. A reliable AI-powered tabla accompaniment tool would give vocalists a responsive practice partner, allowing them to work on their craft without needing a live accompanist. This project also contributes to the intersection of AI and traditional music, opening up the possibility for future innovations in the field.
Throughout this journey, I look forward to exploring the relationship between AI and musical expression. And I'll make sure to post updates as I tackle the technical, musical, and creative challenges involved in developing this system.
Week 1: Groundwork and Reading
Hi everyone! This week, I focused on laying the groundwork for my AI-powered tabla by building some background knowledge with research and starting the process of collecting audio samples.
Research & Techniques:
To understand the technical foundation of sequence generation in music, I explored Generating Sequences with Recurrent Neural Networks by Graves et al. (2013). This paper provided valuable insight into how RNNs, particularly LSTMs, can be used to generate complex sequential data—a core aspect of my project. The discussion on training models to predict the next step in a sequence resonated strongly with my goal of generating real-time tabla accompaniment.
In addition, I read Extending Harmonic-Percussive Separation of Audio Signals by Driedger et al. (2014), which introduced me to methods for isolating percussive elements in audio recordings. Given that tabla accompaniment heavily relies on rhythmic precision, implementing effective harmonic-percussive separation will be crucial in processing training data.
Data Collection:
With these techniques in mind, I began collecting tabla audio samples to build my dataset. My process involved sourcing high-quality recordings of solo tabla performances across different taals and tempos. I prioritized recordings with minimal background noise to ensure cleaner feature extraction.
To further diversify my dataset, I also started segmenting longer performances into shorter, labeled clips that will help train the model to understand rhythmic transitions and patterns. Additionally, I experimented with existing datasets but found that many lacked the specificity needed for my project, so I will likely need to generate more custom recordings in the coming weeks.
Next week, I’ll begin preprocessing the collected data and implementing basic sequence modeling techniques. Stay tuned! (see what I did there)
Week 2: Research and Expanding the Dataset
Hi everyone! This week, I focused on deepening my understanding of the techniques I’ll use and expanding my dataset to prepare for implementation.
Studying RNN-Based Sequence Generation
Building on my reading from last week, I explored Generating Sequences with Recurrent Neural Networks (Graves et al., 2013) in more detail. The paper discusses how RNNs—particularly LSTMs—can generate coherent sequences, making them a strong candidate for real-time tabla accompaniment. I analyzed how LSTM architectures handle long-term dependencies in music generation and considered how these principles could be adapted for rhythmic prediction in my project.
Understanding Harmonic-Percussive Source Separation
I also spent time reviewing Extending Harmonic-Percussive Separation of Audio Signals (Driedger et al., 2014), which presents methods for isolating rhythmic elements from harmonic content. Since tabla accompaniment relies on precise percussive elements, I studied different approaches to harmonic-percussive separation (HPS) and evaluated which techniques might work best for extracting clear tabla strokes from recordings.
Expanding the Dataset
To ensure a strong foundation for training, I continued collecting and organizing tabla recordings. I focused on gathering a diverse set of samples across different taals, tempos, and playing styles. Additionally, I began annotating these recordings with beat and stroke information, which will be crucial for training the model to recognize rhythmic structures accurately.
Next Steps
Next week, I plan to finalize my dataset and start designing the framework for implementing LSTMs in rhythm prediction. I’ll also look into using CNNs for feature extraction to improve accuracy before moving into actual coding.
Thanks for reading—more updates soon!
Week 3: Finishing Data Collection + Pre Processing
This week marked a significant milestone in my project — I'm finally all done with my data collection phase! I now have a total of 800 audio recordings, consisting of 100 recordings for each of the 8 taals I’ve chosen to work with. These include common taals like Teentaal, Rupak, Dadra, and others. I made sure to collect these at various tempos, dynamics, and playing styles to capture the rich diversity present in real-world performances.
To prepare the data for analysis, I spent a good amount of time cleaning and standardizing the recordings using audio editing software. I trimmed unnecessary silences, balanced the volume levels, and ensured that all recordings are clear and consistent. The goal here was to reduce any potential noise that could interfere with feature extraction later.
I’ve decided to start my deeper analysis with Teentaal, one of the most fundamental and widely used taals in Indian classical music. Its straightforward 16-beat cycle and familiar pattern make it an ideal starting point. As a first step, I’ve been manually marking the critical structural points of the taal — specifically the sam (the first beat, which is emphasized) and the khali (an empty, non-emphasized beat that marks a change in the flow). Understanding these markers is crucial because they help define the rhythmic character of a taal and influence how musicians interact with it during a performance.
What I’ve Learned So Far:
Importance of Data Quality: Ensuring clean, high-quality recordings is essential for accurate analysis. Even minor inconsistencies in volume or noise can interfere with feature extraction.
Complexity of Marking: Identifying the sam and khali manually has given me a deeper appreciation for the nuanced structure of Indian classical rhythm. These markers aren’t just technical — they’re expressive, shaping the flow of a performance.
Practical Constraints: Gathering a diverse and representative dataset for each taal took more time and effort than expected. It highlighted the challenge of balancing uniformity for model training with the natural variability of live performances.
What’s Next:
Next week, I’ll begin the feature extraction process on my Teentaal recordings. This involves using signal processing techniques to extract meaningful characteristics like:
Onset detection: Identifying when each beat occurs.
Spectral features: Analyzing frequency components to understand the tonal qualities.
Rhythmic patterns: Capturing the timing and spacing between beats to understand the underlying structure.
The idea is to train the model to understand what makes Teentaal recognizable and distinct. Eventually, the model should be able to generate a Teentaal pattern based on given characteristics, and later, analyze a recording of singing to overlay a suitable rhythmic accompaniment. This could have exciting applications for composers, performers, and educators in the field of Indian classical music.
Week 4: Feature Extraction and Early Testing
This week focused on feature extraction for my tabla dataset. After completing the initial dataset annotations, I developed a script to analyze Sam and Khali beats in the first ten recordings.
Building the Feature Extraction Pipeline
The main goal was to identify key characteristics that distinguish these beats. Using librosa, I wrote a script to extract features from short time windows around each Sam and Khali timestamp. The extracted features include:
RMS Energy – Measures loudness and intensity
Zero-Crossing Rate – Helps detect percussive elements
Spectral Centroid – Identifies where the main frequency is concentrated
Spectral Bandwidth – Measures how spread out the frequencies are
MFCCs – Captures the tonal quality of the beat
All extracted features are stored in a structured JSON file for future analysis. This data will be useful for training models or visualizing beat differences.
Initial Observations
Even without full analysis, some patterns are already emerging. Sam generally has higher energy and a lower spectral centroid, while Khali appears to be softer with a more spread-out frequency profile. These early trends suggest that the extracted features are effectively capturing differences between the beats.
Next Steps
With feature extraction working for the first ten recordings, the next steps will be:
Scaling up feature extraction for all 100 recordings
Visualizing the extracted features to identify trends more clearly
Exploring basic classification methods to test how well these features distinguish beats
Week 5: From Annotating and Extracting to Learning and Predicting
With the initial feature extraction pipeline complete, this week focused on scaling and laying the groundwork for more advanced modeling. Each step is bringing me closer to the ultimate goal: building an AI-powered system that can understand, analyze, and eventually even generate tabla.
Scaling Up + Tuning the Pipeline
I extended the feature extraction process to the full dataset of 100 recordings. This meant optimizing the code for speed and reliability, especially when dealing with noisy or inconsistent audio. I added functionality to automatically detect silence, trim irrelevant segments, and ensure that each window around a sam or khali is precisely aligned.
To ensure consistency, I also normalized the extracted features and updated the JSON structure to include metadata like taal type, tempo, and recording quality—information that could prove useful in downstream modeling.
Feature Comparison & Visualization
Using tools like matplotlib and seaborn, I started visualizing the extracted features across different taals. Some emerging patterns:
Sam beats consistently show higher RMS energy and lower spectral centroids.
Khali beats tend to have more variation in MFCCs, possibly due to their more open or less emphasized character.
Taal structure affects feature distributions more than I expected—teen taal and rupak, for example, show noticeably different profiles.
These comparisons are helping refine my understanding of what makes each beat “recognizable” to both the human ear and machine learning algorithms.
What’s Next:
With features in place, I’m now starting to think about the machine learning pipeline. The short-term goal is a classifier that can label whether a beat is a sam, khali, or other, given its audio features. Longer-term, this could expand into:
Beat tracking: Automatically identifying the rhythmic cycle of a recording.
Taal detection: Determining the taal purely from audio.
Generative models: Creating new tabla compositions or improvisations using AI.
Before that, I plan to:
Train a baseline classifier (e.g., SVM or decision tree) to test the separability of features.
Explore dimensionality reduction techniques (like PCA or t-SNE) to better visualize feature clusters.
Continue improving time alignment, especially around subtle sam entries.
The project is starting to shift from annotating and extracting to learning and predicting.
Week 6: Successful Taal Classification Model Training!
This week marked a major milestone in the Tabla AI project: the development of a complete beat classification system, capable of analyzing tabla recordings and distinguishing between sam and khali with impressive accuracy.
Classifier Development
The core of the week’s work was a new script, train_advanced_classifier.py, which implements a full machine learning pipeline. I trained and evaluated four models—Random Forest, Gradient Boosting, SVM, and a Neural Network—using standardized audio features extracted from the dataset. The pipeline includes hyperparameter tuning (via grid search), cross-validation, and outputs a full set of evaluation metrics: accuracy, precision, recall, confusion matrices, ROC curves, and feature importance plots.
Results were strong: both Random Forest and Gradient Boosting achieved 100% accuracy on the test data, while the SVM performed decently (~75%) and the neural network struggled. Spectral features (especially spectral centroid) and specific MFCC coefficients stood out as highly discriminative.
Interactive Classification Tool
Alongside the training script, I also built classify_beats.py, an interactive beat classification tool. This GUI-based app uses onset detection to locate beats in an audio file and then classifies them using a trained model. The waveform is plotted with color-coded beat labels and confidence scores, allowing for quick validation of model performance. Users can also switch between models to compare predictions.
What This Unlocks
With this system in place, the project now has a working pipeline from raw audio → feature extraction → beat classification → interactive visualization. This proves that sam and khali can be reliably distinguished using machine learning—and sets the stage for much more:
Expanding to more beat types
Real-time performance integration
Eventually, feeding into a generative model for tabla composition
The gap between theory and practice is closing quickly—and for the first time, the system isn’t just extracting or visualizing—it’s learning.
Week 7: Annotating Teentaal in Vocal Music
With the beat classification system in place and performing reliably on isolated tabla recordings, this week I began expanding the project’s scope into more complex and musically rich territory: full vocal performances. The goal was to begin building a dataset that could train a model to recognize sam and khali positions in actual songs, not just percussion recordings.
Dataset Expansion
The first step was collecting a new set of recordings—this time focused on vocal compositions performed in Teentaal. I curated a selection of bandishes and khayal pieces where the rhythmic cycle was clear and consistent. These recordings came from a combination of teaching sessions, practice material, and well-recorded live performances.
Once I had a working dataset, I began manually labeling each taal cycle. For every composition, I annotated the timestamp of the sam—the point of rhythmic resolution—and the khali, which often marks a lighter or unaccented point within the cycle. This process was more nuanced than working with tabla alone: in vocal music, sam is often indicated by melodic cadence or return rather than a distinct percussive hit, and khali can be implied through phrasing or drop in vocal emphasis.
The result was a set of precisely labeled vocal tracks that now mirror the structure of the tabla recordings I worked with earlier, but with the added complexity of musical phrasing, ornamentation, and tempo variation.
Preparing for Feature Extraction
In parallel with annotation, I started planning how to extract relevant features from the vocal audio. These will differ slightly from percussion-focused features. For instance:
RMS energy and spectral centroid still help identify moments of intensity or return.
Pitch contour and delta features can trace melodic phrasing across a cycle.
MFCCs remain valuable for capturing tonal texture, especially in expressive singing.
Onset envelope and tempo estimation will support alignment with taal cycles.
These features, when paired with my manually labeled sam and khali points, will form the training set for a model that can eventually predict rhythmic structure from vocal performance alone.
Why This Matters
This week’s work is a foundational step toward building an AI that can accompany—not just mimic—Indian classical music. It pushes the project toward context-aware analysis, where the system begins to understand how rhythm is expressed through melody, not just percussion. It also opens up exciting future possibilities:
Auto-aligning tabla with a solo vocal recording
Assisting students in finding the sam in a performance
Using vocal phrasing to guide generative tabla improvisation
The complexity is increasing—but so is the depth. The system is starting to listen like a musician.
Week 8: Predicting Sam and Khali in Vocal Music
After weeks of working with isolated tabla recordings, this week marked an important shift: applying what the system has learned to actual music. With the vocal dataset and annotations from Week 7 in place, I focused on training models that could recognize sam and khali points directly from vocal audio—a step that brings the project closer to real-world musical interaction.
Classifying Vocal Beats
The first part of the week was spent preparing the vocal annotations for modeling. Each labeled sam and khali was paired with a segment of audio and its corresponding extracted features, including MFCCs, spectral descriptors, pitch contour, and energy.
Using this data, I trained two baseline models—Random Forest and SVM—to classify beat type (sam, khali, or other) from these features. Even without percussion, the models were able to pick up on subtle shifts in phrasing and energy, especially around sam. Accuracy was promising, especially for the tree-based models.
Introducing Sequence Modeling
Static classification only goes so far when you’re working with music. To capture the flow of a taal, I began building a simple LSTM model that could process sequences of vocal features. The goal was to train the system to understand the broader rhythmic structure—not just isolated beats.
I structured the data into time-aligned sequences based on taal cycles and trained the LSTM to predict the beat type at each step. Early results showed that the model could pick up on repeating patterns and even anticipate sam returns based on melodic buildup.
Visualization and Evaluation
To understand how well the models were performing, I built new visualizations that aligned model predictions with the waveform of the original vocal track. Seeing the predicted sam and khali points plotted on the audio made it easier to assess how closely the system was tracking the actual rhythm of the composition.
This approach also highlighted edge cases—places where phrasing blurred the rhythmic boundaries—and helped identify areas where the model could be improved with additional features or better temporal resolution.
Looking Ahead
With Week 9 approaching, the focus now shifts to integration. The final phase of the project will involve:
Polishing the sequence model
Completing the end-to-end demo: input vocal → predict beat types → display aligned visualization
Preparing documentation and final presentation materials
This week was all about letting the system listen like a musician. Next week, it needs to perform like one.
Week 9: Building the Interface + Facing a Setback
As the project nears completion, this week brought a mix of major progress and unexpected difficulty. On one hand, I built the first full-stack implementation of my tabla AI system—bringing together months of modeling work into a usable web interface. On the other, I encountered a serious roadblock in my vocal music dataset that forced a last-minute pivot.
Building the Full-Stack Interface
The major technical milestone this week was creating a web application that allows users to interact with the system in a clean, intuitive way. The app accepts an uploaded vocal recording, runs it through the model, and returns a visualization of predicted sam and khali points overlaid on the waveform. Under the hood, it uses a Flask backend to handle inference and audio processing, and a React-based frontend for interactivity.
This user-facing layer not only makes the project more accessible—it also helps demonstrate how tabla AI could be used in educational or compositional tools. It’s a shift from research to application, and it finally feels like the system has a real interface with the outside world.
Dataset Setback
Unfortunately, the progress on the web app was matched by a serious issue in the vocal music dataset I’d been working with. After closer inspection, I realized that some of the files had mislabeled taals, inconsistent tempos, or unclear phrasing that made sam and khali annotation unreliable. Since the classifier’s success depends on accurate training data, I had to halt further modeling on this dataset.
To recover, I reached out to researchers who had previously published work on Hindustani classical music and requested access to a curated dataset they had compiled. Their dataset is known for having high-quality annotations and cleaner audio, and I’m hopeful that it can support the next iteration of the model—even if it comes slightly after the formal end of this project phase.
Looking Ahead
While the setback was frustrating, the week still marked an important turning point. I now have:
A working, deployed model interface
A clear path toward improved data quality
A concrete understanding of what the system needs next
In the coming days, I’ll finalize the interface, clean up the backend logic, and evaluate how the model performs on the new dataset once I gain access. Despite the bump in the road, the system is alive, and it’s built to listen.
Week 10: A System That Listens—and Plays
This final week marked the completion of the Tabla AI project. What began as a question—can a machine understand Indian classical rhythm?—has grown into a working system that doesn’t just analyze taal, but actively performs it. The model now listens to real vocal recordings, identifies the rhythmic structure, and overlays tabla accompaniment that aligns with the music. It’s functional, demonstrable, and complete.
Clean Data, Clean Results
After facing data quality issues last week, I reached out to a researcher at IIT Kanpur who generously shared a curated dataset of taal-aligned recordings. Their dataset, complete with accurately marked sam and khali points, was exactly what I needed to finalize and validate my model. With this cleaner data, I retrained and fine-tuned the system, and the model’s performance on vocal recordings improved noticeably—especially in terms of rhythmic alignment and cycle prediction.
The Final App: Overlaying Tabla in Real Time
This week was all about pulling the pieces together into a finished product. I completed the full-stack web app, which now does the following:
Accepts a vocal recording (in Teentaal).
Uses the trained model to identify the structure of the taal—specifically, the positions of sam and khali.
Dynamically overlays a synthesized tabla track that matches the predicted rhythmic cycle.
Reflection
Over ten weeks, the system evolved from basic audio processing to deep musical understanding. It learned to identify rhythmic downbeats, distinguish between types of tabla strokes, analyze melodic phrasing, and now—finally—accompany a performer.
This project brought together everything I love: classical music, technical rigor, machine learning, and creativity. It taught me how to listen more closely, both as a musician and a developer.
The machine now listens. And for the first time—it plays back.