Hello readers, and welcome to my final senior project blog post!!

It’s been a VERY eventful 11 weeks. My senior project experienced lots of setbacks, but also lots of progress. I definitely stepped out of my comfort zone with a project about AI and music, considering I know close to nothing about either subject, nor have I ever done a long-term research project like this! It’s safe to say I learned a lot about both aspects of my project and about the nuances of formal independent research.

My primary focus this week was to write my final product for the project. My final product was a paper detailing my project from start to finish, along with my findings, conclusions, and potential next steps. Since I haven’t talked too much about my findings and conclusions in my blog, I thought I’d share a few pages from my research paper for this week. Here they are:

Discussion

Target Domain Analysis

Ultimately, fine-tuning did not produce a significant improvement in either accuracy or instructional quality for tabla-related prompts. The following are proposed explanations for this observation.

1. Limited Data. The fine-tuning dataset contained 156 question-answer pairs compiled from online sources. This is relatively small compared to the scale of data used to train a base Gemini model. As a result, the additional training data may have had limited influence relative to the model’s extensive pre-existing knowledge. In other words, this level of fine-tuning wasn’t enough to significantly shift the LLM’s learned parameters. Additionally, the Vertex AI training metrics showed a training accuracy of 72%. While this represents an improvement from the initial ~55%, it indicates that the model aligned with the dataset only about 72% of the time, leaving substantial room for improvement through a larger or more diverse dataset. This is likely the primary contributor to the relatively low accuracy scores observed.
2. Overfitting. This is a common phenomenon in machine learning, in which the LLM ‘memorizes’ the question-answer pairs in the dataset, but lacks a thorough comprehension of generalizable patterns. This results in high accuracy scores for prompts closely aligned with the training data, but low scores for everything else. Since the 140 prompts in the target domain were not drawn verbatim from the dataset, lower accuracy scores may reflect the model’s limited ability to generalize beyond memorized examples.
3. Internal Conflict. This occurs when details in the fine-tuning material contradict the knowledge already encoded in the base model. In such cases, models often default to their prior knowledge, particularly when the fine-tuning dataset is relatively small. If such conflicts occurred, they could explain why the base and fine-tuned models produced similar accuracy scores.
4. Learned Response Style. Unlike with accuracy, in some categories, the fine-tuned model showed statistically significant improvement in instructional quality relative to the base model, suggesting that fine-tuning had a stronger effect on response style than on factual accuracy. Though the dataset did not explicitly define how different tabla concepts should be explained to a beginner, the information did carry a beginner-friendly tone and consistently reflected a clear and accessible instructional tone. It’s probable that the model implicitly learned this stylistic pattern during fine-tuning, resulting in responses that were better aligned with beginner-level communication. This would also account for the slight, though not statistically significant, differences observed in instructional quality within the OOD piano prompts. Exposure to beginner tabla instruction may have been generalized to a broader instructional style, meaning the model learned how to instruct a beginner piano student while being trained on tabla information.

Cross-Domain Analysis

A statistically significant difference was observed between the target domain (tabla) and the out-of-domain (OOD) piano set across both accuracy and instructional quality.

1. For accuracy, performance was higher in the OOD (piano) set. This suggests that the 156 question-answer pairs in the fine-tuning dataset were insignificant in comparison to a base model’s pre-existing knowledge of piano. In other words, the results support the interpretation that base models are more extensively trained on widely represented instruments such as the piano than on the tabla, and that the scale of fine-tuning in this study was not large enough to bridge that gap.
2. For IQ, performance was higher in the target domain (tabla). The fine-tuning dataset, compiled from beginner-oriented instructional tabla material, was rich in instructional nuances specific to tabla. These include hand placement, mnemonic syllables (bols), and practice tips, which aren’t as common in general piano theory data that the base model may have been trained on. Through fine-tuning, the model likely learned these nuances, which likely accounts for the higher IQ scores in the target domain compared to the OOD.

Conclusions

The results of this study support the following conclusions:

1. Within the constraints of this study (including a 156-pair fine-tuning dataset, a fixed prompt structure, and evaluation via an LLM-as-a-judge framework), fine-tuning Gemini 2.0 Flash had a more consistent impact on instructional quality than on factual accuracy. Specifically, fine-tuning improved the model’s ability to adopt a beginner-appropriate tone and incorporate tabla-specific terminology in certain responses. However, these improvements were not uniformly observed across all prompt categories.
2. No definitive conclusion can be drawn regarding the effect of fine-tuning on accuracy under these conditions. The relatively small size of the dataset and the moderate training accuracy (~72%) suggest that the fine-tuning may have been too limited to produce a measurable, consistent, and statistically significant improvement in factual accuracy.
3. Under the conditions of this experiment, fine-tuning with a small, domain-specific tabla dataset was not sufficient to achieve the level of accuracy observed for piano-related prompts using the base model. This suggests that the model’s pretraining likely included more extensive or higher-quality representations of piano-related knowledge. While this result is consistent with broader concerns about imbalances in training data across cultures and regions of the world, this study alone does not provide sufficient evidence to attribute the observed difference solely to disparities in cultural representation. However, it does indicate that, even after fine-tuning, Gemini 2.0 Flash demonstrates stronger accuracy on piano than on tabla within the scope of this evaluation.

Future Work

Considering the limitations of this study, particularly those related to time, dataset size, and computational resources, several extensions could strengthen and expand this work.

1. Dataset Expansion. Increasing the size and diversity of the fine-tuning dataset would likely improve both accuracy and instructional quality, particularly, incorporating a larger number of high-quality, domain-specific examples. Additionally, future work could explore multimodal training by including audio or video data, such as recorded demonstrations of tabla techniques. Because instrumental learning is inherently auditory and visual, relying solely on text limits the model’s ability to capture important aspects of performance and technique and limits a student’s ability to meaningfully progress. Adding different modes of media can help prepare the fine-tuned model for use by human tabla students.
2. Real-time Feedback. The current system is limited to text-based input and output, which limits human-like interaction. In contrast, human tabla instructors can offer real-time demonstrations and corrective feedback. Future versions of this project could integrate audio and video input, such as recordings of a student’s playing, and generate real-time feedback and encouragement.
3. Expansion to Other Instruments. In order to gain further insight into disparities in AI training data across different cultures and regions, this project’s framework could be expanded to include other underrepresented instruments. Comparing model performance across these instruments and more globally represented ones (e.g., guitar or drums) would provide a broader basis for evaluating how the origin of training data influences both accuracy and instructional quality.
4. Base Model Comparison. This project focuses on a single base model, Gemini 2.0 Flash. Future research could test and compare base models developed by other companies, like OpenAI or Anthropic. This would help assess how training data differs across base models and how this affects each model’s cultural sensitivity.

It is important to emphasize that, even as this work develops, an AI-based tabla tutor is not intended to replace human instruction. The tabla carries deep cultural and religious significance, and certain aspects of learning the instrument, such as the lineage system and stylistic nuances across regions, cannot be replicated by AI. As this project scales, the final product would be intended for use by students looking to get a high-level understanding of the tabla or refine technical skills without having access to a human teacher.