Tuning the Tutor: Testing the Effectiveness of Fine-tuning AI as a Beginner Tabla Tutor
Advisor: Jordana
Internship Location: Databricks
Onsite Mentor: Lu Wang, Senior Software Engineer, Databricks
My research question is: To what extent does fine-tuning an AI model improve its effectiveness as a beginner tutor of the instrument tabla? The project will compare the tutoring capabilities of a model that receives additional tabla-related data (fine-tuning) with one that does not, while holding the base model and prompt-engineered questions constant. In other words, this project tests whether a base model already has sufficient understanding of tabla to deliver accurate, comprehensible lessons through prompting alone, or whether additional tabla-specific training improves its instructional quality. As the majority of AI training data comes from Europe and the US, this project demonstrates how targeted fine-tuning can make AI a culturally authentic and inclusive tutor, showing that models trained beyond Western-centric data can meaningfully support regionally or culturally-specific topics like the tabla.
The project will consist primarily of observational and experimental research, as I’ll be systematically testing identical questions and comparing each model’s responses. It will also require some internet research to locate tabla-related datasets and conduct the fine-tuning process. My final product will consist of a comprehensive objective analysis on which model performs better and why, along with individual statistics on what kind of questions each model responded better to and in what way. I will also produce an AI model capable of providing beginners, including myself, with an introductory overview of playing tabla.
Project Blog
Week 1: Introduction
Hello! My name is Rinisha (Rini) Golani. Welcome to my senior project blog post. In this entry, I'll be giving a quick introduction to my project and discussing my Week 1 Progress (and struggles!). Feel free to reply with any questions or feedback. Enjoy!
Project Introduction:
This project will assess the effectiveness of fine-tuning with respect to beginner tabla lessons. I plan to create a dataset of tabla-related content and use that to finetune a base AI model. With advice from experienced tabla students, I will create a method to evaluate the responses of the AI model. I will feed the fine-tuned and base models (control group) the same set of prompts and use a third-party LLM to evaluate their accuracy.
This project aims to comment on the level of inclusivity currently present in the field of AI training. In the expected scenario that the base model has a lower accuracy rate than the fine-tuned model, this would mean that base models aren’t trained with enough data on culturally or regionally-specific instruments, posing a threat to misrepresenting their cultural value.
My project is tailored to my personal interests and goals. I am deeply passionate about the social and cultural implications of AI, which this project is aligned with. I also have wanted to learn tabla from a young age, so I hope this project will give me a beginner-level overview of the instrument.
Week 1 Progress:
For my final project for my Capstone Java class this year, I chose to complete the first step of my senior project: fine-tune Gemini 2.0 Flash with tabla-related text data. I initiated the fine-tuning process on Google Colab, a browser-based coding environment that runs and executes Python Code. I used Google Cloud, specifically the Vertex AI API, to interact with the fine-tuned model. As such, going into week 1 of my senior project, I had a headstart since I already had my fine-tuned model ready to test.
During Week 1, when I logged into Vertex AI to check the metrics on my finetuned model, I discovered I was far from ready to start data collection with this model. As pictured below, the accuracy of my model was about 54%.
The graph is a screenshot from Vertex AI in Google Cloud. Metrics are calculated according to Vertex AI’s automated evaluation methods.
Though this graph indicates fine-tuning increases the accuracy of a model, the improvement is too small to show a difference in the quality of responses the model would give when I begin prompting in a couple of weeks. My mission for the rest of week 1 became to increase the model’s accuracy.
After some internet research and a meeting with my advisor, I confirmed that the dataset I used to fine-tune was too small, so the model didn’t “learn” enough new material to show a significant increase in accuracy. The dataset I originally used to fine-tune was compiled by me from a website about tabla, hence its small size.
I started by adding about 16 more pages from another tabla-related website. I then ran the finetuning process again in Colab, only to discover the accuracy increased to just 55%. I was surprised to find that adding so much new data increased the accuracy on Vertex AI by only 1%.
Reflecting on the statistics, I took a step back and did some more internet research on how to increase the accuracy. I learned the issue was that my original dataset and what I added contained roughly the same information, so the model didn’t learn anything new. In fact, adding the new data did nothing but increase compute time. The solution is to add new information. I searched for more articles and blog posts, this time with different information from what I already had. I looked for information on the most famous tabla players in the world, plus how to tune the instrument. When adding this, the accuracy increased to 60%, a significant improvement compared to before.
Next Week's Goal
60% is still lower than I would like. In addition to my main objectives, I plan to continue working with my advisor and doing internet research during Week 2 to understand how I can improve the accuracy further. One idea I have is to increase the number of epochs, or the number of iterations the model goes through the dataset during fine-tuning. Overall, though this has been a significant challenge in my progress thus far, I believe I have a method to counteract it before beginning data collection over the next couple of weeks.
Week 2: The Fight for Higher Accuracy
Last week, my objective was to increase the accuracy of my fine-tuned model. Adding new information to the dataset helped slightly, increasing accuracy by about 5%. This week, improving accuracy continued to be my main goal. However, I took a different approach, focusing on two main strategies: reformatting my dataset and hyperparameter tuning.
In my Colab notebook, the code I wrote accepts a PDF dataset from the user and reformats it into a file type acceptable for fine-tuning, as a plain PDF cannot be used directly. Part of this process includes creating question-answer pairs. That is, examples of prompts I could ask the fine-tuned model, along with correct answers. Rather than feeding the model 100 pages of raw information about tabla, organizing it in this way allows it to find patterns across the examples and learn how questions should be answered.
Previously, my question-answer pairs were too vague, as each question asked the model to “explain the following tabla concept to a beginner.” Although the answers differed, the prompts were identical. This made it harder for the model to recognize what concept each answer referred to and to detect patterns in the data.
Realizing this mistake, I added a distinct question before every new concept in my dataset. I made sure the questions reflected a beginner’s tone: straightforward and without musical jargon. For example, where the dataset began discussing the history of tabla, I added the heading “How and where did the tabla originate?” I then rewrote the data reorganization part of my code to locate these questions in my PDF and their corresponding answers. I expected this to help the model learn from structured examples rather than raw information.
My second strategy for improving accuracy was hyperparameter tuning. Hyperparameters are settings in my code that affect how the model learns. A simple analogy I came across in my research is paraphrased as follows: if a student is studying for a test, the hyperparameters aren’t what the student learns, but their studying techniques. For example, the student might study slowly, review material multiple times, or check their understanding along the way. Hyperparameters control similar learning behaviors for the model.
Based on the regulations of Vertex AI and Gemini 2.0 Flash, there were 2 hyperparameters I could adjust: epochs and learning rate multiplier. Epochs are the number of times the model reads over the dataset. I learned from internet research that more is not necessarily better. More epochs increase the time it takes to fine-tune, and it risks overfitting, which is when the model memorizes correct answers rather than identifying patterns. This means the model wouldn’t be able to answer questions that are not explicitly mentioned in the data. I experimented with the number of epochs to find a sweet spot where the model learns, not memorizes, and I settled on 4 epochs.
The second hyperparameter is the learning rate multiplier, which is a unitless scalar that defines the rate at which the model goes through data. Lowering this value can lead to more precision but a longer computation time. I changed it from 1 to 0.1, but when I didn’t see much improvement, I changed it again to 0.002.
After adjusting these hyperparameters and reformatting my dataset, the accuracy on Vertex AI increased to ~72%, a significant 12% increase from last week. I hope to improve the accuracy further later on, but my next step is to switch focus to creating the rubric I’ll use to evaluate the model’s responses. I’ll do this by chatting with a few experienced tabla students and getting their expertise on what a “good” AI response would look like. This will be the topic of next week’s blog post! Thanks for reading!
Week 3: "Is there a rubric we can use?"
Hello readers, and welcome to my third blog post!
Last week, my primary focus was to increase the accuracy of my fine-tuned model by reformatting my dataset and adjusting hyperparameters. This week, I decided to pause accuracy improvements and shift toward creating a rubric to evaluate model responses.
As a reminder, I will collect data by asking both a fine-tuned and a non-fine-tuned model questions about the instrument tabla and using a third-party LLM as a judge to assess the quality of the responses. This will provide insight into the effectiveness of fine-tuning for this particular subject, which is the ultimate goal of my project.
I am choosing to use a third-party LLM as a judge to grade responses. This decision is influenced by two main reasons: First, it saves time. Manually grading ~100 responses would be tedious and time-consuming. Using an LLM makes the process far more efficient while producing comparable results. Second, and more importantly, it protects the integrity of the research. As a human, I could unintentionally grade the fine-tuned model more favorably since I expect it to do better. Using an LLM reduces this bias and ensures objective evaluation as per the rubric.
To grade the responses with an LLM, I needed to create a rubric defining what counts as a high- or low-quality answer. My first step in doing this was creating categories of prompts/questions related to tabla to ask the models. I expected each category to require slightly different criteria for a good response. For example, the rubric for a question about playing a specific tabla stroke should differ from one about the history of tabla. I created a set of categories and ran it by a classmate who plays tabla to ensure it was comprehensive of what a beginner should learn. Implementing her advice, I finalized the following categories and subcategories.
My next step was to determine what constitutes a “good” answer for each subtopic. I did this by contacting two tabla students who have each been playing for 8+ years. I scheduled a meeting with each, showed them the subcategories, and asked for their thoughts. Instead of asking for the “correct answer” to a hypothetical question in each subtopic, I asked what a strong answer should include, based on their experience as beginner tabla students. For example, they shared that a right answer for the ‘playing strokes’ subtopic should include where to place your hand on the drum, which finger to use, etc. Their input became the foundation of my rubric; when an AI response includes all the required information with full accuracy, it would score full points. Repeating this process for every subtopic, I now have a rubric for every type of question I plan to ask the models.
The next steps are to create prompts for each of the subcategories and feed them to both the fine-tuned and not-fine-tuned models. I will also create a prompt for the LLM as a judge to grade the responses. The prompt will include a role/persona, the question, the model’s response, the rubric, the grading task, and a sample correct answer, which contains accurate information for the judge to compare against the model response. Once these steps are complete, I can begin analyzing responses using the LLM judge and start identifying patterns in the results. Later, I can expand the sample size by adding more prompts and improving model accuracy. This will be the topic of next week’s post. Thank you for reading!
Week 4: The Setup
Welcome back! Hope you’re excited for the 4th installment of my blog.
Last week, my focus was on creating the rubric to use for LLM evaluations. I did this by meeting with a couple of experienced tabla players and asking them, for each type of question, what a high-quality answer by the AI model should include. I recorded their advice and used it to create a structured and objective rubric.
This week, while I didn’t achieve everything I had hoped, I still made meaningful progress. I started by generating 10 questions for each subtopic and created “ground truths” for each. Ground truths are sample correct answers for each subtopic that contain accurate information. My external advisor recommended using Notebook LM to generate the questions and ground truths because this application pulls information directly from the sources I provide it, minimizing potential for hallucinations. This means I can upload my fine-tuning dataset into Notebook LM, and the ground truths will contain information exclusively from the dataset, which I already know is accurate. This way, the third-party LLM-as-a-judge can fact-check the AI responses against these ground truths to generate the accuracy score.
Next, I organized every prompt by category in a Google Sheet. I included the category, subcategory, sample correct answer, accuracy criteria from my rubric, and a prompt for the LLM-as-a-judge, which asks if to grade the response (I’ll show an example of this next week). I also left columns in the spreadsheet for the actual responses to be graded and the LLM-as-a-judge’s notes for my reference.
However, before I can implement the LLM-as-a-judge, I need to feed my newly generated questions to both my fine-tuned and base models, as their responses are what my project evaluates. I currently have 140 questions across 2 models, so 280 total prompts. Manually prompting each one and copying the outputs into the spreadsheet would be time-consuming and inefficient.
To streamline this process, I started writing a script in Colab to access the prompts in my spreadsheet and automatically ask them to both models. However, I've been encountering some bugs in my program, which has inhibited the progress I was hoping to make this week. Though the code itself is running, it's not accessing the models properly, so responses are not being generated.
I want to continue debugging the code this weekend. At the same time, I want to be mindful of whether the time spent troubleshooting outweighs the efficiency gains; if the time spent debugging isn’t worth it, I may switch to manual prompting to stay on track. When I have collected all 280 responses, I will begin implementing the LLM as a judge process! Thank you for reading, and see you next week!
Week 5: Generate, Generate, Generate
Hello readers! Welcome back! Last week, my focus was on creating the set of test questions for my models, organizing them into a spreadsheet, and writing a Colab script to send those questions to both my base and fine-tuned models and log their responses. Though I completed the first two steps, the last wasn’t as productive as I’d hoped because I was working through some bugs in my code.
Fortunately, with some help from my external advisor, I got the code running this past weekend! The script reads all 140 questions from my spreadsheet, sends each one to both models, and places their generated answers into dedicated columns in the same sheet. This saved me loads of time (and sanity), since it eliminated the need to manually paste 140 questions into two models (280 total!) and then copy and paste their answers back into the spreadsheet.
Just to recap, I now have the answers both models produced to the exact same set of tabla-related questions. One model is the standard base Gemini model, while the other has been fine-tuned using additional information about tabla. My project now moves into the evaluation phase, where I test which model produces more accurate and more intelligible answers.
As I discussed in last week’s post, the evaluation process uses a third-party “LLM-as-a-judge.” This means I give a separate AI model all the information it needs to evaluate the models’ answers objectively. I explained my reasoning for using an AI judge instead of human evaluators in my Week 3 blog post.
This week, I began writing the evaluation script. I chose Gemini 1.5 Flash as the judging model because it’s fast, free, and can handle long contextual prompts. The script constructs a detailed instruction prompt that tells the judging model exactly how to evaluate each answer.
The prompt begins with a role description, something like: “You are an expert AI judge evaluating an AI-generated answer for accuracy.” This tells the model what perspective it should adopt. It then defines the task: for example, “Assess the AI answer based on the original question, the expected answer, and specific grading criteria. Provide a score from 1–5 along with a short explanation for your rating.” The explanation portion is especially important because it allows me to verify that the judge’s reasoning actually makes sense.
The prompt also includes several pieces of context:
the original question asked to the models
the model-generated answer being evaluated
the grading criteria from my rubric
a sample accurate answer I created last week using my dataset and NotebookLM
To say the least, this is a very long prompt; it’s much longer than anything I’ve previously asked Gemini to process. Even though the evaluation task itself is straightforward, the judge needs a lot of context to ensure the results are objective and consistent.
I plan to run the script four times:
Evaluate the fine-tuned model’s answers for accuracy
Evaluate the base model’s answers for accuracy
Evaluate the fine-tuned model’s answers for instructional quality
Evaluate the base model’s answers for instructional quality
Each run evaluates all 140 answers, so every iteration requires Gemini to process 140 evaluation prompts. Because of that, a single run takes one to two hours to complete.
You may be wondering: why is the Colab script necessary? Isn’t this just prompting an existing AI model? You’re right, it is. The reason for the script is again for efficiency. It ensures I don’t have to construct each prompt manually by copying and pasting all the context listed above 140*4 times, then loading the score and explanations back into my spreadsheet.
This week, I ran two of the four iterations: both of the accuracy evaluations. After the scores and explanations were organized in my spreadsheet, however, I realized I had overlooked something important: My evaluation prompt asked the judge to assign a score from 1–5, but it didn’t include descriptions of what each number actually represents. In other words, I never defined the scale. Because of this, the judging model effectively created its own interpretation of the scale, which makes the evaluations more subjective than I’d like.
Next week, I’ll add clear definitions for each score in the prompt and rerun all of the evaluations. I expect this to take most of Week 6, since each iteration takes so long and I can only run about one per day without exceeding the request limits for a free Gemini model.
Even though a few hours of work ended up being wasted this week (since I have to scrap the initial evaluations), I’m still happy with the progress I’ve been making. This was a pretty long post, so I truly appreciate those of you who made it to the end :)
Only one more week until a much-deserved spring break! See you all soon!
Week 6: Evaluation Nation
Hello readers! Welcome to blog post number 6.
This week’s post is a continuation of last week’s. In week 5, I began the evaluation process by writing a script in Google Colab to read from my spreadsheet and score responses. Each time the script runs, it generates scores for all 140 model answers in that set based on one specific evaluation criterion. Because I am comparing two models across two evaluation dimensions, I needed to run the script four times:
Evaluate the fine-tuned model’s answers for accuracy
Evaluate the base model’s answers for accuracy
Evaluate the fine-tuned model’s answers for instructional quality
Evaluate the base model’s answers for instructional quality
Last week, I ran the script twice before realizing I forgot to include descriptions of each score (1-5) in the prompt to the LLM-as-a-judge. This week, I added this in and generated responses for all 4 categories. Since running the script takes several hours and there is a daily limit on how many requests I can send to the judging model, this was the bulk of my work this week.
I also realized this week that I planned to implement a control group. In the control, I will ask both models a set of questions unrelated to tabla and evaluate those answers as well. The expectation is that both models should perform equally well, since my fine-tuning dataset only contains information about tabla. In other words, the fine-tuned model should not be more knowledgeable than the base model on topics unrelated to tabla.
If both models receive similar scores on these unrelated questions, it will strengthen my results by showing that any differences in performance on tabla-related questions are likely caused by the fine-tuning process itself, which is exactly what my project aims to test.
I haven’t implemented this set of questions yet, so this will be my focus when I come back from spring break (week 8). I will also begin analyzing the scores. This will involve identifying patterns across different prompt categories and subcategories, and determining where each model performs best and worst. From there, I’ll be able to better understand how fine-tuning affects the model’s performance.
Below, I’ve attached some photos of my spreadsheet, which contains the responses from both models and the output generated from the evaluation process. Hopefully, taking a look at this can clear any doubts my readers may have about my methodology. The first screenshot is the accuracy evaluation, and the second is instructional quality. Note that in both, the prompts and responses stay constant.
Thank you all for reading! If you have any questions, please do leave a comment and I’ll get back to you as soon as I can.
Week 8: It's Under Control
Hello readers! After a refreshing and restful spring break, week 8 of my senior project was quite productive!
This week, my main focus was running my control group test. To do this, I created a small sample set of prompts (20 prompts) completely unrelated to tabla. I gave these prompts to each model (fine-tuned and base) and evaluated the accuracy and instructional quality of their responses.
This is the same process I did with my 140 tabla-related prompts over the past few weeks. However, these 20 prompts do not pertain to tabla. The expectation is that the model responses will be of equal quality and accuracy. If this is the case, it would prove that any discrepancies in my 140 tabla-related prompts can only be due to the fine-tuning process. As a reminder, the fine-tuning process educated the model solely on tabla, as my dataset did not contain information on any other topic.
I chose to create a set of questions related to piano, as I expect another musical instrument will cause a smooth apples-to-apples comparison. To ensure consistency with how I spoke to tabla students to create a rubric, I also spoke to one of my friends who has been playing piano for nearly a decade. I found a beginner’s guide to playing piano online and asked my friend to scroll through it and cross-check the validity of the information based on her learning. With her notes and approval, this article served as the ground truth for piano-playing information on which the accuracy could be scored.
Going through the process of collecting responses and running the script to evaluate them, I filled in my spreadsheet for the 20 piano-related prompts. Just scrolling through my spreadsheet, I can already see that the accuracy scores for piano are much higher than tabla. Though this doesn’t directly affect my conclusions, it does interestingly imply that a base model is much more knowledgeable about piano than tabla, which is a bias I’d like to look more into and discuss in my final presentation.
Ideally, this marks the end of the data collection phase of my project. Next week, my focus will be on actually starting to analyze the scores collected. I’ll be noting outliers, calculating averages per topic, and compiling graphs and findings. At this point, I feel a bit ahead of schedule, but I expect next week’s analysis will provide insight into what worked and what didn’t, so I’m prepared to rerun tests and collect data again if needed. Looking forward to another productive week!
Week 9: Are You Sure About That??
Good morning readers! Last week, I finished off the evaluations for my project. I began analyzing the numbers this week, but found something shocking: the base model had a higher accuracy average than the fine-tuned model.
This was surprising because the fine-tuned model has more tabla-related data than the base model, and both are being evaluated against the data on which the fine-tuned model was trained. So ideally, the fine-tuned model should perform better. Of course, research isn’t meant to always align with my hypothesis, but I do want to be sure this conclusion is as truthful and as accurate as possible, since it did raise some suspicion. These are the three steps I am taking this week to ensure this:
I discovered an inconsistency in the settings of the two models: temperature. In AI, temperature is a setting that determines the ‘randomness’ of an AI response. Lower values produce more focused and consistent answers, while higher values allow for greater variability. I discovered that my base model temperature was set to 0.2 by default, but the fine-tuned model was set to 0.9 by default. To ensure a fair comparison, I adjusted the fine-tuned model’s temperature to 0.2 so that both models would generate responses under the same conditions.
I also added a phrase in my prompts that asks for a response under 150 words. My hope is that this will make the responses more concise and will eliminate any fluff that could be skewing the evaluation. Essentially, it should make the responses more focused and simpler for the judge to evaluate.
After making these adjustments, I regenerated the responses and am currently rerunning the evaluations.
3. Finally, my external advisor recommended that I reconnect with the two tabla students who previously assisted with this project. To further validate my methodology, I created a Google Form containing 14 prompts (10% of the total prompts) along with their corresponding ground truths. I asked the students to review these ground truths and confirm that they are accurate. This step will help ensure that the benchmark answers themselves are reliable.
In addition, I selected a different set of 14 prompts and presented the newly generated responses side by side, asking the students to identify which response is more accurate. This will allow me to compare the AI-based evaluations with human judgment and assess how well they align. To minimize bias, I randomized the order in which the responses were presented.
I am currently waiting for the students to fill out the form and look forward to sharing the results next week. Thanks for reading! See you next week as we make more progress toward the validity of my findings!
Week 10: Three-Step Verification
Hello, lovely readers! We’re back for Week 10 of my senior project.
Last week, I ran a few tests to be sure the evaluation data I collected was as accurate as possible. I did three things:
I ensured the temperature setting was the same across the base and fine-tuned models.
I regenerated model responses that were more concise than before.
I reached out to the two tabla students who’ve been guiding me through the musical part of the project.
This week, my goal was to analyze the impact of these changes. I’ll start with the first two.
After reevaluating responses generated under these controlled conditions, I found that the gap between the base and fine-tuned models became much smaller. In fact, both models earned an identical average accuracy score of 4.15 on the control group. This was an important result: it suggests that once variables like temperature and response length are standardized, the two models perform equally well on prompts outside my fine-tuning dataset. In other words, any differences I observe in the experimental group are much more likely to be due to fine-tuning itself, rather than inconsistencies in the testing setup. *sigh of relief*
As for the experimental group, the fine-tuned model had an average accuracy score of 3.01, and the base model scored 2.99. As I’ve mentioned before, it makes sense that the fine-tuned model would perform slightly better on material related to its training. However, I still need to determine whether this difference is statistically significant.
The third step I took last week, collecting insights from the tabla students, was fruitful in an unexpected way. I selected 10% of my prompts and asked the students to choose which model produced the more accurate response. My project uses LLM evaluation, but this step uses human evaluation. My goal was to see whether a human evaluator would agree with the LLM evaluation and whether a potential discrepancy would reveal flaws in my LLM evaluation process.
The results of talking to the students were split. Out of the 14 prompts I shared, the two students disagreed on the better response 5 times. For the remaining 9 prompts, they preferred the base model 5 times and the fine-tuned model 4 times.
This nearly 50-50 split shows me that, in the eyes of a tabla expert, there isn’t, honestly, much difference between the accuracy of the two models. It may seem like this isn’t a valuable insight, but it’s actually consistent with the LLM evaluation metrics I listed above (2.99 base, 3.01 finetuned). In other words, both human and AI evaluators seem to agree: the models perform very similarly in terms of accuracy.
With all this in mind, I spent a couple of days this week creating graphs to represent the LLM evaluation to use in my final paper and presentation. Here is a screenshot of the accuracy metrics, broken up by category. After further pattern recognition, these visualizations will reveal which categories each model scored better on.
If you’ve been following my blog since the beginning of the evaluation process, you know that I also assessed instructional quality, not just accuracy. The results for this metric differed more than I expected, so I’d like to spend more time interpreting them. I plan to discuss these findings with my advisor and share what I learn in next week’s post.
We’re officially in the final stretch!!
Week 11: That's All Folks!
Hello readers, and welcome to my final senior project blog post!!
It’s been a VERY eventful 11 weeks. My senior project experienced lots of setbacks, but also lots of progress. I definitely stepped out of my comfort zone with a project about AI and music, considering I know close to nothing about either subject, nor have I ever done a long-term research project like this! It’s safe to say I learned a lot about both aspects of my project and about the nuances of formal independent research.
My primary focus this week was to write my final product for the project. My final product was a paper detailing my project from start to finish, along with my findings, conclusions, and potential next steps. Since I haven’t talked too much about my findings and conclusions in my blog, I thought I’d share a few pages from my research paper for this week. Here they are:
Discussion
Target Domain Analysis
Ultimately, fine-tuning did not produce a significant improvement in either accuracy or instructional quality for tabla-related prompts. The following are proposed explanations for this observation.
Limited Data. The fine-tuning dataset contained 156 question-answer pairs compiled from online sources. This is relatively small compared to the scale of data used to train a base Gemini model. As a result, the additional training data may have had limited influence relative to the model’s extensive pre-existing knowledge. In other words, this level of fine-tuning wasn’t enough to significantly shift the LLM’s learned parameters. Additionally, the Vertex AI training metrics showed a training accuracy of 72%. While this represents an improvement from the initial ~55%, it indicates that the model aligned with the dataset only about 72% of the time, leaving substantial room for improvement through a larger or more diverse dataset. This is likely the primary contributor to the relatively low accuracy scores observed.
Overfitting. This is a common phenomenon in machine learning, in which the LLM ‘memorizes’ the question-answer pairs in the dataset, but lacks a thorough comprehension of generalizable patterns. This results in high accuracy scores for prompts closely aligned with the training data, but low scores for everything else. Since the 140 prompts in the target domain were not drawn verbatim from the dataset, lower accuracy scores may reflect the model’s limited ability to generalize beyond memorized examples.
Internal Conflict. This occurs when details in the fine-tuning material contradict the knowledge already encoded in the base model. In such cases, models often default to their prior knowledge, particularly when the fine-tuning dataset is relatively small. If such conflicts occurred, they could explain why the base and fine-tuned models produced similar accuracy scores.
Learned Response Style. Unlike with accuracy, in some categories, the fine-tuned model showed statistically significant improvement in instructional quality relative to the base model, suggesting that fine-tuning had a stronger effect on response style than on factual accuracy. Though the dataset did not explicitly define how different tabla concepts should be explained to a beginner, the information did carry a beginner-friendly tone and consistently reflected a clear and accessible instructional tone. It’s probable that the model implicitly learned this stylistic pattern during fine-tuning, resulting in responses that were better aligned with beginner-level communication. This would also account for the slight, though not statistically significant, differences observed in instructional quality within the OOD piano prompts. Exposure to beginner tabla instruction may have been generalized to a broader instructional style, meaning the model learned how to instruct a beginner piano student while being trained on tabla information.
Cross-Domain Analysis
A statistically significant difference was observed between the target domain (tabla) and the out-of-domain (OOD) piano set across both accuracy and instructional quality.
For accuracy, performance was higher in the OOD (piano) set. This suggests that the 156 question-answer pairs in the fine-tuning dataset were insignificant in comparison to a base model’s pre-existing knowledge of piano. In other words, the results support the interpretation that base models are more extensively trained on widely represented instruments such as the piano than on the tabla, and that the scale of fine-tuning in this study was not large enough to bridge that gap.
For IQ, performance was higher in the target domain (tabla). The fine-tuning dataset, compiled from beginner-oriented instructional tabla material, was rich in instructional nuances specific to tabla. These include hand placement, mnemonic syllables (bols), and practice tips, which aren’t as common in general piano theory data that the base model may have been trained on. Through fine-tuning, the model likely learned these nuances, which likely accounts for the higher IQ scores in the target domain compared to the OOD.
Conclusions
The results of this study support the following conclusions:
Within the constraints of this study (including a 156-pair fine-tuning dataset, a fixed prompt structure, and evaluation via an LLM-as-a-judge framework), fine-tuning Gemini 2.0 Flash had a more consistent impact on instructional quality than on factual accuracy. Specifically, fine-tuning improved the model’s ability to adopt a beginner-appropriate tone and incorporate tabla-specific terminology in certain responses. However, these improvements were not uniformly observed across all prompt categories.
No definitive conclusion can be drawn regarding the effect of fine-tuning on accuracy under these conditions. The relatively small size of the dataset and the moderate training accuracy (~72%) suggest that the fine-tuning may have been too limited to produce a measurable, consistent, and statistically significant improvement in factual accuracy.
Under the conditions of this experiment, fine-tuning with a small, domain-specific tabla dataset was not sufficient to achieve the level of accuracy observed for piano-related prompts using the base model. This suggests that the model’s pretraining likely included more extensive or higher-quality representations of piano-related knowledge. While this result is consistent with broader concerns about imbalances in training data across cultures and regions of the world, this study alone does not provide sufficient evidence to attribute the observed difference solely to disparities in cultural representation. However, it does indicate that, even after fine-tuning, Gemini 2.0 Flash demonstrates stronger accuracy on piano than on tabla within the scope of this evaluation.
Future Work
Considering the limitations of this study, particularly those related to time, dataset size, and computational resources, several extensions could strengthen and expand this work.
Dataset Expansion. Increasing the size and diversity of the fine-tuning dataset would likely improve both accuracy and instructional quality, particularly, incorporating a larger number of high-quality, domain-specific examples. Additionally, future work could explore multimodal training by including audio or video data, such as recorded demonstrations of tabla techniques. Because instrumental learning is inherently auditory and visual, relying solely on text limits the model’s ability to capture important aspects of performance and technique and limits a student’s ability to meaningfully progress. Adding different modes of media can help prepare the fine-tuned model for use by human tabla students.
Real-time Feedback. The current system is limited to text-based input and output, which limits human-like interaction. In contrast, human tabla instructors can offer real-time demonstrations and corrective feedback. Future versions of this project could integrate audio and video input, such as recordings of a student’s playing, and generate real-time feedback and encouragement.
Expansion to Other Instruments. In order to gain further insight into disparities in AI training data across different cultures and regions, this project’s framework could be expanded to include other underrepresented instruments. Comparing model performance across these instruments and more globally represented ones (e.g., guitar or drums) would provide a broader basis for evaluating how the origin of training data influences both accuracy and instructional quality.
Base Model Comparison. This project focuses on a single base model, Gemini 2.0 Flash. Future research could test and compare base models developed by other companies, like OpenAI or Anthropic. This would help assess how training data differs across base models and how this affects each model’s cultural sensitivity.
It is important to emphasize that, even as this work develops, an AI-based tabla tutor is not intended to replace human instruction. The tabla carries deep cultural and religious significance, and certain aspects of learning the instrument, such as the lineage system and stylistic nuances across regions, cannot be replicated by AI. As this project scales, the final product would be intended for use by students looking to get a high-level understanding of the tabla or refine technical skills without having access to a human teacher.
Post 11: Reflections
Hello everyone!
This is the last week of my senior project timeline, which I’m using to prepare for my presentation this weekend. I wanted to share some quick reflections on the project.
This project was a great chance for me to get a glimpse into what I’m going to be studying in college. My current major choice is Artificial Intelligence, but I hadn’t had any thorough exposure to the topic prior to this project. There were so many concepts I had to learn from scratch, like fine-tuning, and every platform I used in this project was completely new to me. Not only do I feel more confident in my ability to study and research AI in college, but I’m also proud of myself for stepping out of my comfort zone.
Also, this was my first time doing research, so I owe a huge thank you to my two advisors, Lu and Ms. Jay, for catching the flaws in my research method and providing advice to fix them.
This was also my first time analyzing this much data. I kind of regret not taking AP stats, but I did figure it out by researching online, and I’m glad I was able to work it out in time and understand enough to put together a research paper.
My original plan for this project was to track my tabla learning, but I received advice during my proposal defense that this wouldn’t be a sound research method. As such, I didn’t learn as much about tabla as I originally had hoped to. I definitely still want to learn how to play the instrument at some point, so I’ll keep an eye out for workshops or intro tabla classes in college!
Though most of my findings were inconclusive and my hypothesis couldn’t yet be confirmed or denied, I still take pride in how much work I did do. If anything, the inconclusive outcome gives me more clarity on how to continue this project in the future. I still had enough findings to put together a quality research paper and confidently say that I learned something, so I’m happy about that.
The findings of my project are aligned with the concern that AI training data overrepresents Western cultures. This is something I’d like to continue looking into through further AI research in college. I truly believe this is a concern worth addressing as AI gains more widespread use all around the world. It should be trained to serve people of all backgrounds equitably and with adequate cultural sensitivity.
To close, I’d like to thank my advisors once again, Lu and Ms. Jay, for their time, guidance, and constant encouragement for the past nearly 9 months. I’d also like to thank Mr. Cunningham for his management of the senior project program and for providing all the necessary resources to complete the project smoothly. Finally, I’d like to specially thank the three musicians, Archita, Shruti, and Abhishek, for their help validating the tabla-related steps in my project. I really appreciate their volunteer time and interest in the project.
Linked below is my final research paper for anyone interested. Hoping to connect at my presentation this Saturday at BISV at 10 AM!
https://docs.google.com/document/d/1uiFyV-q0AT-sVFH6QMXOl2eYHj-hUt4_Bufv-5Hh9xA/edit?usp=sharing