Week 9: Are You Sure About That??

Good morning readers! Last week, I finished off the evaluations for my project. I began analyzing the numbers this week, but found something shocking: the base model had a higher accuracy average than the fine-tuned model. 

This was surprising because the fine-tuned model has more tabla-related data than the base model, and both are being evaluated against the data on which the fine-tuned model was trained. So ideally, the fine-tuned model should perform better. Of course, research isn’t meant to always align with my hypothesis, but I do want to be sure this conclusion is as truthful and as accurate as possible, since it did raise some suspicion. These are the three steps I am taking this week to ensure this:

1. I discovered an inconsistency in the settings of the two models: temperature. In AI, temperature is a setting that determines the ‘randomness’ of an AI response. Lower values produce more focused and consistent answers, while higher values allow for greater variability. I discovered that my base model temperature was set to 0.2 by default, but the fine-tuned model was set to 0.9 by default. To ensure a fair comparison, I adjusted the fine-tuned model’s temperature to 0.2 so that both models would generate responses under the same conditions.
2. I also added a phrase in my prompts that asks for a response under 150 words. My hope is that this will make the responses more concise and will eliminate any fluff that could be skewing the evaluation. Essentially, it should make the responses more focused and simpler for the judge to evaluate.

After making these adjustments, I regenerated the responses and am currently rerunning the evaluations.

3. Finally, my external advisor recommended that I reconnect with the two tabla students who previously assisted with this project. To further validate my methodology, I created a Google Form containing 14 prompts (10% of the total prompts) along with their corresponding ground truths. I asked the students to review these ground truths and confirm that they are accurate. This step will help ensure that the benchmark answers themselves are reliable.

In addition, I selected a different set of 14 prompts and presented the newly generated responses side by side, asking the students to identify which response is more accurate. This will allow me to compare the AI-based evaluations with human judgment and assess how well they align. To minimize bias, I randomized the order in which the responses were presented.

I am currently waiting for the students to fill out the form and look forward to sharing the results next week. Thanks for reading! See you next week as we make more progress toward the validity of my findings!