Week 9: Validating the Results

Having the model and dosage table, I’ve come to the next most important step of this project, validating my results. Building the model is one thing, but demonstrating that it actually works is another, and arguably the most important scientific contribution. This week was dedicated to validation, specifically testing whether my PD model and dosage table held up when challenged against unknown data.

Unfortunately, I could not find another entirely independent dataset to perform my validations against. Instead, I performed something called “leave-one-study-out” cross validation, or LOO. The idea is simple: instead of fitting the model on all of my datasets, I take out one dataset, refit the model on the rest of the datasets, then validate that refit model against the dataset that was taken out. If the model performs well on data that it was never trained on, that is evidence of generalizability.

Overall, the results for the PD model were positive. Across all 4 holdout iterations, the model consistently reproduced the PD model with R^2 values close to the full mode’s value of 0.941. The EC50 and gamma parameters both stayed within a narrow range through all holdouts.

For validating the dosage table, I again used holdout validation, but instead compared the resulting dosage tables against the full model’s dosage table to see how far it differed. The results were quite positive, especially for target effects between 20%-80%, which had very small standard deviations, typically around 1-2mg. The 90% row showed the most variability across LOO iterations, with a standard deviation of nearly 5mg. However, this is again likely due to extrapolation, as during training the model only has data up to 80% subjective effect intensity to work with.

All in all, the validation results gave me reasonable confidence that both the PD model and dosage table reflect a reliable pattern in the data.

Next week I will be focusing on refining my project and creating my presentation,