Blog 6: Continued Research

Welcome back to my blog! Last time, I introduced some of the more basic math behind LDA and covered how LDA would ideally function in my project. Unfortunately, for several reasons, the direct application of LDA that I initially wanted to use in this project is not possible. So today’s blog post is all about why that is and what I’ve been working on to improve my LDA model’s ability to discriminate between specimens.

Nearly all the issues with direct LDA implementation in my project can be traced to the nature of reflectance sampling. Unavoidable discrepancies in light saturation result in random variation in absolute reflectance between readings, and the use of the first derivative to eliminate that issue results in extra noise in the measurements. For me to use LDA effectively despite these issues, I have been forced to be much more selective with what wavelengths I can analyze. 

As you may recall from my last post, each reflectance measurement at a wavelength is a variable fed into my LDA algorithm. This means that even if I only consider looking at wavelengths from 1000 to 2500 nm, I would still have to feed my LDA algorithm 1500 unique variables, expecting the function to then rank and return 1500 “weighting” coefficients. This, especially in files with very little variation at certain wavelengths, only exacerbates my issues with noise. To mitigate this issue, it is necessary to drastically reduce how many variables are fed into the algorithm and to be much more selective with what wavelengths are chosen. Although I am still in the process of picking the correct wavelengths to focus on, Dr. Fine and I have discussed using LDA a second time to first identify which wavelengths are the most important to separation via their weighting coefficients and to then perform the actual LDA analysis using only the most significant wavelengths. This idea, however, is much more difficult to implement than we originally thought, so with the help of Stephanie, we have been developing a simplified algorithm for selecting significant wavelengths before conducting the LDA.

I will share my last updates on the success of this pre-LDA algorithm in my next and possibly final post! I hope you’ve enjoyed learning about my senior project and will tune in for my last post!