Spectroscopic Drug Detection & the Fight Against Fentanyl
Advisor: Ryan Grove
Internship Location: Mitre
Onsite Mentor: Dr. Casey Smith
According to the National Institute on Drug Abuse (NIDA), in 2022, Fentanyl overdose was responsible for the deaths of roughly 80,000 Americans; and that number has been rapidly growing due to increasing amounts of fentanyl smuggled illegally across the US border each year. Fentanyl is a type of synthetic opioid with over 50 times the potency of heroin and over 100 times the potency of morphine. In fact, the Drug Enforcement Administration(DEA) indicates that only 2 milligrams of Fentanyl can serve to be a lethal dose. Its cheap production costs coupled with high toxicity has made it a common component of counterfeit pills. Fentanyl is also found in various street drug samples as a cutting agent for other drugs such as heroin or cocaine. In this way, many fentanyl overdoses can end up being accidental. NIDA highlights that Fentanyl overdose has taken more lives compared to any other drug, making it the center of a national drug health crisis. The fight to keep fentanyl out of American lives starts with preventing illegal fentanyl entry into the US. In recent years, border security has confiscated upwards of 22,000 lbs of fentanyl at the Southwest border alone(DEA). Despite this number being quite large, a lot of Fentanyl continues to go undetected. My project will explore advanced chemical analysis techniques that can be used to detect Fentanyl with a higher accuracy and precision in hopes of evaluating possible improvements in fentanyl detection. The MITRE Corporation is currently working on developing their own advanced sensing technologies in this field. My internship with them provides me with first hand experience with the process of developing an effective drug detection technology. This on-site experience coupled with my independent study/review of spectroscopy techniques in the field of drug detection will allow me to ultimately conclude on the ways we can improve detection for Fentanyl, and potentially other harmful drugs, in the future.
Project Blog
Week 1: Welcome to MITRE
Hello everyone and welcome to my first blog post!
My name is Natasha and my senior project is going to focus on chemical detection-specifically for fentanyl! As some of you may know, fentanyl overdoses have been on the rise in the united states, making it a national crisis. My project will focus on the different chemical analysis tools like, Infrared Spectroscopy or Raman spectroscopy(more to come on these methods in future blog posts!), that may be a good way to detect them in order to prevent their entry into our nation!
This week was week 1 of my internship at The MITRE Corporation where an incredible team of researchers are doing hands on work with the analysis tools I briefly mentioned above!
After being at BIM for 6 years, its easy to forget how it can feel like to start fresh at a new place, but my first week at MITRE has felt incredibly welcoming and exciting!
Starting with orientation on Monday, I set up all of my MITRE accounts and got a badge(which made me feel very official!) On Tuesday, I went to my first ever team meeting, and it was so cool to see everyone collaborating and sharing their thoughts/findings on their research! Everyone on my team(and just generally at MITRE) is so incredibly insightful(and kind!!), and I'm very excited to get to know them and their research more! Tuesday ended off with some preliminary lab training and the rest of the week consisted of catching up on some background literature and shadowing my mentors in the lab while I wait to finish up trainings!
Throughout the week I've honestly learned so much about chemical analysis and the exciting research happening here at MITRE, and I'm more excited than ever to learn more and contribute! This first week has been AMAZING and I can't wait to keep you guys updated on everything to come!
Until next time,
Natasha
Week 2: The IMS That Cried Wolf
Hi everyone and welcome back to my blog!
My second week at MITRE seemed to mark the return(hopefully!) of warmer weather with Tuesday being in the 70s! Daylight savings switched over the weekend too, which was a bit of a struggle to adjust to, but nothing that a little trip to the coffee shop in MITRE 4 couldn’t fix!
As for my actual work, during my internship at MITRE, we will be focusing on different analytical tools used for chemical detection of trace substances in mixtures. The star of this past week has been a device called the Ion Mobility Spectrometer or IMS for short, which basically works by separating the different ions in a mixture based on their mobility through a ‘buffer gas’ in response to an electric field.
For those of you that are frequent flyers, the IMS is actually used for trace detection of controlled substances at airports!
The first step an IMS takes once it gets a sample is to ionize the molecules. This can happen in a lot of different ways depending on the sample’s state. A common technique includes electrospray ionization (which was used in an IMS study I read(Fernandez, 2009)). Once the molecules are ionized, an electric field pulls them through a drift region (filled with the buffer gas). The different types of ions in the mixture will travel through the drift region at different speeds. The IMS typically measures a variable known as drift time(dt) which measures how much time each ion took to travel through the drift region. The device will return certain peaks (that represent the drift times it recorded) and based on those different peaks, scientists should be able to identify which substances were in the mixture. If you’re interested in learning more about how the IMS works, you can check out this link: Ion Mobility Spectrometry: Fundamental Concepts, Instrumentation, Applications, and the Road Ahead - PMC
I got a chance to see an IMS in action this week as Casey and Nate(the lead and co-lead of the project I’m interning for) played around with it to see how it reacts to different compounds and mixtures. The IMS came with different traps (rough pieces of paper like material that you insert into the IMS for it to read). Before starting to sample, we had to enter a calibration trap (which was a strip with a certain calibrant on it) and a verification trap. Once that was set, we were ready to start testing with samples! Casey and Nate put tiny amounts (and I mean super tiny-like maybe half a milligram!!) of powder on glass slides and gently brushed a sampling trap across it. Sometimes, the trap would have such a tiny amount of powder on it that it wouldn’t even be visible! They fed the sample trap to the IMS and within a few seconds it would give us the peak readings. The IMS is super sensitive (which is how it’s able to process such tiny amounts of a sample), so they had to be extra careful to prevent any kind of contamination!
The specific IMS we were working with has a built-in detection system for certain controlled substances, so if it detects a peak that’s close enough to the peak of any of the substances in its system, it should play an alarm indicating that the sample contains controlled substances. One thing we learned early on is that even if the sample doesn’t contain any controlled substances, there’s a chance that the IMS will detect one anyways-kind of like crying wolf! This probably happened because the drift times of the substances we were testing ended up being uncomfortably close to the drift times of the substances it’s trained to detect.
Other studies that have worked with the IMS, specifically the Pacific Northwest National Library’s Performance Assessment of Field Portable Instruments and Assays for Fentanyl and Fentanyl Related Compounds (Bradley 2023), have found similar issues with the IMS being prone to false positives, and we definitely got to see just how true that is! This trait does makes me wonder how they deal with the false positives issue during airport scanning. In addition to the false positives, we also noticed that the IMS may potentially have an issue with consistency in its readings for the same sample. These are definitely important things to consider when assessing the IMS’ feasibility for on field trace detection of drugs.
Looking forward to next week, I finally completed my lab training so hopefully I’ll be able to do some hands-on work in the lab! I’ve learned a lot about lab procedures by watching Casey and Nate work this past week and I’m excited to put what I’ve learned to use!
Until next time,
Natasha
References:
Bradley, A., Mo, K. F., Melville, A., Sanders, D., Jensen, S., Lin, V., Mobberley, J., Doll, C., Orton, D., Heredia-Langner, A., & Ozanich, R. (2023). Performance Assessment of Field Portable Instruments and Assays for Fentanyl and Fentanyl Related Compounds. Pacific Northwest National Library. 20240105_FeRS-Final-Report_CLEARED Public Release
Fernandez-Maestre, R., & Hill, H. H., Jr (2009). Ion Mobility Spectrometry for the Rapid Analysis of Over-the-Counter Drugs and Beverages. International Journal for Ion Mobility Spectrometry. https://doi.org/10.1007/s12127-009-0025-x
Week 3: Rainbows in the Lab
Hi everyone! Welcome back to my blog!
I’ve just completed my third week at MITRE, and I must say, before this internship, I wasn’t necessarily a big tea/warm beverage person, but the free tea employees get here has definitely transformed me! I’ve come to learn that one of the coziest ways to start weekday mornings is with a little trip with your colleagues down to the coffee shop to pick up a warm tea, and I look forward to the many more tea trips to come!
Making tea isn’t the only hands-on activity I’ve been doing during my time here at MITRE! This week was my first week with lab access, so I was finally able to put the lab skills I learned from observing Nate and Casey to use! The star analytical tools of this week were the IR and Raman spectrometers which I actually got to test samples with! Before I tell you about using them, let me explain a little bit about how they work.
IR and Raman spectrometers are types of optical spectrometers, which means they work by measuring a samples response to a certain laser. The IR spectrometer measures the shift in a molecules vibrational state in responses to infrared radiation. Vibrational states is kind of a fancy word, but the overall gist is that certain molecules have a set amount of energy that they can absorb, and if the radiation of the laser matches that energy, then the molecule (or more specifically, the bonds in the molecule) will absorb that energy. If you blast different wavelengths of IR radiation at a sample and identify which ones it absorbed and which ones it didn’t, you could get a pretty good idea of the types of bonds present in the molecule, and potentially even the molecule’s identity itself, and that’s basically what an IR spectrometer does! After testing the samples’ absorbance over a certain range of wavenumbers (which are related to wavelengths), it produces a spectrum, with certain peaks that indicate which wavelengths it absorbed and which ones it didn’t. The spectrum is usually represented in units of wavenumbers (which is the inverse of wavelength). When it comes to analyzing this spectrum, most researchers like to split the spectrum at around 1500 wavenumber. Anything to the right of that (1500-500) is what researchers call the fingerprint region which shows a spectral pattern unique to the substance itself. The spectrum to the left of 1500 wavenumber is the functional group region, meaning the patterns can be traced to certain functional groups within the molecule, but cannot provide definitive information about the identity of the molecule itself. One type of IR spectroscopy is transmission IR spectroscopy (where they shine the laser directly through the sample and measure what wavelengths come out the other side and which ones don’t), but for our project at MITRE, we’re using what’s called Attenuated Total Reflectance (ATR) IR spectroscopy! This essentially measures the same thing; it just uses internal reflectance to do so. Basically, the spectrometer will shine a laser at something with high internal reflectance (the one we have uses a diamond). The light goes through the diamond, reaching the surface and then bouncing back out of the diamond and towards the detector. When we put a sample on the surface of the diamond, some of that light will be absorbed, and that absorption is what the spectrometer will measure.
The Raman spectrometer is kind of like a sister method to the IR, but instead of measuring absorption, it measures scattering. The Raman spectrometer will shoot light of a certain wavelength towards the sample. Some of the photons in this light will basically “scatter” off the sample, but sometimes this scatter is not perfectly elastic, a phenomenon which researchers call Stokes scattering. When this occurs, the sample molecule will be excited to an elevated state but then relaxes to a state that is higher than its original state (because it took some energy from the photon). As a result, the photon that collided with it will scatter off with hv-∆E, where hv represents the proton’s initial energy, and ∆E represents the energy difference between the initial and final state of the molecule. By measuring the scatter of different wavelengths, the Raman spectrometer also creates a spectrum in which different peaks can be characteristic of different substances in the sample. If you want to learn more about how Raman works, you can check out this link: 18.1: Theory of Raman Spectroscopy - Chemistry LibreTexts (its actually very interesting so I highly recommend!)
These techniques are pretty cool, right! But there is a caveat---because the signals that the Raman and the IR measure can be so delicate, they aren’t the best tools to analyze mixtures and determine components with a concentration that’s less than 5%. This could be an issue for detecting drugs that are smuggled in trace amounts in mixtures with other cutting agents.
Now for the exciting part- working hands on with these very cool instruments! My week started with the ATR-IR spectrometer, and collecting spectra on different w/w% samples for further evaluation. The process was pretty simple- I cleaned off the diamond, placed a little amount of the powder mixture onto the diamond, and gently lowered the press to make sure the powder was pressed up against the diamond as much as possible. Then, I clicked run on the connected computer, and in 2-3 minutes, a little squiggly line would show up with peaks at different wavenumbers. We collected three spectra for each sample and tested 12 different w/w% for each mixture, so in a given day, I ran about 36 samples on the IR! Each sample showed up on the screen as a different colored line, so by the time I got to the last sample in my set, the screen was filled with rainbow colored spectra! I also got to work with the Raman spectrometer, which was super cool because it had the ability to test samples through a plastic bag, so I didn’t have to handle the actual powders at all! I basically just had to fasten the bag with the sample against the laser, and it was ready to go! The Raman took about 5 minutes to analyze each sample, which felt kind of long at times compared to the 2-minute IR time, but it was intriguing nonetheless! The spectra I collected are off be run through an algorithm that will analyze the peaks and try to identify the components of the mixtures based off them!
As you can tell, my first week working in the lab at MITRE was amazing! Can’t wait to share more of my lab time with you all next time!
Thanks for reading!
Natasha
Week 4: Branching Out
Hi everyone, and welcome back to my blog! I’m finishing up my 4th week at MITRE, and this week has by far been one of my favorites! I continued to do a lot of the same lab work as last time; although, this week, I got to try weighing samples (which is actually a lot harder than it looks!). To make the mixtures that I’ve spent the last few weeks testing, you have to measure pretty tiny amounts of powders in a fair amount of accuracy! For one of the samples, we had to measure 50mg of a certain powder, and I guess I never really understood how small 50mg is until I put my first scoop of powder onto the scale and watched it go directly up to 64mg(whoops!). It’s safe to say that I’ve learned my lesson-tiny scoops are always the way to go! The rest of the samples went a lot better, and I actually got the measurements pretty precisely on my second go without going over! After we finish making these samples next week, we’ll get a chance to run them through the IR and Raman, collecting even more data!
This week I also got a chance to branch out and talk with some other members of our sensing project that don’t necessarily work in the lab like Casey and Nate! Instead, they work on developing algorithms/methods to process the spectra that we’ve been collecting. Essentially, the problem is that spectroscopy is a really powerful tool for identifying chemicals, especially when they’re pure---but when they’re in a mixture, their spectral signals can start overlapping, so it can be really hard to tell which chemical exactly each peak is referring to too. This problem gets worse when the mixtures start getting down to trace amounts (like 5% of one substance mixed with 95% of another) where you might see the signal from the trace substance almost entirely drowned out by the signal from the diluent. Deconvoluting these mixture spectra and trying to recognize slight patterns and variations that identify the presence of trace substances is currently a big area of research in the field of spectroscopy! Until now, I hadn’t been too familiar with this side of the process, but I got a chance to sit down with the other members on our project, Brendan and Tana, and I learned a lot!
Brendan is an AI engineer and his role in our project is to create an algorithm that can make inferences (based on data that we’ve fed it containing what different spectra look like for different pure substances) on an unknown spectrum and see what combination of known spectra would make a spectrum closest to that. For example, say you were trying to identify a spectrum of 50% substance A and 50% substance B. The algorithm currently knows the spectrum for the pure substance A and the pure substance B in its library, so it would try different combinations of A and B(maybe it would start at 10% A/90% B and then self corrects to 20/80, etc.) until it hopefully gets near 50/50 which should give it a match for the unknown spectrum it’s trying to analyze. The more scientific name for the process I’ve described here is a linear regression model! This was something I kind of recognized from my math classes at school (use this as a sign to lock in during your math classes!). Linear regression can be a form of machine learning that uses training data to create a function (a straight line) that represents the relationship between your parameters(x) and the output you want(y)-basically making a line of best fit. But, you know how in our math classes or on AP practice exams when the question would tell you to “draw a line of best fit through the points” and you would just ‘eyeball it’? Well, the computer doesn’t have that ability…so it actually has to come up with a line of best fit from the data points its given(training data) that minimizes the mathematical value of the loss. Loss is just the difference between the value predicted by the line of best fit, and the actual value from the training data. Remember, the line of best fit looks kind of like y=mx, and the point is to find the m that minimizes loss (which is represented by the function ∑(y’-y) where y is the predicted value and y’ is the actual value). This usually involves differentiating the loss function with respect to m and then adjusting m based on the result of the derivative and a certain increment (learning rate, decided by the researcher)-gosh I feel like such a math teacher. Anyways, this process can be hard when you only have one parameter that you’re using to make your prediction but imagine how tedious it would be to deal with 2, or 5, or 100 parameters! For the model Brendan is developing, we are using upwards of 1,000 parameters, which is typical for most linear regression algorithms nowadays, but don’t worry, the computer can more than handle it! Now that you’ve read through how the model works, let’s apply it to the spectra that we’re collecting.
Once the spectra are collected, we feed it into the algorithm, and the first step the algorithm takes is to “vectorize it.” The algorithm breaks the x-axis(wavenumber) of the spectrum into a thousand little groups and records the value of the spectra at that group to transform the spectrum into numerical data points. Each little group represents a single parameter, and by examining multiple spectra in its library, the computer will create a function that can map out how much of substance A is in an unknown spectrum based on its values at certain parameters. Pretty cool stuff right! Now, this works best for things that definitely always combine linearly in the same way, but spectra can sometimes have variation that can throw these kinds of models off, so naturally, they’re never 100% accurate all the time. Also, in spectroscopy, the difference in the spectrum details between 1% and 0% concentration of a substance can be almost indistinguishable, so for the model, it can be incredibly hard to tell them apart! That’s why these kinds of models can sometimes experience false positives or false negatives when dealing with such tiny concentrations!
Brendan was excellent at explaining all of this, and it’s been really fun getting to dip my toes into the world of AI and machine learning. Brendan offered to help me get my computer set up at some point in the coming weeks so that I can hopefully contribute to this side of the project too, which I’m very excited about!
As for my conversation with Tana, she’s an optical engineer that works with lasers and spectroscopic devices (like Raman and IR), so she’s really knowledgeable in the field we’re working in. She also taught me a lot about the laser side of things, which I will be happy to share with you all next week!
It’s been fun branching out and speaking to different people about the unique work that they’re doing for our project! Not only did I learn more technical stuff, but we also got a chance to just chat about life and their experiences going to college, getting a degree, and navigating their career so far. I’ve learned a lot, and I’m excited to continue doing so!
Thanks for reading (I know it was a lot!)
Natasha
Week 5: Pythons and Pandas
Hi Guys, and welcome back to my blog!
Week 5 of my internship has been the most exciting one yet! This week I got to try my hand at actually making my own linear regression model and using it to analyze the data that I’ve been collecting these past couple of weeks! On Tuesday, I got a chance to meet up with Brendan again, and he helped me get python installed on my computer! He gave me a couple of tutorials to explore so that I get used to the syntax and how everything works, and he also gave me a very useful textbook that walks you through the different types of machine learning models and an example of how they can be implemented on python! I learned that python actually has a lot of very useful data analysis packages that come in handy for doing linear regression! Certain python packages (like pandas) allow you to be able to read data from the data files generated from the spectrometers, and then input that data into a function that will automate the linear regression process for you until it finds the best coefficients to create a model! Coding this process went pretty smoothly and the only major problem I ran into was making sure the data tables were in the right dimensions! It was also a little tricky to find a way to determine an appropriate set of training vs testing data. Basically, when creating a machine learning model, researchers often set aside some real-world data to actually train the model(in our case, this data is what gets used to create the coefficients in the linear regression function), and the rest of the data they get will be used to test the model(by comparing the models predicted value vs the actual real-world value of the given data point). Ultimately my model worked ok and showed results similar to what Brendan had been showing us in the past few sync meetings. It turns out that we’re seeing a high level of accuracy for detecting substances at a concentration of 50% or higher, but it can, at times, get tricky when we try to pinpoint concentrations under/around 5%.
To address the ‘roadblock’, Brendan suggested that we get more data to test the model so that we can confidently assess its performance. Casey and I spent all day on Thursday in the lab preparing and analyzing samples! I ran both the IR and the Raman at the same time (which definitely helped it go by faster!) I think we got a total of 8 samples analyzed 3 times each by both the IR and the Raman, which was a lot! Hopefully we’ll see some more trends in the model’s performance once we tests these samples!
Week 6: The Dance of the Spectrometers
Hi everyone! I’m just finishing up week 6 of my internship, and things are really picking up the pace around here! To help Brendan with the machine learning model, we’ve been on a roll collecting data in the lab. Since I’m more comfortable with the IR and the Raman spectrometers now, I’ve been able to run them at the same time which helps a lot to collect as much data as possible! The IR takes about 3 minutes to complete its analysis and the Raman takes about 5, so I’ve kind of learned the balance between when exactly I should switch between the IR and the Raman. My process is to first run a background scan on both devices. The IR will finish before the Raman so then you start the IR’s first sample. You run it 3 times like normal, and then while the third scan is running, you transfer the remaining sample into a plastic bag and start running it through the Raman. Once the first sample is in the Raman, you go back and start the next sample on the IR. And then once the IR scan is about 70% done, the Raman scan should be complete so you can head back over and start the Raman’s next scan. And then you keep dancing back and forth between them and before you know it, you’ll have finished collecting all your samples on both devices!
I’ve spent most of my mornings collecting data in the lab, but my afternoons have been filled with machine learning and lots of textbook reading! For my individual project, I’ve been exploring different types of models that might be useful for analyzing spectral data and detecting trace amounts of substances. This week, I came across a model known as Linear Discriminant Analysis(LDA), and the math behind it is incredibly complicated and may have gone over my head just a little, but the gist is that this model will supposedly create a ‘decision boundary’ based on the parameters you give it during training. When it receives a new data set, based off how the new parameters compare to the decision boundary, it will classify the data set as belonging to a certain category. LDA is more of a classification-based model compared to linear regression which means that instead of identifying exactly what percentage of a substance is in a mixture, LDA is better for classifying the mixture as containing substance x, substance y, etc. The way I decided to interpret LDA for the identification of trace substances is to set the two ‘classes’ as either “contains target chemical”(which I denoted as “true”) or “does not contain target chemical”(which I denoted as “false”). So I ran it with the data we collected, changing the target chemical each time I ran it, and the results were surprisingly good! It has almost a 100% accuracy, which actually worries me a little because this could mean that its overfitting to the training data. Basically, since the model is trained on mixtures that looks extremely similar to the mixtures that are in its test data set, its very possible that it works well for these mixtures that are similar to what its already seen and has a poorer performance for mixtures that it hasn’t seen before. I’m interested to test it out with some new mixtures as well, and we’ll see how that goes.
Thanks for reading!
Week 7: Is it Normal to Not Normalize?
Hi Guys!
It’s week 7 of my internship and I have spent this week coding away! I shared some of my work with LDA with my team and they pointed out some really useful things for me to test out, like normalizing the data, or specific patterns around when the model seems to fail.
The work I did with LDA last week was mostly with the raw spectral data from the spectrometers. However, in a lot of machine learning model development, researchers often preprocess their data! By ‘preprocessing’ I mean doing things like normalization, standardization, dimensional reduction, etc. These things can help for a variety of reasons! For instance, certain models might struggle to analyze data with over 100 dimensions, so it might help to reduce their dimensions with a technique like PCA. Some models work off the assumption that the mean of the data is 0 and the standard deviation is 1, so it might help to standardize your data to meet that expectation. For my work with LDA, following the suggestions of my team in our last meeting, I tried to normalize the data before running it through the model. The thought behind this was that sometimes, the spectra for the same samples might have variations in their peak intensities, so normalizing each spectrum to a unit vector would help a linear model better handle those variations.
I normalized the data and initially, the model sucked. But then I realized that I forgot to also normalize the test data matrix, so after I did that, its performance seemed a lot more reasonable! I noticed that when using the raw data, if the model had been trained to detect X from the spectrum from a sample of X and Y at 5% and 95%, it would not be able to detect X in spectrum from a sample made of 10% X and 90% Y. Once I started normalizing the data, however, the model started doing a lot better with this task! Unfortunately, some significant positive aspects of this model were lost along the way. Previously, if the model had been trained on a 5/95 w/w% mixture of X and Y, it would be able(with fairly ok accuracy) to detect a 5/95 w/w% mixture of X and some other component Z(given that it has at least seen Z before in some other sample, not necessarily mixed with X though). Once I started normalizing the data, the model started really struggling with these kinds of tasks. Pre normalization, it was also very confidently able to detect a slightly different (because of sample-to-sample variation) spectral reading of 5/95 w/w% X and Y if it had previously seen another spectrum of 5/95 w/w% X and Y. This ability also significantly decreased after normalization. The thing is that the overall variance of the model’s performance from test set to test set decreased with normalization, but the overall performance of the model decreased too. Considering these facts, I think that maybe the model was better using just the raw data! But it’s strange because usually using normalization to simplify the data is useful for a model that works around a fairly basic linear model, like LDA. As I write this, I’m wondering why this might be happening and my current guess is that it might have something to do with how each sample is getting normalized individually, so the specific aspects of one sample relative to the other that the model uses to differentiate between the two classes might be less prominent now?? The importance of the relative aspects of one sample to another might
After these results, I tried to standardize the data instead and that made the results significantly worse in every aspect which is again pretty unexpected. I might try to look into other ways to normalize the data-like min-max normalization, or z-score normalization to see if they can help create more of a middle ground between the raw performance and the unit-vector normalization performance! I’ll let you know how it goes!
As for my lab work, I’m continuing to fly through samples on both the IR and the Raman which means more data to train and test with!
Thanks for reading!
Natasha
Week 8: It’s Definitely Not Normal to Not Normalize
Hi Guys! I’ve just wrapped up week 8 of my internship, and this week was incredibly productive! The week started with me playing around with my models some more---testing them with different data sets, trying to get a better understanding of how it’ll perform! I tried playing around with the normalization technique after last week’s strange struggles with normalization and training/testing the model with different sets of data too see how the performance would vary with each test! I mentioned this briefly in the last post too, but turns out that LDA has a pretty high ‘variance’- a term in the machine learning world that indicates how closely a model’s performance is tied with its training dataset! The higher the variance of your model the lower your ‘bias’- the term that signifies the natural error that one can expect by attempting to simplify complex real-world relationships into simpler models. Au contraire, the lower the variance of your model the higher the bias! This variance-bias tradeoff is a pretty important concept in machine learning and relates very closely to the problem of over/under fitting! Typically, classification models, like LDA, have a higher variance since they define their decision boundaries based very closely on the data that its been presented! Meaning that the data that it has been presented to train on changes, than those decision boundaries could change pretty significantly! I saw this a lot when I tested my LDA model with different training/test data sets, especially since our datasets are smaller than typical classification model datasets that could contain upwards of 1000 data points! For example, if the model was only trained on binary mixtures, it would be pretty good at identifying binary mixtures with similar compositions to the ones it had seen before. However, it struggled A LOT with classifying tri-part mixtures, even if the percentage of the target chemical was something it had seen before. It also started confusing its binary mixture classification as well. When I normalized the data, it seemed to make the ROC curves that I was using to assess its performance worse, which wasn’t normal, so I met with Brendan to see if we could figure out a way to make this problem better! For the tests in which I normalized the data, the model was trained on binary mixtures and tested on tri part mixtures which might have been the reason it looked like normalization made it worse. Future note to self: if you’re going to make an adjustment to your method-like normalizing the data-keep the test/training sets constant before and after to see the actual result of normalizing! We found that normalizing the data and including a significant amount of tri-part mixtures (at least 3 mixtures (i.e. 9 tri-part spectra)) in the training data set seemed to really help with the overall classification skills. The area under the ROC curve went from 0.5 to around 0.8 for each of the targets, which is a big improvement! Although, the LDA’s high dependency on its training set may make it harder to work with for on field detection when it’s bound to come across cutting agents and percentages it’s never seen before!
I also tested out another model called Support Vector Regression(SVR) that is kind of like linear regression, but it introduces this concept of a ‘soft margin’-basically giving the model some set amount of slack for it to get some observations wrong to better fit the rest of the observations as a whole! Basically the ‘loss’ that this algorithm is trying to minimize looks a little different than in simple linear regression and depends pretty closely on the slack that the researcher decides to implement. When coding it in python using the scikitlearn package, this concept of ‘slack’ is denoted by the parameter C-and its usually best to find this parameter using cross-validation techniques (which scikitlearn also has many functions for). SVR can be implemented in different types of function too (like linear, polynomial, etc.) However, for this week I mostly explored linear SVR, and my results were certainly interesting! SVR seemed to perform better for mixtures that it had never come across before, which is not common for machine learning models! So far, SVR seems to have the most promising results of all the models I’ve been playing around with-it has the best ROC curves by far- so I’m excited to pitch it to our team!
Throughout the week I’ve developed a pretty good balance between lab work and computer work! I spend Mondays working on code, and then Tuesdays and Wednesdays working in the lab to get samples! I’m a triple threat in the lab now-prepping, running samples on the IR, and running them on the Raman! Honestly, I’ve come to really love my time in the lab! I feel super productive when I’m running through samples!
As I move forward, I’m probably going to start making a more coherent summary/presentation worthy description of all the model testing and development I’ve been doing for my final presentation!
That’s all for now!
See you next week!
Natasha
Week 9: So Much to Say but How Do I Say It?
Hi Guys! Welcome to week 9 of my internship! There are only a couple weeks left until our final presentation, so I’ve spent this week finalizing the models and actually formalizing all this work I’ve been doing into a presentation! I determined that I am going to be comparing the three models I’ve been focused on the most which include: Linear Regression, Linear Discriminant Analysis, and Support Vector Regression! Without spoiling too much of my final presentation: in order to compare them and assess how feasible they would be for in-field detection of trace narcotics, I came up with a criteria(and corresponding test sets) to assess them on, and will be running the models through the tests a couple of times and then analyzing the results to see which model is best! I’ve just finished running the tests and I have so many excel sheets and scatter plots and ROC curves just filled with data! This is my first time doing an actual ‘data-driven’ research project and I’m finding it just a bit tricky to determine the best ways to actually present my findings in a way that’s concise and easy to follow along! I’m considering doing some sort of statistical test to compare the model performances (maybe a t-test?) and seeing how that goes. I think I’m also going to look more into how other people in the machine learning world have been comparing model performances to see if I can get any hints from that!
Looking forward, I hope to have my slides finalized by Wednesday of next week and then my final presentation for my colleagues at work on the Tuesday after that! Time certainly seems to be flying but I’m so excited to share everything that I’ve been doing!
Until next time,
Natasha
Week 10: Putting It All Together with a Bow
Hi Guys! Welcome to week 10 of my internship! Not including this one, there’s only one blog post left in my senior project! It makes me feel a little bittersweet, but I’ll save the big (and potentially sappy) reflection and goodbye for next week’s post!
Before we get into this week’s work, I just want to acknowledge that on Wednesday of this week I started my morning off with a coffee. This might not seem like a big deal to most, but for me—someone who used to be pretty adamant about how much she doesn’t like the taste of drinking coffee—this was pretty significant. It feels like a coming of age almost. I started this internship drinking nothing in the morning, and then I began drinking tea in the first few weeks and now I’ve reached the level of coffee! I’m not really sure what’s made me progress through the levels of morning caffeine intake like this, but I’m sure it’s probably good practice for college life, right?!
Anyways, regarding the actual work I’ve done this week…I did some more research on how I can analyze/present my results, and I’ve decided to use two types of tests to help me out: a 2 sample t test and an ANOVA test. These two will compare the performances of each model and indicate whether they’re significantly different from each other. The 2-sample t test will be able to provide a bit more detail regarding which test is better than the other (if they are significantly different, of course). I’ve run through all the tests and put it into a slideshow, in a way that I hope doesn’t look too confusing! I’m currently at 27 slides which might seem like a lot for a 20-minute presentation, but I’m optimistic that I can make it work! I’ve sent it over to my team to get another set of opinions. Next week, I will be presenting my presentation to my colleagues at MITRE (outside of just my team), so it’s time to start prepping what I’m actually going to say! I’m a little nervous but there’s nothing that some practice can’t fix! I’ll let you know how it goes!
Until Next Time, (this is technically the last time I can end off my blogs like that lol)
Natasha
Week 11: The End of the Beginning
Hi Everyone!
Well, here we are…the final post in my senior project blog! I’m not the best with big concluding remarks (so bear with me!), but first, a quick recap of my activities during this last week:
On Tuesday, I presented my final presentation to my colleagues at MITRE! But before that, I sat down with my team on Monday and reviewed my presentation slide by slide! They gave me a lot of valuable feedback, like putting more graphics on my slides to highlight certain data points, or to maybe cut out some details on a certain context slide to save more time for the actual bulk of my own research! Peer review is a really massive part of doing professional scientific research, and I’m really grateful my team took the time to introduce me to what that process looks like (and plus, their suggestions helped out A LOT)!
After practicing my presentation a bit more on Monday evening, before I knew it, it was Tuesday afternoon, and I was in a conference room with my colleagues, telling them about the work I’ve been doing for the past 10 weeks! Overall, the presentation went really well! I didn’t stumble over my words too much, and I didn’t go way over time! I also got a few pretty good questions that I hope I answered well!
It felt really nice to show everyone all the work I’ve been doing and even though it may have been like a ‘final culmination’ of my project, it certainly won’t be the end of my journey exploring the professional research world!
Throughout this internship, I’ve learned more than I could have ever imagined! I’ve learned the proper procedures for preparing mixtures of substances, and proper lab safety requirements, and how to operate these really advanced spectrometry devices! I’ve learned about some fundamentals in organic chemistry that are harnessed in optical spectroscopy which are honestly incredibly fascinating (and I certainly hope to look into them more in college)! And of course, outside of the lab, I’ve gotten to immerse myself in this massive field of machine learning-something I was completely unfamiliar with before! I learned fundamentals on developing and training models, how different models can be evaluated, what the risks of machine learning are and how to be careful about them! I have gained SO MANY completely new skill sets from this internship, and I am beyond grateful for having had the opportunity to do so!
In terms of my personal growth, its safe to say that I have come out of this internship as a much more robust, analytical, and well-rounded person than I was on March 3rd! I’ve also learned how to network and connect with people, and I definitely plan on staying in touch with the wonderful scientists I’ve met during my time here!
And of course, it goes without saying, that I consider myself so lucky to have been able to work with my onsite mentor Dr. Casey Smith and the rest of our team! They have showed me what scientific research can look like in a professional setting and have been so very gracious in letting me explore the different parts of the project we’re working on! They were so supportive in everything I took interest in-whether it was meeting with me individually to help me troubleshoot my code or teaching me how to do chemistry in a lab! I couldn’t have asked for better role models and mentors throughout this project!
Ok, now that I’ve gotten through the big reflection, I want to say that even though this is my last blog, and my senior project is at its end, I certainly hope to continue the work I’ve been doing in this field throughout the summer, but also as I start with college! It really is just the end of the beginning, and I can’t wait to see where else this journey will take me!
Thanks for reading!
Natasha