The Use of Artificial Intelligence in Evaluating Clinical and Genomic Predictors of Immune-Related Adverse Events in Cancer Patients Treated with Immune Checkpoint Inhibitors
Advisor: Mr. Hamzawsky
Internship Location: Memorial Sloan Kettering Cancer Center Department of Epidemiology and Biostatistics
Onsite Mentor: Ed Reznik
Immunotherapy has transformed the landscape of cancer treatment, but its toxicities are often lifelong and severe. Furthermore, Immunotherapy related Adverse Events (IrAEs) often necessitate steroid prescription, which makes immunotherapies less effective, and the end of treatment. The biological mechanisms of IrAEs is unknown, and so is the “type” of patient who gets them: currently oncologists have to guess whether a given patient will suffer from an IrAE when on an Immune Checkpoint Inhibitor (ICI). This lack of knowledge about IrAEs is due to a lack of granular, population scale toxicity dataset. My project hypothesizes that the day-to-day experience of cancer patients can be highly informative for biological discovery. I will be using Large Language Models (LLMs) to extract toxicity data from 85,000 patients at Memorial Sloan Kettering Cancer Center, and I will then associate IrAEs with biomarkers through computational analysis. Hopefully, my project will lead to the discovery of a new predictor of IrAEs that Oncologists can test for.
Project Blog
Post 1: Introduction
Immunotherapies have been a breakthrough in cancer treatment, allowing us to cure and extend the lives of tens of thousands of patients. Immunotherapies train a patient's immune system to attack their tumor, but oftentimes the immune system attacks healthy cells too. These toxicities are called Immune related Adverse Events (IrAEs). Right now, doctors have to take a blind guess as to weather a patient is going to suffer from an IrAE or not. We have no idea what the patient who gets an IrAE looks like: if we can find out the sex, age, cancer type and genomic profiles of a patient who gets an IrAE when on an immunotherapy, we can model patient risk and save thousands of lives.
We don't know the characteristics (clinical phenotypes and genomics) of patients who get IrAEs because noone has ever performed a study on them at a large scale. If we want to perform a large scale study of IrAEs, we need data on which patients get IrAEs and which patients dont. My project focuses on creating the first ever integrated IrAE dataset, and using it to discover the type of patient who gets IrAEs.
Prior to my senior project, I had run a computational analysis that discovered HLA DRB1 associations with ICI induced Addison's disease. I want to investigate the biophysical mechanisms driving this interaction and hopefully work towards a biomarker or blood test for adverse event risk.
Week 2: Preliminary Experiments and planning
This week, I started working on my senior project full time. The end goal is to classify all 85,000 patients at MSKCC as having an Immune Related Adverse Event (IrAE), and distinguishing what IrAE they had. Previously, we had explicitly asked the LLM to evaluate 6 IrAEs: Pneumonitis, Colitis, Adrenal Insufficiency, Hyperthyroidism, Hypothyroidism, and Liver toxicity. For these calls, our prompt explicitly stated the conditions, and attributes of each of the adverse events we are looking for. While any IrAE call cannot state what to look for in general, I analyzed these calls to learn how the LLM comprehends clinical notes, and whether it is even capable of extracting the information that I want. The first thing that I did is I generated timelines for LLM positive and negative adrenal insufficiency cases, so I could see how often the LLM is right, and when the LLM predicts patients to have adverse events compared to when patients actually have adverse events. After that, I looked at the prednisone equivalent cumulative steroid dose in the 30 days after the LLM predicts a patient to have an adverse event. If the LLM is right, then in cases where the LLM predicts a patient to have an adverse event, their steroid intake will increase. Overall, after analyzing the LLM output data, I was confident that the current iteration of the LLM-RAG pipeline is accurate enough to scale to any IrAE.
I then started planning the architecture of the IRAE LLM call. My current plan is to have the LLM re-prompt itself in a way similar to a flowchart we can design. My hypothesis is that through filtering and prompting, we can minimize compute usage while maximizing performance and accuracy. I have started running experiments on a small scale, but I hope to have strong preliminary results by the end of next week.
I will soon upload a link to the results of all of my experiments, including the ones that I ran this week to analyze LLM performance.
Post 3: Blog #3: Learning and interpretations
This week was full of learning. I spent most of my time reading research academic papers and working my way through Michael Sipser's MIT course on YouTube “ Theory of computation” ( He is an amazing lecturer, and he has really given me a strong intuition on the principles of computing. I have some hypotheses I want to test later in this project related to cellular automata and deterministic finite automata, so his course has been very helpful)
Besides building intuition for later projects, the general goal was to create an interpretation of the GWAS hits I had previously found on the HLA DRB1 locus. Essentially, I found that when certain parts of the genome are mutated, a patient can become up to 3x more likely to develop adrenal insufficiency ( when they are on an immunotherapy). However, I want some way to describe this set of locations beyond just saying that they are on the HLA heterodimer ( the molecule that all of these amino acids are part of) . If I could back up a claim saying that, for example, all of the significant amino acid changes are in the peptide binding groove of the HLA ( For context, if a peptide binds to a HLA molecule, the cell is signaled to be killed by a T cell, so the shape of the groove of the HLA molecule determines what peptides can bind to it, and therefore what factors will cause the immune system to attack a cell), I could then associate the structure of the binding groove with autoimmune adrenal insufficiency on ICIs. This would be a very interesting clue for researchers who want to know the mechanism of ICI adrenal insufficiency. After reading the current literature, especially many papers from Soumya Raychaudhuri
and Sasha Gusev's lab, I made this plot to analyze where exactly the significant hits are. This is incredibly useful information as we can now investigate the peptide binding groove of the HLA as a determinant of ICI autoimmunity.
I also laid out the experimental design for early iterations of any IrAE call. When discussing with colleagues, we were unsure what the best architecture is, so we needed to design an experiment comparing a dynamic RAG system to some sort of agentic system. We also need to see how the performance of that compares to a system where the LLM reads all of the clinical notes, rather than notes being filtered through a machine learning model. After starting to engineer the prompts for these two sets of experiments, I started learning how to run large scale LLM pipelines on AWS bedrock, so that we can scale this system in the future.
Week 4: Temporality and Severity
This week was very exciting! The main project for this week was to figure out a way to characterize the grade transitions of adverse events over time. The general idea is that, since we know adverse events are biologically different, they likely have different temporal presentations, and the trajectory of their grades ( severity) is unique to each adverse event. To test this, I made several plots showing the percentage of the at risk population of our cohor that had an ae who has that ae at a given point in time. From this, we can plot when each grade of adverse event happens, and get a sense of how adverse events present themselves temporally, and how severity of adverse events progresses.
You can see in the below plots my two attempts at showing the temporality of adverse events visually. Both of them have their flaws, as there is time bias, however there are some very interesting patterns.
This week, I also finalized the design of the any IrAE call. We will feed all clinical notes into a general RAGLLM pipeline that will select only notes relevant to weather or not a patient has an AE, and then a low parameter LLM will rank the likely AEs this patient has. This will then set the keywords of a downstream RAG which will pick up on specific words and feed them into a larger LLM for final classification
Week 5: Reading and organization
This week I waited for new datasets, for existing experiments to finish. I updated myself on the current state of IrAE research, reading literature and watching peoples AACR work. Importantly, I went through all of my code and results and ensured that they were organizable, reproducible, and modular. This week, I also attempted to reproduce the results I found in the MSKCC cohort in other clinical trial cohorts. Many clinical trials report adverse event incidence rates, so for a set of clinical trials i reproduced the features of their cohorts and found the difference in incidence. I found that the incidence rates are very similar in all cases but liver toxicity. Liver toxicity in clinical trials is measured through lab values, while our LLM requires attribution of that toxicity to a systemic therapy, so our requirements are more strict.
Week 6: Temporality
This week I investigated the temporality of adverse events. Most of this project relies on analyses that ask about the type of patient that gets adverse events, as we are uniquely empowered to answer these questions with our dataset. However, another unique feature of our dataset is our temporal resolution. Because our RAG aggregates notes into one prompt per 90 days of clinical notes, we classify a patient as having an ae every 90 days, so we can estimate when that patient had an adverse event, not only if they had an adverse event. Naturally, the first question I tried to ask was “which adverse events occur earliest”, however answering it was not simple. There are two factors that confound this analysis: overall event occurrence, and censoring within our cohorts.
The first analysis to come to mind is a simple kaplan meier or cumulative incidence curve. These plots show the “survival rate” of a cohort over time ( the percent of the cohort that is adverse event positive or free at a given timepoint, depending on the curve). The reason why these cannot answer this question is that they are mainly confounded by overall occurrence. If one event happens at a greater frequency than another, that will alter the “shape” of a kaplan meier curve far more than temporal onset patterns will. Essentially, we need some analysis that normalizes event occurrence to the overall occurrence of that event, we need to plot “the percent of patients who have ever developed an specific adverse event, who developed that event at a given timepoint”.
The second problem is cohort censoring. Cohort's “at risk” population decreases over time, as patients stop treatment, start new therapies, leave trials, die, or get events. This means that the denominator of our equation decreases over time, and we need to select for that. The standard way of doing this, subsetting our cohort to patients with a set minimum follow up time, leads to selection bias, as our cohort would be of patients who did well on therapy. So, our denominator needs to change over time and be a function of the at risk population at a given timepoint for that adverse event.
We ended up with the below plot, and there are very clear patterns. What interests me most is the ability to suggest mechanistic differences from the temporal onset of these adverse events. Because we normalized to get rid of occurrences in our signal, we can see that mathematical features of these curves, like concavity and maxima point, differ between immunotherapy patients and not for some adverse events and others. My preliminary hypothesis is that these mathematical features are a function of the biological mechanism behind these adverse events. For example, colitis stays the same in the ICI and non ICI cohort, suggesting that the mechanism is not immune related in the way adrenal insufficiency is immune related. The curves for adrenal insufficiency are very different. This looks like a promising new research direction to me, and i plan on looking into it next week
Week 7: HLA & peptides
This week I read a lot of interesting literature to inform my project. Below is my interpretation of the three most relevant papers I read. With HLA imputation in hand from last week and structural analysis coming next, I wanted to understand what it means to say an HLA allele confers risk, since the allele only matters through which peptides its binding groove will hold and which T cells will recognize them.
The first was Reynisson et al. in Nucleic Acids Research on NetMHCpan-4.1 and NetMHCIIpan-4.0, pan-specific neural networks that predict peptide binding to MHC class I and class II from the MHC sequence, trained on a mix of binding affinity and mass-spec eluted ligand data, and covering more than eleven thousand class I alleles and close to a thousand class II alleles. This changed how I think about testing an HLA association. I had been treating an HLA allele as a categorical label, where a patient either carries DQB1*06:02 or does not, and the predictor reframes the allele as a function from peptide sequences to binding scores, which means two carriers are equivalent at the allele level but differ in which self-peptide reaches a T cell depending on the proteome they happen to be exposing. The class II coverage matters for me since most of my endocrine candidates are DQ and DP alleles, and the ability to combine alpha and beta chains is what makes a DQA1/DQB1 or DPA1/DPB1 association interpretable at the peptide level at all.
The second was Siebold et al. in PNAS, the crystal structure of HLA-DQ0602 (DQA1*01:02/DQB1*06:02), the same molecule strongly associated with narcolepsy and dominantly protective against type 1 diabetes. Structural comparisons with closely related DQ molecules showed that the P4 pocket volume drives the narcolepsy susceptibility while the P6 and P9 pockets determine the breadth of peptides bound in the protective context. Reading this shifted how I think about the same allele showing up in the hypophysitis literature, since the relevant question moves from whether DQB1*06:02 associates with toxicity to which pocket geometry is doing the work and which self-peptide that pocket can hold. It also makes close-relative comparisons more important than I had thought, since DQB1*06:01 and DQB1*06:02 differ at only a few residues, and the residues that distinguish a risk allele from a protective one are where the mechanism lives.
The third was Prinz in Frontiers in Immunology, a review of the methodological problems with identifying which self-peptides drive autoimmune T-cell responses, given that class II molecules can present peptides from both extracellular and cellular self-proteins and TCRs are polyspecific. This reframed how confident I should be about any mechanistic story I attach to an HLA association. I had been picturing the chain from allele to peptide to T cell as something I could in principle write down for a given hit, and the review makes clear that the immunopeptidome of a class II allele is enormous, that polyspecific TCRs blur which peptide is the relevant one, and that for most HLA-associated autoimmune diseases the driving self-peptide remains unknown despite decades of work. It is a useful check on the temptation to over-narrate from an association, and it gives me a more honest sense of what a structural analysis can and cannot resolve about why a given allele predisposes to a given toxicity.
Week 8
This week I read a lot of interesting literature to inform my project. Below is my interpretation of the three most relevant papers I read. With the SAIGE-based associations coming next, I wanted to understand what a GWAS offers beyond a list of significant variants, since heritability, cross-trait correlation, and polygenic scoring would let me say more about the architecture of toxicity risk than any one hit on its own.
The first was Bulik-Sullivan et al. in Nature Genetics on LD Score Regression, which uses the slope of test statistics against LD scores to estimate SNP heritability and uses the intercept to separate true polygenic signal from confounding by population stratification or cryptic relatedness. Reading it shifted how I think about inflation in test statistics. I had been treating inflation as something to fix or apologize for, and LDSC reframes the intercept as an estimable quantity that tells me whether the inflation is mostly polygenic signal or mostly population structure, which moves the diagnostic question from whether the GWAS worked to what kind of signal it is showing. The heritability estimate also puts a ceiling on how much any germline predictor I build from a phenotype can ever explain, which is a more honest framing than reporting individual hits without anchoring how much of the disease they could plausibly account for.
The second was the companion Nature Genetics paper from the same group on cross-trait LD Score Regression and the atlas of genetic correlations across human diseases and traits, which extended the method to estimate genetic correlation between two phenotypes from summary statistics alone, without requiring overlapping samples. This reframed how I think about the relationship between my different toxicity phenotypes. I had been treating thyroiditis, hypophysitis, and adrenal insufficiency as separate GWAS endpoints that happen to share an organ system, and cross-trait LDSC lets me ask whether they share genetic architecture as a quantitative number rather than as an impression I form from overlapping hits. It also opens a comparison I had not been planning, between my irAE phenotypes and the spontaneous autoimmune diseases they resemble, since I can ask whether checkpoint-induced thyroiditis shares its genetic basis with Graves' or Hashimoto's at the polygenic level rather than only at the few loci that reach significance in either scan.
The third was Ge et al. in Nature Communications on PRS-CS, a polygenic risk score method that uses a Bayesian regression framework with a continuous shrinkage prior on SNP effect sizes and an external LD reference panel, and which improved prediction accuracy over previous methods in the simulations and Partners HealthCare Biobank evaluations they ran. This shifted what I think the deliverable of my project could look like in a clinical setting. I had been thinking in terms of a single large-effect variant, the kind of finding PREDICT-1 used for abacavir, and PRS-CS shows that for a polygenic toxicity the more realistic readout is a continuous score that aggregates many small effects across the genome. The implication for the translational end of my project is that the eventual germline test for irAE risk could end up being a polygenic score computed from the same panel data oncologists already have, which is a different kind of product from the single-allele template PREDICT-1 set and worth understanding the methodology for in advance.
Week 9: Clinicogenomic datasets
This week I read a lot of interesting literature to inform my project. Below is my interpretation of the three most relevant papers I read. I stepped back from the newest extraction work to read the older foundations my pipeline depends on, since the hard parts of turning notes into labels were mostly named long before large language models arrived.
The first was Harkema et al. in the Journal of Biomedical Informatics, which introduced ConText, an algorithm for deciding whether a condition in a clinical note is negated, hypothetical, historical, or about someone other than the patient. This changed how I understand where my extraction errors come from. I had been thinking of the task as finding irAEs in the text, and ConText splits that into two genuinely different problems, noticing that a condition is mentioned and judging whether it is actually asserted of this patient now. Seeing those as separate makes me realize a pipeline can look accurate at finding mentions while being wrong about assertion in a way that would inflate my case counts, and that the second problem is the one that actually threatens my GWAS.
The second was the Snorkel paper by Ratner et al. on weak supervision, the idea of writing many noisy labeling functions and modeling their agreement to produce probabilistic training labels. This reframed how I think about my LLM's output. I had been treating the model as a label I either trust or do not, and weak supervision recasts it as one noisy voter among several, no different in kind from an ICD code or a medication-discontinuation signal. The deeper shift is in how I think about a label itself, since it stops being a hard fact about a patient and becomes a probability with disagreement baked into it, which is a more honest representation of how uncertain note-derived phenotypes really are.
The third was Zehir et al. in Nature Medicine, the foundational MSK-IMPACT paper, describing prospective sequencing of more than 10,000 patients with matched tumor and normal tissue. Reading where my dataset came from changed how I think about its limits. The matched normal is the reason germline analysis is possible at all, and understanding that the panel was built to find somatic tumor mutations across a few hundred cancer genes reframes my germline scan as borrowing a tool designed for something else. It also makes the cohort feel less like a neutral sample of patients and more like a specific population, enriched for advanced and previously treated cancers, which shapes how much I should generalize anything I find and reminds me the dataset was never assembled with my question in mind.