Reddit Fever: Harnessing Reddit Posts for Influenza Case Detection
Advisor: Dr. Allendoerfer
Internship Location: World Bank Group
Onsite Mentor: Shiyong Wang, Senior Health Specialist, The World Bank
The COVID-19 pandemic revealed gaps in infected case reporting due to self-testing for infection and lack of transparency regarding the spread and origin of the disease. Therefore, an alternative method for infectious disease detection that operates independently of official or patient reports, such as scanning social media for posts discussing infection symptoms, would advance epidemiologic surveillance. I will evaluate the potential of Reddit posts to determine the prevalence of influenza-like-illness (ILI) cases as an indicator of influenza infection in the Western United States during the 2024-2025 influenza season; through this project, I hope to validate social media as an accessible, time-efficient way to supplement traditional surveillance methods.
I will train a classification model able to differentiate between posts suggesting the Redditor has ILI from those that indicate other infections or neutrally discuss ILI from sickness-related subreddits (e.g. r/sick, r/flu). I will apply my model to subreddits indicating geographic location (e.g. r/California, r/SanJose) for each 3-month time frame between the start of October 2024 and end of March 2025. After mapping the relative prevalence of ILI per state and comparing with government-issued maps and statistics, I can gauge the potential of Reddit as an informal source for syndromic surveillance. An understanding of using such informal sources would fill gaps in traditional surveillance methods.
Project Blog
Week 1: Welcome!
Hey everyone, thanks for stopping by my blog! I’m Angela and I’m super excited to share my project with you over the next weeks. Today, I want to give you a brief introduction to my project, including the background behind my initiative, my project objective, and a brief description of my goals.
Although our public health systems are very advanced and have been able to track, mitigate, and prevent many infectious disease outbreaks, there are often gaps in case reporting, particularly as seen during the COVID-19 pandemic when many patients tested at home and could not report their results. Certain middle-to-low-income countries may lack adequate resources for traditional surveillance methods, with costs ranging from $63 to over $19,000 in a two-week surveillance period. Additionally, traditional surveillance methods have natural reporting delays, with an up to 40-day delay for pertussis reporting in the United States, which allows the disease to incubate and spread further during the delay period. I’m interested in supplementing traditional infectious disease surveillance methods with more accessible, timely, and cost-effective methods, particularly by using social media as an “informal source” for detecting cases of infection amongst the social media population.
Through my project, I aim to evaluate Reddit as an informal source to detect influenza-like-illness (ILI) cases which can provide insight into the risk of influenza in a community or population during the 2024 - 2025 influenza season. I plan on training a machine learning classification model capable of detecting and differentiating between posts indicating ILI infection in the user and other posts about other illnesses or merely discussing the outbreak itself. After applying my model to subreddits indicating location, I can then gauge the accuracy of my method compared to official reports.
Ultimately, my project evaluates whether or not Reddit is an effective method of informal public health surveillance for ILI. If results are favorable, I hope to further a new method of infectious disease surveillance to supplement traditional methods. At the same time, I hope to learn more about coding and machine learning, which may come in handy in my career.
Week 2: Data Collection
Welcome back to my blog, hope you had a great week! This post will run on the long side, and I will introduce my project inspiration through a mini-literature review, going over my progress last week, and sharing my plans for the next week!
As per my last post, there has been an increasing need for informal sources of public health surveillance, such as the internet. Due to its accessibility, relevance to all demographics, and speed, researchers have tested its potential to provide crucial information on population health. In 2006, Eysenbach et. al. used Google searches to conclude a strong correlation between the influenza cases indicated by searches to official reports [1]. Additionally, Yahoo search queries accurately predicted influenza outbreaks up to 5 weeks in advance, and Google search queries estimated general influenza frequencies much faster than government reports with only a 1-day delay [2, 3].
Along with the internet rose social media, where patients, free from healthcare-related stigma, tended to feel more open to express or vent about their health concerns. This introduces an entirely new digital subpopulation with new opinions and behaviors for public health surveillance. X (previously Twitter) has been used most frequently in the past for public health surveillance, in particular for cholera, E. coli, and Ebola outbreaks [4, 5, 6]. Two studies have also applied X in a similar way to detect influenza cases in real time with an 85% accuracy [7, 8]. Yet, as X faces controversy and decline in post quality, moderation, and user engagement, subsequently resulting in a decrease in relevancy, I was inspired to try a similar approach with other more relevant social media platforms, in particular Reddit. Because Reddit has a huge post character limit, allowing users to freely express themselves, has over 1 billion users worldwide, and faces little controversy, it is the perfect popular, open informal source for surveillance data collection.
I am working on phase 1 of my project, which spans approximately 5 weeks. During phase 1, I am creating and evaluating the ability of 2 machine learning models to detect and differentiate Reddit posts indicating ILI. I will then apply the better model to real-time data from subreddits indicating the geographic location of each state in the Western U.S. and their top 2 most populated cities (e.g. r/California, r/LosAngeles, r/SanDiego). The easternmost states I plan to collect from are Montana, Wyoming, Colorado, and New Mexico, simply because the Western U.S. is geographically most relevant to our geographic location. Because my project is run in real time, I will be collecting data from these subreddits at the end of every month.
Last week I focused on collecting the real-time data I am applying my model to. I used Python Reddit API Wrapper (PRAW), a Python package adhering to Reddit API guidelines, to pull the title and body texts of each subreddit’s posts from October 1, 2024 to December 31, 2024. In total, I collected 156 dataframes, each with anywhere from ten to hundreds of individual posts, cleaned the data frames by filling empty body text rows with the value “0” to eliminate resulting errors, and combined each geographical subreddit’s dataframes for the first period (October to December 2024). While the data collection process was straightforward, I came across some interesting challenges. Firstly, I realized that PRAW could only collect posts that fit under each subreddit’s categories: hot, rising, top, controversial, and new; thus, it would not be able to collect every post that every user publishes in Reddit's entire history. Additionally, PRAW struggled to access posts from more than 3 months ago, resulting in small dataframes for almost all subreddits for October 2024 and some for November 2024. Finally, while some of the largest subreddits (notably, r/California and r/SanDiego) had many posts a day, the majority were links to news about the particular state or city instead of posts from individual users, which would have failed to provide information about ILI infection status.
While the first challenge was due to the timing of my project, I was able to quickly adjust to the others. Firstly, collecting posts every month in real time for January to April will allow me to pull all data for that month as they will not yet be archived. For the subreddits that had a majority of news article links instead of user posts, I simply collected posts from the same time frames in the next most populated cities (e.g. r/SanJose and r/SanFrancisco for California), and I will run my model on these subreddits instead.
As of now, I also have collected all posts for January 2025, but have not yet preprocessed them. Next week I will be collecting February 2025 posts and starting my model training process! I will be deciding on the 5 subreddits from which I am collecting training data from, as well as the keywords I will be filtering my testing and training data with. Thank you for joining me this week, and I hope you’ll check back in for more!
Works Referenced:
[1] Gunther Eysenbach. Infodemiology: Tracking Flu-Related Searches on the Web for Syndromic Surveillance. AMIA Annual Symposium Proceedings 2024, 2006, 244.
[2] Polgreen, Philip M.; Chen, Y.; Pennock, David M.; Nelson, Forrest D. Using Internet Searches for Influenza Surveillance. Clinical Infectious Diseases 2008, 47 (11), 1443–1448. https://doi.org/10.1086/593098.
[3] Ginsberg, J.; Mohebbi, M. H.; Patel, R. S.; Brammer, L.; Smolinski, M. S.; Brilliant, L. Detecting Influenza Epidemics Using Search Engine Query Data. Nature 2009, 457 (7232), 1012–1014. https://doi.org/10.1038/nature07634.
[4] Chunara, R.; Andrews, J. R.; Brownstein, J. S. Social and News Media Enable Estimation of Epidemiological Patterns Early in the 2010 Haitian Cholera Outbreak. The American Journal of Tropical Medicine and Hygiene 2012, 86 (1), 39–45. https://doi.org/10.4269/ajtmh.2012.11-0597.
[5] Diaz-Aviles, E.; Stewart, A. Tracking Twitter for Epidemic Intelligence. Proceedings of the 3rd Annual ACM Web Science Conference on - WebSci ’12 2012. https://doi.org/10.1145/2380718.2380730.
[6] Odlum, M.; Yoon, S. What Can We Learn about the Ebola Outbreak from Tweets? American Journal of Infection Control 2015, 43 (6), 563–571. https://doi.org/10.1016/j.ajic.2015.02.023.
[7] Culotta, A. Lightweight Methods to Estimate Influenza Rates and Alcohol Sales Volume from Twitter Messages. Language Resources and Evaluation 2012, 47 (1), 217–238. https://doi.org/10.1007/s10579-012-9185-0.
[8] Broniatowski, D. A.; Paul, M. J.; Dredze, M. National and Local Influenza Surveillance through Twitter: An Analysis of the 2012-2013 Influenza Epidemic. PLoS ONE 2013, 8 (12), e83672. https://doi.org/10.1371/journal.pone.0083672.
Week 3: Model Training (Part 1)
Hi everyone, welcome back to my blog this week! This week, I focused on collecting and labelling the datasets I used to train my models.
Some background about training machine learning models to detect ILI infection through Reddit posts: each model is trained on a training dataset, which is first manually labeled by the researcher. In my case, I will be labelling the posts as either ILI (1) or not ILI (0), and these labels are binary and categorical. After labelling, the training dataset is split up, with 80% of the data being used by the model program to train itself, and the other 20% of the data being used to gauge the model's accuracy. While the data can be split any way, an 80-20 is the most common. After training, Python produces a F1 score indicating the fit of the model. The F1 score ranges from 0 to 1, with 1 being the most fit and 0 being the least.
Before senior projects officially started, I had done some practicing with model training. I found that typically, increases in the size of training datasets produced lower F1 scores, so I decided to keep my training post dataset relatively small at about 2000 to 3000 posts. I also did some experimentation with two types of models—logistic regression and random forest classifier. While logistic regression is standard for classification models, random forest may be more fit due to the noise of reddit posts, so I am training one of each with the same labels to see which one fits better.
I picked 6 medicine/health-related subreddits to collect my training posts from and made sure I collected enough of both expected ILI and non-ILI posts. I picked r/sick and r/flu for my ILI posts, r/epidemiology and r/epidemic for my non-ILI posts, and r/AskDocs and r/DiagnoseMe for a combination of both. After filtering the subreddit posts for 40 infection-related keywords (see the Data Collection details doc), I collected 2,479 posts as my training dataset. The keywords will be the same as those I filter my experimental data with. After preprocessing the data sets (removing stop words, special characters, lemmatizing, making everything lower-case), I am now in the process of manually labelling each post as ILI or non-ILI, following the criteria I have set here in the detailed doc linked below. While I am following the CDC’s definition of ILI (fever and cough and/or sore throat), I did expand it to encompass more flu-like core symptoms, such as body aches and congestion, as well as secondary symptoms such as headache and runny nose because social media users may not discuss every symptom they have unprompted.
Labelling posts was surprisingly time consuming, as each post took time to scan through for symptoms and some were particularly hard to read after lemmatization. It took me about 5-6 hours to get through half the posts, and I am still working on the other half this week. Although the majority of my labelling is based on the frequencies that certain key symptoms were mentioned, it does also consider subjective context, which AI doesn’t do so as well.
My immediate next steps are to finish labelling by next week and training logistic regression and random forest classifier models. Due to the subjectivity of my labelling, however, my F1 score may be low to start. While a F1 score of >0.5 is considered average, I plan on re-training my model if it dips below 0.65 with more objective criteria, such as strictly following the CDC’s definition of ILI (both fever and cough/sore throat must be present) when labelling, possibly with the help of AI tools which can work with more objective criteria. If all goes well though, I'll try applying my model to real-time data I collected from regional subreddits during October to December 2024.
Thanks for checking in, and see you next week!
Data Collection Details: https://docs.google.com/document/d/1ZxxsMAD5o7VpqsqAu_hB8hGZiZ1b2MgFTsf2ZLL_D6c/edit?usp=sharing
Week 4: Modeling pt. 2
Welcome back! A lot has happened since my last blog post and I’m excited to share them with you, as well as some likely changes I plan to make to my project objectives.
Last week I focused on creating machine learning models to apply to my experimental data. I trained both a logistic regression classification model and a random forest classifier (RFC) model, both with initial F1 scores of 0.83 (see the classification reports below).
While the F1 scores indicate that the model(s) were good to use, their ability to detect ILI posts in the training data was too low, with the random forest model having an accuracy of 53% and the logistic regression model having an accuracy of only 57%. Essentially, if I used this model right away, it would miss a lot of the ILI posts, producing too many false negatives. In the real world, false negatives present more risk than false positives because it allows for the spread of the infectious disease.
To address this, I tried optimizing my models by increasing the class weight of the influenza label class. This essentially tells the model that detecting infected users was more important than detecting healthy users and it would face more risk missing infected users. Although the accuracy and F1 score of my RFC model didn’t change, the F1 score of my logistic regression model increased to 0.87 and the accuracy of detecting ILI posts increased to 82%, quite a significant jump.
Although an 82% accuracy was decent, I wanted to try one more time to see if I can increase the accuracy even further and change my approach a bit. My models performed well, but I had done my labelling with criteria based on the keywords that they mentioned, and ignored the context of the posts. Because of this, sometimes I had to label posts in which the user sounded sufficiently sick with ILI as healthy because they didn’t meet my keyword criteria.
To factor into the context of the posts, I turned to AI labelling and modeling. Large language models such as ChatGPT are capable of identifying ILI based on influenza-related keywords, the CDC’s definition of ILI, users’ sentence structure, and can process semantics, which my previous models don’t do as well. This was a completely new topic I didn’t expect to learn, but with help from Claude.ai to try AI labelling, modeling, and write/check the code, I labelled my training dataset with ChatGPT, which I used to train a BERT model, an AI model capable of learning complex sentence structures. This model generated a F1 score of 0.86 and an ILI-detection accuracy of 87%! I have yet to optimize this model, but it is already quite good for classifying the complex, noisy Reddit posts. Next week I will be conducting a comparison between my models and labeling methods, and evaluating the one I plan on using to classify experimental data.
As I near the end of my model training phase, I’m transitioning to phase 2, where I apply my model to real-time data to plot areas of high risk in the Western US. However, recently, when checking my datasets, I came across a challenge. Because Reddit doesn’t release geographical data in its API, I had to find a workaround by pulling real-time experimental posts from subreddits with geographical location. However, after filtering for flu and infection-related keywords, I ended up with too few posts to apply my model to and come up with any conclusive results.
As of now, I’m changing the direction of my project to rely less on geographical information. However, Reddit API gives extremely precise timestamps for its posts, so it’s promising to conduct temporal analysis on the posts instead. I have yet to finalize my new project direction, but I expect to be focusing on the changes in prevalence of ILI infection and concern surrounding the disease through the flu season as suggested through the changing numbers of ILI posts over the months. By possibly comparing the trends in concern to trends in actual infection cases, I could make connections to media panic and/or possibly early/late preparation for outbreaks.
Next week I will be developing my idea even further and run it by my advisors, as well as finishing up comparisons between my models. Super excited to continue sharing my progress here!
Model Classification Reports: https://docs.google.com/document/d/1-rvCa6ED-hUa-rutpR6kr2oA-3fAEnnu1BadxHWGmuY/edit?usp=sharing
Week 5: New Direction...
Welcome back!!!! This week, I am wrapping up my model training phase, and I have made some insightful comparisons between my models! I also have made some big changes to the second phase of my project, and I’m excited to tell you all about it.
A key part of my model training involved labeling my training dataset in two ways for my models:
Manually (for logistic regression & random forest classifier) - counting specific symptom keywords
ChatGPT AI API (for BERT) - giving ChatGPT the prompt: “Classify this Reddit post as 1 (the post indicates the user has influenza-like illness) or 0 (the post does not indicate the user has influenza-like illness).”
I wanted to see how these two labeling methods differed in their labeling patterns, so I analyzed and graphed their labeling distributions:
I also evaluated the models themselves using ROC (Receiver Operating Characteristic) curves, which measure how well a model distinguishes between true positive and negative ILI cases in posts. A ROC curve plots the model’s true positive rate (sensitivity) against the false positive rate (specificity). Essentially, a perfect model’s ROC curve would hug the top left corner with a high area under the curve (AUC), while a poor model that randomly guesses would produce a straight diagonal ROC line with a low AUC.
Here are the ROC curves I generated for my models! Because my logistic regression model and random forest classifier were trained on the same labeled set and the BERT model on another, I kept the former two together on the same graph for ease of coding.
While all three models performed well, the AI BERT model outperformed the other two by a significant margin. When considering multiple factors - F1 scores, ROC graphs, and accuracy in detecting positive ILI cases - the BERT model, trained on ChatGPT-labeled data, is the best fit for Reddit post datasets. Reddit posts are full of slang, semantics, and informal phrasing that traditional number-based scikit-learn models like Logistic Regression or RFC struggle to understand, but BERT does a very good job understanding the complexity of the English language and thus is much better suited for messy Reddit posts.
Now, for a little more logistical content…
Last week, I mentioned that due to Reddit’s API limitations, I would need to make some changes to my project. When I was going through sickness-related subreddits, I noticed that users posted about their health year-round. From this, I hypothesized that while users post about their health year-round, the flu season would bring about more flu-related posts proportionally compared to the rest of the year (including more users who attribute non-flu symptoms to the flu, showing concern). So, instead of analyzing the geographic location of ILI-related posts, I will focus on how the changes in the prevalence of flu cases and concern surrounding the flu are reflected in Reddit posts, regardless of geographic location. Here are my steps:
I will collect posts from health-related subreddits where users discuss their health conditions often (such as r/medical_advice) and record timestamps
Using my model, I will isolate posts that have ILI, noting the month and year (depending on if I have enough data, I can conduct 2 analyses per month)
I will then determine the proportion of ILI-related posts per month to track trends in flu prevalence and concern over the 2024-2025 flu season.
To put my findings into context, I’ll compare them with Google Trends graphs of flu-related keyword searches(a more representative sample of the global population)
I’ll also compare my findings with government-reported flu cases globally to see how well Reddit concern aligns with real-time outbreak trends
Well, why is my work important?
If Reddit concern trends don’t match with real-time government-reported cases and peak too early, this could be a sign of media-induced panic and indicates more public health efforts are required to improve transparency regarding the impending flu outbreak and dissuade baseless rumors
If the peaks of concern on Reddit shift per year, it could suggest changes in the preparedness of the public to the impending flu season, reflecting the effectiveness of public health teams leading up to the season
If my results match well with true cases and Google Trends, my research could be a further indicator that social media is a great place to provide factual, transparent public health information, and is also a valuable source to gauge the preparedness of a community/population for an upcoming outbreak.
Next week, I will be focusing on readings regarding media-induced panic, specifically how and why the media creates anxiety amongst people about a possible outbreak and ways to quell the panic. I will also find readings on how internet sources (like Google Trends) have been used to gauge public concern surrounding outbreaks and diseases. Towards the end of the week, I will also start collecting my experimental dataset, and preparing my models to run on the data!
Super excited to share what I find in my readings next week! See you soon :)
Week 6: The Media Panic Infodemic
Hello everyone!! Today marks the transition week of my project, aka where I read more background info about my new topic and find interesting connections as the context for my experimental phase and significance for real-world applications. This blog update will be short, but I read lots of interesting papers this week, and I'd like to share them with you!
My project is transitioning from having a purely epidemiologic focus to having more of a social psychology focus, particularly surrounding media-induced panic. At the start of outbreaks, it is natural for news media sources to report on them and include possible health implications as asserted by health professionals. Take the COVID-19 pandemic for instance; think of how many news and video media sources were reporting on the cases, covering everything from case numbers to at-home remedies. Media exposure can be positive; during the H1N1 swine flu outbreak, increased exposure to news reports increased fear and perceived knowledge about the disease amongst viewers, leading to more adoption of preventative measures [1].
However, when the media overreports on an outbreak, it creates a contagion effect where each news post or video creates a wave of concern that may not match the actual severity of the outbreak. For instance, in the 2014 Ebola outbreak in the US, thousands of tweets and internet searches were posted/performed for each media coverage, despite only a few confirmed Ebola cases [2]. This shows the extent to which modern media coverage can cause panic that is not necessarily representative of the scope of each outbreak.
Panic itself is a human instinct, but it also drives misinformation and conspiracy theories, making the situation worse. Firstly, the panic on social media tends to manifest as useless information regarding the outbreak and does not provide practical advice to individuals on how to keep themselves safe [3]. The panic and fear resulting from media exposure can turn into “moral panic”, an intense, sudden, and irrational fear of a certain group of individuals or ideas blamed for the event, in addition to self-speculative self-diagnoses [4]. History is filled with examples of moral panic gone rogue: the Salem Witch Trials, Red Scare, and the War on Drugs often stemmed from fear-driven narratives [4]. In the context of outbreaks, moral panic can lead to stigmatization, alienation, and discrimination. In fact, some researchers argue that this panic spreads faster than the virus itself, further driving rapid misinformation [5].
Researchers assert that the media-driven “infodemic” is just as, if not more, harmful than the outbreak contagions themselves because of the social implications (such as alienation, skepticism) and economic consequences (such as restrictions on travel and imports, which may harm nations’ global economic standings) [6]. Further, researchers urge for spatiotemporal analysis of social media discourse instances, essentially mapping posts across time and geography, to help public health officials target key areas and times for targeted communication and intervention campaigns to address misinformation and conspiracy [6].
Although my project can no longer focus on geographic mapping, I have the data and information to conduct temporal mapping on Reddit posts. I am putting researchers’ propositions to the test, evaluating if Reddit discussion patterns match with trends observed in prior studies and testing whether temporal analysis alone can provide useful insight regarding public concern for targeted public health interventions.
Works Cited:
[1] Zhang, L.; Kong, Y.; Chang, H. Media Use and Health Behavior in H1N1 Flu Crisis: The Mediating Role of Perceived Knowledge and Fear. Atlantic Journal of Communication 2015, 23 (2), 67–80. https://doi.org/10.1080/15456870.2015.1013101.
[2] Towers, S.; Afzal, S.; Bernal, G.; Bliss, N.; Brown, S.; Espinoza, B.; Jackson, J.; Judson-Garcia, J.; Khan, M.; Lin, M.; Mamada, R.; Moreno, V. M.; Nazari, F.; Okuneye, K.; Ross, M. L.; Rodriguez, C.; Medlock, J.; Ebert, D.; Castillo-Chavez, C. Mass Media and the Contagion of Fear: The Case of Ebola in America. PLOS ONE 2015, 10 (6), e0129179. https://doi.org/10.1371/journal.pone.0129179.
[3] Vos, S. C.; Buckner, M. M. Social Media Messages in an Emerging Health Crisis: Tweeting Bird Flu. Journal of Health Communication 2015, 21 (3), 301–308. https://doi.org/10.1080/10810730.2015.1064495.
[4] Ahmed, W.; Bath, P. A.; Sbaffi, L.; Demartini, G. Moral Panic through the Lens of Twitter. Proceedings of the 9th International Conference on Social Media and Society 2018. https://doi.org/10.1145/3217804.3217915
[5] Radwan, E.; Radwan, A. The Spread of the Pandemic of Social Media Panic during the COVID-19 Outbreak. European Journal of Environment and Public Health 2020, 4 (2), em0044. https://doi.org/10.29333/ejeph/8277.
[6] Depoux, A.; Martin, S.; Karafillakis, E.; Preet, R.; Wilder-Smith, A.; Larson, H. The Pandemic of Social Media Panic Travels Faster than the COVID-19 Outbreak. Journal of Travel Medicine 2020, 27 (3). https://doi.org/10.1093/jtm/taaa031.
Week 7: First Model Classifications
Hey all! Welcome back to my blog…without further ado, I’m excited to share with you the experimental results of my model on actual Reddit posts!
As a quick recap, last week I focused on running my AI BERT model on experimental data, which I collected from 7 subreddits where users felt free to express concerns about their health (r/sick, r/medical, r/medical_advice, r/AskDocs, r/AskDoctorSmeeee, r/flu, r/DiagnoseMe). I collected all the available Reddit posts on each subreddit, with the earliest post being from June 2012. Because I wanted to compare the patterns of this flu season with the previous 2 seasons, I used my model to classify 23,403 posts on and after October 2, 2022 (yes, that is huge and time-consuming for my poor computer :( ).
While I did classify all the posts I needed for my project, I could only work on the summary statistics for the 2024-2025 flu season because of the amount of analysis I needed to conduct. Let me show you my findings!
I first analyzed and plotted the proportion of posts from October 1, 2024 to March 31, 2025 (which encompasses the 2024-2025 flu season) that indicated ILI. Out of 11,032 total posts in that time frame, I generated the following summary tables by month and two-week period:
As time goes on, fewer posts become available to PRAW, which explains the generally increasing number of total posts per month nearing March. However, we see that the proportion of ILI posts peaked in January (in particular the January 21 - February 3 period) and dipped significantly starting early March (the March 4 - March 17 period). I plotted these numbers in bar graphs, including error bars for a 95% confidence interval, which represent where the true proportion of ILI prevalence/concern may lie in the general population.
The biweekly graph shows more detailed fluctuations than the monthly graph, but it reveals the same trend. Because March posts have such a large sample size (being the most recent, March posts are still readily available for PRAW), the error is also relatively small. That being said, we can see a clear rise and peak in January 2025 and a drastic fall following.
Next week, I plan to create similar graphs for the past two flu seasons (2022–2023 and 2023–2024) to determine whether the trend observed on Reddit this season also appeared in previous years. Identifying a consistent pattern would help establish Reddit as a more credible source for tracking flu trends. Additionally, I will collect national flu statistics from official government reports for the same periods and incorporate them into my graphs. This comparison will allow me to assess how well my findings align with official data and evaluate the reliability of my approach.
Week 8: Full Dataset Graphs
Welcome back! It’s incredible to say we are nearing the last few weeks of the senior project! While my updates are relatively short this week, I have interesting findings to share with you all regarding the influenza status shown through Reddit.
Last week, I ran my BERT model on my Reddit sample from the 2024-2025 flu season. Because the graph had a nice shape that likely represented a seasonal trend in the proportion of ILI posts, I wanted to see if the same could be shown for previous seasons. I graphed out the proportion of ILI posts detected from all two-week periods since October 2022:
While the graph is slightly blurry, the general trend in flu-related Reddit posts remains consistent throughout the 2022-2023 and 2023-2024 seasons, with peaks in cases around December to January. However, the error margins for the past two seasons are noticeably larger than the most recent season, likely because PRAW was able to access less posts from previous years, contributing to the uncertainty. Another notable observation is that the proportion of posts indicating ILI has increased over the last three years; while this is difficult to explain without reading into confounding factors such as lockdown returns or particular big panics caused by news coverage, we can infer that as the world returns to normalcy from the COVID-19 pandemic and take less respiratory precautions, the number of influenza cases also rises.
My next step is to compare the Reddit flu rates to official flu rates reported by the top 3 English-speaking nations from Reddit’s user demographics to assess whether the timing of concern from Reddit correlates with real-world infection rates. Because most of the graphs published online are line graphs, I also created a line graph version of my bar graph, with a slightly different representation of the 95% confidence interval (shown as the shaded light blue area):
I will also be normalizing the flu rates on Reddit to rates of concern demonstrated through Google Trends. By doing this, I aim to determine how representative of actual public interest/concern Reddit can be.
Although this week’s blog is short, I hope you found these findings just as interesting as I did! Be sure to look out for my next post next Friday!
Week 9: Comparisons with National Data!
Hello again everyone!! Hope you all had a great week :)
This week I also did a lot of graphing and data analysis, but spent most of my time comparing my model’s classification to flu rates reported by large English-speaking countries to see how well my findings can represent (or even predict!) actual flu rates, which may open the door for epidemiologic applications of my approach.
Although Reddit doesn’t release precise geographic data, the user demographic by country is available online. According to the World Population Review, Reddit’s user base is dominated by users from the United States (42.95%), United Kingdom (5.46%), and Canada (5.01%) [1]. Although there is a huge gap between the United States and the other nations, the three nations together account for more than 50% of the Redditor user base. I decided to use these three nations to compare my findings to for the following reasons:
They all have English as an official language. This is important because most posts on Reddit are in English, and thus the sample I took from Reddit is likely representative of these three countries (versus if the country didnt have English as an official language, people from the country may not post on Reddit to begin with and so Reddit is not representative of that country)
All three are in the Northern Hemisphere. The graphs I created show patterns in flu rates that match up to the flu seasons in the Northern Hemisphere (October - March). I explored Australia’s flu trends, but since their flu season starts in April and ends around October, my Reddit sample did not match up well, making the information less useful for analysis.
They make up the majority of Reddit users. Because these three countries collectively represent more than half of Reddit’s user base, it’s reasonable to assume that my Reddit sample most accurately reflects trends in these countries.
While this approach isn’t ideal for capturing worldwide flu trends, it was the most feasible given language, time, and resource limitations. A valuable next step for this project would be to expand the analysis to even more countries, including non-English-speaking countries using multilingual models or translated data.
To plot and make comparisons between my findings and national reports, I first collected flu rates reported every two weeks from national databases from September 29, 2024 to March 29, 2025 to cover the current flu season [2], [3], [4]. I made sure the start date of each two-week period aligned with the dates of my data, and plotted them together on a line graph:
There were a couple of key observations that stood out to me initially:
First, the flu rates estimated by my BERT model were consistently higher than those reported by national databases. This difference stems from the distinct sampling methods. National public health systems rely on lab tests from tens or hundreds of thousands of people each week, including many who test negative. In contrast, my dataset consisted of about 11,000 posts from specific sickness-related subreddits, where users are more likely to post only if they are experiencing symptoms or have already been diagnosed. As a result, my sample naturally skews toward higher flu positivity rates. This doesn't mean my sample is unrepresentative of the general population; rather, it reflects a practical approach. Given my limited time and resources, it was far more feasible to focus on specific subreddits than to sample from all of Reddit.
The general shape of the BERT line closely resembled that of the United States and Canada lines. While it is difficult to explain the resemblance to Canada, the resemblance to the United States is likely due to the fact that over 40% of Reddit users are from the United States.
Because I applied my BERT model to every two-week period on Reddit, I also took national data of the average flu rate every two weeks. However, I was curious about the resemblance of the BERT curve to that of national rate graphs, so I interpolated weekly BERT values (taking average of the two weeks’ rates) and plotted them against the reported weekly flu rates. I found a similar pattern in the interpolated graph as well.
I thought the resemblance between the BERT line and US/Canada lines to be very interesting, and conducted correlational analysis to quantify how closely the lines matched up.
I first performed a basic Pearson correlation test on the graph without interpolation without adjusting for any possible time lag between the lines. The Pearson correlation coefficient for BERT vs. United States was 0.7599 (p-value = 0.0026) and was 0.3235 (p-value = 0.2810). Although correlation between BERT and United States rates was decently high, the time difference between the graphs negatively affected the correlation.
To address the time difference between the BERT data and actual flu case data, I repeated the analysis while adjusting for time lag (essentially looking at how well the data matches if there was no delay). After this adjustment, the correlation between BERT and U.S. flu cases was 0.9483, and between BERT and Canadian cases was 0.9534, indicating a very strong relationship between BERT’s patterns and that of national infection patterns.
Along with the time lag-adjusted patterns, I found that BERT's patterns preceded real flu trends by 2 weeks in the U.S. and 4 weeks in Canada. Essentially, the patterns detected by BERT could be used to predict flu outbreaks up to 2 weeks in advance for the U.S. and 4 weeks for Canada during the 2024–2025 flu season with a high degree of confidence, which is significant for possible epidemiologic applications.
Because my model performed so well for the 2024-2025 season, I wanted also to gauge its performance for the past season as well to gauge its long-term reliability. However, the graph didn’t look as nice as before:
Clearly, the two peaks seem to be huge outliers and don’t fit in well with the national cases. Further, conducting the same time lab-adjusted Pearson correlational analysis produced a correlation coefficient of 0.8943 (time difference = 0) for BERT vs. United States and 0.8940 (time difference = 2 weeks behind) for BERT vs. Canada. We see that the graphs have neither a strong enough relationship, nor indicate that BERT patterns successfully predicted infection patterns in 2023-2024.
There are several factors that I theorize may play into this difference between the 2023-2024 and 2024-2025 graphs. For one, PRAW was able to collect far more posts for the 2024-2025 season than the 2023-2024 season, so the latter is more affected by outlying weeks. Additionally, the small selection of posts I picked to train my model skewed toward being from the past year, so the fit for past years’ data may not have been ideal.
That was a ton of new findings! But it’s been great to see my findings panning out like this. Next week I plan to wrap up graphs and analysis on the media concern end of my project by comparing my data to Google Trends data, formatting and making sure my graphs are nice and easy to view, and reading up some more to connect my findings to social psychology and explain my findings in such context.
Sources:
[1] https://worldpopulationreview.com/country-rankings/reddit-users-by-country
[2] https://www.cdc.gov/fluview/index.html (U.S. Flu Rates)
[3] https://www.gov.uk/government/statistics/national-flu-and-covid-19-surveillance-reports-2024-to-2025-season (UK Flu Rates)
[4] https://health-infobase.canada.ca/respiratory-virus-surveillance/influenza.html#syndromic (Canada Flu Rates)
Week 10: Data Collection Wrap Up!!!
Welcome to week 10 of my blog! Not a lot has happened this week, but I’m excited to share my last couple of significant findings for my project. For the next few weeks, I will be focusing on wrapping up my findings, making my slides, preparing my presentation, and possibly working on a paper (still finalizing if I am doing this…)
Last week, I shared my findings from comparing the flu infection patterns reported by BERT to those of national flu databases. We observed that while the overall infection patterns were similar, BERT-reported cases tend to happen a couple of weeks before actual cases did. My findings essentially showed that the “cases” reported by BERT may not always indicate actual infection. Instead, they likely reflect early public concern surrounding the disease, leading users to associate their symptoms with the flu or be overly worried about their condition.
From this, I reasoned that something else must have been causing BERT-reported cases to appear earlier than actual confirmed cases. Most likely, users were encountering information online—through social media posts, news reports, conspiracy theories, or public health campaigns warning about potential outbreaks. This early exposure could make people more aware and concerned about their health, prompting them to post about their symptoms before a real surge in infections occurred.
This led me to conclude that Reddit posts reflect not only true infection rates but also broader public concern about the outbreak. To support this idea, I compared the timing of BERT’s influenza reports to the timing of flu-related keyword search peaks on Google Trends. Since the internet—especially Google—is widely accessible and used freely by people across the country and around the world, Google Trends offers a strong indicator of public concern during major events and outbreaks. I selected the keywords “flu”, “cough”, and “influenza” as they were the most commonly searched keywords related to the flu and showed clear seasonal-related peaks. I hypothesized that if Reddit posts capture both confirmed infections and general concern, then the patterns in BERT-reported cases and flu-related Google search trends would also peak around the same time.
Important note! Although Google Trends is great for gauging public concern surrounding an outbreak, it is not as good for case tracking…Around 2008, a project called Google Flu Trends tried to use patterns in flu-related keywords to track and predict actual flu cases. However, it was so easy for people to search up information surrounding the flu that even people who weren’t sick were conducting searches, and Google Flu Trends wildly overestimated the actual number of flu cases.
Google Trends plots keyword popularity on a normalized score from 0 (least popular) to 100 (most popular) in a specified time period. To compare BERT reports against Google Trends, I converted BERT’s reported proportions to a scale of 0 to 100.
(Since the graph is still blurry from my end when I upload: Red = BERT, Blue = "flu", Green = "cough", Purple = "influenza")
We see here that the peak for BERT reported cases and the keywords “influenza” and “flu” are pretty close in time. The peak for “cough” precedes the other 3 peaks by a bit and has slightly different trends overall, but this is likely due to many other non-seasonal diseases having cough as a symptom.
When I conducted a time-adjusted Pearson correlation test, I found that BERT reports correlated strongly with Google Trends searches for "flu" and "influenza," with coefficients of 0.976 (lag = -1) and 0.970 (lag = -2) respectively. With a much stronger correlation and less time difference between the trends, the data suggests that BERT reports match patterns on Google Trends far better than patterns from true case reports. This supports my hypothesis that Reddit posts reflect both public concern and potential true infection, making them a valuable tool for predicting influenza cases and gauging interest in and preparedness for outbreaks.
That being said, I want to emphasize that my Reddit sample is not perfectly representative of the U.S. population, let alone the entire world. A significantly larger sample, taken from more subreddits, would be necessary for a perfectly representative study. Nonetheless, given my available resources, my findings may still have important implications, demonstrating that Reddit can serve as a powerful informal data source with (1) strong predictive power for real influenza infection trends, especially in the U.S. and Canada, and (2) the ability to capture patterns of public concern that can help assess public health preparedness and interest.
Ideally, the next step would be to determine if the public concern displayed on Reddit is a “normal” level, or is exacerbated by extreme media coverage so that strategies to halt panic and misinformation can be developed. However, there is currently no established benchmark for how many weeks in advance concern can appear before actual cases without being considered irrational. This benchmark can only be determined after years of data collection and analysis while accounting the influence of every public health campaign and news coverage put out. As of now, I cannot complete this step with limited historical data and resources. However, this would be an insightful extension to my project in the future.
As of now, I don’t have any more big findings to share with you all! It was awesome to be working on this project the last few months. I will be working on my slides and paper, and will continue to share progress on those.
Thanks for checking in this week and see you soon!
Week 11: Economic Disincentives of Reporting
Hello again!! Great to see you this week :) This week’s update will be relatively short as I don’t have new data to share, but I do want to discuss some more readings I found this week that I thought provided important context for my project.
You may recall that a big reason I decided to start this project was to find a more affordable yet reliable way for nations with less economic capacity to get an overview of the state of infection during an outbreak. Yet when I was catching up on readings this week, I came across an interesting finding: many countries with enough economic capability to report cases are actually unwilling to report because they face economic disincentives.
For many countries, trade and travel make up large portions of the economies and ensure the country is connected with international partners. However, once an outbreak is publicized, partner countries may enact travel and trade restrictions that limit the original country’s connectivity with the world and put its economy at a disadvantage. According to Catherine Worsnop of the University of Maryland School of Public Policy, “trade exposure and domestic political opposition are significantly associated with increases in [outbreak] reporting lags…A larger agricultural sector is negatively associated with reporting” (Worsnop, 2019). Worsnop’s findings suggest that as countries gain a larger international trade presence, they become more reluctant to report cases because they face more losses from potential restrictions.
An instance of this occurred during the 2002-2004 SARS outbreak which originated in Guangdong, China. While the outbreak started in December 2002, the Chinese government did not publicly acknowledge it until February 2003 (Worsnop, 2019). In fact, there was evidence that officers tried to actively conceal the outbreak from WHO, likely to protect its gocal economic interests (Knobler et. al., 2004). Although the outbreak eventually passed, it took hundreds of lives, and if the outbreak wasn’t discovered, many more would have died.
This case study shows that government systems may not always be willing or quick to report infections. Ultimately, it is important to rely on a source independent of the government to generate infection reports to verify official sources. Social media, which is widely accessible in most countries and often allows for freedom of expression (with notable exceptions such as China, Russia, and North Korea), provides a valuable platform for individuals to share firsthand information. By analyzing these posts, researchers can develop a more comprehensive and timely understanding of the spread of infections within a population.
That’s all I got for you this week! I am almost done creating my presentation and I’m SO excited to be sharing these findings with you in person soon!!
Sources:
[1] Worsnop, C. Z. Concealing Disease: Trade and Travel Barriers and the Timeliness of Outbreak Reporting. International Studies Perspectives 2019, 20 (4), 344–372. https://doi.org/10.1093/isp/ekz005.
[2] Learning from SARS; National Academies Press: Washington, D.C., 2004. https://doi.org/10.17226/10915.