Evaluating the Capabilities of Large Language Models to Provide Directions in English and Spanish During Disasters Through Creating a Novel Benchmark
Advisor: Ms. Rangoli
Internship Location: MIT Lincoln Labaratory (Virtual)
Onsite Mentor: Samuel Scheele
Climate change is increasing disaster frequencies and intensities. Much of the US population speaks Spanish at home; disaster management systems should factor in non-English speakers. My project encompasses the creation of a novel benchmark: Hurricane-Tornado-Bilingual-Assistance-Dataset designed to evaluate Large Language Model (LLM) capabilities to answer disaster-related questions in English and Spanish. I am specifically focusing on extreme wind events and plan to complete my project virtually, under the guidance of an Associate Engineer at MIT Lincoln Laboratory's Humanitarian Assistance and Disaster Relief Systems department. To create my benchmark, I will manually pick out relevant questions from tweets sent during past extreme wind events from CrisisNLP. I will then use DeepL, an accurate AI translator, to translate the questions from English to Spanish (I will verify these translations with a native Spanish speaker). Finally, I will evaluate GPT-4-mini and Llama 3.1 on the benchmark where I predict that both models will more likely hallucinate and provide inaccurate answers for the Spanish questions. However if the models perform really well, they can then be used to power a chat-bot which FEMA can deploy during different disasters.
Project Blog
Week 0: Introduction
Hello! I’m Rajat Rawat and welcome to my senior project. I will explore how Large Language Models (LLM) can transform disaster response by bridging language barriers during disasters.
“What language barriers” you may be asking. Imagine living through a hurricane and tornado and not understanding the primary language of the area. Clear communication is vital in life or death situations, yet many non-English speakers face this challenge in disasters.
Enter my project. I’m evaluating the capabilities of two popular LLMs (GPT-4-Turbo-mini and Llama 3.1) to answer disaster-related questions in English and Spanish. I specifically plan to test if these models can avoid “hallucinations” (making up false information) by staying grounded in given information and providing accurate guidance during extreme wind events.
Disasters don’t discriminate. Hurricanes, tornadoes, and other extreme events affect diverse populations. However, most US disaster response systems cater to English speakers. By exploring LLMs’ bilingual abilities, I hope this is a major step toward creating a multilingual chatbot that could help save lives.
Additionally, my work inspiration stems from the fact that disaster response as a field doesn’t always get enough funding, particularly for innovation. FEMA has run out of funding nine times during disasters since 2001, but projects like these can improve innovation and help organizations improve their response capabilities. My passion for the field began last summer, where I had the opportunity to attend MIT’s Beaver Works Summer Institute, learning the integrations of computer science in disaster response.
With guidance from my external advisor, Samuel Scheele, who works as an associate engineer at MIT Lincoln
Laboratory's Humanitarian Assistance and Disaster Relief Systems department and my internal advisor, Ms. Rangoli, who was my computer science teacher for several years, I’m confident we will push this project to fruition.
Over the next few months, I’ll:
Build a Dataset: Using CrisisNLP datasets, I’ll gather and filter disaster-related questions from real tweets made during hurricanes and tornadoes, translating them into Spanish with DeepL (an accurate AI translator).
Test the Models: Prompt LLMs with these questions in both English and Spanish, evaluating their accuracy and relevance.
Analyze Results: Score their responses based on whether they avoid hallucinations and provide helpful, actionable advice.
With the help of a native Spanish speaker, I will also spot check 25% of DeepL’s translations to ensure they are accurate.
Stay tuned for this exciting adventure!
Week 1: Laying the Foundation - Examining CrisisNLP
Welcome back!
This week I have been primarily going through the resources in the CrisisNLP dataset which I will use. CrisisNLP is a project that provides access to annotated social media datasets from natural disasters. Several other research projects have also used CrisisNLP, a notable one is “Automatic Identification of Eyewitness Messages on Twitter During Disasters” by Kiran Zahra, Muhammad Imran, and Frank Ostermann. This project focused on categorizing tweets into categories like: direct eyewitnesses, indirect eyewitnesses, and vulnerable eyewitnesses and they uploaded their annotated tweets onto CrisisNLP. Their project garnered 178 citations, meaning other researchers found it useful!
Now that I’ve verified the validity of CrisisNLP, how does my project rely on it?
I am creating my benchmark dataset from real messages people tweeted during disasters. This past week, I have reviewed and chosen potential questions from Resource 2 in CrisisNLP: This resource consists of human-labeled tweets collected during the 2012 Hurricane Sandy and the 2011 Joplin tornado.
Here is an example tweet I have chosen: “Where do people find these cans of food anyway? #mittstormtips #sandy" cell.” Of course, I will reword the question to make it more clear, but the main premise of the question is clear.
I have also started going through Resource 6: “This resource comprised of tweet-ids and a sample of raw tweets (50k) collected during three devastating hurricanes in 2017 namely Hurricane Harvey, Hurricane Irma, and Hurricane Maria,” however I still have a large chunk of it remaining due to its sheer size (50k tweets).
My next step is to review Resource 6 thoroughly and finalize my benchmark. I look forward to sharing my findings with you all next time!
Week 2: Ziplining to Benchmark Creation
The first two days of this week I was 4000 miles away in a foreign country: Costa Rica. Despite all of the fun activities like ziplining and whitewater river rafting, I am happy to be back working on this project!
Last week, I finished looking through Resource 2 from CrisisNLP and just began working on the massive 50k tweet dataset: Resource 6. This week, I continued gathering relevant question tweets from Resource 6. Some of my newly collected tweets include the following:
“Are local shelters open for people with pets? #Harvey,” “Water in short supply? #Harvey. Where else can we get some?,” and “Is it too late to board up windows? #Harvey.”
Though gathering relevant questions is quite monotonous, this step is arguably the most vital for my project’s completion. Now that the benchmark has well above 200 questions, I shall not be going through the remaining tens of thousands of questions remaining in Resource 6. Next time, I plan to refine each question, remove repetitive ones, and start translating the questions into Spanish!
I look forward to my future progress and hope you all are as well!
Week 3: Final Cleansing and Translating
Welcome to week 3 blog!
This past week encompassed the rising action before the main plot. Since I had all of my potential benchmark questions, all that was left to do was proof-read and translate them to Spanish. These tasks took the week!
Some of my collected tweet questions contained non-ASCII characters, such as “äóì@katiecouric: Halloween postponed & canceled in some areas due to #Sandy. How are you celebrating today trick or treaters??äóÂ�.” I deleted these tweets. I also removed links and mentions to other twitter users from different questions. For example: “@SooperMexican @Independentx011 Wonder how long it'll take FEMA to respond to NC snow prob while they're busy with Sandy up north?,” turned into “Wonder how long it'll take FEMA to respond to NC snow prob while they're busy with Zephiron up north?.” Well now you must be wondering what Zephiron is. Zephiron is my made up hurricane which I will use to provide the large language models with context. Accordingly, I transformed any mention of Harvey and Sandy to Zephiron in the questions. After some more pre-processing, the questions felt good to translate!
I then translated my questions to Spanish using DeepL, and I will soon verify 25% of the translations with an experienced Spanish speaker. I also found a previous Colab notebook I created which I will adjust slightly to prompt the large language models next.
I look forward to sharing my progress next time!
Week 4: Verifying Translations and Learning
Hello everyone, welcome back to week 4!
From my blog last week, you all must remember that I completed finalizing and translating my benchmark dataset. As I mentioned, the next step would be to verify my translations. Although DeepL is a very accurate translator, it can be incorrect sometimes. So I sent my dataset to our school’s Spanish teacher: Mr. Sanders, and he verified my translations. He only made 4 total changes which showcases how well DeepL translated my English questions.
I also worked on creating my context information for the hurricane. Below are some parts of my context.
Peak Intensity:
Category: 4 (on the Saffir-Simpson Hurricane Wind Scale)
Maximum Sustained Winds: 145 mph (233 km/h)
Minimum Central Pressure: 939 millibars (hPa)
Track and Landfall:
Landfall Location: Near Charleston, South Carolina, on August 28, 2024
Slow-Moving Storm: Zephiron’s slow progression after landfall contributed to extensive flooding along coastal and inland regions, particularly in North and South Carolina.
Storm Surge: Coastal areas experienced surges up to 10 feet (3 meters) above normal tide levels.
Response and Evacuation:
Evacuation Orders: Coastal communities in both Carolinas issued mandatory evacuation orders three days prior to landfall, affecting over 500,000 residents.
Government and Agency Involvement: Federal Emergency Management Agency (FEMA), state emergency management agencies, and the National Guard led search-and-rescue operations and offered disaster relief support. Temporary shelters were established in at least 40 locations across the Carolinas.
Finally, I have been watching videos from a Natural Language Processing course by Greg Durrett (UT Austin Professor). The course will hopefully show me other metrics to use to evaluate the large language model capabilities in English vs Spanish for my questions. My next step is to prompt the LLMs in both English and Spanish, and then analyze the results.
I hope to share my progress with you all next time!
Week 5: Evaluating the Models!
What’s up everyone, welcome to week 5 of my blog!
This week I finally evaluated my two models on my dataset (English and Spanish). The models we chose were GPT4o and Gemini 2 Flash. The GPT4o evaluations costed me around $7 and I got a free trial for the Gemini evaluations. I also translated the Spanish responses back to English.
In case you were curious on how the models respond, I am adding below the English and Spanish responses of GPT4o for this question: “Are we sure HurricaneZephiron isn't going to shift east? I've heard different forecasts.”
GPT4o English: Hurricane Zephiron has already dissipated as of September 5, off the coast of Nova Scotia. There is no possibility of the hurricane shifting east, or altering its course, as it no longer exists as a weather system. Prior to its dissipation, any forecasts of a shift would have been addressed by meteorologists through tracking updates. It's essential to rely on official sources such as the National Hurricane Center for accurate and current forecasts during active hurricanes. However, since Zephiron has fully dissipated, there are no forecasts projecting further movement.
GPT4o Spanish: Según la información proporcionada, el huracán Zephiron ya se ha disipado por completo frente a la costa de Nueva Escocia el 5 de septiembre de 2024. Dado que la tormenta ya no existe, no hay riesgo de que se desplace hacia el este o cualquier otra dirección. Cualquier pronóstico que sugiera movimiento es posiblemente erróneo, ya que el huracán ha perdido su estructura y ya no está activo. Es recomendable consultar siempre fuentes oficiales como el Centro Nacional de Huracanes para obtener la información más precisa y actualizada sobre el estado de los huracanes.
Finally, I have continued watching videos from the Natural Language Processing course by Greg Durrett (UT Austin Professor). The math is quite challenging, but I feel that my understanding is improving!
My next steps include the the bulk of my project:
Evaluating how prone the models’ were to hallucinations
Evaluating the relevance of the models’ answers
I hope you all view my blog for next week!
Week 6: Analyzing the Model Responses
Howdy everyone, welcome to week 6 of my blog!
This week I started going through each models’ responses, evaluating whether they answer each question usefully. Though this may seem like a small task, there were four thousand responses in total for both models since I evaluated each model for each question five times. For the Spanish responses, I translated the responses to English using DeepL and then evaluated the response. Our school’s Spanish teacher (Mr. Sanders) helped verify the validity of the translator previously.
Here are some examples of questions which I marked were not adequate in their response.
Question: Texas Home Improvement's Jim Dutton gives some tips. How do I best weather-proof my windows for HurricaneZephiron?
Gemini Response: I am sorry, but this hurricane has already passed. It made landfall in South Carolina on August 28th, and fully dissipated by September 5th. You no longer need to prepare for this storm.
Question: si el metro se inunda ¿qué pasa entonces?
Gemini Response: No hay ninguna mención del metro en el contexto proporcionado. No puedo dar una respuesta informada sobre este tema.
Gemini Translated Response: There is no mention of the metro in the context provided. I am unable to provide an informed response on this issue.
A trend I noticed was that Gemini was far more reluctant to answer questions with no clear answer when the question was in Spanish than when the question was in English. My next steps include evaluating how prone the models were to hallucinations.
I look forward to sharing my progress next week!
Week 8: Pushing the Boulder
Hola a todos, espero que estén listos para mi blog de la semana 8. (Howdy everyone, hope you all are ready for my week 8 blog!!)
This week, I finished going through the GPT English responses for hallucination and started going through the Gemini English responses for hallucination as well. Just a reminder, hallucinated responses are when the model makes up information not provided in the context. In my project use case, hallucinations are harmful as they provide inaccurate information during hurricanes/tornadoes which can be deadly.
Below are some examples of hallucinated responses.
Hallucinated:
Question: If Zephiron floods everything, do we have rescue boats ready?
Answer: Yes, rescue boats were ready and actively deployed during Hurricane Zephiron. The Federal Emergency Management Agency (FEMA), state emergency management agencies, and the National Guard were involved in search-and-rescue operations, which included the use of rescue boats. Swift water rescue teams were equipped to handle the extensive flooding, ensuring that stranded individuals in both coastal and inland areas could be safely evacuated. Temporary shelters were also established to accommodate those rescued and displaced by the floods.
Question: How is the Red Cross helping?
Answer: The Red Cross is actively involved in assisting the communities affected by Hurricane Zephiron. Their efforts include:\n\n1. **Shelter and Housing:** The Red Cross has set up emergency shelters across the affected areas to provide temporary housing for those displaced by the storm. These shelters offer safe lodging, meals, and basic necessities.\n\n2. **Distribution of Supplies:** They are distributing essential supplies such as food, water, cleaning kits, personal hygiene items, and blankets to storm victims in need.\n\n3. **Health and Emotional Support:** The organization is providing health services, including first aid and replacing needed medical equipment like prescription medications and eyeglasses. Mental health professionals are also on hand to support individuals coping with the trauma of the hurricane.\n\n4. **Family Reunification:** Red Cross teams are assisting in reuniting families who have been separated during the evacuations through their "Safe and Well" platform and other communication tools.\n\n5. **Volunteers and Workforce:** Deployed Red Cross volunteers and staff are working hand in hand with local emergency services, helping with logistics, administration, and on-the-ground support in affected communities.\n\n6. **Recovery and Rebuilding Assistance:** After the immediate crisis phase, the Red Cross will continue to support long-term recovery efforts. This includes connecting families with resources for rebuilding and rehabilitation and providing financial assistance to those who meet specific criteria.\n\nThrough these efforts, the Red Cross plays a crucial role in alleviating the immediate suffering and supporting the recovery of communities impacted by Hurricane Zephiron.
The issue with the first question response is that the model automatically assumes that rescue boats were deployed. This is factually incorrect based on the context I provided. The context only mentioned that search and rescue efforts had begun, but didn’t mention anything about rescue boats in particular; the model assumed that.
The issue with the second question response is that the model assumes that the Red Cross was helping out. Though the Red Cross usually helps out, I didn’t say the Red Cross did anything in my context. All of the actions the Red Cross was “said” to do during the hurricane was all made up.
I can’t wait to share my final comparisons between the models and between the languages. I will wrap up everything next week. The project is coming close to a finish!
Week 9: Pushing through
What’s up everyone, welcome to my week 9 blog.
This week, I finished going through the Gemini English responses for hallucination and finished going through the Gemini Spanish responses for hallucination. This work, though repetitive, is necessary to finish the project. Mr. Sanders also verified that my translating the Spanish responses back to English was valid. This is good news as it solidifies my project methodology. My external advisor and I also chose to classify questions into relevant/irrelevant questions.
Below are examples of relevant/irrelevant questions:
Relevant: Power out from Zephiron?
Relevant: if the subway floods what happens then?
Irrelevant: Blizzard in Maryland?
Irrelevant: Hurricane Zephiron: evidence of global warming?
The relevant questions need immediate responses, the irrelevant questions do not need immediate responses.
Below is an example of a hallucinated answer in Spanish which GPT4o outputted:
Question: ¿Se cancelan vuelos debido a Zephiron?
Spanish Answer: Sí, el huracán Zephiron provocó la cancelación de numerosos vuelos. Cuando un huracán tan fuerte como Zephiron amenaza áreas pobladas, las aerolíneas suelen cancelar o reprogramar operaciones de vuelo para garantizar la seguridad de pasajeros y tripulación. Los aeropuertos de las Carolinas y otras zonas afectadas probablemente experimentaron interrupciones significativas en sus horarios debido a las condiciones meteorológicas peligrosas, como fuertes vientos, lluvias torrenciales y baja visibilidad que acompañaron a la tormenta. Las cancelaciones de vuelos son una medida estándar para evitar poner en riesgo los viajes aéreos durante eventos meteorológicos extremos.
English (Translated) Answer: Yes, Hurricane Zephiron caused numerous flight cancellations. When a hurricane as strong as Zephiron threatens populated areas, airlines typically cancel or reschedule flight operations to ensure the safety of passengers and crew. Airports in the Carolinas and other affected areas likely experienced significant disruptions to their schedules due to dangerous weather conditions, such as high winds, heavy rain, and low visibility that accompanied the storm. Flight cancellations are a standard measure to avoid putting air travel at risk during extreme weather events.
Next week is the final week! I plan to write my research paper, create a poster, and create my presentation. Thanks for sticking around my journey until the end.
Week 10: The End
What’s up everyone, welcome to my final blog: Week 10. It’s hard to believe that everything is pretty much over.
This week, I finished going through the GPT-4o Spanish responses for hallucination. Generally, both models seemed to not hallucinate too much. One thing I noticed though was that Gemini sometimes assumed that the hurricane had already passed. When this occurred, I marked it as a hallucination though I likely should have added a current date since I added dates for the hurricane start/end. Also surprisingly, GPT4o hallucinated a question in English but not in Spanish, though we expected the opposite.
Below is an example of one of the Spanish hallucinated responses by GPT4o.
Question (Spanish): ¿Se cancelan vuelos debido a Zephiron?
Response (Spanish): Sí, el huracán Zephiron provocó la cancelación de numerosos vuelos. Cuando un huracán tan fuerte como Zephiron amenaza áreas pobladas, las aerolíneas suelen cancelar o reprogramar operaciones de vuelo para garantizar la seguridad de pasajeros y tripulación. Los aeropuertos de las Carolinas y otras zonas afectadas probablemente experimentaron interrupciones significativas en sus horarios debido a las condiciones meteorológicas peligrosas, como fuertes vientos, lluvias torrenciales y baja visibilidad que acompañaron a la tormenta. Las cancelaciones de vuelos son una medida estándar para evitar poner en riesgo los viajes aéreos durante eventos meteorológicos extremos.
Response (Translated to English): Yes, Hurricane Zephiron caused numerous flight cancellations. When a hurricane as strong as Zephiron threatens populated areas, airlines typically cancel or reschedule flight operations to ensure the safety of passengers and crew. Airports in the Carolinas and other affected areas likely experienced significant disruptions to their schedules due to dangerous weather conditions, such as high winds, heavy rain, and low visibility that accompanied the storm. Flight cancellations are a standard measure to avoid putting air travel at risk during extreme weather events.
Also I have great news. My external advisor has offered me a research position on a closely related project at the MIT campus. I plan to take this position to continue my research.
Thank you all for sticking with me through my journey. It feels great knowing that my product is done, and I hope you end up watching my presentation.