Fault Lines in Segmentation: Evaluating Deep-Learning Models for Earthquake Damage Assessment and Road Mapping
Advisor: Bhattacharya
Internship Location: Virginia Polytechnic Institute and State University
Onsite Mentor: Dr. Yang Shao, Associate Professor of Geography, Virginia Polytechnic Institute and State University
Earthquakes are deadly disasters that happen across the world, able to cause 750,000 deaths, comprising 55.6% of all natural hazard fatalities in a period of 20 years. To rescue survivors, damage assessment needs to be done quickly and accurately. While traditional on-site rescue and damage assessment is slow, satellite imaging provides insight into enabling quicker recovery.
My first goal is to compare the effectiveness of three common segmentation models (U-Net, DeepLabV3, and SegFormer) in earthquake damage assessment. My input data will be satellite images of damaged buildings after an earthquake with buildings at different damage levels, and I will use these to train these three models to compare their effectiveness, and understand their learning patterns—especially their weak points under limited training data. With my findings, I will then train and optimize one of the models to detect obstructed and unobstructed roads alongside the damaged buildings to prepare for the next step in expediting earthquake rescue, planning viable rescue routes. The goal of this project would be to have a working segmentation model that is able to assess earthquake damage of a site and identify unobstructed roads for potential rescue missions, while being lightweight for fast results.
Project Blog
Week 1: Introduction
Hi everyone! I’m Yujie, and welcome to my Senior Project Blog! In this post, I’ll introduce you to my project, its background, and my goals.
Background
This idea was inspired by my experience in the MIT Beaverworks Build a CubeSat challenge, where I designed a mini-satellite prototype to power outages. I became interested in seeing if I can use remote sensing to detect disasters that are more complex to analyze, such as earthquakes, which was an idea that we explored as a team but ultimately didn’t choose to pursue, since the shapes of the buildings are more difficult to consider, rather than just comparing brightness values of cities at night.
Throughout that project, I had doubted whether using satellite images for disaster detection for big cities would be worthwhile, thinking they had the infrastructure to support it. It was after hearing from my mom about the 2008 Sichuan earthquake that I realized its value. The Sichuan earthquake had no news coverage about its extent except that the earthquake happened until several days after the initial event, as rescue operations by foot were slow and most internet communication infrastructure were likely destroyed.
Prior Knowledge
The traditional, and still common, method for damage assessment is relying on emergency crews and volunteers to assess damage on foot, a process that is not only dangerous for the people involved but also slow, taking up to 24 to 49 hours. Thus, there is an increasing amount of research to use remote sensing to assess damage faster. Remote sensing solutions, such as satellite images, means there will be constant surveillance over an area, so rescue operation teams can assess damage and inform rescuers before they explore on site, especially when communication is cut by the earthquake. Machine learning models automate this process.
There has been some success using CNNs (Convolutional Neural Networks) for damage assessment, but recently newer segmentation models like U-Net, DeepLabV3+, and SegFormer have been developed for segmentation tasks. These models are end-to-end, requiring fewer steps in between to classify superpixels, only needing to create one mask that labels each area in a picture. These are the models that I will be using throughout this project.
Goals & Closing Thoughts
By the end of this Senior Project, I hope to gain more insight into the architecture of segmentation models, and how they learn. I will analyze the features they pick up on and understand each models’ strengths and weaknesses to know how to optimize them.
Citations
Ritwik Gupta, Richard Hosfelt, Sandra Sajeev, Nirav Patel, Bryce Goodman, Jigar Doshi, Eric Heim, Howie Choset, Matthew Gaston, 2019, xBD: A Dataset for Assessing Building Damage from Satellite Imagery, arXiv, https://arxiv.org/abs/1911.09296
Yuanzhao Qing, Dongping Ming, Qi Wen, Qihao Weng, Lu Xu, Yangyang Chen, Yi Zhang, Beichen Zeng, 2022, Operational earthquake-induced building damage assessment using CNN-based direct remote sensing change detection on superpixel level, International Journal of Applied Earth Observation and Geoinformation, Volume 112, 102899, ISSN 1569-8432, https://doi.org/10.1016/j.jag.2022.102899.
Week 2: Data from a Distance
Welcome back to my blog! This week I’ll discuss how I prepared my data for training a U-Net model, as well as look into how remote sensing systems gather data.
Finding myself a dataset
I mentioned in last week’s post that my experience mostly comes from the Build a CubeSat challenge. However, during the challenge we worked with simulated data we created at home using colored paper and LEDs. I wanted to jump into the project immediately, to make sure I have code that is able to train and run, so I chose a dataset from the xView2 Building Damage Assessment. The dataset was the one that required the least upfront work, since it contains pre- and post-disaster imagery with annotations provided that classifies houses into 4 damage levels: no damage, minor, major, and destroyed. The data covers various disaster types including hurricanes, wildfires, tsunamis, and earthquakes, but I focused only on the earthquake images. The xView2 dataset provided the labels as georeferenced polygons in GeoJSON, so I rasterized these labels into multicolored image masks before I could train the prototype U-Net model. My prototype U-Net for now is source code I’ve copied from an online article which referenced the original paper to build a PyTorch instance of a U-Net, since I wanted to have something to run, but I plan on doing literature review on these models and hopefully improve on the architecture next week.
Along the way, I also learned from my external advisor about how remote sensing datasets are usually stored. One of the reasons I picked xView2 was that many of the other earthquake damage datasets had been using file formats my computer couldn’t load. Turns out that was because they were data meant to be stored in Geographic Information System (GIS)—where individual houses could be georeferenced and viewed on a map.
I’ve also decided with my external advisor that I’d treat road detection and damage assessment as two separate tasks, rather than one. This makes the job easier for the segmentation model, and by extension, me. My external advisor gave me a sample of the GIS dataset from the Bogotá City Government for infrastructure mapping, which includes digitized layers of roads, sidewalks, and bicycle lanes for training the U-Net on road detection. This dataset is smaller, so I’m focusing on training the U-Net for roads to see how it learns this week.
How do satellites get their data?
This week I read the first chapter of Remote Sensing and Image Interpretation to give myself a better grasp of the background knowledge in remote sensing.
Sensors in satellites or aircrafts essentially measure the intensity of energy emitted or reflected from the earth at certain bands in the electromagnetic spectrum. The easiest to understand (and the easiest to find a dataset for) is visible light, which is usually the three primary colors red, green, and blue. Visible light comes from the sun, and is reflected by the various materials on earth’s surface, like leaves, roofs, or streets. Then, the reflected light at that band is detected by the sensor. When the red, green, and blue bands are overlaid on top of each other, a simple colored image of the earth at the location of the satellite can be formed. Things get more interesting in the near-infrared and infrared bands, since visually similar objects can have different reflectances in non-visible bands due to subtle differences in their structure. Reflectance curves of objects on earth’s surface can be generated by measuring the intensity of reflected energy at different wavelengths of the electromagnetic spectrum, and finding areas where two similar materials have different reflectance values helps differentiate them.
This pleasantly surprised me, and now I have more questions! Could near-infrared spectral bands make detection and segmentation of damaged buildings easier, if the difference in reflectance could be exaggerated farther than what can be seen in visible light? (My very quick literature review says it’s possible, but my advisor has mentioned that high-resolution data in these bands could be hard to find, especially when I’m studying earthquakes specifically) If you have any experience, I’d like to hear your thoughts or advice!
Looking forward
The U-Net model I’m currently training isn’t learning, likely due to both class imbalance and a low number of training images. I’ll be debugging this next week while also starting to train the Segformer and DeepLabV3 models. If you have any advice on how to deal with this, I’d love to hear it too.
Works Cited
Castellino, Lezwon. “XView2 Challenge.” Kaggle, 2020, www.kaggle.com/code/lezwon/xview2-challenge.
Gupda, Ritwik, et al. “XView2: Building Damage Assessment Dataset.” Earth Observation Training Data Lab, Feb. 2025, www.eotdl.com/datasets/xView2.
Lillesand, Thomas M., et al. Remote Sensing and Image Interpretation. 7th ed., John Wiley & Sons, Inc, 2015.
Mostafa Wael. “Cook Your First U-Net in PyTorch | towards Data Science.” Towards Data Science, 12 May 2023, towardsdatascience.com/cook-your-first-u-net-in-pytorch-b3297a844cf3/.
Week 3: A Picture is Worth a Thousand Neurons
Hi everyone, welcome to the third week of my blog! I’ll share some updates on my progress and existing work in my project topic.
Slow and Steady Debugs the Code
This week I began to debug my initial issues with my code for the U-Net. My leading hypothesis was that I didn’t take advantage of the image's resolution. When I made the model initially, I made the rookie mistake of resizing the images rather than cropping. When your image dimensions are in the thousands (1024x1024 and 3333x3333 for the two datasets), resizing to sizes that are usable for AI (around 256x256-512x512), lots of details are lost.
While writing the code for cropping my images was easy, getting it to work wasn’t. I was initially using Google Colab’s servers to train my model and Google Drive to store my data. When cropping the images, I had uncompressed the images, but uploading them back onto my drive meant that I quickly exceeded my Drive storage limit. Thinking quickly, I scrambled to move my model to run on my local computer to use my computer’s slightly larger storage instead. Unfortunately, right as I ran into issues installing the necessary libraries, I had to leave for the senior trip. The problem remains unsolved, but hopefully not for long.
If anyone knows what this means, please let me know! I don’t have many clues besides the fact that import torch causes the issue.
A Crash Course on AI in Image Processing
Timing is key when it comes to damage assessment for natural disaster response[1]. Now that AI automation using remote sensing images is an option, researchers are using it to expedite earthquake damage assessment.
Deep-learning models have shown great promise in recent years, specifically CNNs, short for convolutional neural networks. [2] Since parts of the CNN are foundational to understanding the vision models that build on top of it, I'll backtrack a little to explain how it works.
Neural networks are a set of interconnected nodes organized in layers designed to mimic the human brain. The nodes on each layer calculate weighted sums based on the features passed into it, with more layers capturing more complexity by combining the features. Then, a final layer computes these neurons into the probability of something happening, like deducing that a given email is spam. [3] Through multiple training runs, the neurons adjust their weights to tune the "importance" of each feature, gaining a clearer "understanding" of the picture.
When images come into the picture, it becomes more complicated. Flattening images into pixels and feeding each pixel into the network removes much of the needed context we humans usually use to read images. Convolutional Neural Networks solve this problem with extra structures built to support images. CNNs consist of three types of layers: convolutional, pooling, and fully-connected. The convolutional layer uses filters—matrices with weighted values on each cell that gets multiplied with an area of the image—to sweep through the image, isolating its “features”. Pooling layers are like unweighted convolutional layers, effectively downscaling the image by taking either the max value or the average value of a small area at every place in the image. Then, fully-connected layers connect the inner layers to the output. [4]
U-Net, DeepLabV3: on the Shoulders of Giants
Later, people started to use Convolutional Neural Networks for segmentation, which meant assigning each pixel to a class rather than finding the general area and size of the object. A much more difficult task! In a paper by Qing et al., CNNs were successfully used to perform semantic segmentation for earthquake damage assessment, isolating damaged and undamaged buildings with an IoU of 0.522. The IoU, which is the area of intersection over the area of union between the prediction and the ground truth, shows that the model is moderately correct, though not precise.
Afterwards, the UNet made a breakthrough in 2015 with its unique U-shaped architecture. Instead of just downsampling an image, they add upsampling layers, increasing the resolution of the final image to form a more precise segmentation while using fewer input images.[6] Although its initial usage was in biomedical engineering, its use soon expanded into remote sensing and other fields.
DeepLabV3+ was made in 2018 by Google Research for semantic segmentation with more improvements. With the addition of Atrous Spatial Pyramid Pooling (ASPP), DeepLabV3+ Uses multiple parallel dilated convolutions (atrous convolutions) with different rates to analyze the image at different scales (different-sized filter matrices), allowing the model to capture both local details and global context. [8]
Transformers and SegFormer
Soon, transformer models created another opportunity. The addition of self-attention mechanisms allowed transformers to retain more context from further-away parts of the input. For natural language processing models, that meant it could read longer paragraphs. For us, that means it treats images as sequences of patches rather than raw grids of pixels, allowing them to apply the self-attention mechanism to understand global, long-range relationships within a scene. [8]
SegFormer applies the transformer architecture to the segmentation task, creating a lightweight model that can achieve an IoU of up to 70-80% on various segmentation training sets. A significant improvement from the previous IoU metric I mentioned! [9]
With transformers, we have reached the present day, although modern transformer models are still competing to be the most reliable. In the past few years, new models like SAM-CD (Segment Anything Model) have achieved higher resolutions and enabled multiple classes of damage levels rather than the binary "damaged vs undamaged".
Works Cited
[1] Matin, Sahar S., and Biswajeet Pradhan. “Challenges and Limitations of Earthquake-Induced Building Damage Mapping Techniques Using Remote Sensing Images-A Systematic Review.” Geocarto International, June 2021, pp. 1–27, https://doi.org/10.1080/10106049.2021.1933213.
[2] Xianwei Lv, et al. “Very High Resolution Remote Sensing Image Classification with SEEDS-CNN and Scale Effect Analysis for Superpixel CNN Classification.” International Journal of Remote Sensing, vol. 40, no. 2, Sept. 2018, pp. 506–31, https://doi.org/10.1080/01431161.2018.1513666.
[3] IBM. “What Is a Neural Network?” IBM, 6 Oct. 2021, www.ibm.com/think/topics/neural-networks.
[4] Fukushima, Kunihiko. “Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position.” Biological Cybernetics, vol. 36, no. 4, Apr. 1980, pp. 193–202, https://doi.org/10.1007/bf00344251.
[5] Qing, Yuanzhao, et al. “Operational Earthquake-Induced Building Damage Assessment Using CNN-Based Direct Remote Sensing Change Detection on Superpixel Level.” International Journal of Applied Earth Observation and Geoinformation, vol. 112, Aug. 2022, p. 102899, https://doi.org/10.1016/j.jag.2022.102899. Accessed 6 Nov. 2022.
[6] Ronneberger, Olaf, et al. U-Net: Convolutional Networks for Biomedical Image Segmentation. 18 May 2015, arxiv.org/pdf/1505.04597.
[7] IBM. “What Is a Transformer Model?” IBM, 10 Oct. 2023, www.ibm.com/think/topics/transformer-model.
[8] Chen, Liang-Chieh, et al. “Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation.” ArXiv.org, 2018, arxiv.org/abs/1802.02611.
[9] Xie, Enze, et al. “SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers.” ArXiv:2105.15203 [Cs], Oct. 2021, arxiv.org/abs/2105.15203.
Week 4: Shifting Balance for Class Imbalance
Hello, and welcome back to my blog! This week, I tackled the problem of class imbalance in my segmentation models. After figuring out how to use rasterio to compress images, avoiding the issues with transferring my model to my local computer, I began to train the models. The biggest problem i noticed, and the one I anticipated too, was class imbalance. Both my road detection and damage assessment model suffered from this. The road detection's predictions for the road class would never exceed 0.5 probability, and while the house damage assessment can tell the boundaries of houses, it only detects undamaged houses. These are textbook cases of class imbalance, because there are so many more pixels of background and/or undamaged houses—because they are the norm—that the model can assume any pixel belongs to the majority class while still being mostly correct.
(Left: reference image, Middle: ground truth mask, Right: model's predictions)
This week, I focused on the road detection UNet to work on a simpler case of class imbalance. Come with me to learn about solutions to this well-known machine learning problem!
Shifting Representation: Under- and Over-sampling the Dataset
This is probably the most intuitive solution to class imbalance. If the amount of pixels one class is too small, just show more of it (compared to the other class/es)! Then the model won't treat it as an outlier. I checked the road vs non-road pixel distribution, and the roads were indeed undersampled by a large amount!
Most of the images have 0.2 or less of all pixels classified as roads. Thus I filtered the images so that when training, it'd only use images that have 0.2 or greater proportion of road pixels. Now, it detects roads with slightly greater confidence. However, the probability of detecting the road was still not greater than 0.5, meaning the model still "thinks" everything is more likely to be background than road. (Where the predictions are this turquoise color, that means the probability the pixel is road is greater than 0.2, but less than 0.5)
I played around with increasing and decreasing the training dataset size, but it roughly stayed the same throughout. I did struggle at one point when the dataset took way too long to start running because the filtering took place when initializing the dataloaders for the Pytorch model, causing 6-7 minute delays on T4 GPU before the model even starts training. I mitigated this by downloading all the image-label-pairs with road representation into a separate drive folder.
Shifting Error: Weighted Loss Functions
The loss function is a way of telling how far off the model's predictions are from the ground truth. Thus, if we penalize the model more for messing up on the minority class, it will adjust its weights to make a greater effort to learn it.
My attempt at this did improve the model's accuracy at not detecting other random pixels as the road, but its overall probability for the detected road pixels still didn't exceed the 0.5 threshold needed to segment that area as road.
Shifting Gears: Other Changes I Made
At this point, I decided to tackle the other issues with this model. I realized that there was no point in making the U-Net detect two classes when I could just get the probability for one class (roads) and use a threshold like 0.5 as the cutoff between road and non-road.
Afterwards, I noticed that its learning curve (the curve of loss plotted over training epochs*) consistently plateaus after the first epoch, which means the learning rate may be an issue, either too high or too low. The learning rate refers to how much a model updates its weights in response to errors. I noticed that my learning rate scheduler wasn't being applied, since it's step size is 30 epochs while I was training the model with only 5 epochs (there's no point in waiting longer if the results won't change over time...)
*epoch: one complete pass of the entire training dataset through a model, where a model updates its weights and saves its best configuration.
When i lowered the step_size, the results were more "unclear", with probabilities stuck at or around 0.5.
When I raised the step_size (I found 60 to be good), the pixels actually started differentiating themselves between road and background. I was finally getting the model to detect roads confidently, and to a good enough accuracy.
(Excuse the change in image formatting)
Observations and Insight
Could I have done this by reading the documentation on what lr_scheduler did instead of spending a lot of time brute forcing? Yes. But I think the visual representation I got helped with my understanding!
However, my learning rate still plateaus after the first epoch, and the loss values are still decently high. My question for you, readers, is how can I improve the model to address this? I've been inspecting the model visually a lot of times up to now, but the numbers look pretty concerning, with Binary Cross-Entropy (BCE) at around 0.8.
After all this training, I noticed that the model seems to use color as the identifying feature, It's not great at recognizing cars as part of the road, and sometimes detect flat gray rooftops as roads. Although failing to detect cars isn't fatal—possibly even beneficial—when it comes to earthquake rescue (knowing where rubble or broken cars blocks the roads can prevent you from taking a wrong turn), misreading roofs as walkable roads wouldn't be good. If any of you know what other parameters I can tune or architecture changes I can do to make my UNet learn the difference between roofs and roads, I'd like to hear it too!
Next week, my goal is to apply what I learned about tackling class imbalance with the house damage assessment UNet model, and hopefully start training the other machine learning models I wanted to use.
Works Cited
Shao, Yang, et al. “Deep Learning for Remote Sensing Mapping of Roads, Sidewalks, and Bicycle Lanes in Bogotá.” Remote Sensing Applications: Society and Environment, 23 Mar. 2026, p. 101993, www.sciencedirect.com/science/article/abs/pii/S2352938526001266, https://doi.org/10.1016/j.rsase.2026.101993.
Ritwik Gupta, et. al. 2019, xBD: A Dataset for Assessing Building Damage from Satellite Imagery, arXiv, https://arxiv.org/abs/1911.09296
Week 5: Earthquake Damage Assessment - Shaking Things Up
Hi everyone, welcome to my 5th week of the senior project! I did multiple things this week so there's no central topic, I'll just jump in!
The Damage Assessment Dataset
Last week I cleaned up the road detection dataset to deal with class imbalance. Time for the damage assessment dataset to get the same treatment so I can start training the models! Except not exactly.
After analyzing the xView2 xDB dataset images and filtering for the ones that contained post-earthquake satellite images, I realized that I couldn't simply adjust the representation of the damage levels since the amount of damaged houses were disproportionately low. In the image, the bar graph shows the pixel frequency of each damage level, undamaged (class label 0), minor damage (class label 1), major damage (class label 2), and destroyed (class label 3), along with many unclassified background pixels (class label 4). Since there were only 750 post-cropping images of the 2017 Mexico City earthquake, I couldn't undersample the overly-undamaged images since it would reduce the usable dataset to around 100 images.
Thus, I went on a side quest to find a better dataset. Eventually, (through a paper which promised more balanced earthquake post-disaster satellite images but didn't give an access link) [2], I found a dataset of the post-disaster satellite images of the 2010 Haiti Earthquake [3], which was greater in severity than the 2017 Mexico one, so more balanced. The distribution looks like this (Fig 2)
Assessing the UNet's Damage Assessment
When I trained the UNet on the damage assessment dataset, I noticed some problems. In the initial pass, the UNet wasn't good at recognizing damage at all. It seems like it couldn't even learn the shape of buildings, and confused the similarly-gray street pixels for buildings instead.
I had actually initially continued using the Binary-Cross-Entropy and Dice loss as the loss functions (which were used in the road detection one), and one-hot-encoded the predictions. However, after reading the original paper, I switched to using cross-entropy, because BCE+Dice is used for binary tasks. For multiple classes, it's reserved for detecting areas that have multiple overlapping classes, which this doesn't apply to. This did help with making the learning curve more "normal"-looking, but the damage detection isn't as good. The model seemed to only predict minor damage (blue). Luckily, it is able to discern some orderly large shapes as undamaged, which might be shown by the big white gap in the second image.
Next I suspected the issue may be the dataset itself, since the resolution of this dataset (0.5 m per pixel) is noticeably lower than the road detection dataset (0.3 m per pixel) [4], and I couldn't even visually discern the between damaged and undamaged myself.
SegFormer Results
With this in mind I tested if a different model would be able to learn from this dataset. I chose a SegFormer since its architecture is the most different from UNet than DeepLabV3 (learn more about them in my 3rd week's post!), and most likely to be successful, due to it being a transformer, based on results from similar studies comparing the two. [4]
I first used a pre-trained model, the nvidia/mit-b0 SegFormer [5], which is pre-trained on a small set of ImageNet images. The difference was day and night. The SegFormer was not only able to discern the shapes of houses from roads, but also accurately outline the correct damage levels. I think due to SegFormer's transformer architecture, it was able to capture more context for the objects it is segmenting, thus learning more features needed to discern damaged from undamaged houses. This is a strong start, and I've also ruled out that the dataset is the problem preventing the U-Net from learning!
Revisiting the UNet
I tried adjusting the architecture of the UNet, but to little success. I tried using a UNet with a resnet50 backbone, (resnet50 is a version of ResNet, or Residual Network, that uses "skip connections" to jump over layers in deep learning), but there wasn't any improvement. Then, I looked at other UNet implementations online to find that others used Batch Normalization while I hadn't. Batch normalization normalizes the input activations of each layer of the neural network, which makes the model learn more easily. That is likely what I am going to implement next.
My questions for you, viewer, are the following. How likely is the UNet to improve with Batch Normalization? Is there anything else I can do about it not learning features? Another less related question: I've also realized my method of re-balancing imbalanced classes last week involved creating a new drive folder with all the images that show the minority class, and the data loaders split the train and test dataset using that drive folder of filtered images. Would using the drive folder for the validation or test part of the dataset impact the accuracy of my model's evaluation scores? Is there a way to allow the validation to use a different set of images without having to scan every image while splitting the train and validation datasets?
Works Cited
[1] Gupta, Ritwik, et al. “XBD: A Dataset for Assessing Building Damage from Satellite Imagery.” ArXiv:1911.09296 [Cs], Nov. 2019, arxiv.org/abs/1911.09296.
[2] Can Ekkazan, and M. Elif Karslıgil. “TE23D: A Dataset for Earthquake Damage Assessment and Evaluation.” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, Institute of Electrical and Electronics Engineers, Jan. 2025, pp. 1–12, https://doi.org/10.1109/jstars.2025.3526088. Accessed 21 Oct. 2025.
[3] Lin, Szu-Yun, et al. “HaitiBRD: A Labeled Satellite Imagery Dataset for Building and Road Damage Assessment of the 2010 Haiti Earthquake.” Designsafe-Ci.org, DesignSafe-CI, Dec. 2023, https://doi.org/10.17603/ds2-fqat-4v02. Accessed 28 Mar. 2026.
[4] Shao, Yang, et al. “Deep Learning for Remote Sensing Mapping of Roads, Sidewalks, and Bicycle Lanes in Bogotá.” Remote Sensing Applications: Society and Environment, Elsevier, Mar. 2026, p. 101993, https://doi.org/10.1016/j.rsase.2026.101993.
[5] Xie, Enze, et al. “SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers.” ArXiv:2105.15203 [Cs], Oct. 2021, arxiv.org/abs/2105.15203.
Week 6: U-Net Experimentation
Hi everyone, welcome back to my blog! This week, let's attempt to improve the results of the damage assessment U-Net, my worst-performing model (so far).
Motivation
After reading some of the comments I've received in previous weeks and consulting with my internal advisor, I realized that the U-Net not learning may be due to its architecture. I've noticed that by training the U-Net directly on the damage assessment dataset, it often picks gray patches of roads or roofs, and can't tell the difference between roads and buildings. This is likely due to a lack of context, especially since damage in buildings are hard to detect, even visually. Honestly, not even I would have realized the T-shaped building on the right was destroyed if not for the mask—it just looks like a building with many things on the roof!
Another problem is that I've only been training my models on post-disaster images. However, cities in real life are often disorganized. Hard-to-read urban planning or cluttered buildings, and buildings that have already been damaged before the disaster both make only training from post-disaster images more unclear.
So the next step is to see if the U-Net can detect houses pre-disaster—a sanity check to make sure the U-Net is able to pick up the right patterns for houses at all, and possibly a way to help it detect damage better via transfer learning.
U-Net House Detection
After changing some settings of the Road detection U-Net and using the pre-disaster image-label-pairs of the Haiti Earthquake dataset, I was able to get it to separate the shape of the houses. It performed the task very well, actually, which made sense because buildings (in general) are easier to spot.
This is a good sign so far. Next, I wondered how to use the separated buildings to detect damage. I couldn't crop the images and pass it to another model to classify the damage, because the building shapes are often not straight and hard to crop. Additionally, I wanted to stick to the U-Net to avoid extra complexity. Eventually, found online that one way to incorporate both pre-disaster and post-disaster images into damage assessment is a Siamese architecture. I decided to combine that with pre-training my U-Net on house detection,
Siamese U-Net
This model uses two encoders with equal weights that take in different images, one pre-disaster and one post-disaster, then concatenate the dense feature maps to one shared decoder for damage classification [2]. I've taken inspiration from the 1st place-winning Siamese U-Net of the xView2 challenge, and also made it two-stage [3]. I would first train the model on detecting houses with the pre-disaster images, then train it on the combined pre-disaster and post-disaster images for damage classifications. The idea of using a siamese architecture with two encoders that have shared weights is that they will learn the same features,
But does this actually live up to the hype?
Here are the results.
By visual inspection, I think it's improved from without the siamese architecture. The "blobs" are closer to where the houses are, and the destroyed classes are in the right relative location of the actual damage. Of course, it's still not as good as SegFormer, and it's still suffering from making blob shapes instead of recognizing houses. What do the numbers say?
The original pure U-Net damage detection achieved a mean IoU of 0.21 and a mean F1 score of 0.32. Meanwhile, the siamese achieves significantly better results of 0.34 for mean IoU and 0.50 for mean F1 score. It's certainly better, which shows that the Siamese architecture is promising. If I had more time, I would compare my Siamese U-Net to the xView2 winning entry by Victor Durnov [4], and find ways to improve my model to replicate these results. Since I'm running pretty late on my project already, and I have a trip to China coming up, I think this might have to wait. In next week's post, I'll be gathering the metrics to compare the three models—U-Net, DeepLabv3+, and SegFormer, to see how they perform compared to each other.
Works Cited
[1] Lin, Szu-Yun, et al. “HaitiBRD: A Labeled Satellite Imagery Dataset for Building and Road Damage Assessment of the 2010 Haiti Earthquake.” Designsafe-Ci.org, DesignSafe-CI, Dec. 2023, https://doi.org/10.17603/ds2-fqat-4v02. Accessed 28 Mar. 2026.
[2] Chen, Tao, et al. “A Siamese Network Based U-Net for Change Detection in High Resolution Remote Sensing Images.” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 15, 1 Jan. 2022, pp. 2357–2369, https://doi.org/10.1109/jstars.2022.3157648. Accessed 23 July 2023.
[3] Zheng, Zhuo, et al. “Building Damage Assessment for Rapid Disaster Response with a Deep Object-Based Semantic Change Detection Framework: From Natural Disasters to Man-Made Disasters.” Remote Sensing of Environment, vol. 265, Nov. 2021, p. 112636, https://doi.org/10.1016/j.rse.2021.112636.
[4] Durnov, Victor. “GitHub - Vdurnov/Xview2_1st_place_solution: 1st Place Solution for “XView2: Assess Building Damage” Challenge.” GitHub, 2020, https://github.com/vdurnov/xview2_1st_place_solution.
Week 7: Ultimate Battle of Performance Metrics
Hi and welcome back to my blog! This week, I pit the three models I've been working on against each other to find out how well they perform compared to each other! If you've been reading my past few blogs, you won't be surprised by the results.
What are the Metrics
Before I go into the data tables, I'll explain which metrics I'm using so far.
IoU: Intersection over Union
Also known as the Jaccard index, this metric's name is pretty self-explanatory. You would simply take the area of intersection between one class's predictions and that of the labels, next taking the union, then dividing. This is a good measure of how accurate the segmentation area is in computer vision models. If you're more used to using confusion matrices, it would be TP / (TP + FP + FN) where TP is true positive, FP is false positive, and FN is false negative.
F1 Score
The F1 score is a metric more generally used in machine learning that measures a classification model's accuracy by calculating the harmonic mean of precision and recall. The resulting number ranges from 0 to 1, 1 being perfect. Precision measures the accuracy of positive predictions (for a pixel belonging to this class), while recall measures the ability to identify all relevant positive predictions. Precision is calculated as TP / (TP + FP) , while Recall is calculated as TP/ (TP + FN)
Here are the results, with the best-performing model highlighted!
Road Detection Model Comparison
All three models perform very well on road detection, all reaching well above 0.8 IoU and F1 score. Even though SegFormer is the most advanced architecture, surprisingly DeepLabV3+ is the most successful model. Since all the models have metrics very close to each other, only differing at the hundredth point, it could likely be due to random differences in model training, or not optimizing all the parameters for each model.
Damage Assessment Model Comparison
Here it's much more predictable. The SegFormer outperforms both U-Nets and DeepLabV3+ very noticeably, likely because the SegFormer's transformer architecture allows it to capture more context, learning features that are less obvious. Also note that DeepLabV3+ performs slightly worse than Siamese U-Net, but better than normal U-Net. This is likely evidence to show that the Siamese architecture to incorporate pre-disaster data really does help the model learn features better.
U-Net House Detection
Here's the UNet house detection metrics as well. It's pretty successful, just like it would be successful for roads, since the tasks are almost the inverse of each other.
Reflection...
Now that I'm finishing up the first stage of my project, comparing the effectiveness of the three models and learning how they learn, I want to stop and reflect on what I accomplished, what opportunities I missed, and what I'll do going forward.
This is one of the first projects where I'd actually explored and experimented with deep learning models extensively, and I was actually able to see improvement in the models when I make changes to it, which I'm proud of. I learned a lot about the strategies people employ to make stronger, smarter machine learning models. I'm extremely grateful to my internal and external advisors that have guided me and provided critical suggestions along the way.
I now realize the scope of my project is probably more than I should have taken on. I was able to do everything at the end, but I fear my reliance on pre-trained models (specifically SegFormer) and pre-labelled data prevented me from learning more about the remote sensing field and the scientific method.
...and Looking Forward
In the next few weeks, I am going to optimize the models. Since SegFormer is the best so far, I'll focus on that, by attempting to build a SegFormer model from scratch instead of using the current pre-built one. (Thanks Chloe for the suggestion! Context in week 5's comments.)
Additionally, I hope to measure the models' GPU computational efficiency, since it would be important to know how fast a model can respond to satellite data of earthquakes in real time. I've moved my U-Net and DeepLabV3+ models to my local machine's VSCode and I've downloaded a GPU performance tracker which is ready to gather data for. Currently, I am struggling with moving SegFormer into VSCode, because even with the exact same code (and almost the same package versions), I am getting this error in VSCode while not in Google Colab.
RuntimeError: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.
This happens when the model's built-in code runs loss.backward(), and doesn't change whether I use my mac's MPS or CPU. I think this might be fixed if I build my own SegFormer, but if anyone knows a quicker fix to this, I'd be really grateful!
Week 8: SegFormer Swerving Sideways
Hi all, welcome back to my blog! This week I'm re-adjusting from a trip back to China, and starting to improve the SegFormer!
First, some housekeeping. I've decided to make some updates to the upcoming plan I set out in week 7. After a bit of debugging, I concluded that the version I'm using is unable to work on MPS (Metal Performance Shaders, Apple's GPU), since the error happens at the PyTorch autograd C++ engine only while running on MPS but not CPU—and many online sources, along with my advisor, agree that it is a common problem. Thus, I'm calculating all my models' computation efficiency metrics by running the models on CPU. Although it is less realistic, its still useful as a baseline. As for the efficiency metrics themselves, I'm calculating parameter count and latency.
Also, since we are approaching the end of the senior projects soon, I decided with my advisors that instead of creating a SegFormer from scratch, I would attempt to exhaust all the other options with the current SegFormer first. So, let's see if my ideas for optimizing the models worked!
Computational Efficiency Metrics
Inference latency is the time it takes for a model to process an input and generate a prediction. To gather inference latency, I passed 10 input images to the model and measured the time elapsed between starting predictions and ending, then took the average time of the 10 predictions.
Parameters are the number of tuneable weights and biases within the model. Built-in PyTorch tools allowed me to find the parameter count of my models very easily,—the only caveat is that I got the SegFormer's parameter count from the NVIDIA paper the model is based on, because I used a pre-existing checkpoint. (nvidia/segformer-b0-finetuned-ade-512-512)
These parameters show even more support for SegFormer's strength, since it's able to be more lightweight and still outperform in accuracy! I was pleasantly surprised—I guess transformers live up to the hype.
Proposed Improvements to SegFormer
To improve on the model I have so far, I tried two possible solutions this week. I'll first explain why I chose it, and then the results.
Transfer Learning
I noticed that while the SegFormer is able to get the relative location of the damaged building & accurate damage level, the prediction shapes look like blobs and can't capture the edges of buildings. That probably contributed to the low 0.4 IoU. I wanted to see if I could make it learn the features of buildings first, just like I did with the Siamese UNet.
I first trained the model's localization phase on pre-disaster images, and that actually significantly degraded the model.
This confused me, but I suspect it's because the features the SegFormer learned from localizing houses conflicted with the features it picks up when detecting damage, causing it to not learn. I also tried training the localization phase on post-disaster images, but it didn't improve the model much either.
Data Augmentation
Another possible improvement area is the fact the model was overfitting. While training loss was at 0.2 or lower, the val loss was stuck at 0.6. I would combat this by implementing data augmentation, which introduces random transforms like flips or rotations to an image each time it's passed to the trainer.
However, this wasn't too successful either.
Perhaps the artificial increase in data images may have led to the models seeing fewer patterns in the training data learning less. Do you have any thoughts or experiences when these things fail? Anything else for me to try, potentially?
Week 9: Road Work Ahead
Unfortunately I never thought of the "Road Work Ahead" pun until week 9... which is sad because contrary to the title, I'm supposed to be wrapping up all the data (and analysis) that I'll present at the final presentation. In today's blog, we're discussing a critical part of the project I haven't addressed: road detection in actual earthquake images!
Remember from earlier posts that the roads I've been training the models are on are streets in Bogota, not Haiti. I picked this because of convenience and higher data quality. An additional benefit to this choice is that I let the model omit blocked roads. I did this because the HaitiBRD dataset claims that "undamaged and damaged road segments are denoted by line strings". However, these are unsuitable because all their displayed roads are continuous and look exactly like pre-disaster roads, and line strings don't account for street-side blockage that may block rescuers' access to buildings.
However, assessing post-disaster roads is still not as simple! Streets may be littered with debris from broken buildings, the shapes of roads are different across different countries, the resolution is different, and even different lighting may mess up a model's predictions. Thus, I'm performing some informal robustness testing to see if my road detection models hold up in real disaster scenarios.
Road Debris: Methodology
Since I wanted results quick, I have decided not to calculate accuracy metrics and rely on qualitative inspection this week. (As in, I'll just look at the prediction results.) I obviously can't waste the last three weeks of my work labelling data if it won't be a useful way to improve my model first!
I hand-picked three pre- and post-disaster image pairs that are adjacent to more majorly-damaged buildings, a mix, and no majorly-damaged buildings respectively. Then, I looked at how good the models are at recognizing "debris" (or anything that seems to block the road, like cars or trash cans), which I can only use my best judgement to look for since the images' low resolution make it unclear. In the actual dataset, I found that there are many exceptions where surrounding damaged buildings don't have any road clutter. (And conversely, undamaged areas don't have clean roads.) There may not actually be a correlation between debris on roads and nearby damaged buildings, but these are sufficient.
In the results, the top image-prediction-overlay trio will be post-disaster, and the bottom will be pre-disaster.
U-Net
Fig 2. (Lin et. al) U-Net road detection on pre- and post-disaster images. Left (majority damaged buildings), middle (mixed) right (majority undamaged buildings).
U-Net has been the least stable model of the three, and it shows when introduced to a different dataset. Even with pre-disaster models, which matches its training dataset's format the closest, it's only able to correctly identify one road correctly. I suspect the reason may be due to the lighting of the images (brightness and shadow angle). In the mixed-damage road detection results, the U-Net did not recognize non-blocked areas of the road in the post-damage image (top), but did so in the pre-damaged image (bottom). Comparing the images that the U-Net was trained on, it seems to do better with darker colored roads and less shadows. (Though it could be sampling error on my part.)
I think this may indicate that the U-Net is not very robust, and may not be suited for road detection of earthquakes. However, this could possibly be improved if the U-Net trains on the post-disaster dataset as well.
DeepLabV3+
Fig 4. (Lin et. al) DeepLabV3+ road detection on pre- and post-disaster images. Left (majority damaged), middle (mixed) right (majority undamaged).
DeepLabV3+, crowned with the highest accuracy metrics, blows us away again! In the majority damage pre-disaster section, we see that DeepLabV3+ detects a long strip of road in the middle. However, it stops at what might be a cluster of cars. Similarly, the mixed-damage pre-disaster image shows that the model identifies unblocked parts of the road and "notices" the clutter in the middle. DeepLabV3+ seems to fail in the third image, but I don't know why. Maybe it's the abundance of vegetation throwing it off.
SegFormer
I'll preface this by saying I didn't expect SegFormer to do well. It's attention mechanisms are strong, but that means it's much more likely to see clutter as a part of the road instead of an obstacle. It's partially my fault that the model is trained in such a convoluted way due to the model I chose.
Fig 5. (Lin et. al) SegFormer road detection on pre- and post-disaster images. Left (majority damaged), middle (mixed) right (majority undamaged).
With the assumption I wrote in the previous paragraph, I feel like it explains the SegFormer behavior a lot. Changes in prediction masks between pre-disaster and post-disaster look negligible. (Except for obviously explainable stuff like two trees' canopies covering almost the entire road, in the right image, or outright errors like mistaking a roof for road.) And the road detection doesn't stop at road clutter, like in the middle image. (Compare that to DeepLabV3+)
Conclusion
In conclusion, me staring at 18 prediction results is not enough to make a strong statement on each model's accuracy of post-disaster road detection. But with this small sample size, we do see patterns and we can speculate. The U-Net may be too simple and dependent on the training dataset's lighting conditions. The SegFormer was too robust, and learned to recognize types of road clutter like cars which defeated its purpose in detecting roadblocks. DeepLabV3+ seems like the blend that is "just right."
Works Cited
Lin, Szu-Yun, et al. “HaitiBRD: A Labeled Satellite Imagery Dataset for Building and Road Damage Assessment of the 2010 Haiti Earthquake.” Designsafe-Ci.org, DesignSafe-CI, Dec. 2023, https://doi.org/10.17603/ds2-fqat-4v02. Accessed 28 Mar. 2026.
Shao, Yang, et al. “Deep Learning for Remote Sensing Mapping of Roads, Sidewalks, and Bicycle Lanes in Bogotá.” Remote Sensing Applications: Society and Environment, Elsevier, Mar. 2026, p. 101993, https://doi.org/10.1016/j.rsase.2026.101993.
Week 10: More SegFormer
Hi, welcome back to my blog! My research is basically done, so this week I'm starting something new to explore some more options. I've complained previously about using a pre-made SegFormer that doesn't let me look into how it works. So, this week, I found a SegFormer implementation on PyTorch [1] which I'm going reference and play around with. The goal is to analyze and modify the architecture to specialize it for my project.
Transformer this, transformer that, what even is it?
Now that I have the actual code implementation instead of abstract descriptions, let's unpack the SegFormer architecture further. I'm using the SegFormer paper [2] as my primary reference, and I'll try to explain everything as conceptually as I can without going into the equations.
Remember that most image-based neural networks have an Encoder and Decoder. (Most famously, the U-Net) The encoder learns features from the image, and the decoder forms the result.
Encoder: MiT (Mix Transformer)
The MiT was proposed by the original SegFormer paper to be a version of the Vision Transformer tailored for segmentation.
Firstly, overlap patch embeddings converts the image into processable data. Patches of the image are merged into feature maps, similar to a convolution (explained in week 3). The overlapping step is crucial, since it allows the model to preserve spatial continuity
The next steps consist of transformer blocks, which have three sub-steps: efficient self-attention, Mix-FFN, and overlap patch embeddings (again). These steps are done repeatedly in transformer blocks, each step further shrinking the image and creating more features.
In the efficient self-attention step, each pixel is treated as a token, and it essentially looks at the other regions in the image and asks how "important" that region is to understanding what it is. This crucial step allows the transformer to capture the global context throughout the image. Why is it called efficient? Because the SegFormer makes the regions that parttake in self-attention contain multiple pixels, so there are less total queries made between the regions.
Next, the Mix-FFN (Feed-Forward Network) expands and applies a non-linear activation (use a non-linear function, like ReLU, to calculate weights) to the patches, allowing the model to learn local contexts.
Decoder
The decoder uses layers of MLPs (Multi-Layer Perceptrons, they are a type of FFN) to convert the features into an output. First, one MLP is used to unify the channel dimensions of each feature from the encoder (one from each of the four transformer blocks). Then, a second MLP upsamples the features to the same size (H/4 x W/4, where H and W are the height and width of the original image). Afterwards, a third MLP concatenates the four outputs. Finally, the last MLP maps it to the prediction image.
Results
How did our model turn out? Mixed reviews. The model itself worked well in that damage levels were accurately assessed in the rough area of the houses. However, it seems that the model ran into some issues upsampling, so the patches look very blocky. I've attempted to upsample and use interpolation, but it the shapes made less sense with that. I believe it's because the last few upsamples don't take enough context from the original image. I'm working with my advisor to solve it, but if you have any suggestions or disagree with my analysis, I'd love to hear it!
[1] Wang, Phil. “GitHub - Lucidrains/Segformer-Pytorch: Implementation of Segformer, Attention + MLP Neural Network for Segmentation, in Pytorch.” GitHub, 24 Aug. 2021, github.com/lucidrains/segformer-pytorch Accessed 2 May 2026.
[3] Xie, Enze, et al. “SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers” ArXiv:2105.15203 [Cs], 28 Oct. 2021, arxiv.org/abs/2105.15203
[3] Lin, Szu-Yun, et al. “HaitiBRD: A Labeled Satellite Imagery Dataset for Building and Road Damage Assessment of the 2010 Haiti Earthquake” Designsafe-Ci.org, DesignSafe-CI, Dec. 2023, https://doi.org/10.17603/ds2-fqat-4v02 Accessed 28 Mar. 2026.
Week 11: The Importance of Being Pre-trained
Hi everyone, welcome back to my blog! This week, I'm finishing up my presentation, while training and testing the custom SegFormer.
Custom SegFormer Results
I got some help from my advisor to make the SegFormer upscale to the right size with added UpConv layers, however it did not help very much. The shapes looked very much like the results I got from U-Net or DeepLabV3+. I reverted the model and re-ran it, and noticed that I was likely mislead by the results of last post. In this image of another output from the original custom SegFormer, notice how the predictions are all over the place, and still doesn't roughly follow the shape of actual houses in the ground truth labels.
The main difference between the custom SegFormer model and the previous model is the pre-training. The SegFormer model from HuggingFace has been pre-trained, while this model was not. Since the HuggingFace model got to learn more features like how to identify houses or other objects like trees or roads, it might know how to apply these already-learned features into the damage assessment. Our new SegFormer, which is not pre-trained, has to learn all these features from scratch, making it more likely to fail at learning and predicting.
Of course, there could also be architectural differences between the SegFormer models. While both are meant to reference the same paper, differences in implementation make them perform differently too.
Progress on the Presentation
This week, my cohort had a mock presentation where we all presented our slides to middle school engineering classes. There, we gave and received feedback from our advisors and each other. One of the suggestions I got was to represent my accuracy comparison graphically, and I thought it'd be useful to share with you all as well!
Here, you can see the differences in model performance more obviously, and compare which classes tend to be easier to segment. I did notice that the U-Net's major-damage class being 0 is a bit weird for both IoU and F1. It's possible that the U-Net never detected major damage due to it not learning. However, I didn't have time to dig into it thoroughly this week.
Week 12: Aftermath (Reflection Post)
Hi everyone! Thank you to those who made it this far in my project! This is officially the end of my senior project at the 12th week. In this post, I'll do a reflection on what I did over the senior project, what my experience was like, and the impacts and further research directions for my project.
Reflection
Despite the fact that I consider myself a person decently-versed in computer science and going to study electrical and computer engineering, I felt much less confident about my experience in AI compared to my peers. I had only done AI projects in classes. Participating in the senior project allowed me to gain hands-on experience building and training AI models (almost) from scratch. There's a lot more of going through the process of encountering problems and finding a solution for it. For example, my failures with the normal U-Net leading me to investigate the Siamese U-Net.
As a person who can't learn new information unless it is shoved down my throat, challenging myself to understand the models I'm using to the best of my ability was very rewarding. I learned to read research papers and actually digest it, and how to reverse-engineer AI models' code implementation to adjust its architecture. Working with my internal and external advisors gave me a balance of independent solution-seeking and guidance, which I'm really grateful for.
No Regrets... Right?
I'm glad that my research project can end on a positive note. I learned a lot about different segmentation models, and I'm happy with the results I got. However, there are things looking back on that might have made this better. For one, I almost neglected my DeepLab V3+ model because I spent a long time fixating on the U-Net, so I wasn't able to bring it up to its full potential. Same with the SegFormer, where despite it's good performance, I couldn't implement the changes I wanted to with it at the end. I thought the scope I chose was manageable, but I didn't account for how much I'll expand on it while I worked on it.
Impact and Future Work
As for the project itself, my work allows me to create the groundwork for accomplishing similar tasks in remote sensing. It could be used for other disasters, like tornados, hurricanes, or floods, since the procedure for these are the same as for earthquakes. If I could continue with this project, I'd like to implement routing algorithms which prioritize damaged buildings for rescue teams to use. Another direction I'd like to take is to train models on satellite images captured in other spectral bands, where the buildings might reflect differently and make differences in damage levels easier to recognize.
Please look forward to the final presentation tomorrow, or the recording of it which I will post on Sunday!
Week 12: 5: Final Presentation
Hi everyone! To those who attended my final presentation, thank you all for showing your support! If you missed it, I have linked my presentation slides and presentation video below.
https://docs.google.com/presentation/d/1gknLAILIBjfRVQBs4T65DJlCEZtBAnW3bWoAWkKNJaY/edit?usp=sharing
https://youtu.be/RKVEmmd1u8A