Week 3: A Picture is Worth a Thousand Neurons

Hi everyone, welcome to the third week of my blog! I’ll share some updates on my progress and existing work in my project topic.

Slow and Steady Debugs the Code

This week I began to debug my initial issues with my code for the U-Net. My leading hypothesis was that I didn’t take advantage of the image’s resolution. When I made the model initially, I made the rookie mistake of resizing the images rather than cropping. When your image dimensions are in the thousands (1024×1024 and 3333×3333 for the two datasets), resizing to sizes that are usable for AI (around 256×256-512×512), lots of details are lost.

While writing the code for cropping my images was easy, getting it to work wasn’t. I was initially using Google Colab’s servers to train my model and Google Drive to store my data. When cropping the images, I had uncompressed the images, but uploading them back onto my drive meant that I quickly exceeded my Drive storage limit. Thinking quickly, I scrambled to move my model to run on my local computer to use my computer’s slightly larger storage instead. Unfortunately, right as I ran into issues installing the necessary libraries, I had to leave for the senior trip. The problem remains unsolved, but hopefully not for long.

If anyone knows what this means, please let me know! I don’t have many clues besides the fact that import torch causes the issue.

A Crash Course on AI in Image Processing

Timing is key when it comes to damage assessment for natural disaster response[1]. Now that AI automation using remote sensing images is an option, researchers are using it to expedite earthquake damage assessment.

Deep-learning models have shown great promise in recent years, specifically CNNs, short for convolutional neural networks. [2] Since parts of the CNN are foundational to understanding the vision models that build on top of it, I’ll backtrack a little to explain how it works.

Neural networks are a set of interconnected nodes organized in layers designed to mimic the human brain. The nodes on each layer calculate weighted sums based on the features passed into it, with more layers capturing more complexity by combining the features. Then, a final layer computes these neurons into the probability of something happening, like deducing that a given email is spam. [3] Through multiple training runs, the neurons adjust their weights to tune the “importance” of each feature, gaining a clearer “understanding” of the picture.

When images come into the picture, it becomes more complicated. Flattening images into pixels and feeding each pixel into the network removes much of the needed context we humans usually use to read images. Convolutional Neural Networks solve this problem with extra structures built to support images. CNNs consist of three types of layers: convolutional, pooling, and fully-connected. The convolutional layer uses filters—matrices with weighted values on each cell that gets multiplied with an area of the image—to sweep through the image, isolating its “features”. Pooling layers are like unweighted convolutional layers, effectively downscaling the image by taking either the max value or the average value of a small area at every place in the image. Then, fully-connected layers connect the inner layers to the output. [4]

U-Net, DeepLabV3: on the Shoulders of Giants

Later, people started to use Convolutional Neural Networks for segmentation, which meant assigning each pixel to a class rather than finding the general area and size of the object. A much more difficult task! In a paper by Qing et al., CNNs were successfully used to perform semantic segmentation for earthquake damage assessment, isolating damaged and undamaged buildings with an IoU of 0.522. The IoU, which is the area of intersection over the area of union between the prediction and the ground truth, shows that the model is moderately correct, though not precise.

Afterwards, the UNet made a breakthrough in 2015 with its unique U-shaped architecture. Instead of just downsampling an image, they add upsampling layers, increasing the resolution of the final image to form a more precise segmentation while using fewer input images.[6] Although its initial usage was in biomedical engineering, its use soon expanded into remote sensing and other fields.

DeepLabV3+ was made in 2018 by Google Research for semantic segmentation with more improvements. With the addition of Atrous Spatial Pyramid Pooling (ASPP), DeepLabV3+ Uses multiple parallel dilated convolutions (atrous convolutions) with different rates to analyze the image at different scales (different-sized filter matrices), allowing the model to capture both local details and global context. [8]

Transformers and SegFormer

Soon, transformer models created another opportunity. The addition of self-attention mechanisms allowed transformers to retain more context from further-away parts of the input. For natural language processing models, that meant it could read longer paragraphs. For us, that means it treats images as sequences of patches rather than raw grids of pixels, allowing them to apply the self-attention mechanism to understand global, long-range relationships within a scene. [8]

SegFormer applies the transformer architecture to the segmentation task, creating a lightweight model that can achieve an IoU of up to 70-80% on various segmentation training sets. A significant improvement from the previous IoU metric I mentioned! [9] 

With transformers, we have reached the present day, although modern transformer models are still competing to be the most reliable. In the past few years, new models like SAM-CD (Segment Anything Model) have achieved higher resolutions and enabled multiple classes of damage levels rather than the binary “damaged vs undamaged”.

Works Cited

[1] Matin, Sahar S., and Biswajeet Pradhan. “Challenges and Limitations of Earthquake-Induced Building Damage Mapping Techniques Using Remote Sensing Images-A Systematic Review.” Geocarto International, June 2021, pp. 1–27, https://doi.org/10.1080/10106049.2021.1933213.

[2] Xianwei Lv, et al. “Very High Resolution Remote Sensing Image Classification with SEEDS-CNN and Scale Effect Analysis for Superpixel CNN Classification.” International Journal of Remote Sensing, vol. 40, no. 2, Sept. 2018, pp. 506–31, https://doi.org/10.1080/01431161.2018.1513666.

[3] IBM. “What Is a Neural Network?” IBM, 6 Oct. 2021, www.ibm.com/think/topics/neural-networks.

[4] Fukushima, Kunihiko. “Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected by Shift in Position.” Biological Cybernetics, vol. 36, no. 4, Apr. 1980, pp. 193–202, https://doi.org/10.1007/bf00344251.

[5] Qing, Yuanzhao, et al. “Operational Earthquake-Induced Building Damage Assessment Using CNN-Based Direct Remote Sensing Change Detection on Superpixel Level.” International Journal of Applied Earth Observation and Geoinformation, vol. 112, Aug. 2022, p. 102899, https://doi.org/10.1016/j.jag.2022.102899. Accessed 6 Nov. 2022.

[6] Ronneberger, Olaf, et al. U-Net: Convolutional Networks for Biomedical Image Segmentation. 18 May 2015, arxiv.org/pdf/1505.04597.

[7] IBM. “What Is a Transformer Model?” IBM, 10 Oct. 2023, www.ibm.com/think/topics/transformer-model.

[8] Chen, Liang-Chieh, et al. “Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation.” ArXiv.org, 2018, arxiv.org/abs/1802.02611.

[9] Xie, Enze, et al. “SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers.” ArXiv:2105.15203 [Cs], Oct. 2021, arxiv.org/abs/2105.15203.