Week 6: U-Net Experimentation

Hi everyone, welcome back to my blog! This week, let’s attempt to improve the results of the damage assessment U-Net, my worst-performing model (so far).

 

Motivation

After reading some of the comments I’ve received in previous weeks and consulting with my internal advisor, I realized that the U-Net not learning may be due to its architecture. I’ve noticed that by training the U-Net directly on the damage assessment dataset, it often picks gray patches of roads or roofs, and can’t tell the difference between roads and buildings. This is likely due to a lack of context, especially since damage in buildings are hard to detect, even visually. Honestly, not even I would have realized the T-shaped building on the right was destroyed if not for the mask—it just looks like a building with many things on the roof!

Another problem is that I’ve only been training my models on post-disaster images. However, cities in real life are often disorganized. Hard-to-read urban planning or cluttered buildings, and buildings that have already been damaged before the disaster both make only training from post-disaster images more unclear.

So the next step is to see if the U-Net can detect houses pre-disaster—a sanity check to make sure the U-Net is able to pick up the right patterns for houses at all, and possibly a way to help it detect damage better via transfer learning.

 

U-Net House Detection

After changing some settings of the Road detection U-Net and using the pre-disaster image-label-pairs of the Haiti Earthquake dataset, I was able to get it to separate the shape of the houses. It performed the task very well, actually, which made sense because buildings (in general) are easier to spot.

This is a good sign so far. Next, I wondered how to use the separated buildings to detect damage. I couldn’t crop the images and pass it to another model to classify the damage, because the building shapes are often not straight and hard to crop. Additionally, I wanted to stick to the U-Net to avoid extra complexity. Eventually, found online that one way to incorporate both pre-disaster and post-disaster images into damage assessment is a Siamese architecture. I decided to combine that with pre-training my U-Net on house detection,

 

Siamese U-Net

This model uses two encoders with equal weights that take in different images, one pre-disaster and one post-disaster, then concatenate the dense feature maps to one shared decoder for damage classification [2]. I’ve taken inspiration from the 1st place-winning Siamese U-Net of the xView2 challenge, and also made it two-stage [3]. I would first train the model on detecting houses with the pre-disaster images, then train it on the combined pre-disaster and post-disaster images for damage classifications. The idea of using a siamese architecture with two encoders that have shared weights is that they will learn the same features,

But does this actually live up to the hype?

Here are the results.

By visual inspection, I think it’s improved from without the siamese architecture. The “blobs” are closer to where the houses are, and the destroyed classes are in the right relative location of the actual damage. Of course, it’s still not as good as SegFormer, and it’s still suffering from making blob shapes instead of recognizing houses. What do the numbers say?

The original pure U-Net damage detection achieved a mean IoU of 0.21 and a mean F1 score of 0.32. Meanwhile, the siamese achieves significantly better results of 0.34 for mean IoU and 0.50 for mean F1 score. It’s certainly better, which shows that the Siamese architecture is promising. If I had more time, I would compare my Siamese U-Net to the xView2 winning entry by Victor Durnov [4], and find ways to improve my model to replicate these results. Since I’m running pretty late on my project already, and I have a trip to China coming up, I think this might have to wait. In next week’s post, I’ll be gathering the metrics to compare the three models—U-Net, DeepLabv3+, and SegFormer, to see how they perform compared to each other.

 

Works Cited