Week 11: The Importance of Being Pre-trained

Hi everyone, welcome back to my blog! This week, I’m finishing up my presentation, while training and testing the custom SegFormer.

Custom SegFormer Results

I got some help from my advisor to make the SegFormer upscale to the right size with added UpConv layers, however it did not help very much. The shapes looked very much like the results I got from U-Net or DeepLabV3+. I reverted the model and re-ran it, and noticed that I was likely mislead by the results of last post. In this image of another output from the original custom SegFormer, notice how the predictions are all over the place, and still doesn’t roughly follow the shape of actual houses in the ground truth labels.

The main difference between the custom SegFormer model and the previous model is the pre-training. The SegFormer model from HuggingFace has been pre-trained, while this model was not. Since the HuggingFace model got to learn more features like how to identify houses or other objects like trees or roads, it might know how to apply these already-learned features into the damage assessment. Our new SegFormer, which is not pre-trained, has to learn all these features from scratch, making it more likely to fail at learning and predicting.

Of course, there could also be architectural differences between the SegFormer models. While both are meant to reference the same paper, differences in implementation make them perform differently too.

Progress on the Presentation

This week, my cohort had a mock presentation where we all presented our slides to middle school engineering classes. There, we gave and received feedback from our advisors and each other. One of the suggestions I got was to represent my accuracy comparison graphically, and I thought it’d be useful to share with you all as well!

Here, you can see the differences in model performance more obviously, and compare which classes tend to be easier to segment. I did notice that the U-Net’s major-damage class being 0 is a bit weird for both IoU and F1. It’s possible that the U-Net never detected major damage due to it not learning. However, I didn’t have time to dig into it thoroughly this week.