Week #4: Model Training

Intro:

Welcome to Week #4 of my blog! This week, I spent the majority of my time training the wind and fire detection model that I will eventually run on my Raspberry Pi 3 camera. This model I chose to use is the YOLOV8 nano, as I have experience using it from past projects and robotics. It also suits this use case well. If needed, I may use a YOLOV8 small instead after testing the nano. Let me now detail my experience training this model.

Kaggle

This was my first time sourcing datasets off of Kaggle for a personal project. For reference, I built my dataset by combining the D-Fire dataset (https://www.kaggle.com/datasets/sayedgamal99/smoke-fire-detection-yolo) and the FIgLib/HPWREN dataset (https://drive.google.com/drive/folders/1IKXN2-hxTrEQsIIKOxiUAuLgoxubA9Wq). I usually annotate data on Roboflow and download it from there, or hand-create datasets off of sources like Google Earth Engine (for satellite imagery). Kaggle was very convenient to use, and I look forward to using it in the future.

Issue 1: Slow Image Upload:

So now I’ll detail some issues that I’ve faced so far when trying to train my model. I had initially uploaded my entire dataset (combining both D-Fire and FlgLib, so roughly 23,718 images and 31,945 labels) into Google Drive, which took roughly 7 hours. I then mounted the drive and have the notebook pull images and labels from there. However, this led to ludicrously long epochs (see below):

Figure 1. Screenshot of epoch training time. Image by author, 2026.

It was taking 1 hour an 10 minutes to complete 29% of the first epoch of 100. This was simply unacceptable. So, I looked for solutions online, and saw a recommendation. It was to mount my drive, and then move my entire dataset into the runtime’s file system. The notebook would then pull files from there during training, instead of my Google drive. I wasn’t expecting this to work, but surprisingly, it did. It now takes roughly 15 minutes to copy the dataset over, and roughly 7 minutes/epoch to train.

Issue 2: Sleepy Laptop and Disconnecty Runtime:

Another issue kept running into was that the runtime would disconnect everytime my laptop fell asleep, or the browser just felt like it. It was incredibly frustrating. To fix this, I added a feature where the model’s current state is saved in a last.pt file after each epoch. When the runtime gets disconnected, I can input that last.pt file and have the notebook resume training from that checkpoint.

Updated Data Flow Diagram:

As I mentioned in a response to one of the comments on last week’s blog, the Data Flow Diagram I included there was actually slightly outdated, as I had changed some of my methodology. Please find the updated Data Flow Diagram here:

Figure 2. Updated Diagram of the Data Flow (Smoke Detection → Wind Estimation → Thermal Scan) pipeline created using Mermaid. Diagram by author, 2026.

Questions:

The biggest question for me this week would be: does anyone know a way to keep a runtime using a GPU (T4 in my case) from disconnecting when the GPU isn’t being used? What’s happening is that whenever I start the runtime, I set it as a GPU-High RAM runtime. But when I do the initial step of uploading the training data from drive into the runtime’s file system (before training), this process doesn’t use the GPU, prompting Colab to automatically disconnect the runtime. The only current solution I’ve found is buying Colab Pro+, which is $49.99/month. If there is no other solution I’m ready to do this, but I’d like to try a bit first before I do so.

That’s all for this week. Thanks for reading, and I hope to see you next week!