Week 11: Reflection

Thank you so much to everyone who watched my presentation last week! I had a great time showing the results of my work, and I learned so much conducting this project. 

One clear takeaway is that the overall structure of how I approached the project worked well. I approached the problem starting with exploratory data analysis, then benchmarking existing models, then building from scratch, then compressing a production model. Each step gave me information I needed for the next one. If I had jumped straight into compression without understanding what the data looks like and what production models can do, I would have had no way to interpret my results. 

I also learned that roadblocks can come from unexpected places. When I started this project, I expected the main issues to be around finding effective algorithms. However, my main issue involved trying to get Dorado and Bonito working. Dorado’s CRF decoder was built on top of C++. Bonito requires CUDA and does not support macOS. The other thing I would do differently is use a larger dataset. The NanoBaseLib demo dataset has 1,666 reads, which was enough to test compression on RODAN but not enough to train meaningful custom architectures from scratch. 

The final results on the architecture side show how simpler models like RNNs perform surprisingly well while staying small and suggest that chasing more complex buzzwords like “transformer” is unnecessary. For compression, I found a model can be shrunk 4.6x with 0.2% drop in performance, which would help field deployment of these basecalling models with limited processing power. 

Future work in this field should focus on training-aware compression. Every technique I tested was applied after training, to a frozen model. With my best results maintaining good accuracy around 5x compression, making further progress seems to require a training-aware approach that can work with a large amount of data. With a training-aware approach, the model can adapt to the size constraints as it learns.