Week 9: Writing the Report

This week I wrote my technical report. The report documents everything I have done over the course of this project: the baseline model selection, the custom architecture experiments, the compression results, and takeaways.

The first section I wrote was about finding a model to compress. I had three failed attempts before landing on something that worked. Dorado, which is Oxford Nanopore’s production basecaller at 99.96% accuracy, was written in C++, and I could not replicate its decoder in Python. Bonito would not install on my Mac because it requires CUDA. I ended up using RODAN, a fully convolutional RNA basecaller from a 2022 BMC Bioinformatics paper. It is pure Python and PyTorch, uses CTC loss, and has pre-trained weights available. 

The next section covered the eight custom architectures I trained from scratch. The point of this section was to compare which architecture classes showed the most potential. The GRU model came out on top at 57.6% accuracy, and the Minimal RNN, which only has a forget gate, reached 52%. The fact that such a simple recurrent cell came close to the GRU supports the idea that LSTMs have more complexity than this task actually needs.

The compression results section is the most detailed part of the report. I ran six techniques on RODAN and recorded what happened to accuracy and size. Magnitude pruning worked well up to 30% with only a 4% accuracy drop, then collapsed sharply between 30% and 50%. Weight sharing with 128 clusters was nearly lossless and gives a 4.6x reduction in storage size. Low-rank factorization did nothing because RODAN is fully convolutional and its weight matrices are too small for SVD to make a difference. The combined pipelines were the most important negative result. Pruning 20% alone gives 88.4% accuracy, and weight sharing with 64 clusters alone gives 87.7%, but stacking them drops to 13.5%.  Stacking compression without retraining in between does not work.

The report ends with a discussion of limitations and what comes next. The biggest limitation is the dataset size. 1,666 reads is enough to test things, but not enough to train meaningful models from scratch. The other limitation is that I did not get to test training-aware compression, like quantization-aware training or knowledge distillation from RODAN.