Developing a compressed deep learning model to enable nanopore sequencing on edge devices
Advisor: Noble
Internship Location: University of Washington-Seattle
Onsite Mentor: Dr. Cherukumilli
This research project addresses the need for computationally efficient basecalling models to enable nanopore sequencing on edge devices. Nanopores work by measuring the electrical signal change as molecules pass through tiny conductive holes. They are being explored as a cost-effective and accurate method for water quality monitoring in low-resource environments. However, current basecalling models require high-power GPUs to convert raw signals into identifiable sequences. This project investigates how to develop an accurate basecalling neural network that can operate on edge devices with limited processing power, avoiding the need for powerful devices or internet access to cloud servers.
By exploring optimization techniques like model pruning, low-bit quantization, and knowledge distillation, I aim to build accurate models that meet the hardware constraints of field-deployable systems. My methodology involves an iterative development cycle of building and testing models to maximize accuracy while minimizing their computational complexity. The final product will be an open-source, executable machine learning model capable of sequencing on devices with limited computational power.
Project Blog
Week 1: Introduction
Hi everyone. My name is Charan Sridhar, and over the next few months, I'll be working on a machine learning project focused on nanopore sequencing. I'll use this blog to document what I'm doing, what I'm learning, and the problems I run into along the way.
Here's the context. Nanopores are tiny pores that can detect molecules by measuring changes in electrical current as those molecules pass through. The technology was originally developed for DNA sequencing, but researchers have started applying it to other areas like water quality testing. Last summer, I worked with Dr. Cherukumilli at the University of Washington's SWELL Lab on a project that used nanopores to detect contaminants in water. While I was there, I noticed that the software used to interpret the nanopore signal, called a basecaller, requires a lot of computational power. These models are built to run on high-end GPUs in data centers. That's a problem if you want to use a nanopore sensor in the field, in a place without reliable internet or powerful hardware.
That's the problem I'm trying to address. My research question is: how to compress a basecalling neural network without losing meaningful accuracy, so that it can run on low-power devices that can be deployed in the field? I'll be looking at techniques like model pruning, quantization, and knowledge distillation. Most of the work on basecaller optimizations is about making basecallers faster on high-end GPUs, not on small CPU edge devices.
My hypothesis is that using techniques like quantization and pruning, I will be able to build a compressed model that is accurate enough to be practically useful in the field for certain applications like water quality testing, even if it doesn't match the performance of a production-level basecaller like Dorado.
Week 2: Literature Review
While reading the existing research on nanopore basecalling, three papers stood out as most relevant to what I am trying to do. The first is a 2018 paper by Teng et al., which introduced Chiron, one of the first basecallers to use deep learning to directly convert raw nanopore signals into nucleotide sequences. Before this, most pipelines relied on Hidden Markov Models, which required many manual signal-processing steps in between. Chiron showed that an end-to-end neural network approach could beat those older methods. What is relevant to my project is that the authors never considered how computationally expensive the model was, and it was built assuming you had powerful hardware to run the model on.
The second paper, from Frensel, Al-Ars, and Hofstee in 2024, is the most directly related to my project. They applied a technique called structured sparsity to a basecalling model, which essentially means training the model to get rid of weights that are not contributing much, shrinking the model without killing the accuracy. Their results were promising and suggest that compression can work in this domain. The issue is that all their testing was done on GPUs, so the efficiency gains they measured apply only to high-end hardware. Nobody has seriously tested whether these kinds of optimizations actually help when you are running on a low-power CPU.
The third paper is the 2024 BaseNet paper by Li et al., which proposes using a transformer architecture for basecalling instead of the LSTM-based models that most production basecallers use. The argument is that transformers handle long-range dependencies in the signal better. That might be true, but transformers are also expensive to run, and again, the paper only tests on powerful hardware. Looking at these three sources together, the gap becomes obvious. The field has gotten much better at making accurate models, but no one has built something that actually runs well on a small device like a Raspberry Pi. My project aims to build a low-resource model that maintains accuracy even with limited compute power.
Week 3: Data Exploration
This blog post will explain the work I have done over the past 3 weeks.
The first thing I did was prepare my dataset. The dataset I'm using is from NanoBaseLib, a benchmark dataset for nanopore sequencing tasks. After downloading and extracting it, the raw signal data is stored in a multi-read FAST5 file. The signals themselves are stored as 16-bit integers and represent raw ionic current samples. I wrote code to load these signals and convert them to picoamperes using the calibration metadata stored alongside each read. The signals ranged from roughly 53 to 141 pA, with a median read length of about 35,000 samples. What stood out to me right away was how noisy the data is. The signal has a clear step-like structure at a high level, where each step corresponds to a different sequence of bases passing through the pore, but within each step, there is substantial noise, around 3 to 4 pA, that makes the boundaries between steps hard to pin down. Even in the zoomed-in graphs, the transitions are not clean.
I also looked at the eventalign output from Nanopolish, which is the pre-computed alignment of signal segments to reference k-mers. This is the ground truth that basecallers try to learn. Looking at the distribution of event-level mean currents and event durations gave me a better sense of how much variation there is between reads. Some events are very short while others are long, and there is a wide range of current levels across different k-mers.
For week two, I focused on benchmarking existing basecallers to give myself a target to compare against. The main one I tested was Dorado, which is Oxford Nanopore's current production basecaller. I ran Dorado version 0.9.6 on the demo dataset using Apple Metal acceleration (since I am working on a Mac), and it finished in about 285 seconds. Dorado produced results for 4,574 reads with a mean Q-score of 34.5, which corresponds to an estimated accuracy of around 99.96%. Guppy, which is the older ONT basecaller, had a lower mean Q-score of 18.7 and about 99% accuracy.
During Week three, I built four baseline models, all trained using CTC loss, which is the standard approach for basecalling. CTC stands for Connectionist Temporal Classification, and the reason it is used here is that the input signal and the output base sequence have different lengths, and we do not know which signal samples correspond to which bases. CTC handles the alignment implicitly during training, which makes it a natural fit for this problem. The first model was a simple fully connected neural network. The ANN converged to a CTC loss of around 3.9 and basically just predicted the same base for every input.
The second model was a 1D CNN with four convolutional layers and a kernel size of 11, meaning each prediction had context from about 11 neighboring samples. This was significantly better, finishing with a validation loss of around 1.63. The third was a bidirectional LSTM, which processes the entire signal in both directions and can capture longer-range dependencies. It reached a similar validation loss to the CNN, around 1.63. The fourth model was a CNN plus LSTM hybrid. A convolutional front-end extracts local features from the signal, and the LSTM processes the resulting feature sequence to capture longer context. This performed best of the four, ending at a validation loss of about 1.27 on the training set and 1.28 on validation. It also had the smallest parameter count of the non-ANN models, at around 56,000 parameters and 220 KB, compared to the CNN and LSTM, which were both over 500 KB.
Week 4: Creating a list of Algorithms to Test
This week, I worked on figuring out what compression and optimization techniques I should try.
The technique I plan to try first is structured pruning with delayed masking, which comes from a 2024 paper by Frensel, Al-Ars, and Hofstee that I covered in my literature review. The idea is to train the model to eliminate entire neurons that are not contributing much, using a sparsity penalty that is introduced gradually, so the model has time to learn before you start cutting it. They got a 21x reduction in model size with almost no accuracy drop. However, everything in their paper was tested on GPUs, not on low-power CPUs, which is exactly the gap I am trying to fill.
I also looked into quantization, which reduces the precision of the weights from 32-bit floats down to 8-bit integers. Dorado already does this internally, and on ARM hardware like the Raspberry Pi, it is well-supported. I want to test post-training quantization first because it is fast, and then try quantization-aware training if accuracy drops too much. Another technique I plan to test is knowledge distillation, where you train a small student model to mimic the outputs of a larger teacher model, which tends to preserve more accuracy than just compressing the model directly.
Beyond compression, I also looked at architecture changes that could make the model faster from the start. The most promising one is replacing standard convolutions with depthwise separable convolutions, which can cut parameter count in the convolutional layers by about 9x. This has been shown to work well for time-series and audio tasks, but has not been tried for nanopore basecalling. Another option is replacing the LSTM entirely with a temporal convolutional network, which uses dilated convolutions to capture long-range context without the sequential bottleneck that makes LSTMs slow on CPUs.
Week 5: Organizing the Data and Setting up Testing
This week I shifted from research into actually writing code. The goal was to take everything I identified in week 4 and build it into a real, working codebase. Before I could build anything, I needed to clean up how the data gets loaded. In week 3, my data loading code was scattered inside the notebook.
This week I pulled it into its own file, data_loader.py. It reads all 1,666 reads from the FAST5 file and their corresponding base sequences from the label CSV, loads everything into memory, normalizes each signal to zero mean and unit variance, truncates signals longer than 8,000 samples to keep training fast, and splits the data 80/20 into train and validation sets.
The collation function handles the batching, which is a little tricky because signals have different lengths and need padding, while the labels get concatenated in a flat array for CTC loss. Having this in its own file means every other part of the codebase can import it without duplicating logic.
The other big piece of infrastructure I built this week was the benchmarking system in benchmark.py. Every model I test gets evaluated on the same four things: parameter count and model size in kilobytes, validation CTC loss, character error rate, and inference speed measured in signal samples per second. The character error rate is the most meaningful metric. It uses Levenshtein edit distance between the predicted base sequence and the actual one, divided by the actual length.
If the model predicts "ACGTAGT" when the correct answer is "ACGTACGT," the edit distance is 1 and the CER is 1/8, meaning 87.5% accuracy. This is the standard metric used in the nanopore basecalling literature. The benchmarking code also computes a compression ratio for each model relative to the baseline, so I can directly compare how much smaller a model is and what accuracy it gives up to get there.
Week 6: Experimenting with from-scratch Model Architectures
This week, I wrote models.py, which contains nine different model architectures. The idea behind building so many was to test specific hypotheses about what actually matters for this task.
The first is the baseline CNN+LSTM, which is the reference everything else gets compared to. It has three 1D convolutional layers followed by a two-layer bidirectional LSTM and a linear output head. The CNN extracts local patterns from short windows of the signal and the LSTM captures longer-range context. This is a scaled-down version of what Dorado uses.
The second is the depthwise separable CNN+LSTM. This is the same as the baseline, but the convolutional layers after the first one are replaced with depthwise separable convolutions. A standard convolution with 48 channels and kernel size 9 has around 20,000 parameters per layer. The depthwise separable version splits that into a per-channel filter and a pointwise mix step, bringing it down to around 2,700 parameters, roughly an 8x reduction. This idea comes from MobileNet, which is used heavily in mobile computer vision, but has never been applied to basecalling.
The third is the strided CNN+LSTM. The first convolutional layer uses a stride of 3, which means the output sequence is three times shorter than the input. The LSTM then only processes a third as many timesteps, which makes it roughly 3x faster since LSTM computation scales with sequence length. This works because the raw signal has a lot of redundancy. The median samples-per-base ratio in the dataset is around 10 to 15, so downsampling by 3x still leaves plenty of information for the model to work with.
The fourth and fifth are TCN-based models, meaning they have no LSTM at all. A temporal convolutional network uses dilated convolutions with exponentially increasing dilation factors, so the first block sees a small window, the next sees twice as large, then four times, and so on. By the last block the model has a receptive field covering hundreds of timesteps without increasing parameter count. The fifth model is a depthwise separable version of the TCN, which ends up being the smallest architecture at around 30,000 parameters. The reason to try these is that LSTMs are the main computational bottleneck in basecallers, accounting for around 90% of execution time according to the Frensel paper. Replacing them with convolutions that can run in parallel is an attempt to solve that.
The sixth is a GRU(Gated Recurrent Unit) model, which swaps the LSTM for a GRU. A GRU has three gates instead of an LSTM's four, which means roughly 25% fewer recurrent parameters and less computation per timestep. The seventh is a minimal RNN I built from scratch with just a forget gate, based on a 2018 paper that found the forget gate captures most of what LSTMs need. The eighth is an early exit model where each LSTM layer has its own output head, so in theory the model could stop early on easy parts of the signal. The ninth is a shared-weight LSTM that applies one set of LSTM weights three times in sequence, getting the depth of three layers while storing only one layer's worth of parameters.
Week 7: Implementing Compression Techniques and Preliminary Model Results
This week I built compression.py, which implements nine different compression techniques that can be applied on top of any trained model. These are separate from the architecture experiments and operate as post-training or training-time modifications.
The first technique is structured pruning with delayed masking, which comes directly from the Frensel et al. 2024 paper that I covered in my literature review. The model trains normally for a set number of epochs, then a group-lasso sparsity penalty is gradually introduced that pushes neurons toward zero. After training, the neurons with the smallest norms are removed entirely. Frensel got a 21x compression with less than 1.3% accuracy drop on Bonito. My implementation follows the same approach. The second is simpler, unstructured magnitude pruning, which just zeros out the weights with the smallest absolute values. I test this at 30%, 50%, and 70% sparsity.
The third is dynamic INT8 quantization, which reduces weight precision from 32-bit floats to 8-bit integers, giving roughly a 4x size reduction with no retraining needed. The fourth is knowledge distillation, where a smaller student model is trained to mimic the output probabilities of the larger trained baseline rather than learning from hard labels alone. The fifth is weight sharing via k-means clustering, which reduces all weights in each layer to a small number of unique values stored as a lookup table. The sixth is low-rank factorization using SVD, which approximates each weight matrix with a lower-rank version by discarding the smallest singular values.
The seventh is a lottery ticket search, which iteratively prunes and retrains to find the sparse subnetwork that can achieve full accuracy from its original initialization. The eighth trains the early exit model with deep supervision, giving each exit head its own CTC loss during training. The ninth is a combined pipeline that stacks multiple techniques in sequence: pruning, then weight sharing, then low-rank factorization, then quantization.
The main runner script, run_experiments.py, ties all of this together. It trains all the architecture variants from scratch first, then applies compression techniques to the trained baseline. Results are saved to a CSV after each experiment, so if the run gets interrupted I can restart without losing progress. It also handles early stopping, keeping the best checkpoint and stopping if validation loss does not improve for 8 consecutive epochs.
I have some preliminary results from the architecture experiments. The ranking from best to worst accuracy so far is: GRU with fewer gates, then depthwise separable CNN+LSTM, then the baseline CNN+LSTM, then depthwise separable TCN, then standard TCN with no LSTM, then strided CNN+LSTM with stride 3. The GRU outperforming the baseline is interesting because it is also a smaller model. It suggests the fourth gate in the LSTM is not adding much for this task.
The depthwise separable CNN+LSTM beating the baseline is also a good sign since it uses significantly fewer parameters in the convolutional layers. The TCN models landing below the LSTM-based ones suggests that for this dataset, the sequential context from recurrent layers matters more than the parallelism advantage of pure convolutions. The strided model being last makes sense because losing two-thirds of the signal timesteps is a significant information bottleneck even if it speeds things up. I will have full results with compression experiments once training finishes.
Week 8: Debugging Baseline Model and Compression Results
This week was about running compression experiments on a pretrained, already high-performing model.
The whole point of compressing a basecaller is that it needs to start at high accuracy. I needed a pre-trained, open-source model I could load into PyTorch and manipulate directly.
My first attempt was Dorado, which is Oxford Nanopore's current production basecaller and the one I used for benchmarking earlier. The problem is that Dorado is written in C++, not Python, and is built on top of CUDA, a proprietary NVIDIA software. I tried to reconstruct the architecture manually by reading its config files and weight tensors, but this did not work. The real issue was Dorado's decoder. I could not replicate the decoding part of the architecture. After spending a lot of time on this, I gave up.
My second attempt was Bonito, which is ONT's open-source research basecaller written in Python. It failed to install because it requires CUDA, and my Mac has Apple Silicon with no CUDA support. I spent a lot of time trying to find workarounds, but failed.
My third attempt was RODAN, a fully convolutional RNA basecaller published in BMC Bioinformatics. There were three small bugs related to macOS compatibility that I had to fix. After fixing those, RODAN successfully basecalled all 1,666 reads in my dataset. I initially got only 46% accuracy when testing it, but after debugging, I found two issues: I was feeding it raw ADC integers instead of picoampere-scaled values, and the output was being reversed when it should not have been for my dataset. After fixing both, accuracy jumped to 90.2%. RODAN has 10.7 million parameters and takes up about 41 MB. That is the baseline I am now working from.
With RODAN working, I ran all the compression experiments from my week 5 code. Magnitude pruning worked well up to 30% sparsity with only a 4% accuracy drop, but accuracy collapsed fast after that. Weight sharing, where I clustered all weights into 128 unique values per layer, maintained an accuracy of 90%, nearly identical to the baseline, with about 4x storage compression. Even with only 64 clusters, accuracy was 87.7%, which is still usable. Low-rank factorization did nothing. Every level I tested gave the same 90.2% accuracy, which likely means RODAN's fully convolutional architecture does not have weight matrices large enough for SVD to make a real difference. INT8 quantization preserved accuracy but only reduced size by 8% instead of the expected 4x, because PyTorch's dynamic quantization does not apply to convolutional layers, and RODAN is almost entirely convolutions.
Next week, I will start working on my research paper and slideshow presentation.
Week 9: Writing the Report
This week I wrote my technical report. The report documents everything I have done over the course of this project: the baseline model selection, the custom architecture experiments, the compression results, and takeaways.
The first section I wrote was about finding a model to compress. I had three failed attempts before landing on something that worked. Dorado, which is Oxford Nanopore's production basecaller at 99.96% accuracy, was written in C++, and I could not replicate its decoder in Python. Bonito would not install on my Mac because it requires CUDA. I ended up using RODAN, a fully convolutional RNA basecaller from a 2022 BMC Bioinformatics paper. It is pure Python and PyTorch, uses CTC loss, and has pre-trained weights available.
The next section covered the eight custom architectures I trained from scratch. The point of this section was to compare which architecture classes showed the most potential. The GRU model came out on top at 57.6% accuracy, and the Minimal RNN, which only has a forget gate, reached 52%. The fact that such a simple recurrent cell came close to the GRU supports the idea that LSTMs have more complexity than this task actually needs.
The compression results section is the most detailed part of the report. I ran six techniques on RODAN and recorded what happened to accuracy and size. Magnitude pruning worked well up to 30% with only a 4% accuracy drop, then collapsed sharply between 30% and 50%. Weight sharing with 128 clusters was nearly lossless and gives a 4.6x reduction in storage size. Low-rank factorization did nothing because RODAN is fully convolutional and its weight matrices are too small for SVD to make a difference. The combined pipelines were the most important negative result. Pruning 20% alone gives 88.4% accuracy, and weight sharing with 64 clusters alone gives 87.7%, but stacking them drops to 13.5%. Stacking compression without retraining in between does not work.
The report ends with a discussion of limitations and what comes next. The biggest limitation is the dataset size. 1,666 reads is enough to test things, but not enough to train meaningful models from scratch. The other limitation is that I did not get to test training-aware compression, like quantization-aware training or knowledge distillation from RODAN.
Week 10: Preparing the Presentation
This week I worked on my final presentation. I started the presentation with the problem, so the audience understands why any of this matters, before I start explaining model architectures. I started with the background of water quality testing, and how current detection methods require mass spectrometers that are too expensive for low-resource areas.
From there, I walked through how nanopore sequencing works, what a basecaller does, and why current basecallers are a problem for field deployment. Then I explained my approach: explore the data, benchmark existing models, build baselines from scratch, research compression, and apply it to a pre-trained model.
Then came the results section. For magnitude pruning, I focused on the cliff between 30% and 40% sparsity, because that is the most intuitive result to explain. The model tolerates losing some weights, but past a certain point, it collapses fast. For weight sharing, I focused on the K=128 result, 4.6x storage compression with only 0.2% accuracy loss, because that is the clearest win from the whole project.
The conclusion focused on two points. Simpler recurrent architectures like the GRU and Minimal RNN outperformed the LSTM, which suggests production basecallers are overbuilt for this task. And stacking compression techniques without retraining does not work. If I continued the project, the next steps would be training-aware compression, like delayed masking and quantization-aware training, a bigger dataset, and actual testing on a Raspberry Pi.
After putting the slides together, I did a few practice runs to work on timing and make sure the explanations were clear enough for someone without a machine learning background.
Week 11: Reflection
Thank you so much to everyone who watched my presentation last week! I had a great time showing the results of my work, and I learned so much conducting this project.
One clear takeaway is that the overall structure of how I approached the project worked well. I approached the problem starting with exploratory data analysis, then benchmarking existing models, then building from scratch, then compressing a production model. Each step gave me information I needed for the next one. If I had jumped straight into compression without understanding what the data looks like and what production models can do, I would have had no way to interpret my results.
I also learned that roadblocks can come from unexpected places. When I started this project, I expected the main issues to be around finding effective algorithms. However, my main issue involved trying to get Dorado and Bonito working. Dorado's CRF decoder was built on top of C++. Bonito requires CUDA and does not support macOS. The other thing I would do differently is use a larger dataset. The NanoBaseLib demo dataset has 1,666 reads, which was enough to test compression on RODAN but not enough to train meaningful custom architectures from scratch.
The final results on the architecture side show how simpler models like RNNs perform surprisingly well while staying small and suggest that chasing more complex buzzwords like “transformer” is unnecessary. For compression, I found a model can be shrunk 4.6x with 0.2% drop in performance, which would help field deployment of these basecalling models with limited processing power.
Future work in this field should focus on training-aware compression. Every technique I tested was applied after training, to a frozen model. With my best results maintaining good accuracy around 5x compression, making further progress seems to require a training-aware approach that can work with a large amount of data. With a training-aware approach, the model can adapt to the size constraints as it learns.