Complex-Valued Convolutional Neural Networks for Seizure Prediction
Advisor: Jonathan Bielli
Epilepsy is one of the most common neurological disorders with around 50 million people affected worldwide. Epilepsy causes sudden abnormal, excessive electrical discharges among neurons in the brain, leading to dysfunction and symptoms such as convulsions, sensory impairment, and loss of consciousness. While antiepileptic treatments can control seizures for 70% of individuals with epilepsy, around 30% of seizures are not controllable. It stands to reason that an accurate seizure forecasting system would improve the health of individuals with drug-refractory epilepsy. Traditional machine learning seizure prediction methods rely on real-valued representations of EEG signals, failing to exploit the phase and magnitude information present within complex-valued transforms of EEG signals. In this project, I propose the use of a complex-valued convolutional neural network to extract more informative spatial-temporal features within complex-valued representations of EEG signals.
Project Blog
Post 1: Introduction to My Project
This next week my research is starting. My project is on using complex-valued neural networks to improve the phase estimation of interferometric signals. This essentially means that I'm using a machine learning model that is trained directly on the phase and magnitude of signals (since complex numbers are used to represent the phase and magnitude of signals as a number) and trying to improve how those signals are processed in regards to interferometry, an imaging/measurement technique that uses light waves to create interference patterns to measure surfaces or take precise images.
I decided on this project because I learned about interferometry in Astrophysics as it was used in the Michelson-Morley experiments. At the same time, I've always been fascinated by neural networks and their applications. While brainstorming about project ideas, I realized I could combine these two interests into one. With this project, I hope I can learn more about machine learning and the process of research.
As my research proposal dictates, my first steps are to generate a synthetic interferometric dataset. Training my CVNN on the synthetic dataset will ideally give the model flexibility so it can tackle a range of datasets with varying signal to noise ratios (SNR).
In addition to that, it is also in my proposal to work on preprocessing the data. For a neural network, it is important for data to abide to certain guidelines, which include the removal of outlier data, normalizing the data (dividing the data such that it scales between 0 to 1), and splitting the data into training, validation, or test sets. For my project, I plan to randomly divide 70% of my data as a training set, 15% as a validation set, and 15% for test sets.
Week 2: New Challenges and Opportunities
Starting the project has been a lot more challenging than I expected. Preprocessing the data has been pretty simple. I've done this many times before with other projects. However, I probably overestimated myself thinking I could generate a synthetic interferometric dataset with the materials I have.
Without the ability to generate a dataset, this week I decided to focus on strengthening my understanding of complex-valued neural networks (CVNNs) and how they can be applied to applications related to interferometry. One study used CVNNs to process MRI fingerprinting data. The study found that each layer of the network captured phase-sensitive information much more naturally than their real-valued counterparts. It also found that CVNNs improved performance when working with low signal to noise ratio data. I realized that if I couldn't find interferometric data, I could just as well use any other form of phase-sensitive wave data.
I also learned more about how CVNNs are structured differently than traditional neural networks. Researchers have proposed complex versions of activation functions (the functions used to capture nonlinearity within datasets) such as complex versions of ReLU (a function that is zero until the input is greater than zero, at which point the output equals the input) and hyperbolic tangent (an "s" shaped curve). With this knowledge, perhaps programming the CVNN will be easier.
Week 3: Pivoting to Intracranial Seizure Data with Complex-Valued Convolutional Neural Networks
Finding a sizable amount of interferometric data has been proven difficult. Reflecting on the scope and direction of this research, I have concluded that improving interferometric systems does not align with my current research interests. Thus, much of this week was spent finding alternative applications of CVNNs that leverage more accessible forms of data.
One of the first ways I realized I could use CVNNs was for epilepsy prediction. Deep learning has been used in many ways in the last decade for epilepsy prediction, yet nearly no studies have been done using CVNNs for epilepsy prediction.
With the help of a research connection, I was recently able to receive six hours of intracranial electroencephalogram (EEG) recording from four epileptic mice, sampled at 2000 Hz. The data displays dynamic transitions into and out of seizures, and contains seven seizures in total.
Besides looking at data, I spent time this week researching more into different forms of CVNNs. Most likely, I will be using CV-CNNs (complex-valued convolutional neural networks). Like the name suggests, a CV-CNN is a complex-valued form of a convolutional neural network (CNN). A CNN is a type of neural network designed to detect spatial patterns and features in data through the use of filters that are learned. CV-CNNs extend this idea to the complex domain, replacing weights and activations with their complex-valued counterparts. Since CNNs have been used in previous scientific literature for waveform analysis, using its complex form would be logical and straightforward.
Week 4: Understanding EEG Data and Deep Learning
Much of this week was dedicated to developing a stronger understanding of how deep learning (especially convolutional neural networks) are used for seizure prediction. Understanding this will not only allow me to gain a better understanding of how I can code my complex-valued convolutional neural network (CV-CNN) in my project, but also get a baseline of how good current models are at predicting seizures.
One of the most central concepts I encountered throughout the literature is the classification of EEG states into preictal and interictal phases. The ictal phase refers to the actual seizure event, where abnormal electrical activity in the brain causes convulsions, loss of consciousness, or sensory dysfunction. Thus, preictal and interictal respectively refer to the stage before the seizure, lasting minutes to hours, and the stable period between seizures which make up most of a person’s life. The goal of seizure prediction then is to correctly identify the preictal state and not get it confused with the interictal state.
In reviewing the literature of CNN approaches to seizure prediction, I also found that the use of one-dimensional kernels (filters) was almost universally used. In CNN applications to image processing, kernels are applied across height and width, making it two-dimensional. EEG data is different as it runs across a temporal domain. Thus, a 1D kernel is applied to detect changes in amplitude over time. This concept will be useful in my application of CV-CNNs for seizure prediction.
Week 5: Understanding the Statistics Behind Seizure Prediction
Besides working on the introduction, literature review, and methodology, this week I worked on understanding the statistical side of seizure prediction in order to understand how researchers have evaluated and sampled seizure prediction in the past.
One of the most interesting patterns I found in literature this week was how a method called Leave-One-Subject-Out Cross-Validation (LOSO-CV) was commonly utilized. Instead of using the typical deep learning method of setting aside a certain percentage of the samples as test sets, LOSO-CV proposes (like the name suggests) leaving the data from one subject completely out and setting it aside as a test set. Then, this process is repeated until every subject has been set aside as a test set. LOSO-CV allows researchers to more accurately evaluate how well a model will perform with unseen data.
Many research papers in seizure prediction contained the same evaluation methods, including sensitivity, false prediction rate (FPR), and area under the receiver operating characteristic curve (AUC). Sensitivity assesses how well a model makes a correct prediction. FPR is a measure of how often the model makes a false prediction. In EEG research, FPR is often reported in false positives per hour instead of simply a fraction. Finally, AUC describes the balance between sensitivity and FPR. The receiver operating characteristic curve plots sensitivity versus FPR. Thus, the area under that curve reports how well a classification model can distinguish between categories overall.
Week 6: Working Through Gradient Descent Problems
This week, most of my problems came from training the CV-CNN. To understand the problems within training a neural network, it’s helpful to understand loss functions.
A neural network is trained by iteratively reducing the value of a loss function. The loss function is usually the distance between the actual values and the predicted values of a function. For classification problems (of which seizure prediction is a subset), the most common loss function is the cross-entropy loss function, given by the average of the real value distribution multiplied by the log of the predicted class distribution. If a model correctly identifies what it’s supposed to, the specific prediction and label pair will contribute to the model’s loss. If the model doesn’t correctly identify what it’s supposed to, the specific prediction and label pair won’t contribute to the model’s loss. It’s important to note that, as outputs will always be constrained to be probabilities between 0 and 1, the lack of contribution from false positives will not matter because correct predictions will always encourage the model.
As mentioned before, these are then optimized by way of derivatives, which repeatedly try to point the model in a direction that reduces loss by optimizing weights and biases (or, in my case, kernel weights). This “direction” is a vector called the gradient, and the magnitude of this vector can be controlled by way of a hyperparameter called the learning rate. If the learning rate is too large, then the gradients might accidentally place the model at a point with a higher loss than where it started from.
When it comes to my model, its loss was not moving at all. This can be for a variety of factors, which include my learning rate not being large or small enough, or my gradient might be getting smaller and smaller as they hit local minima. Next week, my goal is to fix this issue.
Week 7: Identifying Sources of Stagnant Gradient Descent
This week I focused on trying to diagnose why my CV-CNN’s loss was stagnant. While last week I hypothesized that improper hyperparameter tuning was the culprit, this week I realized other issues may have led to my problem.
While trying to research reasons why my model’s loss was stuck, I realized that the main reason why this problem occurs is because the model is not effectively updating the numbers in its filters. While it could mean that my learning rate was improper, other factors could have come into play. For example, if tensor (the multi-dimensional data type that is operated on in neural networks) types are incorrect while the model is updating its filter numbers via gradient descent, crucial information could be lost, causing the loss function to stay the same. Likewise, if the complex-valued parts of my model could not support the calculation of gradients, then my model’s loss could end up static. In my case, I encountered my model having incompatible tensor types because of its complex nature.
Thus, by the end of this week, I identified that my top priority should be debugging the complex-valued operations to allow for proper gradient descent. Next week, I plan to go layer by layer to verify each layer of the network and confirm that gradients are being computed and applied correctly.
Week 8: Working Through RAM Issues
This past week, I’ve spent most of my time working on problems with gradient descent. While trying to solve that problem, I also ran into another one: my random access memory (RAM) was being used up too quickly in Google Colab (the coding editor I was using).
In Google Colab, you are only allowed a limited amount of memory for use. Because my model was performing thousands of operations on 60,000 data points (as each 30-second EEG recording sample was recorded at 2000 Hz), my RAM depleted almost instantly whenever I ran the program. This caused an error that would completely stop my program.
There are a few methods to solve this issue. One is through the use of the garbage collection (GC) module in Python. Python's primary way to manage memory is through something called reference counting, in which Python counts how many variables are pointing to the data point in question. Once that count reaches zero, Python deletes and frees up that memory. However, this default method does not account for circular references (when two variables are pointing to each other). The GC module looks for these circular references and deletes them, which is especially useful for memory-intensive programs like neural networks. Using the GC module, I was able to free up a lot of RAM.
Another method is through the very basic signal processing technique of downsampling. As the name suggests, downsampling involves reducing the sampling rate of a signal by a certain factor. Naturally, having less data would allow me to decrease my memory usage.
My main goal next week is to find more ways to reduce my memory usage while also improving my model's backpropogation.