Week 3: The struggle is real...

Hey folks, welcome back to my blog!

As you can read in the title, this week has been a struggle. After researching some neural networks to solve the preliminary problem of my project, modeling neural networks to solve simple ODEs, I found two great pytorch libraries. 

First is Pytorch Lightning, which automates the training process in one class so that all I have to worry about is the actual model of the neural network. While this library makes it extremely easy to train neural networks, a lot of the process is thus abstracted away, which makes it harder to debug. While using this library, I have encountered tons of bugs and problems in terms of loading data and training data. I have yet to get used to the new library. 

The next resource I found was Torchdyn, which is a github repository/library dedicated to Neural ODEs, which is a generalization to the Resnet architecture of neural networks. This type of neural network is depth invariant, and is structured in a way such that the depth of the neural network is continuous. Meaning, you can graph the progression of the neural network as it grows deeper. I researched two papers discussing what these neural ODEs are, and how they can be used: https://arxiv.org/abs/2008.02389, https://arxiv.org/abs/1806.07366. These research papers are actually quite new. At first I had a lot of trouble understanding the network structure and the paper in general, but eventually I started to understand the unique depth invariant nature of the structure. After gaining somewhat of an understanding, I followed a few of the tutorials the Torchdyn repository provided to learn how to implement Neural ODEs. Then I tried implementing this structure to model a simple exponential ODE, however the results were not ideal. Weirdly, as I graphed the results in terms of the depth of the neural network, I got an exponential looking graph. However, as I graphed the x to the predicted y, I got a function that is similar to the activation function I used. I have no idea how that occurred. It must be because I might have misunderstood or did not fully understand Neural ODEs. I tried again and again, tinkering with the structure, but nothing worked.

I took a step back, and tried using a MLP neural network with Pytorch Lightning, and still, the results did not match the function I was trying to model. This was very frustrating because this is only the preliminary problem, however, the good news is that because of the multidimensional nature of neural networks, generalizing this to a system of ODEs is a lot simpler than I thought it was. Once I get these neural networks to work with simple ODEs, I believe that this project will move forward a lot faster.

Although this week was less than ideal in terms of results, I feel that I am gaining a much better grasp of what I need to do to get this project finished. It is a lot more challenging than I anticipated, but also a lot more fascinating as well.

Thanks, and see ya guys next week!