Week 7: Dead Ends and Black Magic

Hey folks, welcome back to my blog. As the title above suggests, I have hit an enormous dead end. This does not mean the project is over(I don’t give up that easily), but it does mean I have to try a different approach. Let’s go over how I hit the dead end and how I intend on getting out of it.

Monday was spent researching the properties of different activation functions, loss functions, and optimizers to find one that would fit my problem. I wanted to make the pytorch autograd model work. I altered the activation function of my model from sigmoid to RELU, leaky RELU, ELU, and GELU. While each activation was supposed to be an improvement from the previous one, it made almost no difference in the training process. Then I started tweaking the loss function, however I ended up back at MSE loss, my original loss function, as it was the most stable. Lastly, I tried changing the optimizers…and this is where I saw a big change. But unfortunately, this change was not good. Whether it was Adam, AdaMax, or Adadelta, they would always reach a local minimum loss that was super large. Although each behaved differently, none of them were able to converge near zero. Reaching a dead end here, I started a new notebook to try to figure out torch autograd, which I did. However, upon figuring it out, I am pretty sure my model should’ve worked.

On Tuesday, I tried altering the neural network layers to increase then decrease width taking inspiration from encoder and decoder structure. This did not change the training process at all. With nothing else to try, I gave up on torch autograd. Instead, I started training a neural network based on simulation to learn the Lorenz system. The result was not accurate even after running the training for 2 hours. On Wednesday, I tried tinkering with the model, but I was unable to increase the accuracy significantly. I think this is because the Lorenz system is simply just too difficult for the model to fit.

On Thursday, I felt incredibly demotivated. Pytorch just was not working for me. Both the simulation and autograd methods backfired. When I had my meeting with my external advisor, we went over what could have gone wrong but we were unable to gain a clear answer. After going over everything I have tried, we were genuinely stumped. Pytorch had hidden away too much of its complex functionality for me to effectively debug it. It felt like staring at…black magic. Strange and unexplainable. I am done dealing with Pytorch. I am going to code my own neural network without machine learning libraries. 

Friday. I have a clear mathematical plan on how the algorithm works. By the end of the day, I implemented half of the neural network. I know exactly how I am going to finish this. But…I’ll leave my methods as a mystery for now. Y’all will have to tune in next week for when I reveal some magic of my own. MWAHAHAHA!

(This is how you do a cliffhanger right? Lol see you guys next week!)