Neural Networks for Stock Analysis: Identifying Key Metrics and Predicting Investment Success
Advisor: Mr. Ong
Every year, millions of investors struggle to navigate the complexities of the stock market, and millions more don’t even take advantage of the opportunity to invest. The lack of solid, consistent financial advice leads to suboptimal decisions and missed opportunities, especially for those without access to high-quality financial advisors. My solution is an AI-driven stock-advisory system, designed to analyze historical financial data and identify key factors influencing stock profitability and risk. Using neural network structures, I will develop a system where more people will have access to actionable, data-driven investment decisions, offering an innovative alternative to traditional advisory methods. Furthermore, by comparing my results with the recommendations of financial advisors from publicly available data, I will determine the viability and profitability of my system. This project bridges the gap between advanced financial analysis and individual investors, equipping the everyday investor to make informed investment decisions about stocks.
Project Blog
Week 0: Introduction
Welcome to my blog, where I’ll document the creation of an AI-driven stock advisory system, a project designed to transform how individual investors approach the complexities of the stock market.
Investing can be daunting, especially for those without access to expert financial advice. The stock market is full of risks and opportunities, and making the right decisions often requires analyzing massive amounts of data and understanding subtle market patterns. My goal is to bridge this gap by developing an automated tool that uses machine learning to analyze historical data, identify key factors that drive stock performance, and recommend informed investment decisions.
To start, I’ll focus on collecting and organizing data—gathering historical stock prices, financial statements, and market indicators from reliable sources like Yahoo Finance and Bloomberg. This will lay the foundation for the next few weeks, where I will be training neural networks capable of recognizing patterns and predicting stock profitability. My initial weeks will also involve reviewing existing research on investment strategies and feature selection to ensure my model is well-grounded in proven financial concepts.
I will be working with Mr. Andrew Magee, an expert on the fundamental machine learning concepts I will be applying, and Mr. Daniel Ong, my Corporate Finance teacher and internal advisor.
By the end of this project, I aim to deliver a fully functional AI engine that generates viable stock recommendations. I’ll evaluate its effectiveness by comparing its suggestions to historical advice from financial advisors as well as actual market performance data, ensuring the results are both credible and actionable. Here's to an exciting journey in developing an AI-powered tool to optimize stock market decisions!
Week 1: Laying the Foundation
This week marked the official start of my AI-powered stock advisory project, and I focused on building a strong foundation. I began by exploring how machine learning is used in stock prediction and financial advising. While AI has been applied to forecasting, I found that few projects compare AI-generated stock recommendations with actual financial advisors—one of the key areas I plan to explore. Understanding past research helped me refine my project scope and focus on the most valuable insights.
Next, I identified reliable financial data sources like Yahoo Finance, Bloomberg, and Quandl. These will provide historical stock prices, financial statements, and market indicators—critical for training my model. Since raw financial data is often messy, I also outlined a data preprocessing plan to handle missing values, normalize key metrics, and ensure my dataset is structured for analysis.
To stay organized, I created a detailed project roadmap, breaking down tasks into weekly goals. I also set up my coding environment, installing essential libraries like NumPy, Pandas, and TensorFlow to streamline my workflow. With a clear plan in place, I’m ready to move forward efficiently.
Next week, I’ll begin data collection and preprocessing, making sure my datasets are clean and ready for training. Once that’s complete, I’ll experiment with different machine learning models to predict stock profitability and risk. This week was all about preparation—now it’s time to start building. Stay tuned as I dive into financial data and take my first steps toward training my stock advisory model!
Week 2: Collecting Data
With my research and planning complete, this week I focused on gathering and cleaning the datasets that will power my machine learning models.
I started by extracting historical stock data from sources like Yahoo Finance and NASDAQ Data Link, pulling key metrics such as price movements, trading volume, earnings reports, and fundamental indicators (P/E ratio, market cap, etc.). Since raw financial data is often inconsistent, I spent a significant portion of my time cleaning it—removing duplicates, handling missing values, and normalizing variables to ensure consistency across different stocks and time periods.
To make the data more useful for machine learning, I implemented feature engineering by calculating risk-adjusted return metrics (e.g., Sharpe ratio) and trend indicators (moving averages, volatility measures). These features will help my model better assess stock profitability while considering risk—key for making informed investment decisions.
Next week, I’ll start experimenting with different machine learning models, training them on my dataset to predict stock profitability. I’ll test initial architectures like Feedforward Neural Networks and LSTMs to see which performs best. Additionally, I’ll continue refining my features and exploring ways to integrate real-world market performance data for comparison.
This week was all about building a strong data foundation—next up, I’ll put it to work! Excited to take the first steps toward training my AI model. Stay tuned!
Week 3: Building the Model
This week was a major milestone in my project as I began training my first machine learning models to analyze stock profitability based on risk-adjusted returns. While the past two weeks focused on collecting and preparing data, this was the first real test of whether my approach could work. It’s not perfect yet, but it’s an exciting start!
I started by implementing basic neural networks, experimenting with both Feedforward Neural Networks and LSTMs. After splitting my dataset into training, validation, and testing sets, I ran my first models to see how well they could predict stock profitability. The results were not great—yet. I quickly realized that overfitting was a problem, meaning my model was memorizing patterns in historical data rather than generalizing to new stock trends. To counteract this, I added dropout layers, adjusted hyperparameters like batch size and learning rate, and tested different activation functions.
Another challenge was determining which financial metrics actually contributed the most to stock performance. While I had engineered several features (like moving averages, volatility, and Sharpe ratios), some may not be as useful as I initially thought. Next week, I plan to refine my feature selection process to eliminate unnecessary noise and make the model more streamlined.
Despite these hurdles, this week was a big step forward. I now have a working prototype of my model, and even though it needs improvement, it’s exciting to see machine learning applied to real financial data. Next week, I’ll continue tweaking the model, exploring additional architectures, and improving prediction accuracy. Stay tuned!
Week 4: Debugging
This week was all about debugging and refining my stock prediction model. While I made significant progress, I encountered multiple roadblocks that forced me to dive deeper into the data and code.
One major challenge came from missing values in my calculated features. Initially, I encountered an error that prevented my stock data from loading correctly. After investigating, I found that some datasets didn’t always include expected columns due to API inconsistencies. I resolved this by verifying column names and adjusting my approach to use available price data. Additionally, the rolling averages and volatility measures naturally introduced NaN values at the beginning of the dataset. Instead of simply dropping rows, I refined my handling of missing data to ensure my model wasn’t losing valuable information. Another key breakthrough came when I resolved an error while selecting features for training. I discovered that my feature calculations were happening before the dataset had fully loaded. By restructuring my code, I ensured all necessary columns were properly generated before training the model.
By the end of the week, I successfully loaded stock data, engineered key financial indicators, and prepared a clean dataset for training. Next week, I’ll focus on refining my model, experimenting with different machine learning algorithms, and evaluating their accuracy. Excited to see how the model performs!
Week 5: Refinement and Expanding Features
This week, I focused on refining my stock prediction model by improving data preprocessing and experimenting with different machine learning techniques. Now that my dataset is properly structured, I shifted my attention to optimizing feature selection and fine-tuning my approach to better capture market trends.
One major improvement was expanding my feature set to incorporate additional financial indicators. I explored technical indicators such as the Relative Strength Index (RSI) and Exponential Moving Averages (EMA), which could provide deeper insights into stock movements. I also experimented with different machine learning models beyond the initial neural network approach. Through cross-validation, I began evaluating how well these models generalize to unseen stock data.
Despite these advancements, challenges emerged. Some indicators had highly correlated values, leading to potential redundancy in my model. To address this, I analyzed feature importance scores and adjusted the dataset accordingly. Additionally, training deep learning models proved computationally intensive, requiring me to optimize hyperparameters and streamline model architecture to improve efficiency.
Looking ahead, I plan to finalize my model selection by running comprehensive backtesting on historical stock data. By assessing how well my system would have performed in past market conditions, I’ll determine its predictive accuracy and practical viability. I’m excited to see how these refinements will impact the system’s performance in the coming weeks!
Week 6: Evaluating Performance and Debugging
This week marked a major turning point in my project—transitioning from preparing data and building the initial model to actually testing how well it performs on real market conditions. I spent the first part of the week finalizing my neural network’s architecture, refining how the model processed volatility, technical indicators, and historical price patterns. With the guidance of my external advisor, I experimented with different numbers of hidden layers and activation functions to strike a balance between overfitting and underfitting.
After getting the model to run without errors, I began evaluating its predictions. I compared its performance against baseline strategies like simply holding the S&P 500, and also tracked how often the model made correct directional predictions. While some results were promising, others revealed inconsistencies—especially in volatile market periods. This forced me to go back, tweak some hyperparameters, and pay closer attention to how I split and normalized my training/testing data.
One unexpected challenge came from the way my model handled recent data. At first, it performed worse on newer timeframes. With a bit of debugging and research, I realized that incorporating too many older trends without weighing more recent performance was affecting the output. I’m now working on giving more weight to recent market behavior while keeping long-term trends in context.
Next week, I’ll work on improving prediction accuracy and begin analyzing how this model stacks up against expert recommendations in real-world scenarios. Progress has been steady, and I’m excited to continue fine-tuning and optimizing the system!
Week 7: Analyzing Results
This week was all about tuning and testing. With a functioning neural network model in place, my focus shifted to improving prediction accuracy and understanding why the model behaves the way it does. I ran multiple training sessions using different timeframes and adjusted parameters like batch size, learning rate, and the number of epochs. Through this process, I began to see stronger alignment between my model’s predictions and actual stock price movement, especially in stable market conditions.
I also spent time comparing my model's performance to standard benchmarks, such as a simple buy-and-hold strategy with the S&P 500. Encouragingly, in some shorter time windows, my model showed better returns with lower volatility. However, it still struggled during rapid market shifts, which I’m hoping to address by experimenting with more responsive technical indicators and adding momentum-based features.
Another key part of this week was learning how to visualize model performance. I created graphs showing predicted vs. actual returns, which helped me spot patterns and identify when the model underperforms. These visuals will also be useful for my final presentation. With input from my external advisor, I started exploring ways to evaluate the model not just based on raw accuracy, but in terms of practical investment value, like profit/loss and risk-adjusted return.
Next week, I’ll dive deeper into testing this model on new stock tickers and refining its architecture to increase consistency. It's great to see everything start to come together!
Week 9: Final Tests and Insights
With my final round of testing underway, this week was about making sense of the numbers and preparing to turn raw results into meaningful conclusions. After refining my model and experimenting with grouped financial indicators last week, I now focused on seeing how these strategies held up across different market conditions and stocks.
One of my key tasks was applying the model to fresh stock data—real companies, current trends—to assess how well it generalizes beyond the historical training set. I continued tweaking groupings of features, like combining long-term trends with short-term volatility, in hopes of finding more consistent signals. While some groupings still introduced noise, others showed potential—especially in stable markets where momentum indicators played a bigger role. It’s become increasingly clear that context matters, and no one-size-fits-all feature set works perfectly.
I also began documenting my results in more structured ways: plotting comparative charts, calculating error rates, and evaluating performance against the S&P 500. These visual summaries are helping to spotlight key takeaways—both strengths and limitations—that will shape my final research paper. My external advisor offered valuable input this week on how to frame these findings effectively, encouraging me to look beyond accuracy and think about practical usability.
Next week, I’ll shift my full focus to final deliverables: polishing the paper, designing my presentation, and tying everything together.
Week 10: Wrapping Up
This week, I focused on finalizing and polishing my research paper. I consolidated all major findings, refined the writing across each section, and ensured that my analysis clearly showed how the LSTM model outperformed both the dense neural network and the S&P 500 in terms of cumulative returns and predictive accuracy. I also incorporated visualizations, accuracy curves and return comparisons, to make the results easier to interpret and support my conclusions.
Additionally, I reviewed each section for clarity, tightened transitions between ideas, and double-checked that all figures and tables aligned with the narrative. I made sure to include a strong conclusion and discussion of limitations, providing context for how this work fits into broader financial modeling research. With the paper now complete, I’m looking forward to presenting the project and sharing what I’ve learned!