Exploratory Data Analysis using machine learning on water quality (Temperature, Dissolved Oxygen, Ph, ORP) using machine learning
Advisor: Mr. Andrew Magee
Internship Location: UCSD, Engineering Dept.
Onsite Mentor: Dr. J Garza
The goal of my senior project is to write a research paper by drawing predictions on various types of water quality data from multiple lakes. I will focus on temperature, dissolved oxygen, pH, ORP, and store them in a structured database while removing any inconsistencies or redundancy. I intend to use Exploratory Data Analysis to visualize trends and understand correlations between the different parameters. By combining Exploratory Data Analysis with machine learning algorithms, I will make predictions based on the following four assumptions: Linearity Residuals, Independence Assumption, Constant Variance Assumptions(Homoscedasticity), and Normality Assumption. In addition, I will use multiple graphs and different statistical tests to support and verify these assumptions. For example, I hope to interpret QQ-plot and Shapiro-Wilk test to check Homoscedasticity. From these assumptions, I aim to make predictions on enhancing water quality, thus creating positive impacts on the environment, preventing waterborne diseases and securing clean water for drinking and agriculture. The project will display as a research paper with all the proper procedures.
Project Blog
Week 0: Introduction and Background
Hi everyone, my name is Louis. I am passionate about data science. So I decided to analyze water qualities through multiple lakes in California. My goal is to collect various types of water quality data from multiple lakes from online databases and store them in a structured database with removing any inconsistencies or redundency. Specifically, my data will be focused on temperature, dissolved oxygen, pH, and ORP.
I was first exposed to data science and machine learning with R-studio at an online summer program in UC Berkeley. From there I learned how to create the basics of data analysis through R-studio and python. In school, I studied Java in AP Computer Science A, gaining proficiency in functions, loops, data types, arrays, and algorithms. Outside the classroom, I was able to expand my knowledge in a coding platform called Leetcode. I explored more algorithms through this platform; learning computer algorithms. As I learned more complex algorithms and problems, I also started competing in the USACO contests. There, I learned algorithms and dynamic programming, soon becoming most proficient with the Breadth-First Search (BFS) and Depth-First Search (DFS) graph algorithms. These provided a rich understanding of graph theory, enabling me to analyze and manipulate graphs in-depth.
I also had the opportunity to design a Web Security Platform project in collaboration with the South China University of Technology. My team and I developed an automated static vulnerability detection tool to identify potential errors like SQL injections and packet loss in large websites and applications. We documented all the procedures, testing, results, and future improvements in a research paper. Therefore, I am familiar with all the steps and procedures involved in writing research papers.
In the summer of my junior year, I attended the COSMOS summer program, where I worked on an autonomous surface boat project and displayed water quality on a Python dashboard. For my senior project, I aim to continue this research by collecting water quality data and using Exploratory Data Analysis (EDA) to make crucial suggestions for environmental improvements.
By deciding on the variety of water quality data in multiple lakes, my project aims to display a full research paper providing a comprehensive understanding and structure for the health of the ecosystems. Storing large data files in a database allows fast and efficient access toward a specific data set. Implementing machine learning and exploratory data analysis(EDA) can provide a scientific and a systematic approach for the environment. Through EDA, I aim to draw conclusions that ensure water safety, aiming to prevent waterborne diseases and secure clean water for drinking and agriculture.
Week 1: Data Selection
Hello everyone! Since I was out on a trip on Monday and Tuesday, I began working five hours each day from Wednesday to Friday to fulfill my requirement of 15 hours per week. My week 1 goal was to Select two lakes, the focus will be on collecting data through the California Open Data Portal on temperature, dissolved oxygen, pH, and ORP, which will be structured in a database.
On Wednesday, I started working on my senior project by setting up the environment in Python, which included downloading Anaconda and Visual Studio Code (VSCode). This setup will provide me with a designated space and file to process data analysis and visualization for my selected lakes. Additionally, I worked on configuring my GitHub to link with my VSCode. This allowed me to save all progress coding and files I performed locally onto the web. I also spend some time researching how to perform data visualization on Model Goodness of Fit. ,Model Goodness of Fit determines whether the observed values match with the expected data model. This helps later on in my research to gain a better understanding of which specific data parameters I should select.
On Thursday, I continued my research on data visualization focusing on Model Goodness of Fit. I also begin to work on my weekly goals on selecting two lakes focusing on various types of data parameters. To begin this, I set up a local database using DB Browser to store all the specific parameters. This gives me an advantage because even if I miss a column of data, it won't result in an error during data analysis. So I have to transfer all the data qualities from a CSV file into my local database. I chose Lake Elizabeth, which is just a few minutes away from my school, and Lake Murray in San Diego for my research on an open database website called Hillsborough County Water Atlas to download water quality data, including temperature, dissolved oxygen, pH, and ORP. Currently, I am facing a problem with transferring data into my local database in DB Browser.
On Friday, I started drafting my blog and continued to fix the problem of transferring data into my local database in DB Browser.
Week 2: Data Processing
Hello everyone! Welcome to my second week of blog posts. This week's goal for me was to remove any outliers, correct any errors, and standardize the scale of the data for parameters like temperature and ORP to facilitate comparison. In addition, I will apply transformations if necessary to normalize the distribution of values.
At the beginning of the week, I continued working on transferring the data and finally fixed the problem from last week. I was able to remove any inconsistencies and outliers from the datasets. During data processing, I noticed a significant amount of missing or outlier data in the total dissolved solids columns. So, I simply deleted those entries and replaced them with zero. I also created a Jupyter Notebook file showing all the parameters such as latitude, longitude, temperature, pH, and total dissolved solids (TDS). Additionally, I used a given ID name for sorting every data entry. I was able to find all the parameters for Lake Murray in San Diego. However, I had trouble finding data for dissolved solids and pH values in Lake Elizabeth. Therefore, I decided to continue analyzing my data, focusing only on Lake Murray.
As I met with my external advisor, Dr. Jay, he recommended that I use Flask during data analysis. So, I am planning to include that in week 6 when I start analyzing the data. Next week, I will start visualizing trends and patterns of different parameters as well as drawing some basic analysis from a linear regression summary report.
Week 3: Data Visualization
Hello everyone, welcome to my week 3 blog. This week's goal is to visualize my dataset using scatter plots and linear regression summaries.
From the scatter plots and linear regression summaries, we can draw some basic predictions. For scatter plots that do not show a linear pattern simply means that the Y-axis has no relationship with X-axis. Each Time Series Plot identifies trends, patterns, or outliers in the dataset. In all the scatter plots: Temperature vs. pH, Temperature vs. TDS, and pH vs. TDS, the graph does not seem to show a linear relationship. Therefore, the variables do not have a direct linear relationship between each other. However, in the coefficients for the Temperature vs. pH graph, we observe that for each degree increase in temperature, the pH is expected to increase by about 3.0647, assuming all other factors remain constant. A similar relationship is observed from the coefficient value for pH vs. TDS. Next week, I will choose to focus more on predicting the R-squared value and intend to begin analyzing the data using the Model of Goodness data analysis.
Week 4: Linear Regression Model
Hello everyone, welcome to my week 4 blog. We're almost halfway through! This week, my goal is to continue visualizing my dataset using scatter plots and linear regression summaries.
Last week, I focused on analyzing the coefficient value of each graph. This week, I move on to focus on the R-square value. In simple terms, R-square values determine how well the graph predicts its relationships. If you have an R-square value close to 1, this means the graph is very accurate in terms of predicting the next variables. Similarly, if you have an R-square value close to 0, it means the graph is very inaccurate in terms of predictions.
In the graph of Temperature vs. pH, the analysis showed an R-squared value of 0.597. This means that approximately 59.7% of pH data can be predicted by temperature data. This a strong relationship, suggesting that as temperature changes, pH levels also change in a predictable pattern. In another graph of pH vs. TDS shows an R-squared value of 0.793. This shows stronger and more accurate predictions compared to the previous graph. Both graphs depict strong predictions indicating that as pH changes, both temperature and TDS will change by a similar amount.
Next week, I will move on from Linear Regression to the Model of Goodness, checking the linear assumptions of the graphs!
Week 5: Linearity
Hello everyone, welcome to my week 5 blog. We're at the halfway point. This week's goal was to create a Jupyter notebook to perform Linear Regression on our data, focusing on the Linearity Assumption.
To check linearity, we first needed to plot residual plots for each of our datasets. Residual plot can help understand whether the relationships between temperature and pH, temperature and TDS, and pH and TDS are in a linear relationship. If the residual plots show a random pattern, this tells that the linear model assumptions do not hold. Starting on Monday, I looked up how to plot a residual plot in Python using built-in. On Tuesday, I began using residual plots to analyze my datasets. The plots all displayed a downward trend for all three graphs, showing that as temperature decreases, pH also decreases, and following the same trend for other comparisons. Therefore, the graph shows our first assumption does hold in our Model of Goodness data analysis.
After discussing my progress with my external advisor, he suggested that I use the Durbin-Watson Statistical test as another method to test linearity. So on Wednesday, I began learning about the Durbin-Watson test and how to apply it to my datasets using statsmodels. From the website, I learned that the Durbin-Watson statistic ranges from 0 to 4. A value less than 2 indicates positive autocorrelation. This means that an error in one variable is likely to cause an error in another. A value greater than 2 indicates negative autocorrelation. This also means that an error in one variable is unlikely to affect another variable.
The results of the Durbin-Watson test were as follows:
Durbin-Watson Statistic for Temperature vs. pH: 0.054
Durbin-Watson Statistic for Temperature vs. TDS: 0.073
Durbin-Watson Statistic for pH vs. TDS: 0.066
These results are all significantly lower than 2. This tells us that all three graphs are positive autocorrelation in the residuals model. This draws the same conclusion with my residual plot analysis. This is because the Durbin-Watson test suggests that the variables are highly dependent on each other, which holds true for the linearity assumptions.
Next week, I will start checking the Mean of Residuals Assumptions!
Week 6: Mean of Residuals
Hello everyone, welcome to my week 6 blog. This week’s goal was to continue performing linear regression on our data checking for the following Mean of Residuals Assumptions.
The mean of residuals is typically a horizontal line crossing through the residuals plot, checking for the two following assumptions: Bias and Symmetry. A non-zero residual plot means that the model is biased. This indicates that the model can either be overestimating or underestimating towards both dependent and independent variables. This error could be caused by missing significant data entries or incorrect model forms, and with other errors. On the other hand, symmetry checks for the spread of the data. If you see a graph that evenly spreads above and below the mean of the residuals line, this shows that the model's residuals are randomly distributed and contain no bias.
In my data, I successfully inserted my mean of the residuals function into my residuals graph, marked in a dotted red. This line spreads horizontally across the graph at 0, while containing a good amount of data both above and below. Therefore, we can conclude that my data has no significant bias. I was thinking of using Levene's statistical test to check the symmetry and variance of the data. However, as I was importing the statistical package in Python, I got into a systematic error that I have not fixed yet. Therefore, I will continue fixing the error and hopefully get the result of Levene’s test by next week as well as starting to analyze equal variance.
Week 7: Equal variance/Homoscedasticity
Hello everyone, welcome to my week 7 blog post. This week’s goal was to continue performing linear regression through my lake datasets, checking for Equal variance/Homoscedasticity. This is the second to last assumption, which means that my senior project is coming to an end. I decided to use Levene’s test to display the result of equal variance of residuals/Homoscedasticity assumption.
Levene’s test is a statistical test that allows me to see the variance of the residuals, whether my graph is above, equal or under variance by looking at the p-value. The p-value is a unit for Levene’s test that shows the strength of your data, which also means the probability to observe from existing data points.
After a few days of fixing errors and rereading documentation, I finally imported a statistics package in Python for me to run Levene’s test. My p-value result was 1.0183, which is well above 0.05, demonstrating that my residuals are in equal variance.
On Thursday, I started working on importing Quantile Quantile-plots (QQ-plots) to analyze the normal distributions of my data sets. I first started understanding the structure and its specific functions on Stackoverflow, and then I began to connect it to my datasets. Hopefully, I can finish next week and start to wrap everything up!
Week 8: Normal Distribution
Hello everyone, welcome to my week 8 blog post. This week’s goal is to continue performing linear regression on our data, checking for a normal distribution of residuals.
My original plan was to use quantile-quantile plots to display the normal distributions of residuals in my data analysis. However, after a few days of working, the results led to some display errors. So, I quickly decided to switch to a method using histograms to check for normal distribution of residuals. Histogram is a bar graph showing how data is distributed. The histograms result displayed that all three of my graphs have significantly different distributions. So, my results tell me that my lake quality data is not normally distributed. Part of this reason could be that we only collected parts of the lake and some data were collected in different locations.
Next week, I will start to analyze all the assumptions from my data analysis and draw out a few conclusions on how to improve my water data quality, as well as starting to make my final presentation.
Week 9: Final Predictions
Hello everyone, welcome to my week 9 blog post. It is the last two weeks before the final presentations. My data analysis is about to come to an end. This week, I have mainly worked on comparing my model to a standard water quality dataset and working on the final presentations.
The standard value for temperature is around 4-25 Celsius, and only about 12.6% of my data lies in the standard range. Most of the temperature values are above 25 Celsius, so I will give suggestions on how to reduce the temperature value.
The standard value for pH is around 6.5 to 8.5. Only around 22.7% of my pH data falls in this range. Similarly, the standard value for Total Dissolved Solids is less than 500 ppm, and only around 29.2% falls in the range.
I will draw conclusions on how to enhance water quality data based on the comparison mode and my previous assumptions.