A Better Understanding of Water Consumption Using Machine Learning
Advisor: Swetha Bhattacharya
Internship Location: Yale University
Onsite Mentor: Weicheng Dai
The escalating demand for freshwater, driven by population growth and recurrent water crises, has heightened its significance. Despite California experiencing ample rainfall this year, water conservation has emerged as a critical concern. Facing steep water bills, Californians are actively seeking strategies to reduce their water consumption. In response, I am in the process of developing an AI/ML model that can analyze continuous water usage data, identifying distinct patterns associated with various activities. This data will facilitate the computation of detailed statistics on water consumption per appliance and activity, empowering individuals with insights into the distribution of their monthly bills based on appliance usage like showers, dishwashers, and laundry. The goal is to equip people with knowledge to make more informed decisions, promoting both cost savings and environmental sustainability. Through the utilization of machine learning algorithms, this project aims to offer a comprehensive breakdown of expenditures on different water-consuming activities, such as showering, dish cleaning, and laundry.
Project Blog
Week 1: Introducing My Senior Project: A Dive into Household Water Usage
Welcome to the inaugural blog post for my senior project, where I'll delve into the motivation behind my endeavor, the project's objectives, and the groundwork I've laid thus far.
Background: Growing up in a family passionate about gardening and taking long showers, discussions about our water bill were commonplace. From lengthy showers to hidden leaks, we were often uncertain about our household water usage. Last year, we discovered a hidden leak in our bathroom that had gone unnoticed for some time, highlighting the need for better insights into our water consumption habits. This personal experience prompted me to explore ways to monitor and understand household water usage more effectively.
Global water usage has significantly increased over the past century, reaching over 4 trillion cubic meters in 2022. California is among the top five U.S. states in water consumption, utilizing an astonishing 38 billion gallons of water daily. During the 2012-2016 drought, urban water usage had to be reduced by at least 25%, highlighting the critical need for water conservation. Former Governor Jerry Brown implemented fines of up to $500 for every 748 gallons of wasted water, emphasizing the importance of addressing water wastage. Although current water bills provide overall usage and costs they lack specific details on individual appliance consumption to assist in giving the opportunity for households to take control of their water.
Purpose: Hence, the primary aim of this project is twofold: to contribute to more sustainable water use practices and to address the lack of awareness among individuals regarding their water consumption habits. By developing a system or tool that enables users to monitor and gain insights into their household water consumption, the project seeks to empower individuals to make informed decisions to reduce overall water usage. Ultimately, the initiative aims to promote water conservation and support more efficient water management practices in the face of escalating water scarcity challenges.
Prior Technical Research: For this project, I've decided to employ Python along with machine learning techniques—supervised, semi-supervised, and unsupervised, the choice of which is yet to be determined—to analyze data collected from a Badger Recordall M25 Water Meter, as depicted below. This meter continuously logs water usage in gallons as water flows through it, offering valuable insights into household water consumption patterns.
To retrieve readings from the meter to the Raspberry Pi, I utilize three wires: green (data), black (ground), and red (clock/power), as shown below. While the display on the meter shows the accurate number of gallons, it's not the data transferred. Instead, I've devised a method for decoding the clocked data into gallons and have automated it to continuously record data whenever the meter reading increases (i.e., changes).
Furthermore, I'll be utilizing Raspberry Pi and PyCharm for software implementation and data analysis. My prior experience with Python and machine learning algorithms provides a solid foundation for this project, enabling me to effectively navigate the technical aspects. This week, my goal is to refine my supervised model using a small set of labeled data and to attempt k-means clustering on the same dataset to explore a simple unsupervised method for training this model.
Stay tuned for future updates as I delve deeper into the development process, exploring data collection methods, algorithm implementation, and the integration of the system into my household.
Week 2: Navigating the Waters of Clustering and Label Accuracy
Welcome to the latest update on my senior project, where the journey through data analysis continues. This week's focus has been on further exploring K-means clustering and how it holds up against various representations of our data set.
In the previous installment, I clustered water usage data into seven categories. My aim was to observe whether a more granular approach might yield clearer distinctions between clusters. After experimenting with cluster counts ranging all the way from 1 to 300, I turned my attention back to the sweet spot—clusters numbered at seven.
Interestingly, while the majority of these clusters were well-defined, two overlapped significantly. This observation was confirmed by a stacked bar plot, which should have shown a clear segmentation of water appliance usage within each cluster but instead indicated some inconsistencies (Image 1 & 2).
This brings us to a crucial point: the integrity of our labels. The data was manually annotated, based on individuals’ accounts of their water usage. The margin for human error here cannot be ignored—it could be significant, enough to skew our model's performance.
Upon inspecting the labels more closely, I discovered a discrepancy. The variability in the data was considerable, suggesting that the features identified by the supervised model might be too unique to certain appliances, rendering the labels ineffective for reliable model training.
To address this, I’ve taken a more hands-on approach to data collection, personally ensuring the accuracy of the labels. This new, meticulously gathered data will serve as the foundation for subsequent model evaluations.
The images included show that while the separation of some clusters appears evident at a glance, the underlying distribution of labels tells a different story—a reminder of the challenges we face in this field (Image 1 & 2).
Moving forward, I'll be harnessing this refined data set to test the robustness of my models. Stay tuned as we strive for more precision in our clustering algorithms and a deeper understanding of our data.
And that's the current snapshot of my project—equal parts challenging and exhilarating. Here’s to clearer insights in the weeks to come!
Week 3: Revolutionizing Water Usage Insights with Unsupervised Learning
This week, I embarked on an exploration of a novel data preprocessing platform tailored for my smart water meter data. My journey began in February 2023, when I first installed my smart water meter. This device has been meticulously logging the date, time, and water meter readings in gallons, producing records akin to: ('12/02/2023 17:36:29', '01018257').
Initially, my data processing strategy was straightforward. I grouped 14 readings together based on the timing of an appliance's operation, a method suitable for supervised learning where data is labeled explicitly. However, my ambition to apply k-means clustering—an unsupervised learning technique that operates without predefined labels—prompted a shift in my approach. Given that the duration and volume of water usage significantly hint at the type of appliance in use, I sought a more nuanced data processing method to capture these variations effectively.
To accommodate the unique characteristics of each appliance's water usage, I refined my preprocessing strategy as follows:
- Different Time Segments for Appliances: Segmentation stops when there's no change in the water meter reading during a minute, accurately reflecting appliance usage periods.
- Aggregated Water Usage: Calculation of total or average water usage within a specific timeframe, offering insights into consumption patterns.
- Delta Features: The variation in water usage between consecutive windows, highlighting consumption trends.
- Rolling Statistics: These are calculated over a selected number of time windows to smooth out short-term fluctuations.
- Time Features: Encoding the time of day and day of the week to investigate patterns related to time.
A challenging aspect to consider is the behavior of appliances like dishwashers or laundry machines, which intermittently use and stop using water throughout their cycles. My methodology accounts for this by learning to associate such patterns with the specific appliances, despite the temporary cessation of water flow.
In the upcoming week, I plan to test the effectiveness of these adjustments on the performance of unsupervised modeling techniques. This exploration not only promises to enhance the precision of appliance usage identification but also paves the way for innovative water consumption analytics.
Week 4: Unveiling Water Wastage: A Deep Dive into Leak Detection Through Data Analysis
This week unfolded as a journey into unanticipated challenges, revealing the intricate dance of errors and outliers within my dataset. In a recent endeavor, I embarked on processing my data, categorizing it into groups based on water meter readings that ceased fluctuating for a minute or longer. However, upon delving deeper into the analysis and padding phase, a startling discovery was made: the maximum count of water meter readings in the dataset reached an astonishing 1536.
Initially dismissing this figure as a potential outlier and contemplating its removal, I stumbled upon several similar instances of large counts. It wasn't until a rigorous bout of analysis and debugging that a significant leak was unearthed, dating back to 2023 in our bathroom. This anomaly was starkly highlighted within my water meter reading groups, showcasing a continuous flow of water usage exceeding 12 hours. Although my project's early stages did not prioritize addressing leaks, this revelation underscored the importance of integrating such considerations.
The dilemma then presented itself in the form of data processing methodologies: opting between padding and utilizing time series databases. The latter, while appealing, necessitates a new generator, complicating the incorporation of fresh data. Given the unsupervised nature of the task, padding with zeros might theoretically be inconsequential. The initial approach to identifying leaks relied on a simple count metric, positing that any group exceeding 100 readings likely indicated a leak. This method, however, proved inadequate, as frequent meter readings could inflate the count without necessarily indicating prolonged water use.
Transitioning to a timer metric, which flags any water usage exceeding 60 minutes as a leak, offered a more nuanced perspective. Yet, acknowledging the possibility of legitimate high water usage scenarios, such as lengthy showers, I am now contemplating two innovative approaches: employing k-means clustering with padding up to 1536 or preemptively removing all detected leaks and applying k-means clustering with padding up to 356. This experiment aims to discern whether leaks can be automatically identified within the clustering process or if a preliminary phase of leak removal is requisite for effective unsupervised learning. This exploration not only highlights the technical challenges of data analysis but also underscores the critical environmental issue of water wastage through unnoticed leaks.
Week 5: Unveiling Insights - A Journey through Data Analysis and Automation
A deep analysis of my data is possible with K-means. But before I delve into my successes of this week, I must address the issues I encountered. I created two datasets: one without leaks and one with leaks. While removing leaks from the dataset without leaks, I encountered a problem. I was deleting groups from the same database I was iterating through, resulting in heavily skewed future model numbers. My data removal process was eliminating unnecessary groups but retaining leaks in some cases. I wasn't able to directly identify this error until I attempted to print the maximum count within the groups. To rectify this, I had to create a list of leaks and simply remove them post-iteration through the larger dataset. Furthermore, in earlier iterations of K-means, I wasn’t processing my data. While time values like month, day, year, hour, minute, and second do not need processing, the water gallons meter reading does. It can significantly impact K-means clustering, which likely obscured a clear understanding of the data's distribution. The clusters weren't distinct earlier. However, I normalized my data using min-max normalization to reduce its influence to just a 1-reading difference.
Now, onto the analysis. After printing the values within the K-means clusters, I began to compare them with my own recorded data and noticed two things. First, the groups were accurately formed. Each group closely resembled the manually recorded data, indicating that this automated grouping process worked effectively, eliminating the need for human-made decisions, which is advantageous for unsupervised learning processes. Second, wash groups were categorized as wash groups, laundries as laundries, and few outliers existed. The leaks were also prominently apparent.
The next step in this project is to determine how to automate the grouping of any data to its corresponding appliance, eliminating the need for manual analysis.
Week 6: Navigating through a series of plots
This week was dedicated to tackling an error within my code, particularly concerning my water meter data, which exists in a 3D format with size (10132, 1536, 7). With 10132 being the number of inputs, each input has the size of 1536x7. 7 is the different time components and then the actual water meter reading: "month, day, year, hour, minute, second, VALUE". and 1536 is the number of water meter readings in each. As mentioned in an earlier post I padded the values to fit this size. However the hurdle arose during k-means clustering, requiring the implementation of Principal Component Analysis (PCA) to handle the non-2D data for visualization.
For those not familiar with PCA, it's a technique used in data analysis and machine learning to simplify complex datasets by reducing their dimensionality. Essentially, it finds the directions (or principal components) that capture the most variance in the data and projects the data onto these new axes. This can be particularly useful for visualizing high-dimensional data in a lower-dimensional space.
Working on a MacBook with an M1 processor posed a challenge, as the sklearn library, commonly used for this task, couldn't be easily downloaded due to inherent limitations within the library itself. This limitation stems from the fact that the sklearn library relies on certain dependencies that may not be fully compatible with the architecture of the M1 processor.
To address this obstacle, my initial strategy involved manually coding the PCA function. However, I encountered discrepancies in results across different methods, leading to uncertainty regarding the accuracy of the visual output.
Seeking a reliable solution, I transitioned to Google Colab, which is an online platform that provides free access to computing resources, including GPUs and TPUs, along with pre-installed libraries like sklearn. By leveraging sklearn's PCA and k-means functions in this environment, I was able to overcome the limitations imposed by my local setup.
Surprisingly, this endeavor produced yet another perplexing plot that was very obviously not accurate (image depicted below). While it would be lovely to have a plot like this, I know my data is not this perfect, no data is. Digging deeper into the issue, I realized the underlying problem: the data's size was too large. Despite PCA's attempts at dimensionality reduction, its effectiveness diminished with excessively large datasets. The size 1536x7 means there are 10752 parameters that are being reduced to 2 principal components. This is causing the obscure inaccurate visuals.
Consequently, I've opted to forego the visual representation of the k-means model and instead focus on individually analyzing each data point within its respective group in the upcoming week. This shift in approach will allow for a more granular examination of the data and potentially uncover insights that may have been obscured by the challenges encountered thus far.
Week 7: Unraveling K-Means Clustering: Analyzing Time and Water Meter Readings
I worked on analyzing the results of the k-means of 7 clusters for both the model using time and water meter readings, and the model using just the water meter readings. As a reminder, the dataset contained water meter readings for each time an appliance was used (hypothetically). This process was automated, so no method can guarantee this was done with 100% accuracy. However, the analysis showed it was done with high performance. Anyhow, time could be another indicator for the appliance, so I tested the k-means with and without it. I want to see which model performs better.
Although the goal was to further analyze the results of the k-means last week, I found a big error. The size of the data was too large to reduce into 2 dimensions, causing a big problem with the plots, also hinting at a failure within the k-means algorithm itself.
I experimented to debug the problem. I tested the plots with all the data, including time. Size of torch.Size([10132, 1536, 7]). However, it was suggested that time and gallons might be pushing the model in the incorrect direction, so I tested the model without time and focusing just on gallons. Features might be more stable: torch.Size([10132, 1536, 1]).
The plots are shown below:
When I saw that these were the same, I knew something was wrong, especially since the data should have at least moved. The features are definitely not the same. Hence, I began to review each step and noticed something unfamiliar about the transformation of my tensor from 3D to 2D. Keep in mind the k-means algorithm can only take 2D data, so my data is reshaped. However, I was shaping it to the size of torch.Size([15562752, 1]), where it should instead be torch.Size([10132, 1536]).
Upon fixing this issue, I noticed my plots and clusters change accordingly. The left plot includes time and gallons, the right plot includes only gallons.
This method using PCA with k-means was earlier performed to ensure the PyCharm model was correct. Visually, it is the same plot inverted, hence meaning my algorithms within my other application still work, and I can revert to only one platform. However, for the ease of this project, I will continue to use Google Colab to display the various plots (sklearn is easily usable).
To the analysis of my two accurate mean plots: The plot with time seems to be more curved, and time massively affects the model. But there is no way to perform or prove this without looking at the individual data points within the clusters and testing whether or not they are in the same group.
Within this analysis, I came across yet another problem. My original data was nothing like my new data; it had undergone min-max normalization and was reshaped from 3D to 2D data. To uncover my data would require a lot of meticulousness, and I could not make a single mistake. Instead, I decided my workaround would be the indices. I would get the indices of the data points within the clusters and then take that same index from the original set of data in order to analyze it. Until next time, where more plots and graphs can be seen regarding the results of these endeavors.
Week 8: Challenges in K-Means Clustering for Appliance Recognition
What initially appeared as strides forward in the analytical journey have revealed themselves as missteps, marked by subtle yet significant communication gaps. Central to this week's objectives was a comprehensive analysis encompassing the creation of graphical representations and a meticulous examination of performance metrics. However, amidst this process, a critical issue came to light—the presence of discrepancies in timings within the dataset.
In an endeavor to delve deeper into the intricacies of my final kmeans model, I embarked on an exhaustive exploration of individual data points residing within each cluster. This exploration culminated in the development of an automated function aimed at discerning the underlying appliances associated with each data point. Subsequently, the task at hand pivoted towards the construction of a secondary model intended to provide insights into the interpretation of kmeans results. Yet, amidst this endeavor, a confounding observation emerged: each cluster exhibited high accuracies in representing "flush" events. This discovery prompted a period of profound perplexity and necessitated a thorough investigation.
A meticulous examination of the dataset ensued, with a keen eye for recurring patterns or potential distortions. However, despite the prevalence of "flush" events within the dataset, the varying accuracy percentages observed across clusters defied simple explanations. It was in revisiting the original dataset, which included timing data previously disregarded, that a breakthrough occurred. Upon scrutinizing the results of the revised kmeans model, a glaring truth emerged: the clusters were undeniably influenced by time.
Through the utilization of identical indexing methodologies, a comparative analysis between the time-inclusive kmeans clustering and its time-omitting counterpart was conducted. This analysis yielded a pivotal insight: while the temporal feature had been deliberately omitted in the latter, the clusters remained inherently tethered to chronological sequences. This revelation shed light on the persistent temporal dynamics within the dataset, even in the absence of explicit time-related features.
To summarize, three various datasets are now being tested. The original data containing only the water meter readings (range: 331 to 1018257), the original data + timing of the water meter reading, the change in the original data. Since last week revealed a negative towards the time inclusion we eliminated the second dataset and focused on enhancing the first dataset.
Looking ahead, the focus shifts towards refining a kmeans algorithm that capitalizes on the transitions in water meter readings, rather than relying solely on absolute values. It is through this nuanced approach that further insights into the underlying patterns embedded within the dataset are anticipated to be unearthed, thus paving the way for enhanced analytical precision and depth.
Week 9: Tackling Inaccuracies in Water Usage Data: A Journey from K-Means to Gaussian Mixture Models
When analyzing water usage data for appliance detection, I found that the K-Means clustering algorithm didn't quite meet my needs. The rigid boundaries of K-Means made it difficult to classify data points that could belong to multiple categories. This isn't reflective of real-world scenarios, where things are often more fluid. Thus, I started exploring Gaussian Mixture Models (GMM) as a potentially better solution. While the benefits of GMM aren't fully realized yet, my first step has been to adjust the data to better suit this new approach.
One key issue I faced was the inability to distinguish different appliances based on raw meter readings. Without clear patterns, I couldn't expect accurate model outcomes. The variation in reading frequency made it challenging to separate data into distinct clusters. For instance, some data points had only a few readings while others had over a thousand. It's hard to determine if both should be considered as "sink" appliances. To make sense of this, I shifted focus to flow rates, a more consistent measure. The flow rate per minute, along with the average flow rate, provides a more reliable basis for clustering.
During this process, I discovered that my earlier meter readings were off due to a missing decimal point. The readings lacked the decimal separator, so I simply divided all values by 10 to correct them. This mistake became evident when I calculated flow rates and noticed absurdly high numbers—something you'd never expect from a typical household appliance.
A funny incident happened while analyzing my data this week. We had recently implemented a new sprinkler plan, but I noticed a significant spike in water meter readings every 10 seconds in the middle of the night. Since nobody in my household was using water at that time, it pointed to a potential problem. It turned out that our sprinkler system was broken, leading to continuous water flow. After shutting it off, the anomaly disappeared. So, even if my model doesn't achieve perfection, at least this project helped us identify a major issue with our sprinkler system.
Moving forward, my plan is to refine the data and further explore GMM to see if it can provide better clustering results. While the journey has its challenges, it has already proven useful in unexpected ways.
Week 10: Developing a Better Clustering Strategy for Water Usage Analysis
This week, I made significant progress in the unsupervised segment of my project. As I mentioned last week, one of the major challenges I faced was the randomness of the data. It was chaotic, making it extremely difficult to determine which specific appliances—or even how many appliances—were running. To address this, I took the first step by reducing the data to a set of features that a model could use to identify the appliances.
After analyzing the dataset and researching how water experts determine which appliances are in use, I narrowed down my features to the following:
- Duration
- Total water consumption
- Mean flow rate
- Maximum flow rate
- Minimum flow rate
- Flow rate variance
- Flow rate changes
- Number of spikes
- Duration of high flow
- Time of day
I initially considered removing 'time of day' because it was a text-based item, but then decided to include it for the sake of completeness and later re-evaluate its impact on the model's complexity.
Of these features, I expected that the mean flow rate would play a key role in my model's ability to identify specific appliances. It tends to remain consistent for a given appliance, regardless of when it's used. The main source of confusion occurs when multiple appliances run concurrently or within a similar timeframe. This is where I began developing my models.
Although I knew I would rely on GMM (Gaussian Mixture Model), which allows for softer clusters—a single data point can belong to more than one cluster—I initially attempted K-means clustering to test the "appliance theory." However, I found that K-means wouldn't handle outliers or heavy overlap well. This led me to switch to DBSCAN (Density-Based Spatial Clustering of Applications with Noise), which is robust against noise and outliers and focuses on dense regions where water usage patterns are consistent.
DBSCAN requires two parameters: EPS, which represents the distance at which neighboring points start to significantly separate, and min_samples, which indicates the minimum number of samples within a cluster. Given that appliances should share some commonalities, the min_samples parameter wasn't as critical in my context. To determine the optimal EPS, I plotted the distances and identified the elbow in the curve at 1.5, which became my chosen value for this parameter.
The results from the clustering models were promising. The clustering based on "mean flow rate" showed more diversity, suggesting a useful method for identifying different water appliances. However, I noticed that a lot of data points could belong to multiple categories, suggesting that I might need to revisit GMM for further analysis.
To validate my approach, I tested my models using newly collected data on simpler water appliances. The graph results were encouraging, especially when analyzing "mean flow rate," showing improved diversity in clustering. The outcome was similar to the results from DBSCAN, with some points possibly belonging to two or more clusters. This indicated that the models were indeed flexible and could accommodate various scenarios of water appliance usage.
My goal for next week is to continue refining my models and validate my clustering approach with this new dataset. I also aim to test my models on additional data and explore whether reintroducing 'time of day' into the dataset adds value. With more controlled conditions and further analysis, I hope to gain deeper insights into identifying and categorizing various water appliances.
Week 11: Revolutionizing Water Appliance Detection: Harnessing Calibration and Automated Point Generation for Smart Homes
Enhancing Water Appliance Detection: Leveraging Calibration and Automated Point Generation
This week, I've been immersed in the development of an advanced semi-supervised model aimed at precisely detecting water appliance usage within a household. The crux of this endeavor lies in the meticulous calibration of each appliance and the ingenious automation of data point generation. Let's delve into the intricacies of these concepts and their significance in revolutionizing appliance detection.
The Importance of Calibration: Unveiling Appliance Characteristics
Traditional clustering algorithms like k-means and DBSCAN rely solely on unsupervised learning techniques, making them susceptible to ambiguity and inaccuracies. To overcome this limitation, I've introduced a novel calibration process that sheds light on the unique characteristics of each appliance.
Sequential Appliance Activation: Users are guided through a sequential activation process, where each appliance is turned on individually, mirroring their typical usage patterns. By isolating the operation of each appliance, we obtain clean and informative data, enabling precise characterization of their behavior.
Feature Extraction: During calibration, a rich set of features is extracted from each appliance's operation, including duration, total water consumption, mean flow rate, max flow rate, min flow rate, flow rate changes, number of spikes, and duration of high flow. These features serve as the foundation for understanding and distinguishing between different appliance usages.
Automated Point Generation: Unraveling Appliance Combinations
One of the most challenging aspects of water appliance detection arises when multiple appliances are activated simultaneously. The diverse features of each appliance make it challenging to delineate clear cluster boundaries, leading to ambiguity in classification. To address this challenge, I've devised an innovative solution: automating the generation of data points corresponding to combined appliance usage.
Comprehensive Data Representation: Leveraging the calibration data, I've developed a function that generates all possible combinations of appliance usage. By capturing every conceivable scenario that may occur within the household, this approach ensures a comprehensive understanding of appliance interactions and usage patterns. In other words, my solution is to automate the points so the model is automatically forced to “read” the points where appliances are turned on together. Using the calibration data I developed a function to create all other possible combinations and plot those within a massive graph as visible below. This does not make sense to us as we so so many points but if you single out one appliance such as the dishwasher and add a singlular usage of the sink a new feature point is plotted on PCA, then if you add another sink another point is made on a parabola. It seems that every possible combination has its own graph. This would theoretically allow for every single possiblility that occurs wihtin the household removing the problem of lack of data.
Visualization with PCA: Principal Component Analysis (PCA) serves as our visualization tool of choice, allowing us to reduce the dimensionality of the data to two dimensions and visualize complex relationships. Each point on the PCA plot represents a unique combination of appliance usage, providing valuable insights into the dynamics of water consumption within the household.
Conclusion: Advancing Appliance Detection for Smart Homes
In conclusion, the integration of calibration and automated point generation represents a significant leap forward in water appliance detection for smart homes. By combining supervised calibration with innovative data generation techniques and visualization tools like PCA, I am able to gain deeper insights into appliance usage patterns, paving the way for more accurate and reliable detection models.