Pose Detection for Feedback on Weightlifting Form in the Gym
Advisor: Mr. Andrew Magee
Internship Location: Meta, CA
Onsite Mentor: Jeff Chow
With 39.6% of U.S. adults obese and another 31.6% overweight fitness has never been needed more in the United States of America. Although gyms are accessible and relatively cheap, these statistics show that there are other factors still hindering citizens to workout. One common factor is the lack of knowledge. Working with a personal trainer is a great augmentation to experience in the gym, but not all people have access to this. Personal training can be very expensive, averaging 65 dollars an hour. I aim to develop an open-source app utilizing pose detection in order to substitute for a personal trainer, giving feedback on exercises performed by lifters.
My app builds on an open-source convolutional neural network framework called VideoPose3D. Once trained into a model, this framework can output a skeletal model from an image, extracting the position of detected joints. I will optimize the model's accuracy by training it with diverse exercise datasets to cater to various demographics. My app will calibrate by prompting the user to input height, weight, and goals, comparing video input from the user to a reference video showcasing exemplary form. This reference video will be displayed when selecting a given exercise so the user can learn from an ideal repetition. Real-time feedback based on the lifter’s performance in a given set will be provided, correcting form and adjusting weight based on difficulty of the repetition. To avoid overwhelming the user, the app will prioritize the most crucial feedback first, as a personal trainer would. My goal for this app is to provide assistance to the community on their fitness journeys, increasing the overall quality of their workout along with their longevity.
Project Blog
Week 0: Introduction
Hello, welcome to my first senior project blog! My vision for this project is to develop an app to perform as a personal trainer, aiding lifters to have agency in their fitness journey.
In a nation where 39.6% of adults grapple with obesity, the imperative for fitness has never been more pressing in the United States. Despite the accessibility and affordability of gyms, these alarming statistics suggest that barriers beyond mere availability hinder citizens from engaging in regular exercise. One key issue is the lack of knowledge, exacerbated by the often expensive price of personal training, averaging around $65 per hour. In response, I seek to develop an open-source app employing pose detection technology, designed to serve as a virtual personal trainer, offering insightful feedback on users' workout routines.
Harnessing the capabilities of the open-source convolutional neural network framework, MediaPipe, my app seeks to enhance accuracy through tailored training with diverse exercise datasets, accommodating various demographic needs. The app's calibration process involves users inputting their height, weight, and fitness goals, with subsequent comparison of their workout videos to reference videos showcasing impeccable form. Each exercise selection prompts the display of a reference video, allowing users to learn from an exemplary repetition. Real-time feedback during a workout set will guide users, correcting form and adjusting weights based on the perceived difficulty of each repetition. Prioritizing key feedback akin to a personal trainer, the app aspires to assist and enhance the fitness journeys of individuals within the community, ultimately elevating the quality and sustainability of their workouts.
After considering various options, I have chosen to use the MediaPipe framework developed by Google to extract the skeletal model from the video. I will now try to find data to train the framework into a model and successfully set up my virtual environment to run the code on my PC.
Thank you for reading! Stay tuned for next week’s updates where I will be discussing the dataset I compiled.
Week 1: Setting Up MediaPipe on My Mac
Welcome back to my senior project blog! This week, I'm excited to dive into how I've set up the MediaPipe framework on macOS for my app. Developed by Google, MediaPipe stands out for several reasons that align with the goals of my project. It is optimized for real-time performance, which is compatible with my app’s goal as it must give real-time feedback on repetitions as a personal trainer would. Moreover, this framework’s pre-built models coupled with its flexible architecture allowing customization make MediaPipe an ideal choice as this trait is beneficial to refining the model to enhance accuracy for specific exercises. Lastly, MediaPipe has a growing community of developers keeping it up-to-date, ensuring its accuracy.
To effectively run MediaPipe, I first had to set up the environment. Homebrew was key for this as it enables efficient dependency management, simplifying installation, updates, and removal of my project’s dependencies.
Step 1: Homebrew and Xcode
The first step was installing Homebrew, the package manager that would make it considerably easier to manage software installations on macOS. With a simple command in the Terminal:
(/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)")
Homebrew was ready for use.
I proceeded to download Xcode, Apple’s IDE with high compatibility for macOS, with the command:
xcode-select --install
Step 2: Installing Dependencies
With Homebrew installed, it was time to address the dependencies critical for running MediaPipe: TensorFlow and OpenCV. TensorFlow is a cornerstone for many machine learning applications, including the pose detection technology I plan to utilize, while OpenCV will enable me to process the video inputs.
For TensorFlow: brew install tensorflow
For OpenCV: brew install opencv
Step 3: Running with Bazel
The final step in the setup process involved using Bazel, a build tool required by MediaPipe for compiling its applications. Bazel ensures that dependencies are correctly managed and that builds are reproducible, a crucial aspect of developing a reliable app. Install Bazel:
brew install bazel
Ensure Python >= 3 is installed, if not you can install with:
brew install python
sudo ln -s -f /usr/local/bin/python3.7 /usr/local/bin/python
Install the python ‘six’ library with:
pip3 install –user six
Test Mediapipe by running this command with bazel:
bazel run --define MEDIAPIPE_DISABLE_GPU=1 \
mediapipe/examples/desktop/hello_world:hello_world
This should print out:
Hello World!
Hello World!
Hello World!
Hello World!
Hello World!
Hello World!
Hello World!
Hello World!
Hello World!
Hello World!
I will link some MediaPipe resources below:
Official repository:
https://github.com/google/mediapipe
Troubleshooting:
https://developers.google.com/mediapipe/framework/getting_started/troubleshooting.md
Reflections and Next Steps
Setting up the environment can take time, especially as troubleshooting can be unique on a case-by-case basis. If you run into errors when executing the test, ensure your directory is set up correctly by viewing the contents with ‘ls’ or going into File Explorer/Finder. Try and check the troubleshooting site linked above as well to see if you can find a solution there. With MediaPipe now running on my Mac, the next stage is to dive into experimenting and tailoring the model to fit the needs of our virtual personal trainer app. I will need to find a large quantity of data to train for each exercise, so I plan to start to get the squat working first, as this is a fundamental exercise that requires both full-body and depth recognition from my model.
Thank you for reading! Hope to see you next week for my week 2 blog.
Week 2: A Pivot in Methodology
Hello everyone, welcome back to my senior project blog! This week, I want to share another possible avenue for the logic behind the personal trainer app. After thinking it over. I’ve decided to transition from the initial plan of training another AI model to leveraging hard-coded mathematical algorithms for our app's functionality. Let's dive into why this shift not only makes sense for this project but may also be a more suitable approach given our specific objectives.
Making the Switch
As I ventured deeper into the development process, it became increasingly clear that the requirements of our project could be efficiently met without the complexity of training a new AI model. At first, AI seemed promising and necessary due to its dynamic learning enabling it to adapt to different situations and body types. However, after deeper consideration, I have realized that this task may be accomplished without another model.
Here are some benefits from this approach:
First, mathematical algorithms can offer precise and consistent feedback without the extensive dataset and training phase required by AI models. This saves time and ensures that our app can deliver accurate performance analysis once a sufficient system is adopted. Second, developing and maintaining AI models is resource-intensive, both in terms of computational power and time. This new approach will minimize the time to analyze the form, enabling a clearer focus on the core functionality of providing real-time feedback. Lastly, hard coding with math simplifies the development process. By adopting a hard-coded approach, I can streamline the development process, focusing efforts on refining the user experience in the app development stage and expanding the list of exercises that can be graded.
Implementing Mathematical Models
My approach will still utilize the skeletal tracking capabilities of MediaPipe, as established in previous weeks. However, I will now be attempting to apply mathematical formulas to analyze the user's movements. This process will entail:
Normalizing the Poses: To ensure consistency in the analysis, we first normalize the poses captured by the app. This involves scaling and translating the skeletal model so that it can be compared with the reference pose, regardless of the body type, camera angle, or distance.
Calculating Differences Between Vertices: We then compute the differences between the 3D locations of the vertices corresponding to the user's current normalized pose and the ideal pose. This will allow us to identify deviations from the optimal form for each exercise.
Computing the Sum of Squared Differences: Calculating the sum of squared differences between the user's pose and the ideal pose quantifies the accuracy of the user's form, allowing us to provide targeted feedback to correct specific areas of their exercise technique. These differences will most likely need to be weighted based on the exercise being performed. By defining optimal ranges and thresholds for these parameters, we can effectively guide users towards perfecting their form.
As I move forward with this new direction, my next key steps are finding a way to normalize the poses to ensure inclusivity for various body types, as well as developing formulas to provide an accurate reflection of the biomechanics of various exercises. Next week, I hope to delve deeper into this new approach to further discern if this will be a reliable method for my project.
Stay tuned for more updates, and thanks for following along on this journey!
Week 3: Data Processing and Pose Normalization with MediaPipe
Welcome back to my senior project blog! This week, I’ll be sharing in-depth my plan for normalizing poses within our virtual trainer app using MediaPipe, a crucial step enabling comparison between skeletal models of different individuals. This is necessary to provide accurate fitness advice to individuals of any body type. Let's dive into how we're harnessing MediaPipe to achieve this.
Role of Normalization
At the core of our app's functionality is the ability to provide feedback on the user’s form, which will be made possible by comparing a user's exercise form against a reference model. This comparison will allow us to identify areas of improvement through discrepancies in form. However, for the app to offer tailored advice that aligns with each user's unique physiological makeup, we must find a way to “normalize” the input pose, translating and scaling each segment by a certain weight to enable comparison with a selected reference. To begin normalizing the data, it must be processed first. I will first be developing a working program on my macOS in Python before diving into app development. Our input data will be packets consisting of an image and a string. Below is a walkthrough of a basic usage of MediaPipe for processing data. This Python program is written to process a single packet and must be extended to continuous video input.
Processing Data
Begin by importing MediaPipe:
import mediapipe as mp
Next, define a calculator graph configuration:
config_text = """
input_stream: 'in_stream'
output_stream: 'out_stream'
node {
calculator: 'PassThroughCalculator'
input_stream: 'in_stream'
output_stream: 'out_stream'
}
"""
Next, initialize a CalculatorGraph and set up an listener on the graph's output stream:
graph = mp.CalculatorGraph(graph_config=config_text)
output_packets = []
graph.observe_output_stream(
'out_stream',
lambda stream_name, packet:
output_packets.append(mp.packet_getter.get_str(packet)))
_name, packet:
output_packets.append(mp.packet_getter.get_str(packet)))
Next, add the packets to the input stream:
graph.start_run()
graph.add_packet_to_input_stream(
'in_stream', mp.packet_creator.create_string('abc').at(0))
rgb_img = cv2.cvtColor(cv2.imread('/path/to/your/image.png'), cv2.COLOR_BGR2RGB)
graph.add_packet_to_input_stream(
'in_stream',
mp.packet_creator.create_image_frame(image_format=mp.ImageFormat.SRGB,
data=rgb_img).at(1))
Lastly, close your graph:
graph.close()
Looking Ahead
To extend this to our pose-detection program, we must enable continuous video processing to take the poses within the duration of the set as input. After getting a skeletal model of each frame, the pose must be translated to match our reference pose which should showcase exemplary form. To accurately compare these two, weights will be assigned to various parts of the body to scale our input to our reference and enable comparison. This will be our normalization process.
Stay tuned for next week's update, where I’ll share my progress on processing data, normalization, and selecting references. Thank you for joining me on this exciting journey towards creating a transformative fitness tool!
Reference:
https://developers.google.com/mediapipe/framework/getting_started/python_framework
Week 4: Solidifying MediaPipe Framework Concepts
Welcome back to my weekly blog where I track my journey towards creating a virtual personal trainer app! This week, I took time to gain a deeper understanding of the MediaPipe Framework, laying a robust foundation for normalizing pose landmarks from video frames. Let’s break down the core concepts of MediaPipe with some functions and code examples.
Packets
Packets are the fundamental units of data within MediaPipe, carrying a pointer to an immutable payload tagged with a timestamp. Here's a simple example of creating a packet with a string payload:
import mediapipe as mp
packet = mp.packet_creator.create_string('Hello, MediaPipe!')
Graphs
Graphs define the pathways data takes through the app, specifying how packets are processed and passed between nodes. Here’s a basic graph configuration in protobuf format:
node {
calculator: "PassThroughCalculator"
input_stream: "input_video"
output_stream: "output_video"
}
This configuration sets up a simple graph where video packets pass through unchanged from input to output.
Calculators
Calculators are the nodes of a graph where data is processed and transformed. Each node can have multiple input and output streams.
Streams and Side Packets
Streams carry sequences of packets between nodes, while side packets are used for data that remains constant. Below is an example of how you can add a packet to a stream in Python:
graph = mp.CalculatorGraph(graph_config=config_text)
graph.start_run()
graph.add_packet_to_input_stream(
'input_video',
mp.packet_creator.create_image_frame(
image_format=mp.ImageFormat.SRGB,
data=rgb_img
).at(mp.Timestamp(1))
)
Preparing for Program Development
Our next step is developing a detailed plan for grading the poses. This process should enable us to accurately compare various user poses against ideal forms and give feedback based on the performance. Below is an outline for the program:
Initialize MediaPipe Pose Detection
Load the MediaPipe Pose detection model
Configure the pose detection pipeline with necessary settings (ex static image mode, model complexity)
Capture Video Frame
For each frame in the video input, convert the frame into the format required by MediaPipe (ex RGB)
Detect Poses in Frame
Pass the converted frame to the MediaPipe Pose model
Retrieve pose landmarks from the output
Define Reference Model
Load or define a reference skeletal model that represents the ideal pose for the exercise
The reference model should include normalized positions of key landmarks (ex shoulders, knees, wrists, hips).
Normalize Captured Skeletal Data to
For each detected pose in the frame:
Calculate the scale factor by comparing the distance between specific landmarks (ex shoulder width, height) in the captured pose to the reference model, then scale the model
Align the base landmark of the captured pose with that of the reference model to adjust the position, translating and rotating as necessary
Calculate Deviation
Compare the normalized captured pose to the reference model.
Calculate the deviation for each landmark by measuring the distance between the corresponding landmarks in the normalized pose and the reference model.
Generate Feedback
Based on the calculated deviations, generate specific feedback for the user, placing weights on deviations dependent on the location’s impact on safety and hypertrophy
Repeat for Each Frame
Repeat steps 2-7 for each frame in the video input to continuously analyze and provide feedback on the user's pose throughout their exercise
Conclude Analysis
Once all frames have been processed, compile and present the overall feedback to the user, highlighting areas of improvement and consistency
Next Up:
With a clearer understanding of MediaPipe and preliminary plans for our algorithm, we are now ready to start coding! Thank you for being a part of this journey!
Week 5: Diving Into Pose Detection with MediaPipe
Hello again, and welcome back to my senior project blog! This week, I'm thrilled to share with you my pose detection program for the virtual personal trainer app. I now have a program capable of detecting pose landmarks from input images using the MediaPipe framework. Let's dive into the details of setting up the project environment and understanding the code.
Setting Up the Environment
Before diving into the code, let's talk about the setup. Ensuring compatibility and a smooth running environment is crucial for any project.
Ensure 3.9 < Python < 3.12
I started by creating a new directory with a virtual environment named mediapipe-env in my terminal:
mkdir mediapipe
cd mediapipe
python3 -m venv mediapipe-env
.\mediapipe-env\Scripts\activate
After activating the environment, move your desired input image into the directory. The next step will be to install MediaPipe and download a specific pose landmarker model:
pip install mediapipe
wget -o pose_landmarker.task https://storage.googleapis.com/mediapipe-models/pose_landmarker/pose_landmarker_heavy/float16/1/pose_landmarker_heavy.task
Pose detection program
MediaPipe and OpenCV Initialization: The program begins with importing necessary libraries from MediaPipe and OpenCV, setting the stage for pose detection and image manipulation:
import cv2
from mediapipe import solutions
from mediapipe.framework.formats import landmark_pb2
import numpy as np
import mediapipe as mp
from mediapipe.tasks import python
from mediapipe.tasks.python import vision
Pose Detection Setup: Using MediaPipe's PoseLandmarker and the downloaded pose_landmarker.task model, the program is set to detect human poses:
def draw_landmarks_on_image(rgb_image, detection_result):
pose_landmarks_list = detection_result.pose_landmarks
annotated_image = np.copy(rgb_image)
# Looping through the detected poses to visualize.
for idx in range(len(pose_landmarks_list)):
pose_landmarks = pose_landmarks_list[idx]
# Drawing the pose landmarks.
pose_landmarks_proto = landmark_pb2.NormalizedLandmarkList()
pose_landmarks_proto.landmark.extend([
landmark_pb2.NormalizedLandmark(x=landmark.x, y=landmark.y, z=landmark.z) for landmark in pose_landmarks
])
solutions.drawing_utils.draw_landmarks(
annotated_image,
pose_landmarks_proto,
solutions.pose.POSE_CONNECTIONS,
solutions.drawing_styles.get_default_pose_landmarks_style())
return annotated_image
# Creating an PoseLandmarker object
base_options = python.BaseOptions(model_asset_path='pose_landmarker.task')
options = vision.PoseLandmarkerOptions(
base_options=base_options,
output_segmentation_masks=True)
detector = vision.PoseLandmarker.create_from_options(options)
Loading and Processing the Input Image: An input image is loaded and processed to detect human poses:
# Loading the input image
image = mp.Image.create_from_file("imagen.jpg")
# Detecting pose landmarks from the input image.
detection_result = detector.detect(image)
Visualization: The detected landmarks are then visualized on the image using OpenCV, displaying the pose landmarks:
# Visualizing pose landmarks
annotated_image = draw_landmarks_on_image(image.numpy_view(), detection_result)
cv2.imshow("image", cv2.cvtColor(annotated_image, cv2.COLOR_RGB2BGR))
cv2.waitKey(0)
# closing all open windows
cv2.destroyAllWindows()
Looking Ahead
As we move forward, the next step is comparing our input image to a reference. Our aim is to not just detect poses but to analyze them in the context of exercise form, providing users with personalized recommendations for improvement.
Stay tuned for more updates as I continue developing this program. Your support is greatly appreciated.
See you in the next post!
Week 6: Elevating Our Pose Detection Capabilities with Real-Time Processing
Hello and welcome back to my senior project blog! This week, we've transitioned our pose detection capabilities from static image processing to dynamic, real-time video analysis. Let’s explore the key changes in our approach and how they impact the project.
Transitioning to Real-Time Video Processing
Previously, our application focused on analyzing single images to detect human poses. While effective for static tests, this approach did not cater to the dynamic nature of physical exercises, where movement and continuity play crucial roles.
I have now adapted the program to handle a continuous video stream, accessing the device's camera to capture real-time movement and change in posture as it happens. By processing each frame in real time, the app can provide instantaneous feedback on the user’s exercise form, making it more interactive and beneficial for on-the-spot corrections.
Simplification and Direct Processing
In our previous setup, the detection process involved converting pose landmarks into a protobuf format, and then rendering them onto the image. This method, while precise, introduced unnecessary complexity into our application. Now, I am employing the MediaPipe Solutions API to directly use the output from the pose detection to visualize landmarks on the video frames. This direct approach reduces the complexity of our code and enhances the system’s efficiency. By eliminating intermediary steps, we ensure that our application is faster and more responsive, crucial factors for real-time performance.
Real-Time Interaction Enhancements
Adding real-time video capture and display functionality has transformed our application from a static analyzer into a dynamic training tool. This development allows the app to interact with users, providing a more engaging and effective training experience. With the capability to process video frames in real time, our app is one step closer to being completed.
Looking Ahead
Moving forward, now that we have a working pose-detection program, I will begin to work on the grading system for the squat. Stay tuned for next week's update, where I will delve into optimizing real-time processing capabilities and developing a form analysis algorithm. Thank you for following along on this exciting journey!
Week 7: Pose Analysis with Angular Metrics
Welcome back to my blog tracking this journey in developing the virtual personal trainer app! This week, we made a pivotal shift in our approach to analyzing and normalizing human poses from scaling landmarks to a reference model, to utilizing angular measurements between joints. This change is designed to enhance the efficiency and accuracy of our app, making it more beneficial for users seeking real-time exercise feedback. Let's dive into the details of this transformation and its impact on our project.
Moving Away From Landmark Scaling
Previously, our method for normalizing poses involved scaling each landmark to match a predefined reference model. While this is effective in ensuring a uniform comparison across different body sizes and shapes, this approach was computationally expensive andsometimes resulted in loss of posture details. Scaling each landmark by a different value after translation and proceeding to compare with a reference required unnecessary steps, especially in real-time applications. Additionally, leg movement and position during the squat is much more relevant than that of the upper body, and there are also various different styles of squatting. Therefore, in an analysis only considering position, some “correct” squats might be marked as incorrect.
Embracing Angular Analysis
To address these challenges, I shifted the approach to analyzing the angles formed between various joints. This method relies on the geometric relationships that are less resource-intensive to compute while also providing a better analysis. To summarize the benefits of this approach:
Reduced Computational Load: Calculating angles between joints is significantly less demanding on processing power, facilitating smoother and faster analysis, especially in real-time scenarios.
Preservation of Posture Details: Unlike scaling, which can result in data loss, angular analysis retains the unique aspects of each user's movements, enhancing the personalized feedback our app provides. This will make it easier to cater to users with various body types.
Adaptability Across Different Exercises: Angular measurements maintain versatility in analyzing a wide range of exercises, keeping it simple to adjust the app for various fitness routines
Increased accuracy: This implementation of angular analysis involves calculating the angles between key joints such as elbows, shoulders, knees, and hips during exercises. These angles will be compared against optimal ranges derived from kinesiological standards to assess the correctness of each pose, giving more reliable and accurate feedback
Thank you for your continued support and interest in this project!
Week 8: Enhancing Training Precision with Rep Counting and Angular Thresholds
Hello everyone, and welcome back to another update on my senior project. This week, I focused on counting repetitions accurately. To achieve this, I implemented a method using angular thresholds, ensuring that every rep is not only counted but also meets a defined minimum standard of form. Let's dive into why rep counting is essential and how we're approaching it through angle measurements.
The Importance of Accurate Rep Counting
In any fitness regimen, the number of repetitions of an exercise is a key metric that affects both the intensity and effectiveness of a workout. Accurately counting reps will not only enable the program to grade form but also helps in several other ways:
Progress Tracking: It allows users to track their progress over time, ensuring they are pushing their limits and improving.
Intensity Management: Accurate rep counts help users maintain the right intensity level for their fitness goals, whether they are aiming for strength, endurance, or hypertrophy.
Feedback and Motivation: Knowing how many reps you have completed provides a basis for immediate feedback.
Implementing Angular Thresholds for Squat Reps
Given the importance of precise form in exercises like squats, I implemented an angular threshold method to define what constitutes a repetition in the program. Here’s how we implemented this system:
We start by determining the key angles in a squat that indicate proper form. The angle at the knee is crucial—when a user squats down, the knee angle should decrease, typically going below 90 degrees to indicate sufficient depth.
I set a maximum angle threshold (75 degrees at the knee), which must be reached for a rep to be considered complete. This ensures that all counted reps meet a minimum standard of depth. For example, if the depth reaches only 70 degrees, the rep will not be counted. However, if it reaches around 75-80, the program will recommend a deeper squat but count the rep.
As the user performs squats, the app will continuously monitor the angle of the knee using real-time pose detection technology. Each time the knee angle drops below around 75 degrees and then rises above it again, a rep should be counted. This simple yet effective method guarantees that the user maintains the necessary range of motion for effective training. By using angular thresholds for rep counting, we can enhance the accuracy of exercise tracking while increasing the overall effectiveness of workouts.
Looking Ahead
After I finish implementing and refining this feature, I'll move on to check for injuries and common mistakes using angle thresholds.
Thank you for your continued interest in this project! Stay tuned for next week's update.