Blog Post #10: Down to the Wire

Hello everyone, welcome back to my blog! This was the final week, and I admittedly had more work to do from last week to meet the deadline. Also admittedly, I experienced some tricky problems in the last stretch that I worked to solve.

Problems

The main problem started when I updated my Expo Go app (the app I was using to test my project). This turned out to be a mistake since my project was no longer compatible with the newest Expo version. Furthermore, you can’t install old versions of apps on iOS, so I couldn’t go back. I spent some time testing solutions. Finally, I discovered that you can install old versions of apps on Android… so I had to gratefully borrow my brother’s Android phone to continue. I adjusted my code slightly to work with Android – this was relatively simple since React Native is compatible with both iOS and Android. Finally, I started work on finishing my product.

As a recap, I was previously using two models to make predictions: one called hand-pose-detection from Google Mediapipe, and another that was personally trained on sign data. The hand-pose-detection model worked far more reliably in detecting hand keypoints than my personal model did in predicting signs. I wondered if I could process the keypoints from the hand-pose-detection model and make better predictions.

End Product

I found a package called “fingerpose” to do exactly that. It determines gestures by processing keypoints predicted by a slightly different model called “handpose”. To make predictions, it analyzes the curl and direction of each finger. It also conveniently allows users to make their own gestures by specifying finger weights, curls, and directions, although this has to be self-coded and optimized. (This could probably be a machine learning task in itself.)

So, after coding and testing 10 signs in Malaysian Sign Language/BIM, I finally got it to work. To note, it’s pretty slow since it’s computationally intensive. Its prediction confidences are also kind of arbitrary since I had to assign weights to each sign myself. Still, it predicts signs relatively accurately and translates predictions to both Malay and English, accomplishing what I planned for this project.

I’d like to thank my internal and external advisors Mrs. Shahin and Dr. Fata for being willing to help me solve my numerous problems with my app. I’ve learned a lot from them not only about how to use “AI” and do “Front-end Development”, but also about how to stay committed and persevere through working on my projects. I’d also like to thank Mrs. Nagpal, Ms. Apra, Ms. Oben, and the rest of my teachers, staff, and peers at Basis for supporting me both within the past three months and throughout the past four years.

Until next time(?),
Elysse Ahmad Yusri