Blog Post #9: The Consequences of Opening Pandora’s Box

Hello everyone, welcome to my blog! This week I’ve been trying to wrap up my project and start work on my presentation. I’m going to do more work on my last week, though, to make whatever improvements I can.

Firstly, I’ve been able to successfully convert my model to the tensorflow.js format I need for use in my app by using the tfjs-converter package. I had already gotten Google Mediapipe’s hand-pose-detection package to work and display hand keypoints in my app. Consequently, I wanted to try using the hand-pose-detection model and my own model together: the former would provide visual indication that the app was detecting your hand, and the other would provide the predictions.

But when I imported the necessary files and wrote the corresponding code, I found that my model wasn’t very accurate. I think this could be because of the tfjs-converter package making inaccurate conversions (because I had to circumvent several dependency conflicts to make it work). It might also be because of the way I processed my input in the app, but I don’t think this is likely. My main concern was that it was because I trained an image-classification model on a limited dataset, which wouldn’t generalize well to data in more varied environments.

So what’s next? I’ve trained another gesture-recognizer model from Google Mediapipe, and it seems to perform well on their demo website. The only problem is that it doesn’t work with React Native (which I found out through experience). So, I’m probably going to make a web app and implement it there. Since I’m getting hand keypoints through the hand-pose-detection package, I could also write code to infer signs based on those keypoints. I’m going to do as much as I can in this time.

Until next time,
Elysse Ahmad Yusri