Malaysian Sign Language Translating App using Machine Learning
Advisor: Ms. Nasira Shahin
Internship Location: Wolfspeed, CA
Onsite Mentor: Fata Anvar
Although countries like the United States are making increasing amounts of developments in aiding those with disabilities through making amenities more accessible and facilitating awareness, countries like Malaysia haven’t made as much progress. Around 42,652 people across Malaysia are registered to have some type of hearing problems, although this number is likely to be higher in reality. In addition to this, Malaysia is distinct in that it is ethnically diverse: the population consists of Malay (69.8%), Chinese (22.4%), Indian (6.8%) and others (1.0%). As such, there are many different varieties of Malaysian Sign Language which differ by state and by culture. A combination of Malaysia’s language diversity and less expansive access to technology in some areas has led to disconnect with and a lack of awareness of the deaf community. Far fewer people are aware of Bahasa Insyarat Malaysia (also called BIM, the official sign language of Malaysia) and how to use it, and responsive resources made to assist communication through BIM like courses are either scarce or expensive.
My project seeks to create an easily accessible mobile application that translates words in BIM to Malay and English via camera. I intend my application to not only be used by Malaysian people who are hard of hearing for translating words in daily communication, but also to be used by others interested in learning and practicing BIM. Through this project I seek to accomplish two objectives: 1) to develop skills in using machine learning and in app-developing and 2) to enter, understand, and possibly contribute to the discussion regarding disability and accessibility to technology within different societal contexts. In developing this app, I hope to gain and spread knowledge about deaf communities abroad.
Project Blog
Week 0: Introduction
Hello everyone! Welcome to my senior project. I am Elysse Ahmad Yusri, and my project will be oriented towards making an application to translate Malay Sign Language (Bahasa Isyarat Malaysia, or BIM) via camera to Malay and English. While machine-learning oriented programs exist to help translate other languages, such as American Sign Language, to English, there is less technology in general to aid those learning or communicate through other forms of sign language. Not only do I seek to learn and apply my knowledge in machine learning through this project, but I also seek to learn more about communities centered around disability in my home country.
The first one or two weeks of this project will be oriented around clarifying the scope of my project. A primary concern of mine is to find more suitable datasets to train and test my model with since there are less resources available on BIM. I also plan to reference the official BIM Sign Bank, which contains resources and signals that have been supported by the Malaysian government and various other agencies. I hope to find more official resources like these in this time.
Lastly, I wouldn't be able to do this project without the help of data scientist Dr. Fata Anwar as my external advisor and Computer Science teacher Mrs. Shahin as my internal advisor. With their help, I look forward to learning, creating, and sharing my experiences as I do this project!
Post 2: Blog Post #1: Home Brunches, Time Crunches, Free Lunches
Hello everyone, welcome back to my blog! This week marked the start of my quest for machine learning greatness, although I concede that I embark with a bit less foresight than usual.
For the majority of my time, I’ve been confiding in the first few chapters of the textbook “Hands-On Machine Learning with Scikit-Learn and TensorFlow” to brush up on my skills. Not only is it relatively recent (meaning it doesn’t contain much outdated code), but it also contains many helpful practice exercises.
In addition to learning about coding this week, I also learned about scheduling. Since I’m working from home I’ve had to shift my schedule, so I settled on working in the morning. This way I could use devote my energy from brunch towards looking at code for three hours until lunchtime came.
Speaking of lunchtime. While I was reading about regularization techniques I saw one referred to as a “beautiful free lunch”. But if there’s no such thing as free lunch, how can that be possible? Well, it isn’t: as I’ve learned, each technique has its tradeoffs. The challenge in using these techniques is experimenting and finding the most ideal solutions that fit within your problem space. I’ve gotten a clearer picture of where I want to take my project through reading and practicing, but I aim to take the next few weeks to research tools and read the book so that I have a solid plan of action before I delve completely into the project.
Until next week,
Elysse Ahmad Yusri
Post 3: Blog Post #2: Balancing Learning Rates
Hello everyone, welcome back to my blog! For this week, I’ve been doing a lot more research and reading on machine learning. While I’ve essentially been doing the same tasks as last week, I feel like I’ve learned a lot more about the different model algorithms and selectors available for me to use for my project.
Among the many techniques I’ve learned about, an interesting one that I’ve learned about is called boosting. When training multiple machine learning models in sequence where each improves upon their predecessors, boosting makes these models improve upon previous mistakes specifically. The thing is, how fast these models are made depends on the learning rate: higher rates mean less time is taken, yet the model may make less specific predictions. While I’ve been doing a lot of trial and error through practice problems, I feel it’s about time I outline my specific project with more detail - I feel that I’ll be more efficient in studying the techniques I need given the knowledge I’ve accumulated over these past few weeks. So, I plan to devote next week to doing exactly that.
Until next week,
Elysse Ahmad Yusri
Post 4: Blog Post #3: Getting my Boots on the Ground
Hello everyone, welcome back to my blog! I am nearly ready to start coding, as I’ve been reading about using Tensorflow and Keras. Contrasting with sk.learn’s SVMs, random trees, and k-means clustering (which are different types of machine learning models), Tensorflow specializes in neural networks, which will be especially useful for analyzing images. I’ll be able to preprocess and augment these images through Tensorflow’s data preprocessing resources, like the tf.data API that helps load data efficiently and the data augmentation features built into certain Keras layers. To top it off, I’ll be using it with React Native, the framework I’ll use to make the app. All of these aspects make Tensorflow a helpful framework to use for my project.
While I have spent quite a long time in the researching phase, I feel like I currently have the base to start coding the machine-learning aspect of my project next week. I’ll provide more updates about my project in my next blog, both with regards to my independent work and to the insights I’ll gain through discussions with my advisors.
Until next week,
Elysse Ahmad Yusri
Post 5: Blog Post #4: An Empirical (and probably misapplied) Exploration of Occam’s Razor
Hello everyone, welcome back to my blog! This week I have embarked on the confounding task of model-training. In the first few days I tried following multiple tutorials using different models and techniques, but I encountered many problems along the way. For example, I faced difficulty in installing a package called tflite-model-maker for one tutorial, but it turns out that many others online had the same issue and had resorted to various work-arounds.
I managed to settle on one solution that seems to work, though, and it happens to be the simplest method out of all the tutorials. Most other tutorials I looked into used pre-trained models, or complex models that have already been trained by other people, which one alters to work on more specific tasks. (For example, one could use a model pre-trained on general object detection to classify specific objects like tables and chairs.) I hadn’t been able to get these to work, though, so I just settled on creating my own simple models using tutorials from the Tensorflow website. These models seem to perform reasonably well on one dataset I’m using so far. But is the best solution really the simplest one? I don’t really have the resources to figure that out.
So far, the images in the data don’t have very much variation in them. To alleviate this problem, I plan to use more image augmentation to increase this variability next week. I also want to try incorporating another dataset with more signs into the model.
Until next week,
Elysse Ahmad Yusri
Post 6: Newly-Developed Libraries: A Blessing and a Curse?
Hello everyone, welcome back to my blog! This week I continued to work on training my model. Most importantly, I combined another dataset with my original one because it had more signs - fortunately they were similar in format, but I still had to make some changes. There were less images per sign in the first dataset than the second, so I had to perform data augmentation to make sure that each sign had about the same amount of images associated with them. This is so that the model isn’t biased toward predicting signs that were more populous during training. I also had to retrain my model, which took more time now that I had more data.
Although I finished training the models in the end, I encountered a few problems along the way. Troubleshooting took more time than before because the model took longer to train on the data. Additionally, I had trouble converting my model to a format called TFLite, which is particularly optimized for performance on mobile devices. Since TFLite is still being developed there aren’t any immediately obvious solutions to my problem, so I will have to do a bit more digging before I can continue. I’m hoping that I can resolve this and have enough time to start developing my app next (next) week.
Until next time,
Elysse Ahmad Yusri
Post 7: Blog Post #6: Small Breaks During School Breaks
Hello everyone, welcome back to my blog! Since last post, I've had two breaks: one was a week-long school break, and the other was a day-long break for Eid. Conversely, while I had fun with friends and family on Eid, I made good progress on my project over this week and school break.
For one, I managed to fix my problem with using my GPU to train my models (which probably should have happened earlier). The time taken to train my models has decreased approximately tenfold since my computer doesn’t have to solely rely on my CPU’s computing power. This change has also allowed me to make the proper modifications to my models to convert them to TFLite, a machine-learning format that is optimized for performance on mobile devices (more in this link: https://www.tensorflow.org/lite/guide). The only problem I have now is that Google Colab, the cloud-based platform I use to train my models, enforces limits on how much of its computing resources I use - I’m able to use it for about one or two hours before it restricts access to my GPU. Fortunately, I’ve been able to train and save a few preliminary models in that time.
I’m planning to test these models with a camera via a very simple user interface built using React Native. To do this I’ve found an app called Expo Go, which (in my few days of testing it) makes testing my app on my phone pretty simple. All I have to do is start a development server on my computer to host my app by running a few commands, which returns a QR code for me to scan with my phone. When I have the Expo Go mobile app installed on my phone, it takes this QR code and renders my app on the fly. It even reloads every time I make changes to the app. As a disclaimer I’ve just started to use Expo Go, so there still may be some unforeseen problems I encounter along the way.
As for the future, I’m planning for this app to be more of a proof-of-concept than anything - there are more aspects to app-making that I haven’t considered (i.e. security concerns). Next week, I’m planning to implement camera functionality on my app and possibly to test it with my trained models.
Until next time,
Elysse Ahmad Yusri
Post 8: Blog Post #7: Using a Mobile App to make a Mobile App
Hello everyone, welcome back to my blog! This week, I started work on coding my app in React Native (a software framework used for coding user interfaces). In short I got camera functionality to work, but I had to go through many steps to get there.
First, I created a default React Native app via Expo Go (a mobile app that allows me to test my code on my phone) using a few commands stated here: https://reactnative.dev/docs/environment-setup. It provides a file called App.js, which is the main file used to render content. I used this to make a “wireframe” for my app and lay out my user interface.
After that, I installed a package called expo-camera to access my phone’s camera. I referenced the Expo Documentation website to implement basic functionality. It was actually pretty simple to get it to work and fit within my layout.
At this point, I wanted to try to make predictions using a machine learning model with my camera. Since I didn’t really know how to do that, I referenced two separate tutorials from the Tensorflow website. I was able to clone the github repositories with the code for the tutorials to test them on my own device, but I quickly found out that they were both outdated. So, I went through the simpler tutorial and manually updated the outdated syntax.
In the end, I was able to get the tutorial’s pre-trained model to work on my phone. While it is very power-intensive, it provides relatively stable and accurate predictions - a proof-of-concept that confirms that prediction via camera is possible. Next week I plan to look into the specifics of the tfjs-react-native package, whose components were used by the tutorial, to try to use my own model.
Until next time,
Elysse Ahmad Yusri
Post 9: Blog Post #8: Twaddle with Models
Hello everyone, welcome back to my blog! This week I started on integrating my model with my app for prediction-making, but this involved more trial-and-error than I was expecting.
For one, the package that I previously used to run predictions was called tfjs-react-native. It only works (as far as I know) on models in the tensorflowjs format, so I had to go back and convert my model. But after installing the necessary packages to do so, the accuracy of the model dropped immensely even though I didn’t make any changes to the model. I saw this while testing predictions on my app. While this might be due to mistakes with image processing, I still plan to look into this issue.
Seeing this, I wanted to try other methods of detection. For one, I revisited a pre-made object detection API - instead of trying to classify based on the whole image, the API draws bounding boxes and classifies from within the box. While I felt this would be useful for UI purposes, I unfortunately wasn’t able to get this to work.
Then, I tried a model called hand-pose-detection from the tensorflow-models package. It was created as part of Google’s MediaPipe Solutions, and it functions like the object-detection package that I used previously. So, I was successfully able to predict hand landmarks with it. I also plan to look into performing transfer learning with the model since it performs pretty well on my phone.
One last thing I did this week was attending a senior project workshop. Along with laying out guidelines for our presentation, the workshop also clarified how research posters and demos would be held. While I’m not planning to publish the app, I think I’ll be able to demo using the Expo Go app.
Until next time,
Elysse Ahmad Yusri
Post 10: Blog Post #9: The Consequences of Opening Pandora’s Box
Hello everyone, welcome to my blog! This week I’ve been trying to wrap up my project and start work on my presentation. I’m going to do more work on my last week, though, to make whatever improvements I can.
Firstly, I’ve been able to successfully convert my model to the tensorflow.js format I need for use in my app by using the tfjs-converter package. I had already gotten Google Mediapipe’s hand-pose-detection package to work and display hand keypoints in my app. Consequently, I wanted to try using the hand-pose-detection model and my own model together: the former would provide visual indication that the app was detecting your hand, and the other would provide the predictions.
But when I imported the necessary files and wrote the corresponding code, I found that my model wasn’t very accurate. I think this could be because of the tfjs-converter package making inaccurate conversions (because I had to circumvent several dependency conflicts to make it work). It might also be because of the way I processed my input in the app, but I don’t think this is likely. My main concern was that it was because I trained an image-classification model on a limited dataset, which wouldn’t generalize well to data in more varied environments.
So what’s next? I’ve trained another gesture-recognizer model from Google Mediapipe, and it seems to perform well on their demo website. The only problem is that it doesn’t work with React Native (which I found out through experience). So, I’m probably going to make a web app and implement it there. Since I’m getting hand keypoints through the hand-pose-detection package, I could also write code to infer signs based on those keypoints. I’m going to do as much as I can in this time.
Until next time,
Elysse Ahmad Yusri
Post 11: Blog Post #10: Down to the Wire
Hello everyone, welcome back to my blog! This was the final week, and I admittedly had more work to do from last week to meet the deadline. Also admittedly, I experienced some tricky problems in the last stretch that I worked to solve.
Problems
The main problem started when I updated my Expo Go app (the app I was using to test my project). This turned out to be a mistake since my project was no longer compatible with the newest Expo version. Furthermore, you can’t install old versions of apps on iOS, so I couldn’t go back. I spent some time testing solutions. Finally, I discovered that you can install old versions of apps on Android… so I had to gratefully borrow my brother’s Android phone to continue. I adjusted my code slightly to work with Android - this was relatively simple since React Native is compatible with both iOS and Android. Finally, I started work on finishing my product.
As a recap, I was previously using two models to make predictions: one called hand-pose-detection from Google Mediapipe, and another that was personally trained on sign data. The hand-pose-detection model worked far more reliably in detecting hand keypoints than my personal model did in predicting signs. I wondered if I could process the keypoints from the hand-pose-detection model and make better predictions.
End Product
I found a package called “fingerpose” to do exactly that. It determines gestures by processing keypoints predicted by a slightly different model called “handpose”. To make predictions, it analyzes the curl and direction of each finger. It also conveniently allows users to make their own gestures by specifying finger weights, curls, and directions, although this has to be self-coded and optimized. (This could probably be a machine learning task in itself.)
So, after coding and testing 10 signs in Malaysian Sign Language/BIM, I finally got it to work. To note, it’s pretty slow since it’s computationally intensive. Its prediction confidences are also kind of arbitrary since I had to assign weights to each sign myself. Still, it predicts signs relatively accurately and translates predictions to both Malay and English, accomplishing what I planned for this project.
I'd like to thank my internal and external advisors Mrs. Shahin and Dr. Fata for being willing to help me solve my numerous problems with my app. I've learned a lot from them not only about how to use "AI" and do "Front-end Development", but also about how to stay committed and persevere through working on my projects. I'd also like to thank Mrs. Nagpal, Ms. Apra, Ms. Oben, and the rest of my teachers, staff, and peers at Basis for supporting me both within the past three months and throughout the past four years.
Until next time(?),
Elysse Ahmad Yusri