Burmese-English Machine Translation Accuracy
Advisor: Ms. Canevaro
Internship Location: Frontiir
Onsite Mentor: Phyo Kyaw, Director of Engineering at Frontiir
The vast majority of my research will be done completely digitally. Before beginning my actual project, I will conduct more internet research on more technical details of machine translation as well as the algorithms behind the evaluation metrics I discussed earlier, such as BLEU and RIBES. The project itself will involve translating English and Burmese sentences on different platforms and then evaluating their accuracy using existing metrics. Depending on time, I may also ask bilingual human judges to rate translations. My final product will be a research paper explaining the results.
Project Blog
Week 0: Introduction!
Hello readers!
I began my project as a response to a minor personal problem and a major global problem. In this blog post, I'll be introducing both of these problems and also the project.
The personal:
I'm a native speaker of Burmese and I lived in Myanmar for six years. I've noticed the inaccuracy of Burmese-English machine translations for about as long as I knew they existed—for instance, Facebook's automatic translations of my relatives' posts would mess up on features as simple as numbers. Although I've noticed significant improvements in recent years, I'm still quite wary of trusting these translations, and I thought this would be a good way to gauge the appropriate amount of wariness one should use when approaching them.
The global:
The vast majority of the world's languages are considered low-resource languages, meaning they lack the necessary data to adequately train natural language processing (NLP) models, including machine translation. Burmese is one of these low-resource languages, and the impact of the resulting inaccuracies on its speakers has been far more damaging than most would initially suspect.
Inaccurate Burmese translations on social media platforms have directly harmed marginalized communities before. Most famously, Facebook posts promoting the genocide of Rohingya Muslims were left unmoderated, as English translations of Burmese hate speech were completely inaccurate, and as a result, innocuous. One fairly egregious example was a post that stated, “Kill all the [offensive term] that you see in Myanmar; none of them should be left alive.” but was translated into the seemingly harmless (and incomprehensible), "I shouldn’t have a rainbow in Myanmar." Facebook also lacked any Burmese-speaking content monitors who could manually flag such posts, so their system for managing hate speech was based solely upon these highly flawed translation systems (Stecklow, 2018).
There is also a disproportionately low amount of online content originally written in Burmese, so monolingual Burmese speakers are also forced to depend on potentially unreliable translations to access certain information. Around 0.002% of websites support Burmese (W3Techs, 2024), while Burmese speakers make up approximately 0.4% of the world’s population (Worldmapper, 2024). Having accurate and accessible translation services is vital in bridging this part of the digital divide.
The project itself:
Over the next few months, I'll be evaluating the accuracy of major machine translation platforms (like Google Translate and Meta Translate) via preexisting metrics of translation accuracy, comparing them to each other as well as human translations, and figuring out where they tend to go wrong. Hopefully, I'll be able to produce some usable results on which platforms are the most (and least) reliable, how reliable they are exactly, and where they could potentially be improved in the future. I'll be sharing more detailed information about these metrics, platforms, and errors in the coming weeks.
See you next Friday!
Citations:
Stecklow, S. (2018). “Why Facebook is losing the war on hate speech in Myanmar.” Reuters, https://www.reuters.com/investigates/special-report/myanmar-facebook-hate/.
W3Techs. (2024). “Usage statistics of Burmese for websites.” W3Techs, https://w3techs.com/technologies/details/cl-my-.
Worldmapper. (2024). “Burmese Language.” Worldmapper, https://worldmapper.org/maps/burmese-language/.
Week 1: Translation Accuracy Metrics, Part 1
Hello readers!
A large part of my senior project is quantifying translation accuracy. But how do we quantify something seemingly subjective? My project will feature three different metrics of translation accuracy, and this blog post will introduce two of them.
Adequacy and Fluency
In the early stages of machine translation, accuracy was evaluated by humans on the bases of adequacy and fluency. Adequacy measures how similar a translated sentence is to the original sentence in meaning, while fluency measures how natural or correct the translated sentence sounds syntactically. A panel of native speakers would then rate translated sentences on both these metrics, typically on a scale of 1-5 (White & O'Connell, 1994).
Let's say we have an original sentence that means, "The cat ate the fish." "Cat eat fish" may receive a decent adequacy score, since it conveys a similar meaning, but a low fluency score, as it is not grammatically correct Engish. A sentence like, "The quick brown fox jumps over the lazy dog," which is completely grammatically correct but also completely unrelated in meaning, would receive a very high fluency score but a very low adequacy score.
Bilingual Evaluation Understudy (BLEU)
A more objective and automatic measure of translation accuracy is BLEU, which is currently the standard metric for evaluating machine translation accuracy. BLEU gets its name from how similarly it rates translations to bilingual human judges. It's an algorithm that compares the similarity of a machine-translated sentence to an existing human-translated reference sentence based on the number of times a given word or phrase appears in the translated sentence and the maximum number of times it can appear in a reference sentence. BLEU uses the maximum number of times a word can appear because most sentences can be translated using multiple different wordings without losing their fundamental meaning. "I'm thinking of the beauty of the forest" and "I am thinking about the forest's beauty" could both be valid translations of the same original sentence, but the number of times words like "of," "the," and "am" appear differs.
BLEU also factors in a brevity penalty for sentences that are significantly shorter than all possible references. Sentences that are too long are already penalized by the previous mechanism, as they would have to contain more words than the maximum possible count of each word in the references. BLEU scores range from 0-1, with 1 being completely identical to a reference (Papineni et al., 2002).
In my next post, I'll be talking about the third metric I'm using, RIBES, and my reasons for choosing each one. See you there!
Citations:
Papineni, K., Roukos, S., Ward, T., & Zhu, W. (2002). “BLEU: a Method for Automatic Evaluation of Machine Translation.” Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 311-318, https://aclanthology.org/P02-1040.pdf.
White, J. S., & O’Connell, T. A. (1994). “Evaluation in the ARPA Machine Translation Program: 1993 Methodology.” Human Language Technology, 135-140, https://aclanthology.org/www.mt-archive.info/90/HLT-1994-White.pdf.
Week 2: Translation Accuracy Metrics, Part 2
Hello readers!
This week, we'll be looking at the third and final metric of translation accuracy my project will include and going over why I chose each one.
Rank-based Intuitive Bilingual Score (RIBES)
RIBES was created by Isozaki et al. to resolve an issue with BLEU: since BLEU's calculations are solely based on word frequency, it sometimes gives high scores to mistranslated sentences when they have the correct words in the wrong order. For example, BLEU would give the sentence, "Barry called Adam," a perfect score when compared against the reference sentence, "Adam called Barry," even though it is an inadequate translation, since the words in both sentences are the same. Word order errors like these are uncommon between language pairs such as English and French, which have fairly similar word orders, but are some of the most common errors found in translations between more distant pairs like English and Japanese.
RIBES assigns ranks to each word in a sentence and calculates a rank correlation coefficient based on how many increasing pairs of words there are in the translated sentence. In the reference sentence, "Adam called Barry yesterday," "Adam" would receive a rank of 1, "called" a rank of 2, and so on. This makes the word order list [1, 2, 3, 4]. Ideally, all six pairs would show an increase in rank between the word mentioned first and the word mentioned second: in the reference, we have pairs (1,2), (1,3), (1,4), (2,3), (2,4), and (3,4). If the translated sentence was, "Barry called Adam yesterday," its word order list would be [3, 2, 1, 4]. There would then only be three increasing pairs: (3,4), (2,4), and (1,4). More weight is placed on longer sequences of words that maintain the same ranks. If words from the reference sentence are missing in the translated sentence, its RIBES score receives a precision penalty.
Reasons!
If RIBES is able to capture errors in word order that BLEU fails to, why use BLEU at all? Word order can often be changed without changing the meaning of the sentence—take the sentences, "Last week, Barry and Adam went to the store," and "Adam and Barry went to the store last week"—and RIBES can end up penalizing adequate translations that fall into this category. However, RIBES still outperforms BLEU significantly for distant language pairs where word order errors are more frequent, although BLEU is a slightly more accurate metric for languages with similar word order (Isozaki et al., 2010).
Burmese and English are distant enough to make RIBES a suitable metric (Hlaing et al., 2021). Since BLEU is the standard metric for evaluating accuracy, I thought it would still be useful to include it, as all the prior research I found on the accuracy of Burmese translation included BLEU scores.
Although human ratings of adequacy and fluency are more subjective, the end goal of these translations is ultimately to be understood by humans, so it only makes sense to include human input in their evaluation. Human input could also help resolve differences between BLEU and RIBES scores in certain places.
In my next blog post, I'll discuss some grammatical features of Burmese that may cause difficulties when translating to and from English.
Citations:
Hlaing, Z. Z., Thu, Y. K., Oo, T. M., San, M. E., Usanavasin, S., Netisopakul, P., & Supnithi, T. (2021). “NECTEC’s Participation in WAT-2021.” Proceedings of the 8th Workshop on Asian Translation, pp. 74-82, https://doi.org/10.18653/v1/2021.wat-1.6.
Isozaki, H., Hirao, T., Duh, K., Sudoh, K., & Tsukada, H. (2010). “Automatic Evaluation of Translation Quality for Distant Language Pairs.” Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pp. 944-952, https://aclanthology.org/D10-1092.pdf.
Week 3: I Lied
Hello readers!
So, it turns out that I might not actually be using all the translation accuracy metrics I spent the previous two blog posts explaining. Since prior research (Aye & Soe, 2024; Hlaing et al., 2021) had used these metrics, I thought they would be effective here as well; however, when we went through our data manually, we found multiple issues in our dataset's (the Asian Language Treebank Parallel Corpus) Burmese reference sentences. Some longer sentences were cut off, and some reference translations seemed less accurate than the machine translations they were meant to be compared to. Using these flawed translations as references would've led to more accurate results being punished. For instance, the Burmese reference translation for Pool C of the Rugby World Cup used the word for swimming pool, a mistake that Facebook made, but Google did not, making Google's translation more adequate. However, if we used an automated metric to compare the two translations to the reference, Facebook's translation would've received a higher score. Without accurate references, neither of the automated metrics I had originally planned to use would be accurate either. Therefore, we've pivoted to manually scoring each translation on adequacy, fluency, and handling of context.
Abandoning automated metrics does have some other advantages–it'll allow us to test sentences from a much greater pool of sources. Initially, we were limited to the ALT corpus, as there are very few publicly available datasets of parallel Burmese and English sentences. All the sentences in this dataset come from news articles which were originally written in English, meaning they are all in the formal register in Burmese (something I'll explain in the next blog post), don't use a lot of figurative language, and don't contain many Burmese words that would be difficult to translate to English (Riza et al., 2016). If references are no longer necessary, we can take in data from sources like social media posts, which would not only give us a more comprehensive view of these platforms' weaknesses and strengths, but also have more practical value, as translation errors on social media were the main issue that brought me to this project in the first place.
My next blog post will return to originally scheduled content and discuss a few grammatical features of Burmese that differ from English. See you there!
Citations:
Aye, N. Z. M., & Soe, K. M. (2024). “Neural Machine Translation Between Myanmar and English Languages.” Indian Journal of Computer Science and Engineering, 15(4), 394-405, https://doi.org/10.21817/indjcse/2024/v15i4/241504022.
Hlaing, Z. Z., Thu, Y. K., Oo, T. M., San, M. E., Usanavasin, S., Netisopakul, P., & Supnithi, T. (2021). “NECTEC’s Participation in WAT-2021.” Proceedings of the 8th Workshop on Asian Translation, pp. 74-82, https://doi.org/10.18653/v1/2021.wat-1.6.
Riza, H., Purwaodi, M., Gunarsoh, Uliniansyah, T., Ti, A. A., Aljunied, S. M., Mai, L. C., Thang, V. T., Thái, N. P., Sun, R., Chea, V., Soe, K. M., Nwet, K. T., Utiyama, M., & Ding, C. (2016). "Introduction of the Asian Language Treebank." Oriental COCSODA. https://www2.nict.go.jp/astrec-att/member/mutiyama/ALT/ALT-Parallel-Corpus-20191206/ALT-O-COCOSDA.pdf.
Week 4: Linguistic Features, Part 1
Hello everyone!
This week, I'll be explaining some linguistic features of Burmese that might raise problems when translating to and from English.
Diglossia
Diglossia is when one language community speaks two forms, or registers, of a language in different contexts. In Burmese, these two registers are the literary and the spoken. The two registers use different pronouns, case markers, and conjunctions (Bradley, 2010).
There are a few ways this could cause problems in translation; for one, it may be difficult to determine which register an English sentence would be translated into, since English doesn’t make this distinction. Another potential issue is that as the majority of written material that translation models have been trained on is likely in the literary register, they may struggle to translate sentences in the spoken register. This would affect the quality of translations in contexts such as social media, for example.
Pronouns
The Burmese pronoun system is fairly complex compared to English, taking into account not just gender and plurality, but also factors like age, relationship to speaker, and social status. English sentences often lack this information, and even when it’s included, it’ll be mentioned in other parts of the sentence than the pronoun itself, requiring translation services to be able to use context clues to find the correct pronouns.
The importance of gender also differs from English. In English, first and second person pronouns are gender neutral, while third person pronouns are gendered. The opposite is true in Burmese (Müller & Weymuth, 2017). Therefore, a strong understanding of context clues is necessary when translating in either direction.
My next blog post will be a continuation of this topic.
Citations:
Bradley, D. (2010). The Routledge Handbook of Sociolinguistics Around the World. Routledge.
Müller, A. & Weymuth, R. (2017). How Society Shapes Language: Personal Pronouns in the Greater Burma Zone. Asiatische Studien - Études Asiatiques, 71(1), 409-432. https://doi.org/10.1515/asia-2016-0021
Week 5: Linguistic Features, Part 2
Hello everyone!
This week, I’ll be going over a few other linguistic differences between Burmese and English.
Articles
Like most non-European languages, Burmese does not have articles (Dryer, 2013). Therefore, knowing what articles to use in an English translation is dependent on context. Using "a" instead of "the" rarely has a major impact on the meaning of a sentence, but it's a problem I expect to see fairly regularly in Burmese-to-English translations.
Word Order
While English word order is subject-verb-object (SVO), Burmese word order is subject-object-verb (SOV) (Stewart, 1955). SOV is the most common word order, so I don't think we'll see any particularly blatant errors here; however, word order differences play a large role in inaccuracies even among SOV languages with far more support, such as Japanese (Crystal, 1997; Isozaki et al., 2010).
Reduplication
Reduplication is quite common in Burmese, but the main area where I could see it being problematic in translation is in adjectives. For example, "ချော" on its own means "beautiful," and when reduplicated, "ချောချော," its meaning is intensified. I don't think reduplicated adjectives will be a major issue, as something like "ချောချော" can easily be translated into "very beautiful," but I think it's possible that translations could end up either ignoring the second adjective or translating the phrase directly into English, creating something grammatically incorrect like "beautiful beautiful."
In my next post, I'll be describing some of my preliminary findings.
Citations:
Crystal, D. (1997). The Cambridge Encyclopedia of Language. Cambridge University Press.
Dryer, M. S. (2013). Definite Articles. WALS Online. https://doi.org/10.5281/zenodo.13950591.
Isozaki, H., Hirao, T., Duh, K., Sudoh, K., & Tsukada, H. (2010). “Automatic Evaluation of Translation Quality for Distant Language Pairs.” Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pp. 944-952, https://aclanthology.org/D10-1092.pdf.
Stewart, J. A. (1955). Manual of Colloquial Burmese. Luzac and Company.
Week 6: Preliminary Findings - ALT
Hello everyone!
In this post, I'll be going over a few specific observations I've made regarding Google's and Facebook's translations of ALT sentences.
Expected Results
Both platforms have difficulty translating words with multiple meanings; for example, the word "အမြှုပ်" can refer to both bullet shells and foam, and although it was mentioned in the context of a terrorist attack, hinting toward the first definition, Google still translated it as "foam." Closely related words also tend to get mixed up: "industry," which should be "စက်မှု," was translated to "စက်ရုံ" (factory). As predicted, there are minor errors in articles and determiners, such as mixing up "the" and "this." I also noticed changes in verb tense in some sentences, which was pretty unsurprising to me, as English verb tenses are slightly more complex, and Burmese doesn't really have verb conjugations the same way English does (something I probably should've mentioned in the previous post, oops).
Unexpected Results
There are a few seemingly random words that Google and Facebook translate inconsistently. Google translates the word "စတုတ္ထ" (fourth) to "sixth" and "first," and Facebook translates "မြင်း" (horse) to "cattle" and "sheep," but also gets it right half the time. Numbers are another unexpected issue – in one sentence, Facebook translates နှစ်ဆယ့်ရှစ် (28) to "eight." I've seen Facebook mistranslate numbers on the app before, so I suppose it isn't as unexpected as the other problems in this section, but it still surprises me how it's able to correctly translate most of the rest of the sentence, but not the part I would've assumed would be the easiest.
Facebook was actually doing better than Google for a large portion of the ALT data, which I definitely didn't expect, seeing as they've had the most controversies over inaccurate translations. Both platforms outperformed my expectations – there were very few sentences I rated lower than a 5/10 in adequacy, and practically none were lower than a 5/10 in fluency.
My next post will go more into the translation models themselves.
Week 7: Translation Platforms, Part 1
Hello everyone!
In this blog post (and the next), I'll be going into more specifics about the three translation platforms I'm evaluating in my project: Google, Facebook, and Microsoft, and how they've been impacted by advancements in machine learning as a whole.
Between 2019 and 2020, machine learning improvements enabled Google Translate to implement several features that led to significant increases in BLEU score across all 100+ languages supported on the platform, particularly low-resource languages (Caswell & Liang, 2020). I'll mostly be focusing on updates that were aimed at low-resource languages. Since low-resource languages definitionally lack ample parallel data, which is what models are typically trained on to translate high-resource languages, these methods aim to use more abundant monolingual data to train translation models.
One such method is back-translation. Back-translation takes a human-written sentence in the target language and has a machine translation model generate a translation for it, creating synthetic parallel data that can then be used to train other models. In order to prevent models from learning from their own mistakes, lower-quality synthetic parallel sentences are filtered out using automated scoring systems (Khatri & Bhattacharyya, 2020) or the model's own confidence ratings (Wang et al., 2019). Back-translation primarily leads to gains in fluency (Caswell & Liang, 2020).
Another is M4 (Massively Multilingual, Massive Neural Machine Translation). M4 is the usage of a single model to translate between every supported language and one target language, typically English. Not only is this much more cost-effective and efficient, but it also improves translation quality for low-resource languages because the model can use its findings from related languages with far more resources (Arivazhagan et al., 2019). For instance, Yiddish, a low-resource Germanic language, can be translated more accurately because the model has access to data from higher-resource Germanic languages such as German and Dutch (Caswell & Liang, 2020). M4 also enables zero-shot machine translation, which is when models are able to translate between two languages with no parallel data. Although this doesn't pertain to Burmese-English translation, it's enabled Google Translate to add 110 new languages in 2024 alone (Caswell, 2024).
However, Google Translate's newer models have been slightly more prone to hallucinations than previous models (Caswell & Liang, 2020). Hallucinations can occur when models attempt to come up with meaningful translations from meaningless input, or simply add completely unrelated information. While I haven't experimented much with genuinely nonsensical data, I've observed a few hallucinations in my social media data, which I'll talk about soon.
I ended up saying a lot more about Google than I originally anticipated – my next blog post will be a (hopefully shorter) description of Facebook's and Microsoft's translation models.
Citations:
Arivazhagan, N., Bapna, A., Firat, O., Lepikhin, D., Johnson, M., Krikun, M., Chen, M. X., Cao, Y., Foster, G., Cherry, C., Macherey, W., Chen, Z., & Wu, Y. (2019). "Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges." arXiv, https://arxiv.org/abs/1907.05019.
Caswell, I. "110 new languages are coming to Google Translate." (2024). Google, https://blog.google/products/translate/google-translate-new-languages-2024/.
Caswell, I. & Liang, B. "Recent Advances in Google Translate." (2020). Google Research, https://research.google/blog/recent-advances-in-google-translate/.
Dale, D., Voita, E., Barrault, L., Costa-jussà, M. R. "Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Better." Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pp. 36-50, https://aclanthology.org/2023.acl-long.3.pdf.
Khatri, J. & Bhattacharyya, P. "Filtering Back-Translated Data in Unsupervised Neural Machine Translation." (2020). Proceedings of the 28th International Conference on Computational Linguistics, pp. 4334-4339, https://aclanthology.org/2020.coling-main.383.pdf.
Wang, S., Liu, Y., Wang, C., Luan, H., & Sun, M. "Improving Back-Translation with Uncertainty-based Confidence Estimation." (2019). Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 791-802, https://aclanthology.org/D19-1073.pdf.
Week 8: Translation Platforms, Part 2
Hello everyone!
In this post, I'll be going into more specifics about Facebook's and Microsoft's translation services, and how they differ from Google and each other.
Like Google, Facebook also finds back-translation to be extremely helpful in improving low-resource language translation; however, Facebook argues that M4 is not useful for Burmese-English translation in particular, stating that Burmese lacks significant similarities to any high-resource language (Chen et al., 2019). Not only does its language family, Sino-Tibetan, happen to be one of the most structurally diverse families (Sagart et al., 2019), but the only higher-resource Sino-Tibetan languages are Chinese languages (ex: Mandarin, Cantonese, Shanghainese), which are some of the few Sino-Tibetan languages that use SVO word order instead of Burmese's SOV (Dryer, 2003).
Facebook's primary parallel datasets are the largest two: the University of Computer Studies, Yangon (UCSY) dataset and the Asian Language Treebank (ALT) dataset (Chen et al., 2019). While Google and Bing haven't stated that they used the ALT dataset, it's highly likely that they did as it is the largest publicly available set of parallel Burmese-English sentences. It's possible that the high scores I've been seeing among their translations of ALT data are a partial result of this, which is a limitation I'll definitely keep in mind in my analysis.
Something unique about Facebook's translation research is that it considers toxicity (curse words, threats, racial slurs, etc.) and uses a human-generated toxicity detection list to filter out toxic language in training data. This is in order to prevent translations from including toxic language not present in the original text. While not explicitly stated in the paper, I personally think this added emphasis on toxicity detection is related to their previous bad history with hate speech and translation, and although it's not really part of this project, future research could definitely look at whether this has made Facebook Translate better than other platforms in this regard.
Microsoft
Microsoft also uses a multilingual model similar to M4 – however, instead of training one multilingual model on every language, they use a method called knowledge distillation, where individual "teacher" models are trained on single languages and train a multilingual "student" model to mimic them. Knowledge distillation enhances a model's accuracy for specific languages, making it more accurate than previous multilingual models (Tan et al., 2019).
Unlike either Google or Facebook, Bing Translate distinguishes between standard, formal, and casual tones (Microsoft Translator Blog, 2016), which could improve translation from English to Burmese, as it can actually take diglossia into account.
In my next post, I'll be talking about the datasets I've used to test these platforms.
Citations:
Chen, P., Shen, J., Le, M., Chaudhary, V., El-Kishky, A., Wenzek, G., Ott, M., & Ranzato, M. "Facebook AI’s WAT19 Myanmar-English Translation Task Submission." (2019). Proceedings of the 6th Workshop on Asian Translation, pp. 112-122, https://aclanthology.org/D19-5213/.
Costa-jussà, M. R., Cross, J., Çelebi, O., Elbayad, M., Heafield, K., Heffernan, K., Kalbassi, E., Lam, J., Licht, D., Maillard, J., Sun, A., Wang, S., Wenzek, G., Youngblood, A., Akula, B., Barrault, L., Gonzalez, G. M., Hansanti, P., Hoffman, J., Jarrett, S., Sadagopan, K. R., Rowe, D., Spruit, S., Tran, C., Andrews, P., Ayan, N. F., Bhosale, S., Edunov, S., Fan, A., Gao, C., Goswami, V., Guzmán, F., Koehn, P., Mourachko, A., Ropers, C., Saleem, S., Schwenk, H., Wang, J., "No Language Left Behind: Scaling Human-Centered Machine Translation." (2022). arXiv, https://ai.meta.com/research/publications/no-language-left-behind-scaling-human-centered-machine-translation/.
Dryer, M. "Word order in Sino-Tibetan languages from a typological and geographical perspective." (2003). The Sino-Tibetan languages.
"Fine-tune and customize your translations in Translator for Bing." (2016). Microsoft Translator Blog, https://www.microsoft.com/en-us/translator/blog/2016/11/09/fine-tune-and-customize-your-translations-in-translator-for-bing/.
Sagart, L., Jacques, G., Lai, Y., Ryder, R. J., Thouzeau, V., Greenhill, S. J., & List, J. "Dated language phylogenies shed light on the ancestry of Sino-Tibetan." (2019). Proceedings of the National Academy of Sciences of the United States of America, 116 (21) 10317-10322, https://doi.org/10.1073/pnas.1817972116.
Tan, X., Ren, Y., He, D., Qin, T., Zhao, Z., & Liu, T. "Multilingual neural machine translation with knowledge distillation." (2019). 7th International Conference on Learning Representations, https://openreview.net/pdf?id=S1gUsoR9YX.
Week 9: Datasets
Hello everyone!
In this post, I'll be talking about the datasets I've used for testing. (This is a post I probably should've made much earlier, oops.)
The Asian Language Treebank (ALT) Parallel Corpus
The ALT project was initiated by the Advanced Speech Translation Research and Development Promotion Center in Japan in order to promote NLP research for Asian languages. Its parallel corpus is the largest publicly available corpus of Burmese-English parallel sentences, which is the main reason I chose to use it.
The sentences themselves were originally taken from randomly selected English Wikinews articles and then manually translated into various Asian languages (as well as annotated for features such as parts of speech, but that isn't relevant to my project). For Burmese specifically, human translators were told to exclusively use the formal register, stick to the original English as closely as possible, and avoid figurative language (Thu et al., 2016). While the latter two instructions do make it easier for me to find where the translation and reference diverge, they also make my evaluations of ALT data less applicable to real-world contexts where different sentence structures and figures of speech may be used. As I mentioned in a previous post, the ALT dataset has also been used to train at least one (and most likely all) of the translation models I'm testing (Chen et al., 2019), which may increase their accuracy compared to in real-world scenarios.
My Social Media Dataset
After discovering the issues mentioned in my Week 3 update, I began compiling monolingual English and Burmese datasets from social media. The English data all comes from Twitter/X, with most tweets coming from the U.S. trending page at the time of the dataset's creation. Other tweets came from the profiles of public figures, including but not limited to Barack Obama, Mr. Beast, Neil DeGrasse Tyson, and Charli XCX. I specifically selected tweets that were on the less formal side in order to distinguish this data more from the ALT dataset, and tried to include a wide range of topics. In terms of slang and abbreviations, I avoided more recent terms that most adults would be unfamiliar with (essentially, anything that could be categorized as "brainrot"), but still tried to keep some of the things that make translation on social media a very different task to translating news articles (hashtags, emojis, and more well-known slang like "plz" and "lol," for example). I also included a few tweets with grammar or spelling errors, but not to the extent that they would obscure the meaning of the sentence.
I initially planned to make my Burmese dataset with tweets as well, but I soon found out that the Burmese-speaking community on Twitter was much smaller than I anticipated – most of the trending page after I set my language to Burmese and location to Myanmar was still in other languages. I decided to head to the platform with the largest Burmese-speaking community instead and began taking sentences from Facebook posts and comments (very inefficiently – I had to explore quite a few random groups, since Facebook doesn't have a trending page and I didn't want to exclusively take sentences from my relatives' posts) until my external mentor recommended that I use YouTube comments instead. My Burmese dataset is therefore a mix of Facebook posts and YouTube comments. The Burmese dataset unintentionally contains far less slang, possibly representative of demographic differences between Twitter and Facebook/YouTube, or perhaps just a difference in how Burmese speakers use the Internet (potential future research?). I tried to maintain a wide range of topics here as well.
In my next blog post, I'll be going over some of my earlier data from the social media dataset (no spoilers for final data!).
Citations:
Chen, P., Shen, J., Le, M., Chaudhary, V., El-Kishky, A., Wenzek, G., Ott, M., & Ranzato, M. “Facebook AI’s WAT19 Myanmar-English Translation Task Submission.” (2019). Proceedings of the 6th Workshop on Asian Translation, pp. 112-122, https://aclanthology.org/D19-5213/.
Thu, Y. K., Pa, W. P., Utiyama, M., Finch, A., & Sumita, E. "Introducing the Asian Language Treebank (ALT)." (2016). Proceedings of the Tenth International Conference on Language Resources and Evaluation, pp. 1574–1578, https://aclanthology.org/L16-1249/.
Week 10: Preliminary Findings — Social Media
Hello everyone!
In this post, I'll be sharing some of my observations from my social media dataset.
Expected Results
All three platforms seem to struggle with incomplete and grammatically incorrect sentences the most. Accuracy as a whole is noticeably lower than the ALT dataset. The ALT dataset had few hallucinations, but with social media, they seem to be commonplace. Again, words with multiple meanings are often misinterpreted – a reference to Tulsi Gabbard running was translated literally by Bing (but translated correctly by Google, as there was a mention of voting later in the sentence).
Unexpected Results
Bing Translate's "casual" tone often adds and removes emojis to posts. It doesn't have much of an effect on the accuracy of the translation itself, but it definitely wasn't something I was expecting.
All three platforms leave some words untranslated. This was common for Google with English names in the ALT dataset, and I didn't deduct points for that, but here, some words that definitely have Burmese translations are being left untranslated, such as "fiancé" and "artist," so I began deducting points in fluency. In one instance, Bing decided to transcribe "China" as "ချင်းနဲ့" (pronounced chin-neh and meaning "with ginger") instead of translating it into the actual word for China, causing "CHINA HERE I COME!!!!" to be translated as "You didn't come with the ginger?!!!!"
Bing had several strange text errors, the most egregious example being random Thai characters mixed in with Burmese words. Microsoft does have several research papers discussing the off-target problem with multilingual models, where multilingual models produce input from the wrong language (Chen et al., 2023), so perhaps I shouldn't have been completely surprised, but it was still quite jarring to actually see.
In my next post, I'll be reflecting on my project as a whole and discuss further research and possible improvements. (For final results, you'll have to watch my presentation!)
Citations:
Chen, L., Ma, S., Zhang, D., Wei, F., & Chang, B. "On the Off-Target Problem of Zero-Shot Multilingual Neural Machine Translation." (2023). arXiv, https://arxiv.org/abs/2401.12413.