Designing a Telugu-English Translation Pipeline from Multilingual Radio Transcripts via LLM Interpretation
Advisor: Ms. Sagal
Internship Location: Bolly 92.3 FM, 43010 Christy St, Fremont, CA, 94555
Onsite Mentor: Mr. Kalyan Kattamuri
Language barriers limit access to communication across fields in healthcare, media, public speaking, and other spaces. Bridging the gap in language comprehension often requires manual effort, such as subtitle translation for movies or professional interpreters in a doctor’s office. While there are AI recognition tools that offer some assistance in current translation procedures, such protocols are limited by their performance and inability to translate as well as an experienced human speaker. Recent advances in large language models, however, present an opportunity to iterate and improve AI performance throughout this process. In prior work, I analyzed how multi-agent communication can influence interpretation accuracy within medical systems. I seek to expand this work in the domain of mass media, creating a protocol to interpret audio with multi-lingual dialects into a transcript of any language choice. The methodology involves extracting dialogue from previous radio transcripts, generating raw multilingual transcripts using speech recognition tools, segmenting changes in language, and translating non-English spoken parts to form one full transcript in English. Statistical analysis using the metrics METEOR and BLEU will provide evidence of protocol translationability. Future applications could include expanding datasets across local radio stations for general implementation.
Project Blog
Post 1: Blog 0: Starting the Journey!
Hello there!
My name is Raghav and thank you for following my Senior Project! Right now, you caught me at a time of college interviews and capstone final projects. Before I start my internship and kickstart the 10-week journey, I have a week’s stay in the Galapagos waiting for me! I am incredibly excited to cross the Equator for the first time and rest at the beach.
As a student from an Indian-American household, I’ve connected with my heritage through my mother-tongue and the traditions my parents have passed down. However, “code-switching” or language alternation in sentences is a common pattern within communities with exposure to multiple cultures. I’ve personally seen this change in how my mother-tongue, Telugu, is spoken with family and friends.
The first step to meaningfully studying language alternation has started with an opportunity to work at the Bolly 92.3 FM radio station with their subchannel Virijallu. Their Bay Area broadcast covers segments in Hindi, English, and Telugu, reaching over fifty thousand global listeners. I realized that listeners who aren’t familiar with multiple languages should have the opportunity to hear broadcasts or receive transcripts of streams in their language of preference.
Thus, I was inspired to create a project using tools for audio interpretation and language translation to interpret audio with multi-lingual dialects into a transcript of any language choice. The introduction of machine learning and open-source artificial intelligence APIs are viable tools for reducing the inconvenience of manually translating raw text directly. Artificial intelligence specifically offers an option for multi-lingual translation with extreme efficiency, although its accuracy struggles to match that of manual translation efforts.
It is imperative to address this research topic as multilingual translation and speech comprehension spans not just radio stations, but understanding each other’s voices and thoughts. Without a protocol to address struggles in language comprehension, people would be disconnected.
My AI-integrated protocol will open a larger space for targeted translation efforts as listeners follow a radio broadcast, listening to the dialogue in their language choice of preference. Throughout this process, as an intern at Virijallu, I will access transcripts where hosts alternate speaking between Telugu and English to test my protocol and determine its place in future live-streamed broadcasts.
Stay tuned for my next blog!
Week 1: A Glimpse at the Broadcasts!
Hello there! Welcome back to my Blog Posts! In my Blog 0, I mentioned my Senior Trip to Galapagos before starting the internship work. I just returned from a week of beaches, sightseeing, and hiking in humid weather. Now, I’m excited to share my progress during Week 1 of the Senior Project Internship. This week’s objective was about choosing audio segments from previous interviews that the Virijallu station has done.
First, I narrowed down which recordings and audio files I will use to test my future translation protocol. My on-site advisor shared a list of edited interview clips, so I spent the day listing every file, checking formats (MP3 for most archived files), and analyzing the duration of each track. It was amazing to have first-hand access to these materials as well as listen to a diverse range of interview topics. I further segmented the recordings between thirty seconds and a minute so that speech-to-text, language detection, and translation remain manageable. How frequently the host and guest speakers exchanged dialogue between Telugu and English was another primary factor in the selection process. From these segments, I shortlisted two candidates: an interview segment with the actress Rashmika Mandanna on the movie Animal, as well as an interview with APEX consulting services (a company focused on technology training and solutions).
Next, I defined time windows of each snippet within the larger audio file and wrote a Python script to organize lists of 30-second intervals. The script outputs two CSV files: (1) a formatted table of the snippet, the associated file name, and the time stamps; (2) the previous information alongside columns for the time stamps of Telugu-English language switching. Mapping each instance of language alternation by the second benchmarks the precision required for the translation protocol to be effective. By the end of the day, I could run the script as a way to organize sheets of time-stamped audio segments.
Afterwards, I filled the transcript tables manually while listening to the audio. As I edited the transcript CSVs, it was difficult to time-stamp language switches because the alternation was so rapid and seamless. An example from the “Rashmika” file is when the host says, “super excited ga vundi” (which directly translates to “feeling super excited”). I will reference the final file, including speaker names and the time intervals for comparative analysis against language segmentation tools. It will also aid future audio interpretation efforts based on understanding the dialogue structure.
Throughout the week, I also started reading on how lingual differences between English and Telugu affect sentence structure. One article I enjoyed reading connected linguistic structure to processing and translation implications (link: https://pubmed.ncbi.nlm.nih.gov/32223024/).
Next week, I will pull the audio from the selected segments and run speech-to-text to get raw transcripts for the protocol. Stay tuned for the next blog!
Week 2: A Jumble of Characters on the Screen!
Hello again! This week’s objective shifted from segment selection to audio extraction from interview files. I continued to use the shortlisted interview snippets (the “Rashmika” file and the “APEX” file) for analyzing the effectiveness of the protocol. Additionally, I researched speech processing which clarified the methodology for speech-to-text (STT).
I began audio collection by developing my own extraction script. One common method for audio extraction would be selecting the dialogue through VLC media player. However, my external advisor and I realized the benefits of storing the program in a single, larger program file that uses all the necessary libraries. Thus, my script accesses the Week 1 segment configs, locates the source MP3s for the candidate interviews, and exports 30-second snippets as 16 kHz mono WAV files (a single audio file sampled at 16,000 Hz).
Next, I incorporated the audio using the SpeechRecognition library (one of Python's most popular libraries for recognizing speech). On a MacOS, the SpeechRecognition library with Google’s Web Speech API works without extra system installation beyond Python. For better Telugu accuracy, switching to Google Cloud Speech-to-Text was a valuable alternative. The protocol also included optional stability checks, including running STT on copies of each clip with light noise or reduced quality. After running STT on each audio snippet and generating raw text, I saved each output per clip.
My results contained Telugu and English characters within a single sentence. For example, the APEX interview started as follows: విరిజల్లు శృతులకు నమస్కారం APEX Consulting Services మీరు … (greeting the listeners and introducing APEX Consulting Services). The visible mix in dialogue confirmed that the system was capturing language alternation properly. I was very happy to see this first real instance of a multilingual raw transcript from my own data.
With this script, I completed the main data processing portion of my protocol. Although I haven’t started translation yet, I noted objective observations from speech-to-text results. One challenge throughout the week was the script skipping over words in sentences. At the start, there was noticeable incoherence throughout a sentence. However, to maintain comparison of baseline performance across translation from audio files versus future “live” translation, I never modified the script but iterated script execution. This improved the overall performance while highlighting minor inconsistencies. Another issue I recognized was the grouping of words by phonetic sound, which lead to misinterpretation in the dialogue.
Differences in word choice and sentence structure will be captured numerically by metrics for the respective categories. In the coming weeks, I will further dissect the transcripts produced by speech-to-text and create a spreadsheet with sentences fully in English.
Thanks for reading my blog and see you again next week!
Week 3: Do the Words Make Sense?
Hello! Welcome to another blog! The last two weeks were all about creating the foundation of the protocol and building a multilingual transcript. This week, I focused on sentence coherence within the transcript and had a first look at a full English-translated transcript. I continued reading about machine translation systems, which was especially helpful for producing my first results table.
First, I needed to split the Telugu and English transcribed dialogue to analyze each part. I tested two different approaches: script-based detection and the Python langdetect library. For the first approach, I separated the multilingual text by script using Unicode blocks (ranges of code points that group characters by writing system). Words containing characters in the Telugu block were marked as Telugu words. The rest were marked as English. Running langdetect on Telugu-only and English-only spans helped confirm language splits as well.
To translate the mixed line segments into English, I passed the lines through Google Translate using the deep-translator library. The output directly converts language-mixed dialogue into a continuous English sentence.
I wanted to quantitatively analyze sentence coherence after Telugu to English translation. After some research, I found the metric MUSE (Multilingual Unsupervised and Supervised Embeddings). MUSE aligns word embeddings (representation of words as vectors of numbers) from different languages to compare meaning in multilingual transcripts. Specifically, I color-coded vocabulary across both spreadsheets: blue for the mixed transcript and orange for words from translation. Words that appear in both dialogue sets or have a similar meaning were closely visualized together, indicating that the mixed content and translation conveyed the same idea.
Furthermore, I tested the embeddings of English words within the mixed and translated transcripts by computing cosine similarity (a common test indicating similarity between multiple vectors/data points). For this protocol, the similarity of two vectors indicates how close the words are in meaning. Thus, a higher similarity or a higher cosine value indicates close similarity of word usage within mixed dialogue and the translated dialogue. A value less than or equal to 0.5 marked reevaluation of the translation. Surprisingly, both clips produced cosine values around 0.95, confirming that the translation of Telugu segments to English was extremely successful.
I plan to iterate on these results to confirm the protocol’s translative ability further. Unfortunately, the next stage of my testing will be cut short due to medical leave. This will also slightly delay my Week 4 internship work. However, when I return, I am excited to continue the translation process, analyzing my initial mixed transcript as one fully in English.
Until then, thank you for reading my blog!
Week 4: Expanding My Translation Approach!
Hello! After a week’s break, I’m excited to share my progress in Senior Project! A few weeks ago, I completed my analysis of the Telugu-English translation using Google pipelines. This week, I focused on alternate translation methods using large language models (ex. ChatGPT).
Based on previous segmentation scripts, I isolated Telugu strings per clip to paste fragments into different chatbot models. For general analysis, I used the three most common options in LLM translation: ChatGPT, Claude, and Gemini. A spreadsheet recorded rows of the “mixed” lines for context, the previous Google translations as a baseline, and paired columns for model name plus English text. In my readings on LLM prediction, one topic I found interesting was prompt sensitivity. Addressing the chatbot to translate literally or assume meaning changed word usage. For the purpose of comparing effectiveness across all three models, I phrased the same prompt and requested a literal translation.
I reused my programming scripts for MUSE analysis and calculating cosine similarity (read Week 3’s summarization for more context) to compare model performance. On the filled clips, Google and the LLMs landed in a similar high-cosine range (~0.91-0.95), yet the Google Translate API stood out among the 4 options. This finding, however, doesn’t consider specialized models within the chatbots used, some of which may match or exceed the Google pipeline performance.
Over the last 4 weeks, I have been creating a comprehensive set of scripts that conduct speech-to-text on audio files, segment multi-lingual transcripts, and translate the text entirely into English. Before storing my translations of the two selected audio files and moving on to “live-translation”, I decided to expand my data tables by adding multiple snippets from the same interviews. I re-ran my protocol on the “Rashmika” and “APEX” files for 30 seconds from the previous stopping point, copying all the spreadsheets I previously made and adding rows for the new transcriptions.
Re-testing audio transcription and translation was surprising, especially since having the scripts ready made the process effortless. The same issues arose, such as skipping over words and interpreting similar words by speech sound. After testing live-translation on the same base scripts, this is an issue I will come back to. For now, the general performance remained consistent, which highlights the potential of the protocol with future improvement.
In Week 5, I will complete the first phase of my protocol testing: multi-lingual translation from the audio files. This will include the selection and testing of other interviews that I listed as possible candidates, past the first two files. This way, I will benchmark a comprehensive set of translation outputs against phase 2 of my internship work. Until then, thanks for reading my blog, and stay tuned!
Week 5: Completing Phase 1!
Hello! Week 5 was all about re-testing the entire protocol on new broadcast segments and preparing for Phase 2 of the project: “live-testing”. To help prepare for my Phase 2 work, I read about language segmentation and media input.
In previous blogs, I mentioned the “Rashmika” and “APEX” files which I tested on multiple 30-second segments. From my initial search, I narrowed three other interview candidates I could use for further validation of my protocol before starting Phase 2. These recordings didn’t have as many language alternation points as the first two clips. “Mickey” covers a segment with Mickey J. Meyer, an Indian composer responsible for the soundtrack behind the movie Mahanti. The “Nagaraj” clip describes real estate development in the Bay Area, including personal finances. The third broadcast, titled “Neha”, is a general conversation with actress Neha Shetty about the movie dialogue.
After re-running the protocol from speech-to-text to final translation, I stored the full English transcripts from each snippet in a spreadsheet for future reference. My initial observations were incredibly surprising: the “Neha” and “Micky” 30-second snippet transcribed under 10 words. Unlike previous records of skipped words, these files had skipped sentences. This led to my reevaluation of the audio quality itself rather than the programming script. One difference was that in the introductions for both broadcasts, there was background music mixed with dialogue. Even as a listener, it was a little harder to hear dialogue as the music played. I retested the “Neha” and “Mickey” segments again on the later 30-second segments when the background music slowed down. The protocol regained performance, which confirmed my speculation.
I then stored the full English written transcripts for future comparison of word order and linguistic similarity. With this, I completed phase 1 of my senior project internship. As I understand the difference in performance between live testing and file transcription, I will adapt my protocol to a wider range of audio quality.
Starting phase 2, I researched the best methods to play the audio while running speech-to-text in the background with minimal interference. One reliable approach is digital loopback. For example, BlackHole is an audio driver for the Mac operating system, allowing users to pass audio between apps. I started a small script that records chunks from an input device so I can log results from protocol translation in the next few weeks. While this is only the start of mimicking live translation, it establishes a proper foundation for protocol application in real radio broadcasts.
In Week 6, I will continue with my live-translation approach with the hopes of seeing the first multilingual transcript as I play the audio. Until then, thanks for reading my blog!
Week 6: Messy Sound and Scrambled Words!
Week 6 marked the first full week of phase 2: “live” translation. Last week, I continued to record chunks as I played the broadcast, figuring out the best method for transcription. I also finished my reading on machine generation and simultaneous output, both of which were key to my progress.
Before completing my implementation, I needed to figure out if a digital loopback or an external audio source would be the strongest starting point. The first test was with a small JBL speaker, which I placed next to the MacBook input. Unfortunately, room noise, laptop fans, and speaker echo all leaked into the recording. Thus, I went back to my initial approach of digital loopback, interpreting the audio as a clean input device.
On a MacOS system, BlackHole 2ch is a virtual sink to recollect the audio (read Week 5’s summary on this topic). The missing piece was listening to the broadcast during the collection process. In Mac’s Audio MIDI setup, I created a Multi-Output Device that includes both BlackHole and my computer speakers. I then set my local sound settings to this Multi-Output setup for the best transcription. Throughout the week, I have improved the script to account for sound sensitivity, if the audio setup is functioning, and to write out word-by-word chunks across the 30-second segments.
Compared to the file reading approach, playing the audio while running speech-to-text requires greater computing power. Not only was listening time per chunk much longer than running through the stored file, but following the dialogue required stronger models. My results for this week are individual snippets rather than complete sentences. As I keep re-playing the chunks and improving audio sensitivity, I will reach my goal of a full transcript.
In the meantime, I compared the script performance against Mac Live Captions (Beta). This feature was introduced for local systems, especially if a user required subtitles while playing any audio on their computer. A key observation was the feature’s inability to distinguish multilingual audio, as each interpretation was transcribed as the closest English-sounding word. However, compared to my current script performance, the feature could rapidly transcribe English-spoken segments to near-perfect accuracy. I also tested audio transcription as I played the file in the Whisper Transcription app, matching the trend I saw in Mac Live Captions.
In Week 7, I will improve my “live” transcription script further, learning from previous open source applications and implementing key features. With a full transcript, I will store the full dialogue in English and generate my final results.
Week 7: Completing Live Transcription
Hello! This week, I generated full multilingual transcripts after live-playing the audio and worked on improving the script I built the last few weeks. I also continued my reading on language segmentation and contrast for future comparison between Phase 1 and Phase 2 results.
To transcribe one continuous text, I improved the structure for multilingual recognition using Google speech-to-text. Repurposing the programs I wrote in Phase 1, I analyzed word-by-word dialogue while differentiating between Telugu and English. The terminal tracks output per 30-second segment, with raw speech-to-text, then the translated segments. My initial results were what I expected of testing “live” translation: there was enough lag that not all the sentences were captured. What I did not expect was that the same dialogue repeated several times, almost as if the program was hallucinating.
I synced the speech-to-text outputs with a “live-session” spreadsheet that would label each interview being played and record its respective output. I continued with the full live transcriptions by playing the five interviews in order while analyzing program performance. The final scripts I developed throughout the week were organizational, mimicking the same format I stored generated scripts from Phase 1. This way, I can directly run my statistical comparison of protocol performance across file and live translation as I intended to at the start of the project.
The rest of the week, I returned to the other methods I tried for live translation during Week 6. I researched the backend for MacOS live captions and MacWhisperer for ways to improve my final live-translation script further. One observation I made was the setup both methods rely on: the Apple Neural Engine. The hardware accelerator is designed to advance AI tasks with greater energy efficiency. More importantly, it makes tasks on local MacOS systems that require machine learning or adaptability a lot easier. Although my scripts tune to the correct sound settings and frequency, they will never match the performance of these other options since I’m running everything on a local system.
For the final results comparison, I am creating a comprehensive collection of transcriptions from all three sources so I can compare progress on multiple fronts. In the next few weeks, I will create final results tables and complete my paper detailing the general progress I made towards this project. Until then, see you soon!
Week 8: Wrapping up Phase 2
Hello there! Week 8 was crazy, bouncing between building the last few scripts and making progress on my paper and figures. Last time, I organized live transcriptions of Telugu-English text taken from playing the audio files. This week, I re-tested those scripts again to understand and record performance limits.
I revisited my implementation of MUSE and cosine similarity (read earlier description in Week 3 during Phase 1) to analyze if meaning was similar across the mixed transcription and translated segments. However, unlike earlier scripts where I could analyze translation from the audio file, “live”-playing the clips across chunks meant I needed a separate format for storing dialogue across time stamps. I ran MUSE alongside the live capture script so I could constantly verify the results produced. I was happy to see that rapid transcription and translation didn’t drastically affect the protocol performance compared to the results from Phase 1 (though MUSE did skip over some 30-second segments).
After the final testing, I started working on my other results tables and graphs. Before I started the project, I was interested in analyzing the protocol performance compared to human-transcribed and translated text as well in the same interviews. I already had the difference in cosine similarity across both transcription methods. During my first week of work, my external advisor handed me those scripts for my reference. The metrics I initially used were METEOR (Metric for Evaluation of Translation with Explicit ORdering) and BLEU (Bilingual Evaluation Understudy). METEOR accounts for word matches, stems of words, and synonyms, while BLEU checks word order and patterns.
While I initially thought they were the best options for my baseline protocol performance, I didn’t account for the future challenges, including how often the machine skips or reinterprets words in speech-to-text. METEOR and BLEU are incredibly strict about word-by-word similarity, which made the calculated values extremely low. I felt that, for the stage of the protocol in its timeline, I needed to find metrics that compare meaning similarity rather than exact word-by-word accuracy. After some more research, I found the metric fastText cosine. It refers to the same concept of cosine similarity, where cosine similarity is the general mathematical formula, but fastText accounts for words, misspellings, and unseen text. The other metric I found was BERTScore F1, which measures how similar the file or live translated English is to human text in a transformer’s (machine’s) representation.
The results from BERTScore and fastText were much more promising, with file transcription barely outperforming the live transcription. As an example, on a scale of 0-1, values for the Rashimka file were between 0.7 and 0.9, meaning strong alignment. However, the issues I analyzed throughout this whole process with word skipping and misinterpretation were highlighted again when analyzing the METEOR and BLEU results. For future weeks, I will focus on the last bit of refining so my paper represents the most-refined protocol possible in this Senior Project timespan.
Week 9: Closer and Closer to the Symposium
Hello there! Week 9 was less eventful than previous weeks, focused on preparation for the Senior Project Symposium and preparing the final project. I continued my readings for last-minute tweaks, specifically on word morphology.
At this stage, I have already completed my initial draft of the Introduction with my internal advisor, as well as developed some results graphs. Before writing again, I improved the scripts for result generation, starting with the final comparison across BERTScore F1 and fastText cosine ( explained in previous blogs) as my main indicator of protocol performance. I summarized my general findings and created a graphic to represent them. The graphic indicates a strong ability to transcribe and translate the meaning of spoken dialogue. I also improved the generated graphs of MUSE cosine scores across file and live transcription methods, particularly the organization of bars alongside empty columns (empty scores due to flaws such as skipping over words and heavy noise influence in inputs).
As I was going through my results, one thing was clear: the protocol shows promise but has a long way to go. This came through as I finalized tables detailing a comprehensive report of METEOR/BLEU results across both Phases. I mentioned how these scores were both extremely low due to the captured flaws as well as their prioritization of word-to-word accuracy compared to the human-translated text. However, I didn’t notice that one video showed extremely variable results compared to the rest: the APEX interview. It was the only captured video with a crisp set of dialogue. All external noise in the other stored videos was mixed with the dialogue, making it difficult to separate. However, having the other interviews was a stress test for me to notice imperfections and where to take this project further.
Apart from analyzing the results, I built scripts for figure design, especially my main methodology figure that summarizes all the steps I’ve covered at this point. I took the previously stored tables containing raw translated transcripts from both phases and added them to my paper as a separate appendix. The last portion of my week was organizing all this content into a structured Overleaf document and completing drafts of every section in the paper. During my product preparation, I continued retesting every step to update the results to the product's strongest performance across every thirty-second segment. This way, I can focus all of Week 10 towards poster design, refining the paper itself, and preparing the presentation for the symposium. Although the main design portion is complete, I am extremely happy that I was able to reach each of my initial objectives while having a clear idea of where to take this project.
Thanks for reading my blog and see you for the last week!
Week 10: Presentation Time!
Hello there! Week 10 wraps up my work for Senior Project and I am thrilled with the progress. I am glad to have met all the objectives I noted before the internship. With the final protocol baseline designed and stress tested, I have a clear path towards further development and implementing this tool as a live translation aid.
This week, similar to the last, was focused entirely on symposium and material preparation. I continued working with my Internal Advisor throughout the week to move from rough draft to a final paper. The figures I created for methodology overview and results were all formatted in my LaTeX Overleaf document for final presentation. The last portion was refining my appendix, providing real-time protocol outputs. With that, I shifted my focus to the symposium poster.
I designed the poster as an aid to my research paper. The paper covers all the technical background behind each implementation process, but the poster is the best resource for summarization. I started with detailed bullet points in each section, especially Introduction and Methodology. As I kept improving the visuals, I focused on balancing the text with more data tables and images. One key figure I worked on was a step-by-step display taking a sample mixed-language dialogue and showcasing how speech-to-text, segmentation, and translation works. This complemented the general methods figure which simply connected all the steps from dialogue to translation. I transferred some tables in my appendix directly to the poster itself to display both quantitative and qualitative results.
This entire process, from ideation to writing and formatting, was extremely time-consuming. The last material I needed to prepare was my presentation for the symposium. This visual aid was the most simple to create, taking the best points and graphs from previous works in a set of slides. After I completed my presentation, I looked back at all three to make sure every data point and conclusion I wanted to convey was present. With all the materials complete, I started mock presentations for the actual symposium.
As May 16 approaches, I will continue to refine the protocol and update those charts with any improvements made. Although the Senior Project internship work is complete, I want to continue building this protocol, whether at Virijallu or another radio station that is interested in the product. I’ll go over it in more detail at the symposium. Until then, thank you for reading this blog!