An AI Learning Partner: Utilizing Large Language Models for Memorization Assistance
Advisor: Ms. Sagal
Internship Location: Segmental.AI (Remote)
Onsite Mentor: Jim Yuan
Many people, including myself, struggle with one major aspect of learning: memorization. Having a study buddy is an effective approach to speed up memorization. Unfortunately, some students Are unable to find a peer or tutor. In order to address this problem, I will be building an AI study buddy that mimics a knowledgeable peer. The work will be done remotely under the supervision of a member of Segmental AI, a startup that is working on implementing AI for business and is knowledgeable about AI. I decided on AI due to the prevalence of Large Language Models like GPT-4o and Gemini, and because I had utilized their abilities for prior research. My work will produce a website that will allow the user to interact with a prototype of the AI study buddy and use it to review recitations and answers to generated questions based on a given text. This project will be one step for incorporating AI into the sphere of education as a tool for students to study. Finally, it may also allow me to discover possible solutions for problems with using AI in education.
Project Blog
Week 0: Blog
Hello readers!
Welcome to my week 0 blog post for my Senior Project. In this blog, you will learn a bit about working with existing Large Language Models and Web Development.
Memorization is an aspect of studying that I find challenging, leading me to struggle in classes where it is required, like history.
Because of this, I became interested in finding ways to study, which led me to think about one of the more widespread methods: studying with a friend. However, there have been many times where my friends have been busy, either with clubs or with other work. Situations like this and my previous work with LLMs are what inspired me to try using one to mimic a human peer, allowing someone to have a study buddy at any time.
For this project, I will explore pre-existing Large Language Models, such as GPT, Gemini, and Anthropic Sonnet, in order to find the fastest and most accurate model. Following that, I will test Speech to text programs such as Google Cloud Speech-to-text Service, Amazon Transcribe, and OpenAI’s Whisper for the same purpose. The next stage will be developing a website to handle the system. Finally, I will prepare materials to test the system and then use those materials to demonstrate the system.
For this project, I am being supervised by Jim Yuan from Segmental AI, a startup that focuses on using AI for business. I am also being supported by Ms. Sagal, a teacher at Basis Independent Fremont.
For my next blog post, I should have some information on the specific Large Language Model I have decided on.
Thank you for reading and stay tuned for more!
Week 1: Blog: Testing LLMs
Hello Readers!
For this first week, I have been working on testing different large language models to find which one is the fastest and the most accurate for evaluating a memorized piece of text. To accomplish this, I first found a pdf for my old Physics, Pre-Calculus, Linguistics, and History textbooks. These textbooks are where I found the text blocks I used as the correct memorization or "ground truth" for my experiments.
Following this, I took screenshots of a section in each text book and began writing some example "recitations" for testing purposes. I attempted to replicate speaking in the "recitations" as much as possible, in order to allow for more accurate results since the final product will incorporate speech to text (STT). After finding a good passage on the American Revolution from the History textbook, I created a perfect "recitation", one with only key phrases, a paraphrased "recitation", a partially incorrect "recitation", and a completely incorrect "recitation". For the Pre-Calculus and Physics textbooks, I used the integral definition and Coulomb's law for testing and created similar "recitations". For Linguistics, initially I wanted to test if the model could understand IPA sounds, but I realized I didn't have a good way to obtain IPA from a STT. Thus, I only used one example, where I provided a "recitation" describing the IPA symbols used for English's consonants. These examples were stored in a csv file for ease of testing.
After creating these examples, I began coding the tests. For this, I called the APIs of the following LLMs: GPT-4, Claude, and Gemini, asking each of them to give a score from 0 to 10 based on how accurate the recitation was compared to the image and to provide corrections if necessary. I timed each of these API calls using Pythons time package and saved everything to a csv file for ease of analysis. DeepSeek was not included as I realized the base model did not include image processing capabilities. I also ran into issues providing images to the LLaMA API, likely due to their recent API migration, leading to its exclusion. The results and code will be available here.
Overall, I have managed to stay on schedule and I hope to see you next blog
Week 2: Blog: Deciding on the LLM and Testing STT
Hello Readers!
For the second week, I have decided on the LLM I am going to use. I have also been working on testing Speech to Text models. To decide on the LLM, I looked over the 45 total responses by GPT, Gemini, and Claude to determine which model was the fastest and had the best responses for a given test. After looking over every response I eventually decided on using GPT as it gave the most good answers and was usually the fastest model.
Following this I first began working on testing a Python package I found that could do real time transcriptions called RealtimeSTT. This involved setting up its necessary dependencies like ffmpeg and writing code to allow for real time transcription. Following this, I spoke the sample “recitations” I created last week to see how accurately the Speech to Text model picked up on the text I spoke.
Unfortunately, I was unable to test the other STT services due to them not being real time. This is because the other services, such as Google Cloud Speech-to-text Service, Amazon Transcribe, and OpenAI’s Whisper, require audio files, making them require segmented audio inputs to create real time transcriptions, which is what I want for my website.
Overall, I am a little behind schedule due to being out for a trip, but since I have switched the schedule around a bit I should still be able to catch up to schedule soon.
Week 3: Blog: Working with RealtimeSTT and Beginning Web Development
Hello Readers!
For the third week, I have successfully utilized RealtimeSTT for real time transcription. Last week’s tests were done with regular transcription as I was running into problems with my words not being picked up by the microphone. This week, I began testing different pieces of code to try and fix it and tried reinstalling many of RealtimeSTT’s dependencies to see if they were the problem. On Monday I eventually got a simple piece of code working, although the transcription was relatively slow.
This led me to download the cuda version of pytorch to allow for the package to utilize my GPU for faster transcriptions. However, after doing so, I realized my code had stopped working. After trying many things, I eventually discovered that it was due to my headphones not working as a proper audio input device for the package. After unplugging them from my PC, the package worked perfectly, meaning real time transcription was achieved.
Despite not having perfect transcriptions, RealtimeSTT did provide relatively accurate real time transcriptions, so I decided to begin web application development using it as my Speech to Text service and GPT as my LLM.
Overall, I am now back on schedule due to modifying it, and I hope to see you at the next blog!
Week 4: Blog: Continued Website Development
Hello Readers!
For the fourth week, I have continued developing the website for my project. Last week, I had started working on a basic website using Flask to connect the Python server side to the client side of the website. A good analogy for this would be like a restaurant. The client side would be like waiters the user interacts with, the server side would be the chefs, and Flask would be how the two communicate. However, at the beginning of this week, I realized I wasn’t sure of a good way to implement RealtimeSTT as its implementation had to be server side but audio input had to be client side. This was an issue as in my tests, RealtimeSTT handled both audio input and transcription and I wasn’t sure how to constantly update the client side for real time transcription.
Luckily, after looking into it a little I discovered that RealtimeSTT had an example on how to utilize it for a website. Although I ran into issues running it locally due to the link not working, I eventually learned that you could open the client side just by dragging the html file onto the browser. After testing it and seeing that it worked, I decided to look into how it was implemented. Unfortunately, I found that instead of flask, it utilized packages called Websockets and Asyncio to create a websocket server and client. This type of server allows for more real time communication between the server and the client than one using Flask.
Thus, this week I worked on using these packages to create a basic websocket server and client for my website. Currently, I have been working on a simple UI that will allow me to interact with the key features. In the coming weeks, I will begin implementing the key LLM and Speech to text features.
Overall, I am continuing well on schedule, and I hope to see you at the next blog!
Week 5: Blog: Continued Website Development and Implementing GPT
Hello Readers!
For the fifth week, I have continued working on the website. At the beginning of this week, I implemented some necessary features such as requiring the user to upload a file before inputting their memorization so the AI can be prompted correctly and allowing the user to properly upload a file in the first place. This was done by modifying the CSS and html part of the website.
Following this, I began working on implementing one of the key features of my project: the LLM. Before this, I had just made the response to a user input the same text the user provided, allowing me to more easily implement the AI’s responses when I needed to.
Thus, after reusing the code I wrote prior to test GPT, I was able to implement an actual LLM response. However, I do need to continue working on this aspect of the website as I want the LLM to stream the text, meaning each word is displayed consecutively in its response until it is finished generating, instead of all the words being sent at once like a text message. In the next week, I should be able to implement that after working a bit more with the necessary packages. Following this, I will begin implementing RealtimeSTT for live speech to text transcription.
Overall, I am still on track, and I hope to see you at the next blog!
Week 6: Blog: Continued Website Development, Fixing Bugs, and Implementing GPT Text Streaming
Hello Readers!
For the 6th week, I have once again continued working on the website. At the end of last week, I had encountered a bug where uploading a file would attempt printing the page, but early this week I realized it was due to a line of code I forgot to remove.
I then began implementing GPT text streaming. Since GPT has the ability to stream their model’s responses by default, I just needed a way to get those chunks of text to the client side for them to be displayed. This was done by appending each chunk to the message stored on the client side, which was displayed. During this implementation, I did run into some bugs like the messages not displaying properly and thus the system not working, but I eventually resolved the issues.
Finally, I began making the entire system work, first by making sure the file uploading saved the file and its file type to variables on the server side. Then, I tested sending the file to GPT’s after base 64 encoding it to make sure the file uploading worked. After this, the entire basic system should be working and I can finally move on to incorporating RealtimeSTT into my project.
Overall, I am still on track to complete my project in time, and I hope to see you at the next blog!
Week 7: Blog: Beginning work on RealtimeSTT and Current Progress Overview
Hello Readers!
This week, I did not have as much time to work as I was visiting CalTech until Wednesday night. Thus, I only fixed some bugs and began looking into the example code for implementing RealtimeSTT into a websocket server so I could use it for my website.
Therefore, I will just give a brief overview on my progress so far. When first loading into the website, this prompt will show up:
Once clicking on the “Browse…” button, the user will be directed to upload a file. The currently supported extensions are .pdf, .jpg/.jpeg, and .png. After the user uploads a file the page will look like this:
The user then can send a message, or browse again to set a different material to be the memorization material. The speak button currently has no implementation but it will be used to implement RealtimeSTT. Once the user sends a message, the AI will provide a response like this:
The user can then continue providing memorizations or switch to a different document or image to memorize.
Overall, I am a little behind again due to not having too much time to work this week but I should be able to catch up next week. I hope to see you at the next blog!
Week 8: Blog: Continued Work on Implementing RealtimeSTT
Hello Readers!
This week I’ve been working on implementing the STT functionality into my website.
To properly implement STT, I needed to implement obtaining audio from the client side after pressing a button. I then needed to send the audio chunks to the server in order to transcribe them with RealtimeSTT. After transcription, I needed to send the text back to the client side to modify the text in the textbox. This would allow the user to modify the text in case the transcription had any problems.
I had to modify my system to allow for different types of websocket messages from the server. This was done by sending json strings instead of regular strings to the client side, which allowed from an extra “type” variable which I could read client side and perform different actions depending on its value.
While testing the code I also found a bug which did not allow me to continue sending messages after the first one. After looking into the code I realized I forgot to remove one line of code that closed the websocket connection, leading to the bug.
I have code that is capable of obtaining the audio from the client side. However, I still need to figure out how to use the button press to activate the functionality instead of having it be permanently active. I also have some code to transcribe the audio with RealtimeSTT, but I need to figure out which part of the code is necessary for my purposes and which isn’t as the example code has text color changing which I do not need. I have code to modify the text in the textbox but I need to finish the other parts in order to properly test it.
Overall, I should still be able to complete my project in time, and I hope to see you at the next blog!
Week 9: Blog: Finishing Work on RealtimeSTT
Hello Readers!
This week I have finally gotten RealtimeSTT to work with my system. After reworking the main loop of my server to be able to handle both audio messages in the form of bytes and text strings from the client, I was able to use JavaScript to implement a function that toggled input from the microphone and sent the audio and some metadata in the form of a JavaScript Blob (Binary Large Object).
After the server obtained the audio data, I attempted to send the data after removing the metadata from the Blob to the RealtimeSTT AudioToTextRecorder object which would handle transcription. Unfortunately, I ran into many issues with this step as although it was capable of producing somewhat real time transcription, it was relatively inaccurate at this stage. In the example code, a much slower but more accurate STT model appeared to run simultaneously on a separate thread, replacing the inaccurate transcription with a much better one but taking significantly longer. However, since this is just a prototype, having either of these problems should still be fine for this project. I also ran into some issues with async methods which I eventually resolved.
Overall, I am on track to finish this project and its presentation next week, and I hope to see you next week!
Week 10: Final Bug Improvements and Presentation Work
Hello Readers!
At the end of last week, I successfully implemented the dual STT model system into my website, which involved creating a second thread to run the larger model to have more accurate transcriptions. For the more accurate model, I used a smaller model than the example code so the transcription is much faster.
This week I have finally completed my project and a first draft of the presentation slides that go alongside it. At the beginning of the week, I mostly did some bug fixes and minor UI improvements such as: making the text box bigger by moving half the buttons to a second row, allowing the text chat to display LaTeX code as the LLM sometimes outputs it for equations, retaining existing text after the STT is activated, and other more minor fixes.
After doing so, I created a draft of my presentation, recorded a demo video, and did a practice presentation with my IA. I also set up the code system to work on my laptop so I could display my project.
Thank you for reading my blogs!