LLAMA-P: Designing Locally adapted open-source Large Language Models for AP studying
Advisor: Ms. Shahin
Internship Location: Algoverse (Remote)
Onsite Mentor: Kevin Zhu
Every year in May, thousands of students take Advanced Placement (AP) exams around the world. There Are a wide variety of existing study resources, from books to websites and videos, but there is a lack of true personalized help available. As a result, students may waste a lot of time finding the correct section to study or determining weaknesses. Large Language Models (LLMs) Are a promising alternative to current study materials. LLMs Are chatbots with the ability to remember past messages and form cohesive thoughts, but they often struggle with hallucination of information. In my project, I will try to mitigate these effects, building a study assistant based around the Llama-3 open source model. Through prompting techniques and a novel memory system, I’ll have the model consider both previous user responses and its own responses to best create study plans and materials for the user. My project will keep track of user weaknesses, keep notes of previously incorrectly answered problems, and use these to allow for further personalization in the revision process.
Project Blog
Week 0: Introduction
Hi everyone! Welcome to my senior project blog. Over the next few months, I’ll be using Large Language Models (LLMs) to create a study assistant for AP tests. Through this blog, I’ll document my journey, including my successes, failures, and lessons learned along the way!
Over the past couple of years, I’ve researched LLM reasoning mostly by testing how in-context learning techniques can improve LLM performance and reduce hallucinations. However, with this project, I’ll embark on a different journey: creating a complete LLM system using a model downloaded from the web. Doing this will allow me to exert greater control over the model’s thinking processes and the data it refers to when making decisions. This will be crucial for allowing personalized help for a specific task like AP exams with rigid guidelines and test structure.
Studying for AP exams is a very unique experience. Since it is a standardized test, the format is always the same, and content is very strictly built into units and described in a Course and Exam Description (CED). There are many ways to learn both the content and the test format, ranging from youtube videos to textbooks or websites. However, these resources are often cluttered, and inefficient. From my personal experience, I often found it hard to know what to study (you don’t know what you don’t know). Additionally, the resources aren’t very personalized, so it can be hard to find practice on topics you truly need help on. Overall, a lack of personalization in these resources makes studying inefficient and less helpful. However, interactive LLM chatbots can serve as a solution to this problem, mimicking a human tutor that knows what the student has struggled with in the past, and drawing from their understanding of the AP test to provide guidance.
In my project, I will be building methods to improve LLM efficacy in serving as study assistants, hopefully solving the personalization problem. The model I’ll be working with is Meta’s LLama-3, an open-source model that can be downloaded to local computers (unlike ChatGPT). To improve basic performance, I’ll use existing prompting techniques. To improve the model’s understanding of the subjects and format of AP tests, I’ll finetune it with AP-specific data. Finally, to help with personalization to the user, I’ll have the model keep track of old conversations and use them to inform new ones.
I’m excited to take on this project and hopefully help some AP students this year! Stay tuned for updates.
Week 1: Laying the Groundwork - Building a Uniform Testing System
Welcome back to my blog for the first real week of work!
This week, I’ve been focused on getting all the infrastructure for a successful project into place. I started off by importing the Llama 3 model family, in all of its variants. Afterwards, I began making a testing pipeline for inputting questions and having the model answer them. This pipeline includes the ability to augment the input with reasoning strategies, or change parameters like temperature or model type. I’ve also implemented a save function for the results that pushes the data to my google drive in a file, allowing me to perform analysis on the differences between generations. I’ve tested the entire system, and it is modular and robust from start to finish. Thus, I will be able to test different subjects and methods by changing a few lines of code and running it again! This will greatly help me stay organized and ensure uniformity across testing.
I’ve also been building strong foundations, reading (and re-reading!) papers that might be useful for my project. This has ranged from the Meta Llama-3 technical report to seminal developments like Chain-Of-Thought (having the model think in steps) and Self-Consistency (aggregating multiple model generations). I’ve researched how to develop a fine-tuning pipeline, which will be key when we want the model to understand CollegeBoard’s AP-specific guidelines and requirements. Finally, I’ve also read about strategies like Chain-of-Verification (forcing the model to double-check, provide evidence for its outputs) that serve to reduce the hallucination rate of models. With all this paper reading behind me, I feel confident applying my foundations to the project next week and implementing these strategies while modifying parameters and ideas within them.
Next week, I’ll begin testing different reasoning strategies to see how well they work. I’ll also be keeping a close eye on the hallucination rate, making sure my project gives the user the most accurate information possible. While the system doesn't need to be right every time, it needs to recognize when it can be wrong, and not just make up information to pretend like it understands. Considering overall accuracy and hallucination as metrics, I’ll determine the best strategies and combinations to continue on in the project with. See you next week!
Week 2: Llamas can reason, but slowly - Testing reasoning methods and fighting model limitations
Hello and welcome back to another edition of my blog!
This week was an interesting one, full of twists and turns. I began implementing reasoning techniques to see which methods of augmenting the base model were most effective in generating accurate and confident responses. I worked on 3 methods - Chain-of-thought, Self-consistency, and Chain-of-verification.
Chain-of-thought uses similar examples and example answers with sound reasoning to encourage the model to think for itself, leading to more thought-out answers. Chain-of-verification uses intermediate checks on each reasoning step of the model to ensure it is correct, resulting in a more confidently accurate final response. Finally, self-consistency aggregates multiple independent model runs, taking the consensus response to reduce one-off or probabilistic errors.
Implementing the methods was the smooth part. However, I quickly realized my measly CPU was not going to be practical for running locally. With longer prompts and multiple back-and-forth tasks between the models, it would take minutes just to answer a single question. I pivoted, using the Groq API service as a temporary fix. Groq has designed a special Language Processing Unit (LPU), a chip specifically for Natural Language Processing at lightning speed. However, the free tier has strict rate limits which I still need to figure out.
Progress is coming along well, but I will have to deal with the issue of compute limitations eventually. Next week, I’ll explore quantized Llama-3 models that can be run with less weights on my computer, or other free options. Once I have that in place, I’ll begin to benchmark the capabilities of the different methods. Finally, I will begin to develop fine-tuning which will give our LLM a large swath of AP-specific data. I’ll begin scraping together various AP documents to provide the LLM a good background and guidelines for how to answer. (But remember - we can only give the LLM so much due to compute restraints, so we need to be efficient with what we provide!) This is the first step towards the AP-specific domain knowledge of our final system. Stay tuned!
Week 3: Unexpected problems create uncanny solutions
Hi and welcome to another edition of my blog!
This week was full of challenges and setbacks, as I attempted to make Llama-3 faster on my local device. The first method I tried was quantization, which truncates weights and activations to use less memory space. While the speed improvement was very nice, the output was not. Quantizing an already small model meant that the new model had alarmingly high hallucination rate and sometimes was just nonsensical, so I’ll have to consider another method. The Groq API unfortunately won’t work either since I want to finetune the model, and the free API does not have options to do so.
I was able to successfully scrape the data for fine tuning in the meantime, using LLM PDF extraction. CollegeBoard publishes a Course and Exam Description (CED) for each class, with guidelines, key focus points, and skills. By using LLM text extraction tools and then summarizing the information into a more concise version, I’ll have an efficient set of data for the LLM to learn from and return to when it has questions about the AP-specific format.
Next week, I’ll hopefully find a solution to speeding up model inference while retaining the ability to fine-tune. I’m considering switching different models, or seeing if there are any ways to obtain free GPU credits or things of the sort. Otherwise, I may have to use purely in-context learning strategies with Groq while distributing queries to different models to stay under the ratelimit. Trying to get efficient outputs from good models for free is a unique, multifaceted problem, and I may have to take some wacky patchwork strategies to solve it.
Follow along to see where the journey takes me!
Week 4: Finding a hopeful solution
Hello and welcome to another edition of my blog!
If you remember my post last week, I was running into a lot of problems. Well, I’m happy to announce that I solved them… sort of. After many more unsuccessful attempts at getting fast inference, I’ve come to the conclusion that Groq is my only chance. I’ve also realized that fine-tuning is just too computationally expensive of a method for me to run anyway, so running Groq is fine. Instead of finding ways to optimize hardware for speed and less computational cost, I’ve decided to pivot and downside my methods on the software implementation side. Instead of using fine-tuning, I’ll use another LLM to find relevant ideas from the AP-scraped data to feed to the actual output model as part of the prompt. This will also come in handy later since it can power the memory system too. This will allow me to use the Groq API straight up. For model ratelimits, I plan to use different models on the free tier in a mix to beat the ratelimits. If that still isn't enough, I'll use different accounts and API keys to help further distribute requests. While not exactly the most elegant solution, I hope this will help my model stay online even at the free tier usage rate.
Next week, I’ll start testing out a working version of the system, running the distributed model, and implementing the memory system from there. The code should be similar for the memory system, since the memory system can just pull from the database of past chats the same way my new database of AP information can be pulled from for each prompt. I’m excited to explore this new direction. Stay tuned!
Week 5: Restrictions pass, innovation begins
Hello and welcome to another edition of my blog!
This week, I’ve been able to successfully implement the distributed model I described last week. The requests are pitched to different models, and once API limits are reached the requests redirect to a different model. Since the user will likely need to take time to comprehend each response and come up with new questions (estimated to be around multiple minutes), I feel comfortable with this somewhat patchy solution. I’ve also begun to run the alternate LLM that finds the most relevant AP information to augment the prompt.
Finally, I’ve also begun implementing the memory system. The process should be similar to the summarization process, but just with a different database of data. The only issue is that this database will constantly update. I still have to determine how the database will update, and what information is vital. For example, a user may get a question about the revolutionary war wrong. This information will be added to memory. If they get just one more question about the war correct, should the information be deleted from memory? What if they get it wrong again? Then what?
These are the questions I’ll hope to answer and then implement into my memory system next week. From file management to database maintenance, I’m excited to explore this new domain that I haven’t dove deep in before. Stay tuned!
Week 6: Prototype incoming!
Hello and welcome to another edition of my blog!
This week, I’ve been working on implementing the memory system, and so far the same code initially used for the summarization of AP data has held strong. I have been able to streamline the file management so that the user will be able to store their data between sessions. It’s super exciting to see the features work as intended! Right now, one issue I am running into is that my current summarization techniques for chats result in lengthy paragraphs. Once the user’s usage builds up, this will make the context and prompt length super high, which could be an issue for inference and also just cloud the model with extra unnecessary details. I’m looking into ways we can maintain the memory system and check for redundancy as new details are added.
Next week, I’m hoping to iron out all the details and put together all the parts of the model. I plan to begin running tests to ensure the model is working properly, simulating a student struggling with certain topics. I’ll also test different strategies for maintaining the memory system, from having token limits on summarization outputs as well as archiving old data or merging new data into old data. By the end, hopefully I’ll have a prototype to work on and iterate with for the last 3 weeks. Stay tuned!
Week 7: Extension Required
Hello and welcome to another edition of my blog!
This week, I’m almost done working out all the details. The memory system is a functional skeleton for now, and I’m able to have the LLM generate the summaries into separate files. I am still working on the self-updating nature of the database, but I might have a silver bullet (see below). I’ve been able to string together the different components, from prompting strategy to using context from both the memory and the AP specific data. However, the whole process felt janky, which is when I realized there might be a better way to do everything that requires a fundamental change in my approach to this problem.
This week, I suddenly remembered a chrome extension I used a while ago. It would slot into the user’s chats with ChatGPT, running in the background while appending to the prompt. Replicating this idea would solve a lot of problems that I have been running into. First, this would allow me to explicitly call memory and deposit memory with the model. I could also just append the relevant context for AP data directly to the prompt. Since the fine-tuning process would be too costly anyway, this solution could allow me to hit all the same pros of my existing method without any of the cons.
In the upcoming week, I’ll test this idea and let you all know how it goes. I’m very excited for what’s to come!
Week 8: Working around roadblocks
Hello and welcome back to another week of my blog!
This week has been a rollercoaster, trying to implement the extension I described last week. I initially thought I’d be able to just inject my supplemental data into the prompt via the extension, but it turns out the ChatGPT website has evolved quite a bit and requires complex commands to access and change the site data once “go” is pressed. Thus, my attempts to supply extra prompting to ChatGPT all failed one way or another.
I have two options. I can stick with the extension idea, but make it a pastebin that just reads the site data and runs the system remotely, and then provides a supplementary prompt for the user to paste in themselves. Alternatively, I could just go back to my original idea, submitting a proof of concept for the final product. In the next week, I’ll decide which of these options is more viable and helpful to the goals of my project. I’ll also begin testing all the parts of my system and making them fit together in anticipation of the final presentation. As stated in my syllabus, I want to begin testing by mimicking a user and going through the motions of a potentially studying student. I hope that this will give me the insight I need to tune up my system to make it the best it can be. The conclusion of my senior project is near…Stay tuned!
Week 9: Testing Matters
Hello and welcome back to another edition of my blog.
If there’s one thing I’ve learned through this senior project, it’s that no project ever goes smoothly. I’ve realized the importance of testing again and again, and giving ample time to modify ideas and debug. As the end of my project approaches, I’m going to focus less on adding new features and more on ensuring my existing ones can work together properly. I’ve decided to scrap the extension idea and stick with the original plan.
This week, I’ve been putting all the parts together and testing the whole system. I’ve realized that many of my ideas were overly theorized and I probably should’ve done this earlier, but all I can do now is learn from that and test as extensively as I can. I’ve been mimicking a model student, and asking the system if it can help me with my weaknesses. It does well when explicitly asked to refer to memory, but fails to account for it when not asked to. I’ve changed the prompting process to reflect this, adding more explicit pointers and guiding the model to act as I want it to.
I’ve also decided to change up my memory retrieval algorithm. Right now, the process is still consuming too many tokens, which is why I’ve changed to having an automated script search for keywords instead of having an LLM comb through the entire AP data document. This should help the entire system run faster and save on tokens.
As the project deadline approaches, I’ll continue testing and iterating. See you next week for a final product!
Week 10: Final touches
Hello and welcome to the final edition of my blog!
This week has been very eventful, as I finish up testing in preparation for my final product. I’ve been tailoring the model for specific use cases, simulating end users and their behaviors. One example I’ve been focusing on is a pipeline for practice examples. I failed to think about this earlier, since such a conversation would be two-step and require recent memory of the last turn of the conversation, but also wouldn’t need the entire memory. Thus, I’ve programmed in a few user-friendly modes, so that the user can see some likely use cases right off the bat.
Additionally, I’ve realized that the AP data is too big to be efficiently fed in as one document. The CED is over 540 pages long, and a bit of preprocessing actually went a long way for the model. I’ve split up the CED into different chunks based on unit and structure so that the LLM has a better idea of what it’s parsing. This will hopefully help with retrieval. I’ve also been experimenting with different retrieval modes from LLM-based search to keyword search or even TF-IDF.
I’ve spent this last week adding these features and fixing bugs as they come up. I’m excited to present at the symposium and show you all what I’ve been working on!