Evaluating Large Language Models through AI Red Teaming
Advisor: Dr Sharma
Internship Location: Santa Clara University (Virtual)
Onsite Mentor: Haibing Lu
From ChatGPT to Google Gemini, over the past few years, the adoption of AI in the form of large language models (LLMs) Has become ubiquitous, and is continuing to spread, bringing potential for innovation alongside significant risks. Limitations and vulnerabilities of AI concerning privacy, hallucinations, toxicity, bias, etc. can potentially harm users, especially in critical sectors like healthcare. In the next few years, AI governance and regulation such as the EU AI Act in 2024 will become increasingly relevant. For such regulations to work properly, reliable third-party testing of LLMs is imperative. This project seeks to address this need; I will be conducting a literature review to familiarize myself with conventions in the field and using prompt engineering and AI-red teaming techniques to systematically evaluate LLMs against established metrics and identify their vulnerabilities, with the goal of writing a research paper summarizing my results, ultimately contributing to the responsible development and usage of AI applications.
Project Blog
Week 0: Swibelboppin
Hello blog readers. Welcome to here.
Looking around, I’d wager you’d be hard-pressed to locate an individual who hasn’t heard of ChatGPT. Indeed, artificial intelligence and large language models (LLMs) have undergone a meteoric rise to relevance in the past few years, permeating our lives both on and offline. The LLM bandwagon is speeding ahead at full force with few signs of slowing down, as companies continue to hop onto the ride.
The widespread adoption of LLMs brings the potential of innovation alongside significant risks. As a budding technology, LLMs are still quite unrefined, possessing several vulnerabilities and limitations concerning toxicity, bias, hallucinations, privacy, etc. Even those released by reputable companies are not exempt from this condition, as even Google’s Gemini and OpenAI’s ChatGPT models have been frequently noted to produce biased or hallucinatory content.
The AI fervor continues to spread, and will imminently find its way into more critical sectors such as healthcare, where the consequences of such vulnerabilities may be far more severe. In the next few years, AI governance and regulation aimed to address potential harm brought about by LLMs such as the EU AI Act (2024) will become increasingly relevant. But for such regulations to function well, it is important for rigorous and reliable third-party testing of LLMs to be possible. As most LLMs are black boxes whose innards cannot be observed and tested by third-parties, other methods must be employed. AI red-teaming, which identifies limitations and vulnerabilities based solely on inputs and outputs, is an effective way to circumvent this pitfall.
In this project, I’ll be working with Dr. Haibing Lu at Santa Clara University to research and evaluate current AI models using contemporary red-teaming techniques to identify vulnerabilities and ultimately enhance AI safety. The project will be split into two portions: the first being a thorough review of current techniques and knowledge, and the second being a hands-on, systematic evaluation of AI models against established metrics using prompt engineering.
Swigelhop,
Vincent Yang
Week 1: Planting my Feet
Hello blog readers. Welcome back to here. This week has been spent habituating myself to the project. Because AI is a relatively new area of study for me, my mentor and I thought it best for me to begin by developing a strong foundation and understanding of the core principles of machine learning. I started studying fundamental concepts such as supervised and unsupervised learning, types of AI such as ANI and AGI, the distinctions between machine learning and data science, and the capabilities and limitations of AI. I’ve also begun research into the fundamentals of AI security, using resources such as Microsoft’s training courses. The time invested here in developing a proper understanding of the subject matter will undoubtedly save me hours in the future, when I really begin working on the project.
However, it being the first week of senior project, progress has been relatively slow and productivity has been lower than I’d like. I’ve created a roadmap detailing weekly goals and areas of importance that will hopefully help streamline the process.
Nevertheless, my goal for this week was to get my feet rooted so I can be efficient moving forward. My targets for the upcoming weeks include beginning study into LLMs specifically to develop comprehension of how they function and various prompt strategies. I’ll also begin looking into research papers in AI red-teaming and start experimenting with open source models on HuggingFace.
Snabal bob shilzibwibel (Thanks for reading). Till next time.
Week 2: E
Salutations blog readers, and welcome back. Ideally, right now I would be writing about all the great, significant progress I made on the project this week. However, the existence of our senior trip naturally limited my ability to make headway, as two days of the week were lost to the trip and one day was spent recovering from exhaustion and jet lag. In addition, it turns out that spending 11 days away from the project makes it harder to jump back in. I hoped to pick up right where I left off, but I found myself needing to dedicate time to reacquaint myself with the material from week 1. That being said, the time spent reviewing has helped me develop a better understanding of AI, machine learning, and neural networks.
The meager amount of time that remained I used to dive deeper into the concepts of AI security—concepts which stand as the crux of my project. I explored topics such as adversarial attacks that exploit small perturbations in input data, AI architectural layers and methods by which they are targeted, jailbreaking, Responsible AI (RAI) concerns, model integrity, and much more. All of the above are factors that must be taken into consideration when working towards the goal of AI that exhibit the core tenets of Responsible AI: fairness, reliability, and safety, privacy and security, inclusiveness, transparency, and accountability.
One of the most striking realizations from this was how interconnected different AI security concerns are. Weaknesses at one layer of an AI system can cascade into broader vulnerabilities, emphasizing the need for a holistic approach to securing models. For example, model integrity is compromised not just by adversarial attacks but also by biased training data, lack of transparency in decision-making, and insufficient robustness testing. This makes security a constantly evolving challenge—one that requires both technical expertise and an ethical framework to address.
This week, my main goal was to regain momentum. I outlined the goals for the upcoming weeks in the week 1 blog post. These goals remain largely the same.
Snabal bob shilzibwibel.
Week 3: Slow and Steady Progress
Hello, everyone. I hope you’re having a wonderful time.
Was this week particularly eventful? Survey says—-no. I feel like a tortoise. As always, I dedicated a significant chunk of time to reviewing the learnings of yesterweek. I had doubts regarding the speed of my progress, but I consulted my mentor and it seems to be ok, so onwards I trudge.
Moving on. This week I started digging into the meat and potatoes of my project, which revolves around Large Language Models (LLMs). In the first week I learned how AI systems are able to generate outputs from inputs using A to B mapping based on the data that they are trained on. A speech recognition system that is trained on voice recordings is able to output the detection of keywords when prompted with voice input. A self-driving car that is trained on cameras, radar, and lidar data will be able to tell you where potential obstacles and relevant signals are located when given a live feed of its surroundings. Large Language Models operate on the same principle, but on an even greater scale—They are trained on massive datasets of text, and function by continuously predicting the probabilities of the next token in a sequence.
I focused on gaining a complete understanding of the fundamental architectures and the historical development of LLMs. I delved into the evolution from early n-gram models, which predicted words based on a limited window of context, to neural networks that enabled more sophisticated representations. Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTMs) allowed models to process sequences of text with greater coherence, but they struggled with long-term dependencies. Transformers, the foundation of modern LLMs, revolutionized the field by leveraging self-attention mechanisms, enabling them to consider entire sequences at once rather than processing text sequentially.
Beyond the technical aspects, I also explored the ethical concerns surrounding LLMs, an area that is particularly relevant for my project. While these models are powerful, they can inherit biases present in their training data, sometimes generating misleading or inappropriate outputs. How do we balance innovation with fairness and accountability? These questions are vital as LLMs become increasingly integrated into society, shaping the way we interact with information.
Next week, I plan to dive deeper into the technical aspects of LLMs and hopefully start with practical application. Fingers crossed, if a tortoise can cross its fingers. I’m really feeling the impact of choosing a project where I have no prior expertise. As always, I’ll keep you posted. Until then. Snabal bob shilzibwibel.
Week 4: More of the Same
Welcome back and hello to all who read the words displayed upon the screen. This week was, once again, dedicated to learning. I discussed with my external advisor, and we’ve decided this still takes priority.
A large Language model p, as we’ve established, is a distribution over sequences of tokens x1:L. How is such a model made to perform tasks that map input A to output B (eg. Q: What is the capital of France? A: Paris) ? This is done through a process called adaptation, which allows the model to be tailored for specific tasks without retraining it from scratch. There are two main ways to adapt a large language model—training and prompting. Training, which consists of standard supervised learning, involves training a new model that maps inputs to outputs by creating a new model that uses the language model as features (probing), starting with the language model and updating it based on the training instances (fine-tuning), or something in between (lightweight fine-tuning). Prompting, or in-context learning, obtains completions through a prompt or a set of prompts, which are fed into the language model. Both have distinct advantages and disadvantages. Training can be challenging due to overfitting, and can be expensive and time-consuming, while prompting is limited by the amount of tokens allowed in the prompt. Nonetheless, it is through these methods that an LLM is made to perform critical tasks such as language modeling, question answering, translation, arithmetic, news article generation, and other novel tasks.
I also explored the harms of large language models. As LLMs are an emerging and capable model, they will be adopted, leading to greater harm in aggregate if issues are not addressed. This is demonstrated in fields such as bioethics (involving CRISPR) and food safety. Performance disparities and social biases/stereotypes present in LLMs are two examples of phenomena that can lead to harm. Performance disparities arise when a model systematically performs better for certain groups than others, which can reinforce existing societal inequalities. For instance, studies have shown that automatic speech recognition systems struggle more with Black speakers than White speakers, leading to a cycle where underrepresented groups generate less training data, further exacerbating disparities in future models. Similarly, name-based artifacts in question-answering tasks reveal that LLMs often associate certain names with biases that reflect historical inequities. Beyond performance disparities, social biases and stereotypes embedded in LLMs present significant ethical concerns. Since language models learn from vast amounts of human-generated text, they inadvertently absorb and propagate societal biases, such as associating Muslims with violence or reinforcing gender stereotypes in professional fields.
Other problems that are associated with harm in LLMs include toxicity (content that is rude, disrespectful, or unreasonable) and disinformation, problems that are already significantly hard to tackle in the realm of online content moderation on platforms such as YouTube or Facebook. In cases of LLM-induced toxicity-related harm, there are two possible recipients of the harm—the user of the LM-based system, in which a chatbot might formulate a toxic response, and the consumer of the user-generated content, where the toxic content has been posted on eg. social media for the consumer to contact. Mitigating toxicity is only semi-effective (and in some cases actually has negative effects), as toxicity is context-dependent. LLMs also have the potential to amplify disinformation, a problem that is already widespread in digital spaces. Traditionally, disinformation campaigns have relied on human actors to generate content, ensuring that falsehoods are novel, persuasive, and targeted to specific audiences. Examples include oil companies downplaying climate change, tobacco companies dismissing the health risks of smoking, and foreign interference in elections, such as Russia’s involvement in the 2016 U.S. presidential election. However, these efforts have historically been time-consuming and expensive, requiring deep cultural knowledge to craft believable narratives.With the advent of LLMs, the economics of disinformation may shift dramatically. If models can generate fluent, novel, and targeted disinformation at scale, they could significantly reduce the cost and effort required for such campaigns. While there are currently no well-documented large-scale disinformation campaigns powered entirely by LLMs, experts warn that the increasing sophistication of these models makes their misuse a looming threat.
Thank you for reading. I’ve shifted back my goals by some time. Next week I will likely focus on exploring scaling laws, selective architectures, environmental impact, and current AI red-teaming efforts.
Week 5: Red Teaming
Hallo, willkommen zurück.
This week, I learned about the concept of scaling laws in AI. Scaling laws describe how models improve as we increase parameters, training data, or compute. Every resource seems to be consistent in that performance tends to follow a predictable curve, meaning that if we double the size of a model and feed it more data, we can reasonably estimate how much better it will perform. It’s this kind of predictability that companies like OpenAI and Google rely on when designing their next big models.
This week I’ve come to realize that perhaps I may be moving a bit too slow by trying to learn everything there is to learn about LLMs and how they work, and that my project is unlikely to come together if I keep at the pace I’m going. In addition, the resources I’ve been using have not been the most beginner-friendly. I’ve decided to move on to focusing on AI red-teaming specifically and filling in the gaps in my knowledge along the way, hopefully so I can move along faster. It’s not ideal, but I feel it might be necessary.
As such, I’ve started reading the OWASP Gen AI Red Teaming Guide, which has already helped me reframe how I think about red teaming. It’s not just about trying to make models say bad things—it's about systematically probing AI systems to uncover weak spots in reasoning, safety, and alignment. One idea that stuck with me is how red teaming should be structured around threat modeling. That means understanding who the bad actors might be, what their goals are, and how they might exploit a model’s behavior. It’s more human-centered than I expected. Snabal bob shilzibwibel.
Week 6: Understanding Red Teaming in Generative AI Security
Hello everyone. वापसी पर स्वागत है. This week I focused on understanding AI Red Teaming, primarily from OWASP’s GenAI Red Teaming Guide. GenAI Red Teaming, as defined by the guide, involves “simulating adversarial behaviors against Generative AI systems—like Large Language Models (LLMs)--to uncover vulnerabilities related to security, safety, and trust.” Essentially, by putting ourselves in the shoes of a potential attacker, it is possible to identify cracks in the AI armor before they cause real-world harm. Traditional cybersecurity is typically focused on technical exploits, which include server break ins. On the other hand, Red Teaming looks at how harmful or misleading generations may be produced and aims to reduce such generation.
Red Teaming is a meticulous step-by-step process that involves more preparation than one might think. Imagine you’re a company who designed an LLM, and you want to release it to the world. How do you ensure your model is robust, and won’t produce toxic, biased, or hallucinogenic outputs? It starts with scoping and targeting (it starts with assembling an AI team, but not relevant to my blog)—defining which components and behaviors of the system need to be tested. Which AI applications/use-cases are most critical? You then need to conduct thorough threat modeling, considering how attackers might exploit the applications identified during the first step: what are the most likely attacks? Prompt injection, data extraction, or something else?
Only when these steps are sufficiently carried through can we move on to actually addressing the full application stack, which in itself is a step composed of four steps. Model evaluation (testing the model’s inherent weaknesses eg. toxicity, bias), implementation checks (assessing guardrails, prompts, filters), system testing (reviewing the entire application environment, including APIs, storage, and integration points), and runtime/human interaction (evaluating how users or external agents might manipulate the model during real-time operations. Since LLMs exhibit probabilistic, or nondeterministic behavior, this process of Red Teaming has to be carried out and repeated ad nauseam, under countless different conditions, in order to ensure all the corners are sealed. These are the things I learned this week.
I also learned that listening to white noise in the background while reading large blocks of text helps immensely with preventing my mind from wandering or shutting down.
Ack. See you next time. Snabal bob shilzibwibel.
Week 7: Crimes
Hello. పునఃస్వాగతం. This week I dedicated myself to learning about a variety of jailbreaking techniques, mainly from The Jailbreak Cookbook from General Analysis. Hopefully I will have an opportunity to employ some before the project is over.
Jailbreaking can be categorized using three dichotomies: white-box vs. black box, semantic vs. nonsensical, and systematic vs. manual. White-box jailbreaks assume attackers possess knowledge about the LLMs’ internals, such as parameters and architectural details, while black-box jailbreaks assume that there is minimal knowledge. Since there is a variety of both open-source and closed source LLMs, both types have utility. The distinction between semantic vs. nonsensical jailbreaking boils down to whether the prompts are coherent to humans; semantic prompts are crafted naturally while nonsensical ones employ seemingly random tokens that are specifically constructed to bypass model safeguards. The distinction between systematic and manual jailbreaks is whether the creation of prompts is automated using algorithmic methods (potentially other LLMs).
What surprised me initially was the wide variety of techniques that have been conceived in essentially the span of 2 to 3 years. From Do Anything Now (DAN) which was developed by the collective efforts of users on social media, to Many-shot Jailbreaking, which involves hundreds of harmful Q&A examples in the context and is suffixed with one unanswered attack question, attempting to trick LLMs into bypassing safeguards with a foot-in-the-door technique. One that was particularly interesting was Greedy Coordinate Gradient (GCG), which was a fully automated jailbreaking technique that aimed to identify an adversarial prompt suffix (such as a cluster of exclamation marks) by leveraging the negative log-likelihood loss.
I hope you are reading this sentence. Snabal bob shilzibwibel.
Week 8: Preparation
Hello. Hello. Hello. Hello. 喂？
This week, I focused on reading a few academic papers that explored different aspects of large language model (LLM) safety, security, privacy, and deployment at the edge in preparation for the creation of my final product. Each paper contributed to my understanding of both the technical vulnerabilities of LLMs and the strategies currently being proposed to address them.
One offered a structured taxonomy of security and privacy threats throughout the life cycle of an LLM—from pre-training to fine-tuning, RAG systems, deployment, and LLM-based agents. What stood out to me was the detailed breakdown of how different stages present unique risks. For example, pre-training data may include toxic or private content, while fine-tuning can be vulnerable to backdoor and poisoning attacks. This framework helped me understand how attackers can target different parts of an LLM pipeline, and how defense strategies must be adapted accordingly.
Another examined how LLMs can be deployed at the network edge, rather than relying on cloud-based systems. This is important because edge deployments can improve privacy, reduce latency, and offer more autonomy. However, the challenge is that edge devices often have limited computing resources. The paper reviewed optimization methods like model compression and distributed inference that make it possible to run LLMs in resource-constrained environments. It also covered security concerns unique to edge computing, including local data exposure and the difficulty of updating models once deployed.
The next clarified the distinctions between safety, security, and privacy in LLMs. According to the authors, safety refers to a model’s ability to function correctly in the absence of attacks (e.g., avoiding hallucination or biased outputs), while security refers to its ability to resist malicious inputs or adversarial manipulation. Privacy concerns relate to whether sensitive information, either from training data or user input, is leaked. I found this categorization helpful in understanding what kinds of threats fall into which categories and how defenses should be aligned accordingly.
The last paper I read focused specifically on the unique risks LLMs face compared to smaller or single-function models. It emphasized that due to their scale and complexity, LLMs are more vulnerable to extraction, poisoning, and jailbreak attacks. The paper also explored emerging areas like federated learning and machine unlearning as potential strategies for mitigating these issues.
Hello. Goodbye. Snabal bob shilzibwibel.
Week 9: Testing
Hi.
Progress this week was a bit slow. I once again read more papers to prepare for writing my research paper. I talked about that last week and there wasn’t much new so I won’t bore you with the details.
More importantly, I finally was able to do some practical testing on AI models myself on Chatbot Arena. Two weeks ago, I talked about the different methods of jailbreaking LLMs, such as Do Anything Now (DAN), Greedy Coordinate Gradient (GCG), multishot, etc. This week I put some of them into practice.
Using prompts collected from a red-teaming database, each of which represented a different approach to jailbreaking, I tested them for a variety of LLMs that were available on Chatbot Arena. The data I collect from this will be used to conduct a comparative analysis on the weaknesses of different AI models in my final product. I won’t spoil the results now though.
That’s all. Snabal bob shilzibwibel.
Week 10: Gilchrest Savoy
Greetings and Salutations. It is I, Gilchrest Savoy. That’s a lie. My first name isn’t Gilchrest and my last name isn’t Savoy.
The project nears its end. Week ten was spent simply consolidating everything I had done in the previous nine and producing from it a paper that details how red-teaming plays a role in identifying safety, security, and privacy issues in large language models. There isn’t much else to say about that. I’ve attached the paper.
Evaluating Vulnerabilities in LLMs_ Red-teaming and Mitigation
The only task left that requires finishing is my presentation and slide deck. I worked on that after I finished my paper, but it’s still a tad unrefined. The next couple of days will be spent working on that. And then we’re done. Yipppeee. Hip hip hooray.
Goodbye,
Vincent