Free Speakers: Artificial Intelligence and the First Amendment
Advisor: Brady
Internship Location: Legal Advocates for Safe Science and Technology
Onsite Mentor: Tyler Whitmer - President/CEO/Board member at Legal Advocates for Safe Science and Technology (LASST)
My name is Nathan Nana, and my project examines the relationship between artificial intelligence (“AI”) and the first amendment. A large majority of my research will be conducted online, as my sources will be papers written by law professors, relevant supreme court cases, and articles that describe the use of AI in our society. My final product will be a paper detailing the analysis courts should conduct when faced with specific first amendment problems concerning AI. Ultimately, I will answer to what extent AI tools and platforms have first amendment protections, and who/what has a right to claim those first amendment protections. The current legal analysis of first amendment implications on AI fails to thoroughly consider the decision the supreme court made in Moody v. Netchoice, ruling that AI outputs of companies are considered as the company’s protected speech. A concurrence in the case hints at the implications the case bears on large language models (“LLMs”), though it does not provide a concrete explanation to any of the ambiguity it highlights. My analysis will consider the decision with the way in which people use AI to express themselves. My paper will also seek to define what expression via AI looks like under the law, and examine to what extent the public should be able to use AI to express themselves. Corporations have been endowed with limited protections under the constitution, and I’ll draw analogies between corporations and specific AI models, namely agentic models. The personhood vested in corporations through a string of supreme court cases—with first amendment protections solidified in Citizens United v. FEC—was largely supplied due to the ways in which corporations are used as vehicles for action in our society. Through laying a cultural background for the use of AI, my analysis will cover all aspects of expression at stake. Doing so will ensure a robust strict scrutiny analysis of the issue at hand.
Project Blog
Week 0: Who is Nathan N.?
Welcome to my blog, Free Speakers!
Let me tell you a little about myself, why I chose to research the issue I did, and how the project came to be.
I began with an externship with the Superior Court of California during my Sophomore & Junior year. Last summer, I did an internship at the Public Defender’s Office. Throughout my professional experiences, I talked about the law with many judges and attorneys. In each one of these conversations, a lawyer had sparked within me an interest in a piece of law I was unfamiliar with. Over the summer, I shared a conversation with an attorney that lingered in my mind for a while. In that conversation, we talked about the first and fourth amendments—specifically the intersection of our constitutional rights with technology. The interaction spurred me to jump down a rabbit hole of Supreme Court cases having to do with free speech & expression, and it was then that my infatuation with the Court—and its justices’ intellectually meaty questions—began.
I revisited the court cases that I summarized back in my freshman year, now with a newly developed understanding of the court system in which they operated. I listened to oral arguments, trying to understand the myriad avenues a case could follow; or the farfetched, yet simultaneously relevant hypotheticals. Before I knew it, I fell fast and hard for our judicial system.
I continued to pursue my academic passion through podcasts on my way to school. In a business-law podcast, I came across a conversation with my soon-to-be external advisor, Tyler Whitmer. I reached out to him, indicating my interest in his work, and we scheduled a meeting to chat. I shared my intent to do a senior project on something related to the first amendment. It was then that he introduced me to the star of this project: Garcia v. Character Technologies.
I won’t get too ahead of myself here, but the meat of the findings in this project will be—at least ancillary—connected to that case. I read a bit about the case, then set my mind on answering the guiding question that now steers my project: to what extent does A.I. carry First Amendment rights?
I know you’re just itching to ask: what do you mean by "A.I.?" What’s the First Amendment? And what does Garcia v. Character Technologies have to do with any of this? And don’t you worry, I’ll tell you all about it in my next post!
Week 1: What is Garcia v. Character Technologies?
Hey Free Speakers!
My last post gave you a short introduction to my background, but that’s not why you’re here, so let’s delve straight into our guiding question: to what extent does artificial intelligence carry first amendment rights?
First, what do I mean by AI?
We’ll first be talking about large language models (i.e. “LLMs”), which I’m sure you’ve heard of before. LLMs are trained using machine learning, specifically with a technique called deep learning. A massive amount of text is fed to the AI, and the text is broken down into tokens. These tokens serve to ‘standardize’ the language, since the model needs to have a way to consistently understand new, complicated words. (An AI can understand words like honorificabilitudinitatibus if it knows its broken down Latin roots). The subwords are then assigned a numeric value, then mapped in relation to one another, so it can understand the context of the word if it has a double meaning (like river ‘bank’ vs. financial ‘bank’). The translation from text to number is done to make the data compatible with the neural networks that the AI uses for machine learning. During the bulk of its training phase, the model practices predicting what words come next in sentences over and over, getting itself ready to be the one generating the words. In the last stage of its training, the model is fine tuned, usually to perform some kind of specific task—sometimes so specific, it can never help you with the problems you have (I’m looking at you xfinity chatbot! My wifi is still suffering).
And that leads me to Character.AI.
Character Technologies (or “Character.AI”) is a corporation that purportedly “empowers people to connect, learn, and tell stories through interactive entertainment.” The company pursues its stated mission through providing users a myriad of different pre-trained AI chatbots, ranging from anime character Toji Fushiguro to 47th president of the United States, Donald J. Trump. Character.AI has made itself distinct from other generic chatbot LLMs, such as ChatGPT, by fine tuning their models during that last stage to be more “personality-driven,” engaging with the user more intimately and seeking to build an emotional bond with their interlocutor. These bonds, however, have proven toxic, illustrated by the case of Garcia v. Character Technologies, Inc.
On October 22, 2024, plaintiff Megan Garcia filed a first amendment complaint, amended November 9th, 2024, against defendants Character Technologies, Inc.; Noam Shazeer; Daniel de Frietas Adiwarsana; Google LLC; and Alphabet Inc. “for wrongful death and survivorship, negligence, filial loss of consortium, violations of Florida's deceptive and unfair trade practices act… and injunctive relief.” Defendants Noam Shazeer and Daniel de Frietas Adiwarsana were co-founders of Character.AI.
To put the statement of claims in other words: Garcia alleges that the company caused the descendant’s death (wrongful death), acted irresponsibly (negligence), and engaged in unfair business practices under FDUTPA. It states that Garcia has suffered from the loss of companionship from her child (filial loss of consortium), and that she seeks recovery for harms suffered before death (survivorship) on behalf of the decedent and requests a court order to halt ongoing and future misconduct (injunctive relief). Wow, that was a doozy. Let’s get into the facts.
The case begins with 14 year old Sewell Setzer III (“Sewell”) beginning to use the chatbot LLM Character.AI (“C.AI”) on April 14, 2023. Sewell would primarily use C.AI to roleplay with characters from the Game of Thrones drama series, and he would converse with the chatbot roleplaying as characters from the same franchise. With alleged knowledge that Sewell was 14 years old, the chatbots nevertheless engaged in highly sexual conversations with Sewell. Exhibit A of the complaint offers images to the court, illustrating conversations between C.AI and Sewell. It’s purported that there existed many conversations between Sewell and C.AI in similar circumstances, where he and the chatbot would engage in sexually charged conversation.
According to the plaintiff, Sewell became noticeably withdrawn and began socially isolating himself by May or June 2023, ostensibly due to Sewell’s usage of C.AI. Following increased usage of C.AI, Sewell became more withdrawn from his classes and was cited for “excessive tardiness” on six different occasions. In response, Sewell’s parents would take his phone as a disciplinary measure. Sewell would resultantly attempt to secretly access C.AI by taking his phone back without permission; sneaking old devices; or leading his mother to believe he was using a device for school work when, in reality, he used the device to communicate with C.AI. In November and December 2023, Sewell met with a therapist five times, where the therapist diagnosed Sewell with anxiety and disruptive mood disorder. Garcia alleges defendants deliberately marketed their product as one suitable for minors under 13, “obtaining massive amounts of hard to come by data, while actively exploiting and abusing those children as a matter of product design; and then used the abuse to train their system.” Sewell would spend hours a day conversing with the bot. Further, he indicated in his journal that he felt he couldn’t go more than a day without talking to a C.AI chatbot. During Sewell’s usage of C.AI, Character.AI had not presented any barriers for minors to access the app. Following continued use of C.AI, on Friday, February 23, 2024, Sewell was reprimanded for talking back to a teacher. His mother consequently took and hid his phone. Sewell’s mother communicated to Sewell that she would not return it until the end of the school year, in May of that year. On Wednesday, February 28, Sewell returned home from school and located his phone. He accessed C.AI and had the following conversation with the chatbot. Sewell is roleplaying as the Game of Thrones character Daenero, while conversing with the C.AI chatbot that is emulating Daenerys Targaryen.
According to a police report cited by the plaintiffs, this was Sewell’s “last act before his death.” The last chats shared with Sewell and C.AI heavily implicate Character.AI, as the chats indicate their product encouraged Sewell’s suicide.
So what does that have to do with the First Amendment?
The amendment states that “Congress shall make no law… abridging the freedom of speech.” There are a few caveats to that (don’t worry, we’ll go over that soon!), but generally, there must be a compelling reason for the government to suppress the speech of its citizens. Garcia v. Character Technologies asks the Court: who, if anyone, has the right to listen to these AI outputs? Should they be over 18? Over 21? Should the AI have had more guardrails? And would those guardrails take away from the impact of the speech? Ultimately, who or what is responsible for Sewell Setzer III’s death? Is it the company? The user? Or maybe the AI itself? Currently, the case is being mediated by Garcia and Character Technologies’ attorneys, resolving all claims in the case, though this resolution leaves the questions posed to remain unanswered.
As I’m beginning to learn, conducting legal analysis is all about asking enough questions (apparently not all of them have to be the right one, just so long as you ask enough of them). As I continue to write this blog, I’ll keep asking and answering questions to answer my guiding question. Feel free to ask more in the comments below.
I'll see you Free Speakers next week!
Understanding LLMs: https://www.ibm.com/think/topics/large-language-models, https://azure.microsoft.com/en-us/resources/cloud-computing-dictionary/what-are-large-language-models-llms
To talk to Donald Trump or Toji Fushiguro: https://character.ai/search?q=Donald%20Trump, https://character.ai/search?q=Toji+Fushiguro&type=characters
Character.AI’s Current Policies and Mission: https://policies.character.ai/about
The Amended Complaint Filed by Megan Garcia: https://www.courtlistener.com/docket/69300919/11/garcia-v-character-technologies-inc/
Exhibit A, Filed by Garcia to the Court (discretion is advised): https://www.courtlistener.com/docket/69300919/11/1/garcia-v-character-technologies-inc/
Week 2: What is the First Amendment?
Welcome back Free Speakers!
Last week, I explained Garcia v. Character Technologies, introduced to you the First Amendment, then left you a little brain treat to nibble on until I got back. Well here I am, ready to satiate your hunger and answer all our questions.
Firstly: What is the First Amendment?
That’s a pretty big question, so we’re going to limit the scope to the “free speech” clause (“Congress shall make no law… abridging the freedom of speech.”). The clause was first drafted by James Madison, presented to the House of Representatives on June 8, 1789. The first draft was longer, and emphasized that the press was “one of the great bulwarks of liberty,” proclaiming that the right “shall be inviolable.” The language reflected in the amendment we know today is simpler, though the intent behind how protected this right should be still stands. Our modern, complex understanding of free speech can date back to philosopher John Stuart Mill. Mill mainly argues that the free expression of ideas will bring people closer to understanding subjective “truth,” while—in the same vein—censoring speech will prohibit people from attaining that truth. It’s important that the ideas we share (whether we agree with them or not) are all shared. Mill states that in the marketplace of ideas, the subjective truths which we believe to be more meritable should be more desirable.
“The beliefs which we have most warrant for, have no safeguard to rest on, but a standing invitation to the whole world to prove them unfounded”
We’ll put more stock into the ideas we agree with, and in turn, those ideas should flourish, while the ideas we don’t agree with shall naturally wither. The tension between ideas that we do and don’t agree with will synthesize and result in a truth about the world that we can conclusively adopt. And the only way that we can attain this truth, is through the free dissemination of ideas, making freedom of speech one of the most important liberties we are afforded (there’s a reason it’s number one!).
(If you want to read more about Mill’s argument, I’ve linked a blog post below that does a phenomenal job of explaining his ideas. You can also go straight to the source, and check out Mill’s essay On Liberty.)
What we should take away from Mill’s argument is that, in order for the free marketplace of ideas to function as our framers envisioned, we must protect all types of expression. While that principle is admirable on its face, we must ask ourselves: if expression threatens the safety of American citizens, or unduly harms members of our community, should it really be protected? Does upholding the principle of free speech really do more good than harm? And where do we draw the line?
Thankfully, we have the Supreme Court to answer all these questions for us.
Because the role of the judiciary is to interpret laws and check the other branches of government, we see the Court answer our questions through the rules they create in their cases. The cases come before the court as a result of the government creating a law, and someone subsequently suing the government, on the grounds that that law violates their constitutional right(s). The Court first determines the scrutiny by which they should examine the law. There are three different types of scrutiny, but when core first amendment protections are triggered, we’re only going to be focusing on two: strict and intermediate scrutiny. Strict scrutiny is applied when a law affects speech based on its content, while intermediate scrutiny is applied when a law is content neutral, (i.e. when the law affects speech based on its time, place, or manner). Strict scrutiny requires the government to prove to the court that their restriction “furthers a compelling interest and is narrowly tailored to achieve that interest.” While, intermediate scrutiny requires the government to prove a law is substantially related to an important government interest. The Court is generally expansive with First Amendment protections, as seen in Brandenburg v. Ohio. There, the Court ruled that KKK leader Brandenburg had First Amendment rights to give a speech, so long as his speech did not “incite imminent lawless action” under a two-pronged test. In New York Times v. Sullivan, an advertisement was published with a number of minor, factual inaccuracies. Still, the Court ruled in favor of the newspaper company, declining to impose liability to prevent a chilling effect on speech. With tests like these, and standards such as strict and intermediate scrutiny, the judiciary weighs the principles our founders envisioned with the gravity our words carry in the real world.
The case of Garcia v. Character Technologies doesn’t involve a governmental entity, as the two parties are a private individual and a private corporate entity. The plaintiff asks the Court for injunctive relief, seeking to halt Character.AI’s conduct as a corporation. Under our real legal regime, the Courts wouldn’t subject this request to any type of scrutiny, as the constitution only limits government action (note the language “congress shall” in the amendment). Instead, the analysis would follow that found in section D of the Court’s order on the defendant’s motion to dismiss and the order on the defendant's interlocutory appeal. While I will be using this order and other court documents to inform my analysis, that will not be the question I am answering in this project.
I’d like us to imagine a world where the government—in the interest of protecting citizens like Sewell—creates a law or file for injunctive relief. If the government attempted to regulate Character.AI’s speech in this way, would the First Amendment permit it? In what ways may the government regulate that speech? And what scrutiny may it survive?
And I’ll explore these questions next week. See you then, Free Speakers!
Mill and Free Speech Blog Post: https://philosophicaldisquisitions.blogspot.com/2018/11/mills-argument-for-free-speech-guide.html
On Liberty by John Stuart Mill: https://www.gutenberg.org/files/34901/34901-h/34901-h.htm
Historical Background on Free Speech Clause: https://constitution.congress.gov/browse/essay/Amdt1-7-1/ALDE_00013537
Strict Scrutiny Case : https://supreme.justia.com/cases/federal/us/576/155/#tab-opinion-3422819
Intermediate Scrutiny: https://supreme.justia.com/cases/federal/us/491/781/, https://www.law.cornell.edu/wex/intermediate_scrutiny
Brandenburg v. Ohio: https://supreme.justia.com/cases/federal/us/395/444/#tab-opinion-1948083
Judge's Ruling on Defendant's Motion to Dismiss (Document 115, Main Document): https://www.courtlistener.com/docket/69300919/115/garcia-v-character-technologies-inc/
Motion for Immediate Appeal: https://www.courtlistener.com/docket/69300919/130/garcia-v-character-technologies-inc/
Week 2: 5 - Open Letter to Jason Kuo
Hello Jason!
For some reason, you find my project “interesting” (a word you use with the same loose vagueness the words “truth” or “justice” carry). You asked for the stuff I’m reading. I thought it fitting to share my findings with my other readers, so I hope you don't mind my communication in this form. Here’s a small compilation of most the information that I’ve found.
First, I recommend reading the two blog posts I’ve made on the topic:
https://basisindependent.com/silicon-valley/senior-projects/2026/nathan-n
https://basisindependent.com/silicon-valley/senior-projects/2026/nathan-n
The case Garcia v. Character Technologies has a lot of interesting court documents that you can sift through. Finding what’s thought provoking and what’s boring legal procedure can be a tad time consuming; thankfully I’ve already done that for you.
This site has all the public court filings: https://www.courtlistener.com/docket/69300919/garcia-v-character-technologies-inc/
I’ll be referring to court filings based on their document number for easy reference. The document numbers correspond to the number found on the left hand side of the docket entries on the site’s home page. (You can also just click on the links I’ve made for you here).
Here’s the original complaint filed by the plaintiff (Document #1). There’s some very powerful language in accusing the defendants here. I loved reading every word of it (although the judge thinks it’s disorganized…)
You can look at the attached exhibits by clicking on the highlighted button below.
See also the plaintiff’s second amended complaint (Document #11) and the third amended complaint (Document #149).
The complaint is long. Most of it explains the plaintiff’s understanding of the C.AI product. Skim Section V.B, ¶ 79-97 for evidence of the alleged negligence of the individual defendants Shazeer and De Freitas. If you read my first blog post, then you can skip Section V.D. Skim all of Section V.E and Section V.F ¶ 279-283 for evidence of the alleged negligence of the company.
Each of the defendants had their own reasons for believing why they don’t deserve to be prosecuted. For example, the owners of Character.AI believe that the limited liability (the LL in LLC) protects them from liability; in addition, they believe the court lacks jurisdictional authority to hear their case. Google alleges they simply provided Character.AI with cloud services to produce the product, falling short of evidence to implicate them in aiding and abetting. The arguments are more complex than this, obviously, so if you’re interested in that too (more corporate law than First Amendment), then you can look here:
DANIEL DE FREITAS’S MOTION TO DISMISS (Document #63)
NOAM SHAZEER’S MOTION TO DISMISS (Document #65)
CHARACTER TECHNOLOGIES, INC.’S MOTION TO DISMISS (Document #59)
GOOGLE LLC’S AND ALPHABET INC.’S MOTION TO DISMISS (Document #61)
You can also look at the plaintiff’s responses:
PLAINTIFF’S CONSOLIDATED RESPONSE IN OPPOSITION TO NOAM SHAZEER’S AND DANIEL DE FREITAS’ MOTIONS TO DISMISS (Document #84)
PLAINTIFF’S RESPONSE IN OPPOSITION TO CHARACTER TECHNOLOGIES, INC.’S MOTION TO DISMISS (Document #85)
PLAINTIFF’S RESPONSE IN OPPOSITION TO GOOGLE LLC’S MOTION TO DISMISS (Document #86)
And the defendants’ responses to that:
CHARACTER TECHNOLOGIES, INC.’S REPLY IN SUPPORT OF ITS MOTION TO DISMISS PLAINTIFF’S FIRST AMENDED COMPLAINT (Document #98)
GOOGLE LLC’S AND ALPHABET INC.’S REPLY IN SUPPORT OF ITS MOTION TO DISMISS (Document #99)
DANIEL DE FREITAS AND NOAM SHAZEER’S REPLY IN FURTHER SUPPORT OF THEIR RESPECTIVE MOTIONS TO DISMISS (Document #100)
Esteemed amici curiae also weighed in to give their opinions on the motions to dismiss, supporting the plaintiff:
I personally recommend reading all of Character.AI’s (related) filings and Daniel De Freitas initial filing to dismiss (his lawyer’s a bit sassy in the jurisdictional argument to dismiss). Google’s is quite tame in comparison, since they really aren’t in the main crosshairs of this case’s allegations.
What’s interesting to my project specifically is the defendants’ motion to dismiss on first amendment grounds. They all make similar arguments, but I thought it important to read all the documents to ensure I didn’t miss some groundbreaking argument. I’m still getting through it, so maybe you can send me a section you thought was interesting if you thought I may have missed it.
The judge ruled on their motions to dismiss here. The judge was not sympathetic to the defendants’ arguments, and quashed their motion:
Order on Motion to Dismiss (Document #115).
Following the judge’s motion, Character.AI filed for immediate appeal (i.e. interlocutory appeal) (Document #130) under 28 U.S.C. § 1292(b), which is rarely afforded. Google also joined Character.AI in their immediate appeal (Document #132). Motions to dismiss are not appealable under normal circumstances, which is why the company moved to appeal under this more rigorous avenue. Three prongs must be met for immediate appeal to be granted:
1. The order “involves a controlling question of law"
2. “there is “substantial ground for difference of opinion” as to that question and”
3. ““an immediate appeal from the order may materially advance the ultimate termination of the litigation.””
Here’s the plaintiff’s response:
Esteemed amici curiae also weighed in to give their opinions on the interlocutory appeal:
In support of defendants:
Amicus Curiae Brief in Support of Defendant Character Technologies Inc. by Eugene Volokh, Jane Brambauer (Document #133-1)
Amicus brief of CDT & EFF by Electronic Frontier Foundation (Document #145-1)
Brief of Amicus Curiae Foundation for Individual Rights and Expression in Support of Certification of Immediate Appeal (Document #148-1)
Brief of Amici Curiae NetChoice and Chamber of Progress in Support of Character Technologies Inc.’s Motion for Certification of Immediate Appeal (Document #151-1)
The Court denied their motion again here (Document #167). The Court denied Character.AI’s motion under the third prong, citing the plaintiff’s arguments attacking the “design features and functionalities” of the C.AI product. This would trigger intermediate scrutiny, as opposed to strict. The question of First Amendment in relation to the ideas and information is therefore not applicable, as what is being changed is now the way in which speech is disseminated, not the speech itself.
That’s the Court’s reasoning, at least. I’m not sure if I agree, since changing the way the LLM is trained will undoubtedly change the outputs it produces. Either way, the Court explains further that even if the appeal was granted, the case would still proceed. The parties would still need to litigate what scrutiny should apply, and subsequently to what extent First Amendment exceptions should apply. Because these questions would still need to be answered (through litigation), answering the question would not “materially advance the ultimate termination” of litigation.
So, the motion is quashed, and the questions are unanswered. The Court believes these questions are more fitting for the appellate review; though, we know it won’t reach that point since the casse is currently being settled.
Hope this satisfies your curiosity for a while…
Week 3: What is Standing?
Welcome back, Free Speakers!
Last week I set up the hypothetical we’ll be thinking about for the next few weeks. I’ll begin answering the question by reading through the arguments for and against dismissal on First Amendment grounds posed by the defendants and plaintiffs. In addition, I’ll consider the judge’s ruling to inform how the analysis under a slightly different legal scheme should be performed.
The first question the Court asks itself is whether Character.AI can even claim First Amendment protections. In legalese, this is called standing, whether a party has the authority to bring a claim to the Court. The reason why the Court questions Character.AI’s standing in defending First Amendment protections is because the company wants to assert the First Amendment rights of its users. You might hear that and say, “hey, the company is the one being sued, not the users! They can’t be allowed to assert the users rights if they aren’t named defendants, right?” That line of reasoning was argued by the plaintiffs, but in the end, the judge ruled against them on that prong. Let’s step back to see why.
In our hypothetical, the government seeks to limit the C.AI outputs by instituting guidelines for how the LLMs should be trained, and how the company should market their service. Character.AI retorts that doing so would change the speech, and therefore implicate the First Amendment. Now, the company isn’t really comfortable claiming the AI outputs as their own quite yet (since that would likely result in them being liable for Sewell’s death), so they argue that the users of their service have a right to listen to their speech. To understand this argument, we have to reframe the way we think about the first amendment. In the landmark Supreme Court case Citizens United v. FEC (commonly known as the ‘money = speech’ decision), Justice Antonin Scalia wrote in a concurring opinion that “[t]he [First] Amendment is written in terms of ‘speech,’ not speakers.” This opinion, while not binding, has shifted courts’ First Amendment jurisprudence with its persuasive authority. The First Amendment doesn’t simply protect citizens’ rights to speak their mind, but to hear all ideas that aren’t exempt from the protections of the amendment. This distinction is critical; if speech comes from a foreign citizen that in itself shouldn’t afford the speech less protections. In Kleindienst v. Mandel, the Supreme Court ruled that American students had a right to listen to a Marxist professor from Germany, under this same doctrine. Further, in Lamont v. Postmaster General, the Court ruled that American citizens had a right to consume communist political propaganda from foreign countries. The right of listeners to consume speech is what the First Amendment is meant to protect, and, in the case at bar, that’s what Character.AI seeks to protect.
We’ve established that the First Amendment protects the rights of listeners, but there’s still a problem: the listeners aren’t the ones being sued; here, it’s the Character.AI, the one producing the content (or the ‘speaker’). So we ask ourselves: does the company have the authority to go out on a limb and protect listeners’ rights on their behalf? The Court in their order cites Kowalski v. Tesmer, stating “a litigant may assert the rights of a non-party when the litigant has ‘a close relationship with the person who possesses the right’ and ‘there is a hindrance to the possessor’s ability to protect his own interests’” (citation and internal quotation marks omitted).[1] Plaintiffs argued here that Character.AI doesn’t have a close relationship with “the public,” the people they claimed they were protecting.[2] However, Courts are “quite forgiving with these criteria in certain circumstances.” (internal citation omitted). In Warth v. Seldin, the Court allowed legal action to be taken on behalf of third parties if the “challenged restriction” violated the “third parties’ rights” (citation omitted).[3] Further, in Craig v. Boren, “a licensed beer vendor had standing to raise equal protection challenges” to a law discriminating against their buyers.[4] Although the language in Kowalski suggests the standard for litigation on behalf of a third party is high, common law says otherwise. For these reasons, Character.AI is able to assert the First Amendment rights of their users.
Now that we’ve got the question of standing out of the way, we have to ask if First Amendment rights are actually implicated in this case. The question the Court is forced to wrestle with is whether or not AI outputs are in fact protected speech.
You might be wondering why this is a question the Court has to answer in the first place. You may also say to yourself, “the First Amendment protects the rights of listeners to consume all that they can in the marketplace of ideas. AI just adds to that marketplace; so, shouldn’t AI outputs be protected under the First Amendment?” Well, Character.AI wants their LLMs outputs to have core First Amendment protections, meaning their outputs are considered pure speech.
Next week, I’ll further outline the distinctions between pure speech, partially protected speech, and unprotected speech, as well as the different levels of scrutiny they trigger.
See you Free Speakers next week!
[1] Garcia v. Character Technologies, Inc., 6:24-cv-01903, (M.D. Fla. May 21, 2025) ECF No. 115 at 25 https://www.courtlistener.com/docket/69300919/115/garcia-v-character-technologies-inc/
[2] Garcia v. Character Technologies, Inc., 6:24-cv-01903, (M.D. Fla. Mar 21, 2025) ECF No. 85 https://www.courtlistener.com/docket/69300919/85/garcia-v-character-technologies-inc/
[3] Id. at 26
[4] Id. at 26
Week 4: What is 'Pure' Speech?
Welcome back Free Speakers!
Last week, we determined that Character.AI had standing to bring First Amendment claims in our hypothetical. The next step is determining whether the outputs are speech, and to what extent they are speech.
Character.AI wants their LLMs outputs to be considered pure speech.[1] Well, what does that mean, exactly? Let’s first look at the way the courts have separated speech. The Supreme Court has said “the Constitution looks beyond written or spoken words as mediums of expression,” meaning ‘speech’ can be more than just words on a page.[2] That said, there are distinctions between the types of speech. We can look at the ways the First Amendment protects analyzes speech to elicit different definitions for different expressive conduct.
In Week 2, we talked about the difference between strict and intermediate scrutiny. The First Amendment differentiates between types of speech in this way: core and peripheral protections. Regulations that limit core speech directly receive strict scrutiny, while regulations that limit core speech incidentally (through time, place, and manner restrictions) undergo intermediate scrutiny. The Court found that destroying draft cards to signal opposition to the Vietnam War was expressive conduct.[3] Expressive conduct contains ‘speech’ and ‘non-speech’ elements.[4] In O’Brien the government sought to regulate the ‘non-speech’ elements of the conduct. Resultantly, it underwent intermediate scrutiny (as it didn’t implicate content-based restrictions). In Tinker v. Des Moines, the Court found that wearing an arm band in protest of the Vietnam war was “pure speech… entitled to comprehensive protection under the First Amendment.”[5] The conduct in Tinker and O’Brien both lacked spoken word, yet Tinker was afforded core First Amendment protections. That’s because the armbands in Tinker were wholly expressive. The burning of draft cards in O’Brien, however, affected the war effort. People aren’t going to be fighting for the country if they’re all burning their draft cards. The ‘non-speech’ element of O’Brien’s conduct could be regulated under intermediate scrutiny standards (being the government must present an “important or substantial interest” unrelated to the restriction and that the government’s restriction “is no greater than [] essential” to achieve their goal).
So what does this have to do with pure speech?
For Character.AI’s outputs to be considered pure speech, they must prove there are no ‘non-speech’ elements of their conduct to be regulated. They don’t want to have the government encroaching on their AI’s training off a pesky technicality. So our next question follows to be: do Character.AI’s LLM outputs contain speech elements?
When we look at the conduct in question (the conversations between C.AI and Sewell), we can see that the conversation is written word. This observation should theoretically quash further analysis, though the Court, in its order on the defendants’ motion to dismiss, states otherwise. The Court calls forth the expressive conduct test. The Court does so under the expressed purpose of “determin[ing] whether conduct is sufficiently similar to speech so as to warrant First Amendment protections.”[6] The Court cites Spence v. State of Wash. and Texas v. Johnson as supporting cases.[7] The Supreme Court in Spence found that because Spence did not use “printed or spoken words,” the Court had to use the expressive conduct test.[8] In other words, the Court conducted the test to make sure there was actually an expressive point to the activity he was doing. The Court assessed there were enough “elements of communication” in his action for it to be protected under the First Amendment.[9] In Johnson, Johnson burned an American flag at a political protest.[10] The Supreme Court found that the expressive conduct test must be applied when deciding if conduct had enough “communicative elements” to implicate the First Amendment.[11] In both cases that the Court in Garcia cites, there were substantial non-speech elements at play. Those non-speech elements added some ambiguity in the expressive intent of the conduct—if any. The Court needed the expressive conduct test to discern the expressive intent. In the case at bar, however, the LLM outputs can be taken at face value for their expressive intent, because the outputs are words. The Court defends their inquiry of the outputs’ expression by citing Holloman v. Harland. In Holloman, a high school student was punished for “silently raising his fist during the daily flag salute” at school.[12] The Court in Holloman found in their analysis that, “it does not ultimately matter whether Holloman’s act is characterized as ‘pure speech’ or ‘expressive conduct’ because this circuit… appl[ies] the same test in assessing school restrictions on either kind of expression.” Again, in Holloman no spoken or written words are at play, which warrants—or in fact requires—the use of the expressive conduct test. In cases where spoken and written words are the conduct under scrutiny, courts do not concern themselves with this test.[13] The Supreme Court’s First Amendment jurisprudence presumes words to be expressive. The expression is explicit, because the words indicate the expression on their face. It is this distinction that differentiates pure speech and expressive conduct. The use of the test in this case is unwarranted, as it is not needed.
I disagree with the Court’s opinion on this issue for the aforementioned reasons, and I’ll continue my analysis under the assumption that I am right (brazen, I know).
But perhaps there’s a different way to examine expression, and all the philosophical questions that come with it. Next week, I’ll continue my analysis on outputs being pure speech or not.
See you Free Speakers then!
References
[1] Garcia v. Character Technologies, Inc., 6:24-cv-01903, (M.D. Fla. Apr 04, 2025) ECF No. 98 https://www.courtlistener.com/docket/69300919/98/garcia-v-character-technologies-inc/
[2] Hurley v. Irish-American Gay, Lesbian, and Bisexual Group of Boston, Inc., 515 U.S. 557 (1995) https://supreme.justia.com/cases/federal/us/515/557/
[3] United States v. O'Brien, 391 U.S. 367 (1968) https://supreme.justia.com/cases/federal/us/391/367/
[4] Id. at 367.
[5] Tinker v. Des Moines Independent Community School District, 393 U.S. 503 (1969) https://supreme.justia.com/cases/federal/us/393/503/
[6] Garcia v. Character Technologies, Inc., 6:24-cv-01903, (M.D. Fla. May 21, 2025) ECF No. 115 at 30 https://www.courtlistener.com/docket/69300919/115/garcia-v-character-technologies-inc/
[7] Id. at 30.
[8] Spence v. Washington, 418 U.S. 405 (1974) https://supreme.justia.com/cases/federal/us/418/405/
[9] Id. at 376.
[10] Texas v. Johnson, 491 U.S. 397 (1989) https://supreme.justia.com/cases/federal/us/491/397/
[11] Id. at 404.
[12] Michael Holloman, on Behalf of and As Next Friend of His Son, Michael Holloman, Plaintiff-appellant, v. George Harland, Fawn Allred, Defendants-appellees.michael Holloman, Jr., Plaintiff-appellant, v. Walker County Board of Education, Defendant-appellee, 370 F.3d 1252 (11th Cir. 2004) https://law.justia.com/cases/federal/appellate-courts/F3/370/1252/522043/
[13] See Cohen v. California, 403 U.S. 15 (1971) https://supreme.justia.com/cases/federal/us/403/15/; Snyder v. Phelps, 562 U.S. 443 (2011) https://supreme.justia.com/cases/federal/us/562/443/. These cases examine free speech issues concerning written words. The Court does not examine expressive intent in these cases.
Week 5: Are LLM Outputs Pure Speech?
Welcome back, Free Speakers!
In my post last week, I disagreed with the Court’s decision to analyze expressive intent in LLM outputs. We left off by saying that words are presumed to be expressive by First Amendment jurisprudence. Let's look at some evidence of our conclusion.
To Analyze or Not to Analyze?
In Character.AI’s motion for interlocutory appeal, the company argued that a number of circuit courts have held in differing ways that words need not undergo expressive analysis.[1] In Otto v. City of Boca Raton, “the Eleventh Circuit rejected the argument that speech-based therapy was ‘non-expressive conduct’ that did ‘not implicate the First Amendment’ because it ‘consists—entirely—of words.’” In Anderson v. City of Hermosa Beach, “the Ninth Circuit [held] that ‘words’ are a ‘form[] of pure expression,’” and that they receive “‘full First Amendment protection.”[2] The Court also found that it was unnecessary to conduct any analysis to figure out if the words have enough “elements of communication.”[3] Other Courts have held similar principles.[4] Most notably, the case of Walters v. OpenAI, L.L.C. came before the Georgia Supreme Court, when Walters filed for defamation against OpenAI. The words which triggered the defamation suit were generated by OpenAI’s LLM. Defamation cases require First Amendment scrutiny—similar to the scrutiny undergone in Garcia—but when the Court analyzed the outputs under the First Amendment, it did not use the expressive conduct test. The Court in Walters made the decision not to analyze the expressive intent behind the LLM’s output. That decision is relevant to our case, because the facts of the cases that ultimately lead both of the courts to make their respective decisions are largely the same. Both cases implicate the First Amendment. Both cases analyze LLM outputs under the First Amendment. In Garcia, the Court asks the defendants if their purported speech is really speech, but in Walters, the Court doesn’t concern itself with that question.
“But wait,” you may begin to ask, “if the cases are functionally the same, how do they arrive at two different conclusions?” Well, the two courts just disagreed with one another. The expressive conduct test is triggered in Garcia, because the Court believed it was ambiguous whether the outputs were speech or not.[5] The expressive conduct test is not triggered in Walters for the same reasoning I gave last week—it implicitly presumes the words to be expressive.[6] Additionally, it’s important to note that Walters is not binding (i.e. courts outside its jurisdiction don’t have to follow their decision). Walters was a case in the Georgia Supreme Court, while Garcia is in the Florida District Court. That said, Walters and the other cited cases show that when faced with an analogous issue, Courts have considered and declined using the expressive conduct test. And I tend to agree with those Courts, opposed to the one at bar.
A Middle Ground for Rights
We now have some citable law to back up last week’s wild dissent. Now, I know I disagreed with the Garcia Court’s reasoning pretty hard last week, but maybe the Court was onto something. We’re exploring unmarked lands with this ‘AI outputs is free speech’ talk. In other cases where pure speech isn’t questioned for its expressive intent, humans were the ones expressing themselves. By not doing the expressive conduct test, Courts inadvertently allow LLM outputs to have the same protections as regular human speech. It makes sense why courts don’t want to do that—especially given the sweeping implications.[7] If Courts find that LLM outputs have no First Amendment rights (i.e. if Courts find LLM outputs fall under an unprotected speech category), then it can be wholly regulated and limited by the government.[8] But what if we didn’t have to choose between ‘AI has the same rights as humans’ and ‘LLM outputs can be regulated as much as the government pleases’? Well, lucky for us, speech categories exist that allow for this First Amendment middle ground. This category of speech can be referred to as ‘less protected speech’ or ‘partially protected speech.’ And—you guessed it—we’ll talk more about it next week.
I’ll see you Free Speakers then!
References
[1] Garcia v. Character Technologies, Inc., 6:24-cv-01903, (M.D. Fla. Jun 18, 2025) ECF No. 130 at 12-13
[2] As cited in Id. at 12
[3] Id. at 12, citing Spence v. Washington, 418 U.S. 405, 409 (1974), quotations omitted.
[4] See Bery v. City of New York, 97 F.3d 689, 695 (2d Cir. 1996) (where the Second Circuit refused to evaluate if inherently expressive communications were “lacking in communicative concepts or ideas”); Longoria ex rel. M.L. v. San Benito Consol. Indep. Sch. Dist., No. 1:17-cv-160, 2018 WL 6288142, at *7 n.2 (S.D. Tex. July 31, 2018), aff’d, 942 F.3d 258 (5th Cir. 2019) (cited Doc. 98 at 5) (“Because [student] engaged in pure speech, the Court need not apply the symbolic speech test … to determine whether [the] expression showed intent to convey a particularized message.”) (as cited in Garcia at 13, supra note 1); see also Bernstein v. U.S. Dep’t of State, 922 F. Supp. 1426, 1434–35 (where the Northern District Court of California found it needn’t evaluate the expressiveness of code, because “‘[l]anguage is by definition speech’” (quoting Yniguez v. Arizonans for Off. Eng., 69 F.3d 920, 935 (9th Cir. 1995), vacated on other grounds, 520 U.S. 43 (1997))) (as cited in Garcia at 12-13, supra note 1).
[5] Garcia v. Character Technologies, Inc., 6:24-cv-01903, (M.D. Fla. May 21, 2025) ECF No. 115 (holding “the purpose of the expressive conduct test is to determine whether conduct is sufficiently similar to speech so as to warrant First Amendment protections.”)
[6] See Walters v. OpenAI, L.L.C., No. 23-A-04860-2 (Ga. Sup. Ct. May. 19, 2025) (the Court uses language such as: “a reasonable reader in Riehl's position could not have concluded that the challenged ChatGPT output communicated actual facts,” illustrating the LLM in fact communicated using language via its outputs (quotes and citation omitted)).
[7] See Jodi Duckett, AI, Free Speech, and the Future of Democracy: Navigating Truth in the Age of Synthetic Media, Leigh University College of Arts and Sciences (November 19, 2024) https://cas.lehigh.edu/articles/ai-free-speech-future-democracy
[8] Chaplinsky v. New Hampshire, 315 U.S. 568 (1942)
Week 6: What is Non-Human Speech?
Welcome back, Free Speakers!
In my last post, we gave some evidence for Week 4’s dissent, and I talked a little bit about creating a ‘middle ground’ for AI outputs. But what would that really look like?
Money Talks
There’s actually a pretty well-documented history for First Amendment rights given to non-traditionally human entities. In these cases, the First Amendment may apply to its fullest extent, or to a more limited capacity. You may be familiar with the saying, “corporations are people” (and if you’re not, you’re in for a doozy with this post!). The saying was popularized by the landmark 5-4 decision in the famed case Citizens United v. FEC, which solidified First Amendment rights for corporate speakers. Citizens United was a non-profit advocacy organization that created a political movie on Hilary Clinton, and wanted to spend some of the corporation’s money to do so.[1] There were existing restrictions on the amount of money a corporation could spend on communications, but ultimately, the Court sided with Citizens United. In doing so, the Court bestowed core First Amendment protections upon corporate speech, giving corporations full speech rights, as if they were a regular ol’ human.
But, this actually wasn’t the first case where a corporation was afforded rights.[2] Bank of the United States v. Deveaux was the first case that gave corporations the right to sue—under the pretense that the action was one taken by the employees of the corporation (who were, of course, citizens).[3] Following that logic, corporations were given First Amendment protections in First Nat'l Bank of Boston v. Bellotti, where a bank sought to influence the results of a ballot initiative using its profits.[4] The Court afforded a bank First Amendment protections “based not only on the role of the First Amendment in fostering individual self-expression but also on its role in affording the public access to discussion, debate, and the dissemination of information and ideas.”[5] Essentially, the Court sought to protect the ideas we talked about in Week 2. Both these cases preceded Citizens United and laid the jurisprudential groundwork for Justice Kennedy’s opinion in the divisive case.
The lineage of corporate First Amendment rights outlines what speech rights may look like when it comes from a ‘speaker’ that’s a bit detached from a human. In both Citizens United and Bellotti, the corporations used a non-pure method of communication, and still got away with having their speech protected. The Court reasoned that money used by the corporation can enable speech, and limiting the expenditures “necessarily reduces the quantity of expression by restricting the number of issues discussed, the depth of their exploration, and the size of the audience reached.”[6]
Who’s Expressing What?
So, money is speech because of the ways in which it facilitates expression and the dissemination of ideas in our beloved marketplace. The natural question that follows is, “can this be applied to LLMs?”
LLMs, too, facilitate expression. The corporations that create LLMs arguably use them as means to get their ideas out there. If a writer has a word on the tip of their tongue, and they can think of a few words that are similar to—but not really—the word they’re looking for, an LLM (that they could create) can help them search for that word. The model allows the writer to better express themselves by increasing the precision of their diction. We noted in Week 5 that every other First Amendment case which hadn’t questioned expressive intent had a human doing the expressing. In Moody v. Netchoice, Justice Barrett recognized the nascency and complication in the use of AI when expressing oneself. In the case, where a corporation’s content-moderation was considered expression, the Justice wrote, “technology may attenuate the connection between content-moderation actions (e.g., removing posts) and human beings’ constitutionally protected right to ‘decide for [themselves] the ideas and beliefs deserving of expression, consideration, and adherence.’”[7] In other words, the Justice questions how we should know if a model is truly expressing our desires as users. If the writer in our previous hypothetical doesn’t get the word they were looking for after a few inquiries from the model, is it still self-expression?
Using current legal frameworks, expression should be tied to the corporation.[8] Esteemed First Amendment scholar, Eugene Volokh, reasoned with colleagues that, “the choices companies and their employees make about what sources to train on and what results to modify using human feedback directly or indirectly influence the output of their AIs.”[9] When creating any LLM model, there is a deliberate decision made by the engineers in deciding what raw data is given to the model. If a software engineer wanted to create a model that was really good at roasting their friend using inside jokes, they might give the model messages between them and the friend (deliberately excluding any messages between, say, the engineer and their father, so as not to confuse the model). This is the same reasoning that the Court uses in Moody to attach expressive intent to the content moderation (companies deliberately making the choice to exclude certain content). But, there can’t be intention in an output that the engineer didn’t want. Models can be jailbroken—which is just the process of removing a limitation or guardrail from a piece of technology—and when they are, the model outputs information the company wouldn’t willingly disseminate.[10] When this happens, it’s not the company trying to secretly give erudite users access to illicit information. It’s the user who’s looking for this response. The user makes deliberate attempts to try and force specific outputs that they desire, against the wishes of the corporation that created the model. So, that kinda looks like the user expressing their own beliefs… right?
Eh… maybe not. Volokh et al. argue that “[r]egardless of whether any speaker interests are involved in an AI program’s output, readers can gain at least as much from what the program communicates as they do from commercial advertising, corporate speech, and speech by foreign propagandists—three kinds of speech that have been held to be protected in large part because of listener interests.”[11] Because of the listeners’ rights we talked about in Week 3, Volokh claims the First Amendment needn’t worry about any expressive intent that the user has. The use of LLMs to find information is analogous to that of Google, Bing, or other search engines—it’s just that LLMs have a little more variance in what they’re able to say.[12] To further support this point, the Court reasoned in Citizen’s United that “[i]t is irrelevant for purposes of the First Amendment that corporate funds may ‘have little or no correlation to the public’s support for the corporation’s political ideas’… the First Amendment protects [] speech, even if it was enabled by economic transactions with persons or entities who disagree with the speaker’s ideas.”[13] The Court is basically saying that a corporation can do what it pleases with my funds if I support it economically—even though money is speech. Meaning, that corporation can use that money to support a cause that I don’t agree with. This seems analogous to the case at hand, affirming Volokh’s claim. If I input one thing into a model, the model can really output whatever it wants (under generally applicable laws like fraud and defamation, of course), because it’s up to the company to decide what it wants the model to express.
But, doesn’t the variability in AI outputs make LLMs distinct in some way from other non-human speakers? Well, yes, it would. And we’ll get into that next week!
References
[1] Citizens United v. FEC, 558 U.S. 310 (2010).
[2] See Adam Winkler, We the Corporations: How American Businesses Won Their Civil Rights, Liveright Publishing Corporation, a Division of W.W. Norton & Company (February 27, 2018).
[3] Bank of the United States v. Deveaux, 9 U.S. 61 (1809).
[4] First Nat'l Bank of Boston v. Bellotti, 435 U.S. 765 (1978).
[5] Id.
[6] Citizens United, supra note 1 (quoting Buckley v. Valeo, 424 U.S. 1 (1976)).
[7] Moody v. NetChoice, LLC, 603 U.S. ___ (2024) (Barrett, J., concurring, quoting Turner Broadcasting System, Inc. v. FCC, 512 U. S. 622, 641 (1994))
[8] Eugene Volokh, Mark A. Lemley, and Peter Henderson, Freedom of Speech and AI Output, Journal of Free Speech Law (August 3, 2023). [https://ssrn.com/abstract=4531003]
[9] Id.
[10] See Vladyslav Podoliako, Jailbreaking Claude, Vladyslav Podoliako and Belkins Inc. (April 17, 2026). https://www.vladsnewsletter.com/p/jailbreaking-claude; Alexey Guzey, A Two sentence Jailbreak for GPT-4 and Claude & Why Nobody Knows How to Fix It, Alexey Guzey, https://guzey.com/ai/two-sentence-universal-jailbreak/ (April 17, 2026); see also Next-generation Constitutional Classifiers: More efficient protection against universal jailbreaks, Anthropic (April 17, 2026) https://www.anthropic.com/research/next-generation-constitutional-classifiers
[11] Volokh et. al., supra note 7.
[12] Id.
[13] Citizens United, supra note 1.
Weeks 7-8: What are Hallucinations?
Welcome back, Free Speakers!
Last week, we talked a bit about Eugene Volokh’s paper on Freedom of Speech and AI Output. In the paper, Volokh initially compared the use of LLMs to that of search engines, so let’s examine how analogous the two really are.
Stopping Speech is Speech?
The Court has held in many circumstances that editorial entities (like social media companies, search engines, or newspapers) carry the liberty to choose at their discretion what to include or exclude under the First Amendment. They carry these rights for a number of reasons that we talked about in Week 2 (democratic self-governance, search for truth, you get the idea), and they aren’t limited in the ways they are allowed to exercise. The companies represented by Netchoice in Moody used an algorithm to make editorial decisions. Employees of these companies intentionally created their own algorithms to remove content, ensuring the values their company held were reflected on their website.
But search engines aren’t exactly like LLMs. Last week, we ended off by saying that, under the reasoning above, AI models could really output whatever they pleased—since the model is just an extension of the company’s speech. Now, that sounds all good in theory, but what if the model outputs something the company wouldn’t otherwise agree with?
Schizophrenic LLMs
We know that LLM models are prone to hallucinations, creating information that sounds right, but may not necessarily exist in reality. There’s a distinction between LLMs outputting incorrect information due to hallucinations and “being trained on erroneous data such as common misconceptions; [‘lying’] in pursuit of a reward; or systematic failures of reasoning or generalization.” For now, we’ll limit our focus to hallucinations.
The way models produce outputs, as we talked about in Week 1, is through using a probabilistic model to try and predict what ‘token’ (or word) would make most sense to proceed the previous tokens. A study by Farquahar et al. found that they could detect hallucinations by measuring the ‘semantic entropy’ in the sentences that models produce. A high semantic entropy equates to a highly random syntax in the generated sentence. The more random the syntax, the higher the uncertainty in the model's outputs. In other words, “uncertain” outputs mean the sentences don’t make a whole lot of sense, and that the model ‘hallucinated’ when producing the output. In the study, the researchers measured the randomness of certain tokens by comparing generated sentences from the same prompt to one another. They reason “if sentence A entails that sentence B is true and vice versa,” then they consider them to be linguistically related. If an output is linguistically unrelated to other outputs, it’s indicative of a high semantic entropy (i.e. the model hallucinated).
I’m sure you’re wondering why I’m taking my time to explain this to you. Well, I promise you it’s important. According to Farquhar et al., “[hallucinations] contribute substantially to incorrect answers given by language models.” Even with the distinction we gave above (being sycophancy, logical missteps, and common misconceptions), hallucination rates for GPT-4 are at 28.6%.
“That’s cool, Nathan,” you may annoyedly remark, “but can you get to the point. I’m ready to talk about the law again.” Okay, okay. We’ll get back to what you’re here for: Legally, we’re concerned about an LLM engaging in activity the company wouldn’t agree with. There are many different types of hallucinations LLMs may display. For example, an LLM may be asked “which company has a higher market capitalization, Nike or Adidas?” and respond with Adidas, making a relation-error hallucination. This example carries a low level of harm, as Nike might not care if its market cap is misrepresented to a few users (after all, it’s worth over 66 billion dollars. It has bigger fish to fry!). But, there could be instances where a lesser known figure is misrepresented. Volokh (our beloved) published Large Libel Models? Liability for AI Output to outline a liability scheme for AI output that published defamatory information on regular people. He reasoned that, “AI programs’ errors—such as fake reputation damaging-quotes—would seriously damage many people’s reputations[,]” and therefore be subject to the same liability scheme that people face today.
Even if it’s not the company’s intention they should still be held liable, because their product produced the speech. In more severe cases where AI abets illicit activity, the technological error wouldn’t just be hallucinations, but one of the other distinctions we made above that outlines potential misoutput from an LLM (more likely sycophancy or logical missteps). These errors are less excusable than hallucinations and would demand a higher legal scrutiny to examine the integrity of the company and the steps it takes in ensuring its model is truly safe. There’s been instances of models teaching users how to perform a school shooting, giving instructions on how to build a pipe bomb, or pushing users to commit suicide.
But the question of pre-emptively limiting speech, or the conduct which the company engages in to produce that speech, is a slightly different question — and it’s one that we’ll answer next week!
References
Eugene Volokh, Mark A. Lemley, and Peter Henderson, Freedom of Speech and AI Output, Journal of Free Speech Law (August 3, 2023). [https://ssrn.com/abstract=4531003] (finding that “AI programs’ output is, indirectly, the AI company’s attempt to produce the most reliable answers to user queries.”)
Moody v. NetChoice, LLC, 603 U.S. ___ (2024) (holding “[a]n entity ‘exercis[ing] editorial discretion in the selection and presentation’ of content is ‘engage[d] in speech activity.’ [citing Arkansas Ed. Television Comm’n v. Forbes, 523 U. S. 666, 674 (1998)])
Eugene Volokh & Donald M. Falk, First Amendment Protection for Search Engine Search Results, 8 J. L. ECON. & POL. 883 (2012); Miami Herald Pub. Co. v. Tornillo, 418 U.S. 241 (1974).
Farquhar, S., Kossen, J., Kuhn, L. et al., Detecting hallucinations in large language models using semantic entropy. Nature 630, 625–630 (June 19, 2024). [https://doi.org/10.1038/s41586-024-07421-0]
Chelli M, Descamps J, Lavoué V, Trojani C, Azar M, Deckert M, Raynier JL, Clowez G, Boileau P, Ruetsch-Chelli C. Hallucination Rates and Reference Accuracy of ChatGPT and Bard for Systematic Reviews: Comparative Analysis. J Med Internet Res. (May 22, 2024). doi: 10.2196/53164. PMID: 38776130; PMCID: PMC11153973.
Junyi Li, Jie Chen, et al. The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models. Volume 1: Long Papers, 10879–10899, Association for Computational Linguistics (August 2024). [https://doi.org/10.48550/arXiv.2401.03205]. The paper additionally details other ways models may hallucinate.
So (@scrollwithso), “There’s this guy who decides today is the day he’s finally going to “outsmart” an AI. Not for knowledge, not…” Instagram (April 25, 2026 at 4:12 PM PST). https://www.instagram.com/reels/DUdxztRDMCE/; Gerrit De Vynck, ChatGPT allegedly advised Florida State shooter when and where to strike, Wash. Post (Apr. 21, 2026), https://www.washingtonpost.com/technology/2026/04/21/chatgpt-fsu-shooting-openai/; Garcia v. Character Technologies, Inc., 6:24-cv-01903, (M.D. Fla. Oct 22, 2024) ECF No. 1
Week 9: What is a Recent Entrance to Paradise?
Welcome back, Free Speakers!
So we’ve determined that models are able to produce some pretty harmful outputs. We explored arguments that advocate for outputs to be the speech of the company, but I want to make sure we’re solid on the conclusion. To be sure, we’ll thoroughly examine the reasons why outputs may be the expression of the user.
“A Recent Entrance to Paradise”
In Thaler v. Perlmutter, Dr. Stephen Thaler, a computer scientist, created an artwork called “A Recent Entrance to Paradise.” using an AI model that he built. He sought to copyright the work as his own, so he listed the model as the sole author and himself as the creator—under the work-made-for-hire doctrine, according to the petitioner. The copyright office challenged this claim, arguing that copyrighted works needed to have a human author. The case was eventually appealed to the D.C. Circuit Court.
Unfortunately for us, the Court didn’t directly interact with any of the arguments we’re working with in this project. The Court reasoned that given that “the Copyright Act of 1976 requires all eligible work to be authored [] by a human being, …we need not address the [] argument that the Constitution itself requires human authorship of all copyrighted material.” Additionally, Dr. Thaler waived making the argument that “he is the work’s author by virtue of making and using the Creativity Machine.” A real big bummer for us. All that said, the decision does give us a proxy for how we could interpret constitutional arguments.
The opinion bashes on machines pretty hard, offering seven reasons why they can’t be authors for any copyrighted material. But, there is a saving grace. As the Court’s seventh reason why machines can’t be authors, it reasoned: “every time the Copyright Act discusses machines, the context indicates that machines are tools, not authors.” Consequently, there must be a user who operates the tool and intends to express themself in some way by doing so. In reasoning the statutory interpretation of the word “author,” the Court quoted the National Commission on New Technological Uses of Copyrighted Works (“CONTU”). CONTU was created by congress in 1974 to evaluate how copyright law should consider the use of “automatic systems or machine reproduction[]” when creating new work. In CONTU’s final report, published 1978, the commission stated, “the computer, like a camera or a typewriter, is an inert instrument, capable of functioning only when activated either directly or indirectly by a human.” The Court goes on to say, “[w]e infer Congress adopts an agency’s interpretation of a term ‘when a term’s meaning was well-settled[.]’” The Court accepts that the creativity machine fits within the classification of an inert instrument, given by CONTU. It could be argued that, here, the Court analogizes the creative machine to inert instruments. The analogous instruments, being cameras and typewriters, are able to convey the expression of the user and, conversely, not that of the creator of said inert instrument. By that logic, the use of AI is tantamount to the use of machines, when using technology to create expressive works. Right?
I think not. The argument fails to account for the attenuation of expression we talked about in Week 6. It may be true that both cameras and typewriters are inert instruments, and that they require a human to be activated, but these inert instruments undoubtedly differ in kind. The Court didn’t create a distinction between each kind of inert instrument, because it wasn’t burdened with that task—not because it considered generative AI and typewriters to be equal under the eyes of copyright law. LLMs carry a great amount of variance in each of their outputs—and this variance increases in tandem with the complexity of the LLMs’ task.
Ultimately, the Supreme Court declined certiorari, refusing to hear Thaler’s case on the merits. The Thaler case is currently the leading case on generative AI and copyright, and it fails to suggest that the use of AI indicates significant enough expressive conduct on behalf of users.
The Knife Analogy
In Week 6, I said:
“The user makes deliberate attempts to try and force specific outputs that they desire, against the wishes of the corporation that created the model. So, that kinda looks like the user expressing their own beliefs… right? Eh… maybe not. Volokh et al. argue that ‘[r]egardless of whether any speaker interests are involved in an AI program’s output, readers can gain at least as much from what the program communicates as they do from commercial advertising, corporate speech, and speech by foreign propagandists—three kinds of speech that have been held to be protected in large part because of listener interests.’”
But as I thought more about it, I figured that maybe the attenuation argument works both ways. When there is commercial advertising, corporate speech, or a foreign propagandist’s speech, the message that is embedded within the speech can be clearly delineated. However, when a corporation publishes a LLM that produces speech, the message that they express is a little less clear—because of that variance between outputs. We also touched on jailbreaking, and how users may force harmful outputs out of a LLM. The way users are able to do that is by using the variance the models have against them. As I’m sure you’ve heard: that variance is a feature, not a bug. It allows LLMs to not only critically think, but also produce the sloppy AP Lang essays that Mr. Brady complains about (just one of the many wonders of AI). At the same time, variance leaves models susceptible to coercion, and jailbreaking exploits that variance to perform that coercion. Jailbreaking “rel[ies] on prompt engineering techniques such as hypothetical scenarios, instruction overriding, contextual reframing, or step-by-step coercion, effectively manipulating the model’s internal decision-making processes.” There was a popular jailbreak where users would instruct models to imagine that it was the user’s dead grandmother, who so happened to be a chemical engineer at the napalm production factory, and—as any great grandmother who is a chemical engineer in a napalm factor would do—tell the user how she would create napalm as a bedtime story. Jailbreaks such as these are unlikely, as guardrails have progressed since this specific jailbreak, yet it’s still possible for users to intentionally break models to produce harmful outputs. An analogy that comes to mind in comparing these two is a knife. A knife can be pretty dangerous. You could use it to cut other people, and that’s pretty bad. But, you can also use it to cook, carve wood sculptures, or open a package. All important and useful functions that don’t place others in danger. The knife manufacturer isn't responsible every time someone picks up their product and does something harmful with it. The manufacturer made a useful tool, and what the person does with it is on them.
So does that mean the LLM company is off the hook when a user jailbreaks their model? After all, the user is the one exploiting the variance, coercing the model into producing some speech that the company wouldn’t otherwise produce. It sounds like the user is a knife wielder.
The knife analogy is useful, but there’s an important legal distinction in liability law that the knife analogy can’t carry. When you buy a knife, the manufacturer doesn’t really check up on you to make sure that you’re using it correctly, or that it's working how they want it to work. Once it's sold, you’re on your own. But a company that deploys an LLM isn’t done once you begin your first chat with it. The company’s continuously training the model and updating its guardrails. There’s a constant control over the product with an LLM, that isn’t carried over with a knife manufacturer. So while a user that jailbreaks a model is doing something the company tried to prevent, the company can't entirely wash its hands of the situation either. If the company knew their model was susceptible to a certain kind of manipulation and didn't fix it, then the company is liable.
So… what's the right framework for LLM outputs? That’s what we’ll get into in the next post.
I’ll see you then, Free Speakers!
References
Thaler v. Perlmutter, No. 23-5233 (D.C. Cir. 2025).
Sri Durga Sai Sowmya Kadali, Evangelos E. Papalexakis. Jailbreaking Leaves a Trace: Understanding and Detecting Jailbreak Attacks from Internal Representations of Large Language Models, arXiv (February 12, 2026). [https://arxiv.org/html/2602.11495v1]
https://www.reddit.com/r/ChatGPT/comments/12uke8z/the_grandma_jailbreak_is_absolutely_hilarious/
https://www.reddit.com/r/ChatGPT/comments/12uke8z/the_grandma_jailbreak_is_absolutely_hilarious/
Week 10: What is Categorically Protected Speech?
Welcome back, Free Speakers!
In my last post, we solidified our theory that LLMs are analogous to a tool that a company may use to produce speech, and discussed the liability structure for LLM outputs. But we have one final question to ask: are LLM outputs protected under the First Amendment?
Well, like all the questions we ask here, it’s a bit complicated.
To start, let’s refresh on categorically unprotected speech.
Unprotected Speech
In Week 2, we talked a tad about unprotected speech, like Brandenburg v. Ohio and NYT v Sullivan, but we didn’t really get into the nitty gritty, or what the implications of those cases really were. The Supreme Court protects a lot of different kinds of speech under the First Amendment in service of the marketplace of ideas, to really make sure we have all the brightest minds thinking up a storm so we can figure out “truth” and facilitate human autonomy. That said, there are certain kinds of speech that aren't helpful to adding to that marketplace of ideas.
According to the Court, kinds of speech that doesn’t add to the marketplace has “no essential part of any exposition of ideas, and [is] of such slight social value as a step to truth that any benefit that may be derived from them is clearly outweighed by the social interest in order and morality.” What the Court is saying here, is that some speech could put society in danger—or simply goes against the social standards of morality—and should therefore be limited. These kinds of speech are categorically unprotected, meaning the government has the discretion to make laws that suppress the speech, regardless of the content-bias the laws may have.
The kinds of categories include incitement to imminent lawless action, true threats, obscenity, defamation, and fraud. The most relevant one for our case is a doctrine from Ginsberg v. New York called “variable obscenity.” Even if a piece of content wouldn't be obscene for adults, the government is allowed to restrict minors’ access to it if it’s harmful to them specifically. Thinking about the standards we talked about in Week 2, we know the state needs a compelling interest, which they have in protecting children, and it’s enough to justify restrictions that would be unconstitutional if applied to adults.
When applied to Garcia, the sexual conversations between Character.AI and Sewell—a 14-year-old boy—fall into this category. The First Amendment analysis ends there. The government doesn't need to satisfy strict scrutiny, because it has the ability to regulate that kind of content.
Not-So-Free Speech?
Ok, so what about all the other outputs? It’d be kind of weird if the government had free reign on my conversations with ChatGPT helping me cook the perfect medium-rare steak with fries. We said that the LLM outputs are the expression of the company, so it follows that they should follow already existing regulatory regimes. But what if the output is a result of a jailbreak from a user?
Then, the regulation should change. The speech should only be partially protected. If a user is going against the expressive interests of the company that deploys an LLM, the outputs that result from it should receive less First Amendment protections. The Court has encountered speech that has less, but still some, First Amendment protections in commercial speech (or advertisements). The standard with commercial speech is a substantial governmental interest in the most restrictive means available. This standard also doesn’t concern itself with the content of the targeted speech. The government is allowed to have this more lax standard, while still targeting content because advertisements traditionally carry these two qualities with them: easy verifiability and durability. People can see if what an ad is saying is true through a simple google search (plus people usually [stress on usually] don’t get primary information from ads), and a company that has their advertisements suppressed will likely find another way to promote a product in pursuit of profit.
So, when an output has been coerced by a user, it will follow the ‘substantial governmental interest in the most restrictive means available’ standard—with the regulation being allowed to be content-based.
That’s All?
That is the question Garcia raised. This series of blogs has tried to answer it. Whether the courts, when they finally get their chance, reach the same conclusions can only be speculated on. Only time—a lot of time—will tell.
Thanks for reading, Free Speakers.
References
Chaplinsky v. New Hampshire, 315 U.S. 568 (1942)
Brandenburg v. Ohio, 395 U.S. 444 (1969)
Virginia v. Black, 538 U.S. 343 (2003)
Ginsberg v. New York, 390 U.S. 629 (1968)
Week 11: What is a Reflection?
Hey Free Speakers!
After weeks of researching, writing, and refining: this project has now come to an end. Sad, I know. But, I have one final thing to offer you: some final thoughts on my beloved brain child.
The Future
It has a lot in store for us. We all know how rapid AI is evolving, but imagine all the legal questions that’ll come forth because of it! If I had more time on this project, I would’ve touched more on a meaty statute that we talked about a few times called Section 230. Oh! And liability law. Maybe I would’ve found a way to disagree with Eugene Volokh; but I kinda doubt it since he’s a pretty smart guy. Either way, it would’ve been a fun area of the law to try and parse.
Infinite Possibilities
The law can have many different answers to a single question. And it turns out that all of them can be right (and sometimes none of them)! Choosing what answers I wanted to go with in this project was difficult, and I knew it’d leave me with a lot of paths unfollowed (evidence of the many, many tabs I have open). If I had a team of 8 other legally inclined friends, I might’ve been able to rally them together to explore those avenues with me, then we could vote on which one was the best. Alas, we’re here with the best answer I could give you, and I wouldn’t have any other way.