Constructing [daybreak] - Exploring the narrative power of generative AI in a visual-novel storytelling video game
Advisor: Mrs. Bhattacharya
Internship Location: UC Santa Cruz
Onsite Mentor: Shi Johnson-Bey (game design PhD student at UC Santa Cruz)
My project seeks to explore the narrative potential of generative AI, realizing the broad idea of constructing narratives in real-time through the simplified, detached format of stand-alone short stories.
To accomplish this goal, I will divide my project into two phases. The first phase refers to the scaffolding of my project—I must build a visual novel game in Unity, constructing an architectural system of my own design to best support the degree of freedom offered by the gameplay’s time-travel themed functionality. The completion of the first phase will see a “finished” visual novel game, as well as the necessary framework for the implementation of generative AI into the game. Meanwhile, in the second phase, I must link an online Large Language Model to Unity and engage in automating prompt engineering to obtain relevant narrative output that can be implemented as additional “side quests” to enhance the final game. Lastly, upon the reasonable conclusion of the second phase of this project, I will initiate a round of preliminary playtesting to collect player feedback on the quality of the narratives generated by AI and make appropiate edits.
The final product that I envision at the conclusion of this project would be an entirely functional visual novel-style video game that incorporates generative AI to build short side narratives that the player can explore upon the conclusion of the main narrative playthrough. To fully bring this fictional world to life, I will be collaborating with Nicole Lai, a fellow Senior Project student who will help me in authoring our main narrative, while also providing me with the visuals and animations of our game.
Project Blog
Week 0: [New game]
Hello everyone, and welcome to my senior project, “Constructing [daybreak]—Exploring the narrative power of generative AI in a visual-novel storytelling video game.” In this first post, I will reveal the inspiration behind my project, unveil the plans I have made for the next 12-weeks of work, and explain the final product I envision at the end of my journey as a fledgling game developer.
01. Intro: new game = [daybreak]
As can be inferred from the title of my project, if not the title of this post, the goal of my project is to develop a video game, particularly one focused on storytelling. However, like many exciting stories in this day and age (such as the narratives within video games Honkai: Star Rail and Honkai Impact 3rd as well as web-novel Omniscient Reader’s Viewpoint and animated series Link Click, all of which have served as excellent sources of inspiration), there is a twist—the implementation of generative AI. Perhaps known better in the form of OpenAI’s Chat-GPT and Google’s Gemini, these pretrained AI models will be incorporated into the narrative creation of a video game of my own design, an AI-based game.
02. Background: Why?
One question: Why? For starters, fiction has always been at the core of my interests, from fueling my efforts to learn computer science to developing my hobby of reading. Despite my love for narratives, reading one has always stabbed me with despair over the knowledge that every intriguing story would come to an end. No story can carry on eternally; unless there is a way to continuously create content while the consumer is reading the story, every story must be neatly packaged into a beginning, a middle, and finally, the dreaded end.
However, generative AI offers a solution to this dilemma, albeit one that cannot be harnessed so easily. With the broad range of answers that an AI model such as Chat-GPT may give, it is incredibly difficult to focus AI on the task at hand to generate an appropriate and proper narrative, even more so when players have the opportunity to lead the AI off track with their own inputs.
03. Overview: But what is this project about?
So, let’s take baby steps. Emphasis on ‘baby,’ as while I would like to claim I have accumulated a wealth of experience in game development over the years, the reality is that I am new to almost everything about game development, excluding the act of coding.
Factoring in the time constraints of 12 weeks, I have narrowed the scope of my research into exploring the narrative potential of generative AI via constructing narratives in real-time through the simplified, detached format of stand-alone short stories. By lowering the narrative requirements for AI, I hope to see a similar decrease in cost of implementation, where a success only utilizes prompt engineering—merely a simple set of instructions for the AI to follow, with little additional supporting memory or validation systems.
04. Dedication Page: “Thank you for your support!”
Before I launch into a lengthy description of my plan for the upcoming weeks, I would like to introduce the other contributors to this project. First in line, Nicole Lai, my partner in crime, amazing artist and likewise a story connoisseur who has shouldered the burden of designing and creating the visuals for our game! (She has her own blog post, containing a couple of sneak peeks into the narrative design aspect of the game amongst other morsels of information I may have omitted in favor of explaining the wonders of AI.)
Furthermore, with guidance that has been no less significant, this project would not have been possible without my experienced advisors, Mrs. Bhattacharya and my remote advisor Shi Johnson-Bey.
Lastly, I’d like to offer an additional round of thanks to the beta testers that have been very vocal about their support for this project!
05. Weekly Schedule: “Long Day’s Journey Into Night”
With the hope that I will be able to fully finish building my game in Unity within the next 12 weeks (I’ve had a head start and have coded some of my stems already), I present to you, my planning:
Week 0 (what I’m currently doing): Recreate my current binary tree dialogue choice system with better architecture and quality-of-life (QoL) additions to the user interface (UI), Write the dialogue that the game’s narrative is comprised of
Week 1: Continue updating the binary tree dialogue choice system if not finished, Add point tracking to dialogue choice system, Add necessary parameters to distinguish between scenes
Week 2: Create transitions between scenes, Add necessary structures to incorporate the visual elements that Nicole has developed on her side of the project, Add system to decide main narrative ending based on point values
Week 3: Finish up the writing the dialogue for the main narrative of the game, Create a progress saving feature, Design environment grid for exploration after the main narrative’s conclusion
Week 4: Create secondary choice system and point tracker for “exploration”, Write flashback text for specific locations in “exploration”, Create revival point system in “exploration”
Week 5 (conclude phase 1 of project): Create randomizer and background scene databank for “exploration”, Incorporate exploration endings, Add failsafe text for AI, Begin incorporating Nicole’s visual elements
Week 6: Link Large Language Model (LLM) to Unity via API keys, Learn and attempt prompt engineering by manually accessing the LLM, Continue incorporating Nicole’s visual elements
Week 7: Refine prompt engineering instructions, Create a program to read JSON output from LLM, Continue incorporating Nicole’s visual elements
Week 8: Finish first tier of refining prompt engineering instructions, Automate LLM requests, incorporate LLM output into dialogue system, Export beta version of game
Week 9: Conduct beta test for the entire game, Collect survey data on beta testers’ experience/feedback, Refine tier 2 prompt engineering instructions to create short narratives
Week 10: Resolve any bugs that may have surfaced during beta tests, Implement QoL changes based on beta testers’ feedback, Finish refining prompt engineering instructions, Finish incorporating Nicole’s visual elements
Week 11/12 (buffer weeks): Resolve any lingering bugs/Finish objectives from previous weeks, Export final game, Make presentation slides, Prepare for final presentation
And that’s a wrap for this first post! I hope you enjoyed reading my explanations into the inspirations and intricacies of my project and will continue following my journey through the (still murky) world of game development. I will continue to program my systems in the following weeks, hope to see you then for the first demo!
Week 1: The [game] has already begun
Hello and welcome back to Constructing [daybreak]! Today, I will give a brief overview on how I intend to implement the visual novel section of my game, as well as showcase some of the progress I’ve made in the past weeks.
01. The World Tree
If you’ve been paying close attention to my schedule in the previous post, you might have noticed a recurring term within the first couple of weeks: “binary tree dialogue choice system.” To break down this term, let’s start from the back and work our way to the front.
As its name implies, visual novels rely on a combination of illustration and narrative text, typically dialogue between the characters shown on screen, to convey its plot. While Nicole handles the “visual” part of the visual novel, I must tackle the “novel” part—the dialogue system.
Enter the binary tree, my data structure of choice to help me implement my dialogue system. Comprised of two nodes per level at maximum, this hierarchical data structure holds all dialogue for each scene in sequential order, with each line of dialogue packed into a node (also custom created by yours truly!) and occupying one level inside the binary tree.
02. “Pebble at Crossroads?”
As visual novels also often offer some degree of interaction between the player and the story being told, my dialogue system must also offer players dialogue choices to make, with different storylines unfolding as a result of each choice. In binary tree implementation, choices are indicated as two nodes on the same level, each creating a separate branch below it, as shown in the diagram below (where each node is represented by one box, and each choice node is represented by a blue box).
03. One Way Ticket Down (For Now)
Lastly, a binary tree is easy to create and traverse. By adding line headings to my dialogue and reading in a text file containing that dialogue, I can automate the tree creation process. The tree is also capable of being traversed in real time—when loading a new scene, the tree records its current node as the first node in the tree; as the player clicks to progress to the next line of dialogue, the tree rerecords the current node with the next node in the tree. Similarly, when the player makes a choice, the tree records the current node with the node that the player selected, allowing further progression down the branch chosen.
Add in a bit of display in Unity, and we get this demo (sorry, no visuals yet!):
That’s all for today, everyone! Next week, I’ll explain what the white button that says “Turn Back Time” does (and how I implemented that), as well as more on why I choose to use a binary tree instead of preexisting software (hint: it’s all in the schedule).
Thank you for reading and see you all next week!
Week 2: Retracing Steps
Welcome back, everyone, to Constructing [daybreak]! It has been a little over a week since the last post (oops), but that’s okay because this serves as an apt segue to today’s topic—turning back time and how I implemented it in Unity.
01. (Re:) The World Tree…
If you’ve ever made a mistake in your life, be it major or minor, you may have experienced a varying amount of regret and a fleeting wish to go back in time and undo your mistake by making a different choice. In [daybreak], thanks to the “space fantasy”/sci-fi setting as well as my own objective to go beyond conventional gameplay for a visual novel game, you can.
Now if you rewind to last week for a bit (perhaps just long enough to read my last post), you’ll recall that my choice of data structure for containing the main information in my game [daybreak] is a binary tree. While the binary tree is an intuitive choice, as the gameplay of [daybreak] only allows the player two choices and offers a branching narrative (hence both “binary” and “tree” respectively), it also allows for the additional degree of freedom that going back in time provides through the use of preexisting algorithms that one can apply to the binary tree to traverse and reverse within it.
02. …and the Serpent
Typically, binary trees are traversed using a depth-first search approach—the exploration of entire branches of the tree to visit every node in the tree or until a specific node has been found. The approach can then be split into three methods—pre-order, in-order, and post-order traversal,—but I will only discuss pre-order traversal here, as it is the only relevant method that I took apart to apply to this project.
The pre-order method of traversing or searching a binary tree refers to starting at the root node and traversing down the left branch before returning back up to traverse through the right branch. Using a sample of the story in binary tree form, this is what it essentially looks like:
Due to user interaction, the pre-order method of traversing a binary tree cannot be magically thrown into the game with the expectation that it would work and I must take it apart. Instead of automatically traversing down the entire left branch, the game must wait for the user to select a dialogue option, set that option as the branch to be traversed, and ignore the other branch entirely, at least in traversal.
However, there is one more thing to take away from pre-order traversal, and that is how it reverses back up the tree after traversing the left branch. This reversing process has its own name—recursion, which refers to a function (an encapsulated block of code given a name and a purpose) that repeatedly makes a call to itself until a specific condition is fulfilled. Kind of similar to a snake that bites its own tail, only there’s a second snake inside of the first snake that bites the tail of a third snake inside it…okay, maybe a mirror inside of a mirror analogy would have been better (but I could not resist making a mythological reference given the fact that a tree serves as the basis for my game).
As I need to take input from the user to determine which node I need to reverse to, or rather, “turn back time” to, it is more intuitive to use recursion, repeatedly telling the game to go back a node up the tree until it reaches the node the user wants to return to. While a for-loop can also perform the same job, without knowing how many nodes are between the current node the user is at and the node the user wants to return to, it is considerably harder to set a stopping condition, unlike recursion where the condition is just “until the node in the tree is the same as the one the user selected.”
Of course, nothing is ever that easy. While adapting pre-order traversal itself was not an issue, connecting my code to Unity to enable user interaction is another beast altogether. But, for the sake of the healthy length of this blog post, I’ll push over my experience tackling UI (User Interface, or alternatively, Unity Issues) to next week.
See you then!
Week 3: UI - The Unending Impediments and Unity Issues in creating the User Interface
Hi everyone, and welcome back to Constructing [daybreak]! As the title implies, I have returned from the less than enjoyable journey of learning, creating, and debugging the user interface, or the visual display elements that are part of [daybreak]. But why “less than enjoyable” you ask? It’s better if you see for yourself.
01. Ad break: Version Control
But first, the ad before the video. Or in this case, the other announcement I must make before explaining my less than stellar experience with creating the user interface in Unity: after much debate and several discussions with Nicole and my advisors, Nicole and I have decided to use Unity Version Control, in hopes of not losing all of our work to poorly timed laptop accidents!
Unfortunately, due to Unity’s collaboration limitations, I cannot share a link to our project unlike a GitHub version of version control, which we ultimately decided not to use to avoid having our visual designer with no programming experience use the terminal. You’ll have to settle for the pictures that Nicole (check her blog out too!) and I link in our blog posts.
02. Binary Tree: Book to Movie
Now, return your gaze back to the screen to our regularly scheduled program. As I explained in the last two posts of previous weeks, a binary tree serves as the main data structure in [daybreak], allowing for simple algorithmic traversal and reversal. However, in gameplay, the player never sees the binary tree in action; while the binary tree traverses in the backend, the player only interacts with the buttons and text displayed on the screen—the User Interface (UI).
The UI for traversing the tree is simple enough, with only a dialogue box to display text and two option buttons that pop up whenever the player reaches a point in the binary tree where they must make a choice. You’ve seen it before in my previous posts, but here it is again:
Creating the UI for reversing through the tree is considerably more difficult, as the story that the player traversed through may be different in every playthrough or reversal. Additionally, aside from having no set path nor set path length, each node that the player traversed through must be a clickable option allowing the player to reverse back through the tree up until their chosen destination. Luckily, there are a couple of tools in Unity that can help me out—the prefab and the scrollview.
As I will not know which path the player will choose in their playthrough of the story, I have elected to create the clickable record as the player progresses through the story and traverses down the tree. This means that I spawn a prefab—a packaged game object in Unity of my creation—for every line of dialogue displayed to the player. In this case, my prefab is a button containing the line of dialogue, the character who said that dialogue, and whether it was an option or not:
Meanwhile, the scrollview is used as a handy container for my spawned prefabs. As a single scene may contain a large amount of dialogue, the scrollview, as its name implies, allows the user to scroll through the displayed list of prefabs. Add in a vertical layout group and a content size fitter to help arrange the spawned prefabs on the screen, it should look like this:
Hmm. Well, that’s not what it’s supposed to look like.
03. Wherefore art thou, Problem?
Yes, I am aware that ‘wherefore’ means ‘why.’ However, this is the real question—why is this problem, or rather several problems (that can’t be displayed very well in a screenshot), occurring?
First of all, the prefabs are clearly not being spawned in the correct position. While this can be fixed by choosing the alignment of the vertical layout group to upper center, there are two more pressing problems remaining: a) the prefabs are spawning on top of each other, and b) the scrollview refuses to scroll. Luckily, these problems are in reality one problem. Unluckily, there is a confounding variable in my UI implementation—the content size fitter.
Like its name implies, the content size fitter controls the size of the scrollview so that it can comfortably fit all the spawned prefabs. However, there are no prefabs spawned at the start of the game; prefabs are only spawned when traversing the binary tree. This means that at the start of the game, the content size fitter sets the scrollview size to 0, leading to prefabs overlapping and the inability to scroll…right?
04. AI, Part 0.2…?
Unfortunately for me, no one on the Internet appeared to run into the same problem as me. A couple hours of fruitless Googling revealed that my content size fitter, vertical layout group, and scrollview were all initialized properly in Unity, and that I should have no problems with spawning my prefabs and having my vertical layout group arrange them.
(Have you figured it out yet?) At this point, perhaps it was not the UI elements that were causing me trouble. Removing the content size fitter and manually putting prefabs into the scrollview resulted in the same display issue, finally leading me to the correct conclusion—something in the initially defined prefab was preventing the vertical layout group from automatically organizing the spawned prefabs.
As no one on the Internet seemed to run into the same problem again, I turned to a more efficient search engine: Chat Gpt. With a few quick possibilities from Chat Gpt on what might be going wrong in the prefab definition, I was able to narrow it down to a lack of a layout element telling the prefab how to interact with the vertical layout group and the content size fitter. With that issue resolved, the final UI product:
Unfortunately, while AI is involved and will be implemented into this project, today’s use of AI may not quite be what you expected. But fear not, it won’t all be like wrangling UI.
See you next week for more on the use of AI in games and what I plan to do with it!
Week 4: AI - From [Midnight] to [Mid-morning]
Hello and welcome back to Constructing [daybreak]! With the looming conclusion of the first half of this project—the construction of my custom dialogue system using the binary tree as both inspiration and a tool to organize narrative data—comes the gradual shift to incorporating generative AI. To take the first steps into this shift, we must review the progress already made in the crossover field between AI and the video game industry. Hence, I present to you my obligatory literature review, chronologically ordered by times of day.
01. [Midnight] to before [Dawn]
Despite having been featured in various games since more than a decade prior, early applications of Artificial Intelligence in video games have been limited to tailoring the games’ design to the player’s preferences. From controlling the reactions of non-playable characters (NPCs) towards the player’s choices in The Last of Us to selecting appropriately difficult puzzles that complement each player’s playstyle, AI’s primary purpose has been to accommodate the player to ensure an adaptive experience of the highest caliber.
(Alternatively, game developers themselves have employed the aid of AI in worldbuilding, notably in generating virtually endless environments such as in Minecraft; however, that is not particularly relevant when I seek to explore the interactions between AI and the player.)
02. The Moment of [daybreak]
Enter everyone’s favorite AI chatbot in 2023—Chat-GPT, the first widely-known large language model (LLM) on the face of this earth. With the emergence of Chat-GPT and later on, similar large language models, the functionality provided by AI has only seen improvement—Google’s GameNGen has the capability to not only successfully recreate a functional version of DOOM but also to dynamically adjust each frame based on a player’s performance.
But the uses of AI listed above are all general; what has been done in the narrative side of video games using AI? Not to worry—in the scant couple of years (2 and a half, specifically) since Chat-GPT’s explosion in popularity, several attempts have been made to develop a system that successfully incorporates large language models into a game’s narrative.
03. 1 hour after [daybreak]—are you awake yet?
One such approach is the Procedural Artificial Narrative using Generative AI (PANGeA). Consisting of a novel validation system, a custom memory system, a Unity game engine plug-in, and a server connecting the previous components with local and private LLMs, PANGeA serves as the primary driving force behind the narrative of the prototype turn-based detective game Dark Shadows. Generating NPC dialogue as well as descriptions of each scene’s setting and of the persona behind the character the player plays as, PANGeA relies on its memory and validation systems to respond appropriately to the player's free-form text input.
Aside from designing each NPC according to the Big Five Personality Model to best emulate human personality traits in generated dialogue, the allocation of each NPC’s memory into short-term—a cache of recently generated dialogue—and long-term—a summary of all previous interactions—provides LLMs a method to retain memory of previous relevant information, ensuring continuity and individuality throughout each dialogue exchange between the player and NPCs.
Meanwhile, to avoid the pitfalls of illegal player input, incoming player input is evaluated by PANGeA’s validation system, a set of self-reflection prompting techniques that determine whether any gameplay rules (also generated by PANGeA in Dark Shadows’ game development phase) were broken. Correcting the player’s illegal input through reprompting the player with a gentle reminder of the narrative’s setting and progress, PANGeA prevents LLMs from responding to invalid input, thus preventing the derailing of the narrative while granting more control over the narrative’s direction.
04. [Breakfast] is the most important meal of the day
However, PANGeA is a rather complex approach that is not particularly applicable to my time frame. Luckily, there exists a simpler version of PANGeA in the game “1001 Nights," in which generative AI is again assigned the position of storyteller, taking on the role of the driving force of the narrative.
While “1001 Nights” likewise implemented a simpler “self-reflection” validation system similar to that of PANGeA’s to catch illegal player input, it forgoes the custom memory system. Limiting the number of characters involved to two, the developers of “1001 Nights” simply assigned the AI to the character of the king, who also served as the narrator of the story of “1001 Nights.” As the setting of the game was based on a Persian myth, generative AI had the capability to incorporate external sources, designing its personality around the character of the folktale. With the player acting as the other character, giving limited input that is then vetted by the AI, generative AI collaborates with the player to narrate a story set within the world of the myth.
05. 9 AM, 5 hours after the optimal time to start coding
With the previous examples of PANGeA and “1001 Nights” in mind, I must decide how to wrangle incorporating generative AI into a task of manageable scale in my own visual novel game, preferably without a bundle of memory and input validation systems. Aside from following the model of “1001 Nights” closely, I may continue to decrease the cost of implementing AI by forgoing player input. However, without any additional memory or validation comes the challenge of accuracy to the characters I have established in the non-AI portion of the game—a challenge that I unfortunately will not know the outcome of until I embark on my journey of adding AI to [daybreak].
Lastly, the Unity issues in the non-AI portion will likely continue; in fact, as Nicole begins to add 3D assets and animations to the game (hopefully by next week’s post!), they may double. (I clearly expect pure chaos when animations are added.)
See you next week!
Week 5: Expanding [horizons]
Welcome back to Constructing [daybreak]! It has been a while since my last post, so let’s jump right into the updates.
01. [dawn] approaching by 12 minutes
If you remember my previous posts of how I created the binary tree dialogue system and the corresponding UI, you may notice that the elements I discussed were confined to a single scene. Each binary tree referred to the dialogue for a whole scene, and the list of spawned prefabs (buttons in the dialogue log) similarly contained only the choices selected from one scene. However, as you may have expected from the title of this post, my game has more than one scene; hence, it was time for me to tackle the transitions between scenes.
Thanks to the binary tree, simply reloading the scene in Unity using the dialogue from the next binary tree after reaching the end of the current scene would resolve this lacking function. Unfortunately, there are always exceptions in which such a solution does not work out.
Specifically, the tutorial scene follows a slightly different progression. Instead of loading directly into the next scene at the end of the tutorial scene, the player must use the turn back time button to progress to the second scene. This means that for the tutorial scene only, the turn back time button has a different functionality, while the other scene components must ignore input until the player selects the turn back time button.
While adding some additional conditions to the ending of the scene and the function that runs when a prefab (spawned button) in the turn back time screen is clicked resolves the progression issue, the tutorial scene is still unique in other ways: the starting cutscene—or rather, the newfound workaround due to the lack of a starting cutscene.
02. The State of Animation: [The sun] refuses to rise
If you have been following Nicole’s blog, you may be aware that there have been some ongoing technical issues when we attempted to fuse the parts of our projects together. As we have been struggling to import Nicole’s 3D models and run animations on said models, the animation portion (and running the animations portion) of our project has been significantly delayed.
Unfortunately, this means that some animations that we were originally hoping to accomplish will have to be cut out and instead replaced with the classic “black screen with white text.” Have an example:
Very disappointing, I know. However, we highly doubt that we will get around to modeling a spaceship, much less figuring out how to animate it crashing, so you will have to settle for my increasingly complicated UI setup. (Behind the scenes, I needed to add a fourth layer of UI elements; thankfully, my previous experience from wrangling UI counteracted the unfamiliarity I had with using the imported Unity package DOTween to make the black screen fade in and fade out smoothly, without breaking anything too badly in the process.)
03. AI: Additional Intentions
However, that same animation delay brings me to my closing point for today: how that affects my schedule for implementing AI. I will need time to write the code for playing Nicole’s animations as the narrative playthrough unfolds—a task that must be worked on in tandem with my attempts at prompt engineering AI.
Adding animations also brings up a second question—how will the code know when to play an animation? Since the beginning of the project, I have been using header tags (typically just a letter and a colon) to create a text file containing all the dialogue information (the speaker, the points, the dialogue itself). This text file, such as the example shown below, is then fed into Unity and used to create my binary trees every time the game is played.
However, adding header information for animations clutters an already overwhelming dialogue file, not to mention it being a tedious task for a human to manually go through. Hence, I am considering trying to use AI to automatically add headers to the readable dialogue document I have written.
And that’s a wrap! Potential time saving use of AI aside, next week I will be outlining the more specific plan I have currently formed for implementing AI in my narrative based game.
See you then!
Week 6: An Artificial [Star]’s Trajectory
Hi everyone, and welcome back to Constructing [daybreak]! As promised, this week’s post will explain how exactly I will be harnessing AI for narrative generation.
01. Test Launch
As briefly mentioned in my last post and not at all mentioned in my original schedule (because, surprise, the situation has changed since), in the interest of saving time, I will be attempting to use AI to header my dialogue document.
As a reminder, this is what the test dialogue document that we are currently feeding into Unity looks like:
However, as Nicole and I are constantly updating our main narrative, we need to be able to read the dialogue—or rather, we write dialogue in a format that is readable to us, and not so much to the Unity compiler. As a result, I developed a simple header system that we would be able to label the dialogue with once we finalized the narrative.
Unfortunately, our dialogue document has since grown to such a monstrous size that it is tedious and a massive waste of time to manually header the dialogue into a text file that can be read by my code in Unity. With the dialogue document containing roughly 10 scenes and each scene stretching anywhere from 5 to 15 pages, it would take us far too long to accomplish such a task.
Enter the AI, which will hopefully help to shorten this task from a matter of hours (if not days) to a couple of minutes. With some examples of the current test dialogue text files and the corresponding human-readable dialogue documents as well as the header key, I will pointedly instruct AI via prompt engineering to do the bulk of the header-ing for me, converting this task from a mindless chore to a test run of sorts. Depending on how much editing I would have to do once the AI is finished giving me its best attempt at labeling the dialogue, I will know how to adjust my prompted instructions and where to set my expectations for the following part.
02. Fueling [Stars]
So I plan to “employ generative AI in the creation of short narratives,” but what does that exactly mean? First, let’s talk about “employing generative AI.”
Specifically, I want to prompt generative AI while a player is playing the game. This means that my prompting will have to be automated—I cannot manually prompt the AI on the Internet and copy-paste its response into a text file that is fed into Unity during runtime. By cutting out myself, I am left with generative AI and Unity (and my code that runs in Unity, but I will consider it a part of Unity for now).
Luckily, Unity users have already created a solution for this sort of situation—LLM for Unity, a free Unity plugin that does not require Internet access to connect a local LLaMA large language model (LLM) to Unity. (This plugin also supports other LLMs found on the HuggingFace Github, so I also have the freedom to test the performance of other LLMS aside from LLaMA!)
03. Screaming into the [Void] (of light)
…And praying for a comprehensible answer. Once I set up an LLM inside of Unity, I can begin prompt engineering for a “short narrative.” I will define “short narrative” as a scene consisting of roughly 30 lines of dialogue between the two main characters. I will then prompt the LLM three times, each time asking for 10 lines to create the “beginning,” “middle,” and “end” of each “short narrative.”
As for prompt engineering, I will use a format similar to that of the AI-based games listed in my literature review, such as the one as shown below from “1001 Nights.” In addition to providing the setting of the story and the characters’ profile to the LLM and requesting output in a JSON file, I may try to further enhance the generated result through providing examples of the authored dialogue lines from the main narrative.
Clearly, with the addition of AI into the mix, the narrative playthrough has gained flexibility; at the same time, the uncertainty in its quality and production has also increased. While my current plan may seem simple (only “3” steps!), the chance of AI leading me astray is still significant.
Stay tuned to see what wonders (or horrors) AI will bring me for the remainder of this project, and I’ll see you next week!
Image Sources: Sun, Y., Li, Z., Fang, K., Lee, C. H., & Asadipour, A. (2023a). Language as reality: A co-creative storytelling game experience in 1001 Nights using Generative AI. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, 19(1), 425–434. https://doi.org/10.1609/aiide.v19i1.27539
Week 7: Animation’s Parallel [Universe]
Hello everyone, and welcome back to Constructing [daybreak]! Today’s agenda will consist of a quick—also the first and the last—update on how combining Nicole's visual side of the project with the parts I’ve created is going, as well as some (almost) final visual updates on my end.
01. Flowers on the Tree
If you have been following Nicole’s blog, you probably know by now that upon resolving the Unity issues, Nicole has been pumping out motion capture animation clips to run on our characters. Enter the murky area of animation where our projects overlap: how are we going to run the animations when the story requires it?
Luckily, I only have to resolve half of this problem; at this point in time (assuming this blog post comes out after Nicole’s post), Nicole has already figured out how to play animations using the animator controller. In my code, I can then reference the same animator controller and instruct Unity to play animations according to the narrative. Using the preexisting binary tree structure and simply adding additional information about which animation clip to play as part of each dialogue node in my binary tree, we can achieve this result:
Hopefully, that GIF works on your end. While the animations have much room for improvement, I will be passing back the role of animation production back to Nicole; my job in coding a script to play animations is done.
02. [Day/Night] Cycle : “I fixed it”/”I broke it”
Now, back to our regularly scheduled program of UI and Unity issues! Upon adding a new screen (as shown in the image below) to display all the scenes that the player has played through and allow them to turn back time to a previous scene instead of only turning back time within a scene, the entire game once again broke down.
Fast forward to the end of my detective journey for why the entire game bugged out, and the issue once again lies in a clash between Unity and my spawned game objects, the prefabs. Simply put, Unity clears all of my spawned prefabs every time a new scene starts; here, one can clearly notice the problem, as the functionality I am trying to add requires me to “travel” across scenes, quitting an old scene and starting a new one.
What complicates this issue further is that all of the narrative is meant to be played through in one Unity scene, and “progressing” to a different narrative scene is merely reloading the current Unity scene with the next binary tree containing the dialogue of the narrative. This prevents me from using the traditional DontDestroyOnLoad() function on my prefabs—I am not progressing to a new Unity scene, meaning Unity cannot retain game objects from a previous Unity scene, as the previous Unity scene (currently the title screen) does not actually contain my prefabs.
So, I must find a workaround. Enter more recursion; by adding a boolean (true-false) marker indicating whether I have visited a node on my binary tree or not, I can respawn all of my prefabs every time I select a scene to turn back to, as well as when Unity actually reloads the scene and begins playing the narrative again. For a better representation, see this demo below (also a GIF that I really hope plays):
That’s all for this week! As this project progresses into its final stages, the focus of the remaining blog posts will be on implementing AI. As a result, this may be the last time you get to see Kai and Vixen in all their 3D glory in my blog.
Say goodbye, and I’ll see you next week!
Week 8: Shooting at the [Sun]
Hi everyone, and welcome back to Constructing [daybreak]! This week marks my first attempts at prompt engineering, both in Chat-GPT to reformat my dialogue document as well as in LLM for Unity to generate dialogue scripts for short scenes. With that in mind, I will let the results speak for themselves.
01. Of 10 [Suns], 9 shall be shot down
As the title implies, it took me around 10 attempts before Chat-GPT successfully understood what I was asking it to do. In order for Chat-GPT to reformat my dialogue, I had to give it step-by-step instructions explaining the header system I use to automate reading dialogue files into Unity. However, merely instructing Chat-GPT on how to reformat my document was too confusing of a task for it to accomplish on its first try, as evidenced below.
While Chat-GPT grasped the fact that I had two characters it appeared confused as to what I actually wanted. Not only did it incorrectly reformat every line (turning every line of dialogue into an option), it also decided to rewrite every line, which was not something I was looking for at this point in time.
Enter the few-shot prompt engineering strategy, in which one provides the AI with an example of the result one is looking for as part of the prompt. Using the tutorial scenes as a sample for what the reformatting should look like, Chat-GPT is able to successfully understand the header system I am using:
Only, Chat-GPT is still rewriting all of my dialogue, despite my requests for it to ignore the meaning of the dialogue and to copy paste everything after the colon. Luckily, making my prompt more strict and specific by combining these two instructions into “Ignore all text after the colon,” I was able to finally get the result I desired using the prompt shown below.
02. [Dawn] Rising
In addition to throwing prompts at Chat-GPT, I also decided to try prompt engineering LLM for Unity, to evaluate the quality of writing that the local LLMs can produce. While I have yet to stop LLM for Unity from outputting into the Debug log, here are the prompts and outputs of my two short attempts:
Attempt 1:
Attempt 2:
While the LLM does capture the tone of the two characters decently well in the second attempt, there is still much to be desired in the narrative department, from a setting consistent with the main narrative to the pacing of how the story is told. However, these will not be the final prompts given to the AI; I will be spending the next week workshopping these prompts to create (hopefully) a well-paced side narrative that compliments the main narrative recorded in the binary tree.
See you next week!
Week 9: Catching [Sunrays]
Hello and welcome back to Constructing [daybreak]! This week, I bring you a quick update on my progress in prompt engineering AI in Unity—let’s jump right in!
01. Picking Apart a [Star]
In my test run of LLM for Unity’s generation capabilities from last week’s post, I noticed a couple of issues. First and foremost, due to the vagueness and lack of descriptions present in my first prompts, the Large Language Model (LLM) was not only choosing a new appearance for my characters but also placing them in a setting that was inconsistent with the theme of the game. In addition, the script that the LLM generated included its own scene header, one that doesn’t conform to the formatting used in my other dialogue texts.
Both of the above issues can be fixed by providing the LLM with more context. By adding an explanation for the setting of the game and a short character sheet (backstories, appearance, personality) for both of our characters and setting some constraints for what each character should know, the LLM learns to tone down the aggression between the two characters. Adding instructions on how output should be formatted as well as an example of what it should look like, the LLM finally returns a usable script to us.
02. A Moment [Frozen in Time]
…Except several LLM requests later, it is still the exact same script, word for word. The model’s settings needed tuning, as it clearly repeated itself too often. However, despite turning up the temperature to increase creativity, increasing top P to introduce greater diversity, and lowering the seed into the negatives to reduce reproducibility of each generated output, the LLM was still giving repeat responses.
Unfortunately, one of the problems I ran into was that the LLM I was using was too small. At only roughly 1 billion parameters (tiny, considering the fact that OpenAI’s GPT-3 has 175 billion parameters and GPT-4 is estimated to have 1.7-1.8 trillion parameters), small LLMs tended to get stuck repeating the same output for the same prompt, regardless of how I tuned the settings.
Hence, I introduced a larger LLM, a LLaMA model at 8 billion parameters, considered mid-sized for local LLMs that run without Internet on a computer. However, this came with its own share of problems—ones that I will be discussing next week, as the game is distributed for beta testing.
See you next week!
Week 10: Chasing [Fireflies] amidst an Undreamt [Sky]
Hello everyone, and welcome back to Constructing [daybreak]! As you may or may not have derived from this week’s title, I will be addressing the bugs in the AI portion that I discovered upon distributing a prototype game to my beta testers.
01. Local [Supernova]
Last week, I mentioned that I introduced a larger LLM to the game in an attempt to generate higher quality content. However, an 8 billion parameter AI model came with its own drawbacks—mainly, the amount of CPU power that it requires to run.
While I did not experience much inconvenience running the AI model on my own computer, a rather new M4 Mac, running the same AI model on one of my beta tester’s older M1 Mac resulted in a catastrophic freeze and subsequent crash. As it turns out, running an 8 billion parameter LLM model on an M1 Mac was pushing the upper limits of the hardware, naturally leading to my beta tester’s inability to play the game.
As a tiny model (1 billion parameters) produced repetitive output but a significantly larger model (8 billion parameters) rendered the game unplayable on a number of slightly older computers, I had to settle for a middle ground—a small LLM model at 3 billion parameters.
02. [Laser] point
However, as you may expect at this point, a lower parameter count typically corresponds to lower quality generation. To make up for the drop in quality from using the smaller 3 billion parameter model, I had to narrow down my prompt. Specifically, I had to remove character expressions from the prompt, as the LLM tended to swap out the character’s pose with their lines, leading to sudden gaps in the AI-generated narrative.
In addition, the 3 billion parameter model displayed a tendency to forget the ending header that tells my binary tree script when the scene is over, resulting in no binary tree being fully created. As a result, no dialogue was actually displayed despite the LLM generating a whole scene. Unfortunately, no amount of prompt engineering could resolve this problem; I simply had to add a check at the end of my original binary tree script to create the tree regardless whether the ending header was there or not.
And that’s a wrap for this week! Next week, I will be discussing the final result of this project as well as the beta testers’ opinions and feedback. See you next week!