Measuring Performance of Human-AI Teams in Complex Objective Domains
Advisor: Jaymie Silva
Onsite Mentor: Rob Seater, Jo Kurucar
AI is becoming an increasingly large part of our lives, from healthcare to household chores. Thus, one of the priorities of AI research is finding more effective methods of human-AI collaboration, which could benefit a wide variety of fields. My research is focused on one specific type of human-AI collaboration: balanced human-AI teaming within a decision-making game. In this experiment, the human player is individually responsible for making each decision, but can choose to incorporate suggestions from the AI assistant into their strategy. My final product will consist of the playable decision-making game with the AI assistant, as well as data analysis comparing the human-AI trials with human-only trials. I am working in collaboration with my advisors Dr. Rob Seater and Jo Kurucar, from MIT Lincoln Laboratory, and Ms. Jaymie Silva.
Project Blog
Week 1: Human-AI Teaming Isn't Only for Big Hero 6
Hello and welcome to my senior project blog. Today, I will discuss the background and key questions of my project, as well as the next steps in my research.
Background
AI-human teaming exists in many forms across different fields and applications—for instance, the way you might interact with a Roomba differs from the way you might interact with Siri. In general, human-AI teams can be classified into one of two categories: “balanced” and “unbalanced.” Balanced human-AI teams involve a relatively equal amount of control given to both parties, while unbalanced human-AI teams feature one party occupying a significantly greater role in accomplishing the task.
Since the majority of human-AI teaming is unbalanced (think self-driving cars and crop-harvesting robots), I decided to focus on investigating balanced human-AI teams. Despite being less common today, balanced human-AI teaming has potential to revolutionize a number of fields where unbalanced teaming would not work. A prime example of this is the finance and investment industry: many investors may be averse to the idea of completely entrusting their funds to an AI, but could benefit greatly from an AI assistant giving them suggestions about where and when to invest. To maintain this particular structure of balanced human-AI teaming (AI offering suggestions to a human decision-maker), I am basing my project on the decision-making game that my team and I created at MIT Beaverworks this summer.
Experiment Design + Hypothesis
My initial game simulates an ambulance driver in a zombie apocalypse, whose objective is to squish zombies and save humans in order to gain points. Each time a humanoid appears, the player can take several different actions: skip, squish, save, and scram (return to the hospital). Squishing humans, saving zombies, and skipping injured humans will deduct points. The actions each take up a given amount of time, shown on the in-game clock, and the round ends once a given amount of time (e.g. 12 hours) has passed. Players can store a maximum of 10 humanoids in their ambulance, and the ambulance will be emptied upon a return to the hospital.
For my project, I plan to add an AI feature that is called using a button in the game’s UI. When a new humanoid appears, the player can choose to click the button in order to see how valuable the AI considers the humanoid (from 1 to 10, 10 being the most valuable). I expect that this AI feature will increase the average player score compared to human-only trials.
Next Steps
To begin my project, I will be creating a simpler version of my game as an OpenAI Gym environment and using it to train a Q-learning agent. Once the agent has been trained and tested successfully, I will repeat this process using a different environment that is directly connected to my game (so that the observation space is derived from the current game state).
Thank you for reading and let me know if you have any questions. See you next week!
Week 2: The Coding Journey Begins in Earnest
Hi everyone and welcome back to my blog! This week, I embarked on the path toward coding and training the AI assistant that will (hopefully soon) become a part of my game. Before we start, though, I’d like to define some reinforcement learning (RL) terms that will be used later in today’s blog entry.
Environment: what the agent (AI) must navigate/solve (in this case, the game itself)
State: the current configuration that the environment is in
Action space: the set of all actions that the agent can take
Observation space: the set of all things that the agent knows about a given state
Q-Learning
I decided to model my RL environment and agent off of a sample environment in OpenAI Gym called Blackjack. This environment simulates a game of blackjack, where the agent plays against a dealer and eventually learns to create an optimized strategy. Since blackjack involves drawing randomized cards, it was the most optimal base for my custom environment, which centers around randomly generating humanoids. Using this base, I created the simple version of my game as an RL environment, where the action space consists only of 3 actions (squish, save, skip) and the observation space consists only of 3 humanoids (zombie, injured human, healthy human). I then modified the reward values for each possible action/state combination, and the environment was ready to be used.
After setting up the environment and ensuring it could run with randomly selected actions, I then began creating the Q-learning agent. Q-learning relies on the AI updating a Q-table, a matrix of values corresponding to each action/state combination, as it trains. In order to ensure that the AI continues to train instead of making a new Q-table from scratch each time the code is run, I save the Q-table as a pickle file when the round is terminated.
Github
During our meeting, my advisors suggested I set up a Github to make updates, comments, and feedback more convenient. However, this resulted in much more work than I had expected: I began by creating an empty remote repository and assigning a URL to it. Then, since terminal password authentication for Github is no longer supported, I had to learn about and create a personal access token. After trying to use this token to access and clone my newly created empty repository, I found that the repository somehow never existed. Thus, my next step is to try an alternate method of creating the repository so that I can add my code to it.
Literature Review
This week, I read MIT’s Human-Machine Teaming (HMT) Systems Guide to learn more about how I can potentially improve my game in terms of the AI’s collaboration with players. In particular, the General HMT Requirements section was useful for designing my game and ensuring that the AI assistant aids human players without obstructing their ability to make decisions somewhat independently. Some of the guidelines, involving information presentation and clarity in attention-directing, have made me reconsider the layout of my game’s UI in order to draw more attention to the AI suggestion component. I will also likely remove the time limit on decision-making in order to facilitate greater collaboration between the AI and the human player.
Thank you for reading and see you next week!
Works cited:
Mcdermott, P., Dominguez, C., Kasdaglis, N., Ryan, M., Mitre, I., & Nelson, A. (2018). Bedford, MA Human-Machine Teaming Systems Engineering Guide. https://www.mitre.org/sites/default/files/2021-11/prs-17-4208-human-machine-teaming-systems-engineering-guide.pdf
Week 3: More Coding + A Slight Detour?
Hi everyone and welcome to this week’s blog post! This week, I continued working on the AI aspect of my project, but I also returned to the original game to plan and add some new design features.
The Original Game (and what I modified)
During MIT Beaverworks, my team’s task was to design a game that could determine the extent to which different parameters, such as time limits, affected players’ morality in making decisions. To this end, we implemented moral dilemmas about which humanoids to save. For example, we added doctors and engineers, which have the ability to cure injured people in the hospital and repair the ambulance, respectively. By doing so, we hoped to add pressure against purely saving injured people and squishing zombies. However, doctors and engineers could only be “hired” in exchange for anti-zombie serums, which players could only carry 3 of during an expedition.
Recently, though, my advisors and I met to discuss how the game design could be improved upon. Rather than requiring serums to hire doctors and engineers, we decided to remove their point value, so that having doctors and engineers in the ambulance no longer counts toward a player’s final score. Instead, each doctor and engineer hired will contribute a treatment or damage point. One treatment point can be combined with one injured person for one point in the final score, and one damage point can negate the damage done by squishing one zombie (or human). This new design maintains the same system of conflicting pros and cons for each type of humanoid while being simpler to understand.
This week, the majority of my coding work involved modifying the game according to our new design. Since it consisted of a large number of files, which had been written by several different people, my task proved to be more difficult than anticipated. At this point, I have added the necessary changes to the scorekeeper.py file, which keeps track of the humanoids saved and killed, the number of treatment and damage points, the health of the ambulance, and all of the button functions. In the future, I plan to debug the seven-round mechanism of the game and continue working on the UI-related files.
Next Steps for AI
Last week, I was able to successfully create and train a RL agent on an environment representing a heavily simplified version of my game. Thus, my next steps will involve connecting the AI to my actual game. Unfortunately, this is not as simple as it sounds, since the game was not written in the format of an OpenAI Gym environment and uses tkinter instead of pygame. I expect to spend the next week reformatting my game to fit the Gym environment, which will almost certainly be a (not-unwelcome) challenge.
Literature Review
This week, I read Decision-making authority, team efficiency and human worker satisfaction in mixed human–robot teams by Gombolay et al. The study featured trials in which human-robot or human-only teams of 3 were tasked with building items from a set of parts. After completing their tasks, the human subjects were surveyed about their experiences. Interestingly, the tasks were completed the fastest during completely autonomous distribution of roles, but subjects showed no preference between semi-autonomous distribution (the robot allocates tasks to the rest of the team but not the subject) and completely autonomous distribution (the robot allocates tasks to the entire team). Though my project involves a different format of human-AI teaming, I found this study useful in understanding potential player behavior in accepting or refusing the AI’s suggestions.
Works cited:
Gombolay, Matthew C., Reymundo A. Gutierrez, Shanelle G. Clarke, Giancarlo F. Sturla,
and Julie A. Shah.(October 2015) "Decision-making authority, team efficiency and human worker satisfaction in mixed human–robot teams." Autonomous Robots 39:3 , pp. 293-312. http://dx.doi.org/10.1007/s10514-015-9457-9
Week 4: Creating the Official Environment
Hello and welcome back to my blog! This week, I encountered more difficulties than usual, as I had to create an entire custom Gym environment from scratch based on my original game. Here’s an overview of what I worked on:
__init__: Like in other Python classes, this is the constructor. For a Gym environment, it’s where the action space, observation space, and render mode (how the environment is displayed onscreen) are defined. Other optional parameters can be added, such as the variable “sab” in the sample Blackjack environment for Sutton and Barto rules, but I didn’t include any.
_get_obs: This function returns the observation space, which is useful when training/updating an agent. For my environment, the observation space is a tuple of 3 values: the number of humanoids in the ambulance (out of 10), the humanoid currently being displayed, and the amount of time remaining in the round (out of 720 minutes).
reset: True to its name, reset resets the environment back to its original configuration. For this reason, it’s called when the environment is first created and again when the training is terminated. In my case, the reset function empties the ambulance and resets the timer to 720.
step: Arguably the most important function in the entire class, step details all of the “logic” in the environment. Each time the agent chooses an action from the action space, its action is passed to step, which assigns a reward based on the state of the observation space. Step also determines whether the round is terminated and can modify variables in the observation space. For example, in my environment, each action takes a certain amount of time, so step subtracts the corresponding time from the time remaining.
I faced some minor difficulties involving debugging, since the original game’s scorekeeper file (the equivalent of step) had not been formatted in a way that was conducive to the Gym environment. One main discrepancy was that scorekeeper is able to disable buttons if there isn’t enough time left for that action, but it would be highly impractical to remove and re-add actions to the action space. I settled on invoking a heavy punishment each time the agent attempted to choose an action that it did not have enough time for, which should quickly discourage it from doing so—or suggesting such an action to the player later on.
render: Though render isn’t strictly necessary, it’s quite useful for providing a visual representation of the environment and the agent as it trains. However, render was also the most difficult function to write, since I don’t have much experience with the pygame module. I was also unable to use the original game as a reference, since it was written with tkinter, which differs considerably from pygame. At this point, the render function is the only one that remains incomplete, but I plan to use it to display the current humanoid and potentially also the ambulance capacity and time remaining.
Literature Review
This week, I began reading Sutton and Barto’s Reinforcement Learning: An Introduction. Though I didn’t have much time to read amidst all of the coding and debugging, it was interesting learning about policy optimization from a theoretical point of view. So far, my encounters with optimization have all been code-based, such as learning the best number of learning episodes to produce a good Q-table. Thus, the textbook was useful in illustrating how exactly the difference in episode count and learning rate could produce a different policy.
Works cited:
Sutton, R., & Barto, A. (n.d.). Reinforcement Learning An Introduction second edition.
https://www.andrew.cmu.edu/course/10-703/textbook/BartoSutton.pdf
Week 5: AI Training is Finally Done!
Hello everyone! The end of this week marks a milestone in my senior project: the end of the AI phase. After completing the environment that represents my game last week, I modified the AI agent that I had previously made so that it could train on the environment. Now that the training is complete, I can begin adding the AI policy to the game itself as a UI feature.
But First, Some Modifications…
To my surprise, my AI agent took over 24 hours to train last week. This was a far cry from the 15 minutes that it had spent training on the simpler environment, so I decided to investigate with the help of my advisors. When I printed the Q-table, I realized that it was much larger than before yet mostly populated with 0s. I immediately realized the issue: I had made the observation space unnecessarily specific. For example, rather than setting the ambulance capacity to a specific number, I could make it a boolean—“has_space.” Similarly, the amount of time remaining could be expressed as “high,” “low,” and “none.” After I make these modifications, I expect that the training time will decrease significantly along with the size of the Q-table.
An Update to the Render Function
With the help of the Python API, I eventually finished creating the render function. Most of the difficulty (besides figuring out functions and syntax) was in positioning the text and images in a somewhat organized format. Though render doesn’t serve much of a purpose in terms of training or gameplay, it’s useful as a visual reference:
Next Steps
The original version of my game contained an image-classification CNN which also acted as an assistant. In order to get suggestions from the CNN, players could use two buttons: “Suggest” and “Act.” Suggest originally displayed the type of humanoid the CNN believed was currently onscreen, and Act performs the corresponding action for that type of humanoid. I will be replacing the CNN with the RL agent that I have just finished training, but the buttons will still be used. Instead of suggesting what type of humanoid the player is looking at, though, the “Suggest” button will display the four possible actions with corresponding “confidence” values based on the Q-values from the completed Q-table. The “Act” button will perform the action with the highest confidence value once clicked.
Thank you for reading, and I hope to see you next week!
Week 6: RL Meets Original Game
Hi everyone! I did a lot more work than expected this week—and so did my computer, but I am reaching the completion of my first game prototype. For the past weeks, the AI and game components of my project have remained separate, but I am finally combining them.
AI Training
Throughout the week, I continued training my AI to produce a better Q table. To my surprise, training for a mere 10,000 iterations (as compared to the customary 100,000) took approximately 2 days! Later, I found that this was because the AI trained at a much slower rate when my computer was sleeping, so I adjusted the computer’s settings to make it sleep only after the program had finished running. Though this allows the training to complete after only 14 hours, it also puts a greater strain on my computer, so I have been training the AI in intervals of 10,000 instead of all at once. At the moment, my Q table is not necessarily the most accurate, but it’s enough to work with.
Adding the AI to the Game
The original game from MIT Beaverworks, which I have been working with, already contained a different AI assistant. This assistant was an image classification CNN (convolutional neural network), which identified what humanoid was in the currently displayed image and suggested an appropriate action to take. To obtain the CNN’s wisdom, players could use the “Suggest” and “Act” buttons. I have repurposed the Suggest button to offer my RL agent’s “suggestion” instead. However, to be deliberately ambiguous, I formatted the suggestion as 3 numbers, corresponding to the 3 actions that players could take for each humanoid. Here is what it looks like so far:
Here, the highest value, 9.3 for save, indicates what the agent believes is the best choice, but 8.6 for skip also appears to be a close contender. I am hoping that certain situations will cause dilemmas for players, where two or more of the values are extremely close to each other.
Making the Suggestions Make More Sense
On their own, the raw Q values (which are what I’m displaying) don’t make much sense magnitude-wise, since they simply reflect the AI’s expected reward for that action. For example, I currently have values spanning from 10-4 to 10, which can be confusing: what if the AI suggested 0.1 and 0.2 for the actions? Would that be different from 9 and 9.1?
Thus, my advisors proposed that I process them using a softmax (normalized exponential) function. The softmax function normalizes a set of numbers into a probability distribution, spanning from 0 to 1:
By doing this, I can give the suggestion values a meaning beyond just their magnitudes relative to each other, while narrowing the range of values present in the suggestions. I will most likely spend the rest of the week working on the suggestion values, as well as continuing to train my AI.
Thank you for reading and see you next week!
Works cited:
Goodfellow, Ian; Bengio, Yoshua; Courville, Aaron (2016). "6.2.2.3 Softmax Units for Multinoulli Output Distributions". Deep Learning. MIT Press. pp. 180–184. ISBN 978-0-26203561-3.
Sako, Y. (2018, June 2). “Is the term ‘softmax’ driving you nuts?” Medium; Medium. https://medium.com/@u39kun/is-the-term-softmax-driving-you-nuts-ee232ab4f6bd
Week 7: Finishing Touches + Recruiting Playtesters
Hello everyone! At the moment, I am working on the last few parts of my game before I can start playtesting and collecting data.
The Softmax Function
In order to make my numerical suggestions make more sense to the players, I softmaxed the Q values from my AI policy, which makes them always sum to 1. This is a better indicator of their relative magnitudes to each other, as they behave much like percentages:
I’m also considering changing the numerical values to a bar graph, so that players can read the relative suggestion values more conveniently. The bar graph would also improve the appearance of the game UI.
Incoming UI Changes
This week, I met with my in-school advisor, Ms. Silva, to discuss potential UI improvements I could make to my game. I found her suggestions extremely helpful, especially from the perspective of someone who has been working on this project for months and has since forgotten how it might look to a playtester unfamiliar with the game. Such changes include:
1. Replacing the clock showing the remaining in-game time with a digital clock
2. Improving the appearance of the ambulance capacity meter
3. Rearranging the buttons and features to group them in a way that makes more sense
4. Replacing the humanoid images with better-quality ones made from Unity
Recruiting Playtesters
After I finish the UI improvements, I will begin playtesting. Throughout this week, I plan to recruit playtesters as I work on the UI. Though the prospect of recruiting enough people (30-60) worries me slightly, I hope that beginning recruitment early and potentially finding volunteers at school will help me create an appropriately sized sample pool. (By the way, if you’re reading this around 1-2 weeks after this has been published, and you’re interested in being a playtester, please contact me at mchlin2018@gmail.com).
Thank you and see you next week!
Week 8: Who Would’ve Thought UI Would Be the Most Difficult Part??
Hi everyone! Somehow, this week ended up being the one in which I spent the most time on my project—and the one in which I faced the bottleneck that almost made me abandon my UI-upgrading campaign. However, I somehow survived it with the help of my advisors, Google, and staring at the code for too long, so I expect to formally begin playtesting very soon.
Custom Fonts Are My Greatest Enemy
One of the first things I decided to do was change the in-game clock to a digital one and give it a digital clock font to match. Tkinter doesn’t have a built-in digital clock font, so I had to use my own. However, I believed this wouldn’t be much of a challenge, since I just had to download, import, and load in the font to use it. Right?
Unfortunately, a recurring motif of my senior project has been “easier said than done.” Initially, I tried to use a package called “pyglet” to load the font in, but no matter what I did, the clock still displayed as the default font, not the one I had chosen. Checking whether the font had been loaded in properly yielded “True,” but there seemed to be no way of getting the font to actually render.
After hours of googling and confusedly emailing/texting my advisors, we determined that pyglet doesn’t work on MacOS. Thus, I had to use the package PIL to draw out the font (in the form of my desired numbers) as an ImageDraw object. The ImageDraw could then be converted into a tkinter PhotoImage and displayed onscreen like any other uploaded image.
Updating the clock was a little tricky, as I had to determine how to remove the image and redraw it. I also made sure to take care of special cases such as negative times. In total, working on the digital clock took almost a day and a half, which was the longest I’d ever spent solving one particular issue in my senior project.
Improved Capacity Meter
As Ms. Silva suggested, I added an ambulance image to the background of my capacity meter so that it looks like the humanoids are actually being stored in the back of the ambulance:
Though this addition was not as difficult to implement as the clock, I had to do some research to figure out how to layer the images properly. I also made use of a convenient background remover for the ambulance image, as I have found that searching “transparent” images on Google Images does not actually yield images with transparent backgrounds.
Other Changes
Besides the two main UI changes I made, I also added an HP bar for the ambulance, replaced the zombie images with better ones made in Unity, modified the Suggest button, and adjusted the format of the final score.
I look forward to playtesting, as much as the logistics of it worry me (I can’t believe I’ll have to do a crash course for Statistics as part of my senior project soon). It’s weird knowing that the majority of the coding is over now and I only have to do data collection and analysis from now on.
Thank you for reading and see you next week!
Week 9: Playtesting!
Hi everyone and welcome back! This week felt exceptionally long, even though I had already completed the coding last week. Somehow, arranging times to meet with people and have them play the game isn’t as easy as I thought it would be. Besides the struggle to find enough people interested in participating (I need a minimum of 60 for statistical significance and I’m currently at 45), I also need to coordinate with everyone simultaneously while taking into account potential last-minute scheduling conflicts.
Experimental Design
In order to study the effect of the AI suggestions on player performance, I split test subjects into two different groups: the “baseline” and the “treatment.” People in the baseline group do not have access to the AI feature, while people in the treatment group have access to and are somewhat encouraged to use the AI feature. Regardless of which group they are in, each participant will play the game five times. This mitigates the effect of luck (frequency of certain types of humanoids) on people’s scores, while also giving players a chance to empirically learn the hidden rules of scoring present in the game in order to maximize their score.
After collecting enough data from both groups, I will run two T-tests comparing players’ scores across the two groups. The first T-test will compare baseline and treatment scores for the first two games. Here, I expect a lower p-value (greater difference in average scores), since participants in the baseline group lack assistance in learning how the game works. However, over time, the baseline participants gain more information by observing the effect of their actions on their final score. This is effectively the same information that the treatment group received at the beginning, so I expect the average scores across all 5 games to be more similar.
Qualitative Analysis
In addition to the quantitative data that I plan to analyze next week, I also collected some qualitative information from the players via a post-experiment survey. After participants in the treatment group complete their last game, I ask them to rate the usefulness of the AI suggestion feature on a scale from 1 to 10. They are then given the opportunity to provide additional comments or feedback regarding the game in general. Interestingly, so far I have found that people who give the AI feature a higher rating are not necessarily those with higher scores among the treatment group. I suspect that this is because people who only use the AI briefly in the beginning to learn the game’s rules do not think it plays a large role in their game experience. However, players who rely more on the AI (but are not necessarily intrinsically better at the game) would likely tend to find the AI more useful.
Potential Problems
My experiment’s reliance on volunteer participants is one of its major flaws: I cannot fully control the players, and they lack a strong incentive to behave rationally. As such, I have already noticed that a few players in the treatment group intentionally avoid reading or following the AI’s suggestions in at least one of their games. This pattern of experimentation seems to be present in the baseline group as well, as some players deliberately choose actions that go against the instructions given to them at the beginning of the game.
Next Steps: Data Analysis + Presentation
Since there’s not much time left after playtesting, I will have to start working on my presentation and setting up the data analysis soon. At the moment, I am planning to use the SciPy package to run my T-tests and any other data analysis I might decide to do. As much as I would like to begin working on that now as a much-needed break from coordinating individual meeting times with many people at once, I’m too busy running playtests to do anything else (today I have 9 scheduled, in addition to people spontaneously setting up meetings when they’re available.) Hopefully I’ll be less busy soon…
Thank you so much for reading and see you next week!
Week 10: Playtesting (Part 2)
Hello and welcome back to my blog! I can’t believe I have finally finished both the coding and data collection parts of my project, and now all that’s left is data analysis—besides slide-making, of course.
Recruiting Playtesters: The Struggle Continues
If I’d thought finding enough people to play my game was difficult enough last week, it only became more of a challenge this week. One of my main goals was to somewhat expand the age range of my sample pool, having mainly recruited fellow high school students the week before. However, this was not easy. I unexpectedly managed to recruit quite a few teachers and administrators, through a combination of emails, in-person visits, and help from Ms. Silva, but I didn’t want to stop at only BISV students and teachers. Despite finals week occurring at many universities, a number of college students volunteered to help me, which I am eternally grateful for. (If you’re one of them and you’re reading this, thank you!)
Some of my friends also offered to help me recruit more playtesters, which had an additional benefit besides simply increasing the sample size. Though there may not be direct connections between every volunteer I know personally, I was hoping to include some people who I didn’t know and had no connection with in order to further diversify the sample pool.
Qualitative Analysis
One thing that I noticed while running playtests was that not everyone behaves rationally, even when instructed to in both the written directions and verbal introduction. I considered removing any data from irrational behavior, but I realized I couldn’t afford to and could instead get more interesting results if I included it.
I will elaborate more on the qualitative results during my presentation, but a few observations I made while processing the data include a higher rate of irrational (or experimenting) behavior in the baseline group, a tendency by some participants in the treatment group to avoid using the AI feature during at least one game, and certain patterns in behavior based on age group. Frankly, I didn’t expect to have so many interesting qualitative results to discuss, but the unpredictability of human participants can be quite prominent within a decently-sized sample pool.
Data Analysis
At this point, my Notes app is full of ideas for data analysis that were either brainstormed or suggested to me by my advisors, and I’m afraid I won’t get to them all on the presentation. In the meantime, though, I am working on figuring out how to draw the correct graphs on Google Sheets, which seems determined to misrepresent my data by using the wrong columns for everything.
Here’s a sample graph after I managed to get it to work:
For next week, I’ll be working on the statistics part of the data analysis and my presentation slides. Thank you for reading and see you soon!
Week 11: Playtesting (Part 3?)
Hi everyone and welcome back to my blog! This week, I did not continue playtesting, contrary to the name of this post; if I had, I would probably be running into major time pressure right now, despite having a very good set of data to use.
What Did I Actually Do Then?
For the majority of this week, I conducted data analysis and worked on my presentation. My advisors proposed a number of ways to analyze the data that I had collected, in order to factor qualitative results into the quantitative conclusions. For example, after running a standard T-test comparing the baseline and treatment groups for each of the 5 games, I organized the data in different ways to account for unexpected player behavior, then ran more T-tests on subsets and processed versions of the data.
Here’s a sample of the code I used to obtain my T-test results, courtesy of Jupyter Notebook:
Interestingly, running T-tests was not as simple and straightforward as I expected (“I’ll just need to write a few lines of code, right?”) I had forgotten that Jupyter Notebook requires a kernel to run the code, and that this kernel does not always work. After several “Kernel Unknown” errors and code that failed to run, I was finally able to complete all of my data analysis, which will be a key part of my presentation on Saturday.
The Actual Presentation
Besides working on the data analysis, I also finished making the slides for the presentation, with one slight problem: I had been under the impression that my presentation would be 20-30 minutes long, but I later learned that the maximum time allowed was 15 minutes. By then, I had already completed my whopping 26 slides, which I considered to be a “shorter” version of my presentation. Choosing which slides to omit will be a daunting task, since I had designed the presentation structure with every slide in mind. At least my game demo video isn’t very long…
Next Steps
For the rest of this week, I will be rehearsing my presentation in a desperate attempt to shorten it (or master the art of speaking extremely quickly but coherently; that is, the art of turning my presentation into a rap song without making it sound like one. Maybe this is what the 9th grade AP Government rap assignment had been preparing me for all along.) If you attend my final presentation, you’ll find out whether I have succeeded in this somewhat ambitious endeavor.
Thank you so much and see you either on Saturday or next week!
Week 13: Lucky Number 13 (Is It Really Over Already?)
Credits for the title of this blog post go to Mr. Rocha. Thank you to everyone who attended my presentation on Saturday! I hope you all had fun, and if you playtested for me, I hope you enjoyed finding out the game’s scoring algorithm and learning what your contribution to the project was really for.
What I Learned
This project was an amazing opportunity for hands-on learning, particularly for AI and game design/development. Prior to this, I had never made an AI myself, and the prospect of it intimidated me enough that I didn’t try to do so during MIT Beaverworks, the summer program that provided the basis for my project. Though I worked a bit on the image classification CNN present in the original game, I didn’t make any major adjustments to the code. Thus, the creation of the AI assistant for my senior project allowed me to step out of my comfort zone and accomplish something I never thought I would be able to.
The game design part of my project was also a new and exciting experience, since I haven’t had many opportunities to be on the behind-the-scenes side of games despite playing them during my free time. I enjoyed all of my discussions with my advisors about creating in-game dilemmas for players to navigate. In turn, these discussions also helped me notice similar instances of dilemma-crafting in the games I played.
Of course, I also gained practical Python and Github experience while working on the project, which will be useful for future projects even if they don’t involve the same type of content as this one. The extensive debugging I had to do also helped me recognize common mistakes that I might make and how to avoid them in the future.
Acknowledgements and Sidenotes
I would once again like to thank my external advisors, Rob and Jo from MIT Lincoln Laboratory, who helped me with the AI, UI, and everything else I had questions with, and also trained my AI on their supercomputer in order to achieve the necessary number of iterations (2.5 million) in a short amount of time. I would also like to thank my internal advisor, Ms. Silva, who provided many of the suggestions that inspired my current UI layout. My gratitude also goes to Lydia from MIT Lincoln Laboratory for helping me with my UI issues, and the 60+ playtesters who took time out of their schedules to play my game (and for some, attend my presentation).
Dr. Wang’s Game Theory class also unexpectedly helped me with my project. During T2, we had a group project to design an economics experiment based on a game of our own creation, then conduct the experiment by having several classes of Basis students play the game. Through this project, I learned many skills that were useful for my senior project, including the specifics of designing and conducting game-based experiments and what types of data analysis to use in order to best interpret the data collected.
Next Steps and Future Research
Though my research brought me some answers that I had been looking for, it also opened the doors to more questions, some of which I outlined in my final presentation. For example, I now want to know whether my AI tool will help people who have already had experience playing the game, as well as whether it could hamper players’ ability to learn. To find the answers to these questions and more, I may conduct more “sequel” experiments in the future. I also had a few ideas on how to improve the game’s UI and make it a better experience for players, starting from making the setting more realistic and adding more controls, particularly for movement.
Recently, I was also informed of the possibility of presenting my senior project at Beaverworks 2024, which I’m thrilled about. I would love to showcase the experiment that my Beaverworks project inspired me to make, and hopefully inspire more students this year to do the same. So, if you’re attending Beaverworks for the Serious Games and AI section this summer, I might see you in the near future! (I’m not sure how to phrase this without making it sound ominous).
Thank you so much for accompanying me on this journey of zombie-squishing and human-saving, and I hope you had as much fun as I did when I wasn’t fixing bugs. As per Simon the AI’s suggestions, I have to “scram” now. See you soon!