Finger Coordination Through Gameplay: Assessing Project Sekai as an Introduction to Musical Instruments
Advisor: Ms. Klares
Internship Location: NYU Create Lab Brooklyn (Remote)
Onsite Mentor: Basel Hussein
In recent years, fewer adolescents have shown interest in learning to play an instrument, with studies indicating that the percentage of children who express a desire to learn an instrument declines dramatically as they grow older. The reason? The demand for fast and precise finger and hand coordination, and the lack of motivation to build these skills. However, there may be a solution: rhythm games! These music-based games require players to tap or press notes on beat with their music of choice in increasingly intricate patterns. While these games are primarily designed for fun, their mechanics often require fast and precise finger and hand coordination and resemble the mechanics of an instrument. This research aims to determine whether rhythm games can serve as an engaging bridge to encourage adolescents to learn musical instruments. Specifically, it will analyze how playing the rhythm game Project Sekai: Colorful Stage (PJSK) helps motivate private high school students in Brooklyn to learn to play the piano, which is one of the hardest instruments to master. Through academic research, I hope to establish the connection between learning an instrument and rhythm games. Along with academic research, I will be conducting an experiment consisting of gameplay sessions, keyboard sessions, and observations measuring students’ motivation, under the guidance of my mentor, Basel Hussein, to assess whether or not PJSK can help motivate adolescents to learn the piano. Through my research, I hope to find new strategies for learning instruments and expand research on rhythm games!
Project Blog
Week 0: Introduction
Hello everyone, and welcome to my senior project! My name is Lianna Lee, and I have attended BASIS Independent Brooklyn since 9th grade. For my senior project, I will be analyzing how playing the rhythm game Project Sekai: Colorful Stage (PJSK) can help motivate private high school students in Brooklyn to learn to play the piano, one of the hardest instruments to master. Overall, through my research, I hope to find new strategies for learning instruments and expand research on rhythm games!
For the past 3 years, I have been playing Project Sekai: Colorful Stage (PJSK), a Japanese rhythm game, daily, tapping along to the newest J-pop song on the hardest difficulty. However, I didn't just want my love for rhythm games to be "just a hobby"; I wanted to expand it into something more that can help others. After noticing the improvement in my finger and hand dexterity and the resemblance of rhythm games to instruments, it became clear how I can expand my interest.
Throughout this project, I will be reviewing academic research and conducting an experiment consisting of gameplay sessions and keyboard instructions. During both phases of the experiment, I will be jotting down the students’ expressions to measure their motivation in order to assess whether or not PJSK can help motivate adolescents to learn the piano. During my experimental phase, I will be under the guidance of Basel Hussein, my On-site advisor from NYU CREATE Lab, and Ms. Klares, my faculty advisor, who will help me analyze data and give suggestions throughout the experiment. By the end of my project, I will create a design of a prototype piano-like controller.
If you want to know more about my senior project, you can read more in my proposal. I'm excited to start my research and share my findings with you all. I hope I will be able to recruit more rhythm game lovers!
Week 1: Building the Framework
Hello everyone, and welcome back to my senior project blog! This week, I primarily focused on establishing the foundation of my project. At the beginning of this week, I started talking with my on-site advisor Basel Hussein, a researcher at NYU CREATE Lab. During our meeting, we defined my project’s goals and the key milestones that I aim to complete each week.
Meanwhile, I began reading through my course texts to establish a connection between learning an instrument and rhythm games to help set the basis of my project's research question. To start, I split my research into three parts: (1) the physical challenges beginners have to overcome to learn an instrument, (2) motivation’s role in learning an instrument, and (3) how rhythm games can help.
To identify the physical challenges that beginners have to overcome to learn to play an instrument, I read “Common Challenges for Beginner Musicians (and How to Overcome Them),” a post by Philly Music Lessons, a music school made up of music instructors. From this post, I defined that finger and hand coordination are the two key challenges that beginners have to overcome to learn an instrument.
Afterward, I moved on to the motivational aspects of learning an instrument. I read “Intrinsic, identified, and controlled types of motivation for school subjects in young elementary school children” by Guay et al. and Motivation: A Literature Review by Emily Lai to define the two types of motivation: intrinsic and extrinsic motivation. Additionally, I read “On What Motivates Us: A Detailed Review of Intrinsic v. Extrinsic Motivation” by Morris et al. to emphasize the significance of motivation in learning an instrument, as beginners need to stick to consistent practice to develop finger and hand coordination.
After researching motivation, I moved on to how rhythm games can help. I began to define what a rhythm game is and how its mechanics work, as they have been shown to develop finger and hand coordination and share some similarities with playing an instrument. I found some plausibility in using rhythm games to learn an instrument by reading “Revolutionizing Piano Practice: A Game-Based Learning Rhythm Game to Motivate and Support Learners” by Athalia et al. In this text, the researchers applied the Game-Based learning theory to an experiment where they created a rhythm game specific for teaching the piano. In their results, they found a 100% success rate, meaning that using rhythm games successfully enhanced players’ motivation for practicing piano.
After completing my preliminary research, I began arranging the materials and documents needed for my project’s experiment phase. To recall, since my project works with minors, I started drafting some consent forms for students and parents interested in participating in my experiment. I have also drafted an email to my high school’s directors requesting permission to use the school's iPads to install the rhythm game, as my experiment would need the use of multiple iPads
For next week, I plan on creating a strict set of rules that students must follow in order to take part in my experiment (i.e., they must stay on the rhythm game) and trying to recruit high school students to take part in my experiment. I will also continue to stay in contact with my high school directors to refine the prep for my project’s experiment and create the chart needed to track the high school students’ finger and hand coordination progress and their motivation.
Week 2: Progress, Refinement, and Chaos
Hello everyone! This week, I focused on refining the documents and materials (i.e., iPads and consent forms) needed before starting my experiment phase.
I started by reviewing my project’s consent forms to ensure they are clear and thorough. While reviewing the consent forms, I noticed that there were some potential risks that could unintentionally harm the participants. Because of this, I contacted my on-site advisor, Basel Hussein, to discuss how I could present the risk in a clear way to ensure the participants' safety. We also discussed how I should include a “Exclusion Criteria” to ensure that only participants without certain medical conditions would take part in the study. For an extra precaution, I also had my faculty advisor and high school directors look over my consent forms to ensure everything was fine and addressed.
To minimize and address these potential risks further, I went back and revised the strict set of rules that students must follow in order to take part in my experiment. A major part of this week was developing a clear and structured set of rules for participants, which originally only consisted of one rule: participants must remain on the rhythm game during sessions. This was supposed to help maintain consistency across all participants and improve the accuracy of my results. However, in light of the new potential risks, I made sure that participants had clear guidelines to follow to properly address these risks. For example, I require proper breaks and provide clear instructions that the student must follow before playing a song.
I also created the chart that I will use to track students’ finger and hand coordination and motivation throughout the experiment. This chart will allow me to organize my data effectively and observe any patterns or improvements over time.
Finally, I went to school to begin recruiting potential participants for my experiment. This was an important step because it allows me to know how many iPads I would need as I follow up with my high school directors. While recruiting, I also asked the participants the days and times they would be free, so I could know when to hold the gameplay and keyboard instruction sessions for my senior project.
For next week, I plan on finalizing all the documents and materials needed for my experiment phase, so that I can begin holding my first gameplay sessions!
Week 3: Communication and Finishing Touches
Hello everyone! This week, I focused on communicating and making sure everything is set for my research before I officially start my gameplay/keyboard session. After spending week 2 refining documents, creating charts, and addressing potential risks, I shifted my focus toward confirming materials and reaching out to participants, parents, and my high school directors.
To start, I met up with my high school directors to discuss the logistics and materials needed for my project. Before starting the gameplay sessions, I wanted to make sure I had enough iPads for each participant. After meeting up with my high school director, I was able to confirm that I will have access to 50 iPads.
In addition to confirming the number of iPads available, I was also able to finalize the schedule for my sessions. After noting down when the participants will be available and discussing these dates with my high school directors, I confirmed that I will be holding both gameplay and keyboard instruction sessions on Tuesdays and Thursdays.
Besides these confirmations, I also began distributing consent forms. Since these forms are required before any student can participate, I needed to make sure they were properly shared with both students and parents. For the parents, I discussed with my high school directors to figure out the best way to send out their consent forms efficiently. Based on their advice, I decided to send an email out to students and parents consisting of the consent form, expectations for my senior project, and a brief overview of the sessions.
Along with sending out an email, I also went to school to begin handing out physical consent forms to students. So far, I’ve already handed out all the student consent forms along with some parent consent forms and have started to receive some back.
Hopefully, for next week, I will receive the rest of the consent forms, so that I can begin holding my first game play sessions!
Week 4: Waiting Game, Setbacks, and Adjustments
Hello everyone! Unfortunately, this week is slow compared to the previous weeks. Since the start of this week, I have been playing a “waiting game” of sorts for the consent forms, so I haven’t been able to hold the group gameplay sessions.
For the majority of the week, I have been waiting for the remaining consent forms and sending email reminders to the participants. While I have received some back, I am still missing the majority of the consent forms, so I can’t hold the group gameplay sessions. This delay slowed progress, but it is a necessary step to ensure that my participants are aware of what my research includes.
In the meantime, I checked in again with my high school directors to make sure that all the materials (i.e, iPads) and logistics were on track. During this discussion, I ran into an unexpected problem: although I had access to multiple school iPads, I would not be able to download the rhythm game on them. This was a bit of a setback, since my original plan relied heavily on using those devices to play the rhythm game during gameplay sessions.
Rather than let this stop progress, I adjusted how I would get the participants to play the rhythm game. To the list of rules I made in week 2, I added a new instruction for participants: students will need to download the rhythm game Project Sekai: Colorful Stage (PJSK) on their phones. While this isn’t exactly what I originally planned, it allows me to continue moving forward without relying on the iPads.
Speaking of progress, despite all the challenges that came this week, I was able to hold a mini gameplay session with 3 seniors. Although it was a really small group, it helped me identify small adjustments I may need to make moving forward. Additionally, it was nice to experience what the sessions will be like in a real setting.
Overall, this was a pretty slow week with some unexpected challenges, but I was still able to make some progress. Next week, I hope to receive the rest of the consent forms and hold my first group gameplay session.
Week 5: Tap. Tap. Tap!
Hello everyone! This week has brought some exciting news… We are now officially transitioning to the gameplay sessions!
After the slow yet challenge-filled week 4, I was determined to get all the remaining consent forms in by this week. So, I began by sending out one final reminder email to the participants, letting them know that Tuesday would be the final deadline to either submit their consent forms or withdraw from the sessions.
In the end, I received 3 more consent forms from underclassmen, ranging from ages 14–16. Unfortunately, several other interested participants were unable to join due to upcoming exams and other commitments. While this does limit the size of my participant group, it gave me a clearer idea of who I will be working with and moving forward with for my project.
Now for the gameplay sessions!
In the first half of this week, I continued to hold the mini gameplay sessions with the 4 seniors participating in my senior project. I had them play the rhythm game Project Sekai: Colorful Stage (PJSK) while I tracked their finger and hand coordination progress and noted their expressions and behaviors to track their motivation. Overall, this session was no different from the one in week 4, but it did help me make additional adjustments before hosting the underclassmen gameplay sessions.
After the seniors' gameplay sessions, I began preparing the sessions with the younger participants. The structure of this session was similar to the seniors', except with a few important additions. Since this would be their first-ever gameplay session, I took time to clearly explain the strict set of rules (refer back to my week 2 blog post) and expectations of my senior project. Afterward, I did a quick demo on a random song, showing the participants the different notes (i.e., tap, flick, drag, hold, and stop notes) that PJSK had. This helped ensure that all participants started with a clear understanding of how the game works and how the sessions would run.
In addition to the gameplay sessions, I also made edits to how I tracked the participants' motivation. While noting the participants’ expressions and behaviors, I realized that the way I was tracking their motivation was vague. Originally, motivation was tracked based on behavior and expression alone: laughter meant motivation, and frustration meant no motivation. However, during the sessions, I would see someone laugh while playing PJSK, but would still constantly go on their phone after each song. Due to this predicament, I created a key that clearly depicts which specific behaviors meant increasing motivation and which meant decreasing motivation. Additionally, the key also assigned numeric values to each behavior to allow me to quantify my observations during gameplay sessions. Finally, I transferred my motivation data into a table format to further organize my findings and make analyzing data easier.
Overall, even though my participant pool is smaller than expected, this week represents a major step forward: the official beginning of my gameplay sessions! For next week, I plan to continue holding sessions while watching the trends in motivation and finger and hand coordination.
Week 6: Gameplay Sessions and Organizing Data
Hello everyone! This week, I started quantifying and analyzing my data while continuing to host gameplay sessions.
To start, I hosted more rhythm game gameplay sessions with my participants. Like last week, these sessions consisted of the participants playing the rhythm game Project Sekai: Colorful Stage (PJSK), while I tracked their finger and hand coordination progress and noted down their expressions and behaviors to track their motivation. Basically, nothing different from last week.
However, as I continue collecting more data, I started to focus on how to quantify my results into a more concise value. To clarify, my project takes into account multiple different variables, such as the difficulty of a song, the number of notes a song has, and the number of specific taps (i.e., perfect, great, good, bad, and miss) a participant may get. Although these variables help boost the reliability of my data, it also makes it hard for others to understand how I come to different conclusions. As such, I decided to calculate each participant’s finger and hand coordination accuracy.
To do this, I calculated each participant’s accuracy by first adding together the number of “perfect” and “great” taps they achieved, then dividing that by the total number of notes in the song they selected, and finally multiplying the value by 100. This gave me a clearer, numerical representation of their performance, which allowed me to easily see each participant’s progress.
However, these percentages weren’t enough. With this data now quantified as percentages, I also started working on how to visually represent my data. I began exploring different ways to plot my participants’ accuracy into a chart so that I can more easily identify patterns and track improvements over time. While I am still figuring out the best format to use, this step is important because it will help make my findings more organized and easier to analyze.
For next week, I plan to host the final rhythm game gameplay session, finalize how I will chart my data, and begin preparing to transition to the keyboard gameplay sessions.
Week 7: Creating Graphs and Finalizing Keyboard Sessions
Hello everyone! This week, I took a break from hosting gameplay sessions and instead shifted my focus to figuring out how to visually represent my data and preparing for the first keyboard session.
Unfortunately, I had to cancel this week’s gameplay session because of AP exams. However, since I had already collected a fair amount of gameplay data over the past few weeks, I decided to spend this week continuing to experiment with different ways to visually represent my data. I spent an entire day transferring my data into multiple different types of graphs to determine which type would best show my participants’ finger and hand coordination progress over time. Some of the graphs that I tested include line graphs, bar graphs, column charts, and scatter plots.
After struggling for hours trying to decide which graph best fit my data, I was able to narrow it down to two graphs: the line graph and the scatter plot. However, since I could only pick one, I asked my on-site advisor, Basel Hussein, for advice. After reviewing my data, he recommended using the scatter plot, with the participants’ accuracy percentage (refer back to my week 6 blog post) as the x-axis and the difficulty level as the y-axis. He also suggested color-coding each gameplay session so that future audience members and I can clearly see the participants' progress.
After receiving his advice, I spent the next day trying to figure out how to convert my data into a scatter plot graph. First, I used Google Sheets, as it was where I put all my data. This process was extremely frustrating, especially as a beginner, since I couldn’t figure out how to set the difficulty level as the y-axis. Even when I did figure out how, after proofreading each data point, I realized that they weren’t correct. To clarify, there was one point that was supposed to be plotted at (99.7%, 8), but instead it was plotted at (99.7%, 16).
So, I switched to Canva, as I used this site previously to make graphs. But, after spending a few hours on it, I was deeply humbled again with the same problem that I encountered on Google Sheets. Thus, I switch to Microsoft Excel. Again, the process was full of struggle, but after the help of some friends, family members, and several tutorial videos, I was finally able to successfully convert my data into a scatter plot.
Aside from working with my data, I also began finalizing the preparations for my first keyboard session. To do this, I met with my faculty advisor, Ms. Klares, to confirm the number of keyboards and discuss the logistics for the session. Since there are 6 participants in my project and 4 confirmed keyboards, we decided to let multiple participants share one keyboard. After all, since they’re just starting to learn the keyboard, so they won’t need to use all the keys. Additionally, we discussed her availability, and when the room would be available for the sessions, as well as the goals I hope to achieve by the end of the session. Overall, we were able to organize how we would like the session to play out.
For next week, I plan to host my first keyboard session and create a new chart to track the participants’ finger and hand coordination progress on a keyboard.
Week 8: Keyboard Sessions!
Hello everyone! This week, we are officially transitioning to the keyboard sessions!
Now that AP exams are coming to a close and more participants are available, I can move forward with my research! However, before I talk more about the keyboard sessions, I want to briefly talk about what I worked on this week.
To start, I met up with my on-site advisor, Basel Hussein, to discuss trends that were appearing in my finger and hand coordination data. So far, there’s a trend where the participant's last session results fall between the results of their first and second session. At least that’s the case for the seniors, since they were able to complete 3 rhythm game gameplay sessions.
To clarify, in their first gameplay session, each participant picked a low difficulty level between 5 - 18 and scored in the 70% - 99% accuracy percentage. However, in their second session, each participant challenged themselves by picking a difficulty level between 11- 31 and scored in the 50% - 96% accuracy percentage. By their last session, each participant picked a difficulty level between 15 - 20 and scored in the 69% - 98% accuracy percentage.
In addition to discussing the data, Basel also taught me how to use Google Sheets due to my dilemma from last week.
The very next day, I met with my faculty advisor, Ms. Klares, to finalize preparations for the first keyboard session. We discussed the structure and the goals that I hoped the participant would achieve by the end of the session. Also, HUGE thanks to Ms. Klares for creating the simple piano exercises and beginner songs for the participants to play during the keyboard session!! Not to mention, the hand placement and keys that the participants must know in order to play on the keyboard. These were extremely helpful as these allowed the participants to focus on learning basic keyboard movements and coordination without being too overwhelmed.
Now for the keyboard sessions!
By the second-half of the week, I had the participants play on a keyboard piano. Similar to the gameplay sessions, I continued tracking participants’ motivation by observing their expressions and behavior throughout the activity. Though, I also noted how many times the participants repeated a song for practice to evaluate their motivation.
As for tracking the participants’ finger and hand coordination, instead of continuing to track it by the number of taps they receive, like I did with the rhythm game Project Sekai: Colorful Stage (PJSK) (refer to my week 4, 5, 6 blog posts), I measured how long it took the participants to complete each beginner song. This allowed me to measure the participants’ progress in learning the keyboard over time.
For next week, I plan to continue hosting keyboard sessions while transferring my motivation data into a graph.
Week 9: Motivation and Chords
Hello everyone! This week, I focused on quantifying and transferring my motivation data into a graph while continuing to host keyboard sessions.
To start, I continued hosting more piano keyboard sessions with my participants. Like last week, I had each participant play simple songs and piano exercises on the keyboard. While they played, I noted their expressions and behavior, the number of times they completed, replayed, or restarted a song, and the time it took them to complete a song without mistakes. The only thing different from last week’s session was that I had the participants use both their left and right hands to play a song.
This adjustment made the keyboard sessions more challenging for the participants, especially since the majority of them are new to playing the piano. Though before introducing these two-handed song and exercises, I had each participant replay the same songs and exercises from their first session (refer back to my week 8 blog post). This not only allowed me to observe whether they had improved in speed and confidence while playing, but also gave the participants a chance to warm up before attempting the new pieces.
Other than the keyboard sessions, I also started quantifying and transferring my gameplay motivation data into a graph. Unlike the gameplay data (refer back to my week 6 and 7 blog posts), I had a much easier time transferring my motivation data into a graph. After transferring my data into multiple different graphs, I ended up sticking with the bar graph because it clearly displayed changes in participant motivation.
Additionally, I made another key to track motivation during the keyboard sessions (refer back to my week 5 blog post). Similar to how I tracked the gameplay session’s motivation, this key assigns numeric values to specific behaviors and expressions I notice the participants make. This allows me to quantify the participant's motivation and makes it easier for me to track.
For next week, I plan to continue hosting keyboard sessions while beginning to design the prototype keyboard controller.
Week 10: Final Session, Final Graphs, and Final Presentation.
Hello everyone! This week officially marks the end of the experimental phase of my senior project!
The final keyboard session was, for the most part, identical to the previous two sessions. I had each participant play songs and piano exercises on the keyboard piano. While they played, I tracked their learning progress and noted their expressions and behaviors to track their motivation (refer to my week 8 and 9 blog posts).
However, there was an added challenge to this final session that made it different from the previous sessions. I added two new challenging pieces to the study. One new exercise, and one new song. These additions require bimanual hand coordination (refer to my proposal).
These added pieces made the sessions more challenging for the participants. However, it also allowed them to get used to playing with both hands! Of course, before allowing the participants to attempt these new pieces, I had them each replay the same songs and exercises from their first and second sessions (refer back to my week 8 and 9 blog posts). Again, this gave the participants a chance to warm up before attempting the new pieces and allowed me to observe whether they had improved in speed and confidence as they played.
Before moving on, I want to once again give my thanks to Ms. Klares, my faculty advisor, for composing all the songs and exercises used during the keyboard sessions!
After finally collecting all my data (i.e., keyboard learning progress and motivation data), I began transferring them into graphs. Like the gameplay motivation data (refer to my week 9 blog post), I converted my keyboard motivation data into a bar graph, as it clearly displayed changes in participant motivation.
Everything went smoothly until I had to transfer the keyboard learning progress data into a graph. To put it in short, after fighting Google Sheets and Canva for hours, yet again, I was able to successfully convert the learning progress data into a row graph. I chose this graph because it clearly showed each participant's progress over time. For example, it showed how a participant spent 5 minutes on a song in the first session, yet later spent 5 seconds in their second session.
As the title suggests, I also worked on my senior project final presentation. I added all my graphs and two keys (refer to my week 5 and 9 blog posts), worked on animating my slides, talked about my methodology, added to my long list of struggles/errors, wrote about future research steps, expanded on my conclusion, reminisced about my internship experience, etc. I was making sure that I wasn’t missing anything important, yet I wasn’t yapping away in my presentation.
Finally, I also began working on my final product: a design of a prototype piano-like controller. Well, kind of. I began researching how other prototype gaming controllers' blueprints were made and designed. Specifically, I was analysing how the blueprint of the guitar-like controller from Guitar Hero was structured. After all, I didn’t want to miss anything while designing my piano-like controller. In Guitar Hero’s blueprints, they laid out two important details: (1) the controller’s appearance and (2) the function of each key, button, or switch on the controller. This was extremely helpful as it gave me a guide on how I should create my final product.
For next week, I plan to continue working on my final product while practicing for my senior project final presentation!
Week 12: Show Cleared!
Hello everyone, and welcome to my final blog post! It’s been so much fun working with one of my all-time favorite games, Project Sekai: Colorful Stage (PJSK), and sharing it all with everyone at BASIS Independent Brooklyn! Although I did stutter and blank out at times, I am still happy with how my final presentation came out.
After collecting and analyzing all gameplay and keyboard session data, I was able to work on my final deliverable: a prototype design for a piano-like controller (you can see the design in my final presentation). I believe it was a great step into introducing rhythm games as an alternative way to help teenagers learn the piano and possibly other musical instruments.
Instead of the controller looking exactly like a keyboard piano, it only has 10 different keys. Each key is assigned its own color, which was inspired by the different color keys on the guitar-like controller for the rhythm game Guitar Hero. The purpose of this was to simplify learning the piano so that players can focus on learning things one at a time. What do I mean by this? To clarify, typically when learning to play the piano (or any musical instrument), beginners tend to get overwhelmed as they’re forced to build their finger and hand coordination, learn proper piano hand posture, learn to read sheet music, and understand rhythm and timing all at once.
Instead, this controller will allow the player to first hone their finger and hand coordination skills while getting comfortable with proper piano posture in a more engaging way. The controller combines the challenging yet rewarding nature of rhythm games (and games in general) while also forcing the player to play with proper piano hand posture.
After finishing my final product or deliverable, I also worked on my final presentation. It is pretty long and has so much information. However, thankfully, I didn’t go too over the time-limit in both the rehearsal and final presentation (at least I think I didn’t).
Honestly, I did have fun with my senior project, and this has become quite the bittersweet moment for me. Yes, I did get struck with challenges left and right every single week of my senior project, but I was happy that I got so many people to try out PJSK. Plus, I learned a lot from this experience, like how not to conduct a study during/near exam session and how to use Google Sheets (you can read all about my fight with Google Sheets in my week 7 blog).
With that all said and done, I guess it’s safe to say that I have finally cleared this show!