Designing [daybreak] - Harnessing 3D motion capture to enhance the immersive feel of a visual-novel storytelling video game
Advisor: Mr. Ear
Internship Location: Remote (company: Intelligible Games)
Onsite Mentor: Connor Fallon, game designer at Intelligible Games
My research will be primarily centered around observations of the shapes, forms, quirks, and miniscule movements that make life believable. To accomplish this, I will be utilizing the motion-capture softwares of Rokoko Vision and possibly Remocapp to record human movement and animate the skeletons I have rigged up in my 3D-rendering software (i.e. Blender and Unity). Due to my own lack of prior experience in these specific softwares that I will be using for this project, however, I must begin by researching up on guides and online tutorials in order to properly utilize all these softwares.
Once the project arrives at its conclusion, I plan to have produced every aspect of the visuals of a video game—fully 3-dimensional character designs, character animations, and various rendered backgrounds—that I will be working on alongside Cindy Liu, a fellow Senior Project student and the programmer of this game. This will involve recording multiple videos of real-life subjects via Rokoko Vision (and possibly Remocapp), which will then be imported onto my rigged models in Unity. With the help of my collection of data via motion capture, along with any outside research I may perform to shed some light on the physics of realistic animation, I strive to have my resulting visuals create life as convincingly as possible within the confines of my available softwares.
Project Blog
Post 1: Introduction: The [day breaks] on a New Journey
Mornin', everyone, and welcome to Designing [daybreak]- Harnessing 3D motion capture to enhance the immersive feel of a visual-novel storytelling video game!
A staggering part of my childhood has been spent stringing together bits of original character designs and game storylines within the margins of my notebooks and the fuzzy edges of daydreams; to watch this all come together in a more tangible form feels almost surreal. In this introductory post, I will guide you through all the key points of our journey: my initial sources of inspiration, a brief overview of the game’s planned storyline, a grateful nod to everyone who will be accompanying me on this adventure, and the course I have charted throughout all 12 weeks of this Senior Project.
I. Intro: Just Before the Sun Rises
This visual novel game, which I stretched and molded and pieced together alongside fellow Senior Project student Cindy Liu, was conceptually inspired by the similarly futuristic world of Honkai: Star Rail. Design-wise, my characters’ outfits were considerably influenced by the visuals of Honkai Impact 3rd and the web-novel The King’s Avatar. (On the programming side of things, Cindy’s got her inspirations covered in her own blog, which can be found here!)
II. Brief Overview: A Glimpse of Daybreak
The setting of our game is hidden quietly away on the dark side of the moon: the frozen ghost city of Calais, a failed time travel experiment from long ago. Our two characters are Kai, the synthetic human at the core of the entire experiment, and Vixen, a government agent tasked with retrieving the original time manipulation technology. Little do either of them know, Vixen’s entrance into this frozen city is what triggers its total defrosting—and the player of the game, controlling Kai, must make a number of critical decisions as our two heroes navigate an increasingly life-threatening environment.
As the co-creator at the forefront of the visuals, I will be weaving 3D animation into this visual novel game via motion-capture data collection and 3D rendering softwares, even if such softwares’ capabilities are severely compromised by a paywall (or lack thereof.) In addition, I plan on utilizing softwares of a nature and scope that I have had absolutely zero prior experience in, so a good portion of my project may be dedicated to learning the inner workings of these new softwares—not to mention the biggest thorn in my side, my staggering budget of $0!
III. Dedication Page: To All the Heroes in This Story
This project would have sputtered out and died long ago without my partner in crime and greatest source of inspiration, Cindy Liu. Furthermore, none of this would have been possible without the support of my wonderful faculty advisors, Mr. Ear and Ms. Bhattacharya, along with my remote advisor Mr. Connor Fallon. Last but definitely not least, a heartfelt thank you to all the classmates who have already eagerly lined themselves up as future beta testers for [daybreak]’s final product!
So, reader—come sit by my side, and we will watch from afar as the sunrise slowly melts a city frozen in time.
IV. Weekly Plan: The Day’s Journey
(Week 0: Design the two main characters of the game’s storyline in 2D and then map out their models through T-pose character turnarounds for each)
Week 1: Import the 2D character turnaround (from “Week 0”) for our character #1, Kai, into Blender—in other words, transforming what used to be a completely flat outline into a 3D model.
Week 2: Import the 2D character turnaround for our character #2, Vixen, into Blender.
Week 3: Rig the 3D skeletons of both Kai and Vixen in the separate software of Unity (where I will be later incorporating motion capture directly through the Rokoko Vision add-on); begin importing the un-animated models into the combined Unity project Cindy and I will be using to merge our two realms together
Week 4: Begin incorporating “layer 1” of my motion capture/animation—shifts in posture and the entire upper torso in general—for character #1, Kai
Week 5: Begin incorporating “layer 1” for character #2, Vixen
Week 6: Continue incorporating “layer 1” of my motion capture/animation—subtle arm and hand gestures—for both characters Kai and Vixen, in addition to wrapping up the details of the first layer as a whole
Week 7: Begin creating 3D backdrops for the various different settings of the game’s final product
Week 8: Attempt to incorporate “layer 2” of my motion capture/animation—facial expressions—for character #1, Kai (here is where other softwares like Remocapp may come in, or I may have to manually animate these finer motions without the help of motion capture softwares)
Week 9: Attempt to incorporate “layer 2” of my motion capture/animation for character #2, Vixen; begin conducting beta tests for the entire game alongside Cindy
Week 10: Finish creating 3D backdrops for some of the later parts of the game’s final product (the AI-generated text portions)
Week 11: Wrap up all remaining animations and finish connecting all my visual elements with Cindy’s programming side of things
Week 12: Begin practicing for final presentation (and coordinate accordingly with Cindy for our respective parts to cover)
During this first week, I’ll be plunging headfirst into the world of 3D modeling to bring our first character, Kai, to life. Hope you’ve enjoyed what little sneak peeks I’ve prepared for you this time around, and make sure to stick around to witness the unfurling of our lonely little city!
Week 1: A [very] Cloudy Sunrise
Rise and shine, everyone—hope you're excited to watch the first of my many inevitable catastrophic mistakes unfold before your very eyes! This one's quite hilarious to look at in retrospect, but I will admit I was close to pulling out every last strand of my hair when it happened.
I. How it was Supposed to Unfold
As listed out in my syllabus, my first week was to be fully dedicated to modeling the first character, Kai, in Blender. This all started out smoothly enough—I found an online humanoid model that came with a full-body rig, learned a new Shrinkwrap tool to build the first layer of the character's clothing, successfully troubleshot and smoothed out a handful of other issues like clipping between adjacent meshes—and then I buckled down to tackle the unruly beast that is face sculpting.
II. Stand by, Technical Difficulties
The online model I used did already come with a perfectly fine male face, but it was far from the visual style both Cindy and I were going for. Here is where I entered Sculpt Mode for the first time in this modeling process and attempted to inflate certain parts of the face in order to achieve a less realistically chiseled look...only to realize the Inflate tool was not working at all. Nothing was happening, nothing was "inflating," and I was left staring blankly at the still-frustratingly-chiseled face before me.
I soon had the brilliant idea of applying the rig modifier that was currently connected to the model (since models in Blender cannot be transformed in certain ways unless all previous modifiers/transformations are officially "applied") and promptly came face-to-face with what certainly was the ugliest face I have ever seen. A very stunning photo is attached!
It turned out that all that initial clicking and fiddling with the inflate tool on the original mesh, with its modifier still unapplied and its subsequent transformations appearing to have no effect, did actually register. As soon as I hit "Apply" on that rig modifier, the results of all my previous efforts to sculpt the face suddenly became visible all at once and revealed a face that greatly resembled the aftermath of a serious allergic reaction to shrimp, or a particularly violent fist fight. Of course, I could do nothing but panic—I had clicked and fiddled enough that Ctrl+Z alone was not enough to revert my mesh back to its original chiseled form—but once I realized just how the problem arose, I began sloppily going over all the inflated portions of Kai's beautiful face with the Deflate tool. I have yet to fully restore the face of a human in good health, but also attached is a picture of the progress I have made in calming down the disaster of an allergic reaction my model had earlier.
III. The Day's Journey, Recalibrated
Unfortunately, this does mean my modeling of Kai may bleed into Week 2; however, if you have been paying close attention, you will realize that I no longer need to dedicate an entire week to rigging my skeletons! The Internet has been very generous and has saved me the trouble of separately creating rigs in Week 3 of my original syllabus, so I now have an extra week to work with. I will be using this precious time to get the hang of all this sculpting business—a far more vicious beast than I once judged it to be—and add the outer layer of futuristic hi-tech armor onto Kai's model. (Also attached is a 2D drawing of what Kai's final model should look like!)
Well, that was far from a happy start, but I am definitely glad I learned this fundamental lesson of applying all modifiers before adding further transformations this early on. Enjoy my very flattering pictures of Kai so far, and see you in the next one (hopefully without my model going into anaphylactic shock or sustaining catastrophic facial injuries again)!
Model used: Frederisch H. (2024). Slender Male Mannequin Blender Source (3D Model). Turbosquid. https://www.turbosquid.com/3d-models/slender-male-mannequin-blender-source-2211048
Week 2: The Clouds Melt [beautifully] Away
Hello and welcome to a slightly less eventful episode of I Try To Figure Out A New Software (it is worth mentioning, by the way, that all the softwares I will be using are new to me)! This week, I’ve been chugging along quite smoothly in my character modeling process, and yet…there are still worries lurking in the back of mind.
I. The Shenanigans I Have Been Up to
As promised in my last blog, I’ve been finishing up the model of character #1, Kai. Unfortunately, due to the rather grave injuries his face sustained last week, I decided to perform a drastic surgery and replaced his head with that of an entirely different online 3D model. Although I definitely could have restored his face to some semblance of good health through just sculpting and fiddling around, the vision I had in mind was of a far more contrasting style anyway. Thus, I whisked off the head of a character from a game a lot of you readers may be familiar with—Genshin Impact—and slid it onto the shoulders of my decapitated original model.
Aside from all this face mischief, I got to work dressing Kai up with his clothing and armor, making sure to add a metallic sheen to the armored parts (aren’t they cool!) However, all these additional items and accessories are no longer attached to the original skeleton…
II. The Shenanigans I Will Soon Be Up to
…which brings me to my worries.
I did mention previously that the online model I downloaded comes with its own rig—but getting a model’s clothes to move along with the rest of the body, I discovered, is a lot more complicated than I thought it was. I must rig the clothes to a separate armature, then animate both the body and clothing armatures at once; fortunately, I believe I can simply duplicate and reuse the armature that already exists for the body.
Furthermore, I'll soon have to begin merging my visual elements with Cindy's code via Unity and Github—and it should come as a surprise to no one when I admit that I have never touched either of these two softwares before! Once Cindy and I start combining our two separate halves into one project file, it is inevitable that we will encounter such nightmares as version control issues and totally foreign user interfaces. Safe to say, readers, this week’s placid uneventfulness will not last long. Shame…
As for more of my upcoming tasks, I’ll be sure to make some modifications to Kai’s current hair and facial features before this week ends (just so he doesn’t keep looking like a Genshin Impact character), after which I’ll promptly transition to modeling Vixen. By the end of next week (Week 3), I should have both characters fully clothed and modeled—prepare for loads more glamor shots!
Model used: Frederisch H. (2024). Slender Male Mannequin Blender Source (3D Model). Turbosquid. https://www.turbosquid.com/3d-models/slender-male-mannequin-blender-source-2211048
Week 3: A [storm] Brewing Beyond the Sky
Hey everyone, aren’t you all stoked to see me run headfirst into yet another immovable wall? Although I have been chugging along considerably smoothly upon the path I laid for myself last week, I am fast approaching a gaping pothole in the road from which there is no escape—but only if I continue forward with the same plan I have been following thus far. Come, watch me twist myself into a hundred knots as I confront more of my worries!
II. So, Have I Done Anything Right? (Surprisingly, Yes)
Before I tackle my incompetence, here’s the work I actually have to show for this past week. I finally shifted my attention to Vixen, once again using the same rigged humanoid model as Kai as a foundation to bring together the rest of the character. As you can see below, I have successfully laid down all of Vixen’s clothes and accessories from head to toe.
What do you think of my handiwork? I must confess, I personally really like the head.
II. Is a 20-Billion-Dollar Company After my Head (and the heads of my characters)??
The answer, before you all panic, is no. Hopefully not.
It appears that the official Genshin Impact character model I used for modeling Kai last week has quite a couple of clauses attached to its usage. Thanks to Cindy Liu, my partner in this collaborative project, we discovered from Hoyoverse’s Terms of Service that I cannot “redistribute or disassemble the parts [of the model] to use them in remodeling other models”—which, of course, I did last week. Remember the whole head-swapping fiasco from my previous blog?
Despite the fact that neither of us are planning on using our final game for any commercial purposes, the realization that a globally sensational BEAST of a game company could potentially be after us (and the school) is really quite sobering. Thus, after much discussion with each other and our advisors, we have devised a far safer route—that is, using a different online model of an anime-style head that happens to have none of the pesky caveats that comes with using the official models of Hoyoverse itself! Unfortunately, this does mean I will have to carve into a good amount of my time next week to give both Kai and Vixen new heads. With each week, I appear to lose more and more sense of the original schedule I laid out…
III. Next Week’s Path That I Definitely Will Not Steer Sideways From
As for the next chunk of my project I will be tackling next week, I plan on finally incorporating the actual animation aspect (after I finish giving both characters back their heads, of course). Thus, I may include a good bit of lit review in my next blog post concerning several rather interesting studies I’ve discovered on animation and motion capture. See you in the next one for a much-needed escape from all the 3D modeling headaches we’ve had thus far! (Haha, get it, because they're missing their heads...)
Models used: Frederisch H. (2024). Slender Male Mannequin Blender Source (3D Model). Turbosquid. https://www.turbosquid.com/3d-models/slender-male-mannequin-blender-source-2211048
MarFab. (2023). Anime Head Base Mesh (3D Model). Sketchfab. https://sketchfab.com/3d-models/anime-head-base-mesh-58e291934379438584012d09faefedf6
Week 4: Our First [shaky] Steps Outside
Hey, everyone—I may have bumped up all of your hopes a bit too much last week, because I unfortunately do not have very much to show in the animation department yet (or, more accurately, none at all). There were an unprecedented number of fixes and adjustments I had to implement in both characters before being able to properly begin animation. I have, however, come prepared with a collection of studies relating to the motion capture I will ideally be embarking on by next week—so strap yourselves in and get ready for an onslaught of literature review!
I. Okay, Well, If Not Animating, What Did I Do?
Apologies again for not having begun the promised animation already; I had spent the bigger half of this past week figuring out hair designs for both characters (glamor shots attached below!), then spent the remainder of the days struggling to fix the rigging for the characters’ clothes and faces.
So, remember how I said that rigging wouldn’t be too much of a problem since my online model conveniently came with its own armature? I would like to formally eat my words now, every single last one of them. Rigging the clothing was easy enough—I was able to use the same armature connected to the body to guide the movement of the majority of both characters’ garments—but all the additional accessories proved far less simple to attach to the rig. When rigging any part of a 3D model to an armature, it isn’t quite as simple as assigning one segment of the object to one bone in the armature. Objects need to be “weight painted” to fine-tune and properly dictate their movement, something that applies a lot more to clothing/outerwear than to the bodies themselves. Thus, the problem with applying the same weights as the body to the accessories attached to the body is that…well, accessories don’t quite move like the rest of the body does.
It’s easier to simply show, through pictures, how disastrous the end result can be:
Thus, I’ll spend what little I have left of this week trying to get the hang of weight painting so that I don’t simply have to use the same weights from the body to the rest of each character’s model. After smoothing out these problems, I should be free to finally begin the exciting part of my journey. Onwards!
II. The Aforementioned Lit Review
Before even beginning this entire project, I had stumbled across a handful of rather intriguing articles that echo some aspects of my own animation-related goals.
One article that fortunately touches upon one of my largest dilemmas—my inability to access expensive equipment—attempts to carry out traditional motion capture without markers. Having markers fixed onto the body, the article claims, actually impedes a subject’s movement and disrupts what would have been natural human gestures. Strapping these subjects tight with full-body suits and attaching rigid markers all over their figure diminished the accuracy of recorded data when capturing whole body movements, whereas allowing subjects to keep on their casual wear and comfortable walking shoes contributed to a more natural side-to-side movement of their body’s trunk. Thus, by adopting markerless motion capture for movement analysis, this study considerably reduced the time needed to prepare the subject, provided more authentic assessments of human movement (especially in terms of the human gait), and streamlined the overall process for usage in such fields as clinical practice. Upon discovering the closeness of this article’s premises to my own situation, I was relieved to find that my own lack of professional equipment might not turn out to be as large of an impediment as I once believed.
Another eye-opening source explores a question untouched by the previous few: the possibility of eliminating veterans’ phantom limb pains by digitally animating the missing parts of their body convincingly enough to deceive their brains. In this study, researchers introduced veterans to the following “immersive experience”: using motion capture cameras to record the movement of an existing limb (for example, an arm) as the subject walked on a treadmill, mirroring that arm across the body to “replace” the missing arm, then projecting the subject’s figure in the form of a bodysuit-fitted avatar onto a screen in real time. By recreating this image of a reconstructed, fully intact body before the veteran, the projection onscreen and the veteran’s own movements appear almost to sync, to provide the illusion that the avatar and the veteran are one. Although completely outside of the scope of my own project, this experiment gives me a glimmer of hope that verisimilitude, to the extent of successfully tricking the human brain, is very much possible with 3D motion capture.
III. Itching to Actually Start the Journey?
Well, I guess I’ve had you all wait long enough…but fret no longer, since I truly cannot think of any more unprecedented roadblocks preventing me from formally beginning the animation process next week (one week behind schedule, but hey, could be worse). Brace yourselves for some living, breathing skeletons flailing their way across your screen soon!
Week 5: A [deluge] of Torrential Rainfall Upon Us
Hello, all—hope you’re ready to be absolutely bombarded by photos in this week’s (slightly late) blog post, just like how both Cindy and I were bombarded by error upon hopeless error while we were trying to knit our projects together! Let’s just say this: getting our animations going might be the least of our worries right now.
I. Unity—the Software Where Two Worlds (Struggle to) Unite
At the start of this week, I had only one simple goal in mind: start animating. Did I hit it? Well, technically, no…but I did hit some other goals I didn’t previously know I had!
With an optimistic heart and absolutely no idea what to expect, I opened Unity and attempted to navigate to the shared project Cindy had already begun, only to run smack-dab into…
…this.
After restarting my computer a handful of times (and Unity about a hundred times more than that), uninstalling/reinstalling different Unity versions in an attempt to match Cindy’s own version, banging my head (figuratively) against countless Internet troubleshooting forums, and sweeping multiple other apps on my computer into the trash in a desperate attempt to clear much-needed storage, I finally managed to open and run the project in a Unity version completely separate from Cindy’s—6000.0.42, as opposed to 6000.0.39.
Even then, I ended up encountering several slightly buggy aspects—dialogue lines coming from the wrong character, slow response time, etc—in the project that hadn’t initially been present on Cindy’s side. All of this could very well just be a result of lag from my end (not surprising, considering how large a role my computer’s sheer incompetence has played so far in our struggles), but it still poses an issue of mild concern.
II. What Trial Awaits Me Next?? (Oh no)
Reader, you must be expectantly waiting for the animation snippets that should theoretically be coming right up—in which case I must promptly dash your expectations against the rocks.
The next order of business after successfully opening the project in Unity, of course, was to import my already-finished models into this new software. As my introductory paragraph to the blog must have indicated, however, this process did not go nearly as smoothly as I had hoped.
After importing my rigged model as an .fbx file into Unity and struggling for an embarrassing while to actually make the model appear (turns out, I had been uselessly fiddling with buttons and error messages when all I had to do was drag the model from the Assets folder into the scene), I came to several disheartening realizations:
1. One leg of my pants had gone curiously missing (censored to keep Kai's dignity),
2. One sleeve of my shirt had disappeared in a similar fashion,
and 3. I was looking at the lifeless face of a ghost (bit of a jumpscare warning here).
3 was a simple enough solve—the images I had originally texture-painted onto my Blender model hadn’t been imported into Unity along with the model, so I only had to reattach those images to their corresponding materials.
1 and 2, however, proved to be far more baffling.
Following the instructions of a website Cindy helped me dig up, I recalibrated my model’s axes and recalculated the outside faces (or normals) of its mesh, producing:
Patched-up pant legs and shirtsleeves, but a wholly different missing section to my clothing!
III. I’ve Been Bested…by a Harmless Computer Software
At this point, I was quite beaten and utterly at a loss as to how to tackle each and every new error that popped into view. It’s almost as if every time I tried to put a foot forward into the (actually exciting) realm of animation, Unity stepped in front of me with arms crossed and declared, “No— nope— don’t think so—”
Well, that wraps up this week’s rather upsetting updates! This upcoming week? I’d better clear away all those errors that just determinedly KEEP COMING BACK, and only after the dust has settled around this entire Blender-to-Unity transition can I finally, finally commence animation.
‘Til next time…
Week 6: A [grueling] Uphill Climb
Hello everyone and welcome to our official “start” of animation! Here’s to another episode of “errors I’ve fixed + errors I’ve created” where, like usual, the latter category unfortunately outnumbers the former.
I. A Fresh Face
As you may remember from last week, my character’s model wasn’t looking too hot (inexplicable disappearing chunks in the clothing, a ghastly pale material for the face, etc). Thankfully, however, I was finally able to patch him up and bring the color of life back to his face with the help of one of my external advisor’s colleagues.
Here he is, glorious and whole and FINALLY incorporated into Cindy’s game scenes:
II. No Time To Lose: Next (Baby) Steps!
Take a deep inhale with me as we plunge headfirst into the completely foreign waters of animation…and immediately smash into a wall of rocks. (It truly seems as if nothing is fated to go smoothly in this project).
The first task I attempted to accomplish was record the motion capture data—a silly choice, really, considering I hadn’t even figured out how to link Rokoko Vision to Unity, or how to transfer recorded movement onto my model’s rig. Thus, after performing several basic jumping jacks in front of my computer’s webcam and exporting the resulting clip from Rokoko Studio, it was to my utter dismay that I realized I had absolutely no idea how to link what I just procured to Unity. It turns out, a Rokoko-Unity plugin AND a free version of Rokoko Vision do exist—they just aren’t the same thing. Even though I am able to record whatever I want using Rokoko Vision’s subscription-free motion capture, I cannot stream movement directly from Rokoko Studio to my model in Unity without dropping a couple of dollars.
So…I did what any reasonable person in my situation would do. I refused to pay!
We started this project with a $0 budget and we’re going to STICK with it.
The solution, however, appeared to be an easy enough fix: take the clip, insert it into the character’s animation controller, and hope with what little hope I have left in me that the movement will map perfectly onto my model’s armature (spoiler: it didn’t).
There I was, pressing the play button and eagerly waiting for my character to suddenly begin performing perfect jumping jacks all over the screen—
—only to watch, in horror, as my animation reached the first frame and promptly froze in place.
Hopes dashed and heart in my stomach, I began combing through every single Internet resource even remotely related to my problem in a futile attempt to bring Kai’s jumping jacks to life. At this point, I still have absolutely no idea what mysterious error is barring my model from executing the entire animation, but I do have some troubleshooting plans in place for the upcoming week.
III. Let My Guy Move!!!
In the event that my recorded clip’s movements are unable to map onto the armature of my character’s model (which is very likely), I have decided to test the recording with other basic humanoid models—perhaps some of Unity’s very own sample models, or other “failproof” ones from online—to see if it isn’t just an issue with the specific rig I downloaded. If all these different models continue to fail the jumping-jack test, I’ll wipe everything model- and animation- related from the Unity project and reimport everything (hey, it worked once before).
And if that fails…well, let’s hope it doesn’t. (We’ll cross that bridge when we get to it, but for now I’d like to hope I can find the problem with the simplest possible methods).
See you in the next one, and remember, kids: don’t try to learn 3 new softwares at once and expect any less than a thousand different errors!!
Week 7: Step by [baby] Step
Hey all, do I have good news for you this week! Life is being absorbed into the bones of my model, little by little, and movement has finally begun flashing across my Unity screen. From here on out, the path should be relatively less shrouded in fog!
I. “Have You Tried Turning it Off and Back On Again?”
Ah, yes, the classic IT advice. “That wouldn’t work!!” was my incredulous response when Cindy suggested I should just wipe the project clean of anything model- or animation- related, then reimport everything and start again. It did, however, have some reason to it: to troubleshoot the source of the problem, I’d originally tested out the same model and animation clip in a separate Unity project…and it had worked just fine!
Thus, I dutifully “rebooted” my part of the project by reimporting everything even remotely related to the animations and the rigs. Still completely in doubt, I pressed the “play” button on the project once more—and my eyes widened in disbelief as Kai promptly performed jumping jacks all over my screen. They weren’t the most fluid and attractive jumping jacks ever, mind you, but they were THERE.
Well, I knew what I had to do now!
II. Let the Games Begin
At that point, the rest of what I had to do seemed deceptively simple: catalogue all the different types of emotions present in the dialogue Cindy and I had been working on, determine different poses/movements for each of those emotions, and motion-capture those poses onto the skeleton of my character.
(Foolish of me, really, to hope that anything could be “simple” over the course of this project.)
As I have already clarified several times across my blogs, I will only ever have access to the free versions of all the softwares I plan on using. Now, Rokoko Vision doesn’t provide the most top-tier free services that I’ve ever seen in a software…and that is putting it generously. As I was attempting to film a straightforward idle animation—a rather subtle movement of arms swinging back and forth to imitate exaggerated breathing motions—it was to my utter dismay that I realized Rokoko’s AI-powered tools are pretty much incapable of capturing stillness. In other words, even the parts of my body that weren’t moving, like my legs, would still wiggle around and gyrate in place (not very good for subtle movements, you see!)
I did eventually manage to capture a clip with minimal unnecessary gyrating, but the movement is still far from ideal. Kai looks like he’s about to combust on the spot at any moment with the way he’s noticeably vibrating…but that isn’t even the worst of my worries now. Since my research question was initially intended to investigate the feasibility of executing more subtle bodily movements with completely free mocap softwares, the outcome is currently not looking very promising. First, let’s see how Rokoko Vision fares with movements that are slightly more dynamic than an idle animation!
III. Just Keep Swimming, Just Keep Swimming
There’s not much else for me to do at this point other than determinedly trudge forward in recording the rest of my movements (i.e. “annoyed,” “thoughtful,” “surprised”) despite how awkward and jerky they may be. Anyway, I wasn’t quite sure what I was expecting, going at this ambitious project with a completely nonexistent budget—but I hope Rokoko Vision can at least pull through for the majority of the following full-body movements and provide something even slightly presentable for the final result.
‘Til the next one!
Week 8: Leap by [giant] Leap
I come bearing excellent news, everyone; gather round and gape in amazement at the leaps and bounds of progress my animations have made this past week! (Well, not quite—but this has probably been my smoothest and most worry-free work week yet.)
I. Two Cameras are Better Than One…as are Brains
Cindy and I have taken to meeting up a few times each week so as to promote higher levels of productivity (in theory). This week, our methods actually ended up bearing fruit: with Cindy’s help, I managed to work out Rokoko Vision’s dual-camera motion capture mode, a service I had once believed to be impossible to utilize with my single-webcam Mac. Now, why go through all the complex setup and dual-camera-calibrating pain, you ask?
We have eliminated…most…of the wiggling!
I cannot, unfortunately, convey our progress through video due to the limitations of the blog website, but you’ll have to take my word for it. The addition of a side-view camera has improved the fluidity of our character’s animations tremendously, not to mention its far more accurate positioning of various body parts. Of course, there are still some tricky movements that seem to stump the AI-powered software. The move we currently have for “embarrassed” involves raising a hand to the back of one’s head in one sheepish gesture; however, the software apparently refuses to let the hand go as far back as behind the head and settles instead for waving the hand frustratingly in FRONT of the face.
Anyway, all of this is still a massive improvement for the sorry excuse we had for “motion capture animation” last week. Cindy and I have been diligently fleshing out the dialogue and cataloging all the different emotions/poses we’ll need to be using throughout the entire game (e.g. “smug,” with hands crossed in front of the chest, and “shrug,” with…a shrug). In no time, we’ll have the “Tier 1” of this project done and dusted; it’s the Tier 2, the second layer of the onion, that I’m most uncertain about.
II. Peeling Apart the Onion
Cutting into the onion of my project reveals a frightening inner layer of—drumroll, please—facial motion capture! In order to accomplish this next ambitious goal, I will most likely have to leech off a 30-day free trial of a different online software that can perform facial mocap without any additional costly equipment. Remocapp is currently my top candidate for this role, although I will have to double-check that its services come completely equipment-free.
And so begins the next leg of our ambitious journey—we will now graduate from the heftiness of the body to the delicate features of the face. All I have to wrap up now is the mild clunkiness of some of our already-captured dual-camera animations, and we’re all set to trudge even deeper into the murky waters of Assorted Softwares I Have Never Had Prior Experience With!
Week 9: Scrambling to Find Our [way]
Oh, what high hopes we had last week…
…but hello, everyone! Apparently, the Assorted Softwares I Have Never Had Prior Experience With (mentioned last week) proved to be far more formidable foes than I had originally expected them to be—to no one’s surprise, of course. Sit back and relax as I struggle to find a suitable path with which to continue pushing this project along!
I. Remocapp, How You Have Let Us Down…
The software I had showered with such optimism in my last blog post—a software that should have worked perfectly IN THEORY—ended up bringing us nothing but disappointment this week. Try as we might, Cindy and I simply could not get the second camera to connect or calibrate with the software, despite how smoothly the same process went with Rokoko Vision previously. (It was such a shame, really; Remocapp claimed on its website to capture full-body motion simultaneously with facial expressions, on top of its fluidity and markerless technology. Would have worked perfectly!)
It was back to the mines for me, then, to scour the Internet for possible alternatives. From v-tuber applications to simple personally-designed softwares floating around Reddit forums, I looked through pretty much everything that sparked even the tiniest bit of hope. Eventually, Cindy and I agreed to settle on a rather shaky solution: using Radical Motion, a browser-accessible and normally paywalled service that captures bodily movements alongside facial expressions just like Remocapp theoretically should have. This software’s free version technically only allows one user to export 3 files per year…but Cindy and I figured that, since we were going to rely on free trials and loopholes anyway, we could gather our numerous alternative Google accounts to create numerous alternative Radical Motion accounts!
Thus resumes our conquest of the second layer of our project, the realm of facial mocap. So far, Radical Motion seems to be working as it should, and so are our numerous accounts—but of course, we really don’t have a track record of smoothly progressing through new stages.
II. Cutting Off Vixen’s Wings…
…in a good way.
You see, a previously mentioned problem with our character models was that parts of their clothing (armor, accessories, tricky armpit areas) would move incorrectly as if glued to the wrong parts of the body, thus completely distorting their appearance. I’ll admit, I had been putting off fine tuning this problem for a long while due to the daunting prospect of having to learn weight painting in Blender. As it turns out, learning a singular tool isn’t actually as terrifying as, say, learning an entirely new software. I managed to marginally clean up the “wings” under Vixen’s armpits (pictured below), but there are still some rough bits that evidently need to be polished.
Next victim—Kai!
III. “That Last Lap in Mario Kart Where The Music Gets All Fast and Stressful”
As we enter the home stretch, all the remaining loose ends that still need to be tied up are piling up at an alarming speed. Backgrounds, weight-painting fixes, animation clip tweaks, facial expressions, finishing up the game dialogue alongside Cindy—it all seems almost dizzying. Thankfully, we should at least be able to push out some beta testing later this week, so we’ll at least receive some much-needed feedback on the functions and visuals of our game thus far!
That’s it for this post, and see you one (hopefully very productive) week later!
Week 10: Sprinting [through] the Fog
Progress has indeed been made, everyone! Welcome back to a slightly more polished project and a considerably more optimistic blog post. This week, I’ve tackled most of the “loose ends” I was referring to in my previous post: weight painting, revised/smoother animation clips, and backgrounds…but, unfortunately, no facial expressions yet.
I. Progressing Through Everything BUT the Face
Before we grieve over the sheer impossibility of getting facial animations to work, let us first rejoice over the other areas in which I have made progress this week! After last week, in which I smoothed over Vixen’s webbed-armpit-looking chunks of clothing so that he didn’t transform into a flying squirrel every time he raised his arms, I tackled Kai’s issues next. His most glaring problem was the fact that the plates of armor on his chest did not seem all too rigid when he moved anything in his upper body, since they would twist and mold to the shape of his movements. To make the armor seem more METAL and less FABRIC, I figured out how to parent the armor mesh to only a singular bone so that it would only conform to the movements of that bone rather than those of his skin/clothing. As for his “wings,” my own shoddy workmanship in creating his original clothing mesh proved a fatal handicap in weight painting—I could not manage to smooth out the problematic faces without carving gaping holes or strange undulations in the fabric in the process. Thankfully, Kai’s “wings” were not as egregious as Vixen’s from last week, so after smoothing out all that I could without utterly deforming his clothing, I reimported both characters into Unity and began the process of revising the animations.
With the help of RADiCAL Motion (Remocapp’s replacement software, mentioned last week), I was able to re-record much more fluid versions of some of our choppier Rokoko Vision clips. As you readers might recall, however, obtaining access to this new software wasn’t originally supposed to provide a means to refine old clips—it was intended to pave the way for the elusive “Tier 2” of my project, the movements of the face. Unfortunately, this is where all the animation progress screeches to a grinding halt. Try as I might, I could not get the facial expression data captured in RADiCAL Motion to transfer to my armatures in Unity, despite both characters’ faces already being rigged. This new roadblock definitely calls for another deep dive into internet forums and research, since I cannot come up with any reasons for the animation malfunctions at the moment.
II. A Fresh New Look…with More to Come
On a brighter note, I have FINALLY begun my work with the 2D backgrounds this week, starting with the title screen:
From here, I’ll continue churning out more backgrounds for all the different scenes of the script—and if I can finish all of THAT by the end of next week, I can potentially get started on illustrating the black-screen-white-text “cutscenes” we currently have in the game. On the beta-testing side, Cindy and I will likely be sending out questionnaires to everyone upon their completion of the game (or what we have of it so far). In this way, we’ll be able to evaluate how convincing/immersive each of our contributions were, thus providing us with results for the entire project!
Week 11: Home Stretch in the [setting] Sun
Hello, everyone, and welcome to a glamorous new look for the game! This time, rather than spend another week or two poring over what I likely cannot fix in the time I have left (facial animations), I have invested more effort into polishing the remaining visuals of the game, along with taking in beta testers’ feedback for analysis!
II. Easier on the Eyes
Near the beginning of the week, I had already been approached by a beta tester with critiques of certain spotty areas in my character models—bits of skin peering through Kai’s metal handguards, Vixen’s undershirt being completely rigid and constantly clipping through his neck, etc. Yet another series of model tweaks and reimports later, it was now time to churn out more backgrounds!
Showcased below is the newly incorporated background for the beginning tutorial scenes and the following Scene 3 of the game:
I know you must all be rather dismayed to find that I will not end up conquering the elusive Tier 2 of my project by the time final presentations roll around, but Cindy and I had both agreed it would be the wiser choice to smooth out and embellish what visuals we already had rather than grasp at visuals that seemed impossibly out of reach. (From what research I’ve done on the topic, however, the most likely explanation for my unsuccessful attempts at facial animation is that the simple rig I currently have for the face is far from adequate to accommodate the more complex, subtle movements I captured.)
Thus, our final product for the presentations is complete and raring to go!
II. A Humbling Journey
As mentioned earlier, beta testing has officially begun—and with it, a stream of critique. Our testers’ remarks are invaluable, of course; they’re precisely what both Cindy and I will be basing our results and analysis off of, as the success of our project is entirely determined through the quality of player experience. We did tell our beta testers not to hold back in the honesty of their comments, resulting in such remarks as “Their hands seem unrealistically stiff and their movement is slightly awkward but they get the emotion across.”
From here on out, it’s just a matter of gathering as much critique as we possibly can before spinning it all into analysis in time for the big presentation. Onwards!