Week 4 – Quantifying Skill Level

Welcome to the fourth week of my Senior Project, and my fourth blog to go along with it. I finally finished the entire game, and started thinking more about how the game will relate to my project. I will now need to come up with ways to vary the difficulty in order to attempt to make the game more enjoyable to play. Initially, I had planned to survey players about their thoughts with different difficulties, and attempt to find a difficulty that most people enjoyed. While this may not be a bad idea, there are a few issues I can see with it, including not having a large enough sample size and the fact that different people may enjoy different things. As a result, I also wanted to spend some time doing research on alternative ways that I can have the game automatically adjust the difficulty to fit a player’s “ideal” difficulty.

Fixing Remaining Bugs

Last week, I still had two bugs to fix. I spent some time reading over my code and came up with some ideas to fix them. I ran the program before doing anything, and found that the game worked fine. I’m not really sure how changing king movement fixed these, as the portion of code that was changed shouldn’t have even been executed here, but it does work now.

Systems Used To Classify Skill

Many games are able to use a rating system to quantify a player’s skill. Some systems will use a number to represent rating, with higher numbers signifying better players. Other games will use ranks to indirectly do this, with tiers like bronze, silver, gold, and others, to categorize players based on skill. Additionally, professional sports often use a win-loss ratio each season to determine the best teams. I looked at all three, and how different games use each in order to determine a player’s skill.

Elo Rating

The Elo rating system was created by Arpad Elo to improve the chess rating system used at the time.[1] Players have a rating based on their performance while playing the game. When two people play against each other, comparing their Elo ratings can give the likelihood of each player winning. This isn’t calculated by simply comparing the ratings directly, but by assigning a “score” based on how many times one player would beat another in order to create a non-linear scale. Using the formula 10R/400, where R is a player’s rating, will give each player their score.[2] From there, the two scores can be compared to find a player’s likelihood of beating their opponent. 

For example, if Player A has an Elo rating of 900, while their opponent, Player B, has a rating of 800, their scores can be calculated as 10900/400 and 10800/400. Player A will have a “score” of 177.828, while Player B’s will be 100. These scores can now be directly compared, with Player A expected to win 177.828 games for every 100 that Player B wins, or approximately 64% of the time. As these scores can be proportionally compared, a certain difference in Elo rating will always result in the same expected win rate. A 100 point difference will result in approximately a 64% chance for the higher rated player to win, as shown in the previous example. A 200 point difference will result in approximately a 76% chance for the higher rated player to win, and a 400 point difference will result in approximately a 91% chance for the higher rated player to win.

Tiers And Ranks

Many multiplayer video games with a competitive game mode have ranks to put players into. This separates players by skill, and allows for fairer matchmaking. Players move through the ranks as they improve, and play against continuously better opponents as they move up in rank. When a player gets stuck in a rank, and is unable to move any higher, they have usually found where they “belong.” This is the rank at which they win approximately 50% of the time and play about as well as the other players, resulting in them not moving up or down in rank until their skills either improve or regress, and represents the player’s current skill level. In the game League of Legends, for example, players are classified into 9 tiers, ranging from Iron (lowest) to Challenger (highest). Lower ranked players typically win 46-48% of their games, while the highest ranked players typically win 53-54% of them.[3] Players in the middle ranks, who would be considered average skill, win approximately 49.5-50.5% of the time.

Win-Loss Ratio

Most American professional sports leagues have a regular season followed by playoffs for the top teams. At the start of the season, a team’s goal is typically to make the playoffs. While it can be advantageous to make the playoffs as a higher seed (better regular season teams are higher seeds in the playoffs), doing well in the regular season doesn’t guarantee anything, and not as great teams that still make the playoffs are still able to compete for a championship. And while a playoff team may not win it all, making the playoffs is typically a sign that a team is good. I looked at Major League Baseball, as its long 162-game season gives a much larger sample than basketball and hockey’s 82-game seasons or football’s 17-game season and will allow me to understand what win-loss ratios are good indicators of a team being good in a season. 

The MLB is divided into two conferences of 15 teams each, with each conference further divided into three divisions of 5 teams each. The team with the best record in each division makes the playoffs, along with the next three best overall in each conference (known as wild card teams), for a total of 6 playoff teams per conference, or 12 overall. This is a recent change, as the third wild card was only added in 2022. Additionally, the second wild card was added in 2012, with 1995 being the first year to have any wild card teams at all. Under a system with 2 wild card teams (5 playoff teams per conference), teams need around 89 wins to make the playoffs, and with 3 wild card teams, teams need around 86.[4] On average, this is approximately a 53% to 55% win rate (and very close to the best players’ win rate of 53-54% from the Tiers and Ranks section) to make the playoffs, and for my project, a reasonable win rate to consider a player good at a certain difficulty.

Conclusions

Elo is a great rating system, and there is a reason it has been widely used for so long. Using it would give very precise information about a player’s skill level, and allow me to very accurately provide an opponent of the right difficulty. However in the context of my project, to be effective I would need to know the Elo rating of every difficulty level of my AI, something I am not able to do. As a result, my current plan to implement variable difficulty levels is to adjust it based on how often a player wins. This will combine the tier system along with tracking win-loss ratio, with different difficulty levels representing the different tiers, rather than calling them things like gold, silver, or bronze. 

For example, if difficulty levels range from 1 to 10, let’s assume that Player C is playing at difficulty 4. If they win 7 of their next 10 games (70%), they might get moved to difficulty 6, as they are able to win more often than not at that level. If they win 4 of their next 10 (40%) at difficulty level 6, they aren’t winning as often as the lose, and are likely playing at a level that is too hard for them. Since Player C is better than level 4 but worse than level 6, they might play against a level 5 opponent. If they are able to improve and end up winning more, they might later move back up to level 6. And since they are now also a better player, they might be able to win 50% of the time at this level. The process will repeat, moving the player up or down as needed, until they find a rank at which their skill is matched by the AI’s skill. 

As players play more and improve, the level that they play against will continue to change as well to match their skill, and if a player is able to win approximately 55% of their games, the difficulty they play at will increase. However, if they win under 45%, the difficulty they play at will decrease. And if needed, I can additionally survey players as originally planned, and use their opinions to determine if 45% and 55% are the right values I should be using. For example, I may find that many people consider a 60% win rate to not be too easy, and that the difficulty should be increased at a 65% win rate rather than 55%. Or I may find that many consider only a 50% win rate against a computer to not be enjoyable, and would prefer easier opponents at 50%, rather than 45%.  

Citations

[1] https://en.wikipedia.org/wiki/Elo_rating_system 

[2] https://lichess.org/@/CheckRaiseMate/blog/how-elo-ratings-actually-work/J8UZThlO 

[3] https://leaguefeed.net/good-winrate-lol/ 

[4] https://www.baseballamerica.com/stories/how-many-wins-will-it-take-to-make-expanded-playoffs/