Balance in the System, Part III

Greetings Rokugani!

The Kotei season is finished, and while we still don’t know exactly which icons are going to be saved, we do know all of the results. And that means that we are finally for the third and final installment of this article series which looks at exactly how good the faction balance was during the Spring 2015 Kotei season was as a whole, and how good it is in the Twenty Festivals Arc format in particular.

I will continue to use the metric that I devised and spoke about at length in my previous two articles (which you can find here, and here). I do, however, want to very quickly remind everybody what numbers I’m looking at, and how I ultimately decide what those numbers mean. I look at how often a specific clan was able to make it past the cut as normalized by how well represented that clan was in the tournament. In an ideal world, if 20% of attendance at a tournament was made up one clan, then 20% of the decks that made it past the cut should also come from that clan. This normalization process gets us a ratio of odds. In the dream scenario, each clan’s ration would be 1. The further from 1 a clan goes (either up or down), the less balanced it is. Numbers below 1 indicate that the clan has a hard time getting past the cut (an “underpowered” clan), and numbers above 1 indicate that the clan is making the cut more often than they should (an “overpowered” clan).

I then separate clans into different categories of “balance” based on the thresholds delineated below, which also allows me to assign a certain point value to the Design Team. Although technically the highest possible score in my system would be 18 points (2 for each clan), I think it the difficulties inherent in balancing multiple factions with multiple themes and victory conditions makes this nigh impossible. I instead grade the Design Team based on a high score of 9.

    Any clan within 10% of the ideal (0.9-1.1) is extremely well-balanced, and I expect that most of the variation from the ideal is due to chance, rather than balance issues. Ideally, this is where all clans would be. I will award the Design Team 2 points for each clan that makes it into this category.
    Any clan 11 – 20% of the ideal (0.8-0.9 and 1.1-1.2) is acceptably balanced. Chance and card power play nearly equal roles in the variation from the ideal. Due to the difficulties in balancing so many factions, this is the category I would expect most clans in a well-balanced environment to fall into. I will award the Design Team 1 point for each clan that makes it into this category.
    Any clan 21 – 35% of the ideal (0.65-0.8 and 1.2-1.35) is not balanced. Card power plays a significant role in the variation from the ideal experienced by these clans. I don’t want any clans to make it into this category, but I do realize that even with good design, clans will occasionally slip into this category. The first clan in this category will net the Design Team 0 points, but for each subsequent clan, I will subtract 1 point.
    Any clan that is outside of 35% from the ideal is a problem – it either represents a single clan that wins so often they are format-defining, or a clan that is so badly hampered by their poor card pool that they are effectively unable to compete. As this category represents a major detriment to the game, I will subtract 3 points from the Design Team for each of these clans.

Let’s first revisit the composite data set – the entire 2015 Kotei Season. When I last looked at it (which you can read about here), I found that 7 out of 9 clans were rated as “acceptably balanced” or higher, with only Spider (on the low end), and Mantis (on the high end) being unbalanced. This yielded the Design Team a total of 10 points, indicating that despite the few problems, the season was remarkably well balanced. But that was weeks ago. How do things look now with a complete data set?

2015 Kotei Season Attendance % Made Cut % Odds Ratio Points
Crab 151 9.34 34 10.09 1.08 +2
Crane 188 11.63 35 10.39 0.89 +1
Dragon 179 11.08 34 10.09 0.91 +2
Lion 184 11.39 34 10.09 0.89 +1
Mantis 185 11.45 55 16.32 1.43 -3
Phoenix 185 11.45 47 13.95 1.22 +0
Scorpion 158 9.78 28 8.31 0.85 +1
Spider 201 12.44 29 8.61 0.69 -1
Unicorn 181 11.20 41 12.17 1.09 +2
Total: 1616 337 +5
While 2015 hasn’t been perfect, it deserves acclaim.

Well, the last few weeks of the season clearly decreased the amount of faction balance the season had – clans that were struggling to make the cut (like Crane, Lion, and Scorpion) slipped further, while clans like Phoenix dominated enough that they can no longer be considered balanced. Over all, these changes cause the Design Team to drop from 10 points (as I had previously assigned them), to 5.  While this may seem like a dramatic drop in points, I think it is important to note that 6 of the 9 clans remain “acceptably balanced” or better. The problems are Spider and Phoenix (neither of which was balanced), and Mantis (who were format-warping).

This means that despite the last few weeks doing no favors to this season’s record, it is still the most balanced Kotei season since 2011. Considering how poorly balanced 2014 was, and that the 2014 and 2015 seasons share many cards, I think it is safe to say that the Design Team did a very commendable job at balancing the factions. We can only hope that this upward trend continues into Onyx edition and beyond.

 

Twenty Festivals Shakedown

So just how good were these 20 Festivals?

But what about the Twenty Festivals (20F) Arc format? How did it turn out? Well, before I show you the numbers, I want to talk a little about the data. If that’s not what you’re looking for, then skip right on down to the chart and commentary below.

How did I put together the data set? This set of data includes all of 36 20F Arc Koteis (meaning that it does not include the single 20F Strict event, nor does it include the two 20F Extended events), as well as the North American Championship event run at Origins this year. Together, these events represent the largest set of data that I have access to from a single format. No new cards were released during this time, and no cards were banned. A few cards received an erratum, but these were very minor changes that often just made the card work as intended. As such, it would be safe to conclude that all of these events existed within the same environment.

So, what’s wrong with the data? Well, although this data set comes from a single environment, none of the other data sets I have looked at do. Since we expect mixed data sets to be inherently more balanced than pure data sets (since the deficiencies of one set can be made up for by another), that means that we have to walk into this analysis expecting there to be less balance in this data set than in any of the others I’ve examined. So comparing this data set to any of the previous sets I’ve looked at is murky at best.

Always with the complications!

A second problem is its size. 262 decks made the cut this in 20F Arc events. And while that sounds like a lot, it is small enough to multiple clans are only 1 deck away from either being bumped up or down to the next category of balance. Personally, I am not very comfortable with saying “Clan X is not balanced, but if 1 more deck had made the cut, then it would be” or “Clan X is totally balanced, but if 1 fewer deck had made the cut, then it wouldn’t be.” And this is precisely the position that this data places me in.

So what am I going to do about all of these problems? Honestly, I don’t know what to do, and I’m not sure there is anything that can be done. I will simply run the analysis with my numbers, and turn a blind eye to examples when a single deck makes the difference between two different categories; after all, one must draw lines somewhere! Ok, enough with what we can’t control for, lets look at those numbers!

20F Arc
Attendance % Made Cut % Odds Ratio Points
Crab 117 9.44 28 10.69 1.13 +1
Crane 147 11.86 28 10.69 0.90 +2
Dragon 138 11.14 22 8.40 0.75 +0
Lion 137 11.06 26 9.92 0.90 +2
Mantis 140 11.30 40 15.27 1.35 -1
Phoenix 150 12.11 44 16.79 1.39 -3
Scorpion 123 9.93 20 7.63 0.77 -1
Spider 155 12.51 23 8.78 0.70 -1
Unicorn 132 10.65 31 11.83 1.11 +1
Total: 1239 262 +0
What’s the count?

Well, only four clans (Crab, Crane, Lion, and Unicorn) are rated as “acceptably balanced” or better.  That’s quite the tumble from the 6 or 7 balanced clans that the entire season was showing.  The Scorpion and Dragon clans ended up having a very difficult time in Twenty Festivals and fell from balanced to unbalanced.  These clans actually provide a very nice illustration of the dangers of looking just at the number of Kotei wins for balance – despite their nearly equal odds of making it past the cut, Scorpion won 5 tournaments, and the Dragon won 2.  Looking at that, it would be tempting to say that Scorpion is in a much stronger position than Dragon, but when you look at the larger sample size of who made it past the cut, you see that the clans are actually very evenly matched.

The other point of interest here is that Mantis managed to sneak back into the “not balanced” category (as opposed to “format warping”).  Although I think that has more to do with some very high performing Mantis players (like Jon Palmer and Chris Medico) switching to other clans toward the end of the season, than any signs that the Mantis are actually a more balanced faction.  However, any point gains made by the Design Team due to the slipping of Mantis were negated by the Phoenix honor rocket which was apparently unwilling to end the season as anything but “Format Warping.”

20F was a success, but only if it starts the Design Team down a path to better and better design.

So in the end, the Twenty Festivals Arc format features 4 of 9 clans as being acceptably balanced or better, with another 4 clans being not balanced, and one clan being format warping.  That nets the Twenty Festivals Arc Design Team 0 points.  Ouch.

But remember, we expected that data from a single environment would be less balanced than data taken from a mixed environment.  With that in mind, I would offer the Design Team a passing grade, but suggest that they take the class again.  I think this is an admirable place for the balance of such a complex game to start, but I would also hope to see it get better over time.  It will be very interesting to see to what degree the cards released in the upcoming expansions help to moderate the power of the dominant clans (Mantis and Phoenix), and strengthen the under-performing clans (Dragon, Scorpion, and Spider).  In particular, I’m interested to see how much the fear-boosting cards printed in Thunderous Acclaim are able to help the Spider.

I hope you enjoyed my in-depth at the faction balance of the L5R.  Join me next time when I look at how well attended this Kotei Season was in comparison to the last 4 seasons, and I examine whether the release of 20F actually helped attendance or not.  Until then, enjoy all your matches, and have fun experimenting with new Thunderous Acclaim decks!

Leave a Reply