Activision secretly experimented on 50% of Call of Duty players by 'decreasing' skill-based matchmaking, and determined players like SBMM even if they don't know it

If you ever find yourself in the middle of an argument about Call of Duty and want to toss a tank of gasoline on the fire, recite these four letters in order: S-B-M-M. Skill-based matchmaking is the invisible system by which Call of Duty, as well as most modern multiplayer games, match you with similarly-skilled players so that every matchup is as fair as possible.

Sounds like a win-win for all involved, but there's a legitimate argument against a matchmaker that only cares about high competition. A faction of Call of Duty players argue that it's not fun to always have to "sweat" so hard in an FPS that most people play casually, and believe a random matching system could accommodate a wider range of experiences—matches where sometimes you're way better than your opponents, and sometimes you get totally destroyed. Notably, the free-to-play CoD alternative XDefiant has no SBMM.

Activision has historically stayed on the sidelines during these debates, but that changed earlier this year when the publisher finally lifted the veil on how matchmaking works in every modern Call of Duty game, and explaining its reasoning for choosing SBMM. Building on that, today the company published the first of a series of white papers diving deep (and I mean terminology tables and multi-format graphs deep) on matchmaking.

The new paper, a 25-page document titled "The Role of Skill In Matchmaking," is as impressive as it is jarring. On one hand, it's a little funny that Activision is treating the topic with tact and seriousness, posing such questions as "What is Skill?" and mapping "kills per minute" and "skill buckets" on graphs, when we all know that naysayers will probably just say "SBMM still trash lmao" and move on. But it's also a wild document for a videogame: I don't think we've ever seen a major studio go this in-depth on the intricate criteria and details that its algorithm uses to judge us.

Some of this ground has been covered before. We finally learned for sure back in April that Call of Duty's matchmaker, above all else, values finding matches quickly over ensuring it's perfectly balanced. What players might find most interesting is the discovery that Activision ran a secret test to see what CoD would be like with less SBMM (but not none). Those sneaky devils.

The experiment ran in Modern Warfare 3 in early 2024. Discreetly, Activision toned down the "skill" in skill-based matchmaking for 50% of the North American MW3 population.

The results? Over 90% of players with toned down SBMM played less Call of Duty after the change, with the top 10% of skilled players not as affected. The paper goes on to confirm what has, to me, always been the most obvious reason to have SBMM: random matchmaking primarily benefits really good players, and disproportionately sucks for average-skilled players.

"The use of killstreaks and increased [kills per minute] and [score per minute] shows that the wider lobby skill percentile disparity is disproportionality leveraged by the top 10% of players," it says. "Unfortunately, this increased performance comes at the cost of much greater impact to the much larger 30% of the population toward the bottom of the skill distribution."

Skill issues

Also illuminating is Activision spelling out exactly which factors affect your skill assessment at the end of a match. You probably correctly assumed what most of them are:

• Match Total Kills: This value tells us how well a player did relative to the other opponents in their lobby at the main objective of the game. However, it has poor cardinality, as many players can achieve the same number of kills. This makes updating skill difficult, as many players will appear equally good based on this performance metric. It also does not reflect a player’s ability to survive, which is an important outcome in Call of Duty as well. For example, a player with 10 kills and one death, is better than a player with 10 kills and 20 deaths.
• Kill / Death Ratio: This value has much better cardinality, and it reflects both the primary and secondary objective of the game-mode. However, it does not account for self-kills, which is an easy mechanism of reverse boosting (artificial [sic] dropping your skill to get easier matches)
• Kills / (Deaths by enemy): This value ensures players cannot artificially drop their skill by simply self-killing. However, a large problem remains; the magnitude of this value is the same for a player with 10 kills and one death regardless of if they played the full match or joined in the last minute.

Your "score" apparently isn't a factor. The game isn't taking specific support actions, like healing teammates or capturing objectives, into account directly.

According to Activision, the biggest reason to use SBMM is to prevent blowouts, "which we know are not fun for players on the losing end," the paper continues. It's not just vibes guiding CoD's SBMM—Activision says it has the numbers to back it up.

"We have found that balancing skill against other matchmaking factors quantifiably increases the extent to which most players play and enjoy Call of Duty," the paper reads. "When skill is utilized in matchmaking, 80-90% of players experience better end-of-match placement, stick with the game longer and quit matches less frequently."

The paper goes on (and on and on) to explain in excruciating detail how Call of Duty balances its skill considerations with matchmaking time, which if we all remember from the beginning of class, is ultimately more important for Activision. The gist I'm getting is that the matchmaker strikes as good of a balance as Activision expects, but you can read that section in detail if you'd like to pick it apart.

I've only scratched the surface of this thing—if you're interested in the machinations that influence your late night FPS sesh, this is one of the most thorough accounts of how it all works from the outside (at least where CoD is concerned). The company plans to publish more of these papers in the future. Maybe by the time the next one's out, they'll be experimenting on Black Ops 6 players.