Big Data Mining in Call of Duty – Using Data to Improve Games

Activision allowed two of their most credentialed employees to host a PAX ’15 panel on the role users play in game development. PhDs Justin Shacklette and Spencer Stirling spent nearly an hour explaining how the company is constantly, intelligently collecting data referred to as "Smart Data."

The relatively new (~4 years old) Game Science Division is a group of physicists and mathematicians who also happen to be talented programmers. Their goal is to collect and analyze data to find ways to make the games more fun. Using metrics to make products better is nothing new, and marketing teams are doing this in almost every business in the world; however, as far as Shacklette knew, only Activision and one other company (Riot) have been doing this to improve games instead of just sales.

When it comes to making the next Call of Duty game -- Black Ops 3, in this case -- the Game Science Division is not sitting at the development team's round table and providing some earth shattering new development idea that completely changes the game. Instead, they are using math to make millions of small decisions each day to net a better overall game at the development process. As a matter of fact, when speaking with Stirling after the panel, the expert stated that they really don't work directly with the other teams too often. Instead, they utilize mathematics and programming to automate research to answer questions the other teams have asked.

The team’s job often demands focus on released Call of Duty titles like Advanced Warfare and Black Ops 2, and the networks upon which those games are played to gather information about what the users are doing most frequently. This concept is something we discussed previously with Epic Games, who perform heatmapping to determine usability of maps. Sometimes the information is about good user behavior and sometimes it’s about things they would rather not see happening in the game. Either way, the Game Science Division is able to use metrics to help answer tons of small, yet influential questions brought-up by other teams that need answering.

One example the pair used to help explain what they do was taken from COD: Avanced Warfare and, really, every game in the franchise. It centered on “boosting.” Boosting is recognized as a behavior that most users thought of as a negative behavior. In order to detect it, Activision’s GSD had to create programs based on an advanced mathematics concept referred to as Machine Learning – we’ve written about this here. Simplified, GSD identified key indicators that were associated with boosting, then found ways, through machine learning, to link and track those indicators.

Some of the common user behaviors in COD that identify boosting are players on friends lists that end up on different teams match after match, tactical spawn points where deaths occur after a very short period of time over and over again, and map data that shows a spawning player going back to the same location to die repeatedly. These behaviors and several others are all trackable data points that can be used to determine if COD players are boosting.

Once the data is analyzed and the probability of a negative behavior is determined, the next step is to find a way to try and discourage the negative behavior or, preferably, replace it with a positive behavior. This is where social media and in-game bonuses come in. The GSD team said that they never hop on Twitter and publicly discourage boosting, but they do suggest that it is not good for the game and that other users do not appreciate it. Then, the team waits for the responses to these types of tweets and collects data to see if it has any impact.

The way Activision’s GSD measures the Twitter data is pretty fascinating, too. They collect data on every COD tweet that any COD user puts up. This was demonstrated in a program created during the panel by having us tweet things about PAX Prime, COD, and Smart Data. They then accessed the program in real time and showed us how, based on the content of the tweet, they could tell if the information in the tweet was positive, negative, or basically neutral. Using this system, the team can make decisions about many different aspects of the game and the playerbase. The Twitter data collection system is not limited to just boosting detection. For instance, it can also collect data and serve as an early warning system for networking or programming errors.

Shacklette stated that they use player feedback on social media to monitor activity for Activision teams. Two of the most relatable were Quality of Service (QoS) and game balance. QoS primarily deals with the network connecting COD players together, a component that both presenters agreed should never interfere with the user experience. When negative social media input causes QoS spikes, the system can send alerts to networking teams to check and see if a networking problem is occurring. If an important networking link goes down, it is likely that many Advanced Warfare players will not be able to connect to a server to join matches. The players are then likely to hop on to Twitter or Facebook and post a quick message about how “terrible” the game is because they can never connect. These social media inputs register as negative and the monitoring system spikes sending alerts to the networking team before a Technical Support call is ever made to Activision. By the time a Tech Support call does come through, the users can be reassured that the problem is already being resolved.

Similarly, if tons of Black Ops players are complaining about a specific map or weapon via social media, this also generates a data spike. In this instance, the data is collected and routed to the dev team. The dev team then requests more specific information based on ideas they have to fix the maps or weapons, and the Game Science Division delivers again. In this way, almost uniquely, COD players have more input in the changing of their own game than any other gamers in the market. Both Shacklette and Stirling stated that a gratuitous amount of these types of data points were taken from the users to help determine the creation and direction of Black Ops 3.

In the end, better analytics should equal better gaming. Theoretically, at least.

- Patrick Stone.