Understanding your game through data

Understanding your game through dataSergey Galyonkin

Understanding your game through data

2

Marketing analystWorked for Wargaming, Nival, 1C, handled Ukrainian

distribution for Activision Blizzard, EA, Ubisoft, Take Two and so on.

I also host a podcast about game development “How games are made”

and write a lot on Twitter as @Steam_Spy.

Sergey GalyonkinWho am I?

http://twitter.com/Steam_Spy

http://twitter.com/Steam_Spy


3

What I’m going to talk about

• There are many ways to obtain data that might help you to improve your game.

• You should gather data on all stages of game’s lifecycle, starting with pre-production and going well into support and “$0.99 sales” stages.

• You should start to work with analytics before you even launch the development.

• Not every tool mentioned will work for your game, but most of them will.


“Don’t tell me how to make my game”“I’m a creator, not a corporate slave, and I don’t need marketing research to make something truly unique. Like retro-looking platformer for Steam.”


5

Some arguments against doing your research

• “I’m a creator, I know what I’m doing and I don’t want unwashed masses opinions on that”.

• You’re not asking for advice when you’re doing research. You’re asking questions.

• “I’m making something truly unique, there is no pre-existing market for it”.

• Awesome, but how do you know, if you didn’t do a proper research first? Chances are there have been games with mechanics similar to yours. Don’t you want to learn on their mistakes?

• “I don’t care about the numbers, my goals are qualitative, not quantitative”.

• Qualitative goals can be measured too. Don’t you want to make sure you’ve achieved your targets?


6

Some truisms

• You’re doing research and analytics to make informed decisions about your game.

• You’re the one making decisions. Not the audience, not Steam Spy, not Google Analytics.

• Data doesn’t “tell” anything, nor it is “good” or “bad”. You’re the one interpreting it.

• There are many tools to do the research. Not everything will be applicable to your game.

• One day spent doing the research could save a month in development later.

• While doing research is an actual job (like marketing or art), you can still do some of it yourself without hiring other people.

Before the developmentI don’t even have a game yet, why do I need to do analytics at this stage?


8

Why?

• It’s the best time to research the market, as you are not invested in your game too deep yet – both emotionally and financially.

• You can cancel or change the game after one week of market research – it’s cheaper and easier compared to years of development.

• You might improve your game even before you start working on it.


9

What do I need to research?

• The game’s theme• The game’s core mechanics (genre)• Audience• Market• Competition


10

Theme, mechanics and visual style

• Your game could be described as intersection of theme (sci-fi, fantasy, modern warfare), set of core mechanics (FPS, RPG, side-scroller) and visual style (80’s cartoon, surrealism, gritty, fake retro).

• The theme is the easiest part to research


11

Theme

• Look for movies, TV shows, comic books and books sales and audience engagement stats for your theme.

• Most of it could be obtained through Statista, Box Office Mojo and IMDB. Or even Google Trends. Steam Spy is not so great here.

• E.g. it’s obvious that the interest in sci-fi and space opera is growing recently as we see more movies, books and comic books in this theme.

• It’s hard to lead the trend - create a game that will make everyone interested in sci-fi.

• It’s easier to follow - create a sci-fi game just in time when people have an itch to fly space ships thanks to a movie or book they’ve enjoyed recently.


12

Theme


13

Theme

• Predicting a trend is a bit harder, but it’s worth it. Sci-fi growth was visible right after Avatar success and we saw a number of games successfully using it.

• Look for announcements of big movies with your theme and chart them on a graph so you can see when you can expect their marketing campaigns to start (and help you).

• If your theme is too niche or too small – it’s fine too. You’ll learn more when researching your audience.


14

Core mechanics

• Decompose your game into several core mechanics that are easy to explain and preferably match Steam’s tags. E.g. Turn-Based + Platformer + Clicker.

• Look for games that are using the same core mechanics on Steam Spy or Google.

• Make a list of top games for each mechanics (or better for their combination) and list their strengths and weaknesses (gameplay-wise).

• You will later use this list to avoid making mistakes other people did while developing similar games and to improve your own game.


15

Audience

• This one is tricky. • Your potential audience are people with access

to the platform of your choice interested in your game’s theme and its core mechanics and not scared away by its visual style.

• You’ll probably have to use social media groups or forums. I’d start with the ones dedicated to your theme, but you can go with core mechanics as well.

• Go, introduce yourself and just ask people if they would be interested in playing something like your game. Make a poll if the social media of choice allows it.


16

Caveats

• Don’t ask your friends and don’t promote that poll on your own social media. You want an opinion of your target audience.

• Use several independent forums or groups, not just one.

• Don’t pay too much attention to game ideas people will gladly provide you with. Not at this stage at least.

• Be aware of sampling error (both oversampling and undersampling). If, according to movies and books with the same theme, your game’s theme is more appealing to women over 40, don’t ask twenty-something males about it.


17

Competitors

• Make a list of games that are using the same theme and some (or all) of your core mechanics. Both released and upcoming.

• Make a basic SWOT analysis for all of them, but also add an additional field: “Why our game is different”.

• The key word here is “different” not “better”, so you won’t get caught in wishful thinking “we’ll have better graphics and better balance”.

• You can also check geographical distribution and stats for released games on SteamSpy or AppAnnie, but, frankly, it’s not that useful at this stage.


18

Summary

• A good analysis will take you a week or two, full-time.

• It might seem like overkill, but it’s nothing compared to doing the same during actual game development and then changing the game accordingly.

• Do it yourself, don’t outsource. You might come up with new ideas for your game at this stage by just researching relevant media and audience.


During the developmentYour choices now are becoming more

and more expensive.


20

Why?

• It’s the time when you can get a reaction to your game, not just an idea of it.

• You can change many things without angering your existing audience.


21


• An audience reaction to your visual style• Players behavior in your prototype• User interface


22

Visual style

• If you’ve talked to people online about your game in the previous stage, you should have a pretty good understanding of who is the target audience for your game.

• You can now ask them about your visual style. Usually it’s enough to have several art draft/mockups.

• Don’t ask if they like it or not, ask if they’d play a game that looks like this.

• Focus on your target audience. Every successful game is considered ugly by someone on Internet. Skyrim? Too dirty. Clash of Clans? Too childish. League of Legends? Too bright.


23

Player behavior

• By now you should be adding in-game analytics already. There are many options, you can choose whatever you like.

• But even if it’s not there yet, you can learn a lot by watching people play your prototype.

• The best way is to record a person’s face simultaneously with what’s going on the screen (think “letsplays”), but you could just sit nearby and watch if the person is comfortable.

• Again, when inviting people to try your game, focus on the target audience, not your friends.

• It will get harder in the end, as you’ll be running out of relatives to test the game on.


24

Player behavior

Things to look for when watching people play your game:• What catches their attention?• What do they do first?• How much time passes before the person gets to the

first major point in the game? • Did you expect it to take this long?• Do they understand the game rules?• Do they understand effects of game items or enemy

strategies?• How do they handle controls?

When a person starts playing, don’t talk, don’t help, don’t defend or explain your game. Shut up and watch.


25

Player behavior

• Always do a short interview and a survey after the playtest. It helps to quantify and record the results.

• Determine a net promoter score. Ask “On a scale of 0 to 10, how likely are you to recommend this game to a friend?”

• You can ask if the person liked the game or not, but NPS is more useful, as people tend to lie so they don’t hurt your feelings.

• Ask about what the person found most confusing, most fun, most difficult and most boring.


26

Net Promoter Score

Test Detractors (0-6) Passives (7-8) Promoters (9-10) NPS ScoreIdea test 1 (online poll on forum) 5 20 10 14%Visual Style 1 5 7 6 6%Visual Style 2 7 4 7 0%Visual Style 3 7 6 5 -11%Playtest 1 (studio) 5 7 6 6%Playtest 2 (expo) 20 45 30 11%Playtest 3 (online) 110 200 250 25%

How likely are you to recommend the game to your friends? On a scale from 0 (not likely) to 10 (extremely likely)


27

User Interface

• Prototype on paper. Get a notepad the size of iPad or iPhone (they’re available online) or just typical A4 one and draw stuff.

• If your game is for mobile devices, consider different screen sizes and different hands sizes. Test on different people.

• You’ll be able to eliminate many problems by just handing the piece of paper to a person and asking if he/she can understand how the game is supposed to be played and what’s going on the screen.

• I have a friend who builds elaborate paper UI prototypes with moving parts, but it’s probably too much. Quite fun to use, though.


After the launchThe game is done, fire and forget.

Why are we still analyzing it?


29

Why?

• You have actual users playing the game!• You can still find and fix a lot of problems• You’ll learn a lot for your next game


30


• In-game behavior• In-game economics• In-game balance


31

Behavior

• I hope that by release you have some analytics system implemented – Google Analytics, Unity Analytics or even home-grown solution will do.

• You should be tracking user path through the game.

• Cluster users based on the steps they’re taking in your game. Some will go straight to tutorial, some will go PVP, some will start exploring.

• How much time a person spends in a game per sitting? How often does he play per week?

• Don’t go for averages, always go for brackets or clusters, as averages are deceiving.


32

Economy

• How many resources a person has after 1 hour / 1 chapter into the game? How many people have too much and how many don’t have enough?

• What people usually spend their resources on? Do they go for armor, swords, potions, artillery or tanks? Why?

• Cross-reference the economy with behavior clustering. Do “explorers” buy different items compared to “fighters”? Do they have more resources or less? Should it be like this?

• Research consumables (potions, gems, etc). Do people actually use them or save them for later?


33

Economy

Expected valueDidn’t understand the game at all

Low skill, will have problems later in the game

High skill, might find the game too easy later

Munchkins?

Exploiters?


34

Balance

• Which character/class/role people choose more often? Does it vary based on their behavior in game?

• Based on the character selected are there any differences in resources acquired, damage dealt, time spent in game?

• If the game has PVP, check the win/lose ratio for every class. Again, use brackets or clustering instead of averages, as win ratio might vary a lot depending on users’ skill.


35

BalanceThe 1st chapterClass Warrior Paladin Priest Wizard MonkTotal players 600 200 100 50 50Share of all players 60% 20% 10% 5% 5%Resources after 252 170 150 405 180DPS after 45.2 35.2 17.2 92.5 40.2Average time to beat 01:05 01:30 02:30 00:55 01:15

The 2nd chapterClass Warrior Paladin Priest Wizard MonkTotal players 550 180 70 47 47Share of all players 62% 20% 8% 5% 5%Resources after 630 442 390 810 450DPS after 90.4 70.4 43 148 80.4Average time to beat 01:02 01:20 03:00 00:50 01:10

The 3rd chapterClass Warrior Paladin Priest Wizard MonkTotal players 545 175 35 45 45Share of all players 64% 21% 4% 5% 5%Resources after 1575 1105 975 1782 1125DPS after 180.8 140.8 107.5 236.8 160.8Average time to beat 01:07 01:25 03:30 00:40 01:15

What can we learn here?• Priest class might be severely

underpowered. Players are leaving the game.

• Wizard class is either too strong or more appealing to more skilled players

• Monk and Paladin are all right• Warriors might need some

balancing, so they spend more resources


SummaryA quick recap


37

Key takeaway

Do your research before developing a game.

You’ll both save time and learn a lot.


38

Key takeaway

Test your game on your target audience

even before you have anything playable.


39

Key takeaway

Implement in-game analyticsas early as you can, it makes analyzing

user behavior way easier.


40

Key takeaway

Don’t trust averages, look at the whole picture.


41

Key takeaway

Don’t dismiss analytics as something

for “the big guys only”. You can do a lot of it yourself

or with a small team.


42

Questions?

Skypesergesegal

[email protected]

Twitter@Steam_Spy

What?

How?

When?

Why?

Where?

Are we going to

die or not?

Sergey Galyonkin

Understanding your game through data

Software

Transcript of Understanding your game through data