Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging
description
Transcript of Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging
![Page 1: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/1.jpg)
Maximizing Revenue Through Logging Albert Ho
![Page 2: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/2.jpg)
About Rumble Funded by: Mission: ● Bring AAA quality to F2P ● Play together cross-device/platform
![Page 3: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/3.jpg)
Rumble’s Platform ● Played in >200 countries ● Six data centers worldwide ● Hybrid infrastructure ● Play together on:
![Page 4: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/4.jpg)
What this talk is about…
User Acquisition
Funnel Loss Retention Loss
Your Game Paid Users
Free Users
![Page 5: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/5.jpg)
“Logging” Distinctions ● User behavior
● Game design, retention, virality, monetization
● System behavior ● Maximize potential ● Lower CPA
![Page 6: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/6.jpg)
Thesis “You really aren’t maximizing revenue
unless you can measure system behavior”
![Page 7: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/7.jpg)
![Page 8: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/8.jpg)
Challenges ● 24/7 operations ● Game teams ship code daily ● Being “up” is not good enough
![Page 9: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/9.jpg)
The Long Tail of Revenue Lost: Daily Site Problems Affect Weeks of Revenue
Outage
Problem
Recovery
New Installs/Sales
Revenue
Days
![Page 10: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/10.jpg)
How to Detect Problems
● Solving is Harder than it sounds Signal to noise ratio + Indexing and search + Multiple game teams shipping code daily = BIG Headache and world’s biggest chore
The answers are in the logs!
![Page 11: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/11.jpg)
Rumble Logging 1.0 ● Graylog, Elasticsearch ftw?
● 1 day to setup ● Collecting for 2 game servers ● Few hundred MB a day / 7-day retention time ● Scaling is easy… right?
- Bigger hard disks, more ES nodes
![Page 12: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/12.jpg)
Rumble Logging 1.5 ● 7 days to setup ES cluster ● > 20 servers ● Logs @ TB a day / 3-day retention time ● Log noise incomprehensible ● Very expensive!!! ● Logging cluster broke every day
![Page 13: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/13.jpg)
Key Learnings Log Management is really… “Undifferentiated Heavy Lifting”
We were spending more on LOGGING than HOSTING
There must be an expert that does it better, faster, and cheaper
![Page 14: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/14.jpg)
Rumble Logging 2.0 Requirements ● Must support JSON ● Easy to integrate ● Can scale up and meet retention time goals ● Cost-effective
![Page 15: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/15.jpg)
Enter…….
● No more agents! ● REST API – 21st century please/no syslogs ● Scalable and Responsive ● Superb partner relationship
![Page 16: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/16.jpg)
Sample Log
![Page 17: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/17.jpg)
Loggly Use Cases ● Log clean-up ● Release monitoring and debugging ● Customer service requests ● Trend analysis ● 3rd-party developers
![Page 18: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/18.jpg)
Log Clean-up ● 1000’s errors per minute noise ● DEBUG, INFO, WARN, ERROR? ● Reduce TB’s of logs to GB’s
![Page 19: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/19.jpg)
Release Monitoring ● Log snapshots before and after
![Page 20: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/20.jpg)
Release Monitoring ● Game teams own
process ● All logs in one place ● Easy for PM’s
![Page 21: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/21.jpg)
Customer Service Reactive problem solving ● “Why is customer x slow?” ● “Why did I not get my gems?” !
Proactive debugging ● “These 20 customers were affected by
today’s issue.”
![Page 22: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/22.jpg)
Trend Analysis Game performance analysis over time
Client Lag
Days
![Page 23: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/23.jpg)
3rd Party Development ● Bake logging into games at the beginning
● Including 3rd party ones…
![Page 24: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/24.jpg)
![Page 25: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/25.jpg)
Loggly Live Demo
![Page 26: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/26.jpg)
Loggly Before And After Before After
Log Retention 3 days 15 days
Log Availability 50% uptime >99.9%
Debugging Speed Days Minutes
Happiness Factor
![Page 27: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/27.jpg)
Questions? [email protected] KingsRoad http://kingsroad.com http://apps.facebook.com/kingsroadgame
Ballistic https://apps.facebook.com/ballistic
![Page 28: Rumble Entertainment GDC 2014: Maximizing Revenue Through Logging](https://reader033.fdocuments.in/reader033/viewer/2022051609/547e90a6b4af9f9b158b5723/html5/thumbnails/28.jpg)