Post on 19-Feb-2017
A global consulting and IT solutions leader
Understanding the State of your Web Application
using Apache Kafka, Spark
What if webapps ….
… have feelings?
java.lang.NullPointerException at com.insideout.riley.anger.doAngry(Riley.java:123) at com.insideout.riley.joy.doHappy(Riley.java:456) at com.insideout.riley.sadness.doCry(Riley.java:789) at com.insideout.riley.disgust.doEwwwww(Riley.java:143) at com.insideout.riley.fear.doAaaaahhhh(Riley.java:888) at com.insideout.riley.anger.doAngry(Riley.java:123) at com.insideout.riley.joy.doHappy(Riley.java:456) at com.insideout.riley.sadness.doCry(Riley.java:789) at com.insideout.riley.disgust.doEwwwww(Riley.java:143) at com.insideout.riley.fear.doAaaaahhhh(Riley.java:888) at com.insideout.riley.anger.doAngry(Riley.java:123) at com.insideout.riley.joy.doHappy(Riley.java:456) at com.insideout.riley.sadness.doCry(Riley.java:789) at com.insideout.riley.disgust.doEwwwww(Riley.java:143) at com.insideout.riley.fear.doAaaaahhhh(Riley.java:888) at com.insideout.riley.anger.doAngry(Riley.java:123) at com.insideout.riley.joy.doHappy(Riley.java:456) at com.insideout.riley.sadness.doCry(Riley.java:789) at com.insideout.riley.disgust.doEwwwww(Riley.java:143) at com.insideout.riley.fear.doAaaaahhhh(Riley.java:888) at com.insideout.riley.anger.doAngry(Riley.java:123) at com.insideout.riley.joy.doHappy(Riley.java:456) at com.insideout.riley.sadness.doCry(Riley.java:789) at com.insideout.riley.disgust.doEwwwww(Riley.java:143) at com.insideout.riley.fear.doAaaaahhhh(Riley.java:888) at com.insideout.riley.doPuberty(Riley.java:58)
15/11/09 20:57:42 DEBUGInvalid character in customer name: ‘s@dness’
15/11/09 20:54:56 INFOSuccessfully saved customer record: customer.name=’Joy’
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOfRange(Unknown Source) at java.lang.String.<init>(Unknown Source) at java.io.BufferedReader.readLine(Unknown Source) at java.io.BufferedReader.readLine(Unknown Source) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley.nightmare .overtime(ScaryClown.java:13) at com.insideout.riley.nightmare .deadline(ScaryClown.java:666) at com.insideout.riley .main(ScaryClown.java:1)
Log Data Tell You Everything/Anything/Nothing● Who tried to login to your app?
● Was the login attempt successful?
● Which pages the user visited?
● What did the user do?
● Did ze make any purchases?
● Did the customer encountered any problems while using your app?
● What were they doing at your website when ‘Tamang Panahon’ was ongoing at Philippine Arena?
More Users = More Logs = TMI !!!
How to make sense of it all???
Data Analyst!
All Your Data!
NOW!
Real-Time Insights Are More:● Relevant
● Actionable
LOG DATA
REAL-TIMEINSIGHTS
???
Logback Appender● Natural integration to your webapp
● Take advantage of logback framework (i.e. specific appenders per module, log level, etc.)
● Easier if you use structured data for your kafka messages (i.e. JSON, XML, etc.)
● Couple of OSS appenders available, but doesn’t seem to have an “official logback version”
Apache Kafka● Publish-Subscribe Messaging
● Fast, Scalable, Durable
● Distributed by Design
● Well Integrated to Hadoop Ecosystem
● … makes you yell ‘Hooray!’
Analytics Platform● Apache Spark Streaming (micro-batch)
● Apache Storm (real-time stream processing)
Summary● Application logging is important! Tells the story of your webapp.
● Not all useful data are stored in your database.
● Listening to your webapp will give you better insights to your business.
● Data is becoming the hottest commodity right now. The faster you are able to utilize it, the more edge you will have.
Questions?