Utilizing Groovy based closures for flexible real-time analytics
-
Upload
spring-io -
Category
Technology
-
view
419 -
download
0
description
Transcript of Utilizing Groovy based closures for flexible real-time analytics
© Hortonworks Inc. 2013
Utilizing Groovy based closures for flexible real-time analytics
Mark Johnson [email protected]
Page 1
© Hortonworks Inc. 2013
Who Am I?
• Director Consulting Hortonworks • Distributed Compute fan • Lazy programmer who prefers to reuse rather than re-implement code • President New England Java Users Group
Page 2 Architecting the Future of Big Data
© Hortonworks Inc. 2013 Page 3 Architecting the Future of Big Data
Real – Time Analytics
Discovery Analytics
© Hortonworks Inc. 2013
Real-Time Analytics
Real-time analytics is the use of, or the capacity to use, all available enterprise data and resources when they are needed. It consists of dynamic analysis and reporting, based on data entered into a system less than one minute before the actual time of use.
Page 4 Architecting the Future of Big Data
© Hortonworks Inc. 2013 Page 5 Architecting the Future of Big Data
Real-Time Analytic
Challenges
Lost Bits
Changing Analytic models
too much time to execute
© Hortonworks Inc. 2013
Architectural Tradeoffs
Page 6 Architecting the Future of Big Data
Changeability Requirements: • 24 x 7 processing • Models change • JVM class reload issues
Performance Requirements: • Required reaction time • Resource consumption
© Hortonworks Inc. 2013
Groovy Closures
A Groovy Closure is a code block which is usable via a variable instance for later execution.
Page 7 Architecting the Future of Big Data
© Hortonworks Inc. 2013
Groovy Closure: Example
Page 8 Architecting the Future of Big Data
© Hortonworks Inc. 2013
GroovyShell
Page 9 Architecting the Future of Big Data
© Hortonworks Inc. 2013
Groovy closures and Hotspot
Page 10 Architecting the Future of Big Data
• Groovy script improves with hotspot execution
0
20
40
60
80
100
120
140
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
© Hortonworks Inc. 2013
Architecture
Page 11 Architecting the Future of Big Data
© Hortonworks Inc. 2013 Page 12 Architecting the Future of Big Data
© Hortonworks Inc. 2013
Solution: Dynamic Code deployment
• Embed GroovyShell() into service activator • Reference an externally defined Groovy script from a service
activator base method implementation.
• Groovy script may be located most anywhere: – Property file – Database – External text file
Page 13 Architecting the Future of Big Data
© Hortonworks Inc. 2013
Service Activator Example
Page 14 Architecting the Future of Big Data
© Hortonworks Inc. 2013
Solution: Load Management
Page 15 Architecting the Future of Big Data
© Hortonworks Inc. 2013
Solution: Fast / High Capacity Archival Storage
• Archival data necessary to back test real-time analytics • Sensor Data samples can lead to erroneous conclusions
Page 16 Architecting the Future of Big Data
Real – Time Analytics
Discovery Analytics
© Hortonworks Inc. 2013
Demo
Architecting the Future of Big Data Page 17