JavaOne 2008 - TS-6048 - Complex Event Processing at Orbitz
-
Upload
matt-okeefe -
Category
Technology
-
view
2.688 -
download
0
description
Transcript of JavaOne 2008 - TS-6048 - Complex Event Processing at Orbitz
COMPLEX EVENT PROCESSING AT ORBITZ WORLDWIDE
Matt O’Keefe, Senior ArchitectDoug Barth, Technical Lead
TS-6048
2008 JavaOneSM Conference | java.sun.com/javaone | 2
How to make pager duty suck less
2008 JavaOneSM Conference | java.sun.com/javaone | 3
$10.8 Billion in Gross Bookings in 2007
2008 JavaOneSM Conference | java.sun.com/javaone | 4
Dozens of Apps, Hundreds of VMs and Thousands of Services
2008 JavaOneSM Conference | java.sun.com/javaone | 5
The Need for Abstraction
Webapp
Travel Business Services
Switching Services
ab
stra
ctio
n
Transaction Services
Suppliers
2008 JavaOneSM Conference | java.sun.com/javaone | 6
Complex Event Processing is…
…about sensing and responding to threats and opportunities in real time.
2008 JavaOneSM Conference | java.sun.com/javaone | 7
The Big Picture
Event ProcessorsMonitored Apps
OperationsCenter
Graphite
JMX, ssh
ERMA
SNMP
Portal
2008 JavaOneSM Conference | java.sun.com/javaone | 8
ERMA
Extremely Reusable Monitoring API
2008 JavaOneSM Conference | java.sun.com/javaone | 9
The Monitor Interface
2008 JavaOneSM Conference | java.sun.com/javaone | 10
Using EventMonitors
protected void doValidate(RequestContext context, Object formObject, Errors errors) throws Exception {
super.doValidate(context, formObject, errors);
if (errors.hasErrors()) { EventMonitor validationMonitor = new EventMonitor("ValidationErrors"); validationMonitor.set("errors", errors.getAllErrors()); validationMonitor.fire(); }}
2008 JavaOneSM Conference | java.sun.com/javaone | 11
The CompositeMonitor Interface
2008 JavaOneSM Conference | java.sun.com/javaone | 12
Using TransactionMonitors
public Hotel findHotelById(Long id) { TransactionMonitor monitor = new TransactionMonitor(getClass(), "findHotelById"); monitor.set(“id”, id); try { Hotel hotel = em.find(Hotel.class, id); monitor.succeeded(); return hotel; } catch (RuntimeException e) { monitor.failedDueTo(e); throw e; } finally { monitor.done(); }}
2008 JavaOneSM Conference | java.sun.com/javaone | 13
TransactionMonitorTemplate
public void cancelBooking(final Long id) { TransactionMonitorTemplate.INSTANCE.doInMonitor( getClass(), "cancelBooking", new TransactionMonitorCallback() { public Object doInMonitor(TransactionMonitor monitor) { monitor.set("id", id); Booking booking = em.find(Booking.class, id); if (booking != null) { em.remove(booking); } return null; } });}
2008 JavaOneSM Conference | java.sun.com/javaone | 14
Interceptors
public interface HandlerInterceptor {
boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception;
void postHandle(HttpServletRequest request, HttpServletResponse response, Object handler, ModelAndView modelAndView) throws Exception;
void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex) throws Exception;}
2008 JavaOneSM Conference | java.sun.com/javaone | 15
Listenerspublic interface FlowExecutionListener { public void requestSubmitted(RequestContext context); public void requestProcessed(RequestContext context); public void sessionStarting(RequestContext context, FlowDefinition definition, MutableAttributeMap input); public void sessionCreated(RequestContext context, FlowSession session); public void sessionStarted(RequestContext context, FlowSession session); public void eventSignaled(RequestContext context, Event event); public void stateEntering(RequestContext context, StateDefinition state) throws EnterStateVetoException; public void stateEntered(RequestContext context, StateDefinition previousState, StateDefinition state); public void paused(RequestContext context, ViewSelection selectedView); public void resumed(RequestContext context); public void sessionEnding(RequestContext context, FlowSession session, MutableAttributeMap output); public void sessionEnded(RequestContext context, FlowSession session, AttributeMap output); public void exceptionThrown(RequestContext context, FlowExecutionException exception);}
2008 JavaOneSM Conference | java.sun.com/javaone | 16
Annotations
@Monitoredpublic interface UnderpantsGnomes { void collectUnderpants(); void ?(); void profit(); }
2008 JavaOneSM Conference | java.sun.com/javaone | 17
Spring/AspectJ
<aop:config> <aop:aspect id="transactionMonitorActionAspect" ref="transactionMonitorActionAdvice"> <aop:pointcut id="transactionMonitorActionPointcut“ expression="target(org.springframework.webflow.execution.Action) and args(context)"/> <aop:around pointcut-ref="transactionMonitorActionPointcut“ method="invoke"/> </aop:aspect></aop:config>
<bean id="transactionMonitorActionAdvice" class= "c.o.webframework.aop.aspectj.TransactionMonitorActionAdvice"/>
2008 JavaOneSM Conference | java.sun.com/javaone | 18
Joining Monitors - Across Threads
TransactionMonitor monitor = new TransactionMonitor("Profit"); ErmaCallable callable = new ErmaCallable(new CollectUnderpantsCallable()); try { Future future = executorService.submit(callable); future.get(FUNDING_DURATION, TimeUnit.MILLISECONDS); monitor.addChildMonitor(callable.getMonitor()); monitor.succeeded(); } catch (TimeoutException e) { monitor.failedDueTo(e); throw e; } finally { monitor.done(); }
2008 JavaOneSM Conference | java.sun.com/javaone | 19
Joining Monitors - Distributed Servicespublic DispatcherResponse doFilter(DispatcherRequest request, Object resource, FilterChain chain) throws Throwable { TransactionMonitor monitor = new TransactionMonitor(getName(request)); MonitoringEngine mEngine = MonitoringEngine.getInstance(); Map inheritableAttributes = mEngine.getInheritableAttributes(); Map serializableAttributes = mEngine.makeSerializable(inheritableAttributes); HashMap ermaAttributes = new HashMap(serializableAttributes); request.addParameter(ERMA_FILTER_PARAM_KEY, OLCSerializer.serialize(ermaAttributes));
DispatcherResponse response = chain.doFilter(request, resource);
… return response;}
2008 JavaOneSM Conference | java.sun.com/javaone | 20
Distributed Services (cont.)
public DispatcherResponse doFilter(DispatcherRequest request, Object resource, FilterChain chain) throws Throwable { …
DispatcherResponse response = chain.doFilter(request, resource);
Object responseBlob = response.getParameter(ERMA_FILTER_PARAM_KEY); Monitor responseMonitor = (Monitor) OLCSerializer.deserialize((byte[]) responseBlob); monitor.addChildMonitor(responseMonitor); Throwable throwable = response.getThrowable(); if (throwable == null) { monitor.succeeded(); } else { monitor.failedDueTo(throwable); } return response;}
2008 JavaOneSM Conference | java.sun.com/javaone | 21
The MonitorProcessor Interface
public interface MonitorProcessor { public void startup();
public void shutdown();
public void monitorCreated(Monitor monitor);
public void monitorStarted(Monitor monitor);
public void process(Monitor monitor);}
2008 JavaOneSM Conference | java.sun.com/javaone | 22
A MonitorProcessorAdaptor Examplepublic class ResultCodeAnnotatingMonitorProcessor extends MonitorProcessorAdapter {
public void process(Monitor monitor) { if (monitor.hasAttribute("failureThrowable")) { Throwable t = (Throwable) monitor.get("failureThrowable"); while (t.getCause() != null) { t = t.getCause(); } monitor.set("resultCode", t.getClass().getName()); } else { monitor.set("resultCode", "success"); } }}
2008 JavaOneSM Conference | java.sun.com/javaone | 23
LoggingMonitorProcessor Output
process: c.o.monitoring.api.monitor.TransactionMonitor -> blockedCount = 9 -> blockedTime = 17 -> cpuTimeMillis = 1470.0 -> createdAt = Sun Mar 09 16:13:17 CDT 2008 -> endTime = Sun Mar 09 16:13:30 CDT 2008 -> failed = false -> hostname = oberon -> latency = 13180 -> name = httpIn_/shop/airsearch/search/air/pageView_airResults -> remoteIpAddress = 10.222.186.147 -> sessionId = D417C9CC585F82E1 -> threadId = 2a20ec -> validationFailure = false -> vmid = wl -> waitedCount = 10 -> waitedTime = 10414
2008 JavaOneSM Conference | java.sun.com/javaone | 24
EventPatternLoggingMonitorProcessor Outputwl|httpIn_/shop/airsearch/search/air/pageView_airResults|13180 wl|RoundTripAirSearchAction.resolveRoundTripAirLocations|136 wl|jiniOut_LocationFinderService_findAirports|84 tbs-shop-13.31|jiniIn_LocationFinderService_findAirports|31 tbs-shop-13.31|jiniOut_AirportLookupService_findLocationByIATACode|9 market-8.4|jiniIn_AirportLookupService_findLocationByIATACode|3 tbs-shop-13.31|jiniOut_AirportLookupService_findLocationByIATACode|15 market-8.4|jiniIn_AirportLookupService_findLocationByIATACode|10 wl|jiniOut_LocationFinderService_findAirports|48 tbs-shop-13.31|jiniIn_LocationFinderService_findAirports|16 tbs-shop-13.31|jiniOut_AirportLookupService_findLocationByIATACode|14 market-8.4|jiniIn_AirportLookupService_findLocationByIATACode|10 wl|AirSearchExecuteAction.search|10422 wl|jiniOut_ShopService_createResultSet|9798 tbs-shop-13.31|jiniIn_ShopService_createResultSet|9601 tbs-shop-13.31|com.orbitz.tbs.host.shop.ShopServiceImpl.createResultSet.AIR|9361 tbs-shop-13.31|com.orbitz.tbs.spi.SpiShopService.createResultSet.AIR|9333 tbs-shop-13.31|jiniOut_LowFareSearchService_execute|9175 air-search-7.2.1|jiniIn_LowFareSearchService_execute|9094 air-search-7.2.1|LowFareSearchRequest|9048 air-search-7.2.1|com.orbitz.afo.lib.search.service.LowFareSearchServiceImpl|9038 air-search-7.2.1|com.orbitz.afo.lib.search.service.LowFareSearchServiceImpl.execute|9037 wl|jiniOut_ShopService_viewResultSet|607 tbs-shop-13.31|jiniIn_ShopService_viewResultSet|486 wl|pageView_airResults wl|jsp.render.air200.page|2475
2008 JavaOneSM Conference | java.sun.com/javaone | 25
EventPatternLoggingMonitorProcessor Output
wl|AirSearchExecuteAction.search|NoSearchResultsAvailableException wl|jiniOut_ShopService_createResultSet|NoSearchResultsAvailableException tbs-shop|jiniIn_ShopService_createResultSet|NoSearchResultsAvailableException tbs-shop|c.o.t.h.s.ShopServiceImpl.createResultSet.AIR|NoSearchResultsAvailableException tbs-shop|c.o.t.s.SpiShopService.createResultSet.AIR|NoSearchResultsAvailableException tbs-shop|jiniOut_LowFareSearchService_execute|SearchSolutionNotFoundException air-search|jiniIn_LowFareSearchService_execute|SearchSolutionNotFoundException air-search|LowFareSearchRequest|SearchSolutionNotFoundException
Follow the trail of Exceptions… don’t bother the on-call engineers for the higher layers… save time by narrowing your log search query!
2008 JavaOneSM Conference | java.sun.com/javaone | 26
Event Processing
2008 JavaOneSM Conference | java.sun.com/javaone | 27
EventFlow
2008 JavaOneSM Conference | java.sun.com/javaone | 28
Expression Language
!failed
latency > 1000 && failed
totalFailed/total
sum(int(failed))
stddev(latency)
indexof(name, ‘jini’) == 0
strftime(“%Y%m%d, windowTimestamp)
2008 JavaOneSM Conference | java.sun.com/javaone | 29
Event Processing DemoStreamBase Studio
2008 JavaOneSM Conference | java.sun.com/javaone | 30
Custom Functions
public class HelloWorld { public static String sayHello(String name) { return "Hello, “ + name; }}
2008 JavaOneSM Conference | java.sun.com/javaone | 31
Custom Operators
public void processTuple(int inputPortId, Tuple tuple) throws StreamBaseException {
String name = tuple.getString("name"); Tuple output = outputSchema.createTuple(); output.setField("message", "Hello, "+name); sendOutput(OUTPUT_PORT, output);}
2008 JavaOneSM Conference | java.sun.com/javaone | 32
Visualization
2008 JavaOneSM Conference | java.sun.com/javaone | 33
SNMP
2008 JavaOneSM Conference | java.sun.com/javaone | 34
Graphite
2008 JavaOneSM Conference | java.sun.com/javaone | 35
Graphite RESTful URLstarget=tbs-shop.all.jiniIn.ShopService.createResultSet#all.count
2008 JavaOneSM Conference | java.sun.com/javaone | 36
Graphite RESTful URLs - Wildcardstarget=tbs-shop.all.jiniIn.ShopService.*.count
2008 JavaOneSM Conference | java.sun.com/javaone | 37
Graphite RESTful URLs - Functionstarget=derivative(tbs-shop.all.jiniIn.ShopService.*.count)
2008 JavaOneSM Conference | java.sun.com/javaone | 38
Graphite RESTful URLs – Pie Chartstarget=cpuTime:790&target=waitedTime:4043&target=blockedTime:0&target=other:31
2008 JavaOneSM Conference | java.sun.com/javaone | 39
Graphite RESTful URLs – Raw CSV Datatarget=tbs-shop.all.jiniIn.ShopService.*.count&rawData=true
85 86 82 84 87 83 79 82 79 82 94 91 81
0 0 0 0 0 0 0 0 0 0 0 0 0
None None None None None None None None None None None None None
0 0 0 0 0 0 0 1 0 0 0 0 0
None None None None None None None None None None None None None
85 86 82 84 87 83 79 81 79 82 94 91 81
None None None None None None None None None None None None None
None None None None None None None None None None None None None
None None None None None None None None None None None None None
None None None None None None None None None None None None None
None None None None None None None None None None None None None
89 81 85 83 89 84 78 81 76 84 90 92 84
None None None None None None None None None None None None None
None None None None None None None None None None None None None
None None None None None None None None None None None None None
0 0 0 0 0 0 0 0 0 0 0 0 0
2008 JavaOneSM Conference | java.sun.com/javaone | 40
Graphite CLI - Dashboards
2008 JavaOneSM Conference | java.sun.com/javaone | 41
Monitoring Portal
2008 JavaOneSM Conference | java.sun.com/javaone | 42
RSS
2008 JavaOneSM Conference | java.sun.com/javaone | 43
Command Line Stream Discovery
[mokeefe@egcep03 ~]$ sbc list input-streamsinput-stream DeleteFromThresholdsinput-stream GarbageCollectorStatsinput-stream ListActiveAlarmsInputinput-stream MemoryPoolStatsinput-stream MemoryStatsinput-stream MonitorInputinput-stream MonitoringEngineManager_lifecycleinput-stream ReloadLog4jinput-stream ThreadStatsinput-stream ThresholdInput…
2008 JavaOneSM Conference | java.sun.com/javaone | 44
Command Line Schema Discovery[mokeefe@egcep03 ~]$ sbc describe MonitorInput<stream name=“…" schema=“…" uuid="..."> <schema name="schema:MonitorInput" uuid=“…"> <field name="name" size="256" type="string"/> <field name="vmid" size="128" type="string"/> <field name="hostname" size="128" type="string"/> <field name="threadId" size="32" type="string"/> <field name="createdAt" type="timestamp"/> <field name="endTime" type="timestamp"/> <field name="latency" type="int"/> <field name="cpuTimeMillis" type="int"/> <field name="failureThrowable" size="16384" type="string"/> <field name="failed" type="bool"/> <field name="resultCode" size="128" type="string"/> <field name="sessionId" size="32" type="string"/> <field name="locale" size="24" type="string"/> <field name="remoteIpAddress" size="15" type="string"/> <field name="posCode" size="4" type="string"/> </schema></stream>
2008 JavaOneSM Conference | java.sun.com/javaone | 45
Command Line Queries
[mokeefe@egcep03 ~]$ sbc dequeue MonitorInput --where "sessionId == '570EA1FABEE015B9' and vmid='tbs-shop-13.31'"jiniOut_AirportLookupService_findLocationByIATACode,tbs-shop-13.31,egbsoneg01.prod.o.com,67484c,2008-03-10 15:40:42.489-0500,2008-03-10 15:40:42.489-0500,2008-03-10 15:40:42.502-0500,13,null,null,null,false,null,null,GBP,570EA1FABEE015B9,English (United Kingdom),10.235.1.119,null,EBUKjiniIn_LocationFinderService_findAirports,tbs-shop-13.31,egbsoneg01.prod.o.com,67484c,2008-03-10 15:40:42.480-0500,2008-03-10 15:40:42.480-0500,2008-03-10 15:40:42.503-0500,23,null,null,null,false,null,success,GBP,570EA1FABEE015B9,English (United Kingdom),10.235.1.119,null,EBUK…
2008 JavaOneSM Conference | java.sun.com/javaone | 46
Event Pattern Monitoring
2008 JavaOneSM Conference | java.sun.com/javaone | 47
Event Pattern Monitoring (cont)
2008 JavaOneSM Conference | java.sun.com/javaone | 48
Event Pattern Monitoring Work In Progress
OrbitzWorldwide
Real-time Clickstream Analysisw/Correlation
2008 JavaOneSM Conference | java.sun.com/javaone | 49
Recap
Event ProcessorsMonitored Apps
OperationsCenter
Graphite
JMX, ssh
ERMA
SNMP
Portal
2008 JavaOneSM Conference | java.sun.com/javaone | 50
Future Directions
Orbitz Worldwide will open source ERMAOrbitz Worldwide will open source Graphite Orbitz Worldwide is considering open sourcing of
orbitz-lib-streambaseEsper integration
2008 JavaOneSM Conference | java.sun.com/javaone | 51
For More Information
The Power of Events: An Introduction to Complex Event Processing in Distributed Enterprise Systems, David Luckham, Addison Wesley Professional, May 2002, ISBN: 0201727897 http://www.complexevents.comERMA and Graphite: Watch http://forum.complexevents.com/ for an announcementhttp://corp.orbitz.com/careers/ StreamBase: http://www.streambase.com/Esper: http://esper.codehaus.org/
2008 JavaOneSM Conference | java.sun.com/javaone | 52
Matt O’Keefe, Senior ArchitectDoug Barth, Technical Lead
TS-6048
2008 JavaOneSM Conference | java.sun.com/javaone | 53
Appendix: Filters public void doFilter(ServletRequest servletRequest, ServletResponse
servletResponse, FilterChain filterChain) throws … { TransactionMonitor monitor = new TransactionMonitor(getClass(), "doFilter"); try { filterChain.doFilter(servletRequest, servletResponse); monitor.succeeded(); } catch (IOException e) { monitor.failedDueTo(e); throw e; } catch (ServletException e) { monitor.failedDueTo(e); throw e; } catch (RuntimeException e) { monitor.failedDueTo(e); throw e; } finally { monitor.done(); } }
2008 JavaOneSM Conference | java.sun.com/javaone | 54
Appendix: Interceptors
public class ERMAInterceptor extends HandlerInterceptorAdapter {
public boolean preHandle(…) throws Exception {
TransactionMonitor monitor = new TransactionMonitor("httpIn"); monitor.setInheritable(POSCODE, getPosCode()); monitor.setInheritable(LOCALE, getLocale()); monitor.setInheritable(CURRENCY, getCurrency()); monitor.setInheritable(CHANNEL, getChannel()); monitor.setInheritable(SESSION_ID, getSessionId()); monitor.setInheritable(IP_ADDR, getClientAddress()); }
2008 JavaOneSM Conference | java.sun.com/javaone | 55
Appendix: Interceptors (cont.)
public void postHandle(…) throws Exception { View view = modelAndView.getView(); if (view == null) { String viewName = modelAndView.getViewName(); EventMonitor monitor = new EventMonitor("pageView_" + viewName); monitor.fire(); } else if (RedirectView.class.isAssignableFrom( view.getClass())) { String redirectUrl = extractDispatcherPath(request); EventMonitor monitor = new EventMonitor("redirect_" + redirectUrl); monitor.fire(); }}
2008 JavaOneSM Conference | java.sun.com/javaone | 56
Appendix: Interceptors (cont.)
public void afterCompletion(…, Exception exception) throws Exception {
TransactionMonitor httpInMonitor = (TransactionMonitor) MonitoringEngine.getInstance(). getCompositeMonitorNamed("httpIn"); if (exception == null) { httpInMonitor.succeeded(); } else { httpInMonitor.failedDueTo(exception); } httpInMonitor.set(Monitor.NAME, constructNewMonitorName(httpInMonitor)); httpInMonitor.done();}
2008 JavaOneSM Conference | java.sun.com/javaone | 57
Appendix: Listeners
public void stateEntering(RequestContext context, StateDefinition nextState) throws EnterStateVetoException {
EventMonitor monitor = new EventMonitor("flowExecution.stateEntering"); StateDefinition currentState = context.getCurrentState(); monitor.set("currentStateId", currentState.getId()); monitor.set("nextStateId", nextState.getId()); monitor.fire();}