OUR STORY WITH ELASTICSEARCH

November 2014

WHERE DO WE COME FROM?

•  ActivePivot by QuartetFS: –  InMemory ultra fast business

intelligence tool –  Mainly for traders and market risk

analysts

•  One of the biggest french success (& secret) story

•  We are big data crunchers for a long time now

HOW DID EVERYTHING START?

•  Created in may 2012

•  We wanted to build the perfect tool to understand the social WEB

•  We started with a very famous

NoSQL engine ! But we quickly had problems

(performance, clustering, query/txns overlaps, etc…)

ELASTICSEARCH 2 YEARS AGO

BigData'ready''Easy'to'use/manage''Performance/Scalability''Analy:cal'capabili:es''Primary'document'storage?'

IT DIDN’T MATCH OUR ANALYTICS REQUIREMENTS

•  FieldData cache –  High memory consumption –  Memory is expensive on the cloud

•  No multi-field & multi-metric aggregations –  We could not build the product of our dreams

•  But ElasticSearch is modular –  We decided to implement our own analytics plugin

A SEARCH ENGINE WITH OLAP SKILLS

•  Support lazy loading of fields

•  Multi-Fields & Multi-Metric aggregations

•  Ultra-Fast & Efficient –  Usage of columnar storage with primitive types –  Sub-seconds queries over tens of millions of elements

CUSTOM AGGREGATION EXAMPLE

The'query'

Our'facet'

2'dimensions'

2'metrics'

The'result'

WE BUILT FOCUSMATIC

WE BUILT FOCUSMATIC

WE BUILT FOCUSMATIC

ELASTICSEARCH GETS EVEN BETTER

•  Release 1.0.0 – February 2014 –  First version of the Aggregation Engine –  Introduction of doc_values

•  Release 1.2.0 – May 2014 –  global_ordinals / Faster Aggregations

•  Release 1.4.0 – November 2014 –  Improv. Circuit Breakers / Safer Aggregations –  Improv. doc_values

•  Every releases since 1.0.0 –  More stability –  More aggregation capabilies

•  We had more time to develop other things!

INTRODUCING LOGMATIC LOG GOODNESS POWERED BY ELASTICSEARCH

•  Introduced Logmatic.io in private beta this year –  Beginning of 2014: A lot of logs projects around us –  Our logs experiment: It was an eye opener! (30 VMs / ~6 apps) –  2 friendly startups tried: they went live –  Market Study: 12 projects launched –  We faced new challenges and had to build a new product!!

OUR TYPICAL USER

?

OUR APPROACH

Customer’s applications Our cloud based infrastructure

CTO, devops, developers Tomorrow, we’ll even have business people. We’ll tell you more…

alerts, reports'

queries'

•  Centralises & enriches all data

•  Fully hosted (SaaS) •  Advanced analytics •  Real-Time

•  Beautiful dataviz •  Rapidfire answers

secured connection'

And'more…'

UDP'TCP'(SSL)'

HTTP(S)'

DEMO

INCOMING FEATURES

•  Integrated Grok parsing: –  Logs shipping should be

the only concern –  Logs structuration is done

totally in the cloud –  We extended Grok to

simplify issues like date parsing

INCOMING FEATURES

•  Integrated Grok parsing: –  Logs shipping should be

the only concern –  Logs structuration is done

totally in the cloud –  We extended Grok to

simplify issues like date parsing

•  And much more… –  Security and limited views –  Dimension contexts –  Complex metrics /

formulas

WE CONTINUE WITH ELASTICSEARCH

•  Scalability

•  Heterogeneity

•  Query performance

•  Great analytics

•  Reactivity of the team

logmatic.io @logmatic_ hello@logmatic.io

We’d love to hear from you and answer the questions you might have