MAKE SENSE OF YOUR BIG DATA
-
Upload
treeptik -
Category
Engineering
-
view
31 -
download
0
Transcript of MAKE SENSE OF YOUR BIG DATA
![Page 1: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/1.jpg)
‹#›
make sense of your (BIG) data!
David Pilato Developer | Evangelist@dadoonet
![Page 2: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/2.jpg)
‹#›
![Page 3: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/3.jpg)
3
![Page 4: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/4.jpg)
4
![Page 5: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/5.jpg)
5
![Page 6: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/6.jpg)
6
![Page 7: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/7.jpg)
6
![Page 8: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/8.jpg)
7
![Page 9: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/9.jpg)
7
![Page 10: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/10.jpg)
8
![Page 11: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/11.jpg)
8
![Page 12: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/12.jpg)
9
![Page 13: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/13.jpg)
9
![Page 14: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/14.jpg)
Big data?
10
![Page 15: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/15.jpg)
11
![Page 16: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/16.jpg)
12 Source: http://www.csc.com/insights/flxwd/78931-big_data_just_beginning_to_explode
![Page 17: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/17.jpg)
12 Source: http://www.csc.com/insights/flxwd/78931-big_data_just_beginning_to_explode
35.000.000.000.000.000 mb
![Page 18: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/18.jpg)
13 Source: http://www.domo.com/learn/data-never-sleeps-2
![Page 19: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/19.jpg)
![Page 20: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/20.jpg)
Some dataCREATE TABLE user( name VARCHAR(100), comments VARCHAR(1000));INSERT INTO user VALUES ('David Pilato', 'Developer at elastic');INSERT INTO user VALUES ('Malloum Laya', 'Worked with David at french customs service');INSERT INTO user VALUES ('David Gageot', 'Engineer at Docker');INSERT INTO user VALUES ('David David', 'Who is that guy?');
15
![Page 21: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/21.jpg)
Search on term
SELECT * FROM user WHERE name="David";Empty set (0,00 sec)
INSERT INTO user VALUES ('David Pilato', 'Developer at elastic');INSERT INTO user VALUES ('Malloum Laya', 'Worked with David at french customs service');INSERT INTO user VALUES ('David Gageot', 'Engineer at Docker');INSERT INTO user VALUES ('David David', 'Who is that guy?');
16
![Page 22: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/22.jpg)
Search like
SELECT * FROM user WHERE name LIKE "%David%";+--------------+----------------------+| name | comments |+--------------+----------------------+| David Pilato | Developer at elastic || David Gageot | Engineer at Docker || David David | Who is that guy? |+--------------+----------------------+
INSERT INTO user VALUES ('David Pilato', 'Developer at elastic');INSERT INTO user VALUES ('Malloum Laya', 'Worked with David at french customs service');INSERT INTO user VALUES ('David Gageot', 'Engineer at Docker');INSERT INTO user VALUES ('David David', 'Who is that guy?');
17
![Page 23: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/23.jpg)
Search like
SELECT * FROM user WHERE name LIKE "%David%Pilato%";+--------------+----------------------+| name | comments |+--------------+----------------------+| David Pilato | Developer at elastic |+--------------+----------------------+
INSERT INTO user VALUES ('David Pilato', 'Developer at elastic');INSERT INTO user VALUES ('Malloum Laya', 'Worked with David at french customs service');INSERT INTO user VALUES ('David Gageot', 'Engineer at Docker');INSERT INTO user VALUES ('David David', 'Who is that guy?');
18
![Page 24: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/24.jpg)
Search like with inverted terms
SELECT * FROM user WHERE name LIKE "%Pilato%David%";Empty set (0,00 sec)
INSERT INTO user VALUES ('David Pilato', 'Developer at elastic');INSERT INTO user VALUES ('Malloum Laya', 'Worked with David at french customs service');INSERT INTO user VALUES ('David Gageot', 'Engineer at Docker');INSERT INTO user VALUES ('David David', 'Who is that guy?');
19
![Page 25: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/25.jpg)
Search in two fields
SELECT * FROM user WHERE name LIKE "%David%" OR comments LIKE "%David%";+--------------+---------------------------------------------+| name | comments |+--------------+---------------------------------------------+| David Pilato | Developer at elastic || Malloum Laya | Worked with David at french customs service || David Gageot | Engineer at Docker || David David | Who is that guy? |+--------------+---------------------------------------------+
INSERT INTO user VALUES ('David Pilato', 'Developer at elastic');INSERT INTO user VALUES ('Malloum Laya', 'Worked with David at french customs service');INSERT INTO user VALUES ('David Gageot', 'Engineer at Docker');INSERT INTO user VALUES ('David David', 'Who is that guy?');
20
![Page 26: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/26.jpg)
21
![Page 27: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/27.jpg)
22
![Page 28: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/28.jpg)
23
![Page 29: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/29.jpg)
search engine?
24
![Page 30: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/30.jpg)
search engine?
24
![Page 31: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/31.jpg)
25
![Page 32: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/32.jpg)
25
Lucene
![Page 33: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/33.jpg)
25
REST/JSON Lucene
![Page 34: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/34.jpg)
25
REST/JSON
scalable
Lucene
![Page 35: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/35.jpg)
25
plug & play
REST/JSON
scalable
Lucene
![Page 36: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/36.jpg)
25
plug & play
REST/JSON
scalable
Apache 2 license
Lucene
![Page 37: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/37.jpg)
start…
26
$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.0.0-beta1.tar.gz$ tar -xf elasticsearch-5.0.0-beta1.tar.gz$ ./elasticsearch-5.0.0-beta1/bin/elasticsearch[INFO ][node ][Ghost Maker] version[5.0.0-beta1], pid[72965], …[INFO ][transport][Ghost Maker] publish_address {[/127.0.0.1:9300]}[INFO ][http ][Ghost Maker] publish_address {[/127.0.0.1:9200]}[INFO ][node ][Ghost Maker] started
![Page 38: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/38.jpg)
… and play!
27
$ curl -XPUT localhost:9200/sessions/session/1 -d '{ "title" : "Elasticsearch", "subtitle" : "Make sense of your (BIG) data !", "date" : "2016-10-07T16:30:00", "tags" : [ "realtime", "bigdata" ], "speakers" : [{ "first_name" : "David", "last_name" : "Pilato" }]}'
![Page 39: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/39.jpg)
search!
28
$ curl http://localhost:9200/sessions/session/_search -d' { "query": { "multi_match": { "query": "elasticsearch bigdata david", "fields": [ "title^3", "tags^2", "speakers.first_name" ] } }, "post_filter": { "range": { "date": { "from": "2016-10-01", "to": "2016-11-01" } } } }'
![Page 40: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/40.jpg)
compute?
29
![Page 41: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/41.jpg)
compute!$ curl http://localhost:9200/sessions/session/_search -d' { "query": { ... }, "aggs": { "by_date": { "date_histogram": { "field": "date", "interval": "day", "format" : "dd/MM/yyyy" } } } }'
30
![Page 42: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/42.jpg)
compute!$ curl http://localhost:9200/sessions/session/_search -d' { "query": { ... }, "aggs": { "by_date": { "date_histogram": { "field": "date", "interval": "day", "format" : "dd/MM/yyyy" } } } }'
30
"by_date": [ { "key_as_string": "03/10/2016", "doc_count": 1 }, { "key_as_string": "07/10/2016", "doc_count": 2 }, { "key_as_string": "10/10/2016", "doc_count": 3 } ]
![Page 43: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/43.jpg)
compute!$ curl http://localhost:9200/sessions/session/_search -d' { "query": { ... }, "aggs": { "by_date": { "date_histogram": { "field": "date", "interval": "day", "format" : "dd/MM/yyyy" } } } }'
30
"by_date": [ { "key_as_string": "03/10/2016", "doc_count": 1 }, { "key_as_string": "07/10/2016", "doc_count": 2 }, { "key_as_string": "10/10/2016", "doc_count": 3 } ]
![Page 44: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/44.jpg)
![Page 45: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/45.jpg)
Let’s make sense of …
• logs • twitter • github • marketing data • ... • your data • your big data
32
![Page 46: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/46.jpg)
Let’s make sense of …
• logs • twitter • github • marketing data • ... • your data • your big data
33
{ "name":"Pilato David", "dateOfBirth":"1971-12-26", "gender":"male", "children":3, "marketing":{ "fashion":334, "music":3363, "hifi":2351 }, "address":{ "country":"France", "city":"Paris", "location": [2.332395, 48.861871] } }
![Page 47: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/47.jpg)
Let's inject 1 000 000 marketing documents
34
Demo
![Page 48: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/48.jpg)
‹#›
Demo
35
![Page 49: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/49.jpg)
36
infomercial
![Page 50: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/50.jpg)
37
The only Elasticsearch as a Service offering powered by the creators of the Elastic Stack
• Always runs on the latest software
• One-click to scale/upgrade with no downtime
• Free Kibana and backups every 30 minutes
• Dedicated, SLA-based support
• Easily add X-Pack features: security (Shield), alerting (Watcher), and monitoring (Marvel)
• Pricing starts at $45 a month
infomercial
![Page 51: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/51.jpg)
![Page 52: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/52.jpg)
39
![Page 53: MAKE SENSE OF YOUR BIG DATA](https://reader031.fdocuments.in/reader031/viewer/2022021922/586fd8d91a28ab18428b573d/html5/thumbnails/53.jpg)
‹#›
https://www.elastic.co/subscriptions
Thank you!
David Pilato Developer | Evangelist@dadoonet