Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr...

Post on 11-May-2015

968 views 1 download

Tags:

description

While Solr and Lucene were originally written for full-text search, they are capable and increasingly used for Analytics, as Key Value Stores, NoSQL databases, and more. In this session we'll describe our experience with Solr for Analytics. More specifically, we will describe a couple of different approaches we have taken with SolrCloud for aggregation of massive amounts of performance metrics, we'll share our findings, and compare SolrCloud with HBase for large-scale, write-intensive aggregations. We'll also visit several Solr new features that are in the works that will make Solr even more suitable for Analytics workloads.

Transcript of Solr for Analytics: Metrics Aggregations at Sematext, Presented by Otis Gospodnetic at Lucene/Solr...

+L���

���������������%LJ�GDWD�LV�OLNH�WHHQDJH�VH[������������������HYHU\RQH�WDONV�DERXW�LW������������������QRERG\�UHDOO\�NQRZV�KRZ�WR�GR�LW������������������HYHU\RQH�WKLQNV�HYHU\RQH�HOVH�LV�GRLQJ�LW������������������VR�HYHU\RQH�FODLPV�WKH\�DUH�GRLQJ�LW����

������������������'DQ�$ULHO\��'XNH�8QLYHUVLW\

6ROU�IRU�$QDO\WLFV0HWULFV�$JJUHJDWLRQV�DW�6HPDWH[W

2WLV�*RVSRGQHWLü�#RWLVJ

2*���3UR

«�VLQFH�FLUFD������

«��VLQFH�����

1HHGOH�$GGLFWLRQ

:DWHUPHORQ�$GGLFWLRQ

%UHDG�/RYH

1XWV�

$GGLFWLRQ�0DQLIHVWDWLRQ����

*DWKHULQJ�䘟

6HDUFK�����䘟

$QDO\WLFV��䘟

/I<gEP��[<YsjQEh

$GGLFWLRQ�0DQLIHVWDWLRQ���

*DWKHULQJ�䘟

6HDUFK�����䘟

$QDO\WLFV��䘟

/+!

$GGLFWLRQ�0DQLIHVWDWLRQ���

*DWKHULQJ�䘟

6HDUFK�����䘟

$QDO\WLFV��䘟

]OhI[I

7KH�6XUIDFH

8QGHU�WKH�+RRG

8QGHU�WKH�+RRG

6WULSSHG

�GLPHQVLRQ/LVW�QDPH �VROUBFDFKHVBPHWULFVBFXEH�IRUPXOD �V\VWHP �PRQWK�ZHHN�GD\�KRXU�ILYHBPLQXWHV�PLQXWH� VROUBFDFKHBQDPH KRVW VROUBFRUHBQDPH��!

6FKHPD

�ILHOG�QDPH �LG������������������������W\SH �VWULQJ���LQGH[HG �WUXH����VWRUHG �WUXH��UHTXLUHG �WUXH��PXOWL9DOXHG �IDOVH��!

�ILHOG�QDPH �EDVH5RZ.H\���W\SH �ELQDU\��LQGH[HG �IDOVH��VWRUHG �WUXH��UHTXLUHG �WUXH��PXOWL9DOXHG �IDOVH��!

�ILHOG�QDPH �GLP/LVW,G�����������W\SH �LQW���������LQGH[HG �WUXH����VWRUHG �WUXH��UHTXLUHG �WUXH��PXOWL9DOXHG �IDOVH��!

�ILHOG�QDPH �PHWULFV'DWD������W\SH �ELQDU\��LQGH[HG �IDOVH��VWRUHG �WUXH��UHTXLUHG �WUXH��PXOWL9DOXHG �IDOVH��!

�G\QDPLF)LHOG�QDPH �GLPBVWUB ������W\SH �VWULQJ��LQGH[HG �WUXH���VWRUHG �IDOVH���PXOWL9DOXHG �WUXH��GRF9DOXHV �WUXH��!

�G\QDPLF)LHOG�QDPH �GLPBQXPB ���W\SH �WORQJ���LQGH[HG �WUXH���VWRUHG �IDOVH���PXOWL9DOXHG �WUXH��GRF9DOXHV �WUXH��!

���GLPHQVLRQ/LVW�QDPH �VROUBFDFKHVBPHWULFVBFXEH�����IRUPXOD �V\VWHP �PRQWK�ZHHN�GD\�KRXU�ILYHBPLQXWHV�PLQXWH� VROUBFDFKHBQDPH KRVW VROUBFRUHBQDPH��!

$SSHQG�RQ�:ULWH��&RPSDFW�/DWHU"

&RPSDFWLRQ«�PD\EH"

,GHDOO\

JLWKXE�FRP

���VHPDWH[W���+%DVH+

87

0HUJH�RQ�5HDG

0HUJH�RQ�5HDG

0HWULF�5HWULHYDO

SDUDPV ^VKDUG�NH\V ����B����FROOHFWLRQ SDUWLDO3UHDJJ&ROOHFWLRQZW MDYDELQTW VHOHFWVWDUW �T GLP/LVW,G����IT GLPBQXPB��>��������������72��������������@IT GLPBVWUB��0M0�24��'��'IT GLPBVWUB����0'JZ<7OO1]JW17=N0\��2'KO/:(�1*8W=P0�2'(�1*1N1*1N�YHUVLRQ �URZV �����`KLWV ���VWDWXV ��47LPH ��

:ULWH�)ULHQGO\�3UHVHQW

62/5��������&ORXG6ROU6HUYHU62/5��������6WUHDPLQJ�XSGDWHV�EHWZHHQ�QRGHV62/5��������-DYDELQ�EHWZHHQ�QRGHV62/5�""""���$JJUHJDWLQJ�853

����YV�����

������QR�URXWLQJ�YV�

����URXWLQJ�YLD�&ORXG6ROU6HUYHU

)DVWHU�$)6�IOXVKHV)DVWHU�LQGH[LQJ������0�YV����0�GRFV�LQ����PLQ/RZHU�KHDS��FLUFD����

'RF9DOXHV

�ILHOG�QDPH �LG���������������������������������W\SH �VWULQJ��LQGH[HG �WUXH���VWRUHG �WUXH�����UHTXLUHG �WUXH�������GRF9DOXHV �WUXH��!

�G\QDPLF)LHOG�QDPH �GLPBVWUB ������W\SH �VWULQJ��LQGH[HG �WUXH���VWRUHG �IDOVH���PXOWL9DOXHG �WUXH��GRF9DOXHV �WUXH��!

�G\QDPLF)LHOG�QDPH �GLPBQXPB ���W\SH �WORQJ���LQGH[HG �WUXH���VWRUHG �IDOVH���PXOWL9DOXHG �WUXH��GRF9DOXHV �WUXH��!

)RUZDUG�LQGH[�GDWD�VWUXFWXUH«�LQ�PHPRU\�DQG�RU�RQ�GLVN«�FRPSUHVVLEOH«�RII�KHDS

6HHKWWSV���FZLNL�DSDFKH�RUJ�FRQIOXHQFH�GLVSOD\�VROU�'RF9DOXHVKWWS���VHDUFKKXE�RUJ������������IXQ�ZLWK�GRFYDOXHV�LQ�VROU�����

�'RF9DOXHV�YV��'RF9DOXHV

:LWK�'RF9DOXHV�«�VPDOOHU�KHDS«�PRUH�QXPHURXV��EXW�VPDOOHU�*&V«�URXJKO\�HTXDO�TXHU\�ODWHQF\«�IDVWHU�IDFHWLQJ��VRUWLQJ��DQG�JURXSLQJ��EXW�ZH�DUH�QRW�XVLQJ�WKDW�LQ�630�ZLWK�6ROU�«�GLGQ¶W�VHH�VORZ�GRZQ�LQ�LQGH[LQJ��KPPP�

�'RF9DOXHV�YV��'RF9DOXHV

$QDO\WLFV�)ULHQGO\�)XWXUH

/8&(1(��������1XPHULF�'RF9DOXHV�8SGDWHV62/5��������'LVWULEXWHG�3LYRW�)DFHWV62/5��������$QDO\WLFV�&RPSRQHQW62/5��������&RQILJXUDEOH�&ROOHFWRUV62/5��������3OXJJDEOH�$QDO\WLFV62/5��������)LHOG�&ROODSVLQJ�3RVW)LOWHU

),1���4XHVWLRQV�

������hIZ<jIrj�E]Z�<D]kj�W]Dh������ÃÁÚ�]NN�� .ÃÁÂÄ/+!ÃÁ���hdZ

���������������Y]OhI[I����������������������������������������hI<gEP�<[<YsjQEh

RWLV#VHPDWH[W�FRP³]jQhO��������������������������³hIZ<jIrj

)LOWHU�&ROOHFWLRQ