FireEye Splunk InterFireEye + Splunk: Intermediate Guidemediate Guide
[D-1] Spunk · 2019-12-21 · © 2019 SPLUNK INC. [D-1] Spunk , Staff Sales Engineer, Splunk...
Transcript of [D-1] Spunk · 2019-12-21 · © 2019 SPLUNK INC. [D-1] Spunk , Staff Sales Engineer, Splunk...
© 2019 SPLUNK INC.© 2019 SPLUNK INC.
[D-1] ������Spunk����������� ������� �����������������������������
����, Staff Sales Engineer, Splunk Services Japan
2019/9/6
© 2019 SPLUNK INC.
During the course of this presentation, we may make forward-looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC.
The forward-looking statements made in this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward-looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release.
Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. © 2019 Splunk Inc. All rights reserved.
Forward-Looking Statements
THIS SLIDE IS REQUIRED FOR ALL 3 PARTY PRESENTATIONS.
© 2019 SPLUNK INC.
Schema on The Fly
"�����������������#� ��� ��
!����� or ������
© 2019 SPLUNK INC.
1. Splunk�$)#%
2. Splunk����'"
3. � ���+� (��� vs.������ )
4. �(!�����
5. �*�� Splunk&�������
�����
© 2019 SPLUNK INC.
► AWS► Splunk Version 7.3.1► Search Head x 1 �
• ������: C5.4xlarge (16core,32GB)• ���: 80GiB 400IOPS
► Indexer x 6�• ������: C5.4xlarge (16core,32GB)• ���: 200GiB 800IOPS
►�������: auto (750MB)► limits.conf
• �������
��������
© 2019 SPLUNK INC.
►��������������• ��: 315,508,701�• ��: 2019/7� 1��• ��������
• ��• mac_id:��� �� 443,774 (0.0002%)
• station_id:��� �� 203,824 (0.0005%)
• area_id:��� �� 220 (0.45%)
�����
2019/07/31 23:00:00 mac_id=000:001:053:43:33:104 station_id = K102116 area_id=104 ,2.5ghz,nil,D,534333,,10.304,19,1562.632,1057.999,504.272,67.906,46.392,21.236,25.851,30.09,5.724,7373.742,285.24,7136.532,237.177,8.221,1.986,11.03,7.076,1.413,10.693,0.859,0.665
• �������: 32
© 2019 SPLUNK INC.
��������index=bigdata sourcetype=bigdata_traffic
[bigdata_traffic]BREAK_ONLY_BEFORE_DATE =DATETIME_CONFIG =KV_MODE = autoLINE_BREAKER = ([\r\n]+)NO_BINARY_CHECK = trueSHOULD_LINEMERGE = falseTIME_PREFIX = ^TIME_FORMAT = %Y/%m/%d %H:%M:%STZ = Asia/Tokyocategory = Customdescription = Search time field extractions for bigdatadisabled = falsepulldown_type = trueREPORT-00 = bigdata_traffic_csv_fields
���������
© 2019 SPLUNK INC.
��������index=bigdata_idx sourcetype=bigdata_traffic_idx
[bigdata_traffic_idx]BREAK_ONLY_BEFORE_DATE =DATETIME_CONFIG =KV_MODE = noneLINE_BREAKER = ([\r\n]+)NO_BINARY_CHECK = trueSHOULD_LINEMERGE = falseTIME_PREFIX = ^TIME_FORMAT = %Y/%m/%d %H:%M:%STZ = Asia/Tokyocategory = Customdescription = Index time field extractions for bigdatadisabled = falsepulldown_type = trueTRANSFORMS-00 = bigdata_traffic_mac_idTRANSFORMS-01 = bigdata_traffic_station_idTRANSFORMS-02 = bigdata_traffic_area_idTRANSFORMS-03 = bigdata_traffic_others ������ �����
© 2019 SPLUNK INC.
Splunk��� ���������� Splunk���
© 2019 SPLUNK INC.
�������� ��������$(�&'
► ������• %�� (Dense)
► ��� � �• !.����� �• )#����� �• �� ����� �
► ������• "*���• /,���
► ���-+!.► ���� ����
© 2019 SPLUNK INC.
► ���� � =�! ����#��������
► ����������� � =����� �
����1 Indexer "��������� �
© 2019 SPLUNK INC.
����������
index=bigdata
����������
index=bigdata| streamstats count
����������*
index=bigdata| stats count
�����������������
© 2019 SPLUNK INC.
����������1 Indexer�������������
����� �� ����� �� ������ ��
����� 305K���(172.4�)
24K���(2191.0�)
828K���(63.5�)
����� 20K���(2629.1�)
3K(16745.9�)
20K���(2620.5�)
���� �� >���� �� >����� ��
© 2019 SPLUNK INC.
q K��%�*�� CPUIP���]09('�X\�V�� 1 IndexerG�� 50K� 200K 1:,/Z�'!4:�<_@
q %�*��Ua �/��8-J�QD �$2:-) 0����IP�;^�
q 1',06"+�' 1• %�*�Ua �/��8-J�THFCY�`��• R�SE',7�3:#$2:-��Wf�/��8-���de�;�
• SE',7�3:#$2:-����%�*�?O� 1:,NJ�b�L����B=
• Splunk .�&5:�A��>�[�
SplunkMcIP���
© 2019 SPLUNK INC.
Splunk� ����Splunk� ����������������
© 2019 SPLUNK INC.
► ����$�����"&� ����������#�
► ����!������� ������%�
�������������������
$ cd var/lib/splunk/$ ls -l
auditauthDbbigdatabigdata_idxconf2019conf2018defaultdbhistorydb
© 2019 SPLUNK INC.
► db��"�� �������������� %*
► ($������������������!��������colddb�)&
�S2�#������&'�+��
��������������!���
$ cd bigdata/$ ls -l
colddbdatamodel_summarydbthaweddb
© 2019 SPLUNK INC.
► ����(���"������"��*��
► �"����('�����#$�- ���&,%��"�
�������� =�����"��+)�� ��!��
$ cd db/$ ls –l
.bucketManifestCreationTimedb_1485388800_1483228800_0db_1498867200_1501545599_0hot_v1_0GlobalMetaData
© 2019 SPLUNK INC.
Splunk ��������
© 2019 SPLUNK INC.
������������������"������������!
��$: likes (vodka OR cognac)
(vodka OR cognac) = (4) or (2) = (2,4)
likes = (0,1,2,3,4,5,6)
(2,4) and (0,1,2,3,4,5,6) = (2,4)
2% � 4% ���������
��&#���(����)
01
2
3
4
5
6
© 2019 SPLUNK INC.
���������)/����.%�#����+$
hash(1, ”Armit") == 0hash(2, ”Roger") == 7
▶ Lexicon Term�����(-▶ ����*��������������&"▶ 0! Term����� ,'��� 1���� 0
1 1 0 0 1 0 1 0 0
© 2019 SPLUNK INC.
mac_id� ������*
index=bigdatamac_id=000:001:004:30:17:001
station_id� ������*
index=bigdatastation_id=X209019
area_id� ������*
index=bigdataarea_id=88
�������������������������
© 2019 SPLUNK INC.
��������������"�����-!
mac_id station_id area_id
���� � 12.2( 2.2( 36.2(
���� 1,464& 52,895& 134,029&
����� 418,835& 52,895& 9,789,654&
�� 1,464&�)������ 52,895 &)������,���%'&$���� &$�#+*�����������
© 2019 SPLUNK INC.
q 1&-0@��/�+�Z���B��Xc(+�6)�� Z!�#5;0�HR � �_N/�+ (:$)';)���
q (�,=��:$)';/�+ Ua��^�J�#5;0 S?
q (�,�DM�KPLF���*$7;LF�Y] E��
q `T�+�6 O��:$)';�39�62"9+����b�J��Q[�CM���<\
q 5*048%."* 2• #;/-%*I�G?• =>�G?• #5;0 V?���+�6 O�
Splunk �/�+WA���
© 2019 SPLUNK INC.
���������� vs. �������
� ���������������������
© 2019 SPLUNK INC.
► 1,464 ����419,801�����
►��������������� ��������
�������index=bigdata mac_id=000:001:004:30:17:001
© 2019 SPLUNK INC.
���������� ����"'�2#
���mac_id=000:001:000:17:30:001mac_id=000:001:001:17:30:004mac_id=000:001:004:30:17:001mac_id=000:001:004:30:17:004
Term Posting List000 0,1,2,3001 0,1,2,3004 1,2,317 0,1,2,330 0,1,2,3
���):mac_id=000:001:004:30:17:001
Posting:1,2,3
���� � 1,2,3 ��� ��0�$�KV_MODE=auto����� mac_id�1!%(����� � 1 3 ���*& .���"��,/⇒+-����
0
1
2
3
© 2019 SPLUNK INC.
� ����������������$�-�4����"�'+
���mac_id=000:001:000:17:30:001mac_id=000:001:001:17:30:004mac_id=000:001:004:30:17:001mac_id=000:001:004:30:17:004
Term Posting List000 0,1,2,3001 0,1,2,3004 1,2,317 0,1,2,330 0,1,2,3mac_id::000:001:000:17:30:001 0mac_id::000:001:001:17:30:004 1mac_id::000:001:004:30:17:001 2mac_id::000:001:004:30:17:004 3
���0:mac_id::000:001:004:30:17:001
6!���%� 2�� %��5 . ���$�8&37����#%�2/�
��%����(���$���mac_id=000:001:004:30:17:001��������fileds.conf�,)�1*
0
1
2
3
© 2019 SPLUNK INC.
mac_id��������*
index=bigdata_idxmac_id::000:001:004:30:17:001
station_id��������
index=bigdata_idxstation_id::X209019
area_id��������
index=bigdata_idxarea_id::88
���������� �����������������
© 2019 SPLUNK INC.
������������)$#�!-&��(%9* ( �"-&��(%9*)
mac_id station_id area_id
���� � 2.22 (12.22) 2.22 (2.22) 8.22 (36.22)
���� 1,4640 52,8950 134,0290
����� 1,4640 52,8950 134,0290
station_id� �"-&��(%9*��17�!�')�����4�*�����17�!�')�+���)$#�!-&��(%�,�5����
�� �3 ��Splunk� �"����� �86��
( ).� �"-&��(%9*�5/
© 2019 SPLUNK INC.
��������
���� ������
���������
�������� ���
������� 2 ������������� ��
��������
MAJOR = [ ] < > ( ) { } | ! ; , ' " * \n \r \s \t &
? + %21 %26 %2526 %3B %7C %20 %2B %3D --%2520 %5D %5B
%3A %0A %2C %28 %29
MINOR = / : = @ . -$ # % \\ _
© 2019 SPLUNK INC.
������������+�)�/-".�;5�$�,�=����
����mac_id=000:001:000:17:30:001mac_id=000:001:001:17:30:004mac_id=000:001:004:30:17:001mac_id=000:001:004:30:17:004
Term Posting List000 0,1,2,3001 0,1,2,3004 1,2,317 0,1,2,330 0,1,2,3mac_id=000:001:000:17:30:001 0mac_id=000:001:001:17:30:004 1mac_id=000:001:004:30:17:001 2mac_id=000:001:004:30:17:004 3
�0'& #6*��/(E1������A����-".�$�,�=�����
-".�$�,�24�*��/(=?��<�� �����Splunk�B��:>C��8@�������!�%��mac_id=000:001:004:30:17:001��9���+�)�$�,7D�3���
0
1
2
3
© 2019 SPLUNK INC.
mac_id���� �*
index=bigdataTERM(mac_id=000:001:004:30:17:001)
station_id���� �
index=bigdataTERM(station_id=X209019)
area_id���� �
index=bigdataTERM(area_id=88)
��������TERM �������������������
© 2019 SPLUNK INC.
������������!-%��)$:+�'�(� �&��19
mac_id station_id area_id
������� 2.27 (12.27) - (2.27) 6.17 (36.27)
���� 1,4643 03 134,0293
������ 1,4643 03 134,0293������ ������������� 2.27 2.27 8.27
TERM�5� mac_id��*#"��-%��)$:+�,�04�station_id�����26 0�
mac_id�� �*#"��-%��)$:+��8��
( ).� TERM �5�����8/
© 2019 SPLUNK INC.
station_id������� 0����������
2019/07/29 23:00:00 mac_id=000:001:035:90:82:211 station_id = X209019 area_id=1 ,2.1ghz,15,R,359082,211,0.066,1,0.136,0.083,0,0.162,0.098,0,14.938,14.588,,72.938,5,72.938,5,0.067,0,0.5,,,13.591,,
station_id = X209019station_idX209019
�����
© 2019 SPLUNK INC.
area_id������������������������������ ��
| dbinspect index=bigdata*| stats count as num_buckets sum(sizeOnDiskMB) as sizeOnDiskMB by index| eval sizeOnDiskGB = round(sizeOnDiskMB/1024, 2)| fields index num_buckets sizeOnDiskGB
�� �� ����
bigdata 146 78.53GB
bigdata_idx 333 232.02GB
© 2019 SPLUNK INC.
�����������
������� ��index=bigdata
���������� ��index=bigdata_idx
�����������
������� ��index=bigdata| streamstats count
���������� ��index=bigdata_idx| streamstats count
������������
������� ��index=bigdata| stats count
���������� ��index=bigdata_idx| stats count
������������� ����� vs. ������
© 2019 SPLUNK INC.
���������!1 Indexer���#���������
� ������ �������� ���������
��������"� 305K���� 24K���� 828K����
��� ��������"� 279K���� 24K���� 661K����
"����������� ������$�����!
© 2019 SPLUNK INC.
�����������
������� ��index=bigdata| eval dl_gbytes = (dl_mbyte/1024)
���������� ��index=bigdata_idx| eval dl_gbytes = (dl_mbyte/1024)
�����������
������� ��index=bigdata| streamstatsavg(dl_mbyte)
���������� ��index=bigdata_idx| streamstatsavg(dl_mbyte)
������������
������� ��index=bigdata| stats avg(dl_mbyte)
���������� ��index=bigdata_idx| stats avg(dl_mbyte)
����� ����������� ����� vs. ������
© 2019 SPLUNK INC.
������������� 1 Indexer����"� ������
�� ����� �� ����� ��������
��������!� 134K��� 17K��� 378K���
�����������!� 153K��� 17K��� 560K���
�����!���������� ������#���� �
© 2019 SPLUNK INC.
q 0%+/2")�% 3• .�4&��F�.��4,
• IM�%!15�����5*("%=.��4,Q7�H@��
• IM�%!15 �:9�� TERM�>O������5*("%=.��4,Q7�H@��
• IM�%!15�D;� TERM�>O����:9��5*("%=.��4,Q7�GK
• JC�F�.��4,• �*�-3)��<�JC�F�.��4,��5*("%=.��4,Q7������$�'L?�EN���
• �5*("%=.��4,Q7�8���5*("%.��4�6����GK�#�%BA��$�'=.��4,Q7��P��3%"���
.��4,Q7$�'= vs. �5*("%=
���
© 2019 SPLUNK INC.
���� �������������������
© 2019 SPLUNK INC.
���������������������������%�!�$�"
������������%�
���������%�+
�#� ����
© 2019 SPLUNK INC.
���������������������������%�!�$�"
������������%����������%�
+�#� ����
© 2019 SPLUNK INC.
��������
) 2 1p c I s c I c xpo l D I x po l c c
l l s x W ) - lhk n Ip ) - Ha ( - x po l
n S n p c n c x po l x
po l p W x p nc x po l u i r i r c I p
x po l W i r i ru x Sa F s c D
Sd n x po llhk n I Ha x po l I p
Ii r 422 i rc erto x c
© 2019 SPLUNK INC.
����������
r u e aI r u e a
u e a e H S h 5
uacuac x I e
d ua nu uh I I uh a r nuuh S
d d l SH S I d e
Il u r H ku S Ia d H
c H SdI pu
uh uh
© 2019 SPLUNK INC.
������������
r eTX pr s x 6 DS d a h u d a
s asd a ce
eps h ls I 6d a d e seTX pr s Dd a d h u
se H h
n x
d Xt d a Dks
© 2019 SPLUNK INC.
�����x
t m tkr C )2 Msx t e S m t
s S 3v e
nr 1 I S e Co e l Ce S
1 d C S Me S7 4 d C S Me S
m tIcS em t CLm u
p x
© 2019 SPLUNK INC.
loadjob
1a H 1
a )
c a
b h a de211 2 2jo S b lr l 2
© 2019 SPLUNK INC.
���������
ar ID e f mI n Ho
n m m 1 1 1 /3 1d_ i bI n SH x
t 1 1 P 1 /3 1 6 / I . /3 1 /3 6 1 1s
o ma 1 1 I S cU lU aC ID) C hI S pID
d_ i bI xID
© 2019 SPLUNK INC.
redistribute
dilk ) . R maK 74 3 x.1 ) . Rnu ) . K p Hma
x m mdilk ) . R R I
s u 7 Rh 4 2 71 .1 Rdilk ) . t 2 2 maRx
y m _Sb S.1 e kc R
r ma r
© 2019 SPLUNK INC.
����������
index=bigdata| stats avg(dl_mbyte) by area_id
� ����������
index=bigdata_idx| stats avg(dl_mbyte) by area_id
DMA
| tstatsavg(bigdata_traffic.dl_mbyte) from datamodel=DM_bigdata_traffic_dl_mbyte by bigdata_traffic.area_id
��������� ���������������� vs. � ���� vs. DMA
© 2019 SPLUNK INC.
����������������������)����� vs. ������ vs. DMA
�������� ��� ������ DMA
����� 159.4& 112.8& 64.7&
�������%!��������#"���� (����������$'
© 2019 SPLUNK INC.
q '�"&-� �� 4• �1!���6%��/#O2�� ���6%��/#O2�5M9�&�,1�@�3���FB8CK��• 67�G<���43�5M9
− �).��1!���− 0(�"���-0��,1
• 67G<���JAI:�5M9− !��+!/���-0��,1
• .�/��*>�H=�����5M9− loadjob
• .����ELD;��• redistribute• !��$�&-�1N?9
5M9�&�,1���
© 2019 SPLUNK INC.
������Splunk������
© 2019 SPLUNK INC.
statschart
xyseriesuntable
transponse
makemvmvcombinemvexpand
eval MV ��
rename A_* as B_*foreach
rexstreamstatseventstats
������ ������������ �����!�����& %���
)$�#$� �"' ������� �����(
© 2019 SPLUNK INC.
����
(index=A A.val=*)| join � �����
[| search (index=B B.val=*) ]| stats avg(A.val) as avg_A_val,avg(B.val) as avg_B_valby � �����
����
(index=A A.val=*) OR(index=B B.val=*)| stats avg(A.val) as avg_A_val,avg(B.val) as avg_B_valby � �����
| search avg_A.val=* avg_B.val=*
���������������� vs. ���
���� � ��
����� 11.2� 5.4�
�� 49,172� 120,045�
© 2019 SPLUNK INC.
Splunk ������ �������
Treemap
Sankey Diagram
Punchcard Calendar Heat Map
Parallel Coordinates
Bullet GraphLocation Tracker
Horseshoe Meter
Machine Learning Charts
Timeline
Horizon Chart
Multiple use cases across IT, security, IoT, and business analytics
© 2019 SPLUNK INC.
Box Plot
3D scatter plot
���� �������������
Wordcloud
Donut Chart
Heat Map
© 2019 SPLUNK INC.
�� ���������
Maps+
Custom Cluster Map Missile Map
© 2019 SPLUNK INC.
q Splunk�,&�6%;*39#�bXZ0�-!L�O�?`�����]W�)�.!=�_N��S\
q C��0�-�� ��-#7,-;4�i<AG�E��,&�6%;*39#��"�"L�O�
q �"���!��B��f���0�-49/13$�7�Splunk�
q M�U����c��"�49/15�7���]W�dR�0�-!>g �������
q Splunk�0�-YJ�_N�!JQ��@^I%4+8;�)�.(6;2!HaD��eP �������hF!120%VK
q SplunkT[4:'97https://www.splunk.com/ja_jp/training.html
,&�6%;*39#���
© 2019 SPLUNK INC.
4 Days of Innovation 350 Education Sessions 20 Hours of Networking
“Hands down the most beneficial and attendee focused conference I have attended!”
– Michael Mills, Senior Consultant, Booz Allen Hamilton
�� ��������conf.splunk.com
.conf19October 21-24, 2019
Splunk UniversityOctober19-21, 2019
Las Vegas, NVThe Venetian Sands Expo
October 21-24
© 2019 SPLUNK INC.© 2019 SPLUNK INC.
Thank You.
Schema on The Fly is Always The Best Friend for Your Machine Data� �������������������������� J