From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&!...
Transcript of From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&!...
![Page 1: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/1.jpg)
Copyright © 2014 Splunk Inc.
George Starcher Security Engineer, Peak Hos6ng
From Tool to Team Member: Controlling Systems with Splunk Alert Scripts
![Page 2: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/2.jpg)
About Me ! George Starcher, Informa6on Security Engineer ‒ CISSP, Splunk Cer6fied Knowledge Manager and Splunk
Cer6fied Administrator ! Splunk IRC Channel ! Looking to kick off a Nashville, TN – Splunk User Group • www.georgestarcher.com • www.peakhos6ng.com
2
![Page 3: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/3.jpg)
Agenda
! Splunk from Tool to a Team Member ! How it Works ! GeQng into the Code ‒ Alert Script to Intrusion Preven6on System Control ‒ Alert Script to X-‐ARF Abuse Repor6ng
3
“Using Alert Scripts to take ac6on on our behalf, we can transform Splunk from a tool to a team member.”
![Page 4: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/4.jpg)
Splunk from Tool to Team Member ! Manual Abuse Scanning Process ‒ Reviewed SSH, RDP, VNC etc daily ‒ Consumed 30-‐45 minutes per day ‒ Permanent blacklist entries
! Moved to automated process ‒ Scheduled Splunk Searches driven by any log source ‒ Greatly reduced 6me and sta6c blacklist maintenance ‒ Web Services (REST) calls to the IPS
4
![Page 5: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/5.jpg)
Splunk from Tool to Team Member
5
Outlook Web Access – Phishers
Started Feb 10, 2014
• Blocked for any access from Nigeria every 5 minutes
Expanded Mul6 Country Feb 15, 2014
• Blocked for combina6on from certain countries & a lookup table of hosted providers
Feb 17, 2014
• No6ced unexpected Exchange OWA from Nigeria
![Page 6: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/6.jpg)
Splunk from Tool to Team Member
6
Outlook Web Access – Phishers
Single User by src_ip_country:
Hosted Lookup users by src_ip:
![Page 7: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/7.jpg)
How it Works
![Page 8: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/8.jpg)
How it Works
8
Intrusion Preven6on Appliance
![Page 9: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/9.jpg)
How it Works
9
![Page 10: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/10.jpg)
How it Works
10
h8p://blogs.splunk.com/2011/03/15/storing-‐encrypted-‐credenCals/
h8p://www.georgestarcher.com/splunk-‐alert-‐scripts-‐automaCng-‐control/
![Page 11: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/11.jpg)
How it Works
11
! Setup a service account to own the Alert Searches: svc-‐alert ! Create a role just for the alert account ! That role must have ‘admin_all_objects’ ! The role must have access to all indexes that might have the data for the scheduled search alert
Alert Service Account
![Page 12: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/12.jpg)
How it Works
12
![Page 13: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/13.jpg)
Alert Script in Ac6on
13
! Avoids manual repor6ng ! Ensures 6mely ac6on ! Consistent Repor6ng Format ! Accurate Evidence Data ! Works around the clock and doesn't need coffee
X-‐ARF Abuse Repor6ng
![Page 14: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/14.jpg)
Alert Script in Ac6on
14
X-‐ARF Abuse Repor6ng
![Page 15: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/15.jpg)
Alert Script in Ac6on
15
X-‐ARF Abuse Repor6ng
![Page 16: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/16.jpg)
Crawling into the Code
![Page 17: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/17.jpg)
@SplunkDev Team -‐ THANKS!!
17
@gblock -‐ Glenn Block
@damiendallimore -‐Damien Dallimore
David Noble -‐ Twiner App
![Page 18: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/18.jpg)
Where Can You Get The Code?
18
! Github Repository ‒ hnps://github.com/georgestarcher/Splunk-‐Alert ‒ General Intrusion Preven6on System Example Code ‒ Google Spreadsheet Upload Code ‒ X-‐ARF Abuse Repor6ng Code
! The Google Spreadsheet Example ‒ hnp://www.georgestarcher.com/splunk-‐alert-‐scripts-‐automa6ng-‐control/
![Page 19: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/19.jpg)
Arguments Sent to Alert Scripts
19
h8p://docs.splunk.com/DocumentaCon/Splunk/6.1.3/Alert/Configuringscriptedalerts
! SPLUNK_ARG_0 Script name ! SPLUNK_ARG_1 Number of events returned ! SPLUNK_ARG_2 Search terms ! SPLUNK_ARG_3 Fully qualified query string ! SPLUNK_ARG_4 Name of report ! SPLUNK_ARG_5 Trigger reason (for example, "The number of events was greater than 1")
! SPLUNK_ARG_6 Browser URL to view the report ! SPLUNK_ARG_8 File in which the results for this search are stored (contains raw results)
![Page 20: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/20.jpg)
The Code Modules – IPS
20
! creden6alsFromSplunk.py ‒ A Python class to fetch the saved service account
! targetlist.py ‒ The Python class for data to be handled
! ips.py ‒ The Python class for an IPS rest API interface
! alert_script.py ‒ The main Python alert script
![Page 21: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/21.jpg)
creden6alsFromSplunk.py
21
! A re-‐usable Python class to fetch stored user creden6als from Splunk ! Provide the app where creden6aled is stored: splunkapp ! Provide the purpose name used when saving the creden6als: realm ! Provide the username to be retrieved: username ! Call the getPassword method
![Page 22: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/22.jpg)
creden6alsFromSplunk.py
22
# Define the source in Splunk for the stored credenCal splunkapp = "myadmin" realm = 'ips' username = 'splunk'
# Get the stored credenCal from Splunk try: ipsCredenCal.getPassword(sessionKey) except ExcepCon, e: logError("Splunk CredenCal Error: %s" % str(e)) exitAlertScript(_SYS_EXIT_FAILED_SPLUNK_AUTH)
# Define the ips connecCon ipsCredenCal = credenCal(splunkapp,realm,username)
from alert_script.py
![Page 23: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/23.jpg)
targetlist.py
23
! A simple Python class for a single column list of source IPs ! Populated by the alert search returning only source IPs ! Takes argument of path to the search results to load the list
# Obtain the path to the alert events compressed file and load the search results to the list alertEventsFile = os.environ['SPLUNK_ARG_8']
try: alertTargetList = targetlist(alertEventsFile) except ExcepCon, e: logError("Target File Error: %s" % str(e)) exitAlertScript(_SYS_EXIT_FAILED_TARGET_FILE)
from alert_script.py
![Page 24: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/24.jpg)
ips.py
24
! An example Python class to interface with our Intrusion Detec6on System Rest API
! Setup and retrieve the creden6al from splunk: ipsCredenCal ! Provide the IPS quaran6ne policy name: policy_name ! Provide IP address of the IPS management Interface: ips_ip ! Ac6vate the IPS rest connec6on object ! Loop through the alertTargetList having the IPS quaran6ne each IP
Make your Own REST API wrapper class to control other systems
![Page 25: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/25.jpg)
ips.py
25
# AcCve the ips connecCon object try: ssh_ips = ips(ips_ip,ipsCredenCal.username,
ipsCredenCal.password,policy_name) except ExcepCon, e: logError("IPS Error: %s" % str(e)) exitAlertScript(_SYS_EXIT_FAILED_IPS)
from alert_script.py
![Page 26: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/26.jpg)
alert_script.py
26
! The main script called by Splunk for our alert search ! Imports all our classes ! Parses the sessionKey ! Connects to our IPS ! Pulls in the search result list of IP addresses ! Loops through the IP list and tells the IPS to quaran6ne them
![Page 27: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/27.jpg)
alert_script.py
27
The Hash Bang: #!/opt/splunk/bin/python
# QuaranCne each source ip in the alert results table
for address in alertTargetList.targetlist: try: ssh_ips.addQuaranCne(address) except ExcepCon, e: logError("IPS QuaranCne Error: %s" % str(e)) exitAlertScript(_SYS_EXIT_FAILED_IPS)
# Obtain the Splunk authenCcaCon session key … # Adjust the returned sessionKey text based on Splunk version …
![Page 28: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/28.jpg)
Extended Abuse Repor6ng -‐ X-‐ARF
28
! Much more complex code ! Search results driving results is a table of data not a simple IP list ! Pulls email seQngs from Splunk ! Builds the email body using the Python Mako template (mail merge to search results)
! Improved alert script ac6on logging sending into index=_internal ! Anaches Alert Event Search results from Splunk REST API Calls
h8p://www.x-‐arf.org/
BONUS
![Page 29: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/29.jpg)
The Code Modules -‐ X-‐ARF
29
! abuselist.py ‒ The data to be handled
! emailSplunkXARF.py ‒ A python class to fetch the saved service account
! xarf-‐abuse.tmpl ‒ Abuse report Email mako template
! alert_to_xarf.py ‒ The main alert script
![Page 30: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/30.jpg)
abuselist.py
30
! Method getEvidence holds the evidence search executed against the Splunk REST API
! This method also manipulates the earliest/latest 6mestamp coming from the search results automa6cally to go into the detail evidence search
emailSplunkXARF.py ! Method getMailSeQngs is Splunk REST API call to fetch the seQngs from your Splunk server
![Page 31: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/31.jpg)
alert_to_xarf.py
31
! All the X-‐ARF values are at the top of the script ! Method getSplunkVersion gets the running Splunk version from the REST API to help auto adjust the sessionKey
! Method getSplunkUser gets the username the Alert executed under from Splunk needed for the evidence search fetch
! Logging writes with proper 6mestamp GMT to $SPLUNK_HOME/var/log/splunk/…
You could use this to make your own highly customized alert email based on search results
![Page 32: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/32.jpg)
Thank You!
32
Other resources Splunk IRC ( EFNet #splunk ) Splunk Answers ( hnp://answers.splunk.com ) Splunk community wiki ( hnp://wiki.splunk.com ) hnp://www.georgestarcher.com/ hnp://blog.splunk.com/ hnp://www.meetup.com/Splunk/Nashville-‐TN/
Other “must-‐see” .conf 2014 presentaCons ! Avoid the SSLippery Slope of Default SSL -‐ Duane Waddle and George Starcher ! In Depth With Deployment Server -‐ Dave Shpritz, Aplura ! Using Lesser Known Commands in Splunk Search Processing Language (SPL) -‐ Kyle Smith, The Hershey Company
! Masters of IRC -‐ panel talk on the Splunk Community Stage
![Page 33: From&Tool&to&Team& Member:& …...Splunk&from&Tool&to&Team&Member&! Manual&Abuse&Scanning&Process&‒ Reviewed&SSH,&RDP,&VNC&etc&daily& ‒ Consumed&30S45&minutes&per&day ...](https://reader035.fdocuments.in/reader035/viewer/2022070801/5f02abd67e708231d4056d84/html5/thumbnails/33.jpg)
THANK YOU