“IIS Data Mining with Log Parser 2.X” Alexis Eller Program Manager Internet Information Services...
26
“ “ IIS Data Mining IIS Data Mining with Log Parser with Log Parser 2.X” 2.X” Alexis Eller Alexis Eller Program Manager Program Manager Internet Information Services Internet Information Services Microsoft Microsoft
-
Upload
tabitha-sparks -
Category
Documents
-
view
218 -
download
0
Transcript of “IIS Data Mining with Log Parser 2.X” Alexis Eller Program Manager Internet Information Services...
““IIS Data Mining IIS Data Mining with Log Parser 2.X” with Log Parser 2.X”
Alexis EllerAlexis Eller
Program ManagerProgram Manager
Internet Information ServicesInternet Information Services
MicrosoftMicrosoft
What we will coverWhat we will cover Log Parser BasicsLog Parser Basics
Input Formats, Output FormatsInput Formats, Output Formats FunctionsFunctions Output TemplatesOutput Templates How to get started…How to get started…
Building on Log ParserBuilding on Log Parser Scripting with LogParser.dllScripting with LogParser.dll C# Interop C# Interop
Advanced FeaturesAdvanced Features New ‘CHART’ output formatNew ‘CHART’ output format CheckPointCheckPoint
Helps to have…Helps to have…
Experience with:Experience with: Command Line toolsCommand Line tools SQL Query LanguageSQL Query Language
Familiarity with:Familiarity with: IIS Log Files, Event LogIIS Log Files, Event Log
… … also nice to have experience with:also nice to have experience with: VBScript, JScript, C#VBScript, JScript, C# SQL ServerSQL Server
Log Parser BasicsLog Parser BasicsLog Parser is…Log Parser is…
Freely downloadable, stand-alone tool Freely downloadable, stand-alone tool Version 2.2: www.microsoft.com/download, search Version 2.2: www.microsoft.com/download, search
for “Log Parser”for “Log Parser” Version 2.1: Version 2.1: www.microsoft.com/downloadwww.microsoft.com/download, search , search
for “IIS 6.0 Resource Kit Tools”for “IIS 6.0 Resource Kit Tools”
Developed by Gabriele Giuseppini, former IISDeveloped by Gabriele Giuseppini, former IIS Not officially supported, try Not officially supported, try www.logparser.comwww.logparser.com Provided in two forms:Provided in two forms:
Command line exeCommand line exe Scriptable COM objectScriptable COM object [no GUI available at this time][no GUI available at this time]
Log Parser BasicsLog Parser BasicsKey ConceptsKey Concepts
Log Parser requires…Log Parser requires… Input FormatInput Format
[+options][+options]
QueryQuery Output Format Output Format
[+options][+options]
[+options][+options]
Query Language:Query Language: supports most of the true SQL language syntaxsupports most of the true SQL language syntax extends the language with additional functionsextends the language with additional functions
First Walk-ThroughFirst Walk-Through
Log Parser Cmd-Line Help Simple Log Parser Cmd-Line Help Simple Simple QuerySimple Query
Using Input/Output ParametersUsing Input/Output Parameters
Log Parser BasicsLog Parser BasicsFunctionsFunctions To get a list of functionsTo get a list of functions
logparser –h FUNCTIONSlogparser –h FUNCTIONS Examples:Examples:
Conversion FunctionsConversion Functions TO_TIMESTAMPTO_TIMESTAMP TO_LOCALTIMETO_LOCALTIME TO_INTTO_INT
String HandlingString Handling STRLENSTRLEN SUBSTR SUBSTR STRCAT STRCAT
OtherOther REVERSEDNSREVERSEDNS QUANTIZEQUANTIZE EXTRACT_EXTENSION EXTRACT_EXTENSION CASE CASE
Log Parser BasicsLog Parser BasicsFunction ExampleFunction Example
Convert log file timestamp from UTC time to local time:Convert log file timestamp from UTC time to local time:
SELECT TO_DATE( TO_LOCALTIME( TO_TIMESTAMP(date, time))) AS date, TO_TIME( TO_LOCALTIME( TO_TIMESTAMP(date, time))) AS time,
c-ip, cs-username, s-ip,s-port,cs-method,cs-uri-stem,cs-uri-query,sc-status,sc-win32-status,sc-bytes,cs-bytes,time-taken,cs(User-Agent)
FROM C:\WINDOWS\system32\Logfiles\W3SVC1\ex040308.log
Functions Functions
Top 10 URL'sTop 10 URL's
Convert Log File Time from Convert Log File Time from UTC Time to Local TimeUTC Time to Local Time
Examine Time Taken Per VDirExamine Time Taken Per VDir
Log Parser BasicsLog Parser BasicsOutput TemplatesOutput Templates
logparser -h logparser -h -o:TPL-o:TPL
Consists of:Consists of: Header Header BodyBody FooterFooter
Best for Best for report-style report-style resultsresults
<HTML>
<HEAD><TITLE>Hits/Hour for Ledbury Home Page</TITLE></HEAD><BODY BGCOLOR="#EFEFFF">
<TABLE BORDER="1" CELLPADDING="2“ CELLSPACING="2"><TR> <TH COLSPAN="2" ALIGN="CENTER"> Hits/Hour for Ledbury Home Page </TH></TR><TR> <TH ALIGN="LEFT">Hour</TH> <TH ALIGN="LEFT"># Hits</TH></TR>
<TR><TD>2004-03-07 22:00:00</TD><TD>1</TD></TR><TR><TD>2004-03-09 09:00:00</TD><TD>2</TD></TR><TR><TD>2004-03-10 18:00:00</TD><TD>2</TD></TR>
</TABLE></BODY></HTML>
HE
AD
ER
HE
AD
ER
BO
DY
BO
DY
FO
OT
ER
FO
OT
ER
TEMPLATE OUTPUT
Templates Templates
Broken Links ReportBroken Links ReportEvent Log MessagesEvent Log Messages
Log Parser BasicsLog Parser BasicsImporting Data into SQL ServerImporting Data into SQL Server
Note: Field names in query output are not important -- the field position Note: Field names in query output are not important -- the field position determines what column the data is inserted into in the SQL tabledetermines what column the data is inserted into in the SQL table
SELECT 1,TO_TIMESTAMP(date, time) AS LogTimeStamp,s-ip,cs-method,cs-uri-stem,cs-uri-query,s-port,cs-username,c-ip,cs(User-Agent),sc-status,sc-substatus,sc-win32-status
INTO W3SVC2 FROM ex040309.log
allows for an auto-generated identity column in SQL
database table
Log Parser BasicsLog Parser BasicsHow do I get started?How do I get started?
Find the column names in your data Find the column names in your data source:source: LogParser –h –i:<IISW3C|IIS|EVT>LogParser –h –i:<IISW3C|IIS|EVT> or… “or… “SELECT * FROM <datasource>SELECT * FROM <datasource>”” For TSV or CSV files:For TSV or CSV files:
LogParser –h –i:CSV/TSV <filename>LogParser –h –i:CSV/TSV <filename>
Understand data provided in the columnsUnderstand data provided in the columns Apply any functions necessary to convert Apply any functions necessary to convert
that data into a useful formthat data into a useful form ––queryinfo Optionqueryinfo Option
Log Parser Basics Log Parser Basics -queryinfo-queryinfoC:\DemoScripts>logparser file:SQLDB_Insert.sql -queryinfoWARNING: Output format not specified - using NAT output format.
Query: SELECT 1, TO_TIMESTAMP(date,time) AS LogTimeStamp, [s-ip], [cs-method], [cs-uri-stem], [cs-uri-query], [s-port], [cs-username], [c-ip], [cs(User-Agent)], [sc-status], [sc-substatus], [sc-win32-status], [sc-bytes], [cs-bytes], [time-taken] INTO W3SVC2 FROM ex040309.log
Formats selected: Input Format : IISW3C (IIS W3C Extended Log Format) Output Format: NAT (Native Format)
Query fields: 1 (I) LogTimeStamp (T) s-ip (S) cs-method (S) cs-uri-stem (S) cs-uri-query (S) s-port (I) cs-username (S) c-ip (S) cs(User-Agent) (S) sc-status (I) sc-substatus (I) sc-win32-status (I) sc-bytes (I) cs-bytes (I) time-taken (I)
Building on Log ParserBuilding on Log ParserLog Parser COM ArchitectureLog Parser COM Architecture
MSUtil.LogQueryMSUtil.LogQuery Input and Output Format objectsInput and Output Format objects
MSUtil.LogQuery.IISW3CInputFormatMSUtil.LogQuery.IISW3CInputFormat MSUtil.LogQuery.EventLogInputFormatMSUtil.LogQuery.EventLogInputFormat MSUtil.LogQuery.SQLOutputFormatMSUtil.LogQuery.SQLOutputFormat MSUtil.LogQuery.CSVOutputFormatMSUtil.LogQuery.CSVOutputFormat
Two ways to run query:Two ways to run query: MSUtil.LogQuery.ExecuteMSUtil.LogQuery.Execute
Returns a Returns a LogRecordSetLogRecordSet object that allows the script to object that allows the script to iterate through the query resultsiterate through the query results
MSUtil.LogQuery.ExecuteBatchMSUtil.LogQuery.ExecuteBatch Runs a query with a pre-specified output target (e.g. SQL or Runs a query with a pre-specified output target (e.g. SQL or
CSV)CSV)
Building on Log ParserBuilding on Log ParserInput/Output Format ParametersInput/Output Format Parameters
Log Parser 2.2Log Parser 2.2 Parameters have same name for both Parameters have same name for both
command line tool and COM interfacecommand line tool and COM interface
Log Parser 2.1 and 2.0Log Parser 2.1 and 2.0 Parameters do not always have same name Parameters do not always have same name
for cmd-line version and COM version, for for cmd-line version and COM version, for example:example: logparser file:query.sql –o:NAT –logparser file:query.sql –o:NAT –rtprtp:-1:-1 NativeOutputFormat.NativeOutputFormat.rowsToPrintrowsToPrint
http://www.logparser.com/InstantKB/article.ahttp://www.logparser.com/InstantKB/article.aspx?id=10039spx?id=10039
Building on Log ParserBuilding on Log ParserVBScript VBScript
Create LogQuery object and input/output Create LogQuery object and input/output format objectsformat objects
SET objLogQuery = WScript.CreateObject("MSUtil.LogQuery")
' Get the IIS Input and W3C output formatsSET w3cInputFormat = WScript.CreateObject("MSUtil.LogQuery.IISW3CInputFormat")SET w3cOutputFormat = WScript.CreateObject("MSUtil.LogQuery.W3COutputFormat")w3cOutputFormat.filemode = 0
' Create a SQL query query = "SELECT TOP 20 cs-uri-stem, COUNT(*) as Total " & _ "INTO results.log FROM C:\WINDOWS\system32\Logfiles\W3SVC1\ex*.log " & _ "GROUP BY cs-uri-stem ORDER BY Total DESC "
objLogQuery.ExecuteBatch query, w3cInputFormat, w3cOutputFormat
For an example that uses For an example that uses ExecuteExecute and and LogRecordSetLogRecordSet, see , see \Samples\Scripts\ErrorCodes.js\Samples\Scripts\ErrorCodes.js in Log in Log Parser installation directory (also available on TechNet: Parser installation directory (also available on TechNet: http://www.microsoft.com/technet/community/scriptcenter/logs/logparser/scripts/logpar01.mspx). http://www.microsoft.com/technet/community/scriptcenter/logs/logparser/scripts/logpar01.mspx).
ex05111606.log
Script SamplesScript Samples
List Error Codes and ReasonsList Error Codes and Reasons
Hacker ScanHacker Scan
Building on Log ParserBuilding on Log ParserC# InteropC# Interop
Uses System.Reflection and System.ActivatorUses System.Reflection and System.Activator
Type comLogQueryType = Type.GetTypeFromProgID("MSUtil.LogQuery", true);object comLogQueryObject = Activator.CreateInstance(comLogQueryType);
// Get the IIS Input and W3C output formatsType inputFormatType = Type.GetTypeFromProgID("MSUtil.LogQuery.IISW3CInputFormat", true);object inputFormatObject = Activator.CreateInstance(inputFormatType);Type outputFormatType = Type.GetTypeFromProgID("MSUtil.LogQuery.W3COutputFormat", true);object outputFormatObject = Activator.CreateInstance(outputFormatType);
// Create a SQL query string query = "SELECT TOP 20 cs-uri-stem, COUNT(*) as Total ";query += "INTO results.log FROM C:\\WINDOWS\\system32\\Logfiles\\W3SVC1\\ex*.log ";query += "GROUP BY cs-uri-stem ORDER BY Total DESC ";
// Invoke the ExecuteBatch methodobject[] inputArgs = { query, inputFormatObject, outputFormatObject };comLogQueryType.InvokeMember("ExecuteBatch", BindingFlags.InvokeMethod, null, comLogQueryObject, inputArgs);
Advanced FeaturesAdvanced FeaturesChart Output FormatChart Output Format Uses Microsoft Office Web Uses Microsoft Office Web
Components ChartSpace Components ChartSpace Object ModelObject Model See this link for object model: See this link for object model:
http://msdn.microsoft.com/librhttp://msdn.microsoft.com/library/default.asp?url=/library/en-ary/default.asp?url=/library/en-us/owcvba10/html/octocMSOWus/owcvba10/html/octocMSOWCObjectModels.aspCObjectModels.asp
You will need: You will need: Licensed version of Microsoft Licensed version of Microsoft
Office Web Components, Office Web Components, available in Microsoft Office available in Microsoft Office XPXP©© or better or better
Use Excel to view different Use Excel to view different CHART types:CHART types: Insert Menu, Chart option –Insert Menu, Chart option –
shows the different chart typesshows the different chart types
ChartsCharts
Status Code Breakdown – Pie ChartStatus Code Breakdown – Pie ChartHits Per Hour – Radial Chart Hits Per Hour – Radial Chart
Advanced Features Advanced Features CheckPoint – CheckPoint – incremental parsingincremental parsing
Parse only what has not been parsed before:Parse only what has not been parsed before: ex*.logex*.log … from all log files in a directory… from all log files in a directory <1><1> … from all log files for site 1… from all log files for site 1 SystemSystem … from the System Event Log… from the System Event Log
Requires a CheckPoint file to store state:Requires a CheckPoint file to store state:
logparser "SELECT TimeGenerated, EventTypeName, Strings FROM System WHERE SourceName = 'W3SVC'" -icheckpoint event.lpc
CheckPointCheckPoint
Event Log Update E-mailEvent Log Update E-mail
Session SummarySession Summary
Flexible and powerfulFlexible and powerful As with all data mining, the application starts with a As with all data mining, the application starts with a
question… question…
Building Blocks for Auditing and MonitoringBuilding Blocks for Auditing and Monitoring Log Parser can parse almost any data source on your server… Log Parser can parse almost any data source on your server…
with the additional input and output formats in Log Parser 2.2, with the additional input and output formats in Log Parser 2.2, there’s really nothing more you need to build custom auditing there’s really nothing more you need to build custom auditing and monitoring for your systemand monitoring for your system
Any Text, Any Time Any Text, Any Time LogParser can read most log files, text documents, and LogParser can read most log files, text documents, and
STDOUT/text streams, and generate results in a HUGE number STDOUT/text streams, and generate results in a HUGE number of formats to make them useful and intelligibleof formats to make them useful and intelligible
Log Parser has as many applications Log Parser has as many applications as you can imagine… as you can imagine…
For More InformationFor More Information www.logparser.comwww.logparser.com has:has:
an active forum for questionsan active forum for questions KB articles specifically for Log ParserKB articles specifically for Log Parser
Articles about Log Parser:Articles about Log Parser: SecurityFocus: Forensic Log Parsing with Microsoft's LogParser SecurityFocus: Forensic Log Parsing with Microsoft's LogParser
http://www.securityfocus.com/infocus/1712http://www.securityfocus.com/infocus/1712 TechRepublic: Consolidating Events with Free Log Parser 2.0 Tool TechRepublic: Consolidating Events with Free Log Parser 2.0 Tool
http://techrepublic.com.com/5100-6329-5034923.html http://techrepublic.com.com/5100-6329-5034923.html Windows & .NET Magazine: Using SQL-Like Queries to Extract File-Format Information Windows & .NET Magazine: Using SQL-Like Queries to Extract File-Format Information
http://www.winnetmag.com/Article/ArticleID/39233/39233.html http://www.winnetmag.com/Article/ArticleID/39233/39233.html ComputerWorld ComputerWorld
http://www.computerworld.com/securitytopics/security/story/0,10801,84230,00.html?http://www.computerworld.com/securitytopics/security/story/0,10801,84230,00.html?from=imutopicheads from=imutopicheads
Presentations:Presentations: BlackHat Windows 2004: Forensic Secrets for Windows Servers- BlackHat Windows 2004: Forensic Secrets for Windows Servers-
http://www.blackhat.com/presentations/win-usa-04/bh-win-04-burnett.pdf http://www.blackhat.com/presentations/win-usa-04/bh-win-04-burnett.pdf Blog Mentions:Blog Mentions:
Amazing Log Parser - http://www.larkware.com/Articles/AmazingLogParser.html Amazing Log Parser - http://www.larkware.com/Articles/AmazingLogParser.html Using Log Parser to Read Log Files - Using Log Parser to Read Log Files -
http://geekswithblogs.net/drewby/articles/LogParser.aspx http://geekswithblogs.net/drewby/articles/LogParser.aspx Using LogParser from C# - http://www.furrygoat.com/archives/000858.html Using LogParser from C# - http://www.furrygoat.com/archives/000858.html michaelw.net - http://michaelw.net/Articles/LogParser.html michaelw.net - http://michaelw.net/Articles/LogParser.html
[email protected]@microsoft.com
