Efficient Data Mining with Log Parser 2.2 Chris Adams Program Manager IIS Product Unit Microsoft...
21
Efficient Data Efficient Data Mining with Log Mining with Log Parser 2.2 Parser 2.2 Chris Adams Chris Adams Program Manager Program Manager IIS Product Unit IIS Product Unit Microsoft Corporation Microsoft Corporation
-
Upload
edith-porter -
Category
Documents
-
view
218 -
download
0
Transcript of Efficient Data Mining with Log Parser 2.2 Chris Adams Program Manager IIS Product Unit Microsoft...
Efficient Data Mining with Efficient Data Mining with Log Parser 2.2Log Parser 2.2
Chris AdamsChris AdamsProgram ManagerProgram ManagerIIS Product UnitIIS Product UnitMicrosoft CorporationMicrosoft Corporation
AgendaAgenda
Log Parser BasicsLog Parser Basics
How to Build an IIS Specific QueryHow to Build an IIS Specific Query
IIS Query ExamplesIIS Query Examples
Web Server Troubleshooting ExamplesWeb Server Troubleshooting Examples
Log Parser COM ArchitectureLog Parser COM Architecture
Demos, Demos, DemosDemos, Demos, Demos
Log Parser BasicsLog Parser BasicsGetting started with Log ParserGetting started with Log Parser
Powerful tool for queries against text based dataPowerful tool for queries against text based dataEx: Log files, XML files, and CSV filesEx: Log files, XML files, and CSV files
Log Parser requires:Log Parser requires:1. Input Format 1. Input Format
2. Query2. Query
3. Output Format3. Output Format
Query Language:Query Language:supports most of the true SQL language syntaxsupports most of the true SQL language syntax
extends the language with additional functionsextends the language with additional functions
Log Parser BasicsLog Parser BasicsParsing ModelParsing Model
Log Parser BasicsLog Parser BasicsSyntaxSyntax
Note: Note: Green Green = Required= Required
LogParser LogParser [[ -i:<input_format> -i:<input_format> ]] [[ <input_format_options> <input_format_options> ]]
[[ -o:<output_format> -o:<output_format> ]] [[ <output_format_options> <output_format_options> ]]
<SQL query><SQL query> || file:<query_filename> file:<query_filename>[[?param1=value1+?param1=value1+...]...] [<global switches>] [[<global switches>] [ -queryInfo -queryInfo ]]
Simple example: Logparser “Select * from <1>”Selects all lines in log file for IIS site #1
The Power of Log ParserThe Power of Log Parser
Chris AdamsChris AdamsProgram ManagerProgram ManagerIIS Product UnitIIS Product Unit
How to Build an IIS specific queryHow to Build an IIS specific queryStep 1: Choose input sourceStep 1: Choose input source
IIS related input formatsIIS related input formatsIISIIS
IISODBCIISODBC
IISW3C (Default for IIS)IISW3C (Default for IIS)
BINBIN
NCSANCSA
W3CW3C
HTTPERR (HTTP.SYS error log)HTTPERR (HTTP.SYS error log)
URLSCANURLSCAN
How to Build an IIS specific queryHow to Build an IIS specific query Step 2: Choose fields of interestStep 2: Choose fields of interest
Use “Logparser –h” to display fields for an input Use “Logparser –h” to display fields for an input formatformat
Example: Logparser –h –i:IISW3CExample: Logparser –h –i:IISW3C
How to Build an IIS specific queryHow to Build an IIS specific query Step 2: Choose fields of interestStep 2: Choose fields of interest
Or you can just use Log Parser HelpOr you can just use Log Parser Help
How to Build an IIS specific queryHow to Build an IIS specific query Step 3: Build SQL queryStep 3: Build SQL query
Minimal query must have Minimal query must have SelectSelect and and FromFrom
FromFrom <IIS Site ID> locates log files for an IIS site <IIS Site ID> locates log files for an IIS site
Example: Logparser “Select date from <1>”Example: Logparser “Select date from <1>”
Basic Query SyntaxBasic Query SyntaxSelect, Using, Into, From, Where, Group By, Having, Select, Using, Into, From, Where, Group By, Having, Order ByOrder By
Reduce output data using…Reduce output data using…Expressions: ALL, ANY, BETWEEN, IN, LIKEExpressions: ALL, ANY, BETWEEN, IN, LIKE
Aggregate functions: AVG, COUNT, GROUPING, Aggregate functions: AVG, COUNT, GROUPING, MAX, MIN, PROPCOUNT, PROPSUM, SUMMAX, MIN, PROPCOUNT, PROPSUM, SUM
Functions: ADD, BIT_OR, DIV, MUL, STRLEN, Functions: ADD, BIT_OR, DIV, MUL, STRLEN, STRCAT, TRIM, … (logparser –h FUNCTIONS)STRCAT, TRIM, … (logparser –h FUNCTIONS)
How to Build an IIS specific queryHow to Build an IIS specific query Step 4: Choose Output formatStep 4: Choose Output format
Default format = NAT Default format = NAT format (Tabulated format (Tabulated Columns) Columns)
DATAGRID exampleDATAGRID example
How to Build an IIS specific queryHow to Build an IIS specific query Step 4: Choose Output formatStep 4: Choose Output format
CHART examplesCHART examples
Note: Requires the Microsoft Office Web Components, Note: Requires the Microsoft Office Web Components, which are generally installed with Microsoft Office which are generally installed with Microsoft Office
What will this command do?What will this command do?
logparser "SELECT * INTO LogsTable FROM ex*.log logparser "SELECT * INTO LogsTable FROM ex*.log WHERE TO_LOWERCASE (EXTRACT_EXTENSION(cs-WHERE TO_LOWERCASE (EXTRACT_EXTENSION(cs-uri-stem)) NOT IN ('gif';'jpg';'png') AND sc-status <> 404" –uri-stem)) NOT IN ('gif';'jpg';'png') AND sc-status <> 404" –i:IISW3C –o:SQL –database:LogsDatabase i:IISW3C –o:SQL –database:LogsDatabase
How to Build an IIS specific queryHow to Build an IIS specific query Test Your Log Parser Skill – IIS Query ExampleTest Your Log Parser Skill – IIS Query Example
““SELECT * INTO LogsTable” = Put all info into LogsTable output formatSELECT * INTO LogsTable” = Put all info into LogsTable output format
““FROM ex*.log”FROM ex*.log” = From log files starting in “ex”= From log files starting in “ex”
““TO_LOWERCASETO_LOWERCASE ” = Disregard case” = Disregard case
““(EXTRACT_EXTENSION(cs-uri-stem)) NOT IN ('gif';'jpg';'png')(EXTRACT_EXTENSION(cs-uri-stem)) NOT IN ('gif';'jpg';'png') ” = All ” = All extensions except *.gif, *.jpg, *.pngextensions except *.gif, *.jpg, *.png
““sc-status <> 404” = Requests did not generate 404ssc-status <> 404” = Requests did not generate 404s
HINTSHINTS
Log Parser IIS QueriesLog Parser IIS Queries
Chris AdamsChris AdamsProgram ManagerProgram ManagerIIS Product UnitIIS Product Unit
Using Log Parser to Using Log Parser to Troubleshooting IISTroubleshooting IIS
Chris AdamsChris AdamsProgram ManagerProgram ManagerIIS Product UnitIIS Product Unit
Log Parser COM ArchitectureLog Parser COM Architecture
Requires Log Parser COM objects exported by the Requires Log Parser COM objects exported by the LogParser.dll libraryLogParser.dll library
Must register LogParser.dll on system Must register LogParser.dll on system
Scripting steps (in a nutshell)Scripting steps (in a nutshell)Instantiate the MSUtil.LogQuery object; Instantiate the MSUtil.LogQuery object;
Instantiate the input format object Instantiate the input format object
Instantiate the output format object Instantiate the output format object
Call the ExecuteBatch method of the MSUtil.LogQuery Call the ExecuteBatch method of the MSUtil.LogQuery object, specifying the query text, the input format object, object, specifying the query text, the input format object, and the output format object.and the output format object.
Log Parser COM ArchitectureLog Parser COM Architecture DIM objLogQuery : SET objLogQuery = WScript.CreateObject("MSUtil.LogQuery")
' Get the IIS Input and W3C output formatsDIM w3cInputFormat : SET w3cInputFormat =
WScript.CreateObject("MSUtil.LogQuery.IISW3CInputFormat")DIM w3cOutputFormat : SET w3cOutputFormat =
WScript.CreateObject("MSUtil.LogQuery.W3COutputFormat")w3cOutputFormat.filemode = 0
' Create a SQL query query = "SELECT TOP 20 cs-uri-stem, COUNT(*) as Total " & _ "INTO results.log FROM C:\WINDOWS\system32\Logfiles\W3SVC1\ex*.log " & _ "GROUP BY cs-uri-stem ORDER BY Total DESC "
objLogQuery.ExecuteBatch query, w3cInputFormat, w3cOutputFormat
Using Log Parser COMUsing Log Parser COM
Chris AdamsChris AdamsProgram ManagerProgram ManagerIIS Product UnitIIS Product Unit
Give me more…more…Give me more…more…more!more!
Chris AdamsChris AdamsProgram ManagerProgram ManagerIIS Product UnitIIS Product Unit
ResourcesResources
Where to obtain Log Parser 2.2:Where to obtain Log Parser 2.2:http://www.microsoft.com/downloads/http://www.microsoft.com/downloads/details.aspx?FamilyID=890cd06b-abf8-4c25-details.aspx?FamilyID=890cd06b-abf8-4c25-91b2-f8d975cf8c07&displaylang=en91b2-f8d975cf8c07&displaylang=en
Log Parser 2.2 Overview:Log Parser 2.2 Overview:http://www.microsoft.com/technet/scriptcenter/tools/http://www.microsoft.com/technet/scriptcenter/tools/logparser/default.mspxlogparser/default.mspx
““The Unofficial Log Parser Support Site:”The Unofficial Log Parser Support Site:”http://www.logparser.com http://www.logparser.com
© 2005 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.
