Log Parser and Microsoft Log Parser and Microsoft Exchange Server, the Perfect Exchange Server, the Perfect

Blend!Blend!

Ilse Van CriekingeExchange Trainer & Consultant

(Azlan Training)

ilse@vancriekinge.comwww.Pro-Exchange.be

ObjectivesObjectivesWho needs reports?What kind of data is available?How to gain access to the data?How to present the acquired data?

AgendaAgenda

Introduction

Process flow

Sources of data

Pulling it all together: Joins

Report creation

Automation

IntroductionIntroductionReports are critical to a healthy Exchange org

understand, monitor, and track “who-what-when-where-how” usage data

mail usage by usermessage traffic patternsdelivery timeshistorical usage trendsmessage content reporting...

Many reporting packages are available at a price

OmniAnalyser, StealthAUDIT for Exchange, eIQ Mailanalyzer, Quest MessageStats, Admin Report Kit for Exchange, IMFStats, bv-Control for Exchange, PROMODAG Reports for Exchange, MailMeter Insight, Mail Access Monitor for MS Exchange Server,MailDetective, Sirana AppAnayzer for Exchange, e-nspect real time reporting, Quest Reporter, DYS CONTROL!, Exchange Monitor, bt-LogAnalyzer, ...

But…easy to develop basic reports yourself!

RequirementsRequirementsMicrosoft Exchange ServerActive DirectoryData extraction tools

Microsoft Log Parser 2.2A dash of scripting

To create & publish reportsTo create & publish reports

Microsoft SQL Server 2000 (or better) to store the data

Visual Studio .NET 2003 (or better) to create reports

SQL Reporting Services to publish reports

Why Reporting Services?Why Reporting Services?

Powerful web based reporting toolEasy to create rich, interactive, graphical reportsEnd-users can subscribe to receive reports via email, file share, etc…End-users can export reports to various formats (XLS, XML, CSV, HTML, TIFF, PDF, etc…)Easy to developRendering and processing can be seperated

AgendaAgenda

Introduction

Process flow

Sources of data

Pulling it all together: Joins

Report creation

Automation

Process FlowProcess Flow

1. Extract data from source2. Load data into SQL Server3. Create report in Visual Studio4. Publish report

Process FlowProcess FlowExchange

Message Tracking LogsActive Directory

User & Mailbox InfoExchange

Mailbox InfoDataDataSourcesSources

DataDataAccessAccessMethodMethod

Log Parser CSVDE WMI

Microsoft SQL Server DatabaseDataDataStorageStorage

DataDataOutputOutput Microsoft SQL Server Reporting Services

AgendaAgenda

Introduction

Process flow

Sources of data

Pulling it all together: Joins

Report creation

Automation

Sources of DataSources of Data

Exchange message tracking logs

Active Directory

HomeMDB, quota settings, ...

User Information

WMI providers for Exchange

Source 1:Source 1:Exchange Message Tracking LogsExchange Message Tracking Logs

Available in Exchange 5.5/2000/2003Has to be enabled

Exchange 5.5: Information Store/MTA/Internet Mail ServiceExchange 2000/2003: Server setting

Options:Remove log files [older than (days)]Exchange 2000/2003: Enable subject logging and displayExchange 2003: Location to store log files

Message Tracking Log Message Tracking Log FormatFormat

Note: Tracking logs in Microsoft Exchange 2000 Server have a significantly different format then Microsoft Exchange Server 5.5 tracking logs.

Generally follows the W3C format for log files

First few lines contain directives, tab delimited

One log generated/server/day

Logs roll at midnight GMT

All times in the log are GMT

Message Tracking Log FieldsMessage Tracking Log FieldsField number Field name Field number Field name

1 Date 11 Priority

2 Time 12 Recipient-Report-Status

3 Client-IP 13 Total-bytes

4 Client-hostname 14 Number-recipients

5 Partner-name 15 Time-taken

6 Server-hostname 16 Encryption

7 Server-IP 17 Service-version

8 Recipient-address 18 Linked-MSGID

9 Event-ID 19 Message-subject

10 MSGID 20 Sender-address

Source: http://support.microsoft.com/?kbid=246965

Message tracking event IDs in Exchange Server 20031027: Message submission by store1028: Message deliveryOverview: http://support.microsoft.com/?kbid=821905

Message Tracking Log Message Tracking Log SampleSample

# Message Tracking Log File# Exchange System Attendant Version 6.5.7638.1

# Date Time client-ip Client-hostname Partner-Name Server-hostname server-IP Recipient-AddressEvent-ID MSGID Priority Recipient-Report-Status total-bytes Number-RecipientsOrigination-Time Encryption service-Version Linked-MSGID Message-Subject Sender-Address

2006-2-7 10:28:41 GMT - - - NTS00 - ivcrieki@yahoo.com1027 3ADF255035AF154496E38B1C234B9C5D442F@nts00.matisse.edu 0 0 5111 2006-2-7 10:28:41 GMT 0 - c=US;a= ;p=First Organizati;l=NTS00-060207102841Z-5 Will Public Folders disappear? EX:/O=FIRST ORGANIZATION/OU=FIRST ADMINISTRATIVE GROUP/CN=RECIPIENTS/CN=ILSE -

2006-2-7 10:28:41 GMT - - - NTS00 - ivcrieki@yahoo.com1019 3ADF255035AF154496E38B1C234B9C5D442F@nts00.matisse.edu 0 0 5111 2006-2-7 10:28:41 GMT 0 - - Will Public Folders

disappear? - -

2006-2-7 10:28:41 GMT - - - NTS00 - ivcrieki@yahoo.com1025 3ADF255035AF154496E38B1C234B9C5D442F@nts00.matisse.edu 0 0 5111 2006-2-7 10:28:41 GMT 0 - - Will Public Folders

disappear? - -

2006-2-7 10:28:41 GMT - - - NTS00 - ivcrieki@yahoo.com1024 3ADF255035AF154496E38B1C234B9C5D442F@nts00.matisse.edu 0 0 5111 2006-2-7 10:28:41 GMT 0 - - Will Public Folders

disappear? - -

2006-2-7 10:28:41 GMT - - - NTS00 - ivcrieki@yahoo.com1033 3ADF255035AF154496E38B1C234B9C5D442F@nts00.matisse.edu 0 0 5111 2006-2-7 10:28:41 GMT 0 - - Will Public Folders

disappear? Ilse.VanCriekinge@matisse.edu -

2006-2-7 10:28:41 GMT - - - NTS00 - ivcrieki@yahoo.com1034 3ADF255035AF154496E38B1C234B9C5D442F@nts00.matisse.edu 0 0 5111 2006-2-7 10:28:41 GMT 0 - - Will Public Folders

disappear? Ilse.VanCriekinge@matisse.edu -

Column DataDate 2006-2-7

Time 10:28:41 GMT

Client-IP -

Client-Hostname -

Partner-Name -

Server-Hostname NTS00

Server-IP -

Recipient-Address ivcrieki@yahoo.com

Event-ID 1027

MSGID 3ADF255035AF154496E38B1C234B9C5D442F@nts00.matisse.edu

Priority 0

Recipient-Report-Status 0

total-bytes 511

Number-Recipients 1

Origination Time 2006-2-7 10:28:41 GMT

Encryption 0

Service-Version -

Linked-MSGID c=US;a= ;p=First Organizati;l=NTS00-060207102841Z-5

Message-Subject Will Public Folders disappear?

Sender-Address EX:/O=FIRST ORGANIZATION/OU=FIRST ADMINISTRATIVE GROUP/CN=RECIPIENTS/CN=ILSE

Log ParserLog ParserLog Parser 2.2 can be used to reformat the tracking logs into a format digestible by SQL ServerLog Parser is available for download from the Microsoft Download Center:http://www.microsoft.com/download

Introducing Log ParserIntroducing Log Parser““The world is your database with Log Parser”The world is your database with Log Parser”

Log Parser allows users to treat log files and other information as SQL tables, the rows of which can be queried, processed, and formatted in different ways Born around 2000, as a utility to test the logging mechanisms of IISLatest release = version 2.2“Designed and engineered with the vision of helping users achieve their data-processing goals in a simple, fast, and powerful way.”

Building QueriesBuilding Queries

Data(Input Formats)

Output Records(Output Formats)

Log Parser Query

Building Blocks Log ParserBuilding Blocks Log ParserInput Format

Log Parser Query (dialect of SQL)Output Format

ADS ETW IIS NETMON TSV

BIN EVT IISODBC REG URLSCAN

COM FS IISW3C TEXTLINE W3C

CSV HTTPERR NCSA TEXTWORD XML

CHART IIS SYSLOG W3C

CSV NAT TPL XML

DATAGRID SQL TSV  

Example: Retrieving some Example: Retrieving some fields from the Event Log fields from the Event Log

c:\LogParser –i:EVT –o:NAT “SELECT TimeGenerated, SourceName FROM System”

Or “SELECT TimeGenerated, SourceName INTOmytest.txt FROM System”

TimeGenerated SourceName------------------- -----------------------2005-11-10 12:26:07 Windows Update Agent2005-11-10 12:26:14 Windows Update Agent2005-11-10 15:00:23 Service Control Manager2005-11-10 15:00:23 Service Control Manager2005-11-10 15:00:44 Windows Update Agent2005-11-10 15:01:18 Windows Update Agent2005-11-10 15:01:30 NtServicePack2005-11-10 15:01:36 Windows Update Agent2005-11-10 15:01:50 Windows Update Agent2005-11-10 15:02:12 Windows Update AgentPress a key...

Back to Message Tracking Back to Message Tracking LogsLogs

Command:LogParser.exefile:f:\info\msgtracklog.sql?infile=f:\info\logs\20060207.log+outfile= f:\info\logs\20060207.bcp

-i:W3C -o:TSV

W3C input format parses log files in the W3C Extended Log File FormatTSV output format creates text file formatted according to the Tab-Seperated-Values convention

Log Parser Query SyntaxLog Parser Query Syntax

msgtracklog.sql:

SELECTTO_Timestamp(REPLACE_STR(STRCAT(STRCAT(date,' '), time),'

GMT',''),'yyyy-M-d h:m:s') as DateTime,[client-ip], [Client-hostname], [Partner-name], [Server-hostname], [server-

IP], [Recipient-Address],[Event-ID], [MSGID], [Priority], [Recipient-Report-Status], [total-bytes],

[Number-Recipients],TO_Timestamp(REPLACE_STR([Origination-time], ' GMT',''),'yyyy-M-d h:m:s')

as [Origination Time],Encryption, [service-Version], [Linked-MSGID], [Message-Subject], [Sender-

Address]INTO '%outfile%'FROM '%infile%'WHERE [Event-ID] IN (1027;1028)

Log Parser OutputLog Parser Output

DateTime client-ip Client-hostname Partner-Name Server-hostname server-IP Recipient-AddressEvent-IDMSGID Priority Recipient-Report-Status total-bytes Number-RecipientsOrigination TimeEncryption service-Version Linked-MSGID Message-SubjectSender-Address

2006-02-07 10:28:41 NTS00 ivcrieki@yahoo.com 10273ADF255035AF154496E38B1C234B9C5D442F@nts00.matisse.edu 0 0 511 12006-02-07 10:28:41 0 c=US;a= ;p=First Organizati;l=NTS00-060207102841Z-5Will Public Folders disappear? EX:/O=FIRST ORGANIZATION/OU=FIRST ADMINISTRATIVE GROUP/CN=RECIPIENTS/CN=ILSE -

Column DataDateTime 2006-02-07 10:28:41

Client-IP -

Client-Hostname -

Partner-Name -

Server-Hostname NTS00

Server-IP -

Recipient-Address ivcrieki@yahoo.com

Event-ID 1027

MSGID 3ADF255035AF154496E38B1C234B9C5D442F@nts00.matisse.edu

Priority 0

Recipient-Report-Status 0

total-bytes 511

Number-Recipients 1

Origination Time 2006-02-07 10:28:41

Encryption 0

Service-Version -

Linked-MSGID c=US;a= ;p=First Organizati;l=NTS00-060207102841Z-5

Message-Subject Will Public Folders disappear?

Sender-Address EX:/O=FIRST ORGANIZATION/OU=FIRST ADMINISTRATIVE GROUP/CN=RECIPIENTS/CN=ILSE

Populate SQL with formed DATAPopulate SQL with formed DATA

Create a table to hold the data

Import information into the database

Several tools available, like:

SQL 2000: SQL Query Analyzer, osql utility, bcp utility

SQL 2005: SQL Management Studio, sqlcmd, bcp or osql utility

Create Table to Hold DataCreate Table to Hold DataUSE Analyzing_ExchangeCREATE TABLE MsgTrackingLogs (

[DateTime] datetime NULL ,[Client-IP] varchar (255) NULL ,[Client-Hostname] nvarchar (255) NULL ,[Partner-name] nvarchar (255) NULL ,[Server-hostname] nvarchar (255) NULL ,[Server-IP] varchar (255) NULL ,[Recipient-Address] varchar (512) NULL ,[Event-ID] int NULL ,[MSGID] nvarchar (1024) NULL ,[Priority] int NULL ,[Recipient-Report-Status] int NULL ,[Total-bytes] bigint NULL ,[Number-Recipients] int NULL ,[Origination Time] datetime NULL ,[Encryption] int NULL ,[Service-version] varchar (255) NULL ,[Linked-MSGID] varchar (255) NULL ,[Message-Subject] nvarchar (255) NULL ,[Sender-Address] varchar (255) NULL ,

)

Import Data Into DatabaseImport Data Into Database

Bcp Analyzing_Exchange.dbo.msgtrackinglogs

in f:\info\logs\20060207.bcp

–c

–t”\t”

–T

–F 2

Source 2: Active DirectorySource 2: Active DirectoryEach mailbox is an object in ADSome relevant properties:

legacyExchangeDNhomeMDBmDBUseDefaults (use default quota)mDBStorageQuota (issue warning)mDBOverQuotaLimit (prohibit send)mDBOverHardQuotaLimit (prohibit send/receive)Can also include fields like city, department, etc..

Use CSVDE to export data to CSV fileUse account with Exchange view only admin rights

Introducing CSVDEIntroducing CSVDE

csvde.exe installed on Windows 200X Server by defaultCan be run from Windows 2000 Pro or XP ProfessionalCan be used to import and export data from Active Directory by using files that store data in the comma-separated value (CSV) file format standardAlso supports batch operations that are based on CSV

CSVDE SyntaxCSVDE SyntaxCSVDE

-f file to export to-s servername-d LDAP search root-r LDAP search filter (default objectClass=*)-l list of attributes to export-u Unicode format (important for DBCS)

CSVDE ExampleCSVDE ExampleExtract a specified list of all Person objects in the

Matisse domain:

CSVDE –f f:\info\ad\directory.csv

-s NTSMATISSE

-d "dc=matisse,dc=edu"

-r "(&(objectCategory=Person)(homeMDB=*))"

-l DN,legacyExchangeDN,mail,homeMDB,mDBUseDefaults,mDBOverQuotaLimit,mDBStorageQuota,mDBOverHardQuotaLimit,department

-u

Create Table to Hold DataCreate Table to Hold DataCREATE TABLE [Active_Directory_Info] (

[DN] [varchar] (1000) NULL ,[legacyExchangeDN] [varchar] (512) NULL,[mail] [varchar] (512) NULL ,[homeMDB] [varchar] (1000) NULL ,[mDBUseDefaults] [varchar] (10) NULL ,[mDBOverQuotaLimit] [int] NULL ,[mDBStorageQuota] [int] NULL ,[mDBOverHardQuotaLimit] [int] NULL ,[Department] [varchar] (256) NULL ,

) ON [PRIMARY]

Import Data Into DatabaseImport Data Into DatabaseLogParser

"SELECT DN,legacyExchangeDN,mail,homeMDB,mDBUseDefaults,mDBOverQuotaLimit,mDBStorageQuota,mDBOverHardQuotaLimit,department

into dbo.Active_Directory_Info

FROM f:\info\ad\directory.csv"

-i:csv

-o:SQL -server:servername -database:Analyzing_Exchange -driver:"SQL Server"

Source 3: WMI Source 3: WMI Windows Management InstrumentationManagement technology allowing scripts to monitor and control managed resources throughout the networkResources include hard drives, file systems, operating system settings, processes, services, shares, registry settings, networking components, event logs, users, and groupsBuilt into clients with Windows 2000 or above, and can be installed on any other 32-bit Windows clientWMI is easy to consume via script

Exchange_Mailbox WMI ClassExchange_Mailbox WMI ClassNew class for Exchange 2003

Returns properties of a mailbox

Interesting fields:MailboxDisplayName

LegacyDN (legacyExchangeDN)

ServerName (Exchange server name)

Size (size of mailbox in kb)

TotalItems (total # messages in the mailbox)

DeletedMessageSizeExtended (Size in bytes of deleted messages being retained per deleted items retention policy)

VBScript to Access WMI DataVBScript to Access WMI DatastrWinMgmts =

"winmgmts:{impersonationLevel=impersonate}!//ServerName/root/MicrosoftExchangeV2"

Set objWMIExchange = GetObject(strWinMgmts)Set listExchange_Mailboxes = objWMIExchange.InstancesOf("Exchange_Mailbox")For each objExchange_Mailbox in listExchange_Mailboxes Wscript.echo objExchange_Mailbox.MailboxDisplayName & vbTab _

& objExchange_Mailbox.LegacyDN & vbTab _& objExchange_Mailbox.ServerName & vbTab _& objExchange_Mailbox.Size & vbTab _& objExchange_Mailbox.TotalItems & vbTab _

& objExchange_Mailbox.DeletedMessageSizeExtended & vbTab _ & objExchange_Mailbox.LastLogonTime & vbTab _ & objExchange_Mailbox.LastLogOffTime & vbTab _

& objExchange_Mailbox.LastLoggedOnUserAccountNext

Execute as: cscript //nologo mailboxes.vbs > Mailboxes.txt

Create Table to Hold DataCreate Table to Hold Data

CREATE TABLE [MailboxSizeData] ([displayName] [varchar] (128) NULL ,[legacyExchangeDN] [varchar] (512) NULL ,[ServerName] [varchar] (50) NULL ,[Size] [int] NULL ,[TotalItems] [int] NULL ,[DeletedMessageSizeExtended] [int] NULL ,[LastLogonTime] [varchar] (50) NULL ,[LastLogoffTime] [varchar] (50) NULL ,[LastLoggedOnUserAccount] [varchar] (50) NULL

) ON [PRIMARY]

Import Data Into DatabaseImport Data Into DatabaseMicrosoft SQL Server Management Studio “BULK INSERT [MailboxSizeData] FROM ‘f:\

info\wmi\Mailboxes.txt'“

AgendaAgenda

Introduction

Process flow

Sources of data

Pulling it all together: Joins

Report creation

Automation

Pulling it all togetherPulling it all together

SQL joins let us relate data in one table with data in another tablePowerful feature for rich reportsUse common columns to relate data

Table JoinsTable Joins

MsgTrackingLogs, MailboxSizeData and Active_Directory_Info can all be joinedActive_Directory_Info and MailboxSizeData join on [legacyExchangeDN]Then join Active_Directory_Info to MsgTrackingLogs on [Recipient-Address]

JoinJoin

AgendaAgenda

Introduction

Process flow

Sources of data

Pulling it all together: Joins

Report creation

Automation

Report Creation – Getting Report Creation – Getting StartedStarted

Install Reporting Services Install Reporting Services Client ToolsCreate new “Business Intelligence” projectSelect template: “Report Server Project Wizard”

Six Easy StepsSix Easy Steps

1. Define data source

2. Design query

3. Choose type of report

4. Specify basic layout of report

5. Format the report

6. Deploy the report

DeployDeployCheck project propertiesCheck TargetServerUrRL =http://myserver.mydomain.com/ReportServer

Deploy!When complete, browse with IE:

http://servername/Reports

AgendaAgenda

Introduction

Process flow

Sources of data

Pulling it all together: Joins

Report creation

Automation

AutomationAutomation

Automate data gathering and import with:

SQL 2000: Data Transformation Services (DTS)

SQL 2005: SQL System Integration Services

(SSIS)

SummarySummary

Reports are vital to the health of your messaging infrastructure

Basic reports are fairly easy to develop

Three key data sources: AD, Tracking logs, WMI

This session gave you a very limited view of all the power you have when you use the available tools to create custom reports!

Thank you for your Thank you for your attention!attention!

Ilse Van CriekingeExchange Trainer & Consultant

(Azlan Training)

ilse@vancriekinge.comwww.Pro-Exchange.be

ResourcesResourcesLogParser 2.2

Microsoft Log Parser Toolkit, Gabriele Giuseppini&Mark Burnett, Syngresshttp://www.logparser.com/http://www.microsoft.com/downloads/details.aspx?FamilyID=890cd06b-abf8-4c25-91b2-f8d975cf8c07&displaylang=enhttp://www.microsoft.com/technet/scriptcenter/tools/logparser/default.mspx

Message tracking log field descriptionshttp://support.microsoft.com/default.aspx?scid=kb;en-us;246965

Message tracking log event id definitionshttp://support.microsoft.com/default.aspx?scid=kb;en-us;821905

SQL Server 2000/2005http://www.microsoft.com/sql

Visual Studio 2005http://msdn.microsoft.com/vstudio/

WMI Exchange_Mailbox classhttp://msdn.microsoft.com/library/default.asp?url=/library/en-us/e2k3/e2k3/_wmiref_cl_Exchange_Mailbox.asp

The Connected GenerationThe Connected Generation

7 & 8 March 20067 & 8 March 2006

ICC GentICC Gent