NSD ‘Crash Course’

Gregory Engels, KompurityILUG 2010

Some Rights Reserved

Copyright 2010 by Gregory Engels, Kompurity. Except where otherwise noted, this work is licensed under the Creative Commons Attribution Share Alike 3.0 License

About meGregory EngelsBorn in Moscow, live in Frankfurt, Germany since 1989Been around Lotus Notes since 1998Advanced Certified Domino Developer and Admin and Security R5, R6, R7, R8Co-Author "Migrating Microsoft Exchange 2000/2003 to IBM Lotus Notes 7"Founder Kompurity http://www.kompurity.deSpecialize in Hosting, Infrastructure management

Blog: http://inotes.de

Twitter: dichter

Agenda

• NSD

• ADC

• Upgrading NSD

• LND

• Notes Systems Diagnostic

• binary on W32 and iSeries

• shell script nsd.sh on Unix

What is NSD

Running the NSD

• Manually

• Automatically as part of Fault Recovery

• Enabled by default

• Enable it in the Server document

• Go enable it now.

Fault Recovery

Help is on the way

• nsd -help

Automated Data Collection (ADC)

Enable in the Config Doc

Enable in Desktop Policy

Fault Reports DB

Put this in the nsd.ini

• canceldisable=1

• (BOFH tip of the day)

Real Life

• ADC Enabled 9 Month ago

• User Population c.a. 6000 Users

• 1700 Users have created at least 1 NSD

• c.a. 13000 Client NSD

• 13 Users (0.05%) created 1700 NSD (14%)

How often does Notes Crashes?

0

10

20

30

40

50

60

70

80

90

100

1 5 9 14 21 30 52 75 163 1258

Number of Users that had N CrashesOccurency

Number of NSDs per User

0

375

750

1125

1500

Occurences

NSD’s per Day

0

18

36

54

72

90

22.08.2009 25.08.2009 28.08.2009 31.08.2009 03.09.2009 06.09.2009 09.09.2009 12.09.2009 15.09.2009 18.09.2009 21.09.2009 24.09.2009

Upgrading NSD

• See TN #4013182 for ftp links

• See TN #1233676 for list of SPR’s fixed

• Version of NSD is <NotesVersion><Fixpack>.<YNNN>

• NNN = Days since year begin

• example: NSD 8.0.10.8057 (Release 8.0.1) = Feb 27 2008 (57 Days since Jan 01)

Reading the NSD log

Finding the log

• Logs are stored in the IBM_TECHNICAL_SUPPORT subdirectory

• in the Lotus Notes Domino Fault Reports Database lndfr.nsf

• ND6 - nsd_all_<Platform>_<Host>_MM_DD@HH_MM.log • ND7 -

nsd_<Platform>_<ServerName>_YYYY_MM_DD@HH_MM_SS.log

Contents of the NSD

• Rather verbose

• Process Info (Call Stack)

• Memcheck (Domino Memory Objects)

• System Information

• Environment Information

• Process Information (Call Stacks) – Process Information is composed of the list of all running processes system wide, followed by a list of all Domino specific processes.

• the code path involved in a particular problem.

• Memcheck (Domino Memory Objects) – The Memcheck section dumps information about Domino-specific structures.

• memory pools (both private and shared)

• list of open resources such as open database, open view, open documents, connected users, and open files.

• System Information – This section provides information regarding version of OS,

• kernel configurations, patch information, disk information, network connections, and memory usage.

• Environment Information:

• user environment,

• notes.ini,

• file listing

Finding the fault

• Look for the words FATAL, PANIC, ETC### FATAL THREAD 67/135 [ nSERVER:0908: 2692] ### FP=0x0ae5ec70, PC=0x6018eb23, SP=0x0ae5e2f8 ### stkbase=0ae60000, total stksize=262144, used stksize=2424 ### EAX=0x010d088c, EBX=0x00000000, ECX=0x00ba0000, EDX=0x00ba0000 ### ESI=0x0ae5e904, EDI=0x00001d34, CS=0x0000001b, SS=0x00000023 ### DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010202 Exception code: c0000005 (ACCESS_VIOLATION) Process ID: Thread ID

Reading the Callstack############################################################

### FATAL THREAD 1/4 [ncompact: 3044: 2620]### FP=0x0012e644, PC=0x60193297, SP=0x0012dacc, stksize=2936### EAX=0x00e3088c, EBX=0x00000000, ECX=0x00900000, EDX=0x00900000### ESI=0x0012e1d8, EDI=0x00d495d8, CS=0x0000001b, SS=0x00000023### DS=0x00000023, ES=0x00000023, FS=0x0000003b, GS=0x00000000 Flags=0x00010202Exception code: c0000005 (ACCESS_VIOLATION)############################################################@[ 1] 0x60193297 nnotes._Panic@4+631 (60a30013)@[ 2] 0x600027a6 nnotes._LockHandle@12+294 (3963e26,12e674,12e680)@[ 3] 0x600025e7 nnotes._OSLockObject@4+23 (60c7df40)@[ 4] 0x606c09cc nnotes.CArchSetPolicy::GetPrivateArchiveSettings+3404 (1)@[ 5] 0x606bf1a8 nnotes.CArchSetPolicy::GetArchiveSettings+1032 (0)@[ 6] 0x606be9f9 nnotes.CArchSetPolicy::Init+553 (33a0a99)@[ 7] 0x606c7554 nnotes._PolicyCheckServerArchiveEnabled@4+52 (a9)@[ 8] 0x606e2fc1 nnotes._CheckServerArchiveEnabled+113 (10900a9)@[ 9] 0x00402111 nCompact._CompactFile@16+529 (0,a9,81000000,0)@[10] 0x00401b61 nCompact._AddInMain@12+2913 (3,3,398644)@[11] 0x004028cf nCompact._NotesMain@8+47 (3,400000)@[12] 0x004029d6 nCompact._notes_main+214 (0,0)@[13] 0x004028f6 nCompact._main+22 (3,3b2da0)@[14] 0x00402cc1 nCompact._mainCRTStartup+197 (0,0)@[15] 0x7c816ff7 KERNEL32.RegisterWaitForInputIdle+105 (402bfc,0)

Pid : Tid

Finding the Resources

• Search for “Open Databases”

• Search for “Open Documents”

DBH NOTEID HANDLE CLASS FLAGS IsProf #Pools #Items Size Database

531 7330 0x24ff 0x0001 0x0200 Yes 1 4 2984 d:\notedata\drmail\jsmith.nsf

Open By: CN=John Smith/O=ACME/C=US

Flags2 = 0x0404

Flags3 = 0x0000

OrigHDB = 531

First Item = [ 9471: 836]

Last Item = [ 9471: 1228]

Non-pool size : 0

Note Class Description

• 0x0001 Data Note (document)

• 0x0004 Form Note

• 0x0008 View note

• 0x0040 ACL Note

• 0x0200 Agent Note

• 0x0800 Replication Formula Note

Search for VTread info

• Search on KEYWORDs “Resource Usage” or “Resource Usage Summary”

• Search on KEYWORD “Process” until you located the necessary process ID

• Search on the PHYSICAL THREAD ID of the thread you are interested in (cosection after you have looked at call stacks)

Troubleshooting NSD

• Try NSD -info

• If this runs to completion, perhaps there is a problem with the pid.nbf file

• Try upgrading NSD

• Check if symbols are installed

• lotusnotes.sym

LND Tool

• See TN # 4019151 for download link

• ftp://ftp.software.ibm.com/software/lotus/tools/LND_2.8_Setup.zip

Auto KB Seatch

See the Process Resources

Works on Right-Click in Explorer

Questions

Thank You!

• Please fill out the evaluation (so I can get back next year)

• Contact: Gregory.Engels ※ kompurity.deBlog http://inotes.detwitter: dichter

Backup

NSD parameters

• nsd –stack (collects only call stacks, speeds execution)

• nsd –info (collects only system info)

• nsd –noinfo (collects all but info)

• nsd –memcheck (collects only memcheck info)

• nsd –nomemcheck (collects all but memcheck)

• nsd –perf (collects process memory usage)

• nsd –noperf (collects all but performance data)

• nsd –handles (collects OS level handle info)

• nsd –nohandles (collects all but handle info)

• nsd –kill (kills all notes process and associated memory)

• nsd –monitor (attaches and waits for exceptions)

• nsd –p (runs against a specific process – call stacks only)