Windows NT ® Internals David Solomon David Solomon Expert Seminars Microsoft Corporation.

Windows NTWindows NT®® Internals Internals

David SolomonDavid SolomonDavid Solomon Expert SeminarsDavid Solomon Expert SeminarsMicrosoft CorporationMicrosoft Corporation

AgendaAgenda

IntroductionIntroduction ToolsTools System ArchitectureSystem Architecture Processes and ThreadsProcesses and Threads Memory ManagementMemory Management

About The SpeakerAbout The SpeakerDavid SolomonDavid Solomon 14 years at Digital - the last 10 as a 14 years at Digital - the last 10 as a

developer in the VMS operating developer in the VMS operating system development groupsystem development group

Started Windows NT developer Started Windows NT developer training company in 1992training company in 1992

Author of Author of Inside Windows NT, 2nd Inside Windows NT, 2nd editionedition (Microsoft Press) and (Microsoft Press) andWindows NT for Windows NT for OpenVMS OpenVMS ProfessionalsProfessionals (Digital Press) (Digital Press)

Regular speaker at industry Regular speaker at industry conferences (WinDev, Tech•Ed, conferences (WinDev, Tech•Ed, Software Development, DECUS...)Software Development, DECUS...)

Recipient of past Microsoft MVP Recipient of past Microsoft MVP award for MSWIN32 technical supportaward for MSWIN32 technical support

About The CompanyAbout The Company David Solomon Expert Seminars offers high-quality David Solomon Expert Seminars offers high-quality

Windows developer trainingWindows developer training Taught by well known industry experts and authors Taught by well known industry experts and authors

who develop and teach their own courseswho develop and teach their own courses Instructors include:Instructors include:

Doug Boling, Brian Catlin, Jamie Hanrahan, Jeff Doug Boling, Brian Catlin, Jamie Hanrahan, Jeff Prosise, Jeffrey Richter, and David SolomonProsise, Jeffrey Richter, and David Solomon

Topics include:Topics include: Windows CEWindows CE Windows NT InternalsWindows NT Internals Windows NT and WDM Device DriversWindows NT and WDM Device Drivers Windows NT® Server ApplicationsWindows NT® Server Applications Win32® ProgrammingWin32® Programming Visual C++® and MFCVisual C++® and MFC COM/ActiveX® ProgrammingCOM/ActiveX® Programming

To be notified of new classes and other To be notified of new classes and other developments, join our e-mail interest listdevelopments, join our e-mail interest list

Session GoalsSession Goals GoalsGoals

Explain internal architecture and operation of core Explain internal architecture and operation of core Windows NT componentsWindows NT components

Use various tools that demonstration internal Use various tools that demonstration internal Windows NT behaviorWindows NT behavior

Audience assumptionsAudience assumptions Familiar with basic 32-bit OS conceptsFamiliar with basic 32-bit OS concepts Familiar with Win32 API (processes, threads, Familiar with Win32 API (processes, threads,

memory management)memory management) AcknowledgementsAcknowledgements

Jamie Hanrahan ([email protected] - www.cmkrnl.com), Jamie Hanrahan ([email protected] - www.cmkrnl.com), co-author of the Windows NT internals seminar from co-author of the Windows NT internals seminar from which these slides were takenwhich these slides were taken

Dave Cutler, Helen Custer, John Balciunas, Lou Perazzoli, Dave Cutler, Helen Custer, John Balciunas, Lou Perazzoli, Mark Lucovsky, Steve Wood, Tom Miller, Gary Kimura, Mark Lucovsky, Steve Wood, Tom Miller, Gary Kimura, and Landy Wang for their support and assistance in and Landy Wang for their support and assistance in understanding Windows NT internalsunderstanding Windows NT internals

Device driversDevice drivers

Win32Win32User,GDIUser,GDI

VirtualVirtualMemoryMemory

ProcessesProcesses& Threads& Threads SecuritySecurityCacheCache

ManagerManagerI/O ManagerI/O Manager

Hardware interfaces (buses, I/O, interrupts, timers, Hardware interfaces (buses, I/O, interrupts, timers, clocks, DMA, cache control, etc.)clocks, DMA, cache control, etc.)

ReplicatorReplicatorAlerterAlerter

Service Service ControllerController

WinLogonWinLogon RPCRPC

Environment Environment SubsystemsSubsystems

UserUserApplicationApplication

Subsystem DLLsSubsystem DLLs

POSIXPOSIX

OS/2OS/2

SessionSessionManagerManager

System System ProcessesProcesses ServicesServices ApplicationsApplications

Copyright by Microsoft Corporation. Used by permission.Copyright by Microsoft Corporation. Used by permission.

FileFilesystemssystems Object management / Executive RTLObject management / Executive RTL

KernelKernel

Hardware Abstraction Layer (HAL)Hardware Abstraction Layer (HAL)

EventEventLoggerLogger

UserUserModeMode

SystemSystemThreadsThreads

KernelKernelModeMode

Executive APIExecutive API

Win32Win32

NTDLL.DLLNTDLL.DLL

Windows NT ArchitectureWindows NT Architecture

Windows NT 5.0 Windows NT 5.0 Internal changesInternal changes

In one sense, much is the sameIn one sense, much is the same Basic architecture of many Basic architecture of many

components unchanged:components unchanged: Win32 subsystem, memory manager, process Win32 subsystem, memory manager, process

model, thread scheduling, security model, model, thread scheduling, security model, file systemfile system

But lots of additions of major But lots of additions of major new functionality:new functionality: Active Directory, distributed security, Kerberos, Active Directory, distributed security, Kerberos,

Microsoft management console, IntelliMirrorMicrosoft management console, IntelliMirror™™, , NTFS extensions (content indexing, quotas, reparse NTFS extensions (content indexing, quotas, reparse points, sparse files, link tracking)points, sparse files, link tracking)

Windows NT 5.0 Windows NT 5.0 Internal changesInternal changes

Kernel/core changes include:Kernel/core changes include: I/O system (plug and play and power management)I/O system (plug and play and power management) 64-bit Very Large Memory support for Alpha64-bit Very Large Memory support for Alpha Job objectJob object Integration of Terminal ServerIntegration of Terminal Server

Comparable to level of change from 3.51 to 4.0Comparable to level of change from 3.51 to 4.0 Also many incremental Also many incremental

performance improvements:performance improvements: Object Manager, Memory manager (e.g., working Object Manager, Memory manager (e.g., working

set management algorithms), SMP scalability…set management algorithms), SMP scalability…

AgendaAgenda


tooltool executableexecutable originorigin

Performance Monitor Performance Monitor PerfMonPerfMon Windows NTWindows NT

Registry Editor Registry Editor RegEdt32RegEdt32 Windows NT Windows NT

Windows NT Diagnostics Windows NT Diagnostics WinMSDWinMSD Windows NTWindows NT

Kernel Debugger Kernel Debugger i386kd, i386kd, Widows NT CD \support\debugWidows NT CD \support\debug

alphakdalphakd

Pool MonitorPool Monitor poolmonpoolmon Windows NT CD \support\debugWindows NT CD \support\debug

Global FlagsGlobal Flags gflagsgflags Windows NT Resource KitWindows NT Resource Kit

Open HandlesOpen Handles ohoh Windows NT Resource KitWindows NT Resource Kit

QuickSlice QuickSlice qsliceqslice Windows NT Resource Kit Windows NT Resource Kit

Process Viewer Process Viewer pviewer,pviewer, Windows NT Resource Kit Windows NT Resource Kit pviewpview Platform SDK, VC++Platform SDK, VC++

Process Exploder Process Exploder pviewpview Windows NT Resource Kit 4.0Windows NT Resource Kit 4.0

Process StatusProcess Status pstatpstat Windows NT Resource Kit Windows NT Resource Kit

PmonPmon pmonpmon Windows NT Resource KitWindows NT Resource Kit

Object ViewerObject Viewer WinObjWinObj Platform SDKPlatform SDK

Process Walker Process Walker PWalkPWalk Platform SDKPlatform SDK

Page Fault Monitor Page Fault Monitor PFMonPFMon Platform SDKPlatform SDK

Spy++ Spy++ Visual C++Visual C++

Tools PreviewTools Preview

Windows NT Resource KitsWindows NT Resource Kits

Full “Windows NT 5.0 Resource Kit”Full “Windows NT 5.0 Resource Kit” 250+ utilities250+ utilities Combines what was in the 4.0 Server and Combines what was in the 4.0 Server and

Workstation resource kitsWorkstation resource kits

Subset “Windows NT 5.0 Resource Kit Subset “Windows NT 5.0 Resource Kit Support Tools”Support Tools” 50 utilities50 utilities Ships in \support\reskit on Windows NT CDShips in \support\reskit on Windows NT CD

www.sysinternals.comwww.sysinternals.com

Windows NT internals articles and toolsWindows NT internals articles and tools Some generated using reverse engineering Some generated using reverse engineering

(e.g., no source access)(e.g., no source access) Some examples:Some examples:

winobj - view object manager namespace winobj - view object manager namespace and objectsand objects

nthandlex - show open handles by process nthandlex - show open handles by process ntfilmon - log all file I/O operationsntfilmon - log all file I/O operations ntregmon - log all registry accesses ntregmon - log all registry accesses cpufrob - change thread quantumcpufrob - change thread quantum

Caveat: Most include a device driver, hence Caveat: Most include a device driver, hence you’re added “trusted code”you’re added “trusted code” No warranty on using these on your system!No warranty on using these on your system!

GFLAGS (Global Flags)GFLAGS (Global Flags)

Changes system-wide Changes system-wide or image-wide or image-wide debugging flagsdebugging flags

Poolmon requires Poolmon requires “enable pool taggin”“enable pool taggin”

Oh (open handles) Oh (open handles) requires “maintain a requires “maintain a list of objects for list of objects for each type”each type”

Windows NT Kernel Windows NT Kernel Debugger (1 Of 4)Debugger (1 Of 4) Two versions:Two versions:

Command line: I386KD.EXE, ALPHAKD, etc., shipped with Command line: I386KD.EXE, ALPHAKD, etc., shipped with Windows NTWindows NT In NTcdrom:\support\debug\i386, … \debug\alpha, etc.In NTcdrom:\support\debug\i386, … \debug\alpha, etc. Select directory to match host system (where you will Select directory to match host system (where you will

run the debugger executable); select executable to run the debugger executable); select executable to match target system (system being debugged)match target system (system being debugged)

Also need many DLLs from this directoryAlso need many DLLs from this directory Also need symbol files from NTcdrom:\support\debug\Also need symbol files from NTcdrom:\support\debug\

targetarch\symbols\ …targetarch\symbols\ … Extended via WinDbg shipped with Platform SDK Extended via WinDbg shipped with Platform SDK

(part of MSDN Professional)(part of MSDN Professional) Provides GUI, fully-symbolic, source-level debuggingProvides GUI, fully-symbolic, source-level debugging Needs same DLLs and symbol filesNeeds same DLLs and symbol files

Windows NT Kernel Windows NT Kernel Debugger (2 Of 4)Debugger (2 Of 4) Documentation:Documentation:

Windows NT Workstation Resource Guide Windows NT Workstation Resource Guide (see “Windows NT Debugger”)(see “Windows NT Debugger”)

Windows NT Device Driver Kit (DDK)Windows NT Device Driver Kit (DDK) See i386kd -?See i386kd -? Help within debugger: commands “?” and “!?” Help within debugger: commands “?” and “!?”

and “!help”and “!help”

serial “null modem” cableserial “null modem” cable(for debugger)(for debugger)

hosthost targettarget

Windows NT Kernel Windows NT Kernel Debugger (3 Of 4)Debugger (3 Of 4) Two modes of operation:Two modes of operation:

Open a crash dump file:Open a crash dump file:C:\> set _NT_SYMBOL_PATH= ntcdrom:\support\debug\i386\C:\> set _NT_SYMBOL_PATH= ntcdrom:\support\debug\i386\symbolssymbolsC:\> i386kd -Z dumpfilenameC:\> i386kd -Z dumpfilename

Connect to a live system via null modem cableConnect to a live system via null modem cable(must boot target system with /DEBUG/DEBUGPORT=COMn in (must boot target system with /DEBUG/DEBUGPORT=COMn in boot.ini)boot.ini)C:\> set _NT_SYMBOL_PATH=ntcdrom:\support\debug\i386\C:\> set _NT_SYMBOL_PATH=ntcdrom:\support\debug\i386\symbolssymbolsC:\> set _NT_DEBUG_PORT=COMnC:\> set _NT_DEBUG_PORT=COMn default COM1default COM1C:\> set _NT_DEBUG_BAUD_RATE=nnnnnC:\> set _NT_DEBUG_BAUD_RATE=nnnnn default 19200default 19200C:\> i386kdC:\> i386kd

Windows NT Kernel Windows NT Kernel Debuggers (4 Of 4)Debuggers (4 Of 4) Third-party product: SoftICE for Third-party product: SoftICE for

Windows NT (NuMega)Windows NT (NuMega) Runs on same system - e.g., doesn’t Runs on same system - e.g., doesn’t

require second system for live debuggingrequire second system for live debugging x86 onlyx86 only See www.numega.comSee www.numega.com

AgendaAgenda

IntroductionIntroduction ToolsTools System ArchitectureSystem Architecture

Kernel Mode EnvironmentKernel Mode Environment Executive, Kernel, HAL, DriversExecutive, Kernel, HAL, Drivers Product PackagingProduct Packaging System ThreadsSystem Threads Environment SubsystemsEnvironment Subsystems System Service DispatchingSystem Service Dispatching Process-based Windows NT codeProcess-based Windows NT code SummarySummary

Processes and ThreadsProcesses and Threads Memory ManagementMemory Management

Kernel Mode Versus User ModeKernel Mode Versus User Mode

A processor stateA processor state Controls access to memoryControls access to memory Each memory page is tagged Each memory page is tagged

to show the required mode for to show the required mode for reading and for writingreading and for writing Protects the system from Protects the system from

the usersthe users Protects the user (process) Protects the user (process)

from themselvesfrom themselves System is not protected System is not protected

from systemfrom system Code regions are tagged “no Code regions are tagged “no

write in any mode”write in any mode” Controls ability to execute Controls ability to execute

privileged instructionsprivileged instructions A Windows NT abstractionA Windows NT abstraction

Intel: Ring 0, Ring 3 Intel: Ring 0, Ring 3 PerfMon, Processor: PerfMon, Processor:

“Privileged Time” and “Privileged Time” and “User Time”“User Time”

ComponentsComponents Access modeAccess mode

ApplicationsApplications UserUser

Subsystem processesSubsystem processes UserUser

ExecutiveExecutive KernelKernel

KernelKernel KernelKernel

DriversDrivers KernelKernel

HALHAL KernelKernel

Associated with threadsAssociated with threads Threads can change from user Threads can change from user

to kernel and backto kernel and back Part of saved context, along Part of saved context, along

with registers, etc.with registers, etc. Does not affect schedulingDoes not affect scheduling

Getting Into Kernel ModeGetting Into Kernel Mode

Code is run in kernel mode for one of three reasons:Code is run in kernel mode for one of three reasons:1. Requests from user mode1. Requests from user mode

Via the system service dispatch mechanismVia the system service dispatch mechanism Kernel-mode code runs in the context of the requesting threadKernel-mode code runs in the context of the requesting thread

2. Interrupts from external devices2. Interrupts from external devices Windows NT-supplied interrupt dispatcher invokes the interrupt Windows NT-supplied interrupt dispatcher invokes the interrupt

service routineservice routine ISR runs in the context of the interrupted thread (so-called ISR runs in the context of the interrupted thread (so-called

“arbitrary thread context”)“arbitrary thread context”) ISR often requests the execution of a “DPC routine,” which also ISR often requests the execution of a “DPC routine,” which also

runs in kernel moderuns in kernel mode Time not charged to interrupted threadTime not charged to interrupted thread

3. Dedicated kernel-mode system threads3. Dedicated kernel-mode system threads Some threads in the system stay in kernel mode at all times Some threads in the system stay in kernel mode at all times

(mostly in the “System” process)(mostly in the “System” process) Scheduled, preempted, etc., like any other threadsScheduled, preempted, etc., like any other threads

Interrupt dispatch routineInterrupt dispatch routine

Disable interruptsDisable interrupts

Record machine state (trap Record machine state (trap frame) to allow resumeframe) to allow resume

Mask equal- and lower-IRQL Mask equal- and lower-IRQL interruptsinterrupts

Find and call appropriate Find and call appropriate ISRISR

Dismiss interruptDismiss interrupt

Restore machine state Restore machine state (including mode and (including mode and enabled interrupts)enabled interrupts)

Tell the device to stop Tell the device to stop interruptinginterrupting

Interrogate device state, Interrogate device state, start next operation on start next operation on device, etc. device, etc.

Request a DPCRequest a DPC

Return to callerReturn to caller

Interrupt service routineInterrupt service routine

interrupt !interrupt !

user or user or kernel modekernel mode

codecodekernel modekernel mode

Note, no thread or Note, no thread or process context process context switch!switch!

Interrupt DispatchingInterrupt Dispatching

LowLowAPCAPC

Dispatch/DPCDispatch/DPCDevice 1Device 1

..

..

..Device nDevice n

ClockClockInterprocessor InterruptInterprocessor Interrupt

Power failPower failHighHigh

normal thread executionnormal thread execution

Hardware interruptsHardware interrupts

Deferrable software interruptsDeferrable software interrupts

001122

303029292828

3131

Interrupt Precedence Via IRQLsInterrupt Precedence Via IRQLs

IRQL = Interrupt Request LevelIRQL = Interrupt Request Level The “precedence” of the interrupt The “precedence” of the interrupt

with respect to other interruptswith respect to other interrupts Different interrupt sources have Different interrupt sources have

different IRQLsdifferent IRQLs Not the same as IRQNot the same as IRQ

IRQL is also a state of the IRQL is also a state of the processorprocessor

Servicing an interrupt raises Servicing an interrupt raises processor IRQL to that processor IRQL to that interrupt’s IRQLinterrupt’s IRQL This masks subsequent This masks subsequent

interrupts at equal and lower interrupts at equal and lower IRQLsIRQLs

User mode is limited to IRQL 0User mode is limited to IRQL 0

LowLowAPCAPC

Dispatch/DPCDispatch/DPCDeviceDevice

Device HighDevice HighClockClock

Interprocessor InterruptInterprocessor InterruptHighHigh

001122

665544

77

33

Alpha IRQLsAlpha IRQLs

IRQL on Alpha implemented in PAL codeIRQL on Alpha implemented in PAL code

queue headqueue head DPC objectDPC object DPC objectDPC object DPC objectDPC object

XydriverDpcRtn(DpcObj, XydriverDpcRtn(DpcObj, DfrdCtx,SysArg1,SysArg2)DfrdCtx,SysArg1,SysArg2){{ // ...// ...}}

DfrdCtxDfrdCtxSysArg1SysArg1SysArg2SysArg2

DPCs (Deferred Procedure Calls)DPCs (Deferred Procedure Calls)

A list of “work requests”A list of “work requests” One queue per processor (but processors can run each others’ DPCs)One queue per processor (but processors can run each others’ DPCs) Implicitly ordered by time of request (FIFO)Implicitly ordered by time of request (FIFO)

Used to defer processing from higher (device) interrupt level to a Used to defer processing from higher (device) interrupt level to a lower (dispatch) levellower (dispatch) level Used heavily for driver Used heavily for driver

“after interrupt” functions“after interrupt” functions Used for quantum end and timer expirationUsed for quantum end and timer expiration

Screen snapshot from: Programs | Screen snapshot from: Programs | Administrative Tools | Performance MonitorAdministrative Tools | Performance Monitorclick on “+” button, or select Edit | Add to chart…click on “+” button, or select Edit | Add to chart…

Accounting For Accounting For Kernel-Mode TimeKernel-Mode Time ““Processor Time” = total busy Processor Time” = total busy

time of processor (equal to time of processor (equal to elapsed real time - idle time)elapsed real time - idle time)

““Processor Time” = “User Processor Time” = “User Time” + “Privileged Time”Time” + “Privileged Time”

““Privileged Time” = time Privileged Time” = time spent in kernel modespent in kernel mode

““Privileged Time” includes:Privileged Time” includes: Interrupt TimeInterrupt Time DPC TimeDPC Time

Again note: interrupts and Again note: interrupts and DPCs are not charged to any DPCs are not charged to any process or threadprocess or thread

AgendaAgenda




Windows NT ExecutiveWindows NT Executive

Upper layers of operating systemUpper layers of operating system Provides “generic OS” servicesProvides “generic OS” services

Processes, threads, memory management, Processes, threads, memory management, I/O, interprocess communication, I/O, interprocess communication, synchronization, securitysynchronization, security

Almost completely portable C codeAlmost completely portable C code Exports functions (“services”) which may Exports functions (“services”) which may

be invoked via user-mode APIsbe invoked via user-mode APIs Interface is NTDLL.DLLInterface is NTDLL.DLL E.g., Win32 ReadFile -> executive NtReadFileE.g., Win32 ReadFile -> executive NtReadFile

Most interfaces to executive services not Most interfaces to executive services not documenteddocumented Used by subsystem writersUsed by subsystem writers

Machine Independent CMachine Independent C

AssemblerAssembler

Machine Dep. CMachine Dep. C

Windows NT KernelWindows NT Kernel

Abstracts differences between processor Abstracts differences between processor architecturesarchitectures x86 vs. Alpha vs., etc.x86 vs. Alpha vs., etc.

Main servicesMain services Thread scheduling and context switchingThread scheduling and context switching Generic wait operations Generic wait operations Exception and interrupt dispatchingException and interrupt dispatching Operating system synchronization Operating system synchronization

primitives (MP and UP)primitives (MP and UP)

Not a classic “microkernel”Not a classic “microkernel” shares address space shares address space

withrest of kernel-mode withrest of kernel-mode componentscomponents

A separate loaded binary (c:\winnt\system32\hal.dll)A separate loaded binary (c:\winnt\system32\hal.dll) Several different versions for different motherboards, UP vs. MP, etc.Several different versions for different motherboards, UP vs. MP, etc. Installation procedure selects appropriate HAL for platform and copies Installation procedure selects appropriate HAL for platform and copies

to Hal.Dll on system diskto Hal.Dll on system disk Purpose:Purpose:

Isolate (abstract) Kernel and Executive from platform-specific detailsIsolate (abstract) Kernel and Executive from platform-specific details Present uniform model for ease of driver developmentPresent uniform model for ease of driver development

HAL abstracts:HAL abstracts: I/O system specifics (bus interfaces, DMA…)I/O system specifics (bus interfaces, DMA…) System timers, Cache coherency and flushingSystem timers, Cache coherency and flushing SMP support, Hardware interrupt prioritiesSMP support, Hardware interrupt priorities

OEM Development Kit needed to buildHALsOEM Development Kit needed to buildHALs HAL contains some Executive and HAL contains some Executive and

Kernel subroutinesKernel subroutines

HalGetBusDataHalGetBusDataHalGetBusDataByOffsetHalGetBusDataByOffsetHalAssignSlotResourcesHalAssignSlotResourcesHalSetBusDataHalSetBusDataHalSetBusDataByOffsetHalSetBusDataByOffsetHalTranslateBusAddressHalTranslateBusAddressHalGetInterruptVectorHalGetInterruptVectorHalGetAdapterHalGetAdapterREAD_REGISTER_ULONG READ_REGISTER_ULONG WRITE_PORT_UCHARWRITE_PORT_UCHAR

Sample HAL routines:Sample HAL routines:

HAL - Hardware HAL - Hardware Abstraction LayerAbstraction Layer

Kernel-Mode Device DriversKernel-Mode Device Drivers Separate loadable modules (drivername.SYS)Separate loadable modules (drivername.SYS)

Linked like .EXEsLinked like .EXEs Linked against NTOSKRNL.EXE and HAL.DLLLinked against NTOSKRNL.EXE and HAL.DLL

Only way to add “kernel extensions” or to access Only way to add “kernel extensions” or to access kernel mode system routines kernel mode system routines

Defined in registryDefined in registry Same area as Win32 services (t.b.d.)Same area as Win32 services (t.b.d.) Differentiated by Type valueDifferentiated by Type value

View loaded drivers with pstat.exe, drivers.exeView loaded drivers with pstat.exe, drivers.exe Several types:Several types:

““Ordinary” hardware driversOrdinary” hardware drivers File systemFile system NDIS miniport, SCSI miniport (linked against port drivers)NDIS miniport, SCSI miniport (linked against port drivers) Win32K.Sys - Windowing systemWin32K.Sys - Windowing system

WDM (Win32 Driver Model)WDM (Win32 Driver Model)

Extension to Windows NT driver model Extension to Windows NT driver model to support for Plug and Play and Power to support for Plug and Play and Power ManagementManagement

Allows source/(x86) binary-compatible Allows source/(x86) binary-compatible drivers across Windows 98 and drivers across Windows 98 and Windows NT 5.0Windows NT 5.0

Non trivial additions to existing drivers:Non trivial additions to existing drivers: 3 new major IRP types3 new major IRP types 36 new minor IRPs added36 new minor IRPs added 6 new miniport driver types6 new miniport driver types Supporting WDM affects every area of Supporting WDM affects every area of

a drivera driver

WDM DriversWDM Drivers What’s covered in WDM:What’s covered in WDM:

IEEE 1394 (Firewire)IEEE 1394 (Firewire) Universal Serial Bus (USB)Universal Serial Bus (USB) Audio: Speakers, microphone, CODECAudio: Speakers, microphone, CODEC Human Interface Devices: mouse, keyboard, Human Interface Devices: mouse, keyboard,

monitor controls, game devicesmonitor controls, game devices Still Imaging: Cameras, scannersStill Imaging: Cameras, scanners Video Devices: Video capture, DVDVideo Devices: Video capture, DVD Advanced Power and Configuration Interface Advanced Power and Configuration Interface

(ACPI) BIOS support(ACPI) BIOS support Not covered by WDM:Not covered by WDM:

NetworkNetwork StorageStorage File SystemFile System VideoVideo




AgendaAgenda

Device driversDevice drivers

Win32Win32User,GDIUser,GDI

VirtualVirtualMemoryMemory

ProcessesProcesses& Threads& Threads SecuritySecurityCacheCache

ManagerManagerI/O ManagerI/O Manager

Hardware interfaces (buses, I/O, interrupts, timers, Hardware interfaces (buses, I/O, interrupts, timers, clocks, DMA, cache control, etc.)clocks, DMA, cache control, etc.)

ReplicatorReplicatorAlerterAlerter

Service Service ControllerController

WinLogonWinLogon RPCRPC

EnvironmentEnvironmentSubsystemsSubsystems


Subsystem DLLsSubsystem DLLs

POSIXPOSIX

OS/2OS/2

SessionSessionManagerManager

System ProcessesSystem Processes ServicesServices ApplicationsApplications

Copyright by Microsoft Corporation. Used by permission.Copyright by Microsoft Corporation. Used by permission.

FileFilesystemssystems Object management / Executive RTLObject management / Executive RTL

KernelKernel


EventEventLoggerLogger

UserUserModeMode

SystemSystemThreadsThreads


Executive APIExecutive API

Win32Win32

NTDLL.DLLNTDLL.DLL

Nto

sK

rnl.E

xN

tos

Krn

l.Ex

ee

NTOSKRNL.EXENTOSKRNL.EXE

NTOSKRNL.EXENTOSKRNL.EXE

NTOSKRNL.EXENTOSKRNL.EXE Windows NT executive Windows NT executive

and kerneland kernel

HAL.DLLHAL.DLL Hardware Abstraction Hardware Abstraction

Layer - interface to Layer - interface to hardware platformhardware platform

BOOTVID.DLLBOOTVID.DLL Boot video driverBoot video driver

Naming Convention For Naming Convention For Internal Windows NT RoutinesInternal Windows NT Routines Two- or three-letter component code in beginning of function nameTwo- or three-letter component code in beginning of function name

ExEx - General executive routine- General executive routine ObOb - Object management- Object management ExpExp - Executive private (not exported)- Executive private (not exported) IoIo - I/O subsystem- I/O subsystemCcCc - Cache manager- Cache manager SeSe - Security - Security MmMm - Memory management- Memory management PsPs - Process structure- Process structureRtlRtl - Run-Time Library- Run-Time Library LsaLsa - Security Authentication- Security AuthenticationFsRtlFsRtl - File System Run-Time Lib- File System Run-Time Lib ZwZw - File access, etc.- File access, etc.

KeKe - Kernel- KernelKiKi - Kernel internal (not available outside the kernel)- Kernel internal (not available outside the kernel)

HalHal - Hardware Abstraction Layer- Hardware Abstraction LayerREAD_, WRITE_ - I/O port and register accessREAD_, WRITE_ - I/O port and register access

ExecutiveExecutive

KernelKernel

HALHAL

Multiprocessor SupportMultiprocessor Support Code comprising NTOSKRNL compiled twice: Code comprising NTOSKRNL compiled twice:

Once for uniprocessor, once for multiprocessorOnce for uniprocessor, once for multiprocessor Avoids penalizing uniprocessor systems for Avoids penalizing uniprocessor systems for

added MP complexityadded MP complexity Two files on Windows NT media:Two files on Windows NT media:

UP version: NTOSKRNL.EXEUP version: NTOSKRNL.EXE MP version: NTKRNLMP.EXEMP version: NTKRNLMP.EXE Selected at installation time, but copied to NTOSKRNLSelected at installation time, but copied to NTOSKRNL

All drivers, DLLs, EXEs are built to run on on MPAll drivers, DLLs, EXEs are built to run on on MP Upgrading from Uniprocessor vs MultiprocessorUpgrading from Uniprocessor vs Multiprocessor

See uptomp.exe (in Resource Kit)See uptomp.exe (in Resource Kit) 2 files replaced with different code2 files replaced with different code

NTKRNLMP.EXE replaces NTOSKRNL.EXENTKRNLMP.EXE replaces NTOSKRNL.EXE new HAL replaces HAL.DLLnew HAL replaces HAL.DLL

4 files replaced with same code, but modified image header4 files replaced with same code, but modified image header KERNEL32.DLL, NTDLL.DLL, WINSRV.DLL, WIN32K.SYSKERNEL32.DLL, NTDLL.DLL, WINSRV.DLL, WIN32K.SYS

Screen snapshot from:Screen snapshot from:Programs | Administrative ToolsPrograms | Administrative Tools| Windows NT Diagnostics| Windows NT Diagnostics

Identifying Your NTOSKRNLIdentifying Your NTOSKRNL Build numbersBuild numbers

Incremented each time Incremented each time Windows NT is built from sources Windows NT is built from sources (i.e., different for beta releases)(i.e., different for beta releases)

Service packsService packs Replaces .EXEs (including usually Replaces .EXEs (including usually

NTOSKRNL), .DLLs, etc.NTOSKRNL), .DLLs, etc. Do not change Windows NT Do not change Windows NT

build numberbuild number Free versus Checked buildFree versus Checked build

Free = retail version; Checked = Free = retail version; Checked = debug versiondebug version

Used primarily in driver testingUsed primarily in driver testing Build number is the sameBuild number is the same Recompilation of system with Recompilation of system with

DEBUG flag trueDEBUG flag true Therefore a different Therefore a different

NTOSKRNL.EXENTOSKRNL.EXE Note: MP only (NTOSKRNL and Note: MP only (NTOSKRNL and

NTKRNLMP.EXE identical)NTKRNLMP.EXE identical)

Workstation Vs ServerWorkstation Vs Server

Core operating system executables Core operating system executables are identicalare identical NTOSKRNL.EXE, HAL.DLL, xxxDRIVER.SYS, NTOSKRNL.EXE, HAL.DLL, xxxDRIVER.SYS,

etc., (t.b.d.)etc., (t.b.d.) Windows NT Server a superset of Windows NT Server a superset of

WorkstationWorkstation domains, host-based RAID 5, NetWare gateway, domains, host-based RAID 5, NetWare gateway,

DHCP server, WINS, DNS, full Internet DHCP server, WINS, DNS, full Internet Information Server…Information Server…

Enterprise Server adds yet more functionality Enterprise Server adds yet more functionality (Clusters, 3GB address space)(Clusters, 3GB address space)

Terminal Server enables multi-user thin Terminal Server enables multi-user thin client supportclient support

MP limits: Workstation: 2 CPUs, Server: MP limits: Workstation: 2 CPUs, Server: 4 CPUs, Server Enterprise: 8 CPUs4 CPUs, Server Enterprise: 8 CPUs

Workstation Vs ServerWorkstation Vs Server

Registry indicates system typeRegistry indicates system type HKLM\CurrentControlSet\Control\ProductOptionsHKLM\CurrentControlSet\Control\ProductOptions

ProductType: WinNT=Workstation, ProductType: WinNT=Workstation, ServerNT=Server not a domain controller, ServerNT=Server not a domain controller, LanManNT=Server that is a Domain ControllerLanManNT=Server that is a Domain Controller

ProductSuite: Indicates Enterprise Edition, ProductSuite: Indicates Enterprise Edition, Terminal Server…Terminal Server…

Code in the operating system tests these Code in the operating system tests these values and behaves slightly differently in values and behaves slightly differently in a few placesa few places Licensing limits (number of processors, number Licensing limits (number of processors, number

of inbound network connections, etc.)of inbound network connections, etc.) Boot-time calculations (memory manager)Boot-time calculations (memory manager) Default length of time sliceDefault length of time slice See DDK: MmIsThisAnNtasSystemSee DDK: MmIsThisAnNtasSystem

AgendaAgenda



Processe and ThreadsProcesse and Threads Memory ManagementMemory Management

System ThreadsSystem Threads

Internal worker routines that need thread contextInternal worker routines that need thread context Drivers or Executive can create system threadsDrivers or Executive can create system threads

Always run in kernel modeAlways run in kernel mode Usually associated with the “System” process by defaultUsually associated with the “System” process by default

But can be tied to any processBut can be tied to any process Not non-preemptible (unless they raise IRQL to 2 or above)Not non-preemptible (unless they raise IRQL to 2 or above)

Kernel mode APIs:Kernel mode APIs:

PsCreateSystemThreadPsCreateSystemThread PsTerminateSystemThreadPsTerminateSystemThread KeSetBasePriorityThreadKeSetBasePriorityThread KeSetPriorityThreadKeSetPriorityThread

Screen snapshot from: Programs | Resource Kit |Screen snapshot from: Programs | Resource Kit |Diagnostics | Process ViewerDiagnostics | Process Viewerselect “System” processselect “System” process

Threads In The “System” Threads In The “System” ProcessProcess Note CPU time is 100% Note CPU time is 100%

kernel modekernel mode ““Start address” is Start address” is

address of thread address of thread functionfunction On Intel (at least):On Intel (at least): Addresses 8xxxxxxx will Addresses 8xxxxxxx will

correspond to symbols in correspond to symbols in NtosKrnl.ExeNtosKrnl.Exe

Addresses Axxxxxxx are Addresses Axxxxxxx are routines in Win32K.Sysroutines in Win32K.Sys

Addresses Fxxxxxxx Addresses Fxxxxxxx are routines in loaded are routines in loaded device driversdevice drivers

Threads In The Threads In The “System” Process“System” Process Memory ManagementMemory Management

Modified Page Writer for mapped filesModified Page Writer for mapped files Modified Page Writer for paging filesModified Page Writer for paging files Balance Set ManagerBalance Set Manager Swapper (kernel stack, working sets)Swapper (kernel stack, working sets) Zero page thread (thread 0, priority 0)Zero page thread (thread 0, priority 0)

Security Reference MonitorSecurity Reference Monitor Command Server ThreadCommand Server Thread

NetworkNetwork Redirector and Server Worker ThreadsRedirector and Server Worker Threads

Threads created by drivers for their Threads created by drivers for their exclusive useexclusive use Examples: Floppy driver, parallel port driverExamples: Floppy driver, parallel port driver

Pool of Executive Worker ThreadsPool of Executive Worker Threads Used by drivers, file systems…Used by drivers, file systems… Accessed via ExQueueWorkItemAccessed via ExQueueWorkItem

Threads In System ProcessThreads In System Process(Observed on Intel Windows NT Workstation 4.0 )(Observed on Intel Windows NT Workstation 4.0 )

Routine NameRoutine Name PriorityPriority NotesNotes

Phase1InitializationPhase1Initialization 00 First thread in life of system; becomes zeroFirst thread in life of system; becomes zeropage threadpage thread

ExpWorkerThreadExpWorkerThread 9-169-16 Pool of worker threadsPool of worker threads

MiDereferenceSegmentThreadMiDereferenceSegmentThread 1818 Dereferences segments; also expandsDereferences segments; also expandspaging filepaging file

MiModifiedPageWriterMiModifiedPageWriter 1717 Writes modifed pages to paging fileWrites modifed pages to paging file

KeBalanceSetManagerKeBalanceSetManager 1616 Reclaims memory from processes, with aidReclaims memory from processes, with aidof . . .of . . .

KeSwapProcessOrStackKeSwapProcessOrStack 2323 Scheduled by balance set managerScheduled by balance set manager

FsRtlWorkerThreadFsRtlWorkerThread 16, 1716, 17 Dedicated worker threads for FSDsDedicated worker threads for FSDs

SepRmCommandServerThreadSepRmCommandServerThread 1515 Security Reference Monitor CommandSecurity Reference Monitor CommandServerServer

MiMappedPageWriterMiMappedPageWriter 1717 Writes modified pages to mapped filesWrites modified pages to mapped files

(Win32 threads)(Win32 threads) 1616 routines in Win32K.Sys (0xA0000000)routines in Win32K.Sys (0xA0000000)

(driver threads)(driver threads) variousvarious routines in *driver.Sys (0xF0000000)routines in *driver.Sys (0xF0000000)

AgendaAgenda




Environment SubsystemsEnvironment Subsystems

Expose “native API”Expose “native API” ““Wrap” and extend Windows NT native functionalityWrap” and extend Windows NT native functionality Interfaces to write subsystems not documentedInterfaces to write subsystems not documented

Two main componentsTwo main components Subsystem DLLs - convert documented API to native APISubsystem DLLs - convert documented API to native API Environment Subsystem Process - maintain state of client Environment Subsystem Process - maintain state of client

processes; implement some subsystem APIsprocesses; implement some subsystem APIs

Three provided with Windows NT:Three provided with Windows NT: Win32Win32 PosixPosix

Bare minimum Posix standards, no optional componentsBare minimum Posix standards, no optional components OS/2OS/2

Support for 1.x character-mode applications onlySupport for 1.x character-mode applications only

Subsystem ExtensionsSubsystem Extensions

OS/2OS/2 Microsoft sells an add-on to the Microsoft sells an add-on to the

OS/2 subsystem OS/2 subsystem Supports 1.x Presentation ManagerSupports 1.x Presentation Manager

PosixPosix OpenNT from SoftWayOpenNT from SoftWay More-featured replacement for More-featured replacement for

Posix subsystemPosix subsystem www.opennt.comwww.opennt.com

Subsystem for each .exe specified in image headerSubsystem for each .exe specified in image header See winnt.hSee winnt.h

See Explorer / QuickView (right-click on .exe or .dll file)See Explorer / QuickView (right-click on .exe or .dll file) Or \reskit\exetype image.exeOr \reskit\exetype image.exe

IMAGE_SUBSYSTEM_UNKNOWN 0 // Unknown subsystemIMAGE_SUBSYSTEM_UNKNOWN 0 // Unknown subsystemIMAGE_SUBSYSTEM_NATIVE 1 // Image doesn't require a subsystemIMAGE_SUBSYSTEM_NATIVE 1 // Image doesn't require a subsystemIMAGE_SUBSYSTEM_WINDOWS_GUI 2 // Win32 subsystem (graphical app)IMAGE_SUBSYSTEM_WINDOWS_GUI 2 // Win32 subsystem (graphical app)IMAGE_SUBSYSTEM_WINDOWS_CUI 3 // Win32 subsystem (character cell)IMAGE_SUBSYSTEM_WINDOWS_CUI 3 // Win32 subsystem (character cell)IMAGE_SUBSYSTEM_OS2_CUI 5 // OS/2 subsystemIMAGE_SUBSYSTEM_OS2_CUI 5 // OS/2 subsystemIMAGE_SUBSYSTEM_POSIX_CUI 7 // Posix subsystemIMAGE_SUBSYSTEM_POSIX_CUI 7 // Posix subsystem


Showing .exe Type Showing .exe Type With QuickViewWith QuickView In Explorer:In Explorer:

Right-click on Right-click on an executable an executable file or .DLLfile or .DLL

““Context menu” Context menu” appearsappears

Select Quick Select Quick ViewView

Environment Subsystems Environment Subsystems LoadingLoading

Subsystems to load specified in registry:Subsystems to load specified in registry: \SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems\SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems

Values:Values: RequiredRequired - list of value names for subsystems to load at boot time- list of value names for subsystems to load at boot time OptionalOptional - list of value names for subsystems to load when - list of value names for subsystems to load when

neededneeded WindowsWindows - value giving filespec of Win32 subsystem (csrss.exe)- value giving filespec of Win32 subsystem (csrss.exe)

csrss.execsrss.exe Win32 APIs required (Client Server Runtime Win32 APIs required (Client Server Runtime SubSystem)SubSystem)

os2ss.exeos2ss.exe OS/2 APIsOS/2 APIs optionaloptional

psxss.exepsxss.exe Posix APIsPosix APIs optionaloptional KmodeKmode - value giving filespec of Win32K.Sys - value giving filespec of Win32K.Sys

(kernel-mode component of Win32)(kernel-mode component of Win32) Some Win32 API DLLs are in “known DLLs” registry entry:Some Win32 API DLLs are in “known DLLs” registry entry:

\SYSTEM\CurrenctControlSet\Control\Session Manager\KnownDLLs\SYSTEM\CurrenctControlSet\Control\Session Manager\KnownDLLs

OS/2OS/2Win32Win32

POSIXPOSIX



Subsystem DLLSubsystem DLL

Win32Win32User/GDIUser/GDI

UserUserModeMode

ExecutiveExecutive

Device DriversDevice Drivers KernelKernel



SystemSystemand Serverand ServerProcessesProcesses

NTDLL.DLLNTDLL.DLL

Environment SubsystemsEnvironment SubsystemsComponentsComponents

Subsystem process Subsystem process For Win32: CSRSS.EXEFor Win32: CSRSS.EXE

API DLLs API DLLs For Win32: Kernel32.DLL, Gdi32.DLL, User32.DLL, etc.For Win32: Kernel32.DLL, Gdi32.DLL, User32.DLL, etc.

Kernel-mode extension to executiveKernel-mode extension to executive Win32 only: Win32K.SYSWin32 only: Win32K.SYS

Windows NT Simplified Windows NT Simplified ArchitectureArchitecture(3.51 and earlier)(3.51 and earlier)

OS/2OS/2 Win32Win32 POSIXPOSIX


UserUserModeMode



ExecutiveExecutive



LPCLPC


Subsystem DLLSubsystem DLL11 22

Most Win32 Kernel APIsMost Win32 Kernel APIsAll other Win32 APIs, including User and GDI APIsAll other Win32 APIs, including User and GDI APIs22

11

NTDLL.DLLNTDLL.DLL

OS/2OS/2Win32Win32

POSIXPOSIX


Win32Win32User/GDIUser/GDI

UserUserModeMode

ExecutiveExecutive





11 3322


Subsystem DLLSubsystem DLL

LPCLPC

Most Win32 Kernel APIsMost Win32 Kernel APIsMost Win32 User and GDI APIsMost Win32 User and GDI APIsA few Win32 APIsA few Win32 APIs33

2211

NTDLL.DLLNTDLL.DLL

Windows NT Simplified Windows NT Simplified ArchitectureArchitecture(4.0 and later)(4.0 and later)

(Reduced) Role Of Win32 (Reduced) Role Of Win32 Subsystem ProcessSubsystem Process Process creation and deletionProcess creation and deletion Thread creation and deletionThread creation and deletion Get temporary file nameGet temporary file name Drive lettersDrive letters Security checks for file Security checks for file

system redirectorsystem redirector Window management for console Window management for console

(character cell) applications(character cell) applications Some support for 16-bit DOS support Some support for 16-bit DOS support

(NTVDM.EXE)(NTVDM.EXE)

AgendaAgenda




Invoking System Functions Invoking System Functions From User ModeFrom User Mode Kernel-mode functions (“services”) are invoked from user mode Kernel-mode functions (“services”) are invoked from user mode

via a protected mechanismvia a protected mechanism x86: INT 2E; Alpha: SYSCALL (PALcode)x86: INT 2E; Alpha: SYSCALL (PALcode) I.e., on a call to an OS service from user mode, the last thing that I.e., on a call to an OS service from user mode, the last thing that

happens in user mode is this “change mode to kernel” instructionhappens in user mode is this “change mode to kernel” instruction Causes an interrupt, handled by the system service dispatcher Causes an interrupt, handled by the system service dispatcher

(KiSystemService) in kernel mode(KiSystemService) in kernel mode Return to user mode is done by dismissing the interrupt or exceptionReturn to user mode is done by dismissing the interrupt or exception

The desired system function is selected by the “system The desired system function is selected by the “system service number”service number” Every Windows NT function exported to user mode has a Every Windows NT function exported to user mode has a

unique numberunique number Push this number on the stack just before the Push this number on the stack just before the

“change mode” instruction “change mode” instruction (after pushing the arguments to the service)(after pushing the arguments to the service)

This number is an index into the system service dispatch tableThis number is an index into the system service dispatch table Table gives kernel-mode entry point address and argument list Table gives kernel-mode entry point address and argument list

length for each exported functionlength for each exported function

Invoking System Functions Invoking System Functions From User ModeFrom User Mode All validity checks are done after the user to kernel transitionAll validity checks are done after the user to kernel transition

KiSystemService probes argument list, copies it to kernel-mode stack, KiSystemService probes argument list, copies it to kernel-mode stack, and calls the executive or kernel routine pointed to by the tableand calls the executive or kernel routine pointed to by the table

Service-specific routine checks argument values, probes pointed-to Service-specific routine checks argument values, probes pointed-to buffers, etc.buffers, etc.

Once past that point, everything is “trusted”Once past that point, everything is “trusted” This is safe, because:This is safe, because:

The system service table is in kernel-protected memory; andThe system service table is in kernel-protected memory; and The kernel mode routines pointed to by the system service table are The kernel mode routines pointed to by the system service table are

in kernel-protected memory; therefore:in kernel-protected memory; therefore: User mode code can’t supply the code to be run in kernel mode; it User mode code can’t supply the code to be run in kernel mode; it

can only select from among a predefined listcan only select from among a predefined list Arguments are copied to the kernel mode stack before Arguments are copied to the kernel mode stack before

validation; therefore:validation; therefore: Other threads in the process can’t corrupt the arguments “out from Other threads in the process can’t corrupt the arguments “out from

under” the serviceunder” the service

NTDLL.DLLNTDLL.DLL PUSH of service # and INT 2E are “wrapped” by small “jacket” PUSH of service # and INT 2E are “wrapped” by small “jacket”

procedures in NTDLL.DLLprocedures in NTDLL.DLL These user-mode routines have the same function names and These user-mode routines have the same function names and

arguments as the kernel mode routines they call arguments as the kernel mode routines they call E.g., NtWriteFile in NtDll.Dll invokes NtWriteFile in NtosKrnl.ExeE.g., NtWriteFile in NtDll.Dll invokes NtWriteFile in NtosKrnl.Exe

Therefore exports of NTDLL are the “NT native API”Therefore exports of NTDLL are the “NT native API” Entry points in NtDll.Dll are not supported or documented for use Entry points in NtDll.Dll are not supported or documented for use

from user mode appsfrom user mode apps A few are documented in the DDK for call from kernel modeA few are documented in the DDK for call from kernel mode A few images that come with Windows NT are written to the “native A few images that come with Windows NT are written to the “native

API” exposed by NtDll.Dll (“Windows NT native images”)API” exposed by NtDll.Dll (“Windows NT native images”) See article on www.sysinternals.comSee article on www.sysinternals.com

NTDLL also contains image loader and other support functionsNTDLL also contains image loader and other support functions What about getting to USER and GDI functions in Win32K.SYS?What about getting to USER and GDI functions in Win32K.SYS?

System service wrapper exists in USER32.DLL, GDI32.DLLSystem service wrapper exists in USER32.DLL, GDI32.DLL Does not go through NTDLL.DLLDoes not go through NTDLL.DLL

call WriteFile(…)call WriteFile(…)

call NtWriteFilecall NtWriteFilereturn to callerreturn to caller

do the operationdo the operationreturn to callerreturn to caller

Int 2EInt 2Ereturn to callerreturn to caller

call NtWriteFilecall NtWriteFiledismiss interruptdismiss interrupt

Win32 applicationWin32 application

WriteFile WriteFile in Kernel32.Dllin Kernel32.Dll

NtWriteFileNtWriteFilein NtDll.Dllin NtDll.Dll

KiSystemServiceKiSystemServicein NtosKrnl.Exein NtosKrnl.Exe

NtWriteFileNtWriteFilein NtosKrnl.Exein NtosKrnl.Exe

Tracing An Example Win32 CallTracing An Example Win32 Call

Win32-Win32-specificspecific

used by all used by all subsystemssubsystems

software interruptsoftware interrupt

UU

KK

Source: MSJ, August Source: MSJ, August 1996, page 21 1996, page 21 (by Matt Pietrek)(by Matt Pietrek)

Depends.Exe in Resource Kit and Platform SDKDepends.Exe in Resource Kit and Platform SDK Allows viewing of image->DLL relationships, imports, Allows viewing of image->DLL relationships, imports,

and exportsand exports

Tracing An Example Win32 CallTracing An Example Win32 Call

Examining Symbols Examining Symbols In Key ImagesIn Key Images Examine imports and exports of an .EXE down Examine imports and exports of an .EXE down

to the OSto the OS In Explorer, right mouse click on EXE or DLL, then In Explorer, right mouse click on EXE or DLL, then

“quick view” (built in) or “View Dependencies” “quick view” (built in) or “View Dependencies” (Dependency Walker tool in ResKit and Platform SDK)(Dependency Walker tool in ResKit and Platform SDK)

Or use LINK /DUMP /EXPORTS, /IMPORTSOr use LINK /DUMP /EXPORTS, /IMPORTS

1. Look at imports of \winnt\system32\notepad.exe1. Look at imports of \winnt\system32\notepad.exe

2. Look at exports and imports of kernel32.dll2. Look at exports and imports of kernel32.dll Most of the exports are documented Win32 callsMost of the exports are documented Win32 calls

3. Look at exports and imports of ntdll.dll3. Look at exports and imports of ntdll.dll None of the exports are documentedNone of the exports are documented Some are the same as exports from ntoskrnl.exe, Some are the same as exports from ntoskrnl.exe,

documented in DDK, with identicaldocumented in DDK, with identical

Examining Symbols Examining Symbols In Key ImagesIn Key Images4. Look at exports and imports of ntoskrnl.exe4. Look at exports and imports of ntoskrnl.exe

About 1000 total exported symbolsAbout 1000 total exported symbols About 300 of the exported routine names are About 300 of the exported routine names are

documented in DDK documented in DDK Callable only from kernel modeCallable only from kernel mode

5. Look at all global symbols in ntoskrnl.exe5. Look at all global symbols in ntoskrnl.exe Defined in \support\symbols\xxx\debug\exe\ntoskrnl.dbgDefined in \support\symbols\xxx\debug\exe\ntoskrnl.dbg Quick viewer won’t display - use Kernel Debugger “x *” Quick viewer won’t display - use Kernel Debugger “x *”

with just this .dbg file loadedwith just this .dbg file loaded About 4000 total symbols (Includes executive data cells About 4000 total symbols (Includes executive data cells

in addition to routines)in addition to routines) Exports of ntoskrnl.exe are a subset of this listExports of ntoskrnl.exe are a subset of this list

AgendaAgenda




Process-Based Process-Based Windows NT CodeWindows NT Code Pieces of Windows NT that run in separate Pieces of Windows NT that run in separate

executables (.exe’s), in separate processesexecutables (.exe’s), in separate processes Started by systemStarted by system Not tied to a user logonNot tied to a user logon

Have full process contextHave full process context Three types:Three types:

Environment Subsystems (already described)Environment Subsystems (already described) Win32 ServicesWin32 Services System startup processesSystem startup processes

Note: “system startup processes” is not an Note: “system startup processes” is not an official MS-defined nameofficial MS-defined name

Process Creation HierarchyProcess Creation Hierarchy

tlist.exe (from tlist.exe (from resource kit)resource kit)

tlist /t shows tlist /t shows creation hierarchycreation hierarchy

Creating process Creating process can exit, leaving can exit, leaving created process created process running - hence this running - hence this display does not display does not show all creatorsshow all creators Explorer.exe is Explorer.exe is

actually started by actually started by userinit.exe, which userinit.exe, which then exitsthen exits

Process-Based Process-Based Windows NT CodeWindows NT CodeWin32 servicesWin32 services Win32 .EXEs (applications) that run independently of a Win32 .EXEs (applications) that run independently of a

logged on userlogged on user Start at boot or logon time, survive logoffStart at boot or logon time, survive logoff Defined by CreateService API - view through Control PanelDefined by CreateService API - view through Control Panel See srvany.exe, sc.exe, srvinstw.exe, instsrv.exe in Resource KitSee srvany.exe, sc.exe, srvinstw.exe, instsrv.exe in Resource Kit Typically do not interact with the desktopTypically do not interact with the desktop

Get startup configuration parameters from RegistryGet startup configuration parameters from Registry Log errors to Windows NT Event LogLog errors to Windows NT Event Log

Use some form of IPC mechanism for client communication and controlUse some form of IPC mechanism for client communication and control Services will likely make use of Windows NT security impersonationServices will likely make use of Windows NT security impersonation Remotely manageable (start, stop, user-defined codes)Remotely manageable (start, stop, user-defined codes)

Server Manager allows remote control of servicesServer Manager allows remote control of services Code is the same to control services locally vs. remotelyCode is the same to control services locally vs. remotely

Examples of built-in Windows NT ServicesExamples of built-in Windows NT Services Schedule service (at command), Event Log, Remote Access Server, etc.Schedule service (at command), Event Log, Remote Access Server, etc.

ServiceServiceControllerController

Life Of A ServiceLife Of A Service Install timeInstall time

Setup application tells Service Setup application tells Service Controller about the serviceController about the service

System boot / initializationSystem boot / initialization SCM reads registry, startsSCM reads registry, starts

services as directedservices as directed

Management / maintenanceManagement / maintenance Control panel can start and stop Control panel can start and stop

services and change startup services and change startup parametersparameters

SetupSetupApplicationApplication

CreateServiceCreateService RegistryRegistry

ServiceServiceProcessesProcesses

ControlControlPanelPanel

Where Are Services Defined?Where Are Services Defined? Maintained in Windows NT Registry:Maintained in Windows NT Registry:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\ServicesHKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services One key per installed serviceOne key per installed service

Mandatory information kept on each service:Mandatory information kept on each service: Type of service (Win32, Driver…)Type of service (Win32, Driver…) Imagename of service .EXEImagename of service .EXE

NOTE: Some service .EXEs contain more than one serviceNOTE: Some service .EXEs contain more than one service Start type (automatic, manual, or disabled)Start type (automatic, manual, or disabled)

Optional information:Optional information: Display NameDisplay Name DependenciesDependencies Account and password to run underAccount and password to run under

Can store application-specific configuration parametersCan store application-specific configuration parameters ““Parameters” under service keyParameters” under service key

Process-Based Process-Based Windows NT CodeWindows NT CodeSystem startup processesSystem startup processes Separate processes loaded or started at boot time (not as Separate processes loaded or started at boot time (not as

services or environment subsystems)services or environment subsystems) Names of images are not in registryNames of images are not in registry

““Hardwired” in the source codeHardwired” in the source code Most are Win32 executables, one (smss) is a “native image”Most are Win32 executables, one (smss) is a “native image”

(Idle)(Idle) Process id 0Process id 0Part of the loaded system imagePart of the loaded system imageHome for idle thread(s) (not a real process nor real Home for idle thread(s) (not a real process nor real

threads)threads)Called “System Process” in many displaysCalled “System Process” in many displays

(System)(System) Process id 2Process id 2Part of the loaded system imagePart of the loaded system imageHome for kernel-defined threads (not a real process)Home for kernel-defined threads (not a real process)Thread 0 (routine name Phase1Initialization) Thread 0 (routine name Phase1Initialization)

launches the firstlaunches the first“real” process, running smss.exe…“real” process, running smss.exe……and then becomes the zero page thread…and then becomes the zero page thread

Process-Based Process-Based Windows NT CodeWindows NT CodeSystem startup processesSystem startup processes

smss.exesmss.exeSession ManagerSession ManagerThe first “created” process The first “created” process Takes parameters from \Registry\Machine\System\CurrentControlSet\Takes parameters from \Registry\Machine\System\CurrentControlSet\Control\Session ManagerControl\Session ManagerLaunches required subsystems (csrss) and winlogon Launches required subsystems (csrss) and winlogon

winlogon.exewinlogon.exe Logon processLogon processPresents first login promptPresents first login promptPresents “enter username and password” dialogPresents “enter username and password” dialogLaunches services.exe, lsass.exe, and nddeagnt.exeLaunches services.exe, lsass.exe, and nddeagnt.exeWhen someone logs in, launches userinit.exeWhen someone logs in, launches userinit.exe

services.exeservices.exe Service Controller; also, home for many NT-supplied Service Controller; also, home for many NT-supplied servicesservicesStarts processes for services not part of services.exe (driven by \Registry\Starts processes for services not part of services.exe (driven by \Registry\Machine\System\CurrentControlSet\Services )Machine\System\CurrentControlSet\Services )

lsass.exelsass.exeLocal Security Authentication ServerLocal Security Authentication Server userinit.exeuserinit.exe Started after logon; starts desktop (Explorer.Exe) and Started after logon; starts desktop (Explorer.Exe) and

exitsexits(hence does not show up in tlist output; Explorer appears to be an orphan)(hence does not show up in tlist output; Explorer appears to be an orphan)

explorer.exeexplorer.exe and its children are the creators of all interactive appsand its children are the creators of all interactive apps

AgendaAgenda IntroductionIntroduction ToolsTools System ArchitectureSystem Architecture



Four Contexts For Four Contexts For Executing CodeExecuting Code Full process and thread context:Full process and thread context:

User applicationsUser applications Win32 ServicesWin32 Services Environment subsystem processesEnvironment subsystem processes System startup processesSystem startup processes

Have thread context but no “real” process:Have thread context but no “real” process: Threads in “System” process Threads in “System” process

Routines called by other threads / processes:Routines called by other threads / processes: Subsystem DLLsSubsystem DLLs Executive system services (NtReadFile, etc.)Executive system services (NtReadFile, etc.) GDI routines in Win32K.Sys (and graphics drivers)GDI routines in Win32K.Sys (and graphics drivers)

No process or thread contextNo process or thread context (“Arbitrary thread context”)(“Arbitrary thread context”) Interrupt dispatchingInterrupt dispatching Device driversDevice drivers

Where Is The Code?Where Is The Code? Kernel32.Dll, Gdi32.Dll, User32.DllKernel32.Dll, Gdi32.Dll, User32.Dll

Export Win32 entry pointsExport Win32 entry points NtDll.DllNtDll.Dll

Provides user-mode access to system-space routinesProvides user-mode access to system-space routines Also contains heap manager, image loader, thread startup routineAlso contains heap manager, image loader, thread startup routine

Ntoskrnl.Exe (or Ntkrnlmp.exe)Ntoskrnl.Exe (or Ntkrnlmp.exe) Executive and kernelExecutive and kernel Includes most routines that run as threads in “system” processIncludes most routines that run as threads in “system” process

Win32K.SysWin32K.Sys The loadable module that includes the now-kernel-mode Win32 code The loadable module that includes the now-kernel-mode Win32 code

(formerly in csrss.exe)(formerly in csrss.exe) Hal.DllHal.Dll

Hardware Abstraction LibraryHardware Abstraction Library drivername.Sysdrivername.Sys

Loadable kernel driversLoadable kernel drivers

AgendaAgenda


Per-processPer-processaddress spaceaddress space

Systemwide Systemwide Address SpaceAddress Space

ThreadThread

ThreadThread

ThreadThread

Processes And Processes And ThreadsThreads What is a process?What is a process?

Represents an instance of a running programRepresents an instance of a running program You create a process to run a programYou create a process to run a program Starting an application creates a processStarting an application creates a process

Primary argument to CreateProcess is image Primary argument to CreateProcess is image file name (or command line)file name (or command line)

What is a thread?What is a thread? An execution context within a processAn execution context within a process Primary argument to CreateThread is a Primary argument to CreateThread is a

function entry point addressfunction entry point address All threads in a process share the same per-All threads in a process share the same per-

process address spaceprocess address space Every process starts with one threadEvery process starts with one thread

Running the program’s “main” functionRunning the program’s “main” function Can create other threads in the same processCan create other threads in the same process Can create additional processesCan create additional processes

Tools To Examine ProcessesTools To Examine Processes Task ManagerTask Manager Performance MonitorPerformance Monitor pviewer.exe (pview in Platform SDK): pviewer.exe (pview in Platform SDK):

shows processes, threads within processes, shows processes, threads within processes, memory detailsmemory details

pview.exe (process explode): pview.exe (process explode): thread and process ACLs and tokensthread and process ACLs and tokens

tlist.exe - tlist /t shows parent/child relationshipstlist.exe - tlist /t shows parent/child relationships QuickSliceQuickSlice

qlice.exeqlice.exe CPU usage by process, and by thread within each processCPU usage by process, and by thread within each process

Pulist - process user listPulist - process user list Vadump - dump virtual address space of a processVadump - dump virtual address space of a process

Tools To Examine ProcessesTools To Examine Processes Page fault monitor (pfmon.exe)Page fault monitor (pfmon.exe)

Shows page fault type and origin of subject applicationShows page fault type and origin of subject application Can provide data to working set tuner (part of Platform SDK)Can provide data to working set tuner (part of Platform SDK)

PstatPstat pstat.exe (char mode, no icon)pstat.exe (char mode, no icon) One-time snapshot of systemOne-time snapshot of system Shows state of threads within all processes, with Shows state of threads within all processes, with

wait reasonswait reasons

Kernel debuggerKernel debugger Shows various internal structuresShows various internal structures See Windows NTSee Windows NT®® Workstation Resource Kit documentation Workstation Resource Kit documentation

oh.exe (ResKit), nthandleex oh.exe (ResKit), nthandleex (www.sysinternals.com)(www.sysinternals.com) - show open handles - show open handles

Ntpmon Ntpmon (www.sysinternals.com)(www.sysinternals.com)

Windows NT 5.0 Job ObjectWindows NT 5.0 Job Object

New kernel object to collect a group of New kernel object to collect a group of related processes related processes CreateJobObject/OpenJobObjectCreateJobObject/OpenJobObject

System enforces job quotas System enforces job quotas and security contextand security context Limits: Total and current CPU time, total and active Limits: Total and current CPU time, total and active

processes, per-process and per-job CPU time, min processes, per-process and per-job CPU time, min and max working set, CPU affinity, priority classand max working set, CPU affinity, priority class

Security limits: No administrators token, only Security limits: No administrators token, only restricted token, only specific token, filter token, restricted token, only specific token, filter token, no accessing windows outside the job, no no accessing windows outside the job, no reading/writing the clipboardreading/writing the clipboard

To examine: See new performance counters + To examine: See new performance counters + new !job command in kernel debuggernew !job command in kernel debugger

ProcessProcessobjectobject

Handle tableHandle table

VADVAD VADVAD VADVAD

objectobject

objectobject

Virtual address space descriptorsVirtual address space descriptors

Access tokenAccess token

ThreadThread ThreadThread ThreadThread ……Access tokenAccess token

See kernel debuggerSee kernel debuggercommands:commands:

!processfields!processfields!threadfields!threadfields!process!process!thread!thread!tokenfields!tokenfields!token!token!handle!handle!object!object

Processes And ThreadsProcesses And ThreadsInternal StructuresInternal Structures

Pcb: Pcb: 0x00x0 ExitStatus: ExitStatus: 0x680x68 LockEvent: LockEvent: 0x6c0x6c LockCount: LockCount: 0x7c0x7c CreateTime: CreateTime: 0x800x80 ExitTime: ExitTime: 0x880x88 LockOwner: LockOwner: 0x900x90 UniqueProcessId: UniqueProcessId: 0x940x94 ActiveProcessLinks: ActiveProcessLinks: 0x980x98 QuotaPeakPoolUsage[0]: QuotaPeakPoolUsage[0]: 0xa00xa0 QuotaPoolUsage[0]: QuotaPoolUsage[0]: 0xa80xa8 PagefileUsage: PagefileUsage: 0xb00xb0 CommitCharge: CommitCharge: 0xb40xb4 PeakPagefileUsage: PeakPagefileUsage: 0xb80xb8 PeakVirtualSize: PeakVirtualSize: 0xbc0xbc VirtualSize: VirtualSize: 0xc00xc0 Vm: Vm: 0xc80xc8 LastProtoPteFault: LastProtoPteFault: 0xf80xf8 DebugPort: DebugPort: 0xfc0xfc ExceptionPort: ExceptionPort: 0x1000x100 ObjectTable: ObjectTable: 0x1040x104 Token: Token: 0x1080x108 WorkingSetLock: WorkingSetLock: 0x10c0x10c WorkingSetPage: WorkingSetPage: 0x12c0x12c ProcessOutswapEnabled: ProcessOutswapEnabled: 0x1300x130 ProcessOutswapped: ProcessOutswapped: 0x1310x131 AddressSpaceInitialized: AddressSpaceInitialized: 0x1320x132 AddressSpaceDeleted: AddressSpaceDeleted: 0x1330x133 AddressCreationLock: AddressCreationLock: 0x1340x134

ForkInProgress: ForkInProgress: 0x1580x158 VmOperation: VmOperation: 0x15c0x15c VmOperationEvent: VmOperationEvent: 0x1600x160 PageDirectoryPte: PageDirectoryPte: 0x1640x164 LastFaultCount: LastFaultCount: 0x1680x168 VadRoot: VadRoot: 0x1700x170 VadHint: VadHint: 0x1740x174 CloneRoot: CloneRoot: 0x1780x178 NumberOfPrivatePages: NumberOfPrivatePages: 0x17c0x17c NumberOfLockedPages: NumberOfLockedPages: 0x1800x180 ForkWasSuccessful: ForkWasSuccessful: 0x1840x184 ExitProcessCalled: ExitProcessCalled: 0x1860x186 CreateProcessReported: CreateProcessReported: 0x1870x187 SectionHandle: SectionHandle: 0x1880x188 Peb: Peb: 0x18c0x18c SectionBaseAddress: SectionBaseAddress: 0x1900x190 QuotaBlock: QuotaBlock: 0x1940x194 LastThreadExitStatus: LastThreadExitStatus: 0x1980x198 WorkingSetWatch: WorkingSetWatch: 0x19c0x19c LpcPort: LpcPort: 0x1a00x1a0 InheritedFromUniqueProcessId: 0x1a4InheritedFromUniqueProcessId: 0x1a4 GrantedAccess: GrantedAccess: 0x1a80x1a8 DefaultHardErrorProcessing DefaultHardErrorProcessing 0x1ac0x1ac LdtInformation: LdtInformation: 0x1b00x1b0 VadFreeHint: VadFreeHint: 0x1b40x1b4 VdmObjects: VdmObjects: 0x1b80x1b8 ProcessMutant: ProcessMutant: 0x1bc0x1bc ImageFileName[0]: ImageFileName[0]: 0x1dc0x1dc VmTrimFaultValue: VmTrimFaultValue: 0x1ec0x1ec

!processfields!processfields

Tcb: Tcb: 0x00x0 CreateTime: CreateTime: 0x1b00x1b0 ExitTime: ExitTime: 0x1b80x1b8 ExitStatus: ExitStatus: 0x1c00x1c0 PostBlockList: PostBlockList: 0x1c40x1c4 TerminationPortList: TerminationPortList: 0x1cc0x1cc ActiveTimerListLock: ActiveTimerListLock: 0x1d40x1d4 ActiveTimerListHead: ActiveTimerListHead: 0x1d80x1d8 Cid: Cid: 0x1e00x1e0 LpcReplySemaphore: LpcReplySemaphore: 0x1e80x1e8 LpcReplyMessage: LpcReplyMessage: 0x1fc0x1fc LpcReplyMessageId: LpcReplyMessageId: 0x2000x200 Client: Client: 0x2080x208 IrpList: IrpList: 0x20c0x20c TopLevelIrp: TopLevelIrp: 0x2140x214 ReadClusterSize: ReadClusterSize: 0x21c0x21c ForwardClusterOnly: ForwardClusterOnly: 0x2200x220 DisablePageFaultClustering: DisablePageFaultClustering: 0x2210x221 DeadThread: DeadThread: 0x2220x222 HasTerminated: HasTerminated: 0x2230x223 EventPair: EventPair: 0x2240x224 GrantedAccess: GrantedAccess: 0x2280x228 ThreadsProcess: ThreadsProcess: 0x22c0x22c StartAddress: StartAddress: 0x2300x230 Win32StartAddress: Win32StartAddress: 0x2340x234 LpcExitThreadCalled: LpcExitThreadCalled: 0x2380x238 HardErrorsAreDisabled: HardErrorsAreDisabled: 0x2390x239

!threadfields!threadfields

Looking At Waiting ThreadsLooking At Waiting Threads

pstat.exe (Resource Kit)pstat.exe (Resource Kit) Shows state of every thread in every processShows state of every thread in every process But for threads that are waiting, that’s all But for threads that are waiting, that’s all

we know…we know…

Looking At Waiting ThreadsLooking At Waiting Threads

!thread command in kernel debugger shows !thread command in kernel debugger shows what a thread is waiting onwhat a thread is waiting on

Size TypeState

Wait listhead

Object-type-specific data

DispatcherDispatcherobjectobject

(see \ddk\inc\nttddk.h)(see \ddk\inc\nttddk.h)

Dispatcher ObjectsDispatcher Objects Any kernel object you can wait for is a “dispatcher object”Any kernel object you can wait for is a “dispatcher object”

Some exclusively for synchronizationSome exclusively for synchronization E.g., events, mutexes (“mutants”), semaphores, queues, timersE.g., events, mutexes (“mutants”), semaphores, queues, timers

Others can be waited for as a side effect of their prime function Others can be waited for as a side effect of their prime function E.g., processes, threads, file objectsE.g., processes, threads, file objects

Non-waitable kernel objects are called “control objects”Non-waitable kernel objects are called “control objects” All dispatcher objects have a common headerAll dispatcher objects have a common header All dispatcher objects are in one of two statesAll dispatcher objects are in one of two states

““Signalled” versus “nonsignalled”Signalled” versus “nonsignalled” When signalled, a wait on the object is satisfiedWhen signalled, a wait on the object is satisfied Different object types differ in Different object types differ in

terms of what changes their stateterms of what changes their state Wait and unwait implementation isWait and unwait implementation is

common to all types of dispatcher objectscommon to all types of dispatcher objects

Object-type-Object-type-specific dataspecific data

SizeSize TypeTypeStateState

Wait listheadWait listhead

SizeSize TypeTypeStateState

Wait listheadWait listhead

Object-type-Object-type-specific dataspecific data

DispatcherDispatcherObjectsObjects

Thread ObjectsThread Objects

WaitBlockListWaitBlockListWaitBlockListWaitBlockList

Wait blocksWait blocks

KeyKey TypeTypeNext linkNext link

List entryList entry

ObjectObjectThreadThread







Wait BlocksWait Blocks Represent a thread’s Represent a thread’s

reference to something it’s reference to something it’s waiting for (one per handle waiting for (one per handle passed to WaitFor…)passed to WaitFor…)

All wait blocks from a All wait blocks from a given wait call are chained given wait call are chained to the waiting threadto the waiting thread

Type indicates wait for Type indicates wait for “any” or “all”“any” or “all”

Key denotes argument list Key denotes argument list position for position for WaitForMultipleObjectsWaitForMultipleObjects

AgendaAgenda


Virtual Address Space LayoutVirtual Address Space Layout Process Memory UsageProcess Memory Usage Global System CacheGlobal System Cache System Memory UsageSystem Memory Usage

.EXE code.EXE codeGlobalsGlobals

Per-thread user Per-thread user mode stacksmode stacks

Process heapsProcess heaps.DLL code.DLL code

0000000000000000

7FFFFFFF7FFFFFFF

Exec, Kernel, Exec, Kernel, HAL, drivers, per-HAL, drivers, per-

thread kernel thread kernel mode stacks, mode stacks, Win32K.SysWin32K.Sys

File system cacheFile system cachePaged poolPaged pool

Non-paged poolNon-paged pool

FFFFFFFFFFFFFFFF

8000000080000000

Process page tables,Process page tables,hyperspacehyperspace

C0000000C0000000

4GB Virtual Address Space4GB Virtual Address Space 2 GB per-process2 GB per-process

Address space of one Address space of one process is not directly process is not directly reachable from other reachable from other processesprocesses

2 GB systemwide2 GB systemwide The operating system is The operating system is

loaded here, and appears loaded here, and appears in every process’s in every process’s address spaceaddress space

There is no process for There is no process for “the operating system” “the operating system” (though there are (though there are processes that do things processes that do things for the OS, more or less for the OS, more or less in “background”)in “background”)

Unique per Unique per process, process,

accessible in accessible in user or user or

kernel modekernel mode

System System wide,wide,

accessible accessible only in only in


Per process, Per process, accessible accessible

only in only in kernel modekernel mode

System Space LayoutSystem Space Layout

8000000080000000

System code (NTOSKRNL, HAL, bootSystem code (NTOSKRNL, HAL, bootdrivers); initial nonpaged pooldrivers); initial nonpaged pool

A0000000A0000000 System Mapped Views (e.g. WIN32K.SYS)System Mapped Views (e.g. WIN32K.SYS)or session space (Terminal Server only)or session space (Terminal Server only)

A4000000A4000000 Additional System PTEs (& big cache)Additional System PTEs (& big cache)

C0000000C0000000 Process Page Tables and Page DirectoryProcess Page Tables and Page Directory

C0400000C0400000 Hyperspace and process working set listHyperspace and process working set list

System CacheSystem CacheC1000000C1000000

Paged PoolPaged Pool

EB000000 (min) EB000000 (min)

Non-Paged Pool expansionNon-Paged Pool expansion

FFBE0000FFBE0000

x86x86 Alpha AXPAlpha AXP

C0800000C0800000

System Working Set ListSystem Working Set ListC0C00000C0C00000

Unused No AccessUnused No Access

E1000000E1000000

System PTEsSystem PTEs

Crash dump informationCrash dump information

8000000080000000

System code (NTOSKRNL, HAL,System code (NTOSKRNL, HAL,boot drivers) and initial nonpaged poolboot drivers) and initial nonpaged pool

C0000000C0000000 Process Page Tables and Page DirectoryProcess Page Tables and Page Directory

C1000000C1000000 Hyperspace and process working set listHyperspace and process working set list

System CacheSystem CacheC4000000C4000000

Paged PoolPaged Pool

EB000000 (min) EB000000 (min)

Non-Paged Pool expansionNon-Paged Pool expansion

C2000000C2000000

System Working Set ListSystem Working Set ListC3000000C3000000

Unused No AccessUnused No Access

E1000000E1000000

System PTEsSystem PTEs

FFC00000FFC00000 HAL usageHAL usageFDFEC000FDFEC000 Crash dump information & HAL usageCrash dump information & HAL usage

System Mapped Views (e.g. WIN32K.SYS)System Mapped Views (e.g. WIN32K.SYS)DE000000DE000000

Unique per Unique per processprocess(= per appl.),(= per appl.),user modeuser mode

.EXE code.EXE codeGlobalsGlobals

Per-thread user Per-thread user mode stacksmode stacks

.DLL code.DLL codeProcess heapsProcess heaps

Exec, kernel, Exec, kernel, HAL,HAL,

drivers, etc.drivers, etc.

0000000000000000

BFFFFFFFBFFFFFFF

FFFFFFFFFFFFFFFF

C0000000C0000000

Unique per Unique per process, process,

accessible in accessible in user or user or


3GB Process Space Option3GB Process Space Option Only available on x86 Only available on x86

Server Enterprise EditionServer Enterprise Edition Boot with /3GB option in Boot with /3GB option in

BOOT.INIBOOT.INI Chief “loser” in system Chief “loser” in system

space is file system cachespace is file system cache

Expands per-process Expands per-process address spaceaddress space But image must be But image must be

marked as “large address marked as “large address space aware”space aware”

A stopgap while we wait A stopgap while we wait for 64-bit Windows NT for 64-bit Windows NT (Merced and Alpha; post-(Merced and Alpha; post-Windows NT 5.0)Windows NT 5.0)

System System wide,wide,

accessible accessible only in only in


Per process, Per process, accessible accessible

only in only in kernel modekernel mode

Process page tables,Process page tables,hyperspacehyperspace

2GB user space2GB user space2GB user space2GB user space2GB process space2GB process space

00000000 0000000000000000 00000000

00000000 7FFFFFFF00000000 7FFFFFFF

2GB system space2GB system space

00000007 FFFFFFFF00000007 FFFFFFFF

00000001 0000000000000001 00000000

FFFFFFFF FFFFFFFFFFFFFFFF FFFFFFFF

FFFFFFFF 80000000FFFFFFFF 80000000

Invalid (inaccesible)Invalid (inaccesible)(about 1.8x10^19 (about 1.8x10^19

bytes; not to scale!)bytes; not to scale!)

FFFFFFFF 7FFFFFFFFFFFFFFF 7FFFFFFF

00000008 0000000000000008 00000000

28GB Large 28GB Large Memory AreaMemory Area

64-bit Very Large Memory In 64-bit Very Large Memory In Windows NT 5.0Windows NT 5.0

Alpha Windows NT Server Alpha Windows NT Server Enterprise Edition onlyEnterprise Edition only

Referenced by 64-bit Referenced by 64-bit pointerspointers Cannot be paged out - must Cannot be paged out - must

be resident at all timesbe resident at all times Cannot be used for code, Cannot be used for code,

only data file mappingonly data file mapping New APIs: VirtualAllocVlm, New APIs: VirtualAllocVlm,

MapViewOfFileVlm, MapViewOfFileVlm, Read/WriteFileVlm, Read/WriteFileVlm, Read/WriteProcessMemoryVlRead/WriteProcessMemoryVlm, etc.)m, etc.)

Yet another stopgap prior to Yet another stopgap prior to 64-bit Windows NT64-bit Windows NT

See link/dump/header, or QuickView for .exe’s and .dll’sSee link/dump/header, or QuickView for .exe’s and .dll’s CreateFileMapping, MapViewOfFile simply make the mechanism CreateFileMapping, MapViewOfFile simply make the mechanism

available to application-level codeavailable to application-level code All of these files may simultaneously be mapped by All of these files may simultaneously be mapped by

other processesother processes

0000000000000000

7FFFFFFF7FFFFFFF

.exe.exe

.dll.dllpag

ing

filep

agin

g file

Application Startup Maps Application Startup Maps V.A.S. To Code On DiskV.A.S. To Code On Disk

AgendaAgenda



Process Memory UsageProcess Memory Usage

Working set: All the physical pages Working set: All the physical pages “owned” by a process“owned” by a process Essentially, all the pages the process can Essentially, all the pages the process can

reference without incurring a page faultreference without incurring a page fault Upper limit on size for each processUpper limit on size for each process When limit is reached, a page must be When limit is reached, a page must be

released for every page that’s brought released for every page that’s brought in (“working set replacement”)in (“working set replacement”)

Working set limit: The maximum Working set limit: The maximum pages the process can ownpages the process can own Maximum is calculated as Maximum is calculated as

(available pages - 512 pages)(available pages - 512 pages) Result stored in MmMaximumWorkingSetSizeResult stored in MmMaximumWorkingSetSize

PerfMonPerfMonProcess “WorkingSet”Process “WorkingSet”

newer pagesnewer pages older pagesolder pages

Working Set ListWorking Set ListA FIFO list for each processA FIFO list for each process

PerfMonPerfMonProcess “WorkingSet”Process “WorkingSet”

To standbyTo standbyor modifiedor modified

page listpage list

Working Set ReplacementWorking Set Replacement

When working set “count” = working set size, When working set “count” = working set size, must give up pages to make room for new pagesmust give up pages to make room for new pages

Page replacement is ”modified FIFO”Page replacement is ”modified FIFO” MP x86 and Alpha: no regard to accessed bitMP x86 and Alpha: no regard to accessed bit Windows NT 5.0 on uniprocessor x86 takes into account ageWindows NT 5.0 on uniprocessor x86 takes into account age

Locking PagesLocking Pages Pages may be locked into the process working setPages may be locked into the process working set

Locked pages are guarenteed in physical memory (“resident”) when any thread in process is executingLocked pages are guarenteed in physical memory (“resident”) when any thread in process is executing

Win32:Win32:

status = VirtualLock(baseAddress, size);status = VirtualLock(baseAddress, size);

status = VirtualUnlock(baseAddress, size);status = VirtualUnlock(baseAddress, size);

Number of lockable pages is a fraction of the maximum Number of lockable pages is a fraction of the maximum working set size working set size Changed by SetProcessWorkingSetSizeChanged by SetProcessWorkingSetSize

Pages can be locked into physical memory (by drivers only)Pages can be locked into physical memory (by drivers only) Pages are then immune from outswapping as well as pagingPages are then immune from outswapping as well as paging

MmProbeAndLockPagesMmProbeAndLockPages

Screen snapshot from : Task Manager | Processes tabScreen snapshot from : Task Manager | Processes tab

11

33

22

22

44

11

33

44

Memory Management Memory Management InformationInformationTask manager processes tabTask manager processes tab ““Mem Usage” = physical Mem Usage” = physical

memory used by process memory used by process (working set size, not (working set size, not working set limit)working set limit)

““VM Size” = private (not VM Size” = private (not shared) committed virtual shared) committed virtual space in processesspace in processes

““Mem Usage” in status bar is Mem Usage” in status bar is total of “VM Size” total of “VM Size” column/maximum allowed - column/maximum allowed - i.e., same as “commit i.e., same as “commit charge” in “Performance” tab charge” in “Performance” tab (see next slide) - (see next slide) - notnot same as same as “Mem Usage” column here!“Mem Usage” column here!

““Working Set” = Working Set” = working set size (not limit) working set size (not limit)

““Private Bytes” = same as Private Bytes” = same as “VM Size” from Task “VM Size” from Task Manager Processes listManager Processes list

““Virtual Bytes” = Virtual Bytes” = committed virtual space, committed virtual space, including including shared pagesshared pages

22

11

66

11

22

66

Memory Management Memory Management InformationInformationPerfMon - process objectPerfMon - process object

Screen snapshot from: Performance MonitorScreen snapshot from: Performance Monitorcounters from Process objectcounters from Process object

““Commit charge total” = Commit charge total” = total of private (not total of private (not shared) committed shared) committed virtual space in all virtual space in all processes (i.e. total processes (i.e. total of “VM Size” from of “VM Size” from processes display)processes display)

““Commit charge limit” = Commit charge limit” = sum of available sum of available physical memory + physical memory + free space in free space in paging filepaging file

Memory Management Information Memory Management Information Task manager performance tabTask manager performance tab

Screen snapshot from: Task Manager | Performance Screen snapshot from: Task Manager | Performance tabtab

4433

4433

33

33

44

AgendaAgenda



File System Virtual File System Virtual Block CacheBlock Cache Shared by all file systems (local or remote)Shared by all file systems (local or remote) Caches all filesCaches all files

Including file system metadata filesIncluding file system metadata files

Virtual block cache (not logical block)Virtual block cache (not logical block) Managed in terms of blocks within files, not blocks Managed in terms of blocks within files, not blocks

within partitionwithin partition Uses standard Windows NT virtual memory mechanismsUses standard Windows NT virtual memory mechanisms Coherency maintained between mapped files and Coherency maintained between mapped files and

read/write accessread/write access

Virtual size: 64-512mb (960MB if large cache size set)Virtual size: 64-512mb (960MB if large cache size set) In system virtual address space, so visible to allIn system virtual address space, so visible to all Divided into 256kb “views”Divided into 256kb “views”

ProcessProcessaddressaddressspacespace

SystemSystemaddressaddressspacespace

FileFile

Cached File OperationsCached File Operations Open a file:Open a file:

Find an available viewFind an available view Map the first 256kb of the Map the first 256kb of the

file into the viewfile into the view

Read from or write to a Read from or write to a cached file:cached file: Remap as necessary to Remap as necessary to

map referenced section of map referenced section of file into the cachefile into the cache

Copy data between Copy data between application buffer and application buffer and cache’s virtual address cache’s virtual address spacespace

Actual I/O is due to pagingActual I/O is due to paging

Fast I/OFast I/O

I/O Subsystem API (Ntxxx)I/O Subsystem API (Ntxxx)

DriverDriverSupportSupportRoutinesRoutines(Io, Ex,(Io, Ex,Ke, Mm,Ke, Mm,

Hal, FsRtl,Hal, FsRtl,...)...)

I/O Manager (Ioxxx)I/O Manager (Ioxxx)

HAL I/O access routinesHAL I/O access routines

I/O ports and registersI/O ports and registers

File System driversFile System drivers(e.g. NTFS)(e.g. NTFS)

Disk device driverDisk device driver

CacheCacheManagerManager

Fast I/OFast I/Opathpath

Fast I/O pathFast I/O path Allows executive Allows executive

I/O APIs to access I/O APIs to access cache directlycache directly

Bypasses file Bypasses file system driversystem driver

Bypasses IRP Bypasses IRP generation, probe-generation, probe-and-lock of user and-lock of user buffer, etc.buffer, etc.

Cache SizeCache Size Physical size: Depends on available memoryPhysical size: Depends on available memory

Competes for physical memory with processes, paged Competes for physical memory with processes, paged pool, pageable system codepool, pageable system code

Part of “system working set”Part of “system working set” Automatically expanded / shrunk by systemAutomatically expanded / shrunk by system Normal working set adjustment mechanismsNormal working set adjustment mechanisms

Relies on Memory Manager for global memory Relies on Memory Manager for global memory policypolicy

Performance Monitor: Memory object | System cache Performance Monitor: Memory object | System cache resident bytes shows current physical space resident bytes shows current physical space occupied occupied by cacheby cache

See \SYSTEM\CurrentControlSet\Control\Session See \SYSTEM\CurrentControlSet\Control\Session Manager\ Memory Management\LargeSystemCacheManager\ Memory Management\LargeSystemCache Default is 0 for both Workstation and ServerDefault is 0 for both Workstation and Server 1 = favor system working set vs. process working set1 = favor system working set vs. process working set

also allows cache to be >512MB virtual sizealso allows cache to be >512MB virtual size Can modify with Control Panel->Network->Services->Can modify with Control Panel->Network->Services->

Server propertiesServer properties

Cache Functions Cache Functions And ControlAnd Control Automatic asynchronous readaheadAutomatic asynchronous readahead

Done by separate “Readahead” system threadDone by separate “Readahead” system thread 64kb readaheads by default64kb readaheads by default Predicts next read location based on history of last 3 readsPredicts next read location based on history of last 3 reads Readahead hints can be provided to CreateFile:Readahead hints can be provided to CreateFile:

FILE_FLAG_SEQUENTIAL does 192kb read aheadFILE_FLAG_SEQUENTIAL does 192kb read ahead FILE_FLAG_RANDOM_ACCESS disables read aheadFILE_FLAG_RANDOM_ACCESS disables read ahead

Write-back, not write-throughWrite-back, not write-through Dirty page threshold forces writingDirty page threshold forces writing

Small system: Physical Pages / 8; medium system: Small system: Physical Pages / 8; medium system: Physical Pages / 4Physical Pages / 4

Large system: add above 2 togetherLarge system: add above 2 together ““Lazy writer” thread queues 1/4 of dirty pages every second to Lazy writer” thread queues 1/4 of dirty pages every second to

separate “Write Behind” system thread (note, does not flush separate “Write Behind” system thread (note, does not flush mapped files)mapped files)

Can override via CreateFile with FILE_FLAG_WRITE_THROUGHCan override via CreateFile with FILE_FLAG_WRITE_THROUGH Or explicitly call FlushFileBuffers when you care (does flush Or explicitly call FlushFileBuffers when you care (does flush

mapped files)mapped files)

Cache Functions Cache Functions And ControlAnd Control Can disable cache completely on a Can disable cache completely on a

per-file basisper-file basis CreateFile with CreateFile with

FILE_FLAG_NO_BUFFERINGFILE_FLAG_NO_BUFFERING Requires reads/writes to be done on Requires reads/writes to be done on

sector boundariessector boundaries Buffers must be aligned in memory Buffers must be aligned in memory

on sector boundarieson sector boundaries

AgendaAgenda



System Paged MemorySystem Paged Memory Just as processes have working sets, Windows NT’s pageable Just as processes have working sets, Windows NT’s pageable

system-space code and data lives in the “system working set”system-space code and data lives in the “system working set” Cache is one of 4 components of “system working set”Cache is one of 4 components of “system working set”

Pageable components of system working set:Pageable components of system working set: Paged poolPaged pool Pageable code and data in the execPageable code and data in the exec Pageable code and data in kernel-mode drivers, Win32K.Sys, Pageable code and data in kernel-mode drivers, Win32K.Sys,

graphics drivers, etc.graphics drivers, etc. Global file system data cacheGlobal file system data cache

To get physical (resident) size of these with PerfMon, look at:To get physical (resident) size of these with PerfMon, look at: Memory | Pool Paged Resident BytesMemory | Pool Paged Resident Bytes Memory | System Code Resident BytesMemory | System Code Resident Bytes Memory | System Driver Resident BytesMemory | System Driver Resident Bytes Memory | System Cache Resident BytesMemory | System Cache Resident Bytes Memory | Cache bytes counter is total of these four “resident” Memory | Cache bytes counter is total of these four “resident”

(physical) counters (not just the cache; same as “File Cache” on (physical) counters (not just the cache; same as “File Cache” on Task Manager / Performance tabTask Manager / Performance tab

8000000080000000

System code (NTOSKRNL, HAL, bootSystem code (NTOSKRNL, HAL, bootdrivers); initial nonpaged pooldrivers); initial nonpaged pool

A0000000A0000000 Win32k.sys *8MB)Win32k.sys *8MB)

A0800000A0800000 Session Working Set ListsSession Working Set Lists

x86x86

Mapped Views for SessionMapped Views for Session

Paged Pool for SessionPaged Pool for Session

A0C00000A0C00000

A2000000A2000000

SessionsSessions New memory management object to support New memory management object to support

Windows NTWindows NT®® Server 5.0 Server 5.0 All processes in an interactive session share a:All processes in an interactive session share a:

Session-specific copy of Win32K.SysSession-specific copy of Win32K.Sys Instance of WinlogonInstance of Winlogon Session working setSession working set

Nonpageable components:Nonpageable components: Nonpageable parts of Nonpageable parts of

NtosKrnl.Exe, driversNtosKrnl.Exe, drivers Nonpaged pool (see Nonpaged pool (see

PerfMon, Memory object: PerfMon, Memory object: Pool nonpaged bytes) Pool nonpaged bytes)

To get size of nonpageable To get size of nonpageable system code, run \ntreskit\system code, run \ntreskit\pstat.exe & add columns 1 & 2pstat.exe & add columns 1 & 2non-paged codenon-paged codenon-paged datanon-paged datapageable code+datapageable code+data output of “drivers” (\ntreskit\output of “drivers” (\ntreskit\

drivers.exe) is similardrivers.exe) is similar Win32K.Sys is paged, even Win32K.Sys is paged, even

though it shows up as though it shows up as nonpagednonpaged

System Nonpaged MemorySystem Nonpaged Memory

7

98

7 98

Monitoring Pool UsageMonitoring Pool Usage

Poolmon.exe in \support\debugPoolmon.exe in \support\debug Must first turn on pool tagging with gflagsMust first turn on pool tagging with gflags

““p” to toggle between nonpaged, paged pool, or bothp” to toggle between nonpaged, paged pool, or both

Sorting:Sorting:

““b” to sort by total # of bytesb” to sort by total # of bytes

““a” to sort by # of allocationsa” to sort by # of allocations

““t” to sort by structure tagt” to sort by structure tag

““Free” MemoryFree” Memory

System keeps unassigned physical System keeps unassigned physical pages (those not part of any working pages (those not part of any working set) on five listsset) on five lists Free page listFree page list Modified page listModified page list Standby page listStandby page list Zero page listZero page list Bad page list - pages that failed Bad page list - pages that failed

memory test at system startupmemory test at system startup

Managing Physical PagesManaging Physical Pages

StandbyStandbyPagePageListList

ZeroZeroPagePageListList

FreeFreePagePageListList

ProcessProcessWorkingWorking

SetsSets

pages read pages read from diskfrom disk

demand zero demand zero page faultspage faults

working set working set replacementreplacement

ModifiedModifiedPagePageListList

modifiedmodifiedpagepagewriterwriter

zerozeropagepage

threadthread

““soft”soft”pagepagefaultsfaults

BadBadPagePageListList

““Available” memory = total of Available” memory = total of free, zero, and standby lists free, zero, and standby lists (majority usually are (majority usually are standby pages)standby pages)

““File cache” is really total File cache” is really total physicalphysical size of pageable size of pageable portions of: paged pool, portions of: paged pool, NtosKrnl.Exe code and data, NtosKrnl.Exe code and data, drivers code and data, and drivers code and data, and file system cache (same as file system cache (same as PerfMon “cache PerfMon “cache bytes” counter)bytes” counter)

““Kernel Memory Paged” is Kernel Memory Paged” is resident size of paged poolresident size of paged pool

““Kernel Memory Nonpaged” Kernel Memory Nonpaged” is actual size of is actual size of nonpaged poolnonpaged pool

Screen snapshot from: Task Manager | Performance Screen snapshot from: Task Manager | Performance tabtab

1

2

3

3

21

44

Memory Management InformationMemory Management InformationTask manager performance tabTask manager performance tab

Process working sets Process working sets Perfmon: Process / Working setPerfmon: Process / Working set Note, shared resident pages are Note, shared resident pages are

counted the process working set of counted the process working set of every process that’s faulted them inevery process that’s faulted them in

Hence, the total of all of these may be Hence, the total of all of these may be greater than physical memorygreater than physical memory

Nonpageable system code Nonpageable system code (NTOSKRNL + drivers, including (NTOSKRNL + drivers, including win32k.sys &graphics drivers)win32k.sys &graphics drivers) See total displayed by DRIVERS utility See total displayed by DRIVERS utility

in Windows NT Resource Kitin Windows NT Resource Kit Nonpageable poolNonpageable pool

Perfmon: Memory / Pool Perfmon: Memory / Pool nonpaged bytesnonpaged bytes

Free, zero, and standby page listsFree, zero, and standby page lists Perfmon: Memory / Available bytesPerfmon: Memory / Available bytes Or: Task Manager / Performance tab: Or: Task Manager / Performance tab:

Physical memory: AvailablePhysical memory: Available

Pageable, but currently-resident, Pageable, but currently-resident, system-space memorysystem-space memory Perfmon: Memory / Pool paged Perfmon: Memory / Pool paged

resident bytesresident bytes Perfmon: Memory / System Perfmon: Memory / System

cache resident bytescache resident bytes Perfmon: Memory / System code Perfmon: Memory / System code

resident bytesresident bytes Perfmon: Memory / System Perfmon: Memory / System

driver resident bytesdriver resident bytes Memory | Cache bytes counter is Memory | Cache bytes counter is

really total of these four really total of these four “resident” (physical) counters“resident” (physical) counters

Modified, Bad page listsModified, Bad page lists can only see size with !can only see size with !

memusage command in Kernel memusage command in Kernel DebuggerDebugger

Summary: Accounting For Summary: Accounting For Physical Memory UsagePhysical Memory Usage

Windows NT Internals Windows NT Internals Information SourcesInformation Sources BooksBooks

Inside Windows NT (Solomon, MS Press)Inside Windows NT (Solomon, MS Press) Advanced Windows (Richter, MS Press)Advanced Windows (Richter, MS Press) Windows NT Workstation Resource Guide (MS Press)Windows NT Workstation Resource Guide (MS Press)

MSDN LibraryMSDN Library Platform SDK API documentationPlatform SDK API documentation Windows NT Device Driver Kit (DDK) documentationWindows NT Device Driver Kit (DDK) documentation Win32 Knowledge Base - has some Windows NT internals articlesWin32 Knowledge Base - has some Windows NT internals articles

Past Past Windows NT conferences audio/video tapes (www.mobiletape.com)Windows NT conferences audio/video tapes (www.mobiletape.com) www.sysinternals.com - Windows NT internals articles and toolswww.sysinternals.com - Windows NT internals articles and tools www.microsoft.com/hwdev - hardware developers and driver writerswww.microsoft.com/hwdev - hardware developers and driver writers www.microsoft.com/hwdev/ntifskit - Installable File System Developers Kitwww.microsoft.com/hwdev/ntifskit - Installable File System Developers Kit comp.os.ms-windows.programmer.nt.kernel-mode - drivers newsgroupcomp.os.ms-windows.programmer.nt.kernel-mode - drivers newsgroup www.cmkrnl.com - Windows NT device driver FAQwww.cmkrnl.com - Windows NT device driver FAQ

Windows NT ® Internals David Solomon David Solomon Expert Seminars Microsoft Corporation.

Documents

Transcript of Windows NT ® Internals David Solomon David Solomon Expert Seminars Microsoft Corporation.