How To Write A Device How To Write A Device Driver That Doesn’t Break Driver That Doesn’t Break On Partitioned Systems On Partitioned Systems

Mike TrickerMike TrickerProgram ManagerProgram ManagerWindows KernelWindows KernelMicrosoft CorporationMicrosoft Corporation

Bruce SherwinBruce SherwinSoftware Design EngineerSoftware Design EngineerWindows KernelWindows KernelMicrosoft CorporationMicrosoft Corporation

AgendaAgenda

IntroductionIntroduction

MotivationMotivation

Dynamic Partitioning (DP) on Windows Dynamic Partitioning (DP) on Windows Server codename “Longhorn”Server codename “Longhorn”

Kernel driver remindersKernel driver reminders

Driver considerations on Driver considerations on partitionable systems partitionable systems

Call to actionCall to action

Dynamic PartitioningDynamic Partitioning

Memory

Memory Memory

Memory

IO Bridge

Service Processo

r

1.1. Partition Manager provides the UI Partition Manager provides the UI for partition creation and managementfor partition creation and management

2. Service Processor controls the 2. Service Processor controls the inter processor and IO connectionsinter processor and IO connections

Partition ManagerPartition Manager

3. Platforms partitionable to the socket level; 3. Platforms partitionable to the socket level; virtualization used for sub socket partitioningvirtualization used for sub socket partitioning

4. Support for dynamic partitioning and 4. Support for dynamic partitioning and socket replacementsocket replacement

PCI ExpressPCI Express

Core Core

Cache

… Core Core

Cache

…

Core Core

Cache

…Core Core

Cache

…

. . .

IO Bridge

. . .

IO Bridge

. . .

IO Bridge

. . .

Windows Server Windows Server Longhorn Longhorn dynamic dynamic

hardware partitioninghardware partitioning features are focused features are focused on improving server on improving server

RASRAS

Example Example HardwareHardware Partitionable Server Partitionable Server

Motivation For DPMotivation For DP

This is being driven in part by the availability of This is being driven in part by the availability of multi-core processorsmulti-core processors

In a year or two a 4-processor system will have the performance In a year or two a 4-processor system will have the performance of a 32-processor system todayof a 32-processor system today

And will require Enterprise-class RAS featuresAnd will require Enterprise-class RAS features

In Windows Server Longhorn DP is focused primarilyIn Windows Server Longhorn DP is focused primarilyon Reliability, Availability and Serviceability (RAS)on Reliability, Availability and Serviceability (RAS)

Minimizing unplanned downtime due to failing hardwareMinimizing unplanned downtime due to failing hardwareReplacing hardware that is showing signs of impending failureReplacing hardware that is showing signs of impending failure

Capacity on demandCapacity on demandAdding system resources as neededAdding system resources as needed

What We Are DoingWhat We Are Doing

Windows Server Longhorn 64-bit will add supportWindows Server Longhorn 64-bit will add supportfor Hot Add of processorsfor Hot Add of processors

Together with PCI Express native hot plug of devicesTogether with PCI Express native hot plug of devices

And continues to support Hot Add memoryAnd continues to support Hot Add memory

It will also support Hot Replace of memory It will also support Hot Replace of memory and processorsand processors

Transparent to applicationsTransparent to applications

Drivers receive power IRPsDrivers receive power IRPs

Windows Server Longhorn is planned to supportWindows Server Longhorn is planned to supportHot Add of I/O host bridgesHot Add of I/O host bridges

Supporting devices using line-based interruptsSupporting devices using line-based interrupts

DP will not be available on all Server SKUsDP will not be available on all Server SKUs

Kernel Driver RemindersKernel Driver Reminders

All 64-bit x64 kernel drivers must be signed or All 64-bit x64 kernel drivers must be signed or they will not load on Windows Server Longhorn they will not load on Windows Server Longhorn or Windows Vistaor Windows Vista

Drivers that manipulate critical kernel data Drivers that manipulate critical kernel data structures will cause the system to bugcheck, structures will cause the system to bugcheck, for examplefor example

System service tablesSystem service tables

IDTIDT

GDTGDT

Kernel stacks not allocated by the kernelKernel stacks not allocated by the kernel

Patching any part of the kernelPatching any part of the kernel

Taking Advantage Of DP Taking Advantage Of DP In Your Device DriverIn Your Device Driver

DP-Aware DriversDP-Aware DriversReceiving NotificationsReceiving Notifications

Use Use IoRegisterPlugPlayNotification()IoRegisterPlugPlayNotification()EventCategory isEventCategory is EventCategoryDeviceInterfaceChangeEventCategoryDeviceInterfaceChange

The GUID used in the The GUID used in the DEVICE_INTERFACE_CHANGE_NOTIFICATIONDEVICE_INTERFACE_CHANGE_NOTIFICATION structure will bestructure will be

For Hot Add memory:For Hot Add memory: GUID_DEVICE_MEMORY GUID_DEVICE_MEMORYFor Hot Add processor:For Hot Add processor: GUID_DEVICE_PROCESSOR GUID_DEVICE_PROCESSOR

Include Include WDM.HWDM.H and and POCLASS.HPOCLASS.H

DP-Aware DriversDP-Aware DriversMemory ChangesMemory Changes

Note:Note: Hot Add memory does not affect Hot Add memory does not affect either paged or non-paged pool sizes in either paged or non-paged pool sizes in Windows Server LonghornWindows Server Longhorn

Drivers may attempt to allocate more Drivers may attempt to allocate more physical memory after receiving the Hot physical memory after receiving the Hot Add memory notificationAdd memory notification

Since memory usage is fundamentally Since memory usage is fundamentally competitive they should handle the case competitive they should handle the case of an allocation failure as they (should) of an allocation failure as they (should) do alreadydo already

DP-Aware DriversDP-Aware DriversProcessor ChangesProcessor Changes

Drivers that care about the number of active processors Drivers that care about the number of active processors should callshould call

KeQueryActiveProcessorCountKeQueryActiveProcessorCountto update their countto update their count

Do not use Do not use KeNumberProcessorsKeNumberProcessors – it’s undocumented and is – it’s undocumented and is not staticnot static

If they receive a Hot Add processor notification If they receive a Hot Add processor notification they must call this DDI again to see what they must call this DDI again to see what changedchanged

Which gives them both the updated count and the Which gives them both the updated count and the affinity maskaffinity maskSo they can update their internal countSo they can update their internal count

Affinity Mask ManipulationAffinity Mask Manipulation

Avoid affinity manipulation “by hand”; use Avoid affinity manipulation “by hand”; use appropriate RTL APIs for affinity manipulation appropriate RTL APIs for affinity manipulation when absolutely requiredwhen absolutely required

As mentioned on the previous slideAs mentioned on the previous slide KeQueryActiveProcessorCountKeQueryActiveProcessorCount

should do exactly what you needshould do exactly what you needAs well as returning the count of active processors it As well as returning the count of active processors it also returns the affinity mask:also returns the affinity mask:Count = KeQueryActiveProcessorCount(Count = KeQueryActiveProcessorCount(

&AffinityMask);&AffinityMask);

This will provide you with a current mask that This will provide you with a current mask that you can use as you do todayyou can use as you do today

Processor ScalingProcessor Scaling

If a driver uses per-processor worker threads If a driver uses per-processor worker threads Create new ones per newly added processorCreate new ones per newly added processor

Load balancing algorithms should be Hot Add Load balancing algorithms should be Hot Add processor-aware to ensure scalabilityprocessor-aware to ensure scalability

As processors are added workloads may need to As processors are added workloads may need to rebalance between threadsrebalance between threads

Miniport developers should ask the class driver Miniport developers should ask the class driver owners about possible behavior changes on owners about possible behavior changes on Hot AddHot Add

How the miniport could benefit from the How the miniport could benefit from the additional resourcesadditional resources

Per-Processor DataPer-Processor Data

Using arrays (or “slotted” data structures) with one entry Using arrays (or “slotted” data structures) with one entry per processor works wellper processor works well

Either allocate enough memory to handle the maximum number Either allocate enough memory to handle the maximum number of processors possible for that architecture when creating the of processors possible for that architecture when creating the data structuredata structure

And accept the overhead when the additional processors And accept the overhead when the additional processors are not presentare not present

Or use a data structure that can be grown dynamically as Or use a data structure that can be grown dynamically as processors are addedprocessors are added

So long as your driver gets notified when processors are Hot AddedSo long as your driver gets notified when processors are Hot Added

Provide wrapper functions to hide the complexity of Provide wrapper functions to hide the complexity of accessing the entries using the processor number as accessing the entries using the processor number as the indexthe index

E.g., GetData(Processor) and SetData(Processor)E.g., GetData(Processor) and SetData(Processor)

Using Per-Processor DataUsing Per-Processor Data// Register for PnP notificationstatus = IoRegisterPlugPlayNotification(                EventCategoryDeviceInterfaceChange,                0,                &GUID_DEVICE_PROCESSOR,                DriverObject,                CpuDeviceArrivalNotificationCallback,                NULL,                (PVOID) PerProcessorData                );

NTSTATUSCpuDeviceArrivalNotificationCallback(    IN PVOID NotificationStructure,    IN PVOID Context    ){    MY_DATA_STRUCTURE *PerProcessorData = (MY_DATA_STRUCTURE *)Context;    // Get current processor count and affinity    newNumberOfProcessors = KeQueryActiveProcessorCount(&newActiveProcessorMask)

Using Per-Processor DataUsing Per-Processor Data if (newNumberOfProcessors > oldNumberOfProcessors) {        // Expand existing per-processor data structures to be aware of new //processors        InitializeMyDataStructureWithNewProcessorCount(

PerProcessorData, newNumberOfProcessors);

        // Create a thread that runs with the new processor affinity        PsCreateSystemThread(&thread,                             THREAD_ALL_ACCESS,                             NULL,                             NULL,                             NULL,                             (PKSTART_ROUTINE) MyWorkerThread,                             (PVOID) &newActiveProcessorMask);        ...    }}

VOIDMyWorkerThread(IN OUT PVOID Context){    PKAFFINITY affinityMask = (PKAFFINITY)Context;    // Set thread affinity    KeSetSystemAffinityThread(*affinityMask);    ...}

Interrupt TargetingInterrupt TargetingTo a New ProcessorTo a New Processor

System triggers a tree-wide rebalanceSystem triggers a tree-wide rebalance

Allows interrupt-consuming device Allows interrupt-consuming device drivers to connect interrupts to newly drivers to connect interrupts to newly added processorsadded processors

DPCs queued from an ISR will run on the DPCs queued from an ISR will run on the same processor by defaultsame processor by default

Please don’t fail Please don’t fail QUERY_STOPQUERY_STOP or you or you will prevent the system from distributing will prevent the system from distributing interrupts to new processorsinterrupts to new processors

DP-Aware DriversDP-Aware DriversResource RebalanceResource Rebalance

Started State

Stop-Pending State

Stopped State

IRP_MN_STOP_DEVICE (2)IRP_MN_STOP_DEVICE (2)

IRP_MN_QUERY_STOP_DEVICE (1)IRP_MN_QUERY_STOP_DEVICE (1)

IRP_MN_CANCEL_STOP_DEVICE (4)IRP_MN_CANCEL_STOP_DEVICE (4)

IRP_MN_START_DEVICE (3)IRP_MN_START_DEVICE (3)

Failed Restart (5)Failed Restart (5)

Connecting InterruptsConnecting Interrupts

Windows will include new processor inWindows will include new processor inthe affinity for interrupt resource assigned the affinity for interrupt resource assigned during rebalanceduring rebalance

New resources passed in New resources passed in IRP_MN_START_DEVICEIRP_MN_START_DEVICE

Driver should call Driver should call IoConnectInterruptEx()IoConnectInterruptEx() using this using this new affinitynew affinity

This will connect the device interrupt This will connect the device interrupt to new processorto new processor

Application ImplicationsApplication Implications

Running processes will not change affinity by default Running processes will not change affinity by default for compatibilityfor compatibility

They can receive notifications and change their affinity if desiredThey can receive notifications and change their affinity if desired

RegisterDeviceNotification()RegisterDeviceNotification()GUID_DEVICE_MEMORYGUID_DEVICE_MEMORY

GUID_DEVICE_PROCESSORGUID_DEVICE_PROCESSOR

SetProcessAffinityMask()SetProcessAffinityMask()

SetThreadAffinityMask()SetThreadAffinityMask()

GlobalMemoryStatusEx()GlobalMemoryStatusEx()

New processes will take advantage of new processor(s)New processes will take advantage of new processor(s)

System process affinity will be changed to include the System process affinity will be changed to include the newly added processorsnewly added processors

Worker threads run in system processWorker threads run in system process

User Mode IOCTLs User Mode IOCTLs

Applications may issue custom IOCTLs to a partner Applications may issue custom IOCTLs to a partner device driverdevice driver

~40% of “tier 1” applications we test today install drivers to~40% of “tier 1” applications we test today install drivers tohandle specific taskshandle specific tasks

In this case both need to register for Hot Add processor In this case both need to register for Hot Add processor notifications otherwise they may get out of syncnotifications otherwise they may get out of sync

For example the application may issue IOCTLs on For example the application may issue IOCTLs on a thread running on a processor the driver doesn’t a thread running on a processor the driver doesn’t know aboutknow about

If the driver uses per-processor data, then it needs to registerIf the driver uses per-processor data, then it needs to registerfor hot add processor notification and update its data structuresfor hot add processor notification and update its data structures

Hot Replace FlowHot Replace Flow

System state is migrated in stagesSystem state is migrated in stages1.1. Paged memory is copiedPaged memory is copied

2.2. System is quiescedSystem is quiescedStop DMAStop DMA

Stop interruptsStop interrupts

Devices moved to D3Devices moved to D3

Quiesce duration ~1 secondQuiesce duration ~1 second

3.3. Non-paged memory is copiedNon-paged memory is copiedProcessor state is migratedProcessor state is migrated

4.4. System is resumedSystem is resumedDMA and interrupts resumedDMA and interrupts resumed

Drivers return to D0Drivers return to D0

Hot Replace ImplicationsHot Replace ImplicationsFor DriversFor Drivers

Hot Replace uses a pseudo-S4 state Hot Replace uses a pseudo-S4 state where we temporarily “hibernate” the where we temporarily “hibernate” the system to quiesce devices (stop DMA system to quiesce devices (stop DMA operations and stop interrupts)operations and stop interrupts)

But without actually using a hiberfileBut without actually using a hiberfile

Your driver therefore needs to handle Your driver therefore needs to handle power S IRP and D IRP requestspower S IRP and D IRP requests

IRP_MN_SET_POWER, S4IRP_MN_SET_POWER, S4IRP_MN_SET_POWER, D3IRP_MN_SET_POWER, D3Note:Note: This includes IA-64 This includes IA-64

System ImplicationsSystem ImplicationsFor Hot ReplaceFor Hot Replace

Applications do not see a system change Applications do not see a system change when a Hot Replace operation occurswhen a Hot Replace operation occurs

Amount of physical memory is unchangedAmount of physical memory is unchanged

Number of logical processor is unchangedNumber of logical processor is unchanged

This is deliberate to avoid application This is deliberate to avoid application compatibility issuescompatibility issues

Hot Replace is treated as an Hot Replace is treated as an atomic operationatomic operation

Hot Add ProcessorHot Add Processor

Bruce SherwinBruce SherwinSoftware Design EngineerSoftware Design EngineerWindows KernelWindows Kernel

Related IssuesRelated Issues

NUMA BehaviorNUMA Behavior

NUMA is becoming widespread and NUMA is becoming widespread and should be transparent in most casesshould be transparent in most cases

If the platform supports the _PXM ACPI If the platform supports the _PXM ACPI method you will automatically use NUMA method you will automatically use NUMA local memory at local memory at AddDeviceAddDevice and and StartDeviceStartDevice time time

This ensures your device extensions and This ensures your device extensions and common buffers are allocated out of NUMA common buffers are allocated out of NUMA local memorylocal memory

TestingTesting

Today's DP-capable systems are typically large, Today's DP-capable systems are typically large, restricting access for driver developersrestricting access for driver developers

We expect that to change in the next few yearsWe expect that to change in the next few years

In the meantime if your device needs to work In the meantime if your device needs to work with DP we expect you to work with the OEM on with DP we expect you to work with the OEM on whose system you’ll ship to ensure a great whose system you’ll ship to ensure a great customer experiencecustomer experience

Also work with Microsoft, since we’re also testing Also work with Microsoft, since we’re also testing these systemsthese systems

If you’re already in-box and can run on these If you’re already in-box and can run on these systems you should be coveredsystems you should be covered

Future TechnologiesFuture Technologies

Indicating DP SupportIndicating DP SupportDetecting If DP Is AvailableDetecting If DP Is Available

We’ve been asked how to determine if a system We’ve been asked how to determine if a system actually supports DPactually supports DP

Since OEMs may ship very similar systems with DP Since OEMs may ship very similar systems with DP as a high profile RAS featureas a high profile RAS feature

And thus not enabled on lower cost systemsAnd thus not enabled on lower cost systems

Firmware mechanism to report Firmware mechanism to report platform capabilitiesplatform capabilities

Kernel and user mode API to query Kernel and user mode API to query this capabilitythis capability

Exact mechanism TBDExact mechanism TBD

Supporting Hot RemoveSupporting Hot Remove

Windows Server Longhorn will not support Hot Windows Server Longhorn will not support Hot Remove of memory or processorsRemove of memory or processors

When we do we’re concerned about drivers When we do we’re concerned about drivers and applicationsand applications

Sparse affinity mask and drivers parsing the mask Sparse affinity mask and drivers parsing the mask until finding a missing entry until finding a missing entry

Memory that’s been pinned for DMAMemory that’s been pinned for DMA

Drivers using Drivers using KeNumberProcessorsKeNumberProcessors

We’ll have to be very careful about what drivers We’ll have to be very careful about what drivers we load on such a platformwe load on such a platform

And consider what may need to be blockedAnd consider what may need to be blocked

Call To ActionCall To Action

If you want your 64-bit x64 drivers to load on If you want your 64-bit x64 drivers to load on Windows Server Longhorn (or Windows Vista) Windows Server Longhorn (or Windows Vista) ensure they’re signedensure they’re signed

Ensure that your 64-bit x64 drivers avoid manipulating Ensure that your 64-bit x64 drivers avoid manipulating undocumented private kernel data structures as they undocumented private kernel data structures as they will bugcheck the systemwill bugcheck the system

Please follow the guidelines for good behavior outlined Please follow the guidelines for good behavior outlined in this presentationin this presentation

Realize that DP won’t be “just a high end feature” for Realize that DP won’t be “just a high end feature” for much longermuch longer

That will change during the life of Windows Server LonghornThat will change during the life of Windows Server Longhorn

Ensure that any driver targeting future Windows Server Ensure that any driver targeting future Windows Server releases is DP (and NUMA) awarereleases is DP (and NUMA) aware

Additional ResourcesAdditional Resources

Web resourcesWeb resourcesDynamic Partitioning home pageDynamic Partitioning home page

http://www.microsoft.com/whdc/system/platform/server/dhp.mspxhttp://www.microsoft.com/whdc/system/platform/server/dhp.mspx

White papersWhite papers

http://www.microsoft.com/whdc/system/vista/kernel-en.mspxhttp://www.microsoft.com/whdc/system/vista/kernel-en.mspx

WinHEC 2005 sessionsWinHEC 2005 sessions

http://download.microsoft.com/download/9/8/f/98f3fe47-dfc3-4e74-92a3-0887http://download.microsoft.com/download/9/8/f/98f3fe47-dfc3-4e74-92a3-088782200fe7/TWAR05016_WinHEC05.ppt82200fe7/TWAR05016_WinHEC05.ppt

Kernel patching FAQKernel patching FAQ

http://www.microsoft.com/whdc/driver/kernel/64bitpatch_FAQ.mspxhttp://www.microsoft.com/whdc/driver/kernel/64bitpatch_FAQ.mspx   Related sessionsRelated sessions

Kernel Enhancements for Windows Server Longhorn Kernel Enhancements for Windows Server Longhorn

How to Use the WDK to Develop, Sign and Test Drivers How to Use the WDK to Develop, Sign and Test Drivers

For feedback on all things pertaining to Dynamic Partitioning please For feedback on all things pertaining to Dynamic Partitioning please send mail to:send mail to:

dpfb @ microsoft.comdpfb @ microsoft.com

© 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions,

it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.