Best Practices For Best Practices For Testing Windows DriversTesting Windows Drivers

Dieter AchtelstetterDieter AchtelstetterSoftware Design EngineerSoftware Design EngineerWindows Device Experience GroupWindows Device Experience GroupMicrosoft CorporationMicrosoft Corporation

About This PresentationAbout This Presentation

Fulfill a request from OEM and IHVFulfill a request from OEM and IHVFor best practices on testingFor best practices on testing

Suggestion for Suggestion for Ways to reduce support costs for you and your customers Ways to reduce support costs for you and your customers

Increasing your customer satisfactionIncreasing your customer satisfaction

Finding more bugs earlier in the process, where they costFinding more bugs earlier in the process, where they costa lot less to fixa lot less to fix

This advice is based on the experience of many device This advice is based on the experience of many device testers at Microsofttesters at Microsoft

While this session attempts to cover most While this session attempts to cover most drivers/devices, some points may not be applicabledrivers/devices, some points may not be applicableto your driver/deviceto your driver/device

Agenda Agenda

Overview Overview

ToolsTools

Test configurationsTest configurations

Testing specific driver aspectsTesting specific driver aspects

Types of testingTypes of testing

ResourcesResources

Call to actionCall to action

Overview Of TopicsOverview Of Topics

Testing types

Stress tests

Manual testing

Test strategies

Test early

Plan for testing

Resources

DTM kits

Test Framework

Long haul testing

Scenario testing

Static tools

PREfast

SDV

Test configurations

System

Settings

Run-time tools

Driver Verifier

Driver aspects

Memory

PO(Power Management)

Penetration/Security

I/O Cancel

PnP

Setup

Concurrency

Static Analysis ToolsStatic Analysis Tools

Tools that run against the source codeTools that run against the source code““PREPREffast for Drivers”ast for Drivers”

““Static Driver Verifier”Static Driver Verifier”

Find bugs fast Find bugs fast Quick and inexpensive ways to find bugsQuick and inexpensive ways to find bugs

Must pass these tests for driver signatures or logoMust pass these tests for driver signatures or logo

Run early/run oftenRun early/run oftenUse these tests early in your development cycleUse these tests early in your development cycle

Resolve issues found by the toolsResolve issues found by the tools

Run the tests for every new driver dropRun the tests for every new driver drop

For details For details At WinHEC: “Static Analysis and Verification of Drivers”At WinHEC: “Static Analysis and Verification of Drivers”

Tools and documentation: WDKTools and documentation: WDK

Driver Verifier Driver Verifier

Verifies different aspects of drivers at runtimeVerifies different aspects of drivers at runtime

Built into Windows Built into Windows Type verifier from the Run menu Type verifier from the Run menu

Valuable in your regular testingValuable in your regular testingUse standard settingsUse standard settings

Does Special pool, IRP verification, IRQL checking, etc. Does Special pool, IRP verification, IRQL checking, etc.

Same settings are used for driver signature or logo testingSame settings are used for driver signature or logo testing

Only enable settings for drivers under testOnly enable settings for drivers under testTo ensure proper testing of your driverTo ensure proper testing of your driver

Get high test coverage Get high test coverage By running with a full test pass By running with a full test pass

You must exercise the code; otherwise, it cannot verify the driverYou must exercise the code; otherwise, it cannot verify the driver

For detailsFor detailsTools and documentation: WDKTools and documentation: WDK

Many new features for Windows VistaMany new features for Windows Vista

Test Configuration Test Configuration

Vary your test environmentVary your test environmentDon’t just test your device in a static Don’t just test your device in a static environment – change things aroundenvironment – change things around

System configurationSystem configuration

System environmentsSystem environments

Device settingsDevice settings

Make these part of your regular test passMake these part of your regular test pass

System ConfigurationSystem Configuration

Different platforms: x86, x64, ItaniumDifferent platforms: x86, x64, Itanium

System with S1 and S3 support (for PO testing)System with S1 and S3 support (for PO testing)Ensure system has WDDM video driversEnsure system has WDDM video drivers

So you get hybrid sleep (S3 with Hiberfile)So you get hybrid sleep (S3 with Hiberfile)

Multiprocessor/multi-core machinesMultiprocessor/multi-core machinesOnly way to truly test concurrencyOnly way to truly test concurrency

Hyperthreading doesn't count for this testingHyperthreading doesn't count for this testing

Different chipsetsDifferent chipsets

Checked buildChecked buildSkews timingSkews timing

Has additional assertionsHas additional assertions

See “Using the Checked Build” in the WDKSee “Using the Checked Build” in the WDK

System EnvironmentsSystem Environments

Low memory Low memory See “Boot Parameters to Manipulate Memory” in the WDKSee “Boot Parameters to Manipulate Memory” in the WDK

Large memoryLarge memoryThis is important for both for X86 and 64 bitThis is important for both for X86 and 64 bit

See “Will I need special hardware for memory top-down testing?”See “Will I need special hardware for memory top-down testing?”in the WDKin the WDK

High CPU utilization High CPU utilization From something other than testing the device From something other than testing the device

Stress testing does this Stress testing does this

Skews timingSkews timingCan uncover race conditions in your driverCan uncover race conditions in your driver

Device SettingsDevice Settings

Test your device with all of its settingsTest your device with all of its settingsNot just the default settingsNot just the default settings

Only you know what these settings do Only you know what these settings do

If your device uses hardware resources If your device uses hardware resources (IRQ, I/O Port...), change them (IRQ, I/O Port...), change them

WDK tool “PnP Driver Test”WDK tool “PnP Driver Test”does rebalancingdoes rebalancing

Manually in Device ManagerManually in Device Manager

Device ConfigurationDevice Configuration

Test several of the same deviceTest several of the same devicein a systemin a system

Topologies Topologies Behind a PCI-PCI bridge Behind a PCI-PCI bridge

Connected to a large tree of devicesConnected to a large tree of devicesFor USB or 1394 device, for example For USB or 1394 device, for example

……

Testing Specific Driver AspectsTesting Specific Driver Aspects

Aspects of driver testing that mayAspects of driver testing that maynot usually be your focus not usually be your focus

But tend to have high failure countsBut tend to have high failure counts

Driver aspects

Memory

PO

Penetration/Security

I/O Cancel

PnP

Setup

Concurrency

MemoryMemory

Still a problem area with drivers Still a problem area with drivers

Corruption – testing methodsCorruption – testing methodsDriver Verifier with special pool settingDriver Verifier with special pool setting

Static analysis toolsStatic analysis tools

Code review Code review

Stress testingStress testing

Leaks – testing methodsLeaks – testing methodsCode reviewCode review

Manual testing Manual testing Use “PoolMon” In the WDK to monitor pool allocationsUse “PoolMon” In the WDK to monitor pool allocations

Generally, do repetitive operations, and look for continuedGenerally, do repetitive operations, and look for continuedpool usage increasespool usage increases

Plug And Play – PnPPlug And Play – PnP

WDK testsWDK tests““Disable/Enable with IO”Disable/Enable with IO”

““PnP Driver test”PnP Driver test”

Long haul testingLong haul testingSet up system that tests PnP for several daysSet up system that tests PnP for several days

Test the driver/device in docking stationsTest the driver/device in docking stationsTest undock/dockTest undock/dock

Manual testingManual testingPhysically unplug and replugPhysically unplug and replug

The deviceThe device

The bus/media: USB cable, net cables...The bus/media: USB cable, net cables...

Use your deviceUse your deviceThen disable/enable itThen disable/enable it

Power Management Power Management

Windows Vista = more aggressive PO management Windows Vista = more aggressive PO management Not just for laptops anymoreNot just for laptops anymore

WDK testsWDK tests““Sleep Stress with IO” – cycles through sleep statesSleep Stress with IO” – cycles through sleep statesand does device I/O before and afterwardand does device I/O before and afterward

““Pwrtest” – Logs power eventsPwrtest” – Logs power events

Long haul testingLong haul testingSet up system that tests PO for several daysSet up system that tests PO for several days

Manual testingManual testingUse the device, and thenUse the device, and then

Put the system to sleepPut the system to sleep

Hibernate the systemHibernate the system

PnP And Power ManagementPnP And Power Management

Test both togetherTest both togetherDon’t just test each in isolationDon’t just test each in isolation

This is a real-world scenarioThis is a real-world scenarioFor example, often when you close theFor example, often when you close thelid on a laptop, you also unplug deviceslid on a laptop, you also unplug devices

WDK testsWDK testsCommon Scenario Stress with IOCommon Scenario Stress with IO

Manual testingManual testingCombine existing automated testsCombine existing automated tests

Or do these manuallyOr do these manually

Fault InjectionFault Injection

Test driver code paths that are hard to hit Test driver code paths that are hard to hit under normal useunder normal use

Error handlingError handling

Corner cases Corner cases

Hard to testHard to testYou may need to add test code to the driverYou may need to add test code to the driverduring testingduring testing

Write a test filter driverWrite a test filter driver

Driver Verifier has a Fault Injection settingDriver Verifier has a Fault Injection setting

For USB, try Device Simulation Framework For USB, try Device Simulation Framework

ConcurrencyConcurrency

Do different things to the driver at the same timeDo different things to the driver at the same timeAnalysis of our tests show they do not give very good Analysis of our tests show they do not give very good coverage in this areacoverage in this area

System configurationSystem configurationUse multiprocessor machinesUse multiprocessor machines

TestingTestingUse your device in multiple processesUse your device in multiple processes

Use your device andUse your device andDisable itDisable it

Shut down the systemShut down the system

Put the system to sleepPut the system to sleep

Change the configurationChange the configuration

Get data from it (WMI/IOCTL) Get data from it (WMI/IOCTL)

……

Penetration/SecurityPenetration/Security

Be resilient against mischiefBe resilient against mischiefDon’t assume that only your user-mode code will be calling Don’t assume that only your user-mode code will be calling your deviceyour device

Code review the IOCTL and WMI code pathCode review the IOCTL and WMI code path

Who should have access to your devices: Test for itWho should have access to your devices: Test for it

See “Creating Reliable Kernel-Mode Drivers” in WDKSee “Creating Reliable Kernel-Mode Drivers” in WDKErrors in buffered I/O, etc.Errors in buffered I/O, etc.

Test and code review for these pitfallsTest and code review for these pitfalls

Device path exerciser in WDK test – limited but usefulDevice path exerciser in WDK test – limited but usefulRun with Driver Verifier on target driver enabledRun with Driver Verifier on target driver enabled

For bugs found with this tool, do a code reviewFor bugs found with this tool, do a code reviewLook for similar bugs in other code pathsLook for similar bugs in other code paths

Where bug was found, look for other types of bugsWhere bug was found, look for other types of bugsDon’t expect this tool to find other bugs in this area for youDon’t expect this tool to find other bugs in this area for you

Use static analysis and verification toolsUse static analysis and verification tools

I/O CancellationI/O Cancellation

Still a problem area with drivers Still a problem area with drivers

Hard to testHard to testI/O must be currently active in the driverI/O must be currently active in the driver

Testing Testing Use your device, and then kill the process using itUse your device, and then kill the process using it

Doesn’t guarantee you tested I/O cancellationDoesn’t guarantee you tested I/O cancellation

Set a break point or add tracing on cancellation code Set a break point or add tracing on cancellation code Make sure it gets hit during your testingMake sure it gets hit during your testing

May need to add test code to your driverMay need to add test code to your driver

ExperimentExperiment

Setup PackagesSetup Packages

Test for all scenarios Test for all scenarios OEM preinstall OEM preinstall

During OS installation (for boot drivers) During OS installation (for boot drivers)

After OS installation After OS installation

OS Migration OS Migration Install your driver on Windows XP, then upgrade to Windows Vista Install your driver on Windows XP, then upgrade to Windows Vista and ensure drivers and applets workand ensure drivers and applets work

Same for Windows Vista to Windows Vista upgradesSame for Windows Vista to Windows Vista upgradesFor example, Home Basic to Home Premium SKUsFor example, Home Basic to Home Premium SKUs

Test with multiple packages Test with multiple packages Create 2 packages with 2 different versions of your driverCreate 2 packages with 2 different versions of your driver

Install version 1Install version 1

Upgrade to version 2Upgrade to version 2

Rollback to version 1Rollback to version 1

Un-install Un-install

Tools: ChkInfTools: ChkInf

Testing TypesTesting Types

Testing types

Stress tests Scenario testing

Stress TestingStress Testing

Combines tests to simulate a heavy loadCombines tests to simulate a heavy loadTests your driver under different system conditionsTests your driver under different system conditions

Can find bugs that can be missed in feature testingCan find bugs that can be missed in feature testing

Stress testsStress testsWDK has a stress test WDK has a stress test

Allows you to add your device-specific testAllows you to add your device-specific test

Build your ownBuild your own

Testing – time and configurationTesting – time and configurationRun for at least 24 hours at a timeRun for at least 24 hours at a time

Run on different system configuration Run on different system configuration

Run with different device settingsRun with different device settings

Scenario TestingScenario Testing

Test your devices the way customersTest your devices the way customerswill use themwill use them

Don’t just rely on synthetic Don’t just rely on synthetic tests/automationtests/automation

Focus on corner cases scenariosFocus on corner cases scenariosMainstream scenarios tend to notMainstream scenarios tend to notbe a problem be a problem

Customers always find new waysCustomers always find new waysto use your devicesto use your devices

ResourcesResources

Driver Test Manager kits – DTMDriver Test Manager kits – DTM

Test frameworksTest frameworksWindows Device Testing Framework – WDTF Windows Device Testing Framework – WDTF

Device Simulation Framework – DSFDevice Simulation Framework – DSF

Driver FrameworksDriver FrameworksKernel-mode Driver Framework – KMDF Kernel-mode Driver Framework – KMDF

User-mode Driver Framework – UMDFUser-mode Driver Framework – UMDF

Windows Error ReportingWindows Error Reporting

DTM Test KitsDTM Test Kits

Run early Run early When software, firmware, and hardware changes are still possibleWhen software, firmware, and hardware changes are still possible

Use these tools on a daily basis in your testingUse these tools on a daily basis in your testing

Keep in mind that driving issues to a resolution takes time: Keep in mind that driving issues to a resolution takes time: Plan for 2-4 weeksPlan for 2-4 weeks

Many useful DTM tests for your regular testingMany useful DTM tests for your regular testing

Use DTM automation features in your labUse DTM automation features in your labAdd your own testsAdd your own tests

At WinHEC: WDK and DTM presentationsAt WinHEC: WDK and DTM presentations““How to Use the WDK to Develop, Sign and Test Drivers”How to Use the WDK to Develop, Sign and Test Drivers”

““Using the WDK for Windows Logo and Signature Testing”Using the WDK for Windows Logo and Signature Testing”

Sign up for the WDK Beta: Sign up for the WDK Beta: www.microsoft.com/whdc/driver/wdk/betawdk.mspxwww.microsoft.com/whdc/driver/wdk/betawdk.mspx

See WHDC Web site: See WHDC Web site: www.microsoft.com\www.microsoft.com\whdcwhdc

““Index of Windows Driver Kit Tools” in the WDKIndex of Windows Driver Kit Tools” in the WDK

WDTF – Windows Device WDTF – Windows Device Testing FrameworkTesting Framework

Framework for building driver/device testsFramework for building driver/device testsSome DTM tests are built using WDTFSome DTM tests are built using WDTF

PnP and Power Management testsPnP and Power Management tests

Write your own testsWrite your own tests

For detailsFor detailsAt WinHEC: “Using the Windows DeviceAt WinHEC: “Using the Windows DeviceTesting Framework”Testing Framework”

Tools and documentation: WDKTools and documentation: WDK

DSF – Device SimulationDSF – Device SimulationTest FrameworkTest Framework

Simulate hardware through softwareSimulate hardware through softwareCurrently only for USBCurrently only for USB

For detailsFor detailsAt WinHEC: “Using the Device Simulation At WinHEC: “Using the Device Simulation Framework for Software Simulation of USB Framework for Software Simulation of USB Devices” sessionDevices” session

WDKWDKDSF USB Loop Back Device Simulation DSF USB Loop Back Device Simulation

Windows Driver Frameworks (WDF)Windows Driver Frameworks (WDF)

Generally, testing is same as for other drivers Generally, testing is same as for other drivers

Some features that might helpSome features that might help

FeaturesFeatures UMDFUMDF KMDFKMDF

In Flight recorderIn Flight recorder XX

Integrated VerifierIntegrated Verifier XX XX

Normal Verifier Normal Verifier App VerifierApp Verifier Driver VerifierDriver Verifier

Skew executionSkew execution XX

Object live-time trackingObject live-time trackingFlags leaks on unloadFlags leaks on unload

XX XX

Fault injectionsFault injectionsVia registry key settingsVia registry key settings

XX

Debugger extensionDebugger extension XX XX

Windows Error ReportingWindows Error Reporting

Database of Windows crashesDatabase of Windows crashes

Improve experience/reduce costs Improve experience/reduce costs Improve your customers’ experienceImprove your customers’ experience

Reduce your support costs – and your partners’Reduce your support costs – and your partners’

Engage through Winqual Engage through Winqual Sign up to see crash data for your driversSign up to see crash data for your drivers

Provide solutions through Windows UpdateProvide solutions through Windows Update

Check data regularly Check data regularly For just-released products, check dailyFor just-released products, check daily

For products on the market for a while, check monthlyFor products on the market for a while, check monthly

For detailsFor detailsWinqual Web site: Winqual Web site: https://winqual.microsoft.comhttps://winqual.microsoft.com

Call To ActionCall To Action

Incorporate the testing points from this Incorporate the testing points from this presentation into your regular test passpresentation into your regular test pass

Test early Test early Start in the design and prototyping phaseStart in the design and prototyping phase

Take advantage of the resources listed hereTake advantage of the resources listed here

Engage with Windows Error ReportingEngage with Windows Error Reporting

Engage with driver testers at MicrosoftEngage with driver testers at MicrosoftDrvtest @ microsoft.comDrvtest @ microsoft.com

Additional ResourcesAdditional Resources

WDK: WDK: http://www.microsoft.com/whdc/wdkhttp://www.microsoft.com/whdc/wdk

Testing tools on WHDC: Testing tools on WHDC: http://www.microsoft.com/http://www.microsoft.com/whdc/DevTools/toolswhdc/DevTools/tools//

Driver aspectsDriver aspectsPO: PO: http://www.microsoft.com/whdc/system/sysperf/resumeperf.mspxhttp://www.microsoft.com/whdc/system/sysperf/resumeperf.mspx

Memory: Memory: http://www.microsoft.com/whdc/driver/security/mem-alloc_tst.mspxhttp://www.microsoft.com/whdc/driver/security/mem-alloc_tst.mspx

Setup: “Using Driver Install Frameworks (DIFx)” – WDK Setup: “Using Driver Install Frameworks (DIFx)” – WDK

Microsoft Symbol ServerMicrosoft Symbol ServerAll OS symbols for all releases and patches: All OS symbols for all releases and patches: http://www.microsoft.com/whdc/devtools/debugging/http://www.microsoft.com/whdc/devtools/debugging/

Send email: Windows Driver testing aliasSend email: Windows Driver testing aliasDrvtest @ microsoft.comDrvtest @ microsoft.com

QuestionsQuestions

Tomorrow’s “Ask the expert” sessionTomorrow’s “Ask the expert” session

© 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions,

it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.