QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix
-
Upload
yan-vugenfirer -
Category
Technology
-
view
1.661 -
download
0
description
Transcript of QEMU Development and Testing Automation Using MS HCK - Anton Nayshtut and Yan Vugenfirer, Daynix
QEMU development and testing automation using MS HCK
Anton Nayshtut, [email protected] Yan Vugenfirer, [email protected]
Daynix Computing LTD
Daynix Computing LTD
Agenda
What is WHQL/HCK?
Introducing VirtHCK
Using VirtHCK
Call to action
�2
Daynix Computing LTD
What is HCK/WHQL?The Windows certification program (previously known as the Windows Logo Program) for devices and system
A comprehensive device\system test toolkit by Microsoft
Devices which pass HCK are certified and their binaries are digitally signed
�3
Daynix Computing LTD
HCK test kit
Introduced with Windows 8 (previous kit know as WLK)
Predefined test suites for each device type
Ability to add your own tests for additional testing outside of the certification process
Reproducible scenarios
�4
Daynix Computing LTD
Benefits for QEMU and Linux community
QEMU device testing (not only virtio)
Host subsystems testing, for example networking
Extensive coverage for defined device categories
Regression testing
Stress testing
�5
Daynix Computing LTD
HCK PCI compliance testDevice under test compliance to PCI spec
Applicable for:
New QEMU chipsets
PCIe devices
Next generation of virtio devices
Existing PCI devices
�6
Daynix Computing LTD
HCK NDIS (network) tester (1)
Comprehensive networking test suite
Send/receive,
Stress tests (MPE)
Offload, VLAN, priority, packet filtering
etc
Not once used to test the robustness of the host networking stack
�7
Daynix Computing LTD
HCK NDIS (network) tester (2)Applicable for
QEMU network devices
New devices
New features
Host network components
Bridge
Open vSwitch
TAP
Kernel�8
Daynix Computing LTD
HCK Windows OS tests
Device path exerciser
Identifies drivers that do not correctly handle the different IO calls
Driver verifier
Captures deadlocks, memory leaks, memory corruption
Verified correctness of kernel API usage
�9
Daynix Computing LTD
HCK power management tests
The test cycling through different sleep states
There are variations of IO and PNP (disable/enable) during, before and after power transitions
Applicable for:
QEMU Devices power management support
SeaBIOS
�10
Daynix Computing LTD
Our past experienceTransfer hangs with vhost.
Offload/VLAN related corner cases with network devices (virtio-net, vmxnet3, vmxnet).
vmxnet3 and pvscsi in QEMU
Open vSwitch out of order issues
RSC (Windows GRO)
PCI compliance
�11
Daynix Computing LTD
HCK - great, but frightening
Cumbersome setup requirements
Complex network configuration
Hard to replicate setups
Setup conflicts
No power management tests automation support
�12
Daynix Computing LTD
VirtHCK
�13
Daynix Computing LTD
Introducing VirtHCK
Framework for automating HCK test execution
Based on easily configurable host and guest scripts
Creates isolated virtual network environments
Allows running different HCK scenarios in automated manner
OSS, distributed under BSD license
�14
Daynix Computing LTD
Repository and WiKi
Get VirtHCK from https://github.com/daynix/bricklets.git
VirtHCK wiki with steps for deployment and usage: https://github.com/daynix/bricklets/wiki/VirtHCK-bricklet-documentation
�15
Daynix Computing LTD
HCK issues solved by VirtHCK
Setup replication
Multiple setups sharing the same host
Setup isolation solves the following issues:
HCK controller controls all HCK clients in the network
Inability to use VM templates without manual VM reconfiguration
Automatic network topology setup during network device tests
�16
Daynix Computing LTD
Setup types supported
Network (virtio-net, vmxnet3, vmxnet)
Storage (virtio-block, virtio-scsi, pvscsi)
Simple PCI (virtio-serial, virtio-balloon)
Easily extendable by small changes in configuration files
�17
Daynix Computing LTD
Typical HCK deployment
WHQL control network
Control NIC
Test Client
HCK Controller
Control NIC - e1000
Control NIC
Test Client
Control NIC
Test Client
�18
Daynix Computing LTD
VirtHCK setup
Disconnected from outside world network during test run
Isolated NICs under test network
Isolated WHQL control network
A different number of test clients can run according to test type
�19
Daynix Computing LTD
VirtHCK setup for storage and other devices
WHQL control bridge
“World” bridge Physical NIC
Control NIC - e1000
Test VM 1
Studio VM
Outside world NIC
Control NIC - e1000
�20
Daynix Computing LTD
VirtHCK setup for storage and other devices - running
WHQL control bridge
“World” bridge Physical NIC
Control NIC - e1000
Test VM 1
Studio VM
Outside world NIC
Control NIC - e1000
X�21
Daynix Computing LTD
VirtHCK setup for NICs
Test bridge
WHQL control bridge
“World” bridge Physical NIC
Control NIC - e1000
Test VM 1
NIC under test
Control NIC - e1000
Test VM 2
NIC under test
Studio VM
Outside world NIC
Control NIC - e1000
�22
Daynix Computing LTD
VirtHCK setup for NICs - running
Test bridge
WHQL control bridge
“World” bridge Physical NIC
Control NIC - e1000
Test VM 1
NIC under test
Control NIC - e1000
Test VM 2
NIC under test
Studio VM
Outside world NIC
Control NIC - e1000
X�23
Daynix Computing LTD
VirtHCK - quick startClone VirtHCK from github
Create VM images in /images directory (studio and test client)
Update STUDIO_IMAGE/CLIENT1_IMAGE/CLIENT2_IMAGE variables in hck_setup.cfg
Run VirtHCK - “sudo ./VirtHCK/hck.sh”
Install Microsoft HCK Studio and Clients software according to Microsoft documentation Windows HCK Getting Started and following Checklist for a new client VM guidelines.
�24
Daynix Computing LTD
Directory treeVirtHCK/ - Directory of all VirtHCK scripts, supposed to run in Linux environment
VirtHCK/guest_tools/ - Directory for scripts required for Windows guests
Images directory where VM's images should be located
�25
Daynix Computing LTD
VirtHCK walkthrough - host
Configuration: network, VMs etc
Host scripts
Images
�26
Daynix Computing LTD
hck_setup.cfg
VirtHCK setup configuration file
Environment configuration
VM start up parameters
Configured accordingly to required HCK setup type
Allows multiple setups on one host
�27
Daynix Computing LTD
Running multiple setups
Use UNIQ_ID to configure multiple setups
UNIQ_ID will be used as a base for creation of:
VNC or Spice ports
Bridges names
Network interface names, MAC addresses and etc.
�28
Daynix Computing LTD
Test device types
Network
Storage (also boot)
Balloon
Serial
More can be added
�29
Daynix Computing LTD
Remote access
VNC or Spice
Spice is needed for power management tests
�30
Daynix Computing LTD
Network infrastructure
Linux bridge or Open vSwitch can be used
�31
Daynix Computing LTD
Images
Configure
Studio VM image
Test VMs images
�32
Daynix Computing LTD
VM resources
Number of vCPUs
Memory
NICs for control channels
�33
Daynix Computing LTD
Extra parametersQEMU binary
VHOST, snapshots, cache mode for fast install
ISOs for guest installation
BIOS
�34
Daynix Computing LTD
Configuration summary
�35
Daynix Computing LTD
Host scripts - hck.sh
Main script to run HCK enviroment
Dumps configuration
Creates network configuration for HCK setup
Runs VMs in loop (crucial during PM tests)
Cleans up all the configurations on exit
�36
Daynix Computing LTD
VM execution scripts
Executed by hck.sh
run_hck_studio.sh - starts up Studio VM
run_hck_client.sh -starts up Client VM
Configuration according hck_setup.cfg
�37
Daynix Computing LTD
Network scripts
Passed as parameter to QEMU to add network interface to appropriate bridge
hck_ctrl_bridge_ifup.sh
hck_test_bridge_ifup.sh
hck_world_bridge_ifup.sh
�38
Daynix Computing LTD
Guests configuration
�39
Daynix Computing LTD
Studio VM
Setup HCK according to Microsoft’s documentation
Follow the checklist - https://github.com/daynix/bricklets/wiki/VirtHCK-bricklet-documentation#wiki-Checklist_for_a_new_studio_VM:
Do not run HCK studio directly
Use run_hck_studio.bat or run_hck_studio_8.1.bat located in guest_tools
The above scripts will disconnect the HCK Studio VM from outside network
�40
Daynix Computing LTD
Test client VMs
Follow the checklist for test client - https://github.com/daynix/bricklets/wiki/VirtHCK-bricklet-documentation#wiki-Checklist_for_a_new_client_VM:
Install HCK client according to Microsoft’s documentation from \\hck-studio\HCKInstall\Client\setup.exe
�41
Daynix Computing LTD�42
Daynix Computing LTD
What’s nextHCK automation on the guest
SVVP configuration
Xen support (other hypervisors?)
Adding more device types
Using external configuration
Using libvirt
More guest scripts to automate configuration of test clients
�43
Daynix Computing LTD
HCK automation on the guest
Goal:
Run full test cycle for specific device type without user interaction.
�44
Daynix Computing LTD
Adding more device types
Goal
Test all the devices in QEMU and not only virtio (PCI bridges, USB controllers and etc)
�45
Daynix Computing LTD
More plans
Using external configuration
Multiple setups on one host without replicating VirtHCK and images
Use libvirt
Arbitrary number of test clients on one setup
Xen support
�46
Daynix Computing LTD
Call to action
Test QEMU devices with VirtHCK (not only virtio devices)
Contribute
�47
Q&A�48
Daynix Computing LTD
LinksWiKi - https://github.com/daynix/bricklets/wiki/VirtHCK-bricklet-documentation
Repository - https://github.com/daynix/bricklets.git
HCK kit download - http://msdn.microsoft.com/en-us/windows/hardware/hh833788.aspx
Windows HCK Getting Started - http://msdn.microsoft.com/en-us/library/windows/hardware/jj123537.aspx
Daynix - www.daynix.com
�49