Co-verification Experience
-
Upload
mariko-phelps -
Category
Documents
-
view
34 -
download
2
description
Transcript of Co-verification Experience
![Page 1: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/1.jpg)
Co-verification Experience
Juncao LiSystem Verification LabComputer Science, PSU
01/05/2010
![Page 2: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/2.jpg)
Agenda
• Overview• Sealevel PIO-24 Digital I/O Card• Intel 100M PCI Ethernet Adapter
System Verification Lab @CS, Portland State University 2
![Page 3: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/3.jpg)
PIO-24 Digital I/O Card
• Driver code and device manual links– http://www.osronline.com/article.cfm?article=403– https://www.sealevel.com/uploads/manuals/8018S.pdf
• Driver size: 1724 lines of C code• Device model size: 1232 lines of C code• Issues found in the specification (device manual):
2• Bugs found in the driver: 4• Properties proved in the driver: 2
System Verification Lab @CS, Portland State University 3
![Page 4: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/4.jpg)
Intel 100M PCI Ethernet adapter
• Driver code and device manual links– http://msdn.microsoft.com/en-us/library/dd163298.aspx
• The source code is available in WDK releases.• E.g., the SLAM SD one is in “WDK\src_6001\kmdf\pcidrv”
– http://download.intel.com/design/network/manuals/8255X_OpenSDM.pdf
• Driver size: 14406 lines of C code• Device model size: 3518 lines of C code• Issues found in the specification (device manual):
6• Bugs found in the driver: 3• Properties proved in the driver: 5
System Verification Lab @CS, Portland State University 4
![Page 5: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/5.jpg)
Agenda
• Overview• Sealevel PIO-24 Digital I/O Card• Intel 100M PCI Ethernet adapter
System Verification Lab @CS, Portland State University 5
![Page 6: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/6.jpg)
Specification Issues
• Hardware/Software (HW/SW) interface specifications (manuals) usually have problems:– Incompleteness: what should be clarified is
missing– Inconsistency: multi-places do not consistent
with each other
• Spec issues are found when – We model hardware devices according to the
specs– We verify drivers with the hardware models System Verification Lab @CS, Portland State University 6
![Page 7: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/7.jpg)
Specification Issues
• Issue 1 (Incompleteness): – Location: Page 11, Section: Interrupt Read.– Problem:
• The default value of the register ISRQST1 is not specified. ISRQST1 indicates the interrupt pending status and “should” be 0 when the device is powering on.
– This problem is related to the bug “ProperISR2”
• Issue 2 (Inconsistency): – Location: Page 11, Section: I/O Control Code.– Problem:
• The CW1D1 bit of the Control Word register is never used, but the bit is not defined as “Not Used” in the Section, Register Description, at Page 10.
System Verification Lab @CS, Portland State University 7
![Page 9: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/9.jpg)
Bug1: InvalidRead
• Driver will not read any invalid data• About the bug:
– Driver returns an invalid value to user applications without reading any hardware data register
System Verification Lab @CS, Portland State University 9
![Page 10: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/10.jpg)
Initialize the device model
Test harness entry
![Page 11: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/11.jpg)
Predicates, i.e. HW/SW states
HW state variable: no interrupt pending
HW state variable: interrupt enabled
SW state variable, the inconsistency of the two variables directly causes the bug
Entry the callback function:EvtIoDeviceControl
![Page 12: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/12.jpg)
In the callback function
Process the I/O control code
![Page 13: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/13.jpg)
I/O control codeDriver understands “else” as: interrupt is enabled in hardware
![Page 14: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/14.jpg)
HW runs after this statement, where CurrentRequest and AwaitingInt
become inconsistent
![Page 15: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/15.jpg)
The HW Transaction Function
In Hardware
Simulate the environment or the DIO device behaviors
![Page 16: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/16.jpg)
In RunInterrupt
Fires an interrupt
![Page 17: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/17.jpg)
Call the ISR
![Page 18: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/18.jpg)
In the ISR
In Software
![Page 19: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/19.jpg)
If it is this driver’s interrupt
Because the driver routine EvtIoDeviceControl was interrupted, AwaitingInt is
not “TRUE” yet
![Page 20: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/20.jpg)
“data” is not read from the device
Schedule the DPC. This immediately violates the
rule: “CvIsrCallDpc” (see later slides)
![Page 21: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/21.jpg)
ISR returns, EvtIoDeviceControl continues
![Page 22: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/22.jpg)
EvtIoDeviceControl returns
![Page 23: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/23.jpg)
DPC runs
![Page 24: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/24.jpg)
CurrentReqest is not NULL, so prepare the data for the
user application!
![Page 25: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/25.jpg)
The I/O request is completed with STATUS_SUCCESS, but the data was never
read from the hardware register!
![Page 26: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/26.jpg)
Bug2: CvIsrCallDpc
• Before ISR schedules a DPC, the ISR should read hardware volatile registers first
• About the bug: – Driver does not read the hardware data register but still
requests the DPC
• This bug is already illustrated in the error trace of the “InvalidRead” bug– The actual error trace of the CvIsrCallDpc bug is different
because we used another life circle harness and this bug can happen at various places.
System Verification Lab @CS, Portland State University 26
![Page 27: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/27.jpg)
However …
• InvalidRead and CvIsrCallDpc are different rules– Switch the two lines (in DioEvtDeviceControl) can fix the
“InvalidRead” bug:devContext->CurrentRequest = Request;devContext->AwaitingInt = TRUE;
– To fix the “CvIsrCallDpc” bugWe need to move the WdfInterruptQueueDpcForIsr(...) call into the “if(devContext->AwaitingInt) { ... }” block in ISR.
– These fixes have been proved correct by SDV (CoVer).
System Verification Lab @CS, Portland State University 27
![Page 28: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/28.jpg)
Bug3: ProperISR1
• If ISR returns TURE (i.e. acknowledge the interrupt to OS), the device interrupt status should not be active; otherwise, it may cause the interrupt storm.
• There are two scenarios that can cause this bug: – When the interrupt firing condition is configured as “high
level” (resp. “low level”), the interrupt should be repeatedly fired if the input to Port A (least significant bit) stays high (reps. low)
– When the condition is “rising edge” (resp. falling edge), depending on the input frequency to Port A, the interrupt will be repeatedly fired
• Both of the scenarios can cause interrupt stormSystem Verification Lab @CS, Portland State University 28
![Page 29: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/29.jpg)
In ISR
HW state: interrupt fired
![Page 30: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/30.jpg)
Read the interrupt status and clear the register at
the same time
![Page 31: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/31.jpg)
In hardware, interface event function
Clear the interrupt status register on read
![Page 32: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/32.jpg)
HW transaction function runs after the interrupt status register has being cleared
![Page 33: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/33.jpg)
Low level fires the interrupt
![Page 34: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/34.jpg)
The SLIC rule
![Page 35: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/35.jpg)
About This Bug
• We learned a solution from the Ethernet adapter driver:– If the device may fire interrupts freqently– Disable the interrupt first in ISR and enable it later in
DPC after currect request has been serviced
System Verification Lab @CS, Portland State University 35
![Page 36: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/36.jpg)
Bug4: ProperISR2
• ISR should not return TRUE if the interrupt of the device is not active, i.e. it is not this driver's interrupt.
• About the bug: – ISR acknowledges an interrupt even when the interrupt
of the device is disabled– Related to spec issue 1: ISRQST1 doesn’t have a default
value– A scenario for this bug:
• ISRQST1 is initialized to be 1 during the device power-on• OS registers the DIO driver’s ISR to the PCI bus’ interrupt vector
table • Another PCI device fires an interrupt• The DIO driver may have an “opportunity” to acknowledge this
interrupt System Verification Lab @CS, Portland State University 36
![Page 37: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/37.jpg)
Bug4: ProperISR2
• The causes of this bug:– The default value of ISRQST1 is not clearly stated in the
spec– The driver doesn’t reset the device during the device
power-on– The ISR only checks the ISRQST1 register to decide if
there is an interrupt pending (this bug can be avoided, if the interrupt-enabled register is also checked)
• I have to admit, the chance for this bug seems quite low– However, when it happens, no one will notice!
System Verification Lab @CS, Portland State University 37
![Page 38: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/38.jpg)
Agenda
• Overview• Sealevel PIO-24 Digital I/O Card• Intel 100M PCI Ethernet adapter
System Verification Lab @CS, Portland State University 38
![Page 39: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/39.jpg)
Spec Issues – Some Examples
• Issue 1 (Inconsistency): – Location: Page 38 - 39– Problem:
• Table 15 is inconsistent with Table 14 about the types of CU commands.
• Issue 2 (Inconsistency): – Location: Page 136, Section: 8.2 Transmit
Processing– Problem:
System Verification Lab @CS, Portland State University 39
The word “previous” is missing here, otherwise the logic is wrong
![Page 41: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/41.jpg)
Bug1: DoubleCUC
• The driver should not issue a CU command while the Command Unit (CU) is busy (not zero)– This is clearly stated in the device manual (specification)– Spec Page 37, Section: 6.3.2.2 SCB Command Word
• About the bug: – Driver issues a command to CU regardless the result of
the previous device operation (software reset)
System Verification Lab @CS, Portland State University 41
![Page 42: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/42.jpg)
In EvtDeviceD0Entry callback function
HW states are non-deterministically initialized
![Page 43: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/43.jpg)
Issue a software reset
![Page 44: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/44.jpg)
Write to the “PORT” register to issue the
command
Inside the function HwSoftwareReset
![Page 45: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/45.jpg)
In hardware model
Start the software reset process
![Page 46: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/46.jpg)
Wait for the port reset to complete.
In software
Note: the device manual doesn’t promise that in how long a port reset can complete, so wait is NOT enough!
![Page 47: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/47.jpg)
Issue a command when the reset process is not finished
Out from software reset.
![Page 48: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/48.jpg)
Issue a command without waiting for the CU free
![Page 49: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/49.jpg)
The SLIC rule
![Page 50: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/50.jpg)
About this bug
• I was not sure at first• So I checked the Linux driver of the
same device– http://sourceforge.net/projects/e1000/– The driver is also found in (for example):
• linux-2.6.32/drivers/net/e100.c
System Verification Lab @CS, Portland State University 50
![Page 51: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/51.jpg)
Let’s see how the Linux driver handles “software reset”.
It makes sure that the device is in correct state before issuing any command.
![Page 52: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/52.jpg)
The Linux driver always waits before issuing a new command
![Page 53: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/53.jpg)
Bug2: DevD0Entry
• The callback function EvtDeviceD0Entry returns the value (TRUE or FALSE) that correctly represents the hardware state (initialized or failed)– This rule is clearly stated in MSDN
• About the bug: – Driver returns STATUS_SUCCESS even if the operations
on the device have failed– The error trace illustrates that the driver continues its
attempts to initialize the device even after the previous operations have failed
System Verification Lab @CS, Portland State University 53
![Page 54: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/54.jpg)
In EvtDeviceD0Entry callback function
![Page 55: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/55.jpg)
In NICInitializeAdapter
![Page 56: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/56.jpg)
The command is timeout. Failure starts here
![Page 57: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/57.jpg)
Return the failure status
![Page 58: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/58.jpg)
Do some work and return the failure status that is
returned from NICInitializeAdapter()
![Page 59: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/59.jpg)
The return value of MPSetPowerD0Private() is ignored. The initialization
process goes on as if nothing happened
![Page 60: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/60.jpg)
EvtDeviceD0Entry returns STATUS_SCCESS while the device state is a mess
…
![Page 61: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/61.jpg)
• Once again, how about the Linux driver?
System Verification Lab @CS, Portland State University 61
![Page 62: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/62.jpg)
Corresponding to EvtDeviceD0Entry
System Verification Lab @CS, Portland State University 62
![Page 63: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/63.jpg)
![Page 64: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/64.jpg)
Bug3: DevD0Exit
• The callback EvtDeviceD0Exit returns the value (TRUE or FALSE) that can correctly represent the hardware state (i.e. if the hardware has been properly stopped).– This rule is clearly stated in MSDN
• About the bug: – Driver returns STATUS_SUCCESS even if the operations
on the device have failed.
System Verification Lab @CS, Portland State University 64
![Page 65: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/65.jpg)
Reference• Juncao Li, Fei Xie, Thomas Ball, Vladimir Levin, and Con McGarvey. An Automata-Theoretic
Approach to Hardware/Software Co-verification. To appear in Proc. of International Conference on Foundational Approaches to Software Engineering (FASE)
– Link: http://web.cecs.pdx.edu/~juncao/links/mypapers/cover2010.pdf
System Verification Lab @CS, Portland State University 65
![Page 66: Co-verification Experience](https://reader038.fdocuments.in/reader038/viewer/2022102818/5681331d550346895d99e570/html5/thumbnails/66.jpg)
System Verification Lab @CS, Portland State University 66
Questions ?