SoC Design VerificationVerification Futures Conference
Mark OlenVerification Intrapreneur
Mentor Graphics Corp
March 19, 2013
© 2013 Mentor Graphics Corp.
www.mentor.comHF - February 2013, DVCon Lunch2
� Conducted by Wilson Research Group— Commissioned by Mentor Graphics— Format followed 2002, 2004 Collett studies for trend analysis, as
well as the 2007 FarWest Research Study
� Worldwide study— Overall confidence of 95% plus/minus 4.05%
� This was a blind study! — To eliminate any bias in the results
� This was a balanced study! — No single vendor dominated responses Wilson Research Group
Study Overview
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
Time Design Engineers Spend on Verification vs. Design
54%
2007
46%
2007
47%
2012
53%
2012
Doing Design Doing Verification
Design Engineer Project Time2007 - 2012
WCR DVCon February 20133
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
Verification Consumes Majority of Project Time
0%
5%
10%
15%
20%
25%
1%-20% 21%-30% 31%-40% 41%-50% 51%-60% 61%-70% 71%-80% >80%
Non-FPGA Study Participants
Time (Percent)
Total Project Time Spent in Verification
2007
2010
2012
2007: Mean 49%2010: Mean 56%2012: Mean 56%
Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study
WCR DVCon February 20134
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
More and More Verification Engineers Average Peak Number of Design vs. Verification Engineers
7.8 8.1 8.5
4.8
7.68.4
2007 2010 2012
~ 1-to-1 ratio of peak design and verification
engineers
58% 11%
4% 5%
Verification Engineers
Design Engineers
Source: Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study
WCR DVCon February 20135
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
SOC Verification Challenges Broaden
Verification Futures Conference
Infineon
nVidia
STMicro
Broadcom
ST Ericsson
TI
Wolfson
Dialog
Productivity/Performance � � � � � � � �
Debug � � � �
Verification Complexity � � � �
Coverage & Metrics � � � � �
Low Power Verification � � � � �
SoC & HW/SW Verification � � � � �
Verification IP & Reuse � � � �
Recruiting & Resources � � � �
Source: Mentor Analysis
WCR DVCon February 20136
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
On-Time Completion Remains ConstantNon-FPGA Project’s Schedule Completion Trends
0%
5%
10%
15%
20%
25%
30%
35%
More than 10%EARLY
ON-SCHEDULE 20% 40% >50% BEHINDSCHEDULE
Non-FPGA Study Participants
2007
2010
2012
2007: 67% behind schedule2010: 66% behind schedule2012: 67% behind schedule
Behind ScheduleAhead of schedule
Source: Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study
WCR DVCon February 20137
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
It’s an SoC World~80% of Designs Contain One or More Embedded Processors
21%22%
28%
6%
10% 13%
0%
10%
20%
30%
40%
50%
60%
NONE 1 2 3 4 5 or MORE
Non-FPGA Study Participants
Number of Embedded Processors for Non-FPGA Designs
2004
2007
2010
2012
Source: Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study
WCR DVCon February 20138
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
Embedded Processors per Design Size
0.91
1.86
2.67
0
1
2
3
< 5M 5 - 20M > 20M
Average Number of Embedded
Processors
2012 Design Size (Gate Count Excluding Memories)
Source: Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study
WCR DVCon February 20139
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
MACRO TREND #1:
Standardization of Verification Languages
0%
20%
40%
60%
80%
VHDL Verilog Synopsys Vera System C SystemVerilog Specman e C/C++ OTHERTestbench
Non-FPGA Study Participants
Languages Used for Verification (testbenches)
2007
2010
2012
SystemVerilog grew 8.3% between 2010 and 2012
Source: Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study
WCR DVCon February 201310
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
SystemVerilog Adoption by Design Size
59%
71%
89%
0%
20%
40%
60%
80%
100%
< 5M 5 - 20M > 20M
Non-FPGA Study Participants
SystemVerilog Adoption by Design Size (Gate Count Excluding Memories)
Source: Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study
WCR DVCon February 201311
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
Standardization in Base Class Libraries
0%
10%
20%
30%
40%
AccelleraUVM
OVM MentorAVM
SynopsysVMM
SynopsysRVM
CadenceeRM
CadenceURM
Other
Non-FPGA Study Participants
Testbench Methodologies and Base-Class Libraries
2010
2012
486% UVM growth between 2010 and 201246% UVM projected growth in the next twelve months
Source: Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study
WCR DVCon February 201312
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
MACRO TREND #2:
Standardization of the SoC Verification Flow
� Emerging from ad hoc approaches to systematic processes
Unit Level ConnectivityIP
Integration /Data Path
System Level
WCR DVCon February 201313
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
Aspects of SoC Verification
Block-Level
Verification
Interconnect
Verification
Datapath
Verification
Application/SW
Verification
•Interface-centric•Test under all conditions
•Interconnect is a block unto itself•1000s of connections to check
•SW-driven•Register access•Ensure memory map is correct
•Basic power control
•Couple SW with external stimuli
•CDC•Power control adds levels of complexity
Avoid the “HW works, SW works,
System doesn’t” problem
DVCon 2013 Tutorial - We've Got You Covered14
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
The Verification Paradox
� A good verification process lets you get the most out of best-in-class verification tools
Start ToolsAd Hoc
Processes6-9%
Cost Increase
Start Process Tools20-30%
Cost Savings
Source: Cisco Momentum Research Group
WCR DVCon February 201315
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
Unit-Level Checking
� Goal: Verify the functionality is correct for each IP while achieving high coverage
� Traditional techniques used include
Unit Level ConnectivityIP
Integration /Data Path
System Level
— Simulation with directed, constrained-random, graph-based testbenches— Lots of verification IP (VIP)— Coverage techniques— Assertion-based verification— Formal property checking
WCR DVCon February 201316
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
� GOAL: Verify the functionality is correct for each IP while achieving high coverage
� Goal: Ensure IP blocks are connected correctly
Connectivity Checking
Unit Level ConnectivityIP
Integration /Data Path
System Level
— Verify full connectivity, not simply point-to-point connections— Validate numerous configurations and operation modes
WCR DVCon February 201317
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
IP Integration and Data Path Checking
Unit Level ConnectivityIP
Integration /Data Path
System Level
� Goal: Ensure IP blocks are integrated correctly
— Access all memories— Access all registers, such as control— Verify multiple clock domain crossings
WCR DVCon February 201318
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
� Goal: Performance, power analysis and SoC functionality
System-Level Checking
� SOC ‘features’ that need to be verified
Unit Level ConnectivityIP
Integration /Data Path
System Level
— Verify functional interaction between multiple IP blocks—firmware, middleware, application focused testing
— Requires all IPs to be integrated at this level, including software
— Power control — Functional use-cases to stress interacting IP components— System performance testing— Firmware and other software for signoff
WCR DVCon February 201319
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
MACRO TREND #3: Coverage and Power across All Aspects of Verification
� What gets measured, gets done
� What gets measured, gets improved
� What gets measured, gets managed
– Peter Drucker
WCR DVCon February 201320
www.mentor.com© 2012 Mentor Graphics Corp. Company Confidential
Coverage IntroductionWhat are the Types of Coverage?
Coverage Closure with the Questa Platform21
� Origin of Source
— Specification
— Implementation
� Method of Creation
— Explicit
— Implicit
Explicit
Specification
Implicit
Implementation
Functional
Code
Assertion
Input Output
Protocol
FSM
Power
State
Clock
Domain
Toggle
Branch
Statement
Line
www.mentor.com© 2012 Mentor Graphics Corp. Company Confidential
Coverage IntroductionFundamental Metric Across the Entire Verification Process
22
� Verification Planning— Requirements Mapping— Coverage Planning
� Testbench Creation— Coverage Modeling— Stimulus & Environment Creation
� Simulation-Based Techniques— Dynamic Verification
� Formal-Based Techniques— Static Verification
� Analysis & Reporting— Analyzing— Ranking & Merging— Reporting
ANALYSIS &REPORTING
SIMULATIONBASEDTECHNIQUES
FORMALBASED
TECHNIQUES
TESTBENCHCREATION
VERIFICATIONPLANNING
PLANNING
MODELING
TARGETING
ACHIEVING MEASURING
ANALYZING
RANKING MERGING
REPORTING
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
Unified Coverage Interoperability Standard New Accellera UCIS Standard Announced at DAC 2012
UCIS API
Coverage Database
Simulation
Formal
Emulation
Analysis Testplan
WCR DVCon February 201323
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
Low Power Design Requires Multiple Verification Approaches
Design Functionality
Power Management Architecture
Power Domain Behavior
Power Domain Interactions
Power State Transitions
Power Control Hardware
Power Control Software
System with Power Management
Verify that design functions correctly with power always on
Verify that power management architecture is correct
Verify power up/down and reset/restore on power up for each block/power domains
Verify that interfaces are correctly isolated and level shifted
Verify that HW generates correct control signals in correct sequence
Verify SW Power Control interfaces with HW is correct
Verify full system with HW and SWfor power control
Verify that all transitions are covered and behave correctly with power control
WCR DVCon February 201324
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
Trends in Power Management Verification
0% 20% 40% 60% 80%
Power domain powerdown/power up
Power domain statereset/restoration
Transitions between systempower states
Hardware power controlsequence generation
Interactions between power domains
Operation in each system power state
Application-level power management
Hypervisor/OS control of powermanagement
Aspects of Non-FPGA Power Managed Design That Are Verified
67% of the industry actively manages power
Source: Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study
WCR DVCon February 201325
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
Macro Enablers in VerificationDelivering Order-of-Magnitude Improvements
Intelligent test bench
Multi-engine verification platforms
Application-specific formal
WCR DVCon February 201326
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
Macro Enablers in VerificationDelivering Order-of-Magnitude Improvements
Intelligent test bench
Multi-engine verification platforms
Application-specific formal
WCR DVCon February 201327
www.mentor.com© 2011 Mentor Graphics Corp. Company Confidential
Time
Coverage
Functional Coverage Achieved vs Project Time
� Accelerates Coverage Closure— No wasted stimulus
— Faster ramp to coverage goals
— Fewer simulation resources
� Multiplies Existing Tests— Re-use stimulus model
— Expand coverage model
— Generate even more tests
— Added tests will cover more
CRTDT
iTBA
10X Faster
Questa UltraIntelligent Testbench Automation Benefits
28 Questa Platform - 100X Faster Functional Coverage Closure
CRTDT
iTBA
Time
Coverage
Functional Coverage Achieved vs Project Time
10X Faster
100X More Tests
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
Intelligent Test Bench Technology Accelerates Coverage Closure
OfficeOfficeProducts 3 Days
36 hours on 6 CPUs36 hours on 6 CPUs8 weeks on 6 CPUs8 weeks on 6 CPUs100% coverage100% coverage60% coverage
37 X faster37 X faster+ 40% coverage+ 40% coverage
���� ==
WirelessWirelessNetworking 35 Hours
48 CPU hours48 CPU hours3175 CPU hours3175 CPU hours97% coverage97% coverage95% coverage
66 X faster66 X faster+ 2 % coverage+ 2 % coverage
���� ==
ConsumerConsumerElectronics 1 Day
118 minutes118 minutes>18 hours100% coverage100% coverage100% coverage
���� ==
Storage & Networking 1 Week196,000 tests196,000 tests26,315,000 tests26,315,000 tests100% coverage100% coverage79% coverage
���� ==
BasestationBasestationTelecom unknown
75,000 vectors75,000 vectors825,000 vectors825,000 vectors100% coverage100% coverage100% coverage
10 X faster10 X fasterEqual coverageEqual coverage
���� ==
IndustryConversion
TimeConversion
TimePre iTB Results Results Benefits
SwitchingSwitchingSubsystems 2 Days
400,000 tests400,000 tests10,000 tests 40 X more tests40 X more tests���� ==
WirelessWirelessTelecom 6 Days
45 minutes45 minutes3 days3 days100% coverage100% coverage100% coverage
27 X faster27 X fasterEqual coverageEqual coverage
���� ==
Processors 1 Day30 minutes30 minutes5 hours5 hours
100% coverage100% coverage100% coverage10 X faster10 X faster
Equal coverageEqual coverage���� ==
37 X faster
10 X faster
66 X faster
+ 40% coverage
40 X more tests
27 X faster
10 X faster
Average customer adopting inFactincreased performance by more
than 40X
9.5 X faster9.5 X fasterEqual coverageEqual coverage9.5 X faster
170 X faster170 X faster+ 21% coverage+ 21% coverage170 X faster
+ 21% coverage
WCR DVCon February 201329
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
Macro Enablers in VerificationDelivering Order-of-Magnitude Improvements
Intelligent test bench
Multi-engine verification platforms
Application-specific formal
WCR DVCon February 201330
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
System-Level Checking
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
Hardware/software
co-verification
HW Acceleration &
Emulation
FPGA prototyping Prototyping with
special test chips
No
n-F
PG
A S
tud
y P
art
icip
an
ts
2007
2010
2012
Source: Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study
WCR DVCon February 201331
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
As Design Sizes IncreaseEmulation Up, FPGA Prototyping Down
18%
32%
50%
45%
57%
28%
0%
10%
20%
30%
40%
50%
60%
HW Acceleration/Emulation FPGA Prototyping
Non-FPGA Study Participants
Adoption by design size forHW acceleration/emulation and FPGA Prototyping
< 5M
5 - 20M
> 20M
Source: Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study
WCR DVCon February 201332
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
Design under Test
OVM/UVMSystemC/C++
Intelligent Test
TestbenchAcceleration
Assertions & Checkers
Integrated Verification EnvironmentsSimulation/Formal/Emulation/Software Verification Environments Emerge
WCR DVCon February 201333
Clocks, Connectivity,Coverage
Formal Verification
Virtual Prototype
Offline
JTAG
SW Debug
System Level
Virtual Devices & Transactors
Protocol Solutions
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
Macro Enablers in VerificationDelivering Order-of-Magnitude Improvements
Intelligent test bench
Multi-engine verification platforms
Application-specific formal
WCR DVCon February 201334
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
The Evolution of Formal TechnologyNot Just for Experts Anymore
FormalPropertyChecking
FullyAutomaticFormal
AutomatedApplications
LowEffort
HighEffort
FormalExperts
Everyone
1990s Today2000s
WCR DVCon February 201335
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
Larger Designs Use More Formal
20%
26%
41%
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
< 5M 5 - 20M > 20M
Non-FPGA Study Participants
Formal Property Checking Adoption by Design Size (Gate Count Excluding Memories)
Source: Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study
WCR DVCon February 201336
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
Improve Simulation Coverage by Identifying
Unreachable Code
Identify Full ChipSoC IP Connectivity
Errors
Identify Deadlockand FSM Design
Issues
Today’s Automated Push-Button Formal
Automatic Formal
Applications
Automatic Formal
Applications
Automatic Formal
Applications
IdentifyX-Propagation
RTL Issues
ApplicationSpecificFormal
WCR DVCon February 201337
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
Property Checking
Automated Applications
Fully Automatic
Application-Specific Formal Includes Checking Clock Domain Crossings
� Identify metastability issues for signals crossing multiple
clock domains due to improper synchronization
� Now scales to the full chip level
Clock Domain A Clock Domain B
Synchronizer
WCR DVCon February 201338
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
Average Number of Clock Domains by Design Size
5.2
9.4
11.5
0
2
4
6
8
10
12
14
< 5M 5 - 20M > 20M
Average Number of Clock Domains
Average Number of Clock Domains by Design Size (Gates Excluding Memories)
Source: Wilson Research Group and Mentor Graphics, 2012 Functional Verification Study
WCR DVCon February 201339
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
Accelerating Innovation
Macro trends in verification
Macro enablers in verification
— Standardization of verification languages— Verification flow standardization— Coverage and power across all aspects of verification
— Intelligent test bench— Multi-engine verification platforms— Application-specific formal
WCR DVCon February 201340
© 2013 Mentor Graphics Corp.
www.mentor.com
The Verification Academy
HF - February 2013, DVCon Lunch
10,000 Member Milestone in May 2012!
Today Over 15,000 Members Registered!
© 2013 Mentor Graphics Corp.
www.mentor.com
Verification Academy(1) Online Verification Courses
� AMS
� Assertion-Based Verification
� CDC Verification
� Evolving Capabilities
� FPGA Verification
� OVM Basic & Advanced
� Testbench Acceleration
� Intelligent Testbenches
� UVM Basic & Advanced
� UVM Express & Connect
� Verification Planning
� VHDL 2008HF - February 2013, DVCon Lunch42
© 2013 Mentor Graphics Corp.
www.mentor.com
Verification Academy(3) Methodology Cookbooks
� The UVM Cookbook is the most comprehensive resource
available on UVM today!
� The Coverage Cookbook is
the first of its kind!
UVM Cookbook
Coverage Cookbook
© 2013 Mentor Graphics Corp.
www.mentor.com
Verification Academy
HF - February 2013, DVCon Lunch
www.verificationacademy.com
http://blogs.mentor.com/verificationhorizons/
www.mentor.com© 2013 Mentor Graphics Corp. Company Confidential
Top Related