Testing Strategies for Asymmetric Environments · Use techniques such as testing stubs, using...
Transcript of Testing Strategies for Asymmetric Environments · Use techniques such as testing stubs, using...
V1.0 | 2019-05-29
Jeffrey Fortin, Product Manager, Code Testing
Testing Strategies for Asymmetric Environments
2
1. Asymetric Environments
2. Testing in an Asymetric Environment
3. Summary
4. Questions
Agenda
3
Software Based Systems are Evolving RapidlyAsymetric Environments
• Hundreds of files• Thousands of Lines of Code• Lots of “connections”• Lots of safety concerns
SOFTWAREEMBEDDED SOFTWARE
+SIZE
ENTERPRISESOFTWARE
+Connections
• Thousands of files• Millions of Lines of Code• Few “connections”• Few safety concerns
+Safety
4
Today’s challenge: Meet quality objectives while controlling costAsymetric Environments
5
Today’s challenge: Meet quality objectives while controlling costAsymetric Environments
Code size expandingSame regulatory requirementsSame or higher quality expectationsSame budgets and schedules
A strategy is needed …
6
Environments and DevOpsAsymetric Environments
Development• New Code• Unit Tests• Static Analysis• Code Coverage criteria met
• Development Host
Integration
• Latest code base• Updated continuously
QA• Integration Tests• Regression Tests• Updated Usually Overnight
Staging
• Simulate the production environment
• Perform tests that can’t be done sooner
Production• End user is now your tester
• Bug reports• External monitoring
7
1. Asymetric Environments
2. Testing in an Asymetric Environment
3. Summary
4. Questions
Agenda
8
Multiple ConfigurationsTesting in an Asymetric Environment
Development• Windows Host• Microsoft Visual Studio
Integration• Linux Build Server• GCC Compiler
QA• Docker• Simulator• Cross Compiler
Staging• Sim/Stim• HIL• Signal Generators• Network Simulation• Reference Hardware
Production• Device Management
• Embedded “Target”
9
Typical Embedded Code ExampleTesting in an Asymetric Environment
typedef unsigned long uint32_t;/* Memory Mapped Hardware Registers with *special* properties */uint32_t volatile REG1;static inline uint32_t getREG1();
/******************************************************************************************//* take action based on the value in REG1 and REG2 */int avoid_collision() {
if ((REG1 & 0xFF00) & 0xFE00) {return 1;
}return 0;
}
Can the code logic be tested early?
10
Embedded Code Example Refactored for TestingTesting in an Asymetric Environment
/******************************************************************************************//* Better implementation that allows for stubbing of the special registers */inline uint32_t getREG1() {
/* Using *inline* should give same performance as macros or inline register access */return REG1;
}int avoid_collision_api() {
/* Notice calls through stub-able accessor functions */if ((getREG1() & 0xFF00) & 0xFE00) {
return 1;}return 0;
}
typedef unsigned long uint32_t;/* Memory Mapped Hardware Registers with *special* properties */uint32_t volatile REG1;static inline uint32_t getREG1();
11
Simulating Hardware StatesTesting in an Asymetric Environment
Temperature is a 2’s compliment binary
Using a stub routine, we can easily test for any temperature setting
We can also test for “impossible” bit combinations
12
Injecting FaultsTesting in an Asymetric Environment
Bits 26-30 represent fault conditions
Testing with stubs we can stimulate the code to respond to the fault
We can even set multiple fault conditions
13
Call to getHW_TEMP_STATUS()
Testing ExampleTesting in an Asymetric Environment
TestDriver
ModuleUnder
Test
DummyFunctions
(stubs)
RealFunctions
14
Using Test Automation in the Development EnvironmentTesting in an Asymetric Environment
Development Environment is resource richThe automation system can cycle through
hundreds of test parameters and re-run tests on any code that has changed
Tests can be setup to run on command by anyone
15
Collecting Testing Metrics in the Development EnvironmentTesting in an Asymetric Environment
Get early indication of quality risksTesting coverage is a key metricOther metrics help to make better decisions
16
1. Asymetric Environments
2. Testing in an Asymetric Environment
3. Summary
4. Questions
Agenda
17
Leverage the Development Environment as much as possible
Use techniques such as testing stubs, using multiple compiler configurations and test automation
By doing so, embedded applications can be developed more efficiently and at lower risk than by only testing in the final Production Environment.
ConclusionSummary
18
1. Asymetric Environments
2. Testing in an Asymetric Environment
3. Summary
4. Questions
Agenda
19
Testing Strategies for Asymmetric EnvironmentsQuestions
!?
20 © 2019. Vector Informatik GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.0 | 2019-05-29
Author:Fortin, JeffreyVector
For more information about Vectorand our products please visit
www.vector.com