Agile testing for embedded software development

© COPYRIGHT 2016 COVEROS, INC. ALL RIGHTS RESERVED. 1

Agility. Security. Delivered.

Agile Testing for Embedded and IoT Software Development


About Coveros

• Coveros builds security-critical applications using agile methods.

• Coveros Services• Agile transformations• Agile development and testing• DevOps and continuous integration• Application security analysis

• Agile & Security training

• Government qualifications• DCAA approved rates and accounting• TS facility clearance

Areas of Expertise


What is embedded software

• Embedded software is software that controls a device

• A device is a combination of software, firmware and hardware that make up a system

• The system has hardware components that are either sensors or manipulators

• The software and firmware control the hardware components, using the sensors to discover and the manipulators to carry out actions

• Examples include:• Medical devices• Self driving cars• Fitness wearables• HVAC systems


What is agile testing

• Agile testing is the the set of practices used to build quality in and make sure that software created by the team is of high quality

• The whole team is responsible for quality

• Test automation is integrated into the Continuous Integration and Continuous Delivery pipeline

• Testing of new functionality is conducted during the same sprint or iteration in which the functionality is developed

• Unit testing is used as part of the quality assurance process

• Acceptance testing is conducted using acceptance criteria created directly from the requirements


User Acceptance Testing (UAT)

• UAT is the process of getting stakeholders to write down what results they want to see from the software in order to determine it works as the requirements intended

• Acceptance criteria are written before development of the software begins

• Acceptance criteria are a form of requirements that are used to let the team know what key aspects of the requirements need to be met in order for the stakeholders to feel that they can accept the software increment

• Testing of acceptance criteria happens at the end of an increment and gives the team feedback on how well they have created software that the stakeholders expected.


Behavior Driven Development (BDD)

• BDD is a test automation process for testing software based on the requirements

• The tests are written before the code is written or at least independently of the code

• For some teams BDD is used to automate UAT tests

• Many BDD tools exist such as Cucumber and Behave

• Most tools use the Gherkin language, which uses the following format:• Given initial condition• When some action or trigger• Then resulting action


Test Driven Development (TDD)

• TDD, also called Test First Development, is the process of using unit tests to help design the code for an application

• The developer will write a test of what they intend the code to do and then write the code to make the test pass

• Red, Green, Refactor cycle

• This has a number of positive results:• Makes the code more testable

• Makes the code more modular or self contained

• Makes the code more maintainable

• Gives the code a regression test suite

• Makes the intent of the code easier to understand


Continuous Integration (CI)

• CI is the process of integrating and testing code upon check in

• In CI, integration is the process of integrating a developer’s code with the code already in the repository

• The goal of CI is to make sure the code can build and pass an initial set of regression tests before other testing and quality assurance is applied to a specific build

• CI uses a build server such as Jenkins to do a clean build of the code and run a suite of unit tests

• If the build breaks, the people that checked in since the last successful build are informed and they are expected to get the build back to a working state


Continuous Delivery (CD)

• CD is the process of taking a build through a build pipeline that consists of a CI process followed by several quality gates that verify the code through various quality checks including:• Automated functional tests

• Static code analysis

• Automated non-functional tests like performance and security

• Automated deployment into progressively higher environments

• It could include on demand deployment to a manual test environment

• On demand deployment to a UAT environment

• On demand deployment to a demo environment

• Continuous Deployment is automatically progressing to production including automatically deploying into production


Value Stream Mapping

• Value stream mapping is the process of determining how a product is created in your environment from inception to deployment and use in production

• By creating a value stream map you can figure out how many steps you have in your delivery process and how long each step takes

• Once you have a map of your process you can start to optimize and automate the different steps to streamline your process

• The goal is to reduce the time it takes to move requirements through your process and get new features into production


Build Pipeline

• The build pipeline is the part of your value stream that takes checked in code and makes it into high quality, tested software that is either in production or ready to be distributed to customers

• CI/CD are a subset of your build pipeline

• Automation, including test automation, is the only way to make a build pipeline work in a fast and efficient manner

• Many of the quality gates in a build pipeline are testing focused and require strong testing acumen in order to help deliver high quality software


Version Control

• Version control is the process of managing the code over time by allowing a project to keep track of the state of the code as it progresses

• All code, including testing code and test automation code needs to be checked into version control

• Application code and testing code should be versioned together since the test code acts on the application and changes over time to keep in sync with changes to the application code

• Version control tools include:• Subversion

• Git

• Others


Configuration Management

• Configuration Management are the tools and process used to determine how software is setup in any given environment and how to account for all of the different files needed to make the software work

• This includes binaries, configuration files and environment settings

• Most build pipelines have a component that manages configuration elements for each deployment so that the team can know exactly what is deployed and tested for every environment


Testing Software vs. Hardware vs. System

• With embedded systems you are testing three things:• Software – The code that manages the system

• Hardware – The mechanical and/or electrical parts of the system

• System – The combination of hardware and software

• Sometimes firmware is tested like software, sometimes it is tested like hardware (depending on how firm it is)

• Isolating what you are testing can help you create test automation that covers a much of the product as possible

• Some amount of manual testing might be unavoidable


Software Testing

• During software testing you are testing only the software part of your system

• This means that all input (sensor data) should be supplied by a known source, if possible by recording the data and playing it back, or simulating it

• Output can be ignored or simulated

• Record and playback, or simulation can be less expensive and easier to setup

• You should be focused on testing the business logic of the software, i.e. the calculations and determinations that software makes

• You can also focus on testing the user interface or interactive elements of the system


Hardware Testing

• Hardware testing can be agile as well

• Hardware can have a different cadence than software

• Agile hardware testing is about automating as much of the testing process as possible

• Software tools, both homegrown and third party, can be used to automate hardware testing

• Using the system software the controls that hardware to test the hardware is acceptable but keep the focus on the hardware


System Testing

• System testing is the combination of the software and hardware working together

• Depending on the nature of your embedded system, this can be hard to automated or automate completely

• Depending on the nature of your embedded system there may be a good deal of tooling that can help you test the system or there may be nothing at all

• Testing the system is critical because you don’t know how the software and hardware will interact until you actually have them interact


Test Automation vs. Manual Testing

• Agile testing focuses on test automation and in some types of applications it is possible to automate nearly all of the testing

• With the focus on test automaton it is easy to think that manually testing has no place in agile but that isn’t true

• Manual testing is often a precursor to functional test automation

• Manual testing can be cheaper or more effective for some features of an embedded system

• Exploratory testing can exercise the system in a way that test automation will not and find deeper issues


How much can you automate?

• Look at this from a number of points of view:• What can you afford

• What do you have the skills or abilities to automate

• What is possible

• What is the most beneficial

• Finding a balance between test automation and manual testing will be a challenge that all projects face

• The answer will change over time as you learn and as your code improves


Coding Standards (for tests too)

• Using coding standards means:• Making the coding standards

• Enforcing coding standards using code reviews

• It doesn’t matter what specific coding standards you use so much as you have them and the team agrees to them

• If possible, use an existing coding standard for your technology

• Make coding standards compliance part of your code review

• Better yet, use a code formatter to apply the standard during code commit

• Apply coding standards to your tests as well


Code Inspections

• Code inspections include a number of things:• Code reviews

• Static code analysis

• Secure code reviews

• Format and style reviews

• Automate what you can, format and style are easy to automate

• Static code analysis can be a good input into code reviews

• Code review tools can help, things like Crucible and ReviewBoard

• A secure code review is different than a code review and is a different skill set, you should master both


Unit Testing

• Unit tests are developer tests that are used to make sure that the code that they write behaves as the developer intended

• Unit tests are written in the language and technology platform of the application code and often use a framework like Unity (see http://www.throwtheswitch.org/unity/)

• Unit tests might be run on the dev platform or the target platform, this often depends on how the tooling works

• Sample code:void test_FunctionWhichReturnsLocalVariable_ShouldReturnTheCurrentCounterValueAgain(void)

{TEST_ASSERT_EQUAL(0, FindFunction_WhichIsBroken(78));TEST_ASSERT_EQUAL_HEX(0x5a5a, FunctionWhichReturnsLocalVariable());

}


BDD Sample Code

Scenario: Subtract two numbersGiven I have a calculator onWhen I enter ”25" into the calculatorAnd I enter ”10" into the calculatorAnd I press subtractThen the result should be "15" on the screen

Scenario: Add two numbersGiven I have a calculator onWhen I enter ”25" into the calculatorAnd I enter ”45" into the calculatorAnd I press addThen the result should be ”70" on the screen


Mock Objects

• Mock objects are replacements for the dependencies of the code under test

• They are used to make it easier to test the code by responding the same way all the time

• There are a number of good mock object frameworks including CMock (see http://www.throwtheswitch.org/cmock/)


Test Fixtures

• Test fixtures are the code that creates a fixed state that is used to baseline tests so that the test environment is the same each time the tests are run

• Test frameworks do this by exposing methods or functions that allow for the configuration of dependent objects or for the creation of mock objects

• For a data related test a test fixture might load a specific set of data into the database

• Most unit testing frameworks and many test automation tools have test fixtures build into them


Working with Simulators

• A simulator is software or hardware used for testing that has the same interface as the application or system it is replacing

• Simulators provide a consistent interface for testing, much like test fixtures or mock objects

• Simulators are more full featured and can react to input, in more varied ways that other testing tools

• The idea is that a simulator will react more like the real component in the real world

• Simulators can be expensive to build and maintain but there may be no other option


Testing services for embedded software

• Many Internet of Things (IoT) devices use services

• Services are Internet accessible bits of functionality that do things like:• Collect data

• Calculate results

• Coordinate resources

• Testing these services can be easier to automate then the devices themselves

• Scale can be an issue for services

• Network availability or reliability can be an issue for the devices

• Performance testing is often important for testing services


What to take away from the presentation

• Test automation is the key to agile testing

• Do test automation planning while you do test planning

• Create test suites that test your software separately from your hardware and another suite that tests the system as a whole


Questions?

• Tom Stiehm

• [email protected]

• @thomasstiehm

mailto:[email protected]

Agile testing for embedded software development

Technology

Transcript of Agile testing for embedded software development