Continuous Integration Testing for - TMCnet · PDF file Continuous Integration Testing for IoT...

Click here to load reader

  • date post

  • Category


  • view

  • download


Embed Size (px)

Transcript of Continuous Integration Testing for - TMCnet · PDF file Continuous Integration Testing for IoT...

  • Continuous Integration Testing for IoT Endpoint Integrity

    Arthur “Code Curmudgeon” Hicken


  • Arthur Hicken is Chief Evangelist at Parasoft where he has been involved in automating various software development and testing practices for over 20 years. He has worked on projects including cybersecurity, database development, the software development lifecycle, web publishing and monitoring, and integration with legacy systems and maintains the IoT Hall-of-Shame

    Follow him @codecurmudgeon



  • Agenda

    IoT specifics

    Challenges in quality assurance

    Effectiveness of end-to-end testing

    Doing it better

  • Things are everywhere

    Industrial Automation Smart Health

    Smart Home Smart City

  • Things are vulnerable The IoT Hall-of-shame

  • • Communication • Backbone • Identification • Hardware • Sensors • Protocols • Software • Security • Machine Learning • Data Brokers / Cloud Platforms

    Aspects of IoT

  • IoT specifics • Multi-layered systems

    • Disparate technologies

    • Functionality

    spanning across many


  • Why it is difficult to test IoT • Disparate development technologies hamper

    introducing consistent testing strategies

    • “Not enough code to justify testing” syndrome

    • Limited user interface make the solution to look deceptively simple

    • Inverted testing pyramid problem – Strong temptation to rely entirely on system testing

  • Effectiveness of end-to-end testing

    • Labor intensive

    • Difficult to automate

    • Difficult to start early

    • Requires inter-team synchronization

    • Difficult to test error handlers

    • Failed tests do not provide precise information about failure points

  • Effectiveness of end-to-end testing

    • End-to-End testing for diabetes health system example

    • Difficult to apply stimulation

    • Difficult to validate test result

    • Difficult to test error handlers

  • Effective IoT Testing • Avoid manual end-to-end testing • Invest more into automated tests at component • Follow testing pyramid rules • Measure:

    – test results – test effectiveness

    • Determine: – Risk

  • How to do it better ?

    • Give absolute priority to automated tests

    • Invest time into designing interfaces (API)

    • Use API (service) testing tools to cover interfaces

    • Measure the quality of the test

  • How to do it better ?

    • Use stubs and mocks to improve:

    – Facilitate test automation

    – Increase code penetration

  • How to do it better ?

    • service virtualization to improve test automation

    – Isolate at the message layer

    – Simulate functional scenarios & performance

  • Test scenario example


    Tested System

    Tested System

  • Elements of Continuous Testing

    Continuous Testing

    Risk Assessment

    Policy Analysis

    Requirements Traceability

    Test Environment


    Test Optimization

    Advanced Analysis

    Ensure access to complete test environments

    Automate defect prevention and Policy measurement

    Expand test coverage and measure test effectiveness

    Define actionable practices

    Connect functional with non-functional requirements

    Process improvement opportunities

  • Feature Release



    Customer Support

    Beyond Automation

    Software Feature

    Continuous measurements mean continuous refinement of the process

    Real-time feedback from

    Objective assessment and go/no go Defects are eliminated at the

    point that they are easiest to fix



    Developer Customer Support

    Quality gates: Organizations can automatically promote software through quality gates when business expectations have been met

  • Summary

    IoT adds complexity - It’s not just a device

    Stubs/mocks/virtualization enables testing

    Continuous testing catches it before the user

  • Q&A

    • Arthur Hicken: – Email: [email protected]

    – Web:

    – Twitter: @CodeCurmudgeon

    – Facebook:

    – Blog:

    – IoT Hall-of-Shame:

    mailto:[email protected]