Test Automation for IoT solutions: A Paradigm shift

NRESH REDDY NATHALA – SOLUTION ARCHITECT

Co – Author 1 –Anil Reddy Tatikonda, Test Architect Co – Author 2 –Narasimha Reddy Gujjula, Test Lead

Ideabytes Inc.

AbstractThe internet of Things (IoT) is taking over the world with its promise to connect and "smartify" everyday objects, devices, machines, vehicles etc. with help of sensors, connectivity, cloud, software applications and mobile Apps.

The IoT applications exist in a variety of verticals like Logistics, Health Care, Agriculture, Manufacturing, Retail, and Smart Cities etc.

IoT test automation is challenging the traditional test life cycle models and test automation tools for web and mobile apps, including the current well proven automation frameworks. The reasons are : IoT demands end to end  automation which  involves Devices , Servers , Network connectivity and  real world  events that are triggered asynchronously without any interdependencies. Additionally, testing of IoT applications involves testing of multiple devices and beyond, entering into the real world where we need to test the integration of devices with real time users and Cloud Software applications combined with unreliable network infrastructure like LAN , Wi-Fi GPRS , LoRA  ,ZigBee etc.

This involves lot of asynchronous events, making it difficult for the tester to automate. Challenge involves generation of events in real time and information exchange over different interfaces. So it requires test infrastructure to control multiple test tools and data elements, randomly generating physical world events.

Our Automation Solution brings a paradigm shift, considering these challenges, and presents test approaches and strategies with multiple devices in an Asynchronous environment using our custom test automation framework TSE (Test Suite Express - Ideabytes innovative automation hub)

Introduction to IOTThe Internet of Things (IOT) is all about connecting multiple devices over the network and enableing them to exchange dData between the devices and also between devices and humans to Human and use this data to automate human actions making the life simpler, secure and predictable.

We are soon reaching a phaseIn future, where all the devices we use in day to day may would be connected over network making the interactions and data exchange more complex to develop and at the same time increasinge the benefits to the humans.

A small example on how devices can be interconnected will be the devices and also beneficial- A at the same time.

Assume person travelling on a bike met meets with an accident. Bike immediately notifies to the nearest help center about the accident and the location. Help center automatically notifies the aAmbulance with the location of accident and the nearest hospital. All the Traffic signals on the route to the hospital automatically be turn green till until the ambulance passes. Hospital would be notified of the patient arrival so that the staff would be alert and prepared aanmd also . At the same time, Ppatient’s family would be notified.

Imagine, for all this to happen, in fraction of seconds, there would be many interactions and transfer of information between multiple devices over the internet. This would be IOT in future.

Gartner estimates there will be 50 billion connected devices by 2020

Testing in IOTTo test the IOT applications we have to test all the devices in the Application architecture, communication layer being used to transfer the data, Servers used to Store and analyze the data and the UI interface which user useds for viewing the data and the analysis reports. The architecture couldThis might also involve an admin a common Administration portal to monitor all the devices.

Let’s take an example of a Health Care for elderly and Smart Home IOT application. I, in this there are we will be having devices to monitor visitors to homes using mMotion s Sensors, dDoor sSensors, cCameras, and Light bBulbs. Smart Buttons when clicked, directly notify family Members, WIFI speaker and smart TV speakers to to give reminders on mMedication to be consumed or any other messages or share Pictures from fFamily members d’. Devices to share information like Photos.

To configure and monitor such environment, mobile apps and admin portals will be in operation and also To Monitor all this we will be having Mobile apps, Admin portals and devices such as Smart TV for display of shared information and messages. In these cases,

information exchange would be using WIFI, ZIGBEE protocols and the And all the data is stored in Cloud Server

Communication layer involves WIFI, ZigBee protocols.(this sentence is hanging?)

(Here we will be inserting picture).

To Now to test this whole kind of architecture, it is essential to test individual devices, the communication layer, User Interface components, Gateways and Cloud Server.we have to test the individual devices, Communication layer, UI components, gate ways and Cloud server.

After completion of unit testing of each component, integration testing is conducted. This is the most critical phase. Once we test all these are working fine individually, then we start the integration testing which would be the most critical part. Identification of scenarios for integration testing should not be in the traditional way but this should be from end user’s perspective (Think like a User and Test like a User). In the real world, events generated from all this these devices would not follow any pattern or a standard flow. All are asynchronous events and testers should not be trained to disengage from be working with a the orthodox

testing mindset of Input ->Processing-> step and expected result in a typical sequential step wise operations.

ChallengesAs is evident from the observations made so far in this Paper, iIdentification of Uuse Ccases is always a big challenge in IOT architecture. as iIt involves taking into consideration, multiple devices that trigger events without any syncsynchronisationsynchronization, thus posing a challenge making it difficult forto the tester to in identifying a starting point and subsequent flow of the events.

As the events are asynchronous, tester has to think in all the possible directions and come up with permutation of the events but not restrict with combination of events. This should also combine with the use cases of what would happen if the network is down or delayed.

As the events are asynchronous, it is essential that tester should generate exhaustive Permutations of the events and not be restricted to only Combinations. Also, the Test Plan should include failure Use Cases such as delays over the network, network failure etc.

In order to replicate the identified scenarios in the real world, where the events can be reproduced, IoT lab needs to be set up. Once the use cases are defined and then to replicate them in the real world, testers have to set up IOT labs and reproduce the events.

Another challenge is adopting only manual testing. In case of IoT application testing, many real-world Use Cases would be excluded if only manual testing is carried out. This results in end user reporting many bugs, that tarnishes the credibility of the Solution provider, thus placing the Product marketability at a great risk.

To consider manual testing here, lot of use cases which can happen in the real world would definitely be missed and lot of bugs would be reported by the end user which would doom the entire product and the credibility of the Solution provider making it a very greater risk.

Our Solution(Maintain continuity – challenges are already covered in the previous section) In Traditional testing as the Complexity grows and predictability declines, testers stick to Manual testing abandoning Automation in testing.

To address the challenges, alternate ways need to be identified. But here, with all the challenges in manual testing we cannot go forward with it but need to identify alternate ways. Here comes the Automation testing to our rescue for the unpredictable Uuse Ccases/Eevents of IOT applications.

Our Solution TEST SUITE EXPRESS – AUTOMATION HUB has the orchestration capabilities to interact handle dissimilar SUTs, with multiple devices using the API’s and Mobile devices and Web portals using various leading testing automation tools.

Firstly, individual Methods are developed to trigger each possible event using the device API’s. Here developer’s involvement would be crucial, as most of this activity would already be covered during development Phase.

Libraries providing system functions to stimulate various network conditions, interactions with all the Mobile devices and web portals integrated into the solution

(See how you want to change these 3 paragraphs, you mentioned points that would also get into the ppt)

Testing process first involves listing out all the possible events from each device by going through the API’s of the devices. Methods to trigger each event from the API’s are designed. These methods mostly come from the developers as this would be part of the solution development.

Library methods are developed to vary the network conditions over the period of time.

Library methods to capture the messages to the end user on the Mobile devices working on Android, iOS and Web portals are all developed and integrated into the Solution.

Our custom randomizer interfaces with the external devices using APIs and triggers events by calling the respective methods. We have developed a randomizer to call the multiple devices using the API’s and trigger any event by calling respective method. This would trigger Asynchronous events and these eventswhich are sent input to the IOT eco system that would start working as per the rules.

Results would be captured after every cycle of events.

Results captured after every cycle are displayed to the tester on the UI with all the sequence of events happened and the outcome. Results can be analysedanalyzed and defects raised in bug tracking tool, with a click of a button. There is aAlso with a provision Now the tester can analyze the results and based on the analysis with in a click of a button defect would be raised in Bug Tracking tool and theof automatic defect notification to the Product owner and the Ddeveloper

would be notified of the failed use case with the steps to replicate.

(Here will loop an image)

Summary and ConclusionIOT has been around now for some years and now becoming a very rapid developing market. As per the various reports, a minimum of 50 billion devices would be connected by 2020. With traditional testing approaches almost failing in most of the cases, there is a need for a we have to come up with new dimension in testing to address the needs of IOT testing. Automation previously thought of as robot to act upon as per the directives, now has to be used to make the unpredictable into predictable. Our Solution with Randomizer to trigger asynchronous events and Hub to interact with multiple devices at the same time is an attempt to address the thiscurrent need. Automation, previously thought of as robot to act as per the directives, now has to be used to make the unpredictable into predictable. At the same time, with complexities would only increase day to day solutions needs to be constantly enhanced to make it ready for testing the future.And with rising complexities in the day to day, automation solutions too need to be constantly enhanced to be ready for testing the future.

References & Appendix

http:// www.gartner.com/newsroom/id/2636073

http:// ieeexplore.ieee.org/xpl/RecentIssue.jsp? reload=true&punumber=6488907

https:// en.wikipedia.org/wiki/Internet_of_things

https://nordicapis.com/automated-testing-for-the-internet-of-things /

Wikipedia - IOT

Author Biography

Naresh Nathala works as a Solution Architect at Ideabytes Software India Pvt Ltd, Hyderabad and has 8 years of experience in designing and creating Test Architecture, Integrated Software solutions in Large scale domains with strong expertise in IoT domain. He has unique experience in using multiple Automation tools and evolving QA Strategies in automating Integration testing involving multiple devices, Sensors, Cloud Software and User interface Applications.

THANK YOU!