Testdroid: Which One Works You The Best: In-House or Cloud-Based Development Environment

8 January 2015

Ville-Veikko Helppi Technical Product Manager ville-veikko.helppi@bitbar.com

Which One Works You The Best: In-House or Cloud-Based Development Environment

W E B I N A R

Agenda

•  Comparison of In-house and Cloud-based Setups •  Investments in Mobile App Development and

Testing Environments •  The Characteristics to Use for Comparison of Both

Development Environments •  Calculation Examples •  Testdroid Update •  Q&A

Introduction to Creation of In-House Development Labs

Picture from #GTAC2014

Why Automation?

Smaller coverage, More money burnt & ?me wasted, Error-‐prone

Large coverage, quickly

completed, Less money & ?me wasted, Exact

results.

Manual Automa@on

•  Why Do You Need an In-House Development and Test Lab? –  Pre-market devices (OEMs, ODMs,

integrators ...) –  Prototypes / parts of hardware –  Corporate security policies –  Availability and high usage level –  Special setups (integration w/other

equipments) –  Large set of devices available

In-house Dev&Test Labs

•  What is Required for an In-House Development & Test Lab? –  Server hardware / infrastructure –  Own mobile devices –  Device control servers

•  Hard Disks, RAM, Energy management

–  USB hubs –  WiFi / Network infrastructure –  Continuous Integration/Delivery/Testing –  Build systems / other software tools –  POWER!

Continuous Integration

•  Version control and build revision •  Build automation, instant execution&testing •  All types of testing & integrations enabled •  Frequent commits, Code consolidation ->

Fast builds •  CI integrated with development and testing

provides the best fit for mobile agile process

Core Benefits of CI

•  How to Reach Enterprise-Grade Reliability for In-House Dev&Test Lab – Device management

•  Always run on clean devices •  Create intelligent retry mechanisms •  Create fall-back mechanisms

– System management •  Automatic reconnect of network and USB •  Automatic configuration changes •  Professional-level monitoring system

– Diskspace, network/connections, Power

Issues?

29% of problems are tightly related to devops of an

in-house system

Introduction to Adoption of Cloud-based Development Labs

•  Someone else is taking care of heavy-lifting •  Cloud-based device cloud includes ALL

relevant devices (phones, tablets, cams...) •  You get everything automatically delivered – Data, Results, Logs, Screenshots, Pass/Fail

Reports, Additional data-sets

•  Diversity and Keeping Up-to-Date with SW – New OS versions and OEM updates

•  Easiness of Adoption and Integration

Why Cloud?

Access via Comprehensive API

•  Authentication •  Project Management – Create, Delete, Configure, Managing Users

•  Configuring Access and Assets – Configs, Upload/Download Files (App & Tests) – Device Access & Device Group Configuration

•  Running Tests – Upload App + Tests <-> Download Results/Data – Notifications, Sharing, Pre-processing of Data

Access via Comprehensive API

Investments in Mobile App Development Environment

Investments Aspects

•  What is the Most Cost- Efficient Approach?

•  What is the Total Cost of Ownership?

•  Capital Expenses (CapEx)

•  Operation Expenses (OpEx)

•  CapEx is a business expense incurred to create future benefits or to add to the value of an existing asset

•  OpEx refers to the expenses that the mobile app developer spends in order to get their day-to-day activities going on and to get their infrastructure fully working

CapEx vs. OpEx

•  CapEx: Devices, Servers, Infrastructure •  OpEx: Running & Maintaining the System

Costs: In-House Dev&Test Lab

•  Cost of Service includes all major expenses (both CapEx and OpEx)

Costs: Cloud-based Dev&Test Lab

10 Different Characteristics for Comparing Environments

•  A

Different Characteristics & Benefits

•  Corporate Policies •  Regulation •  Minimizing security

breaches •  Security/Protection

mechanisms •  Monitoring •  Authentication &

User Access

Security 1

•  Needs to be fully configurable

•  Back-ups & Retention •  Preventing non-

authorized use and access •  Needs to available 24/7 •  Transferrable nature of

data, source code, results

Data Ownership and Retention 2

In-house vs.

•  API access – no delays •  Critical/Medium/Low •  DevOps in crucial role •  Various data center

locations •  Hardware failures?

Monitoring? Scale up? •  Average Cloud Up-time:

Performance and Availability 3

•  Support: Email, Phone, Online service systems

•  Intuitive online help: Manuals, How-Tos, Instructions (self-service)

•  Integration with bug-tracking systems (e.g. JIRA)

•  Frequent product updates •  Needs to be available 24/7

Support 4

•  Clear policies on updates, maintenance, monitoring

•  Terms of Service + EULAs •  Manual vs. Automated

enhancements/updates •  Customized features •  Product roadmaps

Enhancements and Update Policy 5

•  Agile process enables agile integrations

•  Tools, Frameworks, Other complementary software

•  API •  Avoid lock-ins to proprietary

frameworks and other SW! •  Can define cost-efficiency of

using either approach

Integrations 6

•  Intuitiveness, Easy-to-Use, Good Help Materials

•  Directly related to TCO •  Live demos and free

trials with the system •  Frequent webinars,

Q&A, "How-To" materials

Usability 7

•  Lock-ins with in-house environments

•  Term or Perpetual licensing model

•  Return-on-Investment?

•  Subscription models – optimal use of service

Contractual Flexibility 8

•  Own devices vs. devices from cloud farm?

•  Scalability with server infrastructure

•  Device control servers – Linux with Android = 15 –  iOS with Mac = 3

•  Easiest scalability comes w/privately hosted cloud

Equipment Scalability 9

•  Case Studies •  Customer References •  Reaching out to user

to ask any details

Recommendations & References 10

Calculation Examples

Diminishing Value Curve

•  Different levels of control and abstraction

•  Definitions aren't always clear – so make sure you understand what you are buying!

SaaS-PaaS-IaaS vs. In-house

•  Assumption – You are building several apps

•  No. of apps is not limited here, as all example infrastructures can handle large number of apps

– You use 50 different devices for testing – You need everything up and running and

someone maintaining the system 24/5 – OR – just business hours

Basic Example

•  In-house Setup –  Devices: ($200 x 50 = $10,000) –  Server infrastructure: ($1,000 per 5 devices -> $10,000) –  Software: ($99/month x 50 = $4,950/mo -> $59,400/year) –  Setup/deployment: 3 biz days ($50/hour x 8 x 3 = $1,200) –  DevOps to manage the system 24/5: ($50/hour x 24 x 5 =

$6,000/week x 50 weeks = $300,000) OR ALTERNATIVELY –  DevOps to manage the system on business hours: ($50 x 8 x

5 = $2,000/week x 50 weeks = $100,000)

•  Majority of costs comes from OpEX

In-House Dev & Test Lab

•  Public Cloud Setup –  Deployment: 1 business day ($50 x 8 = $400) –  Integration: 2 business days ($50 x 8 x 2 = $800) –  License for 50 devices: ($1499/mo -> $17,988/year)

•  Private Cloud Setup –  Deployment: 1 business day ($50 x 8 = $400) –  Integration: 2 business days ($50 x 8 x 2 = $800) –  License for 50 devices: ($199 x 50 -> $9,950/month ->

$119,400/year)

•  Majority of costs comes from CapEx

Cloud-Based Services

•  Head-to-Head: –  In-house (24/5)

costs $380KUSD vs. Private Cloud $120KUSD

•  Privately hosted cloud is 31.5% of in-house system costs

Cost Comparison

Comparable environments:

Significant difference in costs between in-house and cloud

Testdroid Update

Testdroid PrivateCloud - Running user-selected devices at any of our hosting facilities, more cost-efficiently than your own devices!

Option 1:

Public Device Cloud on-demand devices

(multitenant)

Mobile app testing on over 400+ real

Android and iOS devices hosted by

Bitbar

Option 2:

Private Device Cloud reserved devices

Hosted by Bitbar in the

US and/or Europe

Devices chosen by and reserved only for the

Customer

Option 3:

On-premise Device Cloud

Automated mobile app testing on real

Android and iOS devices hosted by the customer, usually 30-500 devices

Testdroid Deployment Options

•  Get to know the essential facts in terms of selecting cloud-based or in-house solution for your needs: – Characteristics – Costs vs. Benefits – Calculation examples

•  Available at www.testdroid.com/library

New EBOOK Available!

Video recording & slides of this webinar will be soon available at Vimeo and Slideshare! Visit at www.testdroid.com for more information.

THANK YOU!

W E B I N A R

We operate the largest global device cloud with over 400 devices

instantly available for developers

Devices from the US, Europe, China, Japan and Korea – global devices for global customers

Testdroid: Which One Works You The Best: In-House or Cloud-Based Development Environment

Technology

Transcript of Testdroid: Which One Works You The Best: In-House or Cloud-Based Development Environment

Learn How AIRVOI Cloud Based Voice Over IP System Works

Cloud Foundry | How it works

159665467 Cloud of Unknowing and Other Works the Anonymous Clifton Wolters

Testdroid: Best Practices in Mobile Game Testing

Testdroid: Build a Large Scale In-House Test Lab for Mobile Apps

How Cloud Web Filtering Works - Web Titan Cloud For WiFi

LDNSE: Testdroid for Mobile App and Web Testing (London Selenium Meetup)

Using Mobility Cloud Service Oracle Field Service Cloud · About Mobility Cloud Service Best Practices Oracle Field Service Mobility Cloud Service works best when it is receiving

Showing cloud collaboration works...regulatory compliance. The cloud allows biopharma ... (GxP) workloads to the cloud, though attitudes are changing. Amazon Web Services, working

How Cloud Load Balancing Works - Infographic

Testdroid Helps Mobile Game Developers Access NVIDIA Tegra K1-Powered Android Devices on Cloud

How It Works: Cloud-Native Protection for Amazon EC2

Testdroid: The Powerful and Comprehensive API for Mobile App Development and Testing

How QuickBooks Hosting works on Cloud

How Cloud Computing Works?

IBM Spectrum Scale 5.0.0: Concepts, Planning, and ... · How Cloud data sharing works ..... . 1 13 How W rite Once Read Many (WORM) storage works ..... . 1 16 Supported cloud pr oviders

9 Ways Cloud Works for Business

Cloud Computing Navigating the Cloud - PwC · 2015-06-03 · Cloud Computing Navigating the Cloud PwC works with technology companies around the world to help them fulfill the promise

Testdroid: How to Leverage Appium in Your Mobile App Testing

Cloud Computing Navigating the Cloud - PwC · Cloud Computing Navigating the Cloud PwC works with technology companies around the world to help them fulfill the promise of their great