Testdroid: Which One Works You The Best: In-House or Cloud-Based Development Environment

© Copyrights by Bitbar Technologies Ltd. 2015 All rights reserved. 1

8 January 2015

Ville-Veikko Helppi Technical Product Manager [email protected]

Which One Works You The Best: In-House or Cloud-Based Development Environment

W E B I N A R


Agenda

•  Comparison of In-house and Cloud-based Setups •  Investments in Mobile App Development and

Testing Environments •  The Characteristics to Use for Comparison of Both

Development Environments •  Calculation Examples •  Testdroid Update •  Q&A


Introduction to Creation of In-House Development Labs


Picture from #GTAC2014


Why Automation?

Smaller coverage, More money burnt & ?me wasted, Error-‐prone

Large coverage, quickly

completed, Less money & ?me wasted, Exact

results.

Manual Automa@on


•  Why Do You Need an In-House Development and Test Lab? –  Pre-market devices (OEMs, ODMs,

integrators ...) –  Prototypes / parts of hardware –  Corporate security policies –  Availability and high usage level –  Special setups (integration w/other

equipments) –  Large set of devices available

In-house Dev&Test Labs


•  What is Required for an In-House Development & Test Lab? –  Server hardware / infrastructure –  Own mobile devices –  Device control servers

•  Hard Disks, RAM, Energy management

–  USB hubs –  WiFi / Network infrastructure –  Continuous Integration/Delivery/Testing –  Build systems / other software tools –  POWER!



Continuous Integration


•  Version control and build revision •  Build automation, instant execution&testing •  All types of testing & integrations enabled •  Frequent commits, Code consolidation ->

Fast builds •  CI integrated with development and testing

provides the best fit for mobile agile process

Core Benefits of CI


•  How to Reach Enterprise-Grade Reliability for In-House Dev&Test Lab – Device management

•  Always run on clean devices •  Create intelligent retry mechanisms •  Create fall-back mechanisms

– System management •  Automatic reconnect of network and USB •  Automatic configuration changes •  Professional-level monitoring system

– Diskspace, network/connections, Power



Issues?

29% of problems are tightly related to devops of an

in-house system


Introduction to Adoption of Cloud-based Development Labs


•  Someone else is taking care of heavy-lifting •  Cloud-based device cloud includes ALL

relevant devices (phones, tablets, cams...) •  You get everything automatically delivered – Data, Results, Logs, Screenshots, Pass/Fail

Reports, Additional data-sets

•  Diversity and Keeping Up-to-Date with SW – New OS versions and OEM updates

•  Easiness of Adoption and Integration

Why Cloud?


Access via Comprehensive API


•  Authentication •  Project Management – Create, Delete, Configure, Managing Users

•  Configuring Access and Assets – Configs, Upload/Download Files (App & Tests) – Device Access & Device Group Configuration

•  Running Tests – Upload App + Tests <-> Download Results/Data – Notifications, Sharing, Pre-processing of Data

Access via Comprehensive API


Investments in Mobile App Development Environment


Investments Aspects

•  What is the Most Cost- Efficient Approach?

•  What is the Total Cost of Ownership?

•  Capital Expenses (CapEx)

•  Operation Expenses (OpEx)


•  CapEx is a business expense incurred to create future benefits or to add to the value of an existing asset

•  OpEx refers to the expenses that the mobile app developer spends in order to get their day-to-day activities going on and to get their infrastructure fully working

CapEx vs. OpEx


•  CapEx: Devices, Servers, Infrastructure •  OpEx: Running & Maintaining the System

Costs: In-House Dev&Test Lab


•  Cost of Service includes all major expenses (both CapEx and OpEx)

Costs: Cloud-based Dev&Test Lab


10 Different Characteristics for Comparing Environments


•  A

Different Characteristics & Benefits


•  Corporate Policies •  Regulation •  Minimizing security

breaches •  Security/Protection

mechanisms •  Monitoring •  Authentication &

User Access

Security 1


•  Needs to be fully configurable

•  Back-ups & Retention •  Preventing non-

authorized use and access •  Needs to available 24/7 •  Transferrable nature of

data, source code, results

Data Ownership and Retention 2

In-house vs.

Cloud


•  API access – no delays •  Critical/Medium/Low •  DevOps in crucial role •  Various data center

locations •  Hardware failures?

Monitoring? Scale up? •  Average Cloud Up-time:

99.5%

Performance and Availability 3


•  Support: Email, Phone, Online service systems

•  Intuitive online help: Manuals, How-Tos, Instructions (self-service)

•  Integration with bug-tracking systems (e.g. JIRA)

•  Frequent product updates •  Needs to be available 24/7

Support 4


•  Clear policies on updates, maintenance, monitoring

•  Terms of Service + EULAs •  Manual vs. Automated

enhancements/updates •  Customized features •  Product roadmaps

Enhancements and Update Policy 5


•  Agile process enables agile integrations

•  Tools, Frameworks, Other complementary software

•  API •  Avoid lock-ins to proprietary

frameworks and other SW! •  Can define cost-efficiency of

using either approach

Integrations 6


•  Intuitiveness, Easy-to-Use, Good Help Materials

•  Directly related to TCO •  Live demos and free

trials with the system •  Frequent webinars,

Q&A, "How-To" materials

Usability 7


•  Lock-ins with in-house environments

•  Term or Perpetual licensing model

•  Return-on-Investment?

•  Subscription models – optimal use of service

Contractual Flexibility 8


•  Own devices vs. devices from cloud farm?

•  Scalability with server infrastructure

•  Device control servers – Linux with Android = 15 –  iOS with Mac = 3

•  Easiest scalability comes w/privately hosted cloud

Equipment Scalability 9


•  Case Studies •  Customer References •  Reaching out to user

to ask any details

Recommendations & References 10


Calculation Examples


Diminishing Value Curve


•  Different levels of control and abstraction

•  Definitions aren't always clear – so make sure you understand what you are buying!

SaaS-PaaS-IaaS vs. In-house


•  Assumption – You are building several apps

•  No. of apps is not limited here, as all example infrastructures can handle large number of apps

– You use 50 different devices for testing – You need everything up and running and

someone maintaining the system 24/5 – OR – just business hours

Basic Example


•  In-house Setup –  Devices: ($200 x 50 = $10,000) –  Server infrastructure: ($1,000 per 5 devices -> $10,000) –  Software: ($99/month x 50 = $4,950/mo -> $59,400/year) –  Setup/deployment: 3 biz days ($50/hour x 8 x 3 = $1,200) –  DevOps to manage the system 24/5: ($50/hour x 24 x 5 =

$6,000/week x 50 weeks = $300,000) OR ALTERNATIVELY –  DevOps to manage the system on business hours: ($50 x 8 x

5 = $2,000/week x 50 weeks = $100,000)

•  Majority of costs comes from OpEX

In-House Dev & Test Lab


•  Public Cloud Setup –  Deployment: 1 business day ($50 x 8 = $400) –  Integration: 2 business days ($50 x 8 x 2 = $800) –  License for 50 devices: ($1499/mo -> $17,988/year)

•  Private Cloud Setup –  Deployment: 1 business day ($50 x 8 = $400) –  Integration: 2 business days ($50 x 8 x 2 = $800) –  License for 50 devices: ($199 x 50 -> $9,950/month ->

$119,400/year)

•  Majority of costs comes from CapEx

Cloud-Based Services


•  Head-to-Head: –  In-house (24/5)

costs $380KUSD vs. Private Cloud $120KUSD

•  Privately hosted cloud is 31.5% of in-house system costs

Cost Comparison

Comparable environments:

216%

Significant difference in costs between in-house and cloud


Testdroid Update


Testdroid PrivateCloud - Running user-selected devices at any of our hosting facilities, more cost-efficiently than your own devices!


Option 1:

Public Device Cloud on-demand devices

(multitenant)

Mobile app testing on over 400+ real

Android and iOS devices hosted by

Bitbar

Option 2:

Private Device Cloud reserved devices

Hosted by Bitbar in the

US and/or Europe

Devices chosen by and reserved only for the

Customer

Option 3:

On-premise Device Cloud

Automated mobile app testing on real

Android and iOS devices hosted by the customer, usually 30-500 devices

Testdroid Deployment Options


•  Get to know the essential facts in terms of selecting cloud-based or in-house solution for your needs: – Characteristics – Costs vs. Benefits – Calculation examples

•  Available at www.testdroid.com/library

New EBOOK Available!


Video recording & slides of this webinar will be soon available at Vimeo and Slideshare! Visit at www.testdroid.com for more information.

THANK YOU!

W E B I N A R


We operate the largest global device cloud with over 400 devices

instantly available for developers

Devices from the US, Europe, China, Japan and Korea – global devices for global customers

Testdroid: Which One Works You The Best: In-House or Cloud-Based Development Environment

Technology

Transcript of Testdroid: Which One Works You The Best: In-House or Cloud-Based Development Environment