CI (cloud): Commit Frequencies
15% less than 5 per week
33% between 5 and 25 per week
33% between 25-50 per week
19% more than 50 per week (~ 10 times a day)
CI: Survey: Top Priorities
48% Improve app performance
41% Faster release/update frequency
11 % Increase security/vulnerability testing
Manual Testing For Mobile
Few Similarities
Zero (for startups)
Dev-only
Beta only
In-house
Often <200, 2000 for banking/etc.
UI Automated
Tests
Many try. Most fail
App changes cause test breaks
Maintenance quickly destroys ROI of UI automation
Often <50 tests. >150 is rare.
DevicesVs
Em/Sim-ulators
Hype cycle: Hardware, then pain, then simulators/emulators
Good news: Google/AMZN device clouds
Perfecto: Expensive, but most complete solution with device access
Smarter: Use engagement metrics for device matrix
Device cloud acquisitions
Good. Subsidized cloud services
Still complaint about connectivity
Perfecto and device anywhere high price but manual access and best of breed
On premise
Few happy with solutions
Cost
Lack of devise matrix . Most debug on hard to find devices
Support insufficient
Test Authoring
Developing test cases is the bottleneck
Developers Don't have time thanks to CI/CD/Agile
Teams can’t afford testers. Those that can...can’t hire or retain them.
Talent
Unicorns: Hard to Find Developer+Tester
Tornado: Harder to retain as they move into Development roles
Expensive!
Frameworks
Espresso and Xctest Solid, but platform specific
Appium still note quite stable, but best long term bet for cross-platform re-use
Calabash ok
SDKs
Install SDK + Code: App and Server
Maintain/Update
Can cause crashes and perf issues themselves
Several in same app can conflict
Limitations of Production APM
Issues found *after* user impact
Difficult to interpret graphs, charts, dials
Measure API and Code--Not User Perception
Why Pre-Production, User-Focused APM?
Catch issues *before* user impact
Measure user-perceived performance
Zero-Code, Zero SDK.
With CI, know which code change impacted performance
We still ‘Hand-Craft’
Missed leveraging the availability and pricing changes in compute, networking and storage.
Tooling is still very much the same: unit test frameworks, mocks, and ‘selenium’-like DOM driving frameworks.
We still ‘hand-craft’
Future is Machine learning + bots
Only hope to deal with the exponential complexity is lots of smart compute
Kick the can down the road about a mile
Large, Centralized, Dedicated Test Team (80%)
Top Related