A testing journey in the age of smart assistants · 2018-09-27 · 10 September 2018 Geoff Meyer,...
Transcript of A testing journey in the age of smart assistants · 2018-09-27 · 10 September 2018 Geoff Meyer,...
10 September 2018
Geoff Meyer, Test Architect
A testing journey in the age of smart assistants
TestExpo2018
Agenda
•Context of DellEMC Server
•AI/Machine Learning 101
•Pinpoint your Testing Painpoints
•Our Superpowers in action
Context at DellEMC Servers
Server Configuration ElementsChassisProcessorMemory DIMMMemory ConfigurationHard Disk Drive (HDD)Non-Volatile Memory (NVM)Embedded Systems ManagementPower Management BIOSPower SupplyBezelNetwork Daughter CardRAID ControllerNetwork Interface Card (NIC)Host Bus Adapter (HBA)Additional PCIe CardsCooling
465 Trillion Test Configurations!!
Analytics Engine
Domain Knowledge
(i.e. Rules)
Data Sources
Data Cleanse
Insights,
Predictions,
Recommendations
Feedback
Data Analytics Modeling
Getting Computers to learn without being explicitly programmed
• Supervised Learning
• Unsupervised Learning
• Reinforcement Learning
Machine Learning
Getting Computers to learn without being explicitly programmed
• Supervised Learning
• Unsupervised Learning
• Reinforcement Learning
Machine Learning
Getting Computers to learn without being explicitly programmed
• Supervised Learning
• Unsupervised Learning
• Reinforcement Learning
Machine Learning
MarI/O - https://www.youtube.com/watch?v=qv6UVOQ0F44
It’s all about the data
“Nobody really goes out of their way to point out the importance of data…”
~ Brian Sletten, Bosatsu Consulting
Data Sources - Product Engineering
What if we had a Smart Assistant?
What are the high-value SUT configurations? What test scripts
should be retired rather than be re-factored?
What tests can detect the maximum number of defects
given the changes in the current build
What is the release risk given the testing that’s been completed?
What’s the optimal coverage for this build/test cycle? What automated test
failures appear to be duplicates?
The Smart Assistant
Selecting our Technology Partners
Q
Challenges
➢ 465 Trillion possible server configs!!!➢ Which are the High Value Configs?➢ How to ensure Optimal Configs
Coverage?
Current Testing Scenario
➢ Quickly predict “best-available” SUT configurations during planning and test execution
➢ Ensure Optimal Configs Coverage➢ Prioritizes High-Value Configs
Objectives
SUT Configuration Model“Q” - System Under Test
Data Sources
Team
Technology Partner: Dell Performance Analytics Group
Q Process FlowDATA SETS
FUNCTIONS
TOOLS & TECH
HIGH VALUE
SUT CONFIGS
FEEDBACK LOOP MACHINE LEARNING
DEFECTS
FEEDBACK LOOP MACHINE LEARNING
TEST HISTORY
100 Configs/RISER+
RISER SLOTS
STORAGE MEMORY THERMAL
BUILD VALID CONFIGS
CONFIGURATION - RESTRICTIONS
PROBABILISTIC RANDOM SAMPLING
100K Configs/RISER
500K Configs/RISER+
CLUSTERING
AS SOLD CONFIGS
NUDDs
SCORING MODEL
GEN OVER GEN
MAPPING
How Does Q Help?
• Increase precision of predictions by extending data sets:
• Test Execution & Defect History (JARVIS integration)
• As-Deployed customer configurations
• As-Tested configurations
• Cost Optimization of prototypes/parts for new platforms
1 Improved Test Coverage - Maximizes Test Coverage given limited availability
Cost Savings - Optimizes number of prototype configurations needed for test
Data-driven - Institutionalizes the Test Configuration Planning process2
3
Time Savings - Reduces Test Configuration Planning from weeks to hours4
Maintains Optimized Coverage – When components/features are delayed1
Automation candidates
De-prioritization candidates
Test Planning Model“JARVIS”
Technology Partner:
Objectives• Use historical test data and defects
as predictors and to expose patterns• Automate deep-think testing tasks• Codify Subject-Matter Expertise• Real-time access to active
repositories
How can I accelerate discovery of break/fix?
Which manual tests are most effective, and should be automated?
Am I over-testing or under-testing?
Which of my test cases appear to be obsolete?
Fast Find of Break/Fix
Reduced test cycle time
Increased Test Capacity
Re-allocate toExploratory Testing
Testbots are hereAI-assisted UI Automation
• Increases UI test coverage
• At substantially less cost of creation and maintenance
Duplicate Defect PredictionDellEMC XtremIO
2018 2019
SUT Config
Develop Deploy Extend
Integrate
JARVIS
Pilot
Deploy (EST)
Test Suite
Workload
AI POC’s Change-Based Regression
AI-Assisted Testbots
Data Curation
Visualization
Visualization
Deploy +
Deploy+
Dell Performance Analytics Group
Cognitive Automation Roadmap
2017
Develop Deploy
Proof-of-Concept
Proof-of-Concept
Staffing
Today
Deploying Smart Assistants
Partner with the Machine
Pinpoint your pain points
Capture your data
Re-imagine Testing
Thank you
ResourcesBooks• Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die: https://www.amazon.com/dp/B019HR9X4U/ref=dp-kindle-redirect?_encoding=UTF8&btkr=1• What To Do When Machines Do Everything: http://www.whenmachinesdoeverything.com/• Weapons of Math Destruction: https://weaponsofmathdestructionbook.com/• Race against the Machine: https://books.google.com/books/about/Race_Against_the_Machine.html?id=IhArMwEACAAJ• Super Freakonomics: http://freakonomics.com/books/• Humans are underrated: http://geoffcolvin.com/books/humans-are-underrated/• Life 3.0: Being Human in the Age of Artificial Intelligence: https://www.amazon.com/Life-3-0-Being-Artificial-Intelligence/dp/1101946598• The Four: http://www.thefourbook.com/
Research• When will AI Exceed Human Performance: https://arxiv.org/pdf/1705.08807.pdf• World Quality Report 2016-17 (Capgemini): https://www.capgemini.com/thought-leadership/world-quality-report-2016-17• World Quality Report 2017-18 (Capgemini): https://www.capgemini.com/thought-leadership/world-quality-report-2017-18• The next era of Human|Machine Partnerships: https://www.delltechnologies.com/en-us/perspectives/realizing-2030.htm• Towards a Reskilling Revolution: A Future of Jobs for All: http://www3.weforum.org/docs/WEF_FOW_Reskilling_Revolution.pdf• Special report: Tech and the future of transportation: http://b2b.cbsimg.net/downloads/Gilbert/SF_feb2018_transport.pdf• How AL will Change Software Development: https://www.slideshare.net/WillyDevNET/how-ai-will-change-software-development-and-applications• 21 Jobs of the future: https://www.cognizant.com/whitepapers/21-jobs-of-the-future-a-guide-to-getting-and-staying-employed-over-the-next-10-years-codex3049.pdf• Wait but why: Artificial Intelligence Revolution Part 1: https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-1.html• Wait but why: Artificial Intelligence Revolution Part 2: https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2.html• What’s Next | Artificial Intelligence Part 1: https://www.youtube.com/watch?v=2br8yji-rcM• What’s Next | Artificial Intelligence Part 2: https://www.youtube.com/watch?v=_WKyiGBYFrU• TensorFlow by Brian Sletten: https://www.youtube.com/watch?v=RlrBKYehcNg• Wolfram Alpha: www.wolframalpha.com
ResourcesArticles• The Future of Jobs: http://www3.weforum.org/docs/WEF_Future_of_Jobs.pdf• This Technology Will Upend the Entire Automotive Industry: https://moneywise411.com/new-automotive-technology/?ppc=743242• 5 ways AI will change software testing - https://techbeacon.com/5-ways-ai-will-change-software-testing• What’s Everybody So Afraid of: http://www.popularmechanics.com/technology/robots/news/a28645/googles-alphabet-astro-teller-ai/• Robots Are Coming for Jobs of as Many as 800 Million Worldwide: https://www.bloomberg.com/news/articles/2017-11-29/robots-are-coming-for-jobs-of-as-many-as-800-million-
worldwide• Self-Driving Cars Could Save 300,000 Lives: https://www.theatlantic.com/technology/archive/2015/09/self-driving-cars-could-save-300000-lives-per-decade-in-america/407956/• The 10 Biggest AI Failures of 2017: https://www.techrepublic.com/article/the-10-biggest-ai-failures-of-2017/?ftag=TRE684d531&bhid=24345184115902224026945549370599• Technology has created more jobs than it has destroyed: https://www.theguardian.com/business/2015/aug/17/technology-created-more-jobs-than-destroyed-140-years-data-
census• AlphaZero Is the New Chess Champion: https://www.extremetech.com/extreme/260215-alphazero-new-chess-champion-harbinger-brave-new-world-ai• The tech industry needs one million workers now: https://www.yahoo.com/finance/news/tech-industry-needs-one-million-workers-now-130452775.html• Only 4% of CIOs have deployed AI:https://cio.economictimes.indiatimes.com/news/business-analytics/only-4-pc-of-cios-have-deployed-ai-despite-huge-interest-levels-in-ai-
technologies/62900459• Towers Watson & Oxford Economics: Global Talent 2021: https://abhishekmittal.com/2012/08/04/towers-watson-oxford-economics-global-talent-2021/• Meet the 13-year-old prodigy taking IBM and AI by storm: http://www.abc.net.au/news/2017-07-26/meet-the-teen-taking-ibm-and-artificial-intelligence-by-
storm/8743880?pfmredir=sm• Testing and Management Efficiency: http://www.developsense.com/blog/2018/02/Efficiency/• How reinventing software testing can transform your business: https://techcrunch.com/2018/03/13/how-reinventing-software-testing-can-transform-your-business-and-change-
the-world/?utm_content=68756890&utm_medium=social&utm_source=linkedin• Top 5: Things to know about AI: https://www.techrepublic.com/article/top-5-things-to-know-about-ai• Test.AI nabs $11M Series A funding: https://techcrunch.com/2018/07/31/test-ai-nabs-11m-series-a-led-by-google-to-put-bots-to-work-testing-apps/• Understanding the differences between AI, machine learning, and deep learning: https://www.techrepublic.com/article/understanding-the-differences-between-ai-machine-
learning-and-deep-learning• AI in software testing has arrived. Here's why robots rule: https://searchsoftwarequality.techtarget.com/feature/AI-in-software-testing-has-arrived-Heres-why-robots-rule• A 5-second test for AI fever: https://www.linkedin.com/pulse/5-second-test-ai-fever-g-%C3%B8stby-sol%C3%A5s/
Title: A testing journey in the age of Smart Assistants
Description: The hype surrounding Artificial Intelligence (AI) has rocketed beyond its second peak. AI has undoubtedly gained traction and delivered
value to traditional businesses by automating away manual processes, tasks and jobs in industry after industry. On top of that, entire industries have been disrupted by those that that have used this opportunity to re-invent processes and business models to create new markets. As the general purpose technology of the 4th Industrial Revolution, the emerging chapters of these fascinating Machines demands our attention as AI continues to be applied in ways that directly affect the workplace, one in which the Test community won’t be immune.
In this latest hype cycle, new products and consultants are everywhere and inundating us with solutions that may or may not be applicable to our organizational context. We find ourselves having to sort out fact from fiction and often underestimate the effort in assessing the viability of these new practices. At times like this, shared insight from a fellow practitioner who’s been down this road could be a most welcome assist.
Geoff shares the "in-progress" journey at DellEMC as they drive to optimize and re-invent their testing practices with the application of data-driven smart assistants, powered by Analytics and Machine Learning. At a macro level, Geoff identifies opportunities across the Product Engineering and Test landscape for the application of Analytics and AI. Key ingredients in moving toward solutions that matter is the identification of organization-specific pain points, their prioritization, and the availability and cleanliness of essential data. Geoff shares the process of experimentation, staffing and implementation that his team approached these new opportunities with and then delves into the Smart Assistants that they’ve created to automate deep-think, cognitive-based testing tasks. “'Q", "JARVIS", and “Goose” automate many of the time-consuming and deeply analytical tasks such as determining high-value test configurations, defining high-value/maximum coverage regression test suites, and identifying market-demanding solution workloads when time is not an ally. Most importantly, Geoff shares insights on the activities that should get the highest levels of attention and those that you might want to de-prioritize to later phases of your own Analytics and AI journey.
Abstract
https://www.linkedin.com/in/geoff-meyer-02b1aa3/
https://twitter.com/geoffrey_meyer
A Test Architect in the Dell EMC Infrastructure Solutions Group, Geoff has 30+ years of industry experience as a software developer, manager, program manager, and director.
He drives the Test Strategy and Architecture for 400+ SW and HW Testers across India, Taiwan, and the United States. His initiatives include Agile Testing, Continuous Testing, Infrastructure as a Service(IaaS), Predictive Analytics and Test.AI
Geoff is a member of the Agile Austin community and is a speaker at Agile, STAR, and international testing conferences. He is an active mentor to Veterans participating in the Vets4Quality.Org program, which provides them an on-ramp to a career in software quality assurance.
Geoff Meyer