Create Your Roadmap to Data-Driven Quality
-
Upload
brittany-price -
Category
Documents
-
view
65 -
download
0
description
Transcript of Create Your Roadmap to Data-Driven Quality
![Page 1: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/1.jpg)
Create Your Roadmap to Data-Driven QualitySeth EliotPrincipal Knowledge Engineer, Test Excellence
April 15, 2014
![Page 2: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/2.jpg)
Services andCloud andData-Driven Engineering and...
A/B Testing of Services
Digital Media Services
Petabytes Processed
About Seth
![Page 3: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/3.jpg)
![Page 4: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/4.jpg)
In This Session, You’ll Learn……how to create your “roadmap” to DDQ
Roadmap Road, Car, Gas At destination
Roadmap DDQ StrategyYou and your...
Application (service or product)EnvironmentEngineering Processes
![Page 5: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/5.jpg)
What is it? Why is it important?
Data-Driven Quality
![Page 6: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/6.jpg)
Data, where have we been?The HiPPO
Highest Paid Person’s Opinion
Engineering DataTest pass/fail resultsBug countsDelivery cadenceCode coverageCode Churn
![Page 7: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/7.jpg)
Data, where are we going?Engineering dataTest results Scoring engines using Bayesian
analysisProduction-quality dataYou may have heard of TiPNot as difficult as you might
thinkLots of solutions for lots of application
types
Real Users
Production Environments
![Page 8: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/8.jpg)
“If you have to kiss a lot of frogs to find a prince, find more frogs and kiss them faster and faster” - Mike Moran
![Page 9: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/9.jpg)
Virtuous cycle of DDQDeplo
y
DetectFix
Deploy
Detect
![Page 10: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/10.jpg)
Determine your questionsDesign for production-quality dataSelect your data sourcesUse the right data toolsGet answers to your questionsLearn new questionsRepeat
The roadmap
![Page 11: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/11.jpg)
AvailabilityPerformance
Usage? GQM - 1994
Start at the beginning
“Big Data” is hot, let’s start with that“Big Data” is hot, let’s start with that
![Page 12: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/12.jpg)
What do you need to know about quality?
Determine your questions
![Page 13: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/13.jpg)
Why not just get data and look for answers?
Do night-lights cause near-sightedness in children?
Quinn, et al, 1999
Does sunscreen increase chance of drowning?
![Page 14: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/14.jpg)
What questions does EXO ask? Monthly
Uptime Percentage
Service Credit
< 99.9% 25% < 99% 50%< 95% 100%
“Downtime” is defined as any period of time when users are unable to send or receive email via all supported mailbox access
![Page 15: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/15.jpg)
Is it available?Often a Pri 0
Critical work stoppage for user“Dialtone”
Is the application (product, service) there for the user?
![Page 16: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/16.jpg)
Application works, but feature does notOccasionally does not render/load properly
違った言語
How do users perceive availability?
![Page 17: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/17.jpg)
Huge ImpactHow is performance?
Power of Production Data
Real usersMultiple environments
End to endScale & geo-diversity
![Page 18: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/18.jpg)
What do users do?
Customer Experience Improvement Program (CEIP)
![Page 19: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/19.jpg)
Your turn - Determine your questions
What type of answer are you looking for?Availability, performance, usage?Prioritize
When does availability NOT come first?
![Page 20: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/20.jpg)
Your turn - Determine your questions
What are the key scenarios for the type you selected?
What is high priority?
SharePoint
![Page 21: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/21.jpg)
Get data from or near production
Design for production-quality data
![Page 22: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/22.jpg)
Two types of data to acquire
Active: syntheticPassive: real (RUM)
Active data in prodFor services only?Client: is the service
there?
![Page 23: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/23.jpg)
Staged data acquisition mitigates riskService
Product (client, on-prem server)Stage 1: Partial
or whole product team
Stage 2: Dogfood
Stage 3: Technology Adoption
Programs (TAP)
Stage 4: Some clients in
production
Stage 5: All Customers
Stage 1: In prod, no users
Stage 2: Dogfood
Stage 3: Some servers in prod
Stage 4: Some more servers in
prod
Stage 5 : World-wide
prod
Deployment Validation
Service Validation Scale Validation
Real-time service quality
![Page 24: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/24.jpg)
Staged data acquisition - Netflix
1B API requests per day
Canary Deployment
![Page 25: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/25.jpg)
Staged Data Acquisition - Facebook
DogfoodIn prod, no users (except internal ones)Some servers in ProductionWorld-wide deploymentFeature flags
![Page 26: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/26.jpg)
Speed of deployment
Deploy
Detect
Usually easy for services
Client apps may have a deployment capability
Or may make use of feature flags
EaaSy - Everything is now connected and thus updatable
![Page 27: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/27.jpg)
Data acquisition for clientsFiltering and aggregation at clientBe kind to the clientDon’t abuse user resources:
BandwidthBatteryDisk
Outlook
![Page 28: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/28.jpg)
Bucket-ized data is aggregated and uses less bandwidth
![Page 29: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/29.jpg)
Data acquisition for clientsFiltering and aggregation at clientBe kind to the client
Pipeline to collect and process dataMake it easy
Staged Data AcquisitionOutlook
Feature Crew
Outlook Team
MS Office Team
Microsoft
Customers
Scale Validation
![Page 30: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/30.jpg)
Your turn - Design for production-quality data
What might be your stages for risk mitigated data acquisition?
Role of active and passive monitoring?
How can you engineer for EaaSy deployment?
![Page 31: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/31.jpg)
Determine the data necessary to answer your questions
Select your data sources
![Page 32: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/32.jpg)
CPU Memory
Storage Network
Infrastructure data
![Page 33: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/33.jpg)
Application data
New and Open
Download Template
Failure Rate
Average Time
![Page 34: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/34.jpg)
Hang and crash dataSpecialized application dataMost frequently encountered conditionsBucket the Big Data and find the offendersGet offending code and function calls (stack)
But it worked in DogfoodCulprit was a old version add-inHarden against this
![Page 35: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/35.jpg)
Usage dataClient-side instrumentationProprietaryJavascript: clicks, hovers [web apps]Get 1x1 GIF: Page Views [web apps]
Combine into more complex scenariosHow did user get to shopping cart checkout?
100%Users
50%Users
50%Users
Control:Existing System
Treatment:Existing System with Feature X
Users interactions instrumented, analyzed & compared
Analyze at the end of the experiment
![Page 36: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/36.jpg)
Feedback data
![Page 37: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/37.jpg)
Feedback data
![Page 38: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/38.jpg)
Feedback data also includes…
Xbox Kinect
Social Media Mining
Customer Support Data
![Page 39: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/39.jpg)
Active monitoring dataData Center X
Data Center YAM2
Service
AM3AM1
Dependency Service(s)
Client
![Page 40: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/40.jpg)
Ease of detection
Deploy
Detect
EaaSy – Rich near real-time telemetry
Again services have it easy
Many clients are always connected
On prem servers (enterprise) require partnerships
![Page 41: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/41.jpg)
Data handling - privacyTransparency and ControlCollection and RetentionDepends on Type anonymous Datapseudonymouspersonally identifiable info (PII)sensitive PII
Depends on Purposeprovide the serviceimprove the current serviceimprove a future version serviceimprove non-associated servicescontent personalizationad targeting
![Page 42: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/42.jpg)
Data handling – non-service productsClient and on-prem server considerations
User owned resources: bandwidth
battery
disk, cpu, etc…
Correlations
end-to-end across clients and services
by user , by session
![Page 43: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/43.jpg)
Your turn - Select your data sources
Infrastructure dataApplication dataHang and crash dataUsage dataFeedback dataActive monitoring data
Design to handle your data
![Page 44: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/44.jpg)
A 50,000 foot view
Use the right data tools
![Page 45: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/45.jpg)
Data storage and processing systemsDatabase Table storage
(SQL)Optimized for CRUD – (create, read, update, delete) of single records
Data Warehouse
Table storage (SQL)
Table structure optimized for queries and bulk insert
OLAP Cubes(Online Analytical Processing)
Multidimensional Cube (MDX)
Aggregations of measures for multiple dimensions
Hadoop / Map-Reduce
Distributed File System (HDFS)
Big Data TB-PB-EBUnstructured Data
![Page 46: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/46.jpg)
Hadoop in 60 Seconds
HDFS – Hadoop Distributed File System
ADD FCA
ECC
ADDADD
ADD
FCAFCA
FCA
ECCECC
ECC
Map-Reduce
1xA1xC1xF
1xA2xD
1xE2xCADDFCAECC
2xA0xB3xC2xD1xE1xF
Hadoop
![Page 47: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/47.jpg)
A common data flowPower BI in
ExcelCube
Telemetry
System
SQL DW
Programmatic accessOther
Visualizations
Real-timeMonitoring
![Page 48: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/48.jpg)
Web Apps integrated with OneDrive, FB, web mail, etc
Real-time monitoring
![Page 49: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/49.jpg)
Twitter Xbox Kinect
Feedback
![Page 50: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/50.jpg)
Your turn - Use the right data processing tools
DB, DW, Cube, Big-Data platformsPut it all together
Do you need real-time monitoring?
…sentiment analysis?
![Page 51: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/51.jpg)
and learn new questions
Get answers to your questions
![Page 52: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/52.jpg)
Outlook.com prioritizes performance
500 Million measurements per month
JSIJavaScript Instrumentation
View Inbox – Page Load Time (PLT) by BrowserAs experienced by actual users
![Page 53: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/53.jpg)
Prioritizes availability
Predict 75% of dips 24 hours ahead of time
Avai
labi
lity
Time
![Page 54: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/54.jpg)
54
Netflix prioritizes perceived availability & performance
![Page 55: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/55.jpg)
Yammer prioritizes usage
What happens to new user retention when you shorten the signup flow?
It goes down!
Don’t ship that feature
![Page 56: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/56.jpg)
Find new questions and repeat
Xbox recommends
Try algorithm
Collect data
Adjust algorithm
Collect data
Repeat
![Page 57: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/57.jpg)
Find new questions and repeat
In Visual Studio 2012 we added asynchronous loading for solutions
We also added Telemetry
We can see how it works
![Page 58: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/58.jpg)
Your turn – Get answers; find more questions
Enjoy the rest of STPCon :-)
Get back to work
Implement your roadmap
Get answers to your quality assessment questions
![Page 59: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/59.jpg)
SummaryDetermine your questionsDesign for production-quality dataSelect your data sourcesUse the right data toolsGet answers to your questionsLearn new questionsRepeatThe most recent version of this deck can be found at http://setheliot.com
![Page 60: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/60.jpg)
Special thanks to these folksRavi VedulaAndrea JesseBill HodgheadDanny ThayerJoseph SefairKitty ThomasAmanda ReinkeHeather Lader
David BrooksMike TholfsenJodie DraperBrian MuellerTara RothDror CohenNathan HalsteadLori Oviatt Thanks!
Monica CatundaLynette SkinnerJoe SchumacherDonny LuuJohn HoeggerAlain Anyouzoa
Any questions?
![Page 61: Create Your Roadmap to Data-Driven Quality](https://reader035.fdocuments.in/reader035/viewer/2022081503/56812ddc550346895d932ee7/html5/thumbnails/61.jpg)
Please fill out the surveyCreate Your Roadmap to Data-Driven QualitySession 101
Seth Eliot?