ReportPortal.io - Open Source experience. Showcase, benefits
-
Upload
comaqaby -
Category
Technology
-
view
189 -
download
1
Transcript of ReportPortal.io - Open Source experience. Showcase, benefits
2CONFIDENTIAL
SPEAKER NAME
Проект Менеджер
10 лет с компанией EPAM Systems. Разработчик в прошломВозглавляю разработку инициатив внутри Центра Компетенций Тестирования.
Report Portal – Product Owner.
DZMITRY HUMIANIUK
EPAM Systems, Project Manager
ДМИТРИЙ ГУМЕНЮК
3CONFIDENTIAL
SPEAKER NAME
DZMITRY HUMIANIUK
EPAM Systems, Project Manager
ДМИТРИЙ ГУМЕНЮК
Проект Менеджер
10 лет с компанией EPAM Systems. Разработчик в прошломВозглавляю разработку инициатив внутри Центра Компетенций Тестирования.
Report Portal – Product Owner.
6CONFIDENTIAL
SPEAKER NAME
Проект Менеджер
10 лет с компанией EPAM Systems. Разработчик в прошломВозглавляю разработку инициатив внутри Центра Компетенций Тестирования.
Report Portal – Product Owner.
DZMITRY HUMIANIUK
EPAM Systems, Project Manager
ДМИТРИЙ ГУМЕНЮК
7CONFIDENTIAL
SPEAKER NAME
Проект Менеджер
10 лет с компанией EPAM Systems. Разработчик в прошломВозглавляю разработку инициатив внутри Центра Компетенций Тестирования.
Report Portal – Product Owner.DZMITRY HUMIANIUK
EPAM Systems, Project Manager
18CONFIDENTIAL
КАК ЧАСТО ВЫ ЗАПУСКАЕТЕ АВТОМАТИЗАЦИЮ?КАКОЙ ПРОЦЕНТ ПРОХОЖДЕНИЯ
ВАШИХ ТЕСТОВ?ПРОСМАТРИВАЕТ ЛИ КОМАНДА ВСЕ РЕЗУЛЬТАТЫ?КАК МНОГО ТЕСТОВ УПАЛО ИЗ-ЗА
ПРОДУКТОВЫХ БАГОВ?НА СКОЛЬКО ВАША АВТОМАТИЗАЦИЯ БЛОКИРУЕТ ПОСТАВКУ?
21CONFIDENTIAL
ИДЕИ В ОСНОВЕ РЕПОРТ ПОРТАЛАLIVE VISIBILITY В СОСТОЯНИЕ
АВТОМАТИЗАЦИИ
СОКРАЩЕНИЕ ЗАТРАТ НА
РАЗБОР РЕЗУЛЬТАТОВ
ПОДДЕРЖКА ПОПУЛЯРНЫХ
ТУЛОВ В АВТОМАТИЗАЦИ
И
КРАСИВЫЕ ГРАФИКИ ПРЯМО ИЗ КОРОБКИ
МОЖЕТ РАБОТАТЬ С
СУЩЕСТВУЮЩЕЙ АВТОМАТИЗАЦИЕ
Й
СОЛЮШЕНЫ ПОМОГАЮТ
ДЕЛАТЬ ДЕНЬГИ
22CONFIDENTIAL
УСПЕХ РЕПОРТ ПОРТАЛА8ВНЕШНИХ
УСТАНОВОКУ
ЗАКАЗЧИКОВ
110+ ПРОЕКТОВ
ИСПОЛЬЗОВАЛИREPORTPORTAL
25CONFIDENTIAL
REPORT PORTAL: ПРОСТАЯ ИНТЕГРАЦИЯ
<4 ЧАСОВ
Без рефакторинга тестов!
TestNGJUnitNunitSoapUICucumber Cucumber-JVMProtractor Node.JSSpecFlowjBehave…
27CONFIDENTIAL
REPORT PORTAL: ПРОСТАЯ ИНТЕГРАЦИЯ
CI tool (Jenkins)
TestNGRP adapter
Report Portal Server
28CONFIDENTIAL
HOW IT WORKS?
EPAM Private Cloud
REST Web Service
Mongo DBNoSQL
UIJava Script
Agents
HTTPs
Domain Credentials,UPSA profiles
31CONFIDENTIAL
“FIRE DETECTOR” METRICSName Why it is important to
monitor thisWhat decision can be
madePeriodicity
Calculation
Execution Frequency
To understand and evaluate how often test cases executed.Goal: Increase execution ratio per day/build.Configure automated execution of TA after commits (smoke, acceptance) and regressions right after nightly builds.
Triggers1 execution per week or less (???)
Target1 per commitInterpretationcode change should be verified not to affect functionality or other code base.
Each day
Measures1. test executions countFormula Unit of measureNumber
% of results analyzed
To understand and evaluate percentage of reviewed fail reports. If fail report not reviewed - there is not any value
Triggers<60%
Target100%Interpretationfailed report(test cases) should be review
Each day
Measures1. No of not analyzed failed test
cases (TI)2. No of failed due to product bug
(PB)3. No of failed due to automation
issues (AB)4. No of failed due to system issue
(SI)
Formula100% - To_Investigate / (PB+AB+SI+TI)
Unit of measure%
32CONFIDENTIAL
“FIRE DETECTOR” METRICSName Why it is
important to monitor this
What decision can be made Periodicity
Calculation
% of product issues (stability of automation)
To understand and evaluate percentage of failed test which corresponds to production issues
Triggerstrend not being positive
Target>90%Interpretationtrend of fails due to test automation instability should decrease (automation issues, system issue, not analyzed failed test cases)
Measures1. No of not analyzed failed test cases
(TI)2. No of failed due to product bug (PB)3. No of failed due to automation
issues (AB)4. No of failed due to system issue (SI)
FormulaPB / (PB+AB+SI+TI)
Unit of measure%
% of automation issues
To understand and evaluate percentage of failed test which corresponds to test automation issues (code issues, test case invalid)
Triggerstrend not being positive
Target<5%Interpretationtrend of fails due to test automation instability should decrease
Measures1. No of not analyzed failed test cases
(TI)2. No of failed due to product bug (PB)3. No of failed due to automation
issues (AB)4. No of failed due to system issue (SI)
FormulaAB / (PB+AB+SI+TI)
Unit of measure%
33CONFIDENTIAL
“FIRE DETECTOR” METRICSName Why it is important
to monitor thisWhat decision can be
madePeriodicit
y
Calculation
% of system issues
To understand and evaluate percentage of failed test which corresponds to system issues (no connectivity, lack of resources and etc.)
Triggerstrend not being positive
Target<5%Interpretationtrend of fails due to system issue instability should decrease
Measures1. No of not analyzed failed test cases (TI)2. No of failed due to product bug (PB)3. No of failed due to automation issues
(AB)4. No of failed due to system issue (SI)
FormulaSI / (PB+AB+SI+TI)
Unit of measure%
Execution Time
To understand and evaluate time of each execution cycle.
TriggersRegression cycles > 15hacceptance cycles >1h
Targetregression <8hacceptance <1hInterpretationtime for execution should not go over described limits
Measures1. Execution duration of regression, hours
(Reg)2. Execution duration of acceptance, hours
(Acp)
FormulaReg < 15hAcp < 1h
Unit of measureHours
34CONFIDENTIAL
SPECIFIC FOR QA AUTOMATION METRICSMetric Name Description Category
Percent Automatable (PA) # of test cases automatable (ATC) / # of total test cases (TC)
Quality
Regression frequency How often does automated regression run? QualityAutomation Progress (AP) # of actual test cases automated (AA) / # of
test cases automatable (ATC)Progress
Percent of Automated Testing Coverage (PTC for Automation)
Automation coverage (AC) / Total Coverage (C) (KLOC, FP, etc)
Coverage
Automated Testing Window
1) How much time does it take to run QA Automation?
2) How much system time/lab time is required to run the test(s) selected?
Cost / time
Testing Results Analyses Windows
How much time does it take to do data analysis?
Cost / time
Auto-Test Development Time
# of automated tests/ time to develop Cost / time
Auto-Test Support Time # of automated tests/ time to support Cost / time