Process Software Validation Virtual - LORE --...
-
Upload
truongnhan -
Category
Documents
-
view
232 -
download
0
Transcript of Process Software Validation Virtual - LORE --...
May 2011 – LMS Confidential
Software validation – LMS Virtual.Lab
2 copyright LMS International - 2011
Outline
2
1 Company profile
Introduction to QA
Virtual.Lab Product Development Process
Software validation in the development phase
Software validation in the QC phase
Customer responsiveness in the deployment phase
3
4
5
6
3 copyright LMS International - 2011
Company Profile
4 copyright LMS International - 2011
LMS, 30 years of Engineering Innovation
� Servicing more than 100.000 R&D engineers … in 5.000 manufacturing companies
� Top talent in 30+ offices Worldwide… almost 1.000 professionals
� Visionary, industry-leading innovator
… 25% of budget in R&D
� Strong financial track record of profitable growth
Leading Partner in
Test & Mechatronic Simulation
5 copyright LMS International - 2011
LMS, serving more than 5000 leading manufacturers
in the auto, aerospace, fabrication and assembly industries
Partnering with automotive manufacturers Partnering with automotive suppliers
Partnering with aerospace companies Partnering with manufacturing companies
6 copyright LMS International - 2011
LMS, Global Talent for Global Customers
R&D Centers
Brasov
Breda ChennaiCoralville Gottingen Kaiserslautern
Leuven LyonPlymouthRoanne
Torino
Sales Offices
Baltimore Bangalore Beijing
Coventry Detroit LeonbergLeuven Los Angeles
Lyon Moscow Munich Novara Paris
Sao Paulo Seoul Shanghai Toulouse
Yokohama
Representatives
Bangkok Bangalore BeckumBucharest Cairo
Kfar Saba Krakow Kuala Lumpur Ljublijana
Madrid Moscow Nilufer BURSA PsárySão Paulo
SarovSt. Petersburg Sydney Veliky Novgorod
Engineering Services
Beijing
Brasov ChennaiCoralville DetroitKaiserslautern
Leuven LyonTorinoYokohama
Europe
40%
Americas
20%
Japan/Korea
20%
RIC
20%
33 15
7 copyright LMS International - 2011
LMS, a unique portfolio of engineering innovation solutions
LMS Tec.ManagerTest data management
LMS Imagine.Lab1D simulation solution
LMS Virtual.Lab3D simulation solutions
LMS Engineering and
Deployment Services
LMS SCADAS
Mobile Recorder Lab
LMS Test.LabTest-based engineering solutions
LMS Test.XpressAnalyzer testing solutions
LMS SCADAS
Mobile Recorder Lab
8 copyright LMS International - 2011
LMS Virtual.Lab, the integrated platformfor 3-D functional performance simulation
Virtual.Lab Rev.9 “For automotive and aerospace design simulation, LMS Virtual.Lab remains one of the most comprehensive, capable, and versatile packages we have seen”
Jeffrey Rowe, Editor
Integrated CAD – Simulation Unified Modeling Supported by Test
Acoustics NVH Motion Durability
9 copyright LMS International - 2011
Introduction QA
10 copyright LMS International - 2011
Introduction to Software Quality Assurance Concept
�The perfect quality of a software is reached when the tests of all the possible combinations between all functionalities provides no error.
�Practically, this is a very expensive process and can’t be applied in software development industry.
�Software Quality Assurance Concept is a set of activities that assures for a certain software the standard of quality expected by customers.
� Fit-for-purpose� Accurate, correct� Performant
� User-friendly
11 copyright LMS International - 2011
Cost of correcting bugs across development lifecycle
Software Project Survival Guide
Steven McConnell, 1998
12 copyright LMS International - 2011
Virtual.Lab Product Creation Process & supporting Software Tools
13 copyright LMS International - 2011
Virtual.Lab Product Creation ProcessDifferent phases and stages – 1 major release per year
�Business scoping
�Planning
�Development cycle with sprints (Agile)
�Quality Control (QC) – internal & external launch
�Deployment and maintenance
+/- 1 year
14 copyright LMS International - 2011
Frontload quality into the sprints of the development cycle
� Improve quality and predictability of release
� Improve development efficiency and workload balance� Developer fixes issues close to when he/she has implemented the functionality� Early start of development of QC scenarios � Facilitates trade-off discussions on improving quality / scope of current content versus adding
more content
� More efficient execution of QC� Reduce backlog at start of QC phase� Reduced number of blocked scenarios in QC period� QC more dominated by regression tests on new and existing functionality� Should allow reducing duration of QC period, automating scenarios and/or preparing for next
release better
Sprint 1 Sprint N-1 Sprint NQC – phase 1
Scenario testing & fixing
QC – phase 2Regression &
stress testing
Sprint exit criteria
Out of Development/start QC
Out of Quality ControlExit criteria
Pre-production
Front load quality
Development of new functionality
15 copyright LMS International - 2011
Software tools supporting the development process
� Defect tracking system Mantis for internal and customer problem reports
� Team responsible
� Priority (escalated, major, …)
� Goes through different stages
• New
• Assigned
Assigned to developer
• Resolved
Fixed by developer and ready for Cross-testing
• Closed
� Web-based planning tool and time logging (internal tool)
� Document management (Lotus Notes)
� Metrics in excel through querying databases
� Source code management system (e.g. CVS, Mercurial)
� Central Software build environment
16 copyright LMS International - 2011
Software validation in development phase
Sprint 1 Sprint N-1 Sprint NQC – phase 1
Scenario testing & fixing
QC – phase 2Regression &
stress testing
Sprint exit criteria
OOD/start QC
OOQCExit criteria
Pre-production
Front load quality
17 copyright LMS International - 2011
Requirements and stories – Approval
� Requirements are provided by product management, application export or customer services
� Development provides feedback and approves the requirement
� Requirement owner and product development have iterations to refine requirement
and break requirement into smaller pieces that have value
each to end-user, so-called stories. Stories describe how the software will actually work for the user in the words of development and typically provide use case.
� Stored in content management system
� Assigned to a development sprint
� Approved by owner of requirement
� Expected to be ready before start of sprint
18 copyright LMS International - 2011
Milestone tests
� Done developments are validated by the product (line) managers, application experts
and/or Customer Services through milestone tests. The milestone test aims at the
following:
� Check before OOD that the functionality is implemented according to approved
specification or story, and that it is correct. This could be done through testing on a
small model.
� Review ease-of-use and the fit-for-purposeness by testing on realistic models
� Review the performance of the implementation by testing on realistic models
� Outcome
� Approval or re-test needed
� Logged Problem Reports (PRs)
19 copyright LMS International - 2011
Break down in stories + effort estimate
Validation of developed story in sprint (development cycle)
Test
ob
jects
Requirement docincluding scenario & data Approval by Product
Development
MS test session (PM-PD)
Fix MS feedback
Work Breakdown Structure -Analysis & Design
Coding
Code review
Prepare for MS test
Re-test/Verify Problem Reports (PRs)
Out of Milestone Test Sprint
Out of Consolidation Sprint
MS approved/story done
Consolidation testing and fixing
Exit criteria
Sprint x –
Develo
pm
ent of sto
ries
Sprint x-1
Demo
Approval by Product Management
20 copyright LMS International - 2011
QC phase
Sprint 1 Sprint N-1 Sprint NQC – phase 1
Scenario testing & fixing
QC – phase 2Regression &
stress testing
Sprint exit criteria
OOD/start QC
OOQCExit criteria
Pre-production
Front load quality
21 copyright LMS International - 2011
Quality Control (QC) Process Description
�QC process is build on Scenario Based Concept : the tester follows a certain suite of steps, usual involving pre-processing, solver and post-processing, and check the validity of the test with a clear and measurable set of exit criteria.
�Different types of tests�Manual tests execution
�It follows a static test plan with milestones – predefined at the beginning of the QC cycle
�Automated tests�Are a collection of automated QC scenarios�Usual procedure is to transform a manual scenario test into a automated scenario test�The automated tests are executed with each QC build, along QC cycle
�Regression Tests�They follow a dynamic test plan – it is defined based on the quality evolution during the tests execution
�Early Visibility Tests� Test session in presence of the customer before software release
�Stress Tests by application experts (out-of-the-box)
�Pre-Production and Installation test
22 copyright LMS International - 2011
QC Phase – Manual tests
� Information about Manual Tests� The number of manual scenarios is increasing each release, to cover the new functionalities.
� Details regarding the Manual Execution� Starting point of QC scenario is a data set to import or database saved with previous version� Also, it is necessary to have the exit criteria data:
� One XML file and one related Excel file per each analysis case� The Virtual.Lab model and solution that generated the exit criteria
� A prototype of the QC report (describing the steps to follow, the exit criteria and other details) is needed to execute the QC scenario
� Deliverables after Manual execution of the QC scenario:� An updated QC report with the conclusions of the tests� The Virtual.Lab model that generated the exit criteria results
�Follow-up progress of test plan in weekly QC meeting
Important :Test enough side paths
in the scenario
Integrate representative
data/scenarios of customers
23 copyright LMS International - 2011
QC Phase - Automated tests – General overview
�Why we need to have Automated Tests part of the QC process?
� AT are testing the whole scenario: pre processing,
solver and post processing.
� AT provide a fast and reliable feedback regarding
the quality of a certain build.� AT don’t experience the human errors, especially
during the execution of some complex scenarios.
�How the AT are running?� The core part of the AT is the Visual Basic
Application, driving Virtual.Lab to execute designed
steps
� Most of the scenarios need a set of initial files.
� The QC report is web based and includes a quantitative report (at the suite level) and qualitative
report (at the model level)
�What are cons for AT?� AT don’t test the GUI� AT take significant effort to develop in Visual Basic
Application
�Challenge� Design of test for keeping cost of maintenance under control (align with evolutions of the software)
24 copyright LMS International - 2011
QC Phase - Automated tests – Individual Exit Criteria
�Individual Exit Criteria from automated tests� Each tested variable is compared against a target set of data
�A graph is automatically generated for
each tested variable containing computed
data and target data
�The graphs are stored as XML files for future reference
25 copyright LMS International - 2011
QC Phase - Early Visibility Tests (customer involved)
� Three types of EVTs (Early Visibility Tests) with customer
� Alpha version
• Scoped in time (e.g. 6 weeks testing period)
• Agree with customer on how training, supporting, reporting and responding to
feedback is handled
• Regular follow-up meetings between PD and customer
� Hands-on (1-2 days on-site)
• On-site feedback - customer uses product
• Participates return to LMS with customer models for QC
� Demo (from few hours to 1 day)
• On-site or through WebEx
26 copyright LMS International - 2011
QC Phase - Manual Regression Tests
QC Regression Manual Tests
� Repeat of manual tests done before to manually verify all source code
changes did not introduce regressions
� The selection of tests is generated based on the statistics of PRs generated
during development and QC execution
� QC regression tests are executed by QC team
Sprint 1 Sprint N-1 Sprint NQC – phase 1
Scenario testing & fixing
QC – phase 2Regression &
stress testing
Sprint exit criteria
OOD/start QC
OOQCExit criteria
Pre-production
Front load quality
27 copyright LMS International - 2011
QC Phase – Stress tests
� Stress tests� Are tests covering a certain functionality, used in a customer specific context
� Tests are done by Customer Service / Product
Management / Experts in the field
� They have not a written a QC scenario or a QC
report; testers are logging PRs and report at the end when the tests are done
28 copyright LMS International - 2011
QC Coordination and Follow-up Progress
� QC Meetings� They have as goal to keep track of the progress of the QC process� At the QC meetings are participating the Product Development Director, Product Development Managers for each
Product Line and QC manager
� QC manager is providing statistics regarding QC execution, bugs fixing and verification, unit tests and automated
tests.
� Using this statistics and also Product Development Managers viewpoints, we conclude on the QC current status for each product line and, if necessary, the actions needed to correct the status.
� The statistics and the conclusions of the QC meetings are compiled in a weekly QC report
�Metrics� Progress scenario execution� Open PRs per severity
� PRs to cross-test
� Success/failure of ODTs
and automated QC tests
� Exit criteria
� GO/NO GO Meeting – Review delayed issues with
� Product Management�Customer Services
� Product development
PRs target Rev 10
1 2 3 4 5 6 7 8 9
QC week number
Am
ou
nt
of
PR
s a
t e
nd
of
we
ek
Incoming S0+S1
Fixed S0+S1
Open S0+S1
Fixed S2+S3
Fixed features
29 copyright LMS International - 2011
Pre-production - Installation And Light Tests
� Installation and Light Tests� These are tests executed during pre-production and using the final
release build
� Light tests (e.g. Tutorials) are executed for each product line
� Various installation tests on all supported platforms are validated
� Tests from master DVD before production
� Tests conclude in short time span (few days only)
Sprint 1 Sprint N-1 Sprint NQC – phase 1
Scenario testing & fixing
QC – phase 2Regression &
stress testing
Sprint exit criteria
OOD/start QC
OOQCExit criteria
Pre-production
Front load quality
Release
30 copyright LMS International - 2011
ODTs
Coverage of ODTs and Scenario Based Testing
Fu
nctio
nality
1
Fu
nctio
nality
x
Full Coverage
Fu
nctio
nality
2
Fu
nctio
nality
n-1
Fu
nctio
nality
n
QC Scenarios Based Testing – Automated and Manual Tests
ODTs - focus on testing each functionality independently
Scenarios Based Testing – focus on testing of more functionalities, organized in a similar way as the customers
may use
. . .. . .
31 copyright LMS International - 2011
QC Coverage Concept
�What is the Coverage Concept?� Identify what test models and scenarios cover what functionality and identify the missing pieces� Reflects also how many functionalities are tested in a certain interval of time or for a certain goal
� Requires tools that track what the tester/using is doing, logging the details of the user actions and link it to the test models/scenarios
� Allows to develop impact analyzer, i.e. a fix for a software issue can be linked to test models/scenarios and re-run those tests.
VL Name Models Items
Accumulator 1 1
Aero Aero 0 0
Aero Hydro 2 2
Aero Vehicle 3 4
Amplifier 27 114
Beam 6 6
Body 426 2317
BodyFromNewPart 61 1660
BodyFromExistingPart 65 608
Bracket Joint 111 1235
Bushing 0 0
Bushing Link 2 2
Radial Bushing 3 3
Standard Bushing 137 1236
Spherical Bushing 0 0
Bump Stop 2 2
Frequency Bushing 0 0
Contact 0 0
Legacy CAD Contact 0 0
CAD Contact 3 12
Check Valve 1 2
Extruded-Surface-to-Revolved-Surface Contact 4 25
Pipe Roller Contact 0 0
Point Ground Contact 0 0
Point-to-Point Contact 7 28
Sphere-to-Rail Contact 0 0
Sphere-to-Extruded-Surface Contact 11 108
Sphere-to-Revolved-Surface Contact 5 80
Sphere-to-Ground Contact 0 0
Control Function 10 69
Csys 0 0
CV Joint 50 127
Cylindrical Joint 151 348
Dead Zone 3 6
32 copyright LMS International - 2011
Deployment and Maintenance
33 copyright LMS International - 2011
Virtual.Lab Product Development – Software in shipment
• Continous improvement (lessons learned)
• Measure success of quality improvement program (frontloading quality into sprints) through comparison with previous release
� Listen to feedback of customers, regions, PM, ES, …
� Responsiveness to customer bugs
• Hot fix procedure for escalated issues
• Metrics on major customer bugs fixed in 5 months
• Metrics on major and minor bugs fixed in in 12 months
� Responsiveness to customer features
� Customer Satisfaction survey
1 2 3 4 5 6 7 8 9 10 11 12 13
Weeks after release (master version)
Am
ou
nt
of
bu
gs
10 Blocking bug
9 Blocking bug
10 Blocking + Major
9 Blocking + Major
34 copyright LMS International - 2011