National Instruments Confidential

Optimizing Test System Development

Wyatt Meek

VISTA Product Manager

Agenda

• Boeing Test Engineering Priorities• V I Engineering Overview• Addressing Common Software Engineering Issues

– Configuration Management– Project Management– Reuse

• Discussion

Boeing Priorities

• What are the top initiatives within the test engineering group?

• How do you currently measure success/failure? • What issues does your group currently face?• Are your suppliers required to follow

process/development guidelines?

History of V I Engineering

• Founded in 1992• Select NI Alliance Member • Four Locations

– Farmington Hills, MI (HQ)– St. Paul, MN– Indianapolis, IN– Los Angeles, CA

• Business Units– Engineering Services– Engineering Information Management Systems (EIM)– VISTA

Multi-Industry Experience

Simplified Waterfall Model for Software Design

research/definition

requirementsspecification

design coding testingship/

maintain

research/definition

requirementsspecification

design coding testingship/

maintain

All Too Often…

LabVIEW makes it very easy to quickly put something together

Developers do not follow a process

Case Study

Source: CMU/SEI-94-TR-013

Case Study

Source: CMU/SEI-94-TR-013

Common Software Engineering Issues

• Process Issues– Configuration Management – Project Management– Requirements Management– Quality Assurance (V&V)

• Development Issues– Scalability – Reusability

This leads to poor code quality and inefficient use of resources

Addressing the Issues

• SEI (Software Engineering Institute www.sei.cmu.edu)– SW-CMM– CMMI

• IEEE Software Engineering Standards (www.standards.ieee.org/software)– 828, 830, 1028, 1220,1058.1

• ISO/IEC– 9001:2000, 12207, 15288

• Industry Guidelines– FDA– DoE– FAA

Process Implementation

• Process Analysis and Definition– Prioritize improvement initiatives– Define development process– Develop templates and guides– Select automation tools

• Implement Automation Tools• Training

– Software development process– Development and process improvement tools– Common programming techniques and architecture

Challenges with Process Implementation for LabVIEW and TestStand• LabVIEW File Handling

– Embedded Source and Object– Static Linkage

• Metrics – Complexity– Size

• Design Methodology– Style – Patterns

Configuration Management

Configuration Management

• Why use Configuration Management?– To know what you’ve built (configuration)– To identify and track needed changes– To track status of configuration items– To deliver known and verified configurations

• Four main Configuration Management Functions– Configuration identification– Configuration control– Configuration status accounting– Configuration authentication

CM Terminology

• SCC – Source Code Control• Configuration Item – files placed in SCC• Baseline – specific versions of CI’s at a specific time in the

product’s lifecycle (ie Beta Release, Rev 2.0.1)• Status Accounting – knowing which version belongs in a

baseline and why• Label – assigned as an attribute to a specific version of a file or

application, may be used to show status accounting (Alpha, Beta, Reviewed, Verified, etc.)

Example CM Plan and Process

• Configuration Items (specifications, code, daq settings, support files, exe’s)

• Configuration Control– User roles (developer, lead developer, project manager, operations)– Permissions (read/write access, labels)– Change procedures

• Status Accounting– Labeling procedures

• Authentication– Audits– Integrity checks

CM Recommendations

• Treat read only VIs as locked • Establish common development environment• Do not allow multiple checkouts (merging impossible)• Avoid branching SCC projects (cross linking and merging) • Do not use LLBs during development (can not locate

modifications within)• Use labels during development (assign status)• Deploy exe or use integrity check

CM Integration Solutions - demos

• Basic– LV 6.0.2 PDS built in SCC– SCC repository independent of LV

• Intermediate– LV 6.0.2 PDS and newer built in SCC with VSS or Perforce

• Advanced– VISTA CM Tools for LV 6.1 (Base, Full, PDS) and newer

with VSS or ClearCase (others in development)

Project Management

Importance of Metrics

• Confirm Estimates• Project Monitoring• Cost Justification • Risk Analysis • Impact of Change • Track Improvement Initiatives

– Reuse– Productivity– Diagnose root cause of project failure

“You can’t manage what you can’t measure”“You can’t manage what you can’t measure”

Standard Metrics (text based languages)

• Hours• SLOC• Function Points• Cyclomatic Complexity• EVA (Earned Value Analysis)

LabVIEW Metrics

• Hours• SLOC Equivalent Metric

– Nodes (VI Metrics)– GOBs (Graphical OBjects calculator)

• Cyclomatic Complexity (possible, but not yet available)• EVA

TestStand Metrics

• Hours• SLOC Equivalent Metric

– Lines in seq file• EVA

Metrics Trials

• LabVIEW Developer Exams– GOBs more reliable than nodes

• V I Engineering– ~10 GOBs/hr for product development and regulated industries– ~25 GOBs/hr for automotive solutions

• Ball Aerospace– 8-10 GOBs/hr for 2 week to 2 man year projects

• General Dynamics (CMMI Level 3)– 8.7 GOBs/hr avg for 8000 man hours of work

Variables

• Style • Design Patterns / Architecture used (GOOP vs event

driven vs traditional)• Quality Level Requirement

– Industry (regulated vs non-regulated)– Application Type (product, manufacturing test system, DVT,

etc.)

Establishing a Project Plan with Metrics

• Review Requirements• Decompose Requirements into Tasks/Components

– Complete preliminary design– MS Project example WBS

• Estimate Development Tasks based on Historical Metrics and Risk (spreadsheet example)

• Select Lifecycle • Identify Quality Goals• Assign Estimate to Non-development Tasks based on

Historical Metrics based on Quality Goals and Lifecycle

Establishing Project Control

• Identify Project Deliverables for each SLM Phase• Build Master Project Schedule• Assign Planned Value to Deliverables with Project

Schedule– SW deliverables back-fire GOBs from hours estimates

Project Monitoring with Metrics

• Track % complete for Deliverables– Use GOBs, SLM phase, status on tasks for objective

feedback on software deliverables• Roll Up % Complete to EV• Track EV vs. Planned Value, Actual Cost

Project Monitoring with Metrics

• Track % complete for Deliverables– Use GOBs, SLM phase for objective

feedback on software deliverables

• Roll Up % Complete to EV• Track EV vs. Planned Value,

Actual Cost• Manage by Budget at Completion

vs. Estimate at Completion Variance, etc.

Development by LifeCycle PhaseWorkspace: Stopwatch Project Rev2.0

Design19%

Development23%

Test28%

Implementation19%

Rework11%

Design

Development

Test

Implementation

Rework

Project Monitoring with Metrics

• Track % complete for Deliverables– Use GOBs, SLM phase for

objective feedback on software deliverables

• Roll Up % Complete to EV• Track EV vs. Planned Value,

Actual Cost• Manage by Budget at

Completion vs. Estimate at Completion Variance, etc.

GOBS vs TimeStopwatch Project

0

200

400

600

800

1000

1 2 3 4 5 6 7 8

Months

GO

BS

(100

0)

Core Classes

Plug Ins

Database

Total

Reuse

33

Coding for Reuse in LabVIEW

• Benefits of LabVIEW Reuse• What is Required for Effective Reuse with LabVIEW?• Reuse Tips

Benefits of LabVIEW Reuse

• Decreased Development Time• Reduced Risk• Improved Quality• Lower Overall Cost

Requirements for Reuse

• Reuse Development Philosophy– Reuse must be a part of the organizational culture

• Coding Standards / Style Guides– Coding style needs to be familiar to entire team

• Training• Source Code Control

36

Reuse Development Philosophy

• Always Evaluate Reuse Potential• Generalize Code; Avoid project specific code• Document for Reuse and Maintenance• Have Reuse Reviews at the Beginning and the End of the

Project

CM and Reuse

• Avoid VIs with the Same Name• Put LabVIEW Reuse Libraries in user.lib (treat them

similar to vi.lib)– Organize VIs based on code categories– Limit check-out/check-in access to reuse librarians

• Categorize LabVIEW Libraries– General reuse, product line reuse, product specific

• Maintain Backward Compatibility– Create a new VI before losing backward compatibility

38

Example Reuse Library Directory Structure in user.lib

• ..\VIE Library.lib\ArrayTools.llb• ..\VIE Library.lib\ConfigUtilities.llb• ..\VIE Library.lib\ControlFunctions.llb• ..\VIE Library.lib\Dialogs.llb• ..\VIE Library.lib\ErrorTools.llb• ..\VIE Library.lib\FileUtilities.llb• ..\VIE Library.lib\GUI.llb• ..\VIE Library.lib\GUI.llb\Soft

Keyboard.llb• ..\VIE Library.lib\NumericTools.llb• ..\VIE Library.lib\OS Utilities• ..\VIE Library.lib\OS Utilities\Windows.llb

• ..\VIE Library.lib\SharedTYPE.llb• ..\VIE Library.lib\Sound.llb• ..\VIE Library.lib\StateMachine.llb• ..\VIE Library.lib\StringTools.llb• ..\VIE Library.lib\Time.llb• ..\VIE Module.lib\AI and Calibration.llb• ..\VIE Module.lib\DTS Engine.llb• ..\VIE Module.lib\Formula Evaluator.llb• ..\VIE Module.lib\SecurityModule.llb• ..\VIE Module.lib\System Integrity Module.llb

• (.llb extensions are used for directory names to be consistent if converted to VI library files)

39

LabVIEW Style Affects Reuse

• Using a common style and good VI documentation can significantly improve reuse

• LabVIEW Development Guidelines, chapter 6– devstyle.pdf in ..LabVIEW\Search the LabVIEW Bookshelf…

• Use a code review checklist• Avoid overly complex code• Use templates to help standardize coding style

between multiple developers

40

Architecture Reuse Tips

• Use layered architectures; limit interaction to adjacent layers

• Avoid project specific dependencies; use loose coupling– Use Graphical Object-oriented Programming (GOOP)– Avoid global variables– Use standard configuration tools– Separate user-interface VIs from the main application

• Think in terms of reusable components– Encapsulate functionality and data (use GOOP)– Remember that how features specific to an architecture are

used may limit reuse to that architecture.

VISTA

Software engineering approach to the management, development and release of LabVIEW and TestStand software.

•Process Consulting•Training•Development and Process Tools

VISTA Configuration and Project Management Tool

• Workspace• Metrics• CM Functionality

– LV and TS integration– Baselines– Build with integrity (exe or native)– File comparison– File load check

VISTA Productivity and Reuse Tools

• Help Generator• Renaming and

Relocation Tool • Documentation Tool• Reuse Library• Style Guide

VISTA Quality Assurance

• Node/Path Tracker• Cyclomatic Complexity• Traceability Tool

VISTA GOOP Tools

• GOOP Inheritance Toolkit– GOOP Wizard 3 with

Inheritance– Icon Editor– Development Distribution

Tool

GOOP Background

• Scalability – this non-OOP is not easy to grow or maintain

GOOP Background

• Scalability – layered, OOP approach

How Can V I Engineering Help You?

• Process Assessment• Custom and Advanced Training• Tools • Project Support• Architecture Design

Learn more at www.vista.viengineering.com