Class 2

9/7/20049/7/2004 Steven Woody, France Antelme, AriSteven Woody, France Antelme, Ario Bigattini, Jaseem Masood, All righo Bigattini, Jaseem Masood, All rights reservedts reserved

11

CES 592 Telecommunications CES 592 Telecommunications System Product VerificationSystem Product Verification

Sonoma State UniversitySonoma State University Fall 2004Fall 2004

Class Lecture 2: Class Lecture 2:

High-Availability Architectures,High-Availability Architectures,

Testing Constraints, Network Testing Constraints, Network Failures, and Test StrategiesFailures, and Test Strategies


22

Telecom System ArchitectureTelecom System Architecture History – Some telecom network History – Some telecom network

failuresfailures High Availability definedHigh Availability defined HW / SW ArchitecturesHW / SW Architectures Unique problems/constraints/goalsUnique problems/constraints/goals

– ISO 9001 & TL 9000ISO 9001 & TL 9000– Other constraintsOther constraints

Testing for High-AvailabilityTesting for High-Availability


33

Communications Network FailuresCommunications Network Failures October 27, 1980 ARPAnet collapse – October 27, 1980 ARPAnet collapse –

self-propagating error required all self-propagating error required all systems to be restarted. 4 hr outagesystems to be restarted. 4 hr outage

January 15, 1990 AT&T nationwide January 15, 1990 AT&T nationwide outage – error in C code caused self-outage – error in C code caused self-propagating 9 hour outage, 5 million propagating 9 hour outage, 5 million blocked callsblocked calls

June 27, 1991, 8 million line outage – June 27, 1991, 8 million line outage – self-propagating error in untested code self-propagating error in untested code patchpatch


44

Communications Network FailuresCommunications Network Failures Famous Fiber Cuts (“Backhoe Fade”)Famous Fiber Cuts (“Backhoe Fade”)

– 11/19/1990, 150,000 phone lines, 11/19/1990, 150,000 phone lines, outage lasted several hoursoutage lasted several hours

– 12/4/1991, 100,000 phone lines, several 12/4/1991, 100,000 phone lines, several hours, interrupted FAA flight control and hours, interrupted FAA flight control and NY Mercantile ExchangeNY Mercantile Exchange

From From Computer Related RisksComputer Related Risks, , by Peter G. Neumannby Peter G. Neumann


55

Telecom System ArchitectureTelecom System Architecture

History – Some telecom network History – Some telecom network failuresfailures

High Availability definedHigh Availability defined HW / SW ArchitecturesHW / SW Architectures Unique problems/constraints/goalsUnique problems/constraints/goals




66

High Availability DefinedHigh Availability Defined 99% Uptime = 3 days downtime / year99% Uptime = 3 days downtime / year 99.9% Uptime = 8 hours downtime / year99.9% Uptime = 8 hours downtime / year 99.95% = 4 hours, 23 minutes / year99.95% = 4 hours, 23 minutes / year 99.99% = 53 minutes / year99.99% = 53 minutes / year 99.999% = 5 minutes / year (‘five nines’)99.999% = 5 minutes / year (‘five nines’) 99.9999% = 30 seconds / year99.9999% = 30 seconds / year

Reference Telcordia GR-1110, TR-332Reference Telcordia GR-1110, TR-332


77

High Availability DefinedHigh Availability Defined

Expectation for reliable communications, Expectation for reliable communications, even in the event of major disasters – even in the event of major disasters – when communications are needed mostwhen communications are needed most

Service Level Agreements (SLA)Service Level Agreements (SLA) Automatic and instantaneous recovery Automatic and instantaneous recovery

from internal & external faultsfrom internal & external faults Very high Mean Time Between Failure Very high Mean Time Between Failure

(MTBF) 100,000+ hours(MTBF) 100,000+ hours


88







99

High Availability Telecom SystemHigh Availability Telecom System HW & SW Architectures HW & SW Architectures

Designed for High AvailabilityDesigned for High Availability– Single fault tolerantSingle fault tolerant– Low probability of double faultLow probability of double fault

Passive backplane (higher reliability – MTBF 1,000,000+ hrs)Passive backplane (higher reliability – MTBF 1,000,000+ hrs) Modular Design - Fault in one card won’t impact other cardsModular Design - Fault in one card won’t impact other cards Separation of control plane and data planeSeparation of control plane and data plane Card Redundancy - “Hot Redundancy” (maintain sync)Card Redundancy - “Hot Redundancy” (maintain sync) On-line HW Replacement – “Hot Swappable”On-line HW Replacement – “Hot Swappable” In-Service errorless SW, FW, FPGA Upgrades/DowngradesIn-Service errorless SW, FW, FPGA Upgrades/Downgrades Alarm logs, audit logs, and provisioned settings preserved Alarm logs, audit logs, and provisioned settings preserved Microprocessor Watchdog Timers / HeartbeatMicroprocessor Watchdog Timers / Heartbeat User errors minimized (major source of outages)User errors minimized (major source of outages) Graceful shutdown / restorationGraceful shutdown / restoration ““Hardened” hardware for operation at extreme power, Hardened” hardware for operation at extreme power,

temperature, humidity, corrosion, ESD, & vibration levelstemperature, humidity, corrosion, ESD, & vibration levels Network designed for Link and Node protectionNetwork designed for Link and Node protection


1010



High Availability definedHigh Availability defined HW / SW ArchitecturesHW / SW Architectures Unique Unique

problems/constraints/goalsproblems/constraints/goals– ISO 9001 & TL 9000ISO 9001 & TL 9000– Other ConstraintsOther Constraints



1111

Unique Problems, Constraints, Goals: Unique Problems, Constraints, Goals: ISO 9001 & TL 9000ISO 9001 & TL 9000

ISO 9001 Covers Quality Assurance in ISO 9001 Covers Quality Assurance in Design/Development, Production, Installation and Design/Development, Production, Installation and ServicingServicing

Certification process:Certification process:– Document quality processes of your organizationDocument quality processes of your organization– Audit by RegistrarAudit by Registrar– Certification and follow-up inspectionsCertification and follow-up inspections

““With ISO 9000 you can still have terrible processes and With ISO 9000 you can still have terrible processes and products. You can certify a manufacturer that makes life products. You can certify a manufacturer that makes life jackets from concrete, as long as those jackets are made jackets from concrete, as long as those jackets are made according to the documented procedures” according to the documented procedures” Richard Buetow, Richard Buetow, Director of Corporate Quality, MotorolaDirector of Corporate Quality, Motorola


1212

Unique Problems, Constraints, Goals: Unique Problems, Constraints, Goals: ISO 9001 & TL 9000ISO 9001 & TL 9000


1313

Unique Problems, Constraints, Unique Problems, Constraints, Goals: ISO 9001 & TL 9000Goals: ISO 9001 & TL 9000

TL 9000 is a quality management TL 9000 is a quality management process for design, development, process for design, development, manufacturing, delivery, installation, manufacturing, delivery, installation, and maintenance of and maintenance of telecommunications hardware and telecommunications hardware and software.software.

““The organization shall establish and maintain The organization shall establish and maintain a method to trace documented a method to trace documented requirements through design and test.”requirements through design and test.”


1414

Unique problems, constraints, goals: Unique problems, constraints, goals: Other Constraints Other Constraints

Can not test on a live operational Can not test on a live operational network, therefore the testing network, therefore the testing configuration must be representativeconfiguration must be representative

Must comply to numerous national Must comply to numerous national and international standards from and international standards from multiple standards bodiesmultiple standards bodies


1515







1616

The Product Development CycleThe Product Development Cycle

System Spec

Software

Spec

New Product

Idea

New Product

Idea

Software Development SW Unit Test

HW-SWIntegrationHardwareSpec Hardware

DevelopmentHW Unit T

est

Product Verification

Engineering Developmentfunctions

Product Verificationfunctions

Customer & marketDriven inputs

Product LineManagement &

Engineeringinputs

Release to

manufacture


1717

Product Verification PhaseProduct Verification Phase

Formal Product Verification Phase

Software Verification

HW-SWIntegration

Test

HW Compliance & Agency approvals

Release to Production

HW Stress Testing

HW Standards/Reqts testing

Software Verification

VolumeProduction


1818

Testing for High AvailabilityTesting for High Availability

““Networks are very complex systems Networks are very complex systems and the only way to test them is to and the only way to test them is to partition them into manageable partition them into manageable layers and functions. Doing this is layers and functions. Doing this is truly an art”truly an art”

- Robert Buchanan, Jr.- Robert Buchanan, Jr.


1919


Systematic, structured testingSystematic, structured testing ANSI / Telcordia / ITU-T Standards-based ANSI / Telcordia / ITU-T Standards-based

testingtesting Environmental testing for hardwareEnvironmental testing for hardware Stress / Load testing for softwareStress / Load testing for software HW & SW fault insertion testingHW & SW fault insertion testing Interoperability testingInteroperability testing Soak testing – continuous operationSoak testing – continuous operation Statistical sampling for manufacturingStatistical sampling for manufacturing


2020

Testing for High AvailabilityTesting for High Availability Systematic, structured testingSystematic, structured testing

– Planned test strategyPlanned test strategy– Thorough, well though out test plan:Thorough, well though out test plan:

Test cases traceable back to specificationsTest cases traceable back to specifications Trade-off decisions made for permutations not performedTrade-off decisions made for permutations not performed Test plan includes positive and negative test casesTest plan includes positive and negative test cases Test case pre-defines unambiguous pass/fail criteriaTest case pre-defines unambiguous pass/fail criteria Test environment is described in detailTest environment is described in detail Risks are anticipated and managed with contingency plansRisks are anticipated and managed with contingency plans

– And yet, controlled randomness in the test caseAnd yet, controlled randomness in the test case– Prioritization: run the important tests earlyPrioritization: run the important tests early– Focus on areas of greatest risk – system state transitionsFocus on areas of greatest risk – system state transitions– Learn from the bugs that you findLearn from the bugs that you find– Use of automation to increase coverage, reduce scheduleUse of automation to increase coverage, reduce schedule– Continuous refining, improving of test plan & test casesContinuous refining, improving of test plan & test cases

Portions from Portions from High Quality Software EngineeringHigh Quality Software Engineering, by Ross Collard, by Ross Collard


2121


Major Causes of Defects NOT Being FoundMajor Causes of Defects NOT Being Found 36% Scope: scenario was beyond the test strategy36% Scope: scenario was beyond the test strategy 21% Permutations: an untested combination failed21% Permutations: an untested combination failed 9% Stochastic: random failure that did not occur during 9% Stochastic: random failure that did not occur during

testingtesting 6% Process: non-compliance in the way the test was 6% Process: non-compliance in the way the test was

performedperformed 5% Oversight: the problem was missed by the tester5% Oversight: the problem was missed by the tester 3% Coverage: the test scenario was not included in the test 3% Coverage: the test scenario was not included in the test

strategystrategy 3% Incomplete test: the scenario was in the test plan, but 3% Incomplete test: the scenario was in the test plan, but

not included in the test casesnot included in the test cases(Percent of all defects which were not found)(Percent of all defects which were not found)

Study by Tellabs, 1998Study by Tellabs, 1998


2222


Systematic, structured testingSystematic, structured testing ANSI / Telcordia / ITU-T Standards-ANSI / Telcordia / ITU-T Standards-

based testingbased testing Environmental testing for hardwareEnvironmental testing for hardware Stress / Load testing for softwareStress / Load testing for software HW & SW fault insertion testingHW & SW fault insertion testing Interoperability testingInteroperability testing Soak testing – continuous operationSoak testing – continuous operation Statistical sampling for manufacturingStatistical sampling for manufacturing


2323


ANSI / Telcordia / ITU-T Standards-ANSI / Telcordia / ITU-T Standards-based testingbased testing– Verify that interfaces meet requirements Verify that interfaces meet requirements

of standards (GR-253…)of standards (GR-253…)– Verify fail-over performance meets Verify fail-over performance meets

standards (60 mS)standards (60 mS)


2424





2525

Elements of Hardware VerificationElements of Hardware Verification

Hardware Verification

Compliance &

Agency Approvals

Stress TestingHALT/HASS

Standards basedTesting

Physical LayerLogical Layer

EMCSafetyNEBSTelecom

Design Stress TestingAccelerated Life-cycle Testing

(Beyond normal operating limits)Where does it break?


2626


Environmental testing for hardwareEnvironmental testing for hardware– Operation over temperature, supply Operation over temperature, supply

voltage, vibrationvoltage, vibration– Monitor software performance (traffic, Monitor software performance (traffic,

alarms) during environmental testingalarms) during environmental testing


2727





2828

Testing for High AvailabilityTesting for High Availability Stress / Load testing for softwareStress / Load testing for software

– Multiple, simultaneous traffic typesMultiple, simultaneous traffic types– Errors on input interface (example: bit errors)Errors on input interface (example: bit errors)– Maximum user activity: db backup, multiple Maximum user activity: db backup, multiple

session launch, multiple data requestssession launch, multiple data requests– Bottleneck / over-subscription of data trafficBottleneck / over-subscription of data traffic– Fail-over testing: single failureFail-over testing: single failure– Fail-over testing: double failureFail-over testing: double failure– Startup under stress, load, & errorsStartup under stress, load, & errors– Alarm hysteresis, holdoff & alarm stormsAlarm hysteresis, holdoff & alarm storms– Goal: more stress/load than SW will ever see Goal: more stress/load than SW will ever see

operationally (find the breaking point)operationally (find the breaking point)


2929





3030


HW & SW fault insertion testingHW & SW fault insertion testing– Any hardware subsystem / module failureAny hardware subsystem / module failure

CPU resetCPU reset Power supply failurePower supply failure Oscillator failureOscillator failure Data bus / Address bus line fail high/lowData bus / Address bus line fail high/low Memory corruptionMemory corruption

– File corruptionFile corruption– Resource exhaustion (memory, file handles, Resource exhaustion (memory, file handles,

sockets, semaphores…)sockets, semaphores…)– User error – software should protect againstUser error – software should protect against


3131





3232

Testing for High AvailabilityTesting for High Availability Interoperability testingInteroperability testing

– Testing interfaces which pass data back and Testing interfaces which pass data back and forthforth

– Compatibility testing with other equipmentCompatibility testing with other equipment– Compatibility testing with other vendor’s Compatibility testing with other vendor’s

equipmentequipment– Consider both hardware and software versionsConsider both hardware and software versions– Compatibility with current, prior, and next Compatibility with current, prior, and next

version of:version of: Operating System (Unix, Solaris, Windows)Operating System (Unix, Solaris, Windows) Java / JREJava / JRE

– Configurations to be tested must be prioritized Configurations to be tested must be prioritized by their importance and riskby their importance and risk


3333





3434


Soak testing / longevity testing – Soak testing / longevity testing – continuous operationcontinuous operation– Telecom equipment designed for continuous Telecom equipment designed for continuous

operation for months, years, even decadesoperation for months, years, even decades– Run in lab for 14 hours? 7 days? 3 weeks?Run in lab for 14 hours? 7 days? 3 weeks?– 1 system for 1 year = 8 systems for 45 days?1 system for 1 year = 8 systems for 45 days?– Mixture of clean and errored trafficMixture of clean and errored traffic– Monitor for traffic interruptions, alarmsMonitor for traffic interruptions, alarms– Monitor performance counters: Monitor performance counters:

Error-free seconds counterError-free seconds counter Bad packet counterBad packet counter


3535



testingtesting Environmental testing for hardwareEnvironmental testing for hardware Stress / Load testing for softwareStress / Load testing for software HW & SW fault insertion testingHW & SW fault insertion testing Interoperability testingInteroperability testing Soak testing – continuous operationSoak testing – continuous operation Statistical sampling for Statistical sampling for

manufacturingmanufacturing


3636


Statistical sampling for Statistical sampling for manufacturingmanufacturing– System testing done with small number System testing done with small number

of prototype hardware cardsof prototype hardware cards– Reliability testing must be done on large Reliability testing must be done on large

sample of production hardware cardssample of production hardware cards– Must be done for a long period to be Must be done for a long period to be

statistically significant: thousands of statistically significant: thousands of hours of total run timehours of total run time


3737

ReferencesReferences““Computer Related Risks”, Computer Related Risks”, Peter G. NeumannPeter G. Neumann

““Code Complete”, Code Complete”, Steve McConnellSteve McConnell

““Software Testing and Quality Assurance”, Software Testing and Quality Assurance”, Ross CollardRoss Collard

True Random Numbers: True Random Numbers: http://www.random.org/nform.htmlhttp://www.random.org/nform.html

Testing Computer Software, Testing Computer Software, by C. Kaner, J. Falk, and H. Nguyenby C. Kaner, J. Falk, and H. Nguyen

IEEE Standard for Software Test Documentation, Std 829 - 1998IEEE Standard for Software Test Documentation, Std 829 - 1998

Black-Box Testing: Techniques for Functional Testing of Software and Systems, Black-Box Testing: Techniques for Functional Testing of Software and Systems, Boris Beizer, Wiley, 1995Boris Beizer, Wiley, 1995

Managing the Testing Process, Managing the Testing Process, Rex BlackRex Black

Classic Testing Mistakes, Classic Testing Mistakes, Brian MarickBrian Marickhttp://http://www.testing.com/writings/classic/mistakes.pdfwww.testing.com/writings/classic/mistakes.pdf

Software QA / Test Resource CenterSoftware QA / Test Resource Centerhttp://http://www.softwareqatest.com/index.htmlwww.softwareqatest.com/index.html

Class 2

Documents

Transcript of Class 2