Post on 12-Jan-2016
description
1JMH Associates © 1998, All rights reserved
A Computer Scientist Looks at the Year 2000 Problem
The Keys to High-Quality Year 2000 System Renovation
Johnson (John) M. Hartjmhart@world.std.com
JMH Associates
“Give us the tools, and we will finish the job.”Winston Churchill to President Roosevelt, 1941
2JMH Associates © 1998, All rights reserved
PretendPretendPretendPretend You are the CIO at a large corporation You are responsible for the Year 2000
readiness of 100s of millions of lines of COBOL code Terabytes of data
The future of the company is in your hands! And - Pretend it’s 1997 or earlier Can computer science and software
engineering help you?
3JMH Associates © 1998, All rights reserved
Y2K Renovation – Getting StartedY2K Renovation – Getting StartedY2K Renovation – Getting StartedY2K Renovation – Getting Started
You are Y2K aware Have a budget Performed inventory and impact analysis The project team is organized You’ve decided on windowing or expansion
Of course you’ll test the renovated systems Now what?
How will you renovate your code and data?
4JMH Associates © 1998, All rights reserved
Renovation: The Neglected StepRenovation: The Neglected StepRenovation: The Neglected StepRenovation: The Neglected Step
How will you renovate your systems? Better renovation: fewer errors to find in test Shift resources from testing to renovation Large systems require automation benefits
Two key steps Date-Sensitive Item Identification System Correction
Caveat: Not entirely automated Some manual intervention
5JMH Associates © 1998, All rights reserved
Common Statements Demonstrate Common Statements Demonstrate Renovation NeglectRenovation Neglect
Common Statements Demonstrate Common Statements Demonstrate Renovation NeglectRenovation Neglect
“It will be good enough” “There’s no ROI in better renovation” “We’ll fix the mistakes in testing” “Not enough testing?
Decrease the renovation budget!” “We’ll test all the paths” “We only need to test what has changed”
6JMH Associates © 1998, All rights reserved
Caution!Caution!Caution!Caution!
Testing misses serious & common errors “Change-based testing” doesn’t test what
should have been changed Good ID & correction: fewer errors to detect You can’t test quality in Do it right the first time
You’ll save time and money! More on this later
7JMH Associates © 1998, All rights reserved
Date-Sensitive Item IdentificationDate-Sensitive Item IdentificationDate-Sensitive Item IdentificationDate-Sensitive Item Identification
Key requirement for quality Y2K renovation What are Date-Sensitive Items?
Variables, constants File & screen fields Anything containing year/century data
Required for ALL renovation strategies You can’t fix what you can’t find Faulty ID leads to defective systems
8JMH Associates © 1998, All rights reserved
DSI ID Investment Pay-OffsDSI ID Investment Pay-OffsDSI ID Investment Pay-OffsDSI ID Investment Pay-Offs
Decreased testing time and expense Decreased risk to operation Decreased system repair time System documentation with long-term value
9JMH Associates © 1998, All rights reserved
…
Faulty DSI ID Example: Original CodeFaulty DSI ID Example: Original CodeFaulty DSI ID Example: Original CodeFaulty DSI ID Example: Original Code
DATA DIVISION.WORKING-STORAGE-SECTION01 TODAYS-DATE PIC 9(6). Correctly identified - yymmdd01 PAYMENT-YEAR-TO-DATE PIC 9(6). False positive -
yymmdd01 PAYMENT-DUE PIC 9(6). Correctly identified - yymmdd01 NEW-PAYMENT PIC 9(6). False negative - is yymmdd01 MAX-YEAR-TO-DATE PIC 9(6) VALUE 100000.
False positive - yymmdd. . .
PROCEDURE DIVISION.IF TODAYS-DATE > PAYMENT-DUE
AND PAYMENT-YEAR-TO-DATE > MAX-YEAR-TO-DATETHEN
MOVE NEW-PAYMENT TO PAYMENT-DUE.
10JMH Associates © 1998, All rights reserved
DATA DIVISION.WORKING-STORAGE SECTION01 TODAYS-DATE PIC 9(8). Correctly identified - yymmdd01 PAYMENT-YEAR-TO-DATE PIC 9(8). False positive - yymmdd01 PAYMENT-DUE PIC 9(8). Correctly identified - yymmdd01 NEW-PAYMENT PIC 9(6). False negative - is yymmdd01 MAX-YEAR-TO-DATE PIC 9(8) VALUE 19100000.
False positive - yymmdd. . .
PROCEDURE DIVISION.IF TODAYS-DATE > PAYMENT-DUE
AND PAYMENT-YEAR-TO-DATE > MAX-YEAR-TO-DATETHEN
MOVE NEW-PAYMENT TO PAYMENT-DUE.
Renovated (Expanded) CodeRenovated (Expanded) CodeRenovated (Expanded) CodeRenovated (Expanded) Code
11JMH Associates © 1998, All rights reserved
Defective DSI ID Example: ResultsDefective DSI ID Example: ResultsDefective DSI ID Example: ResultsDefective DSI ID Example: Results
Defective code and corrupted data PAYMENT-YEAR-TO-DATE must exceed
$19M rather than just $100K PAYMENT-DUE appears to be a first
century date PAYMENT-YEAR-TO-DATE a file field?
Corrupted data! $500,000 becomes $19,500,000 or $20,500,000
Windowing logic correction is just as defective
12JMH Associates © 1998, All rights reserved
Identification: MethodIdentification: MethodIdentification: MethodIdentification: Method
“Knowledge” propagation Start with some “truth” about the system
Initial knowledge seeds the propagation Propagate this knowledge through the code Get seed information from easy places
Files, reports, screens, pattern matching suggestions
Provide attribute information Format (YY, YYMMDD, YYDDD, etc.)
There are some hard cases
13JMH Associates © 1998, All rights reserved
IF TODAYS-DATE > PAYMENT-DUE AND PAYMENT-YEAR-TO-DATE > MAX-YEAR-TO-DATE
THEN MOVE NEW-PAYMENT TO PAYMENT-DUE.
Knowledge PropagationKnowledge PropagationKnowledge PropagationKnowledge Propagation
SEED
Propagate knowledge by applying identification rules at each program statement
14JMH Associates © 1998, All rights reserved
Propagating KnowledgePropagating KnowledgePropagating KnowledgePropagating Knowledge
Year 2000-sensitive I/O fields
Year 2000-sensitive variables and constants
Variables that are not Year 2000-sensitive
Linkage
Module C
Module B
File BFile A
Module A
15JMH Associates © 1998, All rights reserved
DSI ID – Key PointsDSI ID – Key PointsDSI ID – Key PointsDSI ID – Key Points
DSI ID must be automated in large systems Pattern matching: inadequate
Can suggest seeds Cannot make decisions
The complete system must be examined DSI ID must also determine format
(YYMMDD, DDMMYY, YYDDD, ...)
16JMH Associates © 1998, All rights reserved
Total Automation Not Possible Total Automation Not Possible Total Automation Not Possible Total Automation Not Possible
There are always some human decisions “No silver bullet”
Pattern matching has some value Can provide seeding & consistency checks Assessment Inventory Other preparatory phases But, no matter how refined, pattern (and
usage) matching can not decide the dates
17JMH Associates © 1998, All rights reserved
These Techniques Work!These Techniques Work!These Techniques Work!These Techniques Work!
Tools from several vendors in use today Tools can be fast & are getting faster
Best techniques: based on compiler theory Hard work turns the theory into practice
Cost-effective, especially in large systems Why do anything else?
Human knowledge & decisions still necessary Some situations are inherently ambiguous
18JMH Associates © 1998, All rights reserved
Quality DSI ID Can Prevent DefectsQuality DSI ID Can Prevent DefectsQuality DSI ID Can Prevent DefectsQuality DSI ID Can Prevent Defects
Before they get into the system! Faulty DSI ID causes costly defects
Corrupted data Incorrect business decisions
Defect detection through testing is expensive, risky, and will miss many defects We’ll see why later
Identification drives code and data correction
19JMH Associates © 1998, All rights reserved
Automated Year/2000Automated Year/2000Correction is NecessaryCorrection is Necessary
Automated Year/2000Automated Year/2000Correction is NecessaryCorrection is Necessary
Correction must be accurate, repeatable Systems are large and complex Simplified testing and verification Intricate changes must be consistent, reliable Date identification drives correction
20JMH Associates © 1998, All rights reserved
Example: Code Renovation Example: Code Renovation Windowing Logic CorrectionWindowing Logic CorrectionExample: Code Renovation Example: Code Renovation Windowing Logic CorrectionWindowing Logic Correction
(a/k/a Procedural Workaround) Procedure division logic changed
Interpret 2-digit dates in a century window Window range [ 1900+b and 2000+b–1 ] Example: [1950, 2049 ]
Existing files retain 2-digit year fields No changes to existing data files
Variations: Date Expansion, in-line, custom
21JMH Associates © 1998, All rights reserved
Code fragment before correction
Logic Correction Example –Logic Correction Example –Windowing Logic CorrectionWindowing Logic CorrectionLogic Correction Example –Logic Correction Example –Windowing Logic CorrectionWindowing Logic Correction
IF DUE-DATE GREATER THAN START-DATE ANDDUE-DATE LESS THAN END-DATE ORSTART-DATE EQUAL FIRST-DATE
PERFORM SOMETHING.
22JMH Associates © 1998, All rights reserved
Logic Correction Example –Logic Correction Example –Corrected CodeCorrected Code
Logic Correction Example –Logic Correction Example –Corrected CodeCorrected Code
CALL 'CVRT2TO4' USING DUE-DATE YYYYMMDD-1CALL 'CVRT2TO4' USING START-DATE YYYYMMDD-2CALL 'CVRT2TO4' USING DUE-DATE YYYYMMDD-3CALL 'CVRT2TO4' USING END-DATE YYYYMMDD-4IF YYYYMMDD-1 GREATER THAN YYYYMMDD-2 AND
YYYYMMDD-3 LESS THAN YYYYMMDD-4 ORSTART-DATE EQUAL FIRST-DATE
PERFORM SOMETHING.
23JMH Associates © 1998, All rights reserved
ID & Correction: The Keys to QualityID & Correction: The Keys to QualityID & Correction: The Keys to QualityID & Correction: The Keys to Quality
Don’t rely on testing, no matter how thorough It’s expensive It will miss many defects It finds symptoms, not causes
Do it right the first time Sound software engineering Essential for “due diligence”
24JMH Associates © 1998, All rights reserved
Testing Will MissTesting Will MissMost False NegativesMost False Negatives
Testing Will MissTesting Will MissMost False NegativesMost False Negatives
Despite extensive, full-coverage test cases Unknown dates will not be aged “Test what has changed”
Totally ignores false negatives The original code would pass the test!
Also misses many false positives, bad logic Testing finds symptoms, not root causes
25JMH Associates © 1998, All rights reserved
The RealityThe RealityThe RealityThe Reality
Many renovations are poor quality Recent competitive comparison
6 vendors, 100 KLOC, 159 required changes Accuracy ratings:
89%, 75%, 75%, 70%, 60%, 56% 90% means 16 errors on 100 KLOC
1.6 per 10 KLOC
26JMH Associates © 1998, All rights reserved
Answer: Independent Pre-Test AuditAnswer: Independent Pre-Test AuditAnswer: Independent Pre-Test AuditAnswer: Independent Pre-Test Audit
A high-quality tool reports: All detected false positives and negatives All detected bad fix insertions All missing fixes, faulty logic, mixes bases
Fix it now! Before test! You are fixing the root cause Testing only finds a symptom
Due diligence, IV&V
27JMH Associates © 1998, All rights reserved
It Works! It’s Cheap!It Works! It’s Cheap!It Works! It’s Cheap!It Works! It’s Cheap!
Examples: 110 KLOC, tested. Found 33 errors 350 KLOC, tested. Found 200 errors 400 KLOC, tested. Found 70 errors 250 KLOC, carefully renovated.
Found one error High-quality renovation: high-quality result
28JMH Associates © 1998, All rights reserved
A More Dramatic Case StudyA More Dramatic Case StudyA More Dramatic Case StudyA More Dramatic Case Study
Testing found 31 faulty DSI IDs 55 staff-days; 1.77 SD per defect found
Independent audit found 22 more faulty IDs 4 staff-days to find. 0.075 SD per defect
Nearly 25 : 1 cost ratio per defect detected Plus, additional defects were detected Detected root cause, not symptoms
Cost-to-fix ratio is even better No collateral damage from residual defects
29JMH Associates © 1998, All rights reserved
BUDGET ASSUMPTIONS:
1,000,000 LINES OF CODE
$1,000,000 DOLLAR BUDGET
$1.00/LOC
40% PLANNING & ANALYSIS
50% TESTING & FIX
10% RENOVATION
Savings from Independent Savings from Independent Renovation InspectionRenovation Inspection
Savings from Independent Savings from Independent Renovation InspectionRenovation Inspection
Example: Original Budget
30JMH Associates © 1998, All rights reserved
Reduced Budget with IRIReduced Budget with IRIReduced Budget with IRIReduced Budget with IRI
40% PLANNING & ANALYSIS
25% TESTING & FIX
10% RENOVATION
5% IRI
20% PROJECT SAVINGS
ASSUME: IRI FOR 1/2 THE COST OF RENOVATION
AN AVERAGE OF 1 ERROR PER 10,000 LINES OF CODE
ASSUME TESTING UNCOVERS ANOTHER 100 ERRORS
31JMH Associates © 1998, All rights reserved
SummarySummarySummarySummary
Renovation is an exacting, repetitive task Accurate Date ID is essential Changes procedure, data divisions and files
Automated tools are fast and accurate Testing is necessary but not sufficient Independent audits are cost-effective Experience to date is excellent
32JMH Associates © 1998, All rights reserved
More InformationMore InformationMore InformationMore Information
Date Sensitive Item IdentificationYear/2000 Journal, March/April, 1998http://www.y2kjournal.com (see “past issues”)
Automation Assisted Correction Year/2000 Journal, May/June, 1998
Testing Limitations and Auditinghttp://www.itpolicy.gsa.gov/mks/yr2000/y2kconf/paper59fp.htm
http://www.itpolicy.gsa.gov/mks/yr2000/y2kconf/paper60fp.htm
Year/2000 Journal, Sept/Oct, 1998