GLAST Large Area Telescope: AntiCoincidence Detector (ACD) WBS 4.1.6
GLAST Large Area Telescope: Gamma-ray Large Cost/Schedule ... Reviews/8.5... · GLAST LAT Project...
Transcript of GLAST Large Area Telescope: Gamma-ray Large Cost/Schedule ... Reviews/8.5... · GLAST LAT Project...
GLAST LAT Project – Cost and Schedule Review August 5, 2004
AntiCoincidence Detector 1
GLAST Large Area Telescope:GLAST Large Area Telescope:Cost/Schedule ReviewAugust 5, 2004AntiCoincidence Detector (ACD) Subsystem WBS: 4.1.6
David J. ThompsonThomas E. JohnsonNASA Goddard Space Flight CenterSubsystem Manager/Instrument Manager
[email protected]@nasa.gov
GammaGamma--ray Large ray Large Area Space Area Space TelescopeTelescope
GLAST LAT Project – Cost and Schedule Review August 5, 2004
AntiCoincidence Detector 2
Significant AccomplishmentsSignificant Accomplishments
• Completed and delivered all HVBSs• Completed CPT on all flight Front End Electronics Boards.• Corrected mounting hole location error on 11 TDAs. New
holes were drilled and old holes were plugged. TDAs are now being wrapped.
• Completed the assembly of all flight Ribbon Detectors.• Completed the assembly of the clear fiber cables.• Working Photomultiplier Tube glass crack issue. • Received third partial G3/EGSE Test Stand.• Updating grass roots.• Completed assembly of the first Electronics Chassis
(Qual/Spare Unit)
GLAST LAT Project – Cost and Schedule Review August 5, 2004
AntiCoincidence Detector 3
Recent Accomplishments Recent Accomplishments -- Electronic Electronic Component StatusComponent Status
Ready for next assembly
Thermal Testing
Conformal Coating
Functional Testing
Assembly Complete
Quantity RequiredFlight (Spare)
ITEM
2045804580196(40)
Photomultiplier Tube Assembly
282830303024(6)
High Voltage Bias Supply
101114161612(4)
Front End Electronics Boards
GLAST LAT Project – Cost and Schedule Review August 5, 2004
AntiCoincidence Detector 4
Experienced new glass failures during partial life and screening thermal cycling of PMTs
PROBLEM / ISSUE IMPACT ACTIONDATE
ESTAB. COMPLETE
CURRENT STATUS – OPEN
~100 of 250 flight PMTs have been potted/mounted. Potting and PMT sub-assembly work has stopped pending new investigation. Final fight chassis I&T can not be completed without PMTs. This new design already passed previous qualification testing.
7/04 7 or 8/041. Investigate potential causes
including why new design passed initial qualification tests and failed latest tests
2. Determine various solution paths
3. Choose and implement best path
7/04
• Status on Action 1: Root cause has been determined. The mounting design is not robust enough to handle very large variations in material properties and processes. Specifically the Coefficient of Thermal Expansion (CTE) and Poisson’s Ratio of the RTV potting compound vary significantly from batch to batch and even with different cures of the same batch. This results in stresses that can vary by a factor of 25.
• Status on Action 2: There are two fundamental issues. Number one, what should be done with the 100 PMTs that have already been bonded into mechanical housings and secondly how should the remaining flight PMTs be mounted. Multiple solution paths have been developed for both of these issues.
• Status on Action 3: Multiple solutions paths are actively being pursued, others have been but on the “backburner”. We are planning on down selecting the potential solutions on August 17.
ELECTRONICS ELECTRONICS –– Photomultiplier TubesPhotomultiplier Tubes
8/04
7/04 8/15/04
R
GLAST LAT Project – Cost and Schedule Review August 5, 2004
AntiCoincidence Detector 5
Near Term MilestonesNear Term Milestones
Complete!8/3/04AugustComplete Fab of Clear Fiber
Cables
Was completed in July, but rework required on 11 (8 flight + 3 spare) TDAs. Repair complete and TDAsare being wrapped.
8/15/04NovemberComplete Fab of Flight TDAs
Two assembly drawings remain. Designers have been focused on revisions.
8/10/04SeptemberComplete Flight Mechanical
Drawings
Complete!6/1/045/17/04Mechanical Subsystem delivered
to I&T
Unit is assembled and being prepared for testing.8/6/045/20/04Complete Qualification Unit
Electronics Chassis Assembly
First group completed mid-October. See back up material for more information on PMT issue
11/05/04January, 2004Complete PMT Assembly
8/15/04
7/30/04
New Date
All 16 are assembled. 11 ready for assembly into flight electronics chassis.6/30/04Complete Flight FREE Boards
COMPLETE. All 30 are assembled, tested and ready for assembly into Electronics Chassis6/30/04Complete Flight HVBS
Status/NotesDateMilestone Description
GLAST LAT Project – Cost and Schedule Review August 5, 2004
AntiCoincidence Detector 6
ACD MILESTONES vs. TIME1
Variance Explanation: July Variance = -18 ( 14 – PMT Glass, signal wire, capacitor, and Hazardous Material shutdown, 1 – Detectors, 1 – Assembly Drawings, 1 – shippng container, 1 – Micrometeoroid Shield)
0
20
40
60
80
100
120
Nov-0
3Ja
n-04
Mar-04
May-04
Jul-0
4Sep
-04
Nov-0
4Ja
n-05
Mar-05
Cum
mul
ativ
e M
ilest
ones
PLANREPLAN 1/04RECOVERYACTUAL
Sep-03 Oct-03 Nov-03 Dec-03 Jan-04 Feb-04 Mar-04 Apr-04 May-04 Jun-04 Jul-04 Aug-04 Sep-04 Oct-04 Nov-04 Dec-04 Jan-05 Feb-05 Mar-05 Apr-05PLAN 12 21 31 45 58 73 77 85 86 90 92 94 96 99REPLAN 1/04 22 32 43 57 66 79 85 86 89 91 93 96 98 99RECOVERY 68 69 76 82 88 90 93 94 98 99ACTUAL 7 10 15 22 29 34 39 45 56 64 68
GLAST LAT Project – Cost and Schedule Review August 5, 2004
AntiCoincidence Detector 7
InterdependenciesInterdependencies
1. EGSE/G3 – Ongoing development with I&T and Electronics groups. – Last partial EGSE/G3 was received in July. Testing ongoing. – Software updates for software rate counters were received and they are
working. Developing the test scripts.– Full EGSE/G3 delivery planned for August.
2. Delivery of ACD Calibration Unit or subset to LAT I&T – The same electronics chassis being used for G3 commissioning will be used
for the calibration unit. Delivery of calibration unit will occur some time after this testing. Last partial EGSE/G3 was received in July. Testing ongoing.
– Working with LAT I&T to develop a work around that will allow an earlier delivery of a “mini-ACD”.
3. ICD, IDD, etc – Rich Bielawski is helping track a set of needed changes.– Mass CR needs to be released and approved. – Electrical Issues CR needs to be released and approved– IDD (LAT-DS-00309) update. Required for closeout of several open Problem
Reports.4. Funding.
– Recent technical issues are causing cost growth. The new known FY04 liens are provided on the next page.
GLAST LAT Project – Cost and Schedule Review August 5, 2004
AntiCoincidence Detector 8
NEW FY04 KNOWN LIENSNEW FY04 KNOWN LIENS
$32KTDA hole location repair
$38KPMT Purchase (20 PMTs)
$15KPRT contamination issues
$63KVoltage requirement changes (HVBS/GASU)
$37KPMT Issues (screw, wire, and capacitor fixes) plus adding thermal cycling of the PMTs
$507KTOTAL
$260KPMT Glass Crack Recovery
$10KAccelerometer purchase and calibration
$52KTSA Design/Analysis, including instrumentation changes required by LAT Instrumentation Plan
ESTIMATED COSTLIEN
GLAST LAT Project – Cost and Schedule Review August 5, 2004
AntiCoincidence Detector 9
Issues, Concerns, Documentation, & Qual ProgramIssues, Concerns, Documentation, & Qual Program
• Issues– PMTs– Funding
• Concerns– PMTs– Funding– Numerous “minor” technical issues
• Documentation– CM system works great– PR/PFR system working, but since it is a new system team
members are still learning how to use it efficently• Status of PR/PFRs
– 51 PR’s Closed– 21 PR’s Open– 4 PFR’s Open
• Qualification program fully developed
GLAST LAT Project – Cost and Schedule Review August 5, 2004
AntiCoincidence Detector 10
ACD Schedule VariancesACD Schedule Variances
• 4.1.6 – ACD Subsystem (-$425K cum, +$23 current)• 4.1.6.3 - TSA Schedule Variances (-$4K cum, +$5K current)
– Procedures and TDA Assembly and Test• 4.1.6.4 BEA Schedule Variances (-$291K cum, +$20K current)
– ($291K) – Nearly all variance is due to the PMTs.• 4.1.6.7 ACD I&T (-$101K cum, +$1K current)
– ($101K) – Getting started late, but we have begun! Many issues, mainly PMTs, PRTs, TDA’s, and connector polishing have impacted progress.
• 4.1.6.B GSE (-$23K cum, -$3K current)– ($23K) – Shipping container and MMS GSE work being
pushed out to reduce current manpower.
GLAST LAT Project – Cost and Schedule Review August 5, 2004
AntiCoincidence Detector 11
ACD Cost VariancesACD Cost Variances• 4.1.6 ACD (+$189K cum, +453K current period)• 4.1.6.1 ACD Project Management/Sys Eng/Science (+$196K cum,
-$182K current period)– PMCS adjusted for underrun this month– (+$112K) - Labor support lower than planned due to lower than
planned science simulations and test support ($157K), systems engineering being covered by GLAST Project ($48K), Science Support lag in accruals ($65K)
– (-$67K) – Materials. $50K charged to materials instead of lab tax and $17K for software maintenance.
– (+$88K) MPS lower than planned• 4.1.6.2 Safety and Mission Assurance (-$1K cum, -$51K current
period)– Staffing was not sufficient in this area. We have added a QA
person.• 4.1.6.3 Tile Shell Assembly (+$119K cum, +$454K current period)
– CR approved this period– Lag in Fermi invoicing/costing for the TDA’s. Will accrue $182K
in July.
GLAST LAT Project – Cost and Schedule Review August 5, 2004
AntiCoincidence Detector 12
ACD Cost VariancesACD Cost Variances
• 4.1.6.4 Base Electronics Assembly (-$336K cum, +$466K current month)– M&S – (-$17K) PMT issues, (-$63K) HVBS Voltage change, (-$246K) FREE,
HVBS, PMT, and Resistor Network work performed, but have not received credit for work done, (-$46K) SLAC ASIC charges.
• 4.1.6.5 MS/TB (+$23K cum, +$51K current month)– Lag in accruing costs for mechanical designer and MS/TB designer
• 4.1.6.6 ACD Mech Qual and Cal Unit (-$15K cum, -$167K current month)– Final billing for mechanical qualification testing received this month.
PMCS corrected for underrun in this area.• 4.1.6.7 I&T (+$10K cum, -$20K current month)
– (+$10K) - 50/50 earned value• 4.1.6.B Ground Support Equipment (+$146K cum, -$97K current month)
– (+$85K) – Labor. Costs lagging on mechanical design and EGSE support.– (+$59K) – Materials. Not billed for work completed
GLAST LAT ACD Project August 5, 2004
AntiCoincidence Detector 13
Summary of the ProblemFollowing a series of phototube failures (breakage at low temperature) in January, 2004, the mounting for the tubes was changed, reducing stress on the tube by nearly a factor of 10. The new mounting design was qualified with a set of 9 tubes down to temperatures of -40 C (-60 C for some) with no breakage. All tubes were visually inspected and graded. The new design was put into production. 100 of our 240 phototubes were potted into mechanical mounts (194 are needed for flight).
On July 6, 2004, we found out two out of five tubes subjected to a single thermal cycle down to -30 C broke. Both tubes were “fair” quality. A “good” tube and a “worst” tube in the same test survived.
Over that same weekend, a 12-cycle test (to -30 C) of 11 “worst” tubes had 2 break. After 30 cycles, no additional tubes had broken.
On July 9, a set of 25 “worst” tubes was carefully temperature cycled with checks at different temperatures for breakage. Four tubes broke at -15 C; one tube broke at -30 C.
ACD Phototube Breakage Issue
GLAST LAT Project – Cost and Schedule Review August 5, 2004
AntiCoincidence Detector 14
Find the Root CauseFind the Root Cause
Cause outline – how did we pass the original qualification of the new design but fail the follow-on tests? Summary of possibilities:
• A. The Stresses Increased ––RTV material properties changed
•Test new batches for CTE, modulus, Poisson’s ration•Develop new test to more accurately confirm early poisons ratio tests and data sources•Retest old batch if possible for CTE, Poisson’s ratio, modulus•Compare test results with old test results, data sources and assumptions•Use strain gauge tests to test for strain differences between old and new PMT batches and to correlate stress model
–PMTs that failed were unusual – diameter and glass thickness vary between tubes•Look at new centering step data for diameters
–Less likely causes we must still disprove – large bubbles in RTV causing stress concentrations, Thermal vac chambers in error and we actually went much colder, etc
• B. The photo-multiplier tubes became weaker over time ––Exposure to moisture over time has propagated glass flaws or made flaws much easier to propagate at lower stresses–Handling - very unlikely with new handling procedure–Is there any test of bare PMTs that could help see a weakening?
• C. The stress analysis and /or assumptions were flawed AND we were lucky on what were demanding qualification tests
–RTV properties and property tests were wrong to begin with – compare results of first bullet of A above–Hard to understand how 5 tubes could go to multiple cycles at -60C and 4 tubes could go to -40C without failure. Used as controls but none have failed yet. Retest with very low temperature test. Test 7 PMTs potted with old RTV batches but never cycled.–Review stress model again. Devise and use strain gauge tests mentioned in bullet 1 of A above to correlate model. Add non-linear ability to model.
GLAST LAT Project – Cost and Schedule Review August 5, 2004
AntiCoincidence Detector 15
X
X
X
X
XX
X
X
XXPMT Cracks
Glass Weakness
Excessive Stresses
Stress due to mountingStress due to test
Manufacturing cracks
Inside of tube
Outside of tube
Water
Handling
RTV stress
Curing shrinkage Mixing
Manufacturing
High temperatureLow temperature
Thermal rate of changeTest stand
Mu metalHousing
Model deficiencies
Bonding
Construction
Application
Inspection
Tilt Decentering
Curing
Mechanical properties (CTE, Poisson's ratio, Young's modulus)
Thanks to Art Whipple for starting this fishbone (Ishikawa) diagram.
Priming
Fishbone Diagram – Possible Sources of the Problem
X – source unlikely
Known contributors underlined.
GLAST LAT Project – Cost and Schedule Review August 5, 2004
AntiCoincidence Detector 16
Threats to Schedule and CostThreats to Schedule and Cost
1. PMT Assembly2. GASU/G3 EGSE3. Electronics assembly and test
GLAST LAT ACD Project August 5, 2004
AntiCoincidence Detector 17
So• Despite our qualification program, the method for potting ACD
phototubes into housings is still producing breakage of tubes atlow temperatures.
• The number of possible causes of this problem is limited, but noone possibility stands out as most likely. We are investigating all possibilities through a combination of analysis and test.
• Four candidate solution paths have been identified for improvingthe mounting scheme for new tubes and making the already-potted tubes useable for the ACD. Design studies are in progress, but we need more information about cause before selecting one ormore approaches for testing.
ACD Phototube Breakage Issue
GLAST LAT ACD Project August 5, 2004
AntiCoincidence Detector 18
PMT Stress as a function of PMT Stress as a function of CTE and PoissonCTE and Poisson’’s Ratios Ratio
• Raw NASTRAN analysis data– ∆T = -60°C (+20 to -40°C)– Exterior element 4959 beneath mounting flange– These values are based on a FEM with RTV modeled beyond
both edges of the glass tube
RTV-566 Poisson's Ratio CTE 233 ppm/deg C CTE 260 ppm/deg C CTE 290 ppm/deg C CTE 310 ppm/deg C CTE 330 ppm/deg C CTE 350 ppm/deg CTube Stress (psi) Tube Stress (psi) Tube Stress (psi) Tube Stress (psi) Tube Stress (psi) Tube Stress (psi)
0.4 161 191 225 248 270 2930.43 238 282 330 362 394 4260.46 427 550 642 703 764 8250.47 571 670 780 853 926 10000.48 847 993 1154 1262 1369 14770.49 1601 1873 2175 2377 2578 27800.499 8303 9699 11250 12284 13317 14351
RTV-566 Poisson's Ratio and CTE Sensitivity Study for PMT (Tube/RTV-566/Al Housing) Assembly
GLAST LAT Project – Cost and Schedule Review August 5, 2004
AntiCoincidence Detector 19
PMT Stress as a function of PMT Stress as a function of CTE and PoissonCTE and Poisson’’s Ratios Ratio
GLAST LAT Project – Cost and Schedule Review August 5, 2004
AntiCoincidence Detector 20
Stress resultsStress results
• Material properties are all over the place
• Improved Tests show changes between RTV batches and between different cures within same batch – in one case within batch differences (Poisson’s Ratio has biggest effect, we have developed a better test for that now) apparently cause more than a order of magnitude difference
• We identified this as a major risk during the peer review of the first solution. However we had many sources for our assumptions and cost and schedule pressures made many believe that the minimum 2 to 3weeks of additional delay and cost of creating and tweaking a new potting material Poisson’s Ratio test was turning leading us into a ‘technology development’ mode we had no time for.
• We have good confidence in the latest material property results (new poisons ratio test we choose not to pursue last time is working well and has been checked for set up and repeatability, etc)
GLAST LAT Project – Cost and Schedule Review August 5, 2004
AntiCoincidence Detector 21
ACD Phototube Breakage Problem – Root CausePhototube, CTE ~3 µm/mºC
RTV, CTE ~300 µm/mºC
Aluminum housing, CTE ~27 µm/mºC
Because the RTV is used to hold the position of the tube in both transverse and longitudinal directions, it must adhere to both the tube and the housing, i.e. be primed.
Because the Coefficient of Thermal Expansion (CTE) of the RTV is much larger than that of either the tube or the housing, at cold temperatures the shrinkage of the RTV pulls on both surfaces.
If the CTE of the RTV were closer to those of the tube or the housing, we would have no problem. BUT – the actual CTE of the RTV is LARGER than the specifications (and from a number of early sample tests) and variable from batch to batch, and, we are learning, from cure to cure within batch.
If the RTV were compressible enough, it would stretch instead of transmitting the stress. BUT – the actual RTV is almost perfectly incompressible, significantly more so than the specifications. We are learning this also appears to vary from batch to batch and between cures within the same batch. Poisson’s ratio is 0.49 in some samples compared to specifications of no more than 0.46 (risk presented in peer review).
RESULT – Combine the above facts with the fact that glass flaws in our particular PMTs make them 50 to 70% weaker than one might expect, at low temperatures the tube is pulled apart by the shrinking RTV, with cracks starting at these flaws. We believe the qual units did not fail because the earlier batches of RTV tended to have much better material properties and there may have been some de-bonding in the early batches.
GLAST LAT Project – Cost and Schedule Review August 5, 2004
AntiCoincidence Detector 22
ACD Phototube Breakage – Choosing a solution
Approach:
• Basic listing of solution options done. Now lay out the options in sufficient detail to understand the basic issues.
• Lay out basic solution decision criteria.
• Distinguish two cases and the possibility of combine solution
• What we do with the 100 tubes already potted
• What we do with the 140 bare tubes
• Establish top level flow and individual flows for each path with dates
• Pursuit of options with most critical criteria related activates given priority. During pursuit of solutions look for critical missing information or “show stoppers”
• Options that begin to look very hard early on could be put on back burner. This allows eliminate options as quickly as possible as initial prime candidates at least by completing the most critical actions.
GLAST LAT Project – Cost and Schedule Review August 5, 2004
AntiCoincidence Detector 23
PMT Top Level Solution Path flowPMT Top Level Solution Path flow
Interpretation of latest data from failures and materials testing
– preliminary conclusion July 30
Develop general set of solution paths from general
set of failure causesJuly 20
Investigate failure July 6-Aug2
Develop general set of failure causes
July 19
Begin pre-mature solution analysis and feasibility tests of
a few promising concepts, (see separate solution flows)
July 22
Lay out more detailed set of solution options, feasibility results and
selection criteriaAug 5
Complete analysis and preliminary testing
Aug 10
PMT selection solution decisionGoal date -Aug 13
See detailed flow for chosen solution
and actions list
Re-asses solutions based on early
feasibility results and failure
cause analysis
GLAST LAT Project – Cost and Schedule Review August 5, 2004
AntiCoincidence Detector 24
PMT Solution Paths PMT Solution Paths
Possible top level Solution Paths• Modified Potting Solution – understand the new variables and stresses in the potting materials
–RTV or more likely an alternate–Must understand exactly how the latest PMTs failed–Slit-potted design. May even be able to do this to already potted PMTs–Others also require removal methods for already potted PMTs
• Thermal control solution – don’t let the PMTs see the stress of lower temperatures. –Heaters to -5 or 0C, Must determine via test what is warm enough–Allows us to fly PMTs that have already been potted
• Mechanical solution – get out of the potting business since it looks like the material properties vary too much for these flawed tubes.
–Partial CTE compensation design. Uses modified existing housings with inserts, does still add some stress but mostly compressive–Quasi kinematic mounts. Various forms. Simple versions may be able to use existing housings with inserts–Bonding release designs. Releases bonding on one side, grooved housings keep PMT from slipping
• Determining Yield in current design – see if there is a screening test that stresses the PMTs in a way that identifies almost all the PMTs that will fail without consuming lifetime of the PMTs that pass or making them more likely to fail.
–Probably have to screen and partial life test large number of PMTs ( which may be tough to then use as flight) to show it could work.
•Any combination of above with combined with some or all new PMTs without so many glass flaws –First 6 units with modified Hamamatsu process are not flawless but are all dramatically better than any of the original tubes we haveNote : Some solutions also require removal methods for already potted PMTs
GLAST LAT Project – Cost and Schedule Review August 5, 2004
AntiCoincidence Detector 25
Salvage potted PMTs
Screen PMT assemblies
New PMT housing design
Heat PMT assemblies
Fix baseline design
Resume flight PMT assembly
PMTs crackduring thermal
testing
Determine root cause of
failures
Correct problemCorrectable?
Yes
No
Requalifydesign
Develop PMT assembly screening process
Screenable? Failure rateacceptable?
YesModify baseline design
Yes
No No
DevelopLow-stress PMT
mounting
Change baseline design
Develop process to
safely remove potted PMTs
Successful?
Remove potted PMTs
Procure more PMTs
Yes
No
Redefine PMT thermal
requirements
Develop thermal control system & assess impacts
Impactsacceptable?
Modify baseline design
Yes
No
Use current flight PMT assm
Use newflight PMT assm
GLAST LAT Project – Cost and Schedule Review August 5, 2004
AntiCoincidence Detector 26
Selection CriteriaSelection Criteria
• STRESS AND STRAIN - Reliably lowers stress to under 1000 psi for tensile stresses with a goal of < 500 psi. Keeps compressive stresses to TBD (< 2500-5000 psi) compressive stresses. Show by analysis and strain test if possible. Higher compressive stresses must be shown to truly be compressive with no variables that could cause associated tensile stresses
• FEASIBILITY - Passes prototype feasibility tests (assemble-able, controllable, seems repeatable)
• MINIMUM PROTOTYPE TESTS - Passes prototype thermal test, light tight and vibration test.
• LOW VARIABLY RISK – low number of variables or that could possibly effect stresses or ACD performance. Low effect on stress on variables we cant control.
• SCHEDULE (& cost). Could be used as tie breaker for solutions that pass above tests would be cost and schedule, however some solutions will be eliminated sooner solely on that basis.
GLAST LAT Project – Cost and Schedule Review August 5, 2004
AntiCoincidence Detector 27
Alternate Design 1Alternate Design 1–– CTE Comp Mechanical MountCTE Comp Mechanical Mount
Design Prototype,Preliminary
Stress AnalysisJuly 9
FabricateParts
July 17 MechanicalFit CheckJuly 21
Strain GagePMT
July 22
1st PrototypeAssembly
Aug 2
AssemblyProcedure
Aug 2
MarginsOK
Detailed Stress
AnalysisAug 9
ModifyDesign
NO
YESPMT solution
DecisionAug 13
1st ThermalAnd
strain Test – Pass?
August 6
Vibrationtest
StartBuild 20
Qual UnitsAugust 31
Finalize Flight Design
August 26
FabricateParts
August 26
QualTesting
September 12
PMT Production(40/week)
September 20
Thermal ScreenSeptember 27
First Batch Ready For Electronic
Chassis Integration Oct 1