The Practical Aspects of Computational Science and Engineering · The Practical Aspects of...

DISTRIBUTION STATEMENT A. Approved for Public Release. Distribution is unlimited. ITL-14-15, ITL-14-14, ITL-15-62, ITL-16-14

The Practical Aspects of Computational Science and Engineering Frontiers in Computing and Data Science Michigan State University, 2-4 Oct 2016

Dr. Douglass Post, Associate Director for CREATE

The Practical Aspects of Computational Science and Engineering

Page-2 Distribution A: Approved for Public release; distribution is unlimited

HPCMP Ecosystem

DoD Supercomputing Resource Centers (DSRCs) Networking and Security

Acquisition Engineering

Test and Evaluation

A technology-led, innovation-focused program committed to extending HPC to address the DoD’s most significant challenges

U.S. Air Force Research Laboratory DSRC

U.S. Army Research Laboratory DSRC

U.S. Army Engineer Research and Development Center DSRC

Maui High Performance

Computing Center DSRC

U.S. Navy DSRC

Defense Research & Engineering Network (DREN)

Computer Network Defense, Security R&D, and Security Integration

Core Software

Computational Environments

Education and Training

HPC User Support

Results

Software Applications

Acquisition EngineeringScience and Technology

Decision SupportTest and Evaluation

Decision SupportAcquisition Engineering

Acquisition EngineeringDecision SupportDoD Supercomputing Resource Centers (DSRCs)

Acquisition EngineeringDecision SupportNetworking and Security Acquisition EngineeringDecision SupportSoftware Applications



BLUF-Bottom Line Up Front We live at the dawn of a New Era

Today, for first time in history, we can accurately predict

the future – The performance of major physical systems using

physics-based HPC computational tools

Challenge is to develop and deploy the software

Based on an High Performance Computing Ecosystem – With virtual instead of physical Research and Engineering Facilities

Paradigm shift with many sociological as well as technical challenges



HPC Ecosystem Becoming Ubiquitous Tool for Research, Prediction, Testing and Design

Computational modeling is part of experimental and theoretical work in almost all the hard sciences and engineering, and social sciences as well

Hurricane Sandy

Predicted Path

Observed Path

Calculated Tokamak 3-D Turbulent Flow

Calculated Tire Performance Hydroplane test rig

Large Hadron Collider Higgs Boson Tracks

Gene Sequencing in Paleontology

Political Science Agent-based

simulation graph



HPC is the Enabling Technology

1015-18 increase in computer power since 1945 is enabling us to: – Utilize accurate solution methods – Include all the effects we know to be

important multi-physics – Model a complete system – Complete parameter surveys and

analyze data from experiments with usable turn-around times

In ~ 10 years, workstations will be as powerful as today’s high performance computers

Software applications capable of exploiting this computer power are the missing link!

Moore’s “Law”

10-6

0.0001

0.01

1

100

104

106

108

1940 1950 1960 1970 1980 1990 2000 2010 2020

Computing Power For The World's Fastest Computer

Floating-Point Operations/sec

Per

form

ance

(GFL

OP

s/se

c)

Year

Cor

es

High Performance Computers

Workstation Performance



NASA Wind-Tunnel Simulator

CERN, Large Hadron Collider

Wright Brothers Wind-Tunnel Simulator

60 inch cyclotron at UC Berkeley

Small Large

Machines

Teams

Cost

Lifetime

Data

Experimental Facilities Have Grown from 1900 to Now

Complexity



Computational and Experimental Facilities

Douglas XSB2D-1 airplane in NASA 40- by 80-foot tunnel

Kestrel F-18 Simulation

Today Two Types of Facilities – 1. Experimental test and research facilities – 2. Virtual ecosystems for research and testing

Virtual facilities include codes, computers, networks, etc.

Like experimental facilities, virtual facilities require sustainment and modernization support

Supernova Simulation-FLASH

Very Large Array Radio Telescopes

Research Facilities

Test Facilities



Application Paradigm is Changing

Developers Users Developers Users

1960s today 1990s today

In the past, code developers were often the major users – SMEs had detailed understanding of the code’s strengths and weaknesses – Codes not designed for general use (focused V&V, hard to use, minimal

documentation, …)

Now, many SMEs use codes developed and supported by others – Examples: Chemistry and Materials, Fluid Mechanics, Structural

Mechanics, Climate and Weather, … e.g., GAMESS, NWCHEM, OVERFLOW, NASTRAN, WRF, FLASH, ISV codes,…

– SMEs less aware of code strengths and weaknesses



Science-based Code Productivity History Codes developed & used by researchers are generally successful

– Criterion for success is published research results, metrics: published papers

Development of science-based codes for use by others takes a long time (≥ ~10 years) and a large multi-disciplinary team (~5 to 20), longer and larger than pure research codes – E.g., NWCHEM, GAMESS, NNSA Design Codes, ISVs; Case studies (DARPA HPCS Phase II)

Codes developed for use by others have more stringent requirements for success – Useful for others for successful research or product design, measures: level of adoption, impact,… – Need adoption into customer community work flow, measures: level of adoption, impact,… – Higher level of software quality, user support, documentation, …. Very important for success

Evidence is that successful development rate for such codes is much lower – Relatively few research codes graduate to “community” codes, measures: survival rate, # of users – Many contractor-built large-scale complex software projects failed (FBI, NSA, FAA, …) at the $100M to

$2B level – Many ISV products are not profitable



HPCMP Ecosystem: “It Takes a Village”

DoD Supercomputing Resource Centers (DSRCs) Networking and Security

Acquisition Engineering

Test and Evaluation

A technology-led, innovation-focused program committed to extending HPC to address the DoD’s most significant challenges

U.S. Air Force Research Laboratory DSRC

U.S. Army Research Laboratory DSRC

U.S. Army Engineer Research and Development Center DSRC

Maui High Performance

Computing Center DSRC

U.S. Navy DSRC

Defense Research & Engineering Network (DREN)

Computer Network Defense, Security R&D, and Security Integration

Core Software

Computational Environments

Education and Training

HPC User Support

Results

Software Applications

Acquisition EngineeringScience and Technology

Decision SupportTest and Evaluation

Decision SupportAcquisition Engineering

Acquisition EngineeringDecision SupportDoD Supercomputing Resource Centers (DSRCs)

Acquisition EngineeringDecision SupportNetworking and Security Acquisition EngineeringDecision SupportSoftware Applications



Development and Sustainment of “Virtual Research and Test Facilities” Involve Immense Challenges* Scientific and Engineering:

– Trade-off many different strongly interacting effects across many orders-of- magnitude of multiple time- and distance-scales

– Verify and Validate highly-complex applications Wrong results lead to bad science and defective designs

– Develop problem generation methods required for larger, more complex problems – Analyze and visualize larger, more complex datasets - era of “Big Data”

Project: – Evolve from small teams to large, multi-disciplinary and multi-institutional code

development teams – Achieve long-term, stable support for teams and provide customer support

Programming: – Changing computer architectures every 3-4 years – Develop and run-codes for computers that will be 102 to 104 faster with 102 to 103

times more processors and greater memory architecture complexity than today – Achieve efficient performance for next-generation computers – Develop codes and run massively-parallel applications with relatively immature

tools Present research communities have limited experience developing

engineering applications additional challenges

*c.f. The Opportunities, Challenges and Risks of High Performance Computing in Computational Science and Engineering, D.E. Post, R.P. Kendall and R.F. Lucas, Advances in Computers, Quality Software Development, 66, ( 2006), M. Zelkowitz, Ed., Academic Press pp. 239-301.



Examples of Success –V&V on SE—CREATE, GAMESS, FLASH,… DoD Computational Research and Engineering Acquisition

Tools and Environments (CREATE) – 12-year, $300M program to develop design tools for DoD Air Vehicle, Ship and

RF Antenna Major Weapon Systems, initiated in 2005, funding started in FY2008 – 11 separate distributed code teams – Multi-physics, multi-scale, HPC software – After 8 years of development, software being used by ~ 130 DoD Acquisition

Engineering Organizations (government and defense industry)

GAMESS—Community Chemistry Code led by Iowa State (Mark Gordon) – Iowa State (Mark Gordon), ~ 10 staff on site, 150 contributors, ~6000 research

papers written based on GAMESS use, distributed funding

FLASH—Astrophysics (Super-Nova) Code (U. of Chi.) – Funded by DOE NNSA ASCI Alliance and NSF – Community code widely used by astrophysical and HEDP community

Metrics: # of users, impact, publications, longevity, financial support, …



CREATE: Suite of Physics-based HPC Tools for the design and analysis of DoD platforms:

Air Vehicles (AV) - Air Force, Army & Navy – Concept design, High-Fidelity Fixed-Wing and Rotary-Wing

Ships - Navy – Concept design, Shock and Live-Fire Vulnerability, Hydrodynamics

Radio Frequency (RF) Antennas - Air Force, Army & Navy – RF Antenna electromagnetics & integration with platforms

Ground Vehicles (GV) - Army, Marine Corps – Design and evaluation of tactical ground vehicles

Mesh and Geometry (MG) Generation – Rapid generation of geometry representations and meshes

F-35

CREATE tools support all stages of acquisition from rapid early- stage design to full life-cycle sustainment and modifications

CH-47



Design Production Scale-Model Tests Full-Scale Tests Requirements

Analyze & Test Computational

Prototype

Build & Test Physical Product Market

Design iterations

Design & Mesh Computational

Prototype

Flight Radial

Existing DoD Paradigm (design, build, test, fix…)

Goodyear “Innovation Engine” (design, virtual test, fix, build, deploy)

Loren Miller, Simulation-Based Engineering for Industrial Competitive Advantage, computing in Science and Engineering

(2010), May/June, pp. 14-21.

Competitive Advantage

• Reduced product development time from 3+ years to ~9 months or less

• Cut prototype build & test costs by 62%

• “Innovation engine” new products from 10/year to over 60/year



CREATE 6 Projects: 11 Multi-Physics Software Tools Ships - CREATE-Ships

– Rapid Ship Design Environment (RSDE) - Rapid Design and Synthesis Capability – Navy Enhanced Sierra Mechanics (NESM) - Ship Shock and Shock Damage Assessment – NAVYFOAM - Ship Hydrodynamics - Predicts hydrodynamic performance – Integrated Hydro Design Environment (IHDE) - Facilitates access to naval design tools

Air Vehicles - CREATE-AV – DaVinci - Rapid conceptual design – Kestrel - High-fidelity, full-vehicle, multi-physics analysis tool for fixed-wing aircraft – Helios - High-fidelity, full-vehicle, multi-physics analysis tool for rotary-wing aircraft

RF Antenna - CREATE-RF – SENTRi - Electromagnetics antenna design integrated with platforms

Ground Vehicles - CREATE-GV – Mercury - High-fidelity, multi-physics simulation tool for vehicle systems and components – Mobility Analysis Tool (MAT) - Analysis tool to evaluate ground vehicle performance metrics

Meshing and Geometry - CREATE-MG – Capstone - Components for generating geometries and meshes needed for analysis

HPC Portal - Secure access to computers through a browser



HPCMP CREATETM – AV Kestrel v5.x Computational Tool for Full Aircraft Design Analysis and Testing via high-fidelity, multi-disciplinary, physics-based simulation

Key Disciplines (available now in Kestrel v5) − Aerodynamics (Navier-Stokes solvers and full suite of BC’s & turbulence models) − Structural Dynamics (Modal models or FEA for aero-structure interaction) − Flight Control Systems (Control surface movement – deforming geometry or overset) − Propulsion (Engine “cycle-decks” for propulsion effects, or direct engine simulation including inlet

and rotating machinery, nozzle, and moving walls)

Use-Cases Enabled − Materially contribute to the design of next-generation aeronautical weapon systems. − Verify design prior to key decision points (and prior to fabrication of test articles or full-scale prototypes)

− Plan/rehearse wind-tunnel and full-scale flight tests (more bang per test dollar)

− Evaluate planned (or potential) operational use scenarios − Perform flight certifications (e.g., airworthiness, flight envelope expansion, mishap investigation, etc.)

− Generate response surfaces usable in DaVinci, flight-simulators, and other environments that require real-time access to performance data

Kestrel: A CREATE Example



Context The A-10 System Program Office (SPO) is exploring enhancements to the aircraft’s current inboard leading-edge slat system. The A-10 mission requires operation at high angles-of-attack and high sideslip, which increases the likelihood of engine inlet flow distortion.

Performed by AF AEDC using HPCMP CREATETM

A-10 Engine Inlet Flow Distortion

AEDC provided the A-10 SPO with engine inlet distortion data associated with various wing leading-edge designs. Analysis of the simulation results have identified the contributing sources of engine inlet distortion that could not be determined from wind-tunnel data alone. The A-10 enhancement program is still in progress. AEDC engineers have demonstrated that HPCMP CREATETM-AV Kestrel is a valuable tool in design validation testing.

Objective Apply HPCMP CREATE-AVTM Kestrel to simulate the A-10 with baseline and alternative wing leading-edge configurations to assess potential for improvements realizable from the planned enhancement. Explore wind-tunnel scale and full-scale conditions with both flow-through nacelles and the integrated TF34-GE-100 0-D engine model. Jason Klepper

(AF/AEDC)



CREATE Organization—Embedded with the Services

Ground Vehicles Project

Project Manager ERDC

Integrated Hydro Design Environment (IHDE) NSWC, Carderock

Ships Project Project Manager HPCMP, Lorton

Rapid Ship Design Environment (RSDE)

NSWC, Carderock

NavyFOAM NSWC, Carderock

Navy Enhanced Sierra Mechanics

(NESM) NSWC, Carderock

HPCMP Director Dr. David Horner

CREATE Program Assoc. Dir. Dr. Douglass Post

Kestrel 46th Test Wing, Eglin

AFB Quality Assurance NAVAIR, Patuxent

River Helios

Army ADD, Ames

DaVinci HPCMP, WPAFB

Official HPCMP Advisory Panel

Air Vehicles Project Project Manager HPCMP, Lorton

RF Antennas Project

Project Manager (SENTRI)

Sensors Directorate, AFRL, WPAFB

Mesh & Geometry

Project Project Manager

Navy NRL (Capstone)

Distributed, Multi-Organizational,

Multi-Institutional Program

• Embedding enables direct Service input and collaboration

• Aids adoption of tools

Mobility Analysis (MAT) ERDC

Mercury TARDEC

Also senior oversight by Board of Directors for each project (AV, Ships, RF, MG, GV) composed of senior Service acquisition technical staff (SES, Flag Level, O-6)

Aligns customer and developer views and communications



Distributed Teams Can Work But Need Communication, Team meetings,… !

SNL AFRL

SPAWAR

AMES

U of Wy

46th Test Wing

ETI AFLCMC AFRL

AEDC

NAVAIR

MG AV RF Ships

SNL SNL

U of Mich

HPCMP Carderock NAVSEA

Indian Head ONR NRL

HPCMP

CERDEC Penn State

SSI

UIUC CSU

UTx Maui HPCC

U of Mich

Mich State

MIT

GV

ERDC

TARDEC

MSU



Flexible, Agile Software Engineering

– Increased capability annually – Extensive beta-tests of each

release – Rigorous V&V process – Improved scalability for

massively-parallel computers – Improved usability – Responsive to evolving

requirements – Extensive documentation

Software built by government-led teams of 5 to 10 staff • Technical team and team leader embedded in customer institutions • Optimal balance of team agility, structured process, and accountability

Highly Disciplined, but Agile Software Development Processes • Strong emphasis on software quality and accountability • Supportive code development environment - computational clusters, central

servers and code repository, dedicated high performance computers...

Annual releases of each product following a roadmap



Annual CREATE Product Release Cadence

Approximately every year, a fully-tested upgraded code with the new features identified in the roadmap is released

Fiscal Year FY2011 FY2012 FY2013 FY2014 FY2015 FY2016 FY2017* Quarter 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4

AV-DaVinci 1 2 3 4 5

AV-Helios 2 3 4 5 6 7 8

AV-Kestrel 2 3 4 5 6 7 8

MG-Capstone 1 2 3 4 5 6 7

RF-SENTRi 2 3 4 5 6 7

Ships-IHDE 2 3 4 5 6 7 8

Ships-NavyFoam

1 2 3 4 5 6 7

Ships-NESM 1 1.1 2 2.1 3 4 5

Ships-RSDE 0.5 1.0 1.1 1.2 2 3

DP Revised: 6/4/2015

17 June 2016

Planned



CREATE has Defined Core Software Engineering Practices for DoD Physics-based HPC Engineering Software Applications Development Team

1. Lean (<10), close-knit development teams led by technical experts 2. Transparency in development across CREATE projects.

Customer Focus 3. Oversight by senior stakeholder and user representatives 4. Pilots to solicit customer reaction and feedback 5. Frequent reporting to stakeholders

Technical Maturity 6. Proven technologies and customer-defined use-cases 7. VVUQ in alignment with NRC (NAS/NAE) best-practices for scientific codes

Development Methods 8. Milestone-driven workflow management with agile flexible workflow execution and

annual releases 9. Configuration management 10. Code builds based on tests 11. Adequate code documentation

Requirements Definition 12. Reliance on prototypes and use-cases to define requirements



Aerostar & Raven UAVs F/A-18 E/F/G E-2D

NAV

AIR

DDG-1000 CVN-78 Class Ohio SSBN Replacement

LX(R)

NAV

SEA

UH-60 CH-47 (ACRB) Guided Airdrop (RDECOM)

V-22

AR

MY/

US

MC

F-15 SA/DB-110 Strategic Airlift CP&A A-10 B-52

AFL

CM

C

About 130 DoD Acquisition Engineering Organizations (Government and Industry) Now using CREATE Codes



At Last Count, 130 Organizations Use CREATE Tools HCPMP CREATETM Ships: (38) Allion Corporation, Cardinal Engineering, DRS Corporation, DYNAFLOW Corp, General

Dynamics/Electric Boat Division, Hi-Test Laboratory, Northrop Grumman Corp Undersea Systems , Classified Program NSWC Carderock Code 65 , Classified Program NSWC Carderock Code 66 , Naval Underwater Warfare Center, Sandia National Laboratories, Weidlinger & Associates, General Dynamics Land Systems, Hydromechanics Division Naval Surface Warfare Center Carderock Division, Bath Iron Works (shipyard), BMT-Syntek, Bollinger (shipyard), Booz Allen Hamilton, CSC (NAVSEA/PEO engineering contractor), DRS (NAVSEA/PEO engineering contractor), Gibbs and Cox (NAVSEA/PEO engineering contractor), HII (Newport News and Pascagoula shipyards), Lockheed Martin, NASSCO (shipyard), NAVFAC (Naval Facilities), Northrup Grumman, Office of Naval Intelligence, University of Michigan, US Army Corps of Engineers, US Coast Guard, MIT-Department of Naval Architecture , NSWC Carderock Division, Center for Innovative Ship Design, US Coast Guard and Coast Guard Academy, Texas A&M, Naval Postgraduate School, U. of Washington, Virginia Tech, Georgia Tech,

HPCMP CREATE AV: (32) AFLCMC/EN, AFLCMC/XZ, AFAEDC, AFSEO, AF Edwards, AF Hill, AF Holloman, AFRL, NAVAIR/4.3,

NAVAIR/4.10, NAVAIR/Carderock, Army/ADD (Moffett Field), Army/AED (Redstone Arsenal - Aviation), Army/SSDD (Redstone Arsenal - Missiles), Army Research Laboratory (ARL), Army/Nadick Soldier Systems Center), AF Academy (USAFA), AF Institute of Technology (AFIT), USNA, GaTech, BYU, NASA ARC, Boeing Philadelphia/Mesa (Helicopters), Boeing St Louis (Fixed-Wing), Lockheed-Martin, Northrop-Grumman, Raytheon, Sikorsky, Bell Helicopters, Textron, Karem Aircraft, Inc, Mercer Engineering, and Bihrle Applied Research Company

CREATE RF: (55) 57th Intelligence Squadron, 96 Test Wing SK/SKI, AFLCMC, AFRL, AFSEO, Air Force Institute of Technology, Air Force

Research Laboratory (Munitions Directorate), Airborne Threat Simulation Organization (ATSO), AMRDEC-RDMR-SSM-G/Signature Solutions, Inc., Army Research Laboratory, Ball Aerospace & Technologies, Boeing, CERDEC, Cobham Defense Electronics, CSCF, DeposiTech, EP Analytics, Filius, General Atomics Aeronautical Systems, Georgia Tech Research Institute, Global Analytics, Harris Corporation, IERUS Technologies, Leidos, Lockheed Martin Aeronautics, MITRE, NASA Langley Research Center, NASIC/ACNS, NAVAIR, Naval Research Lab, NAWCAD Lakehurst, Northrop Grumman Corporation, NSWC Crane Division, NSWC Dahlgren, Pacific Defense Solutions LLC, Raytheon Company, Raytheon Missile Systems, Raytheon, Space & Airborne Systems, Riverside Research, Rolls-Royce North American Technologies, Signature Solutions, Inc, Sikorsky Aircraft Corporation, Space Command-DoD, Synclesis, Systems Engineering Group, TechFlow, TEDT-WSV-ED, MITRE Corporation, U.S Army CERDEC, U.S. Army Aviation & Missile RDEC, University of Dayton Research Institute, US Army ARDEC, FPAT, METC (RDAR-MEF-E), UTC Pratt & Whitney, Vehicle Technology Directorate, Army Research Lab, Vencore

MG: (5 CREATE AV, Ships and RF) Navy Research Lab (NRL)/ Low-Frequency Broadband (LFBB) Program, NRL/ Strategic

Environmental Research and Development Program (SERDP), NRL/ Jet-noise reduction program, Engineering Research and Design Center(ERDC)-CREEL: Unattended Ground Sensors Programs, ERDC-ITL: Terrain Modeling



BLUF-Bottom Line Up Front We live at the dawn of a New Era

Today, for first time in history, we can accurately predict

the future – The performance of major physical systems using

physics-based HPC computational tools

Challenge is to develop and deploy the software

Based on an High Performance Computing Ecosystem – With virtual instead of physical Research and Engineering Facilities

Paradigm shift with many sociological as well as technical challenges



Questions and Actions “Prediction is very difficult, especially if it's about

the future." – attributed to Niels Bohr

1. What are the advantages of making accurate predictions of the performance of major mechanical systems with physics-based HPC software applications?

2. What challenges do we need to address to be able to make accurate predictions of such systems with physics-based HPC software applications?

3. What do we need to do to overcome those challenges?

4. What are the practical challenges of accurately predicting the future performance of other complex phenomena (weather, climate, biological, material, cultures, etc.)?



Questions?



References 1. Assessing the Reliability of Complex Models: Mathematical and Statistical Foundations of

Verification, Validation, and Uncertainty Quantification, National Academy of Sciences, 2012

2. CREATE: Software Engineering Applications for the Design and Analysis of Air Vehicles, Naval Vessels, and Radio Frequency Antennas, D. Post et al, Computing in Science and Engineering, (2016) 18 (pp.14-24)

3. A Fixed-Wing Aircraft Simulation Tool for Improving DoD Acquisition Efficiency, S. Morton and D. McDaniel, Computing in Science and Engineering, (2016) 18 (pp.25-31)

4. HPCMP CREATE-AV Quality Assurance: Lessons Learned by Validating and Supporting Computation-Based Engineering Software, B. Hallissy, et al, Computing in Science and Engineering, (2016) 18 (pp.52-62)

5. A Risk-Based, Practice-Centered Approach to Project Management for HPCMP CREATE, R. Kendall, et al, Computing in Science and Engineering, (2016) 18 (pp.40-51)

6. Risk-Based Software Development Practices for CREATE Multiphysics HPC Software Applications, R. Kendall, et al, Computing in Science and Engineering, (2016) 18 (pp.2-13)



Questions and Actions “Prediction is very difficult, especially if it's about

the future." – attributed to Niels Bohr

What are the advantages of being able to make accurate performance predictions of major mechanical systems with physics-based HPC software applications?

What challenges are involved in this?

What do we need to do to overcome those challenges?

What are the challenges of accurately predicting the future performance of other complex phenomena (weather, climate, biological systems, materials, cultures, etc.)?

How likely is it that we will succeed with any of these systems.

The Practical Aspects of Computational Science and Engineering · The Practical Aspects of...

Documents

Transcript of The Practical Aspects of Computational Science and Engineering · The Practical Aspects of...