Fundamental concepts and terminology of Verification and Validation

Fundamental concepts and terminology of Verification and ValidationVerification is the process that checks whether mathematical model was implemented accurately enough.Validation is the process that checks that simulation is close enough to reality.This terminology is not universally accepted.1979 figure from Oberkampf and Roy

In order to simulate reality by some calculation, we usually go through two stages. First we postulate a conceptual model of reality, which usually is a mathematical model consisting of a set of algebraic equations and/or differential equations and/or integral equations.

We then implement the solution of these equations, sometimes by hand, more often by writing software that will perform the solution. These two processes are error prone, so we need to check that they were executed properly.

The process of checking that the mathematical model is being solved accurately enough is called verification in our community. The process of comparing the solution to test measurements in order to check if we simulate reality closely enough is called validation.

These terminology is not universally accepted, and is not even well known to most engineers and scientists. However, it is in the process of being established.1Paper Helicopter model qualificationWhen dropped, paper helicopter starts autorotating, which slows down its fall, like a helicopter when it loses power.Has very simple math model: Newtons second law F=maF=Weight-drag

Differential equation

.

2VerificationIn order to verify the computational model and its implementation (e.g. software) we need to test it against analytical solutions or special cases where the solution of the differential equations is known.For the paper helicopter we will first demonstrate with analytical solution, and then with a numerical solution..

Analytical solution of paper helicopter velocity.

4Verifying the analytical solutionWe start by checking that at t=0 the velocity is zero at that at infinity the velocity goes to Vs. Then we substitute solution into the differential equation to check that it is satisfied:

Top hat questionWhat conditions can we apply to check if the expression for dV/dt is correct?We know its value at t=0.We know its value at t=infinity.Both.Neither.Numerical solutionNumerical solution may allow us to relax some of our assumptions, such as constant drag coefficient (may be function of speed).The simplest way to integrate a differential equation is with Euler explicit algorithm with constant time step

Can get also height without the challenge of analytical integrationNow need to implement in software.Will use Matlab and verify against analytical solution.

Matlab codefunction [v,va,h,t]=paper_copter(v0,c,g,T,n)% Calculates trajectory of paper helicopter both % analytically and by Euler explicit integration.dt=T/n; vs=sqrt(g/c);t=linspace(0,T,n+1);va=vs*(1-exp(-2*vs*c*t))./(1+exp(-2*vs*c*t));v(1)=v0;h(1)=0;for i=1:n v(i+1)=(g-c*v(i)^2)*dt+v(i); h(i+1)=v(i)*dt+h(i);end

g=32.2;c=1;T=2;v0=0.;

ButIf we check Matlab code for negative (upward) initial speeds.g=32.2;c=1;T=0.5; v0=-5.7;[v,va,h,t]=paper_copter(v0,c,g,T,10); v -5.7000 -5.7145 -5.7373 -5.7731 -5.8295 -5.9187 -6.0602 -6.2866 -6.6526 -7.2554 -8.2775It is flying into space!What error did I stumble on?

Code verificationIncludes numerical algorithm testing and software quality assurance.However, there are several errors that may create false impression that code is deficient.

Model validationThree stages according to Oberkampf and Trucano

CalibrationModel often includes parameters that need to be estimated from measurements.For paper helicopter we can measure air density and area.We estimate drag coefficient from final speed measurement.We may introduce calibration parameter of fake initial condition to account for period when it falls without spinning.

Distinguishing between alternative modelsConsider the possibility that a model for paper helicopter where drag is proportional to speed instead of square of speed is suggested.That is, Newtons second law isSolution

Top Hat questionWhich of the following is a valid difference between linear and quadratic modelLinear model does not suffer from sign problem when helicopter flies up.Linear model approaches steady state faster.In linear model mass has larger effect on steady state speed.

Equivalent calibrationWe will assume that we measure the steady state speed, and use it for calibration.For quadratic model

For linear model

With errors not be easy to distinguish

Anti-optimization for model discriminationPrinciple: Large errors are easier on debugging than small ones.Example: Comparing finite element and finite strip calculations of buckling loads for stiffened panels I observed 0.3% difference.I knew that the difference should be at most 100 times smaller.I did not have a chance of getting any attention from software developers for this problem.I played with problem parameters until the difference grew to 100%.It became clear that finite-element solution was wrong, and it took developer one day to find the error.

Anti-optimization for composite failure criteriaVan Wamelen A., Haftka, R.T., and Johnson, E.R., "Optimal Design of Laminated Specimens to Evaluate Competing Composite Failure Criteria," Proceedings American Society for Composites, 8th Technical Conference, Technomic Publishing Company, 1993, pp.1045-1055Designed laminate with ply angles that caused two failure criteria that usually give close results to lead to factor of 2 difference.Test validated Tsai-Wu failure criterion, against Hart-Smith criterion.However, developer of competing criterion claimed boundary failure.

Anti-optimization for drag modelsMaximize ratio of quadratic and linear speeds

What can we do to improve experimental conditions for measuring difference?