Ch 2 Systems and Modeling

8/16/2019 Ch 2 Systems and Modeling

1/23

Research Methodology and Presentation – ECEg 4002

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Extractions: by Tezazu Bireda, AAU, Addis Ababa Institute of Technology, ECE Dept May 2012

Chapter 2: Systems and modeling

2.1. Systems: The subject of research as a system concept

The subject on which research is to be done may be conceptualized as a system in its simplest form. A system is generally an individual entity (usually non

isolated) or an assemblage of interacting or interdependent entities (real or abstract) forming an integrated whole and that may have one or more inputs and one

or more recognizable outputs. The input (independent variable) and outputs (dependent variable) associated to the system may be related with known or unknown

(yet to be determined) rules.

In addition to the input and output variables, a system may be subjected to variables of external nature (exogenous) generally called undesired disturbances. The

system itself may be described using system variables (endogenous) or as usually called system parameters.

Fig: Simplified system

Confounding variables are variables with a significant effect on the dependent variable that the researcher failed to control or eliminate - sometimes because the

researcher is not aware of the effect of the confounding variable. The key is to identify possible confounding variables and somehow try to eliminate or control

them.

Most naturally occurring phenomenon and studied in physics, chemistry, biology, nature ; physical occurrences/ processes studied in engineering; phenomenon

in business and society studied in socio-economics; behavioral and social sciences etc all may be visualized and described using the system concept.

A system approach to research can give insights to the researcher on how to make an appropriate experimental setup as well as the modeling and/or simulation of

the problem at hand which helps him intuitively guide what to explore, test, observe and interpret.

Description of the system variables

Depending on the problem at hand, an input may also be called as an excitation, predictor, stimulus, or cause. Correspondingly an output may also be called a

response or an effect.

The input, output, endogenous, exogenous variables as well as the system parameters may be continuous or discrete in time and/or position; time varying

(dynamic) or non-time varying (static/steady); deterministic or stochastic. Further each can be represented either quantitatively and/or qualitatively.

Characterization/types of systems (in the context as a subject of study/reserch )

A system may be of the following character:

Continuous: a continuous system reacts/responds to inputs continuously in time, position and/or other continuous variables.

Discrete: a discrete system reacts/responds to inputs only discretely in time, position and/or other discrete variable. That is reaction occurs only at discrete

instants of time, position and/or other variables.

Time invariant: a time invariant system has parameters or characteristics that do not change with time.

Time variant: a time variant system has at least one parameter or characteristics that changes with time.

Homogenous: the properties or characteristics of a homogenous system components are identical in all locations, times and conditions. As regards to population

study of any type, taking sample from such homogeneity becomes easier. (Stationarity)

Non homogenous: the properties or characteristics of a non-homogenous system components are not identical in all locations, time and conditions. As regards to

population study of any type, taking sample from such non-homogeneity requires a thoughtful strategy. (Non-stationarity)


2/23


-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Dynamic: In a dynamic system, the present output is dependent not only on the present input values but also on the past values represented as initial conditions.

In other words, a dynamic system is said to have memory and thus has associated with it initial conditions. The initial conditions are due to inherent inertial

components which are characteristic of a dynamic system. If describable mathematically, a dynamic system relates its inputs and outputs in the time domain

either by differential equations or difference equations of one or higher order or a state-space method. Frequency domain method can also be used. The

inputs and outputs may vary with time.

Static: In a static system, the present output is dependent only on the present input values and not on the past values. This means that there are no initial

conditions associated to a static system. In other words, a static system is memory-less and has no inertial components. If describable mathematically, a

static system relates its inputs and outputs by algebraic equations that may be linear, quadratic, cubic or any order of polynomials; logarithmic equations,

exponential equations, hyperbolic or trigonometric equations. The inputs and outputs may vary with time. A static system may also be called a zero order

system

Linear: a system is said to be linear if it satisfies the superposition principle. That is, if for input x1 it generates an output y1, and for input x2 it generates an

output y2, etc, then for input x1 + x2 + the output becomes y1 + y2 + …..

Non-linear: a system is said to be nonlinear if it doesn’t satisfy the superposition principle.

Deterministic: for an identical input/s at different instants of time (but with identical initial conditions for dynamic systems) the system generates identical

output/s at the respective time instants.

A deterministic model is one in which every set of variable states is uniquely determined by parameters in the model and by sets of previous states of these

variables. Therefore, deterministic models perform the same way for a given set of initial conditions.

Non-deterministic: for an identical input/s at different instants of time (but with identical initial conditions for dynamic systems) the system generates non-identical (non-unique) output/s [or more specifically random outputs] at the respective time instants. The input and output (as well as the degree of variation

of the output) can only be expressed and determined probabilistically and/or statistically governed by a certain probability distribution.

Causal: a system is said to be causal if it will have out put if and only if after application of input.

Non-causal: If with no input a system has an output then it is said to be non-causal.

Stable: a system is said to be stable if for a finite input it will have a finite out put.

Unstable: a system is said to be unstable if for a finite input it may have an infinite out put.

We may note that a given system may be characterized by a number of (if not all) the above list.

System representation: A system may be mathematically, graphically, and/or qualitatively describable

2.2. Modeling

Scientific model

Scientific modeling is the process of generating abstract conceptual, graphical and/or mathematical model/ description that is supposed to sufficiently (accurately

but simplistically) represent some or most realms of a physical/abstract phenomenon/system.

The Free Dictionary in the Internet defines a model as a description of a system, theory, or phenomenon (in the form of schematics or mathematics) that

accounts for its known or inferred properties and may be used for further study of its characteristics. Examples are the model of an atom, an economic model or a

business model.

Modelling is an essential and inseparable part of all scientific activity, and many scientific disciplines have their own ideas about specific types of modelling.

There is an increasing attention for scientific modelling in fields such as of systems theory, knowledge visualization and philosophy of science.

It is recalled that the system concept may be used to represent a physical, a biological, a socioeconomic as well as abstract entities. Thus, the entity to be studied

in scientific method may be considered as a system.

A scientific model seeks to represent empirical objects, phenomena, and physical processes or generally a system in a logical and objective way.

All models are simplified reflections of reality, but, despite their inherent falsity, they are nevertheless extremely useful. Building and disputing [to question or

doubt the truth or validity of something] models is fundamental to the scientific enterprise. Complete and true representation may be impossible, but scientific

debate often concerns which is the better model for a given task, e.g., which is the more accurate climate model for seasonal forecasting.


3/23


-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Attempts to formalize the principles of the empirical sciences use an interpretation to model reality in the same way logicians axiomatize the principles of logic.

The aim of these attempts is to construct a formal system that will not produce theoretical consequences that are contrary to what is found in reality. Predictions

or other statements drawn from such a formal system mirror or map the real world only insofar as these scientific models are true.

Modelling as a substitute for direct measurement and experimentation

Direct measurement of outcomes under controlled conditions in scientific method will always be more reliable than modelled estimates of outcomes.

Models are typically used when it is either impossible or impractical to create experimental conditions in which scientists can directly measure outcomes.

Models can also be used in companion with empirical experiments in order to facilitate and accelerate a study. By feeding findings of an experiment to a model

under development or an outcome of a model as an insightful manipulation of the experiment, studies can be accelerated towards convergence to solutions and

conclusions faster.

Reasons we do modeling and simulation:

We are constrained by linear thinking: We cannot understand how all the various parts of the system interact and add up to the whole

We cannot imagine all the possibilities that the real system could exhibit

We cannot foresee the full effects of cascading and/or layering upon of events with our limited mental models

We cannot foresee novel events that our mental models cannot even imagine

• We model for insights, not numbers

As an exercise in “thought space” to gain insights into key variables and their causes and effects

To construct reasonable arguments as to why events can or cannot occur based on the model

• We model to make qualitative or quantitative predictions about the future

The process of generating a model

Modelling refers to the process of generating a model as a conceptual representation of some phenomenon. Modelling requires selecting and identifying relevant

aspects of a situation in the real world. Typically a model will refer only to some aspects of the phenomenon in question. Two models of the same phenomenon

may be essentially different, that is to say that the differences between them are more than just a simple renaming of components.

Such differences may be due to differing requirements of the model's end users, or to conceptual or aesthetic differences among the modellers and to contingent

decisions made during the modelling process.

Aesthetic considerations that may influence the structure of a model might be the modeller's preference for a reduced ontology (nature of being), preferences

regarding probabilistic models vis-a-vis deterministic ones, discrete vs continuous time, etc. For this reason, users of a model need to understand the model's

original purpose and the assumptions made that are pertinent to its validity.

The process of evaluating a model

A model is evaluated first and foremost by its consistency to empirical data. Any model inconsistent with reproducible observations must be modified or rejected

One way to modify the model is by restricting the domain over which it is credited with having high validity.


4/23


-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


A case in point is Newtonian physics, which is highly useful except for the very small, the very fast, and the very massive phenomena of the universe. However,

a fit to empirical data alone is not sufficient for a model to be accepted as valid. Other factors important in evaluating a model include:

• Ability to explain past observations

•

Ability to predict future observations

• Cost of use, especially in combination with other models

• Refutability, enabling estimation of the degree of confidence in the model

• Simplicity, or even aesthetic appeal

One may attempt to quantify the evaluation of a model using a utility function.

Simulation and visualization

For the scientist, a model is also a way in which the human thought processes can be amplified. For instance, models that are rendered in software allow scientists

to leverage computational power to simulate, visualize, manipulate and gain intuition about the entity, phenomenon, or process being represented. Such compute

models are called in silico (an expression used to mean "performed on computer or via computer simulation.").

Other types of scientific model may be in vivo (living models, such as laboratory rats), in vitro (in glassware, such as tissue culture) and a lot more types in other

disciplines such as pilot plants.

Simulation

A simulation is the implementation of a model. A steady state simulation provides information about the system at a specific instant in time usually at

equilibrium. A dynamic simulation provides information over time. A simulation brings a model to life and shows how a particular object or phenomenon will

behave. Such a simulation can be useful for testing, analysis, or training in those cases where real-world systems or concepts can be represented by models.

Simulation approach involves the construction of an artificial environment within which relevant information and data can be generated. This permits an

observation of the dynamic behaviour of a system (or its sub-system) under controlled conditions.

The term ‘simulation’ in the context of business and social sciences applications refers to “the operation of a numerical model that represents the structure of a

dynamic process. Given the values of initial conditions, parameters and exogenous variables, a simulation is run to represent the behaviour of the process over

time.”

Simulation approach can also be useful in building models for understanding future conditions.

Visualization is any technique for creating images, diagrams, or animations to communicate a message. Visualization through visual imagery has been an

effective way to communicate both abstract and concrete ideas since the dawn of man. Examples from history include cave paintings, Egyptian hieroglyphs,

Greek geometry, and Leonardo da Vinci's revolutionary methods of technical drawing for engineering and scientific purposes.


5/23


-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


One application of scientific modelling is in the field of "Modelling and Simulation", generally referred to as "M&S". M&S has a spectrum of applications which

range from concept development and analysis, through experimentation, measurement and verification, to disposal analysis.

Examples of some computational, simulation and visualization tools in science and engineering include:

MATLAB,

MathCAD,

Mathematica,

LabView of NI MultiSim,

Proteus

NS or Packet Tracer

etc

Mathematical model

A mathematical model is a description of a system using mathematical language. That is, a mathematical model relates the input variables and output variables of

a system representing a real world phenomenon in order to describe it in the form of one or a set of equations. Variables are abstractions of quantities of interest

in the described systems.

The process of developing a mathematical model is termed mathematical modelling.

Mathematical models are used not only in the natural sciences (such as physics, biology, earth science, meteorology) and engineering disciplines, but also in the

social sciences (such as economics, psychology, sociology and political science).

Physicists, engineers, statisticians, operations research analysts and economists use mathematical models most extensively.

Mathematical models can take many forms, including but not limited to dynamical systems, statistical models, differential equations, or game theoretic models.

These and other types of models can overlap, with a given model involving a variety of abstract structures.

The purpose of the mathematical model of the system may be at least either of the following:

i) Description: i.e. to sufficiently (accurately but simply) describe the characteristics of the real world physical phenomenon described as the model

when subjected to input as well as exogenous variables

ii) (Inferential) to explore how the system parameters can be varied/manipulated in order to obtain an optimal performance (response) for a given set

of input variables and a known set of external factors. Studying predictive (interpolating and extrapolating) accuracy is also another purpose of thi

category.

In an optimization problem, the function to be optimized is called an objective function. The conditions that should be satisfied during optimization are called

constraints of the system. Objective functions will depend on the perspective of the model's user. Depending on the context, an objective function is also known

as an index of performance, as it is some measure of interest to the user. Although there is no limit to the number of objective functions and constraints a model

can have, using or optimizing the model becomes more involved (computationally) as the number increases.

Description and performance optimization are, thus, two important purposes of a mathematical model of a real world physical or abstract system.

Thus, often when engineers are interested to study a system to be controlled or optimized, they use a mathematical model. In analysis, engineers can build adescriptive model of the system as a hypothesis of how the system could work, or try to estimate how an unforeseeable event could affect the system.

Similarly, in control of a system, engineers can try out different control approaches in simulations.

The mathematical model of a system may be classified in accordance to the type of system or operator. Thus we may have linear or non-linear models;

deterministic or non-deterministic (stochastic/probabilistic) model; static or dynamic model etc.


6/23


-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


The system manifests itself as a mathematical operator on the input variables to deliver the output variables as a result. The type of operator depends on the type

of system. Thus we may have an algebraic operator, a differential operator etc. The system itself has parameters called endogenous variables. The system may

also be subjected to external undesired but inevitable factors such as noise manifested and are termed as exogenous variables.

If all the operators in a mathematical model exhibit linearity, the resulting mathematical model is defined as linear. A model is considered to be nonlinear

otherwise.

However, the question of linearity and nonlinearity is dependent on context, and linear models may have nonlinear expressions in them. For example, in a

statistical linear model, it is assumed that a relationship is linear in the parameters, but it may be nonlinear in the predictor variables.

Similarly, a differential equation is said to be linear if it can be written with linear differential operators, but it can still have nonlinear expressions in it.

In a mathematical programming model, if the objective functions and constraints are represented entirely by linear equations, then the model is regarded as a

linear model. If one or more of the objective functions or constraints are represented with a nonlinear equation, then the model is known as a nonlinear model.

Nonlinearity, even in fairly simple systems, is often associated with phenomena such as chaos and irreversibility. Although there are exceptions, nonlinear

systems and models tend to be more difficult to study than linear ones. A common approach to nonlinear problems is linearization, but this can be problematic if

one is trying to study aspects such as irreversibility, which are strongly tied to nonlinearity.

Mathematical models can be divided into either analytical models or numerical models.

An analytical model is a mathematical model that has a closed form expression, i.e. the output and input variables can be related in such a way that the two sets

of variables are disassociated from themselves or expressed as a mathematical analytic function.

On the other hand, a numerical model is a mathematical model that uses some sort of numerical time-stepping procedure to obtain the models behavior over time

Eg recursive methods

A priori information

Mathematical modelling problems are often classified into black box or white box models, according to how much a priori information is available of the system

A black-box model is a system of which there is no a priori information available. A white-box model (also called glass box or clear box) is a system where all

necessary information is available. Practically all systems are somewhere between the black-box and white-box models, so this concept is useful only as an

intuitive guide for deciding which approach to take.

Usually it is preferable to use as much a priori information as possible to make the model more accurate. Therefore the white-box models are usually considered

easier, because if you have used the information correctly, then the model will behave correctly. Often the a priori information comes in forms of knowing the

type of functions relating different variables.

For example, suppose we want to make a model of how a medicine works in a human system knowing the fact that (i.e. a priori information) usually the amount

of medicine in the blood is an exponentially decaying function. But we are still left with several unknown parameters; how rapidly does the medicine amount

decay, and what is the initial amount of medicine in blood? This example is therefore not a completely white-box model. These parameters have to be estimated

through some means (say experiment) before one can use the model.

In black-box models one tries to estimate both the functional form of relations between variables (input and output) and the numerical parameters (of the system)

in those functions. Using a priori information we could end up, for example, with a set of functions that probably could describe the system adequately. If there is

no a priori information we would try to use functions as general as possible to cover all different models.

An often used approach for black-box models are neural networks which usually do not make assumptions about incoming data. The problem with using a largeset of functions to describe a system is that estimating the parameters becomes increasingly difficult when the amount of parameters (and different types of

functions) increases.

Any model which is not pure white-box contains some parameters that can be used to fit the model to the system it is intended to describe. If the modelling is

done by a neural network, the optimization of parameters is called training. In more conventional modelling through explicitly given mathematical functions,

parameters are determined by curve fitting.


7/23


-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Model complexity vs accuracy

In general, model complexity involves a trade-off between simplicity and accuracy of the model. Occam's Razor is a principle particularly relevant to modelling

the essential idea being that among models with roughly equal predictive power, the simplest one is the most desirable.

While added complexity usually improves the realism of a model, it can make the model difficult to understand and analyze, and can also pose computational

problems, including numerical instability. Thomas Kuhn argues that as science progresses, explanations tend to become more complex before a Paradigm shift

offers radical simplification.

That is, when a proposed solution hypothesis of a problem is not sufficient in its explanation of essential scenarios of the problem at hand, then the hypothesis

needs to be either modified or altered by a new one (p aradigm shift).

For example, when modelling the flight of an aircraft, we could embed each mechanical part of the aircraft into our model and would thus acquire an almost

white-box model of the system. However, the computational cost of adding such a huge amount of detail would effectively inhibit the usage of such a model.

Additionally, the uncertainty would increase due to an overly complex system, because each separate part induces some amount of variance into the model. It is

therefore usually appropriate to make some approximations to reduce the model to a sensible size. Engineers often can accept some approximations in order to

get a more robust and simple model.

For example Newton's classical mechanics is an approximated model of the real world. Still, Newton's model is quite sufficient for most ordinary-life situations,

that is, as long as particle speeds are well below the speed of light, and we study macro-particles only.

Model evaluation

A crucial part of the modelling process is the evaluation of whether or not a given mathematical model describes a system accurately. This question can be

difficult to answer as it involves several different types of evaluation.

Fit to empirical data

Usually the easiest part of model evaluation is checking whether a model fits experimental measurements or other empirical data. In models with parameters, a

common approach to test this fit is to split the data into two disjoint subsets: training data and verification data. The training data are used to estimate the model

parameters. An accurate model will closely match the verification data even though this data was not used to set the model's parameters. This practice is referred

to as cross-validation in statistics.

Defining a metric to measure distances between observed (measured) and predicted data is a useful tool of assessing model fit. In statistics, decision theory, and

some economic models, a loss function plays a similar role.

While it is rather straightforward to test the appropriateness of parameters, it can be more difficult to test the validity of the general mathematical form of a

model. In general, more mathematical tools have been developed to test the fit of statistical models than models involving Differential equations. Tools from

nonparametric statistics can sometimes be used to evaluate how well data fits a known distribution or to come up with a general model that makes only minimal

assumptions about the model's mathematical form.

Scope of the model

Assessing the scope of a model, that is, determining what situations the model is applicable to, can be less straightforward. If the model was constructed based on

a set of data, one must determine for which systems or situations the known data is a "typical" set of data.

The question of whether the model describes well the properties of the system between data points is called interpolation, and the same question for events or

data points outside the observed data is called extrapolation.

As an example of the typical limitations of the scope of a model, in evaluating Newtonian classical mechanics, we can note that Newton made his measurements

without advanced equipment, so he could not measure properties of particles traveling at speeds close to the speed of light. Likewise, he did not measure the

movements of molecules and other small particles, but macro particles only. It is then not surprising that his model does not extrapolate well into these domains,

even though his model is quite sufficient for ordinary life physics.


8/23


-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Philosophical considerations

Many types of modelling implicitly involve claims about causality. This is usually (but not always) true of models involving differential equations. As the

purpose of modelling is to increase our understanding of the world, the validity of a model rests not only on its fit to empirical observations, but also on its

ability to extrapolate to situations or data beyond those originally described in the model.

One can argue that a model is worthless unless it provides some insight which goes beyond what is already known from direct investigation of the phenomenon

being studied.

Examples of mathematical models in engineering, science, demography, economics etc

• Population Growth. A simple (though approximate) model of population growth is the Malthusian growth model, sometimes called the simple

exponential growth model, is essentially exponential growth based on a constant rate of compound interest. The model is named after the Reverend

Thomas Malthus, who authored An Essay on the Principle of Population, one of the earliest and most influential books on population and exponential

growth.

where

P0 = initial population, r = growth rate, sometimes also called Malthusian parameter, t = time.

• Model of a particle in a potential-field . In this model we consider a particle as being a point of mass which describes a trajectory in space which is

modeled by a function giving its coordinates in space as a function of time. The potential field is given by a function V : R3 R and the trajectory is

a solution of the differential equation

Note this model assumes the particle is a point mass, which is certainly known to be false in many cases in which we use this model; for example, as a

model of planetary motion.

•

Model of rational behavior of a consumer . In this model we assume a consumer faces a choice of n commodities labeled 1,2,...,n each with a marketprice p1, p2,..., pn. The consumer is assumed to have a cardinal utility function U (cardinal in the sense that it assigns numerical values to utilities),

depending on the amounts of commodities x1, x2,..., xn consumed. The model further assumes that the consumer has a budget M which is used to

purchase a vector x1, x2,..., xn in such a way as to maximize U ( x1, x2,..., xn). The problem of rational behavior in this model then becomes an

optimization problem, that is:

subject to:

This model has been used in general equilibrium theory, particularly to show existence and Pareto efficiency of economic equilibria. However, the

fact that this particular formulation assigns numerical values to levels of satisfaction is the source of criticism (and even ridicule). However, it is not

an essential ingredient of the theory and again this is an idealization.


9/23


-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Statistics and statistical modelling

“... it is only the manipulation of uncertainty that interests us. We are not concerned with the matter that is uncertain. Thus we do not study the mechanism of

rain; only whether it will rain.” Dennis Lindley, "The Philosophy of Statistics", The Statistician (2000).

Overview of statistics

Statistics is the science of the collection, organization, and interpretation of data. It deals with all aspects of this, including the planning of data collection

(generation) in terms of the design of experiments and surveys.

The mathematical foundations of statistics were laid in the 17th century with the development of probability theory by Blaise Pascal and Pierre de Fermat.

Probability theory arose from the study of games of chance. The method of least squares was first described by Carl Friedrich Gauss around 1794. The use of

modern computers has expedited large-scale statistical computation, and has also made possible new methods that are impractical to perform manually.

Because of its empirical roots and its focus on applications, statistics is usually considered to be a distinct mathematical science rather than a branch of

mathematics.

The data and variables mostly dealt with statistics are random or stochastic in their nature. This means that a random or a stochastic data or variable is one whose

magnitude and/or occurrence is uncertain and estimated only probabilistically. Random variables are generated from a random process or equivalently from a

non-deterministic system.

Statistics is thus closely related to probability theory, with which it is often grouped. Randomness is studied using probability theory, which is also a

mathematical discipline. Probability is used to study the sampling distributions of sample statistics and, more generally, the properties of statistical procedures.

The use of any statistical method is valid when the system or population under consideration satisfies the assumptions of the method.

The quality of (collected or generated) data is improved by carefully employing two important techniques: design of experiments and survey sampling. Statistics

is not only used to describe about an entity but also provides tools for prediction and forecasting using data and statistical models.

Statistics is applicable to a wide variety of academic disciplines, including natural and social sciences, government, and business. Statistics also form a key basis

tool in business and manufacturing. It is used to understand measurement systems variability, control processes (as in statistical process control or SPC), for

summarizing data, and to make data-driven decisions. In these roles, it is a key tool, and perhaps the only reliable tool.

Population and sample

In applying statistics to a scientific, industrial, or societal problem, it is necessary to begin with a population or process to be studied. Populations can be diversetopics such as "all persons living in a country" or "every atom composing a crystal". A population can also be composed of observations of a process at various

times, with the data from each observation serving as a different member of the overall group. Data collected about this kind of "population" constitutes what is

called a time series.

For practical reasons, a chosen subset of the population called a sample is studied — as opposed to compiling data about the entire group (an operation called

census). Once a sample that is representative of the population is determined, data is collected for the sample members in an experimental or observational

setting. This data can then be subjected to statistical analysis, serving two related purposes: description and inference.

For a sample to be used as a guide to an entire population, it is important that it is truly a representative of that overall population. Representative sampling

assures that the inferences and conclusions can be safely extended from the sample to the population as a whole. A major problem lies in determining the extent

to which the sample chosen is actually representative. Statistics offers methods to estimate and correct for any random trending within the sample and data

collection procedures. There are also methods for designing experiments that can lessen these issues at the outset of a study, strengthening its capability to

discern truths about the population. Statisticians describe stronger methods as more "robust".

Two main statistical methods (purposes) [for analysis and synthesis]

Descriptive statistics and inferential statistics (a.k.a., predictive statistics) form the two main methods and together comprise applied statistics to which sample

data from experiment or observation are subjected.

• Descriptive statistics: summarizes the population data by describing what was observed in the sample numerically or graphically. Numerical

descriptors include mean, variance and standard deviation for continuous data types (like heights or weights), while frequency and percentage are

more useful in terms of describing categorical data (like race). This method is useful in research, when communicating the results of experiments.


10/23


-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


• Inferential statistics: uses patterns in the sample data to draw inferences (predictions) about the population represented, accounting for randomness.

These inferences may take the form of:

o modeling relationships within the data (regression), extrapolation, interpolation, or other modeling techniques like ANOVA (Analysis of

variance), time series, and data mining. This means patterns in the data may be modeled in a way that accounts for randomness and

uncertainty in the observations, and are then used to draw inferences about the process (system) or population being studied;

o answering yes/no questions about the data (hypothesis testing),

o estimating numerical characteristics of the data (estimation),

o describing associations within the data (correlation),

Inference is a vital element of scientific advance, since it provides a prediction (based on data) for where a theory logically leads. To further prove the guiding

theory, these predictions are tested (i.e hypothesis testing) as well, as part of the scientific method. If the inference holds true, then the descriptive statistics of the

new data increase the soundness of that hypothesis.

The concept of correlation is particularly noteworthy for the potential confusion it can cause. Statistical analysis of a data set often reveals that two variables

(properties) of the population under consideration tend to vary together, as if they were connected. For example, a study of annual income that also looks at age

of death might find that poor people tend to have shorter lives than affluent people. The two variables are said to be correlated; however, they may or may not be

the cause of one another. The correlation phenomena could be caused by a third, previously unconsidered phenomenon, called a lurking variable or confounding

variable. For this reason, there is no way to immediately infer the existence of a causal relationship between the two variables. Correlation does not imply

causation.

Experimental and observational studies

A common goal for a statistical research project is to investigate causality, and in particular to draw a conclusion on the effect of changes in the values of

predictors or independent variables on dependent variables or response.

There are two major types of causal statistical studies: experimental studies and observational studies. In both types of studies, the effect of differences of an

independent variable (or variables) on the behavior of the dependent variable are observed. The difference between the two types lies in how the study is actually

conducted. Each can be very effective.

An experimental study involves taking measurements of the system under study, manipulating the system, and then taking additional measurements using the

same procedure to determine if the manipulation has modified the values of the measurements.

In contrast, an observational study does not involve experimental manipulation. Instead, data are gathered and correlations between predictors and response are

investigated.

Experiments

The basic steps of a statistical experiment are:

1.

Planning the research, including finding the number of replicates of the study, using the following information:

Preliminary estimates regarding the size of treatment effects, alternative hypotheses, and the estimated experimental variability.

Consideration of the selection of experimental subjects and the ethics of research is necessary.

Statisticians recommend that experiments compare (at least) one new treatment with a standard treatment or control, to allow an unbiased estimate of

the difference in treatment effects.

2.

Design of experiments, using blocking to reduce the influence of confounding variables, and randomized assignment of treatments to subjects to

allow unbiased estimates of treatment effects and experimental error.

At this stage, the experimenters and statisticians write the experimental protocol that shall guide the performance of the experiment and that specifies

the primary analysis of the experimental data.


11/23


-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


3. Performing (execute) the experiment and analyzing the data following the experimental protocol.

4.

Further examining the data set in secondary analyses, to suggest new hypotheses for future study.

5. Documenting and presenting the results of the study.

Experiments on human behavior have special concerns. The famous Hawthorne study examined changes to the working environment at the Hawthorne plant of

the Western Electric Company.

The researchers were interested in determining whether increased illumination would increase the productivity of the assembly line workers. The researchers first

measured the productivity in the plant, then modified the illumination in an area of the plant and checked if the changes in illumination affected productivity.

It turned out that productivity indeed improved (under the experimental conditions). However, the study is heavily criticized today for errors in experimental

procedures, specifically for the lack of a control group and blindness. The Hawthorne effect refers to finding that an outcome (in this case, worker productivity)

changed due to observation itself. Those in the Hawthorne study became more productive not because the lighting was changed but because they were being

observed .

Observational study

An example of an observational study is one that explores the correlation between smoking and lung cancer. This type of study typically uses a survey to collect

observations about the area of interest and then performs statistical analysis. In this case, the researchers would collect observations of both smokers and non-

smokers, perhaps through a case-control study, and then look for the number of cases of lung cancer in each group.

Levels of measurement

There are four main levels of measurement used in statistics:

• ratio

• interval,

• ordinal and

• nominal,

They have different degrees of usefulness in statistical research.

Ratio measurements have both a meaningful zero value and the d istances between different measurements defined; they provide the greatest flexibility in

statistical methods that can be used for analyzing the data.

Interval measurements have meaningful distances between measurements defined, but the zero value is arbitrary (as in the case with longitude and temperature

measurements in Celsius or Fahrenheit).

Ordinal measurements have imprecise differences between consecutive values, but have a meaningful order to those values.

Nominal measurements have no meaningful rank order among values.

Because variables conforming only to nominal or ordinal measurements cannot be reasonably measured numerically, sometimes they are grouped together as

categorical variables.

On the other hand ratio and interval measurements are grouped together as quantitative or continuous variables due to their numerical nature.

Key terms used in statistics: Null hypothesis, alternative hypothesis and hypothesis testing

A hypothesis is a suggested explanation of a phenomenon.

Interpretation of statistical information can often involve the development of a null hypothesis in that the assumption is that whatever is proposed as a cause has

no effect on the variable being measured.


12/23


-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


A null hypothesis is a hypothesis which a researcher tries to disprove. Normally, the null hypothesis represents the current view/explanation of an aspect of the

world that the researcher wants to challenge.

Research methodology involves the researcher providing an alternative hypothesis, a research hypothesis, as an alternate way to explain the phenomenon.

The researcher tests the hypothesis to disprove the null hypothesis, not because he/she loves the research hypothesis, but because it would mean coming closer to

finding an answer to a specific problem. The research hypothesis is often based on observations that evoke suspicion that the null hypothesis is not always

correct.

The best illustration for a novice is the predicament encountered by a jury trial . The null hypothesis, H0, asserts that the defendant is innocent, whereas the

alternative hypothesis, H1, asserts that the defendant is guilty.

The indictment comes because of suspicion of the guilt. The H0 (status quo) stands in opposition to H1 and is maintained unless H1 is supported by

evidence “beyond a reasonable doubt”.

However, “failure to reject H0” in this case does not imply innocence, but merely that the evidence was insufficient to convict.

So the jury does not necessarily accept H0 but fails to reject H0. While to the casual observer the difference appears moot, misunderstanding the difference is one

of the most common and arguably most serious errors made by non-statisticians.

Failure to reject the H0 does NOT prove that the H0 is true, as any crook with a good lawyer who gets off because of insufficient evidence can attest to.

While one can not “prove” a null hypothesis one can test how close it is to being true with a power test, which tests for type II errors.

Error (i.e. errors in research)

Working from a null hypothesis, logically there are two basic forms of errors that are recognized when drawing conclusions in research:

• Type I errors:

where the null hypothesis is falsely rejected giving a "false positive".

That is, it is when we accept the alternative hypothesis when the null hypothesis is in fact correct.

• Type II errors:

where the null hypothesis fails to be rejected and an actual difference between populations is missed.

That is, when we reject the alternative hypothesis even if the null hypothesis is wrong.

Error also refers to the extent to which individual observations in a sample differ from a central value, such as the sample or population mean. Many statistical

methods seek to minimize the mean-squared error, and these are called "methods of least squares."

Measurement processes that generate statistical data are also subject to error. Many of these errors are classified as random (noise) or systematic (bias), but other

important types of errors (e.g., blunder, such as when an analyst reports incorrect units) can also be important.

Confidence intervals

Most studies will only sample part of a population and then the result is used to interpret the null hypothesis in the context of the whole population. Any

estimates obtained from the sample only approximate the population value.

Confidence intervals allow statisticians to express how closely the sample estimate matches the true value in the whole population. Often they are expressed as

95% confidence intervals. Formally, a 95% confidence interval of a procedure is a range where, if the sampling an analysis were repeated under the same

conditions, the interval would include the true (population) value 95% of the time. This does not imply that the probability that the true value is in the confidence

interval is 95%. (From the frequentist perspective, such a claim does not even make sense, as the true value is not a random variable. Either the true value is or is

not within the given interval.)


13/23


-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


One quantity that is in fact a probability for an estimated value is the credible interval from Bayesian statistics.

Statistical Significance

Statistics rarely give a simple Yes/No type answer to the question asked of them. Interpretation often comes down to the level of statistical significance applied to

the numbers and often refer to the probability of a value accurately rejecting the null hypothesis (sometimes referred to as the p-value).

Referring to statistical significance does not necessarily mean that the overall result is significant in real world terms. For example, in a large study of a drug it

may be shown that the drug has a statistically significant but very small beneficial effect, such that the drug will be unlikely to help the patient in a noticeable

way.

Some well-known statistical tests and procedures are:

• Analysis of variance (ANOVA)

• Chi-square test

• Correlation

• Factor analysis

• Student's t-test

• Time series analysis

Statistical computing

The rapid and sustained increases in computing power starting from the second half of the 20th century have had a substantial impact on the practice of statistical

science.

Early statistical models were almost always from the class of linear models. However, nowadays, powerful computers, coupled with suitable numerical

algorithms, caused an increased interest in nonlinear models (such as neural networks) as well as the creation of new types, such as generalized linear models

and multilevel models.

Increased computing power has also led to the growing popularity of computationally intensive methods based on resampling, such as permutation tests and the

bootstrap, while techniques such as Gibbs sampling have made use of Bayesian models more feasible.

The computer revolution has implications for the future of statistics with new emphasis on "experimental" and "empirical" statistics. A large number of both

general and special purpose statistical software are now available such as gretl , which is an example of an open source statistical package.

Misuse of statistics

Misuse of statistics can produce subtle, but serious errors in description and interpretation — subtle in the sense that even experienced professionals make such

errors, and serious in the sense that they can lead to devastating decision errors. For instance, social policy, medical practice, and the reliability of structures like

bridges all rely on the proper use of statistics. Even when statistics are correctly applied, the results can be difficult to interpret for those lacking expertise.

The statistical significance of a trend in the data — which measures the extent to which a trend could be caused by random variation in the sample — may or

may not agree with an intuitive sense of its significance. The set of basic statistical skills (and skepticism) that people need to deal with information in their

everyday lives properly is referred to as statistical literacy.

There is a general perception that statistical knowledge is all-too-frequently intentionally misused by finding ways to interpret only the data that are favorable to

the presenter.

The famous saying, "There are three kinds of lies: lies, damned lies, and statistics". which was popularized in the USA by Samuel Clemens, has come to

represent the general mistrust [and misunderstanding] of statistical science.

Harvard President Lawrence Lowell wrote in 1909 that statistics, "...like veal pies, are good if you know the person that made them, and are sure of the

ingredients."

If various studies appear to contradict one another, then the public may come to distrust such studies. For example, one study may suggest that a given diet or

activity raises blood pressure, while another may suggest that it lowers blood pressure. The discrepancy can arise from subtle variations in experimental design,


14/23


-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


such as differences in the patient groups or research protocols, which are not easily understood by the non-expert. (Media reports usually omit this vital

contextual information entirely, because of its complexity.)

By choosing (or rejecting, or modifying) a certain sample, results can be manipulated. Such manipulations need not be malicious or devious; they can arise from

unintentional biases of the researcher. The graphs used to summarize data can also be misleading.

Deeper criticisms come from the fact that the hypothesis testing approach, widely used and in many cases required by law or regulation, forces one hypothesis

(the null hypothesis) to be "favored," and can also seem to exaggerate the importance of minor differences in large studies. A difference that is highly statistically

significant can still be of no practical significance. (See criticism of hypothesis testing and controversy over the null hypothesis.)

One response is by giving a greater emphasis on the p-value than simply reporting whether a hypothesis is rejected at the given level of significance. The p-value

however, does not indicate the size of the effect. Another increasingly common approach is to report confidence intervals. Although these are produced from the

same calculations as those of hypothesis tests or p-values, they describe both the size of the effect and the uncertainty surrounding it.

Statistical modelling

A statistical model is a formalization of relationships between variables in the form of mathematical equations. A statistical model describes how one or more

random variables are stochastically related to one or more random variables.

Statistical models include issues such as statistical characterization of numerical data, estimating the probabilistic future behavior of a system based on past

behavior, extrapolation or interpolation of data based on some best-fit, error estimates of observations, or spectral analysis of data or model generated output.

In mathematical terms, a statistical model is frequently thought of as a pair (Y, P) where Y is the set of possible observations and P is the set of possible

probability distributions on Y. It is assumed that there is a distinct element of P which generates the observed data. Statistical inference enables us to make

statements about which element(s) of this set are likely to be the true one.

Most statistical tests can be described in the form of a statistical model. For example, the Student's t-test for comparing the means of two groups can be

formulated as seeing if an estimated parameter in the model is different from 0. Another similarity between tests and models is that there are assumptions

involved. Error is assumed to be normally distributed in most models.[1]

More formally a statistical model, P, is a collection of probability distribution or density functions. A parametric model is a collection of distributions, each of

which is indexed by a unique finite-dimensional parameter: , where is a parameter and is the feasible region

of parameters, which is a subset of d-dimensional Euclidean space.

A statistical model may be used to describe the set of distributions from which one assumes that a particular data set is sampled. For example, if one assumes that

data arise from a univariate Gaussian distribution, then one has assumed a Gaussian model:

.

Performance of a student class, for instance, can best be modelled using normal distribution.

Arrival of a customer in a bank queue or call making to a telephone network is modelled using Poison (exponentially distributed) distribution.

Other models include Rayleigh distribution, etc.

A non-parametric model is a set of probability distributions with infinite dimensional parameters, and might be written as .

A semi-parametric model also has infinite dimensional parameters, but is not dense in the space of distributions. For example, a mixture of Gaussians with one

Gaussian at each data point is dense is the space of distributions. Formally, if d is the dimension of the parameter, and n is the number of samples, if

as and as , then the model is semi-parametric.


15/23


-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Comparison of statistical models

Models can be compared to each other. This can either be done when you have done an exploratory data analysis or a confirmatory data analysis.

In an exploratory analysis, you formulate all models you can think of, and see which describes your data best.

In a confirmatory analysis you test which of your models you have described before the data was collected fits the data best, or test if your only model fits the

data.

In linear regression analysis you can compare the amount of variance explained by the independent variables, R2, across the different models.

In general, you can compare models that are nested by using a Likelihood-ratio test. Nested models are models that can be obtained by restricting(setting) a

parameter in a more complex model to be zero.

Example

Length and age are probabilistically distributed over humans. They are stochastically related, when you know that a person is of age 7, this influences the chance

of this person being 6 feet tall. You could formalize this relationship in a linear regression model of the following form: lengthi = b0 + b1agei + i, where b0 is the

intercept, b1 is a parameter that age is multiplied by to get a prediction of length, is the error term, and i is the subject.

This means that length starts at some value, there is a minimum length when someone is born, and it is predicted by age to some amount. This prediction is not

perfect as error is included in the model. This error contains variance that stems from sex and other variables.

When sex is included in the model, the error term will become smaller, as you will have a better idea of the chance that a particular 16-year-old is 6 feet tall when

you know this 16-year-old is a girl.

The model would become lengthi = b0 + b1agei + b2sexi + i, where the variable sex is dichotomous. This model would presumably have a higher R2. The first

model is nested in the second model: the first model is obtained from the second when b2 is restricted to zero.

Classification of statistical models

According to the number of the endogenous variables and the number of equations, models can be classified as complete models (the number of equations equals

to the number of endogenous variables) and incomplete models.

Some other statistical models are the general linear model (restricted to continuous dependent variables), the generalized linear model (for example, logistic

regression), the multilevel model, and the structural equation model.


16/23


-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Empirical modeling

Empirical models are those (mathematical or graphical expressions) that are based entirely on data. The important distinction between empirical models and

analytical models is that the empirical models are not derived from assumptions concerning the relationship between variables and they are not based on

physical principles.

The first step in deriving an empirical models is thus to get the scatter-plot of the data.

Interaction between data and models occurs in a number of ways:

Data are needed to suggest a right model. The models called empirical are based entirely on data. [training data]

Data are needed to estimate the values of the parameters appearing in a model. This is sometimes called calibrating a model .

Data are needed to test a model. [verification/ testing data]

It happens very often that the data given at the beginning is not sufficient for making a good model. In these cases further data collection is needed. Considering

the following questions might be useful:

What is the relevant data? Exactly what kind of data is needed?

How can the relevant data be obtained?

In what form do you need the data?

Once the data is collected, you need to decide on the techniques you want to use in order to find an appropriate model. There are two major groups of techniques

based on two different ideas

Interpolation- finding a function that contains all the data points.

Model fitting - finding a function that is as close as possible to containing all the data points. Such function is also called a regression curve.

Sometimes you would need to combine these methods since the interpolation curve might be too complex and the best fit model might not be sufficiently

accurate.

Model Fitting: Modeling using Regressions

Most software applications such as Excel, Matlab and also graphing calculators can be used to find regression curves and a variable monitoring the validity of the

model, the coefficient of determination usually denoted by R2. This coefficient takes values in interval [0,1] and indicates how close the data points are to be

exactly on the regression curve. If R2 is close to 1, the model is reliable. If R2 is close to 0, other model should be considered.

Some possible types models for curve fitting

i. Linear: y = ax+b: Easiest, simplest, used very frequently. A simple test can be performed in order to determine if data is linear: if independent variable

values are equally spaced, simply check if difference of consecutive y-values is the same. If b is sufficiently small, y is said to

be proportional to x i.e. (y x)

ii. Quadratic: y = ax2 + bx + c: Appropriate for fitting data with one minimum or one maximum. To find out if equally spaced data is quadratic, check if the

differences of the successive differences of consecutive y-values are constant. If a > 0, this function is concave up; if a < 0, it is

concave down.

iii. Cubic: y = ax3 + bx2 + cx + d: Appropriate for fitting data with one minimum and one maximum.

iv. Quartic: y = ax4 + bx3 + cx2 + dx + e: Convenient for fitting data with two minima and one maximum or two maxima and one minimum. When working

with polynomial models a thing to keep in mind: balance between complexity and precision. For examples, see section Testing

the effectiveness of a model. Also, monitor the long term behavior and check how realistic it is. For example, see sections

Testing the validity and Choosing the right model.

v. Exponential: y = abx or y = aekx: While a linear function has constant average rate of change (a constant difference of two consecutive y-values), an

exponential function has constant percent (relative) rate of change (a constant quotient of two consecutive y-values). Thus, an

easy test to check if data is linear: if independent variable values are equally spaced, simply check if quotient of consecutive y-values is the same. If k > 0, then the function is increasing and concave up. If k < 0, then the function is decreasing and

concave up. This model is appropriate if the increase is slow at first but then it speeds up (or, if k < 0 if the decrease is fast at

first but then slows down).

vi. Logarithmic: y = a + b ln x: If b > 0, then the function is increasing and concave down. If b < 0, then the function is decreasing and concave up. If the data

indicates an increase, this model is appropriate if the increase is fast at first but then it slows down.

vii. Logistic: y = c/(1+ae-bx) : Increasing for b > 0: In this case, the increase is slow at first, then it speeds up and then it slows down again and approaches the y-

value c for x .


17/23


-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


viii. Power y = ax

b: If a > 0, y is increasing for b > 0 and decreasing for b < 0. It is called a power model since an increase of x by factor of t causes an increase of

y by the power tb of t (for b > 0). Increasing power function will not increase as rapidly as an increasing exponential function.

Connection with linear model: If y = axb, then ln y = ln a + b ln x. So, if y is a power

function, ln y is a linear function of ln x. Note: If y = ab x, then ln y = ln a + x ln b. So, if y is

an exponential function, ln y is a linear function of x.

In conclusion:

ix. Sine y = a sin(bx+c)+d: Appropriate for periodic data. In the formula, a is the amplitude, b/360 the period, c/b the horizontal shift, and d the vertical shift.

Modeling with piecewise defined functions

A piecewise defined function is a function defined by different formulas for different values of the independent variable. Sometimes, it is better to model a given

data using one piecewise defined function than with a single function.

Analytic Methods of Model fitting

In this section, we look into the mathematics behind obtaining regression models and curve fitting.

Least-Squares Criterion

This criterion is the most frequently used curve- fitting criterion. It is based on an idea that in order to fit a set of data points (xi , yi), i = 1, …, m onto a curve y =

f(x), we again want the differences between yi and f(xi) to be as small as possible.

Recall that the square of distance of the vector between the observed value y i and the predicted value f(xi) is given by

Thus, a way to make the differences yi - f(xi) small is to minimize the sum

Least-Squares Line Fit.

Suppose that we would like to fit the points (xi, yi); i = 1, . . ., m onto a line y = ax+b. So, we need to minimize the sum

This will be satisfied when and

We can view the equations above as two linear equations in the unknowns a and b. Solving them for a and b, we obtain


18/23


-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


The two equations above are called the normalizing equations. It is not hard to write a computer code that will compute these values.

Least-Squares Power Fit

Suppose that a positive integer n is fixed and that we would like to fit the points (x i, yi), i = 1, . . ., m onto a curve of the form y = axn. So, we need to minimize

the sum

This will be satisfied when

Solving this equation for a, we obtain

Transformed Least-Squares Fit.

The formula computing the coefficients of the linear and power least-squares fits is not too complex because the partial derivatives considered were linear

functions of the unknown coefficients.

With exponential, logarithmic or power model (with n not fixed in y = axn), the equations determined by the partial derivatives are not linear equations in

unknown coefficients. In cases like this, one may modify the data (considering ln of one or both of the variables instead of the original data) in order to reduce a

non-linear model to a linear one.

Thus

i) Finding an exponential model y = be ax using the transformed data

Taking ln of y = beax, obtain that ln y = ax + ln b = ax + B for ln b = B. Thus, you can find a linear model for (xi, ln yi) instead of (xi, yi), i = 1, …, m. Note that

in this case

ii) Finding a logarithmic model y = a ln x + b using the transformed data

Simply consider (xi, yi) = (ln xi, yi) instead of (xi, yi); i = 1, . . ., m and find the linear model for the transformed set of data. The linear model for y = aX + b is the

logarithmic model that we are looking for. Note that in this case

iii) Finding a power model y = bx a using the transformed data.

Taking ln of y = bxa, gives you ln y = a ln x + ln b. Consider (ln xi, ln yi) instead of (xi, yi), i = 1, …, m and find the linear model for this transformed set of data.

Denote ln b = B. The linear model for Y = ln y = a ln x + ln b = aX + B can be found as:


19/23


-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


An important fact to note is that if the partial derivative equations are used to find the unknown coefficients a and B in y = Beax, y = a ln x + b or y = Bxa without

transforming the original data, we would end up with different models than the models that we get when we transform the data and find a linear regression for the

new data. It is important to keep in mind how different software applications handle the data.

Transforming the original data often simplifies the formulas even for models with less parameters than exponential, power or logarithmic. The next example

illustrates this.

Example 1: Find the formula computing the coefficient a for the quadratic fit y = ax2 and use it to find a transformed quadratic fit for the data below:

Take ln of y = ax2. Obtain then ln y = ln a + 2 ln x. Let us denote ln a by A. Thus, we need to consider ln y = A + 2 ln x and to minimize

The partial derivative is

Setting this derivative to 0 and solving for A gives us

which can be easily solved for a.

Using the data from Example 1, we obtain that A = 1.1432 and so a = 3.1368. Compare this model y = 3.1368x2 with the model y = 3.1869x2 obtained using the

direct power curve fit.

Measuring the validity of a model

Let us first see the Chebyshev Approximation Criterion.

Let us assume that m points (xi, yi), i = 1, …, m are given and that we need to fit them on a curve y = f(x). As we would like the differences between yi and f(xi)

to be as small as possible, the idea is to minimize the largest absolute value |yi - f(xi)|.

Thus, the only difference between the least square fit and Chebyshev’s method is that instead of the sum of squares of yi - f(xi), we are minimizing the maximum

of the absolute values of yi - f(xi).

Following steps achieve this goal.

1. Let ri = yi - f(xi) for i = 1, …, m. The variables ri are called residuals.

2. Find the maximum of |ri| for i = 1, …,m. Let us call this quantity r.

3. Solve the following optimization problem: minimize r subject to constraints -r ri r for i = 1, …, m i.e.

r - ri 0, r + ri 0 for i = 1, ..., m.

Example 2. Suppose that a point B is somewhere inside a line segment AC. Assume that measuring the distances AB, BC and AC, we obtain that AB = 13, BC =

7 and AC = 19. As 13 + 7 = 20 19, we need to resolve the discrepancy.

We can do that using the Chebyshev approximation criterion. Let x1, x2 and x3 stand for exact lengths of AB, BC and AC respectively. Then r1 = x1 – 13, r2 = x2 –

7, and r3 = x1 + x2 – 19.

Let r stands for the maximum of the absolute values of r1, r2 and r3 (note that we don't know which one is r). We would like to minimize r subject to

r - x1 + 13 0, r + x1 - 13 0, r - x2 + 7 0, r + x2 - 7 0


20/23


-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


r - x1 - x2 + 19 0, r + x1 + x2 - 19 0

Using the methods of linear programming, we obtain that r = 1/3, x1 = 12 &2/3 and x2 = 6 & 2/3.

Although in this example linear programming is used, note that in some cases the constraints will not be linear equations and other optimization methods should

be used.

Now let us come back to Measuring the Validity of a Model

In the previously seen examples, two different quadratic power models y = 3.1368x2 and y = 3.1869x2 are obtained for the same set of data. If Chebyshev

criterion is to be used for the same set of data, it would yield yet another model y = 3.17073x2 for the same set of data.

A natural question is: how can we choose the best model?

Before we attempt to answer, let us introduce some notation. If the data (x i, yi), i = 1, . . ., m is to be fit on the curve y = f 1(x) obtained using the Chebyshev

criterion, let ci denote the absolute deviations | yi - f(xi)|. If the same data is to be fit on the curve y = f 2(x) obtained using the least-squares criterion, let d i denote

the absolute deviations |yi - f 2(xi)|. Also, let cmax denote the largest of ci and dmax denote the largest of d i. Since the least-squares criterion is such that the squares

of the deviations are minimal, we have that

Thus,

Also, since the Chebyshev criterion is such that the maximum of the absolute deviations is minimal, we have that cmax dmax. Thus,

D cmax dmax

This last equation can help us determine which model to use: the least-squares criterion is convenient to use but there is always to concern that the difference

between D and dmax might be too large. Balancing these two conditions might help us decide which model to use.

Let us go back to the example with three different quadratic power models. Computing the deviations for all three models we conclude

1. For the model y = 3.1869x2 obtained using the least-squares fit. The largest absolute deviation is 0.347. The smallest is 0.0976.

2. For the model y = 3.1368x2 obtained using transformed least-squares fit. The largest absolute deviation is 0.495. The smallest is 0.0842. This model was the

easiest to compute but has the largest absolute deviation.

3. For the model y = 3.17073x2 obtained using Chebyshev criterion. The largest absolute deviation is 0.282. The smallest is 0.0659. Computationally, this model

was the hardest to get but has the smallest absolute deviation.

However, considering the sum of squares of deviations instead of the maximum of the absolute deviations, the least-squares fit model is better than the

Chebyshev (.2095 for the least squares versus .2256 for Chebyshev).

The conclusion that we can draw from this example is that there is not a single right answer when trying to decide which model is the best. Thus, in each of the

following cases, our choice of models would be different.

1. If it is more important to minimize the sum of squares of deviations than the maximal absolute deviation, the least-squares criterion should be used.

2. If it is more important that the model is computationally simple than to have small maximum or sum of squares of absolute deviations, the

transformed least-squares criterion should be used.

3. If it is more important to minimize the maximal absolute deviation than the sum of squares of deviation, the Chebyshev criterion should be used.

So, we need to decide which model is the best on case-by-case basis, taking all the specifics into account (what is the purpose of the model, how precise should it

be, how accurate is the data, etc).

Linear Regression

Note: regression is a measure of the relation between the mean value of one variable and corresponding values of other variables.)


21/23


22/23


-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Validation: is concerned with building the right model . It is utilized to determine that a model is an accurate representation of the real world system. Validation

is usually achieved through the calibration of the model, an iterative process of comparing the model to actual system behavior and using the discrepancies

between the two, and the insights gained, to improve the model. This process is repeated until model accuracy is judged to be acceptable.

The Model Validation “Gold Standard”

In Physics, The Standard Model is the name given to the current theory of fundamental particles and their interactions

The Standard Model is a good theory because it has been validated

Its predictions have matched experimental data, decimal place

for decimal place, with amazing precision

All the particles predicted by this theory have been found

Can such a theoretical model be found for social systems?

Important Questions about Model Validation

How can the model be validated if…

Controlled experiments cannot be performed on the system, for example, if only a single historical

data set exists?

The real-world system being modeled does not exist?

The model is not deterministic (has random elements)?

How can agent-based models be validated?

Agent behaviors and interaction mechanisms

Adaptive agent behaviors of emergent organizations

Model Validation:

Does the model represent and correctly reproduce the behaviors of the real world system?

Validation ensures that the model meets its intended requirements in terms of the methods employed and the results obtained

The ultimate goal of model validation is to make the model useful in the sense that the model addresses the right problem, provides accurate

information about the system being modeled, and to makes the model actually used

Validation Depends on the Purpose of the Model and Its Intended Use

Reasons we do modeling and simulation:

We are constrained by linear thinking: We cannot understand how all the various parts of the system interact and add up to the whole

We cannot imagine all the possibilities that the real system could exhibit

We cannot foresee the full effects of cascading events with our limited mental models

We cannot foresee novel events that our mental models cannot even imagine

We model for insights, not numbers

As an exercise in “thought space” to gain insights into key variables and their causes and effects

To construct reasonable arguments as to why events can or cannot occur based on the model

We model to make qualitative or quantitative predictions about the future

Practical Validation

Validation exercises amount to a series of attempts to invalidate a model

One recently proposed V&V technique, Active Nonlinear Tests (ANTs), explicitly formulates a series of mathematical tests designed to

“break the model”

Presumably, once a model is shown to be invalid, the model is salvageable with further work and results in a

model having a higher degree of credibility and confidence

The end result of validation

Technically not a validated model, but rather a model that has passed all the validation tests

A better understanding of the model’s capabilities, limitations, and appropriateness for addressing a range of important questions

Establishing Credibility

Unlike physical systems, for which there are well established procedures for model validation, no such guidelines exist for social modeling

In the case of models that contain elements of human decision making, validation becomes a matter of establishing credibility in the model

Verification and validation work together by removing barriers and objections to model use

The task is to establish an argument that the model produces sound insights and sound data based on a wide range of tests and criteria that “stand in”

for comparing model results to data from the real system

The process is akin to developing a legal case in which a preponderance of evidence is compiled about why the model is a valid one for its purported use

Pathways to Validation

Cases


23/23


---------------------------------------------------------------------------------------------

Ch 2 Systems and Modeling

Documents

Transcript of Ch 2 Systems and Modeling