Good Experimental Methodologies and Optimal Design … · Good Experimental Methodologies and...

Universita degli Studi di Genova Consiglio Nazionale delle Ricerche

Good Experimental Methodologiesand Optimal Design Of Experiments

in Marine Robotics

Eleonora Saggini

Dottorato in Matematica e ApplicazioniXXVIII ciclo

Eleonora Saggini DoE in marine robotics 4th November 2015 1 / 31

Autonomous Marine Robots

Figure: Marius AUVFigure: Charlie USV

Figure: Autosub AUV

Long term objective: Integration of robots within civilian scenarios in everyday lifeIssues: Lack of regulation laws

Good Experimental Methodologies in order to

reproduce experiments

assess robot performances

compare robot behaviours


Operational objective

Develope a software tool and a protocol for performance assessment of autonomousmarine robots. Need of

definition of replicable experiments

automatic selection and execution of replicable experiments

online or postprocessing performance evaluation

Issues for experiments

achievement of repeatability including environmental conditions and replicate initialconditions (difficulties to drive a UMV in the pre-defined, starting position andspeed)

time requirements, high cost of data acquisition and data sharing

where and when to evaluate performance (line following task and maneouvringphases: turn, path-approach, transient or overshoot, steady-state)


Repeatability VS Reproducibility

Repeatability concerns the fact that a single result is not sufficient to ensure thesuccess of an experiment

Reproducibility (or replicability) is the possibility to verify, in an independent way,the results of a given experiment.

chemistry, physics,...: to reproduce the same results of a previous experimentmarine robotics: the term replicability is applied to all controllable parameters of theexperiments and not to its results.The obtained results can be easily compared as well as the performance of two or moreplatforms (dependence on non controllable conditions e.g. environmental)


Online Steady State Detection for the heading

When is the heading constant over time?

760 765 770 775 780 785 790 795

809.2

809.4

809.6

809.8

810

810.2

810.4

810.6

810.8

811

t [sec]

ψ[d

eg]

390 395 400 405 410 415 420 425

1

1.2

1.4

1.6

1.8

2

2.2

2.4

2.6

t [sec]

ψ[d

eg]

Method 0: check on the range of data;

Sample correlation coefficient at lag k: ρk =

∑n−ki=1

(Xi − X

) (Xi+k − X

)∑ni=1

(Xi − X

)2

Method I: check on the first ρks one at a time;

Method II: check on the first ρks all together;

Method III: SCARM (Slope Comparing Adaptive Repeated Medians) filter.


Towards Good Experimental Methodologies for Unmanned Marine Vehicles[Caccia et al, 2013]

630 635 640 645 650 655 660 665 670 675

90

95

t1[s]

ψ[d

eg]

Charlie1USV1auto−heading1−1Experiment11:1steady−state1detection

400 405 410 415 420 425 430 435 440 445−2

0246

t*

Charlie1USV1auto−heading1−1Experiment12:1steady−state1detection

t1[s]

ψ[d

eg]

400 405 410 415 420 425 430 435 440 445

00.20.40.60.8

α =10.05

t*

p−va

lues

t1[s]

Charlie1USV1auto−heading1−1Experiment12:1p−values

Test {Xt}t≥0 ∼WN(µ, σ2)in moving time windows [tS , tE ]

H0 : ρ(τ) = 0 for all τ 6= 0H1 : ρ(τ) 6= 0 for some τ 6= 0

ρ(τ) is the theoteticalautocorrelation at lag k

Ljung-Box test statistics Q = N(N + 2)

Kmax∑τ=1

ρ(τ)2

N − τ

Under H0, Q is asimptotically χ2 distributed with Kmax degrees of freedom.


Path following task: need of performance indices

Reference path: R = {(xR,i , yR,i ), i = 1, . . . ,m}

- and f known: f (xR,i , yR,i ) = 0 (polynomial, trigonometric,..)

- and f unknown (e.g. vehicle following)

Vehicle path: V = {(xV ,i , yV ,i ), i = 1, . . . , n}

online computation

accurate computation

general situation- n 6= m- or f is unknown

information that is complementaryin some sense(e.g. accuracy vs consumption)

compound index

DH

DH


Performance indices

Geometric criteria

Area index

DA =area between R and V

total length of R

Hausdorff distanceDH = max{dH(V,R), dH(R,V)},

where dH(V,R) is the directed Hausdorff distance from V to R, defined as

dH(V,R) = maxv∈V{minr∈R

d(v , r)}

and d is the Euclidean distance.→ Maximum of all the distances from a point to in one set to the closest point in theother set


Performance indices

Geometric criteria

Sampson distance

DSm = meanv∈V|f (v)|√

fx(v)2 + fy (v)2DSM = maxv∈V

|f (v)|√fx(v)2 + fy (v)2

→ Distance to the approximating tangent on the curve

Lagrange distance

DLm = meanv∈V minP:f (P)=0 ||v − P||2 DLM = maxv∈V minP:f (P)=0 ||v − P||2

Percentage index of points crossing R

PC = 100Number of points close to f (x , y) = 0

n

→ 2-step procedure

...


The crossing criterion [with Laura Torrente]

1 computation of an algebraic curve f = 0 that approximates the points in R within atolerance ε1 > 0.Low Polynomial Degree (LPA) algorithm

its total degree has to be bounded by the smallest total degree amongst the totaldegrees of all polynomials vanishing at all points in Rthe zero-locus of f lies close to the points of R by less than the tolerance ε1 (thedistance is induced by the given norm)

2 identification of the points in V far from the reference path f = 0 for more than atolerance ε2 > 0.Crossing Cell algorithm (CA)For each p ∈ V

|f (p)| > B1(f , p, ε2) f = 0 does not cross Bε2 (p) → red|f (p)| < B2(f , p, ε2,R) → crosses Bε2 (p) → blackthe crossing problem remains undecided → green

If ε2 < 1: approximated bounds B ′1,B′2 (ACA)


Performance indices: the crossing criterion

x[m]

y[m]

−50 0 50−50

−40

−30

−20

−10

0

10

20

30

40

50

x[m]

y[m]

−10 0 10 20 30−20

−15

−10

−5

0

5

10

15

20

AlgorithmIndex ACA-2 CA-2 ACA−∞PC 50% 63% 13%PNC 29% 29% 33%PU 21% 8% 54%


Performance indices: the crossing criterion

x[m]

y[m]

−50 0 50−50

−40

−30

−20

−10

0

10

20

30

40

50

x[m]

y[m]

10 20 30 40 50−30

−25

−20

−15

−10

−5

0

5

10

AlgorithmIndex ACA-2 CA-2 ACA−∞PC 43% 50% 13%PNC 37% 37% 44%PU 20% 13% 43%


Performance indices

Other criteria

Rudder stress

R =

∑n−1i=1 |∆δi |n − 1

δi is the rudder angle∆δi = δi+1 − δiThrusters’ energy consumption

E =

∑n−1i=1 fu,i∆si

n − 1

fu,i is the commanded thrust force∆si =

√(xV ,i+1 − xV ,i )2 + (yV ,i+1 − yV ,i )2

fuel/battery consumption

time

cross-track error

...


Path following experiment: lack of replicability

−90 −80 −70 −60 −50 −40 −30 −20 −10 0−80

−70

−60

−50

−40

−30

−20

−10

y[m]

x[m]

−90 −80 −70 −60 −50 −40 −30 −20 −10 0−80

−70

−60

−50

−40

−30

−20

−10

0

y[m]

x[m]


Line following experiment: phases and metrics [Saggini et al, 2014]

d

TURN PATHAPPROACH

SETTLING STATE

STEADY STATE

d

H1⊥,H1‖,H2: Hausdorff distances

A1,A2,A∗3 : areas between the two paths

Rudder stress

Cross-track error decreasing rate


Line following experiment: phases and metrics

PATHAPPROACH

TURN SETTLING STATE

STEADY STATE

Lyapunov-based virtual target (LBVT) vs Jacobian-based priority task (JBPT)control architectures

JBPT: select its parameters (Kθ,Ku) ∈ {(0.2, 0.1); (0.4, 0.2); (0.6, 0.3)}comparison between LBVT and JBPT


Line following experiment: lawn mower grids [Saggini et al, 2015]

−160 −140 −120 −100 −80 −60−40

−30

−20

−10

0

10

20

30

40

50

60

y[m]

x[m]

Algorithm DA[m] DH [m] R[deg ] E [J]

MeanLBVT 0.42 1.16 0.51 1.30JBPT 0.17 0.89 0.80 0.60

Std devLBVT 0.31 0.62 0.17 1.13JBPT 0.05 0.05 0.02 0.04


Experiments

TURN

TURN

BACKWARD PATH

INITIALPOSITION

FORWARD PATH

TURN

TURN

BACKWARD PATH

FORWARD PATH

TURN

BACKWARD PATH

FORWARD PATH

run 1 run 2 run n

.........

.........

An experiment is definedby n runs, correspondingto n different paths thatare executed sequentiallyin time

Runs are independentideally

The path is followed backand forth (at least onerepetition)

Repeatability is achievedthrough a suitable turningmanoeuvre

Performance is notmeasured during the turnphases


DeepRuler: Design Execute and Evaluate Path-following Experiments forRobotic Unmanned vehicLes and Repeatability [Sorbara et al, 2015]

A software for both for running experiments on a simulator or during a field trial to

guide the user in the definition of an experiment

conduct and supervise the execution of the experiment

Achieve these two aims

with a step-by-step windowed configurator: receives in input parameters that specifyan experiment (working area, specific path, choice of telemetry and of metrics...)and produces a file reusable to repeat the experiment

by sequencing the experimental phases, collecting the telemetry, computing themetric


DeepRuler

Main features of DeepRuler are:

open source

modular: for path generation, telemetry collection, metrics computation

portable1 written in C++ using well-known library. At the moment compiled and tested under

Linux amd64/i386 and Windows 64/32 bit2 at the moment two communication to/from robot: ROS and boost::serialization


Measurement process for the simulator of Charlie USV and sinusoidal paths

Design space: θ = [θ1, θ2]T ∈ Θ, where Θ = R>0 × Z≥1 ∪ {(0, 0)} identifies a targetpath γ(θ)

For each path θ ∈ Θ an observed path z(θ) is given as a time series

zt(θ) =

[xt(θ)yt(θ)

]=

[x t(θ)y t(θ)

]+ εt ∈ R2, εt ∼ Uniform

where [x t(θ), y t(θ)]T are GPS measurements and t = 1, . . . ,T

Performance measures η1(θ), η2(θ), . . . are associated to {zt(θ)}tFor i = 1, 2, . . . standard assumptions

ηi (θ) = fi (θ) + εi (θ)

where fi (θ) = β i0 + β i

1θ1 + β i2θ2 + β i

12θ1θ2 and εi ∼ GP(0, σ2R(·, ·)) with σ2 a globalvariance and R(θ, θ′) correlation between the θ and θ′ paths


Simulation study

1st and 2nd batch parametersPath: L = 100 mTurn manoeuvre: settling line length = 25 m; turning radius = 14 mPosition noise: 0.2 mParameter space: θ ∈ {(0, 0)} ∪ {5, 10, 15, 20, 25} × {1, 2, 3, 4, 5, 6}

0 5 10 15 20 250

1

2

3

4

5

6

θ1

θ 2

DH,1 and DA,1

0 5 10 15 20 250

1

2

3

4

5

6

θ1

θ 2

DH,2 and DA,2


1st batch: paired sample t-test, back and forth

DFH,1 and DB

H,1 → p = 0.8024; DFA,1 and DB

A,1 → p = 0.4466

5 10 15 20 25 30

0.6

0.5

0.4

0.3

0.2

0.1

0

0.1

0.2

0.3

batch 1: Hausodrff

5 10 15 20 25 30

0.14

0.12

0.1

0.08

0.06

0.04

0.02

0

0.02

0.04

batch 1: Area

2nd batch: paired sample t-test, back and forth

DFH,2 and DB

H,2 → p = 0.2378; DFA,2 and DB

A,2 → p = 0.2559

5 10 15 20 25 30

0

2

4

6

8

10

12

batch 2: Hausdorff

5 10 15 20 25 30

0

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8batch 2: Area


Simulation study: same batch, different order

Paired samples t-test for the 2 sets of data

DH,1 and DH,2 → p = 0.4652;

DA,1 and DA,2 → p = 0.7879

0 5 10 15 20 25 30 35−6

−5

−4

−3

−2

−1

0

1batch 1 e 2: Hausdorff

0 5 10 15 20 25 30 35−0.8

−0.7

−0.6

−0.5

−0.4

−0.3

−0.2

−0.1

0

0.1batch 1 e 2: Area


Design of Experiments: adaptive procedure

Worst performance

1 Space filling designLHSone generator lattice...

2 Performance index computation

3 Optimal local designfor maximizing the performance

4 Kriging re-estimation

Best prediction

1 Space filling designLHSone generator lattice...

2 Performance index computation

3 Kriging estimation

4 Optimal local designfor minimizing the error

MSE of the predictorw.r.t. values computed from a fullfactorial design(simulation/benchmark)

5 Kriging re-estimation


Definition of path following “complexity”?

Sinusoidal path-followingy = θ1 sin(πθ2

Lx)

↓

Maximum curvature inxMC = −θ2/4− L/2

10 20 30 40 501

2

3

4

5

6

7

8

9

10

11

θ1

(amplitude)

θ 2(h

alf−

perio

ds)

Absolutecvaluecofcthecmaximumccurvature

c 0

1

2

3

4

5

6

0 10 20 30 40 500

1

2

3

4

5

6

7

8

9

10

θ1

(amplitude)

θ 2(h

alf−

perio

ds)

Area distance

0

1

2

3

4

5

6


Latin hypercube sampling + Central composite

Worst performance

0 10 20 30 40 500

1

2

3

4

5

6

7

8

9

10

θ1

(amplitude)

θ 2(h

alf−

perio

ds)

LHSC+CCCC(maxCarea)

Best prediction

0 10 20 30 40 500

1

2

3

4

5

6

7

8

9

10

θ1

(amplitude)

θ 2(h

alf−

perio

ds)

LHSC+CCCC(maxCmse)


Kriging prediction

Worst performance Best prediction


Papers

E Saggini, E Zereik, M Bibuli, A Ranieri, G Bruzzone, M Caccia, E Riccomagno,Evaluation and Comparison of Navigation Guidance and Control Systems for2D/Surface Path-Following, Annual Reviews in Control, available online athttp://dx.doi.org/10.1016/j.arcontrol.2015.08.006.

A Sorbara, A Ranieri, E Saggini, E Zereik, M Bibuli, G Bruzzone, M Caccia, Testingthe Waters: Design of Replicable Experiments for Performance Assessment of MarineRobotic Platforms, IEEE Robotics and Automation Magazine 22(3), 2015, 62–71.

E Saggini, L Torrente, M Bibuli, G Bruzzone, M Caccia, E Zereik Assessingpath-following performance for Unmanned Marine Vehicles with algorithms fromNumerical Commutative Algebra, Proceedings of the 22nd Mediterranean Conferenceon Control and Automation (MED’14), June 16–19, Palermo, Italy, 2014.

E Saggini, E Zereik, M Bibuli, G Bruzzone, M Caccia, Performance Indices forEvaluation and Comparison of Unmanned Marine Vehicles’ Guidance Systems,Proceedings of the 19th IFAC World Congress, (IFAC WC 2014), August 24–29,Cape Town, Africa, 2014.


Papers

M Bibuli, G Bruzzone, M Caccia, E Fumagalli, E Saggini, E Zereik, UnmannedSurface Vehicles for Automatic Bathymetry Mapping and Shores’ Maintenance,Proceedings of the MTS/IEEE OCEANS’14 Conference, Taipei, Taiwan, 2014.

M Caccia, E Saggini, M Bibuli, G Bruzzone, E Zereik, E Riccomagno, Towards GoodExperimental Methodologies for Unmanned Marine Vehicles, EUROCAST, Part II,LNCS 8112, Springer, Heidelberg, 2013, 365–372.

In preparation

E Saggini, M L Torrente, A new crossing criterion to assess path-followingperformance for Unmanned Marine Vehicles, Journal of Algebraic Statistics, (to besubmitted in November)

E Saggini, E Riccomagno, Adaptive experimental design for performance assessmentof marine robotic platforms, Computational Statistics & Data Analysis, Special Issueon Design Of Experiments, 2016


Good Experimental Methodologies and Optimal Design … · Good Experimental Methodologies and...

Documents

Transcript of Good Experimental Methodologies and Optimal Design … · Good Experimental Methodologies and...