WP 4.1 Report of the Retrospective Working Group

It is a simple but sometimes forgotten truth that the greatest enemy to present joy and high hopes is the cultivation of retrospective bitterness.

Robert G. Menzies (Australian Prime Minister 1930-1941&1949-1966)

Retro WG Participants

Special thanks to ICES WGMG

Name Affiliation Name Affiliation Name Affiliation Chris Legault (chair) NEFSC Larry Alade NEFSC Paul Nitschke NEFSC Bob Mohn DFO Andy Applegate NEFMC Loretta Obrien NEFSC Larry Jacobson NEFSC Jessica Blaylock NEFSC Bill Overholtz NEFSC Alan Seaver NEFSC Liz Brooks NEFSC Michael Palmer NEFSC

Steve Cadrin NEFSC Paul Rago NEFSC Laurel Col NEFSC Anne Richards NEFSC Steve Corriea MADMF Fred Serchuk NEFSC Deborah Hart NEFSC Gary Shepherd NEFSC Lisa Hendrickson NEFSC Kathy Sosebee NEFSC Joe Idoine NEFSC Jiashen Tang NEFSC Chad Keith NEFSC Mark Terceiro NEFSC Jennifer Martin NEFSC Michele Traver NEFSC Ralph Mayo NEFSC Jim Weinberg NEFSC Tim Miller NEFSC Susan Wigley NEFSC Joshua Moser NEFSC

Retro

• What is it?• How is it measured?• Why is it important?• What causes it?• Can we fix it?

Retro Defined

• The retrospective problem is a systematic inconsistency among a series of estimates of population size, or related assessment variables, based on increasing periods of data. (Mohn 1999)

• Historical vs Within Model• Not just VPA

0

10

20

30

40

50

1970 1975 1980 1985 1990 1995 2000 2005

SSB

(tho

usan

d m

t)

Don’t Always Know It When You See It

Retrospective pattern in GB haddock SSB trendin tons

year

SS

B

50000

100000

150000

1970 1980 1990 2000

Retrospective pattern in GB haddock SSB as percent difference from terminal year

year

per

cent

diff

eren

ce fr

om te

rmin

al y

r

-20

-15

-10

-5

0

1970 1980 1990 2000

A Strong Retrospective

Terminal Year - 2005 Terminal Year - 2004 Terminal Year - 2003

Terminal Year - 2002 Terminal Year - 2001 Terminal Year - 2000

0.00

40.00

80.00

120.00

160.00

200.00

240.00

280.00

320.00

360.00

400.00

440.00

480.00

520.00

560.00

600.00

Freq

uenc

y

0.200000 0.300000 0.400000 0.500000 0.600000 0.700000 0.800000 0.900000Fishing Mortality

Average Fishing Mortality2000

0.0

0.5

1.0

1.5

2.0

1970 1975 1980 1985 1990 1995 2000 2005

F(4+

)

Mohn rho

0 10 20

0.400

0.450

0.500

0.550

F 3-5

25 years in assessment7 terms in rho calculation

Woods Hole rho

0 10 20

0.400

0.450

0.500

0.550

F 3-5

25 years in assessment300 (=24+23+22+...+1) terms in rho calculation

rho Increases with Bias

0 0.5 1.0

0

0.5

1

rho B

Disturbance

Why are retros important?

• Projections too high• Catch advice high• Causes F > Fref• Feedback loop

0

5

10

15

20

25

30

2000 2001 2002 2003 2004 2005 2006 20070

5

10

15

20

25

30

2000 2001 2002 2003 2004 2005 2006 20070

5

10

15

20

25

30

2000 2001 2002 2003 2004 2005 2006 2007

Causes of Retros

• Change in Catch (missing or extra)• Change in M• Change in Survey q

• Closed Areas (sessile)

Missing CatchD1006

0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

7,000,000

1960 1970 1980 1990 2000 2010 2020

SSB

D1006

0

0.2

0.4

0.6

0.8

1960 1970 1980 1990 2000 2010 2020

F

D0610

0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

7,000,000

1960 1970 1980 1990 2000 2010 2020

SSB

D0610

0

0.2

0.4

0.6

0.8

1960 1970 1980 1990 2000 2010 2020

F

Change in MM0204vpa02

0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

7,000,000

1960 1970 1980 1990 2000 2010 2020

SSB

M0204vpa04

02,000,0004,000,0006,000,0008,000,000

10,000,00012,000,00014,000,00016,000,000

1960 1970 1980 1990 2000 2010 2020

SSB

M0402vpa02

0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

7,000,000

1960 1970 1980 1990 2000 2010 2020

SSB

M0402vpa04

0

2,000,000

4,000,000

6,000,000

8,000,000

10,000,000

12,000,000

1960 1970 1980 1990 2000 2010 2020

SSB

Change in Survey qq12

0

2000000

4000000

6000000

8000000

10000000

12000000

14000000

1960 1970 1980 1990 2000 2010 2020

SSB

q12

0

0.2

0.4

0.6

0.8

1

1960 1970 1980 1990 2000 2010 2020

F

q21

0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

7,000,000

1960 1970 1980 1990 2000 2010 2020

SSB

q21

0

0.2

0.4

0.6

0.8

1

1960 1970 1980 1990 2000 2010 2020

F

Closed Area

0 10 20 300

250

500

750

Biom 5+

Year

Retro by Chance Unlikely

1998 1999

2000 2001

2002 2003

2004 2005

True Average F

0.0000

0.3001

0.6002

0.9003

1.2004

1.5005

1.8006

Fis

hing

Mor

talit

y

1983 1986 1989 1992 1995 1998 2001 2004Year

Fishing Mortality RetrospectiveVPA Iteration # 65

Can a Retro be Fixed?

• Yes– Can remove retro pattern in many ways

• No– Do not necessarily get closer to the truth

• Need to known timing and source to get it right– Timing can sometimes be identified– Source cannot be identified

Local Influence Surface

• Cadigan and Farrell (2002, 2005)• Use rho – no good• Use other metric, can sometimes identify

timing

10 20 301

2

3

4

5

6 NPeel 10 F3-5

Year

Age -2.7

-1.3

0.16

1.6

3.1

LIS with rho f(npeels)

Source of Retro 1990

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

1980 1985 1990 1995 2000 2005

Sum

of G

radi

ent o

ver A

ges

npeel4npeel8npeel12npeel16

Source of Retro 2000

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

1980 1985 1990 1995 2000 2005

Sum

of G

radi

ent o

ver A

ges

npeel4npeel8npeel12npeel16

Split Surveys

• GB yt “solution” to retro• Removes retro, but not necessarily to truth

Change Catch (seen previously)D1006

0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

7,000,000

1960 1970 1980 1990 2000 2010 2020

SSB

D1006

0

0.2

0.4

0.6

0.8

1960 1970 1980 1990 2000 2010 2020

F

D0610

0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

7,000,000

1960 1970 1980 1990 2000 2010 2020

SSB

D0610

0

0.2

0.4

0.6

0.8

1960 1970 1980 1990 2000 2010 2020

F

Change Catch Split SurveysD1006

0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

7,000,000

1960 1970 1980 1990 2000 2010 2020

SSB

D1006

0

0.2

0.4

0.6

0.8

1960 1970 1980 1990 2000 2010 2020

F

D0610

0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

7,000,000

1960 1970 1980 1990 2000 2010 2020

SSB

D0610

0

0.2

0.4

0.6

0.8

1960 1970 1980 1990 2000 2010 2020

F

M Change (seen previously)M0204vpa02

0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

7,000,000

1960 1970 1980 1990 2000 2010 2020

SSB

M0204vpa04

02,000,0004,000,0006,000,0008,000,000

10,000,00012,000,00014,000,00016,000,000

1960 1970 1980 1990 2000 2010 2020

SSB

M0402vpa02

0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

7,000,000

1960 1970 1980 1990 2000 2010 2020

SSB

M0402vpa04

0

2,000,000

4,000,000

6,000,000

8,000,000

10,000,000

12,000,000

1960 1970 1980 1990 2000 2010 2020

SSB

M Change Split SurveysM0204vpa02

0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

7,000,000

1960 1970 1980 1990 2000 2010 2020

SSB

M0204vpa04

02,000,0004,000,0006,000,0008,000,000

10,000,00012,000,00014,000,00016,000,000

1960 1970 1980 1990 2000 2010 2020

SSB

M0402vpa02

0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

7,000,000

1960 1970 1980 1990 2000 2010 2020

SSB

M0402vpa04

0

2,000,000

4,000,000

6,000,000

8,000,000

10,000,000

12,000,000

1960 1970 1980 1990 2000 2010 2020

SSB

Change q (seen previously)q12

0

2000000

4000000

6000000

8000000

10000000

12000000

14000000

1960 1970 1980 1990 2000 2010 2020

SSB

q12

0

0.2

0.4

0.6

0.8

1

1960 1970 1980 1990 2000 2010 2020

F

q21

0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

7,000,000

1960 1970 1980 1990 2000 2010 2020

SSB

q21

0

0.2

0.4

0.6

0.8

1

1960 1970 1980 1990 2000 2010 2020

F

Change q Split Surveysq12

0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

7,000,000

1960 1970 1980 1990 2000 2010 2020

SSB

q12

0

0.2

0.4

0.6

0.8

1

1960 1970 1980 1990 2000 2010 2020

F

q21

0

1,000,000

2,000,000

3,000,000

4,000,000

5,000,000

6,000,000

7,000,000

1960 1970 1980 1990 2000 2010 2020

SSB

q21

0

0.2

0.4

0.6

0.8

1

1960 1970 1980 1990 2000 2010 2020

F

Source: Change q, Fix: Change M q12M0203

0

2000000

4000000

6000000

8000000

10000000

12000000

1960 1970 1980 1990 2000 2010 2020

SSB

q12M0203

0

0.2

0.4

0.6

0.8

1

1960 1970 1980 1990 2000 2010 2020

F

q12M02035

0

2000000

4000000

6000000

8000000

10000000

12000000

1960 1970 1980 1990 2000 2010 2020

SSB

q12M02035

0

0.2

0.4

0.6

0.8

1

1960 1970 1980 1990 2000 2010 2020F

q12M0204

0

2000000

4000000

6000000

8000000

10000000

12000000

14000000

1960 1970 1980 1990 2000 2010 2020

SSB

q12M0204

0

0.2

0.4

0.6

0.8

1

1960 1970 1980 1990 2000 2010 2020

F

Retro Fix Resids

0.00

0.25

0.50

-0.25

-0.50

Res

idua

l

1971 1974 1977 1980 1983 1986 1989 1992 1995 1998 2001 2004 2007 2010Years

ResidualsAGE5 5 0.00

0.25

0.50

-0.25

-0.50

Res

idua

l

1971 1974 1977 1980 1983 1986 1989 1992 1995 1998 2001 2004 2007 2010Years

ResidualsAGE5 5

0.00

0.25

0.50

-0.25

-0.50

Res

idua

l

1971 1974 1977 1980 1983 1986 1989 1992 1995 1998 2001 2004 2007 2010Years

ResidualsAGE5 5

Split Survey Series

Original

Alternative “Fixes”

• Adjust NAA in Projections– Assumes retro will persist– Requires MSE to determine how to adjust

• Provide Alternative States of Nature Advice

Conclusions1. Retrospective pattern is an indication something is inconsistent (data

and/or model).2. Lack of a retrospective pattern does not mean all is well. Based on

simulations, data or model inconsistency does not always produce a retrospective pattern. Retrospective patterning is just one diagnostic to be considered when conducting stock assessments.

3. Simulated retrospective patterns can be caused by time trending changes in biological characteristics, catch, survey catchability, or spatial concentration of the population. Multiple sources may occur in assessments.

4. The source(s) of the retrospective pattern can be anywhere in the time series. Some methods were presented to identify when the change took place (moving window, q surface, mean square residual LIS).

5. The true source(s) of a retrospective pattern have not been identified using current methods. Knowledge of events in the fishery or biological information may help identify probable sources.

Conclusions (cont)6. Interventions (correlated errors) are more likely to cause

retrospective patterns than random noise.7. Splitting surveys, changing M, or changing catch may reduce the

retrospective pattern, but do not necessarily produce an assessment closer to the truth, although the other diagnostics for the new assessment may be fine.

8. The retrospective statistic, rho, may be a useful measure of the amount of retrospective pattern. A strong retrospective pattern can be defined by the degree of overlap between confidence intervals from different terminal years.

9. Local influence surface analysis using rho is not useful for diagnosing the timing or source of retrospective patterns.

10. In many stocks, strong retrospective patterns typically persist.

Recommendations1. Always check for the presence of a retrospective pattern.2. If a model shows a retrospective pattern, then consider alternative models

or model assumptions.3. Develop objective and consistent criteria for the acceptance of

assessments with retrospective patterns. 4. A strong retrospective pattern is grounds to reject the assessment model

as an indication of stock status or the basis for management advice. 5. When a moderate retrospective pattern is encountered: (not an

exhaustive list)a. Consider alternative states of nature approach to advice.b. Investigate the performance of alternative methods for retrospective adjustments through management strategy evaluations.

6. Use biological and fishery hypotheses and auxilliary information as a basis for adjustments for retrospective patterns.

7. Consider use of survey swept area numbers instead of mean catch per tow in assessment models.

8. The presence and implications of a retrospective pattern as a source of uncertainty in the assessment should be clearly communicated to managers.