Understanding the Relative Operating Characteristic (ROC) Simon Mason International Research...

Understanding the Relative Operating Characteristic (ROC)

Simon Mason

International Research Institute for Climate Prediction

The Earth Institute of Columbia University

What?

The WMO has included the relative operating characteristic (ROC) as part of its standardized verification system (SVS).

The incomprehensible name is not meant to be intimidating! The name was inherited from signal detection theory. Most meteorologists refer to it simply as ROC.

The ROC is used to measure how good forecasts are, but it is not a single measure of forecast skill like correlation. Instead it is usually presented as a graph.

Why?

The ROC was selected as part of the WMO SVS for many reasons, including the following:

1. It can be used with deterministic and probabilistic forecasts.

2. It is designed to measure how good forecasts are in the context of a very simple decision-making model, and is thus better suited to measure how good forecasts are from the perspective of the user than are many other commonly used measures.

3. It recognizes that forecast quality cannot be measured by a single number.

When?

The ROC works with binary variables. Binary variables are questions that have “yes”/“no” answers. For example?

Will it rain this afternoon?Will it be hot tomorrow?Will this summer be unusually wet?Will this storm spawn tornadoes?

Of course, the second and third questions have to be defined precisely: “hot” could be defined as “above 30°C”; “unusually wet” could mean “more than 500 mm of rain”.

Forecast says event will happen?

Event happens? Yes No

Yes Hit Miss

No False Alarm Correct Rejection

The orange boxes are correct forecasts, the blue boxes are incorrect forecasts.

As we will see later, the forecast does not have to be expressed as a simple “yes” or “no”, but the answer does have to be a simple “yes” or “no”. If the forecasts are expressed as (or converted to) “yes” or “no”, verification is simple:

How?

Forecast says event will happen?

Event happens? Yes No

Yes Hit Miss

No False Alarm Correct Rejection

The ROC addresses the following two questions:

1. For how many of the events were warnings correctly provided?

2. For how many of the non-events were warnings incorrectly provided?

The best way to illustrate is by means of a game…

You will be shown an arrangement of 120 Mahjong tiles arranged in 8 numbered rows and 15 columns (A-O). You will have one minute to remember the locations of 12 “wind” tiles. They are the only tiles that consist solely of large black Chinese characters (shown opposite). There are 3 of each, but do not worry about distinguishing between the tiles, just try to remember the 12 locations. Try to remember as many as you can. Most people will only remember a few.

For example, in the subset below there are 3 wind tiles; they are in C1, B3, and G3.

1. Very confident (you remember this location clearly);

2. Fairly confident (you think there is a wind tile here, but it may be in a neighbouring location);

3. Slightly confident (you think there may be a wind tile somewhere near here);

4. Just guessing (you are listing locations arbitrarily hoping to get one right by luck).

You must also indicate how confident you are that you have remembered the location correctly. Use the following categories:

If you cannot remember all 12, try listing a cluster of locations for those you are not totally sure about. For example, if you think there is a wind tile somewhere in the middle of row 1 but cannot remember the exact column, list the one you think is correct first, and then list locations either side.

Proceed only when you are ready for the test …

Time Up!

Now write down the 12 locations as you remember them, and indicate your level of confidence (1-4). Use the grid references below.

GRID CONFIDENCE

1 2 3 4 5 6 7 8 9 10 11 12

You should now have a list of 12 locations with indications on how confident you are that the locations are correct.

Once you are happy with your responses, and have indicated your level of confidence appropriately, go ahead and check against the answers on the next slide …

The 12 correct locations are:

A8 C7 D4 E1 E4 H8 I2 K3 L2 L5 L7 M4

Count the number you got right and wrong for each level of confidence, and then calculate the totals.

For example, Dr X’s scores were as follows:

VERY

CONFIDENT

FAIRLY

CONFIDENT

SLIGHTLY

CONFIDENT

JUST

GUESSING

ALL

CORRECT

4

2

1

0

7

INCORRECT

1

1

2

1

5

VERY

CONFIDENT

CORRECT

4 out of 12 = 33.3%

INCORRECT

1 out of 108 = 0.9%

In the example, looking first only at the locations for which Dr X is very confident, 4 locations are correct. Since there are 12 wind tiles in total, the locations of 33% of the wind tiles were identified correctly. So Dr X is very confident about the locations of 33% of the wind tiles.

But one location is incorrect. There are 108 non-wind tiles, and so 0.9% of the non-wind tiles were picked erroneously. Dr X is very confident that 0.9% of the non-wind tiles are wind tiles.

Most people could get more of the wind tiles if they include the locations they are fairly confident about.

In the example, there are now 4+2=6 locations correct, so the locations of 50% of the wind tiles were identified correctly. But 1+1=2 locations are incorrect, so 1.9% of the non-wind tiles were picked erroneously.

VERY

CONFIDENT

FAIRLY

CONFIDENT

CORRECT

4 out of 12 = 33.3%

4+2=6 6 out of 12 = 50.0%

INCORRECT

1 out of 108 = 0.9%

1+1=2 2 out of 108

= 1.9%

Similarly you may be able to get even more of the wind tiles if we include the locations you were slightly confident about, and those for which you were just guessing.

However, as we start including locations for which you are less confident, the number of locations picked incorrectly is likely to increase.

So the proportion of wind tiles picked correctly increases as confidence decreases, but so also does the proportion of non-wind tiles picked incorrectly.

Calculate the proportion correct and incorrect at all levels of confidence.

Your table should look something like the following

VERY

CONFIDENT

FAIRLY

CONFIDENT

SLIGHTLY

CONFIDENT

JUST GUESSING

CORRECT

4 out of 12 = 33.3%

4+2=6 6 out of 12 = 50.0%

4+2+1=7 7 out of 12 = 58.3%

4+2+1+0=7 7 out of 12 = 58.3%

INCORRECT

1 out of 108 = 0.8%

1+1=2 2 out of 108

= 1.9%

1+1+2=4 4 out of 108

= 3.7%

1+1+2+1=5 5 out of 108

= 4.6%

We can plot these points on a graph …

VERY FAIRLY SLIGHTLY JUSTCONFIDENT CONFIDENT CONFIDENT GUESSING

CORRECT 0.333 0.500 0.583 0.583INCORRECT 0.009 0.019 0.037 0.046

Mahjong Memory Game Scores

0.00

0.20

0.40

0.60

0.80

1.00

0.00 0.02 0.04 0.06 0.08 0.10

Proportion of non-wind tiles picked incorrectly

Pro

po

rtio

n o

f w

ind

tile

s p

ick

ed

co

rre

ctl

y

Congratulations!!

You have just drawn (part of) an ROC graph!

Before we try to make more sense of the graph let us try to work out how good your score is.

How can we determine how good your score is? One way is to compare your score with the scores of people who had not been given an opportunity to memorize the locations of the tiles. These people would have had to guess all of the locations.

The people guessing also list 12 locations, but they are all listed as “just guessing”. Let us consider how many locations they are likely to get correct …

There are 12 wind tiles and 120 tiles in total:

so 10% of the tiles are wind tiles.

There are 108 = 120 – 12 non-wind tiles:

so 90% of the tiles are non-wind tiles.

1210%

120

120 12 10890%

120 120

Some of the people guessing will be lucky and get a large number correct, but others will be unlucky and get few correct. Assuming we have lots of people “just guessing”, we could average their scores.

NUMBER OF GUESSES

AVERAGE CORRECT

AVERAGE INCORRECT

1 0.1 0.9 2 0.2 1.8 3 0.3 2.7 4 0.4 3.6 5 0.5 4.5 6 0.6 5.4 7 0.7 6.3 8 0.8 7.2 9 0.9 8.1 10 1.0 9.0 11 1.1 9.9 12 1.2 10.8

On average 10% of the locations will be correct, and 90% incorrect. So for any number of guesses we can calculate the average scores.

NUMBER OF GUESSES

AVERAGE CORRECT

AVERAGE INCORRECT

1 0.1 out of 12

= 0.8% 0.9 out of 108

= 0.8%

2 0.2 out of 12

= 1.7% 1.8 out of 108

= 1.7%

3 0.3 out of 12

= 2.5% 2.7 out of 108

= 2.5%

4 0.4 out of 12

= 3.3% 3.6 out of 108

= 3.3%

5 0.5 out of 12

= 4.2% 4.5 out of 108

= 4.2%

6 0.6 out of 12

= 5.0% 5.4 out of 108

= 5.0%

7 0.7 out of 12

= 5.8% 6.3 out of 108

= 5.8%

8 0.8 out of 12

= 6.7% 7.2 out of 108

= 6.7%

9 0.9 out of 12

= 7.5% 8.1 out of 108

= 7.5%

10 1.0 out of 12

= 8.3% 9.0 out of 108

= 8.3%

11 1.1 out of 12

= 9.2% 9.9 out of 108

= 9.2%

12 1.2 out of 12

= 10.0% 10.8 out of 108

= 10.0%

The percentage of wind tiles guessed correctly, and of non-wind tiles guessed incorrectly, are the same.

Let us add these scores to the graph …

The line for any number of guesses is shown, but marks are shown only for direct comparison with the example scores.

Memory scores are better than guessing if there are more wind tiles and fewer non-wind tiles selected, i.e. more of the locations are identified correctly than are guessed correctly.

In the example, 5 tiles were marked “very confident”, and 4 of these were correct. On average, only 0.5 tiles would be correct by guessing, so Dr X’s memory is good.

On the graph, a good memory would show points to the left and above the line for guesses.

Later on we will consider the question of how much better than guessing your scores are. But now let us apply what has been learnt to some climate forecasts.

In the Mahjong example you were given 12 chances to point to the locations of 12 wind tiles out of a total of 120 tiles. You were able to use your memory to improve on guessing, and we were able to identify whether your memory improved upon guesses.

This kind of problem is very common, so now let us take an equivalent climatological example…

This time we have 30 years of rainfall forecasts, and the problem is to identify the 10 “dry” years over the 30-year period. The problem is similar to the Mahjong game, but with the following differences:

30 years instead of 120 tiles10 dry years instead of 12 wind tiles20 non-dry years instead of 108 non-wind tiles

We do not have access to the rainfall data, and so cannot memorize the years. Instead we will use the forecasts to select the “dry” years.

The most logical approach is to use the forecast for the least rainfall (1984 at 0.530) as our most likely indication of a “dry” year, the forecast for the second least rainfall (1963 at 0.729) as our second most likely, and the forecast for the third least rainfall (1966 at 0.796) as our third most likely ...

Year Forecast Year Forecast Year Forecast 1961 1.279 1971 0.872 1981 1.023 1962 1.346 1972 0.886 1982 1.060 1963 0.729 1973 1.134 1983 1.221 1964 1.084 1974 1.071 1984 0.530 1965 0.821 1975 0.850 1985 1.042 1966 0.796 1976 0.933 1986 1.120 1967 0.882 1977 0.884 1987 1.002 1968 0.844 1978 1.025 1988 0.928 1969 0.972 1979 1.039 1989 1.025 1970 1.216 1980 1.032 1990 1.223

We could continue listing all years in order of how confident we are that each year is one of the 10 “dry” years. We would be most confident that 1984 is “dry” and least confident that 1962 is “dry”.

Rank Year Forecast Rank Year Forecast Rank Year Forecast

1 1984 0.530 11 1988 0.928 21 1982 1.060

2 1963 0.729 12 1976 0.933 22 1974 1.071

3 1966 0.796 13 1969 0.972 23 1964 1.084

4 1965 0.821 14 1987 1.002 24 1986 1.120

5 1968 0.844 15 1981 1.023 25 1973 1.134

6 1975 0.850 16 1989 1.025 26 1970 1.216

7 1971 0.872 17 1978 1.025 27 1983 1.221

8 1967 0.882 18 1980 1.032 28 1990 1.223

9 1977 0.884 19 1979 1.039 29 1961 1.279

10 1972 0.886 20 1985 1.042 30 1962 1.346

The actual “dry” years are highlighted above. Let us calculate a score table in the same way as for the memory game. We will use each rank as a decreasing level of confidence.

Rank Year Forecast Rank Year Forecast Rank Year Forecast

1 1984 0.530 11 1988 0.928 21 1982 1.060

2 1963 0.729 12 1976 0.933 22 1974 1.071

3 1966 0.796 13 1969 0.972 23 1964 1.084

4 1965 0.821 14 1987 1.002 24 1986 1.120

5 1968 0.844 15 1981 1.023 25 1973 1.134

6 1975 0.850 16 1989 1.025 26 1970 1.216

7 1971 0.872 17 1978 1.025 27 1983 1.221

8 1967 0.882 18 1980 1.032 28 1990 1.223

9 1977 0.884 19 1979 1.039 29 1961 1.279

10 1972 0.886 20 1985 1.042 30 1962 1.346

Rank Year Correct Incorrect Rank Year Correct Incorrect 1 1984 10% 0% 16 1989 70% 45% 2 1963 10% 5% 17 1978 70% 50% 3 1966 10% 10% 18 1980 70% 55% 4 1965 20% 10% 19 1979 70% 60% 5 1968 20% 15% 20 1985 70% 65% 6 1975 30% 15% 21 1982 70% 70% 7 1971 40% 15% 22 1974 70% 75% 8 1967 40% 20% 23 1964 70% 80% 9 1977 40% 25% 24 1986 70% 85% 10 1972 50% 25% 25 1973 80% 85% 11 1988 60% 25% 26 1970 80% 90% 12 1976 60% 30% 27 1983 90% 90% 13 1969 70% 30% 28 1990 90% 95% 14 1987 70% 35% 29 1961 100% 95% 15 1981 70% 40% 30 1962 100% 100%

Our first selection is correct, so we have identified 1 (10%) of the 10 “dry” years correctly. The second is incorrect, so we have selected 1 (5%) of the 20 “non-dry” years incorrectly. Now we can plot these points …

We can assess how good the forecasts are in exactly the same way as for the memory game. Let us compare the scores for the forecasts with the scores for people who have no forecasts available. These people would have had to guess all of the “dry” years.

There are 10 “dry” years and 30 years in total:

so about 33% of the years are “dry”.

There are 20 = 30 – 10 “non-dry” years:

so about 67% of the years are “non-dry”.

1033%

30

30 10 2067%

30 30

On average about 33% of the years picked will be correct, and 67% incorrect. So for any number of guesses the average scores will be:

NUMBER OF GUESSES

AVERAGE CORRECT

AVERAGE INCORRECT

1 1 33% = 0.33 0.33 out of 10

= 3.3%

1 67% = 0.67 0.67 out of 20

= 3.3%

2 2 33% = 0.67 0.67 out of 10

= 6.7%

2 67% = 1.33 1.33 out of 20

= 6.7%

3 3 33% = 1.00 1.00 out of 10

= 10.0%

3 67% = 2.00 2.00 out of 20

= 10.0%

…

And we can add these scores to the graph …

Remember that for forecasts to be better than guessing, the forecasts should correctly identify more “dry” years and incorrectly identify fewer “non-dry” years than the guesses. For good forecasts the curve will be to the left and above the diagonal line: here the forecasts identify a large proportion of “dry” years while picking only a small proportion of the “non-dry” years incorrectly.

The forecasts seem to be good in the left and middle part of the graph only. What does this mean?

Consider again the table showing the order in which we picked the “dry” years. Notice that most of the “dry” years are identified by our most confident choices. So our most confident selections were fairly successful.


However, after about the 13th selection, the forecasts do not provide useful guidance for identifying any more of the “dry” years.


We can draw the following conclusion from the ROC graph:

When the forecast suggests that conditions are going to be dry we can be reasonably confident that “dry” conditions will occur. However, when the forecast suggests that conditions are going to be normal or wet we cannot make any useful statement about the likelihood of “dry” conditions.

The correlation between the observed and the forecast rainfall is 0.044. Based on the correlation, we would normally ignore these forecasts, but the ROC graph suggests they may be useful in forecasting “dry” conditions.

The ROC graph indicates that the forecasts are better than guesses, but by how much?

We can use a special formula, known as the hypergeometric equation, to calculate the chance of somebody guessing the same number of “dry” years as we forecast correctly.

number of dry years number of non-dry years

number of correct warnings number of incorrect warningschance

number of years

number of warnings

This equation is available as a function in packages such as MS Excel (HYPGEOMDIST).

To use this equation to calculate how good the forecasts are, we must first chose which years we are going to treat as forecasts of “dry” conditions. Let us issue a warning of “dry” conditions when the forecast is less than 1.0. The actual “dry” years are marked in dark blue.

Year Forecast Year Forecast Year Forecast 1961 1.279 1971 0.872 1981 1.023 1962 1.346 1972 0.886 1982 1.060 1963 0.729 1973 1.134 1983 1.221 1964 1.084 1974 1.071 1984 0.530 1965 0.821 1975 0.850 1985 1.042 1966 0.796 1976 0.933 1986 1.120 1967 0.882 1977 0.884 1987 1.002 1968 0.844 1978 1.025 1988 0.928 1969 0.972 1979 1.039 1989 1.025 1970 1.216 1980 1.032 1990 1.223

Now we can define the individual terms of the equation:

number of dry years = 10number of non-dry years = 20number of years = 30number of correct warnings = 7number of incorrect warnings = 6number of warnings = 13

number of dry years number of non-dry years

number of correct warnings number of incorrect warningschance

number of years

number of warnings

The result tells us that only about 3.9% of people who had 13 attempts to guess the 10 “dry” years would get 7 of the years correct.

But the result only tells us the chances of somebody getting exactly the same number correct by guessing as we got using the forecasts. Some people could get more than 7 correct by guessing.

Because some of the people who are purely guessing may have more than 7 years correct, we should count these as well. So we need to calculate the chances of somebody doing as well as, or better than, our forecasts by just guessing.

The chances of getting 8, 9, or 10 by guessing are about 0.58%, 0.02%, and 0.0004% respectively. Adding these, the chances of guessing 7 or more of the “dry” years are about 4.5%.

We would, of course, get a different answer if we used a different criterion for deciding when to issue a warning. You should be wary of the many problems in performing such significance tests.

However, the ROC graph does suggest that these forecasts do contain some useful information, despite a correlation of close to zero.

As a summary measure of the graph, the area under the ROC curve is frequently calculated …

The area beneath the guessing line is 0.5, and so an area greater than 0.5 suggests the forecasts are good.

The area beneath the graph for our forecasts is 0.61.

What does this mean?

The area beneath the graph tells us the probability that the forecast for a “dry” year is drier than the forecast for a “non-dry” year.

If we were given a forecast for one of the “dry” years and one for one of the “non-dry” years, we would identify the “dry” year correctly 61% of the time.

In practice, this information is not very helpful to the user! However, the graph as a whole should be very informative to the forecaster.

FeedbackIn order to monitor the usefulness of this course, and to make revisions and improvements, please could you forward any comments or suggestions.

Dr Simon J. Mason

International Research Institute for Climate Prediction

Columbia University

61 Route 9 W

Palisades, NY 10964-8000

USA

E-mail: [email protected]

Recommended readings

• Murphy, A. H., 1993: What is a good forecast? An essay on the nature of goodness in weather forecasting. Weather and Forecasting, 8, 281–293.

• Wilks, D. S., 1995: Statistical Methods in the Atmospheric Sciences, Academic Press, San Diego. Chapter 7, Forecast verification, pp 233–283.

• Mason, S. J., and N. E. Graham, 1999: Conditional probabilities, relative operating characteristics, and relative operating levels. Weather Forecasting, 14, 713–725.

• Mason, S. J., and N. E. Graham, 2002: Areas beneath the relative operating characteristics (ROC), and levels (ROL) curves: statistical significance and interpretation. Quarterly Journal of the Royal Meteorological Society, 128, 2145–2166.

Understanding the Relative Operating Characteristic (ROC) Simon Mason International Research...

Documents

Transcript of Understanding the Relative Operating Characteristic (ROC) Simon Mason International Research...