Validation of the GLC2000 productsValidation of the GLC2000 products

Philippe Mayaux

GLC 2000 validation strategyGLC 2000 validation strategy

Confidence-building method• Systematic review of the regional products by experts and

comparison with reference data

Design-based method• Quantitative accuracy assessment of the global product using a

stratified random sampling of high-resolution sites (in agreement with the CEOS-IGOS recommendations)

Objectives of the quality controlObjectives of the quality control

• to avoid macroscopic errors before the quantitative accuracy assessment

• Wrong labels & limits

• Inter- and intra-continental inconsistencies

• Location of the errors (thematic or geographical)

• to improve the global acceptance of GLC2000 products – collaboration with local partners

Confidence-building methodConfidence-building method

Systematic descriptive protocol to document the verification per cell (proposed size = 2 by 2 degrees at the equator ~ 50,000 km2)

Use of ancillary data (maps, Landsat & SPOT images, aerial photographs…), expert opinion and intrinsic properties of the dataset (temporal profiles, colour composites…)

Archive the evaluation in a database

The qualitative validation gridThe qualitative validation grid

Qualitative check fieldsQualitative check fields

Name of the expert Type of reference material: high-resolution image, quick-look, thematic

map, aerial photograph, field photograph Spatial pattern: from homogenous to heterogeneous

(4 levels) Overall quality of the GLC product, very good, good, acceptable,

unacceptable LC classes well-identified and LC classes poorly identified Nature of problem: label, limit, missing class, other

Status of the quality controlStatus of the quality control

• Regions covered: Eurasia, Asia, Scandinavia, Africa, Canada

Derived analysisDerived analysis

Overall Quality Heterogeneous Homogeneous

Very heterogeneous

Very homogeneous Total

Good 217 19 29 44 309

Moderate 44 8 11 5 68

Unacceptable 1 1

Very Good 1 4 5

Total 263 27 40 53

An example in RussiaAn example in Russia

An exemple in AsiaAn exemple in Asia

Design-based validationDesign-based validation

Objective: To provide a statistical assessment of the accuracy by class and an overall accuracy of the global map.

Constraints: budget (data, interpretation) time spatial complexity seasonality

Key issues for the validationKey issues for the validation

Sampling issues (scheme, frame, size) Reference material (nature, interpretation) Accuracy statements (contigency matrix, fuzzy logic, double

regression) Single pixels or pixel blocks? Upscaling of legends (mosaic classes) Spatial pattern (linear, massive, diffuse) Rare classes important for several users Universal dataset

Detailed discussion during the CEOS workshop “Validation of Global Land-Cover Products”, 27-28 March

Land-cover class areaLand-cover class area

0.0%

2.0%

4.0%

6.0%

8.0%

10.0%

12.0%

14.0%

16.0%

Tree

cove

r bro

adle

aved

eve

rgre

en

Tree

cove

r bro

adle

aved

dec

iduo

us

Tree

cove

r nee

dlele

aved

eve

rgre

en

Tree

cove

r nee

dlele

aved

dec

iduou

s

Tree

cove

r mix

ed

Flood

ed fo

rest

Shrubla

nd eve

rgre

en

Shrubla

nd dec

iduous

Ope

n-clo

sed g

rass

land

Sparse

grass

land

Regula

rly fl

ooded g

rass

- & s

hrubla

nds

Lichen

& m

osses

Cultiva

ted

& man

aged

are

as

Tree

cove

r / o

ther

nat

ural v

egeta

tion

Cropla

nd / T

ree

cove

r

Cropla

nd / o

ther

nat

ural v

eget

atio

n

Bare

soil

Wate

rbodie

s

Snow & Ic

e

Cities

Regions of interestRegions of interest

North America14.6%

South America14.4%

Africa24.0%

Europe4.8%

Northern Eurasia16.9%

Asia17.0%

Oceania+Insular8.4%

Land cover area per regionLand cover area per region

North & Central America South America Africa Europe

Northern Eurasia Asia

Oceania + Insular SEA Total

Tree cover broadleaved evergreen 4,1% 56,4% 16,5% - 0,1% 6,8% 16,1% 10,0%

Tree cover broadleaved deciduous 12,5% 11,6% 41,5% 5,2% 11,6% 11,9% 5,7% 8,5%

Tree cover needleleaved evergreen 55,4% 0,0% 0,0% 10,3% 23,0% 11,3% - 8,0%

Tree cover needleleaved deciduous - - - - 96,5% 3,5% - 3,6%

Tree cover mixed 55,0% 0,5% - 10,0% 33,8% 0,7% - 3,8%

Flooded forest - 64,0% 29,0% - - 0,2% 6,8% 0,4%

Shrubland evergreen - 26,8% 0,1% 0,4% 9,7% 61,7% 1,5% 2,9%

Shrubland deciduous 0,4% - 43,5% 2,7% 25,3% 4,0% 24,1% 5,0%

Open-closed grassland 19,2% 13,2% 21,2% 3,6% 8,9% 23,3% 10,6% 11,6%

Sparse grassland 17,4% 12,3% 23,2% 1,2% 15,8% 9,8% 20,4% 9,9%

Regularly flooded grass- & shrublands 8,9% 43,0% 10,6% 8,4% 10,8% 12,4% 5,9% 0,7%

Lichen & mosses 32,4% - - 1,3% 65,8% 0,6% - 1,1%

Cultivated & managed areas 14,4% 12,3% 12,4% 16,5% 9,9% 29,1% 5,5% 12,9%

Tree cover / other natural vegetation - - 74,8% 3,3% 21,8% 0,1% - 1,6%

Cropland / Tree cover 5,0% 48,4% 18,1% 4,8% 13,0% 5,8% 5,0% 2,0%

Cropland / other natural vegetation - 22,8% 27,9% 2,3% 19,5% 6,6% 20,8% 3,0%

Bare soil 2,0% 2,9% 51,1% 0,2% 12,5% 29,3% 2,1% 15,1%

Total 14,6% 14,4% 24,0% 4,8% 16,9% 17,0% 8,4%

Random sampling equally distributed per regionRandom sampling equally distributed per region

Option 1: Totally random sampling (N=510)North & Central

America South America Africa EuropeNorthern Eurasia Asia

Oceania+Insular Total %

Tree cover broadleaved evergreen 2 29 8 0 0 3 8 51 10,0%

Tree cover broadleaved deciduous 5 5 18 2 5 5 2 44 8,5%

Tree cover needleleaved evergreen 23 0 0 4 9 5 0 41 8,0%

Tree cover needleleaved deciduous 0 0 0 0 18 1 0 18 3,6%

Tree cover mixed 11 0 0 2 7 0 0 19 3,8%

Flooded forest 0 1 1 0 0 0 0 2 0,4%

Shrubland evergreen 0 4 0 0 1 9 0 15 2,9%

Shrubland deciduous 0 0 11 1 6 1 6 26 5,0%

Open-closed grassland 11 8 13 2 5 14 6 59 11,6%

Sparse grassland 9 6 12 1 8 5 10 50 9,9%

Regularly flooded grass- & shrublands 0 2 0 0 0 0 0 4 0,7%

Lichen & mosses 2 0 0 0 4 0 0 5 1,1%

Cultivated & managed areas 9 8 8 11 7 19 4 66 12,9%

Tree cover / other natural vegetation 0 0 6 0 2 0 0 8 1,6%

Cropland / Tree cover 1 5 2 0 1 1 1 10 2,0%

Cropland / other natural vegetation 0 3 4 0 3 1 3 15 3,0%

Bare soil 2 2 39 0 10 23 2 77 15,1%

Total 75 73 122 24 86 87 43 510

Stratified sampling per classStratified sampling per class

Option 2: Stratified sampling per class and equally distributed per continent (N=510, n=30)

North & Central America South America Africa Europe

Northern Eurasia Asia Oceania+Insular Total %

Tree cover broadleaved evergreen 1 17 5 0 0 2 5 30 5.9%

Tree cover broadleaved deciduous 4 3 12 2 3 4 2 30 5.9%

Tree cover needleleaved evergreen 17 0 0 3 7 3 0 30 5.9%

Tree cover needleleaved deciduous 0 0 0 0 29 1 0 30 5.9%

Tree cover mixed 17 0 0 3 10 0 0 30 5.9%

Flooded forest 0 19 9 0 0 0 2 30 5.9%

Shrubland evergreen 0 8 0 0 3 19 0 30 5.9%

Shrubland deciduous 0 0 13 1 8 1 7 30 5.9%

Open-closed grassland 6 4 6 1 3 7 3 30 5.9%

Sparse grassland 5 4 7 0 5 3 6 30 5.9%

Regularly flooded grass- & shrublands 3 13 3 3 3 4 2 30 5.9%

Lichen & mosses 10 0 0 0 20 0 0 30 5.9%

Cultivated & managed areas 4 4 4 5 3 9 2 30 5.9%

Tree cover / other natural vegetation 0 0 22 1 7 0 0 30 5.9%

Cropland / Tree cover 1 15 5 1 4 2 1 30 5.9%

Cropland / other natural vegetation 0 7 8 1 6 2 6 30 5.9%

Bare soil 1 1 15 0 4 9 1 30 5.9%

Total 68 94 111 21 113 65 37 510

Stratified sampling per classStratified sampling per class

Option 3: Stratified by class with a minimum number for small classes and fixed number for large classes (N=500, n=20 or n=40)

North & Central America South America Africa Europe

Northern Eurasia Asia Oceania+Insular Total %

Tree cover broadleaved evergreen 2 23 7 0 0 3 6 40 8.0%

Tree cover broadleaved deciduous 5 5 17 2 5 5 2 40 8.0%

Tree cover needleleaved evergreen 22 0 0 4 9 5 0 40 8.0%

Tree cover needleleaved deciduous 0 0 0 0 19 1 0 20 4.0%

Tree cover mixed 11 0 0 2 7 0 0 20 4.0%

Flooded forest 0 13 6 0 0 0 1 20 4.0%

Shrubland evergreen 0 5 0 0 2 12 0 20 4.0%

Shrubland deciduous 0 0 17 1 10 2 10 40 8.0%

Open-closed grassland 8 5 8 1 4 9 4 40 8.0%

Sparse grassland 7 5 9 0 6 4 8 40 8.0%

Regularly flooded grass- & shrublands 2 9 2 2 2 2 1 20 4.0%

Lichen & mosses 6 0 0 0 13 0 0 20 4.0%

Cultivated & managed areas 6 5 5 7 4 12 2 40 8.0%

Tree cover / other natural vegetation 0 0 15 1 4 0 0 20 4.0%

Cropland / Tree cover 1 10 4 1 3 1 1 20 4.0%

Cropland / other natural vegetation 0 5 6 0 4 1 4 20 4.0%

Bare soil 1 1 20 0 5 12 1 40 8.0%

Total 70 85 116 22 97 69 42 500

Option 3b: Stratified by class with a minimum number for small classes and variable number for large classes (N=593, n=20 to 77)

North & Central America South America Africa Europe

Northern Eurasia Asia

Oceania +Insular Total

Tree cover broadleaved evergreen 2 29 8 0 0 3 8 51 8,6%

Tree cover broadleaved deciduous 5 5 18 2 5 5 2 44 7,3%

Tree cover needleleaved evergreen 23 0 0 4 9 5 0 41 6,9%

Tree cover needleleaved deciduous 0 0 0 0 19 1 0 20 3,4%

Tree cover mixed 11 0 0 2 7 0 0 20 3,4%

Flooded forest 0 13 6 0 0 0 1 20 3,4%

Shrubland evergreen 0 5 0 0 2 12 0 20 3,4%

Shrubland deciduous 0 0 11 1 6 1 6 26 4,3%

Open-closed grassland 11 8 13 2 5 14 6 59 9,9%

Sparse grassland 9 6 12 1 8 5 10 50 8,5%

Regularly flooded grass- & shrublands 2 9 2 2 2 2 1 20 3,4%

Lichen & mosses 6 0 0 0 13 0 0 20 3,4%

Cultivated & managed areas 9 8 8 11 7 19 4 66 11,1%

Tree cover / other natural vegetation 0 0 15 1 4 0 0 20 3,4%

Cropland / Tree cover 1 10 4 1 3 1 1 20 3,4%

Cropland / other natural vegetation 0 5 6 0 4 1 4 20 3,4%

Bare soil 2 2 39 0 10 23 2 77 13,0%

Total 82 99 141 27 104 93 46 593

Stratified sampling per class Stratified sampling per class

Interpretation of reference materialInterpretation of reference material

Reference material: high spatial resolution interpretations– CNES & NASA / USGS / UNEP-GRID– Existing sources of images

Interpretation by local experts– Very-well focused (better 1 or 2 good experts than a pool of less

qualified experts)– Contracts with GVM Unit when necessary (realised on-site)

Visual interpretation– Efficiency– Accuracy

LCCS-based interpretation of the high-resolution images