Propagating error in land-cover-change analyses: impact of ...danbrown/papers/aburnick_ijgis.pdf ·...

Propagating error in land-cover-change analyses: impact of temporaldependence under increased thematic complexity

Amy C. Burnickia*, Daniel G. Brownb and Pierre Goovaertsc

aDepartment of Geography, University of Wisconsin-Madison, Madison, WI, USA; bSchool ofNatural Resources and Environment, University of Michigan, Ann Arbor, MI, USA;

cBioMedware, Inc., 3526 West Liberty Suite 100, Ann Arbor, MI, USA

(Received 2 March 2009; final version received 23 August 2009)

We examined the impact of temporal dependence between patterns of error in classifiedtime-series imagery through a simulation modeling approach. This research extended theland-cover-change simulation model we previously developed to investigate: (1) theassumption of temporal independence between patterns of error in classified time-seriesimagery; and (2) the interaction of patterns of change and patterns of error in a post-classification change analysis. In this research, the thematic complexity of the classifiedland-cover maps was increased by increasing the number of simulated land-cover classes.Simulating maps with increased categorical resolution permitted the incorporation of: (1)higher-order, more complex spatial and temporal interactions between land-coverclasses; and (2) patterns of error that better reproduce the complex error interactionsthat often occur in time-series classified imagery. The overall modeling framework wasdivided into two primary components: (1) generation of a map representing true change;and (2) generation of a suite of change maps that had been perturbed by specific patternsof error. All component maps in the model were produced using simulated annealing,which enabled us to create a series of map realizations with user-defined spatial andtemporal patterns. Comparing the true map of change to the error-perturbed maps ofchange using accuracy assessment statistics showed that increasing the temporal depen-dence between classification errors did not improve the accuracy of resulting maps ofchange when the categorical scale of the land-cover classified maps was increased. Theincreased structural complexity within the time series of maps effectively inhibited theimpact of temporal dependence. However, results demonstrated that there are interactionsbetween patterns of error and patterns of change in a post-classification change analysis.These interactions played a major role in determining the accuracy associated with themaps of change.

Keywords: complexity; geostatistics; land-use and land-cover change; simulation;uncertainty

1. Introduction

Research in the field of land-use/cover change (LUCC) has increased dramatically over thepast two decades due to the direct relationship between LUCC and important globalprocesses such as loss of biodiversity, biochemical and hydrological cycling, and landproductivity. Much research effort has focused on developing models that successfullylink the driving forces of land-use change to observed changes in land cover (LC)

International Journal of Geographical Information ScienceVol. 24, No. 7, July 2010, 1043–1060

*Corresponding author. Email: [email protected]

ISSN 1365-8816 print/ISSN 1362-3087 online# 2010 Taylor & FrancisDOI: 10.1080/13658810903279008http://www.informaworld.com

Downloaded By: [University of Michigan] At: 18:35 4 August 2010

mailto:[email protected]

http://www.informaworld.com

(Veldkamp and Fresco 1996, Mertens and Lambin 2000, Schneider and Pontius 2001,Serneels and Lambin 2001, Overmars and Verburg 2005). However, the reliability ofthese models directly depends on the accuracy of observed LC changes. Several importantresearch programs operating in the field of LUCC have identified understanding andquantifying the uncertainty associated with observed changes in LC as a key priority forLUCC (Nunes and Auge 1999, NRC 2001, Ojima et al. 2005). Our research contributes tothis goal by examining the impact of varying spatial and temporal patterns of error on a post-classification change analysis.

Post-classification change analysis is the process by which two or more classified LCmaps,representing distinct moments in time, are overlaid to determine whether a change in theassigned LC class has occurred on a pixel-by-pixel basis. The primary determinant of theaccuracy of the resulting map of LC change (LCC) is the accuracy associated with eachindividual LC classification. However, quantifying error in the resulting map of change requiresmore than simply understanding the magnitude and pattern of error present in each individualLC map. It is also necessary to understand how these patterns of error interact over time.

Many studies typically assume temporal independence between errors associated witheach map in the time series when assessing the accuracy of a map of change, that is thepattern of error in an LC map at one period is not correlated to the pattern of error for thesame area at a later time period (Carmel et al. 2001, Sohl et al. 2004). However, recentresearch has demonstrated that classification errors can interact over time (Carmel 2004, Liuand Zhou 2004, van Oort 2005). Furthermore, research has shown that temporal dependencein error within a time series of classified maps leads to an increase in the accuracy associatedwith the resulting map of change (van Oort 2005, Burnicki et al. 2007). Temporal depen-dence concentrates error occurrences in similar regions within the time series. As a result,larger proportions of the mapped area are free from error, thereby increasing the accuracy ofthe map of change and its ability to correctly detect LC transitions.

This research builds upon an LCC simulation model developed by Burnicki et al. (2007).The objective of the initial modeling effort was to quantify the interaction of classificationerrorswithin a time series of LCmaps to improve our understanding of the propagation of errorin analyses of LCC. Specifically, the initial model directly assessed the assumption of temporalindependence between patterns of error in classified time-series imagery. The LCC simulationmodel created multiple versions of a time series of classified maps by investigating threeprimary components of a change-detection analysis: (1) patterns of change that result in an LCmap for the latter time period (time-2); (2) patterns of error associated with each classified LCmap in the time-series (time-1 and time-2); and (3) level of temporal dependence between thepatterns of error in the time series. Burnicki et al. (2007) showed that (1) increasing thestrength of the temporal correlation between patterns of error improved the overall accuracy ofthe resulting map of change and (2) the spatial pattern of change specified in each simulation(e.g., change occurring randomly across the mapped surface versus change centered at LCclass boundaries) played a key role in determining the relative impact increasing temporaldependence had on the accuracy of the final map of change.

A limitation of Burnicki’s et al. (2007) model is that the simulated LC maps consisted ofonly two LC categories, mirroring a forest/non-forest classification. Because many LUCCanalyses involve more than two categories, a natural next step in understanding the propaga-tion of error in analyses of LCC was to extend our model to LC maps comprising more thantwo LC categories. This extension allowed us to determine whether modeling resultsobserved in the two LC class case (i.e., increase in overall change map accuracy resultingfrom an increase in temporal dependence) would also be observed when the complexity ofthe simulated maps in the time series was increased.

1044 A.C. Burnicki et al.


2. Extending the modeling approach

The incorporation of complexity into models of environmental and ecological systems isneeded if models are to reflect the complexity observed in natural systems (Wu and David2002b, Aplin 2006, Cadenasso et al. 2006, Gomez-Hernandez 2006, Wagener and Kollat2007). The adoption of complexity in environmental/ecological models can take severalforms, such as incorporating diverse components, nonlinear interactions, organizationalconnectivity, and spatial and temporal heterogeneity (Wu and David 2002a, Cadenassoet al. 2006). Complexity was incorporated in our modeling effort by increasing the structuralcomplexity of the analysis. Structural complexity refers to the diversity of system elementsand the level of sophistication in their spatial relationships (Wu and David 2002a). In thisresearch, both the spatial extent and the spatial resolution of the simulated time series mapswere held constant from the original simulation model. Therefore, structural complexity wasincorporated by modifying the categorical resolution of the simulated maps (Ju et al. 2005):the number of LC categories was increased from two to three LC classes. Instead ofmirroring a forest/non-forest classification, simulated maps were based on a classificationthat identified the LC classes as Developed, Forest, and Other, where Other representedagriculture, grassland, wetland, and water. Simulating LC maps with three categoriespermitted the incorporation of more complex spatial and temporal interactions betweenthe individual LC classes and the errors associated with these classes. It also enabled a betterreproduction of the complex error interactions existing in real-world time-series maps.

3. Methods

The original experiment (Burnicki et al. 2007) was designed to address two main researchquestions: (1) How do specific patterns of error, particularly patterns with increasingtemporal dependence, affect the ability of a map of change to correctly detect LC transitions;and (2) Does the specified pattern of change interact with the pattern of error to mediate theimpact of increasing temporal dependence on the accuracy of the map of change? Ourcurrent research examined these questions under conditions of increased landscapecomplexity.

Both questions provide important insights for quantifying the level of uncertaintyassociated with a map of change and developing an error-propagation model for post-classification change analysis. The answer to the first research question affects how changemaps are assessed for accuracy and how error-propagation models are constructed. If asignificant increase in overall change map accuracy is observed when a temporal depen-dence between classification errors exists, simply multiplying the overall accuracies asso-ciated with each map in the time series (i.e., assuming temporal independence) willunderestimate the accuracy of the map of change. Furthermore, more complex error-propagation models that incorporate a temporal dependence will be necessary. The answerto the second research question will shed light on whether certain patterns of change arebetter predicted in the presence of various patterns of error.

3.1. Experimental design

The modeling framework used in this research exactly follows the original design developedfor the two LC class case. Readers should refer to Burnicki et al. (2007) for a detaileddescription of the LCC simulation model. However, a brief summary is presented below toorient readers.

International Journal of Geographical Information Science 1045


The goal of the LCC simulation model was to create two versions of a time series ofclassified maps: one conveying a specific pattern of change in the absence of error andanother conveying the same pattern of change in the presence of a specific pattern of error(Figure 1). By creating an error-free and error-perturbed time series of classified maps, wewere able to examine how various patterns of error affected the accuracy of a map of change,particularly patterns with increasing temporal dependence within the time series. Spatialsimulation was used to create the primary components of the LCC simulation model, whichincluded the LCmap for the initial time period (time-1), the pattern of change that resulted inthe LC map for the subsequent time period (time-2), and the patterns of error that resulted inthe error-perturbed versions of the time-1 and time-2 LC maps.

Simulation, as opposed to real-world maps, was utilized in this research because itallowed us to explicitly define the spatiotemporal pattern of error for the classified maps.Burnicki et al. (2007) demonstrated that simulation was useful in modeling error propaga-tion due to the ability to define the structure of the classified maps and their associatedpatterns of error, thereby enabling the direct assessment of increasing temporal depen-dence. The application of simulation in error-propagation research, including researchexamining error propagation in LCC, is well documented (e.g., Goodchild et al. 1992,Heuvelink and Burrough 1993, Arbia et al. 1998, Carmel et al. 2001). Simulating patternsof error provides comprehensive error information for the classified maps, which is neededto construct a spatiotemporal error-propagation model. Comprehensive error informationis often lacking in empirical studies, which present only an overall accuracy measure orerror matrix calculated from a small sample of the mapped surface and seldom report

Time-1classified map

Time-1classificationprobability surfaces

Mulitple realization of time-1error probability surfaces

Mulitple realizations of time-1error-perturbed classification

probability surfaces

Mulitple realizations of time-1error-perturbed classified map

Mulitple realizations of time-2error-perturbed classified map

Mulitple realizationsof error-perturbed

classified map

Mulitple realizations of time-2error-perturbed classification

probability surfaces

Mulitple realization of time-2error probability surfaces

Time-2 classificationprobability surfaces

Change probabilitysurfaces

Time-2classified map

True changemap

OVERLAY

OVERLAY

Figure 1. Overview of the simulation and map processing steps comprising each run of the LCCsimulation model, where map surfaces symbolized by darker shades of gray represent surfaces createdusing simulated annealing.



information regarding a temporal dependence between classification errors (van Oort2005).

Three patterns of error and change were investigated, replicating the patterns examinedin the two-category case:

(1) Patterns of error(a) Error occurred randomly across the surface of the map for all three LC classes

and exhibited a specified level of spatial autocorrelation, with error patterns attime-1 and time-2 independent.

(b) Error occurred randomly across the surface of the map for all three LC classesand exhibited a specified level of spatial autocorrelation, with error patterns attime-1 and time-2 exhibiting a defined level of correlation.

(c) Error exhibited a defined level of correlation to the individual LC classificationboundaries for all three LC classes at time-1 and a defined level of spatialautocorrelation, with error patterns at time-1 and time-2 exhibiting a definedlevel of correlation.

(2) Patterns of change(a) Expansion of LC classes in opposing directions through the addition of trends in

the X direction.(b) Change occurred randomly across the surface of the map for all LC transitions

modeled and exhibited a specified level of spatial autocorrelation.(c) Change exhibited a defined level of correlation to the individual LC classifica-

tion boundaries of the ‘to’ class for all LC transitions modeled and exhibited aspecified level of spatial autocorrelation.

The error-free and error-perturbed time series of classified maps were produced bysimulating classification-, change-, and error-probability surfaces (Figure 1). These surfacesrepresent the probability a pixel belongs to an individual LC class, experiences a specific LCtransition, or experiences an error in classification, respectively. The creation of the LC mapfor the initial time period (time-1) and the LC maps perturbed by error required thesimulation of three classification- or error-probability surfaces, one for each LC class. Thecreation of the LC map for the subsequent time period (time-2) required the simulation offour change-probability surfaces. Only four surfaces were simulated because changes out ofthe Developed class were prohibited, that is no transitions were allowed from Developed toForest or Developed to Other. Any pixel assigned to the Developed class at time-1 remainedin the Developed class at time-2.

3.2. Simulated annealing

All simulated probability surfaces were produced using simulated annealing (Kirkpatricket al. 1983, Deutsch and Journel 1992, Goovaerts 1997). Previous modeling demonstratedthat simulated annealing: (1) provided the flexibility needed to specify the structure of thetime-1 and time-2 classified maps, as well as the structure and magnitude of the associatedclassification errors; and (2) enabled the generation of multiple realizations of each set oftime-series classified maps to thoroughly assess the research questions (Burnicki et al. 2007).

Simulated annealing attempts to find multiple, near-optimal solutions to an optimizationproblem specified by an objective function (de Smith et al. 2007). It utilizes a random searchmethod to gradually modify an initial solution through perturbations, such as changing theLC of a particular pixel. Perturbations that increase the value of the objective function are



also accepted according to a cooling schedule, so that the initial solution space is fullyexplored before focusing in on a particular, promising region (Duh and Brown 2005). In thisresearch, a negative exponential cooling schedule was adopted to permit the acceptance of alarge proportion of unfavorable solutions initially. The probability of acceptance thendecreased slowly over time in order to converge to a single solution. The process iteratesuntil the objective function is reasonably low or further perturbations to the system result inminimal changes to the objective function value (Goovaerts 1997).

Several modifications to this general framework were necessary to incorporate theincreased complexity of three LC classes. The first modification concerned the creation ofthe initial solution. Two constraints were imposed on the initial probability surfaces: (1)individual classification probabilities must range between 0 and 1; and (2) the sum of theclassification probabilities across all three LC classes must sum to 1. To achieve bothconstraints, the initial maps were created by randomly drawing three probability valuesfrom a uniform distribution ranging from 0 to 1 at each pixel location. The resulting vector ofprobabilities were summed and the individual probability values were divided by this sum toensure the vector of rescaled probabilities summed to 1 across all three initial random maps.

The second modification was an adjustment to the method of perturbation. To maintainthe sum-to-one constraint and limit the CPU time required to simulate three classification-probability surfaces, the perturbation method proceeded as follows: (1) a single pixel wasselected at random; (2) a single probability value within the vector of probabilities wasselected at random; (3) the probability value was perturbed by swapping the current value fora new probability value drawn randomly from a uniform distribution ranging from 0 to 1;and (4) the perturbed vector of probabilities was rescaled to sum to 1.

A series of objective functions were defined to produce the initial LC map, maps ofchange, and maps of error. Each objective function was comprised of a series of statistics(i.e., objective function components) that defined the spatial structure of the simulated mapsand the relative proportions of each LC, LCC, or error class. A detailed description anddefinition of the objective function components, and the resulting objective functions, ispresented in the Appendix.

Probability surfaces were produced by modifying the program code of the simulatedannealing program SASim within the software package GSLib (Deutsch and Journel 1992).The resulting probability surfaces, with lattice dimensions 175 · 175, were imported intothe GIS package Idrisi (Eastman 1999) for all raster-based processing (map overlay andclassification). All time-series maps were classified using a maximum likelihood classifica-tion rule, that is each pixel is assigned to the LC class with the highest probability ofoccurrence.

3.3. Simulating component maps

The goal in creating the simulated time-series maps was to produce a series of maps thatreflected real-world patterns and proportions of LC and LCC. To achieve this, Landsat TMimagery (30-m resolution) was acquired for Washtenaw County, Michigan, located in thesoutheast portion of the state. Images were acquired for two time points, separated by aperiod of eight years, and classified into the categories Developed, Forest, and Other. A post-classification change analysis produced a map of change illustrating persistence inDeveloped, Forest, and Other and the following LC transitions: Forest to Developed,Forest to Other, Other to Developed, and Other to Forest.

These maps served as the basis for parameterizing the time-1 classification- and change-probability surfaces and establishing target values for optimization. The parameters for the



error-probability surfaces were not based on the Washtenaw County imagery given the lackof comprehensive error information. Instead, parameters were set to reasonable values, asdescribed below, based on the parameter values assigned to the change-probability surfacesand the values used to parameterize the error-probability surfaces in the two LC classsimulation model.

The two primary parameters determined using the Washtenaw County data were: (1)degree of spatial autocorrelation within and between each LC or LCC class, as defined by therange of fitted semivariogram and cross-semivariogram models; and (2) proportion of eachLC and LCC class (see Appendix). To calculate these parameters, the county was dividedinto 80 rectangular cells following existing quarter township boundaries, delineated prior tosettlement (Figures 2 and 3). The average size of a quarter township was approximately5250 · 5250 m, which divided by the 30-m pixel resolution, resulted in the 175 · 175 gridsize of the simulated surfaces. Semivariogram and cross-semivariogram models were fittedto each LC or LCC class and between each LC or LCC class within each quarter township.The 80 resulting range values for each variogram model were summarized by constructingstatistical distributions. The percentage of each LC or LCC class present within the quartertownships, and the correlation coefficient between each LC or LCC class, was also calcu-lated and summarized with statistical distributions. The resulting parameters specified foreach simulated surface are summarized below.

3.3.1. Initial map

Six semivariogram and cross-semivariogram models were fitted to determine the target levelof spatial autocorrelation for the time-1 classification probability surfaces. The range valueschosen for all six models corresponded to the median of each statistical distribution. Thetarget LC class percentages for the time-1 classified map were specified as:Developed = 0.35, Forest = 0.2, and Other = 0.45. Although a majority of quarter townships

Figure 2. Time-1 land-cover classified map forWashtenaw County, Michigan, with quarter-townshipboundaries overlaid.



were less than 10% developed (Figure 2), we simulated a quarter township with an inter-mediate percentage of developed land by reducing the percentage of the Other class. Thiswas done to create a classified map that had similar LC class proportions as the quartertownships occurring within the suburban regions of Washtenaw County. Additionally, thedegree of correlation between each LC class corresponded to the mean of the statisticaldistributions.

3.3.2. Maps of change

Two scenarios of LCC were simulated. The first scenario had LCC proportions thatreplicated mean percentages observed for each individual transition within WashtenawCounty (Wash County). The highest average proportion of change was from Forest toOther and from Other to Forest (7.3 and 7.4% respectively), with very little change observedbetween the classes: Forest and Developed (mean = 1%). The second scenario reflected asuburban landscape that was experiencing residential growth at a slightly higher rate thanaverage, agricultural abandonment (i.e., no change from Forest to Other), and saw forestgrowth at relatively high rates (Suburban). Each LCC scenario was further divided bysimulating two levels of overall change, i.e., either a low or high rate of overall change.The low and high rates of change were the same for both scenarios and corresponded to 17and 24%, respectively; percentages corresponded to the first and third quartiles of thestatistical distribution.

The first pattern of change (Section 3.1) required change-probability surfaces represent-ing linear trends. These surfaces were created for only the second scenario (Suburban).Transition probabilities for change from Forest to Developed and Other to Developeddeclined from east to west, while probabilities for change from Other to Forest declinedfrom west to east. The change surfaces were created from initial probability surfaces scaledfrom 0 to 1. A program was scripted to find the correct multipliers for each transition toachieve both the correct total and the individual amounts of change.

Figure 3. Land-cover change map illustrating persistence versus change for Washtenaw County,Michigan, with quarter-township boundaries overlaid.



The second and third patterns of change were created using simulated annealing(Section 3.1). All range value distributions for the 10 semivariogram and cross-semivariogram models were skewed to the right. Mean range values were chosen to reflecta relatively high degree of spatial continuity within and between change classes. Thisdecision ensured that the final change surfaces exhibited spatial autocorrelation over longerdistances than the error surfaces (Section 3.3.3) and mirrored the decision made in the twoLC class model (Burnicki et al. 2007). The correlations between LCC classes also corre-sponded to mean values.

To produce change probability surfaces that correlated higher probabilities of change tothe LC class boundaries of the ‘to’ class at time-1 (Section 3.1), a series of secondary mapswere created in which each pixel represented the distance to one of the three LC classes in thetime-1 classified map. Two correlation coefficients (r = -0.3 and -0.7) were chosen toquantify the relationship between the appropriate ‘distance-to-boundary’ map and the LCtransition of interest, following the values used in the two LC class model. A negativecorrelation was specified to associate high change probabilities with short distances to thetargeted LC class; for example, change to Forest is more likely near Forest time-1 classifica-tion boundaries. To summarize, ten sets of change-probability surfaces were generated, fourcorresponding to the Washtenaw County scenario and six corresponding to the Suburbanscenario.

3.3.3. Maps of error

The total percentage of error in the error-perturbed classified maps was set to 25%(i.e., -8.3% per LC class), equal to the misclassification rate specified in the two LC classmodel. The (cross-) semivariogram range values were chosen so that the error-probabilitysurfaces exhibited a smaller degree of spatial autocorrelation than both the change- andinitial-probability surfaces.

The creation of the temporally dependent error-probability surfaces required the speci-fication of a correlation coefficient (Section 3.1). The chosen correlation coefficients (r = 0.2and 0.4) mirrored those used in the two LC class model and reflected values typicallyobserved in time-series imagery (Carmel 2004). For the third pattern of error, whichconditioned time-1 error probabilities based on the distance from boundaries in the time-1classified map, a single correlation coefficient was specified (-0.3). In all, five sets of error-probability surfaces were produced for time-1 and time-2.

3.4. Model analysis

Thirty realizations for both the time-1 and time-2 error-probability surfaces were generated,resulting in 30 error-perturbed maps of change for each model run (Figure 1). In all, 50combinations of change and error patterns were modeled, corresponding to the 10 possiblepatterns of change and the 5 possible patterns of error.

The suite of error-perturbed maps of change was compared to the corresponding truemap of change by calculating two statistics: (1) overall percent correctly classified (PCC);and (2) user’s accuracy for correctly predicting the presence of a change. These statisticsprovided an overall measure of model performance (PCC) and an assessment of the model’sability to reliably predict change (User’s). It is important to separately assess the accuracy ofthe predicted LCCs due to the relative ease associated with predicting stationarity in LC overtime (Pontius et al. 2004). Each statistic was summarized across the 30 error-perturbed mapsof change by calculating the mean and standard deviation for the distribution of values.



Table1.

Meanandstandard

deviationof

overallpercentcorrectlyclassified

(PCC)values

summarizing50

combination

sof

patterns

ofchange

andpatterns

oferror.

Patternsof

error

Rando

matT1and

T2

Rando

matT1and

temporal

correlation=0.2T2

Rando

matT1and

tempo

ral

correlation=0.4T2

Low

correlationto

class

boun

daries

T1andtempo

ral

correlation=0.2

Low

correlationto

class

boun

daries

T1andtempo

ral

correlation=0.4

Rando

matT1and

tempo

ral

correlation=1T2

Patternsof

change

XTrend

swithlow%

ofchange

–subu

rban

51.08

50.41

49.86

48.37

46.21

60.09

0.00

40.00

40.00

30.01

70.01

50.01

6Xtrends

withhigh

%of

change

–subu

rban

51.05

50.39

49.9

48.25

46.57

56.28

0.00

40.00

40.00

30.01

70.01

50.00

9Rando

mwithlow%

ofchange

–WashCou

nty

48.54

48.48

48.68

45.28

43.09

58.7

0.00

50.00

30.00

20.01

70.01

50.01

4Rando

mwithhigh

%of

change

–WashCou

nty

47.23

46.64

46.3

43.9

39.4

54.59

0.00

50.00

40.00

30.01

60.01

40.01

3Rando

mwithlow%

ofchange

–subu

rban

47.93

47.37

46.94

45.36

43.45

56.39

0.00

40.00

50.00

30.01

60.01

40.01

4rand

omwithhigh

%of

change

–subu

rban

48.77

48.28

47.95

46.05

40.88

55.25

0.00

40.00

40.00

20.01

60.01

40.011

Low

Correlation

toclass

boun

daries

–Wash

Cou

nty

49.82

50.1

50.65

46.82

45.11

59.84

0.00

40.00

20.00

20.01

70.01

60.01

2

Highcorrelationto

class

boun

daries

–Wash

Cou

nty

50.3

50.74

51.42

47.43

45.79

60.11

0.00

40.00

20.00

20.01

70.01

50.011

Low

correlationto

class

boun

daries

–subu

rban

49.2

48.73

48.42

46.55

44.58

57.85

0.00

50.00

40.00

20.01

70.01

50.01

4Highcorrelationto

class

boun

daries

–subu

rban

49.33

48.87

48.58

46.64

44.71

57.5

0.00

40.00

40.00

20.01

70.01

50.02

9



4. Results

Unlike results presented elsewhere (Burnicki et al. 2007), increasing the temporal correla-tion between the time-1 and time-2 error-probability surfaces did not increase the overallaccuracy of the maps of change (Table 1). In fact, all but two patterns of change (correlatedchange for theWashtenaw County scenario) experienced a steady decline in the overall PCCvalues when time-2 patterns of error were temporally correlated to their time-1 counterparts.This was true when time-1 patterns of error were either random or correlated to theclassification boundaries at time-1. However, the decline in accuracy was greater in thelatter case. This overall result contradicts findings for the two LC class model, where overallPCC values significantly increased as the temporal correlation increased from 0 to 0.2 to 0.4.

Comparing results among the various patterns of change, it is evident that the scenariounder consideration played a role in determining the impact of increasing temporal correla-tion. TheWashtenaw County scenario, in general, experienced less severe declines in overallPCC values and had two patterns of change that showed significant increases in PCC valuesas compared to the Suburban scenario. Another notable observation was that overallaccuracies varied widely depending on the amount of total change and the underlyingpattern of change. Unlike the two LC class model, there were no overall trends in accuracyvalues among the patterns of change.

Similar results were observed for user’s accuracies in predicting change (Table 2), withdeclines in accuracy even more dramatic than PCC values. Every pattern of changeexperienced a decrease in user’s accuracy as the temporal correlation increased, regardlessof whether the time-1 pattern of error was random or correlated to classification boundaries.And unlike the results observed for PCC values, decreases in user’s accuracy values were notnecessarily greater for the set of error maps that had time-1 patterns of error correlated totime-1 classification boundaries.

The user’s accuracy values confirmed that the LCC scenario mediated the impact ofincreasing temporal correlation. The Washtenaw County scenario experienced greaterdeclines in user’s accuracy values than the Suburban scenario. This was opposite of thetrend observed for PCC values. The patterns of change for the Suburban scenario had higheruser’s accuracy values than their Washtenaw County counterparts for all patterns of error.Additionally, the percentage of total change simulated played a role in determining themagnitude of the user’s accuracy values, unlike PCC value results. All change patternsdisplaying a low percentage of change had significantly lower user’s accuracies than theirhigh change counterparts. User’s accuracy in correctly predicting an LC transition variedbased on the LCC scenario and the amount of change, and it also varied within each of thethree patterns of change.

5. Discussion and conclusions

The presence of a temporal correlation between time-1 and time-2 errors clearly did notimprove the accuracy of the resulting map of change when the categorical scale of the LCclassified maps was increased. It appears that the increased structural complexity within thetime series of maps effectively inhibits the impact of temporal dependence on the accuracy ofLCC maps. In the presence of complex interactions between LC classes and LC transitions,as well as the interactions between LC classes and their associated patterns of error, theeffects of a temporal relationship between error surfaces are severely attenuated. To deter-mine if temporal dependence could affect the accuracy of the maps of change, an extremecase in which errors were perfectly temporally correlated (r = 1) was simulated. The result



Table2.

Meanandstandard

deviationof

user’schange

accuracy

summarizing50

combination

sof

patterns

ofchange

andpatterns

oferror.

Patternsof

error

Rando

matT1and

T2

Rando

matT1and

tempo

ral

correlation=0.2T2

Rando

matT1and

tempo

ral

correlation=0.4T2

Low

correlationto

class

boun

daries

T1andtempo

ral

correlation=0.2

Low

correlationto

class

boun

daries

T1andtempo

ral

correlation=0.4

Rando

matT1and

tempo

ral

correlation=1T2

Patternsof

change

XTrend

swithlow%

ofchange

–subu

rban

21.06

18.6

15.26

19.89

18.55

26.52

0.00

50.011

0.00

90.01

0.00

90.01

3XTrend

swithhigh

%of

change

–subu

rban

36.25

35.32

34.14

34.21

32.88

44.73

0.00

50.00

90.00

80.01

20.01

20.00

6Rando

mwithlow%

ofchange

–WashCou

nty

22.23

20.99

19.39

21.37

20.82

26.95

0.00

30.00

60.00

50.00

60.00

60.00

8Rando

mwithhigh

%of

change

–WashCou

nty

31.84

30.87

29.71

30.64

19.57

38.16

0.00

40.00

60.00

50.00

80.00

50.00

7Rando

mwithlow%

ofchange

–subu

rban

26.68

26.04

25.27

25.77

24.67

34.68

0.00

40.00

70.00

60.01

0.00

90.00

8Rando

mwithhigh

%of

change

–subu

rban

37.56

37.4

37.09

36.12

23.8

470.00

40.00

60.00

60.01

20.00

80.00

8Low

correlationto

class

boun

daries

–Wash

Cou

nty

23.01

21.88

20.41

22.33

21.87

27.95

0.00

30.00

50.00

40.00

70.00

60.00

9

Highcorrelationto

class

boun

daries

–Wash

Cou

nty

23.27

22.25

20.69

22.69

22.23

28.26

0.00

40.00

50.00

40.00

60.00

60.00

7

Low

correlationto

class

boun

daries

–subu

rban

27.37

26.76

26.02

26.44

25.13

36.66

0.00

50.00

80.00

60.011

0.01

0.00

9Highcorrelationto

class

boun

daries

–subu

rban

27.45

26.85

26.14

26.45

25.17

36.83

0.00

50.00

80.00

60.011

0.01

0.00

9



for both overall PCC and user’s accuracy was a significant gain in accuracy (Tables 1 and 2).Overall PCC values increased by an average of 8.3% and user’s accuracies increased by anaverage of 7.1% under perfect temporal correlation. While temporal dependence can affectthe accuracy of a map of change, a correlation level not likely observed in real-world LCCanalyses is needed for maps depicting structurally complex landscapes.

A perplexing result was the continual decline in both PCC values and user’s accuracy asthe temporal correlation increased. Increasing the temporal dependence between error surfacesnot only had no effect on the accuracy of the maps of change, but also decreased the accuracyof the map of change, that is, PCCno . PCCr = 0.2 . PCCr = 0.4. The explanation may lie in themanner in which the time-2 error-probability surfaces were constructed. Time-2 patterns oferror were conditioned on time-1 patterns and did not incorporate the patterns of change. Forcases in which the time-1 error and change surface both had higher probabilities of occurrencein the same general locations, increasing the temporal correlation would ensure that theaddition of the change probability would be severely overwhelmed by the addition of error.

With regard to the second research question (Section 3), results showed that the simulatedpattern of change interacted with the simulated pattern of error to mediate the impact of temporaldependence. The interaction between the change and error pattern played a major role indetermining the accuracy associated with the resulting map of change. Differences wereobserved in both the direction and the magnitude of the change in accuracy values as temporaldependence increasedwhen comparing across the various patterns of change.Models attemptingto quantify the propagation of error in analyses of LCCmust consider, and explicitly incorporate,these change/error interactions to successfully predict the accuracy of a map of change.

The mediating effect of the pattern of change was clearly demonstrated when comparingresults for the LCC scenarios. Increasing the temporal correlation between patterns of errorhad the greatest impact on the Washtenaw County scenario when examining overall PCCvalues and the Suburban scenario when examining user’s accuracies. The mediating effectwas also evident when the temporal correlation was increased to 1. The average gain inuser’s accuracy was 8.4% for the patterns of change in the suburban landscape scenario, butwas only 5.2% for the patterns of change in the Washtenaw County scenario. The primarydifference between the scenarios was that change into the Other class was prevented for theSuburban scenario. Reducing the number of transitions within a pattern of change decreasedthe structural complexity of the time series of classified LCmaps leading to a smaller declinein user’s accuracy. This simplification of the model allowed the impact of the temporaldependence to become more apparent.

Similarly, user’s accuracies were higher under all patterns of error for the suburbanlandscape scenario. This meant that simplifying the LCC model by reducing the number oftransitions played a key role in determining the accuracy of the maps of change. Fewer LCtransitions were modeled with greater certainty regardless of the pattern of error considered.Thus, maps of LCC are more likely to succeed at mapping change when fewer transitions aremodeled and the patterns of change are relatively simple (i.e., fewer interactions betweenLCCs). Finally, it was clear that patterns exhibiting a low percentage of overall change hadsignificantly lower user accuracies than their high percentage counterparts. As the percentageof the mapped surface that experienced an LC transition increased significantly (17–24%), thecorrect prediction of change also increased (average of 12.7%). Clearly, increasing the numberof pixels undergoing an LC transition in the final maps of change led to a higher accuracy inpredicting their occurrence. This general relationship between increased accuracy withincreasing abundance usually holds when trying to map rare phenomena.

The results also lent further support to the usefulness of simulation approaches inmodeling the propagation of error in analyses of LCC. The complex interactions observed



between the patterns of change and patterns of error would be extremely difficult to modelanalytically, and the results for the overall PCC values indicate that the accuracy of the mapsof change do not necessarily follow recognizable trends. The simulation approach allowedus to (1) isolate and test the impact of increasing temporal dependence; and (2) create a seriesof 50 combinations of patterns of change and patterns of error, providing a wealth of data toexplore our research questions. However, one challenge encountered using the simulationapproach was the increased programming and computing time required to produce surfacesreflecting a series of constraints and complex spatiotemporal patterns. It took a desktopcomputer approximately 1.5 h to create 30 realizations of each error-probability surface forthe two LC class case, compared to the 192 h it took a cluster of computers to create 30realizations of each error-probability surface for the three LC class case.

Several conclusions can be drawn from this research. First, temporal dependencebetween patterns of error has a limited impact on analyses of LCC with realistic levelsof categorical detail. While temporal dependence in error increased the accuracy of a mapof change when the classified landscape was relatively simple, its impact was constrainedby the complex interactions present in maps comprising multiple LCs. Therefore, theresults of this research indicate that the simplifying assumption of temporal independencebetween classification errors within a time series may not significantly underestimate theoverall accuracy of a map of change when the maps are composed of several LC classes.Furthermore, results suggest that a simpler error-propagation model for post-classificationchange analysis may be appropriate when time-series classified maps contain multiple LCclasses. In other words, the added complexity of modeling temporal dependence to assessthe uncertainty of a map of change may not be necessary in all cases. However, theseconclusions need to be fully supported by empirical evidence before broad-scale recom-mendations can be made, given previous research that found a positive relationshipbetween temporal dependence and the overall accuracy of a map of change (Burnickiet al. 2007).

The second major conclusion of this work is that the accuracy of a map of changedepends on the interaction between the patterns of change and patterns of error occurringbetween time-series classified maps. The LCC scenario, amount of change, and the type ofchange occurring between the time-1 and time-2 classified maps all affected the resultingoverall PCC and user’s accuracy values. A greater understanding of these interactions isneeded to correctly quantify error propagation in analyses of LCC. This requires movingbeyond statistical summaries like overall PCC and user’s accuracy. Understanding howpatterns of change influence the impact of specified patterns of error requires an analysis ofthe spatial change patterns predicted by the error-perturbed change maps. Pattern-basedcomparisons of the error-perturbed maps of change with the true map of change will providea more meaningful description of the impact of this interaction.

This research highlighted many of the complex issues that affect our ability to provide acomprehensive and accurate assessment of the uncertainty in change-detection products.While the impact of temporal dependence between patterns of error was not as significant oras beneficial as the previous research indicated, it is clear that further research is needed tofully understand how error propagates through analyses of LCC.

Acknowledgements

This research was supported by NASA Headquarters under the Earth System Science (ESS)Fellowship Grant NGT5-05R-0000-0100 and a grant from NASA’s Land Cover Land Use Change(LCLUC) program (NAGS-11271). We would also like to thank the three anonymous reviewers fortheir helpful suggestions.



ReferencesAplin, P., 2006. On scales and dynamics in observing the environment. International Journal of

Remote Sensing, 27, 2123–2140.Arbia, G., Griffith, D., and Haining, R., 1998. Error propagation modelling in raster GIS: overlay

operations. International Journal of Geographic Information Science, 12, 145–167.Burnicki, A. C., Brown, D.G., and Goovaerts, P., 2007. Simulating error propagation in land-cover

change analysis: the implications of temporal dependence. Computers, Environment and UrbanSystems, 31, 282–302.

Cadenasso, M.L., Pickett, S.T.A., and Grove, J.M., 2006. Dimension of ecosystem complexity:heterogeneity, connectivity, and history. Ecological Complexity, 3, 1–12.

Carmel, Y., 2004. Characterizing location and classification error patterns in time-series thematic maps.IEEE Transactions on Geoscience and Remote Sensing, 1, 11–14.

Carmel, Y., Dean, D.J., and Flather, C.H., 2001. Combining location and classification error sources forestimating multi-temporal database accuracy. Photogrammetric Engineering & Remote Sensing,67, 865–872.

de Smith,M.J., Goodchild, M.F., and Longley, P.A., 2007.Geospatial analysis: a comprehensive guideto principles, techniques and software tools. 2nd ed. Leicester: Matador.

Deutsch, C.V. and Journel, A.G., 1992. GSLIB: Geostatistical software library and user’s guide. NewYork: Oxford University Press.

Duh, J.-D. and Brown, D.G., 2005. Generating prescribed patterns in landscape models. In: D.J.Maguire, M. Batty, and M.F. Goodchild, eds. GIS, spatial analysis, and modeling. Redlands, CA:ESRI Press, 423–444.

Eastman, J.R., 1999. Idrisi 32: user’s guide. Worcester, MA: Clark University.Gomez-Hernandez, J.J., 2006. Complexity. Ground Water, 44, 782–785.Goodchild, M.F., Guoging, S., and Shiren, Y., 1992. Development and test of an error model for

categorical data. International Journal of Geographic Information Systems, 6, 87–104.Goovaerts, P., 1997.Geostatistics for natural resources evaluation. New York: Oxford University Press.Heuvelink, G.B.M. and Burrough, P.A., 1993. Error propagation in cartographic modelling using

Boolean logic and continuous classification. International Journal of Geographic InformationSystems, 7, 231–246.

Ju, J., Gopal, S., and Kolaczyk, E., 2005. On the choice of spatial and categorical scale in remotesensing land cover classification. Remote Sensing of Environment, 96, 62–77.

Kirkpatrick, S., Gelatt, C.D., and Vecchi, M.P., 1983. Optimization by simulated annealing. Science,37, 671–680.

Liu, H. and Zhou, Q., 2004. Accuracy analysis of remote sensing change detection by rule-basedrationality evaluation with post-classification comparison. International Journal of RemoteSensing, 25, 1037–1050.

Mertens, B. and Lambin, E.F., 2000. Land-cover-change trajectories in southern Cameroon. Annals ofthe Association of American Geographers, 90, 467–494.

National Research Council: Committee on Grand Challenges in Environmental Sciences, OversightCommission for the Committee on Grand Challenges in Environmental Sciences, 2001. Grandchallenges in environmental sciences. Washington, DC: National Academy Press.

Nunes, C. andAuge, J.I. (eds.), 1999. Land-use and land-cover change (LUCC) implementation strategy,IGBP global change. Stockholm, Sweden: IGBP Secretariat, Report No. 48 and HDPReport No. 10.

Ojima, D., et al., 2005. Global land project science plan and implementation strategy, IGBP globalchange. Stockholm, Sweden: IGBP Secretariat, Report No. 53 and HDP Report No. 19.

Overmars, K.P. and Verburg, P.H., 2005. Analysis of land use drivers at the watershed and householdlevel: linking two paradigms at the Philippine forest fringe. International Journal of GeographicInformation Science, 19, 125–152.

Pontius, R.G., Shusas, E., and McEachern, M., 2004. Detecting important categorical land changeswhile accounting for persistence. Agriculture, Ecosystems and Environment, 101, 251–268.

Schneider, L.C. and Pontius, R.G., 2001. Modeling land-use change in the Ipswich watershed,Massachusetts, USA. Agriculture, Ecosystems and Environment, 85, 83–94.

Serneels, S. and Lambin, E.F., 2001. Proximate causes of land-use change in Narok District, Kenya: aspatial statistical model. Agriculture, Ecosystems and Environment, 85, 65–81.

Sohl, T.L., Gallant, A.L., and Loveland, T.R., 2004. The characteristics and interpretability of landsurface change and implications for project design. Photogrammetric Engineering & RemoteSensing, 70, 439–448.



van Oort, P.A.J., 2005. Improving land cover change estimates by accounting for classification errors.International Journal of Remote Sensing, 26, 3009–3024.

Veldkamp, A. and Fresco, L.O., 1996. CLUE-CR: an integrated multi-scale model to simulate land usechange scenarios in Costa Rica. Ecological Modeling, 91, 231–248.

Wagener, T. and Kollat, J., 2007. Numerical and visual evaluation of hydrological and environmentalmodels using theMonte Carlo analysis toolbox.EnvironmentalModeling&Software, 22, 1021–1033.

Wu, J. and David, J.L., 2002a. Modeling complex ecological systems: an introduction. EcologicalModeling, 153, 1–6.

Wu, J. and David, J.L., 2002b. A spatially explicit hierarchical approach to modeling complexecological systems: theory and applications. Ecological Modeling, 153, 7–26.

AppendixThe ability of simulated annealing to generate a series of map realizations that reproduce user-definedspatial patterns results from the incorporation of statistics into the components of the objective function.The objective function is the weightedmathematical combination of these objective-function components:

OðiÞ ¼XC

c¼1 wcOcðiÞ (A1)

where O(i) is the objective function, Oc(i) is an objective-function component, wc is the weightassigned to component c, C is the total number of objective-function components included in thefunction, and i is the ith perturbation of the simulated annealing procedure. To produce maps withspecific spatial and temporal patterns, five different objective functions were created based on a set offive objective-function components.

The first component incorporated the series of semivariogram and cross-semivariogram modelsdefining the level of spatial continuity within the time series of classified maps. For the initialclassification-probability surfaces and error-probability surfaces, three semivariogram models andthree cross-semivariogram models were required to define the level of spatial autocorrelation withineach LC class and the level of spatial dependence between each pair of LC classes. For the change-probability surfaces, four semivariogram models and six cross-semivariogram models were required.This first component measured the overall difference between each target (detailed below) andexperimental semivariogram or cross-semivariogram value as

O1ðiÞ ¼XL

l¼1

XK

k¼1

XK

k0¼k½�

kk0 ðhlÞ � �ðiÞstd kk0 ðhlÞ�

2

�2kk0ðhlÞ

(A2)

where �kk0 ðhlÞ is the target semivariance or cross-semivariance at lag hl, �ðiÞstd kk0ðhlÞ is the standardized

experimental semivariance or cross-semivariance (described below) at lag hl calculated for the ithperturbation, K is the number of semivariogram models included in the objective-function statistic(i.e., 3 or 4), and L is the number of spatial lags (Goovaerts 1997). Dividing by �2kk0 ðhlÞ ensured thatmore weight was given to semivariogram reproduction for smaller lags. In the above equation, �

ðiÞstd kk0 ðhlÞ

was calculated by comparing the magnitude of the (cross-) semivariance value to the correlationcoefficient between the LC classes of interest. This was done to impose the reproduction of the shapeof the (cross-) semivariogram and not its sill. The sill of the (cross-) semivariogram,which is related to thevariance of each map under consideration, is not known a priori due to the perturbation process.Therefore, the standardized semivariogram values were calculated in the objective-function as follows:

�ðiÞstd kk0ðhlÞ ¼

�ðiÞkk0 ðhlÞrkk0

(A3)

where rkk0 equals the correlation coefficient between the LC classes of interest if k � k0 or 1 if k = k0

(Goovaerts 1997). The key parameter specified for the variogram models was the range, the distancebeyond which pixels no longer exhibit spatial autocorrelation. Specifying the range for each modelcontrolled the overall degree of spatial autocorrelation within and between each LC class or LCtransition.



The second objective-function component incorporated the series of histograms defining thedistribution of values within each probability surface. In order to keep the methodology as similar aspossible between the two and three LC class simulation models, the final probability surfaces needed tofollow a normal distribution. This component measured the discrepancy between the normal distribu-tion and the experimental distribution, after standardization to zero mean and unit variance, summedacross all simulated probability surfaces (e.g., 3 or 4). It was calculated as the following differencebetween a series of quantiles discretizing both the target and experimental distributions:

O2ðiÞ ¼XK

k¼1

XL

l¼1qlk � qðiÞlk�� pl · ð1� pl Þ

(A4)

where qlk is the target pl-quantile of the standard normal cumulative frequency distribution, qðiÞlk is theexperimental pl-quantile of the cumulative frequency distribution calculated for the ith perturbation, Lis the number of quantiles considered (e.g., 99 percentiles), and K is the number of probability surfacesincluded in the objective function (i.e., 3 or 4; Goovaerts 1997). Dividing by pl · (1 – pl) in the aboveequation ensured that more weight was given to the reproduction of the tails of the distribution (i.e.,quantiles corresponding to cumulative probabilities pl close to 0 or 1).

The third and fourth objective-function components concerned the reproduction of user-definedproportions of LC categories. The final time series of classified maps, both error free and errorperturbed, replicated the following proportions: (1) percentage of pixels assigned to each LC class attime-1; (2) percentage of total pixels undergoing any LC transition between time-1 and time-2; (3)percentage of pixels experiencing each of the individual LC transitions between time-1 and time-2; and(4) percentage of total pixels experiencing an error in classification at time-1 and time-2. The totalpercentage of error assigned to the classified maps was equally divided among the three LC classes, soadditional error percentages for each individual LC class were not necessary. The third objective-function component measured the difference between the target global proportion and the experimentalglobal proportion when considering either the total amount of error or the total amount of changedefined as:

O3ðiÞ ¼ s� sðiÞ�� (A5)

where s is the target global proportion and s(i) is the experimental global proportion for the ithperturbation. The fourth objective-function component measured the sum of the differences betweenthe target individual proportions and the experimental individual proportions when considering eitherthe proportions of each LC class at time-1 or the proportions of each LC transition defined as:

O4ðiÞ ¼XK

k¼1 tk � tðiÞk��

�� (A6)

where tk is the target individual proportion, tðiÞ

k is the experimental individual proportion for the ithperturbation, and K is the number of LC classes or transitions included in the objective function (i.e., 3or 4).

The fifth objective-function component controlled the reproduction of the linear correlationcoefficient between the simulated probability surfaces and secondary variables. This constraint wasnecessary to create change- and error-probability surfaces that were correlated to LC classificationboundaries and error-probability surfaces that were temporally dependent. This component measuredthe sum of the differences between the target and experimental correlation coefficients for each map setdefined as:

O5ðiÞ ¼XK

k¼1 rXYk ð0Þ � rðiÞXYk ð0Þ��

�� (A7)

where rXYk ð0Þ is the target linear correlation between the previously generated map surface (X) and theprobability surface being currently simulated (Y), rðiÞXYk ð0Þ is the experimental linear correlationcalculated for the ith perturbation, and K is the number of probability surfaces included in theobjective-function statistic (i.e., 3 or 4; Goovaerts 1997).



Based on components O1 through O5, five different objective functions were specified to produceeach set of probability surfaces simulated for the overall model (Table A1). More weight was assignedto either the reproduction of the (cross-) semivariogram models or the global or individual proportions.This ensured that the level of spatial continuity and percentage of the map experiencing an LCtransition or error replicated user-defined values. Components with relatively small weights (,0.1)were components that were easily reproduced.

Table 3. Objective functions defined for each simulated surface, detailing objective-function com-ponents and component weights.

Simulated surfaceNo. of

components Components Component weights

Temporally independent time-1and time-2 error-probabilitysurfaces

3 (Cross-) Semivariogrammodels, histograms, globalproportion

0.3, 0.3, 0.4

Time-1 classification-probability surfaces

3 (Cross-) Semivariogrammodels, histograms,individual proportions

0.9, 0.09, 0.01

Change-probability surfaces 4 (Cross-) Semivariogrammodels, histograms, globalproportion, individualproportions

0.25, 0.25, 0.15, 0.35

Temporally dependent time-2error-probability surfacesand error-probabilitysurfaces correlated toland-cover boundaries

4 (Cross-) Semivariogrammodels, histograms, globalproportion, correlationcoefficients

0.25, 0.25, 0.25, 0.25

Change-probability surfacescorrelated to land-coverboundaries

5 (Cross-) Semivariogrammodels, histograms, globalproportion, individualproportions, correlationcoefficients

0.2, 0.2, 0.1, 0.3, 0.2



Propagating error in land-cover-change analyses: impact of ...danbrown/papers/aburnick_ijgis.pdf ·...

Documents

Transcript of Propagating error in land-cover-change analyses: impact of ...danbrown/papers/aburnick_ijgis.pdf ·...