Input Data Quality Report - British Columbiaa100.gov.bc.ca/.../documents/r1498/pem_4039_idq02... ·...
Transcript of Input Data Quality Report - British Columbiaa100.gov.bc.ca/.../documents/r1498/pem_4039_idq02... ·...
Input Data Quality Report
Review of Abitibi PEM IDQ Report
Submitted to the Ministry of Sustainable Resource Management
By David E. Moon, CDT Core Decision Technologies Inc.
I
REVIEW OF ABITIBI PEM IDQ REPORT .......................................................................................................... 1
0. EXECUTIVE SUMMARY................................................................................................................................. 1
1. INTRODUCTION................................................................................................................................................ 1 1.1. BACKGROUND ................................................................................................................................................ 1 1.2. RATIONALE FOR THE REVIEW........................................................................................................................ 2 1.3. SCOPE OF THE REVIEW................................................................................................................................... 2 1.4. ORGANIZATION OF THE REVIEW .................................................................................................................... 2
1.4.1. Introduction ........................................................................................................................................... 2 1.4.2. The review proper ................................................................................................................................. 2 1.4.3. Conclusions ........................................................................................................................................... 3
1.5. ELEMENTS OF THE REVIEW............................................................................................................................ 3 1.5.1. Organization and Presentation ............................................................................................................ 3 1.5.2. Field Sampling and Field Data ............................................................................................................ 3 1.5.3. Input Data Layers ................................................................................................................................. 3 1.5.4. Knowledge Base.................................................................................................................................... 7
2. REVIEW................................................................................................................................................................ 7 2.1. ORGANIZATION AND PRESENTATION ............................................................................................................ 7 2.2. FIELD SAMPLING AND FIELD DATA............................................................................................................... 7 2.3. INPUT DATA LAYERS ..................................................................................................................................... 8
2.3.1. TRIM Data............................................................................................................................................. 8 2.3.2. Biogeoclimatic Ecosystem Classifcation ...........................................................................................10 2.3.3. Satellite Imagery..................................................................................................................................12 2.3.4. Terrain Stability Mapping ..................................................................................................................14 2.3.5. Forest Cover........................................................................................................................................16
2.4. KNOWLEDGE BASE AND ALGORITHM REQUIREMENTS..............................................................................18 2.4.1. Entities .................................................................................................................................................18 2.4.2. Knowledge Base Attribute Compilation.............................................................................................18 2.4.3. Belief Matrices ....................................................................................................................................18 2.4.4. Knowledge Base Validation................................................................................................................19
3. CONCLUSIONS ................................................................................................................................................19
1
0. Executive summary
It must be noted that the Input Data Quality standards put forth in the Predictive Eco-system Mapping Inventory Standard were, by design, non-prescriptive and flexible enough for PEM practitioners to meet the intent of the standard without being forced to follow rigid formats, formulas, or procedures. The hope was that the standard would encourage critical evaluation of input data by PEM practitioners and encourage its re-porting in a clear, concise, and complete manner. It is therefore difficult to provide a pass/fail evaluation for IDQ reports.
This said, the Abitibi PEM report does not meet fully the letter or the intent of the PEM Inventory Standard for Input Data Quality Reporting.
1. Documentation of metadata for the input data layers while meeting some of the letter of the standard, was incomplete. Of special concern was that the history or pedigree of Forest Cover map revisions, base map conversion, and retrofit pro-cedures were ignored.
2. Documentation of the attribute extraction/derivation procedures was incomplete. In many cases it was not possible to determine the logic or process of extrac-tion/derivation and it was not possible to evaluate the results.
3. Documentation of quality control and quality assurance was inadequate and while the use of field plots established to support development of the knowledge base as a test for the quality of primary or derived input data was commendable, the lack of documentation on sampling strategy and design, lack of documentation on attribute derivation and extraction procedures, lack of documentation on the evaluation of the nature of mapping entities, map entities (spatial accuracy and thematic resolution and accuracy) limited the utility of the information.
4. Despite relatively high error values in those attributes tested against field data, there was no discussion of the sensitivity of ecosystem prediction to errors in the input data. Failure to provide a simple error analysis identifying the impact of indi-vidual and cumulative attribute errors on predictive error significantly reduces the value of this data.
5. The knowledge base was not documented. The attributes used in the knowledge base were not fully defined, and the interpretive logic of the attributes used in the predictive process were not described.
6. The knowledge base was not evaluated against an independent data set.
The short comings make an adequate evaluation of the quality and application of the input data impossible. Furthermore, the failure to adequately document attribute ex-traction methods, attribute data definitions, and knowledge base structure, values and logic makes assessment of the results impossible and precludes incorporation into a provincial knowledge base.
1
1. Introduction
1.1. Background
The Ministry of Sustainable Resource Management commissioned this report as part of a project to evaluate, against the PEM Inventory Standard, Input Data Quality Re-ports submitted in support of PEM projects. Based on these reviews, the contractor was to prepare and report on a framework and format for future Input Data Quality Reports.
The specification for a PEM standard, against which the IDQ Report is evaluated arose, from four preceding works commissioned by the Terrestrial Ecosystem Map-ping Alternatives Task Force of the Resources Inventory Committee (RIC). These re-ports contained important background information and concepts which for the sake of brevity was not included in the standard but was referenced in the expectation that it would be read. These reports were:
1. Towards the Establishment of a Predictive Ecosystem Mapping Standard: A White Paper, by Keith Jones, R. Keith Jones & Associates; Del Meidinger, BC Ministry of Forests, Research Branch; Dave Clark, BC Ministry of Environment Lands and Parks, Resources Inventory Branch; and Fern Schultz, BC Ministry of Forests, Resources Inventory Branch.
2. Problem Analysis on Data Quality Assessment Issues by Dr. David Moon, CDT–Core Decision Technologies Inc.
3. Situation Analysis for Knowledge-Based Systems by Dr. David Moon, CDT–Core Decision Technologies Inc.
4. Problem Analysis on Reliability, Quality Control and Validation of Predictive Eco-system Mapping (PEM) by Dr. Richard Sims and Jeff Matheson, R.A. Sims & As-sociates.
The standard drew upon four additional reports:
1. Specifications for PEM, version 2.1
2. Mapping entities, draft report
3. Protocol for Quality Assurance and Accuracy Assessment of Ecosystem Maps
4. A Method for Large Scale Biogeoclimatic Mapping in British Columbia.
Because PEM was new and largely untested, the original report was somewhat gen-eral and emphasized documentation of data and procedure rather than prescription. The intent was to ensure that a qualified PEM practitioner would be able to evaluate the quality of the input data and procedures used in the production of a PEM map based on the documentation.
In reality, the standard and its referenced reports proved an intimidating, nebulous, time consuming, and open-ended basis for developing IDQ Reports. The variability in quality and format of the reviewed reports is reflective of this reality.
2
1.2. Rationale for the Review
The review evaluates the submitted Input Data Quality Report in terms of both the let-ter and the intent of the Predictive Ecosystem Mapping Inventory Standard for Input Data Quality Assessment. It evaluates organization, completeness, and adequacy of documentation from the perspective of an inventory specialist evaluating the quality of the input data and the adequacy of the procedures used to produce the PEM. It does not evaluate the adequacy of the report as a contract deliverable and it does not evaluate the quality of the data or procedures used.
The original PEM Inventory Standard was neither prescriptive nor detailed and many of the items in the standard were recommendations rather than requirements. In addi-tion, the original standard had not been tested. Finally, the original standard was not presented or intended as a template for reporting. It is therefore not surprising that submissions to date have been highly variable in terms of both format and complete-ness. The review format has therefore attempted to bring a standard format to the re-view and by extension provide an initial template for future IDQ reports.
1.3. Scope of the Review
The review deals only with input data quality reporting, documentation of input data layer processing, and documentation of the knowledge base. It does not evaluate the adequacy of the data or procedures used, and it does not evaluate the conclusions presented in the report. The review evaluates only whether or not there is sufficient in-formation presented for a knowledgeable PEM practitioner to evaluate the adequacy or efficacy of the input data and approach used.
1.4. Organization of the review
The organization of this review attempts to evaluate and report the elements of the PEM process in the logical order in which they are preformed and represents a devia-tion from the order of presentation in the standard.
1.4.1. Introduction
The introduction to the review presents:
1. Background to the review.
2. Rationale for the review.
3. The organization of the review (this section).
1.4.2. The review proper
The review proper has the following major elements:
1. The organization, presentation, and completeness of the report.
2. The individual input data layers.
3. The derivation/extraction and compilation of knowledge base attributes.
3
4. The knowledge base, and validation of the knowledge base.
1.4.3. Conclusions
A general conclusion as to how well the IDQ report meets the letter and intent of the standard.
1.5. Elements of the Review
1.5.1. Organization and Presentation
The review will evaluate the report for organization and presentation; for ease of use and retrieval of information; for format and presentation of report elements including supporting data, use of tables and figures; and for appropriate summary and conclu-sions (specifically are they present and supported by the content of the report).
1.5.2. Field Sampling and Field Data
The standard has no requirement for field sampling however the intent of the standard requires the following documentation for both third party field data and field data col-lected by the contractor in support of the project:
The development of the knowledge base is based in part on field data where the at-tributes being used in the predictive process are collected at locations where the site-series is known. These sites may be collected by the PEM contractor in support of the project, by a third party in support of non-PEM activities, or the contractor may choose a combination of both. Whatever option is chosen, the IDQ report should include the following documentation.
1. The kind, frequency, and distribution of field samples.
2. The method of sample selection (e.g., random, stratified random, selective/modal, etc.) and if stratified or selective the stratification or selection criteria.
3. Quality control and Quality Assurance protocols applied to the field data.
1.5.3. Input Data Layers
Compilation Metadata
Input map metadata provides basic information to ensure that the nature and limita-tions of input maps are understood before creating PEM input data layers. Most im-portant of these are the original base map, projection, and methods used for compila-tion of the map and the history and nature of changes to the original map. Some con-sultants are unaware that TRIM data is compiled and distributed in Albers projection while forest cover and other maps use Universal Transverse Mercator. If obtained in digital form the difference will be obvious when processing is attempted. However if paper maps are digitized by the contractor and the difference is not known, it is prob-able the map will be digitized using the wrong projection. This difference could pro-duce significant positional discrepancies in larger project areas. The metadata should include the process used to match projections, the magnitude of spatial shifts (rubber
4
sheeting) required during the conversion, the verification of thematic boundary accu-racy (if any) and the nature of any thematic content changes during the update.
Spatial Quality of Input Data
Spatial Data Integrity
Reconciliation to TRIM
The spatial integrity metadata should include the method used to match projections between input maps or confirmation of that the conversion was done. The standard established a procedure for evaluating the consistency of the input data map with TRIM features, particularly hydrography. The standard is inadequate with any maps “retrofitted” to TRIM features (e.g., forest cover) unless the procedures and degree of spatial adjustment required to get conformance of TRIM features is recorded.
Spatial integrity:
The standard requires a measure of error for lines which fail to join at map sheet boundaries and for label consistency for polygons crossing map sheet boundaries.
Spatial Accuracy of thematic boundaries:
A more appropriate measure of spatial data integrity is the measurement of geo-graphic coordinates for non-TRIM features such as cultural features which are com-mon to both maps (e.g., cutblocks and roads).
Thematic Data Input Quality
The standard requires that for all input data layers, the suitability of the input data be evaluated for use in the PEM project. Only the spatial accuracy of TRIM data may be taken as given and even this refers only to TRIM features (including digital elevation points). It does not refer to thematic accuracy or to the interpolation of or derivation of landscape attributes such as slope, aspect, shape, slope position et cetera from TRIM data. The resolution of the TRIM data may not be able to adequately portray the scale of landscape characteristics important to the PEM process. This will be especially true in low relief and complex terrain such as hummocky kame and kettle topography, gla-cial fluvial deposits and others.
Thematic Accuracy
This element, while not required, refers to the validation of thematic attributes by com-paring attributes from know geospatial coordinates to those attributes displayed at those coordinates on the map.
Map Entity Suitability
Mapping Concepts
The standard requires an evaluation for appropriateness of and interpretive issues re-lated to the nature of the mapping entities (things mapped e.g., terrain units) and the nature of the map entities (delineations on the map e.g., simple versus complex composition). The review will look for demonstrated understanding of the nature of the
5
mapping concepts used in the input data layers, for an understanding of the interpre-tive issues related to their use for predictive ecosystem mapping, and for documenta-tion of how the issues were dealt with. Specifically, the review will look for documenta-tion evaluating the resolution/complexity of the mapping and map entities, boundary precision for thematic maps, and boundary accuracy for thematic maps. While not specifically required in the standard, these issues are relevant to predictive ecosystem mapping and should be evaluated by competent practitioners.
Resolution/Complexity In the case of thematic maps, resolution refers to the level of detail attached to the mapping entities used in the input data layer and complexity refers to the number and type of mapping individuals used as components in the map entity. With digital eleva-tion data, resolution refers to the precision of the elevation measurement and the spacing of the elevation data points.
Boundary Precision Boundary precision refers the sharpness of the transition between adjacent map units and/or the confidence with which the mapper has drawn the line. Both will influence the accuracy, precision, and interpretation of any spatial overlay product.
Boundary Accuracy Boundary accuracy refers to how closely the delineation of a boundary on the map corresponds to its true location on the ground.
Quality Control
The PEM standard did not require reference to Standard Operating Procedures (SOPs) or quality control protocols although elements of QC protocols are inherent in some of the metadata requirements. Despite this, the intent of the PEM Inventory Standard would be best met with a combination of SOPs and quality control proce-dures to ensure that the SOPs were applied appropriately.
Standard Operating Procedures
Standard operating procedures refer to procedures that have been extensively docu-mented and tested to ensure that they produce consistent, reliable results when fol-lowed. They may be widely accepted and adopted, as with ISO standards, or they may simply be internal procedures that have been well documented and tested. The principal behind a SOP is that once tested and documented, it is only necessary to confirm that the procedure was followed and quality assurance testing can be signifi-cantly reduced.
Quality Control Protocols
Quality control protocols refer to application of standard procedures used to ensure the quality and integrity of either data or products, or it can refer to a series of quality assurance procedures to ensure that acceptable levels of quality are being met. In the case of SOPs, the procedure is assumed to produce acceptable quality and the intent is to ensure that the standard procedures have been applied. Examples include pro-cedures to ensure that field data is correctly transferred to digital format or colour map edits to ensure that thematic data is correctly attached to polygons. Quality control protocols should consist of defined procedures for the production or editing of data, in-
6
formation, or intermediate or final products and a formal sign-off to confirm that the procedure was implemented as documented.
Quality Assurance
Unlike quality control, quality assurance consists of actual testing of the interim and fi-nal products to ensure that acceptable levels of accuracy or reliability are being achieved. When coupled with standard operating procedures, the level and frequency of quality assurance can be significantly reduced.
Knowledge base validation was the only Quality Assurance procedure required for In-put Data Quality Reporting.
Meta Data
The PEM standard establishes minimum levels of documentation and meta-data re-quired to evaluate the quality of input data, predictive procedures, and output products of PEM. The meta-data specified below meet three needs.
1. They provide sufficient information about the nature of the input entities, input data, predictive procedures, and output products for a qualified PEM practitioner to understand the limitations of these items for PEM applications.
2. Their compilation by the PEM practitioner ensures that the practitioner has re-searched the input data and adequately documented the procedures and output products.
3. A longer-term goal of the PEM standard is the eventual integration of PEM/TEM data, information, and knowledge into a single logical data model and repository. The task of integrating TEM with PEM is beyond the scope of this standard but this section will provide the documentation and meta-data necessary to construct such a repository.
Meta data are required in the following areas.
1. Input map source, base, compilation, and map entities.
2. Input map processing including attribute extraction/derivation.
3. Knowledge base and knowledge processing algorithms.
Thematic Compilation and Derivation
Attribute Collection/Derivation and Compilation
The derivation and/or compilation of thematic data layers requires the implementation of a set of procedures to either the original data capture and compilation or the evaluation and processing of third party input data. The standard requires documenta-tion of these procedures either by reference to appended descriptions or to published documents.
7
Wherever possible these references should be to tested Standard Operating Proce-dures described above and should reference the quality control procedures used to confirm the quality of data or the application of tested Standard Operating Procedures.
Attribute Definition
The standard requires documentation of the attributes used in the PEM process. Ele-ments of this documentation are its definition, domain, scale, and units of measure. The standard provides detailed specifications that should be followed.
1.5.4. Knowledge Base
Documentation of the knowledge base requires the following.
1. The PEM entities being predicted.
2. The attributes being used to predict the entities.
3. The method of compilation of attributes for the spatial entities being predicted.
4. The logic, values, and algorithms used to make the prediction.
The standard provides detailed specifications for documentation of the knowledge base.
2. Review
2.1. Organization and Presentation
The organization is simple and straight forward. It provides a very brief introduction and two major sections, Input Data Spatial Assessment and Input Data Thematic As-sessment. Tables are inserted at appropriate locations in the text but appendixes re-ferred to in the text were not submitted with the report nor were referenced materials on methods and procedures. Additional, useful materials were submitted as unrefer-enced spreadsheets.
2.2. Field Sampling and Field Data
The report identified that 650 combined full ecosystem, ground inspection, and visual inspection plots collected following Field Manual for Describing Terrestrial Ecosys-tems, 1998. Although referred to, no sampling plan or strategy was presented. The reader cannot therefore evaluate the method of sample selection or the kind, fre-quency, and distribution of samples. There was no discussion of third party field plots. As a result, the reader has no information on to support that the contractor was com-petent to evaluate the data used, that it was evaluated, or that the data and evaluation were adequate.
8
2.3. Input Data Layers
2.3.1. TRIM Data
Compilation Meta Data
Citation – Not provided.
Consultant/Department – Provided
Compilation Scale - Provided
Publication scale – NA
Period of compilation – Not provided
Original base map and projection used for compilation – not provided
Modification History including base map and projection conversions and the basis, method, and nature of thematic updates – not provided
Current base map and projection used for compilation – not provided
Projection – Not provided.
Mapping entities – NA
Map entities – NA.
Entity relationships – NA
Spatial Quality of Input Data
Reconcilliation to TRIM Base
Not Applicable.
Spatial Integrity (input processing)
Edge matching – NA
Positional Accuracy of Thematic Boundaries
NA
Thematic Quality of Input Data
Thematic Accuracy
NA
9
Map Entity Suitability
Resolution/Complexity The report makes reference to the resolution of data points being between 45 and 70 m and notes the probable loss of topographic features less than 1 ha in extent. There is no formal evaluation of resolution appropriate to landscape but the contractor did test TRIM derived landscape attributes against field plots.
Accuracy/Precision There was no discussion of the suitability of the accuracy or precision of TRIM data for derived attributes. Only the final attribute classes were tested against field data.
Quality Control
Standard Operating Procedures – NA
Quality Control Protocols – NA
Quality Assurance
Thematic Compilation and Derivation
The information included in this section is essential to the creation of a provincial knowledge base.
Attribute Extraction/Derivation
Quality Control / Quality Assurance
Standard operating procedures - none referenced
Quality Control Protocols – none referenced
In-house Procedures Documentation – none provided
Quality Assurance The contractor tested derived attributes against ground data. Classes (specified ranges of attribute values) of TRIM derived attributes were tested against field plots. The following information is required to fully evaluate the test.
Sampling Strategy/Plan – no strategy or plan provided.
Accuracy/precision of field plot location – none provided.
Sensitivity Analysis – no sensitivity of interpretive error to data error was reported.
Attribute Definitions
[24]. Definition and description – Description provided but no data definition.
[25]. Attribute code – Code provided
[26]. Method – No method of determination provided
10
[27]. Unit of measure – provided where appropriate
[28]. Entity/Relationship described –The report does not define whether the attrib-ute describes the average for the polygon, dominant value of the polygon, or the value of the dominant component of the polygon.
Numeric Data (real or continuous variable data)
[29]. Precision – not provided
[30]. Statistic – not provided
Categorical Data (discontinuous and unranked)
[31]. Valid values – provided
Ordered Classes
[32]. Rank and limits – rank provided
2.3.2. Biogeoclimatic Ecosystem Classifcation
Compilation Meta Data
Citation – None.
Consultant/Department – provided
Publication scale – provided.
Compilation scale – not provided
Period of compilation – not provided
Original base map and projection used for compilation – not provided
Modification History including base map and projection conversions and the basis, method, and nature of thematic updates – not provided
Current base map and projection used for compilation– not provided
Projection – not provided
Mapping entities – not provided
Map entities – not provided.
Entity relationships – not provided
11
Spatial Quality of Input Data
Reconcilliation to TRIM Base
The IDQ report does not make clear whether any existing BEC maps were actually incorporated into localized BEC coverage. It appears that the project used two 1:50K MoF BEC map sheets (93N and 94C) and that the remainder of the area was mapped using rule sets applied to digital elevation data.
[18]. Adjusted control feature shift – not reported.
Spatial Integrity (input processing)
Edge matching – not reported.
[14]. Edge matching error – not reported
[15]. Attribute/Label matching – not reported
[16]. Raster size – NA.
Positional Accuracy of Thematic Boundaries
Not reported
Thematic Quality of Input Data
Thematic Accuracy
Discussed in general terms but there are no reported tests against the available field data.
Map Entity Suitability
Mapping Concepts – not discussed
Resolution/Complexity The contractor offers an opinion but it is not supported by data or reference.
Boundary Precision Not discussed
Boundary Accuracy Unsupported opinion of the contractor is offered..
12
Quality Control
Standard Operating Procedures – none reported
Quality Control Protocols - None are reported
Quality Assurance
None is reported
Thematic Compilation and Derivation
The information included in this section is essential to the creation of a provincial knowledge base.
Attribute Extraction/Derivation
[21]. Attribute derivataion – The report refers to “Final Methods Report for Localizing Biogeoclimatic Linework for the Aiken, Tenakihi, Upper Osilinka, Lower Osilinka, and Swannel Landscape Units”, Atticus 2000) but this report was not appended or avail-able to the reviewer.
2.3.3. Satellite Imagery
Compilation Meta Data
Citation – None.
Consultant/Department – provided
Publication scale – NA.
Period of compilation – provided
Original base map and projection used for compilation - NA
Modification History including base map and projection conversions and the basis, method, and nature of thematic updates - NA
Base map and projection used for compilation – Registered to TRIM
Mapping entities – NA.
Map entities – Na.
Entity relationships – NA.
Spatial Quality of Input Data
Reconcilliation to TRIM Base
[18]. Adjusted control feature shift – not reported.
13
Spatial Integrity (input processing)
Edge matching – not reported.
[14]. Edge matching error – not reported.
[15]. Attribute/Label matching – not reported.
[16]. Raster size – not reported.
Thematic Quality of Input Data
Thematic Accuracy
None reported despite availability of field data.
Map Entity Suitability
Resolution/Complexity Limits to classification including size and type of unit discriminated are discussed in general terms. The nature of the resulting mapping entities are discussed in general terms.
Boundary Precision Issues of probable boundary precision are not discussed..
Boundary Accuracy Issues of probable classification boundary accuracy are not discussed..
Quality Control
Standard Operating Procedures – none are reported
Quality Control Protocols – none are reported.
Quality Assurance
None is reported.
Thematic Compilation and Derivation
The information included in this section is essential to the creation of a provincial knowledge base.
Attribute Extraction/Derivation
[21]. Attribute extraction – no methods are reported, referenced, or discussed for the classification of image data into vegetation types..
14
Attribute Definitions
[24]. Definition and description – none provided except a brief description of the code.
[25]. Attribute code – provided
[26]. Method of Measurement – NA
[27]. Unit of measure – NA
[28]. Entity/Relationship described – a brief description of the map units identifies the vegetation class as a complex of broad vegetation types.
Categorical Data (discontinuous and unranked)
[31]. Valid values – provided
2.3.4. Terrain Stability Mapping
Compilation Meta Data
Citation – none provided
Consultant/Department – provided
Publication scale – Terrain Survey Intensity Level is reported
Period of compilation – provided
Original base map and projection used for compilation – not provided
Modification History including base map and projection conversions and the basis, method, and nature of thematic updates - not provided
Current base map and projection – Not provided
Mapping entities – provincial Terrain Standard is referenced.
Map entities – provincial Terrain Standard is referenced..
Entity relationships – Not porvided
Spatial Quality of Input Data
Reconcilliation to TRIM Base
[18]. Adjusted control feature shift – not reported
Spatial Integrity (input processing)
Edge matching
15
[14]. Edge matching error – not reported
[15]. Attribute/Label matching – not reported
[16]. Raster size – NA.
Positional Accuracy of Thematic Boundaries
Not reported
Thematic Quality of Input Data
Thematic Accuracy
Terrain stability maps were correlated by provincial authority no independent evalua-tion was done despite the availability of 650 geo-referenced plots.
Map Entity Suitability
Mapping Concepts
No demonstrated understanding of the nature of Terrain Stability map units or the im-plications of map unit concepts for PEM production.
Resolution/Complexity Not discussed or reported.
Boundary Precision Not discussed or reported..
Boundary Accuracy Not discussed or reported
Quality Control
Standard Operating Procedures –
The work is completed in accordance with the Standards for Digital Terrain Data Cap-ture in British Columbia (June 1998), and the Guidelines and Standards for Terrain Mapping in British Columbia (RIC, 1996), and Terrain Classification System for British Columbia (Howes and Kenk 1997)
Quality Control Protocols
The survey intensity level which includes frequency of ground inspection is reported for each Terrain Stability Map sheet used but there is no report of ground inspection procedures.
16
Quality Assurance
The contractor provided no independent quality assurance despite the availability of 650 ground inspection forms.
Thematic Compilation and Derivation
The information included in this section is essential to the creation of a provincial knowledge base.
Attribute Extraction/Derivation
[21]. Attribute extraction – the report identifies neither specific terrain features used nor their method of extraction for use in the knowledge tables.
Attribute Definitions
Attributes extracted from Terrain Stability maps are neither reported nor defined.
2.3.5. Forest Cover
Compilation Meta Data
Citation – Specify a reference to a formal, published source of the data, if available.
Consultant/Department – Specify the public or private sector organization(s) re-sponsible for collecting, compiling, and maintaining the data and an appropriate con-tact within the organization(s).
Compilation scale – not provided
Publication scale – Specify the original publication scale of the inventory.
Period of compilation – not provided
Original base map and projection used for compilation – not provided
Modification History including base map and projection conversions and the basis, method, and nature of thematic updates – not provided.
Current base map and projection – provided
Mapping entities – not provided.
Map entities – not provided.
Entity relationships – not provided.
Spatial Quality of Input Data
Reconcilliation to TRIM Base
[18]. Adjusted control feature shift – not provided.
17
Spatial Integrity (input processing)
Edge matching
[14]. Edge matching error – not provided
[15]. Attribute/Label matching – provided
[16]. Raster size – NA.
Positional Accuracy of Thematic Boundaries
Addressed in general terms
Thematic Quality of Input Data
Thematic Accuracy
Leading species, secondary species, height class, and age class were tested using a confusion matrix against field plots. Only brief summary tables were available in the report. Details of the “confusion” matrix were submitted as unreferenced spreadsheets after additional information was requiested.
Map Entity Suitability
Mapping Concepts – not discussed
Resolution/Complexity – not discussed
Boundary Precision – not discussed
Boundary Accuracy – not discussed
Quality Control
Standard Operating Procedures – none provided
Quality Control Protocols – none provided
Quality Assurance
Six hundred and fifty ground samples were collected to provide training data for the knowledge base and for testing the accuracy of thematic information in the input maps. Leading species, secondary species, height class, and age class were tested using a confusion matrix against field plots. The information identified below is neces-sary to evaluate the efficacy of quality assurance procedure.
Sampling Strategy/Plan – no strategy or plan provided.
Accuracy/precision of field plot location – none provided.
Sensitivity Analysis – no sensitivity of interpretive error to data error was reported.
18
Thematic Compilation and Derivation
The information included in this section is essential to the creation of a provincial knowledge base.
Attribute Extraction/Derivation
[21]. Attribute extraction – method not provided.
Attribute Definitions
[24]. Definition and description – not provided
[25]. Attribute code – provided
[26]. Method – not provided.
[27]. Unit of measure – not provided
[28]. Entity/Relationship described – not provided
Numeric Data (real or continuous variable data)
[29]. Precision – not provided.
[30]. Statistic – not provided
Categorical Data (discontinuous and unranked)
[31]. Valid values provided
Ordered Classes
[32]. Rank and limits - provided
2.4. Knowledge Base and Algorithm Requirements
2.4.1. Entities
Entity definitions were provided as an unreferenced spreadsheet. The spread sheet lacked both annotation and a data dictionary.
2.4.2. Knowledge Base Attribute Compilation
[19]. Mapping entity cross product correlation – No information provided
[20]. Sliver adjustment – No information provided.
2.4.3. Belief Matrices
No information provided.
Attribute Values and Condition Sets
19
No information provided
2.4.4. Knowledge Base Validation
No information provided.
3. Conclusions
This report does not meet fully the letter nor the intent of the PEM Inventory Standard for Input Data Quality Reporting.
1. Documentation of metadata for the input data layers was incomplete. Of special concern was that the history or pedigree of Forest Cover map revisions, base map conversion, and retrofit procedures were ignored. Also of concern was that there was no explicit recognition of projection discrepancies between TRIM and other input data layers.
2. Documentation of the attribute extraction/derivation procedures was incomplete.
3. Documentation of quality control and quality assurance was inadequate and while the use of field plots established to support development of the knowledge base as a test for the quality of primary or derived input data was commendable, failure to document sampling strategy and design, failure to document attribute deriva-tion and extraction procedures, failure to evaluate the nature of mapping entities, map entities (spatial accuracy and thematic resolution and accuracy) limited the utility of the information.
4. Despite relatively high error values in those attributes tested against field data, there was no discussion of the sensitivity of ecosystem prediction to errors in the input data. Failure to provide a simple error analysis identifying the impact of indi-vidual and cumulative attribute errors on predictive error significantly reduces the value of this data.
5. The knowledge base was not documented. The attributes used in the knowledge base were not fully defined, the interpretive logic of the attributes used in the pre-dictive process were not described, and the nature of the mapping and map units being predicted were neither defined nor described.
6. The knowledge base was not evaluated against an independent data set.
The short comings make evaluation of the quality and application of the input data im-possible. Furthermore, the failure to adequately document attribute extraction meth-ods, attribute data definitions, and knowledge base structure, values and logic makes assessment of the results impossible and precludes incorporation into a provincial knowledge base.