Aljoscha Requardt, University of Hamburg NEFIS WP5: Evaluation NEFIS – Final Symposium, Ispra...

Al josc ha R

eq uard t, Univ ersi ty o f H

a mb urg

NEFIS WP5: Evaluation NEFIS – Final Symposium, Ispra 28-30th of June 2005

Metadata

Data Rights and Data Rights Management

User Needs and User Expectations - VTK

The Evaluation of the NEFIS Project focused on:

Intro

Al josc ha R


a mb urg


Evaluation based on two approaches:

• metadata and metadata guidelines

• data rights and data rights management

• user needs and user expectations - VTK

• metadata and metadata guidelines

Prof. Dr. Keith Rennolls (Greenwich), Dr. Jarmo Saarikko (Metla)

• keyword list and thesaurus

Renate Prüller (IUFRO), Gillian Petrokofsky (CAB International)

• data rights and data rights management

Andreas Schuck (EFI), Tim Green (EFI)

• EFIS – an electronic system for periodic reporting

Prof. Dr. Michael Köhl (UHH), Aljoscha Requardt (UHH)

Standardised questionnaires

Expert Statements

Intro

Al josc ha R


a mb urg


1. Elaboration of metadata records a step-wise iterative process

2. Questionnaire “Metadata and Metadata Guidelines” - evaluate experienced applicability and usability

Evaluation Metadata

4. Expert Statements- metadata and metadata guidelines- keyword list and thesaurus

Metadata

3. Questionnaire results represented and discussed at the Hamburg Meeting

- preliminary records according previous schema - metadata schema developed further to its final version (D3, D5)

- revise preliminary records make adjustments according new schema

Al josc ha R


a mb urg


Metadata

Questionnaire “Metadata and Metadata Guidelines”

Knowledge and experiences of the respondent

Workload

Metadata Guidelines

NEFIS Metadata Schema

Specific Metadata Elements and Refinements

Metadata Functionality and Structure

Future Perspectives

Al josc ha R


a mb urg


0

1

2

3

4

5

6

7

excellent higher thanaverage

average lower thanaverage

noexperience

(nr)

0

1

2

3

4

5

6

7

excellent higher thanaverage

average lower thanaverage

no experience

(nr)

Partners knowledge and experience of elaborating metadata

Partners knowledge and experience using metadata for data source retrieval

Metadata General

Are the NEFIS metadata the first contact you have had with metadata? 79% Yes

Al josc ha R


a mb urg


0

2

4

6

8

10

12

very high high acceptable low very low

(nr)

Level of complexity to prepare and enter metadata:

Do you consider the workload to enter metadata records as:

Metadata Workload

Time Investment - most partners between 5-10 hours in total

29% = complex71% = acceptable

Al josc ha R


a mb urg


0

1

2

3

4

5

6

very easy easy sufficient difficult very difficult

(nr)

Are the guidelines easy to understand?

Metadata Guidelines

Would a metadata tutorial or training course be helpful and appropriate?

71% Yes

Al josc ha R


a mb urg


0

2

4

6

8

10

12

14

1. T

itle

2. C

reat

or

3. S

ubje

ct

4. D

escr

iptio

n

5. P

ublis

her

6. C

ontri

buto

r

7. D

ate

8. T

ype

9. F

orm

at

10. I

dent

ifier

11. S

ourc

e

12. L

angu

age

13. R

elat

ion

14. C

over

age

15. R

ight

s

16. A

udie

nce

(nr)

Metadata records without problems

Metadata Schema

Al josc ha R


a mb urg


0%

20%

40%

60%

80%

100%

high

acceptable

low

Workload: time to prepare and enter required information

Metadata Schema

Al josc ha R


a mb urg


0%

20%

40%

60%

80%

100%

not relevant low relevance relevant high relevance

Relevance - data and resource documentation

Metadata Schema

Al josc ha R


a mb urg


0%

20%

40%

60%

80%

100%

not relevant low relevance relevant high relevance

Relevance - data retrieval

Metadata Schema

Al josc ha R


a mb urg


• Subject

Metadata Specific Issues

• Type

• Format

• Coverage

• Quality Report

DatasetGeoreferenced

Reference System - encoding scheme for georeferenced data

Point / Box encoding scheme

• Audience

Al josc ha R


a mb urg


0

1

2

3

4

5

6

7

8

fully addressed addressed tosome extent -

sufficient

addressed tosome extent -

deficient

not addressed

(nr)

Named elements which address “Quality” within the DCMI schema:

Creator, Description, Publisher, Coverage, Source, Date, Audience

"Quality" addressed within DCMI elements, refinements and encoding schemes?

Metadata Quality Report

Al josc ha R


a mb urg


0

1

2

3

4

5

6

7

8

9

10

very important important interesting,but not

necessary

not important

(nr)

Relative value of a “Quality Report“

“Quality Report“ under the element “Descriptions“?


Al josc ha R


a mb urg


0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Collection Mandate

Availability of datacollection and data

processing guidelines

Defintions

Sampling methods

Explanatory notes

very important important interesting, but not necessary not important

Value of listed options to describe and structure “Quality Report"


Al josc ha R


a mb urg


• Which of these quantitative measures are relevant and appropriate for which type of data?

• it could become very complicated and time consuming to collect such information in a standardised way.

Value of quantitative measures of quality (standard error, sample size, sampling unit, resampling for measurement control)

0

1

2

3

4

5

6

appropiate to some extent -sufficient

to some extent -deficient

not appropiate

(nr)


Al josc ha R


a mb urg


• sometimes difficult to group the term under a theme

• very time consuming to go through the lists … more structure/ hierarchy would be helpful.

Can the dataset be appropriately described by using the NEFIS terms?

0

1

2

3

4

5

6

7

appropiate to some extent -sufficient

to some extent -deficient

not appropiate

(nr)

Metadata Subject

Al josc ha R


a mb urg


0

1

2

3

4

5

6

7

8

very important important interesting, butnot necessary

not important

(nr)

• it is the way to improve the term lists

• very important during the development phase

•…it allows the provider to demonstrate which additional terms are seen necessary to describe the particular resource more accurately

How important is the option to add "Nominated Terms"

Metadata Subject

Al josc ha R


a mb urg


Metadata Element

1. Title

2. Creator

3. Subject

4. Description

5. Publisher

6. Contributor

7. Date

8. Type

9. Format

10. Identifier

11. Source

12. Language

13. Relation

14. Coverage

15. Rights

16. Audience

Schema - Resume

= easy

?? = difficult (hot spots)

? = acceptable

?

?

?

?

?

??

??

?

??

?

Metadata

Applicability and Usability

Al josc ha R


a mb urg


ConclusionMetadata

Subject, Description and Format caused problems for most partners

low experience by some partners

complicated structure and defined procedures

Obstacles within the challenging tasks as the development of keyword lists

as well as the quality report could have been clearly reflected by the results

of the metadata evaluation

NEFIS metadata schema is applicable in its current form

from the perspective as a data provider – dataset documentation

from the perspective as a data user – data resource retrieval

Al josc ha R


a mb urg


Data Rights

Data provider has to state what data are accessible and for whom, and which type of

data rights and which levels of access should be assigned to which kind of data.

Data provider should have options to assign different levels of access to individual

datasets in order to protect sensitive data by restricting access to them.

Prerequisite to guarantee that legal restrictions

to data access will be satisfied

Al josc ha R


a mb urg


Why to talk about Data Rights?

• within the NEFIS project

• beyond the time of NEFIS project

“Data Rights”, does it concern the actual data? Should not all data presented in NEFIS be “public”?

NEFIS tables are available for free. On the other hand, same information is part of an on-line system to which you have a restricted access. How should we handle this?

Data Rights

Data rights provoked several discussions

Clarify Data Rights

Al josc ha R


a mb urg


Are these “drop down“ options appropriate for your data set?

Which of the "drop down" options (if any) are relevant to your data set?

0 5 10 15 20 25 30

Public

Limited

Restricted - need to purchase ($, €)

Restricted -need to be member

Restricted - need to register

nr - dataset

Data Rights

95% Yes But option “Public” needs further refinements

Al josc ha R


a mb urg


54%

9%

37%the data provider

EFI (Projectcoordinator)

another authorisedbody

Who is responsible for data access management?

Data Rights

68%

3%

29%all data on a server bythe data provider

only certain data on aserver by the dataprovider

all data on a centralserver

Which form of data storage do you prefer for your data set?

Al josc ha R


a mb urg


Would you like to provide more data?

Data Rights

Are there any obstacles to provide additional data?

27%

73%

no

yes

33%

67%

no

yes

cost for the provider !

cost for the user

technical limitations

unclear data rights management

Al josc ha R


a mb urg


Data Rights

Partners demands for further cooperation and data provision

• link data easily to the system without complicating and restricting data management and maintenance

• clear definitions and standards for data format and data content

Financial demands

Technical demands and demands on data management

• capacities for time and personnel investments

Al josc ha R


a mb urg


2. anonymous comments by partners regarding VTK display and analysis options

1. VTK Workshop

3. Test CD

4. Questionnaire “User Needs and User Expectations”

Evaluation „VTK & Information needs“

5. Expert Statement (UHH) “EFIS - an electronic system for periodic reporting”


Al josc ha R


a mb urg



Questionnaire “User Needs and User Expectations”

The respondet

Usability

Performances

VTK features

Tutorial and Help Functions

Improvements

Future Potential

Al josc ha R


a mb urg



19 answered questionnaires

Expirences of useing the VTK = average / less than average

Respondent spent 7 hours using the VTK (5 hours for one explicit dataset)

The respondet

A - familiar with the NEFIS project and the provided dataset

B - only familiar with the provided dataset itself

C - someone who is not familiar with the NEFIS project and the provided dataset, but has some broad knowledge in using GIS applications and statistical programs

• Total

• GIS Knowledge

• Statistical knowledge

Questionnaire evaluated according:

Al josc ha R


a mb urg



0

1

2

3

4

5

6

7

8

9

very easy easy average difficult very difficult

(nr)

How easy is it to find applicable and reasonable functions to

analyse and visualise data which are of explicit interest?

• functions are atractive but it is not easy to find out how to use them and select the one that is needed

• VTK examples are an important addition to get a more transparant view which of the features are useful for which kind of analysis

Al josc ha R


a mb urg



0

2

4

6

8

10

12

very satisfied satisfied average unsatisfied veryunsatisfied

(nr)

Respondent meaning about the VTK menu

• the menu should be developed more simple and informative

• some better “getting started directions” would be very helpful (essential even).

Al josc ha R


a mb urg


• the produced maps are interactive, including traditional GIS interactivity and visual manipulations

• outputs are available as data tables and thematic maps with different techniques for selected geographical area

• it allows production of statistical graphics (box plots, dot plots, scatter plots, etc.) that are dynamically linked to maps


The VTK incorporates automated map design

This could imply that expert knowledge on cartography or GIS is not required

• not necessariliy knowledge in GIS but some knowledge in cartography and data presentation

• the user should be familiar with the data itself

58%

42%agree

disagree

Al josc ha R


a mb urg



0

1

2

3

4

5

6

7

8

1 toogeneric/

superficial

2 3 4 5 toodetailed

(nr)

Consideration of options to analyse available datasets

Consideration of options to visualise available datasets

0

2

4

6

8

10

12

1 toogeneric/

superficial

2 3 4 5 toodetailed

(nr)

Al josc ha R


a mb urg



Ability to recall explicit information from displayed formats

Mapped data

Data plots

Tabular data

high

Figures satisfactory

Units unsatisfactory Lack in the actual dataset itself !!!

Al josc ha R


a mb urg



VTK offers many features, tools and options to analyse and to visualise data

the feature should be applicable to the dataset

the user should understand meaning and relevance of the feature

the user should be able to recall relevant information after using a suitable feature

the appearance of the outputs should be satisfactory for the user

no essential functionality in that feature should be missing

Al josc ha R


a mb urg



0 2 4 6 8 10 12 14 16

thematic mapping

choropleth mapping

chart mapping

visualisation and exploring data over time

dynamic classification

non-cartographical data displays

multiple display and display linking (like: plots or parallelcoordinates)

dynamic query

propagation of object classes

table view and table lens

exploring data regarding their spatial distribution

analysis of grid (raster) data

multi criteria decision making

VTK

feat

ure

(nr)

Feature is not applicable to the dataset Iwas looking

Problem of understanding meaning andrelevance of the feature

Problem of understanding analysed andvisualised data by using this feature

Appearance of output is not satisfactory

Missing function within the feature

Other Problem…

Al josc ha R


a mb urg



Most respondents considered all listed applications as important or as interesting

but the respondent was not always able to test some explicit applications, mainly

due to the lack of help functions and tutorial documents

further tutorial documents and help functions can be seen as

a useful and important asset !

Al josc ha R


a mb urg



Time investment to understand and use VTK features high

Complexitiy to understand available VTK features high

Suggestion:

Develop a set of tutorial tailored at different user groups and level of expertise

Al josc ha R


a mb urg



Basic Version – new users

Advanced Version – experienced users

Develop a VTK for two different groups of users

Reporting

- reporting schemes

- reporting query options

Reduce the set of VTK features or Add new features ???

Suggestion:

Key: User Friendliness

- improved interfaces- improved maneuvering through features- improved guidance for getting started

Al josc ha R


a mb urg



0

1

2

3

4

5

6

very good good suff icient deficient inadequate

(nr)

VTK functional and applicable for your purposes as a data provider?

Al josc ha R


a mb urg



User Backgrounds and Expectations are different

Simplified self explanatory output features

Clearly structured user guidelines

Easy to follow help functions

Although some applications could be developed further or improved, the VTK and

its features provide a solid basis to meet various tasks of a full working EFIS.

Define the purposes of the system – Who are the potential users?

Differentiate the use and applications

Reporting most important purpose - guarantee that VTK outputs not ambiguous in their statement

- contact data user – data provider

Al josc ha R


a mb urg


Thank you

Aljoscha Requardt, University of Hamburg NEFIS WP5: Evaluation NEFIS – Final Symposium, Ispra...

Documents

Transcript of Aljoscha Requardt, University of Hamburg NEFIS WP5: Evaluation NEFIS – Final Symposium, Ispra...