IS 29500 validator and document...

© Fraunhofer Institut FOKUS, Berlin, 2009 1

IS 29500 validator and document library

A project about document interoperability

DII Event, May 18th, London, UK

Jan Ziesing, Ucheoma Ishionwu, Klaus-Peter Eckert

CC Elan, Fraunhofer FOKUS

http://www.fokus.fraunhofer.de/


Agenda

Introduction and Overview

Document test library

Classification dimensions

Definition and evaluation of document categories

Introduction to appropriate technologies

IS29500 validator

Conformance in IS29500

Existing validation tools and outlook

Combining lib and validator

Architecture: Elevator pitch

Conclusion



OpenDocument Format – Office Open XML Translation

ISO/IEC 26300 (OASIS ODF 1.0) and ISO/IEC 29500 eq. ECMA-376 V2 (Office

Open XML) are open formats for saving and exchanging documents, spreadsheets

and presentations.

The possibility to translate between the two standards is of fundamental

importance for document format interoperability. Thus ISO SC34 has formed the

working group WG5 to solve problems which have arisen as a result of this.

FhI FOKUS supports these activities and has organized a group of experts that

has published a technical report at DIN that has been submitted to ISO as PDTR

29166 early 2009

Currently FhI FOKUS is acting as an editor on PDTR 29166 to improve the

document considering input from other member bodies.



Document Interoperability Lab in a Nutshell

Fraunhofer FOKUS has initiated its new Document Interoperability Lab

http://www.fokus.fraunhofer.de/go/is29500-validation

The lab is going to test the validity of documents regarding the document-standard

ISO/IEC 29500 known as Office Open XML.

As part of the lab activities Fraunhofer FOKUS is going to initiate the open source

development of a document validator.

A document and test library consisting of valid template documents will be offered.



Document Test Library in a Nutshell

Main goals:

Create a suite of documents for testing and verifying IS29500 interoperability

Create and maintain web site for document repository

Up-/download documents

Assign category tags to documents

Maintain category tags

Validate documents considering IS29500 and document categories

Announcement and info about the document library is available at

http://www.fokus.fraunhofer.de/go/is29500-validation

The document library is available at www.is29500-interoperability.com



IS29500 Validator in a Nutshell

Main goals:

Develop a tool to validate schema, package conventions and „semantic“

properties and categories of documents considering IS29500

Organize an open source project that develops and maintains a validation tool

Define compliance rules

Evaluate compliance rules

Produce comprehensible error messages

Accessible for users via the document library web page

Project will be lead and hosted by FhI FOKUS at

http://developer.berlios.de/projects/ooxml-validator/

Start of the project mid 2009

Utilization of XML technology

Participants are welcome



berlios.de



developer.berlios.de - Developer Portal



developer.berlios.de/account/register.php – Regestry Form



developer.berlios.de/projects/ooxml-validator/ - The Project Page



Agenda






IS29500 validator





Conclusion



Basic Functionality of the Document Test Library

Retrieving documents

Download valid document templates and examples considering document

categories

Documents are categorized along different dimensions

Adding and categorizing documents

Validate document categories

Add a new document categorized using existing characteristics

Maintaining categories and characteristics

Define, change or remove tags/characteristics describing application-domains,

categories or sub-categories

Maintain definition of document categories

Maintain categorization of documents



Screenshot of the Document Library



Classification



Proposed Document Classification

The document test lib is classifying documents along three dimension:

1. Document types and conformance classes – this is a mandatory property of

all valid documents

2. Application domain – extensible attributes like private, corporate, academic,

package

3. Document categories – extensible set of characteristics or tags

Formal definition using document characteristics

Definition and evaluation of associated rules using XML representation, e.g.

Schematron

Consideration of argument mapping techniques for weaker but comprehensible

asignments of document categories

Sample categories are Memo, Flyer, PhotoBook, Poster, Letter etc.



Classification Hierarchy



Example dokument (Jans Photobook)



Classification Hierarchy – Classification example

Presentation

Document

Domain

Academic

Domain

Corporate

Domain

Private

Category PBPhotoBook

Sub categoryPB + video

Sub categoryPB - video

Category KN

Key Note

CategoryNT

Note Taking

Categoryn.n.

…

Domain

Package

Word-processing

DocumentSpreadsheet Document



Search Paths



Sample Search Path (test document)

Document-Type

Word–Processing Spreadsheet Presentation

Document-Type


Private Corporate Academic Package

Document-Type



Poster Memo Meeting Note Research-Paper Report Proposal

Resume Résums Essay Book PhotoBook Letter Data

Analysis Outline Form Keynote Budgeting-Application …

Select Type

Select Domain

Select Category



Alternative Search Paths (template document)

Category

PhotoBook…


Word-Processing Spreadsheet Presentation

Application-Domain


Word-Processing Spreadsheet Presentation

Poster Memo Meeting Note Research-Paper Report Proposal

Resume Résums Essay Book PhotoBook Letter Data

Analysis Outline Form Keynote Budgeting-Application …

Select Type

Select Domain

Select Category

Select Category

Select Type

Select Domain



? Open Questions ?

The questions we are now addressing:

How can document characteristics be defined?

How can document characteristics be evaluated?

How can document categories (profiles) be defined?

How can document categories (profiles) be verified?



Categories and Characterization



Sample Categories for Presentation Documents

Presentation Category Description

Conference

Presentation

( aka “keynote” )

Presentations that are highly customized, well polished and

represent something larger than the particular individual.

Pitch Book Presentations that are very information dense and often used in the

financial services sector.

Sales Presentation Presentations that are designed to be given to a potential customer

with the goal of distinguishing the presenter/presenter’s

organization from the competition.

Photo Book Presentations that are photo-heavy and provide simple slide shows

with photos as the primary content.

Informal Presentations that are either oriented around informal meetings

such as internal team meetings ( e.g., a simple agenda slide to team

status ).

Note Taking Presentations that are created on the fly and used to capture

meeting minutes or brainstorming results.

Self-Running Presentations that are generally used as kiosks ( both interactive and

not ) as well as self-paced training and education.

Signage Single-slide presentations generally used for creating certificates,

posters and signs.

Based on Microsoft

contribution

Following Example



Characterization of Properties

Characteristic Keynote Pitch

Book

Sales

Presen-

tation

Photo

Book

Infor-

mal

Note

Taking

Self-

Running Signage

# of slides 15-25 25+ 15-25 25+ 1-15 1-15 15-25 1

Info Density Low High Medium Low Low Low Medium Low

Animations Yes No Yes No No No Yes No

Transitions Yes No Yes Yes No No Yes No

Interactivity No No No No No No Yes No

Charts Yes Yes Yes No Yes No Yes No

Tables Yes Yes Yes No Yes No Yes No

OLE Objects No Yes No No Yes No No No

Audio &

Video

Yes No Yes Yes No No Yes No

Photos Yes Yes Yes Yes Yes No Yes Yes

Ink No No No No No Yes No No

Slide Timings No No Yes Yes No No Yes No

Slide Looping No No No Yes No No Yes No

Based on Microsoft

contribution



Characterization of Properties (cont.)


Book

Sales

Presen-

tation

Photo

Book

Infor-

mal

Note

Taking

Self-

Running Signage

Text Size Normal

+

Small Normal

+

Normal

+

Normal

+

Normal Normal

+

Large

Text Quantity Average Lots Average Little Average Lots Average Little

Focus Projec-

tor

Print Both Projec-

tor

Both Both Projec-

tor

Print

Data Facing No Yes Yes No No No No No

Custom Slide

Shows

No No Yes No No No Yes No

Customized

Design

Yes Yes Yes No No No Yes No

Customized

Layout/

Template

Yes Yes Yes No No No Yes No

Formatting

Quantity

Lots Lots Lots Limited Average Limited Lots Lots

Similar tables are

available for word

processing and

spreadsheet docs

Based on Microsoft

contribution



Possible solutions and technologies for the

categorization



Possible solutions for categorization

User-based categorization

Users define domains, categories and sub-categories

Automatic categorization

The categorization is done by analyzing the documents and using rules for the

categorization Demo will follow

Argumentation-based categorization

IT-Assistance for users when they are classifying their documents. Demo

will follow



Schematron Demo


Book

Sales

Presen-

tation

Photo

Book

Infor-

mal

Note

Taking

Self-

Running Signage

# of slides 15-25 25+ 15-25 25+ 1-15 1-15 15-25 1








Audio &

Video






Based on Microsoft

contribution


Evaluation of Document Characteristics using Schematron

<sch:schema xmlns:sch ="http://www.ascc.net/xml/schematron" >

<sch:ns uri="http://schemas.openxmlformats.org/package/2006/relationships" prefix="rel"/>



<sch:pattern name="Doc_Sub_Type" id="DST">

<sch:rule context="/">

<sch:assert test="count(rel:Relationships/rel:Relationship[@Type =

'http://schemas.openxmlformats.org/officeDocument/2006/relationships/slide']) > 25">

Document is not a PHOTOBOOK!</sch:assert>

<sch:report test="count(rel:Relationships/rel:Relationship[@Type =

'http://schemas.openxmlformats.org/officeDocument/2006/relationships/slide']) > 25">

Document is either a PHOTOBOOK or PITCHBOOK (since: it has more than 25 slides)!</sch:report>

</sch:rule>

</sch:pattern>

</sch:schema>


Characteristics of a Presentation Sub Type

<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">

<Relationship Id="rId8"

Type="http://schemas.openxmlformats.org/officeDocument/2006/relationships/slide"

Target="slides/slide7.xml"/>







…..

…..

…..

<Relationships>

Evaluate Schematron rule

on “presentation.xml.rels“ property file

Demo will follow !



Verification of Document Categories - Argumentation Techniques

It has to be discussed if a “black and white logic” is adequate for the given problem

domain. Probably an argument mapping application like Carneades would be a

better choice. Carneades provides for example tools supporting a variety of

argumentation tasks, including:

Argument mapping and visualization

Argument evaluation, applying proof standards and respecting the distribution

of the burden of proof

Argument construction from defeasible rules, precedent cases, ontologies and

testimonial evidence

Argument interchange in XML, using the Legal Knowledge Interchange Format

(LKIF)

Following such an approach the membership of a document to a category can be

verified, even if not all rules are completely satisfied.

http://carneades.berlios.de/



Carneades Demo


Book

Sales

Presen-

tation

Photo

Book

Infor-

mal

Note

Taking

Self-

Running Signage

# of slides 15-25 25+ 15-25 25+ 1-15 1-15 15-25 1








Audio &

Video






Based on Microsoft

contribution



Questions to the Document Library

DocLib Discussion

What do you think of the initial classification schema?

What other use cases next to testing do you see for the library (i.e. templates)?

What do you think of the classifying approaches? Does automation or

argumentation offer a benefit?

What can we do for the provisioning of valid document templates?



Agenda






IS29500 validator





Conclusion



Validation types, project scope and

standard



Validation Types

vs. Standard

Schema or Syntax Validation

Check that all parts of the document package are available and that the XML

documents comply to their schema.

Semantic or Intention Validation

Check that all parts of the standard that have not been specified in a formal way

or are formalized in the XML schema are valid.

vs. Mark Up

Custom Validation

Check that user defined extensions are valid

vs. Features

Feature Validation

Check that features are expressed in a valid way



Validation Types

vs. Standard

Schema or Syntax Validation

Check that all parts of the document package are available and that the XML

documents comply to their schema

Semantic or Intention Validation

Check that all parts of the standard that have not been specified in a formal way

or are formalized in the XML schema are valid.

This includes a kind of semantic validation.

Scope of the Validator Project



Conformance Classes in IS29500

The document types stored in the document library are not simply word

processing, spreadsheet, and presentation documents. Instead, they have to be

compliant to the conformance classes:

1. Document type:

WML Strict or WML Transitional

SML Strict or SML Transitional

PML Strict or PML Transitional

2. Package conventions: OPC

3. Markup compatibility: MCE



Document Validation – the IS29500 Standard

Part 1 of ISO/IEC 29500 defines the following document conformance classes

WML Strict, if the document is a conforming document of category Word

processing that conforms to the strict schema and does not include any features

from Part 4 of ISO/IEC 29500.

SML Strict, if the document is a conforming document of category Spreadsheet

that conforms to the strict schema and does not include any features from Part 4 of

ISO/IEC 29500.

PML Strict, if the document is a conforming document of category Presentation

that conforms to the strict schema and does not include any features from Part 4 of

ISO/IEC 29500.



Document Validation – the IS29500 Standard

Part 2 of ISO/IEC 29500 specifies a set of conventions that are used by Office

Open XML documents to define the structure and functionality of a package in

terms of a package model and a physical model. Part 2 defines the conformance

class OPC - Open Packaging Conventions.

Part 3 of ISO/IEC 29500 describes a set of XML elements and attributes whose

purpose is to collectively enable producers to explicitly guide consumers in their

handling of any XML elements and attributes not understood by the consumer.

Part 3 defines the conformance class MCE - Markup Compatibility and

Extensibility

Part 4 of ISO/IEC 29500 defines features for backward-compatibility and that are

useful for high-quality migration of existing binary documents to ISO/IEC 29500.

Part 4 defines the conformance classes WML Transitional, SML Transitional,

and PML Transitional.



State of the art - validation tools



State of the Art – Validation Tools

Package explorer (part of ooXML SDK)

Open existing Office Open XML packages

View relationship part

Validate package XML

Use custom schema for validating custom XML

These Features will be demonstrated

Package explorer v2



State of the Art – Validation Tools (cont.)

oXygen XML editor

Similar features as package explorer

Supports enhanced document validation using Schematron rules

Supports creation and validation of ooXML and ODF documents

Features will be demonstrated

oXygen



Document Validation Outlook –

Semantic Validation using Schematron



Document Validation Outlook – Semantic validation using Schematron

Semantic Validation

Provides additional information based on syntax and semantic constraints.

Certain semantic co-occurrence constraints* are defined in the OOXML standard.

(*The value/presence of one element is dependent on values/presence of another)

Such constraints can generally NOT be represented in XSD files.

There is a need to manually lookup texts in the standard to be able to validate the

conformance of OOXML documents against such semantic constraints.

Schematron can be used to validate such constraints with comparatively little effort.

effort.




Semantic Validation

A simple .docx file containing a comment.




The IS29500 text provides us with the following restriction:

These elements are found in:

/word/document.xml




The „id“ element:

The comment being referenced here is

located at:

/word/comments.xml




Inside „/word/document.xml“:




Inside „/word/comments.xml“:




Semantic Validation

Schematron rule that checks for OOXML semantic conformance.




Semantic Validation




The IS29500 Validator OSS Project

Use Cases



The IS29500 Validator OSS Project – Use Cases

Use document test lib to:

• Up load Documents

• Download documents

• Share documents

• Contribute to validation

Improve your document / application

Use document test lib to:

• Download OSS IS29500 validator

Use OSS IS29500 validator to:

• Perform document validation

Improve your document / application



The IS29500 Validator OSS Project – Use Cases

Developer

User of

validator

User of

document test library

Library accesses

validator



The IS29500 Validator High Level

Architecture



Validator: High Level Architecture

User front end:

Validator aaS

User front end:

Validator download and

installation support

Validator

Rule engine

Validator

Rule editor

Rule and schema repository

Validator

Operation

Validator

Maintenance



Document Test Library: High Level Architecture

User front end:

Document upload

and categorization

User front end:

Document download

Library

Document categorization

Category repository

Library

Operation

Library

Maintenance

Document repository

Library &

Validator

Rule engine



Next Steps towards the OSS Validator

Agreement on general approach and techniques

Support document upload

Support validator download

Integration with document test library

Supported validation types

Validation considering formal and informal (textual) parts of ISO/IEC 29500

Consideration of implementers’ notes

Support different versions of ISO/IEC 29500 and ECMA 376

Definition and evaluation of document categories considering structure and

content of document

Provision of human understandable validation results

Support for developers of document software

Support for users of probably invalid documents



Questions to the Validator

Validator Discussion

Is a online validation service (SaaS) useful or would privacy / secrecy calculations

make the use difficult?

Do you see better technology approaches then XSD / Schematron?

Further remarks, ideas towards the validator?



Agenda






IS29500 validator





Conclusion



Conclusion – Activity Threads

Prerequisite: Provision of project objectives, documentation and examples

considering document test library and IS29500 validator

Development and maintenance of conformance rules

Development or adaptation of rule editor

Development or adaptation of online rule/validation engine

Development or adaptation of offline, downloadable rule/validation engine

Development of I/O components for lib/validator

Upload/download documents

Download, install and run validator

Publish validation results

Operation and maintenance of lib/validator front end and back end



Participation in Validator OSS Project

FhI FOKUS will be responsible for:

Project hosting and leadership

Maintenance and supervision of project structure and goals

Coordination and integration between validator and library

Operation and maintenance of validator and library

The community is invited to participate on:

Definition of compliance rules

Implementation of rule editor and validation engine

Implementation of I/O components

Provision of valid content for the document test library



Screenshot of Photobook Categorization Demo



Screenshot from Carneades Demo



Screenshot of Semantic Validation Demo



Contact

Klaus-Peter Eckert, (Jan Ziesing)

Fraunhofer FOKUS

Kaiserin-Augusta-Allee 31

D 10589 Berlin

Germany

+49 30 3463 7000

{jan.ziesing|klaus-peter.eckert}@fokus.fraunhofer.de


IS 29500 validator and document...

Documents

Transcript of IS 29500 validator and document...