Http://forge.gridforum.org/projects/dfdl-wg/ DFDL WG Session 1 Summary of Status WG Co-Chairs: Mike...
-
Upload
ami-garrett -
Category
Documents
-
view
230 -
download
0
Transcript of Http://forge.gridforum.org/projects/dfdl-wg/ DFDL WG Session 1 Summary of Status WG Co-Chairs: Mike...
http://forge.gridforum.org/projects/dfdl-wg/
DFDL WG Session 1Summary of StatusWG Co-Chairs:
Mike Beckerle, Ascential Software Alan Chappell, PNNL Martin Westhead
(Two note takers please?)
Tuesday, 2004-09-21 17h30+02:00 Brussels (BE.CEST) in Room D005.
11h30 New York, (US.EDT, UTC-4) 08h30 San Francisco (US.PDT, UTC-7)
http://forge.gridforum.org/projects/dfdl-wg/
Abstract
Progress and Status Review - This session will review the progress since GGF11 which is substantial, and also provide enough context for those interested in getting involved who haven't been following along.
http://forge.gridforum.org/projects/dfdl-wg/
ADF BoF
Sep 22 11h00-12h30ACCESS TO DATA IN FILES-WG CHARTER DISCUSSION BOF (DATA) Calendar: QB (200p)Details:Title: ADF - Access to Data in FilesSession Leaders: Neil Chue Hong, Brian M CollinsAgenda:- Summary of General Focus, Milestones and Objectives- Discussion of Relationships specifically with DFDL and DAIS as well as other GGF Groups- Charter Update
http://forge.gridforum.org/projects/dfdl-wg/
DFDL WG Session 2
Room D008
Thursday, 2004-09-23 11h00+02:00 Brussels (BE.CEST)
05h00 New York (US.EDT, UTC-4)
02h00 San Francisco (US.PDT, UTC-7)
http://forge.gridforum.org/projects/dfdl-wg/
Since GGF11 (June 2004)
Latest working examplesdocument/jdm20040720.zip
DFDL Represenation Properties: Basic Set document/ggf-dfdl-rep-properties-
proposal-basic-001.doc
http://forge.gridforum.org/projects/dfdl-wg/
DFDL-WG Session 1 Summary of Status
Agenda Review of purposes/goals
XML / XSD impact: Data ModelExamples w/latest updates
• as evolved now: different from primer document
General discussion (40 mins) Overview of other sessions (5 mins)
http://forge.gridforum.org/projects/dfdl-wg/
Data Interchange Formats
Prescriptive: Put your data in this format! XML – textual Binary – ASN.1, XDR, NetCDF, HDF, EBML,…
Descriptive: What format is your data in? Commercial products ASN1 Encoding Control Notation
• ITU-T X.692
DFDL
http://forge.gridforum.org/projects/dfdl-wg/
Why Descriptive?
Allows us to achieve two goals simultaneously:
1. Interoperability Modern and Legacy data formats
2. Performance! Density
• Fewest bytes to represent data without resorting to compression
Optimized I/O• Seekable random access• Memory mapped, aligned
• Without sacrificing general access
http://forge.gridforum.org/projects/dfdl-wg/
Why the GGF for DFDL?
Grids are about big-data and big-computation problemsSimplistic solutions like “use XML”
won’t cut it! Grids are about universal data
interchange
http://forge.gridforum.org/projects/dfdl-wg/
General Features
Basic ASCII/Binary Read capabilities Inclusion of static info, e.g. “units” Validated input (from XML Schema) Defaulted input for missing values Reference – use of a previously read value in subsequent
expressions Choice – use of a previously read value to select among format
variations Multi-layer – description of an intermediate representation not
exposed in the final result Multiple input streams Basic Math – in DFDL expressions Basic Math – to generate new representations/values “Push-back” capability New type/transform specification
http://forge.gridforum.org/projects/dfdl-wg/
Desires
Support very efficient parsers Hints about lengths so data can be accessed
randomly Avoid requiring data copies But allow caching (e.g. based on hints) Allow ‘black-box’ transforms (e.g. unzip)
Support round-tripping Link transforms and inverses Infer inverses for lossless transforms
Keep simple cases simple Default transforms and parameters
http://forge.gridforum.org/projects/dfdl-wg/
Related Standards Efforts
Prescriptive systems:W3C binary XML
(http://www.w3.org/XML/Binary/)• Formed, but discussion group has no
items.• CORRECTION - there is activity now.
Descriptive systems:ASN1 Encoding Control Notation
• ITU-T X.692
http://forge.gridforum.org/projects/dfdl-wg/
XML Synergy
Use XSD to describe the logical data Use annotations within the XSD to
describe the representation of it.
http://forge.gridforum.org/projects/dfdl-wg/
XSD Types
Elements A.k.a. fields
Sequence groups, All groups All = unordered group
Choice A.k.a. union, redefine,
Vectors Use element with minOccurs, maxOccurs.
Nillability A.k.a. Nullable values
http://forge.gridforum.org/projects/dfdl-wg/
XML/XSD – basic types
anySimpleType
string QName NOTATION float double decimal boolean base64Binary hexBinary anyURI
normalizedString
token
language Name NMTOKEN
NMTOKENSNCName
ID IDREF ENTITY
IDREFS ENTITIES
integer
long nonPositiveInteger nonNegativeInteger
negativeInteger positiveInteger unsignedLong
unsignedInt
unsignedShort
unsignedByte
int
short
byte
date time dateTime gYear gYearMonth gMonth gMonthDay gDay duration
http://forge.gridforum.org/projects/dfdl-wg/
Example 1: XML
<w>5</w><x>7839372</x><y>8.6E-200</y><z>-7.1E8</z>
http://forge.gridforum.org/projects/dfdl-wg/
Example 1: XSD
<xs:sequence><xs:element name="w" type=“int"/><xs:element name="x" type=“int"/><xs:element name="y" type=“double"/><xs:element name="z" type=“float"/>
</xs:sequence>
http://forge.gridforum.org/projects/dfdl-wg/
Example 1 DFDL - binary
0000 0005 0077 9e8c
169a 54dd 0a1b 4a3f
ce29 46f6
http://forge.gridforum.org/projects/dfdl-wg/
Example 1 DFDL - binary
<xs:complexType name="example1"> <xs:annotation> <xs:appinfo> <repType>binary</repType> <byteOrder>bigEndian</byteOrder> </xs:appinfo> </xs:annotation> <xs:sequence> <xs:element name="w" type="int"/> <xs:element name="x" type="int "/> <xs:element name="y" type="double "/> <xs:element name="z" type="float "/> </xs:sequence> </xs:complexType>
http://forge.gridforum.org/projects/dfdl-wg/
Example 1 DFDL - textual
“5, 7839372, 8.6E-200, -7.1E8”
http://forge.gridforum.org/projects/dfdl-wg/
Example 1 DFDL - textual
<xs:complexType name="example1"> <xs:annotation> <xs:appinfo> <repType>text</repType> <charset>UTF-8</charset> <decimalSeparator>.</decimalSeparator> <separator>,</separator>
</xs:appinfo></xs:annotation><xs:sequence>
<xs:element name="w" type="int"/><xs:element name="x" type="int"/><xs:element name="y" type="double"/><xs:element name="z" type="float"/>
</xs:sequence></xs:complexType>
http://forge.gridforum.org/projects/dfdl-wg/
More Examples
Currently working on a set of examples which illustrate each of the DFDL constructs
http://forge.gridforum.org/projects/dfdl-wg/
Examples (so far)
BasicMathInExpressions.xsd BasicMathInRepresentations.xsd Choice.xsd DefaultValuePropagation.xsd DefaultedInput.xsd IncludeTransform.xsd IncludeType.xsd & NewDFDLType.xsd MultiLayer.xsd MultiStreams.xsd NewDFDLTransform.xsd NewExternalDFDLTransform.xsd NewMixedDFDLTransform.xsd Reference.xsd StaticInformation.xsd ValidatedInput.xsd
http://forge.gridforum.org/projects/dfdl-wg/
Current Status
Fairly clear idea of required features and data types
Large library of simple, real world, and pathological examples
Converging on an annotated XML Schema syntax for DFDL
http://forge.gridforum.org/projects/dfdl-wg/
Status
We think we have:A good handle on requirements/use casesCommon idea of functionality/ overall
designA not quite complete syntax based on
XML SchemaA set of “unit-test” examples to help in
finalizing the syntax
http://forge.gridforum.org/projects/dfdl-wg/
Opportunities
Good time to help with syntax Once that exists:
Standardized language bindings/APIsDFDL implementationsDFDL-based ADF Data Virtualization
ServicesExtensions for specific domains/uses…
http://forge.gridforum.org/projects/dfdl-wg/
Next Steps
Session 2: Charter-revisions discussion Review of specific examples IBM Websphere Business Integrator
DFDL presentation
http://forge.gridforum.org/projects/dfdl-wg/
END