ADT 2010 XQuery Updates in MonetDB/XQuery Other Approaches...

Stefan.Manegold@CWI.nl MonetDB/XQuery: Updates ADT 2010

ADT 2010ADT 2010

XQuery Updates in MonetDB/XQueryXQuery Updates in MonetDB/XQuery

Other Approaches to XQuery ProcessingOther Approaches to XQuery Processing

Stefan ManegoldStefan.Manegold@cwi.nl

http://www.cwi.nl/~manegold/

• 09.11.2010:

•RDBMS back-end support for XML/XQuery (1/2):

•Document Representation (XPath Accelerator, Pre/Post plane)

• 16.11.2010:

•XPath navigation (Staircase Join)

•XQuery to Relational Algebra Compiler:

•Item- & Sequence- Representation

•Efficient FLWoR Evaluation (Loop-Lifting)

•Optimization

• 23.11.2010:

•Updateable Document Representation

•Other (DB-) approaches to XML/XQuery processing

ScheduleSchedule

XQuery Update Facility 1.0 W3C Candidate Recommendation http://www.c3.org/TR/xquery-update-10/

• Categorize updates into• Value updates• Structural updates

(MonetDB/XQuery does not yet support the latest syntax changes made by W3C; for details see

http://monetdb.cwi.nl/XQuery/Documentation/XQuery-Updates.html)

XML/XQuery UpdatesXML/XQuery Updates

do replace value of fn:doc("bib.xml")/books/book[1]/pricewith fn:doc("bib.xml")/books/book[1]/price * 1.1

do replace value of fn:doc(“bib.xml”)/books/book[2]/@isbnwith “90-6196-517-9”

do rename fn:doc(“bib.xml”)/books/book[3]/author[1]into “primary-author”

do rename fn:doc(“bib.xml”)/journals/journal[9]/@isbninto “issn”

=> map directly to simple value updates in relational storage

Value UpdatesValue Updates

do insert attribute isbn {“90-6196-517”}into fn:doc("bib.xml")/books/book[17]

do delete fn:doc(“bib.xml”)/books/book[2]/@wrong

do insert <author>Stefan Manegold</author>after fn:doc(“bib.xml”)/books/book[33]/author[last()]

do replace fn:doc(“bib.xml”)/books/book[44]/author[1]with fn:doc(“bib.xml”)/books/book[33]/author[last()]

do delete fn:doc(“bib.xml”)/books/book[author = “Kermit”]

=> How to implement on pre-/post-encoding?

Structural UpdatesStructural Updates

XML/XQuery XML/XQuery UpdatesUpdates

do insert <k><l/><m/></k> as first into /a/f/g

XML/XML/XQuery XQuery UpdatesUpdates

StaircaseStaircaseJoinJoin

XML Storage RevisitedXML Storage Revisited

N9N8N7

N6N5N4N3N2nullnullN1N0nid

null03

30113010229

306305224

null-121510110

levelsizerid

309308227206145304303222131090

levelsizepre

null-12nullnull3

30113010229208147306305224

1510110

levelsizepre

69j58i77h46g85f14e03d22c31b90a

postpre

post = pre + size - level

Allow holes Define logical pages

XML Storage RevisitedXML Storage Revisited

N5N4N3

null03

14113010309

306225204

null-121510110

levelsizerid

309308227206145304303222131090

levelsizepre

null-12nullnull3

30113010229208147306305224

1510110

levelsizepre

69j58i77h46g85f14e03d22c31b90a

postpre

post = pre + size - level

Allow holes Define logical pages

122100

mappage

rid = pre.swizzle( )

XML Storage RevisitedXML Storage RevisitedUpdate-friendly• rid-table is append-only• rid-tuples may be unused• rid = autoincrement column

MonetDB: • rid not stored but computed (virtual oid)• allows positional lookup/join

Opportunity currently not exploited by other RDBMS

Occurs widely in our XQuery translation.

N5N4N3

null03

14113010309

306225204

null-121510110

levelsizerid

XML Storage RevisitedXML Storage RevisitedUpdate-friendly• rid-table is append-only• rid-tuples may be unused• rid = autoincrement column

MonetDB: • rid not stored but computed (virtual oid)• allows positional lookup/join

Opportunity currently not exploited by other RDBMS

Occurs widely in our XQuery translation.

N5N4N3

null03

14113010309

306225204

null-121510110

levelsizerid

MonetDB/XQueryMonetDB/XQueryOur own XML DBMS with (almost..) full XQuery support.• Built purely on an RDBMS, namely MonetDB

Pathfinder compiler & “staircase join”:– Universität Tübingen (Torsten Grust, et al.)

– Technical University Twente (Maurice van Keulen, et. al.)

MonetDB High-Performance DBMS– CWI Amsterdam (Peter Boncz, Stefan Manegold, ...)

Useful for:

• Large XML databases!

• Querying XML annotations (multimedia, forensic NFI)

• XML information retrieval

• ...

Pathfinder Compiler

RelationalAlgebra

XQuery

(MonetDB)

Research Projects & ExtensionsResearch Projects & Extensions• Value indeces

• Runtime optimization• SIGMOD'09 [Abdel Kader, Boncz, v. Keulen Manegold]

• Algebraic Query Optimization• Grust, Rittinger, et al. (Universität Tübingen)

• Distributed XQuery P2P XQuery• SOAP group communication, XQuery RPC

• VLDB'07 [Zhang, Boncz]

• Benchmarking beyond XMark• ExpDB'06 Workshop [Manegold]

• Support for XML Interval Annotations• XIME-P'06 Workshop [Alink et al.]

• Xquery + Information Retrieval: PF/Tijah

ConclusionsConclusions• Relational approach can be scalable & fast

• MonetDB/XQuery compares favorably with all other available systems

• Techniques that made it work• Property-driven peephole optimization

Order & other properties

• Loop-lifted XPath steps Evaluate Sets of context nodes in a single pass

• Support for dense (autoincrement) keys Positional lookup

• Background Information & Literaturehttp://monetdb-xquery.orghttp://pathfinder-xquery.org

Stefan.Manegold@CWI.nl Other Xquery Processing Approaches ADT 2010

• 09.11.2010:

•Document Representation (XPath Accelerator, Pre/Post plane)

• 16.11.2010:

•XPath navigation (Staircase Join)

•XQuery to Relational Algebra Compiler:

•Item- & Sequence- Representation

•Efficient FLWoR Evaluation (Loop-Lifting)

•Optimization

• 23.11.2010:

•Updateable Document Representation

•Other (DB-) approaches to XML/XQuery processing

ScheduleSchedule

TopicsTopics Other approaches & techniques (selection, far from complete!)

Document storage / tree encoding:

ORDPATH

DataGuides

XPath processing:

Tree patterns, holistic twig joins

Fixed-Width Tree Encodings & UpdatesFixed-Width Tree Encodings & Updates Fixed-width tree encoding (like XPath Accelerator) are

Good for read(-only) processing

small footprint, positional lookup, staircase join

But inherently static

Milo et al., PODS 2002:

“There is a sequence of updates (subtree insertions) for any persistent tree encoding scheme E (where each node keeps its initial encoding label even under updates), such that E needs labels of length (N) to encode the resulting tree of N nodes.”

Milo et al., PODS 2002:

“There is a sequence of updates (subtree insertions) for any persistent tree encoding scheme E (where each node keeps its initial encoding label even under updates), such that E needs labels of length (N) to encode the resulting tree of N nodes.”

XML/XQuery XML/XQuery UpdatesUpdates

MonetDB/XQuery

exploit paging

& mmap trick

updating pg|off

is still O(N)

Non-solutions:

Gaps in the encoding (never large enough)

Encoding based on decimal fractions (limited precision)

Possible solution:

Variable-width tree encodings:

Cheaper updates

At the expense of more expensive read(-only) processing

A Variable-Width Tree Encoding: ORDPATHA Variable-Width Tree Encoding: ORDPATH

O'Neil et al., SIGMOD 2004.

ORDPATH Encoding: ExampleORDPATH Encoding: Example

ORDPATH: Insertion Between SiblingsORDPATH: Insertion Between Siblings

Is ORDPATH suitable for XQuery?Is ORDPATH suitable for XQuery?• Mapping core operations of the XQuery processing model

to operations on ORDPATH labels:

ORDPATH: Variable-Length Node EncodingORDPATH: Variable-Length Node Encoding• For a 10 MB XML sample document, the authors of ORDPATH observed

label lengths between 6 and 12 bytes.

• ORDPATH labels encode root-to-node paths => common prefixes.

=> Label comparisons often need to inspect encoding bits at the far right.

• MS SQL Server employs further path encodings organized in reverse

(node-to-root) order.

• Note: - Preorder ranks fit into CPU registers.- 4 byte pre's sufficient for 232 = 4G nodes (11 GB XMark fits easily).- 8 byte pre's sufficient for 264 nodes, i.e., “the universe” ...

ORDPATH

DataGuides

XPath processing:

DataGuidesDataGuides

XPath Accelerator, ORDPATH & similar encoding schemes

encode the document's tree structure in the node ranks/labels

they assign

DataGuides

Developed in the context of Lore project (DBMS for semi-

structured data)

Stanford University, Goldman & Widom, VLDB 1997

encode the document's tree structure in relation names

Observation:

Each node is uniquely identified by its path from the root

Paths of siblings with equal tag names can be unified,

Provided we keep their relative order (rank) explicitly

Definition

given a semistructured data instance DB, a DataGuide for DB is a graph G s.t.:

- every path in DB also occurs in G

- every path in G occurs in DB

- every path in G is unique

Example:

■ Multiple DataGuides for the same data:

DefinitionLet p, p’ be two path expressions and G a graph; we define

p ≡ G p’ if p(G) = p’(G)

i.e., p and p' are indistinguishable on G.

DefinitionG is a strong dataguide for a database DB if ≡ G is the same as ≡ DB

Example:- G1 is a strong dataguide- G2 is not strong

person.project !≡ DB dept.project

person.project !≡ G1 dept.project

person.project ≡ G2 dept.project

■ Constructing the strong DataGuide G:

Nodes(G)={{root}}

Edges(G)=∅

while changes do

choose s in Nodes(G), a in Labels

add s’={y|x in s, (x -a->y) in Edges(DB)} to Nodes(G)

add (x -a->y) to Edges(G)

• Use hash table for Nodes(G)

• This is precisely the powerset automaton construction.

Monet XML approachMonet XML approach

Early attempt to store and query XML data in MonetDB

By Albrecht Schmidt

Not related to Pathfinder & MonetDB/XQuery

Monet XML approachMonet XML approachMonet XML approachMonet XML approach

Early attempt to store and query XML data in MonetDB

By Albrecht Schmidt

Not related to Pathfinder & MonetDB/XQuery

No XQuery compiler

XMark queries are hand-crafted and -optimized in MIL

Child, Descendant, Parent & Ancestor steps become regular

expressions on the relation names (i.e., catalog)

Open: preceeding & following steps?

ORDPATH

DataGuides

XPath processing:

Twig Join AlgorithmsTwig Join Algorithms

So far: interpreted XPath expressions in an imperative manner

Evaluated XPath expressions step-by-step, as stated in the query

Given /1::1/2::2/.../n::n,

we first evaluated /, then XPath step 1::1, then step 2::2, ...

This may not always be the best choice:

Intermediate results can get very large, even if the final result is

small:

Database context => think in a declarative manner

DBMS optimizer / engine can evaluate query in “best” order

Tree PatternsTree Patterns

In fact, XPath is a declarative language. /descendant::timeline/child::event

“Find all nodes v1, v2, and v3, such that

v1 is a document root,

v2 is a descendant element of v1 and is named timeline, and

v3 is a child element of v2 and named event.

All nodes of type v3 form the query result.

Observe the combination of

(a) predicates on single nodes, and

(b) structural conditions between these nodes.

Structural conditions: Intuitively expressed as tree patterns:

Nodes labeled with node predicates

Structural conditions:

Double line: ancestor/descendant relationships

Single line: parent/child relationships

Arbitrary predicates are allowed, but typical are predicate on tag names:

Nodes labeled with requested tag name

Document root: label /

If not /-node specified:

search for pattern anywhere in the documenttimeline

timeline

Not limited to path patterns

May also be twig patterns

Mapping between tree patterns and XPath is in general not trivial

Examples:

PathStack AlgorithmPathStack Algorithm

PathStack Algorithm: Path PatternsPathStack Algorithm: Path Patterns

timeline

timeline timeline

first timeline node visited

second timeline node visited

first event node visited

timeline

PathStack Algorithm: Twig PatternsPathStack Algorithm: Twig Patterns

So far we only considered path patterns

Can we extend our ideas for efficient twig pattern evaluation?

Decompose twig patterns into multiple path patterns.

All path patterns start from the same root.

Use PathStack for each of them and merge their results.

Example: Decompose twig pattern into path patterns

Summary (1/5)Summary (1/5) XML

Document markup

Data exchange

Semi-structured

Tree model

XML Schema

Navigation, location steps, axes, node tests, predicates, functions

XQuery

Sequences & Iterations (FLWoR expressions)

Summary (2/5)Summary (2/5) XML Data Management

XML file processors

XML databases

XML integration platforms

RDBMS with XML functionality, SQL/XML

Relational XML storage: schema-based vs. schema-oblivious

Summary (3/5)Summary (3/5) Purely Relational XML/XQuery processing: MonetDB/XQuery

Document encoding: XPath Accelerator (pre/post plane)

XPath navigation: Staircase Join

XQuery to Relational Algebra translation

Item- & Sequence-representation

Iterations: Loop-lifting

Loop-lifted staircase join

Peephole Optimization

Order-awareness, sort avoidance

XML/XQuery Update Support

Summary (4/5)Summary (4/5) Other approaches & techniques

Document storage/encoding:

ORDPATH

DataGuides

XPath processing:

Summary (5/5)Summary (5/5) Literature

Slides

Literature references in slides

Literature references on website:

http://www.cwi.nl/~manegold/teaching/adt/html/xquery.html

• Tentamen / Exam:

Tuesday December 21 2010

09:00 – 11:00

Zaal / Room: A1.14

Projects: Join the MonetDB Team!Projects: Join the MonetDB Team!• Own ideas, suggestions, initiative welcome!

• Master Student Projects (6 Months)

• Various projects, each consisting of both research & implementation

• See monetdb.cwi.nl/Development/Research/Projects/ for a sample list

• Feel free to come with your own idea(s)!

• Implementation Projects

• Both short-term & long-term

• E.g. open feature requests: sf.net/tracker/?group_id=56967

• Become owner/maintainer of some (new) part of MonetDB

• We are (desperately) looking for Windows SW-development & system

experts!

• 24x7x365 support & advice

• Membership in a kind & friendly Family-Team of Experts

• Chance to participate in & contribute to a large & successful open-source research project

• Lots of experiences, exiting research & fun

• Desk & workstation at CWI

Fridge, micro-wave, free coffee, free soup, free cake (occasionally)

Master Students only (possibly part-time)

Limited availability => FCFS!

Some pocket money (stage vergoeding)

Master Students only

Limited availability => FCFS!

We Offer...We Offer...

ADT 2010 XQuery Updates in MonetDB/XQuery Other Approaches...

Documents

Transcript of ADT 2010 XQuery Updates in MonetDB/XQuery Other Approaches...

MonetDB/XQuery InfoMgmt 2009 Peter Boncz boncz@cwi.nl (CWI Amsterdam) Querying XML Data Sources using MonetDB/XQuery.

MonetDB, a Column-Store in Midflight

Module 1: Introductionhomepages.cwi.nl/~manegold/teaching/adt/lectures/ADT2010_XML_… · 2 Stefan.Manegold@CWI.nl Introduction to XML, XPath & XQuery ADT 2010 why • Motivation

XQuery Rocks

MonetDB/XQuery Technology Preview 1 Stefan Manegold CWI Amsterdam -

MonetDB/XQuery Reference Manual · PDF file1 This is the reference manual of MonetDB/XQuery, and open-source XQuery database system built on: the open-source MonetDB column-store,

Peter BonczCWI Scientific Meeting 28/4/2006MonetDB/XQuery MonetDB/XQuery: using relational technology to query XML documents Peter Boncz Centrum voor Wiskunde.

ADT 2010 XQuery Updates in MonetDB/XQuery - CWIhomepages.cwi.nl/~manegold/teaching/adt/lectures/... · 13 Stefan.Manegold@CWI.nl MonetDB/XQuery: Updates ADT 2010 XML Storage Revisited

XQuery Reference

XML and Databases XQuery, XSLT and XPath XQuery XQuery XML

Monetdb Intro PPT

ADT 2010 XQuery Updates in MonetDB/XQuery · •XQuery to Relational Algebra Compiler: •Item- & Sequence- Representation •Efficient FLWoR Evaluation (Loop-Lifting) •Optimization

XQuery Extensions

MonetDB/XQuery: Using a Relational DBMS for XML

Chapter 5 Query Execution Pipelined - uni-tuebingen.de · 2018. 3. 9. · MonetDB/X100 MonetDB/X100, developed at CWI, Amsterdam. Principal architect is Peter Boncz. MonetDB/X100

XQuery: a typed functional language for querying XMLhomepages.inf.ed.ac.uk/wadler/papers/xquery-afp/xquery-afp.pdf · XQuery: a typed functional language for ... published before

The MonetDB Architecture Martin Kersten CWI Amsterdam

XML, XPath (& XQuery)homepages.cwi.nl/~manegold/teaching/adt/lectures/ADT2010...10 Stefan.Manegold@CWI.nl Introduction to XML, XPath & XQuery ADT 2010 A Little Bit Of History • Database

XQuery: Joining Content and Data · XQuery 1.0 provides the basics Query, manipulate, render XQuery 1.0 Full Text extensions Essential for human text, esp. multilingual text XQuery

1 XQuery Web and Database Management System. 2 XQuery XQuery is to XML what SQL is to database tables XQuery is designed to query XML data What is XQuery?