Native XML Databases

Copyright 2002-2003, Ronald Bourret, http://www.rpbourret.com

Native XML Databases

Ronald [email protected]://www.rpbourret.com


Overview

• What is a native XML database?

• Use cases

• Code examples

• Implementation and performance

• Normalization and referential integrity

• Common database features


What is aNative XML Database?


Storing XML in a database

1. Build a model» Model the data in the XML document, or...» Model the XML document itself

2. Map the model to the database

3. Transfer data according to the model


Sample XML document

<Order> <Number>1234</Number> <Customer>Gallagher Co.</Customer> <Date>29.10.00</Date> <Item Number="1"> <Part>A-10</Part> <Quantity>12</Quantity> <Price>10.95</Price> </Item> <Item Number="2"> <Part>B-43</Part> <Quantity>600</Quantity> <Price>3.99</Price> </Item></Order>


Modeling data

• Objects in model are specific to XML schema

object Order { number=1234; customer="Gallagher Corp."; date=29.10.00; items={ptrs to Items};}

object Item { number=1; part="A-10"; quantity=12; price=10.95;}

object Item { number=2; part="B-43"; quantity=600; price=3.99;}


Storing data

• Database schema specific to XML schema

OrdersNumber Customer Date1234 Gallagher Co. 291000... ... ...... ... ...

ItemsSONum Item Part Qty Price1234 1 A-10 12 10.951234 2 B-43 600 3.99... ... ... ... ...


Modeling documents

• Objects in model independent of XML schema

Element (Order)

Element Element Element Element Element(Number) (Customer) (Date) (Item) (Item)

Text Text Text ... Element Element Element Attr 1234 Gallagher Co. 29.10.00 (Part) (Quantity) (Price) (Number)

Text Text Text Text

A-10 12 10.95 1


Storing documents

• Database schema independent of XML schema(order columns not shown)

Elements Attributes TextID Name Parent ID Name Parent Parent Value 1 Order -- 13 Number 5 2 12342 Number 1 14 Number 6 3 Gallagher Co.3 Customer 1 4 29.10.004 Date 1 7 A-105 Item 1 8 126 Item 1 9 10.957 Part 5 10 B-438 Quantity 5 11 6009 Price 5 12 3.9910 Part 6 13 111 Quantity 6 14 212 Price 6


Types of databases (take 1)

• Categorize databases based on what they model

• Model data => XML-enabled database

• Model document => Native XML database



• XML adds a new data model to the world» In addition to relational, hierarchical, OO, ...

• The “XML” data model is» A tree of ordered nodes» Nodes have different types (element, attribute, etc.)» Some nodes are labeled (Date, Quantity, Price, etc.)» Data stored in leaf nodes

• Modeling language is an XML schema language» DTD, XML Schemas, RELAX NG, etc.



• XML-enabled databases» Have their own data model (relational, OO, etc.)» Map their data model to XML data model

• Native XML databases» Use XML data model directly


Formal definition¹

• A native XML database:» Defines an XML data model» Uses a document as its fundamental unit of (logical)

storage» Can have any physical storage

¹ Definition from XML:DB Initiative


XML data model

• XML 1.0 did not define a model

• Native XML databases define their own model» Model must include elements, attributes, text, and

document order» Examples are XPath, Info Set, DOM, and SAX» XQuery model will be de facto standard in future?

• Data transferred according to the model


Fundamental unit of storage

• Document is fundamental unit of logical storage» Equivalent structure in RDBMS is a row

• Document usually contains single set of data


Physical storage

• Can use any physical storage

• Examples» Tables for SAX “objects” in relational database» DOM objects in object-oriented database» Binary file format optimized for XPath data model» Hashtables of XPaths and values» Compressed, indexed XML documents in file system


Use Cases


Use case: Tabular data

• Manufacturing data

• Accounting data

• Employee data

• Scientific data

• ...


Use a native XML DB only if:

• You want to lose your job.

• Relational databases excel at storing tabular data

• Systems are mature, fast

• Few reasons to change existing code or DBMS


Use case: Documents

• Product documentation

• Static Web pages

• Presentations

• Advertisements

• ....


Use a native XML DB because:

• Structure too irregular for relational model

• Physical info (entities, CDATA, ...) is important

• Need document-centric queries» Get first chapter with a paragraph containing “XML”» Get headings of all chapters that contain figures

• Want to retain document identity


Use case: Semi-structured data• Some structure, but not regular

» Structured data: white pages» Semi-structured data: yellow pages

• Examples» Health data» Result of EAI (Enterprise Application Integration)» Geneological data» ...


Use a native XML DB because:• Difficult to store in a relational database

» Use single, sparsely populated table or...» Many tables (one per subject area)

• Documents might contain arbitrary extensions


Use case:“Natural” format is XML

• Only format is XML» XSLT stylesheets

• Data stored temporarily as XML» Long-running transactions» Enterprise application integration (EAI)» Documents in a message queue

• Schema-less documents» No schema or schema not known



• Want query language, security, transactions, etc.

• No reason to use an XML-enabled database» Natural format is XML» Mapping arbitrary documents at run-time is

inefficient, error-prone


Use case: Large documents

• Need entire document in memory» DOM tree» XSLT transformation» etc.

• Document too large for memory



• In-memory tree lazily instantiated» Unused nodes kept on disk

• Tree size limited only by:» Disk size» Database address space


Use case: Archiving

• Long-term document storage

• Often required by law» Pharmaceutical industry» Contracts» Financial transactions



• Want query language, security, transactions, etc.» Better than file system or BLOBs

• Retains complete document» Retaining original document might be a problem

• May support versioning through updates or diffs


Use case: Performance

• We all want better performance, right?



• May be faster than XML-enabled databases

• Speed depends on:» Actual query» Query and storage engines» Output format (DOM, SAX, string)


Code Examples


XML:DB API

• Database-neutral API from XML:DB Initiative

• Similar to JDBC, OLE DB, etc.» Implemented by database drivers» Methods to connect to database, execute queries, etc.» Uses separate query language(s)


JDBC example: INSERT

// Instantiate the Sun JDBC-ODBC bridge driver. The driver registers// itself with the driver manager.

Class.forName("sun.jdbc.odbc.JdbcOdbcDriver");

// Get a connection to the Orders database and turn auto-commit off.

Connection conn = DriverManager.getConnection("jdbc:odbc:Orders", "rpbourret", "my_password");conn.setAutoCommit(false);

// Get a Statement.

Statement s = conn.createStatement();


JDBC example: INSERT (cont.)

// Execute INSERT statements to insert new rows into the Orders// and Items tables.

s.executeUpdate("INSERT INTO Orders " + "VALUES ('1234', 'Gallagher Co.', #10/29/00#)");s.executeUpdate("INSERT INTO Items " + "VALUES ('1234', 'A-10', 12)");s.executeUpdate("INSERT INTO Items " + "VALUES ('1234', 'B-43', 600)");

// Commit the transaction. Close the statement and the connection.

conn.commit();s.close();conn.close();


XML:DB example: INSERT// Get the database (driver) for dbXML/Xindice. Register it// explicitly with the database manager.

Class c = Class.forName("org.dbxml.client.xmldb.DatabaseImpl");Database db = (Database) c.newInstance();DatabaseManager.registerDatabase(db);

// Get the Orders collection.

Collection coll = DatabaseManager.getCollection("xmldb:dbxml:///db/Orders", "rpbourret", "my_password");

// Get a transaction service. XML:DB is designed to use "services",// such as transaction, collection management, and query services.

TransactionService txn = (TransactionService)coll.getService("TransactionService", "1.0");


XML:DB example: INSERT (cont.)

// Start a new transaction.

txn.begin();

// Create a new (empty) XMLResource in the Orders collection.

XMLResource res = (XMLResource)coll.createResource("Order1234", "XMLResource");

// Read the contents of the Order1234.xml file into a string.

int b;FileReader r = new FileReader("Order1234.xml");StringWriter w = new StringWriter();

while ((b = r.read()) != -1){ w.write(b);}


XML:DB example: INSERT (cont.)

// Set the contents of the new resource to the string.

res.setContent(w.toString());

// Commit the transaction.

txn.commit();

// Close the collection and deregister the database.

coll.close();DatabaseManager.deregisterDatabase(db);


JDBC example: SELECT

// Instantiate the Sun JDBC-ODBC bridge driver. The driver registers// itself with the driver manager.

Class.forName("sun.jdbc.odbc.JdbcOdbcDriver");

// Get a connection to the Orders database.

Connection conn = DriverManager.getConnection("jdbc:odbc:Orders", "rpbourret", "my_password");

// Execute a SELECT statement to get all orders from 10/29/00.

Statement s = conn.createStatement();ResultSet rs = s.executeQuery("SELECT Number, Date, Customer " + "FROM Orders WHERE Date=#10/29/00#");


JDBC example: SELECT (cont.)

// Get each row in the result set and print it.

while (rs.next()){ System.out.println("Order: " + rs.getString(1)); System.out.println("Customer: " + rs.getString(2)); System.out.println("Date: " + rs.getDate(3)); System.out.println();}

// Close the result set, the statement, and the connection.

rs.close();s.close();conn.close();


XML:DB example: SELECT// Get the database (driver) for dbXML/Xindice. Register it// explicitly with the database manager.

Class c = Class.forName("org.dbxml.client.xmldb.DatabaseImpl");Database db = (Database) c.newInstance();DatabaseManager.registerDatabase(db);

// Get the Orders collection.

Collection coll = DatabaseManager.getCollection("xmldb:dbxml:///db/Orders", "rpbourret", "my_password");

// Get an XPath query service over the Orders collection. XML:DB// is designed to use a variety of "services", such as different// query engines.

XPathQueryService service = (XPathQueryService)coll.getService("XPathQueryService", "1.0");


XML:DB example: SELECT (cont.)

// Execute an XPath query to get all orders from 29.10.00.

ResourceSet docs = service.query("/Order[Date=\"29.10.00\"]");

// Iterate over the returned documents and print each one. We do// not need to know any metadata (such as data types) here. This// is because metadata is contained in the XML document itself.

ResourceIterator iterator = docs.getIterator();while (iterator.hasMoreResources()){ XMLResource doc = (XMLResource)iterator.nextResource(); System.out.println((String)doc.getContent()); System.out.println();}

// Close the collection and deregister the database.

coll.close();DatabaseManager.deregisterDatabase(db);


Implementationand Performance


Text-based storage

• Stores documents as text

• Uses file system, CLOB, etc.» Includes XML-aware text in RDBMS

• May need to parse documents at run time

• Uses indexes to avoid extra parsing, increase search speed


Text-based storage

<Address> <Street>123 Main St.</Street> <City>Chicago</City> <State>IL</State> <PostCode>60609</PostCode> <Country>USA</Country></Address>





Text-based databases

• Indexed files» TextML

• CLOBs» Oracle 9i release 2, DB2


Model-based storage

• Stores documents in “object” form

• Documents parsed when inserted

• For example, store DOM objects in OODBMS

• Underlying storage can be relational, object-oriented, hierarchical, proprietary

• Uses indexes to speed searches


Model-based storage


Element

Element Element Element Element Element

Text Text Text Text Text


Model-based databases

• Proprietary» Tamino, Xindice, Neocore, Ipedo, XStream DB,

XYZFind, Infonyte, Virtuoso, Coherity, Luci, TeraText, Sekaiju, Cerisent, DOM-Safe, XDBM, ...

• Relational» Xfinity, eXist, Sybase, DBDOM

• Object-oriented» eXcelon, X-Hive, Ozone/Prowler, 4Suite, Birdstep


Performance

• No public benchmark data?

• Native vendors say native is faster

• Relational vendors say relational is faster

• Native XML databases rely heavily on indexing» Slows update times

• Some guesses are possible


Whole documents and fragments(Text-based databases)

• Should be very fast» Data is contiguous on disk» Retrieval requires index lookup and single disk read

1. Index lookup2. Position disk head3. Read to here


Whole documents and fragments (Model-based databases)

• Databases with proprietary stores should be fast» Can use physical pointers between nodes

• Databases built on other DBs may be fast or slow» Depends on underlying database and implementation

Node Node Node Node NodeNode

1. Index lookup2. Position disk head3. Follow pointers to end


Unindexed data

• Slow for model-based databases» Must read many elements, not just particular type» Comparisons may be slower due to converting text

• Very slow for text-based databases» Must parse document as well as comparing values

Element

Element Element Element Element

Text Text Text Attr Element ...

Task:Find date 29.10.00

Relational database:1. Search this column

Model-based native XML database:1. Search all elements for Date elements2. Search text for all Date elements

Orders... ... ... 1234 29.10.00 Gallagher Industries ... ... ... ... ... ...


Unindexed data: Optimization

• /Order[Date="29.10.00"]» Only need to search children of Order» Schema may help locate Date element

• //Order[Date="29.10.00"]» Schema may help locate Order and Date elements» Without schema, must search entire document


Query return types

• Text-based databases» Very fast returning strings» Slow returning DOM or SAX due to parsing

• Model-based databases» Probably similar to XML-enabled databases


Queries not following storage hierarchy

• The $64,000 question ...

• Slower than hierarchical queries

• Pointers to parents may help model-based DBs

<Order> <Number>1234</Number> <Customer>Gallagher Co.</Customer> <Date>29.10.00</Date> <Item Number="1"> <Part>A-10</Part> <Quantity>12</Quantity> <Price>10.95</Price> </Item> <Item Number="2"> <Part>B-43</Part> <Quantity>600</Quantity> <Price>3.99</Price> </Item></Order>

Get the dates and numbers ofall sales orders for part “A-10”

1. Index lookup for part “A-10”2. Follow pointers or index to Order3. Search children for Number, Date

<Part Number="A-10" <Order> <Number>1234</Number> <Date>29.10.00</Date> </Order> ...</Part>


Normalization andReferential Integrity


Normalization

• Defined with respect to the relational model

• Performed on schema

• Determines minimal logical structures

• Eliminates duplicate data


Example: Normalization

ORDERS:OrderNum, Date, CustNum, Address,ItemNum, Qty, PartNum, Price

ORDERS:OrderNum, Date, CustNum

CUSTOMERS:CustNum, Address

ITEMS:OrderNum, ItemNum, Qty, PartNum

PARTS:PartNum, Price


First normal form

Relational schema• One value per column

• Define primary key» Means one set of data per row

XML schema• Does not apply; multiple

children allowed

• Root element not a “wrapper”» Means one set of data per doc

» Reduces concurrency problems

» Reduces parse time in text-based databases


First normal form (cont.)

Relational schema XML schema



<!ELEMENT Orders (Order+)><!ELEMENT Order (Number, Date, CustNum, Address, (ItemNum, Qty, PartNum, Price)+)>

<!ELEMENT Order (Number, Date, CustNum, Address, (ItemNum, Qty, PartNum, Price)+)>


Second normal form

Relational schema• Create separate table for

one-to-many fields» Eliminates duplicate "parent"

data

XML schema• Create separate elements for

repeated groups

• Separate document not needed» Hierarchical structure allows

multiple children per parent


Second normal form (cont.)

Relational schema XML schemaORDERS:OrderNum, Date, CustNum, Address,ItemNum, Qty, PartNum, Price

<!ELEMENT Order (Number, Date, CustNum, Address, (ItemNum, Qty, PartNum, Price)+)>

ORDERS:OrderNum, Date, CustNum, AddressITEMS:OrderNum, ItemNum, Qty, PartNum, Price

<!ELEMENT Order (Number, Date, CustNum, Address, Item+)><!ELEMENT Item (ItemNum, Qty, PartNum, Price)>


Third normal form

Relational schema• Create separate table for

many-to-one fields» Eliminates duplicate "child"

data

XML schema• Primary information store:

» Move data to separate document

» Too many docs => use RDBMS

• Secondary information store:» Create separate elements

» Duplicate data often not an issue

» Historic data is read-only


Third normal form (cont.)

Relational schema XML schema (1)ORDERS:OrderNum, Date, CustNum, AddressITEMS:OrderNum, ItemNum, Qty, PartNum, Price

<!ELEMENT Order (Number, Date, CustNum, Address, Item+)><!ELEMENT Item (ItemNum, Qty, PartNum, Price)>

ORDERS:OrderNum, Date, CustNumITEMS:OrderNum, ItemNum, Qty, PartNum CUSTOMERS:CustNum, AddressPARTS:PartNum, Price

<!ELEMENT Order (Number, Date, Customer, Item+)><!ELEMENT Item (ItemNum, Qty, Part)><!ELEMENT Customer EMPTY><!ATTLIST Customer xlink:href CDATA #REQUIRED><!ELEMENT Part EMPTY><!ATTLIST Part xlink:href CDATA #REQUIRED>


Third normal form (cont.)

XML schema (2)<!ELEMENT Order (Number, Date, CustNum, Address, Item+)><!ELEMENT Item (ItemNum, Qty, PartNum, Price)>

<!ELEMENT Order (Number, Date, Customer, Item+)><!ELEMENT Item (ItemNum, Qty, Part)><!ELEMENT Customer (CustNum, Address)><!ELEMENT Part (PartNum, Price)>


Referential integrity• Means logical pointers point to valid data

Case Referential integrityID/IDREF Enforced during validation

Key/keyref Enforced during validation

XLinks, proprietary links [1] Not enforced (problem!) [2]

External entity refs Not enforced (problem!) [2]

[1] Not widely supported[2] Not enforceable for entities outside database


Common Database Features


Round-tripping

• All databases can round-trip documents

• Round-trip level depends on database

• Text-based DBs should do exact round-tripping

• Model-based DBs round-trip at level of model» Minimum level is elements, attributes, PCDATA, and

document order» May be less than canonical XML (comments and

processing instructions may be discarded)


Document collections

• Contain related documents

• Similar to directory in file system ...» Documents can have any schema» Some databases allow nested collections

• ... or similar to table in relational database» Documents restricted to a single schema» Allows indexes and queries to be optimized


Indexes

• All databases use indexes» Structural indexes index element and attribute names» Value indexes index text and attribute values» Full text indexes index tokens in values

• Some databases index everything

• Other databases allow user to specify index fields


Query languages

• Many databases have proprietary languages

• XPath is most common standard language» Includes extensions for multi-document queries

• XQuery will be standard in the future


Updates

• Many databases only replace existing document

• Common update strategies:» Updates through live DOM» Update language. Use XPath to identify node, then:

• Insert data before/after node

• Modify node

• Delete node

» Extensions to XQuery

• Validity of updates often not enforced


APIs

• Similar to ODBC» Query language is separate from API» Connect, execute query, get results, commit/rollback» Results returned as string, DOM tree, or SAX events

• Often available over HTTP

• Most databases have proprietary APIs» XML:DB is database-neutral API» Oracle, IBM have submitted API through Sun JCP


Transactions, locking,and concurrency

• Most databases support transactions

• Locking usually at document (not fragment) level

• Whether this is an issue depends on» What is stored in a single document» Number of concurrent users

• Fragment locking more common in the future?


Links

• Few databases support XLink or proprietary links

• Some links are live (i.e. allow updates)

• Referential integrity not supported


External data

• Some databases can merge external data» Relational data (ODBC, JDBC, OLE DB)» Application data (SAP, PeopleSoft, Excel, etc.)» ...

• Whether data is live depends on database

• In the future, many databases will probably support live external data


External entity storage

• Not clear whether to store entity or URI» Storing entity incorrect if URI points to live data» Storing URI may be incorrect if entity is a snapshot

• Correct answer is probably to let user decide

• Not sure what is currently stored» All databases store URI of external entities?» Some databases also store BLOBs (unparsed entities)


Resources

• Ronald Bourret’s Papers Page» http://www.rpbourret.com/xml/index.htm

• XML / Database Links» http://www.rpbourret.com/xml/XMLDBLinks.htm

• XML:DB Initiative» http://www.xmldb.org/

• The XML Cover Pages: XML and Databases» http://xml.coverpages.org/xmlAndDatabases.html


Questions?

Ronald [email protected]://www.rpbourret.com

Native XML Databases

Documents

Transcript of Native XML Databases