Post on 20-Aug-2015
Agile NoSQL with XRXAgile NoSQL with XRX
Dan McCrearyPresidentPresidentDan McCreary & Associatesdan@danmccreary.com(952) 931 9198
M
D (952) 931-9198D
Metadata Solutions
Session DescriptionWhat if all members of your software development team from Project Managers Business Analysts Testing and Documentation Project Managers, Business Analysts, Testing and Documentation members could create and modify web applications and web services? With traditional SQL solutions this was difficult because f th d t t b t bj t bj t t t bl of the need to convert web pages to objects, objects to tables as
well as the reverse functions. But now with native XML databases and drag-and-drop forms builders, data can flow from the XML model of a web form to the database and back again without translation. This radically simpler process combined with standardized query languages makes it easier for non-standardized query languages makes it easier for nonprogrammers to build and maintain their own applications and web services.
M
D Copyright 2011 Dan McCreary & Associates2 of
N
During this session viewers will:• Understand the challenges associated with traditional four-
translation web/object/RDBMS systemstranslation web/object/RDBMS systems• Understand the elegant simplicity of zero-translation systems• Understand the role of web standards in the XRX environment• See how each component of XRX plays key roles in the
application• See how organizations have leveraged XRX for agile web
development• See variations of XRX architectures• Create and objective process to evaluate the benefits of XRXj p• Get resources for building XRX pilot projects
M
D Copyright 2011 Dan McCreary & Associates3
Executive Summary• Schema-free, zero translation NoSQL
systems have the ability to have a bigimpact on overall system agilityp y g y
• XRX systems dramatically increase overall agility and can also empoweroverall agility and can also empower non-progammers to build and maintain NoSQL systemsNoSQL systems
M
D Copyright 2011 Dan McCreary & Associates4
Background for Dan McCreary• Circuit Designer at Bell Labs• UNIX/Supercomputers• UNIX/Supercomputers• NeXT Computer (Steve Jobs)• Owner of 75-person software consulting firm
with a focus on Object-Oriented systems andwith a focus on Object Oriented systems and Object-relational mapping frameworks
• US Federal data integration (National Information Exchange Model NIEM.gov)
• NativeXML/XQuery for metadata management since 2006
• Advocate of web standards, OpenSource, N SQL d XRX tNoSQL and XRX systems
• W3C invited expert (forms working group)• Likes functional programming
M
D Copyright Kelly-McCreary & Associates, LLC5
Origins: The Humble Data Dictionary
M
D 6Copyright 2011 Kelly-McCreary & Associates
Electronic Certificate of Real Estate
Summer 2006
1 Document= 44 SQL inserts
M
D 7Copyright 2011 Kelly-McCreary & Associates
250 Data ElementsXFormsMockupMockup
M
D 8Copyright 2011 Kelly-McCreary & Associates
Four Translations
T1 T2
T3T4
• T1 HTML into Java Objects
Object MiddleTier
RelationalDatabaseWeb Browser
• T1 – HTML into Java Objects• T2 – Java Objects into SQL Tables
T Tables into Objects• T3 – Tables into Objects• T4 – Objects into HTML
M
D 9Copyright 2011 Kelly-McCreary & Associates
Kurt's SuggestionUse a
A Native XML
Web Form
A Native XMLDatabase!
Save
Web Browser
Kurt Cagle
store($collection, $file-name, $data)eXistM
D 10Copyright 2011 Kelly-McCreary & Associates
eXist
Zero Translation
XForms
• XML lives in the web browser (XForms)
Web Browser XML database
• XML lives in the web browser (XForms)• REST interfaces• XML in the database (Native XML, XQuery)( , y)• XRX Web Application Architecture• No translation!
M
D 11Copyright 2010 Dan McCreary & Associates
Key Question: Impact on Agility• What impact do zero translation
N SQL t hNoSQL systems have on system agility?
• Agility: the ability to quicklyAgility: the ability to quickly react to changing business requirements at any stage of the software development lifecycleQuestion: Big impact or little• Question: Big impact or little impact?
• Answer: Big impactM
D
Answer: Big impact
Copyright 2011 Kelly-McCreary & Associates12
No-Shredding!
My FormData
• Relational databases take a single hierarchical document and shred it into many pieces so it will fit indocument and shred it into many pieces so it will fit in tabular structures
• Many document-oriented NoSQL databases prevent this shredding
M
D Copyright 2008 Dan McCreary & Associates13
g
Is Shredding Really Necessary?
• Every time you takeEvery time you take hierarchical data and put it into a traditionalput it into a traditional database you have to put repeating groups input repeating groups in separate tables and use SQL “joins” touse SQL joins to reassemble the data
M
D Copyright 2008 Dan McCreary & Associates14
Many Processes Today Are Driven By…
The constraints of yesterday…
Challenge:Ask ourselves the question…D t th d f l i bl ith t b l d tDo our current method of solving problems with tabular data…Reflect the storage of the 1950s…Or our actual business requirements?What structures best solve the actual business problem?
M
D Copyright 2008 Dan McCreary & Associates15
p
"Schema Free"• Systems that automatically determine
how to index data as the data is loaded into the database
• No a priori knowledge of data structureN d f f t l i l d t d li• No need for up-front logical data modeling– …but some modeling is still critical
• Adding new data elements or changing g g gdata elements is not disruptive
• Searching millions of records still has sub-second response timesecond response time
M
D 16Copyright 2010 Dan McCreary & Associates
Monoculture and Mono-architecture
M
D 17Copyright 2011 Dan McCreary & Associates
Image Source: Wikipedia Monoculture
Storage Architectural PatternsgTables Trees
T i l StarsTriples Stars
M
D 18Copyright 2010 Dan McCreary & Associates
Finding the Right MatchSchema-Free
Standards Compliant
Mature Query Language
M
D 19Copyright 2010 Dan McCreary & Associates
Use CMU's Architectural Tradeoff and Modeling (ATAM) Process
Architectural SummaryFour Translation Zero Translation
T T
HTML eb pages XForms Client
TT web browser XML databaseweb browser database
• HTML web pages• Object middle tier• RDBMS database
• XForms Client• Native XML Database
RDBMS database
Which system more agile and by how much?M
D 20Copyright 2010 Dan McCreary & Associates
How can this help us manage enterprise metadata?
Origins: The XML Data Dictionary
M
D 21Copyright 2010 Dan McCreary & Associates
Electronic Certificate of Real Estate
Summer 2006
1 Document= 44 SQL inserts
M
D 22Copyright 2010 Dan McCreary & Associates
250 Data ElementsXFormsMockupMockup
M
D 23Copyright 2010 Dan McCreary & Associates
The NO-SQL UniverseDocument StoresKey-Value Stores
XML
Graph Stores
Object Stores
M
D 24Copyright 2010 Dan McCreary & Associates
Three Core Processes1. Add new data to the database
1. Use XML example but JSON and other formats could also be used
2 Q h d2. Query the data3. Create an XML web service
• Analyze the effort for each stepCompare SQL and NoSQL XRX systems• Compare SQL and NoSQL-XRX systems
• Analyze impact on participation of non-programmers• Rate the relative agility of each system
M
D Copyright 2010 Dan McCreary & Associates25
It is Easy to Import DataSQL1. Analyze data for all parent child
relationships and repeating groups
XQuery1. Drag XML files into folder
p p g g p2. Design logical and physical ER
diagrams3. For each table create a Data Definition
File using a data definition language (DDL)(DDL)
4. Create indexes using DDL5. Create one table for each set of
repeating set of data6. Run DDL on database creating tables6. Run DDL on database creating tables
using the appropriate data types7. Create indexes8. Create Insert statements9. Create separate insert statements for
each repeating group10. Run Insert statements on primary
structures in database11. Use primary keys of the first data
inserts as foreign keys of dependantM
D 26
inserts as foreign keys of dependant data structures
The XML File system• XML File system – a way of
storing information in XML thatstoring information in XML that can be quickly searched
• You can drag-and-drop almost fil t thi fil tany files onto this file system
• You access it by using a WebDAV connector
• Microsoft Windows “My Network Places” function
M
D 27
Functional Programming
y = f(x)y = f(x)• Computer programs are like mathematical functions• Developers do not manipulate states and variables (things that
change value), but focus entirely on constants and functions g ) y(things that never change) and transformations of data
• Makes it very easy to build modular programs• Software written in functional programming languages tend to
be very concise and easy to port to highly parallel systemsbe very concise and easy to port to highly parallel systems
M
D Copyright 2011 Kelly-McCreary & Associates28
http://en.wikibooks.org/wiki/Computer_programming/Functional_programming
It's Easy to Query XML Data
SELECT COL1, Col2FROM TABLE
for $r in doc(‘t.xml’)//rowwhere col1=1FROM TABLE
WHERE COL1=1where col1 1
return $r/col1, $r/col2
Col1 Col21 A
<root><row>
<col1>1</col1><col2>A</col2></row>1 A
1 B1 C
<row><col1>1</col1><col2>B</col2>
</row><row>
1 C1 D
<col1>1</col1><col2>C</col2></row><row>
<col1>1</col1> <col2>D</col2>M
D 29
</row></root>
It is Easy to Create A Web Service
1 Learn Java or find a Java Developer 1. All XQuerys are web servicesThe Java/JDBC/SQL Way The XRX Way
1. Learn Java or find a Java Developer2. Install TomCat Web Server3. Install Java AXIS Web Server4. Write a JDBC program that sends
SQL queries to a databaseSQL queries to a database5. Get the results back in Java Result
Object structures6. Go through the Java Results
Structues and use print statements to rap XML tags aro nd the stringsto wrap XML tags around the strings in the result objects
7. Rename your class files to .jws files8. Add the .jws files to the TomCat
deploy foldersdeploy folders9. The WSDL files will automatically be
generated
M
D 30
Insert/Select/Publish ComparisonSQL
SQLSQLJava
TomcatAXIS
logicaldata
modeling
XQuerySQL XQuery
JDBC
XQuery
modeling
Insert Query Web Service NoSQL
M
D 31Total Effort
Steps for Adding a New Element• Add the element to the "new
instance" a form uses and setinstance a form uses and set the default value there
• Add the UI control
Takes about one minute
• Add the UI control• Optionally – run an XQuery
update to update existing dataupdate to update existing data or add an insert to the "enricher" when you get an
Takes about 10 minutes
y ginstance
M
D Copyright 2011 Kell-McCreary & Associates32
Six Translation – Web ServiceWeb Service
T T
T5 T6
T1
T
T2
T
• T1 – HTML into Java Objects
T4T3
Object MiddleTier
RelationalDatabaseWeb Browser
• T1 – HTML into Java Objects• T2 – Java Objects into SQL Tables• T3 – Tables into Objects• T4 – Objects into HTML
T Objects to XMLM
D
• T5 – Objects to XML• T6 – XML to Objects
33Copyright 2011 Dan McCreary & Associates
XML Stored in XForms ModelBrowser Database
model save
view
update
M
D Copyright 2011 Kelly-McCreary & Associates 34
XRX Core ProcessBrowser Database
model save/edit
view
update
M
D Copyright 2011 Kelly-McCreary & Associates35
Code Table ServicesClient Server
model Form Data
viewCode Table Service
C dall-codes.xq
CodeTables
M
DCopyright 2011 Kelly McCreary & Associates 36
Code tables are separated from form instance data
XRX Dynamic Forms GenerationApplication Server
Document
Client Application
User Team
Session
Form Data
XForms Model
Application Server
Form DataCollection
DataElementRegistry
StatusGroupRole
Binding Rules
Code Tables
Views
Code Table ServicesContextfilters
Suggest Services
Required
Read-only
Data TypesBusiness Rules Editor
Submissions
Calculations
Business Rules Editor
Constraints
Calculations Inference
XF Vi
Static Controls
Dynamic Controls
XForms View
Semantic Schemas
Subschema Service
XML Schema Registry
Constraint SchemasM
DDesign Time
Run Time
Semantic SchemasConstraint Schemas
Model Driven• XForms enables the
developer to reuse businessdeveloper to reuse business rules encapsulated in XML Schemas (xsd) and XML
XFormsApplication Schemas (xsd) and XML
Transforms (XQuery Typeswitch)
• XForms reduces duplication and ensures that a change in XML
Schema
MetaData
the underlying business logic does not require rewriting in another language
Schema Registry
M
D Copyright 2011 Kelly-McCreary & Associates 38
another language
View and Model are Trees• The view is a tree of a presentation
data elementModel
data element• Models are comprised of one or
more trees• XForms supplies the control layer
that moves data elements to and from the model
Control (Bind)from the model
• Users don’t have to worry about moving things to and from the
View (Presentation)g g
screen
M
D Copyright 2011 Kelly McCreary & Associates39
Models and View Are Linked with "Bind"
HTMLh d
xf:model
P
headbody
fPerson
Name
form
fieldset
first last
label
inputlabel
input<bind>
• Both the model and the views are trees of d t l t
input<bind>
M
D Copyright 2011 Dan McCreary & Associates40
data elements
Just “Do The Right Thing”HTML
headbody
xf:model
Person
body
form
PersonCurrentOnTaxes type="xs:boolean" fieldset
labellabel
inputlabel
input<bind>
PersonBirthDate type="xs:date"
• Data types from the model just do the right thing• Boolean variables become checkboxes
D t h d t l tM
D Copyright 2008 Dan McCreary & Associates41
• Dates have date selectors
Example of Automatic UI Generation
• All true/false data types (xs:boolean) automatically become a checkbox
• All dates (xs:date) have a date selector to the right of the date fieldthe date field
• All codes can be selected from listsM
D Copyright 2008 Dan McCreary & Associates42
from lists
Structure of a XForms File
• XForms tags are just XML Namespaces tags imbedded in a
standard XHTML file with
Namespaces
CSS Imports (View)
a different namespace• Most HTML form tags are
Model
Constraints (Bindings) gexactly the same but some attributes have been
( g )
UI (View)
promoted to be full elementsMyForm.xhtml
Submit Controls
M
D Copyright 2008 Dan McCreary & Associates43
y
REST• REpresentation State Transfer• Create applications based on well
designed URLs• Take advantage of web caching• Migrate toward Resource-OrientedMigrate toward Resource Oriented
Computing (ROC)• REST evangelists: RESTifarians• REST evangelists: RESTifarians
M
D Copyright 2011 Kelly-McCreary & Associates 44
Five RESTFull Friends
1. In-resident memory cache in your browser
2. You local hard drive cache3. Your local enterprise cache4 The cache on the web server farm4. The cache on the web server farm5. The cache on the database
Please make sure to check with your RESTfullfriends BEFORE you bother the database.
M
D Copyright 2011 Kelly-McCreary & Associates45
Shallow REST vs. Deep REST• You can start taking advantage of ReST
b j t d i ll th ht t URLbuy just doing well thought-out URL design
• To take advantage of deep ReST you must consider the subtleties of the HTTP protocol– GET vs POST vs PUT– DELETE
M
D Copyright 2008 Dan McCreary & Associates46
Benefits of REST• Provides improved response time• Reduced server load• Improves server scalabilityp y• Requires less client-side software• Depends less on vendor dependencies• Depends less on vendor dependencies• Promotes discovery • Provides better long-term compatibility• Better and evolvability
M
D Copyright 2008 Dan McCreary & Associates47
y
Sample XForms
M
D 48Copyright 2011 Dan McCreary & Associates
Requirements Editor
CodeTable
SelectionLists
R ti
Lists
RepeatingElements
M
D 49
Drag-and-Drop XForms Builders• Several options for
d d d fdrag-and-drop forms builder
• Open Source– Orbeon– BetterFORM
• CommercialCommercial– IBM Forms
M
DCopyright 2011 Kelly-McCreary & Associates
50
Application Modularity
M
D 51Copyright 2010 Dan McCreary & Associates
Financial Institution
M
D 52Copyright 2011 Dan McCreary & Associates
Federal Integrator
M
D 53Copyright 2011 Kelly-McCreary & Associates
Minnesota Historical Society
M
D 54Copyright 2010 Dan McCreary & Associates
XRX Apps Demo Site• Demo site for
danmccreary.com• Some programs written
under an hour with student data
• Several utility programs that start with template and add transformations to other formats
• Focus on metadata management
M
D Copyright 2011 Kelly-McCreary & Associates55
Structured Retrieval is Better
Introduction to Information Retrieval
by Christopher D. Manning, y p gPrabhakar Raghavan and Hinrich Schütze
C b id U i iCambridge University Press, 2008
http://nlp stanford edu/IR-book/information-retrieval-book htmlM
D 56
http://nlp.stanford.edu/IR-book/information-retrieval-book.html
Table 10.1 - Revised
RDB search unstructured retrieval
structured retrievalretrieval retrieval
objects records unstructured documents
trees with text at leaves
vector space & model relational model vector space & others XML hierarchy
main data t t table inverted index
trees with node-ids for structure table inverted index ids for document ids
queries SQL free text queries XQuery fulltext
XML - Table 10.1 and structured information retrieval. SQLRDB (relational database) search, unstructured information retrieval
M
D 57
unstructured information retrieval
Retain Document Structure in Search
"Bag of Words" "Retained Structure"
keywords
keywordsdoc-id
'love'
'hate''new'
keywordskeywords
keywordsnew
'fear' keywords
• All keywords in a single container• Only count frequencies are stored
with each word
• Keywords associated with each sub-document component
• Assign higher weight for titles andM
D
with each word Assign higher weight for titles and names
• Set by non-programmer 58
Empower the Non Programmer!Before XRX After XRX
SUPER PM, BA!,
Sorry, we have no ideawhat code 47 means.
Let me just search our XRX system…I'll have your answer in 150 milliseconds.
M
D 59Copyright 2011 Kelly-McCreary & Associates
Using the Right Architecture
FinishStart Finish
Find ways to remove barriers to empoweringthe non programmers on your teamthe non programmers on your team.
M
D 60Copyright 2011 Kelly-McCreary & Associates
Six "S"s of XRX Agility1. Semantics – build around shard metadata
i t iregistry services2. Search – structured search3. Standards – CSS, XML, XPath, XQuery,
XForms, XML Schemas4 S i ll XQ i REST i4. Services – all XQueries are REST services5. Solutions - that are quickly customized6. Super – Empower the non-programmers
M
D 61Copyright 2010 Kelly-McCreary & Associates
XRX Resources• XRX Wikipedia Page
– http://en.wikipedia.org/wiki/XRX(web_application_architecture)p p g ( _ pp _ )• XRX Resources
– http://www.danmccreary.com/xrx/• LinkedIn XRX GroupLinkedIn XRX Group
– http://www.linkedin.com/groups/XRX-Web-Application-Architecture-1056777
• Beginner's Guide to XRX– http://www danmccreary com/xrx/beginners-guide– http://www.danmccreary.com/xrx/beginners-guide
• XRX Wikibook– http://en.wikibooks.org/wiki/XRX
Google Code• Google Code– http://code.google.com/p/xrx/
M
D Copyright 2011 Kelly-McCreary & Associates62
References
XForms
XQXQuery
XRX
Send e-mail to dan@danmccreary com for extended
A Beginner's Guide to XRX
M
D
Send e mail to dan@danmccreary.com for extendedlist of "getting started" resources.
Copyright 2011 Kelly-McCreary & Associates63
Thank You!Dan McCreary, PresidentKelly-McCreary & Associatesdan@danmccreary.com(952) 931-9198
eXist Meeting Prague March 12th, 2010M
D Copyright 2007 Dan McCreary & Associates64
eXist Meeting Prague March 12 , 2010