Post on 01-Nov-2014
description
© 2014 Impetus Technologies - Confidential1
Impetus Technologies Inc.
Why we built Kundera - The Polyglot Object Mapper for
NoSQLs?Vivek Shrivastava
software architect
© 2014 Impetus Technologies - Confidential2
Evolution of Data
© 2014 Impetus Technologies - Confidential3
10 LET A = 1
20 LET B = 2
30 LET C = A + B
40 PRINT C
50 END
Data
First Introduction To Data
© 2014 Impetus Technologies - Confidential4
Relational Databases
• Tables
• Relations
• SQL
• Transactions
• Normalization
© 2014 Impetus Technologies - Confidential5
Current Landscape
http://blogs.the451group.com/information_management/2014/03/18/updated-data-platforms-landscape-map-february-2014/
© 2014 Impetus Technologies - Confidential6
Infrastructure
Datastores Designed To Support Partitions
© 2014 Impetus Technologies - Confidential7
Relationships / Transaction
Datastores Designed To Support Aggregates
Item 1
Order
Item 2
© 2014 Impetus Technologies - Confidential8
Consistency
DC1
N1
DC2
N2
DC3
N3
Transactional Consistency
Replication Consistency
T=0, user1 fetches O
T=1, user2 fetches O
T=2, user1 edits and saves O
T=3, user2 edits and saves O
© 2014 Impetus Technologies - Confidential9
Consistency
Availability
Partition Tolerance
Not Possibl
e
CAP Theorem
Datastores Designed To Support Eventual Consistency
© 2014 Impetus Technologies - Confidential10
Shape
Datastores Designed To Support Richer Data Models
1 David 34
2 John 46
3 Lisa 32
4 Steve 46
5 JP 49
Tables
Graph
Columnar
Key-Value
Document
© 2014 Impetus Technologies - Confidential11
Cluster/Cloud FriendlySchema less Aggregates With
Redefined Transactions
Eventual ConsistencyRicher Data Model With Minimum Impedance
Mismatch
NoSQL
Finally
© 2014 Impetus Technologies - Confidential12
Emergence of a Polyglot World
© 2014 Impetus Technologies - Confidential13
A Single Application Using Multiple Data Storage Technologies
Polyglot Persistence
© 2014 Impetus Technologies - Confidential14
Data streams
Cassandra, HBase
User information
RDBMS
In memory cache/session
Redis
Orders / Articles / Tickets
MongoDB, Couchbase
User Social graph
Neo4J, Titan
Batch processing & analytics
Hadoop, Spark
Polyglot World
© 2014 Impetus Technologies - Confidential15
Decision Implementation
Is It Easy?
No. It’s A Bumpy Ride.
© 2014 Impetus Technologies - Confidential16
Decision
Degree of Polyglot? • X% RDBMS• Y% NoSQLComposition of Polyglot? • Datastores?
Type of Polyglot?
• Don’t choose for pride, passion and trends
• Don’t start with more than 1 NoSQL store
• Don’t start with 100% uses cases
© 2014 Impetus Technologies - Confidential17
Degree of Polyglot
Scale Relational Behavior RDBMS Non-RDBMS
Low Any 100% 0%
High High 90% 10% (New SQL)
High Moderate 75% 25% (NoSQL)
High Low 50% 50-100% (NoSQL)
© 2014 Impetus Technologies - Confidential18
RDBMS NoSQL NewSQL
Datastore + Cache Local + Global Client + Server
Composition of Polyglot - Forms
© 2014 Impetus Technologies - Confidential19
Composition of Polyglot - Datastores
Neo4j, Titan, Objectivity
Cassandra, HBase, BigTable
Oraclekv, Redis, Riak
MongoDB, Couch
base
Graph Columnar
Key-value
Document
Data Model / QueriesConsistency / Performance
Architecture / Non-Functional
© 2014 Impetus Technologies - Confidential20
Operating In The Polyglot World
© 2014 Impetus Technologies - Confidential21
Weak Polyglot
Data LayerReporting Lane
(MySQL)
Orders Lane(MongoDB)
Log Lane(Cassandra)Se
rvic
e La
yer
Web
Lay
er
© 2014 Impetus Technologies - Confidential22
Strong Polyglot
Data Layer
Polyglot MapperReports(MySQL)
+Orders
(MongoDB)+
Logs(Cassandra)Se
rvic
e La
yer
Web
Lay
er
© 2014 Impetus Technologies - Confidential23
Harmonization
Query / Indexing
Operational
Mapping / Interaction
CRUD
Atomicity
Switching / Migrating
© 2014 Impetus Technologies - Confidential24
And We Built Kundera
© 2014 Impetus Technologies - Confidential25
What It Is?
?Object Mapper• RDBMS
• NoSQLs (Cassandra, MongoDB, Hbase, Redis, Oracle NoSQL, CouchDB, Neo4j)
JPA Compliant
Polyglot
© 2014 Impetus Technologies - Confidential26
JPA
Customer(MySQL)
Order 1 (MongoDB)
{Item11, Item12}
Order 2 (MongoDB){Item21, Item22}
One to
many
Aggregate
© 2014 Impetus Technologies - Confidential27
<!-- Persistence unit for Cassandra persistence --><persistence-unit name=“cassandrapu"> <provider>com.impetus.kundera.KunderaPersistence</provider> <class>com.impetus.kvapps.entities.Order</class> <exclude-unlisted-classes>true</exclude-unlisted-classes> <properties> <property name="kundera.nodes" value="localhost" /> <property name="kundera.port" value="9160" /> <property name="kundera.keyspace" value=“orderstore" /> <property name="kundera.dialect" value="cassandra" /> <property name="kundera.client.lookup.class" value="com.impetus.client.cassandra.thrift.ThriftClientFactory" /> <property name="kundera.ddl.auto.prepare" value="create" /> <property name="index.home.dir" value="lucene"/> </properties></persistence-unit>
<!-- Persistence unit for Mongo persistence --><persistence-unit name=“mongopu"> <provider>com.impetus.kundera.KunderaPersistence</provider> <class>com.impetus.kvapps.entities.Invoice</class><exclude-unlisted-classes>true</exclude-unlisted-classes> <properties> <property name="kundera.nodes" value="localhost" /> <property name="kundera.port" value="27017" /> <property name="kundera.keyspace" value=“invoicestore" /> <property name="kundera.dialect" value="mongodb" /> <property name="kundera.client.lookup.class" value="com.impetus.client.mongodb.MongoDBClientFactory" /> <property name="kundera.ddl.auto.prepare" value="create" /></properties> </persistence-unit>
//create entity manager factory. EntityManagerFactory emf = Persistence.createEntityManagerFactory(“cassandrapu,mongopu”, properties); EntityManager em = emf.createEntityManager(); ….. em.persist(order);
Configuration : Persistence.xml
Persist Data
At Its Simplest
© 2014 Impetus Technologies - Confidential28
Supports all major datastores and
Extensible
Polyglot CRUD / Query / Object
Mapping
Schema Generation /
Atomicity / Lazy and Eager fetching
REST Based AccessInterceptors /
Events / Caching / Connection Pool
JPQL / Native Query / Fallback
Indexing Flexibility
Additionally
© 2014 Impetus Technologies - Confidential29
Commitment to the Polyglot
Kundera.js (Polyglot JavaScript Persister)
Polyglot Data Viewer
Support for more datastores
© 2014 Impetus Technologies - Confidential30
Thank you.Questions??
https://github.com/impetus-opensource/Kundera