Using MongoDB to Build a Fast and Scalable Content Repository
Transcript of Using MongoDB to Build a Fast and Scalable Content Repository
![Page 1: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/1.jpg)
H Y P E R S C A L E D I G I T A L A S S E T P L A T F O R M
![Page 2: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/2.jpg)
U S I N G M O N G O D B T O B U I L D
A F A S T A N D S C A L A B L E
C O N T E N T R E P O S I T O R Y
![Page 3: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/3.jpg)
N U X E O P L A T F O R M
We provide a platform that organisations can use to
drive digital transformation by unlocking the full value of digital assets.
https://www.nuxeo.com
![Page 4: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/4.jpg)
LEADER ON DIGITAL LOCAL INFORMATION MANAGEMENT: Addresses, Phone Numbers, Activities of local businesses
USING NUXEO AS A CATALOG FOR INFORMATION ASSETS
HUNDREDS OF MILLIONS DOCUMENTS AND UP TO DAILY 1 M BULK UPDATE
ADAPT THE BUSINESS / INCREASE MARKET POSITION
INFORMATION UPDATE DELIVERY REDUCED FROM 1 YEAR TO 1H
![Page 5: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/5.jpg)
N U X E O P L A T F O R M B E N E F I T S
SIMPLE SOFTWARE ARCHITECTURE
EASY SCALABILITY
HYPERSCALE PERFORMANCE
TRANSACTION MANAGEMENT
The Nuxeo + MongoDB Benefits
![Page 6: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/6.jpg)
Large stream - Large storageattached blobs
Flexible Schema - Write Once/Read ManyAudit Log, activity log
Complex structures - R/W synchronousDocument properties and hierarchy
Flexible Schema - SearchSearch index
S I M P L E A R C H I T E C T U R E
![Page 7: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/7.jpg)
S I M P L E A R C H I T E C T U R E
Large stream - Large storageattached blobs
Flexible Schema - Write Once/Read ManyAudit Log, activity log
Complex structures - R/W synchronousDocument properties and hierarchy
Flexible Schema - SearchSearch index
![Page 8: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/8.jpg)
S C A L A B I L I T Y
![Page 9: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/9.jpg)
H Y P E R S C A L E P E R F O R M A N C E
The One Billion Benchmark
• 1 billion documents import in 8.5 hours
• 32,680 docs/s with peak at 40,400 docs/s
![Page 10: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/10.jpg)
S I M P L E A R C H I T E C T U R E
![Page 11: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/11.jpg)
N U X E O P L A T F O R M
DOCUMENT REPOSITORY
![Page 12: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/12.jpg)
No ORM Hell
S I M P L I C I T Y ?
Single consolidated storage
Out of the box robust deployment
![Page 13: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/13.jpg)
I M P E D A N C E M I S M A T C H
I S S U E
![Page 14: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/14.jpg)
I M P E D A N C E M I S M A T C H
I S S U E
![Page 15: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/15.jpg)
I M P E D A N C E M I S M A T C H
I S S U E
![Page 16: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/16.jpg)
I M P E D A N C E M I S M A T C H
I S S U E
NO LAZY LOADING
NO CACHE
NO INVALIDATION
A LOT OF COMPLEXITY AND PROBLEMS AVOIDED !
![Page 17: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/17.jpg)
I M P A C T O N D E P L O Y M E N T
![Page 18: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/18.jpg)
I M P A C T O N D E P L O Y M E N T
![Page 19: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/19.jpg)
C O N S O L I D A T E D S T O R A G E
Structures
AuditBlobs
Indexes
SINGLE CONSOLIDATED STORAGE Structure, Blobs, Audit & Index
FEWER BUILDING BLOCKS TO PROVISION & CONFIGURE Easier to deploy
![Page 20: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/20.jpg)
E A S Y D E P L O Y M E N T
"BUILT-IN" - DATA REDUNDANCY & FAULT TOLERANCE
Active
Active
![Page 21: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/21.jpg)
S C A L A B I L I T Y
WILL I SCALE BETTER WITH MONGODB ?
![Page 22: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/22.jpg)
S C A L A B I L I T Y
S C A L E O U T R E A D S
S C A L E O U T W R I T E S•Leverage sharding
•Spread Writes
•Leverages replicasets
•Read from secondaries
![Page 23: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/23.jpg)
S C A L A B I L I T Y
S C A L E O U T T E S T
Use massive read operations and queries.
2 Nuxeo nodes + 1 MongoDB node
1850 docs/sMongoDB CPU is the
bottleneck (800%)
![Page 24: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/24.jpg)
S C A L E O U T T E S T
Use massive read operations and queries.
2 Nuxeo nodes + 2 MongoDB nodes 3400 docs/s
(using read preferences) S C A L A B I L I T Y
![Page 25: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/25.jpg)
S C A L A B I L I T Y
S H A R D I N G T E S T
2 Nuxeo nodes +
1 MongoDB ReplicaSet
11,000 docs/s
![Page 26: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/26.jpg)
S C A L A B I L I T Y
S H A R D I N G T E S T
2 Nuxeo nodes +
3 MongoDB Sharded ReplicaSet 27,400 docs/s
![Page 27: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/27.jpg)
H Y P E R S C A L E P E R F O R M A N C E
WILL I BE FASTER WITH MONGODB ?
![Page 28: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/28.jpg)
B U I LT F O R S P E E D
N O I M P E D A N C E I S S U E
D O C U M E N T L E V E L L O C K I N G
• No table level concurrency
• Fewer backend calls
• No invalidation costs
N A T I V E D I S T R I B U T E D A R C H I T E C T U R E
• Easy scale out
![Page 29: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/29.jpg)
S P E E D
https://benchmarks.nuxeo.com/continuous/index.html
Significant RAW Speed improvements for all use cases
More importantly: some use cases are much better handled
![Page 30: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/30.jpg)
M O R E T H A N R A W
P E R F O R M A N C E
• No cache
• Less Memory per connection
• Can handle more connection
• Can handle more concurrent users
Handle more concurrent connections
![Page 31: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/31.jpg)
M O R E T H A N R A W
P E R F O R M A N C E
With SQL, Read and Write operation are competing
![Page 32: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/32.jpg)
M O R E T H A N R A W
P E R F O R M A N C E
Writes are not blocked by Reads
With MongoDB writes operations are not blocked
![Page 33: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/33.jpg)
M O R E T H A N R A W
P E R F O R M A N C E
Large object processing improved
lazy loadingcache trashing
750 documents/s 11,500 documents/s
![Page 34: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/34.jpg)
T R A N S A C T I O N M A N A G E M E N T
![Page 35: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/35.jpg)
MEANS• Different transaction paradigms
• Provide shared mitigation policies for critical use case
NEW STORAGE MODEL• Document Level transaction
• No MVCC isolation
T R A N S A C T I O N M A N A G E M E N T
![Page 36: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/36.jpg)
C O N S I S T E N C Y
I N O U R C O N T E X T
Transactions can not span across multiple documents
• Atomic Document Operations are safe • Large batch updates can not be Atomic
Multi-documents transactions can be problematic Workflows or custom event handlers
FIND A WAY TO MITIGATE APPLICATION LEVEL IMPACT
![Page 37: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/37.jpg)
C O N S I S T E N C Y
![Page 38: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/38.jpg)
C O N S I S T E N C Y
TRANSIENT STATE MANAGER Run all operations in Memory
Populate an Undo Log
Recover Application level Transaction Management• Commit / Rollback model
"Read uncommited" isolation• Need to flush transient state for queries• "uncommited" changes are visible to others
![Page 39: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/39.jpg)
T A K E A W A Y S Nuxeo +
MongoDB
Simple Architecture
Scalable & Performant
Transaction Management
Content Management + MongoDB You should try Nuxeo !
![Page 40: Using MongoDB to Build a Fast and Scalable Content Repository](https://reader035.fdocuments.in/reader035/viewer/2022062503/5871ab861a28abda6a8b589b/html5/thumbnails/40.jpg)
A N Y Q U E S T I O N S ?T H A N K Y O U !
https://github.com/nuxeo nuxeo.com/careers/ @damienmetzler