upfront diet plone conf 2010
-
Upload
roche-compaan -
Category
Technology
-
view
1.592 -
download
1
description
Transcript of upfront diet plone conf 2010
![Page 1: upfront diet plone conf 2010](https://reader035.fdocuments.in/reader035/viewer/2022062710/5597a37f1a28abc9218b477e/html5/thumbnails/1.jpg)
upfront diet for plone
Roché Compaan
![Page 2: upfront diet plone conf 2010](https://reader035.fdocuments.in/reader035/viewer/2022062710/5597a37f1a28abc9218b477e/html5/thumbnails/2.jpg)
![Page 3: upfront diet plone conf 2010](https://reader035.fdocuments.in/reader035/viewer/2022062710/5597a37f1a28abc9218b477e/html5/thumbnails/3.jpg)
About me
● Owner of Upfront Systems● Founded Upfront Systems in 1998● Zope developer for the past 10 years
![Page 4: upfront diet plone conf 2010](https://reader035.fdocuments.in/reader035/viewer/2022062710/5597a37f1a28abc9218b477e/html5/thumbnails/4.jpg)
Experience
● Largest Plone deployments to date: 5FM & MetroFM● 5FM
– 20 million page views per month– 70 thousand members– 6 million objects in ZODB
● Metro FM– 15 million page views per month– 20 thousand members– 2 million objects in ZODB
● Helping Connexions with performance● http://www.markhorner.net/2010/04/29/connexions-authoring-
performance/
![Page 5: upfront diet plone conf 2010](https://reader035.fdocuments.in/reader035/viewer/2022062710/5597a37f1a28abc9218b477e/html5/thumbnails/5.jpg)
![Page 6: upfront diet plone conf 2010](https://reader035.fdocuments.in/reader035/viewer/2022062710/5597a37f1a28abc9218b477e/html5/thumbnails/6.jpg)
![Page 7: upfront diet plone conf 2010](https://reader035.fdocuments.in/reader035/viewer/2022062710/5597a37f1a28abc9218b477e/html5/thumbnails/7.jpg)
Can Plone scale to 10 million documents?
![Page 8: upfront diet plone conf 2010](https://reader035.fdocuments.in/reader035/viewer/2022062710/5597a37f1a28abc9218b477e/html5/thumbnails/8.jpg)
ZODB Benchmarks
● Most common myth spread: “ZODB is a high read, (s)low write database”
● No published ZODB benchmarks● Wrote some benchmarks that test ZODB
read/write speed in 2008● collective.zodbbench
![Page 9: upfront diet plone conf 2010](https://reader035.fdocuments.in/reader035/viewer/2022062710/5597a37f1a28abc9218b477e/html5/thumbnails/9.jpg)
ZODB Write Speed
0 500 thousand 2,5 million 5 million 7,5 million 8,5 million 10 million0
1000
2000
3000
4000
5000
6000
7000
8000
Postgres
ZODB
Number of records
Inse
rts/
seco
nd
![Page 10: upfront diet plone conf 2010](https://reader035.fdocuments.in/reader035/viewer/2022062710/5597a37f1a28abc9218b477e/html5/thumbnails/10.jpg)
ZODB Lookup Speed
Number of Objects Lookup Time100,000 0.003 ms
1,000,000 0.006 ms10,000,000 2 ms
![Page 11: upfront diet plone conf 2010](https://reader035.fdocuments.in/reader035/viewer/2022062710/5597a37f1a28abc9218b477e/html5/thumbnails/11.jpg)
Benchmark Conclusions
● Thanks Jim, ZODB Rocks! ● Large cache size important for writes too, not
only reads● Need a solution for more than 10 million objects● If the ZODB can handle 250 inserts/second at
peak volume why does it take a full minute to add 250 ATDocuments?
![Page 12: upfront diet plone conf 2010](https://reader035.fdocuments.in/reader035/viewer/2022062710/5597a37f1a28abc9218b477e/html5/thumbnails/12.jpg)
A closer look at Plone
● Added 10,000 documents to Plone, while profiling with Dieter Maurer's ZopeProfiler
● Timed “apply_index” method to find indexes performing badly with and without experimental.catalogqueryplan
● Batch import of content with an without references.
● Used fsdump to inspect pickle contents.
![Page 13: upfront diet plone conf 2010](https://reader035.fdocuments.in/reader035/viewer/2022062710/5597a37f1a28abc9218b477e/html5/thumbnails/13.jpg)
Plone Benchmark Conclusions
● portal_catalog remains a bottleneck to scaling a Plone site.
● 1 catalog == many many conflicts● reference_catalog is a major performance
bottleneck● Indexing 10,000 documents shows that 90% of
the objects in Data.fs relate to indexing.
![Page 14: upfront diet plone conf 2010](https://reader035.fdocuments.in/reader035/viewer/2022062710/5597a37f1a28abc9218b477e/html5/thumbnails/14.jpg)
What's in a pickle?
2,51755
81,564
9,872
20,02485
655
1,544
30,455
182,708
7798,088122
Pickle contents of a single transaction adding an ATDocument
object size in bytes
IIBTree
Folder
IOBucket
OIBucket
IIBucket
IOBTree
BTrees.Length
ATDocument
OOBTree
IISet
Archetypes.BaseUnit
OOBucket
IITreeSet
![Page 15: upfront diet plone conf 2010](https://reader035.fdocuments.in/reader035/viewer/2022062710/5597a37f1a28abc9218b477e/html5/thumbnails/15.jpg)
Think before you index!
“Waking up objects is expensive” - this is an irrational fear. And adding another catalog in
response to this fear requires therapy.
![Page 16: upfront diet plone conf 2010](https://reader035.fdocuments.in/reader035/viewer/2022062710/5597a37f1a28abc9218b477e/html5/thumbnails/16.jpg)
The portal_catalog problem
● No navigation, portal tabs, folder contents, review queue if your content type is not in the portal_catalog.
● Your content type is automatically subscribed to 20+ indexes in the portal_catalog. Bad for new developers with hopes of building an enterprise CMS.
● Unless you have 5+ years experience with Zope – it's not so easy to “opt out”
● portal_catalog should only be used for content searches – no functionality must depend on it.
![Page 17: upfront diet plone conf 2010](https://reader035.fdocuments.in/reader035/viewer/2022062710/5597a37f1a28abc9218b477e/html5/thumbnails/17.jpg)
What is upfront.diet?
● It's a healthier and happier approach to Plone development: index only when you really really need to.
● Set of packages that re-implement plone functionality without depending on the portal_catalog and reference_catalog.
● Includes replacements for: portal tabs, navigation portlet, folder contents, archetype references.
● Provides a catalog blacklist/whitelist to customise indexing per content type.
![Page 18: upfront diet plone conf 2010](https://reader035.fdocuments.in/reader035/viewer/2022062710/5597a37f1a28abc9218b477e/html5/thumbnails/18.jpg)
More about upfront.diet
● upfront.navportlet● Separate catalog for navportlet
● upfront.simplereferencefield● Store references directly on object using annotations● still depends uid_catalog
● upfront.foldercontents● Adapter for IFolderish● Index allowedRolesAndUsers directly on folder – less conflicts
● upfront.catalogblacklist● Generic Setup profile with a whitelist and blacklist of indexes per
content type● patches IndexableObjectWrapper, ugh!
![Page 19: upfront diet plone conf 2010](https://reader035.fdocuments.in/reader035/viewer/2022062710/5597a37f1a28abc9218b477e/html5/thumbnails/19.jpg)
upfront.diet todo list
● Packages:● Portal tabs (done but not a separate package yet)● Sitemap● Review queue● News and Events Smart Folders● Site Search
● References without uid_catalog● Comparative benchmarks with and without it
![Page 20: upfront diet plone conf 2010](https://reader035.fdocuments.in/reader035/viewer/2022062710/5597a37f1a28abc9218b477e/html5/thumbnails/20.jpg)
upfront.diet deployments
● Very few at the moment but looks promising● Fullmarks.org.za (half a million objects, Data.fs
< 1GB)
![Page 21: upfront diet plone conf 2010](https://reader035.fdocuments.in/reader035/viewer/2022062710/5597a37f1a28abc9218b477e/html5/thumbnails/21.jpg)
Contact Me
● Email: [email protected]● Blog:
http://upfrontsystems.co.za/Members/roche● facebook.com/rochecompaan● twitter.com/rochecompaan
![Page 22: upfront diet plone conf 2010](https://reader035.fdocuments.in/reader035/viewer/2022062710/5597a37f1a28abc9218b477e/html5/thumbnails/22.jpg)
Questions?