Plone TuneUp challenges
-
Upload
andrew-mleczko -
Category
Technology
-
view
3.084 -
download
1
description
Transcript of Plone TuneUp challenges
![Page 1: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/1.jpg)
1
Plone TuneUp challenges
in search of enterprise intranet solution
Andrew Mleczko (redturtle.net)
European Plone Symposium 2009
Sorrento, Italy
![Page 2: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/2.jpg)
2
PoznanPoznan FerraraFerrara
![Page 3: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/3.jpg)
3
● Co-founder of ACC Poland Ltd., the first Polish
Plone company (past)
● Plone consultant at RedTurtle Technology -
based in Ferrara (present)
● Plone-based and Lotus-based web applications
![Page 4: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/4.jpg)
4
RedTurtle Technology
“Sharing knowledge
about knowledge sharing”
![Page 5: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/5.jpg)
5
So what can you expect...
● Plone is just a CMS?
● Case study – Intranet Provincia di Ferrara
● Loadtests environment
● Plone TuneUp challenges
![Page 6: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/6.jpg)
6
How we can use
Plone?
![Page 7: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/7.jpg)
7
Plone variety
● Community portals
● Document management system (DMS)
● Groupware - Collaboration
● Subsite management system
● Enterprise intranets
● ...
● Combination of all mantioned above
![Page 8: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/8.jpg)
8
Plone as CMS
vs.
Plone as framework
![Page 9: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/9.jpg)
9
Why?
“Plone is NOT a general web development framework.“
/Carlos de la Guardia/
![Page 10: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/10.jpg)
10
Plone can be SLOW*
*when using the wrong way
![Page 11: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/11.jpg)
11
Framework comparision
● Used hardware:
– CPU: AMD OpteronT Processor 146 (2 GHz)
– Memory: 2 GB
● Testing with Siege 2.68
● Read scenario
● Results for 50, 200, 300 Concurent Users (CU)
● More details: www.alrond.com/en/2007/jan/25/performance-test-of-6-leading-frameworks
![Page 12: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/12.jpg)
12
Comparision
Django RubyOnRails TurboGears0
0,5
1
1,5
2
2,5
50 CU200 CU300 CU
Response time, secs.
![Page 13: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/13.jpg)
13
Comparision
Django RubyOnRails TurboGears0
5
10
15
20
25
30
35
50 CU200 CU300 CU
Longest transaction, secs.
![Page 14: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/14.jpg)
14
So where is Plone?
![Page 15: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/15.jpg)
15
Far, far away
Django RubyOnRails TurboGears Plone0
2
4
6
8
10
12
14
16
18
50 CU200 CU300 CU
Response time, secs.
Django RubyOnRails TurboGears Plone0
10
20
30
40
50
60
50 CU200 CU300 CU
Longest transaction, secs.
![Page 16: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/16.jpg)
16
But comparing with Zope2
Django + Psyco RubyOnRails 1.2.1 TurboGears Plone Zope20
2
4
6
8
10
12
14
16
18
50 CU200 CU300 CU
Response time, secs.
![Page 17: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/17.jpg)
17
What can we do...
● Do not treat Plone as a framework
● For some use cases:
use ContentMirror + fast framework (django,
pylons, repoze)
● But what about enterprise intranet solutions?
![Page 18: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/18.jpg)
18
Plone for enterprise intranet
● Large number of users (more than 500) and
documents (more than 100 000)
● A lot of simultaneous write requests (>20 CU/s)
● Using of caching limited
● Slowest acceptable resp. time <10 seconds
the limit for keeping the user's attention focused [Miller 1968; Card et al. 1991]
![Page 19: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/19.jpg)
19
Intranet Provincia di Ferrara
Case study
![Page 20: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/20.jpg)
20
Prerequisites
● 800 users stored in Domino Server (by PAS)
● 170.000 documents
● Yearly growth: ca 25.000 documents
● Peak traffic: 20 CU/s - write transactions
(based on Domino logs)
![Page 21: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/21.jpg)
21
Objective
● User response time (mixed read/write):
– Slowest (5-6 seconds)
– Recommended (2-4 seconds) 90% of all request
– Fast (<2 seconds)
![Page 22: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/22.jpg)
22
Test environment
● Hardware (current):
– 4 dual-core Intel Xeon 2.8 Ghz
– 32 GB RAM
● Amazon EC2 (future)
![Page 23: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/23.jpg)
23
Test environment
● collective.loadtesting
– funkload
– collective.funkload
● PageTemplate Profiler
● mr.bent
● Munin + redturtle.munin
![Page 24: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/24.jpg)
24
class Readonly(testcase.FLTestCase): def setUp(self): self.server_url = 'http://localhost:8080/plone'
def test_ReadOnly(self): server_url = self.server_url self.get(server_url + "/plone/login_form", description="Get /plone/login_form")
self.post(server_url + "/plone/login_form", params=[ ['came_from', server_url + '/plone'], ['form.submitted', '1'], ['cookies_enabled', ''], ['login_name', ''], ['pwd_empty', '0'], ['__ac_name', 'admin'], ['__ac_password', 'admin'], ['submit', 'Accedi']], description="Post /plone/login_form")
collective.loadtesting
![Page 25: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/25.jpg)
25
mr.bent
>>> from mr.bent.wrapper import mkwrapper>>> from mr.bent.tests.test_bent import callcounter>>> from Products.CMFPlone.CatalogTool import \ CatalogTool
>>> mkwrapper(CatalogTool.searchResults, callcounter, "catalogsearches")
![Page 26: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/26.jpg)
26
Test scenario
● Log in
● View random folder
● Create document (2 requests)
● View newly created document
● Publish it
● View home page
● Logout
![Page 27: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/27.jpg)
27
Log in Folder view Create document Save document View Publish Home page Logout
0
10
20
30
40
50
60
5CU20CU60CU
Default clean Plone
Response time, secs.
![Page 28: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/28.jpg)
28
Default clean Plone
Log in Folder view Create document Save document View Publish Home page Logout
0
10
20
30
40
50
60
70
80
90
100
5CU20CU60CU
% Errors (ConflictErrors)
![Page 29: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/29.jpg)
29
Let's start TuneUp and install
the basics:
Varnish + Pound (with sticky session) +
+ 6 ZEO Clients (18 threads) + ZEO Server
![Page 30: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/30.jpg)
30
● Apache – proxy/rewriting urls
● Varnish – cache resources, add http headers
● Pound – balancing with sticky sessison
● 6 ZEO clients
● ZEO Server
![Page 31: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/31.jpg)
31
+ Varnish + Pound + ZEO
Log in Folder view Create document Save document View Publish Home page Logout
0
5
10
15
20
25
30
35
40
5CU20CU60CU
Response time, secs.
![Page 32: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/32.jpg)
32
+ Varnish + Pound + ZEO
Log in Folder view Create document Save document View Publish Home page Logout
0
10
20
30
40
50
60
70
80
90
100
5CU20CU60CU
% Errors (ConflictErrors)
![Page 33: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/33.jpg)
33
Comparing to default Plone
![Page 34: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/34.jpg)
34
Log in Folder view Create document Save document View Publish Home page Logout
0
5
10
15
20
25
30
35
40
Default PloneImprovment
Comparision
Response time, secs., 20 CU
![Page 35: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/35.jpg)
35
Where are the conflict errors
comming from?mainly portal_catalog
![Page 36: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/36.jpg)
36
Let's try:mount portal_catalog to different ZODB
and increase zodb cache on it
![Page 37: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/37.jpg)
37
● Install redturtle.catalogmount● Add zope-conf-additional:
<zodb_db catalog> mount-point /plone/portal_catalog container-class Products.CMFPlone.CatalogTool.CatalogTool cache-size 300000 <zeoclient> server ${zeoserver:zeo-address} storage 2 name catalogstorage var ${buildout:parts-directory}/instance1/var cache-size 400MB </zeoclient> </zodb_db>
![Page 38: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/38.jpg)
38
● And add zeo-conf-additional:
<filestorage 2> path ${buildout:directory}/var/filestorage/CatalogData.fs</filestorage>
![Page 39: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/39.jpg)
39
+ redturtle.catalogmount
Log in Folder view Create document Save document View Publish Home page Logout
0
5
10
15
20
25
5CU20CU60CU
Response time, secs.
![Page 40: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/40.jpg)
40
+ redturtle.catalogmount
Log in Folder view Create document Save document View Publish Home page Logout
0
10
20
30
40
50
60
70
80
90
100
5CU20CU60CU
% Errors (ConflictErrors)
![Page 41: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/41.jpg)
41
Maybe there is a more
stable solution:collective.indexing
experimental.catalogqueryplan
archetypes.schematuning
![Page 42: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/42.jpg)
42
Log in Folder view Create document Save document View Publish Home page Logout
0
2
4
6
8
10
12
14
16
5CU20CU60CU
+ collective.indexing + archetypes.schematuning
Response time, secs.
![Page 43: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/43.jpg)
43
+ collective.indexing + archetypes.schematuning
Log in Folder view Create document Save document View Publish Home page Logout
0
10
20
30
40
50
60
70
80
90
100
5CU20CU60CU
% Errors (ConflictErrors)
![Page 44: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/44.jpg)
44
So what is the improvment?
![Page 45: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/45.jpg)
45
Improvment
Log in Folder view Create document Save document View Publish Home page Logout
0
5
10
15
20
25
30
35
40
fresh Ploneafter TuneUp
Response time, secs., 20 CU
![Page 46: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/46.jpg)
46
Let's try to scale it...
![Page 47: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/47.jpg)
47
ZEO scalability problem
Log in Folder view Create document Save document View Publish Home page Logout
0
5
10
15
20
25
30
35
40
45
1:104:206:1810:30
Response time, secs. 20 CU
![Page 48: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/48.jpg)
48
ZEO scalability problem
Log in Folder view Create document Save document View Publish Home page Logout
0
10
20
30
40
50
60
1:104:206:1810:30
% Errors (ConflictErrors), 20 CU
![Page 49: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/49.jpg)
49
Plone intranet limits
● ZODB is slow when handling concurrent writes
transactions
● Scalability problem using flat ZEO
● Missing 'out of the box' performance
(comparing to other framworks, CMS)
![Page 50: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/50.jpg)
50
How we can solve the problem?
![Page 51: Plone TuneUp challenges](https://reader034.fdocuments.in/reader034/viewer/2022042713/5482b406b07959380c8b4861/html5/thumbnails/51.jpg)
51
Ideas
● RelStorage for scalability problem
● collective.solr
● others ?
● share knowledge at plone-enterprise mailing list