1

Scaling Nuxeo ApplicationsBuilding scalable content apps with Nuxeo

Benoit Delbosc / Thierry Delprat

2

Performance questions ...

What CPU should I use to host 1TB data ?

Will 2 servers be enough for 1000 users ?

Can I run Nuxeo DM with 512 MB of heap ?

Is DELL PExyz ok to host Nuxeo ?

How can I ensure 0.1s response time ?

3

Why is this not that simple ?● Nuxeo EP is an ECM platform

● there are several ways to use it– hundreds of possible user actions– lot of screens

● there are several distributions● there are several possible confgurations

(security, fling plan, doc types ...)

● you can not size without knowing● the hypothesis ● the constraints

4

Agenda● Approach for managing performance● Nuxeo Architecture and performance● Performance testing● Performance tuning● Benchmark fgures

5

Ensuring good performance● Carefully defne your hypothesis

➔ what processing will be needed ?➔ batch processing vs interactive processing

➔ what data you will be processed ?➔ big image transformation vs simple page rendering

➔ what are the expectations ?➔ how many concurrent users ?➔ are users hyper-active ?

➔ be able to defne a usage scenario

6

Ensuring good performance● Defne your architecture according to

● constraints ● initial benchmark➔ mono server, multi-servers, cluster …

● Organize periodic benchmarks➔ ideally integrate in CI chain

● Plan some tuning➔ DB, memory, connections …➔ and code too

➔ this is not a one shot task

7

Impacting factors● Security policies

● ACLs inheritance and custom security policies

● Web Layer● Stateless vs Stateful● JSF traps and screen design

● Document types● complex document types impact DB schema

● Number of active documents● number of rows ● size of indexes vs Memory

8

Impacting factors● Simultaneous access

● how many concurrent requests/s● include batch processing

● Application server● Tomcat 6 is signifcantly faster than JBoss 5

● DB choice● OS

● a 32 bit OS is too limiting for JVM Memory● JVM seems to run faster under Linux 64

9

No (or low) impact factors● Total volume of binary fles

● only network and low level storage are impacted● Average number of document per folder

● VCS has not the same limitations as JackRabbit● Number of parallel sessions

● can only be impacting in JSF● Documents that are almost never accessed

● DB caches should do their job

10

Scaling Nuxeo ApplicationsArchitecture considerations

11

Architecture solutions

● 3 possibles axises

● Simple Clustering

● Spreading services on multiple JVMs

● Multiple repositories

12

VCS Cluster● VCS Cluster is simple

● only one confg parameter to turn on● do not rely on App Server level cluster● “Nuxeo Boxes” are swappable

● VCS Cluster scales well● as long as the backend DB scales !

➔ VCS Cluster is a good solution for both➔ scaling out➔ providing redundancy

13

1 node deployment

FS

DBNuxeoInstance

14

2 nodes deployment

NAS

DB

NuxeoInstance

NuxeoInstance

NLB

15

3 nodes deployment

NAS

DB

NuxeoInstance

NuxeoInstance

NuxeoInstance

NLB

16

Multi VM deployment● Nuxeo Services can be spread across JVMs

● externalize batch processing (mass i/o)● externalize heavy transformations● externalize slow interaction with external app● …➔ build dedicated processing servers

● Nuxeo services can be coupled with a GRID● integration with GRID Gain

17

Multi VM : mono node

FS

DBNuxeoInstance

batch

18

Multi VM : 2 nodes

FS

DBNuxeoInstance

batchNuxeo

Instance

19

Multi VM : GRID

FS

DBNuxeoInstance

batchNuxeo

Instance

batchNuxeo

Instance

batchNuxeo

Instance

GRID

20

Multi VM : perspectives● Technically each service could be accessed

remotely via RMI (on JEE server)● rendering layer● wf engine ● …

● But it's a little bit more complex● Tx management● binding confguration

➔ In most case VCS Cluster is much more simple

21

Multi-Repositories● A single Nuxeo Application can be bound to

several repositories● Each repository

● has its own database and cache➔ scale out solution if DB is the bottleneck

➔ Useful to do data partitioning➔ manage documents with different constraints

(ex: Live documents and archives)

➔ manage documents for different user groups(ex: multi-tenant)

22

Multi-Repositories

Live documents

Archived documents

Repo1

Repo2

Database2

FS2

Database1

FS1

NuxeoServer

23

Scaling Nuxeo ApplicationsPerformance testing

24

Performance testing overview

25

Setup a realistic environment

● Replicate as much as possible the production environment

● Use historical data when possible● Populate the database

● Custom mass importer tools (nuxeo-platform-importer)

● Load generating tools● Feed the database at the SQL level

26

Monitoring● Performance testing without monitor don't bother● Collecting general information:

● Hardware metrics: type and number of cpu, memory, disk usage

● Software versions: os, jvm, middlewar, application● Application confguration: nuxeo.conf● Database confguration and database statistics

27

Monitoring CPU● Is the CPU a bottleneck ?● Is the CPU waiting for IO ?● Does the system use all

the available CPU ?

28

Monitoring Disk● Is there a device

saturation ?● Writing or reading

operations ?

29

Monitoring GC● Is the JVM spend too

much time in the garbage collector ?

30

Monitoring web request processor

● How many requests ?● How much input/output ?

31

Monitoring web thread pool● Is the thread pool a bottle

neck ?

32

Monitoring datasources● Is there enough

connection in the pool ?

33

Monitoring JVM● How much JVM threads ?● After a longevity testing, is

there a memory leak ?

34

Monitoring database● How much time is spent

processing SQL ?● Which query took up the

most time ?

35

Setup the monitoring● Use your production monitoring

(nagios, hyperic, sysstat …)

● Setup GC log in the nuxeo.conf fle:JAVA_OPTS=$JAVA_OPTS -Xloggc:$DIRNAME/../log/gc.log -verbose:gc -XX:+PrintGC

● Setup the monitor JBoss templates in the nuxeo.conf fle :

nuxeo.templates=postgresql,monitor

● Use monitorctl.sh./jboss/bin/monitorctl.sh

Usage: monitorctl.sh (start|stop|status|heapdump [TAG]|info|vacuumdb|help)

36

Load generating tools● The application has to cooperate a bit to ease

test writing● Tools

● In-house tools● Vendor tools● Open source tools

– JMeter (GUI script, java)– FunkLoad (Python)

37

Writing test scriptsTry to simulate the expected user actions

38

Performance report● Speed (response time)

● Throughput (requests/s)

● User satisfaction (Apdex)http://www.apdex.org/

39

Monitoring report● All in one logchart.py

http://svn.nuxeo.org/nuxeo/tools/qa/logchart/trunk

● Other tools:

kSar, pgfouine, gcviewer ...

40

Scaling Nuxeo ApplicationsTuning

41

Tuning process

42

Jboss/Tomcat tuning● JVM, heap size (nuxeo.conf)

JAVA_OPTS=$JAVA_OPTS -Xms2g -Xmx2g

● In special case you may remove SoftRefJAVA_OPTS=$JAVA_OPTS -XX:SoftRefLRUPolicyMSPerMB=0

● Datasource connection pool (nuxeo.conf)nuxeo.db.max-pool-size=40nuxeo.vcs.max-pool-size=40

● HTTP or AJP Thread pooling <Connector port=”8080” … maxThreads=”32” ... acceptCount=”256” .. />

43

Database tuning● Read the Nuxeo KB

https://doc.nuxeo.com/display/KB/Confguring+PostgreSQL

● EXPLAIN ANALYZE helper: http://explain.despez.com

● Check for missing index on custom schema

● PostgreSQL Performance ml mailto:pgsql-performance@postgresql.org

44

Scaling Nuxeo ApplicationsBenchmarking results

45

Mass injection● Using load generating tools:

from 3 to 10 doc/s

works fne up to 100k docs(otherwise it takes to much time)

● Using nuxeo-platform-importer:from 30 to 100 doc/s

works fne up to 1m docs (otherwise it takes to much time)

● Using SQL injection :from 1000 to 3000 doc/s

46

Document retrieval and insertion operations● Nuxeo DM 5.4 / tomcat / Sun JVM 6 (Heap 3G)

● 10M of documents, 1TB of data

● Dell PE 2900 2xQuad-Core 20g RAM / Linux

● http://public.dev.nuxeo.com/~ben/bench-10m-tomcat/

Operation Speed (s) Throughput (req/s) Extrapolation VU

JSF View of a random document

0.6 25 250

JSF View of a cached document

0.2 30 300

Web Engine navigation

0.1 100 1000

JSF Creating new file

0.8 16 160

47

Thank you!