Funambol Server Architecture

Simple Blue Template

Funambol Server
Architecture

Stefano Nichele

Overview

Funambol server components

Synchronization overview

Data synchronization service architecture

Push overview

Push Connection service

InboxListener and PIMListener

Funambol Server Architecture

Core component of the Funambol Server. It provides synchronization and push enginesAllows push mail by checking email accountsAllows pim push by checking the databaseAllows Client TCP PushDataSynchronizationServerInbox ListenerPim ListenerCTP ServerWe will refer to those as services. The DS service is the primary service, without that the Funambol Server cannot run.And we are going to change some names....

DataSynchronizationservicePim ListenerservicePush ConnectionserviceInbox Listenerservice


DataSynchronizationservicePim ListenerservicePush ConnectionserviceInbox ListenerserviceDBMailServer

Synchronization

Push

Synchronization

Process to make equal two sets of data (repository, database)

One-way: data is coming just from one source (i.e. backup, restore)

Two-way: data is coming from both sources

PDABackupRepo

PDAMylaptop

Synchronization

Two different modes:

Full-sync: the sources are synchronized by comparing the whole sets of data (OMA terminology: slow-sync)

Fast-sync: the sources are synchronized by comparing just the changes that occurred since the last sync.

SyncML

Open standard for data synchronization

XML based

Supported by major companies such as Nokia, Motorola....Funambol :-)

Since 2002 defined by Open Mobile Alliance (http://www.openmobilealliance.org)

SyncML an example

1.2 SyncML/1.2 1 3 IMEI:004400061769830 http://syncserver.com/servlets/SyncML ... 1 3 0 SyncHdr http://syncserver.com/servlets/SyncML IMEI:004400061769830 200

Data Synchronization service

It's the core and the only required service of the Funambol server (primary service)

It governs synchronization and push

Extensible and highly configurable

SyncML 1.1 and 1.2


AdminManagerSynchr.EngineEmail Sync SourcePIM Sync SourceGoogle Sync SourceOfficerPipelineManagerNokia SyncletRazr SyncletPushEnginePush SenderWebServicesAPIPluginsManagerPlugin APlugin BFrameworkDevice InventorySMSServiceSMS Provider


PipelineManagerNokia SyncletRazr SyncletIt allows additional processing and manipulation of the messages exchanged between the server and the client.

The kind of processing that is performed in the pipeline is a message level processing, such as the manipulation of the message elements.

Possible applications are:

Encoding/decoding of item content

Item filtering

Item ordering

Message decoration adding/removing elements on a custom basis


The idea behind the message processing pipeline is to be able to modify both incoming and outgoing messages:

in the former case, we want to be able to manipulate the message before it goes into the sync engine

instead, in the latter, we want to be able to change the message returned by the sync engine before sending it to the client.

Input and output pipeline components are called Synclet.

PipelineManagerNokia SyncletRazr Synclet

Synchr.EngineEmail Sync SourcePIM Sync SourceGoogle Sync SourceOfficerSERVLET

Input pipeline

Output pipeline

Pipeline Manager

Rawdata

SyncML

SyncML


How are the synclets to perform selected ?

Synclets for any device

Synclets just for some devices: based on the user-agent of the requests

PipelineManagerNokia SyncletRazr SyncletExample of synclet configuration file

user-agent NokiaE61 com/funambol/server/engine/pipeline/phones-support/bsh/NokiaE65in.bsh


Main responsibilities:

authenticate and authorize the user

identify the sources and the destinations of data to be synchronized

identify what data needs to be updated/added/deleted

determine how updates must be applied

detect conflicts

resolve conflicts

Sync.EngineEmail SSPIM SSGoogle SSOfficer


Deep look at the synchronization engine

Sync.EngineEmail SSPIM SSGoogle SSOfficerSyncSessionHandlerSync4jEngineSync4jStrategyEmail Sync SourcePIM Sync SourceOfficer

SyncMLMessages

It represents the handler for a SyncML session. It coordinates and handles messages as dictated by the protocol.It coordinates the sync processdelegating almost all the synclogic to Sync4jStrategyGiven two sources of data,it performs the real synchronization processIt checks if the credentials provided by the client are validand if the user is authorized.A SyncSource is the interfaceversus the data store


How can you interact with the engine process and how can you synchronize your data repository ?

Officer: it's the component that authenticates and authorizes a user. It's important since implementing your own officer you are able to authenticate a user on your user database.

SyncSource: it's the class used to access to the server data repository. You can develop your own PIM syncsource but also your source for MP3 or for whatever you want synchronization.

For more details, see the Developer Guide.

Sync.EngineEmail SSPIM SSGoogle SSOfficer


SMSServiceSMS ProviderThe Data Synchronization service provides the required infrastructure to allow you an easy integration with your SMS Gateway.

SMS Service

it's the entry point for the SMS infrastructure

no specific SMS gateway logic

SMS Provider

it's the real interface towards your preferred SMS gateway (i.e. SubitoSMS or Clickatell)

SMSServiceSMS ProviderSMS Infrastructure

SMSGateway

Usually this is anHTTP requestSMS MessageRecipientSender


SMSServiceSMS ProviderWhat can I do in the SMS Service ?

decorate the recipient/sender with additional information (like carrier/country)

work as proxy towards more than one SMSProvider selecting the one to use based on the recipient

provide smart functionalities (for example using the delivery detail/status information, a sms can be discharged if the previous one is still pending)

Out of the box we provide a simple SMS Service that forwards requests to an SMS Provider that just logs the messages.


Plug-ins: configurable objects that can be deployed in the DS service in order to extend its functionalities. For instance a simple plug-in could send an email to the system administrator to notify him for startup and shutdown events.

By now the plug-ins are notified just for startup and shutdown events.

As for the other configurable components, they can be deployed without stopping the DS service just copying the xml bean file under com/funambol/server/plugin.

PluginsManagerPlugin APlugin B


It's the core of the administration API. It can be accessed locally or remotely using web services.

It exposes the administration API via web services.

It's the component that handles devices and their capabilities

AdminManagerWebServicesAPIDevice Inventory....the latest DS service components


DataSynchronizationservicePim ListenerservicePush ConnectionserviceInbox ListenerserviceDBMailServer

Push

Push

Push sequence in summary:

1. the Funambol server detects that a device needs to sync

2. the Funambol server sends a push message to the device. This is pkg #0 in the OMA specs

3. the device starts a sync so that its data is updated with the one on the server

FunambolServer

DataRepositoryDataRepository

123

Push

How can the server send the push message to the device ?

SMS seems the easiest way but SMSs are not free of charge

The client can run a TCP server so the server can open a TCP connection and sends the push message over that

The client can open a connection versus the server and keep it always open, ready for push message from the server

Funambol Server supports out of the box connection-less and connection-oriented push but SMS Push is easily implementable.

SMSPush

STPPush

CTPPush

SMSPush

Connection-lessPush

Connection-orientedPush

Push

Who does what ?

DSserviceMailServer

InboxListenerservicePIMListenerservice

PIMDBMailServer

WSAPI

PushEngine

Push Connectionservice

The Inbox Listener at regular time checks on the mail server if there are users to pushIf there are devices to push, the IL and the PIM listener call the DS service admin APIThe Push Engine tries to send the push message using connection-less pushThe Push Connection service tries to push the device using an existingconnectionThe Push Engine asks to the Push Connectionservice to push the device usingconnection-oriented pushThe PIM Listener at regular time checks on the DB if there are devices to push112345


PushEnginePush SenderMain responsibilities:

Handling push :-)

WSAdmin API

NotificationEngineNotificationBuilderNotificationSenderPush Engine

DB

Push request

The NotificationEngineuses a NotificationBuilderto create the Push messageThe NotificationEnginechecks in the DB if there are old push messagesThe sender is the component used to deliver a push message to a deviceIf the sender is not able to deliver a push message, it is stored in the database1234


SMSServiceSMS ProviderBuilder and sender configurable per device

Two available builder implementations (for SyncML 1.1 and for SyncML 1.2)

Just one sender implementation that tries:

connection-less push

connection-oriented push (if the previous one fails)

PushEnginePush Sender

Connection-less push

Main concepts:

the client must be able to run a TCP server waiting for connections from the server

when a push message must be sent, the DS service opens a socket and writes the message

the DS service must know the client IP address

the IP address must be reachable from the DS service


AddressListenerServlet

DB

IP Address

Pending Push

IP Address

If the server is able to send the pending push to the client, the IP address is stored in the DB.123Address notification: process used by the client to notify its IP address

The device must notify its address any time it changes ! So it must be able to detect when it changes.

The device sends its IP address to the serverThe server verifies if the client address is reachable trying to send the pending push messages


Issues:

The client must be able to run a TCP server

The client must be able to detect when its IP address changes

The IP address must be reachable from the server

Connection-oriented push

Main concepts:

The client opens a connection on the server and keeps it open

The server uses that connection to deliver push messages

...really simple ;-)


Push ConnectionService

1The client opens the connection and sends its credentialsDSserviceWSAPIPushEngine2

The Push Connection service calls the DS service WS API to authenticate the client

UDP channel

The Push Sender sends a multicast UDP message with the push message21The Push Connection service receives the UDP message and checks if the recipient is connected3If the device is connected, the push message is written on the already open (by the client)connection

Connecting

Pushing

Push Connection service

Main challenge:

It must be able to handle millions of connections

Features:

Java Standard Application

Based on javax.nio

Based on Apache Mina (network application framework)

Clusterable

InboxListener and PIMListener

Both do the same work, at regular time they must check the data repository

Push ListenerFrameworkInboxListenerTaskInboxListener

Push ListenerFrameworkPimListenerTaskPIMListener

Both based on the PushListener Framework...

Push Framework Listener

Framework ready to use to implement any listener. By now used for:

InboxListener

PIMListener

ExchangeListener

OXListener

Pretty similar to a scheduler with basic functionalities for push and for HA/scalability

Java Standard Application (no J2EE)

Based on Jdk 1.5 thread features

JMX interface

Pluggable architecture


Hey, this is not push.....it's polling !

InboxListenerserviceMailServer

PollingmodeInboxListenerserviceMailServerNotifiablemode

UDP notification

Polling mode works with all mail servers

Notifiable mode can be used just in some cases


DBRegistry Monitor

...

...

User 1

User 2

User 3

User 4

Thread AThread BThread CThread DTask executor

Tasksqueue

Plugin Manager

UDP ListenerAt startup it loads from the database the list of tasks to perform and publish it in the tasks queueThread Pool.Any free thread pickup from the queue a task and starts its execution.It can contain scheduled tasks or One Shot tasksScheduled tasks

One Shottasks

It allows notifiable mode


How does it scale ?

InboxListenerservice AInboxListenerservice CInboxListenerservice BInstance 1 of 3

Instance 2 of 3

Instance 0 of 3

Cluster

Any instance know its index and how many instances are in the clusterIt executes the tasks with (id % 3) = 0

It executes the tasks with (id % 3) = 1


Tasks Repository

The cluster is obtained using JGroups


HA High availability

InboxListenerservice AInboxListenerservice CInboxListenerservice BCluster

Tasks Repository

Instance 1 of 3

Instance 2 of 3

Instance 0 of 3




Instance 1 of 2

Instance 0 of 2



Conclusions

DSserviceMailServer

InboxListenerservicePIMListenerservice

PIMDBMailServer

WSAPI

PushEngine

Push Connectionservice

112345SyncEngine

6Monitoring

Monitoring

WS API


Request for connection-oriented push


Synchronization

References

Funambol Forge - https://www.forge.funambol.org/

Funambol Developer Guide

http://download.forge.objectweb.org/sync4j/funambol-v7-developers-guide.pdf

Funambol design documents

https://core.forge.funambol.org/wiki/FunambolDesignDocuments

Open Mobile Alliance - http://www.openmobilealliance.org

myFunambol - http://my.funambol.com

Funambol mailing lists - https://core.forge.funambol.org/ds/viewForums.do

The End

Click to edit the title text format

Click to edit the outline text format

Second Outline Level

Third Outline Level

Fourth Outline Level

Fifth Outline Level

Sixth Outline Level

Seventh Outline Level

Eighth Outline Level

Ninth Outline Level

Funambol Server Architecture

Education

Transcript of Funambol Server Architecture