Process Orchestration using Streaming API and Heroku

Andrew Fawcett, FinancialForce.com, CTO

@andyinthecloud

Safe harborSafe harbor statement under the Private Securities Litigation Reform Act of 1995: This presentation may contain forward-looking statements that involve risks, uncertainties, and assumptions. If any such uncertainties materialize or if any of the assumptions proves incorrect, the results of salesforce.com, inc. could differ materially from the results expressed or implied by the forward-looking statements we make. All statements other than statements of historical fact could be deemed forward-looking, including any projections of product or service availability, subscriber growth, earnings, revenues, or other financial items and any statements regarding strategies or plans of management for future operations, statements of belief, any statements concerning new, planned, or upgraded services or technology developments and customer contracts or use of our services. The risks and uncertainties referred to above include – but are not limited to – risks associated with developing and delivering new functionality for our service, new products and services, our new business model, our past operating losses, possible fluctuations in our operating results and rate of growth, interruptions or delays in our Web hosting, breach of our security measures, the outcome of intellectual property and other litigation, risks associated with possible mergers and acquisitions, the immature market in which we operate, our relatively limited operating history, our ability to expand, retain, and motivate our employees and manage our growth, new releases of our service and successful customer deployment, our limited history reselling non-salesforce.com products, and utilization and selling to larger enterprise customers. Further information on potential factors that could affect the financial results of salesforce.com, inc. is included in our annual report on Form 10-Q for the most recent fiscal quarter ended July 31, 2012. This documents and others containing important disclosures are available on the SEC Filings section of the Investor Information section of our Web site. Any unreleased services or features referenced in this or other presentations, press releases or public statements are not currently available and may not be delivered on time or at all. Customers who purchase our services should make the purchase decisions based upon features that are currently available. Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements.

All about FinancialForce.com

• #1 Accounting App on Force.com• #1 Professional Services Automation App on Force.com

Leading Native ISV on Force.com

• UNIT4 - $600 million, 33 years building business apps

Backed by Salesforce.com & UNIT4

• San Francisco HQ – 595 Market St.• 145 Employees• Customers in 23 countries

Growing Rapidly

Andrew FawcettCTO

@andyinthecloud

Building declarative applicationsBuilding applications that support Force.com declarative features is critical.

Building applications that consume Force.com declarative features to support your app is smart!

Encourage users admins to use what they have already• Building their solution consisting of part your app and part platform features can be a powerful combo!

Make sure you understand how to be dynamic…• Metadata API, Apex Describe• Dynamic SOQL, Dynamic Visualforce, Fieldsets • Reference Formulas, Rollup Summaries, List Views etc..

Make your applications metadata portable between orgs!

Consultants love our declarative applications!

Our user storyAs A: System Applications Administrator

I want: To be able to link a series of HTTP / REST based API’s to data changes occurring in my organization, based on criteria I provide. Without resorting to Apex coding.

So that: I can harness the power of many HTTP / REST based services to further automate my companies business processes. And leverage my skills as a System Administrator.

Our user case : eBay Integration

eBay Item: 12345678

Invoice Number: INV-000001

eBay Item: 12345678

Streaming API via Bayeux Protocol

Process Manager

Overview Salesforce Streaming API via Bayeux Protocol

1. Create a Push Topic in Force.com• Using the SOQL statement you provide. Salesforce servers will send JSON

messages to listeners. When changes occur to record data that meet the criteria expressed by the SOQL query.

2. Install a Bayeux Client Library• Clients are available from http://cometd.org (none for Apex)

3. Connect and start Listening!• This sample uses the Java Bayeux library

Live Demonstration

Name Query Select From

processchanges Process__c

processlogslinescreated ProcessLogLine__c

{Proccess__c.Id} {SourceObject__c}

NameQueryNotifyForOperationsNotifyForFields

PushTopic (Streaming API)

Admin creates Process

Process__c Listener

On Insert1. Generate PushTopic Query2. Create new PushTopic record3. Subscribe via Streaming API4. Start {SourceObject__c} Listener5. Log to ProcessLogLineItem__c

Invoice Clerk updates Invoice

NameAccount__cAmount__ceBayItemNumber__ceBayCompleteSaleRequest__c

Invoice__c /topic/{Process__c.Id}

On Update1. Read ProcessSteps__c2. Read {SourceObject__c}3. Call {HTMLEndpoint__c} …4. Log to ProcessLogLineItem__c

{SourceObject__c} Listener

NameSourceObject__cWhen__cListViewName__c

Process__c

NameHTMLEndpoint__cHTMLHeader__cHTMLMethod__cHTMLBody__c

ProcessStep__c

Message__cProcessLogLine__c

/topic/processchanges

Live Log Viewer“1. xxxx

2. xxxx”

ProcessLogLine__c Listener

/topic/processlogslinescreated

HTTP POST:<CompleteSaleRequest> <Item>{!eBayItemNumber}</Item></CompleteSaleRequest>

01: public class WorkerProcess 02: {03: // Salesforce API's04: private static final String SALESFORCE_API = "25.0";05: private static final String LOGIN_ENDPOINT = "https://login.salesforce.com/services/Soap/u/" + SALESFORCE_API;06: private static final String REST_ENDPOINT_URI = "/services/data/v" + SALESFORCE_API + "/";07: private static final String STREAMING_ENDPOINT_URI = "/cometd/" + SALESFORCE_API;08: 09: /**10: * Main entry point for the Heroku Worker Process11: * @param args12: * @throws Exception13: */14: public static void main(String[] args) throws Exception 15: {16: // Startup and assignment of listeners to Salesforce Streaming API topics…17: }

/** * Heroku Work Processes are simply Java classes with a main method! */

Java Code : Heroku Worker Process

01: // Connect to Salesforce Streaming API02: final BayeuxClient client = makeStreamingAPIConnection(loginResult);03: 04: // Subscribe to the Process__c topic to listen for new or updated Processes05: client.getChannel("/topic/processchanges").subscribe(new MessageListener() 06: {07: public void onMessage(ClientSessionChannel channel, Message message) 08: {09: HashMap<String, Object> data = (HashMap<String, Object>) JSON.parse(message.toString());10: HashMap<String, Object> record = (HashMap<String, Object>) data.get("data");11: HashMap<String, Object> sobject = (HashMap<String, Object>) record.get("sobject");12: HashMap<String, Object> event = (HashMap<String, Object>) record.get("event");13: // Event type, insert or update?14: String type = (String) event.get("type"); 15: if(type.equals("created"))16: processInsert(loginResult, client, sobject);17: }18: });

/** * Listener : Process__c * Comment : Bayeux Client used connect to Streaming API and implement MessageListener interface * @see makeStreamingAPIConnection method * @see JSON class from Jetty * @see processInsert method */

Java Code : Process__c Listener

Java Code : Process__c Listener

01: // Connection configuration02: ConnectorConfig metadataConfig = new ConnectorConfig();03: metadataConfig.setSessionId(loginResult.getSessionId());04: metadataConfig.setServiceEndpoint(loginResult.getMetadataServerUrl());05: MetadataConnection metadataConnection =06: com.sforce.soap.metadata.Connector.newConnection(metadataConfig);07: 08: // Make a REST connection09: RestConnection restConnection = makeRestConnection(loginResult);10: 11: // Message data from Proccess__c12: String processId = (String) sObject.get("Id");13: String processName = (String) sObject.get("Name");14: String sourceObject = (String) sObject.get("SourceObject__c");15: String listViewName = (String) sObject.get("ListViewName__c");

/** * Listener : Process__c * Step : Connect and Listen * API connections and message data, used WSC manual login to obtain Metadata API base URL from * LoginResult (not normally exposed by the standard connector classes) * @see makeRestConnection method (Rest URL helper) */

Java Code : Process__c Listener

Java Code : Process__c Listener

01: // Retrieve Custom Object Meta data (and thus List View definition) for Source Object02: RetrieveRequest retrieveRequest = new RetrieveRequest();03: retrieveRequest.setSinglePackage(true);04: com.sforce.soap.metadata.Package packageManifest = new com.sforce.soap.metadata.Package();05: ArrayList<PackageTypeMembers> types = new ArrayList<PackageTypeMembers>();06: PackageTypeMembers packageTypeMember = new PackageTypeMembers();07: packageTypeMember.setName("CustomObject");08: packageTypeMember.setMembers(new String[] { sourceObject });09: types.add(packageTypeMember);10: packageManifest.setTypes((PackageTypeMembers[]) types.toArray(new PackageTypeMembers[] {}));11: retrieveRequest.setUnpackaged(packageManifest);12: AsyncResult response = metadataConnection.retrieve(retrieveRequest);13: while(!response.isDone())14: {15: Thread.sleep(1000);16: response = metadataConnection.checkStatus(new String[] { response.getId()} )[0];17: }18: RetrieveResult retrieveResult = metadataConnection.checkRetrieveStatus(response.getId());

/** * Listener : Process__c * Step : Generate PushTopic Query * Comment : Retrieve via Metadata API custom object definition to get at the List View * definition for generating PushTopic Query*/

Java Code : Process__c Listener

Java Code : Process__c Listener

01: // Parse Custom Object metadata for Process Source Object02: CustomObject customObject = new CustomObject();03: byte[] zipBytes = retrieveResult.getZipFile();04: ZipInputStream zipis = new ZipInputStream(new ByteArrayInputStream(zipBytes, 0, zipBytes.length));05: ZipEntry zipEntry = null;06: while((zipEntry = zipis.getNextEntry()) != null) {07: if(zipEntry.getName().endsWith(sourceObject + ".object")) / Process Source Object?08: {09: TypeMapper typeMapper = new TypeMapper();10: XmlInputStream xmlis = new XmlInputStream();11: xmlis.setInput(zipis, "UTF-8");12: customObject.load(xmlis, typeMapper);13: zipis.closeEntry();14: break;15: }16: }17: // Find the List View indicated on the Process to define query for Streaming API Push Topic18: ListView processlistView = null;19: for(ListView listView : customObject.getListViews()) { ... }

/** * Listener : Process__c * Step : Generate PushTopic Query * Comment : Uses the WSC TypeMapper to deserialise the Custom Object definition from the * package zip returned by ‘retrieve’ */

Java Code : Process__c Listener

Java Code : Process__c Listener

01: // Generate SOQL statement for PushTopic based on List View definition02: StringBuilder fieldList = new StringBuilder("Id");03: for(String field : processlistView.getColumns())04: fieldList.append(", " + field);05: // Simple version assumes AND’s, ignores processListView.getBooleanFilter()06: StringBuilder whereClause = new StringBuilder();07: for(ListViewFilter lvFilter : processlistView.getFilters())08: {09: switch (lvFilter.getOperation()) 10: {11: case equals:12: // ...13: }14: }15: // Construct SOQL query statement16: StringBuilder soql = new StringBuilder()17: .append("select ”) .append(fieldList.toString())18: .append("from ”) .append(sourceObject + " ")19: .append("where ”) .append(whereClause.toString());

/** * Listener : Process__c * Step : Generate PushTopic Query * Comment : Parse List View definition to construct SOQL Query for Push Topic. Code shortened * for this slide. */

Java Code : Process__c Listener

Java Code : Process__c Listener

01: // Create PushTopic02: PushTopic pushTopic = new PushTopic();03: pushTopic.Name = processId;04: pushTopic.Query = soql.toString();05: pushTopic.ApiVersion = SALESFORCE_API;06: pushTopic.NotifyForOperations = "All"; // TODO: Interpret the 'When__c' field from Process__c07: pushTopic.NotifyForFields = "Referenced"; 08: restConnection.create(pushTopic);09: 10: // Update Process with Push Topic Name (so that if the worker restarts it can reconnect)11: Process__c process = new Process__c();12: process.PushTopicName__c = pushTopic.Name;13: restConnection.update(process, processId);14: 15: // Start listening on the new Push Topic immediately!16: client.getChannel("/topic/"+pushTopic.Name).subscribe(17: new SourceObjectListener(loginResult, processId, sourceObject));

/** * Listener : Process__c * Step : Create PushTopic and subscribe * Comment : PushTopic Name has to be unique thus we used the Process__c.Id. * It is possible to immediately subscribe once the Push Topic has been created! * @see SourceObjectListener class implements MessageListener interface! */

Java Code : Process__c Listener

Java Code : {SourceObject__c} Listener

/** * Listener : {SourceObject__c} * Step 1 : Parse Source Object Id from message and read current process steps. Determine what HTTP call * information will be provided by the source object record fields. */

01: public static class SourceObjectListener implements MessageListener02: {03: public void onMessage(ClientSessionChannel channel, Message message) 04: {05: // Source Object Message06: HashMap<String, Object> data = (HashMap<String, Object>) JSON.parse(message.toString());07: HashMap<String, Object> record = (HashMap<String, Object>) data.get("data");08: HashMap<String, Object> sobject = (HashMap<String, Object>) record.get("sobject");09: String sourceId = (String) sobject.get("Id")10: // Query Process Steps11: QueryResult processStepsResult = m_partnerConnection.query(12: “select Id, Name, " + 13: "HTTPEndPoint__c, HTTPEndPointFrom__c, " +14: "HTTPMethod__c, HTTPMethodFrom__c, " +15: "HTTPHeader__c, HTTPHeaderFrom__c, " +16: "HTTPBody__c, HTTPBodyFrom__c " +17: ”from ProcessStep__c " +18: "where Process__c = '" + m_processId + "'");

Java Code : {SourceObject__c} Listener

Java Code : {SourceObject__c} Listener

/** * Listener : {SourceObject__c} * Step 2 : Dynamically build query for source object to include fields required * to make HTTP callout contextual */

01: // Construct Source Object SOQL (query formula fields)02: StringBuilder fieldList = new StringBuilder();03: fieldList.append("Id, Name");04: for(String field : sourceFields)05: fieldList.append(", " + field); 06: StringBuilder soql = new StringBuilder()07: .append("select ")08: .append(fieldList.toString() + " ")09: .append("from ")10: .append(m_sourceObject + " ")11: .append("where ")12: .append("id = '%s'"); 13: String sourceObjectQuery = soql.toString();14: 15: // Query Source Object record16: String query = String.format(sourceObjectQuery, sourceId);17: QueryResult sourceObjectResult = m_partnerConnection.query(query);18: SObject sourceRecord = sourceObjectResult.getRecords()[0];

Java Code : {SourceObject__c} Listener

Java Code : {SourceObject__c} Listener

/** * Listener : {SourceObject__c} * Step 3 : Use Jetty HTTP Client to make callout. HTTP call out details are taken from literals on the step * or from formula fields on the source record */

01: // Create the HTTP client02: HttpClient httpClient = new HttpClient();03: httpClient.setConnectorType(HttpClient.CONNECTOR_SELECT_CHANNEL); 04: httpClient.start();05: // HTTP parameters06: String httpEndpoint = ...07: String httpHeader = ...08: String httpMethod = ...09: String httpBody = ...10: processStep.HTTPBodyFrom__c.equals("Literal") ?11: processStep.HTTPBody__c : (String) sourceRecord.getField(processStep.HTTPBody__c);12: // Construct HTTP request13: ContentExchange contentExchange = new ContentExchange(true);14: contentExchange.setMethod(httpMethod);15: contentExchange.setURL(httpEndpoint);16: contentExchange.setRequestHeader(...);17: contentExchange.setRequestContent(new ByteArrayBuffer(httpBody));18: httpClient.send(contentExchange);19: contentExchange.waitForDone();

Java Code : {SourceObject__c} Listener

Java Code : {SourceObject__c} ListenerResources• Other // TODO:’s and Ideas?!

– Record Types for Process Steps?– State / Variables between Steps?

• Source Code and Contact DetailsGitHub: https://github.com/financialforcedevHeroku App (Clone):

https://api.heroku.com/myapps/df12-processmanager/clone

Twitter: andyinthecloud