Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million...

41
Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day

Transcript of Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million...

Page 1: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Behind-the-Scenes at Salesforce.com

Claus Moldt, Salesforce.com

R&D: Powering 150 180+ Million Transactions a Day

Page 2: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Safe Harbor Statement

“Safe harbor” statement under the Private Securities Litigation Reform Act of 1995: This presentation may contain forward-looking statements including but not limited to statements concerning the potential market for our existing service offerings and future offerings. All of our forward looking statements involve risks, uncertainties and assumptions. If any such risks or uncertainties materialize or if any of the assumptions proves incorrect, our results could differ materially from the results expressed or implied by the forward-looking statements we make.

The risks and uncertainties referred to above include - but are not limited to - risks associated with possible fluctuations in our operating results and cash flows, rate of growth and anticipated revenue run rate, errors, interruptions or delays in our service or our Web hosting, our new business model, our history of operating losses, the possibility that we will not remain profitable, breach of our security measures, the emerging market in which we operate, our relatively limited operating history, our ability to hire, retain and motivate our employees and manage our growth, competition, our ability to continue to release and gain customer acceptance of new and improved versions of our service, customer and partner acceptance of the AppExchange, successful customer deployment and utilization of our services, unanticipated changes in our effective tax rate, fluctuations in the number of shares outstanding, the price of such shares, foreign currency exchange rates and interest rates.

Further information on these and other factors that could affect our financial results is included in the reports on Forms 10-K, 10-Q and 8-K and in other filings we make with the Securities and Exchange Commission from time to time. These documents are available on the SEC Filings section of the Investor Information section of our website at www.salesforce.com/investor. Salesforce.com, inc. assumes no obligation and does not intend to update these forward-looking statements, except as required by law.

Page 3: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Claus MoldtVP, Technical Operations

Page 4: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Data Centers

Best of Breed Data Centers

Page 5: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Fully Mirrored Cloud Computing Infrastructure Continued Investments. Unparalleled Confidence.

Maximum Uptime & Performance

Carrier neutral network strategy

No single points of failure

Carrier level scalability

Extensive use of high availability

server and network technologies

Production-ClassProduction-Class R&D Lab

& Tape Archive (CA) Back-UpBack-Up Production

Data Center (VA)

Unmatched Reliability Two (soon to be 3) mirrored

production data centers plus a production-class lab facility

Near real time replicationbetween facilities

Validated disaster recovery MPLS based backbone

Main Production Main Production

Data Center (CA)Data Center (CA)

Asia PacificPacific Production

Datacenter (Singapore

Winter ‘08)

Trusted Security World-class security specs

SAS 70 Type II and SysTrust

Certified

ISO 27001 Certified Secure point-to-point data

replication

Secure custody of customer

data and backups

Page 6: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Security: FacilitiesMaximum Facilities Security

24 x 365 on-site security

All doors, including cages, are secured with biometric hand geometry readers.

Five levels of biometric scanning including man-traps required to reach Salesforce cages

Fully anonymous exteriors

Digital camera (CCTV) coverage of entire facility

Entire perimeter bounded by concrete bollards/planters

A silent alarm and automatic notification of appropriate law enforcement officials protect all exterior entrances.

CCTV integrated with access control and alarm system.

Motion-detection for lighting and CCTV coverage.

Page 7: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

World-Class InfrastructureDelivering leading On-Demand availability

Two mirrored data centers plus a production-scale lab facility

– 18,000 total sq. feet of cage space

– Mirroring is about more than just having a copy of your data

– Salesforce maintains a full-scale replica of the production facility as well as your data

Power: Diesel Generators for backup power supply

Next generation UPS systems (N+1) Five- Hitec Rotary Continuous Power Supplies rated for 4,980kW  (n +1)

Rotating fly-wheel generator provides UPS and Diesel generator start-up Two- Detroit Diesel engine 2mW Generators for a total of 4,980kW (n +1)

Eliminates potentially risky UPS battery maintenance

25,000 gallon diesel fuel tanks supported by two fuel vendors

Cooling

– Precision, N+1 HVAC

– Guaranteed by backup water supply

– On-site dedicated wells

Page 8: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

NetworkIndustry leading performance, scalability and redundancy

Carrier-class and carrier-neutral model: multiple transit vendors

AboveNet

MCI

Level 3

NTT

Equinix Exchange

Sprint

Multi-gigabit IP transit for external customer service

Lightning-fast performance worldwide Data centers located at core Internet hubs

Access to thousands of global Internet peering points delivering global high performance access

Private peering with key carriers and partners (15+)

MPLS/VPLS based backbone Enables near real-time replication for availability and disaster

recovery

Page 9: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Scalability

Highly Scalable POD Architecture

Page 10: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

ENTERPRISEMARKET

ENTERPRISEMARKET

MID-MARKETMID-MARKET

SMALL BUSINESSSMALL BUSINESS

Cloud Computing Serves Companies of All Sizes

~3,200~3,200~3,200~3,200

~65,000~65,000~65,000~65,000 ~30,000~30,000~30,000~30,000Enterprise StdEnterprise StdEnterprise StdEnterprise Std ~30,000~30,000~30,000~30,000

~4,000~4,000~4,000~4,000

Number of Subscribers

~5,800~5,800~5,800~5,800~9,000~9,000~9,000~9,000

~3,000~3,000~3,000~3,000

~5,500~5,500~5,500~5,500~6,300~6,300~6,300~6,300

~3,500~3,500~3,500~3,500

Page 11: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

We built the platform for the cloud

YOU get to focus on

innovation

We doInfrastructure

Services

We doApplication

Services

We doOperations

Services

Build your data model

Build your business logic

Build your user interface

Network

Storage

Operating System

Database

App Server

Web Server

Data Center

Security

Sharing

Integration

Customization

Web Services

API

Multi-Language

Authentication

Availability

Monitoring

Patch Mgmt

Upgrades

Backup

NOC

Force.com allowed us to create and deliver a total of 14 applications – all without the expense and hassles of traditional application development.

“”

Page 12: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

SubscriptionMulti-tenant

The Cloud Computing Model: Multi-tenant, Subscriptions

Faster Vendor InnovationEconomies of Scale ScalabilityAutomatic Upgrades

Page 13: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

The Fastest, Easiest and Lowest Risk Path to IT Success

Client/Server & App Server Platforms

Platform as a Service

Source: 3rd party analyst surveys Source Salesforce.com Customer Relationship Survey conducted in Feb. 2008, by an independent third-party CustomerSat Inc.

Page 14: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Cloud Computing Enables Reactive Innovation26 Major Releases in 9 Years

All Customizations Upgraded Automatically

No Customers left behindEvery customer on the latest

version of salesforce.com

Page 15: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Proven Scalability and PerformanceDelivering 180+ Million Transactions Daily

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

9.0

10.0

11.0

12.0

Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2

0

250

500

750

1,000

1,250

Page Response Time(ms)

Quarterly Transactions

(billions)

2005 2006 2007Fiscal Year 2008

Page 16: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Multi-Tenant Integration = Proven Success

I think API font should be bold and line stronger. Make page

views gray or something to highlight that API transactions is the key thing to focus on.

1,100,000+ Subscribers

-

750,000,000

1,000,000,000

1,250,000,000

1,500,000,000

1,750,000,000

2,000,000,000

2,250,000,000

2,500,000,000

2,750,000,000

3,000,000,000

Q2FY06 Q3FY06 Q4FY06 Q1FY07 Q2FY07 Q3FY07 Q4FY07 Q1FY08

API Transactions

Page Views

Over 2.2 Billion API Transactions per Month

Page 17: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Enterprise Scalability & Performance

Your Company

Your Division

Your Customizations

Query Optimization

Engine

1

Your Sharing ModelMassive Scale

CEO

VP Sales VP MarketingVP Customer

ServiceCOO

VP Professional Services

Director Sales West

Director Sales East

Director Customer Service

Director Support East Services West Services

CEO

VP Sales VP MarketingVP Customer

ServiceCOO

VP Professional Services

Director Sales West

Director Sales East

Director Customer Service

Director Support East Services West Services

CEO

VP Sales VP MarketingVP Customer

ServiceCOO

VP Professional Services

Director Sales West

Director Sales EastDirector Customer

ServiceDirector Support East Services West Services

CEO

VP Sales VP MarketingVP Customer

ServiceCOO

VP Professional Services

Director Sales West

Director Sales EastDirector Customer

ServiceDirector Support East Services West Services

Your Your DataData

2

3

4Immediate Response

Sub-second response time

Billions of Transactions

Page 18: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Scalable Software ArchitectureUtilizing Industry Standard Platforms for High Availability

Database Server: Oracle RAC EE, Dell, Sun

Clustering: SunCluster

Web Site and Application Server: Dell, Resin

Search Server: Jakarta Lucene

Storage Management: Hitachi Data Systems,

Sun

Backup Software: Veritas/RMAN

Operating Systems

– Sun Solaris

– Redhat Linux

Page 19: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Pod Architecture further enhances availability, horizontal scale, and platform for future growth

Network Services

Storage Services

Backup Services

Monitoring Services

NA0 Pod

NA1 Pod

NA2 Pod

NA3 Pod

NA4 Pod

NA5 Pod

EMEA Pod

APAC Pod

Sandbox Pod

EMEA2 Pod

NA6 Pod

NA7 Pod

“N” Pod

Salesforce.com confidential

Threshold User Capacity = Add a POD

Page 20: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

What’s Shared Across Pods

Storage – HDS 9990

SAN – Cisco MDS

Core Network – Force10

Edge Network – Juniper

Search Indexer – Sun SPARC

WWW Services – Dell/Linux

Edge Firewalls – Juniper Netscreens

Load Balancers – F5

Proxy Services – Dell/Linux

IDS

BlueCoat

Performance Monitoring

Email

Ops Stack

– syslog

– bastion

– jump/kick start

– release

– backup

– DNS

– TACACS

– SecurID

Salesforce.com confidential

Page 21: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Redundancy

Network Redundancy/Multiple CarriersLoad Balancing/Fail-OverClustering/RAC EE MirrorForce

Page 22: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

SFDC Built for High Availability

– Multiple Network Carriers

– Redundant Routers at Entry Points

– Fail-over Configured Firewalls

– Redundant & Load Balanced Load Balancers

– Redundant Hubs/Switches at VLANs

– Web, Application, API, Search, Cache, Index, Batch Servers

• Load Balanced, Fail-over or Clustered

– Data Base Servers

• Oracle RAC EE running on 4 way Clustered Nodes

• Sized to sustain Peak Load if Node failure

– Storage

• Multiple paths for reliability

– 4 inter-connects per DBMS Server

» Alternate paths to separate Storage Directors

– 2 Storage Directors per Array

Page 23: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Multiple Network Carriers and Redundancy at the Edge

Sprint

Level3AboveNet

MCI

Edge Routers Edge Routers

EquinixExchange

SiSi SiSi

Server VLAN Switches/Firewalls

RedundantLoad Balancers

RedundantFirewalls

RedundantCore Switches

NTT

Page 24: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Sample POD Architecture – Built for Redundancy

Page 25: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Backup and Disaster Recovery Strategy

Near real time replication

between data centers

Disaster Recovery

Strategy: Failover to full-

scale east coast replica

data center backup

facility.

Production Data Center

Near real time replication between data centers

OC48/MPLS/VPLS Backbone

West Coast

San Francisco

East Coast

Backup DR Data Center

Lab and Tape Archive

* Local 48 Hour lag standby databases

Page 26: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Monitoring

Performance Management Pro-active Monitoring

Page 27: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Performance and MonitoringEnd-to-End Monitoring Guarantees Uptime and Security

Monitoring Strategy: Multi-Tier Monitoring Strategy

Nagios monitoring software

Gomez performance software service

EMC Smarts

Coradiant End-User experience

Custom instrumentation within the Application

Performance Metrics

Average page load times between 250 and 400 milliseconds

180M+ Transactions Daily

Pages served in fiscal Q1 2008: 5.4 Billion

47,600+ Customers

1,100,000+ Subscribers

95% Customer Satisfaction*

Open Communication

http://trust.salesforce.com

*January 2005 independent survey

Cricket

Custom Agents

Page 28: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Example of SFDC Monitoring Pro-Agents

Page 29: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Trust Site - Incident Communications Example

Users on NAX instance may experience latency with Dashboard Refresh, Reporting, and Customer Self Service Portal. The Salesforce.com Technology team is actively working to resolve these issues. Please check back for latest update.

Page 30: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Trust Site - Security Alerts and Examples

Page 31: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Capacity Planning

Log Parsing and AnalysisEngine

OperationsData Store

Breakdown of CustomerTransactions

Capacity Planning and Analysis Model

Basic load management data used to forecast annual growth.

Detailed customer transaction level monitoring for focused and proactive capacity management

Granular performance breakdowns by transaction type

Impact analysis of custom transactional logics

Operationally efficient and scalable

System/App Outputs

Multiple detailed metrics

1. Annual demand growth for all enterprise customers are projected and reviewed every week

2. Three year forecast are predicted for Datacenter capacity

3. Results analyzed and systems scaled appropriately to meet demand growth.

Capacity forecasting

Page 32: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Security

Overview

Page 33: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Security Dedicated Security Organization Strategy/Charter Mitigate risks while complying with legal, statutory, contractual, and internally

developed requirements  Develop and enforce policies and procedures

– Design and secure information systems using

security domains, defense in-depth and least privilege principles– Develop and integrate security architecture into business processes (CobiT,

ISO27001)– Conduct employee security awareness training classes– Perform regular vulnerability assessments and audits

Addresses all layers– Physical Security– Logical Network Security– Host Security– Transmission Level Security– Database Security

Page 34: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Internal Vulnerability Assessments

Salesforce.com implements a multi-prong approach to ensure the software we release is secure.  Specifically, we perform the following tasks to assure security in the development lifecycle.

Architecture Reviews Salesforce.com architects (including security team) meet regularly to discuss features that could be considered high risk.

Development  Salesforce.com developers follow coding best practices such as those specified in OWASP.  All code prior to check in is reviewed.  Code quality and security tools (Findbugs, Checkmarx.) are run frequently to detect possible program anomalies. All developers receive application security training to help them write secure code.

Quality Assurance Salesforce.com QA testers analyze their features through both positive and negative testing.  Salesforce.com also employs several black box analysis tools (Appscan, Peros, etc.) to help in identification of security vulnerabilities.

Information Security Salesforce.com InfoSec tests medium and high risk features. (Proprietary fuzzers, Burp Suite) Periodically brings in third parties to perform code reviews, blackbox analysis and design reviews (iSEC Partners, etc.)

Page 35: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

• Cross-Site Scripting• Input validation• Buffer Overflow• SQL Injection• Directory Traversal• Parameter Overflow• Path Manipulation• Command Execution• Path Truncation• Character Encoding• Character Stripping• Site Search• Application Mapping• Automatic Form-Filling• Configuration Management• Proxy Support• Parameter Injection

External Vulnerability Assessments

• Directory Enumeration• Authentication and Session Management• Web Server Assessment• HTTP Compliance• SSL Support and Strength• Certificate Analysis• Content Investigation• Spam Gateway Detection• Developer Comments• Absolute Path Detection• Error Handling• Permissions Assessment• Brute Force Authentication attacks• Known Attacks• Session Hijacking• Horizontal Attacks• Insecure Storage

MSSPs include SPI Dynamics, Solutionary, Symantec

Network Assessments and Application Assessments

Assessments cover the following:

Executive Summaries available upon request

Page 36: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Managing Change

Release Management/Change ManagementMaintenance Windows

Page 37: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

SFDC Release Testing/Managing Quality & Change

SFDC Testing is focused to ensure transparency of

changes– Intense Functional and System Testing prior to release

– Forward and backward compatibility of all standard API’s

– Review Teams

– Metrics and Reporting

• Quality Targets

• System metric/trends

All production changes logged in cases – Includes rollback, validation and expected impact

Page 38: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Salesforce.com Releases

Salesforce.com confidential

Break fix errors

End user experience

enhancement with

146.8 release

Planned 146 release

Example

Unscheduled on as

needed basis

Weekly for first 3-4

weeks after major

release

Every other week

there after

Approx 3 - 4 per year

Frequency

(No Downtime)

Wednesday evening

(No Downtime)

Friday night,

Saturday

TYPICAL Time of the

week

Fix production

vulnerabilities

Bug fixes or minor

functionality

enhancement

Significant new

functionality and

enhancement

Release objective

E Release

Patch/Dot

Major

Release

Page 39: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Maintenance Windows are Designed to Minimize Business Disruption to Customers

Established based on analysis of our customer usage patterns and traffic

4 hour windows reserved for routine maintenance– 1st & 3rd Saturdays

– 7pm to 11pm Pacific Time all NA & EU Instances except NA2

– 12am to 0400am Sunday Pacific Time NA2 only

– 10am to 2pm Saturday Pacific Time AP0 only

– Plans for EMEA instance can be adjusted to fit their time zones (for maintenance of non-shared infrastructure)

– Maintenance of Shared Infrastructure 1st & 3rd Saturdays 7pm to 11pm Pacific Time

– Windows are planned conservatively

– Not all reserved windows are utilized

– Actual maintenance downtime is a fraction of declared window

Future roadmap to minimize and eventually eliminate downtime

Note: Product release updates (3 per year) typically occur on a separate schedule on Friday nights and have longer windows

Actual

Declared

Reserved ~4 Hours

~30 Minutes

~15 Minutes

Page 40: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Maintenance Windows are Declared 1 Week in AdvanceSample Notification

Page 41: Behind-the-Scenes at Salesforce.com Claus Moldt, Salesforce.com R&D: Powering 150 180+ Million Transactions a Day.

Thank You

Q&A