LESSONS LEARNED – BUILDING PAYPAL CLOUD

22
Chinmay Naik Lead Software Engineer, Cloud Engineering Anand Palanisamy Manager, Software Development, Cloud Engineering (OpenStack Summit – Hong Kong – 2013) LESSONS LEARNED – BUILDING PAYPAL CLOUD

description

LESSONS LEARNED – BUILDING PAYPAL CLOUD. Chinmay Naik Lead Software Engineer, Cloud Engineering Anand Palanisamy Manager, Software Development, Cloud Engineering ( OpenStack Summit – Hong Kong – 2013). About paypal. 137,000,000 Users. - PowerPoint PPT Presentation

Transcript of LESSONS LEARNED – BUILDING PAYPAL CLOUD

Page 1: LESSONS LEARNED – BUILDING  PAYPAL CLOUD

Chinmay Naik Lead Software Engineer, Cloud Engineering

Anand Palanisamy Manager, Software Development, Cloud Engineering

(OpenStack Summit – Hong Kong – 2013)

LESSONS LEARNED – BUILDING PAYPAL CLOUD

Page 2: LESSONS LEARNED – BUILDING  PAYPAL CLOUD

Confidential and Proprietary2

137,000,000 Users.

$300,000 Payments processed by PayPal each minute.

193 markets / 26 currencies.

PayPal is the World’s Most Widely Used Digital Wallet.

ABOUT PAYPAL

Page 3: LESSONS LEARNED – BUILDING  PAYPAL CLOUD

Confidential and Proprietary3

Challenges we are trying to address

Why OpenStack has emerged as a problem solver ?

Getting Openstack ready for production primetime

Success stories

STRUCTURE OF THE PRESENTATION

Page 4: LESSONS LEARNED – BUILDING  PAYPAL CLOUD

Confidential and Proprietary4

WHAT ARE WE TRYING TO SOLVE ?

Page 5: LESSONS LEARNED – BUILDING  PAYPAL CLOUD

Confidential and Proprietary5

Seamless On-Demand Infrastructure Capacity

Do we really want those hundred tickets to deploy a service ?

Drive developer agility

Provide self-service tool for application life cycle mgmt

Provide a platform to enable faster innovation.

SOME OF OUR CHALLENGES

Page 6: LESSONS LEARNED – BUILDING  PAYPAL CLOUD

Confidential and Proprietary6

WHO WILL GET US THERE ?

Page 7: LESSONS LEARNED – BUILDING  PAYPAL CLOUD

Confidential and Proprietary7

Solves Infrastructure-as-a-Service

Its open source

No specific vendor lock-ins

Fast growing developer community

Open standards and api driven

Industry best practices, prevent reinventing the wheel

OPENSTACK IS THE WINNER

Page 8: LESSONS LEARNED – BUILDING  PAYPAL CLOUD

Confidential and Proprietary8

OPEN SOURCE CANNOT ALWAYS BE USED OFF THE SHELF

Page 9: LESSONS LEARNED – BUILDING  PAYPAL CLOUD

Confidential and Proprietary9

OUR TECHNOLOGY STACK

Operations PortalAsgard, Horizon, Ceilometer

PD Deployment Portal

Orchestration Engine Cloud Formation (Heat)

Nova, Cinder, Swift, Keystone, Neutron, Horizon

CobblerISC DHCP

Salt Bind RHEL 6.x Hypervisor Zabbix

x86 Compute Local Storage Network Load Balancer

User Interface

Orchestration

Foundational Services

Software Infrastructure

Hardware Infrastructure

Traffic Mgmt Monitoring Metering Stages Workflow Monitoring

LBaaS, DNSaaS FWaaS

PP Specific

Page 10: LESSONS LEARNED – BUILDING  PAYPAL CLOUD

Confidential and Proprietary10

Scheduling enhancements for failure and availability domains Custom PayPal filter scheduler

Tenant based Compute Zone filters with Folsom

Host Aggregate filtering in Grizzly

25% distribution among different fault zone for HA

TUNING NOVA FOR HIGH AVAILABILITY

Page 11: LESSONS LEARNED – BUILDING  PAYPAL CLOUD

Confidential and Proprietary11

Instance host naming uniqueness

Auto assigning floating IPs to VMs

Rack aware networking

Leveraging config-drive

Nova conductor - security vs. load on rabbit

NOVA CHANGES

Page 12: LESSONS LEARNED – BUILDING  PAYPAL CLOUD

Confidential and Proprietary12

Integrating keystone with LDAP

Auto tenancy feature

Tenant based hostnames & dns zones

Client side token caching

Team admin feature

KEYSTONE CHANGES

Page 13: LESSONS LEARNED – BUILDING  PAYPAL CLOUD

Confidential and Proprietary13

Automatic

Project based zones

Floating IPs

DNS-AS-A-SERVICE INTEGRATION

Page 14: LESSONS LEARNED – BUILDING  PAYPAL CLOUD

Confidential and Proprietary14

Registration and auto discovery

Rich tenant and operator facing apis

Propagating changes to multiple LBs

Change Management Integration

LOAD BALANCER-AS-A-SERVICE

Page 15: LESSONS LEARNED – BUILDING  PAYPAL CLOUD

Confidential and Proprietary15

OTHER SUCCESS STORIES

Page 16: LESSONS LEARNED – BUILDING  PAYPAL CLOUD

Confidential and Proprietary16

Ease of use

Adoption

Multi Version

Multi Region

USER EXPERIENCE

Page 17: LESSONS LEARNED – BUILDING  PAYPAL CLOUD

Confidential and Proprietary17

USER INTERFACE SCREEN SHOTS

Page 18: LESSONS LEARNED – BUILDING  PAYPAL CLOUD

Confidential and Proprietary18

USER INTERFACE SCREENSHOTS

Page 19: LESSONS LEARNED – BUILDING  PAYPAL CLOUD

Confidential and Proprietary19

Devstack != Production

Keeping up with trunk

Single keystone service

Performance & Scalability

Error Handling

DEPLOYMENT PAIN POINTS

Page 20: LESSONS LEARNED – BUILDING  PAYPAL CLOUD

20 Confidential and Proprietary

[email protected]

Page 21: LESSONS LEARNED – BUILDING  PAYPAL CLOUD

Confidential and Proprietary21

http://www.123rf.com/

http://www.trashionista.com/

http://www.coreytowe.com/

http://www.birst.com/

COURTESIES FOR IMAGES USED

Page 22: LESSONS LEARNED – BUILDING  PAYPAL CLOUD

Confidential and Proprietary22

THANK YOU