Satellite at Scale - Microsoft · 2018-05-10 · Partnership Two companies working together on a...

Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant

Darin Lively, Brian Ameling, & John Henley 2018-05-08

© Wal-Mart Stores, Inc. 2018


This presentation contains specific configuration and parameters optimized to Walmart's environment. These are likely not the exact configurations or parameters that you will use in your environment. But if they can scale, we know you can, too.

Any reference in this presentation

to any specific commercial

product, process, or service, or the

use of any trade, firm or

corporation name is for

information and convenience

purposes only, and does not

constitute an endorsement or

recommendation by Walmart Inc.

PLEASE NOTE

PHOTO DOWNLOADED FROM WALMART CORPORATE

https://corporate.walmart.com/photos/customers-checkout-after-shopping-walmarts-black-friday-event-on-thursday-nov-26-in-rogers-ark

SCALE W A L M A R T


Walmart’s Satellite 6 Journey

PART ONE

PART TWO

PART THREE

PART FOUR

PART FIVE

Walmart & Red Hat Joint Engineering Large-Scale Satellite Operations Satellite Infrastructure Considerations Managing Satellite Clients at Scale Operational Improvements


Partnership

● Two companies working together on a common goal to meet and exceed their business demands.

Collaboration ● collaborative relationship with Red hat that can be mirrored by any size customer

○ RHT listens, helps pull together sat releases that benefit the larger customer base (whatever scale you're at)

■ BZs & RFEs are here and the process is open to anyone.

○ Walmart + Red Hat had a positive Red Hat field Engineering experience

■ this is over, but we’re creating some programs to continue the benefits

JOINT ENGINEERING

EFFORT

COLLABORATIVE PARTNERSHIP

REPEATABLE PROCESS, OPEN

TO ANYONE

IMPROVING SATELLITE FOR

ALL


Large-Scale Satellite Operations


11,700 STORES

150+ DISTRIBUTION CENTERS

28 COUNTRIES

2.3M ASSOCIATES

WHAT WALMART BUSINESS SCALE LOOKS LIKE


270M CUSTOMERS/WEEK

234K PACKAGES

182K PACKAGE

DOWNLOADS/ HOUR PEAK

46 ACTIVATION

KEYS

22 CAPSULES

60K HOSTS

64 CONTENT VIEWS

9 LIFECYCLE

ENVIRONMENTS

3.9K PULP REPOS

WHAT WALMART SATELLITE SCALE LOOKS LIKE

1 SATELLITE


Satellite Infrastructure Considerations


SECURE

Secure

HOME OFFICE

STORES

DCs

Cloud

GeC

DMZ

DMZ

END-POINT LINUX

SERVERS – RHEL OR SLES

DC1

SECURE

FOREMAN- PROXY

SERVICES & DMZ HOST

CAPSULE

DC1

DC2

DC3

...

Secure

CLIENT TO CAPSULE: SERVICE/PORT

http/80 https/443

ampq/5647 https/8000

https/8443 https/9090

tftp/69

PXE/TFTP

tftp/69




http/80 https/443

https/8443 ampq/5647

LOAD BALANCER

Satellite-capsule VIP


http/80 https/443

https/8443

ampq/5647 https/8000 https/9090

Infoblox

DHCP Next-server

foreman-proxy-capsule

SATELLITE TO CAPSULE: SERVICE/PORT

https/443 https/9050 ampq/5646 ampq/5647

USER AUTHENTICATION

SATELLITE

DATABASE

MASTER SATELLITE

Active Directory

USER ACCESS

Web User API User

idaps:tcp/636

http/80 https/443

PUPPET FACTS/REPORTS

Puppet Masters

VERSION CONTROLLED

TEMPLATES

Git/Bitbucket

https/443

https/443

PERFORMANCE METRICS

grafana

http/2003

Third Party Repos

Red Hat CDN

SUSE CDN

INTERNET REPOSITORIES AND CONTENT DELIVERY

NETWORKS

EXTERNAL PROXY

Corporate Proxy

SMT Server

Squid Proxy

Internal Repos

SLES MIRRORED REPOS

http/80 https/443

http/80 https/443 http(s)/3128


http/80 https/443

ampq/5646 ampq/5647 https/9090

http/80 https/443

http/80 https/443

http/80 https/443

https/443

https/443

http/80 https/443

SECURE

Secure

HOME OFFICE

STORES

DCs

Cloud

GeC

DMZ

DMZ

END-POINT LINUX


DC1

SECURE

FOREMAN- PROXY

SERVICES & DMZ HOST

CAPSULE

DC1

DC2

DC3

...

Secure


http/80 https/443



tftp/69

PXE/TFTP

tftp/69




http/80 https/443


LOAD BALANCER



http/80 https/443

https/8443


Infoblox

DHCP Next-server




USER AUTHENTICATION

SATELLITE

DATABASE

MASTER SATELLITE

Active Directory

USER ACCESS

Web User API User

idaps:tcp/636

http/80 https/443


Puppet Masters

VERSION CONTROLLED

TEMPLATES

Git/Bitbucket

https/443

https/443

PERFORMANCE METRICS

grafana

http/2003

EXTERNAL PROXY

Corporate Proxy

SMT Server

Squid Proxy

Internal Repos

SLES MIRRORED REPOS

http/80 https/443



http/80 https/443


http/80 https/443

http/80 https/443

http/80 https/443

https/443

https/443

http/80 https/443

Third Party Repos

Red Hat CDN

SUSE CDN


NETWORKS

EXTERNAL PROXY

Corporate Proxy

SMT Server

Squid Proxy

Internal Repos

SLES MIRRORED REPOS

http/80 https/443

http/80 https/443

http/80 https/443

http/80 https/443

https/443

https/443

http/80 https/443

SECURE

Secure

HOME OFFICE

STORES

DCs

Cloud

GeC

DMZ

DMZ

END-POINT LINUX


DC1

SECURE

FOREMAN- PROXY

SERVICES & DMZ HOST

CAPSULE

DC1

DC2

DC3

...

Secure


http/80 https/443



tftp/69

PXE/TFTP

tftp/69




http/80 https/443


LOAD BALANCER



http/80 https/443

https/8443


Infoblox

DHCP Next-server




USER AUTHENTICATION

SATELLITE

DATABASE

MASTER SATELLITE

Active Directory

USER ACCESS

Web User API User

idaps:tcp/636

http/80 https/443


Puppet Masters

VERSION CONTROLLED

TEMPLATES

Git/Bitbucket

https/443

https/443

PERFORMANCE METRICS

grafana

http/2003

EXTERNAL PROXY

Corporate Proxy

SMT Server

Squid Proxy

Internal Repos

SLES MIRRORED REPOS

http/80 https/443



http/80 https/443


http/80 https/443

http/80 https/443

http/80 https/443

https/443

https/443

http/80 https/443

Third Party Repos

Red Hat CDN

SUSE CDN


NETWORKS

USER AUTHENTICATION

SATELLITE

DATABASE

MASTER SATELLITE

Active Directory

USER ACCESS

Web User API User

idaps:tcp/636

http/80 https/443


Puppet Masters

VERSION CONTROLLED TEMPLATES

Git/Bitbucket

https/443

https/443

PERFORMANCE METRICS

grafana

http/2003

SECURE

Secure

HOME OFFICE

STORES

DCs

Cloud

GeC

DMZ

DMZ

END-POINT LINUX


DC1

SECURE

FOREMAN- PROXY

SERVICES & DMZ HOST

CAPSULE

Secure


http/80 https/443



tftp/69

PXE/TFTP

tftp/69




http/80 https/443


LOAD BALANCER



http/80 https/443

https/8443


Infoblox

DHCP Next-server




USER AUTHENTICATION

SATELLITE

DATABASE

MASTER SATELLITE

Active Directory

USER ACCESS

Web User API User

idaps:tcp/636

http/80 https/443


Puppet Masters

VERSION CONTROLLED

TEMPLATES

Git/Bitbucket

https/443

https/443

PERFORMANCE METRICS

grafana

http/2003

Third Party Repos

Red Hat CDN

SUSE CDN


NETWORKS

EXTERNAL PROXY

Corporate Proxy

SMT Server

Squid Proxy

Internal Repos

SLES MIRRORED REPOS

http/80 https/443



http/80 https/443


http/80 https/443

http/80 https/443

http/80 https/443

https/443

https/443

http/80 https/443

DC1

DC2

DC3

...

DC1

DC2

DC3

...

SECURE

Secure

HOME OFFICE

STORES

DCs

Cloud

GeC

DMZ

DMZ

END-POINT LINUX



http/80 https/443



tftp/69

PXE/TFTP

tftp/69




http/80 https/443


LOAD BALANCER



http/80 https/443

https/8443


Infoblox

DHCP Next-server




USER AUTHENTICATION

SATELLITE

DATABASE

MASTER SATELLITE

Active Directory

USER ACCESS

Web User API User

idaps:tcp/636

http/80 https/443


Puppet Masters

VERSION CONTROLLED

TEMPLATES

Git/Bitbucket

https/443

https/443

PERFORMANCE METRICS

grafana

http/2003

Third Party Repos

Red Hat CDN

SUSE CDN


NETWORKS

EXTERNAL PROXY

Corporate Proxy

SMT Server

Squid Proxy

Internal Repos

SLES MIRRORED REPOS

http/80 https/443



http/80 https/443


http/80 https/443

http/80 https/443

http/80 https/443

https/443

https/443

http/80 https/443

SECURE

DC1

DC2

DC3

...

DC1

DC2

DC3

...

DC1 FOREMAN-

PROXY SERVICES

& DMZ HOST CAPSULE

Secure

SECURE

DC1

FOREMAN- PROXY SERVICES

& DMZ HOST CAPSULE

Secure

SECURE

Secure

HOME OFFICE

STORES

DCs

Cloud

GeC

DMZ

DMZ

END-POINT LINUX


DC1

SECURE

FOREMAN- PROXY

SERVICES & DMZ HOST

CAPSULE

DC1

DC2

DC3

...

Secure


http/80 https/443



tftp/69

PXE/TFTP

tftp/69




http/80 https/443


LOAD BALANCER Satellite-capsule VIP


http/80 https/443

https/8443


Infoblox

DHCP Next-server




USER AUTHENTICATION

SATELLITE

DATABASE

MASTER SATELLITE

Active Directory

USER ACCESS

Web User API User

idaps:tcp/636

http/80 https/443


Puppet Masters

VERSION CONTROLLED

TEMPLATES

Git/Bitbucket

https/443

https/443

PERFORMANCE METRICS

grafana

http/2003

Third Party Repos

Red Hat CDN

SUSE CDN


NETWORKS

EXTERNAL PROXY

Corporate Proxy

SMT Server

Squid Proxy

Internal Repos

SLES MIRRORED REPOS

http/80 https/443



http/80 https/443


http/80 https/443

http/80 https/443

http/80 https/443

https/443

https/443

http/80 https/443

SECURE

Secure

HOME OFFICE

STORES

DCs

Cloud

GeC

DMZ

DMZ

END-POINT LINUX


DC1

SECURE

FOREMAN- PROXY

SERVICES & DMZ HOST

CAPSULE

DC1

DC2

DC3

...

Secure


http/80 https/443



tftp/69

PXE/TFTP

tftp/69




http/80 https/443


LOAD BALANCER



http/80 https/443

https/8443


Infoblox

DHCP Next-server




USER AUTHENTICATION

SATELLITE

DATABASE

MASTER SATELLITE

Active Directory

USER ACCESS

Web User API User

idaps:tcp/636

http/80 https/443


Puppet Masters

VERSION CONTROLLED

TEMPLATES

Git/Bitbucket

https/443

https/443

PERFORMANCE METRICS

grafana

http/2003

Third Party Repos

Red Hat CDN

SUSE CDN


NETWORKS

EXTERNAL PROXY

Corporate Proxy

SMT Server

Squid Proxy

Internal Repos

SLES MIRRORED REPOS

http/80 https/443



http/80 https/443


http/80 https/443

http/80 https/443

http/80 https/443

https/443

https/443

http/80 https/443

DC1

DC2

DC3

...

5 6

Hiera customizations

Apache overrides

Limits files

Kernel tunables

Postgres

Logrotate

Pulp concurrency

CUSTOMIZATIONS


HIERA CUSTOMIZATIONS

/etc/foreman-installer/custom-hiera.yaml


APACHE OVERRIDES

/etc/httpd/conf.d/zzz-custom-overrides.conf


LIMITS FILES

/etc/systemd/system/*


KERNEL TUNABLES

/etc/sysctl.conf


POSTGRES

/var/lib/pgsql/data/postgresql.conf


LOGROTATE


PULP CONCURRENCY (Reduced Published Times)


Managing Satellite Clients at Scale


USING PUPPET TO AUTOMATE CLIENT SIDE OPERATIONS

PACKAGING FILES SERVICES

PUPPET MODULES


USING PUPPET ENTERPRISE TO ROLLOUT IN SCALE

ENTERPRISE CONSOLE

CLASSIFICATION

GROUPS

MIGRATION LOGIC

6 5

PUPPET ENTERPRISE


MODIFIED PUPPET MODULE TO RUN MULTIPLE LINUX CLIENTS

UPSTREAM SOURCE

SUSE ENTERPRISE LINUX


HOW SCALE IMPACTED ROLLOUT

WHERE: WHAT: WHEN: HOW:

Started in one location Touched 60,000 boxes Delivered on an aggressive timeline Rolled out successfully


Operational Improvements


Walmart had challenges, but they turned them into efficiency gains with customization

Maxing Passenger, Memory, Swap Provisioning Templates Migration to Satellite 6 Publishing


Added automation to roll out to a huge number of clients, multiple locations, small team

1/5 1/6 1/8 1/9 1/10 1/11 1/12 1/13 1/16 1/17 1/18 1/19 2/14 3/1 3/3

BUSINESS CHALLENGE

Need to get clients migrated in a reasonable amount of time. This required automation.



BUSINESS CHALLENGE

During client registration, Satellite defaults out of the box are not optimal for large scale, and they get maxed out.

Added randomization to even the distribution

BEFORE

AFTER



BUSINESS CHALLENGE

Users need new content available faster.

Performed customizations to dramatically reduce publish times

Initially impossible to publish all in one day (manually through the UI)

Scripted – still more than 24 hours because of capsule syncs

Added shared filesystem = 12 hour publish

Multi-threaded script = 4 hour publish

6.3 = 1.5 hour publish

OVER TIME


PUBLISH TIME (HOURS)


BUSINESS CHALLENGE

Walmart needed version control. Template changes from one user changed the template for everyone.

Moved to fully version-controlled template process to improve workflow



BUSINESS CHALLENGE

Need visibility into the operational health of the infrastructure


collectd > graphite > grafana

Ansible playbook to redeploy grafana dashboards

IMPORTANT METRICS TO COLLECT

METRICS

standard cpu/memory/disk/network

foreman tasks active (per task type)

foreman tasks per minute (per task type)

katello event queue

qpid queue depths

dynflow plans

dynflow orphaned execution plans

pulp tasks running

pulp tasks state

pulp repo count

postgres connections

postgres query length

apache processes

passenger processes

apache scoreboards

capsule apache busy servers

Scalability

Build capacity for IT of any size

Collaboration

Share your ideas and optimizations

Infrastructure

Create a reference architecture then customize

SCALING SUCCESS

Migration

Know your environment and track key metrics


THANK YOU plus.google.com/+RedHat

linkedin.com/company/red-hat

youtube.com/user/RedHatVideos

facebook.com/redhatinc

twitter.com/RedHatNews


Satellite at Scale - Microsoft · 2018-05-10 · Partnership Two companies working together on a...

Documents

Transcript of Satellite at Scale - Microsoft · 2018-05-10 · Partnership Two companies working together on a...