Polyp-detection in Colonoscopy Stefan Ameling 2008 [email protected].
Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT...
Transcript of Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT...
![Page 1: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/1.jpg)
Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant
Darin Lively, Brian Ameling, & John Henley 2018-05-08
© Wal-Mart Stores, Inc. 2018
![Page 2: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/2.jpg)
© Wal-Mart Stores, Inc. 2018
This presentation contains specific configuration and parameters optimized to Walmart's environment. These are likely not the exact configurations or parameters that you will use in your environment. But if they can scale, we know you can, too.
Any reference in this presentation
to any specific commercial
product, process, or service, or the
use of any trade, firm or
corporation name is for
information and convenience
purposes only, and does not
constitute an endorsement or
recommendation by Walmart Inc.
PLEASE NOTE
![Page 3: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/3.jpg)
PHOTO DOWNLOADED FROM WALMART CORPORATE
https://corporate.walmart.com/photos/customers-checkout-after-shopping-walmarts-black-friday-event-on-thursday-nov-26-in-rogers-ark
SCALE W A L M A R T
© Wal-Mart Stores, Inc. 2018
![Page 4: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/4.jpg)
Walmart’s Satellite 6 Journey
PART ONE
PART TWO
PART THREE
PART FOUR
PART FIVE
Walmart & Red Hat Joint Engineering Large-Scale Satellite Operations Satellite Infrastructure Considerations Managing Satellite Clients at Scale Operational Improvements
© Wal-Mart Stores, Inc. 2018
![Page 5: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/5.jpg)
Partnership
● Two companies working together on a common goal to meet and exceed their business demands.
Collaboration ● collaborative relationship with Red hat that can be mirrored by any size customer
○ RHT listens, helps pull together sat releases that benefit the larger customer base (whatever scale you're at)
■ BZs & RFEs are here and the process is open to anyone.
○ Walmart + Red Hat had a positive Red Hat field Engineering experience
■ this is over, but we’re creating some programs to continue the benefits
JOINT ENGINEERING
EFFORT
COLLABORATIVE PARTNERSHIP
REPEATABLE PROCESS, OPEN
TO ANYONE
IMPROVING SATELLITE FOR
ALL
© Wal-Mart Stores, Inc. 2018
![Page 6: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/6.jpg)
Large-Scale Satellite Operations
© Wal-Mart Stores, Inc. 2018
![Page 7: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/7.jpg)
11,700 STORES
150+ DISTRIBUTION CENTERS
28 COUNTRIES
2.3M ASSOCIATES
WHAT WALMART BUSINESS SCALE LOOKS LIKE
© Wal-Mart Stores, Inc. 2018
270M CUSTOMERS/WEEK
![Page 8: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/8.jpg)
234K PACKAGES
182K PACKAGE
DOWNLOADS/ HOUR PEAK
46 ACTIVATION
KEYS
22 CAPSULES
60K HOSTS
64 CONTENT VIEWS
9 LIFECYCLE
ENVIRONMENTS
3.9K PULP REPOS
WHAT WALMART SATELLITE SCALE LOOKS LIKE
1 SATELLITE
© Wal-Mart Stores, Inc. 2018
![Page 9: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/9.jpg)
© Wal-Mart Stores, Inc. 2018
![Page 10: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/10.jpg)
Satellite Infrastructure Considerations
© Wal-Mart Stores, Inc. 2018
![Page 11: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/11.jpg)
SECURE
Secure
HOME OFFICE
STORES
DCs
Cloud
GeC
DMZ
DMZ
END-POINT LINUX
SERVERS – RHEL OR SLES
DC1
SECURE
FOREMAN- PROXY
SERVICES & DMZ HOST
CAPSULE
DC1
DC2
DC3
...
Secure
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5647 https/8000
https/8443 https/9090
tftp/69
PXE/TFTP
tftp/69
CLIENT TO CAPSULE: SERVICE/PORT
https/8000 https/9090
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443 ampq/5647
LOAD BALANCER
Satellite-capsule VIP
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443
ampq/5647 https/8000 https/9090
Infoblox
DHCP Next-server
foreman-proxy-capsule
SATELLITE TO CAPSULE: SERVICE/PORT
https/443 https/9050 ampq/5646 ampq/5647
USER AUTHENTICATION
SATELLITE
DATABASE
MASTER SATELLITE
Active Directory
USER ACCESS
Web User API User
idaps:tcp/636
http/80 https/443
PUPPET FACTS/REPORTS
Puppet Masters
VERSION CONTROLLED
TEMPLATES
Git/Bitbucket
https/443
https/443
PERFORMANCE METRICS
grafana
http/2003
Third Party Repos
Red Hat CDN
SUSE CDN
INTERNET REPOSITORIES AND CONTENT DELIVERY
NETWORKS
EXTERNAL PROXY
Corporate Proxy
SMT Server
Squid Proxy
Internal Repos
SLES MIRRORED REPOS
http/80 https/443
http/80 https/443 http(s)/3128
SATELLITE TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5646 ampq/5647 https/9090
http/80 https/443
http/80 https/443
http/80 https/443
https/443
https/443
http/80 https/443
![Page 12: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/12.jpg)
SECURE
Secure
HOME OFFICE
STORES
DCs
Cloud
GeC
DMZ
DMZ
END-POINT LINUX
SERVERS – RHEL OR SLES
DC1
SECURE
FOREMAN- PROXY
SERVICES & DMZ HOST
CAPSULE
DC1
DC2
DC3
...
Secure
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5647 https/8000
https/8443 https/9090
tftp/69
PXE/TFTP
tftp/69
CLIENT TO CAPSULE: SERVICE/PORT
https/8000 https/9090
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443 ampq/5647
LOAD BALANCER
Satellite-capsule VIP
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443
ampq/5647 https/8000 https/9090
Infoblox
DHCP Next-server
foreman-proxy-capsule
SATELLITE TO CAPSULE: SERVICE/PORT
https/443 https/9050 ampq/5646 ampq/5647
USER AUTHENTICATION
SATELLITE
DATABASE
MASTER SATELLITE
Active Directory
USER ACCESS
Web User API User
idaps:tcp/636
http/80 https/443
PUPPET FACTS/REPORTS
Puppet Masters
VERSION CONTROLLED
TEMPLATES
Git/Bitbucket
https/443
https/443
PERFORMANCE METRICS
grafana
http/2003
EXTERNAL PROXY
Corporate Proxy
SMT Server
Squid Proxy
Internal Repos
SLES MIRRORED REPOS
http/80 https/443
http/80 https/443 http(s)/3128
SATELLITE TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5646 ampq/5647 https/9090
http/80 https/443
http/80 https/443
http/80 https/443
https/443
https/443
http/80 https/443
Third Party Repos
Red Hat CDN
SUSE CDN
INTERNET REPOSITORIES AND CONTENT DELIVERY
NETWORKS
EXTERNAL PROXY
Corporate Proxy
SMT Server
Squid Proxy
Internal Repos
SLES MIRRORED REPOS
http/80 https/443
http/80 https/443
http/80 https/443
http/80 https/443
https/443
https/443
http/80 https/443
![Page 13: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/13.jpg)
SECURE
Secure
HOME OFFICE
STORES
DCs
Cloud
GeC
DMZ
DMZ
END-POINT LINUX
SERVERS – RHEL OR SLES
DC1
SECURE
FOREMAN- PROXY
SERVICES & DMZ HOST
CAPSULE
DC1
DC2
DC3
...
Secure
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5647 https/8000
https/8443 https/9090
tftp/69
PXE/TFTP
tftp/69
CLIENT TO CAPSULE: SERVICE/PORT
https/8000 https/9090
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443 ampq/5647
LOAD BALANCER
Satellite-capsule VIP
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443
ampq/5647 https/8000 https/9090
Infoblox
DHCP Next-server
foreman-proxy-capsule
SATELLITE TO CAPSULE: SERVICE/PORT
https/443 https/9050 ampq/5646 ampq/5647
USER AUTHENTICATION
SATELLITE
DATABASE
MASTER SATELLITE
Active Directory
USER ACCESS
Web User API User
idaps:tcp/636
http/80 https/443
PUPPET FACTS/REPORTS
Puppet Masters
VERSION CONTROLLED
TEMPLATES
Git/Bitbucket
https/443
https/443
PERFORMANCE METRICS
grafana
http/2003
EXTERNAL PROXY
Corporate Proxy
SMT Server
Squid Proxy
Internal Repos
SLES MIRRORED REPOS
http/80 https/443
http/80 https/443 http(s)/3128
SATELLITE TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5646 ampq/5647 https/9090
http/80 https/443
http/80 https/443
http/80 https/443
https/443
https/443
http/80 https/443
Third Party Repos
Red Hat CDN
SUSE CDN
INTERNET REPOSITORIES AND CONTENT DELIVERY
NETWORKS
USER AUTHENTICATION
SATELLITE
DATABASE
MASTER SATELLITE
Active Directory
USER ACCESS
Web User API User
idaps:tcp/636
http/80 https/443
PUPPET FACTS/REPORTS
Puppet Masters
VERSION CONTROLLED TEMPLATES
Git/Bitbucket
https/443
https/443
PERFORMANCE METRICS
grafana
http/2003
![Page 14: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/14.jpg)
SECURE
Secure
HOME OFFICE
STORES
DCs
Cloud
GeC
DMZ
DMZ
END-POINT LINUX
SERVERS – RHEL OR SLES
DC1
SECURE
FOREMAN- PROXY
SERVICES & DMZ HOST
CAPSULE
Secure
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5647 https/8000
https/8443 https/9090
tftp/69
PXE/TFTP
tftp/69
CLIENT TO CAPSULE: SERVICE/PORT
https/8000 https/9090
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443 ampq/5647
LOAD BALANCER
Satellite-capsule VIP
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443
ampq/5647 https/8000 https/9090
Infoblox
DHCP Next-server
foreman-proxy-capsule
SATELLITE TO CAPSULE: SERVICE/PORT
https/443 https/9050 ampq/5646 ampq/5647
USER AUTHENTICATION
SATELLITE
DATABASE
MASTER SATELLITE
Active Directory
USER ACCESS
Web User API User
idaps:tcp/636
http/80 https/443
PUPPET FACTS/REPORTS
Puppet Masters
VERSION CONTROLLED
TEMPLATES
Git/Bitbucket
https/443
https/443
PERFORMANCE METRICS
grafana
http/2003
Third Party Repos
Red Hat CDN
SUSE CDN
INTERNET REPOSITORIES AND CONTENT DELIVERY
NETWORKS
EXTERNAL PROXY
Corporate Proxy
SMT Server
Squid Proxy
Internal Repos
SLES MIRRORED REPOS
http/80 https/443
http/80 https/443 http(s)/3128
SATELLITE TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5646 ampq/5647 https/9090
http/80 https/443
http/80 https/443
http/80 https/443
https/443
https/443
http/80 https/443
DC1
DC2
DC3
...
DC1
DC2
DC3
...
![Page 15: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/15.jpg)
SECURE
Secure
HOME OFFICE
STORES
DCs
Cloud
GeC
DMZ
DMZ
END-POINT LINUX
SERVERS – RHEL OR SLES
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5647 https/8000
https/8443 https/9090
tftp/69
PXE/TFTP
tftp/69
CLIENT TO CAPSULE: SERVICE/PORT
https/8000 https/9090
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443 ampq/5647
LOAD BALANCER
Satellite-capsule VIP
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443
ampq/5647 https/8000 https/9090
Infoblox
DHCP Next-server
foreman-proxy-capsule
SATELLITE TO CAPSULE: SERVICE/PORT
https/443 https/9050 ampq/5646 ampq/5647
USER AUTHENTICATION
SATELLITE
DATABASE
MASTER SATELLITE
Active Directory
USER ACCESS
Web User API User
idaps:tcp/636
http/80 https/443
PUPPET FACTS/REPORTS
Puppet Masters
VERSION CONTROLLED
TEMPLATES
Git/Bitbucket
https/443
https/443
PERFORMANCE METRICS
grafana
http/2003
Third Party Repos
Red Hat CDN
SUSE CDN
INTERNET REPOSITORIES AND CONTENT DELIVERY
NETWORKS
EXTERNAL PROXY
Corporate Proxy
SMT Server
Squid Proxy
Internal Repos
SLES MIRRORED REPOS
http/80 https/443
http/80 https/443 http(s)/3128
SATELLITE TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5646 ampq/5647 https/9090
http/80 https/443
http/80 https/443
http/80 https/443
https/443
https/443
http/80 https/443
SECURE
DC1
DC2
DC3
...
DC1
DC2
DC3
...
DC1 FOREMAN-
PROXY SERVICES
& DMZ HOST CAPSULE
Secure
SECURE
DC1
FOREMAN- PROXY SERVICES
& DMZ HOST CAPSULE
Secure
![Page 16: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/16.jpg)
SECURE
Secure
HOME OFFICE
STORES
DCs
Cloud
GeC
DMZ
DMZ
END-POINT LINUX
SERVERS – RHEL OR SLES
DC1
SECURE
FOREMAN- PROXY
SERVICES & DMZ HOST
CAPSULE
DC1
DC2
DC3
...
Secure
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5647 https/8000
https/8443 https/9090
tftp/69
PXE/TFTP
tftp/69
CLIENT TO CAPSULE: SERVICE/PORT
https/8000 https/9090
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443 ampq/5647
LOAD BALANCER Satellite-capsule VIP
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443
ampq/5647 https/8000 https/9090
Infoblox
DHCP Next-server
foreman-proxy-capsule
SATELLITE TO CAPSULE: SERVICE/PORT
https/443 https/9050 ampq/5646 ampq/5647
USER AUTHENTICATION
SATELLITE
DATABASE
MASTER SATELLITE
Active Directory
USER ACCESS
Web User API User
idaps:tcp/636
http/80 https/443
PUPPET FACTS/REPORTS
Puppet Masters
VERSION CONTROLLED
TEMPLATES
Git/Bitbucket
https/443
https/443
PERFORMANCE METRICS
grafana
http/2003
Third Party Repos
Red Hat CDN
SUSE CDN
INTERNET REPOSITORIES AND CONTENT DELIVERY
NETWORKS
EXTERNAL PROXY
Corporate Proxy
SMT Server
Squid Proxy
Internal Repos
SLES MIRRORED REPOS
http/80 https/443
http/80 https/443 http(s)/3128
SATELLITE TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5646 ampq/5647 https/9090
http/80 https/443
http/80 https/443
http/80 https/443
https/443
https/443
http/80 https/443
![Page 17: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/17.jpg)
SECURE
Secure
HOME OFFICE
STORES
DCs
Cloud
GeC
DMZ
DMZ
END-POINT LINUX
SERVERS – RHEL OR SLES
DC1
SECURE
FOREMAN- PROXY
SERVICES & DMZ HOST
CAPSULE
DC1
DC2
DC3
...
Secure
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5647 https/8000
https/8443 https/9090
tftp/69
PXE/TFTP
tftp/69
CLIENT TO CAPSULE: SERVICE/PORT
https/8000 https/9090
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443 ampq/5647
LOAD BALANCER
Satellite-capsule VIP
CLIENT TO CAPSULE: SERVICE/PORT
http/80 https/443
https/8443
ampq/5647 https/8000 https/9090
Infoblox
DHCP Next-server
foreman-proxy-capsule
SATELLITE TO CAPSULE: SERVICE/PORT
https/443 https/9050 ampq/5646 ampq/5647
USER AUTHENTICATION
SATELLITE
DATABASE
MASTER SATELLITE
Active Directory
USER ACCESS
Web User API User
idaps:tcp/636
http/80 https/443
PUPPET FACTS/REPORTS
Puppet Masters
VERSION CONTROLLED
TEMPLATES
Git/Bitbucket
https/443
https/443
PERFORMANCE METRICS
grafana
http/2003
Third Party Repos
Red Hat CDN
SUSE CDN
INTERNET REPOSITORIES AND CONTENT DELIVERY
NETWORKS
EXTERNAL PROXY
Corporate Proxy
SMT Server
Squid Proxy
Internal Repos
SLES MIRRORED REPOS
http/80 https/443
http/80 https/443 http(s)/3128
SATELLITE TO CAPSULE: SERVICE/PORT
http/80 https/443
ampq/5646 ampq/5647 https/9090
http/80 https/443
http/80 https/443
http/80 https/443
https/443
https/443
http/80 https/443
DC1
DC2
DC3
...
![Page 18: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/18.jpg)
5 6
Hiera customizations
Apache overrides
Limits files
Kernel tunables
Postgres
Logrotate
Pulp concurrency
CUSTOMIZATIONS
© Wal-Mart Stores, Inc. 2018
![Page 19: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/19.jpg)
HIERA CUSTOMIZATIONS
/etc/foreman-installer/custom-hiera.yaml
© Wal-Mart Stores, Inc. 2018
![Page 20: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/20.jpg)
APACHE OVERRIDES
/etc/httpd/conf.d/zzz-custom-overrides.conf
© Wal-Mart Stores, Inc. 2018
![Page 21: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/21.jpg)
LIMITS FILES
/etc/systemd/system/*
© Wal-Mart Stores, Inc. 2018
![Page 22: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/22.jpg)
KERNEL TUNABLES
/etc/sysctl.conf
© Wal-Mart Stores, Inc. 2018
![Page 23: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/23.jpg)
POSTGRES
/var/lib/pgsql/data/postgresql.conf
© Wal-Mart Stores, Inc. 2018
![Page 24: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/24.jpg)
LOGROTATE
© Wal-Mart Stores, Inc. 2018
![Page 25: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/25.jpg)
PULP CONCURRENCY (Reduced Published Times)
© Wal-Mart Stores, Inc. 2018
![Page 26: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/26.jpg)
Managing Satellite Clients at Scale
© Wal-Mart Stores, Inc. 2018
![Page 27: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/27.jpg)
USING PUPPET TO AUTOMATE CLIENT SIDE OPERATIONS
PACKAGING FILES SERVICES
PUPPET MODULES
© Wal-Mart Stores, Inc. 2018
![Page 28: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/28.jpg)
USING PUPPET ENTERPRISE TO ROLLOUT IN SCALE
ENTERPRISE CONSOLE
CLASSIFICATION
GROUPS
MIGRATION LOGIC
6 5
PUPPET ENTERPRISE
© Wal-Mart Stores, Inc. 2018
![Page 29: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/29.jpg)
MODIFIED PUPPET MODULE TO RUN MULTIPLE LINUX CLIENTS
UPSTREAM SOURCE
SUSE ENTERPRISE LINUX
© Wal-Mart Stores, Inc. 2018
![Page 30: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/30.jpg)
HOW SCALE IMPACTED ROLLOUT
WHERE: WHAT: WHEN: HOW:
Started in one location Touched 60,000 boxes Delivered on an aggressive timeline Rolled out successfully
© Wal-Mart Stores, Inc. 2018
![Page 31: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/31.jpg)
Operational Improvements
© Wal-Mart Stores, Inc. 2018
![Page 32: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/32.jpg)
Walmart had challenges, but they turned them into efficiency gains with customization
Maxing Passenger, Memory, Swap Provisioning Templates Migration to Satellite 6 Publishing
© Wal-Mart Stores, Inc. 2018
![Page 33: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/33.jpg)
Added automation to roll out to a huge number of clients, multiple locations, small team
1/5 1/6 1/8 1/9 1/10 1/11 1/12 1/13 1/16 1/17 1/18 1/19 2/14 3/1 3/3
BUSINESS CHALLENGE
Need to get clients migrated in a reasonable amount of time. This required automation.
Maxing Passenger, Memory, Swap Provisioning Templates Migration to Satellite 6 Publishing
© Wal-Mart Stores, Inc. 2018
![Page 34: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/34.jpg)
BUSINESS CHALLENGE
During client registration, Satellite defaults out of the box are not optimal for large scale, and they get maxed out.
Added randomization to even the distribution
BEFORE
AFTER
Maxing Passenger, Memory, Swap Provisioning Templates Migration to Satellite 6 Publishing
© Wal-Mart Stores, Inc. 2018
![Page 35: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/35.jpg)
BUSINESS CHALLENGE
Users need new content available faster.
Performed customizations to dramatically reduce publish times
Initially impossible to publish all in one day (manually through the UI)
Scripted – still more than 24 hours because of capsule syncs
Added shared filesystem = 12 hour publish
Multi-threaded script = 4 hour publish
6.3 = 1.5 hour publish
OVER TIME
Maxing Passenger, Memory, Swap Provisioning Templates Migration to Satellite 6 Publishing
PUBLISH TIME (HOURS)
© Wal-Mart Stores, Inc. 2018
![Page 36: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/36.jpg)
BUSINESS CHALLENGE
Walmart needed version control. Template changes from one user changed the template for everyone.
Moved to fully version-controlled template process to improve workflow
Maxing Passenger, Memory, Swap Provisioning Templates Migration to Satellite 6 Publishing
© Wal-Mart Stores, Inc. 2018
![Page 37: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/37.jpg)
BUSINESS CHALLENGE
Need visibility into the operational health of the infrastructure
© Wal-Mart Stores, Inc. 2018
collectd > graphite > grafana
Ansible playbook to redeploy grafana dashboards
IMPORTANT METRICS TO COLLECT
METRICS
standard cpu/memory/disk/network
foreman tasks active (per task type)
foreman tasks per minute (per task type)
katello event queue
qpid queue depths
dynflow plans
dynflow orphaned execution plans
pulp tasks running
pulp tasks state
pulp repo count
postgres connections
postgres query length
apache processes
passenger processes
apache scoreboards
capsule apache busy servers
![Page 38: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/38.jpg)
Scalability
Build capacity for IT of any size
Collaboration
Share your ideas and optimizations
Infrastructure
Create a reference architecture then customize
SCALING SUCCESS
Migration
Know your environment and track key metrics
© Wal-Mart Stores, Inc. 2018
![Page 39: Satellite at Scale - Red Hat · Satellite at Scale How Walmart manages the massive IT infrastructure at the heart of the retail giant Darin Lively, Brian Ameling, & John Henley](https://reader036.fdocuments.in/reader036/viewer/2022070803/5f03292e7e708231d407d732/html5/thumbnails/39.jpg)
THANK YOU plus.google.com/+RedHat
linkedin.com/company/red-hat
youtube.com/user/RedHatVideos
facebook.com/redhatinc
twitter.com/RedHatNews
© Wal-Mart Stores, Inc. 2018