StackiFest16: Stacki 1600+ Server Journey - Dave Peterson, Salesforce

Stacki – The 1600+ Server Journey

David PetersonLead Systems Engineer

[email protected]

Agenda

▪ Why Stacki

▪ Hardware and provisioning requirements

▪ Stacki configuration with chef integration

▪ ZFS and data safe re-provisioning

▪ Detecting issues and adhoc reporting

Why Stacki

Why Stacki▪ Managing thousands of servers is easy (csv)

▪ HP raid controller support

▪ Easy out of the box provisioning but deep customization available

▪ Ability to re-provision without losing data

▪ Easy network/subnet configuration

▪ YUM repo support

▪ Command line, command line, command line

▪ Support

Hardware and Provisioning Requirements

Hardware and Provisioning RequirementsFirewall/ACL Ports

▪ bootpc and bootps (dhcp): UDP 67 and 68 ▪ tftp: UDP 69▪ tftp ephermeral: UDP 32765-65535▪ http/https: TCP 80 and 443

Hardware and Provisioning RequirementsHost RAID LUNs and Partition Setup

LUNs:▪ 4x 2TB SATA disks => RAID 10

▪ 200GB LUN (sda)▪ 7.7TB LUN (sdb)

▪ 2x 480GB SSD disks => RAID 0▪ 960GB LUN (sdc)

Hardware and Provisioning RequirementsHost RAID LUNs and Partition Setup

Partitions:▪ sda

▪ /boot, ext4 (sda) => 500MB▪ Swap => 5GB▪ /, ext4 => ~195GB

▪ sdb▪ No partitions

▪ sdc▪ sdc1 => 10GB, non-formatted▪ sdc2 => 200GB, non-formatted

Hardware and Provisioning RequirementsLatest LT Kernel and ZFS

▪ Kernel LT => 3.10.95-1▪ ZFS => 0.6.5.2

Hardware and Provisioning RequirementsChef Integration

▪ End to end server provisioning with chef▪ Chef configured on each server, host added to

chef server, and a chef-client run to apply base roles

Stacki Configuration

Stacki ConfigurationConcurrent kickstart limitation

▪ /export/stack/sbin/kickstart.cgi:L154

# Use a semaphore to restrict the number of concurrent kickstart# file generators. The first time through we set the semaphore# to the number of CPUs (not a great guess, but reasonable).

▪ semaphore = stack.lock.Semaphore('/var/tmp/kickstart.semaphore')

[root@stacki]# echo 200 > /var/tmp/kickstart.semaphore

Stacki ConfigurationCustom RAID Controller Setup

▪ /export/stack/site-profiles/prod/2.0/nodes/replace-storage-controller-client.xml

/export/stack/site-profiles/prod/2.0/nodes/replace-storage-controller-client.xml

<?xml version="1.0" standalone="no"?>

<kickstart><pre>if [ "&nukecontroller;" == "true" ]then /opt/stack/sbin/hpssacli ctrl slot=0 delete forced override

/opt/stack/sbin/hpssacli ctrl slot=0 create type=ld \ drives=1I:1:1,1I:1:2,1I:1:3,1I:1:4 raid=1+0 size=200000 /opt/stack/sbin/hpssacli ctrl slot=0 create type=ld \ drives=1I:1:1,1I:1:2,1I:1:3,1I:1:4 raid=1+0

/opt/stack/sbin/hpssacli ctrl slot=0 create type=ld \ drives=2I:0:5,2I:0:6 raid=0

fi</pre>



<pre><eval>/opt/stack/bin/stack set host attr &hostname; attr=nukecontroller value=false</eval></pre>

</kickstart>

/export/stack/site-profiles/prod/2.0/nodes/replace-storage-controller-client.xml

<pre cond="appliance in [’rabbitmq']">

if [ "&nukecontroller;" == "true" ]then /opt/stack/sbin/hpssacli ctrl slot=0 delete forced override

/opt/stack/sbin/hpssacli ctrl slot=0 create type=ld \ drives=1I:1:1,1I:1:2,1I:1:3,1I:1:4 raid=1+0 size=500000 /opt/stack/sbin/hpssacli ctrl slot=0 create type=ld \ drives=1I:1:1,1I:1:2,1I:1:3,1I:1:4 raid=1+0

/opt/stack/sbin/hpssacli ctrl slot=0 create type=ld \ drives=2I:0:5,2I:0:6 raid=1fi

</pre>

/export/stack/site-profiles/prod/2.0/nodes/extend-backend.xml

<boot order="post">

# Set HP RAID controller cache settings/opt/stack/sbin/hpssacli ctrl slot=0 ld 1 modify arrayaccelerator=enable/opt/stack/sbin/hpssacli ctrl slot=0 ld 2 modify arrayaccelerator=enable/opt/stack/sbin/hpssacli ctrl slot=0 modify cacheratio=80/20/opt/stack/sbin/hpssacli ctrl slot=0 array b modify ssdsmartpath=disable/opt/stack/sbin/hpssacli ctrl slot=0 logicaldrive 3 modify arrayaccelerator=enableecho "y" | /opt/stack/sbin/hpssacli ctrl slot=0 modify dwc=enable

</boot>

Stacki ConfigurationCustom Partitions

▪ /export/stack/site-profiles/prod/2.0/nodes/extend-partition.xml

/export/stack/site-profiles/prod/2.0/nodes/extend-partition.xml

<?xml version="1.0" standalone="no"?><kickstart><post><![CDATA[/sbin/fdisk /dev/sdc << EOFdwEOF

/sbin/fdisk /dev/sdc << EOFnp11+10GwEOF

/sbin/fdisk /dev/sdc << EOFnp2

+200GwEOF]]></post></kickstart>

Stacki ConfigurationCustom Appliance Types

▪ /export/stack/site-profiles/prod/2.0/nodes/extend-backend.xml

[root@stacki]# stack list applianceAPPLIANCE MEMBERSHIP PUBLICfrontend: Frontend nobackend: Backend yesrabbitmq: Rabbitmq yesredis: Redis yesmysql: Mysql yes

[root@stacki]# stack add appliance loadbalancer[root@stacki]# stack set appliance attr loadbalancer attr=managed value=true[root@stacki]# stack set appliance attr loadbalancer attr=kickstartable value=true[root@stacki]# stack set appliance attr loadbalancer attr=node value=backend


<boot order="post" cond="appliance == ’rabbitmq' and (nukecontroller or nukedisks)">/sbin/zfs create data/rabbitmq/sbin/zfs set mountpoint=/var/lib/rabbitmq data/rabbitmq</boot>

<boot order="post" cond="appliance == ’redis' and (nukecontroller or nukedisks)">/sbin/zfs create data/redis/sbin/zfs set mountpoint=/var/lib/redis data/redisadduser -r redis -Uchown redis:redis /var/lib/redis</boot>

<boot order="post" cond="appliance == 'mysql' and (nukecontroller or nukedisks)">

# Disabling THP<![CDATA[echo never > /sys/kernel/mm/transparent_hugepage/enabledecho never > /sys/kernel/mm/transparent_hugepage/defrag

sed -i 's/kernel.* console=ttyS0,19200n8$/& transparent_hugepage=never/' /boot/grub/grub.confsed -i 's/kernel.* crashkernel=auto$/& transparent_hugepage=never/' /boot/grub/grub.conf]]>

/sbin/zfs create data/mysql/sbin/zfs create data/mysql-log/sbin/zfs create data/mysql-tmp/sbin/zfs set recordsize=16K data/mysql/sbin/zfs set mountpoint=/var/lib/mysql data/mysql/sbin/zfs set mountpoint=/var/log/mysql data/mysql-log/sbin/zfs set mountpoint=/var/lib/mysql/tmp data/mysql-tmpadduser -r mysql -Uchown mysql:mysql /var/lib/mysql /var/log/mysql /var/lib/mysql/tmp</boot>

Stacki ConfigurationCustom Appliance Types

Stacki ConfigurationChef Cart

▪ /export/stack/carts/chef/nodes/cart-chef-backend.xml

/export/stack/carts/chef/nodes/cart-chef-backend.xml

<?xml version="1.0" standalone="no"?><kickstart>

<description> chef cart backend appliance extensions

</description> <package>chef</package><post>mkdir -p /etc/chef /var/log/chef /var/run/chef</post>

<post cond="not ‘proxy’ in hostname"><file name="/etc/chef/client.rb"><![CDATA[## Chef Client Config File## Dynamically generated by Stacki#log_level :infolog_location STDOUTchef_server_url "#CHEF_SERVER#"validation_client_name ”chef-validator"validation_key "/etc/chef/validation.pem"client_key "/etc/chef/client.pem"ssl_verify_mode :verify_nonehttp_proxy 'http://proxy1:3128'https_proxy 'http://proxy2:3128'no_proxy ’test1,localhost,127.0.0.1'environment 'production'# Using default node name (fqdn)node_name "#HOSTNAME#”Ohai::Config[:plugin_path] << '/etc/chef/ohai']]></file>

# Need to add the chef server and client hostname to the client.rb filesed -i 's,#CHEF_SERVER#,&chef_server;,g' /etc/chef/client.rbsed -i 's/#HOSTNAME#/&hostname;.&domainname;/g' /etc/chef/client.rb

</post>

/export/stack/carts/chef/nodes/cart-chef-backend.xml

<post><file name="/etc/chef/first-boot.json">{ "run_list": [

"role[base_role]", "role[dc_sfo]" ]}</file></post>

# If we are nuking disks we are assuming this is a new server# or the chef client/node has been deleted out of the chef server if it existed.<boot order="post" cond="nukedisks">

# Run chef-client for the first time/usr/bin/chef-client -j /etc/chef/first-boot.json -L /var/log/chef/chef.log

# Make a backup of the chef private key in case we need to re-provision/upgrade a servermkdir -p /data/chef-backupchown root:root /data/chef-backupchmod 700 /data/chef-backup

cp -a /etc/chef/* /data/chef-backup</boot>

# If we are not nuking the disks we are assuming we are re-loading or upgrading# the OS and need to keep the client.pem chef key so chef-client can run properly<boot order="post" cond="not nukedisks">

cp /data/chef-backup/client.pem /etc/chef//usr/bin/chef-client -L /var/log/chef/chef.log</boot>

Stacki ConfigurationRCS Issues

▪ Stacki installs foundation-rcs package on provisioned servers

▪ Caused issues for our rsyslog daemon because of RCS config files being loaded. Other daemons were affected as well.

▪ Let’s remove it and clean up all the RCS directories


<boot order="post">

# Remove rcs rpm and cleanup RCS directoriesrpm -e foundation-rcsfind / -type d -name 'RCS' -print0 |xargs -0 rm –rf

</boot>

ZFS and Data Safe Provisioning

ZFS and Data Safe ProvisioningWhat is ZFS?

▪ A combined file system and logical volume manager▪ Data integrity▪ Software raid▪ Storage pools▪ Sophisticated caching: ARC (RAM MFU/MRU), L2ARC

(SSDs), ZIL/SLOG▪ Snapshots and Clones▪ Compression

ZFS and Data Safe ProvisioningZFS and Latest Kernel Installation

▪ YUM repos imported into Stacki▪ http://elrepo.org/▪ http://zfsonlinux.org/

<?xml version="1.0" standalone="no"?><kickstart>

<package>kernel-lt</package><package>kernel-lt-devel</package><package>kernel-lt-headers</package><package>zfs</package>


http://elrepo.org/

http://elrepo.org/

http://zfsonlinux.org/

http://zfsonlinux.org/


<post>

# Enable kernel 3.xsed -i 's/^default=.*/default=1/g' /boot/grub/grub.conf

# Add zfs module config optionsecho "options zfs zfs_arc_max=34359738368" >> /etc/modprobe.d/zfs.confecho "options zfs zfs_nocacheflush=1" >> /etc/modprobe.d/zfs.confecho "options zfs zfs_read_chunk_size=1310720" >> /etc/modprobe.d/zfs.confecho "options zfs zfs_prefetch_disable=1" >> /etc/modprobe.d/zfs.confecho "options zfs zil_slog_limit=104857600" >> /etc/modprobe.d/zfs.conf

</post>


<boot order="post" cond="nukedisks">

/sbin/modprobe zfs

/sbin/zpool create -f data sdb log sdc1 cache sdc2/sbin/zfs set atime=off data/sbin/zfs set compression=lz4 data

# Add /opt filesystem/sbin/zfs create data/opt/bin/mv /opt/* /data/opt//bin/rm -rf /opt/sbin/zfs set mountpoint=/opt data/opt

# Add /var/log/httpd filesystem/sbin/zfs create data/httpd-log/sbin/zfs set mountpoint=/var/log/httpd data/httpd-logchmod 700 /var/log/httpd

# Add /var/log/logstash filesystem/sbin/zfs create data/logstash/sbin/zfs set mountpoint=/var/log/logstash data/logstashadduser -r logstash -Uchown logstash:logstash /var/log/logstash

echo "create zfs data pool..." > /tmp/zfs-create.log

</boot>


<boot order="post" cond="not nukedisks">

# We need to empty/move the data in /opt before we can import zfsmkdir /tmp/optmv /opt/* /tmp/opt/

/sbin/modprobe zfs/sbin/zpool import -d /dev/disk/by-path/ dataecho "Importing zfs data pool..." > /tmp/zfs-import.log

mv /tmp/opt/* /opt/rm -rf /tmp/opt

</boot>

Detecting Issues and AdHoc Reporting

Detecting Issues and AdHoc ReportingWhat? We have Issues?▪ Stacki is great at provisioning but getting the status of a provisioned

or currently being provisioned server is a little harder.▪ A couple different ways but at various stages in the provisioning

process:1. Tailing /var/log/messages for DHCP requests and acks2. Watching the nukecontroller and nukedisks attributes3. Tailing /var/log/httpd/access_log for rpm downloads4. Watching the boot action flag5. iftop6. Chef node entry

▪ Note: Tailing log files for a couple servers is fine but when provisioning hundreds of servers at a time, it is not viable.

Detecting Issues and AdHoc ReportingWhat? We have Issues?

▪ Watching the nukecontroller and nukedisks attributes

[root@stacki]# stack list host attr chef1-1 |grep nukechef1-1: -------------------- nukecontroller true Hchef1-1: -------------------- nukedisks true H

192.168.10.50 - - [09/Feb/2016:20:39:52 -0700] "GET /install/sbin/public/setDbPartitions.cgi HTTP/1.1" 200 1

/var/log/httpd/ssl_access_log


▪ Tailing /var/log/httpd/access_log for rpm downloads

192.168.10.50 - - [09/Feb/2016:17:09:32 -0700] "GET /install/distributions/prod/x86_64/RedHat/RPMS/gtk2-2.24.23-6.el6.x86_64.rpm HTTP/1.1" 200 3339880 "-" "-"192.168.10.50 - - [09/Feb/2016:17:09:31 -0700] "GET /install/distributions/prod/x86_64/RedHat/RPMS/hdparm-9.43-4.el6.x86_64.rpm HTTP/1.1" 200 83060 "-" "-”192.168.10.50 - - [09/Feb/2016:17:09:32 -0700] "GET /install/distributions/prod/x86_64/RedHat/RPMS/libXext-1.3.2-2.1.el6.x86_64.rpm HTTP/1.1" 200 35644 "-" "-"192.168.10.50 - - [09/Feb/2016:17:09:32 -0700] "GET /install/distributions/prod/x86_64/RedHat/RPMS/filesystem-2.4.30-3.el6.x86_64.rpm HTTP/1.1" 200 1057228 "-" "-"192.168.10.50 - - [09/Feb/2016:17:09:32 -0700] "GET /install/distributions/prod/x86_64/RedHat/RPMS/NetworkManager-0.8.1-99.el6.x86_64.rpm HTTP/1.1" 200 1185212 "-" "-"192.168.10.50 - - [09/Feb/2016:17:09:32 -0700] "GET /install/distributions/prod/x86_64/RedHat/RPMS/libstoragemgmt-1.2.3-1.el6.x86_64.rpm HTTP/1.1" 200 211068 "-" "-"

tail –f /var/log/httpd/access_log | grep –E “192.168.10.50|192.168.10.51”


▪ Watching the boot action flag

[root@stacki]# stack list host boot chef1-*HOST ACTIONchef1-2: installchef1-1: os

192.168.10.50 - - [09/Feb/2016:20:39:52 -0700] "GET /install/sbin/public/setPxeboot.cgi?params={\"action\":\"os\"} HTTP/1.1" 200 1

/var/log/httpd/ssl_access_log


▪ Issues we encountered• TORs ip helper-address not set properly• ACL mis-match between racks causing DHCP/TFTP to be

blocked• Mis-configured host networks causing the gateways to be wrong

which prevented DHCP/PXE from working properly• Post boot zfs commands not running properly due to hardware

missing drives

Detecting Issues and AdHoc ReportingAdHoc Reporting

▪ Find all hosts that still have the “install” flag and generate a report

for h in `stack list host boot |grep -w install|awk '{print $1}'|sed s/://`; \do for ip in `stack list host interface $h|grep eth0|awk '{print $5}'`; \do echo -e "Host: $h\nChecking for IP: $ip"; echo ""; \cat /var/log/messages /var/log/httpd/ssl_access_log /var/log/httpd/access_log|grep -iw $ip; echo ""; \done; done > host_report.txt

Host: test1Checking for IP: 192.168.10.50

Feb 9 19:32:12 stacki-host dhcpd: DHCPOFFER on 192.168.10.50 to ba:c2:3d:c3:ab:13 via 192.168.10.1Feb 9 19:32:12 stacki-host dhcpd: DHCPOFFER on 192.168.10.50 to ba:c2:3d:c3:ab:13 via 192.168.10.1Feb 9 19:32:16 stacki-host dhcpd: DHCPREQUEST for 192.168.10.50 (192.168.10.5) from ba:c2:3d:c3:ab:13 via 192.168.10.1Feb 9 19:32:16 stacki-host dhcpd: DHCPACK on 192.168.10.50to ba:c2:3d:c3:ab:13 via 192.168.10.1Feb 9 19:32:16 stacki-host dhcpd: DHCPREQUEST for 192.168.10.50 (192.168.10.5) from ba:c2:3d:c3:ab:13 via 192.168.10.1Feb 9 19:32:16 stacki-host dhcpd: DHCPACK on 192.168.10.50 to ba:c2:3d:c3:ab:13 via 192.168.10.1192.168.10.50 - - [09/Feb/2016:19:32:54 -0700] "GET /install/sbin/kickstart.cgi?arch=x86_64&np=40 HTTP/1.1" 200 96101192.168.10.50 - - [09/Feb/2016:19:33:13 -0700] "GET /install/distributions/prod/x86_64/images/updates.img HTTP/1.1" 404 329 "-" "-”192.168.10.50 - - [09/Feb/2016:19:33:33 -0700] "GET /install/distributions/prod/x86_64/images/product.img HTTP/1.1" 200 782336 "-" "-"192.168.10.50 - - [09/Feb/2016:19:33:35 -0700] "GET /install/distributions/prod/x86_64/images/install.img HTTP/1.1" 200 236163072 "-" "-"



Detecting Issues and AdHoc ReportingAdHoc Reporting

▪ Find the top racks with the most un-provisioned hosts. Helps us identify racks with potential ACL issues.

[root@stacki]# stack list network|awk '{print $1}’NETWORKrack1-prod_vlan1:rack2-prod_vlan2:rack3-prod_vlan1:rack4-prod_vlan2:rack5-prod_vlan2:

[root@stacki]# for h in `stack list host boot |grep -w install|awk '{print $1}'|sed s/://`; do stack list host interface $h; done |grep eth0|awk '{print $3}'|cut -d- -f 1|sort|uniq -c|sort -rn|head 40 rack2 9 rack3 7 rack5 6 rack1 6 rack4

Lessons Learned

▪ With thousands of servers, you need a standard naming convention for hosts, networks, appliance types, etc.

▪ Standardized servers saves you time and headaches.

▪ Created custom scripts to augment stacki functionality and reduce human errors• create-stack-appliances.sh: This script will look for appliance types in the extend-backend.xml file, check to

see if they already exist and if not, create them in Stacki.• create-stack-networks.sh: This script will import a list of networks from a csv file you specify.• stack-hosts.sh: This script enables or disables provisioning of hosts listed in a file and can optionally set the

nuke attributes.

▪ Stacki by default does not allow you to have a high number of concurrent kickstart sessions.

▪ When making config changes, verify proper syntax and expected output by running: stack list host profile <hostname> | less

thank y u

StackiFest16: Stacki 1600+ Server Journey - Dave Peterson, Salesforce

Software

Transcript of StackiFest16: Stacki 1600+ Server Journey - Dave Peterson, Salesforce