CentOS, QA and OpenNebula

Opennebula Conference 2014Berlin

December 2nd to 4th

Christoph Galuschkatigalch@tigalch.org

Who am I

● Christoph Galuschka, age 37● from Innsbruck, Austria● working at TIWAG Tiroler Wasserkraft AG (utility company) in the IT

operations department● Using Linux since 1998, CentOS since 2005● Part of the CentOS-QA-Team since April 2012● Mainly working on release-QA, t_functional git repo for automated testing

and the testing infrastructure, wiki and forums

CentOS CI

● created in june 2011● an approach to automate QA-release testing● create a set of tests to run for each point release QA-cycle ● use git as a common repository for other users (yes you too) to contribute

new tests● currently contains more then 430 scripts/functional/trademark tests for more

than 110 packages● additionally uses the Linux Test Project

● upcoming: to test every update before public release

History

● First setup of CI: lost from memory :) and in a different DC● Second setup - move to opennebula● started with 4.0, 4.2 later (RPMs from CentOS-Developers)● 4.8 since November 2014● only “issue” during update: ruby gem● ‘/usr/share/one/install_gems’ vs. rpm

Setup I● 3 Dell 1950III with 4 cores, 16GB RAM and 2x300GB disks (raid1) each

● Hypervisor#1 (K6) primarily runs the jenkins VM, the other two Hypervisors are solely used for test VMs

Setup II

Setup III

● OpenNebula is used as the cloud controller● The sunstone interface is available on Hypervisor1● All VMs instantiated only available on the private network● Datastore transport → SSH● Hosts → KVM● The ci.dev.centos.org site runs as VM in that setup● nginx is used to forward http(s) from the public internet to the VM● local mirror for all CentOS-releases● mimics the setup of mirror.centos.org for all VMs● Jenkins is used to handle all running tests/VMs

Jenkins

● 4 different testgroups

● docker image (C6/C7 only), t_functional, LTP and reimzul

● run on a daily basis

● Infrastructure is hosted by GoDaddy

Jenkins II

Docker image

● pulls and runs current C5/6/7 docker image

● checks installation and command execution inside image

LTP

● Linux Test Project1

● comprehensive checks of basic linux functionality

t_functional

● functional tests of packages

● also checks ™ issues if valid

● 14 contributors● C7 running

since Nov. 7th

reimzul

● called on de- mand via IRC

● used for verifiying merge requests

● All tests run the current versions of CentOS (C5.11, C6.6, C7.0)

1) visit http://ltp.sourceforge.net/

Images used for CI● Created by Karanbir Singh and Jaime Melis● Creates images for C5 and C6 with generic kickstarts and

configurations● Images are

custombuilt on the fly and are downloadable afterwards

● C7 image comes from cloud.centos.org

KS-Sample for prjrd.netinstallurl --url=http://repohost.raindrops.centos.org/centos/6/os/x86_64/lang en_US.UTF-8keyboard uknetwork --device eth0 --bootproto dhcprootpw --iscrypted $1$ZCDym637$Q8tqR6iCTag8t7a3RA5nI.firewall --service=sshauthconfig --enableshadow --passalgo=sha512 --enablefingerprintselinux --enforcingtimezone --utc Europe/Londonbootloader --location=mbr --driveorder=sdarepo --name="CentOS" --baseurl=http://repohost.raindrops.centos.org/centos/6/os/x86_64/ --cost=100repo --name="ONE-context" --baseurl=http://repohost.raindrops.centos.org/opennebula/6/x86_64/ --cost=100zerombr yesclearpart --all --initlabelpart /boot --fstype ext3 --size=250part pv.2 --size=5000 --growvolgroup VolGroup00 --pesize=32768 pv.2logvol / --fstype ext4 --name=LogVol00 --vgname=VolGroup00 --size=1024 --growlogvol swap --fstype swap --name=LogVol01 --vgname=VolGroup00 --size=256 --grow --maxsize=512reboot%packages@Base@Coreopennebula-context%end%post --log=/root/post.log --nochroot...

KS-Sample for prjrd.netsed -i "s/^ACTIVE_CONSOLES=\/dev\/tty\[1-6\]/ACTIVE_CONSOLES=\/dev\/tty1/" /mnt/sysimage/etc/sysconfig/initsed -i '/HWADDR/d' /mnt/sysimage/etc/sysconfig/network-scripts/ifcfg-eth\*rm -f /mnt/sysimage/etc/udev/rules.d/70-persistent-net.rulesfor f in /boot/grub/grub.conf; do /bin/sed -i "s/^serial.*$//" /mnt/sysimage/${f} /bin/sed -i "s/^terminal.*$//" /mnt/sysimage/${f} /bin/sed -i "s/console=ttyS0,115200//" /mnt/sysimage/${f}donerm -rf /mnt/sysimage/etc/rc.d/rc*.d/S*sshdcat >> /mnt/sysimage/etc/rc.d/rc.local <<EOF/sbin/restorecon -Rv /root/.sshif [ ! -e /var/update_done ] then yum update -y touch /var/update_done chkconfig sshd on chkconfig auditd on rebootfiEOF%end

● This kickstart creates an image with disabled ssh-server. After boot it pulls all updates, enables ssh and reboots once (ensuring current kernel for all things to come for this VM).

introducing t_functional

● Git repo hosting all the functional tests (packages, roles)● Using #!/bin/bash (but other scripting languages are possible!)● Hosted on gitorious and git.centos.org (for QA)● Community driven (yes, you too)● Daily results at: http://ci.dev.centos.org● https://git.centos.org/summary/sig-core!t_functional.git● https://gitorious.org/testautomation/t_functional

Distro and functional tests

● Is release X.y installable?● Through ISO?● Through network (nfs/http/ftp) + pxe?● Mixed (netinstall.iso + network)?● Usb image?● UEFI vs Bios?

● Are all packages in the distro doing what they're supposed to do ?● Still working with different sets of packages (php vs php53 for example in

CentOS 5.x)

t_functional git workflow

gitoriousClone & change

Request to merge

Jenkins CI merge request

No? No?Fix it

gitorious push git.centos.org pull QA-Env

further opennebula usage

● Two further setups in DCs in London● controls four hosts in both cases● runs seven.c.o, planet.c.o, some testing-VMs, IPv6 to IPv4 Reverse Proxy,

some internal stuff● www and forum site to come● uses 4.8● GlusterFS over Infiniband● more on that: http://arrfab.net/blog/?p=485

Any Questions?

Thank you for your time