OpenNebulaConf 2014 - CentOS, QA and OpenNebula - Christoph Galuschka
OpenNebulaConf 2016 - OpenNebula, a story about flexibility and technological agnosticism by ...
-
Upload
opennebula-project -
Category
Technology
-
view
68 -
download
1
Transcript of OpenNebulaConf 2016 - OpenNebula, a story about flexibility and technological agnosticism by ...
2016 Todo en Cloud S.L. www.todoencloud.com
Todo En CloudArchitecture as a Service
OpenNebula, a story about flexibility and technological agnosticism
Alberto Picón Couselo
CIO Todoencloud
25/10/2016
2016 Todo en Cloud S.L. www.todoencloud.com
Cloud providers are constantly
addressing the technology
limitations on their
infrastructures, which must be
overcome to meet customer
needs.
The technological agnosticism
and flexibility of OpenNebula
has allowed Todoencloud to
provide the most efficient
solutions to the needs of its
customers
“Opennebula’sTechnological Agnosticism”
2016 Todo en Cloud S.L. www.todoencloud.com
Why do they choose us?
Our clients choose us for our
technical expertise, our stability
and our Disaster Recovery
capacity, our management,
High Security (certificates in
27001) and overall
management capacity of our
clients’ business architecture.
Todo En Cloud
2016 Todo en Cloud S.L. www.todoencloud.com 4
Public Cloud Solution
Thanks to our capabilities,
to our human, technical and
technological resources, we
are a reference in the
delivery of
“Architecture as a service”.
Our cloud infrastructure is largely
based on Free and Open Source
Software, which allows us to be more
efficient and competitive.
We do not subcontract nor delegate
the management of the Cloud
infrastructure to any third parties,
which means we can improve the
Agreement Standards in our services
with our clients, guaranteeing them
that their data will not be passed on to
any business competitors or
governments.
All the infrastructure is located in two
Data Centers.
.
A NEUTRAL Spanish Data Center
(Tier IV), by which we can guarantee
that it complies with LOPD
standards, and a second CPD Tier II
in order to provide DRP services.
Besides, its information is in a
completely clusterized environment.
It is possible to carry out complete
system copies within personalised
frequency periods.
Our communication backbone is of
10Gbe and of 40Gbe both in its core
and in INTERNET communications.
TodoenCloud Services
2016 Todo en Cloud S.L. www.todoencloud.com
The New Cloud Generation
Our cloud core infrastructure is
always based on Free Software
and Open Source technology
solutions
This approach provide us the
chance to explore and deploy
technology without vendor lock-in
issues
TodoEnCloudTechnological
Approach
2016 Todo en Cloud S.L. www.todoencloud.com 6
Opennebula. Our Cloud Orchestrator
� Opennebula since version 3.2
� Opennebula provide us the flexibility and technology agnosticism we need for
our cloud management core and to deploy services to our customers
� Few Components, robust and rock solid cloud orchestrator
Xen is our Core Virtualization Technology
� Technologically Neutral (Software and Hardware)
� High Performance and total VM isolation
� Low impact between VM Workloads
ZFS storage clusters
� Our storage core is based on ZFS storage clusters
� SSD and SATA with SSD R/W Cache Tiers
� VMs (DataStore Level) and NFS FileSystem Snapshots
� ZFS Send/Receive for remote replication
Todoencloud Technological Evolution (I)
2016 Todo en Cloud S.L. www.todoencloud.com 7
Xen Virtualization and Opennebula
� Opennebula Xen Support is great!
� However:
More and more features are supported for Libvirtd Linux KVM World
� Limiting disk IOPs and network bandwidth on Xen is not supported
� Xen version support and migration can be a REAL challenge
� Xen live migration require same hardware architecture between
hypervisors
� Some Linux Distributions decided to remove Xen Support on new
releases (yes, those Xen-* packages are not used at all, you know)
Todoencloud Technological Evolution (II)
2016 Todo en Cloud S.L. www.todoencloud.com 8
ZFS Known Limitations
ZFS Read and Write Speed is limited by:
� Maximum I/O of the slowest disk
� Maximum throughput of the ZIL and Read Cache
� RAIDZ design and number of disks per pool
ZFS scalability is expensive:
� Capacity expansion per tier can be very expensive and
delivers performance degradation
ZFS High Availability solutions can fail:
� ZFS export and import processes between
active/passive nodes can fail during takeover
� Migration impact over all ZFS pools on the active node
Todoencloud Technological Evolution (IV)
2016 Todo en Cloud S.L. www.todoencloud.com 9
Opennebula Upgrade
� Migration to Opennebula with latest Ceph Features
Add Full KVM support (without loosing Xen hypervisors)
� Using Paravirtualized hardware support: virtio-scsi virtio-blk and virtio-net
� Disk IOPs Limitation using Opennebula Sunstone Templates and Libvirtd
� Network Bandwidth Limitation using Opennebula Hook VM control
� New! FreeBSD now supports KVM VirtIO disk and networking drivers!
� KVM Live migration is fully supported
Add a CEPH Cluster (without loosing ZFS Shared Model)
� Opensource Technology
� SSD Pool and SATA Ceph Pool with SSD Journals
� Scripting for Scheduled VM Snapshotting and snap rotation
� CephFS Support!!
� HA NFS-Ganesha VM servers
� HA Samba-VFS CephFS VM servers
Todoencloud Technological Evolution (V)
2016 Todo en Cloud S.L. www.todoencloud.com 10
� Ceph design, deployment and maintenance is not an easy task (not at all!)
� Efficient CRUSHMap design and Node/Component Failure Testing is a MUST!
� 3 Ceph Monitor servers are required
1 CPU Intel(R) Xeon(R) CPU E5-2640 and 32 GB are enough
� At least 2 Ceph MDS servers (active/passive) for CephFS are required
1 CPU Intel(R) Xeon(R) CPU E5-2640
RAM: The bigger the better!
4Kbytes/inode -> 64GB available RAM ~ 16M of inodes on memory
� Ceph requires a minimum number of OSD nodes to be usable
10 OSD nodes per tier should be considered the minimum
1 Logical CPU (for instance, E5-2640) and 2GB RAM / OSD process/disk
� Special attention to the number of rotational disks per journal SSD disk
� Setting up the size of a SSD journal partition for a rotational SATA/SAS disk:
SSD partition size should not exceed 5s of maximum network throughput.
For example: 10GB NIC -> 6~8GB journal partition
Ceph v10.2.2. Deployment Challenges (I)
2016 Todo en Cloud S.L. www.todoencloud.com 11
CephFS v10.2.2. Deployment Challenges (I)
� CephFS looks really cool, why not?. Let’s try it!
� What?!!, CephFS in Production!!!?
� CephFS “Jewel” is considered production ready since April 21st 2016
� However, CephFS Snapshots are NOT SUPPORTED -> FS CORRUPTION!!!
� Our customers need a scalable shared Filesystem to interconnect their VMs
� NFS servers using conventional LVM RBD disks for VM storage are dangerous too
2016 Todo en Cloud S.L. www.todoencloud.com 12
CephFS v10.2.2. Deployment Challenges (II)
� CephFS Solution consists of the following:
Two MDS servers 1 CPU Intel(R) Xeon(R) CPU E5-2640 128GB RAM Active/Passive
CephFS Metadata over SSD Disks, CephFS Data over SATA/SSD Multitenant FS Structure
NFS-Ganesha 2.3, Samba 4, and CephFS Kernel driver active/passive PaceMaker cluster
� Tested during a whole month of an extreme 24/7 file copying process
45TB of files copied, more than 3500M of files
� Ceph metadata is read/written from CephFS clients using MDS Server memory:
4KBytes/inode -> 30M of inodes ~ 120GB of RAM
� MDS Ceph master configuration file /etc/ceph/ceph.conf:
[mds]mds_cache_size = 30000000
� MDS max inodes can be changed live using:
ceph daemon mds.mds_server_name config set mds_cache_s ize 30000000
� Directories and files on a CephFS can be assigned to a specific pool using setfattr/getfattr tool:
setfattr -n ceph.dir.layout.pool -v <ceph_pool> <direc tory>setfattr -n ceph.file.layout.pool -v <ceph_pool> <fil e>
� These attributes will be assigned to NEW files and nested directories created on CephFS
� Stablishing a correct directory hierarchy from start is VERY IMPORTANT
2016 Todo en Cloud S.L. www.todoencloud.com 13
� NFS-Ganesha Compilation on Setup procedure:
apt-get install git-core cmake build-essential portm ap libcephfs-dev bison flex libkrb5-dev libtirpc1 uuid-dev apt-get install attr nfs-common libnfsidmap-dev
cd /usr/srcgit clone https://github.com/nfs-ganesha/nfs-ganesha .gitcd nfs-ganeshagit checkout -b V2.3-stable origin/V2.3-stablegit submodule update --init
mkdir buildcd buildcmake -DUSE_NFSIDMAP=ON ../srcmake; make install
� NFS-Ganesha DBUS Setup procedure:
cp ../src/scripts/ganeshactl/org.ganesha.nfsd.conf /e tc/dbus-1/system.d/
CephFS v10.2.2. NFS-Ganesha Compilation and Setup (I)
2016 Todo en Cloud S.L. www.todoencloud.com 14
� /etc/ganesha/ganesha.conf
# NFS-Ganesha Master configuration fileNFSv4{
IdmapConf = /etc/idmapd.conf;}
EXPORT{
Export_ID = 1;Path = "/customer1";Pseudo = "/";Access_Type = RW;Protocols = 4;Squash = None;Transports = TCP;SecType = sys;FSAL {
Name = CEPH;}
}
CephFS v10.2.2. NFS-Ganesha Compilation and Setup (II)
2016 Todo en Cloud S.L. www.todoencloud.com 15
� Systemd service file, /etc/systemd/system/nfs-ganesha.service
[Unit]Description=NFS GaneshaWants=network.target network-online.targetAfter=network.target network-online.target
[Service]Type=simpleExecStart=/usr/bin/ganesha.nfsd -F -f /etc/ganesha/g anesha.conf -L /tmp/ganesha.logExecStop=/bin/kill -SIGTERM $MAINPIDExecReload=/bin/kill -HUP $MAINPIDKillMode=processRestart=on-failureRestartSec=5s
[Install]WantedBy=multi-user.target
CephFS v10.2.2. NFS-Ganesha Compilation and Setup (III)
2016 Todo en Cloud S.L. www.todoencloud.com 16
~# systemctl enable nfs-ganesha.service~# systemctl start nfs-ganesha.service~# systemctl systemctl status nfs-ganesha.service
� Add rpc.idmapd to /etc/rc.local:
/usr/sbin/rpc.idmapd
� Set NFS Domain to Idmap service, /etc/idmap.conf
Domain = localdomain
� Set IDMAPD service as a needed service, /etc/default/nfs-common
# Do you want to start the idmapd daemon? It is only needed for NFSv4.NEED_IDMAPD="yes“
� Create /etc/ceph/secret.conf
File contents the password hash of the user specified in /etc/fstab to mount CEPHFS target)
� Add CephFS mount point using native CephFS Kernel driver in /etc/fstab.conf file:
mon01,mon02,mon03:/cephfs/customer1 /mnt/cephfs/cu stomer1 cephname=customer1,secretfile=/etc/ceph/secret.conf,noa time,nodiratime,_netdev0 1
CephFS v10.2.2 Deployment Challenges (II). NFS-Ganesha Compilation and Setup (IV)
2016 Todo en Cloud S.L. www.todoencloud.com 17
� Opennebula uses SSH access to KVM hypervisors to manage RBD images
� Ubuntu LTS 14.04 KVM hypervisor installation is easy and straightforward
� VM Live migration problems due to apparmor daemon and libvirtd security profiles:
ln -s /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmo r.d/disable/ln -s /etc/apparmor.d/usr.lib.libvirt.virt-aa-helpe r /etc/apparmor.d/disable/apparmor_parser -R /etc/apparmor.d/usr.sbin.libvirtdapparmor_parser -R /etc/apparmor.d/usr.lib.libvirt.v irt-aa-helperinvoke-rc.d apparmor reload/etc/init.d/libvirt-bin restart
� RBD Ceph v.10.2.1 exclusive lock feature led to KVM random VM crash during
scheduled RBD snapshot execution (librbd/ExclusiveLock.cc: 197: FAILED assert)
https://bugs.launchpad.net/ubuntu/+source/ceph/+bug/1607694
� Workaround: Disable exclusive lock for registered RBD images and adding global option
“rbd default features = 33 ” to ceph.conf on KVM hypervisors
� Bug seems to be corrected on Ceph v.10.2.3 (Test Pending)
KVM and Ceph Challenges (I)
2016 Todo en Cloud S.L. www.todoencloud.com 18
� Opennebula does not support BW limitation per interface through Libvirtd
� It is possible to limit BW using a VM Hook during VM Deployment when it reaches
RUNNING state
� VM Hook on /etc/one/oned.conf:
VM_HOOK = [name = "set_vm_network_speed“on = "RUNNING“command = “net_bwlimit.sh“arguments = "$ID $TEMPLATE" ]
� We can transfer TEMPLATE VARIABLES to our script using the following code:
XPATH="/var/tmp/one/datastore/xpath.rb -b $2“unset i j XPATH_ELEMENTSwhile IFS= read -r -d '' element; do XPATH_ELEMENTS[i++]="$element" done < <($XPATH /VM/TEMPLATE/VARIABLE) VARIABLE="${XPATH_ELEMENTS[j++]}"
KVM and Ceph Challenges (II)Opennebula Network Interface Bandwidth limitation through VM Hook (I)
2016 Todo en Cloud S.L. www.todoencloud.com 19
� Create /var/lib/one/remotes/hooks/net_bwlimit.sh script:#!/bin/shVM_ID=$1XPATH="/var/tmp/one/datastore/xpath.rb -b $2“unset i j XPATH_ELEMENTSwhile IFS= read -r -d '' element; do XPATH_ELEMENTS[i++]="$element"
done < <($XPATH /VM/TEMPLATE/NETBW) NETBW="${XPATH_ELEMENTS[j++]}“if [ $(sudo virsh domiflist one-${VM_ID} | awk '/vnet/ { print $1 }' | wc -l) -ge 1 ]then
for iface in $(sudo virsh domiflist one-${VM_ID} | awk ' /vnet/ { print $1 }')do
sudo virsh domiftune one-${VM_ID} ${iface} --live --in bound $NETBW,$NETBW,$NETBW --outbound $NETBW,$NETBW,$NETBW
donefiexit 0
� chmod 755 /var/lib/one/remotes/hooks/net_bwlimit.sh� chown oneadmin:oneadmin /var/lib/one/remotes/hooks/ne t_bwlimit.sh
� Execute “onehost sync ” as user oneadmin on Opennebula node to sync script to HVs
KVM and Ceph Challenges (II)Opennebula Network Interface Bandwidth limitation through VM Hook (II)
2016 Todo en Cloud S.L. www.todoencloud.com 20
� Use a CUSTOM Tag to set NETBW variable to the VM Template (value in KBps):
� After a successful VM deployment, the network interface bandwidth is correctly set-up:
~# virsh domiftune one-477 vnet0inbound.average: 131000inbound.peak : 131000inbound.burst : 131000inbound.floor : 0outbound.average: 131000outbound.peak : 131000outbound.burst : 131000
KVM and Ceph Challenges (IV)Opennebula Network Interface Bandwidth limitation through VM Hook (III)
2016 Todo en Cloud S.L. www.todoencloud.com
The New Cloud Generation.
We are going to describe our
product and how it is
consumed, both from the
standpoint of an IT department
as from that of a business
department.
Cloud Bursting As
Business
2016 Todo en Cloud S.L. www.todoencloud.com
More Information about us:
www.todoencloud.com
+34910801233
Thank you!