MADE TO TRADE METRO.CLOUD - Experiences with OpenStack in ... · METRO.CLOUD - Experiences with...

MADE TO TRADE.

METRO.CLOUD - Experiences with OpenStack in Metro

Guido Schrörs: June 2016 | © METRO SYSTEMS GmbH 2016Classification Level: Public

Guido Schrörs: June 2016 | © METRO SYSTEMS GmbH 2016MADE TO TRADE. METRO.CLOUD

.

Classification Level: Public

Made for success: METRO GROUP at a glance.

n One of the world’s leading retail and wholesale companiesn Successful stock-listed companyn Presence in over 2,000 locations in 29 countriesn More than 220,000 employees from 171 nationsn Three sales lines with leading positions

n METRO Cash & Carry: cash-and-carry wholesalen Media-Saturn: consumer electronics storesn Real: hypermarkets

n All sales lines are closely linked to online shopsn Geared towards sustainable positive earnings development


.


Group structure – profiles at a glance.

METRO AG

A leading international playerin cash-and-carrywholesale trade

Europe’s No. 1 consumer electronics

retailer; one of thelargest discount

providers in German-language

online sites

One of the leading operators

of hypermarkets in Germany

Companies

Status: 1 October 2015


.


Up to 3 x 10Gbps Internet connection

3

METRO SYSTEMS

METRO SYSTEMS is the internal IT service provider of METRO GROUP

Located in Düsseldorf + Bucharest, 1.500 employeesApplication development & operation, infrastructure provider

Datacenters in Düsseldorf, Frankfurt, Hongkong

3.000+ Intel server (Bare metal Linux + VMWare)

60 IBM Power server

25 PB Storage

8.000 Router60 Datacenter firewalls

1.200 Store firewalls

100.000 Mailboxes1.000 Domaincontroller


.


4

Challenges

Getting ready for multichannel retail business

Redesign/rebuilding applications (Front & Backend)

New modern non-monolithic application design

Microservice architecture

DevOps approach

Continuous delivery / continuous deployment


.


5

Challenges

n IT is changing fast (and will change even faster in the future)n New technologies are coming up (and disappear) frequentlyn Open source is a key factorn Reduce vendor and technology lock-in wherever possible

IT landscape

Operate, improve, optimize

New technologies

Experiment, fail, learn, repeat


.


6

Where we want to go:

“Build a Cloud Infrastructure for METRO Group

based on open source software”


.


7

METRO.CLOUD – The big picture

Platform layer

…

StorageCompute Network

Applications

Infrastructure layer

FirewallLB.

DNS…

Core network / hardware (cabling, housing, server etc.)

Storage


.


8

Challenges

“Look & feel” like an IaaS

Cloud provider based on

OpenStack

Implement OpenStack in multiple

datacenters

Start small but future proof

High performance

network architecture

Network segregation

Block and object

storage

Enable Internet access

Enable Legacy IT

access

Keep it as simple as possible


.


9

OpenStack network architecture (simplified)

Legacy IT landscape METRO.CLOUD

Metro InterCloud backbone

ExternalOpenStack

ProviderMETRO.CLOUDMETRO.CLOUD


.


10

OpenStack distribution

n SUSE OpenStack Cloud as distribution for the OpenStack setup

n SUSE Enterprise Storage for block and object storage


.


11

OpenStack extensions we made

Configuration changes

• Network adjustments (2x10 Gbps bonding)

• Network separation in (external, tenant, storage, admin, ceph-replication, etc.)

• Nova patch for availability zones and host aggregates

• Configured Glance & Cinder to use ceph copy-on-write

• …

Additional features implemented

• DNS: auto-registration and deregistration of VM hostnames (no designate in juno)

• Cinder integration for ceilometer, direct connect to RadosGW for obj. storage accounting

Some bug fixes

• Several SSL Bug fixes in different components

• Fix a bug to get the Cinder volume manager HA ready

• …


.


12

OpenStack deployment method

n Driven by Ansible playbooks, mixture of chef and Ansible automation tasks to automatically setup a complete OpenStack cloud environment (duration <= 2 days)

Ansible

SUSE Util ServerMETRO.CLOUDOpenStack cloud

environment

Run chefModify

cookbooks

Run Ansible


.


13

Basic Openstack Architecture

SAN

Compute

Ceph Monitor

Control Nodes Util Server Monitoring

Logging

Active Directory

Accounting


.


14

OpenStack Resources today

Infrastructure

90TB local Storage

17.000 vCores

17TB RAM

360TB Block Object


.


15

METRO.CLOUD Infrastructure

METRO.CLOUDInfrastructure

METRO.CLOUDDUS10

OpenStackbasedResources

IntercloudBackbone

METRO.CLOUDDUS20


METRO.CLOUDFFM10


ExternalCloud’s


METRO.CLOUDMoscow


METRO.CLOUDShanghai


Productivesince11/2015

Productivesince01/2016

Productivein05/2016

ExperimentalStatus

PlanedforEarly2017

PlanedforEarly2017


.


16

Performance

n Metro OpenStack ceph HDD pool + SSD pool vs Rackspace block storage SATA + SSD

n Method: fio from VM on 100GB volume

n Results: IOPS / higher is better


.


17

Performance

n Metro OpenStack local storage (SAN) vs Rackspace IO30 SSD data-diskn Method: fio from VM on 100GB Volume / 300 GB SSD Data-Disk

n Results: IOPS / higher is better


.


18

Operation

24 x 7 METRO.CLOUD Infrastructure operations

1st level core operations team (monitoring & alerting)2nd level operations team (incident handling and resolving)

Open Monitoring Distribution (check_mk)

BMC Service Impact Manager

ElasticSearch, Logstash,

Kibana

JIRA + Confluence

InstanaEvaluation

http://instana.io


.


19

Operation: Standard monitoring


.


20

Operation: Service monitoring


.


21



.


22



.


23

Cost & usage reports

n Collect resource usage per tenant for reporting and accounting/billingn Running instancesn Storage usage (Block & Object)n IP addressesn Per tenant & per hour

n Custom solution: use Ceilometer + Rados + Neutronn Ceilometer: instance and block storage usagen Rados: object storage usagen Neutron: IP addresses

n Data collection runs every day at midnight


.


24

The team

n We started with OpenStack in January 2015

n Mixed internal team of specialists from several different infrastructure units (network, serverhardware, storage, linux)

n Experimental phase (test, build up knowledge, design, proof of concept) till august

n Team up with codecentric OpenStack consultants in september 2015

n Buildup METRO.CLOUD OpenStack infrastructure in 3 datacenters from october 2015 – May 2016

n Currently extending team (http://metro.cloud/team)


.


25

How we work

n Infrastructure as Code

n Openstack is installed automated (ansible, crowbar, chef)

n Everything we do is based on a central git / ansibleworkflow

n Git is used for checking in software installations and configuration changes based on ansible playbooks

n We have an agile working environment based on weekly sprints in JIRA and daily standups.

n We document everything we do in confluence


.


26

Upcoming

n Team, Team, Team!

n Vanilla vs. Distribution

n Upgrade to Mitaka or Newton

n Hardware optimization

n Continuously adding new features


.


27

Contact.

METRO SYSTEMS GmbH

Guido SchrörsMetro-Strasse 1240235 DuesseldorfGermany

E-Mail: [email protected]: http://metro.cloud


.


28

Contact.

METRO SYSTEMS GmbH

Guido SchrörsMetro-Strasse 1240235 DuesseldorfGermany

E-Mail: [email protected]: http://metrosystems.net

http://metro.cloud

MADE TO TRADE METRO.CLOUD - Experiences with OpenStack in ... · METRO.CLOUD - Experiences with...

Documents

Transcript of MADE TO TRADE METRO.CLOUD - Experiences with OpenStack in ... · METRO.CLOUD - Experiences with...