Ceph TestDrive Module3 v16

RED HAT CEPH STORAGE

Lab Guide – Ceph Object Gateway & Ceph Pools

Lab Description

Welcome to the third installment in the Red Hat Ceph Storage Test Drive series.

In this lab we will be focusing on the installa?on and configura?on of:

•  Ceph Object Storage Gateway •  Replicated Storage Pool

•  Erasure Coded Storage Pool

AGENDA

•  Register for Red Hat Storage Test Drive •  Launch Lab : Red Hat Ceph Object Gateway and Ceph Pools •  AWS EC2 Lab Deployment Architecture Overview •  Accessing Servers •  Lab Setup Commands •  Setup/Configure Ceph Object Store Gateway •  Setup/Configure Replicated Storage Pool

•  Generate/View object(s) •  Snapshot pool •  Restore from Snapshot

•  Setup/Configure Erasure Coded Storage Pool

Register for Red Hat Storage Test Drive Go to the URL: h+ps://engage.redhat.com/aws-‐test-‐drive-‐201308271223

Class Selection

Once logged in you will see that courses available to you. In this Test Drive we are focusing on the Ceph Training modules. Select Ceph Training (highlighted) to get started.

Lab Launch Page

Select the Lab you wish to work on by selecNng Start Lab bu+on.

Lab - Initializing

Lab - Tabs

Each tab has specific informaNon relaNng to the lab you are taking

Ceph Concepts: This tab contains detail on the components that make up a Ceph storage cluster

Lab Instruc?ons: In this tab you will find instrucNons on how to set up and run the lab Connec?on: The SSH key pairs can be found in this tab. Only needed if you have connecNon problems

ADDL. INFO: IP Addresses of the nodes in the lab

LAB IP INFORMATION Under ADDL INFO tab, you should see the IP addresses of all the nodes. Note: The ManagementNodePublicIP in your lab will likely be different than appears here.

Lab Concepts

Ceph Calamari: Calamari is a graphical management and monitoring system for Ceph storage cluster.

Ceph Monitor: A Ceph monitor maintains a master copy of the Ceph storage cluster map with the current state of the storage cluster.

Ceph OSD (Object Storage Device): Ceph OSDs store data on behalf of Ceph clients. AddiNonally Ceph nodes perform data replicaNon, rebalancing, recovery, monitoring and reporNng funcNons. Ceph PG (Placement Groups): PGs are logical containers for objects.

Ceph Pools: A Pool is a logical parNNon for storing objects via the PGs assigned to a pool.

Ceph Object Storage Gateway (RADOSGW): Object storage gateway supporNng S3 and SwiZ compaNbility built on top of librados.

Replicated Storage Pool: Ceph pool that establishes the number of object copies to be stored (default 3). Erasure Coded Storage Pool: Ceph pool where objects are divided into chunks to be stored. The chunks(n) are then wri+en to mulNple OSDs using K data chunks and M coded chunks (where n = k + m). For more details see the Ceph Concepts guide located at the main lab page.

Administra?on / Calamari -‐ mgmt 10.100.0.72/External IP

mon1 10.100.2.11

mon2 10.100.2.12

OSD node1 10.100.1.11

OSD node2 10.100.1.12

OSD node4 10.100.1.14

CLUSTER NETWORK

AWS resources launched as part of the Lab: Red Hat Ceph Storage – Fundamentals 1

client1 10.100.2.14

mon3 10.100.2.13

SSD

HDD

HDD

HDD

OSD node3 10.100.1.13

PUBLIC NETWORK

SSD

HDD

HDD

HDD

SSD

HDD

HDD

HDD

SSD

HDD

HDD

HDD

Ceph Diagram Here

OSD2 OSD3 OSD1 OSD5 OSD6 OSD4

Objects in Pools

Placement Groups

Ceph Nodes: OSD Hosts Monitors (mons)

Obj Obj

Obj Obj

Obj

Obj

Obj Obj

Obj Obj

Obj Obj

Obj

Obj

Obj Obj

RADOSGW RBD

Client Interface Layer

CRUSH Ruleset

PG PG PG PG PG PG PG PG PG PG PG PG




Pool ID. (Hash(Object) %Num of PGs)

Pool ID. (Hash(Object) %Num of PGs)

CRUSH Map

Mon1 Mon2 Mon3

RADOS LIBRADOS

Logical view of Red Hat Ceph Storage cluster components

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 !::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 !# !10.100.0.72 mgmt.ec2.internal mgmt ip-10-100-0-72.ec2.internal ip-10-100-0-72 !10.100.0.80 gateway.ec2.internal gateway ip-10-100-0-80.ec2.internal ip-10-100-0-80!# !10.100.1.11 node1.ec2.internal node1 ip-10-100-1-11.ec2.internal ip-10-100-1-11 !10.100.1.12 node2.ec2.internal node2 ip-10-100-1-12.ec2.internal ip-10-100-1-12 !10.100.1.13 node3.ec2.internal node3 ip-10-100-1-13.ec2.internal ip-10-100-1-13 !10.100.1.14 node4.ec2.internal node4 ip-10-100-1-14.ec2.internal ip-10-100-1-14 !# !10.100.2.11 mon1.ec2.internal mon1 ip-10-100-2-11.ec2.internal ip-10-100-2-11 !10.100.2.12 mon2.ec2.internal mon2 ip-10-100-2-12.ec2.internal ip-10-100-2-12 !10.100.2.13 mon3.ec2.internal mon3 ip-10-100-2-13.ec2.internal ip-10-100-2-13 !# !10.100.2.14 client1.ec2.internal client1 ip-10-100-2-14.ec2.internal ip-10-100-2-14 !

Lab Setup This is the reference host file for this lab

A note about networking This lab uses a very simple networking schema to be+er facilitate the learning acNviNes. Your corporate networking setup will likely differ. Make sure the Admin node has connecNvity to the server nodes (Mons, OSD’s).

Lab Setup First we need to InstanNate the lab environment. Use your SSH client of choice to access the management node via the public ip address (found at the bo+om of the ADDL. INFO tab on the launch page): ssh ceph@ManagementNodePublicIP // password = Redhat12 !Example: ssh [email protected] ! You can login to each individual node using: User – ceph (or root) !Password – Redhat12! Note: If your SSH connecNon fails, do the following -‐ Select the CONNECTION tab from the main lab page -‐ Select the Download PEM/PPK bu+on -‐ Select Download PEM (note the download locaNon of the PEM file) -‐ Change to the locaNon of the downloaded PEM file and execute the following:

ssh -‐i name-‐of-‐file.pem ceph@ip-‐address-‐of-‐ManagementNodePublicIP Example: ssh -i qwikLABS-L12-1652.pem [email protected] (your keypair and IP will differ)!

Add OSDs Login to the mgmt node as user ceph with password Redhat12 and change directory to ceph-‐config. Execute the mgmt1.sh script under /home/ceph/scripts directory to add eight OSDs to the Ceph cluster. You will be adding the disks xvdg and xvdh for the four nodes. ![ceph@mgmt ~]$ cd /home/ceph/ceph-config![ceph@mgmt ceph-config]$ /home/ceph/scripts/mgmt1.sh ! Bonus: Add “xvdi” to all 4 nodes to increase the number of OSDs in the cluster to twelve.

Virtual disk(s) on EC2

Verify Lab

Username – root Password – Redhat12

Using a browser go to the URL of the management node. h+p://ManagementNodePublicIP

Verify Lab - Add Hosts to Calamari

Select the Add bu+on (highlighted) to establish the Ceph cluster and setup the Monitors

Once the process completes, return to the command prompt and enter: [ceph@mgmt ceph-config]$ sudo salt '*' state.highstate !

Type the following administraNon commands to check the status of the cluster, lisNng of the OSDs & pools and the status of quorum among the monitors. [ceph@mgmt ~]$ ceph health !Three possible values: HEALTH_OK, HEALTH_WARN, and HEALTH_ERR [ceph@mgmt ~]$ ceph –w A more concise view of what is occurring in the cluster (use ctrl-‐c to end) [ceph@mgmt ~]$ ceph osd tree !A tree lisNng of the OSD’s and their status [ceph@mgmt ~]$ ceph osd lspools !Displays the pools configured in the cluster [ceph@mgmt ~]$ ceph quorum_status --format json-pretty !Displays the quorum status of the Monitors Note: Try the previous “quorum_status” command without “json-pretty” formanng and compare the two [ceph@mgmt ~]$ ceph quorum_status !

Ceph Management/Monitoring – CLI

[mgmt] [gateway] [client 1]

mgmt1.sh gateway1.-‐h+p.sh

Or gateway1.-‐h+p.sh

mgmt2.sh

mgmt3.sh

gateway2.-‐h+p.sh

gateway3.-‐h+p.sh

gateway4.-‐h+p.sh

gateway5.-‐h+p.sh

client1.sh

boto-‐scripts/*

Sequence of execuNon

Servers

Flow Diagram for configuring a RadosGW

Install and configure the Ceph Object Gateway. SSH to the gateway node as ceph user. You can SSH using the public IP address for the Gateway node or SSH from the mgmt node. [ceph@mgmt ~]$ ssh ceph@gateway!Enter password – Redhat12 Execute the gateway1-‐h]p.sh script under /home/ceph/scripts directory. This script will install Apache, copy the configuraNon file and start the service for non-‐secure access. The Rados Gateway packages are also installed by the script. Answer y when prompted when prompted by YUM. Follow instrucNons on next slide if you want to enable secure access of HTTP using SSL. [ceph@gateway ~]$ /home/ceph/scripts/gateway1-http.sh ! !

Configure Ceph Object Gateway (RADOSGW)

Skip this slide if you do not require secure access of HTTP using SSL. SSH to the gateway node as ceph user. You can SSH using the public IP address for the Gateway node or SSH from the mgmt node. [ceph@mgmt ~]$ ssh ceph@gateway!Enter password – Redhat12 Execute the gateway1-‐h]ps.sh script under /home/ceph/scripts directory. This script will install Apache, install SSL packages, copy the configuraNon file, generate x509 cerNficate and start the service for secure access. The Rados Gateway packages are also installed by the script. Answer y when prompted when prompted by YUM. [ceph@gateway ~]$ /home/ceph/scripts/gateway1-https.sh ! !

Configure Ceph Object Gateway (RADOSGW) contd.

Return to the mgmt node from the gateway node and make sure you are logged in as ceph user on the mgmt node. [root@gateway ~]# exit !Execute the mgmt2.sh script under /home/ceph/scripts directory. [ceph@mgmt ceph-config]$ /home/ceph/scripts/mgmt2.sh ! The script uses the mgmt node to create a keyring. Then it creates a client user name and key which is then added to the Ceph storage cluster. Finally, the key ring is distributed to the gateway node. !!!!!!!!!!!!!!!!!!


On the mgmt node as ceph user execute the mgmt3.sh script under /home/ceph/scripts directory. [ceph@mgmt ceph-config]$ /home/ceph/scripts/mgmt3.sh ! The script adds the Ceph Object Gateway configuraNon to the Ceph configuraNon file (ceph.conf) on the mgmt node. The configuraNon informaNon includes hostname of the gateway server, a keyring, socket path for FastCGI and a log file. Pull the updated ceph.conf to the ceph-‐config directory on the mgmt node and then push out the ceph.conf file to all the nodes in the cluster including the gateway node. !!!!!!!!!!!!!!!!!


Log in to the gateway node as user ceph. On the gateway node as ceph user execute the gateway2.sh script under /home/ceph/scripts directory. [ceph@gateway ~]$ /home/ceph/scripts/gateway2.sh ! The script copies a pre-‐built CGI wrapper script that provides the interface between the webserver and the radosgw process. The CGI wrapper script should be in a web accessible locaNon and should be executable. The helper script creates the data directory for the radosgw daemon on the gateway node, sets the correct directory permission, log file ownership and starts the radosgw service. !!!!!!!!!!!!!!!!!


On the gateway node as ceph user execute the gateway3.sh script under /home/ceph/scripts directory. [ceph@gateway ~]$ /home/ceph/scripts/gateway3.sh ! The script copies a pre-‐built Ceph Gateway configuraNon file rgw.conf that is needed for the radosgw service to the /etc/h]pd/conf.d directory. The script then restarts the h+pd service to accept the new configuraNon. !!!!!!!!!!!!!!!!!


On the gateway node as ceph user execute the gateway4.sh script under /home/ceph/scripts directory. [ceph@gateway ~]$ /home/ceph/scripts/gateway4.sh ! The script creates a Ceph Object Gateway user for the S3 interface. Note down the access_key and secret_key values from the output as they will be used by the Python example code for access validaNon. !!!!!!!!!!!!!!!!!


Please log into the client1 node using login ceph and password Redhat12 in order to test the Rados Gateway access using the newly created S3 user. On client1 node, execute the script client1.sh script under /home/ceph/scripts directory. [ceph@client1 ~]$ /home/ceph/scripts/client1.sh !!The script installs Boto which is a Python package that provides interfaces to Amazon Web Services. The script also install all the packages needed for the SwiZ access. !!!!!!!!!!!!!!On client1 node, under boto-‐scripts directory, you will find sample Python test scripts for verifying S3 access to the Rados Gateway. The test scripts will connect to the radosgw service, create a new bucket, create an object within the bucket and list all buckets and the objects it contains. !


In order to test SwiZ access to the Rados Gateway, we need to log back into the gateway node as ceph user. On gateway node, execute the script gateway5.sh script under /home/ceph/scripts directory. [ceph@gateway ~]$ /home/ceph/scripts/gateway5.sh !!The script creates a SwiZ subuser in a two step process. The first step is to create the user, and the second step is to create the secret key. Note the secret_key corresponding to testuser:swi` in the output from the second key generaNon step. The secret key will be used to get access to radosgw. !!!!!!!!!! You can now log into the client1 node to test the SwiZ access using the following command: ![ceph@client1 ~]$ swift -A http://10.100.0.80/auth/v1.0 ! -U testuser:swift -K '<secret_key>' list !!


Replicated Storage Pool - CLI

Create Replicated Storage Pool – CLI CreaNng a Ceph storage pool requires a pool name, PG/PGP values, and a pool type; replicated or erasure coded (replicated is the default type). We will create a replicated pool rep-‐pool-‐cli with 64 PG/PGP and 4 replicas. First here is a view ways to view the exisNng pools [ceph@mgmt ~]$ ceph osd lspools ![ceph@mgmt ~]$ rados lspools ![ceph@mgmt ~]$ ceph osd dump | grep -i pool ! Create replicated pool [ceph@mgmt ~]$ ceph osd pool create rep-pool-cli 64 64 !Again run ceph osd dump | grep -i pool and locate the rep-pool-cli pool. How many replicas did the rep-pool-cli pool default too? !Modify exisNng replica count to 4 [ceph@mgmt ~]$ ceph osd pool set rep-pool-cli size 4 !Verify the replicated size is now 4 [ceph@mgmt ~]$ ceph osd dump | grep -i pool !

Replicated Storage Pool - CLI

Create Objects and take snapshot of the newly created pool Create an object rep-‐obj in rep-‐pool-‐cli [ceph@mgmt ~]$ rados -p rep-pool-cli put rep-obj /etc/hosts ![ceph@mgmt ~]$ rados lspools !Verify the object in the storage cluster [ceph@mgmt ~]$ rados -p rep-pool-cli ls !Snapshot the pool and verify [ceph@mgmt ~]$ rados mksnap snapshot01 -p rep-pool-cli ![ceph@mgmt ~]$ rados -p rep-pool-cli ls !Remove the object [ceph@mgmt ~]$ rados -p rep-pool-cli rm rep-obj ![ceph@mgmt ~]$ rados -p rep-pool-cli ls !Rollback the pool to the snapshot [ceph@mgmt ~]$ rados -p rep-pool-cli listsnaps rep-obj ![ceph@mgmt ~]$ rados rollback -p rep-pool-cli rep-obj snapshot01 !Verify the object is once again available [ceph@mgmt ~]$ rados -p rep-pool-cli ls !

Replicated Storage Pool - Calamari

Now lets create a Replicated Storage Pool using the Calamari management interface Bring up Calamari and navigate to Manage > Pools. Select the Add bu+on (highlighted)

Replicated Storage Pool - Calamari

Create Replicated Storage Pool – Calamari Add pool NAME, REPLICAS, PGs, CRUSH RULESET. Press the Add bu+on to commit

Erasure Coded Storage Pool

Create Erasure Coded Storage Pool CreaNng a Ceph storage pool requires a pool name, PG/PGP numbers, and a pool type; replicated or erasure code (replicated is the default type). We will create an Erasure Coded pool named ec-‐pool with senngs k=3,m=2 (5 different OSDs will be used to store data and coded objects). First view the exisNng pools [ceph@mgmt ~]$ ceph osd lspools ![ceph@mgmt ~]$ rados lspools ![ceph@mgmt ~]$ ceph osd dump | grep -i pool !


Erasure Code is defined by a profile. The profile dictates the number of data chunks (represented as k) and the number of coding chunks (represented by m), along with the erasure code plugin in use and coding technique used. Default Profile: ceph osd erasure-‐code-‐profile get default directory=/usr/lib64/ceph/erasure-‐code k=2 m=1 plugin=jerasure technique=reed_sol_van View exisNng erasure code profile [ceph@mgmt ~]$ ceph osd erasure-code-profile get default !Create a new erasure code profile [ceph@mgmt ~]$ ceph osd erasure-code-profile set ec-profile ruleset-failure-domain=osd k=3 m=2 !Verify the new profile [ceph@mgmt ~]$ ceph osd erasure-code-profile ls!View new profile contents [ceph@mgmt ~]$ ceph osd erasure-code-profile get ec-profile !


Create Erasure Coded Storage Pool -‐ CLI Create a Ceph erasure coded pool using the profile we just created [ceph@mgmt ~]$ ceph osd pool create ec-pool 64 64 erasure ec-profile !Verify status/details of the newly created pool [ceph@mgmt ~]$ ceph osd dump | grep -i pool ! For extra credit: Add data to and take a snapshot of the new erasure coded pool (hint: use the same procedures described in the Replicated Pool – CLI secNon.

End Lab If you have completed all the lab exercises and wish to end your session, please select the End bu+on from the launch page. Thank You!

Ceph TestDrive Module3 v16

Documents

Transcript of Ceph TestDrive Module3 v16