MySQLwithDRBD/Pacemaker/CorosynconLinux

DefinitionofDRBD:DRBD(DistributedReplicatedBlockDevice)

DRBDsynchronizesdataattheblockdevice(typicallyaspinningorsolidstatedisk)transparenttotheapplication,databaseandeventhefilesystem.DRBDrequirestheuseofajournalingfilesystemsuchasext3orext4.ForthissolutionitactsinanactivestandbymodethismeansthatatanypointintimethedirectoriesbeingmanagedbyDRBDareaccessibleforreadsandwritesonexactlyoneofthetwohostsandinaccessible(evenforreads)ontheother.AnychangesmadeontheactivehostaresynchronouslyreplicatedtothestandbyhostbyDRBD.

:TheConcept:

Theconceptofanactive/passivefailoverClusteristhefollowing:

Twoservers(nodes).

Theycommunicateoveraclustersoftware(Heartbeat ,Corosync ,Pacemaker)

TheyarerunningonDRBDfailoverstoragesystem.

MySQLisonlyrunninginMASTERnode(active),theotheristhePASIVEnode.

YoureachMySQLoveraVirtualIP(ClusterIP)

IncaseofaproblemtheclusterfailovertheresourcesincludingtheVIPtothepassivenode.

Thisfailoveristransparentfortheapplication(aliteSERVICEDOWN).

NetworkandServersettings:

NetworkstaticipsetonBothNode

Deactivateselinux(/etc/sysconfig/selinux)

Tosimplificationtheconfigurationusehostnames.(/etc/sysconfig/network)

Addthetwonodesinthe/etc/hostsfileonBothNode

likethis:

127.0.0.1localhost

192.168.0.251Cluster1

192.168.0.252Cluster2

PasswordlesscommunicationwithBothNode

InstallmysqlwiththehelpofYum(OnBothNodeformountingDRBDcreateddevice)

DRBDInstallation&ConfigurationStepbyStep:

SimpleDownloadelreporelease65.el6.elrepo.noarch.rpmforDRBDpackageinstallation.

RequiredPackages:

1. drbd83utils.x86_64

2. kmoddrbd83.x86_64

Note:Thispackageinstallationwiththehelpofyum

Copythedistconfiguration:

#cppr/etc/drbd.conf/etc/drbd.confDIST

The/etc/drbd.confistheconfigurationfilefordrbd,heremyconfiguration:

looklikethis:

#Youcanfindanexamplein/usr/share/doc/drbd.../drbd.conf.example

include"drbd.d/global_common.conf";

resourceDISK1{

protocolC;

net{

cramhmacalgsha1;

sharedsecret"8a6cxxxxxxxxxxxxxxxxxxxxx49xxxxxxxxfb3";

aftersb0pridiscardzerochanges;

aftersb1pridiscardsecondary;

aftersb2pridisconnect;

rrconflictdisconnect;

}

device/dev/drbd0;

disk/dev/sda5;

metadiskinternal;

onCluster1{

address192.168.0.251:7789;

}

onCluster2{

address192.168.0.252:7789;

}

}

AsyoucanseeIspecifythisparameters:

RESOURCE:Thenameoftheresource

PROTOCOL:InthiscaseCmeanssynchronous

NET:TheSHA1key,thathavethesameinthetwonodes

aftersb0pri:WhenaSplitBrainocurrs,andnodatahavechanged,thetwonodesconnectnormally.

aftersb1pri:Ifsomedatahavebeenchanged,discardthesecondarydataandsynchronizewiththeprimary

aftersb2pri:Ifthepreviousoptionisimpossibledisconnectthetwonodes,inthiscasemanuallySplitBrainsolutionisrequired

rrconflict:Incasethatthepreviousstatementsdontapplyandthedrbdsystemhavearoleconflict,thesystemdisconnectautomatically.

DEVICE:Virtualdevice,thepatchtothefisicaldevice.

DISK:Fisicaldevice

METADISK:Metadataarestoredinthesamedisk(sdc1)

ON:Thenodesthatformthecluster

Creatingtheresource

Thiscommandsinbothnodes

#fdisk/dev/sda

Createpartition

Createthepartitionwithoutformat

[root@node1~]#drbdadmcreatemdDISK1

Writingmetadata...

initializingactivitylog

NOTinitializingbitmap

Newdrbdmetadatablocksuccessfullycreated.

[root@node2~]#drbdadmcreatemdDISK1

Writingmetadata...

initializingactivitylog

NOTinitializingbitmap

Newdrbdmetadatablocksuccessfullycreated.

ActivatetheResource

Besurethatthedrbdmoduleisload(lsmod),ifnotloadit:

#modprobedrbd(OnBothNode)

NowactivatetheresourceDISK1:

[root@node1~]#drbdadmupDISK1

[root@node2~]#drbdadmupDISK1

Synchronize

Onlyinthemasternode,wellsaythatthenode1istheprimary:

#drbdadmoverwritedataofpeerprimaryDISK1

Wellseethatthediskssynchronizationareinprogress,adnthestateisUpToDate/Inconsistent

CheckSynchronizationstatus

#/etc/init.d/drbdstatus(OnBothnode)

FormattheResource

Onlyinthemasternode:

#mkfs.ext4/dev/drbd0

Testing

Mounttheresourceinthenode1

[root@node1~]#mount/dev/drbd0/var/lib/mysql

Ok,nowumountandmarkthenode1likesecondary

[root@node1~]#umount/var/lib/mysql

[node1~]#drbdadmsecondaryDISK1

MarkNode2likePrimaryandmount:

[root@node2~]#drbdadmprimaryDISK1

[root@node2~]#mount/dev/drbd0/var/lib/mysql

IntroductionofCorosync&PacemakerPacemakerandCorosynccombinetoprovidetheclusteringlayerthatsitsbetweentheservicesandtheunderlyinghostsandoperatingsystems.Pacemakerisresponsibleforstartingandstoppingservicesensuringthattheyrerunningonexactlyonehost,deliveringhighavailabilityandavoidingdatacorruption.CorosyncprovidestheunderlyingmessaginginfrastructurebetweenthenodesthatenablesPacemakertodoitsjob;italsohandlesthenodesmembershipwithintheclusterandinformsPacemakerofanychanges.

InstallationofCorosync&Pacemakerstepbystep

elreporelease65.el6.elrepo.noarch.rpm(.repofile)forcorosync&Pacemakerpackageinstallation

Packageinstallwiththehelpofyum

#yuminstallpacemaker.x86_64corosync.x86_64

configurecorosynctousecryptographictechniquestoensureauthenticityandprivacyofthemessages,youwillneedtogenerateaprivatekey.CorosyncClusterEngineAuthenticationkeygenerator.Gathering1024bitsforkeyfrom/dev/random.

CorosyncKeyInonenodecreatethecorosyncsecuritycomunicationkey.

Corosyncforthefirsttime,youneedtocreatetheauthkeyfileforauthenticationwithinclustercommunication.

[root@node1]#corosynckeygen

CorosyncClusterEngineAuthenticationkeygenerator.Gathering1024bitsforkeyfrom/dev/random.Presskeysonyourkeyboardtogenerateentropy.Writingcorosynckeyto/etc/corosync/authkey.

needtocopythatfiletoallofyournodesandputitin/etc/corosync/withuser=root,group=rootandmode0400.permissionsto400

[root@node1]#scp/etc/corosync/authkeynode2:/etc/corosync/

[root@node1]#ll/etc/corosync/authkeyr1rootroot128july1710:26/etc/corosync/authkey[root@node2]#ll/etc/corosync/authkeyr1rootroot128july1710:27/etc/corosync/authkey

Nowconfigurethe/etc/corosync/corosync.conf

Looklikethis:

[root@node1]#vi/etc/corosync/corosync.conf

#Pleasereadthecorosync.conf.5manualpage

compatibility:whitetank

totem{

version:2

secauth:off

interface{

member{

memberaddr:192.168.0.251

}

member{

memberaddr:192.168.0.252

}

ringnumber:0

mcastaddr:239.255.255.255

bindnetaddr:192.168.0.0

mcastport:5405

ttl:1

}

transport:udpu

}

logging{

fileline:off

to_logfile:yes

to_syslog:yes

debug:on

logfile:/var/log/cluster/corosync.log

debug:off

timestamp:on

logger_subsys{

subsys:AMF

debug:off

}

}

NowCopycorosync.confconfiguredfileonNode02

[root@node1]#scp/etc/corosync/corosync.confnode2:/etc/corosync/.

Corosynctoloadthequorumandmessaginginterfacesneededbypacemaker,create/etc/corosync/service.d/pcmkwiththefollowingfragment.

service{

#LoadthePacemakerClusterResourceManagername:pacemakerver:1}

NowcopyonNode02

[root@node1]#scp/etc/corosync/service.d/pcmknode2:/etc/corosync/service.d/.

crmcommand(clustermanagmentforpacemaker)notfoundinlatestCentos6

#yuminstallcrmsh.x86_64(OnBothNode)

NowcreateLogDirectoryandfileunder/var/log/(BothNode)

#mkdirp/var/log/cluster

#vi/var/log/cluster/corosync.log

Corosyncprocesswouldlaunchpacemaker,thisisnolongerthecase.PacemakermustbelaunchedafterCorosynchassuccessfullystarted.(BothNode)

#servicecorosyncstart

#servicepacemakerstart

#chkconfigcorosyncon

#chkconfigpacemakeron

crm_monisrunwiththe1optiontoindicatethatitshouldreportonceandthenreturn.Arecommendationwouldbetoalsorunitwithouttheoption(onbothservers)sothatyougetacontinuallyrefreshedviewofthestateoftheclusterincludinganymanagedresources.

CheckOnlineconfiguredClusternodewithcrmcommand(OnBothNode)

[root@node01]#crm_mon1============

Outputshowlooklikethis:

Lastupdated:MonFeb2717:51:102012

Lastchange:MonFeb2717:50:252012viacrmdonhost1.localdomain

Stack:openais

CurrentDC:host1.localdomainpartitionwithquorum

Version:1.1.63.el6a02c0f19a00c1eb2527ad38f146ebc0834814558

2Nodesconfigured,2expectedvotes

0Resourcesconfigured.

============

Online:[Node01Node02]

ConfiguringandManagingClusterResources(CommandLine)

Usethefollowingcommandstosettheoptionsforatwonodeclustersonly:

[root@node01~]#crmconfigurepropertynoquorumpolicy=ignore

Pacemakerusesresourcestickinessparameterstodeterminewhenresourcesshouldbemigratedbetweennodestheabsolutevaluesarenotimportant,ratherhowtheycomparewiththevaluesthatwillsubsequentlybeconfiguredagainstspecificevents;herewesetthestickinessto100:

[root@node01~]#crmconfigurersc_defaultsresourcestickiness=100

STONITH(ShootTheOtherNodeInTheHead)otherwiseknownasfencingreferstoonenodetryingtokillanotherintheeventhatitbelievestheotherhaspartiallyfailedandshouldbestoppedinordertoavoidanyriskofasplitbrainscenario.Weturnthisoffasthissolutionwillrelyoneachnodeshuttingitselfdownintheeventthatitlosesconnectivitywiththeindependenthost:

[root@node01~]#crmconfigurepropertystonithenabled=false

NowconfigureVirtualIPresourceforthisnodes.Nodeshavetocheckeachotherevery20seconds:

[root@node1]#crmconfigurecrm(live)configure#primitiveClusterIPocf:heartbeat:IPaddr2paramsip=192.168.1.100cidr_netmask=32

DRBD Nowthefilesystem,addDISK1tothecluster

crm(live)configure#primitivedrbd_mysqlocf:linbit:drbdparamsdrbd_resource="DISK1"opmonitorinterval="15s"opstarttimeout="240s"

Definethemountpoint

crm(live)configure#primitivefs_mysqlocf:heartbeat:Filesystemparamsdevice="/dev/drbd0"directory="/var/lib/mysql"fstype="ext4"

DefineonlyoneMasternodecrm(live)configure#msms_drbd_mysqldrbd_mysqlmetamastermax="1"masternodemax="1"clonemax="2"clonenodemax="1"notify="true"

MySQL

Nowthemysqlserver

crm(live)configure#primitivemysqldocf:heartbeat:mysqlparamsbinary="/usr/bin/mysqld_safe"config="/etc/my.cnf"user="mysql"group="mysql"log="/var/log/mysqld.log"pid="/var/run/mysqld/mysqld.pid"datadir="/var/lib/mysql"socket="/var/lib/mysql/mysql.sock"opmonitorinterval="60s"timeout="60s"opstartinterval="0"timeout="180"opstopinterval="0"timeout="240"

Groups&Colocations

Withthisgroupweensurethatthedrbd,mysqlandVIPareinthesamenode(master)andtheordertostopandstartiscorrectly:

start:fs_mysql>mysqld>ClusterIPstop:ClusterIP>mysqld>fs_mysql

crm(live)configure#groupgroup_mysqlfs_mysqlmysqldClusterIPmetamigrationthreshold="5"

Thegroupgroup_mysqlallwaysintheMASTERnode

crm(live)configure#colocationmysql_on_drbdinf:group_mysqlms_drbd_mysql:Master

MysqlstartallwaysafterdrbdMASTER

crm(live)configure#ordermysql_after_drbdinf:ms_drbd_mysql:promotegroup_mysql:start

Howtoconfigurepacemakerapachefailoversystemwithpacemaker

crm(live)configure#crmconfigureprimitiveAPACHEocf:heartbeat:apacheparamsconfigfile="/etc/httpd/conf/httpd.conf"statusurl="http://localhost/serverstatus"opmonitorinterval="40s"

###APACHEresourcename###configfilepathtoapacheconfigurationfile###statusurlurltostatuspage(belowhowtoconfiguresone)###intervaltimebetweenchecks

###InApacheConfigurationfile:

#vi/etc/httpd/conf/httpd.conf

ExtendedStatusOn

SetHandlerserverstatusOrderdeny,allowAllowfromall

Topreventsituationwhenresourceapachemigratetonode002andresourceIPstaysatnode001(Ithappenswhenapacheatnode001hungbutnetworkstackworkswell)weneedtomakecolocation

[root@node1]#crmconfigurecolocationWEB_SITEinf:APACHEClusterIP

TomakepacemakerstartupapacheonlyafterIPissetup.Inotherwordsdescribestartuporderrun:

[root@node1]#crmconfigureorderSTART_ORDERinf:ClusterIPAPACHE

NowcheckconfigurationonbothNode

#crm_mon1#crmconfigureshow#crm_mon

By:AlokRaj

SystemAdministrator

SaigunTechnologiesPvtLtd