Redis High availability and fault tolerance in a multitenant environment

REDIS IN A MULTI-TENANT ENVIRONMENT Iccha Sethi, Software Developer

Twitter: @IcchaSethi

2

IcchaSethi.__dict__

{ “Name”: “Iccha Sethi”, “Occupation”: “Software Developer”, “Team”: “Redis”, “Company”: “Object Rocket, Rackspace”, “Fun Fact”: “loading….” }

Why does High Availability matter for Redis?

3

Why does High Availability matter for Redis?

4

Outline

• Architecture • Security and Isolation • Know thy environment

5

Outline


6

7

Architecture

Outline

8

Redis Master

Access HA Redis: Dns: read/write port 6379 password

Outline

9

Redis Master

Redis Slave

Access HA Redis: DNS: read/write port 6379 password

Outline

10

VIP

LB1 (Master) HAProxy

Redis Master

Redis Slave


Outline

11

VIP


LB2 (Backup) HAProxy

keepalived

Redis Master

Redis Slave


12

Sentinels

13

Highly Available REDIS

STASH NOVA

Proxy API

Sentinels

Host 0

Host 1

Host 2

Host 3

Redis Master

Redis Slave

KeepAliveD HA Proxy

KeepAliveD HA Proxy

Proxy Agent

Proxy Agent

Agent

Agent

Customer :6379

HA Create

14

Architecture - Opportunities

Opportunities

• Failover is easy • Easy to introduce changes • Reusable architecture

15

Opportunities


16

17

Fail Over – Redis Nodes

STASH NOVA

Proxy API

Sentinels

Host 0

Host 1

Host 2

Host 3

Redis Master

Redis Slave

KeepAliveD HA Proxy

KeepAliveD HA Proxy

Proxy Agent

Proxy Agent

Agent

Agent

Customer :6379

18


STASH NOVA

Proxy API

Sentinels

Host 0

Host 1

Host 2

Host 3

Redis Master

Redis Slave

KeepAliveD HA Proxy

KeepAliveD HA Proxy

Proxy Agent

Proxy Agent

Agent

Agent

Customer :6379

19


STASH NOVA

Proxy API

Sentinels

Host 0

Host 1

Host 2

Host 3

Redis Master

Redis Master

KeepAliveD HA Proxy

KeepAliveD HA Proxy

Proxy Agent

Proxy Agent

Agent

Agent

Customer :6379

20


STASH NOVA

Proxy API

Sentinels

Host 0

Host 1

Host 2

Host 3

Redis Master

Redis Master

KeepAliveD HA Proxy

KeepAliveD HA Proxy

Proxy Agent

Proxy Agent

Agent

Agent

Customer :6379

21


STASH NOVA

Proxy API

Sentinels

Host 0

Host 1

Host 2

Host 3

Redis Master

Redis Master

KeepAliveD HA Proxy

KeepAliveD HA Proxy

Proxy Agent

Proxy Agent

Agent

Agent

Customer :6379

22


STASH NOVA

Proxy API

Sentinels

Host 0

Host 1

Host 2

Host 3

Redis Master

Redis Master

KeepAliveD HA Proxy

KeepAliveD HA Proxy

Proxy Agent

Proxy Agent

Agent

Agent

Customer :6379

23


STASH NOVA

Proxy API

Sentinels

Host 0

Host 1

Host 2

Host 3

Redis Master

Redis Master

KeepAliveD HA Proxy

KeepAliveD HA Proxy

Proxy Agent

Proxy Agent

Agent

Agent

Customer :6379

Opportunities


24

25

Redis Resize – Vertical Scaling

STASH

Sentinels

Redis Master

Redis Slave

HA Proxy

HA Proxy

Customer :6379

26


STASH

Sentinels

Redis Master

Redis Slave

HA Proxy

HA Proxy

Customer :6379

New Redis Master

New Redis Slave

27


STASH

Sentinels

Redis Master

Redis Slave

HA Proxy

HA Proxy

Customer :6379

New Redis Master

New Redis Slave

28


STASH

Sentinels

HA Proxy

HA Proxy

Customer :6379

New Redis Master

New Redis Slave

Opportunities


29

Highly Available MySQL

30

31

Architecture - Challenges

Challenges

• Global sentinels • Redis drivers

32

Global Sentinels

33

34

Sentinels

Redis Drivers/Clients

35

36

Architecture - Case Study

Tale of ‘Resize Why you no work?’

37

38


STASH

Sentinels

Redis Master

Redis Slave

HA Proxy

HA Proxy

Customer :6379

New Redis Master

New Redis Slave

Outline

• Architecture • Security and Isolation • Know your environment

39

40

Security and Isolation

ACLs

41

42

VIP


ACLS


ACLs

keepalived

Redis Master

Redis Slave

Access HA Redis: VIP: read/write port 6379

ACLs

SSL

43

44

VIP


ACLs SSL certs


ACLs SSL certs

keepalived

Redis Master

Redis Slave

Access HA Redis: VIP: read/write port 6379 SSL port 6380

SSL

Containers

45

46

VIP



keepalived

Redis Master

Redis Slave


Containers

47

Security and Isolation – Case Study

48

VIP



keepalived

Redis Master

Redis Slave


Tale of the Customer with 120K connections

Outline


49

Monitoring, Monitoring, Monitoring

• What do you monitor? • How often is it monitored? • Who does the monitoring? • What action is taken?

51


• What do you monitor? –  Front door check –  Connection limit –  Memory usage –  Failover –  Failed failover –  Redis process down –  No valid slaves

• How often is it monitored? • Who does the monitoring? • What action is taken?

52


• What do you monitor? • How often is it monitored?

–  Front door check –  Memory usage –  Failover –  Failed failover –  Redis process down –  No valid slaves

• Who does the monitoring? • What action is taken?

53


• What do you monitor? • How often is it monitored? • Who does the monitoring? • What action is taken?

54


• What do you monitoring? • How often is it monitored? • Who does the monitoring? • What action is taken?

57


• What do you monitoring? • How often is it monitored? • Who does the monitoring? • What action is taken?

–  Front door check –  Memory usage –  Failover –  Failed failover –  Redis process down –  No valid slaves

59

60

Know Thy Environment - Challenges

62

Know Thy Environment – Case Studies

Tale of the customer saved by Zendesk

63

Outline


64

Redis High availability and fault tolerance in a multitenant environment

Technology

Transcript of Redis High availability and fault tolerance in a multitenant environment