Post on 15-Jan-2015
description
Dean Samuels
Technical lessons on how to doBackup and Disaster Recovery in the Cloud
Solution Architect, Amazon Web Services
Agenda
• Prologue The need for Backup and DR
• Customer Story Ausenco Limited
• Lessons Backup and DR with AWS
• Conclusion Q&A
The need for Backup & DR
• Archiving is the process of moving data that is no longer actively used to a separate data storage device for long-term retention. Data archives are indexed and have search capabilities so that files and parts of files can be easily located and retrieved.
• A backup or the process of backing up is making copies of data which may be used to restore the original after a data loss event. The primary purpose is to recover data after its loss, be it by data deletion or corruption. The secondary purpose of backups is to recover data from an earlier time.
• Disaster recovery (DR) is the process, policies and procedures related to preparing for recovery or continuation of technology infrastructure critical to an organization after a natural or human-induced disaster.
Some Definitions
Where is this…. Brisbane, QLD
Source: http://www.abc.net.au/news/specials/qld-floods/
Where is this…. Brisbane, QLD - January 13th 2011
Ausenco
Source: http://www.abc.net.au/news/specials/qld-floods/
Milton, QLD (Lang Park)
Source: http://www.abc.net.au/news/specials/qld-floods/
Milton, QLD (Lang Park) – January 13th 2011
Source: http://www.abc.net.au/news/specials/qld-floods/
Coronation Drive, Milton, QLD
Source: http://www.abc.net.au/news/specials/qld-floods/
Coronation Drive, Milton, QLD – January 13th 2011
Source: http://www.abc.net.au/news/specials/qld-floods/
Backup, Disaster Recovery and Data Tiering with AWS
Mike Latsky
Director, IT Operations
Ausenco
About Ausenco
We provide:Engineering, Procurement, Construction Management andOperational Servicesto the energy, resource, and process industries
Presentation title | DD Month Year | 12
IT Perspective• Engineers generate large 2D and 3D
CAD files
• CAD performance demands close network proximity
• Drawings are Ausenco’s deliverable and MUST be protected
• ASX200 company
• Head office in Brisbane
• 29 Offices in 19 countries
• 3500 Employees
Ausenco Offices
Our IT Environment
• Completely virtualised
• Windows
• ERP - Oracle SaaS
• Email, Lync, SharePoint – MS Office 365 SaaS
• Onsite storage
• 2 x primary data centres
• 1 x secondary data centre
Our Challenge
• Backup – Unreliable
• Disaster Recovery– Sites were exposed
• Local Disk Storage– At capacity
Possible Solutions
An Enterprise Backup Solution?
Secondary Data Centres?
More Disk Storage?
Our initial approach
• Large Integrators
• Leading backup providers
• Leading archiving providers
• Hardware providers
• Data centre hosting providers
• Leading cloud providers
Paradigm Shift
• Why use traditional technologies?
• How can we better leverage the AWS cloud?
• How do we do more with less?
• Address the complete data lifecycle
• Flexible, Scalable, Cost Effective
Disaster Recovery Archiving Backup
Solution
The Result
• Low effort• Incremental rollout• Cloud migration
• Efficient, effective usage of bandwidth• Application/database servers• Regionalised deployment
Our Roadmap
• Wider deployment
• Decommission costly secondary data centre
• Migration of production systems to AWS
Thank-you
Backup and DR with AWS
Backup & DR Lessons
Business and Technical Drivers….
Reduce costs
Slash DR budgets by up to 50%
Consolidate sites
Eliminate the need to run a secondary site
Reduce on-premise
Eliminate 30%+ of on-premise physical equipment
Remove aging technologies
Eliminate tape for backup and archive
The fundamental economic model…
Utility, on-demand datacenter
Primary SiteRouters
Firewalls
Network
Application Licenses
Operating Systems
Hypervisor
Servers
SAN
Primary Storage
Backup
Archive
AWSRouters
Firewalls
Network
Application Licenses
Operating Systems
Hypervisor
Servers
SAN
Snapshot Storage
Backup
Archive
Secondary site costs
Backup and DR with AWS
Backup Lessons
Backup Lessons – My backup should be accessible
Source: http://www.abc.net.au/news/specials/qld-floods/
a.k.a. the pain of physical data transfer
AWS Direct ConnectDedicated bandwidth between
your site and AWS
Amazon Storage GatewayShrink-wrapped gateway for volume
synchronization
AWS Import/ExportPhysical transfer of media into and
out of AWS
Getting data into the cloud
Simple Storage ServiceHighly scalable object storage
1 byte to 5TB in size
99.999999999% durability
Elastic Block StoreHigh performance block storage device
1GB to 1TB in size
Mount as drives to instances with
snapshot/cloning functionalities
GlacierLong term object archive
Extremely low cost per gigabyte
99.999999999% durability
Storage Options
Very fast ‘instance’ disks
Slow, rare accessFast web object storage
Cost ($/GB/month) Performance Durability
Amazon S3 0.095
Amazon Glacier 0.01
AWS Storage Gateway 0.095(+ 125/GW)
Amazon EBS 0.10
Amazon EBS (PIOPS) 0.125
*AWS Oregon pricingStorage Options
Snapshot of local volumes
Restoration from snapshots
Backup Lessons – My backup should be accessible
VPN
Public / AWS Direct Connect
AWS Import/Export
z
Backup Lessons – My backup should be accessible
GW-Cached volumes
GW-stored volumes
“Cool” storage
“Cold”
Backup Lessons – My backup should be accessible
Backup Lessons – My backup should be accessible
• “Infinite” scale with Amazon S3 and Amazon Glacier
• Scale to multiple regions
• Seamless
• No need to provision
• Cost tiers (cheaper at scale)
Backup Lessons – My backup should be able to scale
• SSL Endpoints (Amazon S3 and Amazon Glacier)
• Signed API calls
• Store encrypted files
• Server-side encryption
• Multiple copies across different data centers
• Local/cloud with AWS Storage Gateway
Backup Lessons – My backup should be safe
• Easy to integrate within AWS or Hybrid
• AWS Storage Gateway: Run services on Amazon EC2 (DR)
• Clear costs
• Reduced costs
• I decide redundancy/availability in relation to costs
Backup Lessons – My backup should work with a DR policy
Backup Lessons – My backup should work with a DR policy
• Clear ownership
• Permissions with IAM: Users, groups roles
• Logs
• AWS support
Backup Lessons – Someone should care about it
1. My backup should be accessible
2. My backup should be able to scale
3. My backup should be safe
4. My backup should work with a DR policy
5. Someone should care about it
Backup Lessons
Backup and DR with AWS
DR Lessons
DR is part of a wider set of policies and controls…
DR & business continuity
It’s not an all or nothing thingChoose what needs to failover and what does not
Some things more important than othersSome things will still be working
High availability Backup Disaster recovery
Keep your applications
running 24x7
Make sure your data is safe Get your applications and
data back after a major
disaster
Each set of IT assets will have different requirements…
DR & business continuity
Recovery Time Objective (RTO)
How quickly you need this asset to be
recovered?
e.g. 1min? 15min? 1hr? 4hrs? 1day?
Recovery Point Objective (RPO)
How ‘fresh’ the recovery must be for the
asset?
e.g. zero data loss, 15mins out of date?
Assets will sit on a spectrum of technical complexity…
DR & business continuity
Rebuild when required from offsite backup
Run hot-hot configuration with
auto-failover
Custo
mer
facin
g tra
nsac
tiona
l
web ap
plica
tion
Inte
rnal
com
mun
icatio
ns
syste
m
Daily
sche
duled
proc
esse
s & sy
stem
s
Back
end
repo
rting
syste
m
& dat
abas
e
DR Lessons – You NEED a DR plan in place
DR with High Availability
App DR with Standby
DR Lessons – You NEED a DR plan in place
DR Lessons – Testing your DR
• Dev/test in the cloud is super easy
• Spin up capacity only for the test
• Regularly test your DR
• Cost is minimal
• What about data transfer speed?
s3cmd ls --recursive
s3://datasets.elasticmapreduce/ngra
ms/books/ | awk '{print $4;
sub(/s3:\/\/datasets.elasticmapredu
ce/, "/array", $4); print $4}' |
parallel -j0 -N2 --progress
/usr/bin/s3cmd --no-progress get
{1} {2}Copying 2.4 TB down from 48 hours to 9 hours (5x faster)
DR Lessons – Reducing Costs
• Dev/test in the cloud is super easy
• Spin up capacity only for the test
• Regularly test your DR
• Cost is minimal
• What about data transfer speed?
DR Lessons – You can have different DR solutions
• Easy to integrate existing vendors with DR on AWS
• Approach: One vendor/hybrid/multiple vendors
• One region/multi-regions (if you need geo-diversity)
• Different DR Architectures
Backup & Restore Pilot light
Warm standby in AWS
Multi-site solution in AWS & on-
premise
Pilot light
Stopped instances
Pilot light
Running instances
1. You NEED a DR plan in place
2. Testing your DR
3. Reducing costs
4. You can have different DR solutions
DR Lessons
aws.amazon.comget started on the free tier
Technical lessons on how to doBackup and Disaster Recovery in the Cloud