Getting Started with MongoDB on Amazon Web Services
-
Upload
crcsmnky -
Category
Technology
-
view
9.638 -
download
6
Transcript of Getting Started with MongoDB on Amazon Web Services
Sandeep ParikhTechnical Product [email protected]
Getting Started with MongoDB and Amazon Web Services
What We’ll Cover
• MongoDB components » AWS instances• Deployment configurations• Storage configuration• Production tips• Setting up a three-node replica set
MongoDB Components
• 64-bit instances• Where your data lives• Needs RAM and Disk I/O
mongod
• Stateless request router• Lives on your app server• Minimal data needs
mongos
• Metadata server for sharded configurations• Minimal data needs• 64-bit instances
config
Amazon EC2 Components• mongod
– Standard: Large or Extra Large (typical)
– Hi-Memory: XL, XXL or 4XL (large data sets)
– Cluster instances provide increased capacity and bandwith
• mongos– Deploy onto your app server (32
or 64-bit)• config
– Standard instances– Micro instance is sufficient
DEPLOYMENT CONFIGURATIONS
Single Node
• mongod– 64-bit EC2 instance
• Storage– Multiple EBS volumes– RAID 10– Configured using mdadm
mongod
RAID 10
Replica Set
mongodprimary
RAID 10
mongodsecondary
RAID 10
mongodsecondary
RAID 10
Replica Set – Using Zones
mongodprimary
RAID 10
mongodsecondary
RAID 10
mongodsecondary
RAID 10
Zone 3Zone 2Zone 1
Replica Set – Multiple Regions
app
Region 1 Region 2
mongodprimary
RAID 10
mongodsecondary
RAID 10
mongodsecondary
RAID 10
Replica Set – Security Groups
mongodprimary
RAID 10
mongodsecondary
RAID 10
mongodsecondary
RAID 10
app “application”
“database”
Sharded Deployment
Shard 1 Shard 2 Shard 3
config
config
config
app server
mongos
app server
mongos
app server
mongos
CONFIGURING STORAGE
Recommendations
• EBS-backed storage vs. instance-based– Persistent vs. ephemeral
• RAID 10: “striped mirrors”
• 4-8 EBS volumes for best performance
http://en.wikipedia.org/wiki/Nested_RAID_levels#RAID_1_.2B_0
DEPLOYMENT NOTES
Some Tips
• Know your deployment– Security– Instances– Storage
• Configure filesystem as Ext4 or XFS• Reduce I/O overhead– noatime, nodiratime
• Raise file descriptor limits
More Tips
• EBS snapshots are an easy way to back up data
• Deploy in a trusted environment– Consider authentication
REPLICA SET DEPLOYMENT STEPS
Security Groups
• Create groups for the following– app servers– MongoDB components
• Ports to remember– mongod: 27017, 28017 (web-based status)– mongod shard server: 27018 – mongos: 27017– config: 27019
• Security groups act as ACLs• Instances are accessible via SSH keys
Single Node
• We’ll use this as a starting point
• Single EC2 instance• 4 EBS volumes• RAID10
– Example:• 100 GiB total• 50 GiB usable
mongod
RAID 10
Creating the Components
$ ec2-run-instances ami-41814f28 -n 1 -g database -k cluster-keypair -t m1.large -z us-east-1a
AMI
count
group
keypair
size
zone
Create an instance
Create storage volumes (4x)$ ec2-create-volume –s 25 -z us-east-1a
size
zone
Storage Configuration
$ ec2-attach-volume vol-e796108a -i i-11eee072 -d /dev/sdh1
volume
instance
device
Attach storage (4x)
Start the RAID$ sudo mdadm --create -l10 -n4 /dev/md0 /dev/sdh*
type
new device
devices
Storage Configuration
$ sudo fdisk /dev/md0$ sudo mkfs.ext4 /dev/md0p1
Partition device and make filesystem
$ sudo mkdir /data$ sudo chown `id -u` /data
Create mount point and set ownership
$ sudo echo ‘/dev/md0p1 /data auto noatime,noexec,nodiratime 0 0’ >> /etc/fstab$ sudo mount –a /dev/md0p1 /data
Update the filesystem table and mount
Install MongoDB
$ echo "[10gen]name=10gen Repositorybaseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64gpgcheck=0" | sudo tee -a /etc/yum.repos.d/10gen.repo
Update local repo settings
$ sudo yum -y install mongo-10gen-server$ sudo yum -y install sysstat
Install MongoDB and tools
$ sudo chkconfig --level 3 mongod off$ sudo chkconfig --level 5 mongod off
Change auto-start settings
MongoDB Configuration
$ sudo nano /etc/mongod.conf...dbpath=/data...
Set the dbpath
$ sudo chkconfig mongod on$ sudo /etc/init.d/mongod start
Set to autostart at boot and start the server now
$ mongoMongoDB shell version: 2.0.2connecting to: test>
Connect to MongoDB
Adding Additional Nodes
• For each node, first repeat steps for– Creating instances and
volumes– Configuring storage
• Include replica set parameter in MongoDB configuration
• Start mongod
MongoDB Configuration
$ sudo nano /etc/mongod.conf...dbpath=/datareplSet=replicaSetName...
Set the dbpath
$ sudo chkconfig mongod on$ sudo /etc/init.d/mongod start
Set to autostart at boot and start the server now
replica set name
Initialize Replica Set
$ mongoMongoDB shell version: 2.0.2connecting to: test>
Connect to MongoDB
> rs.initiate(){ "info2" : "no configuration explicitly specified -- making one", "me" : "ip-10-127-127-91:27017", "info" : "Config now saved locally. Should come online in about a minute.", "ok" : 1}
Initialize the replica set
> rs.add(“ec2-abc.def.amazonaws.com”){ “ok” : 1 }
Add each replica set member
external DNS name
Initialize Replica Set
>
The mongo prompt should go from this
PRIMARY>
To this
SECONDARY>
Or this
BACKUP/RESTORE
Backing Up Your Data
$ mongoSECONDARY> use adminSECONDARY> db.runCommand({fsync:1,lock:1});{ "info" : "now locked against writes, use db.$cmd.sys.unlock.findOne() to unlock", "ok" : 1}
Lock the database
$ ec2-create-snapshot vol-1234abcd --description "MongoDB RAID backup1"
volume description
Create a snapshot for each attached volume
$ mdadm --detail /dev/md0p1
Run mdadm and note the UUID
Backing Up Your Data
$ mongoSECONDARY> db.$cmd.sys.unlock.findOne();{ "ok" : 1, "info" : "unlock requested" }
Unlock the database
Restoring Your Data
$ ec2-create-volume --availability-zone us-east-1a --snapshot vol-1234abcd$ ec2-attach-volume vol-1234abcd -i i-aa3bc4c8 -d /dev/sdh1
Create and attach a volume from each snapshot
$ sudo mdadm --assemble --auto-update-homehost -u 07552c4d…a9c2f2fc --no-degraded /dev/md0mdadm: /dev/md0 has been started with 4 drives.
Reassemble the RAID
$ sudo mkdir /data2$ sudo chown `id -u` /data2
Create mount point and set ownership
UUID from mdadm
Restoring Your DataUpdate the filesystem table and mount
$ sudo echo ‘/dev/md0p1 /data2 auto noatime,noexec,nodiratime 0 0’ >> /etc/fstab$ sudo mount –a /dev/md0p1 /data2
$ mongod --dbpath /data2$ mongoMongoDB shell version: 2.0.2connecting to: test> use yourDb> db.yourCollection.validate({full:true})
Verify the restore (optional, requires time and disk space)
$ sudo cp /data2 /data
Set /data2 to /data
Recommended Backup Methods
• Create backups on secondary servers when using replication– Doesn’t block on the primary– Secondary updates once unlocked
• Use mongodump to backup live data– Collections, databases, everything– Restore with mongorestore
QUESTIONS?
Learn More
• MongoDB on Amazon Web Services white paper: http://www.10gen.com/white-papers
• Amazon EC2 info at MongoDB wiki: http://www.mongodb.org/display/DOCS/Amazon+EC2
• Scripts for working with EBS (creating, attaching, etc.): https://gist.github.com/1482182
Contact Info
• MongoDB– http://www.mongodb.org– Downloads, Docs, Forums, etc.
• 10gen– http://www.10gen.com/contact– [email protected]– Consulting, Support, etc.
• Sandeep Parikh– [email protected]