MongoDB at Gilt Groupe
Transcript of MongoDB at Gilt Groupe
![Page 1: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/1.jpg)
February 2013Sean Sullivan
@
![Page 2: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/2.jpg)
• software engineer
• ~2 years at Gilt
• work in Gilt’s Portland office
• back office applications
About me
![Page 3: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/3.jpg)
• Gilt Groupe
• Gilt’s technology stack
• MongoDB at Gilt
• Q&A
Agenda
![Page 4: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/4.jpg)
Gilt Groupehttp://www.gilt.com
![Page 5: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/5.jpg)
flash sales
![Page 6: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/6.jpg)
Everyday at 12 noon
![Page 7: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/7.jpg)
what does Gilt sell?
![Page 8: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/8.jpg)
Apparel
![Page 9: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/9.jpg)
![Page 10: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/10.jpg)
![Page 11: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/11.jpg)
Kids merchandise
![Page 12: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/12.jpg)
![Page 13: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/13.jpg)
Home furnishings
![Page 14: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/14.jpg)
![Page 15: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/15.jpg)
Food
![Page 16: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/16.jpg)
![Page 17: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/17.jpg)
Local deals
![Page 18: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/18.jpg)
![Page 19: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/19.jpg)
Travel
![Page 20: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/20.jpg)
![Page 21: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/21.jpg)
![Page 22: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/22.jpg)
Gilt technology stack
![Page 23: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/23.jpg)
![Page 24: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/24.jpg)
Gilt architecture
![Page 25: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/25.jpg)
monolithicapplication
service-oriented architecture
2007 2013
![Page 26: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/26.jpg)
2007
![Page 27: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/27.jpg)
2013
service A
service B
service C
service D
service E
legacyweb app
gilt.com
![Page 28: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/28.jpg)
Data storage @ Gilt
![Page 29: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/29.jpg)
@
![Page 30: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/30.jpg)
Why MongoDB?
• Ease of use
• Horizontal scaling
• High availability
• Automatic failover
![Page 31: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/31.jpg)
Why MongoDB?
• Stability
• Support
• Drivers
![Page 32: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/32.jpg)
• MongoDB 2.0
• sharded and non-sharded data
• Solid State Drives
• MMS for monitoring
![Page 33: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/33.jpg)
Use case #1: user profiles
![Page 34: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/34.jpg)
• user data in Postgres
• legacy Rails app expects to find user data in Postgres
• we wanted Gilt’s customer facing applications to retrieve user data from MongoDB
Challenges
![Page 35: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/35.jpg)
• keep user data in both MongoDB and Postgres
• replicate from MongoDB to Postgres
Solution
![Page 36: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/36.jpg)
Replicating user data
user service
legacyweb app
replicationservice
![Page 37: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/37.jpg)
Replication service
• listens for RabbitMQ messages
‣ UserCreated message
‣ UserUpdated message
• retrieve data using REST API
• write data to Postgres using JDBC
![Page 38: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/38.jpg)
Use case #2: feature configuration
Goal
manage the release of new features on gilt.com
How
feature configuration persisted in MongoDB
![Page 39: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/39.jpg)
Rolling out a new feature
1. deploy new application code to production
2. enable feature for Gilt Tech employees
3. ... then enable for all Gilt employees
4. ... then enable for a subset of users
5. gradually ramp up to 100% of users
![Page 40: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/40.jpg)
Feature configservice
Feature configuration
gilt.com
![Page 41: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/41.jpg)
Use case #3: favorite brands
![Page 42: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/42.jpg)
userpreference
service
AJAX
Favorite brands
![Page 43: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/43.jpg)
Application development
![Page 44: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/44.jpg)
mongo-java-driver
Morphia
Casbah
![Page 45: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/45.jpg)
MongoDB Java driver
• main class: com.mongodb.Mongo
• Mongo object maintains a pool of connections
• Latest version: 2.10.1
![Page 46: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/46.jpg)
MongoDB Java driver connection pool tuning
MongoOptions opts = new MongoOptions();
// example values
opts.connectionsPerHost = 50;
opts.threadsAllowedToBlockForConnectionMultiplier = 5;
![Page 47: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/47.jpg)
com.mongodb.WriteConcern
• NORMAL
• SAFE
• REPLICAS_SAFE
• MAJORITY
• FSYNC_SAFE
• JOURNAL_SAFE
![Page 48: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/48.jpg)
Morphia library
![Page 49: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/49.jpg)
Morphia
• object-document mapper for Java
• built on top of mongo-java-driver
• map fields using Java annotations
![Page 50: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/50.jpg)
// Morphia example
import com.google.code.morphia.annotations.*;
@Entity(value="features", noClassnameStored = true)
public class Feature {
@Id
ObjectId featureId;
@Property("feature_key") @Indexed(unique=true)
String featureKey;
@Property("release_to_percentage")
int releaseToPercentage;
}
![Page 51: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/51.jpg)
http://code.google.com/p/morphia/
![Page 52: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/52.jpg)
Casbah library
![Page 53: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/53.jpg)
Casbah
• Scala toolkit for MongoDB
• built on top of the mongo-java-driver
• current version: 2.5.0
![Page 54: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/54.jpg)
Casbah
• Scala idioms
• Scala collections
• fluid query syntax
![Page 55: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/55.jpg)
https://twitter.com/max4f/status/230503836958199808
![Page 56: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/56.jpg)
Gilt Scala code
![Page 57: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/57.jpg)
trait MongoWriteConcern {
def withSession[T](f: => T)(implicit mongoDb: MongoDB) {
mongoDb.requestStart()
mongoDb.requestEnsureConnection()
try {
f
} finally {
mongoDb.requestDone()
}
}
}
![Page 58: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/58.jpg)
Best Practices
![Page 59: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/59.jpg)
Best practices
Connection tuning
explicitly configure Mongo Java Driver connection pool size
![Page 60: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/60.jpg)
Best practices
Use caution when creating new indexes
“creating a new index on a production mongo server can basically cause it to stop working while it builds the index”
(source: Gilt production incident report)
![Page 61: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/61.jpg)
http://docs.mongodb.org/manual/administration/indexes/#index-building-replica-sets
Minimizing impact of building a new index
• remove one secondary from replica set
• create/rebuild index on this instance
• rejoin replica set
• repeat on all remaining secondaries
• run rs.stepDown() on primary member
![Page 62: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/62.jpg)
Best practicesuse short names for fields to avoid wasted space
http://christophermaier.name/blog/2011/05/22/MongoDB-key-names
{
city: “Portland”,
state: “Oregon”,
country: “US”
}
{
ci: “Portland”,
st: “Oregon”,
co: “US”
}
vs
![Page 63: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/63.jpg)
Best practices
use explain() during development
db.collection.find(query).explain()
![Page 64: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/64.jpg)
Best practices
use caution when choosing a shard key
“It is generally not a good idea to use the default ObjectId as the shard key”
source: http://stackoverflow.com/questions/9164356/sharding-by-objectid-is-it-the-right-way
![Page 65: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/65.jpg)
Future
• shipping addresses
• discounts
• SKU’s and brands
More Gilt data in MongoDB
• tag aware sharding
Upgrade to MongoDB 2.2
![Page 66: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/66.jpg)
Gilt Groupe is hiring!http://techjobs.gilt.com
![Page 67: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/67.jpg)
Questions?
![Page 68: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/68.jpg)
[email protected]@tinyrobots
![Page 69: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/69.jpg)
The end
![Page 70: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/70.jpg)
Bonus slides
![Page 71: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/71.jpg)
Gilt tech talks
• Apache Camel and MongoDB @ Gilt http://bit.ly/OYO37K
• Deploying new features @ Gilt http://slidesha.re/OoCYfd
• Voldemort @ Gilt
http://bit.ly/b9Qhib
![Page 72: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/72.jpg)
https://twitter.com/stripe/status/298858032421535744
![Page 73: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/73.jpg)
![Page 74: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/74.jpg)
![Page 75: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/75.jpg)
![Page 76: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/76.jpg)
![Page 77: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/77.jpg)
![Page 78: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/78.jpg)
![Page 79: MongoDB at Gilt Groupe](https://reader031.fdocuments.in/reader031/viewer/2022020110/554f7c61b4c905435d8b4775/html5/thumbnails/79.jpg)