MongoSF 2012
-
Upload
monica-wilkinson -
Category
Technology
-
view
4.360 -
download
0
Transcript of MongoSF 2012
© 2012 VMware, Inc. All rights reserved
Activity Streams on Cloud Foundry with MongoDB and Node.jsMongoSF - 2012
By Monica Wilkinson
Friday, May 4, 12
CONFIDENTIAL
About Monica Wilkinson
2
Loves the web and data portability.
Developer Advocate @ Cloud Foundry12 years development experience.
Last 5 years in Social WebOpen Web Standards Advocate
Contact Me: @[email protected]
Friday, May 4, 12
CONFIDENTIAL3
Agenda
1.Overview of Cloud Foundry2.What is Activity Streams ?3.Building an Activity Stream Engine
Friday, May 4, 12
CONFIDENTIAL
About Cloud Foundry
4
The first Open PaaS
Multi(n) Languages, Frameworks, Services & Clouds
Open Source
Friday, May 4, 12
5
Deploy and scale applications in seconds, without locking yourself
Simple, Open,
Flexible, Scalable
The Open Platform as a Service
Friday, May 4, 12
Cloud Foundry open PaaS - Choice of frameworks
Friday, May 4, 12
Cloud Foundry open PaaS - Choice of frameworks
OSS community
Friday, May 4, 12
Applica'on Service Interface
Data Services
Other Services
Msg Services
Cloud Foundry open PaaS - Choice of application services
Friday, May 4, 12
Applica'on Service Interface
Data Services
Other Services
Msg Services
Cloud Foundry open PaaS - Choice of application services
vFabric Postgres
vFabric RabbitMQTM
Additional partners services …
Friday, May 4, 12
Applica'on Service Interface
Data Services
Other Services
Msg Services
Cloud Foundry open PaaS - Choice of clouds
Private Clouds
PublicClouds
MicroClouds
Cloud Provide
r Interface
Friday, May 4, 12
Applica'on Service Interface
Data Services
Other Services
Msg Services
Cloud Foundry open PaaS - Choice of clouds
Private Clouds
PublicClouds
MicroClouds
Cloud Provide
r Interface
Avoid
Lock-in
Friday, May 4, 12
9
Multi-cloud flexibility is critical to your long-term success
Friday, May 4, 12
9
Multi-cloud flexibility is critical to your long-term success
§ Make use of both public and private clouds without rewriting your applications
Friday, May 4, 12
9
Multi-cloud flexibility is critical to your long-term success
§ Make use of both public and private clouds without rewriting your applications
§ Protect against vendor lock-in
Friday, May 4, 12
9
Multi-cloud flexibility is critical to your long-term success
§ Make use of both public and private clouds without rewriting your applications
§ Protect against vendor lock-in
§ Meet different compliance and geographical needs
Friday, May 4, 12
9
Multi-cloud flexibility is critical to your long-term success
§ Make use of both public and private clouds without rewriting your applications
§ Protect against vendor lock-in
§ Meet different compliance and geographical needs
§ Accommodate peak loads while optimizing costs
Friday, May 4, 12
9
Multi-cloud flexibility is critical to your long-term success
§ Make use of both public and private clouds without rewriting your applications
§ Protect against vendor lock-in
§ Meet different compliance and geographical needs
§ Accommodate peak loads while optimizing costs
§ Manage your growth and changing needs over time
Friday, May 4, 12
10
CloudFoundry.COM - Multi-tenant PaaS operated by VMware
Runtimes & Frameworks
Services
vCenter / vSphere
CloudFoundry.COM (beta)
Infrastructure
Friday, May 4, 12
11
Micro Cloud FoundryTM – Industry first downloadable PaaS
Runtimes & Frameworks
Services
Your Laptop/PC
Micro Cloud Foundry
Single VM instance of Cloud Foundry that runs on a developer’s MAC or PC
Friday, May 4, 12
12
CloudFoundry.ORG - Community open-source project
CloudFoundry.ORG
DownloadCode
Setup Environment
Deploy Behind Firewall
Setup Scripts
Apache2 license
Your Infrastructure
Friday, May 4, 12
Cloud Foundry: you can trade-off effort vs flexibility
13
Public PaaS
Private PaaS
Custom Private PaaS
Less
More
Less
More
.COM....
....
git clone git://github.com/cloudfoundry/
Effort Flexibility
Friday, May 4, 12
CONFIDENTIAL14
Why PaaS Matters
Friday, May 4, 12
Container (e.g. Tomcat)
Traditional web application architecture
15
Desktop Browser
Apache
Web Application
MySQL Database
developtestdeployscale
ShippingService
AccountingService
InventoryService
StoreFront
Simple to
Friday, May 4, 12
Container (e.g. Tomcat)
Traditional web application architecture
15
Desktop Browser
Apache
Web Application
MySQL Database
developtestdeployscale Or is it?
ShippingService
AccountingService
InventoryService
StoreFront
Simple to
Friday, May 4, 12
?Let’s imagine you want to run a performance test...
How long to get the servers?Who is going to set up the servers?Who is going to set up Apache, Tomcat, and MySQL?
Friday, May 4, 12
And things are changing:
This simple, monolithic architecture is inadequate
17
Friday, May 4, 12
18
Smart phones overtake PCs in Q4 2010
New kinds of clients
Friday, May 4, 12
Users expect a rich, dynamic and interactive experience on mobile devices and desktop
19
Web ApplicationDesktop Browser
HTTP Request
HTML/Javascript
Friday, May 4, 12
Users expect a rich, dynamic and interactive experience on mobile devices and desktop
19
Web ApplicationDesktop Browser
HTTP Request
HTML/Javascript
Old style UI architecture isn’t good enough
Friday, May 4, 12
Need to handle massive loads and the data explosion
20
Need scalable architectures
§ Application tier:•Replicated/clustered servers•Modular so that components can be scaled differently•Asynchronous architecture - communication via a message
broker§ Database tier:•Replication•Sharding•Polyglot persistence: Relational, NoSQL, NewSQL
databases
Friday, May 4, 12
Scaling development
21
§ Strongly coupled components makes development difficult
§ Forces multiple developers/teams to synchronize development efforts
§ Obstacle to frequent, independent deployments
§ Increases risk of failure - need to redeploy everything to change one thing
!=Scalable development
Web application
ShippingService
AccountingService
InventoryService
StoreFront
Friday, May 4, 12
Desktop Browser
NodeJS front-end application
Native Mobile Application
HTML5 mobile application
RabbitMQ
NodeJS
MySQL
Mongo
Redis
ShippingService
AccountingService
billing web application
shipping web application
InventoryService
inventory web application
StoreUI
StoreUI
StoreUI
StoreUI
22
Modern application architecture
Friday, May 4, 12
Developing and testing these applications is even more challenging
23
Friday, May 4, 12
Let’s imagine you are fixing a bug and want to run some JUnit integration tests...
Who is going to setup your sandbox: MySQL, RabbitMQ, MongoDB, ....??
Friday, May 4, 12
Let’s imagine you have fixed a bug and want to run some functional tests...
How long to purchase the servers?Who is going to set up the servers?Who is going to setup MySQL, RabbitMQ, MongoDB, ....??
Friday, May 4, 12
26
Platform-as-a-Service is the solution
Easy deployment
Application management
Easy scaling up and down
+SQL databases
NoSQL databases
Message Broker
Deployment Services
Friday, May 4, 12
CONFIDENTIAL27
Getting Started on Cloud Foundry
Friday, May 4, 12
CONFIDENTIAL
STEP 1 - Get a Cloud Foundry Account
§ https://my.cloudfoundry.com/signup/hack
28
Friday, May 4, 12
29
STEP 1 - Fetch password from email
Friday, May 4, 12
CONFIDENTIAL
STEP 2 - Login to Cloud Foundry
•sudo gem install vmc
•vmc login <username>
30
Friday, May 4, 12
CONFIDENTIAL
STEP 3 - Push your code to Cloud Foundry
vmc push
• Detects runtime automatically. • In addition if you have manifest.yml vmc will read:
• App Name • App Instances• Services to bind to App
•Demo
31
Friday, May 4, 12
CONFIDENTIAL32
Services on Cloud Foundry
Friday, May 4, 12
Current Services Available on Cloud Foundry
33
Friday, May 4, 12
Creating a service instance$ vmc create-service mysql --name mysql1
Creating Service: OK
$ vmc services
============== System Services ==============+------------+---------+---------------------------------------+| Service | Version | Description |+------------+---------+---------------------------------------+| mongodb | 1.8 | MongoDB NoSQL store || mysql | 5.1 | MySQL database service || postgresql | 9.0 | PostgreSQL database service (vFabric) || rabbitmq | 2.4 | RabbitMQ messaging service || redis | 2.2 | Redis key-value store service |+------------+---------+---------------------------------------+
=========== Provisioned Services ============
+-------------+---------+| Name | Service |+-------------+---------+| mysql1 | mysql |+-------------+---------+
Friday, May 4, 12
$ vmc push cer-spring --path web/target/Application Deployed URL [cer-spring.cloudfoundry.com]: Detected a Java SpringSource Spring Application, is this correct? [Yn]:
Memory Reservation (64M, 128M, 256M, 512M, 1G) [512M]:
Creating Application: OKWould you like to bind any services to 'cer-spring'? [yN]: y
Would you like to use an existing provisioned service? [yN]: yThe following provisioned services are available
1: mysql1
2: mysql-135e0Please select one you wish to use: 1
Binding Service [mysql1]: OKUploading Application:
Checking for available resources: OK
Processing resources: OK Packing application: OK
Uploading (12K): OK
Binding a service to an application
Friday, May 4, 12
$ vmc push cer-spring --path web/target/Application Deployed URL [cer-spring.cloudfoundry.com]: Detected a Java SpringSource Spring Application, is this correct? [Yn]:
Memory Reservation (64M, 128M, 256M, 512M, 1G) [512M]:
Creating Application: OKWould you like to bind any services to 'cer-spring'? [yN]: y
Would you like to use an existing provisioned service? [yN]: yThe following provisioned services are available
1: mysql1
2: mysql-135e0Please select one you wish to use: 1
Binding Service [mysql1]: OKUploading Application:
Checking for available resources: OK
Processing resources: OK Packing application: OK
Uploading (12K): OK
Binding a service to an application
Would you like to bind any services to 'cer-spring'? [yN]: yWould you like to use an existing provisioned service? [yN]: yThe following provisioned services are available1: mysql12: mysql-135e0Please select one you wish to use: 1Binding Service [mysql1]: OK
Friday, May 4, 12
$ vmc push cer-spring --path web/target/Application Deployed URL [cer-spring.cloudfoundry.com]: Detected a Java SpringSource Spring Application, is this correct? [Yn]:
Memory Reservation (64M, 128M, 256M, 512M, 1G) [512M]:
Creating Application: OKWould you like to bind any services to 'cer-spring'? [yN]: y
Would you like to use an existing provisioned service? [yN]: yThe following provisioned services are available
1: mysql1
2: mysql-135e0Please select one you wish to use: 1
Binding Service [mysql1]: OKUploading Application:
Checking for available resources: OK
Processing resources: OK Packing application: OK
Uploading (12K): OK
Binding a service to an application
Friday, May 4, 12
Binding a service to an application
$ vmc bind-service mysql1 myappBinding Service [mysql1]: OKStopping Application 'myapp': OKStaging Application 'myapp': OK Starting Application 'myapp': OK$
Friday, May 4, 12
CONFIDENTIAL37
Every day development on Cloud Foundry
Friday, May 4, 12
CONFIDENTIAL
Every day development§ Debugging and accessing the data locally
• Caldecott --> Service tunneling. Access your Cloud Foundry service as if it was local.
38
Friday, May 4, 12
CONFIDENTIAL
Tunneling
39
gem install caldecott
vmc tunnel <mongodb>
Friday, May 4, 12
CONFIDENTIAL
Using your favorite tools
40
Friday, May 4, 12
CONFIDENTIAL41
Friday, May 4, 12
CONFIDENTIAL42
What is Activity Streams ?
Friday, May 4, 12
CONFIDENTIAL43
http://activityStrea.ms§ Activity Streams is a simple open specification used to describe social
actions around the web.
§ The goal of Activity Streams is data portability
§ Some publishers of Activity Streams: Socialcast, MySpace, Facebook and G+
§ The default format is JSON but Atom is also supported. The JSON format was added after we saw some initial limitations with JSON
§ Core concepts are: Actor, Verb, Object and Target
Friday, May 4, 12
© 2012 VMware, Inc. All rights reserved
Actor -> Monica
Verb -> Pinned(Bookmarked)
Object -> Rosa Parks’ Bio Page
Target -> Female Leaders Board
Source -> Pinterest
Published -> MLK Day 2012
Friday, May 4, 12
CONFIDENTIAL45
Example Activity
John posted “SCALA 2012 Recap” on his blog “John OSS”10 minutes ago
Friday, May 4, 12
CONFIDENTIAL
Activity in JSON
46
{ "published": "2011-02-10T15:04:55Z", "actor": { "url": "http://example.org/john", "objectType" : "person", "id": "tag:example.org,2011:john", "image": { "url": "http://example.org/john/image", "width": 250, "height": 250 }, "displayName": "John Smith" }, "verb": "post", "object" : { “objectType” : “blog-entry”, “url": "http://example.org/blog/2011/02/entry", "id": "tag:example.org,2011:abc123/xyz" }, "target" : { "url": "http://example.org/blog/", "objectType": "blog", "id": "tag:example.org,2011:abc123", "displayName": "John OSS" } }
Friday, May 4, 12
CONFIDENTIAL47
What is an Activity Stream Engine ?§ An activity stream engine allows you to publish events and subscribe to
events.
§ Many activity stream engine support aggregation of events via Streams.
§ Many activity stream engines support fanning out activities to subscribers.
§ Many activity stream engines tailor the stream to the reader providing Key Stats and Contextual Info. Example: Monica with 1000 followers or Bank of America established in 1989
Friday, May 4, 12
CONFIDENTIAL48
Building an Activity Streams Engine
Friday, May 4, 12
CONFIDENTIAL49
What should we use to store Activities ?
ColumnKey-Value Document Graph
Redis, Riak
Cassandra,HBase
MongoDB Neo4J
Friday, May 4, 12
CONFIDENTIAL50
ColumnKey-Value Document Graph
MongoDB (so easy)
Mongo DB schema-less documents are perfect for Activities
Friday, May 4, 12
CONFIDENTIAL
About MongoDB
51
1. No-SQL database
2. Stores JSON-style documents with embedded documents
3. Horizontally scalable
4. Full Indexing Support
5. Open Source so great ORMs and drivers.
development : {
tools: many,
language_support: superb,
agility: high
},
production: {
speed: fast
fault_tolerance: true
scalability: high
=
Friday, May 4, 12
CONFIDENTIAL
Step 1- Build the persistence Modelvar MediaLinkHash = {
duration: Number, height: Number, width: Number, url: String };
var ActivityObjectHash = { id: {type: String}, image: MediaLinkHash, displayName: {type: String}, summary: {type: String}, content: {type: String}, url: {type:String}, author: {type: ObjectId, ref: 'activityObject'}, published: {type: Date, default: Date.now}, updated: {type: Date, default: Date.now}, objectType: {type: String}, attachments: [{type: ObjectId, ref: 'activityObject'}], upstreamDuplicates: [String], downstreamDuplicates: [String] };
var ActivityObjectSchema = new Schema(ActivityObjectHash) this.ActivityObject = mongoose.model('activityObject', ActivityObjectSchema);
52
Friday, May 4, 12
CONFIDENTIAL
Step 1- Persistence Model var ActivitySchema = new Schema({ id: {type: String}, verb: {type: String}, url: {type: String}, title: {type: String}, content: {type: String}, icon: MediaLinkHash, object: ActivityObjectHash, actor: ActivityObjectHash, target: {type: ObjectId, ref: 'activityObject'}, published: { type: Date, default: Date.now}, updated: { type: Date, default: Date.now}, inReplyTo: {type: ObjectId, ref: 'activity'} });
this.Activity = mongoose.model('activity', ActivitySchema);
return this;};
53
Friday, May 4, 12
CONFIDENTIAL
Step 2 - Expose helpers for the queries§ The most important is to be able to list the activities in
descending order by published time.§ It is also important to hydrate any objects for which we
have references
§ With Mongoose you can do: § this.getActivityStream = function(n, fx) { Activity.find().sort('published', 'descending').limit(n).populate('target').run(fx);
}
§ More at: https://github.com/cloudfoundry-samples/activity-streams-mongoose
54
Friday, May 4, 12
CONFIDENTIAL
How do we make our engine faster and more scalable ?§ Add support for PubSub§ In a PubSub model as soon as an event is published it
gets sent to all the subscribers.§ The Publisher doesn’t know who the subscribers are, it
simply knows where to publish.§ Redis has nice support for Pub Sub http://redis.io/topics/
pubsub
55
Friday, May 4, 12
CONFIDENTIAL56
Exposing the Engine as Service
Friday, May 4, 12
CONFIDENTIAL
Using Socket.io
§ Server Side:• Subscribe to a Stream• When there is a new event send it to the client as JSON
§ Client Side• Add support to publish activities to the Stream• Change the client rendering to use ActivityStrea.ms
§ Benefits• Richer messaging• Activity Syndication
57
Friday, May 4, 12
CONFIDENTIAL58
Demo
http://asms.cloudfoundry.com
Friday, May 4, 12
© 2012 VMware, Inc. All rights reserved
Thank You - Questions ?Cloud Foundry Activity Streams Libraries are at:
https://github.com/organizations/cloudfoundry-samples
http://cloudfoundry.com
Questions: @cloudfoundry - @ciberch or [email protected]
Friday, May 4, 12