Ducksboard - A real-time data oriented webservice architecture
66
Ducksboard a real-time data oriented webservice architecture [email protected] @aitorciki on twitter
-
Upload
ducksboard -
Category
Technology
-
view
4.231 -
download
3
description
Talk by @aitorciki at BcnDevCon'11.
Transcript of Ducksboard - A real-time data oriented webservice architecture
What is Ducksboard?
● it's a web-based dashboard...
What is Ducksboard?
● it's a web-based dashboard...
● to display SaaS / your own metrics...
What is Ducksboard?
● it's a web-based dashboard...
● to display SaaS / your own metrics...
● with heavy focus on UX...
What is Ducksboard?
● it's a web-based dashboard...
● to display SaaS / your own metrics...
● with heavy focus on UX...
● huge bet on REAL real-time updates...
What is Ducksboard?
● it's a web-based dashboard...
● to display SaaS / your own metrics...
● with heavy focus on UX...
● huge bet on REAL real-time updates...
● and much more in the near future!
What is Ducksboard?
Architectural Goals
● data grabber / visualizer independence
Architectural Goals
● data grabber / visualizer independence
● real-time updates (push vs pull)
Architectural Goals
● data grabber / visualizer independence
● real-time updates (push vs pull)
● scalability (web scale, bitches!)
Architectural Goals
● data grabber / visualizer independence
● real-time updates (push vs pull)
● scalability (web scale, bitches!)
● eat your own duck food
Architectural Goals
Architecture Concepts
● Back-end (Twisted)
Architecture Concepts
● Back-end (Twisted)
● Parser: obtains data from external services
Architecture Concepts
● Back-end (Twisted)
● Parser: obtains data from external services
● Gateway: stores data
Architecture Concepts
● Back-end (Twisted)
● Parser: obtains data from external services
● Gateway: stores data
● API: provides read access to stored data
Architecture Concepts
● Front-end
Architecture Concepts
● Front-end
● web application
Architecture Concepts
● Front-end
● web application
● mobile web application (not yet available)
Architecture Concepts
● Front-end
● web application
● mobile web application (not available)
● mobile / desktop apps?
Architecture Concepts
Common Components
● PostgreSQL (the “DB”)
Common Components
● PostgreSQL (the “DB”)
● stores application configuration
Common Components
● PostgreSQL (the “DB”)
● stores application configuration
● stores data for historic queries
Common Components
● PostgreSQL (the “DB”)
● stores application configuration
● stores data for historic queries
● starts / stops components (PL/Python!)
Common Components
● RabbitMQ (the “queue”)
Common Components
● RabbitMQ (the “queue”)
● handles RT updates
Common Components
● RabbitMQ (the “queue”)
● handles RT updates
● simplifies publisher / consumer scenarios
Common Components
Architecture Overview
Backend Components: Parser
● a parser per service and user
Backend Components: Parser
● a parser per service and user
● many API models supported (RT preferred)
Backend Components: Parser
● a parser per service and user
● many API models supported (RT preferred)
● configuration stored in the DB
Backend Components: Parser
● a parser per service and user
● many API models supported (RT preferred)
● configuration stored in the DB
● obtained data sent to the gateway for storage
Backend Components: Parser
Backend Components: Gateway
● stores data in the DB for persistence
Backend Components: Gateway
● stores data in the DB for persistence
● pushes data to the queue for RT notifications
Backend Components: Gateway
● stores data in the DB for persistence
● pushes data to the queue for RT notifications
● receives updates from all the parsers
Backend Components: Gateway
● stores data in the DB for persistence
● pushes data to the queue for RT notifications
● receives updates from all the parsers
● exposes webservice for custom metrics
Backend Components: Gateway
Backend Components: API
● gets historic data from the DB and ...
Backend Components: API
● gets historic data from the DB and ...
● … RT updates from the queue
Backend Components: API
● gets historic data from the DB and ...
● … RT updates from the queue
● exposes HTTP interface for data retrieval
Backend Components: API
● gets historic data from the DB and ...
● … RT updates from the queue
● exposes HTTP interface for data retrieval
● exposes Websocket server for RT updates
Backend Components: API
Frontend Components: WebApp
● Django application
Frontend Components: WebApp
● Django application
● Nginx + uWSGI
Frontend Components: WebApp
● Django application
● Nginx + uWSGI
● uses the DB as a normal webapp (SQLAlchemy)
Frontend Components: WebApp
● Django application
● Nginx + uWSGI
● uses the DB as a normal webapp (SQLAlchemy)
● Celery handles async operations
Frontend Components: WebApp
● Backbone.js empowers the UI
Frontend Components: WebApp
● Backbone.js empowers the UI
● Raphaël draws the widgets
Frontend Components: WebApp
● Backbone.js empowers the UI
● Raphaël draws the widgets
● a little bit of jQuery everywhere :)
Frontend Components: WebApp
● Backbone.js empowers the UI
● Raphaël draws the widgets
● a little bit of jQuery everywhere :)
● Websocket or Flash fallback for RT updates
Frontend Components: WebApp
Architecture Overview
Scale (or die trying)
● every component is standalone (no code deps)
Scale (or die trying)
● every component is standalone (no code deps)
● components talk using HTTP (Nginx + DNS win!)
Scale (or die trying)
● every component is standalone (no code deps)
● components talk using HTTP (Nginx + DNS win!)
● RabbitMQ allows m-gateways / n-apis scenarios
Scale (or die trying)
Security
● HTTPS everywhere
Security
● HTTPS everywhere
● pgcrypto to provide credentials encryption
Security
● HTTPS everywhere
● pgcrypto to provide credentials encryption
● bcrypted webapp passwords
Security
● HTTPS everywhere
● pgcrypto to provide credentials encryption
● bcrypted webapp passwords
● relational integrity ensures data sanity
Security
Well, that's it! Questions?
Thanks for listening!and buying ;)
