ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья...
-
Upload
geekslab -
Category
Data & Analytics
-
view
115 -
download
1
description
Transcript of ManetoDB: Key/Value storage, BigData in Open Stack_Сергей Ковалев, Илья...
MagnetoDB Key-Value Storageservice for OpenStack
Mirantis, 2014
Serge.Kovaleffgmail.comlinkedin.comfacebook.com
@
Odessa, GeeksLab 4/12/2014@
isviridov at FreeNode
WII FM
WII F
WII FM
What the heck is OpenStack?
What the heck is OpenStack?
to be continued ...
Is it Big?
Community
1227 active contributors in IceHousemore than 166 organizations
Communication
44 active official IRC channels2887 e-mails in March
Is it Big?
Is it about data?
CERN- 3 PB per custer- 50000 + 35000 core
PayPal 80000 VMs (according to Forbes)
OpenStack Big Data
OpenStack Sahara - Elastic Hadoop clusters provisioning and management on OpenStack and elastic data processing (on-demand Hadoop job workflow)
OpenStack MagentoDB - key-value storage service
What is MagnetoDB?
MagnetoDB - OpenSource Amazon DynamoDB API implementation for OpenStack, the key-value database service for storing any amount of data with seamless scalability and predictable performance.
MagnetoDB key features ● Easy-to-integrate REST-like API (AWS SDK, boto
clients)● OpenStack standards following REST API● Schemaless, non-relational table-based model● Put/get/query/scan item operations● Eventual and strong consistency reads● Local Secondary indexes● Batch read/write operations● Designed to handle any amount of data and any level
of request traffic● Seamless throughput and storage scaling● Fault tolerance
MagnetoDB use cases
● High request traffic level applications
● Time series data
● Queries are distributed across the data
● Any other applications
● Migration of workloads between AWS and
OpenStack
Architecture highlights
● Layered architecture
● Horizontally scalable
● Pluggable database backend
● Uses native OpenStack infrastructure
○ OpenStack HEAT for provisioning
○ OpenStack Neutron LoadBalancer
○ Integration with OpenStack Keystone planned
Overall architecture
Integration with OpenStack services
Current status
○ 2.0.2 version released
○ Open source, Apache 2 license
○ Launchpad homepage: https://launchpad.net/magnetodb
○ Code published on github: https://github.
com/stackforge/magnetodb
○ Deployable to Vanilla OpenStack
○ Compatible with existing DynamoDB clients
MagnetoDB Pilot Features
● Table CRUD API○ ListTables○ DescribeTable○ CreateTable○ DeleteTable
● Item CRUD API○ PutItem○ UpdateItem○ DeleteItem○ GetItem
● Data querying API○ Query○ Scan
● Error Handling API
MagentoDB use-case: dynamodb-geo
“Geo Library for Amazon DynamoDB allows to easily create and query geospatial data”
Next steps● General
○ Move the development to OpenStack Infra - done○ Integration with OpenStack Keystone - done
● MagnetoDB API, following OpenStack tenets - in progress● DynamoDB API
○ Atomic counters○ Asynchronous table creation○ Atomic conditional write operations○ Batch operations - in progress
● Other features○ Management API○ Quota management
Summary
● The 2.0.2 implementation is available on GitHub in public
repository
● MagnetoDB is compatible with AWS SDK and boto library
● It is now possible to run applications which use DynamoDB as
a storage on top of OpenStack using MagnetoDB
Links and Resources
○ Wiki: https://wiki.openstack.org/wiki/MagnetoDB
○ Launchpad: https://launchpad.net/magnetodb/
○ Source code: https://github.com/stackforge/magnetodb
○ IRC: #magnetodb @ FreeNode
○ AWS DynamoDB http://aws.amazon.com/dynamodb/
○ dynamodb-geo https://github.com/awslabs/dynamodb-geo
Q&A
Got Questions?