GettingStarted With- Elasticsearch by-IBM-ComposeElasticsearch:+Use+Cases •...

Elastic Stack in A DayMilano – 16 Giugno 2016

Getting Started With Elasticsearch by IBM Compose

Disclaimer© Copyright IBM Corporation 2016. All rights reserved.U.S. Government Users Restricted Rights -‐ Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON IBM'S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, NOR SHALL HAVE THE EFFECT OF, CREATING ANYWARRANTIES OR REPRESENTATIONS FROM IBM (OR ITS SUPPLIERS OR LICENSORS), OR ALTERING THE TERMS AND CONDITIONS OF ANY AGREEMENT ORLICENSE GOVERNING THE USE OF IBM PRODUCTS AND/OR SOFTWARE.

IBM's statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM's sole discretion. Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision. The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.

IBM, the IBM logo, ibm.com, Information Management, DB2, DB2 Connect, DB2 OLAP Server, pureScale, System Z, Cognos, solidDB, Informix, Optim, InfoSphere, and z/OS are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. If these and other IBM trademarked terms are marked on their first occurrence in this information with a trademark symbol (® or ™), these symbols indicate U.S. registered or common law trademarks owned by IBM at the time this information was published. Such trademarks may also be registered or common law trademarks in other countries. A current list of IBM trademarks is available on the Web at “Copyright and trademark information” at www.ibm.com/legal/copytrade.shtml

Other company, product, or service names may be trademarks or service marks of others.

• Introducing IBM Compose• Services & Deployments• Elasticsearch from Compose• Creating an Elasticsearch deployment• Migrating MongoDB to Elasticsearch with Transporter

Agenda

Introducing IBM Compose

• Compose is a cloud managed platform of DBaaS that delivers Public Multi-‐Tenant (MT) Open Source services that can be adopted individually or as part of a fit-‐for-‐purpose production data strategy• Compose databases are delivered as a complete, production-‐ready configuration• Daily backups and weekly/monthly snapshots• Intuitive dashboards with integrated monitoring and alerting

• Compose integrates open-‐source databases into a unified layer with powerful tools for provisioning, scaling, and connecting data

5

Leader in the field of JSON document databases.

An open-‐source, blazingly fast, low maintenance, key/value store, often used for data caching

Combines the power of a full text search engine with the indexing strengths of a JSON document database

A powerful, open source object-‐relational database that is highly customizable.

A JSON document based, distributed database with an integrated administration and exploration console

Asynchronously handles the messages between your applications and databases, allowing you to ensure separation of the data and application layers

This key/value store holds the always-‐correct data you need to coordinate and manage your distributed applications server clusters

RabbitMQ

etcd

Managed Platform of DBaaS Offerings

Customers Using A Broad Scope of Open Source Databases

E-‐Commerce Application

Product Catalog

MongoDB

Search Index

Elasticsearch

User Sessions

Redis

Financial Data

PostgreSQL

Our SolutionFrom a customer’s perspective

One click provisioning Unified system

Expert team Freeing the data

Checks all the marks Ease of operational management

Developers extend their knowledge

Transforms data from engine to engine

NoSQL (BSON)Document DB

Data Caching Key-Value DB

Full-Text Search Indexing Engine

Extensible and Secure Object Relational DB

Scalable JSON Database for

Real-Time Apps

Asynchronous Messaging Layer

Key-Value Database for

Distributed DBs

IBM Compose

MongoDB Redis Elasticsearch PostgreSQL RethinkDB RabbitMQ etcd

IBM Compose -‐ Services

IBM SoftLayer Amazon AWS

Compose services are deployable to

both SoftLayerand AWS

Available on SoftLayer as:§ IBM-Managed service§ Public Multi-Tenant

Available on AWS as:§ IBM-Managed service§ Self-Hosted service§ Public Multi-Tenant

IBM Compose


** Self-Hosted services coming soon

IBM Compose -‐ Deployments

IBM Compose

IBM SoftLayer

CloudantAnalytics

for Apache Spark

BigInsights on Cloud dashDB DB2 on

CloudObject

Storage DataWorks Watson Analytics


Compose services complementhosted and managed services from

IBM Cloud Data Services (CDS)

IBM Graph

Complementing IBM Cloud Data Services

Elasticsearch from Compose

• Elasticsearch from Compose uses the Compose Transporter connector to intelligently import data– as a one-‐off task or a continuous stream –from other DBs into Elasticsearch for aggregation, search and analysis• Synchronize Elasticsearch with MongoDB for deeper analysis of NoSQL data

• Available to Enterprise customers as IBM-‐Managed or Self-‐Hosted platform• Guaranteed RAM, IOPs, and CPU that autoscale as your Elasticsearch data grows• RAM autoscales 0.1 GB for every 1 GB of storage• 3-‐node cluster with SSDs, HA, and automatic failover• Automatic daily and on-‐demand (no-‐cost) backups• Plugins: elastichq and kibana installed by default (Compose Enterprise customers can tailor plugins)


Creating an Elasticsearch deployment• From your Compose.io dashboard, click the Deployments tab from the icons on the left side of the dashboard. Next Create Deployment.

Creating an Elasticsearch deployment• Within the Create Deployment tab select Elasticsearch and configure the size of your multi-‐tenant Elasticsearch service.

Elasticsearch: Service Summary

• Elasticsearch is a hosted and fully-‐managed search engine that can be integrated with applications

• Elasticsearch combines the power of a full text search engine with the indexing strengths of a JSON document database to create a powerful tool for rich data analysis on large volumes of data• With Elasticsearch your searching can be scored for exactness letting you dig through your data set for those close matches and near misses which you could be missing

• With Compose, you can bring Elasticsearch in as part of your full data stack• Compose's Transporter technology allows you to intelligently import data, either as a one off task or a continuous stream of updates, from other databases into Elasticsearch for aggregation, search and analysis

Elasticsearch: Use Cases

• Customers often look to a solution like Elasticsearch when they need to include full-‐text search in their application. Also, when the document remains primarily static, but needs to be retrieved frequently and quickly, to extract new values from the data• As an example, if a customer finds their analysis options in MongoDB limited, Transporter allows you to create a synchronized Elasticsearch store ready for the customer to dig deeper

• Other use cases• Compare large numbers of documents containing diverse data elements to display real-‐time results• Real-‐time queries tied to interactive dashboards• Full-‐text search and analytics on that data

Pairing Elasticsearch and other services

• Elasticsearch is a document data store; however, it is best considered as a specialized application solution (vs. a data solution)• In particular, it is lacking in data resiliency & partitioning tolerance found in other document databases such as Cloudant and

MongoDB

• We recommend Elasticsearch from Compose in scenarios where documents are not changing often, but need to retrieved quickly and frequently (such as in a search engine)

• Industry best-‐practice pairings of Elasticsearch and other Compose.io services include:

Elasticsearch

MongoDB

Elasticsearch

RethinkDB

Elasticsearch provides faster, more advanced search capabilities than are

available in document data stores

Migrating MongoDB to Elasticsearch with Transporter

• Assume a MongoDB database contains Twitter data that we want to be made available for analysis using Elasticsearch

• Transporter Setup:1. Elasticsearch dashboard on Compose.io è select Transporter tab è Transporter configuration page2. Select the MongoDB database and enter the name of the collection containing our captured Tweets3. Press Start Transport to begin data migration into Elasticsearch immediately OR Add a Transformation Function to

apply a transformation to the MongoDB data before ingest into Elasticsearch database• e.g. Map of Twitter abbreviations & their expansions to standardize a particular field or attribute• e.g. Scan doc fields for a list of keywords & pull those as input into the Elasticsearch database

4. Continuous Synchronization between MongoDB and Elasticsearch can be enabled (via MongoDB oplog monitoring) to ensure Elasticsearch data is kept in sync with MongoDB records

GRAZIE

https://www.compose.io

GettingStarted With- Elasticsearch by-IBM-ComposeElasticsearch:+Use+Cases •...

Documents

Transcript of GettingStarted With- Elasticsearch by-IBM-ComposeElasticsearch:+Use+Cases •...