Github Projects Overview and IBM Streams V4.1

© 2015 IBM Corporation

Github Projects Overview

IBM Streams 4.1

Samantha Chan

IBM Streams Community Architect

For questions about this presentation contact: [email protected]

2 © 2015 IBM Corporation

Important Disclaimer

THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONALPURPOSES ONLY.

WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THEINFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTYOF ANY KIND, EXPRESS OR IMPLIED.

IN ADDITION, THIS INFORMATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY,WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE.

IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OROTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.

NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:

• CREATING ANY WARRANTY OR REPRESENTATION FROM IBM (OR ITS AFFILIATES OR ITS ORTHEIR SUPPLIERS AND/OR LICENSORS); OR

• ALTERING THE TERMS AND CONDITIONS OF THE APPLICABLE LICENSE AGREEMENTGOVERNING THE USE OF IBM SOFTWARE.

IBM’s statements regarding its plans, directions, and intent are subject to change orwithdrawal without notice at IBM’s sole discretion. Information regarding potentialfuture products is intended to outline our general product direction and it should notbe relied on in making a purchasing decision. The information mentioned regardingpotential future products is not a commitment, promise, or legal obligation to deliverany material, code or functionality. Information about potential future products maynot be incorporated into any contract. The development, release, and timing of anyfuture features or functionality described for our products remains at our solediscretion.

THIS INFORMATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE.

IBM SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.


Agenda

IBMStreams Organization

What’s new in Streams Github Projects?

Getting Started with Streams 4.1


IBMStreams

http://ibmstreams.github.io

Open-source organization established on Github in March 2014

Goals:– Provide a platform and foster a community to extend and share Streams

programming resources (toolkits, samples, performance benchmark,

utilities, etc.)

– Allow us to deliver new toolkit functions in a more open, agile and rapid

manner

– Improve visibility of Streams programming resources and make them more

easily accessible.


IBMStreams March 2014

– Started the 3 repositories:• 3 toolkits from the Streams product (HDFS,

Messaging, Inet)

April 2015– Total: 31 Projects

Dec 2015– 1 Language Integration

– 9 Adapters

– 6 Parsers and Formatters

– 12 Processing and Analytics

– 6 Utilities

– 8 Demos / Samples / Docs

– Total: 42 Projects


What’s New? – Language Integration

Topology Project (streamsx.topology)– Shipped as part of Streams v4.1

– A project that supports building streaming topologies (applications) for

Streams in different programming languages, such as Java and Scala.

– An experimental version of Python is available.


What’s New? - Adapters HBase Toolkit (streamsx.hbase)

– Bluemix Support

HDFS Toolkit (streamsx.hdfs)

– Sequence File Support (Preview)

– Bluemix Support

Messaging Toolkit (streamsx.messaging)

– RabbitMQ (Preview)

– Kafka 0.9

– MessageHub

Inet Toolkit (streamsx.inet)

– The Inet toolkit provides support for common internet protocols. Supported protocols

include FTP, WebSocket, HTTP.

– Native Functions for httpPut, httpGet, httpPost, httpDelete

– Map visualization


What’s New? - Adapters

Multi-Connection TCP Server Toolkit (streamsx.tcp)– This toolkit contains a TCPServer operator which allows for multi-

connections, and is a multi-threaded source operator. The operator accepts

and sends text or binary data from one or more TCP sockets.

– Added support for full duplex TCP communication. Operator can now

accept and send data via a TCP socket connection.


What’s New? - Adapters

New repositories:

CDC Toolkit (streamsx.cdc)– Provides support for efficiently reading and writing data from InfoSphere

Change Data Capture (CDC)

GraphDB (streamsx.graphdb)– Under Construction

– Repository created to provide support for reading and writing data to graph

databases (e.g. IBM System G / Apache Tinkerpop)

JDBC (streamsx.jdbc)– JDBCRun operator to allow IBM Streams application to run queries from DB

via JDBC


What’s New? – Parsers and Formatters

Parquet Toolkit (streamsx.parquet)–Parquet is a columnar storage format for Hadoop. This repository is created for

hosting operators for reading and writing data in Parquet format.


What’s New? – Analytics and Processing

SparkMLLib (streamsx.sparkmllib)– This repository contains a toolkit for real-time scoring using SparkMLLib.

Healthcare Toolkit (streamsx.health)– This repository is under construction. This toolkit is intended to provide

building blocks and operators for real-time healthcare related analysis.

Social Toolkit (streamsx.social)– This repository provides support to ingest data from social media. Initial

support includes ingesting from Twitter


What’s New? – Analytics and Processing

Anomaly Detection (streamsx.anomalyDetection)– This repository contains operators and functions for anomaly detection. Initial

contribution would be based on the Local Outlier Function (LOF)

Internet of Things Foundation Toolkit (streamsx.iotf)– Provides the ability to have IBM Streams application to easily interact with

IoTF, either in Bluemix or on-premises.


Github Projects Overview – Utilities

Network Toolkit (streamsx.network)– Contains operators and functions for processing network data.

Transform Toolkit (streamsx.transform)– Contains building block operators to transform input data from one format to

another.

– Contains a high-performance Modify operator that can be used as a

replacement for Functor


Github Projects Overview – Samples / Demos / Docs

Cybersecurity Starter Applications

(streamsx.cybersecurity.starterApps)– Contains starter applications to get started with the IBM Streams

Cybersecurity Toolkit quickly

Tutorials– Contains Streams Introduction and SPSS labs

Streamsx.documentation– Community-based documentation

– Contains various getting started guides for Streams v4.1 features


Getting Started! - StreamsDev

https://developer.ibm.com/streamsdev/

https://developer.ibm.com/streamsdev/


Streamsx.documentation

Community-based documentation project

Complements StreamsDev and Knowledge Center

Provides Getting Started Guides for new features in Streams

Community contribution and feedback is much appreciated!

http://ibmstreams.github.io/streamsx.documentation/

http://ibmstreams.github.io/streamsx.documentation/


Get Involved!

Many new and cool projects to try!– E.g. Python Support, JDBC, Social

We want your feedback and input!– Report any issues you have found

– If you think it’s cool, let us know!

– Which toolkit do you use? How do you use it?

– Do you want the toolkit to be included in the product?

Contribute code, samples and docs!– Got a clever way to do things? Contribute it to the samples project!

– Need a new parameter? Need support for a new type of server / data format?

Work with us to try to implement it!

– Get a better way to explain things? Contribute to streamsx.documentation

Got an idea?– Propose a new project / new feature.


Questions?

Github Projects Overview and IBM Streams V4.1

Data & Analytics

Transcript of Github Projects Overview and IBM Streams V4.1