What's New in IBM Streams V4.2

© 2016 IBM Corporation

What’s New in IBM Streams V4.2

Mike Spicer

Distinguished Engineer - Lead Architect, IBM Streams

© 2016 IBM Corporation2

Important Disclaimer

2

The information contained in this presentation is provided for informational purposes only.

While efforts were made to verify the completeness and accuracy of the information contained in this

presentation, it is provided “as is”, without warranty of any kind, express or implied.

In addition, this information is based on IBM’s current product plans and strategy, which are subject to

change by IBM without notice.

IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this

presentation or any other documentation.

Nothing contained in this presentation is intended to, or shall have the effect of:

Creating any warranty or representation from IBM (or its affiliates or its or their suppliers and/or

licensors); or

Altering the terms and conditions of the applicable license agreement governing the use of IBM

software.

IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal

without notice at IBM’s sole discretion. Information regarding potential future products is intended to

outline our general product direction and it should not be relied on in making a purchasing decision.

The information mentioned regarding potential future products is not a commitment, promise, or legal

obligation to deliver any material, code or functionality. Information about potential future products

may not be incorporated into any contract. The development, release, and timing of any future

features or functionality described for our products remains at our sole discretion.


IBM Streams V4.2 Overview

Edge to Center analytics for IoT use cases

• Integration with Apache Edgent and Watson IoT Platform

Development with Python & Jupyter Notebooks

• Python application API to develop Streams apps entirely in Python

• Support for Python operators

Business Rules integration

• Business rules editor and associated SPL Type generation

• Business Rules compiler for optimized execution

Performance & application deployment improvements

Automatic fusion & dynamic threading

Security, serviceability & administration improvements

New and improved toolkits

• Speech to text


Edge to Center analytics for IoT use cases

Apache Edgent for analytics at the edge

• Programming model and lightweight embeddable runtime

• Functional Flow API

• Connectors and Runtime (executable graph of oplets)

• Dynamic control (Change queue size, change out of band data limits)

• Metrics (Optional, rich set of metrics using open source package)

• Open Source Apache Incubation Project

• Developed in the open at Apache using github and JIRA

Streams Integration for Center to Edge Analytics

• Streams pluggable “Device Hub” interface with IoT Platform implementation

• Manage Edgent apps by device type from Streams

• Control Edgent apps from Streams apps

• Streams console shows summary health of devices and Edgent apps

• Edgent app connects to Streams via IoTP (automatically configured)

IBM Confidential


Streams, Edgent & Watson IoT Platform Integration

Edgent for edge analytics – on device or gateway• Lightweight embedded streaming analytics runtime

• Analyze events locally on the edge

• Reduce communication costs by only sending relevant events

IBM Watson IoT Platform for optional device hub• Device management

• Message broker (MQTT)

• Public device hub API supports custom device hub

• Includes message brokers like Kafka

IBM Streams for streaming analytics• High performance, full featured streaming analytics

• Build windows of state and correlate across devices

• Have access to data-of-record systems, e.g. medical history

• Control edge device based upon analytics

• Central job management/health summary

• Automatic application connectivity

IBM Watson IoT PlatformDevice Hub

IBM StreamsStreaming Analytics

Edgent on Edge Devices and Gateways

Cluster

Gateway

Edge

DeviceEdge

Device

Messaging(MQTT, Kafka etc.)


Business Rules Integration

Deep Integration of ODM Business Rules

Tooling Integration

• Embedded Business Rules project in Streams Studio

• Business rules editor (ODM Rules Designer) in Streams Studio

• Generates SPL types from business rules

Rules Compiler

• Business rules compiled to C++ operator(s)

• Can generate an Operator per rules package

Dynamic update of rules for running applications

• Compiler generates new library for rules

• Ruleset Operator loads new library dynamically


Python Topology API

Implement Streams applications entirely in Python

• Functional Flow API similar to Java Topology API

• Define topology with source, sink and operations

• Submit to standalone and distributed contexts

• Use existing Python modules

• Can use and invoke popular modules like SciKits

• Support for Python V3.5 with Anaconda

• Support for Python 2.7 planned

Support for Jupyter notebooks

• Build and submit Python Streams application directly from notebook

• View stream data directly in python notebook

• Generate dynamically updating visualisations

Support for Python functions as Streams operators


Python Topology API


Performance and Application Deployment Improvements

Parallel Region Improvements

• Nested parallel regions

• Parallel split to send tuples to all channels

Consistent Region Improvements

• Asynchronous non-blocking checkpoints

• Increase concurrency of establishing and restoring consistent states

• Streams Hyperstate Accelerator support for checkpoints

Scheduling Improvements

• Better utilization across different sized resources

• The ability to reserve application resources for a specific purpose

Job Configuration Overlays

• Submission time Configuration & parameters in separate config file

• Same application can be run with different configurations with no code changes

Secure Application Configuration

• A set of properties accessible to applications by name

• Update notifications through JMX

• Scoped at Streams Domain or Instance level


Automatic Fusion and Dynamic Threading

Pre V4.2 Behaviour

• Fusion determined at compile time

• Changes to fusion require a recompile

• Default 1 PE process per Operator

• Threads defined manually and statically

• Threaded ports

New V4.2 Behaviour

• Fusion determined at submission time

• Default automatic fusion

• ~ 1 PE process per 4 Cores

• Based on resources available

• Optional directed fusion

• Specify number of PEs

• 1 to 1 Legacy behaviour

• Manual placement constraints take precedence

• Automatic threading determined at runtime

• Dynamically adjusts based on throughputAll fused

No fusing


Security, Serviceability and Administration Improvements

Security Improvements• Inter host communication encryption

• Management services and Inter PE connection using TLS/SSL

• Kerberos support• Single Sign On (SSO) authentication

• Microsoft Active Directory Authentication through Kerberos Tokens

Serviceability Improvements• Improved defensive coding and error case handling

• Improved logging and troubleshooting

• Improved metrics and zookeeper monitoring

Rolling Upgrades• Domain can be upgraded to new version with instances and jobs running

• Each instance can be at a different version and upgraded independently• Rolling upgrade of instance with jobs running planned for future release

Console custom templates for hyperlink targets

Streamtool Improvements• Persistent command history in console mode

• Manage jobs by jobname


New and Improved Toolkits

Speech to Text toolkit• Converts audio input stream to text output

• Uses Watson Speech to text technology

Open Source toolkits added to Product in V4.2• Datetime Toolkit

• IoT Toolkit• Connectivity with IoT platform

• JDBC toolkit

• JSON toolkit

• Network Toolkit• Ingest, parse and process network packets

• Map ip addresses to locations, reassemble files from packets

Geospatial toolkit enhancements• Support for shared maps for matching points to a map (PointMapMatcher)

Cybersecurity toolkit enhancements• QRadar integration

• Improved algorithms

Telecommunications Event Data Analytics (TEDA) toolkit enhancements

DPS toolkit support for Streams Hyperstate accelerator


Streaming Analytics Service in Bluemix

Provides the capabilities of Streams in the Bluemix Cloud

• Deploy any Streams application to run in the cloud

• The service has been generally available since October 2015

• Available in the US South, United Kingdom, and Sydney* Bluemix Regions

Capabilities at the initial GA included:

• Integration with key Bluemix services such as Watson IoT, Message Hub, and Cloudant.

• Sample, demo and reference applications.

• Support for many Streams toolkits

Users can create their Streams instance in seconds from the bluemix.net

web portal.

• Streams instances can be used interactively through the Streams web console or

programmatically via the Streaming Analytics REST API

• Instances are created initially with two dedicated application nodes, but users can

dynamically resize their instance

• Streams instances in Bluemix are highly available

Streams apps must be developed locally before deploying to the cloud

[ * syndicated in Sydney catalog with deployment in London]


Bluemix Streams Analytics – What’s new since Streams 4.1?

New features of the service

• Streaming Analytics Development Guide

• Enabled use of Streams product REST API in the cloud

• Subscription Price Plan

• Auto-stop of idle PayGo instances

• Real-time Hangout Detection (Geospatial Analytics Service)

Integration with other Bluemix services

• HDFS and HBase (BigInsights on Cloud)

• Object Storage

• RabbitMQ

• DashDB and SQL Database

• Secure Gateway

• Message Connect (experimental)

Additional demo and sample applications

• Twitter Smackdown Sample for Development Guide

• Water Conservation Starter Kit (IoT use case with Apache Edgent and IoT platform)

• Airport Sentiment Demo


Bluemix Streaming Analytics – Exploiting new Streams 4.2 features

Most new Streams 4.2 features are relevant to and can be used with the

Streaming Analytics service in Bluemix:

• Edge to Center analytics for IoT use cases

• Development with Python & Jupyter Notebooks

• Compiled business rules

• Performance & application deployment improvements

• Automatic fusion & dynamic threading

• Inter host communication encryption

• Toolkit improvements

Target for Streams 4.2 availability in Bluemix is October 19th.


Find out more and download V4.2 QSE at STREAMSDEV

https://developer.ibm.com/streamsdev/

Streaming Analytics Service in Bluemix

https://console.ng.bluemix.net/catalog/services/streaming-analytics

Get involved in the Apache Edgent Community

http://edgent.incubator.apache.org/

THANK YOU

https://developer.ibm.com/streamsdev/

https://console.ng.bluemix.net/catalog/services/streaming-analytics

http://edgent.incubator.apache.org/

What's New in IBM Streams V4.2

Data & Analytics

Transcript of What's New in IBM Streams V4.2