Pentaho Dev Day 30th October 2013 - Meetupfiles.meetup.com/1804355/Pentaho_Zaponet_DevDay... · 1...

© 2013, Pentaho. All Rights Reserved. pentaho.com. Worldwide +1 (866) 660-7555 1

Pentaho Dev Day

30th October 2013

Sébastien Cognet

Sales Engineer EMEA

@opentoile


Agenda

PENTAHO Introduction

Analytic suite and more…

USER CONSOLE Architecture

Report, Analyze & Dashboard

Datasource management

Mobility

Demo #1

DEV TOOLS Metadata

Metadata Editor

Schema Workbench

Report Designer

CTools

INTEGRATION Graphic Design

Agile development

Demo #2

OEM Embedded analytics

Many architectures

BIG DATA Visual MapReduce

Big Data Layer

Bended Data

Demo #3

DATAMINING Weka

PDI


PENTAHO CORP

PRESENTATION


Pentaho Mission Big Data Analytics without Boundaries

Modern, unified data integration and business

analytics platform

• Native integration into big data ecosystem

• Embeddable, cloud-ready analytics

Critical mass achieved

• Over 1,000 commercial customers

• Over 10,000 production deployments

Fast and Broad Innovation

• Open source development model

• Extensible by customer, partner & community


Analytic suite and more …

Enterprise Analytic Platform Easy of Use / Administration / Audit

Data Integration

Traditional

Big Data Layer (framework)

Blended Data (as a service)

OEM

One value

CTOOLS framework

Multi-Tenant

Embedded

VISUALIZATION

Report

Analyze

Dashboard

Data Wizard

Services

Training

Workshop

Checkpoint

Consulting

Subscription

Productivity

Garanty

ASSISTANCE

Conciergerie

Network

JIRA

Help-Desk

Infocenter

Community / Forums


Pentaho Business Analytics Platform Our approach addresses the challenges

Operational

Data

Big

Data

Data

Stream

Public/

Private Clouds

Multi-Tenant Ready Open API’s 100% Java

Access Integrate Cleanse Enrich

Score

Forecast

Connect Visualize

Report Dashboard

Analyze/Explore

DBA ETL/BI Developer

Business Users Executives

Analysts Data Scientists

Embed

Use Case Segments

Big Data

Business Analytics


Data Ingestion

Manipulation

Integration

Enterprise &

Ad Hoc Reporting

Data Discovery

Visualization

Predictive Analytics

Complete Big Data Analytics &

Visual Data Management

Relational Hadoop NoSQL Analytic

Databases

Pentaho Big Data Analytics


Pentaho Data Integration

• Visual development for big data

• Broad connectivity

• Data quality & enrichment • Integrated scheduling

• Security integration

• Visual data exploration

• Ad hoc analysis

• Interactive charts & visualizations

DASHBOARD DESIGNER

• Self-service dashboard builder

• Content linking & drill through

• Highly customized mash-ups

Pentaho Data Mining / Predictive Analytics

• Model construction & evaluation

• Learning schemes

• Integration with 3rd part models using PMML

Pentaho Product Components

INTERACTIVE REPORT

• Both ad hoc & distributed reporting

• Drag & drop interactive reporting

• Pixel-perfect enterprise reports

Pentaho for Big Data MapReduce & Instaview

• Visual Interface for Developing MR

• Self-service big data discovery

• Big data access to Data Analysts

ANALYZER


Pentaho Business Analytics

Modern Platform Built for the Future of

Analytics

100% Java, cross-platform Modular, lightweight, pluggable High-performance, scalable

Service-oriented architecture for easy integration Standards-based, highly extensible, easy to embed

Reporting, dashboards, analysis, data mining, predictive Power tools for business users, analysts, & data scientists

Structured, unstructured & NoSQL data Native support for emerging Big Data sources

End-to-end platform – unified data integration and business analytics Agile approach for fast prototyping & iterations Low cost subscription

Modern Architecture

Embedded Analytics

Broadest Spectrum of Insight

Diverse Data

Integrated, Low Cost Platform


Pentaho Core Competencies

Information Consumers

Powerful Reporting and Visualization Business Users

Power Users, Developers &

DBAs

Data Integration and Big Data

Advanced Analytical

Professionals Data Mining (Predictive Analytics)

Knowledge Workers/

Business Users

Self-Service Analysis & Queries



Any Report Executive dashboards

Management / Operational Reports

Financial Reports

Any Format HTML – for the web

PDF – for printing

Excel / CSV – for finance or sharing

Anytime / Anywhere On-demand, scheduled

Event driven – manage by exception

Access via web portal, email or mobile

Information Consumers

Powerful Reporting and Visualization Business Users



Easy and Intuitive Content Creation Drag and drop

Metadata intelligence

Context-sensitive, right-click interface

Web-based Interactivity Drill down, drill thru, slice & dice, pivoting, lasso filtering

User-defined calculations

Rich Visualizations Scatter plots

Geo-mapping

Heat grids

Knowledge Workers/

Business Users

Self-Service Analysis & Queries



Deep Integration with multiple sources and analytics output

Inputs: RDBMS, files, web services, NoSQL, Analytical DBs, Hadoop

Output: ETL is tightly coupled with analytics

Scalability Scale up – multi-threaded

Scale out - clustering

Workflow Scheduling

Monitoring

Alerting

Power Users, Developers &

DBAs

Data Integration and Big Data



Full data mining lifecycle support Preparation of input data

Statistical evaluation of learning schemes

Visualization of input data and result of learning

Visualization, classification and clustering capabilities

Explorer - data exploration/visualization, model construction and export, preliminary evaluation

118 classification/regression algorithms

11 clustering algorithms

Integrated with PDI ETL Execute Weka and R predictive models inside of a PDI transformation

Append probabilities dynamically to each row in the data flow

Retrain Weka models using the KnowledgeFlow plugin for PDI

Advanced Analytical



Refs

../../Pentaho_Corp/Sales/References/Pentaho_BrusselsAirport_fr.pdf

../../Pentaho_Corp/Sales/References/Pentaho_StAntonius_fr.pdf

../../Pentaho_Corp/Sales/References/Pentaho_Lufthansa_us.pdf

http://www.pentaho.com/customers/44/marketo

http://www.pentaho.com/customers/20/mozilla

../../Pentaho_Corp/Sales/References/Pentaho-Nexway_fr.pdf

../../Pentaho_Corp/Sales/References/Pentaho_Sisteer_fr.pdf

http://ww2.pentaho.com/customers/62/union-investment/

http://www.pentaho.com/customers/100/edo-interactive

http://www.pentaho.com/customers/46/controlpay


Click Stream Analytics From buying patterns to revenue

360o View of Customer

• Monetize buying patterns hidden in billions of

data points

• Quickly analyze multi-channel click stream data

Pentaho Benefits

• Reduced ETL time to analyze blended data

from Hadoop, Hbase & data warehouse

• Use of big data analytics to grow revenue from

targeted campaigns


Device Data Analytics Big Data for Net App

Business Challenge

• Affordably scale machine data from storage

devices for customer support app

• Predict device failure

• Enhance product performance

Pentaho Benefits

• Easy to use ETL & analysis for Hadoop, Hbase,

& Oracle data sources

• 15x cost improvement

• Stronger performance against customer SLA’s


Data Warehouse Optimization Cost effective, fast processing

Business Challenge

• Gain competitive advantage through intraday

balance reporting for commercial customers

• Use Hadoop and relational data stores to

process huge volumes 15x faster

to develop

10x faster

to execute

No coding

Integrate

with existing

Easy to find

resources

Pentaho Benefits

• Graphical orchestration for Hadoop, Hbase &

DB2 data integration workloads

• 15x faster to develop, 10x faster to execute


Telecom Use Cases Big Data Analytics and the Customer Experience

Expand Net Promoter Score beyond traditional

survey methods

• Add social media data

• Focus on highly valued customers

• Refine predictive models

Better track and manage overall IT

infrastructure for telecom services

• Capacity planning and forecasting

Use emerging sensor/device data to enable

services for a customer’s connected lifestyle

• Connected car, digital life, and mobile wallet

On-Line Ad Performance

• YP.com


USER CONSOLE

PRESENTATION


Plateform

CENTRAL ADMINISTRATION, AUDITING & MONITORING

DELIVER When & Where Users Need It

STREAMLINE Information Delivery

VISUALIZE & Report Information In Any Style

ACCESS All Data Sources

ISV & Packaged Applications

SaaS / Cloud Applications

INTEGRATION

Web

Mobile

Print

E-Mail

STANDALONE

‣ Advanced &


DATA MINING

‣ Proactive

‣ Operational

‣ Enterprise

REPORTING

‣ Ad hoc Exploration

‣ Multi-Dimensional

ANALYSIS

‣ Interactive Metrics

‣ Rich Visualizations

DASHBOARDS

ERP / CRM /

Enterprise Apps (e.g. SAP, Oracle)

Hadoop, NoSQL Data & Analytical

Unstructured &

semi-structured (XML, Excel, Files, etc.)

Traditional Relational Data

Cloud (e.g. Salesforce,

Amazon, Dell)

‣ Direct Access

‣Data Integration

‣ Hadoop Clustering

‣ Graphical ETL Designer

‣ Enterprise Scalability

INTEGRATE, CLEANSE, & ENRICH DATA

‣ In Memory Caching

‣ High Performance

‣ Relational OLAP Cubes

METADATA LAYER


Generic architecture

Unstructured data

EDW

Structured data

Technology BIG DATA

and/or

Staging Area


Collect


Cleansing

Transformation

Change Data Capture

Data Warehouse Management

PDI PDI Metadata

Dashboard

Report

Analyzer


Cache

Complete management

Ad-hoc Data

Data Mart(s) / Entrepôts

Alertes SMS, eMail & pièces jointes

PDI Metadata

Dashboard

Rapport

Analyse

Technology BIG DATA

and/or

Staging Area

PDI

Collect, Transform, Load and Alert

Structured data

Unstructured data




J2EE Container (Tomcat/JBoss)

Data Integration

Architecture: Components Se

rver

W

ork

stat

ion

Th

in c

lien

t

Business Analytics

• Analytics • Reporting • Dashboards

Data Discovery and Advanced Analytics

Reporting, Ad Hoc Query, Dashboards, Mobile

Pu

blish

H

TTP/HTTP

S

• ETL • Data profiling, cleansing, quality • Job Design/Orchestration

• Scheduling • Administration • Content Storage

Report Designer PDI Designer Metadata Design


Architecture: Technology

J2EE Container (Tomcat/JBoss)

Data Integration

Serv

eur

Stat

ion

de

trav

ail

Clie

nt

lége

r

Business Analytics

• Analytics • Reporting • Dashboards

Data Discovery and Advanced Analytics

Reporting, Ad Hoc Query, Dashboards, Mobile

• ETL • Data profiling, cleansing, quality • Job Design/Orchestration

• Scheduling • Administration • Content Storage

Report Designer PDI Designer Metadata Design

• JavaScript • Dojo • GWT

• Swing • SWT

• Java • J2EE Web

Application

Technologies


Extensible Architecture

BUSINESS ANALYTICS PLUGIN TYPES:

❯ Platform plugins - security integration, new components

❯ User Console – new editors, analytic displays

❯ Analyzer visualizations – integrate 3rd party visualizations

❯ Dashboard Framework – filter control types, visualizations

❯ Dashboard Designer – additional widget types

DATA INTEGRATION

❯ Transformation Steps – connectors, transformation elements

❯ Job Entries – process/orchestration elements

❯ Perspectives – integrated design or analytic environments


Architectural Components


User Console

DEMO


2013 Performance Goals

1. Increase subscription revenue by analyzing call data to upsell PAYG customers to subscriptions

2. Improve store profitability by holding store managers accountable by bursting store income statements

3. Reduce stock outs with real-time inventory report delivered on an Ipad.

4. Maximize profits by profiling users with high average call duration

5. Maximize revenue by analyzing e-commerce clickstream data in MongoDB to profile purchasing users

6. Improve supply chain by giving phone manufacturers and resellers web-based reports


3 Calling Plans

• Nationwide

• PAYG

• Prepaid 50

2 Business

Units

• B2B

• B2C

7 Retail Stores

7 Product

Lines

3 Websites

Clear Wireless – Wireless Carrier

10 Resellers

9 Phone

Manf

Red River Mobile

Apple

• San Francisco • Boston • NYC • Paris • Tokyo • Sydney • London

• Smartphones • Home Phones • Wifi Devices • Modems • Notebooks • Tablets • Accessories

• Ecommerce Site • Reseller Portal • Manufacturer Portal

EXTERNAL INTERNAL

IFrame Integration

Custom Widget Embedding


Pentaho pour iPad & Android Instant Visualization & Analysis for Mobile Users

INSTANT AND INTERACTIVE VISUALIZATION

❯ Attractive dashboards, analysis, operational &

enterprise reports

❯ Touch filtering, drill-thru to details

POWER TO CREATE NEW ANALYSIS ON

THE GO

❯ Unique to Pentaho

❯ Highly interactive vs. a read-only access to static

content

EASY TO DEPLOY, EASY TO EMBED

❯ IT-free, create once, access anywhere

❯ Web-based, easily embeddable into mobile apps


DEV TOOLS


Translation

YOU CAN LOCALIZE THE PENTAHO USER CONSOLE AND DEVELOPMENT TOOL IN ANY LANGUAGE USING ISO FORMAT

CONCEPT:

❯ Application:

❯ Specific message bundles within the Pentaho Web application

❯ Message bundles are dynamically adjusted according to browser locale

❯ Metadata:

❯ Reporting & Analysis metadata development tool contain specific localization functionalities

❯ Data:

❯ As in your database. Can use multi Tenant Id to switch beetwen different content.


Metadata Translation

METADATA

EDITOR

❯ All data can

be translated

SCHEMA

WORKBENCH

❯ All levels can

be translated


Metadata

Data Mart(s) / Warehouse

Metadata

Dashboard

Rapport

Analyse

PDI Datasource

Operations Mart


Schema Workbench

Pentaho Server

Mondrian Schema

Metadata Schema

MDX

SQL

Metadata Editor

Analyzer

Interactive Reporting

Report Designer

Architecture Metadata

2 TYPES OF METADATA

❯ Metadata Report

❯ Metadata Olap


END USER CAN CREATE HIS OWN METADATA

❯ From a file

❯ With Sql Statement

❯ From database

Architecture Metadata

Data Source Wizard

Pentaho Server

Mondrian Schema

Metadata Schema

MDX

SQL

Analyzer

Interactive Reporting

Report Designer


Schema Workbench & Aggregation Designer

CALCULATION, VIRTUAL CUBES AND AGGREGATION TOOL


Metadata Editor

CREATE METADATA FOR YOUR END-USERS

Modeling

Properties


PENTAHO DATA INTEGRATION


PDI Composants

Extract

Transform

Load


PDI Components

SPOON ❯ Graphical environment for modeling

❯ Transformations are metadata models describing the flow of data

❯ Jobs are workflow-like models for coordinating resources, execution and dependencies of ETL activities

PAN ❯ Command line tool for executing

transformations modeled in Spoon

KITCHEN ❯ Command line tool for executing

jobs modeled in Spoon

… AND OF COURSE KETTLE ❯ The Engine itself

KDE ETTL Environment

Spoon Interface – Designing a Transformation

Job Example


PDI Components

ENTERPRISE EDITION DATA INTEGRATION SERVER

❯ Execution and remote monitoring

❯ Integrated scheduling

❯ Enterprise Security options

❯ Enhanced content management including revision history and locking

❯ Remote distributed cluster based processing


Any Format of Data


Overall Management

Not just processing … A key element once in a production environment


Access Controls

PREVENT UNAUTHORIZED USERS FROM VIEWING DATA

TRANSFORMATION RULES AND POSSIBLY CONNECTION

CREDENTIALS (E.G. DATABASE LOGINS / PASSWORDS)

❯ Integrate with existing security (e.g. LDAP / Active Directory)

PROVIDE ACCESS TO TRANSFORMATIONS AND JOBS ON A

“NEED TO KNOW” BASIS’

PROTECT DATABASE LOGINS AND PASSWORDS


Version Control

AUTOMATICALLY SAVE MULTIPLE REVISIONS OF A

TRANSFORMATION OR JOB

ELIMINATE THE RISK OF “FAT FINGERS” … ACCIDENTAL

DELETION OR CHANGES

EXPERIMENT WITH DIFFERENT ETL DESIGNS WHILE PRESERVING

THE ORIGINAL

RESTORE TRANSFORMATIONS AND JOBS FROM AN EARLIER

VERSION


Production Control

IDENTIFY THE CORRECT VERSIONS OF TRANSFORMATIONS AND

JOBS TO RUN

ALLOW “EXECUTE ONLY” BY IT OPERATIONS PERSONNEL

LOCK TRANSFORMATIONS WITH COMMENTS

SCHEDULE JOBS TO RUN ON A CENTRAL SERVER AT

PREDETERMINED TIMES



STEP BASED PROCESSING ENGINE WITH INSTANT VISUALISATION

OF RESULTS



• AGILE BI METHODOLOGY

• Load

• Modeling

• Visualize


Traditional DB

DATA INTEGRATION ANALYSIS

etc etc etc


Broadest Support for Big Data Platforms

Hadoop NoSQL Analytic Databases


BIG DATA


Big Data?

http://www.youtube.com/watch?v=QV3t-3QIf1E






CENTRAL ADMINISTRATION, AUDITING & MONITORING

DELIVER When & Where Users Need It

STREAMLINE Information Delivery

VISUALIZE & Report Information In Any Style

ACCESS All Data Sources

ISV & Packaged Applications

SaaS / Cloud Applications

INTEGRATION

Web

Mobile

Print

E-Mail

STANDALONE

‣ Advanced &


DATA MINING

‣ Proactive

‣ Operational

‣ Enterprise

REPORTING

‣ Ad hoc Exploration

‣ Multi-Dimensional

ANALYSIS

‣ Interactive Metrics

‣ Rich Visualizations

DASHBOARDS

ERP / CRM /

Enterprise Apps (e.g. SAP, Oracle)

Hadoop, NoSQL Data & Analytical

Unstructured &

semi-structured (XML, Excel, Files, etc.)

Traditional Relational Data

Cloud (e.g. Salesforce,

Amazon, Dell)

‣ Direct Access

‣Data Integration

‣ Hadoop Clustering

‣ Graphical ETL Designer

‣ Enterprise Scalability

INTEGRATE, CLEANSE, & ENRICH DATA

‣ In Memory Caching

‣ High Performance

‣ Relational OLAP Cubes

METADATA LAYER

Big Data with Pentaho

BIG DATA Discovery

‣ Instaview


Classic usage

Web Application

User Behavior JavaScript, Java, PHP,

Embedded Specialist Tool

CRM Style Data


ORCHESTRATE

ERP DW

Processing

CRM

Pig, Oozie, Flume, Hive, Hbase, Sqoop

Raw Data

Parsed Data

Analytic Datasets

Transform & visualize

Master Data

Analysis & Reporting

A

N

A

L

Y

Z

E

Unstructured Data

Structured Data

INGEST

Ingestion

VISUAL MAP REDUCE

Data Integration Analytics


Pentaho Big Data Strategy

• VISUAL MAP REDUCE • Graphic development

• Technical architecture near from Hadoop

• BIG DATA LAYER • Framework included all Big Data distribution

• Technical partnership (Cloudera, HortonWorkd, MongoDB, …)

• BLENDED DATA • JDBC Driver to use our ETL like a datasource

• Data as a service


Pentaho Visual Development Eliminates need for complex coding

Would you rather do this?

Integrate, Manipulate, Ingest

… or this?

Schedule

Model


Pentaho Visual MapReduce Drag & Drop then run in the cluster

Parallel execution as MapReduce

in the Hadoop cluster

Up to 15x faster than hand-

written code


VISUAL MAP REDUCE

UNIQUEMENT DES DÉVELOPPEURS ETL

The main part of your transformation doesn’t change… only a new first and last steps


Big Data Orchestration

• Scheduling, Event management • Use your existing scripts (cf. scripts Pig) • All Db’s and File system – Hadoop, NoSQL, RDBMS, …


Adaptive Big Data Layer Leadership in Big Data Integration and Analytics

• Insulates from changing versions, vendors, data stores

• Give customers broad flexibility of choice, rapid time to value, reduced risk

• Provides native integration into the big data ecosystem

• Broadest, deepest Big Data Support

Transparent Access to & Integration of Big Data


BIG DATA LAYER

ALL MAIN HADOOP DISTRIBUTION

NOSQL CONNECTORS

ACCESS TO AMAZON REDSHIFT & SPLUNK.


How Pentaho helps you?

• WE REDUCE COMPLEXITY WHICH SIMPLIFIES MIGRATION TO NEW VERSIONS OF

HADOOP

• BECAUSE WE DON’T GENERATE CODE, WE REDUCE THE RISK OF OBSOLESCENCE AS

HADOOP EVOLVES


Data Ingestion

Manipulation

Integration

Enterprise &

Ad Hoc Reporting

Data Discovery

Visualization


Complete Big Data Analytics &

Visual Data Management

Relational Hadoop NoSQL Analytic

Databases

Pentaho Big Data Analytics


Orchestration Engine

Applications

Databases

Files

Job (.KJB)

Plug-In Job

Entries

Monitoring Logs

Flume

SQL

Files FTP

Sqoop

Folder

Oozie

Email

Sub-Job

Analytic DB

NoSQL

Hadoop Cluster

PDI Engine


Transformation Engine

Data Node

Task Tracker


Data Node

Task Tracker


Data Node

Task Tracker


JobTracker

Orchestration

Distributed Cache

Transformation (.KTR)



BLENDED DATA

Data sources SQL

Datawarehouse

Location

Network

Web

Social Media

WebServices NoSql

Hadoop


• Takes time • Requires IT • Target database is updated as

transformations are run

How do we integrate data today?


Location

Network

Web

Social Media

We put it in: Hadoop NoSQL Analytic DB’s

Where do we store big data?


Location

Network

Web

Social Media

How can we bring it together?

ETL


Location

Network

Web

Social Media

PDI

What if a user could bring together both types

of data on demand?


❯ Just in time blending of data from multiple sources for a complete

picture

❯ Connect, combine and transform data from multiple sources

❯ Query data directly from any transformation

❯ Access architected blends with the full spectrum of Pentaho Analytics

❯ Manage governance and security of data for on-going accuracy

Accurate, Blended Big Data Analytics

EDW

Existing ETL Tool

or PDI Custom

er

Billing

Provisioning

NoSQL Network

Location

PDI

PDI Analytics

Just in time blending


Evolving Big Data Architectures

Just-in-Time Integration P

D

I

PDI

Analytic

DB

Location

Web

Social Media

Network

Existing

Process

or PDI Hadoop

Cluster

NoSQL

Existing

ETL Tool

or PDI

EDW Data

Marts

Analytics

Existing

ETL Tool

or PDI

Customer

Provisioning

Billing

Other BI Tools


Improve operational effectiveness

• Machines/sensors: predict failures, network attacks

• Financial risk management: reduce fraud, increase security

Reduce data warehouse cost

• Integrate new data sources without increased database cost

• Provide online access to ‘dark data’

Drive incremental revenue

• Predict customer behavior across all channels

• Understand and monetize customer behavior

• Begin to monetize data as a service

Customer Value from Big Data

MONETIZING BIG DATA-DRIVEN USE CASES DRIVING NEED TO BLEND DATA


Analytics

Analyze quality of service: • Network outages

• Dropped calls

• Poor quality

• Calls to support center

For profiles of customers: • Up for renewal

• Profitable

• Multiple agreements/services

• In competitive area

Determine best action to take: • Billing Credit

• Customer Coupon

• No Action

EDW

Existing

ETL

Tool

or PDI

Customer

Billing

Provisioning

Customer Financial Data:

• Billing

• Payment

• Usage

NoSQL Network

Location

PDI

Customer Experience Data:

• Outages

• Drops

• Service Quality

PDI

Blend revenue-related and

quality-of-service data

together to find customers at

risk

Why Blending at the Source Matters Customer Experience Analytics for Loyalty and Revenue


Optimisation Data Warehouse

Data Sources Big Data Architecture

Data Warehouse (Master & Transactional Data)

ERP

CRM

CDR

Analytic Data Mart(s)



Logs Logs

Other Data

Raw Data

Parsed Data

Analytic Datasets

Master Data

Tape

Archive


Our Big Data solutions B

ig D

ata

Mgm

t.

NoSQL Databases

• Data Integration • Job Orchestration • Workflow


• Scheduling • High Performance • Visual IDE

Dat

a In

tegr

atio

n

Analytic Databases Hadoop Java MapReduce, Pig Pentaho MapReduce

Big

An

alyt

ics

3RD Party Tools

• R • 3rd Party BI Tools • Applications

Business Analytics Embedded Analytics


MongoDB & Pentaho

../../Pentaho_Corp/Pre_Sales/MongoDB/MongoPentaho_Infographic.pdf


OEM & ISV


OEM pattern

Pentaho BI Server

Your Application

Pentaho

Your functions

Your application

Pentaho components


OEM

Bundled Mashup Extended Embedded

Value Fastest Way to Get Analytics that Have

Your Look & Feel

An Integrated Experience for Yours

End User

Customizing Pentaho for Your

Experience

Ultimate Integration and Customization

What it Takes?

• Pentaho is a separate app, branded with Partner’s logo, look & feel

• Optional: Partner app may include links to Pentaho reports, analysis and dashboards (popping new window)

• Optional: Single sign-on creates a seamless experience

• Pentaho & Partner app have the same UI • Pentaho User Console, or individual reports, analysis or dashboards are included in partner app

• Single sign-on creates a seamless experience

• Pentaho’s core functionality is extended through plug-ins. Examples: - Connecting to custom data sources - Adding new visualizations - Customizing security - Replacing Pentaho rules engine

• Integrate with Partner’s App Server

• Directly embedding Pentaho into your app

• Calling Pentaho Java APIs from your App

Skill Level • Limited HTML skills • HTML skills • HTML skills • Java skills

• HTML skills • Java skills • Knowledge of Pentaho architecture


COMPLETE ISOLATION OF ALL CONTENT INCLUDING:

ARTIFACTS (REPORTS, DASHBOARDS, ETC.)

DATA SOURCES

SCHEDULER

PLUGINS?

“VIRTUAL SERVER PER ORGANIZATION”

Pentaho BI Server

Use Case – Share Nothing

Artifacts

Data Sources

Schedules

Configuration

Artifacts

Data Sources

Schedules

Configuration

Organizations

Company B Company A


•ONE OR MORE COMMON DATABASES

•DATA ‘STRIPED’ WITH TENANT ID

Pentaho BI Server

Use Case – Shared Data, tagged with Tenant ID

Artifacts

Data Sources

Schedules

Configuration

Artifacts

Data Sources

Schedules

Configuration

Organizations

Company B Company A

Shared Database


•COMMON DATA SOURCE DEFINITION

•ORGANIZATION DATA IN ISOLATED DATABASES WITH THE SAME DATA

MODEL

•CONNECTIONS ‘ROUTED’ BY TENANT ID

Pentaho BI Server

Use Case – Shared Data Source (data isolation)

Artifacts

Data Sources

Schedules

Configuration

Artifacts

Data Sources

Schedules

Configuration

Organizations

Company B Company A

Data Sources


•COMMON PENTAHO ARTIFACTS – REPORTS, DASHBOARDS, ANALYSIS

•EACH TENANT CAN VIEW SHARED AND TENANT SPECIFIC CONTENT

Pentaho BI Server

Use Case – Shared Content

Artifacts

Data Sources

Schedules

Configuration

Artifacts

Data Sources

Schedules

Configuration

Organizations

Company B Company A

Artifacts


DATAMINING


Class Photo

…And that, in simple terms, is how data mining works.



Full data mining lifecycle support Preparation of input data

Statistical evaluation of data mining models

Visualization of inputs and results of model learning

Visualization, classification and clustering capabilities

118 classification/regression algorithms

11 clustering algorithms

Integrated with PDI ETL Execute Weka and R predictive models inside of a PDI transformation

Append probabilities dynamically to each row in the data flow

Retrain Weka models using the KnowledgeFlow plugin for PDI

Advanced Analytical



3 Calling Plans

• Nationwide

• PAYG

• Prepaid 50

2 Business

Units

• B2B

• B2C

7 Retail Stores

7 Product

Lines

3 Websites

Clear Wireless – Wireless Carrier

• San Francisco • Boston • NYC • Paris • Tokyo • Sydney • London

• Smartphones • Home Phones • Wifi Devices • Modems • Notebooks • Tablets • Accessories

• Ecommerce Site • Reseller Portal • Manufacturer Portal

Call Detail Records

Retail Sales

Website Clickstream

Databases



Increase subscription revenue

Improve store profitability

Eliminate inventory stock outs

Leverage big data to maximize profits

Profile and target profitable customers

Improve supply chain visibility for partners



93

Goals Objectives Enabler

Increase subscription revenue

Analyze call data to upsell PAYG customers to subscriptions

Improve store profitability

Hold store managers accountable by pushing store income statements to email

Eliminate inventory stock outs

Empower store employees with iPads and real-time inventory reports

Profile and target profitable customers

Profile mobile plan customers with high average call duration

Leverage big data to maximize profits

• Analyze e-commerce clickstream data in MongoDB to profile purchasing users

• Use predictive technologies to improve marketing effectiveness

Improve supply chain visibility for partners

Give phone manufacturers and resellers web access to secure sales reports


Data Mining Lifecycle

Phases

a visual guide to CRISP-DM methodology

SOURCE CRISP-DM 1.0

http://www.crisp-dm.org/download.htm

DESIGN Nicole Leaper

http://www.nicoleleaper.com

Generic Tasks

Specialized Tasks

(Process Instances)

Determine Business

Objectives

Background

Business Objectives

Business Success Criteria

(Log and Report Process)

Assess Situation

Inventory of Resources,

Requirements, Assumptions,

and Constraints

Risks and Contingencies

Terminology

Costs and Benefits


Determine Data Mining

Goals

Data Mining Goals

Data Mining Success Criteria


Produce Project Plan

Project Plan

Initial Assessment of Tools and

Techniques


Collect Initial Data

Initial Data Collection Report


Describe Data

Data Description Report


Explore Data

Data Exploration Report


Verify Data Quality

Data Quality Report


Data Set

Data Set Description


Select Data

Rationale for Inclusion/

Exclusion


Clean Data

Data Cleaning Report


Construct Data

Derived Attributes

Generated Records


Integrate Data

Merged Data


Format Data

Reformatted Data


Select Modeling

Technique

Modeling Technique

Modeling Assumptions


Generate Test Design

Test Design


Build Model Parameter

Settings

Models

Model Description


Assess Model

Model Assessment

Revised Parameter


Evaluate Results

Align Assessment of Data

Mining Results with



Approved Models

Review Process

Review of Process


Determine Next Steps

List of Possible Actions

Decision


Plan Deployment

Deployment Plan


Plan Monitoring and

Maintenance

Monitoring and

Maintenance Plan


Produce Final Report

Final Report

Final Presentation


Review Project

Experience

Documentation


Modeling

manipulate data and

draw conclusions

Evaluation

evaluate model and

conclusions

Deployment

apply conclusions to

business

Business Understanding

identify project objectives

Data Understanding

collect and review data

Data Preparation

select and cleanse data

Data Mining Life Cycle



WEBSITE CLICKSTREAM DATA

INCREASE REVENUE THROUGH TARGETED MARKETING

LIMITED DIRECT MARKETING BUDGET

PREDICT A WEB USER’S PROPENSITY TO PURCHASE BASED ON THEIR ONLINE CLICKSTREAM BEHAVIOR

SEND EXPENSIVE PROMOTIONAL OFFERS TO WEB USERS MOST LIKELY TO MAKE A PURCHASE

ASSUMPTIONS: $5/MAILING FOR $500 PURCHASE

❯ Expected Benefit of true positive prediction: $495 ($500 – mailing cost)

❯ Expected Benefit of false negative: $0 (no gain & no loss)

❯ Expected Cost of a false positive: $5 (cost of mailing)


Data Understanding Phases


SOURCE CRISP-DM 1.0




Generic Tasks

Specialized Tasks

(Process Instances)

Determine Business

Objectives

Background

Business Objectives



Assess Situation



and Constraints


Terminology

Costs and Benefits



Goals

Data Mining Goals




Project Plan


Techniques





Describe Data



Explore Data



Verify Data Quality

Data Quality Report


Data Set



Select Data


Exclusion


Clean Data



Construct Data

Derived Attributes

Generated Records


Integrate Data

Merged Data


Format Data

Reformatted Data


Select Modeling

Technique

Modeling Technique




Test Design



Settings

Models

Model Description


Assess Model

Model Assessment

Revised Parameter


Evaluate Results


Mining Results with



Approved Models

Review Process

Review of Process




Decision


Plan Deployment

Deployment Plan


Plan Monitoring and

Maintenance

Monitoring and

Maintenance Plan



Final Report

Final Presentation


Review Project

Experience

Documentation


Modeling

manipulate data and

draw conclusions

Evaluation

evaluate model and

conclusions

Deployment


business



Data Understanding


Data Preparation



Use ecommerce website clickstream log data stored in a MongoDB database

Key Source Fields: [id_user], [date], [event_name]

Events


Data Understanding/Data Preparation

WEBSITE CLICKSTREAM DATA USE WEBSITE CLICKSTREAM LOG DATA STORED IN MONGODB

NEED TO AGGREGATE, TRANSFORM, AND ENRICH THAT DATA

NEED TO PIVOT THE DATA INTO TABULAR FORMAT FOR PREDICTIVE MODELS

CREATE A FILE IN AARF FORMAT FOR PENTAHO DATA MINING

USE PENTAHO DATA MINING TO DERIVE “PROPENSITY SCORE” FOR EACH USER

Phases


SOURCE CRISP-DM 1.0




Generic Tasks

Specialized Tasks

(Process Instances)

Determine Business

Objectives

Background

Business Objectives



Assess Situation



and Constraints


Terminology

Costs and Benefits



Goals

Data Mining Goals




Project Plan


Techniques





Describe Data



Explore Data



Verify Data Quality

Data Quality Report


Data Set



Select Data


Exclusion


Clean Data



Construct Data

Derived Attributes

Generated Records


Integrate Data

Merged Data


Format Data

Reformatted Data


Select Modeling

Technique

Modeling Technique




Test Design



Settings

Models

Model Description


Assess Model

Model Assessment

Revised Parameter


Evaluate Results


Mining Results with



Approved Models

Review Process

Review of Process




Decision


Plan Deployment

Deployment Plan


Plan Monitoring and

Maintenance

Monitoring and

Maintenance Plan



Final Report

Final Presentation


Review Project

Experience

Documentation


Modeling

manipulate data and

draw conclusions

Evaluation

evaluate model and

conclusions

Deployment


business



Data Understanding


Data Preparation




Data Understanding/Data Preparation Phases


SOURCE CRISP-DM 1.0




Generic Tasks

Specialized Tasks

(Process Instances)

Determine Business

Objectives

Background

Business Objectives



Assess Situation



and Constraints


Terminology

Costs and Benefits



Goals

Data Mining Goals




Project Plan


Techniques





Describe Data



Explore Data



Verify Data Quality

Data Quality Report


Data Set



Select Data


Exclusion


Clean Data



Construct Data

Derived Attributes

Generated Records


Integrate Data

Merged Data


Format Data

Reformatted Data


Select Modeling

Technique

Modeling Technique




Test Design



Settings

Models

Model Description


Assess Model

Model Assessment

Revised Parameter


Evaluate Results


Mining Results with



Approved Models

Review Process

Review of Process




Decision


Plan Deployment

Deployment Plan


Plan Monitoring and

Maintenance

Monitoring and

Maintenance Plan



Final Report

Final Presentation


Review Project

Experience

Documentation


Modeling

manipulate data and

draw conclusions

Evaluation

evaluate model and

conclusions

Deployment


business



Data Understanding


Data Preparation



Key Source Fields: [id_user], [date], [event_name]

PARSE CLEAN AND FORMAT

GROUP AND AGGREGATE ENRICH w OTHER DATA



USE PDI ROW DENORMALIZER STEP TO PREPARE DATE FOR PREDICTIVE MODEL

• Pivots events from records into columns (tabular format)

Phases


SOURCE CRISP-DM 1.0




Generic Tasks

Specialized Tasks

(Process Instances)

Determine Business

Objectives

Background

Business Objectives



Assess Situation



and Constraints


Terminology

Costs and Benefits



Goals

Data Mining Goals




Project Plan


Techniques





Describe Data



Explore Data



Verify Data Quality

Data Quality Report


Data Set



Select Data


Exclusion


Clean Data



Construct Data

Derived Attributes

Generated Records


Integrate Data

Merged Data


Format Data

Reformatted Data


Select Modeling

Technique

Modeling Technique




Test Design



Settings

Models

Model Description


Assess Model

Model Assessment

Revised Parameter


Evaluate Results


Mining Results with



Approved Models

Review Process

Review of Process




Decision


Plan Deployment

Deployment Plan


Plan Monitoring and

Maintenance

Monitoring and

Maintenance Plan



Final Report

Final Presentation


Review Project

Experience

Documentation


Modeling

manipulate data and

draw conclusions

Evaluation

evaluate model and

conclusions

Deployment


business



Data Understanding


Data Preparation



Data Output: • Parsed • Cleaned; Formatted • Grouped; Aggregated • Enriched



The learning algorithms require each row of data to be an independent example of what is to be learned. This allows us to provide an example which summarizes all of the possible user's events over the hour in a single record. This “presence/absence” of event types are then used as predictors for "add to cart".

Why do we have to pivot the records into tabular format?


Data Preparation – Final Step

• Create a file in Weka’s ARFF format


Modeling

Phases


SOURCE CRISP-DM 1.0




Generic Tasks

Specialized Tasks

(Process Instances)

Determine Business

Objectives

Background

Business Objectives



Assess Situation



and Constraints


Terminology

Costs and Benefits



Goals

Data Mining Goals




Project Plan


Techniques





Describe Data



Explore Data



Verify Data Quality

Data Quality Report


Data Set



Select Data


Exclusion


Clean Data



Construct Data

Derived Attributes

Generated Records


Integrate Data

Merged Data


Format Data

Reformatted Data


Select Modeling

Technique

Modeling Technique




Test Design



Settings

Models

Model Description


Assess Model

Model Assessment

Revised Parameter


Evaluate Results


Mining Results with



Approved Models

Review Process

Review of Process




Decision


Plan Deployment

Deployment Plan


Plan Monitoring and

Maintenance

Monitoring and

Maintenance Plan



Final Report

Final Presentation


Review Project

Experience

Documentation


Modeling

manipulate data and

draw conclusions

Evaluation

evaluate model and

conclusions

Deployment


business



Data Understanding


Data Preparation




Modeling

BUSINESS GOAL: INCREASE REVENUE THROUGH TARGETED MARKETING

❯ Would like to focus on who is most likely to purchase and try to persuade them to spend more (or perhaps commit to spending if they are borderline) with seductive marketing, special offers etc.

DATA MINING

❯ Build a model that will predict “Added_item_to_cart” with higher accuracy than random selection or any hand-crafted business rules

❯ Based on data characteristics – few attributes; small number of instances – try some “likely suspects” first

❯ Naïve Bayes (linear)

❯ Logistic regression (linear)

❯ Decision tree (non-linear)

❯ K nearest neighbors (non-linear)


Modeling

• Take a quick look at summary info in the Explorer

Phases


SOURCE CRISP-DM 1.0




Generic Tasks

Specialized Tasks

(Process Instances)

Determine Business

Objectives

Background

Business Objectives



Assess Situation



and Constraints


Terminology

Costs and Benefits



Goals

Data Mining Goals




Project Plan


Techniques





Describe Data



Explore Data



Verify Data Quality

Data Quality Report


Data Set



Select Data


Exclusion


Clean Data



Construct Data

Derived Attributes

Generated Records


Integrate Data

Merged Data


Format Data

Reformatted Data


Select Modeling

Technique

Modeling Technique




Test Design



Settings

Models

Model Description


Assess Model

Model Assessment

Revised Parameter


Evaluate Results


Mining Results with



Approved Models

Review Process

Review of Process




Decision


Plan Deployment

Deployment Plan


Plan Monitoring and

Maintenance

Monitoring and

Maintenance Plan



Final Report

Final Presentation


Review Project

Experience

Documentation


Modeling

manipulate data and

draw conclusions

Evaluation

evaluate model and

conclusions

Deployment


business



Data Understanding


Data Preparation




Knowledge Flow


Evaluation

Phases


SOURCE CRISP-DM 1.0




Generic Tasks

Specialized Tasks

(Process Instances)

Determine Business

Objectives

Background

Business Objectives



Assess Situation



and Constraints


Terminology

Costs and Benefits



Goals

Data Mining Goals




Project Plan


Techniques





Describe Data



Explore Data



Verify Data Quality

Data Quality Report


Data Set



Select Data


Exclusion


Clean Data



Construct Data

Derived Attributes

Generated Records


Integrate Data

Merged Data


Format Data

Reformatted Data


Select Modeling

Technique

Modeling Technique




Test Design



Settings

Models

Model Description


Assess Model

Model Assessment

Revised Parameter


Evaluate Results


Mining Results with



Approved Models

Review Process

Review of Process




Decision


Plan Deployment

Deployment Plan


Plan Monitoring and

Maintenance

Monitoring and

Maintenance Plan



Final Report

Final Presentation


Review Project

Experience

Documentation


Modeling

manipulate data and

draw conclusions

Evaluation

evaluate model and

conclusions

Deployment


business



Data Understanding


Data Preparation




Logistic Regression: Results

MODEL IS A LINEAR FUNCTION THAT PREDICTS THE LIKELIHOOD

(PROBABILITY) OF A PERSON “ADDING ITEM TO CART”

RELATIVE MAGNITUDES OF THE COEFFICIENTS GIVE AN

INDICATION OF IMPORTANCE

Function for label “0” – i.e. wont add to cart


Logistic Regression: Results

Negative Impact on Purchase Positive Impact on Purchase

Subscribed to Email Gender - Male

Commented on Blog Language – French

Visited Site Language – Spanish

Signup Newsletter Primary Use - Personal

Watched Video Referring URL - Ebay

Tweeted Item Referring URL - Google

Tweeted Blog Posts Referring URL – Live.com

Signup Free Offer


Cost/Benefit Analysis

LOGIBOOST LOGISTIC

REGRESSION

LEFT CURVE:

CUMULATIVE GAINS

RIGHT CURVE: BENEFIT

CURVE

❯ y axis: benefit

❯ x axis: sample size

EXPECTED BENEFIT $ 21,475

❯ Also shows

expected benefit if

we just chose a

random subset of

this size from the

total pop.

❯ $ -3,036


Cost/Benefit Analysis

ALGORITHM: LOGIBOOST LOGISTIC REGRESSION

EXPECTED BENEFITS IF WE USE MACHINE RECOMMENDED MAILER

RECIPIENTS

❯ $ 21,475

EXPECTED BENEFIT IF WE JUST CHOSE A RANDOM SUBSET OF THIS

SIZE FROM THE TOTAL POP.

❯ $ (-3,036)


Modeling

Phases


SOURCE CRISP-DM 1.0




Generic Tasks

Specialized Tasks

(Process Instances)

Determine Business

Objectives

Background

Business Objectives



Assess Situation



and Constraints


Terminology

Costs and Benefits



Goals

Data Mining Goals




Project Plan


Techniques





Describe Data



Explore Data



Verify Data Quality

Data Quality Report


Data Set



Select Data


Exclusion


Clean Data



Construct Data

Derived Attributes

Generated Records


Integrate Data

Merged Data


Format Data

Reformatted Data


Select Modeling

Technique

Modeling Technique




Test Design



Settings

Models

Model Description


Assess Model

Model Assessment

Revised Parameter


Evaluate Results


Mining Results with



Approved Models

Review Process

Review of Process




Decision


Plan Deployment

Deployment Plan


Plan Monitoring and

Maintenance

Monitoring and

Maintenance Plan



Final Report

Final Presentation


Review Project

Experience

Documentation


Modeling

manipulate data and

draw conclusions

Evaluation

evaluate model and

conclusions

Deployment


business



Data Understanding


Data Preparation




Marketing

HOW MANY MARKETING PEOPLE DOES

IT TAKE TO SCREW IN A LIGHTBULB?

None….they’ve automated it.


YOUR PROJET


Project method

Prepare Explore Designe Develop Deployed

Installation et

Configuration

Formation Agile BI

Etudes & Infrastructure

Architecture

Kick-off projet Revue de développement

Créer du contenu

Explorer

les données

Identifier les besoins

Revue itérative

Publier

Développer le contenu

Extraire et

charger les

données

Affiner le modele de données

Tester, recetter et déployer

Collecte besoins métier

Etendre

Mise en production

Réunion Go/NoGo

Définition Projet

Suite BI et formations avancées

Planning Projet

Plan Projet

Modèle de données

Cahier des charges

Procédure de mise en

production

Specification

Plans test

Formation et documentation

utilisateurs

Revue itérative


How are you going to manage your project?

Vos efforts

Assistance Pentaho

Ask yourself: What resources do we have?

What competences do we have? What is our project timeline?

What’s our project complexity? Do we have the infrastructure?


Thanks

blog.pentaho.com

@Pentaho

Facebook.com/Pentaho


JOIN THE CONVERSATION. YOU CAN FIND US ON:

Pentaho Dev Day 30th October 2013 - Meetupfiles.meetup.com/1804355/Pentaho_Zaponet_DevDay... · 1...

Documents

Transcript of Pentaho Dev Day 30th October 2013 - Meetupfiles.meetup.com/1804355/Pentaho_Zaponet_DevDay... · 1...