HADOOP & SAS Data Loader for HADOOP...Nov 18, 2014 · SAS Forum Twitter Contest –Tweet to win...

Turning Data into Value

Hadoop & SAS Data Loader for HadoopSebastiaan SchaapFrederik Vandenberghe

What’s Hadoop

SAS Data management:Traditional In-Database In-Memory

The Hadoop analytics lifecycle

SAS Data Loader for Hadoop

Agenda

Market trends

TODAY $115

2010 $270

2005 $3,720

2000 $33,000

1995 $3,360,000

1990 $33,600,000

1985 $315,000,000

1980 $1,312,500,000

How much does this drive

Tech trends

4.17 hr.

2.5 min

15 sec

1 disk

100 disks

1000 disks

How long does it take to

read 3 TB?

What is it?

• Distributed processing of large data sets across clusters of computers using simple programming models

• Single or multiples machines

• Data processing framework and a distributed file system for data storage (HDFS)

Data Store

In-Database

Data Store

Traditional SAS

Even with In-Database processing there will still be some

work performed on the SAS server

Traditional vs. In-Database vs. In-memory

These approaches are complementary & can be combined for maximum effect

Data Store

In-Memory

MemoryData

Even with In-Database processing there will still be some

work performed on the SAS server

SAS and Hadoop

SAS accesses and extracts data from Hadoop to a SAS server for processing,

and writes results back

SAS processes data directly in the Hadoop cluster

SAS accesses and processes Hadoop data on SAS Servers while keeping the data and

computations massively parallel

IDENTIFY /

FORMULATE

PROBLEM

PREPARATION

EXPLORATION

TRANSFORM

& SELECT

VALIDATE

DEPLOY

EVALUATE /

MONITOR

RESULTS

SAS Visual Analytics

SAS Visual Statistics

SAS In-Memory Statistics for Hadoop

Done using either the Data

Preparation, Data Exploration

or Build Model Tools

SAS High Performance Analytics Offerings

supported by relevant clients like SAS

Enterprise Miner, SAS/STAT etc.

Done using the Build Model

Tools and other checks

SAS Scoring Accelerator for Hadoop

SAS Code Accelerator for Hadoop

SAS Visual Analytics

SAS/Access to Hadoop

SAS DI & Federation Server

SAS ESP

SAS Data Loader

SAS DI & Federation Server

SAS Data Loader

SAS DQ Accelerator for Hadoop

SAS Code Accelerator for Loader SAS/Access to Hadoop

MANAGE

DEVELOP

MODELS

• SAS/ACCESS

• SAS Data Management

• SAS Federation Server

• SAS Event Stream Processing

• SAS Data Loader for Hadoop SAS Data Quality Accelerator for

Hadoop

SAS Code Accelerator for Hadoop

• SAS Data Loader for Hadoop

• SAS Visual Analytics

• SAS In-memory Statistics for Hadoop

• SAS Scoring Accelerator

for Hadoop

• SAS Decision Manager

• SAS Visual Analytics

• SAS High Performance Analytics Products

• SAS Visual Statistics

• SAS In-memory Statistics for Hadoop

SAS Data Management Platform works seamlessly across Hadoop

Web Based DM

interface for

Hadoop

Self-service data

manipulation in

Hadoop + Loading

into LASR

SAS/Access to Hadoop, SAS/Access to Impala,

BASE SAS, SAS Federation Server

Access to HDFS,

Hadoop scripting (Pig,

Map Reduce…) and

HIVE/Cloudera Impala

through SAS coding and

GUI + Reuse of DQ and

ETL/ELT processing

SAS DI Studio

On-Hadoop data processing

Data virtualization &

masking across Hadoop

and other data stores

All other DM Clients

Hadoop Accelerated Clients BAU SAS DM clients

Third party clients +

SAS BI + SAS

Analytics + SAS

Solutions

Other clients

Bring streaming data from various sources

into Hadoop and/or the RDBMS or generate

events before data hits downstream store

SAS Data Loader for Hadoop

• Code Accelerator for Hadoop

• Data Quality Accelerator for Hadoop

• Data Loader, the UI

Scoring Accelerator for Hadoop• Separately licensed product

SAS IN-DATABASE FOR HADOOP

Sas® data loader for Hadoop

a new SAS Web-based Business user interface

Point & Click

User Menus

Little or no Hadoop

experience neededSelf-Service UI HTML 5 Interface

Enables Self-Service approach to managing data in Hadoop environment

Web Based Data Management interface for Hadoop

Capabilities

Browser-based + point and click

No knowledge of Hadoop or SAS is required)

Access and view data in Hadoop

Query, filter, transform, summarize the data

Load data into tables as well as SAS LASR

SAS Data Quality Accelerator for Hadoop

Benefits

self-service approach

enable the casual user

Improve data quality

Minimize movement of data

SAS Data Quality Accelerator for Hadoop and SAS Code Accelerator for Hadoop run in the Hadoop cluster

SAS® Data Loader for HadoopWhat is it?

Web-based interface

• Easy-to-use

• HTML5

Execute code on the Hadoop cluster• DS2, Hive and Data Quality

Load data into SAS LASR server

Non-IT or Business person

Easy to configure (small configuration list)

SAS® Data Loader for HadoopWhat is it?

vApp What is a vApp

vApp stands for virtual Application

Fully functional appliance containing a specific set of

SAS Software

Plug-and-Play environment

Some vApp examples : SAS University Edition, SAS Data

Loader and Visual Analytics 6,2 (Cloud only)

Storage

Network

vApp Ledger

SAS Solution

vAppHow does it integrate with the rest of the environment?

Metadata

Instructions/queries

Desktop

SAS Data Loader

For Hadoop

Registers Loaded

LASR tables only

Instructions

Laptop or desktop running Windows 7 (64-bit)

8 GB RAM minimum (16 GB preferred)

HyperThreading enabled in the BIOS (VT-x or AMD-v)

20 GB of free disk space

Capable of installing and running VMware 6 or 6+

Internet Explorer 9+, Firefox 14+, or Chrome 21+

Sas® data loader for Hadoop

Client-Side requirements

Installation

Pre-requisites Deploy Integrate Test

VMWare Player

SAS Software Depot

Hadoop Cluster

SAS Embedded Process

Firewall

Shared Folder

VM Configuration &

deploy

Startup

Apply SAS License

Application page

Hadoop configuration

inside the Data Loader

Optional : LASR

Configuration

Navigate in Hadoop

Do a transformation

Filter & query

Run SAS Code

Load to LASR

SAS® Data Loader for HadoopInstallation process

Key take-aways

SAS Data Management graphical user interfaces accelerate the

adoption of Hadoop

SAS Data Management provides the flexibility to work with Hadoop as

a new data store alongside traditional data stores using a single

platform

Existing SAS customers can leverage their SAS skills and existing data

management assets developed with SAS when using Hadoop

Big Data + Hadoop =

Big Data Collection for the technical user

Big Data + Hadoop + SAS =

Accessibility for everybody in the organization

• Business users consume the big Hadoop data

• Business analysts explore & visualize

• Data Scientists develop and deploy analytical models

Decisions built on fact based analytical insights into all of the data

NEW workshop SAS & Hadoop, getting the value out of Big Data

18 Nov. 2014

All details on www.sas.com/belux/training

SAS & Hadoop, getting the value out of Big Data

Big Data + Hadoop =

Big Data Collection for the technical user

Big Data + Hadoop + SAS =

Accessibility for everybody in the organization

• Business users consume the big Hadoop data

• Business analysts explore & visualize

• Data Scientists develop and deploy analytical models

Decisions built on fact based analytical insights into all of the data

NEW workshop SAS & Hadoop, getting the value out of Big Data

18 Nov. 2014

All details on www.sas.com/belux/training

SAS & Hadoop, getting the value out of Big Data

Twitter Contest – Tweet to win prizes!SAS Forum

A. HDFS & Hive

B. Cloudera & HDFS

C. HDFS & MapReduce

5. Which are the 2 core components of every Hadoop installation?

Tweet your answer:

Example: @spicyanalytics 5X

Prizes to win:

1st prize: a ticket for Analytics 2015

2nd prize: a book of Prof Bart Baesens: “Analytics in a big

data world”

3rd to 30th prize: chocolates with pepper

Winners will be contacted post-Forum !

Start of your tweet Question # Your answer

SAS® Data Loader for HadoopA new SAS Web-based Business user interface

Point & Click

User Menus

Little or no Hadoop

experience neededSelf-Service UI HTML 5 Interface

Enables Self-Service approach to managing data in Hadoop environment

SAS® Data Loader for HadoopTransform Data in Hadoop

Filtering RulesColumn

SelectionsAggregation

No coding, scripting or specialized skills required

SAS® Data Loader for HadoopQuery Hadoop data

Select

Source Tables

Apply Query

Criteria

See subset of data in

Table Viewer

Simple Drag & Drop approach to Query Data inside Hadoop

SAS® Data Loader for HadoopProfile Hadoop Data

Select

Source Table

View Reports in

Column Display

Run standard metrics on data inside Hadoop and generate reports

View Reports in

Table Display

View Data

SAS® Data Loader for HadoopCopy Data to distributed sas® lasr server

Select

Source Table

Explore Hadoop data quickly and easily for faster insights

Copy Data To distributed

SAS® LASR Servers

SAS® Visual Analytics

Optional

Visualize Data

HADOOP & SAS Data Loader for HADOOP...Nov 18, 2014 · SAS Forum Twitter Contest –Tweet to win...

Documents

Transcript of HADOOP & SAS Data Loader for HADOOP...Nov 18, 2014 · SAS Forum Twitter Contest –Tweet to win...

Hadoop Interacting with HDFS

Hadoop, HDFS, MapReduce and Pig

Hadoop Distributed File System (HDFS)eldawy/19FCS226/slides/CS226-03-HDFS.pdf · Hadoop Distributed File System (HDFS) 1. HDFS Overview A distributed file system Built on the architecture

Apache Hadoop - A Deep Dive (Part 1 - HDFS)

Hadoop & HDFS Final

HDFS and Hadoop

Hdfs 2016-hadoop-summit-san-jose-v4

Hadoop with Python · 2018. 7. 19. · Hadoop Distributed File System (HDFS) The Hadoop Distributed File System (HDFS) is a Java-based dis‐ tributed, scalable, and portable filesystem

Hadoop & HDFS for Beginners

Session2 - Hadoop Distributed File System...Hadoop Distributed File System (HDFS) What For Today!!! HDFS Features & Design Goals HDFS Operation Principle Data Locality, Rack Awareness

Hadoop Training #2: MapReduce & HDFS

Hadoop HDFS and Oracle

Tutorial Hadoop HDFS MapReduce

JerrinJoseph Hadoop ppt - · PDF fileRelies on principles of Distributed File System. HDFS have a Master-Slave ... Non-Posted Write by Hadoop HDFS OPERATION. HDFS ... Architecture

HDFS: Hadoop Distributed File Systemeecs.csuohio.edu/~sschung/cis612/LectureNotes_HadoopFinal_1.pdf · Hadoop Distributed File System (HDFS) p: HDFS • HDFS Consists of data blocks

Hadoop Distributed File System(HDFS) : Behind the scenes

Hadoop with Python - apphosting.io · 2016-10-11 · Hadoop Distributed File System (HDFS) The Hadoop Distributed File System (HDFS) is a Java-based dis‐ tributed, scalable, and

Hdfs 2016-hadoop-summit-dublin-v1

Strata + Hadoop World 2012: HDFS: Now and Future

What's hot: Big Data Analytics met Hadoop · Pig Hive Map Reduce HDFS Base SAS & SAS/ACCESS® to Hadoop™ In-Memory Data Access Pig Hive SAS® Data Management SAS® Visual Analytics