Big Data: SQL on Hadoop - Introduction to Big SQL for SF Bay Area MeetUp, March 13, 2014
Big Data: Big SQL Application Development with BigInsights
-
Upload
cynthia-saracco -
Category
Technology
-
view
1.469 -
download
0
description
Transcript of Big Data: Big SQL Application Development with BigInsights
Developing, publishing and deploying your first Big SQL application with InfoSphere BigInsights
Cynthia Saracco
Senior Solution Architect
IBM Silicon Valley Lab
2 © 2013 IBM Corporation
Big Data Application Ecosystem
Eclipse
App library
MapReduce, "
Text Analytics
Query
App Development
• Code application program, and generate
associated App
• Deploy Apps to Enterprise ManagerApp
Development
Publish
Data integration scenario:
Pre-defined work flows simplify
loading data from various
sources
•Work flows can be configured,
deployed, executed and
scheduled
Development tooling:
•Text analytics
•MapReduce
•Query languages
• . . .
Application scenarios (web log,
email, social media, �):
• Samples provide starting
point, speed time to value
Big Data Web Console
3 © 2013 IBM Corporation
BigInsights Applications Catalog (Web Console)
� Browse available applications � Manage and deploy applications (administrators only) � Execute (or schedule execution of ) a deployed application � Monitor job (application) status� Link or chain applications for sequential execution
4 © 2013 IBM Corporation
Overview of Application Development Lifecycle
� Configure your Eclipse environment (one-time set up) � Develop your application using BigInsights tools � Test your application � Package and publish your application � Deploy your application on the cluster
5 © 2013 IBM Corporation
Configure your Eclipse environment
� One-time set up � Download and install BigInsights tools (Eclipse plug-ins)
– Welcome tab of BigInsights Web console includes pre-req info, download &
installation instructions
6 © 2013 IBM Corporation
Develop your application – Big SQL example
� Open the BigInsights perspective in Eclipse� Create a BigInsights project
7 © 2013 IBM Corporation
Develop your application – Big SQL example (cont’d)
� Create a BigInsights program with a SQL script (file)
8 © 2013 IBM Corporation
Develop your application – Big SQL example (cont’d)
� Populate your SQL file with the desired code
create table if not exists media_csv
(id integer not null,
name varchar(50),
url varchar(50),
contactdate string)
row format delimited
fields terminated by ','
stored as textfile;
load hive data local inpath
'/home/biadmin/sampleData/DBMS/RDBMS_data.csv'
-- overwrite
into table media_csv;
9 © 2013 IBM Corporation
Test your application
� Run your application from Eclipse
10 © 2013 IBM Corporation
Publish your application to the BigInsights catalog
� Package and publish your application from Eclipse � Specify application name, workflow requirements, etc.
11 © 2013 IBM Corporation
Deploy your application on the cluster
� Access the Applications tab of the Web console� “Manage” the published applications � Locate your new application and deploy it � Create credentials store file in DFS (if needed) -- see next chart � Optionally, execute the application after it’s been deployed
12 © 2013 IBM Corporation
Run your Big SQL application on the cluster
13 © 2013 IBM Corporation
Upgrade your application (optional)
� Satisfy evolving business requirements, improve flexibility – Example: add input parm(s)
� Modify SQL code and re-package, re-publish, re-deploy
create table if not exists $TABLE(id integer not null,
name varchar(50),
url varchar(50),
contactdate string)
row format delimited
fields terminated by ','
stored as textfile;
load hive data local inpath '$FILE' -- overwrite
into table $TABLE;
14 © 2013 IBM Corporation
Upgrade your application (cont’d)
� Re-publish your application from Eclipse � Adjust workflow specs for input parm(s)
15 © 2013 IBM Corporation
Summary
� Eclipse tools simplify big data application development for BigInsights– Wizards
– Context-sensitive help
– Oozie workflow generation
– Built-in test environment
– Etc.
� Application catalog provides easy way to locate and launch apps of interest– Developers use Eclipse tools to package/publish their applications to this
catalog
– Application upgrades easily managed