High-level languages for Big Data Analytics (Presentation)
-
Upload
jose-luis-lopez-pino -
Category
Technology
-
view
1.258 -
download
7
description
Transcript of High-level languages for Big Data Analytics (Presentation)
BI Seminar project: High-level languages for Big Data AnalyticsJanani ChakkaradhariJose Luis Lopez Pino
Outline1. Introduction
1.1 The Map Reduce programming model1.2 Hadoop
2. High Level Languages2.1 Pig Latin2.2 Hive2.3 JAQL2.4 Other Languages
3. Comparison of HLLs3.1 Syntax Comparison3.2 Performance3.3 Query Compilation3.4 JOIN Implementation
4. Future Work4.1 Machine Learning4.2 Interactive Queries
5. Conclusion
Introduction
The MapReduce model• Introduced in 2004 by Google• This model allows programmers without any experience in
parallel coding to write the highly scalable programs and hence process voluminous data sets.
• This high level of scalability is reached thanks to the decomposition of the problem into a big number of tasks.• The Map function produces a set of key/value pairs, taking a
single pair key/value as input.• The Reduce function takes a key and a set of values related to this
key as input and it might also produce a set of values, but commonly it emits only one or zero values as output
The MapReduce model• Advantages:• Scalability• Handle failures and balance the system
• Pitfalls• Complicated to code some tasks.• Some tasks are very expensive.• Difficulties to debug the code.• Absence of schema and indexes.• A lot of bandwidth might be consumed.
Hadoop• An Apache Software foundation open source
project• Hadoop – HDFS + Map Reduce• DFS – Partitioning data & Storing in separate
machine• HDFS – Stores large files, running on commodity
clusters of hardware and typically 64 MB for per block• Both FS and Map reduce are Co-Designed
Hadoop• No separate storage network and processing network• Moving compute to the data node
High level languages
High level languages• Two different types• Created specifically for this model.• Already existing languages
• Languages present in the comparison• Pig Latin• HiveQL• Jaql
• Interesting languages• Meteor• DryadLINQ
Pig Latin• Executed over Hadoop.• Procedural language.• High level operations similar to those that we can find in SQL• Some interesting operators:• FOREACH to process a transformation over every tuple of the set.
To make possible to parallelise this operation, the transformation of one row should depend on another.
• COGROUP to group related tuples of multiple datasets. It is similar to the first step of a join.
• LOAD to load the input data and its structure and STORE to save data in a file.
Pig Latin• Goal: to reduce the time of development. • Nested data model.• User-defined functions• Analytic queries over text files (not need of loading the data)
• Procedural language -> control over the execution plan• The user can speed the performance up.• It makes easier the work of the query optimiser.• Unlike SQL.
HiveQL• Open-source DW
solution built on top of Hadoop• The queries looks
similar to SQL and also has extensions on it• Complex column
types- map, array, struct as data types• It stores the metadata
in RDBMs
HiveQL
• The Metastore acts as the system catalog for Hive• It stores all the information about the tables,
their partition, the schema and etc.,• Without the system catalog it is not possible to
impose a structure on hadoop files.• Facebook uses MySQL to store this metadata.
Reason: Since these information has to be served fast to the compiler
JAQL• What is Jaql?• Declarative scripting programming language.• Used over Hadoop’s MapReduce framework• Included in IBM’s InfoSphere BigInsights and Cognos Consumer Insight
products.• Developed after Pig and Hive.• More scalable.• More flexible• More reusable.
• Data model• Simple: similar to JSON.
• Values as trees.• No references.• Textual representation very similar.
• Flexible• Handle semistructured documents.• But also structured records validated against a schema.
JAQL• Control over the evaluation plan.• The programmer can work at different levels of abstraction
using Jaql's syntax:• Full definition of the execution plan.• Use of hints to indicate to the optimizer some evaluation
features.• This feature is present in most of the database engines that use SQL
as query language.• Declarative programming, without any control over the flow.
Other languages: Meteor• Stratosphere stack• Pact:
• Programming model• It extends MapReduce with new second-order functions• Cross: Cartesian product• CoGroup: group all the records with the same key and process them.• Match: similar to CoGroup but pairs with the same key could be processed
separately.
• Sopemo:• Semantically rich operator model• Extensible
• Meteor: query language• Optimization• Meteor code• Logical plan using Sopemo operators -> Optimized• Pact final program -> Physically optimized
Other languages: DryadLINQ• Coded embedded in .NET programming languages• Operators• Almost all the operators available in LINQ.• Some specific operators for parallel programming.• Develop can include their own implementations.
• DryadLINQ code is translated to a Dryad plan• Optimization• Pipeline operations• Remove redundancy• Push aggregations• Reduce network traffic
Comparison HLLs
Comparing HLLs
• Different design motivations• Developers preferences• Write concise code -> Expressiveness• Efficiency -> Performance
• Criteria that impact performance• Join implementation• Query processing
• Some other are not included• Language paradigm• Scalability
Comparison Criteria
• Expressive power• Performance• Query Compilation• JOIN Implementation
Expressive power
• Three categories by Robert Stewart:• Relational complete• SQL equivalent (aggregate functions)• Turing complete• Conditional branching• Indefinite iterations by means of recursion• Emulation of infinite memory model
Expressive power
• Three categories by Robert Stewart:• Relational complete• SQL equivalent (aggregate functions)• Turing complete• Conditional branching• Indefinite iterations by means of recursion• Emulation of infinite memory model
Expressive power
• Three categories by Robert Stewart:• Relational complete• SQL equivalent (aggregate functions)• Turing complete• Conditional branching• Indefinite iterations by means of recursion• Emulation of infinite memory model
Expressive power
Expressive power
• But this do not mean that they are SQL, Pig Latin and HiveQL are the same.• HiveQL• Is inspired by SQL but it does not support the full
repertoire included in the SQL-92 specification• Includes features notably inspired by MySQL and
MapReduce that are not part of SQL.• Pig Latin• It is not inspired by SQL.• For instance, do not have OVER clause
SQL Vs. HiveQL (2009)
SQL HiveQLTransactions Yes NoIndexes Yes NoCreate table as select Not SQL-92 YesSubqueries In any clause
Correlated or notOnly in FROM clauseOnly noncorrelated
Views Yes Not materializedExtension with map/reduce scripts
No Yes
Query Processing
Query Processing
• In order to make a good comparison we should have the basic knowledge on how these HLQL are working. • How the abstract user representation of the
query or the script is converted to map reduce jobs?
Query Processing – Pig Latin• The goal of writing
Pig Latin script is to produce an equivalent map reduce jobs that can be executed in the Hadoop environment• Parser first checks for
the syntactic errors
Query Processing – Pig Latin
Query Processing – Pig Latin
Query Processing - Hive
• It gets the Hive SQL string from the client• The parser phase converts it into parse tree
representation• The logical query plan generator converts it into
logical query representation. Prunes the columns early and pushes the predicates closer to the table.• The logical plan is converted to physical plan and
then map reduce jobs.
Query Processing - JAQL
• JAQL includes two higher order functions such as mapReduceFn and mapAggregate
• The rewriter engine generates calls to the mapReduceFn or mapAggregate
QP - Summary
• All these languages has its own methods • All supports syntax checking usually done by the
compiler• Pig currently misses out on optimized storage
structures like indexes and column groups• HiveQL provides more optimizations• it prunes the buckets that are not needed• Predicate push down
• Query rewriting is the future work of JAQL (Projection push-down )
JOIN Implementation
JOIN in Pig Latin
• Pig Latin Supports inner join, equijoin and outer join. The JOIN operator always performs inner join. • Join can also be achieved by COGROUP operation
followed by FLATTEN• JOIN creates a flat set of output records while
COGROUP creates a nested set of output records• GROUP – when only one relation • COGROUP – when multiple relations are involved• FLATTEN - (a, {(b,c), (d,e)}) (a, b, c) and (a, d, e)
JOIN in Pig Latin
• Fragment Replicate joins• Trivial case, only possible if one of two relations are
small enough to fit into memory• JOIN is in Map phase
• Skewed Joins• Not equally distributed data• Basically computes histogram of the key space and
uses this data to allocate reducers for a given key• JOIN in reduce phase
• Merge Joins• Only possible if the relations are already sorted
JOIN in Pig Latin• The choice of join strategy can be specified by the user
JOIN in Hive
• Normal map-reduce Join• Mapper sends all rows with the same key to a
single reducer• Reducer does the join
• SELECT t1.a1 as c1, t2.b1 as c2 FROM t1 JOIN t2 ON (t1.a2 = t2.b2);• Map side Joins• small tables are replicated in all the mappers
and joined with other tables
JOIN in JAQL
• Currently JAQL supports equijoin• The join expression supports equijoin of 2 or
more inputs. All of the options for inner and outer joins are also supported
joinedRefs = join w in wroteAbout, p in products
where w.product == p.nameinto { w.author, p.* };
JOIN - Summary
• Both Pig and Hive has the possibility to performs join in map phase instead of reduce phase
• For skewed distribution of data, the performance of JAQL for join is not comparable to other two languages
Performance
Benchmarks
• Pig Mix is a set of queries to test the performance. These set checks the scalability and latency• Hive’s benchmark is mainly based on the queries
that are specified by Pavlo et al (selection task, Aggregation task and a Join task)• Pig-Latin implementation for the TPC-H queries
and HiveQL implementation of TPC-H queries
Performance - Summary
• The paper describes Scale up, Scale out and runtime• For skewed data, Pig and Hive seems to be more
effective in handling it compared to JAQL runtime • Pig and Hive better in utilizing the increase in the
cluster size compared JAQL• Pig and Hive allows the user to explicitly specify
the number of reducers task• This feature has significant influence on the
performance
Machine Learning
• What page will the visitor next visit?• Twitter has extended Pig’s support of ML by
placing learning algorithms in Pig Storage functions• Hive - the machine learning is treated as UAFs• A new data analytics platform Ricardo is
proposed combines the functionalities of R and Jaql.
Interactive queries• One of the main problems of MapReduce all the languages built on top of this
framework (Pig, Hive, etc.) is the latency.• As a complement of those technologies, some new frameworks that allow
programmers to query large datasets in an interactive manner have been developed• Dremel by Google• The open source project Apache Drill.
• How to reduce the latency?• Store the information as nested columns.• Query execution based on a multi-level tree architecture.• Balance the load by means of a query dispatcher.
• Not too many details of the query language• It is based on SQL • It includes the usual operations (selection, projection, etc.) • SQL-like languages features: user define functions or nested subqueries• The characteristic that distinguish this languages is that it operates with nested
tables as inputs and outputs.
Conclusions• The MapReduce programming model have big pitfalls.• Each programming language try to solve some of these
disadvantages in a different way.• No single language beat all the other options.• Comparison• Jaql is expressively more powerful.• JAQL is at the lower level in case of performance when compared to
Hive and Pig• HiveQL and Pig Latin supports map phase JOIN.• HiveQL use more advanced optimization techniques for query
processing
• New technologies to solve those problems:• Languages: Dremel and Apache Drill• Libraries: Mahaut
Thank you very much!