Online aggregation

Joseph M. Hellerstein University of California,

Berkley

Peter J. Haas IBM Research Division

Helen J. Wang University of California,

Berkley

Presented By:Arjav Dave

Overview• Introduction• Goals• Implementation• Performance Evaluation• Future Work

Problems with Aggregation• Aggregation is performed in batch mode: a query

is submitted, the system processes a large volume of data over a long period of time and then the final answer is returned.

• Users are forced to wait without feedback while the query is being processed.

• Aggregate queries are generally used to get a rough picture of a large body, yet they are computed with painstaking precision.

What is Online Aggregation and Why?• Permits users to observe the progress of their

aggregation.• Controls the execution on the fly.

Example• Consider the query:

select avg(final_grade) from grades where course_name=‘cse186’

• Without index the query will scan all records before returning the answer.

Example using online aggregation• The user can stop the query processing if the

result is within specified confidence interval.

Interface with groups• Consider a query with group by clause having 6

groups in output• There are 6 stop signs each for one group. They

can be used to stop the group processing.• Such an interface is easy for the non-statistical

users.

Usability Goals• Continuous Observation: Statistical, Graphical

and other intuitive interfaces. Interfaces must be extensible for each aggregate function.

• Control of Time/Precision: User should be able to terminate processing at any time controlling trade off between time and precision.

• Control of Fairness/Partiality: Users can control the relative rate at which different running aggregates are updated.

Performance Goals• Minimum time to Accuracy: Minimize time

required to produce useful estimate of the final answer.

• Minimize time to completion: Minimize the time required to produce the final answer.

• Pacing: The running aggregates should be updated at regular rates, without being so frequent that they overburden the user or user interface.

Building a system for Online AggregationTwo Approaches:•Naïve Approach•Modifying a DBMS

Naive Approach• Consider the query:

select running_avg(final_grade),running_confidence(final_grade),running_interval(final_grade) from grades;

• POSTGRES supports user defined function which can be defined for simple aggregates. For complex aggregates performance and functional issues arise.

Modifying a DBMS• Modify the database engine to support online

aggregation.

• Random access to Data:▫ Necessary to get the statistically meaningful estimates

of the precision of running aggregates.• Access Methods:

▫ Heap Scan▫ Index Scan▫ Sampling from Indices

Types of Access Methods• Heap Scan:

▫ Generally stored in random order, so easy to fetch▫ In clustered files there may be some logical order, so

choose other access method for queries over that attributes.

• Index Scan:▫ Returns tuples based on some attribute or in groups

based on some attribute.• Sampling from Indices:

▫ Ideal for producing meaningful confidence intervals. ▫ Less efficient than other two.

Fair, Non-Blocking Group By• Traditional technique does sorting and then

grouping. But sorting blocks• Instead hash the input relation on its grouping

columns. But does not perform better as the number of groups increases.

• Hybrid hashing or its optimized version Hybrid Cache can be used.

• For DISTINCT queries same technique can be used.

Index Striding• Update for the group with few members will be

very infrequent.• Index Striding: It uses Round-Robin technique to

fetch tuples from different groups fairly.• Advantages:

▫ Output is updated according to default or user settings.▫ Delivery of tuples of a group can be stopped, so delivery

of tuples for other groups is much faster.

Non-Blocking Join Algorithms• Sort-Merge Join: Sorting blocks.• Merge Join: Not acceptable for access methods

that fetch tuples in sorted order.• Hybrid Hash Join: Acceptable if inner relation is

small, as it blocks to hash the inner relation.• Pipeline Hash: Less Efficient than hybrid hash

join. But efficient if both the relations are large.• Nested-Loops join: Not useful for joins with large

inner relation non-indexed.

Optimization• Avoid Sorting

• Function to calculate cost for Blocking sub-operations (e.g. Hashing inner relation in hybrid hash join) according to processing time should be exponential.

• Maximize user control.

• Trade off need to be evaluated between the output rate and the time to completion.

Aggregate Functions• New aggregate functions must be defined to

return running confidence intervals.

• Query Executor must be modified to provide running aggregate values for display.

• An API must be provided to control the rate e.g. stopGroup, speedupGroup, slowDownGroup, setSkipFactor.

Running Confidence Intervals• Precision of running aggregates is given by

running confidence intervals• Three types:

▫ Conservative Confidence Interval▫ Large Sample Intervals▫ Deterministic Confidence Interval

Future Work• Enhancing User Interface• Support for Nested Queries• Checkpointing and Continuation• Tracking online queries

QUESTIONS?