NiagaraCQ : A Scalable Continuous Query System for Internet Databases Jianjun Chen et al

Computer Sciences Dept. University of Wisconsin-Madison

SIGMOD 2000 Talk by S. Sudarshan

Continuous QueriesExample

Inform me when there is a new publication related to multi-query optimization

A broad classification Change based Timer based

NiagaraCQA CQ system for the InternetContinuous Queries on XML data setsScalable CQ processingIncremental group optimizationHandles both change based and timer based queries in a uniform way

NiagaraCQ command language

Creating a CQCreate CQ_name XML-QL queryDo action { START start_time} { EVERY

time_interval} { EXPIRE expiration_time}

Delete CQ_name

Expression SignatureQuery examples Where <Quotes><Quote><Symbol>INTC</></></> element_as $g in “http://www.stock.com/quotes.xml”

construct $g

Where <Quotes><Quote><Symbol>MSFT</></></> element_as $g in “http://www.stock.com/quotes.xml”

construct $g

Expression signatures

Quotes.Quote.Symbol in quotes.xml

constant

=

Query plansTrigger Action I Trigger Action J

File Scan

Select Symbol = “MSFT”

Select Symbol = “INTC”

File Scan

quotes.xml quotes.xml

GroupGroup Signature

Common signature of all queries in the group

Group constant table Constant_value

Dest_buffer

INTC Dest. I MSFT Dest. J

The group plan

Incremental Grouping AlgoWhen a new query is submitted

If the expression signature of the new query matches that of existing groups

Break the query plan into two partsRemove the lower partAdd the upper part onto the group

planelse create a new group

Query split with materialized intermediate files

Why not use a pipeline scheme ? Split operator may block simple queries Gives a single complicated execution plan A large portion of query plan may not need to be

executed at each invocation Does not work for grouping timer based queries

Using intermediate files Cut query plan into 2 parts at split operator Add a file scan operator to upper part to read

intermediate file Intermediate files are monitored just like other

data sources

The query split scheme

Trade-offsOther advantages of materialized intermediate files

Only the necessary queries are executed Uniform handling of intermediate files and

original data source files

Disadvantages Split operator becomes a blocking

operator Extra disk I/Os

Range PredicatesE.g. R.a < val or val1 < R.a < val2Multiple such rangesProblem

Intermediate files may contain duplicate tuples

Idea: Virtual intermediate files Use an index to implement this

Incremental grouping of selection predicates

Multiple selection predicates in a query CNF for predicates on same data source Incremental grouping

Choose the most selective conjunct and implement virtual file on this conjunct

Evaluation of other predicates Upper levels of continuous query

Example queryWhere <Quotes><Quote><Symbol>”INTC”</> <Current_Price>$p</></> element_as $g </> in “quotes.xml”, $p < 100Construct $g

Incremental grouping of join operators

A join queryQuotes.Quote.Change_Ratio constant in “quotes.xml”Where <Quotes><Quote><Symbol>$s</></>

element_as $g </> in “quotes.xml”,<Companies><Company><Symbol>$s</></>

element_as $t</> in “companies.xml”construct $g, $t

Queries that contain both join and selection

Example query :Where <Quotes><Quote><Symbol>$s</><Industry>”Computer Service”</></>element_as $g </> in “quotes.xml”,<Companies><Company><Symbol>$s</></>element_as $t</> in “companies.xml”construct $g, $t

Where to place the selection operator ? Below the join

Eliminates irrelevant tuples Above the join

Allows sharing Pick based on cost model

Grouping timer-based queries

Challenge Sharing common computation

Event List Stores time events sorted in time order

Incremental evaluationInvoke queries only on changed data

For each source file, NiagaraCQ keeps a delta file Also for the intermediate files Time stamp store the each tuple

Incremental evaluation of join operators requires complete data files

Memory CachingThousands of continuous queries can’t fit in memoryWhat should we cache ?

Grouped query plans What about non-grouped queries ?

Favor small delta files Front part of the event list

System Architecture

CQ processing

Experimental ResultsExample query :

Where <Quotes><Quote><Symbol>”INTC”</></>element_as $g </> in “quotes.xml”, construct $g

N = number of installed queriesF= number of fired queriesC = number of tuples modified

Performance ResultsCase 1: F=N, C=1000Case 2: F=100, C=1000

Performance ResultsF=N=2000, vary data size

Thank You

OutlineGeneral strategy of incremental group optimizationQuery split with materialized intermediate filesIncremental grouping of selection and join operatorsSystem architectureExperimental results

Incremental group optimization

General Strategy

Why can’t we regroup all queries when a new query is added ?

Use of expression signatures for grouping

Same syntax structure Different constant values