Stream SQL, Rules, Subscriptions: It’s All The Same

Hans-Arno JacobsenBell University Laboratory Chair

Middleware Systems Research GroupUniversity of Toronto

MIDDLEWARE SYSTEMSRESEARCH GROUP

http://www.padres.msrg.utoronto.ca

Go PSQL !

Event Processing with PADRES• PADRES Event Language

– A1: [class = item], [shipID = *], [weight < 10]– P1: [class, item],[shipID, ACME123],[weight, 4]– S1: [class = item], [weight > 3]

• Initially implemented with JESS (i.e., as rules)– LHS statements encode atomic subscriptions– LHS encode composite subscriptions– RHS encode notification semantics (e.g., forwarding logic)

• IF LHS op LHS op LHS THEN RHS

• PADRES SQL (PSQL) modeled after SQL for event and event stream processing

PSQL Schema DefinitionCREATE TABLE (attr op val [, attr op val]* )

CREATE TABLE (class = invoke, service = *, item = *, id = *) Advertisement: [class, =, invoke],[service, = ,*],[item, =, *],[id, =, *]

CREATE TABLE (class = result, service = *, item = *, id = *, state = *) Advertisement: [class, =, result],[service, = ,*],[item, =, *], [id, =, *],[state,=,*]

PublishingINSERT ( attr [, attr]* ) VALUES ( val [, val]* )

INSERT (class, service, id) VALUES (invoke, ItemView, a0012)

Publication: [class,invoke],[service,ItemView],[id, a0012]

Subscribing

Querying and subscribing to dataSELECT [ attr | function]*

[ FROM event pattern ] WHERE [ attr op val ]*

[ GROUP BY attr* ][ HAVING function* ]

[ … ] clause is optional

We support AVG, MIN, MAX, COUNTas AVG(attr,N)

ExampleAtomic and hybrid subscription:

SELECT * WHERE class = invoke,

service = ItemView, item = T-Shirt, start_time = NOW – 1h,end_time = NOW + 1h

[class, =, invoke], [service, =, ItemView],[item, =, T-Shirt], [start_time, =, NOW – 1h], [end_time, =, NOW + 1h]

ExampleComposite and hybrid subscription

SELECT e2.service, e2.item, e2.id FROM e1 AND e2WHERE e1.class = result,

e1.service = ItemShipped,e1.state = 2, e2.class = invoke, e2.service = ItemView, e1.start_time = NOW – 2months, e1.item = e2.item

[class,=,result],[service, =, ItemShipped],[item,=,$X],[state, =, 2],[start_time, =, NOW – 2 months],AND[class,=, invoke],[service, =, ItemView],[item,=,$X]

Historic Event Query

DB Admin

Publisher

Subscriber

P1 P2

P3 P4 P5

Publication space P(A)

1

3

2

4

6

5P

S

• Partition the publication space• Assign partitions

Partition

1

2

3

458CANOE Summer School, Norway, 2009

Assigning A Partition

Advertisement Subscription Publication

DB Admin

Publisher

Subscriber

[class,DB_CONTROL],[command,STORE],[db,B3-DB],[class,DB_CONTROL],[command,STORE],[db,B3-DB],[partition_spec,`[class,=,reading],[level,>,0]`][partition_spec,`[class,=,reading],[level,>,0]`][class,=,reading],[level,>,0][class,=,reading],[level,>,0]

Partition

1

2

3

4

5

1 1

2

• Partitioning

• Partial replication

• Full replication

1

2

3 4

6

5P

S

[class,DB_CONTROL],[command,STORE],[db,B3-DB],[class,DB_CONTROL],[command,STORE],[db,B3-DB],[partition_spec,” [partition_spec,” [class,=,reading],[level,>,0] [class,=,reading],[level,>,0] ”]”]

9CANOE Summer School, Norway, 2009

Challenges & Obstacles Remaining

• Stream SQL ≈ Rules ≈ Subscriptions

• A common formal framework– A query algebra or calculus

• Simple (i.e., can be described in 2 pages or less)

• Algorithmic issues in processing expressions

The PSQL LanguageSchema or type definition

CREATE TABLE (attr op val [, attr op val]* )

CREATE TABLE (class = invoke, service = *, item = *, id = *) Adv: [class, =, invoke],[service, = ,*],[item, =, *],[id, =, *]

CREATE TABLE (class = result, service = *, item = *, id = *, state = *) Adv: [class, =, invoke],[service, = ,*],[item, =, *],[id, =, *],[state,=,*]

Produce data INSERT ( attr [, attr]* ) VALUES ( val [, val]* )

INSERT (class, service, id) VALUES (invoke, ItemView, a0012) Pub: [class,invoke],[service,ItemView],[id, a0012]

Retail Scenario

B

B

B

B

BCustomer

Customer

Customer

Web Services

Web Services

Manager

Delivery Services