Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators:...
-
Upload
lucinda-newman -
Category
Documents
-
view
215 -
download
0
Transcript of Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators:...
![Page 1: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/1.jpg)
Republishing Mechanisms for R-GMA
Benefits and Approaches.
Talk by: Alasdair Gray
Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt
Heriot-Watt University
![Page 2: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/2.jpg)
Summary
Current situation in R-GMA. Example of a republisher hierarchy
that users want. Problems of creating and
maintaining a hierarchy of republishers.
Other open issues.
![Page 3: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/3.jpg)
Current Situation in R-GMA: Primary Producers
Continuous Queries: Stream Producer. Resilient Stream Producer. Circular Buffer Producer (Deprecated).
Snapshot Query: Latest Producer. Canonical Producer.
History Query: Database (History) Producer. Canonical Producer.
![Page 4: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/4.jpg)
Current Situation in R-GMA: Republishers Currently called an Archiver. Constructed in two ways:
1. Archiver(rdms, user, pass)
Database Producer
1. Archiver(insertable) Stream Producer Latest Producer Database Producer
![Page 5: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/5.jpg)
Summary
Current situation in R-GMA. Example of a republisher hierarchy
that users want. Problems of creating and
maintaining a hierarchy of republishers.
Other open issues.
![Page 6: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/6.jpg)
Example Scenario Producers for cpuLoad running on individual machines.
These are very small streams (burns/brooks) of data. Aim: combine these burns to form larger streams. Three levels of views: SELECT * FROM cpuLoad
Producers of burns:
V1: WHERE country=‘britain’ AND loc=‘hw’ AND machineID=42
Confluent of burns at site level:
V2: WHERE country=‘britain’ AND loc=‘hw’Confluent of streams at national level:
V3: WHERE country=‘britain’
![Page 7: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/7.jpg)
Limits of Current R-GMA RepublishersIn the current R-GMA system we could not have: A stream republisher for V3 consuming from
V2. Forced to choose type of republisher when
created.
Why can’t V3 consume from V2? No mechanisms to make sure that:
1. Republishers don’t consume from themselves.
2. Loops of republishers are not created.
![Page 8: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/8.jpg)
The Scenario
Hierarchy of republishers. i.e. a republisher can consume from another republisher. A republisher cannot consume from itself. Based on cpuLoad example. Illustrates:
1. Difficulties that arise.2. Possible approaches.3. Benefits of creating hierarchies.
Assumptions:1. Republishers are complete with respect to their view
definition.2. All relevant producers are stream producers.
![Page 9: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/9.jpg)
The Set Up
country=‘britain’
NationalRepublisher
site=‘hw’
site=‘ral’
Local/siteRepublisher
Primary Producers
ral hw
Producer
Consumer
Key
![Page 10: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/10.jpg)
Summary
Current situation in R-GMA. Example of a republisher hierarchy
that users want. Problems of creating and
maintaining a hierarchy of republishers.
Other open issues.
![Page 11: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/11.jpg)
Question: How do we add a new producer?
country=‘britain’
site=‘hw’
site=‘ral’1. Site and national
republishers.
A new machine is added at ral.
Which consumers should be informed?There are two options:
or2. Site republisher
only.
ral hw
Producer
Consumer
Key
![Page 12: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/12.jpg)
Efficiency
Option 1: connect producer to all relevant republishers.
Easy to implement: simply find all relevant consumers and
start streaming.
Duplication of tuples. Option 2: connect producer to most specific republisher.
Provides performance gains due to: Lower load on new producer. Lower network bandwidth. No duplicate tuples (in general).
Requires more sophisticated logic.
![Page 13: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/13.jpg)
Issues of Implementing Option 2 How does the system know
which republishers to inform and which to ignore?
Which component makes this decision? The republisher agents. The consumer agents. The registry.
What information is needed to make this decision?
Where is this information stored?
country=‘britain’
site=‘hw’
site=‘ral’
ral hw
![Page 14: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/14.jpg)
What else happens if we choose option 2?
Need to consider the process of adding / removing an intermediary republisher.
Effects on links between producers and other republishers.
![Page 15: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/15.jpg)
Requirements for Republishers
No duplication of tuples. Duplicates cause a problem for:
Aggregation queries. Users performing statistical analysis.
Completeness issues:1. No tuples lost.
2. Republishes all tuples that conform to its view definition.
Tuples in chronological order of timestamps.
![Page 16: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/16.jpg)
Question: How do we add an intermediary republisher?
country=‘britain’
site=‘hw’
site=‘ral’
ral hwibm
![Page 17: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/17.jpg)
Question: How do we add an intermediary republisher?
country=‘britain’
site=‘hw’
site=‘ral’
site=‘ibm’
ral hwibm
![Page 18: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/18.jpg)
Steps Involved in Adding an Intermediary Republisher
Involves the following tasks: Creating the new republisher. Start the republisher consuming
from relevant producers. Start the republisher producing
tuples. Find relevant higher level
republishers. Remove any existing channels
between producers and higher level republishers.
country=‘britain’
site=‘hw’
site=‘ral’
ral hwibm
site=‘ibm’
![Page 19: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/19.jpg)
Adding Intermediary Republishers is Difficult Links between producers and higher level
republisher may only be
removed after
the intermediary republisher is in place
… otherwise we may lose tuples. However, this may lead to duplicates.
![Page 20: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/20.jpg)
Question: How do we remove an intermediary republisher?
country=‘britain’
site=‘hw’
site=‘ral’
site=‘ibm’
ral hwibm
![Page 21: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/21.jpg)
Question: How do we remove an intermediary republisher?
country=‘britain’
site=‘hw’
site=‘ral’
ral hwibm
![Page 22: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/22.jpg)
Steps Involved in Removing an Intermediary Republisher
Involves the following tasks: Creating links between producers
and relevant higher level republishers.
Stopping the intermediary republisher from consuming and publishing.
Removing the intermediary republisher.
country=‘britain’
site=‘hw’
site=‘ral’
ral hwibm
![Page 23: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/23.jpg)
Removing an Intermediary Republisher is Difficult Intermediary republisher can only be
removed after
links between producers and higher level republishers are in place
… otherwise we may lose tuples. However, this may lead to duplicates.
![Page 24: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/24.jpg)
Requirements for the Protocol to Change the System Has to deal with:
Addition of new producers. Addition of new intermediary republishers. Removal of intermediary republishers.
Has to achieve: No loss of tuples No generation of duplicate tuples.
![Page 25: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/25.jpg)
Summary
Current situation in R-GMA. Example of a republisher hierarchy
that users want. Problems of creating and
maintaining a hierarchy of republishers.
Other open issues.
![Page 26: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/26.jpg)
Other Issues: Completeness
When is a republisher complete? Simple if all its sources are
registered as complete. What if a source is a latest
producer over a private stream, then can a republisher be complete that uses this source? What if it ignores this source?
![Page 27: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/27.jpg)
Other Issues: Duplicates
Will users be bothered?Possibly if conducting statistical
analysis of tuples.
Should we:1. Filter duplicate tuples out.
Requires duplicate tuple detection.
2. Ignore them and leave in the stream.
![Page 28: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/28.jpg)
Other Issues: Tuple Arrival Order
Should the republisher receive:
1. All tuples from producer 1 in a burst, then producer 2, and then producer 3.
2. Apply some interleaving of tuple arrival.
1 2 3
![Page 29: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/29.jpg)
Other Issues: Queries
Which producer / republisher does the query ask for the answer? The one that is the closest match. All relevant producers and
republishers.Defeats point of
hierarchy. Should the user be able to restrict
the types of query that a republisher can answer?
![Page 30: Republishing Mechanisms for R-GMA Benefits and Approaches. Talk by: Alasdair Gray Collaborators: Andy Cooke, Lisha Ma, and Werner Nutt Heriot-Watt University.](https://reader030.fdocuments.in/reader030/viewer/2022032604/56649e6d5503460f94b6c1f5/html5/thumbnails/30.jpg)
Discussion Points
Ideally, how should the system behave?
What system behaviour can the users live with?
What are the user requirements from WP2?