HPSearch

Design & Development via ScriptingHarshawardhan Gadgil

Dr. Geoffrey Fox, Dr. Marlon Pierce

Why Script Web Services

Scripting Advantages Rapid prototyping

Scripting Web Services Manipulate data streams, files and web-services using

scripting language. Equivalent to shell and Perl in UNIX. Allows us to control the invocation and execution of

disparate services using scripting. Equivalent to programming a workflow

Setting up data-flow among components by dynamically joining disparate data streams

Rapidly deploy applications (under development) We have developed a tool for binding URIs to a scripting

language that manipulates the local / remote resource identified by the URI - HPSearch

HPSearch Binds URI to a scripting language

We use Mozilla Rhino (A Javascript implementation, Refer: http://www.mozilla.org/rhino), but the principles may be applied to any scripting language such as Perl, Python etc…

Every Resource may be identified by a URI and HPSearch allows us to manipulate the resource using the URI. For e.g. Read from a web address and write it to a local file

x = “http://trex.ucs.indiana.edu/data.txt”;y = “file:///u/hgadgil/data.txt”;Resource r = new Resource(“Copier”);r.port[0].subscribeFrom(x); // read from r.port[0].publishTo(y); // write to

f = new Flow();f.addStartActivities(r);f.start(“1”);

HPSearch (contd.) Currently provide bindings for the following

file:// socket://ip:port http://, ftp:// topic:// jdbc:

Host-objects to do specific tasks WSDL – invoke web-services using SOAP PerfMetrics – Bind NaradaBrokering performance metrics. Store

published metrics and allow querying Resource – Every data source / filter / sink is a resource. Flow – To create a data flow between resources. Useful for

streaming applications. For more information, visit

http://www.hpsearch.org

NaradaBrokering

Event- brokering system designed to run on a large network of co-operating brokers.

Guarantees delivery in presence of failures Implements high-performance protocols (message

transit time < 1 ms per broker) Order-preserving optimized message transport Interface with reliable storage for persistent events Fault tolerant data transport Support for different underlying transport

implementations such as TCP, UDP, Multicast, SSL, RTP

HPSearch uses NaradaBrokering to route data streams

Consider the script as followsResource a = new Resource(“ResourceA”);a.port[0].subscribeFrom(“file:///u/hgadgil/data.txt”);a.port[0].publishTo(“topic:///x”); // Optional

Resource b = new Resource(“ResourceB”);b.port[0].publishTo(“file:///tmp/rawData.dat”);b.port[0].subscribeFrom(a.port[0].publish);

f = new Flow();f.addComponents(b);f.addStartActivities(a);f.start(“1”);

HPSearch + NaradaBrokering

Resource A

Resource B

HPSearch Shell

WFE

WFE

Broker Network

HPSearch - WSProxy

WFE

WFE

WSProxyClient

WSProxy

Service

WSProxyClient can control the WSProxy using simple

Web-service calls

WSProxy is a simple web-service that listens for requests from the client (WSProxyClient) and steers the service

Request Handler

WFE

HPSearch Shell

The WSProxy connects to a broker and handles all the data communication on behalf of the

service

Pipe DB ConnectorFlow Handler

Flood Modeling Example using Web Service scripting: Components

Rainfall Data Inches of rainfall gathered by the sensor network at specific intervals

(Simulated to be in intervals of 30 min in our example) WebMap Server

OpenGIS based terrain data repository Generates Watershed model

Compute Service – Runoff Model Computes the runoff model using the Hydrostatic watershed model and

Hydrodynamic rainfall data Visualization Service

Shows the visual representation of the runoff model for the terrain in question Each of these components run as a separate Web service (Either direct or

wrapped as a web service) Scripting to manage the chain of these disparate services into one complete

application Each of these components is wrapped using WSProxy which is then

controlled by the Workflow Engine

Crisis Grid - Flood Modeling

Compute Service is the basic control On start up, MapService sends the

spatial information for the terrain under consideration to the Compute service.

Compute Service requests the Rainfall publisher to send the next rainfall reading (pulls the next reading).

The Compute service then operates the run off model for flood analysis.

The computed results are then sent to a Visualization service which displays the computed predicted model results in form of an animation

Map Service Rainfall Publisher

Compute Service

Visualization

maps = "/CGL/FLOODMODEL/MAPS";rain = "/CGL/FLOODMODEL/RAINFALL";visu = "/CGL/FLOODMODEL/VISUAL";

mapServer = "org.hpsearch.demo.floodModel.MapService";mapServerLoc = "http://156.56.104.170:9090/axis/services/WSSConnector?wsdl";mapSource = new WebServiceHandler(mapServer);mapSource.setEndPointURI(mapServerLoc);mapSource.setParameter("MAPSERVERTOPIC", maps);

rainFallPublisher = "org.hpsearch.demo.floodModel.RainfallPublisher";rainFallPublisherLoc = "http://156.56.104.170:6060/axis/services/WSSConnector?wsdl";rainFall = new WebServiceHandler(rainFallPublisher);rainFall.setEndPointURI(rainFallPublisherLoc);rainFall.setParameter("RAINFALLTOPIC", rain);rainFall.setParameter("VISUALIZATIONTOPIC", visu);

computeService = "org.hpsearch.demo.floodModel.ComputeService";computeServiceLoc = "http://156.56.104.170:7070/axis/services/WSSConnector?wsdl";compute = new WebServiceHandler(computeService);compute.setEndPointURI(computeServiceLoc);compute.setParameter("MAPSERVERTOPIC", maps);compute.setParameter("RAINFALLTOPIC", rain);compute.setParameter("VISUALIZATIONTOPIC", visu);

crisisGridFlow = new Flow();crisisGridFlow.addComponents(compute);crisisGridFlow.addStartActivities(mapSource, rainFall);crisisGridFlow.start("1");

HPSearch Java ScriptCode for creating and initializing the WMS Component

Code for creating and initializing the Rainfall data Component

Code for creating and initializing the Compute service

Code for registering the components of the flow and starting it

How to initialize… First Define the service to be invoked

mapServer = "org.hpsearch.demo.floodModel.MapService";

The location of the service. This should be normally done via a registry lookup. For For E.g. NBDiscover.discover(“MapService”, specs); Its hard-coded as of nowmapServerLoc = "http://156.56.104.170:9090/axis/services/WSSConnector?wsdl";

Initialize a WebService Handler. This object invokes a WSProxyClient that interacts with the WSProxy which is a wrapper on the actual servicemapSource = new WebServiceHandler(mapServer);

Set the End-point URImapSource.setEndPointURI(mapServerLoc);

Set the service initialization properties. This is optional, but allows for more customization. This could include QoS properties, security tokens and any other service specific parametersmapSource.setParameter("MAPSERVERTOPIC", maps);

How to Initialize (contd.) Finally we create a new Flow Object. This object helps to serialize the

requests for setting up a new Flow. The FlowHandler component of the WorkflowEngine then distributes these requests to other engines and the flow is finally started.crisisGridFlow = new Flow();

We add the components of the flow. All components which generate the data to be processed are startActivities and are started after the other components are started. This enables the components to start processing the data as soon as the startActivities publish the datacrisisGridFlow.addComponents(compute);

crisisGridFlow.addStartActivities(mapSource, rainFall);

Finally we start the flow. The ”1” is for debugging purposes. Ignore it for nowcrisisGridFlow.start("1");

In Short…(w.r.t. Flood model)

Handle file-based data. Read from static files (archived data),

databases, For E.g. The terrain data is static Support streaming data sources. For E.g.

Rainfall data read in real-time in a stream. In an actual system we may expect the

frequency of observation (and hence data generation) to be about 5 to 20 minutes.

In our system, we assume just one source for rainfall data but in an actual scenario, multiple data sources possible

Wrapping a Simple Service

Two implementations of ProxyWebService RunnableProxyWebService

Provides operations START, STOP, SUSPEND, RESUME.

Service is provided the required Input and Output streams from where it will read and write.

Good for applications which work on a chunk of data at a time

Easiest way to create your first service WrapperProxyWebService

Provides just the 4 operations as above. The application handles its own communication

Example (next slide)

package mypackage;

public class StringProcessingService extends RunnableProxyWebService {

BufferedReader input; BufferedWriter output;

public void initService(InputStream[] in, OutputStream[] out) { input = new BufferedReader(new InputStreamReader(in[0])); output = new BufferedWriter(new OutputStreamWriter(out[0])); }

public String[] serviceExceptions() { String[] exceptions = { "java.io.EOFException",

"java.io.IOException“};

return exceptions; }

Coding a simple string reversal service

MUST initialize the service in this fashion

Enables the WSProxyClient to add error specific handlers. You may return an empty array also.

Coding a simple string reversal service

public void process() throws Exception { String s, newS; try { s = input.readLine(); if (s == null) { serviceFinished();

return; } } catch (IOException e) {

throw e; }

newS = reverseOf(s);output.write(newS);output.newLine();

output.flush(); }public String reverseOf(String s) {

…}

}

Your code that processes the data

Deploy using AXIS

Create a deployment descriptor for AXIS Or use one which already exists

Deploy the service Finally run the script to start using the Filter

Invoking our simple service

service = “mypackage.StringProcessingService”;serviceLoc = “http://...path to WSDL”;stringproc = new WebServiceHandler(service);stringproc.setEndpointURI(serviceLoc);stringproc.setInput(“input URI”);stringproc.setOutput(“output URI”);

f = new Flow();f.addStartActivities(stringproc);f.start(“1”);

Useful links for additional information

WSProxy http://www.hpsearch.org/notes/wsproxy

Technical Report on the flood model http://www.hpsearch.org/documents/crisisGridFlow.pdf

Timing Analysis etc… (incomplete as of now) http://www.hpsearch.org/notes/notes/9-timing.html

Credits…

Dr. Sunghoon Ko (CGL) and Dr. Jin-Yong Choi (Purdue Univ.) for the Flood Demo

NB – team @ CGL, Dr. Pierce, Dr. Fox