Event Processing and Integration with IAS Data Processors
-
Upload
invenire-aude -
Category
Technology
-
view
85 -
download
3
Transcript of Event Processing and Integration with IAS Data Processors
![Page 1: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/1.jpg)
InvenireAude.com Copyright (C) 2015 Invenire Aude Ltd.
Practical Guide to
Data Processors
![Page 2: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/2.jpg)
Data Processors
Data processors can be configured to act as:
Data transformation nodes, using PASCAL-like script language, Gateways and bridges (e.g. HTTP/JSON and Queues/XML), SQL Database interfaces using the data mapping script extension.
You can configure and use the Data Processors as single threaded programs but you can define many logic implementations and run them in parallel as threads.
You can choose the transaction support from the three available modes: auto-commit, single phase (independent) commits, distributed two phase commit with XA when the supported coordination software is used.
And last but not least, one can find the Data Processors as a very helpful command line admin's tool.
![Page 3: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/3.jpg)
Architecture
![Page 4: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/4.jpg)
Concept
The Data Processor is designed to process the incoming messages calling desired action accordingly.
1. Begin remote transactions (optional),2. Receive message,3. Parse content (if configured) and call prerequisit actions (if defined).4. Invoke the configured logic.5. Commit or rollback transaction (in case of an exception).
The program can interact with databases and send messages according to the requirements.
![Page 5: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/5.jpg)
First Usage
export IAS_LANG_XSD=${your_project}/data/customer.xsd
File operations:
ias_qs_processor i file:customer.xml o file:stdout l fwd
ias_qs_processor i file:cust.xml o file:stdout_fmt l fwd
ias_qs_processor i file:c1.xml o file:somedir/c2.xml l fwd
l fwd vs l fwd! Turns the parsing off.
ias_qs_processor i file:customer.xml \ o file:stdout_fmt?format=JSON l fwd
![Page 6: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/6.jpg)
Directories
ias_qs_processor i dir:data/xml o file:output l fwd
The above command will display subsequent messages on the terminal output. In order to direct the messages into series of files use the dir protocol in output specification.
ias_qs_processor i dir:data/xml \o 'dir:output/${ID}.${COUNT}.out' l fwd
The parameter is a directory name (output in this case) and the file name pattern.
![Page 7: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/7.jpg)
Timer
The next command will produce two messages with a one second (1000ms) delay.
ias_qs_processor N 2 i timer:clock?timeout=1000 \ o file:stdout l fwd
![Page 8: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/8.jpg)
Queue Operations
ias_qs_processor i shm://sample/Q.TEST \o file:stdout l fwd
By default, the messages (and files) are read in the shared browser mode.
ias_qs_processor i shm://sample/Q.TEST?mode=input \o file:stdout l fwd
Note that if the transport layer does not support browsing (e.g. HTTP/FCGI) omitting the modifier yields in an error.
By default, if there are no messages available, the reading ends immediately. You can specify the timeout parameter (a number of milliseconds or -1 for an infinite wait).
ias_qs_processor \ i shm://sample/Q.TEST?mode=input?timeout=10000 \ o file:stdout l fwd
![Page 9: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/9.jpg)
Network Operations 1
When the receiver is going to be a server, one can start it with the command:
ias_qs_processor i nets://localhost:10000/?mode=input o file:stdout_fmt l fwd
and invoke a client with the matching address specified on its output:
ias_qs_processor i file:data/xml/customer.xml o netc://127.0.0.1:10000/ l fwd
![Page 10: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/10.jpg)
Network Operations 2
The data flow in the other direction is also possible.
Server:
ias_qs_processor o nets://localhost:10000/?mode=input i file:data/xml/customer.xml l fwd
Client:
ias_qs_processor i netc://127.0.0.1:10000/?mode=input o file:stdout_fmt l fwd
![Page 11: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/11.jpg)
Modes
Typical operation patterns are:
● Producer,● Consumer,● Requester,● Responder,● Publisher,● Subscriber,● Browser,● Shared Worker Browser (default!)
![Page 12: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/12.jpg)
Producer and Consumer
![Page 13: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/13.jpg)
Requester and Responder
![Page 14: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/14.jpg)
Publish-Subscribe
The application calls the send() and receive() procedures and is not aware of such a configuration.
Not all transports support this pattern.
![Page 15: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/15.jpg)
Implementing a Service
In this case API allows to perform send() and receive() calls on both i/o descriptors.
One can specify the same or different names for the requester input and output.
![Page 16: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/16.jpg)
Browser
autoRecycle specifies if the processing should resume after end of data condition occurs.
● never Browser is not reset when no message is available.
● next Browser will be reset where no message is available but the exception is thrown this time. The next call will grab a new message.
● immediate Browser will be reset where no message is available and the program attempts to read a message immediately.
![Page 17: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/17.jpg)
Shared Browser
This time, the dedicated thread reads messages on behalf of all logic threads. This pattern allows to take full advantage of parallel processing.
![Page 18: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/18.jpg)
Built-in Actions
Typical operation patterns are:
● Forward.● Execute.● Input data type match execute.● Attribute based selection execute.● Job Daemon.● Synchronous proxy.● Asynchronous proxy.
● Message preview.● Message extraction.● Queue System statistics producer.● Writing custom logic.● Java Script execute.● Simplistic Process Server.
![Page 19: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/19.jpg)
Considerations
![Page 20: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/20.jpg)
Performance Notes
Options:
T <no. of threads>
C <commit count>
N <no. of messages to process> O <no. of messages to skip>
![Page 21: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/21.jpg)
Parsing
export IAS_LANG_XSD=${your_project}/data/customer.xsd
Note the exclamation mark right after the logic name:
l fwd vs l fwd! Turns the parsing off.
![Page 22: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/22.jpg)
Command Line Hacks
![Page 23: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/23.jpg)
Data Format Change
ias_qs_processor i mqm://MQTEST/Q1?mode=input \ o mqm://MQTEST/Q2?format=JSON l fwd
![Page 24: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/24.jpg)
Working with Meta Attributes
ias_qs_processor i mqm://MQTEST/Q1?mode=input \ o 'mqm://MQTEST/Q2?*Folder.ATTR1=VALUE' l fwd
The '*' attributes provided in URL are considered as message meta attributes and will be seen in RFH2 or message properties according to the application or IBM WebSphere MQ version.
![Page 25: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/25.jpg)
Protocol Gateway
![Page 26: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/26.jpg)
Example Queue Setup
DEF QL(GATEWAY.IN)DEF QL(SERVICE.IN)DEF QA(GATEWAY.OUT) TARGQ(SERVICE.IN)
![Page 27: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/27.jpg)
Echo Service
If you do not have a service for this example just start the following command to setup the echo program.
ias_qs_processor \ i 'mqm://MQTEST/SERVICE.IN?mode=responder&timeout=1' \ o dummy: l fwd!
![Page 28: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/28.jpg)
Gateway Synchronous Proxy
ias_qs_processor \ i 'srvhttp://localhost:50000/mode=input&responderName=input' \ o 'mqm://MQTEST/GATEWAY.OUT&mode=requester&inputDestination=GATEWAY.IN&timeout=20000' \ l proxy! m server
Note the srvhttp service and the proxy logic.
![Page 29: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/29.jpg)
Gateway Asynchronous Proxy
ias_qs_processor \ i 'asrvhttp://localhost:50000/mode=input&responderName=input' \ o 'mqm://MQTEST/GATEWAY.OUT&mode=requester&inputDestination=GATEWAY.IN&timeout=1' \ l aproxy!
Note the asrvhttp service and the aproxy logic.
![Page 30: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/30.jpg)
Testing
curl X POST d "{ 'key': 'value1234' }" \ H "ContentType: application/json" \ http://localhost:50000/
You can use the HTTP client of your choice.
![Page 31: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/31.jpg)
Modes
processor Each processor thread opens a session and processes incoming messages until the end conditions are met. The threads are not restarted and program terminates when all threads terminate. This is a default mode.
server The session processing is as in the processor mode. However when the thread encounters the end of data condition, the session is recreated. This mode is useful for the synchronous network servers and when one wants to automatically release resources after some period of inactivity.
![Page 32: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/32.jpg)
Transaction Control
NONTXN No transaction call are executed.
TXN After each iteration the commit (or rollback) API call will be executed on success (error).
XA No transaction control calls are executed and it is assumed that proper XA resource manger is configured.
![Page 33: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/33.jpg)
Configuration Files
ias_qs_processor f <configuration_file>
Configuration (XML or JSON)
- input and output specification (Queues, HTTP(S), Files)
- data sources (data bases)
- logic(s) configuration
You can split the configuration into separate definitions of inputs, outputs or/and datasources.
ias_qs_processor i inputs.xml o outputs.xml d ds.xml
![Page 34: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/34.jpg)
Environment Specific Settings
![Page 35: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/35.jpg)
Writing Data Processor Configuration
Follow the XSD files from the IAS-QsystemLib project. You can write in XML or JSON.
File name Contents
qsystem.service.xsd QSystem service data schema.qsystem.workers.xsd QSystem main processor schema.qsystem.workers.logic.xsd QSystem processor logic schema.qsystem.workers.ds.xsd QSystem data sources (SQL) schema.qsystem.workers.io.xsd QSystem input/output (messaging) schema.qsystem.workers.txm.xsd QSystem XA/TX schema.qsystem.workers.spec.xsd QSystem combined specification schema.qsystem.workers.cache.xsd QSystem cache module schema.
![Page 36: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/36.jpg)
Input Configuration
<inputSpec> <inputs xsi:type="io:ConsumerInput" inputName="input" responderName="output"> <connection alias="qs.esb"/> <destination>CIF.INQ.IN</destination> <txnMode>NONTXN</txnMode> <timeout>-1</timeout> </inputs> </inputSpec>
The above configuration creates a server.
The responderName field creates corresponding output to send replies.
The connection details can be specified here or are referenced via an alias name from the registry file (as shown above).
![Page 37: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/37.jpg)
Output Configuration
The above configuration creates a client.
The inputDestination field specified where the underlying protocol expects the replies .
<outputSpec> <outputs xsi:type="io:RequesterOutput" outputName="output"> <connection alias="qs.esb"/> <destination>WWW.INQ.OUT</destination> <timeout>2000</timeout> <attrUpdates name="FMT" value="XML" override="true"/> <inputDestination>WWW.INQ.IN</inputDestination> </outputs> </outputSpec>
![Page 38: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/38.jpg)
Exit Configuration
The above configuration specifies that the cb::ValidateCustomer program is to be called.The exit program parameters are matched against the input data type.
Available error actions are: IGNORE_ALL, IGNORE_PROGRAM, IGNORE_NONE.
<inputs xsi:type="io:ConsumerInput" inputName="input" responderName="output" > <connection> <protocol>mqm</protocol> <host>MQTSTBRK</host> </connection> <destination>DB.INQ.IN</destination> <txnMode>TXN</txnMode> <timeout>-1</timeout> <attrUpdates name="DB" value="@timestamp@" override="true"/>
<onSend> <program>cb::ValidateCustomer</program> <errorAction>IGNORE_ALL</errorAction> </onSend>
</inputs>
![Page 39: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/39.jpg)
Data Source
<datasourceSpec> <datasources name="ds.mcif"> <connection> <protocol>db2</protocol> <user>user</user> <password>password</password> <location>CIFDB</location> </connection> <txnMode>TXN</txnMode> </datasources> </datasourceSpec>
This example provides all configuration without referencing the registry.
![Page 40: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/40.jpg)
Logic – Explicit Execute
In this scenario the processor will start two threads.Note how “input” and “output”, “error” i/o destinations are specified.
<logicSpec> <logics xsi:type="logic:ExplicitExecute" error="brk.error" instances="2"> <load>esb::broker::inq::routeInput</load> </logics> <logics xsi:type="logic:ExplicitExecute"
input="CIF" output="output" error="brk.error"> <load>esb::broker::inq::routeOutput</load> </logics> </logicSpec>
![Page 41: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/41.jpg)
Logic – Mach Execute
<logicSpec> <logics xsi:type="logic:MatchExecute"> <load>mcif::svc::party::getParties</load> <load>mcif::svc::role::getContextRoles</load> <load>mcif::svc::role::getPartyProducts</load> <load>mcif::svc::comm::getCommunicators</load> </logics> </logicSpec>
In this scenario the processor will find programs that match the input data.
![Page 42: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/42.jpg)
Logic – Input and Output
Each logic has at least the following defaults:- input- output- error (optional)
Above are used by the predefined logic implementations and the script execution framework.
You can specify many input and output definitions in the configuration file. Then you can either assign them as defaults forthe desired logic or use them explicitly via std::send() or std::receive()Script calls.
![Page 43: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/43.jpg)
Basics of Scripting
![Page 44: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/44.jpg)
Environment
Data model:IAS_LANG_XSD=/home/ias/data/xsd/model.xsd
Script Sources:IAS_LANG_SRC_DIRS=/home/ias/lang
![Page 45: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/45.jpg)
First Script
vi /home/ias/lang/update_msgs.y
The Eclipse source highlighting plugin is available.
See the Script Language for more details (for example on writing procedures accessing databases and files and more).
![Page 46: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/46.jpg)
Processing Data
Testing, browsing and displaying result on the screen:
ias_qs_processor i mqm://TEST/Q1 \ o file:stdout_fmt l exe update
Processing, moving and saving changed content in the output queue:
ias_qs_processor i mqm://TEST/Q1?mode=input \ o mqm://TEST/Q2 l exe update
![Page 47: Event Processing and Integration with IAS Data Processors](https://reader034.fdocuments.in/reader034/viewer/2022042821/55d380fabb61eb49048b45a2/html5/thumbnails/47.jpg)
Learn More
http://www.invenireaude.com/content/books/QSystemProc/index.html?show=chapters/ref/index.html