Post on 13-Dec-2015
FIRMS: Federation & Implementation of Reliable Messaging Specifications for
Web Services
Geoffrey Fox (managerial contact)Shrideep Pallickara (technical contact)
October 21 2004 Southampton
Staffing and Contract Update
• Contract still waiting negotiation of terms and conditions– e.g. is the State of Indiana part of British Empire and
subject to its laws
• Have interviewed two good software engineers– Shrideep gave them each a one day exam
FIRMS and FINS• Are built on NaradaBrokering Software with a different
leaner deployment – Can use original deployment if need additional features
Broker Broker
Broker Broker
Publisher
Subscriber
Subscriber
Subscriber
Original
FIRMS
Service NB as Appl. Logic Handler
NB as ServiceHandler Appl. Logic
Multiple protocol transport supportIn publish-subscribeParadigm with differentProtocols on each link
Transport protocols supported include TCP, Parallel TCP streams, UDP, Multicast, SSL, HTTP and HTTPS.Communications through authenticating proxies/firewalls & NATs. Network QoS based RoutingAllows Highest performance transport
Subscription Formats Subscription can be Strings, Integers, XPath queries, Regular Expressions, SQL and tag=value pairs.
Reliable delivery Robust and exactly-once delivery in presence of failures
Ordered delivery Producer Order and Total Order over a message type. Time Ordered delivery using Grid-wide NTP based absolute time
Recovery and Replay Recovery from failures and disconnects.Replay of events/messages at any time. Buffering services.
Security Message-level WS-Security compatible security
Message Payload options
Compression and Decompression of payloadsFragmentation and Coalescing of payloads
Messaging Related Compliance
Java Message Service (JMS) 1.0.2b compliant Support for routing P2P JXTA interactions.
Grid Feature Support NaradaBrokering enhanced Grid-FTP. Bridge toGlobus GT3.
Web Service reliability Prototype implementation of WS-ReliableMessaging
Current NaradaBrokering Features
Forthcoming Features
• Production implementations of WS-Eventing, WS-Notification, WS-RM and WS-Reliability.
• Time Differential Services: Preserve time spacing between events, that are time-stamped using high-resolution timers.
• Active replay support: Pause and Replay live streams.
• Replicated storage support for fault tolerance and resiliency to storage failures.
WS-Reliable Messaging
• Specification from IBM, and Microsoft• Leverages the WS-Addressing and WS-Policy
specifications.• Provides support for both positive and negative
acknowledgements.• Provides operations for explicit creation and
termination of sequences.• Delivery assurance mode supported: at-least-
once, at-most-once, exactly-once, and ordered delivery
WS-Reliability
• Specification from Fujitsu, Oracle, and Sun• Provides support for positive acknowledgements.• Provides support for a variety of message-exchange
patterns.– Request/Response, One-way, Polling
• Delivery assurance mode supported– Unreliable, at-least-once, ordered-and-exactly-once
• Under consideration by the OASIS to be a standard
Mechanisms for Reliable Messaging I
• There are essentially sequence numbers on each message• Unreliable transmission detected by non-arrival of a
message with a particular sequence number• This is “just some TCP reliability” built at application level
(Service level Internet)• One can either use ACK’s – Receiver (service B) positively
acknowledges messages when received– Service A fully responsible for reliability
• Or NAK’s – Service B is partially responsible and tracks message numbers – sends a NAK if sequence number missing
Service B Service A
M(n+1)M(n)
Mechanisms for Reliable Messaging II• Each message has a retransmission time; messages are
retransmitted if ACK’s not received in time– Uses some increasing time delay if retransmit fails
• Note need to be informed (eventually) that OK to throw away messages at sender; pure NAK insufficient
• Note this is reliability from final end-point to beginning end-point: TCP reliability is for each link and has different grain size and less flexible reliability mechanisms
• There are several efficiency issues– Divide messages into groups and sequence within groups– Do not ACK each message but rather sequences of messages
• NAK based system attractive if high latency (some mobile devices) on messaging from receiver back to sender
Custom Message Reliability
NaradaBroker
Filter 1
Filter 2
WS-RM
WirelessOptimizedWS-RM
WS-Reliability
2 second PDA reply latency!
Different endpoints may well need different reliability schemes. Another reason to use application layer.Need to define easy touse “standard reliabilityprofiles
Handlers and Filters in-memory Processing
Container
Service
Filter pipeline
Filter
Service method
ContainerComponent
Container Component
BytePackets
Support Classes
SOAP Message
Component Classes
Client-side
Application Logic
Filter pipeline
Filter
SupportClasses
Support Classes
Byte processor
Type mappings
SOAP object
SOAPString
ProcessWS-RM
Built-inHandlers
Deployment Issues for “System Services”• “System Services” are ones that act before the real
application logic of a service• They gobble up part of the SOAP header identified
by the namespace they care about and possibly part or all of the SOAP body– e.g. the XML elements in header from the WS-RM
namespace
• They return a modified SOAP header and body to next handler in chain
WS-RMHandler
WS-……..Handler
Header
Body
e.g. ……. Could be WS-Eventing WS-Transfer ….
Proxy Distributed WS-RM Processing• A handler is like an in memory “service” so one
can build handlers that can alternatively be deployed “outside” application service and look like a WS-RM service.
• Comments on handlers as services paradigm?– Will capture this as a deployment memo – Handlers could be just part of application logic – more
natural for WS-Eventing than WS-RM
WS-RM enabledSOAP
WS-RMService
LegacyService
WS-RM removedfrom SOAP Header
WSRM Implementation: Overview
SOAP Parser
WSRMProcessor
ProtocolImplementation
Sequence Policy Information
ProtocolConstants
StableStorage
SOAP Parser
WSRMProcessor
Stable Storage• This is where messages are stored until receiver
indicates that message has been successfully processed– In memory, Flat Files, MySQL supported– In memory (default?) or Flat File is sufficient for
WS-RM messages that do not require sophisticated search
• Can store messages at one or more distributed NaradaBrokering sites– Could keep messages for a long time!
• All “new” communication will be done using SOAP and WSDL defined ports
Complicated NaradaBrokering
• NaradaBrokering has more capabilities than OMII needs– We will make them optional to reduce code bloat
• NaradaBrokering could implement Handler chains for protocols and WSDL bindings not supported by AXIS– UDP plus WS-RM (or equivalent) has been shown
to outperform traditional TCP over high bandwidth high latency links
– Also supports parallel TCP and GridFTP
WS-Reliability WS-ReliableMessaging
Defines Defines elements and attributes in the header block of a SOAP envelope.
An XML based schema for elements that are needed for reliable messaging.
Related Specifications
SOAP, WS-Security WS_addressing, WS-Policy, WS-Security
Companies Involved Sun, Oracle, Fujitsu IBM, Microsoft, BEA
Submission Status Under consideration by OASIS to be a standard
Not submitted yet.
Delivery modes supported
Unreliable, at-least-once, ordered-and-exactly-once
At most once, at least once, ordered and exactly-once.
WS-Reliability WS-ReliableMessaging
Groups of messages
Identified by GroupId information associated with every message in sequence. Individual messages in the group also have a SequenceNumber which increases monotonically.
Grouped together using Sequence element, which has a unique identifier, and a message number which increments sequentially.
Support for creation and termination of message groups
NO YES
Ordering Is guaranteed only within a group of messages.
Is guaranteed only within a group of messages
Ordering and Duplication dependence
Order is always tied to Guaranteed delivery and cannot be separately specified.
Order is not necessarily tied to guaranteed delivery
WS-Reliability WS-ReliableMessaging
Exchange and Specification of protocol constants
Through an abstract concept referred to as Agreement. The spec does not recommend one.
WS-Policy.
Numbering info associated with first element in a group of messages.
SequenceNumber starts at 0 for the first message in a group.
MessageNumber starts at 1 for the first message in a group.
Defines Message-Exchange patterns
Request/Response, One-way and Polling
Not defined
Support for negative acknowledgements
NO YES. This enables receiver-initiated error corrections.
Support for acknowledging range of messages
YES YES
Requesting acknowledgements
The AckRequested element is REQUIRED in every message for which reliable delivery needs to be ensured.
AckRequested is used to request the receiving entity to acknowledge the message received.
WS-Reliability WS-ReliableMessaging
Time based expiry
removerAfter: Receiver Maintains value of GroupId until the end of the sequence is received or after the expiry of a specified time. Plays a role in order/duplicate detection.ExpiryTime: Defines the expiration time associated with an individual message.
Expires: Provides an indication of the expiry time for the sequence specified in the Sequence element.There is also an Inactivity timeout associated with sequences. This is specified in milliseconds.
Retransmissions
Triggered after receipt of a set of acknowledgements. In the event an acknowledgement is not received, the message is retransmitted until a specified number of resend attempts have been made.
Triggered by either positive or negative acknowledgments. Specification of a Retransmission Interval for a sequence. This effects every message within a sequence of messages. The interval can also be adjusted based on the exponential backoff algorithm.
WS-Reliability WS-ReliableMessaging
Security Relies on WS-Security and assorted specifications
Relies on WS-Security and assorted specifications
Errors Are notified through SOAP faults.
Are notified through SOAP faults. Fault processing is more sophisticated since one can leverage WS-Addressing’s message information headers.
WSRM/WSR Similarities
• Messages are part of a sequence/group of messages.• Addresses issues pertaining to ordering and duplicate
detection.• Quality of service constraints are applied to groups of
messages.• Recommends message-level security: specifically
WS-Security for secure delivery of messages.• Provides framework for reporting faults/errors in
processing between endpoints involved in reliable delivery
• Provide support for acknowledging multiple range of messages received within a group/sequence.
WSRM/WSR Differences• WSR has no support for negative acknowledgements.
Retransmissions are always initiated by the source of messages. WSRM supports negative acknowledgements.
• WSRM has an explicit operation for the creation of sequence and associated sequence identifier. WSR has no such operation, a new group begins when a receiver has encountered a new groupID.– disadvantage: difficult to resolve collisions in group id namespace
• WSRM uses WS-Policy for specifying and exchanging featured info. WSR does not advocate any specification though it alludes to an abstract concept of agreements.
• Order is always tied to duplication elimination in WSR. WSRM allows order and duplication detection to exist independent of each other
• WSR incorporates a HTTP binding for its specification. WSRM currently does not have one; though one can simply use SOAP’s HTTP binding.
• WSRM has explicit termination of sequences. WSR groups cannot be terminated. They are first closed and then removed.
Federation, Why?
• WSRM being supported by powerful group: IBM/Microsoft.
• WSR being actively considered for recommendation as a standard by OASIS
• It is quite possible that these specification may continue to co-exist for a while.
• Federation would allow end-points belonging to different specifications to communicate with each other.
Federation, How?• Mapping of physical (XML) elements and semantics associated
with these specifications.– Mapping of sequence numbering. WSRM starts numbering at 1, WSR starts at
0.– NAKs in WSRM messages will simply be ignored, since WSR does not support
it.• Mapping of policy elements and rules associated with
where/when and the combination in which multiple policy/agreement elements may occur.
• Enforcing constraints on delivery. WSR provides a subset of the delivery modes available in WSRM
• Mapping of faults/error reporting• Creation/Termination of sequence in WSRM have no equivalents
in WSR.– So terminate-sequence in WSRM will trigger multiple requests/retransmission
to ensure WSR has received everything. Group expiry time then needs to be updated at the WSR side.
FINS: Federation & Implementation of Notification Specifications for Web
Services
Geoffrey Fox (managerial contact)
Shrideep Pallickara (technical contact)
WS-Eventing
• Delivery Modes: Push– Pull and Trap (UDP Notifications) through WS-Management
• Subscription Manager can be a separate Web service or part of handler bundled with Source Web Service when it gives as in OGSI special ports for each Source service
Source SinkSubscription
Manager
Subscribe
Unsubscribe
Renew
getStatus
Subscription End
Notifications
WS-Notification
• WS-BaseNotification, WS-BrokeredNotification, WS-Topics, WS-ResourceProperties and WS-ResourceLifetime
ProducerSubscription
Manager Subscriber Consumer
Subscribe
Get current message
Pause subscription
Resume subscription
Notify
NotificationBroker
Publisher Subscriber
NotifyRegisterPublisher
Subscribe
WS-Eventing/Notification Issues• IBM (Sun ..) joined Microsoft (BEA. Tibco) in new August
2004 specification • WS-Eventing will presumably replace WS-BaseNotification
as core of WS-BrokeredNotification• We made WS-Eventing as our highest priority• Eventually support WS-Eventing, WS-Notification and JMS
(Java Message Service) in a federated model• WS-Metadata needed to query WS-Eventing sources
– Not in current OMII plans but clear how to add• Note FINS will use FIRMS in all messages if desired
WS-Notification WS-Eventing
Delegated Management of subscriptions
Yes. Through the subscription manager interface.
Yes. Through the subscription manager interface.
Support for replay like features
One can get last message to a topic. A sink can also retrieve messages issued between the pausing and resumption of a subscription.
No.
Subscription operations
Subscribe, Pause and Resume. (There is NO exchange to unsubscribe).
Subscribe, Renew, Unsubscribe and Subscription End.
Support for filters to narrow notifications
YES YES
Subscription lifetimes
Defined using the WS-Resource Lifetime specification.
Contained within the Subscribe and Renew exchanges.
WS-Notification WS-Eventing
Notification filters and topic expressions supported
Topic Expressions supported: QName, “/” separated Strings, and XPath path expressions.
Filter supported is XPath.User defined filters allowed
Hierarchical topics and Wildcards support
Yes. Supports * and // wildcards for selection of topic descendants in a topic tree.
No.
Topic space management
Defined using WS-Topics. The topic space will also support exchanges as defined by the WS-ResourceProperties specification.
No formal recommendation regarding topic management.
Advertisement of supported topics
Yes. The NotificationProducer interface allows inspection of available topics.
No.
WS-Notification WS-Eventing
On demand publishing
YES. This is supported through the WS-Brokered Notification specification. This allows a publisher to publish ONLY if there is a consumer interested in receipt of notifications.
No.
Notification messages
Provides support for both a Notify message as well as “raw” application specific message,
Does not define any special Notification message type.Application can define special ports for different notifications distinct from “real” messages
Suggested Security
WS-Security and assorted specifications.
WS-Security & assorted specifications.
WS-Eventing and WS-NotificationWS-Notification WS-Eventing
Related Specifications
SOAP, WS-Addressing, WS-BaseNotification, WS-Brokered Notification, WS-Topics, WS-Resource Properties and WS-ResourceLifetime
SOAP, WS-Addressing
Support for loosely coupled notifications. (Producers need not know consumers)
Yes. The intermediary called Notification Broker and the exchanges that need to be supported are defined in the WS-Brokered Notification specification.
No.
Delivery modes supported
Push PushPull and Trap (UDP transmissions) defined in WS-Management