c++ Ntw Programming

“hs˙bo2001/page i

i

i

i

i

i

i

i

i

C++ Network Programming

Systematic Reuse with ACE and Frameworks

Dr. Douglas C. Schmidt Stephen D. Huston


i

i

i

i

i

i

i

i

“hs˙b2001page

i

i

i

i

i

i

i

i

Contents

About this Book vii

1 Object-Oriented Frameworks for Network Programming 11.1 An Overview of Object-Oriented Frameworks . . . . . . . . 11.2 Applying Frameworks to Network Programming . . . . . . 41.3 An Overview of the ACE Frameworks . . . . . . . . . . . . . 71.4 Comparing Frameworks with Other Reuse Techniques . . 101.5 Example: A Networked Logging Service . . . . . . . . . . . 161.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

2 Service and Configuration Design Dimensions 232.1 Service Design Dimensions . . . . . . . . . . . . . . . . . . 23

2.1.1 Short- vs. Long-Duration Services . . . . . . . . . . 242.1.2 Internal vs. External Services . . . . . . . . . . . . . 242.1.3 Stateful vs. Stateless Services . . . . . . . . . . . . . 262.1.4 Layered/Modular vs. Monolithic Services . . . . . . 272.1.5 Single- vs. Multi-Service Servers . . . . . . . . . . . 302.1.6 One-shot vs. Standing Servers . . . . . . . . . . . . . 31

2.2 Configuration Design Dimensions . . . . . . . . . . . . . . . 332.2.1 Static vs. Dynamic Naming . . . . . . . . . . . . . . . 332.2.2 Static vs. Dynamic Linking . . . . . . . . . . . . . . . 332.2.3 Static vs. Dynamic Configuration . . . . . . . . . . . 35

2.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

iii


i

i

i

i

i

i

i

i

iv CONTENTS

3 The ACE Reactor Framework 393.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393.2 The ACE Time Value Class . . . . . . . . . . . . . . . . . . . 423.3 The ACE Event Handler Class . . . . . . . . . . . . . . . . . 443.4 The ACE Timer Queue Classes . . . . . . . . . . . . . . . . 543.5 The ACE Reactor Class . . . . . . . . . . . . . . . . . . . . . 613.6 The ACE Select Reactor Class . . . . . . . . . . . . . . . . . 733.7 The ACE TP Reactor Class . . . . . . . . . . . . . . . . . . . 813.8 The ACE WFMO Reactor Class . . . . . . . . . . . . . . . . 843.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

4 The ACE Service Configurator Framework 914.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 914.2 The ACE Service Object Class . . . . . . . . . . . . . . . . . 944.3 The ACE Service Repository and ACE Service Repository Iterator

Classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 994.4 The ACE Service Config Class . . . . . . . . . . . . . . . . . 1094.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

5 The ACE Task Framework 1215.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1215.2 The ACE Message Queue Class . . . . . . . . . . . . . . . . 1235.3 The ACE Task Class . . . . . . . . . . . . . . . . . . . . . . . 1395.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150

“hs˙bo2001/page v

i

i

i

i

i

i

i

i

List of Figures

1.1 Defining Characteristics of a Framework . . . . . . . . . . . . 31.2 Levels of Abstraction for Network Programming . . . . . . . . 51.3 The Layered Architecture of ACE . . . . . . . . . . . . . . . . 81.4 The Frameworks in ACE . . . . . . . . . . . . . . . . . . . . . 91.5 Class Library vs. Framework Architectures . . . . . . . . . . 111.6 Applying Class Libraries to Develop and Use ACE Frameworks 141.7 Component Architecture . . . . . . . . . . . . . . . . . . . . . 141.8 Processes and Daemons in the Networked Logging Service . 18

2.1 Internal vs. External Services . . . . . . . . . . . . . . . . . . 252.2 Layered/Modular vs. Monolithic Services . . . . . . . . . . . 272.3 Single-service vs. Multi-service Servers . . . . . . . . . . . . . 302.4 One-shot vs. Standing Servers . . . . . . . . . . . . . . . . . . 322.5 Static Linking vs. Dynamic Linking . . . . . . . . . . . . . . . 34

3.1 The ACE Reactor Framework Classes . . . . . . . . . . . . . . 403.2 The ACETime Value Class . . . . . . . . . . . . . . . . . . . . 433.3 The ACEEvent Handler Class . . . . . . . . . . . . . . . . . . 463.4 The ACE Timer Queue Classes . . . . . . . . . . . . . . . . . . 553.5 The ACEReactor Class . . . . . . . . . . . . . . . . . . . . . . 623.6 The ACE Reactor Class Hierarchy . . . . . . . . . . . . . . . . 683.7 Architecture of Reactor-based Logging Server . . . . . . . . . 703.8 The ACESelect Reactor Framework Internals . . . . . . . . 753.9 The ACESelect Reactor Notification Mechanism . . . . . . 78

v


i

i

i

i

i

i

i

i

vi LIST OF FIGURES

4.1 The ACE Service Configurator Framework Classes . . . . . . 924.2 The ACEService Object Class . . . . . . . . . . . . . . . . . 954.3 The ACEService Repository Class . . . . . . . . . . . . . . 994.4 The ACEService Repository Iterator Class . . . . . . . . 1024.5 The ACEService Config Class . . . . . . . . . . . . . . . . . 1114.6 BNF for the ACEService Config Scripting Language . . . . 1164.7 A State Diagram for Configuring the Server Logging Daemon 1184.8 A State Diagram for Reconfiguring the Server Logging Daemon 119

5.1 The ACE Task Framework Classes . . . . . . . . . . . . . . . 1225.2 The ACEMessage Queue Class . . . . . . . . . . . . . . . . . . 1255.3 The Structure of an ACEMessage Queue . . . . . . . . . . . . 1265.4 Multi-threaded Client Logging Daemon . . . . . . . . . . . . . 1325.5 The ACETask Class . . . . . . . . . . . . . . . . . . . . . . . . 1405.6 Task Activate Behavior . . . . . . . . . . . . . . . . . . . . . . 1425.7 Passing Messages Between ACETask Objects . . . . . . . . . 1425.8 Architecture of the Thread Pool Logging Server . . . . . . . . 144


i

i

i

i

i

i

i

i

About this Book

Writing high-quality networked applications is hard—it’s expensive, com-plicated, and error-prone. The patterns, C++ features, and object-orienteddesign principles presented in [SH02], help to minimize complexity andmistakes in concurrent networked applications by re-factoring [FBB+99]common structure and functionality into reusable wrapper facade class li-braries. If these class libraries are rewritten for each new project, however,many benefits of reuse will be lost.

Historically, many network software projects began by designing andimplementing demultiplexing and dispatching infrastructure mechanismsthat handle timed events and I/O on multiple socket handles. Next, theyadded service instantiation and handling mechanisms atop the demulti-plexing and dispatching layer, along with message buffering and queueingmechanisms. Finally, application-specific behavior was implemented usingthis ad hoc host infrastructure middleware.

The development process outlined above has happened many timesin many companies, by many projects in parallel and, worse, by manyprojects serially. Regrettably, this continuous rediscovery and reinventionof core concepts and software has kept costs unnecessarily high through-out the development lifecycle. This problem is exacerbated by the inherentdiversity of today’s hardware, operating systems, compilers, and commu-nication platforms, which keep shifting the foundations of networked ap-plication software.

Object-oriented frameworks [FJS99a, FJS99b] are one of the most flexi-ble and powerful techniques for addressing the problems outlined above. Aframework is a reusable, “semi-complete” application that can be special-

vii


i

i

i

i

i

i

i

i

viii About this Book

ized to produce custom applications [JF88]. Frameworks help to reduce thecost and improve the quality of networked applications by reifying provensoftware designs and patterns into concrete source code. By emphasizingthe integration and collaboration of application-specific and application-independent classes, moreover, frameworks enable larger-scale reuse ofsoftware than is possible by reusing individual classes or stand-alone func-tions.

In 1992, Doug Schmidt started the open-source ACE project at the Uni-versity of California, Irvine and Washington University, St. Louis. Over thenext decade, the ACE toolkit yielded some of the most powerful and widelyused object-oriented frameworks written in C++. By applying reusable soft-ware patterns and a lightweight OS portability layer, the ACE toolkit pro-vides frameworks for synchronous and asynchronous event handling, ser-vice initialization, concurrency, connection management, and hierarchicalservice integration.

ACE has changed the way complex networked applications and mid-dleware are being designed and implemented on the world’s most popularoperating systems, such as AIX, HP/UX, Linux, MacOS X, Solaris, andWindows, as well as real-time embedded operating systems, such as Vx-Works, LynxOS, ChorusOS, QNX, pSOS, and WinCE. ACE is being usedby thousands of development teams ranging from large Fortune 500 com-panies to small startups. Its open-source development model and self-supporting culture is similiar in spirit and enthusiasm to Linus Torvald’spopular Linux operating system.

Intended Audience

This book is intended for “hands-on” C++ developers or advanced stu-dents interested in understanding how to design and apply object-orientedframeworks to program concurrent networked applications. We show youhow to enhance your design skills and take advantage of C++, frameworks,and patterns to produce flexible and efficient object-oriented networkedapplications quickly and easily. The code examples we use to reinforce thedesign discussions illustrate how to use the frameworks in ACE. These ex-amples help you begin to apply key object-oriented design principles andpatterns to your concurrent networked applications right away.


i

i

i

i

i

i

i

i

About this Book ix

Structure and Content

This book describes a family of object-oriented network programming frame-works provided by the ACE toolkit. These frameworks help reduce thecost and improve the quality of networked applications by reifying provensoftware designs and implementations. ACE’s framework-based approachexpands reuse technology far beyond what can be achieved by reusing in-dividual classes or even class libraries. We describe the design of theseframeworks, show how they can be applied to real networked applications,and summarize the design rules and lessons learned that ensure effectiveuse of these frameworks.

This book’s primary application is a networked logging service thattransfers log records from client applications to a logging server, whichstores the records in a file or database. We use this service as a runningexample throughout the book to

� Show concretely how ACE frameworks can help achieve efficient, pre-dictable, and scalable networked applications and

� Demonstrate key design and implementation considerations and so-lutions that will arise when you develop your own concurrent object-oriented networked applications.

Chapter 1 introduces the concept of an object-oriented framework andshows how frameworks differ from other reuse techniques, such as classlibraries, components, and patterns. We also outline the frameworks inthe ACE toolkit that are covered in subsequent chapters. The ACE frame-works are based on a pattern language [BMR+96, SSRB00] that has beenapplied to thousands of production and research networked applicationsand middleware systems world-wide.

Chapter 2 presents a domain analysis of service and configuration de-sign dimensions that address key networked application properties, suchas duration and structure, how networked services are identified, and thetime at which they are bound together to form complete applications.

Chapter 3 describes the design and use of the ACE Reactor framework,which implements the Reactor pattern [SSRB00] to allow event-driven ap-plications to demultiplex and dispatch service requests that are deliveredto an application from one or more clients.

Chapter 4 describes the design and use of the ACE Service Config-urator framework, which implements the Component Configurator pat-

“hs˙bo2001/page x

i

i

i

i

i

i

i

i

x About this Book

tern [SSRB00] to allow an application to link/unlink its component im-plementations at run-time without having to modify, recompile, or relinkthe application statically.

Chapter 5 describes the design and effective use of the ACE Task frame-work, which can be used to implement key concurrency patterns, such asActive Object and Half-Sync/Half-Async [SSRB00].

Chapter ?? describes the design and effective use of the ACE Acceptor-Connector framework, which implements the Acceptor-Connector pattern[SSRB00] to decouple the connection and initialization of cooperating peerservices in a networked system from the processing they perform onceconnected and initialized.

Chapter ?? describes the design and use of the Proactor framework,which implements the Proactor pattern [SSRB00] to allow event-driven ap-plications to efficiently demultiplex and dispatch service requests triggeredby the completion of asynchronous operations.

Chapter ?? describes the design and use of the ACE Streams frame-work, which implements the Pipes and Filters pattern [BMR+96] to providea structure for systems that process a stream of data.

Chapter ?? describes the design and use of the ACE Logging Serviceframework, which uses the ACE Reactor, Service Configurator, Task, andAcceptor-Connector frameworks to implement and configure the variousprocesses that constitute the networked logging service.

Appendix ?? describes the design rules to follow when using the ACEframeworks. Appendix ?? summarizes the lessons we’ve learned duringthe past decade developing the reusable object-oriented networked appli-cation software in the ACE toolkit and deploying ACE successfully in a widerange of commercial applications across many domains. Appendix ?? pro-vides a synopsis of all the ACE classes and frameworks presented in thetwo volumes of C++ Network Programming.

The book concludes with a glossary of technical terms, an extensive listof references for further research, and a general subject index.

Related Material

This book focuses on abstracting commonly recurring object structures,behaviors, and usage patterns into reusable frameworks to make it easierto develop networked applications more efficiently and robustly. The first


i

i

i

i

i

i

i

i

About this Book xi

volume in this series—C++ Network Programming: Mastering Complexitywith ACE and Patterns [SH02]—is primarily concerned with using object-oriented patterns and language features to abstract away the details ofprogramming low-level APIs. We recommend that you read the Volume 1first before reading this book.

This book is based on ACE version 5.2, released in October, 2001. ACE5.2 and all the sample applications described in our books are open-sourcesoftware that can be downloaded at http://ace.ece.uci.edu and http://www.riverace.com . These sites also contain a wealth of other materialon ACE, such as tutorials, technical papers, and an overview of other ACEwrapper facades and frameworks that aren’t covered in this book. We en-courage you to obtain a copy of ACE so you can follow along, see the actualACE classes and frameworks in complete detail, and run the code examplesinteractively as you read through the book. Pre-compiled versions of ACEcan also be purchased at a nominal cost from http://www.riverace.com .

To learn more about ACE, or to report any errors you find in the book,we recommend you subscribe to the ACE mailing list, [email protected] . You can subscribe by sending email to the Majordomo listserver at [email protected] . Include the following com-mand in the body of the email (the subject is ignored):

subscribe ace-users [emailaddress@domain]

You must supply emailaddress@domain only if your message’s From ad-dress is not the address you wish to subscribe.

Archives of postings to the ACE mailing list are available at http://groups.yahoo.com/group/ace-users . Postings to the ACE mailing listare also forwarded to the USENET newsgroup comp.soft-sys.ace .

Acknowledgements

Champion reviewing honors go to ..., who reviewed the entire book andprovided extensive comments that improved its form and content substan-tially. Naturally, we are responsible for any remaining problems.

Many other ACE users from around the world provided feedback ondrafts of this book, including Vi Thuan Banh, Kevin Bailey, Alain Decamps,Dave Findlay, Don Hinton, Martin Johnson, Nick Pratt, Eamonn Saunders,Michael Searles, Kalvinder Singh, Henny Sipma, Leo Stutzmann, TommySvensson, Dominic Williams, Johnny Willemsen, and Vadim Zaliva.


i

i

i

i

i

i

i

i

xii About this Book

We are deeply indebted to all the members, past and present, of theDOC groups at Washington University in St. Louis and the University ofCalifornia, Irvine, as well as the team members at Object Computing Inc.and Riverace Corporation, who developed, refined, and optimized many ofthe ACE capabilities presented in this book. This group includes...

We also want to thank the thousands of C++ developers from over fiftycountries who’ve contributed to ACE during the past decade. ACE’s ex-cellence and success is a testament to the skills and generosity of manytalented developers and the forward looking companies that have had thevision to contribute their work to ACE’s open-source code base. With-out their support, constant feedback, and encouragement we would neverhave written this book. In recognition of the efforts of the ACE open-source community, we maintain a list of all contributors that’s availableat http://ace.ece.uci.edu/ACE-members.html .

We are also grateful for the support from colleagues and sponsors ofour research on patterns and development of the ACE toolkit, notably thecontributions of ...

Very special thanks go to....Finally, we would also like to express our gratitude and indebtedness to

the late W. Richard Stevens, the father of network programming literature.His books brought a previously-unknown level of clarity to the art andscience of network programming. We endeavor to stand on his virtualshoulders, and extend the understanding that Richard’s books broughtinto the world of object-oriented design and C++ programming.

Steve’s Acknowledgements

Doug’s Acknowledgements

“hs˙bo2001/page 1

i

i

i

i

i

i

i

i

CHAPTER 1

Object-Oriented Frameworks forNetwork Programming

CHAPTER SYNOPSIS

Object-oriented frameworks help reduce the cost and improve the qualityof networked applications by reifying software designs and pattern lan-guages that have proven effective in particular application domains. Thischapter illustrates what frameworks are and shows how they compare andcontrast with other popular reuse techniques, such as class libraries, com-ponents, and patterns. We then outline the frameworks in ACE that arethe focus of this book. These frameworks are based on a pattern languagethat has been applied to thousands of production networked applicationsand middleware systems world-wide.

1.1 An Overview of Object-Oriented Frameworks

Although computing power and network bandwidth have increased dra-matically in recent years, the design and implementation of networked ap-plication software remains expensive, time-consuming, and error-prone.The cost and effort stems from the increasing demands placed on net-worked software and the continual rediscovery and reinvention of coresoftware design and implementation artifacts throughout the software in-dustry. Moreover, the heterogeneity of hardware architectures, diversity ofOS and network platforms, and stiff global competition are making it in-creasingly hard to build networked application software from scratch while

1


i

i

i

i

i

i

i

i

2 Section 1.1 An Overview of Object-Oriented Frameworks

also ensuring that it has the following qualities:

� Portability, to reduce the effort required to support applications acrossheterogeneous OS platforms, programming languages, and compilers

� Flexibility, to support a growing range of multimedia data types, traf-fic patterns, and end-to-end quality of service (QoS) requirements

� Extensibility, to support successions of quick updates and additionsto take advantage of new requirements and emerging markets

� Predictability and efficiency, to provide low latency to delay-sensitivereal-time applications, high performance to bandwidth-intensive ap-plications, and usability over low-bandwidth network, such as wire-less links

� Reliability, to ensure that applications are robust, fault tolerant, andhighly available and

� Affordability, to ensure that the total ownership costs of softwareacquisition and evolution are not prohibitively high.

Developing application software that achieves these qualities is hard;systematically developing high quality reusable software middleware fornetworked applications is even harder [GHJV95]. Reusable software is in-herently abstract, which makes it hard to engineer its quality and to man-age its production. Moreover, the skills required to develop, deploy, andsupport reusable networked application software have traditionally been a“black art,” locked in the heads of expert developers and architects. Whenthese technical reuse impediments are combined with the myriad of non-technical impediments, such as organizational, economical, administra-tive, political, and psychological factors, it’s not surprising that significantlevels of software reuse have been slow to materialize in most projects andorganizations [Sch00c].

During the past decade, we’ve written hundreds of thousands of linesof C++ code developing widely reusable middleware for networked appli-cations as part of our research and consulting with dozens of telecom-munication, aerospace, medical, and financial service companies. As aresult of our experience, we’ve documented many patterns and patternlanguages [SSRB00] that guided the design of the middleware and applica-tions. In addition, we’ve taught hundreds of tutorials and courses on reuse,middleware, and patterns for thousands of developers and students. De-spite formidable technical and non-technical challenges, we’ve identified asolid body of work that combines design knowledge, hands-on experience,


i

i

i

i

i

i

i

i

Section 1.1 An Overview of Object-Oriented Frameworks 3

and software artifacts that can significantly enhance the systematic reuseof networked application software.

At the heart of this body of work are object-oriented frameworks [FJS99a,FJS99b], which are a powerful technology for achieving systematic reuseof networked application software.1 Figure 1.1 illustrates the following

NETWORKING

DATABASE

GUI

EVENT

LOOP

EVENT

LOOP

EVENT

LOOP

APPLICATION-SPECIFIC

FUNCTIONALITY

CALL

BACKS

DOMAIN-SPECIFIC

FRAMEWORK

CAPABILITIES

CALLBACKS

CALLBACKS

Figure 1.1: Defining Characteristics of a Framework

framework characteristics:1. A framework provides an integrated set of domain-specific struc-

tures and functionality. Systematic reuse of software depends largelyon how well a framework captures the pressure points of stability andvariability in an application domain, such as business data process-ing, telecom call processing, graphical user interfaces, or distributedobject computing middleware.

2. A framework exhibits “inversion of control” at run-time via call-backs. A callback is an object registered with a dispatcher that callsback to a method on the object when a particular event occurs, suchas a connection request or data arriving on a socket handle. Inversionof control decouples the canonical detection, demultiplexing, and dis-patching steps within a framework from the application-defined eventhandlers managed by the framework. When events occur, the frame-work calls back to hook methods in the registered event handlers,which then perform application-defined processing on the events.

3. A framework is a “semi-complete” application that programmerscustomize to form complete applications by inheriting and instantiat-

1In the remainder of this book we use the term framework to mean object-oriented frame-work.


i

i

i

i

i

i

i

i

4 Section 1.2 Applying Frameworks to Network Programming

ing framework classes. Inheritance enables the features of frameworkbase classes to be shared selectively by subclasses. If a base classprovides default implementations of its methods, application develop-ers need only override those virtual methods whose default behavioris inadequate.

These characteristics of a framework yield the following benefits:

� Since a framework reifies the key roles and relationships of classes inan application domain, the amount of reusable code is increased andthe amount of code rewritten for each application is decreased.

� Since a framework exhibits inversion of control, it simplifies applica-tion design since the framework—rather than an application—runsthe event loop that detects events, demultiplexes events to event han-dlers, and dispatches hook methods on the handlers to process theevents.

� Since a framework is a semi-complete application, it enables larger-scale reuse of software than reusing individual classes or stand-alonefunctions because it integrates application-defined and application-independent classes. A framework abstracts the canonical controlflow of applications in a particular domain into families of relatedclasses, which then collaborate to integrate generic and customizableapplication-independent code with concrete customized application-defined code.

Appendix ?? evaluates the pros and cons of frameworks in more detail.

1.2 Applying Frameworks to Network Programming

One reason why it’s hard to write robust, extensible, and efficient net-worked applications is because developers must master many complexnetworking programming concepts and mechanisms, including:

� Network addressing and service identification/discovery� Presentation layer conversions, such as encryption, compression, and

marshaling/demarshaling, to handle heterogeneous end-systems withalternative processor byte-orderings

� Local and remote inter-process communication (IPC) mechanisms� Synchronous and asynchronous event demultiplexing and event han-

dler dispatching and


i

i

i

i

i

i

i

i

Section 1.2 Applying Frameworks to Network Programming 5

� Process/thread lifetime management and synchronization.

Application programming interfaces (APIs) and tools have evolved over theyears to simplify the development of networked applications and middle-ware. Figure 1.2 illustrates the IPC APIs available on many OS platforms,such as UNIX and many real-time operating systems. This figure shows

LEVEL OF

ABSTRACTION

USER

SPACE

KERNEL

SPACE

HOST INFRASTRUCTURE MIDDLEWARE

SOCKETS & TLI

open()/close()/putmsg()/getmsg()

STREAMSFRAMEWORK

TPINPIDLPI

HI

LO

Figure 1.2: Levels of Abstraction for Network Programming

how applications can access networking APIs for local and remote IPC atseveral levels of abstraction. Below, we outline each level of abstraction,starting from the low-level kernel APIs to the native OS user-level network-ing APIs and host infrastructure middleware.

Kernel-level networking APIs. Lower-level networking APIs are availablein an OS kernel’s I/O subsystem. For example, the UNIX putmsg() andgetmsg() system calls can be used to access the transport provider in-terface (TPI) [OSI92b] and the data-link provider interface (DLPI) [OSI92a]available in System V STREAMS [Rit84]. It’s also possible to develop net-work services, such as routers [KMC+00], network file systems [WLS+85],or even Web servers [JKN+01], that reside entirely within an OS kernel.

Programming directly to kernel-level networking APIs is rarely portablebetween different OS platforms. It’s often not even portable across differ-ent versions of the same OS! Since kernel-level programming isn’t usedin most networked applications, we won’t discuss it further in this book.See [Rag93] and [SW93] for coverage of these topics in the context of Sys-tem V UNIX and BSD UNIX, respectively.

User-level networking APIs. Networking protocol stacks in most com-mercial operating systems reside within the protected address space of theOS kernel. Applications running in user-space access the kernel-residentprotocol stacks via OS IPC APIs, such as the Socket or TLI APIs. These


i

i

i

i

i

i

i

i

6 Section 1.2 Applying Frameworks to Network Programming

APIs collaborate with an OS kernel to access the capabilities shown in thefollowing table:

Capability DescriptionLocal contextmanagement

Manage the lifetime of local and remote communication end-points.

Connectionestablishmentandconnectiontermination

Enable applications to establish connections actively or pas-sively with remote peers and to shutdown all or part of theconnections when transmissions are complete.

Optionsmanagement

Negotiate and enable/disable certain options.

Data transfermechanisms

Exchange data between connected peer applications.

Networkaddressing

Convert humanly-readable names to low-level network ad-dresses and vice versa.

These capabilities are described in Chapter 2 of [SH02] in the context ofthe Socket API.

Many IPC APIs are modeled loosely on the UNIX file I/O API, whichdefines the open() , read() , write() , close() , ioctl() , lseek() , andselect() functions [Rit84]. Networking APIs provide additional function-ality that’s not supported directly by the standard UNIX file I/O APIs dueto syntactic and semantic differences between file I/O and network I/O.For example, the pathnames used to identify files on a UNIX system aren’tglobally unique across hosts in a heterogeneous distributed environment.Different naming schemes, such as IP host addresses and TCP/UDP portnumbers, have therefore been devised to uniquely identify communicationendpoints used by networked applications.

Host infrastructure middleware frameworks. Many networked applica-tions exchange messages with clients using various types of synchronousand asynchronous request/response protocols in conjunction with host in-frastructure middleware frameworks. Host infrastructure middleware en-capsulates OS concurrency and IPC mechanisms to automate many te-dious and error-prone aspects of networked application development, in-cluding:

� Connection management and event handler initialization� Synchronous and asynchronous event detection, demultiplexing, and

event handler dispatching


i

i

i

i

i

i

i

i

Section 1.3 An Overview of the ACE Frameworks 7

� Message framing atop bytestream protocols, such as TCP� Presentation conversion issues involving network byte-ordering and

parameter (de)marshaling� Concurrency models and synchronization of concurrent operations� Networked application composition from dynamically configured ser-

vices and� Management of QoS properties, such as scheduling access to proces-

sors, networks, and memory.

The increasing availability and popularity of high-quality and affordablehost infrastructure middleware is helping to raise the level of abstractionat which networked application developers can work effectively. Even so,it’s wise to understand how lower-level IPC mechanisms work to fully com-prehend the challenges that will arise when designing your networked ap-plications.

1.3 An Overview of the ACE Frameworks

The ADAPTIVE Communication Environment (ACE) is a highly portable, widely-used, open-source host infrastructure middleware that can be downloadedfrom http://ace.ece.uci.edu/ or http://www.riverace.com . The ACElibrary contains �240,000 lines of C++ code and �500 classes. To separateconcerns, reduce complexity, and permit functional subsetting, ACE is de-signed using a layered architecture [BMR+96], shown in Figure 1.3. Thefoundation of the ACE toolkit is its combination of an OS adaptation layerand C++ wrapper facades [SSRB00], which together encapsulate core OSconcurrent network programming mechanisms. The higher layers of ACEbuild upon this foundation to provide reusable frameworks, networked ser-vice components, and standards-based middleware. Together, these mid-dleware layers simplify the creation, composition, configuration, and port-ing of networked applications without incurring significant performanceoverhead.

The ACE wrapper facades for native OS IPC and concurrency mech-anisms and the ACE standards-based middleware based upon and bun-dled with ACE were described in [SH02]. This book focuses on the ACEframeworks that help developers produce portable, scalable, efficient, androbust networked applications. These frameworks have also been usedto build higher-level standards-based middleware, such as The ACE ORB


i

i

i

i

i

i

i

i

8 Section 1.3 An Overview of the ACE Frameworks

PROCESSES/THREADS

DYNAMICLINKING

SHAREDMEMORY

SELECT/IO COMP

FILE SYSAPIS

WIN32 NAMEDPIPES & UNIXSTREAM PIPES

UNIXFIFOS

CCAAPPIISS

SOCKETS/TLI

CCOOMMMMUUNNIICCAATTIIOONNSSUUBBSSYYSSTTEEMM

VVIIRRTTUUAALL MMEEMMOORRYY && FFIILLEESSUUBBSSYYSSTTEEMM

GGEENNEERRAALL OOPPEERRAATTIINNGG SSYYSSTTEEMM SSEERRVVIICCEESS

PPRROOCCEESSSS//TTHHRREEAADDSSUUBBSSYYSSTTEEMM

FFRRAAMMEEWWOORRKKLLAAYYEERR

AACCCCEEPPTTOORR CCOONNNNEECCTTOORR

NNEETTWWOORRKKEEDDSSEERRVVIICCEE

CCOOMMPPOONNEENNTTSSLLAAYYEERR

NAMESERVER

TOKENSERVER

LOGGINGSERVER

GATEWAYSERVER

SOCK SAP/TLI SAP

FIFOSAP

LOGMSG

SSEERRVVIICCEEHHAANNDDLLEERR

TIMESERVER

CC++++WWRRAAPPPPEERRFFAACCAADDEELLAAYYEERR SPIPE

SAP

CCOORRBBAAHHAANNDDLLEERR

FILESAP

SHAREDMALLOC

THE ACE ORB(TAO)

JAWS ADAPTIVEWEB SERVER

SSTTAANNDDAARRDDSS--BBAASSEEDD MMIIDDDDLLEEWWAARREE

RREEAACCTTOORR//PPRROOAACCTTOORR

PROCESS/THREADMANAGERS

SSTTRREEAAMMSS

SSEERRVVIICCEECCOONNFFIIGG--UURRAATTOORR

SYNCHWRAPPERS

MEMMAP

OOSS AADDAAPPTTAATTIIOONN LLAAYYEERR

Figure 1.3: The Layered Architecture of ACE

(TAO) [SLM98], which is a CORBA-compliant [Obj01] Object Request Bro-ker (ORB) implemented using the frameworks in ACE. Figure 1.4 illustratesthe ACE frameworks described in this book, which implement a patternlanguage for programming concurrent object-oriented networked applica-tions.

ACE Reactor and Proactor frameworks. These frameworks implementthe Reactor and Proactor patterns [SSRB00], respectively. Reactor is an ar-chitectural pattern that allows event-driven applications to synchronouslyprocess requests that are delivered to an application from one or moreclients. Proactor is an architectural pattern that allows event-driven appli-cations to process requests triggered by the completion of asynchronousoperations to achieve the performance benefits of concurrency without in-curring many of its liabilities. The Reactor and Proactor frameworks auto-mate the detection, demultiplexing, and dispatching of application-definedhandlers in response to various types of I/O-based, timer-based, signal-based, and synchronization-based events. Chapter 3 describes the Reactorframework and Chapter ?? describes the Proactor framework.


i

i

i

i

i

i

i

i

Section 1.3 An Overview of the ACE Frameworks 9

ServiceConfigurator

Acceptor-Connector

Half-SyncHalf-Async

MonitorObject

ProactorReactor

StreamsActiveObject

Figure 1.4: The Frameworks in ACE

ACE Service Configurator framework. This framework implements theComponent Configurator pattern [SSRB00], which is a design pattern thatallows an application to link/unlink its component implementations atrun-time without having to modify, recompile, or relink the applicationstatically. The ACE Service Configurator framework supports the config-uration of applications whose services can be assembled dynamically latein the design cycle, i.e., at installation-time and/or run-time. Applicationswith high availability requirements, such as mission-critical systems thatperform on-line transaction processing or real-time industrial process au-tomation, often require such flexible configuration capabilities. Chapter 4describes this framework in detail.

ACE Task concurrency framework. This framework implements variousconcurrency patterns [SSRB00], such as Active Object and Half-Sync/Half-Async:

� Active Object is a design pattern that decouples the thread that exe-cutes a method from the thread that invoked it. Its purpose is to en-hance concurrency and simplify synchronized access to objects thatreside in their own threads of control.

� Half-Sync/Half-Async is an architectural pattern that decouples asyn-chronous and synchronous processing in concurrent systems, to sim-plify programming without reducing performance unduly. This pat-tern introduces two intercommunicating layers, one for asynchronousand one for synchronous service processing. A queueing layer medi-ates communication between services in the asynchronous and syn-chronous layers.


i

i

i

i

i

i

i

i

10 Section 1.4 Comparing Frameworks with Other Reuse Techniques

Chapter 5 describes the ACE Task framework in detail.

ACE Acceptor-Connector framework. This framework leverages the Re-actor and Proactor frameworks by reifying the Acceptor-Connector pat-tern [SSRB00]. Acceptor-Connector is a design pattern that decouples theconnection and initialization of cooperating peer services in a networkedsystem from the processing they perform once connected and initialized.The Acceptor-Connector framework decouples the active and passive ini-tialization roles from application-defined service processing performed bycommunicating peer services after initialization is complete. Chapter ??describes this framework in detail.

ACE Streams framework. This framework implements the Pipes and Fil-ters pattern [BMR+96], which is an architectural pattern that provides astructure for systems that process a stream of data. The ACE Streamsframework simplifies the development and composition of hierarchically-layered services, such as user-level protocol stacks and network manage-ment agents [SS94]. Chapter ?? describes this framework in detail.

When used together, these ACE frameworks enable the development ofnetworked applications that can be updated and extended without the needto modify, recompile, relink, or restart running applications. ACE achievesthis unprecedented flexibility and extensibility by combining

� C++ language features, such as templates, inheritance, and dynamicbinding [Bja00]

� Patterns, such as Strategy [GHJV95] and Component Configurator[SSRB00] and

� OS mechanisms, such as event demultiplexing, IPC, dynamic linking,multithreading, and synchronization [Ste99].

1.4 Comparing Frameworks with Other Reuse Tech-niques

Object-oriented frameworks don’t exist in isolation. Many other reuse tech-niques are in widespread use, such as class libraries, components, andpatterns. In this section, we compare frameworks with these other reusetechniques to illustrate their similarities and differences.


i

i

i

i

i

i

i

i

Section 1.4 Comparing Frameworks with Other Reuse Techniques 11

Comparing Frameworks and Class Libraries

Class libraries represent the most common first-generation object-orientedreuse technique [Mey97]. A class provides a general-purpose, reusablebuilding block that can be applied across a wide range of applications.Class libraries support component reuse-in-the-small more effectively thanfunction libraries since classes enhance the cohesion of data and methodsthat operate on the data. Their scope is somewhat limited, however, sincethey don’t capture the canonical control flow, collaboration, and variabilityamong families of related software artifacts. Developers who apply classlibrary-based reuse must therefore reinvent and reimplement the overallsoftware architecture and much of the control logic for each new applica-tion.

Frameworks are second-generation reuse techniques that extend thebenefits of class libraries in the following two ways:

1. Frameworks are “semi-complete” applications that embody domain-specific object structures and functionality. Class libraries are oftendomain-independent and provide a fairly limited scope of reuse, e.g., theC++ standard library [Bja00] provides classes for strings, vectors, and var-ious abstract data type (ADT) containers. Although these classes can bereused in most application domains, they are relatively low-level. For ex-ample, application developers are responsible for (re)writing the “glue code”

CALL

BACKS

DATABASE

CLASSES

NETWORK

IPCCLASSES

MATH

CLASSES

ADTCLASSES

GUICLASSES


FUNCTIONALITY

EVENT

LOOP

GLUE

CODE

LOCAL

INVOCATIONS

NETWORKING

DATABASE

GUI

EVENT

LOOP

(1) CLASS LIBRARY ARCHITECTURE

EVENT

LOOP

EVENT

LOOP

APPLICATION-SPECIFIC EVENT HANDLER

FUNCTIONALITY

(2) FRAMEWORK ARCHITECTURE

CALLBACKS

CALLBACKS

Figure 1.5: Class Library vs. Framework Architectures

that performs the bulk of the application control flow and class integration


i

i

i

i

i

i

i

i


logic, as shown in Figure 1.5 (1). As a result, the total amount of reuseis relatively small, compared with the amount of application-defined codethat must be rewritten for each application.

In contrast, classes in a framework collaborate to provide a reusablearchitecture for a family of related applications. Frameworks can be clas-sified by the techniques used to extend them, which range along a con-tinuum from whitebox frameworks to blackbox frameworks [HJE95], asdescribed below:

� Whitebox frameworks—In this type of framework, extensibility isachieved via object-oriented language features, such as inheritanceand dynamic binding. Existing functionality can be reused and cus-tomized by inheriting from framework base classes and overridingpre-defined hook methods [Pre94] using patterns such as TemplateMethod [GHJV95], which defines an algorithm with some steps sup-plied by a derived class. Application developers must have someknowledge of a whitebox framework’s internal structure in order toextend it.

� Blackbox frameworks—In this type of framework, extensibility isachieved by defining interfaces that allow objects to be plugged intothe framework via composition and delegation. Existing functional-ity can be reused by defining classes that conform to a particularinterface and then integrating these classes into the framework us-ing patterns such as Bridge and Strategy [GHJV95], which providea blackbox abstraction for selecting one of many algorithms. Black-box frameworks may be easier to use than whitebox frameworks sinceapplication developers needn’t have as much knowledge of the frame-work’s internal structure. Blackbox frameworks can be harder to de-sign, however, since framework developers must define crisp inter-faces that anticipate a broad range of potential use-cases.

ACE supports both whitebox and blackbox frameworks. For example,its Acceptor-Connector framework described in Chapter ?? defines twotypes of factories that initialize a new endpoint of communication in re-sponse to a connection request from a peer connector:

� The ACE_Acceptor uses the Template Method pattern, which pro-vides a whitebox approach to extensibility, whereas

� The ACE_Strategy_Acceptor uses the Bridge and Strategy patterns,which provide a blackbox approach to extensibility [Sch00b].


i

i

i

i

i

i

i

i


Complete networked applications can be composed by customizing theclasses and frameworks in ACE.

2. Frameworks are active and exhibit “inversion of control” at run-time. Classes in a class library are typically passive, i.e., they performtheir processing by borrowing the thread of control from self-directed ap-plications that invoke their methods. As a result, developers must continu-ally rewrite much of the control logic needed to “glue” the reusable classestogether to form complete networked applications.

In contrast, frameworks are active, i.e., they direct the flow of controlwithin an application via various callback event handling patterns, suchas Reactor, Proactor [SSRB00], and Observer [GHJV95]. These patternsinvert the application’s flow of control using a design technique known asthe Hollywood Principle, i.e., “Don’t call us, we’ll call you” [Vli98]. Sinceframeworks are active and manage the application’s control flow, they canperform a broader range of activities on behalf of applications than is pos-sible with passive class libraries.

All the ACE frameworks provide inversion of control via callbacks, asshown in the following table:

ACE Framework Inversion of ControlReactor andProactor

Call back to application-supplied event handlers to per-form processing when events occur synchronously andasynchronously, respectively.

Service Configurator Calls back to application-supplied service objects to ini-tialize, suspend, resume, and finalize them.

Task Calls back to an application-supplied hook method toperform processing in one or more threads of control.

Acceptor-Connector Calls back to service handlers in order to initialize themafter they’ve been connected.

Streams Calls back to initialize and finalize tasks when they arepushed and popped from a stream.

In practice, frameworks and class libraries are complementary tech-nologies. As shown in Figure 1.6 for instance, the ACE toolkit simplifies theimplementation of its frameworks via its class libraries of containers, suchas queues, hash tables, and other ADTs. Likewise, application-definedcode invoked by event handlers in the ACE Reactor framework can usethe ACE wrapper facades presented in [SH02] and the C++ standard li-brary classes [Jos99] to perform IPC, synchronization, file management,and string processing operations.


i

i

i

i

i

i

i

i


AC

ES

tream

s

ACE Reactor

EVENT

LOOP

EVENT

LOOP

APPLICATION-SPECIFIC EVENT HANDLER

FUNCTIONALITY

CALL

BACKSLOCAL

INVOCATIONS

IPCCLASSES

ADTCLASSES

ACE Task

EVENT

LOOP

CALLBACKS

CALLBACKS

Figure 1.6: Applying Class Libraries to Develop and Use ACE Frame-works

Comparing Frameworks and Components

A component is an encapsulated part of a software system that implementsa specific service or set of services. A component has one or more interfacesthat provide access to its services. Components serve as building blocks forthe structure of a system and can be reused based solely upon knowledgeof their interface protocols. Components can also be plugged in and/orscripted together to form complete applications, as shown in Figure 1.7.Common examples of components include ActiveX controls, CORBA object

NAMING

TRADING

LOCKING


FUNCTIONALITY

EVENT

LOOP

GLUE

CODE

REMOTE OR

LOCAL

INVOCATIONS

LOGGING

TIME

Figure 1.7: Component Architecture

services [Obj98], and JavaBeans [CL97].Components are less lexically and spatially coupled than frameworks.

For example, applications can reuse components without having to sub-class them from existing base classes. In addition, by applying patternslike Proxy [GHJV95] and Broker [BMR+96], components can often be dis-


i

i

i

i

i

i

i

i


tributed to servers throughout a network and accessed by clients remotely.ACE provides naming, event routing [Sch00a], logging, time synchroniza-tion, and network locking components in its networked service componentslayer outlined in Chapter 0 of [SH02].

The relationship between frameworks and components is highly syner-gistic, with neither subordinate to the other [Joh97]. For example, a mid-dleware framework can be used to develop higher-level application com-ponents, where component interfaces provide a facade for the internalclass structure of the framework. Likewise, components can be used aspluggable strategies in blackbox frameworks [HJE95]. Frameworks areoften used to simplify the development of middleware component mod-els [Ann98, BEA99], whereas components are often used to simplify thedevelopment and configuration of networked application software.

Comparing Frameworks and Patterns

Developers of networked applications must address design challenges re-lated to complex topics like connection management, service initialization,distribution, concurrency control, flow control, error handling, event loopintegration, and dependability. These challenges are often independentof application-defined requirements. Successful developers resolve thesechallenges by applying the following types of patterns [BMR+96]:

� Design patterns, which provide a scheme for refining componentsof a software system or the relationships between them and describea commonly-recurring structure of communicating components thatsolves a general design problem within a particular context.

� Architectural patterns, which express fundamental structural or-ganization schemas for software systems and provide a set of prede-fined subsystems, specify their responsibilities, and include rules andguidelines for organizing the relationships between them.

� Pattern languages, which define a vocabulary for talking about soft-ware development problems and provide a process for the orderly res-olution of these problems.

Traditionally, patterns and pattern languages have been locked in theheads of expert developers or buried deep within complex system sourcecode. Allowing this valuable information to reside only in these locationsis risky and expensive, however. Capturing and documenting patterns fornetworked applications therefore helps to


i

i

i

i

i

i

i

i

16 Section 1.5 Example: A Networked Logging Service

� Preserve important design information for programmers who en-hance and maintain existing software. If this information isn’t docu-mented explicitly it’ll be lost over time. In turn, this increases softwareentropy and decreases software maintainability and quality since sub-stantial effort may likewise be necessary to reverse engineer the pat-terns from existing source code.

� Guide design choices for developers who are building new networkedapplications. Since patterns document the common traps and pitfallsin their domain, they help developers select suitable architectures,protocols, algorithms, and platform features without wasting timeand effort (re)implementing solutions that are known to be inefficientor error-prone.

Knowledge of patterns and pattern languages helps to reduce develop-ment effort and maintenance costs. Reuse of patterns alone, however, isnot sufficient to create flexible and efficient networked application soft-ware. Although patterns enable reuse of abstract design and architectureknowledge, software abstractions documented as patterns don’t yield re-usable code directly. It’s therefore essential to augment the study of pat-terns with the creation and use of frameworks. Frameworks help develop-ers avoid costly reinvention of standard software artifacts by implementingcommon pattern languages and refactoring common implementation roles.

1.5 Example: A Networked Logging Service

Throughout this book, we illustrate key points and ACE capabilities by ex-tending and enhancing the networked logging service example presented in[SH02]. This service collects and records diagnostic information sent fromone or more client applications. Unlike the versions in [SH02], which werea subset of the actual networked logging service in ACE, this book illus-trates the full complement of capabilities, patterns, and daemons providedby ACE. Sidebar 1 defines what a daemon is and explains the daemonprocess.

Figure 1.8 illustrates the application processes and daemons in ournetworked logging service, which are described below.

Client application processes run on client hosts and generate log recordsranging from debugging messages to critical error messages. The logging


i

i

i

i

i

i

i

i

Section 1.5 Example: A Networked Logging Service 17

Sidebar 1: Daemons and Daemonizing

A daemon is a server process that runs continuously in the backgroundperforming various services on behalf of clients [Ste98]. Daemonizing aUNIX process involves the following steps:

1. Dynamically spawning a new server process2. Closing all unnecessary I/O handles3. Changing the current filesystem directory away from the initiating

user’s4. Resetting the file access creation mask5. Disassociating from the controlling process group and the controlling

terminal and6. Ignoring terminal I/O-related events and signals.

An ACE server can convert itself into a daemon on UNIX by invoking thestatic method ACE::daemonize() . A Win32 Service [Ric97] is a form ofdaemon and can be programmed in ACE using the ACE_NT_Serviceclass.

information sent by a client application indicates the following:

1. The time the log record was created2. The process identifier of the application3. The priority level of the log record and4. A string containing the logging message text, which can vary in size

from 0 to a maximum length, such as 4 Kbytes.

Client logging daemons run on every host machine participating in thenetworked logging service. Each client logging daemon receives log recordsfrom client applications via some form of local IPC mechanism, such asshared memory, pipes, or sockets. The client logging daemon convertsheader fields from the received log records into network-byte order anduses a remote IPC mechanism, such as TCP/IP, to forward each record toa server logging daemon running on a designated host.

Server logging daemons collect and output the incoming log records theyreceive from client applications via client logging daemons. A server log-ging daemon can determine which client host sent each message by usingaddressing information it obtains from the underlying Socket API. There’s


i

i

i

i

i

i

i

i

18 Section 1.6 Summary

int spawn (void) { if (ACE_OS::fork () == -1) ACE_ERROR (LM_ERROR, "unable to fork in function spawn");

SERVER

CLIENTTANGO

CLIENTMAMBO

LOCAL IPC CLIENT

LOGGING

DAEMON

P1

P3P2

HOST A HOST BTANGO MAMBO

SERVER LOGGING

DAEMON

STORAGE DEVICE

PRINTER

LOCAL IPC

CLIENT

LOGGING

DAEMON

P1

P3

P2

Oct 31 14:50:28 [email protected]@18352@2@drwho::sending request to server tangoOct 31 14:48:13 [email protected]@38491@7@client::unable to fork in function spawn

CCOONNSSOOLLEE

NETWORK

if (Options::instance ()->debug()) ACE_DEBUG ((LM_DEBUG, "sending request to server %s", server_host));

TCP CONNECTION

TCP

CO

NN

ECTI

ON

Figure 1.8: Processes and Daemons in the Networked Logging Service

generally one server logging daemon per system configuration, though theycan be replicated to enhance fault tolerance.

1.6 Summary

The traditional method of continually re-discovering and re-inventing coreconcepts and capabilities in networked application software has kept thecosts of engineering these systems unnecessarily high for too long. Object-oriented frameworks are crucial to improving networked application devel-opment processes by reducing engineering cycle time and enhancing soft-ware quality and performance. A framework is a reusable, “semi-complete”application that can be specialized to produce custom applications [JF88].

Frameworks can be applied together with patterns and componentsto improve the quality of networked applications by capturing success-ful software development strategies. Patterns systematically capture ab-stract designs and software architectures in a format that’s intended to becomprehensible to developers. Frameworks and components reify concrete


i

i

i

i

i

i

i

i

Section 1.6 Summary 19

patterns, designs, algorithms, and implementations in particular program-ming languages.

The ACE toolkit provides over a half dozen high-quality frameworks thathave been developed and refined over scores of person years. As shown inlater chapters, ACE serves as an excellent case study of how to reap thebenefits of—and reduce your exposure to the risks of—framework usage.As you read the remainder of the book, keep in mind the following points:

The ACE frameworks are much more than a class library. The ACEframeworks are a collection of classes that collaborate to provide semi-complete networked applications. Whereas the ACE library container classesdescribed in [?] and the wrapper facades described in [SH02] are passive,the ACE frameworks are active and exhibit inversion of control at run-time. The ACE toolkit provides both frameworks and a library of classesto help programmers address a broad range of challenges that arise whendeveloping networked applications.

The ACE frameworks provide a number of benefits, including improvedreusability, modularity, and extensibility. The inversion of control inthe ACE frameworks augments these benefits by separating application-defined concerns, such as event processing, from core framework con-cerns, such as event demultiplexing and event handler dispatching. A lesstangible—but no less valuable—benefit of the ACE frameworks is the trans-fer of decades of accumulated knowledge from ACE framework developersto ACE framework users in the form of expertise embodied in well-tested,easily reusable C++ software artifacts.

Framework developers must resolve many design challenges. One ofthe most critical design challenges is determining which classes in a frame-work should be stable and which should be variable. Insufficient stabilitymakes it hard for users to understand and apply the framework effectively,which makes the framework hard to use and may not satisfy the QoSrequirements of performance-sensitive applications. Conversely, insuffi-cient variation makes it hard for users to customize framework classes,which results in a framework that can’t accommodate the functional re-quirements of diverse applications.

Framework development and training can be expensive. It took scoresof person years to develop and mature the ACE frameworks. This develop-ment effort has been amortized over thousands of users, however, who can


i

i

i

i

i

i

i

i


take advantage of the expertise available from the core ACE developmentteam and experienced programmers throughout the ACE community. Thisleveraging of expertise is one of the key benefits of open-source projects.Developers can become more proficient with ACE by

� Incrementally learning enough to complete the tasks at hand via themany examples and tutorials in ACE and

� Leveraging the goodwill and knowledge of the ACE user communityvia the ACE mailing lists and USENET newsgroup, which are de-scribed at http://ace.ece.uci.edu/ACE/ .

Foresight and experience help with integratability. Getting frameworksto play together nicely with other frameworks, class libraries, and legacysystems can be hard. ACE’s layered architecture provides a good exampleto follow since it enables integration at several levels of abstraction. Forexample, applications can integrate with ACE at its wrapper facade level,its framework level, or at its standards-based middleware level. Differentapplications benefit from integrating with ACE at different levels.

Framework maintenance and validation is challenging. A framework’sextensibility features, such as hierarchies of abstract classes and templateparameterization, can make a priori validation hard. Moreover, frameworkevolution requires detailed knowledge of the areas being worked on, as wellas how various framework and application classes collaborate and inter-act. Addressing these challenges effectively requires experienced develop-ers and a large set of automated regression tests. ACE’s open-source de-velopment model is also useful since the large ACE user base provides ex-tensive field testing, along with a collective body of experienced and skilleddevelopers to suggest fixes for problems that arise.

Micro-level efficiency concerns are usually offset by other macro frame-work benefits. The ACE framework’s virtual methods and additional lev-els of redirection may incur some micro-level performance degradation oncertain OS/compiler platforms. The expertise applied to the ACE frame-work’s design and implementation, however, often makes up for this over-head. For example, it may be possible to substitute completely differ-ent concurrency and synchronization strategies without affecting appli-cation functionality, thereby providing macro-level optimizations. ManyACE frameworks also contain novel optimizations that may not be com-mon knowledge to application developers. Finally, the added productivity


i

i

i

i

i

i

i

i


benefits of employing the ACE framework usually offset any minor perfor-mance overheads.


i

i

i

i

i

i

i

i


i

i

i

i

i

i

i

i

CHAPTER 2

Service and Configuration DesignDimensions

CHAPTER SYNOPSIS

A service is a set of functionality offered to a client by a service provideror server. A networked application can be created by configuring its con-stituent services together at various points of time, such as compile-time,static link-time, installation/boot-time, or even at run-time. This chapterpresents a domain analysis of service and configuration design dimensionsthat address key networked application properties, including duration andstructure, how networked services are identified, and the time at whichthey are bound together to form complete applications.

2.1 Service Design Dimensions

This section covers the following service design dimensions:

� Short- vs. long-duration services

� Internal vs. external services

� Stateful vs. stateless services

� Layered/modular vs. monolithic services

� Single- vs. multi-service servers

� One-shot vs. standing servers

23


i

i

i

i

i

i

i

i

24 Section 2.1 Service Design Dimensions

2.1.1 Short- vs. Long-Duration Services

The services offered by network servers can be classified loosely as short-duration or long-duration. These two different time durations help to deter-mine which protocols to use, as outlined below. The time reflects how longthe service holds system resources, and should be evaluated in relation toserver startup and shutdown requirements.

Short-duration services execute in brief, often fixed, amounts of time andoften handle only a single request. Examples of relatively short-durationservices include computing the current time of day, resolving the Ethernetnumber of an IP address, and retrieving a disk block from a network fileserver. To minimize the amount of time spent setting up a reliable connec-tion, these services are often implemented using connectionless protocols,such as UDP/IP.

Long-duration services execute for extended, often variable, lengths oftime and may handle numerous requests during their lifetime. Examplesof long-duration services include transferring a large file via FTP, down-loading streaming video over HTTP, accessing host resources remotely viaTELNET, or performing remote file system backups over a network. To im-prove efficiency and reliability, these services are often implemented withconnection-oriented protocols, such as TCP/IP.

Logging service ) From the standpoint of an individual log record, ourserver logging daemon appears to be a short-duration service. The sizeof each log record is bounded by its maximum length (e.g., 4 Kbytes) andmost messages are much smaller. The actual time spent handling a logrecord is relatively short. Since a client may transmit many log records ina session, however, we optimize performance by designing client loggingdaemons to open connections with their peer server logging daemons andthen reusing these connections for many logging requests. It would bewasteful and time-consuming to set up and tear down a socket connectionfor each logging request, particularly if they are sent frequently. Thus, wemodel our server logging daemon as a long-duration service.

2.1.2 Internal vs. External Services

Services can be classified as internal or external. The primary tradeoffsin this dimension are service initialization time, safety of one service from


i

i

i

i

i

i

i

i

Section 2.1 Service Design Dimensions 25

another, and simplicity.

Internal services execute within the same address space as the server thatreceives the request, as shown in Figure 2.1 (1). As described in Section ??,

(2) EXTERNAL SERVICES

select()

(1) INTERNAL SERVICES

DISPATCHER PROCESS DISPATCHER PROCESS

SVC1 SVC2 SVC3

SVC1 SVC2

select()

Figure 2.1: Internal vs. External Services

an internal service can run iteratively or concurrently in relation to otherinternal services.

External services execute in different process address spaces. For in-stance, Figure 2.1 (2) illustrates a master service process that monitors aset of network ports. When a connection request arrives from a client, themaster accepts the connection and then spawns a new process to performthe requested service externally.

Some server frameworks support both internal and external services.For example, system administrators can choose between internal and ex-ternal services in INETD by modifying the inetd.conf configuration file asfollows:

� INETD can be configured to execute short-duration services, such asECHO and DAYTIME, internally via calls to functions that are staticallylinked into the INETD program and

� INETD can also be configured to run longer-duration services, such asFTP and TELNET, externally by spawning separate processes.

Internal services usually have lower initialization latency, but also mayreduce application robustness since separate functions within a processaren’t protected from one another. For instance, one faulty service cancorrupt data shared with other internal services in the process, which mayproduce incorrect results, crash the process, or cause the process to hang


i

i

i

i

i

i

i

i


indefinitely. To increase robustness, therefore, mission-critical applicationservices are often implemented externally in separate processes.

Logging service ) Most of our logging servers examples in this book aredesigned as an internal service. As long as only one type of service isconfigured into our logging server we needn’t protect it from harmful side-effects of other services. There are valid reasons to protect the processing ofdifferent client sessions from each other, however, so Chapter ?? illustrateshow to implement our logging server as an external service.

2.1.3 Stateful vs. Stateless Services

Services can be classified as stateful or stateless. The amount of state,or context, information that a service maintains between requests impactsboth service and client complexity and resource consumption.

Stateful services cache certain information, such as session state, au-thentication keys, identification numbers, and I/O handles, in a serverto reduce communication and computation overhead. For instance, Webcookies enable state to be preserved on a Web server across multiple pagerequests.

Stateless services retain no volatile state within a server. For example, theNetwork File System (NFS) provides distributed data storage and retrievalservices that don’t maintain volatile state information within a server’s ad-dress space. Each request sent from a client is completely self-containedwith the information needed to carry it out such as the file handle, bytecount, starting file offset, and user credentials.

Stateful and stateless services trade off efficiency and reliability, withthe right choice depending on a variety of factors, such as the probabilityand impact of host and network failures. Some common network appli-cations, such as FTP and TELNET, don’t require retention of persistent ap-plication state information between consecutive service invocations. Thesestateless services are generally fairly simple to configure and reconfigurereliably. Conversely, a middleware service like CORBA Naming [Obj98]manages various bindings whose values must be retained even if the servercontaining the service crashes.

Logging service ) Our networked logging service is a stateless service. Thelogging server processes each record individually without respect to any


i

i

i

i

i

i

i

i


previous or possible future request. The need for any possible requestordering is not a factor since TCP/IP is used, providing an ordered, reliablecommunication stream.

2.1.4 Layered/Modular vs. Monolithic Services

Service implementations can be classified as layered/modular or mono-lithic. The primary tradeoffs in this dimension are service reusability, ex-tensibility, and efficiency.

Layered/modular services decompose into a series of partitioned and hi-erarchically-related tasks. For instance, application families, such as PBXnetwork management services [SS94], can be specified and implementedas layered/modular services, as illustrated in Figure 2.2 (1). Each layer

(1) LAYERED/MODULARSERVICES

(2) MONOLITHICSERVICES

MODULE2

MODULE3

MODULE1

MODULE4

SVC1W SVC1R

SVC2W

SVC4W

SVC2R

SVC4R

SVC3W SVC3R

SVC1

SVC2

SVC4

SVC3

GLOBAL DATA

MSG

MSG

MSG

Figure 2.2: Layered/Modular vs. Monolithic Services

can handle a self-contained portion of the overall service, such as inputand output, event analysis and service processing, and event filtering.Inter-connected services can collaborate by exchanging control and datamessages for incoming and outgoing communication.

Over the years, many communication frameworks have been devel-oped to simplify and automate the development and configuration of lay-ered/modular services [SS93]. Well-known examples include Systems VSTREAMS [Rit84], the x-kernel [HP91], and the ACE Streaming frameworkcovered in Chapter ??. In general, these frameworks decouple the protocol


i

i

i

i

i

i

i

i


and service functionality from the following non-functional service designaspects:

1. Compositional strategies, such as the time and/or order in whichservices and protocols are composed together

2. Concurrency and synchronization strategies, such as task- vs.message-based architectures (described in Section ??) used to exe-cute services at run-time [SS95b].

Monolithic services are tightly coupled clumps of functionality that aren’torganized hierarchically. They may contain separate modules of function-ality that vaguely resemble layers, but are most often tightly data coupledvia shared, global variables, as shown in Figure 2.2 (2). They are also of-ten tightly functionally coupled, with control flow diagrams that look likespaghetti. Monolithic services are hard to understand, maintain, and ex-tend. While they may sometimes be appropriate in short-lived, ‘throw-away’ prototypes1 [FY99], they are rarely suitable for software that mustbe maintained and enhanced by multiple developers over time.

Developers can often select either layered or monolithic service archi-tectures to structure their networked applications. There are several ad-vantages to designing layered/modular services:

� Layering enhances reuse since multiple higher-layer application com-ponents can share lower-layer services

� Implementing applications via an inter-connected series of layeredservices enables transparent, incremental enhancement of their func-tionality

� A layered/modular architecture facilitates macro-level performanceimprovements by allowing the selective omission of unnecessary ser-vice functionality and

� Modular designs generally improve the implementation, testing, andmaintenance of networked applications and services.

There can also be some disadvantages with using a layered/modulararchitecture to develop networked applications:

� The modularity of layered implementations can introduce excessiveoverhead, e.g., layering may cause inefficiencies if buffer sizes don’t

1After you become proficient with the ACE toolkit it’ll be much faster to build a properlylayered prototype than to hack together a monolithic one.


i

i

i

i

i

i

i

i


match in adjacent layers, thereby causing additional segmentation,reassembly, and transmission delays

� Communication between layers must be designed and implementedproperly, which can introduce another source of errors and

� Information hiding within layers can make it hard to allocate andmanage resources predictably and dependably in applications withstringent real-time and dependability requirements.

It may seem odd to consider the tradeoffs between layered/modular andmonolithic services in a book on advanced object-oriented design. Yet manyservices are implemented monolithically—without regard to structure orgood design principles—even after years of evangelizing higher-level designmethods based on advances in object-oriented techniques. While mono-lithic service designs may occasionally make sense, e.g., when prototypingnew capabilities, they are ultimately like the goto —a powerful mechanismthat can easily hurt you.

Logging service ) By carefully separating design concerns, our server log-ging daemon is designed using a layered/modular architecture that con-sists of the following layers:

1. Event infrastructure layer—The classes in this layer perform applic-ation-independent strategies for detecting and demultiplexing eventsand dispatching them to their associated event handlers. Chapter 3describes how the Reactor pattern and ACE Reactor framework canbe applied to implement a generic event infrastructure layer.

2. Configuration management layer—The classes in this layer performapplication-independent strategies to install, initialize, control, andterminate service components. Chapter ?? describes how the Com-ponent Configurator pattern and ACE Service Configurator frameworkcan be applied to implement a generic configuration managementlayer.

3. Connection management layer—The classes in this layer performapplication-independent connection and initialization services thatare independent of application functionality. Chapter ?? describeshow the Acceptor-Connector pattern and the ACE Acceptor-Connectorframework can be applied to implement a generic connection manage-ment layer.

4. Application layer—The classes in this layer customize the application-independent classes provided by the other layers to create concrete


i

i

i

i

i

i

i

i


objects that configure applications, process events, establish connec-tions, exchange data, and perform logging-specific processing. Theremaining chapters in this book illustrate how to implement theseapplication-level capabilities using the ACE frameworks.

2.1.5 Single- vs. Multi-Service Servers

Protocols and services rarely operate in isolation, but instead are accessedby applications within the context of a server. Servers can be designed aseither single-service or multi-service. The tradeoffs in this dimension areresource consumption vs. robustness.

Single-service servers offer only one service. As shown in Figure 2.3 (1),a service can be internal or external, but there’s only a single service per

INTERNAL

SERVICE

INTERNAL

SERVICE

EXTERNAL

SERVICE EXTERNAL

SERVICE EXTERNAL

SERVICE EXTERNAL

SERVICE

(2) MULTI-SERVICE SERVER(1) SINGLE-SERVICE SERVERS

SLAVES

INTERNAL

SERVICE

EXTERNAL

SERVICE

EXTERNAL

SERVICE

MASTERINTERNAL

SERVICE

Figure 2.3: Single-service vs. Multi-service Servers

process. Examples of single-service servers include:

� The RWHO daemon (RWHOD), which reports the identity and numberof active users, as well as host workloads and host availability.

� Early versions of UNIX ran standard network services, such as FTP

and TELNET, which ran as distinct single-service daemons that wereinitiated at OS boot-time [Ste98].

Each instance of these single-service servers execute externally in a sep-arate process. As the number of system servers increases, however, thisstatically configured, single-service per-process approach incurred the fol-lowing limitations:


i

i

i

i

i

i

i

i


� It consumed excessive amounts of OS resources, such as virtual mem-ory and process table slots

� It caused redundant initialization and networking code to be writtenseparately for each service program

� It required running processes to be shutdown and restarted manuallyto install new service implementations and

� It led to ad hoc and non-uniform administrative mechanisms beingused to control different types of services.

Multi-service servers address the limitations with single-service serversby integrating a collection of single-service servers into a single adminis-trative unit, as shown in Figure 2.3 (2). This multi-service design yieldsthe following benefits:

� It reduces the consumption of OS resources by allowing servers to bespawned on-demand

� It simplifies server development and reuses common code by auto-matically (1) daemonizing a server process (as described in Sidebar 1),(2) initializing transport endpoints, (3) monitoring ports, and (4) de-multiplexing/dispatching client requests to service handlers

� It allows external services to be updated without modifying existingsource code or terminating running server processes and

� It consolidates network service administration via a uniform set ofconfiguration management utilities, e.g., the INETD super-server pro-vides a uniform interface for coordinating and initiating external ser-vices, such as FTP and TELNET, and internal services, such as DAYTIME

and ECHO.

Logging service ) Our implementations of the networked logging servicein [SH02] used a pair of single-service servers, i.e., one for the client loggingdaemon and one for the server logging daemon. In this book, we’ll enhanceour implementation so that the various entities in the networked loggingservice can be configured via a multi-service super-server similar to INETD.

2.1.6 One-shot vs. Standing Servers

Servers can also be designed as either one-shot or standing. The primarytradeoffs in this dimension involve how long the server runs and uses sys-tem resources, and should be evaluated in relation to server startup andshutdown requirements.


i

i

i

i

i

i

i

i


One-shot servers are spawned on-demand, e.g., by a super-server likeINETD. They perform service requests in a separate thread or process, asshown in Figure 2.4 (1). One-shot servers terminate after the request that

SVC4

SVC3

(1) ONE-SHOT SERVER (2) STANDING SERVER

SUPER-SERVER PROCESS

LOCALIPC

SUPER-SERVER PROCESS

SVC1

SVC2

SVC2SVC3

SVC1

Figure 2.4: One-shot vs. Standing Servers

triggered their creation completes. This design strategy can consume fewersystem resources, such as virtual memory and process table slots, sinceservers don’t remain in system memory when they are idle.

Standing servers continue to run beyond the lifetime of any particularservice request they process. Standing servers are often initiated at boot-time or by a super-server after the first client request. They may receiveconnection and/or service requests via local IPC channels, such as namedpipes or sockets, that are attached to a super-server, as shown in Fig-ure 2.4 (2). Alternatively, a standing server may take ownership of, or in-herit, an IPC channel from the original service invocation. Standing serverscan improve service response time by amortizing the cost of spawning aserver process or thread over a series of client requests. In addition, theyenable applications to reuse endpoint initialization, connection establish-ment, endpoint demultiplexing, and service dispatching code.

Logging service ) We implement the client and server logging daemonsin our networked logging service as standing servers since they run long-duration services that process requests from many client applications be-fore terminating. In general, the choice between one-shot or standingservers is orthogonal to the choice between short- or long-duration ser-vices described in Section 2.1.1. The former design alternative reflects OSresource management constraints, whereas the latter design alternative isa property of a service.


i

i

i

i

i

i

i

i

Section 2.2 Conguration Design Dimensions 33

2.2 Configuration Design Dimensions

This section covers the following configuration design dimensions:

� Static vs. dynamic naming� Static vs. dynamic linking� Static vs. dynamic configuration

2.2.1 Static vs. Dynamic Naming

Applications can be categorized according to whether their services arenamed statically or dynamically. The primary tradeoff in this dimensioninvolves efficiency vs. flexibility.

Statically named services associate the name of a service with objectcode that exists at compile-time and/or static link-time. For example, IN-ETD’s internal services, such as ECHO and DAYTIME, are bound to staticallynamed built-in functions stored internally within the INETD program.

Dynamically named services defer the association of a service namewith the object code that implements the service. Thus, code needn’t beidentified—nor even be written, compiled, and linked—until an applicationbegins executing the corresponding service at run-time. A common ex-ample of dynamic naming is demonstrated by INETD’s handling of TELNET,which is an external service. External services can be updated by modi-fying the inetd.conf configuration file and sending the SIGHUP signal tothe INETD process. When INETD receives this signal, it re-reads its con-figuration file and dynamically rebinds the services it offers to their newexecutables.

2.2.2 Static vs. Dynamic Linking

Applications can also be categorized according to whether their servicesare linked into a program image statically or dynamically. The primarytradeoffs in this dimension involve efficiency, security, and extensibility.

Static linking creates a complete executable program by binding togetherall its object files at compile-time and/or static link-time, as shown inFigure 2.5 (1).


i

i

i

i

i

i

i

i

34 Section 2.2 Conguration Design Dimensions

(1) STATIC LINKING (2) DYNAMIC LINKING

OBJECT

CODE

MODULES

OBJECT

CODE

STUBS

SHARED

OBJECTS

IN DLLS

OBJECT

CODE

STUBS

Figure 2.5: Static Linking vs. Dynamic Linking

Dynamic linking inserts object files into and removes object files fromthe address space of a process when a program is invoked initially or up-dated at run-time, as shown in Figure 2.5 (2). Modern operating systemsgenerally support both implicit and explicit dynamic linking:

� Implicit dynamic linking defers most address resolution and reloca-tion operations until a function is first referenced. This lazy evalua-tion strategy minimizes link editing overhead during server initializa-tion. Implicit dynamic linking is used to implement shared libraries,also known as dynamic-link libraries (DLLs) [Sol98]. Ideally, only onecopy of DLL code exists, regardless of the number of processes thatexecute library code simultaneously. DLLs can therefore reduce thememory consumption of both a process in memory and its programimage stored on disk.

� Explicit dynamic linking allows an application to obtain, use, and/orremove the run-time address bindings of certain function- or data-related symbols defined in DLLs. Common explicit dynamic linkingmechanisms include

– The POSIX/UNIX dlopen() , dlsym() , and dlclose() functionsand

– The Win32 LoadLibrary() , GetProcAddress() , and FreeLibrary()functions.

Developers must consider tradeoffs between flexibility, time and spaceefficiency, security, and robustness carefully when choosing betweendynamic and static linking [SSRB00].


i

i

i

i

i

i

i

i

Section 2.2 Conguration Design Dimensions 35

2.2.3 Static vs. Dynamic Configuration

As described in Section 2.1, networked applications generally offer or usecommunication services. Popular services available on the Internet todayinclude

� Web browsing and content retrieval services, e.g., Alta Vista, Apache,and Netscape’s HTTP server

� Software distribution services, e.g., Castanet� Electronic mail and network news transfer services, e.g., sendmail

and nntpd

� File access on remote machines, e.g., ftpd

� Network time protocols, e.g., ntpd

� Payment processing services, e.g., Cybercash and� Streaming audio/video services, e.g., RealAudio, RealSystem, and Re-

alPlayer.

By combining the naming and linking dimensions described above, we canclassify these types of networked application services as being either stat-ically or dynamically configured. The primary tradeoffs in this dimensioninvolve efficiency, security, and extensibility.

Static configuration refers to the process of initializing an application thatcontains statically named services, i.e., developing each service as a sepa-rate function or class and then compiling, linking, and executing them ina separate OS process. In this case, the services in the application aren’textensible at run-time. This design may be necessary for secure applica-tions that contain only trusted services. Statically configured applicationsmay also be more efficient since code generated by compilers can eliminateindirections required to support relocatable code.

However, statically configuring services can also yield non-extensibleapplications and software architectures. The main problems with staticconfigurations are:

� It tightly couples the implementation of a particular service with theconfiguration of the service with respect to other services in a net-worked application and

� It severely limits the ability of system administrators to change theparameters or configuration of a system to suit local operating condi-tions or changing network and hardware configurations.


i

i

i

i

i

i

i

i

36 Section 2.2 Conguration Design Dimensions

Dynamic configuration refers to the process of initializing an applica-tion that offers dynamically named services. When combined with ex-plicit dynamic linking and process/thread creation mechanisms, the ser-vices offered by dynamically configured applications can be extended atinstallation/boot-time or even during run-time. This degree of extensibilityhelps facilitate the following configuration-related activities:

� Functional subsetting—Dynamic configuration simplifies the stepsnecessary to produce subsets of functionality for application familiesdeveloped to run on a range of OS platforms. Explicit dynamic linkingenables the fine-grain addition, removal, or modification of services.In turn, this allows the same framework to be used for space-efficientembedded applications and for large enterprise distributed applica-tions. For example, a web browsing application may be able to run onPDAs, PCs, and/or workstations by dynamically configuring subsets,such as image rendering, Java capability, printing, or direct phonenumber dialing.

� Application workload balancing—It’s often hard to determine therelative processing characteristics of application services a priori sinceworkloads can vary at run-time. It may therefore be necessary to ex-periment with alternative load balancing techniques [OOS01] and sys-tem configurations that locate application services on different hostmachines throughout a network. For example, developers may havethe opportunity to place certain services, such as image processing,on either side of a client/server boundary. Bottlenecks may result ifmany services are configured into a server application and too manyactive clients access these services simultaneously. Conversely, con-figuring many services into clients can result in a bottleneck if clientsexecute on cheaper, less powerful machines.

� Dynamic service reconfiguration—Highly available networked ap-plications, such as mission-critical systems that perform on-line trans-action processing or real-time process control, may require flexibledynamic reconfiguration management capabilities. For example, itmay be necessary to phase new versions of a service into a server ap-plication without disrupting other services that it’s already executing.Re-configuration protocols [SSRB00] based on explicit dynamic link-ing mechanisms can enhance the functionality and flexibility of net-worked applications since they enable services to be inserted, deleted,


i

i

i

i

i

i

i

i


or modified at run-time without first terminating and restarting theunderlying process or thread(s) [SS94].

Logging service ) The implementations of our networked logging servicein [SH02] and Chapter 3 are all configured statically. In Chapter ?? in thisbook, we describe

� The ACE_DLL class, which is a wrapper facade class that portablyencapsulates the ability to load/unload dynamically linked libraries(DLLs) and find symbols in them and

� The ACE Service Configuration framework, which can be used to con-figure application services dynamically.

Starting with Chapter ??, all our examples are configured dynamically.

2.3 Summary

The way in which application services are structured, instantiated, andconfigured has a significant impact on how effectively they use system andnetwork resources. Efficient resource usage is closely linked to applica-tion response time, and overall system performance and scalability. Asimportant as performance and scalability are, however, a coherent andmodular design is often just as important to maintain and extend applica-tions over time. Fortunately, performance vs. modularity tradeoffs needn’tbe an either/or proposition. By studying the design dimensions carefullyin this chapter and applying the ACE wrapper facades and frameworksjudiciously, you’ll be able to create well-designed and highly efficient net-worked applications.

Configuration and service location/naming are important topics to un-derstand and use in the quest to reduce the effects of inherent complexity.In this chapter, we described the two key design dimensions in this areaas

1. Identifying a particular set of services and2. Linking these services into the address space of one or more applica-

tions.

The remaining chapters in the book describe the ACE frameworks that reifythese design dimensions.


i

i

i

i

i

i

i

i


i

i

i

i

i

i

i

i

CHAPTER 3

The ACE Reactor Framework

CHAPTER SYNOPSIS

This chapter describes the design and use of the ACE Reactor frame-work. This framework implements the Reactor pattern [SSRB00], whichallows event-driven applications to process events delivered from one ormore clients. In this chapter, we show how to implement a logging serverwith a reactor that detects and demultiplexes different types of connectionand data events from various event sources and dispatches the events toapplication-defined handlers that process the events.

3.1 Overview

The ACE Reactor framework simplifies the development of event-drivenprograms, which characterize many networked applications. Commonsources of events in these applications include I/O operations, signals,and expiration of timers. In this context, the ACE Reactor framework isresponsible for

� Detecting the occurrence of events from various event sources� Demultiplexing the events to their pre-registered event handlers and� Dispatching methods on the handlers, which process the events in an

application-defined manner.

This chapter describes the following ACE Reactor framework classesthat networked applications can use to detect the occurrence of eventsand then demultiplex and dispatch the events to their event handlers:

39


i

i

i

i

i

i

i

i

40 Section 3.1 Overview

ACE Class DescriptionACE_Time_Value Provides a portable representation of time that uses

C++ operator overloading to simplify the time-relatedarithmetic and relational operations.

ACE_Event_Handler Defines an abstract interface for processing varioustypes of I/O, timer, and signal events.

ACE_Timer_HeapACE_Timer_ListACE_Timer_WheelACE_Timer_Hash

Implementations of various ACE timer queues.

ACE_Reactor The public interface to the ACE Reactor framework.ACE_Select_Reactor An ACE_Reactor implementation that uses the

select() synchronous event demultiplexer function todetect I/O and timer events.

ACE_TP_Reactor An ACE_Reactor implementation that uses theselect() function to detect events and theLeader/Followers pattern to process the events ina pool of threads.

ACE_WFMO_Reactor An ACE_Reactor implementation that uses theWaitForMultipleObjects() event demultiplexerfunction to detect I/O, timer, and synchronizationevents.

Figure 3.1: The ACE Reactor Framework Classes

The most important relationships between the classes in the ACE Re-actor framework are shown in Figure 3.1. The Reactor pattern describedin [SSRB00] divides its participants into two layers:

� Event infrastructure layer classes that perform application-inde-pendent strategies for demultiplexing indication events to event han-dlers and then dispatching the associated event handler hook meth-ods. The infrastructure layer components in the ACE Reactor frame-work include the various implementations of the ACE_Reactor andthe ACE timer queue.

� Application layer classes that define concrete event handlers thatperform application-defined processing in their hook methods. In theACE Reactor framework, all application layer components are descen-dants of ACE_Event_Handler .

The ACE Reactor framework provides the following benefits:


i

i

i

i

i

i

i

i

Section 3.1 Overview 41

� Broad portability—The framework can be configured to use a widerange of OS synchronous event demultiplexing mechanisms, such asselect() , which is available on UNIX, Win32, and many real-timeoperating systems, and WaitForMultipleObjects() , which is avail-able only on Win32.

� Automates event detection, demultiplexing, and dispatching—By eliminating the reliance on non-portable native OS synchronousevent demultiplexing APIs, the ACE Reactor framework provides ap-plications with a uniform object-oriented event detection, demulti-plexing, and dispatching mechanism. Event handlers can be writtenin C++ and registered with the ACE_Reactor to process sets of desiredevents.

� Transparent extensibility—The framework employs hook methodsvia inheritance and dynamic binding to decouple

– Lower-level event mechanisms, such as detecting events on mul-tiple I/O handles, expiring timers, and demultiplexing and dis-patching methods of the appropriate event handler to processthese events, from

– Higher-level application event processing policies, such as connec-tion establishment strategies, data marshaling and demarshal-ing, and processing of client requests.

This design allows the ACE Reactor framework to be extended trans-parently without modifying or recompiling existing application code.

� Increase reuse and minimize errors—Developers who write pro-grams using native OS synchronous event demultiplexing operationsmust reimplement, debug, and optimize the same low-level code foreach application. In contrast, the ACE Reactor framework’s eventdetection, demultiplexing, and dispatching mechanisms are genericand can therefore be reused by many networked applications, whichallows developers to focus on higher-level application-defined eventhandler policies, rather than wrestling repeatedly with low-level mech-anisms.

� Efficient demultiplexing—The framework performs its event demul-tiplexing and dispatching logic efficiently. For instance, the ACE_Select_Reactor presented in Section 3.6 uses the ACE_Handle_Set_Iterator class described in Chapter 7 of [SH02], which uses an opti-


i

i

i

i

i

i

i

i

42 Section 3.2 The ACE Time Value Class

mized implementation of the Iterator pattern [GHJV95] to avoid exam-ining fd_set bitmasks one bit at a time. This optimization is based ona sophisticated algorithm that uses the C++ exclusive-or operator toreduce run-time complexity from O(number of total bits) to O(number ofenabled bits), which can substantially improve run-time performancefor large-scale applications.

The remainder of this chapter motivates and describes the capabilitiesof each class in the ACE Reactor framework. We illustrate how this frame-work can be used to enhance the design of our networked logging server.Section ?? describes design rules to follow when applying the ACE Reactorframework.

3.2 The ACE Time Value Class

Motivation

Different operating systems provide different functions and data structuresto access and manipulate date and time information. For example, UNIXplatforms define the timeval structure as follows:

struct timeval {long secs;long usecs;

};

Different date and time representations are used on other OS platforms,such as POSIX, Win32, and proprietary real-time operating systems. Ad-dressing these portability differences in each application is unnecessarilycostly, which is why the ACE Reactor framework provides the ACE_Time_Value class.

Class Capabilities

The ACE_Time_Value class applies the Wrapper Facade pattern [SSRB00]and C++ operator overloading to simplify the use of portable time-relatedoperations. This class provides the following capabilities:

� It provides a standardized representation of time that’s portable acrossOS platforms


i

i

i

i

i

i

i

i

Section 3.2 The ACE Time Value Class 43

Figure 3.2: The ACETime Value Class

� It uses operator overloading to simplify time-based comparisons bypermitting standard C++ syntax for time-based arithmetic and rela-tional expressions and

� Its constructors and methods normalize time quantities by convertingthe fields in a timeval structure into a canonical encoding schemethat ensures accurate comparisons.

The interface for the ACE_Time_Value class is shown in Figure 3.2 andits key methods are outlined in the following table:

Method DescriptionACE_Time_Valueset()

Constructors and methods that convert from various timeformats, such as timeval or long , to a normalized ACE_Time_Value .

sec() Return the number of seconds in an ACE_Time_Value .usec() Return the number of microseconds in an ACE_Time_

Value .operator +=operator -=operator *=

Arithmetic methods that add, subtract, and multiply anACE_Time_Value .

In addition to the methods shown above, the following binary operators arefriends of the ACE_Time_Value class that define arithmetic and relationaloperations:

Method Descriptionoperator +operator -

Arithmetic methods that add and subtract two ACE_Time_Value s.

operator ==operator !=

Methods that compare two ACE_Time_Value s for equality andinequality.

operator <operator >operator <=operator >=

Methods that determine relationships between two ACE_Time_Value s.

All ACE_Time_Value constructors and methods normalize the time val-ues they operate upon. For example, after normalization, the quantity


i

i

i

i

i

i

i

i

44 Section 3.3 The ACE Event Handler Class

ACE_Time_Value(1,1000000) will compare equal to ACE_Time_Value(2) .In contrast, a direct bitwise comparison of these non-normalized class val-ues won’t detect their equality.

Example

The following code creates two ACE_Time_Value objects, which are con-structed by adding user-supplied command-line arguments to the currenttime. The appropriate ordering relationship between the two ACE_Time_Value s is then displayed:

#include "ace/OS.h"

int main (int argc, char *argv[]){

if (argc != 3)ACE_ERROR_RETURN ((LM_ERROR,

"usage: %s time1 time2\n", argv[0]), 1);

ACE_Time_Value curtime = ACE_OS::gettimeofday ();ACE_Time_Value t1 =

curtime + ACE_Time_Value (ACE_OS::atoi (argv[1]));ACE_Time_Value t2 =

curtime + ACE_Time_Value (ACE_OS::atoi (argv[2]));

if (t1 > t2) cout << "timer 1 is greater" << endl;else if (t2 > t1) cout << "timer 2 is greater" << endl;else cout << "timers are equal" << endl;return 0;

}

This program behaves identically on all OS platforms where ACE has beenported. Sidebar 2 describes how to build the ACE library so that you canexperiment with the examples we present in this book.

3.3 The ACE Event Handler Class

Motivation

Networked applications are often written as reactive servers, which re-spond to various types of events, such as I/O events, timer events, andsignals. One way to program reactive applications is to define a separate


i

i

i

i

i

i

i

i

Section 3.3 The ACE Event Handler Class 45

Sidebar 2: Building ACE and Programs that Use ACE

ACE is open-source software: you can download it from http://ace.ece.uci.edu and build it yourself. Here are some tips to help you under-stand the source examples we show, and how to build ACE, the examples,and your own applications:

� ACE should be installed into an empty directory. The top-level direc-tory in the distribution is named ACE_wrappers . We refer to this top-level directory as “$ACE_ROOT.” You should create an environmentvariable by that name containing the full path to the top-level ACEdirectory.

� The ACE source and header files reside in $ACE_ROOT/ace.� This book’s networked logging service example source and header

files reside in $ACE_ROOT/examples/C++NPv2 .� When compiling your programs, the $ACE_ROOTdirectory must be

added to your compiler’s file include path, which is often designatedby the -I or /I compiler option.

� The $ACE_ROOT/ACE-INSTALL.html file contains instructions onbuilding and installing ACE and programs that use ACE.

You can also purchase a prebuilt version of ACE from Riverace at a nom-inal cost. A list of the prebuilt compiler and OS platforms supported byRiverace is available at http://www.riverace.com .

function for each type of event. This approach can become unwieldy, how-ever, since application programmers are responsible for explicitly

1. Keeping track of which functions correspond to which events and

2. Associating data with the functions.

To alleviate both problems, the ACE Reactor framework defines the ACE_Event_Handler base class.

Class Capabilities

The ACE_Event_Handler base class is the root of all event handlers inACE. This class provides the following capabilities:


i

i

i

i

i

i

i

i


Figure 3.3: The ACEEvent Handler Class

� It defines hook methods for input events, output events, exceptionevents, timer events, and signal events1

� It simplifies the association of data with methods that manipulate thedata

� It centralizes how event handlers can be destroyed when they are nolonger needed and

� It holds a pointer to the ACE_Reactor that manages it.

The interface for the ACE_Event_Handler class is shown in Figure 3.3and its key methods are outlined in the following table:

Method Descriptionhandle_input() Hook method called when input events occur, e.g., con-

nection or data events.handle_output() Hook method called when output events are possible,

e.g., when flow control abates or a non-blocking con-nection completes.

handle_exception() Hook method called when an exceptional event occurs,e.g., a SIGURG signal.

handle_timeout() Hook method called when a timer expires.handle_signal() Hook method called when signaled by the OS, either via

POSIX signals or when a Win32 synchronization objecttransitions to the signaled state.

handle_close() Hook method that performs user-defined terminationactivities when one of the handle_*() hook meth-ods outlined above returns �1 or when a remove_handler() method is called explicitly to unregister anevent handler.

get_handle() Returns the underlying I/O handle.reactor() Accessors to get/set the ACE_Reactor associated with

an ACE_Event_Handler .

Applications can inherit from ACE_Event_Handler to create concreteevent handlers, which have the following properties:

1On Win32, an ACE_Event_Handler can also handle synchronization events, suchas transitioning from the non-signaled to signaled state with Win32 mutexes orsemaphores [Sol98].


i

i

i

i

i

i

i

i


� They override one or more of the ACE_Event_Handler ’s handle_*()virtual hook methods to perform application-defined processing in re-sponse to the corresponding types of events.

� They are registered with an ACE_Reactor , which then uses the ACE_Event_Handler interface to dispatch hook methods on event han-dlers that process events.

� Since concrete event handlers are objects rather than functions, it’sstraightforward to associate data with a handler’s hook methods tohold state across multiple callbacks by an ACE_Reactor .

When an application registers a concrete event handler with a reactor, itmust indicate what type(s) of event(s) the event handler should process. Todesignate these event types, ACE defines a typedef called ACE_Reactor_Mask and the ACE_Event_Handler defines the following set of enumera-tion literals that can be passed as the values of the ACE_Reactor_Maskparameter:

Event Type DescriptionREAD MASK Indicates input events, such as data on a socket or file han-

dle. A reactor dispatches the handle_input() hook methodto process input events.

WRITE MASK Indicates output events, such as when flow control abates.A reactor dispatches the handle_output() hook method toprocess output events.

EXCEPT MASK Indicates exceptional events, such as urgent data on asocket. A reactor dispatches the handle_except() hookmethod to process exceptional events.

ACCEPT MASK Indicates passive-mode connection events. A reactor dis-patches the handle_input() hook method to process con-nection events.

CONNECT MASK Indicates a non-blocking connection completion. A reactordispatches the handle_output() hook method to processnon-blocking connection completion events.

All these * MASK enumeration values are defined as powers of two so theirbits can be “or’d” together efficiently to designate a set of events.

Concrete event handlers used for I/O events provide a handle, suchas a socket handle, that can be retrieved via the get_handle() hookmethod. When an application registers a concrete event handler with areactor, the reactor calls back to the handler’s get_handle() method toretrieve the underlying handle. This method can be left as a no-op if a con-crete event handler only handles time-driven events. For example, the ACE


i

i

i

i

i

i

i

i


timer queue classes described in Section 3.4 use concrete event handlersto process time-driven events. When a timer managed by this mechanismexpires, the handle_timeout() method of the associated event handler isinvoked by the reactor.

The reactor interprets the return values of the handle_*() hook meth-ods as follows:

� Return value of 0—When a handle_*() method returns 0 this in-forms the reactor that the event handler wishes to remain registeredwith the reactor. The reactor will therefore continue to include thehandle of this event handler next time its handle_events() methodis invoked. This behavior is common for event handlers whose lifetimeextends beyond a single handle_*() method dispatch.

� Return greater than 0—When a handle_*() method returns a valuegreater than 0 this informs the reactor that the event handler wishesto be dispatched again before the reactor blocks on its event demul-tiplexer. This feature is useful for cooperative event handlers to en-hance overall system fairness since it allows one event handler to per-form a limited amount of computation, relinquish control, and thenallow other event handlers to be dispatched before it retains controlagain.

� Return less than 0—When a handle_*() method returns a valueless than 0 this informs the reactor that the event handler wants to beremoved from the reactor’s internal tables. In this case, the reactor in-vokes the event handler’s handle_close() hook method and passesit the ACE_Reactor_Mask value corresponding to the handle_*()hook method that returned �1. In addition to the event types shownin the table on page 47, the reactor can pass the following enumera-tion values defined in ACE_Event_Handler :

Event Type DescriptionTIMER MASK Indicates time-driven events. A reactor dispatches the

handle_timeout() hook method to process timeoutevents.

SIGNAL MASK Indicates signal-based events (or synchronization-based events on Win32). A reactor dispatches thehandle_signal() hook method to process signalevents.

The handle_close() method can perform user-defined terminationactivities, such as deleting dynamic memory allocated by the object

‘‘hs _2001 /page

i

i

i

i

i

i

i

i


or closing log files. After the handle_close() method returns, thereactor will remove the associated concrete event handler from itsinternal tables if the handler is no longer registered for any events.

Example

We’ll implement our networked logging server by subclassing from ACE_Event_Handler and driving its processing via the ACE_Reactor ’s eventloop. We have two types of events to handle:

1. Events indicating the arrival of new connections and2. Events indicating the arrival of log records from connected clients.

We therefore define two types of event handlers:

� Logging_Handler_Adapter —This class processes log records from aconnected client. It uses the ACE_SOCK_Streamclass from Chapter 3in [SH02] to receive log records from its client.

� Logging_Acceptor —This class is a factory that dynamically allo-cates a Logging_Handler_Adapter and initializes it with a newlyconnected client. It uses the ACE_SOCK_Acceptor class from Chapter3 in [SH02] to initialize the Logging_Handler_Adapter ’s ACE_SOCK_Stream .

Both of these classes inherit from ACE_Event_Handler , which enablestheir handle_input() methods to be dispatched automatically by an ACE_Reactor , as described in Section 3.5.

We start by creating a file called Logging_Acceptor.h that includesthe necessary header files:

#include "ace/Event_Handler.h"#include "ace/INET_Addr.h"#include "ace/Log_Record.h"#include "ace/Reactor.h"#include "ace/FILE.h"#include "ace/SOCK_Acceptor.h"#include "ace/SOCK_Stream.h"#include "Logging_Handler.h"

The Logging_Handler.h file was defined in Chapter 4 of [SH02] and allother files are defined in the ACE toolkit.

We next show the interface and portions of the implementation of meth-ods in Logging_Acceptor . Although we don’t show much error handling


i

i

i

i

i

i

i

i


code in this example, a production implementation should take appropri-ate corrective action if failures occur.

class Logging_Acceptor : public ACE_Event_Handler{public:

Logging_Acceptor (ACE_Reactor *r = ACE_Reactor::instance ()): ACE_Event_Handler (r) {}

// Initialize a passive-mode acceptor socket. The <local_addr>// is the address that we’re going to listen for connections on.virtual int open (const ACE_INET_Addr &local_addr) {

return peer_acceptor_.open (local_addr);}

// Return the connected socket’s I/O handle.virtual ACE_HANDLE get_handle () const{ return peer_acceptor_.get_handle (); }

// Called by a reactor when there’s a new connection to accept.virtual int handle_input (ACE_HANDLE h = ACE_INVALID_HANDLE);

// Called when object is destroyed, e.g., when it’s removed// from a reactor.virtual int handle_close (ACE_HANDLE = ACE_INVALID_HANDLE,

ACE_Reactor_Mask = 0) {peer_acceptor_.close ();

}

// Returns a reference to the underlying <peer_acceptor_>.ACE_SOCK_Acceptor &acceptor () const {

return peer_acceptor_;}

private:// Factory that connects <ACE_SOCK_Stream>s passively.ACE_SOCK_Acceptor peer_acceptor_;

};

As shown on page 72, a Logging_Acceptor will be registered with anACE_Reactor to handle ACCEPT events. Since the passive-mode socket inthe ACE_SOCK_Acceptor becomes active when a new connection can be ac-cepted, the reactor dispatches the Logging_Acceptor::handle_input()

‘‘hs _2001 /page

i

i

i

i

i

i

i

i


method automatically. We’ll show this method’s implementation on page 52after defining the following Logging_Handler_Adapter class:

class Logging_Handler_Adapter : public ACE_Event_Handler{protected:

// File where log records are written.ACE_FILE_IO log_file_;

// Connection to remote peer.Logging_Handler logging_handler_;

This class inherits from ACE_Event_Handler and adapts the Logging_Handler defined in Chapter 4 of [SH02] for use with the ACE Reactorframework. In addition to a Logging_Handler , each Logging_Handler_Adapter contains an ACE_FILE_IO object to keep separate log files for eachconnected client.

The public methods in the Logging_Handler_Adapter class are shownbelow.

public:// Constructor.Logging_Handler_Adapter (ACE_Reactor *r)

: ACE_Event_Handler (r),logging_handler_ (log_file_) {}

// Activate the object.virtual int open ();

// Get the I/O handle of the contained Logging_Handler.virtual ACE_HANDLE get_handle () const{ return peer ().get_handle (); };

// Called when input events occur, e.g., connection or data.virtual int handle_input (ACE_HANDLE h = ACE_INVALID_HANDLE);

// Called when object is destroyed, e.g., when it’s removed// from an ACE_Reactor.virtual int handle_close (ACE_HANDLE = ACE_INVALID_HANDLE,

ACE_Reactor_Mask = 0);

// Get a reference to the contained ACE_SOCK_Stream.

‘‘hs _2001 /page

i

i

i

i

i

i

i

i


ACE_SOCK_Stream &peer () const{ return logging_handler_.peer (); };

};

Now that we’ve shown the interface of Logging_Handler_Adapter , wecan implement Logging_Acceptor::handle_input() , which is called backautomatically by a reactor whenever a new connection can be accepted.This handle_input() method is a factory method that performs the stepsnecessary to create, connect, and activate a Logging_Handler_Adapter ,as shown below:

1 int Logging_Acceptor::handle_input (ACE_HANDLE)2 {3 Logging_Handler_Adapter *peer_handler;45 ACE_NEW_RETURN (peer_handler,6 Logging_Handler_Adapter (reactor ()),7 -1);89 if (peer_acceptor_.accept (peer_handler->peer ()) == -1) {

10 delete peer_handler;11 return -1;12 }13 else if (peer_handler->open () == -1)14 peer_handler->close ();15 return 0;16 }

Lines 3–7 Create a new Logging_Handler_Adapter that will process thenew client’s logging session.

Lines 9–12 Accept the new connection into the socket handle of theLogging_Handler_Adapter .

Lines 13–14 Activate the connected Logging_Handler_Adapter by in-voking its open() method:

1 int Logging_Handler_Adapter::open ()2 {3 char filename[MAXHOSTNAMELEN + sizeof (".log")];4 ACE_INET_Addr logging_peer_addr;56 logging_handler_.peer ().get_remote_addr (logging_peer_addr);


i

i

i

i

i

i

i

i


7 logging_peer_addr.get_host_name (filename, MAXHOSTNAMELEN);8 strcat (filename, ".log");9

10 ACE_FILE_Connector connector;11 connector.connect (log_file_,12 ACE_FILE_Addr (filename),13 0, // No timeout.14 ACE_Addr::sap_any, // Ignored.15 0, // Don’t try to reuse the addr.16 O_RDWR|O_CREAT|O_APPEND,17 ACE_DEFAULT_FILE_PERMS);1819 return reactor ()->register_handler20 (this, ACE_Event_Handler::READ_MASK);21 }

Lines 3–8 Determine the host name of the connected client and use thisas the filename of the log file.

Lines 10–17 Create or open the file that’s used to store log records froma connected client.

Lines 19–20 Use the ACE_Reactor::register_handler() method toregister this event handler for READ events with the same reactor usedby the Logging_Acceptor . This method is described on page 62 in Sec-tion 3.5.

When log records arrive from clients, the reactor will dispatch Logging_Handler_Adapter::handle_input() automatically. This method processesa single log record by calling Logging_Handler::log_record() , whichreads the record from the socket and writes it to the log file associated withthe client connection, as shown below:

int Logging_Handler_Adapter::handle_input (ACE_HANDLE){

return logging_handler_.log_record ();}

Since logging_handler_ maintains its own socket handle, the handle_input() method ignores its ACE_HANDLEparameter.

Whenever an error occurs or a client closes a connection to the loggingserver, the log_record() method returns �1, which the handle_input()method then passes back to the reactor that dispatched it. In turn, this


i

i

i

i

i

i

i

i

54 Section 3.4 The ACE Timer Queue Classes

value causes the reactor to dispatch the Logging_Handler_Adapter::handle_close() hook method, which closes both the socket to the clientand the log file and then deletes itself, as follows:

int Logging_Handler_Adapter::handle_close (ACE_HANDLE,ACE_Reactor_Mask)

{logging_handler_.close ();log_file_.close ();delete this;return 0;

}

The use of delete this in handle_close() is valid since this Logging_Handler_Adapter object is allocated dynamically and will no longer be ref-erenced or used by the reactor or any other part of the program. Additionalrules for managing the removal of concrete event handlers are described inSection ?? on page ??.

3.4 The ACE Timer Queue Classes

Motivation

Many networked applications require support for time-driven dispatching.For example, web servers require watch-dog timers that release resourcesif clients don’t send an HTTP GET request shortly after they connect. Like-wise, the Windows NT Service Control Manager [Sol98] requires servicesunder its control to report their status periodically via “heartbeat” mes-sages so it can restart services that have terminated abnormally. To allevi-ate application developers from the burden of developing efficient, scalable,and portable time-driven dispatchers in an ad hoc manner, the ACE Reac-tor framework defines a family of timer queue classes.

Class Capabilities

The ACE timer queue classes allow applications to register time-driven con-crete event handlers derived from ACE_Event_Handler . These classes pro-vide the following capabilities:


i

i

i

i

i

i

i

i

Section 3.4 The ACE Timer Queue Classes 55

� They allow applications to schedule event handlers whose handle_timeout() hook method will be dispatched efficiently and scalably atcaller-specified times in the future, either once or at periodic intervals

� They allow applications to cancel a particular timer associated withan event handler or all timers associated with an event handler and

� They provide a means to configure a timer queue so it can use varioustime sources, such as ...

Figure 3.4: The ACE Timer Queue Classes

The interfaces and relationships of all the ACE timer queue classes areshown in Figure 3.4. The key methods in these classes are outlined in thefollowing table:

Method Descriptionschedule() Schedule an event handler whose handle_timeout() method

will be dispatched at a caller-specified time in the future, eitheronce or at periodic intervals.

cancel() Cancel a timer associated with a particular event handler or alltimers associated with an event handler.

expire() Dispatch the handle_timeout() method of all event handlerswhose expiration time is less than or equal to the current timeof day, which is represented as an absolute value, e.g., 2001-09-11-08.45.00.

The schedule() method is passed a pointer to an ACE_Event_Handlerand a reference to an ACE_Time_Value indicating the absolute point oftime in the future when the handle_timeout() hook method is invoked onthe event handler. This method can also be passed the following optionalparameters:

� A void pointer that’s stored internally by the timer queue and passedback unchanged when the handle_timeout() method is dispatched.This pointer can be used as an asynchronous completion token (ACT)[SSRB00], which allows an application to efficiently demultiplex andprocess the responses of asynchronous operations it invokes on ser-vices. This capability allows the same event handler to be registeredwith a timer queue at multiple dispatching times in the future.


i

i

i

i

i

i

i

i


� An ACE_Time_Value that designates the interval at which the eventhandler should be dispatched periodically. If this parameter is omit-ted, the event handler’s handle_timeout() method is just dispatchedonce.

When a timer queue dispatches an event handler’s handle_timeout()method, it passes the current time and the void pointer ACT that waspassed as a parameter to the schedule() method when the event handlerwas scheduled originally.

The return value of schedule() uniquely identifies each event handlerthat’s scheduled with a timer queue. Sidebar 3 explains the recycling policyfor these timer identifiers. Applications can pass the unique timer identifierto the cancel() method to remove a particular event handler before itexpires. Applications can also pass the address of the handler to cancel()to remove all timers associated with a particular event handler. If a non-NULL void pointer is passed to cancel() , it’s assigned the ACT passedby the application when the timer was scheduled originally. This makes itpossible to allocate ACTs without dynamically incurring memory leaks.

Sidebar 3: Recycling of Timer Identifiers

The ACE timer queue classes combine design patterns, hook methods,and template arguments to provide the following timer queue implementa-tions:

� ACE_Timer_Heap , which is a partially-ordered, almost-complete bi-nary tree implemented in an array [Rob99]. Its average- and worst-case performance for scheduling, canceling, and expiring a concreteevent handler is O(lgn). Heap-based timer queues are therefore use-ful for applications [BL88] and middleware [SLM98] that require pre-dictable and low-latency time-driven dispatching, which is why it’sthe default timer queue mechanism in the ACE Reactor framework.

� ACE_Timer_Wheel , which uses timing wheels [VL97] that contain acircular buffer designed to schedule, cancel, and dispatch timers inO(1) time in the average case, but O(n) in the worst-case.

� ACE_Timer_Hash , which uses a hash table to manage the queue. Likethe timing wheel implementation, the average-case time required toschedule, cancel, and expire timers is O(1) and its worst-case is O(n).


i

i

i

i

i

i

i

i


� ACE_Timer_List , which is implemented as a linked list of absolutetimers ordered by increasing deadline. Its average and worst-caseperformance for scheduling and canceling timers is O(n), but it usesthe least amount of memory of the ACE timer queue implementations.

The ACE Reactor framework allows developers to use any of these timerqueue implementations to achieve the functionality needed by their appli-cations, without burdening them with a “one size fits all” implementation.Since the methods in the ACE_Timer_Queue base class are virtual, appli-cations can provide different implementations of other timer queue mech-anisms, such as delta lists [Com84]. Like the ACE_Timer_List , a delta liststores event handlers in a list ordered by increasing deadline. Rather thanbeing absolute time, however, the delay for each event is computed fromexpiration of previous event. The first element of the list can therefore bechecked/dispatched in O(1) time, though insertion and deletion of a timermay require O(n) time.

Another example of flexibility in the ACE Reactor framework is the timesource used by an ACE timer queue. Most ACE timer queues internallyuse the absolute time of day. While this implementation is fine for manyapplications, ACE also provides a hook method that allows applicationsto configure a different time source for a timer queue. For example, theperformance counter on Win32 bases the time on system uptime ratherthan on wallclock time. System uptime is useful in situations where thesystem’s time-of-day can be adjusted, but timers must not be affected bychanges to the time-of-day clock.

Example

Although the Logging_Acceptor and Logging_Handler_Adapter eventhandlers in Section 3.3 implement the logging server functionality cor-rectly, they may consume system resources unnecessarily. For example,clients can connect to a server and then not send log records for long pe-riods of time. In the example below, we illustrate how to apply the ACEtimer queue mechanisms to reclaim resources from those event handlerswhose clients log records infrequently. Our design is based on the Evictorpattern [HV99], which describes how and when to release resources, suchas memory and I/O handles, to optimize system resource management.

We use the Evictor pattern in conjunction with the ACE Reactor frame-

‘‘hs _2001 /page

i

i

i

i

i

i

i

i


work’s timer queue mechanisms to check periodically when each registeredevent handler has received its last client log record. If the time the last logrecord was received exceeds a designated threshold, the event handler isdisconnected from its client, its resources are returned to the OS, and it’sremoved from the reactor. Clients are responsible for detecting these dis-connections and re-establishing them when they need to send more logrecords.

To implement the Evictor pattern, we extend the Logging_Acceptorand Logging_Handler_Adapter classes shown in Section 3.3 to create theLogging_Acceptor_Ex and Logging_Handler_Adapter_Ex classes. Wethen register a timer for every instance of Logging_Handler_Adapter_Ex .Since the default ACE timer mechanism (ACE_Timer_Heap ) is highly scal-able and efficient, its scheduling, cancellation, and dispatching overheadis minimal regardless of the number of registered timers.

We start by creating a new header file called Logging_Acceptor_Ex.hthat contains the new subclasses. The changes to the Logging_Acceptor_Ex class are minor. We simply override and modify the handle_input()method to create a Logging_Handler_Adapter_Ex class rather than aLogging_Handler_Adapter , as shown below:

#include "Logging_Acceptor.h"

class Logging_Acceptor_Ex : public Logging_Acceptor{public:

int handle_input (ACE_HANDLE) {Logging_Handler_Adapter_Ex *peer_handler;

ACE_NEW_RETURN (peer_handler,Logging_Handler_Adapter_Ex (reactor ()),-1);

// ... same as Logging_Acceptor::handle_input()}

};

In Chapter ??, we’ll illustrate how the ACE Acceptor-Connector frameworkcan be used to add new behavior to an event handler without copying ormodifying existing code.

We then extend the Logging_Handler_Adapter to create the followingLogging_Handler_Adapter_Ex class:

‘‘hs _2001 /page

i

i

i

i

i

i

i

i


class Logging_Handler_Adapter_Ex: public Logging_Handler_Adapter

{private:

// Time when a client last sent a log record.ACE_Time_Value time_of_last_log_record_;

// Maximum timeout.const ACE_Time_Value max_client_timeout_;

// Max timeout is an hour.enum { MAX_CLIENT_TIMEOUT = 3600 };

// Private destructor ensures dynamic allocation.˜Logging_Handler_Adapter () {}

We implement the Evictor pattern by adding an ACE_Time_Value thatkeeps track of the time when a client last sent a log record. The meth-ods in the public interface of Logging_Handler_Adapter_Ex are shownbelow:

public:Logging_Handler_Adapter_Ex

(ACE_Reactor *r,const ACE_Time_Value &max_client_timeout

= ACE_Time_Value (MAX_CLIENT_TIMEOUT)): Logging_Handler_Adapter (r),

time_of_last_log_record (0),max_client_timeout_ (max_client_timeout) {}

// Activate the object.virtual int open ();

// Called when input events occur, e.g., connection or data.virtual int handle_input (ACE_HANDLE);

// Called when a timeout expires to check if the client has// been idle for an excessive amount of time.virtual int handle_timeout (const ACE_Time_Value &tv,

const void *act);

// Called when object is destroyed, e.g., when it’s removed// from an ACE_Reactor.

‘‘hs _2001 /page

i

i

i

i

i

i

i

i


virtual int handle_close (ACE_HANDLE h = ACE_INVALID_HANDLE,ACE_Reactor_Mask close_mask = 0);

};

The handle_input() method notes the time when a log record is re-ceived from the connected client and then forwards to its parent’s handle_input() method to process the log record, as shown below:

int Logging_Handler_Adapter_Ex::handle_input (ACE_HANDLE h){

time_of_last_log_record_ = ACE_OS::gettimeofday ();return Logging_Handler_Adapter::handle_input (h);

}

The open() method is shown next:

int Logging_Handler_Adapter_Ex::open (){

int result = Logging_Handler_Adapter::open ();if (result != -1)

result = reactor ()->schedule_timer(this,

max_client_timeout_,0,max_client_timeout_);

return result;}

This method first forwards to its parent’s open() method, which is definedon page 52. It then calls the ACE_Reactor::schedule_timer() method(described on page 65) to schedule this event handler to be dispatchedperiodically to check whether its client sent it a log record recently. Weschedule the initial timer to expire in max_client_timeout_ seconds andalso request that it re-expire periodically every max_client_timeout_ sec-onds. When a timer expires, the reactor uses its timer queue mechanismto dispatch the following handle_timeout() hook method automatically:

int Logging_Handler_Adapter_Ex::handle_timeout(const ACE_Time_Value &tv, const void *act)

{if (ACE_OS::gettimeofday () - time_of_last_log_record_

> max_client_timeout_)


i

i

i

i

i

i

i

i

Section 3.5 The ACE Reactor Class 61

reactor ()->remove_handler (this, ACE_Event_Handler::READ_MASK);return 0;

}

This method checks if the time elapsed between the current time andwhen the last log record was received by this event handler is greater thandesignated max_client_timeout_ threshold. If so, it calls the remove_handler() method, which triggers the reactor to call the following handle_close() hook to remove the event handler from the reactor:

int Logging_Handler_Adapter_Ex::handle_close (ACE_HANDLE,ACE_Reactor_Mask)

{reactor ()->cancel_timer (this);return Logging_Handler_Adapter::handle_close ();

}

This method cancels the timer for this event handler and then calls itsparent’s handle_close() method, which closes the socket to the clientand the log file, and then deletes itself.

3.5 The ACE Reactor Class

Motivation

Event-driven networked applications have historically been programmedusing native OS mechanisms, such as the Socket API and the select()synchronous event demultiplexer. These implementations are often inflex-ible, however, since they tightly couple low-level event detection, demulti-plexing, and dispatching code with application processing code. Develop-ers must therefore rewrite all this code for each new networked application.This approach is tedious, expensive, error-prone, and unnecessary, how-ever, since event detection, demultiplexing, and dispatching can be per-formed in an application-independent manner. To decouple this reusablecode from the application-defined event processing code, the ACE Reactorframework defines the ACE_Reactor class.

Class Capabilities

The ACE_Reactor class implements the Facade pattern [GHJV95] to definean interface that applications can use to access the various capabilities of


i

i

i

i

i

i

i

i

62 Section 3.5 The ACE Reactor Class

the ACE Reactor framework. This class provides the following capabilities:

� It centralizes event loop processing in a reactive application� It detects events using a synchronous event demultiplexer, such as

select() or WaitForMultipleObjects() , provided by the underly-ing OS

� It demultiplexes events to event handlers when the synchronous eventdemultiplexer indicates the occurrence of designated events

� It dispatches the appropriate methods on registered event handlers toperform application-defined processing in response to the events and

� It enables other threads in a program to notify a reactor.

By encapsulating low-level OS event demultiplexing mechanisms withinan object-oriented C++ interface, the ACE Reactor framework simplifiesthe development of correct, compact, portable, and efficient event-drivennetworked applications. Likewise, by separating event detection, demulti-plexing, and dispatching mechanisms from application-defined event pro-cessing policies, the ACE Reactor framework enhances reuse, improvesportability, and enables the transparent extensibility of event handlers.

Figure 3.5: The ACEReactor Class

The interface for the ACE_Reactor class is shown in Figure 3.5. TheACE_Reactor has a rich interface that exports all the features in the ACEReactor framework. We therefore group the description of its methods intothe six categories described below.

1. Reactor lifecycle management methods. The following methods ini-tialize and terminate an ACE_Reactor :

Method DescriptionACE_Reactoropen()

These methods create and initialize instances of a reactor.

˜ACE_Reactorclose()

These methods clean up the resources allocated when a reac-tor was initialized.

2. Event handler management methods. The following methods regis-ter and remove concrete event handlers from an ACE_Reactor :


i

i

i

i

i

i

i

i


Method Descriptionregister_handler() Methods that register concrete event handlers with a

reactor.remove_handler() Methods that remove concrete event handlers from a

reactor.mask_ops() Perform operations that get, set, add, and clear the

event type(s) associated with an event handler and itsdispatch mask.

schedule_wakeup() Add the designed masks to an event handler’s en-try, which must already have been registered viaregister_handler() .

cancel_wakeup() Clears the designated masks from an event handler’sentry, but doesn’t not remove the handler from the re-actor.

The register_handler() methods can be used with either of the fol-lowing signatures:

� Two parameters—In this version, the first parameter identifies theevent handler and the second indicates the type of event(s) the eventhandler has registered to process. The method’s implementation uses“double-dispatching” [GHJV95] to obtain a handle by calling back tothe event handler’s get_handle() method. The advantage of thisdesign is that the wrong handle can’t be associated with an eventhandler accidentally. Most examples in this book therefore use thetwo parameter variant of register_handler() .

� Three parameters—In this version, another first parameter is addedto pass the handle explicitly. Although this design is potentiallymore error-prone than the two-parameter version, it allows an ap-plication to register the same event handler for multiple handles,which helps conserve memory if an event handler needn’t maintainper-handle state. The client logging daemon example in the Exam-ple portion of Section 5.2 illustrates the three parameter variant ofregister_handler() .

Both ACE_Reactor::remove_handler() methods remove an event han-dler from a reactor so that it’s no longer registered for one or more typesof events. The signatures of these methods can be passed either an eventhandler and/or a handle, just like the two register_handler() methodvariants described above. When removing a handler, applications also passa bit-mask consisting of the enumeration literals defined in the table on


i

i

i

i

i

i

i

i


page 47 to indicate which event types are no longer desired. The eventhandler’s handle_close() method is called soon afterwards to notify it ofthe removal.2 After handle_close() returns and the concrete event han-dler is no longer registered to handle any events, the ACE_Reactor removesthe event handler from its internal data structures.

The mask_ops() method performs operations that get, set, add, andclear the event type(s) associated with an event handler and its dispatchmask. Since mask_ops() assumes that an event handler is already presentand doesn’t try to remove it, it’s more efficient than using register_handler() and remove_handler() . The schedule_wakeup() and cancel_wakeup() methods are simply “syntactic sugar” for common operations in-volving mask_ops() .

The mask_ops() , schedule_wakeup() , and cancel_wakeup() meth-ods don’t cause the reactor to re-examine its set of handlers, i.e., the newmasks will only be noticed the next time the reactor’s handle_events()method is called. If there’s no other activity expected, or you need imme-diate re-examination of the wait masks, you’ll need to call ACE_Reactor::notify() after calling one of these methods or use the ACE_Reactor ’sregister_handler() or remove_handler() methods instead.

3. Event-loop management methods. After registering its initial con-crete event handlers, an application can manage its event loop via thefollowing methods:

2This callback can be prevented by adding the ACE_Event_Handler::DONT_CALL valueto the mask passed to remove_handler() which instructs a reactor not to dispatch thehandle_close() method when removing an event handler.


i

i

i

i

i

i

i

i


Method Descriptionhandle_events() Wait for events to occur and then dispatch the

event handlers associated with these events.A timeout parameter can bound the timespent waiting for events, so that it won’t blockindefinitely if events never arrive.

run_reactor_event_loop() Run the event loop continuously until thehandle_events() method returns �1 or theend_reactor_event_loop() method is in-voked.

end_reactor_event_loop() Instruct a reactor to terminate its event loopso that it can shut down gracefully.

reactor_event_loop_done() Returns 1 when the reactor’s event loop hasbeen ended, e.g., via a call to end_reactor_event_loop() .

The handle_events() method gathers the handles of all registeredconcrete event handlers, passes them to the reactor’s synchronous eventdemultiplexer, and then blocks for an application-specified time intervalawaiting the occurrence of various events, such as data events, to arriveon socket handles or for timer deadlines to expire. When I/O events occurand the handles become active, this method dispatches the appropriatepre-registered concrete event handlers by invoking their handle_*() hookmethod(s) defined by the application to process the event(s). Sidebar 4 out-lines the order in which different types of event handlers are dispatched byhandle_events() .

4. Timer management methods. By default, the ACE_Reactor usesthe ACE_Timer_Heap timer queue mechanism described in Section 3.4to schedule and dispatch event handlers in accordance to their timeoutdeadlines. The timer management methods exposed by the ACE_Reactorinclude:

Method Descriptionschedule_timer() Register a concrete event handler that will be executed

at a user-specified time in the future.cancel_timer() Cancel one or more event handlers that were registered

previously.

The ACE_Reactor provides additional capabilities to those defined inthe ACE timer queue classes. For example, it integrates the dispatching


i

i

i

i

i

i

i

i


Sidebar 4: The Dispatching Order in the ACE Reactor Framework

The ACE Reactor framework can dispatch five different types of events,which are processed in the following order:

1. Time-driven events2. Notifications3. Output I/O events4. Exception I/O events5. Input I/O events

It’s generally a good idea to design applications whose behavior is inde-pendent of the order in which the different types of events are dispatched.There are situations, however, where knowledge of the dispatching orderof events is necessary.

of timers with the dispatching of other types of events, whereas the ACEtimer queue classes just dispatch time-driven events. Moreover, unlikethe schedule() methods described in Section 3.4, the ACE_Reactor::schedule_timer() uses relative time, not absolute time. As a result, weuse slightly different names for the ACE_Reactor timer management meth-ods.

5. Notification methods. The following methods manage the notificationqueue that application threads can use to communicate with and controla reactor:

Method Descriptionnotify() Pass an event handler to a reactor and

designate which handle_*() methodwill be dispatched in the context of thereactor. By default, the event han-dler’s handle_except() method is dis-patched.

max_notify_iterations() Set the maximum number of event han-dlers that a reactor will dispatch fromits notification queue.

purge_pending_notifications() Purge a specified event handler or allevent handlers that are in the reactor’snotification queue.


i

i

i

i

i

i

i

i


The ACE_Reactor::notify() method can be used for several pur-poses:

� It can wake up a reactor whose synchronous event demultiplexerfunction is waiting for I/O events to occur.

� It can pass event handlers to a reactor without associating the eventhandlers with I/O or timer events.

By default, the reactor’s notification mechanism will dispatch all eventhandlers in its notification queue. The max_notify_iterations() methodcan be used to change the number of event handlers dispatched. Settinga low value will improve fairness and prevent starvation, at the expense ofincreasing dispatching overhead.

6. Utility methods. The following methods are also provided by the ACE_Reactor :

Method Descriptioninstance() A static method that returns a pointer to a singleton ACE_

Reactor , which is created and managed by the Singleton pat-tern [GHJV95] combined with the Double-Checked Locking Opti-mization [SSRB00].

owner() Assigns a thread to “own” a reactor’s event loop.

The ACE_Reactor can be used in two ways:

� As a singleton [GHJV95] via the instance() method shown in thetable above.

� By instantiating one or more instances. This capability can be usedto support multiple reactors within a process. Each reactor is oftenassociated with a thread running at a particular priority [Sch98].

Some reactor implementations, such as the ACE_Select_Reactor de-scribed in Section 3.6, only allow one thread to run their handle_events()method. The owner() method changes the identity of the thread that ownsthe reactor to allow this thread to run the reactor’s event loop.

The ACE Reactor framework uses the Bridge pattern [GHJV95] to de-couple the interface of a class from its various implementations. TheACE_Reactor defines the interface and all the actual event detection, de-multiplexing, and dispatching is performed by implementations of this in-terface. The Bridge pattern allows applications to choose different reactorswithout changing their source code, as shown in Figure 3.6.


i

i

i

i

i

i

i

i


Figure 3.6: The ACE Reactor Class Hierarchy

The most common implementations of ACE_Reactor —the ACE_Select_Reactor , ACE_TP_Reactor , and ACE_WFMO_Reactorclasses—are describedin Sections 3.6 through 3.8, respectively. The ACE_Select_Reactor is thedefault implementation of the ACE_Reactor on all platforms except Win32,which uses the ACE_WFMO_Reactorby default. For situations where thedefault reactor implementation is inappropriate, developers can choose adifferent reactor at compile-time or at run-time. The method names andoverall functionality provided by the ACE_Reactor interface remain thesame, however. This uniformity stems from the modularity of the ACEReactor framework’s design, which enhances its reuse, portability, andmaintainability.

The design of the ACE Reactor framework also enhances extensibil-ity above and below its public interface. For example, it’s straightfor-ward to extend the logging server’s functionality, e.g., to add an “authen-ticated logging” feature. Such extensions simply inherit from the ACE_Event_Handler base class and selectively implement the necessary vir-tual method(s). It’s also straightforward to modify the underlying eventdemultiplexing mechanism of an ACE_Reactor without affecting existingapplication code. For example, porting the Reactor-based logging serverfrom a UNIX platform to a Win32 platform requires no visible changes toapplication code. In contrast, porting a C implementation of the networkedlogging service from select() to WaitForMultipleObjects() would betedious and error-prone.

Additional coverage of the ACE Reactor framework’s design appears inthe Reactor pattern’s Implementation section in Chapter 3 of [SSRB00].Sidebar 5 describes how you can extend the ACE Reactor framework tosupport other event demultiplexers.

Sidebar 5: Extending the ACE Reactor Framework


i

i

i

i

i

i

i

i


Example

Now that we’ve described the classes in the ACE Reactor framework, weshow how they can be integrated to implement a complete version of thenetworked logging server. This server listens on a TCP port number definedin the OS network services file as ace_logger , which is a practice used bymany networked servers. For example, the following line might appear inthe UNIX /etc/services file:

ace_logger 9700/tcp # Connection-oriented Logging Service

Client applications can optionally specify the TCP port and the IP ad-dress where the client application and logging server should rendezvous toexchange log records. If this information is not specified, however, the portnumber is located in the services database, and the hostname is assumedto be the ACE DEFAULT SERVER HOST, which is defined as “localhost ” onmost OS platforms.

The version of the logging server shown below offers the same capabili-ties as the Reactive_Logging_Server_Ex version in Chapter 7 of [SH02].Both logging servers run in a single thread of control in a single process,handling log records from multiple clients reactively. The main difference isthat this version uses the Reactor pattern to factor out the event detection,demultiplexing, and dispatching code from the logging server applicationinto the ACE Reactor framework, which dispatches incoming log recordsreceived from clients in a round-robin fashion. This refactoring helps ad-dress limitations with the original Reactive_Logging_Server_Ex imple-mentation, which contained the classes and functionality outlined belowthat had to be re-written for each new networked application:

Managing various mappings: The Reactive_Logging_Server_Ex servercontained two data structures that performed the following mappings:

� An ACE_Handle_Set mapped socket handles to Logging_Handlerobjects, which encapsulate the I/O and processing of log records inaccordance with the logging service’s message framing protocol.

� An ACE_Hash_Map_Manager mapped socket handles to their associ-ated ACE_FILE_IO objects, which write log records to the appropriateoutput file.

We’ve removed all the mapping code from this chapter’s logging server ap-plication and refactored it to reuse the capabilities available in the ACEReactor framework. Since the framework now provides and maintains thiscode, you needn’t write it for this or any other networked application.


i

i

i

i

i

i

i

i


Event detection, demultiplexing, and dispatching: To detect connec-tion and data events, the Reactive_Logging_Server_Ex server used theACE::select() synchronous event demultiplexer method. This design hasthe following drawbacks:

� It works only as long as the OS platform provides select()

� The code for setting up the fd_set structures for the call and inter-preting them upon return is error-prone and

� The code that called ACE::select() is hard to reuse for other appli-cations.

The version of the logging server shown below uses the ACE Reac-tor framework to detect, demultiplex, and dispatch I/O- and time-basedevents. This framework also supports the integration of signal handling inthe future as the need arises. In general, applications use the followingsteps to integrate themselves into the ACE Reactor framework:

1. Create concrete event handlers by inheriting from the ACE_Event_Handler base class and overriding its virtual methods to handle var-ious types of events, such as I/O events, timer events, and signals

2. Register concrete event handlers with an instance of ACE_Reactorand

3. Run an event loop that demultiplexes and dispatches events to theconcrete event handlers at run-time.

Figure 3.7 illustrates the Reactor-based logging server architecture thatbuilds upon and enhances our earlier logging server implementations from[SH02]. To enhance reuse and extensibility, the classes in this figure are

Figure 3.7: Architecture of Reactor-based Logging Server

designed to decouple the following aspects of the logging server’s architec-ture, which are described from the bottom to the top of Figure 3.7:

Reactor framework classes. The classes in the Reactor framework en-capsulate the lower-level mechanisms that perform event detection andthe demultiplexing and dispatching of events to concrete event handlerhook methods.

Connection-oriented ACE Socket wrapper facade classes. The ACE_SOCK_Acceptor and ACE_SOCK_Stream classes presented in Chapter 3

‘‘hs _2001 /page

i

i

i

i

i

i

i

i


of [SH02] are used in this version of the logging server. As in previous ver-sions, the ACE_SOCK_Acceptor accepts network connections from remoteclients and initializes ACE_SOCK_Streamobjects. An initialized ACE_SOCK_Stream object then processes data exchanged with its connected client.

Concrete logging event handler classes. These classes implement thecapabilities specific to the networked logging service. As shown in the Ex-ample portion of Section 3.4, The Logging_Acceptor_Ex factory uses anACE_SOCK_Acceptor to accept client connections. Likewise, the Logging_Handler_Adapter_Ex uses an ACE_SOCK_Stream to receive log recordsfrom connected clients. Both classes are ancestors of ACE_Event_Handler ,so their handle_input() methods receive callbacks from an ACE_Reactor .

Our implementation resides in a header file called Reactor_Logging_Server.h , which includes several header files that provide the various ca-pabilities we’ll use in our logging server.

#include "ace/ACE.h"#include "ace/Reactor.h"#include "Logging_Acceptor_Ex.h"

We next define the Reactor_Logging_Server class.

template <class ACCEPTOR>class Reactor_Logging_Server : public ACCEPTOR{protected:

// Pointer to the reactor.ACE_Reactor *reactor_;

This class inherits from its ACCEPTORtemplate parameter. To vary certainaspects of Reactor_Logging_Server ’s connection establishment and log-ging behavior, subsequent examples will instantiate it with various types ofacceptors, such as the Logging_Acceptor_Ex on page 58. The Reactor_Logging_Server also contains a pointer to the ACE_Reactor that it usesto detect, demultiplex, and dispatch I/O- and time-based events to theirevent handlers.

The signatures of the public methods in Reactor_Logging_Server classare shown below.

public:// Constructor uses the singleton reactor by default.Reactor_Logging_Server


i

i

i

i

i

i

i

i


(int argc,char *argv[],ACE_Reactor *r = ACE_Reactor::instance ());

// Destructor.virtual ˜Reactor_Logging_Server ();

};

This interface differs from the Logging_Server class defined in Chap-ter 4 of [SH02]. In particular, the Reactor_Logging_Server uses theACE_Reactor::handle_events() method to drive application processingvia upcalls to instances of Logging_Acceptor and Logging_Handler_Adapter . We therefore don’t need the wait_for_multiple_events() ,handle_connections() , and handle_data() hook methods that were usedin the reactive logging servers from [SH02].

The Reactor_Logging_Server ’s constructor performs the steps neces-sary to initialize the Reactor-based logging server:

1 template <class ACCEPTOR>2 Reactor_Logging_Server<ACCEPTOR>::Reactor_Logging_Server3 (int argc, char *argv[], ACE_Reactor *reactor)4 : reactor_ (reactor) {5 u_short logger_port = argc > 1 ? atoi (argv[1]) : 0;6 ACE::set_handle_limit ();78 typename ACCEPTOR::PEER_ADDR server_addr;9 int result;

1011 if (logger_port != 0)12 result = server_addr.set (logger_port, INADDR_ANY);13 else14 result = server_addr.set ("ace_logger", INADDR_ANY);15 if (result == -1) return -1;1617 open (server_addr); // Calls ACCEPTOR::open();1819 reactor_->register_handler20 (this, ACE_Event_Handler::ACCEPT_MASK);21 }

Line 5 Set the port number that’ll be used to listen for client connections.

Line 6 Raise the number of available socket handles to the maximumsupported by the OS platform.

Lines 8–17 Set the local server address and use this to initialize thepassive-mode logging acceptor endpoint.


i

i

i

i

i

i

i

i

Section 3.6 The ACE Select Reactor Class 73

Lines 19–20 Register this object with the reactor to ACCEPT events.

The destructor of the Reactor_Logging_Server class removes thisobject from the reactor, which triggers its inherited ACCEPTOR::handle_close() hook method to close the underlying ACE_SOCK_Acceptor .

template <class ACCEPTOR>Reactor_Logging_Server<ACCEPTOR>::˜Reactor_Logging_Server () {

reactor_->remove_handler (this,ACE_Event_Handler::ACCEPT_MASK);

}

We conclude with the main() program, which instantiates the Reactor_Logging_Server template with the Logging_Acceptor_Ex class definedon page 58. It then defines an instance of this template class and usesthe singleton reactor to drive all subsequent connection and data eventprocessing.

typedef Reactor_Logging_Server<Logging_Acceptor_Ex>Server_Logging_Daemon;


ACE_Reactor *reactor = ACE_Reactor::instance ();Server_Logging_Daemon server (argc, argv, reactor);

if (reactor->run_reactor_event_loop () == -1)ACE_ERROR_RETURN ((LM_ERROR, "%p\n",

"run_reactor_event_loop()"), 1);return 0;

}

Since all event detection, demultiplexing, and dispatching is handled bythe Reactor framework, the reactive logging server implementation shownabove is much shorter than the equivalent ones in [SH02].

3.6 The ACE Select Reactor Class

Motivation

At the heart of a reactive server is a synchronous event demultiplexer thatdetects and reacts to events from clients continuously. The select() func-tion is the most common synchronous event demultiplexer. This system


i

i

i

i

i

i

i

i

74 Section 3.6 The ACE Select Reactor Class

function waits for specified events to occur on a set of I/O handles.3 Whenone or more of the I/O handles become active, or after a designated periodof time elapses, select() returns to its caller. The caller can then processthe events indicated by information returned from select() . Additionalcoverage of select() is available in Chapter 6 of [SH02] and in [Ste98].

Although select() is available on most OS platforms, programming tothe native OS select() C API requires developers to wrestle with manylow-level details, such as:

� Setting and clearing bitmasks� Detecting events and responding to signal interrupts� Managing internal locks and� Dispatching functions to process I/O, signal, and timer events.

To shield programmers from the accidental complexities of these low-leveldetails, the ACE Reactor framework defines the ACE_Select_Reactor class.

Class Capabilities

The ACE_Select_Reactor class is an implementation of the ACE_Reactorinterface that uses the select() synchronous event demultiplexer func-tion to detect I/O and timer events. It’s the default implementation ofACE_Reactor on all platforms except for Win32. In addition to supportingall the features of the ACE_Reactor interface, the ACE_Select_Reactorclass provides the following capabilities:

� It supports re-entrant reactor invocations, i.e., applications can callhandle_events() from within event handlers that are themselves be-ing dispatched by the reactor and

� It can be configured to be either synchronized or non-synchronized.� It preserves fairness by dispatching all active handles in the handle

sets before calling select() again.

The ACE_Select_Reactor class is a descendant of ACE_Reactor_Impl ,as shown in Figure 3.6. It therefore can serve as a concrete implementationof the ACE_Reactor , which plays the role of the interface participant inthe Bridge pattern. Internally, the ACE_Select_Reactor is structured asshown in Figure 3.8 and described below:

3The Win32 version of select() only works on socket handles.


i

i

i

i

i

i

i

i


Figure 3.8: The ACESelect Reactor Framework Internals

� It contains three instances of ACE_Handle_Set , one each for read,write, and exception events, which are all passed to select() whenan application calls the ACE_Reactor::handle_events() method.As described in Chapter 7 of [SH02], ACE_Handle_Set provides anefficient C++ wrapper facade for the underlying fd_set bitmask datatype.

� It also contains three arrays of ACE_Event_Handler pointers, whichstore pointers to registered concrete event handlers that process var-ious types of events specified by applications.

The select() -based implementation of the ACE Reactor framework isbased upon the ACE_Select_Reactor_T template that uses the Strate-gized Locking pattern [SSRB00] to configure the following two types of re-actors:

� Non-synchronized—This version is designed to minimize the over-head of event demultiplexing and dispatching for single-threaded ap-plications. It can be configured by parameterizing the ACE_Select_Reactor_T template with an ACE_Null_Mutex .

� Synchronized—This version allows multiple threads to invoke meth-ods on a single ACE_Reactor that’s shared by all the threads. Italso allows multiple ACE_Reactor s to run in separate threads withina process. The ACE_Select_Reactor is an instantiation of ACE_Select_Reactor_T that uses a recursive locking mechanism calledan ACE_Token to prevent race conditions and intra-class method dead-lock. Sidebar 6 outlines the ACE_Token class.

Only one thread can invoke ACE_Select_Reactor::handle_events()at a time. To enforce this constraint, each ACE_Select_Reactor storesthe identity of the thread that owns its event loop. By default, the owner ofan ACE_Reactor is the identity of the thread that initialized it. The ACE_Select_Reactor::owner() method sets ownership of the ACE_Select_Reactor to the calling thread. This method is useful when the threadrunning the reactor’s event loop differs from the thread that initialized thereactor.


i

i

i

i

i

i

i

i


Sidebar 6: The ACE Token Class

ACE_Token is a lock whose interface is compatible with other ACE synchro-nization wrapper facades, such as ACE_Thread_Mutex or ACE_RW_Mutexfrom [SH02], but whose implementation differs as follows:

� It implements recursive mutex semantics, where a thread that ownsthe token can reacquire it without deadlocking. Before a token canbe acquired by a different thread, however, its release() methodmust be called the same number of times that acquire() was called.

� Each ACE_Token maintains two FIFO lists that are used to queue uphigh- and low-priority threads waiting to acquire the token. Threadsrequesting the token using ACE_Token::acquire_write() are keptin the high-priority list and take precedent over threads that callACE_Token::acquire_read() , which are kept in the low-priority list.Within a priority list, threads that are blocked awaiting to acquire atoken are serviced in FIFO order as other threads release the token,which ensures the fairness among waiting threads. In contrast, UNIXInternational and Pthread mutexes don’t strictly enforce thread ac-quisition order.

� The ACE_Token::sleep_hook() hook method allows a thread to re-lease any resources it’s holding, before it waits to acquire the token.This capability allows a thread to avoid deadlock, starvation, and pri-ority inversion. The sleep_hook() is only invoked if a thread can’tacquire a token immediately.

ACE_Select_Reactor and its derived Reactors use a subclass of ACE_Token to synchronize multithreaded access to a reactor. The FIFO serv-ing order of ACE_Token ensures threads waiting on a reactor are servedfairly. Requests to change the internal states of a reactor use ACE_Token::acquire_write() to ensure other waiting threads see the changes assoon as possible. The ACE_Token subclass also overwrites the defaultsleep_hook() method to notify the reactor of pending threads via its no-tification mechanism.

The ACE_Select_Reactor::owner() method is useful when the threadthat initializes an ACE_Select_Reactor differs from the thread that ulti-mately runs the reactor’s event loop.


i

i

i

i

i

i

i

i


Example

The reactive logging server on page 73 is designed to run continuously.There’s no way to shut it down gracefully, however, other than to terminateit abruptly, e.g., by sending its process a “kill -9” from a UNIX login consoleor ending the process via the Win2K process manager. In this example,we show how to use the ACE_Select_Reactor::notify() mechanism toshut down the logging server gracefully and portably.

The ACE_Select_Reactor implements its notification mechanism viaan ACE_Pipe , which is a bi-directional IPC mechanism described in Side-bar 7. The two ends of a pipe play the following roles:

� The writer role—The ACE_Select_Reactor::notify() method ex-poses this end of the pipe to applications, which use the notify()method to pass event handler pointers to an ACE_Select_Reactorvia its notification pipe.

� The reader role—The ACE_Select_Reactor registers this end of thepipe internally with a READ MASK. When the reactor detects an eventin the read-side of its notification pipe it wakes up and dispatchesa user-configurable number of event handlers from the pipe. Unlikeother concrete event handlers registered with a reactor, these han-dlers needn’t be associated with I/O-based or timer-based events,which helps improve the ACE_Select_Reactor ’s dispatching scala-bility. In particular, there’s no requirement that a handler whosepointer you give to ACE_Reactor::notify() has ever been, or everwill be, registered with that reactor.

Figure 3.9 illustrates the structure of the reader and writer roles withinan ACE_Select_Reactor .

We can use the ACE_Select_Reactor ’s ACE_Pipe -based notificationmechanism to shut down our Reactor_Logging_Server gracefully via thefollowing steps:

1. We’ll spawn a controller thread that waits for an administrator to passit commands via its standard input.

2. When the “quit ” command is received, the controller thread passes aspecial concrete event handler to the singleton reactor via its notify()method and then exits the thread.

3. The reactor dispatches this event handler by invoking its handle_except() method, which calls end_reactor_event_loop() and thendeletes itself.

‘‘hs _2001 /page

i

i

i

i

i

i

i

i


Sidebar 7: The ACE Pipe Class

The ACE_Pipe class provides a portable, bi-directional IPC mechanism thattransfers data within an OS kernel. This class is implemented as follows onvarious OS platforms:

� Via a STREAM pipe on modern UNIX platforms� Via a socketpair() on legacy UNIX platforms or� Via a connected socket on Win32 platforms.

After initializing an ACE_Pipe , applications can obtain its “read” and“write” handles via access methods and invoke I/O operations to receiveand send data. These handles can also be included in ACE_Handle_Set spassed to ACE::select() or to an ACE_Select_Reactor .

Figure 3.9: The ACESelect Reactor Notification Mechanism

4. The next time the ACE_Reactor::run_reactor_event_loop() methodcalls handle_events() internally it will detect that reactor_event_loop_done() is now true, which will cause the main server thread toexit gracefully.

The C++ code below illustrate these four steps. The revised main() func-tion is shown first:

1 #include "ace/Auto_Ptr.h"2 // Forward declarations.3 void *controller (void *);4 void *event_loop (void *);56 typedef Reactor_Logging_Server<Logging_Acceptor_Ex>7 Server_Logging_Daemon;89 int main (int argc, char *argv[])

10 {11 auto_ptr <ACE_Select_Reactor> holder (new ACE_Select_Reactor);12 auto_ptr <ACE_Reactor> reactor (new ACE_Reactor (holder.get ()));13 ACE_Reactor::instance (reactor.get ());1415 Server_Logging_Daemon server (argc, argv, reactor.get ());1617 ACE_Thread_Manager::instance ()->spawn18 (event_loop, ACE_static_cast (void *, reactor.get ()));

‘‘hs _2001 /page

i

i

i

i

i

i

i

i


1920 ACE_Thread_Manager::instance ()->spawn21 (controller, ACE_static_cast (void *, reactor.get ()));2223 return ACE_Thread_Manager::instance ()->wait ();24 }

Lines 1–7 We include the relevant ACE header file, define some forwarddeclarations, and instantiate the Reactor_Logging_Server template withthe Logging_Acceptor_Ex class from page 58 to create a Server_Logging_Daemon type definition.

Lines 11–13 We set the implementation of the singleton ACE_Reactor tobe an ACE_Select_Reactor .

Lines 15–18 We then create an instance of Server_Logging_Daemon anduse the ACE_Thread_Manager singleton described in Chapter 9 of [SH02]to spawn a thread that runs the following event_loop() function:

void *event_loop (void *arg){

ACE_Reactor *reactor = ACE_static_cast (ACE_Reactor *, arg);

reactor->owner (ACE_OS::thr_self ());

return reactor->run_reactor_event_loop ();}

Note how we set the owner of the reactor to the id of the thread that runsthe event loop. Section ?? explains the design rule governing the use ofthread ownership for the ACE_Select_Reactor .

Lines 20–21 We spawn a thread to run the controller() function, whichis the next function shown below.

Line 23 We wait for the other threads to exit before returning from themain() function.

The controller function is implemented as follows:

1 void *controller (void *arg)2 {3 ACE_Reactor *reactor = ACE_static_cast (ACE_Reactor *, arg);45 Quit_Handler *quit_handler = new Quit_Handler (reactor);67 for (;;) {8 std::string user_input;


i

i

i

i

i

i

i

i


9 getline (cin, user_input, ’\n’);10 if (user_input == "quit")11 return reactor->notify (quit_handler);12 }13 return 0;14 }

Lines 3–6 After casting the void pointer argument back into an ACE_Reactor pointer, we create a special concrete event handler called Quit_Handler , which is shown shortly below.

Lines 7–12 We then go into a loop that waits for an administrator totype “quit ” on the standard input stream. When this occurs, we passthe quit_handler to the reactor via its notify() method and exit thecontroller thread.

We finally define the Quit_Handler class. Its handle_except() andhandle_close() methods simply shut down the ACE_Select_Reactor ’sevent loop and delete the event handler, respectively, as shown below:

class Quit_Handler : public ACE_Event_Handler {public:

Quit_Handler (ACE_Reactor *r): reactor_ (r) {}

virtual int handle_except (ACE_HANDLE) {reactor_->end_reactor_event_loop ();return -1; // Trigger call to handle_close() method.

}

virtual int handle_close (ACE_HANDLE, ACE_Reactor_Mask) {delete this;

}

private:ACE_Reactor *reactor_;

// Private destructor ensures dynamic allocation.˜Quit_Handler () {}

};

The implementation shown above is portable to all ACE platforms thatsupport threads. Section 3.8 illustrates how to take advantage of Win32-specific features to accomplish the same behavior without needing an ad-ditional controller thread.


i

i

i

i

i

i

i

i

Section 3.7 The ACE TP Reactor Class 81

3.7 The ACE TP Reactor Class

Motivation

Although the ACE_Select_Reactor is quite flexible, it’s limited since onlyits owner thread can call its handle_events() event loop method. TheACE_Select_Reactor therefore serializes event processing at the demul-tiplexing layer, which may be overly restrictive and non-scalable for cer-tain networked applications. One way to solve this problem is to spawnmultiple threads and run the event loop of a separate instance of ACE_Select_Reactor in each of them. This design can be hard to program,however, since it requires programmers to keep track of tedious bookkeep-ing information, such as which thread and which reactor an event han-dler is registered. A more effective way to address the limitations withACE_Select_Reactor is to use the ACE_TP_Reactor class provided by theACE Reactor framework.

Class Capabilities

The ACE_TP_Reactor class is another implementation of the ACE_Reactorinterface. This class implements the Leader/Followers architectural pat-tern [SSRB00], which provides an efficient concurrency model where mul-tiple threads take turns calling select() on sets of I/O handles to de-tect, demultiplex, dispatch, and process service requests that occur. Inaddition to supporting all the features of the ACE_Reactor interface, theACE_TP_Reactor provides the following capabilities:

� It enables a pool of threads to call its handle_events() method,which can improve scalability by handling events on multiple han-dles concurrently.

� Compared to other thread pool models, such as the Half-Sync/Half-Async model in Chapter 5 of [SH02], the leader/followers implementa-tion in ACE_TP_Reactor provides the following performance enhance-ments:

– It enhances CPU cache affinity and eliminates the need to allo-cate memory dynamically and share data buffers between threads

– It minimizes locking overhead by not exchanging data betweenthreads


i

i

i

i

i

i

i

i

82 Section 3.7 The ACE TP Reactor Class

– It can minimize priority inversion because no extra queueing isintroduced in the server and

– It doesn’t require a context switch to handle each event, whichreduces event dispatching latency.

The ACE_TP_Reactor class is a descendant of ACE_Reactor_Impl , asshown in Figure 3.6. It can therefore serve as a concrete implementationof the ACE_Reactor interface. The ACE_TP_Reactor is structured inter-nally much like the ACE_Select_Reactor . For example, ACE_TP_Reactoruses the same instances of ACE_Handle_Set and the arrays of ACE_Event_Handler pointers described on page 74.

The fundamental difference between the ACE_TP_Reactor and ACE_Select_Reactor is the way they acquire and release the ACE_Token de-scribed in Sidebar 6 on page 76. In the ACE_TP_Reactor , the thread thatacquires the token is the leader and threads waiting to acquire the tokenare the followers. The leader thread calls select() to wait for events tooccur on a set of I/O handles. After select() returns, the leader threaddoes the following:

� It picks one event and dispatches its associated event handler hookmethod.

� It suspends the I/O handle so that other threads can’t detect eventson that handle. Suspension involves removing the handle from thehandle set the reactor uses to wait on during select() .

� It releases the ACE_Token.

� When the original leader thread returns from its dispatching upcall,it resumes the suspended handle, which adds the handle back to theappropriate handle set.

After the leader thread releases the token to process an event, a followerthread becomes the new leader. The ACE_TP_Reactor can therefore allowmultiple threads to process events on different handles concurrently. Incontrast, the ACE_Select_Reactor holds the token while it dispatches toall handlers whose handles were active in the handle set, which serializesthe reactor’s dispatching mechanism.

Given the added capabilities of the ACE_TP_Reactor , you may wonderwhy anyone would ever use the ACE_Select_Reactor . There are severalreasons:

‘‘hs _2001 /page

i

i

i

i

i

i

i

i

Section 3.7 The ACE TP Reactor Class 83

� Less overhead—Although the ACE_Select_Reactor is less power-ful than the ACE_TP_Reactor it also incurs less overhead. More-over, single-threaded applications can instantiate the ACE_Select_Reactor_T template with an ACE_Null_Mutex to eliminate the over-head of acquiring and releasing tokens.

� Implicit serialization—The ACE_Select_Reactor is particularly use-ful when application-level serialization is undesirable. For example,applications that have pieces of data coming over different handlesmay prefer handling the data using the same event handler for sim-plicity.

Example

To illustrate the power of the ACE_TP_Reactor , we’ll revise the main()function from page 78 to use a pool of threads that use the Leader/Followerspattern to take turns sharing the Reactor_Logging_Server ’s I/O han-dles.

1 #include "ace/Auto_Ptr.h"2 // Forward declarations3 void *controller (void *);4 void *event_loop (void *);56 typedef Reactor_Logging_Server<Logging_Acceptor_Ex>7 Server_Logging_Daemon;89 const size_t N_THREADS = 4;

1011 int main (int argc, char *argv[])12 {13 size_t n_threads = argc > 1 ? atoi (argv[1]) : N_THREADS;1415 auto_ptr <ACE_TP_Reactor> holder (new ACE_TP_Reactor);16 auto_ptr <ACE_Reactor> reactor (new ACE_Reactor (holder.get ()));17 ACE_Reactor::instance (reactor.get ());1819 Server_Logging_Daemon server (argc, argv, reactor);2021 ACE_Thread_Manager::instance ()->spawn_n22 (n_threads,23 event_loop,24 ACE_static_cast (void *, reactor.get ()));2526 ACE_Thread_Manager::instance ()->spawn


i

i

i

i

i

i

i

i

84 Section 3.8 The ACE WFMO Reactor Class

27 (controller, ACE_static_cast (void *, reactor.get ()));2829 return ACE_Thread_Manager::instance ()->wait ();30 }

Lines 1–7 We include the relevant ACE header file, define some forwarddeclarations, and instantiate the Reactor_Logging_Server template withthe Logging_Acceptor_Ex class from page 58 to create a Server_Logging_Daemontype definition.

Line 13 We determine the number of threads to include in the threadpool.

Lines 15–17 We set the implementation of the singleton ACE_Reactor tobe an ACE_TP_Reactor .

Lines 19–24 We create an instance of the Server_Logging_Daemon tem-plate and then spawn n_threads , each of which runs the event_loop()function shown on page 79.

Lines 26–27 We spawn a thread to run the controller() function shownon page 79. Note that the ACE_TP_Reactor ignores the owner() methodthat’s called in this function.

Line 29 We wait for the other threads to exit before exiting the main()function.

3.8 The ACE WFMO Reactor Class

Motivation

Although the select() function is available on most operating systems,it’s not always the most efficient or most powerful event demultiplexer onan OS platform. In particular, select() has the following limitations:

� On UNIX platforms, it only supports demultiplexing of I/O handles,such as regular files, terminal and pseudo-terminal devices, STREAMS-based files, FIFOs and pipes. It does not support demultiplexing ofsynchronizers, threads, or System V Message Queues.

� On Win32, select() only supports demultiplexing of socket handles.� It can only be called by one thread at a time for a particular set of I/O

handles, which can degrade potential parallelism.


i

i

i

i

i

i

i

i

Section 3.8 The ACE WFMO Reactor Class 85

Sidebar 8: The Win32 WaitForMultipleObjects() Function

The Win32 WaitForMultipleObjects() system function is similar to theselect() system function. It blocks on an array of handles until one ormore of them become active (which is known as being “signaled” in Win32terminology) or until the interval in its timeout parameter elapses. It canbe programmed to return to its caller either when any one of the handlesbecomes active or when all the handles become active. In either case,it returns the index location of the lowest active handle in the array ofhandles passed to it as a parameter. Unlike the select() function thatonly demultiplexes I/O handles, WaitForMultipleObjects() can wait forany type of Win32 object, such as a thread, process, synchronizer (e.g.,event, semaphore, and mutex), I/O handle, named pipe, socket, directorychange notification, console input, or timer.

To alleviate problems with select() , Win32 defines WaitForMultiple-Objects() , which is described in Sidebar 8. Not only can this functionwork with all types of Win32 I/O handles, multiple threads can call itconcurrently on the same set of handles, thereby enhancing potential par-allelism. The WaitForMultipleObjects() function is tricky to use cor-rectly [SS95a], however, for the following reasons:

� Each WaitForMultipleObjects() call only returns a single activehandle. Multiple WaitForMultipleObjects() calls must thereforebe invoked to achieve the same behavior as select() , which returnsa set of active I/O handles.

� WaitForMultipleObjects() doesn’t guarantee a fair distribution ofnotifications, i.e., the lowest active handle in the array is always re-turned, regardless of how long other handles further back in the arraymay have had pending events.

To shield programmers from these low-level details, while preserving fair-ness and exposing the power of WaitForMultipleObjects() on Win32platforms, the ACE Reactor framework defines the ACE_WFMO_Reactorclass.

Class Capabilities

The ACE_WFMO_Reactorclass is yet another implementation of the ACE_Reactor interface. It uses the WaitForMultipleObjects() function to


i

i

i

i

i

i

i

i


wait for events to occur on a set of event sources. This class is the defaultimplementation of the ACE_Reactor on Win32 platforms. In addition tosupporting all the features of the ACE_Reactor interface, the ACE_WFMO_Reactor class provides the following capabilities:

� It enables a pool of threads to call its handle_events() method con-currently and

� It allows applications to wait for a wide range of events, includingI/O events, general Win32 synchronization events (such as mutexes,semaphores, and threads), and timer events.

� It preserves fairness by dispatching all active handles in a handlearray before starting at the beginning of the handle array.

ACE_WFMO_Reactorinherits from ACE_Reactor_Impl , as shown in Fig-ure 3.6. It can therefore serve as a concrete implementation of the ACE_Reactor interface. Just as the internals of the ACE_Select_Reactor aredesigned to leverage the capabilities of select() , ACE_WFMO_Reactor isdesigned to leverage the capabilities of WaitForMultipleObjects() . Itstwo most significant differences from the ACE_Select_Reactor are:

� Socket event detection—Although socket handles aren’t part of the“handle space” supported by WaitForMultipleObjects() , Win32 pro-vides the following functions to enable the use of socket handles withWaitForMultipleObjects() :

Win32 Function DescriptionWSACreateEvent() Creates a Win32 event object whose han-

dle can be passed to WaitForMultiple-Objects() .

WSAEventSelect() Associates a set of event types on a givensocket handle with an event handle. Theoccurrence of any of the events causes theevent handle to become signaled.

WSAEnumNetworkEvents() Retrieves the set of events that have oc-curred on a socket handle after its associ-ated event handle is signaled.

The ACE_WFMO_Reactor implementation associates an event handlewith a registered socket handle. To maintain the same programminginterface and semantics for the ACE_Event_Handler class, the eventsare demultiplexed internally and mapped to the appropriate callbackmethods, e.g., ACCEPT, CLOSE, and READ Win32 events map to the


i

i

i

i

i

i

i

i


ACE_Event_Handler::handle_input() callback hook method. Sincethe set of events that WaitForMultipleObjects() can wait for isbroader than that for select() , this mapping isolates the Win32-specific mechanisms from the portable interface exported to appli-cation designers via the ACE Reactor framework. As a result, theACE_WFMO_Reactor class requires neither the three arrays of ACE_Event_Handler pointers nor the ACE_Handle_Set class used by theACE_Select_Reactor . Instead, it allocates a single array of ACE_Event_Handler pointers and an array of handles that it uses to storethe concrete event handlers registered by applications.

� Multiple event loop threads—It’s legal for multiple threads to ex-ecute WaitForMultipleObjects() concurrently on the same set ofI/O handles. This feature complicates how ACE_WFMO_Reactorregis-ters and unregisters event handlers since multiple threads accessinga set of registered handlers may be affected by each change. To ex-ecute changes to the registered handler set correctly, the ACE_WFMO_Reactor therefore defers changes until the internal action stablizesand the changes can be made safely.

The manner in which the ACE_WFMO_Reactordefers changes makesone aspect of its behavior different from the ACE_Select_Reactor . Inparticular, when an event handler’s handle_close() hook method isinvoked (e.g., due to one of the handle_*() methods returning �1

or by calling ACE_Reactor::remove_handler() ), the call to handle_close() is deferred until the ACE_WFMO_Reactor’s internal recordsare updated. This update may not occur until some time after thepoint at which the application requests the event handler’s removal.This means that an application can’t delete an event handler imme-diately after requesting its removal from an ACE_WFMO_Reactorsincethe handle_close() method may not have been called on the eventhandler yet.

The two areas of difference described above show that two dissimilarevent demultiplexing mechanisms can be used effectively and portably byapplying patterns and abstraction principles to decouple the common in-terface from the divergent implementations.

‘‘hs _2001 /page

i

i

i

i

i

i

i

i


Sidebar 9: WRITE MASK Semantics on Win32 vs. UNIX

On UNIX platforms, if select() detects a WRITE event and the applicationdoesn’t handle the event, select() will renotify the application of theevent next time it’s called. In particular, UNIX platforms will generate WRITE

events as long as it’s possible to write data without flow controlling. Incontrast, Win32 platforms don’t generate multiple write events when theyare able to write. Instead, they only generate a single write event. UnderWin32 therefore you must continue to write until you get flow controlled orthe link drops.

Example

This example illustrates how to use the I/O handling capabilities of theACE_WFMO_Reactorto shut down the Reactor_Logging_Server withoutthe need for an additional controller thread. To accomplish this, we definea different Quit_Handler class than the one shown on page 80.

class Quit_Handler : public ACE_Event_Handler {private:

// Must be implemented by the <ACE_WFMO_Reactor>.ACE_Reactor *reactor_;

// Private destructor ensures dynamic allocation.˜Quit_Handler () {}

public:

Like the earlier Quit_Handler , this class inherits from ACE_Event_Handler .It’s used in an entirely different manner, however. For example, the con-structor of our new Quit_Handler registers itself with a reactor to receivea notification when an event occurs on the standard input.

Quit_Handler (ACE_Reactor *r): reactor_ (r) {reactor_->register_handler (this, ACE_STDIN);

}

When an event occurs on standard input, the ACE_WFMO_Reactor dis-patches the following handle_signal() method, which checks to see ifan administrator wants to shut down the reactor’s event loop.

‘‘hs _2001 /page

i

i

i

i

i

i

i

i


virtual int handle_signal (int, siginfo_t *, ucontext_t *) {std::string user_input;getline (cin, user_input, ’\n’);if (user_input == "quit") {

reactor_->end_reactor_event_loop ();return -1; // Trigger removal of handler from reactor.

}return 0;

}};

The main() function is similar to the one shown on page 83, with themain differences being

� The ACE_WFMO_Reactoris used instead of ACE_TP_Reactor� The controller thread is replaced by an instance of Quit_Handler

registered with the reactor� The calls to the ACE_WFMO_Reactor::handle_events() in different

threads can actually run concurrently, rather than being serializedusing the Leader/Followers pattern as is the case with the ACE_TP_Reactor and

� The program will only run on Win32 platforms, instead of all ACEplatforms.

The complete main() function is shown below.

#include "ace/Auto_Ptr.h"

void *event_loop (void *); // Forward declaration.

typedef Reactor_Logging_Server<Logging_Acceptor_Ex>Server_Logging_Daemon;

const size_t N_THREADS = 4;


size_t n_threads = argc > 1 ? atoi (argv[1]) : N_THREADS;

auto_ptr <ACE_WFMO_Reactor> holder (new ACE_WFMO_Reactor);auto_ptr <ACE_Reactor> reactor (new ACE_Reactor (holder.get ()));ACE_Reactor::instance (reactor.get ());

Server_Logging_Daemon server (argc, argv, reactor.get ());

Quit_Handler quit_handler (reactor.get ());


i

i

i

i

i

i

i

i


ACE_Thread_Manager::instance ()->spawn_n(n_threads,

event_loop,ACE_static_cast (void *, reactor.get ()));

return ACE_Thread_Manager::instance ()->wait ();}

3.9 Summary

This chapter shows how the ACE Reactor framework helps to simplify thedevelopment of event-driven networked applications by applying

� Patterns, such as Wrapper Facade, Facade, Bridge, and Iterator, with� C++ features, such as classes, inheritance, and dynamic binding.

The ACE Reactor framework provides reusable classes that perform all thelower-level event detection, demultiplexing, and event handler dispatch-ing. Only a small amount of application-defined code is therefore requiredto implement event-driven applications, such as the logging server shownin Sections 3.3 through 3.5. For example, the code in Sections 3.3 and 3.4is concerned with application-defined processing activities, such as receiv-ing log records from clients. All applications that reuse the ACE Reactorframework can leverage the knowledge and experience of its skilled mid-dleware developers, as well as its future enhancements and optimizations.

The ACE Reactor framework uses dynamic binding extensively sincethe dramatic improvements in clarity, extensibility, and modularity pro-vided by the ACE Reactor framework compensate for any decrease in effi-ciency resulting from indirect virtual table dispatching [HLS97]. The Re-actor framework is often used to develop networked applications wherethe major sources of overhead result from caching, latency, network/hostinterface hardware, presentation-level formatting, dynamic memory allo-cation and copying, synchronization, and concurrency management. Theadditional indirection caused by dynamic binding is therefore usually in-significant by comparison [Koe92]. In addition, good C++ compilers canoptimize virtual method overhead away completely via the use of “adjustorthunks” [Sta96].


i

i

i

i

i

i

i

i

CHAPTER 4

The ACE Service ConfiguratorFramework

CHAPTER SYNOPSIS

This chapter describes the design and use of the ACE Service Configura-tor framework. This framework implements the Component Configuratorpattern [SSRB00], which increases system extensibility by decoupling thebehavior of services from the point of time when implementations of theseservices are configured into application processes. We illustrate how theACE Service Configurator framework can help to improve the extensibilityof our networked logging server.

4.1 Overview

Section 2.2 described the naming and linking design dimensions that de-velopers need to consider when configuring networked applications. Anextensible strategy for addressing these design dimensions is to apply theComponent Configurator design pattern [SSRB00]. This pattern allows anapplication to link and unlink its services at run-time without having tomodify, recompile, or relink an application statically. This pattern alsosupports the reconfiguration of services in an application process withouthaving to shut down and restart a running process.

The ACE Service Configurator framework is a portable implementationof the Component Configuration pattern that allows applications to deferconfiguration and/or implementation decisions about their services until

91


i

i

i

i

i

i

i

i


late in the design cycle, i.e., at installation-time or even during run-time.In this chapter, we examine the following classes in the ACE Service Con-figurator framework:

ACE Class DescriptionACE_Service_Object Defines a uniform interface that the

ACE Service Configurator frameworkuses to configure and control the typeof application service or functional-ity provided by a service implementa-tion. Common control operations in-clude initializing, suspending, resum-ing, and terminating a service.

ACE_Service_Repository Manages all the services offered bya Service Configurator-based applica-tion and allows an administrator tocontrol the behavior of application ser-vices.

ACE_Service_Repository_Iterator Provides a portable mechanism for it-erating through all the services in arepository.

ACE_Service_Config Coordinates the (re)configuration ofservices by implementing a mech-anism that interprets and executesscripts specifying which services to(re)configure into the application, e.g.,by linking and unlinking dynamicallylinked libraries (DLLs), and which ser-vices to suspend and resume.

Figure 4.1: The ACE Service Configurator Framework Classes

The most important relationships between the classes in the ACE Ser-vice Configurator framework are shown in Figure 4.1. The ComponentConfigurator pattern described in [SSRB00] divides its participants intotwo layers:

� Configuration management layer classes that perform application-independent strategies to install, initialize, control, and terminateservice objects. The classes in the configuration management layerin the ACE Service Configurator framework include ACE_Service_


i

i

i

i

i

i

i

i

Section 4.1 Overview 93

Config , ACE_Service_Repository , and ACE_Service_Repository_Iterator .

� Application layer classes that implement concrete services to per-form application-defined processing. In the ACE Service Configura-tor framework, all application layer classes are descendants of ACE_Service_Object .

The ACE Service Configurator framework provides the following bene-fits:

� Uniformity—The framework imposes a uniform interface for manag-ing the (re)configuration of networked services. This uniformity al-lows services to be treated as building blocks that can be composedto form complete applications. Enforcing a common interface acrossall services also ensures that they support the same management op-erations, such as initializing, suspending, resuming, and terminatinga service.

� Centralized administration—The framework groups one or more ser-vices in an application into a single administrative unit. By default,the framework configures an application process by reading com-mands from a file called svc.conf . Alternative configuration filescan be specified by command-line options or by supplying commandsto ACE_Service_Config directly.

� Modularity, testability, and reusability—The framework improvesapplication modularity and reusability by decoupling the implementa-tion of services from the manner in which the services are configuredinto application processes. This flexibility allows service implemen-tations the freedom to evolve over time largely independent of config-uration issues, such as which services should be collocated or whatconcurrency model will be used to execute the services. In addition,each service can be developed and tested independently, which sim-plifies service composition.

� Enhanced dynamism and control—By decoupling the application-defined portions of an object from the underlying platform configu-ration mechanisms, the framework enables a service to be reconfig-ured dynamically without modifying, recompiling, or statically relink-ing existing code. It may also be possible to reconfigure a servicewithout restarting the service itself or other services with which it’s


i

i

i

i

i

i

i

i

94 Section 4.2 The ACE Service Object Class

collocated. Such dynamic reconfiguration capabilities are often re-quired for applications with high availability requirements, such asmission-critical systems that perform on-line transaction processingor real-time industrial process automation.

� Tuning and optimization—The framework increases the range ofservice configuration alternatives available to developers by decou-pling service functionality from service execution mechanisms, whichenables the optimization of certain service implementation or config-uration parameters. For instance, depending on the parallelism avail-able on the hardware and operating system, it may be either more orless efficient to run multiple services in separate threads or separateprocesses. The Service Configurator framework enables applicationsto select and tune these behaviors flexibly at run-time, when moreinformation is available to help match client demands and availablesystem processing resources.

The remainder of this chapter motivates and describes the capabilitiesof each class in the ACE Service Configurator framework. We also illus-trate how this framework can be used to enhance the extensibility of ournetworked logging server.

4.2 The ACE Service Object Class

Motivation

Enforcing a uniform interface across all networked services enables themto be configured and managed consistently. In turn, this consistency sim-plifies application development and deployment by simplifying the cre-ation of reusable administrative configuration tools and promoting the“principle of least surprise.” To provide a uniform interface between theACE Service Configurator framework and the application-supplied ser-vices, each service must be a descendant of a common base class calledACE_Service_Object .

Class Capabilities

The ACE_Service_Object class provides a uniform interface that allowsall service implementations to be configured and managed by the ACE Ser-


i

i

i

i

i

i

i

i

Section 4.2 The ACE Service Object Class 95

Figure 4.2: The ACEService Object Class

vice Configurator framework. This class provides the following capabilities:

� It provides hook methods that initialize a service and shut down aservice and clean up its resources.

� It provides hook methods to suspend service execution temporarilyand to resume execution of a suspended service.

� It provides a hook method that reports key information about theservice, such as its purpose and the port number where it listens forclient connections.

The interface for the ACE_Service_Object class is shown in Figure 4.2.By inheriting from ACE_Event_Handler and ACE_Shared_Object , sub-classes of ACE_Service_Object can be dispatched by the ACE Reactorframework and can be linked/unlinked dynamically from a DLL, respec-tively. The key methods of ACE_Service_Object are outlined in the fol-lowing table:

Method Descriptioninit() A hook method used by the Service Configurator framework to in-

struct a service to initialize itself. A pair of “argc /argv ”-style pa-rameters can be passed to init() and used to control the initial-ization of a service.

fini() A hook method used by the Service Configurator framework toinstruct a service to terminate itself. This method typically per-forms termination operations that release dynamically allocatedresources, such as memory, synchronization locks, or I/O descrip-tors.

suspend()resume()

Hook methods used by the Service Configurator framework to in-struct a service to suspend and resume its execution.

info() A hook method used by the Service Configurator framework toquery a service for certain information about itself, such as itsname, purpose, and network address. Clients can query a server toretrieve this information and use it to contact a particular servicerunning in a server.

These hook methods collectively impose a uniform contract between theACE Service Configurator framework and the services that it manages.

Application services that inherit from ACE_Service_Object can selec-tively override its hook methods, which are called back at the appropriate


i

i

i

i

i

i

i

i


time by the ACE Service Configurator framework in accordance to variousevents. This highly extensible technique allows applications to defer the se-lection of a particular service implementation until late in the design cycle,i.e., at installation-time or even during run-time. Developers can there-fore concentrate on a service’s functionality and other design dimensionswithout committing themselves prematurely to a particular service config-uration. Developers can also configure complete applications by compos-ing multiple services that are developed independently and therefore don’trequire a priori global knowledge of each other, yet can still collaborateeffectively.

Example

To illustrate the ACE_Service_Object class, we reimplement our Reactor-based logging server from the Example portion in Section 3.5. This revi-sion can be configured dynamically by the ACE Service Configurator frame-work, rather than configured statically into the main() program shown onpage 3.5. To accomplish this, we’ll create the following Reactor_Logging_Server_Adapter template:

template <class ACCEPTOR>class Reactor_Logging_Server_Adapter : public ACE_Service_Object{public:

// Hook methods inherited from <ACE_Service_Object>.virtual int init (int argc, char *argv[]);virtual int fini ();virtual int info (ACE_TCHAR **, size_t) const;virtual int suspend ();virtual int resume ();

private:Reactor_Logging_Server<ACCEPTOR> *server_;

};

This template inherits from ACE_Service_Object and contains a pointerto the Reactor_Logging_Server template defined on page 71. We’ve in-stantiated this template with the ACCEPTORparameter to defer our choiceof acceptor until later in the design cycle.

‘‘hs _2001 /page

i

i

i

i

i

i

i

i

Section 4.2 The ACE Service Object Class 97

When an instance of Reactor_Logging_Server_Adapter is configureddynamically, the ACE Service Configurator framework invokes its init()method:

template <class ACCEPTOR> intReactor_Logging_Server_Adapter<ACCEPTOR>::init

(int argc, char *argv[]){

server_ = new Reactor_Logging_Server<ACCEPTOR>(argc,

argv,ACE_Reactor::instance ());

return 0;}

This method creates an instance of Reactor_Logging_Server and keepstrack of it with the server_ pointer.

When instructed to remove the Reactor_Logging_Server_Adapter ,the ACE Service Configurator framework invokes its fini() method:

template <class ACCEPTOR> intReactor_Logging_Server_Adapter<ACCEPTOR>::fini (){

delete server_;return 0;

}

This method deletes the instance of Reactor_Logging_Server created ininit() .

The info() method is shown next:

1 template <class ACCEPTOR> int2 Reactor_Logging_Server_Adapter<ACCEPTOR>::info3 (ACE_TCHAR **bufferp, size_t length) const {4 ACE_INET_Addr sa;5 server_->acceptor ().get_local_addr (sa);67 ACE_TCHAR buf[BUFSIZ];8 sprintf (buf,9 "%d/%s %s",

10 sa.get_port_number (),11 "tcp",12 "# Reactor-based logging server\n");


i

i

i

i

i

i

i

i


13 *bufferp = ACE_OS::strnew (buf);14 ACE_OS::strcpy (*bufferp, buf);15 return ACE_OS::strlen (buf);16 }

Lines 1–5 Obtain the network address object from the instance of ACE_SOCK_Acceptor that’s used by the Reactor_Logging_Server .

Lines 7–12 Format an informative message that explains what the ser-vice does and how to contact it.

Lines 13–15 Allocate a new memory buffer dynamically, store the for-matted description string into this buffer, and return the buffer’s length.The caller is responsible for deleting the buffer.

The suspend() and resume() methods are similar to each other, asshown below:

template <class ACCEPTOR> intReactor_Logging_Server_Adapter<ACCEPTOR>::suspend (){

return ACE_Reactor::instance ()->suspend_handler (server_);}

template <class ACCEPTOR> intReactor_Logging_Server_Adapter<ACCEPTOR>::resume (){

return ACE_Reactor::instance ()->resume_handler (server_);}

Since Reactor_Logging_Server is a descendant of ACE_Event_Handler ,the server_ object can be passed to the singleton reactor’s suspend_handler() and resume_handler() methods, which double-dispatch toReactor_Logging_Server::get_handle() to extract the underlying passive-mode socket handle. These methods then use this socket handle to tem-porarily remove or resume the Reactor_Logging_Server from the list ofsocket handles handled by the singleton reactor.

The Example portion of Section 4.4 shows how the Reactor_Logging_Server_Adapter can be configured into and out of a generic server appli-cation dynamically.


i

i

i

i

i

i

i

i

Section 4.3 The ACE Service Repository and ACE Service Repository Iterator Classes 99

Figure 4.3: The ACEService Repository Class

4.3 The ACE Service Repository and ACE Service Repos-itory Iterator Classes

Motivation

The ACE Service Configurator framework supports the configuration ofboth single-service and multi-service servers. To simplify run-time ad-ministration of these servers, it’s often necessary to individually and/orcollectively access and control the service objects that comprise a server’scurrently active services. Rather than expecting application developers toprovide these capabilities in an ad hoc way, the ACE Service Configura-tor framework defines the ACE_Service_Repository and ACE_Service_Repository_Iterator classes.

Class Capabilities

The ACE_Service_Repository implements the Manager pattern [Som97]to control the life-cycle of—and the access to—service objects configured bythe ACE Service Configurator framework. This class provides the followingcapabilities:

� It keeps track of all service implementations that are configured intoan application and maintains each service’s status, such as whetherit is active or suspended and

� It allows the ACE Service Configurator framework to control when aservice is configured into or out of an application.

The interface for the ACE_Service_Repository class is shown in Fig-ure 4.3 and its key methods are outlined in the following table:


i

i

i

i

i

i

i

i

100 Section 4.3 The ACE Service Repository and ACE Service Repository Iterator Classes

Method DescriptionACE_Service_Repository()open()

Initialize the repository and allocate its dy-namic resources.

˜ACE_Service_Repository()close()

Close down the repository and release its dy-namically allocated resources.

insert() Add a new service into the repository.find() Locate an entry in the repository.remove() Remove an existing service from the reposi-

tory.suspend() Suspend a service in the repository.resume() Resume a suspended service in the repository.instance() A static method that returns a pointer to a sin-

gleton ACE_Service_Repository .

A search structure within the ACE_Service_Repository binds together

� The name of a service, which is represented as an ASCII string, and� An instance of ACE_Service_Type , which is the class used by the

ACE Service Configurator framework to link, initialize, suspend, re-sume, remove, and unlink AC_Service_Object s from a server stati-cally or dynamically.

Each ACE_Service_Type object contains:

� Accessors that set/get the type of service that resides in a repository.There are three types of services:

1. ACE_Service_Object_Type (ACE SVC OBJ T)—This type providesaccessors that set/get a pointer to the associated ACE_Service_Object .

2. ACE_Module_Type (ACE MODULE T)— TBD3. ACE_Stream_Type (ACE STREAM T)— TBD

� For dynamically linked service objects, an ACE_Service_Type storesthe handle of the underlying DLL where the object code resides. TheACE Service Configurator framework uses this handle to unlink andunload a service object from a running server when the service itoffers is no longer required.

Sidebar 10 shows how an application can use the ACE_Dynamic_Serviceclass to retrieve services from the ACE_Service_Repository programmat-ically.


i

i

i

i

i

i

i

i


Sidebar 10: The ACE Dynamic Service Template

The ACE_Dynamic_Service template can be used to retrieve services reg-istered with the ACE_Service_Repository . We use the C++ templateparameter to ensure that a pointer to the appropriate type of service isreturned, as shown below:

template <class TYPE>class ACE_Dynamic_Service{public:

// Use <name> to search the <ACE_Service_Repository>.static TYPE *instance (const ACE_TCHAR *name) {

const ACE_Service_Type *svc_rec;if (ACE_Service_Repository::instance ()->find

(name, &svc_rec) == -1) return 0;const ACE_Service_Type_Impl *type = svc_rec->type ();if (type == 0) return 0;void *obj = type->object ();return ACE_dynamic_cast (TYPE *, obj);

}};

An application can use the ACE_Dynamic_Service template as follows:

typedef Reactor_Logging_Server_Adapter<Logging_Acceptor>Server_Logging_Daemon;

Server_Logging_Daemon *logging_server =ACE_Dynamic_Service<Server_Logging_Daemon>::instance

("Server_Logging_Daemon");

cout << logging_server->info () << endl;

The ACE_Service_Repository_Iterator implements the Iterator pat-tern [GHJV95] to provide a way to access the ACE_Service_Type itemsin an ACE_Service_Repository sequentially without exposing its internalrepresentation. The interface for the ACE_Service_Repository_Iteratorclass is shown in Figure 4.4 and its key methods are outlined in the fol-lowing table:

‘‘hs _2001 /page

i

i

i

i

i

i

i

i


Figure 4.4: The ACEService Repository Iterator Class

Method DescriptionACE_Service_Repository_Iterator() Initialize the iterator.next() Passes back the next unseen ACE_

Service_Type in the repository.done() Returns 1 when all items have been

seen, else 0.advance() Move forward by one item in the

repository.

It’s important not to delete entries from an ACE_Service_Repository that’sbeing iterated upon since the ACE_Service_Repository_Iterator is notdesigned as a robust iterator [Kof93].

Example

The following example illustrates how ACE_Service_Repository and ACE_Service_Repository_Iterator can be used to implement a Service_Reporter class. This class provides a “meta-service” that clients can useto obtain information on all the services that the ACE Service Configuratorframework has configured into an application statically or dynamically. Aclient interacts with a Service_Reporter as follows:

� A client establishes a connection to the TCP port number where theService_Reporter object is listening.

� The Service_Reporter returns a list of all the server’s services to theclient.

� The client and Service_Reporter close their respective ends of theTCP/IP connection.

Sidebar 11 describes how the Service_Reporter class differs from theACE_Service_Manager class that’s bundled with the ACE toolkit.

The interface and implementation of the Service_Reporter class aredescribed below. We first create a file called Service_Reporter.h thatcontains the following class definition:

class Service_Reporter : public ACE_Service_Object{public:


i

i

i

i

i

i

i

i


Sidebar 11: The ACE Service Manager Class

ACE_Service_Manager provides clients with access to administrativecommands that publish and manage the services currently offered by anetwork server. These commands “externalize” certain internal attributesof the services configured into a server. During server configuration, anACE_Service_Manager is typically registered at a well-known communi-cation port, e.g., port 9411, accessible by clients. Clients can connect toan ACE_Service_Manager and send it the following commands:

� Report services—If the command “help ” is sent, a list of all ser-vices configured into an application via the ACE Service Configuratorframework is returned to the client.

� Reconfigure—If the command “reconfigure ” is sent, reconfigura-tion is triggered that will reread the local service configuration file.

� Process directive—If neither “help ” nor “reconfigure” is sent,the client’s command is passed to the ACE_Service_Config::process_directive() method, which is described on page 112.This feature enables remote configuration of servers via command-line instructions like

% echo "suspend My_Service" | telnet hostname 9411

// Hook methods inherited from <ACE_Service_Object>.virtual int init (int argc, char *argv[]);virtual int fini ();virtual int info (ACE_TCHAR **, size_t) const;virtual int suspend ();virtual int resume ();

private:// Acceptor instance.ACE_SOCK_Acceptor acceptor_;

enum { DEFAULT_PORT = 9411 };};

Service_Reporter can be configured by the ACE Service Configuratorframework since it inherits from ACE_Service_Object .

‘‘hs _2001 /page

i

i

i

i

i

i

i

i


The implementations of the Service_Reporter hook methods are placedinto the file Service_Reporter.cpp . The ACE Service Configurator frame-work calls the following Service_Reporter::init() hook method whenan instance of Service_Reporter is configured into an application:

1 int Service_Reporter::init (int argc, ACE_TCHAR *argv[]) {2 ACE_INET_Addr local_addr (Service_Reporter::DEFAULT_PORT);34 // Start at argv[0].5 ACE_Get_Opt get_opt (argc, argv, "p:", 0);67 for (int c; (c = get_opt ()) != -1;)8 switch (c) {9 case ’p’:

10 local_addr.set ((u_short) ACE_OS::atoi (get_opt.optarg));11 break;12 }1314 acceptor_.open (local_addr);15 return ACE_Reactor::instance ()->register_handler16 (acceptor_.get_handle (),17 this,18 ACE_Event_Handler::ACCEPT_MASK);19 }

Line 2 Initialize the local_addr to the default TCP port number used byService_Reporter .

Lines 5–12 Parse the service configuration options using the ACE_Get_Opt class described in Sidebar 12. If the “-p ” option is passed into init()then the local_addr port number is reset to that value.

Lines 14–18 Initialize the ACE_SOCK_Acceptor to listen on the local_addr port number and then register the instance of Service_Reporterwith the singleton reactor.

When a connection request arrives from a client, the singleton reac-tor dispatches the following Service_Reporter::handle_input() hookmethod:

1 int Service_Reporter::handle_input (ACE_HANDLE) {2 // Connection to the client (we only support3 // one client connection at a time).


i

i

i

i

i

i

i

i


Sidebar 12: The ACE Get Opt Class

The ACE_Get_Opt class is an iterator for parsing command-line argumentsthat provides the following features:

� It provides a C++ wrapper facade for the standard C librarygetopt() function. Unlike the standard getopt() function, however,each instance of ACE_Get_Opt contains its own state, so it can beused reentrantly.

� It also supports so-called “long” option formats, which ...

4 ACE_SOCK_Stream peer_stream;5 acceptor_.accept (peer_stream);67 ACE_Service_Repository_Iterator iterator8 (*ACE_Service_Repository::instance (), 0);9

10 for (const ACE_Service_Type *st;11 iterator.next (st) != 0;12 iterator.advance ()) {13 iovec iov[3];14 iov[0].iov_base = st->name ();15 iov[0].iov_len = strlen (iov[0].iov_base);16 iov[1].iov_base =17 st->active () ? " (active) " : " (paused) ";18 iov[1].iov_len = strlen (" (active) ");19 iov[2].iov_len = st->type ()->info (&iov[2].iov_base);20 peer_stream.sendv_n (iov, 3);21 delete [] iov[2].iov_base;22 }2324 peer_stream.close ();25 return 0;26 }

Lines 4–5 Accept a new client connection.

Lines 7–8 Initialize an ACE_Service_Repository_Iterator , which we’lluse to report all the active and suspended services offered by the server.Passing a 0 as the second argument to the constructor instructs it to returninformation on suspended services, which are ignored by default.

‘‘hs _2001 /page

i

i

i

i

i

i

i

i


Lines 10–22 Iterate through each service, invoke their info() methodto obtain a descriptive synopsis of the service, and send this informationback to the client via the connected socket.

Line 24 Close down the connection to the client and release the sockethandle.

The Service_Reporter::info() hook method passes back a stringthat explains what the service does and how to connect to it:

int Service_Reporter::info (ACE_TCHAR **bufferp,size_t length = 0) const {

ACE_INET_Addr sa;acceptor_.get_local_addr (sa);

ACE_TCHAR buf[BUFSIZ];sprintf (buf,

"%d/%s %s",sa.get_port_number (),"tcp","# lists all services in the daemon\n");

*bufferp = ACE_OS::strnew (buf);ACE_OS::strcpy (*bufferp, buf);return ACE_OS::strlen (buf);

}

As with the info() method on page 97, a caller must delete the dynami-cally allocated buffer.

The Service_Reporter ’s suspend() and resume() hook methods for-ward to the corresponding methods in the reactor singleton, as follows:

int Service_Reporter::suspend () {return ACE_Reactor::instance ()->suspend_handler (this);

}

int Service_Reporter::resume () {return ACE_Reactor::instance ()->resume_handler (this);

}

The Service_Reporter::fini() method is shown below:

int Service_Reporter::fini () {acceptor_.close ();


i

i

i

i

i

i

i

i


ACE_Reactor::instance ()->remove_handler(this, ACE_Event_Handler::ACCEPT_MASK);

}

This method closes the ACE_SOCK_Acceptor endpoint and removes theService_Reporter from the singleton reactor. We needn’t delete thisobject in handle_close() since the ACE Service Configurator frameworkis responsible for deleting a service object after calling its fini() hookmethod.

Finally, we add the necessary ACE service macros to the Service_Reporter implementation file. These macros create an instance of Service_Reporter and register it with the ACE_Service_Repository , as describedin Sidebar 13 on page 108.

1 ACE_SVC_FACTORY_DEFINE (Service_Reporter);23 ACE_STATIC_SVC_DEFINE {4 Service_Reporter,5 "Service_Reporter",6 ACE_SVC_OBJ_T,7 &ACE_SVC_NAME (Service_Reporter),8 ACE_Service_Types::DELETE_THIS |9 ACE_Service_Types::DELETE_OBJ,

10 0 // This object is not initially active.11 };1213 ACE_STATIC_SVC_REQUIRE (Service_Reporter);

Line 1 Use the ACE SVC FACTORY DEFINE macro to automatically generatethe following factory function:

extern "C" ACE_Service_Object *make_Service_Reporter (){ return new Service_Reporter; }

The macro that generates make_Service_Reporter() defines the functionwith extern "C" linkage. This C++ feature simplifies the design and im-proves the portability of the ACE_Service_Config implementation sinceit can fetch the make_Service_Reporter() function from a DLL’s symboltable without requiring knowledge of the C++ compiler’s name manglingscheme.


i

i

i

i

i

i

i

i


Sidebar 13: The ACE Service Factory Macros

The ACE Service Configurator framework creates new service objects byinvoking factory functions. ACE defines the following macros in the ace/OS.h header file to simplify the creation and use of factory functions:

ACE Macro DescriptionACE_SVC_FACTORY_DECLARE(T) Used in a header file to declare a factory

method for creating service objects of typeT. In this case, T is the name of a descen-dant of ACE_Service_Object .

ACE_SVC_FACTORY_DEFINE(T) Used in an implementation file to de-fine a factory function that will createa new object of type T. This functionwill match the declaration created byACE SVC FACTORY DECLARE() and will alsodefine an exterminator function for deletingthe object created by the factory.

ACE_STATIC_SVC_DECLARE(T) Used in a header file to declare that type Twill be used as a statically linked service.

ACE_STATIC_SVC_DEFINE(T,NAME,TYPE,FUNC,FLAGS,ACTIVE)

Used in an implementation file to de-fine an object of type ACE_Static_Svc_Descriptor . Parameter T is the class be-ing defined as a static service and the re-maining parameters match the attributesof the ACE_Static_Svc_Descriptor class.In order to use the function defined byACE SVC FACTORY DEFINE() as the allocatorfunction the value assigned to FUNC shouldbe &ACE_SVC_NAME(T).

ACE_STATIC_SVC_REQUIRE(T) Used in an implementation file to addthe ACE_Static_Svc_Descriptor definedby the ACE STATIC SVC DEFINE macro tothe ACE_Service_Config ’s list of static ser-vices.

Lines 3–11 The ACE STATIC SVC DEFINE macro initializes an instance ofACE_Static_Svc_Descriptor , which is a class that stores the informa-tion needed to describe a statically configured service. Service_Reporteris the service object’s class name and "Service_Reporter" is the nameused to identify the service. ACE SVC OBJ T is the type of service ob-


i

i

i

i

i

i

i

i

Section 4.4 The ACE Service Cong Class 109

ject container and &ACE_SVC_NAME(Service_Reporter) is the address ofthe make_Service_Reporter() factory function that creates an instanceof Service_Reporter . The DELETE THIS and DELETE OBJ are enumer-ated literals defined in the ACE_Service_Types class that ensure the con-tainer and the service object are deleted after the Service_Reporter isdestroyed.

Line 13 The ACE STATIC SVC REQUIRE macro automatically registers theinstance of the Service_Reporter ’s ACE_Static_Svc_Descriptor objectwith the ACE_Service_Repository .

The Example portion of Section 4.4 shows how a Service_Reportercan be configured statically into a server application.

4.4 The ACE Service Config Class

Motivation

Before a service can execute, it must be configured into an application’s ad-dress space. One way to configure the services that comprise a networkedapplication is to statically link the functionality provided by its variousclasses and functions into separate OS processes. We used this approachin the logging server examples in Chapter 3 and throughout [SH02], wherethe logging server program runs in a process that handles log records fromclient applications. Although our use of the ACE Reactor framework in theprevious chapter improved the modularity and portability of the networkedlogging server, the following drawbacks arose from statically configuratingthe Reactor_Logging_Server class with its main() program:

� Service configuration decisions are made prematurely in the de-velopment cycle, which is undesirable if developers don’t know thebest way to collocate or distribute services a priori. Moreover, the“best” configuration may change as the computing context changes.For example, an application may write log records to a local file whenit’s running on a disconnected laptop computer. When the laptop isconnected to a LAN, however, it may forward log records to a cen-tralized logging server.1 Forcing networked applications to commit

1If “wait a minute—you don’t need a logging service for local disk file logging!” comesto mind, you’re correct. ACE makes the switch easy via the ACE_Logging_Strategy classthat’s covered in Appendix ??.


i

i

i

i

i

i

i

i

110 Section 4.4 The ACE Service Cong Class

prematurely to a particular service configuration impedes their flexi-bility and can reduce performance and functionality, as well as incurcostly redesign and reimplementation later in a project’s lifecycle.

� Modifying a service may adversely affect other services if the im-plementation of a service is coupled tightly with its initial configura-tion. To enhance reuse, for example, a logging server may initiallyreside in the same program as other services, such as a name service.If the name service lookup algorithm is changed, however, all exist-ing code in the server would require modification, recompilation, andstatic relinking. Moreover, terminating a running process to changeits name service code would also terminate the collocated logging ser-vice. This disruption in service may not be acceptable for highly avail-able systems, such as telecommunication switches or customer carecall centers [SS94].

� System performance may scale poorly since associating a separateprocess with each service ties up OS resources, such as I/O descrip-tors, virtual memory, and process table slots. This design is partic-ularly wasteful if services are often idle. Moreover, processes can beinefficient for many short-lived communication tasks, such as askinga time service for the current time or resolving a host address requestvia the Domain Name Service (DNS).

To address these drawbacks, the ACE Service Configurator framework de-fines the ACE_Service_Config class.

Class Capabilities

The ACE_Service_Config class implements the Facade pattern [GHJV95]to integrate the other ACE Service Configurator classes and coordinate theactivities necessary to manage the services in an application process. Thisclass provides the following capabilities:

� It provides mechanisms to dynamically link and unlink service imple-mentations into and out of an application process.

� It interprets a simple scripting language that allows applications oradministrators to provide the ACE Service Configurator frameworkwith commands, called “directives,” to locate and initialize a ser-vice’s implementation at run-time, as well as to suspend, resume,re-initialize, and/or terminate a component after it’s been initialized.This interpretation can (re)configure an application process either


i

i

i

i

i

i

i

i


Figure 4.5: The ACEService Config Class

– In batch-mode, i.e., via a series of directives in a configurationfile, known as the svc.conf file or

– Interactively, i.e., by passing directives via strings.

The interface for the ACE_Service_Config class is shown in Figure 4.5.The ACE_Service_Config has a rich interface since it exports all the fea-tures in the ACE Service Configurator framework. We therefore group thedescription of its methods into the three categories described below.

1. Service Configurator lifecycle management methods. The followingmethods initialize and terminate the ACE_Service_Config :

ACE Class DescriptionACE_Service_Config()open()

These methods create and initialize the ACE_Service_Config .

˜ACE_Service_Config()close()

These methods shut down and finalize all the con-figured services and deletes the resources allo-cated when the ACE_Service_Config was initial-ized.

There is only one instance of the ACE_Service_Config in a processsince this class is a variant of the Monostate pattern [CB97], which ensuresa unique state for all objects by declare all data members to be static.Moreover, the ACE_Service_Config methods are also declared as static .

The open() method is the most common way to initialize the ACE_Service_Config . It parses arguments passed in the argc and argv pa-rameters, which are illustrated in the following table:


i

i

i

i

i

i

i

i


Option Description’-b’ Turn the application process into a daemon.’-d’ Turn on debugging mode, which displays diagnostic information as

directives are processed.’-f’ Supply a file containing directives other than the default svc.conf

file. This argument can be repeated to process multiple configurationfiles.

’-n’ Don’t process any static directives, which eliminates the need toinitialize the ACE_Service_Repository statically.

’-s’ Designate the signal to be used to cause the ACE_Service_Config toreprocess its configuration file. By default, SIGHUP is used.

’-S’ Supply a directive to the ACE_Service_Config directly. This argu-ment can be repeated to process multiple directives.

’-y’ Process static directives, which requires the static initialization ofthe ACE_Service_Repository .

2. Service configuration methods. After parsing all its argc /argv ar-guments, the ACE_Service_Config::open() method calls one or both ofthe following methods to configure the server:

Method Descriptionprocess_directives() Process a sequence of directives that are stored

in a designated script file, which defaults to svc.conf . This method allows multiple directives tobe stored persistently and processed iteratively inbatch-mode. This method executes each serviceconfiguration directive in a svc.conf file in the or-der in which they are specified.

process_directive() Process a single directive passed as a string param-eter. This method allows directives to be created dy-namically and/or processed interactively.

The following table summarizes the service configuration directives thatcan be interpreted by these two ACE_Service_Config methods:

Directive Descriptiondynamic Dynamically link and initialize a service.static Initialize a service that was linked statically.remove Remove a service completely, e.g., unlink it from the application

process.suspend Suspend service without completely removing it.resume Resume a service that was suspended earlier.stream Initialize an ordered list of hierarchically-related modules.


i

i

i

i

i

i

i

i


We describe the syntax and semantics for the tokens in each of these di-rectives below.

� Dynamically link and initialize a service: dynamic svc_nameService_Object * DLL:factory_func() ["argc/argv options"]

The dynamic directive is used to dynamically link and initialize a ser-vice object. The svc_name is the name assigned to the service. DLL isthe name of the dynamic link library that contains factory_func() ,which is an extern "C" function that ACE_Service_Config invokesto create an instance of a service. The factory_func() must returna pointer to an object derived from ACE_Service_Object .

The DLL can either be a full pathname or a filename without a suffix.If it’s a full pathname, the ACE_DLL::open() method described inSidebar 14 is used to dynamically link the designated file into theapplication process. If it’s a filename, however, ACE will use OS-dependent mechanisms to locate the file, as follows:

– DLL filename expansion—It will determine the name of the DLLby adding the appropriate prefix and suffix, such as the lib pre-fix and .so suffix for Solaris shared libraries or the .dll suffixfor Win32 DLLs, and

– DLL search path—It will search for the designated expanded DLLfilename using the platform’s DLL search path lookup environ-ment variable, e.g., $LD LIBRARY PATH on many UNIX systems or$PATH on Win32.

After ACE_Service_Config locates the file, it dynamically links it intothe address space of the process via ACE_DLL::open() . The dynamicdirective can be used portably across a range of OS platforms sinceACE encapsulates these platform-dependent DLL details.

The argc /argv options are a list of parameters that can be suppliedto initialize a service object via its init() method. The ACE_Service_Config substitutes the values of environment variables that are in-cluded in an options string.

� Initialize a statically linked service: static svc_name ["argc/argv options"]

Although the ACE_Service_Config is commonly used to configureservices dynamically, it also supports the static configuration of ser-vices via the static directive. The svc_name and argc /argv options


i

i

i

i

i

i

i

i


Sidebar 14: The ACE DLL Class

Dynamically linking and unlinking DLLs presents the following problems thatare similar to those with native OS Sockets API in Chapter 2 of [SH02]:

– Non-uniform DLL programming interface that’s much lessportable than the Socket API. Whereas the BSD Socket API is nearlyubiquitous, the API for linking and unlinking DLL varies greatly be-tween platforms.

– Unsafe types that invite errors and misuse because various DLLmechanisms return a weakly-typed handle that’s passed to DLL func-tions, such as looking up symbols and unlinking the DLL. Since there’snothing that distinguishes these handles from other handles, however,their use is error-prone.

To address these problems, the ACE_DLLwrapper facade class abstractsthe functionality necessary to use a DLL object itself, rather than dealingwith procedural concepts and types. The ACE_DLLclass eliminates theneed for applications to use weakly-typed handles and also ensures re-sources are released properly on object destruction.

The interface of ACE_DLLis shown in the figure above and its key meth-ods are outlined in the following table:

ACE Class DescriptionACE_DLL()open()

Opens and dynamically links a designated DLL.

˜ACE_DLLclose()

Closes the DLL.

symbol() Return a pointer to the designated symbol in the symbol ta-ble of the DLL.

error() Return a string explaining which failure occurred.

are the same as those in the dynamic directive. The syntax is sim-pler, however, since the service object must be linked into the programstatically, rather than linked dynamically. Static configuration tradesflexibility for increased security, which may be useful for certain typesof servers that must contain only trusted, statically linked services.

� Remove a service completely: remove svc_name

The remove directive causes the ACE_Service_Config to query the


i

i

i

i

i

i

i

i


ACE_Service_Repository for the designated svc_name service. If itlocates this service, it invokes its fini() hook method, which per-forms the activities needed to clean up resources when the serviceterminates. If a service destruction function pointer is associated withthe service object, it’s called to destroy the service object itself. TheACE SVC FACTORY DEFINE macro defines this function automatically.Finally, if the service was linked dynamically from a DLL, it’s unlinkedvia the ACE_DLL::close() method.

� Suspend a service without removing it: suspend svc_name

The suspend directive causes the ACE_Service_Config to query theACE_Service_Repository for the designated svc_name service. Ifthis service is located, its suspend() hook method is invoked. A ser-vice can override this method to implement the appropriate actionsneeded to suspend its processing.

� Resume a service suspended previously: resume svc_name

The resume directive causes the ACE_Service_Config to query theACE_Service_Repository for the designated svc_name service. Ifthis service is located, its resume() hook method is invoked. A servicecan override this method to implement the appropriate actions neededto resume its processing, which typically reverse the effects of thesuspend() method.

� Initialize an ordered list of hierarchically-related modules: streamsvc_name ’f’ module-list ’g’

The stream directive causes the ACE_Service_Config to initializean ordered list of hierarchically-related modules. Each module con-sists of a pair of services that are interconnected and communicateby passing ACE_Message_Block s. The implementation of the streamdirective uses the ACE Streams framework described in Chapter ??.

The complete Backus/Naur Format (BNF) syntax for svc.conf files parsedby the ACE_Service_Config is shown in Figure 4.6. Sidebar 15 describeshow to specify svc.conf files using XML syntax.

3. Utility methods. The key methods are outlined below:

Method Description() .

‘‘hs _2001 /page

i

i

i

i

i

i

i

i


<svc-config-entries> ::= svc-config-entries svc-config-entry | NULL<svc-config-entry> ::= <dynamic> | <static> | <suspend> |

<resume> | <remove> | <stream><dynamic> ::= dynamic <svc-location> <parameters-opt><static> ::= static <svc-name> <parameters-opt><suspend> ::= suspend <svc-name><resume> ::= resume <svc-name><remove> ::= remove <svc-name><stream> ::= stream <svc-name> ’{’ <module-list> ’}’<module-list> ::= <module-list> <module> | NULL<module> ::= <dynamic> | <static> | <suspend> |

<resume> | <remove><svc-location> ::= <svc-name> <type> <svc-initializer> <status><type> ::= Service_Object ’*’ | Module ’*’ | Stream ’*’ | NULL<svc-initializer> ::= PATHNAME ’:’ FUNCTION ’(’ ’)’<svc-name> ::= STRING<status> ::= active | inactive | NULL<parameters-opt> ::= STRING | NULL

Figure 4.6: BNF for the ACEService Config Scripting Language

Sidebar 15: Using XML to Configure Services

Example

This example illustrates how to apply ACE_Service_Config and the otherclasses in the ACE Service Configurator framework to configure a serverthat behaves as follows:

� It statically configures an instance of Server_Reporter .� It dynamically configures the Reactor_Logging_Server_Adapter tem-

plate from the Example portion of Section 4.2 into the address spaceof a server.

We then show how to dynamically reconfigure the server to support a dif-ferent implementation of a reactor-based logging service.

Initial Configuration. The main() program below configures the Service_Reporter and Reactor_Logging_Server_Adapter services into an appli-cation process and then runs the reactor’s event loop.


i

i

i

i

i

i

i

i


#include "ace/Service_Config.h"#include "ace/Reactor.h"


ACE_Service_Config::open (argc, argv);

ACE_Reactor::instance ()->run_reactor_event_loop ();

return 0;}

There are no service-specific header files or code in the main() program.The genericity of this server illustrates the power of the ACE Service Con-figurator framework.

When ACE_Service_Config::open() is called, it uses the ACE_Service_Config::process_directives() method to interpret the following svc.conf file:

1 static Server_Reporter "-p $SERVER_REPORTER_PORT"23 dynamic Server_Logging_Daemon Service_Object *4 SLD:make_Server_Logging_Daemon() "$SERVER_LOGGING_DAEMON_PORT"

Line 1 Initialize the Server_Reporter instance that was linked stati-cally together with the main() program. The ACE STATIC SVC REQUIRE

macro used in the Service_Reporter.cpp file on page 107 ensures theService_Reporter object is registered with the ACE_Service_Repositorybefore the ACE_Service_Config::open() method is called.

Line 3–4 Dynamically link the SLD DLL into the address space of theprocess and use ACE_DLLto extract the make_Server_Logging_Daemon()factory function from the SLDsymbol table. This function is called to obtaina pointer to a dynamically allocated Server_Logging_Daemon . The frame-work then calls the Server_Logging_Daemon::init() hook method onthis pointer, passing in the "$SERVER_LOGGING_DAEMON_PORT"string asits argc /argv argument. This string designates the port number where theserver logging daemon listens for client connection requests. If init() suc-ceeds, the Server_Logging_Daemon pointer is stored in the ACE_Service_Repository under the name "Server_Logging_Daemon" .

The SLD DLL is generated from the following SLD.cpp file:


i

i

i

i

i

i

i

i


typedef Reactor_Logging_Server_Adapter<Logging_Acceptor>Server_Logging_Daemon;

ACE_SVC_FACTORY_DEFINE (Server_Logging_Daemon);

This file defines a typedef called Server_Logging_Daemon that instanti-ates the Reactor_Logging_Server_Adapter template with the Logging_Acceptor shown on page 50 of Section 3.3. The ACE SVC FACTORY DEFINE

macro is then used to generate the make_Server_Logging_Daemon() fac-tory function automatically.

The UML state diagram in Figure 4.7 illustrates the steps involved inconfiguring the server logging daemon based on the svc.conf file shownabove. When the OPEN event occurs at run-time, the ACE_Service_Config

Figure 4.7: A State Diagram for Configuring the Server Logging Dae-mon

class calls process_directives() , which consults the svc.conf file.When all the configuration activities have been completed, the main()

program invokes the ACE_Reactor::run_reactor_event_loop() method,which in turn calls the Reactor::handle_events() method continuously.As shown in Figure 4.7, this method blocks awaiting the occurrence ofevents, such as connection requests or data from clients. As these eventsoccur, the reactor dispatches the handle_input() method of concreteevent handlers automatically to perform the designated services.

Reconfigured Server. The ACE Service Configurator framework can beprogrammed to reconfigure a server at run-time in response to externalevents, such as the SIGHUP or SIGINT signal. At this point, the frame-work rereads its configuration file(s) and performs the designated direc-tives, such as inserting or removing service objects into or from a server,and suspending or resuming existing service objects. We now illustratehow to use these features to dynamically reconfigure our server loggingdaemon.

The initial configuration of the logging server shown above used theLogging_Acceptor implementation from page 50 of Section 3.3. This im-plementation didn’t timeout logging handlers that remained idle for longperiods of time. To add this capability without affecting existing code orthe Service_Reporter service in the process, we can simply define a new


i

i

i

i

i

i

i

i


svc.conf file and instruct the server to reconfigure itself by sending it theappropriate signal.

1 remove Server_Logging_Daemon23 dynamic Server_Logging_Daemon Service_Object *4 SLDex:make_Server_Logging_Daemon_Ex() "$SERVER_LOGGING_DAEMON_PORT"

Line 1 Remove the existing server logging daemon from the ACE_Service_Repository and unlink it from the application’s address space.

Lines 3–4 Configure a different instantiation of the Reactor_Logging_Server_Adapter template into the address space of the server loggingdaemon. In particular, the make_Server_Logging_Daemon_Ex() factoryfunction shown in the SLDex.cpp file below instantiates the Reactor_Logging_Server_Adapter template with the Logging_Acceptor_Ex shownon page 58 of Section 3.4.

typedef Reactor_Logging_Server_Adapter<Logging_Acceptor_Ex>Server_Logging_Daemon_Ex;

ACE_SVC_FACTORY_DEFINE (Server_Logging_Daemon_Ex);

The UML state diagram in Figure 4.8 illustrates the steps involved inreconfiguring the server logging daemon based on the svc.conf file shownabove.

Figure 4.8: A State Diagram for Reconfiguring the Server Logging Dae-mon

The dynamic reconfiguration mechanism in the ACE Service Configu-rator framework enables developers to modify server functionality or fine-tune performance without extensive redevelopment and reinstallation ef-fort. For example, debugging a faulty implementation of the logging servicecan simply involve the dynamic reconfiguration of a functionally equiva-lent service that contains additional instrumentation to help identify theerroneous behavior. This reconfiguration process may be performed with-out modifying, recompiling, relinking, or restarting the currently executingserver logging daemon. In particular, this reconfiguration needn’t affectthe Service_Reporter that was configured statically.


i

i

i

i

i

i

i

i


4.5 Summary

This chapter described the ACE Service Configurator framework, which al-lows services to be initiated, suspended, resumed, and terminated dynam-ically and/or statically. This framework helps to improve the extensibilityand performance of networked application software by deferring serviceconfiguration decisions until late in the design cycle, i.e., at installation-time and/or at run-time, without changing the application or server imple-mentations.

We applied the ACE Service Configurator pattern to enhance the net-worked logging service example described in previous chapters. The resultis a networked logging service that can be configured and deployed in vari-ous ways via the ACE Service Configurator framework. The logging serviceprovides a good example of why it’s useful to defer configuration decisionsuntil run-time. The extensibility afforded by the ACE Service Configura-tor framework allows operators and administrators to dynamically selectthe features and alternative implementation strategies that make the mostsense in a particular context, as well as make localized decisions on howbest to initialize them.


i

i

i

i

i

i

i

i

CHAPTER 5

The ACE Task Framework

CHAPTER SYNOPSIS

This chapter describes the design and use of the ACE Task framework,which can be used to implement common concurrency patterns [SSRB00],such as Active Object and Half-Sync/Half-Async. We show how to applythe ACE Task framework to enhance the concurrency of various parts ofour networked logging service.

5.1 Overview

The ACE_Thread_Manager wrapper facade class described in Chapter 9of [SH02] implements portable multithreading capabilities. This class onlyoffers a function-oriented interface, however, since programmers pass anentry point function to its spawn() and spawn_n() methods. To pro-vide a more powerful and extensible object-oriented concurrency mecha-nism, ACE defines the ACE Task concurrency framework. This frameworkprovides a facility for spawning threads in the context of an object anda flexible queueing mechanism for transferring messages between tasksefficiently.

The ACE Task framework can be applied to implement common con-currency patterns [SSRB00], such as:

� The Active Object pattern, which decouples the thread used to invokea method from the thread used to execute a method. Its purpose isto enhance concurrency and simplify synchronized access to objectsthat reside in their own threads.

121


i

i

i

i

i

i

i

i


� The Half-Sync/Half-Async pattern, which decouples asynchronousand synchronous processing in concurrent systems to simplify pro-gramming without reducing performance unduly. This pattern in-troduces three layers: one for asynchronous processing, one for syn-chronous service processing, and a queueing layer that mediates com-munication between the asynchronous and synchronous layers.

This chapter explains how these patterns—and the ACE Task frameworkthat reifies them—can be applied to develop concurrent object-orientedapplications at a level higher than existing C APIs and the ACE_Thread_Manager C++ wrapper facade. We focus on the following ACE Task frame-work classes that networked applications can use to spawn, manage, andcommunicate between one or more threads within a process:

ACE Class DescriptionACE_Message_Queue Provides a powerful message queueing facility that en-

ables applications to pass and queue messages be-tween threads in a process.

ACE_Task Allows applications to create active objects that canqueue and process messages concurrently.

ACE_Thread_Manager ACE_Task0..1 *

SYNCH

ACE_Message_Queue

SYNCH

ACE_Message_Block* 1

Figure 5.1: The ACE Task Framework Classes

The most important relationships between the classes in the ACE Taskframework are shown in Figure 5.1. This framework provides the followingbenefits:

� Improve the consistency of programming style by enabling de-velopers to use C++ and object-oriented techniques throughout their


i

i

i

i

i

i

i

i

Section 5.2 The ACE Message Queue Class 123

concurrent networked applications. For example, the ACE_Task classprovides an object-oriented programming abstraction that associatesOS-level threads with C++ objects.

� Manage a group of threads as a cohesive collection. Networkedapplications that use multithreading often require multiple threadsto start and end as a group. The ACE_Task class therefore provides athread group capability that allows other threads to wait for an entiregroup of threads to exit before proceeding with their processing.

This chapter motivates and describes the capabilities of each class inthe ACE Task concurrency framework. We illustrate how this frameworkcan be used to enhance the concurrency of our client and server loggingdaemons.

5.2 The ACE Message Queue Class

Motivation

Although some operating systems supply intra-process message queuesnatively, this capability isn’t available on all OS platforms. When it isoffered, moreover, it’s either:

� Highly non-portable, e.g., VxWorks message queues and/or� Inefficient, tedious, and error-prone to use, e.g., System V IPC

message queues.

It may be possible to encapsulate the spectrum of available message queuemechanisms within a wrapper facade that emulates missing capabilitieswhere needed. Since ACE already supplies the convenient, efficient, andportable ACE_Message_Block class described in Chapter 4 of [SH02], how-ever, it’s easier and more portable to create a lightweight message queueingmechanism that can be adapted easily to an OS platform’s threading ca-pabilities. The result is the ACE_Message_Queue class.

Class Capabilities

The ACE_Message_Queueclass is a platform-independent, lightweight intra-process message queueing mechanism that provides the following capabil-ities:


i

i

i

i

i

i

i

i

124 Section 5.2 The ACE Message Queue Class

� It allows messages (which are instances of ACE_Message_Block ) tobe enqueued at the front of the queue, the rear of the queue, or inpriority order. Messages are dequeued from the front of the queue.

� Its use of ACE_Message_Block s provides an efficient message buffer-ing mechanism that minimizes dynamic memory allocation and datacopying.

� It can be instantiated for either multi-threaded or single-threadedconfigurations, which allows programmers to trade off strict synchro-nization for less overhead when concurrent access to a queue isn’trequired.

� In multi-threaded configurations it supports configurable flow con-trol, which prevents fast message producer threads from swampingthe resources of slower message consumer threads.

� It allows timeouts to be specified on both enqueue and dequeue oper-ations to avoid indefinite blocking.

� It provides allocators that can be strategized so the memory usedby messages can be obtained from various sources, such as sharedmemory, heap memory, static memory, or thread-specific memory.

The interface for the ACE_Message_Queue class is shown in Figure 5.2and its key methods are outlined in the following table:

Method DescriptionACE_Message_Queue()open()

Initialize the queue.

˜ACE_Message_Queue()close()

Shutdown the queue and release its resources.

is_empty()is_full()

Checks if the queue is empty/full.

enqueue_tail() Insert a message at the back of the queue.enqueue_head() Insert a message at the head of the queue.enqueue_prio() Insert a message according to its priority.dequeue_head() Remove and return the message at the front of the

queue.

The ACE_Message_Queue has powerful semantics. We therefore groupthe description of its capabilities into the three categories described below.

1. Message buffering and flow control. The semantics of the ACE_Message_Queue class are based on the design of the message buffering and


i

i

i

i

i

i

i

i


+ open (high : size_t, low : size_t) : int+ is_empty () : int+ is_full () : int+ enqueue_tail (item : ACE_Message_Block *, timeout : ACE_Time_Value *) : int+ enqueue_head (item : ACE_Message_Block *, timeout : ACE_Time_Value *) : int+ enqueue_prio (item : ACE_Message_Block *, timeout : ACE_Time_Value *) : int+ dequeue_head (item : ACE_Message_Block *&, timeout : ACE_Time_Value *) : int+ close () : int

# head_ : ACE_Message_Block *# tail_ : ACE_Message_Block *# high_water_mark_ : size_t# low_water_mark_ : size_t

ACE_Message_QueueSYNCH_STRATEGY

Figure 5.2: The ACEMessage Queue Class

queueing facilities in System V STREAMS [Rag93]. Since messages passedbetween threads using a message queue are instances of ACE_Message_Block , two kinds of messages can be placed in an ACE_Message_Queue:

� Simple messages, which contain a single ACE_Message_Block .� Composite messages, which contain multiple ACE_Message_Block

objects that are linked together in accordance with the Compositepattern [GHJV95], which provides a structure for building recursiveaggregations. Composite messages generally consist of a control mes-sage followed by one or more data messages, which play the followingroles:

– A control message contains bookkeeping information, such asdestination addresses and length fields and

– The data message(s) contain the actual contents of a compositemessage.

Figure 5.3 illustrates how simple and composite messages can be linkedtogether to form an ACE_Message_Queue. To optimize insertion and dele-tion in a queue, ACE_Message_Block messages are linked bi-directionally


i

i

i

i

i

i

i

i


nneexxtt__pprreevv__ccoonntt__

AACCEE__DDaattaa__BBlloocckk

AACCEE__MMeessssaaggee__BBlloocckk

SSYYNNCCHH SSTTRRAATTEEGGYY




AACCEE__MMeessssaaggee__QQuueeuuee

hheeaadd__ttaaiill__







Figure 5.3: The Structure of an ACEMessage Queue

via a pair of pointers that can be obtained via the ACE_Message_Blocknext() and prev() accessor methods. Composite messages are chainedtogether uni-directionally via the continuation pointer in each ACE_Message_Block , which can be obtained via its cont() accessor method. Side-bar 16 describes the ACE_Message_Queue_Ex class, which is a variant ofACE_Message_Queue that’s more strongly typed.

Sidebar 16: The ACE Message Queue Ex Class

The ACE_Message_Queue enqueues and dequeues ACE_Message_Blockobjects since they provide a highly flexible and dynamically extensiblemeans of representing messages. For situations where strongly-type mes-saging is required, ACE provides the ACE_Message_Queue_Ex class, whichenqueues and dequeues messages that are instances of a MESSAGE_TYPEtemplate parameter, rather than an ACE_Message_Block .

An ACE_Message_Queue contains a pair of high and low water markvariables to implement flow control, which prevents a fast sender threadfrom overrunning the buffering and computing resources of a slower re-ceiver thread. The high water mark indicates the total number of messagebytes an ACE_Message_Queue is willing to buffer before it becomes flowcontrolled, at which point enqueue_*() methods will block until a suffi-


i

i

i

i

i

i

i

i


cient number of bytes of messages are dequeued. The low water mark in-dicates the number of message bytes at which a previously flow controlledACE_Message_Queue is no longer considered full. The Example portion ofSection 5.3 illustrates the use of high and low water marks to exert flowcontrol within a multithreaded logging server.

2. Parameterized synchronization strategies. If you look carefully atthe ACE_Message_Queue template in Figure 5.2 you’ll see that it’s param-eterized by a SYNCH_STRATEGYclass. This design is based on the Strate-gized Locking pattern [SSRB00], which parameterizes the synchronizationmechanisms that a class uses to protect its critical sections from concur-rent access. Internally, the ACE_Message_Queue class uses the followingtraits from its SYNCH_STRATEGYtemplate parameter:

template <class SYNCH_STRATEGY>class ACE_Message_Queue{

// ...protected:

// C++ traits that coordinate concurrent access.typename SYNCH_STRATEGY::MUTEX lock_;typename SYNCH_STRATEGY::CONDITION notempty_;typename SYNCH_STRATEGY::CONDITION notfull_;

};

These traits enable the ACE_Message_Queue synchronization strategy to becustomized to suit particular needs. Sidebar 17 describes the C++ traitsidiom.

Sidebar 17: The C++ Trait Idiom

A trait is a type that conveys information used by another class or algorithmto determine policies or implementation details at compile time. A traitsclass [Jos99] is a useful way to collect a set of characteristics that should beapplied in a given situation to alter another class’s behavior appropriately.

The C++ traits idiom and traits classes are widely used throughout theC++ standard library [Aus98]. For example, ...

Sets of ACE’s synchronization wrapper facades can be combined to formtraits classes that define customized synchronization strategies. ACE pro-vides the following two traits classes that pre-package the most commonsynchronization traits:


i

i

i

i

i

i

i

i


� ACE_NULL_SYNCH—The traits in this class are implemented in termsof “null” locking mechanisms, as shown below.

class ACE_NULL_SYNCH{public:

typedef ACE_Null_Mutex MUTEX;typedef ACE_Null_Mutex NULL_MUTEX;typedef ACE_Null_Mutex PROCESS_MUTEX;typedef ACE_Null_Mutex RECURSIVE_MUTEX;typedef ACE_Null_Mutex RW_MUTEX;typedef ACE_Null_Condition CONDITION;typedef ACE_Null_Semaphore SEMAPHORE;typedef ACE_Null_Semaphore NULL_SEMAPHORE;

};

The ACE_NULL_SYNCHclass is an example of the Null Object pat-tern [Woo97], which simplifies applications by defining a “no-op” place-holder that removes conditional statements in a class implementa-tion. ACE_NULL_SYNCHis often used in single-threaded applications,or in applications where the need for inter-thread synchronizationhas been either eliminated via careful design or implemented via someother mechanism.

� ACE_MT_SYNCH—The traits in this pre-defined class are implementedin terms of actual locking mechanisms, as shown below:class ACE_MT_SYNCH{public:

typedef ACE_Thread_Mutex MUTEX;typedef ACE_Null_Mutex NULL_MUTEX;typedef ACE_Process_Mutex PROCESS_MUTEX;typedef ACE_Recursive_Thread_Mutex RECURSIVE_MUTEX;typedef ACE_RW_Thread_Mutex RW_MUTEX;typedef ACE_Condition_Thread_Mutex CONDITION;typedef ACE_Thread_Semaphore SEMAPHORE;typedef ACE_Null_Semaphore NULL_SEMAPHORE;

};

The ACE_MT_SYNCHtraits class defines a strategy with portable, ef-ficient synchronization classes suitable for multi-threaded applica-tions.

Parameterizing the ACE_Message_Queue template with a traits classprovides the following benefits:

� It allows ACE_Message_Queues to work correctly and efficiently in


i

i

i

i

i

i

i

i


both single-threaded or multi-threaded configurations, without re-quiring changes to the class implementation and

� It allows the synchronization aspects of a particular instantiation ofthe ACE_Message_Queue template to be changed whole-sale via theStrategized Locking pattern.

For example, if the ACE_NULL_SYNCHstrategy is used, the ACE_Message_Queue’s MUTEXand CONDITIONtraits resolve to ACE_Null_Mutex and ACE_Null_Semaphore , respectively. In this case, the resulting message queueclass behaves like a non-synchronized message queue and incurs no syn-chronization overhead.

In contrast, if an ACE_Message_Queue is parameterized with the ACE_MT_SYNCHstrategy, its MUTEXand CONDITIONtraits resolve to ACE_Thread_Mutex and ACE_Condition_Thread_Mutex , respectively. In this case, theresulting message queue class behaves in accordance with the MonitorObject design pattern [SSRB00], which

� Synchronizes concurrent method execution to ensure that only onemethod at a time runs within an object and

� Allows an object’s methods to schedule their execution sequences co-operatively.

3. Blocking and timeout semantics. When an ACE_Message_Queuetemplate is instantiated with ACE_MT_SYNCHits synchronized enqueue anddequeue methods support blocking, non-blocking, and timed operations.If a synchronized queue is empty then calls to its dequeue_head() methodwill block until a message is enqueued and the queue is no longer empty.Likewise, if a synchronized is full calls to its enqueue_head() or enqueue_tail() method will block until a message is dequeued and the queue is nolonger full. The default blocking behavior can be modified by passing thefollowing types of ACE_Time_Value values to these methods:


i

i

i

i

i

i

i

i


Value BehaviorNULL ACE_Time_Valuepointer

Indicates that the enqueue or dequeue methodshould wait indefinitely, i.e., it will block untilthe method completes, the queue is closed, or asignal occurs.

Non-NULL ACE_Time_Valuepointer whose sec() andusec() methods return 0

Indicates that enqueue and dequeue methodsshould perform a “peek,” i.e., if the methoddoesn’t succeed immediately, return �1 and seterrno to EWOULDBLOCK.

A non-NULLACE_Time_Value pointerwhose sec() or usec()method returns > 0

Indicates that enqueue or dequeue methodshould wait until the absolute time elapses, re-turning �1 with errno set to EWOULDBLOCK ifthe method does not complete by this time.

The Client_Logging_Daemon example on page 132 illustrates the use ofthe ACE_MT_SYNCHtraits class.

In summary, the ACE_Message_Queue implementation applies the fol-lowing patterns and idioms from POSA2 [SSRB00]:

� Strategized Locking—C++ traits are used to strategize the synchro-nization mechanism in accordance with the Strategized Locking pat-tern [SSRB00].

� Monitor Object—When ACE_Message_Queue is parameterized by ACE_MT_SYNCHits methods behave as synchronized methods in accor-dance with the Monitor Object pattern [SSRB00].

� Thread-Safe Interface—The public methods acquire locks and dele-gate to the private implementation methods, which assume locks areheld and actually enqueue/dequeue messages.

� Scoped Locking—The ACE_GUARD*macros from Chapter 10 of [SH02]ensure that any synchronization wrapper facade whose signature con-forms to the ACE_LOCK*pseudo-class is acquired and released auto-matically in accordance with the Scoped Locking idiom [SSRB00].

These and other patterns from the POSA2 book are used to implement keyportions of the ACE Task framework. A message queue implementationsimilar to the ACE_Message_Queue is also shown in Chapter 10 of [SH02].

Example

The following example shows how to use the ACE_Message_Queue to im-plement a client logging daemon. As described in Section 1.5 on page 17,


i

i

i

i

i

i

i

i


a client logging daemon runs on every host participating in the networkedlogging service and performs the following tasks:

� It receives log records from client applications running on the samehost via some type of local IPC mechanism, such as shared memory,pipes, or loopback sockets.

� It uses a remote IPC mechanism, such as TCP/IP, to forward logrecords to a server logging daemon running on a designated host.

Our example uses two threads to implement a bounded buffer [BA90] usinga synchronized ACE_Message_Queue, i.e., one whose SYNCH_STRATEGYisinstantiated using the ACE_MT_SYNCHtraits class shown on page 128.

In our client logging daemon implementation, a receiver thread usesthe ACE Reactor framework to read log records from sockets connectedto client applications via the network loopback device. It queues each logrecord in a synchronized ACE_Message_Queue. The forwarder thread runsconcurrently, performing the following steps continuously:

1. Dequeueing messages from the message queue,2. Buffering the messages into larger chunks and then3. Forwarding the chunks to the server logging daemon over a TCP con-

nection.

The relationships between the receiver thread, the forwarder thread,and the ACE_Message_Queue that connects them are shown in Figure 5.4.By using a synchronized message queue, the receiver thread can continueto read requests from client applications as long as the message queueisn’t full. Overall server concurrency is therefore enhanced, even if theforwarder thread blocks on send operations when the connection to thelogging server is flow controlled.

We start our implementation by including the necessary ACE headerfiles.

#include "ace/OS.h"#include "ace/Message_Queue.h"#include "ace/Synch.h"#include "ace/Thread_Manager.h"#include "ace/SOCK_Connector.h"#include "ace/Reactor.h"#include "Logging_Acceptor.h"

Since the client logging daemon design uses different logic than thelogging server we can’t reuse our Reactor_Logging_Server class from the

‘‘hs _2001 /page

i

i

i

i

i

i

i

i


LOGGING

SERVER

NETWORKCLIENTTANGO

LOCAL IPC

P1

P3

P2

CCOONNSSOOLLEE

CLIENT LOGGING DAEMON

MessageQueueLogging

Handlers

LoggingAcceptor

TCPCONNECTION

CLIENT

APPLICATIONS

LoggingHandler

Figure 5.4: Multi-threaded Client Logging Daemon

Example portion of Section 3.5. Instead, we define the following Client_Logging_Daemon class:

class Client_Logging_Daemon : public ACE_Service_Object{public:

// Reactor hook methods.virtual int handle_input (ACE_HANDLE handle);virtual int handle_close (ACE_HANDLE = ACE_INVALID_HANDLE,

ACE_Reactor_Mask = 0);virtual ACE_HANDLE get_handle () const;

// Service Configurator hook methods.virtual int fini ();virtual int init (int argc, char *argv[]);virtual int info (ACE_TCHAR **bufferp, size_t length = 0) const;virtual int suspend ();virtual int resume ();

// Forward log records to the server logging daemon.virtual int forward ();

protected:// Entry point into forwarder thread of control.static void *run_svc (void *arg);


i

i

i

i

i

i

i

i


// A synchronized <ACE_Message_Queue> that queues messages.ACE_Message_Queue<ACE_MT_SYNCH> msg_queue_;

// Factory that passively connects <ACE_SOCK_Stream>s.ACE_SOCK_Acceptor acceptor_;

};

ACE_SVC_FACTORY_DECLARE (Client_Logging_Daemon);

The Client_Logging_Daemon class inherits from ACE_Service_Object .It can therefore be linked dynamically by the ACE Service Configuratorframework. It can also use the ACE Reactor framework to accept con-nections and wait for log records to arrive from any client applicationsconnected to the client logging daemon via loopback TCP sockets.

The methods are defined in the Client_Logging_Daemon.cpp file. Whena connection request or a log record arrives at the client logging daemon,the singleton reactor dispatches the following Client_Logging_Daemon::handle_input() hook method:

1 int Client_Logging_Daemon::handle_input (ACE_HANDLE handle) {2 if (handle == acceptor_.get_handle ()) {3 ACE_SOCK_Stream peer_handler;4 peer_acceptor_.accept (peer_handler);5 return ACE_Reactor::instance ()->register_handler6 (peer_handler.get_handle (),7 this,8 ACE_Event_Handler::READ_MASK);9 } else {

10 ACE_Message_Block *mblk;11 Logging_Handler logging_handler (handle);1213 if (logging_handler.recv_log_record (mblk) != -1)14 // Just enqueue the log record data, not the hostname.15 if (msg_queue_.enqueue_tail (mblk->cont ()) != -1)16 return 0;17 else mblk->release ();1819 return -1;20 }21 }

Lines 2–8 If the handle argument matches the ACE_SOCK_Acceptor fac-tory’s handle, we accept the connection and then use the three parametervariant of register_handler() to register the Client_Logging_Daemon

‘‘hs _2001 /page

i

i

i

i

i

i

i

i


object with the singleton reactor for READ events. This reactor method en-ables the client logging daemon to reuse a single C++ object for its acceptorfactory and all of its logging handlers.

Lines 10–19 If the handle argument is not the ACE_SOCK_Acceptor fac-tory handle then it must be a handle to a connected client applicationsocket. In this case, we read a log record out of the socket handle parame-ter, store the record into an ACE_Message_Block , and insert this messageinto the synchronized queue serviced by the forwarder thread.

If a client application disconnects or if a communication error occurs,the handle_input() hook method returns �1. This value triggers thereactor to call the following handle_close() hook method that cleans upall the Client_Logging_Daemon ’s resources:

int Client_Logging_Daemon::handle_close (ACE_HANDLE,ACE_Reactor_Mask) {

if (acceptor_.get_handle () != ACE_INVALID_HANDLE) {message_queue_.close ();acceptor_.close ();

// The close() method sets the handle to ACE_INVALID_HANDLE.}return 0;

}

We first check to see if the ACE_SOCK_Acceptor ’s handle has the value ofACE INVALID HANDLE since the handle_close() method can be called mul-tiple times. For example, it will be called when handle_input() returns�1, as well as when the following Client_Logging_Daemon::fini() hookmethod is invoked by the ACE Service Configurator framework:

int Client_Logging_Daemon::fini () {return handle_close ();

}

Note that we needn’t delete this object in handle_close() since the ACEService Configurator framework is responsible for deleting a service objectafter calling its fini() hook method.

We next show the Client_Logging_Daemon::forward() method, whichestablishes a connection with the server logging daemon and forwards logrecords to it, as shown below:

1 int Client_Logging_Daemon::forward () {2 // Connection establishment and data transfer objects.


i

i

i

i

i

i

i

i


3 ACE_SOCK_Stream logging_peer;4 ACE_SOCK_Connector connector;5 ACE_INET_Addr server_addr;67 server_addr.set ("ace_logger", LOGGING_SERVER_HOST);8 connector.connect (logging_peer, server_addr);9

10 int bufsiz = ACE_DEFAULT_MAX_SOCKET_BUFSIZ;11 logging_peer.set_option (SOL_SOCKET,12 SO_SNDBUF,13 (void *) &bufsiz,14 sizeof (bufsiz));1516 // Max # of <iov>s OS can send in a gather-write operation.17 iovec iov[ACE_IOV_MAX];18 int i = 0;1920 for (ACE_Message_Block *mblk = 0;21 msg_queue_->dequeue_head (mblk);22 ) {23 iov[i].iov_base = mblk->rd_ptr ();24 iov[i].iov_len = mblk->length ();2526 // Don’t delete the data in the message.27 mblk->set_flags (ACE_Message_Block::DONT_DELETE);28 mblk->release ();2930 if (i++ >= ACE_MAX_IOVLEN) {31 // Send all buffered log records in one operation.32 logging_peer_.sendv_n (iov, i);3334 // Clean up the buffers.35 for (--i; ; --i) {36 delete [] iov[i].iov_base;37 if (i == 0) break;38 }39 }40 }4142 if (i > 0) {43 // Send all remaining log records in one operation.44 logging_peer_.sendv_n (iov, i);4546 for (--i; i >= 0; --i) delete iov[i].iov_base;47 }48 return 0;49 }


i

i

i

i

i

i

i

i


Lines 3–8 We use the ACE Socket wrapper facades from Chapter 4 of[SH02] to establish a TCP connection with the server logging daemon.

Lines 10–14 We increase the socket send buffer to its largest size to max-imize throughput over high-speed networks.

Lines 17–48 We then run an event loop that dequeues a pointer to thenext ACE_Message_Block from the msg_queue_ and stores the log recorddata in a buffer of size ACE MAX IOVLEN. Whenever this buffer fills up, weuse the ACE_SOCK_Stream::sendv_n() call to transfer all log record datato the server logging daemon in one gather-write operation. Since an ACE_Message_Block is reference counted, its release() method deletes all theresources associated with mblk except the log record data, which will bedeleted after the entire buffer of ACE MAX IOVLEN log records is transmitted.

The forward() method runs in a separate thread of control that’sspawned in the Client_Logging_Daemon::init() hook method shownbelow:

1 int Client_Logging_Daemon::init (int argc, char *argv[]) {2 ACE_INET_Addr local_addr (ACE_CLIENT_LOGGING_DAEMON_PORT);34 // Start at argv[0].5 ACE_Get_Opt getopt (argc, argv, "p:", 0);67 for (int c; (c = getopt ()) != -1;)8 switch (c) {9 case ’p’:

10 local_addr.set ((u_short) ACE_OS::atoi (getopt.optarg));11 break;12 }1314 acceptor_.open (local_addr);15 ACE_Reactor::instance ()->register_handler16 (this, ACE_Event_Handler::ACCEPT_MASK);1718 ACE_thread_t thread_id;19 return ACE_Thread_Manager::instance ()->spawn20 (&Client_Logging_Daemon::run_svc,21 ACE_static_cast (void *, this),22 THR_SCOPE_SYSTEM,23 &thread_id);24 }

Line 2 Initialize the local_addr to the default TCP port number used bythe client logging daemon.

‘‘hs _2001 /page

i

i

i

i

i

i

i

i


Lines 5–12 Parse the service configuration options using the ACE_Get_Opt class described in Sidebar 12 on page 105. If the “-p ” option is passedinto init() then the local_addr port number is reset to that value.

Lines 14–16 Initialize the ACE_SOCK_Acceptor to listen at the local_addr port number and then register this object with the singleton reac-tor to accept new connections. The reactor will call back to the follow-ing Client_Logging_Daemon::get_handle() method to obtain the ac-ceptor’s socket handle:

ACE_HANDLE Client_Logging_Daemon::get_handle () const {return acceptor_.get_handle ();

}

When a connect request arrives, the reactor will dispatch the Client_Logging_Daemon::handle_input() method shown on page 133.

Lines 18–23 We finally use the ACE_Thread_Manager from Chapter 9of [SH02] to spawn a system-scoped thread that executes the Client_Logging_Daemon::run_svc() static method concurrently with the mainthread of control. The run_svc() static method casts its void* argumentto a Client_Logging_Daemon pointer and then delegates its processing toforward() method, as shown below:

void *Client_Logging_Daemon::run_svc (void *arg) {Client_Logging_Daemon *client_logging_daemon =

ACE_static_cast (Client_Logging_Daemon *, arg);

return client_logging_daemon->forward ();}

For completeness, the suspend() , resume() , and info() hook meth-ods are shown below:

int Client_Logging_Daemon::info (ACE_TCHAR **bufferp,size_t length = 0) const {

ACE_INET_Addr sa;acceptor_.get_local_addr (sa);

ACE_TCHAR buf[BUFSIZ];sprintf (buf,

"%d/%s %s",sa.get_port_number (),"tcp",

‘‘hs _2001 /page

i

i

i

i

i

i

i

i


"# client logging daemon\n");*bufferp = ACE_OS::strnew (buf);ACE_OS::strcpy (*bufferp, buf);return ACE_OS::strlen (buf);

}

int Client_Logging_Daemon::suspend () {return ACE_Reactor::instance ()->suspend_handler (this);

}

int Client_Logging_Daemon::resume () {return ACE_Reactor::instance ()->resume_handler (this);

}

Note that the ACE_Reactor ’s suspend_handler() and resume_handler()will call back to the get_handle() method to obtain the socket handle ofthe ACE_SOCK_Acceptor factory.

We can now place the ACE SVC FACTORY DEFINE macro into the imple-mentation file.

ACE_SVC_FACTORY_DEFINE (Client_Logging_Daemon);

This macro automatically defines the make_Client_Logging_Daemon()factory function, which is used in the following svc.conf file:

dynamic Client_Logging_DaemonService_Object *CLD:make_Client_Logging_Daemon() "-p $CLIENT_LOGGING_DAEMON_PORT"

This file configures the client logging daemon by dynamically linking theCLD DLL into the address space of the process and using ACE_DLL toextract the make_Client_Logging_Daemon() factory function from theCLD symbol table. This function is called to obtain a pointer to a dy-namically allocated Client_Logging_Daemon . The framework then callsClient_Logging_Daemon::init() hook method on this pointer, passingin the "$CLIENT_LOGGING_DAEMON_PORT"string as its argc /argv argu-ment. This string designates the port number where the client loggingdaemon listens for client application connection requests. If init() suc-ceeds, the Client_Logging_Daemon pointer is stored in the ACE_Service_Repository under the name "Client_Logging_Daemon" .

We’re now ready to show the main() function, which is identical to theone on page 116 in Section 4.4.


i

i

i

i

i

i

i

i

Section 5.3 The ACE Task Class 139





return 0;}

5.3 The ACE Task Class

Motivation

Although the ACE_Thread_Manager class provides a portable threading ab-straction, the threads it spawns and manages are not object-oriented, i.e.,they are C-style functions rather than objects. C-style functions make ithard to associate data members and methods with a thread. To resolvethese issues, ACE provides the ACE_Task class.

Class Capabilities

The ACE_Task class is the basis of ACE’s object-oriented concurrency frame-work. It provides the following capabilities:

� It uses an instance of ACE_Message_Queue from Section 5.2 to queuemessages that are passed between tasks.

� It can be used in conjunction with the ACE_Thread_Manager to be-come an active object [SSRB00] and process its queued messages inone or more threads of control.

� Since it inherits from ACE_Service_Object , instances of ACE_Taskcan be linked/unlinked dynamically via the ACE Service Configuratorframework from Chapter 4, which implements the Component Con-figurator pattern [SSRB00].

� Since ACE_Service_Object inherits from ACE_Event_Handler , in-stances of ACE_Task can serve as concrete event handlers via theACE Reactor framework from Chapter 3, which implements the Reac-tor pattern [SSRB00].


i

i

i

i

i

i

i

i

140 Section 5.3 The ACE Task Class

� It can be subclassed to create application-defined methods that queueand/or process messages.

Our focus in this section is on the ACE_Task capabilities for message pro-cessing. It obtains the event handling and dynamic linking/unlinking ca-pabilities by inheriting from the ACE_Service_Object class described inthe previous two chapters.

The interface for the ACE_Task class is shown in Figure 5.5 and its keymethods are shown in the following table:

Method Descriptionopen()close()

Hook methods that perform application-defined initialization andtermination activities.

put() A hook method that can be used to pass a message to a task,where it can be processed immediately or queued for subsequentprocessing in the svc() hook method.

svc() A hook method run by one or more threads to process messagesthat are placed on the queue via put() .

getq()putq()

Insert and remove messages from the task’s message queue (onlyvisible to subclasses of ACE_Task).

thr_mgr() Get and set a pointer to the task’s ACE_Thread_Manager .activate() Uses the ACE_Thread_Manager to convert the task into an active

object that runs the svc() method in one or more threads.

ACE_Task must be customized by subclasses to provide application-defined functionality by overriding its hook methods. For example, ACE_Task subclasses can override its open() and close() hook methods toperform application-defined ACE_Task initialization and termination activ-ities. These activities include spawning and canceling threads and allocat-ing and freeing resources, such as connection control blocks, I/O handles,and synchronization locks. ACE_Task subclasses can perform application-defined processing on messages by overriding its put() and svc() hookmethods to implement the following two message processing models:

1. Synchronous processing. The put() method is the entry point intoan ACE_Task, i.e., it’s used to pass messages to a task. To minimize over-head, pointers to ACE_Message_Block objects are passed between tasks toavoid copying their data. Task processing can be performed synchronouslyin the context of the put() method if it executes solely as a passive object,i.e., if its caller’s thread is borrowed for the duration of its processing.

‘‘hs _2001 /page

i

i

i

i

i

i

i

i


+ open (args : void *) : int+ close (flags : u_long) : int+ put (mb : ACE_Message_Block *, timeout : ACE_Time_Value *) : int+ svc () : int+ getq (mb : ACE_Message_Block *&, timeout : ACE_Time_Value *) : int+ putq (mb : ACE_Message_Block *, timeout : ACE_Time_Value *) : int+ activate (flags : long, threads : int) : int+ thr_mgr (): ACE_Thread_Manager *+ thr_mgr (mgr : ACE_Thread_Manager *)

+ thr_mgr_ : ACE_Thread_Manager *+ thr_count_ : size_t+ msg_queue_ : ACE_Message_Queue *

ACE_Task

Figure 5.5: The ACETask Class

2. Asynchronous processing. The svc() method can be overriddenby a subclass and used to perform application-defined processing asyn-chronously with respect to other activities in an application. Unlike put() ,the svc() method is not invoked by a client of a task directly. Instead,it’s invoked by one or more threads when a task becomes an active ob-ject, i.e., after its activate() method is called. This method uses theACE_Thread_Manager associated with an ACE_Task to spawn one or morethreads, as follows:

template <class SYNCH_STRATEGY> intACE_Task<SYNCH_STRATEGY>::activate (long flags,

int n_threads,/* Other params omitted */)

{// ...thr_mgr ()->spawn_n (n_threads,

&ACE_Task<SYNCH_STRATEGY>::svc_run,ACE_static_cast (void *, this),flags,/* Other params omitted */);

// ...


i

i

i

i

i

i

i

i


11.. AACCEE__TTaasskk::::aaccttiivvaattee (())22.. AACCEE__TThhrreeaadd__MMaannaaggeerr::::ssppaawwnn ((ssvvcc__rruunn,, tthhiiss));;33.. CCrreeaatteeTThhrreeaadd ((00,, 00,, ssvvcc__rruunn,, tthhiiss,, 00,, &&tthhrreeaadd__iidd));; RRUUNN--TTIIMMEE

TTHHRREEAADD SSTTAACCKK

44.. tteemmppllaattee <<SSYYNNCCHH__SSTTRRAATTEEGGYY>> vvooiidd ** AACCEE__TTaasskk<<SSYYNNCCHH__SSTTRRAATTEEGGYY>>::::ssvvcc__rruunn ((AACCEE__TTaasskk<<SSYYNNCCHH__SSTTRRAATTEEGGYY>> **tt)) {{ //// ...... vvooiidd **ssttaattuuss == tt-->>ssvvcc (());; //// ...... rreettuurrnn ssttaattuuss;; //// RReettuurrnn vvaalluuee ooff tthhrreeaadd.. }}

Figure 5.6: Task Activate Behavior

}

The ACE_Task::svc_run() method is a static method used as an adapterfunction. It runs in the newly spawned thread(s) of control, which pro-vide an execution context for the svc() hook method. Figure 5.6 illus-trates the steps associated with activating an ACE_Task using the Win32CreateThread() function to spawn the thread. Naturally, the ACE_Taskclass shields applications from any Win32-specific details.

When an ACE_Task subclass executes as an active object, its svc()method often runs an event loop that waits for messages to arrive on thetask’s ACE_Message_Queue. This queue can be used to buffer a sequenceof data messages and control messages for subsequent processing by atask’s svc() method. As messages arrive and are enqueued by a task’sput() method, the svc() method runs in separate thread(s) dequeueingthe messages and performing application-defined processing concurrently,as shown in Figure 5.7. Sidebar 18 compares and contrasts the ACE_Task

22:: ppuuttqq ((mmssgg))

11:: ppuutt ((mmssgg))

33:: ssvvcc (())44:: ggeettqq ((mmssgg))55:: ddoo__wwoorrkk((mmssgg))

66:: ppuutt ((mmssgg))

:: AACCEE__MMeessssaaggee__QQuueeuuee

tt11 :: SSuubbTTaasskk

:: TTaasskkSSttaattee







Figure 5.7: Passing Messages Between ACETask Objects

capabilities with the Java Runnable interface and Thread class.


i

i

i

i

i

i

i

i


Sidebar 18: ACE_Task vs. Java Runnable and Thread

If you’ve used Java’s Runnable interface and Thread class [Lea99], theACE_Task design should look familiar. The following are the similarities anddifferences between the two designs:

� ACE_Task::activate() is similar to the Java Thread::start() , i.e.,they both spawn internal threads. Java Thread::start() onlyspawns one thread, however, whereas activate() can spawn mul-tiple threads within the same ACE_Task, which makes it easy to im-plement thread pools, as shown in Section 5.3.

� ACE_Task::svc() is similar to the Java Runnable::run() method,i.e., both methods are hooks that run in newly spawned thread(s) ofcontrol.

� ACE_Task also contains a message queue, which allows applicationsto exchange and buffer messages. In contrast, this queueing capa-bility must be added by Java developers explicitly.

Example

This example shows how to combine the ACE_Task and ACE_Message_Queue classes with the ACE_Reactor from Chapter 3 and the ACE_Service_Config from Chapter ?? to implement a concurrent logging server. Thisserver design is based on the Half-Sync/Half-Async pattern [SSRB00] andthe eager spawning thread pool strategy described in Chapter 5 of [SH02].As shown in Figure 5.8, a pool of worker threads is pre-spawned when thelogging server is launched. Log records can be processed concurrently untilthe number of simultaneous client requests exceeds the number of workerthreads in the pool. At this point, additional requests are buffered in a syn-chronized ACE_Message_Queue until a worker thread becomes available.

The ACE_Message_Queue plays several roles in our thread pool loggingserver’s half-sync/half-async concurrency design:

� It decouples the main thread from the pool of worker threads—This design allows multiple worker threads to be active simultane-ously. It also offloads the responsibility for maintaining the queuefrom kernel-space to user-space.

� It helps to enforce flow control between clients and the server—When the number of bytes in the queue reaches its high-water mark,


i

i

i

i

i

i

i

i


NETWORK

TCPCONNECTIONS

P1

P3

P2

CCOONNSSOOLLEE

SERVER LOGGING DAEMON

MessageQueue

LoggingHandlers

LoggingAcceptor

CLIENT

LOGGING

DAEMONS

LOGGING

SERVER

worker threads

svc() svc() svc()

Figure 5.8: Architecture of the Thread Pool Logging Server

its flow control protocol blocks the main thread. As the underlyingTCP socket buffers fill up, this flow control propagates back to theserver’s clients, thereby preventing them from establishing new con-nections or sending log records. New connections and log records willnot be accepted until after the worker threads have a chance to catchup and unblock the main thread.

Pre-spawning and queueing help to amortize the cost of thread creationand bound the use of OS resources, which can improve server scalabilitysignificantly.

We start by including the necessary ACE header files and defining somehelper functions:

#include "ace/Synch.h"#include "ace/Task.h"#include "Reactor_Logging_Server.h"

The following table outlines the classes that we’ll cover in the examplebelow:


i

i

i

i

i

i

i

i


Class DescriptionTP_Logging_Task Runs as an active object processing and printing log

records.TP_Logging_Acceptor A factory that accepts connections and creates TP_

Logging_Handler objects.TP_Logging_Handler Target of upcalls from the ACE_Reactor that receives

log records from clients.TP_Logging_Server A facade class that integrates the other three classes

together.

TP Logging Task. This class inherits from ACE_Task and is configured touse a synchronized ACE_Message_Queue and a pool of worker threads thatall run the same svc() method.

class TP_Logging_Task :public ACE_Task<ACE_MT_SYNCH>

// Instantiated with an MT synchronization trait.{public:

enum { MAX_POOL_THREADS = 4 };

// ...Methods defined below...};

The TP_Logging_Task::open() hook method calls the ACE_Task::activate() method to convert this task into an active object.

virtual int open (void *) {return activate (THR_NEW_LWP | THR_DETACHED,

MAX_POOL_THREADS);}

When activate() returns, the TP_Logging_Task::svc() method will berunning in MAX POOL THREADS separate threads. We show the Logging_Task::svc() method implementation on page 147 after we describe theTP_Logging_Acceptor and TP_Logging_Handler classes.

The TP_Logging_Task::put() method inserts a message block con-taining a log record into the queue.

virtual int put (ACE_Message_Block *mblk, ACE_Time_Value *) {return putq (mblk);

}

The TP_Logging_Task::close() hook method closes the message queue,which will cause the threads in the pool to exit.

virtual close (u_int) { msg_queue ()->close (); }

‘‘hs _2001 /page

i

i

i

i

i

i

i

i


TP Logging Acceptor. This class inherits from the Logging_Acceptoron page 50 of Section 3.3 and overrides its handle_input() method tocreate instances of TP_Logging_Handler .

class TP_Logging_Acceptor : public Logging_Acceptor{public:

TP_Logging_Acceptor (ACE_Reactor *r = ACE_Reactor::instance ()): Logging_Acceptor (r) {}

virtual int handle_input (ACE_HANDLE) {TP_Logging_Handler *peer_handler;ACE_NEW_RETURN (peer_handler,

TP_Logging_Handler (reactor ()),-1);

if (peer_acceptor_.accept (peer_handler->peer ()) == -1) {delete peer_handler;return -1;

}else if (peer_handler->open () == -1)

peer_handler->close ();return 0;

}};

TP Logging Handler. This class inherits from Logging_Handler_Adapteron page 53 in Section 3.3 and is used to receive log records from a con-nected client.

class TP_Logging_Handler : public Logging_Handler_Adapter{public:

TP_Logging_Handler (ACE_Reactor *r): Logging_Handler_Adapter (r) {}

// Called when input events occur (e.g., connection or data).virtual int handle_input (ACE_HANDLE h);

};

ACE_SVC_FACTORY_DECLARE (TP_Logging_Handler);

The difference between Logging_Handler_Adapter::handle_input()and TP_Logging_Handler::handle_input() is that the latter doesn’t pro-cess a log record immediately after we receive it. Instead, it combines thelog record with a message block containing the client’s log file and insertsthe resulting composite message at the end of the task’s message queue,as shown below:


i

i

i

i

i

i

i

i


1 int TP_Logging_Handler::handle_input (ACE_HANDLE) {2 ACE_Message_Block *mblk;3 if (logging_handler_.recv_log_record (mblk)) != -1) {4 ACE_Message_Block *log_blk =5 new ACE_Message_Block6 (ACE_reinterpret_cast (char *, &log_file_));9 log_blk->cont (mblk);

10 logging_task_.put (log_blk);11 return 0;12 } else13 return -1;14 }

Lines 2–3 First read a log record from a socket into an ACE_Message_Block .

Lines 4–9 Create another ACE_Message_Block called log_blk that con-tains a pointer to the log_file . Attach the mblk to log_blk ’s continuationchain to form a composite message.

Line 10 Insert the composite message into the TP_Logging_Task ’s mes-sage queue.

Now that we’ve shown the TP_Logging_Handler class, we can showthe TP_Logging_Task::svc() method, which runs concurrently in eachof the worker threads. This method runs its own event loop that blockson the synchronized message queue. After a message is enqueued by theTP_Logging_Handler::handle_input() method, it’ll be dequeued by anavailable worker thread, demarshaled, and written to the appropriate logfile corresponding to their client.

1 int TP_Logging_Task::svc () {2 for (ACE_Message_Block *log_blk; getq (log_blk) != -1; ) {3 ACE_FILE_IO *log_file =4 ACE_reinterpret_cast (ACE_FILE_IO *, log_blk->rd_ptr ());56 Logging_Handler logging_handler (log_file);7 logging_handler.write_log_record (log_blk->cont ());89 log_blk->cont (0);

10 log_blk->release ();11 }12 return 0;13 }

‘‘hs _2001 /page

i

i

i

i

i

i

i

i


Lines 2–4 Call the getq() method, which blocks until a message block isavailable. Each message block is actually a composite message that con-tains the following three message blocks chained together via their contin-uation pointers:

1. The ACE_FILE_IO object that’s used to write the log record2. The marshaled log record contents and3. The hostname of the connected client

Lines 6–7 Initialize a Logging_Handler with the log_file and then callits write_log_record() method, which writes the log record to the log file.The write_log_record() method is responsible for releasing its messageblocks, as shown on in Chapter 4 of [SH02].

Lines 9–10 After the log record is written, we release the log_blk mes-sage block to reclaim the message block allocated to store the ACE_FILE_IOpointer. We first set the continuation field to NULL so we just release themblk . Note that we don’t delete the ACE_FILE_IO memory since it’s “bor-rowed” from the TP_Logging_Handler rather than allocated dynamically.

TP Logging Server. This facade class contains an instance of TP_Logging_Task and Reactor_Logging_Server .

class TP_Logging_Server : public ACE_Service_Object{protected:

// Contains the reactor, acceptor, and handlers.typedef Reactor_Logging_Server<TP_Logging_Acceptor>

LOGGING_DISPATCHER;LOGGING_DISPATCHER *logging_dispatcher_;

// Contains the pool of worker threads.TP_Logging_Task *logging_task_;

public:// Other methods defined below...

};

The TP_Logging_Server::init() hook method enhances the reactor-based logging server implementation in Chapter 3 by pre-spawning a poolof worker threads that process log records concurrently.

virtual init (int argc, char *argv[]) {logging_dispatcher_ =

new TP_Logging_Server::LOGGING_DISPATCHER


i

i

i

i

i

i

i

i


(argc, argv, ACE_Reactor::instance ();logging_task_ = new TP_Logging_;return logging_task->open ();

}

The TP_Logging_Server::fini() method is shown next:

1 virtual int fini () {2 logging_task_->close ();34 logging_task_->thr_mgr ()->wait ();56 delete logging_dispatcher_;7 delete logging_task_;8 return 0;9 }

Line 2 Close the TP_Logging_Task , which signals the worker threads inthe pool to exit.

Lines 4–8 Use the ACE_Thread_Manager ’s barrier synchronization fea-ture to wait for the pool of threads to exit and then delete the dynamicallyallocated data members and return from fini() .

For brevity, we omit the suspend() , resume() , and info() hook meth-ods, which are similar to those shown in earlier examples.

We place the following ACE SVC FACTORY DEFINE macro into the TPLS.cpp implementation file.

ACE_SVC_FACTORY_DEFINE (TP_Logging_Server);

This macro automatically defines the make_TP_Logging_Server() factoryfunction that’s used in the following svc.conf file:

dynamic TP_Logging_Server Service_Object *TPLS:make_TP_Logging_Server() "-p $TP_LOGGING_SERVER_PORT"

This file configures the thread pool logging server by dynamically link-ing the TPLS DLL into the address space of the process and using ACE_DLL to extract the make_TP_Logging_Server() factory function from theTPLS symbol table. This function is called to obtain a pointer to a dy-namically allocated TP_Logging_Server . The framework then calls theTP_Logging_Server::init() hook method on this pointer, passing inthe "$TP_LOGGING_SERVER_PORT"string as its argc /argv argument. Thisstring designates the port number where the logging server listens for clientconnection requests. If init() succeeds, the TP_Logging_Server pointer


i

i

i

i

i

i

i

i


is stored in the ACE_Service_Repository under the name "TP_Logging_Server" .

We’re now ready to show the main() function, which is identical to theones we’ve shown earlier in this chapter and in Chapter 4.





return 0;}

5.4 Summary

The ACE Task framework allows developers to create and configure concur-rent networked applications in a powerful and extensible object-orientedfashion. This framework provides the ACE_Task class that integrates mul-tithreading with object-oriented programming and queueing. The queue-ing mechanism in the ACE_Task is based on the ACE_Message_Queue classthat transfers messages between tasks efficiently. Since ACE_Task derivesfrom the ACE_Service_Object class in Section 4.2, it’s easy to design ser-vices that can run as active objects and be dispatched by the ACE Reactorframework.

This chapter illustrates how the ACE Reactor framework can be com-bined with the ACE Task framework to implement variants of the Half-Sync/Half-Async pattern [SSRB00]. The ACE Task framework classes canalso be combined with the ACE_Future , ACE_Method_Request , and ACE_Activation_List classes to implement the Active Object pattern [SSRB00],as shown in the supplemental material at the ACE website http://ace.ece.uci.edu .


i

i

i

i

i

i

i

i

Bibliography

[Ann98] Anne Thomas, Patricia Seybold Group. Enterprise JavaBeansTechnology. java.sun.com/products/ejb/white paper.html, December1998. Prepared for Sun Microsystems, Inc.

[Aus98] Matt Austern. Generic Programming and the STL: Using and Extendingthe C++ Standard. Addison-Wesley, 1998.

[BA90] M. Ben-Ari. Principles of Concurrent and Distributed Programming.Prentice Hall International Series in Computer Science, 1990.

[BEA99] BEA Systems, et al. CORBA Component Model Joint RevisedSubmission. Object Management Group, OMG Documentorbos/99-07-01 edition, July 1999.

[Bja00] Bjarne Stroustrup. The C++ Programming Language, 3rd Edition.Addison-Wesley, 2000.

[BL88] Ronald E. Barkley and T. Paul Lee. A Heap-Based CalloutImplementation to Meet Real-Time Needs. In Proceedings of the USENIXSummer Conference, pages 213–222. USENIX Association, June 1988.

[BMR+96] Frank Buschmann, Regine Meunier, Hans Rohnert, Peter Sommerlad,and Michael Stal. Pattern-Oriented Software Architecture – A System ofPatterns. Wiley and Sons, 1996.

[CB97] John Crawford and Steve Ball. Monostate Classes: The Power of One.C++ Report, 9(5), May 1997.

[CL97] Patrick Chan and Rosanna Lee. The Java Class Libraries: java.applet,java.awt, java.beans, Volume 2. Addison-Wesley, Reading,Massachusetts, 1997.

[Com84] Douglas E. Comer. Operating System Design: The Xinu Approach.Prentice Hall, Englewood Cliffs, NJ, 1984.

151


i

i

i

i

i

i

i

i

152 BIBLIOGRAPHY

[FBB+99] Martin Fowler, Kent Beck, John Brant, William Opdyke, and DonRoberts. Refactoring - Improving the Design of Existing Code.Addison-Wesley, Reading, Massachusetts, 1999.

[FJS99a] Mohamed Fayad, Ralph Johnson, and Douglas C. Schmidt, editors.Object-Oriented Application Frameworks: Problems & Perspectives.Wiley & Sons, New York, NY, 1999.

[FJS99b] Mohamed Fayad, Ralph Johnson, and Douglas C. Schmidt, editors.Object-Oriented Application Frameworks: Applications & Experiences.Wiley & Sons, New York, NY, 1999.

[FY99] Brian Foote and Joe Yoder. Big Ball of Mud. In Brian Foote, NeilHarrison, and Hans Rohnert, editors, Pattern Languages of ProgramDesign. Addison-Wesley, Reading, Massachusetts, 1999.

[GHJV95] Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides.Design Patterns: Elements of Reusable Object-Oriented Software.Addison-Wesley, Reading, Massachusetts, 1995.

[HJE95] Herman Hueni, Ralph Johnson, and Robert Engel. A Framework forNetwork Protocol Software. In Proceedings of OOPSLA ’95, Austin,Texas, October 1995. ACM.

[HLS97] Timothy H. Harrison, David L. Levine, and Douglas C. Schmidt. TheDesign and Performance of a Real-time CORBA Event Service. InProceedings of OOPSLA ’97, pages 184–199, Atlanta, GA, October 1997.ACM.

[HP91] Norman C. Hutchinson and Larry L. Peterson. The x-kernel: AnArchitecture for Implementing Network Protocols. IEEE Transactions onSoftware Engineering, 17(1):64–76, January 1991.

[HV99] Michi Henning and Steve Vinoski. Advanced CORBA Programming WithC++. Addison-Wesley, Reading, Massachusetts, 1999.

[JF88] R. Johnson and B. Foote. Designing Reusable Classes. Journal ofObject-Oriented Programming, 1(5):22–35, June/July 1988.

[JKN+01] Philippe Joubert, Robert King, Richard Neves, Mark Russinovich, andJohn Tracey. High-Performance Memory-Based Web Servers: Kerneland User-Space Performance. In Proceedings of the USENIX TechnicalConference, Boston, MA, June 2001.

[Joh97] Ralph Johnson. Frameworks = Patterns + Components.Communications of the ACM, 40(10), October 1997.

[Jos99] Nicolai Josuttis. The C++ Standard Library: A Tutorial and Reference.Addison-Wesley, Reading, Massachusetts, 1999.


i

i

i

i

i

i

i

i

BIBLIOGRAPHY 153

[KMC+00] Eddie Kohler, Robert Morris, Benjie Chen, John Jannotti, andM. Frans Kaashoek. The Click Modular Router. ACM Transactions onComputer Systems, 18(3):263–297, August 2000.

[Koe92] Andrew Koenig. When Not to Use Virtual Functions. C++ Journal, 2(2),1992.

[Kof93] Thomas Kofler. Robust iterators for ET++. Structured Programming,14(2):62–85, 1993.

[Lea99] Doug Lea. Concurrent Java: Design Principles and Patterns, SecondEdition. Addison-Wesley, Reading, Massachusetts, 1999.

[Mey97] Bertrand Meyer. Object-Oriented Software Construction, Second Edition.Prentice Hall, Englewood Cliffs, NJ, 1997.

[Obj98] Object Management Group. CORBAServices: Common Object ServicesSpecification, Updated Edition, 95-3-31 edition, December 1998.

[Obj01] Object Management Group. The Common Object Request Broker:Architecture and Specification, 2.5 edition, September 2001.

[OOS01] Ossama Othman, Carlos O’Ryan, and Douglas C. Schmidt. An EfficientAdaptive Load Balancing Service for CORBA. IEEE Distributed SystemsOnline, 2(3), March 2001.

[OSI92a] OSI Special Interest Group. Data Link Provider Interface Specification,December 1992.

[OSI92b] OSI Special Interest Group. Transport Provider Interface Specification,December 1992.

[Pre94] Wolfgang Pree. Design Patterns for Object-Oriented SoftwareDevelopment. Addison-Wesley, Reading, Massachusetts, 1994.

[Rag93] Steve Rago. UNIX System V Network Programming. Addison-Wesley,Reading, Massachusetts, 1993.

[Ric97] Jeffrey Richter. Advanced Windows, Third Edition. Microsoft Press,Redmond, WA, 1997.

[Rit84] Dennis Ritchie. A Stream Input–Output System. AT&T Bell LabsTechnical Journal, 63(8):311–324, October 1984.

[Rob99] Robert Sedgwick. Algorithms in C++, Third Edition. Addison-Wesley,1999.

[Sch98] Douglas C. Schmidt. Evaluating Architectures for Multi-threadedCORBA Object Request Brokers. Communications of the ACM specialissue on CORBA, 41(10), October 1998.


i

i

i

i

i

i

i

i

154 BIBLIOGRAPHY

[Sch00a] Douglas C. Schmidt. Applying a Pattern Language to DevelopApplication-level Gateways. In Linda Rising, editor, Design Patterns inCommunications. Cambridge University Press, 2000.

[Sch00b] Douglas C. Schmidt. Applying Design Patterns to Flexibly ConfigureNetwork Services in Distributed Systems. In Linda Rising, editor,Design Patterns in Communications. Cambridge University Press, 2000.

[Sch00c] Douglas C. Schmidt. Why Software Reuse has Failed and How to Makeit Work for You. C++ Report, 12(1), January 2000.

[SH02] Douglas C. Schmidt and Stephen D. Huston. C++ NetworkProgramming: Resolving Complexity Using ACE and Patterns.Addison-Wesley, Reading, Massachusetts, 2002.

[SLM98] Douglas C. Schmidt, David L. Levine, and Sumedh Mungee. TheDesign and Performance of Real-Time Object Request Brokers.Computer Communications, 21(4):294–324, April 1998.

[Sol98] David A. Solomon. Inside Windows NT, 2nd Ed. Microsoft Press,Redmond, Washington, 2nd edition, 1998.

[Som97] Peter Sommerland. The Manager Design Pattern. In Robert Martin,Frank Buschmann, and Dirk Riehle, editors, Pattern Languages ofProgram Design. Addison-Wesley, Reading, Massachusetts, 1997.

[SS93] Douglas C. Schmidt and Tatsuya Suda. Transport System ArchitectureServices for High-Performance Communications Systems. IEEE Journalon Selected Areas in Communication, 11(4):489–506, May 1993.

[SS94] Douglas C. Schmidt and Tatsuya Suda. An Object-OrientedFramework for Dynamically Configuring Extensible DistributedCommunication Systems. IEE/BCS Distributed Systems EngineeringJournal (Special Issue on Configurable Distributed Systems), 2:280–293,December 1994.

[SS95a] Douglas C. Schmidt and Paul Stephenson. Experiences Using DesignPatterns to Evolve System Software Across Diverse OS Platforms. InProceedings of the 9

th European Conference on Object-OrientedProgramming, Aarhus, Denmark, August 1995. ACM.

[SS95b] Douglas C. Schmidt and Tatsuya Suda. Measuring the Performance ofParallel Message-based Process Architectures. In Proceedings of theConference on Computer Communications (INFOCOM), pages 624–633,Boston, MA, April 1995. IEEE.

[SSRB00] Douglas C. Schmidt, Michael Stal, Hans Rohnert, and FrankBuschmann. Pattern-Oriented Software Architecture: Patterns for


i

i

i

i

i

i

i

i

BIBLIOGRAPHY 155

Concurrent and Networked Objects, Volume 2. Wiley & Sons, New York,NY, 2000.

[Sta96] Stan Lippman. Inside the C++ Object Model. Addison-Wesley, 1996.

[Ste98] W. Richard Stevens. UNIX Network Programming, Volume 1: NetworkingAPIs: Sockets and XTI, Second Edition. Prentice Hall, Englewood Cliffs,NJ, 1998.

[Ste99] W. Richard Stevens. UNIX Network Programming, Volume 2:Interprocess Communications, Second Edition. Prentice Hall, EnglewoodCliffs, NJ, 1999.

[SW93] W. Richard Stevens and Gary Wright. TCP/IP Illustrated, Volume 2.Addison Wesley, Reading, Massachusetts, 1993.

[VL97] George Varghese and Tony Lauck. Hashed and Hierarchical TimingWheels: Data Structures for the Efficient Implementation of a TimerFacility. IEEE Transactions on Networking, December 1997.

[Vli98] John Vlissides. Pattern Hatching: Design Patterns Applied.Addison-Wesley, Reading, Massachusetts, 1998.

[WLS+85] D. Walsh, B. Lyon, G. Sager, J. M. Chang, D. Goldberg, S. Kleiman,T. Lyon, R. Sandberg, and P. Weiss. Overview of the SUN Network FileSystem. In Proceedings of the Winter USENIX Conference, Dallas, TX,January 1985.

[Woo97] Bobby Woolf. The Null Object Pattern. In Robert Martin, FrankBuschmann, and Dirk Riehle, editors, Pattern Languages of ProgramDesign. Addison-Wesley, Reading, Massachusetts, 1997.


i

i

i

i

i

i

i

i


i

i

i

i

i

i

i

i

Back Cover Copy

Software for networked applications must possess the following all-too-rare qualities to be successful in today’s competitive, fast-paced computingindustry:

� Efficiency, to provide low latency to delay-sensitive applications, highperformance to bandwidth-intensive applications, and predictabilityto real-time applications

� Flexibility, to support a growing range of multimedia datatypes, trafficpatterns, and end-to-end quality of service (QoS) requirements with-out rewriting applications from scratch and

� Extensibility, to support successions of quick updates and additionsto take advantage of new requirements and emerging markets.

This book describes how frameworks can help developers navigate betweenthe limitations of

1. Lower-level native operating system APIs, which are inflexible andnon-portable and

2. Higher-level distributed computing middleware, which often lacks theefficiency and flexibility for networked applications with stringent QoSand portability requirements.

This book illustrates how to develop networked applications using ACEframeworks, and shows how key patterns and design principles can beused to develop and deploy successful object-oriented networked applica-tion software.

If you’re designing software and systems that must be portable, flexible,extensible, predictable, reliable, and affordable, this book and the ACE

157


i

i

i

i

i

i

i

i

158 BIBLIOGRAPHY

frameworks will enable you to be more effective in all of these areas. You’llget first hand knowledge from Doug Schmidt, the original architect anddeveloper of ACE, the reasons and thought processes behind the patternsand ideas that went into key aspects of the ACE frameworks. Co-authorSteve Huston’s expertise and knowledge from his extensive background inhardware and software environments and hands-on involvement in ACEsince 1996 will take you further in understanding how to effectively tapthe rich features embodied in the ACE framework.

Dr. Douglas C. Schmidt is an Associate Professor at the Universityof California, Irvine, where he studies patterns and optimizations for dis-tributed real-time and embedded middleware. Doug led the developmentof ACE and TAO, which are widely-used open-source middleware contain-ing a rich set of reusable frameworks that implement key concurrency andnetworking patterns. He’s also been the editor-in-chief of the C++ Reportmagazine and co-authors the Object Interconnections column for the C/C++Users Journal.

Stephen D. Huston is President and CEO of Riverace Corporation, thepremier provider of technical support and consulting services for compa-nies looking to work smarter and keep software projects on track usingACE. Mr. Huston has more than twenty years of software development ex-perience, focusing primarily on network protocol and networked applica-tion development in a wide range of hardware and software environments.He has been involved with ACE’s development since 1996 and has enjoyedhelping many companies develop clean and efficient networked applica-tions quickly using ACE.

c++ Ntw Programming

Documents

Transcript of c++ Ntw Programming