Galaxy Communicator 3.3 Documentationcommunicator.sourceforge.net/sites/MITRE/... · Summary . . ....

Galaxy Communicator 3.3 Documentation

The DARPA Communicator Teamat

The MITRE Corporation202 Burlington Rd.Bedford, MA 01730

http://fofoca.mitre.org

Table of Contents

.................. 1Toplevel Index

.................. 2Getting help

................... 2Tutorial

................. 3Advanced topics

................... 3Reference

.................. 4Administration

............... 6Introduction and Overview

................... 6History

.................. 6What you get

................. 7What you don’t get

... 8Prerequisites: technical background, hardware and software requirements

................ 8Technical overview

.................... 12Tutorial

.................. 13Getting Started

............... 13Reference environment

................ 13Start up X windows

.............. 14Set up your shell environment

................. 14Modify your path

....... 16How the Hub and Server Communicate, and What They Say

................... 16Frames

................... 17Messages

.......... 20Starting up a Galaxy Communicator system

........... 23Running a Galaxy Communicator system

................... 25Summary

.............. 27How to Use the Process Monitor

.............. 27Starting the process monitor

.................. 28The menu bar

................ 29The process panes

................ 30Using the menu bar

.............. 31A Sample End-to-End System

.................. 31The servers

.............. 32Setting up the toy travel demo

............... 37Step 1: Audio available

........... 39Step 2: Reroute to handle general input

............... 41Step 3: Send to dialogue

............. 43Step 4: Dialogue consults backend

......... 44Step 5: Dialogue reply to generation and synthesis

i

.................. 45Step 6: Audio output

.................... 47Summary

............... 48Introducing Frames and Objects

.................. 48What’s in a frame?

................ 50Making and freeing frames

.............. 51Making, using and freeing objects

.................... 54Summary

.................... 55Server Basics

............... 55How complex can a server get?

................ 56Six steps to building a server

.................... 62Summary

................. 63How to Use the Unit Tester

............... 63Starting the unit tester as a Hub

.............. 66Interacting with the Parser server

.............. 67Starting the unit tester as a server

............ 68Interacting with the Parser server via the Hub

.................... 71Summary

............... 72Writing a Basic Hub Program File

............. 73A little bit about Hub program file syntax

............ 74Global information and server declarations

................. 77Hub program directives

................. 80Rules and namespaces

................... 84Rule conditions

................. 86Managing flow of control

.................... 88Summary

.................... 90Error Handling

.............. 90Environments and message returns

................. 91How to raise an error

................. 94How to catch an error

.................... 99Summary

............... 101Sending New Messages to the Hub

.............. 101An overview of the Dialogue server

................ 102Asynchronous messages

................. 103Synchronous messages

.................. 112Putting it all together

.................... 115Summary

............. 116Setting Up a Brokered Audio Connection

.............. 117Recognizer, Synthesizer and Audio

................. 117Brokering in four steps

.................. 121Brokering details

.............. 127Why the dependency on :call_id ?

.................... 128Summary

.................. 129Creating a UI Server

............... 129The Communicator object model

................ 130Event-driven programming

.................. 132The event monitor

.................. 134Adding UI elements

ii

.................... 135Summary

............ 137Setting Up the Hub to Log Your Interactions

.................. 137Introducing logging

.................. 138Setting up the log

............... 139Inserting information into the log

.................. 141Looking at the log

................ 142The "real" toy travel demo

.................. 143Working with logs

.................... 144Summary

..................... 145Sessions

.................. 145What sessions are

.................. 146Why use sessions?

................. 147How sessions propagate

.................. 149Creating sessions

................... 150Ending sessions

.................... 150Summary

............... 152Creating an End-to-End System

................. 152Selecting your servers

.................. 153Contacting the Hub

........... 154Guidelines for writing servers and Hub programs

.................. 154The Builtin server

................. 156Debugging strategies

................... 157Avoiding trouble

.................... 159Summary

................... 161Advanced topics

................... 162Server Locations

................. 162The server locations file

............... 164Using the Hub in Scriptless Mode

................. 164Omitting Hub programs

..................... 165Logging

.............. 166Writing your own scripting module

.................... 167Hub Listeners

.................. 167Setting up the Hub

................. 168Setting up the server

................. 169The handshake protocol

.................... 169API support

.................. 170The Timed Task Loop

............... 170What the timed task loop does

................. 170Setting up timed tasks

................. 173Managing timed tasks

.................. 173Internal functions

..................... 174Threads

................ 175For backward compatibility

......... 176Managing Session Information and Using Continuations

.................. 176What is a session?

............... 176Starting and ending sessions

................ 177Keeping information straight

iii

................. 178Administrative functions

................... 178Continuations

.................. 181Hub continuations

....... 183Controlling Hub Server and Session Properties from the Server

.................. 183Server properties

.................. 184Session properties

................... 186Special Main Loops

................. 186External loop records

................. 190ELR support functions

................... 193The ugly details

............. 203Using main loops with listener-in-Hub

................. 206Using non-MIT Makefiles

.................... 208Signal Handling

..................... 209Binary Data

................. 209Passing arbitrary data

......... 212Command Line Argument Parsing and Printing Utilities

............... 212Command line argument parsing

................... 212Program Model

.................... 213Overview

.................. 213Variable Arguments

..................... 214Help

..................... 215Printing

....... 217How to Use the Communicator Library in Arbitrary Executables

..................... 218Reference

................ 219Frequently Asked Questions

.................... 219Installation

................. 220Server and Hub setup

.................... 220Brokering

.................. 220MIT Hub scripting

............... 222The Hub and Server Executables

.................... 222The Hub

.................... 223The servers

................. 224Running the executables

............ 225Frames, Property Lists, and Typed Objects

.................. 226Frames and Objects

.................. 229Frame Functions

................... 231Property Lists

................. 234Typed Object Functions

................... 241Frame Utilities

................ 245MIT Linguistic Frame Tools

................. 247Deprecated functions

............ 249Building a Communicator-Compliant Server

....... 249Steps to Building and Using a Communicator-Compliant Server

............... 249Implement the Server Functions

.............. 259Modify the Makefile Stub Template

................ 260Declare server information

................. 266Create a Hub Program

iv

....... 267Conditionalize your server for multiple Communicator versions

................ 268Communicator Hub Programs

............... 268The Hub Program File Structure

..................... 269Values

..................... 270Subfiles

.................. 270Expression syntax

.................. 277Global Declarations

.................... 279Initial Token

.................. 280Logging Parameters

.................. 282Server Declarations

................... 287Hub Programs

................... 288Program Name

..................... 288Rules

.................... 289Parameters

............ 297Alphabetical index of program file elements

...................... 299Logging

............... 299What Can Be Logged and Why

........... 300Adding Logfile Instructions to the Program File

................... 304Audio Logging

.............. 304When and How Data Get Logged

.............. 304Where Logfiles and Data are Stored

................. 306Brokering and Audio Data

.................. 306Sending broker data

................. 309Streaming broker data

................. 311Receiving broker data

................. 314Backward compatibility

................... 316Server Architecture

................. 316Simulating a main loop

.................. 317Argument packages

............... 319Manipulating the server object

.............. 321Manipulating the connection object

................ 323Event-driven programming

................. 326Listener-in-Hub support

................. 328Deprecated functions

.................... 330Reserved Keys

................... 330Hub/server keys

.................... 333Hub keys

..................... 335API Index

.................... 367Core Library

................. 367String Utility Functions

................. 368Print Utility Functions

............... 368Program File Parsing Functions

.................. 372Stream Functions

................... 373Buffer Functions

................ 374Frame Reading Functions

.................. 375Hash Table Utilities

................ 376Dispatch Function Utilities

v

................... 377Random utilities

................... 378Galaxy I/O Library

.................. 378Brokering functions

................ 379Comm structure functions

................ 380Server structure functions

................. 382Flow of control functions

................. 383Socket queue functions

................... 384Server Stub Library

.................... 386Utilities Library

..................... 386Sockets

............... 387Command line argument parsing

..................... 388Threads

................... 389Miscellaneous

..................... 390Glossary

................... 394The Double Server

.................. 396Audio and Brokering

..................... 398Main Loops

..................... 400Binary Data

........... 402Using non-MIT Makefiles with the Double Server

.............. 403A Multiple-Connection Double Server

................... 404Turn Management

........... 404Using Turn Management for Dialogue Control

.......... 404Implement the Server Dialogue Control Functions

............... 406Create a Dialogue Control File

................ 406Command Line Arguments

................. 406Dialogue Control Script

................... 407The Header File

................ 409Initialize Turn Management

............... 409Turn Management Message Set

................... 412Dialogue Control

.................. 412Dialogue Functions

.................. 413Domain Functions

................... 415Control file tools

.................. 416The Program Verifier

..................... 416Usage

................... 417The Builtin Server

..................... 417Usage

................... 417Message Set

................. 430Installing and Accessing

................. 430Compiling the utilities

............... 430Compiling against the utilities

............... 431Using the utilities in your code

.................... 432Frame Tools

..................... 433Binary Data

................. 433Defining the data type

.............. 434Storing and accessing the data type

vi

..................... 437Brokering

................. 437Defining type handlers

............... 438Adding outgoing broker support

............... 438Adding incoming broker support

..................... 439Stdin Poll

............... 439Creating and activating a poll

.............. 440Doing without the timed task loop

..................... 441API Index

.................... 443MITRE Tools

................... 443Server scripts

................. 445Process Monitor Reference

................... 445Command line

.................. 445Command line flags

................ 447Command line flag example

.................. 447Configuration file

................ 449Configuration file example

.................. 451Unit Tester Reference

.................... 451Unit tester

................... 452MITRE Log Tools

.............. 452Background and common arguments

.................. 457XML log generation

................... 458XML annotation

............. 465XML human annotation stub generation

............. 468Unifying automatic and human XML files

.................. 468XML summarization

.................. 473XML log validation

.................... 476XML scoring

............. 480Batch generation of NIST sclite input

................. 483Extraction of landmarks

............... 487MITRE Log Reference Documents

.................. 488Toy Travel "System"

................. 488The configuration file

.................... 491The servers

................... 492The program file

.................. 497Running the demo

.................. 498Examining the logs

.................. 500MITRE Java Bindings

................. 500Enabling Java Bindings

................ 500Documentation Summary

.................. 501Creating a Server

................ 502Command Line Arguments

................... 504Continuations

.......... 505Calling Dispatch Functions From Incoming Brokers

............... 506Sending Messages to the Hub

............... 506Contacting a Hub at Runtime

............... 507Server and Session Properties

.................. 508Setting Session Id

vii

.................. 508Postponing a Reply

.................... 508Change Log

................. 510MITRE Python Bindings

................ 510Enabling Python bindings


..................... 511Details

................... 516Equivalences

.............. 534MITRE Allegro Common Lisp Bindings

................ 534Enabling Allegro bindings


..................... 535Details

.................... 540Known Bugs

................... 540Equivalences

.................... 559Administration

.............. 560Installing the DARPA Communicator

................... 560Preliminaries

.................. 562Site configuration

.................... 568Compilation

.................... 569Installation

.................. 570Cross compilation

.................... 572New Features

.................... 572Version 3.3

.................... 573Version 3.2

.................... 573Version 3.1

.................... 574Version 3.0

................. 577Upgrading from 3.2 to 3.3

............ 577Step 1: Updating cross-compilation support

........... 577Step 2: Upgrading to the Open Source Toolkit

........ 578Step 3: Upgrading flags which set Java Hub client behavior


.......... 579Step 1: Updating session management in the Hub

......... 580Step 2: Upgrading flags which set Hub client behavior


............... 581Step 1: Configuring your system

........... 581Step 2: Updating external mainloop references

........... 582Step 3: Upgrading GAL_VERBOSE references

............... 582Step 4: Updating Java bindings



............. 585Step 2a: Updating outgoing brokering

........ 587Step 2b: Upgrading memory management in broker callbacks

........... 588Step 2c: Upgrading incoming brokering (optional)

........ 591Step 2d: Using broker callbacks for completion and aborting

.............. 596Step 3: Upgrading Python bindings

............... 599Step 4: Upgrading Java bindings

............. 606Step 5: Upgrading server listen status

.......... 607Step 6: Upgrading references to the call environment

viii

........... 608Step 7: Upgrading calls to GalIO_SetServerData

..... 609Step 8a: Upgrading references to the Communicator main loop (optional)

........ 610Step 8b: Upgrade your own main loop (advanced, optional)

........ 620Step 9: Upgrade timed task invocation (advanced, optional)

....... 622Step 10: Upgrading session management (advanced, optional)



........... 627Step 2: Check your use of none! and destroy!

....... 627Step 3: Check your use of SET: and DEL: with dispatch_to_main

........ 628Step 4: Upgrade your own main loop (advanced, optional)

.......... 632Step 5: Upgrade timed task invocation (advanced)

........... 633Step 6: Upgrade your local Makefile (advanced)

............... 635Upgrading from 1.2 or 1.3 to 2.0

................. 635Step 0: Updating to 1.3


............... 636Step 2: Updating your headers

...... 636Step 3: Updating your command line argument parsing and printing

.............. 637Step 4: Updating your header files

........ 639Step 5: Upgrading server access inside a dispatch function

....... 641Step 6: Upgrading server access inside the reinitialize message

......... 642Step 7: Upgrading server access from broker handlers

......... 645Step 8 (optional): Upgrading outgoing brokering creation

............ 646Step 9 (advanced): Enabling signal handling

.......... 646Step 10 (advanced): Updating your own main loop

......... 652Step 11 (advanced): updating your use of stdin polling

.................... 655Release Notes

................ 655January 2002: Version 3.3

................ 657October 2001: Version 3.2

................. 659June 2001: Version 3.1

................ 662January 2001: Version 3.0

................ 675Release Notes for Version 2.x

................ 675August 2000: Version 2.1

.............. 678February 14, 2000: Version 2.0.1

............... 681October 30, 1999: Version 2.0

................ 691Release Notes for Version 1.x

............... 691August 17, 1999: Version 1.3

................ 693July 15, 1999: Version 1.2

................... 695March 26, 1999

............... 695January 29, 1999: Version 1.1.1

............... 696January 15, 1999: Version 1.1

.............. 697November 6, 1998: Version 1.0.1

............... 697November 1, 1998: Version 1.0

................... 698Directory Structure

.................... 698Top Level

................ 699bin and lib Subdirectories

................. 699The docs Subdirectory

................. 699The src Subdirectory

ix

................. 699The MITRE Subdirectory

................. 699The Galaxy Subdirectory

.................... 700Templates

.................. 700ARCH-OS Support

.................. 701Guide to the Makefiles

.................... 701Introduction

................ 701Structure and Organization

.................... 703Windows port

.................... 703Introduction

................. 703What has been ported?

............ 704What libraries do I need to compile the HUB?

................. 705How do I write a server?

............. 705Differences between Unix and Windows

.................... 706Thread Notes

..................... 706Signals

................... 707Platform notes

x

Toplevel Index

The Galaxy Communicator software infrastructure is a distributed, message-based,hub-and-spoke infrastructure optimized for constructing spoken dialogue systems. The GalaxyCommunicator infrastructure is an extension and evolution of the distributed infrastructure for the MIT Galaxy system, and is being developed and maintained by the MITRE Corporation underthe aegis of the DARPA Communicator program. The Galaxy Communicator infrastructure isavailable under a liberal open source license from the Galaxy Communicator Sourceforge site.This is version 3.3 of the Galaxy Communicator distribution.

1

http://communicator.sourceforge.net/

http://www.darpa.mil/ito/research/com/index.html

http://fofoca.mitre.org/

http://www.sls.lcs.mit.edu/sls/whatwedo/architecture.html

This documentation is divided into four parts:

a tutorial, which will introduce you to the fundamentals of the Galaxy Communicatorinfrastructure and teach you enough to get started; a set of advanced topics, which describe the more complex capabilities of the infrastructure; a reference section, which provides additional details about the basic and advanced topics; and an administrative section, which deals with installing and upgrading the infrastructure.

Before you begin, please read the introduction and overview, which will help you decide whetherthe Galaxy Communicator infrastructure can help you.

Getting helpThere are two primary ways of getting help with Galaxy Communicator. First, there are publicmailing lists for Galaxy Communicator which you can subscribe to. Second, you can send amessage to [email protected]. Please do not send messages directly to theindividual developers; it’s easier for us to track problems and respond promptly if you use themailing lists or the bug queue.

TutorialIf you intend to run the examples in the tutorial rather than simply read it, make sure that theGalaxy Communicator distribution has been installed so that the tutorial can be run.

Please note that the tutorial currently only runs on Unix, and that it assumes familiarity with theUnix command line.

Getting started: setting up your environment The basics: how the Hub and server communicate, and what they say Our first tool: the process monitor A sample end-to-end system: MITRE’s toy travel demo Introducing frames and objects Server basics Our second tool: the unit tester Writing a basic Hub program file Error handling Sending new messages to the Hub Setting up a brokered audio connection Creating a UI server Setting up the Hub to log your interactions Sessions Assembling an end-to-end system

2

http://communicator.sourceforge.net/lists.html

http://communicator.sourceforge.net/lists.html

Advanced topicsThese topics are also presented in tutorial style, where possible.

Coordinating locations of Hub and server listeners How to use the Hub in scriptless mode How to set up listeners in the Hub for servers to contact How to use the timed task library How to manage session information and use continuations How to control Hub server and session properties from the server How to use a different main loop How to use non-MIT makefiles How to do signal handling How to work with binary data How to use the command line argument parsing and printing facilities How to use the Communicator library in arbitrary executables

ReferenceFrequently asked questions The Hub and server executables Frame and object reference Building a Communicator-compliant server Program file reference Logging reference Broker reference The structure of a server Reserved frame keys in Galaxy Communicator API indexes

Alphabetical index of API calls Other calls in the core library API Other calls in the I/O library API Other calls in the ServerStub library API Other calls in the utilities library API

Glossary Examples

Basic server example: double Exemplifies the basics of creating a server, declaring servers and message signatures,sending new messages to the Hub, raising and handling errors, and defining server andconnection initialization functions, in C, Python, Java and Allegro Common Lisp. Alsoexemplifies using command line argument parsing, scriptless interaction, run-time scripterror checking, logging, listener-in-Hub functionality, and how to use continuations.

3

Audio and brokering Exemplifies creating and managing broker connections in C, Python, Java and AllegroCommon Lisp. Also exemplifies interacting with the MITRE broker packaging utilities. Alternative main loops Exemplifies how to write Communicator-compliant servers using a non-Communicatormain loop. Binary objects for audio and data Exemplifies how to use binary (8-bit) data for audio in frames (instead of brokering) andfor opaque encoding of arbitrary objects (including MITRE support). Using non-MIT makefiles Exemplifies how to compile Communicator-compliant servers without using the MITmakefile templates. Running a multiple-connection server Illustrates how to run multiple Hubs against the same server.

MIT executables and facilities Using turn management for dialogue control The dialogue control library verify_program builtin

MITRE utilities Installing and using the utilities Alphabetical index of MITRE utilities API calls Convenient tools for specifying frames Tools for exploiting MIT’s binary data type Tools for brokering Tools for adding a poll on stdin to the server loop

MITRE tools MITRE server scripts MITRE process monitor MITRE unit tester MITRE log tools

MITRE toy travel demo reference Programming language bindings

Java Python Allegro Common Lisp

AdministrationAll bug reports and requests for help should be sent to [email protected] do not contact members of the Communicator team directly.

4

Installing the distribution (significantly enhanced for 3.3) New features in the current distribution Upgrading

Upgrading from 3.2 to 3.3 Upgrading from 3.1 to 3.2 Upgrading from 3.0 to 3.1 Upgrading from 2.1 to 3.0 Upgrading from 2.0 to 2.1 Upgrading from 1.2 or 1.3 to 2.0

Release notes for version 3.x Version 2.x Version 1.x

Details Directory organization Guide to the Makefiles Using Communicator on NT Thread notes

Last updated January 4, 2002

5

Introduction and Overview

History What you get What you don’t get Prerequisites: technical background, hardware and software requirements Technical overview

History

What you getThe Galaxy Communicator infrastructure is a hub-and-spoke architecture. The distribution contains:

6

A Hub, implemented in C, which mediates connections between Communicator servers(such as speech recognition and synthesis, parsing, dialogue management, etc.) Server libraries for constructing Communicator-compliant servers in C (and C++), Java,Python, and Allegro Common Lisp Examples illustrating basic and advanced functionality for creating servers and setting upthe Hub to communicate with them Extensive documentation (which you’re now reading) Example servers such as wrappers for the TrueTalk and Festival speech synthesizers, andfor Oracle and PostGres database clients

What you don’t get

You don’t get an end-to-end dialogue systemThe Galaxy Communicator distribution is not an end-to-end dialogue system; it provides youwith tools for constructing such a system out of a suite of servers. You can obtain the serversyou need in a number of ways:

You may find some of them in the Galaxy Communicator distribution, as noted above; You may be able to obtain some of them from willing participants in the Communicator program; You may choose to write some of them yourself.

MITRE intends in FY02 to assemble an open-source toolkit of Communicator-compliant serverswhich make up an end-to-end dialogue system, but this effort is in its infancy. At the moment, ifyou want to use this infrastructure to build an end-to-end dialogue system, you’ll need manypieces that aren’t provided here, such as audio support, speech recognition, language parsing,dialogue management and language generation.

You don’t get run-time semantic standardsThe Galaxy Communicator infrastructure provides a sophisticated and general transport layer forconnecting servers and Hubs, as well as a message syntax, but does not provide anyspecifications about the semantics of the messages which travel between the servers andHubs. That is, there’s no standard run-time API for speech recognizers, or audio devices, orparsers. While the MITRE team has explored adapting existing APIs or message sets for usewith Communicator-compliant servers, nothing in the infrastructure endorses any of theseexisting APIs or message sets.

You don’t get configuration-time semantic standardsThe Galaxy Communicator infrastructure provides no support or specifications for standards forconfiguring individual servers. For instance, the W3C Voice Browsers group is proposingspecifications for speech recognition grammars. Such proposals are completely compatible withthe Galaxy Communicator infrastructure, but the infrastructure does not endorse any particular

7

http://www.w3c.org/voice

proposal.

Prerequisites: technical background, hardware andsoftware requirements

Technical backgroundThe core Galaxy Communicator infrastructure is written in C, and detailed documentation isprovided only for the C libraries, so familiarity with C is fairly important. Since the GalaxyCommunicator infrastructure is a distributed infrastructure, some background in distributedprocessing is preferable (e.g., RPC, CORBA, Java RMI), since distributed processing is a fairlydistinct programming paradigm. Object-oriented programming experience is not needed unlessyou’ll be using the Java or Allegro Common Lisp server bindings. Finally (and fairly obviously,since you’re already reading this), a reasonable command of English is required forunderstanding the documentation.

Supported platformsThe current version of the Galaxy Communicator infrastructure is actively supported on SparcSolaris, Intel Linux and Win32. Consult the installation instructions for more details.

Software requirementsThe Galaxy Communicator infrastructure requires the GNU gcc compiler and GNU make.Consult the installation instructions for more details.

Technical overview

8

A Communicator-compliant dialogue system consists of a process called the Hub, together witha set of servers. Servers and the Hub communicate with each other using named attribute-valuestructures called frames. These frames form the basis of all structured communication in theGalaxy Communicator infrastructure.

Hub

9

Almost all communication between Communicator-compliant servers passes through the Hub.The Hub has a number of significant capabilities:

The Hub maintains connections to servers (parser, speech recognizer, backend, etc.), androutes messages among them. The Hub can be configured to log aspects of this message traffic. The message traffic routing in the Hub can be programmed via a scripting language thatcontrols the flow through each dialogue turn; the default scripting language is the MITscripting language, but users can opt to use no scripting language or incorporate their own. The Hub also incorporates an internal server named Builtin to implement user-visibleadministrative tasks

10

Server libraries

The server libraries provide a number of convenient capabilities for managing the data andcommunications. Detailed documentation is provided for the C bindings; the documentation forother programming language bindings contains representative examples and equivalence tables.

The server libraries provide a set of commonly-available command line arguments andsupport for defining dispatch functions, which are invoked in response to frames from the Hub. The server libraries provide support for backchannel connections for high-bandwidth datathat can be passed directly from server to server.

Last updated September 28, 2001

11

Tutorial

12

Getting Started

The tutorial portion of the Galaxy Communicator documentation makes a few assumptions aboutyou, your Galaxy Communicator installation, and your computing environment. Two of thesehave already been noted:

There are special instructions for installing the distribution so that the tutorial can be run. The tutorial currently only runs on Unix, and it assumes familiarity with the Unix command line.

In the remainder of this document, we’ll describe what else you need to do to get yourself started.

Reference environmentWe believe that the training course should work on any Unix environment on which the GalaxyCommunicator installation can be installed. However, for reference purposes, here is theconfiguration of our development environment for the tutorial:

Hardware: Sparc Ultra 10 OS: Solaris 2.7 X implementation: /usr/openwin Window manager: fvwm Shell: tcsh Editor: emacs 20.4.1 Python version: 1.5

If your behavior differs from ours, it may be due to a difference in environment.

Start up X windowsThe tutorial assumes that you’re running X. Since you’re reading this documentation, you’realmost certainly in X already if you’re on a Unix machine.

13

Set up your shell environmentFor convenience and generality, we use the shell variable $GC_HOME to refer to the root of yourGalaxy Communicator installation. We recommend that you set this value in your shell. So ifyour Galaxy Communicator installation root is /usr/local/GalaxyCommunicator-3.2 , do this:

csh/tcsh:

% setenv GC_HOME /usr/local/GalaxyCommunicator-3.2

sh/bash:

$ GC_HOME=/usr/local/GalaxyCommunicator-3.2; export GC_HOME

At this point, you should be able to run the examples in the tutorial.

If you add these settings to your shell startup file file, you can run the examples from any Xterminal. Otherwise, you will only be able to run the examples in the X terminal in which youmake these settings, because the PATH value you’ll set in the next step is usually overwrittenwhen you start up a new shell.

Modify your pathIn order to save heaps of typing, we recommend that you add a couple directories to your Unixsearch path. One is the location of the main Hub executable, and the other is a directory whichhosts a number of useful MITRE tools.

csh/tcsh:

% setenv PATH $GC_HOME/bin:$GC_HOME/contrib/MITRE/tools/bin:$PATH

sh/bash:

$ PATH=$GC_HOME/bin:$GC_HOME/contrib/MITRE/tools/bin:$PATH; export PATH

In all the examples, we’ll assume that the path has been updated as shown here.

If you add these settings to your shell startup file file, you can run the examples from any Xterminal. Otherwise, you will only be able to run the examples in the X terminal in which youmake these settings, because the PATH value is usually overwritten when you start up a new shell.

14

Next : The basics: how the Hub and server communicate, and what they say


15

How the Hub and Server Communicate,and What They Say

An instance of the Galaxy Communicator infrastructure consists of potentially many processes,perhaps running on separate computers. These processes are arranged in a hub-and-spokeconfiguration. The spokes are known as servers , while the central hub is named (obviouslyenough) the Hub (capitalized throughout the documentation). Because none of these processesoperates independently, it sometimes seems like it’s necessary to learn everything at once. Inthis first lesson, we’ll try to explain how these processes communicate, and how a configurationof servers and Hub get started.

Frames Messages

When a server receives a new message When the Hub receives a new message A simple message sequence

Starting up a Galaxy Communicator system Servers start up Hub reads program file Hub and servers contact each other

Running a Galaxy Communicator system When the Hub receives a new message: tokens and rules First new message source: the initial token Second new message source: reinitialize Third new message source: user gestures

Summary

FramesAlmost all communication in the Galaxy Communicator infrastructure is in the form of a datastructure called a frame . A frame is an attribute-value structure consisting of a name, a frametype (always "c" for clause, for historical reasons), and a set of pairs of keys and associatedtyped values. The keys, by convention, start with a colon (":"), but nothing in the infrastructureenforces this convention; the types for the values are drawn from a fairly standard defined set oftypes (integer, float, string, frame, list, etc.). These frames can be constructed using API calls, or

16

parsed from a formatted representation, illustrated here:

In its printed representation, a frame is delimited by curly brackets "{}"; the type "c" appearsimmediately after the opening bracket, followed by the whitespace-delimited name, and finallythe alternating keys and values are listed. We’ll look at the details of the frame API and itsprinted representation later; all that’s important right now is that you recognize the printedrepresentations of frames when you see them.

MessagesA message is just a frame which is passed from the server to the Hub, or from the Hub to theserver. These messages can be new messages, which initiate an action, or a reply to a newmessage. Either the Hub or server can send or receive new messages. In this section, we’ll talkabout what happens when a process receives a new message; later on,we’ll talk about how aprocess might send a new message.

When a server receives a new messageFrom the point of view of the Hub, servers provide bits of behavior which the Hub can invoke bysending a new message to the server. These bits of behavior are called dispatch functions .When a server receives a new message, it looks for a dispatch function which it supports whosename matches the name of the incoming message frame:

17

The name of the incoming frame can be qualified by an "address", which tells the Hub whichserver to send the message to, or it can be unqualified. In the illustration here, the frame whichthe speech recognition server received has the address Recognizer and the dispatch functionname Recognize . The server finds the named dispatch function and invokes it.

Dispatch functions can do a wide variety of things. They can send message replies, send newmessages and optionally await message replies, or set up callbacks for backchannelconnections called brokers . We’ll cover all these topics later.

When the Hub receives a new messageIn addition to routing message traffic among the various servers, the Hub can be provided withsimple scripts called Hub programs . These Hub programs are sequences of rules which dictatewhich dispatch functions are invoked on which servers, what order they’re invoked in, and underwhat conditions.

18

When the Hub receives a new message, it searches for a Hub program whose name matchesthe name of the incoming message. If it finds a program with that name, it invokes the program.This is called a scripted interaction . If the Hub doesn’t know of a program with the given name,it searches for a server which provides a dispatch function with the given name, and relays themessage it received directly to the server which provides the dispatch function. This is called a scriptless interaction . If neither a program nor a server is found, the new message is discarded.

This illustration provides an example of a scripted interaction:

In this illustration, the audio server sends a message named FromAudio to the Hub, whichknows of a program with that name. The program has a rule which tells the Hub to send amessage named Recognizer.Recognize .

A simple message sequenceIf you’ve guessed that the message the Hub is sending in these examples is the same as themessage the server is receiving, you’re right:

19

So firing a rule in a Hub program is one way that the Hub can send a new message. Later on inthis lesson, we’ll learn about the other ways that new messages can be sent.

Starting up a Galaxy Communicator systemIn order to run a configuration Galaxy Communicator processes, you have to start up all theprocesses: all the servers, and the Hub. The most convenient order for starting these processesdepends on the Hub. In some cases, the Hub will contact the servers, and in other cases, theservers will contact the Hub. The details of this initialization sequence can be controlled in anumber of complex ways, but we’ll describe a simple case here.

Step 1: Servers start upTypically, the first thing that happens is that the servers which the Hub will contact start up:

20

These servers will set up a port to listen for connections from a Hub.

Step 2: Hub reads program fileTypically, the Hub starts up next. When the Hub starts up, it reads a program file (specified onthe command line with the -pgm_file directive), which contains:

a list of servers to communicate with (and whether the Hub should contact the servers or theservers will contact the Hub) global information such as logging instructions zero or more Hub programs

21

If the Hub is supposed to contact a server, the program file will list the host and port the Hubshould contact the server at. In this case, the Recognizer server is listening on port 15000 onthe local host.

Step 3: Hub and servers contact each otherNow, the Hub contacts the servers it’s supposed to contact, and starts listening for connectionsfrom servers which are supposed to contact it. When the Hub and server make a connection (nomatter which side initiates it), the Hub sends a connection-specific initialization message named reinitialize , which executes the reinitialize dispatch function on the server (if it’s defined).

22

In this illustration, the Hub is contacting all the servers except for Audio , which contacts the Hubat some later point.

Running a Galaxy Communicator systemOnce the Hub has contacted the servers it’s supposed to contact, the system begins its normaloperation. During normal operation, the Hub and servers wait for the arrival of new messages.So where do these new messages come from?

When the Hub receives a new message: tokens and rulesBefore we talk about where new messages come from, we need to go into a little more detailabout what happens when the Hub receives a new message.

Each new message specifies whether the process that sent it wants an answer back or not. Foreach new message, the Hub creates a token , which is initially just a copy of the incomingmessage frame:

23

At this point, the Hub looks for an appropriately named Hub program. If the Hub doesn’t findsuch a program (a scriptless interaction), the Hub relays the message to a server which providesa dispatch function named by the incoming message, and if the original sender wants an answerback, the Hub relays the reply from the dispatch function provider back to the original sender. Inthis case, the only information in the token that the Hub uses is whether the original senderwants an answer back.

If the Hub does find an appropriate Hub program (a scripted interaction), the token becomes thememory state of the Hub program, which is updated, step by step, by the rules in the program.The rules contain conditions under which the rules should be fired, instructions for constructing anew message to be sent to a server, and instructions for how to use the reply from the server toupdate the token. When the program terminates, the current token is returned to the originalsender, if the original sender wants an answer back.

First new message source: the initial tokenSo we’ve already seen the two ways way the Hub generates a new message for a server: byfiring a rule in a Hub program, or by relaying a message to a server in a scriptless interaction. Inmost cases, the new message which causes the Hub to do this comes from a server, but there isone crucial case in which the message comes from the Hub itself: the initial token .

The initial token is a frame, declared in the Hub program file, which is processed immediatelyafter the Hub contacts the servers it’s supposed to contact. This frame is treated as a newmessage received by the Hub, and is processed accordingly. So one way to generate a newmessage is to specify an initial token.

24

This is a fairly crude and unreliable way to generate a new message, though. First, the initialtoken is only processed once, and like all new messages, if it can’t be processed, it’s discarded.So if you have servers which contact the Hub which are crucial to the execution of the initialtoken, chances are they won’t connect in time, and the wrong things will happen. So while theinitial token is valuable for one-time system initializations (and for demonstration purposes), it’snot a reliable run-time source of new messages.

Second new message source: reinitialize

We noted that when a server receives a new message and executes a dispatch function, it canin turn send a new message to the Hub. But this process can seem circular: the Hub sends anew message to a server, which sends a new message to the Hub, which sends a new messageto a server, and on and on. We’ve seen one way this process can start: via the initial token. Canthis process start from dispatch functions?

The answer is yes. Remember that when the Hub and server establish a connection, the Hubsends an initialization message to the server named reinitialize , which will then invoke the reinitialize dispatch function. This means that this dispatch function is called without theHub needing to receive a message.

Since reinitialize is called every time a connection is established, it’s more reliable than theinitial token. First, it happens more often, and second, the timing of the dispatch function isclearly defined relative to server connection time. So for example, a batch processor could fireoff its first batch processing request from the reinitialize dispatch function, and fire off eachsuccessive request upon receiving the result of the current request, until the batch process iscomplete. But this source of new messages is still limited; it’s once per connection establishment.

Third new message source: user gesturesThe third source of new messages is by far the most common: user gestures. The GalaxyCommunicator infrastructure is designed for human-computer interaction, and most suchinteractions ultimately originate with the user, via a mouse click, microphone input, typed input,pen gesture, or some other UI action. The Galaxy Communicator infrastructure is designed tointeract gracefully with processes which are connected both to a Hub and to the user in someway, and you can write these processes to translate user gestures into Galaxy Communicator messages.

SummaryIn this introductory lesson, you’ve learned the basics about a wide range of elements of theGalaxy Communicator infrastructure:

servers the Hub frames

25

messages dispatch functions, including reinitialize brokers Hub programs and program rules scripted and scriptless interactions program files tokens, including the initial token

In the remainder of this tutorial, you’ll learn enough about each of these elements to allow you tobuild your own Communicator-compliant system.

Next : Our first tool: the process monitor


26

How to Use the Process Monitor

The process monitor is a tool which allows the user to control and monitor system processes in avariety of configurations. This tools is used extensively in the tutorial and in the examples in theGalaxy Communicator distribution. In this lesson, we learn about the various parts of the processmonitor and how to use them.

Starting the process monitor The menu bar The process panes Using the menu bar

Once you feel comfortable with the process monitor, if you want to know more, you can consultthe process monitor reference.

Starting the process monitorThere are a couple different ways to start the process monitor, but we’ll be starting it with a filewhich describes the configuration of processes that it will run. The process monitor executablecan be found at $GC_HOME/contrib/MITRE/tools/bin/process_monitor . Let’s start itup to run a couple of simple Unix applications.

[Process monitor exercise 1]

% process_monitor $GC_HOME/tutorial/process_monitor/dir.config &

(If the shell can’t find the process monitor application, make sure you’ve configured yourenvironment correctly.) You’ll get a window which looks like this:

27

The menu barThe process monitor contains a menu bar, which controls the overall configuration of the processmonitor and the collection of processes.

The File menuThe File menu contains the Quit command, which shuts down all the processes and exits theprocess monitor.

The Process Control menuThe Process Control menu contains three commands:

Stop all, which stops all the processes Clear all, which resets all the scrollable output histories Restart all, which stops all the processes which are running and then starts all the processes

28

The Configuration menuThe Configuration menu allows you to choose between two different configurations:

Column configuration, which displays all the processes in one, two or three columns Compressed configuration, which displays only a single process at a time, which you canswitch between with a set of radio buttons; you can select one, two or three rows of buttons

The process panesThe process monitor presents each process as four panes:

A command bar, which includes the Hide/Show button as well as the title of the process A scrollable output history, which contains the command to be executed, in square brackets,and the output of the command (both standard output and standard error) An editable command line A control bar, which contains the Start/Stop button, the Keep alive/Let die button, the Clearbutton and the Pause output/Resume output button

Whether or not all these panes are visible depends on the Hide/Show button. In the examplehere, there are two processes, one which has all four panes visible, and one which has only thecommand bar.

Hide/ShowPress the button marked "Hide" on the command bar titled "Current directory". See how thelower three panes for the process vanish. Like the command bar titled "Directory contents", therun status now appears on the command bar on the far right, and this button now reads "Show".Press it again to restore the configuration.

Start/StopPress the button marked "Start". The process monitor will now run pwd, and you’ll see somethinglike the following output in the scrollable output history:

[Scrollable output pane]

[exec pwd 2>&1] /usr/GalaxyCommunicator-3.2 ======================================

Note the row of equals signs ("="). This indicates that the process has halted. You may alsohave noticed that for the instant the process is running, this button reads "Stop". While the buttonreads "Stop", you can press it to stop the process.

29

Keep alive/Let diePress the button marked "Keep alive". The process will now be automatically restarted if it diesof its own accord. The button now reads "Let die". Press "Start" and notice how pwd is calledover and over. Press "Stop" to halt this sequence, and "Let die" to disable automatic restart.

ClearPress the button marked "Clear". The scrollable output history is cleared and the command to beexecuted is rewritten to the output pane.

Pause output/Resume outputThis final toggle button pauses and resumes the output. Press "Keep alive" again, and then"Start". As the process is running over and over, press "Pause output", and note that nothing isnow being added to the scrollable output pane. The button now reads "Resume output". Press"Resume output", then "Stop", then "Let die".

Using the menu barPress "Show" on the command bar titled "Directory contents". Select "Process Control -> Restartall". Both processes run and terminate. Select "Process Control -> Clear all" to clear all thescrollable output panes.

Next, we’ll switch configurations. Select "Configuration -> Compressed configuration -> Onebutton row". Now the process monitor displays a single output history, command line and controlbar, with a row of buttons directly below the menu bar to select the displayed process. Tryswitching between the two processes, and start, stop and clear the processes to convinceyourself that the process displayed in the pane corresponds to the selected process in the button row.

The "Detach this pane" button at the bottom allows you to view a given process in its ownwindow with its own set of panes. Press this button. The button corresponding to the currentprocess disappears from the button row, and the new window which appears has a no menu barand a button at the bottom which says "Reattach this pane". When the separate window isreattached, the corresponding button is re-added to the button row on the main window.

Once you’re done exploring the process monitor, select "File -> Quit" to exit.

Next : A sample end-to-end system: MITRE’s toy travel demo


30

A Sample End-to-End System

The best way to understand how a configuration of Galaxy Communicator servers might worktogether is to watch an example in action. For pedagogical purposes, we’ve constructed a set ofdummy components which follow a script of message sequences. In this lesson, we’ll follow asingle interaction with this set of components, to illustrate one plausible flow of control formessages in a Communicator-compliant system, which we call our toy travel system .

The servers Setting up the toy travel demo

Starting the process monitors Understanding what you see: server startup Controlling how much you see Understanding what you see: Hub startup Starting the interaction

Step 1: Audio available Understanding what you see: Hub processing

Step 2: Reroute to handle general input Understanding what you see: server processing

Step 3: Send to dialogue Step 4: Dialogue consults backend Step 5: Dialogue reply to generation and synthesis Step 6: Audio output Summary

The serversWe will focus our attention on seven dummy servers, plus the Hub’s Builtin server:

31

There are two other dummy servers in the toy travel system. One is a dummy server which isintended to emulate typed input and output; we will not use this server in this course. The otherserver, called IOMonitor, monitors both input and output and reports what is said by each side;we do use this server in this lesson, but we omit it from the illustrated flow of control for the sakeof simplicity.

Setting up the toy travel demo

Starting the process monitorsWe’ll be using the process monitor once more. Start up the demonstration as follows:

[Toy travel demo exercise 1]

% process_monitor $GC_HOME/tutorial/toy-travel/short-toy-travel.config

32

You’ll get a process monitor in its compressed configuration, with three button rows of threeprocesses each. Select "Process control -> Restart all". You’ll start all the processes in the orderof the presentation of the buttons. The processes are started in the order described in the tutorialon starting up a Galaxy Communicator configuration: first the servers which are listening forconnections, then the Hub, then the servers (in this case, the dummy audio server), which willconnect to the Hub. More precisely, in the case of the audio server, you’ll get a second processmonitor window which will allow you to start the audio server separately (we’ll do that in just a minute).

At this point, you should have two process monitors on the screen, like so:

33

Understanding what you see: server startupThe "Recognizer" button is selected in the "Toy travel single exchange" process monitor, so thispane shows the output of the recognizer. We can see three lines of output. The last two tell us,first, that the recognizer server is available for connections on port 11000, and second, that therecognizer server has accepted a connection (from the Hub; remember, it’s started up already).

You’ll find that if you select any of the first six processes, you’ll see a variation of this output; allof these servers will have started listening (each on a different port, of course), and all of theseservers will have accepted a connection. However, the last three processes will look different.

Controlling how much you seeSelect the "IOMonitor" button. The only indication that the server is running is that the Start/Stopbutton now reads "Stop"; there will be nothing in the output pane except the original command line:

[IOMonitor pane]

[exec $DEMO_ROOT/bin/IOMonitor -verbosity 0 2>&1]

The reason that there’s no other output is because we’ve limited the verbosity of the server to 0.

For Communicator-compliant servers, there are six levels of verbosity. The status messages inthe Galaxy Communicator infrastructure can be made sensitive to the verbosity level. 0 is themost severe; no verbosity-sensitive status messages of any sort are printed. 3 is the default; atthis level, you’ll see normal status messages, indicating when connections are established andlost, and what messages are being sent and received. 6, the most verbose, provides debugginginformation and a full dump of the encoded message traffic. In this tutorial, we’ll only useverbosity levels 0 or 3 (the default).

34

You can control the verbosity in two ways. First, as shown here, Communicator-compliantservers all accept the -verbosity command-line argument, as does the Hub. Second, youcan set the environment variable GAL_VERBOSE to the verbosity level you desire. In this tutorial,we’ll only use the command-line argument. (Notice that the command line for the audio server inthe "Audio client" process monitor also limits the verbosity to 0.)

The IOMonitor will print out a transcription of the dialogue you’re about to trace through. Makesure that the "IOMonitor" button is still selected, and press "Detach this pane" to detach the IOMonitor.

Understanding what you see: Hub startupNow select the "Hub" button in the "Toy travel single exchange" process monitor. We’ll nowexamine the output of the Hub up to this point. Detach the Hub pane, and enlarge it for easierviewing. First you’ll see printouts informing you that the Hub is reading and loading the program file:

[Hub pane]

Reading /usr/local/GalaxyCommunicator-3.2/tutorial/toy-travel/toy-travel.pgm Done reading/usr/local/GalaxyCommunicator-3.2/tutorial/toy-travel/toy-travel.pgm(270 lines) 9 service types 7 service providers 9 programs

Notice that the Hub distinguishes between service types , which are named collections ofbehavior (e.g., the service named Parser provides the operation Parse), and service providers ,which are actual processes which are instances of service types. So the Parser server in ourconfiguration, from the Hub’s point of view, is a provider for the Parser service.

Next, you’ll see printouts informing you that the Hub is listening for connections from the serverswhich might contact it (the audio server, in this case). Observe that the Hub reports setting upthis connection listener just as the Recognizer server does:

[Hub pane]

-------------------------------------------------- service type: Audio : 2800 Opening listener Audio Trying to set up listener on port 2800 reset_listener called GalIO_ServerStart: Listening on port 2800

35

Finally, the Hub contacts all the other servers, and exchanges some crucial connectivityinformation (which we don’t care about). Here’s an example for the dummy Parser server:

[Hub pane]

provider: for Parser @ localhost:10000 GalIO_CommDispatchFrame: sending new message: {c handshake :conn_type 1 :protocol_version 1 } GalIO_CommDispatchFrame: got reply: {c Parser :signatures ( ( "Parse" ( ) 2 6 ( ) 2 ) ) :properties {c server_properties } :extra_service_types ( ) :protocol_version 1 } connected to provider for Parser @ localhost:10000.

There are two things which commonly appear which we haven’t seen so far: processing theinitial token in the Hub, and processing a reinitialize message in the server. We’ll see boththese things in later lessons.

Starting the interactionAt this point, we’re ready to begin. The message script we’re going to follow models the followingsimple dialogue:

System: Welcome to Communicator. How may I help you? User: I WANT TO FLY FROM BOSTON TO LOS ANGELES System: American Airlines flight 115 leaves at 11:44 AM, and Unitedflight 436 leaves at 2:05 PM

We’re not going to step through the initial greeting in detail; we’re going to pay detailed attentiononly to the user request and system reply. In the following lessons, we’ll learn in detail how thisexchange is constructed.

Press the "Start" button on the "Audio client" process monitor to start the dummy audio server,which will contact the Hub. The reason the audio server contacts the Hub is that we want tomanage access to our speech-based service dynamically, so that users can contact it from anyone of a number of phone lines or desktop microphones. Since every one of the audioconnections needs to be a Communicator-compliant server, we can either choose a fixednumber (from fixed locations) when we write the Hub program file, or we can set up a listener sothat any appropriate service can contact the Hub.

36

Now that you’ve pressed the "Start" button, a number of things have happened. The Audioserver sent a message which caused the Dialogue server to send a greeting. The Hub hasreported its message processing, which we will ignore for the moment. The detached"IOMonitor" window contains the resulting textual output, and the "Audio client" process monitorshows the (dummy) audio segments which are being streamed to the user:

[IOMonitor window]

In session session-998661263.227: system said "Welcome toCommunicator. How may I help you?"

[Audio client process monitor]

[Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (184 samples)] [Audio data to user is finalized (14520 samples).] Hit <return> to send speech:

The "Audio client" process monitor has a button called "Input <return>". You’ll use this button tomove the dialogue forward, to send the next user utterance. The "Hub" pane also has an "Input<return>" button; the program file the Hub is running has breakpoints inserted at crucial points,and you’ll use this button to step through these breakpoints.

Step 1: Audio availablePress the "Input <return>" button on the "Audio client" process monitor, which simulates telling apush-to-talk audio server that it should start listening. As a result, the Audio server informs theHub that audio is available, and the Hub informs the Recognizer to start processing:

37

At this point, the Hub will be paused at a breakpoint.

Understanding what you see: Hub processingFor this first step, let’s take a detailed look at the Hub output. First, the printout informs us thatthe Hub has received a new message from the Audio server. This output will be at the verybottom of the Hub pane. The Hub constructs a token from this new message, and the token index of the new token is 4 (that is, it’s the fourth new message the Hub has received sincestartup). This new token contains seven key-value pairs, which will will ignore for the moment:

[Hub pane]

Got new message from provider for Audio (id 8)

----------------[ 4]---------------------- {c FromAudio :host "129.83.10.107" :port 15010 :call_id "129.83.10.107:1121:0" :sample_rate 8000 :encoding_format "linear16" :session_id "session-998661263.227" :tidx 4 } ----------------------------------------

38

When the Hub receives a new message, it looks for a program by the same name, and startscomparing the new token state with the conditions on the rules in that program. When it finds arule whose conditions are satisfied by the token state, it invokes the associated dispatch function(which the Hub calls an operation ) by constructing and sending a message to the appropriate server.

In some conditions, the Hub can fire multiple rules in immediate sequence, and in this case, theHub finds and fires two rules: the first invokes the Recognizer.Recognize message, and thesecond invokes Builtin.hub_break , which forces the breakpoint. In each case, there is onlyone service provider for the relevant service type.

[Hub pane]

found operation: Recognizer.Recognize found operation: Builtin.hub_break ---- serve(Recognizer@localhost:11000, token 4 op_name Recognize(sole provider)) ---- serve(Builtin@<none>:-1, token 4 op_name hub_break (sole provider))

Finally, the Builtin server within the Hub receives the hub_break message, and triggers the breakpoint:

GalSS_FrameHandler: input frame: {c Builtin.hub_break :session_id "session-998661263.227" :hub_opaque_data {c admin_info ... } } FnWithData: hub_break (h for help, c or <return> to continue) -->

From this point on, we’ll ignore the calls to hub_break .

Press the "Input <return>" button on the "Hub" pane to continue.

Step 2: Reroute to handle general inputOnce the recognizer is done processing the audio, it sends a new message reporting its results.This is one form of textual input. This configuration of servers, under other circumstances, canalso handle typed input and output. In this next step, we use the Builtin.call_programdispatch function to invoke a new Hub program, which unifies the processing of text input.

39

We see this reflected in the Hub printout:

[Hub pane]

Got new message from provider for Recognizer @ localhost:11000

----------------[ 5]---------------------- {c FromRecognizer :input_string "I WANT TO FLY FROM BOSTON LOS ANGELES" :session_id "session-998661263.227" :tidx 5 } ----------------------------------------

found operation: Builtin.call_program ---- serve(Builtin@<none>:-1, token 5 op_name call_program (sole provider))

Understanding what you see: server processingWe can take this dispatch to the Builtin server as an opportunity to examine a little more closelywhat server-side printouts look like. Immediately after the Hub reports that it’s fired the rulewhich calls Builtin.call_program , the Builtin server "takes over" the printout and reportshow it processes the message. First, the server reports the message it receives, and the factthat it’s found a dispatch function for the message:

40

[Hub pane]

GalSS_FrameHandler: input frame: {c Builtin.call_program :session_id "session-998661263.227" :program "UserInput" :input_string "I WANT TO FLY FROM BOSTON LOS ANGELES" :hub_opaque_data {c admin_info ... } } FnWithData: call_program

Next, the server prints out anything related to what it does to process the message. In this case,it sends a new message to the Hub:

[Hub pane]

GalIO_CommWriteFrame: sending new message: {c UserInput :session_id "session-998661263.227" :input_string "I WANT TO FLY FROM BOSTON LOS ANGELES" :hub_opaque_data {c admin_info :session_id "session-998661263.227" } }

Finally, it reports its return value. In this case, there is none:

[Hub pane]

_GalSS_FrameReturnHandler: no result frame

This structure for printouts is identical to the printout for any server at the default level of verbosity.

Press the "Input <return>" button on the "Hub" pane to continue from the current breakpoint.

Step 3: Send to dialogueAt this point, the Hub will receive the new message sent by Builtin.call_program , named UserInput , and will route this message through the parser to the dialogue manager:

41

At each point where the Hub receives a message reply, it will print out the token state for theupdated token. So we can extract the following sequence from the Hub output (there will bevarious other messages interspersed, such as calls to the IOMonitor to print out the output, tothe breakpoint function, and replies which the Hub program does not need and will ignore):

[Hub pane]

Got new message from provider for Builtin

----------------[ 6]---------------------- {c UserInput :session_id "session-998661263.227" :input_string "I WANT TO FLY FROM BOSTON LOS ANGELES" :tidx 6 } ----------------------------------------

found operation: Parser.Parse ---- serve(Parser@localhost:10000, token 6 op_name Parse (sole provider))

42

Got reply from provider for Parser @ localhost:10000 : token 6

----------------[ 6]---------------------- {c UserInput :session_id "session-998661263.227" :input_string "I WANT TO FLY FROM BOSTON LOS ANGELES" :tidx 6 :frame {c flight ... } } ----------------------------------------

You can see that the token state is evolving as the program proceeds; so token 6 has a value forthe :frame key after the call to the Parser which it didn’t have before.

Press the "Input <return>" button on the "Hub" pane to continue from the current breakpoint.

Step 4: Dialogue consults backendAt this point, the Dialogue server will consult the Backend server and retrieve a databaseresponse. This is accomplished by the Dialogue server sending a new message to the Hub withan indication that it wants a response. The Hub creates a token, finds an appropriate Hubprogram, and executes the program, and returns the updated token state to the Dialogue serverwhen the program ends:

43

The current breakpoint is immediately after the Backend responds to the Hub, but immediatelybefore the end of the program. Press the "Input <return>" button on the "Hub" pane to continuefrom the breakpoint; you’ll see the token state returned to the Dialogue server immediately afterward:

[Hub pane]

done with token 7 --> returning to owner Dialogue@localhost:18500 [Destroying token 7]

Step 5: Dialogue reply to generation and synthesisNow, the Dialogue manager decides that it’s time to say something to the user (in this case, tolist flights). So it sends a new message to the Hub, which is routed through the Generator serverto the Synthesizer server:

44

At this point, the IOMonitor has been notifed what the system is about to say, and you should beable to see the entire three-turn dialogue in the IOMonitor window.

The current breakpoint is set immediately before the call to the Synthesizer server. Press the"Input <return>" button on the "Hub" pane to continue to the final step.

Step 6: Audio outputFinally, the Synthesizer server produces a new message to notify the Audio server that audio isavailable, and the Audio server fetches the audio:

45

You can see the result in the "Audio client" process monitor:

[Audio client process monitor]

[Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (1024 samples)] ... [Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (252 samples)] [Audio data to user is finalized (35068 samples).] Hit <return> to send speech:

At this point, the exchange is over. Press the "Input <return>" button on the "Audio client processmonitor"; since the server has no further inputs in its message script, it reports that audio is nolonger available and shuts down.

46

Select "File -> Quit" in the "Toy travel single exchange" process monitor to shut down the toytravel demo.

SummaryIn this lesson, you’ve seen how a plausible exchange between a user and system might proceedin a Communicator-compliant system. You’ve also learned a few more terms:

verbosity service type service provider token index operation

In the next lessons, we’ll learn more about these terms, and about how this demo is constructed.

Next : Introducing frames and objects

Last updated August 23, 2001

47

Introducing Frames and Objects

As we’ve already learned, the basic structure which is passed between server and Hub is calleda frame . In this lesson, we’ll learn how to manipulate frames, and also a little about how they’re built.

What’s in a frame? Making and freeing frames Making, using and freeing objects

Wrapping data in objects Creating and accessing key-value pairs in frames

Summary

Once you learn enough, you can consult the frame and object reference.

What’s in a frame?A frame consists of

a type, which will always be a clause for us; a name; a set of pairs of keys and their associated values

By convention, the names of keys start with a colon (":"). There are many available types for thevalues, but we will focus on integers, strings, lists, and frames. Here’s what the printedrepresentation of all these types looks like:

48

A frame in C is a structure called a Gal_Frame. The values are all different basic C types:integers are C int , frames are C Gal_Frame , strings are C char * . In order to be able tomanipulate these values with a common interface, they are wrapped in a C structure called aGal_Object, which contains the data and a constant of type Gal_ObjectType indicating the datatype. Lists, for instance, are implemented as arrays of Gal_Objects, so that it’s possible to putelements of different types in the same list. Here’s approximately what the frame above lookslike, structurally:

Here’s a table of the properties of the four types we’ll be discussing.

49

Gal_ObjectType C typeprinted representation

comments

GAL_FRAME Gal_Frame{<type> <name><key> <value>... }

whitespace between openingbracket and type is forbidden;whitespace between elements (type,name, keys, values) is obligatory

GAL_STRING (char *) "foo"whitespace in strings is preserved;escape character is backslash (\)

GAL_INT 32 bit integer 5explicit sign prefixes (+,-) also recognized

GAL_LISTan array of Gal_Objects

( <obj> <obj><obj> )

lists are heterogeneous (multipleobject types can appear in a singlelist); whitespace between elementsis obligatory

Making and freeing framesThe fundamental functions for creating a frame is Gal_MakeFrame. In C, because you have todo your own memory management, you must also free the frames you use with the function Gal_FreeFrame.

/* Create the frame */ Gal_Frame f = Gal_MakeFrame("output", GAL_CLAUSE);

/* Do something with the frame */ ... /* Free the frame */ Gal_FreeFrame(f);

Alternatively, you can use the function Gal_ReadFrameFromString to convert the stringrepresentation of a frame into a Gal_Frame structure:

/* Create the frame */ Gal_Frame f = Gal_ReadFrameFromString("{c output }");

/* Do something with the frame */ ... /* Free the frame */ Gal_FreeFrame(f);

50

Making, using and freeing objects

Wrapping data in objectsFor each type, there’s a function for creating an appropriate Gal_Object, as well as a function forextracting that datatype from the created Gal_Object:

Gal_ObjectType Creating a Gal_Object Extracting the data from the Gal_Object

GAL_FRAME Gal_FrameObject Gal_FrameValue

GAL_STRING Gal_StringObject Gal_StringValue

GAL_INT Gal_IntObject Gal_IntValue

GAL_LIST Gal_ListObject Gal_ListValue

All these objects can be freed using the function Gal_FreeObject. Here are some examples withframes, strings and integers:

Gal_Object frame_obj, string_obj, int_obj, list_obj; Gal_Object list_array[3]; Gal_Frame output_frame; char *foo; int five; int list_length; Gal_Object *list_contents;

/* Create a frame and wrap an object around it */ frame_obj = Gal_FrameObject(Gal_ReadFrameFromString("{c output }"));

/* Wrap an object around a string */ string_obj = Gal_StringObject("foo");

/* Wrap an object around an integer */ int_obj = Gal_IntObject(5);

/* Extract the frame from the object */ output_frame = Gal_FrameValue(frame_obj);

/* Extract the string from the object */ foo = Gal_StringValue(string_obj);

/* Extract the integer from the object */ five = Gal_IntValue(int_object);

51

/* Populate the list array */ list_array[0] = frame_obj; list_array[1] = string_obj; list_array[3] = int_obj;

/* Wrap a list around the array. Printed representation of the list is

( {c output } "foo" 5 ) */ list_obj = Gal_ListObject(list_array, 3);

/* Extract the list from the object. The list_length integer will contain the length of the list (i.e., 3) */ list_contents = Gal_ListValue(list_array, &list_length);

/* Free the list object. See the memory management note. */ Gal_FreeObject(list_obj);

Memory management note . The rules of memory management for frames and objects is a littlecomplicated, and we won’t go into it here. The only point worth noting at this point is that for theset of functions we use here, when frames and objects are freed, all the frames and objects inthem are freed. The full reference has complete details, as well as alternative functions whichgive you more control over memory management.

Creating and accessing key-value pairs in framesThe two basic functions for manipulating objects in frames are Gal_SetProp and Gal_GetObject.Let’s use strings as an example:

Gal_Object string_obj; char *string; /* Create the frame */ Gal_Frame f = Gal_MakeFrame("output", GAL_CLAUSE);

/* Add a string. Printed representation of the frame is now

{c output :output_string "hello" } */ Gal_SetProp(f, ":output_string", Gal_StringObject("hello"));

/* Extract the string object. */ string_obj = Gal_GetObject(f, ":output_string"); string = Gal_StringValue(string_obj);

To extract the data from the frames directly, there’s a set of shorthand functions that wefrequently use. These functions find the appropriate key-value pair and check to see that thevalue is of the appropriate type:

52

Gal_ObjectType Extracting the data from the frame

GAL_FRAME Gal_GetFrame

GAL_STRING Gal_GetString

GAL_INT Gal_GetInt

GAL_LIST Gal_GetList

So the example above could be rewritten as follows:

char *string; /* Create the frame */ Gal_Frame f = Gal_MakeFrame("output", GAL_CLAUSE);

/* Add a string. Printed representation of the frame is now

{c output :output_string "hello" } */ Gal_SetProp(f, ":output_string", Gal_StringObject("hello"));

/* Extract the string object. */ string = Gal_GetString(f, ":output_string");

Lists and integers have some special properties. For integers, the problem is that all thesefunctions return NULL when they don’t find the appropriate key-value pair. This is a problem forGal_GetInt, because NULL is equivalent in C to 0; as a result, Gal_GetInt can’t distinguishbetween a value of 0 and the absence of a key. So we recommend using Gal_GetObjectfollowed by Gal_IntValue if the distinction is important to you.

For lists, there is an additional function Gal_GetListObject which extracts the nth element of a listin a key-value pair:

Gal_Frame f = Gal_ReadFrameFromString("{c listframe :list ( 5 6 7 ) }"); Gal_Object int_obj;

/* int_obj will be a Gal_Object wrapped around the integer 7 */ int_obj = Gal_GetListObject(f, ":list", 2);

53

SummaryThere are many more ways of manipulating frames and objects. There are functions to

compare frames and objects copy frames and objects create objects wrapped around expandable lists and arrays extract the name and type of frames create frames and objects using explicit memory management manipulate many other data types, such as arrays of 8-, 16-, 32- and 64-bit integers print frames and objects in a number of different formats extract and check the type of objects and more

You can find all this information in the frame and object reference. However, for the purposes ofthis course, we’ve introduced in this lesson all the functions you’ll need to understand howframes and objects are being manipulated:

frames: Gal_MakeFrame, Gal_FreeFrame, Gal_ReadFrameFromString frames in other frames: Gal_FrameObject, Gal_FrameValue, Gal_GetFrame strings: Gal_StringObject, Gal_StringValue, Gal_GetString integers: Gal_IntObject, Gal_IntValue, Gal_GetInt lists: Gal_ListObject, Gal_ListValue, Gal_GetList, Gal_GetListObject objects: Gal_FreeObject, Gal_SetProp, Gal_GetObject

In the next lesson, we’ll see how these are used in writing Communicator-compliant servers.

Next : Server basics


54

Server Basics

In this lesson, we’ll learn how to build the simplest of servers: one which receives a messagefrom the Hub and returns a reply. In later lessons, we’ll study successively more complex typesof servers. All our examples will be drawn from the toy travel system we’ve already used.

How complex can a server get? Parser, generator, backend Dialogue Recognizer, synthesizer Audio

Six steps to building a server Step 1: Set up your headers Step 2: Write your dispatch functions Step 3: Write your server initialization function Step 4: Write and incorporate your server declarations Step 5: Write your Makefile and compile Step 6: Write your documentation

Summary

You can consult the complete server reference when you need to.

How complex can a server get?We can divide the seven servers in the toy travel demo into four groups, in order of complexity.

Parser, generator, backendThese three servers all exhibit the simplest functionality possible: they receive a message andsend back a response. We will use the Parser server as our example in this exercise.

DialogueThe Dialogue server is slightly more complex. Instead of simply responding to messages, theDialogue server sends new messages to the Hub, and at times waits for the response. We’ll turnto this server after we learn a little about the Hub scripting language.

55

Recognizer, synthesizerThese servers, in addition to responding to messages, also communicate with the Audio serverusing the brokering backchannel. We’ll learn about brokering in the context of these two servers.

AudioFinally, in addition to responding to messages and communicating with the Recognizer andSynthesizer servers using brokering, the Audio server serves as a UI element, which needs tomonitor an I/O device in addition to its connections to the Hub. We’ll study this server last.

Six steps to building a serverThere are four basic steps to building a Communicator-compliant server.

Step 1: Set up your headers Step 2: Write your dispatch functions Step 3: Write your server initialization function Step 4: Write and incorporate your server declarations Step 5: Write your Makefile and compile Step 6: Write your documentation

Step 1: Set up your headersThis step is simple. All the public Communicator header files can be loaded from $GC_HOME/include/galaxy/galaxy_all.h . For historical reasons having to do with theway the Galaxy Communicator Makefile template is written, we usually include the header file asfollows, and we recommend you do the same:

#include "galaxy/galaxy_all.h"

Step 2: Write your dispatch functionsAll dispatch functions have the same function signature. Let’s use the Parse dispatch functionas an example.

Gal_Frame Parse(Gal_Frame frame, void *server_data);

The frame in the argument list is the incoming message frame. The server_data is actually astructure encapsulating the environment of the dispatch function invocation. We’ll talk about thisargument more when we talk about error handling; for now, you can ignore it.

There are three basic regions in each dispatch function:

56

message decomposition and type checking core processing reply construction

Dispatch functions have the additional complexities of message decomposition, type checkingand reply construction because our messages are in the form of sets of key-value pairs ratherthan ordered sequences of typed elements (as in a C message signature). So the dispatchfunction code needs to do some of the work that a compiler might do in other circumstances. Inthis current discussion, we’re going to cover message decomposition and reply construction;we’ll ignore type checking for the moment, because it relates to the server_data argument.

For example, the core processing in the Parser server is a call to the function ParseSentence :

ParseTree *ParseSentence(char *input_string);

This function takes a string and produces an element of type ParseTree * , which must besomehow translated into something which can be included in a reply frame. We choose totranslate it into a Gal_Frame. Here’s a simplified illustration of the wrapped function:

Let’s take a closer look at each of these regions.

Message decomposition

We need to pass a string to the ParseSentence function, and that string is stored in theincoming message in the :input_string key. We extract it using one of the tools from our last lesson:

57

char *input_string = Gal_GetString(frame, ":input_string");

Core processing

Now that we have the string, we can parse the sentence:

p = ParseSentence(input_string);

Reply construction

Now we need to create a reply message frame, convert the tree to a frame, wrap a Gal_Objectaround the frame, and insert it into the reply message frame. The name of the reply messagedoesn’t matter, since the reply is returned with all the appropriate information about what it’s areply to.

reply = Gal_MakeFrame("reply", GAL_CLAUSE); Gal_SetProp(reply, ":frame", Gal_FrameObject(ParseTreeToFrame(p))); return reply;

The infrastructure takes care of freeing both the input and reply frames after the dispatchfunction is processed. It’s perfectly legal for these frames to be the same; so our simplifiedfunction might have looked like this:

Gal_Frame Parse(Gal_Frame frame, void *server_data) { char *input_string = Gal_GetString(frame, ":input_string"); ParseTree *p;

p = ParseSentence(input_string); Gal_SetProp(frame, ":frame", Gal_FrameObject(ParseTreeToFrame(p))); return frame; }

In this circumstance, all you need to remember is that all the keys in the incoming message willalso be in the reply (unless you overwrite them when you construct the reply).

The reinitialize dispatch function

One of the dispatch functions you can define is reinitialize . As we discussed in the lessonon how the infrastructure works, this dispatch function is called whenever the the Hub and serverestablish a connection. It can also be called explicitly, like any other dispatch function. The onlydifference between the two calling contexts is that when the Hub and server establish aconnection, the value returned from reinitialize is ignored.

The Parser server does not define the reinitialize dispatch function.

58

Step 3: Write your server initialization functionThe next step is to write your server initialization function. This step is optional; the infrastructurewill skip this processing step if the developer doesn’t specify it. The function is called _GalSS_init_server :

void *_GalSS_init_server(GalIO_ServerStruct *server, int argc, char **argv);

This function is called once, when the server starts up. You can use this function to loadgrammars or other models, or otherwise initialize the server state. In the toy travel demo, all theservers must load the message script, so we do that in the server initialization function. Thisfunction uses Galaxy Communicator library functions to parse the argument list and report the result; we won’t be discussing these functions in the tutorial.

void *_GalSS_init_server(GalIO_ServerStruct *server, int argc, char **argv) { char *data_file = (char *) NULL;

if (GalUtil_OACheckUsage(argc, argv, oas, NULL) == 0) exit(1);

if (GalUtil_OAExtract(argc, argv, oas, "-parser_data", GAL_OA_STRING, &data_file) == 0) { GalUtil_Warn("No -parser_data argument. Exiting."); exit(1); } InitializeParser(data_file); return (void *) NULL; }

Without going into the details, this function expects the Parser server to be called as follows:

% Parser -parser_data <file>

The initialization function will pass the string specified by <file> to the function InitializeParser , which in the toy travel demo simply loads the message script.

Step 4: Write and incorporate your server declarationsThe next step is to create and incorporate a header file called a server declaration file. This filecontains macros which specify, among other things,

a name for the server (which in most cases doesn’t have to be the name that the Hub knowsit as) a default port for the listener the server sets up when it listens for connections the available dispatch functions

59

The complete list of specifications you can make in your server declaration file can be found inthe full server reference.

We’ll continue to use the Parser server as an example. First, we’ll create the server declarationsfile and name it Parser_server.h . It looks like this:

GAL_SERVER_NAME(Parser) GAL_SERVER_PORT(10000) GAL_SERVER_OP(Parse)

These declarations announce that the server knows its name to be Parser (GAL_SERVER_NAME), that its default listener port is 10000 (GAL_SERVER_PORT), and that if theserver receives a message whose name is Parse , it should invoke the function with the samename. All dispatch functions must be registered in this way, including reinitialize ;otherwise, the infrastructure has no way of knowing that the function is available.

Second, we’ll tell the server about the server declaration file. You can do this using the following idiom:

#define SERVER_FUNCTIONS_INCLUDE "Parser_server.h" #define USE_SERVER_DATA #include "galaxy/server_functions.h"

The first #define declares the name of the server declaration file. The second #define forcesthe server to use the correct signature for dispatch functions (this is a necessary historical relic).Finally, the #include causes the server declaration file to be loaded and the macros expandedin the appropriate way.

The following picture illustrates all the dependencies involved in the server declarations:

60

Step 5: Write your Makefile and compileThe next step is to construct a Makefile to compile your server. The Galaxy Communicatorinfrastructure provides a sophisticated Makefile template to use to compileCommunicator-compliant servers. Here’s a simplified version of the Parser server Makefile:

# Specify the name of the Makefile.

MAKEFILE = Parser.make

# Specify the location of the Communicator distribution.

ROOT_DIR = $(GC_HOME) MITRE_ROOTDIR = $(ROOT_DIR)/contrib/MITRE TEMPLATES = $(ROOT_DIR)/templates

# The file archos.make contains information about OS and platform.

include $(TEMPLATES)/archos.make

# The name of the server executable that you want to compile.

SERVER = Parser

61

# Put the executable in EXECDIR (if you don’t specify it, # the executable will go in the default Galaxy Communicator # bin directory, inside the distribution)

EXECDIR = bin/

# List all your source .c files.

SOURCES = Parser.c component_engine.c

# Include the Galaxy Communicator template.

include $(TEMPLATES)/rules.make

# Include automatically generated source dependency file.

ifneq ($(findstring clean,$(MAKECMDGOALS)),clean) include $(MAKEFILE).depend endif

The full set of configuration possibilities for the templates can be found in the full server reference. It’s also fairly straightforward to use your own Makefiles, but you’ll lose some of thebuilt-in transparency of managing multiple platforms that the Galaxy Communicator templates provide.

Step 6: Write your documentationFinally, we encourage you to document the message set your server supports. We’ve developeda fairly simple HTML template, which we exemplify using the Parser server.

SummaryIn this lesson, we’ve described the levels of complexity that servers can reach, and we’veexemplified a six-step process for constructing a server using the least complex type of server.In the next lesson, you’ll learn about a tool called the unit tester which we’ll use to understandbetter how all these types of servers work.

Next : Our second tool: the unit tester


62

How to Use the Unit Tester

The unit tester is a GUI tool which simulates either a server or a Hub. As a Hub, it connects to aserver and allows the user to send messages to the server and monitor responses just as theHub would. As a server, it connects to a listener in the Hub and gives the user the appropriatefacilities to interact with the Hub as a server would. In both case, the unit tester can send newmessages, or respond to new messages it receives.

Starting the unit tester as a Hub Interacting with the Parser server Starting the unit tester as a server Interacting with the Parser server via the Hub Summary

Once you feel comfortable with the unit tester, you can also consult the unit tester reference.

Starting the unit tester as a HubIn order to demonstrate the functionality of the unit tester, we’ll perform a simple interaction withthe Parser server from the toy travel demo, which we explored in detail in our server basics lesson. We’ll start with the simplest case: starting the unit tester as a Hub. We’ll use the processmonitor to control the interaction.

[Unit tester exercise 1]

% process_monitor $GC_HOME/tutorial/servers/parse-hub.config &

You’ll see a process monitor window with two panes, one of which starts the Parser and theother of which starts the unit tester. The command line for the unit tester looks like this:

[Unit tester hub command line]

$TOOLS_BIN/unit_tester --as_hub --frames$GC_HOME/tutorial/servers/parse.frames localhost 10000

The --as_hub argument configures the unit tester to pretend to be a Hub; the --framesargument provides a file containing a list of frames which the unit tester will be able to use asnew messages; and the final two arguments are the host and port of the server to contact.

63

Start the Parser server and then the unit tester. You’ll get a window like this:

As the unit tester sends and receives messages, it will update its scrollable output history.

Press the "Send new message" button. You’ll get a second window:

64

This window allows you to enter a new message to send to the Hub. You can either type in thenew message in the new message typein window using the printed representation of frames, oryou can select a frame from the mouseable frame history using the left mouse button. This framewill be entered in the new message typein window, and you can edit the frame or use it as is.

The "Reply required" radio button allows you to control whether the message you’re sendingrequires a reply or not. You’ll learn more about how to control this aspect of the message later.We’ll ignore the "Session ID" typein window and the "Lock Session" radio button for the moment.

65

Interacting with the Parser serverTo start your first interaction, select the frame named Parser.Parse with the left mouse buttonin the mouseable frame history, press the Reply Required button, and then press OK. You’ll seea couple of things happen.

First, the scrollable output history will now have the following contents:

[Scrollable output history]

[Sending: new message] {c Parser.Parse :input_string "I WANT TO FLY FROM BOSTON TO LOS ANGELES" } [Received: reply message] {c Parser.Parse :frame {c flight :origin "BOSTON" :destination "LOS ANGELES" } }

This history indicates that the unit tester sent the message described, and received a replycontaining the :frame key which contains the parse. We can see the same thing if we look atthe process monitor pane for the Parser:

[Parser process monitor pane]

GalSS_FrameHandler: input frame: {c Parser.Parse :input_string "I WANT TO FLY FROM BOSTON TO LOS ANGELES" :hub_opaque_data {c admin_info ... } } FnWithData: Parse _GalSS_FrameReturnHandler: result frame: {c Parser.Parse :frame {c flight ... } :hub_opaque_data {c admin_info ... } }

We’ve seen this sort of output before, in the server processing for the toy travel demo. In bothcases, we can see that the input and output correspond to the documentation for the Parser server.

Next, press the "Send new message" button again. Notice that the mouseable frame history nowhas the message reply in its history, along with the message it sent. Select the first Parser.Parse frame again, and delete Parser. from the frame name in the new messagetypein window (you can do this by selecting the text and pressing the <Del> key on yourkeyboard). The contents of the new message typein window should now look like this:

[New message typein window]

66

{c Parse :input_string "I WANT TO FLY FROM BOSTON TO LOS ANGELES" }

Now press the Reply Required button and then OK. The result is the identical, except that theframes are named Parse instead of Parser.Parse . This interaction demonstrates how theserver ignores the prefixes.

Finally, press the "Send new message" button again, and select the frame named UserInput(you may have to scroll back to the top of the mouseable frame history). You may recognize thisname as the name of the program which handles general input in the toy travel demo. Press theReply Required button and then OK. The result this time is very different. The message historynow contains a different type of response:


[Sending: new message] {c UserInput :input_string "I WANT TO FLY FROM BOSTON TO LOS ANGELES" } [Received: error message] {c system_error :err_description "Function UserInput does not exist" :errno 1 }

So instead of providing a normal response, the server responded with an error, because thename of the message the server received doesn’t correspond to any dispatch function it knowsabout. (We’ll learn more about error handling later.)

Select "File --> Quit" in the process monitor to end this exercise.

Starting the unit tester as a serverAs we said earlier, the unit tester can function as a server or as a Hub. When it functions as aserver, it contacts the Hub, rather than setting up a listener for the Hub to contact. In this secondexercise, we’ll use the unit tester as a server to interact with the Parser server via the Hub.

[Unit tester exercise 2]

% process_monitor $GC_HOME/tutorial/servers/parse-server.config &

You’ll see a process monitor with three panes: one for the Parser server, as before; one for theHub; and third for the unit tester, acting as a server. The command line for the unit tester shouldlook like this:

[Unit tester server command line]

$TOOLS_BIN/unit_tester --as_server --service_type UI --frames$GC_HOME/tutorial/servers/parse.frames localhost 14500

67

The --as_server argument instructs the unit tester to act as a server; the --service_typeargument tells the unit tester which service type the Hub has set up the listener for; the rest ofthe arguments are as before. Start the Parser, then the Hub, and finally the unit tester.

When you start the unit tester, you’ll get the toplevel window containing the scrollable outputhistory, as before, but you’ll also get a second window, which looks like this:

This window is the dialogue which allows you to respond to new messages which the unit testerreceives. The reason the unit tester is receiving a new message, of course, is because it’s justcontacted the Hub, and the Hub has sent it a reinitialize message. This window contains apane which displays the incoming message, an option to send a new message to the Hub, and aset of buttons to allow you to select the type of reply to send to this dispatch function. Since theresponse to the reinitialize message is ignored, select the "Dummy reply" button and pressOK. You’re now connected to the Hub.

Interacting with the Parser server via the HubWe’re going to try our three interactions again, but in a slightly different order.

Press the "Send new message" button and select the frame named UserInput . Although wewon’t look at it in detail until the next lesson, there’s a Hub program named UserInput definedin the Hub script we’re currently running, and this program calls the Parser.Parse operation.

68

Press "Reply Required" and then OK. The Hub pane in the process monitor should looksomething like this:

[Hub pane]

Got new message from provider for UI (id 2) M-M dialogue for provider for UI (id 2)

----------------[ 1]---------------------- {c UserInput :input_string "I WANT TO FLY FROM BOSTON TO LOS ANGELES" :session_id "Default" :tidx 1 } ----------------------------------------

------ find next op ---------------------- {c UserInput :input_string "I WANT TO FLY FROM BOSTON TO LOS ANGELES" :session_id "Default" :tidx 1 } --------------------------------------------



----------------[ 1]---------------------- {c UserInput :input_string "I WANT TO FLY FROM BOSTON TO LOS ANGELES" :session_id "Default" :tidx 1 :frame {c flight ... } } ----------------------------------------

done with token 1 --> returning to owner UI@<remote>:-1 [Destroying token 1]

We should already be familiar with a number of aspects of this interaction. We see that the Hubreceived a new message, which it used to instantiate a token with index 1. It found a programnamed UserInput (although this isn’t obvious from the printout), found the Parser.Parseoperation, dispatched the message, updated the token with the :frame key which was returned,and, after the program was complete, returned the result to the unit tester (which had contactedthe listener for the UI service type). This is a scripted interaction.

69

The scrollable output history in the unit tester shows the other side of this interaction:


[Sending: new message] {c UserInput :input_string "I WANT TO FLY FROM BOSTON TO LOS ANGELES" } [Received: reply message] {c UserInput :frame {c flight :origin "BOSTON" :destination "LOS ANGELES" } :session_id "Default" :input_string "I WANT TO FLY FROM BOSTON TO LOS ANGELES" }

Notice that the response here is a little different than in the case where the unit tester ispretending to be the Hub. In this case, we get the entire contents of the token back, whichcontains both the original :input_string key and also the :frame response.

Next, press "Send new message", and select the frame named Parser.Parse . In this case,there isn’t going to be a program named Parser.Parse , but there is a server named Parserwhich supports the Parse message. So in this case, we’ll have a scriptless interaction. Press"Reply Required" and then OK. In this case, we’ll have a slightly different output. First, the Hub pane:

[Hub pane]

Got new message from provider for UI (id 2) M-M dialogue for provider for UI (id 2)

----------------[ 2]---------------------- {c Parser.Parse :input_string "I WANT TO FLY FROM BOSTON TO LOS ANGELES" :session_id "Default" :tidx 2 } ----------------------------------------

---- serve(Parser@localhost:10000, token 2 op_name Parse (sole provider))


----------------[ 2]---------------------- {c Parser.Parse :frame {c flight ... } :session_id "Default" :tidx 2 } ----------------------------------------

70


The interaction is a little simpler. Notably, there’s no accumulated token state, because there’sno program matched. The state of the token is simply the response from the dispatch function,which contains :frame but not :input_string , as we saw in the first exercise. Again, thescrollable output pane confirms this:

[Scrollable output pane]

[Sending: new message] {c Parser.Parse :input_string "I WANT TO FLY FROM BOSTON TO LOS ANGELES" } [Received: reply message] {c Parser.Parse :frame {c flight :origin "BOSTON" :destination "LOS ANGELES" } :session_id "Default" }

The response received by the unit tester contains only the :frame key.

Finally, press "Send new message", and select the first Parser.Parse frame again, and delete Parser. from the frame name as you did in exercise 1. Press "Reply Required" and then OK.The result should be exactly the same as if you hadn’t removed the Parser. from the framename. The reason for this, you may recall, is that server name "addresses" are optional, and theHub will try to find a server which supports the named message. In this case, there’s only one(Parser), and the Hub uses it.

Now, select "File --> Quit" in the process monitor to end this exercise.

SummaryIn this lesson, you’ve learned about the unit tester, a powerful and flexible tool which allows us tointeract with the Hub and servers in very controlled ways. In the next lesson, we’ll start learning alittle bit about how Hub program files are constructed.

Next : Writing a basic Hub program file


71

Writing a Basic Hub Program File

In our hub-and-spoke configuration, we’ve seen that it’s the Hub which maintains connections toall the servers and routes the message traffic. We’ve already learned a great deal about howHub program files work, even though we haven’t looked yet at the technical details. In thislesson, we’ll learn the basics of how program files are constructed. We’ll learn about other Hubprogram functionality in future lessons.

A little bit about Hub program file syntax Global information and server declarations

PGM_SYNTAX: Atomic string values Service types and service providers SERVICE_TYPE: CLIENT_PORT: OPERATIONS: SERVICE_PROVIDER: SERVER: HOST: PORT: LOCATION:

Hub program directives PROGRAM: RULE: IN: OUT: Summary

Rules and namespaces Specifying the namespace explicitly Distinguishing between source and target namespaces Using $in for source and target Literal values Summary

Rule conditions Managing flow of control

The none! value

72

The destroy! value Summary

A little bit about Hub program file syntaxA line in a Hub program file can be

a directive entry , which consists of a directive (like PROGRAM: or RULE:) and its value; a comment, which is indicated by a semicolon (;) in column 0; or a blank line, which has no meaning except to terminate certain directive entry blocks like RULE:.

If the final character of a physical line in a directive entry is a backslash (\), the line break isignored and the next line is treated as part of the same directive entry.

There are three types of information in the program file:

global information server declarations programs

There are lots of directives, and each directive expects a value of a certain type or format. We’reonly going to look at a few of the directives and values. You can consult the complete documentation if you want to know more.

73

Global information and server declarationsAs a first example, we’ll begin with the program file we used in the second unit tester exercise. Inthat exercise, you’ll recall, there were two servers: the Parser server from the toy travel demo,and the unit tester, acting as a server. The Hub contacted the Parser server, while the unit testercontacted the Hub. We discussed this process in our lesson on how the Hub and servers communicate. Here’s the relevant fragment of the program file parse.pgm :

[parse.pgm]

1: ;; Use extended syntax (new in version 3.0). 2: 3: PGM_SYNTAX: extended 4: 5: SERVICE_TYPE: UI 6: CLIENT_PORT: 14500 7: 8: SERVER: Parser 8: HOST: localhost 10: PORT: 10000 11: OPERATIONS: Parse

Let’s look first at the syntax of this fragment.

Line 1 starts with a semicolon, and so it is a comment. Line 2 is blank. It is ignored, because there’s no block to terminate. Line 3 is a directive entry. The directive is PGM_SYNTAX:, and the value is "extended". Line 4 is blank. It is ignored, because there’s no block to terminate. Lines 5-6 are a directive entry block. The first directive, SERVICE_TYPE:, determines whattype of block it is. Line 7 is blank. It terminates the directive entry block. Lines 8-11 are a directive entry block. The first directive, SERVER: determines what type ofblock it is.

Now that we know what we’re looking at, let’s take a closer look at the directive entries.

PGM_SYNTAX:In version 3.0 of the Galaxy Communicator infrastructure, we extended the syntax of the Hubprogram file and made it more consistent overall. However, since these modifications were notbackward compatible, we chose to enable them through an explicit directive entry. Alwaysbegin your program file with this entry .

74

Atomic string valuesThe value of the PGM_SYNTAX: directive, as well as a number of the other directives we’lldiscuss here, is a string. Typically, values for directives have the same printed form as they do in frames. So we’d expect line 3 to look like this:

PGM_SYNTAX: "extended"

However, for readability purposes, the Hub program file parser also recognizes these values(and sequences of these values) without their delimiting quotation marks. This is the lastinterpretation considered; so, for instance, a directive value which is a number will be interpretedas such, rather than a string consisting entirely of digit characters.

Service types and service providersYou may recall from our lesson on the toy travel demo that the Hub distinguishes between service types and service providers . The idea is that the Hub might want or need access tomultiple servers of the same conceptual type: multiple recognizers for load balancing, multipleaudio servers for multiple simultaneous users. The conceptual type corresponds to the servicetype. Service types have a name and a set of operations; the actual servers which implementthese operations are the service providers.

Each of these is defined by a separate block, terminated by a blank line (or the first directiveentry of another block of the same type). The two directive entry blocks in this Hub program filefragment illustrate two dimensions of this distinction.

SERVICE_TYPE:Each service type has a name, which is the value of the SERVICE_TYPE: directive. A blockwhich begins with the SERVICE_TYPE: directive may also contain an OPERATIONS: directiveand a CLIENT_PORT: directive, among others. The value of this directive is a string.

CLIENT_PORT:A service type may set up a listener to await connections from providers for this service type.The CLIENT_PORT: directive is a number which specifies the port to set up the listener on. Soin this fragment, the UI service type will set up a listener on port 14500.

OPERATIONS:Each service type may declare a set of operations, which are dispatch functions it expects theproviders to implement. The Hub will choose a service provider to use in a rule only if the Hubprogram file has declared that the corresponding service type supports the operation, via theOPERATIONS: directive. The value of this directive is a sequence of strings. In this fragment,the UI service type declares no operations (so the only messages it will receive from the Hub arereplies to messages it sends).

75

SERVICE_PROVIDER:Each service provider block corresponds to a service provider the Hub will contact. The providerimplements a service type (sometimes more than one); the value of the SERVICE_PROVIDER:directive is the name of the service type(s) it implements. This block may also contain a HOST:directive, a PORT: directive, and a LOCATION: directive, among others.

Since each service type defines a set of supported operations, all providers for a given servicetype must support the same operations (i.e., define the appropriately named dispatch function).So in the following example, both providers must define the Parse dispatch function:

SERVICE_TYPE: Parser OPERATIONS: Parse

SERVICE_PROVIDER: Parser HOST: localhost PORT: 15000


Multiple service types can define the same operation. So, for instance, you might choose todistinguish between parsers for French and parsers for Chinese by classifying them as differentservice types (there are better ways to do this, but let’s assume it for the moment). They canboth define the Parse operation, and any providers for these types must support that operation:

SERVICE_TYPE: ChineseParser OPERATIONS: Parse

SERVICE_TYPE: FrenchParser OPERATIONS: Parse

SERVER:The SERVER: block is almost identical to the SERVICE_TYPE: block, except it may also containthe locations of various service providers for that type. In other words, lines 8-11 of this fragmentare equivalent to

SERVICE_TYPE: Parser OPERATIONS: Parse


76

HOST:Each service provider needs to specify the location of the listener that the Hub will contact. Thevalue of the HOST: directive is a string naming the machine where the service provider’s listener is.

PORT:The value of the PORT: directive is an integer corresponding to the port number that the serviceprovider’s listener is listening on. So in this fragment, the Hub expects to find a service providerfor the Parser service type listening on port 10000 on the local host.

LOCATION:A shorthand for specifying HOST: and PORT: simultaneously. The value of this directive is astring <host>:<port> .

So this fragment declares two service types, UI and Parser. It tells the Hub to set up a listenerfor the UI service type, and to contact a service provider for the Parser service type at the localhost, port 10000. Here’s an illustration:

Hub program directivesFor a quick introduction to programs in Hub program files, let’s consider the program in parse.pgm :

PROGRAM: UserInput

RULE: :input_string --> Parser.Parse IN: :input_string OUT: :frame

In many ways, this is about a simple as programs ever get. It corresponds approximately to asmall subsection of the toy travel demo.

77

PROGRAM:You should remember from the basics lesson that when the Hub receives a new message, ittries to match the name of the incoming frame with the name of a Hub program. This directiveentry begins a Hub program, and the value of the directive is a string which is the name of theprogram. Programs are terminated by the end of the Hub program file or by another PROGRAM:directive entry.

RULE:Each Hub program is a sequence of rules. Each rule is a directive entry block terminated by ablank line. The RULE: directive entry indicates the beginning of a rule. This block may alsocontain IN: and OUT: directive entries, among others. The value of the RULE: directive is fairlycomplicated; it consists of a (possibly empty) set of conditions, an implication arrow (--> ), andan operation name, which can be specified either as <operation> or <service_type>.<operation> .

78

The set of conditions is evaluated against the state of the current token. The condition here is asimple atomic condition requiring the presence of the :input_string key (with any value) inthe frame comprising the token state. If this condition is satisfied, the Hub fires the rule, whichmeans that the Hub finds a service provider for a service type which implements the namedoperation (in this case, Parse ) and sends the provider an appropriate message. If the servicetype is specified in the operation name (in this case, Parser ), only providers for the namedservice type will be considered.

IN:When the Hub decides to fire a rule, it must construct a message to send. The name of themessage will be the name of the operation (in this case, Parser.Parse ). The value of the IN:directive describes how to construct the message given the current token. In this example, theinstruction :input_string means that the Hub will look for a key-value pair in the token whosekey is :input_string , and, if present, copy it into the message.

OUT:When the server has finished executing the operation, it typically returns the result to the Hub.The value of the OUT: directive describes how to update the current token given the messagereturn. In this example, the :frame instruction means that the Hub will look for a key-value pairin the message return whose key is :frame , and, if present, copy it into the current token.

SummaryHere’s an illustration of the overall process:

79

Rules and namespacesIn order to understand the possible complexity of rules (and other program file directives that wewon’t be discussing here), it’s important to understand the notion of namespace . We’ve seenthat the Hub uses frames to maintain at least two distinct types of information states: tokens andmessages. These types are interrelated:

incoming messages constitute the initial state of tokens the token state is used to construct new messages to be sent to servers (the IN: directive) the token state is updated based on message returns from servers (the OUT: directive)

We’ll refer to these types of information states as namespaces . In addition to the message andtoken namespaces, there are two others which we’ll encounter:

the session namespace, which is the set of key-value pairs specific to the current user’sdialogue with the system (more on this later) the global namespace, which is a globally-accessible set of key-value pairs

For more on namespaces, see the program file reference.

Each directive in a rule directive entry block has a namespace or namespaces associated with it.The RULE: directive is associated with the token namespace, for instance. The IN: and OUT:directives have both a source namespace (the memory state from which the pairs are drawn)and a target namespace (the memory state which is updated). The IN: directive’s sourcenamespace is the token namespace (that is, that’s where the values come from), and its target

80

namespace is the message namespace (that is, that’s where the values go). For OUT:, it’s theother way around.

For this discussion, let’s assume a situation where our new UserInput message arrives at theHub in a context where the global namespace contains the key-value pair :fragments_permitted 1 and the session namespace contains the key-value pair :language "English" :

Specifying the namespace explicitlyIt’s possible to specify a namespace explicitly, using an operator named $in . You can use thisoperator to override the namespace defaults associated with a directive. So, for instance, thefollowing two RULE: directive entries are equivalent, since the default namespace associatedwith the RULE: directive is the token namespace:

RULE: :input_string --> Parser.Parse

RULE: $in(:input_string token) --> Parser.Parse

81

If we want to fire a rule only if the global namespace contains a key-value pair whose key is :fragments_permitted , as in our example, we can write this rule as follows:

RULE: $in(:fragments_permitted global) --> Parser.Parse

So essentially, a frame key in a directive is a shorthand for a reference to that key in the default namespace.

Distinguishing between source and target namespacesFor those directives like IN: and OUT: which have both source and target namespaces, it’spossible to distinguish between the source and target using a list of length two, where the firstelement is the target and the second element is the source. So the following two directives are equivalent:

IN: :input_string IN: (:input_string :input_string)

In both cases, the directive looks for a key-value pair with the key :input_string in thesource namespace and inserts the value in the target namespace under the key :input_string .

As you may have guessed at this point, you can use this syntax to refer to different keys in thesource and target. So let’s suppose that the Parse dispatch function expects the key :string ,rather than :input_string , and returns the result in a key :parse, rather than :frame . If wewish to preserve the token state shown previously, we could rewrite our rule as follows:

RULE: :input_string --> Parser.Parse IN: (:string :input_string) OUT: (:frame :parse)

The value of :input_string in the source namespace for IN: (namely, the token) will becopied to :string in the target namespace (namely, the message), and the value of :parse inthe source namespace for OUT: (namely, the message return) will be copied to :frame in thetarget namespace (namely, the token). Here’s an illustration:

82

Using $in for source and targetUnsurprisingly, you can put these two things together, by overriding either the source or thetarget in the mapping list. Let’s say we want to copy the value of the :language key in thesession namespace to the :lang key in the message. The IN: directive entry would look like this:

IN: (:lang $in(:language session))

If the $in operator isn’t part of a pair in IN: or OUT:, it’s treated as the source key andnamespace. The key is also used as the key for the target namespace, and the targetnamespace is the default. So the following directive entries are equivalent:

IN $in(:language session) IN: (:language $in(:language session)) IN: ($in(:language message) $in(:language session))

Literal valuesThe pair notation for IN: and OUT: can also be used to specify literal values. These literal valuesappear in the source position. If, for example, we want to insert a literal confidence threshold inthe message sent to the parser, we can do it as follows:

83

IN: (:threshold .8)

These literal values can be any frame key value, just like the values of directives, and the keyscan be explicit namespace references.

SummaryThis illustration unifies all these principles.

Rule conditionsSo far, the only rule condition we’ve encountered is an existence condition, i.e., whether or notthere’s a key-value pair for the given key in the appropriate namespace:


The existence condition can be negated:

RULE: !:parse_completed --> Parser.Parse

The basic conditions also include numeric comparisons (>, <, =) and string equality (=) andtheir negations:

84

RULE: :threshold > .8 --> Parser.Parse RULE: :input_string != "ignore" --> Parser.Parse

These conditions can be recursively combined using conjunction (&) and disjunction (|):

RULE: :input_string & (:threshold > .8) --> Parser.Parse RULE: (:threshold > .8) | (:input_string != "ignore") --> Parser.Parse

The program file reference discusses rule conditions in considerably more detail.

Let’s do a simple example to illustrate what happens when conditions don’t match. Rememberour simple program:

PROGRAM: UserInput


Now let’s send a message that doesn’t match.

[Hub program exercise 1]

% process_monitor $GC_HOME/tutorial/program_file/nomatch.config

Start the Parser, then the Hub, then finally the unit tester. (When you start the unit tester server,you won’t be asked for a reply for the reinitialize message, as you were in the unit tester tutorial. This is because the unit tester is being started with the --ignore_reinitialize flag.See the unit tester reference for more details.) Select "Send new message", select the framenamed UserInput , press "Reply Required" and then OK. You’ll see the following in the Hub pane:

[Hub pane]

------ find next op ---------------------- {c UserInput :string "I WANT TO FLY FROM BOSTON TO LOS ANGELES" :session_id "Default" :tidx 1 } --------------------------------------------


And the unit tester interaction history will show that the result is almost identical to what was sent:

85

[Interaction History pane]

[Sending: new message] {c UserInput :string "I WANT TO FLY FROM BOSTON TO LOS ANGELES" } [Received: reply message] {c UserInput :string "I WANT TO FLY FROM BOSTON TO LOS ANGELES" :session_id "Default" }

It should be clear what has happened. The UserInput program has a single rule, which fires ifthe token contains the :input_string key. The message we sent contains the :string key,but not the :input_string key, and so the token which is instantiated from the new messagedoesn’t contain the :input_string key. As a result, the rule in the program doesn’t fire, whichmeans that the Parser server isn’t invoked, and there are no updates to the token state. Whenthe program ends (trivially, since it didn’t fire the only rule), it returns the token state to the unittester, since it asked for a reply.

Select "File --> Quit" to end this exercise.

Managing flow of controlTypically, the Hub evaluates a Hub program by considering each rule in turn. When a rulecondition is satisfied, the Hub fires the rule and waits for the response from the server. When theHub receives the response, it resumes the process at the rule immediately after the rule it fired,until it reaches the end of the program.

However, it’s possible both to ignore the response from the server and to terminate the programbefore the final rule is considered. The simplest way to do this is through two special values ofthe OUT: directive, none! and destroy! .

The none! valueIf the value of OUT: is none! , then the Hub will not wait for the server to respond, and willimmediately move on to consider the next rule. If the server does respond, the response will beignored. In essence, then, this rule and any subsequent rules which match will be firedsimultaneously. We’ve already seen an example of this behavior.

You may recall from the toy travel demo that the input string was printed out by the IOMonitorserver immediately before it was passed to the Parser server. The corresponding program filefragment looks like this:

PROGRAM: UserInput

RULE: :input_string --> IOMonitor.ReportIO IN: (:utterance :input_string) (:who "user") OUT: none!

86


By now, we can read the IN: directive entry for the first rule. The value for the :input_stringkey in the source (token) namespace is copied to the :utterance key in the target (message)namespace, and the :who key in the target namespace is given the literal string value "user" .The value OUT: directive informs us that the response from the server will be ignored. So if thetoken has an :input_string key-value pair, both these rules will be fired at the same time.

We can see this behavior in action in the following exercise:

[Hub program exercise 2]

% process_monitor $GC_HOME/tutorial/program_file/none.config

You’ll see a process monitor with four panes. Start the Parser, IOMonitor, Hub and finally theunit tester server. Select "Send new message", and select the frame named UserInput , asbefore. Press "Reply required", and then OK.

Now take a look at the Hub pane. You should see something like this:

[Hub pane]

------ find next op ---------------------- {c UserInput :input_string "I WANT TO FLY FROM BOSTON TO LOS ANGELES" :session_id "Default" :tidx 1 } --------------------------------------------

found operation: IOMonitor.ReportIO found operation: Parser.Parse ---- serve(IOMonitor@localhost:10050, token 1 op_name ReportIO (sole provider)) ---- serve(Parser@localhost:10000, token 1 op_name Parse (sole provider))


----------------[ 1]---------------------- {c UserInput :input_string "I WANT TO FLY FROM BOSTON TO LOS ANGELES" :session_id "Default" :tidx 1 :frame {c flight ... } } ----------------------------------------

87


You can see that the two operations in the program matched essentially at the same time, andwere sent to their respective servers at essentially the same time. The Parser server provided aresponse, and the token state is updated as before.


The destroy! valueThe destroy! value also tells the Hub to ignore the response from the server, but it also tellsthe Hub to terminate executing the program. For instance, we might want to notify the IOMonitorand abort the execution of the UserInput program if the :input_string key was not present:

PROGRAM: UserInput

RULE: !:input_string --> IOMonitor.ReportIO IN: (:utterance "<no input string found>") (:who "user") OUT: destroy!



If there’s no operation you want to fire, there are other ways of destroying the token; we’ll talkabout these when we talk about the Builtin server and building end-to-end systems.

The program file reference has more details about flow of control.

SummaryIn this lesson, we learned about the following Hub program directives:

PGM_SYNTAX: SERVICE_TYPE: CLIENT_PORT: OPERATIONS: SERVICE_PROVIDER: SERVER: HOST: PORT:

88

LOCATION: PROGRAM: RULE: IN: OUT:

Now that we understand the basics of writing both servers and Hub program files, we can moveon to study functionality which involves both additional server capabilities and correspondingfunctionality in the Hub program file.

Next : Error handling


89

Error Handling

So far, in almost all the examples we’ve run, everything has worked as expected. However,that’s a very unrealistic assumption. Almost every programming language and distributedprocessing infrastructure has some facility for error handling, and the Galaxy Communicatorinfrastructure does as well. In this lesson, we’ll learn how errors are handled.

Environments and message returns How to raise an error How to catch an error

"Passive" error handling "Active" error handling

Summary

Environments and message returnsUp to this point, we’ve ignored the second argument of each dispatch function:

Gal_Frame Parse(Gal_Frame frame, void *server_data );

This argument, although represented as a void * for historical reasons, is actually a pointer toa call environment structure, of type GalSS_Environment. The call environment is theappropriate conduit for all communication with the Hub, since it embodies a good deal ofcontextual information which the communication requires, such as the current session, theconnection through which to communicate, and the Hub program and rule which caused thedispatch function to be called.

The environment also keeps track of whether the Hub expects a response (which we saw wecould control via Hub programs), and whether a response has been provided. So far, the onlyway we’ve seen to provide a response is by returning a frame from the dispatch function. Youcan also provide a normal message response using the function GalSS_EnvReply, which takesa call environment as an argument. So the Parse dispatch function we studied in the serverbasics lesson could have also been written as follows:

Gal_Frame Parse(Gal_Frame frame, void *server_data) { /* Message decomposition */ char *input_string = Gal_GetString(frame, ":input_string");

90

ParseTree *p; Gal_Frame reply;

/* Core processing */ p = ParseSentence(input_string);

/* Reply construction */ reply = Gal_MakeFrame("reply", GAL_CLAUSE); Gal_SetProp(reply, ":frame", Gal_FrameObject(ParseTreeToFrame(p))); GalSS_EnvReply((GalSS_Environment *) server_data, reply); return (Gal_Frame) NULL; }

Note that after we send the reply, we return NULL from the dispatch function. This isn’t strictlynecessary, because call environments will only send one response back to the Hub; any framereturned from this dispatch function would be ignored, since the response has already been sent.

In most circumstances, there’s no reason to use GalSS_EnvReply. However, it serves as aconvenient introduction to call environments and explicit message returns. In particular, inaddition to normal returns, we can also return errors. We’ll explore this in the next section.

How to raise an errorWe’ve already seen a simple error. When we interacted with the Parser server using the unittester, we tried sending the server a frame named UserInput . Since the Parser server doesn’tknow of any dispatch function named UserInput , it responded with an error, as we saw in theoutput history of the unit tester:


[Sending: new message] {c UserInput :input_string "I WANT TO FLY FROM BOSTON TO LOS ANGELES" } [Received: error message] {c system_error :err_description "Function UserInput does not exist" :errno 1 }

So one way to raise an error is to send a server a message it doesn’t understand.

All errors in the Galaxy Communicator infrastructure originate in servers. In particular, errors areparticular types of message responses. Most of these errors are signalled by the programmerfrom a dispatch function using the function GalSS_EnvError. This function, likeGalSS_EnvReply, sends an explicit message return via a call environment.

91

So what sorts of errors might you want to report? You may recall that, for instance, wepostponed the discussion of type checking when we discussed server basics; this was becausein order to signal a type checking error, we needed to talk about environments first. So that’s onetype of error. In fact, we might encounter errors at any point in the dispatch function processing:

The :input_string key might not be present in the input, or its value might not be astring (type checking) The input string might fail to parse (core processing) It might not be possible to convert the result into a frame (reply construction)

Let’s look at the "real" version of the Parse dispatch function, with error handling revealed:

Gal_Frame Parse(Gal_Frame frame, void *server_data) { char *input_string; ParseTree *p; Gal_Frame reply; Gal_Frame parse_frame;

/* Deconstruction and type checking */ input_string = Gal_GetString(frame, ":input_string"); if (!input_string) { GalSS_EnvError((GalSS_Environment *) server_data, "no input string"); return (Gal_Frame) NULL; }

/* Core processing */ p = ParseSentence(input_string);

/* Reply construction */ if (!p) { GalSS_EnvError((GalSS_Environment *) server_data, "no parse"); return (Gal_Frame) NULL; } parse_frame = ParseTreeToFrame(p); if (!parse_frame) { GalSS_EnvError((GalSS_Environment *) server_data, "can’t convert parse to frame"); return (Gal_Frame) NULL; } reply = Gal_MakeFrame("reply", GAL_CLAUSE); Gal_SetProp(reply, ":frame", Gal_FrameObject(parse_frame)); return reply; }

92

Now let’s use the process monitor to see some of these errors in action.

[Error handling exercise 1]

% process_monitor $GC_HOME/tutorial/errors/hub-error.config

Start the Parser and then the unit tester. Select "Send new message" in the unit tester window,select the first frame (the one with the :string key), press "Reply Required", and then pressOK. You should see the following exchange in the interaction history pane:


[Sending: new message] {c Parser.Parse :string "I WANT TO FLY FROM BOSTON TO LOS ANGELES" } [Received: error message] {c system_error :err_description "no input string" :errno 0 }

(If you’re adventurous, you’ll discover that when the unit tester is acting as a Hub, you’ll get areply most of the time, whether or not you select "Reply Required". This is because in thecurrent implementation, the server library forces a reply if a reply is required, but doesn’t suppress a reply if a reply isn’t required. This may change in a future release. If the unit testeris acting as a server, it will not get a reply if a reply isn’t required.)

Note that the error you get corresponds to the absence of an :input_string key. Next, select"Send new message" again, and select the second frame (the one with the float value for :input_string ), press "Reply Required", and then OK. You’ll now see the following exchange:


[Sending: new message] {c Parser.Parse :input_string 5.600000e+00 } [Received: error message] {c system_error :err_description "no input string" :errno 0 }

The error is the same; this is because the :input_string key still doesn’t contain a string.You should be able to match the error description to the argument to GalSS_EnvError, as follows:

93

Finally, select "Send new message", select the third frame, press "Reply Required", and thenOK. You’ll now see the following exchange:


[Sending: new message] {c Parser.Parse :input_string "I WANT TO FLY TO LOS ANGELES" } [Received: error message] {c system_error :err_description "no parse" :errno 0 }

This error corresponds to the parse failure.

Select "File --> Quit" in the process monitor to conclude this example.

How to catch an errorAt this point, it should be clear how errors are raised. Now, we’re going to look at how to catchthose errors.

94

"Passive" error handlingIn normal circumstances, when the Hub sends a message to a server which results in an errorbeing raised, the Hub treats the error as terminating the current program and token, and returnsthe error to the originating server if a reply was required. An example will illustrate.


% process_monitor $GC_HOME/tutorial/errors/server-error.config

Start the Parser, then the Hub, and finally the unit tester. The program we’re running is the samesimple program we used in the Hub basics lesson:

PROGRAM: UserInput


We’re going to try two of the frames we tried in the last exercise. Select "Send new message",select the first frame, press "Reply required" and then OK. You’ll see the following in the Hub pane:

[Hub pane]

------ find next op ---------------------- {c UserInput :input_string 5.600000e+00 :session_id "Default" :tidx 1 } --------------------------------------------


Got error from provider for Parser @ localhost:10000: token 1 {c system_error :errno 0 :err_description "no input string" :session_id "Default" :tidx 1 } done with token 1 --> returning to owner UI@<remote>:-1 [Destroying token 1]

So the Hub recognizes that the Parser server returned an error. It terminates the program (whichwas about to terminate anyway) and instead of sending the token state back to the unit tester asthe reply, it sends the error:

95


[Sending: new message] {c UserInput :input_string 5.600000e+00 } [Received: error message] {c system_error :session_id "Default" :err_description "no input string" :errno 0 }

Here’s an illustration of the process:

If you press "Send new message" again, select the second frame, press "Reply required" andthen OK, you’ll see the analogous behavior with the "no parse" error description as a result.


96

"Active" error handlingIt’s also possible to explicitly catch an error, using the ERROR: directive in the program file. Solet’s use a slightly different program:

PROGRAM: UserInput

RULE: :input_string --> Parser.Parse IN: :input_string OUT: :frame ERROR: (:encountered_error 1) :err_description

The interpretation of OUT: and ERROR: are almost identical: they have the same source(message) and target (token) namespaces, and they’re processed after the message return. Thedifference is that OUT: is evaluated when there’s a normal reply, and ERROR: is evaluatedwhen there’s an error reply. If ERROR: is present, it also serves to "catch" the error; the programdoesn’t terminate, but rather proceeds normally. We expect, then, that if we redo the lastexercise, the unit tester will receive a normal message reply. So let’s check that.


% process_monitor $GC_HOME/tutorial/errors/catch-server-error.config

As before, start the Parser, Hub and unit tester. Select "Send new message", select the firstframe, press "Reply required" and then OK. You’ll see the following in the Hub pane:

[Hub pane]

------ find next op ---------------------- {c UserInput :input_string 5.600000e+00 :session_id "Default" :tidx 1 } --------------------------------------------


Got error from provider for Parser @ localhost:10000: token 1 {c system_error :errno 0 :err_description "no input string" :session_id "Default" :tidx 1 }

97

----------------[ 1]---------------------- {c UserInput :input_string 5.600000e+00 :session_id "Default" :tidx 1 :encountered_error 1 :err_description "no input string" } ----------------------------------------


So in this case, the Hub printed the resulting token state (which it does at every normal step in aHub program). The token state is returned to the unit tester:


[Sending: new message] {c UserInput :input_string 5.600000e+00 } [Received: reply message] {c UserInput :encountered_error 1 :session_id "Default" :err_description "no input string" :input_string 5.600000e+00 }

Note that the unit tester states that it received a normal reply, rather than an error. Note also thatas in other normal replies, the name of the reply frame is identical to the original message sent,and the original contents of the message are preserved in the message reply. Finally, theERROR: directive entry has updated the token state with a setting for :encountered_errorand for :err_description . Here’s an illustration of the process:

98

If you’re curious, you can confirm this behavior by sending a second new message.


SummaryIn this lesson, you’ve learned about the following concepts and components:

Call environment structures Error replies The functions GalSS_EnvReply and GalSS_EnvError The ERROR: program file directive

Up to this point, we’ve been able to send new messages to servers either from the Hub or byusing the unit tester. However, we don’t know yet how to program dispatch functions to sendnew messages to the Hub. In the next lesson, we’ll learn how to do this.

Next : Sending new messages to the Hub

99


100

Sending New Messages to the Hub

Up to this point, we’ve studied only the simplest type of servers, which we exemplified using theParser server. As we described in the server basics lesson, the Generator and Backend also areno more complex: all three servers accept messages and return a response. In this lesson, we’regoing to look at a slighly more complex server, one which doesn’t return any responses, butrather sends new messages (and sometimes waits for their replies). This server is the Dialogue server.

An overview of the Dialogue server Asynchronous messages Synchronous messages

From the unit tester From the Dialogue server

Putting it all together Summary

An overview of the Dialogue serverThe Dialogue server supports two dispatch functions. The first is DoGreeting , which is calledwhen a user connects, and sends a greeting to the user. The second is DoDialogue , whichdecides what to do with each user input. In some cases, the Dialogue server will contact theBackend server to retrieve data from the database, and present the result to the user; in othercases, the Dialogue server might decide that it needs more information from the user before itcontinues. The full toy travel demo contains a small dialogue which exemplifies all three of these outputs:

# Greeting System: Welcome to Communicator. How may I help you? User: I WANT TO FLY TO LOS ANGELES # Additional information System: Where are you traveling from? User: BOSTON # Database query and response presentation System: American Airlines flight 115 leaves at 11:44 AM, and Unitedflight 436 leaves at 2:05 PM

101

The way the Dialogue server is implemented, all of these interactions involve sending a newmessage to the Hub. In addition, the database query involves sending a new message to theHub and waiting for the response. We’ll examine each of these elements in turn.

Asynchronous messagesThe first message type we’ll look at is asynchronous; that is, we don’t ask for a response, andwe don’t wait for one. We’ll choose the DoGreeting dispatch function to illustrate this.

Gal_Frame DoGreeting(Gal_Frame frame, void *server_data) { char *s = "{c FromDialogue :output_frame {c greeting }:is_greeting 1 }"; Gal_Frame greeting = Gal_ReadFrameFromString(s);

GalSS_EnvWriteFrame((GalSS_Environment *) server_data, greeting, 0); Gal_FreeFrame(greeting); return (Gal_Frame) NULL; }

The function we use to write new frames is GalSS_EnvWriteFrame. This function takes threearguments: a call environment, a frame to use as the message, and a flag indicating whether towait for the frame to be written before returning. So when the Dialogue server receives a framenamed DoGreeting , it constructs a frame named FromDialogue , writes it to the Hub, freesthe frame, and returns NULL, which means that no frame will be returned (unless a reply isrequired, in which case a dummy will be sent). We can illustrate this with an example.

[New messages exercise 1]

% process_monitor $GC_HOME/tutorial/messages/greeting.config

In this example, the unit tester will pretend to be the Hub. Start the Dialogue server and then theunit tester. Select "Send new message", choose the DoGreeting frame, and press OK. Theinteraction history will show a new message:


[Sending: new message] {c DoGreeting } [Received: new message] {c FromDialogue :output_frame {c greeting } :is_greeting 1 }

102

Note that we neither asked for a reply to the message we sent, nor did we receive one.

(If you’re curious, you can repeat this exercise with "Reply required" selected, and you can seehow the unit tester receives a dummy reply in addition to the new message.)


Synchronous messagesThe process through which a server sends a message to the Hub and gets a reply is a little morecomplicated, although we’ve actually seen it happen a number of times; every time we’ve usedthe unit tester as a server and sent a message which requires a reply, we’ve done exactly thesame thing that any other server does in the same circumstances. One such circumstance, aswe’ve already described, is when the Dialogue server retrieves information from the database.The Dialogue server sends a new message named DBQuery, which invokes the correspondingHub program:

PROGRAM: DBQuery

RULE: :sql_query --> Backend.Retrieve IN: :sql_query OUT: :column_names :nfound :values

When the program is done, the result is returned to the originating server. This interactioncorresponds to the following small subsection of the toy travel demo:

103

From the unit testerBefore we look at the Dialogue server itself, let’s try invoking this behavior from the unit tester.


% process_monitor $GC_HOME/tutorial/messages/backend.config

The unit tester will be functioning as a server in this example. Start the Backend server, the Hub,then the unit tester. Select "Send new message", select the first frame named DBQuery , press"Reply required" and then OK. You should see something like this in the unit tester interaction history:


104

[Sending: new message] {c DBQuery :sql_query "select airline, flight_number, departure_datetimefrom flight_table where departure_aiport = ’BOS’ and arrival_airport =’LAX’" } [Received: reply message] {c DBQuery :nfound 2 :values ( ( "AA" "115" "1144" ) ( "UA" "436" "1405" ) ) :sql_query "select airline, flight_number, departure_datetimefrom flight_table where departure_aiport = ’BOS’ and arrival_airport = ’LAX’" :column_names ( "airline" "flight_number" "departure_datetime" ) :session_id "Default" }

If you look at the Hub pane, you’ll see that the new message matched the appropriate program,fired the rule to invoke the Backend server, and then terminated normally, returning the tokenstate to the originating server (the unit tester):

[Hub pane]

------ find next op ---------------------- {c DBQuery :sql_query "select airline, flight_number, departure_datetime ..." :session_id "Default" :tidx 1 } --------------------------------------------

found operation: Backend.Retrieve ---- serve(Backend@localhost:13000, token 1 op_name Retrieve (sole provider))

Got reply from provider for Backend @ localhost:13000 : token 1

----------------[ 1]---------------------- {c DBQuery :sql_query "select airline, flight_number, departure_datetime ..." :session_id "Default" :tidx 1

105

:column_names ( "airline" ... ) :nfound 2 :values ( ( "AA" ... ) ... ) } ----------------------------------------


Formally, this interaction is identical to our previous interactions with the Parser server: the unittester sends a new message to the Hub, the appropriate program is matched, the rules are firedand the token state updated, and the token state returned when the program ends.

If the Backend server raises an error, of course, the error will be forwarded to the originatingserver, as we saw in the last lesson. (Notice that there’s no ERROR: directive in the Hubprogram rule, so the program won’t continue.) We can see this again if we select "Send newmessage", select the second frame named DBQuery (the one with arrival_airport = ’SFO’ ), press "Reply required" and then OK. This time, the uniit tester interaction history willshow an error reply:


[Sending: new message] {c DBQuery :sql_query "select airline, flight_number, departure_datetimefrom flight_table where departure_aiport = ’BOS’ and arrival_airport= ’SFO’" } [Received: error message] {c system_error :session_id "Default" :err_description "no DB result" :errno 0 }

Select "File --> Quit" to end this exercise

From the Dialogue serverNow, let’s take a look at the Dialogue server itself, to see how it does the same thing. This is avastly simplified version of the actual dispatch function:

Gal_Frame DoDialogue(Gal_Frame f, void *server_data) { Gal_Frame msg_frame = Gal_MakeFrame("DBQuery", GAL_CLAUSE); Gal_Frame r_f; GalIO_MsgType reply_type; char *sql_query;

106

GalSS_Environment *env = (GalSS_Environment *) server_data;

/* ... */

Gal_SetProp(msg_frame, ":sql_query", Gal_StringObject(sql_query); response_frame = GalSS_EnvDispatchFrame(env, msg_frame, &reply_type); Gal_FreeFrame(msg_frame); switch (reply_type) { case GAL_ERROR_MSG_TYPE: /* Relay error */ /* ... */ break; case GAL_REPLY_MSG_TYPE: /* Construct presentation of database response */ /* ... */ } }

Let’s look at this a little more closely:

107

As before, we construct the message. This time, we use the function GalSS_EnvDispatchFrame,which takes three arguments: the call environment and new message, as before, and also apointer to a message type variable. The message we receive will be either a normal reply (GAL_REPLY_MSG_TYPE) or an error reply (GAL_ERROR_MSG_TYPE), and the final argumentcaptures this information. The frame which this function returns is the final token state of the Hubprogram invoked by the new message, or an error which the Hub is returning.

So when the Dialogue server receives a frame which causes it to consult the Backend server,this is how it gets the result from the Backend server via the Hub.

In this next example, the unit tester is going to act as a server. We’re going to use the followingprogram file:

;; Use extended syntax (new in version 3.0).

PGM_SYNTAX: extended

SERVICE_TYPE: UI CLIENT_PORT: 14500 OPERATIONS: FromDialogue

SERVER: Dialogue HOST: localhost PORT: 18500 OPERATIONS: DoDialogue DoGreeting

108

SERVER: Backend HOST: localhost PORT: 13000 OPERATIONS: Retrieve

PROGRAM: DBQuery


We’re going to send a message named DoDialogue , and the Dialogue server will respond witha message named FromDialogue , as in the DoGreeting case. Now, neither of thesemessages corresponds to a Hub program by that name, but both of them correspond tooperations, and the Hub will find the server with the appropriately named operation and send themessage to it. The Dialogue server provides DoDialogue , and the UI server (!) provides FromDialogue .

This last bit is a little sneaky. The UI is listening for connections from the unit tester, whichdoesn’t implement any dispatch functions. We’re just relaying this message to that server so thatyou can see what the result of the interaction is. In effect, we’re performing a slightly largersubset of the toy travel demo than just the interaction with the Backend; the unit tester isproviding input to the Dialogue server, and digesting output from the Dialogue server:

109

Let’s try it:


% process_monitor $GC_HOME/tutorial/messages/hub_backend.config

Start the Backend, Dialogue, Hub and then the unit tester. Select "Send new message", selectthe first frame named DoDialogue , and press OK. If you scroll through the output in the Hubpane, you’ll see the following steps:

1. Hub receives the DoDialogue message 2. Hub dispatches the DoDialogue message to the Dialogue server, and destroys the token

(because there was no program that matched, and no reply requested) 3. Hub receives the DBQuery message 4. Hub dispatches the DBQuery message to the Backend server 5. Hub receives the reply from the Backend server, relays the result to the Dialogue server,

110

and destroys the token (because the DBQuery program has ended) 6. Hub receives the FromDialogue message from the Dialogue server 7. Hub dispatches the FromDialogue message to the UI (unit tester), and destroys the token

(because there was no program that matched, and no reply requested)

And the unit tester interaction history will reveal the result:


[Sending: new message] {c DoDialogue :frame {c flight :origin "BOSTON" :destination "LOS ANGELES" } } [Received: new message] {c FromDialogue :tidx 3 :output_frame {c db_result :tuples ( ( "AA" "115" "1144" ) ( "UA" "436" "1405" ) ) :column_names ( "airline" "flight_number" "departure_datetime" ) } :session_id "Default" }

Now, let’s look at how errors might be handled here. Remember, the DoDialogue dispatchfunction doesn’t send a reply. This means that it can’t send an error reply (well, it could, butsince the Hub isn’t waiting for a reply, it will discard whatever reply it gets). So the Dialogueserver will have to send a new message when it gets an error from the Backend server. Soselect "Send new message", select the second frame named DoDialogue , and press OK. TheHub and servers will follow exactly the same steps, except that the Hub will receive an errorreply from the Backend server, and the new message the unit tester receives will describe the error:


[Sending: new message] {c DoDialogue :frame {c flight :origin "BOSTON" :destination "SAN FRANCISCO" } } [Received: new message] {c FromDialogue

111

:tidx 6 :output_frame {c error :description "error consulting backend" } :session_id "Default" }


Putting it all togetherAt this point, we can add in the Generator and Parser servers and perform an even largersubsection of the toy travel demo: we can provide strings as input and see strings as output, asillustrated here:

112

We’ll continue to expand our Hub program file, and we’ll once again use the trick of assigning aphony operation (ReportIO ) to the UI server (i.e., the unit tester). For simplicitly, we’ll only showthe Hub programs here. Note that some of these Hub programs have multiple rules:

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; ;; MAIN INPUT BODY ;; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

PROGRAM: UserInput


RULE: :frame --> Dialogue.DoDialogue IN: :frame OUT: none!

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; ;; MAIN OUTPUT BODY ;; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

PROGRAM: FromDialogue

RULE: :output_frame --> Generator.Generate IN: :output_frame OUT: :output_string

RULE: :output_string --> UI.ReportIO IN: :output_string OUT: none!

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; ;; DB SUBQUERY ;; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

PROGRAM: DBQuery


113

Let’s do one more exercise:


% process_monitor $GC_HOME/tutorial/messages/string_io.config

The two available inputs in this exercise correspond to the two inputs in exercise 3. There are anumber of processes in this exercise; you should be able to start them all by selecting "ProcessControl --> Restart all". When the unit tester window appears, select "Send new message",select the first frame named UserInput , and press OK. You’ll see the following interaction inthe interaction history:


[Sending: new message] {c UserInput :input_string "I WANT TO FLY FROM BOSTON TO LOS ANGELES" } [Received: new message] {c UI.ReportIO :session_id "Default" :output_string "American Airlines flight 115 leaves at 11:44 AM,and United flight 436 leaves at 2:05 PM" }

If you like, you can select the Hub pane in the process monitor and trace the history of thisinteraction; at this point in the tutorial, you should be able to do that without assistance.

To induce the error, select "Send new message" in the unit tester, select the second framenamed UserInput , and press OK, and then look at the interaction history:


[Sending: new message] {c UserInput :input_string "I WANT TO FLY FROM BOSTON TO SAN FRANCISCO" } [Received: new message] {c UI.ReportIO :session_id "Default" :output_string "I’m sorry, but I can’t get your answer from thedatabase" }

Again, inspect the history of the interaction in the Hub pane if you like.


114

SummaryIn this lesson, you’ve learned about the following concepts and components:

Synchronous and asynchronous new messages The functions GalSS_EnvWriteFrame and GalSS_EnvDispatchFrame

At this point, we’ve learned enough about servers and program files to understand a substantialsubsection of the toy travel demo. In the next lesson, we’ll take a look at the next level ofcomplexity: backchannel audio connections, known as "brokers".

Next : Setting up a brokered audio connection


115

Setting Up a Brokered Audio Connection

Up to this point, all the communication between servers has been mediated by the Hub. Thisarrangement has its advantages: for instance, the Hub provides a central location for flow ofcontrol and routing. However, it also has its disadvantages: in order for a message to get fromone server to another, it has to be decoded, analyzed, and reconstructed in the Hub. While thisprocess appears to be quite efficient, there are circumstances in which you might prefer to senddata directly: for example, in situations involving high bandwidth or streaming, such as audioinput or output.

To support this alternative, the Galaxy Communicator infrastructure supports a peer-to-peerconnection called a broker . The Hub mediates the establishment of this connection, but all datawhich flows through the connection goes directly from server to server. In this lesson, we’ll learnhow to set up a broker connection and how it works.

Recognizer, Synthesizer and Audio Brokering in four steps

Step 1: Establish the broker listener Step 2: Notify the Hub and client Step 3: Establish the broker connection, transmit data Step 4: Process data, shut down broker listener

Brokering details Source server: broker setup, message dispatch Hub: broker client notification Target server: set up broker client, establish callback About broker data The two ends of the broker connection An example

Why the dependency on :call_id ? Summary

Once you feel comfortable with this lesson, you can consult the broker reference documentation.

116

Recognizer, Synthesizer and AudioUp to this point, we’ve examined four servers in the toy travel demo: the Parser, Backend andGenerator servers, which represent the simplest type of server, and the Dialogue server, whichrepresents the next level of complexity. The first three servers simply send responses todispatch functions, while the Dialogue server sends new messages to the Hub and occasionallywaits for the response. The last three servers take advantage of all these tools, and furthermoreinteract with each other via broker connections.

In addition, the Audio server also exhibits the complexities of UI programming, which we’llexplore in the next lesson.

Brokering in four stepsThe setup and operation of brokers can be broken down into four steps. We’ll describe andillustrate these steps briefly, and then look at some code. We’ll focus on the interaction betweenthe Synthesizer and Audio servers as an example.

Step 1: Establish the broker listenerIn the first step, the server which will be the source of the broker data establishes a listener forthe broker. This listener, in most cases, is the same listener that’s listening for connections fromthe Hub; the protocol is constructed so that the listener can distinguish between the two types ofconnections. The server assigns each broker a unique ID, so that the server can listen formultiple brokers simultaneously. The server also assigns each broker a timeout; it will acceptmultiple connections for the broker, until the timeout is reached.

117

Step 2: Notify the Hub and clientIn the second step, the source server sends a new message to the Hub which contains thebroker contact information. This information consists of the host, port and unique ID associatedwith this particular broker. This message is a message like any other, and can contain additionalinformation (such as encoding format and sample rate for audio).

118

Step 3: Establish the broker connection, transmit dataWhen the target server receives this information, it establishes a connection to the source serverusing the host, port and unique ID, and establishes a callback to receive the data. The sourceserver can now send data over the connection, and the callback will be invoked in the targetserver whenever data is received.

119

Step 4: Process data, shut down broker listenerFinally, when the source server is done sending data, it notifies the target server that it’s done,and the broker connection is terminated. The target server does whatever it wants to do with thedata (e.g., send audio to the audio device). When the source server’s broker reaches its timeout,it stops accepting connections for that broker; when all the connections for the broker are done,the source server shuts down the broker.

120

Now that we’ve seen these steps in the abstract, let’s take a look at each one of them in detail.

Brokering details

Source server: broker setup, message dispatchIn the first and second brokering steps, the source server sets up a broker listener and sends amessage to the Hub announcing the broker. Here’s how that happens.

121

There are three functions involved.

The function GalIO_BrokerDataOutInit creates the broker listener. The first argument of thisfunction is the connection which the call environment contains (which can be retrieved fromthe call environment using the function GalSS_EnvComm). This connection, and the serverit’s associated with, will host the broker and its listener. The second argument is how often(or whether) to allow the toplevel Communicator loop to poll this broker to see if there’sanything to do; 0 means use the default poll value. The third argument is the timeout, inseconds; after this amount of time, the broker will stop accepting new connections. The function GalIO_GetBrokerListenPort can be used to check that the broker listener hasbeen set up correctly. If it returns 0, there’s a problem. The function GalIO_BrokerPopulateFrame is used to insert the appropriate brokerinformation into the message being sent to the Hub. The host and port will be stored in theindicated keys; the unique call ID is always stored in the :call_id key.

After the message is populated with the appropriate brokering information, it is sent to the Huband freed, just like any other message.

Hub: broker client notificationThe Hub portion of the process is the simplest, and you can probably guess what it looks like:

PROGRAM: FromSynthesizer

RULE: :host & :port & :call_id --> Audio.Play IN: :host :port :encoding_format :sample_rate :call_id

122

The Hub matches the incoming message with the appropriate program, shown here, and checksthe rule. All three keys are present in the token: :host and :port were specified in the call to GalIO_BrokerPopulateFrame, and the :call_id key is the key which GalIO_BrokerPopulateFrame always uses for the unique broker ID. So the rule fires, and therelevant data is sent to the Audio.Play dispatch function.

Target server: set up broker client, establish callbackWhen the Audio server invokes the Audio.Play dispatch function, it creates a broker client, as follows:

There are three functions involved in setting up the client correctly.

The function GalSS_EnvBrokerDataInInit creates the broker client. It requires a callenvironment object (in case the callback needs the environment to write new messages tothe Hub), the host and port obtained from the message frame, the message frame itself (toget the unique ID from), and a data callback function which is invoked whenever the targetserver receives data over the broker connection (and a few other arguments we won’tdiscuss). If successful, this function returns a connected broker client. The function GalIO_AddBrokerCallback allows the programmer to associate additionalbehavior with various events associated with the broker (more on this later). The importantaspect of this call here is that the programmer can associate behavior with the conclusion ofa broker connection. The function GalIO_SetBrokerActive activates the broker. Under some circumstances,broker clients might need to be evaluated in a particular order (say, if you want to guaranteethat audio output is played in the order in which the brokers are set up). One way to controlthis is by ensuring that broker clients don’t do anything unless specifically requested to. As aresult, all broker clients are created inactive, and you must activate them to have them doanything. This aspect of broker clients is a historical idiosyncracy, and we recommend thatyou activate all broker clients when you create them, and use your own mechanisms to

123

impose any ordering constraints.

At this point, the source and target servers have established a broker connection, and the brokerclient can begin to consume data.

About broker dataWhen we discussed frames and objects, we talked about the object types that could appear inframes, and concentrated on frames, strings, integers and lists. We suggested at that point thatthere were many other datatypes which could appear in frames, including arrays of integers ofvarious sizes. Any type that can appear in a frame can be sent across a broker connection, andvice versa, but we won’t be looking at frames, string, integers and lists in our broker example;we’ll be looking at arrays of integers.

When data is written to a broker server, it is encoded and cached in an output queue. If anybroker clients are currently connected, it is also written to the client; and when a new clientconnects, the server transmits the contents of the output queue to the new client. In other words,all clients which connect to the broker server are guaranteed to see all the same data in thesame order, no matter when they connect. This is one of the reasons that broker servers are setup to expire; if a broker server didn’t expire, it would never be possible to free its output queue.

The two ends of the broker connectionOn the source server side, the server writes data to the broker, and then indicates that it’s done,approximately as follows:

void *data; int num_samples; GalIO_BrokerStruct *b; SynthesisRecord *s;

/* ... */

data = PollSynthesis(s, &num_samples);

while (data) { GalIO_BrokerWriteInt16(b, data, num_samples); free(data); data = PollSynthesis(s, &num_samples); } if (SynthesisIsDone(s)) { GalIO_BrokerDataOutDone(b); /* ... */

There are two functions involved in this example.

The function GalIO_BrokerWriteInt16 is one of a family of functions which write typed datato a broker. This particular function writes an array of 16-bit integers to the broker. The

124

second argument is the array of data, and the third argument is the number of elements inthe array. This function can be called as many times as desired, and data of different types canbe mixed in the same broker connection (although typically isn’t). When there’s no more data to send, the function GalIO_BrokerDataOutDone must becalled. This call causes a special message to be sent over the broker connection which indicatesthat all data has been written. If this function is not called, the broker clients will never terminateand disconnect, and the broker server will never die.

On the target server side, the data callback function is invoked each time a block of data isreceived (each block corresponds to a single call to one of the GalIO_BrokerWrite* functions).The data callback function looks approximately like this:

static void __AudioOutputCallback(GalIO_BrokerStruct *b, void *data, Gal_ObjectType data_type, int n_samples) { /* ... */

switch(data_type) { case GAL_INT_16: printf("[Audio data to user (%d samples)]\n", n_samples); /* Do something with the audio */ free(data); break; default: GalUtil_Warn("__AudioOutputCallback: unexpected data type %s\n", Gal_ObjectTypeString(data_type)); free(data); break; } }

Each data callback function has the same signature. The first argument is the broker client. Thesecond argument is a pointer to the data, which is interpreted according to the data type in thethird argument. So, for instance, if the data type is GAL_FRAME, then the data is really aGal_Frame structure; if the data type is GAL_STRING, the data is really a char * . In this case,all the data which the source server wrote is an array of 16-bit integers; the corresponding typeis GAL_INT_16. The fourth object is interesting only for the array and list types; it’s the numberof elements in the array or list.

So when the target server receives a block of data, it invokes the data callback, which decideswhat to do with the data based on the data type. The data is allocated as new memory before it’spassed to the data callback function, so once it’s done with the data, it frees it. This happens foreach block of data (that is, each GalIO_BrokerWrite* call on the source side), until the brokerclient receives the termination message, at which point it will do whatever it’s supposed to dowhen it’s done (see the call to GalIO_AddBrokerCallback here), and terminate.

125

An exampleLet’s watch a broker in action. This is slightly complicated by the fact that the unit tester cannothost broker servers or clients. So we need to use the unit tester, acting as a server, to send amessage to the Hub which will cause the Synthesizer server to produce a broker connection.This is actually pretty simple; we’ll just send a Synthesizer.Synthesize message to theHub, and populate its program file with the program we saw here, to handle the new messagefrom the Synthesizer server.

[Brokering exercise 1]

% process_monitor $GC_HOME/tutorial/brokering/synth.config

Select "Process Control --> Restart all", then select "Send new message" in the unit tester.Select the message named Synthesizer.Synthesize , and press OK. You’ll see the result inthe Audio pane:

[Audio client pane]

[Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (1024 samples)] [Audio data to user (184 samples)] [Audio data to user is finalized (14520 samples).]

Each of these printouts corresponds to a single call to the data callback function. The Hub showshow the information was conveyed from the source server to the target server:

[Hub pane]

------ find next op ---------------------- {c FromSynthesizer :host "129.83.10.107" :port 15500 :call_id "129.83.10.107:24405:0" :sample_rate 8000 :encoding_format "linear16"

126

:session_id "Default" :tidx 3 } --------------------------------------------

found operation: Audio.Play ---- serve(Audio@<remote>:-1, token 3 op_name Play (sole provider))

Got reply from provider for Audio (id 2) : token 3

----------------[ 3]---------------------- {c FromSynthesizer :host "129.83.10.107" :port 15500 :call_id "129.83.10.107:24405:0" :sample_rate 8000 :encoding_format "linear16" :session_id "Default" :tidx 3 } ----------------------------------------

[Destroying token 3]

The FromSynthesizer message constructed by the Synthesizer server contains the host, portand unique broker ID, and the program shown here sends the appropriate message to the Audioserver. Finally, the Synthesizer pane shows the evidence of the broker client connecting, theunique ID matching, and the transmission of the special conclusion message:

[Synthesizer pane]

__GalIO_ServerContactHandler: connected to 129.83.10.107 (sock=6) _GalIO_BrokerListenerHandshakeHandler: reference frame matched GalIO_BrokerDataOutCallbackHandler: done sending data.


Why the dependency on :call_id ?At this point, you may be a little puzzled about why this mechanism is so complex. In particular,there are three keys which must be sent from source to target server, and one of the keys (:call_id ) is a "magic" value. In addition, there’s no way to tell that there’s a broker requestbeing passed around; nothing guarantees that the :call_id key is used only for brokers.Finally, the mechanism seems too flexible for the normal case; while brokers support sendingarbitrary data of different types through a broker connection, in most circumstances all the datawill be the same type.

127

There are at least two circumstances in which this is not just inelegant, but a realimplementational problem. For example, let’s say we want to define an HTTP bridge whichwould allow us to transmit Communicator interactions through a firewall over HTTP. In order todo this in a general way, we’d have to be able to identify broker requests automatically, becausethe broker data would have to be consumed and encoded over the same HTTP connection.Since we can’t do that given the current implementation, we can’t build a general HTTP bridge. Asecond case is the reason the unit tester doesn’t deal with brokers. In order to establish brokerconnections, the unit tester, acting as a broker client, would have to ask the user to determinewhether a broker request is present, and to identify the keys which correspond to the brokerinformation in the incoming message if a broker request is present. This process might or mightnot be able to happen before the broker timeout if the user has to be involved.

In fact, there’s an obvious alternative. In addition to types like GAL_STRING, etc., we couldintroduce a new type - call it GAL_PROXY - which encapsulates the appropriate brokerinformation, and, in the simplest cases, the type of the data being sent. In the best ofcircumstances, the client server would establish the broker connection and access the brokerdata without the programmer even having to know about it.

The reason we haven’t implemented this yet is, frankly, because we haven’t had the time. Butwe have been planning ahead. As of Galaxy Communicator 3.0, we’ve added support for theGAL_PROXY type in the communication protocol. But we haven’t yet had a chance to implementthe APIs which would enable access to it. We’re hoping to do that very soon.

SummaryIn this lesson, you’ve learned about the concept and the details of broker connections, including:

The broker server functions GalIO_BrokerDataOutInit, GalSS_EnvComm, GalIO_GetBrokerListenPort, GalIO_BrokerPopulateFrame The broker client functions GalSS_EnvBrokerDataInInit, GalIO_AddBrokerCallback, GalIO_SetBrokerActive The data transmission functions GalIO_BrokerWriteInt16, GalIO_BrokerDataOutDone

In the next lesson, we’ll learn about the final and most complex type of server: one whichmonitors user interaction.

Next : Creating a UI server


128

Creating a UI Server

Up to this point, we haven’t talked much about the engineering model behind the GalaxyCommunicator infrastructure, and we haven’t needed to. However, in order to learn how tointegrate UI elements into a Communicator-compliant system, it’s important to understand howthings happen in the infrastructure, and how to control how things happen. This lesson is moreabout computer science than anything else, and we won’t look at any code; but at its end, youshould understand enough about how the Communicator infrastructure works to integrate a UIaspects into a Communicator-compliant server.

The Communicator object model Event-driven programming The event monitor Adding UI elements Summary

The Communicator object modelThere are several object types involved in flow of control in the Galaxy Communicatorinfrastructure. The first Galaxy Communicator object we came across in looking at server codewas the call environment. In the last lesson, on brokering, we encountered the broker object. Webriefly encountered the server object when discussing the server initialization function, and webriefly encountered the connection object when setting up outgoing brokers. Here’s the completeobject model:

129

The reason this object model is important is because the server, connection, and broker objectsare all associated with separate communications channels, and these channels are monitored bythe Galaxy Communicator infrastructure using an event-driven programming model. This modelis at the heart of the flexibility of the Galaxy Communicator infrastructure, and it’s worth spendingsome time understanding it better.

Event-driven programmingThere are three major components to an event-driven programming model:

events; callbacks, which are instructions about what to do when an event occurs; and an event monitor, which observes the events and invokes the callbacks.

In a communications system like Galaxy Communicator, some of the events are typically theavailability of data to read from or write to a communications channel; we’ll call these events data events . Another type of event is a timer event , which marks the conclusion of apreviously-determined time interval. These events can be used to schedule activities (forexample, inspect the state of a connection every 10 milliseconds).

For each type of object in the Galaxy Communicator object model, there’s a correspondingcallback. The server object can establish a new connection when it has a request to do so; theconnection object can invoke a dispatch function when it has message data; the outboundbroker can feed data to its connected clients; and the inbound broker can read typed data whenit’s available.

130

So we can see now that the data callbacks for the broker clients we learned about in the lastlesson are actually part of this larger model. They’re not exactly the low-level data callbackswe’re talking about here, since the data has been read and digested long before the brokercallback sees it; rather, it’s one of a set of higher-level callbacks based on what we’ll call logical events that the Galaxy Communicator infrastructure supports. In general, it’s these logicalevents which are visible to the average developer, rather than the lower-level data and timerevents. Not surprisingly, dispatch functions also count as a type of logical callback:

There are a number of other logical events which the various objects support, some of which area direct result of communications events (for instance, when a server creates a new connectionobject), and some of which are under programmer control (for instance, when an outbound orinbound broker is created). The full set of logical events can be shown as follows:

131

The programmer can associate callbacks with any of these logical events; in fact, we saw a brief example of this behavior when our target server associated a data completion callback with itsbroker client.

The power of the event-driven programming model is embodied in its extremely conservativeassumptions about ordering and scheduling. Because events can occur randomly, theappropriate behavior in response to any event must be carefully encapsulated. This means thatthe resulting system is maximally embeddable in other software, because all the requiredbehavior is completely separate from the scheduling and flow of control.

In other words, the event monitor is the least important dimension of the model; you can replaceit with another event monitor without any sacrifice in functionality. In fact, we provide an examplein the distribution which substitutes a different event monitor for the default GalaxyCommunicator event monitor. Furthermore, the Python and Allegro Common Lisp programminglanguage bindings use their own event monitors, but invoke the same underlying C callbacks asthe default C implementation. So even before we look at UI programming, we can derive somesignificant benefits from this model.

The event monitorThere are two strategies for implementing event monitors we’ll look at: threads and event loops.We’re only going to talk about threads in passing, but it’s a valuable conceptual step to the event loop.

132

In the abstract, a system like this has a number of parallel threads of execution going on. Someof these events are events which happen at particular intervals, and others are events whichhappen when data is available on a particular communications channel. These threads areconceptually separate. You could think of a callback which is fired every 10 milliseconds as onewhich is governed by an infinite loop which fires the callback, sleeps for 10 milliseconds, andrepeats. Similarly, you could think of each data callback as governed by an infinite loop whichconstantly tries to read data from the communications channel, and succeeds only when data isavailable (a so-called blocking read ).

In a multithreaded programming situation, this is exactly what happens. Each logical monitor isassigned to a programming thread. Examples of this are when a Communicator C server is runin threaded mode (which we won’t talk about in this tutorial), or in the Java or Allegro CommonLisp programming language bindings. In these latter cases, the threaded model is theconceptually natural model for the language.

If you don’t use threads (and the default C server doesn’t), life is a little more complicated. Sinceyou can only have one infinite loop, these conceptually independent threads of execution have tobe collapsed into one:

Obviously, in this case we can’t sleep or do blocking reads; if we block on one channel, we’llmiss events on another channel. So we have to use an event loop . Periodically, the event loopchecks the communications channels it’s been told to monitor, and checks the expiration time ofthe timers it’s been told to monitor. When it discovers that data is available, or that a timer has

133

expired, it fires the appropriate callback. When the callback finishes, the event loop continues.The default Galaxy Communicator main loop is an event loop.

Adding UI elementsWhen you add a UI element, what you’re doing, essentially, is adding another type of event tomonitor. You may be monitoring standard input, or perhaps input from audio:

In cases like this, the appropriate thing to do is to add these events to the Galaxy Communicatormain loop:

A well-designed main loop will allow the programmer to add and remove callbacks for timer anddata events at will; the Galaxy Communicator event loop provides these capabilities. In fact,there’s even a utility in the Galaxy Communicator distribution which allows you to add a standardinput monitor to the Galaxy Communicator main loop fairly easily. This is the utility whichmonitors the carriage returns in the Audio server.

However, the reason this is simple to do is that monitoring standard input, or an audio device,usually doesn’t come with its own main loop. There are many other common tools, such asCORBA and windowing systems, which have their own main loop. For instance, you’d encounterthis problem if you wanted to build a Communicator-compliant Motif or Tcl/Tk server:

134

This is exactly the same problem we faced with multiple parallel threads of execution: unlessyou’re actually using threads, you can’t have more than one main loop running. So in thesecases, either the Galaxy Communicator main loop has to go, or the windowing main loop has to go.

Fortunately, almost all of these main loops are well-designed, in that they support the ability toadd and remove timer and data event callbacks. And since we know that Galaxy Communicatoris designed so that all the work is done in the callbacks, and we have no idea whether the sameis true for these other systems, the appropriate thing to do is to toss out the GalaxyCommunicator main loop and monitor all the appropriate events in the other main loop:

This is how the unit tester works, actually; it monitors the Communicator events in the Python/Tkmain window loop.

For external main loops implemented in C, there’s a well-defined way of doing this embedding ofthe Galaxy Communicator callbacks, called external loop records. The example in thedistribution which substitutes a different main loop for the Galaxy Communicator main loopshows how this mechanism works.

SummaryThis lesson has been fairly abstract, out of necessity; an appropriately detailed examination ofrelevant code examples is far beyond the scope of this tutorial. We believe, though, that theGalaxy Communicator distribution contains enough examples and documentation to allow you tolearn the details on your own.

135

This brings us to the end of our discussion of the structure of servers. We still have three morelessons to go. These lessons will address more details in the Hub which will finally allow you toassemble an entire end-to-end system.

Next : Setting up the Hub to log your interactions


136

Setting Up the Hub to Log Your Interactions

As we’ve learned more and more about servers, we’ve seen that our knowledge covers moreand more of the operation of the toy travel demo that we saw in the very beginning of thistutorial. In the final two lessons, we’ll cover the remaining details of the toy travel demo, most ofwhich have to do with the operation of the Hub.

Introducing logging Setting up the log Inserting information into the log

Logging messages Logging key-value pairs Logging audio

Looking at the log The "real" toy travel demo Working with logs Summary

Once you’ve worked through this lesson, you can take a look at the logging reference.

Introducing loggingTo introduce logging, we’ll start by logging our little interaction with the Parse server from the unittester lesson.

[Logging exercise 1]

% process_monitor $GC_HOME/tutorial/logging/parse-server.config

Start the Parser, then the Hub, and finally the unit tester. Press "Send new message", select thefirst message, select "Reply required" and then OK. After the interaction, press the "Stop" buttonassociated with the Hub pane, and then select "File --> Quit". You’ve now created a log.

137

Here’s the relevant portion of the program file $GC_HOME/tutorial/logging/parse.pgm :

LOG_VERSION: "Parser, version 1"

;; This means that the log directory hierarchy ;; will start in the directory where the Hub is run.

LOG_DIR: .

TIMESTAMP: Parse UserInput

PROGRAM: UserInput

RULE: :input_string --> Parser.Parse IN: :input_string OUT: :frame LOG_IN: :input_string LOG_OUT: :frame

In the next few steps, we’ll learn about the information in the program file which helps organizethe log, and then we’ll look at the log itself.

Setting up the logThere are two lines in the program file which set up the log:

LOG_VERSION: "Parser, version 1"

;; ...

LOG_DIR: .

The first line is optional. LOG_VERSION: is a directive which allows you to associate arbitrarynames with your logs. These names are intended to correspond to system or program fileversions, so you can keep track of what to expect in your log (for example, you may change thenames of programs, dispatch functions or keys in your system, and this might affect your abilityto extract the appropriate information from the logs for, say, summarization).

The second line is obligatory. Without a LOG_DIR: directive entry, the Hub will not create a log.The value of LOG_DIR: is where the log directory hierarchy should live. This value is interpretedrelative to the directory where the Hub was started from. In this directory, the log can be found in sls/ date/ counter/sls- date- counter-hublog.txt , where date and counterguarantee a unique identifier for each log. (You can change the string sls by using theUSER_ID: directive entry, but we won’t cover this in the tutorial. See the logging reference.)

138

Inserting information into the log

Logging messagesYou can log incoming new messages and their returns, and also outgoing new messages andtheir returns. The TIMESTAMP: directive entry does this for you:

TIMESTAMP: Parse UserInput

This entry covers both program names (incoming messages) and operation names (outgoingmessages). Each of these messages is recorded in the log along with a timestamp in milliseconds:

139

Logging key-value pairsYou can log key-value pairs using the LOG_IN: and LOG_OUT: directive entries. Here’s whatthe rule which invokes the Parser server really looks like:

RULE: :input_string --> Parser.Parse IN: :input_string LOG_IN: :input_string OUT: :frame LOG_OUT: :frame

This process can be illustrated as follows:

These two directive entries, like IN: and OUT:, have two namespaces, sort of. The sourcenamespace for both these directive entries is the message (message send for LOG_IN:,message return for LOG_OUT:), but the target "namespace" isn’t really a namespace; all thedata is written directly to the log. So you can map to different keys, or log from differentnamespaces, or log literals:

LOG_IN: (:string :input_string) (:user_id $in(:user_id global))(:log_successful 1)

If you log key-value pairs for a message which isn’t timestamped, the message will automaticallybe timestamped for you.

140

Logging audioSince most audio is brokered and doesn’t pass through the Hub, the Hub can’t log it. Up to now,in the Communicator program, logging of audio has been left up to the individual audio servers.Your audio server should write the audio to a file and then report the name of the file to the Hub,so that the Hub can log it. If the audio server doesn’t share a file system with the Hubexecutable, you may need to do some postprocessing to get the audio and the logfile into acommonly accessible location.

Looking at the logIn general, you won’t be looking at the log, at least not in its raw form. But just so you have anidea of what’s happening, let’s look at the log we recorded. You should find it in sls/ date/000/sls- date-000-hublog.txt , since it’s the first log you’ve recorded in thisdirectory on this data.

LOGFILE_FORMAT_VERSION: 1.0

LOG_VERSION: Parser, version 1

SESSION_ID: Default

:BEGIN_UTT (-01) [Timestamp (-01): read:GAL_MESSAGE_MSG_TYPE UI(<remote>:-1) 1UserInput at 10:17:14.58 on 27-SEP-2001] [Timestamp (-01): send:GAL_MESSAGE_MSG_TYPE Parser(localhost:10000)1 Parse at 10:17:14.59 on 27-SEP-2001] :input_string (-01): "I WANT TO FLY FROM BOSTON TO LOS ANGELES" [Timestamp (-01): read:GAL_REPLY_MSG_TYPE Parser(localhost:10000) 1Parse at 10:17:14.59 on 27-SEP-2001] [Begin :frame (-01)] {c flight :destination "LOS ANGELES" :origin "BOSTON" } [End :frame (-01)] [Timestamp (-01): send:GAL_REPLY_MSG_TYPE UI(<remote>:-1) 1UserInput at 10:17:14.59 on 27-SEP-2001]

You can see that both the read and send (new message and reply) for UserInput weretimestamped, along with the send and read (new message and reply) for the Parse message. Inboth cases, the server’s name and location were also recorded, as well as the token index (inthis case, 1). You can also see that the :input_string key-value pair was logged, as was the :frame key-value pair.

141

As for administrative information, you can see that the log contains the version we provided inthe logfile, as well as a format version number (in case the format changes), and an indication ofthe current session (again, more on this later; for this reason, also ignore :BEGIN_UTT and the-01 that appear in the log).

We provide a set of tools to postprocess these logs in a number of ways. To illustrate this, we’regoing to use the "real" toy travel demo.

The "real" toy travel demoThe version of the toy travel demo we used differs from the default version in the distribution infour ways:

It uses a shorter interaction script It inserts breakpoints, which we used to pause the Hub It allows only the Audio dummy server to connect, and not the UI dummy server It doesn’t do logging

Now, we’re going to address the last of these points. So let’s run the "real" toy travel demo. Toprepare for this, please delete the sls directory which contains the simple log we just generated:


% rm -rf sls % setenv TTDEMO $GC_HOME/contrib/MITRE/demos/toy-travel % process_monitor $TTDEMO/toy-travel.config --$TTDEMO/example.frames $TTDEMO/toy-travel.pgm

Select "Process Control --> Restart all". This will start all the servers, then the Hub, and finallytwo additional process monitors, one for the Audio server and one for the UI server. In the"Audio client" process monitor, press "Start". Each time you see the prompt "Hit <return>to send speech:" in the scrollable output history, press the "Input <return>" button, until theserver exits. In the toy travel process monitor, press the "IOMonitor" button. You should seesomething like the following in the scrollable output history:

[Toy travel scrollable output history]

[exec $DEMO_ROOT/bin/IOMonitor -verbosity 0 2>&1] In session session-1001366584.434: system said "Welcome toCommunicator. How may I help you?" In session session-1001366584.434: user said "I WANT TO FLY LOS ANGELES" In session session-1001366584.434: system said "Where are youtraveling from?" In session session-1001366584.434: user said "BOSTON" In session session-1001366584.434: system said "American Airlines

142

flight 115 leaves at 11:44 AM, and United flight 436 leaves at 2:05 PM"

Select "File --> Quit" in the toy travel process monitor to end the interaction.

Working with logsIn the DARPA Communicator project, we’ve established a scoring procedure which reports arange of metrics based on these log files. MITRE has developed a set of tools (which NIST isnow maintaining, but we continue to distribute) to help generate these metrics. For details onthese tools and metrics, see the log tools reference. in the remainder of this exercise, we willdescribe this procedure and give you an illustration.

Most of the steps in this procedure are automated. The tools convert the raw Hub log into XML,and then use a hand-generated rules file to add significant "landmarks" to the log (e.g., the :input_string value in the Parse message corresponds to the output of the recognizer).These tools add both these annotations and a set of hand annotations (such as audiotranscriptions) to the log, and the result can be used for summarization, validation or scoring:

Let’s examine a summary for the interaction you just performed. We’ve provided a set of rules toadd the landmarks, but no hand annotations, so the summary will show the "recognizer output"but not the audio "transcription". We’ll summarize the entire contents of the sls subdirectory,since all the tools work recursively:

143


% process_monitor $GC_HOME/tutorial/logging/show-log.config

Press "Start", and review the output. You’ll see the summary of the interpreted log: when theuser and system started and stopped speaking, what each of them said, and when user andsystem turns began and ended. When you’re done examining the summary, select "File --> Quit".

SummaryIn this lesson, we’ve explored the basics of configuring the program file to generate a raw Hublog, and what we can do with that log. In the next lesson, we’ll talk about how sessions help youmanage your interactions, and in the final lesson, we’ll cover all the random odds and endswhich you need to understand to finally assemble an end-to-end system.

Next : Sessions


144

Sessions

Before we look at the final details of putting together an end-to-end system, there’s a subjectwhich we’ve mentioned numerous times but have yet to cover: sorting interactions into sessions.In this lesson, we’ll learn what sessions are, how they’re created, how they’re managed, andwhat they can do for you.

What sessions are Why use sessions? How sessions propagate Creating sessions Ending sessions Summary

What sessions areAt various points throughout this tutorial, we’ve referred to the notion of session :

In our toy travel interaction, we saw reports in the IO monitor which referred to input andoutput within a particular session. We saw that the unit tester utility has a typein window for session IDs, and a radio button tolock the session (whatever that means) We saw that one of the namespaces in the Hub refers to the session. We saw that the call environment argument to dispatch functions contains the currentsession as part of its contextual information. We saw that the session ID turns up in the Hub log.

A session is a interaction between the user and system, with a clear beginning and end. Forinstance, when a user calls a system on the phone and the system answers, that’s the beginningof that session with the user; when the user or system hangs up, that’s the end of the session.Sessions are distinguished by strings called session IDs, which we’ve seen referred to in theHub log and the unit tester.

Sessions distinguish users from each other and group applications associated with the sameusers together. So a single Hub might support multiple users, each using different dedicatedchannels (such as audio), and multimodal interactions by each user, using multiple dedicatedapplications (say, a GUI and audio channel):

145

Why use sessions?There are a number of reasons that sessions are valuable.

First, they provide the boundaries on separate interactions. There’s a one-to-one mappingbetween sessions and Hub log files when logging is enabled in the Hub, and this allows us tosort out statistics about user interactions appropriately for scoring and analysis.

Second, they provide a, well, session-level memory state in the Hub (i.e., the sessionnamespace) for each user interaction which spans all the tokens generated for that userinteraction. So if a Hub program needs to record crucial information about the interaction (say,the interaction language, or the media available for output), the session is the appropriate place(and, perhaps, the only appropriate place) to store it. This is especially important if your systemanticipates supporting multiple simultaneous users.

Third, and most dramatic, sessions provide a way of controlling the eligibility of service providersin the Hub. Servers can lock themselves to particular sessions (we’ll talk about this in a minute)so that they’re constrained to send messages to the Hub only for a particular session, or (moreimportant) to receive messages associated only with a particular session. Imagine, for amoment, if this were not the case, and the system had two simultaneous users. If we could notuse sessions to distinguish between the various UI elements, and one user provided some input,we wouldn’t necessarily be able to guarantee which user received the response! This is becauseif the response is a new message, rather than a message reply to the original input, the systemwould try to select an appropriate UI element to handle the output, and it wouldn’t be able to tellwhich UI element was associated with which user. You may respond, "Well, why doesn’t it justkeep track?", and the answer is: that’s what sessions do.

We can see this behavior by using the unit tester. The unit tester allows you to specify thesession associated with a new message, and also optionally allows you to lock the session. If wedon’t set the lock and we have two unit tester servers connected to the same Hub (say, to thetoy travel demo), it should be random which server receives the response to any input fromeither server. And indeed, that’s exactly what happens.

[Sessions exercise 1]

% process_monitor $GC_HOME/tutorial/sessions/unit-tester.config

146

Select "Process Control --> Restart all". Two unit testers should start up. In one of them (itdoesn’t matter which), press "Send new message", and then select the first message (the onenamed UserInput ), and then OK. Repeat this process several times, and watch the interactionhistory of both unit testers carefully. In each case, the unit tester you interacted with will have thefollowing in its interaction history:


[Sending: new message] {c UserInput :input_string "I WANT TO FLY FROM BOSTON TO LOS ANGELES" }

And one or the other of the interaction history panes will have the response in its interaction history:

[Interaction history pane]

[Received: new message] {c UI.Print :session_id "Default" :output_string "American Airlines flight 115 leaves at 11:44 AM,and United flight 436 leaves at 2:05 PM" }

However, which interaction history gets the response should be essentially random.

Now, repeat the procedure, but this time type "Session1" in the "Session ID" entry box and pressthe "Lock session" button before you press OK. It may already say "Default"; delete it if it does(more on this in a minute.) (You’ll only have to type "Session1" the first time; more on this in aminute too). As long as you’ve provided a session ID and locked the session, the responseshould always come back to the same unit tester.


Locks are a little complicated. You can control which dimensions of the message traffic arelocked, and whether the lock is permanent or not. You can find more details about locks in the Hub properties documentation.

How sessions propagateSo why did the "Session ID" box say "Default", and why did we only have to type "Session1" thefirst time? First, Default is the name of the default session, which is used when no session isspecified. When a new message arrives at the Hub, it is assigned a session to the resultingtoken, either one it’s requested (more on this in a minute) or Default . Then the Hub processesthis token. If the Hub sends a message to a server, and that server sends a new message as aresult, the chances are overwhelming that the new message ought to be associated with thesame session as the original message that was sent. For instance, in the toy travel demo, theuser input is ultimately routed to the Dialogue server which (as we saw in the new messages

147

lesson) sends a new message to the Hub which constitutes the system’s response. Thisresponse is part of the same session (user interaction) as the user input:

This is accomplished as follows. When the Hub sends a message to a server, it sends thecurrent session ID of the current token along with the message, which is incorporated into thecall environment. Whenever a user writes a new message through a call environment which hasa session ID, the new message will automatically be associated with that session. In this way,session IDs automatically percolate from token to token:

The call environment serves this purpose in multiple situations.For instance, we saw in the brokering lesson that the broker client is set up with a call environment, just in case the clienthas to send a new message to the Hub when it’s done. This happens in the toy travel demo,when the Audio server sends a message to the recognizer, which receives the audio via a brokerclient and ultimately sends a new message from the broker callback containing the recognizedstring. This recognition result is part of the same session (user interaction) as the audionotification, and the call environment ensures that the same session is used:

148

So once a session is established, the call environment ensures that it is preserved andpropagated appropriately.

Creating sessionsSo where do session IDs come from? Typically, they start with your UI elements, since it’s theuser interaction which the sessions are encapsulating.

If your UI element is a server which the Hub has contacted, you can update the session IDin the call environment using the function GalSS_EnvUpdateSessionID., or GalSS_EnvSetSession to update the session ID and establish the session lock. If your UI element is a server which contacts the Hub, you can use the -session_id command line argument to set a session ID for a given interaction. This command lineargument establishes the session lock for you; you can get more details in the advanceddocumentation on hub listeners. You can also set the session ID programmatically beforethe server contacts the Hub; this is how the Audio server works in the toy travel demo, butthe procedure is a little too complicated to discuss here.

When the Hub receives a message with a session ID, it looks for a session with that ID. If itdoesn’t find one, it creates it. So it’s not necessary to create a session explicitly; sessions arecreated in the Hub by virtue of being mentioned.

149

Ending sessionsOnce a session is created, it survives until it is explicitly ended. There’s no inference, forinstance, that if a particular UI element sends a message which causes a session to be created,and that UI element disconnects, the session is over. There are obvious situations where thisinference would be wrong. For example, let’s say the user wishes to suspend a session andresume later. The user could simply disconnect now and reconnect later, perhaps from adifferent location and a different UI element, as long as the same session ID is used.

It’s important to end a session for all sorts of reasons, especially efficiency. There’s a lot ofmemory associated with the session, including the session namespace and all the informationrelated to session locks, and this memory can’t be freed until the session ends. It’s alsoimportant to close the Hub log file, which remains open until the session ends. So it’s importantto end the session.

You can end a session by invoking the operation Builtin.end_session in the Hub. Thisdispatch function is implemented by the Builtin server, which we’re going to discuss in the next(and final) lesson. If you look at the Audio server, it sends a message to the Hub with this namewhen it disconnects:

/* ... */ case AUDIO_UNAVAILABLE: /* The device shut down. End the session and exit. */ /* We ran out of inputs. Disconnect. */ printf("Audio no longer available. Disconnecting.\n");

/* First, end the session. */ output_f = Gal_MakeFrame("Builtin.end_session", GAL_CLAUSE); GalSS_EnvWriteFrame(p->env, output_f, 0); Gal_FreeFrame(output_f);

/* Next, shut down the connection. This ought to reset the poll. */ GalIO_SetCommDone(GalSS_EnvComm(p->env)); /* ... */

Assuming the Hub doesn’t have a program named Builtin.end_session , when it receivesthis message, it will invoke the appropriate Bulitin server operation.

SummaryIn this lesson, we’ve learned all about sessions: what they are, why to use them, and how tocreate, manage and end them. In the next (and final) lesson, we’ll cover all the random odds andends which you need to understand to finally assemble an end-to-end system.

150

Next : Assembling an end-to-end system

Last updated October 4, 2001

151

Creating an End-to-End System

We’re almost there. We can build servers, write Hub program files, handle errors, set up brokerbackchannels, understand the design of UI elements, log our activity, and keep track of multipleusers over multiple interactions. In this last lesson, we’ll address a number of remaining details.

Selecting your servers Contacting the Hub Guidelines for writing servers and Hub programs The Builtin server Debugging strategies Avoiding trouble Summary

Selecting your serversThe toy travel demo illustrated a representative set of servers you might need in your end-to-end system:

An audio server which supports some sort of telephony hardware, such as ComputerFoneor Dialogic; A recognition server which maps audio samples into text strings; A parsing server which maps text strings into semantic frames; A dialogue manager server which incorporates each incoming semantic frame into thecurrent context and decides how to act on the resulting context; An information server, such as a relational database; A language generation server which maps outgoing frames into text strings; A speech synthesis server which maps text strings into audio samples.

There are a wide range of variations on this configuration. For instance:

In some cases, you may choose to "federate" audio, recognition and synthesis in the sameserver. You might do this, for instance, because your recognizer can only read input fromthe audio device (this is the case with JSAPI-compliant recognizers in Java 1.2, due tolimitations of Javasound in that version of Java). You may have a separate server for mapping dialogue representations into representationsthe information server can understand (an SQL generation server, for instance).

152

You may have a different information server, such as a script which harvests informationfrom Web pages, or you may have multiple information servers, either for cacheing, redundancy,or to cover multiple domains. Your recognizer may return more structured output, such as n-best lists or word lattices, orprovide confidence measures along with its recognition results. You may choose to separate the dialogue server’s function of incorporation into contextfrom its function of choosing what to do with the resulting context. You may choose to provide a user profile server, either for security or user modeling orboth, which might register a user by voice or by touchtone input. You may choose to have multiple UI elements, such as a Web browser and a telephoneconnection working in tandem.

The Galaxy Communicator architecture places no restrictions on any of these configurations(although in its current form, it may accidently or intentionally make some configurations easierthan others).

We plan in FY02 to develop an open source toolkit of all the available Communicator-compliantwrappers and servers developed by the program participants which are free and available to use.

Contacting the HubOne thing we haven’t talked about in detail is how to set up servers which contact the Hub. It’sactually trivial. There’s a command line argument -contact_hub which is available by defaultto all Communicator-compliant servers which allows you to specify host:port where a Hubhas set up the appropriate listener. You can find detailed documentation about this in the Hublistener documentation. The short version is that there are two steps.

First, you should use the CLIENT_PORT: program file directive to set up the Hub listener, as wesaw in our program file tutorial. Second, you should start up the server using -contact_hub .As an example, here’s the command line to start up the Audio server from the toy travel demo:

% $DEMO_ROOT/bin/Audio -audio_data$GC_HOME/tutorial/toy-travel/short-example.frames -contact_hublocalhost:2800 -verbosity 0

The Hub program file has the appropriate listener set up:

SERVICE_TYPE: Audio CLIENT_PORT: 2800 OPERATIONS: Play

Under normal operating circumstances, you should be able to connect as many servers as youwant to the Hub listener; there is no limit imposed. If you want the listener locked to a particularsession, you can use the -session_id command line argument we talked about in the session lesson.

153

Guidelines for writing servers and Hub programs

Always use the environment to write messages to the HubThere are publicly available functions in the Galaxy Communicator API which allow you to writemessages to the Hub using the connection directly, instead of the environment. But if you usecall environments, you’re always guaranteed of passing along the correct session information.We recommend always using the call environment.

Make sure to declare your dispatch functions and Hub operationsIf you write a dispatch function in a server, you have to declare it using GAL_SERVER_OP if youwant the server to know about it. Similarly, if you want the Hub to know that the server has adispatch function, you have to declare it in the appropriate OPERATIONS: directive entry.

Make sure your messages are distinctThe Hub programming language is very forgiving, in the sense that it will ignore or pay attentionto messages which invoke programs depending on the keys in the message. So, for example,you could use main for the name of every message you send to the Hub via GalSS_EnvWriteFrame() or GalSS_EnvDispatchFrame(), and use the keys in the message todecide what to do with them. However, this strategy makes the program file extremely hard toread. We have learned, from studying MIT’s program files, that it’s much more straightforward todistinguish your messages at least by the server that sent them. Our toy travel exampledemostrates this strategy.

Use the listener-in-Hub capability at least for your UI elementsThe easiest way to manage UI connections to the Hub (that is, GUIs, audio servers, and the like)is to have them contact the Hub instead of the other way around. If you set things up using the listener-in-Hub capability, you’ll be able to have UI elements contact the Hub on an ad-hoc basis,and there are simple hooks for keeping the sessions separate as well. You can uselistener-in-Hub for all your server connections, if you choose, but it’s most desirable for UI elements.

The Builtin serverThe Builtin server is a special server which is implemented as part of the Hub itself. This serveris always available, which means that it doesn’t need to be declared in the Hub program file. Ithas special access to the internals of the Hub operations, so that it can be used to provideinformation about the Hub state (such as the servers which are available, or the state of variousnamespaces). The Builtin server is a grab bag of functionality, and we’re not going to talk aboutmost of it. We’ll concentrate here on a small number of potentially interesting and relevant functions.

154

nopThe dispatch function nop is a no-op. It simply returns its input frame. The reason you mightwant to use nop is that there may be some times when you may want simply to update akey-value pair in a namespace when a certain condition is met:

RULE: :output_parse == "PARSE_FAILED" --> Builtin.nop OUT: (:parse_failed 1)

The Hub scripting language probably ought to provide a case where you can omit the operation,but for now, using nop is the appropriate strategy.

call_programThis dispatch function can be used to construct a new message to the Hub. The name of theprogram should be passed in as the value of the :program key. All the other key-value pairspassed to call_program will appear in the new message. The new message will be processedlike any other new message; it will either invoke a program with the appropriate name, berelayed to a server which supports an operation with the appropriate name, or be discarded. Thisdispatch function will also return the result of the executed program, if requested.

For example, the toy travel demo unifies the processing of its typed input and the "output" of theRecognizer server by routing both to the same program (named UserInput ), as follows:

PROGRAM: FromRecognizer

RULE: :input_string --> Builtin.call_program IN: (:program "UserInput") :input_string OUT: none!

In this example, no result is requested, so none will be provided.

new_sessionThis dispatch function creates or resets a session. For creation, this dispatch function is reallynot needed, because sessions are created automatically when they’re mentioned, but thisdispatch function is crucial if you try to reuse a session ID. For instance, you might be running asystem which will never have multiple simultaneous users, and you might be using the defaultsession as your session every time. This is not recommended . However, if you must do this,new_session will reset the current session state and start a new log file in the appropriatecircumstances (i.e., if you’re calling new_session on this session for the second time, or later).

end_sessionThis dispatch function ends the current session. We discussed in the session lesson howimportant this is.

155

destroyThis dispatch function destroys the current token. We alluded to this dispatch function when wetalked about the special destroy! value of OUT: in the program file tutorial. This method ofdestroying the token is somewhat less efficient than the OUT: value, but it’s an available alternative.

hub_exitThis dispatch function causes the Hub to exit, in case you ever want this sort of thing underprogram control.

Debugging strategiesThere are a number of things you can do to make it easier for you to understand what’s going on(and what’s going wrong) with your end-to-end system. In this section, we describe some ofthem briefly.

Builtin.hub_breakThis Builtin dispatch function suspends the execution of the Hub and enters a loop where youcan inspect the state of various namespaces. We used this functionality to halt the Hub in ourinitial lesson on the toy travel demo. If there’s a Hub program rule that isn’t getting fired and youthink it should be (or vice versa), you may find it useful to insert a call to Builtin.hub_breakat the appropriate point in your Hub program.

The -debug argument to the HubYou can also access this same breakpoint behavior by using the -debug Hub command line argument. This argument will cause the Hub to break after it sends each new message. You canshut this behavior off as the Hub is running if you choose to, by typing a capital C at thebreakpoint prompt.

Exploit your verbosity settingsAs we described in the toy travel demo lesson, the Hub and servers support 6 levels of verboseoutput, from nothing (level 0) to 6 (everything). These levels can be controlled by setting the GAL_VERBOSE environment variable in your shell, or by using the -verbosity command linearguments for the Hub and servers. The default is 3. If you choose a level above 3, you’ll getmore information. These levels are not clearly defined yet, but among the most reliable are:

At level 4, you’ll get a full printout of the message traffic in the servers instead of thetruncated messages with ellipses that you usually see. At level 6, you’ll get the actual XDR encodings which are written to and read from theservers and the Hub. It’s not necessarily advisable to use this level if you have brokerconnections, since the broker data encoding will be dumped as well.

156

In the future, we’ll be setting up the verbosity levels to be more clearly defined and more useful,but you may find them useful right now.

MODE: pedanticIn normal circumstances, the Hub will report errors in program files, but a number of those errorswill not cause the Hub to exit (for instance, a reference to an undeclared operation). If you insertthe directive entry "MODE: pedantic" into your program file, the Hub will always exit when itencounters these errors. This is a good way to find typos in your server and operation names.

Fake desktop audioIf you’re building a telephony application, we strongly recommend constructing versions of youraudio server to handle desktop audio, and one to handle text I/O. Both of these should presentas similar as possible a set of functionality as the telephony server. MITRE is currently leadingan effort to develop an open-source audio solution, but it is not yet available. We’re also hopingto cooperate with other providers of open-source audio servers to settle on a common message set.

Avoiding trouble

Consider using the -assert command line argumentIf a server is starting up a listener for Hub connections (as opposed to using the listener-in-Hubcapability), it will use the first port it finds available. If the requested port isn’t available, it will trythe requested port + 1, and so on until it finds a port it can start up on. This behavior cansometimes lead to unexpected consequences. For instance, if you start up a server on port 6000and there’s already a server running on that port, the new server will happily start up on port6001. If you don’t notice this, your Hub may try to contact the wrong server, or you might not beable to make a connection at all. One way to avoid this is to use the -assert server commandline argument, which will force the server to exit if the requested port isn’t available.

Don’t forget about the initial tokenYou may remember that we talked about the initial token when we talked about how to send newmessages to the Hub. This initial token can be specified using the INITIAL_TOKEN: directiveentry in the Hub program file. While we don’t really recommend using the initial token foranything besides global initializations and simple tests and examples, you should be aware thatyou don’t have complete control over the appearance and content of the initial token. Inparticular, for historical reasons, even if you don’t specify INITIAL_TOKEN:, an initial token willbe created if there’s a program named main . If your main program is being invokedunexpectedly on a token with token index 0, you should check to see if there is some accidentaloverlap between the keys you expect and the keys in the default initial token.

157

Turn off the Hub pacifierIn all of our exercises, we’ve used the -suppress_pacifier command line argument to theHub. If this flag is not provided, the Hub will print out a period (".") each second when it is idle. Ifyou’re trying to scan back through the Hub output, many displays (X terminals and the processmonitor, for instance) will cause problems because they scroll to the end whenever there’soutput. In the process monitor, you can press the "Pause" button; but in general, if you’re havingproblems with this, use -suppress_pacifier .

Don’t forget to activate your broker clientWhen we talked about broker clients, we emphasized the importance of GalIO_SetBrokerActive.It’s important to reemphasize that if you don’t do this, the broker client will never try to read datafrom the broker server. This feature is a historical idiosyncracy which we hope to remove in thefuture, but for now, this can cause problems.

Builtin.increment_utterance and loggingYou may recall when we talked about the content of the Hub log, we told you to ignore thefeatures :BEGIN_UTT and the reference to -01 throughout the log. These elements have to dowith the way the Hub was originally designed to handle utterance boundaries. The Hub has anutterance counter, which can be incremented using the Builtin dispatch function increment_utterance . The original designers assumed that a Communicator-compliantsystem would begin with a presentation to the user (utterance -1), followed by a call to increment_utterance , and then a call to increment_utterance each time the Hubprogram writer determined an utterance boundary was reached.

The utterance counter has had three uses in the history of the Hub:

It is used to index entries in a Hub-internal database which is local to each session. You canuse the STORE: directive entry to store key-value pairs in a frame associated with thecurrent utterance counter in the current session, and use the RETRIEVE: directive entry toretrieve key-value pairs from a frame associated with (by default) the previous utterancecounter in the current session. If you have a strong commitment to stateless servers, youmay find this functionality useful as a storage device. However, it clearly depends on theutterance counter being incremented; otherwise, you’d just be able to use the session namespace. The original MIT audio server used this counter to help name the audio files logged for thegiven session. We believe there are better ways to do this; in particular, we recommend thatthe audio server determine its own file names, and report them to the Hub so that they canbe logged in the Hub log. Incrementing the utterance counter has the effect of ending an utterance in the log andstarting a new one. When we designed the XML DTD for the XML version of the Hub log,we introduced a tag called GC_TURN which correspond this boundary. However, wesubsequently moved to a different model of introducing "meaningful" information into thelog, through a system of annotation rules which "decorate" existing tags with meaningfullandmarks which indicate things like the beginning and end of turns. This was because we

158

didn’t want people to have to write their program files a particular way in order to guaranteelogs that could be scored. As a result, the GC_TURN tag is mostly ignored in the log analysisprocess, with a single exception: it’s possible to write annotation rules which rely on GC_TURNas a scoping device, and under some circumstances, it’s difficult to write your rules if you don’thave enough scoping levels. This is a flaw which ought to be addressed eventually, but has notbeen yet.

In summary, unless you want to use STORE: and RETRIEVE:, you can usually ignore theutterance counter. But you should be aware of the issues involved.

Other historical artifactsThere are a few other elements which are worth mentioning which may surprise you or can getyou into trouble:

The frame type (the c at the beginning of the frame’s string representation) is a relic of howframes were originally used (and are still used in the MIT servers): as hierarchicalrepresentations of parses and semantic interpretations. The c stands for "clause"; there arealso predicates ("p") and topics ("q" for quantificational element). Frames without the leadingtype (e.g., { foo :bar 1 } ) are interpreted as clauses. We only use the clause type inthe infrastructure proper. The Galaxy Communicator header include line for C code should always be -I$GC_HOME/include , and the main header file should always be #include <galaxy/galaxy_all.h> . This is because all the Galaxy Communicator header filesassume that the include lines end with the include/ directory, and will include other fileswhich are referenced as, say, #include <galaxy/util.h> . This is a historicalidiosyncracy which will never be changed. When you define a server, we told you that you should always specify #define USE_SERVER_DATA. Originally, dispatch functions only had one argument, the frame, andit’s still technically possible to define dispatch functions which don’t have the second callenvironment argument. There are a few servers still floating around in the Communicatorcommunity which are implemented this way. Don’t do this . You won’t have the appropriateaccess to the channels to write new messages through, or to probe the call environment inany way. We’re hoping that in time, this declaration will be superfluous, but for now, it’s crucial. Finally, you may wonder why the second argument of dispatch functions is a void * , ratherthan a call environment pointer. This, again, is a historical relic, but because of the vagariesof typing in C and the base of installed code already finished, it’s something that we can’t change.

SummaryCongratulations! You’ve completed the Galaxy Communicator tutorial. Not only should you haveenough information to understand the remainder of the documentation, you should also knowenough by now to actually get started building your own end-to-end system. Good luck!

159


160

Advanced topics

161

Coordinating Locations of Hub andServer Listeners

By default, servers start up listeners on their default ports, and the Hub contacts them.Alternatively, you can set up a listener in the Hub and have the server contact the Hub instead.In both these cases, the programmer needs to make sure that the server entries in the Hubprogram file, the default server header, and the server command line arguments all correspondto each other, so that the Hub and servers can connect. Not only can this be a difficult andunreliable procedure, but it’s also possible that multiple instances of the same Communicatorsystem might be running at the same site on different sets of machines. In this latter case, thescripts and program files for the system would typically have to be modified for each invocation(or separate versions would need to be maintained). The overhead of this chore can beextremely frustrating.

In version 3.0, we’ve implemented a simple solution to address this issue.

The server locations fileIn version 3.0, both the Hub and servers accept a command line argument -server_locations_file which names a file which describes where the listeners arelocated. The format of each meaningful line of this file whould be as follows:

<server_name> <listener_host_or_ip>:<listener_port> [hub|server]

On the Hub side, <server_name> is the name of a service type; on the server side, it’s thename the server knows itself as. When the Hub is the listener, the Hub uses the port as thelisten port for the named service type; when it’s the client, it uses the host and port as thelocation of a service provider for the named service type. So it’s important for the Hub and serverto know the server by the same name. The <listener_host_or_ip>:<listener_port> isthe location of the listener for this connection, and the third element is either hub or serverdepending on whether the listener is in the Hub or the server.

For example, the following server locations file describes a situation where the Hub has alistener on port 3600 listening for connections from a server named multiply , and a servernamed double opens up its own listener on port 5200. In addition, the multiply server will tryto find that Hub listener on bolcom.mitre.org (so the Hub had better be running on thatmachine), and the Hub will try to find the double listener on feta.mitre.org (so the double

162

server had better be running on that machine):

multiply bolcom.mitre.org:3600 hub double feta.mitre.org:5200 server

So as long as the Hub starts up on bolcom.mitre.org and is passed this server locations file,and as long as the double server starts up on feta.mitre.org and is passed this serverlocations file, and as long as the multiply server starts up some where and is passed thisserver locations file, this system configuration will initialize itself appropriately:

bolcom.mitre.org$ hub -server_locations_file double.loc --------------------------------------------------------- feta.mitre.org$ double -server_locations_file double.loc ---------------------------------------------------------- somewhere.mitre.org$ multiply -server_locations_file double.loc

The result is that it’s possible to configure a suite of servers and Hub with a single configurationfile, and no scripts or program files need to be changed.

Note : this utility does not currently provide a facility for ensuring that the listener is started on thecorrect host. The crudest version of this support would be to shut down if the specified hostwasn’t the host the listener was running on, but it’s not clear whether this would work for DHCPconnections, for instance.

Locations file syntaxIn addition to the meaningful lines described above, all lines consisting entirely of whitespace areignored, as are all lines whose first non-whitespace character is #. There can be any nonemptyamount of whitespace in between the three tokens in each meaningful line.

Server locations files and other command line argumentsThe server locations files override all other port information. On the Hub side, it supersedes thehost and port information in the program file and in the -locations command line argument.On the server side, it supersedes the information in the server declaration header file, as well asthe -port and -contact_hub command line arguments. On both the Hub and server sides,the specified port for the listener is required; the listener startup process will not search for thefirst available port, but rather fail if the specified port is not available.

Note : this override mechanism only works in the Hub when there is at most one service providerfor the named service type.

Last updated December 6, 2000

163

Using the Hub in Scriptless Mode

As of version 2.0, the Hub does not require Hub programs to run. The Hub still requires aconfiguration file, because it still must know how to do its logging and how to contact servers. Inthis document, we discuss some of the options available to the developer for scriptless interaction.

Omitting Hub programsCurrently, we have chosen to implement scriptless interaction by making Hub programs optional.That means that a Hub program file need not contain any PROGRAM: or RULE: declarations inorder for the Hub to function. The user may still choose to provide any of the global declarations; server declarations are still required, and the only way to enable logging is to use the logging parameters.

When the Hub receives a message and there is no program to claim it, the Hub searches its listof servers for a server which has declared an operation which is identical to the message name.Let’s say the Hub receives a message of this form:

{c parse :input_string "I want to fly to Denver" }

If there is a Hub program named parse , that program will claim the message and process it,regardless of whether there’s an operation named parse . In the following program file, themessage sent to the parser server will be parse_string , not parse :

SERVER: parser OPERATIONS: parse parse_string

PROGRAM: parse

RULE: :input_string --> parser.parse_string IN: :input_string OUT: :frame

However, if there were no program named parse , the Hub would pass the incoming messagedirectly to the parser server, and the parse operation would be invoked.

164

What this means is that the developer doesn’t need to choose fully scripted or fully scriptless; it’spossible to construct a "partially scripted" program file, where some new messages are claimedby programs and others are not.

If multiple servers implement the same operation, the Hub makes a random choice among theservers (there is no broadcast option). If you want to direct the operation to a particular server,use the dot notation which is used in rule targets:

{c parser.parse :input_string "I want to fly to Denver" }

LoggingPrevious to version 2.0, it was possible to log only the invocation of a program rule. This meantthat logging in scriptless mode would have been impossible. In 2.0, it is possible to log incomingmessages and their returns to the calling server. We give a brief summary here; see the logging documentation for details.

First, the TIMESTAMP: declaration now applies indiscriminately to operations and programs:


TIMESTAMP: parse parse_string

PROGRAM: parse


In our example here, the log will contain a timestamp for when the parse request arrived on theHub, as well as when the parse_string operation was sent to the parser server. Becausetimestamps are indiscriminate, the program doesn’t have to be present in order for a timestampto happen; so if the parse program is removed, the incoming timestamp will still happen.

Second, it is now possible to add LOG_IN: and LOG_OUT: declarations outside the scope of arule, and the new MESSAGE: declaration supports in and out logging for scriptless rules. Soeither of the following program files will result in a time being recorded for the incoming parse message:



165

PROGRAM: parse LOG_IN: :input_string


--------------------------------------------------



MESSAGE: parse LOG_IN: :input_string

In this way, all the possibilities for logging are still available for scriptless interaction.

Writing your own scripting moduleIn principle, it is possible to write your own scripting module, by altering the SCRIPTINGvariables in config.make and providing your own library which implements the scripting API.However, we have not tested this functionality. We do know that because we use MIT scriptingto provide server and logging declarations, your scripting module would currently have to supplyequivalent declaration facilities.

Last updated November 8, 2000

166

Setting Up Listeners in the Hub

By default, each Communicator server sets up a socket-based listener for the Hub to contact. Asof version 2.1, it is possible to reverse the direction of this connectivity, by setting up listeners inthe Hub instead. This has a number of clear advantages:

If you have more Hubs than servers, it reduces the number of open TCP listeners; It becomes trivial to implement UI elements which connect to a running Hub, either from thecommand line or as part of a process which monitors multiple potential system connections(such as phone lines)

This functionality supersedes dispatch servers, which have been removed in version 3.0.

Setting up the HubThe Hub needs to know that it’s supposed to set up a listener. To do this, use the CLIENT_PORT: declaration in your SERVICE_TYPE: declaration:

SERVICE_TYPE: parser OPERATIONS: parse CLIENT_PORT: 15000

When the Hub starts up, it will start a listener on port 15000 (or the next available port) and waitfor connections. You can have as many or as few listeners as you want, and as many servers asyou want can share a listener:

SERVICE_TYPE: parser OPERATIONS: parse CLIENT_PORT: 15000

SERVICE_TYPE: recognizer OPERATIONS: recognize CLIENT_PORT: 15000

SERVICE_TYPE: database OPERATIONS: retrieve CLIENT_PORT: 14000

167

SERVER: generator OPERATIONS: generate HOST: localhost PORT: 18000

In this example, the Hub will start up two listeners, one on port 15000 (shared by the parser andrecognizer) and one on port 14000 (used exclusively by the database), and attempt to contactthe generator on its own listener on port 18000. Listeners are shared based on the requestedclient port; so if port 15000 is taken, the Hub will start up a shared listener for the parser andrecognizer on the next available port.

Important : The Hub relies on the connecting server to identify itself by name in order todetermine which of the server types to identify the new connection with. Therefore, unlike innormal server entries, the value of SERVER: or SERVICE_TYPE: must match the name thecontacting server knows itself as.

Setting up the serverSetting up the server is also simple. It involves the server command line arguments -contact_hub and -session_id . The -contact_hub argument provides a space-delimitedsequence of host:port entries to contact Hubs at; the -session_id argument locks theserver connection to the named user session (and creates the session if it does not alreadyexist). So the parser or database might contact the Hub configured above as follows:

% parser -contact_hub "localhost:15000"

If the server in question is a UI element (say, an audio connection), you might want to associateit with a specific session:

% audio -contact_hub "localhost:2400" -session_id 20000711session1

If you use the -session_id command line argument, it’s equivalent to setting all the sessionlock flags for this server.

You can contact several Hubs at once:

% database -contact_hub "localhost:15000 remotehost:25000"

The server will poll the connection location until it establishes a connection, and will start pollingagain when the Hub disconnects; it’s possible to customize this behavior if you write your ownmain function.

168

The handshake protocolThe initial handshake between the Hub and servers is slightly different in the listener-in-Hubcase. In particular, since the Hub needs to know immediately what the server’s name is, thehandshake begins with the server sending a message to the Hub, which allows the Hub to set upthe connection appropriately or reject it. Once this stage of the handshake is complete, thenormal initialization procedure continues, with the Hub sending the reinitialize message to the server.

API supportFor API support for this functionality, see the server structure documentation.


169

The Timed Task Loop

In order to control the execution of events, MIT provides a timing loop which it uses in both theHub and servers. We document this loop here. As of 3.0, the functions Gal_AddTimedTask, etc.are no longer recommended; use Gal_AddTask, etc. instead.

What the timed task loop doesIn the MIT main loop, the timed task loop schedules and executes server and Hub pollingevents. When the timed task loop fires a poll, it may be scheduled to fire again if appropriateconditions are met. The loop can be used to schedule other events at the programmer’sdiscretion. The loop runs until Gal_TimedTasksLoopExit is called. We describe the structure ofthe MIT main loop here. In some cases, the programmer might choose to replace the MIT mainloop, and schedule the polls and other events by other means; the tools to accomplish this aredescribed here.

It is now possible to use a version of the timed task API to produce threads instead.

Setting up timed tasksGal_TaskPkg *Gal_AddTask (void (*task)(Gal_TaskPkg *), void *refcon, long num_millisecs,int read_blocking_available, void (*cleanup_fn)(void *)) Use this function to add a timed task. When threads are enabled, this function will create athread and associate a task with it; when threads are not being used, this function adds a task tothe list of timed tasks to fire. The task is a function which takes a Gal_TaskPkg * structure as anargument (in fact, the one returned by Gal_AddTask), and returns void. The num_millisecs ishow long in milliseconds after the task is added (or, if the loop hasn’t been started, how longafter the loop is started) the task should be executed. The read_blocking_available argumentshould indicate whether the task has the capacity to do a blocking read in the threaded case; ifread blocking is available, num_milliseconds is ignored in the threaded case.

The Gal_TaskPkg * structure argument to the task provides five things:

A hook for removing the task (see Gal_RemoveTask). A hook for refiring the task (see Gal_ReAddTask). A way of accessing the refcon, which is available via the Gal_TaskPkgData function.

170

A way of determining the read blocking properties that the task should respect, via the Gal_TaskPkgBlocking function. The return value of this function indicates whether the taskshould read input in blocking or non-blocking mode. In the timed task case, the argument willalways be 0; in the threaded case, it will be the same as read_blocking_available. Whatthese two features together do is insure that blocking reads are done exactly when it is possibleto do so (when threading is enabled and blocking reads are possible). Each task is responsibleitself for responding appropriately to the value of this function (so for instance, they themselvesmust set the blocking mode on the file descriptors they read from); all Gal_AddTask() does ispass the flag around. A way of determining why the task was fired (see Gal_TaskPkgRunReasons).

Finally, the cleanup_fn is run on refcon if refcon is present, whenever the task is not refired orwhen refcon or cleanup_fn is replaced during refiring (if refiring passes the identical arguments,the function is not run). Immediately before the task is executed, it is removed from the list oftimed tasks (or, in the threaded case, it is marked for destruction); so typically the last thing the task will do is call Gal_ReAddTask if it intends the task to be refired.

Gal_TaskPkg *Gal_AddTaskWithSocketIO (void (*task)(Gal_TaskPkg *), void *refcon,long num_millisecs, int read_blocking_available, GAL_SOCKET *read_socket, GAL_SOCKET *write_socket, void (*cleanup_fn)(void *)) Gal_TaskPkg *Gal_AddTaskWithFileIO (void (*task)(Gal_TaskPkg *), void *refcon, long num_millisecs, int read_blocking_available, FILE *read_file, FILE *write_file, void (*cleanup_fn)(void *)) Adds the task with sensitivity to I/O on sockets and file descriptors, respectively. Otherwise,identical to Gal_AddTask(). Note that file I/O polling is not available on Win32.

typedef int (*Gal_TaskConditionFn )(void *caller_data); Gal_TaskPkg *Gal_AddTaskExtended (void (*task)(Gal_TaskPkg *), void *caller_data, long num_millisecs, int read_blocking_available, GAL_SOCKET *read_socket, GAL_SOCKET *write_socket, GAL_SOCKET *err_socket, FILE *read_file, FILE *write_file, FILE*err_file,Gal_TaskConditionFn condition, void (*cleanup_fn)(void *)) Adds the task with sensitivity to the entire range of elements, including I/O on sockets and filedescriptors, the additional possibility of error streams and task conditions. Task conditions arepassed the caller_data, and if the task condition returns nonzero, the task will be executed inadvance of its normal time expiration. Otherwise, identical to Gal_AddTask(). Note that file I/Opolling is not available on Win32.

void Gal_ReAddTask (Gal_TaskPkg *p, void *refcon, long num_millisecs, int read_blocking_available, void (*cleanup_fn)(void *)) This function is identical to Gal_AddTask(), except it is to be used when tasks are supposed tobe refired, as described in Gal_AddTask(). In the non-threaded case, it just adds theGal_TaskPkg * structure back to the list of timed tasks; in the threaded case, the thread iscontinued. This function is intended to be called from inside the task which p was passed to. Ifthe task is not refired, the task package is freed; the programmer is responsible for managing thememory associated with refcon.

171

void Gal_ReAddTaskWithSocketIO (Gal_TaskPkg *p, void *refcon, long num_millisecs,int read_blocking_available, GAL_SOCKET *read_socket, GAL_SOCKET *write_socket, void (*cleanup_fn)(void *)) void Gal_ReAddTaskWithFileIO (Gal_TaskPkg *p, void *refcon, long num_millisecs, int read_blocking_available, FILE *read_file, FILE *write_file, void (*cleanup_fn)(void *)) Re-adds the task with sensitivity to I/O on sockets and file descriptors, respectively. Otherwise,identical to Gal_ReAddTask(). Note that file I/O polling is not available on Win32.

void Gal_ReAddTaskExtended (Gal_TaskPkg *p, void *caller_data, long num_millisecs,int read_blocking_available, GAL_SOCKET *read_socket, GAL_SOCKET *write_socket,GAL_SOCKET *err_socket, FILE *read_file, FILE *write_file, FILE *err_file,Gal_TaskConditionFn condition, void (*cleanup_fn)(void *)) Re-adds the task with sensitivity sensitivity to the entire range of elements, including I/O onsockets and file descriptors, the additional possibility of error streams and task conditions.Otherwise, identical to Gal_ReAddTask(). Note that file I/O polling is not available on Win32.

int Gal_TaskPkgRunReasons (Gal_TaskPkg *pkg) Returns the logical OR of the reasons the package was run. Possible reasons are:

value description

GAL_SOCKET_READABLE The task was fired because a read socket had data

GAL_SOCKET_WRITABLE The task was fired because a write socket had data

GAL_SOCKET_ERR_READABLEThe task was fired because an error socket had data

GAL_FILE_READABLE The task was fired because a read file pointer had data

GAL_FILE_WRITABLE The task was fired because a write file pointer had data

GAL_FILE_ERR_READABLE The task was fired because an error file pointer had data

GAL_TIMER_EXPIREDThe task was fired because it had slept for the requirednumber of milliseconds

GAL_CONDITION_SATISFIEDThe task was fired because its task condition functionreturned nonzero

GAL_THREAD_READABLEThe task was fired because a blocking read had data in thethreaded case

So you can check, for instance, to see if your task fired because of a readable file as follows:

if (Gal_TaskPkgRunReasons(pkg) & GAL_FILE_READABLE) { ... }

172

voidGal_RemoveTask (Gal_TaskPkg *task_id) Removes the task task_id.

int Gal_TaskPkgBlocking (Gal_TaskPkg *pkg) If 1, the task should set its read file descriptors to blocking mode; if 0, non-blocking.

void *Gal_TaskPkgData (Gal_TaskPkg *pkg) Returns the data associated with the package. This is the same as the value of the refconargument to Gal_AddTask and Gal_ReAddTask.

Managing timed tasksThese functions manage the invocation and operation of the timed task loop.

void Gal_TimedTasksLoop (void) This function initiates the timed task loop. It loops until Gal_TimedTasksLoopExit is called.

void Gal_TimedTasksLoopExit (void) This function halts the timed tasks loop. All tasks in the queue are completed, but no other tasksare queued.

typedef void (*Gal_IdleFunction )(void *client_data); int Gal_AddIdleFunction (Gal_IdleFunction func, void *client_data) Adds a function to the list of functions which the timed task loop will execute when there are notasks to fire. The client_data argument is passed to func when it’s called. Unlike tasks, which areremoved after they’re fired, idle functions remain installed until they’re removed with Gal_RemoveIdleFunction().

void Gal_RemoveIdleFunction (Gal_IdleFunction func) Removes func from the list of functions which the timed task loop executes when there are notasks to fire.

void Gal_RunIdleFunctions () Runs the idle functions. Called by the timed task loop, but can also be called explicitly.

Internal functionsint Gal_TimedTasksLoopHandler (struct timeval *tv) This is the internal function which determines when to execute the timed tasks.

173

ThreadsThe Galaxy Communicator library is thread-safe (we believe) on a number of platforms (see the thread notes). One way we’ve tested this is by running multiple-connection servers in threadedmode. We have made this experimental facility available to the community via the -threadargument described in the generic server arguments. This argument is available when threadsare enabled. The following functions can be used to exploit this functionality.

If you want to use the timed task loop in a threaded context, it should work (we haven’t tested itextensively). In particular, you can only run the timed task loop from one thread at a time, andyou can only shut down the loop from that thread. However, be warned that if you use the -thread argument, the timed task loop will not be started automatically; you will have to start it yourself.

void Gal_EnableTimedTaskThreads () This function controls whether tasks added via Gal_AddTask() and Gal_ReAddTask() arehandled via timed tasks or via threads. Once timed task threads are enabled, they can onlydisabled by Gal_EndTasks(). Gal_EnableTimedTaskThreads() is the function which is invokedwhen you provide the -thread argument to servers when threads are enabled. If the user iswriting his/her own main() and wants to avoid the timed task loop for server, connection andbroker handling but would like threads to handle these facilities, the user can enable this facilitywith this function.

int Gal_TimedTaskThreadsEnabled () Returns 1 if timed tasks threads have enabled via Gal_EnableTimedTaskThreads(), 0 otherwise.The user should never need to call this function.

void Gal_TimedTaskLoopThreadWaiter (void) Waits until all the tasks are completed. This function is the threaded equivalent of Gal_TimedTasksLoop. The main server loop uses it in the threaded case to delay the actions ofthe main thread so that GalSS_RunServer only returns when the server is finished.

void Gal_EndTasks (int immediate) This is a more general version of Gal_TimedTasksLoopExit. It causes one of two types of exits:immediate and deferred. An immediate exit will cancel tasks at the most immediate possible exitpoint, while a deferred exit will cancel tasks which are already "scheduled" to happen. If immediate is non-zero, this call causes an immediate exit, otherwise a deferred exit. Thisfunction halts both threads (if running) and the timed task loop (if running).

void Gal_MaybeEndTask (int immediate, int deferred) This function introduces an exit point into a long-running task. For instance, if your (threaded)task is looping forever and doing blocking reads, you might choose to call Gal_MaybeEndTask()to check to see if another thread has scheduled an exit via Gal_EndTasks. If it finds animmediate scheduled exit and immediate is nonzero, or if it finds a deferred scheduled exit and deferred is nonzero, the thread will terminate.

174

For backward compatibilityThese functions represent an older, simpler way of invoking timed tasks. They still work, but theydo not allow you to take advantage of many of the newer features. We have no plans todeprecate them, but we strongly discourage their use.

int Gal_AddTimedTask (void *task, void *refcon, long num_millisecs) Use this function to add a timed task. The task is a function which takes refcon as its argumentand returns void. The num_millisecs is how long in milliseconds after the task is added (or, if theloop hasn’t been started, how long after the loop is started) the task should be executed. Oncethe task is executed, it is removed from the list of timed tasks; so typically the last thing the taskwill do is call Gal_AddTimedTask again. This function is implemented in terms of the internalfunction Gal_AddTask.

int Gal_AddTimedTaskWithFileIO (void *task, void *refcon, long num_millisecs, FILE *read_file, FILE *write_file) Use this function to add a timed task which is sensitive to input and output on file descriptors. Ifany of the elements in either of the file descriptor sets is ready for reading and writing before num_milliseconds has elapsed, the task will nonetheless be executed. Note that file I/O polling isnot available on Win32.

int Gal_AddTimedTaskWithSocketIO (void *task, void *refcon, long num_millisecs,GAL_SOCKET *read_socket, GAL_SOCKET *write_socket) Use this function to add a timed task which is sensitive to input and output on sockets. If any ofthe elements in either of the file descriptor sets is ready for reading and writing before num_milliseconds has elapsed, the task will nonetheless be executed.

int Gal_RemoveTimedTask (void *task, void *refcon) Removes the given timed task. Note that tasks are distinguished by task and refcon, so it’s not agood idea to define multiple timed tasks with the same task and refcon.


175

Managing Session Information andUsing Continuations

What is a session? Starting and ending sessions Keeping information straight Administrative functions Continuations Hub continuations

What is a session?The Galaxy Communicator infrastructure is designed to support multiple simultaneous sessions .By session we mean an individual conversation with a Communicator system, such as thesequence of exchanges between the time the system answers the phone and greets anindividual user and the time the system or user hangs up. A single Communicator system mightsupport a large number of phone lines, and thus a large number of simultaneous sessions.

Although some servers connected to the Hub might be dedicated to a particular session (such aswhen they connect using -contact_hub and -session_id), most servers support multiple sessionsthrough the same connection to the Hub. So, for instance, a single parser may be responsible forparsing all the user utterance strings for all the sessions currently connected to the Hub.

Starting and ending sessionsThe Hub will start a session whenever it receives a session ID it does not recognize. If logging isenabled in the Hub program file, the Hub will open a log for that session as well. The Builtinserver function new_session (somewhat misnamed, at this point) can be called to reset thesession’s utterance counter and open a new log. Reusing session IDs in this way is not recommended , however, it is supported for convenience. The most common case of reuse iswhen no session ID is actually declared, and the Hub uses the Default session ID as a result.

If new_session opened a new log every time it was called, any program which calls it would endup writing a very short log which covered the time from when the session was created(essentially, when the Hub first saw the session ID) to when new_session was called. Since this

176

behavior would be unavoidable when new_session is used correctly, new_session has beenimplemented not to open a new log the first time it is called on a given session .

If you want to end a session explicitly, you can call the Builtin server function end_session.

If you want to control the memory state of a given session, you can do that from the server as well.

Keeping information straightIn the documentation on adding a server, we noted that the call environment structure embodiesthe information required to communicate properly with the Hub. These call environments arepassed to dispatch functions, where they can be used to provide replies or send new messagesto the Hub. But dispatch functions aren’t restricted to providing a reply to an incoming messageor sending new messages; they can also set up callbacks or timed tasks which may send newmessages later. When these messages are sent from callbacks or timed tasks, the callenvironment in the dispatch function needs to be saved away cleanly, so that it isn’t freed untilthe new message is sent.

To support this functionality, the Galaxy Communicator library provides environment-aware broker callback and timed task setup, as well as a way of maintaining an environment throughsuccessive calls to dispatch functions. In this section, we describe some of these functions.

void GalSS_TaskSetEnvironment (Gal_TaskPkg *p, GalSS_Environment *env) Sets the environment associated with the task p to env.

GalSS_Environment *GalSS_TaskGetEnvironment (Gal_TaskPkg *p) Returns the environment associated with the task p.

Gal_TaskPkg *GalSS_EnvAddTask (GalSS_Environment *env, void (*task)(Gal_TaskPkg*), void *caller_data, long num_millisecs, int read_blocking_available, GAL_SOCKET *read_socket, GAL_SOCKET *write_socket, GAL_SOCKET *err_socket, FILE *read_file, FILE *write_file, FILE*err_file, Gal_TaskConditionFn condition, void (*cleanup_fn)(void *)) This function provides all the functionality of Gal_AddTaskExtended, but also saves away thecall environment env for access using the function GalSS_TaskGetEnvironment. A task thusestablished can be refired in any of the normal ways; see the timed task documentation for details.

At times, connections may be associated with UI elements, which can issue new messages. Sothe UI itself is an element outside the scope of dispatch functions where it would be helpful tohave a call environment to use to issue new messages. It’s not enough to set up theenvironment when the connection is established; crucial properties of the UI interaction, such asthe session ID, may be changed by other servers in the course of evaluation. The safest thing todo is to create an environment for the UI when the connection is established, update it everytime a dispatch function is called, and free the environment when the connection is shut down.The function GalSS_EnvMaintainInLocation does this for you.

177

void GalSS_EnvMaintainInLocation (GalIO_CommStruct *gcomm, char *initial_session_id, GalSS_Environment **env_loc) The connection gcomm is the connection to the Hub which is associated with a UI element forthe current session, and hosts the dispatch functions which are fired. This function seeds thelocation env_loc with a new environment and updates its session ID to initial_session_id. It setsa dispatch function callback via GalIO_AddConnectionDispatchFnCallback to keep the locationcurrent, and sets up a shutdown callback to free the environment.

Administrative functionsThese are the functions the infrastructure uses to manage call environments. You should neverneed to use these functions directly.

GalSS_Environment *GalSS_EnvCreate (GalIO_CommStruct *gcomm) Creates an environment from the connection gcomm.

void GalSS_EnvUpdateSessionID (GalSS_Environment *env, char *session_id) Sets the session_id as the session for this environment. The session_id is copied. This functioncan safely be called repeatedly on the same call environment to change the session context.

char *GalSS_EnvGetSessionID (GalSS_Environment *env) Returns the session_id for this environment. The session_id is not copied. It may be NULL.

void GalSS_EnvLock (GalSS_Environment *env) Increments the reference counter on the environment env to indicate that it should not be freed.

void GalSS_EnvUnlock (GalSS_Environment *env) Decrements the reference counter on the environment env. When the reference counter reaches0, the environment is freed.

int GalSS_EnvReturnRequired (GalSS_Environment *env) Returns 1 if the call environment represents a dispatch function invocation for which a return isexpected, or 0 if it does not.

ContinuationsIn the vast majority of cases, the server’s reply to a message from the Hub coincides with thereturn value from the dispatch function invoked by that message. However, as we’ve alreadyseen, we can provide that return value before the dispatch function ends (by calling one of thefunctions GalSS_EnvDestroyToken, GalSS_EnvError, or GalSS_EnvReply). In addition, it is alsopossible to postpone returning a value and provide it later. This is desirable in a number of situations.

178

Situation 1

Consider a case where the Hub sends a message to a recognizer that audio input is availablevia brokering. The recognizer server, in the appropriate dispatch function, sets up an incomingbroker to capture the audio and send the result to the Hub. In the typical arrangement,exemplified by the example for GalSS_EnvBrokerDataInInit, the server writes a new message tothe Hub. This new message is associated with a new token; that is, the request for recognitionoccurs at the end of one Hub program, and the recognition results invoke another Hub program:

PROGRAM: FromAudio

...

RULE: :audio_host & :audio_port & :call_id --> Recognizer.Recognize IN: :audio_host :audio_port :call_id OUT: none!


RULE: :input_string --> Parser.Parse ....

However, the programmer might prefer that the call to the recognizer appear to the Hub to be a synchronous call; that is, that the input string be the reply to the Recognize message:

PROGRAM: FromAudio

...

RULE: :audio_host & :audio_port & :call_id --> Recognizer.Recognize IN: :audio_host :audio_port :call_id OUT: :input_string

RULE: :input_string --> Parser.Parse ....

This is easy to do, using the function GalSS_EnvPostponeReply.

int GalSS_EnvPostponeReply (GalSS_Environment *env) Informs the Hub that the response to its message will be delayed, but that in the meantime it’savailable for other incoming messages. It does this by sending a pacifier message of type GAL_POSTPONE_MSG_TYPE.

The programmer would then set up the broker as follows:

Gal_Frame Recognize(Gal_Frame f, void *server_data) { GalSS_Environment *env = (GalSS_Environment *) server_data;

179

/* ... */ GalSS_EnvPostponeReply(env); GalSS_EnvBrokerDataInInit(env, host, port, f, env_recognition_handler, env_recognition_finalizer, (void *)NULL, 0); return (Gal_Frame) NULL; }

static void env_recognition_handler(GalSS_Environment *env, GalIO_BrokerStruct *broker_struct, void *data, Gal_ObjectType data_type, int n_samples) { if (data_type == GAL_INT_16) { /* ... gather the audio ... */ } }

static void env_recognition_finalizer(GalSS_Environment *env, GalIO_BrokerStruct *broker_struct) { /* Send the reply. The name doesn’t matter because it will be set by the environment as the original name of the incoming message. */ Gal_Frame f = Gal_MakeFrame("foo", GAL_CLAUSE);

/* ... */

Gal_SetProp(f, ":input_string", Gal_StringObject(recognized_string)); GalSS_EnvReply(env, f); }

Observe that the dispatch function postpones the response, and then the environment-awarefinalizer explicitly sends the reply when recognition is done. This same strategy could be usedwith timed task callbacks as well, if the appropriate situation arose.

Situation 2

Consider a case where the server sends a request to the Hub using GalSS_EnvDispatchFrame.This function returns when the Hub returns a reply. However, during that time, the server isinaccessible for other dispatch function invocations from that Hub, and in addition, a deadlockwill result if the Hub needs to contact the originating server in the course of satisfying the request(if, for instance, the program that is invoked by the incoming message ends up calling a dispatch

180

function in the originating server). In this situation, the programmer might prefer to dispatch therequest, and postpone the processing of the reply to the request using a continuation funciton.The programmer can accomplish this using the function GalSS_EnvDispatchFrameWithContinuation.

typedef Gal_Frame (*GalSS_ContinuationFn )(Gal_Frame, GalIO_MsgType,GalSS_Environment *, void *); This is the type of the continuation function.

int GalSS_EnvDispatchFrameWithContinuation (GalSS_Environment *env,Gal_Frame frame, GalSS_ContinuationFn fn, void *continuation_state, void (*continuation_state_free_fn)(void *)) This function sends frame to the Hub using the environment env, and indicates that it expects areply. It then notifies the Hub that the dispatch server response will be postponed using GalSS_EnvPostponeReply, and stores away the environment, along with the continuationfunction fn and an arbitrary state continuation_state, which the programmer can use to storearbitrary data for use in the continuation function. The data will be freed using the continuation_state_free_fn. The continuation function is invoked as the continuation of thedispatch function; in particular, if the continuation function returns a frame, it will be treated asthe reply to the original dispatch function. It is called with the frame and message type of thereply to the original request to the Hub, the environment env, and the continuation_state.

Hub continuationsUnder some circumstances, you might want to do something comparable on the Hub scriptingside. In some cases, you may wish to send a message to a server, and then monitor that serverfor a new message which you want to treat as a reply or error (if, for instance, the server inquestion uses an asynchronous programming paradigm and you prefer a synchronous one inyour Hub scripts). This allows you to achieve the same results as in situation 1 above withoutmodifying the server. There are two ways to do this.

Hub continuations in the scripting languageThe best way of doing this is to use the CONTINUE_REPLY: or CONTINUE_ERROR: Hub directives:

PROGRAM: FromAudio

...

RULE: :audio_host & :audio_port & :call_id --> Recognizer.Recognize IN: :audio_host :audio_port :call_id CONTINUE_REPLY: {c FromRecognizer } OUT: :input_string

181


The Hub will monitor the Recognizer server for a new message (in the current session) named FromRecognizer , and treat it as the reply to the call to Recognizer.Recognize . Theprocessing proceeds normally from that point. The CONTINUE_ERROR: directive does thesame for an error reply. Both directives can be repeated in a single rule.

Hub continuations using the Builtin serverThe other way of doing this, which is more powerful but less focused, is to use the Builtinfunction hub_continue. This function should be called immediately after a rule which does notwait for a reply, so it will be processed essentially simultaneously. This function allows theprogrammer to specify a list of possible reply and continuation frames, as well as a service typeand/or service provider to monitor. While on the one hand, this mechanism allows theprogrammer to treat as replies new messages which come from other servers, it does not allowthe programmer to automatically "inherit" the identity of the server the previous message wassent to:

PROGRAM: FromAudio

...

RULE: :audio_host & :audio_port & :call_id --> Recognizer.Recognize IN: :audio_host :audio_port :call_id OUT: none!

RULE: --> Builtin.hub_continue IN: (:reply_matches ( {c FromRecognizer } ) ) (:service_type "Recognizer") OUT: :input_string


The Hub will monitor any provider of the Recognizer service for a new message (in the currentsession) named FromRecognizer , and treat it as the reply to the call to Builtin.hub_continue . The processing proceeds normally at that point.


182

Controlling Hub Server and SessionProperties from the Server

The Hub maintains state information for all its sessions and service providers. This informationcan be updated in the Hub, but it can also be updated by the server.

Server propertiesThe Hub keeps track of a set of properties for each service provider. These properties are notnecessarily known by the servers themselves, but the Hub can use them locally to help selectthe appropriate service provider given the current session and token states. It is possible for theserver to modify the properties the Hub associates with it.

Gal_Frame GalIO_ServerProperties (GalIO_ServerStruct *server) Returns the frame constituting the server properties as the server currently knows them locally.Not thread-safe; use only in _GalSS_init_server.

void GalIO_AddServiceType (GalIO_ServerStruct *server, char *stype) Use in _GalSS_init_server to add a service type to the default type (which is identical to the server name).

Gal_Frame GalSS_EnvGetServerProperties (GalSS_Environment *env, char **keys) Fetches the properties from the Hub which the Hub has associated with the current server. keysis a NULL-terminated array of strings. The return value contains a key-value pair for each key in keys that the Hub has a value for; the name of the returned frame is not interesting. This functionultimately calls the Builtin server function get_properties.

void GalSS_EnvSetServerProperties (GalSS_Environment *env, Gal_Frame properties) Updates the properties the Hub associates with the current server with the key-value pairs in properties. The name of properties is ignored by the Hub. The local server properties are alsoupdated; see GalIO_ServerModifyProperties. This function ultimately calls the Builtin serverfunction modify_properties.

183

void GalSS_EnvDeleteServerProperties (GalSS_Environment *env, char **keys) Deletes the properties in keys from the properties the Hub associates with the current server. keys is a NULL-terminated array of strings. The local server properties are also updated; seeGalIO_ServerModifyProperties. This function ultimately calls the Builtin server function modify_properties.

void GalSS_EnvModifyServerProperties (GalSS_Environment *env, Gal_Frame properties_to_set, char **properties_to_delete) A combination of GalSS_EnvDeleteServerProperties and GalSS_EnvSetServerProperties. Theproperties in properties_to_delete are deleted before the properties in properties_to_set areadded. This function ultimately calls the Builtin server function modify_properties.

void GalIO_ServerModifyProperties (GalIO_ServerStruct *server, Gal_Frame new_properties, char **delete_properties) Modifies the server properties locally. The name of the new_properties frame is ignored. Theproperties in delete_properties are deleted before the properties in new_properties are added. delete_properties is a NULL-terminated array of strings. This function is called by GalSS_EnvModifyServerProperties, etc.

Session propertiesThe Hub also keeps track of a set of properties for each session. Some of these properties are preset, such as those associated with logging. However, the programmer can also control themfrom the server.

Gal_Frame GalSS_EnvGetSessionProperties (GalSS_Environment *env, char **keys) Fetches the properties from the Hub which the Hub has associated with the current session. keys is a NULL-terminated array of strings. The return value contains a key-value pair for eachkey in keys that the Hub has a value for; the name of the returned frame is not interesting. Thisfunction ultimately calls the Builtin server function get_properties.

void GalSS_EnvSetSessionProperties (GalSS_Environment *env, Gal_Frame properties) Updates the properties the Hub associates with the current session with the key-value pairs in properties. The name of properties is ignored by the Hub. This function ultimately calls the Builtinserver function modify_properties.

void GalSS_EnvDeleteSessionProperties (GalSS_Environment *env, char **keys) Deletes the properties in keys from the properties the Hub associates with the current session. keys is a NULL-terminated array of strings. This function ultimately calls the Builtin serverfunction modify_properties.

void GalSS_EnvModifySessionProperties (GalSS_Environment *env, Gal_Frame properties_to_set, char **properties_to_delete) A combination of GalSS_EnvDeleteSessionProperties and GalSS_EnvSetSessionProperties.

184

The properties in properties_to_delete are deleted before the properties in properties_to_set areadded. This function ultimately calls the Builtin server function modify_properties.

void GalSS_EnvSetSession (GalSS_Environment *env, char *session_name, int lock_info) Changes the session ID for the current environment and sends a Hub message whichguarantees that the session will exist. This function calls the Builtin function set_session. Thelock information should be a logical OR of the following flags:

flag description

GAL_SERVER_READS_ONLY_FROM_SESSIONthe Hub will not send any messages forany sessions other than this one to this server

GAL_SESSION_WRITES_ONLY_TO_SERVER the Hub will ensure that any messages inthis session which might be sent to thisserver will be

GAL_SERVER_WRITES_ONLY_TO_SESSIONthe Hub will assume that any messagereceived from this server is assigned tothis session

GAL_PERMANENT_LOCKthe Hub will not free these locks except inresponse to another call to Builtin.set_session.

If lock_info is not -1, this function frees all previous locks; otherwise, it inherits the locks from theprevious session, if possible.


185

Using a Different Main Loop

We can use the elements of the toplevel server loop to provide Communicator serverfunctionality to other systems which have their own main loops, such as scripting languageinterpreters (Tcl and Python, for instance), window systems (Tk and X, for instance), anddistributed object systems (e.g., Xerox PARC’s ILU). In this document, we show how to do this.

External loop recordsIn our anatomy of the main loop, we showed how the default server is set up, and how thedispatch functions access the connection to the Hub. However, we didn’t describe how thefunctions actually get called, except to make reference to the timed task loop. But if you can’tuse the timed task loop, because you’re using a single-threaded application which is using a GUIor CORBA event loop, you need to set up your server in another way. In version 3.1, we’veencapsulated this process in a new set of utilities based an an object called an external loop record (ELR). We have tested this mechanism with our own external main loop example, as wellas with the Tk, GTK and Motif window systems.

Recall the Communicator main loop in the timed task case:

/* The main() here is essentially the same main() as in the Communicator library. */

int main(int argc, char **argv) { GalIO_ServerStruct *server;

server = GalSS_CmdlineSetupServer(argc, argv); if (!server) { GalUtil_Fatal("Failed to set up server!\n"); } GalSS_StartAndRunServer(server); exit(0); }

Here’s the main function from simple_mainloop_elr.c, in the main loop example whichdemonstrates the use of ELRs:

186

int main(int argc, char **argv) { Looper *l = SM_NewLooper(); GalSS_ELR *elr = GalSS_ELRSetupServer((GalSS_ServerArgs *) NULL, argc, argv, _simple_mainloop_set_timer, _simple_mainloop_unset_timer, _simple_mainloop_set_fd, _simple_mainloop_unset_fd, _simple_mainloop_behavior_fn, (void *) l, NULL, 1);

if (!elr) { exit(1); } SM_Mainloop(l); exit(0); }

The basic steps in this latter case are

performing initializations for the main loop you’re using; creating the ELR; starting your main loop.

The ELR object encapsulates several types of information.

First, in order to configure an external main loop to handle Communicator events appropriately,the external main loop must support both timer-triggered events and file-descriptor-triggeredevents. The callbacks for a Communicator-compliant server cannot rely exclusively on filedescriptors, because sometimes there’s material in the incoming or outgoing internal queueswhich a file descriptor poll would not catch. In principle, timer-triggered events would probablysuffice, but the ELR registration is not configured to guarantee that possibility yet. So the ELRobject must record functions to set up and cancel both types of events, using the chosen mainloop. Here are the appropriate functions for our simple main loop example. Let’s start with timers:

static void _simple_mainloop_timer_callback(void *client_data) { GalSS_ELRDoCallback((GalSS_ELR *) client_data, GALSS_ELR_TIMER); }

void *_simple_mainloop_set_timer(GalSS_ELR *elr, int ms) { return (void *) SM_AddTimerCallback((Looper *) GalSS_ELRGetLoopData(elr),

187

ms, _simple_mainloop_timer_callback, (void *) elr); }

void _simple_mainloop_unset_timer(GalSS_ELR *elr, void *tag) { SM_RemoveTimerCallback((TimerCallback *) tag); }

The setting function takes an ELR object and a timer duration in milliseconds, and returns a tagthat can be used to cancel the timer. The unsetting function takes the ELR object and the tag. Inorder to complete the implementation, the setter needs to refer to a callback function which hasa function signature appropriate for the external main loop you’re using. This function should call GalSS_ELRDoCallback. Now we do the same thing for file descriptors:

static void _simple_mainloop_fd_callback(void *client_data) { GalSS_ELRDoCallback((GalSS_ELR *) client_data, GALSS_ELR_FD); }

void *_simple_mainloop_set_fd(GalSS_ELR *elr, GAL_SOCKET fd) { SM_AddFDCallback((Looper *) GalSS_ELRGetLoopData(elr), fd, _simple_mainloop_fd_callback, (void *) elr); return (void *) fd; }

void _simple_mainloop_unset_fd(GalSS_ELR *elr, void *tag) { SM_RemoveFDCallback((Looper *) GalSS_ELRGetLoopData(elr), (GAL_SOCKET) tag); }

In addition, the ELR object records whether the timer in the main loop is persistent or not. Insome cases, a timer resets itself automatically after it’s fired, and continues firing until it’sexplicitly cancelled (that is, it’s persistent). In other cases, the timer must be explicitly reset (thatis, it’s not persistent). For instance, tasks in the Communicator timed task loop are not persistent.

Second, the ELR object records behavior associated with Communicator events. For example,you may require special things to happen when a connection starts up or shuts down (you mayneed to store it somewhere that the GUI can get to it, perhaps), or when a server starts up orshuts down. This is the behavior function in the simple main loop example:

188

void _simple_mainloop_behavior_fn(GalSS_ELR *elr, int event) { switch (event) { case GAL_SERVER_LISTENER_SHUTDOWN_EVENT: GalUtil_Warn("The server has failed.\n"); SM_RemoveAllFDCallbacks((Looper *) GalSS_ELRGetLoopData(elr)); SM_RemoveAllTimerCallbacks((Looper *) GalSS_ELRGetLoopData(elr)); break; case GAL_SERVER_DESTRUCTION_EVENT: GalUtil_Warn("The server has been destroyed.\n"); SM_LooperExit((Looper *) GalSS_ELRGetLoopData(elr)); break; } }

In this particular case, the main loop is shut down when the server exits.

Finally, the ELR object records arbitrary external data which the ELR object may need to do itsjob. In this case, we see in the behavior function that the ELR object requires a reference to theexternal main loop.

enum {GALSS_ELR_TIMER, GALSS_ELR_FD}; typedef void *(*GalSS_ELTimerSetFn )(GalSS_ELR *, int ms); typedef void *(*GalSS_ELFDSetFn )(GalSS_ELR *, GAL_SOCKET fd); typedef void (*GalSS_ELUnsetFn )(GalSS_ELR *, void *tag); typedef void (*GalSS_ELBehaviorFn )(GalSS_ELR *, int condition);

GalSS_ELR *GalSS_ELRSetupServer (GalSS_ServerArgs *external_arg_pkg, int argc,char **argv, GalSS_ELTimerSetFn timer_set_fn, GalSS_ELUnsetFn timer_unset_fn,GalSS_ELFDSetFn fd_set_fn, GalSS_ELUnsetFn fd_unset_fn, GalSS_ELBehaviorFn behavior_fn, void *loop_data, void (*loop_data_free_fn)(void *), int timer_is_persistent) This function performs a number of tasks. First, it analyzes the command line arguments usingthe argument package tools, treating argc, argv, and external_arg_pkg as in GalSS_ExtractCmdlineServerArgs. It fixes the loop type to be external using GalSS_SAFixLoopType. It then calls GalSS_SetupServer, builds the ELR structure using GalSS_ELRCreate (the five function arguments and timer_is_persistent), sets the loop data (ifpresent) using GalSS_ELRSetLoopData (the loop_data and loop_data_free_fn arguments), andcalls GalIO_ServerStart. At this point, the server is configured and running, and the appropriatecallbacks should be enabled in the external main loop.

void GalSS_ELRDoCallback (GalSS_ELR *elr, int timer_or_fd) This function should be invoked by all external loop timer and file descriptor callbacks. The timer_or_fd argument should be GALSS_ELR_TIMER or GALSS_ELR_FD as appropriate.

189

void GalSS_ELRSetLoopData (GalSS_ELR *elr, void *loop_data, void (*loop_data_free_fn)(void *)) This function sets the arbitrary ELR data. If loop_data is not NULL and loop_data_free_fn is notNULL, loop_data_free_fn will be called on loop_data when elr is freed.

void *GalSS_ELRGetLoopData (GalSS_ELR *elr) Retrieves the arbitrary ELR data.

ELR support functionsUnder normal circumstances, you won’t need to know anything more about ELR objects.However, there are a number of customizations you may want to add.

GalSS_ELR *GalSS_ELRCreate (GalIO_ServerStruct *scomm, GalSS_ELTimerSetFn timer_set_fn, GalSS_ELUnsetFn timer_unset_fn, GalSS_ELFDSetFn fd_set_fn,GalSS_ELUnsetFn fd_unset_fn, GalSS_ELBehaviorFn behavior_fn, int timer_is_persistent) This function creates the basic ELR object. The scomm is a server object created using GalSS_SetupServer or some similar function. The timer_set_fn, timer_unset_fn, fd_set_fn, and fd_unset_fn are the timer and file descriptor setting and unsetting functions illustrated here. The behavior_fn is the function which is called when various Communicator events are fired; for anygiven situation, it is called after the corresponding timer and file descriptor setting and unsettingfunctions are called (so, for instance, when a new connection object is created, the timer and filedescriptors are registered for that object, and then the behavior_fn is called). The timer_is_persistent argument should be 1 if the timers in the external main loop reset themselvesautomatically when they’re fired, 0 otherwise.

When you create an ELR object, you’re actually registering global information for manyELR objects, each of which is associated with a specific server, connection or broker object.When a connection is created, for instance, a local ELR object is create for it which shares theglobal information with all the other local ELR objects; when the connection is destroyed, thelocal ELR object is freed. The functions GalSS_ELRSetLoopData and GalSS_ELRGetLoopDataaccess the shared global information, so you don’t need to know about this local/globaldistinction in the normal case. However, under some circumstances you may want to know whatsort of local ELR object you’re currently holding in a callback. You can do this by using thefollowing three functions.

GalIO_ServerStruct *GalSS_ELRSComm (GalSS_ELR *elr) Returns the server object associated with this ELR object.

GalIO_CommStruct *GalSS_ELRGComm (GalSS_ELR *elr) Returns the connection object associated with this ELR object, or NULL if this ELR object isassociated with a server.

190

GalIO_BrokerStruct *GalSS_ELRBroker (GalSS_ELR *elr) Returns the broker object associated with this ELR object, or NULL if this ELR object isassociated with a server or connection.

There is currently no support for distinguishing between inbound and outbound brokers.

void GalSS_ELRUpdatePollIntervals (GalSS_ELR *elr, int server_client_poll_ms, int conn_ms, int broker_ms) There are currently three circumstances where timers are used in the ELR infrastructure: whenpolling server clients trying to contact a Hub, when polling connections, and when pollingbrokers. The default timer intervals for these polls are 1000 ms, 50 ms, and 50 ms, respectively.If you want to change the polling interval, you can do so using this function. An argument of -1for any of the three polling arguments here will leave the existing poll untouched.

In rare cases, the programmer will want to change the callback behavior for a specific objecttype. While this is not without risks, it can be done, with one of the following five functions. Thecallback function must return 1 if the object is still pollable, 0 otherwise. It is stronglyrecommended that you begin from the existing implementations of these callback functions andmodify them as little as possible.

typedef int (*GalSS_ELCallbackFn )(GalSS_ELR *elr, int timer_or_fd);

void GalSS_ELRSetConnectionCallback (GalSS_ELR *elr, GalSS_ELCallbackFn fn) Sets the behavior of GalSS_ELRDoCallback when the object associated with the ELR is aconnection object. The current definition is as follows:

static int __GalSS_ELRDoConnectionCallback(GalSS_ELR *elr, int timer_or_fd) { if ((timer_or_fd == GALSS_ELR_FD) || GalIO_CommReadReady(elr->local_info->gcomm) || GalIO_CommWriteReady(elr->local_info->gcomm)) { return GalIO_ConnectionCallbackHandler(elr->local_info->gcomm, 0); } else { return 0; } }

void GalSS_ELRSetBrokerOutCallback (GalSS_ELR *elr, GalSS_ELCallbackFn fn) Sets the behavior of GalSS_ELRDoCallback when the object associated with the ELR is anoutbound broker object. The current definition is as follows:

static int __GalSS_ELRDoBrokerOutCallback(GalSS_ELR *elr, int timer_or_fd) { if (GalIO_BrokerWriteReady(elr->local_info->broker)) {

191

return GalIO_BrokerDataOutCallbackHandler(elr->local_info->broker); } else { return 0; } }

void GalSS_ELRSetBrokerInCallback (GalSS_ELR *elr, GalSS_ELCallbackFn fn) Sets the behavior of GalSS_ELRDoCallback when the object associated with the ELR is aninbound broker object. The current definition is as follows:

static int __GalSS_ELRDoBrokerInCallback(GalSS_ELR *elr, int timer_or_fd) { if ((timer_or_fd == GALSS_ELR_FD) || GalIO_BrokerReadReady(elr->local_info->broker)) { returnGalIO_BrokerDataInCallbackHandler(elr->local_info->broker, 0); } else { return 0; } }

void GalSS_ELRSetServerListenerCallback (GalSS_ELR *elr, GalSS_ELCallbackFn fn) Sets the behavior of GalSS_ELRDoCallback when the object associated with the ELR is aserver object and the current ELR was set up to monitor connections from the Hub. The currentdefinition is as follows:

static int __GalSS_ELRDoServerListenerCallback(GalSS_ELR *elr, int timer_or_fd) { int res = GalIO_ServerCallbackHandler(elr->local_info->scomm, 0, (GalIO_CommStruct **) NULL); /* 1 means it got a connection and everything’s OK. */ if (res == 1) res = 0; return res; }

void GalSS_ELRSetServerClientCallback (GalSS_ELR *elr, GalSS_ELCallbackFn fn) Sets the behavior of GalSS_ELRDoCallback when the object associated with the ELR is aserver object and the current ELR was set up to monitor connections to the Hub. The currentdefinition is as follows:

192

static int __GalSS_ELRDoServerClientCallback(GalSS_ELR *elr, int timer_or_fd) { GalIO_ServerCheckHubContacts(elr->local_info->scomm); return 0; }

In all cases, the functions that are called are the toplevel functions that an external main loopwould call to monitor the relevant object.

GalSS_ELR *GalSS_ELRCopy (GalSS_ELR *source) Copies an ELR object. Programmers should not need this function.

void GalSS_ELRShutdown (GalSS_ELR *elr) Shuts down all callbacks associated with this ELR. Programmers should not need this function.

void GalSS_ELRDestroy (GalSS_ELR *elr) Frees the ELR. If the ELR is the "root" ELR, the global information is freed as well.

The ugly detailsThe ELR infrastructure was abstracted from a number of instances of embedding theCommunicator infrastructure in external main loops. While the event-driven programming model,new in 3.0, made this process more explicit and complete, it also made it more complicated. Inthis section, we illustrate the 3.0 version of our simple main loop example, which illustrates ingory detail the details underlying the ELR infrastructure.

The main loopLet’s begin with the main() function.

int main(int argc, char **argv) { GalaxyCallbackRecord *gcr = (GalaxyCallbackRecord *) NULL; int new_argc; char **new_argv; GalSS_ServerArgs *arg_pkg;

gcr = (GalaxyCallbackRecord *) malloc(sizeof(GalaxyCallbackRecord)); gcr->timer_cb = (TimerCallback *) NULL; gcr->l = SM_NewLooper();

/* If you want to use the built-in server arguments, you can use GalSS_ExtractCmdlineServerArgs. Otherwise, you can just call GalSS_InitializeServerToplevel(). */

193

arg_pkg = GalSS_DefaultServerArgs();

/* Make sure it knows that we’re using our own main loop. We set this before we ever parse the server arguments, because we don’t even want the arguments pertaining to the loop type enabled for the user. */

GalSS_SAFixLoopType(arg_pkg, GAL_LOOP_EXTERNAL); arg_pkg = GalSS_ExtractCmdlineServerArgs(arg_pkg, argc, argv, &new_argc, &new_argv);

if (!arg_pkg) { /* Something bad happened, or -help was passed. */ exit(1); }

/* Now, we call GalSS_InitializeServerFromServerArgs, and we don’t have to worry about the signature of GalSS_InitializeServerToplevel. */

gcr->scomm = GalSS_SetupServer(arg_pkg, new_argc, new_argv); GalSS_FreeArgPkg(arg_pkg);

if (!gcr->scomm) { fprintf(stderr, "Couldn’t create a server\n"); fflush(stderr); exit(1); }

/* Set the connect callback for the server. This gets called whenever a new connection is established. */

GalIO_AddServerConnectCallback(gcr->scomm, GCRAddConnectionCallback, (void *) gcr); /* The server can be a listener when it starts out, or it can become a listener when an outgoing broker starts up. So we set a callback to handle whenever this happens. */ GalIO_AddServerCallback(gcr->scomm, GAL_SERVER_LISTENER_STARTUP_EVENT, GCRSetupServerListener, (void *) gcr); /* Similarly, if someone calls GalIO_ContactHub, it may lead to a new poller starting up. So we should deal with that as a callback too. */ GalIO_AddServerCallback(gcr->scomm,

194

GAL_SERVER_CLIENT_POLL_STARTUP_EVENT, GCRSetupServerClient, (void *) gcr); /* And now, something that will shut down the loop when the server is destroyed. */ GalIO_AddServerCallback(gcr->scomm, GAL_SERVER_DESTRUCTION_EVENT, GCRDestroyServer, (void *) gcr);

/* Now, start the server, and then the main loop. */ if (!GalIO_ServerStart(gcr->scomm)) { fprintf(stderr, "Couldn’t start the server\n"); fflush(stderr); exit(1); } SM_Mainloop(gcr->l); exit(0); }

This example contrasts with the default main loop in that we’re using a different main loop; theway the server object is created is essentially the same. There are three sections to this main() function.

In the first section, we allocate the structures we need for this type of mainloop. In the secondsection, we parse the command line arguments, using the Galaxy Communicator library utilities.Notice that before we parse the arguments, we fix the loop type, so that the arguments to controlthe loop type will not be used. Then, we populate the package of arguments, and then set up aserver based on those arguments. In the final section, we set up the callbacks which will handlethe server. We do this by setting up four callbacks: one for when the listener starts up, one forwhen the client poll starts up, one for when a connection is established, and one for when theserver is destroyed.

The server callbacksThere are three callbacks here (we’ll discuss the connection establishment callback below).First, we consider the listener startup callback.

static void GCRServerListenerHandler(void *client_data) { GalaxyCallbackRecord *gcr = (GalaxyCallbackRecord *) client_data;

GalIO_ServerCallbackHandler(gcr->scomm, 0, (GalIO_CommStruct **) NULL); }

static void GCRShutdownServerListener(GalIO_ServerStruct *scomm, void *callback_data) { GalaxyCallbackRecord *gcr = (GalaxyCallbackRecord *)

195

callback_data;

GalUtil_Warn("The server has failed.\n"); SM_RemoveAllFDCallbacks(gcr->l); SM_RemoveAllTimerCallbacks(gcr->l); }

static void GCRSetupServerListener(GalIO_ServerStruct *scomm, void *callback_data) { GalaxyCallbackRecord *gcr = (GalaxyCallbackRecord *) callback_data; /* You only need a file descriptor callback here, since there will be no connection requests in any internal queue. */ SM_AddFDCallback(gcr->l, GalIO_GetServerListenSocket(scomm), GCRServerListenerHandler, (void *) gcr); GalIO_AddServerCallback(scomm, GAL_SERVER_LISTENER_SHUTDOWN_EVENT, GCRShutdownServerListener, (void *) gcr); }

The function GCRSetupListenerServer is called when the listener starts up. It adds ashutdown callback to the server, and adds a file descriptor callback for the local main loop whichcalls GalIO_ServerCallbackHandler.

int GalIO_ServerCallbackHandler (GalIO_ServerStruct *scomm, int read_blocking,GalIO_CommStruct **new_conn_ptr) Polls the server scomm and sets *new_conn_pointer to the new connection, if one isestablished. If new_conn_ptr is NULL, the new connection will not be returned. The read_blocking flag should be 1 if the handler should do a blocking read, 0 otherwise. Returns 1 ifthere’s a new connection, 0 if there isn’t, -1 if an error was encountered and the listener was shutdown, -2 if an error was encountered and the server was destroyed.

Next, let’s look at the client startup callbacks:

/* This function is used when the server is subscribing to Hub listeners. */

static void GCRServerClientHandler(void *client_data) { GalaxyCallbackRecord *gcr = (GalaxyCallbackRecord *) client_data;

GalIO_ServerCheckHubContacts(gcr->scomm); }

196

static void GCRShutdownServerClient(GalIO_ServerStruct *scomm, void *callback_data) { TimerCallback *cb = (TimerCallback *) callback_data;

SM_RemoveTimerCallback(cb); }

static void GCRSetupServerClient(GalIO_ServerStruct *scomm, void *callback_data) { GalaxyCallbackRecord *gcr = (GalaxyCallbackRecord *) callback_data; TimerCallback *cb;

/* Set up a periodic task to check the hub contacts. */ cb = SM_AddTimerCallback(gcr->l, 10, GCRServerClientHandler, (void*) gcr); /* Add a shutdown callback now. */ GalIO_AddServerCallback(scomm, GAL_SERVER_DESTRUCTION_EVENT, GCRShutdownServerClient, (void *) cb); }

This is exactly parallel to the listener case, except the client shutdown callback is associated withserver destruction. The local main loop callback calls GalIO_ServerCheckHubContacts.

void GalIO_ServerCheckHubContacts (GalIO_ServerStruct *scomm) Polls the server scomm to make sure that all client connections to Hub listeners areappropriately established.

Next, we have the server destruction callback, which simply exits the local loop.

static void GCRDestroyServer(GalIO_ServerStruct *scomm, void *callback_data) { GalaxyCallbackRecord *gcr = (GalaxyCallbackRecord *) callback_data;

GalUtil_Warn("The server has been destroyed.\n"); SM_LooperExit(gcr->l); }

197

The connection callbacksThe server connect callback AddConnectionCallback sets up both a timer and a file descriptorcallback for the connection in the local main loop, and stores away a disconnect handler for theconnection which will shut down both callbacks, as well as broker startup callbacks. The reasonfor two local main loop callbacks is that the file descriptor callback is more efficient (processinghappens immediately when data is available), but it misses some of the cases, because if extradata is read in and is available in the internal queue for the connection, a file descriptor callbackwon’t see it.

typedef struct __connection_container { GalIO_CommStruct *gcomm; GalaxyCallbackRecord *gcr; TimerCallback *t; GAL_SOCKET fd; } ConnectionContainer;


/* GalIO_ConnectionCallbackHandler(): -1 means an error was encountered and the connection has been destroyed. 0 means we’re in the midst of things. 1 means we’re done and the connection has been destroyed. */

static void GCRConnectionDisconnect(GalIO_CommStruct *gcomm, void *caller_data) { ConnectionContainer *c = (ConnectionContainer *) caller_data; GalaxyCallbackRecord *gcr = c->gcr;

SM_RemoveFDCallback(gcr->l, c->fd); SM_RemoveTimerCallback(c->t); free(c); }

/* The loop cleanup is handled in the disconnect callback. */

static void GCRConnectionHandler(void *client_data) { ConnectionContainer *c = (ConnectionContainer *) client_data; GalIO_CommStruct *gcomm = c->gcomm;

198

GalIO_ConnectionCallbackHandler(gcomm, 0); }

static void GCRConnectionTimerHandler(void *client_data) { /* This is called from the timer. We could go ahead and just call the normal connection handler, which would try to read from the file descriptor, but since there’s already a file descriptor callback which triggers that handler, we’ll only do something if there’s stuff in the internal queues. */ ConnectionContainer *c = (ConnectionContainer *) client_data; GalIO_CommStruct *gcomm = c->gcomm;

if (GalIO_CommReadReady(gcomm) || GalIO_CommWriteReady(gcomm)) { GCRConnectionHandler(client_data); } }

static void GCRAddConnectionCallback(GalIO_ServerStruct *scomm, GalIO_CommStruct *gcomm, void *callback_data) { GalaxyCallbackRecord *gcr = (GalaxyCallbackRecord *) callback_data; ConnectionContainer *c = (ConnectionContainer *) calloc(1, sizeof(ConnectionContainer));

c->gcr = gcr; c->gcomm = gcomm; c->fd = GalIO_GetCommSocket(gcomm);

/* We’ll use the file descriptor callback to check the file descriptor, and the timer callback to check the internal queue. */ SM_AddFDCallback(gcr->l, c->fd, GCRConnectionHandler, (void *) c); c->t = SM_AddTimerCallback(gcr->l, 5, GCRConnectionTimerHandler, (void *) c); /* Finally, to support brokers, and to deal with disconnections, we need to use the data slot for the connection. */ /* Make sure you stop polling when the connection dies. */ GalIO_AddConnectionCallback(gcomm, GAL_CONNECTION_SHUTDOWN_EVENT, GCRConnectionDisconnect,

199

(void *) c); /* And now, add the callbacks for the broker setups. */ GalIO_AddConnectionBrokerCallback(gcomm, GAL_CONNECTION_BROKER_OUT_STARTUP_EVENT, GCRSetupBrokerOut, (void *) gcr); GalIO_AddConnectionBrokerCallback(gcomm, GAL_CONNECTION_BROKER_IN_STARTUP_EVENT, GCRSetupBrokerIn, (void *) gcr); }

int GalIO_ConnectionCallbackHandler (GalIO_CommStruct *gcomm, int read_blocking) Polls the connection gcomm for dispatch function requests to process. Does a blocking read if read_blocking is nonzero. Returns -1 if an error was encountered and the connection has beendestroyed, 1 if the connection is done and it’s been destroyed, 0 otherwise.

int GalIO_CommWriteReady (GalIO_CommStruct *gcomm) Returns 1 if the connection gcomm has data in its internal outbound queue waiting to be written,0 otherwise.

intGalIO_CommReadReady (GalIO_CommStruct *gcomm) Returns 1 if the connection gcomm has data in its internal inbound queue waiting to beprocessed, 0 otherwise.

The outgoing broker callbackThe broker callbacks are completely parallel to these last cases. Let’s start with the outgoing broker.

typedef struct __broker_container { GalaxyCallbackRecord *gcr; GalIO_BrokerStruct *b; TimerCallback *t; GAL_SOCKET fd; } BrokerContainer;

/* GalIO_BrokerDataOutCallbackHandler() returns 1 if the broker is done and has been destroyed, 0 if not done */

/* The timer disconnect is handled in the loop data finalizer. */

200

static void GCROutBrokerHandler(void *client_data) { BrokerContainer *c = (BrokerContainer *) client_data;

if (GalIO_BrokerWriteReady(c->b)) { GalIO_BrokerDataOutCallbackHandler(c->b); } }

void GCRBrokerShutdown(GalIO_BrokerStruct *b, void *loop_data) { BrokerContainer *c = (BrokerContainer *) loop_data;

if (c->fd != GAL_INVALID_SOCKET) SM_RemoveFDCallback(c->gcr->l, c->fd); SM_RemoveTimerCallback(c->t); free(c); }

void GCRSetupBrokerOut(GalIO_CommStruct *gcomm, GalIO_BrokerStruct *b, void *caller_data) { BrokerContainer *c = (BrokerContainer *) calloc(1, sizeof(BrokerContainer)); c->b = b; c->gcr = (GalaxyCallbackRecord *) caller_data; c->fd = GAL_INVALID_SOCKET;

/* There’s no point in an fd callback for the outgoing broker, since it piggybacks off of the server listener. */

c->t = SM_AddTimerCallback(c->gcr->l, 1, GCROutBrokerHandler, (void *) c);

/* Use the caller data to set up the loop finalizer. */ GalIO_AddBrokerCallback(b, GAL_BROKER_DESTRUCTION_EVENT, GCRBrokerShutdown, (void *) c); }

The outgoing broker sets up a timer callback for the local main loop (the server is alreadymonitoring the connection requests, so all the outgoing broker needs to do is flush data). Inaddition, the broker adds a shutdown callback for itself.

201

int GalIO_BrokerDataOutCallbackHandler (GalIO_BrokerStruct *b) This function invokes the broker b and destroys the broker in the appropriate circumstances.This function returns 1 if the broker is done and has been destroyed, 0 if not done.

int GalIO_BrokerWriteReady (GalIO_BrokerStruct *b) This function returns 1 when the broker has data to write in its outbound queue, 0 otherwise.

The incoming broker callbackThe incoming brokers are completely parallel, as long as you’ve use GalIO_CommBrokerDataInInit instead of the older GalIO_BrokerDataInInit.

/* Next, the incoming broker. This is much more like the connection. */

/* GalIO_BrokerInCallbackHandler() returns 1 if the broker is done and has been destroyed, 0 if not done, -1 if error was encountered and the broker has been destroyed. */

static void GCRInBrokerHandler(void *client_data) { /* The timer disconnect must be handled in the finalizer. Make sure that’s set up correctly in the examples. */ BrokerContainer *c = (BrokerContainer *) client_data; GalIO_BrokerStruct *b = c->b;

GalIO_BrokerDataInCallbackHandler(b, 0); }

static void GCRInBrokerTimerHandler(void *client_data) { /* This is called from the timer. We could go ahead and just call the normal in broker handler, which would try to read from the file descriptor, but since there’s already a file descriptor callback which triggers that handler, we’ll only do something if there’s stuff in the internal queues. */ BrokerContainer *c = (BrokerContainer *) client_data; GalIO_BrokerStruct *b = c->b;

if (GalIO_BrokerReadReady(b)) { GCRInBrokerHandler(client_data); } }

202

void GCRSetupBrokerIn(GalIO_CommStruct *gcomm, GalIO_BrokerStruct *b, void *caller_data) { BrokerContainer *c = (BrokerContainer *) calloc(1, sizeof(BrokerContainer)); c->b = b; c->gcr = (GalaxyCallbackRecord *) caller_data; c->fd = GalIO_GetBrokerSocket(b);

SM_AddFDCallback(c->gcr->l, c->fd, GCRInBrokerHandler, (void *) c); c->t = SM_AddTimerCallback(c->gcr->l, 1, GCRInBrokerTimerHandler, (void *) c);

/* Use the caller data to set up the loop finalizer. */ GalIO_AddBrokerCallback(b, GAL_BROKER_DESTRUCTION_EVENT, GCRBrokerShutdown, (void *) c);

The callback sets up a local file descriptor and timer callback, and sets up a shutdown callbackto destroy the local callbacks.

int GalIO_BrokerReadReady (GalIO_BrokerStruct *b) This function returns 1 when the broker has data to read in its inbound queue, 0 otherwise.

int GalIO_BrokerDataInCallbackHandler (GalIO_BrokerStruct *b, int read_blocking) This function polls the broker b and destroys the broker in the appropriate circumstances. Thisfunction returns 1 if the broker is done and has been destroyed, 0 if not done, -1 if error wasencountered and the broker has been destroyed.

Using main loops with listener-in-HubWhen we write our own main loops to take advantage of the listener-in-Hub capability, we cando some very fancy things. For instance, under some circumstances, we don’t want to contactthe Hubs we list immediately. We can use the function GalSS_SAFixServerListenStatus() tocontrol how the Hub client behaves, and we can use the function GalSS_SAFixContactHubInfo()to help us collect the information and use it later. Here’s an example of a toplevel loop for asimple server which also listens for non-Communicator input:

int main(int argc, char **argv) { GalIO_ServerStruct *s; GalSS_ServerArgs *arg_pkg; GalIO_ServerLocation *locs; char *loc_string;

203

char *default_session_id; MGal_StdinPoll *p; InputState *a; int new_argc; char **new_argv;


/* Make sure it knows that we’re using the timed tasks. */ GalSS_SAFixLoopType(arg_pkg, GAL_LOOP_TT);

/* Set it up to be a Hub client, and not to retry the initial connection and to shutdown when the last connectionexits. */ GalSS_SAFixServerListenStatus(arg_pkg, GAL_HUB_CLIENT | GAL_HUB_CLIENT_DISCONNECT_SHUTDOWN | GAL_HUB_CLIENT_CONNECT_FAILURE_NOOP);

/* If you want to use the built-in server arguments, you can use GalSS_ExtractCmdlineServerArgs(). Otherwise, you can just call GalSS_InitializeServerToplevel(). */ arg_pkg = GalSS_ExtractCmdlineServerArgs(arg_pkg, argc, argv, &new_argc, &new_argv);


/* Now that we’ve captured the arguments, we "recapture" the locations, because we don’t want to use them immediately. This will return the current value of the server locations, and set the new value to NULL, before we digest the arguments. We make sure there’s something in it before we continue. */ loc_string = GalSS_SAFixContactHubInfo(arg_pkg, (char *) NULL, (char *) NULL, &default_session_id); locs = GalIO_DigestServerLocations(loc_string); if (!locs) { /* Without Hubs to contact, I can’t deal with anything. */ GalUtil_Fatal("No Hubs to contact!"); }

204

s = GalSS_SetupServer(arg_pkg, new_argc,new_argv); if (!s) { GalUtil_Fatal("Failed to initialize server!"); }

/* Before I start up the server, I store away locs in some safe place, and I set up my non-Communicator listeners (perhaps a GUI button callback, or a telephone line connection, or something like that). When the listener is activated, it will use locs to pick a Hub to contact. */

...

GalSS_StartAndRunServer(s); exit(0); }

GalIO_ServerLocation *GalIO_DigestServerLocations (char *client_pair_string) Digests the client_pair_string, which is a whitespace-separated sequence of host:port pairs, andstores the syntactically legal ones in an array of locations.

int GalIO_NumServerLocations (GalIO_ServerLocation *locs) Returns the length of the server locations array.

char *GalIO_NthHostAndPort (GalIO_ServerLocation *locs, int i, unsigned short *port) Retrieves the ith host and port in locs. The host is returned; the port is set as an outarg. The usershould not free the returned host value. If i is out of range, NULL is returned.

GalIO_ServerLocation *GalIO_GetServerLocations (GalIO_ServerStruct *scomm) Retrieves the server locations array which the server scomm has.

Last updated June 5, 2001

205

Using non-MIT Makefiles

As of Galaxy Communicator 2.0, no stubber is required to generate headers. This means that it’seasy to use non-MIT makefiles to compile Communicator-compliant servers. We provide an example of this.

You need to declare a number of things in order to compile the server:

You need the value of the Communicator root directory. You need the value of ARCHOS, to choose the correct library. You need to determine the value of LIBEXT, to make sure you specify the proper library dependency. You need the current source directory in your includes. You need the Communicator header directory in your includes. You need the Communicator library in your libraries. The library must follow your source orobject files in your link command, because the Communicator library contains defaultimplementations of functions which are defined in your server headers.

Here’s an example:

# This example doesn’t use either the MIT or the MITRE # Makefile templates, as an illustration.

ROOTDIR = /packages/GalaxyCommunicator

TEMPLATES = $(ROOTDIR)/templates

# I need to get the architecture to find the MIT libraries. # But I need to load config.make to find out what the ARCH prefix # is, and config.make to find out what ARCHOS_PREFIX is.


CPPFLAGS = -I. -I$(ROOTDIR)/include

LIBS = -L$(ROOTDIR)/lib/$(ARCHOS) -lGalaxy

206

ifeq ($(OS),solaris) LIBS += -lsocket -lnsl -lposix4 endif

include $(TEMPLATES)/shared_libs.make

# If we’re using shared libraries, add the run-time dependency on # the library to the LIBS line.

ifdef SHARED_LIBS ifdef XLINKER_RTFLAG LIBS += -Xlinker $(XLINKER_RTFLAG) -Xlinker $(ROOTDIR)/lib/$(ARCHOS) endif endif

LIBDEPS = $(ROOTDIR)/lib/$(ARCHOS)/libGalaxy$(LIBEXT)

all: double

double: double.c $(LIBDEPS) gcc -g $(CPPFLAGS) -o double.$(ARCHOS).bin double.c $(LIBS)

Note that the libraries rely on the Posix and network libraries on Solaris.


207

Signal Handling

Signal handling is complicated by threads. In order to enforce the proper signal handling acrossthreaded and non-threaded applications, we provide a simple interface to signal handling whichhides the appropriate details. It is strongly recommended that you use this interface to installsignal handlers.

void Gal_InitializeSignals () This function is called by GalSS_CmdlineInitializeServer and GalSS_InitializeServerToplevel. Inthe non-threaded environment, it does nothing; if threads are enabled, it sets up a separatethread for signal handling and enables a few default handlers.

int Gal_SignalsInitialized () Returns 1 if signals have been initialized. In the non-threaded case, always returns 1.

void Gal_AddSignalHandler (int sig, void (handler)(int )) This function is like calling signal(), except it handles signals appropriately in threaded contexts.It supports the System V interpretation of signal(), whereby the signal handler is cleared after thesignal is caught. If you want the signal to be caught again, you must callGal_AddSignalHandler() inside your signal handler. This function will do the right thing inside _GalSS_init_server and _GalSS_InitializeDefaults. (although in the threaded case, no signalscan be counted on to be handled appropriately until immediately before _GalSS_init_server is called).


208

Binary Data

In addition to brokering, MIT provides a way of moving binary data among servers, using the binary frame object. We can use this object either to send around audio data, or to encodearbitrary structures which remain opaque to the Communicator Hub. (Be aware that unlike brokering connections, big_endian/little_endian issues are not addressed in the binary datatools, since only 8 bit binary data can be encoded.)

We provide an example of using binary objects for audio to compare with our example of usingbrokering for audio. In this section, we show how one might use binary objects to encodearbitrary structures.

Passing arbitrary dataThe key aspect of using binary objects to encode arbitrary data is to write encoding anddecoding functions for the data. In the example we provide, we encode and decode a simplestructure, whose slots are initialized in the reinitialize() message:

typedef struct __MTestStruct { int first; int last; char *msg; } MTestStruct;

static void *encode_test_struct(MTestStruct *s, int *len) { char *encode_buf;

encode_buf = (char *) malloc(sizeof(char) * (strlen(s->msg) + 64)); sprintf(encode_buf, "%d %d %s", s->first, s->last, s->msg); *len = strlen(encode_buf); return (void *) encode_buf; }

209

static MTestStruct *decode_test_struct(void *data, int len) { MTestStruct *s = (MTestStruct *) malloc(sizeof(MTestStruct)); int where;

sscanf(data, "%d %d %n", &(s->first), &(s->last), &where); s->msg = (char *) malloc(sizeof(char) * (1 + (len - where))); strncpy(s->msg, data + where, len - where); return s; }

The important step here is for the server or servers which need access to this arbitrary data toagree on a binary (or string) encoding of the data. The data will be passed untouched. In ourexample, the reinitialize() message creates a structure, encodes it, and passes it to the Hub, andthe receive_binary() message decodes the data, recreates the structure, and prints out thecontents of the structure:

Gal_Frame receive_binary(Gal_Frame f, void *server_data) { int size = 0; void *buf = Gal_GetBinary(f, ":binary_data", &size); MTestStruct *s = decode_test_struct(buf, size);

sls_pinfo1("Decoded buf is ‘%s’\n", buf); sls_pinfo1("First is %d, last is %d, msg is ‘%s’\n", s->first, s->last, s->msg); return (Gal_Frame) NULL; }

Gal_Frame reinitialize(Gal_Frame f, void *server_data) { Gal_Frame fr; int len; void *data; MTestStruct *s = (MTestStruct *) malloc(sizeof(MTestStruct));

s->first = Gal_GetInt(f, ":test_first"); s->last = Gal_GetInt(f, ":test_last"); s->msg = Gal_GetString(f, ":test_msg");

/* Create a new message. */ fr = Gal_MakeFrame("main", GAL_CLAUSE); data = encode_test_struct(s, &len); Gal_SetProp(fr, ":binary_data", Gal_BinaryObject(data, len)); GalSS_EnvWriteFrame((GalSS_Environment *) server_data, fr, 0); Gal_FreeFrame(fr); return (Gal_Frame) NULL; }

210

Note how the data and size established by the encoder are used to create a binary object, andthen how the data and size in the binary object are passed to the decoder to recreate thestructure. In this way, data structures which the Hub knows nothing about can be passedbetween servers without needing to recursively duplicate the structure of the data in frames.


211

Command Line Argument Parsing andPrinting Utilities

As part of establishing a single Communicator library, we have provided copies of MITfunctionality for command line argument parsing and printing. This documentation has beenlargely adapted from the MIT documentation, and we’ve included their copyright below.

Command line argument parsingThe SLS online argument (oa) facility provides a simple interface to standard online argumentparsing. It allows easy specification of program usage and easy extraction of the actual optionsspecified by the user at runtime. It performs some simple usage checking, and also providesdocumentation of the program by printing out the usage line along describing each programoption. This documentation has been broken down into the following areas:

Program Model Overview Variable Arguments Help

Program ModelThe program model is as follows:

program [options] [variable arguments]

When a program is invoked, first options (if any) are specified, and then variable arguments (ifany) are specified. Their is a finite set of options, and each option consists of a key (a string thatstarts with a ‘-’) followed by a fixed number of arguments for that key. For example, an optionmight be "-copy from to" . Here the key is -copy, and the two arguments are from and to.The variable arguments must appear after all options, and is just an arbitrarily long list of strings(often zero).

When a program is called with no online arguments, or with the -help option, it should print out ahelp line indicating its usage. If a user of this program gives too few or too many arguments to agiven key, the program will do its best to detect this and signal some kind of warning or error.

212

OverviewThe usage of a program is specified with a single char * array, typically called oas and defined atthe very top of a program (after the includes):

char *oas[] = {"-lex file", "the file containing the lexicon", "default.lex", "-subtract file1 file2", "remove the words in file2 from file1", NULL, NULL, "-set_weight N", "word weight", "5.0", "-disable", "turn off the default checking", NULL };

The oas string contains one entry for each key (usually one entry per line, but not required),where an entry can have a variable number of strings. The first string in an entry is the keystring, which contains the key and the arguments needed by the key, separated by spaces. Thelibrary determines how many arguments a key takes from this string. Next is the description ofwhat the key does, to be used when printing out the usage line. This description should besuccint and helpful. Following the description is a default string for each of the arguments of thekey. These are the defaults that are to be used if the user doesn’t specify the key on thecommand line. Each default can optionally be NULL, meaning that argument does not have adefault. There must be as many default strings as there are arguments to the key. The oas stringmust be terminated with a single NULL in the slot where the next key would otherwise be.

Before calling any other OA functions, the program should first call GalUtil_OACheckUsage().This function checks first the programmer’s usage in writing the oas list, and second the user’susage (i.e., whether the user used a key with too few arguments, or whether an extra argumentwas inserted between keys, etc). This function will print out the usage line, and return 0 if theusage failed, and return 1 otherwise. If it returns zero, the program should just exit (since GalUtil_OACheckUsage() will print the usage line). The function GalUtil_OaPrintUsage() printsthe usage help line.

To actually extract arguments, use the function GalUtil_OAExtract(). This function takes arc,argv, and the oas list, followed by the key you’d like to extract, followed by pairs of a typespecification, and a pointer to where the argument should be stored. For each argument of thekey, oa_extract will set the variables according to the requested type. For example, using theabove oas one could call:

GalUtil_OAExtract(argc, argv, oas, "-set_weight", GAL_OA_INT, &n);

Variable ArgumentsVariable arguments must appear after all of the keys when the program is invoked. The functionGalUtil_OACheckUsage() can be used to determine the argc where the variable arguments firstbegin. It will check to be sure that the variable arguments are not intermingled with program options.

213

To specify a description of variable arguments, use the special key "REST". This is the only legalkey without a ‘-’, and the OA library treats is specially.

HelpThe program should detect if argc is 1 and print the usage line, if that is appropriate. The OAlibrary will automatically detect when the user specifies -help, and print the usage lineaccordingly (GalUtil_OACheckUsage() does this). Furthermore, this will automatically be printedin the usage line (the -help key), and need not be specified in the oas list. Also, if there is anerror in GalUtil_OACheckUsage(), it will print the usage line and return 0.

int GalUtil_OACheckUsage (int argc, char **argv, char **oas, int *first_real_arg) This function quickly looks through the oas list and through the actual online argumentsspecified, and verifies that the usage is correct (to a first approximation). It checks thatarguments are not missing nor that there are extra arguments, and also verifies that the oas listis formatted correctly. This function must be called once at the start of the program beforeGalUtil_OAExtract(). If first_real_arg is non-NULL, this integer will be set to the argc of the firstfixed argument. The function returns 0 if there is an error and 1 if usage is correct.

void GalUtil_OAPrintUsage (int argc, char **argv, char **oas) This function prints the help line of the program. It should be called when the user has not calledthe program properly.

int GalUtil_OAExtract (int argc, char **argv, char **oas, char *key, ... ) This function is used to extract the arguments specified by the user. This function is called for asingle key, and will return the arguments of that key casted to a specific type. The number oftype/p_value pairs must be equal to the number of arguments specified by the key. For example,for the key "-copy from to", there must be two pairs of type/value. The exception to this is abinary key (one with zero arguments, for example "-debug"). In this case, GalUtil_OAExtract()takes one pair of type/value, and it will set the value to be GAL_TRUE or GAL_FALSEdepending on whether the key was specified. This is just an alternative to looking at the returnvalue of GalUtil_OAExtract() to determine if the user specified the key. If the user did not specifythe key, then the values are set to the defaults specified on the oas list. The allowed types are:

GAL_OA_CHAR GAL_OA_SHORT GAL_OA_INT GAL_OA_FLOAT GAL_OA_DOUBLE GAL_OA_STRING

214

Memory management

GalUtil_OAExtract() does not allocate any memory, so when the type is GAL_OA_STRINGthe pointer will be exactly the corresponding argv.

void GalUtil_OAExtractAsserting (int argc, char **argv, char **oas, char *key, ... ) This is the same as the above, but it aborts (exits) if the user did not specify the argument.

PrintingThese functions provide the developer with control over how messages get displayed orprocessed, and the user with run time control over the verbosity of messages. This isaccomplished by defining levels or classes of messages, each with an associated function.Developers using this facility decide at which level a message is printed by calling the functioncorresponding to that level (e.g. GalUtil_Error(), GalUtil_Warn(), GalUtil_Debug(), etc).

Whether or not the message passed to that function gets printed is determined at runtime by theuser’s GAL_VERBOSE environment variable, or by the -verbosity arguments of the Hub or server. This allows the user to have run time control over the number and nature of messageswhich get printed. Setting GAL_VERBOSE to 0 turns off all printing. Incrementing it by 1 turnson printing of additional message classes, until 6, at which point all printing is turned on.Increasing GAL_VERBOSE beyond 6 has no effect. The default verbosity is 3.

Note that MIT servers use the MIT equivalent of these facilities in places, and you should alsoset SLS_VERBOSE as well.

int GalUtil_SetVerbose (int verbose_level) This function sets the verbosity level for the executable. It can be called at any time. It returnsthe new verbosity level. Input values less than zero will be normalized to the default verbosity of 3.

The following functions are available for increasing degrees of verbosity:

GAL_VERBOSE > 0

void GalUtil_Fatal (char *format, ... ) void GalUtil_Error (char *format, ... ) void GalUtil_Assert (int truth, char *format, ... )

Note that GalUtil_Fatal() and GalUtil_Assert() will quit the program in addition to providing anerror message.

215

GAL_VERBOSE > 1void GalUtil_Warn (char * format, ... )

GAL_VERBOSE > 2

void GalUtil_PInfo1 (char *format, ... ) void GalUtil_CPInfo1 (int fore, int back, char *format, ... )

GAL_VERBOSE > 3

void GalUtil_PInfo2 (char *format, ... ) void GalUtil_CPInfo2 (int fore, int back, char *format, ... )

GAL_VERBOSE > 4void GalUtil_Debug1 (char * format, ... )

GAL_VERBOSE > 5void GalUtil_Debug2 (char * format, ... )

In addition, there are two functions which allow the user to set a minimum verbose level.

void GalUtil_Print (int level, char *format, ... ) void GalUtil_CPrint (int level, int fore, int back, char *format, ... ) Prints in color or b/w if GAL_VERBOSE exceeds level.

void GalUtil_VerboseUseBW (void ) Print for a black and white xterm. Default.

void GalUtil_VerboseUseColor (void ) Print for a color xterm.


216

How to Use the Communicator Library inArbitrary Executables

At times, you may be interested in using the Galaxy Communicator C library in executablesbesides the Hub and servers. For instance, you may be interested in writing code whichpretends to be a Hub and contacts a server directly, or you may be interested in parsing andmanipulating Communicator frames.

GalIO_CommStruct *GalIO_ClientConnect (char *name, char *host, unsigned short port, int silent, Gal_Frame welcome_frame, Gal_Frame *reply_frame) Contact the server located at host and port and use welcome_frame as the contents of the reinitialize message. The name and silent arguments are for debugging purposes. The serverthat is contacted will behave exactly as if a Hub had contacted it.

void Gal_InitializeStatics (void ) This function initializes the verbosity printing and all the mutexes required for threadedinteraction. If you write an application which doesn’t create a GalIO_ServerStruct object, youmust call this function.


217

Reference

218

Frequently Asked Questions

We intend to expand this FAQ considerably, as people start asking us more questions :-). Thequestions below are a rough beginning.

Installation"When I type ’make’, it fails with an error like ’Unexpected end of line seen’.

Be sure you’re using GNU make. Try "make -v" to confirm that it’s GNU make. GNU make willprint out the version; other makes will probably not even recognize the -v flag.

"I’m trying to build a server in my own directory, using the MIT Makefile template, butmake can’t find the paths."

In order to compile a server outside the MIT directory stack, be sure you do the following in yourMakefile, at the top:

# Modify this to point to the package root directory

ROOT_DIR = <root of Communicator>

# Need these to provide something useful for MIT rules.make.

TEMPLATES = $(ROOT_DIR)/templates

This will establish the appropriate context for the compilation.

"I want to use the Python/Java/Allegro servers, but when I compile thecontrib/MITRE/ subdirectory, the Python/Java/Allegro servers aren’t created."

Check templates/<arch>-<os>/config.make to see if the appropriate language binding variablesare uncommented and correct.

"I want to use the database/TrueTalk server, but it isn’t compiled in the default installation."

219

Check templates/<arch>-<os>/config.make to make sure that you’ve created the appropriatesettings for the appropriate Oracle or TrueTalk libraries.

Server and Hub setup

"My server won’t start up on the port I request."

The servers are configured to search for the first available port. Most likely, the port yourequested is already in use. If you want to force the server to start up on the request port or fail,use -assert as an argument to the server.

"I’ve started up my server, but the Hub can’t find it."

Make sure the Hub is looking for the server on the appropriate port, and that the server startedup on the same port.

"I want to read some of the Hub output in the process monitor, but my Hub window keepsscrolling back to the bottom."

MITRE provides a process monitor to aid in controlling processes. Each pane of the processmonitor has a "Pause" button which pauses the output. Normally, the Hub prints out a pacifiercharacter every second when it’s not receiving messages. So in order to review the Hub outputyou must press the "Pause" button, or restart the Hub with the -suppress_pacifiercommand line directive.

Brokering"I’ve set up a callback for my incoming broker to do something when it’s done, but it’snever doing anything."

First, make sure that your outgoing broker is calling GalIO_BrokerDataOutDone. If it doesn’t, theincoming broker will never run the appropriate callbacks. Second, make sure you call GalIO_SetBrokerActive on the incoming broker. If you don’t, the broker will never read from its connection.

MIT Hub scripting"I wanted my program to loop forever, so I made a single rule with identical IN: and OUT:variable lists, but it exits after a single iteration."

There is no looping construct in the Hub scripting language. Execution resumes at the pointwhere the message returns. Your program terminated because there were no following rules. Ifyou want to simulate an infinite loop, you must do it by tail recursion: introduce a new messagefor each "iteration". See the double server example.

220

"Rules that shouldn’t be firing are firing when the Hub starts up."

There’s probably something in your initial token which is causing the rules to fire. Customizeeither the rules or the initial token to rule out the offending condition.


221

The Hub and Server Executables

The HubThe Hub executable manages the interactions among the Communicator servers.

Command line

<GALAXY_ROOT>/bin/hub ...

Command line arguments

static char *oas[] = { "-debug", "run in debug mode", "-color", "color output", "-pgm_file file", "specify hub program file", NULL, "-init string", "hub variables to be added to welcome frame: -init\":domain \\\"Pegasus\\\"\"", NULL, "-validate", "validate server operation lists and message sendsand returns against message signatures", "-locations locs", "space-delimited sequence ofservice_type@host(:location) overrides", NULL, "-server_locations_file file", "a server locations file of linesserver host:port [hub|server]", NULL, "-verbosity level", "set the verbosity of the Hub", NULL, "-gui listener_loc@host:port", "optional gui (listener_loc iseither hub or server)", NULL, "-suppress_pacifier", "don’t print the pacifier dots", NULL };

The -debug argument will cause the Hub to enter a loop which allows the user to explore thestate of the Hub, each time the Hub is about to dispatch a message to a server. The -pgm_fileshould be the user’s program file. The -init argument will add key-value pairs to the framewhich seeds the initial token., and also contribute to the key-value pairs in the welcome messagepassed to the reinitialize dispatch function in each server. The -validate key will use themessage signatures reported by the servers and dynamically validate each message call. The -locations key allows overrides of the server declarations in the Hub script, using the samesyntax as the SERVER: and SERVICE_PROVIDER: declarations respect.

222

The -server_locations_file key allows a similar sort of override, but in a file format thatservers also understand. Note that both these override mechanisms will only apply when there isat most one listed provider for the named service type. This argument is identical to the -server_locations argument of a server executable.

The -verbosity key controls how much information is printed out when the Hub executableruns. See the section on running the executables.

The -gui key allows the user to specify a server which will receive a series of messagesindicating the status of the Hub, which can be used for visualization and data collection. Thisfunctionality is still experimental. The message set will be documented in a future release. Asimple Python server partially illustrating the message set can be found in contrib/MITRE/tools/src/hub_gui .

The -suppress_pacifier key suppresses the printing of the pacifier dot which is usuallyprinted for every 1-second interval the Hub doesn’t receive any messages.

The serversIf you build Communicator servers in the default way, they will accept a range of common arguments. Among these arguments are:

static char *oas[] = { ... "-verbosity level", "set the verbosity of this server", NULL, "-server_locations file", "a server locations file of lines serverhost:port [hub|server]", NULL, ... NULL };

Note that these arguments have correspondences in the Hub executable.

The -server_locations key allows a similar sort of override, but in a file format that serversalso understand. Note that both these override mechanisms will only apply when there is at mostone listed provider for the named service type. This argument is identical to the -server_locations_file argument of the Hub executable.

The -verbosity key controls how much information is printed out when the server executableruns. See the section on running the executables.

223

Running the executablesNeither the Hub nor the server executables require any shell environment variables to be set.However, it is possible to control the verbosity (i.e., the amount of run-time informationpresented) using the GAL_VERBOSE environment variable, in addition to the -verbosity keyon the command line of the Hub or servers. See the documentation on printing utilities for more details.

It is also possible to use the MITRE process monitor to help control the execution of arbitraryprograms, including the Hub and servers. There is a shell version of this tool, but the versionthat’s strongly preferred is written in Python, using Python’s Tk support. The process monitor isused in almost all the MITRE examples.


224

Frames, Property Lists, and Typed Objects

Frames and Objects Frame Subtypes Frame Syntax Object Types Object Syntax

Frame Functions Frame Create, Free, and Copy Functions Frame Comparison Predicates Frame Name and Type Operations

Property Lists Property List Operations Miscellaneous

Typed Object Functions Object Creation Functions Object Free and Copy Functions Type Checking Predicates Object Comparison Predicates Object Value Extraction List Object Operations Binary Object Operations

Frame Utilities Frame to String Conversion Frame Printing Functions Frame Reading Functions

MIT Linguistic Frame Tools General Utilities Frame Searching Functions Predicate List Operations

Deprecated functions

225

Frames and ObjectsFrames are the general data structure which the Galaxy Communicator servers and Hub use tocommunicate. A frame has the following characteristics:

a type a name a property list of key-value pairs a list of predicate frames

Frame SubtypesThere are three frame subtypes:

frame Gal_FrameType

clause GAL_CLAUSE

topic GAL_TOPIC

predicate GAL_PRED

Developers of Communicator-compliant systems will use the GAL_CLAUSE type exclusively,and will not use the list of predicate frames at all. The remaining frame types and predicate listare used internally by MIT in their parsing and semantic interpretation. We provide a briefsummary of their use of these types as a point of interest; these distinctions are ignored entirelyin the Galaxy Communicator infrastructure.

Frame SyntaxThe printed representation of a frame is used both for presentation by functions like Gal_PrFrame and for digestion by functions like Gal_ReadFrameFromString.

{<type> <name> <key> <value> } Legal types are c (GAL_CLAUSE), q (GAL_TOPIC), p (GAL_PRED) The whitespace between the final value and the closing bracket is no longer required There can be no space between the opening bracket and the type The name is obligatory; the type is optional (default is GAL_CLAUSE) Where whitespace is permitted, any amount of whitespace is permitted (except in GAL_BINARY)

The convention for keys is that they should be alphanumeric and begin with a colon (:). Nothingin the system enforces these conventions, though.

226

Note that the keys :pred and :PRED are special to Gal_ReadFrameFromString and its relatives.

Examples

{c foo }

{p mod }

{ object }

Object TypesEach value of a frame key in a key-value pair is a structure called a Gal_Object, which is a typeddata structure. The following table lists the available types. All types not only used internally areavailable for both brokering and frames.

227

data type Gal_ObjectType C type

frame GAL_FRAME Gal_Frame

string GAL_STRING (char *)

integer GAL_INT 32 bit integer

float GAL_FLOAT 32 bit float

list GAL_LIST a Gal_Object list

symbol GAL_SYMBOL (only used internally)

pointer GAL_PTR (only used internally)

topic frame GAL_TOPIC_FRAME Gal_Frame

clause frame GAL_CLAUSE_FRAME Gal_Frame

predicate frame GAL_PREDICATE_FRAME Gal_Frame

binary data GAL_BINARY array of 8 bit integers

16 bit integer GAL_INT_16 array of 16 bit integers



32 bit float GAL_FLOAT_32 array of 32 bit floats

64 bit float GAL_FLOAT_64 array of 64 bit floats

keyword GAL_KEYWORD (only used internally)

tag GAL_TAG (only used internally)

token GAL_TOKEN (only used internally)

Object syntaxThe printed representation of an object is used both for presentation by functions like Gal_PrObject and for digestion by functions like Gal_ReadObjectFromString.

228

Gal_ObjectType format example comments

GAL_STRING "foo"strings are delimited by double quotes, andinternal whitespace is preserved

GAL_FRAME{<type> <name> <key><value> ... }

frame values have the same form as frames,and thus support arbitrary recursion

GAL_INT 5 positive and negative values handled

GAL_FLOAT 5.6 positive and negative values handled

GAL_LIST( <obj> <obj> <obj> )

elements of the list can be any object type (sodifferent object types can be mixed in the samelist). Whitespace after opening parenthesis andbefore closing parenthesis is no longerobligatory, although whitespace between listelements probably is

GAL_BINARY%% <decoded_length><encoded_length> <data>

whitespace separating the four elements mustbe exactly one space; the data is encodedusing a variant of uuencode (see source codefor algorithm)

Other types do not have any printed representations (GAL_INT_16, etc.).

Examples

{c main :initialize 1 }

{c rec :status "typed" :confidence_measures ( 0.5 4.6 2.756 ) }

{c error :error_description {c system_unavailable :timeout 40 } }

The types GAL_TOKEN, GAL_SYMBOL and GAL_KEYWORD are not intended to be used forvalues in frames, as we understand it. However, if the string representation of your framecontains a value consisting of a single token without delimiting quotes, it may be recognized asone of these elements.

Frame Functions

Frame Create, Free, and Copy Functions

Gal_Frame Gal_MakeFrame (char *name, Gal_FrameType type); Creates a frame of the specified type, type. The name should begin with an alphabetic characterand contain no spaces.

229

Memory management

The frame is allocated from an internal supply of frames. The name is copied using strdup.

void Gal_FreeFrame (Gal_Frame fr); "Frees" the frame, fr.

Memory management

All the values of the keys in the frame are freed using Gal_FreeObject, as are all theelements in the predicate list. The frame is returned to the internal supply of frames for reuse.

Gal_Frame Gal_CopyFrame (Gal_Frame fr); Copies the frame, fr, and all its contents.

Memory management

All the values of the frame keys are copied using Gal_CopyObject, as are all the elementsof the predicate list.

Gal_Frame Gal_MakeClauseFrame (char *name)

Creates a frame of type GAL_CLAUSE. See Gal_MakeFrame.

Frame Comparison Predicates

int Gal_FrameEqual (Gal_Frame sf1, Gal_Frame sf2); Returns 1 if the frames have the same name and type and have matching property andpredicate lists. Having matching property lists currently means that sf1 has all the keys in sf2and the corresponding values are judged equal using Gal_ObjectEqual. Having matchingpredicate lists currently means that sf2 has predicates that match all the predicates of sf1.

int Gal_MatchFrame (Gal_Frame sf1, Gal_Frame sf2); Returns 1 if the frames have the same name and type and have matching property lists. Havingmatching property lists currently means that sf1 has all the keys in sf2 and the correspondingvalues are judged equal, where equality for GAL_FRAME is judged recursively byGal_MatchFrame, equality for GAL_LIST is such that each element in the key value for sf2 has amatch in the corresponding value for sf1 (where the match is judged recursively on the sameterms as the toplevel), and equality for all other types is judged by Gal_ObjectCaseEqual. Thename of either frame can be *, indicating a wildcard match. Predicates associated with framesare not considered in this comparison.

230

Frame Name and Type Operations

Gal_Frame Gal_SetFrameName (Gal_Frame fr, char *name); Sets the name of the frame, fr, to name.

Memory management

The name is copied using strdup.

char *Gal_FrameName (Gal_Frame fr); Returns the name of the frame, fr.

Memory management

The string which is returned is the string in the frame, not a copy.

int Gal_FrameNameEq (Gal_Frame fr, char *name); Returns 1 if the frame has name name.

int Gal_FrameNamesEq (Gal_Frame fr1, Gal_Frame fr2) Returns 1 if the two frames have the same name.

Gal_Frame Gal_SetFrameType (Gal_Frame fr, Gal_FrameType type); Sets the type of the frame, fr, to type.

Gal_FrameType Gal_GetFrameType (Gal_Frame fr); Returns the type of the frame, fr.

int Gal_FrameIsType (Gal_Frame fr, Gal_FrameType type); Returns 1 if the frame, fr, is of the specified type, type.

Property ListsFrames contain property lists, or lists of key-value pairs which describe some knowledge aboutthe domain. A key-value pair is a relation which maps domain elements to elements of a range. All keys are optional. Keys used in frames should not have global impact. Keys should beginwith a colon and contain no spaces. Values of keys are typed objects as described below.

Property List Operations

int Gal_NumProperties (Gal_Frame fr); Returns the number of properties in the property list. This function counts properties with NULL values.

231

int Gal_NumNonNullProperties (Gal_Frame fr); Returns the number of properties in the property list. This function skips properties with NULL values.

Gal_Object Gal_SetProp (Gal_Frame fr, char *key, Gal_Object obj); Adds the object, obj, to the property list. The key, key, should begin with a colon (:) and containno spaces. If the key is found in the property list, the associated object will be freed using Gal_FreeObject and replaced with the new object.

Memory management

The obj is not copied when stored in the frame, in spite of the fact that objects it replacesare freed. The key is copied using strdup, if the key is not yet present.

Gal_ObjectGal_GetObject (Gal_Frame fr, char *key); Returns the object associated with the key specified by key.

Memory management

The Gal_Object which is returned is not copied from the frame.

Gal_ObjectGal_RemProp (Gal_Frame fr, char *key); Removes the property, specified by key, from the property list and returns the object. The calleris responsible for freeing the object.

Memory management

The Gal_Object is returned, not freed.

int Gal_DelProp (Gal_Frame fr, char *key) Removes the property, specified by key, from the property list and calls Gal_FreeObject on it.Returns 1 if an element was found to free, 0 otherwise.

char **Gal_GetProperties (Gal_Frame fr, int *nkeys); Returns a list of all the keys in the property list. If successful, nkeys will be set to the number of properties.

Memory management

The array of char * pointers is allocated using calloc, but each element in the array is apointer to the actual name of the key, not a copy.

void Gal_DoProperties (Gal_Frame fr, int (*prop_fn)(char *, Gal_Object , void *), void *caller_data) Applies the function prop_fn to every key-value pair in the property list of the frame fr whosevalue is not NULL. The first argument of prop_fn is the key, the second is the Gal_Object valuewhich would be returned by Gal_GetObject, and the third argument is the caller_data. The prop_fn should return 1 to continue, 0 to halt.

232

Miscellaneous FunctionsThe following functions are provided for convenience and compile time type checking.

Memory management

These functions have the same memory management properties as the object value extraction functions.

Gal_Frame Gal_GetFrame (Gal_Frame fr, char *key); Returns the frame value specified by the key, key, or NULL if the key is not present or its value isnot a frame.

Gal_Frame Gal_GetTopicFrame (Gal_Frame fr, char *key) Returns the topic frame value specified by the key, key., or NULL if the key is not present or itsvalue is not a topic frame.

char *Gal_GetString (Gal_Frame fr, char *key); Returns the string value specified by the key, key, or NULL if the key is not present or its value isnot a string.

int Gal_GetInt (Gal_Frame fr, char *key); Returns the integer value specified by the key, key, or NULL if the key is not present or its valueis not an integer.

float Gal_GetFloat (Gal_Frame fr, char *key); Returns the float value specified by the key, key, or NULL if the key is not present or its value isnot a float.

Gal_Object *Gal_GetList (Gal_Frame fr, char *key, int *length); Returns the list value specified by the key, key. length is set to the length of the list object. If thekey is not present, or the object is not a list, NULL is returned and length is not set.

void *Gal_GetBinary (Gal_Frame fr, char *key, int *size); Returns the binary object value specified by the key, key. size is set to the length of the binaryobject. If the key is not present, or the object is not a binary object, NULL is returned and size isnot set.

void *Gal_GetInt16 (Gal_Frame fr, char *key, int *size) Returns the GAL_INT_16 object value specified by the key, key. size is set to the number ofelements in the object. If the key is not present, or the object is not a GAL_INT_16 object, NULLis returned and size is not set.


233


void *Gal_GetFloat32 (Gal_Frame fr, char *key, int *size) Returns the GAL_FLOAT_32 object value specified by the key, key. size is set to the number ofelements in the object. If the key is not present, or the object is not a GAL_FLOAT_32 object,NULL is returned and size is not set.

void *Gal_GetFloat64 (Gal_Frame fr, char *key, int *size) Returns the GAL_FLOAT_64 object value specified by the key, key. size is set to the number ofelements in the object. If the key is not present, or the object is not a GAL_FLOAT_64 object,NULL is returned and size is not set.

Typed Object FunctionsFor type checking, GAL_CLAUSE_FRAME, GAL_TOPIC_FRAME, GAL_PRED_FRAME, andGAL_FREE (for NULL frame) are valid.

The types GAL_TOKEN, GAL_SYMBOL and GAL_KEYWORD are not intended to be used forvalues in frames, as we understand it. However, if the string representation of your framecontains a value consisting of a single token without delimiting quotes, it may be recognized asone of these elements.

Gal_ObjectType Gal_GetObjectType (Gal_Object to); Returns the object type of the typed object, to.

Gal_ObjectType Gal_GetDetailedType (Gal_Object to); Returns the object type of the typed object, to. If the object is of type GAL_FRAME, the detailedframe type is returned: one of GAL_CLAUSE_FRAME, GAL_TOPIC_FRAME,GAL_PRED_FRAME, or GAL_FREE.

char *Gal_ObjectTypeString (Gal_ObjectType object_type); Returns a character string representing the name of the object type, object_type. This can be a Gal_Object type, a Gal_Frame type, or a binary data type.

Memory management

The string which is returned is a constant which should not be freed.

char *Gal_GetObjectTypeString (Gal_Object to) Returns a character string representing the name of the type of the object to.

234

Memory management

The string which is returned is a constant which should not be freed.

Object Creation FunctionsThe following functions are for converting values into objects. They are most frequently usedwith Gal_SetProp, as in the following example:

void set_clause_topic(Gal_Frame clause, Gal_Frame topic) { Gal_SetProp(clause, ":topic", Gal_FrameObject(topic)); }

Gal_Object Gal_FrameObject (Gal_Frame val); Converts the value, val, into a GAL_FRAME object.

Memory management

Creation: the frame argument is not copied; the Gal_Object which is returned containsa pointer to the frame passed in. The Gal_Object is allocated from an internal store of Gal_Objects. Freeing: the frame which the Gal_Object points to is freed via Gal_FreeFrame, eventhough it was not copied when the object was created. The Gal_Object is returned tothe internal store of Gal_Objects. Copying: the frame is copied using Gal_CopyFrame. A new object is created using Gal_FrameObject.

Gal_Object Gal_CreateFrameObject (Gal_Frame value, int manage_memory) Converts the value, value, into a GAL_FRAME object. If manage_memory is nonzero, the valuewill be freed when the object is freed, otherwise not.

Gal_Object Gal_StringObject (char *val); Converts the value, val, into a GAL_STRING object.

Memory management

Creation: the string is copied using strdup(). The Gal_Object is allocated from aninternal store of Gal_Objects. Freeing: the string is freed. The Gal_Object is returned to the internal store of Gal_Objects. Copying: a new object is created using Gal_StringObject. See Gal_CreateStringObject for memory management options.

Gal_Object Gal_CreateStringObject (char *cp, int manage_memory) Converts the value, cp, into a GAL_STRING object. If manage_memory is nonzero, the valuewill be freed when the object is freed, otherwise not.

235

Gal_Object Gal_IntObject (int val); Converts the value, val, into a GAL_INT object.

Memory management

Creation: the Gal_Object is allocated from an internal store of Gal_Objects. Freeing: the Gal_Object is returned to the internal store of Gal_Objects. Copying: a new object is created using Gal_IntObject.

Gal_Object Gal_FloatObject (float val); Converts the value, val, into a GAL_FLOAT object.

Memory management

Creation: a placeholder for a float is allocated using calloc(), and the value passed in isstored in that location. The Gal_Object is allocated from an internal store of Gal_Objects. Freeing: the float placeholder is freed. The Gal_Object is returned to the internal storeof Gal_Objects. Copying: a new object is created from the contents of the float placeholder using Gal_FloatObject. See Gal_CreateFloatObject for memory management options.

Gal_Object Gal_CreateFloatObject (float *value, int manage_memory) Converts the value, value, into a GAL_FLOAT object. The value must point to allocated memory.If manage_memory is nonzero, the allocated memory will be freed with the object is freed,otherwise not.

Gal_Object Gal_ListObject (Gal_Object *values, int n); Converts the values, values, into a GAL_LIST object, where n is the number of objects in the list.

Memory management

Creation: an array of Gal_Objects (which are actually pointers to structs, not structsthemselves) is allocated using calloc(), and the elements in that array are made to pointto the elements in values; so the array is copied, but not its members. If n is 0 or valuesis NULL, no memory is allocated. The Gal_Object is allocated from an internal store of Gal_Objects. Freeing: each object in the array is freed using Gal_FreeObject, even though theobjects were not copied when the list was created. The array itself is freed. TheGal_Object is returned to the internal store of Gal_Objects. Copying: an array of Gal_Objects is allocated using calloc(), and the elements of theinternal array in the source object are copied using Gal_CopyObject. A new Gal_Objectis allocated from an internal store of Gal_Objects; the Gal_ListObject function is notused directly. See Gal_CreateListObject for explicit memory management options.

236

Gal_Object Gal_ListObjectFromElements (int n, ... ) Converts n values into a GAL_LIST object. Unlike Gal_ListObject, does not require the user tocreate an array of Gal_Objects first. The variable number of values must all be Gal_Objects. Thesame memory management comments apply.

Gal_Object Gal_CreateListObject (Gal_Object *values, int n, void (*free_fn)(void *), int manage_memory) Converts the values, values, into a GAL_LIST object, where n is the number of objects in the list.The resulting list is expandable using the function Gal_ListObjectAdd. The free_fn is a functionwhich can be used to free the elements of the list. There is no default freeing function, so unlike Gal_ListObject, the elements aren’t automatically freed when the list object is freed. If the manage_memory argument is nonzero, the list of objects itself will be freed when the enclosingobject is freed, otherwise not.

int Gal_ListObjectAdd (Gal_Object obj, Gal_Object elt) Add object elt to the dynamic list object obj.

Gal_Object Gal_BinaryObject (void *data, int size); Converts the data, data, into a GAL_BINARY object, where size is the size of the data in bytes.While frames allow binary data to be transported, brokering is a more flexible tool for transmittingaudio data; binary data can be used for encoding arbitrary objects.

Memory management

Creation: an array the same size as data is allocated using malloc(), and the contentsof data are copied using memcpy(). Freeing: the data memory is freed. The Gal_Object is returned to the internal store of Gal_Objects. Copying: a new object is created from the data array using Gal_BinaryObject. See Gal_CreateBinaryObject for explicit memory management options.

Gal_Object Gal_CreateBinaryObject (void *data, int size, int manage_memory) Gal_Object Gal_CreateInt16Object (void *data, int num_int_16, int manage_memory) Gal_Object Gal_CreateInt32Object (void *data, int num_int_32, int manage_memory) Gal_Object Gal_CreateInt64Object (void *data, int num_int_64, int manage_memory) Gal_Object Gal_CreateFloat32Object (void *data, int num_float_32, int manage_memory) Gal_Object Gal_CreateFloat64Object (void *data, int num_float_64, int manage_memory) Converts the data, data, into an array object of the appropriate type (GAL_BINARY,GAL_INT_16, GAL_INT_32, GAL_INT_64, GAL_FLOAT_32, GAL_FLOAT_64), where size isthe size of the data in elements. The data must be of the appropriate type (array of bytes, arrayof 16-bit integers, etc.), coerced to a void *. The resulting object is expandable usingGal_ArrayObjectAdd. If manage_memory is nonzero, data will be freed when the object is freed,otherwise not.

237

int Gal_ArrayObjectAdd (Gal_Object obj, void *data, int size) Adds the data to the array object obj, where size is the number of elements of data. The obj canbe of type GAL_BINARY, GAL_INT_16, GAL_INT_32, GAL_INT_64, GAL_FLOAT_32, or GAL_FLOAT_64.

Object Free and Copy Functions

void Gal_FreeObject (Gal_Object obj); Frees the object, obj, and its value.

Memory management

See the object creation functions for the behavior of objects when they’re freed.

void _gal_free_object (void *obj) Wrapper for Gal_FreeObject for use with functions where memory management is required,such as Gal_CreateListObject.

Gal_Object Gal_CopyObject (Gal_Object obj); Copies the object, obj, and its value.

Memory management

See the object creation functions for the behavior of objects when they’re copied. If one ofthe Gal_Create... functions has been used to create obj, then a "deep copy" is performed,and the new object contains a copy of the underlying data.

Type Checking Predicates

int Gal_Framep (Gal_Object obj); Returns 1 if the object, obj, is a frame, of type GAL_FRAME.

int Gal_Clausep (Gal_Object obj); Returns 1 if the object, obj, is a clause frame, of type GAL_CLAUSE_FRAME.

int Gal_ClauseFramep (Gal_Frame fr) Returns 1 if the frame is of type GAL_CLAUSE.

int Gal_Stringp (Gal_Object obj); Returns 1 if the object, obj, is a string, of type GAL_STRING.

int Gal_Intp (Gal_Object obj); Returns 1 if the object, obj, is an integer, of type GAL_INT.

int Gal_Floatp (Gal_Object obj); Returns 1 if the object, obj, is a float, of type GAL_FLOAT.

238

int Gal_Listp (Gal_Object obj); Returns 1 if the object, obj, is a list, of type GAL_LIST.

int Gal_Binaryp (Gal_Object obj); Returns 1 if the object, obj, is binary data, of type GAL_BINARY.

int Gal_Int16p (Gal_Object to) Returns 1 if the object, to, is binary data, of type GAL_INT_16.



int Gal_Float32p (Gal_Object to) Returns 1 if the object, to, is binary data, of type GAL_FLOAT_32.

int Gal_Float64p (Gal_Object to) Returns 1 if the object, to, is binary data, of type GAL_FLOAT_64.

Object Comparison Predicates

int Gal_ObjectEqual (Gal_Object obj1, Gal_Object obj2); Returns 1 if the objects, obj1 and obj2, are equivalent. The table below lists the functions usedfor equivalence testing of each object type:

object type equivalence function

GAL_FRAME Gal_FrameEqual

GAL_STRING strcmp

GAL_INT, GAL_FLOAT

==

GAL_LISTrecursive comparison using Gal_ObjectEqual; must match one-to-one(not in order)

GAL_BINARY, etc. memcmp

int Gal_ObjectCaseEqual (Gal_Object obj1, Gal_Object obj2); The same as Gal_ObjectEqual except that strings are compared with strcasecmp.

239

Object Value ExtractionEach of the supported types has an extraction function. If the object is not of the expected type,a warning is printed and NULL or zero is returned, as appropriate.

Memory management

None of these functions copy or free the contents of the Gal_Object; they just return thepointer which the Gal_Object contains. This is also true of Gal_UnwrapValue, which returnsthe Gal_Object structure to its internal store, but does not free the contents.

Gal_Frame Gal_FrameValue (Gal_Object to); Extracts a frame from the object to. If the object is not of type GAL_FRAME, a warning is printedand NULL is returned.

Gal_Frame Gal_ClauseValue (Gal_Object to) Extracts a frame of type GAL_CLAUSE from the object to. If the object is not of the appropriatetype, a warning is printed and NULL is returned.

char *Gal_StringValue (Gal_Object to) Extracts a string from the object to. If the object is not of type GAL_STRING, a warning isprinted and NULL is returned. Also works for objects of type GAL_TOKEN, as a special case.

int Gal_IntValue (Gal_Object to) Extracts an integer from the object to. If the object is not of type GAL_INT, a warning is printedand 0 is returned.

float Gal_FloatValue (Gal_Object to) Extracts a float from the object to. If the object is not of type GAL_FLOAT, a warning is printedand 0 is returned.

Gal_Object *Gal_ListValue (Gal_Object obj, int *n) Extracts an array of Gal_Objects (that is, a list) from the object to, and stores the length in n. Ifthe object is not of type GAL_LIST, a warning is printed and 0 is returned.

void *Gal_BinaryValue (Gal_Object obj, int *size) void *Gal_Int16Value (Gal_Object obj, int *size) void *Gal_Int32Value (Gal_Object obj, int *size) void *Gal_Int64Value (Gal_Object obj, int *size) void *Gal_Float32Value (Gal_Object obj, int *size) void *Gal_Float64Value (Gal_Object obj, int *size) Returns the array data and sets size to the size of the data in elements, if the object is of theappropriate type.

240

void Gal_FreeWrapper (Gal_Object to) This function returns the Gal_Object wrapper to its internal store and detaches it from its internalvalue. It only frees the contents if the contents are a float pointer. It does not return the value.

List Object OperationsThe following additional functions are provided for accessing GAL_LIST object elements:

int Gal_ListLength (Gal_Object obj) Returns the length of the list.

Gal_Object Gal_GetListObject (Gal_Object obj, int n) Returns the nth object in the list (counting from 0). Returns NULL if n is out of range. Returns obj if it is not a list.

Memory management

A pointer to the object in the list is returned, not a pointer to a copy of the object.

Binary Object OperationsThe following additional functions are provided for manipulating binary data.

int Gal_BinarySize (Gal_Object to) int Gal_Int16Size (Gal_Object to) int Gal_Int32Size (Gal_Object to) int Gal_Int64Size (Gal_Object to) int Gal_Float32Size (Gal_Object to) int Gal_Float64Size (Gal_Object to) Returns the number of elements of the array object, to, assuming it is of the appropriate type.

int Gal_ObjectByteCount (Gal_Object obj) Returns the actual byte count for the object, obj, which can be GAL_STRING, GAL_BINARY,GAL_INT_16, GAL_INT_32, GAL_INT_64, GAL_FLOAT_32, GAL_FLOAT_64.

Frame Utilities

Frame to String ConversionThese functions are for converting frames and objects to strings for printing and for HUB-servercommunications. There are three forms:

Pr (print) functions generate space-separated tokens. {c a_clause :topic {q the_topic :name "topic_name" :number 3 } } PP (pretty-print) functions generate a human-readable format with line breaks and

241

indentation. {c a_clause topic {q the_topic :name "topic_name" :number 3 } } The outline function truncates the frame: {c a_clause :topic {q the_topic ... } }

Both the Pr and PP formats can be converted back to the original frames with Gal_ReadFrameFromString. char *Gal_PrFrameToString (Gal_Frame fr, char *buf, int *bufsizeptr); Converts a frame to Pr format.

Memory management

This function expands the buffer as necessary or will create a buffer if buf is NULL. If thebuffer isreallocated, the new buffer is returned. Otherwise, the original buffer is returned.

char *Gal_PPFrameToString (Gal_Frame fr, char *buf, int *bufsizeptr); Converts a frame to PP format.

Memory management

This function expands the buffer as necessary or will create a buffer if buf is NULL. If thebuffer is reallocated, the new buffer is returned. Otherwise the original buffer is returned.

char *Gal_ObjectToString (Gal_Object to) Converts an object to Pr format.

Memory management

A new string is returned.

enum {GAL_PP_PRINT , GAL_PR_PRINT };

char *Gal_PrintFrameToString (Gal_Frame fr, char *irpbuf, int *bufsizeptr, int how) Converts a frame to a string according to the format specified by how (either GAL_PP_PRINT orGAL_PR_PRINT). Other arguments are as for Gal_PPFrameToString or Gal_PrFrameToString.

void Gal_PrintFrameToFile (Gal_Frame fr, FILE *fp, int how) Writes a frame to a file according to the format specified by how (either GAL_PP_PRINT or GAL_PR_PRINT).

void GalUtil_PrintObject (int gal_verbose_level, Gal_Object to, int how) Prints a printable representation of the object on standard output according to the formatspecified by how (either GAL_PP_PRINT or GAL_PR_PRINT), if GAL_VERBOSE exceeds gal_verbose_level.

242

Frame Printing FunctionsThe following functions use Gal_PrFrameToString and Gal_PPFrameToString to convert theframe, then print to the specified output device, and analogously for objects.

void Gal_PrFrame (Gal_Frame fr); Prints the contents of the frame, fr, to standard output.

void Gal_PrObject (Gal_Object obj); Prints the object, obj, to standard out.

void Gal_PPFrame (Gal_Frame fr); Pretty prints the contents of the frame, fr, to standard output.

void Gal_PPObject (Gal_Object to) Pretty prints the contents of the object, to, to standard output.

voidGal_PrObjectToFile (Gal_Object obj, FILE *fp); Prints the object, obj, to a file.

void Gal_PPObjectToFile (Gal_Object to, FILE *fp) Pretty prints the contents of the object, to, to the file fp.

void GalUtil_PrObject (int sls_verbose_level, Gal_Object to) Prints the object, to, using sls_verbose_level.

void GalUtil_PPFrame (int sls_verbose_level, Gal_Frame fr) Pretty prints the frame, fr, using sls_verbose_level.

void GalUtil_PPObject (int sls_verbose_level, Gal_Object to) Pretty prints the object, to, using sls_verbose_level.

void GalUtil_CPPFrame (int sls_verbose_level, int fore, int back, Gal_Frame fr) Pretty prints the frame, fr, using sls_verbose_level, using embedded color Xterm directives.

void GalUtil_CPPObject (int sls_verbose_level, int fore, int back, Gal_Object to) Pretty prints the object, to, using sls_verbose_level, using embedded color Xterm directives.

void Gal_PrFrameToFile (Gal_Frame fr, FILE *fp); Prints the contents of the frame, fr, to a file or stream.

void Gal_PPFrameToFile (Gal_Frame fr, FILE *fp); Pretty prints the contents of the frame, fr, to a file or stream.

243

void Gal_OutlineFrame (Gal_Frame fr, int sls_verbose_level); Prints a truncated frame to standard output.

void Gal_OutlineObject (Gal_Object to, int sls_verbose_level) Prints a truncated object to standard output.

Frame Reading FunctionsThese functions read frames or objects from strings and files.

Note : the keys :pred and :PRED are special to these functions when converting strings toframes. For unavoidable historical reasons, their values are treated as elements of the list ofpredicate frames associated with each frame. This list of predicates is used extensively by MIT,but has no function in the general Galaxy Communicator infrastructure. These two keys,therefore, should be avoided.

Gal_Frame Gal_ReadFrameFromString (char *buf); This function creates a frame from Pr or PP format.

Gal_Object Gal_ReadObjectFromString (char *buf) This function creates a Gal_Object from either format.

Gal_Frame Gal_ReadFrameFromFile (FILE *fp) Reads a frame from the file pointed to by fp.

Gal_ObjectGal_ReadObjectFromFile (FILE *fp) Reads an object from a file.

Gal_Frame Gal_VAReadVarFrameFromString (char *buf, int num_pairs, ... ) This function reads a frame from a string. The frame may contain key values of the form $... ,which indicates a variable. The num_pairs are the number of variable mappings which appear asvariable arguments. The first element of each pair is a string whose first character is $, and thesecond element is a Gal_Object. So for example, the invocation

Gal_VAReadVarFrameFromString("{c foo :string $a :int $b }", 2, "$a", Gal_StringObject("bar"), "$b", Gal_IntObject(5));

yields the frame

{c foo :string "bar" :int 5 }

Gal_VarMapping * Gal_CreateVarMapping (int num_pairs, ... ) Gal_Frame Gal_ReadVarFrameFromString (char *buf, Gal_VarMapping *map) An alternative means of incorporating variables. The map should be freed after being createdand used, using free .

244

MIT Linguistic Frame ToolsIn the MIT Galaxy system, frames are used as interpretations of natural language utterances aswell as general-purpose attribute-value structures. As interpretations of natural languageutterances, the different frame types have the following form and purpose:

Clause frames correspond to sentences, and typically have a :topic key-value pair (typicallyfilled by a topic frame) and a :pred key-value pair (filled by a predicate frame). The topiccorresponds to the subject of the sentence, and a single element in the predicate list whichcorresponds to the verb phrase. Manipulating the predicate list is different than setting thevalue of a key named :pred, but since the frame encoder/decoder treats these as the samething, it only makes a difference within a single program. Topic frames correspond to noun phrases. They may have one or more predicates, whichcorrespond to modifiers (adjective phrases, prepositional phrases). Predicate phrases correspond to predicates of all sorts: modifiers (adjective phrases,prepositional phrases, adverbial phrases) and verb phrases. These phrases may havepredicates themselves, and may also have a :topic key-value pair (typically filled by a topicframe) which corresponds to the internal argument (the object of the verb or preposition, for instance).

General Utilities

Gal_Frame Gal_MakeTopicFrame (char *name) Gal_Frame Gal_MakePredFrame (char *name)

Creates a frame of the specified type. See Gal_MakeFrame.

int Gal_Topicp (Gal_Object obj); Returns 1 if the object, obj, is a topic frame, of type GAL_TOPIC_FRAME.

intGal_Predp (Gal_Object obj); Returns 1 if the object, obj, is a predicate frame, of type GAL_PRED_FRAME.

int Gal_TopicFramep (Gal_Framefr) int Gal_PredFramep (Gal_Frame fr) Returns 1 if the frame is of the appropriate frame type.

Gal_Frame Gal_TopicValue (Gal_Object to) Gal_Frame Gal_PredValue (Gal_Object to) Extracts a frame of the appropriate type from the object to. If the object is not of the appropriatetype, a warning is printed and NULL is returned.

245

Frame Searching Functions

Gal_Object Gal_FindKey (Gal_Frame fr, char *key_name); Recursively searches the frame and its contents for the key specified by key_name, using thenormal property list and predicate list operations. If found, the corresponding object is returned.

Gal_Object Gal_MatchKeyValue (Gal_Frame fr, char *key_name, Gal_Object match); Recursively searches the frame and its contents for the key specified by key_name, using thenormal property list and predicate list operations. If found, the object is compared with match using Gal_ObjectEqual. If the objects match, the corresponding object is returned.

Gal_Frame Gal_FindPred (Gal_Frame fr, char *pred_name); Recursively searches the frame for a predicate with name pred_name, using the normal property list and predicate list operations. If found, the predicate frame is returned.

Gal_Frame Gal_FindPredParent (Gal_Frame frame, char *name, Gal_Frame parent, int findpar, int nth); Recursively searches for a predicate with the specified name, name, using the normal property list and predicate list operations. The parent of the first predicate found is returned.

Gal_Frame Gal_FindTopic (Gal_Frame fr, char *topic_name); Recursively searches the frame for a topic with name topic_name, using the normal property listand predicate list operations. If found, the topic frame is returned.

void Gal_DeletePreds (Gal_Frame fr, char *pred_name); Recursively searches the frame for predicates with name pred_name , removes the predicatefrom the frame using Gal_RemPredByName, and deletes the predicate using Gal_FreeObject.

Predicate List Operations

int Gal_NumPreds (Gal_Frame fr); Returns the number of predicates in the predicate list.

Gal_Frame Gal_AddPred (Gal_Frame fr, Gal_Frame pred); Creates a Gal_Object from pred using Gal_FrameObject and adds the object to the predicate listin the frame, fr. The frame must be of type GAL_PRED.

Gal_Frame Gal_GetPred (Gal_Frame fr, int i); Retrieves a predicate frame from the predicate list of fr by its index, specified by i.

Memory management

The retrieved element is the element in fr, not a copy.

246

Gal_Frame Gal_GetPredByName (Gal_Frame fr, char *name); Retrieves a predicate frame from the predicate list of fr by its name, specified by name.

Memory management

The retrieved element is the element in fr, not a copy.

Gal_ObjectGal_RemPred (Gal_Frame fr, int i); Removes a predicate from the predicate list of fr by its index, i, and returns it.

Memory management

The retrieved element is the element from fr, not a copy.

Gal_Object Gal_RemPredByName (Gal_Frame fr, char *name); Removes a predicate from the predicate list by its name, name, and returns it.

Memory management

The retrieved element is the element from fr, not a copy.

void Gal_DelPred (Gal_Frame fr, int i); Removes a predicate from the predicate list by its index, i, and frees it using Gal_FreeObject.

void Gal_DelPredByName (Gal_Frame fr, char *name); Removes a predicate from the predicate list by its name, name, and frees it using Gal_FreeObject.

void Gal_ClearPreds (Gal_Frame fr); Clears the predicate list of fr and frees all predicate objects using Gal_FreeObject.

void Gal_DoPreds (Gal_Frame fr, int (*pred_fn)(Gal_Object , void *), void *caller_data) Applies the function pred_fn to all the predicates in the frame fr. The first argument of pred_fnwill be a Gal_Object containing a frame which is the predicate; caller_data is passed as thesecond argument of pred_fn. The pred_fn should return 1 to continue, 0 to halt.

Deprecated functionsThese functions will be removed in the next major release.

void *Gal_GetListValue (Gal_Object to, int n, Gal_ObjectType type) Calls Gal_GetListObject, ensures the result is of the type type, and returns the contents of theobject as a void *. If the element is not of the appropriate type, NULL is returned. Use of thisfunction is strongly discouraged.

247


248

Building a Communicator-Compliant Server

Steps to Building and Using a Communicator-Compliant Server

Implement the Server Functions Modify the Makefile Stub Template Declare Server Information Create a Hub Program Conditionalize your server for multiple Communicator versions

As an illustration of how to build a Communicator-compliant server, we will use the double serverexample. The double server double.c (together with its dispatch functions, defined in double_core.c) is a simple server that doubles the value of an integer. The complex version ofthe example uses the server multiply.c to exemplify a slightly more complicated interaction.

Implement the Server FunctionsSetting up the headers Initialization Dispatch functions and call environments Writing a Dispatch Function The special reinitialize function Invoking the Hub Replying to the Hub

The server interacts with the Hub in the following ways:

an optional server initialization function _GalSS_init_server() an optional special dispatch function reinitialize functions which invoke the Hub

GalSS_EnvWriteFrame GalSS_EnvDispatchFrame

249

functions which provide replies to the Hub GalSS_EnvError GalSS_EnvDestroyToken

In addition, servers rely on a structure called a call environment to embody the context of eachcall to a dispatch function (the connection through which the call was made, the administrativeinformation that needs to be returned from that call, etc.). As the programmer, you’ll touch theserver object when you initialize the server, and in almost all other cases, you’ll be interactingwith the call environment. The semantics and applicability of call environments has beenradically extended, simplified and improved for version 3.0. See the session managementdocumentation for more details.

Setting up the headersThe header file galaxy/galaxy_all.h declares the appropriate C++ conditionalizations andloads the following four header files:

galaxy/util.h - this provides headers and types for the utilities library galaxy/galaxy.h - this provides headers and types for frames, timed tasks, and more(prefix Gal_) galaxy/galaxy_io.h - this provides headers and types for brokering and more (prefix GalIO_) galaxy/generic-server.h - this provides headers and types for functions like GalSS_EnvWriteFrame() (prefix GalSS_)

Your header information will declare server information by declaring a server declarations fileand then providing instructions for how it should be interpreted. See the section on declaringserver information for details.

Initialization

_GalSS_init_server reinitialize _GalSS_print_usage

There are thre crucial differences between _GalSS_init_server and reinitialize .

_GalSS_init_server is called when the server starts up, while reinitialize is calledeach time the Hub connects to the server _GalSS_init_server is passed initialization information from the command line, while reinitialize is passed initialization information from the Hub _GalSS_init_server is passed a server object while reinitialize is passed a callenvironment object

250

Being aware of these three differences should guide you in choosing where to put various initializations.

void *_GalSS_init_server (GalIO_ServerStruct *server, int argc, char **argv)

Each MIT server accepts the following arguments:

static char *oas[] = { "-port port", "run as a server, listening for client connectionson this port", NULL, "-assert", "exit unhappily if we fail to get the desired port", "-color", "enable color printing in a cxterm", #ifdef _REENTRANT "-thread", "use threads instead of timed tasks (experimental)", #endif /* _REENTRANT */ "-nottloop", "do not use the timed_tasks_loop mechanism", "-ttloop", "obsolete -- maintained for compatibility only", "-maxconns max", "maximum number of connections for ttloop", "1", "-validate", "validate each message send and return against themessage signature", "-verbosity level", "set the verbosity of this server", NULL, "-contact_hub \"host:port...\"", "run as client, contacting Hubsat the specified host and port pairs (overrides default portsetting, but not -port)", NULL, "-server_locations file", "a server locations file of lines serverhost:port [hub|server]", NULL, "-session_id id", "if -contact_hub is used, lock this client tothe specified Hub session", NULL, NULL };

If both -port and -contact_hub are present, the server will start up as both client and listener.

See the oa library for details on how these flags are processed. These arguments will beremoved from the arglist before it is passed to _GalSS_init_server . For more details on howto run this server, see the section on the server executables.

When the server is first run, the generic server library will initialize the server by calling _GalSS_init_server . If the function is not defined, the default in the library will be used. Theargument of this function is a GalIO_ServerStruct *. This is the server structure whichmaintains information about the default listen port, the dispatch functions, the server name, thenumber of connections, etc. For more details about what functions can be used to manipulatethe server object, see the server architecture documentation.

251

The _GalSS_init_server function in double_core.c sets the initial increment, extracted fromthe command line.

static char *oas[] = { "-increment i", "set the initial value", "1", NULL };

void *_GalSS_init_server(GalIO_ServerStruct *s, int argc, char **argv) { int i, increment = 1;

if (!GalUtil_OACheckUsage(argc, argv, oas, &i)) exit(1); GalUtil_OAExtract(argc, argv, oas, "-increment", GAL_OA_INT, &increment); GalIO_SetServerData(s, (void *) increment, NULL); return (void *) NULL; }

Returning a non-NULL value from _GalSS_init_server has the same effect as calling GalIO_SetServerData, although using this capability is strongly discouraged.

void _GalSS_print_usage (int argc, char **argv) This is one way for the server to print out its usage information. The default way of defining thisfunction is

void _GalSS_print_usage(int argc, char **argv) { GalUtil_OAPrintOptions(argc, argv, oas, NULL); printf("\n"); }

The function GalUtil_OAPrintOptions is provided in connection with the oa library, and can beused when the appropriate argument array is defined. See the example at the beginning of this section.

Dispatch functions and call environmentsThe appropriate execution of dispatch functions requires maintaining a variety of administrativeinformation: the token index by which the Hub knows the message it sent, the sessionassociated with that token index, whether or not the Hub expects a return value, the location inthe Hub script that the Hub issued the message from, etc. This information is embodied in astructure known as a call environment . This call environment, of type GalSS_Environment * , is the second argument to every dispatch function (although, for historical reasons, theargument is typed as a void * ).

252

It’s important that every reply to a dispatch function include the administrative informationassociated with that call. The various ways of providing a dispatch function reply all guaranteethat this information will be included (returning a frame from a dispatch function, as well ascalling the functions GalSS_EnvDestroyToken, GalSS_EnvError, or GalSS_EnvReply, all ofwhich we’ll discuss in a moment). The call environment is also used to send new messages,using the functions GalSS_EnvWriteFrame and GalSS_EnvDispatchFrame.

Although the call environment is almost always the proper channel through which tocommunicate with the Hub, the actual communication is done via the connection object, which isstored in the call environment and is accessible via the function GalSS_EnvComm.

For more details about functions which manipulate the connection object, see the server architecture documentation. For more details on the use and manipulation of call environments, see the session management documentation.

Writing a Dispatch FunctionDispatch functions take frames as arguments, and return frames. The call environment cachesthe appropriate administration information and add it to the dispatch function reply. The returnvalue from the dispatch function is always treated as the reply, unless a reply has beenpreviously provided, in which case it will be ignored. Similarly, as of 3.0, if the dispatch functionreturns NULL, and the Hub requires a reply, the server bindings will write back a pacifier reply.

multiply.c defines a dispatch function, multiply , which multiplies a number in the :int slot bya predefined factor and returns the number.

Gal_Frame multiply(Gal_Frame frame, void *server_data) { int i = Gal_GetInt (frame, ":int");

if ((INT_MAX / i) < Factor) { /* If we’re about to overflow... */ GalSS_EnvError ((GalSS_Environment *) server_data, "multiply would overflow MAXINT"); return (Gal_Frame) NULL; } else { Gal_SetProp (frame, ":int", Gal_IntObject (Factor * i)); return frame; } }

253

Memory management

Both the frame which is passed to the dispatch function and the frame which is returned (ifdifferent) are freed by the toplevel loop using Gal_FreeFrame. No dispatch function shouldfree the incoming frame. See the frame documentation for detailed memory managementcomments about frames.

If you need to cache information per connection across calls to dispatch functions, it is possibleto set and get connection-specific data directly through the environment object.

void *GalSS_EnvGetCommData (GalSS_Environment *env) Retrieves the connection-specific data from the connection object stored in env. See also GalIO_GetCommData.

void GalSS_EnvSetCommData (GalSS_Environment *env, void *data, void (*free_fn)(void *)) Sets the data specific to the connection stored in env.. If free_fn is non-NULL, it will be called onthe data when the data is reset or the connection is destroyed. See also GalIO_SetCommData.

If you’ve cached information specific to a given dispatch function, it is possible to access thisinformation directly through the environment object. void *GalSS_EnvGetClientData (GalSS_Environment *env, char *name) Retrieves the information specific to the dispatch function name from the connection objectstored in env. See also GalIO_GetCommClientData.

Invoking the HubIn some cases, you might want to send a new message to the Hub, or query the Hub and waitfor a response (we call these latter synchronous calls server-to-server subdialogues ). Themessage and response will be in the form of a frame.

You should always use the call environment to send these messages. It is possible to send themusing the connection alone (accessible via the call environment using the function GalSS_EnvComm(); see the server structure documentation for details), but important sessioninformation will be lost. When you’re inside a dispatch function, using the call environment isstraightforward; if you’re outside a call environment (if you’ve set up a brokering callback or atimed task), you must take special precautions to make sure the call environment isn’t freed afterthe dispatch function exits. See the session management documentation for more details.

When a message is written to or read from the Hub, the type of the message is also specified.The possible types are:

254

GAL_MESSAGE_MSG_TYPE A new message

GAL_REPLY_MSG_TYPE A normal reply to a message

GAL_ERROR_MSG_TYPE An error reply to a message

GAL_DESTROY_MSG_TYPEA destroy request for the specified token (also counts as amessage reply)

GAL_POSTPONE_MSG_TYPEA reply from the server informing the Hub that the messagereturn will be provided later (see the documentation on continuations)

If the message type is GAL_ERROR_MSG_TYPE, the accompanying frame will currently havethe following form (use GalIO_GetError to access the values):

{c system_error :errno <num> :err_description <string> ...}

At this point, possible errors are:

GAL_APPLICATION_ERROR Some server generated an error via GalSS_EnvError

GAL_TRANSMISSION_ERROR An error was encountered in sending a frame

GAL_RECEPTION_ERROR An error was encountered in reading a frame

GAL_NO_OPNAME_ERROR Some server did not implement a requested operation

GAL_SERVER_DOWN_ERROR The Hub could not contact a required server

GAL_NO_FRAME_ERROR A dispatch was requested on an empty frame

int GalSS_EnvWriteFrame (GalSS_Environment *env, Gal_Frame frame, int do_block) This function writes frame to the Hub through the connection object stored in the env. If do_blockis set, the function guarantees that the write has happened before returning. The message typewritten is always GAL_MESSAGE_MSG_TYPE.

double_core.c defines a dispatch function, twice , which takes the value of :int , doubles it,introduces a new message to the Hub with the doubled integer, and returns NULL. If it is aboutto overflow MAX_INT, or if the number it’s supposed to double is 0, then it generates an error.

Gal_Frame twice (Gal_Frame frame, void * server_data) { int i = Gal_GetInt (frame, ":int"); char *program_name = Gal_GetString (frame, ":program");

255

Gal_Frame new_f;

if (!program_name) program_name = "main";

if (i == 0) { /* We’ll loop forever. */ GalSS_EnvError ((GalSS_Environment *) server_data, "i is 0"); return (Gal_Frame) NULL; } else if ((INT_MAX / i) < 2) { /* If we’re about to overflow... */ GalSS_EnvError ((GalSS_Environment *) server_data, "double would overflow MAXINT"); return (Gal_Frame) NULL; } else { new_f = Gal_MakeFrame (program_name, GAL_CLAUSE); Gal_SetProp (new_f, ":int", Gal_IntObject (2 * i)); Gal_SetProp (new_f, ":program", Gal_StringObject (program_name)); GalSS_EnvWriteFrame ((GalSS_Environment *) server_data, new_f, 0); Gal_FreeFrame (new_f); return (Gal_Frame) NULL; } }

Gal_Frame GalSS_EnvDispatchFrame (GalSS_Environment *env, Gal_Frame frame,GalIO_MsgType *t) This function implements a server-to-server subdialogue with the Hub. It sends the framethrough the connection object stored in the env and waits for a reply. This function isimplemented in terms of GalIO_DispatchViaHub. The message type written is alwaysGAL_MESSAGE_MSG_TYPE. The type of the reply is stored in t. The only values for *t you’llever see are *t are GAL_REPLY_MSG_TYPE and GAL_ERROR_MSG_TYPE; for all others, awarning message is printed and NULL is returned.

int GalIO_GetError (Gal_Frame f, char **err_desc) Retrieves an error from a error frame returned from GalSS_EnvDispatchFrame. If the frame isnot a well-formed error frame, this function will return -1 and set *err_desc to NULL if err_desc isprovided. If the frame is a well-formed error frame, this function will return an error code, and set *err_desc if err_desc is provided. If an error description is present in the frame, the descriptionwill be a string, otherwise NULL.

double_core.c defines a dispatch function, complex_twice, which doubles the result it gets fromthe multiply server:

256

Gal_Frame complex_twice(Gal_Frame frame, void *server_data) { Gal_Frame new_f = Gal_MakeFrame ("multiply", GAL_CLAUSE); Gal_Frame res_f; GalIO_MsgType t; int i;

Gal_SetProp (new_f, ":int", Gal_IntObject ( Gal_GetInt (frame, ":int"))); res_f = GalSS_EnvDispatchFrame ((GalSS_Environment *) server_data,new_f, &t); Gal_FreeFrame (new_f);

if (!res_f) { GalUtil_Warn ("Didn’t hear back from multiply"); return (Gal_Frame) NULL; }

switch (t) { case GAL_REPLY_MSG_TYPE: prog_name = Gal_GetString (frame, ":program"); if (!prog_name) prog_name = "main"; Gal_SetProp (new_f, ":program", Gal_StringObject (program_name)); return twice(res_f, server_data); case GAL_ERROR_MSG_TYPE: return (Gal_Frame) NULL; default: return (Gal_Frame) NULL; } }

Replying to the HubTypically, the return value of a dispatch function will be treated as the reply to the message theHub sent. However, the programmer can provide a reply in advance of returning from thedispatch function. In all cases where a reply is provided in advance, the return value of thedispatch function is ignored.

Note that it is also possible to postpone the reply to a message beyond the scope of the dispatchfunction. The functions for doing this and the motivation for it are discussed extensively in the session management documentation.

int GalSS_EnvError (GalSS_Environment *env, char *description) Reports an error as the return value. The message type of the message written is alwaysGAL_ERROR_MSG_TYPE; it is annotated with the administrative information of the framepassed in. If you use this function, the return value from the dispatch function will be ignored.

257

int GalSS_EnvDestroyToken (GalSS_Environment *env) Makes a destroy request. The token information is taken from the frame passed in. Themessage type of the message written is always GAL_DESTROY_MSG_TYPE. If you use thisfunction, the return value from the dispatch function will be ignored.

int GalSS_EnvReply (GalSS_Environment *env, Gal_Frame f) Provides the frame f as a normal reply of type GAL_REPLY_MSG_TYPE. This function isprovided mostly for completeness, since it duplicates the functionality embodied in returning aframe from the dispatch function. However, if for some reason you want a dispatch function tokeep running past its "natural" reply point, you can use this function to satisfy the Hub’sexpectation of a reply.

Gal_Frame reinitialize (Gal_Frame frame, void *server_data);

The dispatch function reinitialize is special. It is called when the Hub first connects to theserver. Any connection-specific initializations should be put in the reinitialize dispatchfunction. Like all other dispatch functions, the second argument is typed to void * forbackward compatibility, but can reliably be cast to GalSS_Environment * .

As of version 3.0, reinitialize is no longer restricted in its use. However, because it is usedto initialize a connection, it’s worth observing that while is convenient to use the reinitializemessage to "seed" the Hub by providing a new token via GalSS_EnvWriteFrame(), GalSS_EnvDispatchFrame should not be used, for three very good reasons:

since the Hub does not start processing new messages until all the reinitializemessages for all the servers are done, the request will be queued and never return;therefore, the reinitialize message itself will hang if the server did not wait for a response, it would get one later, during normal processing,when it did not expect it since reinitialize messages are handled essentially in parallel as far as the Hub isconcerned, it is not illegitimate to attempt to contact other servers (which may not have beeninitialized yet).

The frame passed to reinitialize at initialization time is constructed very similarly to the initial token.

1. The frame is seeded with same global values as the implicit initial token, including thevalues from the INITIAL_TOKEN: directive

2. The frame is augmented as follows:

frame key source type default value

<frame name> (specified in code; cannot be changed) string reinitialize

:server_typethe name of the server as the Hub knows it, ifnot already specified

string <none>

258

3. The frame is augmented (or overwritten) by the values specified in the INIT: directive forthe server in the program file.

4. The frame is augmented (or overwritten) with the contents of the -init command lineargument to the Hub executable, if present.

Modify the Makefile Stub TemplateIn the server directory, copy templates/Makefile.stub to use as the server Makefile. Make theappropriate modifications, for example, setting the root directory. (For more details on themakefiles, see the Guide to the Makefiles.) We provide a table here of all the elements you cancustomize in the MIT Makefile, what they do, and whether there is a "slot" for them in the stub Makefile:

Variable What it does Obligatory? Slot?

MAKEFILEDeclares the name of the file, in order togenerate the Makefile dependencies correctly

yes yes

ROOT_DIRThis is the root of the Communicator distribution

yes yes

CPPFLAGS

The usual C preprocessor flags. MIT providesa special preprocessor directive-DREPLACE_PRINTF which will cause allcalls to printf() to be replaced with calls to GalUtil_Pinfo1().

no yes

LDFLAGS The usual C linker flags. no yes

COMMON_LIBSDefined here and augmented in templates/Galaxy.make . Third party libraries without variants.

no yes

SPECIAL_LIBS, SPECIAL_LIBS_D, SPECIAL_LIBS_P

Libraries with variants (_debug, _profile) whichare particular to the current executable. If anyof the variants are non-standard, set SPECIAL_LIBS_D and/or SPECIAL_LIBS_P .These will be arguments to the link line.

no yes

LIBDEPS LIBDEPS_D, LIBDEPS_P

A list of libraries, which will be Makefiledependences (so real pathnames, not -lvalues). These should correspond to thespecial and common libraries.

noyes for LIBDEPS

259

LIBTARGET, EXECTARGETS, SERVER, APPLET

The primary target types. Exactly one of thesemust be set. The SERVER is the option forbuilding a Communicator-compliant serverwhich automaticaly generates the operationsheader file.

yes yes

LIBDIR

The directory to put a library in. Defaults to $(ROOT_DIR)/lib/ . Note the trailing slash.The library will be put in the appropriate $(ARCHOS) subdirectory. We advise settingthis so as not to write to the Communicator distribution.

no no

EXECDIR

The directory to put an executable or server in.Defaults to $(ROOT_DIR)/bin/ . Note thetrailing slash. The executable or server will beput in the appropriate $(ARCHOS)subdirectory. We advise setting this so as notto write to the Communicator distribution.

no no

THREAD_SAFE

Whether the server or library supportsthread-safe compilation. If you uncomment thisline, the compilation instruction "make thread"will compile a threaded version of the library orserver. If you leave this line uncommented, thisinstruction will compile the normal version. Seethe thread notes.

no yes

SUBDIRSA list of subdirectories where some of thesources may be.

no yes

SOURCES The source files. yes yes

Declare server informationGalaxy Communicator 2.0 and later uses the C preprocessor to generate server information,especially mappings between dispatch function names and dispatch functions.

Basics Incorporating other software packages The headerless alternative

260

BasicsThere are two steps involved in declaring server information.

Create the declaration file

The declaration file contains several types of information:

A declaration GAL_SERVER_NAME(name), which contains the name of the server. Thisname potentially corresponds to the name of a service type in the Hub. This server namedoesn’t need to match the name the Hub has in its SERVICE_TYPE: declaration, except inthe case of listeners in the Hub. A declaration GAL_SERVER_PORT(port), which contains the default port number. A declaration GAL_SERVER_OP(dispatch_fn) or GAL_SERVER_OP_SIGNATURE(dispatch_fn, in_keys, allow_other_in_keys, reply_status,out_keys, allow_other_out_keys) for each operation. A declaration GAL_SERVER_SW_PACKAGE(pkg), which may contain a name of a software package to use. A declaration GAL_SERVER_MAX_CONNECTIONS(numconns), which contains the defaultmaximum number of simultaneous connections permitted for this server. This can beoverridden by the -maxconns command line argument. If neither is specified, the default isone connection.

The double server declaration file lists twice as one of its available operations:

GAL_SERVER_NAME(double) GAL_SERVER_PORT(2800) GAL_SERVER_OP(twice) GAL_SERVER_OP(complex_twice) GAL_SERVER_OP(reinitialize)

Here’s a version of the double server declaration file using the extended signatures:

GAL_SERVER_NAME(double) GAL_SERVER_PORT(2800) GAL_SERVER_OP_SIGNATURE(twice, GAL_SERVER_OP_KEYS(":int" _ GAL_INT _ GAL_KEY_ALWAYS), GAL_OTHER_KEYS_NEVER, GAL_REPLY_NONE, NULL, GAL_OTHER_KEYS_NEVER) GAL_SERVER_OP_SIGNATURE(complex_twice, GAL_SERVER_OP_KEYS(":int" _ GAL_INT _ GAL_KEY_ALWAYS), GAL_OTHER_KEYS_NEVER, GAL_REPLY_NONE, NULL,

261

GAL_OTHER_KEYS_NEVER) GAL_SERVER_OP_SIGNATURE(reinitialize, NULL, GAL_OTHER_KEYS_NEVER, GAL_REPLY_PROVIDED, NULL, GAL_OTHER_KEYS_NEVER)

The extended version which specifies the signature works as follows:

262

argument description legal values

dispatch_fnthe name of a dispatch function

in_keys

the keyswhich thisdispatchfunction expects

Either NULL or a declarationGAL_SERVER_OP_KEYS(key _ type _ obligatory ... ),where underscores are used in place of commasbecause of the idiosyncracies of the C preprocessor.Here key is a string, type is a legal object type, andobligatory is either GAL_KEY_ALWAYS orGAL_KEY_SOMETIMES. This declaration may containany number of key declarations (that is, its argumentsmust be a multiple of 3, all separated by underscores).The type GAL_FREE is used as a wildcard match.

allow_other_in_keys

whether ornot the list ofin_keysprovided is complete

either GAL_OTHER_KEYS_MAYBE or GAL_OTHER_KEYS_NEVER

reply_status

whether ornot thedispatchfunctionreturns anything

one of GAL_REPLY_PROVIDED,GAL_REPLY_NONE, GAL_REPLY_UNKNOWN

out_keys

the keyswhich thisdispatchfunction returns

(see in_keys)

allow_other_out_keys

whether ornot the list ofout_keysprovided is complete

(see allow_other_in_keys)

These signatures can be used to validate calls to the dispatch functions if you pass the -validate flag to the server, and will be used in the future to pass to the Hub and synchronize signatures.

263

Reference the declaration file

Instead of adding "server.h" to exactly one file in your server sources, add the appropriatereference to the declaration file. The declaration file will be #included in a context whichtranslates the declarations in the declaration file into the appropriate C code. This reference#defines SERVER_FUNCTIONS_INCLUDE, and then includes galaxy/server_functions.h .In addition, the reference should #define USE_SERVER_DATA. Both the #defines must be donebefore the header file is #included.

Here is the header text from double.c:

#include "galaxy/galaxy_all.h" #define SERVER_FUNCTIONS_INCLUDE "double_server.h" #define USE_SERVER_DATA #include "galaxy/server_functions.h"

This technique for providing server information is compatible with non-MIT Makefiles (as well asbeing cross-platform-friendly). If not using the MIT Makefile templates to build your servers, besure that you have -I. on your compile line to ensure that the compiler can find the declaration file.

Note : the value of SERVER_FUNCTIONS_INCLUDE is included multiple times in server_functions.h . Therefore, the usual header programming practice of ensuring theheader is loaded only once will cause the declaration macros to fail. In other words, don’t do this:

#ifndef __MY_SERVER_H__ #define __MY_SERVER_H__

GAL_SERVER_NAME(my_server) ... #endif

Incorporating other software packagesYou can use the GAL_SERVER_SW_PACKAGE() macro in the server header to declare asoftware package you would like to use. If you do this, the compiler will look for a function _GalSS_configure_<name> , where <name> is the argument of the macro. Typically, a libraryyou link against will provide this function; in the case of MIT dialogue processing, it’s provided inthe dialogue headers. The function should take a single GalIO_ServerStruct * argumentand return void . For an example, see the turn management documentation.

The headerless alternativeAll the header incorporation does, really, is define the references to the dispatch functions andcreate a function called _GalSS_InitializeDefaults, which is called by the toplevel initializationprocedure. This function stores the initialization information in the server in the appropriate way.You can do the same thing yourself. The file headerless_double.c exemplifies this strategy:

264

/* We don’t need any signatures because the functions have alreadybeen defined. */

void _GalSS_InitializeDefaults(GalIO_ServerStruct *s) { GalSS_InitializeServerDefaults(s, "double", 2800); GalSS_AddDispatchFunction(s, "twice", twice, NULL, GAL_OTHER_KEYS_MAYBE, GAL_REPLY_UNKNOWN, NULL, GAL_OTHER_KEYS_MAYBE); GalSS_AddDispatchFunction(s, "complex_twice", complex_twice, NULL, GAL_OTHER_KEYS_MAYBE, GAL_REPLY_UNKNOWN, NULL, GAL_OTHER_KEYS_MAYBE); GalSS_AddDispatchFunction(s, "reinitialize", reinitialize, NULL, GAL_OTHER_KEYS_MAYBE, GAL_REPLY_UNKNOWN, NULL, GAL_OTHER_KEYS_MAYBE); }

void _GalSS_InitializeDefaults (GalIO_ServerStruct *scomm) User-defined function which declares static server information. Usually generated automaticallyduring server declaration generation.

The following typedefs are relevant to the functions below:

typedef Gal_Frame (*Gal_FrameFnPtr)(Gal_Frame frame); typedef Gal_Frame (*Gal_FrameDataFnPtr)(Gal_Frame frame, void *data);

void GalSS_InitializeServerDefaults (GalIO_ServerStruct *scomm, char *name, unsignedshort port) Stores the name and port and function_map in the server scomm. The name corresponds to aHub service type.

void GalSS_AddDispatchFunction (GalIO_ServerStruct *i, char *name,Gal_FrameDataFnPtr fn, Gal_DispatchFnSignatureKeyEntry *in_key_array, int allow_other_in_keys, int reply_provided, Gal_DispatchFnSignatureKeyEntry *out_key_array, int allow_other_out_keys) Adds a fn indexed by name to the server i. This function takes a frame and a void * pointer(actually a GalSS_Environment * ) and returns a frame. By convention, the index and thename of the function are the same, and that’s the way the header strategy implements things,but this is not required (as long as the programmer is willing to assume responsibility fornegotiating the digression). The possible values of allow_other_in_keys, reply_provided, andallow_other_out_keys are the same as for the GAL_SERVER_OP_SIGNATURE() macro. Thein_key_array and out_key_array are created by the function Gal_CreateDispatchFnKeyArray().

265

Gal_DispatchFnSignatureKeyEntry *Gal_CreateDispatchFnKeyArray (int ignore, ... ) This function takes an arbitrary number of arguments. The first argument is ignored; it is presentbecause the ANSI C mechanism for variable arguments requires at least one listed argument.The argument list must terminate with a NULL; otherwise, it’s identical to the arguments to theGAL_SERVER_OP_KEYS() macro. Here’s an example:

Gal_CreateDispatchFnKeyArray(0, ":int", GAL_INT, GAL_KEY_ALWAYS, NULL);

Memory management

The function GalSS_AddDispatchFunction copies the key array, so if you create an array topass to GalSS_AddDispatchFunction, you must free it using Gal_FreeDispatchFnKeyArray.

void Gal_FreeDispatchFnKeyArray (Gal_DispatchFnSignatureKeyEntry *entry) Frees a key array created by Gal_CreateDispatchFnKeyArray.

Create a Hub ProgramIn order to use this server, a Hub needs to know how to find it. To use the server we’ve definedhere, the Hub would typically be passed a program file containng information like the following:

SERVER: double-server PORT: 2800 HOST: localhost OPERATIONS: twice

This entry instructs the Hub to contact the server in question on port 2800 on the local machine,and informs the Hub that the server supports a dispatch function named twice . (It’s alsopossible to have the server contact the Hub instead.) In addition, you might want to tell the Hubwhat to do with messages the server sends to it:

PROGRAM: main

RULE: :int --> twice IN: :int OUT: none!

This program sends the value of the :int key in the current token to the server which supportsthe twice dispatch function, when the :int key is present in the current token. (It’s alsopossible for the Hub to run without a program.)

266

Conditionalize your server for multiple Communicator versionsThings unavoidably change between releases, and you may wish to retain the option ofcompiling your servers against multiple versions of the Galaxy Communicator distribution. TheGC_VERSION definition is designed to do this for you. Here’s a simple illustration of how to use it:

#include "galaxy/galaxy_all.h"

#if defined(GC_VERSION) && (GC_VERSION >= 0x30000) char *version = "3 point 0"; #else #if defined(GC_VERSION) && (GC_VERSION >= 0x20000) char *version = "between 2 point 0 and 3 point 0"; #else #ifndef GC_VERSION char *version = "before 2 point 0"; #endif #endif #endif

The value of GC_VERSION is a number in hexadecimal form, such that the lowest two hex digitsare the "subminor" version, the next two hex digits are the minor version, and everything abovethat is the major version. So Galaxy Communicator version 4.2.3 (if we ever get there) will havea GC_VERSION of 0x40203.


267

Communicator Hub Programs

The Galaxy Communicator uses a configurable Hub. This configurability is accomplished byhaving the Hub read in a script upon start-up. This script is written in a simple, Hub-specificscripting language. The Hub program, or script, contains information about available servers(name and location) and their externally accessible operations, as well as possible flow of control instructions.

The program file must be specified via the command line option: -pgm_file filename. If no file isspecified, the Hub will exit.

Each directive of the program file must contain no more than one expression, and begin with avalid, system defined identifier name. Blank lines are permitted, and are also used to terminatesmall declaration blocks (see the rest of the documentation for details). A comment is introducedwhen the first non-whitespace character of a line is either a semicolon (;). Line breaks indirectives are permitted using the \ (backslash) character. Directives are terminated by anunescaped newline.

Identifiers are case sensitive. However, by convention, identifiers are all upper-case with atrailing colon, :. Identifiers can be used to specify:

global variables in the form of key-value pairs server declarations (service type and provider definitions) named programs (rules and associated parameters)

There is no macro expansion or evaluation. There is also no nesting of rules or scoping,although modules are supported via the PROGRAM: identifier.

Note: any newlines in the examples below are a side effect of presentation; newlines arepermitted in entries in program files only if they are escaped with the backslash character.

The Hub Program File StructureA Hub program file contains the following constructs:

Values Subfiles Expression syntax

268

Directives Simple values Key-value pairs

Optional value format Obligatory value format

Rule conditions Extended syntax mode

These constructs are used to form the following logical blocks:

Global Declarations Logging Parameters

Server Declarations Service Types Service Providers Other server directives

Hub Programs Name Rules Parameters

Although global declarations can be defined anywhere, servers must be defined beforeprograms. Typically a script will specify global variables first, then servers. The file may alsocontain one or more program definitions. The frame names of incoming messages which are notreplies to messages sent by the Hub names are treated as program names if the namedprogram exists; otherwise, the incoming message is treated as an invocation of a serveroperation. If no matching program or server operation is found, the message is discarded. Seethe discussion of how a system works for more details, as well as the documentation on usingthe Hub in scriptless mode.

The distribution contains a number of program files (default extension .pgm ) as examples. See contrib/MITRE/example and contrib/MITRE/demos.

ValuesAs of version 1.3, the program file reader uses the same underlying reader that’s used to digest string representations of frames. As a result, there is a well-defined system of value parsingthroughout the program files.

Frames, ints, floats and lists should be formatted just as their corresponding types are inside frames. Strings should be formatted just as GAL_STRINGs are (that is, with double quotes aroundthem). As a special case, a string which consists of a single token (no whitespace) does notneed to be delimited by double quotes.

269

SubfilesA program file can be decomposed into subfiles. To include a subfile into the main program file use:

#include <filename>

The filename should be a string, so double quotes are optional (since it’s a single token). If <filename> includes no directory information, the current file directory is used as the default. Nested includes are permissible. The contents of included files are incorporated as read in, so#includes can be places anywhere within the program file. If the filename is a relative pathname,it will be interpreted relative to the directory where the program file which contains the referenceis located.

Note: as of version 1.3, logging instructions are no longer ignored in subfiles.

Expression syntaxIn addition to the syntax of values, program files can contain a number of different constructs.Here’s a fragment of a program file which exemplifies the different constructs, which we will useto elaborate.

SERVICE_TYPE: parser OPERATIONS: parse_string INIT: :speed "medium" (:accuracy "high")

SERVICE_PROVIDER: parser LOCATION: localhost:14000 IN: (:language "English")

PROGRAM: main

RULE: :input_string && :language == "English" --> parser.parse_string IN: (:string :input_string) :user

DirectivesAll directive entries in the program file consist of a single logical line. These logical lines cancontain newlines if they are escaped with a backslash (\), which must be the last character in thephysical line. The first token in each directive entry is the directive, which always ends with acolon (:). Examples in the program file fragment here are SERVICE_TYPE:, INIT:,OPERATIONS:, etc. For readability, we recommend leaving at least one space in between thedirective and its value; however, it is not necessary to do so.

270

Simple valuesDirectives like PROGRAM: and SERVICE_TYPE: accept a quoted or unquoted string as avalue. This value is interpreted as a string regardless. If a single value is expected, the directivevalue must be quoted. If multiple values are expected, a sequence of tokens, quoted orunquoted, will be interpreted as the appropriate sequence of string.

Key-value pairsThe Hub maintains a number of memory states, which we call namespaces.

State descriptionNamespace name

Each token is associated with a state, which is initially populated by a newincoming message and updated by the Hub program

token

Each session is associated with a state, which can be updated by servers orby Hub program instructions

session

Each server (actually, service provider) the Hub knows about is associatedwith a state, which can be updated by the server or, in extended syntax mode,by Hub program instructions

server

Each message, incoming and outgoing, counts as a state message

Global settings are stored as their own state global

All key-value pairs in the Hub program language have a target namespace, and in some cases asource namespace as well. There are two types of key-value pair formats in program files.

Optional value format

In optional value format, exemplified by the IN: lines in the example program file, there are threeformat options:

271

Example Interpretation

:user

A single key in optional value format is an instruction to draw the valuefrom the source namespace and store it under the identical key in thetarget namespace. In the case of IN:, the source namespace is thetoken, and the target namespace is the message.

(:string :input_string)

A parenthesized pair where the second element is an unquoted string isan instruction to draw the value from the source namespace under thekey named by the second element (:input_string , here) and store itin the target namespace under the key named by the first element (:string , here). In the case of IN:, the source namespace is the token,and the target namespace is the message.

(:language "English")

A parenthesized pair where the second element is a literal value is aninstruction to store that value ("English" , here) in the targetnamespace under the key named by the first element (:language ,here). In the case of IN:, the target namespace is the message.

Obligatory value format

In obligatory value format, exemplified by the INIT: line in the example program file, only literalvalues are permitted. There are two format options.

Example Interpretation

:speed "medium"

An unparenthesized pair where the second element is a literal value is aninstruction to store that value ("medium" , here) in the target namespaceunder the key named by the first element (:speed , here). In the case ofINIT:, the target namespace is the reinitialize message.

(:accuracy "high")

A parenthesized pair where the second element is a literal value is aninstruction to store that value ("high" , here) in the target namespace underthe key named by the first element (:accuracy , here). In the case of INIT:,the target namespace is the reinitialize message.

Rule conditionsThe RULE: directive, as exemplified in the program file, accepts a complex rule condition expression.

Constructing a rule condition

The grammar for condition expressions is as follows:

<conditions> = <condition> (<connective> <condition>)* <connective> = & | && | | | || (that is, (2) ampersand(s) or (2)

272

vertical bar(s)) <condition> = (!) <key> | <key> <restriction> <condition> = ( <conditions> ) (that is, conditions surrounded by parentheses) <restriction> = (!) <int_comparison> <int> <int_comparison> = > | < | >= | <= | =(=) <restriction> = (!) <string> | (!) <string_comparison> <string> <string_comparison> = % | ^ | =(=)

Whitespace is required between the key and the restriction, but whitespace is optional betweenthe comparison and the value. As noted, parentheses can be used to group the elementsexplicitly, and these new conditions can be negated, etc.

Note: The % comparison operator still works, but will be deprecated in favor of the ^ comparisonin the future.

Examples of valid conditions

condition interpretation

:foo the key is present in the target namespace

!:foo ! :foo

the key is not present in the target namespace

:foo = 5 :foo =5 :foo == 5 :foo 3 etc.

the key value in the target namespace is the specified integer (note thatthe right hand side can be a key referring to an integer in extendedsyntax mode)

:foo > 5 :foo < 5 :foo >5 :foo <5

the key value in the target namespace is an integer which is greater(less) than the specified integer (note that the right hand side can be akey referring to an integer in extended syntax mode)

:foo IN ( "a""b" )

the key value in the target namespace is a member of the specified list,using Gal_ObjectEqual for comparison (note that the right hand sidecan be a key referring to a list in extended syntax mode)

:foo % bar :foo ^ bar :foo % "bar" :foo ^ "bar" :foo %"bar" :foo ^bar etc.

the key value in the target namespace is a string with the specifiedsubstring (note that unquoted literal values are interpreted as keys in extended syntax mode)

273

:foo ! % bar :foo !^ bar :foo !% "bar" :foo ! ^ "bar" etc.

the key value in the target namespace is a string which does notcontain the specified substring (note that unquoted literal values areinterpreted as keys in extended syntax mode)

:foo bar foo = bar :foo ="bar" :foo == bar etc.

the key value in the target namespace is the specified string (note thatquoteless integer values are interpreted as integers, not strings, andthat unquoted literal values are interpreted as keys in extended syntax mode)

:foo ! bar :foo !"bar" foo != bar :foo !="bar" etc.

the key value in the target namespace is not the specifed string (notethat quoteless integer values are interpreted as integers, not strings,and that unquoted literal values are interpreted as keys in extendedsyntax mode)

<condition> & <condition> <condition> && <condition> (<condition> & <condition>) etc.

conjunction of conditions

<condition> | <condition> <condition> || <condition> (<condition> | <condition>) etc.

disjunction of the immediately adjacent conditions

Special cases

an integer comparison against a list value uses the length of the list a string comparison against a frame value uses the name of a frame a string comparison against a list value recurses on the first element of the list

Note: these special cases are not supported in extended syntax mode.

Extended syntax modeAs of version 3.0, there is a variant syntax for Hub programs, which we call extended syntax mode , which makes available a wider range of expressions to the Hub program writer andeliminates a number of special cases in the default (or normal ) syntax mode. You can switch tothis mode using the global directive PGM_SYNTAX:. While many other compatible

274

enhancements are planned, the one significant enhancement available in extended syntax modein version 3.0 is the ability to refer to namespaces explicitly. These references have the following form:

$in(<key> <namespace>)

These references are currently available in the directives IN:, OUT:, DEL:, PARAM:, SET:, ERROR:, LOG_IN:, LOG_OUT:, CONDITIONS:, and in rule conditions. Here’s a variation on ourexample program file:


SERVICE_TYPE: parser OPERATIONS: parse_string INIT: :speed "medium" (:accuracy "high")

SERVICE_PROVIDER: parser LOCATION: localhost:14000 PROPERTIES: (:language "English")

PROGRAM: main

RULE: :input_string && :language == "English" --> parser.parse_string IN: (:string :input_string) :user $in(:language server)

In this example, we associate a property with the service provider, and import it into themessage on the IN: line. These namespace references can occur anywhere that keys can occur(in the directives in which they’re available). When present, the specified namespace overridesthe default source or target namespace. As a special case, when these namespace referencesoccur in optional value mode in the same position as bare keys, the specified namespaceoverrides the default source namespace and retains the default target. In this example here, itoverrides the default source namespace for IN:.

Consider another example:

RULE: :input_string && :language == "English" --> parser.parse_string IN: ($in(:lang session) $in(:language server))

This IN: line doesn’t add anything to the message; rather, it takes the value of the :languagekey in the server namespace and stores it as the value of the :lang key in the sessionnamespace, before the message is sent.

These namespace references are available in rule conditions as well:

275


SERVICE_TYPE: parser OPERATIONS: parse_string INIT: :speed "medium" (:accuracy "high") CONDITIONS: :language == $in(:language session)

The CONDITIONS: line here imposes a restriction on the selection of any service provider forthe given service type; namely, that the value of the :language key in the default namespace(which is the server namespace for CONDITIONS:) is equal to the value of the :languagekey in the session namespace. In other words, choose the parser that understands thelanguage the session is being conducted in.

As a side effect of implementing this construct, it is now possible in rule conditions to comparethe values of any two keys, which means that unquoted tokens on the right hand side ofcomparisons are not coerced into strings. Here’s an example:

RULE: :a == :b --> foo

In default syntax mode, this rule will be fired if the value of the key :a in the token is the string ":b" ; that is, the test is equivalent to :a == ":b" . In extended syntax mode, this rule will befired if the value of the key :a is the same as the value of the key :b .

Summary of differences from default syntax mode

Some of these differences will not make sense until you’ve read the complete scriptingdocumentation. They are listed here for completeness. Some of the differences refer topreviously undocumented exceptional behavior in the default syntax mode, and in many casesare previous attempts to support references to namespaces other than the default namespace.

The $in constructor is available in extended syntax mode, but not in default syntax mode. Unquoted strings on the right hand side of comparisons in conditions are interpreted asframe keys rather than string literals. The special case comparisons in conditions are not supported in extended syntax mode. The :hub_session_ prefix for keys is recognized as an exceptional reference to thesession namespace in IN: and OUT: in default syntax mode, but is not available in extendedsyntax mode. The special logfile information keys are retrieved from the session rather than the tokenwhen they appear in the IN: line in default syntax mode, but this exception is eliminated inextended syntax mode. In default syntax mode, a key in the source position in IN: which is not found in the token(the default source namespace) will also be looked for in the message, in case a RETRIEVE: directive added it. This exception is eliminated in extended syntax mode. In default syntax mode, a colon is prepended to every supposed frame key which does notbegin with a colon. This behavior is eliminated in extended syntax mode.

276

Note that because of the nature of Hub program execution, not all namespaces are available inall directives where the construct is recognized. For each directive where extended syntaxmakes a difference, the restrictions are noted.

Global DeclarationsThere are a number of global identifiers, which have different effects on the function of the Hub.These identifiers can be used to configure the initial token which is executed by the Hub, or toset global values for logging, or to set other global defaults.

global variable identifier description typevisible in global namespace?

DOMAIN:

Domain name, used by MIT topartition the space of queries intodifferent topic areas, and passedto all servers via the reinitialize message.Default value is Unspecified .

string yes

INITIAL_REPLY:

Used by the Hub to "seed" theinitial token with a string to bepresented to the user as awelcome message. It is called a"reply" because in the MIT scripts,the same rules which handlereplies are intended to handle thisstring. This is veryidiosyncratic, and should notbe used.

string yes

INITIAL_TOKEN:

A frame which can override someof the default values of the initialtoken, especially the name of theprogram invoked. See the sectionon the initial token.

frame no

277

MODE:

There are three legal values for MODE:.

The first, singlethread , allowsthe user to specify that only thefirst rule which matches shouldfire. See the lesson on how theHub processes incoming messages. As of version 2.1, thisentry is superfluous, because it isset by default and cannot be disabled.

The second, pedantic , forcesthe Hub to exit if it encounters amissing server or operation(default behavior is to warn andcontinue). But see the IGNORE: directive.

The third, global_read_only ,locks the global namespace for writing.

string no

USER_ID:

Distinguished string whichidentifies the "user". This issomething of a misnomer, sincethe Hub can be used by manydifferent people at the same time.It is passed both during reinitialize and in the initialtoken. It can be used to identifythe connecting site to the server.

string yes

PGM_SYNTAX:

The single legal value extendedspecifies that all subsequentmaterial in the program file shouldbe interpreted using the rules forextended syntax (new in 3.0)

string no

Here’s an example global declaration block:


USER_ID: MITRE INITIAL_TOKEN: {c initialize } MODE: pedantic

278

Extended syntax noteThe global namespace keys are the directive names themselves. Here’s the global namespacewhich results from the sample global declaration block:

{c global DOMAIN: "Unspecified" USER_ID: "MITRE"}

Here’s a sample rule:

RULE: $in(DOMAIN: global) --> do_it IN: (:domain $in(DOMAIN: global)) ($in(LOCATION: global) "US")$in(COUNTRY: global)

This rule will fire if the global namespace contains an entry for the DOMAIN: key. It will pass inthat value as the :domain key to the do_it operation. It will attempt to set the global LOCATION: to "US" , which will have no effect because the global namespace is not writable,and it will attempt to pass in the value of the COUNTRY: key in the global namespace, which willhave no effect because the global namespace does not have an entry for that key.

In version 3.0, the global namespace could not be written to, so specifying it as a targetnamespace had no effect. In 3.1, the global namespace is writable, and it can be write-lockedusing the global_read_only value of MODE:.

Initial TokenThe initial token is constructed as follows:

1. The frame is first seeded with global default information, as follows:

frame key source type default value

<frame name> (specified in code) main

:domain value of DOMAIN: program file identifier string Unspecified

:initialize integer 1

:reply_stringvalue of INITIAL_REPLY: program fileidentifier, if present

string <none>

:user_idvalue of USER_ID: program file identifier, if present

string sls

2. The frame is augmented (or overwritten) from the specification of the INITIAL_TOKEN: entry, if it exists. At this point, the initial token is cached to feed the construction of the reinitialize message.

279

3. The frame is augmented (or overwritten) with the contents of the -init command lineargument to the Hub executable, if present.

On startup, the Hub processes this token as soon as it’s finished its initial round of contactingservers (some of which might be unsuccessful). See the discussion on how the Hub works.

Logging ParametersLogging parameters can also be defined globally. These instruct what to log and when. Logfileinstructions are useful for spot checking the system, and the resulting logs display attributesimportant for debugging. For details, see the logging documentation.

Timestamps are recorded whenever the listed operations are called. Keys are logged when theinput message is sent, or a return is received. This behavior supersedes previous behavior,where keys were logged only when increment_utterance was called.

logging identifier description type examplevisible inglobal namespace?

LOG_IN:

Keys to berecordedwhen a ruleis fired or amessage issent to the Hub

(see the rule entry)

See the logging documentation.

(not global)

LOG_OUT:

Keys to berecordedwhen a rulereturn isreceived ora reply issent fromthe Hub

(see the rule entry)


(not global)

LOG_DIR:Logfile directory

string LOG_DIR: /usr/logs yes

280

LOG_HUB_ACTIVITY:

Control oflogging ofHub eventslike alarmexpiration,servers notfound, etc.

sequenceof strings

LOG_HUB_ACTIVITY: system_errors


no

LOG_VERSION:

Userversionstring todifferentiatelogging strategies

string

LOG_VERSION: "traveldemo, version 2.3"


yes

TIMESTAMP:

Operationsormessagesto log withtimestamps (dot notation is legal)

sequenceof strings

TIMESTAMP: create_framenl.paraphrase_requestjupiter.turn_management paraphrase_reply

no

MESSAGE:

A messagedeclarationto hostmessageLOG_IN:andLOG_OUT: keys

stringSee the logging documentation.

(not global)

(Note that any newlines in this table are a side effect of presentation; newlines are permitted inentries in program files only if they are escaped with the backslash character.)

Special logging session keysIf logging is enabled, each session namespace contains four special keys:

281

key value

:hub_logdir the value of the LOG_DIR: directive

:hub_log_prefix the prefix for the logfile name

:hub_pwd the directory the Hub is running in

:hub_logfile the name of the logfile for the session

So on Unix, the logfile name is <:hub_pwd>/<:hub_logdir>/<:hub_log_prefix>-hublog.txt , or, alternatively, <:hub_logfile> . In extended syntax mode, you can pass these keys to the server as follows:

IN: $in(:hub_logdir session) $in(:hub_log_prefix session)$in(:hub_pwd session) $in(:hub_logfile session)

In default syntax mode, these keys are exceptionally available in the IN: line.

Server DeclarationsThe Hub keeps track of a set of service types, each of which is associated with a set of availableoperations, and a set of service providers which implement these operations. These serviceproviders correspond to Communicator-compliant servers.

Service typesThe service type block contains the following directives:

282

system identifier

description typeextended syntax notes

SERVICE_TYPE:unique name of the service being provided

string

OPERATIONS:list of operations supported by allproviders of this service

sequenceof strings

INIT:initializes server-specific keys tospecified values in the reinitialize() message.

obligatory value format

CLIENT_PORT:a port for the Hub to set up a listeneron (alternates with PORT:)

integer

PORT: synonym of CLIENT_PORT: integer

PROPERTIES:

initializes a set of properties which theHub associates with each serviceprovider for this service type (can beoverridden by a PROPERTIES:declaration on the service provider, orby the server itself)


CONDITIONS:

specifies a set of conditions associatedwith when to select a given serviceprovider; relatively useless unlessextended syntax is enabled

rule condition

Default namespace is server. Allnamespaces available.

IN:

specifies a set of directives to beappended to the IN: line for any rulefired for any service provider for thisservice type (see the main entry for IN:)

optional value format

Default sourcenamespace is server , default targetnamespace is token.All namespaces available.

Here’s an example service type block:

SERVICE_TYPE: parser OPERATIONS: create_frame context_tracking_in INIT: :default_language "Spanish" CLIENT_PORT: 11000 PROPERTIES: :dialect "standard" CONDITIONS: :language == $in(:language session) IN: :dialect

Note : this block may not contain any blank lines.

283

Service providersThe program file can also specify a list of service providers for the Hub to contact at startup time(service providers can also contact the Hub using the listener-in-Hub facilities, but these are notlisted in the Hub program). The Hub will attempt to maintain a connection with these servers; if itcannot find them, or they disconnect after startup, the Hub will periodically attempt to reconnect.

system identifier description typeextended syntax notes

SERVICE_PROVIDER:

a sequence of tokens corresponding todeclared service types. The serviceprovider must implement the union ofoperations specified for those service types.

sequenceof strings

HOST:

service provider host name or IPaddress (see also LOCATION:). Notethat the program file parser will notinterpret IP addresses as strings unlessthey’re explicitly delimited.

string

INIT:

initializes server-specific keys tospecified values in the reinitialize() message. Individualkey-value pairs override any INIT:declarations on the parent service types


LOCATION:

server host name/IP address and port,separated by a colon; used as analternative to HOST: and PORT:. Notethat the program file parser will notinterpret IP addresses as strings unlessthey’re explicitly delimited.

string

PORT: server port (see also LOCATION:) integer

PROPERTIES:

initializes a set of properties which theHub associates this service provider(individual key-value pairs override any PROPERTIES: declarations on theparent service type, and can also beupdated by the server itself)


284

CONDITIONS:

specifies a set of conditions associatedwith when to select this serviceprovider; relatively useless unlessextended syntax is enabled. If specifiedon a service provider, all CONDITIONS: on service type parentsare ignored.

rule condition

Defaultnamespace is server. Allnamespaces available.

IN:

specifies a set of directives to beappended to the IN: line for any rulefired for any service provider for thisservice type (see the main entry for IN:). Individual key-value pairs overrideany IN: declarations on the parentservice types.

optional value format

Default sourcenamespace is server , defaulttargetnamespace is token. Allnamespaces available.

Here’s a sample service provider block. Note that the content of INIT:, PROPERTIES:,CONDITIONS: and IN: override the service type values, if present. Also note that theCONDITIONS: line requires extended syntax to be enabled.

SERVICE_PROVIDER: parser LOCATION: localhost:11000 INIT: :default_language "Spanish" PROPERTIES: :dialect "standard" CONDITIONS: :language == $in(:language session) IN: :dialect

Note : this block may not contain any blank lines.

Other server directivesThere are three other directives which are provided.

285

system identifier

description type

SERVER:

Introduces a block which defines a service type and its providerssimultaneously. All service type and service provider directives areaccepted; multiple LOCATION: directives are interpreted as multipleservice providers. INIT:, CONDITIONS:, IN:, and PROPERTIES: areassociated with the service type rather than any of the providers.

string

SERVERS:

Provides a list of service types and providers which are active in thissession. If present, the subset of SERVICE_TYPE: andSERVICE_PROVIDER: declarations present in this directive will beused. This can be used to share a single file of type and providerdeclarations among multiple systems and program files, not all ofwhich are used in any given system configuration. This declarationmust precede all type and provider declarations. Types are specifiedby name; providers are specified using legal values for the LOCATION: directive.

sequenceof strings

IGNORE:

Provides a list of operations which the Hub can disregard whenreading in a program file. Normally, if the Hub encounters a rule withan operation which is not defined, it will issue a warning (or exit in pedantic mode). However, if the operation is in the IGNORE: list, theHub will not complain. This feature is useful when including programfiles in many other program files, where the programmer is certain thatsome of the programs or rules in the included file will never be encountered.

sequenceof strings

Examples:

SERVERS: parser localhost:17500

SERVER: parser LOCATION: localhost:17600 OPERATIONS: parse_string INIT: :default_language "Spanish"

IGNORE: parser.create_frame context_in

Note : the SERVER: block may not contain any blank lines.

The Builtin serverThe Hub provides a special server called Builtin which is always available (that is, it does notneed to be declared). This server provides support for null operations, session management, etc.

286

Hub ProgramsA Hub program is basically a set of rules that describes a sequence of operations. Hub programsprovide an optional level of scripting which developers may exploit to exert additional controlover the flow of control of messages. Typically, there is a main program (conventionally named"main") which controls the toplevel flow of control; other uses of subprograms might be to controlserver-initiated communication for executing subtasks. For example, a program called db_querymight control the following sequence of operations: turn_management --> database --> turn_management.

All programs (main and subprograms) consist of:

Program Name (PROGRAM:) Rules (RULE:)

Constructing a Rule Examples of Rules

Parameters Input / Output / Merge Parameters ( IN: / OUT: / SET: / PARAM:/ DEL: ) Database Parameters (RETRIEVE: / STORE:) Session keys Alarms

There is an order associated with when operations and identifiers are evaluated.

A program begins when it is named and ends when the next program is named. All rules andassociated keys in between are part of the program.

The following table lists the valid identifiers used in defining programs. Identifiers which arespecific to rules are listed later.

program identifier

description extended syntax notes

ALARM:enabling/disabling time outs.See the section on alarms.

PROGRAM:name of Hub program (main or subprogram)

RULE:the specified conditions underwhich the rule will be executed

Default namespace is token . Allnamespaces except server available(provider hans’t been selected yet).

287

Program NameA Hub program name is indicated by the PROGRAM:identifier followed by the program name:

PROGRAM: main

The program may have any name.

As of Galaxy Communicator 2.0, the program may also have LOG_IN: and LOG_OUT:designations, which cause keys and values to be written to the Hub log when the program isinitially invoked and when it returns a server-to-server subdialogue reply to the server whichinvoked the program via GalSS_EnvDispatchFrame.

RulesA Hub program consists of a sequence of rules. A rule specifies an operation to be executed bya server if and when appropriate conditions are met. The scripting language supports a subsetof boolean operators for composing conditions in a logical expression. A rule is similar to anif-then statement.

Here is an example of a simple rule:

RULE: !:thin_frame & !:parse_frame & :input_string --> create_frame

This rule states that if the :thin_frame key and the :parse_frame key do not exist but :input_string does, then call create_frame .

Rules are fired in the order in which they appear in the program file. Typically, the first rule whichmatches is the one which is fired, except in the case of certain special control directives. Notethat the order in which tokens are returned from multiple rules is not defined.

It is possible to direct the rule to a particular server if multiple servers support the sameoperation, by prepending the server name to the operation name, separated by a period:

RULE: :request_frame & :key_value & :domain Jupiter --> jupiter.turn_management

This rule directs the token to the server named jupiter, in case multiple servers support the turm_management operation.

Constructing a Rule

The RULE: identifier is used to indicate a rule. It uses the rule condition expressions:

<rule_expr> = RULE: (<conditions>) --> <operation>

288

Examples of Valid Rules

RULE: :history_frame & :reply_frame --> context_tracking_out

If there is a :history_frame and a :reply_frame then execute context_tracking_out .

RULE: !:history_frame & !:reply & :reply_frame --> paraphrase_reply

If there is not a :history_frame and not a :reply but there is a :reply_frame thencall paraphrase_reply .

ParametersOrder of application Input Parameters (IN: ) Output Parameters ( OUT:/ ERROR:) Merge Parameters ( SET: / PARAM:/ DEL: ) Database Parameters (RETRIEVE: / STORE:) Other rule parameters

Order of OperationThere is an order associated with when operations and identifiers are evaluated. All theseidentifiers are evaluated after the condition matching is checked. The overall order is:

key value processing value format comments

(conditions match)

Default namespaceis token . Allnamespaces but server available(service providernot chosen yet).

Extended syntax available.

RETRIEVE:From Hub DB tomessage frame

list of tokens (but seenote on database parameters)

Extended syntax not available.

PARAM:Target namespaceis message .

obligatory valueExtended syntax available.

289

IN:

Default sourcenamespace is token , defaulttarget namespaceis message . Seealso the server directives.

optional valueExtended syntax available.

LOG_IN:

Default sourcenamespace is message . Writes to log.

optional valueExtended syntaxavailable for source namespace.

CONTINUE_REPLY: (n/a) frame

For discussion, see thesection on Hub continuations. Can be repeated.

CONTINUE_ERROR: (n/a) frame

For discussion, see thesection on Hub continuations. Can be repeated.

(dispatch function called)

LOG_OUT:

Default sourcenamespace is message(message return).Writes to log.

optional valueExtended syntaxavailable for source namespace.

STORE:From messagereturn to Hub DB

list of tokens

Note that STORE: doesnot work with none!,since there’s nomessage return, but itdoes work with destroy!.Extended syntax not available.

DEL: From token list of tokens

Note that DEL: workswith both none! and destroy!. Extendedsyntax available.

290

OUT:

Default sourcenamespace is message(message return),default targetnamespace is token .

optional valueExtended syntax available.

ERROR:

Default sourcenamespace is message(message return),default targetnamespace is token .

optional value

Evaluated if the dispatchfunction call returns anerror reply, in which caseOUT: is not evaluated.Extended syntaxavailable. See thediscussion of output parameters.

SET: From rule to token obligatory value

Note that SET: workswith both none! and destroy!. Extendedsyntax available.

CONTROL: (n/a)

zero or more of thesymbols :return , :asynchronous ,and :no_result

For discussion, see the special control directives

The INIT: directive in the server specification also uses the obligatory value value format.

Note : the RULE: block may not contain any blank lines.

Input ParametersWhen a given operation is executed, input parameters are passed in the token to the server. The IN: identifier is used to specify the list of input parameters. Any keys not included in theparameter lists are probably ignored by the server. IN: uses the optional value value format; thekey-mapping subcase of this format will map keys set by RETRIEVE: and PARAM:

For example,

IN: :foo (:bar 5) :baz (:remote_key :local_key) would mean that the frame sent to the server contains:

the key :foo , containing the token’s value of :foo the key :bar , containing 5 the key :baz , containing the token’s value of :baz the key :remote_key , containing the token’s value of :local_key

291

Note that explicit namespace references provided in extended syntax mode are available for IN:.

Output ParametersExpected output parameters are identified with OUT: or ERROR:. Output parameters are usedto update the token state. Output parameters use the optional value value format.

For example,

OUT: :foo (:bar 5) :baz (:local_key :remote_key) would mean that after the update, the token contains:

the key :foo , containing the reply frame’s value of :foo the key :bar , containing 5 the key :baz , containing the reply frame’s value of :baz the key :local_key , containing the reply frame’s value of :remote_key

The server is marked as busy when there are expected output parameters, and the Hub willsend no further messages to this server until it receives a reply.

If the reply from the dispatch function is an error, the default behavior is for the Hub to terminateexecution of the current token (and return the token state to the originating server, in the case ofa server-to-server subdialogue). In this case, all the appropriate updates, including OUT:, areperformed before the token execution is terminated. If the ERROR: directive is present and thereply is an error, then the ERROR: directive is evaluated instead of the OUT: directive and tokenexecution continues (that is, it’s as if the reply was a normal reply instead of an error). However,if a reply is not expected, then ERROR: has no effect.

Note that explicit namespace references provided in extended syntax mode are available foroutput parameters.

Special control directivesThe CONTROL: directive allows the user to specify how the program is to proceed after the ruleis fired.

A value of :return means that the script should stop running the program when this rule is complete. A value of :asynchronous means that the script should not wait for the operation tocomplete before starting the next applicable rule (SET: and DEL: will be run immediately,before additional rules are tried). When you use the builtin server dispatch functions dispatch_to_main or dispatch_token with CONTROL: :asynchronous, a new token iscreated for the dispatch, but rules continue to be tried for the current token. A value of :no_result tells the hub that the server will not return a value from theoperation. If you use :no_result , you should also use :asynchronous .

292

Three special values of the OUT: directive, none! , abort! and destroy! , are shorthands forsets of CONTROL: values.

OUT: none! is equivalent to CONTROL: :no_result :asynchronous OUT: abort! is equivalent to CONTROL: :return OUT: destroy! is equivalent to CONTROL: :no_result :asynchronous :return

Hub continuationsBy specifying values for CONTINUE_REPLY: and CONTINUE_ERROR: in a rule, theprogrammer can force the Hub to treat new messages from the selected server as reply or errormessages. These directives should be used when the programmer wants the Hub to use asynchronous programming model, but the server in question doesn’t provide replies. The serveris notified that a real reply isn’t expected (in fact, the real reply will be discarded if the serverprovides it), and the Hub won’t process the message return until a new message of one of thespecified forms is received from the server. The matching is done using Gal_MatchFrame. Thenew message is treated as a normal reply or error, and all the return parameters (OUT:,STORE:, etc.) are processed normally. See also the Builtin.hub_continue function and thediscussion of continuations.

Merge ParametersNote that explicit namespace references provided in extended syntax mode are available formerge parameters.

Set

The SET: identifier allows token keys to be set to specific values upon return from the server. SET: uses the obligatory value value format.

For example,

SET: :enable_input_ok 1 :reply_string "Thanks_for_using_Jupiter":speaker "paul" sets :enable_input_ok to int 1 and :reply_string to the string"Thanks_for_using_Jupiter" and :speaker to the string "paul".

Note that as a special case, none! works with SET:, even though there’s no message return.

Param

The PARAM: identifier allows token keys to be set to specific values before being sent to theserver. PARAM: uses the obligatory value value format.

For example,

293

PARAM: :reply_string "The phone has disconnected. Terminating session." sets :reply_string to "The phone has disconnected. Terminating session"

Del

The DEL: identifier indicates which keys to delete from the token state upon return from the operation.

Note that as a special case, none! works with DEL:, even though there’s no message return.

Database Parameters

STORE: is used to store the named keys in the Hub-internal database. RETRIEVE: is used to retrieve the named keys from the Hub-internal database.

For example,

RETRIEVE: :history_frame IN: :selected_thin_frame OUT: :request_frame :history_frame :domain STORE: :history_frame

The RETRIEVE: operation is performed before the argument list is composed for serverdispatch, and the STORE: operation is performed after the rule returns, but before the token isupdated with the server response. See the overall order. New in version 1.2: STORE: willsearch "unmapped" versions of key-mapped OUT: entries if it can’t find the value in the mapped values.

RETRIEVE: and STORE: are related to the underlying utterance ID of the Hub (see the increment_utterance message in the builtin server). STORE: stores an element in the slice of theHub DB associated with the current utterance ID. RETRIEVE:, by default, retrieves an elementfrom the slice of the Hub DB associated with the previous utterance ID. The RETRIEVE: entryallows a optional value version in which the second element is the relative index into the HubDB. For example, (:a 0) will retrieve the value of :a from the DB associated with the currentutterance ID, while (:a -2) will retrieve the value of :a from the DB associated with the currentutterance ID - 2. Default is -1.

Note that STORE: works with destroy! but not with none!, since there’s no message return in thelatter case. However, both work with DEL: and SET:, since although there’s no message return,neither DEL: nor SET: depend on the return message.

Other rule parameters

294

rule identifier

description example / syntax

ALARM:enabling/disabling time outs. Used both globallyand in rules. See the section on alarms.

ALARM: alarm_name enable| disable | <num_secs> ALARM: global_alarm 20 ALARM: global_alarm disable ALARM: global_alarmenable global_alarm 20turn_alarm enableturn_alarm 60

LOCK:

one of two possible keys(:hub_get_session_lock,:hub_serve_this_session_only) which controlsession locking

(not documented yet)

Session keysThe session maintains its own session state. By default, the session is populated with log information. It also initializes the :utterance_id key, which is incremented by the builtinserver dispatch function increment_utterance. However, it’s convenient to store additionalinformation on the session, which is specific to the current interaction but persists acrossmessages. It’s possible to control these values from the server itself, and in extended syntax mode, it’s easy to set and retrieve such values using namespace references. In default syntaxmode, it’s a little trickier.

In default syntax mode, in SET:, IN: and OUT:, keys which start with :hub_session_ arestored in or retrieved from the session rather than the token. The hub_session_ portion of thekey is stripped off. Here’s an example of this capability in action, first a program file fragment andthen the resulting status:

LOG_DIR: .

USER_ID: MITRE

PROGRAM: main

[...]

RULE: --> Builtin.nop SET: :hub_session_current_status "adequate"

-------------------------------------

295

{c session :utterance_id -1 :hub_log_prefix "MITRE-20001206-007" :hub_logdir "./MITRE/20001206/007/" :hub_pwd "/usr/local/GalaxyCommunicator/src" :current_status "adequate" }

As a special case, a Hub program may use the special values hub_increment_value and hub_decrement_value as values of these session keys in a SET: directive to increment anddecrement integer values.

AlarmsAlarms are session-specific variables used for setting time outs in a production system. Alarmscan have three states:

disabled enabled with time set enabled without time set

If an alarm is enabled with the time set, a system error will be generated when the time reacheszero. In the normal course of execution, the time should be reset as each rule fires. If a serverdies, a system error will be generated. The error can be used to end the dialogue by informingthe user that the system is down.

The syntax is:

ALARM: alarm_name enable | disable | <num_secs>

The same line may contain enable/disable and <num_secs> for the same alarm. Two examples are:

ALARM: global_alarm 20 ALARM: global_alarm enable global_alarm 20 turn_alarm enableturn_alarm 60

Processing occurs from left to right with the enable/disables processed first, then the<num_secs>. The first example sets the already-enabled global_alarm to 20 seconds. Thesecond example enables both the global_alarm and the turn_alarm and then sets the first to 20seconds, the second to 60 seconds.

296

Alarm Enable

Upon intiation, all alarms are enabled but they are not set unless the <num_secs> is specified. Once an alarm is enabled, any future change of <num_secs> to a non-zero number is valid. Setting an alarm with zero <num_secs> is equivalent to the alarm not being set. For example,

ALARM: global_alarm 20 sets the already-enabled global_alarm to 20 seconds.

ALARM: global_alarm 40 sets the same alarm to 40 seconds.

ALARM: global_alarm 0 is equivalent to: ALARM: global_alarm disable

Alarms can be set globally so that they do not have to be set in each rule. To set an alarmglobally, set the alarm before any rules are defined. Any particular server can override theglobal setting with a specific, local alarm.

Alarm Disable

Disabling an alarm is a higher-level override that sets <num_secs> to zero. The alarm will notchange until the next enable. Disabling an alarm is useful because there is no easy way to testfor an alarm having a time set before resetting it. For example, one might want to turn off allalarms while waiting for user input. Disabling an alarm is also useful when trying to debug code.

Alphabetical index of program file elements; (comment) # (comment) #include ALARM: CLIENT_PORT: CONDITIONS: CONTINUE_REPLY: CONTINUE_ERROR: CONTROL: DEL: DOMAIN: ERROR: HOST: IGNORE: IN: INIT: INITIAL_REPLY:

297

INITIAL_TOKEN: LOCATION: LOCK: LOG_DIR: LOG_HUB_ACTIVITY: LOG_IN: (see also the logging documentation) LOG_OUT: (see also the logging documentation) LOG_VERSION: MESSAGE: MODE: OPERATIONS: OUT: PARAM: PGM_SYNTAX: PORT: PROPERTIES: PROGRAM: RETRIEVE: RULE: SERVER: SERVERS: SERVICE_PROVIDER: SERVICE_TYPE: SET: STORE: TIMESTAMP: USER_ID:


298

Logging

What Can Be Logged and Why Adding Logfile Instructions to the Program File

Global Parameters LOG_DIR: LOG_VERSION: LOG_HUB_ACTIVITY: TIMESTAMP: USER_ID:

Parameters within the Scope of a Rule LOG_IN: LOG_OUT:

Parameters within the Scope of a Program Parameters within the Scope of a Message

Audio Logging When and How Data Get Logged

Key Values and Frame Contents Timestamps Audio

Where Logfiles and Data are Stored

What Can Be Logged and WhyExecution of server operations, contents of key values, and spoken utterances can be logged forpurposes of debugging or playback. Logfile instructions are useful for spot checking the system,and the resulting logs display attributes important for debugging.

As of version 2.0, it is now possible to log incoming Hub messages and keys in both scripted andscriptless modes, as well as Hub activity such as alarm expirations, etc. Also, it is now possibleto version your logs.

299

Adding Logfile Instructions to the Program FileThere is no simple way to "turn on" logging, but one can tailor a program file to log specific data.

Global Parameters

Some logging parameters are defined globally in the program file. These instruct what to logand when. The following table contains the logfile instruction parameters that need to be definedglobally in the program file. Note that any line breaks in these entries are the result of browserformatting; no linebreaks are permitted in these entries.

logging variable description what can be logged example

LOG_DIR:

top level logging directory path can be relativeor absolute logging will notoccur if not declared(no default value) if this directory doesnot exist, it is created

n/aLOG_DIR: /usr/users/communicator/logs

LOG_HUB_ACTIVITY:list of Hub events to log

legal values are serve_any_session serve_this_session_only get_session_lockrelease_session_lockalarm_activity system_errors

LOG_HUB_ACTIVITY:serve_any_session system_errors

LOG_VERSION:user string to tracklog versions

an arbitrary string which canbe used to differentiatedifferent strategies for logginga system (if, for instance, youchange the names of yourkeys or operations and yourautomatic annotation toolsneed to be able to tell whichstrategy is being used)

LOG_VERSION: "travel demo, version 2.3"

300

TIMESTAMP:

server operations orprograms/messagesto log with timestamps timestamps arerecorded wheneverthe specifiedoperations areexecuted orprograms areinvoked (see below) cannot use dotnotation withoperation names only one TIMESTAMP: lineper program file(subsequentdeclarations override)

server operations

TIMESTAMP: create_framecontext_tracking_inparaphrase_request turn_management paraphrase_reply

USER_ID:

user identification name of loggingsubdirectory (under <LOG_DIR:>) used as prefix to the logfile if not specified,default value is ’sls’

n/a USER_ID: travel_audio

Parameters within the Scope of a Rule

Other logging parameters can be defined within the scope of a given rule. The variables belowallow specific key values to be logged in association with the execution of a particular operation.

301

logging variable

descriptionwhat can be logged

example

LOG_IN:

associated with a particularprogram file rule or program/message keys are specified withinthe scope of a rule values are stored before therule is fired keys are logged frommessage, not from token

values ofkeys, frame contents

RULE: !:thin_frame &!:parse_frame &:input_string --> nl.create_frame IN: :input_string :domain LOG_IN: :input_string OUT: :thin_frame :parse_status

LOG_OUT:

associated with a particularprogram file rule or program/message keys are specified withinthe scope of a rule values are stored upomreturn from the server keys are logged frommessage, not from token

values ofkeys, frame contents

RULE: !:thin_frame &!:parse_frame &:input_string --> nl.create_frame IN: :input_string :domain OUT: :thin_frame :parse_status LOG_OUT: :thin_frame :parse_status

It is also possible to use specify mappings between local and remote keys, as you can with IN:and OUT:

LOG_IN: (:parse_frame :input_frame)

Both normal and error replies will be logged if you have LOG_OUT: keys specified. All errorreplies are logged.

Parameters within the Scope of a Program

As of version 2.0, it is possible to log the invocation of programs (or incoming messages andtheir replies in scriptless mode). You can timestamp them by listing them on the TIMESTAMP:list. For example:

SERVER: parser OPERATIONS: parse_nl

TIMESTAMP: main parse_nl

PROGRAM: main

RULE: :string --> parser.parse_nl

302

If you want to log the incoming and outgoing keys associated with a program, simply useLOG_IN:/LOG_OUT: at the program level:



PROGRAM: main LOG_IN: :string

RULE: :string --> parser.parse_nl

This will cause the value of the :string key to be logged when the program is first invoked. If youwant to log the keys for a program return to a server which has called GalSS_EnvDispatchFrame, just use LOG_OUT:.

Parameters within the Scope of a Message

In scriptless mode, the timestamp will be recorded without any declaration at all beyond the oneon the TIMESTAMP: list.

If you want to log incoming and outgoing keys in scriptless mode, use LOG_IN: and/orLOG_OUT: under the MESSAGE: declaration:

logging variable

descriptionwhat can be logged

example

MESSAGE:the name of a message which theHub may receive

n/aMESSAGE: parse_string

This declaration might be used as follows:



MESSAGE: parse_nl LOG_IN: :string

When the Hub receives the parse_nl message, it will log the :string key value before it passesthe message on to the appropriate server.

303

Audio LoggingThe MIT audio server implements a fairly complex way of logging audio. We no longer documentthis method, because the MIT audio server is no longer being distributed with GalaxyCommunicator. If you’re writing an audio server, this server needs to do some basic things:

It should capture the audio and write it to a file, allowing the filename path and prefix to beset by calling dispatch functions It should report the filename back to the Hub

The fact that audio is typically shipped around by brokering can cause some difficulty, becausethe Hub can’t log the audio (since it doesn’t pass through the Hub), and if the Hub and audioservers can’t access the same file system, the audio files will have to be moved to theappropriate location by some outside process.

When and How Data Get LoggedKey Values and Frame Contents Timestamps Audio

Key Values and Frame Contents

Parameters (key values and frame contents) are logged either before or after the associated ruleis fired. LOG_IN: parameters are stored before the operation is executed. LOG_OUT:parameters are logged upon return from the server.

Timestamps

Timestamps are recorded whenever the listed operations are called or programs/messages are invoked.

Audio

Audio should be logged by the audio server when the audio server is done processing it.

Where Logfiles and Data are StoredText logfiles are stored in the logfile directory as <LOG_DIR:>/<USER_ID:>/ date/ counter/<USER_ID:>- date- counter- hublog.txt , where:

304

USER_ID:user identification as specified in program file default value is ’sls’

date date of log in yyyymmdd format

counter 3-digit session identifcation in ddd format

hublog.text default suffix

The location of audio log files depends on the configuration of your audio server. The value ofLOG_DIR: and the remainder of the logfile prefix are available in the Hub scripting language topass to your servers if necessary.


305

Brokering and Audio Data

In Communicator, there are two ways of moving data from server to server. First, you canpackage the data into a frame and send it to another server via the Hub, using the standardframe dispatch functions. Alternatively, you can use the Hub to establish a direct connectionbetween servers (called a broker connection), through which you can send data. This lattertechnique is used in the MIT architecture for transmitting audio data from the device server to therecognizer, and from the synthesizer to the device server. In this document, we describe andexemplify this brokering tool.

In some cases, it’s not possible to use the MIT timed task loop. For information on how to handlebrokering in those cases, look here.

Sending broker dataWe’ve constructed an example which consists of two sides of a broker connection whichtransmits the contents of a file containing raw 8-bit audio data and and outputs the data to theaudio device. Here’s a code fragment which illustrates how the sender uses the broker connection:

static Gal_Frame prepare_audio_frame(GalIO_CommStruct *gcomm, char *filename) { Gal_Frame f = Gal_MakeFrame("main", GAL_CLAUSE); int total; char *buf; GalIO_BrokerStruct *b;

/* In the code omitted here, the data from the named file is read into buf, and total is the number of bytes in buf */

/* .... */

/* Now that we have the audio, we write the binary data through the broker. */

306

b = GalIO_BrokerDataOutInit(gcomm, 0, 10); if (b && (GalIO_GetBrokerListenPort(b) > 0)) { GalIO_BrokerPopulateFrame(b, f, ":binary_host", ":binary_port"); GalIO_BrokerWriteBinary(b, buf, total); GalIO_BrokerDataOutDone(b); } free(buf); return f; }

The important steps here are to set up the outgoing broker stream, constructing a well-formedframe for initiating a broker connection, sending the frame to the Hub (in this case, by a call toGalSS_EnvWriteFrame, not shown), writing the appropriate data to the connection, andannouncing that no more data will be sent.

Step 1: setting up the outgoing broker stream

GalIO_BrokerStruct *GalIO_BrokerDataOutInit (GalIO_CommStruct *gcomm, int poll_ms,int timeout_seconds) The outgoing broker uses the server’s listener as its listener port. If the server is a Hub client anddoesn’t have an open listener, the broker sets up a listener for the server to host. The outgoingbroker accesses the server via gcomm. The poll_ms is the time in milliseconds that the timedtask loop is supposed to check the broker connection; as with GalIO_ServerInit, 0 is the default(100 ms), a positive number is an actual number of milliseconds, and < 0 means not to set up atimed task. The timeout_seconds is how long before the broker expires; -1 means never expire,0 means use the default (which is 10 seconds). You can write to an expired broker, but it willaccept no more connections; when all the data is written to the existing connections, the brokerwill be destroyed.

It’s important to set the timeout intelligently, because outgoing brokers can now accept multipleconnections. This means that they must cache the data which is written through them forsubsequent subscribers. If the broker never times out, significant memory bloat might result.

Step 2: constructing a well-formed frameThe server needs to announce the availability of broker data to the Hub, and it does so bypopulating a frame with the appropriate information. A well-formed frame for initiating a brokerconnection contains the following keys:

The key designated by GAL_BROKER_CALL_ID_FRAME_KEY, which sets a unique ID whichallows the server to distinguish between connections some key which stores the host name, which the connection receiver must expect some key which stores the port number, which the connection receiver must expect

307

As of version 3.0, it is no longer the programmer’s responsibility to construct a unique call ID forthe brokers. In addition to being more convenient, much existing brokering code doesn’t take intoaccount the possibility that servers may have multiple outgoing brokers simultaneously available.

void GalIO_BrokerPopulateFrame (GalIO_BrokerStruct *b, Gal_Frame f, char *host_key,char *port_key) Populates the frame f with the appropriate broker information, including the call ID, host and port.The host_key is the key to store the hostname in the frame, and the port_key is the key to storethe port in. The call ID is stored in the key designated by GAL_BROKER_CALL_ID_FRAME_KEY.

unsigned short GalIO_GetBrokerListenPort (GalIO_BrokerStruct *b) Returns the port number the broker b is listening on. If this value is not 0, the broker has beenset up correctly.

char *GalIO_GetBrokerCallID (GalIO_BrokerStruct *b) Returns the call ID the broker b is using. You should not typically need to use this function.

void GalIO_FrameSetBrokerCallID (Gal_Frame f, char *call_id) Sets the unique ID for broker confirmation. You should not typically need to use this function.

char *GalIO_IPAddress (void) This function returns the IP address of the host, for use in creating contact information for thebroker connection. You should not typically need to use this function.

Step 3: writing data to the brokerSince brokers now accept multiple connections, all data which is written to an outgoing broker iscached until the broker expires. Each new client is guaranteed of getting all the data written tothe broker, in the order in which it was written.

int GalIO_BrokerWriteFrame (GalIO_BrokerStruct *b, Gal_Frame frame) int GalIO_BrokerWriteInt (GalIO_BrokerStruct *b, int i) int GalIO_BrokerWriteFloat (GalIO_BrokerStruct *b, float f) int GalIO_BrokerWriteList (GalIO_BrokerStruct *b, Gal_Object *elts, int n_elts) int GalIO_BrokerWriteString (GalIO_BrokerStruct *b, char *str) int GalIO_BrokerWriteBinary (GalIO_BrokerStruct *b, void *data, int n_bytes) int GalIO_BrokerWriteInt16 (GalIO_BrokerStruct *b, void *data, int n_ints) int GalIO_BrokerWriteInt32 (GalIO_BrokerStruct *b, void *data, int n_ints) int GalIO_BrokerWriteInt64 (GalIO_BrokerStruct *b, void *data, int n_ints) int GalIO_BrokerWriteFloat32 (GalIO_BrokerStruct *b, void *data, int n_floats) int GalIO_BrokerWriteFloat64 (GalIO_BrokerStruct *b, void *data, int n_floats) These functions send data through the broker connection. The encoding for these elements isidentical to their encoding within frames during message transport. These functions correspondin the obvious way to the Communicator object types.

308

int GalIO_BrokerWriteObject (GalIO_BrokerStruct *b, Gal_Object o) Write a Gal_Object of any type through the broker connection. The encoding for these elementsis identical to their encoding within frames during message transport.

Step 4: announcing the end of the data

void GalIO_BrokerDataOutDone (GalIO_BrokerStruct *b) This function should be called on the broker structure when all the data has been sent. If you donot call this function, the incoming broker on the other end of the broker connection will never terminate.

void GalIO_ForceBrokerExpiration (GalIO_BrokerStruct *b) This function causes the broker to expire immediately. You can use this in conjunction with GalIO_AddBrokerCallback to force a broker to expire after it’s accepted a certain number ofconnections, for instance.

int GalIO_BrokerIsDone (GalIO_BrokerStruct *b) Returns 1 if the broker is already marked as done, 0 otherwise.

Streaming broker dataThere is a minor problem with the example we’ve presented here: the broker setup bothestablishes the broker and writes the data. In an application without threads, this means that thereceiving broker won’t be able to connect to the sending broker until all the data is written to theoutgoing broker (because the server won’t have a chance to accept connections for the outgoingbroker until the broker setup returns).

For some applications, this may be acceptable, but it’s almost certainly the case that you’ll wantto "stream" audio data, so that the receiving broker can start making use of it as soon aspossible (say, to start speech recognition). In these cases, you need to exploit the timed task loop to send outbound data at periodic intervals (say, by polling the audio device and relayingthe data when it’s available), so that the server has a chance to accept connections.

Here’s a version of the broker setup which exemplifies this strategy. This is a bit unnatural, sinceall the data is immediately available in a file, but it illustrates the streaming:

static Gal_Frame prepare_audio_frame(GalIO_CommStruct *gcomm, char *filename) { Gal_Frame f = Gal_MakeFrame("main", GAL_CLAUSE); FILE *fp; GalIO_BrokerStruct *b; OutData *o = (OutData *) calloc(1, sizeof(OutData)); /* In the code omitted here, the file is opened */

309

/* .... */ /* If we’re the sending direction, we read binary data from the file, relaying it to the broker, until we find EOF. */ b = GalIO_BrokerDataOutInit(gcomm, 0, 10); if (b && (GalIO_GetBrokerListenPort(b) > 0)) { GalIO_BrokerPopulateFrame(b, f, ":binary_host", ":binary_port"); o->fp = fp; o->b = b; Gal_AddTask(__write_data, (void *) o, 10, 0, NULL); }

return f; }

Note that we use Gal_AddTask to set up a task to start writing the data in 10 milliseconds.Here’s the task:

static void __write_data(Gal_TaskPkg *p) { OutData *o = (OutData *) Gal_TaskPkgData(p); FILE *fp = o->fp; GalIO_BrokerStruct *b = o->b; char *buf = (char *) malloc(BLOCKSIZE * sizeof(char)); int count; count = fread(buf, sizeof(char), BLOCKSIZE, fp); if (count) { GalIO_BrokerWriteBinary(b, buf, count); } free(buf); if (count == BLOCKSIZE) { /* Not done yet. */ Gal_ReAddTask(p, (void *) o, 10, 0, NULL); } else { GalIO_BrokerDataOutDone(b); fclose(fp); free(o); } }

We keep resetting the task until we run out of data.

310

Receiving broker dataHere’s the other end of the connection from our example. The DataHandler structure is not partof the Communicator package; we use it for illustration:

Gal_Frame receive_audio(Gal_Frame f, void *server_data) { DataHandler *d = (DataHandler *) malloc(sizeof(DataHandler)); GalIO_BrokerStruct *b; char *host = Gal_GetString(f, ":binary_host"); int port = Gal_GetInt(f, ":binary_port");

d->data_buf = (char *) NULL; d->size = 0; if (host && port) { b = GalSS_EnvBrokerDataInInit((GalSS_Environment *) server_data, host, port, f, env_audio_handler, 0, d, __FreeDataHandler); if (b) { GalIO_SetBrokerActive(b); GalIO_AddBrokerCallback(b, GAL_BROKER_ABORT_EVENT, __report_abort, (void *) NULL); GalIO_AddBrokerCallback(b, GAL_BROKER_DATA_DONE_EVENT, __report_done, (void *) NULL); } } else { free(d); } return (Gal_Frame) NULL; }

When the server receives the receive_audio message, it invokes this dispatch function. If itcan find the appropriate host and port set, it sets up a broker receiving connection, makes itactive, and exits. This connection takes a callback function and data to be stored in the brokerstructure for the callback function to use. If the broker is set up appropriately, this function willalso set up other callbacks which are invoked when the broker is notified of a broker abort or thatdata is done (see the documentation on GalIO_AddBrokerCallback and the event-drivenprogramming model in general for further details).

GalIO_BrokerStruct *GalSS_EnvBrokerDataInInit (GalSS_Environment *env, char *host,unsigned short port, Gal_Frame frame, GalIO_BrokerDataHandler fnptr, int poll_ms, void *refptr,void (*free_fn)(void *)) The call environment env is stored away and made available to the broker data handler fnptr viathe function GalSS_BrokerGetEnvironment. The host and port are the location of the server tocontact. The frame is the frame which delivered the connection message, which is forwarded to

311

the other end of the broker connection to verify that the correct connection is being made. It mustcontain the :call_id key. The fnptr is a function of type GalIO_BrokerDataHandler which willbe called when there is data to be read, and the caller_data is arbitrary data which will be storedin the broker structure (for instance, an object to collect the binary data, as show in the examplehere) and will be freed when the broker is destroyed using the caller_data_free_fn. This data canbe retrieved using the function GalIO_GetBrokerData, as illustrated below. As usual, poll_ms isthe number of milliseconds for the timed task; 0 means 100 ms, > 0 is a value in ms, < 0 meansnot to set up a timed task.

Memory management

The data passed to the callback function is not freed by the caller. It is up to the callbackfunction itself to free the data when it’s done with it.

Here’s an example of a callback function:

static void env_audio_handler(GalIO_BrokerStruct *broker_struct, void *data, Gal_ObjectType data_type, int n_samples) { DataHandler *d = (DataHandler *) GalIO_GetBrokerData(broker_struct);

switch (data_type) { case GAL_BINARY: if (d->data_buf) d->data_buf = (char *) realloc(d->data_buf, n_samples + d->size); else d->data_buf = (char *) malloc(n_samples + d->size); bcopy(data, d->data_buf + d->size, n_samples); d->size += n_samples; free(data); break; default: GalUtil_Warn("Unknown data type %s\n", Gal_ObjectTypeString(data_type)); } }

You can see how the types which are checked correspond to the types of the data which aresent. You can also see how the DataHandler object is retrieved from the broker structure.Finally, observe that the callback function frees the data it’s passed after processing it.

Here’s an example of a general broker callback, in this case fired when the broker is notified ofan abort:

312

void __report_abort(GalIO_BrokerStruct *b, void *data) { Gal_Frame f; GalSS_Environment *env = GalSS_BrokerGetEnvironment(b);

f = Gal_MakeFrame("notify", GAL_CLAUSE); Gal_SetProp(f, ":notification", Gal_StringObject("Audio aborted.")); GalSS_EnvWriteFrame(env, f, 0); Gal_FreeFrame(f); }

You can see here how the callback uses GalSS_BrokerGetEnvironment to send a message withthe appropriate context to the Hub.

void GalIO_SetBrokerActive (GalIO_BrokerStruct *b) Enables the broker to process the data handler. Nothing will happen with the data until thisfunction is called on the broker structure. This is to allow the user to control when the brokerstructures are activated (if, for instance, they play audio output as a side effect).

typedef void (*GalIO_BrokerDataHandler )(GalIO_BrokerStruct *broker_struct, void *object, Gal_ObjectType object_type, int object_count) This is the type of the callback functions. The broker_struct is the broker structure which hasbeen set up to receive the data. The object is the data itself. The type is the type of the object,and the object_count is how many elements of the data there are (this is interesting mostly in thecase of array data sent by GalIO_BrokerWriteBinary, GalIO_BrokerWriteInt16, etc.).

void *GalIO_GetBrokerData (GalIO_BrokerStruct *b) This function retrieves from the broker structure b the data which was passed in to the refptr argument to GalIO_CommBrokerDataInInit.

void GalIO_SetBrokerData (GalIO_BrokerStruct *b, void *caller_data, void (*free_fn)(void *)) If you need to update the caller data at any point, this function will free the old data using thepreviously set free function and update the caller data to caller_data and the free function to free_fn.

void GalIO_BrokerDataDone (GalIO_BrokerStruct *b) This function should be called on the broker structure if the client decides it’s done before all thedata is read.

void GalSS_BrokerSetEnvironment (GalIO_BrokerStruct *b, GalSS_Environment *env) Sets the environment associated with the broker b to env. You should not typically need to usethis function.

313

GalSS_Environment *GalSS_BrokerGetEnvironment (GalIO_BrokerStruct *b) Retrieves the environment associated with the broker b.

Gal_Frame GalIO_GetBrokerFrame (GalIO_BrokerStruct *b) Returns the internal copy of the frame passed to GalIO_CommBrokerDataInInit. You should nottypically need to use this function.

If you’re dealing with multiple incoming broker objects simultaneously, you might wish to use thebroker queues. Broker objects can be connected in a doubly-linked list for the purposes ofordering their processing.

GalIO_BrokerStruct *GalIO_BrokerStructQueueAppend (GalIO_BrokerStruct *b,GalIO_BrokerStruct *bqueue) Adds the broker structure b to the end of the queue bqueue. If bqueue is NULL, b is returned,otherwise bqueue is returned.

GalIO_BrokerStruct *GalIO_BrokerStructQueuePop (GalIO_BrokerStruct *bqueue) Removed all finished broker structure from the queue and returns the broker structure whichcorresponds to the new head of the active queue. The broker structure is destroyed.

GalIO_BrokerStruct *GalIO_BrokerStructDequeue (GalIO_BrokerStruct *b,GalIO_BrokerStruct *bqueue) Removes a broker structure from the queue and returns the new queue.

Backward compatibilityThese functions are provided for backward compatibility with previous releases of the GalaxyCommunicator infrastructure. They are all inadequate in one way or another, and should beavoided. They will not be deprecated or removed yet, however.

GalIO_BrokerStruct *GalIO_BrokerDataInInit (char *host, unsigned short port, Gal_Frame frame, GalIO_BrokerDataHandler fnptr, void *caller_data, int poll_ms) Like GalIO_CommBrokerDataInInit, but doesn’t allow the programmer to pass in the hostconnection or a free function for the data.

void *GalIO_GetBrokerCallerData (GalIO_BrokerStruct *b) Identical to GalIO_GetBrokerData but does not conform to consistent naming conventions.

typedef void (*GalIO_BrokerDataFinalizer )(GalIO_BrokerStruct *, void *caller_data) void GalIO_BrokerSetFinalizer (GalIO_BrokerStruct *b, GalIO_BrokerDataFinalizer finalizer) Sets a finalizer for the broker structure. This finalizer is called when the broker structure isdestroyed. This can be used, for instance, to pop a broker queue and activate the next elementin the queue. The finalizer is called with the contents of the refptr argument passed in toGalIO_BrokerDataInInit. This function has been superseded by the addition of a free function in

314

GalIO_CommBrokerDataInInit and by the callback architecture described here.

GalIO_BrokerStruct *GalIO_CommBrokerDataInInit (GalIO_CommStruct *host_gcomm,char *host, unsigned short port, Gal_Frame frame, GalIO_BrokerDataHandler fnptr, int poll_ms,void *caller_data, void (*caller_data_free_fn)(void *)) This function provides almost the same functionality as GalSS_EnvBrokerDataInInit, except thatthe host_gcomm is the connection which hosts the callback from which the broker was created;it is provided to support the event-driven programming model. All other arguments are as theyare for GalSS_EnvBrokerDataInInit.


315

The Toplevel Server Loop

The toplevel server loop in the MIT Communicator server library exploits a complex array ofcallback and object types. Understanding this structure is crucial to being able to do "unusual"things with the Communicator system. In this section, we’ll describe the rough outline of thetoplevel loop and how it relates to the dispatch functions.

Simulating a main loop Argument packages Manipulating the server object Manipulating the connection object Event-driven programming Listener-in-Hub support

Simulating a main loopHere’s a main loop which exposes the basic functionality. This code is taken from basic_double.c, in the simple_mainloop example. As the comment notes, this main loop isidentical to the main loop in the Communicator library.

/* The main() here is essentially the same main() as in the Communicator library. */



316

GalIO_ServerStruct *GalSS_CmdlineSetupServer (int argc, char **argv) This function creates and returns a server, according to the parameters provided in argc and argv. It first uses the MIT command line argument parsing library to parse the argument list andremove the general server arguments. Next, it calls the function _GalSS_InitializeDefaults, whichis typically generated automatically from server information. It creates a server according to allthis information, initializes signal handling, and calls _GalSS_init_server on the remainingarguments and stores the return value for later retrieval by GalIO_GetCommServerData.

voidGalSS_StartAndRunServer (GalIO_ServerStruct *server) This function sets up the server listener and starts the timed task loop.

Argument packagesIn order to allow the user to configure servers in the most extensible and flexible way possible,the Communicator infrastructure supports argument packages which can be manipulated beforeserver instantiation.

GalIO_ServerStruct *GalSS_SetupServer (GalSS_ServerArgs *arg_pkg, int new_argc, char **new_argv) Like GalSS_CmdnlineSetupServer, but sets up a server given the argument packagearg_pkg.

GalSS_ServerArgs *GalSS_ExtractCmdlineServerArgs (GalSS_ServerArgs *arg_pkg,int argc, char **argv, int *new_argc_ptr, char ***new_argv_ptr) void GalSS_FreeArgPkg (GalSS_ServerArgs *arg_pkg) In case you want to have access to all the standard server arguments, but you need to changesome of the settings programmatically before you initialize the server, you can extract the serverarguments using GalSS_ExtractCmdlineServerArgs and then pass the result toGalSS_SetupServer. GalSS_ExtractCmdlineServerArgs peels off the standard server argumentsand returns the remainder via new_argc_ptr and new_argv_ptr. IfGalSS_ExtractCmdlineServerArgs returns NULL, something’s gone wrong and you should exit.Here’s what the main loop would look like if you did that:

int main(int argc, char **argv) { GalIO_ServerStruct *server; GalSS_ServerArgs *arg_pkg; int new_argc; char **new_argv;

arg_pkg = GalSS_ExtractCmdlineServerArgs((GalSS_ServerArgs *) NULL, argc, argv, &new_argc, &new_argv); if (!arg_pkg) { GalUtil_Fatal("Failed to parse command line arguments!\n");

317

} /* Do whatever you need to do */ /* ... */ server = GalSS_SetupServer(arg_pkg, new_argc, new_argv); if (!server) { GalUtil_Fatal("Failed to initialize server!\n"); } GalSS_FreeArgPkg(arg_pkg); GalSS_StartAndRunServer(server); exit(0); }

GalSS_ServerArgs *GalSS_DefaultServerArgs () Sometimes, you may want to use the built-in argument parsing mechanism, but you might wantto set some of the arguments so that the user can’t modify them. In this case, you can create adefault GalSS_ServerArgs * object with GalSS_DefaultServerArgs and use the functions belowto set the appropriate arguments. In all these cases, the appropriate command line argumentswill be disabled, and the values you provide will be used.

All these following functions can be used to modify the server args either before or after GalSS_ExtractServerArgs() is called. If called before, they disable the appropriate command linearguments, set a fixed value, and return the defaults (or a previously set value). If called after,they still disable the appropriate command line arguments (although this has no effect, since thevalues have already been extracted), but their primary effect is to allow the programmer tocapture and overwrite information from the arguments before the server is created with GalSS_SetupServer(). See the alternate mainloop documentation for a complex example.

unsigned short GalSS_SAFixPort (GalSS_ServerArgs *arg_pkg, unsigned short port) Sets the port to try to start the server listener on to port. 0 uses the default found in the serverdeclaration information. Disables the -port command line argument. Returns the old value.

int GalSS_SAFixMaxConns (GalSS_ServerArgs *arg_pkg, int max_conns) Sets the maximum number of connections permitted to max_conns. A non-positive integer usesthe default found in the server declaration information, or 1 if there is no default. Disables the -maxconns command line argument. Returns the old value.

int GalSS_SAFixVerbosity (GalSS_ServerArgs *arg_pkg, int verbosity) Sets the verbosity level. If verbosity is not -1, it is used as the verbosity level for the server.Disables the -verbosity command line argument. Returns the old value.

int GalSS_SAFixColor (GalSS_ServerArgs *arg_pkg, int color) If color is > 0, printing will be set up for a color xterm; if 0, black and white, if < 0, it’s assumedthat printing is initialized elsewhere. Use 0 for this argument. Disables the -color commandline argument. Returns the old value.

318

int GalSS_SAFixAssert (GalSS_ServerArgs *arg_pkg, int assert) If assert is non-zero, the server initialization will fail if the declared port is not available, instead ofsearching for an available port. Disables the -assert command line argument. Returns the old value.

int GalSS_SAFixValidate (GalSS_ServerArgs *arg_pkg, int validate) If validate is non-zero, the server will check each dispatch function call against the messagesignature. Disables the -validate command line argument. Returns the old value.

int GalSS_SAFixLoopType (GalSS_ServerArgs *arg_pkg, int loop_type) Sets the loop type. The loop_type argument can have one of three values:

GAL_LOOP_TT: use the timed task loop GAL_LOOP_THREADS: use threads instead of the timed task loop GAL_LOOP_EXTERNAL: don’t use any loop (the caller will supply one; see the special mainloop documentation)

Disables the -thread , -ttloop , -nottloop command line arguments. Returns the old value.

int GalSS_SAFixServerListenStatus (GalSS_ServerArgs *arg_pkg, int server_listen_status) Sets the listen status (values documented in GalIO_ServerListenStatus()). If the listen status isfixed and it’s a server listener, then -contact_hub and -session_id are disabled. Returnsthe old value.

char *GalSS_SAFixContactHubInfo (GalSS_ServerArgs *arg_pkg, char *client_pair_status, char *session_id, char **old_session_id_ptr) Sets the Hub contact information, if the server is supposed to contact the Hub. The client_pair_status and session_id arguments have the same form as the command linearguments described in the listener-in-Hub documentation. Disables the -contact_hub , -session_id command line arguments. Returns the old values. The returned memory is up tothe caller to free. If old_session_id_ptr is NULL, the old session ID string is freed internally andnot returned.

char *GalSS_SAFixServerLocations (GalSS_ServerArgs *arg_pkg, char *server_locations_file) Sets the server location file. Disables the -server_locations_file command lineargument. Returns the old value.

Manipulating the server objectThere are a number of functions which can be used to manipulate the server object.

unsigned short GalIO_GetServerListenPort (GalIO_ServerStruct *scomm) This function returns the port number that the server is actually listening on.

319

void *GalIO_GetServerData (GalIO_ServerStruct *scomm) void GalIO_SetServerData (GalIO_ServerStruct *scomm, void *data, void (*free_fn)(void *)) These functions get and set the server data. This data can also be set by returning a value from _GalSS_init_server, although this behavior is strongly discouraged. The server data is alsoavailable for retrieval through individual connections using GalIO_GetCommServerData.

void *GalIO_GetCommServerData (GalIO_CommStruct *gcomm) This function returns the value which was set by GalIO_SetServerData.

void GalIO_SetCommClientData (GalIO_CommStruct *gcomm, char *name, void *client_data) void *GalIO_GetCommClientData (GalIO_CommStruct *gcomm, char *name) void GalIO_SetServerClientData (GalIO_ServerStruct *server, char *name, void *client_data) void *GalIO_GetServerClientData (GalIO_ServerStruct *server, char *name) These functions get and set client_data associated with the dispatch function named by name.The same repository of client data information is accessible through the server or any of itsconnections. This functionality is useful when there’s persistent data that’s relevant to thedispatch function which it doesn’t make sense to make global.

char *GalIO_GetServerName (GalIO_ServerStruct *scomm) void GalIO_SetServerName (GalIO_ServerStruct *scomm, char *name) These function get and set the name by which the server is known. Typically, this information isset in _GalSS_InitializeDefaults.

int GalIO_GetServerMaxConnections (GalIO_ServerStruct *scomm) void GalIO_SetServerMaxConnections (GalIO_ServerStruct *scomm, int max) These functions get and set the maximum number of connections the server may accept.Typically, the default case is handled before _GalSS_init_server is called, so you may reset itthere if you choose. Note that this function can be used to reduce the number of maximumconnections permitted, but it will silently fail if the number of active connections exceeds themaximum requested.

int GalIO_GetServerNumConnections (GalIO_ServerStruct *scomm) Returns the number of connections currently connected to the server.

GalIO_CommStruct *GalIO_GetUniqueConnection (GalIO_ServerStruct *scomm) This function will return the single active connection when there is an active connection and themaximum number of connections permitted is one . This is present mostly for backwardcompatibility. In releases before 2.0, there was no distinction between the server and theconnection object, and only one connection at a time was permitted. In those situations, the codeassumes easy access to the single permitted connection, given the server. This functionprovides that access.

320

unsigned short GalIO_GetServerDefaultPort (GalIO_ServerStruct *scomm) void GalIO_SetServerDefaultPort (GalIO_ServerStruct *scomm, unsigned short port) These functions get and set the default port for the server. Typically, the default case is handledbefore _GalSS_init_server is called, so you may reset it there if you choose.

int GalIO_ServerUsesTimedTasks (GalIO_ServerStruct *server) This function returns 1 if the server is configured to use the timed task loop (the default case), 0 otherwise.

void GalIO_EnableDispatchFnValidation (GalIO_ServerStruct *scomm) This function is called when the validate argument to GalSS_InitializeServerToplevel is set. Allconnections spawned by this server will validate each dispatch function call. Validation cannotbe disabled.

Manipulating the connection objectWhen the server accepts a connection, it creates a GalIO_CommStruct object to handle thatconnection. This object can be accessed from the second argument of each dispatch functionusing the function GalSS_EnvComm. It is possible to use this connection object to access serverinformation or to send information to the Hub. These functions, however, do not provideenough of a context to support appropriate session management . We strongly recommendusing the environment-aware versions of these functions. See the documentation on adding a server and session management.

For example, the function GalIO_CommWriteFrame sends a new message to the Hub. Weillustrate with a simplified variant of the double example:

Gal_Frame twice(Gal_Frame frame, void *server_data) { Gal_Frame new_f = Gal_MakeFrame("main", GAL_CLAUSE); Gal_SetProp(new_f, ":int", Gal_IntObject(2 * Gal_GetInt(frame, ":int"))); GalIO_CommWriteFrame(GalSS_EnvComm((GalSS_Environment *)server_data), new_f, 0); Gal_FreeFrame(new_f); return (Gal_Frame) NULL; }

We see here that we cast the server_data back to a GalSS_Environment* to access theconnection to send the message.

We exemplify the corresponding behavior for server-to-server subdialogues using a variant ofthe complex_twice function in the double example. Here, before the server doubles the input andsubmits a new token, it invokes a "multiply" message to multiply the input by some factor (set inthe server which provides the "multiply" message):

321

Gal_Frame complex_twice(Gal_Frame frame, void *server_data) { Gal_Frame new_f = Gal_MakeFrame("multiply", GAL_CLAUSE); Gal_Frame res_f; GalIO_MsgType t;

Gal_SetProp(new_f, ":int", Gal_IntObject(Gal_GetInt(frame, ":int"))); res_f = GalIO_DispatchViaHub(GalSS_EnvComm((GalSS_Environment *)server_data), new_f, &t);

if (!res_f) { GalUtil_Warn("Didn’t hear back from multiply"); return (Gal_Frame) NULL; } Gal_FreeFrame(new_f); /* Ignoring the message type return for the sake of the example... */ new_f = Gal_MakeFrame("main", GAL_CLAUSE); Gal_SetProp(new_f, ":int", Gal_IntObject(2 * Gal_GetInt(res_f, ":int"))); GalIO_CommWriteFrame(GalSS_EnvComm((GalSS_Environment *)server_data), new_f, 0); Gal_FreeFrame(new_f); return (Gal_Frame) NULL; }

Once again, we see here that we cast the server_data back to a GalSS_Environment* to accessthe connection to send the message.

GalIO_CommStruct *GalSS_EnvComm (GalSS_Environment *env) Retrieves the connection object from the environment.

int GalIO_CommValidating (GalIO_CommStruct *gcomm) Returns 1 if the connection is validating dispatch function calls, 0 otherwise.

void *GalIO_GetCommData (GalIO_CommStruct *gcomm) void GalIO_SetCommData (GalIO_CommStruct *gcomm, void *data, void (*free_fn)(void *)) These functions get and set the data specific to a connection. If free_fn is non-NULL, it will becalled on the data when the data is reset or the connection is destroyed.

char *GalIO_GetCommServerName (GalIO_CommStruct *gcomm) This function retrieves the name by which the server is known from gcomm. This informationwas originally set by GalIO_SetServerName.

322

int GalIO_CommWriteFrame (GalIO_CommStruct *gcomm, Gal_Frame frame, int do_block) This function writes a frame to the Hub through the gcomm connection. The type of the messageis always GAL_MESSAGE_MSG_TYPE. See the section on message types.

Gal_Frame GalIO_DispatchViaHub (GalIO_CommStruct *gcomm, Gal_Frame frame,GalIO_MsgType *msg_type_ptr) This function implements a server-to-server subdialogue with the Hub. It sends the frame andwaits for a reply. The type of the reply is stored in msg_type_ptr. If the reply type isGAL_DESTROY_MSG_TYPE or GAL_MESSAGE_MSG_TYPE, this function prints a warningand returns NULL; therefore, the only legal values for *msg_type_ptr areGAL_REPLY_MSG_TYPE and GAL_ERROR_MSG_TYPE. See the section on message types.

Memory management

None of the frames related to these two functions are freed.

Event-driven programmingThe Communicator infrastructure has a rich set of event-driven programming capabilities.Programmers can register callbacks to be run at any one of these events. This event-drivenmodel is used to implement the Communicator main loop, as well as external main loops andprogramming language bindings. Using this model to its fullest, it is possible to write dispatchfunctions identically for the Communicator main loop and external main loops.

These event callbacks should not have any interdependencies among them. They are notguaranteed to be called in the order they are defined.

323

event description

GAL_SERVER_LISTENER_STARTUP_EVENT

The server has just opened a listener ona port, either because it’s listening forserver connections or an outgoingbroker requires it

GAL_SERVER_LISTENER_SHUTDOWN_EVENTThe server is about to shut down its listener

GAL_SERVER_CLIENT_POLL_STARTUP_EVENTThe server hast just started attemptingto contact the Hub

GAL_SERVER_DESTRUCTION_EVENT The server is about to be destroyed

GAL_SERVER_CONNECTION_CREATION_EVENTThe server has just created a new connection

GAL_CONNECTION_BROKER_OUT_STARTUP_EVENTThe connection has just created anoutgoing broker

GAL_CONNECTION_BROKER_IN_STARTUP_EVENTThe connection has just created anincoming broker

GAL_CONNECTION_DISPATCH_FN_EVENTThe connection is about to invoke adispatch function

GAL_CONNECTION_SHUTDOWN_EVENT The connection is about to shuts down

GAL_CONNECTION_DESTRUCTION_EVENT The connection is about to be destroyed

GAL_BROKER_DATA_DONE_EVENT

The broker has just determined it isdone, either by receiving a terminationmessage or via the call GalIO_BrokerDataDone.

GAL_BROKER_ABORT_EVENTThe broker is about to be destroyedbefore determining it is done

GAL_BROKER_DESTRUCTION_EVENT The broker is about to be destroyed

GAL_BROKER_CONNECTION_EVENTThe (outgoing) broker has just accepteda connection

IMPORTANT. These callbacks cannot be reentrant. If these callbacks call themselves, you’llget a deadlock in the threaded version of the library.

typedef void (*GalIO_ServerCallbackFn )(GalIO_ServerStruct *, void *); GalIO_Callback *GalIO_AddServerCallback (GalIO_ServerStruct *scomm, int callback_event, GalIO_ServerCallbackFn fn, void *callback_data) This function adds the callback fn to the server scomm. The possible values for callback_event

324

are GAL_SERVER_LISTENER_STARTUP_EVENT,GAL_SERVER_LISTENER_SHUTDOWN_EVENT,GAL_SERVER_CLIENT_POLL_STARTUP_EVENT, GAL_SERVER_DESTRUCTION_EVENT. The fn is invoked with the server scomm and callback_data. The callback is returned.

typedef void (*GalIO_ServerConnectCallbackFn )(GalIO_ServerStruct *,GalIO_CommStruct *, void *); GalIO_Callback *GalIO_AddServerConnectCallback (GalIO_ServerStruct *scomm,GalIO_ServerConnectCallbackFn connect_callback, void *callback_data) This function adds the callback fn to the server scomm. The event is GAL_SERVER_CONNECTION_CREATION_EVENT. The fn is invoked with the server scomm, thenew connection, and callback_data. The callback is returned.

void GalIO_RemoveServerCallback (GalIO_ServerStruct *scomm, GalIO_Callback *cb) Removes the callback cb from the server scomm.

typedef void (*GalIO_ConnectionCallbackFn )(GalIO_CommStruct *, void *); GalIO_Callback *GalIO_AddConnectionCallback (GalIO_CommStruct *gcomm, int callback_event, GalIO_ConnectionCallbackFn connect_callback, void *callback_data) This function adds the callback fn to the connection gcomm. The possible values for callback_event are GAL_CONNECTION_SHUTDOWN_EVENT, GAL_CONNECTION_DESTRUCTION_EVENT. The fn is invoked with the connection gcomm and callback_data. The callback is returned.

typedef void (*GalIO_ConnectionBrokerCallbackFn )(GalIO_CommStruct *,GalIO_BrokerStruct *, void *); GalIO_Callback *GalIO_AddConnectionBrokerCallback (GalIO_CommStruct *gcomm,int callback_event, GalIO_ConnectionBrokerCallbackFn connect_callback, void *callback_data) This function adds the callback fn to the connection gcomm. The possible values for callback_event are GAL_CONNECTION_BROKER_OUT_STARTUP_EVENT, GAL_CONNECTION_BROKER_IN_STARTUP_EVENT. The fn is invoked with the connection gcomm, the new broker, and callback_data. The callback is returned.

typedef void (*GalIO_ConnectionDispatchFnCallbackFn )(GalSS_Environment *,Gal_Frame, void *); GalIO_Callback *GalIO_AddConnectionDispatchFnCallback (GalIO_CommStruct *gcomm, GalIO_ConnectionDispatchFnCallbackFn dispatch_callback, void *callback_data) This function adds the callback fn to the connection gcomm. The event is GAL_CONNECTION_DISPATCH_FN_EVENT. The fn is invoked with the environment of thedispatch function call, the frame comprising the message, and callback_data. The callback is returned.

void GalIO_RemoveConnectionCallback (GalIO_CommStruct *gcomm, GalIO_Callback *cb) Removes the callback cb from the connection gcomm.

325

typedef void (*GalIO_BrokerCallbackFn )(GalIO_BrokerStruct *, void *); GalIO_Callback *GalIO_AddBrokerCallback (GalIO_BrokerStruct *b, int callback_event,GalIO_BrokerCallbackFn fn, void *callback_data) This function adds the callback fn to the broker b. The possible values for callback_event areGAL_BROKER_DATA_DONE_EVENT, GAL_BROKER_ABORT_EVENT,GAL_BROKER_DESTRUCTION_EVENT, GAL_BROKER_CONNECTION_EVENT. The fn is invokedwith the broker b and callback_data. The callback is returned.

void GalIO_RemoveBrokerCallback (GalIO_BrokerStruct *b, GalIO_Callback *cb) Removes the callback cb from the broker b.

Listener-in-Hub supportAs of version 2.1, it is possible to set up Hubs and servers so that servers contact listeners in the Hub, instead of Hubs contacting listeners in servers. This new functionality requires API support.

GalIO_CommStruct *GalIO_ContactHub (char *host, unsigned short port,GalIO_ServerStruct *scomm, char *session_id, int client_poll_flags) This function is used to contact a Hub on the given host and port, and associate the resultingconnection with the given server scomm. If the session_id argument is not NULL, the connectionwill use the given session ID instead of the default provided to GalSS_InitializeServerToplevel()(also accessible via the function GalIO_ServerSessionID()). This function can be used to addnew connections to a server after the server starts up. The new connection will be processedaccording to the listener status associated with the server (see GalIO_ServerListenStatus()). Ifthe client_poll_flags are not -1, they provide specific control over whether this connection isrestarted or retried (see GalIO_ServerListenStatus).

void GalIO_OperateOnConnections (GalIO_ServerStruct *scomm, void *arg, void (*op)(GalIO_CommStruct *, void *)) This function applies the operation op to each connection associated with the server scomm.The operation is also passed arg.

int GalIO_ServerListenStatus (GalIO_ServerStruct *scomm) The listener status is an integer whose bits correspond to various aspects of the server behavior.One set of bits control whether the server is listening for connections and/or brokers from theHub and/or connecting as a client to the Hub; another set of bits is whether, as a Hub client, theserver should retry an connection if it fails to connect; and a final set of bits control whether, as aHub client the server should attempt to reconnect after a disconnect, do nothing, or shutdownwhen the last Hub disconnects. You can set this status using GalSS_SAFixServerListenStatus().The constants and their masks are as follows:

Constant Mask Description

326

GAL_CONNECTION_LISTENER GAL_SERVER_TYPE_MASKServerlistens for connections

GAL_BROKER_LISTENER GAL_SERVER_TYPE_MASKServerlistens for brokers

GAL_HUB_CLIENT GAL_SERVER_TYPE_MASKServerconnects to Hub

GAL_HUB_CLIENT_CONNECT_FAILURE_RETRY GAL_HUB_CLIENT_CONNECT_FAILURE_MASK

Serverconnectingto Hubretries if itcan’testablish aninitial connection

GAL_HUB_CLIENT_CONNECT_FAILURE_SHUTDOWNGAL_HUB_CLIENT_CONNECT_FAILURE_MASK

Serverconnectingto Hubshuts downif it can’testablish aninitial connection

GAL_HUB_CLIENT_CONNECT_FAILURE_NOOP GAL_HUB_CLIENT_CONNECT_FAILURE_MASK

Serverconnectingto Hubdoesnothing if itcan’testablish aninitial connection

GAL_HUB_CLIENT_DISCONNECT_RETRY GAL_HUB_CLIENT_DISCONNECT_MASK

Serverconnectingto Hubretries after disconnect

327

GAL_HUB_CLIENT_DISCONNECT_SHUTDOWN GAL_HUB_CLIENT_DISCONNECT_MASK

Serverconnectingto Hub exitsafter last disconnect

GAL_HUB_CLIENT_DISCONNECT_NOOP GAL_HUB_CLIENT_DISCONNECT_MASK

Serverconnectingto Hubdoesnothingafter disconnect

Typically, you won’t need to probe these flags directly, if you need them at all. The functions GalIO_ServerIsClient(), GalIO_ServerIsListener(), etc. will allow you to determine the server’sstatus. Under some circumstances, you might need more complex information. For instance, ifyou want to know if your server will shutdown after the last disconnect, you can use the test

(GalIO_ServerListenStatus(scomm) & GAL_HUB_CLIENT_DISCONNECT_MASK) == GAL_HUB_CLIENT_DISCONNECT_SHUTDOWN

And so on. The default listen status is GAL_CONNECTION_LISTENER; when the server typeincludes GAL_HUB_CLIENT, the default client bits are GAL_HUB_CLIENT_CONNECT_FAILURE_RETRY | GAL_HUB_CLIENT_DISCONNECT_RETRY.If you set multiple values for the client connect and disconnect statuses (which you should neverdo, because it’s pointless, but just in case), the precedence is SHUTDOWN > NOOP > RETRY.

char *GalIO_ServerSessionID (GalIO_ServerStruct *scomm) Returns the default session ID for the server, as determined by GalSS_InitializeServerToplevel().The actual session ID is returned, not a copy.

Deprecated functionsThese functions will still be supported, but will be removed in a future release. Any newcommand line arguments available to Communicator-compliant servers will not be available ifyou use these functions.

328

GalIO_ServerStruct *GalSS_CmdlineInitializeServer (int argc, char **argv) Like GalSS_CmdlineSetupServer, but also starts up the appropriate listeners.

GalIO_ServerStruct *GalSS_InitializeServerToplevel (unsigned short server_port, int max_conns, int use_color, int do_assert, int loop_type, int validate, int verbosity, int server_listen_status, char *client_pair_string, char *session_id, int new_argc, char **new_argv) This function was originally called internally by GalSS_CmdlineInitializeServer when theserver-relevant information had been extracted from the arglist, but it became obvious that thesignature of this function would have to be changed every time an argument was added to theservers; therefore, the functionality here has been superseded by argument packages. Thearguments correspond to the various GalSS_SAFix* functions.

GalIO_ServerStruct *GalSS_InitializeServer (unsigned short server_port, int max_conns,int use_color, int do_assert, int use_ttloop, int validate, int new_argc, char **new_argv) This function is an old version of GalSS_InitializeServerToplevel() which does not provide forinitialization of verbosity or listener-in-Hub support. Provided for backward compability.

GalIO_ServerStruct *GalSS_InitializeServerFromServerArgs (GalSS_ServerArgs *arg_pkg, int new_argc, char **new_argv)

GalSS_ServerArgs *GalSS_ExtractServerArgs (int argc, char **argv, int *new_argc_ptr,char ***new_argv_ptr) A version of GalSS_ExtractCmdlineServerArgs() which doesn’t allow a default argumentpackage. Provided for backward compatibility.

void GalSS_RunServer (GalIO_ServerStruct *server) This function starts the timed task loop.


329

Reserved Keys

All information in the Galaxy Communicator system is stored in frames. There are a number ofreserved keys which are used to pass important information between the Hub and servers, orwithin the Hub. This list is provided only for reference; these keys are not guaranteed to remainthe constant, although every effort will be made to keep them so. The programmer should notneed to refer to any of them directly.

Hub/server keysThese keys are used both in the Hub and servers. Programming language bindings must respectthese keys. These keys are defined in include/galaxy/distinguished_keys.h.

variable name value description functions

GAL_ERROR_NUMBER_FRAME_KEY :errnothe errornumber of an error

GalIO_CreateErrorFrame, GalIO_GetError, GalIO_CommDispatchFrame

GAL_ERROR_DESCRIPTION_FRAME_KEY:err_descriptionthe error description

GalIO_CreateErrorFrame, GalIO_GetError

GAL_SESSION_ID_FRAME_KEY :session_id

the ID of thecurrent session(used inopaque data,deprecated at toplevel)

GAL_TOKEN_INDEX_FRAME_KEY :tidx

the uniqueindex of the Hub tokenassociated withthis message;used in opaque data

330

GAL_SERVER_TOKEN_INDEX_FRAME_KEY:server_tidx

the uniqueindex ofserver-to-serversubdialoguesgenerated bythe servers,which isreturned to theserver with thereply; used inopaque data

GalIO_CommDispatchFrame

GAL_HUB_OPAQUE_DATA_FRAME_KEY :hub_opaque_data

opaque datacarried alongby servermessages forthe Hub

GAL_SIGNATURES_FRAME_KEY :signatures

the server message signaturesreported backto the Hubduring the handshake

GAL_SERVER_PROPERTIES_FRAME_KEY:properties

the serverpropertiesreported backto the Hubduring the handshake

GAL_SERVICE_TYPE_FRAME_KEY :extra_service_types

the extraservice typesreported backto the Hubduring the handshake

GAL_BROKER_CALL_ID_FRAME_KEY :call_id

the unique"cookie" tovalidate broker connections

GalIO_FrameSetBrokerCallID

331

GAL_ROUND_TRIP_FRAME_KEY :reply_requested

indicates thatthe serverwants ananswer to thismessage, orthat the Hubwants ananswer to thismessage; usedin opaque data

GalIO_DispatchViaHub

GAL_CONNECTION_TYPE_FRAME_KEY :conn_type

holds theconnection type(broker ornormalconnection)during ahandshakebetween aclient and listener

GAL_HUB_NEW_SESSION_FRAME_KEY :hub_new_session

indicates thatthe Hub shouldcall the Builtinfunction new_session (deprecated)

GAL_RESET_SESSION_FRAME_KEY :reset_session

indicates thatthe Hub shouldcall the Builtinfunction new_session (in opaque data)

GAL_PROTOCOL_VERSION_FRAME_KEY:protocol_version

the currentversion of thecommunicationprotocol theserver or Hub isrunning,exchanged inthe handshake

332

GAL_UTTERANCE_ID_FRAME_KEY :utterance_idID of thecurrent utterance

GAL_LOCK_MASK_FRAME_KEY :lock_mask

value of thesession lockmask; inopaque data

GAL_LOCK_VALUE_FRAME_KEY :lock_value

value of thesession lockvalue; inopaque data

GalSS_EnvSetSession

Hub keysThese keys are used within the Hub to record information and to pass information between theHub core and its scripting module.

333

variable name value description

GAL_HUB_SESSION_USER_HUB_FRAME_KEY :hub_session_userstores theuser for this session

GAL_HUB_SERVER_UNAVAILABLE_HUB_FRAME_KEY:hub_server_unavailable

indicatesthat theserver is unavailable

GAL_HUB_SESSION_UTTERANCE_ID_HUB_FRAME_KEY:hub_session_utterance_id

internalstorage ofutterance ID

GAL_NO_ABORT_HUB_FRAME_KEY :no_abort

indicatesthat theprogramcannot be aborted

GAL_SESSION_ERROR_COUNT_HUB_FRAME_KEY :session_error_count

record ofnumber oferrors in thecurrent session

GAL_OPAQUE_SCRIPT_MODULE_HUB_FRAME_KEY :hub_program_info

opaquedata thescriptingmodule maintains


334

Alphabetical Index of API Calls

GalIO_CommStruct *GalIO_AcceptUniqueConnection (GalIO_ServerStruct *scomm)

GalIO_Callback *GalIO_AddBrokerCallback (GalIO_BrokerStruct *b, int callback_event,GalIO_BrokerCallbackFn fn, void *callback_data)

void GalIO_AddCommDispatchFunctionEntry (GalIO_CommStruct *gcomm, char *name, Gal_FrameFnPtr fn, Gal_FrameDataFnPtr fn_with_data,Gal_DispatchFnSignatureKeyEntry *in_key_array, int allow_other_in_keys, int reply_provided,Gal_DispatchFnSignatureKeyEntry *out_key_array, int allow_other_out_keys)

GalIO_Callback *GalIO_AddConnectionBrokerCallback (GalIO_CommStruct *gcomm,int callback_event, GalIO_ConnectionBrokerCallbackFn connect_callback, void *callback_data)

GalIO_Callback *GalIO_AddConnectionCallback (GalIO_CommStruct *gcomm, int callback_event, GalIO_ConnectionCallbackFn connect_callback, void *callback_data)

GalIO_Callback *GalIO_AddConnectionDispatchFnCallback (GalIO_CommStruct *gcomm, GalIO_ConnectionDispatchFnCallbackFn dispatch_callback, void *callback_data)

GalIO_Callback *GalIO_AddServerCallback (GalIO_ServerStruct *scomm, int callback_event, GalIO_ServerCallbackFn fn, void *callback_data)

GalIO_Callback *GalIO_AddServerConnectCallback (GalIO_ServerStruct *scomm,GalIO_ServerConnectCallbackFn connect_callback, void *callback_data)

void GalIO_AddServerDispatchFunctionEntry (GalIO_ServerStruct *server, char *name, Gal_FrameFnPtr fn, Gal_FrameDataFnPtr fn_with_data,Gal_DispatchFnSignatureKeyEntry *in_key_array, int allow_other_in_keys, int reply_provided,Gal_DispatchFnSignatureKeyEntry *out_key_array, int allow_other_out_keys)

GalIO_ServerLocation *GalIO_AddServerLocation (GalIO_ServerStruct *scomm, char *host, unsigned short port, int client_poll_flags)

void GalIO_AddServiceType (GalIO_ServerStruct *server, char *stype)

335

void GalIO_BrokerDataDone (GalIO_BrokerStruct *b)

int GalIO_BrokerDataInCallbackHandler (GalIO_BrokerStruct *b, int read_blocking)

int GalIO_BrokerDataInHandler (GalIO_BrokerStruct *b)

GalIO_BrokerStruct *GalIO_BrokerDataInInit (char *host, unsigned short port, Gal_Frame frame, GalIO_BrokerDataHandler fnptr, void *refptr, int poll_ms)

int GalIO_BrokerDataOutCallbackHandler (GalIO_BrokerStruct *b)

void GalIO_BrokerDataOutDone (GalIO_BrokerStruct *b)

int GalIO_BrokerDataOutHandler (GalIO_BrokerStruct *b)

GalIO_BrokerStruct *GalIO_BrokerDataOutInit (GalIO_CommStruct *gcomm, int poll_ms,int timeout_seconds)

int GalIO_BrokerIsDone (GalIO_BrokerStruct *b)

void GalIO_BrokerPopulateFrame (GalIO_BrokerStruct *b, Gal_Frame f, char *host_key,char *port_key)

int GalIO_BrokerReadReady (GalIO_BrokerStruct *b)

void GalIO_BrokerSetFinalizer (GalIO_BrokerStruct *b, GalIO_BrokerDataFinalizer finalizer)

GalIO_BrokerStruct *GalIO_BrokerStructDequeue (GalIO_BrokerStruct *b,GalIO_BrokerStruct *bqueue)

GalIO_BrokerStruct *GalIO_BrokerStructQueueAppend (GalIO_BrokerStruct *b,GalIO_BrokerStruct *bqueue)

GalIO_BrokerStruct *GalIO_BrokerStructQueuePop (GalIO_BrokerStruct *bqueue)

int GalIO_BrokerWriteBinary (GalIO_BrokerStruct *b, void *data, int n_bytes)

int GalIO_BrokerWriteFloat (GalIO_BrokerStruct *b, float f)

int GalIO_BrokerWriteFloat32 (GalIO_BrokerStruct *b, void *data, int n_floats)

int GalIO_BrokerWriteFloat64 (GalIO_BrokerStruct *b, void *data, int n_floats)

336

int GalIO_BrokerWriteFrame (GalIO_BrokerStruct *b, Gal_Frame frame)

int GalIO_BrokerWriteInt (GalIO_BrokerStruct *b, int i)

int GalIO_BrokerWriteInt16 (GalIO_BrokerStruct *b, void *data, int n_ints)



int GalIO_BrokerWriteList (GalIO_BrokerStruct *b, Gal_Object *elts, int n_elts)

int GalIO_BrokerWriteObject (GalIO_BrokerStruct *b, Gal_Object o)

int GalIO_BrokerWriteReady (GalIO_BrokerStruct *b)

int GalIO_BrokerWriteString (GalIO_BrokerStruct *b, char *str)

GalIO_CommStruct *GalIO_ClientConnect (char *name, char *host, unsigned short port, int silent, Gal_Frame welcome_frame, Gal_Frame *reply_frame)

GalIO_CommStruct *GalIO_ClientInit (char *host, unsigned short port, Gal_Frame frame,GalIO_FrameHandler fnptr, int poll_ms)

void GalIO_CloseCommSocket (GalIO_CommStruct *gcomm)

void GalIO_CloseServerListenSocket (GalIO_ServerStruct *scomm)

GalIO_BrokerStruct *GalIO_CommBrokerDataInInit (GalIO_CommStruct *host_gcomm,char *host, unsigned short port, Gal_Frame frame, GalIO_BrokerDataHandler fnptr, int poll_ms,void *caller_data, void (*caller_data_free_fn)(void *))

Gal_Frame GalIO_CommDispatchFrame (GalIO_CommStruct *gcomm, Gal_Frame dispatch, GalIO_MsgType *msg_type_ptr, char *reply_key)

void GalIO_CommFlushOutQueue (GalIO_CommStruct *gcomm)

int GalIO_CommReadFrame (GalIO_CommStruct *gcomm, Gal_Frame *frame, int do_block)

int GalIO_CommReadMessage (GalIO_CommStruct *gcomm, Nframe *frame,GalIO_MsgType *msg_type_ptr, int do_block)

int GalIO_CommReadReady (GalIO_CommStruct *gcomm)

337

GalIO_ServerStruct *GalIO_CommServer (GalIO_CommStruct *gcomm)

int GalIO_CommValidating (GalIO_CommStruct *gcomm)

int GalIO_CommWriteFrame (GalIO_CommStruct *gcomm, Gal_Frame frame, int do_block)

int GalIO_CommWriteMessage (GalIO_CommStruct *gcomm, Gal_Frame frame,GalIO_MsgType msg_type, int do_block)

int GalIO_CommWriteReady (GalIO_CommStruct *gcomm)

int GalIO_ConnectionCallbackHandler (GalIO_CommStruct *gcomm, int read_blocking)

int GalIO_ConnectionPoll (GalIO_CommStruct *gcomm)

GalIO_CommStruct *GalIO_ContactHub (char *host, unsigned short port,GalIO_ServerStruct *scomm, char *session_id, int client_poll_flags)

Gal_Frame GalIO_CreateErrorFrame (int errnum, char *error)

GalIO_SockQueueStruct *GalIO_CreateSockQueue (GAL_SOCKET sockfd, int in_size, int out_size)

void GalIO_DestroyBrokerStruct (GalIO_BrokerStruct *b)

void GalIO_DestroyCommStruct (GalIO_CommStruct *gcomm)

void GalIO_DestroyServerStruct (GalIO_ServerStruct *scomm)

void GalIO_DestroySockQueue (GalIO_SockQueueStruct *q)

GalIO_ServerLocation *GalIO_DigestServerLocations (char *client_pair_string)

Gal_Frame GalIO_DispatchViaHub (GalIO_CommStruct *gcomm, Gal_Frame frame,GalIO_MsgType *msg_type_ptr)

void GalIO_EnableDispatchFnValidation (GalIO_ServerStruct *scomm)

void GalIO_ForceBrokerExpiration (GalIO_BrokerStruct *b)

void GalIO_FrameSetBrokerCallID (Gal_Frame f, char *call_id)

char *GalIO_GetBrokerCallID (GalIO_BrokerStruct *b)

338

void *GalIO_GetBrokerCallerData (GalIO_BrokerStruct *b)

void *GalIO_GetBrokerData (GalIO_BrokerStruct *b)

Gal_Frame GalIO_GetBrokerFrame (GalIO_BrokerStruct *b)

unsigned short GalIO_GetBrokerListenPort (GalIO_BrokerStruct *b)

GAL_SOCKET GalIO_GetBrokerListenSocket (GalIO_BrokerStruct *b)

GAL_SOCKET GalIO_GetBrokerSocket (GalIO_BrokerStruct *b)

void *GalIO_GetCommClientData (GalIO_CommStruct *gcomm, char *name)

void *GalIO_GetCommData (GalIO_CommStruct *gcomm)

Gal_DispatchFnPkg *GalIO_GetCommDispatchFnPkg (GalIO_CommStruct *gcomm)

Gal_Frame GalIO_GetCommFrame (GalIO_CommStruct *gcomm)

char *GalIO_GetCommHost (GalIO_CommStruct *gcomm)

void *GalIO_GetCommServerData (GalIO_CommStruct *gcomm)

char *GalIO_GetCommServerName (GalIO_CommStruct *gcomm)

GAL_SOCKET GalIO_GetCommSocket (GalIO_CommStruct *gcomm)

int GalIO_GetError (Gal_Frame f, char **err_desc)

void *GalIO_GetServerClientData (GalIO_ServerStruct *server, char *name)

void *GalIO_GetServerData (GalIO_ServerStruct *scomm)

unsigned short GalIO_GetServerDefaultPort (GalIO_ServerStruct *scomm)

Gal_DispatchFnPkg *GalIO_GetServerDispatchFnPkg (GalIO_ServerStruct *scomm)

unsigned short GalIO_GetServerListenPort (GalIO_ServerStruct *scomm)

GAL_SOCKET GalIO_GetServerListenSocket (GalIO_ServerStruct *scomm)

GalIO_ServerLocation *GalIO_GetServerLocations (GalIO_ServerStruct *scomm)

339

int GalIO_GetServerMaxConnections (GalIO_ServerStruct *scomm)

char *GalIO_GetServerName (GalIO_ServerStruct *scomm)

int GalIO_GetServerNumConnections (GalIO_ServerStruct *scomm)

int GalIO_GetSockQueueError (GalIO_SockQueueStruct *q)

GalIO_CommStruct *GalIO_GetUniqueConnection (GalIO_ServerStruct *scomm)

int GalIO_HubHandler (GalIO_CommStruct *gcomm, int silent, Gal_Frame *reply)

char *GalIO_IPAddress (void )

int GalIO_InHandler (GalIO_CommStruct *gcomm)

GalIO_ServerStruct *GalIO_ListenerCreate (unsigned short port, int require_port,GalIO_FrameHandler foo_ptr, void *server_data, int poll_ms, int max_connections)

char *GalIO_MsgTypeToName (GalIO_MsgType mt)

char *GalIO_NthHostAndPort (GalIO_ServerLocation *locs, int i, unsigned short *port)

int GalIO_NumServerLocations (GalIO_ServerLocation *locs)

void GalIO_OperateOnConnections (GalIO_ServerStruct *scomm, void *arg, void (*op)(GalIO_CommStruct *, void *))

int GalIO_OutHandler (GalIO_CommStruct *gcomm)

int GalIO_QueueProcessOut (GalIO_CommStruct *gcomm, int do_block)

void GalIO_RemoveBrokerCallback (GalIO_BrokerStruct *b, GalIO_Callback *cb)

void GalIO_RemoveConnectionCallback (GalIO_CommStruct *gcomm, GalIO_Callback *cb)

void GalIO_RemoveServerCallback (GalIO_ServerStruct *scomm, GalIO_Callback *cb)

int GalIO_ResetCommSockets (GalIO_ServerStruct *scomm)

int GalIO_ServerCallbackHandler (GalIO_ServerStruct *scomm, int read_blocking,GalIO_CommStruct **new_conn_ptr)

340

void GalIO_ServerCheckHubContacts (GalIO_ServerStruct *scomm)

GalIO_ServerStruct *GalIO_ServerCreate (unsigned short port, int require_port,GalIO_FrameHandler fnptr, void *server_data, int poll_ms, int max_connections)

int GalIO_ServerHandler (GalIO_ServerStruct *scomm, GalIO_CommStruct **new_conn_ptr)

GalIO_ServerStruct *GalIO_ServerInit (unsigned short port, int require_port,GalIO_FrameHandler fnptr, void *server_data, int poll_ms, int max_connections)

int GalIO_ServerIsClient (GalIO_ServerStruct *scomm)

int GalIO_ServerIsListener (GalIO_ServerStruct *scomm)

int GalIO_ServerListenStatus (GalIO_ServerStruct *scomm)

int GalIO_ServerListensForBrokers (GalIO_ServerStruct *s)

int GalIO_ServerListensForConnections (GalIO_ServerStruct *s)

void GalIO_ServerModifyProperties (GalIO_ServerStruct *server, Gal_Frame new_properties, char **delete_properties)

int GalIO_ServerPoll (GalIO_ServerStruct *scomm)

Gal_Frame GalIO_ServerProperties (GalIO_ServerStruct *server)

char *GalIO_ServerSessionID (GalIO_ServerStruct *scomm)

GalIO_ServerStruct *GalIO_ServerStart (GalIO_ServerStruct *scomm)

int GalIO_ServerUsesTimedTasks (GalIO_ServerStruct *server)

void GalIO_SetBrokerActive (GalIO_BrokerStruct *b)

void GalIO_SetBrokerData (GalIO_BrokerStruct *b, void *caller_data, void (*free_fn)(void *))

void GalIO_SetCommClientData (GalIO_CommStruct *gcomm, char *name, void *client_data)

void GalIO_SetCommData (GalIO_CommStruct *gcomm, void *data, void (*free_fn)(void *))

341

void GalIO_SetCommDispatchFnPkg (GalIO_CommStruct *gcomm, Gal_DispatchFnPkg *pkg)

void GalIO_SetCommDone (GalIO_CommStruct *gcomm)

void GalIO_SetDisconnectCallback (GalIO_CommStruct *gcomm, void (*disconnect_callback)(GalIO_CommStruct *server_data))

void GalIO_SetServerClientData (GalIO_ServerStruct *server, char *name, void *client_data)

void GalIO_SetServerData (GalIO_ServerStruct *scomm, void *data, void (*free_fn)(void *))

void GalIO_SetServerDefaultPort (GalIO_ServerStruct *scomm, unsigned short port)

void GalIO_SetServerDisconnectCallback (GalIO_ServerStruct *scomm, void (*disconnect_callback)(GalIO_CommStruct *server_data))

void GalIO_SetServerDispatchFnPkg (GalIO_ServerStruct *scomm, Gal_DispatchFnPkg *pkg)

void GalIO_SetServerDone (GalIO_ServerStruct *scomm)

void GalIO_SetServerListenStatus (GalIO_ServerStruct *scomm, int server_listen_status,char *client_pair_string, char *session_id)

void GalIO_SetServerMaxConnections (GalIO_ServerStruct *scomm, int max)

void GalIO_SetServerName (GalIO_ServerStruct *scomm, char *name)

void GalIO_SetSockQueueComment (GalIO_SockQueueStruct *q, char *comment)

int GalIO_SockQueueProcessReceive (GalIO_SockQueueStruct *q)

int GalIO_SockQueueProcessSend (GalIO_SockQueueStruct *q)

int GalIO_SockQueueRead (GalIO_SockQueueStruct *q, char *data, int max_data_len, int *data_len)

int GalIO_SockQueueReadAtom (GalIO_SockQueueStruct *q, char *data, int data_len)

int GalIO_SockQueueWrite (GalIO_SockQueueStruct *q, void *data, long len)

int GalIO_StartClient (GalIO_ServerStruct *scomm)

342

int GalIO_StartListener (GalIO_ServerStruct *scomm, int additional_flags)

int GalIO_VerificationHandler (GalIO_CommStruct *gcomm)

void GalSS_AddBareDispatchFunction (GalIO_ServerStruct *i, char *name,Gal_FrameFnPtr fn, Gal_DispatchFnSignatureKeyEntry *in_key_array, int allow_other_in_keys,int reply_provided, Gal_DispatchFnSignatureKeyEntry *out_key_array, int allow_other_out_keys)

void GalSS_AddDispatchFunction (GalIO_ServerStruct *i, char *name,Gal_FrameDataFnPtr fn, Gal_DispatchFnSignatureKeyEntry *in_key_array, int allow_other_in_keys, int reply_provided, Gal_DispatchFnSignatureKeyEntry *out_key_array, int allow_other_out_keys)

GalSS_Environment *GalSS_BrokerGetEnvironment (GalIO_BrokerStruct *b)

void GalSS_BrokerSetEnvironment (GalIO_BrokerStruct *b, GalSS_Environment *env)

Gal_Frame GalSS_CallServerFunction (Gal_Frame frame, GalSS_Environment *env, int optional)

GalIO_ServerStruct *GalSS_CmdlineInitializeServer (int argc, char **argv)

GalIO_ServerStruct *GalSS_CmdlineSetupServer (int argc, char **argv)

GalSS_ServerArgs *GalSS_DefaultServerArgs ()

GalSS_ServerLocationEntry *GalSS_DigestServerLocationFile (char *filename)

GalIO_BrokerStruct *GalSS_ELRBroker (GalSS_ELR *elr)

GalSS_ELR *GalSS_ELRCopy (GalSS_ELR *source)

GalSS_ELR *GalSS_ELRCreate (GalIO_ServerStruct *scomm, GalSS_ELTimerSetFn timer_set_fn, GalSS_ELUnsetFn timer_unset_fn, GalSS_ELFDSetFn fd_set_fn,GalSS_ELUnsetFn fd_unset_fn, GalSS_ELBehaviorFn behavior_fn, int timer_is_persistent)

void GalSS_ELRDestroy (GalSS_ELR *elr)

void GalSS_ELRDoCallback (GalSS_ELR *elr, int timer_or_fd)

GalIO_CommStruct *GalSS_ELRGComm (GalSS_ELR *elr)

void *GalSS_ELRGetLoopData (GalSS_ELR *elr)

343

GalIO_ServerStruct *GalSS_ELRSComm (GalSS_ELR *elr)

void GalSS_ELRSetBrokerInCallback (GalSS_ELR *elr, GalSS_ELCallbackFn fn)

void GalSS_ELRSetBrokerOutCallback (GalSS_ELR *elr, GalSS_ELCallbackFn fn)

void GalSS_ELRSetConnectionCallback (GalSS_ELR *elr, GalSS_ELCallbackFn fn)

void GalSS_ELRSetLoopData (GalSS_ELR *elr, void *loop_data, void (*loop_data_free_fn)(void *))

void GalSS_ELRSetServerClientCallback (GalSS_ELR *elr, GalSS_ELCallbackFn fn)

void GalSS_ELRSetServerListenerCallback (GalSS_ELR *elr, GalSS_ELCallbackFn fn)

GalSS_ELR *GalSS_ELRSetupServer (GalSS_ServerArgs *external_arg_pkg, int argc,char **argv, GalSS_ELTimerSetFn timer_set_fn, GalSS_ELUnsetFn timer_unset_fn,GalSS_ELFDSetFn fd_set_fn, GalSS_ELUnsetFn fd_unset_fn, GalSS_ELBehaviorFn behavior_fn, void *loop_data, void (*loop_data_free_fn)(void *), int timer_is_persistent)

void GalSS_ELRShutdown (GalSS_ELR *elr)

void GalSS_ELRUpdatePollIntervals (GalSS_ELR *elr, int server_client_poll_ms, int conn_ms, int broker_ms)

Gal_TaskPkg *GalSS_EnvAddTask (GalSS_Environment *env, void (*task)(Gal_TaskPkg*), void *caller_data, long num_millisecs, int read_blocking_available, GAL_SOCKET *read_socket, GAL_SOCKET *write_socket, GAL_SOCKET *err_socket, FILE *read_file, FILE *write_file, FILE*err_file, Gal_TaskConditionFn condition, void (*cleanup_fn)(void *))

GalIO_BrokerStruct *GalSS_EnvBrokerDataInInit (GalSS_Environment *env, char *host,unsigned short port, Gal_Frame frame, GalIO_BrokerDataHandler fnptr, int poll_ms, void *refptr,void (*free_fn)(void *))

GalIO_CommStruct *GalSS_EnvComm (GalSS_Environment *env)

GalSS_Environment *GalSS_EnvCreate (GalIO_CommStruct *gcomm)

void GalSS_EnvDeleteServerProperties (GalSS_Environment *env, char **keys)

void GalSS_EnvDeleteSessionProperties (GalSS_Environment *env, char **keys)

int GalSS_EnvDestroyToken (GalSS_Environment *env)

344

Gal_Frame GalSS_EnvDispatchFrame (GalSS_Environment *env, Gal_Frame frame,GalIO_MsgType *t)

int GalSS_EnvDispatchFrameWithContinuation (GalSS_Environment *env,Gal_Frame frame, GalSS_ContinuationFn fn, void *continuation_state, void (*continuation_state_free_fn)(void *))

int GalSS_EnvError (GalSS_Environment *env, char *description)

int GalSS_EnvErrorOfType (GalSS_Environment *env, int type, char *description)

void *GalSS_EnvGetClientData (GalSS_Environment *env, char *name)

void *GalSS_EnvGetCommData (GalSS_Environment *env)

Gal_Frame GalSS_EnvGetServerProperties (GalSS_Environment *env, char **keys)

char *GalSS_EnvGetSessionID (GalSS_Environment *env)

Gal_Frame GalSS_EnvGetSessionProperties (GalSS_Environment *env, char **keys)

void GalSS_EnvLock (GalSS_Environment *env)

void GalSS_EnvMaintainInLocation (GalIO_CommStruct *gcomm, char *initial_session_id, GalSS_Environment **env_loc)

void GalSS_EnvModifyServerProperties (GalSS_Environment *env, Gal_Frame properties_to_set, char **properties_to_delete)

void GalSS_EnvModifySessionProperties (GalSS_Environment *env, Gal_Frame properties_to_set, char **properties_to_delete)

int GalSS_EnvPostponeReply (GalSS_Environment *env)

int GalSS_EnvReply (GalSS_Environment *env, Gal_Frame f)

int GalSS_EnvReturnRequired (GalSS_Environment *env)

void GalSS_EnvSetCommData (GalSS_Environment *env, void *data, void (*free_fn)(void *))

void GalSS_EnvSetServerProperties (GalSS_Environment *env, Gal_Frame properties)

void GalSS_EnvSetSession (GalSS_Environment *env, char *session_name, int lock_info)

345

void GalSS_EnvSetSessionProperties (GalSS_Environment *env, Gal_Frame properties)

void GalSS_EnvUnlock (GalSS_Environment *env)

void GalSS_EnvUpdateSessionID (GalSS_Environment *env, char *session_id)

int GalSS_EnvWriteFrame (GalSS_Environment *env, Gal_Frame frame, int do_block)

GalSS_ServerArgs *GalSS_ExtractCmdlineServerArgs (GalSS_ServerArgs *arg_pkg,int argc, char **argv, int *new_argc_ptr, char ***new_argv_ptr)

GalSS_ServerArgs *GalSS_ExtractServerArgs (int argc, char **argv, int *new_argc_ptr,char ***new_argv_ptr)

GalSS_ServerLocationEntry *GalSS_FindServerLocationEntry (char *filename, char *server_name)

int GalSS_FrameHandler (GalIO_CommStruct *gcomm, Gal_Frame frame)

void GalSS_FreeArgPkg (GalSS_ServerArgs *arg_pkg)

GalIO_ServerStruct *GalSS_InitializeServer (unsigned short server_port, int max_conns,int use_color, int do_assert, int use_ttloop, int validate, int new_argc, char **new_argv)

void GalSS_InitializeServerDefaults (GalIO_ServerStruct *gcomm, char *name, unsignedshort port)

GalIO_ServerStruct *GalSS_InitializeServerFromServerArgs (GalSS_ServerArgs *arg_pkg, int new_argc, char **new_argv)

GalIO_ServerStruct *GalSS_InitializeServerToplevel (unsigned short server_port, int max_conns, int use_color, int do_assert, int loop_type, int validate, int verbosity, int server_listen_status, char *client_pair_string, char *session_id, int new_argc, char **new_argv)

void GalSS_RunServer (GalIO_ServerStruct *server)

int GalSS_SAFixAssert (GalSS_ServerArgs *arg_pkg, int assert)

int GalSS_SAFixColor (GalSS_ServerArgs *arg_pkg, int color)

char *GalSS_SAFixContactHubInfo (GalSS_ServerArgs *arg_pkg, char *client_pair_status, char *session_id, char **old_session_id_ptr)

346

int GalSS_SAFixLoopType (GalSS_ServerArgs *arg_pkg, int loop_type)

int GalSS_SAFixMaxConns (GalSS_ServerArgs *arg_pkg, int max_conns)

unsigned short GalSS_SAFixPort (GalSS_ServerArgs *arg_pkg, unsigned short port)

int GalSS_SAFixServerListenStatus (GalSS_ServerArgs *arg_pkg, int server_listen_status)

char *GalSS_SAFixServerLocations (GalSS_ServerArgs *arg_pkg, char *server_locations_file)

int GalSS_SAFixValidate (GalSS_ServerArgs *arg_pkg, int validate)

int GalSS_SAFixVerbosity (GalSS_ServerArgs *arg_pkg, int verbosity)

int GalSS_SessionDecodeLocks (Gal_Frame fr, int *mask_ptr, int *value_ptr)

GalIO_ServerStruct *GalSS_SetupServer (GalSS_ServerArgs *arg_pkg, int new_argc, char **new_argv)

void GalSS_StartAndRunServer (GalIO_ServerStruct *server)

GalSS_Environment *GalSS_TaskGetEnvironment (Gal_TaskPkg *p)

void GalSS_TaskSetEnvironment (Gal_TaskPkg *p, GalSS_Environment *env)

void GalUtil_Assert (int truth, char *format, ... )

void GalUtil_CPInfo1 (int fore, int back, char *format, ... )

void GalUtil_CPInfo2 (int fore, int back, char *format, ... )

void GalUtil_CPPFrame (int gal_verbose_level, int fore, int back, Gal_Frame fr)

void GalUtil_CPPObject (int gal_verbose_level, int fore, int back, Gal_Object to)

void GalUtil_CPrint (int level, int fore, int back, char *format, ... )

void GalUtil_ConditionBroadcast (GalUtil_ConditionVar *condVar)

void GalUtil_ConditionWait (GalUtil_ConditionVar *condVar, GalUtil_LocalMutex *mutex)

GalUtil_LocalMutex*GalUtil_CreateLocalMutex (void )

347

void GalUtil_Debug1 (char *format, ... )

void GalUtil_Debug2 (char *format, ... )

void GalUtil_DestroyConditionVar (GalUtil_ConditionVar *condVar)

void GalUtil_DestroyLocalMutex (GalUtil_LocalMutex *mutex)

void GalUtil_Error (char *format, ... )

void GalUtil_Fatal (char *format, ... )

int GalUtil_IP4Address (unsigned long *inaddr)

int GalUtil_IPFromHostname (char *hostname)

void GalUtil_InitConditionVar (GalUtil_ConditionVar *condVar)

void GalUtil_InitMainWin32Thread ()

int GalUtil_InitVerbose ()

void GalUtil_MilliSleep (int msecs)

int GalUtil_Mkdirp (char *pathname, mode_t mode)

int GalUtil_OACheckUsage (int argc, char **argv, char **oas, int *first_real_arg)

GalUtil_OASDescription *GalUtil_OADigest (char **oas)

int GalUtil_OAExtract (int argc, char **argv, char **oas, char *key, ... )

void GalUtil_OAExtractAsserting (int argc, char **argv, char **oas, char *key, ... )

void GalUtil_OAPrintOptions (int argc, char **argv, char **oas, char *tag)

void GalUtil_OAPrintUsage (int argc, char **argv, char **oas)

int GalUtil_OASplitArgs (int argc, char **argv, char **oas, int *first_real_arg, int *in_argc,char ***in_argv, int *ex_argc, char ***ex_argv)

int GalUtil_OAUiCheckUsage (char *progname, int argc, char **argv, char **oas, int *first_real_arg)

348

int GalUtil_OAUiExtract (int argc, char **argv, char **oas, char *key, ... )

void GalUtil_OAUiExtractAsserting (int argc, char **argv, char **oas, char *key, ... )

void GalUtil_OAUiPrintUsage (char *progname, int argc, char **argv, char **oas)

void GalUtil_PInfo1 (char *format, ... )

void GalUtil_PInfo2 (char *format, ... )

void GalUtil_PPFrame (int gal_verbose_level, Gal_Frame fr)

void GalUtil_PPObject (int gal_verbose_level, Gal_Object to)

void GalUtil_PrObject (int gal_verbose_level, Gal_Object to)

void GalUtil_Print (int level, char *format, ... )

void GalUtil_PrintObject (int gal_verbose_level, Gal_Object to, int how)

int GalUtil_RemoveOASArg (GalUtil_OASDescription *descr, char *key)

int GalUtil_SAExtract (int argc, char **argv, char *key, int num, ... )

int GalUtil_SetVerbose (int verbose_level)

int GalUtil_SockBlockOff (GAL_SOCKET sock)

int GalUtil_SockBlockOn (GAL_SOCKET sock)

long GalUtil_SockBytesWaiting (GAL_SOCKET sock)

int GalUtil_SockCanRead (GAL_SOCKET sock)

int GalUtil_SockCanWrite (GAL_SOCKET sock)

int GalUtil_SockClose (GAL_SOCKET sock)

GAL_SOCKET GalUtil_SockCreateClient (char *host, unsigned short port)

GAL_SOCKET GalUtil_SockCreateServer (unsigned short port, int n_listen)

int GalUtil_SockSelect (int maxfd, fd_set *read_fdset, fd_set *write_fdset, fd_set *err_fdset,struct timeval *tp, int *interrupted)

349

GAL_SOCKET GalUtil_SockServerAcceptConnection (GAL_SOCKET sock, char *host)

int GalUtil_SockServerBlockingCheckForConnection (GAL_SOCKET sock, char *host, int *client_sock)

int GalUtil_SockServerCheckForConnection (GAL_SOCKET sock, char *host, int *client_sock)

int GalUtil_SockServerHostnameToInaddr (char *hostname, unsigned long *inaddr)

void GalUtil_SockServerInaddrToString (unsigned long inaddr, char *s)

int GalUtil_ThreadCancel (GalUtil_ThreadID tid)

void GalUtil_ThreadCreate (GalUtil_ThreadID *tid, void*(*func)(void*), void *arg)

void GalUtil_ThreadExit ()

int GalUtil_ThreadPopCleanup (int execute)

void GalUtil_ThreadPushCleanup (void (*handler)(void*), void *arg)

void GalUtil_ThreadTestCancel ()

int GalUtil_TryLockLocalMutex (GalUtil_LocalMutex *mutex)

void GalUtil_VerboseUseBW (void )

void GalUtil_VerboseUseColor (void )

void GalUtil_Warn (char *format, ... )

Gal_DispatchFnPkg *Gal_AddDispatchFunctionEntry (Gal_DispatchFnPkg *table, char *name, Gal_FrameFnPtr fn, Gal_FrameDataFnPtr fn_with_data,Gal_DispatchFnSignatureKeyEntry *in_key_array, int allow_other_in_keys, int reply_provided,Gal_DispatchFnSignatureKeyEntry *out_key_array, int allow_other_out_keys)

void Gal_AddErrorTags (Gal_TagMap *tmap)

int Gal_AddIdleFunction (Gal_IdleFunction func, void *client_data)

Gal_Frame Gal_AddPred (Gal_Frame fr, Gal_Frame pred)

350

void Gal_AddProgramTags (Gal_TagMap *tmap)

void Gal_AddSignalHandler (int sig, void (*handler)(int ))

Gal_TaskPkg *Gal_AddTask (void (*task)(Gal_TaskPkg *), void *caller_data, long num_millisecs, int read_blocking_available, void (*cleanup_fn)(void *))

Gal_TaskPkg *Gal_AddTaskExtended (void (*task)(Gal_TaskPkg *), void *caller_data, long num_millisecs, int read_blocking_available, GAL_SOCKET *read_socket, GAL_SOCKET *write_socket, GAL_SOCKET *err_socket, FILE *read_file, FILE *write_file, FILE*err_file,Gal_TaskConditionFn condition, void (*cleanup_fn)(void *))

Gal_TaskPkg *Gal_AddTaskWithFileIO (void (*task)(Gal_TaskPkg *), void *caller_data,long num_millisecs, int read_blocking_available, FILE *read_file, FILE *write_file, void (*cleanup_fn)(void *))

Gal_TaskPkg *Gal_AddTaskWithSocketIO (void (*task)(Gal_TaskPkg *), void *caller_data, long num_millisecs, int read_blocking_available, GAL_SOCKET *read_socket,GAL_SOCKET *write_socket, void (*cleanup_fn)(void *))

int Gal_AddTimedTask (void *task, void *caller_data, long num_millisecs)

int Gal_AddTimedTaskWithFileIO (void *task, void *caller_data, long num_millisecs, FILE *read_file, FILE *write_file)

int Gal_AddTimedTaskWithSocketIO (void *task, void *caller_data, long num_millisecs,GAL_SOCKET *read_socket, GAL_SOCKET *write_socket)

void Gal_AddToTagArray (Gal_TagMap *tag_map, Gal_TagArray tag_array)

void Gal_AddToTagTable (Gal_TagMap *tag_map, Gal_HashTable ht)

int Gal_ArrayObjectAdd (Gal_Object obj, void *data, int size)

Gal_Object Gal_BinaryObject (void *data, int size)

int Gal_BinarySize (Gal_Object to)

void *Gal_BinaryValue (Gal_Object obj, int *size)

int Gal_Binaryp (Gal_Object to)

int Gal_ByteBufferAppend (Gal_StringBuffer *b, char *s, int size)

351

char *Gal_ByteBufferBytes (Gal_StringBuffer *buf)

int Gal_ByteBufferSize (Gal_StringBuffer *buf)

int Gal_ClauseFramep (Gal_Frame fr)

Gal_Frame Gal_ClauseValue (Gal_Object to)

int Gal_Clausep (Gal_Object to)

void Gal_ClearHash (Gal_HashTable hp)

void Gal_ClearPreds (Gal_Frame fr)

void Gal_CloseControlFile (Gal_FileStruct *fs)

Gal_DispatchFnSignatureKeyEntry *Gal_CopyDispatchFnKeyArray (Gal_DispatchFnSignatureKeyEntry *array)

Gal_Frame Gal_CopyFrame (Gal_Frame fr)

Gal_HashTable Gal_CopyHash (Gal_HashTable old_ht)

Gal_Object Gal_CopyObject (Gal_Object to)

Gal_Object Gal_CreateBinaryObject (void *data, int size, int manage_memory)

Gal_DispatchFnSignatureKeyEntry *Gal_CreateDispatchFnKeyArray (int ignore, ... )

Gal_DispatchFnSignature *Gal_CreateDispatchFnSignature (char *name,Gal_DispatchFnSignatureKeyEntry *in_key_array, int allow_other_in_keys, int reply_provided,Gal_DispatchFnSignatureKeyEntry *out_key_array, int allow_other_out_keys)

Gal_Object Gal_CreateFloat32Object (void *data, int num_float_32, int manage_memory)

Gal_Object Gal_CreateFloat64Object (void *data, int num_float_64, int manage_memory)

Gal_Object Gal_CreateFloatObject (float *value, int manage_memory)

Gal_Object Gal_CreateFrameObject (Gal_Frame value, int manage_memory)

Gal_Object Gal_CreateInt16Object (void *data, int num_int_16, int manage_memory)


352


Gal_Object Gal_CreateListObject (Gal_Object *values, int n, void (*free_fn)(void *), int manage_memory)

Gal_ProgramEntity *Gal_CreateNamespaceProgramEntity (char *key,Gal_ProgramParser *pp)

Gal_ProgramEntity *Gal_CreateProgramEntity (ProgramObject **pptr_ptr,Gal_ProgramParser *pp, int allow_complex, int allow_literal, int allow_key)

Gal_Object Gal_CreateStringObject (char *cp, int manage_memory)

Gal_Object Gal_CreateTokenObject (char *cp, int manage_memory)

Gal_VarMapping *Gal_CreateVarMapping (int num_pairs, ... )

int Gal_DataBufferAppend (Gal_StringBuffer *b, char *s, int size)

int Gal_DataBufferByteCount (Gal_StringBuffer *buf)

char *Gal_DataBufferData (Gal_StringBuffer *buf)

int Gal_DataBufferSize (Gal_StringBuffer *buf)

Gal_DispatchFnSignature *Gal_DecodeDispatchFnSignatures (Gal_Object maybe_list)

void Gal_DelPred (Gal_Frame fr, int i)

void Gal_DelPredByName (Gal_Frame fr, char *name)

int Gal_DelProp (Gal_Frame fr, char *key)

void Gal_DeletePreds (Gal_Frame fr, char *pred_name)

void Gal_DeleteProgramEntityLocation (Gal_ProgramEntity *e, Gal_Frame *namespace_array)

int Gal_DigitStringp (char *word)

void *Gal_DispatchFnPkgGetClientData (Gal_DispatchFnPkg *pkg, char *op_name)

void Gal_DispatchFnPkgSetClientData (Gal_DispatchFnPkg *pkg, char *op_name, void *data)

353

void Gal_DoPreds (Gal_Frame fr, int (*pred_fn)(Gal_Object , void *), void *caller_data)

void Gal_DoProperties (Gal_Frame fr, int (*prop_fn)(char *, Gal_Object , void *), void *caller_data)

void Gal_EnableTimedTaskThreads ()

Gal_Object Gal_EncodeDispatchFnPkgSigs (Gal_DispatchFnPkg *pkg)

Gal_Object Gal_EncodeDispatchFnSignatures (Gal_DispatchFnPkg *pkg)

void Gal_EndTasks (int immediate)

char *Gal_ErrorTagString (int tag)

char *Gal_FileCurrentLine (Gal_InputStream gs, char *line, int max)

char *Gal_FileNextLine (Gal_InputStream gs, char *line, int max)

int Gal_FindDispatchFunctionEntry (Gal_DispatchFnInvocation *i)

Gal_Object Gal_FindKey (Gal_Frame fr, char *key_name)

Gal_DispatchFnSignature *Gal_FindNamedSignature (Gal_DispatchFnSignature *sigs,char *name)

Gal_NamespaceEntry *Gal_FindNamespaceEntry (char *name, Gal_NamespaceEntry *ne)

Gal_Frame Gal_FindPred (Gal_Frame fr, char *pred_name)

Gal_Frame Gal_FindPredParent (Gal_Frame frame, char *name, Gal_Frame parent, int findpar, int nth)

Gal_Frame Gal_FindTopic (Gal_Frame fr, char *topic_name)

int Gal_Float32Size (Gal_Object to)

void *Gal_Float32Value (Gal_Object obj, int *size)

int Gal_Float32p (Gal_Object to)

int Gal_Float64Size (Gal_Object to)

354

void *Gal_Float64Value (Gal_Object obj, int *size)

int Gal_Float64p (Gal_Object to)

Gal_Object Gal_FloatObject (float value)

int Gal_FloatStringp (char *word)

float Gal_FloatValue (Gal_Object to)

int Gal_Floatp (Gal_Object to)

char *Gal_FormatEntity (Gal_ProgramEntity *e, int pedantic, Gal_StringBuffer **bufptr)

int Gal_FrameEqual (Gal_Frame sf, Gal_Frame sem)

int Gal_FrameIsType (Gal_Frame fr, Gal_FrameType type)

char *Gal_FrameName (Gal_Frame fr)

int Gal_FrameNameEq (Gal_Frame fr, char *name)

int Gal_FrameNamesEq (Gal_Frame fr1, Gal_Frame fr2)

Gal_Object Gal_FrameObject (Gal_Frame value)

Gal_Frame Gal_FrameValue (Gal_Object to)

int Gal_Framep (Gal_Object to)

void Gal_FreeByteBuffer (Gal_StringBuffer *buf)

void Gal_FreeConditionStruct (Gal_ConditionStruct *cond)

void Gal_FreeDispatchFnKeyArray (Gal_DispatchFnSignatureKeyEntry *entry)

void Gal_FreeDispatchFnSignature (Gal_DispatchFnSignature *sigs)

void Gal_FreeFrame (Gal_Frame fr)

void Gal_FreeHash (Gal_HashTable hp)

void Gal_FreeObject (Gal_Object to)

355

void Gal_FreePointerBuffer (Gal_PointerBuffer *buf)

void Gal_FreeProgramEntity (Gal_ProgramEntity *e)

void Gal_FreeStringBuffer (Gal_StringBuffer *buf)

void Gal_FreeStringOutputStream (Gal_OutputStream gs)

void Gal_FreeTests (Gal_TestClause *c)

void Gal_FreeWrapper (Gal_Object to)

void *Gal_GetBinary (Gal_Frame fr, char *key, int *size)

Gal_ObjectType Gal_GetDetailedType (Gal_Object to)

float Gal_GetFloat (Gal_Frame fr, char *key)

void *Gal_GetFloat32 (Gal_Frame fr, char *key, int *size)

void *Gal_GetFloat64 (Gal_Frame fr, char *key, int *size)

Gal_Frame Gal_GetFrame (Gal_Frame fr, char *key)

Gal_FrameType Gal_GetFrameType (Gal_Frame fr)

Gal_Object Gal_GetHash (char *str, Gal_HashTable hp)

int Gal_GetInt (Gal_Frame fr, char *key)

void *Gal_GetInt16 (Gal_Frame fr, char *key, int *size)



Gal_Object *Gal_GetList (Gal_Frame fr, char *key, int *length)

Gal_Object Gal_GetListObject (Gal_Object to, int n)

void *Gal_GetListValue (Gal_Object to, int n, Gal_ObjectType type)

Gal_Object Gal_GetObject (Gal_Frame fr, char *key)

356

Gal_ObjectType Gal_GetObjectType (Gal_Object to)

char *Gal_GetObjectTypeString (Gal_Object to)

Gal_Frame Gal_GetPred (Gal_Frame fr, int i)

Gal_Frame Gal_GetPredByName (Gal_Frame fr, char *name)

Gal_Object Gal_GetProgramEntity (Gal_ProgramEntity *e, Gal_Frame *namespace_array)

int Gal_GetProgramTag (char *tag_name)

Gal_HashTable Gal_GetProgramTagHash (void )

char **Gal_GetProperties (Gal_Frame fr, int *nkeys)

char *Gal_GetString (Gal_Frame fr, char *key)

char *Gal_GetTagArrayTag (int tag, Gal_TagArray tag_array)

Gal_Object Gal_GetTagObject (char *str)

Gal_HashTable Gal_GetTagObjectHash (void )

Gal_Frame Gal_GetTopicFrame (Gal_Frame fr, char *key)

int Gal_HashHasKey (char *key, Gal_HashTable ht)

char *Gal_HashListNext (Gal_HashTable hp, Gal_Object *value)

void Gal_HashListRewind (Gal_HashTable hp)

void Gal_InitializeProgramParser (Gal_ProgramParser *pp, Gal_InputStream gs,Gal_FileStruct *fs, int default_tag, int control_flags)

void Gal_InitializeSignals ()

void Gal_InitializeStatics (void )

Gal_TagArray Gal_InitializeTagArray (Gal_TagMap *tag_map)

Gal_HashTable Gal_InitializeTagTable (Gal_TagMap *tag_map, int size)

int Gal_InputStreamNextChar (Gal_InputStream gs)

357

void Gal_InputStreamRewind (Gal_InputStream gs, int i)

int Gal_Int16Size (Gal_Object to)

void *Gal_Int16Value (Gal_Object obj, int *size)

int Gal_Int16p (Gal_Object to)







Gal_Object Gal_IntObject (int value)

int Gal_IntValue (Gal_Object to)

int Gal_Intp (Gal_Object to)

int Gal_IsFullPath (char *path)

Gal_Object Gal_KeywordObject (char *value)

char *Gal_KeywordValue (Gal_Object to)

int Gal_Keywordp (Gal_Object to)

int Gal_ListLength (Gal_Object to)

Gal_Object Gal_ListObject (Gal_Object *values, int n)

int Gal_ListObjectAdd (Gal_Object obj, Gal_Object elt)

Gal_Object Gal_ListObjectFromElements (int n, ... )

Gal_Object *Gal_ListValue (Gal_Object obj, int *length)

358

int Gal_Listp (Gal_Object to)

Gal_StringBuffer *Gal_MakeByteBuffer (char *buf, int bufpos, int bufsize, int manage_memory, int dynamic, int increment, int padding)

Gal_Frame Gal_MakeClauseFrame (char *name)

Gal_StringBuffer *Gal_MakeDataBuffer (char *buf, int bufpos, int bufsize, int manage_memory, int dynamic, int increment, int padding, int chunk_size)

Gal_InputStream Gal_MakeFileInputStream (FILE *fp)

Gal_Frame Gal_MakeFrame (char *name, Gal_FrameType type)

Gal_HashTable Gal_MakeHash (int size)

Gal_PointerBuffer *Gal_MakePointerBuffer (void **buf, int pointer_type, int bufpos, int bufsize, int manage_memory, int dynamic, void (*free_fn)(void *), int increment, int padding)

Gal_Frame Gal_MakePredFrame (char *name)

Gal_StringBuffer *Gal_MakeStringBuffer (char *buf, int bufsize)

Gal_InputStream Gal_MakeStringInputStream (char *sp)

Gal_OutputStream Gal_MakeStringOutputStream (char *buf, int bufsize)

Gal_Frame Gal_MakeTopicFrame (char *name)

int Gal_MatchFrame (Gal_Frame sf, Gal_Frame sem)

Gal_Object Gal_MatchKeyValue (Gal_Frame fr, char *key_name, Gal_Object match)

void Gal_MaybeEndTask (int immediate, int deferred)

Gal_ConditionStruct *Gal_NewCondition (ProgramObject *values, Gal_ProgramParser *pp,int *error_ptr)

Gal_TestClause *Gal_NewTest (ProgramObject **values, int *error_ptr, int in_group,Gal_ProgramParser *pp)

char *Gal_NextToken (char *sent, char *tok, int toksize)

int Gal_NumNonNullProperties (Gal_Frame fr)

359

int Gal_NumPreds (Gal_Frame fr)

int Gal_NumProperties (Gal_Frame fr)

int Gal_ObjectByteCount (Gal_Object obj)

int Gal_ObjectCaseEqual (Gal_Object to, Gal_Object sem)

int Gal_ObjectEqual (Gal_Object to, Gal_Object sem)

char *Gal_ObjectString (Gal_Object to, Gal_StringBuffer **buf)

char *Gal_ObjectToString (Gal_Object to)

char *Gal_ObjectTypeString (Gal_ObjectType object_type)

Gal_FileStruct *Gal_OpenControlFile (char *filename, char *directory)

void Gal_OutlineFrame (Gal_Frame fr, int gal_verbose_level)

void Gal_OutlineObject (Gal_Object to, int gal_verbose_level)

void Gal_PPFrame (Gal_Frame fr)

void Gal_PPFrameToFile (Gal_Frame fr, FILE *fp)

char *Gal_PPFrameToString (Gal_Frame fr, char *irpbuf, int *bufsizeptr)

void Gal_PPObject (Gal_Object to)

void Gal_PPObjectToFile (Gal_Object to, FILE *fp)

int Gal_PointerBufferAdd (Gal_PointerBuffer *b, void *s)

int Gal_PointerBufferAddMultiple (Gal_PointerBuffer *b, void **s, int size)

Gal_PointerBuffer *Gal_PointerBufferCopy (Gal_PointerBuffer *b, void *(*copy_fn)(void *),void (*free_fn)(void *))

void *Gal_PointerBufferNthElement (Gal_PointerBuffer *b, int i)

void **Gal_PointerBufferPointers (Gal_PointerBuffer *buf)

void Gal_PointerBufferRemove (Gal_PointerBuffer *b, void *s)

360

int Gal_PointerBufferSize (Gal_PointerBuffer *buf)

Gal_Object Gal_PointerObject (void *val)

void *Gal_PointerValue (Gal_Object to)

int Gal_Pointerp (Gal_Object to)

Gal_FileStruct *Gal_PopProgramFile (Gal_FileStruct *current_files)

void Gal_PrFrame (Gal_Frame fr)

void Gal_PrFrameToFile (Gal_Frame fr, FILE *fp)

char *Gal_PrFrameToString (Gal_Frame fr, char *irpbuf, int *bufsizeptr)

void Gal_PrObject (Gal_Object to)

void Gal_PrObjectToFile (Gal_Object to, FILE *fp)

int Gal_PredFramep (Gal_Frame fr)

Gal_Frame Gal_PredValue (Gal_Object to)

int Gal_Predp (Gal_Object to)

void Gal_PrintCondition (FILE *fp, Gal_ConditionStruct *condition, char *tag, int pedantic)

void Gal_PrintEntity (Gal_ProgramEntity *e, int pedantic)

void Gal_PrintFrameToFile (Gal_Frame fr, FILE *fp, int how)

char *Gal_PrintFrameToString (Gal_Frame fr, char *irpbuf, int *bufsizeptr, int how)

void Gal_PrintProgramError (Gal_FileStruct *fs, int error)

void Gal_PrintProgramObject (ProgramObject *obj)

void Gal_PrintProgramWarning (Gal_FileStruct *fs, int error)

void Gal_PrintTests (FILE *fp, Gal_TestClause *tests, int pedantic)

int Gal_ProgramEntitiesEqual (Gal_ProgramEntity *e1, Gal_ProgramEntity *e2)

361

char *Gal_ProgramStringValue (Gal_Object obj)

int Gal_ProgramStringp (Gal_Object o)

char *Gal_ProgramTagString (int tag)

Gal_FileStruct *Gal_PushProgramFile (char *filename, FILE *fp, Gal_FileStruct *current_files)

void Gal_ReAddTask (Gal_TaskPkg *p, void *caller_data, long num_millisecs, int read_blocking_available, void (*cleanup_fn)(void *))

void Gal_ReAddTaskExtended (Gal_TaskPkg *p, void *caller_data, long num_millisecs,int read_blocking_available, GAL_SOCKET *read_socket, GAL_SOCKET *write_socket,GAL_SOCKET *err_socket, FILE *read_file, FILE *write_file, FILE *err_file,Gal_TaskConditionFn condition, void (*cleanup_fn)(void *))

void Gal_ReAddTaskWithFileIO (Gal_TaskPkg *p, void *caller_data, long num_millisecs,int read_blocking_available, FILE *read_file, FILE *write_file, void (*cleanup_fn)(void *))

void Gal_ReAddTaskWithSocketIO (Gal_TaskPkg *p, void *caller_data, long num_millisecs, int read_blocking_available, GAL_SOCKET *read_socket, GAL_SOCKET *write_socket, void (*cleanup_fn)(void *))

Gal_Frame Gal_ReadFrameFromFile (FILE *fp)

Gal_Frame Gal_ReadFrameFromString (char *buf)

Gal_Frame Gal_ReadNextFrameFromFile (FILE *fp, int bol)

Gal_Object Gal_ReadObjectFromFile (FILE *fp)

Gal_Object Gal_ReadObjectFromString (char *buf)

ProgramObject *Gal_ReadProgramLine (Gal_ProgramParser *pp, int *count_ptr)

char *Gal_ReadToken (Gal_InputStream gs, char *tok, int toksize, char *stop_chars, int do_rewind)

Gal_Frame Gal_ReadVarFrameFromString (char *buf, Gal_VarMapping *map)

Gal_Object Gal_RemPred (Gal_Frame fr, int i)

Gal_Object Gal_RemPredByName (Gal_Frame fr, char *name)

362

Gal_Object Gal_RemProp (Gal_Frame fr, char *key)

void Gal_RemoveIdleFunction (Gal_IdleFunction func)

void Gal_RemoveTask (Gal_TaskPkg *task_id)

int Gal_RemoveTimedTask (void *task, void *caller_data)

void Gal_RunIdleFunctions ()

Gal_Frame Gal_SetFrameName (Gal_Frame fr, char *name)

Gal_Frame Gal_SetFrameType (Gal_Frame fr, Gal_FrameType type)

Gal_Object Gal_SetHash (char *str, Gal_Object val, Gal_HashTable hp)

void Gal_SetProgramEntityLocation (Gal_ProgramEntity *e, Gal_Object o, Gal_Frame *namespace_array)

Gal_Object Gal_SetProp (Gal_Frame fr, char *key, Gal_Object val)

void Gal_SetTagObject (char *key, Gal_Object value)

void Gal_SetTagObjectHash (Gal_HashTable ht)

int Gal_SignalsInitialized ()

char *Gal_SplitLocation (char *location_string, int *port)

char *Gal_SplitOperationName (char *op_name, char **server_name)

char *Gal_SplitServerName (char *server_string, char **hostname, int *port)

char *Gal_StringBufferString (Gal_StringBuffer *buf)

int Gal_StringBufferWrite (Gal_StringBuffer *b, int increment, char *s, ... )

int Gal_StringBufferWriteString (Gal_StringBuffer *b, char *s)

int Gal_StringCaseEq (char *str1, char *str2)

int Gal_StringEq (char *str1, char *str2)

char *Gal_StringInputStreamString (Gal_InputStream gs)

363

void Gal_StringLeftTrim (char *str, char *trim)

Gal_Object Gal_StringObject (char *val)

char *Gal_StringOutputStreamString (Gal_OutputStream gs)

void Gal_StringRightTrim (char *str, char *trim)

int Gal_StringStreamWriteString (Gal_OutputStream gs, int increment, char *s, ... )

void Gal_StringTrim (char *str, char *trim)

char *Gal_StringValue (Gal_Object to)

int Gal_Stringp (Gal_Object to)

Gal_Object Gal_SymbolObject (char *value)

int Gal_Symbolp (Gal_Object to)

Gal_Object Gal_TagObject (char *value)

int Gal_Tagp (Gal_Object to)

int Gal_TaskPkgBlocking (Gal_TaskPkg *pkg)

void *Gal_TaskPkgData (Gal_TaskPkg *pkg)

int Gal_TaskPkgRunReasons (Gal_TaskPkg *pkg)

Gal_Boolean Gal_TestCondition (Gal_Frame frame, Gal_TestClause *tests)

Gal_Boolean Gal_TestConditionInNamespaces (Gal_TestClause *tests, Gal_Frame *frame_array, int default_namespace)

void Gal_TimedTaskLoopThreadWaiter (void )

int Gal_TimedTaskThreadsEnabled ()

void Gal_TimedTasksLoop (void )

void Gal_TimedTasksLoopExit (void )

int Gal_TimedTasksLoopHandler (struct timeval *tv)

364

Gal_Object Gal_TokenObject (char *val)

int Gal_Tokenp (Gal_Object to)

int Gal_TopicFramep (Gal_Frame fr)

Gal_Frame Gal_TopicValue (Gal_Object to)

int Gal_Topicp (Gal_Object to)

void Gal_UpdateFrameProperties (Gal_Frame target_frame, Gal_Frame source_properties, char **delete_properties)

Gal_Frame Gal_VAReadVarFrameFromString (char *buf, int num_pairs, ... )

void Gal_ValidateDispatchFnInput (char *op_name, Gal_Frame frame,Gal_DispatchFnSignature *sig, char **exclusions)

void Gal_ValidateDispatchFnOutput (char *op_name, Gal_Frame frame,Gal_DispatchFnSignature *sig, char **exclusions)

void _GalIO_SetCommDispatchFnAccess (GalIO_CommStruct *gcomm,Gal_DispatchFunctionSelector s, Gal_DispatchFnSignatureLister l,Gal_DispatchFunctionInvoker i, void *invocation_client_data)

void _GalIO_SetServerDispatchFnAccess (GalIO_ServerStruct *server,Gal_DispatchFunctionSelector s, Gal_DispatchFnSignatureLister l,Gal_DispatchFunctionInvoker i, void *invocation_client_data)

void *_GalSS_init_server (GalIO_ServerStruct *server, int argc, char **argv)

void _GalSS_print_usage (int argc, char **argv)

void _GalUtil_FreeOASDescription (GalUtil_OASDescription *descr)

Gal_DispatchFnSignatureKeyEntry *_Gal_CreateEmptyDispatchFnKeyArray (int i)

Gal_DispatchFnPkg *_Gal_DispatchFnPkgSetAccess (Gal_DispatchFnPkg *pkg,Gal_DispatchFunctionSelector s, Gal_DispatchFnSignatureLister l,Gal_DispatchFunctionInvoker i, void *invocation_client_data)

Gal_Frame _Gal_InvokeDispatchFn (Gal_DispatchFnInvocation *i)

Gal_DispatchFnSignature *_Gal_ListDispatchFnSignatures (Gal_DispatchFnPkg *pkg)

365

void _Gal_PopulateDispatchFnKeyArrayCell (Gal_DispatchFnSignatureKeyEntry *array, int index, char *key, Gal_ObjectType t, int obligatory)

Gal_Object _Gal_ReadObjectWithTags (Gal_InputStream gs, Gal_SpecialTokenizer special_tokenizer)

void *_gal_copy_object (void *obj)

void _gal_free_object (void *obj)


366

Core Library

This documentation briefly describes functions in the directory src/libGalaxy/galaxy which arepresent but probably will not be used by the developer. The signatures of these functions maychange from release to release.

String Utility Functions Print Utility Functions Program File Parsing Functions Stream Functions Buffer Functions Frame Reading Functions Hash Table Utilities Dispatch Function Utilities Random utilities

String Utility Functionsint Gal_DigitStringp (char *word) Returns 1 if the string, word, represents an integer.

int Gal_FloatStringp (char *word) Returns 1 if the string word represents a float.

void Gal_StringRightTrim (char *str, char *trim) Removes characters in trim from the right edge of str, inserting a null character at theappropriate place.

void Gal_StringLeftTrim (char *str, char *trim) Removes characters in trim from the left edge of str, updating the underlying pointer.

367

void Gal_StringTrim (char *str, char *trim) Removes characters in trim from both edges of str using Gal_StringLeftTrim and Gal_StringRightTrim.

int Gal_StringEq (char *str1, char *str2); Returns 1 if the strings, str1 and str2, are equivalent (using strcmp). Otherwise, 0 is returned.

int Gal_StringCaseEq (char *str1, char *str2); Returns 1 if the strings, str1 and str2, are of the same case. Otherwise, 0 is returned.

Print Utility Functionschar *Gal_ObjectString (Gal_Object to, Gal_StringBuffer **buf) A static version of Gal_ObjectToString. Also returns strings corresponding to keyword, symboland tag objects.

Program File Parsing FunctionsPart of program files which is new in 1.3 is the notion of a "tag", which is a distinguished symbolsuch as RULE: or destroy!. Tags can be used for errors, directives in Hub programs, or other uses.

char *Gal_SplitOperationName (char *op_name, char **server_name)

char *Gal_SplitServerName (char *server_string, char **hostname, int *port)

char *Gal_SplitLocation (char *location_string, int *port)

Gal_DispatchFnSignature *Gal_FindNamedSignature (Gal_DispatchFnSignature *sigs,char *name)

void Gal_SetTagObjectHash (Gal_HashTable ht) Internal function to establish the hash table for tag objects.

Gal_HashTable Gal_GetTagObjectHash (void ) Internal function to retrieve the tag object hash table.

Gal_Object Gal_GetTagObject (char *str) Internal function to retrieve a tag object for str. Depends on Gal_SetTagObject having beencalled on the string str, so it is not yet really used.

void Gal_SetTagObject (char *key, Gal_Object value) Internal function to store the tag object for a given string. Not yet used.

368

Gal_HashTable Gal_InitializeTagTable (Gal_TagMap *map, int size) Initializes the hash table mapping from tag names to tag integers.

void Gal_AddToTagTable (Gal_TagMap *map, Gal_HashTable ht) Adds a set of pairs to the hash table.

Gal_TagArray Gal_InitializeTagArray (Gal_TagMap *map) Initializes the mapping from tag integers to tag names.

void Gal_AddToTagArray (Gal_TagMap *map, Gal_TagArray tag_array) Adds a set of pairs to the tag array mapping.

char *Gal_GetTagArrayTag (int tag, Gal_TagArray tag_array) Retrievs the tag name corresponding to a given tag integer.

void Gal_AddErrorTags (Gal_TagMap *map)

char *Gal_ErrorTagString (int tag)

void Gal_AddProgramTags (Gal_TagMap *map)

char *Gal_ProgramTagString (int tag)

int Gal_GetProgramTag (char *tag_name)

Gal_HashTable Gal_GetProgramTagHash (void )

Gal_FileStruct *Gal_PushProgramFile (char *filename, FILE *fp, Gal_FileStruct *current_files)

Gal_FileStruct *Gal_PopProgramFile (Gal_FileStruct *current_files)

char *Gal_ProgramStringValue (Gal_Object obj)

int Gal_ProgramStringp (Gal_Object o)

Gal_Object Gal_GetProgramEntity (Gal_ProgramEntity *e, Gal_Frame *namespace_array)

void Gal_SetProgramEntityLocation (Gal_ProgramEntity *e, Gal_Object o, Gal_Frame *namespace_array)

void Gal_DeleteProgramEntityLocation (Gal_ProgramEntity *e, Gal_Frame *namespace_array)

369

void Gal_InitializeProgramParser (Gal_ProgramParser *pp, Gal_InputStream gs,Gal_FileStruct *fs, int default_tag, int control_flags)

Gal_Boolean Gal_TestConditionInNamespaces (Gal_TestClause *tests, Gal_Frame *frame_array, int default_namespace)

Gal_Object _Gal_ReadObjectWithTags (Gal_InputStream gs, Gal_SpecialTokenizer special_tokenizer)

int Gal_IsFullPath (char *path)

Gal_ConditionStruct *Gal_NewCondition (ProgramObject *values, Gal_ProgramParser *pp,int *error_ptr)

ProgramObject *Gal_ReadProgramLine (Gal_ProgramParser *pp, int *count_ptr)

void Gal_PrintProgramWarning (Gal_FileStruct *fs, int error)

void Gal_PrintProgramError (Gal_FileStruct *fs, int error)

void Gal_PrintProgramObject (ProgramObject *obj)

Gal_FileStruct *Gal_OpenControlFile (char *filename, char *directory)

void Gal_CloseControlFile (Gal_FileStruct *fs)

void Gal_PrintTests (FILE *fp, Gal_TestClause *tests, int pedantic)

void Gal_PrintCondition (FILE *fp, Gal_ConditionStruct *condition, char *tag, int pedantic)

Gal_Boolean Gal_TestCondition (Gal_Frame frame, Gal_TestClause *tests)

The types GAL_PTR, GAL_KEYWORD, GAL_TAG and GAL_TOKEN are used in the programfile parser.

Gal_Object Gal_PointerObject (void *val); Converts the value, val, into a GAL_PTR object. This object is used internally in hash tables todeal with objects in program file parsing.

Memory management

Creation: the Gal_Object is allocated from an internal store of Gal_Objects. Freeing: the Gal_Object is returned to the internal store of Gal_Objects. Copying: a new object is created using Gal_PointerObject.

370

void *Gal_PointerValue (Gal_Object to) Extracts a void * from the object to. If the object is not of type GAL_PTR, a warning is printedand NULL is returned.

int Gal_Pointerp (Gal_Object to) Returns 1 if the object, obj, is a void *, of type GAL_PTR.

Gal_Object Gal_KeywordObject (char *value) char *Gal_KeywordValue (Gal_Object to) int Gal_Keywordp (Gal_Object to) Returns 1 if the object, obj, is a symbol, of type GAL_KEYWORD.

Gal_Object Gal_TagObject (char *value) int Gal_Tagp (Gal_Object to) Returns 1 if the object, obj, is a symbol, of type GAL_TAG.

To access the value of this object use, Gal_KeywordValue.

Gal_Object Gal_TokenObject (char *val) Gal_Object Gal_CreateTokenObject (char *cp, int manage_memory) int Gal_Tokenp (Gal_Object to) Returns 1 if the object, obj, is a symbol, of type GAL_TOKEN.

to access the value of this object, use Gal_StringValue.

Gal_Object Gal_SymbolObject (char *value) int Gal_Symbolp (Gal_Object to); Returns 1 if the object, obj, is a symbol, of type GAL_SYMBOL.

To access the value of this object use, Gal_KeywordValue.

Gal_TestClause *Gal_NewTest (ProgramObject **values, int *error_ptr, int in_group,Gal_ProgramParser *pp)

Gal_ProgramEntity *Gal_CreateProgramEntity (ProgramObject **pptr_ptr,Gal_ProgramParser *pp, int allow_complex, int allow_literal, int allow_key)

int Gal_ProgramEntitiesEqual (Gal_ProgramEntity *e1, Gal_ProgramEntity *e2)

Gal_NamespaceEntry *Gal_FindNamespaceEntry (char *name, Gal_NamespaceEntry *ne)

Gal_ProgramEntity *Gal_CreateNamespaceProgramEntity (char *key,Gal_ProgramParser *pp)

371

char *Gal_FormatEntity (Gal_ProgramEntity *e, int pedantic, Gal_StringBuffer **bufptr)

void Gal_PrintEntity (Gal_ProgramEntity *e, int pedantic)

void Gal_FreeConditionStruct (Gal_ConditionStruct *cond)

void Gal_FreeTests (Gal_TestClause *c)

void Gal_FreeProgramEntity (Gal_ProgramEntity *e)

Stream FunctionsGal_InputStream Gal_MakeStringInputStream (char *s) Creates a stream from a string.

Memory management

The string is not copied.

int Gal_InputStreamNextChar (Gal_InputStream gs) Returns the next char from an input stream.

void Gal_InputStreamRewind (Gal_InputStream gs, int i) Rewinds the input stream by count.

char *Gal_StringInputStreamString (Gal_InputStream gs) Returns a pointer into the string used to create the string stream.

Gal_InputStream Gal_MakeFileInputStream (FILE *fp) Creates a stream from a file pointer.

char *Gal_FileNextLine (Gal_InputStream gs, char *line, int max) Returns the next line from a file stream, or NULL if the stream is not a file stream.

char *Gal_FileCurrentLine (Gal_InputStream gs, char *line, int max) Returns the current line from a file stream, or NULL if the stream is not a file stream.

Gal_OutputStream Gal_MakeStringOutputStream (char *buf, int bufsize) Creates an output stream with a string as its target. It uses Gal_MakeStringBuffer to construct itsinternal store.

void Gal_FreeStringOutputStream (Gal_OutputStream gs) Frees the output stream if the stream is a string stream. Calls Gal_FreeStringBuffer.

372

char *Gal_StringOutputStreamString (Gal_OutputStream gs) Returns a pointer to the current contents of the string buffer, or NULL if the stream is not a string stream.

int Gal_StringStreamWriteString (Gal_OutputStream gs, int increment, char *s, ... ) Formats s with its arguments to the output stream. It attempts to ensure that there is at least increment bytes open in the internal buffer. If increment is -1, the length of s is used. If thestream is not a string stream, or the string buffer does not have the requested room availableand can’t be expanded, 0 is returned; otherwise, 1 is returned.

Buffer FunctionsGal_StringBuffer *Gal_MakeStringBuffer (char *buf, int bufsize) Returns a string buffer suitable for writing to. If buf is NULL, a dynamically extendable stringbuffer is allocated from an internal store of such buffers. If buf is not NULL, buf is used as abuffer of immutable size bufsize.

void Gal_FreeStringBuffer (Gal_StringBuffer *buf) Frees the string buffer. If the buffer is immutable, the structure is freed; if it is dynamic, the bufferis zeroed and returned to the internal store of string buffers.

char *Gal_StringBufferString (Gal_StringBuffer *buf) Returns the string associated with the string buffer. The internal buffer is not copied.

int Gal_StringBufferWrite (Gal_StringBuffer *b, int increment, char *s, ... ) Formats s with its arguments to the string buffer. It attempts to ensure that there is at least increment bytes open in the internal buffer. If increment is -1, the length of s is used. If the stringbuffer does not have the requested room available and can’t be expanded, 0 is returned;otherwise, 1 is returned.

int Gal_StringBufferWriteString (Gal_StringBuffer *b, char *s) "Safe" version of Gal_StringBufferWrite, where characters which appear to be format directivesbut aren’t (say, stray % characters) aren’t interpreted incorrectly.

Gal_StringBuffer *Gal_MakeByteBuffer (char *buf, int bufpos, int bufsize, int manage_memory, int dynamic, int increment, int padding)

void Gal_FreeByteBuffer (Gal_StringBuffer *buf)

char *Gal_ByteBufferBytes (Gal_StringBuffer *buf)

int Gal_ByteBufferSize (Gal_StringBuffer *buf)

373

int Gal_ByteBufferAppend (Gal_StringBuffer *b, char *s, int size)

Gal_StringBuffer *Gal_MakeDataBuffer (char *buf, int bufpos, int bufsize, int manage_memory, int dynamic, int increment, int padding, int chunk_size)

char *Gal_DataBufferData (Gal_StringBuffer *buf)

int Gal_DataBufferSize (Gal_StringBuffer *buf)

int Gal_DataBufferByteCount (Gal_StringBuffer *buf)

int Gal_DataBufferAppend (Gal_StringBuffer *b, char *s, int size)

Gal_PointerBuffer *Gal_MakePointerBuffer (void **buf, int pointer_type, int bufpos, int bufsize, int manage_memory, int dynamic, void (*free_fn)(void *), int increment, int padding)

void Gal_FreePointerBuffer (Gal_PointerBuffer *buf)

void **Gal_PointerBufferPointers (Gal_PointerBuffer *buf)

int Gal_PointerBufferSize (Gal_PointerBuffer *buf)

int Gal_PointerBufferAddMultiple (Gal_PointerBuffer *b, void **s, int size)

int Gal_PointerBufferAdd (Gal_PointerBuffer *b, void *s)

Gal_PointerBuffer *Gal_PointerBufferCopy (Gal_PointerBuffer *b, void *(*copy_fn)(void *),void (*free_fn)(void *))

void *_gal_copy_object (void *obj)

void *Gal_PointerBufferNthElement (Gal_PointerBuffer *b, int i)

void Gal_PointerBufferRemove (Gal_PointerBuffer *b, void *s)

Frame Reading FunctionsGal_Frame Gal_ReadNextFrameFromFile (FILE *fp, int bol) Identical to Gal_ReadFrameFromFile, but bol controls whether the frame must start at thebeginning of a line or not (so it’s possible to "comment out" a frame by adding a space before theinitial left curly bracket).

374

char *Gal_ReadToken (Gal_InputStream gs, char *tok, int toksize, char *stop_chars, int do_rewind) Reads a token from a stream.

char *Gal_NextToken (char *sent, char *tok, int toksize) Uses Gal_ReadToken to read a token from a string.

Hash Table UtilitiesGal_HashTable Gal_MakeHash (int size); Allocates space for and returns a symbol hash table of size, size. If the operation is notsuccessful, NULL is returned.

Gal_Object Gal_GetHash (char *str, Gal_HashTable hp); Looks up the symbol, str, in the hash table, hp, and returns the value. If the lookup is notsuccessful, NULL is returned.

Gal_Object Gal_SetHash (char *str, Gal_Object val, Gal_HashTable hp); Inserts and returns the object, val, identified by str, into the hash table. If the symbol alreadyexists in the table, the function sets the new value and returns it. If insertion is unsuccessful,NULL is returned.

void Gal_ClearHash (Gal_HashTable hp); Clears the hash table, hp.

void Gal_FreeHash (Gal_HashTable hp) Clears the hash table and frees it.

Gal_HashTable Gal_CopyHash (Gal_HashTable old_ht) Copies the hash table.

void Gal_HashListRewind (Gal_HashTable hp) Initializes a hash table for sequential search.

char *Gal_HashListNext (Gal_HashTable hp, Gal_Object *value) Returns the current key and sets *value to the current value. Returns NULL at the end.

int Gal_HashHasKey (char *key, Gal_HashTable ht) Returns 1 if the hash table has the given key, 0 otherwise.

375

Dispatch Function UtilitiesGal_DispatchFnSignatureKeyEntry *Gal_CopyDispatchFnKeyArray (Gal_DispatchFnSignatureKeyEntry *array)

Gal_DispatchFnPkg *Gal_AddDispatchFunctionEntry (Gal_DispatchFnPkg *table, char *name, Gal_FrameFnPtr fn, Gal_FrameDataFnPtr fn_with_data,Gal_DispatchFnSignatureKeyEntry *in_key_array, int allow_other_in_keys, int reply_provided,Gal_DispatchFnSignatureKeyEntry *out_key_array, int allow_other_out_keys)

int Gal_FindDispatchFunctionEntry (Gal_DispatchFnInvocation *i)

void Gal_ValidateDispatchFnInput (char *op_name, Gal_Frame frame,Gal_DispatchFnSignature *sig, char **exclusions)

void Gal_ValidateDispatchFnOutput (char *op_name, Gal_Frame frame,Gal_DispatchFnSignature *sig, char **exclusions)

Gal_Object Gal_EncodeDispatchFnPkgSigs (Gal_DispatchFnPkg *pkg)

Gal_Object Gal_EncodeDispatchFnSignatures (Gal_DispatchFnPkg *pkg)

Gal_DispatchFnSignature *Gal_DecodeDispatchFnSignatures (Gal_Object maybe_list)

Gal_DispatchFnSignatureKeyEntry *_Gal_CreateEmptyDispatchFnKeyArray (int i)

void _Gal_PopulateDispatchFnKeyArrayCell (Gal_DispatchFnSignatureKeyEntry *array, int index, char *key, Gal_ObjectType t, int obligatory)

void Gal_DispatchFnPkgSetClientData (Gal_DispatchFnPkg *pkg, char *op_name, void *data)

void *Gal_DispatchFnPkgGetClientData (Gal_DispatchFnPkg *pkg, char *op_name)

void Gal_FreeDispatchFnSignature (Gal_DispatchFnSignature *sigs)

Gal_DispatchFnSignature *Gal_CreateDispatchFnSignature (char *name,Gal_DispatchFnSignatureKeyEntry *in_key_array, int allow_other_in_keys, int reply_provided,Gal_DispatchFnSignatureKeyEntry *out_key_array, int allow_other_out_keys)

Gal_DispatchFnPkg *_Gal_DispatchFnPkgSetAccess (Gal_DispatchFnPkg *pkg,Gal_DispatchFunctionSelector s, Gal_DispatchFnSignatureLister l,Gal_DispatchFunctionInvoker i, void *invocation_client_data)

376

Gal_Frame _Gal_InvokeDispatchFn (Gal_DispatchFnInvocation *i)

Gal_DispatchFnSignature *_Gal_ListDispatchFnSignatures (Gal_DispatchFnPkg *pkg)

Random utilitiesvoid Gal_UpdateFrameProperties (Gal_Frame target_frame, Gal_Frame source_properties, char **delete_properties)


377

I/O Library

This documentation briefly describes functions in the directory src/libGalaxy/io which are presentbut probably will not be used by the developer. The signatures of these functions may changefrom release to release.

Brokering functions Comm structure functions Server structure functions Flow of control functions Socket queue functions

Brokering functionsGAL_SOCKET GalIO_GetBrokerSocket (GalIO_BrokerStruct *b) Returns the socket that the broker connection is using to transmit or receive broker data.

GAL_SOCKET GalIO_GetBrokerListenSocket (GalIO_BrokerStruct *b) Returns the socket that the outgoing broker connection is using to listen for connections on.

int GalIO_BrokerDataOutHandler (GalIO_BrokerStruct *b) Once you’ve set up the outgoing broker connection, call this function to poll it. It returns 1 if thethe connection is done, 0 if it is not done, -1 if there is an error. If the result is not 0, you shouldstop polling and call GalIO_DestroyBrokerStruct to clean up.

int GalIO_BrokerDataInHandler (GalIO_BrokerStruct *b) Once you’ve set up the incoming broker connection, call this function to poll it. It returns 1 if thethe connection is done, 0 if it is not done, -1 if there is an error. If the result is not 0, you shouldstop polling and call GalIO_DestroyBrokerStruct to clean up.

378

void GalIO_DestroyBrokerStruct (GalIO_BrokerStruct *b) Destroys the broker structure and terminates the connection.

Comm structure functionsvoid GalIO_DestroyCommStruct (GalIO_CommStruct *gcomm)

GAL_SOCKET GalIO_GetCommSocket (GalIO_CommStruct *gcomm)

void GalIO_CloseCommSocket (GalIO_CommStruct *gcomm)

char *GalIO_GetCommHost (GalIO_CommStruct *gcomm)

Gal_Frame GalIO_GetCommFrame (GalIO_CommStruct *gcomm)

void GalIO_SetCommDone (GalIO_CommStruct *gcomm)

void GalIO_CommFlushOutQueue (GalIO_CommStruct *gcomm)

int GalIO_QueueProcessOut (GalIO_CommStruct *gcomm, int do_block)

int GalIO_CommReadFrame (GalIO_CommStruct *gcomm, Gal_Frame *frame, int do_block)

Gal_Frame GalIO_CommDispatchFrame (GalIO_CommStruct *gcomm, Gal_Frame dispatch, GalIO_MsgType *msg_type_ptr, char *reply_key) This function implements a module-to-module subdialogue with the Hub. It sends the dispatchand waits for a reply. This function corresponds to GalSS_DispatchViaHub, which implementsthis functionality for the default "container". This function requires the reply_key which will notifythe Hub that this a reply is expected; use :hub_reply_requested. The msg_type_ptr is a pointerto one of the GalIO_MsgType values which describe the content of the response.

int GalIO_CommWriteMessage (GalIO_CommStruct *gcomm, Gal_Frame frame,GalIO_MsgType msg_type, int do_block) See the discussion on message types.

int GalIO_CommReadMessage (GalIO_CommStruct *gcomm, Nframe *frame,GalIO_MsgType *msg_type_ptr, int do_block) See the discussion on message types.

void GalIO_AddCommDispatchFunctionEntry (GalIO_CommStruct *gcomm, char *name, Gal_FrameFnPtr fn, Gal_FrameDataFnPtr fn_with_data,Gal_DispatchFnSignatureKeyEntry *in_key_array, int allow_other_in_keys, int reply_provided,Gal_DispatchFnSignatureKeyEntry *out_key_array, int allow_other_out_keys)

379

Gal_DispatchFnPkg *GalIO_GetCommDispatchFnPkg (GalIO_CommStruct *gcomm)

void GalIO_SetCommDispatchFnPkg (GalIO_CommStruct *gcomm, Gal_DispatchFnPkg *pkg)

void _GalIO_SetCommDispatchFnAccess (GalIO_CommStruct *gcomm,Gal_DispatchFunctionSelector s, Gal_DispatchFnSignatureLister l,Gal_DispatchFunctionInvoker i, void *invocation_client_data)

Gal_FrameGalIO_CreateErrorFrame (int errnum, char *error) Used by GalSS_EnvError and other functions to create a well-formed error frame.

GalIO_ServerStruct *GalIO_CommServer (GalIO_CommStruct *gcomm)

char *GalIO_MsgTypeToName (GalIO_MsgType mt)

int GalIO_ConnectionPoll (GalIO_CommStruct *gcomm) This function handles an established connection. It returns 0 if normal processing occurred(either it had no input or it processed the input it had), 1 if the connection is finished and wasterminated, and -1 when the connection encountered an error and was terminated. Theconnection object will have been freed by the time the error is reported, so if you need to recoverany data from the object (such as the socket so that you can shut off a poll), you should do itbefore you call GalIO_ConnectionPoll.

It’s also possible to set up a server which is polled at intervals instead of being activated by filedescriptor input. It also uses these two handlers, but its structure is much simpler. The main loop example also contains an example of this. This alternative example polls both server and all itsconnections, using GalIO_ServerPoll.

void GalIO_SetDisconnectCallback (GalIO_CommStruct *gcomm, void (*disconnect_callback)(GalIO_CommStruct *server_data)) Associates with the connection gcomm a function disconnect_callback which is called when theconnection is shut down. The callback is called with the connection to be shut down as its soleargument. This function should no longer be used; it has been superseded by the event-drivenprogramming model.

Server structure functionsint GalIO_ResetCommSockets (GalIO_ServerStruct *scomm)

GAL_SOCKET GalIO_GetServerListenSocket (GalIO_ServerStruct *scomm)

380

void GalIO_CloseServerListenSocket (GalIO_ServerStruct *scomm)

void GalIO_AddServerDispatchFunctionEntry (GalIO_ServerStruct *scomm, char *name, Gal_FrameFnPtr fn, Gal_FrameDataFnPtr fn_with_data,Gal_DispatchFnSignatureKeyEntry *in_key_array, int allow_other_in_keys, int reply_provided,Gal_DispatchFnSignatureKeyEntry *out_key_array, int allow_other_out_keys)

void _GalIO_SetServerDispatchFnAccess (GalIO_ServerStruct *server,Gal_DispatchFunctionSelector s, Gal_DispatchFnSignatureLister l,Gal_DispatchFunctionInvoker i, void *invocation_client_data)

Gal_DispatchFnPkg *GalIO_GetServerDispatchFnPkg (GalIO_ServerStruct *scomm)

void GalIO_SetServerDispatchFnPkg (GalIO_ServerStruct *scomm, Gal_DispatchFnPkg *pkg)

GalIO_CommStruct *GalIO_AcceptUniqueConnection (GalIO_ServerStruct *scomm)

void GalIO_SetServerListenStatus (GalIO_ServerStruct *scomm, int server_listen_status,char *client_pair_string, char *session_id) Sets the listener status and a variety of other information relevant to the listener status. The client_pair_string has the same format as the -contact_hub command line argument, and thesession ID is the name of the session the connection(s) should be locked to.

int GalIO_ServerHandler (GalIO_ServerStruct *scomm, GalIO_CommStruct **new_conn_ptr) This function listens for connections on the socket initialized by GalSS_InitializeServerToplevel.It returns 1 if a connection has been established, 0 if not, -1 if error. Once the connection hasbeen established, GalIO_ConnectionPoll should be used for further handling. On error, theserver should be marked as done and shut down.

void GalIO_SetServerDone (GalIO_ServerStruct *scomm) Marks the server as done. GalIO_DestroyServerStruct will not actually destroy the server untilit’s marked as done.

void GalIO_DestroyServerStruct (GalIO_ServerStruct *scomm) Destroys a server as soon as it’s marked as done.

int GalIO_ServerPoll (GalIO_ServerStruct *scomm) Polls the server and all connections associated with it. The status returned is the one associatedwith the server.

int GalIO_ServerIsClient (GalIO_ServerStruct *scomm) Returns 1 if the server will contact a Hub listener to set up connections, 0 otherwise.

381

int GalIO_ServerIsListener (GalIO_ServerStruct *scomm) Returns 1 if the server has its own listener which the Hub will contact to set up connections, 0 otherwise.

int GalIO_ServerListensForConnections (GalIO_ServerStruct *s) Returns 1 if the server has its own listener and that listener is listening for "regular" connections(that is, connections which will fire dispatch functions), 0 otherwise.

int GalIO_ServerListensForBrokers (GalIO_ServerStruct *s) Returns 1 if the server has its own listener and that listener is listening for broker connections, 0 otherwise.

Flow of control functionsGalIO_ServerStruct *GalIO_ServerInit (unsigned short port, int require_port,GalIO_FrameHandler fnptr, void *server_data, int poll_ms, int max_connections) This function creates a structure which contains the appropriate information for interacting withthe Hub: the file descriptors, ports, appropriate buffers, and information for callbacks. The port isthe port the server should start on. If require_port is 0, the function will start the server on thefirst port number which is available, starting from port; otherwise, the function will return NULL ifthe port is unavailable. The fnptr is always GalSS_FrameHandler.

The poll_ms indicate how often the state of the server should be checked. If the number is 0 orgreater, this function will set up a timed task to check the state of the server. If the number is 0,the system uses the default (100 ms, currently); positive values are taken to be poll intervals inmilliseconds. If the number is less than 0, no timed task is set up, and the developer isresponsible for polling the system.

Finally, the server_data defines the information which can be retrieved using GalIO_GetCommServerData and GalIO_GetServerData.

GalIO_ServerStruct *GalIO_ServerCreate (unsigned short port, int require_port,GalIO_FrameHandler fnptr, void *server_data, int poll_ms, int max_connections)

GalIO_ServerStruct *GalIO_ServerStart (GalIO_ServerStruct *scomm)

int GalIO_StartListener (GalIO_ServerStruct *scomm, int additional_flags) Ensures that a listener is started. Used internally by the outgoing broker setup.

int GalIO_StartClient (GalIO_ServerStruct *scomm)

GalIO_CommStruct *GalIO_ClientInit (char *host, unsigned short port, Gal_Frame frame,GalIO_FrameHandler fnptr, int poll_ms)

382

intGalIO_OutHandler (GalIO_CommStruct *gcomm)

int GalIO_InHandler (GalIO_CommStruct *gcomm)

int GalIO_HubHandler (GalIO_CommStruct *gcomm, int silent, Gal_Frame *reply)

GalIO_ServerStruct *GalIO_ListenerCreate (unsigned short port, int require_port,GalIO_FrameHandler foo_ptr, void *server_data, int poll_ms, int max_connections)

void GalIO_SetServerDisconnectCallback (GalIO_ServerStruct *scomm, void (*disconnect_callback)(GalIO_CommStruct *server_data))

int GalIO_VerificationHandler (GalIO_CommStruct *gcomm)

GalIO_ServerLocation *GalIO_AddServerLocation (GalIO_ServerStruct *scomm, char *host, unsigned short port, int client_poll_flags)

Socket queue functionsGalIO_SockQueueStruct *GalIO_CreateSockQueue (GAL_SOCKET sockfd, int in_size, int out_size)

void GalIO_DestroySockQueue (GalIO_SockQueueStruct *q)

void GalIO_SetSockQueueComment (GalIO_SockQueueStruct *q, char *comment)

int GalIO_GetSockQueueError (GalIO_SockQueueStruct *q)

int GalIO_SockQueueProcessSend (GalIO_SockQueueStruct *q)

int GalIO_SockQueueProcessReceive (GalIO_SockQueueStruct *q)

int GalIO_SockQueueWrite (GalIO_SockQueueStruct *q, void *data, long len)

int GalIO_SockQueueRead (GalIO_SockQueueStruct *q, char *data, int max_data_len, int *data_len)

int GalIO_SockQueueReadAtom (GalIO_SockQueueStruct *q, char *data, int data_len)


383

Server Stub Library

This documentation briefly describes functions in the directory src/libGalaxy/ServerStub whichare present but probably will not be used by the developer. The signatures of these functionsmay change from release to release.

void GalSS_AddBareDispatchFunction (GalIO_ServerStruct *i, char *name,Gal_FrameFnPtr fn, Gal_DispatchFnSignatureKeyEntry *in_key_array, int allow_other_in_keys,int reply_provided, Gal_DispatchFnSignatureKeyEntry *out_key_array, int allow_other_out_keys)

Gal_Frame GalSS_CallServerFunction (Gal_Frame frame, GalSS_Environment *env, int optional) Calls the server function specified in frame. Both frame and server_data are passed to thefunction. If the function is not optional, a warning is printed if it does not exist. The result frameis returned.

int GalSS_EnvErrorOfType (GalSS_Environment *env, int type, char *description)

int GalSS_FrameHandler (GalIO_CommStruct *gcomm, Gal_Frame frame) This function is always the third argument of GalIO_ServerInit, and it is called with connection asan argument. The frame is a message which will be passed to the appropriate dispatch function.The return value is

1 if the function both evaluated the dispatch function and wrote a reply; 0 if the function evaluated the dispatch function, but no reply was necessary; -1 if an error was encountered and the connection was closed

GalSS_ServerLocationEntry *GalSS_FindServerLocationEntry (char *filename, char *server_name)

GalSS_ServerLocationEntry *GalSS_DigestServerLocationFile (char *filename)

384

int GalSS_SessionDecodeLocks (Gal_Frame fr, int *mask_ptr, int *value_ptr)


385

Utilities Library

This documentation briefly describes functions in the directory src/libGalaxy/util and elsewherewhich are present but probably will not be used by the developer. The signatures of thesefunctions may change from release to release.

Sockets Command line argument parsing Threads Miscellaneous

Socketsint GalUtil_IPFromHostname (char *hostname) Produces an IPv4 address from hostname.

int GalUtil_IP4Address (unsigned long *inaddr) Tries to find a non-loopback IP address (not on Win32).

GAL_SOCKET GalUtil_SockCreateServer (unsigned short port, int n_listen) Creates a server on port with a queue of length n_listen.

GAL_SOCKET GalUtil_SockCreateClient (char *host, unsigned short port) Creates a client connection to host at port.

GAL_SOCKET GalUtil_SockServerAcceptConnection (GAL_SOCKET sock, char *host)

int GalUtil_SockServerCheckForConnection (GAL_SOCKET sock, char *host, int *client_sock)

int GalUtil_SockServerBlockingCheckForConnection (GAL_SOCKET sock, char *host, int *client_sock)

386

void GalUtil_SockServerInaddrToString (unsigned long inaddr, char *s)

int GalUtil_SockServerHostnameToInaddr (char *hostname, unsigned long *inaddr)

int GalUtil_SockClose (GAL_SOCKET sock)

int GalUtil_SockBlockOn (GAL_SOCKET sock)

int GalUtil_SockBlockOff (GAL_SOCKET sock)

int GalUtil_SockCanWrite (GAL_SOCKET sock)

int GalUtil_SockCanRead (GAL_SOCKET sock)

long GalUtil_SockBytesWaiting (GAL_SOCKET sock)

int GalUtil_SockSelect (int maxfd, fd_set *read_fdset, fd_set *write_fdset, fd_set *err_fdset,struct timeval *tp, int *interrupted) Cross-platform select() call. If select() returned because the system call was interrupted by asignal, the integer interrupted points to is set to 1 (if interrupted is not NULL).

Command line argument parsingvoid GalUtil_OAPrintOptions (int argc, char **argv, char **oas, char *tag)

int GalUtil_OASplitArgs (int argc, char **argv, char **oas, int *first_real_arg, int *in_argc,char ***in_argv, int *ex_argc, char ***ex_argv)

int GalUtil_OAUiCheckUsage (char *progname, int argc, char **argv, char **oas, int *first_real_arg)

void GalUtil_OAUiPrintUsage (char *progname, int argc, char **argv, char **oas)

int GalUtil_OAUiExtract (int argc, char **argv, char **oas, char *key, ... )

void GalUtil_OAUiExtractAsserting (int argc, char **argv, char **oas, char *key, ... )

intGalUtil_SAExtract (int argc, char **argv, char *key, int num, ... )

void _GalUtil_FreeOASDescription (GalUtil_OASDescription *descr)

GalUtil_OASDescription *GalUtil_OADigest (char **oas)

387

int GalUtil_RemoveOASArg (GalUtil_OASDescription *descr, char *key)

ThreadsThese functions and macros are intended to provide a common interface to Posix and Win32threads. See the thread notes.

GalUtil_LocalMutex*GalUtil_CreateLocalMutex (void )

void GalUtil_DestroyLocalMutex (GalUtil_LocalMutex *mutex)

void GalUtil_InitLocalMutex (GalUtil_LocalMutex *mutex)

void GalUtil_LockLocalMutex (GalUtil_LocalMutex *mutex)

void GalUtil_UnlockLocalMutex (GalUtil_LocalMutex *mutex)

GalUtil_ThreadID GalUtil_CurrentThreadID ()

int GalUtil_EqualThreadID (GalUtil_ThreadID x, GalUtil_ThreadID y)

void GalUtil_ThreadPushCleanup (void (*handler)(void*), void *arg)

int GalUtil_ThreadPopCleanup (int execute)

intGalUtil_ThreadCancel (GalUtil_ThreadID tid)

void GalUtil_ThreadTestCancel ()

void GalUtil_ThreadCreate (GalUtil_ThreadID *tid, void*(*func)(void*), void *arg)

void GalUtil_InitConditionVar (GalUtil_ConditionVar *condVar)

void GalUtil_DestroyConditionVar (GalUtil_ConditionVar *condVar)

void GalUtil_ConditionWait (GalUtil_ConditionVar *condVar, GalUtil_LocalMutex *mutex)

void GalUtil_ConditionBroadcast (GalUtil_ConditionVar *condVar)

void GalUtil_ThreadExit ()

void GalUtil_InitMainWin32Thread ()

388

int GalUtil_TryLockLocalMutex (GalUtil_LocalMutex *mutex)

Miscellaneousint GalUtil_Mkdirp (char *pathname, mode_t mode) Cross-platform recursive directory creation function. Used in logging in the Hub.

void GalUtil_MilliSleep (int msecs)

int GalUtil_InitVerbose () Called to initialize values for verbosity levels.


389

Glossary

broker

A server-to-server connection, unmediated by the Hub, through which typed data can be transmitted.

call environment

A structure passed to every dispatch function which contains the environment of theinvocation, such as the current session, the connection through which the message wasreceived, and whether or not a response has been provided yet.

dispatch function

A procedure invoked in a server in response to a rule in a Hub program .

environment

See call environment .

frame

A general-purpose data object which is passed between servers and the Hub. It consists ofa frame type, a frame name, and a collection of attributes, or keys , and their associatedvalues. See the frame documentation.

global variables, program file

See program file global variables .

Hub

An executable which controls the interaction among a number of servers , using a Hub script as its controller.

Hub script

Initialization file for the Hub which contains information about available servers (name,location, externally accessible operations), global variables, and possibly flow of controlinstructions. Also known as a program file.

390

Hub program

A set of rules that describes a sequence of operations specifying the flow of control in the Hub .

initial token

A token specified in the program file which is executed immediately after the Hub contactsits servers.

key-value pair

An attribute and its associated value that describes some domain entity. Sets of key-valuepairs are stored in frames .

logfile

Text file containing logs of timestamped server operations and specified key values.

message

A frame used as a vehicle for communication to and from servers. The key-value pairswhich comprise the message sent to the server are drawn from the Hub program rules andthe current state of the token . The message causes a dispatch function to be invoked inthe server. Any response from the dispatch function is used to update the token state. See frame , token .

namespace

One of five types of frames comprising a memory state during the execution of a Hub program .

operation

A dispatch function supported by a server , as viewed by a service type in a program file .

program, Hub

See Hub program .

program file

See Hub script .

program file global variables

Program file declarations used to set key-value pairs in the initial token and to set globalinformation about the Hub. Examples include DOMAIN:, USER_ID: ,LOG_DIR: .

391

rule

An element of a Hub program which specifies a condition for execution and instructions forconstructing a message to a server and for updating the state of the token .

scripted interaction

A new incoming message which the Hub processes according to the instructions in a Hub program whose name matches that of the incoming message.

scriptless interaction

A new incoming message which the Hub processes by dispatching directly to a server whichsupports a dispatch function whose name matches that of the incoming message.

scripting module

A module compiled into the Communicator Hub which assumes responsibility for theprocessing of a subset of incoming new messages. MIT provides the default scriptingmodule, which allows users to specify programs to implement this control.

server

An executable configured to accept messages from the Hub and respond to dispatchfunctions invoked by these messages.

server-to-server subdialogue

A call to GalSS_EnvDispatchFrame, which uses the Hub to invoke operations on anotherserver or servers and retrieves the result. The Hub may retrieve this information via a Hubprogram, or it may relay the request directly to another server.

service provider

A specific process on a specific host which qualifies as an instance of a service type .

service type

A named collection of operations in a program file . Each operation is expected tocorrespond to a dispatch function implemented by any service provider which is aninstance of this service type.

session

An instance of an interaction of a user with a Hub.

token

392

A frame which represents the evolving state of the execution of a Hub program . A token issimilar to the local memory state of a function or a procedure. See also frame , message .

utterance ID

Internal counter maintained by the builtin server. This counter can be updated only bycalling builtin.increment_utterance . This counter cannot currently be accessedexcept by calling this builtin function.

verbosity

A setting provided to a Hub or server which controls the amount of information printed out.Zero verbosity is silence; the highest level of verbosity is 6; the default is 3.


393

Double server example

In this example, we demonstrate the basic functionality of the Communicator server architecture.

This example can be found in the distribution in contrib/MITRE/examples/double .

Source filesdouble_core.c: demonstrates a number of basic capabilities and important details of writing aCommunicator server, including

using the command line argument parsing utilities writing a simple dispatch function sending a message to the Hub manipulating frames defining _GalSS_init_server() defining the special reinitialize() dispatch function raising and handling errors defining message signatures (see double_server.h) setting up continuations

multiply.c: demonstrates the use of the INIT: keyword in the program file, as well asimplementing the second server for the complex double example.

headerless_double.c: illustrates the same basic capabilities as double.c, but in addition showshow to declare server information without using the server headers.

Scriptsdouble.csh: runs the double server and the Hub, using the double.pgm program file. You canplay with the headerless_double server by substituting it into this script.

complex_double.csh: runs the double server, the multiply server, and the Hub, using thecomplex_double.pgm program file.

validating_complex_double.csh: runs the double server, the multiply server, and the Hub, usingthe wrong_complex_double.pgm program file. This program file contains several errors, warningfor which are printed out by the Hub, which is started with the -validate flag. See the program filefor details.

394

scriptless_complex_double.csh: runs the double server, the multiply server, and the Hub, usingthe scriptless_complex_double.pgm program file. This program file has no programs.

logging_complex_double.csh: runs the double server, the multiply server, and the Hub, using thelogging_complex_double.pgm program file. This demo exemplifies partial scripting and logging.

client_complex_double.csh: runs the double server, the multiply server, and the Hub, using theclient_complex_double.pgm program file. This demo exemplifies the listener-in-Hub capability.To run this example, start the Hub first, then the multiply server, then the double server.

continuation_double.csh: runs the double server and the Hub, using thecontinuation_double.pgm program file. This example demonstrates how to use continuations,and in particular how they can be used to resolve deadlocks. See the comments in the programfile for details.

Language bindingsThe python/ subdirectory contains Python versions of the double and multiply servers. Theseservers are completely interchangeable with the C versions. In order to play with them, you cansubstitute them into the scripts.

The acl/ subdirectory contains Allegro Common Lisp versions of the double and multiply servers.These servers are completely interchangeable with the C versions. In order to play with them,you can substitute them into the scripts.

The java/ subdirectory contains a Java version of the double and multiply servers. These serversare completely interchangeable with the C versions. In order to play with them, you cansubstitute them into the scripts. Note that the command line arguments for the Java version ofthe listener-in-Hub capability are slightly different than the other bindings.

CompilationTyping "make" in this directory will create the servers, MIT-style Makefiles, scripts, and Python,Java and Allegro Common Lisp servers.

The C version of this server contains a MSVC project for Windows NT.

DiscussionSee the documentation on adding a server.


395

Brokering example

In this example, we demonstrate the basic principles of brokering, including multiple brokerclients and environment-aware brokers.

This example can be found in the distribution in contrib/MITRE/examples/audio .

Source fiilestestaudio_send.c: sets up an outgoing broker for 8-bit audio data.

testaudio_stream_send.c: like testaudio_send.c, but uses the timed task loop to allow the clientto connect before the outgoing broker is done writing data.

env_testaudio_receive.c: sets up an incoming broker for 8-bit audio in response to a Hubmessage. Dumps the result to the audio device and writes a notification message to the Hub.

testaudio_receive.c: like env_testaudio_receive.c, but uses the bare connection instead of thecall environment for handling the callback.

testaudio_abort_send.c: terminates a broker without completing it appropriately. This servertriggers the broker abort behavior in the receiving servers.

Scriptsaudio.csh: runs the testaudio_send and testaudio_receive servers and the Hub, using theaudio.pgm program file.

env_audio.csh: runs the testaudio_send and env_testaudio_receive servers and the Hub, usingthe env_audio.pgm program file. Note that the Hub receives a session ID along with thenotification message at the end of the interaction, in contrast to the audio.csh example.

double_audio.csh: runs the testaudio_send server, two testaudio_receive servers, and the Hub,using the double_audio.pgm program file. Observe that both broker clients receive the same audio.

client_double_audio.csh: like double_audio.csh, except that the testaudio_send server is startedup as a Hub client. Start this server last.

396

Language bindingsThe python/ subdirectory contains a Python version of the testaudio_send, testaudio_receiveand env_testaudio_receive servers. These servers are completely interchangeable with the Cversions. In order to play with it, you can substitute it into the scripts.

The acl/ subdirectory contains an Allegro Common Lisp version of the testaudio_send,testaudio_receive and env_testaudio_receive servers. These servers are completelyinterchangeable with the C versions. In order to play with it, you can substitute it into the scripts.

The java/ subdirectory contains a Java version of the testaudio server. This server is completelyinterchangeable with the C version. In order to play with it, you can substitute it into the scripts.However since it uses /dev/audio to play sounds it should only work on sparc-solaris machines.

CompilationTyping "make" in this directory will create the servers, MIT-style Makefiles, scripts and Python,Java and Allegro Common Lisp servers.

DiscussionSee the documentation on audio and brokering, as well as the session management documentation.


397

Main loop example

In this example, we show how the Communicator main loop works, and how to recreateCommunicator server functionality in other main loops.

This example can be found in the distribution in contrib/MITRE/examples/simple_mainloop .

Source filessimple_mainloop.c: provides a crude main loop for illustration purposes, which allows the user toset callbacks on file descriptors and on "timers" (actually, loop iterations, but it serves thepurpose). This file defines the functions SM_AddTimerCallback, SM_RemoveTimerCallback,SM_AddFDCallback, SM_RemoveFDCallback, and SM_Mainloop ("SM" stands for "simple mainloop").

simple_mainloop_elr.c: uses this crude mainloop to run a Communicator-compliant server.

mainloop_double.c: the standard double server from the double example, using simple_mainloop_elr.c.

basic_double.c: a double server with an explicit main loop (essentially a copy of the one in theGalaxyCommunicator core library), to contrast with the other servers in this example.

Scriptsmainloop_double.csh: runs the mainloop_double server and the Hub, using the double programfile from the double example.

mainloop_audio.csh: runs the mainloop_testaudio_send and mainloop_testaudio_receiveservers and the Hub, using the audio program file from the audio example.

basic_double.csh: runs the basic double server and the Hub, using the double program file fromthe double example.

398

CompilationTyping "make" in this directory will create the servers, MIT-style Makefiles, and scripts.

DiscussionSee the documentation on server structure and alternative main loops.


399

Binary data example

In this example, we demonstrate how to use the binary data object.

This example can be found in the distribution in contrib/MITRE/examples/binary_data .

Source fiilesaudio_binary.c: routes 8-bit audio data from itself to itself, using a binary object. Dumps theresult to the audio device. Compare this to the brokering version.

data_binary.c: creates a data encoding and decoding for a simple structure and passes itthrough the Hub using binary objects.

mgal_data_binary.c: creates a data encoding and decoding for two simply structures and passesthem through the Hub using the MITRE facility for manipulating binary objects.

Scriptsaudio.csh: runs the audio_binary server and the Hub, using the audio.pgm program file.

binary.csh: runs the data_binary server and the Hub, using the binary.pgm program file.

mgal_binary.csh: runs the mgal_data_binary server and the Hub, using the binary.pgm. program file.

Language bindingsThe python/ subdirectory contains Python versions of the audio_binary, data_binary andmgal_data_binary servers. These servers are completely interchangeable with the C versions. Inorder to play with them, you can substitute them into the scripts.

The acl/ subdirectory contains an Allegro Common Lisp version of the audio_binary server. Thisserver is completely interchangeable with the C version. In order to play with it, you cansubstitute it into the script.

The java/ subdirectory contains Java versions of the testaudio and echo servers. These areinterchangeable with the C versions. In order to play with them, you can substitute them into thescripts. However since they use /dev/audio to play sounds they should only work onsparc-solaris machines.

400

CompilationTyping "make" in this directory will create the servers, MIT-style Makefiles, scripts and Python,Java and Allegro Common Lisp servers.

DiscussionSee the documentation on binary objects and the documentation on the MITRE binary object tools.


401

Using non-MIT Makefiles in the double server

In this example, we present a version of the double server which does not use the MITMakefiles. This example relies on the source code in the double server example.

This example can be found in the distribution in contrib/MITRE/examples/local_makefile .

Source filesNone. Uses the source code in the double example.

Scriptsdouble.csh: runs the double server and the Hub, using the double.pgm program file in the double example.

CompilationTyping "make" in this directory will create the script.

DiscussionSee the documentation on using non-MIT Makefiles.


402

Multiple connection server example

In this example, we use the simple double server to illustrate a multiple connection server.Multiple connections are available in the core, so all we need is to tell the server to acceptmultiple connections, using the -maxconns argument. This example uses the server andprogram file from the double example.

This example can be found in the distribution in contrib/MITRE/examples/many_conns .

Source filesNone.

Scriptsdouble.csh: runs the double server and the Hub, using the double.pgm program file from thedouble example.

CompilationTyping "make" in this directory will create the script.

DiscussionSee the documentation on setting maximum number of connections.


403

Turn Management

MIT provides a way of controlling servers using Turn Management, a set of resources which wasone of the initial inspirations for the Hub program files. The Galaxy library provides a dispatchfunction named turn_management which implements this toplevel control. We describe itbriefly here. At least part of this documentation will take the form of server documentation, sincethis is a good deal of the behavior from the programmer’s point of view.

Using Turn Management for Dialogue ControlImplement the Server Dialogue Control Functions Create a Dialogue Control File

Command Line Arguments Dialogue Control Script Header File

Initialize Turn Management Turn Management Message Set

turn_management

As an illustration of how to use Turn Management for dialogue control, we will use the pegasusserver, a flight reservations server.

Implement the Server Dialogue Control FunctionsImplement the functions to handle the dialogue control. All dialogue control functions areassumed to be of the form:

int function_name (GAL_DOMAIN_SVR *dc);

Valid return values are:

404

DIALOGUE_CONTINUE DIALOGUE_STOP DIALOGUE_RESTART

See below for an explanation of the implications of these return values.

The file containing the functions should be kept in the server directory. A header file, containingdialogue control function prototypes, will be automatically generated at compile time. (See below.)

As an example, pegasus has a function called default_action which handles the undefinedcase by replying with a "cant_handle" clause.

int default_action (GAL_DOMAIN_SVR *dc) { if (!dc->reply_frame) dc->reply_frame = Gal_MakeFrame ("cant_handle", GAL_CLAUSE);

return(DIALOGUE_STOP); }

See below for an example rule which states the required conditions in order for default_action to be executed.

Another example is the need_destination function. A "need_destination" clause is createdfor the given topic and then added to the system initiative.

int need_destination (GAL_DOMAIN_SVR *dc) { Nframe topic; topic = Gal_GetFrame (dc->request_frame, ":topic"); dc->reply_frame = Gal_MakeFrame ("need_destination", GAL_CLAUSE); fr_setsprop(dc->reply_frame, ":topic", make_tframe(copy_nframe(topic))); Gal_AddSystemInitiative (dc, "destination", NULL); return(DIALOGUE_STOP); }

See below for an example rule which states the required conditions in order for need_destination to be executed.

405

Create a Dialogue Control FileThe dialogue control file dictates the behavior of the dialogue control. It contains a single scriptof rules.

The default dialogue control file is assumed to be ../System/<server_name>.dctl. The dialoguecontrol file can also be specified as a command line argument. If no file is specified and thedefault file does not exist, execution is aborted.

A header file, based on the dialogue control file, is automatically generated at compile time. Thisheader file contains the function prototypes for each operation listed in the dialogue control file. See below for more details.

Command Line ArgumentsThe dialogue control flow is regulated via the dialogue script which can be provided as acommand line argument. If not provided, the dialogue file defaults to "../System/<server_name>.dctl"

static char *oas[] = { "-dialogue_script name", "script for dialogue control in domainserver", NULL, NULL };

This server also accepts the standard server arguments. See the oa library for details.

Dialogue Control ScriptThe dialogue control file contains a dialogue control script. The dialogue control script is similarto a Hub program in that it contains a script consisting of a set of rules. Each rule specifiesconditions that must be satisfied in order for a given operation to be executed. When turn_management is called, each rule is evaluated, and the subsequent operations are called.

Each operation returns a value which has implications for the flow of control through the script. Valid return values are:

DIALOGUE_CONTINUE --> resume flow of control immediately after operation was executed DIALOGUE_STOP --> do not resume flow of control DIALOGUE_RESTART --> resume flow of control at top of script

406

A rule in a dialogue control script is similar to a rule in a Hub program except that the RULE:identifier is not used. Otherwise, the syntax is identical:

<alphanumeric expression> --> <operation>

where the alphanumeric expression is ( [!] [X %] <key> [comparison] [<value>] [&] [|])

See the table of Valid Operators for Constructing Rules.

A dialogue control script can also contain comments. Any line beginning with a semicolon isinterpreted as a comment.

Examples of Valid Rules

clause help --> give_help ; If clause is equal to help then execute the give_help operation.

on_time which | departure_time %which | arrival_time %which --> mark_status ; on_time must be equal to which, or departure_time must contain which, or arrival_time must contain which in order for the mark_status operation to be called.

destination & filtered !% to --> filter_to_airport ; If destination exists and filtered does not contain to , then call execute filter_to_airport .

num_found >1 --> speak_multiple ; If num_found is greater than 1, execute speak_multiple .

!sql_query & !destination & !flight_number --> need_destination ; If there is no sql_query and no destination and no flight_number , execute need_destination .

--> default_action ; If no other rule’s conditions have been satisfied, execute default_action . In order forthis to work as the default, the rule has to be the last rule in the set.

The Header FileThe new style for generating header files for dialogue control is described here. The old style isno longer supported. See the upgrade notes.

407

The new styleIn the new style, the user creates a declarations file of this form:

GAL_DIALOGUE_FUNCTION(<op_name>) GAL_DIALOGUE_FUNCTION(<op_name>) ...

with a GAL_DIALOGUE_FUNCTION declaration for each server dialogue control function. Thenthe user adds the following header information to exactly one of the server source files:

#define SERVER_FUNCTIONS_INCLUDE "<server_declaration_file_name>" #include <galaxy/server_functions.h>

#include <domain_svr/domain_svr.h> #define DIALOGUE_FUNCTIONS_INCLUDE "<dctl_declarations_file_name>" #include <domain_svr/dialogue_functions.h>

where <server_declaration_file_name> and <dctl_declaration_file_name> arereplaced with the appropriate names of the server declarations file and the file created here,respectively., As with the old style of server header generation, the server header file includesthe appropriate behavior for the dialogue header file. Note that this new style works inconjunction with the new style of declaring server information; in particular, theGAL_SERVER_SW_PACKAGE() macro is used to incorporate the dialogue control module.

Here is an example of what the MIT pegasus server would look like if it used the new style of headers:

------ pegasus-dialogue.h

GAL_DIALOGUE_FUNCTION(give_help) GAL_DIALOGUE_FUNCTION(repeat_response) ...

------ pegasus-server.h

GAL_SERVER_NAME(pegasus) GAL_SERVER_PORT(11100) GAL_SERVER_OP(turn_management) GAL_SERVER_SW_PACKAGE(MITDialogue)

------ functions.c

... #define SERVER_FUNCTIONS_INCLUDE "pegasus-server.h" #include <galaxy/server_functions.h> #include <domain_svr/domain_svr.h> #define DIALOGUE_FUNCTIONS_INCLUDE "pegasus-dialogue.h" #include <domain_svr/dialogue_functions.h>

408

...

Initialize Turn ManagementIn order to use turn management, your domain server must call an initialization function, Gal_DialogueInitServer , provided by the Galaxy library.

void *_GalSS_init_server(GalIO_ServerStruct * server, int argc, char ** argv) { return( Gal_DialogueInitServer ( GalIO_GetServerName ( server), argc, argv)); }

void *Gal_DialogueInitServer (char *server_name, int argc, char **argv) Initializes and returns the server specified by server_name. This function calls Gal_InitTM.

GAL_DOMAIN_SVR *Gal_InitTM (int argc, char **argv) Initializes the domain server. The dialogue script may be specified by the -dialogue_scriptargument shown above; otherwise, the dialogue file defaults to "../System/<server_name>.dctl". If no dialogue script is provided and the default file is not found, the function returns a fatal error. This function is called by Gal_DialogueInitServer.

You can also add support for printing out usage information as follows:

void _GalSS_print_usage(int argc, char **argv) { Gal_DialoguePrintUsage(argc, argv); }

void Gal_DialoguePrintUsage (int argc, char **argv) Prints out usage information for turn management.

Turn Management Message Setturn_management

turn_management runs the dialogue control tools (via the dialogue control file) andgenerates a domain-specific response.

409

parameter type optionaldepends on

description/constraints

IN: :key_value string yes key/value paraphrase of input

:kv_lang string yes

key/value paraphrase language;if not present in input, will be setfrom system’s default key-valuelanguage, which is "dialogue"

:parse_frame frame parsed input string

:request_frame frame input plus context

:tm_state frame yes previous state of turn manager

OUT: :reply_frame frame

will have the form {... :domain<string> :comment_tlist <list>:continuant <frame> :db_tlist<list> ...}

:list yes (no examples)

:map yes (no examples)

:image yes (no examples)

:tm_state frame yes previous dialogue state

:filter_list frame

value is {... :Flist <list of strings>:Ftype <string> :Fcategory<string> :Fkey <string> ....}.Describes words which ought tobe present in the :parse_frame. Used by a number of servers toprovide preferences forchoosing among frames.

:discourse_update frame yes frame for context tracking out toadd to the discourse history

:system_initiative frame yes query to present to the user

Because many servers utilitize the Turn Management dialogue control capabilities, it isrecommended that the dot notation be used for specific servers when the turn_managementoperation is included in a Hub program rule:

410

RULE: :request_frame & :key_value & :domain Pegasus --> pegasus.turn_management RULE: :request_frame & :key_value & :domain Jupiter --> jupiter.turn_management


411

libGalaxyEnhancements

This documentation briefly describes functions in the MIT dialogue control library.

Dialogue Functions Domain Functions

Dialogue Functionsvoid Gal_InitializeDialogueDefaults (GAL_DIALOGUE_FUNCTION_MAP *function_map); Initializes the default dialogue function map to function_map.

Gal_IntFnPtr Gal_GetDialogueFunction (char *fn_name) Retrieves the dialogue function fn_name from the dialogue function map.

GAL_DIALOG_CONTROL *Gal_InstantiateSystem (char *dialogscriptfilename); Instantiates the dialogue control and calls Gal_LoadDialogueControl. If dialog_script_filename isnot specified, it is assumed that the dialogue control script is ../System/server_name.dctl. Ifunsuccessful, NULL is returned.

GAL_INSTRUCTION **Gal_LoadDialogueControl (GAL_DIALOG_CONTROL *dc); Loads dialogue control and validates the requested functions. This function is called by Gal_InstantiateSystem.

Gal_Frame Gal_FillDialogueState (GAL_DIALOG_CONTROL *dc, int clear); This function is called by turn_management.

int Gal_DialogueLoop (GAL_DIALOG_CONTROL *dc, Gal_Data data); Executes the loaded dialogue control commands. This function is called by turn_management.

412

void Gal_VarSetValue (Gal_Frame dialogue_state, char *key, char *value, int int_value); If value is not NULL, this function sets the value of key to value, otherwise it sets it to int_value.

int Gal_VarUnsetValue (Gal_Frame dialogue_state, char *key); Removes the object specified by key.

intGal_VarIsSet (Gal_Frame dialogue_state, char *key); Returns 1 if key exists in the dialogue_state, 0 otherwise.

char *Gal_VarGetValue (Gal_Frame dialogue_state, char *key); Returns the value of the object specified by key.

int Gal_VarMatchValue (Gal_Frame dialogue_state, char *key, char *match_value); Returns 1 if the value of the object, specified by key, matches match_value. Otherwise, returns 0.

void Gal_VarAugmentValue (Gal_Frame dialogue_state, char *key, char *value); Appends value to the value of the object specified by key. This applies only to string objects.

void Gal_Para2Fact (Gal_Frame dc, char *paraphrase); This converts the key value paraphrase, paraphrase, into an eform frame structure.

Gal_Frame GalSS_DispatchQueryToIServer (char *par_lang, Gal_Frame query_frame,Gal_Frame intermediate_reply)

Gal_Frame GalSS_RefillDialogueState (GAL_DIALOG_CONTROL *dc, Gal_Frame request_frame, char *para_lang, int clear)

Domain FunctionsGal_Frame Gal_DeclareFilterList (char *Ftype, char *Fcategory, char *Fkey, Gal_Object *tarray, int n); Converts the values of tarray into an filter list, where n is the number of objects in the list. Theclause frame that is returned contains the filter list, the filter type, Ftype, and the filter key, Fkey. If Fcategory is "ignore", this function does nothing and returns NULL.

Gal_Frame Gal_FetchFilterList (Gal_Frame prev_state); This function returns any prior filter list for n-best selection filtering on categories.

Gal_Frame Gal_FillParaphraseKeys (char *clause_name, Gal_Frame frame, char *domain, char *language); This function fills the paraphrase keys of the frame, fr, with values specified by clause, domain,and language.

413

Gal_Frame Gal_GrovelForPredLike (Gal_Frame ref_frame, char *match_string, char *reject_string, int exact); Looks in ref_frame and finds a match to match_string, rejecting if it contains reject_string, andrequiring an exact match if exact is set. Returns the matching predicate or NULL if no match is found.

Gal_Frame Gal_GrovelForPred (Gal_Frame ref_frame, char *pred_name); Looks in ref_frame and finds an exact match to pred_name. Returns the matching predicate orNULL if no match is found.

Gal_Frame Gal_GrovelForTopic (Gal_Frame ref_frame, char *topic_name); Looks in ref_frame and finds an exact match to topic_name. Returns the frame containing thematching topic or NULL if no match is found.

Gal_Frame Gal_IsNew (Gal_Frame frame, Gal_Frame thin_frame); Looks in thin_frame for an existing topic or predicate whose name matches that of frame. If amatch is found, that topic or predicate frame is returned. Otherwise, NULL is returned.

void Gal_AddComment (GAL_DOMAIN_SVR *dc, Gal_Frame clause); Adds parenthetical user comments in the frame, clause, to the domain server.

void Gal_AddSystemInitiative (GAL_DOMAIN_SVR *dc, char *pred_name, Gal_Frame frame); If frame is provided, this function copies frame to the system initiative under the name, pred_name. If no frame exists, a predicate frame is created and added under the name, pred_name.

void Gal_FillResponse (GAL_DOMAIN_SVR *dc, Gal_Frame res); Fills the domain server with values in the result frame, res.

void Gal_InitDService (Gal_Frame big_frame, GAL_DOMAIN_SVR *domain_svr); Initializes the domain server with values in the big_frame: domain information, the operationname, the key_value paraphrasing, the parse_frame, and the request_frame.

void Gal_InitializeContents (Gal_Object *array, int nmax); Initializes the contents of the object, array, where nmax is the size of the array.

Gal_Frame Gal_RepeatResponse (Gal_Frame prev_state) Repeats the previous response, if the prev_state has one.

414

Control file toolsGal_ConditionStruct **Gal_LoadDialogueControl (char *filename, int *num_conditions_ptr)

void Gal_PrintDialogueControl (FILE *fp, Gal_ConditionStruct **conditions, int num_conditions)


415

The Program Verifier

The executable verify_program will check the syntax of a Hub program file and report errors.

Usage

Command line

<GALAXY_ROOT>/contrib/MIT/galaxy/bin/verify_program ...

Command line arguments

static char *oas[] = { "-in filename", "program file to verify", NULL, "-out filename", "regenerate program to file (- is stdout)", NULL, "-debug", "verbose mode", "-color", "enable color output", NULL, };

The -in argument is the input program file. Errors will be reported for this file. If you wish a"canonical" program file to be written (for instance, for seeing how the Hub will interpret aprogram file), use the -out argument.

Last updated November 2, 2000

416

The Builtin Server

The builtin server is built into the Hub. It provides administrative support, as well as an additionalbody of functionality the MIT group has found useful.

UsageBuilt into the Hub. Does not need to be declared in the program file.

Message Sethub_turn_management dispatch_token dispatch_to_main call_program increment_utterance abort_main new_session end_session destroy nop reset_history log_keys modify_properties get_properties set_session hub_print hub_break hub_exit hub_raise_error hub_available_servers hub_token_owner hub_pause hub_set_verbosity

417

hub_continue

hub_turn_management invokes a Hub-local version of turn management, which relies onactions identified by the parser to determine its behavior. Unlike BIANCA-style turnmanagement, only one action applies. In the MIT program files, this message is invoked whenthe domain that the parser identifies is "local".



IN: :parse_frame frame the name of the frame is the action


the name of the frame is the return valueof the action. If the :parse_framecontains a :domain key/value pair, the:reply_frame is also guaranteed to havethe same value for :domain. If the returnvalue of the action is NULL, then the:reply_frame is retrieved from thetoplevel frame (the action will have set it).

The actions are:

scratch_that: removes the last utterance from processing and the discourse history.



IN: :session_id string yes string which identifies the session;defaults to "Default"

Returns one of the strings "scratched", "unscratched". clear_history: clears the discourse history.




Returns the string "history_cleared".

418

close_off: interpretation of user’s utterance "goodbye"

parameter type optional depends on description/constraints

OUT: :close_off integer value is 1

Returns the string "goodbye". greetings: interpretation of user’s utterance "hello"


OUT: :greetings integer value is 1

Returns the string "greetings". praise: interpretation of user’s praise of the system


OUT: :praise integer value is 1

Returns the string "praise". vacuous: interpretation of vacuous user utterance


OUT: :vacuous integer value is 1

Returns the string "vacuous". domain_switch: switches the current domain for the session

419



IN: :session_id string yes string which identifies thesession; defaults to "Default"

:parse_frame frame

the :parse_frame should havethe form {... :topic {... :name "<name>" ... } ... }

OUT: :new_domain string Domain being switched to.

:reply_frame frame yes If the domain being switched towas previously encountered, thisvalue is set (a continuation?).

:welcome_frame frame yes If the domain being switched towas not previously encountered,this value is set.

Returns NULL. unknown_word:


OUT: :reply_frame frame copies the value of :parse_frame

Returns NULL. no_parse

420





will be {c rejection_<num> :no_parse 1 :domain <domain> } :domain will be set if the sessionhas a domain specified; <num> isan internal index of failed parses.

Returns NULL. call_me



IN: :parse_frame frame

the parse frame should have the form ... :topic {... :country_code "<country_code>" :area_code "<area_code>" :exchange "<exchange>" :core "<core>" } :calling_domain <optional string:provides domain for session> ... } where the phone number in the result will be <country_code>-<area_code>-<exchange>-<core>.

OUT: :callback_channel string value is "telephone"

:callback_address string value is "<country_code>-<area_code>-<exchange>-<core>"

Returns the string "answer_phone". erase_all_history Clears the Hub database history. Returns the string "history_cleared".

This message returns a frame.

dispatch_token handles the details of invoking the program named in the :program key(default is "main") using the current message. It creates a new message by changing the nameof the frame to the program name and removing the internal Hub state information from the

421

frame. It updates session information from :out_lang, :para_lang, :kv_lang and :domain. Thedispatch function dispatch_to_main simply deletes its incoming :program key and calls dispatch_token.

These messages are special because they will relay arbitrary keys to the new message, notjust the keys which are listed here.


IN: :session_id string yes string which identifies the session

:program string yes program name to dispatchto; default is main

:server_id frame yesabsence of :session_id

should be {...:server_name <string>:host <string> :port <int>:sock <int> ...}. All keys areoptional. Used to computevalue of :session_id, whichultimately defaults to "Default"

:out_lang string yes output language for the session

:para_lang string yes paraphrase language forthe session

:synth_lang string yes

synthesizer "language" forthe session (that is, thename of the Genesiscatalog for transductionfrom a string to a legalsynthesizer input)

:kv_lang string yes key-value "language" forthe session

:domain string yes name of the domain.

:utterance_id integer yes the utterance index

:reply_clause_name string yes

for generating a:reply_frame for the "local"domain (that is, the builtin server)

422

:no_abort(not checked)

yes

value of:session_idand:utterance_id(aftercomputationof output values)

blocks a previously-setabort for all utterances withthis :utterance_id

This message returns NULL.

call_program is a streamlined version of dispatch_token, which does not do any specialprocessing on the current message beyond copying it, removing the Hub state information, andinvoking the named program (default is "main"). It returns the final token state of the called program.

This message is special because it relays arbitrary keys to the called program, and in fact doesnot pass the :program key at all. It also returns arbitrary keys.



IN: :session_id string yes string which identifies the session

:program string yes program name to dispatch to; default is main


increment_utterance increments the various indices for recording how many utteranceshave been made.




:domain string yes name of the domain.

OUT: :utterance_id integer new utterance index

423


abort_main sets an abort for a specified utterance. This function returns NULL.




:server_id frame yesabsence of :session_id

should be {... :server_name <string>:host <string> :port <int> :sock <int>...}. All keys are optional. Used tocompute value of :session_id, whichultimately defaults to "Default"

:utterance_id integer yes utterance index; defaults to theutterance index of the located session

This message does not return a frame.

new_session resets a session (opens a new logfile, resets utterance counter). If the sessiondoes not exist, it will be created. To make sure that logfiles are managed correctly when sessionIDs are reused, this function only opens a new logfile the second time it’s called and eachsubsequent time. See the session management documentation for details.



IN: :session_id string yes string which identifies the session; defaultsto "Default"


end_session terminates the specified session. If a logfile is open, it will be closed.



IN: :session_id string yes string which identifies the session; defaultsto "Default"

424


destroy destroys the token.

This message returns a destroy token message reply. It does not return a normal reply.

nop is a no-op. It returns what is passed in.

reset_history saves away the existing history when a domain switch happens, and restores apreviously stored history for the new domain (if it exists).



IN: :session_id string yes string which identifies the session; defaultsto "Default". If there’s no session, returnswithout switching

:new_domain string name of the new domain


log_keys invokes the logger. This message is special because it logs whatever keys are inthe frame passed to it. The keys will be logged under a timestamped instance of the log_keysoperation. This message returns the frame passed to it.

modify_properties updates properties in either the session or server Hub states. It deletesbefore it sets.



IN: :namespace string identifies the Hub namespaceto retrieve from (options are server or session )

:properties_to_delete list yes a list of keys to delete from thespecified namespace

:properties_to_set frame yes

a frame containing key-valuepairs to add to the specifiednamespace (the name of theframe is ignored).

425

This message returns NULL. See the Hub properties documentation.

get_properties retrieves properties for either the session or server Hub states.



IN: :namespace string identifies the Hub namespace to retrievefrom (options are server or session )

:properties list a list of keys to retrieve from the specified namespace

This message returns a frame containing the specified keys. See the Hub properties documentation.

set_session possibly creates a new session, and optionally assigns some session locks to it.



IN: :session_id string yes string which identifies thesession; defaults to "Default".

:lock_info integer yes

a logical OR of flags controllinghow this session should belocked. See GalSS_EnvSetSession. Ifabsent, the locks from theprevious session will be used,if present.

:previous_session_id string yes a session to transfer locksfrom, if appropriate


hub_print prints a message or the contents of a Hub namespace.

426



IN: :namespace string yes identifies the Hub namespace todisplay the contents of

:message string yes a string to display

:verbosity integer yes if present, restricts the printing tosituations where GAL_VERBOSEexceeds the specified verbosity

:session_locks integer yes if nonzero, causes the Hub to print thestatus of all session locks

:db integer yes if nonzero, causes the Hub to print thecontents of the current sessionutterance history DB


hub_break enters a loop which allows the user to explore the current state of the Hub.



OUT: :disable_break integer if 1, indicates that the user exitedthe loop requesting no further breaks

This message returns a frame

hub_exit causes the Hub to exit gracefully. This message returns NULL.

hub_raise_error raises an error using GalSS_EnvError. This function can be useful if you’vecaught an error using the ERROR: directive in a Hub program but you still want to raise an errorfor the caller.



IN: :error_description string the description of the error to bepassed to GalSS_EnvError.

427


hub_available_servers returns the active service types in the Hub. This list may not matchthe list of defined service types due to the use of the SERVERS: directive in the program file.



OUT: :serverslist of strings

no a list of names of active service types


hub_token_owner returns the name of the service type of the provider originating thecurrent token.



OUT: :owner string The name of the service type for the provideroriginating the current token


hub_pause sets a pause after each message dispatch. This can be useful for demoing and debugging.



IN: :milliseconds integer yes If present, the length of the pause in milliseconds


hub_set_verbosity sets the verbosity level for the Hub, in addition to the command lineargument to the Hub and the GAL_VERBOSE environment variable.


IN: :verbosity integer Verbosity level (0 to 6)

428


hub_continue listens for a new message of the specified form and treats it as a reply orerror. See the section on Hub continuations for a discussion.



IN: :reply_matcheslist of frames

yes

If present, a sequence of framesone of which should match anincoming new message, which willthen be treated as a normal reply.Matching is done using Gal_MatchFrame. One of this and :error_matches must be specified.

:error_matcheslist of frames

yes

If present, a sequence of framesone of which should match anincoming new message, which willthen be treated as an error reply.Matching is done using Gal_MatchFrame. One of this and :reply_matches must be specified.

:service_type string yes

If present, the name of a servicetype known to the Hub, which willserve as a filter on which providerswill be monitored for new messages

:service_provider string yes

If present, the identity or location ofa service provider (e.g, audio@localhost:14000 or localhost:14000 ) which willserver a s afilter on whichproviders will be monitored for new messages

This message returns the frame which matches one of the elements in :reply_matches or :error_matches .


429

Installing and Accessing MITRE Utilities

Compiling the utilitiesTo compile the MITRE utilities, assuming that ROOTDIR is set to the root of your installation:

% cd $(ROOTDIR)/contrib/MITRE % make

You must do this after you compile the Galaxy system as a whole, since the MITRE examplesuse MIT libraries (of course). If you’ve done a toplevel compilation and the MITRE subdirectoryis present, this compilation will be done for you already.

Compiling against the utilitiesIn your Makefile, you must put the link and include information:

# For instance, if your include information is in CPPFLAGS # and your link information in LIBS, and if ROOTDIR is set to # the root of the installation:

# First, make sure that MIT’s architecture settings are loaded. # The TEMPLATES variable must be set to the location of archos.make.

TEMPLATES = $(ROOTDIR)/templates


CPPFLAGS += -I$(ROOTDIR)/contrib/MITRE/utilities/include LIBS += -L$(ROOTDIR)/contrib/MITRE/utilities/lib/$(ARCHOS) -lMITRE_galaxy

430

Using the utilities in your codeInclude the header "MITRE_galaxy.h". In addition, because some global thread properties needto be dynamically initialized, you must either call the function MGal_InitializeStatics or includethe following line in your server declarations file:

GAL_SERVER_SW_PACKAGE(MITREUtilities)

See the documentation on adding a server.

void MGal_InitializeStatics (void) Initializes the thread variables.

Last updated February 8, 2000

431

Frame Tools

It is sometimes convenient for the user to create a frame for sending a message, or for someother purpose, in a single call, rather than creating the empty frame and adding values one byone. This tool provides that functionality.

Gal_Frame MGal_CreateFullFrame (char *name, int type, int num_pairs, ... ) The name and type are as they are for Gal_MakeFrame. The num_pairs is the number ofkey-value pairs which follow; the first element of each pair must be a string, and the second aGal_Object. Here’s an example:

MGal_CreateFullFrame("main", GAL_CLAUSE, 4, ":input_string", Gal_StringObject(str), ":utterance_id", Gal_IntObject(UttIdx), ":hyp_id", Gal_IntObject(0), ":mode", Gal_StringObject("typed"));

The frame which is created is a garden-variety Gal_Frame, and subsequent calls to Gal_SetProp, etc., work as expected.

See the page on compiling and accessing these utilities.


432

Binary Data Tools

The MIT binary data object can be used to transmit arbitrary structures from server to server,among other things. In this set of tools, we provide a facility for automatically encoding anddecoding these structures. We provide an example of this facility in the context of the binary data example.


Defining the data typeThe MIT system supports a set of predefined datatypes which are used when determining thetypes of information transmitted over broker connections and stored in keys of frames. The toolshere allow the user to "extend" this set ot types.

typedef void *(*MGal_BinaryDataEncoder )(void *obj, int *size) typedef void *(*MGal_BinaryDataDecoder )(void *data, int size)

void MGal_AddBinaryDataType (int data_type, MGal_BinaryDataEncoder encoder,MGal_BinaryDataDecoder decoder) This function associates the encoder and decoder functions with the data_type. The encodertakes an arbitrary structure and returns something that will be treated as an array of bytes oflength size. The decoder takes an array of bytes of length size and returns an arbitrarystructure. The programmer defines the encoder and decoder; typically, the types are declared in _GalSS_init_server:

enum {TEST1_DT, TEST2_DT};

/* The Test2 type is elided here; see example */

typedef struct __TestStruct1 { int first; int last; char *msg; } TestStruct1;

433

static void *encode_test_struct1(void *data, int *len) { char *encode_buf; TestStruct1 *s = (TestStruct1 *) data;

encode_buf = (char *) malloc(sizeof(char) * (strlen(s->msg) + 64)); sprintf(encode_buf, "%d %d %s", s->first, s->last, s->msg); *len = strlen(encode_buf); return (void *) encode_buf; }

static void *decode_test_struct1(void *data, int len) { TestStruct1 *s = (TestStruct1 *) malloc(sizeof(TestStruct1)); int where;

sscanf(data, "%d %d %n", &(s->first), &(s->last), &where); s->msg = (char *) malloc(sizeof(char) * (1 + (len - where))); strncpy(s->msg, data + where, len - where); s->msg[1 + (len - where)] = ’\0’; return (void *) s; }

void *_GalSS_init_server(GalIO_ServerStruct *s, int argc, char **argv) { MGal_AddBinaryDataType(TEST1_DT, encode_test_struct1, decode_test_struct1); return (void *) NULL; }

Storing and accessing the data typeWe provide facilities for storing and accessing this data. The functions here are intended to beparallel to functions like Gal_IntObject/Gal_GetInt, etc.

Gal_Object MGal_OpaqueObject (void *obj, int data_type) Given an object whose encoder and decoder have been associated with data_type, return anobject suitable for use as a value of a frame key. This function constructs a frame which containsthe data_type and the binary encoding of the object:

{c __opaque__ :type <datatype> :data <binary_encoding>}

434

void *MGal_GetOpaque (Gal_Frame fr, char *key, int *data_type) Looks for a frame of the structure just described as the value of key in frame fr, decodes itaccording to its user-defined atatype, and sets the data_type. If the key does not exist, or thevalue is not a frame of the form described, or if there is no decoder for the datatype the framecontains, this function returns NULL and does not set data_type.

void *MGal_GetOpaqueWarn (Gal_Frame fr, char *key, int data_type) Like MGal_GetOpaque, but in addition returns NULL if the element found is not of the required data_type.

In the example provided, the reinitialize dispatch function creates an object and dispatches it tothe Hub, which returns it to othe receive_binary dispatch function, which prints out the structurebased on its decoded type:

Gal_Frame receive_binary(Gal_Frame f, void *server_data) { int dt; void *obj = MGal_GetOpaque(f, ":binary_data", &dt); TestStruct1 *s1;

if (obj) { switch (dt) { case TEST1_DT: s1 = (TestStruct1 *) obj;

sls_pinfo1("First is %d, last is %d, msg is ‘%s’\n", s1->first, s1->last, s1->msg); break; /* ... */ } } return (Gal_Frame) NULL; }

Gal_Frame reinitialize(Gal_Frame f, void *server_data) { Gal_Frame fr = Gal_MakeFrame("main", GAL_CLAUSE); TestStruct1 *s1;

s1 = (TestStruct1 *) malloc(sizeof(TestStruct1));

s1->first = Gal_GetInt(f, ":test_first"); s1->last = Gal_GetInt(f, ":test_last"); s1->msg = Gal_GetString(f, ":test_msg"); Gal_SetProp(fr, ":binary_data", MGal_OpaqueObject((void *) s1, TEST1_DT));

435

/* ... */

/* Satisfy the response to reinitialize() */ GalSS_WriteFrameToHub(f, server_data, 0); /* Send the new message */ return fr; }


436

Brokering Tools

MITRE offers a utility which bundles up some of the brokering information and slightly simplifiessetting up a brokering connection. This utility involves three steps: defining type handlers for thebroker, adding outgoing broker support, and adding incoming broker support.

The brokering example contains servers which illustrate this technique in addition to thetechnique support directly by the MIT API. This tool is completely compatible with the MIT API;it’s possible to use the MITRE tools on one end of the connection and the MIT API on the other,and the MITRE tool works without the timed task loop in the same way as the MIT API does.


Defining type handlersA type handler accepts the broker structure, the data, and the data length as arguments:

typedef void (*MGal_BrokerDTHandler )(GalIO_BrokerStruct *broker_struct, void *data, int n_samples)

These type handlers should be registered in _GalSS_init_server:

void *_GalSS_init_server(GalIO_ServerStruct *s, int argc, char **argv) { MGal_AddBrokerDTHandler(GAL_STRING, (void *) AUDIO_SERVER_NEW_UTT, audio_in_new_utt); MGal_AddBrokerDTHandler(GAL_STRING, (void *) AUDIO_SERVER_SAMPLES_DONE, audio_in_done); MGal_AddBrokerDTHandler(GAL_STRING, (void *) AUDIO_SERVER_SAMPLES_ABORT, audio_in_abort); MGal_AddBrokerDTHandler(GAL_INT_16, (void *) NULL, audio_in_int16_data); return (void *) NULL; }

437

void MGal_AddBrokerDTHandler (Gal_ObjectType dt, void *val, MGal_BrokerDTHandler h) The dt is an integer data type. The val is what the data needs to match (currently, this is onlyimplemented for strings). The handler h is a callback function which will perform the appropriateactions for this type and match value.

Adding outgoing broker supportIn the usual case, the programmer needs to populate the frame with the appropriate brokerinformation, much of which is predictable. The MITRE utility handles this case.

GalIO_BrokerStruct *MGal_AddOutgoingBrokering (GalIO_CommStruct *gcomm,Gal_Frame fr, int poll_ms, int timeout_ms) The arguments for this function are identical to those of GalIO_BrokerDataOutInit. In addition,this function computes the :call_id and records the appropriate host and port in the keys:broker_host and :broker_port.

Adding incoming broker supportGalIO_BrokerStruct *MGal_AddIncomingBrokering (Gal_Frame fr, int poll_ms, void *caller_data, int activate) The fr and poll_ms arguments are identical to those of GalIO_BrokerDataInInit; the caller_dataargument corresponds to the refptr argument there. In addition, this function accesses the hostand port and establishes the dispatch function for selecting among the type handlers. If activateis nonzero, GalIO_SetBrokerActive will be called on the broker structure.


438

Polling Stdin

Sometimes, the user might find it convenient to poll stdin periodically, as a way of getting inputfrom the user or as a way of allowing the user to control when certain events occur (such aswhen an audio device starts to listen). We provide a way of doing a non-blocking poll on stdin,and controlling the behavior when a line is read. This functionality is available with the MIT timedtask loop or without, but is not available on Win32.

We have provided a complex example of this functionality in our toy travel end-to-end "system".


Creating and activating a pollWe distinguish between creating a poll object and activating it. Although servers which poll stdinshould only accept a single connection at a time, the poll object should be stored in a connectionrather than a server, because of the type of its argument.

typedef Gal_Frame (*MGal_StdinFrameCreator )(char *str)

MGal_StdinPoll *MGalIO_CreateStdinPoll (char *prompt, GalIO_CommStruct *comm,MGal_StdinFrameCreator fn, int ms, int activate) The prompt is the string which will be printed out to stdout every time the poll is activated. The comm is a connection to the Hub. The fn is a function which is called whenever a line of text isread, which creates a frame which will be sent to the Hub as a new message. The ms are thenumber of milliseconds for the Galaxy timed task; 0 is the default (100 ms), -1 means don’t setup a timed task, and anything else is taken as a number of milliseconds. The activate argumentshould be 1 if the poll should be activated immediately, 0 if the poll will be activated later.

The fn may choose not to return a frame, in which case polling continues.

MGal_StdinPoll *MGalSS_EnvCreateStdinPoll (char *prompt, GalSS_Environment *env,MGal_StdinFrameCreator fn, int ms, int activate) The environment-aware version of MGalIO_CreateStdinPoll. See the documentation on session management.

439

void MGal_ActivateStdinPoll (MGal_StdinPoll *poll_struct) This function activates the poll, which means that the prompt is printed out and a timed task iscreated, if the polling milliseconds are not -1. If you’re in a dispatch function and you return thepoll structure from _GalSS_init_server, you can access the poll structure using GalIO_GetServerData:

Gal_Frame reinitialize(Gal_Frame f, void *server_data) { /* set up poll for stdin */ MGal_ActivateStdinPoll((MGal_StdinPoll *) GalIO_GetServerData(server_data)); return f; }

Typically, you’ll reactivate the poll every time you want to start listening again. It is possible tocall MGal_ActivateStdinPoll within the frame creation function to restart the poll immediately,whether or not the frame creation function returns a frame.

void MGal_SetStdinPollData (MGal_StdinPoll *poll_struct, void *data) This function sets the data for the poll object to some arbitrary element for later retrieval.

void *MGal_GetStdinPollData (MGal_StdinPoll *poll_struct) This function retrieves the arbitrarily stored data.

void MGal_SetStdinPollPrompt (MGal_StdinPoll *poll_struct, char *prompt) Resets the poll’s prompt. This does not take effect util MGal_ActivateStdinPoll is called the next time.

void MGal_FreeStdinPoll (MGal_StdinPoll *poll_struct) Use this function to free a poll structure.

Doing without the timed task loopAs with other polling functions, it is possible to use this functionality without the normal MITserver structure. Two problems must be addressed. First, we need a function which creates astdin poll not for a GalIO_ServerStruct, but for a GalIO_CommStruct. Second, we need a pollingfunction to use ourselves.

int MGal_PollStdin (MGal_StdinPoll *poll_struct) This is the polling function to use with the polling structure. It returns 1 if it read a linesuccessfully, 0 if it hasn’t encountered a line yet.


440

Alphabetical Index of API Calls

MGal_StdinPoll *MGalIO_CreateStdinPoll (char *prompt, GalIO_CommStruct *comm,MGal_StdinFrameCreator fn, int ms, int activate)

MGal_StdinPoll *MGalSS_EnvCreateStdinPoll (char *prompt, GalSS_Environment *env,MGal_StdinFrameCreator fn, int ms, int activate)

void MGal_ActivateStdinPoll (MGal_StdinPoll *poll_struct)

void MGal_AddBinaryDataType (int data_type, MGal_BinaryDataEncoder encoder,MGal_BinaryDataDecoder decoder)

void MGal_AddBrokerDTHandler (Gal_ObjectType dt, void *val, MGal_BrokerDTHandler h)

GalIO_BrokerStruct *MGal_AddIncomingBrokering (Gal_Frame fr, int poll_ms, void *caller_data, int activate)

GalIO_BrokerStruct *MGal_AddOutgoingBrokering (GalIO_CommStruct *gcomm,Gal_Frame fr, int poll_ms, int timeout_ms)

Gal_Frame MGal_CreateFullFrame (char *name, int type, int num_pairs, ... )

void MGal_FreeStdinPoll (MGal_StdinPoll *poll_struct)

void *MGal_GetOpaque (Gal_Frame fr, char *key, int *data_type)

void *MGal_GetOpaqueWarn (Gal_Frame fr, char *key, int data_type)

void *MGal_GetStdinPollData (MGal_StdinPoll *poll_struct)

void MGal_InitializeStatics (void )

Gal_Object MGal_OpaqueObject (void *obj, int data_type)

441

int MGal_PollStdin (MGal_StdinPoll *poll_struct)

void MGal_SetStdinPollData (MGal_StdinPoll *poll_struct, void *data)

void MGal_SetStdinPollPrompt (MGal_StdinPoll *poll_struct, char *prompt)


442

MITRE Tools

MITRE is providing a number of support tools for the community to use. Some of these toolswere developed in the context of the MITRE training course.

Process monitor Tutorial Reference

Unit tester Tutorial Reference

Server scripts MITRE log tools

Server scriptsA number of the MIT servers require certain environment variables to be set, and require you tobe in certain directories. MITRE provides a set of scripts which attempt to encapsulate theserequirements and make the appropriate options available on the command line.

Command line

<GC_HOME>/contrib/MITRE/tools/bin/run_server [-verbosity <int>] <server_and_args>...

This command takes a server and its arguments and runs the server in an appropriateenvironment. The primary advantage of this tool is to ensure that GAL_VERBOSE and SLS_VERBOSE is set to a reasonable value. SLS_VERBOSE is set for backward compatibility withMIT servers. The level is 3 by default; it can be controlled using the -verbosity argument.

Command line

<GC_HOME>/contrib/MITRE/tools/bin/run_silent_server <server_and_args>...

This command is identical to run_server except that it ensures that SLS_VERBOSE/GAL_VERBOSE are 0 (that is, minimal information will be printed out).

443


444

Process Monitor Reference

Command line<GC_HOME>/contrib/MITRE/tools/bin/process_monitor <flags> <GC_HOME>/contrib/MITRE/tools/bin/process_monitor <config_file>[<config_name_or_index>] [-- <arg>*] <GC_HOME>/contrib/MITRE/tools/bin/csh_process_monitor <flags>

Both these tools manage the behavior of the Hub and multiple servers (although they can beused to manage any arbitrary executable). The process_monitor is a Python GUI whichprovides a pane for each managed executable; the panes can be hidden and redisplayed, andeach process can be stopped and restarted. The csh_process_monitor is a version whichprovides a separate xterm for each process (so it does not provide some of the processmanagement support). If <GC_HOME>/templates/config.make has not enabled Python atcompilation time, process_monitor will be identical to csh_process_monitor .

There are two different ways of configuring the Python GUI: through command line flags orthrough a configuration file. You may use one or the other, but not both. For the csh tool, onlythe command line flag option is available.

You can also consult the process monitor tutorial.

Command line flagsThe relevant arguments for the two monitors differ slightly, but both processes will accept theentire set of arguments (and ignore the ones that aren’t relevant). The command line argumentsare "chunked"; there is a set of arguments for each process, for which the command lineargument -c terminates each "chunk".

445

ArgumentApplicable to Python

Appicableto csh

Description

-T <string> yes yes The title of the particular process

--open yes Should the pane start open? (default isclosed in Python, always open in csh)

--start yes Should the process be startedautomatically? (default is no in Python,always started in csh)

--keep_alive yes If specified, restart this process if it dies

--input_line yes Should a typein window be provided for thisprocess? (default is no in Python, alwaysavailable in csh)

--input_return yes Should a button be provided for inputting acarriage return? (default is no in Python,always available in csh)

-c <string> yes yesThe command line of the process. Thiscommand line should always be somethingthat can be preceded with "exec".

In addition to these arguments there are five other, global arguments:

--ncols <int> (Python): how many columns should the process panes be arranged in?This argument can appear anywhere in the command line. Default is 1. --compressed (Python): if specified, start up in a compressed configuration, with a singleprocess viewer and buttons to switch the viewer between processes. When --compressedis specified, --ncols becomes the number of button rows. This argument can appearanywhere in the command line. --clear_history <int> (Python): if specified, clears the history of each pane after thespecified number of lines are written. This means that long-running processes won’t grind toa halt because of Tcl update delays. --master_title <title> (Python): if specified, sets the title of the main window(default is "Process monitor"). --sleep <int> (csh): how many seconds should the xterm sleep before starting theprocess? This argument can be interspersed anywhere in the command line, and applies tothe current "chunk" and all subsequent chunks; this argument can be repeated. Initialdefault is 0.

In the Python version, the command line is echoed both in the history window and in a typeinwindow. You can modify the command line in the typein window and the changes will berecognized on restart. Also, the "keepalive" behavior and the window configuration can bereconfigured after startup via the process monitor GUI.

446

Command line flag exampleThis example shows how you might use the process monitor in a C shell to start up aconfiguration of servers to interact with a database. Each of the command lines is a single line,in spite of the wrapping imposed by your browser.

% setenv MITRE_ROOT $GC_HOME/contrib/MITRE % setenv DEMO_ROOT $MITRE_ROOT/examples/sql_explore % $MITRE_ROOT/tools/bin/process_monitor --master_title "DBInteraction" \ -T "DB" --open -c "$MITRE_ROOT/tools/bin/run_server$GC_HOME/contrib/MITRE/servers/bin/db_server -dblogin galaxy/galaxy" \ -T "SQL Monitor" --open --input_line -c"$MITRE_ROOT/servers/bin/c_ui -prompt ’Type a query: ’ -verbosity 0" \ -T "Hub" --open -c "$GC_HOME/bin/hub -pgm_file $DEMO_ROOT/bare_query.pgm"

This invocation of the process monitor creates three panes, all of which are open at startup, all ofwhich must be started by the user by hand.

The first is a database server, which is started by the run_server script. The second is a typein server which has a space to type in a line. The third is the Galaxy Communicator Hub executable.

Configuration fileAll the information which can be specified on the command line can be specified in theconfiguration file, and more. A configuration file consists of a number of process blocks, as wellas global information. Each of the process blocks may contain multiple processes, and eachprocess block corresponds to a possible invocation of the process monitor. If there are multipleprocess blocks, and the name or index of the desired block has not been specified on thecommand line, the process monitor will list the available blocks and exit.

All arguments after a double dash ("--") on the command line are passed to the configuration file,to be substituted into the command lines in the same way that the EXPAND: directive does.These arguments are made available as $1, $2, etc., or as the name listed in the correspondingposition in the ARGUMENTS: directive.

All entries in the configuration file are a single line long. Lines starting with a pound sign ("#") areignored, as are lines consisting entirely of whitespace. The first token in the line is the directive,and the remaining data is the value. By convention, all the directives end with a colon (":").

447

Directive Value ScopeStarts block

Description

PROCESS: yes process yes

The command line of the process. Thiscommand line should always besomething that can be preceded with"exec". Identical to -c .

PROCESS_KEEP_ALIVE: no process nof specified, restart this process if itdies. Identical to --keep_alive .

PROCESS_TITLE: yes process noThe title of the particular process.Identical to -T .

PROCESS_MONITOR_ARGS:yes process no

Additional command line arguments tospecify, such as --input_return ,etc. These will be digested as theywould in the corresponding set ofarguments describing a process on thecommand line.

TITLE: yesprocess block

yes

The name of the process block. Thisname can be used as the secondargument to the process monitor tospecify one of a number of processblocks in a configuration file. Identicalto --master_title .

EXPAND: yes global no

A pair of tokens, the first being a stringto be substituted, the second a stringto substitute for. This can be usedanywhere in the values for anysubsequent entry, but are primarilyintended to serve as shorthand in thecommand line. These expansions canalso be chained; that is, subsequent EXPAND: entries can refer to previous EXPAND: entries. Note that theexpansion of $GC_HOME to the root ofthe Galaxy Communicator installationis always available. By convention,these expansions start with a dollarsign ("$"), to evoke the notion of shell variables.

448

SUPPRESS_EXPANSIONS: no global no

If present, the expansions specified in EXPAND: will only be applied when thecommand is actually executed; fordisplay purposes, the unexpandedstring will be used

NUM_DIVISIONS: yes global noHow many columns should theprocess panes be arranged in?Identical to --ncols .

COMPRESSED: no global no

if specified, start up in a compressedconfiguration, with a single processviewer and buttons to switch theviewer between processes. When COMPRESSED: is specified, NUM_DIVISIONS: becomes thenumber of button rows. This argumentcan appear anywhere in the commandline. Identical to --compressed .

CLEAR_HISTORY: yes global no

If specified, clears the history of eachpane after the specified number oflines are written. This means thatlong-running processes won’t grind toa halt because of Tcl update delays.Identical to --clear_history .

ARGUMENTS: yes global no

If specified, provides a sequence oftokens which corresponds to thearguments expected after the doubledash ("--" ). No optional argumentsare permitted. If not all arguments arepresent, the process monitor will fail. Ifthere are extra arguments, they will beignored. The resulting boundarguments can be referred to by nameor position, with a prefixed dollar sign ($).

Configuration file exampleThe following configuration file does exactly the same thing as the command line flag example.Again, all directives are single lines, in spite of the wrapping in your browser:

449

EXPAND: $MITRE_ROOT $GC_HOME/contrib/MITRE EXPAND: $DEMO_ROOT $MITRE_ROOT/examples/sql_explore

TITLE: DB Interaction PROCESS: $MITRE_ROOT/tools/bin/run_server$MITRE_ROOT/servers/bin/db_server -dblogin galaxy/galaxy PROCESS_TITLE: DB PROCESS_MONITOR_ARGS: --open PROCESS: $MITRE_ROOT/servers/bin/c_ui -prompt ’Type a query: ’-verbosity 0 PROCESS_TITLE: SQL Monitor PROCESS_MONITOR_ARGS: --open --input_line PROCESS: $GC_HOME/bin/hub -pgm_file $DEMO_ROOT/bare_query.pgm PROCESS_MONITOR_ARGS: -T "Hub" --open


450

Unit Tester Reference

Unit tester

Command line

<GC_HOME>/contrib/MITRE/tools/bin/unit_tester --as_hub [--retry][--frames file] host port [welcome_frame] <GC_HOME>/contrib/MITRE/tools/bin/unit_tester --as_server [--retry][--frames file] [--ignore_reinitialize] [--service_type t]+ host port

This utility, implemented in Python (and only available if <GC_HOME>/templates/config.make has enabled Python at compilation time), provides agraphical "fake" Hub or server which connects to a server or Hub listener, respectively, locatedat host and port .

As a Hub, the unit tester accepts an optional welcome_frame which should be the contents ofa reinitialize message to be sent to the server. The name of the welcome frame will bechanged to reinitialize if necessary.

As a server, the unit tester must provide at least one value for --service_type whichcorresponds to a listener for that service type which the Hub has set up. If --ignore_reinitialize is present, the unit tester will not pop up a response dialogue forthe reinitialize message, but rather provide an automatic dummy reply.

For both, the --retry flag instructs the unit tester to keep trying to contact the listener if it’s notinitially available; the default behavior is to exit if the listener is not found. The --framesargument is a file which contains the printed representation of a single list of frames, such as:

( {c frame1 } {c frame2 } )

These frames are used as "seeds" for the frame history, which is mouseable and allows you toselect frames to use as new messages.


451

MITRE Log Tools

MITRE is providing a number of support tools for the community to use for manipulating logs andpreparing them for evaluation. These tools are useful for people generating logs from raw Hublogs, or writing their own XML. All these tools are written in Python.

This version of the log tools was originally distributed with GalaxyCommunicator 2.0.1p2; nochanges were made for version 2.1. However, a set of errors was discovered in the finalpreparations for the 2000 evaluation which led to patches both for 2.0.1 and 2.1. Forconvenience, a set of reference documents has been included with this distribution; however,you are encouraged to consult MITRE’s DARPA Communicator web site for the most current versions.

Background and common arguments XML log generation XML annotation XML human annotation stub generation Unifying automatic and human XML files XML summarization XML log validation XML scoring Batch generation of NIST sclite input Extraction of landmarks

Background and common argumentsAll the MITRE log analysis tools operate on sets of log directories . These log directories havean expected form, and all the tools in this suite of tools can be instructed about where to findthese log directories and what to expect their form to be.

All these tools are written in Python, and currently use the XML parser in the standard Pythondistribution. This XML parser isn’t really fast enough. In the near future, we’ll be looking intoother XML parsers currently being produced by the Python XML SIG, which should yieldconsiderably enhanced performance (as well as providing DTD validation). We apologize for the inconvenience.

452

http://fofoca.mitre.org/logstandard/index.html

Common command line

<GC_HOME>/contrib/MITRE/tools/bin/<tool_name> \ [--raw_txt_pat regexp] [--raw_xml_pat regexp] [--ann_xml_patregexp] [--human_xml_pat regexp] \ [--force] [--write_file] [--print_output] [--write_cache][--read_cache] \ [--help] \ [--start first_dir] [--end last_dir] log_root...

We’ll consider these arguments in groups.

Group 1: Directory structure

Each log directory must contain either a raw MIT text log, a raw XML log, or an annotated XMLlog. In addition, the tool suite recognizes a fourth distinguished file containing humanannotations. The directory may have an arbitrary number of additional files, but it must containno directories besides a directory called .cache , which is created by the tool suite under certain circumstances. The default names of the files in the log directory are:

File type Perl regular expressionCommand lineargument for override

raw MITtext log

^.+-\d{8,8}-\d{3,3}-hublog[.]txt$ --raw_txt_pat

raw XML log

^.+-\d{8,8}-\d{3,3}-hublog-raw[.]xml$ --raw_xml_pat

annotatedXML log

^.+-\d{8,8}-\d{3,3}-hublog-annotated[.]xml$ --ann_xml_pat

humanannotation file

^.+-\d{8,8}-\d{3,3}-hublog-human[.]xml$ or ^.+-\d{8,8}-\d{3,3}-human[.]xml$

--human_xml_pat

The default pattern for the raw MIT text log is identical to what the Hub logger produces, so ifyou’re using this tool suite to generate the raw and annotated XML, you don’t have to worryabout filenames, except to observe that your human annotations file should be generated byreplacing hublog.txt in your raw MIT text log filename with hublog-human.xml .

Overriding file names

If you are not using the Hub logger, you have the option of providing different regularexpressions for the four file types. You can use the arguments listed here to provide differentpatterns for the file types. By convention, the file names follow a consistent pattern, and you cantake advantage of this in your overrides. So for instance, if you provide a pattern only for the raw

453

MIT text log, and the tool finds the raw MIT text log but can’t find the other files based on thedefault patterns, it will also search for files based on the pattern you provided for the raw MITtext log. The algorithm works as follows:

The tool first searches for a file matching the default or override pattern for each file type. If the tool finds a raw MIT text log <base>.<ext> , it will attempt to find any missing filesaccording to the following patterns: <base>-raw.xml , <base>-annotated.xml,<base>-human.xml . If the tool finds a raw XML log <base>-raw.<ext> , it will attempt to find any missing filesaccording to the following patterns: <base>-annotated.xml,<base>-human.xml . If thefilename without extension does not end in -raw , it will treat the entire filename withoutextension as the base. If the tool finds an annotated XML log <base>-annotated.<ext> , it will attempt to find amissing human annotation file according to the pattern <base>-human.xml . If the filenamewithout extension does not end in -raw , it will treat the entire filename without extension asthe base.

So if your raw MIT text log pattern is ^mylog.text$ , the tool will also accept mylog-raw.xml , mylog-annotated.xml and mylog-human.xml as the names of the other files if it can’t findthe defaults.

Group 2: Processing directives

Each analysis step has preconditions on its execution, and may produce a log as a result of operating.

454

Analysis step Precondition Produces

XML log generation

raw MIT text lograw XML logconforming to the logfile standard

XML annotationraw XML log conforming to the logfile standard (+ rules)

annotated XML logconforming to the logfile DMA standard

XML summarization

annotated XML log conforming to the logfile DMA standard (+ human annotation file conforming to the human annotation logfile standard, if annotations arenot included in the annotated XML log)

XML log validation


XML scoring


If the required preconditions of each step are not met, the tool suite is capable of backwardchaining to generate it. So if you ask for an annotation, but all you currently have is the raw MITtext log, the annotation tool will force the generation of the raw XML log (or, rather, an internalrepresentation of it).

455

http://fofoca.mitre.org/members/CAC/human_annot.html

http://fofoca.mitre.org/members/CAC/logstandard-dmas-v3.html









http://fofoca.mitre.org/members/CAC/log-standard-versions.html


Flag Purpose

--force

Causes the tool to backward chain as far as possible, regardless of whichintermediate files are present. So if you’ve provided a raw MIT text log, allthe analysis steps will start with this file, no matter whether you’ve writtenout intermediate results.

--write_fileCauses the tool to write out intermediate results to anappropriately-named file, using the same naming generalizations that ituses to search for files. The default is not to write files.

--print_outputCauses the tool to print out each intermediate result to standard output.The default is not to print.

--read_cacheCauses the tool to read from a file in the .cache directory, if present. Thedefault is not to consult the cache.

--write_cacheCauses the tool to write intermediate results to a file in the .cache directory. The default is not to write intermediate results to the cache.

The cache exists because of the current speed problems with the default Python XML parser.The cache uses a fairly simple XDR encoding of the XML tree structure. This cache takes awhile to write (considerably longer than writing the XML files themselves), but currently seemsnoticeably faster to load than the raw XML (we haven’t done timings, though). If you’re doing agreat deal of batch processing, it may be valuable to write the cache for the annotated XML logand read from the cache for subsequent validation, summarization and scoring. The cache maygo away in the future, if the XML parsing speeds up sufficiently.

Group 3: --help

Prints out the command line and exits.

Group 4: Finding the log directories

The tools in this suite are capable of processing multiple log directories at the same time. Eachtool can accept a sequence of directory names as arguments, and it will search hierarchicallythrough those directories to find any directories which meet the qualifications for being log directories. As an example, assume your directory stack looks like this:

mylogs/ 19991120/ 000/ 001/ 002/ notes/ 19991123/ 000/ 001/ 002/

456

003/ 004/ notes/ 19991124/ 000/ 001/ 002/ notes/

Assume the notes/ directories don’t contain logs, but all the other leaf directories do. If youwant to produce raw XML for all the logs and write out the file, all you need to do is this:

% xmlize --write_file mylogs

which is equivalent to

% xmlize --write_file mylogs/19991120 mylogs/19991123 mylogs/19991124

The --start and --end command line arguments allow you to control begin and end pointswhen you provide a single directory root as an argument. The --start and --end values aretreated as subdirectories of the root, and only directory names which are alphanumericallygreater than or equal to the start and less than or equal to the end are processed. So if you justwant to do the last two days in your directory stack, you could use the following command line:

% xmlize --write_file --start 19991123 mylogs

If you wanted to start with the final log on the first day and end with the first log on the final day,you could use the following command line:

% xmlize --write_file --start 19991120/002 --end 19991124/000 mylogs

If you’re using the Hub logger, the alphanumeric order of the directory names in your directorystack is conveniently also the date order.

XML log generation

Command line

<GC_HOME>/contrib/MITRE/tools/bin/xmlize <common flags>[--default_user_version version_string] log_root...

This tool translates MIT logs into XML which conforms to the unannotated portion of the mostrecent version of the MITRE logfile standard proposal. The file <GC_HOME>/contrib/MITRE/tools/src/xmlizer/log_standard.dtd contains the XMLDTD which the result of this utility conforms to. The resulting log can be printed or written to anappropriately-named file using the common flags. All status messages are written to stderr.

457


The --default_user_version argument allows you to add a version name to the GC_LOGtag; this is normally inherited from the program file if you use LOGFILE_VERSION:, but thisallows you to provide a default version for this log if there is no LOGFILE_VERSION: in yourprogram file. The XML annotation tool is sensitive to this version name; if it is present, the namemust match one of the GC_LOG_VERSION tag contents specified in the rules file (describedbelow), but if it is absent, no checking will be done.

XML annotationOnce you have a raw XML file, you may annotate it using the XML annotation tool. This tooluses a special rules file.

Command line

<GC_HOME>/contrib/MITRE/tools/bin/xml_annotate <common flags> \ [--default_user_version version_string] [--rule_base rule_file] log_root...

This tool produces an annotated XML document. The resulting log can be printed or written to anappropriately-named file using the common flags. The raw XML log in each directory mustconform to the logfile standard. The rule_file is an XML file describing legal augmentationsof the raw XML. You may pass in as many rule files as you wish; the first one whose versionmatches will be used. The resulting document conforms to the logfile DMA standard. Note thatthis operation also takes the command line arguments for XML log generation, in casebackchaining is required. All status messages are written to stderr.

Here’s an example of a rules file (this is actually the current rules file for MITRE’s travel demo):

<RULES> <GC_LOG_VERSION>travel, version 2.0 cfone</GC_LOG_VERSION>

<RULE> <GC_MESSAGE name="filelog"> <GC_DATA key=":synth_log_filename" dtype="string" new:type_audio_file="system"/> </GC_MESSAGE> </RULE>

<RULE> <GC_MESSAGE name="filelog"> <GC_DATA key=":utt_log_filename" dtype="string" new:type_audio_file="user"/> </GC_MESSAGE> </RULE>

458



<RULE occurrences="first"> <GC_OPERATION name="speak_output" new:type_start_task="total"/> </RULE>

<RULE occurrences="first"> <GC_OPERATION name="nop" new:type_start_task="task"> <GC_DATA key=":listening_has_begun"/> </GC_OPERATION> </RULE>

<RULE occurrences="last"> <GC_OPERATION name="nop" new:type_end_task="true"> <GC_DATA key=":playing_has_ended"/> </GC_OPERATION> </RULE>

<RULE> <GC_OPERATION name="paraphrase_reply" new:type_new_turn="system"/> </RULE>

<RULE> <GC_OPERATION name="nop" new:type_new_turn="user"> <GC_DATA key=":listening_has_begun"/> </GC_OPERATION> </RULE>

<RULE> <GC_OPERATION name="speak_output"> <GC_DATA key=":reply_string" dtype="string" new:type_utt_text="system"/> </GC_OPERATION> </RULE>

<RULE> <GC_OPERATION name="gather"> <GC_DATA key=":parse_frame"> <GC_FRAME> <GC_DATA key=":input_string" dtype="string" new:type_utt_text="asr"/> </GC_FRAME> </GC_DATA> </GC_OPERATION> </RULE>

459

<RULE> <GC_OPERATION name="nop" new:type_end_utt="user"> <GC_DATA key=":recording_has_ended"/> </GC_OPERATION> </RULE>

<RULE> <GC_OPERATION name="nop" new:type_start_utt="system"> <GC_DATA key=":playing_has_begun"/> </GC_OPERATION> </RULE>

<RULE> <GC_OPERATION name="nop" new:type_end_utt="system"> <GC_DATA key=":playing_has_ended"/> </GC_OPERATION> </RULE>

</RULES>

Tags

We attempt to describe the rules file here.

460

Tag name Description Legal children

RULESToplevel tag in the rules file. Exactly one must appear.

RULE,GC_LOG_VERSION, OR

GC_LOG_VERSION

The content of this tag is log file version thisset of rules applies to. This must match thevalue of the logfile_version attribute ofthe GC_LOG tag in your raw XML log file, ifthat attribute is present. This tag may berepeated, in which case the rules are appliedif at least one of the versions matches.

RULE

An individual rule. The immediate children ofthis tag must match tags which share animmediate parent. The children of this tagdescribe the pattern which must match, andthe augmentations to apply. This tag may berepeated. It accepts the attributes occurrences="first" and occurrences="last" , which apply to thefirst and last tags to match the pattern, respectively.

GC_TURN,GC_OPERATION,GC_MESSAGE,GC_EVENT, GC_DATA

ORA disjunction of rules. The first rule thatsucceeds for a given tag in the raw XML logfile satisfies the OR.

RULE

Both tags and attributes may have the special namespace prefixes new: (material to add) or not: (material to ensure is not present), which we’ll discuss in a moment. First, let’s review thebehavior of the rule engine. The rule engine behaves as follows:

At each level, each sequence of children in the RULE pattern are matched with as manynonoverlapping child sequences as possible. So if your rule pattern contains <A><B/><C/></A> , each subpattern of <B/><C/> within the same <A> will match. The patterns enforce a subset match rather than an exact match; so the last rule in oursample set says that if you find a GC_OPERATION tag whose attributes include name="nop" , and one of whose immediate children is a GC_DATA tag whose attributesinclude key=":playing_has_ended" , add the attribute type_end_utt="system" tothe parent tag. You can add attributes anywhere in the pattern, and you can add as many attributes as youwant to a tag. The entire pattern in a given RULE must match for any of the attributes or tags to be added. Data is unordered; everything else is ordered. The effect of this generalization is that if allthe children are GC_DATA and GC_LIST (the only possible manifestations of dataaccording to the DTD), the children will be treated as unordered, unless the name of their

461

parent tag is GC_LIST (whose children are data but are ordered). If you specify a tag withmultiple order-sensitive children, the pattern children must be in the same order as the childrenof the tag the pattern matches. If you specify a tag with multiple order-insensitive children, thepattern children can be in any order relative to the children of the tag the pattern matches, but no"target" child can match more than one pattern child.

In the context of this behavior, the namespace prefixes have the following meaning:

Prefix Tag meaning Attribute meaning

new:Add this tag, with its attributes and children,at the end of the list of children of the tag itsparent matches.

Add this attribute-value pair to the tagwhich this attribute’s tag matches.

not: Fail if this tag matches an eligible child.Fail if this attribute-value pair matchesany attribute-value pair on the tag being checked.

For unordered tag children, the interpretation of the not: prefix is straightforward: if any childmatches, fail. So the pattern <GC_DATA a="b"/><NOT:GC_DATA a="b"/> will always fail.However, for ordered tag children, the interpretation is a little more subtle: in this case, theeligible children are only those children between the nearest positive matches (or the beginningor end of the list of children). So the pattern <GC_OPERATION a="b"/><NOT:GC_OPERATION a="b"/> may very well match a sequence of children which has exactly one tag in it of the form <GC_OPERATION a="b"/> .

Note that for both prefixes, the closing tags must also bear the prefixes.

The annotator will warn you if you specify a "new" attribute it doesn’t know about; it will not fail,however. For known attributes, on the other hand, the annotator enforces certain generalizationsabout what tags the attributes can be associated with. So tags which implicitly refer totimestamps must be associated with timestamped tags (GC_OPERATION, GC_MESSAGE,GC_EVENT), while tags which implicitly refer to data must be associated with GC_DATA tags.The annotator will print a notification if it encounters such a rules file, and then exit.

A note about GC_TURN

Currently, the GC_TURN tag is created in the raw XML log as a result of the execution of theHub builtin server operation builtin.increment_utterance . Because there’s norequirement that you use this operation, we can’t rely on GC_TURN tags. Accordingly, thesubsequent summarization, validation and scoring steps rely not on GC_TURN but on theplacement of the type_new_turn attribute that results from the annotation step. We anticipatethat GC_TURN will eventually vanish from the log standard DTD.

462

Matching and copying attributes and data

The values of attributes, and tag content, in the RULE patterns are actually Perl regularexpressions (which Python also recognizes). These expressions are required to match the entireattribute value or tag content. So if you want to tag any data element whose key starts with ":error" as a type_error_msg , you can write the following rule:

<RULE> <GC_DATA key=":error.*" new:type_error_msg="true"/> </RULE>

Perl regular expressions provide a grouping operation (using parentheses) which allowssubsequent reference to the match by numeric index. Python provides an extension of thisoperation by which these groups can be named, as well as numbered. You can use this facilityto copy information from one place in the pattern to another (this will be useful, for instance, torecord values for the type_prompt attribute signalling a system reprompt). The Python syntaxis as follows:

(?P<name>...)

The location to copy the matching text to can be specified using the "old:" namespace,referencing the pattern name. The scope of these names is the entire rule; it is an error to haveduplicate names in the patterns, or to reference an undefined pattern name.

So let’s say we record the prompted slot in our frames in the :prompted_slot key, as follows:

{c tts :output_string "What city would you like to depart from?" :prompted_slot "departure_city" }

We’d like to be able to copy the value of :prompted_slot to the value of type_prompt as follows:

<RULE> <GC_OPERATION> <GC_DATA key=":prompted_slot" dtype="string"> (?P<slot_name>.*) </GC_DATA> <GC_DATA key=":output_string" dtype="string" new:type_prompt="old:slot_name"/> </GC_OPERATION> </RULE>

The Python extension, regrettably, delimits the pattern name using angle brackets, which are (ofcourse) special in XML. So we actually need to use XML entity references in the pattern:

463

<RULE> <GC_OPERATION> <GC_DATA key=":prompted_slot" dtype="string"> (?P<slot_name>.*) </GC_DATA> <GC_DATA key=":output_string" dtype="string" new:type_prompt="old:slot_name"/> </GC_OPERATION> </RULE>

So the XML reader translates the entities to their textual representations (open and close anglebracket), and the postprocess which digests the rules recognizes the resulting string as adelimited name. This general approach applies to any other characters in Perl regularexpressions which are also special in XML.

Using order to conditionalize

Let’s say you have two operations, start_speaking_A and start_speaking_B , and theycan occur in any order, and the first one should indicate the start of system speech. Becauseannotations aren’t added unless the entire rule is matched, you can use order to conditionalizethe assignment of system speech onset, as follows:

<RULE> <GC_OPERATION name="start_speaking_A" new:type_start_utt="system"/> <GC_OPERATION name="start_speaking_B"/> </RULE>

<RULE> <GC_OPERATION name="start_speaking_B" new:type_start_utt="system"/> <GC_OPERATION name="start_speaking_A"/> </RULE>

Using negation

Consider a situation where the system reports beginning and end of audio output, but in somecases fails to report the end of audio output because a call hangup occurred. You can use not:to ensure that the hangup is recognized as the end of audio output just when there’s no actualend reported as follows:

<RULE> <GC_MESSAGE name="audio_status" new:type_start_utt="system"> <GC_DATA key=":playback_start"/> </GC_MESSAGE> </RULE>

<RULE> <GC_MESSAGE name="audio_status"> <GC_DATA key=":playback_start"/>

464

</GC_MESSAGE> <GC_MESSAGE name="audio_status" new:type_end_utt="system"> <GC_DATA key=":playback_end"/> </GC_MESSAGE> </RULE>

<RULE> <GC_MESSAGE name="audio_status"> <GC_DATA key=":playback_start"/> </GC_MESSAGE> <NOT:GC_MESSAGE name="audio_status"> <GC_DATA key=":playback_end"/> </NOT:GC_MESSAGE> <GC_MESSAGE name="call" new:type_end_utt="system"> <GC_DATA key=":hangup"/> </GC_MESSAGE> </RULE>

XML human annotation stub generationSometimes human annotations, such as speech transcription, need to be hand-generated. Wedo not yet have a GUI tool which will generate these files for you. However, we do provide astubber, which produces a "seed" containing the SR output as the default transcription, and aplace for the judgment of task completion.

Command line

<GC_HOME>/contrib/MITRE/tools/bin/xml_human_stubber <common flags> \ [--default_user_version version_string] [--rule_base rule_file] \ [--range [all_turns | overall_task | on_task]] log_root...

This tool produces a stub for human annotations from an annotated XML log. The annotatedXML log must conform to the logfile DMA standard. The generated stub conforms to the humanannotation logfile standard. This utility writes its stub to standard output (rather than saving itdirectly). All status messages are written to stderr. When you save the output of this utility, besure that you respect the filename conventions. Although this utility, like all the others, operateson an entire repository, you’ll almost certainly want to use it one log directory at a time.

Note that this operation also takes the command line arguments for XML log generation and annotation, in case backchaining is required.

465




Log ranges

This utility also takes the --range command line argument. This argument allows the user tospecify what subinterval of the session this utility (and other utilities) applies to. The default is overall_task unless otherwise indicated for the tool.

Value Description

all_turnsAll annotated data starting with the first new turn (marked with the type_new_turn attribute) will be considered.

overall_task

Only annotated data between the overall start and end task markers (type_start_task=true, type_start_task=total,type_end_task=true, type_end_task=total ), inclusive, will beconsidered. If a turn is underway when the start marker is encountered, thesubset of that turn which follows the marker will be considered.

on_task

Only annotated data between the on_task start and end markers (type_start_task=true, type_start_task=task,type_end_task=true, type_end_task=task ), inclusive, will beconsidered. If a turn is underway when the start marker is encountered, thesubset of that turn which follows the marker will be considered.

When you write a human annotations file, you can specify what the annotation is anchored to by turnid or by tidx . Here’s an example hand-generated human annotations file from a MITRE log:

<GC_LOG_ANNOTATIONS> <GC_SESSION> <GC_ANNOT type_task_completion="0"/> <GC_DATA type_utt_text="transcription" turnid="0" dtype="string"> I’d like to fly to Newark </GC_DATA> <GC_DATA type_utt_text="transcription" turnid="1" dtype="string"> St. Louis </GC_DATA> <GC_DATA type_utt_text="transcription" turnid="2" dtype="string"> Could I leave from New Orleans </GC_DATA> <GC_DATA type_utt_text="transcription" turnid="3" dtype="string"> I’d like to leave from New Orleans </GC_DATA> <GC_DATA type_utt_text="transcription" turnid="4" dtype="string"> I’d like to leave at midnight on Friday </GC_DATA> <GC_DATA type_utt_text="transcription" turnid="5" dtype="string"> I’d to leave around dinner on Friday

466

</GC_DATA> <GC_DATA type_utt_text="transcription" turnid="6" dtype="string"> I want to leave around six _p_m on Friday </GC_DATA> <GC_DATA type_utt_text="transcription" turnid="7" dtype="string"> Goodbye </GC_DATA> </GC_SESSION> </GC_LOG_ANNOTATIONS>

The value of the turnid attribute is anchored to the appropriate turn in the original XML file(and see the note about GC_TURN). The stub output for this same session would look like this:

<GC_LOG_ANNOTATIONS> <GC_SESSION> <GC_ANNOT type_task_completion="?"> <GC_DATA type_utt_text="transcription" dtype="string" turnid="0" tidx="35"> i+d like to fly to newark </GC_DATA> <GC_DATA type_utt_text="transcription" dtype="string" turnid="1" tidx="81"> say ends </GC_DATA> <GC_DATA type_utt_text="transcription" dtype="string" turnid="2" tidx="127"> could i will be from new orleans </GC_DATA> <GC_DATA type_utt_text="transcription" dtype="string" turnid="3" tidx="158"> i+d like to leave from new orleans </GC_DATA> <GC_DATA type_utt_text="transcription" dtype="string" turnid="4" tidx="211"> i+d like to leave around midnight on friday </GC_DATA> <GC_DATA type_utt_text="transcription" dtype="string" turnid="5" tidx="261"> i+d like to leave around dinner on friday </GC_DATA> <GC_DATA type_utt_text="transcription" dtype="string" turnid="6" tidx="296"> i want to leave at around six p m on friday </GC_DATA> <GC_DATA type_utt_text="transcription" dtype="string" turnid="7" tidx="341"> goodbye

467

</GC_DATA> </GC_SESSION> </GC_LOG_ANNOTATIONS>

Unifying automatic and human XML filesThere’s no requirement that human annotations be placed in a separate file; it’s just sometimeseasier to store the human annotations separately. All the tools in this package will operatereliably if no human annotations file is present; of course, in those circumstances, theappropriate human annotations need to be in the annotated XML file in order for summarization,validation and scoring to work appropriately. If, for some reason, you wish to merge a humanannotations file into an annotated XML file, you can use this utility.

Command line

<GC_HOME>/contrib/MITRE/tools/bin/xml_unify <common flags> \ [--default_user_version version_string] [--rule_base rule_file] log_root...

This tool produces an annotated XML log which contains all the auxiliary human annotationsfound in a human annotations file in the same directory. The annotated XML log must conform tothe logfile DMA standard. Note that this operation also takes the command line arguments for XML log generation and annotation, in case backchaining is required. If a human annotation fileis present in the log directory, it must conform to the human annotation logfile standard (you cangenerate a stub for this file using the human stubber).

As usual, in order to print out the result, use the --print common flag. In order to save the file,use the --write_file common flag. Note that the human annotations file is not removed .This utility adds the attribute human_annotations_included="1" to the output file, whichcauses utilities which use the annotated XML files to ignore the human annotations file.

XML summarizationOnce you have your annotated XML file, you can summarize it using this tool. This tool will workfor any conformant XML file, whether or not it was generated using the previous tools. If youhave a separate human annotations file, it will be folded in here.

Command line

<GC_HOME>/contrib/MITRE/tools/bin/xml_summarize <common flags> \ [--default_user_version version_string] [--rule_base rule_file] \ [--range [all_turns | overall_task | on_task]] log_root...

This tool summarizes an annotated XML log. The annotated XML log must conform to the logfileDMA standard. If a human annotation file is present in the log directory, it must conform to the human annotation logfile standard (you can generate a stub for this file using the human

468






stubber). This utility writes its summary to standard output. All status messages are written to stderr.

Note that this operation also takes the command line arguments for XML log generation and annotation, in case backchaining is required. In addition, this utility allows you to specify a log range.

Example

Here’s an example of a MITRE log summarized using this method. Notice that this summaryincludes references to the audio files, which are annotated using the optional type_audio_file="user" and type_audio_file="system" .

% xml_summarize --force --rule_base travel_2_0_rules.xml MITRE/19991101/004 Checking: MITRE/19991101/004 Reading raw Hub log: MITRE/19991101/004/travel_cfone-19991101-004-hublog.txt ...read. Converting to XML: MITRE/19991101/004/travel_cfone-19991101-004-hublog.txt ...converted. Reading rule file: travel_2_0_rules.xml Ignoring Comment element Ignoring Comment element ...read. Applying rules: travel_2_0_rules.xml ...succeeded. Reading human XML annotations: MITRE/19991101/004/travel_cfone-19991101-004-hublog-human.xml ...read. Resegmenting... Incorporating human annotations: MITRE/19991101/004/travel_cfone-19991101-004-hublog-human.xml ...resegmented. Mon Nov 1 1999 at 17:09:57.40: Task-specific portion started. Mon Nov 1 1999 at 17:10:08.83: Overall task started. Mon Nov 1 1999 at 17:11:58.32: Task-specific portion and overall task ended. Task completion status: not completed.

Turn 0 (system) Mon Nov 1 1999 at 17:09:57.36 to Mon Nov 1 1999 at 17:09:57.38: Newsystem turn began.

469

System said: Hi! Welcome to Mitre’s Travel demonstration. This call isbeing recorded for system development. You may hang up or ask for helpat any time. How can I help you? Mon Nov 1 1999 at 17:09:58.43: System started speaking. System audio file: travel_cfone-19991101-004-synth--01-001.wav Mon Nov 1 1999 at 17:10:08.66: System finished speaking.

Turn 1 (user) Mon Nov 1 1999 at 17:10:08.83: New user turn began.

Mon Nov 1 1999 at 17:10:09.36: User started speaking. User audio file: travel_cfone-19991101-004-000.wav Mon Nov 1 1999 at 17:10:12.77: User finished speaking. Recognizer heard: i+d like to fly to newark User said: I’d like to fly to Newark


System said: What city does the flight depart from? Mon Nov 1 1999 at 17:10:18.72: System started speaking. System audio file: travel_cfone-19991101-004-synth-000-001.wav Mon Nov 1 1999 at 17:10:20.83: System finished speaking.


Mon Nov 1 1999 at 17:10:21.80: User started speaking. User audio file: travel_cfone-19991101-004-001.wav Mon Nov 1 1999 at 17:10:24.14: User finished speaking. Recognizer heard: say ends User said: St. Louis


System said: i am sorry i did not understand that Mon Nov 1 1999 at 17:10:30.95: System started speaking. System audio file: travel_cfone-19991101-004-synth-001-001.wav Mon Nov 1 1999 at 17:10:33.43: System finished speaking.


470

Mon Nov 1 1999 at 17:10:34.66: User started speaking. User audio file: travel_cfone-19991101-004-002.wav Mon Nov 1 1999 at 17:10:37.89: User finished speaking. Recognizer heard: could i will be from new orleans User said: Could I leave from New Orleans


System said: could you please repeat that Mon Nov 1 1999 at 17:10:45.13: System started speaking. System audio file: travel_cfone-19991101-004-synth-002-001.wav Mon Nov 1 1999 at 17:10:46.91: System finished speaking.


Mon Nov 1 1999 at 17:10:47.92: User started speaking. User audio file: travel_cfone-19991101-004-003.wav Mon Nov 1 1999 at 17:10:51.36: User finished speaking. Recognizer heard: i+d like to leave from new orleans User said: I’d like to leave from New Orleans


System said: Can you provide the approximate departure or arrival time? Mon Nov 1 1999 at 17:10:55.96: System started speaking. System audio file: travel_cfone-19991101-004-synth-003-001.wav Mon Nov 1 1999 at 17:10:59.54: System finished speaking.


Mon Nov 1 1999 at 17:11:00.51: User started speaking. User audio file: travel_cfone-19991101-004-004.wav Mon Nov 1 1999 at 17:11:04.49: User finished speaking. Recognizer heard: i+d like to leave around midnight on friday User said: I’d like to leave at midnight on Friday


471

System said: i am sorry i could not understand you Mon Nov 1 1999 at 17:11:11.64: System started speaking. System audio file: travel_cfone-19991101-004-synth-004-001.wav Mon Nov 1 1999 at 17:11:13.99: System finished speaking.


Mon Nov 1 1999 at 17:11:14.72: User started speaking. User audio file: travel_cfone-19991101-004-005.wav Mon Nov 1 1999 at 17:11:18.88: User finished speaking. Recognizer heard: i+d like to leave around dinner on friday User said: I’d like to leave around dinner on Friday


System said: please try rephrasing what you said Mon Nov 1 1999 at 17:11:25.29: System started speaking. System audio file: travel_cfone-19991101-004-synth-005-001.wav Mon Nov 1 1999 at 17:11:27.43: System finished speaking.


Mon Nov 1 1999 at 17:11:28.40: User started speaking. User audio file: travel_cfone-19991101-004-006.wav Mon Nov 1 1999 at 17:11:32.87: User finished speaking. Recognizer heard: i want to leave at around six p m on friday User said: I want to leave around six _p_m on Friday


System said: I have no information about a flight from New Orleans thatdepart to Newark that depart friday around 6 o’clock p m . Mon Nov 1 1999 at 17:11:40.08: System started speaking. System audio file: travel_cfone-19991101-004-synth-006-001.wav Mon Nov 1 1999 at 17:11:47.26: System finished speaking.


Mon Nov 1 1999 at 17:11:48.50: User started speaking. User audio file: travel_cfone-19991101-004-007.wav Mon Nov 1 1999 at 17:11:50.62: User finished speaking.

472

Recognizer heard: goodbye User said: Goodbye


System said: Good bye. Thank you for using Mitre’s Travel demonstration. Mon Nov 1 1999 at 17:11:54.40: System started speaking. System audio file: travel_cfone-19991101-004-synth-007-001.wav Mon Nov 1 1999 at 17:11:58.32: System finished speaking.

XML log validationOnce you have your annotated XML file, you can validate it using this tool. This tool will work forany conformant XML file, whether or not it was generated using the previous tools. If you have aseparate human annotations file, it will be folded in here.

Command line

<GC_HOME>/contrib/MITRE/tools/bin/xml_log_validate <common flags> \ [--default_user_version version_string] [--rule_base rule_file] log_root...

This tool validates an annotated XML log. The annotated XML log must conform to the logfileDMA standard. If a human annotation file is present in the log directory, it must conform to the human annotation logfile standard (you can generate a stub for this file using the human stubber). This utility writes its summary to standard output. All status messages are written to stderr.

Note that this operation also takes the command line arguments for XML log generation and annotation, in case backchaining is required.

This utility now analyzes all turns. It reports which turns lie outside the task boundaries, as wellas more specific information about how to locate misplaced elements.

Example

Here’s an example of a MITRE log validated using this method. Notice that this log is notcompletely valid; we have not yet completed writing our rules to catch all the appropriatetermination conditions.

473




% xml_log_validate --force --rule_base travel_2_0_rules.xml MITRE/19991101/004 Checking: MITRE/19991101/004 Reading raw Hub log: MITRE/19991101/004/travel_cfone-19991101-004-hublog.txt ...read. Converting to XML: MITRE/19991101/004/travel_cfone-19991101-004-hublog.txt ...converted. Reading rule file: travel_2_0_rules.xml Ignoring Comment element Ignoring Comment element ...read. Applying rules: travel_2_0_rules.xml ...succeeded. Reading human XML annotations: MITRE/19991101/004/travel_cfone-19991101-004-hublog-human.xml ...read. Resegmenting... Incorporating human annotations: MITRE/19991101/004/travel_cfone-19991101-004-hublog-human.xml ...resegmented.

# # Log MITRE/19991101/004: #

Global elements:

No errors found.

Possibly misplaced turns:

Turn 17 (turnid 8, tidx 365) is completely outside the overall task boundaries.

Turn 0 (system)

No errors found.

Turn 1 (user)

No errors found.

474

Turn 2 (system)

No errors found.

Turn 3 (user)

No errors found.

Turn 4 (system)

No errors found.

Turn 5 (user)

No errors found.

Turn 6 (system)

No errors found.

Turn 7 (user)

No errors found.

Turn 8 (system)

No errors found.

Turn 9 (user)

No errors found.

Turn 10 (system)

No errors found.

Turn 11 (user)

475

No errors found.

Turn 12 (system)

No errors found.

Turn 13 (user)

No errors found.

Turn 14 (system)

No errors found.

Turn 15 (user)

No errors found.

Turn 16 (system)

No errors found.

Turn 17 (user)

Error for one of the attributes type_start_utt="user": At least one required.

XML scoringOnce you have your annotated XML file, you can score it using this tool. This tool will work forany conformant XML file, whether or not it was generated using the previous tools. If you have aseparate human annotations file, it will be folded in here.

Command line

<GC_HOME>/contrib/MITRE/tools/bin/xml_score <common flags> \ [--default_user_version version_string] [--format html | csv][--rule_base rule_file] \ [--range [all_turns | overall_task | on_task]] [--ref_postprocessscript] log_root...

476

This tool scores an annotated XML log. The annotated XML log must conform to the logfile DMA standard. If a human annotation file is present in the log directory, it must conform to the humanannotation logfile standard (you can generate a stub for this file using the human stubber). Thisutility writes its score output to standard output. All status messages are written to stderr. Thedefault format is HTML output; --format csv will produce comma-delimited output suitable forloading into a spreadsheet (no formula cells, unfortunately).

Note that this operation also takes the command line arguments for XML log generation and annotation, in case backchaining is required. In addition, this utility allows you to specify a log range. The default in this case is to report all intervals separately, rather than just the total task interval.

The scorer now produces all automatically computable DMAs yet (except word error rate, whichthe xml_nist_batch tool generates input for the NIST scoring package for). It also currentlyprovides output as HTML tables and comma-delimited flat files for spreadsheet input.

Postprocessing scripts

This utility allows you to specify a script to postprocess the user transcriptions via --ref_postprocess . This script should be a normal Unix executable, of any kind. It can beused to strip out transcription markup in preparation for scoring. The executable should loop,reading lines from stdin and writing the altered string to stdout, including a trailing newline. Besure to flush stdout after each write. See the extraction of landmarks for examples.

If no postprocess is provided, any data corresponding to use transcriptions will be postprocessedaccording to the NIST transcription guidelines for the Communicator program. This postprocessremoves any tokens delimited by square brackets ([ ]), since they are intended to indicate noiseunder the guidelines. If you provide your own postprocess, this step will be skipped; so be sureyou duplicate this behavior if you write your own postprocess.

Example

Here’s the command line for scoring a set of MITRE logs, and the corresponding HTML tableswhich result. Note the use of multiple rule files, due to the fact that the two logs processed hereuse different program file formats and thus have different log versions.

% xml_score --force --default_user_version "travel, version 1" \ --rule_base travel_1_0_rules.xml --rule_base travel_2_0_rules.xml MITRE Checking: MITRE Checking: MITRE/19991101 Checking: MITRE/19991101/004 Checking: MITRE/19990923 Checking: MITRE/19990923/000 Reading raw Hub log: MITRE/19991101/004/travel_cfone-19991101-004-hublog.txt ...read. Converting to XML:

477





MITRE/19991101/004/travel_cfone-19991101-004-hublog.txt ...converted. Reading rule file: travel_1_0_rules.xml ...read. Reading rule file: travel_2_0_rules.xml Ignoring Comment element Ignoring Comment element ...read. Applying rules: travel_1_0_rules.xml ...wrong version. Applying rules: travel_2_0_rules.xml ...succeeded. Reading human XML annotations: MITRE/19991101/004/travel_cfone-19991101-004-hublog-human.xml ...read. Resegmenting... Incorporating human annotations: MITRE/19991101/004/travel_cfone-19991101-004-hublog-human.xml ...resegmented. Reading raw Hub log: MITRE/19990923/000/travel_cfone-19990923-000-hublog.txt ...read. Converting to XML: MITRE/19990923/000/travel_cfone-19990923-000-hublog.txt ...converted. Applying rules: travel_1_0_rules.xml ...succeeded. Reading human XML annotations: MITRE/19990923/000/travel_cfone-19990923-000-hublog-human.xml ...read. Resegmenting... Incorporating human annotations: MITRE/19990923/000/travel_cfone-19990923-000-hublog-human.xml ...resegmented.

Complete sessions (on task)

Duration (secs)

Turnsin

interval

Error messages

Help messages

Responselatency (mean

in secs/variance)

Userwords(noise

stripped)

System words

Prompt percentage

Numberof

reprompts

MITRE/19990923/000 58.71 8 1 0 5.35/0.1189 21 62 0.25 0

Overall 58.71 8.00 1.00 0.00 5.35/0.1189 21 62 0.25 0.00

478

Complete sessions (total task)

Duration (secs)

Turnsin

interval

Error messages

Help messages


in secs/variance)

Userwords(noise

stripped)

System words

Prompt percentage

Numberof

reprompts

MITRE/19990923/000 70.35 9 1 0 5.35/0.1189 21 92 0.20 0

Overall 70.35 9.00 1.00 0.00 5.35/0.1189 21 92 0.20 0.00

Complete sessions (all turns)

Turnsin

interval

Error messages

Help messages


in secs/variance)

Userwords(noise

stripped)

System words

Prompt percentage

Numberof

reprompts

MITRE/19990923/000 10 1 0 5.35/0.1189 21 92 0.20 0

Overall 10.00 1.00 0.00 5.35/0.1189 21 92 0.20 0.00

Incomplete sessions (on task)

Duration (secs)

Turnsin

interval

Error messages

Help messages

Userwords(noise

stripped)

System words

Prompt percentage

Numberof

reprompts

MITRE/19991101/004 120.92 1 0 0 0 30 0.00 0

Overall 120.92 1.00 0.00 0.00 0 30 0.00 0.00

Incomplete sessions (total task)

Duration (secs)

Turnsin

interval

Error messages

Help messages


in secs/variance)

Userwords(noise

stripped)

System words

Prompt percentage

Numberof

reprompts

MITRE/19991101/004 109.49 17 0 0 6.14/1.4842 47 105 0.22 0

Overall 109.49 17.00 0.00 0.00 6.14/1.4842 47 105 0.22 0.00

Incomplete sessions (all turns)

Turnsin

interval

Error messages

Help messages


in secs/variance)

Userwords(noise

stripped)

System words

Prompt percentage

Numberof

reprompts

MITRE/19991101/004 18 0 0 6.14/1.4842 47 105 0.22 0

Overall 18.00 0.00 0.00 6.14/1.4842 47 105 0.22 0.00

479

All sessions (on task) Task

completedDuration

(secs)Turns

in interval

Meanuser

wordsper turn(noise

stripped)

Meansystemwords

per turn

Error messages

Help messages


in secs/variance)

Userwords(noise

stripped)

System words

Prompt percentage

Numberof

reprompts

Meansystem

utteranceduration (secs)

Meansystem

turnduration (secs)

Meansystem

turnsilence (secs)

Systemturn

silencepct

MITRE/19990923/000 1 58.71 8 5.25 15.50 1 0 5.35/0.1189 21 62 0.25 0 5.35 5.35 0.00 0.00

MITRE/19991101/004 0 120.92 1 - 30.00 0 0 - 0 30 0.00 0 10.23 10.23 0.00 0.00

Overall 0.50 89.82 4.50 5.25 18.40 0.50 0.00 5.35/0.1189 21 92 0.20 0.00 6.33 6.33 0.00 0.00

All sessions (total task) Task

completedDuration

(secs)Turns

in interval

Meanuser

wordsper turn(noise

stripped)

Meansystemwords

per turn

Error messages

Help messages


in secs/variance)

Userwords(noise

stripped)

System words

Prompt percentage

Numberof

reprompts

Meansystem


Meansystem

turnduration (secs)

Meansystem

turnsilence (secs)

Systemturn

silencepct

MITRE/19990923/000 1 70.35 9 5.25 18.40 1 0 5.35/0.1189 21 92 0.20 0 6.33 6.33 0.00 0.00

MITRE/19991101/004 0 109.49 17 5.88 11.67 0 0 6.14/1.4842 47 105 0.22 0 3.97 3.97 0.00 0.00

Overall 0.50 89.92 13.00 5.67 14.07 0.50 0.00 5.88/1.1692 68 197 0.21 0.00 4.82 4.82 0.00 0.00

All sessions (all turns) Task

completedTurns

in interval

Meanuser

wordsper turn(noise

stripped)

Meansystemwords

per turn

Error messages

Help messages


in secs/variance)

Userwords(noise

stripped)

System words

Prompt percentage

Numberof

reprompts

Meansystem


Meansystem

turnduration (secs)

Meansystem

turnsilence (secs)

Systemturn

silencepct

MITRE/19990923/000 1 10 5.25 18.40 1 0 5.35/0.1189 21 92 0.20 0 6.33 6.33 0.00 0.00

MITRE/19991101/004 0 18 5.88 11.67 0 0 6.14/1.4842 47 105 0.22 0 3.97 3.97 0.00 0.00

Overall 0.50 14.00 5.67 14.07 0.50 0.00 5.88/1.1692 68 197 0.21 0.00 4.82 4.82 0.00 0.00

Batch generation of NIST sclite inputOnce you have your annotated XML file, you can generate input for the NIST sclite scoringtool using the xml_nist_batch tool. This tool will work for any conformant XML file, whether ornot it was generated using the previous tools. If you have a separate human annotations file, itwill be folded in here.

Command line

<GC_HOME>/contrib/MITRE/tools/bin/xml_nist_batch <common flags> \ [--default_user_version version_string] [--rule_base rule_file] \ [--hyp_postprocess script] [--ref_postprocess script][--wav_postprocess script] \ [--outdir file_location] [--outprefix file_prefix] \ [--range [all_turns | overall_task | on_task]] log_root...

This tool generates three files, given an annotated XML log. The annotated XML log mustconform to the logfile DMA standard. If a human annotation file is present in the log directory, itmust conform to the human annotation logfile standard (you can generate a stub for this fileusing the human stubber).

480




This utility saves three files: a speech hypothesis file, a speech transcription file, and (possibly) afile of audio files. The --outprefix names the file prefix, if present. The default file prefix is sr .The three files are named:

<file prefix>.hyp (speech hypothesis) <file prefix>.ref (speech transcriptions) <file prefix>.wav (names of the audio files, if the optional MITRE landmarks for audiofiles are used)

These files are written to:

the directory named by --outdir , or the directory corresponding to the log root, if exactly one log root was given, or the current working directory.

If any of the arguments --hyp_postprocess , --ref_postprocess , or --wav_postprocess are present, the hypothesis, transcription, or audio file pathname will bepassed through the executable named by the corresponding argument. These executables areidentical in form to the postprocessing scripts described above. The default for --ref_postprocess is the same as in xml_score ; the defaults for the others are not to do postprocessing.

The hypothesis and transcription files can be passed to the sclite utility as follows:

% xml_nist_batch --force --default_user_version "travel, version 1" \ --rule_base travel_1_0_rules.xml --rule_base travel_2_0_rules.xml \ --range all_turns MITRE Checking: MITRE Checking: MITRE/19991101 Checking: MITRE/19991101/004 Checking: MITRE/19990923 Checking: MITRE/19990923/000 Reading raw Hub log: MITRE/19991101/004/travel_cfone-19991101-004-hublog.txt ...read. Converting to XML: MITRE/19991101/004/travel_cfone-19991101-004-hublog.txt ...converted. Reading rule file: travel_1_0_rules.xml ...read. Reading rule file: travel_2_0_rules.xml Ignoring Comment element

481

Ignoring Comment element ...read. Applying rules: travel_1_0_rules.xml ...wrong version. Applying rules: travel_2_0_rules.xml ...succeeded. Reading human XML annotations: MITRE/19991101/004/travel_cfone-19991101-004-hublog-human.xml ...read. Resegmenting... Incorporating human annotations: MITRE/19991101/004/travel_cfone-19991101-004-hublog-human.xml ...resegmented. Reading raw Hub log: MITRE/19990923/000/travel_cfone-19990923-000-hublog.txt ...read. Converting to XML: MITRE/19990923/000/travel_cfone-19990923-000-hublog.txt ...converted. Applying rules: travel_1_0_rules.xml ...succeeded. Reading human XML annotations: MITRE/19990923/000/travel_cfone-19990923-000-hublog-human.xml ...read. Resegmenting... Incorporating human annotations: MITRE/19990923/000/travel_cfone-19990923-000-hublog-human.xml ...resegmented. % sclite -r MITRE/sr.ref -h MITRE/sr.hyp -i rm sclite: 2.2 TK Version 1.2 Begin alignment of Ref File: ’MITRE/sr.ref’ and Hyp File: ’MITRE/sr.hyp’ Alignment# 4 for speaker travel_cfone19990923000hublogannotated Alignment# 8 for speaker travel_cfone19991101004hublogannotated

SYSTEM SUMMARY PERCENTAGES by SPEAKER

,------------------------------------------------------------------------------------------------.

| MITRE/sr.hyp | |------------------------------------------------------------------------------------------------|

| SPKR | # Snt # Wrd | Corr Sub Del Ins Err S.Err | |----------------------------------------+-------------+-----------------------------------------|

| travel_cfone19990923000hublogannotated | 4 21 | 76.2 23.8

482

0.0 9.5 33.3 100.0 | |----------------------------------------+-------------+-----------------------------------------|

| travel_cfone19991101004hublogannotated | 8 47 | 80.9 19.1 0.0 6.4 25.5 87.5 | |================================================================================================|

| Sum/Avg | 12 68 | 79.4 20.6 0.0 7.4 27.9 91.7 | |================================================================================================|

| Mean | 6.0 34.0 | 78.5 21.5 0.0 8.0 29.4 93.8 | | S.D. | 2.8 18.4 | 3.3 3.3 0.0 2.2 5.5 8.8 | | Median | 6.0 34.0 | 78.5 21.5 0.0 8.0 29.4 93.8 | ‘------------------------------------------------------------------------------------------------’

Successful Completion

Extraction of landmarksOnce you have your annotated XML file, you can extract the values of arbitrary landmarks usingthe xml_extract_landmarks tool. This tool will work for any conformant XML file, whether ornot it was generated using the previous tools. If you have a separate human annotations file, itwill be folded in here.

Command line

<GC_HOME>/contrib/MITRE/tools/bin/xml_extract_landmarks <common flags> \ [--default_user_version version_string] [--rule_base rule_file] \ --landmark attr=val[:script] [--landmark attr=val[:script]]* \ [--outdir file_location] [--outprefix file_prefix] log_root...

For each annotated XML log, this tool generates a file containing the landmarks specified, givenan annotated XML log. The annotated XML log must conform to the logfile DMA standard. If ahuman annotation file is present in the log directory, it must conform to the human annotationlogfile standard (you can generate a stub for this file using the human stubber).


You must specify at least one landmark. Each landmark is specified as shown in the commandline; so if you want to extract all instances of system text, you’d specify --landmark type_utt_text=system . If you’d like the landmark postprocessed before it’s written out, youcan specify a postprocess executable.

483




These files are written to:

the directory named by --outdir , or the directory corresponding to the log root, if exactly one log root was given, or the current working directory.

Each file is named <file_prefix>-<annotated_XML_filename>.txt . The default fileprefix is landmark-<UTC_time> . Because the annotated XML filename is used to differentiatebetween sessions, this utility will probably not work too well if the names of your annotated XMLfiles are all the same (e.g., if you’ve overriden the defaults and named all your files annotated.xml ).

Here’s an example of collecting alternating user and system text using this utility. The scripts add_u.py and add_s.py add the prefix "U: " and "S: " , respectively, to each line. Thescripts are displayed at the end of the example.

% xml_extract_landmarks --landmarktype_utt_text=transcription:/tmp/add_u.py \ --landmark type_utt_text=system:/tmp/add_s.py --force \ --default_user_version "travel, version 1" \ --rule_base travel_1_0_rules.xml --rule_base travel_2_0_rules.xml \ --outprefix landmark MITRE Checking: MITRE Checking: MITRE/19991101 Checking: MITRE/19991101/004 Checking: MITRE/19990923 Checking: MITRE/19990923/000 Reading raw Hub log: MITRE/19991101/004/travel_cfone-19991101-004-hublog.txt ...read. Converting to XML: MITRE/19991101/004/travel_cfone-19991101-004-hublog.txt ...converted. Reading rule file: travel_1_0_rules.xml ...read. Reading rule file: travel_2_0_rules.xml Ignoring Comment element Ignoring Comment element ...read. Applying rules: travel_1_0_rules.xml ...wrong version. Applying rules: travel_2_0_rules.xml ...succeeded. Reading human XML annotations: MITRE/19991101/004/travel_cfone-19991101-004-hublog-human.xml ...read. Resegmenting...

484

Incorporating human annotations: MITRE/19991101/004/travel_cfone-19991101-004-hublog-human.xml ...resegmented. Reading raw Hub log: MITRE/19990923/000/travel_cfone-19990923-000-hublog.txt ...read. Converting to XML: MITRE/19990923/000/travel_cfone-19990923-000-hublog.txt ...converted. Applying rules: travel_1_0_rules.xml ...succeeded. Reading human XML annotations: MITRE/19990923/000/travel_cfone-19990923-000-hublog-human.xml ...read. Resegmenting... Incorporating human annotations: MITRE/19990923/000/travel_cfone-19990923-000-hublog-human.xml ...resegmented. % more MITRE/landmark-* :::::::::::::: MITRE/landmark-travel_cfone-19990923-000-hublog-annotated.txt :::::::::::::: S: Hi! Welcome to Mitre’s Travel demonstration. This call is beingrecorded for system development. You may hang up or ask for help at any time. Howcan I help you? U: i+d like an flight from boston to detroit S: Can you provide the approximate departure or arrival time? U: i+d like to leave at seven a m S: north west flight 381 departs Boston at seven thirty a m andarrives Detroit at nine forty nine a m . Is there something else I can do for you? U: is that flight serve breakfast S: That flight does not serve breakfast. What else would you like to know? U: goodbye now S: I heard your words, but I can’t understand them correctly. :::::::::::::: MITRE/landmark-travel_cfone-19991101-004-hublog-annotated.txt :::::::::::::: S: Hi! Welcome to Mitre’s Travel demonstration. This call is beingrecorded for system development. You may hang up or ask for help at any time. Howcan I help you? U: I’d like to fly to Newark

485

S: What city does the flight depart from? U: St. Louis S: i am sorry i did not understand that U: Could I leave from New Orleans S: could you please repeat that U: I’d like to leave from New Orleans S: Can you provide the approximate departure or arrival time? U: I’d like to leave at midnight on Friday S: i am sorry i could not understand you U: I’d to leave around dinner on Friday S: please try rephrasing what you said U: I want to leave around six _p_m on Friday S: I have no information about a flight from New Orleans that depart toNewark t hat depart friday around 6 o’clock p m . U: Goodbye S: Good bye. Thank you for using Mitre’s Travel demonstration. % more /tmp/add_u.py /tmp/add_s.py :::::::::::::: add_u.py :::::::::::::: #!/usr/bin/python

import string, sys

r = string.strip(sys.stdin.readline()) while r: print "U:", r sys.stdout.flush() r = string.strip(sys.stdin.readline()) :::::::::::::: add_s.py :::::::::::::: #!/usr/bin/python

import string, sys

r = string.strip(sys.stdin.readline()) while r: print "S:", r sys.stdout.flush() r = string.strip(sys.stdin.readline())


486

MITRE Log Reference Documents

MITRE is providing a number of reference documents for use with its log tools. Some of theselog tools implement standards which are being used in the evaluation process in the DARPACommunicator program. The most current version of these reference documents can always befound here.

MITRE logfile standard (version 14) (cf. the xmlize utility) Logfile implementation of DMA metrics (version 4) (cf. the xml_annotate utility) Sample human annotations file (cf. the xml_human_stubber utility) Sample automatic annotation rules file (cf. the xml_annotate utility) XML DTD for MITRE logfile standard (version 3) (cf. the xmlize utility) XML DTD for human annotations file (version 2) (cf. the xml_human_stubber utility)

Note that there is no DTD for the automatic annotation rules files, because the rules frameworkfor automatic annotation allows arbitrary attributes in the new: and not: namespaces, whichcannot be validated against a DTD


487

http://fofoca.mitre.org/logstandard

Toy Travel "System"

In order to illustrate how to assemble an end-to-end system, we’ve constructed an examplewhich operates according to a preset sequence of messages. Below, we describe theconfiguration of servers, how to run them, and what they illustrate.

The configuration file The servers The program file Running the demo Examining the logs

The configuration fileThe configuration file which is distributed with the distribution looks like this:

( ( {c dialogue_output :frame {c greeting } } {c generator_output :output_string "Welcome to Communicator. How may I help you?" } {c synthesizer_output :sample_rate 8000 :encoding_format "linear16" :num_samples 14520 } )

( {c audio_input :sample_rate 8000 :encoding_format "linear16" :num_samples 16560 } {c text_input :input_string "I WANT TO FLY TO LOS ANGELES" } {c recognizer_output :input_string "I WANT TO FLY LOS ANGELES" } {c parser_output :frame {c flight :destination "LOS ANGELES" } }

488

{c dialogue_output :frame {c query_departure } } {c generator_output :output_string "Where are you traveling from?" } {c synthesizer_output :sample_rate 8000 :encoding_format "linear16" :num_samples 9560 } )

( {c audio_input :sample_rate 8000 :encoding_format "linear16" :num_samples 4580 } {c text_input :input_string "BOSTON" } {c recognizer_output :input_string "BOSTON" } {c parser_output :frame {c flight :city "BOSTON" } } {c backend_query :sql_query "select airline, flight_number, departure_datetimefrom flight_table where departure_aiport = ’BOS’ and arrival_airport =’LAX’" } {c backend_output :column_names ( "airline" "flight_number" "departure_datetime" ) :nfound 2 :values ( ( "AA" "115" "1144" ) ( "UA" "436" "1405" ) ) } {c dialogue_output :frame {c db_result :column_names ( "airline" "flight_number""departure_datetime" ) :tuples ( ( "AA" "115" "1144" ) ( "UA" "436" "1405" ) ) } } {c generator_output :output_string "American Airlines flight 115 leaves at 11:44AM, and United flight 436 leaves at 2:05 PM" } {c synthesizer_output :sample_rate 8000 :encoding_format "linear16" :num_samples 35068 } )

)

489

The file is a list of lists. Each list corresponds to the consequences of a single user input (either"typed input" or "spoken input"). Each user input list is a sequence of frames. The name of eachframe corresponds to a processing stage, and each processing stage has a set of key-valuepairs which are recognized. Here are the steps and keys:

Step Frame name keys

Audio gesture from theAudio server to Recognizer

audio_input:sample_rate (integer), :encoding_format (string), :num_samples (integer)

Text gesture fromUI server to Parser

text_input :input_string (string)

From Recognizer to Parser

recognizer_output :input_string (string)

From Parser to Dialogue

parser_output :frame (frame)

Query from Dialogueto Backend

backend_query :sql_query (string)

Result of query to Backend

backend_output:column_names (list of strings), :nfound(integer), :values (list of lists of strings)

New message to userfrom Dialogue serverto to Generator

dialogue_output :frame (frame)

From Generator, toSynthesizer or UI

generator_output :output_string (string)

From Synthesizer to Audio

synthesizer_output:sample_rate (integer), :encoding_format (string), :num_samples (integer)

Each list is treated as a single set of actions. Each server works by looking for an action setwhich contains the data corresponding to the input it gets in the appropriate step. If it finds theappropriate input, and the action set also contains an appropriate output, it produces the output.

For the Parser, Generator and Backend servers, the process is maximally simple, sincethere’s only one possible input and one possible output for each of these servers. For the Dialogue server, its output depends on whether the action set has a query to thebackend or not. If it does, it will query the backend before sending something to the user;otherwise, it will send something to the user regardless. For the UI and Audio servers, each searches the action list in order, looking for actionswhich contain audio or text gestures. When the server runs out of qualifying action sets, it

490

exits. Whether the output from the generator goes to the UI or to the Audio server depends onwhich server connected to that session. This is handled in the Hub programs OpenAudioSession and OpenTextSession , below.

The serversThis demo consists of several sample servers. None of these servers actually recognize speech,or parse text, or track dialogue; they illustrate only the ways such servers might interact, andhow these servers might be constructed (especially in the more complex cases). The servers are:

A recognizer A synthesizer A parser A generator A database A dialogue module An audio server A text in/text out module

We’ve made a serious attempt to make the wrappers for these servers (Parser.c , Generator.c , etc.) into plausible wrappers for the appropriate functionality. The file component_engine.c contains the code which "implements" the appropriate functionality anddefines a plausible API for it. This file also contains the code which digests the configuration file.

Parser, generator, database, dialogueAll these servers work fairly straightforwardly. The first three servers simply receive a frame andreturn a response. The Dialogue server both exercises server-to-server subdialogues using GalSS_EnvDispatchFrame() and issues the result as a new message to the Hub.

Recognizer, synthesizerBoth these servers use brokering to interact with the audio server. These servers exchange 16bit samples with the audio server, consisting of random data whose length is dictated by theconfiguration file. These files might serve as useful templates for constructing brokered serversof this type. Notice in the recognizer that when audio data is brokered, whatever processing isperformed on the data (such as running the recognizer and sending the results to the Hub) hasto happen in a broker callback or a broker finalizer, which means that it must contact the Hubwith a new message when it’s done.

491

Audio, text in/text outThese servers are the most complex, because they must monitor some input source other thanthe Hub. We’ve used the MITRE stdin polling mechanism to stand in for the audio or GUI inputsource. When you start the server, it will connect to the Hub; each you press the carriage return,it will send the next predetermined input; and so on until it runs out of inputs, at which point it willdisconnect and shut down. You can restart the server and rerun these inputs as many times asyou like; each time a separate session will be created.

The program fileThe toy travel demo uses the following program file, which we have annotated extensively.

;; The value of LOG_VERSION: will be used in ;; the annotation rules.

LOG_VERSION: "toy travel, version 1"

;; Use extended syntax (new in version 3.0).


;; This means that the log directory hierarchy ;; will start in the directory where the Hub is run.

LOG_DIR: .

;; Both audio and UI will be HUB clients, and ;; they will share a port.

SERVICE_TYPE: Audio CLIENT_PORT: 2800 OPERATIONS: Play

SERVICE_TYPE: UI CLIENT_PORT: 2800 OPERATIONS: Print

SERVER: Parser HOST: localhost PORT: 10000 OPERATIONS: Parse

SERVER: Dialogue HOST: localhost PORT: 18500 OPERATIONS: DoDialogue DoGreeting

492

SERVER: Generator HOST: localhost PORT: 16000 OPERATIONS: Generate

SERVER: Backend HOST: localhost PORT: 13000 OPERATIONS: Retrieve

SERVER: Recognizer HOST: localhost PORT: 11000 OPERATIONS: Recognize

SERVER: Synthesizer HOST: localhost PORT: 15500 OPERATIONS: Synthesize

SERVER: IOMonitor HOST: localhost PORT: 10050 OPERATIONS: ReportIO

;; We use four crucial functions in the Builtin server.

SERVER: Builtin OPERATIONS: new_session end_session call_program nop hub_break

;; For logging, I will timestamp everything. Since ;; I’m also logging all the relevant keys, I really ;; don’t need to timestamp, since they’ll be added ;; automatically, but it’s harmless and good practice.

TIMESTAMP: Play Print Parse DoDialogue DoGreeting Generate Retrieve \ Recognize Synthesize new_session end_session call_program \ FromAudio OpenAudioSession OpenTextSession \ FromRecognizer FromUI UserInput \ FromDialogue DBQuery FromSynthesizer

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; ;; AUDIO INPUT ;; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

493

;; This first program handles input from the audio ;; server. The :host and :port and :call_id are ;; typically the information required to establish ;; a brokering connection.

PROGRAM: FromAudio

RULE: :host & :port & :call_id --> Recognizer.Recognize IN: :host :port :encoding_format :sample_rate :call_id LOG_IN: :host :port :encoding_format :sample_rate :call_id OUT: none!

;; This program handles opening ;; an audio session. It marks audio available for the session ;; by using a key whose prefix is :hub_session_.

PROGRAM: OpenAudioSession

;; Notice that we use the nop dispatch function to ;; "host" a call to OUT: to set a session variable.

RULE: --> Builtin.nop OUT: ($in(:audio_available session) 1)

;; Now we create a session. This is not technically ;; necessary, since the audio server creates a ;; session by virtue of how it connects using the ;; listener-in-Hub functionality.

RULE: --> Builtin.new_session

;; Finally, I kick off the system greeting.

RULE: --> Dialogue.DoGreeting

;; This program dispatches recognizer results to ;; the general program which handles user input.


RULE: :input_string --> Builtin.call_program IN: (:program "UserInput") :input_string OUT: none!

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; ;; TEXT INPUT ;; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

494

;; This program handles opening ;; a text in/text out session. It creates a session ;; and kicks off the user greeting.

PROGRAM: OpenTextSession

RULE: --> Builtin.new_session

RULE: --> Dialogue.DoGreeting

;; This function relays the typed input to the ;; main body of the input processing.

PROGRAM: FromUI

RULE: :input_string --> Builtin.call_program IN: (:program "UserInput") :input_string

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; ;; MAIN INPUT BODY ;; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; This program handles the main input text processing. ;; It passes the result of the parsing to the dialogue ;; manager, and if there’s a response, it relays it ;; to the output processing program.

PROGRAM: UserInput


RULE: :input_string --> Parser.Parse IN: :input_string OUT: :frame LOG_IN: :input_string LOG_OUT: :frame

;; We’re not waiting for a reply, so the errors will be ;; signalled by new messages.

RULE: :frame --> Dialogue.DoDialogue IN: :frame LOG_IN: :frame OUT: none!

495

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; ;; MAIN OUTPUT BODY ;; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; This program handles the response from the dialogue ;; manager which may take one of a number of ;; forms (database tuples to be described, or perhaps ;; an already-formatted frame).

PROGRAM: FromDialogue

RULE: :output_frame --> Generator.Generate IN: :output_frame OUT: :output_string LOG_IN: :output_frame LOG_OUT: :output_string

RULE: :output_string --> IOMonitor.ReportIO IN: (:utterance :output_string) (:who "system") OUT: none!

;; At this point, we need to decide whether to respond ;; using audio or not. We condition this on our Hub ;; session variable.

RULE: $in(:audio_available session) & :output_string --> Synthesizer.Synthesize IN: :output_string LOG_IN: :output_string OUT: none!

RULE: ! $in(:audio_available session) & :output_string --> UI.Print IN: :output_string LOG_IN: :output_string OUT: none!

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; ;; DB SUBQUERY ;; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; This program handles the server-to-server subdialogue ;; through which the dialogue manager queries the database.

496

PROGRAM: DBQuery

RULE: :sql_query --> Backend.Retrieve IN: :sql_query OUT: :column_names :nfound :values LOG_IN: :sql_query LOG_OUT: :column_names :nfound :values

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;; ;; ;; AUDIO OUTPUT ;; ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;; Finally, the synthesizer chooses to produce a new ;; message. We could also accomplish the same ;; result by returning a value from the synthesizer and ;; adding more rules to the FromDialogue program above.

PROGRAM: FromSynthesizer

RULE: :host & :port & :call_id --> Audio.Play IN: :host :port :encoding_format :sample_rate :call_id :num_samples LOG_IN: :host :port :encoding_format :sample_rate :call_id :num_samples

Running the demoThe process monitor configuration file contrib/MITRE/demos/toy-travel/toy-travel.config can be used to start up aprocess monitor in compressed mode (a single pane with buttons to select the differentprocesses). This process monitor handles the Hub and creates additional process monitors forthe the UI elements. Start it up like this:

% cd contrib/MITRE/demos/toy-travel % ../../tools/bin/process_monitor toy-travel.config --example.frames toy-travel.pgm &

If you select "Process Control --> Restart all", all the servers will be started in order, and theHub. Also, two additional process monitors will be started for the UI and Audio servers. Boththese servers use the interaction paradigm described for audio and text. Both these servers canbe running at the same time, and both of them can be connected at the same time; the sessionswill be distinguished appropriately.

497

Please note that in the case of audio simulation, the audio data being passed around is an arrayof randomly generated bytes. The computer’s audio device is not involved, and you will not hearany audio played at any point.

Examining the logsAs a final step, we’ve also included an annotation rules file which supports the log analysis tools.So you can look at the results using these tools. You should also consult the logging documentation to learn more about how logs are created and where they’re stored.

% cd contrib/MITRE/demos/toy-travel % ../../tools/bin/process_monitor toy-travel.config --example.frames toy-travel.pgm & [...after running the example...] % ../../tools/bin/xml_summarize --rule_base annotation_rules.xml sls/[datedir]/[sessionnum] Checking: sls/20000804/018 Reading raw Hub log: sls/20000804/018/sls-20000804-018-hublog.txt ...read. Converting to XML: sls/20000804/018/sls-20000804-018-hublog.txt ...converted. Reading rule file: annotation_rules.xml ...read. Applying rules: annotation_rules.xml ...succeeded. Resegmenting... ...resegmented. Fri Aug 4 2000 at 21:40:21.89: Task-specific portion and overalltask ended.

Fri Aug 4 2000 at 21:40:16.54: New system turn began.

Fri Aug 4 2000 at 21:40:16.56 to Fri Aug 4 2000 at 21:40:16.58:System started speaking. Fri Aug 4 2000 at 21:40:16.56 to Fri Aug 4 2000 at 21:40:16.58:System finished speaking. System said: Welcome to Communicator. How may I help you?

Fri Aug 4 2000 at 21:40:17.78: New user turn began.

498

Fri Aug 4 2000 at 21:40:17.78: User started speaking. Fri Aug 4 2000 at 21:40:17.78: User finished speaking. Recognizer heard: I WANT TO FLY LOS ANGELES


Fri Aug 4 2000 at 21:40:17.95 to Fri Aug 4 2000 at 21:40:17.96:System started speaking. Fri Aug 4 2000 at 21:40:17.95 to Fri Aug 4 2000 at 21:40:17.96:System finished speaking. System said: Where are you traveling from?

Fri Aug 4 2000 at 21:40:19.02: New user turn began.

Fri Aug 4 2000 at 21:40:19.02: User started speaking. Fri Aug 4 2000 at 21:40:19.02: User finished speaking. Recognizer heard: BOSTON


Fri Aug 4 2000 at 21:40:19.23 to Fri Aug 4 2000 at 21:40:19.25:System started speaking. Fri Aug 4 2000 at 21:40:19.23 to Fri Aug 4 2000 at 21:40:19.25:System finished speaking. System said: American Airlines flight 115 leaves at 11:44 AM, andUnited flight 436 leaves at 2:05 PM

The timestamps for the start and end of speech are identical because our audio server doesn’tsend separate notifications for audio start and end. It also doesn’t log the audio and report thelogfile back to the Hub. A true audio server would do both these things.


499

MITRE Java Bindings

The Java bindings which MITRE is providing are a modification of Java bindings produced byIntel Corp. for a previous version of the Communicator implementation. We have updated andenhanced these bindings.

Enabling Java BindingsThe configuration templates/<arch-os>/config.make is where the setting for JDK_HOME shouldbe placed. The bindings and the examples require Java 2 (i.e., JDK 1.2) or higher. Please mailall bugs and comments to [email protected].

Documentation SummaryThese bindings implement a subset of the what the C libraries support, including brokering.Support for dialog control is not available in this distribution.

For documentation of the API, see:

Java bindings API documentation

The following demos contain Java server examples. All examples can be found in the java/subdirectory of these demos:

Basic server example: double, multiply Audio and brokering Binary objects for audio and data

Among the major differences from the C bindings are:

The examples that play sound files will only work on Sparc-Solaris machines (or othersystems that use /dev/audio as their sound device). The API itself should be platform independent. Servers accept multiple connections by default in a multi-threaded environment. The bindings do not implement the full range of functionality available in the MIT servers. Inparticular, the BIANCA turn management facilities are not yet available in Java.

500

At the moment, the source code and examples will serve as the primary documentation, thoughthe release notes and upgrade notes also contain useful information on the bindings.

Creating a ServerThere are two main classes of interest when building a Java Galaxy Communicator server. Thefirst is galaxy.server.MainServer . This class is responsible for creating and managingconnections to Hubs. The connections are embodied by the second class, galaxy.server.Server . Subclasses of this class implement the dispatch functions that areinvoked by the Hub directly (e.g., based on a program file) and indirectly (e.g., on the behalf ofanother Galaxy Communicator server). Currently, an instance of MainServer can only start up Server s of one type. The Server type is set programmatically using MainServer.setServerClassName (see the example below).

An instance of MainServer can be configured to run in one of three modes. In the first (default)mode (using the -port command line argument described below or relying on the default port),the server listens on a port for connection requests from Hubs and instantiates a new Serverfor each new connection. In the second mode, MainServer can be configured to connect toone or more Hubs (using the -contact_hub command line argument). For each connection, itinstantiates a Server to handle the connection. In the third mode (using both -port and -contact_hub) , MainServer acts as a Hub listener and client concurrently.

As mentioned above, the Server class encapsulates a connection to a Hub. When you extendthis class, all operations defined by serverOp OperationName will be interpreted asoperations that a Hub or another Galaxy Communicator server can call. serverOp OperationName is equivalent to "operation_name " in Hub program file syntax.When a Java server receives an operation name in a frame, it follows these steps to convert theoperation into the name of a Java method. First, it removes all underscore (_) characters andconverts the letter after the underscore to uppercase (e.g., "operation_name" becomes "operationName" ). Second, it converts the first letter of the operation name to uppercase ("operationName" becomes "OperationName" ). Finally, it prepends "serverOp" to theoperation name ("OperationName" becomes "serverOpOperationName" ). Keep theserules in mind when specifying operation names in program files and Java code.

Below is an example of a Java main method that starts a Galaxy Communicator server. Theinstance of MainServer is initialized with the command line arguments (see descriptionsbelow), and the Server type for the server is set by the call to MainServer.setServerClassName . Calling MainServer.start starts the server.

public static void main(String[] args) { MainServer mainServer = new MainServer( server name, args); mainServer.setServerClassName( name of a subclass of galaxy.server.Server);

501

try { mainServer.start(); } catch(Exception e) { System.out.println(e.toString()); } }

A third class, galaxy.server.ui.ServerUI , is useful for embedding a server in a simpleuser interface. This class instantiates a MainServer and provides a user interface with loggingcapability and the ability to start and stop the server. Below is an example of a Java mainmethod that creates an instance of ServerUI .

public static void main(String[] args) { ServerUI ui = new ServerUI(args) { protected void init() { serverClassName = name of a subclass of galaxy.server.Server; port = optional default port number; serverName = server name; } }; }

The examples in this distribution tend to use ServerUI , though the source code for the basicJava server example does contain examples of how to created UI-less servers.

Command Line ArgumentsThe following are of the built-in command line arguments for Java Galaxy Communicator servers:

-port <port number>Specifies the port on which this server will listen for connections.

-verbosity <verbosity level>

The logging verbosity level. Currently, the only valid valuesare 0 (no logging) or 1 (all logging on, the default).

-assertPrevent the server from searching for a free port if thespecified port (or default port if there is no -portargument) is not available.

-maxconns <max number of connections>

Maximum number of client connections supported by thisserver’s listener (if not > 0, no limit is set). If this argumentis used, but no value is specified, a default of 1 is used.

-validate Validate Server method signatures.

502

-contact_hub <host:port ...>

Hosts and ports of Hubs to contact on startup. Note thathost/port pairs are separated by whitespace, and multiplehost/port pairs must be enclosed in double quotes, e.g., -contact_hub "localhost:123 localhost:456".

-session_id <id>If -contact_hub is used, lock this client to the specified Hub session.

-server_locations_file <file>

A file of lines of the form: <server name> <host:port>[hub|server]. This file defines the locations of the listenersto use (be they in servers or Hubs).

-ui

Starts the server’s user interface. If this argument is notused, the user interface is not displayed and all output issent to standard error (typically the display screen). Thisargument is only processed by ServerUI -based servers.

-no_outputStarts the server and doesn’t print any output anywhere.This argument is only processed by ServerUI -based servers.

-startAutomatically starts the server (you don’t need to click the Start button in the user interface). This argument is onlyprocessed by ServerUI -based servers.

-log <log file name>Sets the name of the log file. This argument is onlyprocessed by ServerUI -based servers.

-append_logIf the log file already exists, append to it. This argument isonly processed by ServerUI -based servers.

-main_server_class<server class name>

The name of the MainServer class to start. Thisargument is only processed by ServerUI -based servers.

-helpDisplays descriptions of the above command line arguments.

Subclasses of MainServer and ServerUI can access their parent class’s instance of galaxy.server.ServerArgParser (for subclasses of MainServer , this parser object isonly useful if the constructor that takes the argument list was used). The methods on theinterface of ServerArgParser (the useful methods are actually implemented by its parentclass, galaxy.util.ArgParser ) can be used to examine the argument list (e.g., to extractthe values of custom-defined arguments). The ServerArgParser constructor that takes asecond string argument allows developers to add information on custom command linearguments to the usage message that is displayed by the -help command line argument.

503

ContinuationsWhen you wish to call a dispatch function from a server and allow the Hub to send the sendingserver other messages BEFORE the reply to the dispatch function is sent back to the server, youmust call the dispatchFrameWithConinuation method on the server’s current callenvironment (of type galaxy.server.Environment ). The current environment can beobtained from a galaxy.server.Server via the server’s getCurrentEnvironment method:

public boolean dispatchFrameWithContinuation(GFrame frame, Continuation continuation, Object continuationState)

This method’s arguments include the frame that is to be dispatched to the Hub, a continuationobject (discussed below), and state data used by the continuation. A continuation objectimplements the galaxy.server.Continuation interface:

public interface Continuation { public GFrame run(GFrame frame, int msgType, ObjectcontinuationState, Environment env); }

When the server receives the reply to a dispatch function that was called with dispatchFrameWithConinuation , it looks up the appropriate continuation object andinvokes its run method, passing in the Hub’s reply frame, the message type of the message thatcontained the reply frame, the state data that was passed into the original call to dispatchFrameWithConinuation , and the call environment from when the server originallydispatched the frame (i.e., no necessarily the server’s current call environment). The runmethod is responsible for processing the reply just as if it had been returned immediately afterthe server called the dispatch function. The code segment below (from DoubleServer.java)illustrates this new functionality.

Example

public class DoubleServer extends galaxy.server.Server {

...

public GFrame serverOpContinuationComplexTwice(GFrame frame) { GFrame newFrame = new Clause("multiply"); int i = ((Integer)frame.getProperty(":int")).intValue(); newFrame.setProperty(":int", i); ContinueComplexTwice continueComplexTwice = new ContinueComplexTwice(); getCurrentEnvironment().dispatchFrameWithContinuation(newFrame,continueComplexTwice, frame);

504

return (GFrame) null; }

class ContinueComplexTwice implements Continuation { public GFrame run(GFrame frame, int msgType, ObjectcontinuationState, Environment env) { GFrame origFrame = (GFrame) continuationState; GFrame returnFrame; GFrame twiceFrame; String progName;

if (frame == null) { logWarning("Didn’t hear back from multiply"); return (GFrame) null; }

switch (msgType) { case GalaxyMessage.GAL_REPLY_MSG_TYPE: progName = origFrame.getString(":program"); if (progName == null) progName = "main";

twiceFrame = new Clause("twice"); twiceFrame.setProperty(":program", progName); int i = ((Integer)frame.getProperty(":int")).intValue(); twiceFrame.setProperty(":int", i); returnFrame = serverOpTwice(twiceFrame); return returnFrame; case GalaxyMessage.GAL_ERROR_MSG_TYPE: return (GFrame) null; default: return (GFrame) null; } } }

Calling Dispatch Functions From Incoming BrokersIncoming brokers can now call dispatch functions on servers by using the Hub connection of the galaxy.server.Server that created the broker. This is done by calling the dispatchFramemethod on the interface of the broker’s galaxy.server.Environment object (NOTE: use thebroker’s local environment object; do NOT use the current environment of the galaxy.server.Server that created the broker). In addition, the broker can register with the

505

galaxy.server.Server object that created it by calling galaxy.server.DataInBroker.registerWithServer . This registration will force theserver to destroy the broker if the server loses its connection to the Hub. This prevents thebroker from trying to send messages to the Hub on a connection that no longer exists. Use galaxy.server.DataInBroker.unregisterFromServer to unregister from the server.

Sending Messages to the HubThe new, preferred way to send new messages and replies to the Hub from an instance of galaxy.server.Server is to use galaxy.server.Server.getCurrentEnvironmentto get a reference to the current call environment and use the methods on its interface (writeFrame , reply , error , error (second version), destroyToken , dispatchFrame ) tosend messages to the Hub. This replaces the use of writeMessage , writeErrorMessage , writeFrame , writeDestroyMessage and dispatchFrame on the interface of galaxy.server.Server (these methods have been deprecated as of 3.0).

Contacting a Hub at RuntimeIt is now possible to establish a client connection to a Hub at runtime using the contactHubmethod on the interface of galaxy.server.MainServer . This would typically be done from aclass that extends galaxy.server.Server , but it can be done from any class that has areference to a MainServer . If a connection is established, a new Server object of the typesupported by the MainServer instance is created. However, the reference to this newconnection is not returned to the caller due to the variability in the connection policies supportedby contactHub (discussed below).

Example

MainServer mainServer; ... mainServer.contactHub("localhost", 12345, "foo", MainServer.GAL_HUB_CLIENT_CONNECT_FAILURE_NOOP | MainServer.GAL_HUB_CLIENT_DISCONNECT_SHUTDOWN);

The first two arguments to contactHub are the host name and port of the Hub to contact. Thethird argument is the initial session id of the connection that is to be created. The fourthargument is a flag that specifies the connection policy to use (if this is -1, the connection policy ofthe invoked MainServer is used). This last argument is either set to MainServer.DEFAULT_HUB_CONTACT_POLICY or an or’ed (using the Java bitwsie or, |,operator) combination of one of

MainServer.GAL_HUB_CLIENT_CONNECT_FAILURE_RETRY: Indicates server should keeptrying to contact Hub until initial connection is established. MainServer.GAL_HUB_CLIENT_CONNECT_FAILURE_SHUTDOWN: Indicates server shouldshutdown if initial connection to Hub can not be established. MainServer.GAL_HUB_CLIENT_CONNECT_FAILURE_NOOP: Indicates server should donothing if initial connection to Hub can not be established.

506

and one of

MainServer.GAL_HUB_CLIENT_DISCONNECT_RETRY: Indicates server should keep tryingto re-establish contact to Hub if the connection is lost. MainServer.GAL_HUB_CLIENT_DISCONNECT_SHUTDOWN: Indicates server shouldshutdown if connection to Hub is lost. MainServer.GAL_HUB_CLIENT_DISCONNECT_NOOP: Indicates server should do nothing ifconnection to Hub is lost.

The default policy (MainServer.DEFAULT_HUB_CONTACT_POLICY ) is to keep trying toestablish the initial connection to the Hub until successful. If the connection to the Hub is laterlost, keep trying to reconnect. The default policy of a MainServer can be set and get with themethods setHubContactPolicy and getHubContactPolicy and you can set and get thepolicy of a given Server object with its own setHubContactPolicy and getHubContactPolicy methods.

Server and Session PropertiesThe Hub keeps track of properties for servers and sessions. There are various methods on theinterfaces of galaxy.server.Server and galaxy.server.Environment for accessingand manipulating these values. See the discussion of properties in the C binding documentationfor more details on the use of properties.

Methods on the interface of galaxy.server.Server :

getServerProperties: Returns the frame constituting the server properties as the servercurrently knows them locally. setServerProperties: Sets the properties in the server’s local property repository. deleteServerProperties: Deletes the specified properties from the server’s local property repository. addServiceType: Adds a new service type to a server’s local service type repository. getServiceTypes: Returns the list constituting the server service types as the servercurrently knows them locally.

Methods on the interface of galaxy.server.Environment (note that these methods act onthe server and session associated with the invoked environment object):

getServerProperties: Returns the frame constituting the server properties as the Hubcurrently knows them. setServerProperties: Sets the server properties in the Hub. This also sets the properties inthe server’s local property repository. deleteServerProperties: Deletes the specified server properties in the Hub. This also deletesthe properties from the server’s local property repository. modifyServerProperties: Allows a set of server properties to be modified (added andremoved) in the Hub. This also updates the server’s local property repository. getSessionProperties: Returns the frame constituting the session properties as the Hubcurrently knows them.

507

setSessionProperties: Sets the session properties in the Hub. deleteSessionProperties: Deletes the specified session properties in the Hub. modifySessionProperties: Allows a set of session properties to be modified (added andremoved) in the Hub.

Setting Session IdThere are now methods on the interface of galaxy.server.Environment that allow theenvironment’s session id to be changed:

setSessionId: Updates the environment’s local session id and informs the Hub of the change. updateSessionId: Updates the environment’s local session id. The Hub is not informed ofthe change.

Postponing a ReplyWhen a galaxy.server.Server dispatch function is invoked by the Hub, the server can call galaxy.server.Environment.postoneReply on the current environment to inform theHub that the response to its message will be delayed, but that in the meantime it’s available forother incoming messages.

Change Log

January 2002

Updated Hub/client connection policy flags.

June 2001

Added initial support for -verbosity command line argument. Changes to management of Hub/client connection policies.

December 2000

Extensive changes to brokering support. Extensive changes to thread management. Extensive changes to interface for writing frames to the Hub. Added support for continuations. Added support for configurable connection policies for Hub clients. Added support for server and session properties. Added support for new command line arguments.

508

August 2000

The Java bindings have been modified to work with the new listener-in-Hub capability. Inparticular, support for the -contact_hub and -session_id command line arguments has been added. Various changes to Java bindings implementation to improve thread synchronization.

February 2000

Removed references to Thread.stop() from Server.java and MainServer.java Added serverClassName field to ServerUI.java to allow more efficient implementation ofjava servers. (For an example see the double server)

January 1999

Server.java has changed. Operations now should be defined as GFrameserverOpOperationName(GFrame) as opposed to void serverOpOperationName(GFrame). 1/99 All demos have been modified accordingly. 1/99 Server.java accepts null as a return value from an operation. In this case the Server willnot return anything to the hub. 1/99 reinitialize is now called serverOpReinitialize and does not return anything. Theserver bindings will return a message to the HUB when reinitialize is called. If you wish tosend further information please use galaxy.server.Server.writeMessage(GalaxyMessage msg) or galaxy.server.Server.writeFrame(GFrame) . 10/28 All Classes in galaxy.lang are now children of galaxy.lang.GalaxyObject. 10/28 DataInBroker and DataOutBroker: notice that the i/o streams in galaxy.server.Server havebeen made private. Brokers must use the writeMessage, writeFrame and writeDestroyMessage() from brokers now. 10/28 For other changes please view the change log for this distribution.

Last modified January 9, 2002

509

MITRE Python Bindings

Enabling Python bindingsSee the installation documentation for how to enable the Python bindings. Python 1.5 or later is required.

Documentation SummaryThe bindings are divided into files which correspond approximately to subdirectories oflibGalaxy: Galaxy.py (galaxy), GalaxyIO.py (io and ServerStub), MGalaxy.py (MITRE’slibMITREgalaxy), and SLSUtil.py (util). There is a final module, cGalaxy, which is implemented inC and provides a wrapper around the core Galaxy Communicator library.

The following demos contain Python server examples. All examples can be found in the python/subdirectory of these demos:

Basic server example: double Audio and brokering Binary objects for audio and data

The Python versions in these examples are, to the greatest extent possible, exact parallels of theC versions, so you can study the two versions side by side.


Frames are implemented as a subclass of the UserDict class, which means that most of theframe manipulation can be handled by the Python mapping interface. Servers require explicit mappings for dispatch functions

At the moment, the source code and examples will serve as the primary documentation.However, we provide details and a table of equivalences as a guide.

510

DetailsWe will follow the general outline of the tutorial. Most of the lessons apply just as well to thePython bindings as to the core C library. We address the important differences here.

Introducing frames and objectsIn general, because Python is a dynamically typed language and supports dictionaries whoseobjects can be of heterogeneous types, the wrappers for building objects are superfluous.

GAL_FRAME objects are instances of the Galaxy.Frame class, which inherits from theUserDict class, which is a class customization of the basic dictionary type and handles theappropriate dictionary syntax GAL_STRING objects are objects of type types.StringType GAL_INT objects are objects of type types.IntType GAL_LIST objects are objects of type types.ListType, and the elements of the list can beany of the types specified here

Here are some of the examples from the frame tutorial, translated into Python:

import Galaxy

# {c output :output_string "hello" }

f = Galaxy.Frame("output", Galaxy.GAL_CLAUSE) f[":output_string"] = "hello"

# or

f = Galaxy.Frame("output", Galaxy.GAL_CLAUSE, {":output_string": "hello"})

# {c listframe :list ( 5 6 7 ) }

f = Galaxy.Frame("listframe", Galaxy.GAL_CLAUSE) f[":list"] = [5, 6, 7]

# or

f = Galaxy.Frame("listframe", Galaxy.GAL_CLAUSE, {":list": [5, 6, 7]})

Server basicsIn order to access the Python bindings, we recommend the following idiom:

import sys, os

511

sys.path.insert(0, os.path.join(os.environ["GC_HOME"], "contrib", "MITRE", "templates"))

import GC_py_init

import Galaxy, GalaxyIO, SLSUtil

Alternatively, instead of assuming the presence of GC_HOME in your environment, you could findanother way to make the GalaxyCommunicator root directory available to your program.

Each dispatch function takes an environment object and a frame and returns a dictionary orframe. In the case of Python, the call environment is the first argument, so that methods onclasses inherited from GalaxyIO.CallEnvironment also have the same signature.

Here’s a Python equivalent of the Parse dispatch function:

def Parse(env, frame): input_string = frame[":input_string"] p = ParseSentence(input_string) return {":frame": ParseTreeToFrame(p)}

Note that we can return a dictionary, instead of a named frame, since the name of the reply isignored anyway. The dictionary will be coerced into an appropriate reply frame.

Python doesn’t use macros to generate the declaration of dispatch functions or the name anddefault port information for the server. This is all handled via explicit function calls:

class ParseServer(GalaxyIO.Server): ...

s = ParseServer(sys.argv, "Parser", default_port = 10000) s.AddDispatchFunction("Parse", Parse)

Note that because the dispatch functions are explicitly associated with their names, the name ofthe function and the name of the message that invokes it can differ.

Here’s another way to set up your server, by using methods as dispatch functions:

class ParseEnvironment(GalaxyIO.CallEnvironment): def Parse(self, frame): input_string = frame[":input_string"] p = ParseSentence(input_string) return {":frame": ParseTreeToFrame(p)}

...

s = ParseServer(sys.argv, "Parser", default_port = 10000, env_class = ParseEnvironment) s.AddDispatchFunction("Parse", ParseEnvironment.Parse)

512

You can use the specialized classes for the call environment or for the server to store arbitrarydata as instance variables.

By default, Python servers accept approximately the same set of default arguments as Cservers. The differences are:

The -color argument is not supported. The -thread argument is not supported. Neither the -nottloop nor the -ttloop argument is supported The -debug argument provides for a complete printout of any errors captured by thePython server, instead of simply reporting them back in an error frame

The Python bindings include a set of functionality equivalent to the core C argument parsing library. This functionality is exemplified in the Python version of the double server:

OAS = [("-increment i", "initial increment")]

# Write a wrapper for the usage check.

class DoubleServer(GalaxyIO.Server): def CheckUsage(self, oas_list, args): global InitialIncrement data, out_args = GalaxyIO.Server.CheckUsage(self, OAS +oas_list, args) if data.has_key("-increment"): InitialIncrement = data["-increment"][0] del data["-increment"] return data, out_args

Note that we check command line arguments by creating a wrapper around the CheckUsagemethod of the server object. This method returns a dictionary whose keys are the argumentnames and whose values are a list of elements, as well as a sequence of unanalyzedarguments. If the argument list does not parse (if, for instance, some expected parameters of agiven command line argument are missing), the server will exit. Here’s a more complex exampleof an initialization string:

OAS = [("-utts utts_file num", "text utterance file for sequencing", (types.StringType, types.IntType), (Utterance_File, 0)] ("-stdin", "Read sentences from stdin instead of a file")]

The first element is the argument, with spaces delimiting the parameters. The second is adescription. Both of these elements are as in the the oa library. The third element, if present, is asequence of types, and the fourth element, if present, is a sequence of defaults. If there aren’tenough types, types.StringType is used; if there is a defaults list but not enough defaults, theappropriate null value is used (0, 0.0, or ""). If the fourth element is present, the return dictionarywill contain the defaults if the argument is not present in the arglist.

513

Error handlingUnlike C, Python has try/except for raising and catching errors. If you raise an error in a dispatchfunction (intentionally or otherwise), the Python bindings will catch the error and convert it to anerror reply. Here’s the Parse function again:

import types

ParseError = "ParseError"

def Parse(env, frame): try: input_string = frame[":input_string"] except KeyError: input_string = None if (type(input_string) is not type("")): raise ParseError, "no input string" p = ParseSentence(input_string) if not p: raise ParseError, "no parse" p_frame = ParseTreeToFrame(p) if not p_frame: raise ParseError, "can’t convert parse to frame" return {":frame": p_frame}

Sending new messages to the HubSending asynchronous and synchronous messages to the Hub works pretty much as you mightanticipate. Let’s start with the DoGreeting case:

def DoGreeting(env, frame): s = "{c FromDialogue :output_frame {c greeting } :is_greeting 1 }" greeting = Galaxy.Frame(str = s) env.WriteFrame(greeting) return None

If you want to return no frame to the Hub, you must return None; an empty dictionary will returnan empty frame.

Now, here’s the DoDialogue example:

def DoDialogue(env, frame): msg_frame = Galaxy.Frame("DBQuery", Galaxy.GAL_CLAUSE)

# ...

514

msg_frame[":sql_query"] = sql_query try: response_frame = env.DispatchFrame(msg_frame) # Equivalent of GAL_REPLY_MSG_TYPE. # Construct presentation of database response # ... except GalaxyIO.DispatchError: # Equivalent of GAL_ERROR_MSG_TYPE # Relay error back to Hub # ...

If the Hub relays an error back to the server, the Python bindings will raise the error locally. Theerror raised will be GalaxyIO.DispatchError , as shown here.

Setting up a brokered audio connectionIn order to set up a broker server, you can use the class GalaxyIO.BrokerDataOut directly,most of the time. For the broker client, you should subclass GalaxyIO.BrokerDataIn . Themethods EnvHandleFrame , EnvHandleString , EnvHandleBinary , EnvHandleInt16 ,etc., can be subclassed on the broker client to handle the appropriate class of incoming data.Outgoing data is handled using the Write method; there is no need for separate functions towrite each datatype.

Here’s the equivalent of the Synthesize dispatch function, exemplifying the source serversetup. If the broker setup fails, an error is raised:

def Synthesize(env, frame): # ... try: b = GalaxyIO.BrokerDataOut(env.conn, 10) output_f = Galaxy.Frame("FromSynthesizer", Galaxy.GAL_FRAME, {":sample_rate": s.sample_rate, ":encoding_format": s.encoding_format}) b.PopulateFrame(output_f, ":host", ":port") env.WriteFrame(output_f) except GalaxyIO.BrokerInitError, m: # ....

Now, here’s the setup for the client:

class AudioBroker(GalaxyIO.BrokerDataIn): # ...

def Play(env, frame): host = frame[":host"] port = frame[":port"] p = AudioPkg()

515

# ...

try: p.in_broker = AudioBroker(env, host, port, frame) p.in_broker.AddCallback(GalaxyIO.GAL_BROKER_DATA_DONE_EVENT, lambda b = p.in_broker: b.__AudioOutputCallback) # ... except GalaxyIO.BrokerInitError, m: # ...

The data exchange works as follows. First, on the source server:

data = PollSynthesis(s)

while data: b.Write(data) data = PollSynthesis(s) if SynthesisIsDone(s): b.DataDone()

On the client, it’s handled as methods associated with the specialized broker class:

class AudioBroker(GalaxyIO.BrokerDataIn): def EnvHandleInt16(self, env, obj): print "[Audio data to user (%d samples)]" % len(obj)

Unhandled data types raise GalaxyIO.BrokerProcessingError .

EquivalencesThese functions are arranged in alphabetical order of the corresponding C functions, arrangedby the documents in the advanced and reference sections of the documentation which describethose functions.


GalUtil_CPPFrame(sls_verbose_level, fore, back, fr)

[not implemented]

GalUtil_CPPObject(sls_verbose_level, fore, back, o)

[not implemented]

516

GalUtil_PPFrame(verbose_level, fr)SLSUtil.OStream().write_level(fr.PPrint(), verbose_level)

GalUtil_PPObject(verbose_level, o)SLSUtil.OStream().write_level(Galaxy.OPr(o,PP_TYPE), verbose_level)

GalUtil_PrObject(verbose_level, o)SLSUtil.OStream().write_level(Galaxy.OPr(o), verbose_level)

GalUtil_PrintObject(verbose_level, o, how)SLSUtil.OStream().write_level(Galaxy.OPr(o, how), verbose_level)

Gal_AddPred(fr, pred) fr.preds.append(pred)

Gal_ArrayObjectAdd(obj, data, size) obj = obj + data

Gal_BinaryObject(data, size) Galaxy.BinaryObject(Galaxy.GAL_BINARY, data)

Gal_BinarySize(obj) len(obj)

Gal_BinaryValue(obj, size) Galaxy.ValueWarn(obj, Galaxy.GAL_BINARY)

Gal_Binaryp(obj)Galaxy.GetObjectType(obj) == Galaxy.GAL_BINARY

Gal_ClauseFramep(fr)Galaxy.GetDetailedType(fr) == Galaxy.GAL_CLAUSE

Gal_ClauseValue(o) [not implemented]

Gal_Clausep(obj)Galaxy.GetDetailedType(obj) == Galaxy.GAL_CLAUSE

Gal_ClearPreds(fr) fr.preds = []

Gal_CopyFrame(fr) fr.Copy()

Gal_CopyObject(obj) [not implemented]

Gal_CreateBinaryObject(data, size, manage_memory)

Galaxy.BinaryObject(Galaxy.GAL_BINARY, data)

Gal_CreateFloat32Object(data, num_float_32, manage_memory)

Galaxy.BinaryObject(Galaxy.GAL_FLOAT_32, data)


Galaxy.BinaryObject(Galaxy.GAL_FLOAT_64, data)

Gal_CreateFloatObject(value, manage_memory) [not needed]

Gal_CreateFrameObject(value, manage_memory) [not needed]

517

Gal_CreateInt16Object(data, num_int_16, manage_memory)

Galaxy.BinaryObject(Galaxy.GAL_INT_16, data)





Gal_CreateListObject(values, n, free_fn , manage_memory)

[not needed]

Gal_CreateStringObject(cp, manage_memory) [not needed]

Gal_CreateVarMapping(num_pairs, ...) [not implemented]

Gal_DelPred(fr, i) del f.preds[i]

Gal_DelPredByName(fr, name) p = f.GetPredByName(name); f.preds.remove(p)

Gal_DelProp(fr, key) del f[key]

Gal_DeletePreds(fr, pred_name) [not implemented]

Gal_DoPreds(fr, pred_fn , caller_data) map(pred_fn, fr.preds)

Gal_DoProperties(fr, prop_fn , caller_data) map(prop_fn, fr.items())

Gal_FindKey(fr, key_name) [not implemented]

Gal_FindPred(fr, pred_name) [not implemented]

Gal_FindPredParent(frame, name, parent, findpar, nth)

[not implemented]

Gal_FindTopic(fr, topic_name) [not implemented]

Gal_Float32Size(o) len(o)

Gal_Float32Value(obj, size) Galaxy.ValueWarn(obj, Galaxy.GAL_FLOAT_32)

Gal_Float32p(o)Galaxy.GetObjectType(o) == Galaxy.GAL_FLOAT_32

Gal_Float64Size(o) len(o)

Gal_Float64Value(obj, size) Galaxy.ValueWarn(obj, Galaxy.GAL_FLOAT_64)

Gal_Float64p(o)Galaxy.GetObjectType(o) == Galaxy.GAL_FLOAT_64

Gal_FloatObject(val) [not needed]

Gal_FloatValue(o) Galaxy.ValueWarn(o, Galaxy.GAL_FLOAT)

518

Gal_Floatp(obj)Galaxy.GetObjectType(obj) == Galaxy.GAL_FLOATOR type(obj) is types.FloatType

Gal_FrameEqual(sf1, sf2) sf1.Equal(sf2)

Gal_FrameIsType(fr, type) fr.type == type

Gal_FrameName(fr) fr.name

Gal_FrameNameEq(fr, name) fr.name == name

Gal_FrameNamesEq(fr1, fr2) fr1.name == fr2.name

Gal_FrameObject(val) [not needed]

Gal_FrameValue(o) Galaxy.ValueWarn(o, Galaxy.GAL_FRAME)

Gal_Framep(obj)Galaxy.GetObjectType(obj) == Galaxy.GAL_FRAME

Gal_FreeFrame(fr) [not needed]

Gal_FreeObject(obj) [not needed]

Gal_FreeWrapper(to) [not needed]

Gal_GetBinary(fr, key, size) fr.GetValue(key, Galaxy.GAL_BINARY)

Gal_GetDetailedType(o) Galaxy.GetDetailedType(o)

Gal_GetFloat(fr, key) fr.GetValue(key, Galaxy.GAL_FLOAT)

Gal_GetFloat32(fr, key, size) Galaxy.ValueWarn(obj, Galaxy.GAL_FLOAT_32)

Gal_GetFloat64(fr, key, size) Galaxy.ValueWarn(obj, Galaxy.GAL_FLOAT_64)

Gal_GetFrame(fr, key) fr.GetValue(key, Galaxy.GAL_FRAME)

Gal_GetFrameType(fr) fr.type

Gal_GetInt(fr, key) fr.GetValue(key, Galaxy.GAL_INT)

Gal_GetInt16(fr, key, size) Galaxy.ValueWarn(obj, Galaxy.GAL_INT_16)



Gal_GetList(fr, key, length) fr.GetValue(key, GAL_LIST)

Gal_GetListObject(obj, n) obj[n]

Gal_GetListValue(o, n, type) Galaxy.ValueWarn(o[n], type)

Gal_GetObject(fr, key) fr[key]

519

Gal_GetObjectType(o) Galaxy.GetObjectType(o)

Gal_GetObjectTypeString(o) Galaxy.ObjectTypeString(Galaxy.GetObjectType(o))

Gal_GetPred(fr, i) fr.preds[i]

Gal_GetPredByName(fr, name) fr.GetPredByName(name)

Gal_GetProperties(fr, nkeys) fr.keys()

Gal_GetString(fr, key) fr.GetValue(key, Galaxy.GAL_STRING)

Gal_GetTopicFrame(fr, key) [not implemented]

Gal_Int16Size(o) len(o)

Gal_Int16Value(obj, size) Galaxy.ValueWarn(obj, Galaxy.GAL_INT_16)

Gal_Int16p(o) Galaxy.GetObjectType(o) == Galaxy.GAL_INT_16







Gal_IntObject(val) [not needed]

Gal_IntValue(to) Galaxy.ValueWarn(obj, Galaxy.GAL_INT)

Gal_Intp(obj)Galaxy.GetObjectType(obj) == Galaxy.GAL_INTOR type(obj) is types.IntType

Gal_ListLength(obj) len(obj)

Gal_ListObject(values, n) [not needed]

Gal_ListObjectAdd(obj, elt) obj.append(elt)

Gal_ListObjectFromElements(n, ...) [not needed]

Gal_ListValue(obj, n) Galaxy.ValueWarn(obj, Galaxy.GAL_LIST)

Gal_Listp(obj)Galaxy.GetObjectType(obj) == Galaxy.GAL_LISTOR type(obj) is types.ListType

Gal_MakeClauseFrame(name)Galaxy.Frame(name = name, type = Galaxy.GAL_CLAUSE)

520

Gal_MakeFrame(name, type) Galaxy.Frame(name = name, type = type)

Gal_MakePredFrame(name)Galaxy.Frame(name = name, type = Galaxy.GAL_PRED)

Gal_MakeTopicFrame(name)Galaxy.Frame(name = name, type = Galaxy.GAL_TOPIC)

Gal_MatchFrame(sf1, sf2) [not implemented]

Gal_MatchKeyValue(fr, key_name, match) [not implemented]

Gal_NumNonNullProperties(fr) len(fr)

Gal_NumPreds(fr) len(fr.preds)

Gal_NumProperties(fr) len(fr)

Gal_ObjectByteCount(obj) [not implemented]

Gal_ObjectCaseEqual(obj1, obj2) Galaxy.ObjectEqual(obj1, obj2, ignore_case = 1)

Gal_ObjectEqual(obj1, obj2) Galaxy.ObjectEqual(obj1, obj2)

Gal_ObjectToString(o) Galaxy.OPr(o)

Gal_ObjectTypeString(object_type) Galaxy.ObjectTypeString(object_type)

Gal_OutlineFrame(fr, sls_verbose_level) [not implemented]

Gal_OutlineObject(to, sls_verbose_level) [not implemented]

Gal_PPFrame(fr) print fr.PPrint() OR fr.PP()

Gal_PPFrameToFile(fr, fp) fp.write(fr.PPrint())

Gal_PPFrameToString(fr, buf, bufsizeptr) fr.PPrint()

Gal_PPObject(o) print Galaxy.OPr(o, Galaxy.PP_TYPE)

Gal_PPObjectToFile(o, fp) fp.write(Galaxy.OPr(o, Galaxy.PP_TYPE))

Gal_PrFrame(fr) print fr.Print() OR f.rPr()

Gal_PrFrameToFile(fr, fp) fp.write(fr.Print())

Gal_PrFrameToString(fr, buf, bufsizeptr) fr.Print()

Gal_PrObject(obj) print Galaxy.OPr(obj) OR Galaxy.PrObject(obj)

Gal_PrObjectToFile(obj, fp) fp.write(Galaxy.OPr(obj))

Gal_PredFramep(fr)Galaxy.GetDetailedType(obj) == Galaxy.GAL_PRED

521

Gal_PredValue(to) [not implemented]

Gal_Predp(obj)Galaxy.GetDetailedType(obj) == Galaxy.GAL_PRED

Gal_PrintFrameToFile(fr, fp, how) fp.write(fr._Print(how))

Gal_PrintFrameToString(fr, irpbuf, bufsizeptr, how) fr._Print(how)

Gal_ReadFrameFromFile(fp) [not implemented]

Gal_ReadFrameFromString(buf) Galaxy.Frame(str = buf)

Gal_ReadObjectFromFile(fp) [not implemented]

Gal_ReadObjectFromString(buf) Galaxy._read_irp_value(buf)

Gal_ReadVarFrameFromString(buf, map) [not implemented]

Gal_RemPred(fr, i) p = fr.preds[i]; del fr.preds[i]; return p

Gal_RemPredByName(fr, name)p = fr.GetPredByName(name); fr.preds.remove(p);return p

Gal_RemProp(fr, key) p = fr[key]; del fr[key]; return p

Gal_SetFrameName(fr, name) fr.name = name

Gal_SetFrameType(fr, type) fr.type = type

Gal_SetProp(fr, key, obj) fr[key] = obj

Gal_StringObject(val) [not needed]

Gal_StringValue(to) Galaxy.ValueWarn(obj, Galaxy.GAL_STRING)

Gal_Stringp(obj)Galaxy.GetObjectType(obj) ==Galaxy.GAL_STRING OR type(obj) is types.StringType

Gal_TopicFramep(fr)Galaxy.GetDetailedType(obj) == Galaxy.GAL_TOPIC

Gal_TopicValue(to) [not implemented]

Gal_Topicp(obj)Galaxy.GetDetailedType(obj) == Galaxy.GAL_TOPIC

Gal_VAReadVarFrameFromString(buf, num_pairs, ...)

[not implemented]

_gal_free_object(obj) [not needed]

522


GalIO_GetError(f, err_desc) [not needed; just catch GalaxyIO.DispatchError]

GalSS_AddDispatchFunction(i, name, fn,in_key_array, allow_other_in_keys, reply_provided,out_key_array, allow_other_out_keys)

i.AddDispatchFunction(...)

GalSS_EnvDestroyToken(env) env.DestroyToken()

GalSS_EnvDispatchFrame(env, frame, t) env.DispatchFrame(frame)

GalSS_EnvError(env, description) [not needed; just raise an error]

GalSS_EnvGetClientData(env, name) [not needed; use instance attributes]

GalSS_EnvGetCommData(env) [not needed; use instance attributes]

GalSS_EnvReply(env, f) env.Reply(f)

GalSS_EnvSetCommData(env, data, free_fn ) [not needed; use instance attributes]

GalSS_EnvWriteFrame(env, frame, do_block) env.WriteFrame(frame)

GalSS_InitializeServerDefaults(scomm, name, port)

[not implemented]

Gal_CreateDispatchFnKeyArray(ignore, ...) [not needed]

Gal_FreeDispatchFnKeyArray(entry) [not needed]

_GalSS_InitializeDefaults(scomm) [not needed; provided to scomm during initialization]

_GalSS_init_server(server, argc, argv)[not needed; perform operations after initializationand before starting server]

_GalSS_print_usage(argc, argv) [not implemented]


GalIO_BrokerDataDone(b) b.DataDone()

GalIO_BrokerDataInInit(host, port, frame, fnptr,caller_data, poll_ms)

[not implemented]

GalIO_BrokerDataOutDone(b) b.DataDone()

523

GalIO_BrokerDataOutInit(gcomm, poll_ms, timeout_seconds)

b = GalaxyIO.BrokerDataOut(gcomm, timeout_seconds)

GalIO_BrokerIsDone(b) [not implemented]

GalIO_BrokerPopulateFrame(b, f, host_key, port_key)

b.PopulateFrame(f, host_key, port_key)

GalIO_BrokerSetFinalizer(b, finalizer) [not implemented]

GalIO_BrokerStructDequeue(b, bqueue) [not needed; no broker queues in Python]

GalIO_BrokerStructQueueAppend(b, bqueue) [not needed; no broker queues in Python]

GalIO_BrokerStructQueuePop(bqueue) [not needed; no broker queues in Python]

GalIO_BrokerWriteBinary(b, data, n_bytes) b.Write(data)

GalIO_BrokerWriteFloat(b, f) b.Write(f)

GalIO_BrokerWriteFloat32(b, data, n_floats) b.Write(data)

GalIO_BrokerWriteFloat64(b, data, n_floats) b.Write(data)

GalIO_BrokerWriteFrame(b, frame) b.Write(f)

GalIO_BrokerWriteInt(b, i) b.Write(i)

GalIO_BrokerWriteInt16(b, data, n_ints) b.Write(data)



GalIO_BrokerWriteList(b, elts, n_elts) b.Write(elts)

GalIO_BrokerWriteObject(b, o) b.Write(o)

GalIO_BrokerWriteString(b, str) b.Write(s)

GalIO_CommBrokerDataInInit(host_gcomm, host,port, frame, fnptr, poll_ms, caller_data,caller_data_free_fn )

b = GalaxyIO.BrokerDataIn(host_gcomm, host, port, frame)

GalIO_ForceBrokerExpiration(b) [not implemented]

GalIO_FrameSetBrokerCallID(f, call_id) [not implemented]

GalIO_GetBrokerCallID(b) b.call_id

GalIO_GetBrokerCallerData(b) [not needed; use instance attributes]

GalIO_GetBrokerData(b) [not needed; use instance attributes]

GalIO_GetBrokerFrame(b) [not implemented]

524

GalIO_GetBrokerListenPort(b) b.port

GalIO_IPAddress() GalaxyIO.IPAddress()

GalIO_SetBrokerActive(b) [not needed; no broker queues in Python]

GalIO_SetBrokerData(b, caller_data, free_fn ) [not needed; use instance attributes]

GalSS_BrokerGetEnvironment(b) b.env

GalSS_BrokerSetEnvironment(b, env) [not needed; pass environment to BrokerDataIn]

GalSS_EnvBrokerDataInInit(env, host, port, frame,fnptr, poll_ms, refptr, free_fn )

b = GalaxyIO.BrokerDataIn(env, host, port, frame)

Server Architecture

GalIO_AddBrokerCallback(b, callback_event, fn, callback_data)

b.AddCallback(callback_event, fn)

GalIO_AddConnectionBrokerCallback(gcomm,callback_event, connect_callback, callback_data)

[not needed; specialize __init__ method]

GalIO_AddConnectionCallback(gcomm,callback_event, connect_callback, callback_data)

[not implemented]

GalIO_AddConnectionDispatchFnCallback(gcomm,dispatch_callback, callback_data)

[not implemented]

GalIO_AddServerCallback(scomm, callback_event,fn, callback_data)

[not implemented]

GalIO_AddServerConnectCallback(scomm,connect_callback, callback_data)

[not implemented]

GalIO_CommValidating(gcomm) gcomm.Validating()

GalIO_CommWriteFrame(gcomm, frame, do_block)

gcomm.WriteFrame(frame)

GalIO_ContactHub(host, port, scomm, session_id, client_poll_flags)

[not implemented]

GalIO_DispatchViaHub(gcomm, frame, msg_type_ptr)

gcomm.DispatchFrame(frame)

GalIO_EnableDispatchFnValidation(scomm) scomm.EnableValidation()

GalIO_GetCommClientData(gcomm, name) [not needed; use instance attributes]

525

GalIO_GetCommData(gcomm) [not needed; use instance attributes]

GalIO_GetCommServerData(gcomm) [not needed; use instance attributes]

GalIO_GetCommServerName(gcomm) gcomm.server.ServerName()

GalIO_GetServerClientData(server, name) [not needed; use instance attributes]

GalIO_GetServerData(scomm) [not needed; use instance attributes]

GalIO_GetServerDefaultPort(scomm) [not implemented]

GalIO_GetServerListenPort(scomm) scomm.ListenPort()

GalIO_GetServerMaxConnections(scomm) scomm.MaxConnections()

GalIO_GetServerName(scomm) scomm.ServerName()

GalIO_GetServerNumConnections(scomm) len(scomm.conns)

GalIO_GetUniqueConnection(scomm) [not implemented]

GalIO_OperateOnConnections(scomm, arg, op ) map(op, scomm.conns.values())

GalIO_RemoveBrokerCallback(b, cb) [not implemented]

GalIO_RemoveConnectionCallback(gcomm, cb) [not implemented]

GalIO_RemoveServerCallback(scomm, cb) [not implemented]

GalIO_ServerListenStatus(scomm) [not implemented]

GalIO_ServerSessionID(scomm) [not implemented]

GalIO_ServerUsesTimedTasks(server) [not implemented; polling by select]

GalIO_SetCommClientData(gcomm, name, client_data)

[not needed; use instance attributes]

GalIO_SetCommData(gcomm, data, free_fn ) [not needed; use instance attributes]

GalIO_SetServerClientData(server, name, client_data)

[not needed; use instance attributes]

GalIO_SetServerData(scomm, data, free_fn ) [not needed; use instance attributes]

GalIO_SetServerDefaultPort(scomm, port) [not implemented]

GalIO_SetServerMaxConnections(scomm, max) scomm.MaxConnections(max)

GalIO_SetServerName(scomm, name) scomm.ServerName(name)

GalSS_CmdlineInitializeServer(argc, argv) [not implemented]

GalSS_CmdlineSetupServer(argc, argv) [not implemented]

526

GalSS_DefaultServerArgs() [not needed]

GalSS_EnvComm(env) env.conn

GalSS_ExtractCmdlineServerArgs(arg_pkg, argc,argv, new_argc_ptr, new_argv_ptr)

[not needed]

GalSS_ExtractServerArgs(argc, argv,new_argc_ptr, new_argv_ptr)

[not needed]

GalSS_FreeArgPkg(arg_pkg) [not needed]

GalSS_InitializeServer(server_port, max_conns,use_color, do_assert, use_ttloop, validate,new_argc, new_argv)

[not implemented]

GalSS_InitializeServerFromServerArgs(arg_pkg,new_argc, new_argv)

[not implemented]

GalSS_InitializeServerToplevel(server_port,max_conns, use_color, do_assert, loop_type,validate, verbosity, server_listen_status,client_pair_string, session_id, new_argc, new_argv)

[not implemented]

GalSS_RunServer(server) [not implemented]

GalSS_SAFixAssert(arg_pkg, assert) [not implemented]

GalSS_SAFixColor(arg_pkg, color) [not implemented]

GalSS_SAFixContactHubInfo(arg_pkg,client_pair_status, session_id, old_session_id_ptr)

[not implemented]

GalSS_SAFixLoopType(arg_pkg, loop_type) [not implemented]

GalSS_SAFixMaxConns(arg_pkg, max_conns) [not implemented]

GalSS_SAFixPort(arg_pkg, port) [not implemented]

GalSS_SAFixServerListenStatus(arg_pkg, server_listen_status)

[not implemented]

GalSS_SAFixServerLocations(arg_pkg, server_locations_file)

[not implemented]

GalSS_SAFixValidate(arg_pkg, validate) [not implemented]

GalSS_SAFixVerbosity(arg_pkg, verbosity) [not implemented]

GalSS_SetupServer(arg_pkg, new_argc, new_argv)

GalaxyIO.Server(...)

527

GalSS_StartAndRunServer(server) s.RunServer()

The Timed Task Loop None of these functions are implemtened, becausePython doesn’t use the timed task loop.

Gal_AddIdleFunction(func, client_data) [not implemented]

Gal_AddTask( task , refcon, num_millisecs,read_blocking_available, cleanup_fn )

[not implemented]

Gal_AddTaskExtended( task , caller_data,num_millisecs, read_blocking_available,read_socket, write_socket, err_socket, read_file,write_file, err_file, condition, cleanup_fn )

[not implemented]

Gal_AddTaskWithFileIO( task , refcon,num_millisecs, read_blocking_available, read_file,write_file, cleanup_fn )

[not implemented]

Gal_AddTaskWithSocketIO( task , refcon,num_millisecs, read_blocking_available,read_socket, write_socket, cleanup_fn )

[not implemented]

Gal_AddTimedTask(task, refcon, num_millisecs) [not implemented]

Gal_AddTimedTaskWithFileIO(task, refcon,num_millisecs, read_file, write_file)

[not implemented]

Gal_AddTimedTaskWithSocketIO(task, refcon,num_millisecs, read_socket, write_socket)

[not implemented]

Gal_EnableTimedTaskThreads() [not implemented]

Gal_EndTasks(immediate) [not implemented]

Gal_MaybeEndTask(immediate, deferred) [not implemented]

Gal_ReAddTask(p, refcon, num_millisecs,read_blocking_available, cleanup_fn )

[not implemented]

Gal_ReAddTaskExtended(p, caller_data,num_millisecs, read_blocking_available,read_socket, write_socket, err_socket, read_file,write_file, err_file, condition, cleanup_fn )

[not implemented]

Gal_ReAddTaskWithFileIO(p, refcon,num_millisecs, read_blocking_available, read_file,write_file, cleanup_fn )

[not implemented]

528

Gal_ReAddTaskWithSocketIO(p, refcon,num_millisecs, read_blocking_available,read_socket, write_socket, cleanup_fn )

[not implemented]

Gal_RemoveIdleFunction(func) [not implemented]

Gal_RemoveTask(task_id) [not implemented]

Gal_RemoveTimedTask(task, refcon) [not implemented]

Gal_RunIdleFunctions() [not implemented]

Gal_TaskPkgBlocking(pkg) [not implemented]

Gal_TaskPkgData(pkg) [not implemented]

Gal_TaskPkgRunReasons(pkg) [not implemented]

Gal_TimedTaskLoopThreadWaiter() [not implemented]

Gal_TimedTaskThreadsEnabled() [not implemented]

Gal_TimedTasksLoop() [not implemented]

Gal_TimedTasksLoopExit() [not implemented]

Gal_TimedTasksLoopHandler(tv) [not implemented]

Managing Session Information and Using Continuations

GalSS_EnvAddTask(env, task , caller_data,num_millisecs, read_blocking_available,read_socket, write_socket, err_socket, read_file,write_file, err_file, condition, cleanup_fn )

[not implemented]

GalSS_EnvCreate(gcomm)GalaxyIO.CallEnvironment(gcomm, None, create_p= 1)

GalSS_EnvDispatchFrameWithContinuation(env,frame, fn, continuation_state,continuation_state_free_fn )

env.DispatchFrameWithContinuation(frame, fn)

GalSS_EnvGetSessionID(env) env.GetSessionID()

GalSS_EnvLock(env) [not needed]

GalSS_EnvMaintainInLocation(gcomm,initial_session_id, env_loc)

[not implemented]

529

GalSS_EnvPostponeReply(env) [not implemented]

GalSS_EnvReturnRequired(env) env.ReturnRequired()

GalSS_EnvUnlock(env) [not needed]

GalSS_EnvUpdateSessionID(env, session_id) env.UpdateSessionID(session_id)

GalSS_TaskGetEnvironment(p) [not implemented]

GalSS_TaskSetEnvironment(p, env) [not implemented]


GalIO_AddServiceType(server, stype) server.AddServiceType(stype)

GalIO_ServerModifyProperties(server,new_properties, delete_properties)

server.ModifyProperties(new_properties, delete_properties)

GalIO_ServerProperties(server) [not implemented]

GalSS_EnvDeleteServerProperties(env, keys)env.ModifyServerProperties(properties_to_delete = keys)

GalSS_EnvDeleteSessionProperties(env, keys)env.ModifySessionProperties(properties_to_delete= keys)

GalSS_EnvGetServerProperties(env, keys) env.GetServerProperties(keys)

GalSS_EnvGetSessionProperties(env, keys) env.GetSessionProperties(keys)

GalSS_EnvModifyServerProperties(env,properties_to_set, properties_to_delete)

env.ModifyServerProperties(properties_to_set, properties_to_delete)

GalSS_EnvModifySessionProperties(env,properties_to_set, properties_to_delete)

env.ModifySessionProperties(properties_to_set, properties_to_delete)

GalSS_EnvSetServerProperties(env, properties)env.ModifyServerProperties(properties_to_set = properties)

GalSS_EnvSetSession(env, session_name, lock_info)

env.SetSession(session_name, lock_info)

GalSS_EnvSetSessionProperties(env, properties)env.ModifySessionProperties(properties_to_set = properties)

530

Special Main Loops

The basic Python main loop works differently thanthe C main loop, so customizing it and embedding itworks differently as well. Most of these functionsdon’t make any sense in Python.

GalIO_BrokerDataInCallbackHandler(b, read_blocking)

[not implemented]

GalIO_BrokerDataOutCallbackHandler(b) [not implemented]

GalIO_BrokerReadReady(b) [implemented but not visible]

GalIO_BrokerWriteReady(b) [implemented but not visible]

GalIO_CommReadReady(gcomm) [implemented but not visible]

GalIO_CommWriteReady(gcomm) [implemented but not visible]

GalIO_ConnectionCallbackHandler(gcomm, read_blocking)

[not implemented]

GalIO_DigestServerLocations(client_pair_string) [not implemented]

GalIO_GetServerLocations(scomm) [not implemented]

GalIO_NthHostAndPort(locs, i, port) [not implemented]

GalIO_NumServerLocations(locs) [not implemented]

GalIO_ServerCallbackHandler(scomm,read_blocking, new_conn_ptr)

[not implemented]

GalIO_ServerCheckHubContacts(scomm) [not implemented]

GalSS_ELRBroker(elr) [not implemented]

GalSS_ELRCopy(source) [not implemented]

GalSS_ELRCreate(scomm, timer_set_fn,timer_unset_fn, fd_set_fn, fd_unset_fn,behavior_fn, timer_is_persistent)

[not implemented]

GalSS_ELRDestroy(elr) [not implemented]

GalSS_ELRDoCallback(elr, timer_or_fd) [not implemented]

GalSS_ELRGComm(elr) [not implemented]

GalSS_ELRGetLoopData(elr) [not implemented]

GalSS_ELRSComm(elr) [not implemented]

531

GalSS_ELRSetBrokerInCallback(elr, fn) [not implemented]

GalSS_ELRSetBrokerOutCallback(elr, fn) [not implemented]

GalSS_ELRSetConnectionCallback(elr, fn) [not implemented]

GalSS_ELRSetLoopData(elr, loop_data,loop_data_free_fn )

[not implemented]

GalSS_ELRSetServerClientCallback(elr, fn) [not implemented]

GalSS_ELRSetServerListenerCallback(elr, fn) [not implemented]

GalSS_ELRSetupServer(external_arg_pkg, argc,argv, timer_set_fn, timer_unset_fn, fd_set_fn,fd_unset_fn, behavior_fn, loop_data,loop_data_free_fn , timer_is_persistent)

[not implemented]

GalSS_ELRShutdown(elr) [not implemented]

GalSS_ELRUpdatePollIntervals(elr,server_client_poll_ms, conn_ms, broker_ms)

[not implemented]

Signal Handling

Gal_AddSignalHandler(sig, handler ) [not implemented]

Gal_InitializeSignals() [not implemented]

Gal_SignalsInitialized() [not implemented]


Streams for printout are attached to callenvironments and connections in Python.

GalUtil_Assert(truth, format, ...) env.ostream.p_assert(truth, msg)

GalUtil_CPInfo1(fore, back, format, ...) [not implemented]


GalUtil_CPrint(level, fore, back, format, ...) [not implemented]

GalUtil_Debug1(format, ...) env.ostream.debug1(msg)

GalUtil_Debug2(format, ...) env.ostream.debug2(msg)

532

GalUtil_Error(format, ...) env.ostream.error(msg)

GalUtil_Fatal(format, ...) env.ostream.fatal(msg)

GalUtil_OACheckUsage(argc, argv, oas, first_real_arg)

GalaxyIO.Server.CheckUsage(scomm, oas, argv)

GalUtil_OAExtract(argc, argv, oas, key, ...) SLSUtil.OAExtract(oas, argv, ostream)

GalUtil_OAExtractAsserting(argc, argv, oas, key, ...)

[not implemented]

GalUtil_OAPrintUsage(argc, argv, oas) SLSUtil.OAPrintUsage(oas)

GalUtil_PInfo1(format, ...) env.ostream.pinfo1(msg)

GalUtil_PInfo2(format, ...) env.ostream.pinfo2(msg)

GalUtil_Print(level, format, ...) env.ostream.write_level(level, msg)

GalUtil_SetVerbose(verbose_level) env.ostream.set_verbosity(verbose_level)

GalUtil_VerboseUseBW() [not implemented]

GalUtil_VerboseUseColor() [not implemented]

GalUtil_Warn(format, ...) env.ostream.warn(msg)


GalIO_ClientConnect(name, host, port, silent,welcome_frame, reply_frame)

GalaxyIO.ClientConnection(host, port, welcome_frame)

Gal_InitializeStatics() [not needed; called when the bindings are loaded]


533

MITRE Allegro Common Lisp Bindings

We have prepared a (crude) set of Communicator bindings for Common Lisp. We do not provideas much support as we do for Python at the moment. There are fewer demos, and there is nosupport for MITRE utilities equivalents yet. These bindings are specific to Allegro Common Lisp,since the Common Lisp language has no official networking support and we have no experiencewith other Common Lisp dialects.

Enabling Allegro bindingsSee the installation documentation for how to enable the Allegro bindings. Allegro 5.0 or later is required.

Documentation SummaryThe bindings are divided into files which correspond approximately to subdirectories oflibGalaxy: Galaxy.cl (galaxy), GalaxyIO.cl (io, ServerStub), and SLSUtil.cl (util). MGalaxy.cl(MITRE’s libMITREgalaxy) is currently not implemented. There are two other files, cGalaxy.cland acl_callbacks.c, which provided wrappers around the core Galaxy Communicator C library.

The following demos contain Allegro Common Lisp server examples. All examples can be foundin the acl/ subdirectory of these demos:

Basic server example: double Audio and brokering Binary objects for audio and data

The Allegro versions in these examples are, to the greatest extent possible, exact parallels of theC versions, so you can study the two versions side by side.


Servers require explicit mappings for dispatch functions

At the moment, the source code and examples will serve as the primary documentation.However, we provide details and a table of equivalences as a guide.

534

DetailsWe will follow the general outline of the tutorial. Most of the lessons apply just as well to theAllegro bindings as to the core C library. We address the important differences here.

Introducing frames and objectsIn general, because Allegro is a dynamically typed language and supports association listswhose objects can be of heterogeneous types, the wrappers for building objects are superfluous.

GAL_FRAME objects are instances of the galaxy::gal-frame class, which has asubclass galaxy::gal-clause-frame which is used for all the frames used in theinfrastructure itself GAL_STRING objects are Allegro strings GAL_INT objects are Allegro integers GAL_LIST objects are Allegro lists, and the elements of the list can be any of the typesspecified here

Here are some of the examples from the frame tutorial, translated into Allegro. Note that the datain the frame is stored as an association list.

;; {c output :output_string "hello" }

(setf f (make-instance ’galaxy::gal-clause-frame :name "output")) (set-prop f ":output_string" "hello")

;; or

(setf f (make-instance ’galaxy::gal-clause-frame :name "output" :data ’((":output_string" . "hello"))))

;; {c listframe :list ( 5 6 7 ) }

(setf f (make-instance ’galaxy::gal-clause-frame :name "listframe")) (set-prop f ":list" (list 5 6 7))

;; or

(setf f (make-instance ’galaxy::gal-clause-frame :name "listframe" :data ’((":list" . (5 6 7)))))

Server basicsIn order to access the Allegro bindings, we use a fairly complicated set of instructions whichloads the shared object library and the native Allegro code built on top of the library, andconfigures the read-eval-print loop for silent operation. The location of the libraries is specified bythe LIBDIR environment variable. This initialization file is located at

535

$GC_HOME/contrib/MITRE/templates/acl-init.cl . We load this file as we start upAllegro, as follows:

% setenv LIBDIR $GC_HOME/contrib/MITRE/bindings/clisp % acl -qq -batch -backtrack-on-error -e ’(setf lisp:*load-verbose*nil)’ -e ’(load "$GC_HOME/contrib/MITRE/templates/acl-init.cl") --[cmdline_args] < [server_code_file]

There are almost certainly less cumbersome ways of doing this, but they’ll probably involvehaving more verbose printing at startup than you’re interested in.

Each dispatch function takes an environment object and a frame and returns a frame. In thecase of Allegro, the call environment is the first argument, so that methods on classes inheritedfrom galaxy-io::call-environment also have the same signature.

Here’s an Allegro equivalent of the Parse dispatch function:

(defun Parse (env frame) (let* ((input-string (galaxy::get-object frame ":input_string")) (p (parse-sentence input-string))) (make-instance ’galaxy::gal-clause-frame :name "reply" :data ‘((":frame" . ,(parse-tree-to-frame p))))))

Allegro doesn’t use macros to generate the declaration of dispatch functions or the name anddefault port information for the server. This is all handled via explicit function calls:

(setf s (make-instance ’galaxy-io::server :name "Parser" :argv (sys:command-line-arguments:application t) :default-port 10000)) (galaxy-io::add-dispatch-function s "Parse" #’Parse)

Note that because the dispatch functions are explicitly associated with their names, the name ofthe function and the name of the message that invokes it can differ.

Here’s another way to set up your server, by using methods as dispatch functions:

(defclass parse-environment (galaxy-io::call-environment) ())

(defmethod Parse ((env parse-environment) frame) (let* ((input-string (galaxy::get-object frame ":input_string")) (p (parse-sentence input-string))) (make-instance ’galaxy::gal-clause-frame :name "reply" :data ‘((":frame" . ,(parse-tree-to-frame p))))))

536

You can use the specialized classes for the call environment or for the server to set up instancevariables to store arbitrary data.

By default, Allegro servers accept approximately the same set of default arguments as Cservers. The differences are:

The -color argument is not supported. The -thread argument is not supported. Neither the -nottloop nor the -ttloop argument is supported The -debug argument provides for a complete printout of any errors captured by theAllegro server, instead of simply reporting them back in an error frame

The Allegro bindings have no functionality equivalent to the core C argument parsing library.

Error handlingUnlike C, Allegro has explicit errors, and handlers for catching them. If you raise an error in adispatch function (intentionally or otherwise), the Allegro bindings will catch the error and convertit to an error reply. Here’s the Parse function again:

(defun Parse (env frame) (let ((input-string (galaxy::get-object frame ":input_string")) (p nil) (p-frame nil)) (if (not (stringp input-string)) (error "no input string")) (setf p (parse-sentence input-string)) (if (null p) (error "no parse")) (setf p-frame (parse-tree-to-frame p)) (if (null p-frame) (error "can’t convert parse to frame")) (make-instance ’galaxy::gal-clause-frame :name "reply" :data ‘((":frame" . ,p-frame)))))

Sending new messages to the HubSending asynchronous and synchronous messages to the Hub works pretty much as you mightanticipate. Let’s start with the DoGreeting case:

(defun DoGreeting (env frame) (let* ((s "{c FromDialogue :output_frame {c greeting }:is_greeting 1 }") (greeting (galaxy::read-frame-from-string s))) (galaxy-io::write-frame env greeting) nil))

537

If you want to return no frame to the Hub, you must return nil ; if the last expression evaluatedin the function is not nil , it will be treated as the reply.

Now, here’s the DoDialogue example:

(defun DoDialogue (env frame) (let ((msg-frame (make-instance ’galaxy::gal-clause-frame :name "DBQuery")) (response-frame nil)) ;; ...

(galaxy::set-prop msg-frame ":sql_query" sql_query)

(handler-case (let ((response-frame (galaxy-io::dispatch-frame env msg-frame))) ;; Equivalent of GAL_REPLY_MSG_TYPE. ;; Construct presentation of database response ;; ... ) (galaxy-io::dispatch-error (e) ;; Equivalent of GAL_ERROR_MSG_TYPE ;; Relay error back to Hub ;; ... ))))

If the Hub relays an error back to the server, the Allegro bindings will raise the error locally. Theerror raised will be galaxy-io::dispatch-error , as shown here.

Setting up a brokered audio connectionIn order to set up a broker server, you can use the class galaxy-io::broker-data-outdirectly, most of the time. For the broker client, you should subclass galaxy-io::broker-data-in . The methods galaxy-io::env-handle-frame , galaxy-io::env-handle-string , galaxy-io::env-handle-binary , galaxy-io::env-handle-int-16 , etc., can be redefined on the subclassed broker client tohandle the appropriate class of incoming data. Outgoing data is handled using the galaxy-io::write-object method; there is no need for separate functions to write each datatype.

Here’s the equivalent of the Synthesize dispatch function, exemplifying the source serversetup. If the broker setup fails, an error is raised:

(defun Synthesize (env frame) ;; ... (handler-case (let ((b (make-instance ’galaxy-io::broker-data-out

538

:connection (galaxy-io::env-connection env) :timeout 10)) (output-f (make-instance ’galaxy::gal-clause-frame :name "FromSynthesizer" :data ‘((":sample_rate" .,(sample-rate s)) (":encoding_format" .,(encoding-format s)))))) (galaxy-io::populate-frame b output-f ":host" "port") (galaxy-io::write-frame env output-f)) (galaxy-io::broker-connected (e) ;; ... )))

Now, here’s the setup for the client:

(defclass audio-broker (galaxy-io::broker-data-in) ())

;; ...

(defun Play (env frame) (let ((host (galaxy::get-object frame ":host")) (port (galaxy::get-object frame ":port")) (p (make-instance ’audio-pkg)))

;; ...

(handler-case (progn (setf (in-broker p) (make-instance ’audio-broker :environment env :host host :port port)) (galaxy-io::add-callback (in-broker p) galaxy-io::*GAL-BROKER-DATA-DONE-EVENT* #’(lambda ()(audio-output-callback (in-broker p)))) ;; ... ) (galaxy-io::broker-connected (e) ;; ... ))))

The data exchange works as follows. First, on the source server:

(do ((data (poll-synthesis s) (poll-synthesis s))) ((null data)) (galaxy-io::write-object b data))

539

(if (synthesis-is-done s) (galaxy-io::data-done b))

On the client, it’s handled as methods associated with the specialized broker class:

(defmethod galaxy-io::env-handle-int-16 ((b audio-broker) env obj) (format t "[Audio data to user (~d samples)]~%" (length obj)))

Unhandled data types raise galaxy-io::broker-data-not-handled .

Known BugsWriting to /dev/audio in the audio demos seems to be somewhat unreliable. However, sincewe doubt that anyone will be using Allegro Common Lisp to write an audio server, this is nota problem.

EquivalencesThese functions are arranged in alphabetical order of the corresponding C functions, arrangedby the documents in the advanced and reference sections of the documentation which describethose functions.


GalUtil_CPPFrame(sls_verbose_level, fore, back, fr)

[not implemented]

GalUtil_CPPObject(sls_verbose_level, fore, back, to)

[not implemented]

GalUtil_PPFrame(verbose_level, fr)(sls-util::gal-write-level ostream(galaxy::pp-frame fr nil) verbose_level)

GalUtil_PPObject(verbose_level, o)(sls-util::gal-write-level ostream(galaxy::pr-object o nil :pp) verbose_level)

GalUtil_PrObject(verbose_level, o)(sls-util::gal-write-level ostream(galaxy::pr-object o nil) verbose_level)

GalUtil_PrintObject(gverbose_level, o, how) (sls-util::gal-write-level ostream(galaxy::pr-object o nil how) verbose_level)

Gal_AddPred(fr, pred) (push pred (galaxy::frame-preds fr))

Gal_ArrayObjectAdd(obj, data, size) (galaxy::augment-binary-object obj data)

540

Gal_BinaryObject(data, size)(galaxy::binary-object ’galaxy::gal-binary:data data)

Gal_BinarySize(o) (length o)

Gal_BinaryValue(obj, size) (galaxy::value-warn obj ’galaxy::gal-binary)

Gal_Binaryp(obj) (typep obj ’galaxy::gal-binary)

Gal_ClauseFramep(fr) (typep fr ’galaxy::gal-clause-frame)

Gal_ClauseValue(o)(galaxy::value-warn o ’galaxy::gal-clause-frame)

Gal_Clausep(obj) (typep obj ’galaxy::gal-clause-frame)

Gal_ClearPreds(fr) (setf (galaxy::frame-preds fr) nil)

Gal_CopyFrame(fr) [not implemented]

Gal_CopyObject(obj) [not implemented]

Gal_CreateBinaryObject(data, size, manage_memory)

(galaxy::binary-object ’galaxy::gal-binary:data data)


(galaxy::binary-object ’galaxy::gal-float-32:data data)


(galaxy::binary-object ’galaxy::gal-float-64:data data)

Gal_CreateFloatObject(value, manage_memory) [not needed]

Gal_CreateFrameObject(value, manage_memory) [not needed]


(galaxy::binary-object ’galaxy::gal-int-16:data data)


(galaxy::binary-object ’galaxy::gal-int-32:data data)


[not implemented]

Gal_CreateListObject(values, n, free_fn , manage_memory)

[not needed]

Gal_CreateStringObject(cp, manage_memory) [not needed]

Gal_CreateVarMapping(num_pairs, ...) [not implemented]

Gal_DelPred(fr, i) (galaxy::rem-pred fr i)

Gal_DelPredByName(fr, name) (galaxy::rem-pred-by-name fr name)

541

Gal_DelProp(fr, key) (galaxy::del-prop fr key)

Gal_DeletePreds(fr, pred_name) [not implemented]

Gal_DoPreds(fr, pred_fn , caller_data) (mapcar pred_fn (galaxy::frame-preds fr))

Gal_DoProperties(fr, prop_fn , caller_data) (mapcar prop_fn (galaxy::frame-data fr))

Gal_FindKey(fr, key_name) [not implemented]

Gal_FindPred(fr, pred_name) [not implemented]

Gal_FindPredParent(frame, name, parent, findpar, nth)

[not implemented]

Gal_FindTopic(fr, topic_name) [not implemented]

Gal_Float32Size(o) (length o)

Gal_Float32Value(obj, size)(galaxy::value-warn obj ’galaxy::gal-float-32)

Gal_Float32p(o) (typep o ’galaxy::gal-float-32)

Gal_Float64Size(o) (length o)

Gal_Float64Value(obj, size)(galaxy::value-warn obj ’galaxy::gal-float-64)

Gal_Float64p(o) (typep o ’galaxy::gal-float-64)

Gal_FloatObject(val) [not needed]

Gal_FloatValue(to) (galaxy::value-warn obj ’galaxy::gal-float)

Gal_Floatp(obj) (typep obj ’galaxy::gal-float)

Gal_FrameEqual(sf1, sf2) [not implemented[

Gal_FrameIsType(fr, type) (eq (galaxy::frame-type fr) type)

Gal_FrameName(fr) (galaxy::frame-name fr)

Gal_FrameNameEq(fr, name) (string= (galaxy::frame-name fr) name)

Gal_FrameNamesEq(fr1, fr2)(string= (galaxy::frame-name fr1)(galaxy::frame-name fr2))

Gal_FrameObject(val) [not needed]

Gal_FrameValue(to) (galaxy::value-warn obj ’galaxy::gal-frame)

Gal_Framep(obj) (typep obj ’galaxy::gal-frame)

Gal_FreeFrame(fr) [not needed]

542

Gal_FreeObject(obj) [not needed]

Gal_FreeWrapper(to) [not needed]

Gal_GetBinary(fr, key, size) (galaxy::get-value fr key ’galaxy::gal-binary)

Gal_GetDetailedType(o) (galaxy::get-detailed-type o)

Gal_GetFloat(fr, key) (galaxy::get-value fr key ’galaxy::gal-float)

Gal_GetFloat32(fr, key, size)(galaxy::get-value fr key ’galaxy::gal-float-32)

Gal_GetFloat64(fr, key, size)(galaxy::get-value fr key ’galaxy::gal-float-64)

Gal_GetFrame(fr, key) (galaxy::get-value fr key ’galaxy::gal-frame)

Gal_GetFrameType(fr) (galaxy::frame-type fr)

Gal_GetInt(fr, key) (galaxy::get-value fr key ’galaxy::gal-int)

Gal_GetInt16(fr, key, size) (galaxy::get-value fr key ’galaxy::gal-int-16)

Gal_GetInt32(fr, key, size) (galaxy::get-value fr key ’galaxy::gal-int-32)

Gal_GetInt64(fr, key, size) [not implemented]

Gal_GetList(fr, key, length) (galaxy::get-value fr key ’galaxy::gal-list)

Gal_GetListObject(obj, n) (elt obj n)

Gal_GetListValue(o, n, type) (galaxy::value-warn (elt o n) type)

Gal_GetObject(fr, key) (galaxy::get-object fr key)

Gal_GetObjectType(o) (galaxy::get-object-type o)

Gal_GetObjectTypeString(o)(galaxy::object-type-string(galaxy::get-object-type o))

Gal_GetPred(fr, i) (elt (galaxy::frame-preds fr) i)

Gal_GetPredByName(fr, name) (galaxy::get-pred-by-name fr name)

Gal_GetProperties(fr, nkeys)(galaxy::gal-get-properties fr)[get-properties is a CL internal function]

Gal_GetString(fr, key) (galaxy::get-value fr key ’galaxy::gal-string)

Gal_GetTopicFrame(fr, key)(galaxy::get-value fr key ’galaxy::gal-topic-frame)

Gal_Int16Size(o) (length o)

543

Gal_Int16Value(obj, size) (galaxy::value-warn obj ’galaxy::gal-int-16)

Gal_Int16p(o) (typep o ’galaxy::gal-int-16)

Gal_Int32Size(o) (length o)

Gal_Int32Value(obj, size) (galaxy::value-warn obj ’galaxy::gal-int-32)

Gal_Int32p(o) (typep o ’galaxy::gal-int-32)

Gal_Int64Size(to) [not implemented]

Gal_Int64Value(obj, size) [not implemented]

Gal_Int64p(to) [not implemented]

Gal_IntObject(val) [not needed]

Gal_IntValue(o) (galaxy::value-warn o ’galaxy::gal-int)

Gal_Intp(obj) (typep obj ’galaxy::gal-int)

Gal_ListLength(obj) (length obj)

Gal_ListObject(values, n) [not needed]

Gal_ListObjectAdd(obj, elt) (nconc obj (list elt))

Gal_ListObjectFromElements(n, ...) [not needed]

Gal_ListValue(obj, n) (galaxy::value-warn obj ’galaxy::gal-list)

Gal_Listp(obj) (typep obj ’galaxy::gal-list)

Gal_MakeClauseFrame(name)(make-instance ’galaxy::gal-clause-frame:name name)

Gal_MakeFrame(name, type)

(make-instance ’galaxy::gal-topic-frame:name name) OR (make-instance’galaxy::gal-clause-frame :name name) OR(make-instance ’galaxy::gal-pred-frame:name name)

Gal_MakePredFrame(name)(make-instance ’galaxy::gal-pred-frame:name name)

Gal_MakeTopicFrame(name)(make-instance ’galaxy::gal-topic-frame:name name)

Gal_MatchFrame(sf1, sf2) [not implemented]

Gal_MatchKeyValue(fr, key_name, match) [not implemented]

Gal_NumNonNullProperties(fr) (galaxy::num-properties fr)

544

Gal_NumPreds(fr) (length (galaxy::frame-preds fr))

Gal_NumProperties(fr) (galaxy::num-properties fr)

Gal_ObjectByteCount(obj) [not implemented]

Gal_ObjectCaseEqual(obj1, obj2) [not implemented]

Gal_ObjectEqual(obj1, obj2) [not implemented]

Gal_ObjectToString(o) (galaxy::pr-object o nil)

Gal_ObjectTypeString(object_type) (galaxy::object-type-string object_type)

Gal_OutlineFrame(fr, sls_verbose_level) [not implemented]

Gal_OutlineObject(to, sls_verbose_level) [not implemented]

Gal_PPFrame(fr) (galaxy::pp-frame fr T)

Gal_PPFrameToFile(fr, fp) (galaxy::pp-frame fr fp)

Gal_PPFrameToString(fr, buf, bufsizeptr) (galaxy::pp-frame fr nil)

Gal_PPObject(o)(galaxy::pr-object o T galaxy::*GAL-PR-PRINT*)

Gal_PPObjectToFile(o, fp)(galaxy::pr-object o fp galaxy::*GAL-PP-PRINT*)

Gal_PrFrame(fr) (galaxy::pr-frame fr T)

Gal_PrFrameToFile(fr, fp) (galaxy::pr-frame fr fp)

Gal_PrFrameToString(fr, buf, bufsizeptr) (galaxy::pr-frame f nil)

Gal_PrObject(obj) (galaxy::pr-object obj T)

Gal_PrObjectToFile(obj, fp) (galaxy::pr-object obj fp)

Gal_PredFramep(fr) (typep fr ’galaxy::gal-pred-frame)

Gal_PredValue(o)(galaxy::value-warn obj ’galaxy::gal-pred-frame)

Gal_Predp(obj) (typep obj ’galaxy::gal-pred-frame)

Gal_PrintFrameToFile(fr, fp, how) (galaxy::print-frame fr fp how)

Gal_PrintFrameToString(fr, irpbuf, bufsizeptr, how) (galaxy::print-frame fr nil how)

Gal_ReadFrameFromFile(fp) [not implemented]

Gal_ReadFrameFromString(buf) (galaxy::read-frame-from-string buf)

545

Gal_ReadObjectFromFile(fp) [not implemented]

Gal_ReadObjectFromString(buf) (galaxy::read-irp-value buf) (approximately)

Gal_ReadVarFrameFromString(buf, map) [not implemented]

Gal_RemPred(fr, i) (galaxy::rem-pred fr i)

Gal_RemPredByName(fr, name) (galaxy::rem-pred-by-name fr name)

Gal_RemProp(fr, key) (talaxy::rem-prop fr key)

Gal_SetFrameName(fr, name) (setf (galaxy::frame-name fr) name)

Gal_SetFrameType(fr, type) (setf (galaxy::frame-type fr) type)

Gal_SetProp(fr, key, obj) (galaxy::set-prop fr key obj)

Gal_StringObject(val) [not needed]

Gal_StringValue(to) (galaxy::value-warn obj ’galaxy::gal-string)

Gal_Stringp(obj) (typep obj ’galaxy::gal-string)

Gal_TopicFramep(fr) (typep obj ’galaxy::gal-topic-frame)

Gal_TopicValue(to) [not implemented]

Gal_Topicp(obj) (typep obj ’galaxy::gal-topic-frame)

Gal_VAReadVarFrameFromString(buf, num_pairs, ...)

[not implemented]

_gal_free_object(obj) [not needed]


GalIO_GetError(f, err_desc)[not needed; catch galaxy-io::dispatch-error]

GalSS_AddDispatchFunction(i, name, fn,in_key_array, allow_other_in_keys, reply_provided,out_key_array, allow_other_out_keys)

(galaxy-io::add-dispatch-function ...)

GalSS_EnvDestroyToken(env) (galaxy-io::destroy-token env)

GalSS_EnvDispatchFrame(env, frame, t) (galaxy-io::dispatch-frame env frame)

GalSS_EnvError(env, description) [not needed; just raise error]

546

GalSS_EnvGetClientData(env, name)[not needed; specialize class and addinstance variable]

GalSS_EnvGetCommData(env)[not needed; specialize class and addinstance variable]

GalSS_EnvReply(env, f) (galaxy-io::reply env f)

GalSS_EnvSetCommData(env, data, free_fn )[not needed; specialize class and addinstance variable]

GalSS_EnvWriteFrame(env, frame, do_block) (galaxy::write-frame env frame)

GalSS_InitializeServerDefaults(scomm, name, port)

[not implemented]

Gal_CreateDispatchFnKeyArray(ignore, ...) [not needed]

Gal_FreeDispatchFnKeyArray(entry) [not needed]

_GalSS_InitializeDefaults(scomm)[not needed; provided to scomm during initialization]

_GalSS_init_server(server, argc, argv) [not needed]

_GalSS_print_usage(argc, argv) [not implemented]


GalIO_BrokerDataDone(b) (galaxy-io::data-done b)

GalIO_BrokerDataInInit(host, port, frame, fnptr,caller_data, poll_ms)

[not implemented]

GalIO_BrokerDataOutDone(b) (galaxy-io::data-done b)

GalIO_BrokerDataOutInit(gcomm, poll_ms, timeout_seconds)

(make-instance ’galaxy-io::broker-data-out:connection gcomm :timeout timeout_seconds)

GalIO_BrokerIsDone(b) [not implemented]

GalIO_BrokerPopulateFrame(b, f, host_key, port_key)

(galaxy-io::populate-frame b f host_key port_key)

GalIO_BrokerSetFinalizer(b, finalizer) [not implemented]

GalIO_BrokerStructDequeue(b, bqueue) [not needed; no broker queues in Allegro]

547

GalIO_BrokerStructQueueAppend(b, bqueue) [not needed; no broker queues in Allegro]

GalIO_BrokerStructQueuePop(bqueue) [not needed; no broker queues in Allegro]

GalIO_BrokerWriteBinary(b, data, n_bytes) (galaxy-io::write-object b data)

GalIO_BrokerWriteFloat(b, f) (galaxy-io::write-object b f)

GalIO_BrokerWriteFloat32(b, data, n_floats) (galaxy-io::write-object b data)

GalIO_BrokerWriteFloat64(b, data, n_floats) (galaxy-io::write-object b data)

GalIO_BrokerWriteFrame(b, frame) (galaxy-io::write-object b f)

GalIO_BrokerWriteInt(b, i) (galaxy-io::write-object b i)

GalIO_BrokerWriteInt16(b, data, n_ints) (galaxy-io::write-object b data)



GalIO_BrokerWriteList(b, elts, n_elts) (galaxy-io::write-object b data)

GalIO_BrokerWriteObject(b, o) (galaxy-io::write-object b o)

GalIO_BrokerWriteString(b, str) (galaxy-io::write-object b str)

GalIO_CommBrokerDataInInit(host_gcomm, host,port, frame, fnptr, poll_ms, caller_data,caller_data_free_fn )

(make-instance ’galaxy-io::broker-data-in:host host :port port :frame frame:connection host_gcomm)

GalIO_ForceBrokerExpiration(b) [not implemented]

GalIO_FrameSetBrokerCallID(f, call_id) [not implemented]

GalIO_GetBrokerCallID(b) (galaxy-io::broker-data-out-call-id b)

GalIO_GetBrokerCallerData(b) [not needed; use child class]

GalIO_GetBrokerData(b) [not needed; use child class]

GalIO_GetBrokerFrame(b) [not implemented]

GalIO_GetBrokerListenPort(b) (galaxy-io::broker-data-out-port b)

GalIO_IPAddress() (galaxy-io::ip-address)

GalIO_SetBrokerActive(b) [not needed; no broker queues in Allegro]

GalIO_SetBrokerData(b, caller_data, free_fn ) [not needed; use child class]

GalSS_BrokerGetEnvironment(b) (galaxy-io::broker-data-in-environment b)

548

GalSS_BrokerSetEnvironment(b, env)[not needed; initialize in broker with environment]

GalSS_EnvBrokerDataInInit(env, host, port, frame,fnptr, poll_ms, refptr, free_fn )

(make-instance ’galaxy-io::broker-data-in:host host :port port :frame frame:environment env)

Server Architecture

GalIO_AddBrokerCallback(b, callback_event, fn, callback_data)

(galaxy-io::add-callback b callback_event fn)

GalIO_AddConnectionBrokerCallback(gcomm,callback_event, connect_callback, callback_data)

[not needed; specialize initialization methods]

GalIO_AddConnectionCallback(gcomm,callback_event, connect_callback, callback_data)

[not implemented]

GalIO_AddConnectionDispatchFnCallback(gcomm,dispatch_callback, callback_data)

[not implemented]

GalIO_AddServerCallback(scomm, callback_event,fn, callback_data)

[not implemented]

GalIO_AddServerConnectCallback(scomm,connect_callback, callback_data)

[not implemented]

GalIO_CommValidating(gcomm)(galaxy-io::server-validate(galaxy-io::connection-server gcomm))

GalIO_CommWriteFrame(gcomm, frame, do_block)

(galaxy-io::write-frame gcomm frame)

GalIO_ContactHub(host, port, scomm, session_id, client_poll_flags)

[not implemented]

GalIO_DispatchViaHub(gcomm, frame, msg_type_ptr)

(galaxy-io::dispatch-frame gcomm frame)

GalIO_EnableDispatchFnValidation(scomm) (setf (galaxy-io::server-validate s) t)

GalIO_GetCommClientData(gcomm, name) [not needed; use child class]

GalIO_GetCommData(gcomm) [not needed; use child class]

GalIO_GetCommServerData(gcomm) [not needed; use child class]

549

GalIO_GetCommServerName(gcomm)(galaxy-io::server-name(galaxy-io::connection-server gcomm))

GalIO_GetServerClientData(server, name) [not needed; use child class]

GalIO_GetServerData(scomm) [not needed; use child class]

GalIO_GetServerDefaultPort(scomm) (galaxy-io::server-default-port scomm)

GalIO_GetServerListenPort(scomm) (galaxy-io::server-port scomm)

GalIO_GetServerMaxConnections(scomm) (galaxy-io::server-maxconns scomm)

GalIO_GetServerName(scomm) (galaxy-io::server-name scomm)

GalIO_GetServerNumConnections(scomm) (length (galaxy-io::server-conns scomm))

GalIO_GetUniqueConnection(scomm) [not implemented]

GalIO_OperateOnConnections(scomm, arg, op )(mapcar #’(lambda (a) (op (cdr a))(galaxy-io::server-conns scomm))

GalIO_RemoveBrokerCallback(b, cb) [not implemented]

GalIO_RemoveConnectionCallback(gcomm, cb) [not implemented]

GalIO_RemoveServerCallback(scomm, cb) [not implemented]

GalIO_ServerListenStatus(scomm) [not implemented]

GalIO_ServerSessionID(scomm) [not implemented]

GalIO_ServerUsesTimedTasks(server) [not implemented; polling by select]

GalIO_SetCommClientData(gcomm, name, client_data)

[not needed; use child class]

GalIO_SetCommData(gcomm, data, free_fn ) [not needed; use child class]

GalIO_SetServerClientData(server, name, client_data)

[not needed; use child class]

GalIO_SetServerData(scomm, data, free_fn ) [not needed; use child class]

GalIO_SetServerDefaultPort(scomm, port)(setf (galaxy-io::server-default-port scomm) port)

GalIO_SetServerMaxConnections(scomm, max)(setf (galaxy-io::server-maxconns scomm) max)

GalIO_SetServerName(scomm, name)(setf (galaxy-io::server-name scomm) name)

GalSS_CmdlineInitializeServer(argc, argv) [not implemented]

550

GalSS_CmdlineSetupServer(argc, argv) [not implemented]

GalSS_DefaultServerArgs() [not implemented]

GalSS_EnvComm(env) (galaxy-io::env-connection c)

GalSS_ExtractCmdlineServerArgs(arg_pkg, argc,argv, new_argc_ptr, new_argv_ptr)

[not needed]

GalSS_ExtractServerArgs(argc, argv,new_argc_ptr, new_argv_ptr)

[not needed]

GalSS_FreeArgPkg(arg_pkg) [not needed]

GalSS_InitializeServer(server_port, max_conns,use_color, do_assert, use_ttloop, validate,new_argc, new_argv)

(make-instance ’galaxy-io::server ... )

GalSS_InitializeServerFromServerArgs(arg_pkg,new_argc, new_argv)


GalSS_InitializeServerToplevel(server_port,max_conns, use_color, do_assert, loop_type,validate, verbosity, server_listen_status,client_pair_string, session_id, new_argc, new_argv)


GalSS_RunServer(server) [not implemented]

GalSS_SAFixAssert(arg_pkg, assert) [not implemented]

GalSS_SAFixColor(arg_pkg, color) [not implemented]

GalSS_SAFixContactHubInfo(arg_pkg,client_pair_status, session_id, old_session_id_ptr)

[not implemented]

GalSS_SAFixLoopType(arg_pkg, loop_type) [not implemented]

GalSS_SAFixMaxConns(arg_pkg, max_conns) [not implemented]

GalSS_SAFixPort(arg_pkg, port) [not implemented]

GalSS_SAFixServerListenStatus(arg_pkg, server_listen_status)

[not implemented]

GalSS_SAFixServerLocations(arg_pkg, server_locations_file)

[not implemented]

GalSS_SAFixValidate(arg_pkg, validate) [not implemented]

GalSS_SAFixVerbosity(arg_pkg, verbosity) [not implemented]

551

GalSS_SetupServer(arg_pkg, new_argc, new_argv)


GalSS_StartAndRunServer(server) (galaxy-io::run-server s)

The Timed Task LoopNone of these functions are implemtened,because Allegro doesn’t use the timed task loop.

Gal_AddIdleFunction(func, client_data) [not implemented]

Gal_AddTask( task , refcon, num_millisecs,read_blocking_available, cleanup_fn )

[not implemented]

Gal_AddTaskExtended( task , caller_data,num_millisecs, read_blocking_available,read_socket, write_socket, err_socket, read_file,write_file, err_file, condition, cleanup_fn )

[not implemented]

Gal_AddTaskWithFileIO( task , refcon,num_millisecs, read_blocking_available, read_file,write_file, cleanup_fn )

[not implemented]

Gal_AddTaskWithSocketIO( task , refcon,num_millisecs, read_blocking_available,read_socket, write_socket, cleanup_fn )

[not implemented]

Gal_AddTimedTask(task, refcon, num_millisecs) [not implemented]

Gal_AddTimedTaskWithFileIO(task, refcon,num_millisecs, read_file, write_file)

[not implemented]

Gal_AddTimedTaskWithSocketIO(task, refcon,num_millisecs, read_socket, write_socket)

[not implemented]

Gal_EnableTimedTaskThreads() [not implemented]

Gal_EndTasks(immediate) [not implemented]

Gal_MaybeEndTask(immediate, deferred) [not implemented]

Gal_ReAddTask(p, refcon, num_millisecs,read_blocking_available, cleanup_fn )

[not implemented]

Gal_ReAddTaskExtended(p, caller_data,num_millisecs, read_blocking_available,read_socket, write_socket, err_socket, read_file,write_file, err_file, condition, cleanup_fn )

[not implemented]

552

Gal_ReAddTaskWithFileIO(p, refcon,num_millisecs, read_blocking_available, read_file,write_file, cleanup_fn )

[not implemented]

Gal_ReAddTaskWithSocketIO(p, refcon,num_millisecs, read_blocking_available,read_socket, write_socket, cleanup_fn )

[not implemented]

Gal_RemoveIdleFunction(func) [not implemented]

Gal_RemoveTask(task_id) [not implemented]

Gal_RemoveTimedTask(task, refcon) [not implemented]

Gal_RunIdleFunctions() [not implemented]

Gal_TaskPkgBlocking(pkg) [not implemented]

Gal_TaskPkgData(pkg) [not implemented]

Gal_TaskPkgRunReasons(pkg) [not implemented]

Gal_TimedTaskLoopThreadWaiter() [not implemented]

Gal_TimedTaskThreadsEnabled() [not implemented]

Gal_TimedTasksLoop() [not implemented]

Gal_TimedTasksLoopExit() [not implemented]

Gal_TimedTasksLoopHandler(tv) [not implemented]

Managing Session Information and Using Continuations

GalSS_EnvAddTask(env, task , caller_data,num_millisecs, read_blocking_available,read_socket, write_socket, err_socket, read_file,write_file, err_file, condition, cleanup_fn )

[not implemented]

GalSS_EnvCreate(gcomm)(make-instance ’galaxy-io::call-environment:connection gcomm)

GalSS_EnvDispatchFrameWithContinuation(env,frame, fn, continuation_state,continuation_state_free_fn )

(galaxy-io::dispatch-frame-with-continuationenv frame fn)

GalSS_EnvGetSessionID(env) (galaxy-io::get-session-id env)

553

GalSS_EnvLock(env) [not needed]

GalSS_EnvMaintainInLocation(gcomm,initial_session_id, env_loc)

[not implemented]

GalSS_EnvPostponeReply(env) [not implemented]

GalSS_EnvReturnRequired(env) [not implemented]

GalSS_EnvUnlock(env) [not needed]

GalSS_EnvUpdateSessionID(env, session_id)(galaxy-io::update-session-id env session_id)

GalSS_TaskGetEnvironment(p) [not implemented]

GalSS_TaskSetEnvironment(p, env) [not implemented]


GalIO_AddServiceType(server, stype) (galaxy-io::add-service-type server stype)

GalIO_ServerModifyProperties(server,new_properties, delete_properties)

(galaxy-io::modify-properties server:properties-to-set new_properties:properties-to-delete delete_properties)

GalIO_ServerProperties(server) [not implemented]

GalSS_EnvDeleteServerProperties(env, keys)(galaxy-io::modify-server-properties env:properties-to-delete keys)

GalSS_EnvDeleteSessionProperties(env, keys)(galaxy-io::modify-session-properties env:properties-to-delete keys)

GalSS_EnvGetServerProperties(env, keys) (galaxy-io::get-server-properties env keys)

GalSS_EnvGetSessionProperties(env, keys) (galaxy-io::get-session-properties env keys)

GalSS_EnvModifyServerProperties(env,properties_to_set, properties_to_delete)

(galaxy-io::modify-server-properties:properties-to-set properties_to_set:properties-to-delete properties_to_delete)

GalSS_EnvModifySessionProperties(env,properties_to_set, properties_to_delete)

(galaxy-io::modify-session-properties:properties-to-set properties_to_set:properties-to-delete properties_to_delete)

GalSS_EnvSetServerProperties(env, properties)(galaxy-io::modify-server-properties:properties-to-set properties_to_set)

554

GalSS_EnvSetSession(env, session_name, lock_info)

(galaxy-io::set-session env session_name lock_info)

GalSS_EnvSetSessionProperties(env, properties)(galaxy-io::modify-session-properties:properties-to-set properties_to_set)

Special Main Loops

The basic Allegro main loop worksdifferently than the C main loop, socustomizing it and embedding it worksdifferently as well. Most of these functionsdon’t make any sense in Allegro.

GalIO_BrokerDataInCallbackHandler(b, read_blocking)

[not implemented]

GalIO_BrokerDataOutCallbackHandler(b) [not implemented]

GalIO_BrokerReadReady(b) [not implemented]

GalIO_BrokerWriteReady(b) [not implemented]

GalIO_CommReadReady(gcomm) [not implemented]

GalIO_CommWriteReady(gcomm) [not implemented]

GalIO_ConnectionCallbackHandler(gcomm, read_blocking)

[not implemented]

GalIO_DigestServerLocations(client_pair_string) [not implemented]

GalIO_GetServerLocations(scomm) [not implemented]

GalIO_NthHostAndPort(locs, i, port) [not implemented]

GalIO_NumServerLocations(locs) [not implemented]

GalIO_ServerCallbackHandler(scomm,read_blocking, new_conn_ptr)

[not implemented]

GalIO_ServerCheckHubContacts(scomm) [not implemented]

GalSS_ELRBroker(elr) [not implemented]

GalSS_ELRCopy(source) [not implemented]

GalSS_ELRCreate(scomm, timer_set_fn,timer_unset_fn, fd_set_fn, fd_unset_fn,behavior_fn, timer_is_persistent)

[not implemented]

555

GalSS_ELRDestroy(elr) [not implemented]

GalSS_ELRDoCallback(elr, timer_or_fd) [not implemented]

GalSS_ELRGComm(elr) [not implemented]

GalSS_ELRGetLoopData(elr) [not implemented]

GalSS_ELRSComm(elr) [not implemented]

GalSS_ELRSetBrokerInCallback(elr, fn) [not implemented]

GalSS_ELRSetBrokerOutCallback(elr, fn) [not implemented]

GalSS_ELRSetConnectionCallback(elr, fn) [not implemented]

GalSS_ELRSetLoopData(elr, loop_data,loop_data_free_fn )

[not implemented]

GalSS_ELRSetServerClientCallback(elr, fn) [not implemented]

GalSS_ELRSetServerListenerCallback(elr, fn) [not implemented]

GalSS_ELRSetupServer(external_arg_pkg, argc,argv, timer_set_fn, timer_unset_fn, fd_set_fn,fd_unset_fn, behavior_fn, loop_data,loop_data_free_fn , timer_is_persistent)

[not implemented]

GalSS_ELRShutdown(elr) [not implemented]

GalSS_ELRUpdatePollIntervals(elr,server_client_poll_ms, conn_ms, broker_ms)

[not implemented]

Signal Handling

Gal_AddSignalHandler(sig, handler ) [not implemented]

Gal_InitializeSignals() [not implemented]

Gal_SignalsInitialized() [not implemented]


556

GalUtil_Assert(truth, format, ...)(sls-util::gal-assert (galaxy-io::ostreamconn) truth msg)



GalUtil_CPrint(level, fore, back, format, ...) [not implemented]

GalUtil_Debug1(format, ...)(sls-util::debug1 (galaxy-io::ostream conn) msg)

GalUtil_Debug2(format, ...)(sls-util::debug2 (galaxy-io::ostream conn) msg)

GalUtil_Error(format, ...)(sls-util::gal-error (galaxy-io::ostream conn) msg)

GalUtil_Fatal(format, ...)(sls-util::fatal (galaxy-io::ostream conn) msg)

GalUtil_OACheckUsage(argc, argv, oas, first_real_arg)

[not implemented]

GalUtil_OAExtract(argc, argv, oas, key, ...) [not implemented]

GalUtil_OAExtractAsserting(argc, argv, oas, key, ...)

[not implemented]

GalUtil_OAPrintUsage(argc, argv, oas) [not implemented]

GalUtil_PInfo1(format, ...)(sls-util::pinfo1 (galaxy-io::ostream conn) msg)

GalUtil_PInfo2(format, ...)(sls-util::pinfo2 (galaxy-io::ostream conn) msg)

GalUtil_Print(level, format, ...)(sls-util::gal-write-level (galaxy-io::ostreamconn) msg level)

GalUtil_SetVerbose(verbose_level) [not implemented]

GalUtil_VerboseUseBW() [not implemented]

GalUtil_VerboseUseColor() [not implemented]

GalUtil_Warn(format, ...)(sls-util::gal-warn (galaxy-io::ostream conn) msg)


557

GalIO_ClientConnect(name, host, port, silent,welcome_frame, reply_frame)

[not implemented]

Gal_InitializeStatics()[not needed; called when bindings are loaded]


558

Administration

559

Installing the Galaxy Communicator Distribution

Preliminaries Site Configuration Compilation Cross compilation

Preliminaries

Supported and semi-supported platformsThe Galaxy Communicator distribution is actively supported on the following platforms:

Sparc Solaris version 2.7 (sparc-solaris) Red Hat Linux for x86 6.2 (x86-linux) Win32 (x86-nt)

Labels in parentheses correspond to the ARCHOS values described below.

The Galaxy Communicator distribution has also been recently tested to some degree or anotheron the following platforms:

SGI IRIX 32- and 64-bit (mips-irix, mips64-irix) FreeBSD 4.3 (x86-FreeBSD) MacOS X 10.1/Darwin 1.4 (powerpc-Darwin)

These platforms are not actively supported, but we will continue to test on these platforms whenpossible and incorporate appropriate fixes.

To the best of our knowledge, the following table reflects the status of the distribution on thesesix platforms. In all cases, the Hub and core C libraries, compiled statically, are known to workand pass our test suite.

560

Static libraries

Shared libraries

Python bindings

Allegro bindings

Java bindings

Threads64-bit integerXDR encode/decode

SparcSolaris 2.7

yes yes yes yes yes yes not tested

Red HatLinux for x86

yes yes yes unknown yes yes not tested

Win32 yes no no unknown yes yes not tested

SGI IRIX yes yes

no (don’tknowhow tododynamicloading yet)

unknown unknown yes not tested

FreeBSD yes yes yes unknown unknown unknownno (xdr_hyper missing)

MacOS X/Darwin

yes no

no (don’tknowhow tododynamicloading yet)

unknown yesno (pthreadimplementation incomplete)

no (xdr_hyper missing)

For more details on how to configure these various options, see the configuration section.

Unsupported platformsThe Galaxy Communicator distribution is not yet supported on the following current platforms,because we don’t have access to them, but may work just fine on them:

Sparc Solaris 2.8 Red Hat Linux for x86 7.0 and later

The Galaxy Communicator distribution, at one point or another, has also worked on the followingplatforms, and may still work, although these platforms are no longer tested or supported.

561

Sparc SunOS 4.3.1_U1 (sparc-sunos) Alpha OSF V3.2 (alpha-osf) Red Hat Linux for x86 4.2, 6.0, 6.1 Solaris for x86 2.5, 2.6 (x86-solaris) mkLinux and LinuxPPC (ppc-linux) SPARC Solaris version 2.5.1

Third-party sourcesThe Galaxy Communicator distribution requires gcc and GNU make. We also recommend usingGNU mkdir and GNU ar for best results. Sources for the GNU libraries and tools can be obtainedfrom GNU:

gcc: on Sparc Solaris, we currently use gcc-2.95.3, but we believe gcc 2.8.1 and later willwork, as will egcs 1.1.1 or later GNU make: make-3.75.tar.gz or later GNU ar: binutils-2.9.1.tar.gz or later GNU mkdir: fileutils-3.16.tar.gz or later

Python 1.5 or later with Tkinter support is recommended for a number of the MITRE demos,examples and utilities. Python is available from www.python.org:

version 1.5

We have tested the Python bindings with distributions up to 2.1, and we believe that they stillwork, but we haven’t upgraded our day-to-day development to the current versions of Python yet.

Setting up the tutorialIf you intend to work through the tutorial, the following three points are crucial:

Python with Tkinter support is required rather than recommended, since the tutorial reliesextensively on tools implemented in Python, such as the process monitor. Because of this Python requirement, the tutorial will currently not run on Win32, becausethere are some crucial Win32 incompatibilities with Python which have yet to be resolved. Finally, be sure that when you configure and compile the distribution on the platform youintend to run the tutorial on, you enable compilation of the MITRE contributions by leavinguncommented the COMPILE_MITRE Makefile variable.

Site configurationIn 2.0, we began a slow migration toward GNU configure . As of version 3.3, this migration isfinally complete. The old configuration method still works, but there are new options which makeit even easier to preserve configuration information, and all aspects of the configuration can becontrolled by arguments to GNU configure . On Win32, we’ve provided header files which

562

http://www.python.org/1.5/

http://www.python.org/

ftp://prep.ai.mit.edu/pub/gnu/fileutils-3.16.tar.gz

ftp://prep.ai.mit.edu/pub/gnu/binutils-2.9.1.tar.gz

ftp://prep.ai.mit.edu/pub/gnu/make-3.75.tar.gz

http://egcs.cygnus.com/egcs-1.1/egcs-1.1.1.html

ftp://prep.ai.mit.edu/pub/gnu/gcc/gcc-2.95.3.tar.gz

http://www.gnu.org/

reflect the values which GNU configure would have generated for Windows. The instructionshere are for Unix.

A note about platform and operating system settings[This information is also provided in file config.make , described below.]

The Galaxy Communicator infrastructure uses an MIT-developed system for maintaining librariesand binaries for multiple architectures and operating systems under a single source tree.Environment/make variables specify the machine architecture (ARCH), the operating system(OS), and their combination (ARCHOS = $(ARCH)-$(OS)). At compile time, object files, libraries,and binaries are created in ARCHOS subdirectories. The ARCHOS-independent executablesare links to a script which determines the appropriate ARCHOS-dependent binary to call at runtime based on the name of the link and the environment variable which contains the ARCHOSinformation (configurable at compile time). So there’s no make install for GalaxyCommunicator; all the binaries and libraries are sorted out in place. For more details, see the directory structure documentation.

The variables for ARCH and OS are determined by the scripts templates/set_arch.cmdand templates/set_os.cmd . The values of these scripts will be the Makefile variables$(ARCH) and $(OS), respectively, and $(ARCHOS) = $(ARCH)-$(OS). The C files in thedistribution will also be compiled with the preprocessor flags -D__$(ARCH)__ -D__$(OS)__ ifyou use the MIT Makefile templates and these flags aren’t already provided by the preprocessor;if you don’t use the MIT Makefile templates, you may still find these settings useful (if you don’tuse GNU configure and you’re trying to do cross-platform development, for instance). Hereare the settings:

Solaris 2/Solaris 7 (SunOS 5): on Sparc: sparc-solaris on x86/Pentium: x86-solaris

Solaris 1 (SunOS 4): on Sparc: sparc-sunos on x86/Pentium: x86-sunos

Linux: on x86/Pentium: x86-linux

SGI Irix: 32-bit: mips-irix 64-bit: mips64-irix

MacOS X/Darwin: on PowerPC: powerpc-Darwin

In almost all other cases, ARCH = ‘uname -m‘ and OS = ‘uname -s‘. Windows doesn’t use theMakefile structure, and its preprocessor flag is WIN32, as usual. You can run the two scripts byhand to verify that the ARCH, OS, and ARCHOS variables will be set correctly.

563

New configuration methodThe new configuration method works basically like this:

% cd templates % ./configure <args> % cd .. % make

The possible arguments to configure are described below.

The first thing the configure script will do is compute the appropriate values for your architecture and operating system. Then, the configure script will check your environment for arange of features, and generate two files which it will place in templates/<arch>-<os> . Thefirst, config.make , contains compilation and configuration settings which you may change afteryou run configure , if you have to (for example, you may want to enable the Python bindings,but forget to pass the appropriate arguments to configure to do so). The second, GC_config.h , is a header file which sets the appropriate compilation flags for your platform; donot edit this file.

Old configuration methodThe old method does not allow you to pass arguments to configure, at least in its simplest form(there’s a way to do it which we’ll describe below). The old configuration method works as follows:

% cd templates # optional % make site-init # invokes the configure script, among other things % cd .. # if you did cd templates % make

After it runs configure, make site-init will exit to allow you to make your changes to templates/<arch>-<os>/config.make before system compilation.

Arguments to configure and contents of config.makeThe file config.make contains all the configuration information you will need. It is organizedinto two main sections, one for the core distribution and the other for MITRE extensions such asthe examples and programming language bindings. Each section is arranged into a series ofsteps. All the documentation for these steps is also found in config.make . All of the settingswhich are user-settable are also controllable using the configure script.

Core distribution

Stepconfig.make variables

configure arguments description

564

Configureyour csh environment

ARCHOS_PREFIX(none)

The Communicator system mirrorsthe compiler variables in some ofthe csh shell scripts it uses. Insome cases, the variable namesclash with the environment on theinstalled machine. By default, weenable a prefix for these variablesusing the ARCHOS_PREFIXvariable, which is GALAXY_ bydefault; this setting makes the cshvariables GALAXY_ARCH, GALAXY_OS, and GALAXY_ARCHOS. Change thisprefix if you prefer.

Configureyour utilities

CC, AR,MKDIR, CHMOD, etc.

(none)

Here, you have the opportunity tochange the values of CC, AR, MKDIR, CHMOD, etc. based on yourplatform. These settings should beright in almost all cases.

Configureyourcompiler options

CFLAGS_O,CFLAGS_P, CFLAGS_D, etc.

(none)

Here, you have the opportunity tomodify your CPPFLAGS, CFLAGS,etc. for your platform. Most ofthese settings should be correct.You may wish to change theoptimization flags for CFLAGS_O(normal compilation), CFLAGS_D(debug compilation) and CFLAGS_P (profile compilation).

Configureshared libraries

SHARED_LIBS --enable-shared

If you want to compile theCommunicator libraries as sharedlibraries, use the --enable-shared argument to configure or uncomment thisline. This will significantly reducethe size of your executables;however, the executables will thendepend on the presence of theshared libraries to run. Thelocation of the shared libraries isautomatically compiled in to theexecutables by the Makefile templates.

565

ConfigurePurify and Insure

INSURE, PURIFY

--with-insure=<insure_binary>, --with-purify=<purify_binary>

MIT’s Makefile template hasbuilt-in support for make purify andmake insure", which will createinstrumented executables forRational’s Purify tool andParasoft’s Insure tool. Thesesettings are disabled by default. Ifyou have either of these tools andwant to make use of the MITMakefile support, use the --with-insure or --with-purify arguments to configure or uncomment (andperhaps modify) the appropriate lines.

Choose yourHubscripting module

(none) (none)

By default, the distribution isconfigured to use the MIT Hubscripting module. Currently, theHub will not compile without atleast some scripting moduleprovided. You can probably writeyour own, but we haven’tdocumented that process yet. Ifyou comment out COMPILE_MIT inthe next step, the scripting modulewill still be compiled.

Enable contributions

COMPILE_MIT, COMPILE_MITRE

--enable-MIT-extensions, --enable-MITRE-extensions

MIT has contributed usefullibraries, and MITRE hascontributed examples, tools, etc.Both are compiled by default, ifpresent in the distribution. Youmay disable either or both by usingthe appropriate arguments to configure (e.g., --enable-MIT-extensions=noor --disable-MIT-extensions )or commenting out the COMPILE_MIT or COMPILE_MITRE variables.

566

MITRE extensions

If you have commented out COMPILE_MITRE, you may ignore this section.

Stepconfig.make variables

configure arguments description

Locateyour xterm program

XTERM--with-x, --with-xterm=<xterm_binary>

MITRE uses xterm for processmonitoring under some circumstances.

Locateyour Java installation

JDK_HOME --with-jdk=<dir_above_bin/javac>

MITRE uses Java for theCommunicator Java bindingsand Java examples. The MITREJava bindings require at leastJDK 1.2.

EnablePythonbindingsand examples

PYTHONBIN, PYINCLUDE

--with-python=<python_executable>, --with-pycppflags=<cmdline_args_to_cpp>

If you want to enable the MITREPython bindings, servers anddemos, use with --with-python argument to configure or uncomment PYTHONBIN and set it to theappropriate path. Communicatorrequires Python 1.5 or later.Your Python installation mustbe built with Tkinter (Python’sbindings for Tcl/Tk).

Note: As of 3.0, the Pythonbindings use C wrappers aroundthe core Galaxy Communicatorlibrary. In order to compilePython, you must have PYINCLUDE or the --with-pycppflagsargument to configure set tothe include flags which providethe location of the Pythonheader files (typically,something like -I/usr/include/python1.5 parallel ). On Linux, you mayhave to install the python-devel RPM.

EnableAllegro CLbindingsand examples

ALLEGROBIN --with-allegro=<allegro_executable>

If you want to enable the MITREAllegro Common Lisp bindings,servers and demos, use the --with-allegro argument to configure or uncomment ALLEGROBIN and set it to theappropriate path. Communicatorrequires Allegro 5.0 or later.

567

Storing and reusing your configuration informationBy default, when you run the configure script and you provide arguments, those argumentsare cached in the file templates/<arch>-<os>/config.args . The --with-cached-argsargument to the configure script will reuse these stored arguments (and also won’t overwrite config.args ). When you call configure via make (as, for instance, in the old configuration method), the --with-cached-args argument is always used, so if you run configure byhand once, the settings will be available to you no matter which configuration method you usesubsequently (in case you need or want to reconfigure, or the compilation script detects a needto run the configuration script again).

At MITRE, we cache our config.make and config.args files for each platform and insertthem instead of running either configuration process. If you do this, you can use make partial-site-init in the templates subdirectory to generate the remainder of the site fileswithout overwriting config.make or config.args .

CompilationTo compile the entire distribution:

% cd $ROOT_DIR (the top of the distribution tree)% make

make will automatically stop to allow editing of newly-created site configuration files. This willcompile the Hub and support libraries, all the MIT servers, and all the MITRE examples, servers,demos and tools.

Each directory supports the following targets:

opt: the default target, makes optimized executables. Object files go in OBJ/<archos>/O,executables and libraries have no suffix. debug: makes executables with -g suitable for using gdb with. Object files go inOBJ/<archos>/D, executables and libraries have the _debug suffix. profile: makes executables suitable for use with gprof. Object files go in OBJ/<archos>/P,executables and libraries have the _profile suffix. insure: makes executables with the Insure++ tool. In order for this target to work, you musthave uncommented the INSURE settings in config.make and customized themappropriately. Object files go in OBJ/<archos>/I, executabes and libraries have the _insure suffix. purify: makes executables with the Purify tool. This target only works for executables, sincePurify isn’t needed for libraries. In order for this target to work, you must have uncommentedthe PURIFY settings in config.make and customized them appropriately. Purify relies on the"debug" target for its libraries and object files. Executables have the _purify suffix.

568

There are also five thread-specific targets, one for each non-threaded target:

thread: corresponds to opt thread-debug: corresponds to debug thread-profile: corresponds to profile thread-insure: corresponds to insure thread-purify: corresponds to purify

If threads are enabled in a Makefile, these targets will create threaded versions of the library orexecutable. If threads are not enabled, the targets will default to the non-threadedcorrespondents, in order to keep the recursive compilation from breaking.

Additional toplevel targets are listed below:

init: Make sure site-init, links, and Makefile dependencies have been made. depend: generate source file dependencies for all makefiles libs/libs-debug: make opt/debug GALAXY libraries (libGalaxy) hub/hub-debug: make opt/debug GALAXY binaries (verify_program, hub) contrib/contrib-MIT/contrib-MITRE: make the appropriate contrib directories clean: remove object files for the current architecture and operating system veryclean: clean plus

remove libraries and binaries for the current architecture and operating system distclean: veryclean plus

remove all OBJ directories remove automatically generated scripts and links remove top-level lib and bin directories and their contents remove backup files automatically generated by emacs and CVS clear automatically generated dependencies from all makefiles

InstallationAs we noted, there’s no make install for the Galaxy Communicator infrastructure. Once youcompile, you’re done.

Note : In some ways this is convenient, but in other ways in can be an obstacle. This is becausein some cases, it’s necessary to compile absolute paths into executables. For example, if wewant to avoid environment variables like GC_HOME, we must encode the location of the compileddistribution into shell scripts which run Python, Allegro or Java servers, so that these serversknow where to find the various Galaxy Communicator code libraries which implement the GalaxyCommunicator infrastructure for that language. Similarly, if we want to avoid the LD_LIBRARY_PATH environment variable for shared libraries, it’s necessary to compile thelocation of the shared libraries into Galaxy Communicator C executables.

569

What this means is that in many cases, these executables can’t be moved. Any installationprocess would have to address these issues; it’s straightforward to see how to address it for theshell scripts, but it’s something of a mystery how to address it for the C executables short ofrelinking them. And even then, the resulting installation also couldn’t be moved. Therefore, we’vechosen not to worry about the complexities of make install.

Cross compilationBecause of the changes to the configuration process in version 3.3, we’ve had to makesignificant changes to the experimental Unix support for cross-compilation, introduced in version 3.1.

This support is now enabled via an environment variable visible to configure and make called GC_CROSS. This variable is the name of a /bin/sh file which contains settings for the hostvariable (which should be the <arch>-<os> value for the platform being compiled for), for anyvariables in config.make which must be specified for the target platform and can’t be derivedfrom the environment (such as CC), and for some special cross-compilation variables whosevalues couldn’t be derived without executing code. The value of GC_CROSS must be an absolutepathname; configure will refuse to run otherwise.

For instance, for compiling for the StrongARM processor running Linux, we provide the followingsettings. This sample configuration can be found in templates/cross_compilation/armv4l-linux-config.sh .

# This is a /bin/sh file which sets variable values. # It is sourced in two places.

# It must always have a value for host. This value should # be a value for ARCH-OS.

host=armv4l-linux

# We need to set the following tools: CC, CXX, AR.

CC=/usr/local/arm-linux/bin/arm-linux-gcc CXX=/usr/local/arm-linux/bin/arm-linux-g++ AR="/usr/local/arm-linux/bin/arm-linux-ar crsv"

# These variables are special cross-compilation variables whichcould not # be derived without executing code.

ac_cc_sizeof_char=1 ac_cc_sizeof_short=2 ac_cc_sizeof_int=4 ac_cc_sizeof_long=4 ac_cc_sizeof_long_long=8

570

ac_cc_sizeof_float=4 ac_cc_sizeof_double=8 ac_cc_c_bigendian=no

The special cross-compilation variables shown here, like host , are required; if configureconcludes that it’s cross-compiling and it doesn’t find these values, it will refuse to run. As aconvenience, we’ve provided a C file, templates/cross_compilation/configure_values.c , which will compute and print outthese settings. Just run your cross-compiler on that file, run the resulting executable on thetarget platform, and paste the output directly into your GC_CROSS file. You only need to do this once.

If the value of GC_CROSS is visible to configure , it will create the appropriate subdirectory in templates to hold the configuration information. After configure runs, you’ll have the usualopportunity to edit config.make . For the StrongARM, we edit templates/armv4l-linux/config.make to set the STRIP makefile variable to arm-linux-strip to reduce the size of the executables.

When you compile after configuring, make sure that the value of GC_CROSS is visible to make. Ifit is, make will use the correct config.make and create and use the appropriate set of subdirectories.


571

New Features in the Current Distribution

Version 3.3The major purpose of this distribution is to clean up the installation process and move serversover to the new open source toolkit.

Improved installation processWe have moved the entire configuration process over to GNU configure, a migration whichbegan in version 2.0. As an added benefit, we’ve been able to compile and test the infrastructureeasily on three additional platforms: SGI IRIX, Intel FreeBSD, and MacOS X/Darwin.

Hub continuationsWe have introduced Hub continuations, which allow the programmer capture new messagesfrom servers and treat them as replies to previous messages, in order to reconcile conflicting(asynchronous vs. synchronous) programming models.

Open Source ToolkitVersion 3.3 marks the debut of the Open Source Toolkit (OSTK), which we hope will become amust-have collection of servers and wrappers for dialogue system development. We haveinitially populated the Open Source Toolkit with our own servers and wrappers, including somepreviously unreleased software. The OSTK is available from http://communicator.sourceforge.net.

Experimental Hub visualization supportIn this version, we’re releasing experimental work in progress to support message streams whichreport Hub state to separate servers, for visualization or other monitoring. See the -guicommand line argument of the Hub executable. A corresponding experimental Hub visualizationtool can be found in the Open Source Toolkit.

572


Version 3.2The major purpose of this distribution is to add a new, improved tutorial to the reorganized documentation.

New tutorialIn the past, MITRE has held training courses on the Galaxy Communicator architecture, andprovided a self-guided version of the training materials for download. These materials had fallenout of date, and were no longer being distributed. In this new version, we’ve completely updatedthe tutorial, integrated it into the main documentation, and redesigned it to be more consistent,easier to run, and easier to understand.

Improved unit testerIn support of the new tutorial, the unit tester has been completely redesigned andreimplemented. It now allows you to connect to a Hub pretending to be a server, or to a serverpretending to be a Hub. It also provides a graphical interface to allow you to respond to incomingmessages and to send new messages of your own, and keeps a mouseable history of frames foryou to select from if you choose.

Version 3.1This version is mostly a bug fix distribution, but there is also some new functionality provided.For details, see the release notes.

New Builtin server functionsThe Builtin server now has functions to list the available servers and to call subprograms so thatthey return values.

Cross-compilation supportThis version includes experiment support for cross-compiling executables for other platforms,such as the StrongARM.

Embedding supportThis version provides a new abstraction for the event-driven programming model used to embedCommunicator servers in external main loops.

573

Hub scripting improvementsThe Hub scripting language has a new directive (IGNORE:), a new comparison (IN), and betterwrite-locking for the global namespace.

Version 3.0

Session managementIn versions 2.1 and previous, there was poor support for session management. For instance, ifyou wanted to send a new message to the Hub in the current session, you were required to addthe session ID to the outgoing message by hand. This requirement was even more onerous inthe case of sending messages from timed tasks and broker callbacks. Also, the Hub createdsessions for new session IDs in an odd, inconsistent set of circumstances. These problems havebeen fixed. The Hub now always creates a session if it doesn’t know about it, and we’veimplemented far more robust session management support.

ContinuationsIn versions 2.1 and previous, once the Hub sent a message to a server, the server was busy, asfar as that Hub was concerned, and it could send no more messages to it until the serverresponded. This sometimes led to deadlock situations. In version 3.0, you can use continuationsto tell the Hub that you’ll send an answer later.

Server location filesIn versions 2.1 and previous, it was hard to reassign ports and host locations for a configurationof servers. The -locations command line Hub argument helped, as did the -port and -contact_hub command line server arguments. However, there was no way to change whocontacts whom without editing the program file, and the command line arguments were quiteunwieldy. In 3.0, you can define a server locations file which the Hub and all servers can refer tovia the -server_locations_file command line argument, which specifies the location of alllisteners and who contacts whom. These settings override the Hub program file, so no programfile needs to be edited.

GC_VERSION declarationBefore version 2.0, there was no support for distinguishing between Communicator releases.This was a potential problem in source code which needed to be compiled against multiplereleases. The GC_VERSION C declaration was introduced in version 2.0, but it was notdocumented, and there was no corresponding support for Python, Allegro and Java. In 3.0,we’ve documented this feature and made it available in Allegro (compile/read time feature) andPython (run-time variable). Java is a compiled language which has no compile-timeconditionalizations that we can find, so this feature is unfortunately unavailable in Java.

574

Multiple events in C timed tasksIn versions 2.1 and previous, the implementation of timed tasks in C did not allow you toassociate a single task with multiple events (input available, error input available, etc.). In version3.0, the functions Gal_AddTaskExtended and Gal_TaskPkgRunReasons support thisfunctionality correctly.

C frame library improvementsVersion 3.0 features three major improvements to the frame library. First, you can now createdynamically expandable lists and binary data objects (see the functions Gal_CreateListObjectand Gal_CreateBinaryObject, for example). Second, we’ve defined a parallel API which givesyou better memory management control over objects (see the Gal_Create... family of functions).Finally, there are no longer any restrictions on the types of objects you can insert into frames; allarray types can now be sent, in addition to binary data (see the function Gal_CreateInt16Object,for instance).

Improved event-driven programming modelIn version 2.1 and previous, there were functions which allowed you to register "callback"behavior (for instance, when a broker was destroyed), but there was no consistent event-drivenmodel underlying these functions, and the implementation was quite incomplete. In version 3.0,we’ve defined an event-driven model which we use for the default Communicator main loop, forevent registration in external main loops, and for embedding the C library in the Python andAllegro bindings. See the upgrade notes for more details.

Improvements in the Hub syntax and conceptual modelIn a number of circumstances, it’s important to be able to exert control over how and whenparticular servers are used by a given Hub. In order to support this degree of control, we’vereorganized the Hub model of its resources. The Hub now groups servers (known to the Hub as "service providers") into equivalence classes of service types. These providers can haveproperties associated with them, and the Hub programmer can impose conditions on the choiceof provider, based on the service provider’s properties and the properties of the current sessionand token.

These improvements are accompanied by a new extended syntax mode, which broadens therange of expressive possibilities and eliminates some of the idiosyncracies of the default Hubsyntax. The primary improvement in extended syntax mode is the ability to refer transparently tosession variables, global Hub state, server properties, etc.

Finally, the Hub now has greater debugging and error-handling support, including expandedfunctionality for the -debug Hub command line argument, introduction of the ERROR: Hubprogram directive, and the new Builtin server dispatch functions hub_break, hub_exit and hub_print.

575

Brokering improvementsIn version 2.1 and previous, the set of types which could be brokered and the set of types whichcould be included in frames overlapped, but were not identical. In 3.0, thanks to the new unifiedXDR encoding, there is finally no distinction between these sets. Also, outgoing brokers nowaccept multiple subscribers by default.

Protocol improvementsThe interaction between Hub and servers has been made more robust in a number of ways.First, the new XDR encoding has eliminated a number of nagging bugs, such as the inability tosend arrays of floats from one platform to another. Second, both Hub and server indicate in theirmessages whether a response is expected or not, and the Hub and bindings either eliminateresponses or add pacifier responses to guarantee that the expectations are fulfilled. As a result,the Hub will no longer deadlock waiting for a response from a server which isn’t configured toprovide one. Third, the infrastructure-specific data is now encapsulated in a single frame key,which will make upward compability much simpler in the future. And finally, the initial protocolhandshake now contains a version number.


576

Upgrading from 3.2 to 3.3

Upgrading to Galaxy Communicator 3.3 is simple. There are only two upgrades, and the firstaffects only a small number of users.

This document describes only the steps required or recommended to upgrade existing featuresof Galaxy Communicator. You can find a list of new features here. You can find the completerelease notes here.

Step Who’s affected Status

Step 1: Updatingcross-compilation support

All users of cross-compilation Obligatory

Step 2: Upgrading to the OpenSource Toolkit

All users of servers or wrappers previouslyfound in contrib/MITRE/servers

Obligatory

Step 3: Upgrading flags whichset Java Hub client behavior

All users of Java Hub clients Obligatory

Step 1: Updating cross-compilation supportThe new cross-compilation support requires any users of this facility to change their approachcompletely. Previously, the value of the GC_CROSS environment variable was a directory whichcontained files which allowed the user to describe the system architecture and operating systemof the target platform. This information is now contained directly in a single /bin/sh file which isthe value of GC_CROSS. Because this functionality was previously experimental, we will notprovide step-by-step upgrade instructions; see the documentation for a detailed example.

Step 2: Upgrading to the Open Source ToolkitIf you are using any of the servers in contrib/MITRE/servers , be advised that they havebeen removed from the distribution in order to form the initial basis of the Open Source Toolkit,available from http://communicator.sourceforge.net.

577


Step 3: Upgrading flags which set Java Hub client behaviorThe connection policy flags for Java Hub clients has been updated to coincide with support inthe other language bindings. See the Java notes for details.


578


Upgrading to Galaxy Communicator 3.2 is simple. For almost all users, there are no obligatory upgrades.



Step 1: Updating session management inthe Hub

All users of session locks andmultiple sessions

Obligatory

Step 2: Upgrading flags which set Hubclient behavior

All users of Hub clients (not for Java yet)

Obligatory

Step 1: Updating session management in the HubThe Hub chooses service providers based on a number of considerations: whether they supportthe requested operation, whether they’re free, whether they’re up, and what sessions they maybe restricted to supporting. In version 3.1, being the only provider for a particular serviceoutweighed the session the provider was restricted to supporting, so that if there was, say, onlyone audio server, and it was locked to session 1, programs running tokens in session 2 wouldstill be allowed to select that server, because it was the only one. This behavior was incorrect,and it has been fixed.

However, if you have been experimenting with session locks and multiple sessions, it’s possiblethat at this point your system might stop working, because you’ve unwittingly relied on this bug in3.1. So if in 3.2, your system informs you that there are no providers available when you believethat there are, you should consider the possibility that you’ve misused session locks.

579

Step 2: Upgrading flags which set Hub client behaviorWhen you set up a Hub client programmatically, it’s possible to define the behavior of the Hubclient when it cannot establish a connection, or when it loses one. As of version 3.2, it’s possibleto specify that the process should shut down if it can’t establish a connection, where before itwas not possible to do so. It’s also possible to check the conditions more mnemonically thanbefore, because new masks and constants have been added. In some cases, because thevalues of the constants have changed, this change is obligatory.

For instance, if you want to see if the server was set up to retry, you would previously have needto do this:

Version 3.1

if (!(GalIO_ServerListenStatus(scomm) &GAL_HUB_CLIENT_CONNECT_FAILURE_NOOP)) { ... }

This will now do the wrong thing. The proper (and more mnemonic) test is

Version 3.2

if ((GalIO_ServerListenStatus(scomm) &GAL_HUB_CLIENT_CONNECT_FAILURE_MASK) == GAL_HUB_CLIENT_CONNECT_FAILURE_RETRY) { ... }

See the documentation on GalIO_ServerListenStatus for more details.


580


Upgrading to Galaxy Communicator 3.1 is simple. In fact, except for one minor change to Javaservers, there are no obligatory upgrades.



Step 1: Configuring your system All users Optional

Step 2: Updating externalmainloop references

All users of the C API who have embeddedCommunicator in an external main loop

Optional

Step 3: UpgradingGAL_VERBOSE references

All users Optional

Step 4: Updating Java bindings Users of the Java bindings Required

Step 1: Configuring your systemPerl is no longer required for configuration or compilation. The appropriate changes have beenmade in the file templates/config.make.in . If you have a configuration file you’ve storedsomewhere which you use during compilation, you may wish to update that configuration file tobe consistent with the template.

Step 2: Updating external mainloop referencesIn version 3.0, it was necessary to provide extensive, detailed callback code to embed theCommunicator infrastructure in an external main loop. After considerable experience with thisprocess, we have been able to abstract and simplify it. If you’ve already gone through theheadache of making your external embedding 3.0-compliant, you may not want to go back andmake yet another upgrade; however, the new abstraction deals with the vast majority of the

581

details you’d otherwise need to maintain yourself, and it may be valuable to you to use it. The3.0 API underlies this new abstraction, and will continue to be supported.

Step 3: Upgrading GAL_VERBOSE referencesIn previous versions of Galaxy Communicator, the default value for GAL_VERBOSE wasundefined, which translated into the largest amount of data being printed. We have changed thedefault value of GAL_VERBOSE to be 3, which prints out a reasonable, but not overwhelming,amount of information. Many of the tools distributed with the Galaxy Communicator infrastructurealready impose this value. In the unlikely case that you run C, Allegro or Python servers insituations which assume that GAL_VERBOSE is not set (and, therefore, that the serversmaximally verbose), you’ll have to explicitly set GAL_VERBOSE to 0.

Step 4: Updating Java bindingsThe method MainServer.contactHub has been modified. It no longer takes a Server argument.Rather, if the Hub contact policy argument is set to -1, the default policy of the invoked MainServer is used.


582


The 3.0 Galaxy Communicator distribution contains major improvements to outgoing brokering.There are obligatory upgrades associated with these improvements; in particular, APIs haveunavoidably changed. In almost all other ways, the 3.0 upgrade should be transparent for Cprogrammers; except for outgoing brokering, the programmer will probably not have exploitedthe functionality which needs to be changed. For Java programmers, a significant internalreorganization of the Java bindings was required to bring its functionality in line with the Clibrary, and there are a number of unavoidable API changes which are required. For Python andAllegro programmers, the changes are much more modest, but in many cases still necessary.



Step 1: Configuring your system

Users of the Python and Allegrobindings, users of the MITREPostGres DB wrapper server

Required

Step 2a: Upgradingoutgoing brokering

All server developers Required

Step 2b: Upgradingmemory management inbroker callbacks

Users of incoming brokering in thelibGalaxy C library

Optional, but highlyrecommended for efficient performance

Step 2c: Upgradingincoming brokering

Users of incoming brokering in thelibGalaxy C library

Optional

Step 2d: Using brokercallbacks for completionand aborting

Users of brokering Optional

Step 3: UpgradingPython bindings

Users of the Python bindings Required

583

Step 4: Upgrading Java bindings

Users of the Java bindings Required

Step 5: Upgrading serverlisten status

Users of the server listen statusfunctionality in the libGalaxy Clibrary (introduced in 2.1)

Required

Step 6: Upgradingreferences to the call environment

Users of the libGalaxy C library whoinadvisedly exploited the odd returnproperties in 2.1 of GalSS_EnvErrorand GalSS_EnvDestroyToken

Required

Step 7: Upgrading callsto GalIO_SetServerData

Users of the libGalaxy C library whouse either use GalIO_SetServerData to storearbitrary data in the server, orreturned values from _GalSS_init_server for storage

Required(GalIO_SetServerData),recommended (_GalSS_init_server)

Step 8a: Upgradingreferences to theCommunicator main loop

Users of the libGalaxy C library whohave written servers with their own main()

Optional

Step 8b: Upgrade yourown main loop

Users of the libGalaxy C library whohave written servers with their ownevent loops (embedded in Tcl/Tk orCORBA, for instance)

Optional but highly recommended

Step 9: Upgrade timedtask invocation

Users of the libGalaxy C library whoset up their own timed tasks

Optional

Step 10: Upgradingsession management

Users of the libGalaxy C library

Optional but highlyrecommended, especially forthose who intend to runmultiple parallel sessions

Even though a number of these steps are optional, we strongly encourage you to consider them.The functionality they replace has not been deprecated, but in many cases, future upgrades willnot extend the older functionality. In addition, steps 8-10 feature radically improved strategies forembedding Communicator servers in external main loops, and for session handling; while theseupgrades are optional, these improvements are so dramatic that we strongly recommend thateveryone adopt them.

584

Step 1: Configuring your systemThe only obligatory change in configuring your system comes in the section where Python isenabled. Because we’re now implementing the Python bindings on top of the core C library, thePython bindings require the Python header files. In order to compile Python, you must havePYINCLUDE in templates/<ARCHOS>/config.make set to the location of Python.h (typically include/python1.5 parallel to bin/ ). On Linux, you may have to install the python-devel RPM.

In addition, the 3.0 Galaxy Communicator distribution requires Allegro 5.0.

Finally, we have added the PostGres DB server to the default compilation. If you want to enablethis server, uncomment the POSTGRES_INCLUDE and POSTGRES_LIB variables in yourconfig file and set them to the appropriate values.

Step 2a: Updating outgoing brokeringBrokering has been improved in two major ways. First, outgoing brokers now share a listenerport with the main server, which reduces the number of open ports and also fixes a bug involvingport reuse inside processes. Second, outgoing brokers can now provide data to an arbitrarynumber of broker clients. This second improvement now makes timeout management ofoutgoing brokers considerably more important, since outgoing brokers now cache their data, andmemory bloat can result if they’re not destroyed promptly.

Here’s the old way of doing outgoing brokering:

static Gal_Frame prepare_audio_frame(char *filename) { Gal_Frame f = Gal_MakeFrame("main", GAL_CLAUSE); int total; char *buf; GalIO_BrokerStruct *b; char host_pid[1024];


/* .... */

/* Now that we have the audio, we add a binary element. */ /* These must be added to the broker frame BEFORE the broker object is initialized. */ sprintf(host_pid, "%s:%d", GalIO_IPAddress(), (int) getpid()); GalIO_FrameSetBrokerCallID(f, Gal_StringObject(host_pid)); b = GalIO_BrokerDataOutInit(3900, f, 0, 0); if (b && (GalIO_FrameGetBrokerPort(f) > 0)) {

585

Gal_SetProp(f, ":binary_host", Gal_StringObject(GalIO_IPAddress())); Gal_SetProp(f, ":binary_port", Gal_IntObject(GalIO_FrameGetBrokerPort(f))); GalIO_BrokerWriteBinary(b, buf, total); GalIO_BrokerDataOutDone(b); } return f; }

Here’s the new way

static Gal_Frame prepare_audio_frame(GalIO_CommStruct *gcomm, char *filename) { Gal_Frame f = Gal_MakeFrame("main", GAL_CLAUSE); int total; char *buf; GalIO_BrokerStruct *b; char host_pid[1024];


/* .... */

/* Now that we have the audio, we write the binary data through the broker. */

b = GalIO_BrokerDataOutInit(gcomm, 0, 10); if (b && (GalIO_GetBrokerListenPort(b) > 0)) { GalIO_BrokerPopulateFrame(b, f, ":binary_host", ":binary_port"); GalIO_BrokerWriteBinary(b, buf, total); GalIO_BrokerDataOutDone(b); } free(buf); return f; }

Note the following contrasts:

It is no longer necessary to create the unique call ID; the broker does this for you. GalIO_BrokerDataOutInit no longer takes a port number as its first argument; instead, ittakes a connection object, through which it can associate itself with an existing serverlistener (or start up a listener for the server). The third argument to GalIO_BrokerDataOutInit is still a timeout, but it is much moreimportant to set this timeout appropriately, because the semantics of the timeout havechanged. Previously, when brokers accepted only a single connection, the broker would

586

destroy itself after it received and satisfied that single connection, and the timeout was onlyactivated if no connection was ever received. Now, since brokers can accept multipleconnections, the broker will not be destroyed until the timeout is reached and all currentlyconnected broker clients have received all their data. A broker which never expires will never befreed, and all the data it is intended to send will also remain unfreed, since the broker mustcache its data for subsequent connections. The host and port for the broker connection are now inserted by GalIO_BrokerPopulateFrame.

All these changes are mandatory. Before 3.0, setting the broker properties (host, port, call ID)exhibited some complicated and unnecessary ordering dependencies which have now beeneliminated; in addition, the outgoing broker didn’t need the connection, because it created itsown server listener.

Corresponding modifications are required in Allegro, Java and Python.

Step 2b: Upgrading memory management in broker callbacksAs part of general memory management cleanup, we have observed that before version 3.0, wehad failed to document appropriately the fact that data passed to broker handlers used by GalIO_CommBrokerDataInInit is not freed by the Galaxy Communicator library. Therefore, allbroker handlers should take responsibility for freeing the data they’re passed. We made thismistake in the 2.1 version of one of our audio examples; here’s the comparison:

Version 2.1

static void audio_handler(GalIO_BrokerStruct *broker_struct, void *data, Gal_ObjectType data_type, int n_samples) { DataHandler *d = (DataHandler *) GalIO_GetBrokerCallerData(broker_struct);

switch (data_type) { case GAL_BINARY: if (d->data_buf) d->data_buf = (char *) realloc(d->data_buf, n_samples + d->size); else d->data_buf = (char *) malloc(n_samples + d->size); bcopy(data, d->data_buf + d->size, n_samples); d->size += n_samples; break; default: GalUtil_Warn("Unknown data type %s\n",

587

Gal_ObjectTypeString(data_type)); } }

Version 3.0

static void audio_handler(GalIO_BrokerStruct *broker_struct, void *data, Gal_ObjectType data_type, int n_samples) { DataHandler *d = (DataHandler *) GalIO_GetBrokerCallerData(broker_struct);


Step 2c: Upgrading incoming brokering (optional)As part of cleaning up our data storage and callback models, we’ve made some modifications tothe preferred way that incoming brokers are set up. First, to support the new event-drivenprogramming paradigm more reliably, we recommend that everyone change their incomingbroker setup function from GalIO_BrokerDataInInit to GalIO_CommBrokerDataInInit, as follows:

Version 2.x


588

d->data_buf = (char *) NULL; d->size = 0;

if (host && port) { b = GalIO_BrokerDataInInit(host, port, f, audio_handler, d, 0); if (b) { GalIO_SetBrokerActive(b); } } else { free(d); } return (Gal_Frame) NULL; }

Version 3.0


d->data_buf = (char *) NULL; d->size = 0; d->gcomm = GalSS_EnvComm((GalSS_Environment *) server_data);

if (host && port) { b = GalIO_CommBrokerDataInInit(d->gcomm, host, port, f, audio_handler, 0, d, __FreeDataHandler); if (b) { GalIO_SetBrokerActive(b); } } else { free(d); } return (Gal_Frame) NULL; }

Actually, our best recommendation is to use GalSS_EnvBrokerDataInInit, as shown in this upgrade step, but if you choose not to, we recommend the upgrade here.

Second, we now discourage the user of GalIO_BrokerSetFinalizer, which has been supersededby the event-driven programming model: Note that this example also illustrates the upgrade toGalSS_EnvBrokerDataInInit, but we won’t highlight that contrast here:

589

Version 2.x

void __AudioSupport_RetrieveAudio(char *host, int port, int sample_rate, Gal_Frame f, GalIO_BrokerDataFinalizer finalizer, void *callback_data) { GalIO_BrokerStruct *b;

if (host && port && (sample_rate > 0)) { DataContainer *c = (DataContainer *) calloc(1, sizeof(DataContainer)); c->data_buf = (char *) NULL; c->size = 0; c->callback_data = callback_data; b = GalIO_BrokerDataInInit(host, port, f, __AudioCallback, c, 0);

if (b) {

/* Set the broker finalizer. */ GalIO_BrokerSetFinalizer(b, finalizer);

/* Make the broker active. Without this step, nothing will happen. This utility can arguably be used to manage multiple incoming audio signals, but it’s probably inadequate. */ GalIO_SetBrokerActive(b); } } }

Version 3.0

void __AudioSupport_RetrieveAudio(GalSS_Environment *env, char *host, int port, int sample_rate, Gal_Frame f, GalIO_BrokerCallbackFn finalizer, void *callback_data) { GalIO_BrokerStruct *b;

if (host && port && (sample_rate > 0)) { DataContainer *c = (DataContainer *) calloc(1, sizeof(DataContainer));

590

c->data_buf = (char *) NULL; c->size = 0; c->callback_data = callback_data; b = GalSS_EnvBrokerDataInInit(env, host, port, f, __AudioCallback, 0, (void *) c, NULL);

if (b) {

GalIO_AddBrokerCallback(b, GAL_BROKER_DATA_DONE_EVENT, finalizer, (void *) NULL);

/* Make the broker active. Without this step, nothing will happen. This utility can arguably be used to manage multiple incoming audio signals, but it’s probably inadequate. */ GalIO_SetBrokerActive(b); } } }

The special finalizer has been replaced by a more general callback, and in this case the eventthe callback is associated with is when the broker is done with its data.

Step 2d: Using broker callbacks for completion and abortingIn version 2.1 and previous, there was no way of marking completion of a broker connection or atermination before completion without sending string semaphores through the broker connection.In version 3.0, you can use broker callbacks to implement this behavior.

With semaphores

#define AUDIO_END "end" #define AUDIO_ABORT "abort"

static void audio_handler(GalIO_BrokerStruct *broker_struct, void *data, Gal_ObjectType data_type, int n_samples) { DataHandler *d = (DataHandler *) GalIO_GetBrokerData(broker_struct);

591

switch (data_type) { case GAL_STRING: if (!strcmp((char *) data, AUDIO_END)) { GalIO_BrokerDataDone(broker_struct); __report_done(broker_struct, (void *) NULL); } else if (!strcmp((char *) data, AUDIO_ABORT) { __report_abort(broker_struct, (void *) NULL); } else { GalUtil_Warn("Unknown message %s\n", (char *) data); } free(data); break; case GAL_BINARY: if (d->data_buf) d->data_buf = (char *) realloc(d->data_buf, n_samples + d->size); else d->data_buf = (char *) malloc(n_samples + d->size); bcopy(data, d->data_buf + d->size, n_samples); d->size += n_samples; free(data); break; default: GalUtil_Warn("Unknown data type %s\n", Gal_ObjectTypeString(data_type)); } }

void __FreeDataHandler(void *data) { DataHandler *d = (DataHandler *) data; free(d->data_buf); free(d); }

void __report_abort(GalIO_BrokerStruct *b, void *data) { DataHandler *d = (DataHandler *) GalIO_GetBrokerData(b); Gal_Frame f;

f = Gal_MakeFrame("notify", GAL_CLAUSE); Gal_SetProp(f, ":notification", Gal_StringObject("Audio aborted.")); GalIO_CommWriteFrame(d->gcomm, f, 0); Gal_FreeFrame(f); }

592

void __report_done(GalIO_BrokerStruct *b, void *data) { DataHandler *d = (DataHandler *) GalIO_GetBrokerData(b); Gal_Frame f; FILE *fp = fopen("/dev/audio", "w");

if (!fp) { GalUtil_Warn("Couldn’t open /dev/audio"); } else { fwrite(d->data_buf, sizeof(char), d->size, fp); fflush(fp); fclose(fp); }

f = Gal_MakeFrame("notify", GAL_CLAUSE); Gal_SetProp(f, ":notification", Gal_StringObject("Audio received.")); GalIO_CommWriteFrame(d->gcomm, f, 0); Gal_FreeFrame(f); }




593

Without semaphores

static void audio_handler(GalIO_BrokerStruct *broker_struct, void *data, Gal_ObjectType data_type, int n_samples) { DataHandler *d = (DataHandler *) GalIO_GetBrokerData(broker_struct);


void __FreeDataHandler(void *data) { DataHandler *d = (DataHandler *) data; free(d->data_buf); free(d); }

void __report_abort(GalIO_BrokerStruct *b, void *data) { DataHandler *d = (DataHandler *) GalIO_GetBrokerData(b); Gal_Frame f;

f = Gal_MakeFrame("notify", GAL_CLAUSE); Gal_SetProp(f, ":notification", Gal_StringObject("Audio aborted.")); GalIO_CommWriteFrame(d->gcomm, f, 0); Gal_FreeFrame(f); }

594

void __report_done(GalIO_BrokerStruct *b, void *data) { DataHandler *d = (DataHandler *) GalIO_GetBrokerData(b); Gal_Frame f; FILE *fp = fopen("/dev/audio", "w");

if (!fp) { GalUtil_Warn("Couldn’t open /dev/audio"); } else { fwrite(d->data_buf, sizeof(char), d->size, fp); fflush(fp); fclose(fp); }

f = Gal_MakeFrame("notify", GAL_CLAUSE); Gal_SetProp(f, ":notification", Gal_StringObject("Audio received.")); GalIO_CommWriteFrame(d->gcomm, f, 0); Gal_FreeFrame(f); }



if (host && port) { b = GalIO_CommBrokerDataInInit(d->gcomm, host, port, f, audio_handler, 0, d, __FreeDataHandler); if (b) { GalIO_SetBrokerActive(b); GalIO_AddBrokerCallback(b, GAL_BROKER_ABORT_EVENT, __report_abort, (void *) NULL); GalIO_AddBrokerCallback(b, GAL_BROKER_DATA_DONE_EVENT, __report_done, (void *) NULL); } } else { free(d); } return (Gal_Frame) NULL;

595

}

We believe that the version without semaphores is considerably more reliable than the versionwith semaphores; in addition to the sender and receiver not having to agree on the semaphoresthemselves, the detection of an incomplete connection covers all the cases where the sendingserver died unexpectedly and could not have sent an abort semaphore. We have upgraded allMITRE examples to use the callback-driven approach.

Step 3: Upgrading Python bindingsIf you write Python servers, you’ll need to make a few tiny modifications for 3.0.

First, because Python now uses the C library, a number of the elements you could set andaccess via instance attributes now require you to call a method. In particular:

the port attribute of the GalaxyIO.Server class is now the ListenPort() method the validate attribute of the GalaxyIO.Server class is now the Validating() and EnableValidation() methods the server_name attribute of the GalaxyIO.Server class is now the ServerName() method the max_connections attribute of the GalaxyIO.Server class is now the MaxConnections() method

So for example, you should now set the maximum number of connections as follows:

Version 2.x

s.max_connections = 5

Version 3.0

s.MaxConnections(5)

See the source code for more details.

Also, because Python now uses the C library:

The method ReadFrameFromString() of the Galaxy.Frame class has been removed. The Galaxy.OPr() function no longer accepts the indent keyword.

Next, with an eye on simplifying server startup under Windows NT, we’ve revised the "preamble"which tells the server how to locate the Python libraries:

Version 2.x

596

import os, sys

sys.path.insert(0, os.environ["LIBDIR"])

import Galaxy, GalaxyIO

Version 3.0

import os, sys

sys.path.insert(0, os.path.join(os.environ["GC_HOME"], "contrib", "MITRE", "templates"))

import GC_py_init

import Galaxy, GalaxyIO

In this new method, the only thing the environment needs to provide is the identity ofGC_HOME; all the other path functionality, etc., is provided in the GC_py_init module.

Finally, the Python bindings now fully embrace the new call environments (see thedocumentation on adding a server and session management). The GalaxyIO.Connectionclass, which was "secretly" a call environment in 2.x, has now been renamed GalaxyIO.CallEnvironment. The impact of this change is minimal in non-class-basedservers. We use a simplified version of the multiply server from the double example to illustrate:

Version 2.x

def Multiply( conn , dict): return {":int": dict[":int"] * Factor}

def Welcome( conn , dict): global Factor try: Factor = dict[":factor"] except: pass return None

def main(): s = GalaxyIO.Server(sys.argv, "multiply", default_port = 2900, conn_class = GalaxyIO.Connection ) s.AddDispatchFunction("multiply", Multiply) s.AddDispatchFunction("reinitialize", Welcome) s.RunServer()

main()

597

Version 3.0

def Multiply( env , dict): return {":int": dict[":int"] * Factor}

def Welcome( env , dict): global Factor try: Factor = dict[":factor"] except: pass return None

def main(): s = GalaxyIO.Server(sys.argv, "multiply", default_port = 2900) s.AddDispatchFunction("multiply", Multiply) s.AddDispatchFunction("reinitialize", Welcome) s.RunServer()

main()

For mnemonic reasons, we’ve changed the name of the first argument of dispatch functions from conn to env in this example; this of course has no functional import. However, observe also thatthe conn_class keyword to the creation method for GalaxyIO.Server has been removed,because it is no longer recognized, and in this case it was superfluous anyway. The realdifferences emerge when we compare class-based versions of the same example:

Version 2.x

class MultiplyConnection ( GalaxyIO.Connection ):

def Multiply(self, dict): return {":int": dict[":int"] * self.conn.Factor}

def Welcome(self, dict): try: self.conn.Factor = dict[":factor"] except: self.conn.Factor = 1 return None

def main(): s = GalaxyIO.Server(sys.argv, "multiply", default_port = 2900, conn_class = MultiplyConnection ) s.AddDispatchFunction("multiply", MultiplyConnection .Multiply) s.AddDispatchFunction("reinitialize", MultiplyConnection .Welcome) s.RunServer()

598

main()

Version 3.0

class MultiplyEnvironment ( GalaxyIO.CallEnvironment ):

def Multiply(self, dict): return {":int": dict[":int"] * self.conn.Factor}

def Welcome(self, dict): try: self.conn.Factor = dict[":factor"] except: self.conn.Factor = 1 return None

def main(): s = GalaxyIO.Server(sys.argv, "multiply", default_port = 2900, env_class = MultiplyEnvironment ) s.AddDispatchFunction("multiply", MultiplyEnvironment .Multiply) s.AddDispatchFunction("reinitialize", MultiplyEnvironment .Welcome) s.RunServer()

main()

Notice that because the call environment is an "ephemeral" object, permanent data (such as thevalue of Factor) must be stored on the underlying connection. This is true in both 2.x and 3.0, butit’s clearer in 3.0 why this is necessary. The conn_class keyword has been replaced by the env_class keyword, etc.

Step 4: Upgrading Java bindingsIf you write Java servers, you’ll need to make some modifications for 3.0.

Brokering

Java broker classes have been modified. First, the galaxy.server.DataOutBroker constructor has changed:

Version 2.x

public DataOutBroker(Server server, int port, GFrame frame) throws Exception

599

public DataOutBroker(Server server, int port, GFrame frame, booleanrport) throws Exception

Version 3.0

public DataOutBroker(Server server, int pollMilliseconds, inttimeoutSeconds) throws Exception

Java outgoing brokers now rely on the listener in the main server to handle broker clientconnection requests, so the broker no longer needs its own listener (i.e., port). Also, the GFrameis no longer needed (it provided the broker’s call ID) since DataOutBroker now generates itsown call ID (accessible via DataOutBroker.getCallId ). The new constructor’s argumentsinclude a reference to the galaxy.server.Server that started the broker, a polling time inmilliseconds, and a timeout in seconds. The polling time controls how often the broker checks fornew data to send to its clients (it also checks for new clients, to which it sends any cached data ithas). The timeout determines when (if ever) the broker stops accepting new client connections.Once a timeout expires, no new connections are accepted, and the broker will exit once it hasbeen instructed to disconnect from its clients (via DataOutBroker.close ) . The old constructors are still available but have been deprecated. They now only use the Serverparameter (default polling time and timeout values are used internally).

When sending messages from a server to a Hub, informing the Hub that brokered data isavailable, use DataOutBroker.populateFrame to add the broker contact informationautomatically to the frame.

The codes segments below illustrate how code for creating an outgoing broker and creating anotification frame differs between Galaxy Communicator 2.x and 3.0.

Version 2.x

public class TestAudio extends galaxy.server.Server { { ...

Clause fr = new Clause("main"); fr.setProperty(":call_id", this.getCallId()); DataOutBroker broker=null; try { broker = new DataOutBroker(this, 3900, fr); } catch (Exception ex) { logAndSendError("Error opening broker connection: " + ex.toString()); return; }

... // prepare brokered data

600

GFrame result = new Clause("main"); result.setProperty(":binary_port", broker.getPort()); result.setProperty(":call_id", this.getCallId()); result.setProperty(":binary_host", this.getHostAddress());

... // send the frame

Version 3.0

public class AudioOut extends galaxy.server.Server { ...

DataOutBroker broker; int pollMs = 50; int timeoutSecs = 50; try { broker = new DataOutBroker(this, pollMs, timeoutSecs); } catch (Exception e) { logAndSendError("Error opening broker connection: " + e.toString()); return; }

... // prepare brokered data

GFrame result = new Clause("main"); broker.populateFrame(result, ":binary_host", ":binary_port");

... // send the frame

The manner in which an incoming broker detects and processes the end of a brokered datastream has also changed. Subclasses of galaxy.server.DataInBroker should implementthe disconnectReceived method. This method is called automatically when the incomingbroker receives a disconnect message from the outgoing broker, indicating that no morebrokered data will be sent. Incoming brokers can also choose to implement abortReceived ,which is called when the broker receives an unexpected exception while receiving brokered data.

Version 2.x

// Outgoing broker code segment

DataOutBroker broker;

... // create broker

601

broker.write("start");

// write brokered data here

broker.write("end"); broker.close();

// Incoming broker code segment

public void receivedString(String str) { if (str.equals("start")) { System.out.println("About to start receiving data"); } else if (str.equals("end")) {

... // process data

Version 3.0

// Outgoing broker code segment

DataOutBroker broker;

... // create broker

// write brokered data here

broker.close();

Note that the outgoing broker no longer has to send explicit markers for the beginning and end ofthe brokered data.

// Incoming broker code segment

protected void disconnectReceived() { ... // process data

As a result of these and other changes to brokering, there are a number of incompatabilitiesbetween 2.x and 3.0 code:

The galaxy.server.DataInBroker thread is no longer started in the DataInBrokerconstructor. Instead, DataInBroker.start must be called explicitly (e.g., at the end of a DataInBroker subclass constructor). This is required in order to ensure that a DataInBroker does not attempt to retrieve and process any brokered data until all custom

602

initialization has taken place:

Version 3.0

public class AudioInBroker extends galaxy.server.DataInBroker { private GBinary bdata;

public AudioInBroker(Server server, InetAddress ip, int port,GFrame frame) throws Exception { super(server,ip,port,frame); bdata = new GBinary();

// You must explicitly call start() once your broker is initialized. // For example, you can call it at the end of theconstructor, as is // done here. start(); } . . .

In order to enable all Galaxy data types to be available via brokering, a DataInBrokermust implement four additional methods: receivedList , receivedInteger , receivedFloat , and receivedSymbol .

603

Since out brokers now rely on the main server listener to receive client connection requests,the following DataOutBroker methods have been removed:

setPort getPort startListening init flush stopListening

In addition, galaxy.server.Server.getCallID has been removed. Use DataOutBroker.getCallId instead.

Threads

Thread management has been redesigned in the Java bindings, especially with regards tostopping various threads. As a result, the following methods have been removed:

galaxy.server.DataInBroker.close (use DataInBroker.stop to stop the broker) galaxy.server.MainServer.prepareToStop (use MainServer.stop to stop themain server thread) galaxy.server.Server.start (this did not do anything in 2.1, and it is not needed in 3.0)

Also, Server.prepareStop is now private. Use Server.stop to stop the server. Finally,these MainServer methods have been removed and reimplemented elsewhere:

startListener (by default the listener thread, now encapsulated in galaxy.server.ListenerThread , is managed internally, but galaxy.server.MainServer.startListenerThread can be used to start the listenerif it is not already running) connectToHubs (should not impact Galaxy Communicator developers) createServer (should not impact Galaxy Communicator developers) startServer (should not impact Galaxy Communicator developers)

Command Line Arguments

The syntax of command line arguments has changed in the Java bindings. Equal signs (=) areno longer needed when assigning values (e.g., -port=123 is now -port 123). Also, double quotes(") are now needed when listing multiple values for one argument (e.g., -contact_hub=localhost:123,localhost:456 is now -contact_hub "localhost:123 localhost:456").

The -noui argument has been replaced with -ui (i.e., graphical user interfaces based on ServerUI , if present, are now off by default; use -ui to turn them on).

604

Finally, the arguments appendLog and -mainServerClass have been renamed to -append_log and -main_server_class respectively.

See the Java binding documentation for details on new command line arguments.

Miscellaneous

This section contains information on other potential 2.x/3.0 incompatibilities in addition to thosediscussed above.

These classes have been removed from 3.0:

galaxy.server.WrongTypeInDispatch (should not impact Galaxy Communicator developers) galaxy.io.GalaxyStreamable (should not impact Galaxy Communicator developers) galaxy.io.GalaxyStream (should not impact Galaxy Communicator developers) galaxy.util.HostPort (should not impact Galaxy Communicator developers) galaxy.lang.Symbols (symbol definitions are now public static members of galaxy.lang.Symbol ) galaxy.lang.NullFrameException (should not impact Galaxy Communicator developers)

These methods have been removed from 3.0 or renamed:

galaxy.server.MainServer.getAllowMultipleConnections galaxy.server.MainServer.setAllowMultipleConnections (use the new -maxconns command line argument or MainServer.getMaxConnections and setMaxConnections to configure the number of maximum connections) galaxy.server.MainServer.setHostPortCollection (this functionality has beenreplaced by MainServer.setHubContactInfoCollection , and there should not beany impact Galaxy Communicator developers) galaxy.lang.Symbol.setOrdinal galaxy.lang.Symbol.getOrdinal galaxy.util.ArgParser.getArgCollections (should not impact GalaxyCommunicator developers) galaxy.io.GalaxyInputStream.write(ObjectStorage) (should not impact GalaxyCommunicator developers)

The signatures of these methods have changed in 3.0:

galaxy.io.GalaxyInputStream.readBinary now returns galaxy.lang.GBinaryinstead of galaxy.lang.BinaryObject (which has been deprecated as of 3.0) The galaxy.util.ArgParser.getArg methods now return java.util.List (was java.lang.String ) since one command line argument can have a list of associatedvalues (e.g., -contact_hub ). In a related change, the version of getArg that takes adefault argument value now expects a java.util.List (was java.lang.String ). The galaxy.lang.Gframe constructor has been changed. It takes a

605

galaxy.lang.Symbol and an integer that represents the frame type (types are nowdefined in galaxy.lang.GFrame : see GAL_NULLFRAME, GAL_TOPIC, GAL_CLAUSE, GAL_PRED).

The default constructors for the array objects (galaxy.lang.Float32 , galaxy.lang.Float64 , galaxy.lang.Int16 , galaxy.lang.Int32 , and galaxy.lang.Int64 ) now initialize numerical arrays (e.g., the default Int16 constructorcreates an empty array of shorts). In 2.1, the default constructors created empty byte arrays, butusing the array objects to hold byte arrays has been deprecated in 3.0.

Finally, the Java method galaxy.server.Server.serverOpReinitialize is now publicsince the reinitialize method is treated like any other dispatch function in 3.0.

See the Galaxy Communicator release notes for additional information on changes and deprecations in the Java bindings.

Step 5: Upgrading server listen statusIn 2.1, we began to develop a more complex model of determining the server listen status, sincethe listener-in-Hub functionality raised the possibility that servers might not have listenersassociated with them. In 3.0, the situation is even more complex, since outgoing brokers share alistener with the main server. So the server listener may be listening for normal connections, forbroker connections, for both, or for neither. As a result, the flags which help determine this havechanged, and the most common queries have been implemented as API calls. For instance, theways to determine if a server has Hub client connections or if it has an active listener have changed:

Version 2.1

if ((GalIO_ServerListenStatus(scomm) & GAL_SERVER_TYPE_MASK) == GAL_HUB_CLIENT)) ...

if ((GalIO_ServerListenStatus(scomm) & GAL_SERVER_TYPE_MASK) == GAL_LISTENING_SERVER) ...

Version 3.0

if (GalIO_ServerIsClient(scomm)) ...

if (GalIO_ServerIsListener(scomm)) ...

606

Also, in 3.0, it’s possible for servers to be both clients and listeners; in 2.1, servers were one orthe other. See also the functions GalIO_ServerListensForConnections and GalIO_ServerListensForBrokers.

Step 6: Upgrading references to the call environmentThe semantics of call environments have been significantly clarified in version 3.0. Thisclarification has a small but possibly significant impact on users of the functions GalSS_EnvErrorand GalSS_EnvDestroyToken.

In version 2.1, the call environment could "force a reply", which would cause a message to besent to the Hub which counted as the reply to the Hub message which caused the dispatchfunction to fire. All subsequent messages which were written through the call environment, including the return value from the dispatch function , counted as new messages to the Hub.This functionality supported some idiosyncratic properties of the reinitialize dispatch function. Westrongly discouraged people from exploiting this functionality in other cases. So for instance,while we also used this reply forcing technique to provide error and destroy replies to Hubmessages, we advised programmers not to send any further "replies" through that callenvironment, since they would be treated as new messages. So the recommended use ofGalSS_EnvError, for instance, was as follows:

Version 2.1

Gal_Frame dispatch_fn(Gal_Frame f, void *server_data) { GalSS_Environment *env = (GalSS_Environment *) server _data;

...

if (something_bad_happened) { GalSS_EnvError(env, "Something bad happened"); return (Gal_Frame) NULL; }

... }

If the programmer had sent additional messages through the call environment (includingdispatch function replies), all the additional messages would have been treated by the Hub asnew messages:

Version 2.1

Gal_Frame dispatch_fn(Gal_Frame f, void *server_data) { GalSS_Environment *env = (GalSS_Environment *) server _data;

607

...

if (something_bad_happened) { GalSS_EnvError(env, "Something bad happened"); GalSS_EnvError(env, "It was really bad"); GalSS_SetProp(f, ":details", Gal_StringObject("Really, really bad")); return f; }

... }

Although this was documented functionality, programmers were strongly discouraged fromexploiting it.

This behavior was incoherent and difficult to understand, and it has been changed in 3.0, as partof a general clarification and extension of call environments. In 3.0, a call environment will neversend more than one return message to the Hub. Only four things count as return messages: areturn value from a dispatch function, a call to GalSS_EnvError, a call toGalSS_EnvDestroyToken, or a call to GalSS_EnvReply. After the first reply is sent, allsubsequent replies are ignored . So in the second example here, the second call toGalSS_EnvError will have no effect, and the frame returned from the dispatch function will be ignored.

Step 7: Upgrading calls to GalIO_SetServerDataIn three cases (brokers, connections, and servers), it is possible to store caller data for laterretrieval and use inside the context of these objects. It is important that functions to free this databe provided (for when the object itself is freed, among other things), but in some cases, this wasnot happening. The function GalIO_SetServerData was changed to accommodate this requirement.

As a consequence of this observation, we recommend that users now avoid returning data from _GalSS_init_server, as shown in this pre-3.0 example:


if (!GalUtil_OACheckUsage(argc, argv, oas, &i)) exit(1); GalUtil_OAExtract(argc, argv, oas, "-increment", GAL_OA_INT, &increment); signal(SIGINT, exit_gracefully); /* Return value is server data. */

608

return (void *) increment; }

The effect of this code is to store the return value as data via GalIO_SetServerData; but inaddition to this being counterintuitive, it also doesn’t allow the user the opportunity to provide adata freeing function. As of now, we recommend that users call GalIO_SetServerData explicitlyin their server initialization functions, as shown here:


if (!GalUtil_OACheckUsage(argc, argv, oas, &i)) exit(1); GalUtil_OAExtract(argc, argv, oas, "-increment", GAL_OA_INT, &increment); signal(SIGINT, exit_gracefully); GalIO_SetServerData(s, (void *) increment, NULL); return (void *) NULL; }

The server library has been rewritten so that a NULL return value does not overwrite the existingvalue. The original version still works, but it is strongly discouraged.

Step 8a: Upgrading references to the Communicator mainloop (optional)In 2.1 and previous releases, the preferred way of writing a main() which used theCommunicator main loop looked like this:


server = GalSS_CmdlineInitializeServer(argc, argv); if (!server) { GalUtil_Fatal("Failed to initialize server!\n"); } GalSS_RunServer(server); exit(0); }

However, the function GalSS_CmdlineInitializeServer actually starts up the server listener, whichdoesn’t allow us to reconfigure a number of features on an instantiated server which is not yetrunning. As a result, we now recommend a different way of writing main() for servers in version

609

3.0 and later:



The old way still works (and will continue to work for the foreseeable future), but it is dispreferred.

Step 8b: Upgrade your own main loop (advanced, optional)For those of you who are working with main loops other than the Communicator main loop,we’ve repeatedly worked to improve the embedding properties of the Galaxy Communicator API.In 3.0, as a byproduct of embedding the API in the Python and Allegro bindings, we provideanother round of recommended improvements. This is an optional upgrade ; however, it is sucha clear improvement that we recommend that everyone do it. The current strategy is discussedin detail in the external mainloop documentation; here, we focus on the contrasts with version 2.1.

The main loopIn the main loop, we provide a cleaner set of callbacks, with the following changes:

The main loop now sets up event callbacks which will set up the local callbacks, instead ofdoing it directly. This is because the server startup can happen in a number of different places. The server now has a connection event callback which handles the loop-specificinitializations when a server accepts a connection. The new strategy reflects the fact that servers can now simultaneously be listeners and Hub clients. The server now sets up the server and later runs it explicitly, rather than initializing it (settingit up and running it), as discussed in this upgrade step. The new strategy gives us uniformaccess to the event callbacks.

610

Note that it is not possible to embed the Communicator API with file descriptor callbacks alone,since sometimes extra data can be read into a connection’s inbound queue which is notimmediately processed. Therefore, at a minimum, timer callbacks must be used to handle brokerand connection readers, although file descriptor based callbacks can also be used for efficiency.

Version 2.1



/* If you want to use the built-in server arguments, you can use GalSS_ExtractServerArgs. Otherwise, you can just call GalSS_InitializeServerToplevel(). */


/* Make sure it knows that we’re using our own main loop. */ GalSS_SAFixLoopType(arg_pkg, GAL_LOOP_EXTERNAL); arg_pkg = GalSS_ExtractServerArgs(arg_pkg, argc, argv, &new_argc, &new_argv);


gcr->gcomm = GalSS_InitializeServerFromServerArgs(arg_pkg, new_argc, new_argv); GalSS_FreeArgPkg(arg_pkg);

if (!gcr->gcomm) { fprintf(stderr, "Couldn’t create a server\n"); fflush(stderr); exit(1); }

611

/* If the server is acting as a client, we’ll want to set up callbacks for the connections; if it has its own listener, we’ll want to set up a callback for the server, which will set up the listeners for the connections as they’re received. */

if ((GalIO_ServerListenStatus(gcr->gcomm) & GAL_SERVER_TYPE_MASK) == GAL_LISTENING_SERVER) { gcr->server_sock = GalIO_GetServerListenSocket(gcr->gcomm); SM_AddFDCallback(gcr->l, gcr->server_sock, DoubleServerHandler, (void *) gcr); } else { GalIO_ServerCheckHubContacts(gcr->gcomm); if (GalIO_GetServerNumConnections(gcr->gcomm) == 0) { fprintf(stderr, "Couldn’t contact any Hubs\n"); fflush(stderr); exit(1); } /* Add a callback for the connections. */ GalIO_OperateOnConnections(gcr->gcomm, (void *) gcr, __AddConnectionCallback); } SM_Mainloop(gcr->l); exit(0); }

Version 3.0



/* If you want to use the built-in server arguments, you can use GalSS_ExtractCmdlineServerArgs. Otherwise, you can just call GalSS_InitializeServerToplevel(). */

612


/* Make sure it knows that we’re using our own main loop. We set this before we ever parse the server arguments, because we don’t even want the arguments pertaining to the loop type enabled for the user. */

GalSS_SAFixLoopType(arg_pkg, GAL_LOOP_EXTERNAL); arg_pkg = GalSS_ExtractCmdlineServerArgs(arg_pkg, argc, argv, &new_argc, &new_argv);



gcr->scomm = GalSS_SetupServer(arg_pkg, new_argc, new_argv); GalSS_FreeArgPkg(arg_pkg);

if (!gcr->scomm) { fprintf(stderr, "Couldn’t create a server\n"); fflush(stderr); exit(1); }

/* Set the connect callback for the server. This gets called whenever a new connection is established. */

GalIO_AddServerConnectCallback(gcr->scomm, GCRAddConnectionCallback, (void *) gcr); /* The server can be a listener when it starts out, or it can become a listener when an outgoing broker starts up. So we set a callback to handle whenever this happens. */ GalIO_AddServerCallback(gcr->scomm, GAL_SERVER_LISTENER_STARTUP_EVENT, GCRSetupServerListener, (void *) gcr); /* Similarly, if someone calls GalIO_ContactHub, it may lead to a new poller starting up. So we should deal with that as a callback too. */ GalIO_AddServerCallback(gcr->scomm,

613

GAL_SERVER_CLIENT_POLL_STARTUP_EVENT, GCRSetupServerClient, (void *) gcr); /* And now, something that will shut down the loop when the server is destroyed. */ GalIO_AddServerCallback(gcr->scomm, GAL_SERVER_DESTRUCTION_EVENT, GCRDestroyServer, (void *) gcr);

/* Now, start the server, and then the main loop. */ if (!GalIO_ServerStart(gcr->scomm)) { fprintf(stderr, "Couldn’t start the server\n"); fflush(stderr); exit(1); } SM_Mainloop(gcr->l); exit(0); }

The server callbacksBesides the fact that we’ve generalized this mainloop example and changed the names of thecallbacks as a result, the following substantive changes can be found. We can’t illustrate thechanges in boldface, since they’re so extensive.

The startups for both client and listener are handled in parallel ways, using the eventcallbacks to set up local main loop callbacks as well as subsequent event callbacks (suchas destruction callbacks). There’s a more general callback handler (GalIO_ServerCallbackHandler) which calls theconnect callback for the server. Therefore, our server callback handler doesn’t need to setup the connection. GalIO_ServerCallbackHandler also takes care of destroying the server in the appropriate circumstances. Because the main loop requires both timer and file descriptor callbacks, both need to becleared when the server dies. There’s now a distinction between the server shutting down and the server being destroyed,which is reflected in an addition possible value returned from GalIO_ServerCallbackHandler.This is also reflected in the different event callbacks that are set up.

Version 2.1

/* This function is only used when the server is running its own listener. */

static void DoubleServerHandler(void *client_data) { int status; GalaxyCallbackRecord *gcr = (GalaxyCallbackRecord *) client_data;

614

GalIO_CommStruct *new_conn = (GalIO_CommStruct *) NULL;

status = GalIO_ServerHandler(gcr->gcomm, &new_conn);

switch (status) { case 1: /* We’ve got a connection. */ __AddConnectionCallback(new_conn, (void *) gcr); break; case -1: /* An error has occurred. */ GalUtil_Warn("The server has failed.\n"); SM_RemoveAllFDCallbacks(gcr->l); GalIO_SetServerDone(gcr->gcomm); GalIO_DestroyServerStruct(gcr->gcomm); break; default: break; } }

Version 3.0

static void GCRServerListenerHandler(void *client_data) { GalaxyCallbackRecord *gcr = (GalaxyCallbackRecord *) client_data;

GalIO_ServerCallbackHandler(gcr->scomm, 0, (GalIO_CommStruct **) NULL); }

static void GCRShutdownServerListener(GalIO_ServerStruct *scomm, void *callback_data) { GalaxyCallbackRecord *gcr = (GalaxyCallbackRecord *) callback_data;

GalUtil_Warn("The server has failed.\n"); SM_RemoveAllFDCallbacks(gcr->l); SM_RemoveAllTimerCallbacks(gcr->l); }

static void GCRSetupServerListener(GalIO_ServerStruct *scomm, void *callback_data) { GalaxyCallbackRecord *gcr = (GalaxyCallbackRecord *) callback_data; /* You only need a file descriptor callback here, since

615

there will be no connection requests in any internal queue. */ SM_AddFDCallback(gcr->l, GalIO_GetServerListenSocket(scomm), GCRServerListenerHandler, (void *) gcr); GalIO_AddServerCallback(scomm, GAL_SERVER_LISTENER_SHUTDOWN_EVENT, GCRShutdownServerListener, (void *) gcr); }

/* This function is used when the server is subscribing to Hub listeners. */

static void GCRServerClientHandler(void *client_data) { GalaxyCallbackRecord *gcr = (GalaxyCallbackRecord *) client_data;

GalIO_ServerCheckHubContacts(gcr->scomm); }

static void GCRShutdownServerClient(GalIO_ServerStruct *scomm, void *callback_data) { TimerCallback *cb = (TimerCallback *) callback_data;

SM_RemoveTimerCallback(cb); }

static void GCRSetupServerClient(GalIO_ServerStruct *scomm, void *callback_data) { GalaxyCallbackRecord *gcr = (GalaxyCallbackRecord *) callback_data; TimerCallback *cb;

/* Set up a periodic task to check the hub contacts. */ cb = SM_AddTimerCallback(gcr->l, 10, GCRServerClientHandler, (void*) gcr); /* Add a shutdown callback now. */ GalIO_AddServerCallback(scomm, GAL_SERVER_DESTRUCTION_EVENT, GCRShutdownServerClient, (void *) cb); }

static void GCRDestroyServer(GalIO_ServerStruct *scomm,

616

void *callback_data) { GalaxyCallbackRecord *gcr = (GalaxyCallbackRecord *) callback_data;

GalUtil_Warn("The server has been destroyed.\n"); SM_LooperExit(gcr->l); }

The connection callbacksThe differences here are considerable, and the structure is so different that highlighting thecontrasts is impossible. The differences are:

In 2.1, we called GalIO_ConnectionPoll, while in 3.0, we call the more general GalIO_ConnectionCallbackHandler. In 2.1, we shut down the server in the listener-in-Hub case when the last connection died,while we leave the client contact timer callback running in 3.0. In 2.1, we only used a file descriptor callback for connections, while we’ve clarified in 3.0that a timer callback is also needed. This addresses the kludge reflected in the comment in2.1 beginning "Sometimes, God help us, there’s already data". In 3.0, we make use of the disconnect callback to shut down the file descriptor and timercallbacks for the connection, while in 2.1, we did it explicitly in the connection callback. In 3.0, we can also control broker startup with event callbacks, while that option was notavailable in 2.1.

Version 2.1

static void DoubleConnectionHandler(void *client_data) { GalIO_CommStruct *new_conn = (GalIO_CommStruct *) client_data; GalaxyCallbackRecord *gcr = (GalaxyCallbackRecord *) GalIO_GetCommData(new_conn); int fd = GalIO_GetCommSocket(new_conn);

/* GalIO_ConnectionPoll returns 1 or -1 when the client should be terminated. */ switch (GalIO_ConnectionPoll(new_conn)) { case 1: /* Done, stop polling. */ case -1: /* Error, stop polling. */ SM_RemoveFDCallback(gcr->l, fd); /* If this is the last connection, and the server is acting as a client, exit. */ if (((GalIO_ServerListenStatus(gcr->gcomm) &

617

GAL_SERVER_TYPE_MASK) == GAL_HUB_CLIENT) && ((GalIO_ServerListenStatus(gcr->gcomm) &GAL_HUB_CLIENT_DISCONNECT_MASK) == GAL_HUB_CLIENT_DISCONNECT_SHUTDOWN) && (SM_NumFDCallbacks(gcr->l) == 0)) { SM_LooperExit(gcr->l); } break; default: break; } }

static void __AddConnectionCallback(GalIO_CommStruct *gcomm, void *arg) { GalaxyCallbackRecord *gcr = (GalaxyCallbackRecord *) arg; int client_sock = GalIO_GetCommSocket(gcomm);

GalIO_SetCommData(gcomm, gcr, NULL); SM_AddFDCallback(gcr->l, client_sock, DoubleConnectionHandler, (void *) gcomm);

/* Sometimes, God help us, there’s already data coming in. Particularly if this is being called when the listener is in the Hub. */ DoubleConnectionHandler((void *) gcomm); }

Version 3.0



618

/* GalIO_ConnectionCallbackHandler(): -1 means an error was encountered and the connection has been destroyed. 0 means we’re in the midst of things. 1 means we’re done and the connection has been destroyed. */

static void GCRConnectionDisconnect(GalIO_CommStruct *gcomm, void *caller_data) { ConnectionContainer *c = (ConnectionContainer *) caller_data; GalaxyCallbackRecord *gcr = c->gcr;

SM_RemoveFDCallback(gcr->l, c->fd); SM_RemoveTimerCallback(c->t); free(c); }

/* The loop cleanup is handled in the disconnect callback. */

static void GCRConnectionHandler(void *client_data) { ConnectionContainer *c = (ConnectionContainer *) client_data; GalIO_CommStruct *gcomm = c->gcomm;

GalIO_ConnectionCallbackHandler(gcomm, 0); }

static void GCRConnectionTimerHandler(void *client_data) { /* This is called from the timer. We could go ahead and just call the normal connection handler, which would try to read from the file descriptor, but since there’s already a file descriptor callback which triggers that handler, we’ll only do something if there’s stuff in the internal queues. */ ConnectionContainer *c = (ConnectionContainer *) client_data; GalIO_CommStruct *gcomm = c->gcomm;

if (GalIO_CommReadReady(gcomm) || GalIO_CommWriteReady(gcomm)) { GCRConnectionHandler(client_data); } }

static void GCRAddConnectionCallback(GalIO_ServerStruct *scomm, GalIO_CommStruct *gcomm, void *callback_data) { GalaxyCallbackRecord *gcr = (GalaxyCallbackRecord *)

619

callback_data; ConnectionContainer *c = (ConnectionContainer *) calloc(1, sizeof(ConnectionContainer));

c->gcr = gcr; c->gcomm = gcomm; c->fd = GalIO_GetCommSocket(gcomm);

/* We’ll use the file descriptor callback to check the file descriptor, and the timer callback to check the internal queue. */ SM_AddFDCallback(gcr->l, c->fd, GCRConnectionHandler, (void *) c); c->t = SM_AddTimerCallback(gcr->l, 5, GCRConnectionTimerHandler, (void *) c); /* Finally, to support brokers, and to deal with disconnections, we need to use the data slot for the connection. */ /* Make sure you stop polling when the connection dies. */ GalIO_AddConnectionCallback(gcomm, GAL_CONNECTION_SHUTDOWN_EVENT, GCRConnectionDisconnect, (void *) c); /* And now, add the callbacks for the broker setups. */ GalIO_AddConnectionBrokerCallback(gcomm, GAL_CONNECTION_BROKER_OUT_STARTUP_EVENT, GCRSetupBrokerOut, (void *) gcr); GalIO_AddConnectionBrokerCallback(gcomm, GAL_CONNECTION_BROKER_IN_STARTUP_EVENT, GCRSetupBrokerIn, (void *) gcr); }

Finally, we provide explicit examples of how to embed brokering in 3.0, while in 2.1 it wasunclear. For more details, consult the external mainloop documentation.

Step 9: Upgrade timed task invocation (advanced, optional)In version 2.1, the basic timed task was a simple function which took a void * argument. Thistask was set up using the functions Gal_AddTimedTask, Gal_AddTimedTaskWithFileIO, etc.

620

There was a more complex version, represented by Gal_AddTask, etc., which manipulated astructure called a Gal_TaskPkg ; this set of functions was used internally throughout the GalaxyCommunicator library, but was built on top of the simpler functions, and there was no particularreason for application programmers to use them for setting up timed tasks.

However, the timed task mechanism had a glaring shortcoming in version 2.1: while it waspossible to associate both read and write sockets or both read or write file pointers with a timedtask, it was not possible to tell which one led the task to be fired. Using the simpler tasks as anunderlying representation, it was impossible to address this shortcoming; however, we were ableto address it by "inverting" the implementation, making the complex tasks basic andimplementing the simple tasks on top of them. Users of Gal_AddTask, etc., can now determinewhy a task was fired using the function Gal_TaskPkgRunReasons.

In version 3.0, we recommend that everyone upgrade to complex tasks. See the timed tasks documentation for full details on the functions. Here’s the sort of change we recommend making(from the MITRE stdin polling utility):

Version 2.1

static void stdin_poll(void *server_data);

Gal_AddTimedTaskWithFileIO(stdin_poll, (void *) poll_struct, poll_struct->ms, stdin, (FILE *) NULL);

Version 3.0

static void stdin_poll(Gal_TaskPkg *p);

Gal_AddTaskWithFileIO(stdin_poll, (void *) poll_struct, poll_struct->ms, 1, stdin, (FILE *) NULL, NULL);

Note also that when you reset this task, you should call Gal_ReAddTask, etc., on the existingtask rather than create a new task. This makes a considerable difference if you ever use thethreaded toplevel loop.

Version 2.1

static void stdin_poll(void *server_data) { ... Gal_AddTimedTaskWithFileIO(stdin_poll, (void *) poll_struct, poll_struct->ms, stdin, (FILE *) NULL);

621

... }

Version 3.0

static void stdin_poll(Gal_TaskPkg *p) { ... Gal_ReAddTaskWithFileIO(p, (void *) poll_struct, poll_struct->ms, 1, stdin, (FILE *) NULL, NULL); ... }

Step 10: Upgrading session management (advanced, optional)As discussed in detail in the new session management documentation, it’s very important tokeep sessions separate, especially if you expect multiple users to interact with your systemsimultaneously. If all your dispatch functions do is return values, and they never set up othercallbacks, like timed tasks or broker callbacks, and they never write new messages to the Hub,you don’t need to worry about anything, since the server wrappers have always handled thatcase cleanly. However, all the other cases have been poorly supported up to this point.

While call environments embodied the dispatch function invocation context (including thesession information), they couldn’t be saved outside the context of their dispatch function(such as in broker callbacks or timed tasks). In order to forward the session information appropriately, the programmer needed to storeaway the session ID and add it to any new messages which were sent from broker callbacksor timed tasks. The behavior of call environments was idiosyncratic in other ways (see, for instance, thesection on upgrading call environments for version 3.0).

All these problems have been fixed in version 3.0, as far as we know, and we stronglyencourage people to take advantage of the improvements. For instance, the function GalSS_EnvBrokerDataInInit allows the programmer to set up an environment-aware broker callback.

Without environments

static void audio_handler(GalIO_BrokerStruct *broker_struct, void *data, Gal_ObjectType data_type, int n_samples) { DataHandler *d = (DataHandler *)

622

GalIO_GetBrokerCallerData(broker_struct); Gal_Frame f;

...

f = Gal_MakeFrame("notify", GAL_CLAUSE); Gal_SetProp(f, ":notification", Gal_StringObject("Audio received.")); GalIO_CommWriteFrame(d->gcomm, f, 0); Gal_FreeFrame(f);

... }

void __FreeDataHandler(GalIO_BrokerStruct *b, void *data);




With environments

static void env_audio_handler(GalIO_BrokerStruct *broker_struct, void *data, Gal_ObjectType data_type, int n_samples) { DataHandler *d = (DataHandler *) GalIO_GetBrokerCallerData(broker_struct);

623

Gal_Frame f; GalSS_Environment *env = GalSS_BrokerGetEnvironment(broker_struct);

...

f = Gal_MakeFrame("notify", GAL_CLAUSE); Gal_SetProp(f, ":notification", Gal_StringObject("Audio received.")); GalSS_EnvWriteFrame(env, f, 0); Gal_FreeFrame(f);

... }

void __FreeDataHandler(GalIO_BrokerStruct *b, void *data);


d->data_buf = (char *) NULL; d->size = 0;

if (host && port) { b = GalSS_EnvBrokerDataInInit((GalSS_Environment *) server_data, host, port, f, env_audio_handler, 0, d, __FreeDataHandler); if (b) { GalIO_SetBrokerActive(b); } } else { free(d); } return (Gal_Frame) NULL; }

At this point, we know of no reason for the programmer to use GalIO_CommWriteFrame and GalIO_DispatchViaHub. , and we strongly encourage everyone to replace all calls to thesefunctions with calls to GalSS_EnvWriteFrame and GalSS_EnvDispatchFrame. We’ve providedsupport for the following contexts:

624

Gal_AddTask can be replaced with GalSS_EnvAddTask, for those cases whereenvironments are needed for writing in timed tasks. The stdin polling function MGalIO_CreateStdinPoll should be replaced with MGalSS_EnvCreateStdinPoll in all cases where the stdin callback returns a frame, becausethat frame will be written as a new message to the Hub.

If you discover a circumstance in which it is not possible to use an environment for writing,please notify us.


625


In almost all cases, upgrading from 2.0 to 2.1 will be completely transparent.

Step 1: Configuring your system Step 2: Check your use of none! and destroy! Step 3: Check your use of SET: and DEL: with dispatch_to_main Step 4: Upgrade your own main loop (advanced, optional) Step 5: Upgrade timed task invocation (advanced) Step 6: Upgrade your local Makefile (advanced)

Step 1: Configuring your systemThe 2.1 distribution contains two major changes in your config.make file.

First, because the Galaxy Communicator distribution no longer contains any MIT servers, thereare no longer any settings for contrib/MIT. Second, it is now possible to compile GalaxyCommunicator libraries as shared libraries, which reduces the size of executables, butintroduces a run-time dependency on the shared library. MITRE’s experience suggests thatusing shared libraries can reduce the size of your Galaxy Communicator installation by 50%.See the installation notes for details.

It’s possible that you’ve included explicit dependencies on static libraries (.a ) files in yourMakefiles. Any examples you have which use the MITRE utilities library may have suchreferences. If you intend to preserve the option of compiling with shared libraries, you will needto change those references. For instance, here’s a fragment of a Makefile based on theCommunicator Makefile template, in version 2.0 and version 2.1:

Version 2.0

LIBDEPS += $(MITRE_ROOTDIR)/utilities/lib/$(ARCHOS)/libMITRE_galaxy.a

626

Version 2.1

LIBDEPS += $(MITRE_ROOTDIR)/utilities/lib/$(ARCHOS)/libMITRE_galaxy$(LIBEXT)

Although we haven’t tried it, config.make files for 2.0 should still work in 2.1; you simply won’thave the option of shared libraries.

Step 2: Check your use of none! and destroy!In version 2.0, the program file special directives none! and destroy! had the following interpretations:

OUT: none! indicated no response was expected from server operation; token referencecount was decremented OUT: destroy! indicated that the Hub expected a response from the server; token wasdestroyed after the server responded

In version 2.1, the behavior of these directives has been redefined in terms of the CONTROL:directive. As far as we can tell, none! still behaves identically, but destroy! now behavesidentically to none! , except that the program stops executing immediately after the rule.Because destroy! no longer waits until the message return to stop execution, certain rulesequences will no longer work as before. For instance, destroy! followed by none! will fireboth rules in 2.0, but only the first in 2.1:

RULE: ... ... OUT: destroy!

RULE: ... ... OUT: none!

This is a very subtle change, so inspect your program files carefully.

Step 3: Check your use of SET: and DEL: with dispatch_to_mainAs a special case in versions previous to 2.1, the SET: and DEL: directives applied to themessage, not the token, when the dispatch function was builtin.dispatch_to_main. This specialcase has been eliminated in version 2.1. The following program file illustrates the contrast:

SERVER: builtin OPERATIONS: dispatch_to_main

627

INITIAL_TOKEN: {c foo :baz 6 }

PROGRAM: foo

RULE: --> dispatch_to_main IN: :baz SET: :bar 5 DEL: :baz

PROGRAM: main

Previous to version 2.1, :baz was deleted from the message and :bar was added, with theeffect that :bar and not :baz was transmitted to the main program. In 2.1, neither changeapplies to the message, so :baz and not :bar is transmitted to the main program. This is avery subtle change, so inspect your program files carefully. The clean and proper way to transmitvalues from the foo program to the main program is and has always been to include them inthe IN: directive:

RULE: --> dispatch_to_main IN: :baz (:bar 5)

Step 4: Upgrade your own main loop (advanced, optional)In order to support the new listener-in-Hub functionality, the function GalSS_InitializeServerToplevel() is now preferred to GalSS_InitializeServer(). In addition, it isnow much more straightforward to customize server settings both before and after you parsecommand line arguments.

Unless you’re embedding a Communicator-compliant server in another top-level loop, you won’tneed to worry about this; but if you do, there are a number of differences in 2.1 that you shouldbe aware of. Here is the main() from the MITRE example fd_double.c, with the potentialdifferences in boldface:

Version 2.0

int main(int argc, char **argv) { GalaxyCallbackRecord *gcr = (GalaxyCallbackRecord *) NULL;

gcr = (GalaxyCallbackRecord *) malloc(sizeof(GalaxyCallbackRecord)); gcr->timer_cb = (TimerCallback *) NULL; gcr->server_sock = -1; gcr->l = SM_NewLooper();

628

gcr->gcomm = GalSS_InitializeServer(-1, 10, 0, 1,GAL_LOOP_EXTERNAL, 0, argc, argv);

if (!gcr->gcomm) { fprintf(stderr, "Couldn’t create a server\n"); fflush(stderr); exit(1); } gcr->server_sock = GalIO_GetServerListenSocket(gcr->gcomm); SM_AddFDCallback(gcr->l, gcr->server_sock, DoubleServerHandler, (void *) gcr); SM_Mainloop(gcr->l); exit(0); }

If you aren’t interested in the new listener-in-Hub functionality, nor interested in automaticparsing of Communicator-specific command line arguments, this example will function in 2.1. Ifyou are interested in making a minimal change to obtain access to the listener-in-Hubfunctionality, you can do this:

Version 2.0/2.1 (not recommended)


gcr = (GalaxyCallbackRecord *) malloc(sizeof(GalaxyCallbackRecord)); gcr->timer_cb = (TimerCallback *) NULL; gcr->server_sock = -1; gcr->l = SM_NewLooper(); #if GC_VERSION < 0x20100 gcr->gcomm = GalSS_InitializeServer(-1, 10, 0, 1,GAL_LOOP_EXTERNAL, 0, argc, argv); #else gcr->gcomm = GalSS_InitializeServerToplevel(-1, 10, 0, 1, GAL_LOOP_EXTERNAL, 0, -1, GAL_LISTENING_SERVER, (char *) NULL, (char*) NULL, argc, argv); #endif

if (!gcr->gcomm) { fprintf(stderr, "Couldn’t create a server\n"); fflush(stderr); exit(1);

629

}


if (GalIO_ServerListenStatus(gcr->gcomm) == GAL_LISTENING_SERVER) { gcr->server_sock = GalIO_GetServerListenSocket(gcr->gcomm); SM_AddFDCallback(gcr->l, gcr->server_sock, DoubleServerHandler, (void *) gcr); } else { /* Add a callback for the connections. */ GalIO_OperateOnConnections(gcr->gcomm, (void *) gcr, __AddConnectionCallback); }

SM_Mainloop(gcr->l); exit(0); }

You can now modify the call to GalSS_InitializeServerToplevel() to modify the listen status anddeclare the host and port you wish to contact. The second change is sensitive to the listenstatus, and will start up the server main loop appropriately. However, these changes aren’tparticularly useful unless you can pass in the host and port as command line arguments, andsince the Communicator libraries already provide this functionality, we’ve made it easy in 2.1 toexploit it.

Version 2.1 (recommended)



630

/* If you want to use the built-in server arguments, you can use GalSS_ExtractServerArgs. Otherwise, you can just call GalSS_InitializeServerToplevel(). */


/* Make sure it knows that we’re using our own main loop. We set this before we ever parse the server arguments, because we don’teven want the arguments pertaining to the loop type enabled for the user. */ GalSS_SAFixLoopType(arg_pkg, GAL_LOOP_EXTERNAL); arg_pkg = GalSS_ExtractCmdlineServerArgs(arg_pkg, argc, argv, &new_argc, &new_argv);



gcr->gcomm = GalSS_InitializeServerFromServerArgs(arg_pkg, new_argc, new_argv); GalSS_FreeArgPkg(arg_pkg);

if (!gcr->gcomm) { fprintf(stderr, "Couldn’t create a server\n"); fflush(stderr); exit(1); }


if ((GalIO_ServerListenStatus(gcr->gcomm) & GAL_SERVER_TYPE_MASK) == GAL_LISTENING_SERVER) { gcr->server_sock = GalIO_GetServerListenSocket(gcr->gcomm); SM_AddFDCallback(gcr->l, gcr->server_sock, DoubleServerHandler,

631

(void *) gcr); } else { GalIO_ServerCheckHubContacts(gcr->gcomm); if (GalIO_GetServerNumConnections(gcr->gcomm) == 0) { fprintf(stderr, "Couldn’t contact any Hubs\n"); fflush(stderr); exit(1); } /* Add a callback for the connections. */ GalIO_OperateOnConnections(gcr->gcomm, (void *) gcr, __AddConnectionCallback); } SM_Mainloop(gcr->l); exit(0); }

In this final version, the first difference is that 2.1 includes support for using the built-in commandline argument parsing tools to "peel off" the standard server arguments, using the functions GalSS_DefaultServerArgs(), GalSS_InitializeServerFromServerArgs(), and GalSS_FreeArgPkg(). Before you extract the server args with GalSS_ExtractServerArgs(), youcan make whatever changes to the argument package you choose; here, for instance, we’retelling the server that it’s not going to run its own main loop. Because of this functionality, you willnot need to worry about the function signature for GalSS_InitializeServerToplevel(), nor do youhave to recapitulate the Communicator argument parsing if you desire that level of flexibility. Ifyou don’t want to give the user access to some of these settings, of course, you can eitheroverwrite them between the first two function calls, or simply useGalSS_InitializeServerToplevel() directly.

The second difference is that once we permit the listener-in-Hub functionality, we need to checkthe listen status of the server and initialize our own listeners appropriately. In this case, if theserver has a listener in it, we add a file descriptor callback for the server listener; otherwise, if theserver is contacting Hubs, we check the Hub contacts and then call GalIO_OperateOnConnections() with an operation which makes sure that reinitialize messagesthat are queued as part of the listener-in-Hub handshake are actually processed.

Notice that because the default C behavior for listener-in-Hub functionality is to retry both forinitial connections and after disconnects, this loop doesn’t conform to the default C behaviorbecause it’s only possible to implement that sort of retry using timers, and this example onlyuses file descriptor callbacks.

Step 5: Upgrade timed task invocation (advanced)Version 2.0 had a logic problem involving the interaction of sockets and the timed task loop onWin32. To fix this bug, the function Gal_AddTimedTaskWithIO() had to be eliminated, andreplaced with the new functions Gal_AddTimedTaskWithSocketIO() and Gal_AddTimedTaskWithFileIO(). The chances are extremely slim that you have used the

632

function Gal_AddTimedTaskWithIO() in your own code; however, if you have, you must replaceit with the appropriate version, depending on whether you’re using it to poll files or sockets. Forinstance, the MITRE stdin polling utility had to be changed as follows:

Version 2.0

fd_set readfd; unsigned int fd = fileno(stdin);

FD_ZERO(&readfd); FD_SET(fd, &readfd); Gal_AddTimedTaskWithIO(stdin_poll, (void *) poll_struct, poll_struct->ms, &readfd, (fd_set *) NULL);

Version 2.1

Gal_AddTimedTaskWithFileIO (stdin_poll, (void *) poll_struct, poll_struct->ms, stdin, (FILE *) NULL);

In addition, because select() doesn’t support polling on file descriptors under Win32, the file I/Opolling is not available on Win32.

Step 6: Upgrade your local Makefile (advanced)If you don’t use the C Makefile templates included in the distribution, and you opt for sharedobject compilation when you install Galaxy Communicator, you will have to update yourreferences to the Galaxy Communicator libraries in your C Makefiles if you’ve chosen to encodelibrary dependencies. We illustrate with a relevant subsection of the Makefile in our localMakefile example:

Version 2.0


LIBDEPS = $(ROOTDIR)/lib/$(ARCHOS)/libGalaxy.a

SOURCES = ../double/double.c ../double/double_core.c

double: ../double/double.c $(LIBDEPS) gcc -g $(CPPFLAGS) -I../double -o double.$(ARCHOS).bin$(SOURCES) $(LIBS)

633

Version 2.1


# Depending on whether there are shared libraries or not, # we want to pick the right one library extension, etc.

include $(TEMPLATES)/shared_libs.make

ifdef SHARED_LIBS ifdef XLINKER_RTFLAG LIBS += -Xlinker $(XLINKER_RTFLAG) -Xlinker $(ROOTDIR)/lib/$(ARCHOS) endif endif

LIBDEPS = $(ROOTDIR)/lib/$(ARCHOS)/libGalaxy$(LIBEXT)

SOURCES = ../double/double.c ../double/double_core.c

double: ../double/double.c $(LIBDEPS) gcc -g $(CPPFLAGS) -I../double -o double.$(ARCHOS).bin$(SOURCES) $(LIBS) -lm

The file templates/shared_libs.make is guaranteed to be created when you install andcompile the Galaxy Communicator system, and it contains the appropriate setting for LIBEXT foryou to use in other Makefiles, as well as the proper linker flag to pass via gcc which will set therun-time location of the shared library in the executable.


634

Upgrading from 1.2 or 1.3 to 2.0

For most applications, upgrading from 1.2 to 1.3 will be fairly transparent. The release notesshould provide enough detail. However, the upgrade from 1.3 to 2.0 is much more complex.Here, we attempt to provide a step-by-step guide to updating your systems.

Step 0: updating to 1.3 Step 1: configuring your system Step 2: updating your headers Step 3: updating your command line argument parsing and printing Step 4: updating your header files Step 5: upgrading server access inside a dispatch function Step 6: upgrading server access inside the reinitialize message Step 7: upgrading server access from broker handlers Step 8 (optional): upgrading outgoing brokering creation Step 9 (advanced): enabling signal handling Step 10 (advanced): updating your own main loop Step 11 (advanced): updating your use of stdin polling

Step 0: Updating to 1.3First, be sure you’ve consulted the release notes for version 1.3 and updated appropriately.Things to watch out for:

The function GalIO_SetBrokerActive must now be called on an incoming broker connectionin order for it to be processed. Also, the function to be called when all the data has beenwritten to a broker connection has changed from GalIO_BrokerDataDone to GalIO_BrokerDataOutDone. The syntax of program files has been rationalized and generalized, and in the process anumber of the more arcane constructs have been removed. Consult the release notes for details. A few minor functions have different signatures; none of these functions should be functionsyou use.

635

Step 1: Configuring your systemIn previous releases, someone configuring a Communicator distribution was required to editthree files: templates/init.make, templates/site.make, and templates/links.setenv.<archos>. Eachof these files was represented in the distribution by a .in file which had to be copied andmodified. Furthermore, the file templates/sysdep.make had additional platform information whichinteracted with some of these configuration files but had no .in equivalent. Finally, the first twoconfiguration files were Makefiles, while the third was a shell script. In Galaxy Communicator2.0, all this information is centralized in a single file templates/config.make.in, which also hasdetailed instructions interleaved. See the installation notes for details.

Step 2: Updating your headersThe Communicator distribution now uses a single header file galaxy/galaxy_all.h. Significantly,the header file sls/util.h has been replaced by galaxy/util.h in the set of headers thatCommunicator servers use. Change all references to sls/util.h to galaxy/util.h, or better yet, justreplace the set of four headers with a single reference to galaxy/galaxy_all.h.

Step 3: Updating your command line argument parsingand printingThe old functions and types stand in a one-to-one correspondence with the new ones; only thenames have been changed, to guarantee that the core library won’t have symbol clashes withthe MIT utilities library and to provide naming consistency.

old new

FALSE GAL_FALSE

TRUE GAL_TRUE

BOOL Gal_Boolean

OA_CHAR GAL_OA_CHAR

OA_SHORT GAL_OA_SHORT

OA_INT GAL_OA_INT

OA_FLOAT GAL_OA_FLOAT

OA_STRING GAL_OA_STRING

OA_DOUBLE GAL_OA_DOUBLE

636

oa_check_usage GalUtil_OACheckUsage

oa_print_usage GalUtil_OAPrintUsage

oa_extract GalUtil_OAExtract

oa_extract_asserting GalUtil_OAExtractAsserting

SLS_VERBOSE GAL_VERBOSE

sls_fatal GalUtil_Fatal

sls_warn GalUtil_Warn

sls_error GalUtil_Error

sls_print GalUtil_Print

sls_cprint GalUtil_CPrint

sls_pinfo1 GalUtil_PInfo1

sls_pinfo2 GalUtil_PInfo2

sls_cpinfo1 GalUtil_CPInfo1

sls_cpinfo2 GalUtil_CPInfo2

sls_debug1 GalUtil_Debug1

sls_debug2 GalUtil_Debug2

sls_assert GalUtil_Assert

sls_verbose_use_bw GalUtil_VerboseUseBW

sls_verbose_use_color GalUtil_VerboseUseColor

Step 4: Updating your header filesIn GalaxyCommunicator 1.2, you needed a stubber to generate the headers "server.h" and"functions.h", which was not particularly portable across platforms because of the dependenceon the power of GNU make. In GalaxyCommunicator 1.3, you had the option of using the newstyle of header declaration, which relies only on the C preprocessor; the old style was stillavailable, but deprecated. In Galaxy Communicator 2.0, the old style is no longer supported, andyou must upgrade.

Here’s how the old style worked. For server declarations, the operations file contains theinformation about the server: server name, port, and operations. This file was examined atserver compile time to determine which operations would be called. Based on the operationslisted, a header file named "server.h" was automatically generated which mapped the name of

637

each operation to the corresponding function in the server. Port was needed to set the defaultport (unlike the program file, host was unnecessary). Here’s a sample operations file from thedouble server:

SERVER: double-server PORT: 2800 OPERATIONS: twice reinitialize

The automatically generated header file "server.h" was added to exactly one of the source filesin the server source code, and the operations file was declared in the Makefile. Finally, theSERVER_DATA directive instructed the stubber to generate headers for functions which took asecond argument (actually the connection object).

For dialogue control headers, the file "functions.h" was automatically generated which containedboth the dialogue control function prototypes and a function map. The function prototypes weredictated by the operations listed in the dialogue control file rules. The function map mapped thefunction name to the actual function call. The location of the dialogue control file was declared inthe Makefile using DCTL_FILE, and USE_DCTL was set to 1.

Step 4a

First, generate the new headers. In version 2.0, MITRE provided a utility to help upgrading fromthe old to the new style. A detailed description of the new header structure for servers can befound here.; for dialogue control, look here.

Step 4b

Next, remove the OPERATIONS_FILE, SFUNC.H, SERVER_DATA, USE_DCTL, DCTL_FILE,and DFUNC.H declarations from your Makefile. Here’s what they used to do:

638

Variable What it does Obligatory? Slot?

OPERATIONS_FILE

The name of the operations file you need todefine if you’re using the old style of declaringserver information. Defaults to galaxy/System/servers .

no yes

SFUNC.HThe name of the operations header file togenerate, used for the old style of declaringserver information. Defaults to server.h .

no no

SERVER_DATA

Value should be -server_data if you want togenerate dispatch function signatures whichcontain an argument for the server structure. Youshould always use this if you’re using the old style of declaring server information.

no, but use itanyway ifyou’re usingthe old style

yes

USE_DCTL

If the server uses the dialogue controlmechanism, the value of this variable should be1, using the old style of generating dialoguecontrol headers.

no yes

DCTL_FILE

This variable specifies the dialogue control file forgenerating the dialogue function map, using the old style of generating dialogue control headers(the equivalent of server.h for the dialoguecontrol mechanism). The default is galaxy/System/$(SERVER).dctl .

no yes

DFUNC.H

The name of the header file to generate for thedialogue function map, using the old style ofgenerating dialogue control headers. Defaults to functions.h .

no no

Step 5: Upgrading server access inside a dispatch functionIn Galaxy Communicator 2.0, multiple connection support is built in, and as a result we havesignficantly reorganized the connection management "under the hood" to provide a moreconsistent infrastructure. The distribution now makes a clear distinction between connectionsand servers, and introduces the notion of a call environment to embody the context of each callto a dispatch function (the administrative information that needs to be returned from that call,etc.). This object also provides access to the connection object (and to the server object). As aresult, you will probably need to tease apart the different types of access to the connectioninfrastructure that you’ve made in your code.

639

In particular, the following important generalization applies:

If you write or dispatch frames or access the connection object inside a dispatch function ,you may use the call environment (functions which start with GalSS_Env ). If you write or dispatch frames or access the connection object outside a dispatch function (that is, in brokering callbacks or other timed tasks), you must use the connectionobject itself. In particular, you cannot save away a call environment object inside a dispatchfunction and expect it to be accessible outside the dispatch function.

We will elaborate on these distinctions below.

The first of these changes in the distribution has to do with the second argument of dispatchfunctions. In GalaxyCommunicator 1.3 and previous, the server_data argument of dispatchfunctions was not reliably a connection object. In 2.0, this argument is guaranteed to be aGalSS_Environment * (the call environment of the dispatch function). All functions which handledthis unreliability have been deprecated. See the section on adding a server.

In your dispatch functions, make the following modifications:

GalSS_WriteFrameToHub(frame, server_data, 0); becomes GalSS_EnvWriteFrame((GalSS_Environment *) server_data, frame, 0); GalSS_DispatchViaHub(frame, server_data); becomes GalSS_EnvDispatchFrame((GalSS_Environment *) server_data, frame, &msg_type);

Note that the order of the frame and connection arguments is reversed, and that theserver-to-server subdialogues now return the type of the message return.

GalSS_GetServerData(server_data); becomes GalIO_GetCommServerData(GalSS_EnvComm((GalSS_Environment *) server_data));

More details are available in the server architecture documentation.

Note that parallel comments apply to the Python, Java and Common Lisp bindings.

In Python and Common Lisp, we’ve provided backward compatibility at the expense of a slightname misnomer; the Connection object is now actually a call environment object. If all you do iswrite frames to the Hub, access the server object and access the server object’s output stream,these operations are all still transparently supported and your code should not need to change.

Note that the comments here apply only to dispatch functions. When you build aCommunicator-compliant server, you have many opportunities to access the connection objectfor writing, etc. In addition to dispatch functions, you may set up another timed task whosecallback function writes to the Hub (for instance, the MITRE stdin polling utility does this), or you

640

might write a broker callback which writes to the Hub (as in the MITRE echo server). Even if youhave access to a GalSS_Environment * object when you set up these callbacks, you cannotstore this object away; as far as the programmer is concerned, the environment object does notpersist past the scope of the dispatch function it is passed to. Instead, you must save away theconnection object directly (accessible from the environment object via the GalSS_EnvComm()function. See the upgrade section on broker callbacks for a detailed comparison between 1.3and 2.0 in such a case.

Step 6: Upgrading server access inside the reinitialize messageIn GalaxyCommunicator 1.3 and previous, the reinitialize message behaved quitedifferently from other dispatch functions, in that the first frame it sent back to the Hub wasinterpreted as its return value, rather than the actual return from the dispatch function. Thismeant that if you wanted to send a new message from the reinitialize dispatch function, you hadto write back a "dummy" frame first. Furthermore, at least one frame had to be sent, either as anexplicit send or as the function return, because the Hub needed some response from thereinitialize message (even though it ignored it). These idiosyncracies have been eliminated.From the programmer’s standpoint, the behavior of message dispatches in reinitialize is nowidentical to other dispatch functions. No return is required, and messages sent from inside reinitialize will be reliably interpreted as new messages, as they should. (Warning:reinitialize messages are still special "under the hood", and the reinitialize message can’t yet becalled anywhere except at startup. We will fix this problem in a subsequent release.)

Because the previous logic was obscure, and people may not want to take the time toreconstruct the proper logic, we have introduced the function GalSS_EnvForceReturn, whichexplicitly forces its frame argument to be treated as the dispatch function return. All messagessent to the Hub after a forced return are treated as new messages.This function can be used inany dispatch function, not just in reinitialize; most significantly, however, it can be used as asimple explicit substitute for the dummy message send. We illustrate the mapping with the reinitialize dispatch function from double.c.

Old:

Gal_Frame reinitialize(Gal_Frame frame, void *server_data) { Gal_Frame new_f = Gal_MakeFrame("main", GAL_CLAUSE);

/* Reply to reinitialize. */ GalSS_WriteFrameToHub(frame, server_data, 0); /* New message. */ Gal_SetProp(new_f, ":int", Gal_IntObject(InitialIncrement)); return new_f; }

641

Transparent substitution of GalSS_EnvForceReturn:


/* Reply to reinitialize. */ GalSS_EnvForceReturn((GalSS_Environment *) server_data, f, 0); /* New message. */ Gal_SetProp(new_f, ":int", Gal_IntObject(InitialIncrement)); return new_f; }

Corrected logic:


/* New message. */ Gal_SetProp(new_f, ":int", Gal_IntObject(InitialIncrement)); GalSS_EnvWriteFrame((GalSS_Environment *) server_data, new_f, 0); Gal_FreeFrame(new_f); return (Gal_Frame) NULL; }


In Python and Common Lisp, the required changes are exactly parallel. You may eithertransparently force a return, or reorganize the logic to match other dispatch functions.

Step 7: Upgrading server access from broker handlersIf you access server data or send messages to the Hub from broker callback functions, your taskis slightly complicated than before, due to the new server/connection/environment paradigm.This example is drawn from the audio template demo.

Old:

static void synth_data_handler(GalIO_BrokerStruct *broker_struct,void *data, Gal_ObjectType data_type, int n_elements) { Gal_Frame fr;

642

switch (data_type) { case GAL_FRAME: printf("Answer is: ‘%s’\n", Gal_GetString((Gal_Frame) data, ":reply_string")); GalIO_BrokerDataDone(broker_struct);

/* Now send playing is done */ fr = Gal_MakeFrame("audio", GAL_CLAUSE); Gal_SetProp(fr, ":playing_has_ended", Gal_IntObject(1)); GalSS_WriteFrameToHub(fr, (void *) NULL, 0); Gal_FreeFrame(fr); break; default: sls_warn("synth_data_handler: unexpected data type %s\n", Gal_ObjectTypeString(data_type)); } }

Gal_Frame handle_synth_data(Gal_Frame frame, void *server_data) { char *host; int port; GalIO_BrokerStruct *b;

host = (char *)Gal_GetValue(frame, ":synth_host", GAL_STRING); port = (int)Gal_GetValue(frame, ":synth_port", GAL_INT);

if (host && port) { b = GalIO_BrokerDataInInit(host, port, frame, synth_data_handler, server_data, 0); if (b) GalIO_SetBrokerActive(b); } return frame; }

New:

static void synth_data_handler(GalIO_BrokerStruct *broker_struct,void *data, Gal_ObjectType data_type, int n_elements) { Gal_Frame fr;

switch (data_type) { case GAL_FRAME: printf("Answer is: ‘%s’\n", Gal_GetString((Gal_Frame) data, ":reply_string"));

643

GalIO_BrokerDataDone(broker_struct);

/* Now send playing is done */ fr = Gal_MakeFrame("audio", GAL_CLAUSE); Gal_SetProp(fr, ":playing_has_ended", Gal_IntObject(1)); GalIO_CommWriteFrame((GalIO_CommStruct *)GalIO_GetBrokerCallerData(broker_struct), fr, 0); Gal_FreeFrame(fr); break; default: GalUtil_Warn("synth_data_handler: unexpected data type %s\n", Gal_ObjectTypeString(data_type)); } }

Gal_Frame handle_synth_data(Gal_Frame frame, void *server_data) { char *host; int port; GalIO_BrokerStruct *b;

host = (char *)Gal_GetValue(frame, ":synth_host", GAL_STRING); port = (int)Gal_GetValue(frame, ":synth_port", GAL_INT);

if (host && port) { b = GalIO_BrokerDataInInit(host, port, frame, synth_data_handler, GalSS_EnvComm((GalSS_Environment *)server_data), 0); if (b) GalIO_SetBrokerActive(b); } return frame; }

Previously, the argument passed to the dispatch function was the server argument or NULL; nowit is guaranteed to be a GalSS_Environment *. However, the environment object is reclaimedafter the dispatch function is called, and as a result it can’t be passed to the broker callbackfunction to be stored. So the right thing to do is to store the actual connection object, accessedwith the GalSS_EnvComm function, and send a message from the broker callback using GalIO_CommWriteFrame. If you want to access connection or server data, the lesson is thesame: store the actual connection object and use GalIO_GetCommServerData and GalIO_GetCommData.


644

In Python and Common Lisp, the broker data must be passed the connection object, but thedispatch function argument is now a call environment object instead. You must access theconnection object from it. See the brokering examples in MITRE’s audio example.

Step 8 (optional): Upgrading outgoing brokering creationThere are a number of reserved keys which various aspects of the Communicator library rely onfor communication. We’re trying to isolate those keys and protect them with functions when theyare visible to the programmer. One instance of this is in setting up an outgoing brokeringconnection, in which the keys which store the broker port and the unique call ID must currentlybe set by the programmer to ensure a well-formed brokering connection. MITRE recommendsthe following upgrade. This upgrade affects outgoing brokering connections only. The example istaken from one of MITRE’s audio examples, and is the same as the one used in the brokering documentation.

Old:

static Gal_Frame prepare_audio_frame(char *filename) { /* .... */

/* Now that we have the audio, we add a binary element. */ /* These must be added to the broker frame BEFORE the broker object is initialized. */ sprintf(host_pid, "%s:%d", GalIO_IPAddress(), (int) getpid()); Gal_SetProp(f, ":call_id", Gal_StringObject(host_pid)); b = GalIO_BrokerDataOutInit(3900, f, 0, 0); if (b && ( Gal_GetInt(f, ":broker_port") > 0)) { Gal_SetProp(f, ":binary_host", Gal_StringObject(GalIO_IPAddress())); Gal_SetProp(f, ":binary_port", Gal_IntObject( Gal_GetInt(f, ":broker_port") )); GalIO_BrokerWriteString(b, AUDIO_START); GalIO_BrokerWriteBinary(b, buf, total); GalIO_BrokerWriteString(b, AUDIO_END); GalIO_BrokerDataOutDone(b); } return f; }

New:

static Gal_Frame prepare_audio_frame(char *filename) { /* .... */

645

/* Now that we have the audio, we add a binary element. */ /* These must be added to the broker frame BEFORE the broker object is initialized. */ sprintf(host_pid, "%s:%d", GalIO_IPAddress(), (int) getpid()); GalIO_FrameSetBrokerCallID(f, host_pid); b = GalIO_BrokerDataOutInit(3900, f, 0, 0); if (b && ( GalIO_FrameGetBrokerPort(f) > 0)) { Gal_SetProp(f, ":binary_host", Gal_StringObject(GalIO_IPAddress())); Gal_SetProp(f, ":binary_port", Gal_IntObject( GalIO_FrameGetBrokerPort(f) )); GalIO_BrokerWriteString(b, AUDIO_START); GalIO_BrokerWriteBinary(b, buf, total); GalIO_BrokerWriteString(b, AUDIO_END); GalIO_BrokerDataOutDone(b); } return f; }

Step 9 (advanced): Enabling signal handlingIf you set signal handlers in your code anywhere, you should replace all calls to signal() andsigset() with calls to Gal_AddSignalHandler. This is in order to ensure that signals are handledcorrectly under threads. If you never intend to compile your servers with threads, you can omitthis step.

Step 10 (advanced): Updating your own main loopIf you have written applications with their own main loops, you will have to make some extensivechanges, once again due to the new server/connection/environment paradigm. The crucial stepsare outlined here. We’ll use the fd_double example.

Step 10a

First, you need to change the way your server is created.

Old:


gcr = (GalaxyCallbackRecord *) malloc(sizeof(GalaxyCallbackRecord)); gcr->timer_cb = (TimerCallback *) NULL;

646

gcr->server_sock = -1; gcr->client_sock = -1; gcr->l = SM_NewLooper();

Gal_InitializeServerDefaults(ServerName, DefaultPort, SvrFunctionMap); gcr->gcomm = GalIO_ServerInit(Gal_GetServerPort(), 0, GalSS_FrameHandler, (void *) gcr, -1); if (!gcr->gcomm) { fprintf(stderr, "Couldn’t create a server\n"); fflush(stderr); free(gcr->l); free(gcr); exit(1); } gcr->server_sock = GalIO_GetCommListenSocket(gcr->gcomm); SM_AddFDCallback(gcr->l, gcr->server_sock, DoubleServerHandler, (void *) gcr); SM_Mainloop(gcr->l); exit(0); }

New:



gcr->gcomm = GalSS_InitializeServer(-1, 10, 0, 1,GAL_LOOP_EXTERNAL, 0, argc, argv);

if (!gcr->gcomm) { fprintf(stderr, "Couldn’t create a server\n"); fflush(stderr); free(gcr->l); free(gcr); exit(1); } gcr->server_sock = GalIO_GetServerListenSocket(gcr->gcomm); SM_AddFDCallback(gcr->l, gcr->server_sock, DoubleServerHandler, (void *) gcr);

647

SM_Mainloop(gcr->l); exit(0); }

Observe the following differences:

You should no longer call Gal_InitializeServerDefaults (which has been replaced by GalSS_InitializeServerDefaults anyway), nor do you need to call GalIO_ServerInit. Thisfunctionality has all been wrapped into GalSS_InitializeServer and GalSS_CmdlineInitializeServer. It’s no longer necessary to use the (removed) function Gal_GetServerPort to get the defaultport; handling the default appropriately is dealt with in GalSS_InitializeServer. It’s no longer possible to change what the frame handler function is (we believe this is agood thing). The server socket is now retrieved by GalIO_GetServerListenSocket, which replacesGalIO_GetCommListenSocket (removed). Although it’s not apparent from the example here, the type of the object returned byGalSS_InitializeServer is a GalIO_ServerStruct *, not a GalIO_CommStruct *. The latter isreserved for connections. The callback record is no longer stored as server data by virtue of being passed to GalIO_ServerInit. Finally, notice that I no longer track the client_sock in my callback record, because therecan be more than one connection active at any time (I’ve specified a maximum of 10, as youcan see).

Step 10b

Next, you need to handle the way server callbacks are handled.

Old:

static void DoubleServerHandler(void *client_data) { int status; GalaxyCallbackRecord *gcr = (GalaxyCallbackRecord *) client_data;

status = GalIO_ServerHandler(gcr->gcomm); if ((status != -1) && (gcr->client_sock == -1)) { gcr->client_sock = GalIO_GetCommSocket(gcr->gcomm); SM_AddFDCallback(gcr->l, gcr->client_sock, DoubleConnectionHandler, (void *) client_data); } }

648

New:

static void DoubleServerHandler(void *client_data) { int status; GalaxyCallbackRecord *gcr = (GalaxyCallbackRecord *) client_data; GalIO_CommStruct *new_conn = (GalIO_CommStruct *) NULL; int client_sock;

status = GalIO_ServerHandler(gcr->gcomm, &new_conn);

switch (status) { case 1: /* We’ve got a connection. */ client_sock = GalIO_GetCommSocket(new_conn); GalIO_SetCommData(new_conn, gcr, NULL); SM_AddFDCallback(gcr->l, client_sock, DoubleConnectionHandler, (void *) new_conn); break; case -1: /* An error has occurred. */ GalUtil_Warn("The server has failed.\n"); SM_RemoveAllFDCallbacks(gcr->l); GalIO_SetServerDone(gcr->gcomm); GalIO_DestroyServerStruct(gcr->gcomm); break; default: break; } }


GalIO_ServerHandler now takes a pointer to a new connection as an argument, which it fillsif a new connection is established. The server data for the connection is updated using the new function GalIO_SetCommData.The callback record is connection-specific data now, not server-specific data; I’m storing thecallback record there so I can get at it in the connection callback. The return status for GalIO_ServerHandler has changed slightly. -1 now unambiguouslymeans that the server has failed and the server should be marked as done and destroyed. 1means that a connection has been established, as before. It is no longer necessary to check the state of the client socket, because we’re not tracking itdue to the possibility of multiple connections. We must be sure to set up polling for the connection on the connection object, rather thanthe server object.

649

If you’re using timers instead of file descriptor callbacks, the comparison is even more striking.

Old (from timer_double.c):

static void DoubleHandler(void *client_data) { GalaxyCallbackRecord *gcr = (GalaxyCallbackRecord *) client_data;

if (!gcr->server_connected) { switch (GalIO_ServerHandler(gcr->gcomm)) { case 1: gcr->server_connected = 1; } } else { switch (GalIO_ConnectionPoll(gcr->gcomm)) { case 1: /* Shut down the server */ SM_RemoveTimerCallback(gcr->l, gcr->timer_cb); gcr->server_sock = -1; gcr->server_connected = 0; gcr->client_sock = -1; break; case -1: /* the connection is done */ gcr->server_connected = 0; gcr->client_sock = -1; break; } } }

New:

static void DoubleHandler(void *client_data) { GalaxyCallbackRecord *gcr = (GalaxyCallbackRecord *) client_data;

GalIO_ServerPoll(gcr->gcomm); }

The new function GalIO_ServerPoll takes care of polling all the server’s connections if the timedtask loop is not enabled. You could also use this function in the file descriptor case, but it’s less efficient.

650

Step 10c

Next, you need to change the way connection callbacks are handled.

Old:

static void DoubleConnectionHandler(void *client_data) { GalaxyCallbackRecord *gcr = (GalaxyCallbackRecord *) client_data;

/* GalIO_ConnectionPoll returns 1 when the server and client should be terminated, -1 when the client should be terminated. */ switch (GalIO_ConnectionPoll(gcr->gcomm)) { case 1: /* Server disconnected */ SM_RemoveFDCallback(gcr->l, gcr->server_sock); SM_RemoveFDCallback(gcr->l, gcr->client_sock); gcr->client_sock = -1; gcr->server_sock = -1; gcr->gcomm = (GalIO_CommStruct *) NULL; break; case -1: /* Client disconnected */ SM_RemoveFDCallback(gcr->l, gcr->client_sock); gcr->client_sock = -1; break; } }

New:

static void DoubleConnectionHandler(void *client_data) { GalIO_CommStruct *new_conn = (GalIO_CommStruct *) client_data; GalaxyCallbackRecord *gcr = (GalaxyCallbackRecord *) GalIO_GetCommData(new_conn); int fd = GalIO_GetCommSocket(new_conn);

/* GalIO_ConnectionPoll returns 1 or -1 when the client should be terminated. */ switch (GalIO_ConnectionPoll(new_conn)) { case 1: /* Done, stop polling. */ case -1: /* Error, stop polling. */ SM_RemoveFDCallback(gcr->l, fd);

651

break; default: break; } }


A return value of 1 from GalIO_ConnectionPoll no longer means that the server should beshut down, but only that the client has been shut down. GalIO_ConnectionPoll takes care ofremoving the connection; as before, all the handler has to do is remove the callback on thesocket. Note also that the function GalIO_ConnectionPoll is now restricted to contexts inwhich the timed task loop is not used, since it no longer resets its own timed task. This is aninternal change which should not be visible to any users.

Step 11 (advanced): updating your use of stdin pollingIf you use the MITRE stdin polling tool, you may want to modify it to reflect the fact that serverscan accept more than one connection at a time. At the very least, you’d want to set the servermaximum connections to 1; if you really want to be clean, you should modify your use of stdinpolling to ensure that the poll is stored in the connection object, not in the server object. Althoughonly one connection at a time is being accepted, this step ensures that the poll object isterminated at the appropriate points (that is, when connections die). This example is taken fromthe MITRE UI server.

Old:

void *_GalSS_init_server(GalIO_ServerStruct *s, int argc, char **argv) { char *prompt; int i; char *prog_name; MGal_StdinPoll *poll_obj;

if (!oa_check_usage(argc, argv, OAS, &i)) exit(1); if (!oa_extract(argc, argv, OAS, "-prompt", OA_STRING, &prompt)) prompt = DefaultPrompt; if (!oa_extract(argc, argv, OAS, "-prog_name", OA_STRING, &prog_name)) prog_name = DefaultProgram;

poll_obj = MGalSS_CreateStdinPoll(prompt, (void *) s, CreateTextFrame,500, 0);

652

MGal_SetStdinPollData(poll_obj, (void *) prog_name);

return (void *) poll_obj; }

Gal_Frame reinitialize(Gal_Frame f, void *server_data) { if (Gal_GetObject(f, ":greeting")) { sls_print(-1, "[Greeting] %s\n", Gal_GetString(f, ":greeting")); } /* set up poll for stdin */ MGal_ActivateStdinPoll((MGal_StdinPoll *) GalSS_GetServerData(server_data)); return f; }

New:

void *_GalSS_init_server(GalIO_ServerStruct *s, int argc, char **argv) { int i;

if (!GalUtil_OACheckUsage(argc, argv, OAS, &i)) exit(1); if (!GalUtil_OAExtract(argc, argv, OAS, "-prompt", GAL_OA_STRING, &Prompt)) Prompt = DefaultPrompt; if (!GalUtil_OAExtract(argc, argv, OAS, "-prog_name",GAL_OA_STRING, &ProgName)) ProgName = DefaultProgram; GalIO_SetServerMaxConnections(s, 1); return (void *) NULL; }

Gal_Frame reinitialize(Gal_Frame f, void *server_data) { GalSS_Environment *env = (GalSS_Environment *) server_data;

if (Gal_GetObject(f, ":greeting")) { GalUtil_Print(-1, "[Greeting] %s\n", Gal_GetString(f, ":greeting")); }

GalSS_EnvSetCommData(env, (void *) MGalIO_CreateStdinPoll(Prompt, GalSS_EnvComm(env), CreateTextFrame, 500, 0), FreePoll); MGal_SetStdinPollData(GalSS_EnvGetCommData(env), (void *)

653

ProgName); /* set up poll for stdin */ MGal_ActivateStdinPoll((MGal_StdinPoll *) GalSS_EnvGetCommData(env)); return f; }


Previously, the poll object was created in _GalSS_init_server, returned, and stored asserver data. In the new implementation, the poll object is created in reinitialize and stored asconnection-specific data using GalSS_EnvSetCommData. The deprecated function GalSS_GetServerData is no longer used to retrieve the poll object.It has been replaced by GalSS_EnvGetCommData. The server is now set to take no more than a single connection, which is a natural restrictionon a program which polls stdin. The poll object will be freed with the FreePoll function, which will be called when theconnection is dropped. The function MGalSS_CreateStdinPoll has been eliminated in favor of the preexisting MGalIO_CreateStdinPoll.


654

Release Notes

This document briefly lists all the changes from one Galaxy Communicator distribution to thenext for the current major version of Galaxy Communicator. You can find a list of new features here. You can find a detailed description of the steps required or recommend to upgrade existingfeatures of Galaxy Communicator here.

You can also inspect release notes for previous major version.

Release notes for version 2.x Release notes for version 1.x

January 2002: Version 3.3

Core distribution

Enhancement . The distribution can now be configured for compilation using GNU configure exclusively, although the old method still works. All references to specificplatform dependencies are now in include/galaxy/sysdep.h , and many files andheaders throughout the distribution have been modified. Old cruft relating to platforms we nolonger have access to for testing (e.g., DEC Alpha OSF/1 and HP/UX) has been removed. Enhancement . As a byproduct of the new configuration process, we can easily test andcompile on three additional platforms: SGI IRIX, Intel FreeBSD, and MacOS X/Darwin. Enhancement . New Builtin server functions hub_set_verbosity, hub_raise_error. Enhancement . A couple of unnecessary bottlenecks in the transport layer have beenremoved, resulting in significantly higher message throughput. Enhancement . We have confirmed that we can cross-compile the Galaxy Communicatorinfrastructure for Linux on the iPaq. Enhancement . We have released experimental support for external Hub visualizationservers. These servers can subscribe to special message streams which indicate the statusof the Hub. See the -gui command line flag for the Hub executable. Enhancement . New Hub command line flag -suppress_pacifier . Clarification . We’ve clarified the precedence when inconsistent listen status flags are set. Clarification . We’ve clarified in the non-MIT makefile documentation that the Communicatorlibraries should follow the source or object files in your link line. Clarification . We’ve clarified that the use of #ifndef to ensure that header files are only

655

loaded once is counterproductive in the case of server declaration files. Bug fix . To support BSD operating systems, we identified and fixed a bug where BSDplatforms couldn’t determine their own IP address using the existing code. Bug fix . We finally changed the name of a file in the templates subdirectory whose namediffered from another file only by capitalization. Bug fix . A bug in the Hub was identified and fixed where sessions could have been freedprematurely if multiple servers were locked to it. Bug fix . A bug in the Hub was identified and fixed where server messages were sometimesnot being freed, and one case where the server messages were potentially referenced after theywere freed. Bug fix . A bug in the Hub was identified and fixed where messages queued for a server inthe Hub were not all getting sent if some of them didn’t require a reply. Bug fix . A bug in the Hub was identified and fixed where round-trip messages from a serverwhich were processed in scriptless mode by the Hub were never returning when the destinationserver did not provide a reply. Bug fix . A bug in the core C library was identified and fixed where threaded servers couldpotentially fail to free a mutex if the server was contacting the Hub. Incompatible change . As a consequence of the new configuration process, theexperimental cross-compilation support has been completely changed.

MIT contributions

Enhancement . We have added the capability in the Hub scripting language of monitoring aserver for a new message and interpreting it as a reply. We call this capability Hub continuations. It is available as scripting language directives and as a Builtin function. Bug fix . Fixed a bug where the Builtin function wasn’t being declared if there were noprograms or messages.

MITRE contributions

Enhancement . It is now possible to compile and use the Python process monitor withoutcompiling the Python bindings. Enhancement . The Java bindings now support the same Hub connection policy flags as theother language bindings. Enhancement . The cleanup method has been added to the MainServer class, allowingsubclasses to run arbitrary code before the server shuts down. Enhancement . The process monitor now makes arguments passed to its configuration filesavailable by name as well as position. Bug fix . While we are no longer officially maintaining the XML log manipulation code, wehave addressed a bug where partial match information in the XML annotation rule matcherwas being preserved by accident. We do not have a test suite for this section of code, so wecan’t be absolutely sure that we haven’t broken something else, but the fix is very local andwe can’t imagine what other effects it might have beyond addressing this bug. Bug fix . When a Java server that is acting only as a Hub client starts an outgoing broker,the listener thread that is started as a result no longer accepts regular connections from the

656

Hub. Incompatible change . In MainServer, a number of methods have been renamed:setActsAsClient is now setActsAsHubClient, isClient is now isHubClient, setActsAsListeneris now setActsAsHubListener, isListener is now isHubListener, startListenerThread is nowstartHubListenerThread (and it was changed from protected to private), and startClientThread isnow startHubClientThread (and it was also changed from protected to private). Finally, the MainServer connection bitmask flag values (GAL_CONNECTION_LISTENER, GAL_BROKER_LISTENER, GAL_HUB_CLIENT, GAL_SERVER_TYPE_MASK) have beenmade protected (were public). Incompatible change . In order to establish the Open Source toolkit, all servers previouslyin contrib/MITRE , along with their documentation and supporting examples, have beenremoved from the distribution.

Known bugsTwo known bugs from 3.2 remain:

In rare cases, if two broker servers attempt to contact each other simultaneously, adeadlock will arise. This bug is fixable, but would require some serious internalreorganization. (Observed 3/01) Line breaks inside Galaxy Communicator list objects in program files do not work.(Observed 2/01)

We judge that fixing these two bugs is not a high priority at the moment. All bugs which havebeen reported since the release of 3.1 have been fixed.

October 2001: Version 3.2

Core distribution

Enhancement . The documentation has been completely reorganized into four sections:tutorial, advanced topics, reference and administrative information. The tutorial material ismostly new, and represents a completely updated version of the original Communicatortraining course (previously distributed separately). This tutorial is designed to be self-guided,and is generously illustrated. Enhancement . The flags which control server behavior when they contact the Hub didn’thave a value which allowed the user to specify that the server should shut down if it couldn’tcontact the Hub in the first place. In addition, there were some intuitively mnemonic namesand values (for masks, for instance) which were missing. These problems have beenremedied (not in Java yet). Enhancement . For advanced programmers, there is now an undocumented capability ofoverriding the default method for selecting dispatch functions. We use this to allow thePython bindings to use native dictionary lookup as its selection method, which also allowsus to defeat the selection method entirely for the new Python-based unit tester. Enhancement . For 3.2, we will be using a different, better tool to generate the PDF version

657

of the documentation. Due to expanded documentation and illustrations, this document hasgrown to 10MB, and will be distributed separately. Enhancement . Added the functions GalSS_EnvReturnRequired and GalUtil_SetVerbose. Bug fix . In 3.1, service providers in the Hub which were providers for multiple service typeswere being added multiple times to the global list of service providers, leading to a segmentationfault when the provider disconnected. This has been fixed. Bug fix . In 3.1, session locks were being misinterpreted by the Hub; restrictions onmessage received were being interpreted as restrictions on messages sent, and vice versa. Thishas been fixed. Bug fix . Because of a flaw in the Makefile templates, it was not possible to maintainsimultaneously different settings for whether to compile shared libraries on different platforms.This bug has been fixed. Bug fix . MIT discovered and helped fix a bug where under unusual circumstances involvingqueued brokers in C which were not activated, the brokers would loop forever. Bug fix . MIT discovered and fixed a bug where read streams associated with strings wouldrewind incorrectly under unusual circumstances. Bug fix . We have finally fixed a longstanding bug where program files with Unix lineterminations could not be read properly on Windows. Bug fix . We have discovered and fixed an arcane Hub bug where extraneous unexpectedreply messages from asynchronous Hub calls could destroy tokens by accident. Bug fix . With MIT’s help, we have discovered and fixed a bug where server and sessionlock release requests were not being transmitted from server to Hub. Bug fix . We have fixed a bug where older servers which use bare connections to writemessages to the Hub couldn’t control server and session locking correctly. Incompatible change . The way service provider eligibility in the Hub is determined haschanged slightly. In particular, in 3.1, service providers which are the only provider for aparticular service type were being used because they were the only available server, even whenthey were locked to other sessions. Service providers which are locked to sessions can now beused only by that session, without exception. Incompatible change . The signatures of the undocumented functionsGal_FindDispatchFunctionEntry and Gal_EncodeDispatchFnSignatures have changed. Incompatible change . The function Gal_GetListValue has been deprecated, and will beremoved in the next major release.

MIT contributions

Bug fix . In 3.1, any operations referenced in Hub program rules which had no explicitservice type (e.g., Parse instead of Parser.Parse ) were erroneously assigned a servicetype at Hub startup time (in general, the last service type mentioned which supported theoperation). As a result, if multiple service types supported the same operation, only oneservice type would ever be considered, even though the script writer intended the referenceto be ambiguous. This bug has been fixed, along with a number of related bugs which hadbeen hidden by this bug. However, if only one service type supports a given operation, thatservice type is still explicitly assigned at read time to operations without a service type. Enhancement . MIT has provided minor enhancements to its dialogue tools.

658

MITRE contributions

Enhancement . In support of the new tutorial, the toy travel demo has been completelyredesigned and reimplemented, so that the code as well as the program file arepedagogically appropriate. Enhancement . The process monitor now accepts a configuration file as an argument, as analternative to the sequence of flags it has previously accepted (and still accepts). Thisfunctionality makes visible some process monitor extensions we had previously only usedinternally in our test harness. This enhancement makes it significantly easier to maintainand start up examples in the tutorial, as well as making it simpler to port the process monitorto Windows. Enhancement . In support of the new tutorial, the unit tester has been completelyredesigned and reimplemented. It now allows you to connect to a Hub pretending to be aserver, or to a server pretending to be a Hub. It also provides a graphical interface to allowyou to respond to incoming messages and to send new messages of your own, and keeps amouseable history of frames for you to select from if you choose. Enhancement . We have augmented the audio example and brokering documentation withexamples of outbound streaming broker data. Bug fix . In the Python bindings, the equivalent of GalIO_ClientConnect did not supportdisconnect callbacks appropriately. This has been fixed. Bug fix . In the Python bindings, it was not possible to reset the verbosity level for theunderlying C library after startup. This has been fixed.

Known bugsTwo known bugs from 3.1 remain:

In rare cases, if two broker servers attempt to contact each other simultaneously, adeadlock will arise. This bug is fixable, but would require some serious internalreorganization. (Observed 3/01) Line breaks inside Galaxy Communicator list objects in program files do not work.(Observed 2/01)

We judge that fixing these two bugs is not a high priority at the moment. All bugs which havebeen reported since the release of 3.1 have been fixed.

June 2001: Version 3.1Core distribution MIT contributions MITRE contributions Known bugs

659

Core distribution

Enhancement . Perl is no longer required by the Communicator infrastructure. Enhancement . New API functions Gal_NumNonNullProperties, Gal_PrintEntity Enhancement . The Makefile infrastructure now includes experimental Unix support for cross-compilation. Enhancement . The nightmarishly complex process of embedding Communicator in externalmain loops has been vastly simplified. Enhancement . The Hub builtin server now has a number of new functions: call_program,which eliminates the idiosyncracies in invoking subprograms and supports returns fromsubprograms; hub_available_servers, which provides access to the list of service types thatare currently enabled; hub_token_owner, which provides the service type of the originatingserver for the current token; and hub_pause, which introduces a pause after each Hubmessage dispatch for demo purposes. Incompatible change . The default verbosity level is now 3; it was previously undefined,which was equivalent to 6 (the most verbose). Clarification . If you use Python, it must be configured with Tkinter support. Clarification . The function Gal_NumProperties counts properties with NULL values. Usethe new Gal_NumNonNullProperties to skip NULL-valued properties. Clarification . For reasons of backward compatibility, the keys :pred and :PRED are specialto Gal_ReadFrameFromString and its relatives, and should be avoided by sites other than MIT. Clarification . The 3.1 distribution also contains undocumented experimental hooks forsupporting an external Hub visualization tool. These hooks are disabled because thefunctionality is only partially implemented. Bug fix . In 3.0, if a frame contained a key-value pair with a NULL value, the property wascounted but not encoded in the XDR transport, causing the number of properties in theframe to be encoded incorrectly in the XDR transport. This bug is now fixed. Bug fix . There is now a more reliable way of avoiding name collisions with OS-level Clibrary functions which only exist on a subset of supported platforms. Bug fix . In 3.0, the automatically generated Makefile dependency files incorrectly expandedthe ARCHOS Makefile variable, partially invalidating the dependencies on platforms otherthan the one they were generated on. This bug has been fixed, along with a number of otherlurking bugs in the dependency generation. Bug fix . In 3.0, tokens were not consistently being returned to their originating servers whenthey died, due to poor code organization. This facility has been completely reorganized. Bug fix . In 3.0, a null OPERATIONS: directive in service type declarations in the Hubprogram file could lead to apparently random server choices for operations, includingservers which did not support the operation. This bug has been fixed. Bug fix . In 3.0, servers which were configured to shut down after their connection to theHub was lost were using the wrong facility to shut down, with the result that the wrong thingwas happening in external main loops. This bug has been fixed. Bug fix . In 3.0, there were a couple cases where sessions were not being createdautomatically in the Hub. This bug has been fixed. Bug fix . In 3.0, under some circumstances sessions could be destroyed without their

660

corresponding database history being freed, leading to potential memory leaks. This bughas been fixed.

MIT contributions

Enhancement . New Hub scripting IGNORE: directive, IN comparison, Enhancement . The Hub scripting global namespace is now writable, and there is a newvalue for the MODE: directive called global_read_only which will write-lock the namespace. Enhancement . The current logfile name is now available as a session key in the Hub scripting. Bug fix . The DEL: Hub scripting directive seg faulted when only simple keywords were listed.

MITRE contributions

Enhancement . Running Python servers in debug mode no longer causes an exit when anerror is encountered, but rather prints out better debugging information. Enhancement . The process monitor now takes two new arguments: --master_title , toset the title of the process monitor window, and --clear_history , to automatically clearthe history of each pane after a given number of lines are printed. Enhancement . The following methods have been added to the Java bindings:

galaxy.server.MainServer setVerbosityLevel (public) getVerbosityLevel (public) setHubContactPolicy (public) getHubContactPolicy (public)

Enhancement . Added initial support for the -verbosity command line argument in the Java bindings. Clarification . We have determined that the Python bindings work with Python 2.0. Incompatible change . The Java method MainServer.contactHub has been modified. It nolonger takes a Server argument. Rather, if the Hub contact policy argument is set to -1, thedefault policy of the invoked MainServer is used. Bug fix . "make distclean" at the toplevel no longer operates on contrib/MITRE twice. Bug fix . In 3.0, the Allegro and Python bindings were missing the functionality provided by GalSS_EnvGetSessionID and related functions. This has been fixed. Bug fix . In the Python Tk bindings, the application wasn’t exiting appropriately when theserver exited. This is now fixed. Bug fix . The PostGres wrapper now handles SQL queries other than insert and delete appropriately.

661

Known bugs

In rare cases, if two broker servers attempt to contact each other simultaneously, adeadlock will arise. This bug is fixable, but would require some serious internalreorganization. (Observed 3/01) If you ask the Hub on NT to read a program file which was written on Unix, the Hub will faildue to the difference in line breaks between platforms. This bug has apparently been lurkingfor quite a while. (Observed 3/01) Line breaks inside Galaxy Communicator list objects in program files do not work.(Observed 2/01) The tool which automatically generates the PDF documentation from the HTMLdocumentation does a good but not great job, and in some cases the formatting is quite odd.(Observed 3/01)

January 2001: Version 3.0Core distribution MIT contributions MITRE contributions Known bugs Wish list

Core distribution

We have documented and radically improved session management support. This includes abetter defined use of call environments. We have introduced continuations to allow better management of asynchronouscommunication with the Hub. We have introduced server locations files to allow dynamic control of the location of Huband server listeners. We have documented the GC_VERSION header definition, which allows you toconditionalize your servers against different versions of the Galaxy Communicator distribution. The timed task facilities now allow you to register error streams as well as input and outputstreams for monitoring, and also give you access to the reasons the task is being run. Seethe functions Gal_AddTaskExtended and Gal_TaskPkgRunReasons. Due to this change,we now recommend that programmers upgrade from the "simple" timed task API. See the upgrade notes. The core frame library now includes support for dynamically expandable lists and binaryobjects. See the functions Gal_CreateListObject and Gal_CreateBinaryObject. The distribution now includes versions of object creation functions which allow explicitcontrol of memory management, as well as creation functions for all the array types whichcould only be brokered in previous releases. As a side benefit of using the C library as the basis of the Python and Allegro language

662

bindings, the API for embedding Communicator servers in other main loops has beentremendously improved. You are strongly encouraged to take advantage of these modifications.See the upgrade notes. Thread startup for timed tasks now works on Windows NT. The syntax of Hub scripts has been extended in a number of powerful and consistent ways.It is now possible to refer to the memory states of sessions, etc. in a transparent way. Inaddition, the Hub now organizes its known servers into service types and service providers,which simplifies and clarifies the process of selecting a server to deliver a message to. Finally,the Builtin server no longer needs to be declared. See the extensively revised scripting documentation for details. Both brokering and message traffic have been reimplemented using the XDR dataencoding. There is no longer any distinction between the frame types which can be brokered andthe frame types which can be included in frames. In addition, brokering floats between platformsfinally works. Also, there is no longer any possibility of confusing values of the :pred keywith elements of the list of predicates in a frame. The reinitialize dispatch function can now be called as a normal dispatch function. It is now possible to build frames from string representations with embedded variables. Seethe function Gal_VAReadVarFrameFromString. The Hub now informs the server whether it needs a reply or not, and the server bindingsnow return a pacifier reply when the dispatch function doesn’t return a frame. This means theHub will never block waiting on a reply from a server. Servers now provide a unique index for each server-to-server request via GalSS_EnvDispatchFrame, and appropriately postpone new requests while they’re waitingfor a response. This more robust behavior is superfluous under normal circumstances, but iscrucial for proper functioning when two threads are using the same connection. A bug in internal socket management in 2.1 has been fixed, in which mutex protection forsockets was not fine-grained enough. This caused multiple-connection servers to deadlock inthreaded mode in 2.1. Naming conventions for non-public functions in the core library have been cleaned up and simplified. We have clarified in the documentation that incoming broker callbacks have responsibilityfor freeing the data they’re passed. See the upgrade notes. We have cleaned up the event-driven programming and callback data storage models forservers, connections and brokers considerably. Some of this cleanup has resulted inrecommended upgrades for incoming brokering. See the upgrade notes and the newsection on event-driven programming. We’ve done some internal cleanup and reorganization of the headers. This should betransparent to the programmer. The distribution now includes a PDF version of the documentation. Gal_ObjectEqual,Gal_MatchFrame and Gal_ObjectCaseEqual now work on list objects. A long-standing bug where error keys were not reliably logged in the Hub has been fixed. Deprecation . It’s become clear that we can’t keep changing the signature of GalSS_InitializeServerToplevel every time we add a new command line argument to theserver bindings. The following functions have been deprecated.

GalSS_InitializeServerToplevel

663

GalSS_CmdlineInitializeServer GalSS_InitializeServer GalSS_InitializeServerFromServerArgs GalSS_ExtractServerArgs GalSS_RunServer

Deprecation . The frame key :hub_new_session in new messages currently forces theBuiltin server function new_session to be called. This mechanism will be replaced in asubsequent release. The frame key :session_id can currently be used to access andmodify the session associated with a message. This behavior is now deprecated; use GalSS_EnvGetSessionID and GalSS_EnvUpdateSessionID instead. Incompatible change . As part of cleaning up the callback data storage model, thesignature of GalIO_SetServerData has changed. Incompatible change . Dispatch server support has been removed from the distribution. Incompatible change . The following deprecated functions have been removed:

sls_pp_frame Gal_GetValue sls_cpp_frame sls_pp_object sls_pr_object sls_cpp_object Gal_UnwrapValue Gal_Value Gal_ValueWarn Gal_ValueWarnWithKey Gal_ValueSilent GalIO_SetServerDisconnectCallback GalIO_ReconnectInit GalIO_ServerReconnect GalSS_SetHubDisconnect GalSS_GetServerData GalSS_GetServerName GalSS_GetServerSock GalSS_CloseServerSock GalSS_ReadFrameFromHub GalSS_FlushHubWriteQueue GalSS_CloseHubConnection GalSS_HubReconnectInit GalSS_ForceReturn GalSS_EnvForceReturn

The following functions have been moved from the core into the MIT dialogue control library,until they can be replaced by MIT:

Gal_SetServerName Gal_GetServerName

The following functions remain in the core library to support MIT dialogue control, until they

664

can be replaced by MIT: GalSS_DispatchViaHub GalSS_WriteFrameToHub

Incompatible change . Due to improvements in outgoing brokering, the function GalIO_BrokerDataOutInit has a new argument list, a number of new functions have beenadded, and the function GalIO_FrameGetBrokerPort has been removed. See the upgrade notes. Incompatible change . Due to a number of protocol changes to support better handshakingand improved outgoing brokering, the 3.0 distribution is not on-the-wire compatible with 2.x.Some of these changes involve modified handshakes and message encoding, and othersinvolve some modifications to the distinguished keys used in the protocol. Incompatible change . Due to the new reorganization in the Hub of servers into servicetypes and providers, the -locations command line argument no longer overwrites CLIENT_PORT:directives. Incompatible change . The flags controlling server listener status, first introduced in version2.1, have been changed slightly in connection with upgraded outgoing brokering. See the upgrade notes. Incompatible change . In connection with improving session management, the semantics of GalSS_EnvError and GalSS_EnvDestroyToken have changed slightly; the return value froma dispatch function which calls them is now ignored, rather than being treated as a newmessage. See the upgrade notes. Incompatible change . Due to the revised handshaking protocol, broker clients can nolonger contact a broker server running in the same server as the broker client innon-threaded mode. This restriction can be fixed if it turns out to be important. Incompatible change . The internal reader functions Gal_ReadObjectLine,Gal_ReadBinary, Gal_ReadString, Gal_ReadFrame, Gal_ReadList, and Gal_ReadObjecthave been made static and are no longer externally visible. These functions were neverintended to be used by the programmer, and we doubt that anyone uses them. Incompatible change . As a result of cleaning up the internal processing in the Hub, anumber of functions in the API now have slightly different function signatures. Thesefunctions should only have been used internally anyway, so no user programs should beaffected. The functions are:

Gal_ReadProgramLine Gal_PrintTests Gal_PrintCondition Gal_TestCondition Gal_NewCondition Gal_ReadProgramLine

In addition, the types Gal_FileStruct and Gal_ConditionStruct no longer have the aliasesFileStruct and ConditionStruct. Incompatible change . The following utility functions have been removed because they arenot used, and were only intended to be used internally:

__GalUtil_MaybeByteSwapVector __GalUtil_LittleEndianMaybeByteSwapVector GalUtil_SockReadIntWithTimeout

665

GalUtil_SockWriteFloat GalUtil_SockWriteAtom GalUtil_SockReadShortArray GalUtil_SockReadAtomWithTimeout GalUtil_SockReadInt GalUtil_SockReadFloat GalUtil_SockWriteInt GalUtil_SockAllocReadString GalUtil_SockReadString GalUtil_SockReadAtom

Incompatible change . The deprecated construct using the :timestamp key to indicate thetimestamping of a bare key in Hub logging has been removed.

MIT contributions

The Hub program verifier now allows you to print the result to standard output instead of a file.

MITRE contributions

It is now possible to conditionalize Allegro and Python servers using the *features* list inthe same way that C servers can be conditionalized using the #define for GC_VERSION.Like C, Allegro supports read/compile time conditionalization via features; the Allegrofeature for 3.0 is :gc-version-3 . Python only supports runtime conditionalization; thePython feature is Galaxy.GC_VERSION , the value of which is identical to the C value. Youcan distinguish in Python between 3.0 and pre-3.0 libraries by checking for the presence of GC_VERSION in the Galaxy module. Java is a compiled language which has nocompile-time conditionalizations that we can find, so this feature is unfortunately unavailablein Java. In order to better ensure consistency across programming languages, the Allegro andPython bindings now rely on the C library. This change should be invisible to the serverdeveloper. We have preserved Java as a natively implemented binding. As a result of relying on the C library for Allegro bindings (see below), the longstanding bugwhere Allegro servers don’t allow their sockets to be reused promptly has been fixed. The MITRE toy travel demo has been upgraded to use all the current session management utilities. We’ve made significant improvements to the MITRE PostGres database wrapper. This distribution contains the very latest XML log analysis tools distributed in 2.0p3 and 2.1p1. We have fixed an insidious bug in the stdin polling utility which manifested itself during theupgrade to 3.0. In 2.1, there was an inconsistency between the behavior of programming language bindingsin the listener-in-Hub case and the C library, where the programming language bindingswould shut down after the last Hub disconnected. As a result of relying on the C library forPython and Allegro bindings, this inconsistency has been resolved.

666

Brokering of floats now works correctly in Allegro. It is now possible to initialize Python and Allegro server objects with keyword arguments aswell as command line arguments. The following classes have been added to the Java bindings:

galaxy.server.Environment galaxy.lang.PostponeMessage galaxy.server.Continuation galaxy.server.ServerArgParser galaxy.server.ServerCreationThread (internal class that Galaxy Communicatordevelopers do not use directly) galaxy.server.ListenerThread (internal class that Galaxy Communicator developers donot use directly) galaxy.server.ClientThread (internal class that Galaxy Communicator developers do notuse directly) galaxy.server.FrameDispatcher (internal class that Galaxy Communicator developersdo not use directly) galaxy.server.ContinuationData (internal class that Galaxy Communicator developersdo not use directly) galaxy.io.XdrInBuffer (internal class that Galaxy Communicator developers do not use directly) galaxy.io.XdrOutBuffer (internal class that Galaxy Communicator developers do not use directly) galaxy.util.FifoMutex (internal class that Galaxy Communicator developers do not use directly) galaxy.util.HubContactInfo (internal class that Galaxy Communicator developers do notuse directly)

The following methods have been added to the Java bindings: galaxy.server.MainServer

MainServer(java.lang.String name) (public) MainServer(java.lang.String name, java.lang.String[] args) (public) MainServer(java.lang.String.name, java.lang.String[] args, java.lang.String helpText) (public) setActsAsListener (public) isListener (public) setActsAsClient (public) isClient (public) getSessionId (public) getMaxConnections (public) setMaxConnections (public) contactHub (public) listenerHasPort (protected) startListenerThread (protected) startClientThread (protected) registerOutBroker (protected)

667

unregisterOutBroker (protected) galaxy.server.Server

getCurrentEnvironment (public) getCopyOfCurrentEnvironment (public) getServerProperties (public) setServerProperties (public) deleteServerProperties (public) addServiceType (public) getServiceTypes (public) getMainServer (public) getSessionId (public) setHubContactPolicy (public) getHubContactPolicy (public) cleanup (protected) getContinuationData (protected) addContinuationData (protected) registerOutBroker (protected) unregisterOutBroker (protected)

galaxy.server.DataInBroker DataInBroker(galaxy.server.Server, java.lang.String, int, galaxy.langGFrame) (public) stop (public) registerWithServer (public) unregisterFromServer (public) start (public) isStopped (public) receivedList (public abstract) receivedInteger (public abstract) receivedFloat (public abstract) receivedSymbol (public abstract) cleanup (protected) disconnectReceived (protected) abortReceived (protected)

galaxy.server.DataOutBroker getCallIdIndex (public static) getCallId (public) isExpired (public) stop (public) expire (public) populateFrame (public) isStopped (public)

galaxy.io.GalaxyInputStream tryReadInt16 (public)

668

tryReadInt32 (public) tryReadInt64 (public) tryReadFloat32 (public) tryReadFloat64 (public) tryReadBinary (public) tryReadString (public) tryReadFrame (public) tryReadMessage (public) tryReadObject (public) readList (public) tryReadList (public) readInteger (public) tryReadInteger (public) readFloat (public) tryReadFloat (public) readSymbol (public) tryReadSymbol (public) getMutex (public)

galaxy.lang.GFrame GFrame(galaxy.lang.Symbol name, int frameType) (public) getFrameType (public) setFrameType (public) encodeFrame (public) encodeObject (public static) decodeFrame(public static) decodeObject (public static)

galaxy.lang.GBinary GBinary(byte[] data, boolean makeCopy) (public)

galaxy.lang.Float32 Float32(float[] floatArray) (public) Float32(float[] floatArray, boolean makeCopy) (public) getFloatArray (public) append (public) getSize (public) getBytes (public)

galaxy.lang.Float64 Float64(double[] doubleArray) (public) Float64(double[] doubleArray, boolean makeCopy) (public) getDoubleArray (public) append (public) getSize (public) getBytes (public)

galaxy.lang.Int16

669

Int16(short[] shortArray) (public) Int16(short[] shortArray, boolean makeCopy) (public) getShortArray (public) append (public) getSize (public) getBytes (public)

galaxy.lang.Int32 Int32(int[] intArray) (public) Int32(int[] intArray, boolean makeCopy) (public) getIntArray (public) append (public) getSize (public) getBytes (public)

galaxy.lang.Int64 Int64(long[] longArray) (public) Int64(long[] longArray, boolean makeCopy) (public) getLongArray (public) append (public) getSize (public) getBytes (public)

The following methods are deprecated as of 3.0: galaxy.server.MainServer

getRequirePort (use galaxy.server.MainServer.isRequiredPort) galaxy.server.Server

Server(galaxy.server.MainServer, java.net.Socket, java.lang.String, java.lang.String) (use galaxy.server.Server(galaxy.server.MainServer, java.net.Socket) instead) writeMessage (use galaxy.server.Environment.error, galaxy.server.Environment.error (second version), galaxy.server.Environment.writeFrame, galaxy.server.Environment.destroyToken,or galaxy.server.Environment.reply instead) writeErrorMessage (use galaxy.server.Environment.error or galaxy.server.Environment.error (second version) instead) writeFrame (use galaxy.server.Environment.writeFrame instead) writeDestroyMessage (use galaxy.server.Environment.destroyToken instead) dispatchFrame (use galaxy.server.Environment.dispatchFrame instead) register (use galaxy.server.Server.registerOutBroker instead) unregister (use galaxy.server.Server.unregisterOutBroker instead)

galaxy.io.GalaxyOutputStream write(galaxy.lang.GFrame, int) (use galaxy.io.GalaxyOutputStream.writeObject instead) write(java.lang.String, int) (use galaxy.io.GalaxyOutputStream.writeObjectinstead) write(galaxy.lang.BinaryObject, int) (use galaxy.io.GalaxyOutputStream.writeObject instead)

670

writeFrame (use galaxy.io.GalaxyOutputStream.writeObject instead) galaxy.server.DataOutBroker

DataOutBroker(galaxy.server.Server, int, galaxy.lang.GFrame, boolean) (use DataOutBroker(galaxy.server.Server, int, int) instead) DataOutBroker(galaxy.server.Server, int, galaxy.lang.GFrame) (use DataOutBroker(galaxy.server.Server, int, int) instead)

galaxy.lang.Float32 Float32(byte[] data) toFloatArray (use galaxy.lang.Float32.getFloatArray in conjunction with Float32(float[] floatArray))

galaxy.lang.Float64 Float64(byte[] data) toDoubleArray (use galaxy.lang.Float64.getDoubleArray in conjunction with Float64(double[] doubleArray))

galaxy.lang.Int16 Int16(byte[] data) toShortArray (use galaxy.lang.Int16.getShortArray in conjunction with Int16(short[] shortArray))

galaxy.lang.Int32 Int32(byte[] data) toIntArray (use galaxy.lang.Int32.getIntArray in conjunction with Int32(int[] intArray))

galaxy.lang.Int64 Int64(byte[] data) toLongArray (use galaxy.lang.Int64.getLongArray in conjunction with Int64(long[] longArray))

galaxy.lang.BinaryObject has been deprecated as of 3.0. As a result, the byte[] -based constructors for the array objects have also been deprecated as have thevarious to<data type>Array() methods. The default constructors for the array objects havebeen changed to create empty arrays of the appropriate type (e.g., galaxy.lang.Int32objects have arrays of ints). New local append methods for the array objects have alsobeen added to accept the appropriate data types, getBytes returns the array as an arrayof bytes, and getSize will return the length of the data type array (if the object wasinitialized with a byte array, the length of that array is returned). Finally, the use of these BinaryObject methods on array objects are deprecated:

getBytes toString toEncodedString toFormattedString

The Java class galaxy.lang.GBinary has taken on galaxy.lang.BinaryObject ’s"binary data" role. In particular, GBinary now has these local methods:

getBytes (public) getSize (public) toString (public)

671

toEncodedString (public) toFormattedString (public) append (public) readBinaryFile (static public)

In the Java bindings, the environment object has been moved out of galaxy.server.Server and implemented as the object galaxy.server.Environment . A Java server can now postpone a reply to a Hub message. See the Java bindings documentation for details. See the Java bindings documentation for the new, preferred way to send new messagesand replies to the Hub from a server or incoming broker. galaxy.server.MainServer has a new constructor that takes in a server name and thearray of command line arguments. It can initialize itself with the values of -port, -assert, -maxconns, -contact_hub, -session_id, and -validate. This simplifies the development ofservers that do not rely on a user interface (e.g., as provided by galaxy.server.ServerUI ). The Java class galaxy.util.ArgParser has a new method getStringArg foraccessing the string value of a command line argument (if appropriate). Additional frame key type definitions have been added to the Java class galaxy.lang.GFrame . There is now support for the -maxconns command line argument in the Java bindings. Also,if both the -port and -contact_hub command line arguments are specified, the server will actas a listener AND Hub client. Finally, if the -assert command line argument is specified, and -port has also been specified, the server will fail to establish a listener if it cannot obtain thespecified port (or the default port if the -port argument is not used with -assert ). Enabled support for appending to a log file in the Java bindings (this had beenunimplemented previously). Added support for the -server_locations_file command line argument in the Java bindings. Incompatible change . The Java bindings no longer treat the colon (:) character in anyspecial way when processing frames. In particular, colons are no longer prepended to thenames of keys and frame names are no longer prevented from beginning with a colon. Incompatible change . As a result of relying on the C library for Python and Allegrobindings, the installation procedure is slightly different. The Allegro bindings now requireAllegro 5.0 or later, and the Python bindings require access to the Python header files. Seethe installation notes. Incompatible change . As a result of improving outgoing brokering, the API for setting up anoutgoing broker in the various programming language bindings has changed, as well as theargument list for MGal_AddOutgoingBrokering(). See the upgrade notes. Incompatible change . For consistency with the rest of the system, the values controlling"raw" vs. pretty printing in the Allegro bindings have been changed from :pr and :pp to galaxy::*GAL-PR-PRINT* and galaxy::*GAL-PP-PRINT* . Incompatible change . As a result of relying on the C library for Python bindings, themethod ReadFrameFromString() of the Galaxy.Frame class has been removed. The Galaxy.OPr() function no longer accepts the indent keyword. A number of attributes ofthe GalaxyIO.Server class have been replaced with methods to access the underlying C

672

functionality: the port attribute is now the ListenPort() method the validate attribute is now the Validating() and EnableValidation() methods the server_name attribute is now the ServerName() method the max_connections attribute is now the MaxConnections() method

Finally, as part of the deprecation process, the Galaxy.ValueWarn() function has beenremoved. See the upgrade notes and the implementation of the Python bindings for more details. Incompatible change . The (undocumented) TkMGalaxy module has been cleaned up, andsome non-central functionality has been moved elsewhere. See contrib/MITRE/servers/src/ui/py_gui_ui.py . If you import this module, yourcode will need to be updated. Incompatible change . In support of more robust session support, and to reduce confusion,the Python GalaxyIO.Connection class has finally been renamed GalaxyIO.CallEnvironment , which has been its actual function since GalaxyCommunicator 2.0. See the upgrade notes. Incompatible change . Looking forward to better Python support on Windows NT, we havesimplified and changed the way Python servers are configured to get access to the corePython libraries. See the upgrade notes. Incompatible change . See the upgrade notes for changes to the procedure for starting galaxy.server.DataInBroker threads. Incompatible change . See the upgrade notes for details on new methods that must beimplemented by subclasses of galaxy.server.DataInBroker . Incompatible change . See the upgrade notes for a list of methods removed from theinterface of galaxy.server.DataOutBroker . Incompatible change . See the upgrade notes for changes in thread management in theJava bindings. Incompatible change . See the upgrade notes for changes to the command line argumentsused in the Java bindings. Incompatible change . See the upgrade notes for other miscellaneous changes to the Java bindings.

Known bugs

In rare cases, if two broker servers attempt to contact each other simultaneously, adeadlock will arise. This bug is fixable, but would require some serious internal reorganization. If you ask the Hub on NT to read a program file which was written on Unix, the Hub will faildue to the difference in line breaks between platforms. This bug has apparently been lurkingfor quite a while. Some of the specialized Makefile behavior seems to exhibit some small bugs (for instance,"make distclean" appears to do contrib/MITRE twice). The tool which automatically generates the PDF documentation from the HTMLdocumentation does a good but not great job, and in some cases the formatting is quite odd.

673

Wish listThis list contains some of the features we had hoped to include in version 3.0 but didn’t havetime for.

The Hub program file reader does not yet warn developers when they appear to be trying touse extended syntax features in non-extended mode. The Hub session history DB has not been incorporated into the extended syntax paradigm. The dispatch_token Builtin dispatch function is not capable of returning the result state. The extended syntax mode in Hub programs does not implement things like explicitreference to frame types and names, attribute paths through frames, etc., although theinfrastructure to support for these references is already present. Server locations files do not support servers contacting multiple Hubs. It should be possible to provide a much better encapsulation of broker information. We haveincluded protocol support for a proxy data type in frames, but no API support yet. We had hoped to provide real Python support for NT. We had hoped to improve the PostGres wrapper in a number of ways we haven’t had achance to address yet. We have not yet updated the correspondences between C and Python/Allegro API calls.

Last modified January 18, 2002

674

Release Notes for Version 2.x

This document briefly lists all the changes from one Galaxy Communicator distribution to thenext for versions 2.x.

Version 2.x is not the current version of Galaxy Communicator. For the current distribution, youcan find a list of new features here. You can find a detailed description of the steps required orrecommend to upgrade existing features of Galaxy Communicator here.

August 2000: Version 2.1

Core distribution

Shared object compilation is now supported. See the installation notes. As a result, you mayneed to do something slightly different if you’re not using Communicator Makefile templates.See the upgrade notes. The distribution is now provided under an open source license. All copyright notices havebeen updated. We have dramatically simplified and reorganized the internal Makefile templates. Thischange should be completely transparent to existing applications. The dispatch server functionality is now deprecated, and should be replaced by the new listener-in-Hub capability. Dispatch servers will probably be removed in the 3.0 distribution. It is now possible to control the built-in command line argument parsing in a much morefine-grained manner when you’re writing your own main (). See GalSS_DefaultServerArgs()and the upgrade notes. A number of utility functions inherited from the MIT sls_util library which are not used inthe core distribution have been removed. Thanks to MIT, the Hub logger now uses an API call GalUtil_Mkdirp instead of a shell call to mkdir . MIT fixed a bug in #include ordering which was leading to incorrect macro expansions. MIT made changes to the socket code which ensure that servers on machines which areassigned IP addresses via DHCP will work correctly. AT&T helped debug a problem involving the configuration of MITRE’s Windows NT libraries.Substitution functions such as close() are now defined in Win32 only if the __STDC__flag is present when the Galaxy core library is compiled; this helps ensure that thesefunctions are not defined when other NT libraries define them.

675

The timed task loop now supports the notion of idle functions, which are invoked periodicallywhen there are no tasks to fire. MIT has repaired a large number of memory leaks in the Hub executable involving framesand frame elements which were not being freed. As part of this fix, the function Gal_DelProp() now returns a value indicating whether it found an element to delete or not. An insidious bug in the Hub under which queued messages would sometimes be sent to thewrong server has now been fixed. MIT fixed a bug in the Hub under which messages in server connection output buffers weresometimes not being written out in a timely manner (or being lost). The Hub executable now takes the -verbosity command line argument. MIT added better support for handling SIGPIPE. Servers now support the notion of client data, which is data associated with a particulardispatch function rather than a particular connection. A bug in incoming brokering which was causing the finalizer not to be run in some caseshas been fixed. MIT has added support for automatic logging when the Hub loses its connection withservers and when it reestablishes it. Enhancement . As a consequence of support for the listener-in-Hub capability, the function GalSS_InitializeServer() should be replaced by GalSS_InitializeServerToplevel(). Similarly,the function GalSS_ExtractServerArgs() should be replaced by GalSS_ExtractCmdlineServerArgs(). See the upgrade notes. Incompatible change . MITRE and AT&T discovered a major logic mistake in the interactionof sockets and the timed task loop on Win32. As a result, the functionGalIO_AddTimedTaskWithIO has been removed, because the timed task loop needs todistinguish clearly between file I/O and socket I/O. Furthermore, since select() can’t poll filedescriptors on Win32, file I/O polling is not available on Win32. See the new functions Gal_AddTimedTaskWithSocketIO, Gal_AddTimedTaskWithFileIO, and the upgrade notes.

MIT contributions

The program file setting MODE: singlethread is now the default. Since MITRE has alwaysrecommended using this setting, this change should not affect any code. Incompatible change . In order to ensure the consistency and timeliness of MIT software,MIT and MITRE have agreed that MITRE will no longer distribute any MIT servers orapplication libraries. MITRE will continue to distribute MIT open source code, such as theHub scripting library. One attractive consequence of this new arrangement is that the entireGalaxy Communicator distribution is now open source. Incompatible change . MIT has clarified and slightly modified the semantics of the specialtokens destroy! and none! . See the upgrade notes. Incompatible change . The effect of SET: and DEL: directives in program file rules nolonger are transmitted in dispatch_to_main. See the upgrade notes.

676

MITRE contributions

The MITRE process monitor has been significantly enhanced for 2.0, with options forrun-time screen real estate management and built-in support for automatic restarting ofdead servers (so-called "keepalive" functionality). Thanks to a number of sites, a number of bugs in the MITRE Oracle server have been fixed. As a consequence of clarifying the distinction between file I/O and socket I/O polling in thetimed task loop, the MITRE stdin polling utility is now disabled under Win32. The double example now exemplified the new listener-in-Hub capability. Since the new listener-in-Hub capability supersedes the old dispatch function capability, thedispatch function support in the MITRE utilities has been deprecated for 2.1, to be removedin 3.0. The Java bindings have been modified to work with the new listener-in-Hub capability. Inparticular, support for the -contact_hub and -session_id command line arguments hasbeen added. Various changes to Java bindings implementation to improve thread synchronization. Incompatible change . The header update utility, which updated server headers fromversion 1.2 to version 1.3, has been removed. Incompatible change . To avoid conflicts with the new listener-in-Hub capability, the(previously undocumented) -session_id command line argument to the MITRE dispatchserver has been changed to -dispatch_session_id . Incompatible change . In the Java bindings, references to the Thread.stop() method havebeen removed from Server.java and MainServer.java. Also, the serverClassName field hasbeen added to ServerUI.java. See the Java bindings. Incompatible change . The command line of the (not very well documented) MITRE frametransducer has been changed, and the (undocumented) rule format has been completely redone. Incompatible change . The MITRE log analysis tools were completely rewritten andsignificantly enhanced for 2.0.1p2, and further bug fixes were added for 2.1. Incompatible change . The Java bindings now require the use of JDK 1.2.x (previously,some versions of JDK 1.1.x with Swing support could be used).

Full list of deprecations scheduled for 3.0Some of these deprecations have been pending since 2.0. All will be removed in 3.0.

GalSS_ReadFrameFromHub (use GalIO_CommReadFrame instead) GalSS_GetServerSock (apparently unused; use GalIO_GetCommSocket instead) GalSS_CloseServerSock (apparently unused; use GalIO_CloseCommSocket instead) GalSS_SetHubDisconnect (use GalIO_SetDisconnectCallback instead) GalSS_CloseHubConnection (use GalIO_DestroyCommStruct instead) GalSS_FlushHubWriteQueue (use GalIO_CommFlushOutQueue instead) GalSS_GetServerName (use GalIO_GetServerName and GalIO_GetCommServerName instead) The functions sls_pp_frame, sls_cpp_frame, sls_pp_object, sls_cpp_object, and

677

sls_pr_object have been deprecated in favor of GalUtil_PPFrame, GalUtil_CPPFrame, GalUtil_PPObject, GalUtil_CPPObject, and GalUtil_PrObject. GalSS_WriteFrameToHub (use GalSS_EnvWriteFrame instead) GalSS_DispatchViaHub (use GalSS_EnvDispatchFrame instead) GalSS_GetServerData (use GalIO_GetCommServerData instead) The functionality represented by GalSS_ForceReturn and GalSS_EnvForceReturn will be removed. All functionality relating to dispatch servers will be removed, including

contrib/MITRE/servers/bin/dispatcher MGal_InitializeDispatchServer MGal_DispatchServerProcessHubConn __MGal_HandleConnection MGal_FindDispatchUIArgs MGal_SetupDispatchUI MGal_DispatchUIMain GalIO_ReconnectInit GalIO_ServerReconnect GalSS_HubReconnectInit

Gal_Value Gal_ValueWarnWithKey Gal_UnwrapValue Gal_GetValue Gal_ValueWarn Gal_ValueSilent Gal_GetServerName, GalSS_GetServerName (use GalIO_GetServerName and GalIO_GetCommServerName instead) Gal_SetServerName GalSS_GetServerSock (use GalIO_GetCommSocket instead) GalSS_CloseServerSock (use GalIO_CloseCommSocket instead) GalSS_SetHubDisconnect (se GalIO_SetDisconnectCallback instead)

February 14, 2000: Version 2.0.1This distribution contains a number of bug fixes and small enhancements which resulted fromthe use of 2.0.

Core distribution

MIT identified and fixed a bug in the socket code which accidently permitted a server tocontact itself in Linux. This fix has also been made in the MIT socket library. AT&T reported a bug in which the Hub would fail ungracefully if a new token arrived with the :tidx slot already set. This bug has been fixed; :tidx values on new messages are nowignored, as they should be.

678

A bug where dispatch functions were able to overwrite administrative information has been fixed. A small typo in templates/Makefile.stub has been fixed (init.make -> archos.make ). Some small errors in the logging documentation have been fixed, and it has been slightlyreorganized for clarity. The deprecated comparison operator % in Hub scripting rules was unintentionally removedin 2.0. It has been replaced. Due to an oversight, the evaluation of parenthesized expressions in Hub program conditionswas never implemented correctly. This has been fixed. Negations of conditions grouped byparentheses also works. Thanks to MIT, the core Communicator library is now thread-safe on Windows NT. Thanks to MIT, a number of tiny inconsistencies in the NT port have been fixed. All references to ftime() have been removed from the distributed source code, and thecompatibility code kindly provided by AT&T has been removed as well. The bug where clean_headers.pl was not being specialized to platforms correctly has been fixed. MIT copyrights have been changed to open source copyrights in the appropriate places. The standard server arguments now include -verbosity , which is the command-lineequivalent of setting the GAL_VERBOSE environment variable. The function GalSS_RunServer() previously called exit() when the timed task loop exits,which defeated the opportunity for the caller to do cleanups, etc. This call to exit() has beenremoved. The equivalent problem in the threaded case has also been fixed. Thanks to MIT forpointing this out. The function Gal_TimedTasksLoopExit() was not fine-grained enough, because some of thepending tasks were being executed after this function was called. This has been fixed, and the Gal_EndTasks() has been added for the threaded case. In addition, thread cancellation isnow enabled during the execution of tasks in threaded mode, so that blocking reads can beinterrupted by Gal_EndTasks(). All Win32 projects define _REENTRANT when compiling executables and libraries. A newtoplevel workspace named GC.dsw has been created for easier maintainability.

MIT contributions

MIT identified and fixed a bug in the socket code which accidently permitted a server tocontact itself in Linux. This fix has also been made in the core socket library. MIT has provided minor updates to their GUI server and the dialogue management library. Two toplevel MIT program files, pegasus.pgm and hub.pgm, are no longer being distributedbecause they are outdated. There is a new MODE: value in the program file, pedantic , which will force the Hub to exitif it encounters a missing server or operation. Default behavior is still to warn and continue.

679

MITRE contributions

Support for MIT’s new FST recognizer has been added to the MITRE run_summit_server script. The process monitor now interpolates stdout and stderr better. The Hub scripting documentation has been updated to reflect the fact that the test :foo 3now interprets the unquoted sequence of digits as a number, not a string. Java bindings are now known to compile under JDK 1.2. The distribution now ships withpre-compiled byte code and galaxy.jar. The system is still compatible with jdk1.1.x and nowuses Swing 1.1, instead of 1.0 - the import headers have been modified to javax.swing. The XML logfile generation tool has been improved in a number of ways.

It now does the right thing with multiline string data It now recurses on lists in frames correctly It now compensates as much as possible for missing operation invocations. It has an expanded argument list, and takes either a pathname or a directory as input -note that the command line has changed substantially It now works on logs generated with Communicator versions previous to 2.0. If the user did not provide a LOG_VERSION:, there is no longer an empty versionattribute produced. It can now write to a file as well as stdout.

Fixed a number of bugs in processing message signatures in the Python and Allegro bindings. Fixed a bug in Python, Allegro and C where if reinitialize wasn’t defined, the messagesignatures weren’t reported to the Hub. MITRE has provided an expanded suite of log manipulation tools. The new -verbosity argument has been added to the Python and Allegro bindings (theJava bindings don’t have verbosity levels).

Known bugs

All 2.0 bugs remain with the exception of the clean_headers.pl bug, which has been fixed.This status report includes Java audio clipping, which is now a bit more important to fixbecause there’s genuine interest in Java audio. We’ll work with interested sites to fix this. Servers and connections do not shut down their sockets when they exit due to threadcancellation in threaded mode. This should not be too much of a problem, since theapplication should exit almost immediately afterward, but this should be cleaned up eventually.

Omissions and yet-to-be-honored requests

All 2.0 omissions remain, with the exception of progress on threading in NT. While the NTport is now thread-safe, the -thread server argument is not yet supported. IBM reported problems with logging resulting from their audio server not reporting events inreal time. CMU has requested more transparent access to session keys, and AT&T has requested

680

better support for retrieving the session’s log directory. We will soon issue a revised set of logging tools which expand the annotation rule language,provide for tab-delimited score output, introduce support for the type_dialogue_stateattribute, support interpolation of human annotations into the automated annotation file, andintroduce new scoring columns for quantity and percentage of silence within the system turn.

October 30, 1999: Version 2.0This distribution of Galaxy Communicator features a number of modifications requested by thecommunity. In addition, it reflects a reorganization of the archive to support an anticipated opensource development effort. This version of the release notes is organized by section of thedocumentation, instead of by section of the code.

"Directory organization"

All core Communicator libraries and executables are now in the toplevel src/ and include/directories. These include the libGalaxy library (an amalgamation of the three MIT GalaxyCommunicator libraries libgalaxy, libgalaxy_io and libserverstub, as well as the subset oflibsls_util which is used in Communicator) and the Hub executable (excluding MIT scripting). The MITRE documentation is now in the toplevel doc/ directory. MITRE-specific contributions besides the documentation are now in contrib/MITRE. MIT-specific servers and support libraries (dialogue control, scripting) are now incontrib/MIT, the directory structure of which corresponds to the directory structure of the MITslice of the 1.3 distribution. A range of functions originally in libgalaxy have been moved into the MIT-specific librarylibGalaxyEnhancements. These functions deal with the MIT dialogue mechanism. A range of functions originally in the Hub have been moved into the MIT-specific librarylibGalaxyHubControl. These functions deal with the MIT scripting control.

"Installing the distribution"

If you don’t set PERL in your configuration file, make will now fail clearly instead of mysteriously. Incompatible change : the core distribution now uses GNU configure, in part, to generateits configuration files for each platform. The settings you need to modify have beenaggregated into a single file, instead of the three previously required, and users now haveclearer access to a wider range of customizations.

"Running a configuration of servers"

Incompatible change : the locations of the executables and MITRE servers have changeddue to the directory reorganization. Also, the GAL_VERBOSE environment variablesupersedes the SLS_VERBOSE environment variable due to moving a copy of the MITcommand line argument parsing library into the Communicator core library.

681

"Thread notes"

The core library is now thread-safe.

"Overview: invocation, execution summary"

There is a new command line argument -locations for the Hub executable which is astring which is a space-delimited sequence of server declarations of the form <server>@<host>(:<port>) , which override the host (and perhaps port) declarations inthe Hub script. The Hub also accepts the -validate command line argument, which dynamicallyvalidates each message sent to servers, as well as the status of messages to be returned to servers. Incompatible change : the handling of the construction of the initial token and reinitialize message has been rationalized. In general, the -init command line argument now takesprecedence over INIT: and INITIAL_TOKEN:, which takes precedence over default globalinformation in the program file. In 1.3, the precedence order for the two was inconsistent. Incompatible change : the Hub no longer has a default for the program file. The -pgm_fileargument is now obligatory.

"How to write Hub scripts"

MIT program files now accept the syntax <server>@<host>(:<port>) as an argumentof the SERVER: declaration. It is no longer necessary to use DISPATCH_PORT: to identify a server as a dispatch server.You can pass the :is_dispatch key from the server to the Hub. Incompatible change : the handling of the construction of the initial token and reinitialize message has been rationalized. In general, the -init command line argument now takesprecedence over INIT: and INITIAL_TOKEN:, which takes precedence over default globalinformation in the program file.

"Using the Hub in scriptless mode"

MIT program files no longer require programs. Any messages sent to the Hub which are nothandled by the scripting module are sent to whatever servers implement them. Thisimplements scriptless interaction (and partial scripting). As a corollary, the initial token is nolonger created implicitly if there is no program named main .

"(How to...) Add a server"

The header galaxy/galaxy_all.h now includes all relevant headers and C++conditionalizations, and is now the preferred way of specifying Communicator headers. Servers now accept multiple simultaneous connections. Server declaration files nowrecognize the GAL_SERVER_MAX_CONNECTIONS() macro, and servers accept the -maxconns command line argument. See the documentation on adding a server. Server declaration files now recognize the GAL_SERVER_OP_SIGNATURE() macro tospecify the array of input and output keys, value types, etc. for dispatch functions, and

682

servers accept the -validate command line argument which will check each call to adispatch function against its signature. See the documentation on adding a server. The function _GalSS_init_server() no longer needs to be defined, since there is now adefault in the core library. The reinitialize message no longer is required to return a frame; this requirement is nowenforced by the server library. If the reinitialize message does not provide a frame, a dummyframe will be sent instead. The administrative information (token index, session ID, etc.) in the frame sent to a dispatchfunction is now cached and inserted into the response. It is no longer necessary to return thesame frame that was passed in in order for this information to be passed back to the Hub correctly. The Hub now supports remote error reporting (see the function GalSS_EnvError). Thisnecessitates some slight complications in server-to-server subdialogues, supported by the newfunction GalSS_EnvDispatchFrame. Incompatible change: relevant MIT functions in libsls_util have been imported into the corelibGalaxy library. These symbols include the MIT socket library (sock_...), the MIT commandline argument parsing library (oa_...), and the MIT printing library (sls_...). Servers no longerlink against libsls_util by default, and as a result all such symbols must be changed. Theequivalence table can be found here. Also, the environment variable which controls theverbosity of printing is now called GAL_VERBOSE, not SLS_VERBOSE. If you choose, you mayalso link against the MIT libsls_util library and keep the old functionality; the symbols will notclash, and SLS_VERBOSE still controls the verbosity in MIT libsls_util. Incompatible change : the header file sls/util.h will no longer be in the defaultCommunicator include path, because it’s no longer used. Replace it with galaxy/util.h, orbetter yet, just use galaxy/galaxy_all.h and remove the headers galaxy.h, galaxy_io.h and serverstub.h. Incompatible change : the second argument of dispatch functions is now reliably a GalSS_Environment * object. See the documentation on adding a server, as well as theupgrade guide on accessing the Hub from dispatch functions and setting up broker callbacks. Note : this change has analogous effects on the Python, Java and Common Lisp bindings. Incompatible change : the reinitialize message no longer behaves idiosyncraticallywith respect to sending new messages. See the upgrade guide. Note : this change hasanalogous effects on the Python, Java and Common Lisp bindings. Incompatible change : because administrative information is now cached by the serverbindings, it is no longer possible to send a new message to the Hub by returning a frame from adispatch function which has the administrative information deleted. Use GalSS_EnvWriteFrame instead. Incompatible change : the old method of declaring server and dialogue functioninformation, which was deprecated in 1.3, is now no longer supported. See the upgrade guide. Incompatible change and deprecation : the function GalSS_GetServerName has beenchanged (and deprecated). It is now restricted for use with server objects, not with connections.This is to support backward compability for its use, which was almost exclusively in the serverstartup phase. Use GalIO_GetServerName and GalIO_GetCommServerName instead. Deprecation : in the context of standardizing the names for printing, the functions

683

sls_pp_frame, sls_cpp_frame, sls_pp_object, sls_cpp_object, and sls_pr_object have beendeprecated in favor of GalUtil_PPFrame, GalUtil_CPPFrame, GalUtil_PPObject, GalUtil_CPPObject, and GalUtil_PrObject. Deprecation : The following functions have been deprecated because dispatch functions arenow reliably passed connection arguments. See the upgrade guide. These functions will beremoved in a future release.

GalSS_WriteFrameToHub (use GalSS_EnvWriteFrame instead) GalSS_DispatchViaHub (use GalSS_EnvDispatchFrame instead) GalSS_GetServerData (use GalIO_GetCommServerData instead)

"(How to...) Use the frame library"

The function Gal_ListObjectFromElements has been added. This function accepts avariable number of arguments.

"(How to...) Use the timed task loop"

The timed task loop now supports thread-based connection management. The timed task loop now fully exploits an optimization based on select() which MIT hadimplemented but not used aggressively, with the result that server message handling issignificantly faster.

"(How to...) Use MIT logging"

It is now possible to log program invocations and returns using LOG_IN: and/or LOG_OUT:at the program level. In scriptless mode, it is possible to log when the Hub receives a message and returns areply using the MESSAGE: declaration together with the LOG_IN: and/or LOG_OUT: declarations. It is now possible to invoke the logger directly from a server using the log_keys builtinfunction. The operation must be declared in your program file. It is now possible to timestamp program invocations and scriptless message arrivals usingthe TIMESTAMP: declaration. It is now possible to log Hub events like alarm expiration, server access errors, etc., usingthe LOG_HUB_ACTIVITY: declaration. It is now possible to log an arbitrary string to track versions of your logfiles using theLOG_VERSION: declaration. Deprecation : the (:timestamp <key>) entry for LOG_IN: and LOG_OUT: has beendeprecated, because all keys will be logged with an operation. Incompatible change : the deprecated keys LOG_FIRST:, LOG_LAST: and LOG_ALL:have been removed. Incompatible change : the format of the MIT logfiles has been modified.

Format version number and optional user version number have been added. No keys are logged without a timestamped operation. The operations now contain the message type and the token index. The form of the data has been rationalized to conform with the format of data inside

684

frames. In particular, strings now are delimited by double quotes.

"(How to...) Set up a brokered audio connection"

Incompatible change : if you refer to your connection in your broker callbacks, you’llprobably have to make a small change due to the change in the type of the secondargument of dispatch functions. See the section on setting up broker callbacks in theupgrade guide. Deprecation : the reserved keys required for brokering can now be set and accessed usingthe functions GalIO_FrameGetBrokerPort and GalIO_FrameSetBrokerCallID. Users areencouraged to modify their code. See the upgrade guide.

"(How to...) Understand the structure of a server"

Incompatible change : the underlying structure of the toplevel loop has been revisedextensively to accommodate multiple connection servers. If you have written servers withtheir own main loops, see the upgrade guide.

"(How to...) Use a different main loop"

Incompatible change : due to the requirements of multiple connection servers, the behaviorof the functions GalIO_ServerInit, GalIO_ServerHandler, and GalIO_ConnectionPoll havechanged in subtle and important ways. If you have written servers which have their ownmain loop, you should consult the upgrade guide. Note also that the functionGalIO_ConnectionPoll is now restricted to contexts in which the timed task loop is not used,since it no longer resets its own timed task. This is an internal change which should not bevisible to any users. Incompatible change : the behavior of Hub disconnects has changed. Previously, if aserver set a function to run when the Hub disconnected, the server data would be passed tothe function. For complex reasons, it must be the connection object which is passed in thecurrent release. This affects a single MIT server, and we doubt it affects anyone else. Thefunctions have changed their signatures to reflect the fact that a GalIO_CommStruct * is theargument of the disconnect function: GalIO_SetServerDisconnectCallback, GalIO_SetDisconnectCallback, GalIO_ReconnectInit, GalIO_ServerReconnect,GalSS_SetHubDisconnect, GalSS_HubReconnectInit.

"(How to...) Use non-MIT Makefiles"

Incompatible change : because the Communicator support code has been aggregated intoa single library and include directory, the include and link directives have changed slightly.

"(How to...) Use the command-line argument parsing and printing facilities"

Incompatible change: as noted in the release notes for adding a server, these utilities havebeen newly introduced into the Communicator library from the MIT sls_util library. Theirnames have been changed in order to avoid symbol clashes.

685

"(How to...) Do signal handling"

Incompatible change : due to the requirements of signal handling in threads,Communicator now has a special signal handler installation function. It is stronglyrecommended that you use this function to install signal handlers.

"(How to...) Connect multiple users using dispatch servers"

MITRE has finally documented this facility.

"Reserved frame keys in Galaxy Communicator"

MITRE has introduced #defines for the C versions of these keys, and documented the list of keys.

"Basic server example: double"

The double example now halts when it reaches the largest integer, thanks to remote error reporting. The double example now demonstrates run-time validation of Hub scripts, scriptlessinteraction, and logging.

"MITRE utilities"

A bug has been fixed in the MITRE stdin utility which caused it to fail when the Hub haltedand reconnected. Incompatible change : MITRE’s dispatch function utility has been incorporated into the coredistribution. The functions MGal_CreateServerInfo(), MGal_AddDispatchFunction(),MGal_InitializeServerDefaults() have been removed. Incompatible change : MITRE’s multiple connection server utility has been incorporatedinto the core distribution. The functions MGal_PollMultiConnServer(),MGal_CreateMultiConnServer(), MGal_SetCommWrapperData(),MGal_GetCommWrapperData(), MGal_GetCommWrapperComm(),MGal_GetMultiConnServerData() have been removed. Incompatible change : the way the stdin utility is used has changed fairly substantially,since it used to store its information in the server/connection but now needs to store itsinformation as connection-specific data. See the upgrade guide. Incompatible change : the utility which implements the Hub as an output device has beentemporarily removed, because it is not thread-safe and did not interact appropriately withmultiple connection servers. The supporting example has been removed as well. Incompatible change : MITRE’s hub stream utility, which compensated for the unusualbehavior of the reinitialize dispatch function, has been incorporated into the core distribution.The functions MGalIO_CreateHubStream, MGal_HSSendFrame, MGal_HSSendMessage,MGal_HSReturnFrame, and MGal_HSReturn have been removed. The supporting examplehas been removed.

686

"Tools for constructing dispatch servers and UIs"

MITRE has documented and enhanced the dispatch server protocol and provided libraryfacilities for constructing for using it. MITRE has also provided a general-purpose dispatchserver proxy for use with dispatch server UI elements, and enhanced its UI server withdispatch protocol capabilities.

"MITRE tools"

The MITRE process monitor now provides a typein window which allows one to edit thecommand line before restarting. An extremely evil bug in the process monitor has now been fixed in which the CPUconsumption of an idle Hub slowly climbed to almost 100% of the CPU. The MITRE run_server script now takes a -verbosity argument. MITRE now provides a utility which translates MIT logs into XML.

Other

Unless you’ve been seriously messing with the guts of the Communicator library, none of thesechanges should be visible to you.

The function Gal_CopyHash has been added. The syntax <server>.<opname> as the name of a new message is now recognized by theHub in the same way it is recognized in MIT scripting. When MIT scripting claims themessage, the server dimension is ignored; in the scriptless case, it is taken as a directive toforward the message to the specified server. The bug in Allegro 5.0 having to do with brokering in the Allegro bindings seems not to arisein Solaris 2.7. The (undocumented) HUB_stub Python class in tools/lib/python has been moved intoGalaxyIO.py and renamed ClientConnection. Internal deprecation : the function GalSS_ForceReturn has been simultaneously introducedand deprecated. Use GalSS_EnvForceReturn instead. Internal deprecation : the following functions have been deprecated because dispatchfunctions are now reliably passed connection arguments. See the upgrade guide.Thesefunctions will be removed in a future release.

GalSS_ReadFrameFromHub (use GalIO_CommReadFrame instead) GalSS_GetServerSock (apparently unused; use GalIO_GetCommSocket instead) GalSS_CloseServerSock (apparently unused; use GalIO_CloseCommSocket instead) GalSS_SetHubDisconnect (use GalIO_SetDisconnectCallback instead) GalSS_CloseHubConnection (use GalIO_DestroyCommStruct instead) GalSS_FlushHubWriteQueue (use GalIO_CommFlushOutQueue instead)

Incompatible internal change : the (apparently unused) function Gal_GetServerPort hasbeen removed. It has been replaced by GalIO_GetServerDefaultPort, which requires aserver object argument. Incompatible internal change : the function Gal_GetServerName is now deprecated, andits use has been restricted to contexts in which the programmer has called

687

Gal_SetServerName directly. In the vast majority of circumstances, it will no longer work.Use GalIO_GetServerName instead. Incompatible internal change : the following internal functions have been removed.

GalIO_GetCommListenSocket (use GalIO_GetServerListenSocket instead) GalIO_GetCommListenPort (use GalIO_GetServerListenPort instead) Gal_FileRewind (now internal) Gal_MakeStringStream (use Gal_MakeStringInputStream instead) Gal_StringStreamString (use Gal_StringInputStreamString instead) Gal_FileNextChar (now internal) GalIO_CloseCommListenSocket (use GalIO_CloseServerListenSocket instead) Gal_StringNextChar (now internal) Gal_StringRewind (now internal) Gal_MakeFileStream (use Gal_MakeFileInputStream instead) Gal_InitializeServerDefaults (use GalSS_InitializeServerDefaults instead)

Incompatible internal change : the arglists of the internal functions Gal_ReadObject,Gal_ReadFrame, Gal_ReadString, Gal_ReadList, Gal_ReadBinary, Gal_ReadObjectLine, Gal_ObjectString, Gal_FileNextLine, Gal_FileCurrentLine, and Gal_ReadToken have allchanged due to improved encapsulation of input streams. Incompatible internal change : the function Gal_CallServerFunction has been replaced by GalSS_CallServerFunction., due to a change in the API. Incompatible internal change : the function Gal_GetProgramTagHash() now returns acopy of the hash table it’s passed, for thread safety reasons. Incompatible internal change : the name of the operation to be invoked in the server isnow the name of the frame, instead of the contents of the op_name slot. Also, the name ofthe frame intended to invoke the reinitialize message has been changed from welcome to reinitialize . Finally, there is a special key for script-control-specific information called :hub_program_info which takes the place of :ridx . All of these changes are intendedto support the consistency of incoming and outgoing messages, and to make scriptlessmessage invocation as transparent as possible in the Hub. Incompatible internal change : the communications protocol has been extended slightly toallow encoding of various metalevel message types: new message, reply, error, etc. Ifyou’ve written native programming language bindings for Communicator, you’ll want to payattention to this. Incompatible internal change : the communications protocol change now subsumes the"bounce" message which was used to reject connections above the maximum numberpermitted. If you’ve written native programming language bindings for Communicator, you’llwant to pay attention to this.

Known bugs in 2.0

The Java bindings sometimes truncate audio when interacting with the audio device. Wedoubt that people are planning on writing a Java audio server, so we’re not in a terrible hurryto track this one down. The MIT audio server sometimes exhibits repeated clicking on short outputs using the OSSsound drivers on Linux or X86 Solaris. We don’t know whether this is a problem with the

688

MIT audio server or with the OSS drivers. There are also some odd cases if you leave theaudio server running when you shut down and restart the Hub where the audio server will notflush its output buffer. There is a bug in logging for error replies. If there are no LOG_OUT: keys specified inRULE:, PROGRAM: or MESSAGE: for an incoming reply from a server or a server-to-serversubdialogue, error replies will not be logged unless you put "system_error" in your TIMESTAMP:list. This bug will be fixed in a later release. Because of the way the Java bindings map dispatch function names to method names, theJava servers will not generate correct default message signatures to report back to the Hub, andthe -validate flag may erroneously report that the server isn’t aware of a particular message. Ifyou want signatures reported correctly, do not rely on the automatic generation of default signatures. The Allegro bindings still "hang on" to server ports for a while if the server dies inelegantly.We know that this has to do with REUSEADDR in Allegro, but we haven’t upgraded to the newsocket library in 5.0 which makes it easy to fix this problem. It’s not a particularly good idea to try to compile multiple platforms simultaneously in thesame distribution. In particular, clean_headers.pl must refer directly to your Perl executable,which may be in different places on different platforms. There are other clashes that might ariseas well.

Significant omissionsThere are a number of things which we had hoped to get to but didn’t, which render thedistribution somewhat inconsistent.

Although we are fairly certain that the core Communicator library is thread-safe, it will notwork without modifications on Windows NT, at least at the moment. There is a Cygnus-supplied DLL which one can obtain which provides a POSIX threads wrapperaround the Windows native threading, but we haven’t had a chance to download and test it;furthermore, as of October 1999 there were a couple functions we use (such as sigwait())which they’ve documented as not working yet. Although we have invested a great deal of effort into segmenting the Communicator Huband MIT Hub scripting, we haven’t had a chance to try writing our own scripting module, norhave we documented the API for those who wish to try writing their own scripting module.Our best advice is to examine the MIT scripting module in contrib/MIT/galaxy/src/libGalaxyHubControl. Although we made signfiicant progress in segmenting the Communicator Hub and MIT Hubscripting, the job isn’t quite finished. The builtin server, in particular, most likely still hasMIT-specific functionality in it. Although we’ve finally provided documentation for dispatch functions, we still haven’tdocumented the MIT gui server, nor have we provided well-supported scripts for setting it up(it’s not at all obvious). We have still not had the chance to upgrade our documentation on how to build anend-to-end system.

689

http://sourceware.cygnus.com/pthreads-win32/

Last modified September 28, 2001

690

Release Notes for Version 1.x

This document briefly lists all the changes from one Galaxy Communicator distribution to thenext for versions 1.x.

Version 1.x is not the current version of Galaxy Communicator. For the current distribution, youcan find a list of new features here. You can find a detailed description of the steps required orrecommend to upgrade existing features of Galaxy Communicator here.

August 17, 1999: Version 1.3All reported bugs have been fixed.

Galaxy Communicator 1.3

MIT has deprecated the functions Gal_GetValue, Gal_Value, Gal_ValueWarn,Gal_ValueWarnWithKey, Gal_ValueSilent, and Gal_UnwrapValue. They will be removed ina subsequent release. Incompatible change: To support appropriate manipulation of audio output, the brokeringmechanism now maintains a queue of brokering objects. Each object in the queue must nowbe made active in order to work correctly. Also, the function to be called when all the datahas been written to a broker connection has changed from GalIO_BrokerDataDone to GalIO_BrokerDataOutDone. See the brokering documentation. MIT has provided significant improvements to the program file parsing, including a verification program. All previously reported program file parsing bugs have been fixed. Anumber of changes in the program file syntax have been introduced:

Incompatible change: all multi-token string values of toplevel declarations now requiredouble quotes, as in the value of INITIAL_REPLY:. The #include directive now permits quotes around its value. The backslash character (\) can now be used to escape linebreaks. Incompatible change: The order of declarations in the program file is now morerestrictive. While global declarations can still appear anywhere, servers must be definedbefore programs. Logging information may now appear in included files. Incompatible change: the comparisons ":foo EXISTS", ":foo X", and ":foo -" no longercount as existence tests for the key :foo in rule conditions. The comparison ":foo NULL"no longer counts as a nonexistence test. The comparison ":foo #5" no longer counts as

691

an integer identity test. The comparison ":foo ^ bar" is a new substring comparison in rule conditions. MITintends to deprecate the % comparison. The connectives "&&" and "||" now work as well as "&" and "|". Incompatible change: the HOST: directive no longer permits server IP addresseswithout delimited double quotes. The spacing rules in rule conditions are now more liberal; whitespace is now permittedbetween the comparison and its value, and between the negation and the comparison it precedes. It is no longer necessary to delimit the opening and closing brackes of frame and list representations with whitespace. It is now possible to group clauses explicitly using parentheses. The default "closestbinding" for disjunction remains in effect in the absence of explicitly grouping. Incompatible change: keywords in program files must now begin with a colon (:),except in rule conditions, where a colon will be prepended if one is missing.

MIT has introduced a new method of generating headers for its servers and dialogue controlwhich use the C preprocessor and are thus portable to platforms like Windows NT which donot typically use GNU make. We strongly recommend updating to this new method, sincewe are deprecating the old method and may not support it in the future. As a side effect of program file parsing, MIT has added the Gal_Boolean type, with valuesGAL_TRUE and GAL_FALSE. Incompatible change: MIT has changed the argument order of sls_pp_frame to render itconsistent with other functions. MIT has updated its audio server to incorporate most of the functionality from the MITREversion. MITRE has made some slight modifications to the result in order to eliminate theMITRE audio server altogether. It is no longer necessary to define a program named main in the Hub program file. MIT has rationalized the behavior of destroy! and none! so that both now respect SET: andDEL: directives in rules; previously, only destroy! did so. Incompatible change: the following functions have been removed: Gal_PatternMatches, Gal_FillConditions. Incompatible change: the following functions have changed their signatures, in addition tosls_pp_frame: Gal_ReadObject, Gal_ReadFrame, Gal_HashListNext, GalIO_BrokerSetFinalizer. MIT has changed the printed encoding of floats in the print representation of frames from %f(e.g., 56.78) to %e (e.g., 5.678000e+01). All language bindings now reflect this change. Theframe reader should still recognize both formats, though, so if you have frames saved tofiles which you parse using the Galaxy utilities, and these frames have floats in them, youshould not have any problems with the data. MIT has introduced a new C preprocessor directive -DREPLACE_PRINTF which will causeall calls to printf() to be replaced with calls to sls_pinfo1(). MIT has made significant updates to its batchmode and nl servers. However, due to thelarge volume of changes, the MITRE documentation is now rather incomplete. Incompatible change: MIT has updated the administrative information keys in its framesfrom tidx and ridx to :tidx and :ridx . This change should be invisible to anyone who

692

wasn’t trying to do anything complicated. However, tools which rely on data which storesframes from the Hub may need to be updated if these tools rely on the identities of theseadministrative information keys. MITRE has introduced a distinction between the 32- and 64- bit Irix architectures, followinga suggestion by AT&T.

MITRE support 1.3

MITRE has finally documented session keys in the Hub. Incompatible change: in conjunction with MIT’s change to incoming brokering, the MITREfunction MGal_AddIncomingBrokering now takes an additional activate argument. MITRE has updated Python, ACL and Java language bindings to be consistent with MIT’suse of %e instead of %f encoding for floats in frames. MITRE has provided a Python utility for updating to the new method of generating server headers. MITRE has finally documented its push-to-talk server. Incompatible change : this serverhas been moved from MITRE/examples/speech_explore to MITRE/servers/bin. MITRE has successfully compiled the Allegro bindings with Allegro 5.0, but there is a bug inbrokering in 5.0 which we have not yet had time to track down. The MITRE audio server is no more. MIT has incorporated the relevant functionality intotheir audio server.

July 15, 1999: Version 1.2Almost all reported bugs have been fixed (but see the list of known bugs). Some program fileparsing bugs have been fixed in the latest MIT distribution, which we will make available in mid-August.


MIT has fixed a bug where the environment variables ARCH, OS and ARCHOS weresometimes clashing with local shell configurations. See the installation instructions and templates/init.make.in MIT has resolved collisions between its STOP constant in the libgalaxy library and the PerlSTOP constant. MIT has improved automatic generation of lib and include links. This fix has necessitated amodification of the installation procedure. MIT has added a GUI server and necessary support files (Web directory, Java client applet,icons, etc.) to this distribution. Documentation will follow in MITRE support 1.2.1. MIT has improved logging capabilities. MIT has improved x86-linux audio support In previous releases, the only way to access the value of the utterance ID was by calling thedispatch function builtin.increment_utterance . The utterance ID is now alwaysavailable for access in :hub_session_utterance_id .

693

New dispatch function builtin.dispatch_token allows tokens to move from oneprogram to another, not just to main . Logging now writes keys to the log file when they are encountered, not when builtin.increment_utterance is called. Mapping IN: keywords will now remap keys previously set by PARAM: and RETRIEVE:. STORE: now searches "unmapped" key names in the OUT: keys if the named keys are notfound directly. The MIT audio playbox now sets levels automatically by sampling line noise. It is now possible to use SFUNC.H in the MIT Makefile to customize the name of theoperations header file, normally named server.h . This will allow multiple servers in thesame directory. A couple of subtle bugs discovered by AT&T in the timed tasks loop have been fixed. Thanks in part to AT&T and CMU, MITRE and MIT have made the core Communicatorfunctionality available on Windows NT. It is now possible to specify values for the initial token on the command line. LOG_ALL:, LOG_FIRST: and LOG_LAST: have been superseded by LOG_IN: andLOG_OUT:. See the logging documentation. A bug reported by CMU which involves memory management problems in loggingtimestamps has been fixed. A bug reported by CSLU where the audio server seg faults on long utterances has been fixed.

Known bugs

If the MIT audio playbox receives two broker output requests, it will output the audio which itreceives first, not necessarily in the order it received the broker requests. This bug has beenfixed in the MITRE version of the MIT audio playbox, and will be fixed in the next MIT distribution. A string consisting of a single space as a key value in the program file will cause the Hub toseg fault. This bug will be fixed in the next MIT distribution. A server name which contains a dash will cause the Hub to do the wrong thing when theserver name is used explicitly in the consequent of a program file rule (e.g., --> foo-server.bar ). This bug should be fixed in the next MIT distribution. There may still be some infelicities in the IRIX support which will be resolved with the nextMIT release.

MITRE support 1.2

Local MITRE change: removed support for devel in MITRE/install. A fatal bug discovered by SRI in the MITRE example for using external main loops has been fixed. A memory error in the hub stream utility which caused a seg fault on Linux has been fixed. A bug discovered by SRI in the frame parsing for the Allegro Common Lisp bindings hasbeen fixed. A bug in the frame printing for the Python bindings and the Allegro Common Lisp bindings

694

has been fixed. MITRE has finally documented the frame syntax. The multiple connection server utilities now work correctly when no server data is passed toframe writing functions. MITRE now provides the process monitor which was developed for the MITRE trainingcourse. The process monitor is currently used in some (but not all) of the MITRE examples. The frame documentation and the documentation on adding a new server now includeinformation on memory management. The new SFUNC.H customization has been incorporated into all MITRE examples. The documentation of the Galaxy Communicator system is now up-to-date. MITRE has added a number of servers, including a general-purpose dispatch server (not yetdocumented), a Festival server (provided by Alan Black; not yet documented), a MITRE versionof the MIT audio playbox, an audio replay server (not yet documented), a frame transducer (notyet documented), and a PostGres DB server (not yet documented). MITRE has added new tools, including a set of scoring and log manipulation tools (not yetdocumented), a unit tester, server scripts and a process monitor. The process monitor wasdeveloped for the MITRE training course, and is currently used in some (but not all) of theMITRE examples.

Known bugs

There is a string quoting bug in the frame encoding process in the Python bindings whichhas not been fixed. We will hold off on fixing this bug until we incorporate the string quotingbug fixes from the MIT July distribution.

March 26, 1999

MITRE support 1.1.2

Added expanded usage instructions to all server pages. Ensured that every dispatch function in the MITRE servers returns a frame, asrecommended in the training course. Fixed some dead links in the documentation for the language bindings. Corrected some errors in the documentation. Fixed a buffer flushing problem in the speech monitor server.

January 29, 1999: Version 1.1.1

695

Galaxy Communicator 1.1.1

Added new rule-based logging keys LOG_IN:, LOG_OUT:. Documentation to follow. Added new global program file key INITIAL_TOKEN:. Documentation to follow. Fixed typo in ComputerFone library. Added Makefile dependencies to force recompilation of clean_headers.pl in the appropriate circumstances. Added i586 as an architecture. Fixed a bug in the Hub in which type mismatches between tokens and program file valuescaused a segmentation fault. Fixed a bug in the Hub in which empty rule conditions were occasionally being read incorrectly.

MITRE support 1.1.1

Fixed bug where Java subdirectories were not being appropriately ignored when Java wasnot enabled. Changed Java API slightly. See the change log. Added to FAQ. Enhanced logging documentation. Fixed documentation errors: PARAM, DB server API Fixed flaw in TrueTalk server relating to how session IDs are passed around. Documented MGal_GetStdinPollData and MGal_SetStdinPollData. Following arecommendation from CMU, modified the stdin polling utility to allow immediate restarts andto allow the prompt to be changed.

January 15, 1999: Version 1.1


The binary data frame printout problem has been fixed. MIT has made improvements to their Makefiles. MIT has enhanced the Communicator API: GalIO_AddTimedTaskWithIO. Linux support is now included.

MITRE support 1.1

We have added Allegro Common Lisp bindings. These bindings currently lack thefunctionality of the MITRE utilities library. We have significantly improved the Python bindings, and revised their API to conform asclosely as possible with the C libraries. These bindings are incompatible with 1.0.1. We have added a number of MITRE servers to the distribution. We have added two demos to the distribution. We have included support for PPC Linux and SGI Irix.

696

November 6, 1998: Version 1.0.1

Galaxy Communicator 1.0.1

We have fixed a significant bug in audio brokering, which will cause audio brokering to fail. We have fixed a number of bugs in preparation for distributing the Pegasus Lite demo. Webelieve that all that we’re missing is the documentation about how to run the demo, althoughwe have not yet tested the audio version. We are aware of an infelicity in the printing of binary data in frames, due to a poorimplementation of the encoding algorithm. This will be fixed in the next release.

MITRE support 1.0.1

We have improved the layout of the example documentation. We have added Python bindings and examples of Python servers. Documentation issomewhat spare; it will be improved soon.

November 1, 1998: Version 1.0


We have not yet had a chance to learn about and document a number of features of theGalaxy Communicator system:

The BIANCA turn management tools Logging in the Hub script Dispatch servers

We will document these as soon as we can. Some of the MIT servers may have public functions whose names are too "generic" andlead to symbol clashes with other libraries. We will be working with them to fix theseproblems when they arise.

MITRE support 1.0

We have not yet had a chance to update and bulletproof the Java bindings. These will beforthcoming soon.

Last modified September 28, 2001

697

Directory Structure

Top LevelHere we describe the relevant organization beneath the top level Galaxy Communicator directory.

docs/ bin/ lib/ src/

libGalaxy/ HUB/

templates/ contrib/

MITRE/ bindings/ demos/ docs/ examples/ servers/ templates/ tools/ utilities/

MIT/ galaxy/

src/ verify_program/, verify_dctl/ libGalaxyEnhancements/ libGalaxyHubControl/

698

bin and lib Subdirectoriesbin/ and lib/ directories are created at compile time as are bin/ and lib/ directories in all thesubdirectories. bin/ and lib/ directories contain executables and libraries respectively. There’s alittle more to the bin/ and lib/ directories which is described more fully below in the section on ARCH-OS.

The docs SubdirectoryThis subdirectory contains MITRE-produced documentation on the distribution, the core libraries,and the Hub.

The src SubdirectoryThis subdirectory contains the sources for the core of the Communicator distribution.

HUB/ C code for the Hub itself. The Hub acts as an intelligent router whose behavior can becontrolled by a script. libGalaxy/ C code for the core Communicator support library, including frame processing, HUB-servercommunication, printing, command line argument parsing, and the default server wrapper.

The MITRE SubdirectoryThe MITRE subdirectory includes examples of Communicator functionality and utilities whichuse the Communicator libraries in illustrative ways. MITRE also provides a set of programminglanguage bindings (for Python, Java, and Allegro Common Lisp), servers (DB, UI, frametransducer), and tools (process monitor, tools for parsing MIT log files, scripts for running MITservers). See the main index for a comprehensive list of these resources.

The Galaxy SubdirectoryHere’s an overview of the directories under galaxy/src/.

verify_program/, verify_dctl_file/ Hub script and dialogue control file verifiers. libGalaxyEnhancements/ MIT enhancements to the Communicator core, including turn management. libGalaxyHubControl/ MIT Hub scripting module (used by the Hub by default)

699

TemplatesThe templates/ subdirectory contains configuration and support files which are included in thevarious Makefiles. Essentially that’s where we consolidate stuff which is common to all the Makefiles.

ARCH-OS SupportThe only thing to note is that the Makefiles will put things under subdirectories of the form$ARCH-OS (e.g. sparc-sunos, sparc-solaris). The bin/ directory also contains a csh script called fat_binary, which determines the correct executable to exec. For example, making an executablenamed a.out will create a symbolic link in bin/ to fat_binary (i.e. bin/a.out -> bin/fat_binary). fat_binary determines the $ARCH-OS of the machine it’s running on, looks at the name by whichit was called, and invokes the correct executable.


Copyright (c) 1998, 1999, 2000 The MITRE Corporation

ALL RIGHTS RESERVED

700

http://www.mitre.org/

Guide to the Makefiles

IntroductionThe Make organization is designed to support multiple versions of object files in parallel. It doesthis by separating .o files, libraries, and executables into separate directories based upon themachine architecture (processor) and operating system. It also separates the .o files intoseparate directories according to compile type (e.g. optimized, debug, profile, etc.), andseparates libraries and executables by appending an appropriate identifier. For example, alibrary libfoo.a would have a debug form of libfoo_debug.a, and the debug version of theexecutable a.out would be a.out_debug. The Makefiles are designed to link automatically againstthe correct versions of object files and libraries.

In addition to being separated by directories according to machine architecture and OS(ARCH-OS), each executable has a symbolic link in the top level bin directory which points to thefile fat_binary. fat_binary is a csh script which figures out the ARCH-OS of the machine itsrunning on, looks at the name by which it was invoked, and executes the appropriateARCH-OS-specific executable.

We currently support compiling optimized, debug, and profiled versions of everything, and maysupport purify’d versions in the future. As for ARCH-OS, MIT typically builds on Sparc and x86Solaris, and MITRE typically builds on Sparc Solaris and Linux; these systems can beconsidered "supported." See the section on supported architectures.

The next section will describe how to build the system "out of the box." You should read thesection on installing first.

Structure and Organization(Note: the following discussion is augmented by the section on directory organization. TheMakefile structure of the core library is inherited directly from MIT.)

All source code resides under src/, and is typically separated so that each directory correspondsto either one library, one server, or one or more executables. There is a top level Makefile in src/,which provides a simple way to ’make’ various targets in one or more directories. (e.g. libs, hub,etc.). This Makefile basically invokes the appropriate targets in the subdirectory Makefile(s).

701

Each subdirectory Makefile is essentially a slightly modified version of templates/Makefile.stubwith the appropriate variables defined. Makefile.stub is designed to be as simple as possible.This is accomplished by hiding almost all of the Makefile complexity in files in the templates/directory. Makefile.stub ’includes’ the file rules.make, which in turn includes config.make. Theformer file defines a variety of symbolic variables, as well as all the rules and targets. The latterprovides all the details of your configuration as described in the installation documentation.

All Makefiles (except for the top-level Makefile) support the same set of global targets, which aredefined in templates/rules.make For a description of these and toplevel targets, see the installation documentation.


702

Windows NT Port

Introduction What has been ported? What libraries do I need to compile the HUB? How do I write a server? Differences between Unix and Windows

IntroductionThe Windows NT port of the HUB and support libraries is a joint effort between AT&T, CMU andMITRE. However, all questions and bug reports should be sent to [email protected].

Our plan is to distribute this port as both source and pre-compiled executables and libraries. Thesource and pre-compiled versions should be available through our regular download method.We have tried to follow the guidelines by placing the libraries and executables in the samefashion as their Unix counterparts, except the "arch-os" directory is named x86-nt . So forexample, the HUB executable compiled using our project settings would be placed under GalaxyCommunicator/bin/x86-nt/HUB.exe . The same applies for the libraries.

What has been ported?Only a subset of the Galaxy Communicator distribution has been ported. We have concentratedon the HUB executable and on the libraries necessary for writing servers compatible with theWindows platform. Microsoft’s Visual Studio 6.0, which includes Visual C++, was used as acompiler, and the project and workspace files (Microsoft’s alternative to Makefiles) are alsoVC++ 6.0 compliant. We have generated and added to the archive makefiles with .makextensions as well. The same source base is used on all platforms (Unix and Windows). If yourequire Makefile support for different versions of MSVC++ and/or alternative compilers, pleasecontact us at [email protected] system has only been tested on WindowsNT 4.0 SP 5, we would appreciate feedback on other platforms in which this system wassuccessfully (or unsuccessfully) used on. A new workspace has been created that holds all the Win32 projects. It can be found at: GalaxyCommunicator/GC.dsw

703

http://fofoca.mitre.org/download/

The following are the libraries that were ported, their equivalent workspace files and where thecompiled libraries and/or executables are placed:

Core library ported:

GalaxyCommunicator/src/libGalaxy libGalaxy.(dsw, dsp, mak) GalaxyCommunicator/lib/x86-nt/libGalaxy.lib

MIT library ported:

GalaxyCommunicator/contrib/MIT/galaxy/src/libGalaxyHubControl libGalaxyHubControl.(dsw, dsp, mak) GalaxyCommunicator/contrib/MIT/galaxy/lib/x86-nt/libGalaxyHubControl.lib

MITRE library ported:

GalaxyCommunicator/contrib/MITRE/utilities/src libMITRE_galaxy.(dsw, dsp, mak) GalaxyCommunicator/contrib/MITRE/utilities/lib/x86-nt/libMITRE_galaxy.lib

Executables ported:

GalaxyCommunicator/src/HUB HUB.(dsw, dsp, mak) GalaxyCommunicator/bin/x86-nt/HUB.exe

GalaxyCommunicator/contrib/MITRE/examples/double double.(dsw, dsp, mak), complex_double.dsp, multiply.dsp GalaxyCommunicator/contrib/MITRE/examples/double/bin/x86-nt/(double,complex_double, multiply).exe

GalaxyCommunicator/contrib/MITRE/examples/local_makefile nt_build.(dsw. dsp, mak) GalaxyCommunicator/contrib/MITRE/examples/local_makefile/bin/x86-nt/nt_build.exe

GalaxyCommunicator/contrib/MIT/galaxy/src/verify_program verify_program.(dsw, dsp, mak) GalaxyCommunicator/contrib/MIT/galaxy/bin/x86-nt/verify_program.exe

What libraries do I need to compile the HUB?The HUB executable depends on the libGalaxy.lib library.

704

How do I write a server?The following are the libraries that most servers require:

libGalaxy.lib libMITRE_galaxy.lib (possibly)

There is no longer any difference between platforms in writing a server. As documented in thesection on adding a server, the new style of header generation relies only on the Cpreprocessor, so it is portable across platforms. All MITRE examples have been upgraded to usethis new style. As a demonstration, the double example has been ported to NT.

Differences between Unix and WindowsThe functionality differs slightly between Unix and Windows versions. Below are some of theknown limitations and differences.

Reading with time-out (in sock.c from libsls_util) was replaced with regular reads underWIN32 since SIGALARMS are not supported. Logging to syslog wasn’t ported since the syslog facility does not exist under NT. Under NT sockets can not be treated as file descriptors, therefore one cannot poll stdin.This renders the MITRE stdin poll utility unusable. Logging using the HUB will not lock files before writing to them.

Last modified July 13, 2000

705

Thread Notes

The core Galaxy Communicator library is known to be thread-safe on the following platforms:

Sparc Solaris 2.5.1 and above RedHat Linux 6.0 (kernel version 2.2.5, glibc version 2.1.1) Windows NT (thanks to MIT for making this work)

The thread implementation in Communicator uses POSIX threads on Unix, and native NTthreads on NT.

On Unix, if you wish to have the option of compiling a threaded version of a library or executable,insert the line

THREAD_SAFE = 1

into the MIT Makefile before rules.make is loaded. This will enable the compilation targets thread , thread-debug , thread-profile , thread-purify and thread-insure . IfTHREAD_SAFE has not been set, these targets will be equivalent to the non-threaded versions.MITRE has provided an (experimental) option for toplevel server operation which uses threadsinstead of timed tasks for server and connection management. As of version 3.0, this optionworks on Windows as well as Unix.

On Windows NT, make sure the -D_REENTRANT flag is in your compilation line.

Due to some backward compatibility issues, we have let remain a situation in the output codewhere the variable which controls verbosity levels may be initialized (harmlessly) more than once.

The Hub is not completely thread-safe, due to calls to rand() and localtime(). These will be fixedin a subsequent distribution.

SignalsThe Communicator library provides two functions for signal handling, which are guaranteed to dothe right things in both threaded and non-threaded cases. Please use these functions to installsignal handlers (and to initialize signal handling, if you’re writing your own toplevel loop).

706

Platform notes

SolarisThe Communicator library uses the following reentrant versions of functions:

gethostbyname_r() strtok_r()

The library also uses a homemade version of inet_ntoa() due to some unfortunate interactionswith Tcl observed by the MIT folks. inet_ntoa_r() is available in Solaris, but is not documented.

RedHat Linux 6.0The Communicator library uses the following reentrant versions of functions:

gethostbyname_r() strtok_r()

The library also uses a homemade version of inet_ntoa() due to some unfortunate interactionswith Tcl observed by the MIT folks. The library also provides a thread-safe version of strerror(),which appears not to be thread-safe in this version of Linux.

It appears that RedHat 5.2 might also support a thread-safe compilation of the core library.RedHat 4.2 is pre-glibc and may not have the appropriate thread support.

Windows NTThe Windows NT functions gethostbyname(), inet_ntoa() and strtok() are thread-safe. The NTdocumentation does not make it clear whether strerror() is thread-safe.

Other platformsIf you enable threads for the core Communicator compilation and your compilation fails becauseany of these functions are missing, the chances are you will not be able to enable threads. Evenif you write a wrapper function with the appropriate mutex protection, you will not be able toprotect the calls in the core Communicator library and the results will be unsafe behavior. If youneed support enabling threads on a platform not described here, please contact us.


707

Galaxy Communicator 3.3 Documentationcommunicator.sourceforge.net/sites/MITRE/... · Summary . . ....

Documents

Transcript of Galaxy Communicator 3.3 Documentationcommunicator.sourceforge.net/sites/MITRE/... · Summary . . ....