Intel Technology Journal Q2, 1998...Intel Technology Journal Q2 ‘98 An Overview of Advanced...

Intel Technology Journal Q2, 1998

Preface Lin Chao Editor Intel Technology Journal This Q2'98 issue of the Intel Technology Journal focuses on failure analysis testing methodologies used at Intel and software-based research in computer vision, floating-point operations, and H.323 set of standards for multimedia communications. Failure analysis is one of the key competencies in Intel and is essential to Intel's steep product ramp and high-volume manufacturing. The first paper in this issue describes failure analysis techniques used to detect and localize flaws in the silicon manufacturing process. Intel actively pursues technologies that expand computing and telecommunication capabilities in PCs. The other papers in this issue describe new software-based research that contributes to this expansion. The second paper is on computer vision. Helping computers to "see" is just one aspect of broader research into a "perceptual interface" for PCs where computers can "speak", "sense" touch, and in the case of this paper, "see." This paper describes the development of a 4-degree of freedom color object tracker. Included with this paper are three video clips showing the abilities of the head-tracking software. Floating-point divide, remainder, and square root are three important operations performed by computers today. The third paper describes research into software alternatives to the hardware implementation of floating-point operations. This paper describes some of the general properties of floating-point computations, and proves the IEEE correctness of iterative algorithms that calculate the square root of a floating-point number. The fourth and fifth papers describe an important industry standard, H.323, for multimedia communication over packet-based networks. The fourth paper, co-authored by the Chair of the H.323 Interoperability Group, presents an overview of the H.323 core components and functionality. IP telephony is an important industry trend, and the role of H.323 procedures in deploying IP telephony are explained. The fifth paper presents a characterization of video and audio traffic transported over the Internet by video conferencing applications following the H.323 standards. This paper looks at the multimedia traffic sources of a H.323 terminal. Issues such as packet format and multiplexing of audio and video frames at the host are studied.

Copyright © Intel Corporation 1998. This publication was downloaded from http://www.intel.com/. Legal notices at http://www.intel.com/sites/corporate/tradmarx.htm

http://www.intel.com/technology/itj/chao_bio.htm

An Overview of Advanced Failure Analysis Techniques for Pentium and Pentium ProMicroprocessors 1

An Overview of Advanced Failure Analysis Techniques forPentium and Pentium Pro Microprocessors

Yeoh Eng Hong, Intel Penang Microprocessor Failure Analysis Department, MalaysiaLim Seong Leong, Intel Penang Microprocessor Failure Analysis Department, MalaysiaWong Yik Choong, Intel Penang Microprocessor Failure Analysis Department, MalaysiaLock Choon Hou, Intel Penang Microprocessor Failure Analysis Department, MalaysiaMahmud Adnan, Intel Penang Microprocessor Failure Analysis Department, Malaysia

Index words: failure analysis, DFT, DFFA, e-beam, DPM, RTL

AbstractFailure analysis (FA) is one of the key competencies inIntel. It enables very rapid achievement of world classmanufacturing standards, resulting in excellentmicroprocessor time-to-market performance. This paperdiscusses the evolution of FA techniques from onegeneration of microprocessors to another.

According to Moore’s law, transistor count doubles astransistor dimensions are reduced in half every 18 months,allowing for more complex microprocessor architecturedesigns. For example the Intel486DX™ microprocessorhad 1.2 million transistors while the Pentium®microprocessor contains 3.1 million transistors. Withrapid technological advances such as more complexmicroprocessor architecture, an increasing number ofinterconnect layers, and flip-chip packaging technologyfor products like the Pentium® and Pentium® IImicroprocessors, conventional FA techniques, in use sincethe Intel386DX™ processor generation, are no longereffective. These conventional techniques require in-depthknowledge of the processor’s architecture, and theyinvolve exhaustive e-beam probing work, which typicallyresults in very long FA throughput times. Other traditionaldefect localization techniques, such as emissionmicroscopy from the frontside of the die, are alsobecoming less successful due to the increased number ofmetal interconnect layers that obscure local circuitry.

This paper provides insight into FA techniques that havebeen adopted at Intel. It discusses the evolution ofsoftware fault isolation techniques based on Design ForTestability (DFT) features, and other special FAtechniques. In this paper, we will discuss these techniquesand show how they are effectively used to produce fast FAsupport turnaround for both silicon debug and

manufacturing. We will also review their technical meritsand return on investment, as well as the cost of eachtechnique to Intel. The main focus of this paper iselectrical fault isolation techniques, as opposed to physicaldefect localization techniques such as liquid crystalanalysis and emission microscopy.

In the context of this paper, Fault Localization and FaultIsolation (FI) are synonymous. These are defined as thetask of electrically isolating the location of a defect inlogical space. Another approach, termed DefectLocalization, refers to the task of isolating the location ofa defect in physical space.

IntroductionFailure analysis plays a very important role in thesemiconductor industry in enabling timely product time-to-market and world-class manufacturing standards(greater than 95% manufacturing yields, lower than 100defects per million, or DPM). At Intel, the situation iseven more compelling: quick failure analysis turnaround isnecessary to support Intel’s steep product ramp and high-volume manufacturing, where annual microprocessorproduction volume is in the range of tens of millions ofunits.

However, the increasing complexity of microprocessors inthe form of more complex architecture designs, shrinkingtransistor feature sizes, and new packaging technologieshave significantly increased the failure analysis challengeson Intel’s Pentium and Pentium Pro family ofmicroprocessors. A roadmap recently published by theSemiconductor Industry Association (SIA) predicts thatby the year 2000, microprocessor transistor counts willexceed 21 million transistors. The same industry roadmapalso predicts that by that time, microprocessors will utilize

Intel Technology Journal Q2 ‘98


full flip-chip technology, instead of current wirebondassembly and packaging technology. Traditional faultisolation techniques using intensive e-beam probing andassembly code minimization as well as die frontside defectlocalization with liquid crystals or emission microscopyare no longer sufficient even for today’s complexmicroprocessors like the Pentium and Pentium Promicroprocessors.

Newer FA techniques based on design-for-testability(DFT) and design-for-failure-analysis (DFFA) featureshave proven to be highly successful for the Pentium andPentium Pro microprocessors, as evidenced by highanalysis success rates (>90%) and short analysisthroughput time. Without these new FA techniques thatpinpoint the exact failing location, detection of sub-micron defects such as silicon dislocation, as shown inFigure 1 below, is very difficult if not impossible.

Figure 1: A TEM micrograph of silicon dislocation

This paper provides insight into the various FA techniquesbased on the DFT and DFFA features that have beensuccessfully developed for the Pentium and Pentium Promicroprocessors. Two other advanced FA techniques andtools will also be discussed. These two techniques are thelow-cost personal computer (PC) system-level testersolution and the processor cartridge-level (SECC) FAtechnique.

Failure Analysis OverviewFigure 2 illustrates a typical FA flow used at Intel. Thefirst step is to verify the failure on a functional tester in thefailure analysis lab to ensure that the failure’s electricalsignature observed on the production tester can beduplicated on the lab’s functional tester. Fault localizationor defect localization is the next step. Its function is tonarrow down the fault to a small block of circuitry. In thiscase, a small block of circuitry could just be a via, atransistor, a gate, or even a functional unit block (FUB).Reverse engineering, or physical FA as it is better known,is the next step. Individual interconnect layers areselectively removed either mechanically or chemically andexamined for defects. Occasionally, defectcharacterization is introduced to gain more insight into thedefect’s behavior over time and under stress. This iscrucial in order to develop the best possible defect screen

to maintain a high quality product. A case is consideredclosed when a root cause is determined and correctiveaction is put in place. A corrective action could be a minorchange such as the addition of new test screens, or it couldalso involve major re-engineering via process and designfixes.

Failure Verification

Fault Isolation & DefectLocalization

Physical FA/DefectCharacterization

Root Cause Determination

Corrective action

Figure 2: A typical failure analysis flow

From the flow in Figure 2, it is clear that the key todetermining the overall FA success rate actually lies in thefault localization and defect localization steps. These stepsprovide the spatial information on where the defect is, andthey control the rest of the steps.

Fault LocalizationAs the name implies, fault localization is the technique oflocalizing a fault in a failing circuit functionally andlogically. Traditionally, this is a two-step process thatinvolves coarse-level isolation using assembly codeminimization followed by fine-level isolation using e-beam probing. With increasing device architecturecomplexity, coarse-level isolation using assembly codeminimization becomes very time consuming andineffective. Figure 3 below depicts the evolution of thetwo-step fault localization flows for different generationsof Intel’s microprocessors.

DFT and DFFA features built into newer microprocessorsto accelerate the silicon debug and production testingprocesses can also be employed as coarse-level fault-localization tools. The use of DFT and DFFA features asFA tools in conjunction with e-beam probing is proven tobe very successful on the Intel486DX and Pentiumprocessors where the internal signals of interest are stillaccessible for e-beam probing. However, the increasingnumber of interconnect layers (Intel’s Pentium® 90/100MHz processor has four metal layers, and the Pentium® II



333 MHz processor has five metal layers) and changedpackaging technologies from frontside wirebondtechnology to full flip-chip technology makes the e-beamprobing process even more challenging. The use ofRegister Transfer Level (RTL) simulation as a fine-levelisolation tool addresses this challenge.

Figure 3: An overview of various fault-isolationapproaches for the Intel386DX™ processor (top left), theIntel486DX™ and Pentium® processors (top right), andthe Pentium® Pro and Pentium II® processors (bottom)

DFT-Based Fault Isolation TechniquesThe following sections elaborate on the new faultlocalization techniques adopted from DFT featurescurrently available on the Pentium and Pentium Promicroprocessors.

MicropatchingMicrocodes are integrated instructions that dictate theprocessor’s internal operation. In traditional architecturedesign, access to built-in microcodes is restricted.However, an ability to create new microcodes and controltheir flow would be very useful for fault isolation. Withsuch a tool, the failure analyst would be able to createcustomized microcode subroutines and control microcodeflow. This capability is now realized on the Pentium Promicroprocessor family through the introduction of theMicropatching DFT feature. This capability has causedsignificant breakthroughs in debugging processors withmicrocode failures. For example, with micropatching, it isnow possible to systematically perform fault isolation on

failures in the processor’s reset subroutine, which isimplemented in microcode.

The Micropatching DFT feature consists of two keyelements: the microcode patch RAM and several pairs ofMatch and Destination registers. The microcode patchRAM stores externally programmed microcodes. (Fromhere on, programmed microcodes that reside in themicrocode RAM will be called external microcodes andbuilt-in microcodes that reside in the microcode ROM willbe called internal microcodes.)

The Match and Destination registers are used forcontrolling the microcode flow. Whenever a microcodeaddress in Microcode Instruction Pointer (UIP) matchesthe content of a Match register, the UIP will be reloadedwith a new address from the Destination register.

In order to control the microcode flow, the Matchregisters are loaded with the UIP that the user intends tojump from. Similarly, the Destination register must be setto the UIP that the user wishes to jump to.

Besides its usefulness in electrical fault isolation, theMicropatching FA tool can also be used in conjunctionwith liquid crystal analysis and emission microscopy.Without Micropatching, the processor may need toexecute many other instructions in the test patterncompletely unrelated to the failure before reaching theinstruction that causes the failure. By then, the CPU wouldhave generated a lot of heat and made the liquid crystalanalysis less sensitive to the heat generated by the failinginstruction (i.e., the heat generated by the defectivecircuitry). With Micropatching, the UIP for the resetsubroutine can be set in the Match register to point to thefailing UIP, thereby bypassing the reset subroutinealtogether. This minimizes the generation of surroundingheat that could result in genuine hot spots, caused by thedefect, to be hidden.

Array DumpMemory arrays are important elements in a processor.Memory arrays are usually used for storing data, controlsignals, processor status, etc. Visibility into these arrayshelps a failure analyst understand the internal operation ofthe device thereby speeding up the fault isolation processand reducing e-beam probing time. It is almost impossibleto analyze a dynamic execution machine such as thePentium Pro processor without knowing the contents ofthe arrays. With the Array Dump tool, the contents ofmany important arrays can be dumped out to producesnapshots of the arrays at any core clock. In order to savedata analysis time and avoid human error, a post-processing program was developed to compare theacquired data with that from RTL simulation. Mismatchesare automatically highlighted.

Coarse-level FI:assembly codeminimization

Fine-level FI:e-beam probing

Coarse-level FI:DFT based fault

localization

Fine-level FI:e-beam probing

Coarse-level FI:DFT based fault

localization

Fine Level FI:RTL simulation



The Array Dump FA tool is developed by applying twonew DFT features in the Pentium Pro processor family.The first DFT feature, called Micro Breakpoint, is one ofthe built-in debug trigger-response mechanisms thatallows the processor to execute certain tasks in responseto preprogrammed trigger events. An external debugbreakpoint pin is set up to trigger a processor “micro-breakpoint” event when the pin is asserted. In response tothis trigger event, the processor executes apreprogrammed debug command called “All ArrayFreeze” which is the second key DFT feature that makesArray Dump possible. The “All Array Freeze” featurecauses all major memory arrays in the device to freezetheir normal execution. The Array Dump feature iscapable of acquiring array data at any core cycle relativeto a processor’s external bus cycle.

ScanoutAccessibility to internal control and datapath signals alsogreatly enhances FA capability. Scanout is the FA tooldeveloped at Intel for monitoring internal control signals.Scanout has already been successfully and regularly usedin Pentium microprocessor failure analysis. One of thePentium processor’s external pins is used to trigger a TestAccess Port (TAP) instruction that causes the data fromselected control and datapath signals to be latched intoScanout data buffers. The data captured and stored inthese buffers are then shifted out serially through the TAPcontroller’s Test Data Output (TDO) pin.

The Scanout FA tool’s implementation on the Pentium Promicroprocessor is triggered differently from the Pentiumprocessor’s method. This is necessitated by theintroduction of odd bus-to-core clock ratios. To overcomethis problem, an innovative approach that uses the MicroBreakpoint DFT feature was developed similar to the onein the Array Dump tool. This Micro Breakpoint trigger-response mechanism makes it possible to capture Scanoutdata on every core cycle of the processor’s execution.

A total of around 2000 internal nodes are available forobservation. This coverage is much wider than what wasavailable on the original Pentium processor (about 200Scanout nodes) and newer versions of the Pentiumprocessor family (about 400 Scanout nodes). A programwas also written to compare the Scanout data with RTLsimulation to aid in interpreting the data. The toolautomatically highlights Scanout mismatches.

Memory DAT and LYA ModeDirect Access Test (DAT) is a special test mode that isspecifically intended for manufacturing use. Newerimplementations of the Pentium Pro processors containDAT capability to enhance the testability of most of themajor core memory structures. Once DAT mode is

enabled, the processor will behave like a pipelinedSRAM. It accepts DAT commands every bus clock andreturns data for a DAT memory read command to theexternal output bus a few clocks later.

Figure 4: Example of a cache raster bitmap displayshowing a cache column failure

In DAT mode, the processor only understands DATinstructions. These instructions incorporate the memoryarray’s address information, input data, and array accesscommands to specify a particular cache operation. Theinformation is directly fed to the array under test. Thisdirect access bypasses the decoding circuitry used innormal operation. Thus, it reduces the amount of effortrequired to determine whether a cache failure is caused bya problem in the memory array itself or a problem in thesupporting circuitry such as the address decoders.

Because only a subset of the external pins are needed inDAT mode, it is very easy to develop a small but effectivecache-testing pattern to raster the entire cache line withinone memory array. Also, the entire set of advanced cache-testing algorithms used in production testing can be easilywritten into a small test pattern. If a failure occurs inproduction, the rastering program can easily provide thefailing set and the way and bit information in a bitmapform for failure analysis purposes. Figure 4 above showsan example of the cache rastering program’s bitmapdisplay. (In this figure, bits (columns) are arrangedhorizontally, and cache lines (rows) are arrangedvertically.)

Columnfailure



However, although DAT mode testing provides the failingset and way and bit information, it provides littleinformation about the failure mechanism. In the latestversion of the Pentium Pro processor, a Low YieldAnalysis (LYA) mode was added as an enhancement toDAT mode for the largest cache structure.

Figure 5: Schematic diagram of a 6-transistor SRAM cellfor LYA mode illustration

When LYA mode is enabled, the bitline (BL) and bitline#(BL#) signals of a memory cell are connected to externalpins. By having this direct access through the externalpins, DC measurements on the memory cell can be carriedout easily. LYA mode testing produces cell transistorcurrent readings, as well as cell trip points and other DCmeasurements. From these LYA “signatures” acquiredfrom a failing device, the failure can be quicklycategorized into a few different failure mechanisms, suchas open transistor, missing contact, or shorted transistor.With this information, the defect location can be moreaccurately located, and the physical FA can focus on thatlocation. This results in shorter analysis throughput timeand improved success rates in handling cache failures.

Figure 5 above shows a 6-transistor memory cell. WhenLYA mode is enabled, the BL and BL# signals of thismemory cell are directly accessible from the external pins.Enabling the word line select (WL) signal activates thismemory cell. In this case, both transistors M1 and M6 areturned on to allow various DC measurements to beperformed on this memory cell.

PBISTProgrammable Built-In Self-Test (PBIST) is a memoryDFT feature that incorporates all the required test systemsinto the chip itself. The test systems implemented on-chipare as follows:

• algorithmic address generator

• algorithmic data generator

• program storage unit

• loop control mechanisms

PBIST was originally adopted by large memory chips thathave high pin counts and operate at high frequencies,thereby exceeding the capability of production testers. Thepurpose of PBIST is to avoid developing and buying moresophisticated and very expensive testers.

The interface between PBIST, which is internal to theprocessor, and the external tester environment is throughthe standard TAP controller pins. Algorithms and controlsare fed into the chip through the TAP controller’s TestData Input (TDI) pin. The final result of the PBIST test isread out through the TDO pin.

PBIST supports the entire algorithmic memory testingrequirements imposed by the production testingmethodology. In order to support all of the required testalgorithms, PBIST must have the capability to store therequired programs locally in the device. It must also beable to perform different address generation schemes,different test data pattern generation, looping schemes,and data comparisons.

The program storage structure is used to store the testalgorithm. The algorithm includes the types of operationsto be performed (e.g., memory read, memory write), thetypes of address generation modes, the data to be writteninto and read out from the memory array, and the types oflooping schemes.

The address generator is responsible for generating thememory address where the next data are read from orwritten into. Correct address generation is very importantbecause the physical mapping of the array is alwaysdifferent from its logical mapping. In order to achieve therequired test coverage, the address generator needs to beable to generate addresses in different fashions in order toaccommodate different kinds of addressing flows such asMarch-C, Galloping patterns, Address Complements, FastX, and Fast Y.

The data generator plays a very similar role to the addressgenerator. In order to get the inverse data for eachphysically adjacent cell, data has to be generated based onthe logical-to-physical mapping of the memory array andthe data background that is required, such ascheckerboard, reverse checkerboard, column stripe, rowstripe, and diagonal.

The loop control system is the major sequencing logic inPBIST that allows testing of the entire memory arrayusing only a few program steps. Without the loopingcontrol mechanism, test programs of thousands of linesneed to be written in order to test an entire array. With



PBIST, testing of the large on-chip memory arraysbecomes a lot simpler. Also, PBIST can be used for TestDuring Burn-In (TDBI) where the processor is tested inthe burn-in oven. Other cache-testing methods cannot beused because only the TAP controller pins and a few othercontrol pins are active in the burn-in environment whilethe rest of the pins are tristated.

RTL Simulation For FAThe motivation for doing fault isolation based on RTLsimulation is driven by the need for detailed informationabout the internal workings of the processor. Thisinformation is readily available from the RTL model.Furthermore, as described in the previous sections, theimplementation of more sophisticated DFT and debugfeatures in the Pentium, Pentium Pro, and Pentium IIgenerations of microprocessors have helped to promotethis technique by providing better observability of theinternal signals, thus resulting in less dependency on e-beam probing.

In previous generations of Intel’s microprocessors, RTLsimulation was used in the failure analysis flow primarilyfor test pattern generation, test modification, and signaltracing. But the fault isolation process largely dependedon extensive e-beam waveform probing work to trace thefailure from an architectural starting point, movingupstream until the signals acquired at the inputs of a logicblock were correct, while the output was faulty.

Because of the lack of observability of the internal nodes,e-beam probing became the necessary means to gatheringthe detailed internal signal information of amicroprocessor. However, relying on e-beam probingalone has become increasingly difficult or even futile oncurrent generations of microprocessors as more metalinterconnect layers are used (obscuring most local signalsthat run only in Metal1 and Metal2 layers). Additionally,while products are beginning to move into C4 packaging,next-generation waveform probers that allow probing ofinternal signals through the backside of the die are still notwidely available.

Figure 6: A high-level block diagram of the RTL modelsimulation environment

Advances and improvements in today’s RTL simulationtools used for Intel microprocessors result in a better andmore efficient environment for performing fault isolationthan that of the previous generation of RTL simulationtools. More sophisticated capabilities and user-friendlyfeatures have been added to the RTL simulationenvironment, such as more extensive node coverage; thecapability to use application programming interface (API)programs; an interactive simulation mode; and reliablesimulation state, save, and restore functions. Employingother design tools such as schematic viewers, layoutdatabases, and circuit-level simulation to complement theRTL simulation analysis further helps in the fault isolationprocess. Figure 6 provides an overview of the keycomponents of the RTL model simulation environment.

The simulation tool’s API enables application programs toaccess signal information from the full chip RTL model ofthe microprocessor during the model's execution. Theapplication program runs as a separate process and usesroutines provided in the API library to obtain signalinformation from the model. The primary purpose of theAPI is to isolate the model from user events, enabling thedesigner (as well as failure analyst) to quickly modifyapplication programs without having to relink the model.The advantage of such an API is its reusability. It isplatform independent and can run with several models.For the average user, the API is easier to support than auser event, and it is simpler to write and debug.

A very useful RTL simulation API is the p6watch tool,which enables visualization of large amounts of RTLsignal values by displaying them in a human format. Forexample, instead of displaying a set of control signals fora finite state machine (FSM) in binary or hexadecimalformat, the user can use p6watch to convert the signals tostrings that represent the individual states of the FSM.Another example would be to use p6watch to display the

RTL Simulation

Functional Model:HDL sources

Stimuli:ASM code

User Interface:internal or external

signal controls

Output (Trace file for test pattern orinternal signal tracing during

b )



field contents of an array in terms of their decoded orfunctional meanings.

In this method of performing fault isolation, the failurelocation is deduced based on extensive analysis of the full-chip RTL model. The failure analyst is also required tohave an understanding of the processor’smicroarchitecture to aid in interpreting the HDLinformation and the RTL simulation results, andcomparing these data with the behavior observed in thefailing device. This fault isolation method is very similarto RTL debugging during the design phase.

The RTL simulator provides crucial information on theexpected behavior of the processor. By running the teststhat cause the DUT to fail using the RTL simulator, thefailure analyst can observe the processor’s pipe stages andsee how instructions and data are being transacted andpropagated. It is also useful to run passing tests tocompare the differences in control and data handling.Among the things to look for at the high level is the flowof data along the datapath blocks and the control signalsthat arbitrate the machine pipeline. The gatheredinformation will help to determine the logic block that ismost likely to have caused the failure.

When the logic block has been isolated, more detailedRTL signal analysis is performed within that logic block.By isolating the failure to a small logic block, the analystcan generate more exhaustive and more focused tests forthat block of logic. These new focused tests are then runon the failing device to see how it responds to each testcase. For example, if a failure is isolated to a multi-portedmemory block, then the focused tests are targeted at all ofthe available ports to see if there is any specificdependency of the failure on a particular port. Other testcases would involve testing the address generation logic.

Moreover, when running a test it is possible to introduce a“defect” into the model and simulate its effects in terms ofthe processor’s response to do a “what-if” type ofanalyses.

E-beam-Less Fault IsolationE-beam-less fault isolation can produce results faster andreduce analysis by focusing on test data collection, dataanalysis, and comparison with simulation results. The costof the analysis is reduced by avoiding the e-beam probingstep in the fault isolation process. E-beam probing workinvolves many hours of sample preparation time (toperform depassivation and prepare probe holes using theFIB) as well as many more hours of waveform acquisitiontime. Avoiding the e-beam probing step can save asignificant amount of analysis time. However, in order toskip e-beam probing work, an alternative method is

required to collect valuable information from the faileddevice. This alternative approach is described in thefollowing paragraphs in this section.

First of all, the failing signature needs to be identified inorder to understand why the unit is failing from thestandpoint of the processor architecture. Then, DFT toolsare used to collect internal node information to furtherunderstand the failure mode. Later, RTL simulation isperformed to verify the hypothesis made based on thecollected data and to identify the failing node. As RTLsimulation is usually good enough to anticipate theprocessor’s internal operation and build a hypothesisabout the failure location, probing is not necessary. Whencompared to the traditional approach where probing isneeded to confirm the failing node, which takes anaverage of 31 days, this e-beam-less approach takes anaverage of only 15 days throughput time. This newapproach has been shown to produce a very accurateestimation of the defect location with greater than 95%success rate on the Pentium and Pentium Promicroprocessors. The high success rate can be attributedto the avoidance of the high-risk steps involved in e-beamprobing, especially during sample preparation. Thismethod readily and efficiently supports Intel’s virtualfactory concept where fault isolation work is easily sharedamong all participating factories. By using the FA tools,data collection can be done at the site where the failingdevice is located. These data are then sent to the sitewhere the product FI expertise resides for in-depthanalysis. The predicted defect location is then sent back tothe original site for physical FA.

Advanced FI TechniquesThe next sections of this paper present the advanced faultisolation techniques used on the Pentium and Pentium Promicroprocessor families. Specifically, these techniquesinvolve the PC FA tool and the SECC FA tool, both ofwhich were developed to address issues that will bediscussed in the following sections, and to overcome thechallenges and roadblocks described in the introduction.

PC-based FA Tester PlatformWith the increasing complexity of Intel’s microprocessordesigns, failure analysis TPT would be longer without theadoption of DFT and DFFA features in fault isolationtechniques. However, most of the DFT-based faultisolation tools and techniques have been developed foruse on expensive functional testers. To reduce thedependency on these expensive lab testers, a newapproach has been developed based on the personalcomputer (PC) platform. The key elements of the PC FAtester platform are shown in Figure 7.



Figure 7: A simplistic block diagram of the PC-based FA platform configuration

The In-Target Probe (ITP) tool has been widely used forsystem-level debug and validation. On this test platform, ahost PC is used to control a target PC where the deviceunder test (DUT) resides. The host PC is used to controlthe DUT and to upload the test program that the DUT willexecute. The ITP is developed around the processor’sProbe Mode debug feature and is able to access allprogrammer-visible and non-visible registers in the DUT.This gives the ITP the necessary control mechanisms tohalt the DUT, modify its internal state, and resume normalprogram execution at any chosen instruction boundary.

In this new approach, the ITP is used as the controller ofthe target PC where the failing pattern can be uploaded tothe target PC through the TAP controller interface. Thehost-target PC configuration acts like a tester where itdrives the necessary data to the input pins of themicroprocessor. However, the output pins of themicroprocessor are not strobed to compare with expecteddata. Rather, the output data is monitored by a hardwareboard that sends out a signal when certain conditions aremet. This signal can be used as a trigger signal, such asthe e-beam trigger signal, to enable hooking up the DUTto the e-beam, although this has only been provenexperimentally.

In order to do e-beam probing, the DUT needs to executethe test program in a loop. To achieve this, anotherhardware board is used to monitor the activities of theDUT. When a preset condition is met, the hardware boardsends out a signal that is used to reset the DUT. Oncereset, the DUT restarts execution from the normal boot-upaddress and reruns the entire test pattern.

FA tools based on the processor’s DFT features have alsobeen implemented on the PC test platform where the hostPC is used as an interface to the DUT. Commands toinvoke the DFT features are fed into the DUT through theTAP controller interface on the ITP board. The results ofthe test are dumped to the host PC through the same TAPcontroller interface.

FA tools that employ the DFT features such as Scanout,Array Dump, and cache rastering have been developed forthe PC test platform. To use tools such as Scanout andArray Dump, the DUT is initially halted. The requiredregister for stopping the DUT from execution isprogrammed through the ITP. The trigger condition forthe breakpoint is set, and the desired processor response tothe breakpoint is also programmed. Next, the originalarchitectural state is restored, and the DUT is allowed toresume execution from the point where it was halted.When the preprogrammed breakpoint condition is met, theDUT will stop, and as a result of the response to thebreakpoint, the data are shifted out through the ITP to thehost PC. These data provide information on the internalstate of the DUT to enable further fault isolation.

To perform cache rastering on the DUT, a databackground for the processor’s internal cache is preset bywriting into each memory location. Examples of thepatterns typically used for cache rastering includecheckerboard and reverse checkerboard patterns. Next, thedata in the memory array are read out and compared withthe data that were originally written. If there is anydiscrepancy between the acquired and expected data, thefailing memory cell is identified. The PC is a perfectplatform for data-retention type of memory testingbecause all commands and data are shifted seriallythrough the TAP controller. This method takes longer to



perform cache rastering as compared to lab functionaltesters, but it is cheaper, and it frees up the utilization timeof the lab testers.

By providing capabilities similar to a lab tester such asproviding stimuli to drive the DUT, enabling interface tothe e-beam, and supporting various FA tools, the PC testplatform can be used for functional FA work, a task whichhad been performed predominantly by lab testers. Thecomplete test platform is significantly cheaper and morecompact than the traditional workstation and testercombination.

A few FA cases have been successfully resolved using thisPC test platform. E-beam waveform of a clock signal hasbeen acquired indicating the feasibility of using the PC toreplace a tester.

SECC Form-Factor TestingThe concept of processor bus fractions was introduced tomake it possible to increase the processor’s core clockfrequency while at the same time maintaining the externalbus clock frequency at a lower speed. To support thisidea, a frequency multiplier is designed into the processorto multiply the external bus clock to produce a higherfrequency clock in the processor core. The term “1:2 busfraction” refers to a clock configuration in which theinternal processor frequency is twice as fast as the speedof the external bus. For example, for an external processorbus clock of 66 MHz, the internal clock frequency is 133MHz. In this situation, there are two clock domains: onerunning at 66 MHz and another running at 133 MHz.Extra care is required when developing test programs forproducts that support bus fractions: input and output datathat cross the clock boundaries must be correctly aligned.The Pentium processor only has one bus clock domainand one core clock domain. These two different clockdomains are isolated from each other by the processor’sinput and output buffers. In this case, data alignment ishandled by the processor itself, thus reducing the amountof complexity associated with aligning the data in the testpattern.

The Pentium II processor introduces a new bus fractionconcept, which increases the complexity of the test patterndevelopment and testing. The Pentium II processorimplements two separate sets of bus frequencies: afrontside and a backside. The frontside bus is connected toexternal components such as chipsets and other peripheralinterface devices. Alternatively, the backside bus is a localbus for the Pentium II processor that is connected only tothe level-2 (L2) memory. These two buses run at differentclock speeds. The backside bus frequency is always equalto or faster than the frontside bus frequency. Togetherwith the internal core clock, there are a total of threedifferent clock domains in this implementation. In this

case, the processor’s bus fraction configuration isdescribed in terms of all three clock domains relative toeach other. The term “1:4:2 bus fraction” refers to aprocessor with a backside bus running 2x faster than thefrontside bus, and a core frequency that is 4x faster thanthe frontside bus or 2x faster than the backside bus. Thedata alignment between the frontside bus and the corelogic is handled by the frontside input and output buffers.Meanwhile, the data alignment between the backside busand the core logic is handled by the backside input andoutput buffers. The user, or the test program developer, isresponsible for managing the data alignment between thefrontside and backside buses.

For a round numbered (or even) bus fraction configurationbetween the frontside bus and the backside bus, the dataalignment can be easily handled by the test program. Thisis illustrated in Figures 8 and 9 below.

Frontsidedata 1

Frontsidedata 2

Backside

Backside

Backside

Backside

Backside

Backside

Figure 8: A processor’s frontside and backside dataswitching and alignment for a 1:3 (frontside-to-backside)

bus fraction

Frontside

Backside

Backside

Backside

Backside

Frontside

Backside

Backside

Backside

Backside

Figure 9: A processor’s frontside and backside dataswitching and alignment for a 1:4 (frontside-to-backside)

bus fraction

In the examples shown in both of these figures, frontsidedata switches at the same time as backside data. Thisboundary can be used as the alignment point for both thefrontside and backside data. However, in a fractional (orodd) bus fraction configuration, data switching and dataalignment become more complicated. An example of afractional frontside-to-backside bus fraction is a 4:7(frontside to backside) configuration.

In the 4:7 example shown in Figure 10 below, when thefrontside data FD1 switches to FD2, the data are notaligned with backside data switching. The same situation



occurs during the transitions from FD2 to FD3 and FD3 toFD4.

FD1

FD22

FD 3 FD 4

BD1

BD2

BD3

BD4

BD5

BD6

BD7

Figure 10: The die frontside and backside data switchingand alignment for 4:7 (frontside-to-backside) bus fraction

The functional tester needs to simultaneously drive inputdata to the processor and strobe for the output data onboth the frontside and the backside bus. Advancedproduction testers are capable of handling these situationsbecause they offer very flexible timing capabilities wherethe rising and failing edges of the data can be placedanywhere within a tester period. This allows the edgeplacement of the first cycle to be different from the edgeplacement of the second cycle. Unfortunately, the failureanalysis test platform employed by the FA labs does notsupport advanced flexible timing capabilities. Therefore,functional testing of a Pentium II processor in a labenvironment becomes very difficult.

The fractional bus testing problem is addressed bymounting the Pentium II processor onto a Single EdgeContact Cartridge (SECC) card. The backside bus, whichused to be accessible externally, now only communicateswith the on-board L2 cache. Only the frontside bus isaccessible to the external tester through the edgeconnector.

In the SECC card form factor testing, the FA tester onlyneeds to handle the data transactions occurring on thefrontside bus; the backside bus data transactions aretransparent to the test platform. Hence, the complexfrontside:core:backside bus fraction is reduced to asimple frontside:core bus fraction. This setup enables thenormal FA functional tester to execute test vectors that usecomplex bus fractions.

To support SECC testing on a normal FA tester, a patternconversion tool is needed to convert a pattern fromcomponent format into SECC format. This includesextracting the frontside bus data and realigning it intoSECC format. This concept has been proven to be verysuccessful in analyzing the latest Pentium IImicroprocessors.

Future Directions In Failure AnalysisTechniquesThe advent of more sophisticated DFT and debugfeatures, coupled with faster RTL simulators and bettercomputer-aided fault debugging tools increases the usageof the fault isolation methods described in this paper. Thetrend towards more extensive use of computer basedanalysis tools is continuing. With the introduction of newDFT features which provide high degrees ofcontrollability in addition to observability in nextgeneration Intel processors, the concept of performingcomputer-aided fault diagnosis at the logic or RTL level istaken to the next abstraction level of the design, wherefault diagnosis is done at the gate level using intelligentcomputer-aided fault diagnostic tools. By having bothobservability and controllability of many importantinternal signals, the concept of virtual probing will berealized.

ConclusionAs device complexity and interconnect layers increase,new fault isolation techniques need to be continuouslydeveloped in order to maintain a high analysis success rateand short throughput time. As discussed in this paper,performing FA using traditional methods has been shownto lower FA success rates. By quickly isolating the defectlocation and identifying the failure mechanism that iscausing low manufacturing yields or abnormal failurerates, problems can be fixed in a short amount of time toimprove manufacturing yields. In addition, the correctiveactions developed as a result of these FA techniques aredocumented and proliferated to newer products in theform of design rules. This will prevent the problems fromrecurring, which in turn enables even a steeper volumeramp. The innovative FA techniques developed for thePentium and Pentium Pro processors have been one of thekey elements in the continued exceptional performance ofIntel’s product time-to-market and high-volumemanufacturing.

References[1] Yeoh Eng Hong, Martin Tay, “The Application of

Novel Failure Analysis Techniques for AdvancedMulti-Layered CMOS Devices,” International TestConference, 1997.

[2] Adrian Carbine, Derek Feltham, “Pentium ® ProProcessor Design for Test and Debug,” InternationalTest Conference, 1997.

Authors’ BiographiesYeoh Eng Hong graduated from Monash University in1992. He joined Intel as a microprocessor failure analysis



engineer. He is now the microprocessor product FAmanager in Intel Penang. His main technical interests aredeveloping new FA/FI techniques and studying newfailure mechanisms. His e-mail address [email protected].

Seong Leong, Lim graduated from University ScienceMalaysia in 1992. He joined Intel as a microprocessorfailure analysis engineer. He is now leading the PentiumPro microprocessor Fault Isolation group in Intel Penang.His main technical interests are microprocessorarchitecture and fault isolation. His e-mail address [email protected].

Wong Yik Choong graduated from the University ofMalaya in 1994. He joined Intel as a microprocessorfailure analysis engineer. He is now the lead Pentium IIprocessor FA engineer in the Penang Microprocessor FAdepartment. His main technical interests are computerarchitecture and networking, and software development.His e-mail address [email protected]

Lock Choon Hou graduated from the University ofMalaya in 1996. He joined Intel as a microprocessorproduct failure analysis engineer. His e-mail address [email protected]

Mahmud Adnan received a BSEE from BradleyUniversity, Peoria, Illinois in 1990. He joined IntelPenang in 1991 as a microprocessor failure analysisengineer. He is currently working on Merced™processor’s DFT design validation and Merced debug andFA tools development. His e-mail address [email protected].

Computer Vision Face Tracking For Use in a Perceptual User Interface 1

Computer Vision Face Tracking For Use in a Perceptual UserInterface

Gary R. Bradski, Microcomputer Research Lab, Santa Clara, CA, Intel Corporation

Index words: computer vision, face tracking, mean shift algorithm, perceptual user interface, 3Dgraphics interface

AbstractAs a first step towards a perceptual user interface, acomputer vision color tracking algorithm is developed andapplied towards tracking human faces. Computer visionalgorithms that are intended to form part of a perceptualuser interface must be fast and efficient. They must beable to track in real time yet not absorb a major share ofcomputational resources: other tasks must be able to runwhile the visual interface is being used. The newalgorithm developed here is based on a robust non-parametric technique for climbing density gradients tofind the mode (peak) of probability distributions called themean shift algorithm. In our case, we want to find themode of a color distribution within a video scene.Therefore, the mean shift algorithm is modified to dealwith dynamically changing color probability distributionsderived from video frame sequences. The modifiedalgorithm is called the Continuously Adaptive Mean Shift(CAMSHIFT) algorithm. CAMSHIFT’s tracking accuracyis compared against a Polhemus tracker. Tolerance tonoise, distractors and performance is studied.

CAMSHIFT is then used as a computer interface forcontrolling commercial computer games and for exploringimmersive 3D graphic worlds.

IntroductionThis paper is part of a program to develop a PerceptualUser Interface for computers. Perceptual interfaces areones in which the computer is given the ability to senseand produce analogs of the human senses, such asallowing computers to perceive and produce localizedsound and speech, giving computers a sense of touch andforce feedback, and in our case, giving computers anability to see. The work described in this paper is part of alarger effort aimed at giving computers the ability tosegment, track, and understand the pose, gestures, andemotional expressions of humans and the tools they mightbe using in front of a computer or settop box. In this paperwe describe the development of the first core module in

this effort: a 4-degree of freedom color object tracker andits application to flesh-tone-based face tracking.

Computer vision face tracking is an active and developingfield, yet the face trackers that have been developed arenot sufficient for our needs. Elaborate methods such astracking contours with snakes [[10][12][13]], usingEigenspace matching techniques [14], maintaining largesets of statistical hypotheses [15], or convolving imageswith feature detectors [16] are far too computationallyexpensive. We want a tracker that will track a given facein the presence of noise, other faces, and handmovements. Moreover, it must run fast and efficiently sothat objects may be tracked in real time (30 frames persecond) while consuming as few system resources aspossible. In other words, this tracker should be able toserve as part of a user interface that is in turn part of thecomputational tasks that a computer might routinely beexpected to carry out. This tracker also needs to run oninexpensive consumer cameras and not require calibratedlenses.

In order, therefore, to find a fast, simple algorithm forbasic tracking, we have focused on color-based tracking[[7][8][9][10][11]], yet even these simpler algorithms aretoo computationally complex (and therefore slower at anygiven CPU speed) due to their use of color correlation,blob and region growing, Kalman filter smoothing andprediction, and contour considerations. The complexity ofthe these algorithms derives from their attempts to dealwith irregular object motion due to perspective (nearobjects to the camera seem to move faster than distalobjects); image noise; distractors, such as other faces inthe scene; facial occlusion by hands or other objects; andlighting variations. We want a fast, computationallyefficient algorithm that handles these problems in thecourse of its operation, i.e., an algorithm that mitigates theabove problems “for free.”

To develop such an algorithm, we drew on ideas fromrobust statistics and probability distributions. Robuststatistics are those that tend to ignore outliers in the data(points far away from the region of interest). Thus, robust



algorithms help compensate for noise and distractors inthe vision data. We therefore chose to use a robust non-parametric technique for climbing density gradients tofind the mode of probability distributions called the meanshift algorithm [2]. (The mean shift algorithm was neverintended to be used as a tracking algorithm, but it is quiteeffective in this role.)

The mean shift algorithm operates on probabilitydistributions. To track colored objects in video framesequences, the color image data has to be represented as aprobability distribution [1]; we use color histograms toaccomplish this. Color distributions derived from videoimage sequences change over time, so the mean shiftalgorithm has to be modified to adapt dynamically to theprobability distribution it is tracking. The new algorithmthat meets all these requirements is called CAMSHIFT.

For face tracking, CAMSHIFT tracks the X, Y, and Areaof the flesh color probability distribution representing aface. Area is proportional to Z, the distance from thecamera. Head roll is also tracked as a further degree offreedom. We then use the X, Y, Z, and Roll derived fromCAMSHIFT face tracking as a perceptual user interfacefor controlling commercial computer games and forexploring 3D graphic virtual worlds.

Choose initialsearch window

size and locationHSV Image

Set calculationregion at searchwindow centerbut larger insize than thesearch window

Color histogram look-up in calculation

region

Color probabilitydistribution

Find center of masswithin the search

window

Center search windowat the center of massand find area under it

ConvergedYES NOReport X,Y, Z, and

Roll

Use (X,Y) to setsearch windowcenter, 2*area1/2

to set size.

Figure 1: Block diagram of color object tracking

Figure 1 summarizes the algorithm described below. Foreach video frame, the raw image is converted to a colorprobability distribution image via a color histogram modelof the color being tracked (flesh for face tracking). Thecenter and size of the color object are found via theCAMSHIFT algorithm operating on the color probabilityimage (the gray box is the mean shift algorithm). Thecurrent size and location of the tracked object are reportedand used to set the size and location of the search windowin the next video image. The process is then repeated forcontinuous tracking.

Video DemonstrationsThe following three videos demonstrate CAMSHIFT inaction.

1. FaceTrack_Fast.avi

2. FaceTrack_Distractors.avi

3. FaceTrack_HandOcclusion.avi

The first video shows CAMSHIFT tracking rapid facemovements. The second video shows CAMSHIFTtracking a face with other faces moving in the scene. Thethird video shows CAMSHIFT tracking a face throughhand occlusions. These videos are available from thispaper on the Web in the Intel Technology Journal Q2’98under the site http://developer.intel.com/technology/itj.

Color Probability DistributionsIn order to use CAMSHIFT to track colored objects in avideo scene, a probability distribution image of thedesired color (flesh color in the case of face tracking) inthe video scene must be created. In order to do this, wefirst create a model of the desired hue using a colorhistogram. We use the Hue Saturation Value (HSV) colorsystem [5][6] that corresponds to projecting standard Red,Green, Blue (RGB) color space along its principlediagonal from white to black (see arrow in Figure 2). Thisresults in the hexcone in Figure 3. Descending the V axisin Figure 3 gives us smaller hexcones corresponding tosmaller (darker) RGB subcubes in Figure 2.

HSV space separates out hue (color) from saturation (howconcentrated the color is) and from brightness. We createour color models by taking 1D histograms from the H(hue) channel in HSV space.

For face tracking via a flesh color model, flesh areas fromthe user are sampled by prompting users to center theirface in an onscreen box, or by using motion cues to findflesh areas from which to sample colors. The hues derivedfrom flesh pixels in the image are sampled from the Hchannel and binned into an 1D histogram. When samplingis complete, the histogram is saved for future use. Morerobust histograms may be made by sampling flesh hues



from multiple people. Even simple flesh histograms tendto work well with a wide variety of people without havingto be updated. A common misconception is that differentcolor models are needed for different races of people, forexample, for blacks and whites. This is not true. Exceptfor albinos, humans are all the same color (hue). Dark-skinned people simply have greater flesh color saturationthan light-skinned people, and this is separated out in theHSV color system and ignored in our flesh-tracking colormodel.

During operation, the stored flesh color histogram is usedas a model, or lookup table, to convert incoming videopixels to a corresponding probability of flesh image as canbe seen in the right-hand image of Figure 6. This is donefor each video frame. Using this method, probabilitiesrange in discrete steps from zero (probability 0.0) to themaximum probability pixel value (probability 1.0). For 8-bit hues, this range is between 0 and 255. We then trackusing CAMSHIFT on this probability of flesh image.

When using real cameras with discrete pixel values, aproblem can occur when using HSV space as can be seenin Figure 3. When brightness is low (V near 0), saturationis also low (S near 0). Hue then becomes quite noisy,since in such a small hexcone, the small number ofdiscrete hue pixels cannot adequately represent slightchanges in RGB. This then leads to wild swings in huevalues. To overcome this problem, we simply ignore huepixels that have very low corresponding brightness values.This means that for very dim scenes, the camera mustauto-adjust or be adjusted for more brightness or else itsimply cannot track. With sunlight, bright white colors cantake on a flesh hue so we also use an upper threshold toignore flesh hue pixels with corresponding highbrightness. At very low saturation, hue is not defined sowe also ignore hue pixels that have very lowcorresponding saturation (see Implementation Detailssection below).

Originally, we used a 2D color histogram built fromnormalized red green (r,g) space (r = R/(R+G+B), g =G/(R+G+B)). However, we found that such color modelsare much more sensitive to lighting changes sincesaturation (which is influenced by lighting) is notseparated out of that model.

YellowGreen

Red

MagentaBlue

CyanWhite

Figure 2: RGB color cube

Green,1200

Red,00

Blue,2400

Yellow

Magenta

Cyan 1.0White

0.0Black

H S

V

Figure 3: HSV color system

CAMSHIFT DerivationThe closest existing algorithm to CAMSHIFT is known asthe mean shift algorithm [2][18]. The mean shift algorithmis a non-parametric technique that climbs the gradient of aprobability distribution to find the nearest dominant mode(peak).

How to Calculate the Mean Shift Algorithm

1. Choose a search window size.2. Choose the initial location of the search window.3. Compute the mean location in the search window.4. Center the search window at the mean location

computed in Step 3.5. Repeat Steps 3 and 4 until convergence (or until the

mean location moves less than a preset threshold).

Proof of Convergence [18]Assuming a Euclidean distribution space containingdistribution f, the proof is as follows reflecting the stepsabove:1. A window W is chosen at size s.



2. The initial search window is centered at data point pk

3. Compute the mean position within the search window

∑∈

=Wj jp

WWkp .

1)(ˆ

The mean shift climbs the gradient of f(p)

.)(

)()(ˆ

kpf

kpf

kpWkp′

≈−

4. Center the window at point

).(ˆ Wpk

5. Repeat Steps 3 and 4 until convergence.Near the mode ,0)( ≅′ pf so the mean shift algorithm

converges there.

For discrete 2D image probability distributions, the meanlocation (the centroid) within the search window (Steps 3and 4 above) is found as follows:

Find the zeroth moment

∑∑=x y

yxIM ).,(00

Find the first moment for x and y

∑∑∑∑ ==x yx y

yxyIMyxxIM ).,(. );,( 0110

Then the mean search window location (the centroid) is

; ;00

01

00

10

M

My

M

Mx cc ==

where I(x,y) is the pixel (probability) value at position(x,y) in the image, and x and y range over the searchwindow.

Unlike the Mean Shift algorithm, which is designed forstatic distributions, CAMSHIFT is designed fordynamically changing distributions. These occur whenobjects in video sequences are being tracked and theobject moves so that the size and location of theprobability distribution changes in time. The CAMSHIFTalgorithm adjusts the search window size in the course ofits operation. Initial window size can be set at anyreasonable value. For discrete distributions (digital data),the minimum window size is three as explained in theImplementation Details section. Instead of a set orexternally adapted window size, CAMSHIFT relies on thezeroth moment information, extracted as part of theinternal workings of the algorithm, to continuously adaptits window size within or over each video frame. One canthink of the zeroth moment as the distribution “area”

found under the search window. Thus, window radius, orheight and width, is set to a function of the the zerothmoment found during search. The CAMSHIFT algorithmis then calculated using any initial non-zero window size(greater or equal to three if the distribution is discrete).

How to Calculate the Continuously Adaptive MeanShift Algorithm

1. Choose the initial location of the search window.2. Mean Shift as above (one or many iterations); store

the zeroth moment.3. Set the search window size equal to a function of the

zeroth moment found in Step 2.4. Repeat Steps 2 and 3 until convergence (mean

location moves less than a preset threshold).

In Figure 4 below, CAMSHIFT is shown beginning thesearch process at the top left step by step down the leftthen right columns until convergence at bottom right. Inthis figure, the red graph is a 1D cross-section of an actualsub-sampled flesh color probability distribution of animage of a face and a nearby hand. In this figure, yellow isthe CAMSHIFT search window, and purple is the meanshift point. The ordinate is the distribution value, and theabscissa is the horizontal spatial position within theoriginal image. The window is initialized at size three andconverges to cover the tracked face but not the hand in sixiterations. In this sub-sampled image, the maximumdistribution pixel value is 206 so we set the width of thesearch window to be 2*M0/206 (see discussion of windowsize in the Implementation Details section below). In thisprocess, CAMSHIFT exhibits typical behavior: it finds thecenter of the nearest connected distribution region (theface), but ignores nearby distractors (the hand).

1 4 7

10 13 16 19 22

0

50

100

150

200

250

Step 1

1 4 7

10 13 16 19 22

0

50

100

150

200

250

Step 2

1 3 5 7 9

11 13 15 17 19 21 23

0

50

100

150

200

250

Step 3

1 3 5 7 9

11 13 15 17 19 21 230

50

100

150

200

250

Step 4

1 3 5 7 9

11 13 15 17 19 21 23

0

50

100

150

200

250

Step 5

1 3 5 7 9

11 13 15 17 19 21 23

0

50

100

150

200

250

Step 6

Figure 4: CAMSHIFT in operation down the left thenright columns



Figure 4 shows CAMSHIFT at startup. Figure 5 belowshows frame to frame tracking. In this figure, the red colorprobability distribution has shifted left and changed form.At the left in Figure 5, the search window starts at itsprevious location from the bottom right in Figure 4. In oneiteration it converges to the new face center.

1 4 7

10 13 16 19 22

0

50

100

150

200

250

Step 1

1 4 7

10 13 16 19 22

0

50

100

150

200

250

Step 2

Figure 5: Example of CAMSHIFT tracking starting fromthe converged search location in Figure 4 bottom right

Mean Shift Alone Does Not WorkThe mean shift algorithm alone would fail as a tracker. Awindow size that works at one distribution scale is notsuitable for another scale as the color object movestowards and away from the camera. Small fixed-sizedwindows may get lost entirely for large object translationin the scene. Large fixed-sized windows may includedistractors (other people or hands) and too much noise.

CAMSHIFT for Video SequencesWhen tracking a colored object, CAMSHIFT operates ona color probability distribution image derived from colorhistograms. CAMSHIFT calculates the centroid of the 2Dcolor probability distribution within its 2D window ofcalculation, re-centers the window, then calculates thearea for the next window size. Thus, we needn’t calculatethe color probability distribution over the whole image,but can instead restrict the calculation of the distributionto a smaller image region surrounding the currentCAMSHIFT window. This tends to result in largecomputational savings when flesh color does not dominatethe image. We refer to this feedback of calculation regionsize as the Coupled CAMSHIFT algorithm.

How to Calculate the Coupled CAMSHIFT Algorithm

1. First, set the calculation region of the probabilitydistribution to the whole image.

2. Choose the initial location of the 2D mean shiftsearch window.

3. Calculate the color probability distribution in the 2Dregion centered at the search window location in anarea slightly larger than the mean shift window size.

4. Mean shift to convergence or for a set number ofiterations. Store the zeroth moment (area or size) andmean location.

5. For the next video frame, center the search window atthe mean location stored in Step 4 and set the window

size to a function of the zeroth moment found there.Go to Step 3.

For each frame, the mean shift algorithm will tend toconverge to the mode of the distribution. Therefore,CAMSHIFT for video will tend to track the center (mode)of color objects moving in a video scene. Figure 6 showsCAMSHIFT locked onto the mode of a flesh colorprobability distribution (mode center and area are markedon the original video image). In this figure, CAMSHIFTmarks the face centroid with a cross and displays itssearch window with a box.

Figure 6: A video image and its flesh probability image

Calculation of Head RollThe 2D orientation of the probability distribution is alsoeasy to obtain by using the second moments during thecourse of CAMSHIFT’s operation where (x,y) range overthe search window, and I(x,y) is the pixel (probability)value at (x,y):

Second moments are

).,( );,( 220

220 yxIxMyxIxM

x yx y∑∑∑∑ ==

Then the object orientation (major axis) is

2

2

arctan2

00

022

00

20

00

11

−−

−

−

=cc

cc

yM

Mx

M

M

yxM

M

θ

The first two Eigenvalues (major length and width) of theprobability distribution “blob” found by CAMSHIFT maybe calculated in closed form as follows [4]. Let

,2

00

20cx

M

Ma −=

,200

11

−= cc yx

M

Mb

and



,2

00

02cy

M

Mc −=

Then length l and width w from the distribution centroidare

,2

)()( 22 cabcal

−+++=

.2

)()( 22 cabcaw

−+−+=

When used in face tracking, the above equations give ushead roll, length, and width as marked in Figure 7.

Figure 7: Orientation of the flesh probability distributionmarked on the source video image

CAMSHIFT thus gives us a computationally efficient,simple to implement algorithm that tracks four degrees offreedom (see Figure 8).

Video Camera

x

y

z

First 4 tracked degrees of freedom: Head movement

Roll

Figure 8: First four head tracked degrees of freedom: X,Y, Z location, and head roll

How CAMSHIFT Deals with Image ProblemsWhen tracking color objects, CAMSHIFT deals with theimage problems mentioned previously of irregular objectmotion due to perspective, image noise, distractors, andfacial occlusion as described below.

CAMSHIFT continuously re-scales itself in a way thatnaturally fits the structure of the data. A colored object’spotential velocity and acceleration scale with its distanceto the camera, which in turn, scales the size of its colordistribution in the image plane. Thus, when objects areclose, they can move rapidly in the image plane, but theirprobability distribution also occupies a large area. In thissituation, CAMSHIFT’s window size is also large and so

can catch large movements. When objects are distant, thecolor distribution is small so CAMSHIFT’s window sizeis small, but distal objects are slower to traverse the videoscene. This natural adaptation to distribution scale andtranslation allows us to do without predictive filters orvariables–a further computational saving–and serves as anin-built antidote to the problem of erratic object motion.

CAMSHIFT’s windowed distribution gradient climbingcauses it to ignore distribution outliers. Therefore,CAMSHIFT produces very little jitter in noise and, as aresult, tracking variables do not have to be smoothed orfiltered. This gives us robust noise tolerance.

CAMSHIFT’s robust ability to ignore outliers also allowsit to be robust against distractors. Once CAMSHIFT islocked onto the mode of a color distribution, it will tend toignore other nearby but non-connected color distributions.Thus, when CAMSHIFT is tracking a face, the presenceof other faces or hand movements in the scene will notcause CAMSHIFT to loose the original face unless theother faces or hand movements substantially occlude theoriginal face.

CAMSHIFT’s provable convergence to the mode ofprobability distributions helps it ignore partial occlusionsof the colored object. CAMSHIFT will tend to stick to themode of the color distribution that remains.

Moreover, when CAMSHIFT’s window size is setsomewhat greater than the root of the distribution areaunder its window, CAMSHIFT tends to grow toencompass the connected area of the distribution that isbeing tracked (see Figure 4). This is just what is desiredfor tracking whole objects such as faces, hands, andcolored tools. This property enables CAMSHIFT to notget stuck tracking, for example, the nose of a face, butinstead to track the whole face.

Implementation Details

Initial Window Size and PlacementIn practice, we work with digital video images so ourdistributions are discrete. Since CAMSHIFT is analgorithm that climbs the gradient of a distribution, theminimum search window size must be greater than one inorder to detect a gradient. Also, in order to center thewindow, it should be of odd size. Thus for discretedistributions, the minimum window size is set at three. Forthis reason too, as CAMSHIFT adapts its search windowsize, the size of the search window is rounded up to thecurrent or next greatest odd number. In practice, at startup, we calculate the color probability of the whole sceneand use the zeroth moment to set the window size (seesubsection below) and the centroid to set the windowcenter.



Setting Adaptive Window Size Function

Deciding what function of the zeroth moment to set thesearch window size to in Step 3 of the CAMSHIFTalgorithm depends on an understanding of the distributionthat one wants to track and the goal that one wants toachieve. The first consideration is to translate the zerothmoment information into units that make sense for settingwindow size. Thus, in Figure 4, the maximum distributionvalue per discrete cell is 206, so we divide the zerothmoment by 206 to convert the calculated area under thesearch window to units of number of cells. Our goal isthen to track the whole color object so we need anexpansive window. Thus, we further multiply the result bytwo so that the window grows to encompass the connecteddistribution area. We then round to the next greatest oddsearch window size so that the window has a center.

For 2D color probability distributions where the maximumpixel value is 255, we set window size s to

.256

*2 00Ms =

We divide by 256 for the same reason stated above, but toconvert the resulting 2D region to a 1D length, we need totake the square root. In practice, for tracking faces, we setwindow width to s and window length to 1.2s since facesare somewhat elliptical.

Comments on Software Calibration

Much of CAMSHIFT’s robustness to noise, transientocclusions, and distractors depends on the search windowmatching the size of the object being tracked—it is betterto err on the side of the search window being a little toosmall. The search window size depends on the function ofthe zeroth moment M00 chosen above. To indirectlycontrol the search window size, we adjust the colorhistogram up or down by a constant, truncating at zero orsaturating at the maximum pixel value. This adjustmentaffects the pixel values in the color probability distributionimage which affects M00 and hence window size. For 8-bithue, we adjust the histogram down by 20 to 80 (out of amaximum of 255), which tends to shrink the CAMSHIFTwindow to just within the object being tracked and alsoreduces image noise.

HSV brightness and saturation thresholds are employedsince hue is not well defined for very low or highbrightness or low saturation. Low and high thresholds areset off 10% of the maximum pixel value.

Comments on Hardware Calibration

To use CAMSHIFT as a video color object tracker, thecamera’s field of view (zoom) must be set so that it coversthe space that one intends to track in. Turn off automatic

white balance if possible to avoid sudden color shifts. Tryto set (or auto-adjust) AGC, shutter speed, iris or CCDintegration time so that image brightness is neither toodim nor saturating. The camera need not be in focus totrack colors. CAMSHIFT will work well with cheapcameras and does not need calibrated lenses.

CAMSHIFT’S Use as a Perceptual Interface

Treatment of CAMSHIFT Tracking Variablesfor Use in a Perceptual User InterfaceFigure 8 above shows the variables X, Y, Z, and Rollreturned by the CAMSHIFT face tracker. For game andgraphics control, X, Y, Z, and Roll often require a“neutral” position; that is, a position relative to whichfurther face movement is measured. For example, if thecaptured video image has dimensions (Y, X) of 120x160,a typical neutral position might be Y=60, X=80. Then if X< 80, the user has moved 80-X left; if X > 80, the user hasmoved X-80 right and so on for each variable.

Piecewise Linear Transformation of Control Variables

To obtain differential control at various positionsincluding a jitter-damping neutral movement region, eachvariable’s relative movement (above or below the neutralposition) is scaled in “N” different ranges.

In the X variable example above, if X is in range #1, Xwould be scaled by “X scale 1”; if X is in range #2, Xwould be scaled by “X scale 2” and so on.

The formula for mapping captured video head position Pto a screen movement factor F is

F = min(b1,P)s1 + [min(b2-b1, P-b1)]+s2 + … + [min(b(i+1)-

bi, P-bi)]+s(i+1)+ … + [P-b(N-1)]

+sN, (control equation 1)

where [#]+ equals “#” if # > 0, and zero otherwise; min(A,B) returns the minimum of A or B; b1-bN represents thebounds of the ranges, s1-sN are the corresponding scalefactors for each range, and P is the absolute value of thedifference of the variable’s location from neutral.

This allows for stability close to the neutral position, withgrowing acceleration of movement as the distance awayfrom neutral increases, in a piecewise linear manner asshown in Figure 9.



3D C

ontr

ol, F

Visually tracked position, P

Figure 9: Piecewise linear transformation of CAMSHIFTposition P interface control variable F

Frame Rate Adjustment

If the graphic’s rendering rate R can be determined, thefinal screen movement S is

S = F/R (control equation 2)

Computer graphics and game movement commands S canbe issued on each rendered graphic’s frame. Thus, it isbest for the movement amount S to be sensitive to framerate. In computer-rendered graphic or game scenes, simpleviews (for example, looking up at blue sky) are renderedmuch faster than complex views (for example, texturemapped city skylines). The final rate of movement shouldnot depend on the complexity of the 3D view if one wantsto achieve satisfying motion when immersed in a 3Dscene.

Y Variable Special Case for Seated User

If a user sits facing the camera (neutral X,Y) and thenleans left or right (X movement) by pivoting on his/herchair, Y will decrease as shown in Figure 10 below.

Y

X

13.75

x

A

b

c

Figure 10: Lean changes Y and head roll

In order to overcome this, we use an empirical observationthat the 2nd Eigenvalue (face half width) of the local faceflesh color distribution length is proportional to face size(see Figure 7), which is, on average, often proportional tobody size. Empirically, the ratio of the 2nd Eigenvector totorso length from face centroid is

1 to 13.75 (2 inches to 27.5 inches).(control equation 3)

Given lean distance x (in 2nd Eigenvector units), andseated size of 13.75, as in Figure 10 so that sin(A) =x/13.75. Then,

A = sin-1(x/13.75), (control equation 4)

so c = 13.75cos(A), and

b = 13.75( 1 - cos(A)) (control equation 5)

in units of 2nd Eigenvectors. This is the Y distance tocorrect for (add back) when leaning.

Roll Considerations

As can be seen from Figure 10, for seated users lean alsoinduces a change in head roll by the angle A. Thus, forcontrol that relies on head roll, this lean-induced rollshould be corrected for. Correction can be accomplishedin two ways:

• Make the first range boundary b1 in control equation1 large enough to contain the changes in faceorientation that result from leaning. Then use a scalevalue s1 = 0 so that leaning causes no roll.

• Subtract the measured roll from the lean-induced roll,A, calculated in control Equation 4 above.

Another possible problem can result when the user looksdown too much as shown in Figure 11. In this case, theuser is looking down at the keyboard. Looking down toomuch causes the forehead to dominate the view which inturn causes the face flesh color “blob” to look like it isoriented horizontally.

Figure 11: Extreme down head pitch causes a corruptedhead roll value

To correct for such problems, we define a new variable, Qcalled “Roll Quality.” Q is the ratio of the first twoEigenvalues, length l and width w, of the distributioncolor “blob” in the CAMSHIFT search window:

Q = l/w. (control equation 6)

For problem views of the face such as in Figure 11, weobserve that Roll Quality is nearly 1.0. So, for face



tracking, roll should be ignored (treated as vertical) forquality measures less than 1.25. Roll should also beignored for very high quality scores greater than 2.0 sincesuch distributions are un-facelike and likely to haveresulted from noise or occlusions.

CAMSHIFT’s Actual Use as an InterfaceCAMSHIFT is being used as a face tracker to controlgames and 3D graphics. By inserting face controlvariables into the mouse queue, we can controlunmodified commercial games such as Quake 2 shown inFigure 12. We used left and right head movements to slidea user left and right in the game, back and forth headmovements to move the user backwards and forwards, upor down movements to let the user shoot (as if ducking orgetting jolted by the gun), and roll left or right to turn theuser left or right in the game. This methodology has beenused extensively in a series of demos with over 30different users.

Head tracking via CAMSHIFT has also been used toexperiment with immersive 3D graphics control in whichnatural head movements are translated to moving thecorresponding 3D graphics camera viewpoint. This hasbeen extensively tested using a 3D graphics model of theForbidden City in China as well as in exploring a 3Dgraphics model of the big island of Hawaii as shown inFigure 13. Most users find it an enjoyable experience inwhich they naturally pick up how to control the graphicsviewpoint movement.

Figure 12: CAMSHIFT-based face tracker used to playQuake 2 hands free by inserting control variables into the

mouse queue

Figure 13: CAMSHIFT-based face tracker used to “fly”over a 3D graphic’s model of Hawaii

CAMSHIFT Analysis

Comparison to PolhemusIn order to assess the tracking accuracy of CAMSHIFT,we compared its accuracy against a Polhemus tracker.Polhemus is a magnetic sensor connected to a system thatmeasures six degrees of spatial freedom and thus can beused for object tracking when tethered to an object. Theobserved accuracy of Polhemus is +/- 1.5cm in spatiallocation and about 2.5o in orientation within 30 inches ofthe Polhemus antenna. We compared Polhemus trackingto CAMSHIFT color object tracking using a 320x240image size (see Figure 14a-d). The coordinate systems ofPolhemus and the camera were carefully aligned prior totesting. The object tracked was pulled on a cart in a settrajectory away from the Polhemus origin. Thecomparison between CAMSHIFT and Polhemus in eachof X, Y, Z, and Roll yielded the results shown Table 1.

TrackingVariable

X Y Z Roll

StandardDeviationofDifference

0.27cm 0.58cm 3.4cm 2.4o

Table 1: Standard deviation of Polhemus vs. CAMSHIFTtracking differences

Z exhibited the worst difference because CAMSHIFTdetermines Z by measuring color area, which is inherentlynoisy. X, Y, and Roll are well within Polhemus’s observedtracking error and therefore indistinguishable. Z is about2cm off. Except for Z, these results are as good or betterthan much more elaborate vision tracking systems [17],although CAMSHIFT does not yet track pitch and yaw.



��

��

��

��

��

��

��

��

��

X - axis translationStandard Deviation: 0.27

0

5

10

15

20

25

30

35

40

45

50

0 5 10 15 20 25 30 35 40 45

Actual Distance (cm)

Mea

sure

d D

ista

nce

(cm

)

Figure 14a: Comparision of X tracking accuracy

��

��

��

��

��

��

��

��

��

Y - axis translationStandard Deviation: 0.58

0

5

10

15

20

25

0 5 10 15 20


Mea

sure

d D

ista

nce

(cm

)

Figure 14b: Comparision of Y tracking accuracy

Z - axis translationStandard Deviation: 3.44

-90

-80

-70

-60

-50

-40

-30

-20

-10

0

0 6 12 18 24 30 36 42 48 54 60 66 72 78 84 90


Mea

sure

d D

ista

nce

(cm

)

Figure 14c: Comparision of Z tracking accuracy

��

��

��

��

��

��

��

��

��

��

��

��

T - axis rotationStandard Deviation: 2.46

0

20

40

60

80

100

120

140

160

Actual Distance (deg)

Mea

sure

d D

ista

nce

(deg

)

Figure 14d: Accuracy comparison of Polhemus andCAMSHIFT tracking for roll.

Tracking in NoiseCAMSHIFT’s robust ability to find and track the mode ofa dynamically changing probability distribution also givesit good tracking behavior in noise. We videotaped a headmovement sequence and then played it back adding 0, 10,30, and 50% uniform noise. Figure 15 shows 50% noiseadded to the raw image on the left, and the resulting colorprobability distribution on the right. Note that the use of acolor model greatly cuts down the random noise sincecolor noise has a low probability of being flesh color.



Nevertheless, the flesh color model is highly degraded andthere are many spurious flesh pixels in the colorprobability distribution image. But CAMSHIFT is stillable to track X, Y, and Roll quite well in up to 30% whitenoise as shown in Figure 16a-d. Z is more of a problembecause CAMSHIFT measures Z by tracking distributionarea under its search window, and one can see in Figure15 that area is highly effected by noise. Y shows anupward shift simply because the narrower chin regionexhibits more degradation in noise than the widerforehead. Roll tracks well until noise is such that thelength and width of the face color distribution areobscured. Thus, CAMSHIFT handles noise well withoutthe need for extra filtering or adaptive smoothing.

Figure 15: Tracking in 50% uniform noise

X Noise

-60

-50

-40

-30

-20

-10

0

10

20

30

40

0 20 40 60 80

Time

Dis

tanc

e fr

om S

tart

Pos

ition

0% |Avg Dif|

10% 2.9

30% 1.2

50% 7.6

Figure 16a: X accuracy in 0-50% uniform noise

Y Noise

-30

-20

-10

0

10

20

30

40

0 20 40 60 80

Time

Dis

tanc

e fr

om S

tart

Pos

ition

0% |Avg Dif|

10% 1.9

30% 1.5

50% 7.6

Figure 16b: Y accuracy in 0-50% uniform noise

Z Noise

0

10

20

30

40

50

60

70

80

0 20 40 60 80

Time

Dis

tanc

e F

rom

Sta

rt P

ositi

on

0% |Avg Dif|

10% 2.7

30% 10.6

50% 21.5

Figure 16c: Z accuracy in 0-50% uniform noise

T Noise

0

20

40

60

80

100

120

140

160

0 20 40 60 80

Time

Fac

e O

rient

atio

n (d

eg)

0% |Avg Dif|

10% 2.6deg

30% 4.1deg

50% 8.3deg

Figure 16d: Roll accuracy in 0-50% uniform noise



Tracking in the Presence of DistractionsCAMSHIFT’s search window converges to span thenearest dominant connected probability distribution. If weadjust the nature of the probability distribution byproperly setting the HSV brightness and saturationthreshold (see Implementation Details section above), thesearch window will tend to stay just within the objectbeing tracked as shown in the marked image at top left inFigure 17. In such cases, CAMSHIFT is robust againstdistracting (nearby) distributions and transient occlusions.This robustness occurs for distractors because the searchwindow rarely contains the distractor as shownsequentially down the left, then right, columns of Figure17.

Figure 17: Tracking a face with background distractorfaces (sequence: down left then right columns)

Table 2 shows the results collected from 44 sample pointon five tracking runs with active background facedistraction such as that shown in Figure 17. Since thedistracting face rarely intersects much of CAMSHIFT’ssearch window, the X, Y, and Z tracking variables areperturbed very little. Roll is more strongly affected sinceeven a small intersection of a distractor in CAMSHIFT’ssearch window can change the effective orientation of theflesh pixels as measured by CAMSHIFT.

TrackedVariable

Average Std.Deviation

Maximum Std.Deviation

X (pixels) 0.42 2.00

Y(pixels) 0.53 1.79

Z(pixels) 0.54 1.50

Roll (degrees) 5.18 46.80

Table 2: Perturbation of CAMSHIFT tracking variablesby face distractors

CAMSHIFT tends to be robust against transient occlusionbecause the search window will tend to first absorb theocclusion and then stick with the dominant distributionmode when the occlusion passes. Figure 18 demonstratesrobustness to hand occlusion in sequential steps down theleft, then right columns.

Figure 18: Tracking a face in the presence of passinghand occlusions (sequence: down left then right columns)

Table 3 shows the results collected from 43 sample pointson five tracking runs with active transient hand occlusionof the face. Average perturbation is less than three pixelsfor X, Y, and Z. Roll is more strongly effected due to thearbitrary orientation of the hand as it passes through thesearch window.



TrackedVariable

Average Std.Deviation

Maximum Std.Deviation

X (pixels) 2.35 7.17

Y(pixels) 2.81 6.29

Z(pixels) 2.10 4.65

Roll (degrees) 14.64 34.40

Table 3: Perturbation of CAMSHIFT tracking variablesby passing hand occlusion

We see from the above table that CAMSHIFT gives uswide tolerance for distraction and occlusion “for free” dueto the statistically robust workings of the algorithm.

PerformanceThe order of complexity of CAMSHIFT is Ο(αN2) whereα is some constant, and the image is taken to be NxN. α ismost influenced by the moment calculations and theaverage number of mean shift iterations until convergence.The biggest computational savings come through scalingthe region of calculation to an area around the searchwindow size as previously discussed.

CAMSHIFT was run on a 300 MHz Pentium® IIprocessor, using an image size of 160x120 at 30 framesper second (see Figure 19). CAMSHIFT’s performancescales with tracked object size. Figure 19 shows the CPUload from the entire computer vision thread includingimage acquisition, conversion to color probabilitydistribution, and CAMSHIFT tracking. In Figure 19 whenthe tracked face is far from the camera, the CAMSHIFTthread consumes only 10% of the CPU cycles. When theface fills the frame, CAMSHIFT consumes 55% of theCPU.

Figure 19: Performance scales inversely with trackedobject size (ordinate is percent of CPU used)

Figure 20 traces computer vision thread’s performance inan actual control task of “flying” over a 3D model ofHawaii using head movements. In this case, the averageCPU usage was 29%. VTUNETM analysis showed that theactual CAMSHIFT operation (excluding image capture,color conversion or image copying) consumed under 12%of the CPU. CAMSHIFT relies on Intel’s MMX™technology optimized Image Processing Library availableon the Web [3] to do RGB-to-HSV image conversion andimage allocation. MMX technology optimized imagemoments calculation has recently been added to the ImageProcessing Library, but not in time for publication. Theuse of such optimized moment calculations will boostperformance noticeably since this forms part of the innermean shift calculation loop. Even without thisimprovement, CAMSHIFT’s current average actual useefficiency of 29% allows it to be used as a visual userinterface.



Figure 20: In actual use, CAMSHIFT consumed anaverage of 29% of one 300 MHz Pentium® II CPU when

used to control a 3D graphic’s Hawaii fly through

DiscussionThis paper discussed a core tracking module that is part ofa larger effort to allow computers to track and understandhuman motion, pose, and tool use. As such, the modulewas designed to be simple and computationally efficient.Yet, this core module must still handle the basiccomputer-vision problems outlined in this paper. We’veseen that CAMSHIFT handles these problems as follows:• Irregular object motion : CAMSHIFT scales its

search window to object size thus naturally handlingperspective-induced motion irregularities.

• Image noise: The color model eliminates much of thenoise, and CAMSHIFT tends to ignore the remainingoutliers.

• Distractors: CAMSHIFT ignores objects outside itssearch window so objects such as nearby faces andhands do not affect CAMSHIFT’s tracking.

• Occlusion: As long as occlusion isn’t 100%,CAMSHIFT will still tend to follow what is left of theobjects’ probability distribution.

• Lighting variation: Using only hue from the HSVcolor space and ignoring pixels with high or lowbrightness gives CAMSHIFT wide lighting tolerance.

CAMSHIFT’s simplicity does cause limitations however.Since CAMSHIFT derives Z from object area estimates, Zis subject to noise and spurious values. The effects ofnoise are evident in Figure 16c. That CAMSHIFT can getspurious area values is evident in Figure 11.

Since CAMSHIFT relies on color distributions alone,errors in color (colored lighting, dim illumination, toomuch illumination) will cause errors in tracking. Moresophisticated trackers use multiple modes such as featuretracking and motion analysis to compensate for this, but

more complexity would undermine the original designcriterion for CAMSHIFT.

CAMSHIFT also only detects four (X, Y, Z, and Roll) ofthe six modes of freedom (above plus pitch and yaw).Unfortunately, of the six degrees of head movementpossible, Roll is the least useful control variable since it isthe least “natural” head movement and is thereforefatiguing for the user to use constantly.

ConclusionCAMSHIFT is a simple, computationally efficient faceand colored object tracker. While acknowledging thelimitation imposed by its simplicity, we can still see thatCAMSHIFT tracks virtually as well as more expensivetethered trackers (Polhemus) or much more sophisticated,computationally expensive vision systems [17], and ittracks well in noisy environments. Thus, as we haveshown, even though CAMSHIFT was conceived as asimple part of a larger tracking system, it has many usesright now in game and 3D graphics’ control.

Adding perceptual interfaces can make computers morenatural to use, more fun for games and graphics, and abetter medium of communication. These new featuresconsume more MIPs and so will take advantage of moreMIPs available with future Intel® CPUs.

In this project, we designed a highly efficient face trackingalgorithm rather than a more complex, higher MIPs usagealgorithm. This was done because we want to be able todemonstrate compelling applications and interfaces ontoday’s systems in order to prepare the way for the futureuse of computer vision on PCs. CAMSHIFT is usable as avisual interface now, yet designed to be part of a morerobust, larger tracking system in the future. CAMSHIFTwill be incorporated into larger, more complex, higherMIPs-demanding modules that provide more robusttracking, posture understanding, gesture and facerecognition, and object understanding. In this way, thefunctionality of the computer vision interface will increasewith increasing Intel CPU speeds. A user will thus be ableto upgrade their computer vision interface by upgrading tohigher speed Intel CPUs in the future.

AcknowledgmentsMany thanks to Mark Holler for countless discussions andfeedback leading to the development of this work. Specialthanks to Ryan Boller for rapid implementation of theCAMSHIFT game control demo and for major help inimplementing the testing routines for CAMSHIFT.



References[1] D. Comaniciu and P. Meer, “Robust Analysis of

Feature Spaces: Color Image Segmentation,”CVPR’97, pp. 750-755.

[2] K. Fukunaga, “Introduction to Statistical PatternRecognition,” Academic Press, Boston, 1990.

[3] MMX TM technology optimized libraries in image,signal processing, pattern recognition and matrixmath can be downloaded fromhttp://developer.intel.com/design/perftool/perflibst/index.htm)

[4] W.T. Freeman, K. Tanaka, J.Ohta, and K. Kyuma,“Computer Vision for Computer Games,” Int. Conf.On Automatic Face and Gesture Recognition, pp.100-105, 1996.

[5] A.R. Smith, “Color Gamut Transform Pairs,”SIGGRAPH 78, pp. 12-19, 1978.

[6] J.D. Foley, A. van Dam, S. K. Feiner and J.F.Hughes, “Computer graphics principles and practice,”Addison-Wesley, pp. 590-591.

[7] P. Fieguth and D. Terzopoulos, “Color-based trackingof heads and other mobile objects at video framerates,” In Proc. Of IEEE CVPR, pp. 21-27, 1997.

[8] C. Wren, A. Azarbayejani, T. Darrell, A.Pentland,“Pfinder: Real-Time Tracking of the Human Body,”SPIE Vol. 2615, 1995.

[9] M. Hunke and A. Waibel, “Face locating and trackingfor human-computer interaction,” Proc. Of the 28th

Asilomar Conf. On Signals, Sys. and Comp., pp.1277-1281, 1994.

[10] K. Sobottka and I. Pitas, “Segmentation and trackingof faces in color images,” Proc. Of the Second Intl.Conf. On Auto. Face and Gesture Recognition, pp.236-241, 1996.

[11] M. Swain and D. Ballard, “Color indexing,” Intl. J. ofComputer Vision, 7(1) pp. 11-32, 1991.

[12] M. Kass, A. Witkin D.Terzopoulos, “Snakes: Activecontour Models,” Int. J. o f Computer Vision (1) #4,pp. 321-331, 1988.

[13] C. Vieren, F. Cabestaing, J. Postaire, “Catchingmoving objects with snakes for motion tracking,”Pattern Recognition Letters (16) #7, pp. 679-685,1995.

[14] A. Pentland, B. Moghaddam, T. Starner, “View-basedand Modular Eigenspaces for face recognition,”CVPR’94, pp. 84-91, 1994.

[15] M. Isard, A. Blake, “Contour tracking by stochasticpropagation of conditional density,” Proc. 4th

European Conf. On Computer Vision, Cambridge,UK, April 1996.

[16] T. Maurer, and C. von der Malsburg, “Tracking andlearning graphs and pose on image sequence of

faces,” Proc. Of the Second Intl. Conf. On Auto. Faceand Gesture Recognition, pp. 176-181, 1996.

[17] A. Azabayejani, T. Starner, B. Horowitz, and A.Pentland, “Visually Controlled Graphics,” IEEETran. Pat. Anal. and Mach. Intel. pp. 602-605, Vol.15, No. 6, June 1993.

[18] Y. Cheng, “Mean shift, mode seeking, andclustering,” IEEE Trans. Pattern Anal. MachineIntell., 17:790-799, 1995.

Author’s BiographyDr. Gary R. Bradski received a Ph.D. in patternrecognition and computer vision from Boston University.He works in computer vision research in theMicrocomputer Research Labs at Intel’s Mission Collegecampus. His interests include segmenting and trackingpeople in visual scenes; perceptual computer interfaces;applying visual 3D structure to 3D graphics; patternrecognition; biological perception and self-organization.His e-mail is [email protected].

1

Proving the IEEE Correctness of IterativeFloating-Point Square Root, Divide, and Remainder

Algorithms

Marius Cornea-Hasegan, Microprocessor Products Group, Hillsboro, OR, Intel Corp.

Index words: floating-point, IEEE correctness, divide, square root, remainder

Abstract

The work presented in this paper was initiated as part of astudy on software alternatives to the hardwareimplementations of floating-point operations such asdivide and square root. The results of the study proved theviability of software implementations, and showed thatcertain proposed algorithms are comparable inperformance to current hardware implementations. Thispaper discusses two components of that study:

(1) A methodology for proving the IEEE correctness ofthe result of iterative algorithms that implement thefloating-point square root, divide, or remainderoperation.

(2) Identification of operands for the floating-pointdivide and square root operations that lead to resultsrepresenting difficult cases for IEEE rounding.

Some general properties of floating-point computationsare presented first. The IEEE correctness of the floating-point square root operation is discussed next. We showhow operands for the floating-point square root that leadto difficult cases for rounding can be generated, and howto use this knowledge in proving the IEEE correctness ofthe result of iterative algorithms that calculate the squareroot of a floating-point number. Similar aspects areanalyzed for the floating-point divide operation, and wepresent a method for generating difficult cases forrounding. In the case of the floating-point divideoperation, however, it is more difficult to use thisinformation in proving the IEEE correctness of the resultof an iterative algorithm than it is for the floating-pointsquare root operation. We examine the restrictions on themethod used for square root. Finally, we present possiblelimitations due to the finite exponent range.

IntroductionFloating-point divide, remainder, and square root are threeimportant operations performed by computing systemstoday. The IEEE-754 Standard for Binary Floating-PointArithmetic [1] requires that the result of a divide or squareroot operation be calculated as if in infinite precision, andthen rounded to one of the two nearest floating-pointnumbers of the specified precision that surround theinfinitely precise result (the remainder will always beexact).

Most processor implementations to date have usedhardware-based implementations for divide, remainder,and square root. Recent research has led to software-basediterative algorithms for these operations that are expectedto always generate the IEEE-correct result. Severaladvantages can be envisioned. Firstly, software-basedalgorithms lead to a possibly higher throughput thanbefore because they are capable of being pipelined;secondly, they are easier to modify; and finally, theyreduce the actual chip size because they do not have to beimplemented in hardware.

Different algorithms are used depending on the targetprecision of the result, or on the particular architecture ofthe processor. In either case, performance and IEEEcorrectness have to be ensured. The expected levels ofperformance are possible due to improvements in thefloating-point architecture of most modern processors.Proving the IEEE correctness of the results generated bythe divide, remainder, and square root operations, which isof utmost importance for modern processors, is the objectof this paper. A new methodology for achieving this goalis presented. Operands that lead to "difficult" cases forrounding are also identified, allowing better testing of theimplementations for these operations. The methodpresented was verified through a mix of mathematicalproofs, sustained by Mathematica [2], C language, andassembly language programs.

Intel Technology Journal Q2’98

2

Correctness proofs following methods similar to thosepresented in this paper, in conjunction with careful andthorough verification and testing of the actualimplementation, should ensure flawless floating-pointdivide, remainder, and square root operations on modernprocessors that adopt iterative, software-based algorithms.

Some of the mathematical notations used in the followingsections are explained at the end of this paper.

General Properties of Floating-PointComputations and IEEE Correctness

Floating-point numbers are represented as a concatenationof a sign bit, an M-bit exponent field, and an N-bitsignificand field. Mathematically

f = σ⋅s⋅2e

where σ =±1, s∈[1,2), e∈[emin, emax] ∩ Z, s = 1+k /2N-1,k∈{0,1,2,…,2N-1-1}, emin = -2M-1+2, and emax = 2M-1–1. LetFN be the set of floating-point numbers with N-bitsignificands and unlimited exponent range, and let FM,N bethe set of floating-point numbers with M-bit exponentsand N-bit significands (no special values included). Notethat FM,N ⊂ FN ⊂ Q*.

The IEEE-754 Standard for Binary Floating-PointArithmetic allows several formats, but the most commonare single precision (M=8, N=24), double precision(M=11, N=53), and double-extended precision (M=15,N=64). Even though there is only a finite number of realvalues that can be represented as floating-point numbers(which constitute a finite subset of the rational numbers),the total of 2⋅2M⋅2N floating-point numbers or specialvalues for a given precision can be huge. Verifyingcorrectness of a binary floating-point operation fordouble-extended precision by exhaustive testing on astate-of-the-art workstation could easily take 250 years.The only operation that lends itself to such testing, fromamong those considered herein, is the single precisionsquare root.

The variable density of the floating-point numbers on thereal axis implies that it would be difficult to have arequirement regarding the absolute error whenapproximating real values by floating-point numbers.Instead, most iterative algorithms are analyzed by tryingto limit the relative error of the result being generated.

The IEEE-754 standard requires that the result of a divideor square root operation be calculated as if in infiniteprecision, and then rounded to one of the two nearest

floating-point numbers of the specified precision thatsurround the infinitely precise result. Special cases occurwhen this is outside the supported range. The IEEEstandard specifies four rounding modes: to nearest (rn), tonegative infinity (rm), to positive infinity (rp), and towardzero (rz).

In order to determine whether an iterative algorithm for anIEEE operation yields the correctly rounded result (in anyrounding mode), we have to evaluate the error that occursdue to rounding in each computational step. Twomeasures are commonly used for this purpose. The first isthe error of an approximation with respect to the exactresult, expressed in fractions of an ulp, or unit in the lastplace. For the floating-point number f = σ⋅s⋅2e∈FN givenabove, one ulp has the magnitude

1 ulp = 2e-N+1

An alternative is to use the relative error. If the realnumber [ is approximated by the floating-point number D,then the relative error ε is determined by

a = x⋅(1+ε)

The non-linear relationship between ulps and thecorresponding relative error is illustrated in Figure 1(where the outer envelopes mark the minimum andmaximum values of the relative error), by Theorem 1, andalso by Corollaries 1a and 1b of this theorem. Forexample, if [ > 1.000 in Figure 4, but [ is much closer to1.000 than to 1.001, then if we approximate [ by D=1.001,the relative error is close to the largest possible (whenstaying within 1 ulp of the approximation), i.e., close to ε= 1/8 (1/8 = 2-N+1 for N = 4).

Figure 1: Relative error when approximating within1 ulp, positive real numbers by floating-point numbers

in F4

Theorem 1 (ulp-s versus relative error) Let x∈R* be areal number, and a∈FN, a = σ⋅s⋅2e, |σ| = 1, s = 1+k/2N-1,k∈Z,0 ≤ k ≤ 2N-1-1, e∈Z and m∈R, 0 < m ≤ 1. Then


3

(a) For k≠0: a = σ⋅(1+k/2N-1)⋅2e ≅ x within m ulp of x ⇔a = x⋅(1+ε), ε ∈ (-m /(2N-1+k+m), m/(2N-1+k–m))

(b) For k=0: a = σ⋅2e ≅ x within m ulp of x ⇔a = x⋅(1+ε), ε∈(-m/(2N-1+m), m/(2N–m))

(For the sake of brevity, no proofs are included in thispaper.)

Several corollaries can be derived from this property, butthe following two are often useful.

Corollary 1a. Let x∈R*, and a∈FN*. If a ≅ x, within1 ulp of x, then a = x⋅(1+ε), with |ε| < 1/2N-1 .

Corollary 1b. Let x∈R*, and a∈FN*. If a = x⋅(1+ε), with|ε| < 1/2N , then a ≅ x, within 1 ulp of x.

Some properties used in IEEE correctness proofs areformulated in terms of errors expressed in ulp-s. However,it is easier to evaluate the errors introduced by eachcomputational step in an iterative algorithm as relativeerrors. The two properties above, together with othersimilar ones, are used in going back and forth from oneform to the other, as needed.

In order to show that the final result generated by afloating-point algorithm represents the correctly roundedvalue of the infinitely precise result [, one has to showthat the exact result and the final result of the algorithmbefore rounding, \, lie within such an interval that,through rounding, they end up at the same floating-pointnumber. Figure 2 illustrates these intervals over a binade(an interval delimited by consecutive integer powers of thebase 2) when (a) only rounding to nearest, and (b) anyIEEE rounding mode is acceptable. In this figure, thefloating-point numbers are marked on the bottom realaxis.

First we will discuss the square root, and then we willexamine the divide and remainder operations.

Figure 2: Intervals that condition (a) xrn = yrn, and(b) xrnd = yrnd

IEEE Correctness Proofs for Iterative Floating-Point Square Root Algorithms

We will first present the means to determine difficultcases for rounding in floating-point square rootoperations, and then we will show how to use thisknowledge in proving the IEEE correctness of the result ofiterative algorithms implementing the floating-pointsquare root operation.

Iterative Algorithms for Floating-Point Square Root

Among the most common iterative algorithms forcalculating the value of the square root of a floating-pointnumber are those based on the Newton-Raphson method.Such algorithms are suggested in [3]. Starting with aninitial guess of 1/√D, a very good approximation of 1/√D isderived first in a number of iterations. From this, the valueof √D can be calculated. For example, an algorithm thatstarts by determining the value of the root off ([) = 1/[2 - D , could have the following iteration:

ei = (1/2 – 1/2⋅a⋅yi2)rn

yi+1 = (yi + ei⋅yi )rn

where ei is the error term, yi+1 is the next betterapproximation, and the subscript rn denotes the IEEErounding to nearest mode.

No matter what iterative algorithm is being used, one caneasily evaluate the relative errors introduced in eachcomputational step. We will show that if the relative errorthat we can derive is small enough, then the IEEEcorrectness of the generated result is proven. In practicethough, the relative error of the final result is more oftenthan not larger than required. We can compensate for thedifference between what we can determine in terms ofrelative error and what is needed, by knowing the valuesof input arguments that lead to the most difficult cases forrounding. The following subsections will show how toachieve this.

Difficult Cases for Rounding

The lemma shown below allows difficult cases forrounding to be determined. For rounding to nearest, theseare represented by values of the argument D of the squareroot function such that √D is different from, but very closeto, a midpoint between two consecutive floating-pointnumbers. For rounding toward negative infinity, positiveinfinity, or zero, √D has to be different from, but very

2 2 2

2p+1

2p

p-N p-N+1 p-N+2

(b)

(a)


4

close to, a floating-point number. We will show next howto determine the most difficult cases in each category.

Lemma 1. (a) Let a∈FN, a > 0, a = σ⋅s⋅2e. Then its valuecan be written as

a = A ⋅ 2e-2N+2 for e = 2⋅u, u∈Z, ora = A ⋅ 2e-2N+1 for e = 2⋅u+1, u∈Z

where A∈[22N-2, 22N-1), A ≡ 0 (mod 2N-1) for e=2⋅u, u∈Z,and A∈[22N-1, 22N), A ≡ 0 (mod 2N) for e=2⋅u+1, u∈Z.

(b) √a∈FN if and only if √A∈[2N-1, 2N-1/2 )∩Z for e=2⋅u,u∈Z, and √a∈FN if and only if √A∈[2N–1/2, 2N )∩Z fore=2⋅u+1, u∈Z (note that √A has to be an integer in bothcases).

(c) √a is a midpoint between two consecutive floating-point numbers in FN if and only if √A is an integer + 1/2,√A∈[2N-1, 2N).

(d) Let a∈FN. If √a∉FN, then √a∉FN+1 (this is to say that√a cannot be a midpoint between two consecutivefloating-point numbers in FN; the observation is necessary,as FN ⊂ FN+1).

The properties above will help us determine the difficultcases for rounding. Consider first the cases of roundingtoward negative infinity, positive infinity, or zero. If√D∉FN (it is not representable as a floating-point numberwith N bits in the significand), then how close can it be toa floating-point number I∈FN? Because the values of √$fall in [2N-1, 2N), where floating-point numbers with N-bitsignificands are integers and 1 ulp = 1, this can be re-formulated as “how close can √$ be to an N-bit integernumber )?” The answer is given in Figure 3.

Figure 3: The distance G of $ to the square of aninteger )∈[2N-1, 2N)

This figure shows that our question is directly related to“how close can A be to the square of an N-bit number in[2N-1, 2N)?” To find out, we have to solve the equation

(2N-1 + k)2 = A + δ

for increasing values of δ∈Z*, and for 0 ≤ k ≤ 2N-1–1,k∈Z. This leads to two cases, depending on whether theexponent H of D is even or odd.

If H is even, according to Lemma 1, the equation abovebecomes

(2N-1 + k)2 = 22N-2 + m ⋅ 2N-1 + δ

for 0 ≤ m ≤ 2N-1–1, m∈Z.

If H is odd, according also to Lemma 1, the equationbecomes

(2N-1 + k)2 = 22N-1 + m ⋅ 2N + δ

for 0 ≤ m ≤ 2N-1–1, m∈Z.

Solving the diophantine equations above (equations ininteger numbers) for δ = 1,-1,2,-2,3,-3,… we find at mosttwo acceptable solutions for each value of δ. For example,for H even and δ = 1, the only solution isA = 22N-2+2⋅2N-1, which is at a distance d = δ = 1 of (2N-1+1)2, the square of F = 2N-1+1. Thevalue of D�that corresponds to this $ (D�obtained as shownin Lemma 1), has a significand of 1.00…010B, and itconstitutes a difficult case for rounding toward negativeinfinity, positive infinity, or zero, as √D is different from,but very close to, a floating-point number in FN (it is infact the “most difficult” such case). For example, for N =24 and H = 0

√1.00000000000000000000010B =1.00000000000000000000000111111…B

is very close to, but less than,

1.00000000000000000000001B.

Consider next the case of rounding to nearest. If √D∉FN

(it is not representable as a floating-point number with Nbits in the significand), then how close can it be to amidpoint between two consecutive floating-point numbersin FN? This can be re-formulated as “how close can √$ beto an N-bit integer number plus 1/2?” The answer is givenin Figure 4. This figure shows that our question is directlyrelated to “how close can $ be to the square of a midpointbetween two consecutive N-bit integer numbers inFN∩[2N-1, 2N)?” To find out, we have to solve theequation

(2N-1 + k + 1/2)2 = A + 1/4 + δ

F2

A 22N2N-2

2

22N-1

A

Af(A) =

d

2N

FN-1/22

A

2 N-1


5

Figure 4: The distance d’ of A to the square of M,M = integer +1/2∈[2N-1, 2N)

for increasing values (in absolute value) of δ∈Z, and for0 ≤ k ≤ 2N-1–1, k∈Z. This leads again to two cases,depending on whether the exponent H of D is even or odd.

If H is even, according to Lemma 1, the equation abovebecomes

(2N-1 + k + 1/2)2 = 22N-2 + m ⋅ 2N-1 + 1/4 + δ

for 0 ≤ m ≤ 2N-1–1, m∈Z.

If H is odd, according also to Lemma 1, the equationbecomes

(2N-1 + k + 1/2)2 = 22N-1 + m ⋅ 2N + 1/4 + δ

for 0 ≤ m ≤ 2N-1–1, m∈Z.

Solving the diophantine equations above for δ = 0,1,-1,2,-2,3,-3, … we find again at most two acceptable solutionsfor each value of δ. For example, for H even and δ = 0, theonly solution is A = 22N-2+2N-1, which is at a distanced = δ+1/4 = 1/4 of (2N-1+1/2)2, the square ofM = 2N-1+1/2. The value of D that corresponds to this $ (Dobtained as shown in Lemma 1), has a significand of1.00…001B, and it constitutes a difficult case for roundingto nearest, as √D is different from, but very close to, amidpoint between two consecutive floating-point numbersin FN (it is in fact the “most difficult” such case). Forexample, for N = 24 and H = 0

√1.00000000000000000000001B =1.000000000000000000000000111111… B

is very close to, but less than,

1.00000000000000000000000B + 1/2 ulp.

Not all solutions to the equations above can berepresented by simple formulas. For example, for H evenand δ = -2, we obtain

A = 7e08a5000000H and a = 1.f82294H⋅2 e if N = 24(√a very close to 1.673f4aH⋅2e/2+1/2 ulp)

A = 1d407bb3641da50000000000000H and a = 1. d407bb3641da5H⋅2 e if N = 53 (√a veryclose to 1.5a24e31b39fa5H⋅2e/2+1/2 ulp)

A = 4d7f90be2ec18ed98000000000000000H anda = 1.35fe42f8bb063b66H⋅2e if N = 64 (√a veryclose to 1.19b4bb639c98c0b4H ⋅ 2 e/2 + 1/2 ulp)

This method of determining values of the argument D thatlead to values of √D that are difficult to round is alsouseful in generating good quality test vectors for anyimplementation of the floating-point square rootoperation, not just for iterative algorithms (these valuescan be added to others chosen by different criteria). Inaddition, this method can help in proving the IEEEcorrectness of the result of iterative algorithms thatcalculate the square root of a floating-point number, asshown below.

General Properties

Two main properties, Theorems 2 and 3, are used in theproposed method of proving the IEEE correctness of theresult of iterative floating-point square root algorithms.

Theorem 2. Let a∈FN, a>0, a = σ⋅s⋅2e. If √a∉FN, and Ais determined by scaling a as specified in Lemma 1, thenfor any integer F∈[2N-1, 2N), and for any f∈FN

(a) The distance w√A between √A and F satisfies w√A = | √A – F | > 1/2N+1

(b) The distance w√a between √a and f satisfies w√a = | √a - f | > 2e/2-2N , if e = 2⋅u, andw√a = | √a - f | > 2e/2-2N-1/2, if e = 2⋅u+1, u ∈ Z

This theorem states that if √D is not representable as afloating-point number with an N-bit significand, then thereare exclusion zones of known minimal width around anyfloating-point number, within which √D cannot exist. Theminimum distance between √D and I, or equivalently,between √$ and ), was determined by examining insteadthe distance between $ and )2, as shown in Figure 3. Theminimal width of the exclusion zones can be extended bygradually eliminating the difficult cases for roundingtoward negative infinity, positive infinity, or zero (startingwith the most difficult cases). For example, if we solve thediophantine equations that determine these cases (as

2A 2

2N2N-22

22N-1

A

Af(A) =

2 N

N-1/22

A

2 N-1

M

M

d’


6

shown above) just for δ = 1,-1,2,-2,3, and -3, twosolutions are found for any N, and statements (a) and (b)in Theorem 2 are modified to reflect exclusion zones thatare four times wider (in (b), the two inequalities werereplaced by the more restrictive one)

(a) The distance w√A between √A and F satisfiesw√A = | √A - F | > 1/2N-1, except forA1 = 22N-2 + 2⋅2N–1 if e = 2⋅u, u∈ Z, andA2 = 22N - 2⋅2N if e = 2⋅u+1, u∈Z

(b) The distance w√a between √a and f satisfiesw√a = | √a - f | > 2e/2-2N+3/2, except fora1 = (1+2–N+2 ) ⋅ 2e if e = 2⋅u, u∈Z, anda2 = (2–2–N+2 ) ⋅ 2e if e = 2⋅u+1, u∈Z

This property is used in conjunction with the one fromTheorem 3.

Theorem 3. Let a∈FN, a > 0, a = σ⋅s⋅2e. If √a∉FN, and Ais determined by scaling a as specified in Lemma 1, thenfor any midpoint m between two consecutive numbers inFN, and for any midpoint M between two consecutiveintegers in [2N-1, 2N), M = k+1/2∈[2N-1, 2N), k∈Z

(a) The distance w√A’ between √A and M satisfiesw√A’ = | √A - M | > 1/2N+3

(b) The distance w√a’ between √a and m satisfiesw√a’ = | √a - m | > 2e/2-2N-2, if e = 2⋅u, andw√a’ = | √a - m | > 2e/2-2⋅N-5/2, if e = 2⋅u+1, u∈Z

This theorem states that if √D is not representable as afloating-point number with an N-bit significand, then thereare exclusion zones of known minimal width around anymidpoint between two consecutive floating-pointnumbers, within which √D cannot exist. The minimumdistance between √D and P, or equivalently, between √$and 0, could be determined by examining instead thedistance between $ and 02, as shown in Figure 4. Just asin the case of Theorem 2, the minimal width of theexclusion zones can be extended by gradually eliminatingthe difficult cases for rounding toward nearest (startingwith the most difficult cases). For example, if we solve thediophantine equations that determine these cases (asshown above) just for δ = 0,1,-1,2,-2,3,-3, and –4, sevensolutions are found for N=24, N=53, and N=64 (but notalways from the same equation for all the three values ofN), and statements (a) and (b) in Theorem 3 are modifiedto reflect exclusion zones that are 17 times wider (in (b),the two inequalities were replaced by the more restrictiveone):

(a) The distance w√A' between √A and M satisfiesw√A' = | √A - M | > 17/2N+3

except for a set of known values A1' through A7'.

(b) The distance w√a' between √a and m satisfiesw√a' = | √a - m | > 17⋅2e/2-2N-5/2

except for a set of known values a1' through a7'.

Some of the actual values of ai' and of the correspondingAi' (not shown here) were determined directly frommathematical expressions, but others were determined byC programs, using recursion. Note that it is not necessaryto have the same number of solutions for different valuesof N.

This property is used, as explained below, in conjunctionwith that in the preceding Theorem 2.

IEEE Correctness of Iterative Floating-Point SquareRoot Algorithms

In order to prove that an iterative algorithm for calculatingthe square root of a floating-point number D generates aresult that is IEEE correct, we have to show that the resultR* before rounding and the exact value of √D lead,through rounding, to the same floating-point number. Ifthe result R* before rounding and the exact √D are closerto each other than half of the minimum width of anyexclusion zone determined as shown in Theorems 2 and 3,then (R*)rnd = (√a)rnd for any IEEE rounding mode rnd.This is illustrated in Figure 2 (b) above and in Figure 5.

Figure 5: Exclusion zones around floating-pointnumbers I and around midpoints P between

consecutive floating-point numbers

As in Theorem 2 (a), 1/2N+1 = 1/2N+1 ulp in FN = 1 ulp inF2N + 1, and in Theorem 3 (a), 1/2N+3 = 1/2N+3 ulp in FN =1 ulp in F2N+3, the above can be expressed in the followingcorollaries of these two theorems:

Corollary 2. If a∈FN, a > 0, R*∈R and |R* - √a| ≤ 1 ulpin F2N+1, then (R*)rnd = (√a)rnd for any rounding modernd∈{ rm, rp, rz}

Corollary 3. If a∈FN, a > 0, R*∈R and |R* - √a| ≤ 1 ulpin F2N+3, then (R*)rn = (√a)rn

In practice, the inequalities to be verified are

m f m f m f m

R*a

a

δ

a2 w2 w’


7

|R* - √a| ≤ (w√a)min

|R* - √a| ≤ (w√a’)min

The method of proving the IEEE correctness of the resultof an iterative algorithm that calculates the square root ofa floating-point number can then be summarized as asequence of steps:

Step 1. Evaluate the relative error ε of the final resultbefore rounding

R* = √a⋅(1+ε)

where an upper bound for |ε| is known as |ε| ≤ 2-p, forsome given p∈R, p > 0.

Step 2. Evaluate |R* - √a| = |√a⋅ε| ≤ √a⋅2-p and determinethe minimum widths (w√a)min and (w√a’)min for which thefollowing hold

|R* - √a| ≤ (w√a)min

|R* - √a| ≤ (w√a’)min

Step 3. For Theorems 2 and 3, determine a sufficientnumber of difficult cases for rounding, to allowaugmenting the widths of the exclusion zones to the valuesdetermined in Step 2 (the smaller this new minimumwidth, the fewer the points we have to determine). Exceptpossibly for these values, the iterative algorithm forcalculating √D generates a result that is IEEE correct.Alternatively, instead of steps 1, 2, 3 above, we could useTheorem 1 to verify that the conditions in Corollaries 2and 3 are satisfied.

Step 4. Verify directly that the algorithm generates IEEEcorrect results for the special points determined in Step 3.Once this is done, the algorithm is proven to generateIEEE correct results for all the possible input values of theargument.

The method presented above was applied to severalsoftware-based iterative algorithms for calculating thesquare root of a floating-point number. It proved to be ofgood practical value, as the number of special points to beverified directly was never larger than 20. It thereforerepresents a viable option for verifying correctness of thisclass of algorithms.

IEEE Correctness Proofs for IterativeFloating-Point Divide and RemainderAlgorithms

As we did for square root, we will first present a methodfor determining difficult cases for rounding in floating-

point divide operations. We will show, however, that aperfect parallel with the square root is not possible.

The IEEE correctness of the floating-point remainderoperation (which is always exact, and therefore notaffected by the rounding mode) is a direct consequence ofthe IEEE correctness of the result of the floating-pointdivide operation, as

rem(a,b) = a – b⋅near(a/b)

where near([) is the nearest integer to the real number [.Two problems arise. First, we must verify that near(a/b)fits in an integer ([4] explains this). Second, we mustcounter the possibility of a double rounding error becausewhat we calculate is actually

rem(a,b) = a – b⋅near ((a/b)rn)

To do this, we just need to compare (a/b)rn and (a/b)rz, andapply a correction if needed. Note that this might occuronly in the tie cases for the rounding to nearest performedin near((a/b)rn).

It is thus straightforward to prove the IEEE correctness ofthe floating-point remainder operation, once we haveproved it for the floating-point divide. Therefore, fromthis point on, we will focus on the floating-point divideoperation.

Iterative Algorithms for Floating-Point Divide

Just as for square root, among the most common iterativealgorithms for calculating the value of the quotient of twofloating-point numbers, D�E, are those based on theNewton-Raphson method. Such algorithms are suggestedalso in [3]. Starting with an initial guess of 1/E, a verygood approximation of 1/E is derived first in a number ofiterations. From this, the value of D�E can be calculated.For example, an algorithm that starts by determining thevalue of the root of f ([) = E - 1/[, could have thefollowing iteration:

ei = (1 – b ⋅ yi)rn

yi+1 = (yi + ei ⋅ yi)rn

where ei is the error term, and yi+1 is the next betterapproximation.

Regardless of the particular iterative algorithm being used,one can evaluate the relative errors introduced in eachcomputational step. We will show that if the relative errorthat we can derive is small enough, then the IEEEcorrectness of the generated result is easily proven. Inpractice though, the relative error of the final result is


8

larger than desired. In the case of the divide (unlike for thesquare root), we cannot easily compensate for thedifference between what we can determine in terms ofrelative error and what is needed. In such cases,alternative methods have to be used, such as thosepresented in [3].

Difficult Cases for Rounding

For rounding to nearest, the difficult cases are representedby values of the arguments D and E of the divide operationsuch that D�E is different from, but very close to, amidpoint between two consecutive floating-pointnumbers. For rounding toward negative infinity, positiveinfinity, or zero, D�E has to be different from, but veryclose to, a floating-point number. We will show next howto determine the most difficult cases in each category.

The following lemma shows how to scale D and E to $and % respectively in order to make reasoning easier (Dand E are assumed to be positive here).

Lemma 2. Let N∈Z, N ≥ 3, a,b∈FN*, and rnd any IEEErounding mode. Then

(a) a/b can be expressed as a/b = A/B⋅2E, where A∈Z∩FN,A ≡ 0 (mod 2N-1) , B∈[2N–1,2N)∩Z and (A/B)rnd∈[2N-1,2N)∩Z. Let A = 2k⋅A1, such thatA1∈[2N–1, 2N)∩Z. If A1<B then k = N, and if A1>B,then k = N–1.

(b) a/b∈FN if and only if A/B∈[2N-1,2N )∩Z.

(c) a/b is a midpoint between two consecutive floating-point numbers in FN if and only if A/B is aninteger + 1/2 in [2N-1,2N ).

(d) If a/b∉FN, then a/b∉FN+1 (this is to say that a/b cannotbe a midpoint between two consecutive floating-pointnumbers in FN; the observation is necessary, as FN⊂FN+1).

Note that above, A1/B = sa/sb (ratio of the significands).

We can now determine the most difficult cases forrounding.

Consider first the cases of rounding toward negativeinfinity, positive infinity, or zero. If D�E∉FN (it is notrepresentable as a floating-point number with N bits in thesignificand), then how close can D�E be to a floating-point number f∈FN? Because the values of A/B fall in [2N-

1,2N), where floating-point numbers with N-bitsignificands are integers, and 1 ulp = 1, this can be re-formulated as “how close can $�% be to an N-bit integernumber )?” To answer this, we have to solve the equation

A/B = q + δ/B

for increasing values of δ∈Z*, and for q∈[2N–1,2N)∩Z.This leads to two cases, depending on whether A1<B, orA1>B (see Lemma 2 (a) above).

If A1<B, according to Lemma 2, the equation abovebecomes

2N⋅A1 = B⋅q + δ

If A1>B, according also to Lemma 2, the equationbecomes

2N-1⋅A1 = B⋅q + δ

In both cases, q∈[2N–1, 2N)∩Z.

We can solve the diophantine equations above forδ = 1, -1,2,-2,3,-3, … and B fixed. The method weapplied was to use Euclid’s algorithm to determine thegreatest common divisor of 2k and B, gcd (2k, B) (wherek=N or k=N-1), and to express it as a linear combinationof B and 2k. This yields a base solution, from which all theadmissible solutions can be derived (see [5] for usingEuclid’s algorithm to determine above solutions). Themost difficult cases are obtained for δ=1 and δ=-1. Thenumber of solutions is very large (unlike for the squareroot), which makes it impractical to verify them all for agiven divide algorithm. For example, for N=24, A1<B,and δ=1, the number of solutions is 1289234. Oneexample is

0.a49d25H / 0.ff7e75H = .a49e2300000100018b…H

which is very close to, but greater than, .a49e23H.

Consider next the case of rounding to nearest. If a/b ∉ FN

(it is not representable as a floating-point number with Nbits in the significand), then how close can it be to amidpoint between two consecutive floating-point numbersin FN? This can be re-formulated as “how close can A/Bbe to an N-bit integer number plus 1/2?” To find out, wehave to solve the equation

A/B = q + 1/2 + δ/B

for increasing values (in absolute value) of δ, 2⋅δ∈Z*, andfor q∈[2N–1,2N)∩Z. We can solve the diophantineequations above for δ = 1/2,-1/2,1,-1,3/2,-3/2, … and Bfixed. The method applied was similar to that describedfor the other three rounding modes. The most difficultcases for rounding are obtained for δ = 1/2 and δ = -1/2,when B is odd. The number of solutions is again verylarge. For example, for N = 24, A1<B, and δ = -1/2, thenumber of solutions is 1285649. One example is

0.c8227b H / 0.e73317H = .dd9a537fffff 724506a…H

which is very close to, but less than, .dd9a53H + 1/2 ulp.


9

For N=24, A1 < B, and δ=1/2, the number of solutions is1287219. An example is

0.ac1228 H / 0.b461d1H = .f43467800000b5a8…H

which is very close to, but greater than, .f43467H+1/2 ulp.

The method of determining values of the arguments D andE that generate values of D�E that are difficult to round isalso useful in generating good quality test vectors for anyimplementation of the floating-point divide operation (notjust for iterative algorithms), that can be added to otherschosen by different criteria. In addition, this method canhelp in proving IEEE correctness of certain iterativealgorithms for calculating the result of the floating-pointdivide operation, as shown below.

General Properties

Two main properties, Theorems 4 and 5, are used in theproposed method of proving the IEEE correctness of theresult generated by iterative floating-point dividealgorithms. They are expressed in terms of the scaledvalues $ and % of the arguments D and E, as explained byLemma 2 above.

Theorem 4. Let a,b∈FN, a=σa⋅sa⋅2e_a , b=σb⋅sb⋅2e_b. Ifa/b∉FN, and A and B are determined by scaling a and brespectively as specified in Lemma 2, then for any f∈FN,and for any integer F∈[2N-1,2 N)

(a) The distance wA/B between A/B and F satisfieswA/B = | A/B - F | > 1/2N

(b) The distance wa/b between a/b and f satisfieswa/b = | a/b - f | > 2e_a–e_b–2N, if sa < sb, and

wa/b = | a/b - f | > 2e_a–e_b–2N+1, if sa > sb

This theorem states that if D�E is not representable as afloating-point number with an N-bit significand, then thereare exclusion zones of known minimal width around anyfloating-point number, within which D�E cannot exist.The minimal width of the exclusion zones cannot be easilyextended in this case. If we try to gradually eliminate thedifficult cases for rounding toward negative infinity,positive infinity, or zero (starting with the most difficultcases), we find too many cases to make it practical toverify directly.

Theorem 5. Let a,b∈FN, a=σa⋅sa⋅2e_a , b=σb⋅sb⋅2e_b. Ifa/b∉FN, and A and B are determined by scaling a and brespectively as specified in Lemma 2, then for anym∈FN+1-FN (midpoint between two consecutive floating-point numbers in FN), and for any M, integer+1/2∈[2N-1,2N)

(a) The distance wA/B’ between A/B and M satisfieswA/B’ = | A/B - M | > 1/2N+1

(b) The distance wa/b’ between a/b and m satisfieswa/b’ = | a/b - m | > 2e_a–e_b–2N-1 , if sa < sb, and

wa/b’ = | a/b - m | > 2e_a – e_b–2 N , if sa > sb

This theorem states that if D�E is not representable as afloating-point number with an N-bit significand, then thereare exclusion zones of known minimal width around anymidpoint between two consecutive floating-pointnumbers, within which D�E cannot exist. Just as in thecase of Theorem 4, the minimal width of the exclusionzones cannot be easily extended.

This property is used, as explained below, in conjunctionwith that in the preceding Theorem 4.

IEEE Correctness of Iterative Floating-PointDivide Algorithms

In order to prove that the result of an iterative algorithmfor calculating the quotient D�E is IEEE correct, we haveto show that the result R* before rounding and the exactvalue of D�E lead, through rounding, to the same floating-point number. If the result R* before rounding and theexact D�E are closer to each other than half of theminimum width of any exclusion zone determined asshown in Theorems 4 and 5, then (R*)rnd = (a/b)rnd forany IEEE rounding mode rnd. Figure 5 above, whichillustrates this idea for the square root, is applicable in thiscase too.

As in Theorem 4 (a), 1/2N = 1/2N ulp in FN = 1 ulp in F2N,and in Theorem 5 (a), 1/2N+1 = 1/2N+1 ulp in FN = 1 ulp inF2N+1, the above can be expressed in the followingcorollaries of these two theorems:

Corollary 4. If a,b∈FN, R*∈R and |R*-a/b| ≤ 1 ulp in F2N,then (R*)rnd = (a/b)rnd for any rounding mode rnd ∈ {rm,rp, rz}

Corollary 5. If a,b∈FN, R*∈R and |R*-a/b| ≤ 1 ulp inF2N+1, then (R*)rn = (a/b)rn

In practice, the inequalities to be verified are

|R* - a/b| ≤ (wa/b)min

|R* - a/b| ≤ (wa/b’)min

The method of proving the IEEE correctness of the resultof an iterative algorithm that calculates the quotient of twofloating-point numbers can then be summarized as asequence of steps:


10

Step 1. Evaluate the relative error ε of the final resultbefore rounding:

R* = a/b⋅(1+ε)

where an upper bound for |ε| is known as |ε| ≤ 2-p, for somegiven p∈R, p > 0.

Step 2. Evaluate |R* - a/b| = |(a/b)⋅ε| ≤ |a/b|⋅2-p. If thefollowing hold

|a/b| ⋅ 2-p ≤ (wa/b)min

|a/b| ⋅ 2-p ≤ (wa/b’)min

then the algorithm generates IEEE-correct results for anyinput values D and E.

The method presented above was applied to software-based iterative algorithms for calculating the quotient oftwo floating-point numbers, when sufficient extraprecision existed for the intermediate computational stepsto allow the conditions above to be satisfied. Wheneverthis is not possible, alternative methods, such as the onepresented in [3], have to be used.

It is worth mentioning that Corollaries 4 and 5 above werealso obtained independently, starting from the followingLemma:

Lemma 3. Let a,b∈FN, b≠0. Then

(a) Either a/b∈FN (the exact a/b is representable with Nbits in the significand), or a/b is periodic.

(b) If a/b∉FN (it is periodic), then its infiniterepresentation in binary cannot contain more than N-1consecutive zeroes past the first binary one, and it cannotcontain more than N-1 consecutive ones.

The Effect of the Limited Exponent Range

A note should be made here regarding the effect of thelimited exponent range in iterative algorithms. If weanalyze the algorithm by assuming valid inputs in a givenfloating-point format, and a certain precision (significandsize) is ensured for all the computational steps, we maydraw the conclusion (possibly using methods such as thosepresented above) that the final result is IEEE correct. It ispossible though for an intermediate computational step tooverflow, underflow (due to the limited exponent range),or to lose precision (due to the limited exponent range andto the limited significand size). Such cases have to beidentified by carefully examining every computation stepand, if they exist, the input operands that can lead to themhave to be singled out. For such operands, alternatesoftware algorithms are expected to be used in order togenerate the IEEE-correct results, most likely by

appropriate scaling of the input values and correspondingscaling back of the results.

ResultsThere are two main results of the work described in thispaper.

First, a new method of proving IEEE correctness of theresult of iterative algorithms that calculate the square rootof a floating-point number was devised. This includedestablishing theoretical properties that also alloweddetermination of difficult cases for any of the four IEEErounding modes, and for any desired precision. Themethod proposed for proving the IEEE correctness wassuccessfully used for several algorithms that calculate thesquare root of single precision, double precision, anddouble-extended precision floating-point numbers.

Second, a parallel was drawn for iterative algorithms thatcalculate the quotient of two floating-point numbers.Again, the theoretical study allowed determination ofdifficult cases for rounding for any precision (single,double, or double-extended). The method proposed forproving the IEEE correctness of the result wassuccessfully used this time only for algorithms thatcalculate the quotient of two single precision floating-point numbers. The reason for this limitation was clearlyidentified.

Overall, the study helped to demonstrate that efficient,IEEE-correct, software-based algorithms can competewith hardware-based implementations for floating-pointsquare root and divide.

DiscussionThe methods proposed are general and can be applied toany iterative algorithm that implements the floating-pointsquare root or divide operation. Their usefulness wasverified for software-based algorithms.

The method proposed for proving the IEEE correctness ofthe result for the floating-point square root operationshould work for any iterative algorithm. Based on thismethod, it is possible to completely automate theverification process, and to build an IEEE correctnesschecker that would take as input the algorithm to beverified, together with the precision and rounding modefor every step. The method proposed for proving the IEEEcorrectness of the result for the floating-point divideoperation will only work if there is sufficient extraprecision in the intermediate calculations with respect tothe precision of the final result. This will hold mostly forsingle-precision computations, but might also be true forhigher precisions, depending on the particularimplementation.


11

Methods to generate cases of difficult operands forrounding were also described, which are independent ofany particular algorithm. They can be used to generate testvectors for any desired precision.

ConclusionThe two goals stated in the beginning of this paper werereached: a methodology for proving the IEEE correctnessof the results for iterative algorithms that implement thesquare root, divide, and remainder operations wasestablished, and operands that lead to results thatconstitute difficult cases for rounding were identified.This work is important to Intel Corporation because itproves the existence and viability of alternatives to thehardware implementations of these operations.

Notation

Some of the mathematical notations used in this paper aregiven here for reference:

N the set of natural numbers

Z the set of integer numbers

Q the set of rational numbers

R the set of real numbers

FN the set of floating-point numbers with N-bitsignificands and unlimited exponent range

FM,N the set of floating-point numbers with M-bitexponents and N-bit significands

D∈S D is a member of the set S

S∩T the intersection of sets S and T

S∪T the union of sets S and T

S⊂T the set S is a subset of set T

S−T the difference of sets S and T

[O�K] the set of real values [, O ≤ [ ≤ K

(O�K) the set of real values [��O�< [�<�K

[ ≅ \ the real number [ is approximatelyequal to the real number \

S ⇔ T statement S is equivalent to statement T

m ≡ n (mod p) integers m and n yield the sameremainder when divided by the positiveinteger p

In all the cases of sets, an asterisk (*) indicates theabsence of the value 0, e.g., R* = R \ {0}.

AcknowledgmentsThe author thanks Roger Golliver from Intel Corporation,Peter Markstein from Hewlett-Packard Company, andMarius Dadarlat from Purdue University for their support,ideas, and/or feedback regarding parts of the workpresented in this paper.

References[1] ANSI/IEEE Std 754-1985, IEEE Standard for Binary

Floating-Point Arithmetic, IEEE, New York, 1985.

[2] Wolfram, S., Mathematica – A System for DoingMathematics by Computer, Adison-Wesley PublishingCompany, 1993.

[3] Markstein, P., Computation of Elementary Functionson the IBM RISC System/6000 Processor, IBMJournal, 1990.

[4] Pentium® Pro Family Developer’s Manual, IntelCorporation, 1996, pp. 11-152 to 11-154.

[5] Stark, H. M., An Introduction to Number Theory, TheMIT Press, Cambridge MA, 1995, pp. 16-50.

Author’s BiographyMarius Cornea-Hasegan is a staff software engineer withMicrocomputer Software Labs at Intel Corporation inHillsboro, OR. He holds an M.Sc. in ElectricalEngineering from the Polytechnic Institute of Cluj,Romania, and a Ph.D. in Computer Sciences from PurdueUniversity, in West Lafayette, IN. His interests includemainly floating-point architecture and algorithms,individually or as parts of a computing system, and formalverification. His e-mail address [email protected].

Demystifying Multimedia Conferencing Over the Internet using the H.323. Set of Standards 1

Demystifying Multimedia Conferencing Over the InternetUsing the H.323 Set of Standards

James Toga, Emerging Products Division, Intel Architecture Labs, Intel CorporationHani ElGebaly, Emerging Products Division, Intel Architecture Labs, Intel Corporation

Index words: H.323, conferencing, Internet, multimedia, gatekeeper

AbstractThe Telecommunication Sector of the InternationalTelecommunication Union (ITU-T) has developed a set ofstandards for multimedia conferencing over packet-basednetworks. These standards are aggregated under astandard umbrella termed Recommendation H.323.Recommendation H.323 describes terminals, equipment,and services for multimedia communication over networkssuch as the Internet.

H.323 terminals and equipment carry real-time voice,video, and data, in any combination thereof. Terminalssignal calls using Q.931-derived procedures defined inRecommendation H.225.0. After the call signaling phase,terminals proceed to the call control phase where theyexchange capabilities and logical channel informationusing the H.245 protocol defined in RecommendationH.245. Once the call has been established, audio andvideo (if supported) are initiated. Both media types usethe Real Time Protocol (RTP) defined by the InternetEngineering Task Force (IETF) as their TransportProtocol. Procedures for audio and video packet formatand transport are described in Recommendation H.225.0.Recommendation H.323 allows for the use of a variety ofvideo codecs (e.g., H.261, H.263, H.263+) and audiocodecs (e.g., G.711, G.723.1). Data collaboration is alsoallowed using Protocol T.120. We provide an overview ofthese protocols and explain the H.323 call scenario.

Other entities such as gatekeepers, Multipoint ControlUnits (MCUs), and gateways are also addressed inRecommendation H.323. These entities allow networkmanagement, centralized multipoint, and interoperabilitywith other conferencing standards. We explain each ofthese entities briefly and provide some scenarios of theirinteraction with the H.323 terminals.

IP telephony has become an important driver for packet-based communications. We address the role of H.323procedures in deploying IP telephony, and describe how

new H.323 features such as supplementary services andsecurity facilitate this purpose.

IntroductionRecommendation H.323 [8] describes the procedures forpoint-to-point and multipoint audio and videoconferencing over packet-switched networks. In additionto video conferencing terminals, Specification H.323describes other H.323 entities including gateways,gatekeepers, and MCUs. Gateways allow interoperation ofH.323 systems with other audio/video conferencingsystems on integrated services digital networks (ISDN),plain old telephone systems (POTS), asynchronoustransfer mode (ATM), and other transports. Gatekeepersprovide admission control and address translation toH.323 endpoints. MCUs can engage more than two H.323endpoints in a centralized multipoint conference.

Recommendation H.323 comprises a number of relateddocuments that describe terminals, equipment, services,and interactions. Examples of other core standards that arereferred to in Recommendation H.323 are H.225.0 [5](procedures for call signaling, media packet format, andsynchronization); H.245 [7] (procedures for capabilityexchange, channel negotiation, and flow control); H.450.x[11] (procedures for supplementary services); H.246 [8](procedures for terminals’ interoperability throughgateways); and H.235 (security and encryptionprocedures). Other referenced standards include mediacodecs for audio (such as, G.711, G.723.1, G.729, andG.722) and video (such as, H.261 and H.263).

The purpose of this paper is to present an overview of theH.323 core components and functionality and the currentindustry trends with respect to H.323, such as IPtelephony and corporate conferencing. Towards this goal,we briefly describe the H.323 architecture, theresponsibilities of H.323 entities, and basic call scenarios.The main obstacles to the adoption and deployment of theH.323 standard in industry are also explained. Finally,



H.323 zone management, multipoint operation, telephonyservices features, and security are highlighted.

H.323 Architecture

Video I/O

Audio I/O

Delay

User App

Video

Network

Interface

Pkt H.225

T.120, etc.

H.245

Scope of Recommendation H.323

RecvPath

G.711

G.729G.723.1

Layer

Control

User

Interface

System

H.261

H.263

Q.931

RAS

Audio

G.722

G.728

Figure 1: H.323 architecture

The architecture of an H.323 terminal is shown in Figure1. The scope of Recommendation H.323 is limited to thedefinition of the media compression standard, packetformat, signaling, and flow control. Media capturing suchas video capturing schemes, audio recording, or user dataapplications are outside the scope of RecommendationH.323. Initial H.323 implementations targeted IPnetworks; however, Recommendation H.323 supportsalternative transports such as IPX. The Transport layercarries control and media packets over the network; thereare no specific requirements for the underlying transportexcept for support for reliable and unreliable packetmodes. An example of a well-known transport is UserDatagram Protocol (UDP) over Internet Protocol (IP).

The video codec (e.g., H.261, etc.) encodes the videofrom the video source (i.e., camera) for transmission anddecodes the received video code that is output to a videodisplay. The mandatory video codec for an H.323 terminalis H.261 [3] with quarter common intermediate format(QCIF) resolution. (QCIF is a video picture size.) Othercodecs such as H.263 may be supported. H.263 [4] hasbetter picture quality and more options. A terminal canalso support other picture sizes such as CIF and SQCIF.During the terminal capability exchange phase, the video

codecs, resolution, bitrate, and algorithm options areexchanged between terminals, using the H.245 protocol.Terminals can open channels only with parameters andoptions chosen from the intersection of the capability sets.In general, the receiver always specifies what thetransmitter may send.

The audio codec (G.711, etc.) encodes the audio signalfrom the microphone for transmission and decodes thereceived audio code that is output to the loudspeaker.G.711 0 is the mandatory codec for an H.323 terminal. Aterminal may be capable of optionally encoding anddecoding speech using Recommendations G.722, G.728,G.729, MPEG1 audio, and G.723.1. Since G.711 is ahigh-bitrate codec (64Kb/s or 56Kb/s), it cannot becarried over low-bitrate (< 56 kbps) links. G.723.1 [2] isthe preferred codec in this situation because of itsreasonably low rate (5.3Kb/s and 6.4 Kb/s). H.323terminals open logical channels using a commoncapability that is supported by all entities and exchangedduring the H.245 capability exchange phase.

The Data Channel supports telematic applications such aselectronic whiteboards, still image transfer, file exchange,database access, audiographics conferencing, etc. Thestandardized data application for real-time audiographicsconferencing is T.120. Other applications and protocolsmay also be used via H.245 negotiation such as chattingand fax.

The System Control Unit (H.245, H.225.0) providessignaling and flow control for proper operation of theH.323 terminal. H.245 [7] is the media control protocolthat allows capability exchange, channel negotiation,switching of media modes, and other miscellaneouscommands and indications. The H.225 standard describesthe Call-Signaling Protocol used for admission controland for establishing connection between two or moreterminals. The Connection Establishment Protocol isderived from the Q.931 specification [12]. The H.225.0[5] Layer also formats the transmitted video, audio, data,and control streams into messages for output to thenetwork interface, and it retrieves the received video,audio, data, and control streams from messages that havebeen input from the network interface, using the RealTime Transport Protocol (RTP) and its companion, theReal Time Control Protocol (RTCP). The RTP performslogical framing, sequence numbering, timestamping,payload distinction, source identification, andoccasionally, error detection and correction as appropriateto each media type. The RTCP provides reporting andstatus that may be used by both senders and receivers tocorrelate performance on the media streams.



H.323 Entities and Responsibilities

Terminal 1 Terminal 2

Terminal 3MCU

Gatekeeper

Gateway

Packet Switched Network

MC MP

Figure 2: H.323 entities

Figure 2 shows the different entities that can be involvedin an H.323 conference. A gatekeeper is an optionalelement in an H.323 conference that provides call controlservices to the H.323 endpoints such as addresstranslation, admission control, bandwidth control, andzone management. The gatekeeper may also provide otheroptional functions such as call authorization and callaccounting information. A gatekeeper cannot be specifiedas a destination in a call. Address translation is the methodby which an alias address (e.g., e-mail address) istranslated to a transport address. Admission control is away of limiting H.323 access to a network, or to aparticular conference using Request, Admission, andStatus (RAS) messages defined in RecommendationH.225.0. A gatekeeper also manages bandwidth allocationto H.323 endpoints using RAS messages. Zonemanagement defines the scope of entities over which agatekeeper has control. These include endpoints,gateways, and MCUs. In addition, a gatekeeper can allowsecure access to the conference using variousauthentication mechanisms. Q.931 and H.245 messagescan be routed through the gatekeeper, and statisticalinformation about the calls in progress can be collected.Gatekeepers may also perform telephony serviceoperations such as call forwarding and call transfer.

H.323 endpoints can interact with each other directly in apoint-to-point or multipoint conference if no gatekeeper ispresent. When a gatekeeper is present, all endpoints haveto register with it.

A gateway operates as an endpoint on the network thatprovides real-time, two-way communication betweenH.323 terminals on the packet-based network and otherITU terminals on a switched-circuit network, or to anotherH.323 gateway. Other ITU terminals include thosecomplying with Recommendations H.310 (B-ISDN),H.320 (ISDN), H.321 (ATM), H.322 (GQoS-LAN),H.324 (GSTN), H.324M (Mobile), and POTS. Thegateway provides the appropriate translation betweentransmission formats (for example, H.225.0 of an H.323endpoint to/from H.221 of an H.320 endpoint) andbetween communication procedures (for example, H.245

of an H.323 endpoint to/from H.242 of an H.320endpoint). This translation is specified inRecommendation H.246. The gateway also performs callsetup and clearing on both the network side and theSwitched-Circuit Network (SCN) side. Translationbetween video, audio, and data formats may also beperformed in the gateway. In general, the purpose of thegateway is to complete the call in both directions betweenthe network endpoint and the SCN endpoint in atransparent fashion.

The MCU is an endpoint on the network, which providesthe capability for three or more terminals or gateways toparticipate in a multipoint conference. It may also connecttwo terminals in a point-to-point conference, which maylater develop into a multipoint conference. The MCUconsists of two parts: a mandatory Multipoint Controller(MC) and optional Multipoint Processors (MP). The MCprovides the capability for call control to negotiate withall terminals to achieve common levels of communication.It is this element that is required for all multipointconferences. The MP allows mixing, switching, or otherprocessing of media streams under the control of the MC.The MP may process a single media stream or multiplemedia streams depending on the type of conferencesupported. In the simplest case, an MCU may consist onlyof an MC with no MPs.

The following section provides an overview of the basicoperation of an H.323 endpoint in a point-to-pointconference without a gatekeeper and then with agatekeeper.

H.323 General OperationFigure 3 shows call establishment and tear down stepsbetween two H.323 endpoints without a gatekeeper. All ofthe mandatory Q.931 and H.245 messages exchanged arelisted. Note that some of these messages may beoverlapped for increased performance. Each message hasan assigned sequence number at the originating endpoint.Endpoint A starts by sending a Setup message (1) toendpoint B containing the destination address. Endpoint Bresponds by sending a Q.931 Alerting message (2)followed by a Connect message (3) if the call is accepted.At this point, the call establishment signaling is complete,and the H.245 negotiation phase is initiated. Bothterminals will send their terminal capabilities (4) in theterminalCapabilitySet message. The terminal capabilitiesinclude media types, codec choices, and multiplexinformation. Each terminal will respond with aterminalCapabilitySetAck (5) message. The terminals’capabilities may be resent at any time during the call. TheMaster/Slave determination procedure (6-8) is thenstarted. The H.245 Master/Slave determination procedureis used to resolve conflicts between two endpoints that can



both be the MC for a conference, or between twoendpoints that are attempting to open bi-directionalchannels at the same time. In this procedure, twoendpoints exchange random numbers in the H.245masterSlaveDetermination message to determine themaster and slave endpoints. H.323 endpoints are capableof operating in both master and slave modes. Followingmaster/slave determination procedures, both terminalsproceed to open logical channels (9-10). Both video andaudio channels are unidirectional while data is bi-directional. Terminals may open as many channels as theywant. Only one logical channel is shown in Figure 3, yetthe same procedure applies if more channels are opened.

123

4

5

11

1. Setup2. Alerting

3. Connect

4. termCapSet

5. termCapAck

6. mSDet

7. mSDetAck

9. openReq

10. openAck

11. endSession

12. ReleaseComplet

Q.931 messagesH.245 messages

A B456

8.mSDetConfirm

11

10

789

910

12

Figure 3: Q.931 and H.245 messages exchanged betweentwo H.323 endpoints

Other H.245 control messages may be exchanged betweenthe endpoints to change media format, request video keyframes, change the bitrate, etc.

Termination of a call is initiated by one endpoint sendingan endSession message (11). Endpoint B, on receiving theendSession command, will respond with anotherendSession message (11) to Endpoint A. Endpoint A willfinally send a Q.931 ReleaseComplete message (12), andthe call is terminated.

Endpoint 1 Endpoint 2

Gatekeeper Cloud

123

84

5 6 7

910

11 12

12

3 4

9 10

1112

1314

13 14

1. GRQ

2. GCF\GRJ

3. RRQ

4. RCF\RRJ

5. ARQ

6. ACF\ARJ

7. ARQ

8. ACF\ARJ

9. BRQ

10. BCF\BRJ

11. DRQ12. DCF\DRJ

13. URQ

14. UCF\DRJ

Q.931

H.245

Figure 4: Gatekeeper interaction in an H.323 call

Figure 4 shows messages exchanged between a gatekeeperand an H.323 endpoint. Before the conference starts, bothendpoints look for a gatekeeper by multicasting aGatekeeperDiscovery (GRQ). Request. The gatekeeperwill reply either with a GatekeeperConfirm (GCF)message or with a GatekeeperReject (GRJ) message. Bothendpoints will then register their alias names with thegatekeeper using the RegistrationRequest (RRQ) message.The gatekeeper will acknowledge by sending aRegistrationConfirm (RCF) message or will deny it usinga RegistrationReject (RRJ) message. Registering aliasnames with the Gatekeeper allows endpoints to call eachother using user-friendly addresses such as e-mail, etc.,rather than the transport address. The discovery andregistration procedure is valid until the gatekeeperindicates otherwise.

An endpoint or gatekeeper can request the location ofanother endpoint using its alias name by using aLocationRequest (LRQ) message, and the gatekeeperreplies with a LocationConfirm (LCF) message containingthe resolved address for the alias name.

When a user places a call from an endpoint, the endpointstarts by requesting admission from the gatekeeper usingan AdmissionRequest (ARQ) message. The gatekeeper canaccept (ACF) or deny the request (ARJ). If the call isaccepted, the endpoint sends a Q.931 Setup message tothe remote destination. The recipient of the Setup messagein turn requests admission from its gatekeeper by sendingan ARQ. When the call is accepted, the Q.931 callsignaling sequence is completed followed by the H.245message negotiation. The AdmissionRequest (ARQ)message carries the initial bandwidth the endpoint requiresfor the duration of the conference. If during H.245 logicalchannel negotiation, an endpoint requires more



bandwidth, it issues a BandwidthRequest (BRQ) messageto the gatekeeper. If the request is accepted, thegatekeeper replies with a BandwidthConfirm (BCF)message; otherwise, it replies with a BandwidthReject(BRJ) message.

When the call is terminated, both endpoints send aDisengageRequest (DRQ) message to inform thegatekeeper that a call is being terminated. The gatekeeperreplies with a confirm (DCF) or reject (DRJ) message.Alternatively, endpoints may unregister from thegatekeeper by sending an UnregisterRequest (URQ)message. The gatekeeper replies with anUnregisterConfirm (UCF) message or anUnregisterReject (URJ) message.

H.323 Deployment ObstaclesIn order to achieve H.323 deployment in real networks,limitations at both the network level and the clientplatform level must be resolved at the H.323 client. Theclient should scale performance based on the availablebandwidth. Given the inconsistencies of networks withbest effort traffic, (i.e., no guaranteed Quality of Sevice(QoS), the Internet), it is extremely important to providemechanisms for fault tolerance and error resiliency at theclient platform, if it is to be used on an unmanagednetwork such as the Internet. At the network level, broadconnectivity, policy management, and security areconsidered the major issues in the deployment of a newtechnology such as H.323. Switched Circuit Networkconnectivity is achieved in the H.323 context by usinggateways for H.320, H.324, H.323, POTS, and otherendpoints on other networks. Policy management isachieved using gatekeepers to provide call admission,authentication, and zone management. Deployment ofQoS protocols such as RSVP can also help with policymanagement. The security of media streams is anotherimportant factor in the success of H.323 deployment,especially in unsecured environments such as the Internet.

The platform consists of two main components: theoperating system and the CPU. Many media compressionalgorithms are currently limited by the machine speed.(We expect this issue to improve as more powerfulprocessors hit the market.) Moreover, popular desktopoperating systems do not supply consistent real-timeservices. Multitasking can adversely affect the quality ofaudio and video in an H.323 conference.

H.323 ApplicationsA number of applications can take advantage of H.323technology both in the corporate environment and in thehome-user environment. One obvious application is video

conferencing between two or more users on the network.In this application, the user is expecting the same servicesas those provided by a telephone. The quality of serviceshould be equal to or better than POTS. The Internet doesnot appear to be readily addressing these issues; however,there is work in progress at the corporate Intranet level toimprove the quality of multimedia communication.

Multimedia call centers are another application for H.323.An H.323 call center provides a well-integratedenvironment for Web access and other data/voice businesssituations. The call centers are used by banks for customerservice, shops for extra retail outlets, etc. The call centercan just be an H.323 terminal or an MCU, or it can be afull featured endpoint with a gatekeeper, a gateway, andMCU capability. The front end of the legacy call centermay be a gateway, which allows installed systems tooperate with minimal disruption.

Another compelling application for H.323 istelecommuting. Telecommuters can attend meetings attheir companies, check their mail, or talk with someone atthe company while on the road or at home.

Finally, IP telephony is another area in which H.323 hasfound a significant role; this will be covered in a latersection.

H.323 Zone ManagementGatekeepers fulfill a required set of operationalresponsibilities and may offer a number of optionalfunctions to entities within their zone. Before we describehow zones are managed, we will review some of theseresponsibilities and functions.

A gatekeeper acts as a monitor of all H.323 calls within itszone on the network. It has two main responsibilities: callapproval and address resolution. An H.323 client thatwants to place a call can do so with the assistance of thegatekeeper. The gatekeeper provides the addressresolution to the destination client. (This division of workis due to alias name registration procedures.) During thisaddress resolution phase, the gatekeeper may also makepermissioning decisions based upon available bandwidth.The gatekeeper can act as an administrative point on thenetwork for IT/IS managers to control H.323 traffic onand off the network.

Strictly speaking, a gatekeeper zone is defined by what itcontains: it is defined by all of the endpoints, gateways,and MC(U)s that are or will be registered with agatekeeper. Another way to describe a gatekeeper zone isto call it an “administrative domain,” although the formalITU-T recommendation text uses zone. An example ofgatekeeper zones is given in Figure 5.



Some important aspects of zone coverage are summarizedas follows:

• Zones are defined by all H.323 devices registered to asingle gatekeeper.

• Zone design may be independent of physicaltopology.

• Each zone has only one gatekeeper.

• Zone definition is implementation-specific.

• Gatekeeper zones are logical in nature.

• Network topology and administrative scope are bothfactors in zone design.

• Resources such as gateways and proxies may affectthe partitioning of zones.

Figure 5: Gatekeeper zones

Multipoint Conferences: Centralized VersusDecentralizedMultipoint conferences are defined as calls between threeor more parties. During these conferences, call control andmedia operations can become considerably more complexthan during a simple point-to-point conference. Thecoordination and notification of participants entering andleaving a conference along with the marshalling of themedia streams require the presence of at least a Multipoint

Control (MC). In other situations, an MCU is required; wewill explain why in the next section.

The two broad models of multipoint, centralized anddecentralized, differ in their handling of real-time mediastreams (audio and video). The centralized model operatesin the same fashion as other circuit-based conferencing(e.g., H.320 or telephony “bridges”). In this model, all ofthe audio and video is transmitted to a central MCU thatmixes the multiple audio streams, selects thecorresponding video stream, and re-transmits the result toall of the participants. Figure 6 illustrates this procedure.

A

B

C

D

E

F

Centralized Multi point

MCU

Audio

Video

Figure 6: Centralized model

Note that the MCU is only ‘logically’ at the center of theconference configuration; the physical topology may bevery different. The operational model of the conference issuch that each endpoint is exchanging media controlsignaling (H.245), audio, and video directly with theMCU. In order to prevent echo, the MCU must providethe current speaker with a custom audio mix that does notcontain the speaker’s own audio. The remainingparticipants may all receive the same audio and videomedia. In some implementations, the MCU may actuallydecode the video streams and combine them in a mixingoperation to provide continuous presence of allparticipants on the endpoint displays. The amount ofprocessor speed required for this operation increasessignificantly.

The decentralized model shares common controlcharacteristics with the centralized model, but the mediastreams are handled differently. One of the participatingentities must be an MC, but it will typically be co-locatedwith one of the endpoints. All of the H.245 connectionswill have one end terminating at the MC just as with the

H.323

H.323

Zone A

Zone B

Gatekeeper

Gatekeeper

H.323

H.323 H.323

H.323

H.323



MCU in the centralized model. Whereas the MCU doesthe media processing in the centralized model, the mediastreams are sent and received by all participating entitieson a peer-to-peer basis in the decentralized model. AsFigure 7 demonstrates, the logical configuration for this isa bus. There is no MCU to process the multiple streams;each entity is responsible for its own audio mixing andvideo selection. The media may be sent between allentities utilizing either multicast, or a multiple uni-cast ifthe underlying network does not support multicast.

A B C D E F

Decentralized Multipoint

MC

Audio

Video

Figure 7: Decentralized model

There are a number of advantages and disadvantages toboth the centralized and the decentralized models. In thecentralized model, the endpoints that participate in thistype of conference are not required to be as powerful as inthe decentralized model. Each endpoint has only toencode its locally produced media streams and decode theset sent by the MCU. The MCU may provide specialized,or higher performance conferencing. The MCU’s abilityto provide custom-mixed media streams can allowotherwise constrained endpoints to participate inconferences. For example, if an endpoint is connected tothe conference by a low-bitrate connection, it may onlyhave the capacity to transmit and receive one mediastream. If this same endpoint were connected to adecentralized conference, it would have no ability todetect which speaker was the focus when the focuschanged. In most situations, the MCU will choose thelargest common set of attributes within which to operatethe conference. This may lead to a conference that isoperated in a least common denominator mode, such asQCIF, rather than CIF or lower video resolution.

By definition, the decentralized multipoint model does notrequire the presence of an MCU, a potentially expensiveand limited resource. It allows for disproportionateprocessing at the endpoints with each running at its own

level. The decentralized multipoint model does requirethat one of the participating entities contain an MC. If theendpoint that has the MC leaves the conference, the MCmust stay active or the conference is terminated. In orderfor this model to be bandwidth efficient, the underlyingnetwork should support multicast. If it doesn’t, eachendpoint can send multiple uni-cast streams to all others,but this becomes increasingly inefficient with more thanfour entities participating in a conference.

H.323 Features for IP TelephonyThe initial environment envisioned for H.323 was thecorporate network environment, primarily local areanetworks. Wide Area Network (WAN) access was to begained by using gateways to H.320/ISDN. During theimplementation of Revision 1 of H.323, it became clearthat IP telephony was gaining popularity and relevance asvarious infrastructure elements were improved upon. Anumber of proprietary IP-based telephones were creatingmany small islands that could not communicate with oneanother. Recommendation H.323 provided a good basisfor establishing a universal IP voice and multimediacommunication in larger, connected networks. WithRevision 2 of the Recommendation, new additions andfurther extensions were added specifically to make it moresuitable for IP telephony. These changes will be describedin a later section.

By using Q.931 as its basis for establishing a connection,H.323 allows for relatively easy bridging to the publicswitched telephone networks (PSTN) and circuit-basedphones. The required voice codec of G.711 also allows foreasy connections to the legacy networks of telephones.The uncompressed 64kb/sec stream can easily betranslated between digital and analog media. One of theaddressing formats provided in Recommendation H.323 isthe E.164 address. This is another ITU Recommendationthat specifies standard telephone numbers (e.g., the digits0-9, * and #). These addresses, which ultimately map ontothe IP addresses for the H.323 endpoints, allow regulartelephones to ‘dial’ them. Gatekeepers provide the finalimportant element for IP telephony. Gatekeepers supplythe ability to have integrated directory and routingfunctions within the course of the call setup. Theseoperations are important for real-time voice or video whenresources must be balanced, and points of connectivity arehighly dynamic. The gatekeeper functions, which providecall permissioning and bandwidth control, enable loadmonitoring, provisioning, and ultimately, commercial-grade IP telephony service.

InteroperabilityRecommendation H.323 comprises a number ofinterrelated documents and sub-recommendations.



Therefore, customers are faced with a number of options,which they may or may not implement. These designchoices, in conjunction with differing interpretations ofthe required elements, can lead to implementations thatcannot interoperate. Stated more simply, compliance withthe recommendations does not always implyinteroperability. The complexity and flexibility of H.323essentially requires that implementations be testedtogether to ensure interoperability.

To this end, the International Multimedia TechnicalConsortium (IMTC) has focussed largely on this area ofinteroperability. The testing events sponsored by thisconsortium have provided a venue for more than thirtycompanies to participate in “bake-offs” to test theinteroperability of their product. The methodology that isfollowed provides for a gradual testing of componentprotocol stacks that then leads to higher level end-endscenario testing. Figure 8 illustrates the grouping of thecomponents. Each horizontal level provides increasingH.323 functionality, and each vertical group indicates aspecific stack function that is required.

Much of the mass interoperability testing has occurred inface-to-face events. In theory, the H.323 protocol shouldallow for testing from remote sites across the Internet; inpractice, however, remote testing has not turned out to beuseful. The lack of a controlled environment makesdistinguishing interoperability issues from simple networkproblems extremely difficult.

Examples for initial interoperability problems encounteredduring some of the testing events include misaligned bitfield and byte-ordering issues. These problems arecommon in the early development stage of protocols. Forexample, there are a number of adaptations that arespecified in the Q.931 protocol used by H.323, whichmake the Protocol Data Units (PDUs) slightly differentfrom the protocol messages used in the circuit world (suchas ISDN).

Porting of the existing Q.931 code to the packetenvironment provided mis-matches. During the initialcourse of development, there were a small number ofdefects in the Recommendation that were discovered bythe implementers. These defects were recorded in adocument called the Implementer’s Guide and eventuallycorrected in the revision of the base Recommendation. Inaddition to the low-level control protocol interoperability,the media stream is another potentially problematic area.Recommendation H.245 allows the receiver to specify themaximum bitrate that may be sent, but currently there isno way in which to specify the maximum Real TimeProtocol (RTP) packet size. This can lead tointeroperability problems if a receiver implementationmakes assumptions about the buffering required and thencannot decode the packet. Although H.323 specifiesG.711 and H.261 as its baseline codecs (audio and videorespectively), a large number of initial H.323implementations were targeted for low-bitrate connectivitywhere these codecs could not operate. The result of thiswas that the audio codec, G.723.1, was chosen due to itslow (5.3-6.4kb/s) data rate. This low bitrate allowedH.263 video to operate in an acceptable fashion across a33.6kb connection.

Implementation choices that are made as a result ofoperating environments elevate the interoperability issuesto the next level. At an operational level, endpointimplementations may select to support an asymmetricalmedia model for optimal performance; otherimplementations may not. The ability to enter extendedinformation at the user interface may determine whetheran endpoint can operate with certain gateways or H.323proxies. Gatekeepers may or may not provide intelligentzone control, which allows them to operate in a networkconnected to other activated gatekeepers. With theaddition of a range of security options to H.323, strictpolicy constraints may prevent H.323 entities frominteroperating while still complying with all therecommendations.



VideoCodec

Layer 0

Layer 1

Layer 2

VideoCodec TPKT

RTP/RTCP

H.245Control

TPKT

Q.931

RASAudioCodec

RTP/RTCP

T120

GroupA

GroupB Group

C

AudioCodecRTP/RTCP

GroupD

GroupE

GroupF

GroupG

GroupH

GroupI

Layer 3

VideoCodec

AudioCodec

RTP/RTCP

H.245Control

TPKT

Q.931

Group N

T120VideoCodec

AudioCodec

RTP/RTCP

H.245Control

TPKT

Q.931

Group M

RASLayer 4

M C

TPKT

VideoCodec

AudioCodec

RTP/RTCP

H.245Control

TPKT

Q.931

Group L

H.245Control

Q.931

TPKT

Group J

H.245Control

Q.931

TPKT

Group K

RAS

Figure 8: Testing matrix1

1 In this figure TPKT refers to the layer that segments a TCP stream into separate message ‘packets’ to be delivered to theH.323 application.

New FeaturesRevision 2 (1998) of Recommendation H.323 contains anumber of improvements for IP Telephony, among otherareas. A new Recommendation (H.235) was developed toprovide a full security framework for H.323 and othermultimedia systems. It may provide the services ofAuthentication (which can be used for authorization),Privacy, and Integrity. The system can utilize underlyingsecurity protocols such as Internet Protocol Security(IPSEC) or Transport Layer Security (TLS) as establishedin the IETF. A number of new features and options wereintroduced regarding interactions with gatekeepers.During the process of discovering gatekeepers, endpointsmay actually receive a number of gatekeeper addresses toutilize. This redundancy in the protocol will eliminate thesingle point of failure if a gatekeeper becomesinoperative. When endpoints register their current addresswith a gatekeeper, they may specify multiple addresses;

this allows a ‘line hunting’ mode of operation. Keep alivetypes of messages called Request In Progress (RIP) canstop premature retries on lengthy operations.

The call setup has a new method called FastStart thatestablishes bi-directional media in one round trip time ofmessages (discounting the establishment of the actual TCPconnection). Figure 9 shows FastStart messagesexchanged. The OpenLogicalChannels messages thatoccur after H.245 is established in the regular case arepiggy-backed on the Setup-Connect exchange. Thisfacility allows instant audio connection that resembles theregular phone call model as opposed to the standardlengthy H.323 startup procedures.



TCP connection

SETUP

CONNECT (H245 Address)

TCPconnection

H.245Messages

ReleaseComplete

(RTCP & RTP addresses)

Callee“B”

Caller“A”

RTP streamRTP streamRTCP stream

(RTCP & RTP addresses)(H245 Address)

Figure 9: FastStart

Many new explicit address types have been added.including RFC822 (e-mail) formatted names and URLs,which network-based users find more familiar and easierto remember. Each endpoint may register under a numberof these aliases. New unique call identifiers were added tobetter track calls as they traverse multiple network nodesand topologies. A number of new features were added toH.245 also, which allow for a much richer set of mediapossibilities. In addition to layered codecs, which divideup the video stream into additive layers of detail, Qualityof Service parameters (e.g., RSVP) may now be signaledwhen opening up media streams. Finally, the ability topass the media on other transports such as ATM (whilekeeping the rest of call and media control on IP) havebeen added.

In addition, a new family of recommendations has beendeveloped to set up a framework for supplementaryservices. The H.450.1, H.450.2, and H.450.3 are looselybased on the Q-signals (QSIG) protocol, which is used byPBX and switch vendors. The initial services that havebeen defined are call transfer and call forward. Both ofthese features are a peer-peer option requiring no specifichelp from a centralized network entity (such as the PBX inthe circuit world). A number of standardizedsupplementary services are expected to be developed inthe future.

Along with the new features, the latest development ofH.323 also includes an expanded topology model in theH.332 document. Recommendation H.332 allows anH.323 conference model to expand to literally thousandsof participants. The tightly controlled H.323 ‘panel’ issurrounded by a very large number of ‘listeners.’

Members may participate in the conference by joining thepanel. They may also leave as they wish. This H.323 panelis analogous to a panel on a stage in a large auditorium.Occasionally participants might get up from the audienceand join the panel and others might leave the panel andjoin the audience. This movement of participants occursusing the standard Q.931 Invite and Join signaling asdescribed in the H.225.0 document.

ConclusionThe H.323 system and its associated recommendationsprovide a useful and flexible system for multimediacommunication. The factors that allow the protocols toeasily bridge data and voice networks also make H.323scalable. The ability of most, if not all of the system, tooperate on a general-use platform such as a personalcomputer allows the H.323 system to scale withunderlying processing power. As the processor’s speedbudget increases, the H.323 system can provide a betterend-user experience. The dynamic exchange ofcapabilities allows the communications to change ifneeded during a call and adapt to any environmental orendpoint constraints.

Recommendation H.323 has become the single standards-based solution for a complete array of communicationsystems from simple point-to-point telephony to a richmultimedia conference with data sharing. Throughcontinued effort by the ITU-T study group,Recommendation H.323 continues to evolve and adapt tonew situations. Many of the real world difficulties inutilizing H.323 come about from infrastructure issues orother problems that are being resolved. Globallycoordinated addressing and consistent QoS are two areaswhere we expect to see great improvements in the future.Some of these improvements will be facilitated byexpected higher level communications betweengatekeepers and gateways as their interaction isstandardized.

References

[1] ITU-T Recommendation G.711 (1988)“ Pulse CodeModulation (PCM) of Voice Frequencies.”

[2] ITU-T Recommendation G.723.1 (1996) “Dual RateSpeech Coders for Multimedia CommunicationTransmitting at 5.3 & 6.3 kb/s.”

[3] ITU-T Recommendation H.261 (1993) “Video Codecfor Audiovisual Services at p X 64 kb/s.”



[4] ITU-T Recommendation H.263 (1996) “VideoCoding for Low Bit-rate Communication.”

[5] ITU-T Recommendation H.323 (1998) “Packet BasedMultimedia Communications Systems.”

[6] ITU-T Recommendation H.225.0 (1998) "CallSignaling Protocols and Media Stream Packetizationfor Packet Based Multimedia CommunicationsSystems."

[7] ITU-T Recommendation H.245 (1998) "ControlProtocol for Multimedia Communication."

[8] ITU-T Recommendation H.246 (1998) “Interworkingof H-Series Multimedia Terminals.”

[9] ITU-T Recommendation H.235 (1998) “Security andEncryption of H series (H.323 and other H.245 based)Multimedia Terminals.”

[10] ITU-T Recommendation H.332 (1998) “LooselyCoupled H.323 Conferencing.”

[11] ITU-T Recommendation H.450.1 (1998) “GenericFunctional Protocol.”

[12] ITU-T Recommendation Q.931 (1993) “DigitalSubscriber Signaling System No. 1 (DSS 1)-ISDNUser-Network Interface Layer 3 Specification forBasic Call Control.”

Authors’ BiographiesJim Toga holds a B.Sc in Chemistry from Tufts Universityand a M.Sc in Computer Science from NortheasternUniversity. Before joining Intel, he was the principalengineer on StreetTalk∗ Directory with Banyan Systemswhere he designed and developed the Yellow Pagesservice. Presently, he is a senior staff software architectfor the Standards and Architecture Group in the IntelArchitecture Labs. He coordinates product groups givingguidance on architecture and standards. His primary tasksare H.323/Internet Telephony, Directory, and real-timesecurity issues.

∗ All other trademarks are the property of their respectiveowners.

Outside of Intel, Mr. Toga develops standards andstandards-based products within ITU-T, IETF, and IMTC.He is also involved in the following related activities:

Editor of ITU-T H.323 Implementors GuideH.235 Security Standard

Chair of IMTC “Packet Networking Activity Group”Chair of H.323 Interoperability GroupHe has also written numerous articles for trade magazines.His e-mail is [email protected].

Hani ElGebaly is the project technical lead for theH.323/H.324 protocols development team within theConferencing Products Division of Intel ArchitectureLabs. He received an M.Sc. in Computer Science from theUniversity of Saskatchewan, Canada, and a B.Sc. inElectrical Engineering from Cairo University, Egypt. Mr.ElGebaly is currently pursuing a Ph.D. with the Universityof Victoria, Canada. His primary areas of interest includemultimedia conferencing protocols, embeddedprogramming, fault tolerant systems, and computerarchitecture. His e-mail is [email protected].

Characterization of Multimedia Streams of an H.323 Terminal 1

Characterization of Multimedia Streams of an H.323Terminal

Hani ElGebaly, Emerging Products Development, Intel Architecture Labs, Intel Corporation

Index words: H.323, Conferencing, Internet, G.723.1, H.263

AbstractA study of multimedia performance over the Internetrequires accurate representation of the workload model.Measurements of multimedia conferencing applicationsprovide a snapshot of real workloads. This paper presentsa characterization for video and audio traffic transportedover the Internet by multimedia conferencing applicationsfollowing the H.323 set of standards defined by theInternational Telecommunication Union (ITU-T).

Our goal is to investigate multimedia traffic multiplexingissues at the host. We are not concerned with the Internetinfrastructure or the store and forward technologydeployed. The traces collected in this study are done at themultimedia source host. They provide information aboutpacket length, inter-arrival time, jitter, overhead, andburstiness.

These traces help in understanding local-induced effectson the multimedia traffic mix. Many of these effects areprimarily due to limitations of operating systems,bandwidth sharing, or the choice of traffic parameters.These local effects contribute significantly to traffic delayor the abuse of network bandwidth and congestion.

A few tradeoffs are involved in the choice of multimediatraffic parameters. For example, a tradeoff exists betweenthe network protocol overhead and the local latency ofmultimedia packets. Another tradeoff exists between thevideo packet size and the burstiness of the video packets.In this paper, the various tradeoffs involved in generatingaudio and video packets are discussed. We focus on theaudio and video codecs defined in RecommendationsG.723.1 and H.263 of the ITU, respectively. Both codecsare extensively utilized by H.323 developers because oftheir efficiency, popularity, and suitability fortransmission over low bandwidth pipes. We derive anoptimal operating point for both audio and video trafficfor better bandwidth efficiency and acceptable latency.

We also discuss the interaction of conferencing mediacomponents as they are multiplexed on the host to be

transmitted to a peer terminal. Lack of appropriatemultiplexing algorithms can lead to one or more mediacomponents oversubscribing to the shared bandwidth andpenalizing other participants. A new performance qualifieris introduced in this paper. This qualifier is the number ofinterleaved video bytes scheduled for transmissionbetween audio packets. This number turned out to be agood local indicator for audio jitter and latency.

IntroductionAdvances in computer technology, such as fasterprocessors, faster modems, Intel MMX™ technology, andbetter data compression schemes have made it possible tointegrate audio and video data into the computingenvironment. A new type of video conferencing is nowpossible: desktop video conferencing. Desktop videoconferencing applications include telecommuting,corporate meetings to cut travel costs, family gatherings,Call Centers, etc. Banks, shopping centers, retail centers,and so on can be more efficient and cut a lot of overheadby providing video-conferencing customer service centersfor customers to dial into.

One of the most prominent enabling technologies fordesktop video conferencing is the H.323 standarddeveloped by the ITU-T. Recommendation H.323.[6]describes terminals, equipment, and services formultimedia communication over networks such as theInternet.

This paper addresses issues of the multimedia trafficsources of an H.323 terminal. Issues such as packet formatand multiplexing of audio and video frames at the host arestudied. The traces collected in this study were done at thenetwork edge. These traces provide information about thepacket length, inter-arrival time, jitter, protocol overhead,and burstiness. Local-induced effects on the conferencingtraffic can occur due to limitations of the operatingsystem, bandwidth sharing, and lack of an accurateperformance qualifier for choice of traffic parameters. The



collected traces help to understand these effects andeliminate them.

Measurement MethodologyThis paper studies the packet format parameters andmultiplexing issues of audio and video traffic at thetransport layer before reaching the network. Trafficmeasurements capture packet overhead imposed by theTransport, Network, and Datalink layers’ protocols. It alsoprovides information on the local inter-arrival time,latency, and jitter.

We focused our study on audio and video codecs definedin the G.723.1 specification [3] and the H.263specification [5], respectively. Although the mandatorycodecs for the H.323 standard are G.711 [2] and H.261[4], most of the H.323 terminals that connect to theInternet via modems through Internet service providersuse G.723.1 and H.263 codecs as their preferredconferencing codecs. G.723.1 has lower bit rate (5.3/6.4kb/s) than G.711 (64 kb/s). Hence, it is more suitable fortransmission over low bandwidth links (e.g., 28.8 kb/s and33.6 kb/s modems). However, for video conferencing,H.263 has better quality than H.261.

It is important to capture the worst-case behavior of thenetwork such that the solutions proposed can have wideapplicability. It is also important to observe the mostcongested time of the day over the Internet. We used avariety of comparable Internet service providers forestablishing conferencing sessions.

Media Trace Format

Audio FormatG.723.1 [3] is a dual-rate speech-coding standard, whichoperates in low bit rate while maintaining high perceptualquality. It is recommended as the preferred speech codecfor the ITU H.323 [6] conferencing standard over theInternet when the access link to the Internet has limitedbandwidth (e.g., an ISP connection using a modem). TheG.723.1 codec has two bit rates associated with it. Theserates are 5.3 and 6.4 kb/s, the latter being of better quality.Both rates are a mandatory part of the encoder anddecoder.

The collected measurements are applied to a trafficscheduler. The scheduler accepts audio and video packetsand schedules them for transmission using the schedulingalgorithm of interest. During the simulation run,measurements of packet sizes, inter-arrival time, latency,and jitter are computed. The scheduler is depicted inFigure 1.

The scheduler is a multithreaded application. Each trafficsource has its own thread. The scheduler also has its own

separate thread. Additional traffic sources such as T.120data or other video or audio codecs can be easily hookedto the scheduler.

Audiosource

Videosource

Scheduleralgorithm

Queue

Measurementstatistics report

MUX

Figure 1: Architecture of the traffic scheduler

The G.723.1 encoder provides one frame of audio every30 milliseconds. The audio frame size is 20 bytes ofpulse-coded modulation (PCM) samples for the low rate(5.3 kb/s) and 24 bytes for the high rate (6.4 kb/s). Eachapplication is set for a particular number of frames peraudio packet before the conference starts. These valuesare negotiated when the call is established, and the leastnumber prevails. Both applications are forced to use thisnumber as the maximum number of frames incorporated inan audio packet.

Increasing the number of frames per audio packetimproves the bandwidth utilization and decreases networkpacket overhead. However, it also introduces additionaldelay to the audio playback since a packet has to wait forall the audio frames to be accumulated before sending itacross. This host-induced delay can be even more harmfulespecially with the timeliness requirement of real-timeaudio telephony.

Each audio packet has a Real-time Protocol (RTP) [7]header (12 bytes) that carries time-stamps, sequencenumbers, a payload type, and a synchronization sourceidentifier. In addition, a User Datagram Protocol (UDP)header is needed to carry UDP information about thepacket (8 bytes). Then an Internet Protocol (IP) header of20 bytes is also needed to carry routing information.Finally, since we are strictly constrained in bandwidth aswe are expected to connect to the Internet via Internetservice providers, an additional 5 bytes of point-to-pointprotocol (PPP) header is also required.

Video FormatThe H.263 codec [5] is designed for a wide range of bitrates (10kb/s - 2 Mb/s). The codec supports five differentresolutions. In addition to Common Intermediate Format



(CIF) and Quarter Common Intermediate Format (QCIF)that were supported by H.261, there is sub QCIF (SQCIF),four times CIF (4CIF), and sixteen times CIF (16CIF).SQCIF is approximately half the resolution of QCIF. 4CIFand 16CIF are four and sixteen times the resolution ofCIF, respectively. Support for 4CIF and 16CIF allows theH.263 codec to compete with other higher bit-rate videocoding standards such as the MPEG standards. H.263 is ahierarchical codec, i.e., some performance and errorrecovery options can be sacrificed for lower bit rate.

It is important to understand how the video sourcegenerates video data in order to analyze video-generatedtraffic. Video pictures are fragmented into multiple videofragments. Each fragment forms a video packet. Eachpacket is sent across the network after RTP, UDP, IP, andPPP headers have been added. Video is different fromaudio in the sense that the generated bit rate can bevariable while the audio bit rate is usually constant. TheITU-T standard left this issue up to the application as towhether to use a fixed or variable bit rate for video [6].Another important difference between video and audiomedia types is fragmentation. Video can be fragmentedbecause a video-generated frame can be quite large; audiois not fragmented. Fragmentation means a video frame isdivided into multiple fragments. Each fragment forms avideo packet that is sent across the network. There is anupper limit on the maximum fragment size used by thefragmentation process.

A third major difference between audio and video is theburstiness of the video source due to fragmentation, whichcauses multiple fragments to cluster in a small interval oftime. This burst of packets can lead to packet loss,latency, or at least some amount of unfavorable jitter.

In this section, we focus on the characterization ofgenerated video traffic after fragmentation and the impliedtradeoffs when enforcing different maximum fragmentsizes for video packets.

Smaller fragment size results in shorter inter-arrival timeyet bandwidth efficiency decreases because of largerpacket overhead. However, larger fragment size results inlonger inter-arrival time yet maintains good utilization ofbandwidth. As the maximum fragment size limit increases,larger video packets will occur more often. These packetswill require more time to be transmitted and hence thevideo latency increases.

An experiment was conducted to study the video packetsizes (number of bytes per packet) for different maximumfragment limits as generated by our video source. We setthe maximum fragment size to four different values (128,256, 512, and 750 bytes). We ran four experiments, eachwith a different maximum fragment limit and collected the

corresponding video packet sizes generated after thefragmentation module.

Figure 2 shows how video packet sizes are distributed fordifferent maximum fragment sizes. The figure shows thatby using a maximum fragment size of 750 bytes, 70% ofthe video packets had a size in the range of 50-250 bytes.Less than 10% of the total generated video packets weremore than 512 bytes. For a fragment size of 512 bytes,75% of the packets had sizes between 50-250 bytes. Bydecreasing the fragment size further to 256 bytes, morethat 60% of the packets were in the range of 150-250bytes. For a 128 maximum fragment size, the video packetsizes were equally distributed in the range of 50-100bytes, and 100-150 bytes. Generally, the majority of thepackets for all maximum fragment sizes were less than256 bytes. This can mean that the maximum fragment sizechoice should be in the range of 256-512 bytes.

05

101520253035404550

0-50

100-

150

200-

250

> 51

2

128 bytefragment %

256 bytefragment %

512 bytefragment %

750 bytefragment %

Figure 2: Byte distribution for video fragment sizes

Determining Traffic Parameters

Audio TrafficIt has been demonstrated that the choice of the number offrames per packet affects both local latency and protocoloverhead (i.e., bandwidth utilization). We are primarilyconcerned with the latency induced by the audio packetpreparation manager that buffers the captured audioframes and synthesizes them into audio packets. Thislatency is computed before the packet is actually sent onthe network, and it only accounts for local capture andpacket preparation delay. The choice of the number offrames per audio packet should minimize both locallatency and packet overhead.

Results for Uncompressed Audio Packet Header

Figure 3 shows the percentage packet overhead for boththe high and low bit rate G.723.1 audio codec. The packet



overhead decreases as the number of frames per packetincreases. Intuitively, high bit-rate audio has slightly lessoverhead when compared to low bit-rate packets.

Figure 3: Packet overhead for low and high G.723.1audio

In Figure 4, the number of audio frames per packet isplotted for both rates against normalized packet overheadand latency. The latency is normalized against themaximum latency value exhibited by the largest audiopacket (24 audio frames per packet) used in ourexperiment. The packet overhead is normalized againstthe maximum overhead exhibited by an audio packet. Anaudio packet with only one audio frame has the maximumpacket overhead as given in Figure 3. As the number ofaudio frames per packet increases, the packet overheaddecreases. The overhead decrease is explained by moreinformation bytes (audio frames) being included in apacket with a fixed header (RTP+UDP+PPP+IP).

The latency increases as the number of frames per packetincreases. This is expected since more audio framesrequire more time to be captured and buffered. Hence, atradeoff exists between packet overhead and local latencyfor audio packet transfer. The intersection of the latencyand the overhead curves provides the choice of thenumber of frames that has the minimum latency and leastoverhead.

Figure 4: Packet overhead versus latency

The optimal point for the number of frames based on thisplot is between seven and eight for both G.723.1 audio bitrates. This value should be used by terminals that targetpacket efficiency as well as low latency for audio packets.

Sometimes, the audio local latency is reduced at theexpense of packet overhead, in order to achieve latenciesacceptable to users. In fact, a few applications are willingto sacrifice some protocol overhead for achieving betteraudio latency. Some H.323 terminals use a value of 3 or 4for the number of frames per audio packet in order toreduce this latency.

Results for Compressed Audio Packet Header

Protocol header compression has been an active researcharea for the past couple of years especially after thematurity of the protocols and standards that drive audioand video streaming over the Internet. Besides IP andUDP, researchers have also investigated RTP headercompression. Jacobson and Casner [1] proposed anapproach for compressing RTP, UDP, and IP headers tobe used over low-speed serial connections to the Internet.Examples of these links include low speed (e.g., 28.8kb/s) modem connections to the Internet through Internetservice providers.

This approach seems ideally suited to better bandwidthutilization of the media streams since the protocoloverhead is significantly reduced. A few companies arecurrently working on the deployment of this approachinside the network interface infrastructure in order toimprove multimedia conferencing over the Internet forterminals connected via low speed links.



Figure 5: Packet overhead versus latency withcompressed IP/UDP

Figure 5 shows a plot of the packet overhead versuslatency after applying IP/UDP header compression. Thecurve suggests that the optimal number of audio frames isalmost five for low bit-rate audio and six for high bit-rateaudio.

Video TrafficAnother major decision in video source parameters is thechoice of an appropriate maximum fragment size. Themaximum fragment size contributes to the latency,network overhead, and the amount of generated traffic. Asthe size increases, larger video packets are generated.Large packets lead to increased latency and less protocoloverhead. Reducing the maximum fragment size will leaddirectly to more packets being generated for the samenumber of video frames.

Video Packet Length Effect

Histograms of the video traffic are shown in figures 6-8.The choice of a maximum fragment size should capturemost of the data clustering and at the same time should notpenalize audio packets by generating large video packetsthat induce huge delays.

We chose 10 kb/s for the video bit rate, which is suitablefor ISP-based Internet video conferencing terminals. Weused the Quarter Common Intermediate Format (QCIF)for video streaming. Each trace lasted 15 minutes. Thetrace shows the video fragments generated from the H.263codec after the RTP header is added. Measurements aretaken before the addition of any UDP or IP packet header.

Figure 6: Video histogram for a maximum fragment sizeof 128 bytes

Figure 6 shows a plot of the video packet sizes againsttime with the maximum fragment size set to 128 bytes.The packets varied in size from as small as 25 bytes to themaximum value allowed, which is 128 bytes. The plotseemed too crowded as more packets are generated tomake up for the small value of the maximum fragmentsize.


Figure 7 shows a plot of the video packet size against timewhere the maximum fragment size is set to 256 bytes.Packets varied in size from as high as 256 bytes to as lowas 50 bytes. Figure 7 appears to be less crowded thanFigure 6.




Figure 8 shows a plot of the video packet size against timewith the maximum fragment size set to 512 bytes. Thepacket sizes ranged from 75 bytes to 512 bytes. The largerswing in the packet size range allowed for a less densehistogram compared to those in Figure 6 and Figure 7.

Table 1 shows the maximum fragment size and thecorresponding mean, standard deviation, and number ofgenerated fragments of the video traffic source.

MaxFrag

Meansize

StandardDev

Mean arrival Fragno.

500 191.64 127.2 275.027 3680

256 169.38 51.358 224.40 4464

128 94.581 21.46 132.568 6382

Table 1: Mean and standard deviation for differentfragment sizes

In Table 1, the smaller the maximum fragment size, thesmaller the mean size of generated fragments andconsequently the smaller the delay (smaller inter-arrivaltime). However, by examining the number of fragmentsgenerated for each maximum fragment size in Table 1, wenote that as the maximum fragment size decreases, thenumber of generated fragments increases. This isexpected, however, since the number of video frames tobe fragmented and formed into packets is almost fixed forall maximum fragment sizes with which we experimented.In addition, as the maximum fragment size decreases, thepacket overhead increases, and the bandwidth utilizationdecreases. Normally, it doesn’t take any longer to sendand receive fewer but larger video packets than it does to

send and receive more but smaller video packets, if thesame amount of video data is played. In fact, the extrapacket handling overhead may cause the reverse.However, larger video packets can cause audio latency asthe bandwidth is shared, and this is the major concern.

Video Packet Inter-Arrival Time and Jitter Effect

Another important measure we can deduce from the dataprovided by these graphs is the mean inter-arrival time.This measure provides an estimate for video packetlatency induced locally at the host. We compute the rate ofchange of the inter-arrival time, which is a measure ofvideo packet jitter. As the maximum fragment sizedecreases, it is expected that the number of packetsgenerated within the same interval of time will increase.The measurements for inter-arrival time and inter-arrivaljitter for maximum fragment sizes of 128, 256, and 512are shown in figures 9-11.

Figure 9 shows a plot of the inter-arrival time and jitter ofvideo packets against time where the maximum fragmentsize is set to 128 bytes. Note that many packets aregenerated with variant inter-arrival time and hence thejitter between packets is significant. As the maximumfragment size is increased to 256 bytes (as in Figure 10),the jitter is less significant as a smaller number of packetswith less delay variations is generated. Increasing themaximum fragment size further to 512 bytes decreases thejitter significantly as shown in Figure 11.

Figure 9: Inter-arrival time and jitter for video maximumfragment size of 128 bytes



Figure 10: Inter-arrival time and jitter for videomaximum fragment size of 256 bytes

In general, local jitter increases as the maximum fragmentsize decreases. This phenomenon is explained by the factthat more packets are generated for the smaller maximumfragment size during the same time interval. More packetswith a variety of inter-arrival times result in a greaterlikelihood of inter-arrival time variation and hence, jitter.

Theoretically, as the maximum fragment size decreases,the sizes of the packets are normalized and jitter should bereduced. However, the increase in jitter in this case is alsodue to packet handling overhead in the transport stackfrom the increased number of packets.

Figure 11: Inter-arrival time and jitter for videomaximum fragment size of 500 bytes

Multiplexing Video and Audio PacketsAudio and video packets are generated by their respectivesources and then funneled through a sharedcommunication medium. Each media component shouldsubscribe to a sufficient amount of bandwidth to meetquality standards. Lack of appropriate multiplexingalgorithms can lead to one or more media componentsoversubscribing to the shared bandwidth and penalizingother participants.

A typical example of this multiplexing problem is whenthe scheduler dispatches all media types through a FirstCome First Served (FCFS) queue. All packets are servicedin the order they arrive without any attention being paid totheir priority or timeliness. In this example, if the videodata is bursty and a train of video packets is generatedduring an audio silent (inter-arrival) period, audio packetswill be blocked for the duration of the video packet train.This duration may be sufficiently long to disturb thecontinuity of the audio speech and make itincomprehensible. To illustrate this example, we fed audioand video traffic to the scheduler shown in Figure 1. Wechose a worst- case audio source that continuouslygenerates audio packets of four frames each every 120milliseconds. Video is of variable bit rate with maximumfragment size set to 256 and 512, respectively. We areinterested in the number of video bytes that residebetween audio packets during periodic silence intervals. Iftoo many video bytes accumulate between twoconsecutive audio packets, severe delay (and jitter) maybe experienced for the blocked audio packet.

We computed the mean number of video bytesinterleaving audio packets from the plot. We depicted thehost’s extra inter-arrival time penalty to audio packets astime incurred by interleaving video bytes. Further, byknowing the maximum number of video interleaved bytes,we can show the maximum penalty incurred on the inter-arrival time of audio packets that will show at the receiverside due to the local multiplexing effect. We can alsodeduce the mean local audio jitter by computing the rateof change of the interlaced video packets over theexperiment interval.

Figure 12 plots the number of video bytes residingbetween consecutive audio packets against time using anH.263 video source of maximum fragment size 256.Although it is commonly believed that using a smallermaximum video fragment size will lead to better audioperformance, our experience proved otherwise. It isobserved that the mean interleaved video bytes betweenaudio packets for a video maximum fragment size of 256is larger than the mean of the corresponding trace for amaximum fragment size of 512 as shown in Figure 13.Hence, reducing the video fragment size alone will nothelp the audio performance. On the contrary, it may lead



to irregular spacing in time between audio packets, whichrenders audio playback unintelligible.

Figure 12: Interleaved video bytes between audio packets(maximum video fragment size 256)

Figure 13: Interleaved video bytes between audio packets(maximum video fragment size 500)

The Problem with Media Multiplexing on theHostMultiplexing audio and video packets over a sharedbandwidth on the local host has its drawbacks. One majordrawback we observed was where video oversubscribes itsbandwidth. This leads to significant variable gaps betweenaudio packets that causes audio inter-arrival delay andjitter.

There are two reasons for this problem. The first reason isdirectly related to the variable size of the video packet or

fragment. This can be controlled by fixing the size of thevideo packets, which can be inefficient. The secondreason for the problem is related to the burstiness of thevideo source. Burstiness can be caused by the encoder,the packet fragmenter, or both. The remedy for theburstiness problem is a traffic regulator that controls theflow of video packets and maintains quality of service forall media types on the network. Other solutions include arecovery mechanism at the remote end for audio jittereffects, careful traffic monitoring and remote control ofthe violating traffic source, and use of priority schemes foraudio at the network level and below. These solutions arebeyond the scope of this paper.

ConclusionReal video traffic workload is collected and analyzed inthis paper. Worst-case G.723.1 audio traffic is used as theaudio source input. The measurements collected duringtraffic analysis are packet length, inter-arrival time, jitter,and packet overhead.

Analysis of audio traffic revealed a tradeoff between theaudio latency and the network bandwidth utilization.Increasing the number of frames decreases the packetoverhead and increases the network utilization. However,increasing the number of frames per packet also increasesaudio local-induced latency by the time taken to buffer theframes before assembling them into packets. The optimalnumber of frames for minimum latency and minimumoverhead is approximately seven per packet for anuncompressed header. This number will add latencyoverhead to the audio packet of 210 ms. Analysis of audiopackets with compressed UDP and IP headers revealedthe optimal number of frames to be five and six for lowand high bit-rate audio, respectively. These numbersinduce a local delay on the audio packet of 150 ms for lowand 180 ms for high bit rates.

Analysis of the video traffic revealed that the mean packetsize is less than 250 bytes for all fragment size limits. Thestandard deviation of packet size for all fragment sizes ofchoice is less than 180 bytes. The majority of the videopacket sizes lay in the range of 50-250 bytes. Thisinformation suggests that a maximum fragment size ofbetween 256 and 512 bytes is an appropriate choice.

It was also shown that as the maximum fragment sizeincreases, the mean inter-arrival time between videopackets increases, the number of generated fragmentsdecreases, and the inter-arrival jitter decreases. In theoryjitter should be reduced as the fragment size decreases.However, the increase in jitter in this case is also due tothe packet-handling overhead (from the increased numberof packets) in the transport stack by the packet assembler.



The local audio latency and jitter experienced by theaudio-video scheduler are due to variations in videofragment size and burstiness of the video source. A newperformance measurement was introduced. Thismeasurement is the number of interleaved video bytesscheduled between audio packets. This number is a goodindicator for audio latency and jitter at the host.

These results help assess H.323 terminal traffic locally atthe host during a video conference. They provide usefulinformation for tuning media traffic parameters. Theresults also provide a better understanding of theaudio/video multiplexing problem over a sharedbandwidth medium, and they enable the development ofeffective solutions.

AcknowledgmentsThe author would like to thank all his colleagues at IntelArchitecture Labs. Special thanks and acknowledgmentare extended to Steve Ing and Jose Puthenkulam of IALfor productive discussions during the various stages of thiswork.

References[1] Casner S., Jacobson V., “Compressing IP/UDP/RTP

Headers for Low-Speed Serial Links.” Internet Draft,draft-ietf-avt-crtp-03.txt, July 1997.

[2] ITU-T, Recommendation G.711 (1988)–Pulse CodeModulation (PCM) of Voice Frequencies.

[3] ITU-T Recommendation G.723.1 (1996)–Dual RateSpeech Coders for Multimedia CommunicationTransmitting at 5.3 & 6.3 kb/s.

[4] ITU-T, Recommendation H.261 (1993)–Video Codecfor Audiovisual Services at p X 64 kb/s.

[5] ITU-T, Recommendation H.263 (1996 –VideoCoding for Low Bit-rate Communication.

[6] ITU-T Recommendation H.323 (1996 –Terminal forLow Bit-rate Multimedia Communication over Non-Guaranteed Bandwidth Networks.

[7] Schulzrinne H., Casner S., “RTP: A TransportProtocol for Real-Time Applications,” draft-ietf-avt-rtp-04.txt, October 1993.

Author’s BiographyHani ElGebaly is the project technical lead for the H.32xprotocols development within the conferencing productsdivision at Intel Architecture Labs. He received an M.Sc.in Computer Science from the University ofSaskatchewan, Canada, and a B.Sc. in ElectricalEngineering from Cairo University, Egypt. Mr. ElGebalyis currently pursuing a Ph.D degree with the University of

Victoria, Canada. His primary areas of interest includemultimedia conferencing protocols, embeddedprogramming, fault tolerant systems, and computerarchitecture. His e-mail address is [email protected]

Intel Technology Journal Q2, 1998...Intel Technology Journal Q2 ‘98 An Overview of Advanced...

Documents

Transcript of Intel Technology Journal Q2, 1998...Intel Technology Journal Q2 ‘98 An Overview of Advanced...