Guide Commodity DataServer Data and · PDF fileCOMMODITY DATASERVER DATA AND DEVELOPMENT GUIDE...

Commodity DataServer Data and DevelopmentGuide

Copyright © 2011Morningstar, Inc. All rights reserved. The Morningstar name, Morningstar Commodity Data name and logo are registered marks of Morningstar.Marks used in conjunction with Morningstar Commodity Data products or services are the property of Morningstar or its subsidiaries.

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDETable of Contents

iii

Table of Contents

Conventions........................................................................................................................................ xiiiCOMMODITY DATASERVER OVERVIEW................................................................................................1

Chapter 1: Commodity DataServer...............................................................................................3Introduction.......................................................................................................................... 3

Chapter 2: Commodity DataServer Architecture........................................................................... 5System Highlights.................................................................................................................. 5

Architecture..................................................................................................................6Database Server Architecture.................................................................................................. 7Database Types..................................................................................................................... 7

Ticker Symbol Databases................................................................................................ 8Time-Series Databases................................................................................................. 10Putting It All Together................................................................................................... 11Database Illustration..................................................................................................... 12

Query Execution Engine......................................................................................................... 13Application Programming Interfaces (APIs)................................................................................ 14

Chapter 3: Database and Data Guidelines.................................................................................. 17Structure of a Commodity DataServer Database.........................................................................17Building Your Own Database.................................................................................................. 18Using Unique Identifiers........................................................................................................ 18Using Your Company Name................................................................................................... 20Creating an Empty Database.................................................................................................. 20

API...................................................................................................................................................23Chapter 4: BMIM Scripting Language........................................................................................ 25

Introduction........................................................................................................................ 25Invoking the BMIM Client...................................................................................................... 25Features and Conventions......................................................................................................26Handling Multiple Databases.................................................................................................. 26Updating a Database............................................................................................................ 27

Locking and Unlocking the Database................................................................................27Adding, Modifying, and Deleting Relations........................................................................ 28Adding, Modifying, and Deleting Columns......................................................................... 33Adding, Modifying, and Deleting Relation Columns..............................................................35Reading and Writing ASCII Data Files...............................................................................40Deleting Facts............................................................................................................. 48Assigning Holiday Schedules.......................................................................................... 49Adding Relation Aliases.................................................................................................50Adding Formulas.......................................................................................................... 51Adding Correction Audits to the Table Facility.................................................................... 52Generating Continuous Contracts.................................................................................... 53Executing Saved Queries............................................................................................... 53

iv


Specifying Data Files/Partitioning..................................................................................... 54Printing the Database Partition Table................................................................................ 57Printing the Location of a Relation and Column.................................................................. 57Printing Database Schema Information............................................................................. 58Putting a Database into Safe Mode................................................................................. 61

Miscellaneous Commands..................................................................................................... 62Unix Interface..............................................................................................................62Setting Global Options.................................................................................................. 62Using Data Compression............................................................................................... 63Printing Expiration Dates................................................................................................64Entitlements................................................................................................................64

Chapter 5: C/C++ API............................................................................................................. 67The Basics..........................................................................................................................67

Invoking a Server......................................................................................................... 67Connecting and Disconnecting from a Server.....................................................................70Dealing with Optional Arguments.................................................................................... 72Metadata Terminology...................................................................................................74

Data Retrieval..................................................................................................................... 75Browsing the Relation Hierarchy..................................................................................... 75Examining the Columns of a Relation............................................................................... 79Retrieving Data Records................................................................................................ 81Writing Data to a File................................................................................................... 93Executing Queries........................................................................................................ 98XML Results..............................................................................................................104

Data Updating................................................................................................................... 107Locking the Database..................................................................................................107Adding Relations, Columns, and Relation Columns............................................................ 108Storing a Vector of Values........................................................................................... 111Loading Data From a File............................................................................................. 113High Frequency Updating............................................................................................. 114Adding Formulas (Dynamic Time Series)......................................................................... 122

Memory Management.........................................................................................................123Data Source Table: A Complete Application............................................................................. 125

Creating/Updating the Table......................................................................................... 125Querying the Table......................................................................................................128Utility Routines.......................................................................................................... 129

The Complete Interface: A Reference..................................................................................... 134Type Definitions..........................................................................................................134General Argument Handling.......................................................................................... 146Initialization and Termination......................................................................................... 147Supporting Multiple Databases..................................................................................... 148BMIM Command Equivalents........................................................................................ 151Schema Browsing...................................................................................................... 168Reading and Writing Data............................................................................................ 175High Frequency Updating............................................................................................. 189Executing Queries.......................................................................................................191


v

Table Facility..............................................................................................................193Correction Audits....................................................................................................... 200Custom Search Facility................................................................................................ 206Loading the Specification File ...................................................................................... 210Memory Management.................................................................................................211Rollover Specifications.................................................................................................212Global Options........................................................................................................... 221Date/Time Utility Routines............................................................................................224Logging Utility Routines............................................................................................... 227Error Handling............................................................................................................ 228Accessing Server Information....................................................................................... 228Linking Libraries in C/C++ on Windows........................................................................ 229

Chapter 6: Java API .............................................................................................................. 235Required Software..............................................................................................................235API Library Files................................................................................................................. 235Getting Started.................................................................................................................. 235The Basics........................................................................................................................ 237

Metadata Terminology................................................................................................. 237Connecting and Disconnecting from a Server................................................................... 238Changes to the Interface............................................................................................. 239

Data Retrieval....................................................................................................................240Browsing the Relation Hierarchy....................................................................................240Examining the Columns of a Relation............................................................................. 242Retrieving Data Records.............................................................................................. 243Executing Queries.......................................................................................................250

Data Updating................................................................................................................... 254Locking the Database..................................................................................................254Adding Relations, Columns, and Relation Columns............................................................ 255Storing a Vector of Values........................................................................................... 256Loading Data From a File............................................................................................. 257High Frequency Updating............................................................................................. 258

Sample Programs............................................................................................................... 260GetRelation............................................................................................................... 260GetRelChildren........................................................................................................... 262QueryExecute............................................................................................................ 264

Error Messages..................................................................................................................269ONC/RPC Error Message..............................................................................................269

Chapter 7: .NET API................................................................................................................ 271Installation Requirements..................................................................................................... 271

Configuration with Microsoft Visual Studio...................................................................... 271Using the .NET API............................................................................................................. 272

Sample C# Programs................................................................................................. 272Error Messages..................................................................................................................291

ONC/RPC Error Message..............................................................................................291Chapter 8: WebServices API................................................................................................... 293

Overview.......................................................................................................................... 293

vi


Meta Data Retrieval Service................................................................................................. 293Java Example............................................................................................................ 294

Data Retrieval Services....................................................................................................... 296Java Example............................................................................................................ 296

Data Upload Service........................................................................................................... 299Java Example............................................................................................................ 300

Chapter 9: VBA API................................................................................................................ 303Installing the Commodity DataServer VBA API..........................................................................303

Commodity DataServer VBA API Installation Instructions.................................................... 303Introducing VBA API........................................................................................................... 303

VBA API Object Types................................................................................................. 303Programming with VBA API.................................................................................................. 304

Overview.................................................................................................................. 304XMIMServer.............................................................................................................. 306XMIMRecords and XMIMRecordsDouble......................................................................... 313XMIMRelation............................................................................................................ 340XMIMColumn............................................................................................................ 351XMIMRelCol.............................................................................................................. 356XMIMShowWhen and XMIMShowWhenDouble............................................................... 365XMIMExecution and XMIMExecutionDouble..................................................................... 375XMIMSchema............................................................................................................384XMIMTable................................................................................................................393XMIMCurrentTick........................................................................................................403XMIMCustomSearch................................................................................................... 411XMIMUtils (Constants)................................................................................................ 416XMIMUtils (Utilities)................................................................................................... 426

Commodity DataServer Visual Basic for Applications API Examples...............................................429Introduction............................................................................................................... 429Referencing the VBA API Library................................................................................... 429Extracting Data: XmimRecordsDouble & getdata()............................................................. 429Replicating XMIM Queries: XmimShowWhenDouble.......................................................... 432Accessing the Children of a Relation: XmimSchema ......................................................... 434Relation and Data Loading........................................................................................... 436

FRAMEWORK EXTENSIONS.............................................................................................................441Chapter 10: C Stored Function Framework............................................................................... 443

Description........................................................................................................................443Structure of a C Stored Function........................................................................................... 444Stored Function Framework.................................................................................................. 447

Regular Stored Functions............................................................................................. 447Built-in Stored Functions.............................................................................................. 453

Stored Function Life Cycle....................................................................................................454Example: Providing a C Stored Function Wrapper for an Existing C Function.................................... 454

Straightforward Approach............................................................................................ 454Lazy Evaluation Approach.............................................................................................462

Complete Predictor Code..................................................................................................... 463Complete Lazy Predictor Code.............................................................................................. 467


vii

Chapter 11: MIM/SQL External Database Implementation..........................................................471Introduction....................................................................................................................... 471Key Commodity DataServer Concepts.................................................................................... 471Implementation.................................................................................................................. 472Accessing Time Series........................................................................................................ 472Accessing Meta Data Information..........................................................................................473Hierarchy, Types and Categories............................................................................................ 473Column Model................................................................................................................... 474Relation, Column and Data Presence...................................................................................... 474

Chapter 12: Correction Audits................................................................................................. 475Introduction....................................................................................................................... 475Correction Audits............................................................................................................... 475Adding Correction Audits to the Table Facility.......................................................................... 475

BMIM Synopsis......................................................................................................... 476Commodity DataServer C/C++ API Synopsis.................................................................. 476

Retrieving Correction Audits from the Table Facility................................................................... 477Commodity DataServer C/C++ API Synopsis.................................................................. 478

Deleting Correction Audits from the Table Facility..................................................................... 478Commodity DataServer C/C++ API Synopsis.................................................................. 479

Correction Audits File Format................................................................................................479Accessing Data with Correction Audits Applied........................................................................ 480Using the Table Facility to Retrieve or Update Corrections.......................................................... 481

Daily Table................................................................................................................ 482Intraday Table............................................................................................................ 482

LOADING AND EXTRACTING DATA.................................................................................................. 483Chapter 13: Best Practices for Data........................................................................................ 485

Best Practices for Loading Data............................................................................................ 485Loading Data using BMIM and the Package Maker............................................................485Loading Data using the File Loader................................................................................ 485Loading Data using the Formula Loader.......................................................................... 486Loading Futures Data.................................................................................................. 486Loading Data using the Commodity Add-in...................................................................... 486Loading Data using the APIs.........................................................................................486

Extracting/Reading Data.......................................................................................................486Commodity Query Software..........................................................................................487Commodity Charts Software......................................................................................... 487Commodity Add-in Software.........................................................................................487BMIM Scripting Language............................................................................................ 487Application Programming Interfaces (APIs)...................................................................... 487

Chapter 14: Software Installations for Loading Data................................................................. 489Product Bundle – (Formula Loader, Package Maker and File Loader).............................................. 489

Installation Bundle Instructions...................................................................................... 489Chapter 15: Loading Data using the Package Maker................................................................. 491

Data Packages................................................................................................................... 491Building Data Package Components....................................................................................... 492

The make_data File.................................................................................................... 492

viii


Data File................................................................................................................... 494Running the Package Maker......................................................................................... 495Loading Data Packages and Checking Logs...................................................................... 495

Chapter 16: Loading Futures Type Data (Monthly Curves)..........................................................497Overview.......................................................................................................................... 497

Futures Data Concepts................................................................................................ 498Summation............................................................................................................... 499

Getting Started.................................................................................................................. 500Futures Type Data Naming Conventions.................................................................................. 501

futures_contract Naming Conventions............................................................................ 501futures_continuous Naming Conventions.........................................................................502

Creating a Futures Type “make_data” File............................................................................... 503Creating Categories, Symbols and Columns..................................................................... 503Creating Continuous Contracts...................................................................................... 505

Creating the Futures Type Data File(s).................................................................................... 506Testing the Results............................................................................................................. 506Generating Continuous Contracts...........................................................................................506

Chapter 17: Loading Data from a File (File Loader)................................................................... 509Overview.......................................................................................................................... 509The .csv File Format............................................................................................................509The XML File Format...........................................................................................................510Running the File Loader....................................................................................................... 511

Command Prompt Usage............................................................................................. 511Final Ouput for .csv Example.........................................................................................512Final Output for .xml Example....................................................................................... 512

Chapter 18: Formula Loader.................................................................................................... 513Overview.......................................................................................................................... 513Running the Program.......................................................................................................... 513Creating Formula Query Files................................................................................................ 513

Creating the Query..................................................................................................... 513Saving the Query....................................................................................................... 514Creating Nested Formulas............................................................................................ 515Formula Loader Script................................................................................................. 515

Running Queries for Multiple Relations using LET...................................................................... 515Creating the LET Query................................................................................................516

Final Ouput....................................................................................................................... 517Chapter 19: Units of Measurement (UOM)............................................................................... 519

Create the Units of Measurement File.................................................................................... 519MimicUM_Units......................................................................................................... 519MimicUM_CategoryUnits............................................................................................. 519MimicUM_RelColConversion......................................................................................... 520MimicUM_Conversions................................................................................................520MimicUM_Currencies..................................................................................................520

XMIM_LOCAL_UNITS..........................................................................................................521Running the “units_measure_load” Command..........................................................................521

SERVER ADMIN GUIDE................................................................................................................... 523


ix

Chapter 20: Invoking the Commodity DataServer...................................................................... 525Chapter 21: Commodity DataServer Security............................................................................ 529

Security Overview.............................................................................................................. 529Server Security.................................................................................................................. 529

Global Setting............................................................................................................ 530xmimsvr.acl File......................................................................................................... 530Free Pass..................................................................................................................531If Connection is Denied............................................................................................... 532

Database Security.............................................................................................................. 532Default Setting.......................................................................................................... 532Free Pass..................................................................................................................532xmimlock.acl File Format..............................................................................................533If lock_files is Denied.................................................................................................. 533

Data Security (aka Entitlements)........................................................................................... 534Concepts.................................................................................................................. 534Usage Guidelines........................................................................................................535Free Pass..................................................................................................................532Enabling/Disabling Entitlements..................................................................................... 536If Access is Denied.................................................................................................... 536Entitlement File Format................................................................................................537Importing Existing Entitlements..................................................................................... 538

Chapter 22: Commodity DataServer Logging............................................................................. 539Introduction....................................................................................................................... 539Log Formats...................................................................................................................... 539Logging Levels................................................................................................................... 540What is Logged................................................................................................................. 540Sample Reports................................................................................................................. 542Resetting the Log File......................................................................................................... 542Reference Documents......................................................................................................... 543

Report 1: Web Server Statistics (Week of 05/09/99 to 05/15/99).........................................543Report 2: Web Server Statistics (Week of 05/09/99 to 05/15/99).........................................549

Chapter 23: Using Multiple Databases..................................................................................... 557Introduction....................................................................................................................... 557Initial Specification of Databases........................................................................................... 557Ordering of Databases.........................................................................................................557Semantics of Multiple Databases.......................................................................................... 558Managing the Current Database Pool/View.............................................................................. 558

Creating, Opening and Closing Databases........................................................................559Changing the Database View........................................................................................ 560Listing the Current Database View.................................................................................561

Partitioning Data Types Across Databases............................................................................... 561Chapter 24: MIM/SQL External Database Implementation..........................................................471

Introduction....................................................................................................................... 471Key Commodity DataServer Concepts.................................................................................... 471Implementation.................................................................................................................. 472Accessing Time Series........................................................................................................ 472

x


Accessing Meta Data Information..........................................................................................473Hierarchy, Types and Categories............................................................................................ 473Column Model................................................................................................................... 474Relation, Column and Data Presence...................................................................................... 474

UTILITIES........................................................................................................................................569Chapter 25: xmim_get Utility................................................................................................... 571Chapter 26: xmim_get_options Utility...................................................................................... 573Chapter 27: xmim_trading_info Utility...................................................................................... 575Chapter 28: xmim_svr_info Utility............................................................................................577Chapter 29: xmim_find Utility.................................................................................................. 579Chapter 30: xmim_get_rel_paths Utility................................................................................... 581Chapter 31: xmim_get_col_paths Utility................................................................................... 583Chapter 32: xmim_put_milli Utility........................................................................................... 585Chapter 33: MIMDBCP Utility (Copying Database Contents)...................................................... 587

How to Setup MIMDBCP..................................................................................................... 588Methods for Running the MIMDBCP Utility.............................................................................. 588How it Works.................................................................................................................... 588

Chapter 34: MIMDBCK Utility (Database Integrity Check)..........................................................591How to Setup MIMDBCK..................................................................................................... 592How it Works.................................................................................................................... 592

Delete Option............................................................................................................ 592Schema Check...........................................................................................................593Percent Complete Indicator.......................................................................................... 593

Chapter 35: Delta Publisher Utility........................................................................................... 595Overview.......................................................................................................................... 595

Chapter 36: Index Constituent Utility....................................................................................... 597Chapter 37: dbcompare Utility................................................................................................. 601Chapter 38: xmim_pair_gen Utility........................................................................................... 603

Overview.......................................................................................................................... 603How to Use the Utility........................................................................................................ 603Managing Size................................................................................................................... 605Table of Iterations...............................................................................................................605One to Many Comparison.................................................................................................... 606

APPENDIX...................................................................................................................................... 607Chapter 39: Using the Package Maker..................................................................................... 609

Requirements.................................................................................................................... 609How to Use the Customer Data Package Maker....................................................................... 609Example Usage.................................................................................................................. 609

Creating the upd Package File.......................................................................................609Unpacking the upd Package File.................................................................................... 610

Chapter 40: Commodity DataServer .xmimrc File...................................................................... 611Locating the .xmimrc File..................................................................................................... 611Specifications.................................................................................................................... 611

Example .xmimrc File.................................................................................................. 612License File Location................................................................................................... 612Databases Used.........................................................................................................612


xi

Directory Locations..................................................................................................... 613Custom Search Spec File Location.................................................................................613NaN Handling............................................................................................................ 613Printing Formats......................................................................................................... 613Global Options........................................................................................................... 614Activate Delta Publisher...............................................................................................614Print Directly from prn File in facts_read......................................................................... 614Entitlements Usage.....................................................................................................615Read Locking.............................................................................................................615Publish Continuous Daily Contract Changes......................................................................615

Executing the .xmimrc File as a Script.................................................................................... 616Index.............................................................................................................................................. 617

xii


COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEConventions

xiii

Conventions

This guide uses the following conventions to convey instructions and information:

Command descriptions use these inputs:

● Text that you see in the application (for example, windows and menus) and keywords are in boldface.

● Arguments for which you supply values are in italic.

● Programming code and Information that you must type exactly as it appears is in typewriter font.

Notes, tips, cautions and warnings use the following conventions and symbols:

Means reader take note. Notes contain helpful suggestions or references to materials not contained inthis document.

Means the following information will help you solve a problem. The tips information might not betroubleshooting or even an action, but could be useful information.

Means reader be careful. In this situation, you might do something that could result in equipmentdamage or loss of data.

This warning symbol means danger. You are in a situation that is dangerous for your equipment, maydamage your data or may cause bodily injury.

xiv

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEConventions

Part I: Commodity DataServer Overview

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 1: Commodity DataServer

3

CHAPTER1

Commodity DataServer

IntroductionThe Commodity DataServer is an enterprise data warehouse used to retrieve time-series data at high speeds.It provides a consistent platform for storing, organizing, categorizing and manipulating disparate raw data. TheCommodity DataServer is an excellent solution for data management and administration.

4

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 1: Commodity DataServer

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 2: Commodity DataServer Architecture

5

CHAPTER2

Commodity DataServer Architecture

This chapter outlines the components that define the Commodity DataServer and details how the server and itsexecution engine work together to provide the capability to analyze and manipulate raw data and perform such tasksas: event driven analysis, cross-market analysis, P & L studies and more all in a near-English language query format.

System HighlightsThe following list details the major components and features that make up the Commodity DataServer environment:

● C/C++, Java, .NET, Web Service APIs● The BMIM Scripting Language● Server platforms are Solaris and Linux● Client platforms include Solaris, Linux, and Windows 7/XP/Vista● Built-in knowledge of financial markets● Specialized tools for equities, futures, options, fundamental data● Sophisticated rollover language● Holiday schedules● Distinction between types of NaNs and different fill options● Real tick, intraday and daily data● Aggregation and frequencies● Dynamic schema (extensional database)● Entitlements● Database utilities● High-frequency data handling● Weekend data● Multi-valued data and Multi-field data● Time-varying trading patterns● Non-contiguous trading patterns● Multiple databases and External databases● Custom search facilities● Multiple access paths to ticker symbols

6


ArchitectureThe following shows the client-server architecture of the DataServer.

The client/server process functions as follows:

● The process is made up of four components – (1) Master Server (2) Slave Servers (3) Clients (4) Database● The Master Server is the controlling process. Requests are made from the clients and the Master Server divides the

workload among the slave servers.● The Slave Servers access the database performing the bulk of the work.● Once the client process has been assigned a Slave Server, communication continues between the client and Slave

Server until the request is complete.● Apart from WebServices, all clients use the rpc protocol to communicate with the server.● If a request is made for data, the Master Server will return the simple hostname and Slave Server port number to

the client. The client will then establish communications with the Slave Server● There may be multiple Master Servers running on a single machine, each with a unique port number.

It is very important that the client be able to resolve the simple hostname that the Master Serverreturns.


7

Database Server ArchitectureThe features highlighted in this section include:

● Database server with master/slave architecture● Database integrity (locking, reliable commit)● Entitlements● Query language execution engine● Multiple databases● External databases

The Commodity DataServer database server is a high-performance, scalable engine designed to process time seriesdata. Clients can connect through one of several APIs. Multiple clients can connect simultaneously to a given server.The server then routes the request to one of a pool of slave servers, thus achieving parallelism on distributed or multi-processor environments.

For some time critical applications, such as high-frequency updating from a real-time feed, the master server canconnect the client directly to a dedicated slave server. This significantly reduces the overhead for all clients connectingto that server.

The server supports simple retrieval of the data stored in the database. It also supports a sophisticated query languagewhich was designed specifically for market historical research.

The Commodity DataServer database itself resides on the UNIX file system. The data is kept in a proprietary formatdesigned specifically for efficient retrieval of time series data, featuring compression of historical data and fastupdating of real-time data. The database can be split across multiple disks and/or file systems.

The Commodity DataServer is capable of reading in multiple databases and presenting them to the user as a singledatabase. Multiple databases facilitate segmenting of the data into separate namespaces such that, for instance, thecustomer is able to keep their own proprietary data separate from data provided by Morningstar Commodity Data. Anyuser can extend the server so that it understands a different database format, e.g., using a call-level interface into arelational database. Thus, existing DataServer client applications can be leveraged by tying in different databases.

Database TypesThe database server has to maintain a lot of different types of information with different access patterns. For example,the database knows that DELL stands for DELL Inc., that it trades on the NASDAQ Mon-Fri from 9:00am to 4:00pm (ET), and that on 5/8/96 it closed at 1.4062. To support these different types of data and to optimize for thedifferent expected usage, the server manages four different databases. The use of these different databases is totallytransparent to the user and many parts of the system itself.

The following details the two different database categories used. The first category, Ticker Symbol Database, managesall the information needed for each ticker symbol. The second category, Time Series Database, manages the timeseries data.

8


Ticker Symbol Databases

In-memory (Memory-mapped) Schema Database

● Object-oriented: manages lots of different types of objects● Supports inheritance

Extensional Database (Table Facility)

● Manages extensions to the schema● Relational in nature (but not SQL)● Not in-memory; uses optimized B-tree storage to effect fast disk retrieval● Power lies in fast selects with inheritance (property p of relation r)

Time Series Databases

Historical Tick Database

● Manages historical data● Stored on disk● Optimized for fast data retrieval (assumes update frequency low)● Daily, intraday tick (minute bars), real tick (second bars and prices)

Current Tick Database

● Manages high-frequency data (e.g., from a data feed)● Stored on disk● Optimized for fast update (at expense of slower retrieval time)● Intraday and real tick optimized

Ticker Symbol DatabasesThere is a lot of information associated with each symbol in the database (name, description, exchange, trading time,etc.). All this information is stored either in the in-memory schema database or in the extensional database. Some ofthe information is necessary for the system to be operational in a minimal form and is needed all the time:

● Name (Ticker Symbol) - used for parsing queries, displaying menus, etc.● Inheritance Tree - because of searches

Since these items are used so frequently, the information is stored in-memory. Auxiliary information where the accessneed is much less critical/frequent is not kept in-memory but is stored on disk instead. This auxiliary information is keptin the extensional database.


9

There is a balance between what needs to be stored in-memory for efficiency versus what should be stored on disk.The DataServer uses memory mapped files to minimize the impact on swap space requirements.

The schema is an object-oriented database supporting inheritance. It manages a set of relations, corresponding toticker symbols (DELL); columns, corresponding to attributes of relations (Close); and relation columns, correspondingto the historical data for a ticker symbol (Close of DELL). The extensional database, on the other hand, is relational innature, utilizing an optimized B-tree variant to effect very fast disk retrieval.

The relations and columns are organized in a hierarchical tree structure which is used to inherit both attributes oftickers and values for these attributes. For example, when SP is added to Futures, it inherits price columns as well asVolume and Open Interest. Other inherited properties are the data type of the relation columns (e.g., Volume of DELL isan integer), and the exchange where it trades, etc. Some of the information is inherited via multiple paths.

Trading times and trading patterns are not inherited.

Multiple relation and column hierarchies can be defined by using aliases. This allows, for example, symbols to beorganized by region as well as by industry. As many paths to a given symbol as desired can be defined.

There are many other objects and types. Some examples include:

● Description - String providing user-defined explanation of a relation.● Data Source - User defined data for a relation column.● Rollover Policy - Sophisticated language for specification of rollover switching date and policy (e.g., rollover on

second Friday of expiration month using a 3-month logarithmic perpetual policy). Rollover information is associatedwith relations and stored in the database. Continuous contracts are generated automatically and the systemupdates these automatically.

● Trading Patterns - Non-standard trading ranges, including weekend days and non-contiguous trading days/times(e.g., Mon, Wed, Fri, Sat.; 8am-2pm, 3pm-5pm).

● Segments - Time-varying treatment of trading periods (e.g., Mon-Sat until 1960 and Mon-Fri after that).● Aggregation Rules - Rules for combining data into desired frequency (e.g., averaging, sum, last value, first value,

high, low).● Sparse Data - Optimized storage and handling for this data type since it is conceptually a different object: the last

value is applicable until a new one comes out (e.g., CPI report).● Exchanges - Where a relation trades.● Time Zones - Normalization factor for when a relation trades.● Holiday Schedules - Alternative holiday schedules (e.g., foreign) available on a per-relation or per-exchange

basis. System differentiates between holidays and missing data and allows different filling specifications (NaN, fill-forward, fill-backward, linear/geometric/logarithmic interpolation).

● Entitlements - Read/write access privileges to data defined for specific users or groups of users (e.g., who hasaccess to IBES data or energy futures data or even down to the granularity of a specific column of a specificrelation, e.g., Cash of CL).

● Corrections History - Old values of the data maintained for risk evaluation.

10


● User-defined - Storage for other information defined by the user (e.g., website, address of company).

Time-Series DatabasesA time series is a function that takes a date/time and returns a value. This simple concept is complicated in twoimportant ways. First, a time series need not have a value for a particular date and time. In fact, it should behave asif certain dates and times do not exist, e.g., weekends. Secondly, a time series may have more than one value for aspecific date and time, e.g., data coming off a real-time feed. When these complications are factored in, a generaltime series resembles a relation instead of a function or even a partial function. From an implementation viewpoint, thesimple conceptual model yields a much more tractable solution.

● Ignore the extremes. Implement only the more common classes (daily, M-F) and ignore the rest. This approach iseasy to code and gives order (constant speed) access time.

● Implement the worst case. Treat all time series uniformly to achieve full-blown generality. This approach is veryflexible but results in slower (order log(n)) access for all time series because there is no way to exploit the regularityof the data. This is the relational approach.

● Implement special cases. Program special cases by writing lots of classes to exploit the regularity of the data,choosing the appropriate implementation for a specific time series at run-time. In some cases the performancewill be as good as the best case and in others, it will be comparable to the general case. The actual performancedepends on the specific time series. Probabilistically, the performance will be quite good since, by assumption,the more common cases can be accessed in constant time. This approach is both fast and flexible but difficult toprogram.

The database implementation takes the third approach. The implementation breaks time series into two orthogonalhalves, a time component and a storage component, each with many special cases.

● Time component: Calendar Abstraction

● Compresses the date/time space into a more rational sequence: converts date/time into indices by compressingholes.

● Calendar is the key of time series so important to do a good job of it.

● Perfect place to exploit regularity due to all the variety: easy calendar (M-F) is so common and nice that get largebang for buck by special casing.

● Calendar handles: different frequencies: n seconds, minutes, hours, days, weeks, months, quarters, years. - datearithmetic (e.g., 10 days ago) - contiguous/noncontiguous trading days and times - multi-segment (time-varyingtrading patterns)

● Conceptually since days are converted to indices, all that’s left is an array lookup. This is true in some cases but notin the more complicated ones.

● Storage component


11

● Gets location and figures out a value.

● Handles: - multi-value data (e.g., real tick where multiple values per time stamp are stored) - multi-field data(multiple fields per value) - sparse/non-sparse data - float, integer, double data types for single-field - grossgranularity (tick storage different than daily)

● When retrieving, time series sometimes kept in memory and sometimes not (daily fits but tick does not) so someread in all and process whereas others read in pages at a time.

● Depending on how much data involved and type will have arrays, hashes, nested arrays, B-trees, etc. Some timeseries are as simple as arrays; some are disk-based objects that are trees so lookup involves traversing a tree withall the complications of virtual pager/paging policies, caching in and out, modification of buffers, functions to findthe right bucket, etc.

● Options time series have their own special storage class because they have strike prices, calls/puts, expirationdates to handle. - Could use regular storage class to store options but replicating information if do this so we use aspecial storage class to renormalize and avoid replication. - Options are inherently multi-valued at the physical layer(in one date/time slot can have more than one option trading).

● Compression of data is an interesting issue because one cannot just simply take time series history, compress itand store it on a disk. - Some history is very large and one would not want to compress/uncompress it every timethere is an update. - We break it up into chunks and access via discrimination nets. Buckets are compressed byusing the tree to walk down the hierarchy until a chunk of the right size is obtained. To read, the appropriate bucketis found, read into memory and uncompressed. Block sizes are tunable in the system. - We compress tick data butnot daily. This has empirically been determined to be the correct trade-off. It is a big win to compress tick data dueto the magnitude of data: the savings in I/O time more than makes up for the overhead of compression. But, withdaily data, you lose in terms of speed with no appreciable gain so compression is not used. - We use a standardcompression algorithm which has proven to be quite acceptable. As an example, compression was turned off foran update and the database grew to 350MB vs. 80MB when compression was employed. - Compression can bespecified on a per-time series basis and thresholds set for the data size, below which compression will not be used.

By splitting up time series into these two components, we achieve a great deal of code reuse. Rather than specialcase all possible combinations of calendar and storage classes, we need only implement each calendar and storageclass and then combine the two. We can achieve even more code reuse by using inheritance (both is-a and has-a) and templates to implement the calendar as well as the storage classes. For example, a has-a hierarchy is a naturalway to handle multi-segment calendars, with care taken around segment boundaries. This benefits not only thecoding, but also the execution, since one or more of the segments may be special cased (e.g., M-F).

Putting It All TogetherThe net result is a very complex virtual C++ class hierarchy implementing time series. However, once an objectis created, interactions with it are simple since the class hierarchy implements a unified interface. Moreover, oncea public virtual call is made, nested calls are non-virtual; hence, the performance penalty of using virtual classes isminimized. The complex task of instantiating a specific time series object is delegated to a time series factorywhich, among other things, caches the more frequently used time series. The factory also interacts with the currenttick database to ensure that the newly-created time series has the most recent values.

12


Consider a simple time series example:

● Single-valued, float● Daily● Mon-Fri

This time series could represent the closing price of DELL. Since the trading pattern is so regular, a constant ordercalendar can be used. Moreover, a 10 year history of this time series requires only 2500 values so an array is theappropriate storage choice.

In contrast, consider the following worst case example:

● multi-value data● multi-field non-aligned (e.g., float, char(1), int – not mod 4 address so can’t use integer to integer copy in the

system to get the value)● multi-segment (trades Mon-Sat and then switches to Mon-Fri)● noncontiguous in both dimensions (daily: Mon, Wed, Thur, Fri, Sat; tick: 8-2, 3-5)

There is nothing to exploit with this time series. Hence, both a general calendar and storage class are required.

To improve performance, the system relies on caching. Most time series processing accesses the time serieselements either sequentially or in sequential windows, hence caching can result in significant speed-ups. For example,the calendar class caches the “current” segment in a multi-segment time series. The storage class caches thelocation of a value in a multi-valued time series, which avoids doing an order log (n) search for nearby values.

In other cases, where the time series has some but not all elements of the simple case, we can still exploit whateverregularity exists. For example, if a time series is single-valued, float, with a single non-contiguous segment, we canuse a simple storage class with a non-contiguous, single-segment calendar class. We pay the penalty for using a non-contiguous trading period, but we can still optimize the remainder of the time series, resulting in constant-access time.

The time series class is designed to provide fast access, and does so at the expense of slower update speed. Tobridge the gap with real-time feeds, where update speed is the most critical factor, the server uses a separatedatabase specialized for extremely fast update (approximately 1000 transactions per second.) This database is usedby the time series factory to fetch the latest values when a time series is created. Thus, the users of the time seriesclass are unaware of the current tick database’s existence.

The data stored in the current tick database can be permanently merged with the historical time series data. Thisfacilitates a way to do updates during the day (either intraday tick or real tick) and, at the end of the day either migratethe current tick data into the historical data or discard it in favor of an update that has been processed and “cleaned”.Folding into the historical data can be done selectively and the data can be folded into daily, intraday or real tick withaggregation done automatically by the system as appropriate.

Database IllustrationTo illustrate how all the system’s databases inter-operate, consider a simple query:


13

SHOW 1: BAR of DELL WHEN Date is within 100 days

First of all, the system will use the in-memory schema to look up the symbol DELL and decide to use the DELL calendarto execute the query (since it is the only symbol present in the query). The choice of calendar gives meaning to thephrase 100 days. Then the in-memory schema and extensional database are accessed to get useful information onDELL such as:

● Precision to be displayed in.

● Holiday schedule and NaN filling rules.

● Whether the data is sparse or non-sparse.

● The system will then go to disk and read the time series from storage.

● Then the current tick database will be accessed to find if there are any recent prices for DELL that need to beincluded.

● Caching is involved such that if any of the Bar prices for DELL are asked for again, the same thing will be returned,i.e., the most recent prices of DELL affect all four components of the Bar, but it would be a gross mistake toconsider a more recent price when computing the Close component of DELL than when computing its High.

● The execution code actually does not know of the four different databases, as far as it is concerned, everything is inmemory.

● The execution engine has all it needs:

○ Time series objects - can ask for value in this range

○ Calendar object - figure out what date is within 100 days

After all the information is accessed then it is just a matter of report generation.

Query Execution EngineThe Commodity DataServer Query Language is very high-level, extremely powerful and rich.

Among other things, it supports:

● Morningstar Commodity Data’s patented Show-When technology● LIM’s patented analog indicators● event-driven analysis● cross-market analysis

14


● P&L studies○ Good equity model○ Sophisticated signals and order types○ Comprehensive statistics○ Capital balancing○ Trading of baskets w/report for basket combined and individual in any desired frequency○ Specification of #contracts traded in variable amounts based on a condition (e.g., buy more if volatility is high)

● P&L studies● highly sophisticated date language● macro language● looping/optimization including ranking and filtering of results

The DataServer Query Execution Engine is built on top of the time series abstraction so it uses a lot of the time seriesnotions. Some of the real power stems from directly using the time series support in the server. For example,

● In the query language, data can be looked at in any granularity (3-day, 5- week, 2-month, 10-year, etc.). Days,months, etc. can be mixed in a single query. The aggregation of data is directly supported by the time series class.

● The calendar class provides sophisticated date arithmetic operations, which are also needed by the executionengine. For example, in computing a 20- day average, the calendar class is used to compute the endpoints of themoving 20-day window. The calendar class also provides direct support for the implementation of time offsets inthe query language (e.g., Close of DELL 3 weeks ago).

● The query language supports rollover on-the-fly. This requires creation of an intermediate temporary time series anduse of information from the schema to get the rollover policy. In fact, the actual rollover computation is also handledby the time series updating code, where historical rollover relations are maintained automatically by the database.

The need to support the query language drove some of the sophisticated technology in the server so there wasconsiderable synergy between the execution engine and server efforts. The Execution Engine adds its own level ofsophistication as well and some clients use both our client and server as a stand-alone system.

Application Programming Interfaces (APIs)Our applications have an open architecture with C/C++, Java, .NET, Web Service and the BMIM Scripting Language,allowing customization using APIs. Another use for APIs is to access data via third-party products.

The following list outlines some distinguishing API features and uses:

● C++ safe; compatible with both ANSI C and K&R C● Simple to use● Great for data retrieval and update for all four databases● Primary model is one of select● Get parent of DELL● Get Bar of DELL since 1980


15

● Issue request and get immediate answer● Have raw access to data● Have access to entire query language● Subsets of API ported to Java

16


COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 3: Database and Data Guidelines

17

CHAPTER3

Database and Data GuidelinesIt is important to understand and follow the guidelines listed below when loading proprietary data into your CommodityDataServer. Doing so will insure that your data integrity is not compromised.

Structure of a Commodity DataServer DatabaseUnderstanding the structure of a Commodity DataServer database begins with the relation. Relation is a special namefor a symbol or item in the database. Examples of relations are: DELL, HOUSTON.TX, or CHICAGO.IL. The term relationcomes from the mathematical concept of a function mapping from one set of items to another.

Relations can have multiple fields. The fields in a database are called columns. Examples of columns are: Open, Close,HighTemp, MidPoint, etc. Columns can be shared by many relations.

To store data as a time-series in the database, a relation and column are chosen to name the times-series data.For example, to store the prices for DELL, we can choose the relation-column “Close of DELL”. Any relation can beassociated with any column when relation-columns are created. A column can only be associated with a relation onetime. There cannot be two close columns of a single relation.

The relation, column and relation columns allow the database to relate information about the data to the users of thedata. In addition to a name, the relations and columns also have a description and other meta data items. For example,the relation DELL can have the description “Dell Incorporated, NASDAQ”. The column High can have the description“High Price”.

The fundamental unit of storage in the database is the time series. Time series are collections of the following items:

Relation, Column, Date, Time, and a numeric value.

An example of a time series:AUSTIN.TX, HighTemp, 7/1/2002, 97.0AUSTIN.TX, HighTemp, 7/2/2002, 96.0AUSTIN.TX, HighTemp, 7/3/2002, 94.0

The time component of a time series is optional. If the time is used, then the time series is called intraday. If the timecomponent is not used, then the time series is called daily. The Commodity DataServer keeps separate time series fordaily and intraday data. The database also can store data at resolutions smaller than 1 second, in that case the data isstored in a third form: tick-by-tick data.

All of the time series in the database are independent. The database understands that when two time series arerequested in a single operation, then the date and time indexes are properly aligned.

18


Most databases have many relations. In order to present the data in an organized way, there are relations without datacalled “category” relations. The categories in the database are similar to folders on a disk drive. Categories can containcategories or “normal” relations. By combining categories we create a hierarchy of data. The start of the hierarchy isalways the special category “TopRelation”. Normally, the hierarchy is depicted as a tree structure.

For example:TopRelation Weather UnitedStates Illinois CHICAGO.IL Texas AUSTIN.TX HOUSTON.TX

All relations have a complete pathname in the database. For example, HOUSTON.TX can have the path nameTopRelation:Weather:UnitedStates:Texas:HOUSTON.TX. Sometimes it is necessary to use pathnames to completelyspecify data items in order to move them in the hierarchy.

Normally pathnames are not used and the relation name used is a short name. In order for the short names to workthey need to be unique with a database. It is possible for a relation to have more than one name. This is achieved byusing “alias” relations. It is not necessary for category relations to have unique names. For example, the category,“UnitedStates”, could appear in a database many times.

The relation hierarchy serves two purposes in a database. One purpose is to provide the server information aboutspecific types of data such as futures contracts. The other purpose is to lend structure to the data for end users.

There is also a hierarchy for columns in the database. The column hierarchy always begins with “TopColumn”. Thecolumn hierarchy also provides information structure to the server. For example, composite columns are created usingthe parent child relationships of columns in the column hierarchy. Using column categories it is possible to refer tomany columns at the same time.

Building Your Own DatabaseIt is very important to not load any proprietary data into any Morningstar database whenever possible. If you absolutelymust load proprietary data into a database (such as adding new columns to existing series) please contact customersupport for further assistance. Doing so could compromise the integrity of the database.

Using Unique IdentifiersThe database is set up in a hierarchy style format which means any series you load will be loaded into a tree structure.Any series that you create for your database should have unique series names. Additionally, any category (folder) you

mailto:[email protected]


19

create must also have a unique name. This means that not only do your series names have to be completely unique ofother series names, but they also have to be unique of category names as well. The rules are:

1. Category names do not need to be unique from other category names.

2. Category names cannot match symbol names.

3. Symbol names must be unique from category names and other symbols present within the same category.

4. The following special characters cannot be used:* asterick+ plus sign- minus sign/ forward slash= equals sign< less than sign> greater than sign% percent sign# pound sign^ caret( left parenthesis) right parenthesis{ left curly bracket} right curly bracket[ left brace] right brace\ backwards slash| pipe: colon; semi colon’ single quotation mark

Note the following requirements:

1. New symbols cannot conflict with existing Morningstar symbols. A Morningstar symbolis considered different if it falls under a different path/tree/parent. For example,TopRelation:Equitites:GlobalInsight:Nasdaq:ExchangeTickers:d:de:DELL would be different in comparisonwith TopRelation:Energy:DELL.

2. A symbol cannot have a parent of TopRelation like TopRelation:Symbol. Instead it should be at a minimumTopRelation:Some_Parent:Symbol.

Standard case usage is that category names (folder names) are always mixed case with the folder name starting withan uppercase letter. Symbol names are always upper case and must begin with a letter or an underscore. No spacesare allowed. A symbol name may contain these special characters along with any special characters not listed on theexclusion list above:

● . period

20


● $ dollar sign● @ at sign● _ underscore

The following examples show correct naming usage:

● Equities:WidgetsInc:WID

● Equities:WidgetsInc:WID_AUSTIN

● Energy:WidgetsInc:WID@NY – Note how you can repeat a folder name under a different hierarchy path

The following is an example of an erroneous category and symbol:

● CompanyData:US:MORNINGSTAR.US – The category US is a symbol name and cannot be a category name aswell.

Using Your Company NameTo help insure that all categories and symbols are kept unique, one technique is to prefix them with your companyname. For example, since US already exists, Widgets Incorporated may call their new series WID.US. For a Futurescategory, use WidgetsFutures. The following is a good example of proper symbology for proprietary data loads:

● WidgetsIncorporated:WidgetsFutures:WID.US

All of the category names used in this example are unique as well as the symbol names. You do not have to create aunique category name but any symbol names that you add to a Commodity DataServer must be unique.

Creating an Empty DatabaseEvery Commodity DataServer is equipped with a program which will create an empty database. From the DBA homedirectory (normally /home/lim), run the command:

$ make_db_dirs [database name]

where database name is the name of the empty database to be built. Generally, custom database names follow theformat of data. and then company name. So, for Widgets Incorporated, the database name would be data.widgets.

Create the database data.widgets in the DBA home directory (normally /home/lim) using the following command:

$ xmim/bin/make_db_dirs data.widgets


21

Once you have created the new data.widgets database directory, the Commodity DataServer will need to be able toaccess its contents. This is done via the .xmimrc file, located in the DBA home directory.1

$ cd $ vi .xmimrc

A typical .xmimrc file’s contents look similar to this:

#!/bin/shecho "license: $LIMHOME/license/xmim.ids"echo "database: $LIMHOME/data/xmim.mim"echo "database: $LIMHOME/data.cust/xmim.mim"echo "database: $LIMHOME/data.gii/xmim.mim"echo "datesDir: $LIMHOME/dates/user"echo "holidayDir: $LIMHOME/dates/holidays"echo "tempDir: $LIMHOME/tmp"echo "entitlements: no"echo "read_lock: yes"if [ "$XMIMHOME" = "" ]then XMIMHOME="$LIMHOME/xmim" XMIMSLAVESERVERPATH=$XMIMHOME export XMIMHOMEfiecho "libraryDir: $XMIMHOME/library"for file in $LIMHOME/.xmimrc.local $LIMHOME/xmimrc/xmimrc.format.???do if [ -f "$file" ] then cat $file

Each line that begins with echo is a database that the Commodity DataServer is currently reading. Insert a linewith your new database beneath the $LIMHOME/data/xmim.mim in the .xmimrc file and save it. Adding thedata.widgets database then looks like this:

The following example assumes that data was loaded into $LIMHOME.

#!/bin/shecho "license: $LIMHOME/license/xmim.ids"echo "database: $LIMHOME/data/xmim.mim"echo "database: $LIMHOME/data.widgets/xmim.mim"echo "database: $LIMHOME/data.cust/xmim.mim"echo "database: $LIMHOME/data.gii/xmim.mim"echo "datesDir: $LIMHOME/dates/user"echo "holidayDir: $LIMHOME/dates/holidays"echo "tempDir: $LIMHOME/tmp"echo "entitlements: no"echo "read_lock: yes"if [ "$XMIMHOME" = "" ]

1 This example uses the “vi” text editor. For assistance editing the .xmimrc file, please contact a Client Services Support Representative.

http://customers.lim.com/Contact.htm

22


then XMIMHOME="$LIMHOME/xmim" XMIMSLAVESERVERPATH=$XMIMHOME export XMIMHOMEfiecho "libraryDir: $XMIMHOME/library"for file in $LIMHOME/.xmimrc.local $LIMHOME/xmimrc/xmimrc.format.???do if [ -f "$file" ] then cat $file

Now that the empty database is created and the .xmimrc file is altered, restart your server so that it can read thenew database. Verify that no users are running any queries or are using the database as restarting it will kill any usersessions. Do this by issuing the following command:

$ xmim_svr_info

This command will show which slave servers are running. If there are no slave servers running then it is free of users.

You will also need to verify that your database is not loading data when you reboot it. If you restart your server whileload_updates.sh is running, you can cause serious data corruption. The load_updates.sh process is slave port4091, and you can check to see if the writing slave process is running by issuing the following command:

$ ps –ef | grep 4091

If it returns processes for your Commodity DataServer (there could be multiple servers on multiple machines), thenwait until the processes complete. If nothing is returned then the server may be restarted:

$ start.server

To verify that the server is reading the database, run the following command:

The following example assumes the user is using port 0.

$ xmim_svr_info –p 0

The header will show the database list:

master server 4 on database is running process id: 29411 owner: lim log_level: 0 log_file: /home/lim/tmp/.xmim_server_4.log database 0: /home/lim/data/xmim.mim database 1: /home/lim/data.widgets/xmim.mim database 2: /home/lim/data.cust/xmim.mim database 3: /home/lim/data.gii/xmim.mim license: /home/lim/license/xmim.ids version: Version 4.4.13 Compiled Mon Mar 22 15:50:35 CST 2004

Part II: API

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 4: BMIM Scripting Language

25

CHAPTER4

BMIM Scripting Language

IntroductionThe BMIM facility allows users to interact with the Commodity DataServer in batch mode. BMIM is primarily designedas a facility to perform databases batch updates. It can also be used to perform other tasks, such as executing savedqueries or generating schema information.

Invoking the BMIM ClientThis write-up describes how to invoke the BMIM client process.

The BMIM client is invoked as follows:bmim_client [-h] [-v] [-s] [-p server_number] [-q server_host] [-d target_db1] [-d target_db2] ... [-d target_dbN] [- | cmd_files]

where:-h prints out a usage message

-v prints version and compilation information

-s specifies to execute the BMIM commands silently; that is, do not print out a message about each commandexecuted.

-p specifies the server_number used to identify the master server. The default server number is 0.

-q specifies the server_host for the master server. The default is the local host.

-d specifies the databases to be used, as given by target_db1, target_db2, etc. The databases specified must bea subset of the current database pool for the client and this will narrow the view to that subset. The currentdatabase pool consists of those databases that are specified in the client xmimrc (.cxmimrc) or, if no .cxmimrcexists, the server xmimrc (.xmimrc). The order that the databases are specified will set the search precedencewith symbol resolution to the first database in which the symbol is found. If one of the databases referenced isnot in the client's current database pool, the database names from that pool will be printed and the commandwill exit.

The cmd_files refer to BMIM-script files and these file names must be specified at the end of the command line.If no file name is specified, standard input is assumed. A hyphen (-) character may also be used to specify standardinput. Thus, the BMIM client, like BMIM itself, can be used to process BMIM-script files or can be used interactively.

26


An exit status of 3 is used by the BMIM client in the case of a timeout, server crash or, in general, any fatal error.

Features and ConventionsA valid identifier in BMIM consists of a letter or underscore (“_”) followed by any number of letters, digits,underscores, or the following characters: period (“.”), ampersand (“&”), dollar sign (“$”), percent sign (“%”), at sign(“@”). Relation names and column names specified in BMIM commands must be valid identifiers.

As a general convention throughout this document, string arguments are always quoted in a synopsis, just as theymust be when used in BMIM. When a string argument is a valid identifier, it may optionally be used in BMIM withoutthe quotes.

Whenever a filename appears as an argument to a BMIM command, the filename specification may include the fullUnix pathname or may consist of a relative pathname. If the pathname is relative then it is assumed to be relativeoff of the current directory of the BMIM process. If characters other than asterick (“ * ”), tilde (“ ~ ”), period (“ . ”),underscore (“ _ ”), backslash (“ / ”), upper/lowercase letters or digits are used, then the filename must be quoted.

Comments, denoted with the # character, can be used in a BMIM script. All characters following the # character tothe end of the line are ignored.

Handling Multiple DatabasesUpon startup, the Commodity DataServer either creates or opens the databases specified in the .xmimrc file as theinitial database pool. If a database specified in .xmimrc does not exist at DataServer startup, it is created as long as itsdirectory structure exists. Use the command make_db_dirs <databaseName> to create the directory structure ofa new database.

Use of the BMIM open/create/close commands will add or remove the specified databases to/from the pool ofdatabases accessible to this invocation of the BMIM client. Descriptions of these facilities may be found in Chapter 40,“Commodity DataServer .xmimrc File”.

Note that the order of specification for databases (i.e., the order in which the initial database pool is specified andthen in which open/new commands are used) is important as the databases will be opened in the order specified andmetadata information will be accessed in that order as well. Thus, for example, if the same relation exists in more thanone open database, the first one specified will be the one used unless disambiguating database name/path informationis provided. The databases to be used most should always be specified first.

The BMIM client supports temporarily restricting the databases viewed by narrowing the focus to a specified subsetof the currently open databases and then expanding back to entire open database pool. The synopses for thesecommands are as follows:database_narrow {


27

database = dbase name 1; database = dbase name 2; ... }database_widen

The database_narrow command specifies the list of databases that will be available as the current view ofdatabases. The databases specified must be a subset of the current open database pool. Databases that are notspecified will not be closed, they are just restricted from the view until a subsequent database_narrow commandincludes them or a database_widen command is given. Subsequent database_narrow commands will undothe effect of the previous database_narrow (i.e., the narrow command is not nested). The database_widencommand resets the view to the current pool of all open databases.

See “Specifying Data Files/Partitioning” for a discussion of partitioning daily, intraday and tick-by-tick data acrossdatabases and an example of using the database_narrow and database_widen commands.

In incorporating the use of multiple databases throughout this document, it is important to note that multipledatabases are viewed as though they were a single database and so the applied semantics is as if dealing with asingle database. If a database is thought of as a "tree" of nodes, then a multi-database is a tree that contains all pathsthat exist in any one of the single databases. Each path appears only once in the multi-database. Note that the nodesin the multi-database tree do not correspond to any one node in the single databases. Rather, a node in the multi-database tree is an amalgamation of the corresponding nodes in the single databases, e.g., it may have both daily andtick data, even though database #1 only has daily data and database #2 only has tick data.

In general, in the presence of multiple databases, any BMIM commands resulting in the addition and/or modification ofschema information will result in the information being added to all applicable database schemas. For example, if IBMis added to Equities:i:ib, then IBM is added to all database schemas that have a category called Equities:i:ib. Datainformation, however, is added to the first applicable database. For example, if daily data is added to Close of IBM, itis added to the first database that has a relation called IBM and a column called Close. Any BMIM commands havingto do with schema or data retrieval will retrieve information from the first applicable database. Delete commands willresult in schema or data information being deleted from all applicable databases.

For more information see Chapter 23, “Using Multiple Databases”.

Updating a Database

Locking and Unlocking the DatabaseBefore a database can be updated, it must be locked for exclusive write access. This is done using the lock_filescommand. Commodity DataServer supports a time series level granularity of locking for data updates and a per-bmimcommand level granularity of locking for schema updates but this is performed transparently to the user. The use oflock_files essentially puts the system in a mode whereby locking will occur as needed. After the database ismodified, the database can be unlocked using the unlock files command. This command serves as the final commit

28


and is when rollover computations will be initiated. There is an implicit unlock_files command at the end of everyBMIM-script file. The synopses for these commands are as follows:

lock_filesunlock_files

Adding, Modifying, and Deleting RelationsA relation can be added to the Commodity DataServer database using the relation_add command. Its synopsis isgiven below.

relation_add{ name = relname; description = "description"; parent = parent_relation; type = relation_type; trading_pattern = MTWHFSD; expiration_day = expiration_date; first_notice_day = first_notice_date; rollover = rollover_data_type; rollover_date = "rollover_date"; rollover_policy = "rollover_policy"; exchange = "exchange"; time_zone = "time_zone"; start_time = starting_trading_time; end_time = ending_trading_time; contract_units = contract_units;}

The relname is the name (i.e., ticker symbol) of the relation to be added. In Commodity DataServer, there is nounique name requirement for relations, even leaf relations (e.g., those with data associated) within the samedatabase. The description is a string containing a user-defined explanation of the relation. Typically, it might consistof the full corporation name or an expansion of the ticker symbol (e.g., International Business Machines Corp., S&P500 Composite Index). This attribute is used only for documentation purposes. If it is omitted, Commodity DataServerdefaults the description to be the same as the relname. The parent_relation is the name of the category to which therelation is being added. If the category name is ambiguous (i.e., there is more than one category with that name), a“path” to the correct category can be specified, using colons as separators. For example, the Indices sub-categoryof Futures can be unambiguously stated as Futures:Indices. When multiple databases are open, the relation willbe added to all database schemas that have the category specified by parent_relation. When non-unique relationsoccur within the same database, they must have distinct paths and the path specification must be provided when therelations are referenced so as to disambiguate them. For Commodity DataServer, the user may construct the entirerelation hierarchy, including the specification of the top-most categories. By default, the top-most relation categorieswill be Futures, Equities, Indices, Other, EconomicIndicators and MonetaryIndicators. TopRelation servesto identify the top-most categories, as in TopRelation:Futures. Only relname and parent_relation must bespecified for the relation_add command; all other arguments are optional.

Use the trading_pattern command to modify the trading days using the format “MTWHFSD” either withrelation_add or relation_modify.

29

The relation type specifies the type of the relation. The possibilities are as follows:

● category● normal● futures● futures_continuous● futures_contract

Relations of type category are used to group other relations, not to store data. Normal relations include all non-futuresleaf relations (e.g., indices, equities or options). Normal will be used as the default type for relations

Options data is stored just like a regular relation (relation type=normal). However, options must be stored underneatha special options relation category. For example, the options for DELL is stored under the DELL_Options category andoptions for IBM is stored under the IBM_Options category.

All options relations are required to have XML in the description field as such:<plist> <type> option_type </type> <strike> strike_price </strike> <expir> expiration_date </expir> </plist>

where:

● option_type is either put or call● strike_price is a numeric field (e.g., 30.0)● expiration_date is a date (e.g., 3/17/2003)

The <type>, <strike> and <expir> fields are required and may be entered in any order. All date entries must have4 digit years.

The description is entered all on one line.

Example:relation_add{ name = IBM_Options; type = category; parent = IBM;}relation_add{ name = IBM.199308.CALL.40; description = "<plist><expir>08/31/1993</expir><strike>40</strike><type>call</type></plist> parent = IBM_Options;}relation_add {

30


name = IBM.199307.PUT.60; description = "<plist><expir>07/31/1993</expir><strike>60</strike><type>put</type></plist> parent = IBM_Options;}

The futures type is reserved for the futures in the database (e.g., US or SP). A relation of this type correspondsto the main continuous contract for a future. It can also be the parent for any number of futures_contract relations.These are the individual contracts for the future (e.g., US_1994M). In addition, a relation of type futures may bethe parent for any number of futures_continuous relations. These relations are the different continuous contractscorresponding to the various rollover policies specified for the given futures.

A relation of type futures_contract may have an expiration_date and/or a first_notice_date; thesedates are specified in the format mm/dd/yy[yy]. They may also be specified in one of the other Commodity DataServerdate formats by quoting the specification. Relations of type futures or futures_continuous may containinformation pertaining to the rollover computation for the continuous contract. The rollover_data_type is usedto indicate whether rollover computations should take place for daily data, tick data, both tick and daily or neither. Thepossibilities are specified as:

● none● daily● tick● both

The default will be to compute rollover policies for both tick and daily. The syntax for rollover_policy and rollover_date is described in the document Rollovers. If not specified, the default rollover date used will be thelast day on which there is data for each contract and the default rollover policy will be to roll forward using the actualprices for the front contract.

The exchange is the name of the exchange where the security is trading, and the time_zone is the stringindicating the time zone for this exchange. Currently, these attributes are used for documentation purposes only.The starting_trading_time and ending_trading_time specify the time in the day at which the securitybegins and ends trading. The format for the times is given by hh:mm[am|pm]. The contract_units specify thedollar value (i.e., dollar equivalent of a one point move) for the contracts of a future. This feature is used for Profit/Loss queries. This contract_units specification is relevant only for relations of type futures; it is ignored for allother types. The exchange, time zone, and contract units are all arguments that are inherited from the parent relation.Binding is delayed as much as possible, such that they will remain inherited values until they are explicitly set. In thecase of the trading time arguments, when tick data is read in for any column of the relation, the attributes will benecessarily fixed2. It is, thus, imperative that these arguments be set to the desired values prior to reading tick data forthe relation. If no values are available to inherit (i.e., no values are specified for any ancestor relation), the system willuse 1 as the default value for contract_units.

The command in the following example adds the relation SP500.

relation_add { name = SP500; description = "S&P 500 Composite Index";

2 When tick data is read in, the trading times are used such that values that fall outside of the specified range are ignored.

http://customers.lim.com/pdfdocs/rollover_orig.pdf


31

parent = Indices; type = normal; time_zone = "US/Central"; start_time = 8:30; end_time = 15:15;}

The following commands add the futures relation SP.

relation_add { name = Indices; description = "Index Futures"; parent = Futures; type = category;}relation_add { name = SP; description = "S&P 500 Composite Index Futures"; parent = Futures:Indices; type = futures; rollover = daily; rollover_date = "15 days before last data day"; rollover_policy = "actual prices"; time_zone = "US/Central"; start_time = 8:30; end_time = 15:15;}

It is important to note that in order to add a child relation (e.g., SP), the parent must already exist (e.g.,Futures:Indices). Also, adding the Indices category to Futures creates ambiguity because the name Indicesis no longer sufficient to specify the category; instead, Futures:Indices has to be specified. Furthermore, ifanother relation is added to the original Indices category, the parent must now be given as TopRelation:Indicesbecause of the ambiguity. Indices is one of the top-most categories and TopRelation identifies it as such. Asmuch of the ancestry as desired can be specified, though only that needed to dis-ambiguate is necessary (e.g.,TopRelation:Futures:Indices could have been specified above, though only Futures:Indices was actuallynecessary). In general, the colon (:) character is used when a parent and an immediate child are specified, as shownabove. A double colon (::) may be used to avoid specifying the entire path since it is used to indicate a descendant,but is not limited to an immediate descendant. For example, TopRelation::Indices could be used, but would notserve to disambiguate since that would match both TopRelation:Indices and TopRelation:Futures:Indices.

Finally, some futures contracts and alternative continuous contracts can be created as follows:

relation_add { name = SP_1994M; parent = SP; type = futures_contract; expiration_day = 06/17/94;}relation_add { name = SP_3mo; parent = SP; type = futures_continuous; rollover_date = "15 days before last data day"; rollover_policy = "3 month perpetual";

32


}

The system enforces that all relations of types futures_contract and futures_continuous have names thatbegin with the name of the futures parent followed by an underscore (i.e., _) character. Thus, in the example above,the SP_constituent of the names was mandatory.

The required syntax for futures_contract:

● the first part of the name must be the entire parent’s contract name● an underscore must follow the parent contract name● a 2-digit or 4-digit year must follow the underscore (a 2-digit year is assumed to be 19xx)● one of the following letters must follow the year: F G H J K M N Q U V X Z

These codes represent January, February…December. Also note that the letter designates the endof the relation name.

Example: parentcontractname_YYYYM

The required syntax for futures_continuous is:

1. entire parent’s relation name2. an underscore3. name of the continuous contract

All three parts are required. The non Y2K form, NG_99Z is not supported.

Example: parentname_continuouscontractname

An existing relation can be modified using the relation_modify command. Its synopsis is as follows:relation_modify oldrelname { name = newrelname; description = "description"; parent = parent_relation; type = relation_type; trading_pattern = ..MTWHFSD; expiration_day = expiration_date; first_notice_day = first_notice_date; rollover = rollover_data_type; rollover_date = "rollover_date"; rollover_policy = "rollover_policy"; exchange = "exchange"; time_zone = "time_zone"; start_time = starting_trading_time; end_time = ending_trading_time; contract_units = contract_units;}


33

The oldrelname is the original name of the relation. The remaining arguments are all optional. They contain thedesired modifications and are identical to the arguments of relation_add. When multiple databases open,the relation will be modified in all database schemas where it exists. The oldrelname must incorporate a pathspecification as necessary to disambiguate in the case of a relation being modified that is non-unique within a singledatabase. Certain restrictions apply when modifying relations. The type cannot be modified if the relation has childrenor has a type of futures_continuous or futures_contract. In addition, the type cannot be changed to eitherone of these two types. The parent of a relation may only be changed if the old and new parents are both of typecategory. Of course, the system will not allow changing the parent such that a cycle is created. If the name of arelation of type futures is changed, the name change will propagate down to all children futures_contract andfutures_continuous relations. Thus, the following commandrelation_modify SP { name = SP500;}

will result not only in the renaming of SP to SP500, but also in SP_1994M being renamed to SP500_1994M andSP_3mo being renamed to SP500_3mo. Note that these are the only instances where a modification to a parentrelation is propagated down to the children. This propagation is necessary to retain consistency in the database.

A relation can be deleted using the command relation_delete. Its synopsis is given by the following:relation_delete relname

If the relation name is not unique, then the user must enter the full path of the relation.

For example:relation_delete TopRelation:Equities:g:gc:GCX

Any data stored in relname will also be deleted by this command. Moreover, if relname contains any other relations,they will be deleted as well. As with the modify command, the specification of a relation that is non-unique withina single database is accommodated by allowing a pathname in the specification for relname. The relation will bedeleted from all currently open databases where it exists.

Adding, Modifying, and Deleting ColumnsA column can be added using the column_add command. Its synopsis is as follows:column_add{ name = colname; description = "description"; parent = parent column; type = column type; fields = field_name1 [ " field_description" ], field_name2 [ " field_description2" ], ...;

34


}

The colname is the name of the column to be added, and description is an arbitrary descriptive text used only fordocumentation purposes. The parent_column specifies where in the column hierarchy the new column is tobe added. When multiple databases are open, the column will be added to all database schemas that have thecategory specified by parent_column. As with relations, if the category name is ambiguous a “path” to the correctcategory can be specified, using colons as separators. When a non-unique column occurs within the same database,it must have a distinct path specification associated with it that can be used to disambiguate when referencing it. ForCommodity DataServer, the user may construct the entire column hierarchy, including the specification of the top-mostcolumn categories. By default, the top-most column categories will be Price, VolumeCat, and FundamentalCat.The keyword TopColumn serves to distinguish the top-most columns if needed. The column type specifies the type ofthe column. This must be either category, normal or composite.

A column of type category is used to group other columns or categories, but may not be used to store data.Conversely, a normal column may contain data, but not other columns or categories. A column of type composite is amulti-field column in that it is used to store data for which there is more than one field. For example, it can be used tostore tick-by-tick data consisting of an arbitrary number of user-defined fields for each tick/transaction. Conceptually,composite columns can be thought of as homogeneous relational tables. If the column is of type composite, the fieldsargument must be specified. This argument provides the field names and, optionally, field descriptions for each field inthe composite column. Field names need not be unique across columns in Commodity DataServer but, if they are notunique within the same database, disambiguating paths will need to be used when referring to them. In CommodityDataServer, columns of type composite may be used for storing tick-by-tick, intraday or daily data. Columns of typecomposite, like those of type normal, contain data, but not other columns or categories. Normal is the default typeused for columns. Only the colname and parent column arguments are required in the column_add command.

The following command adds the column Yield to the Commodity DataServer database:column_add { name = Yield; description = "Bond Yields"; parent = Price; type = normal;}

The following command adds the composite column Quotes to the Commodity DataServer database:column_add { name = Quotes; parent = Price; type = composite; fields = Bid "Bidding Price", Ask "Asking Price";}

The command above would be used if the corresponding tick-by-tick data had both the bid and ask prices associatedwith each transaction. If the tick-by-tick data was structured such that there was a separate transaction for each bidand each ask, there would be two normal columns:column_add { name = Bid; description = "Bidding Price"; parent = Price; type = normal;


35

}column_add { name = Ask; description = "Asking Price"; parent = Price; type = normal;}

A column can be modified using the column_modify command. The synopsis for this command is:column_modify oldcolname { name = newcolname; description = "description"; parent = parent_column; type = column_type;}

The oldcolname is the original name of the column. When multiple databases are open, the column will be modifiedin all database schemas where it exists. The oldcolname must incorporate a path specification as necessary todisambiguate in the case of a column being modified that is non-unique within a single database. The remainingarguments are all optional. They contain the desired modifications and are identical to the arguments of column_add.The type of a column can only be changed from category to normal if the column does not have any children. The typeof a composite column cannot be changed, nor can a category or normal column be changed to a composite column.If the parent of a column is changed, the new parent must be of type category. The fields argument for compositecolumns cannot be modified and, therefore, does not appear in the synopsis above.

A column can also be deleted, using the column_delete command. Its synopsis is:column_delete colname

This command will also delete all references to colname in all relations, as well as all data associated withcolname. As with the modify command, the specification of a column that is non-unique within a single database isaccommodated by allowing a pathname in the specification for colname. The column will be deleted from all currentlyopen databases where it exists.

Adding, Modifying, and Deleting Relation ColumnsA relation column creates a binding between a column and a relation; for example, if IBM is a relation and Close isa column, then Close of IBM is a Commodity DataServer relation column. Only relation columns may store data. Arelation column can be added to the Commodity DataServer by using the relation_column_add command. It hasthe following synopsis:relation_column_add { relation = relname; column = colname; type = relation_column_type; object_type = object_type; constant = value; aggregation = aggregation_rule; delta = point_size; multiplicity_daily = multiplicity;

36


multiplicity_intraday = multiplicity; multiplicity_tick = multiplicity; multiplicity_millisecond = multiplicity;}

The relname and colname specify the relation and the column which is being added to it, respectively. If thespecified relname and colname exists in more than one currently open database, then the relation column willbe added to the occurrence in each applicable database. In Commodity DataServer, there is no uniqueness namingrequirement for any relations or columns (even those that have data associated with them). Non-unique relations orcolumns that reside in the same database, however, must have distinct paths. To accommodate the specificationof non-unique relations and columns in the relation_column_add command, pathnames are allowed in thespecifications for the relname and colname arguments.

Both relname and colname may be categories and, thus, have children. In the case that colname is of typecategory, the column hierarchy will be traversed to find all normal and composite columns that are descendantsof colname and these will be the columns added to relname. If relname contains children relations, eitherdue to being of type category or future, the appropriate columns will be added not only to relname, but also toall of its children relations. When this propagation takes place, the attributes of the relation column (i.e., type,object_type, . . . ) for the children relations are left unspecified such that they can later be inherited from the parent.

When data is added to a relation column, any attributes not explicitly specified, are inherited from the parent andbound to those values at that time. This provides a flexible mechanism whereby columns can be added to a relationor group of relations obviating the need to explicitly specify the relation column with its associated attributes for everyrelation. Note that when a relation is added to the database, all of the columns of the parent relation are inherited and,just as explained above, the attributes of the corresponding relation columns are left unspecified such that they canlater be inherited from the parent. The relname and colname arguments are the only arguments required for therelation_column_add command; the remaining arguments are optional and have default values associated withthem.

The relation_column_type is used to specify the type of the new relation column. This must be one of thefollowing:

● base● sparse

Base is the type used for regular time series data. Sparse is the type used for data with values that appearsporadically. Between the specified values of sparse data, the value for the relation column is assumed to stay thesame. That is, Commodity DataServer will always return the previously known value for the relation column. A goodexample of a sparse relation column is the prime rate, since its value remains constant until the Federal Reservechanges its policy. If the type is not specified, base will be assumed. A relation column of type constant is used tostore a user-defined field for the particular relation (e.g., a unique id for each relation). When the type is constant, thevalue is used to specify the constant value of the relation column.

Queries executed through the query language will automatically forward fill for relation-columns that aredefined as sparse. However, the get records API calls (i.e., XMIMGetRecords, getRecords, GetRecords)will not automatically forward fill those relation-columns. It will only return the actual data points thathave been loaded. Because of this, when sparse data is merged with trading calendar data, there will beNaNs.


37

The object_type specifies the type of the data stored in the database. The possible types are the following:

● float● double● integer

The float and double types represent single and double precision floating point, respectively. If the object_type isnot specified, float is assumed.

The aggregation_rule specifies how data of one granularity is to be converted to other granularities; for example,how daily data is combined to form weekly data. The possible aggregation rules are as follows:

● open● high● low● close● sum● average

The default aggregation_rule is close, which specifies that the last value for the period should be used. Similarly,open specifies to use the first value for the period, high the highest value, and low the lowest value. For example,close would be used for the Close column of a relation such that the weekly close would be the last value for the week(i.e., the close for the last day of the week). When sum or average are specified, the sum or average of the values forthe period are used. For example, sum would be used for the Volume column of a relation such that the weekly volumewould be the sum of the daily volumes for that week. Commodity DataServer will always aggregate on whatever datais available for the specified period. For example, if the aggregation period is weekly and on a given week, Friday is aholiday, the aggregation will take place over the data for Monday through Thursday so that the weekly close will beThursday's close.

The point_size specifies the increment size for the relation column data. The default point_size used is none.

The multiplicity specifies individually whether the daily, intraday, tick and millisecond data for the relation columnwill be singlevalued or multi-valued. A multi-valued relation column is one in which multiple values can be stored fora single time slot as opposed to a single-valued relation column where there is only a single value for each time slot.Tick-by-tick data is typically multi-valued. The possible specifications for multiplicity are:

● single● multi● multiple

where either multi or multiple can be used to indicate multi-valued. By default, daily, intraday and millisecond timeseries will be single-valued and tick-by-tick will be multi-valued.

The command in the following example adds the relation column BondRating of IBM.

relation_column_add { relation = IBM; column = BondRating;

38


type = sparse; object_type = float; aggregation = close; delta = none; multiplicity_daily = single; multiplicity_tick = multi;}

The object_type, aggregation and delta need not have been specified since the values given are the same asthe default values. Likewise for the multiplicity specifications. The command in the following example adds the columnYield to the futures relation US and to any of its children continuous or individual futures contracts.

relation_column_add { relation = US; column = Yield; type = base; delta = 32;}

The type need not have been specified since base is the default value. If Yield was a column of type category, theeffect of the above command would be to add all the leaf descendants of Yield to US and any children relations. Thus,if YieldToCall, YieldToPut, and YieldToMaturity were columns of type normal with parent being Yield,the relation columns added for US would be YieldToCall of US, YieldToPut of US, and YieldToMaturity ofUS.

If the column to be added to the relation is of type composite, the object_type, constant, aggregation anddelta attributes will be specified for each field of the composite column. Thus, these attributes will appear as lists inthe relation_column_add command whenever the colname specified is of type composite:

relation_column_add { relation = relname; column = colname; type = relation_column_type; object_type = object_type1, object_type2, ...; constant = value1, value2, ...; aggregation = aggregation_rule1, aggregation_rule2, ...; delta = point_size1, point_size2, ...; multiplicity_daily = multiplicity; multiplicity_intraday = multiplicity; multiplicity_tick = multiplicity; multiplicity_millisecond = multiplicity;}

Again, the relname and colname arguments are the only required arguments; all remaining arguments are optionaland have default values associated with them. Any of the lists that are specified must have a length that is the sameas the number of fields specified in the column_add command used for colname.

The command in the following example adds the relation column Quotes of IBM .199307.PUT.60.

relation_column_add { relation = IBM.199307.PUT.60; column = Quotes; type = base;


39

multiplicity_tick = multi; object_type = float, float; aggregation = close, close; delta = none, none;}

Since the column Quotes is a composite column with two fields (see example in “Adding, Modifying, and DeletingColumns”), object_type, aggregation and delta are given as lists of length two. The first element of eachlist will be used for the Bid field of Quotes and the second element of each list will be used for the Ask field. Thus,the type of data for both Bid and Ask is float. Again, the multiplicity_tick argument need not be specifiedsince it is given as multi-valued and that is the default. However, if this relation column was to be used for daily data,multiplicity_daily would need to be specified if multi-valued was desired as single-valued is the default fordaily data. Options data is inherently multi-value by nature so it would make sense to specify it as such even for dailydata.

The Commodity DataServer database is initialized such that there are certain relation columns already defined foreach of the default top-level relation categories. Of course, this is all subject to modification by the user. To determinewhat relation columns exist in the initial state, simply open a newly created database and use the print_schemacommand. In Commodity DataServer, EconomicIndicators and MonetaryIndicators can have explicitcolumns associated with them just like any other relation. There is, however, a default sparse column named_Implicit_ that is assigned to these relations initially by Commodity DataServer. In previous versions of CommodityDataServer, this default column was the only column allowed for these relations.

A relation column may be modified using the relation_column_modify command.

Its synopsis is as follows:relation_column_modify relname colname { type = relation_column_type; object_type = object_type; constant = value; aggregation = aggregation_rule; delta = point_size; multiplicity_daily = multiplicity; multiplicity_intraday = multiplicity; multiplicity_tick = multiplicity; multiplicity_millisecond = multiplicity;}

As with the relation_column_add command, the specification of non-unique relations and columns areaccommodated by allowing pathnames in the specifications for relname and colname. Note that it is not possibleto modify the relation or column of a relation column. While relname may correspond to a relation with children,colname must be of type normal or composite. Furthermore, modifications will not be explicitly propagated to anychildren relations3. Of course, if a relation column is inherited from a parent and the attributes are not yet bound, and,thus, are to be inherited from the parent, the modifications will effectively be propagated. Likewise, any relationsadded to the database that inherit a relation column from their parent relation after the relation column has beenmodified, will inherit the new specifications. If the specified relation column exists in more than one currently opendatabase, the modification will take place for each applicable database.

3 Propagating modifications of a relation column could be extremely dangerous in the presence of existing data since the modifications sometimes requirethat the actual data values be changed.

40


The relation_column_modify command to be used for composite columns has the following synopsis:

relation_column_modify relname colname { type = relation_column_type; object_type = object_type1, object_type2, ...; constant = value1, value2, ...; aggregation = aggregation_rule1, aggregation_rule2, ...; delta = point_size1, point_size2, ...; multiplicity_daily = multiplicity; multiplicity_intraday = multiplicity; multiplicity_tick = multiplicity; multiplicity_millisecond = multiplicity;}

As with the relation_column_add command, if colname is a composite column, the object_type,constant, aggregation and delta attributes must be specified for each field of the composite column and,thus, appear as lists. The length of these lists cannot be modified; they must stay the same length as in the originalrelation_column_add command.

Deleting a relation column is accomplished using the relation_column_delete command. Its synopsis is asfollows:

relation_column_delete relname colname

This command will also delete any data associated with the relation column. As with the modify and add commands,the specification of non-unique relations and columns are accommodated by allowing pathnames in the specificationsfor relname and colname. As with modifying relation columns, the colname specified must be a column of typenormal or composite. Only the specified relation column is deleted. Thus, if relname is a relation with children, thechildren will be unaffected (i.e., their corresponding relation columns will not be deleted). The relation column will bedeleted from all currently open databases where it exists.

Reading and Writing ASCII Data FilesData can be loaded into the database from an ASCII data file using the facts_read command. Its synopsis is asfollows:

facts_read { file = filename; relation = relname; [ fixed | variable ]; [ merge | replace ]; [ tick_intraday | tick_by_tick | tick_convert | tick_millisecond ]; default_format = <format>; column = col1<format1>, col2<format2>, ...;}

The file to be read is specified by filename. This should be an ASCII file, with one record per line. The filenamespecification may include the full Unix pathname or may consist of a relative pathname. The relation to be updated isspecified by relname. Relname must not be of type category since categories do not have data associated with

41

them. If no relation is specified, then each record in the file must contain the name of the relation being modified. Thisallows a single file to update multiple relations, which is useful for updating a small number of values (e.g., a singleday) for many relations.

The modifiers fixed and variable specify whether the records in the file have fixed-or variable-sized fields. The defaultis to assume variable-sized (i.e., free-format) fields. The modifiers merge and replace specify whether the data read inwill serve as an update for or a replacement to the existing Commodity DataServer data for the given relation columns.The default is to merge the data with any existing Commodity DataServer data. While it is possible to use thefacts_read command to update more frequently than daily, the most efficient way to do so is to use the CommodityC/C++ API (see Chapter 5, “C/C++ API”). The keywords tick_intraday, tick_by_tick, tick_convert andtick_millisecond, are used to specify that the file contains tick data. If tick_millisecond is specified, thenthe file is assumed to contain millisecond data. If tick_by_tick or tick_convert is specified, then the file isassumed to contain real tick (second) data. When tick_convert is used, BMIM will automatically aggregate thedata into intraday (minute) form. If tick_intraday is specified, the file is assumed to be in intraday form already.Note that one of these keywords must be specified if tick data is being read since daily data will be read by default.

The column list specifies which fields are in each record of the file. An element in the column list consists of a columnfield name followed by an optional format specification. The format specification is required when fixed format is used.The column field names col1, col2, ..., can be the name of an Commodity DataServer column, in which caseBMIM expects the corresponding field to update the given relation column. Alternatively, each column field name maybe one of the following keywords:

● ignore● relation● date● time

The special column field name ignore is used to specify a field which is to be completely ignored by BMIM. This isuseful mainly in fixed-format ASCII files. The special column field name relation is used to specify a field containingthe name of the relation to which the current record should be added. This field must be present somewhere in therecord if relname is not specified. The date and time fields are used to specify the date and time for the record.All records must have a date field, and they should also have a time field if the file is being read in tick mode. Thecolumn list may contain any Commodity DataServer columns appropriate for the given relation, including compositecolumn fields from any composite column of the given relation. If any of the fields of a composite column are specifiedin the column list then any unspecified fields will be assigned default values. If a composite column, itself, is givenin the column list then information for all fields of the composite column will be read. In the presence of CommodityDataServer composite columns, fixed format cannot be used; files must be read using variable format.

When a file is read using variable format, fields may be separated by either spaces or commas, and BMIM willsimply read in the fields as given. When the file is read using fixed format, the format specifiers <format1>,<format2>, . . . , must be used to specify how the data is stored in the file4. The possible formats for CommodityDataServer columns are Fw.d or Iw , where w is the number of characters taken up by the field (i.e., the width), and dspecifies the number of digits assumed to be to the right of the decimal point. The format Iw is an alias for Fw.0 and isused for reading in integer numbers. The Fw.d format is used for reading in floating point numbers5. If the number readin already contains a decimal point, the d specifier is ignored. In the special case that d is negative, it is interpreted as

4 These format specifiers will simply be ignored when using variable format except as necessary to differentiate between date formats.5 The format Ew.d can be used as an alias for Fw.d.

42


the point size of the data. For reading facts, the available point sizes are 8, 16, 32, and 64. For writing facts, any pointsize may be specified. Some sample data values with the corresponding point size adjusted value are given below.

Value 8ths 16ths 32nds 64ths62.250 62' 2 62' 04 62' 08 62' 1662.375 62' 3 62' 06 62' 12 62' 2462.500 62' 4 62' 08 62' 16 62' 3262.875 62' 7 62' 14 62' 28 62' 56

The point adjusted values can be read in the format shown above or without the ' character. For example, the value316 is interpreted as 31.6 if the format is F3.1. If the format is F3. -32, it is interpreted as 3.5 instead. Since the pointsize used is 32nds, the last 2 digits are interpreted as or .5. Similarly, a format of F3. -8 is interpreted as 31.75.

Here, the point size is 8ths and so the last digit is interpreted as or .75. Using the point size in a format specifier issimilar to setting the delta argument of relation_column_add except that here it only affects the reading or writing ofdata. In general, format specifiers will always override a delta specification.

Typically, most of the relation columns in a file will have the same format specifier (e.g., all F8.3). To facilitate this,BMIM uses the default_format entry. If this is specified, then it is used for any relation column fields that do not havean explicit format.

The format specifier used for ignore fields is wX, where w is the number of characters to be ignored. Note that aformat specifier with no specified column field name is assumed to be ignore. Relation fields use the format specifierAw, where w is the number of characters in the relation name.

The formats for time fields are specified as follows:

Time Format Examplehhmm 2315

hh:mm 09:05

hhmmam 0905am

hhmmAM 0905AM

hh:mmam 01:15pm

hh:mmAM 01:15PM

hhmmss 231501

hh:mm:ss 09:05:15

hhmmssam 090515am

hhmmssAM 090515AM

hh:mm:ssam 01:15:01pm

hh:mm:ssAM 01:15:01PM

hhmmssddd 231501333

hh:mm:ss.ddd 09:05:15.333

hhmmssdddam 090515333am

hhmmssdddAM 090515333AM

hh:mm:ss.dddam 01:15:01.333pm

hh:mm:ss.dddAM 01:15:01.333PM


43

The formats for date fields are specified as follows:

Date Format ExampleYymmdd 940615

Yyyymmdd 19940615

dd-MMM-yy 15-Jun-94

dd-MMM-yyyy 15-Jun-1994

dd-MMMMM-yy 15-June-94

dd-MMMMM-yyyy 15-June-1994

mm/dd/yy 06/15/94

mm/dd/yyyy 06/15/1994

dd/mm/yy 15/06/94

dd/mm/yyyy 15/06/1994

mm-dd-yy 06-15-94

mm-dd-yyyy 06-15-1994

dd-mm-yy 15-06-94

dd-mm-yyyy 15-06-1994

MMM dd, yy Jun 15, 94

MMM dd, yyyy Jun 15, 1994

MMM. dd, yy Jun. 15, 94

MMM. dd, yyyy Jun. 15, 1994

MMMMM dd, yy June 15, 94

MMMMM dd, yyyy June 15, 1994

In fixed format mode, when one of the date formats using MMMMM is applied, it is necessary to use spaces to padthe actual month names in the file being read out to 9 characters (which is the length of the longest month name) inorder to ensure fixed-length fields. In variable format mode, Commodity DataServer automatically determines whichof the above data formats is being used. Since there is no way for Commodity DataServer to distinguish between themm/dd/yy[yy] format and the dd/mm/yy[yy] format, mm/dd/yy[yy] will always be assumed. If the dd/mm/yy[yy] formatis desired, the format specification must be used with the date column field name as is done in fixed format mode.Likewise, for the formats mm-dd-yy[yy] and dd-mm-yy[yy] where mm-dd-yy[yy] will be assumed. Please note that thedefault century that is used for a 2-digit year specification is 1900. The 4-digit year specification is the recommendedusage. When using variable format, it is not necessary to specify the date, time, and relation fields. If not specified, thedate field is automatically inserted at the beginning of the field specification list. Similarly, the time field (in tick mode)and the relation field (if relname is not specified) are inserted after the date.

These formats are similar to the date formats given previously except that only the month and year are specified, notthe day. The regular date formats may also be used to specify the expiration day.

44


The contents of the example file ibm.prn are given below:

19940601 63.000 63.625 62.500 63.500 168960019940602 63.375 63.500 61.875 62.000 247590019940603 61.750 61.875 60.750 61.000 238140019940606 61.250 63.500 61.125 62.625 234420019940607 62.750 63.250 62.625 62.625 139910019940608 62.875 63.000 61.500 61.625 165150019940609 61.500 62.125 61.500 62.000 108770019940610 62.250 62.875 62.250 62.625 132490019940613 62.375 64.125 62.250 63.750 216450019940614 63.875 65.000 63.875 65.000 2778600

The following command will read data from this file as free-format records.

facts_read { file = ibm.prn; relation = IBM; column = Open, High, Low, Close, Volume;}

Since the date field is not specified in the command above, it is assumed to be the first field of each record. The datawill be merged with any existing data since that is the default mode.

The same file can be read as fixed-format records using the following command:

facts_read { file = ibm.prn; relation = IBM; default_format = <F7.3> column = Date<yyyymmdd>, Open, High, Low, Close, Volume<I8>; fixed;}

Note that the format specifications use field widths that are actually one more than used for the corresponding fieldin the file. This is necessary in order to account for the spaces between fields. Alternatively, the 1X format could havebeen used to account for the spaces as shown below:

facts_read { file = ibm.prn; relation = IBM; default_format = <F6.3> column = Date<yyyymmdd>, <1X>, Open, <1X>, High, <1X>, Low, <1X>, Low, <1X>, Close, <1X>, Volume<I7>; fixed;}

45

Similarly, the following file can be read in to update both IBM and DELL:

19940610 IBM 62.250 62.875 62.250 62.625 132490019940612 IBM 62.250 62.875 62.250 62.625 132490019940613 IBM 62.375 64.125 62.250 63.750 216450019940614 IBM 63.875 65.000 63.875 65.000 277860019940615 IBM 64.500 64.750 63.500 63.750 178930019940616 IBM 63.250 63.500 62.875 62.875 143260019940610 DELL 20.250 20.375 19.625 19.875 92700019940613 DELL 19.625 20.500 19.625 20.250 57920019940614 DELL 20.125 20.250 19.625 19.875 97090019940615 DELL 19.750 19.750 19.375 19.625 92410019940616 DELL 19.500 20.125 19.375 20.125 839900

facts_read { file = ibm_dell.prn; default_format = <F7.3> column = Date<yyyymmdd>, Relation<A4>, Open, High, Low, Close, Volume<I8>; fixed;}

Note that the second line of the input file has data for 6/12/1994, which falls on a Sunday. Commodity DataServer willignore this line of data since it falls on a non-trading day. In a similar manner, when tick data is being read, if the timegiven falls outside of the starting and ending trading times for the corresponding relation, that tick is ignored.

The following command will read data for an economic indicator.

facts_read { file = cpi.prn; relation = CPI;}

Since economic and monetary indicators do not necessarily have explicit columns associated with them, the columnlist does not need to be specified if free-format is used. BMIM will automatically add the implicit default column afterthe date field and any other applicable default fields. If columns other than the default implicit column are desired, thenall columns must be specified. If fixed-format is used for reading economic or monetary indicators, the default columnname (_Implicit_) must be explicitly specified, if desired.

As a final example, the following file fragment contains real tick data for a relation of type options.

930701,083216,10.13,10.63930701,083244,10.63,11.13930701,083248,10.73,11.13930701,083253,11.13,11.63

The relation name is IBM.199307.PUT.60. It has a composite column, Quotes that has fields named Bid and Ask.The following facts_read command will read data from this file as free-format records.

facts_read { relation = IBM.199307.PUT.60; column = Date, Time, Bid, Ask; file = "IBM-july93-put-60.prn";}

46


Alternatively, the following facts_read command can be used to read the same real tick options data. It utilizes thedefault insertion capabilities for the Date and Time columns. It also uses the composite column name rather than thefield names. BMIM will expand the composite column name such that all fields of the column are read.facts_read { relation = IBM.199307.PUT.60; column = Quotes; file = "IBM-july93-put-60.prn";}

Comments and blank lines are allowed in both fixed and variable format modes. Comments are denoted with the #character. All characters following the # character to the end of the line are ignored. While the comment can beginanywhere in a line when variable format is used, it must be the first non-space character of a line when fixed format isused.

ASCII files can also be created by BMIM. This is accomplished using the facts_write command. Its synopsis is asfollows:facts_write { file = filename; relation = rel1, rel2, ...j default_format = <format>; column = col1<format1>, col2<format2>, ...; from_date = from_date; to_date = to_date; from_time = from_time; to_time = to_time; units = units ending at [ end_date | end_time ];}

The name of the file to be written by BMIM is specified in filename. Data is written for the relations rel1, rel2, . . .The relations can be of any type except category.

The columns to be written out are specified in the column list. All records written by BMIM are in a fixed-format,so the format specifiers are used to determine the form of the output. If the format specifiers (i.e., <format1>,<format2>, . . . ) are not given, Commodity DataServer will first default to the format specifier in the .xmimrc fileand, if the appropriate specifier is not given there, will then default to F10.2 for column fields, A10 for the relation field,mm/dd/yyyy for the date format, and hh:mmam for the time format.

The from_date, to_date, from_time and to_time are used to narrow the range of data that is output tofilename. The dates are specified in the format mm/dd/yyyy. The times are specified in the format hh:mm:ssAM[am]/PM[pm]. Note that the finest granularity allowed for the time range is 1 second as milliseconds are notallowed in the time range specification. For each day in the date range given, data will be output for the time rangegiven. If only the dates are specified, then data will be aggregated over the entire trading period of each date in thedate range. Likewise, if only the times are specified, then data will be aggregated for the time range specified forall days available in the database. If no date or times are specified, BMIM prints all available data. Of course, thefrom_time and to_time arguments only make sense in the context of intraday or real tick or millisecond data. Ifthe range specified is incorrect (e.g., to_date is before from_date), then no data will be printed.

The units specification is used to determine what type of data is desired (e.g., 1 week, 10 minute, . . . ). The timeperiod over which to aggregate data is specified by units. For daily data, units is specified as an integer followed

47

by either days, weeks, months, quarters or years. For millisecond data, units is specified as an integer followed bymilliseconds. For real tick data, units is specified as an integer followed by seconds. For intraday tick data, unitsis specified as an integer followed by either minutes or hours6. If minutes or hours is specified but no intraday dataexists, the system will aggregate real tick data (if available) into the desired frequency; if no real tick data exists,millisecond data (if available) will be aggregated into the desired frequency. Since the usage of intraday data willbe preferred for aggregating into minutes or hours, if both real tick and intraday data exist and the use of real tickis desired, the units should be specified in terms of seconds (e.g., 120 seconds in lieu of 2 minutes). Likewise, foraggregating into seconds, if both real tick and millisecond data exist and the use of millisecond is desired, the unitsmust be specified in terms of milliseconds (e.g., 2000 milliseconds in lieu of 2 seconds). Optionally, units maybe followed by the keywords ending at and then a date (for daily data) or a time (for tick data). This date or timeprovides Commodity DataServer with a point of reference for the desired aggregation period. For example, if the unitsspecification is 1 hour ending at 8:30, aggregation will be hourly on the half hour such that if the trading range startsat 7:00am, the times shown will be: 7:30am, 8:30am, 9:30am, . . . . These times correspond to the hourly periodsof 6:31am to 7:30am, 7:31am to 8:30am, 8:31am to 9:30am, . . . . Notice that the end_time corresponds to theend of the aggregation period and that times emanate from it both backwards and forwards. Similarly, if the unitsspecification were 1 week ending at 6/1/94, since 6/1/94 is a Wednesday, the weekly aggregation will take placefrom Thursday to Wednesday. If the end_date or end_time is not specified, Commodity DataServer defaults willbe used (e.g., on the hour for hourly, Monday to Friday for weekly, last day of the year for yearly). The default unitsspecification is 1 day.

The following command writes this year's IBM and DELL data to a new file using the Commodity DataServer defaultformats for date and relation, while using a point size of 8ths and field width of 7 to write the data for Open, High, Lowand Close. Volume data is written in an integer format with a field width of 8. The generated output (given that thedata is read in from the previous sample file, ibm_dec.prn) is also provided.

facts_write { file = ibm_dell.out; relation = IBM, DELL; default_format = <F7.-8>; column = Open, High, Low, Close, Volume<I8>;}

06/10/1994, IBM, 62'2, 62'7, 62'2, 62'5, 132490006/13/1994, IBM, 62'3, 64'1, 62'2, 63'6, 216450006/14/1994, IBM, 63'7, 65'0, 63'7, 65'0, 277860006/15/1994, IBM, 64'4, 64'6, 63'4, 63'6, 178930006/16/1994, IBM, 63'2, 63'4, 62'7, 62'7, 143260006/10/1994, DELL, 20'2, 20'3, 19'5, 19'7, 92700006/13/1994, DELL, 19'5, 20'4, 19'5, 20'2, 57920006/14/1994, DELL, 20'1, 20'2, 19'5, 19'7, 97090006/15/1994, DELL, 19'6, 19'6, 19'3, 19'5, 92410006/16/1994, DELL, 19'4, 20'1, 19'3, 20'1, 839900

The following command writes out the tick data for DJIA on 1/4/1993, aggregating in 1 minute units. A 10 minutesegment of the output generated is also provided.

facts_write { file = djia.out;

6 Aggregation for tick data does not take place across day boundaries, so if the milliseconds, seconds, minutes or hours specified is beyond the scope of asingle day, the aggregation will stop at the end of the day.

48


relation = DJIA; default_format = <F7.2>; column = Date<dd-MMM-yyyy>, Time<hh:mm>, Open, High, Low, Close, Volume<I3>; units = 1 minute; from_date = 1/4/1993; to_date = 1/4/1993;}

04-Jan-1993, 08:34, 3301.10, 3303.80, 3301.10, 3303.80, 704-Jan-1993, 08:35, 3305.20, 3309.20, 3305.20, 3309.00, 1104-Jan-1993, 08:36, 3308.70, 3309.20, 3308.70, 3309.20, 1004-Jan-1993, 08:37, 3310.00, 3314.10, 3310.00, 3314.10, 904-Jan-1993, 08:38, 3314.10, 3314.90, 3314.10, 3314.10, 1104-Jan-1993, 08:39, 3314.40, 3314.40, 3313.80, 3314.40, 1004-Jan-1993, 08:40, 3314.90, 3314.90, 3314.60, 3314.90, 1004-Jan-1993, 08:41, 3315.20, 3315.70, 3315.20, 3315.20, 1004-Jan-1993, 08:42, 3315.40, 3316.00, 3315.40, 3315.70, 1004-Jan-1993, 08:43, 3315.70, 3316.20, 3315.40, 3316.20, 9 26

The following command writes out the tick data for DJIA on 1/4/1993, aggregating in hour increments ending 15minutes after the hour. The output that is generated is also given below.

facts_write { file = djia.out; relation = DJIA; default_format = <F7.2>; column = Date<mm-dd-yyyy>, Time<hh:mmam>, Open, High, Low, Close, Volume<I3>; units = 1 hour ending at 8:15; from_date = 1/4/1993; to_date = 1/4/1993;}

01-04-1993, 09:15am, 3301.10, 3318.90, 3301.10, 3304.40, 41501-04-1993, 10:15am, 3304.10, 3310.00, 3299.20, 3309.50, 59701-04-1993, 11:15am, 3309.50, 3310.30, 3303.00, 3304.60, 59601-04-1993, 12:15pm, 3304.90, 3310.00, 3304.10, 3304.60, 59801-04-1993, 01:15pm, 3304.90, 3309.20, 3302.70, 3307.90, 59701-04-1993, 02:15pm, 3307.60, 3310.60, 3299.00, 3304.60, 59701-04-1993, 03:15pm, 3304.90, 3310.80, 3301.10, 3309.20, 543

When multiple databases are open, the facts_read command will load data to the first applicable database; thatis, if daily data is added to Close of IBM then it is added to the first database that has the relation IBM with a Closecolumn. Likewise, in the presence of multiple databases, facts_write will retrieve data from the first database thathas data for the specified relation column.\

Deleting FactsSometimes it is desirable to delete all the data from a relation column. The delete_facts command can be used forthis purpose. The synopses for the two forms of this command are listed below.

The following shows the first form of the command:delete_facts relname [ colname ]


49

relname must be a normal relation. The colname argument is optional. If colname is not given, the effectwill be to delete data for all existing columns of relname. Note that this command is different from therelation_column_delete command in that only the data is deleted, the relation column will still exist in thedatabase.

When multiple databases are open, delete_facts will delete the specified data from all applicable databases.

The following shows the second form of this command:

delete_facts { relation = relname; column = col1, col2, ...; [ daily | tick_intraday | tick_by_tick ]; [from_date = from_date]; [to_date = to_date]; [from_time = from_time]; [to_time = to_time];}

The relation name and at least one column must be specified, the rest is optional. The data type will default to alltypes of data, from_date to the first date for which there is data and to_date to the last date for which thereis data, from_time to the start of the trading day and to_time to the end of the trading day. If both the dateand time ranges are specified, for each day in the date range given, data will be deleted for the time range given. Ofcourse, the from_time and to_time arguments only make sense in the context of intraday or real tick data. Notethat this command is different from the relation_column_delete command in that only the data is deleted, therelation column will still exist in the database.

When multiple databases are open, delete_facts will delete the specified data from all applicable databases.

Assigning Holiday SchedulesThe set_holiday_schedule command is used to associate a holiday schedule with a relation or an exchange.Utilizing this command, foreign holiday schedules can be introduced into Commodity DataServer. The synopsis forset_holiday_schedule is:

set_holiday_schedule [ exchange | relname ] filename

The first argument is optional and indicates which exchange or relname that the holiday schedule defined infilename is to be assigned to. The filename specification may include the full Unix pathname or may consist of arelative pathname. Regardless, Commodity DataServer must be able to establish the location of the holiday schedulefile such that it resides in the same Unix subtree as the schema file (i.e., xmim.sch). If the system cannot resolvethe filename as such, an error message will be issued. Note that the system will not attempt to resolve soft links. Iffilename is the only argument specified, then the default holiday schedule is set. If the relname to be specified isalso an existing exchange, it must be disambiguated by giving the “path” for the relation, using colons as separatorsas outlined in “Database Server Architecture”. The holiday schedule used for a particular relation will be determinedfirst by checking if a holiday schedule has been assigned directly to the relation, and, failing that, checking the relationhierarchy for the closest ancestor that has been assigned a holiday schedule. If no holiday schedule is found in that

50


manner, then the system checks for a holiday schedule having been assigned to the exchange for that relation. Notethat the exchange is an attribute that can be inherited from a parent relation so if no exchange exists for the givenrelation, the relation ancestry is searched to determine the correct exchange. The default holiday schedule will be usedif none other can be found.

Each holiday file must consist of a series of dates, given one per line. The date format must be one of the validCommodity DataServer date formats. A date range may also be specified by providing two dates with the hyphen(“-”) character in between, again on a single line. The date range specification may be useful for foreign holidays thatconsist of multiple days. Each line may optionally have a character string that will be taken as a comment. A fragmentof a file containing US holidays is shown below:

650531,MEMORIAL_DAY650705,INDEPENDENCE_DAY651125,THANKSGIVING660530,MEMORIAL_DAY660704,INDEPENDENCE_DAY661124,THANKSGIVING

Adding Relation AliasesThe relation_alias_add command is used to add an alias for a given relation. The synopsis is as follows:

relation_alias_add { name = alias_name; parent = alias_parent; target = orig_relation;}

The name simply specifies the alias to be used for the relation. The parent supplies where the alias will reside and thetarget what the alias points to. The parent and target need not be paths unless the names alone are ambiguous andthe path is necessary to disambiguate. If there is ambiguity and path names are not specified, an error message willbe returned.

In the following example, TopRelation:Equities:IBM exists and an alias is added providing an alternate path to accessthe relation IBM:

relation_alias_add { name = IBM; parent = US; target = TopRelation:Equities:IBM;}

The target in the above example, could have been specified as simply IBM (as long as IBM is unique).

In the following example, the path TopRelation:Equities:IBM exists and an alias is added so that IBM can alsobe accessed via the path TopRelation:US:Equities:IBM:

relation_alias_add { name = IBM;


51

parent = US:Equities; target = TopRelation:Equities:IBM;}

Here, paths had to be specified for parent and target to disambiguate.

The following example demonstrates adding an alias for IBM of cusip_1111 (with the same path):

relation_alias_add { name = cusip_1111; parent = Equities; target = IBM;}

Note that if more than one database is open and a relation alias is added for a relation that exists in more than onedatabase with the same path, the alias will be added to each occurrence. For example, if TopRelation:Equities:IBMexists in two different databases and the above relation_alias_add command is given, then cusip_1111 willbe added as an alias for IBM in both databases. To add the alias to only one of the databases, a database_narrowcommand must be used.

Adding FormulasThe formula_add command is used to add a formula for a given relation-column. Formulas are used to create a"computed" time series. A relation-column that has a formula has no actual data for that relation-column. The data forthe "Formula time series" is computed dynamically based on a formula that refers to data in other relation-columns.

The synopsis of the formula_add command follows:

formula_add { relation = relation_name; column = column_list; formula = formula_filename;}

The relation and column name together provide the name of the relation-column(s) that is the computed time series.The relation, columns and relation-columns must all exist before invoking the formula_add command.

In the following example, we have added a new relation-column called avg_bar (without data) to NG. Now, we aregoing to specify a formula to compute avg_bar.

formula_add { relation = NG; column = avg_bar; formula = /tmp/formula.txt;}

Where the contents of /tmp/formula.txt is:

SHOW 1: (Open of NG + High of NG + Low of NG + Close of NG) / 4

52


A single formula can compute more than one time series by having more than one SHOW label. For each time seriesto be computed by a formula query, there would be one column-name in the column_list.

In the second example, we have added two new relation-columns for NG: avg_bar and my_mid. Now, we are goingto specify a formula to compute avg_bar and my_mid.

formula_add { relation = NG; column = avg_bar, my_mid; formula = /tmp/formula2.txt;}

Where the contents of /tmp/formula2.txt is:

SHOW 1: (Open of NG + High of NG + Low of NG + Close of NG) / 4 2: (High of NG + Low of NG ) / 2

Another command for manipulating formulas is the formula_modify command. The formula_modify commandis exactly the same as the formula_add command.

formula_modify { relation = relation_name; column = column_list; formula = formula_filename;}

Adding Correction Audits to the Table FacilityMorningstar Commodity Data uses correction audits to track changes made to historical data points. Clients can usecorrection audits to examine when and what data points in the database were modified over time. It can also beused to pose queries to see how data would have appeared on a particular day. Internally, the corrections audit trail isstored in the Commodity DataServer Table Facility. The Table Facility is a generic storage area used to hold data that isnot time-series in nature.

See Chapter 12, “Correction Audits” for more information on correction audits and the Table Facility.

The corrections_add command is used to add daily and intraday correction audit data to the Table Facility withinthe database. There are two tables that store corrections data: XMIM_CORRECTIONS_INTRADAY stores intraday dataand XMIM_CORRECTIONS stores daily data.

The synopsis is as follows:

corrections_add filename

The argument filename is the name of the file containing the corrections data.


53

The file specifying the correction audit data must be composed of one correction per line. Correction audit data is ofthe following form for intraday data:20030421, 0000, APX.DAHOURLY, Val, 20030422, 1400, 1.25

Where this line indicates that on April 21, 2003 a corrected value for Val of APX.DAHOURLY on April 22, 2003 at 2:00PM was placed in the database and the previous value for that series was 1.25.

If two corrections for the same relation on the same date and time are received, only the last one is kept.

Correction audit data is of the following form for daily data:20030421, APX.DADAILY, Val, 20030422, 1.25

Data may be accessed via XmimVaGetRecords in such a manner as to take into consideration any correctionsmade up through a specified date, but none made after that date. For example, use XmimVaGetRecords withXMIM_CORRECTIONS_DATE to set the data point back to a certain value as appeared on a specific date. Thiscapability is available only through the C/C++ API and the Visual Basic for Applications APIs and is only available fordaily data.

Generating Continuous ContractsContinuous contracts will be automatically created and updated by Commodity DataServer. Whenever possible,Commodity DataServer optimizes the updating of continuous contracts such that most types of continuous contractsare updated incrementally and are not completely re-generated with each update. Sometimes it is desirable to havethe continuous contracts for a given relation completely re-computed. The regen_continuous command can beused for this purpose. Its synopsis is:regen_continuous [ relname ]

The relname argument is optional. If it is not given, continuous contracts will be re-computed for all futures andfutures_continuous relations in the database. If relname is of type category or futures, then continuous contracts willbe re-computed for all futures and futures_continuous relations occurring in that sub-tree of the relation hierarchy. Thecontinuous contracts will be re-computed immediately, wherever the regen_continuous command appears in theBMIM script. By contrast, the automatic computation/recomputation of continuous contracts is performed after theentire BMIM script is processed.

Executing Saved QueriesSaved Commodity DataServer queries can be executed from BMIM using the query_execute and query_graphcommands. For more information, see “XMIMExecution and XMIMExecutionDouble” (Commodity DataServer

54


Visual Basic for Applications API), “Executing Saved Queries” (Commodity DataServer C/C++ API) and“MimData.queryExecute” (Commodity DataServer Java API).

Their synopsis is as follows:

query_execute { query = query_filename; file = output_filename; [ append | overwrite ]; print; [ verbose | veryverbose ];}query_graph { query = query_filename; file = output_filename; [ append_mode | overwrite_mode ]; print; [ verbose | veryverbose ];}

The saved query is specified in query_filename. The output will be written to output_filename, if it is specified.The modifiers append_mode and overwrite_mode are used to determine whether the output is appended tooutput_filename, or whether it should replace output_filename. The default is to replace output_filename.In addition, if the print option is specified, BMIM will send the output to the printer. Note that it is possible to writethe output to a file and print with the same command. The verbose and veryverbose modifiers are used to specifywhether the text of the query executed should be included in the output7. If verbose is specified, the query text isincluded. Moreover, if veryverbose is specified, the query text and options will be included. By default, only theresults of the query execution are output.

For example, the following command executes the query metal_currency.query once for each pair of Metals andCurrencies futures. Each of the metal futures are substituted for GC, while each of the currency futures are substitutedfor DM.

query_execute { query = metal_currency.query; file = metal_currency.out; verbose; set GC = "*Metals"; set DM = "*Currencies";}

Specifying Data Files/PartitioningBy default, the Commodity DataServer stores the daily, intraday tick and real-tick data to files specified by the system.To override this behavior and specify the filename where the Commodity DataServer should store the data for a givenrelation, the database_def_daily_file, database_def_intraday_file and database_def_tick_fileand database_def_millisecond_file commands can be used. The synopses are:

7 The verbose and veryverbose options are available beginning with version 4.x.


55

database_def_daily_file [ relname [ filename ] ]database_def_intraday_file [ relname [ filename ] ]database_def_tick_file [ relname [ filename ] ]database_def_millisecond_file [ relname [ filename ] ]

The tilde escape is not implemented in the database_def_daily_file,database_def_intraday_file or database_def_tick_file ordatabase_def_millisecond_file.

An alternative syntax follows:

For example, if the following file is needed:

/home/lim/data/xmim.mimdb/daily/part_a.fac

then enter:

database_def_daily_file TopRelation:PartA daily/part_a.fac

The file is located relative to the xmim.mimdb directory. This way the database def commands are all relative and haveno dependency on the true database location.

These commands can be used to partition the data across multiple disks/partitions. Partitioning the data isaccomplished simply by using multiple instances of the above commands with the appropriate relname andfilename specifications. Of course, relname need not be a leaf relation and, thus, the above specificationscan be used to specify where to store the data for an entire hierarchy of relations. TopRelation canbe used to indicate all relations. All facts_read commands following a database_def_daily_file,database_def_intraday_file, database_def_tick_file or database_def_millisecond_filecommand and pertaining to the given relation or a child of that relation will store data to the given filename.

Note that, while it is possible to partition the data within a relation (split up the columns), it isnot possible to partition the data within a column of a relation (i.e., a time series). As with theset_holiday_schedule command, the filename specification may include the full Unix pathnameor may consist of a relative pathname and the path must resolve to the same Unix subtree as theschema file or an error message will be issued. When these commands are specified with only therelname as an argument, the Commodity DataServer will return to using the system defaults forstoring the remaining data for that relation (and it's children). When these commands are used with noarguments, the effect is as if TopRelation had been specified; that is, the Commodity DataServer willreturn to using the system defaults for storing the remainder of the data for all relations.

The special filename NONE can be used in the database_def_daily_file, database_def_intraday_file,database_def_tick_file and database_def_millisecond_file commands to indicate that daily, intraday,tick or millisecond data should not be allowed for the given relation. In the context of multiple databases this is usefulif it is desired to place the daily data in a separate database from the tick data for given relations. The use of NONE willensure that data be loaded in the appropriate database.

56


To see where this becomes useful, suppose, for example, that there are two databases open: /home/lim/daily_data/xmim.mim and /home/lim/tick_data/xmim.mim. As the names suggest, /home/lim/daily_data/xmim.mim has dailydata and /home/lim/tick_data/xmim.mim has tick data (both intraday and tick-by-tick). Consider a sequence of BMIMcommands whereby a new relation (e.g., IBM) is added followed by reading daily data and then reading tick datafor that relation. Since IBM does not exist, it is added to both the /home/lim/daily_data/xmim.mim and /home/lim/tick_data/xmim.mim databases. When the system tries to read daily data, it looks for the first database that alreadyhas daily data, so as to append to that database. Neither database does, so it will have to pick the first one, namely /home/lim/daily_data/xmim.mim. Then, when the process is repeated for the tick data, the same algorithm will pick /home/lim/daily_data/xmim.mim as the database to add the data to. This would not be correct given the intentions for /home/lim/daily_data/xmim.mim to only have daily data and tick data to reside in /home/lim/tick_data/xmim.mim. Thespecial NONE filename can be used as such to prevent this from happening:database_narrow { database = /home/lim/daily_data/xmim.mim;}database_def_intraday_file TopRelation NONEdatabase_def_tick_file TopRelation NONEdatabase_narrow { database = /home/lim/tick_data/xmim.mim;}database_def_daily_file TopRelation NONEdatabase_widenrelation_add { name = IBM; parent = Equities;}facts_read { file = daily_data; relation = IBM;}facts_read { file = intraday_data; relation = IBM; tick_intraday;}facts_read { file = tick_data; relation = IBM; tick_by_tick;}database_widen

This BMIM sequence first narrows the focus to the daily database and then specifies that no intraday tick or tick-by-tick data is allowed in that database. The focus is then narrowed to the tick database and a command given tospecify that no daily data is allowed in that database. After that, the database_widen command is used to set thedatabase pool back to the original, and the commands to add the IBM relation (will be added to both databases) andthen load daily, intraday and tick data are given with the results being that the appropriate data goes to the appropriatedatabase.

Note that this same behavior could have been accomplished with the following BMIM sequence:relation_add { name = IBM;

57

parent = Equities;}database_narrow { database = /home/lim/daily_data/xmim.mim;}facts_read { file = daily_data; relation = IBM;}database_narrow { database = /home/lim/tick_data/xmim.mim;}facts_read { file = intraday_data; relation = IBM; tick_intraday;}facts_read { file = tick_data; relation = IBM; tick_by_tick;}database_widen

The database_narrow command should be used to set the focus such that data goes to the appropriate databaseand the database_def_daily_file, etc. commands need not be used. However, it is anticipated that thedatabase_def_daily_file, etc. commands will be used when a database is first created and the convenience ofnot needing to be very cautious about narrowing the focus to specify the appropriate database will be found useful.

Printing the Database Partition TableUse the database_def_write command to print the partition table of a single database. You must narrow to adatabase first:database_def_write <file>

This is where all new relations will be located. Existing relations are not affected by changes to the partition table,so the partition table does not always reflect the truth of where a relation resides. Use one of the print_*_filecommands (see “Printing the Location of a Relation and Column”) to print the actual location of a relation and column.

Printing the Location of a Relation and ColumnUse the following family of print_*_file BMIM commands to print the actual location of a relation and column. Youmust lock the database first. The database does not have to be narrowed.print_daily_file <relation> <column>print_intraday_file <relation> <column>print_tick_file <relation> <column>

58


Printing Database Schema InformationThe print_schema command is used to print information about the data existing in the database. The synopses forthe two forms of this command are:print_schema [ relname ]print_schema { relation = rel1, rel2, ...; file = output filename; max_depth = depth; all_contracts; [ append | overwrite ]; print; verbose;}

The verbose option is available beginning with version 4.x.

The print_relation command is interchangeable with the print_schema command.

The relation for which information is to be printed is specified by relname. Note that if this is a relation category,the command will recursively print all information pertaining to the children of the category as well as the informationabout the category itself. Thus, the entire relation hierarchy beneath the specified category will be printed. If relnameis not specified, information will be printed for the entire schema relation hierarchy.

The second form of the command allows for multiple relation names to be specified, resulting in the printing ofinformation for all given relations. When this form of the print_schema command is used, a number of additionalarguments may also be specified, all of which are optional. The max_depth and all_contracts arguments canbe used to limit the level of descendants printed for a relation category. If the depth given is 0, only informationpertaining to the category itself is printed. If the depth is 1 then information for the immediate children will alsobe printed (but not for their descendants). In general, depth specifies the number of generations of children forwhich information will be printed. If the max_depth argument is not given, the system will default to printingthe information for all descendants as described above for the first form of the print_schema command. Theall_contracts flag is relevant for futures relations only. By default, only the relevant continuous contracts areprinted for futures relations, the individual contracts are not. If the all_contracts flag is specified, information willadditionally be printed for all individual futures contracts existing in the hierarchy beneath the futures relation. Notethat even in the presence of the all_contracts flag, for the individual contracts to be printed, the max_depthargument must be absent or such that the individual contracts generation is included. Thus, to print information forall futures contracts, both continuous and individual, the all_contracts flag should be used but the max_depthargument should be left unspecified.

When more than one database is open, hierarchy information is merged as consistent with viewing multiple databasesas though they were a single database. Where duplication exists, the schema information shown will be that from thefirst applicable database.


59

The output will be written to output_filename, if it is specified. The output can also be sent to a printer byspecifying the print option. If no output_filename is given and the print option is not used, the output will be sendto standard output. If the output_filename is specified, append_mode and overwrite_mode can be used toappend to the given file or replace it, respectively. The default will be to replace the file.

By default, the information printed for each specified relation consists of the relation name (ticker symbol), columnsdefined for the relation and the starting and ending dates for the data. The verbose option can be used to print allpertinent relation and relation column information. In general, attributes that are inherited from the parent and have notyet been bound are not printed. For example, the following print_schema command produces verbose output forthe futures relation AD and its children continuous contracts.print_schema { relation = AD; file = AD outfile; verbose;}

The output is more suitable for human consumption if the verbose option is utilized and more suitable forprogrammatic use if the verbose option is not specified.

A partial listing of the generated output follows. Since the all_contracts option was not specified, no outputwas produced for the individual contracts of AD. Notice that only the parent futures relation has Cash data. The Cashrelation column is defined for all the children contracts to AD, but print_schema will not print any relation columninformation (i.e., type, object_type, . . . ) since these attributes are set to be inherited from the parent relation. Valueswill be assigned to these attributes only by explicitly setting them using relation_modify or when the Cash datais added. Also note that none of the relations have tick data.Futures: Futures:Currencies:AD (Australian Dollar) Exchange: IMM Time Zone: US/Central Trading Hours: 7:20am -2:00pm RolloverPolicy: front, year to year, actual prices. RolloverDate: Last Trading Day. Column: Open (Open) Type: Base ObjectType: Float Aggregation: Open Daily Data: 01/13/1987 -06/13/1994 Column: High (High) Type: Base ObjectType: Float Aggregation: High Daily Data: 01/13/1987 -06/13/1994 Column: Low (Low) Type: Base ObjectType: Float Aggregation: Low Daily Data: 01/13/1987 -06/13/1994 Column: Close (Close) Type: Base ObjectType: Float Aggregation: Close Daily Data: 01/13/1987 -06/13/1994 Column: Cash (Cash)

60


Type: Base ObjectType: Float Aggregation: Close Daily Data: 08/09/90 -06/15/1994 Column: Volume (Volume) Type: Base ObjectType: Integer Aggregation: Sum Daily Data: 01/13/1987 -06/13/1994 Column: OpenInterest (OpenInterest) Type: Base ObjectType: Integer Aggregation: Close Daily Data: 01/13/1987 -06/13/1994Continuous: Futures:Currencies:AD:AD_H (Australian Dollar (March)) RolloverPolicy: front, March, actual prices. RolloverDate: Last Trading Day. Column: Open (Open) Type: Base ObjectType: Float Aggregation: Open Daily Data: 01/13/1987 -03/14/1994 Column: High (High) Type: Base ObjectType: Float Aggregation: High Daily Data: 01/13/1987 -03/14/1994 Column: Low (Low) Type: Base ObjectType: Float Aggregation: Low Daily Data: 01/13/1987 -03/14/1994 Column: Close (Close) Type: Base ObjectType: Float Aggregation: Close Daily Data: 01/13/1987 -03/14/1994 Column: Cash (Cash) Column: Volume (Volume) Type: Base ObjectType: Integer Aggregation: Sum Daily Data: 01/13/1987 -03/14/1994 Column: OpenInterest (OpenInterest) Type: Base ObjectType: Integer Aggregation: Close Daily Data: 01/13/1987 -03/14/1994Continuous: Futures:Currencies:AD:AD_M (Australian Dollar (June)) RolloverPolicy: front, June, actual prices. RolloverDate: Last Trading Day. Column: Open (Open) Type: Base ObjectType: Float Aggregation: Open Daily Data: 01/13/1987 -06/14/1993

61

Column: High (High) Type: Base ObjectType: Float Aggregation: High Daily Data: 01/13/1987 -06/14/1993 Column: Low (Low) Type: Base ObjectType: Float Aggregation: Low Daily Data: 01/13/1987 -06/14/1993 Column: Close (Close) Type: Base ObjectType: Float Aggregation: Close Daily Data: 01/13/1987 -06/14/1993 Column: Cash (Cash) Column: Volume (Volume) Type: Base ObjectType: Integer Aggregation: Sum Daily Data: 01/13/1987 -06/14/1993 Column: OpenInterest (OpenInterest) Type: Base ObjectType: Integer Aggregation: Close Daily Data: 01/13/1987 -06/14/1993

Putting a Database into Safe ModeUse the writer_wait command to put the database in a "safe" (also known as "quiesced") state and pause thewriter slave.

writer_wait { flag_file = <filename>; max_wait = <seconds>;}

The writer_wait command is designed for use with external backup solutions. The database (the schema and thedata files) is guaranteed not to be modified for the duration of the "wait". All other requests to the writer slave (i.e.,from database lock owners) will block until the wait is over. The wait terminates when one of the following occurs:

● the timer expires

● the flag file is removed externally

The parameters include:

● flag_file - The server creates this file once safe state is achieved and removes it when the wait is over. Removingthis file will cause the wait to end. If the file already exists the command will fail with an error.

62


● max_wait - Timer value in seconds. Setting this to zero will cause the server to wait forever (until the flag file isremoved).

Ideally, the writer_wait command should be placed in a make_data file by itself, packaged and processed viaload_updates.sh. Whoever/whatever is requesting the wait state should wait for the flag file to appear.

It is possible to do this by hand from a parallel BMIM session. The caveat is that if there is load_updates running and inthe middle of loading a package when writer_wait is called, then the load will fail. In this case the package shouldbe simply reloaded.

Miscellaneous Commands

Unix InterfaceThe system command can be used to execute an arbitrary Unix command or list of commands from within a BMIMscript. The synopses for the two forms of this command are:

system "unix command"system { "unix_command_1"; "unix_command_2"; ...;}

The chdir command is used to change the current working directory from within a BMIM script. It is analogous to theUnix chdir command. Its synopsis is as follows:

chdir path

The directory pointed to by path will become the current working directory.

Setting Global OptionsIf the suppress_duplicate_add flag is set, subsequent _add commands will only be performed if thecorresponding relation, column or relation_column does not already exist. When this flag is used, if anattempt is made to add a previously existing item, Commodity DataServer will quietly ignore the duplicate add, and,thus, no error messages will be produced. The synopsis is simply:

suppress_duplicate_add


63

If the suppress_duplicate_delete flag is set, subsequent _delete commands will only be performed if thecorresponding relation, column or relation column exists. When this flag is used, if an attempt is made to delete a non-existing item, Commodity DataServer will quietly ignore the delete, and, thus, no error messages will be produced. Thesynopsis is simply:suppress_duplicate_delete

If the suppress_bad_date_time flag is set, subsequent facts_read commands will not print warning messageswhen data values are ignored by Commodity DataServer because they fall on invalid dates or times (e.g., the datecorresponds to a weekend or holiday). The synopsis is:suppress_bad_date_time

If the suppress_relation_not_found flag is set, the error message “Relation does not exist” is suppressed whenusing the relation_modify command.

This flag only suppresses relation_modify error messages. The synopsis is:

suppress_relation_not_found

Using Data CompressionBy default, real and intraday tick data that is loaded into the database will be compressed, while daily data willnot be compressed. To override this behavior and specify that a given type of data be compressed or not, theset_compress and unset_compress commands are used. The synopses are:set_compress [ data type ]unset_compress [ data type ]

The data_type argument is optional. If specified, it must be one of the following keywords:

● tick● intraday

Data compression is only supported for intraday tick and real tick data. All facts_read commands following aset_compress command will store the specified type of data in a compressed format. Likewise, all facts_readcommands following an unset_compress command will store the specified type of data in regular uncompressedformat. Thus, it is possible to specify whether or not to compress the data on a per-time series basis by utilizingmultiple instances of the set_compress and unset_compress commands. Note that the compression is notperformed on a file basis, and, thus, how the data is split up into files in no way affects the system performanceduring data retrieval. If the data_type argument is not given, the result will be to turn compression on or off for allsubsequent tick and intraday tick data loaded.

In fact, due to the significant reduction in I/O processing time when compression is used, data retrievalmay actually be faster on compressed data even with the time to uncompress factored in.

64


Some extremely sparse time series do not benefit from compression and, in these cases, it may be desirable to ignorethe compression settings, leaving the time series uncompressed. By default, Commodity DataServer will choose athreshold for the data size, below which data will not be compressed.

The current data compression threshold used by Commodity DataServer is 256 bytes.

The set_compress_threshold command can be used to change the compression threshold.

set_compress_threshold [ data size ]

If specified, the data_size argument must be a non-negative integer. All data to be compressed is checked on a perday, per time series basis and if the data size is below the number of bytes given by data_size then that particulardata will not be compressed. If 0 is used for data_size then the effect will be to not use a threshold (i.e., all data,regardless of size, will be compressed). If the set compress threshold command is used without an argument, thethreshold will be reset to the default one used by Commodity DataServer.

Printing Expiration DatesThe dump_expiration_dates command is used to retrieve and print all contracts and their associated expirationdates for all specified futures relations. This command is available only in the client version of BMIM. Enter thefollowing command if you want contract/expiration output for every futures relation in the database:

dump_expiration_dates dirname

The dirname argument specifies a directory for the output information. One file for each futures relation will becreated in this directory.

If you want contract/expiration output for a specific futures relation, then enter the following:

dump_expiration_dates { file = dirname; relation = relname;}

EntitlementsThis section describes the BMIM commands for setting entitlements in a non WebServices environment. For moreinformation on entitlements see “Data Security (aka Entitlements)”. For a complete description of the WebServicesentitlements facilities see the Commodity Data WebServices API User's Guide.

The following BMIM commands are used for running entitlements:

http://www.morningstarcommodity.com/sites/default/files/WebService_API_2.3.2.pdf


65

entitlements_clearThe BMIM command entitlements_clear clears all previous entitlement information.

entitlements_clear

entitlements_readThe BMIM command entitlements_read loads new entitlement information from the batch input file. See“Entitlement File Format”

entitlements_read

entitlements_writeThe BMIM command entitlements_write dumps entitlement for the database that has been narrowed towardsto an output file.

entitlements_write

entitlements_write_allThe BMIM command entitlements_write_all dumps entitlement to output files named ent_0.out, ent_1.out,ent_2.out etc. for each database on the Commodity DataServer. Each file has the name of the database inside it.Before running the command you must first create a directory (this directory will contain the output files). See theexample below.

entitlements_write_all

Entitlements ExampleThe following is an example script:

#clear all previous entitlements informationentitlements_clear;#load new entitlements information from batch input fileentitlements_read /home/lisa/entitlements1.txt;#dump entitlements to output fileentitlements_write /home/lisa/entitlements1_out.txt;#dump entitlements for each database on the MIM Server to output files in designated directory!"mkdir /home/lisa/temp/entdir"entitlements_write_all /home/lisa/temp/entdir;

66


COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 5: C/C++ API

67

CHAPTER5

C/C++ API

The BasicsThe Commodity DataServer C/C++ Application Programming Interface (API) supports a client/server model. It doesthis through a Remote Procedure Call (RPC) mechanism, so that the client and the server need not reside on the samemachine. The server must be brought up before any applications can utilize it and must be brought down manually.

The Commodity DataServer C/C++ API is not thread-safe. For applications that require multiplethreads, Morningstar Commodity Data recommends using the Commodity DataServer Java, CommodityDataServer .NET or Commodity DataServer WebServices API.

Invoking a ServerThe Commodity DataServer utilizes an architecture whereby the workload is distributed by the main (master) server toadditional (slave) servers. In this manner, data access and query execution requests can be processed simultaneouslywithout sequential first in, first out limitations. Thus, a large request no longer needs to hold up the processing ofsmaller requests. With this architecture, database updates can be efficiently processed without hindering the abilityof the server to process data access requests. Moreover, multiple database writers are now permitted such that onewriter does not lock out all others. While multiple clients are allowed to write to the database at the same time, themaster server will assign only a single slave server as the historical write request processor, with all historical writerequests from all clients directed to that slave server such that they are processed sequentially. Because, at any onetime, only a single slave server will be allowed to write to the historical database, deploying additional slave serverswill not speed up updates to the database.

Note that when the high frequency (current tick) data facility (see “High Frequency Updating”) is used,the master server will automatically invoke a new slave server to serve as the dedicated writer for thecurrent tick store. This slave server will be distinct from any other slaves used to process access/queryrequests or updates to the historical database.

The architecture is best utilized by having a single master server per database, with as many slave servers as desired.Unless network access is extremely fast, it is strongly recommended that the slaves all run on the same machinewhere the database is physically located. All Commodity DataServers associated with a particular database load theschema file into memory using memory-mapped IO. Whenever any server modifies the schema information, all theother servers will notice the change.

68


The user has the option of manually starting up slave servers or leaving it to the Commodity DataServer master serverto do so as necessary. Even when it is left to the Commodity DataServer master server to invoke the slave servers,the user is responsible for specifying on which hosts to run the slave servers and how many of them can potentiallybe dispatched8 . This information is specified when the Commodity DataServer master server is invoked. Of course,all command line arguments are optional and the default is a single slave server on the local host. The CommodityDataServer (master) is invoked as follows:hostname% xmim_server [-h] [-v] [-p server_number] [-l log_level] [-f log_file] [-k] [-i pid_file] [-t idle_time] [-q host] [-r min max] [-w] [-M]

where:-h prints out the usage.

-v prints out the version number and compilation time.

-p assigns a unique integer identifier, server_number, to the server. This is useful when multiple servers are desireda.For example, if the following commands are given at the shell, hostname% xmim_server -p 1 hostname% xmim_server -p 2

then there will be two servers running, known to the user as server 1 and server 2. Before invoking server 2, the user maywish to change the xmimrc environment variable to point to a different .xmimrc file in order to use a different databasefor the second serverb. The default server number is 0.

-l specifies that the server perform logging, with level log_level. Logging levels 1 through 7 are supported. If a levelof 0 is specified then the process that has been granted connection along with the time of connection, userid, hostnameand type of client is recorded. If a level of 1 is specified then, in addition to the level 0 logging, each API functionexecuted by the server will be recorded. A level of 2 will additionally provide the timing of each function executed. A levelof 3 will, in addition to the logging from previous levels, log query executions and data access. The information loggedwith level 3 logging includes the query or API function name, the server access time, all relation columns accessed and,for each relation column, the type of data (i.e., daily, intraday or tick) and data range, and a status including the errormessage for failed requests. Log levels 4 and higher support the Common Log Format (CLF). This format is used by webservers and there are many third party analyzing tools available (including wwwstat and wusage). Log levels 6 and 7display the execution times, start and finish, of all function or API calls received. See Chapter 22, “Commodity DataServerLogging” for log level details. The slave server generates all the logging information with the exception of informationregarding the starting and killing of slaves which is logged by the master server. All information is logged to a single file(see the -f switch below).

-f specifies the file name, log_file, to be used for logging. The file .xmim_server_server number.log is the default logfile where the server_number is the one specified by the -p option (or 0 by default). So, for example, if the master serveris invoked as:hostname% xmim_server -p 1 -l 1

then the log file will be .xmim_server_1.log. If the -f option is specified but not the -l option then logging will beperformed using a log level of 0. The log files will reside in either the temporary directory specified by the user inthe .xmimrc filec, the current directory (if it is writable), or /tmp.

-k specifies that if a server already exists with the given server number, then this existing server will be (silently) terminated.By default, the existing server will not be terminated. Instead, this invocation will print a message indicating that a serverwith the specified server number already exists and will exit.

-i specifies to print out the process id numbers for all servers (master and slave) when invoked to the file pid_file.

-t forces the server to terminate slave servers that are idle for more than idle_time minutes. By default, slave servers, onceinvoked, will not be brought down.

8 The size of the slave server executables is significant so attention should be paid to this size verses the limitations of the hardware where the servers arerunning before too many slave servers are specified.


69

-q -r is an alternative way to specify where the slaved servers are to be started. The host (defaulting to localhost) is thehostname where the slave servers should run, and the min and max (both defaulting to 0) specify the range of portnumbers to use.

Note that the slave server port numberse are distinct from the master server port numbers, hence, bydefault, the master server port number will be 0 and so will the default single slave server used. Thisspecification will override the -u option.

-w specifies that slave servers will be started manually instead of being invoked by xmim_server as needed. Forexample, two slaves could be started for one server as follows: xmim_server -w xmim_slave_server -p 1 xmim_slave_server -p 2

-M stops registration with the portmapper.a On a single host, there can be up to 255 master servers (server numbers 0-254).b The database(s) pointed to in the .xmimrc file of the user who invokes the Commodity DataServer will be loaded and used as the current database.c For a description of the .xmimrc file, refer to Chapter 40, “Commodity DataServer .xmimrc File”.d There can be up to 4096 slave servers (slave numbers 0-4095) associated with a single master server.e These are not actually raw port numbers but serve as a means of identifying servers.

The Commodity DataServer slave server is invoked as follows: hostname% xmim_slave_server [-h] [-v] [-p slave_number] [-l log_level] [-f log_file] [-i pid_file] [-n master_server_number] [-o] [-M]

where:-h prints out the usage.


-p specifies the number of the slave (will default).

-l specifies that the server perform logging, with level log_level. Logging levels 1 through 7 are supported. If a level of 0 is specifiedthen the process that has been granted connection along with the time of connection, userid, hostname and type of client is recorded.If a level of 1 is specified then, in addition to the level 0 logging, each API function executed by the server will be recorded. A level of 2will additionally provide the timing of each function executed. A level of 3 will, in addition to the logging from previous levels, log queryexecutions and data access. The information logged with level 3 logging includes the query or API function name, the server access time,all relation columns accessed and, for each relation column, the type of data (i.e., daily, intraday or tick) and data range, and a statusincluding the error message for failed requests. Log levels 4 and higher support the Common Log Format (CLF). This format is used byweb servers and there are many third party analyzing tools available (including wwwstat and wusage). Log levels 6 and 7 display theexecution times, start and finish, of all function or API calls received. See Chapter 22, “Commodity DataServer Logging” for log leveldetails. The slave server generates all the logging information with the exception of information regarding the starting and killing of slaveswhich is logged by the master server. All information is logged to a single file (see the -f switch below).

-f specifies the file name, log file, to be used for logging. The file .xmim_server_server_number.log is the default log file where theserver_number is the one specified by the -p option (or 0 by default). So, for example, if the master server is invoked as:hostname% xmim_server -p 1 -l 1

then the log file will be .xmim_server_1.log. If the -f option is specified but not the -l option then logging will be performed using a loglevel of 0. The log files will reside in either the temporary directory specified by the user in the .xmimrc filea, the current directory (if it iswritable), or /tmp.


-n specifies the server number of the master (same as -p in xmim_server command).

70


-M stops the registration with portmapper.a For a description of the .xmimrc file, refer to Chapter "Commodity DataServer .xmimrc File".

The Commodity DataServer performs signal handling such that the following command will gracefully bring down themaster server for which the pid is given and all of it's slave servers.

hostname% kill -TERM pid_of_master_server

It is very important that this command be used in lieu of a kill -9 command as the master server is able to clean upafter itself when the kill -TERM is used.

The xmim_svr_info utility program can also be used to gracefully kill a server.

The Commodity DataServer also processes the -INT signal which results in the master pinging all slave servers andresetting so as to remove any slaves that have died from the list of currently used slave servers. This will allow themaster server to put a slave server back in use when it is needed.

hostname% kill -INT pid_of_master_server

Connecting and Disconnecting from a ServerIn this section, we will describe the basic structure of a Commodity DataServer API client program. First, a client mustinclude the header file xmim_api.h. Next, it should include a call to XmimConnect. The routine XmimConnectestablishes the connection with the desired Commodity DataServer. Finally, the client must call XmimDisconnectbefore terminating. Thus, the following example shows the minimal Commodity DataServer client:

#include "xmim_api.h"#include <stdio.h>main (){ XmimClientHandle handle; if (XmimConnect ("hostname", 0, &handle) != XMIM_SUCCESS) { XmimPrintError("XmimConnect"); return 1; } XmimDisconnect (handle); return 0;}

XmimConnect has the following calling sequence:

XmimReturnCode XmimConnect (XmimString serverName, int serverNumber,

71

XmimClientHandle *handle);

The serverName specifies the hostname of the machine on which the Commodity DataServer is running. If thisargument is NULL, the client tries to connect to a server on the local machine. The serverNumber argument is usedto specify which server to connect with. When a server is invoked, the server number can be supplied by the user.The default server number is 0. If there is no server running on the specified host, XmimConnect fails. If the callsucceeds, XmimConnect implicitly returns, through an output parameter, a client handle consisting of a positiveinteger. This client handle serves to uniquely identify the server connection enabling connections to (and, therefore,utilization of) different servers by a client. All Commodity DataServer C/C++ API routines, with the exception of theutility/convenience routines, require this client handle to be specified as an argument so as to identify which serverto utilize. The explicit return type for the XmimConnect routine is XmimReturnCode. This type is defined with thefollowing enumeration:

typedef enum { XMIM_ERROR, XMIM_SUCCESS, XMIM_WARNING} XmimReturnCode;

All Commodity DataServer C/C++ API routines, again, with the exception of the utility/convenience functions, willutilize XmimReturnCode for their return type. Thus, each routine in the Commodity DataServer C/C++ API returnsa standard indication of failure, success, or success with information (i.e., warning). In the cases where XMIM_ERRORor XMIM_WARNING is returned, the XmimPrintError routine can be used to obtain the error or warning message.XMIM_WARNING will be returned if a routine is used that retrieves data or schema information and the number ofrecords, children, etc. returned is 0. The warning message will indicate that no results were found.

XmimDisconnect has the following synopsis:

XmimReturnCode XmimDisconnect (XmimClientHandle handle);

The handle argument must be the client handle returned by the XmimConnect routine and identifies which server todisconnect from. The server is not brought down by an XmimDisconnect even if there are no other clients connectedto the server. It is very important that clients call XmimDisconnect before exiting for any reason (error condition ornormal exit) in order for the client handle space not to be exhausted.

Note the use of the Commodity DataServer special defined types in the above synopsis.

The XmimString type is used for character strings and XmimBoolean is used when the argument is a flag.

A simple client to ping the server is shown below:

#include "xmim_api.h"#include <stdlib.h>#include <stdio.h>#define usage fprintf (stderr, "Usage: %s [-h host]\n", argv[0])main(int argc, char **argv){

72


XmimClientHandle handle; char *host = NULL; char c; extern char *optarg; while ((c = getopt(argc, argv, "h")) != -1) { switch (c) { case 'h': host = optarg; break; default: usage; return 1; } } if (XmimConnect(host, 0, &handle) != XMIM_SUCCESS) { printf("xmim_server not running\n"); return 1; } else { printf("xmim_server is running\n"); XmimDisconnect(handle); return 0; }}

The following commands compile and run the program, assuming /home/xmim is the full path name of the directorywhere the DataServer was installed:

hostname% cc -c -I/home/xmim/include ping_server.chostname% cc -o ping_server ping_server.o -L/home/xmim/lib -lxmimhostname% ping_server

Under Solaris 2.x you will need to add two additional libraries when linking:

hostname% cc -o ping_server ping_server.o -L/home/xmim/lib -lxmim -lnsl -lform

Dealing with Optional ArgumentsMany of the API commands take a large number of arguments, most of which can be defaulted. The CommodityDataServer C/C++ API provides a convenient way to default arguments, through a name-value parameter passingmechanism. All API commands have both a traditional positional interface, as described above, as well as a name-value interface. The name-value interface of the function is accessed by prefixing the name of the function withXmimVa instead of Xmim. For example, the minimal Commodity DataServer client program above could be written asfollows:

#include "xmim_api.h"#include <stdio.h>main (){


73

XmimClientHandle handle; if (XmimVaConnect (XMIM_SERVER_NAME, "hostname", XMIM_SERVER_NUMBER, 0, XMIM_CLIENT_HANDLE, &handle, XMIM_END_ARGS) != XMIM_SUCCESS) { XmimPrintError("XmimVaConnect"); XmimVaDisconnect (XMIM_CLIENT_HANDLE, handle, XMIM_END_ARGS); return 1; } XmimVaDisconnect (XMIM_CLIENT_HANDLE, handle, XMIM_END_ARGS); return 0;}

The arguments are specified by using a naming token (e.g., XMIM_SERVER_NAME) which then takes one or morearguments. The exact number of arguments is determined by the token. The name-value pairs can occur in any order,and any of these that have default values may be omitted. The special token XMIM_END_ARGS is used to delineate theend of the argument list. Clearly, the XmimVa interface does not present an advantage when the number of argumentspassed is small, as in the case of the two routines above. However, it is a significant improvement when the routinetakes a large number of arguments. Subsequent sections of this document provide examples of such routines.

The recommended programming style is to use the XmimVa version of routines. This alleviates the necessity ofchanging client source code if a future version of the API requires the addition of new arguments to an existing routine(e.g., to add additional functionality). Since the XmimVa routines provide for omitting arguments with default values,an existing client would only need to be changed if the new functionality was to be incorporated into the client. In thisand future versions of the API we will attempt to provide for backward source code compatibility for existing clientsby leaving non-Va routines as is, instead adding any new arguments to the XmimVa routines only. Thus, clients will beeasier to extend to utilize new functionality if the XmimVa routines are used.

Throughout this document, the synopsis for each routine will include both an ANSI C header prototype for theregular Xmim routine and an actual call for the XmimVa specification. We will use the following convention to specifythe arguments for the XmimVa calls. All arguments will be listed in the synopsis. Optional arguments will havetheir default value listed as the argument of the call, whereas required arguments will use the same name as thecorresponding argument of the regular Xmim routine specification. The types for arguments in the XmimVa routinescan be determined from the specified types for the equivalent arguments of the corresponding ANSI C headerprototype. Because the XmimVa specification is an actual call, the return arguments in this specification will bepreceded by the address operator (&). The XmimVa synopses for connecting to and disconnecting from the server areas follows:XmimReturnCode XmimVaConnect (XMIM_SERVER_NAME, NULL, XMIM_SERVER_NUMBER, 0, XMIM_CLIENT_HANDLE, &handle, XMIM_END_ARGS);XmimReturnCode XmimVaDisconnect (XMIM_CLIENT_HANDLE, handle, XMIM_END_ARGS);

Notice that all of the input arguments for the XmimVaConnect routine have default values and, therefore, areoptional. Thus, only the handle return argument, along with the XMIM_END_ARGS end of list specification, are actuallyrequired for this routine. Likewise, only the handle argument is required for the XmimVaDisconnect routine, though,in this case, it is an input argument.

74


The XmimVa interface also provides a flexible mechanism for specifying multiple columns, relations, etc. as arguments.The following section addresses this capability.

Metadata TerminologyThis section provides some introductory terminology that may prove helpful in understanding the organization of theCommodity DataServer database and, therefore, this document. In the DataServer, a relation corresponds to a tradinginstrument (i.e., equity, future, etc.) and a column corresponds to specific information for an instrument (i.e., economicor monetary values, market activity, etc.). Often, the assigned relation name will correspond to the ticker symbolfor that instrument. A column can be associated with a relation resulting in what is termed a relation column. Forexample, the relation IBM could have the columns Open, High, Low, Close and Volume associated with it to form therelation columns Open of IBM, High of IBM, etc. A relation column, thus, corresponds to an actual time series. Data isactually added only to relation columns.

In the DataServer, both relations and columns are organized hierarchically. When a relation or column is added, theparent must always be specified, with TopRelation and TopColumn denoting the roots of the relation and column trees,respectively. Special relations and columns typed as categories are used to group relations and columns; they have allthe properties of relations and columns but will never have data associated with them. Relations and columns that arenot solely used for grouping and do have associated data are referred to in this document as leaf-level relations andcolumns because they typically correspond to terminal leaves in the tree hierarchy.

Neither leaf-level relations and columns, nor category relations and columns are required to have unique names.However, in specifying non-unique relations or columns, a disambiguating path must be provided. This path mustconsist of the ancestors (i.e., parent, grandparent, etc.) of the relation or column going back as far as necessary todisambiguate. Colons are used as separators if the path is specified with immediate descendants; a double colon isused as a separator to indicate a descendant that is not necessarily an immediate descendant. For example, if a singledatabase exists with both Corn and Chrysler and they both use C for the relation name, Corn could be referencedunambiguously as Futures:C if Futures was the parent and could be referenced unambiguously as Futures::C if Cwas a descendant of Futures but not necessarily an immediate descendant. Likewise, Chrysler might be referencedunambiguously as Equities:C if Equities was the parent and as Equities::C is Equities was an ancestor but notnecessarily the parent.

In the context of multiple databases, relations and columns that are not unique across databases need not be referredto with a disambiguating path name. In that case, the order of loading the databases provides a resolution to theambiguity in that databases will be searched in the order loaded and the first one with the specified relation or columnname will be used. Refer to “Supporting Multiple Databases” for a description of using multiple databases.

The relation hierarchy is not required to be a strict single-path tree hierarchy. By using the relation alias facility, multiplepaths to a relation can be assigned. Multiple names can also be associated with a relation via this facility. Refer to“Adding Relation Aliases” for a description of using the relation alias facility.

The relation and column hierarchies together with all information pertaining to the attributes of existing relations,columns and relation columns comprise the metadata of the database: termed the DataServer schema. Thus, thestructure of the database (consisting of the actual time series data values) is described by the DataServer schema.


75

Data Retrieval

Browsing the Relation HierarchyIn this section, we show how the Commodity DataServer C/C++ API can be used to write a client thatbrowses the data in an Commodity DataServer database. The relation hierarchy can be browsed using the callsXmimGetRelChildren and XmimGetRelType. The synopses for these calls are as follows:

XmimReturnCode XmimGetRelChildren (XmimClientHandle handle, XmimString relName, int *numChildren, XmimString **children);XmimReturnCode XmimVaGetRelChildren (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_RELATION_ARRAY, &numChildren, &children, XMIM_END_ARGS);XmimReturnCode XmimGetRelType (XmimClientHandle handle, XmimString relName, XmimRelType *type);XmimReturnCode XmimVaGetRelType (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_TYPE, &type, XMIM_END_ARGS);

The XmimGetRelChildren and XmimGetRelType routines require the relation name (ticker symbol) to be specifiedas an argument. If the relation name is not unique then a disambiguating path name must be specified with the colon(:) character used as a separator (e.g., Futures:Indices). XmimGetRelChildren returns, as arguments, a pointerto an array of relation names consisting of the immediate children of the given relation as they exist in the relationhierarchy and the number of immediate children. The XmimGetRelType routine returns a type argument, indicatingthe type of the relation, as in the following enumeration:

typedef enum { XMIM_REL_INVALID, /* No Such Relation XMIM_REL_CATEGORY, /* "Menu" of Relations XMIM_REL_NORMAL, /* Standard Leaf Relation XMIM_REL_FUTURES, /* Main Futures Continuous XMIM_REL_FUTURES_CONTRACT, /* Futures Individual Contract XMIM_REL_FUTURES_CONTINUOUS /* Futures Continuous Contract} XmimRelType;

The routine XmimGetDataRange can be used to determine the dates for which data is available for a given relation.The synopsis for this call is as follows:

XmimReturnCode XmimGetDataRange (XmimClientHandle handle, XmimString relName, int numColumns, XmimString *colNames, XmimUnits units, XmimDate *fromDate,

76


XmimDate *toDate);XmimReturnCode XmimVaGetDataRange (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_COLUMN, colName, XMIM_COLUMN_LIST, colName1, colName2, ...., NULL, XMIM_COLUMN_ARRAY, numColumns, colNames, XMIM_COLUMN_FILE, fileName, &numEntries, &colName XMIM_UNITS, XMIM_DAYS, XMIM_FROM_DATE, &fromDate, XMIM_TO_DATE, &toDate, XMIM_END_ARGS);

The arguments of the XmimVaGetDataRange routine require some explanation. In general, whenever multiplecolumns appear as input arguments for a routine, the XmimVa call will accept the column list specified in any of thefollowing ways. First, the columns may be specified individually, each one preceded by the XMIM_COLUMN token.Secondly, a list of columns may be specified after the XMIM_COLUMN_LIST token. This list must be terminated witha NULL. An example using list specifications is provided in “Accessing Data for Multiple Relations”. Third, the list ofcolumns can be specified in an array. In this case, the arguments after the XMIM_COLUMN_ARRAY will be the numberof elements in the array, followed by the array itself. An example using the array specification is provided in “AccessingData with Relations and Columns Specified in Arrays”.

Note that the array specification is also useful as a return argument, when an array and the number ofelements in that array needs to be returned.

Finally, the list of columns can be written in a file, and then this file name specified after the XMIM_COLUMN_FILEtoken. When the file version is used, the numEntries argument returns the number of entries in the file specifiedand the colNames argument returns the column names that were read from the file. It is often critical in the clientside of the code to have access to these returned arguments. An example using the file specification is provided in“Accessing Data for Relations Specified in a File”.

These methods of specifying multiple columns can be combined. That is, the XMIM_COLUMN token can beused to specify one column, and then the XMIM_COLUMN_ARRAY can be used to add an array of columns.There can also be more than one of the same token specifiers (e.g., two XMIM_COLUMNs in the same call).Subsequently in this document, we will only indicate XMIM_COLUMN_LIST where all four tokens/methods canbe used. This will also apply to XMIM_RELATION_LIST, XMIM_SHOW ATTR_LIST, XMIM_CONDITION_LIST,XMIM_FORMAT_LIST, XMIM_FIELD_LIST, XMIM_DESCRIPTION_LIST, XMIM_OBJ_TYPE_LIST,XMIM_DELTA_LIST, XMIM_AGGR_RULE LIST, XMIM_QUERY_SUBSTITUTION_LIST, XMIM_DATABASE_LIST.In the case where there is a list of numbers, either integer or real, the list specification will not be allowed becausea value of 0 (or 0.0) will conflict with the NULL list terminator. In these cases, the array specification will be usedfor documentation purposes, though the multiple individual specifications or the file specification may also be used.Currently, this applies only to XMIM_CONSTANT_ARRAY. Notice that the XmimGetRelChildren routine does notaccept multiple relations, so the XMIM_RELATION_LIST approach does not apply there, and, therefore, only a singleXMIM_RELATION specification is given.

The types XmimDate, XmimTime, and XmimDateTime are specified as follows:

typedef struct {

77

unsigned year; unsigned month; unsigned day; unsigned hour; unsigned minute; unsigned second; unsigned millisecond;} XmimDateTime;

typedef struct { unsigned year; unsigned month; unsigned day;} XmimDate;

typedef struct { unsigned hour; unsigned minute; unsigned second;} XmimTime;

XmimDateTime should be used when both the date and the time need to be specified, XmimDate when only thedate information is relevant and XmimTime when only the time information is relevant. These structures are definedusing bit fields such that they require only a minimal number of bytes. Therefore, the user must be careful not toset any of the member fields to a value greater than that expected because the result may be to overwrite the nextmember field. Refer to Section for more detail on the date/time bit fields. The API also provides the special constantsXMIM_INVALID_DATE, XMIM_INVALID_TIME, and XMIM_INVALID_DATE_TIME. These constants are used asdefault arguments for the corresponding types and may be used to initialize the date/time structures. For example, thefollowing code could be used to provide a time of 1:15pm:

XmimTime time;

time = XMIM_INVALID_TIME;time.hour = 13;time.minute = 15;

The following functions are provided to facilitate comparisons with the date/time structures returned by the API:

int XmimDateCmp (XmimDate date1, XmimDate date2);int XmimTimeCmp (XmimTime time1, XmimTime time2);int XmimDateTimeCmp (XmimDateTime datetime1, XmimDateTime datetime2);

These routines compare the two given date/time structures and return zero if they are identical, a negative value if thefirst date/time is before the second and a positive value if the first date/time is after the second. If the client is writtenin C++, the comparison operators (==, !=, <, <=, >, >=) can also be used.

Note that the client handle is not necessary for these routines since they are merely convenienceroutines.

78


XmimGetDataRange returns the earliest (fromDate) and latest (toDate) days over which a given type ofdata is available. The units argument indicates which type of data, with XMIM_SECONDS indicating tick-by-tick,XMIM_MINUTES indicating intraday tick and XMIM_DAYS indicating daily. When numColumns is non-zero, colNamescan be used to limit the columns considered for determining the data range of a relation; otherwise (or if no columnsare specified in the XmimVa routine), all columns for the relation will be considered.

As an example, the following program traverses the Commodity DataServer relation hierarchy, printing the relationnames as well as the type of data currently available for each leaf relation.

#include "xmim_api.h"#include <stdio.h>#define EXIT \XmimPrintError(NULL); XmimDisconnect(handle, False); return 1main (int argc, char **argv){ XmimClientHandle handle; int i; XmimRelType reltype; int num_relname XmimString *relnames; XmimDate daily_from; XmimDate daily_to; XmimDate intraday_from; XmimDate intraday_to; XmimDate tick_from; XmimDate tick_to; if (argc != 2) { fprintf (stderr, "Usage: %s relation\n", argv[0]); return 1; } if (XmimConnect ("hostname", 0, &handle) != XMIM_SUCCESS) { XmimPrintError("XmimConnect"); return 1; } if (XmimGetRelChildren (handle, argv[1], &num_relnames, &relnames) != XMIM_SUCCESS) { EXIT; } for ( i = 0; i < num_relnames; i++) { if (XmimGetRelType (handle, relnames[i], &reltype) != XMIM_SUCCESS) EXIT; } printf ("%-10s ", relnames[i]); if (reltype == XMIM_REL_CATEGORY) printf ("(menu)\n"); else { if(XmimVaGetDataRange (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relnames[i], XMIM_UNITS, XMIM_DAYS, XMIM_FROM_DATE, &daily_from, XMIM_TO_DATE, &daily_to, XMIM_END_ARGS) == XMIM_ERROR || XmimVaGetDataRange (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relnames[i], XMIM_UNITS, XMIM_MINUTES, XMIM_FROM_DATE, &intraday_from,


79

XMIM_TO_DATE, &intraday_to, XMIM_END_ARGS) == XMIM_ERROR || XmimVaGetDataRange (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relnames[i], XMIM_UNITS, XMIM_SECONDS, XMIM_FROM_DATE, &tick_from, XMIM_TO_DATE, &tick_to, XMIM_END_ARGS) == XMIM_ERROR) { EXIT; } if (XmimDateCmp (daily_from, XMIM_INVALID_DATE) != 0) if (XmimDateCmp (intraday_from, XMIM_INVALID_DATE) != 0) if (XmimDateCmp (tick_from, XMIM_INVALID_DATE) != 0) printf ("(daily,intraday,tick)\n"); else printf ("(daily,intraday)\n"); else if (XmimDateCmp (tick_from, XMIM_INVALID_DATE) != 0) printf("(daily,tick)\n"); else printf ("(daily)\n"); else if (XmimDateCmp (intraday_from, XMIM_INVALID_DATE) != 0) if (XmimDateCmp (tick_from, XMIM_INVALID_DATE) != 0) printf ("(intraday,tick)\n"); else printf ("(intraday)\n"); else if (XmimDateCmp (tick_from, XMIM_INVALID_DATE) != 0) printf ("(tick)\n"); else printf ("(no data)\n"); } } XmimDisconnect (handle); return 0;}

Notice that the code is written such that if no children are found for the relation given (i.e., there is no hierarchy toprint), the warning message returned by XmimGetRelChildren will be printed since the warning and error cases aretreated the same. However, in the case of the XmimVaGetDataRange calls, the code checks only for the presenceof an error, handling the warning as a success so that the program will continue in the presence of no data. Also notethe use of the #define for handling exiting from the routine in the presence of an error. In general, this is a convenientmethod to simplify the code if the client makes several API calls, since the return code for Commodity DataServer C/C++ API routines should always be checked and the exit code is usually the same for all routines: XmimDisconnectmust always be called before exiting and printing the error message is generally desirable.

Examining the Columns of a RelationThe example above illustrated how to browse the relation hierarchy. The next step is to examine, in detail, a particularrelation. For example, we may want to find out which columns the relation has (e.g., Open, High, Low, Close), or wemay want to extract the data for a given set of columns. The next example shows how we can determine the columnsin a relation, using the XmimGetRelColumns routine. XmimGetRelColumns provides return arguments with the

80


number of columns for a given relation (numColumns argument) and the names of all the columns (colNamesargument). Its synopsis is as follows:XmimReturnCode XmimGetRelColumns (XmimClientHandle handle, XmimString relName, int *numColumns, XmimString **colNames);XmimReturnCode XmimVaGetRelColumns (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_COLUMN_ARRAY, &numColumns, &colNames, XMIM_END_ARGS);

Note the use of XMIM_COLUMN_ARRAY in the XmimVaGetRelColumns routine to return the number ofcolumns and the column names.

This example program also uses the XmimGetDataRange routine and prints the resulting daily and intraday tick dataranges.#include "xmim_api.h"#include <stdio.h>#define EXIT \XmimPrintError(NULL); XmimDisconnect(handle); return 1main (int argc, char **argv){ XmimClientHandle handle; int i; int num_relname XmimString *colnames; XmimDate daily_from; XmimDate daily_to; XmimDate intraday_from; XmimDate intraday_to; if (argc != 2) { fprintf (stderr, "Usage: %s relation\n", argv[0]); return 1; } if (XmimConnect ("hostname", 0, &handle) != XMIM_SUCCESS) { XmimPrintError("XmimConnect"); return 1; } if (XmimGetRelColumns (handle, argv[1], &num_columns, &colnames) != XMIM_SUCCESS) { EXIT; } for ( i = 0; i < num_columns; i++) { if(XmimVaGetDataRange (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, argv[i], XMIM_COLUMN, colnames[i], XMIM_UNITS, XMIM_DAYS, XMIM_FROM_DATE, &daily_from, XMIM_TO_DATE, &daily_to, XMIM_END_ARGS) == XMIM_ERROR || XmimVaGetDataRange (XMIM_CLIENT_HANDLE, handle,


81

XMIM_RELATION, argv[i], XMIM_COLUMN, colnames[i], XMIM_UNITS, XMIM_MINUTES, XMIM_FROM_DATE, &intraday_from, XMIM_TO_DATE, &intraday_to, XMIM_END_ARGS) == XMIM_ERROR) { EXIT; } printf ("%-10s ", colnames[i]); if (XmimDateCmp (daily_from, XMIM_INVALID_DATE) != 0) printf ("(daily (%02d/%02d/%02d-%02d/%02d/%02d) ", daily_from.month, daily_from.day, daily_from.year%100, daily_to.month, daily_to.day, daily_to.year%100); if (XmimDateCmp (intraday_from, XMIM_INVALID_DATE) != 0) printf ("intraday (%02d/%02d/%02d-%02d/%02d/%02d) ", intraday_from.month, intraday_from.day, intraday_from.year%100, intraday_to.month, intraday_to.day, intraday_to.year%100); printf ("\n"); } XmimDisconnect (handle); return 0;}

Retrieving Data Records

Accessing Data for a RelationNext, we show how to write a simple program that returns the bars for a particular relation. There are several ways ofaccessing the data for a relation. We start with the routine XmimGetRecords, with the following synopsis:XmimReturnCode XmimGetRecords (XmimClientHandle handle, int numRelations, XmimString *relNames, int numColumns, XmimString *colNames, XmimDate fromDate, XmimDate toDate, XmimTime fromTime, XmimTime toTime, int numUnits, XmimUnits units, XmimFillOption missDataFillOption, XmimSkipAllNaN skipAllNaNRecords, int limit, XmimLimitMode limitMode, int *numRecords, XmimDateTime **dates, float **values);XmimReturnCode XmimVaGetRecords (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION_LIST, relName1, relName2, .. NULL,

82


XMIM_COLUMN_LIST, NULL, XMIM_FROM_DATE, XMIM_INVALID_DATE, XMIM_TO_DATE, XMIM_INVALID_DATE, XMIM_FROM_TIME, XMIM_INVALID_TIME, XMIM_TO_TIME, XMIM_INVALID_TIME, XMIM_UNITS, 1, XMIM_DAYS, XMIM_FILL_OPTION XMIM_FILL_NAN, XMIM_FILL_NAN, XMIM_SKIP_NONE, XMIM_LIMIT, 0, XMIM_LIMIT_BY_RECORD XMIM_CURRENT_TICK_USAGE, XMIM_APPEND_TO_NONE, XMIM_NUM_RECORDS, &numRecords, XMIM_DATE_TIMES, &dates, XMIM_VALUES, &values, XMIM_END_ARGS);

The relNames and colNames arguments specify the relations and columns from which to get data. If no columnsare specified (numColumns is given as 0) then the bars (Open, High, Low, Close) for the relations will be used as thedefault columns. Notice that for the XmimVa version of this routine, the XMIM_COLUMN_LIST argument is shownwith a default value of NULL to indicate that it is an optional argument since the bars will be used by default. Ingeneral, throughout this document, if a list specification in a XmimVa routine is optional, it will be shown with avalue of NULL. The dates (fromDate and toDate) allow the caller to specify the appropriate range for the data; ifthese are set to XMIM_INVALID_DATE (or left as default arguments), then all the data will be retrieved. This rangecan be as small as desired: specifying fromDate to be the same as toDate will retrieve the single record for thatdate. Of course, it is much more efficient to retrieve blocks of data and use the XmimGetRecordIndex routine(described below) to access a particular record. Likewise, the times (fromTime and toTime) allow the caller tospecify the appropriate trading time range over which to retrieve data for each date in the date range; if these are setto XMIM_INVALID_TIME, then data will be retrieved for the entire trading time range.

Note that the finest granularity allowed for the time range is 1 second as milliseconds are not allowed inthe time range specification. Of course, the from/to time arguments only make sense in the context ofretrieving intraday, real tick or millisecond data.

The units and numUnits arguments specify what type of data to retrieve (e.g., 10 minute or 5 day bars). TheXmimUnits type is defined as follows:typedef enum { XMIM_UNITS_INVALID, XMIM_SECONDS, XMIM_MINUTES, XMIM_HOURS, XMIM_DAYS, XMIM_WEEKS, XMIM_MONTHS, XMIM_QUARTERS, XMIM_YEARS, XMIM_MILLISECONDS} XmimUnits;

When XMIM_MILLISECONDS is used, millisecond data is retrieved and aggregated as specified. WhenXMIM_SECONDS is used, real tick data (i.e., tick-by-tick) is retrieved and aggregated as specified. If real tick data is


83

not available, then millisecond data (if available) will be used and aggregated appropriately. When XMIM_MINUTESor XMIM_HOURS is used, intraday tick data will be retrieved and aggregated. If intraday data is not available, then realtick data (if available) will be used and aggregated appropriately. When XMIM_DAYS, XMIM_WEEKS, XMIM_MONTHS,XMIM_QUARTERS or XMIM_YEARS are used, daily data will be retrieved and aggregated into the desired frequency.

When you extract two re-cols that have different trading calendars using XmimGetRecords, then theresulting matrix may contain NaNs. The reason for this is that the filling of NaNs for each rel-col ishandled independently (i.e., when one time series has data for a particular time stamp, but the othertime series does not).

A special type of column referred to as a multi-field column, while available for use with any type of data, is particularlyuseful for real tick data. Multi-field columns have one or more fields defined, which, for real tick data, wouldcorrespond to data fields associated with the same tick transaction. When retrieving data for multi-field columns,individual field names may be specified as:

rnf_column_name:field_name

For example, if Bar were a multi-field column with a field named Open, the specification would be Bar:Open. If the fieldname is unique across all columns in the database such that it can be unambiguously resolved, the field name may bespecified alone, without the multi-field column name as a qualifier. If the multi-field column name itself is specified,data will be retrieved for all fields of the multi-field column.

The holidayFillOption and missDataFillOption allow the caller to specify different behaviors for dealing withthe two types of NaNs or invalid data in the database. For each type of NaN, the caller can ask the database server toreturn the NaNs as part of the answer array, to carry over the previous known value, to use the first following value, orto interpolate using either linear, geometric or logarithmic interpolation. This is specified using the following data type:

typedef enum { XMIM_FILL_INVALID, /* Use Default setting XMIM_FILL_NAN, /* Use NaN for missing value XMIM_FILL_FORWARD, /* Use first previous closing value XMIM_FILL_BACKWARD, /* Use first following opening value XMIM_FILL_INTERP_LIN, /* Interpolate linearly XMIM_FILL_INTERP_GEO, /* Interpolate geometrically XMIM_FILL_INTERP_LOG, /* Interpolate logarithmically} XmimFillOption;

When XMIM_FILL_FORWARD is specified, the last closing value is used for the entire bar; that is, open, high, low andclose are all set to the previous close. Likewise, when XMIM_FILL_BACKWARD is specified, the next opening value isused for the entire bar. When any of the interpolate fill options are specified, the interpolation is performed for eachrelation column resulting in a new (interpolated) bar.

An additional option available to the user for handling NaNs is to skip entire rows of NaNs in the answer array. TheskipAllNaNRecords argument is used for this purpose and is specified using the following data type:

typedef enum { XMIM_SKIP_INVALID, /* Use default setting XMIM_SKIP_ALL_NAN, /* Skip rows consisting of all NaNs

84


XMIM_SKIP_NONE /* Do not skip NaNs} XmimSkipAllNaN;

If this argument is set to XMIM_SKIP_ALL_NAN, the answer array will be condensed by eliminating rows that consistentirely of NaNs.

Note that this will apply for both types of NaNs. The XmimVa default behavior will be to use NaNs in theanswer array for both holidays and missing data. For tick data, rows of NaNs will be skipped, however,they will not be for daily data.

The limit and limitMode allow the caller to limit the result to a certain amount of memory or number of records.If this amount is exceeded, the results will be cut off such that the least recent data is retrieved. Depending on themode, limit specifies either the number of kilobytes of memory or the number of records. The XmimLimitMode datatype is defined using the following enum:

typedef enum { XMIM_LIMIT_INVALID, /* Use default setting XMIM_LIMIT_BY_RECORDS, /* Skip rows consisting of all NaNs XMIM_LIMIT_BY_MEMORY /* Do not skip NaNs} XmimLimitMode;

The XmimVaGetRecords routine has an extra argument not available with the non-Va version. This argument,XMIM_CURRENT_TICK_USAGE, is used to indicate whether historic data only should be retrieved or whetherdata from the high-frequency current tick store (see “High Frequency Updating”) should also be retrieved. Thecorresponding data type for this argument is:

typedef enum { XMIM_APPEND_TO_ALL, /* Append current tick to historic XMIM_APPEND_TO_NONE, /* Do not use current tick XMIM_APPEND_TO_DAILY, /* Append current tick to daily data XMIM_APPEND_TO_TICK /* Append to intraday and real tick} XmimCurrentTickUsage;

The default is XMIM_APPEND_TO_NONE such that data from the current tick store will not be utilized. IfXMIM_APPEND_TO_ALL is specified and the ending date/time range specified extends beyond what is available inthe historic database, then data from the current tick store (if available) will be aggregated into the appropriate unitsand appended to the historic data. XMIM_APPEND_TO_DAILY and XMIM_APPEND_TO_TICK can be used to specifythat current tick data should be appended only to daily data retrieval or only to tick (both intraday and real tick) dataretrieval, respectively. Thus, if historic data is available ending yesterday, the current tick facility is being updated via atick feed, daily bars are retrieved with the default toDate and XMIM_APPEND_TO_ALL or XMIM_APPEND_TO_DAILYis specified, the result will include historic daily data with an additional daily bar constructed from today's data storedso far in the current tick facility. Likewise, for intraday or real tick data retrieval, the current tick store will be accessed(if so specified) and the data stored so far will be aggregated and appended to the historic data.

Note that since the high-frequency data store is optimized for fast data storage at the expense of slowerdata retrieval, if appending from current tick storage is indicated, the data retrieval will be much slowerthan if only historical data is retrieved (historical data is optimized for quick retrieval).

85

Finally, the dates and values arrays are used to return the records found. A record consists of all the values for allspecified columns of all specified relations for a given date and time. So, the number of records (numRecords) willbe exactly the number of entries in the dates array. Each record corresponds to numRelations times numColumnsfloating-point values. Therefore, there are many more entries in the values array than in the dates array. In somecases, these numbers will not be known at compile time, for example when the relations are read in from a file. Inthese cases, the calling program will have to perform index arithmetic on the values array. In most cases, however,this information is previously known, so the client can hide these details using C structs as shown in the followingexample, which prints the daily bars of a relation.

#include "xmim_api.h"#include <stdio.h>main (int argc, char **argv){ XmimClientHandle handle; int i; XmimDate from_date; int num_records; XmimDateTime *dates; struct { float open; float high; float low; float close; } *values; if (argc != 2) { fprintf (stderr, "Usage: %s relation\n", argv[0]); return 1; } if (XmimConnect ("hostname", 0, &handle) != XMIM_SUCCESS) { XmimPrintError("XmimConnect"); return 1; } from_date.month = 1; from_date.day = 1; from_date.year = 1994; if (XmimVaGetRecords (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, argv[1], XMIM_COLUMN_LIST, "Open", "High", "Low", "Close" NULL, XMIM_FROM_DATE, from_date, XMIM_NUM_RECORDS, &num_records, XMIM_DATE_TIMES, &dates, XMIM_VALUES, (float **) &values, XMIM_END_ARGS) != XMIM_SUCCESS) { XmimPrintError("XmimVaGetRecords"); XmimDisconnect (handle); return 1; } for (i = 0; i < num_records; i++) { XmimPrintDateTime (dates[i], False, False); if (XmimTestNaN (handle, values[i].open)) printf (", NaN"); else printf (", %8.3f", values[i].open); if (XmimTestNaN (handle, values[i].high)) printf (", NaN");

86


else printf (", %8.3f", values[i].high); if (XmimTestNaN (handle, values[i].low)) printf (", NaN"); else printf (", %8.3f", values[i].low); if (XmimTestNaN (handle, values[i].close)) printf (", NaN"); else printf (", %8.3f", values[i].close); printf ("\n"); } XmimDisconnect (handle); return 0;}

Notice that XMIM_FROM_DATE was set to 1/1/94, but XMIM_TO_DATE was left unspecified, so the code returnsall values since the beginning of 1994 (historic data only since appending from the current tick facility was notspecified). Also notice how XmimTestNaN was used to test the returned values, to make sure that they were valid.The synopses for XmimTestNaN and related routines are as follows:

XmimBoolean XmimTestNaN (XmimClientHandle handle, float value);XmimBoolean XmimTestNaNType (XmimClientHandle handle, float value, XmimNaNType nanType);XmimReturnCode XmimGetNaN (XmimClientHandle, handle, XmimNaNType nanType, float *value);XmimReturnCode XmimSetNaN (XmimClientHandle, handle, float value, XmimNaNType nanType);

The XmimTestNaN routine returns True if the given value is invalid (NaN) for any reason and False otherwise.Because the Commodity DataServer is able to differentiate between two types of NaNs, namely those resulting frommissing data and those resulting from holidays, the remaining routines take a nanType argument. The XmimNaNTypedata type is defined using the following enum:

typedef enum { XMIM_NAN_TYPE_MISS_DATA, /* NaN due to missing data XMIM_NAN_TYPE_HOLIDAY, /* NaN due to holiday XMIM_NAN_TYPE_ALL /* all NaNs} XmimNaNType;

The XmimTestNaNType routine can be used to check specifically for holiday or missing data invalid values. IfXMIM_NAN_TYPE_ALL is used, the result will be identical to that of XmimTestNaN. The routines XmimGetNaNand XmimSetNaN allow the user to change the default value of either the holiday or the missing data NaN usedby the system. Of course, the client should ensure that the new value of NaN is one that is not used in thedatabase. Moreover, checking a user-defined NaN for equality with a NaN value returned by the system may notwork correctly, because of subtleties in the floating-point normalization hardware. The routine XmimTestNaN (orXmimTestNaNType) should still be used in these cases.

Note how the XmimPrintDateTime routine was used in the example to print the date. This call has the same effectas the following print statement:

printf ("%02d/%02d/%04d", dates[i].month, dates[i].day, dates[i].year);


87

See “Date/Time Utility Routines” for details on this routine as well as variants used to print expiration dates, times anddate/times with default formatting (as in the example) or user-specified formatting.

Accessing a Particular Record

The example above retrieved the data and then proceeded to walk through all the elements in it. Sometimes,it is convenient to access a particular record of the arrays returned by XmimGetRecords. The callXmimGetRecordIndex serves this purpose. Its synopsis is as follows:

int XmimGetRecordIndex (XmimDateTime getDate, int numRecords, XmimDateTime *dates, XmimFillOption fillOption);int XmimVaGetRecordIndex(XMIM_DATE_TIME, getDate, XMIM_NUM_RECORDS, numRecords, XMIM_DATE_TIMES, dates, XMIM_FILL_OPTION, XMIM_FILL_NAN, XMIM_END_ARGS);

The getDate argument specifies the date (and time for tick data) of the desired record. The numRecords and datesarguments are those returned by the XmimGetRecords routine. The fillOption argument is used to determinewhat should be returned by XmimGetRecordIndex in the case where no record exists for the given date/time. Of theavailable ll options, only XMIM_FILL_FORWARD, XMIM_FILL_BACKWARD or XMIM_FILL_NAN are valid in this usageand indicate that the previous record, the next record or -1, respectively, will be returned in the case where no recordexists for the given date/time. For example, to get the data for 4/15/94, the client could execute the following code:

XmimDateTime apr_15 = { 1994, 4, 15, 0, 0, 0, 0 };if (XmimVaGetRecords (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, argv[1], XMIM_COLUMN_LIST, "Open", "High", "Low", "Close", NULL, XMIM_NUM_RECORDS, &num_records, XMIM_DATE_TIMES, &dates, XMIM_VALUES, (float **) &values, XMIM_END_ARGS) != XMIM_SUCCESS) { XmimPrintError ("XmimVaGetRecords"); XmimDisconnect (handle); return 1;}idx = XmimGetRecordIndex (apr_15, num_records, dates, XMIM_FILL_FORWARD);if (idx >= 0) printf ("Close on 4/15/94 is %8.3f\n", values[idx].close);

Note the use of XMIM_FILL_FORWARD, so that if there is no record for 4/15/94, then the record returned is the lastrecord prior to 4/15/94. Also note that the time fields of apr_15 were set to 0. This is important because the clientis expecting XmimGetRecordIndex to ignore the times, since the data is in daily mode. XmimGetRecordIndexis a utility function and, therefore, no connection handle is necessary and the standard return code type is not used,instead, the record index is returned by the function call.

88


Accessing Data for Multiple RelationsThe following example illustrates dealing with multiple relations, as well as using intraday tick data. Nested structsfor values provide a convenient way to access the returned data values using relation and column name specifiers.#include "xmim_api.h"#include <stdio.h>main (int argc, char **argv){ XmimClientHandle handle; int i; XmimDate from_date; int num_records; XmimDateTime *dates; struct { struct { float open; float close; } us, sp; } *values; if (argc != 1) { fprintf (stderr, "Usage: %s\n", argv[0]); return 1; } if (XmimConnect ("hostname", 0, &handle) != XMIM_SUCCESS) { XmimPrintError("XmimConnect"); return 1; } from_date.month = 1; from_date.day = 1; from_date.year = 1994; if (XmimVaGetRecords (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION_LIST, "US", "SP", NULL, XMIM_COLUMN_LIST, "Open", "Close", NULL, XMIM_FROM_DATE, from_date, XMIM_UNITS, 1, XMIM_MINUTES, XMIM_NUM_RECORDS, &num_records, XMIM_DATE_TIMES, &dates, XMIM_VALUES, (float **) &values, XMIM_END_ARGS) != XMIM_SUCCESS) { XmimPrintError("XmimVaGetRecords"); XmimDisconnect (handle); return 1; } for (i = 0; i < num_records; i++) { XmimPrintDateTime (dates[i], True, False); if (XmimTestNaN (handle, values[i].us.open)) printf (", NaN"); else printf (", %8.3f", values[i].us.open); if (XmimTestNaN (handle, values[i].us.close)) printf (", NaN"); else 26 printf (", %8.3f", values[i].us.close); if (XmimTestNaN (handle, values[i].sp.open)) printf (", NaN");

89

else printf (", %8.3f", values[i].sp.open); if (XmimTestNaN (handle, values[i].sp.close)) printf (", NaN"); else printf (", %8.3f", values[i].sp.close); printf ("\n"); } XmimDisconnect (handle); return 0;}

Accessing Data for Relations Specified in a File

In the following example, we show the case when the relations are specified indirectly (i.e., they appear as entries in afile). Since the contents of the file may change over time, the client can not know at compile time how many relationsare being read. Therefore, the client has to do the index arithmetic.

#include "xmim_api.h"#include <stdio.h>main (int argc, char **argv){ XmimClientHandle handle; int i; int j; XmimDate from_date; int num_relnames; XmimString *relnames; int num_records; XmimDateTime *dates; struct { float open; float high; float low; float close; } *values; if (argc != 2) { fprintf (stderr, "Usage: %s relation_file\n", argv[0]); return 1; } if (XmimConnect ("hostname", 0, &handle) != XMIM_SUCCESS) { XmimPrintError("XmimConnect"); return 1; } from_date.month = 1; from_date.day = 1; from_date.year = 1994; if (XmimVaGetRecords (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION_FILE, argv[1], &num_relnames, &relnames, XMIM_COLUMN_LIST, "Open", "High", "Low", "Close", NULL, XMIM_FROM_DATE, from_date, XMIM_NUM_RECORDS, &num_records, XMIM_DATE_TIMES, &dates, XMIM_VALUES, (float **) &values,

90


XMIM_END_ARGS) != XMIM_SUCCESS) { XmimPrintError("XmimVaGetRecords"); XmimDisconnect (handle); return 1; } for (i = 0; i < num_records; i++) { for(j = 0; j < num_relnames; j++) { XmimPrintDateTime (dates[i], False, False); printf (" %-8s", relnames[j]); if (XmimTestNaN (handle, values[i*num_relnames+j].open)) printf (", NaN"); else printf (", %8.3f", values[i*num_relnames+j].open); if (XmimTestNaN (handle, values[i*num_relnames+j].high)) printf (", NaN"); else printf (", %8.3f", values[i*num_relnames+j].high); if (XmimTestNaN (handle, values[i*num_relnames+j].low)) printf (", NaN"); else printf (", %8.3f", values[i*num_relnames+j].low); if (XmimTestNaN (handle, values[i*num_relnames+j].close)) printf (", NaN"); else printf (", %8.3f", values[i*num_relnames+j].close); printf ("\n"); } } XmimDisconnect (handle); return 0;}

Accessing Data with Relations and Columns Specified in Arrays

In the following example, we show the most general case which exemplifies accessing data when neither therelations nor columns are previously known, and, thus, the client must do all the index arithmetic on the values array.The relations and columns appear below in arrays.

#include "xmim_api.h"#include <stdio.h>#include <stdlib.h>main (int argc, char **argv){ XmimClientHandle handle; int i; int j; int k; int idx; XmimDate from_date; int num_relnames; XmimString *relnames; int num_columns; XmimString *columns; int num_records;

91

XmimDateTime *dates; float *values; if (argc != 1) { fprintf (stderr, "Usage: %s\n", argv[0]); return 1; } if (XmimConnect ("hostname", 0, &handle) != XMIM_SUCCESS) { XmimPrintError("XmimConnect"); return 1; } num_relnames = 2; relnames = {XmimString *) malloc (num_relnames * sizeof (XmimString)); relnames[0] = "IBM"; relnames[1] = "DELL"; num_columns = 5; columns = (XmimString *) malloc (num_columns * sizeof (XmimString)); columns[0] = "Open"; columns[1] = "High"; columns[2] = "Low"; columns[3] = "Close"; columns[4] = "Volume"; from_date.month = 1; from_date.day = 1; from_date.year = 1994; if (XmimVaGetRecords (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION_ARRAY, num_relnames, relnames, XMIM_COLUMN_ARRAY, num_columns, columns, XMIM_FROM_DATE, from_date, XMIM_NUM_RECORDS, &num_records, XMIM_DATE_TIMES, &dates, XMIM_VALUES, &values, XMIM_END_ARGS) != XMIM_SUCCESS) { XmimPrintError("XmimVaGetRecords"); XmimDisconnect (handle); return 1; } for ( i = 0; i < num_records; i++) { for (j = 0; ; < num_relnames; j++) { XmimPrintDateTime (dates[i], False, False); printf (" %-8s", relnames[j]); for (k = 0; k < num_columns; k++) { idx = (i*num_relnames*num_columns) + j*num_columns + k; if ( XmimTestNaN (handle, values[idx])) printf (", NaN"); else printf (", %8.3f", values[idx]); } printf ("\n"); } } XmimDisconnect (handle); return 0;}

92


Accessing Data for an Options RelationIn the following example, we demonstrate the use of the routine to access options data. The routine used isXmimGetRecordsOption. The synopsis is given in “Accessing Data Records”. The program takes the relation(option) name as a command-line argument. The arguments to restrict the expiration date range, the strike price rangeand the option type are not specified and, thus, all options with daily data for the given relation and columns will beretrieved.

#include "xmim_api.h"#include <stdio.h>#include <stdlib.h>#define usage \ fprintf (stderr, "Usage: %s %s\n", argv[0], \ "relation")#define EXIT \ XmimPrintError(NULL); XmimDisconnect(handle); return 1main (int argc, char **argv){ XmimClientHandle handle; int i, j; int width; static XmimString cols[] = { "Open", "High", "Low", "Close" }; int numOfCols = sizeof (cols) / sizeof(cols[0]); int num_records; XmimDateTime *dates; XmimDate *expir_dates; XmimOptionType *option_types; float *strikes; float *values; char *host = NULL; if (argc != 2) { usage; return 1; } if (XmimConnect ("hostname", 0, &handle) != XMIM_SUCCESS) { XmimPrintError("XmimConnect"); return 1; } if (XmimVaGetRecordsOption ( XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, argv[1], XMIM_COLUMN_ARRAY, numOfCols, cols, XMIM_NUM_RECORDS, &num_records, XMIM_DATE_TIMES, &dates, XMIM_EXPIRATION_DATES, &expir_dates, XMIM_OPTION_TYPES, &option_types, XMIM_STRIKE_PRICES, &strikes, XMIM_VALUES, &values, XMIM_END_ARGS) != XMIM_SUCCESS) { EXIT; } /* Print out headings using the column names */ width = 10 + 12 + 6 + 10; printf("%*s", width, ""); for (i = 0; i < numOfCols; i++)

93

printf("%10s", cols[i]); printf("\n\n"); /* Print out data records */ for (i = 0; i < num_records; i++) { XmimPrintDateTime (dates[i], False, False); printf(", "); XmimPrintExpDate(expir_dates[i]); printf(", %4s", ( option_types[i] == XMIM_OPTION_CALL ? "Call" : "Put " )); printf (", %8.3f", strikes[i]); for (j = 0; j < numOfCols; j++) if (XmimTestNaN (handle, values[i*numOfCols+j])) printf (", %8s", "NaN"); else printf (", %8.3f", values[i*numOfCols+j]); printf("\n"); } printf("\nTotal number of records is: %10d\n", num_records); XmimDisconnect (handle); return 0;}

Writing Data to a FileAnother way of extracting data from the Commodity DataServer database is to use the API equivalent of the BMIMfacts_write command. This is the method that must be used when the client wishes the server to write the datadirectly to a file. Its synopsis is as follows:XmimReturnCode XmimWriteFacts (XmimClientHandle handle, XmimString fileName, int numRelations, XmimString *relNames, int numColumns, XmimString *colNames, int numFields, XmimFieldFormat *fieldFormats, int numUnits, XmimUnits units, XmimDateTime unitsFrom, XmimDate fromDate, XmimDate toDate, XmimTime fromTime, XmimTime toTime);XmimReturnCode XmimVaWriteFacts (XMIM_CLIENT_HANDLE, handle, XMIM_FILENAME, NULL, XMIM_RELATION_LIST, relName1, relName2, ..., NULL, XMIM_COLUMN_LIST, NULL, XMIM_FORMAT_LIST, NULL, XMIM_UNITS, 1, XMIM_DAYS, XMIM_DATE_TIME, XMIM_INVALID_DATE_TIME, XMIM_FROM_DATE, XMIM_INVALID_DATE,

94


XMIM_TO_DATE, XMIM_INVALID_DATE, XMIM_FROM_TIME, XMIM_INVALID_TIME, XMIM_TO_TIME, XMIM_INVALID_TIME, XMIM_END_ARGS);

The fileName argument specifies the output file. If the fileName is not specified (or specified as NULL), thenstdout will be used. The next four arguments specify the relations and columns for which data is to be written.If no columns are specified, then Open, High, Low and Close will be used as the default columns. The last fourarguments can be used to narrow the range of data that is written; if these are set to XMIM_INVALID_DATE andXMIM_INVALID_TIME or left as defaults, then all the data will be written.

The units and numUnits arguments give the time period over which to aggregate the data (e.g., 15 minute or1 week bars). When the units specified is milliseconds, millisecond data will be used (i.e., aggregated into thedesired frequency and written out). When the units specified is seconds, tick-by-tick (second) data will be used; ifno second data exists, millisecond data (if available) will be aggregated into the desired frequency. When the unitsspecified is minutes or hours, intraday tick data will be used; if no intraday data exists, second data (if available) willbe aggregated into the desired frequency; if no second data exists, millisecond data (if available) will be aggregatedinto the desired frequency. Since the usage of intraday data will be preferred for aggregating into minutes or hours,if both tick-by-tick data and intraday data exist and the use of tick-by-tick is desired, the units must be specified interms of seconds (e.g., 120 seconds in lieu of 2 minutes). Likewise, for aggregating into seconds, if both second andmillisecond data exist and the use of millisecond is desired, the units must be specified in terms of milliseconds(e.g., 2000 milliseconds in lieu of 2 seconds). When the units specified is days, weeks, months, quarters or years,daily data will be aggregated appropriately. The unitsFrom argument can be either a time (for tick data) or a date(for daily data) and provides a point of reference for the desired aggregation period. For example, if the unitsspecification is 1 hour and the unitsFrom is 8:30, the aggregation will be hourly ending on the half hour. Thus,the date or time given corresponds to the end of the aggregation period and dates or times emanate from it bothbackwards and forwards. If the unitsFrom argument is set to XMIM_INVALID_DATE_TIME then the CommodityDataServer system defaults will be used (e.g., on the hour for hourly, Monday to Friday for weekly, last day of the yearfor yearly).typedef enum { XMIM_FIELD_DATE, /* Date Field */ XMIM_FIELD_TIME, /* Time Field */ XMIM_FIELD_RELATION, /* Relation Field */ XMIM_FIELD_COLUMN, /* Column Field */ XMIM_FIELD_EXP_DATE, /* Expiration Date Options Field */ XMIM_FIELD_OPT_TYPE, /* Options Type Options Field */ XMIM_FIELD_STRIKE, /* Strike Price Options Field */ XMIM_FIELD_FILLER, /* Filler (Ignore) Field */} XmimFieldType;typedef struct { XmimFieldType field; /* type of field written out */ short width; /* number of characters in field */ short decimal; /* number of decimal digits printed */} XmimFieldFormat;

The field value is used to specify what type of field is written out. The XMIM_FIELD_FILLER type indicates blankspaces; the XMIM_FIELD_DATE and XMIM_FIELD_TIME types indicate the strings corresponding to date and time;the XMIM_FIELD_RELATION type indicates the string corresponding to a relation name; the XMIM_COLUMN_FIELDtype indicates actual data values. For fields of type column, the corresponding entry in the column list specifies whichcolumn to print data for. The XMIM_FIELD_EXP_DATE, XMIM_FIELD_OPT_TYPE and XMIM_FIELD_STRIKE types


95

are used for options relations. The XMIM_FIELD_EXP_DATE type indicates the string corresponding to the expirationdate; XMIM_FIELD_OPT_TYPE indicates the string corresponding to the options field type (i.e., call or put, c or p);the XMIM_FIELD_STRIKE type indicates the actual data value for the options strike price and is specified as theXMIM_FIELD_COLUMN types are. All three of these types must be specified if the relation being written is of typeoptions.

The width member of XmimFieldFormat specifies the number of characters in the corresponding fieldfor the column, relation, filler, option type or strike price field types. For fields of type date, time or expirationdate, width is used to identify the appropriate format from the XmimDateFormat, XmimTimeFormat orXmimExpDateFormat enumerations. These enumerations are listed at the end of this section. The decimal memberspecifies the number of digits to print after the decimal points for the appropriate column. The decimal memberis only applicable for the column and strike price field types since they are the only types with data values. Theconstants XMIM_FORMAT_DATE_FIELD, XMIM_FORMAT_TIME_FIELD, XMIM_FORMAT_RELATION_FIELD,XMIM_FORMAT_COLUMN_FIELD, and XMIM_FORMAT_FILLER_FIELD can be used when the clientwishes to default the width and decimal values. The constants XMIM_FORMAT_EXP_DATE_FIELD,XMIM_FORMAT_OPT_TYPE_FIELD, and XMIM_FORMAT_STRIKE_PRICE_FIELD can be used as defaults for optionsfields.

If no fieldFormats are specified, the API assumes that the field formats consist of a default column field entry foreach column in colNames, with a default date field and, if necessary, default time and relation fields, in front. Forexample, the call:XmimVaWriteFacts (XMIM_CLIENT_HANDLE, handle, XMIM_FILENAME, "ibm.prn", XMIM_RELATION, "IBM", XMIM_COLUMN_LIST, "Open", "High", "Low", "Close", NULL XMIM_END_ARGS);

will write the bars of IBM, to the file ibm.prn. The format will consist of the date, followed by values for Open, High,Low, and Close. Similarly, the call:XmimVaWriteFacts (XMIM_CLIENT_HANDLE, handle, XMIM_FILENAME, "ibm_dell.prn", XMIM_RELATION_LIST, "IBM", "DELL", NULL, XMIM_COLUMN_LIST, "Open", "High", "Low", "Close", NULL XMIM_END_ARGS);

will write the data for IBM and DELL in the format date, relation, then Open, High, Low, Close.

The same result could have been achieved by specifying the name of the multi-field column in the column list insteadof the field names for that column. As is the case in XmimGetRecords, if a field name is not unique, it must bequalified by prepending the multi-field column name followed by a colon (:) character.

The next example shows how to use the fieldFormats structure.XmimFieldFormat fieldFormats[] = { { XMIM_FIELD_DATE, XMIM_DATE_yymmdd, 0 }, { XMIM_FIELD_COLUMN, 8. 3 }, { XMIM_FIELD_COLUMN, 8, 3 }, { XMIM_FIELD_COLUMN, 8, 3 }, { XMIM_FIELD_COLUMN, 8, 3 }, { XMIM_FIELD_COLUMN, 8, 0 } };XmimString ohlcv[] = { "Open", "High", "Low", "Close", "Volume" };if (XmimVaWriteFacts (XMIM_CLIENT_HANDLE, handle,

96


XMIM_FILENAME, "djia.prn", XMIM_RELATION, "DJIA", XMIM_COLUMN_ARRAY, 5, ohlcv, XMIM_FORMAT_ARRAY, 6, fieldFormats, XMIM_END_ARGS) != XMIM_SUCCESS) { XmimPrintError("XmimVaWriteFacts"); XmimDisconnect (handle); return 1;}

Note the use of XMIM_COLUMN_ARRAY in the XmimVaWriteFacts routine. The XmimVa array argument passingstyle is most useful when a group of arguments, such as Open, High, Low, Close, Volume, are commonly passed byseveral routines. Note also how the constant from the XmimDateFormat enumeration is used as the width memberof fieldFormats to specify the desired date format. The corresponding integer value could also have been used (9in this case)9.

If the XMIM_FORMAT_FILE specification is used, then the fieldFormats are read in from the given file. The filemust have one format per line where each format consists of the field name (i.e., date, time, relation,column, filler, expir_day, option_type or strike_price)10 followed by two integers, the firstspecifying the width and the second specifying the number of decimal place digits. In the case of date, time andexpir_day, the width value corresponds to one of the constants from the appropriate enumeration and the decimalfield need not be specified. Commas should be used as delimiters. For example, the file specifications corresponding tothe fieldFormat array used above would be:date, XMIM_DATE_yymmddcolumn, 8, 3column, 8, 3column, 8, 3column, 8, 3column, 8, 0

The XmimSetDateFormat, XmimSetExpDateFormat and XmimSetTimeFormat routines can be used to set thedate, expiration date and time formats used by XmimWriteFacts in lieu of setting them using fieldFormats. ThefieldFormats specification will, of course, override the formats set by these routines. If no specification at all isgiven for the date, expiration date and time formats, the system defaults will be used. The synopses for these routinesare given below along with the date, expiration date and time format enumerations.XmimReturnCode XmimSetDateFormat (XmimClientHandle handle, XmimDateFormat dateFormat);XmimReturnCode XmimGetDateFormat (XmimClientHandle handle, XmimDateFormat *dateFormat);XmimReturnCode XmimSetExpDateFormat (XmimClientHandle handle, XmimExpDateFormat dateFormat);XmimReturnCode XmimGetExpDateFormat (XmimClientHandle handle, XmimExpDateFormat *dateFormat);XmimReturnCode XmimSetTimeFormat (XmimClientHandle handle, XmimTimeFormat timeFormat);XmimReturnCode XmimGetTimeFormat (XmimClientHandle handle, XmimTimeFormat *timeFormat);typedef enum { XMIM_DATE_mmsddsyy = 1, /* 03/10/84 */

9 Notice that the enumerations for date, time and expiration date all have 1 as the first value instead of 0.10 The constant names from the XmimFieldType enumeration can be used in lieu of the field names shown (e.g., XMIM_FIELD_COLUMN instead of column).


97

XMIM_DATE_mmsddsyyyy, /* 03/10/1984 */ XMIM_DATE_ddsmmsyy, /* 10/03/84 */ XMIM_DATE_ddsmmsyyyy, /* 10/03/1984 */ XMIM_DATE_mm_dd_yy, /* 03-10-84 */ XMIM_DATE_mm_dd_yyyy, /* 03-10-1984 */ XMIM_DATE_dd_mm_yy, /* 10-03-84 */ XMIM_DATE_dd_mm_yyyy, /* 10-03-1984 */ XMIM_DATE_yymmdd, /* 840310 */ XMIM_DATE_yyyymmdd, /* 19840310 */ XMIM_DATE_mon_dd_yy, /* Mar 10, 84 */ XMIM_DATE_mon_dd_yyyy, /* Mar 10, 1984 */ XMIM_DATE_monp_dd_yy, /* Mar. 10, 84 */ XMIM_DATE_monp_dd_yyyy, /* Mar. 10, 1984 */ XMIM_DATE_month_dd_yy, /* March 10, 84 */ XMIM_DATE_month_dd_yyyy, /* March 10, 1984 */ XMIM_DATE_dd_mon_yy, /* 10-Mar-84 */ XMIM_DATE_dd_mon_yyyy, /* 10-Mar-1984 */ XMIM_DATE_dd_month_yy, /* 10-March-84 */ XMIM_DATE_dd_month_yyyy, /* 10-March-1984 */} XmimDateFormat;

typedef enum { XMIM_DATE_mmsddsyy = 1, /* 03/10/84 */ XMIM_DATE_mmsddsyyyy = 2, /* 03/10/1984 */ XMIM_DATE_ddsmmsyy = 3, /* 10/03/84 */ XMIM_DATE_ddsmmsyyyy = 4, /* 10/03/1984 */ XMIM_DATE_mm_dd_yy = 5, /* 03-10-84 */ XMIM_DATE_mm_dd_yyyy = 6, /* 03-10-1984 */ XMIM_DATE_dd_mm_yy = 7, /* 10-03-84 */ XMIM_DATE_dd_mm_yyyy = 8, /* 10-03-1984 */ XMIM_DATE_yymmdd = 9, /* 840310 */ XMIM_DATE_yyyymmdd = 10, /* 19840310 */

XMIM_DATE_mon_dd_yy = 11, /* Mar 10, 84 */ XMIM_DATE_mon_dd_yyyy = 12, /* Mar 10, 1984 */ XMIM_DATE_monp_dd_yy = 13, /* Mar. 10, 84 */ XMIM_DATE_monp_dd_yyyy = 14, /* Mar. 10, 1984 */ XMIM_DATE_month_dd_yy = 15, /* March 10, 84 */ XMIM_DATE_month_dd_yyyy = 16, /* March 10, 1984 */ XMIM_DATE_dd_mon_yy = 17, /* 10-Mar-84 */ XMIM_DATE_dd_mon_yyyy = 18, /* 10-Mar-1984 */ XMIM_DATE_dd_month_yy = 19, /* 10-March-84 */ XMIM_DATE_dd_month_yyyy = 20 /* 10-March-1984 */

} XmimDateFormat;

typedef enum { XMIM_TIME_hh_mm_am = 1, /* 9:05am */ XMIM_TIME_hh_mm_AM = 2, /* 11:15AM */ XMIM_TIME_hhmm_am = 3, /* 905am */ XMIM_TIME_hhmm_AM = 4, /* 1115AM */ XMIM_TIME_hh_mm = 5, /* 09:05 */

98


XMIM_TIME_hhmm = 6, /* 2315 */ XMIM_TIME_hh_mm_ss_am = 7, /* 9:05:01am */ XMIM_TIME_hh_mm_ss_AM = 8, /* 11:15:01AM */ XMIM_TIME_hhmmss_am = 9, /* 90501am */ XMIM_TIME_hhmmss_AM = 10, /* 111501AM */ XMIM_TIME_hh_mm_ss = 11, /* 09:05:01 */ XMIM_TIME_hhmmss = 12, /* 231501 */ XMIM_TIME_hh_mm_ss_uuu_am = 13 /* 9:05:01 123 am */ } XmimTimeFormat;

For example, to have the Commodity DataServer display date/time formats as 01/31/94 15:09:00 the following callsmust be made:XmimSetDateFormat (handle, XMIM_DATE_mmsddsyy);XmimSetTimeFormat (handle, XMIM_TIME_hh_mm_ss);

Executing Queries

Executing Saved QueriesThe Commodity DataServer C/C++ API allows the client to execute and graph Commodity DataServer queries. Oneway of doing this is to use the API equivalent of the BMIM query_execute and execute_graph commands. TheXmimExecQuery routine is used to execute a standard Commodity DataServer query that has been saved to a file.The results may be saved to a file or printed in the same manner as is done by query_execute (i.e., report title,column headings, . . . , will be printed as specified by the reporting options). Likewise, XmimGraphQuery sends thepostscript commands to graph a saved Commodity DataServer query to a file or sends the graph directly to a printer.For more information on similar code using the BMIM, Commodity DataServer Visual Basic for Applications and theCommodity DataServer Java APIs, see “Executing Saved Queries” on query_execute (BMIM API), “XMIMExecutionand XMIMExecutionDouble” (Commodity DataServer Visual Basic for Applications API) and “MimData.queryExecute”on MimData.queryExecute (Commodity DataServer Java API).

The synopses for these routines are as follows:XmimReturnCode XmimExecQuery (XmimClientHandle, handle, XmimString queryFilename, XmimString outputFilename, int numSubstitutions, XmimSubstitution *substitutions, XmimAppendMode appendMode, XmimBoolean print, XmimVerboseMode verboseMode);XmimReturnCode XmimVaExecQuery (XMIM_CLIENT_HANDLE, handle, XMIM_QUERY_FILE_NAME, queryFilename, XMIM_QUERY_OUT_FILE_NAME, NULL, XMIM_QUERY_SUBSTITUTION_LIST, NULL, XMIM_APPEND_MODE, XMIM_REPLACE, XMIM_PRINT, False, XMIM_QUERY_VERBOSE_MODE, XMIM_QUERY_SILENT, XMIM_END_ARGS);XmimReturnCode XmimGraphQuery (XmimClientHandle handle,


99

XmimString queryFilename, XmimString outputFilename, int numSubstitutions, XmimSubstitution *substitutions, XmimAppendMode appendMode, XmimBoolean print, XmimVerboseMode verboseMode);XmimReturnCode XmimVaGraphQuery (XMIM_CLIENT_HANDLE, handle, XMIM_QUERY_FILE_NAME, queryFilename, XMIM_QUERY_OUT_FILE_NAME, NULL, XMIM_QUERY_SUBSTITUTION_LIST, NULL, XMIM_APPEND_MODE, XMIM_REPLACE, XMIM_PRINT, False, XMIM_QUERY_VERBOSE_MODE, XMIM_QUERY_SILENT, XMIM_END_ARGS);

The queryFilename is the name of the file where the query is stored. The outputFilename, if not NULL, is thename of the file to which the results of the query should be written. The print option specifies whether the outputshould be sent to a printer.

Note that one or both of these options should always be set.

The appendMode specifies whether the output should overwrite or append to outputFilename. The enum isdefined as follows: typedef XmimMergeMode XmimAppendMode;

The verboseMode determines whether the query should be included in the output. The enum is given as follows: typedef enum { XMIM_QUERY_SILENT = 0, /* Do not include query */ XMIM_QUERY_VERBOSE = 1, /* Include Query */ XMIM_QUERY_VERY_VERBOSE = 2 /* Include Query and options */ } XmimVerboseMode;

The substitutions argument is used to give a list of substitutions to be performed before executing the query. Thisallows a query to be repeatedly executed, once for each item in the substitution list. The XmimSubstitutionstructure is defined as follows: typedef struct { char *leftHandSide; /* Item to substitute for */ int numRightHandSides; /* Number of substitutions */ char **rightHandSides; /* Items to substitute in */ } XmimSubstitution;

The leftHandSide element is the string to be substituted for and rightHandSides are the strings which areto be substituted in. The code segment below executes the Commodity DataServer query that is saved in the filemetal_currency.query, once for each future existing under the Metals category and once for each future existingunder the Currencies category. Each of the metal futures are substituted for GC, while each of the currency futures aresubstituted for DM. The convention used where a category name prefaced with a “*" is taken to mean all relations inthat category.

100


XmimSubstitution subs[2]; subs[0].leftHandSide = "GC"; subs[0].numRightHandSides = 1; subs[0].rightHandSides = (XmimString *) malloc (sizeof (XmimString)); subs[0].rightHandSides[0] = "*Metals"; subs[1].leftHandSide = "DM"; subs[1].numRightHandSides = 1; subs[1].rightHandSides = (XmimString *) malloc (sizeof (XmimString)); subs[1].rightHandSides[0] = "*Currencies"; if (XmimExecQuery (handle, "metal_currency.query", "metal_currency.out", 2, subs, XMIM_REPLACE, False, XMIM_QUERY_VERBOSE) != XMIM_SUCCESS) { XmimPrintError("XmimExecQuery"); XmimDisconnect (handle); return 1; }

For convenience, the XMIM_QUERY_SUBSTITUTION_LIST argument used in XmimVaExecQuery consists of inlineexpansions of the substitutions structure. The inline expansions are of the form: leftHandSide, rightHandSide1, rightHandSide2, ..., NULL

Any number of these may be specified with the entire list terminated by NULL. For example, the following call willexecute the Commodity DataServer query in query_file four times, substituting GC and SI for GC, and DM and JYfor DM.XmimVaExecQuery (XMIM_CLIENT_HANDLE, handle, XMIM_QUERY_FILE_NAME, "query_file", XMIM_QUERY_OUT_FILE_NAME, "query_file.out", XMIM_QUERY_SUBSTITUTION_LIST, "GC", "GC", "SI", NULL, "DM", "DM", "JY", NULL, NULL, XMIM_END_ARGS);

The substitutions specification used for XMIM_QUERY_SUBSTITUTION_FILE is handled similarly. Only onesubstitution per line is allowed, each of the form:leftHandSide = rightHandSide1, rightHandSide2, ...

Executing SHOW-WHEN QueriesXmimSelectRecords

Another way to execute a query from the API is to use the XmimSelectRecords routine. This is equivalent to usingthe XmimExecQuery routine to execute a SHOW-WHEN query, but has the distinct advantage of avoiding the use offiles for input and output. The synopsis is as follows:XmimReturnCode XmimSelectRecords (XmimClientHandle handle,


101

int numConds, XmimString *whenConds, int numShowAttrs, XmimString *showAttrs, int beforeRepeat, int afterRepeat, int numExecUnits, XmimUnits execUnits, int *numRecords, XmimBoolean **whenSuccess, XmimDateTime **dates, float **values); XmimReturnCode XmimVaSelectRecords (XMIM_CLIENT_HANDLE, handle, XMIM_CONDITION_LIST, NULL, XMIM_SHOW_ATTR_LIST, showAttr1, showAttr2, ..., NULL, XMIM_SHOW_BEFORE_REPEAT, 0, XMIM_SHOW_AFTER_REPEAT, 0, XMIM_UNITS, 1, XMIM_DAYS, XMIM_FILL_OPTION, XMIM_FILL_NAN, XMIM_FILL_NAN, XMIM_SKIP_NONE, XMIM_CURRENT_TICK_USAGE, XMIM_APPEND_TO_NONE, XMIM_UNION_DATE, 0, XMIM_NUM_RECORDS, &numRecords, XMIM_CONDITION_SUCCESS, &success, XMIM_DATE_TIMES, &dates, XMIM_VALUES, &values, XMIM_END_ARGS);

This routine returns an array of records containing the desired showAttrs for each date/time satisfying thewhenConds. In addition, for each record satisfying the whenConds, it returns the previous beforeRepeat andsubsequent afterRepeat records. The whenSuccess array is used to indicate which records actually satisfied thewhenConds as opposed to those that are included because a repeat factor was specified. Thus, the whenSuccessargument is not needed if the repeat values are zero.

Note that, while the number of conditions can be 0, there has to be at least one show attributespecified.

The Va version of this routine accepts arguments to allow the fill options for holidays and missing data and the skipNaN option to be specified. These arguments are not available in the non-Va version of the routine so the Va versionmust be used if these arguments need to be specified locally.11

The Va version also provides the Union Date option named XMIM_UNION_DATE. When this option is turned on (set to1), the union of two dates is applied. For example, consider the date range for A+B where: “A” ranges from Januaryto June and “B” ranges from March to September. With the Union Date option turned on, the resulting range will befrom January to September.

11 The XmimSetFillOption routine can be used to set these options globally and will be applicable unless local options are specified.

102


The default is to have Union Date (XMIM_UNION_DATE is set to 0) turned off and the intersection of thedates will be applied. The resulting intersection for this example would be March to June.

The Va version also provides another argument which is not available in the non-Va version. This argument,XMIM_CURRENT_TICK_USAGE, can be used to indicate that the historical data to be used for query execution shouldbe supplemented with data from the high-frequency current tick store. The corresponding data type for this argumentis: typedef enum { XMIM_APPEND_TO_ALL = 0, /* to both daily and intraday */ XMIM_APPEND_TO_NONE = 1, /* do not use current tick */ XMIM_APPEND_TO_DAILY = 2, /* current tick to daily only */ XMIM_APPEND_TO_TICK = 3 /* current tick to subday data only */ } XmimCurrentTickUsage;

The default is to use only historical data as this will provide for much faster query execution. (The high-frequencystore, unlike the historical store, is optimized for data storage rather than data retrieval.) XMIM_APPEND_TO_ALLis used to indicate that data be appended from the current tick store for both tick and daily data, whereasXMIM_APPEND_TO_DAILY and XMIM_APPEND_TO_TICK are used to specify that daily data be supplemented but nottick and vice-versa. This is useful since types of data can be mixed in a query.

The following code segment will return the Close and a 5-day average of the Volume of SP when IBM drops 2 ormore points on a Monday. It also returns the values for the days before and after, since the beforeRepeat andafterRepeat arguments are non-zero.int i; XmimString condition = "Close of IBM is down more than 2 and date is Monday"; XmimDateTime *dates; XmimBoolean *whenSuccess; struct { float close; float vol_avg; } *values; int num_values; if (XmimVaSelectRecords (XMIM_CLIENT_HANDLE, handle, XMIM_CONDITION, condition, XMIM_SHOW_ATTR_LIST, "Close of SP", "5 day average of Volume of SP", NULL, XMIM_SHOW_BEFORE_REPEAT, 1, XMIM_SHOW_AFTER_REPEAT, 1, XMIM_CONDITION_SUCCESS, &whenSuccess, XMIM_NUM_RECORDS, &num_values, XMIM_DATE_TIMES, &dates, XMIM_VALUES, (float **) &values, XMIM_END_ARGS) != XMIM_SUCCESS) { XmimPrintError("XmimVaSelectRecords"); XmimDisconnect (handle); return 1;

103

} for (i = 0; i < num_values; i ++) if (whenSuccess[i]) printf ("%02d/%02d/%02d %8.3f %8.3f %8.3f %8.3f %8.3f %8.3f\n", dates[i].month, dates[i].day, dates[i].year%100, values[i-1].close, values[i-1].vol_avg, values[i].close, values[i].vol_avg, values[i+1].close, values[i+1].vol_avg);

While the above code writes the results to standard output (stdout), it could just as easily have saved them to a file.By using the XmimSelectRecords routine to execute a query, the user has complete control over the disposition ofthe results and, thus, the flexibility of writing the output to a file in whatever format is desired.

There are no restrictions on the conditions possible in whenConds. However, there are two restrictions on the possibleshowAttrs. First, the showAttrs should not refer to Bar directly. If the Bar is desired, this can be achieved bygetting the Open, High, Low, and Close separately. Secondly, they should not contain any repeat statements. Theserestrictions ensure that the number of columns returned in the values structure is exactly equal to numShowAttrs sothat the corresponding indices match up.

XmimQueryExecute

The XmimQueryExecute can be used to run queries and retrieve the results programmatically. The completerange of Commodity DataServer query results are available by accessing the report block handles. This code issimilar to the actions of BMIM query_execute (see “Executing Saved Queries”). See “XMIMExecution andXMIMExecutionDouble” for the code equivalent (XMIMExecution) using the Commodity DataServer Visual Basicfor Applications API, and “MimData.queryExecute” for the code equivalent (MimData.queryExecute) using theCommodity DataServer Java API.

The following shows the synopsis:

XmimReturnCode XmimQueryExecute(XmimClientHandle handle, XmimString query, XmimBoolean hasClntMacroP, int multiple, XmimUnits units, XmimDateTime endingDateTime, XmimFillOption holidayFill, XmimFillOption missDataFill, XmimCurrentTickUsage ctdUsage, XmimQueryResultHandle *rsltSetHandle); XmimReturnCode XmimVaQueryExecute (XMIM_CLIENT_HANDLE, handle, XMIM_QUERY, query_string, XMIM_QUERY_FILE_NAME, query_file, XMIM_QUERY_HAS_CLIENT_MACROS, has_macros, /* value is 1 for yes, 0 for no */ XMIM_UNITS, units_multiple, units, XMIM_FILL_OPTION, holidayFill, missDataFill, /* both of type XmimFillOption*/ XMIM_CURRENT_TICK_USAGE, ctdUsage, XMIM_DATE_TIME, unitsFrom, XMIM_QUERY_RESULT_HANDLE, queryResultHandle, XMIM_UNION_DATE, union_date_desired,

104


/*value is 1 for yes, 0 for no */ XMIM_END_ARGS);

Example:XmimReturnCode XmimQueryExecute(XmimClientHandle handle, XmimString query, XmimBoolean hasClntMacroP, int multiple, XmimUnits units, XmimDateTime endingDateTime, XmimFillOption holidayFill, XmimFillOption missDataFill, XmimCurrentTickUsage ctdUsage, XmimQueryResultHandle *rsltSetHandle); XmimReturnCode XmimVaQueryExecute (XMIM_CLIENT_HANDLE, handle, XMIM_QUERY, query_string, XMIM_QUERY_FILE_NAME, query_file, XMIM_QUERY_HAS_CLIENT_MACROS, has_macros, /* value is 1 for yes, 0 for no */ XMIM_UNITS, units_multiple, units, XMIM_FILL_OPTION, holidayFill, missDataFill, /* both of type XmimFillOption*/ XMIM_CURRENT_TICK_USAGE, ctdUsage, XMIM_DATE_TIME, unitsFrom, XMIM_QUERY_RESULT_HANDLE, queryResultHandle, XMIM_UNION_DATE, union_date_desired, /*value is 1 for yes, 0 for no */ XMIM_END_ARGS);

XML Results

XML API RoutinesThere are three API calls that provide XML output:XmimReturnCode XmimExecQueryXML(XmimClientHandle handle, char *query, XmimAppendMode appendMode, XmimBoolean print, XmimVerboseMode verboseMode, int* len, char** val);XmimReturnCode XmimFindRelationsXML(XmimClientHandle handle, char *pattern, XmimBoolean byName, XmimSearchFilter filter, XmimBoolean caseSensitive, char **xmlString);XmimReturnCode XmimGetXMLRelationInfo(XmimClientHandle handle, char *rel,

105

char **xml_string);

XmimExecQueryXML() allows a user to pass in a query and get the output back in an XML format. The followingDocument Type Definition (DTD) describes the basic format of the output: char* dtd = "<!DOCTYPE mimreport [\n" \ "<!ELEMENT mimreport (error | mimanswersets)>\n" \ "<!ELEMENT error (#PCDATA)>\n" \ "<!ELEMENT mimanswersets (mimanswerset+)>\n" \ "<!ELEMENT mimanswerset (mimtitle, mimlabels, mimvariables?, mimrows, mimstatistics?)>\n" \ "<!ELEMENT mimtitle (#PCDATA)>\n" \ "<!ELEMENT mimlabels (mimlabel*)>\n" \ "<!ELEMENT mimvariables (mimvariable*)>\n" \ "<!ELEMENT mimrows (mimrow*)>\n" \ "<!ELEMENT mimstatistics (mimsum?, mimavg?, mimavgpos?, mimavgneg?, mimpctpos?, mimpctneg?, mimmaximum?, mimminimum?, mimstddev?, mimzstat?, mimvariance?)>\n" \ "<!ELEMENT mimlabel (#PCDATA)>\n" \ "<!ELEMENT mimvariable EMPTY>\n" \ "<!ATTLIST mimvariable\n name CDATA #REQUIRED\n value CDATA #REQUIRED>\n" \ "<!ELEMENT mimrow (mimdate, mimtime?, mimvalue+)>\n" \ "<!ELEMENT mimdate (#PCDATA)>\n" \ "<!ELEMENT mimtime (#PCDATA)>\n" \ "<!ELEMENT mimvalue (#PCDATA)>\n" \

"<!ELEMENT mimsum (mimstatvalue+)>\n" \ "<!ELEMENT mimavg (mimstatvalue+)>\n" \ "<!ELEMENT mimavgpos (mimstatvalue+)>\n" \ "<!ELEMENT mimavgneg (mimstatvalue+)>\n" \ "<!ELEMENT mimpctpos (mimstatvalue+)>\n" \ "<!ELEMENT mimpctneg (mimstatvalue+)>\n" \ "<!ELEMENT mimmaximum (mimstatvalue+)>\n" \ "<!ELEMENT mimminimum (mimstatvalue+)>\n" \ "<!ELEMENT mimstddev (mimstatvalue+)>\n" \ "<!ELEMENT mimzstat (mimstatvalue+)>\n" \ "<!ELEMENT mimvariance (mimstatvalue+)>\n" \ "<!ELEMENT mimstatvalue (#PCDATA)>\n" \ "]>\n\n";

XmimFindRelationsXML() finds all the relations that match a given pattern. If byName is set, the search willbe performed on relation names. Otherwise, byName will search the relation descriptions. The filter argumentcan be used to restrict the number of relations returned. Possible values for filter are: XMIM_SEARCH_LEAVES,XMIM_SEARCH_NON_LEAVES, and XMIM_SEARCH_ALL. If caseSensitive is true the search will be case sensitive.Otherwise, case will be ignored while searching. The output is returned in xmlString. The DTD that describes theXML output is listed here:<!DOCTYPE relations [<!ELEMENT relations (name*)><!ELEMENT name (#PCDATA)>]>

Finally, XmimGetXMLRelationInfo() fetches all the metadata associated with a relation along with someinformation about it’s children (if it has any children). The rel argument contains the name of the relation you want

106


information about. The result includes the relation name, description, type, path, children, and columns. The XML forthis routine is described by this DTD: char* dtd = "<!DOCTYPE mim [" \ "<!ELEMENT mim (relation | error)>" \ "<!ELEMENT relation (name, description, type, path, trading-patterns, futures-info, num_children, children?, columns)> " \ "<!ELEMENT name (#PCDATA)> " \ "<!ELEMENT description (#PCDATA)>" \ "<!ELEMENT type (#PCDATA)>" \ "<!ELEMENT path (#PCDATA)>" \ "<!ELEMENT trading-patterns (trading-pattern*)>" \ "<!ELEMENT trading-pattern (trading-pattern-date?, trading-pattern-mask, trading-pattern-mask-int, trading-pattern-time-periods)>" \ "<!ELEMENT trading-pattern-time-periods (trading-pattern-time-period*)>" \ "<!ELEMENT trading-pattern-time-period " \ "(trading-pattern-time-period-begin?, trading-pattern-time-period-end?)>" \ "<!ELEMENT trading-pattern-date (#PCDATA)>" \ "<!ELEMENT trading-pattern-mask (#PCDATA)>" \ "<!ELEMENT trading-pattern-mask-int (#PCDATA)>" \ "<!ELEMENT trading-pattern-time-period-begin (#PCDATA)>" \ "<!ELEMENT trading-pattern-time-period-end (#PCDATA)>" \ "<!ELEMENT futures-info (futures-expiration-date?, futures-first-notice-date?, futures-contract-units, futures-rollover-day?, futures-rollover-policy?)>" \ "<!ELEMENT futures-expiration-date (#PCDATA)>" \ "<!ELEMENT futures-first-notice-date (#PCDATA)>" \ "<!ELEMENT futures-contract-units (#PCDATA)>" \ "<!ELEMENT futures-rollover-day (#PCDATA)>" \ "<!ELEMENT futures-rollover-policy (#PCDATA)>" \ "<!ELEMENT children (child*)>" \ "<!ELEMENT child (name, description, type)>" \ "<!ELEMENT columns (column*)>" \ "<!ELEMENT column (name, description, (units, unit-conversion*, currency, currency-conversion*)?)>" \ "<!ELEMENT units (#PCDATA)>" \ "<!ELEMENT unit-conversion EMPTY>" \ "<!ELEMENT currency (#PCDATA)>" \ "<!ELEMENT currency-conversion EMPTY>" \ "<!ELEMENT error (#PCDATA)>" \ "<!ATTLIST unit-conversion units CDATA #REQUIRED>" \ "<!ATTLIST unit-conversion formula CDATA #REQUIRED>" \ "<!ATTLIST currency-conversion currency CDATA #REQUIRED>" \ "<!ATTLIST currency-conversion formula CDATA #REQUIRED>" \ "]>";

Extra ExamplesBelow is the fast cgi program that simply calls XmimGetXMLRelationInfo() and returns the output. If theconnection ever fails (i.e., the Commodity DataServer server was restarted), it attempts to reconnect before callingXmimGetXMLRelationInfo().#include "cgi-lib.h" #include "html-lib.h"


107

#include "xmim_api.h"XmimClientHandle handle;int main(){ char* path = NULL; LIST *head = NULL; int iterations = 0; if (XmimConnect("localhost", 4, &handle) != XMIM_SUCCESS) { return -1; } while(FCGI_Accept() >= 0) { iterations++; head = cgi_input_parse(); mime_header("text/html"); if (head) { char *rel = find_val(head, "rel"); if (rel) { if (XmimGetXMLRelationInfo(handle, rel, &path) != XMIM_SUCCESS) { if (XmimConnect("localhost", 4, &handle) != XMIM_SUCCESS) { return -1; } XmimGetXMLRelationInfo(handle, rel, &path); } if (path) printf("%s\n", path); } } if (head) list_clear(head); if (iterations > 500) { FCGI_Finish(); return 0; } } XmimDisconnect(handle);

Data Updating

Locking the DatabaseIn this section, we show how the Commodity DataServer C/C++ API can be used to update the data in anCommodity DataServer database. Before any updates can be performed, the client must lock the CommodityDataServer database, using the XmimLockDatabase function. Its synopsis, along with that for unlocking thedatabase, is given below:XmimReturnCode XmimLockDatabase (XmimClientHandle handle); XmimReturnCode XmimVaLockDatabase (XMIM_CLIENT_HANDLE, handle,

108


XMIM_END_ARGS); XmimReturnCode XmimUnlockDatabase (XmimClientHandle handle); XmimReturnCode XmimVaUnlockDatabase (XMIM_CLIENT_HANDLE, handle, XMIM_END_ARGS); XmimReturnCode XmimIsDatabaseLocked (XmimClientHandle handle, XmimBoolean *isLocked); XmimReturnCode XmimVaIsDatabaseLocked (XMIM_CLIENT_HANDLE, handle, XMIM_DB_LOCKED *isLocked); XMIM_END_ARGS);

If the Commodity DataServer can not acquire a write-lock for the database, XmimLockDatabase will fail. TheCommodity DataServer supports a time series level granularity of locking but this is performed transparently to theuser. The use of XmimLockDatabase essentially puts the system to a mode whereby locking on a time seriesbasis will occur as needed. XmimUnlockDatabase serves as the final commit for changes to the database and iswhen rollover computations will be performed. If multiple clients have locked the database, rollover computationswill be performed only after the last lock is released. Both updates to existing time series (e.g., adding or deletingdata for existing relation columns) and updates that require modifications to the DataServer schema will be visibleimmediately following the update operation; unlocking is not necessary for these changes to be available. TheXmimIsDatabaseLocked routine can be used to determine if a database is currently locked or not. These routinesare applicable for locks pertaining to the historical database; for locks specific to the high-frequency data store see“High Frequency Updating”.

Adding Relations, Columns, and Relation ColumnsRelations are added to the database using the XmimAddRelation routine. Columns are added to the database usingthe XmimAddColumn routine and the XmimAddMFColumn routine, depending on whether the column to be addedis a multi-field column or not. Relation columns are added to the database using the XmimAddRelColumn and theXmimAddMFRelColumn routines, depending on whether the relation column to be added is a multi-field relationcolumn or not. “BMIM Command Equivalents” details each of these routines. The following example demonstrateshow to add the components necessary to incorporate real tick options data into the system. A multi-field column withtwo fields, namely Bid and Ask, is used; therefore, the multi-field relation column has two object types specified, onefor each field.#include "xmim_api.h" #include <stdio.h> #define EXIT \ XmimPrintError(NULL); XmimDisconnect(handle); return 1 main (int argc, char **argv) { XmimClientHandle handle; if (argc != 1) { fprintf (stderr, "Usage: %s \n", argv[0]); return 1; } if (XmimConnect ("hostname", 0, &handle) != XMIM_SUCCESS) { XmimPrintError("XmimConnect"); return 1; } if (XmimLockDatabase(handle) != XMIM_SUCCESS) {

109

EXIT; } if (XmimVaAddRelation(XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, "GII.IBM.NYSE", XMIM_PARENT, "TopRelation:Equities:GlobalInsight:Nyse:Tickers:i:ib", XMIM_TYPE, XMIM_REL_NORMAL, XMIM_END_ARGS) != XMIM_SUCCESS) { EXIT; } if (XmimVaAddMFColumn(XMIM_CLIENT_HANDLE, handle, XMIM_COLUMN, "quotes", XMIM_PARENT, "Price", XMIM_FIELD_LIST, "Bid", "Ask", NULL, XMIM_END_ARGS) != XMIM_SUCCESS) { EXIT; } if (XmimVaAddMFRelColumn (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, "GII.IBM.NYSE", XMIM_COLUMN, "quotes", XMIM_TYPE, XMIM_RELCOL_BASE, XMIM_OBJ_TYPE_LIST, XMIM_RELCOL_OBJ_FLOAT, XMIM_RELCOL_OBJ_FLOAT, NULL, XMIM_END_ARGS) != XMIM_SUCCESS) { EXIT; } if (XmimUnlockDatabase(handle) != XMIM_SUCCESS) { EXIT; } XmimDisconnect (handle); return 0; }

The example below illustrates adding a relation where both the trading days and the trading times vary over time and,thus, multiple segments must be defined for the relation. “Adding, Modifying and Deleting Relations” provides detailsregarding establishing trading patterns and defining multiple segments. In this example, three different segments aredefined for the time series. The time series is defined to trade on Monday, Wednesday and Friday from 9:30am to12:00 and from 1:00pm to 2:00pm starting 1/1/1994; on Monday-Saturday from 8am to 6pm starting 1/1/1995; onMonday-Friday from 8:30am to 3pm starting 1/1/1996.

#include <xmim_api.h> #include <stdio.h> #include <stdlib.h> #include <unistd.h> #define EXIT \ XmimPrintError(NULL); XmimDisconnect(handle); return 1

main(int argc, char **argv) { XmimClientHandle handle; char *rel; XmimRelType type; int i; XmimRelSegment segments[3]; XmimTimePeriod time_periods_0[1];

110


XmimTimePeriod time_periods_1[2]; XmimTimePeriod time_periods_2[1]; if (argc != 2) { fprintf(stderr, "Usage: %s rel\n", argv[0]); return 1; } else rel = argv[1]; segments[0].segStartDate = XmimSetDate(1995, 1, 1); segments[0].tradingPattern = (XmimTradingPattern)(XMIM_MTWTF | XMIM_SATURDAY); segments[0].numPeriods = 1; segments[0].timePeriods = time_periods_0; time_periods_0[0].fromTime = XmimSetTime(8, 0, 0); time_periods_0[0].toTime = XmimSetTime(18, 0, 0); segments[1].segStartDate = XmimSetDate(1994, 1, 1); segments[1].tradingPattern = XMIM_MONDAY | XMIM_WEDNESDAY | XMIM_FRIDAY; segments[1].numPeriods = 2; segments[1].timePeriods = time_periods_1; time_periods_1[0].fromTime = XmimSetTime(9, 30, 0); time_periods_1[0].toTime = XmimSetTime(12, 0, 0); time_periods_1[1].toTime = XmimSetTime(14, 0, 0); segments[2].segStartDate = XmimSetDate(1996, 1, 1); segments[2].tradingPattern = XMIM_MTWTF; segments[2].numPeriods = 1; segments[2].timePeriods = time_periods_2; time_periods_2[0].fromTime = XmimSetTime(8, 30, 0); time_periods_2[0].toTime = XmimSetTime(15, 0, 0); if (XmimConnect (NULL, 0, &handle) != XMIM_SUCCESS) { EXIT; } if (XmimLockDatabase (handle) != XMIM_SUCCESS) { EXIT; } if (XmimGetRelType(handle, rel, &type) == XMIM_ERROR) { if (XmimVaAddRelation(XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, rel, XMIM_PARENT, "Indices", XMIM_TYPE, XMIM_REL_NORMAL, XMIM_SEGMENTS, 3, segments, XMIM_END_ARGS) != XMIM_SUCCESS) { EXIT;` } } else fprintf(stderr, "Relation %s already exists.\n", rel); if (XmimUnlockDatabase (handle) != XMIM_SUCCESS) { EXIT; } XmimDisconnect (handle); }

Note that the segments defined do not need to be chronologically ordered. Also notice the check (usingXmimGetRelType) to ensure that the relation does not already exist in the database before adding it.

111

Storing a Vector of ValuesA simple way to update the Commodity DataServer data is to use the XmimPutRecords function, which is theconverse of XmimGetRecords. Its synopsis is as follows:XmimReturnCode XmimPutRecords (XmimClientHandle handle, int numRelations, XmimString *relNames, int numColumns, XmimString *colNames, XmimUnits units, int numRecords, XmimDateTime *dates, float *values); XmimReturnCode XmimVaPutRecords (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION_LIST, relName1, relName2, ..., NULL, XMIM_COLUMN_LIST, NULL, XMIM_UNITS, XMIM_DAYS, XMIM_HISTORIC_ONLY, True, XMIM_NUM_RECORDS, numRecords, XMIM_DATE_TIMES, dates, XMIM_VALUES, values, XMIM_END_ARGS);

This routine should be used to update data in the database for the specified relations when the data is alreadyavailable in memory (i.e., already exists in an array). If no columns are specified, then the bars (Open, High,Low, Close) will be used as default columns. To update millisecond data, units should be specified asXMIM_MILLISECONDS.To update real tick data, units should be specified as XMIM_SECONDS. Likewise, unitsshould be XMIM_MINUTES for intraday tick data and XMIM_DAYS for daily data. The number of records to update isgiven by numRecords and the dates and values supplied in the dates and values arrays. The number of records mustcorrespond to the number of entries in the dates array. If data already exists for a record specified by an entry in thedates array, the effect will be to replace the existing record.

XmimVaPutRecords has an additional argument, XMIM_HISTORIC_ONLY, that is not available in the non-Va version.By default, this argument is True such that data will be added to the historical data store only. If this argument isset to False, data will be updated both in the historical database and the current tick data store (refer to “HighFrequency Updating” for a description of this facility), as appropriate given the corresponding date/time ranges. Thus,if XMIM_HISTORIC_ONLY is specified as False, the current tick facility is being used and the date/time range of thedata being added falls within (or is newer than) the date/time range of the current tick store, the data will be addedthere. Any data being added that is not within the current tick store range will be added to the historic data store. Ifboth the historic database and the current tick store are being updated then both a database lock and a current tickstore lock will be required.

Options data is stored just like a regular relation (relation type=normal). However, options must be storedunderneath a special options relation category. For example, the options for DELL is stored under the DELL.Optionscategory and options for IBM is stored under the IBM.Options category.

All options relations are required to have the following fields:<plist>

112


<type>option_type</type> <strikep>strike_price_with_decimal_price</strikep> <strike>strike_price</strike> <expir>expiration_date</expir></plist>

where:

● option_type is either put or call● strike_price_with_decimal_place adds the decimal to the strike price (e.g., 30.0).● strike_price is a numeric field (e.g., 300)● expiration_date is a date (e.g., 3/17/03)

The property list can include other items (e.g., fields for root code, special settlement) as necessary but the <type,<strike> and <expir> fields are required.

XmimPutRecords can be used to load data into the Commodity DataServer when the data exists in a file that isnot in the ASCII format readable by the Commodity DataServer (e.g., a binary file). For example, the following routinereads data from a binary file. The file is organized as an array of records.load_binary_data (FILE *file, XmimClientHandle handle) { int i; int num_recs; struct stat stat_buf; struct { short year; char month; char day; float open; float high; float low; float close; } file_rec; XmimDateTime *dates; struct bar{ float open; float high; float low; float close; } *values; fstat (fileno (file), &stat_buf); num_recs = (int) stat_buf.st_size / sizeof (file_rec); dates = (XmimDateTime *)malloc (num_recs * sizeof (*dates)); values = (struct bar *)malloc (num_recs * sizeof (*values)); for (i=0; i<num_recs; i++) { fread (&file_rec, sizeof (file_rec), 1, file); dates[i].year = file_rec.year; dates[i].month = file_rec.month; dates[i].day = file_rec.day; values[i].open = file_rec.open; values[i].high = file_rec.high; values[i].low = file_rec.low; values[i].close = file_rec.close;


113

} if (XmimLockDatabase (handle) != XMIM_SUCCESS) { XmimPrintError("XmimLockDatabase"); return 0; } if (XmimVaPutRecords (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, "IBM", XMIM_COLUMN_LIST, "Open", "High", "Low", "Close", NULL, XMIM_NUM_RECORDS, num_recs, XMIM_DATE_TIMES, dates, XMIM_VALUES, values, XMIM_END_ARGS) != XMIM_SUCCESS) { XmimPrintError("XmimVaPutRecords"); return 0; } if (XmimUnlockDatabase (handle) != XMIM_SUCCESS) { XmimPrintError("XmimUnlockDatabase"); return 0; } free (dates); free (values); return 1; }

Loading Data From a FileWhen the data to be added to the Commodity DataServer is stored in a file in an ASCII format supported by theCommodity DataServer, use the API equivalent of the BMIM facts_read command, the XmimReadFacts function.Its synopsis is as follows:XmimReturnCode XmimReadFacts (XmimClientHandle handle, XmimString fileName, XmimString relName, int numColumns, XmimString *colNames, int numFields, XmimFieldFormat *fieldFormats, XmimRecordFormat recordFormat, XmimMergeMode mergeMode, XmimTickMode tickMode); XmimReturnCode XmimVaReadFacts (XMIM_CLIENT_HANDLE, handle, XMIM_FILENAME, fileName, XMIM_RELATION, NULL, XMIM_COLUMN_LIST, NULL, XMIM_FORMAT_LIST, NULL, XMIM_RECORD_FORMAT, XMIM_RECFMT_FREE, XMIM_MERGE_MODE, XMIM_MERGE, XMIM_TICK_MODE, XMIM_TICK_NONE, XMIM_END_ARGS);

The relName specifies the relation for which data is being read. If this is NULL, the Commodity DataServer assumesthe relation is specified as part of the records in the file. This allows data for more than one relation to be read in from

114


a single file. As with XmimWriteFacts, if no columns are specified, Open, High, Low and Close will be used as thedefault columns. The recordFormat specifies the style of formatting used in the file. This is defined as follows: typedef enum { XMIM_RECFMT_FREE = 0, XMIM_RECFMT_FIXED = 1 } XmimRecordFormat;

When XMIM_RECFMT_FREE is used, the Commodity DataServer accepts either commas or whitespace as delimitersand the data is read in exactly as it appears in the file. On the other hand, if XMIM_RECFMT_FIXED is used, therecords are assumed to be in a fixed format. In these cases, the fieldFormats argument must be used to specifyhow to interpret the data read from the file by providing the type, width and, if applicable, number of decimaldigits for each field. For an explanation of the usage of fieldFormats, refer to “Writing Data to a File”. WhenXMIM_RECFMT_FREE is used, the width and decimal members of the fieldFormats are ignored12. When optionsdata is being read, the fieldFormats must include specifications for the expiration date, the option type and thestrike price. When multi-field column data is read, free-format must be used if the multi-field column name is specifiedinstead of the fields.

Note that when some, but not all, of the fields of a multi-field column are read, those fields that are notspecified will be filled with NaNs.

The mergeMode argument specifies whether the data read in is to be merged with any existing data for that relation,or whether it should replace the existing data. The enum is defined as follows: typedef enum { XMIM_RECFMT_FREE = 0, XMIM_RECFMT_FIXED = 1 } XmimRecordFormat;

The tickMode specifies whether daily or tick data is being read. Moreover, if tick data is being read, it indicateswhether the data is millisecond, real tick (second) or intraday data. Real tick will be automatically converted intointraday data by the system, if so specified. The enum is defined as follows: typedef enum { XMIM_TICK_NONE = 0, /* Daily data */ XMIM_TICK_PRE_SAMPLED = 1, /* Intraday data */ XMIM_TICK_BY_TICK = 2, /* Real tick data */ XMIM_TICK_CONVERT = 3, /* Real tick converted to intraday */ XMIM_TICK_MILLISECOND = 4 /* Millisecond data */ } XmimTickMode;

High Frequency UpdatingThe Commodity DataServer C/C++ API also provides a mechanism for high-frequency updating of the tick data.The routines XmimPutRecords and XmimReadFacts pre-process the historical data so that it can be efficiently

12 Note that unexpected results may occur if the filler specification is used with free format since this will result in the corresponding field being totally ignored.


115

retrieved, while minimizing the storage requirements. However, to efficiently facilitate high-frequency updating,the time of insertion into the database needs to be minimized as a first priority. For this purpose, the CommodityDataServer provides the XmimUpdateCurrentTick function, with the following synopsis:XmimReturnCode XmimUpdateCurrentTick (XmimClientHandle handle, XmimDate date, int numRelations, XmimString *relNames, int numColumns, XmimString *colNames, XmimBoolean realTick, int numRecords, XmimTime *times, float *values); XmimReturnCode XmimVaUpdateCurrentTick (XMIM_CLIENT_HANDLE, handle, XMIM_DATE, date, XMIM_RELATION_LIST, relName1, relName2, ..., NULL, XMIM_COLUMN_LIST, NULL, XMIM_REAL_TICK, False, XMIM_NUM_RECORDS, numRecords, XMIM_TIMES, times, XMIM_VALUES, values, XMIM_END_ARGS);

The date argument specifies the date for which tick data is to be updated in the high-frequency data store andthe relNames and colNames give the relations and columns to be updated. As with the XmimPutRecordsand XmimReadFacts routine, if no columns are specified, then Open, High, Low, and Close will be used as thedefault columns. The high frequency updating facility is available both for real tick data and for intraday tick data; therealTick argument indicates which type of tick data is involved (False indicates intraday and True indicates realtick). The number of records to update is given by numRecords and the times and values supplied in the times andvalues arrays. The number of records must correspond to the number of entries in the times array.

An additional routine, XmimUpdateCurrentTickBar, is available for use in cases where the tick data updateconsists of a single relation with a single price value (i.e., there is only a single update record) and it is desired topropagate the update to the entire bar such that when the data is retrieved, bar charts can be constructed. Utilizingthis routine represents a significant optimization for the updating process such that it should always be used if theconditions apply. The synopsis is as follows:XmimReturnCode XmimUpdateCurrentTickBar (XmimClientHandle handle, XmimString relName, XmimDate date, XmimTime time, float value); XmimReturnCode XmimVaUpdateCurrentTickBar (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_DATE, date, XMIM_TIME, time, XMIM_VALUE, value, XMIM_END_ARGS);

The relName argument gives the relation to be updated in the high-frequency data store. The date and timearguments specify the date and time of the update and the value supplies the price value. This routine can be used for

116


updating either real tick or intraday tick data but, in the case of intraday data, the user must make sure to zero out theseconds field (or use XMIM_INVALID_TIME prior to setting the minute and hour fields). It is recommended that toobtain the best possible performance, the non-Va version of this routine be used13.

The Commodity DataServer maintains the data added by XmimUpdateCurrentTick (orXmimUpdateCurrentTickBar) separately from the historical tick data. The store used for this high-frequencydata is referred to in this document as the current tick store. A database lock must be obtained prior to any updates;however, in the case of current tick data updates, a special lock that is specific to the current tick store is providedso that the historical data may remain unlocked. The routines for locking and unlocking the current tick store are asfollows:

XmimReturnCode XmimLockCurrentTick (XmimClientHandle handle); XmimReturnCode XmimVaLockCurrentTick (XMIM_CLIENT_HANDLE, handle, XMIM_END_ARGS); XmimReturnCode XmimUnlockCurrentTick (XmimClientHandle handle); XmimReturnCode XmimVaUnlockCurrentTick (XMIM_CLIENT_HANDLE, handle, XMIM_END_ARGS);

If the Commodity DataServer cannot acquire a write-lock for the current tick store, XmimLockCurrentTick will fail.In general, updates to the current tick store will be immediately visible to any clients reading from the database.

During data retrieval, when the historical tick data and the current tick data intersect, the Commodity DataServeralways uses the historical data, under the assumption that historical tick data has been processed and “cleaned,"whereas the current tick data is being directly accessed from a potentially noisy feed. The client may choose tomigrate the current tick data into the historical tick data. This is accomplished by using the XmimFoldCurrentTickfunction. The synopsis is as follows:

XmimReturnCode XmimFoldCurrentTick (XmimClientHandle handle, XmimDate fromDate, XmimDate toDate, XmimBoolean realTick); XmimReturnCode XmimVaFoldCurrentTick (XMIM_CLIENT_HANDLE, handle, XMIM_FROM_DATE, XMIM_INVALID_DATE, XMIM_TO_DATE, XMIM_INVALID_DATE, XMIM_REAL_TICK, False, XMIM_END_ARGS);

Alternatively, the client may choose to discard the current tick data without folding it into the historical data. This isaccomplished using the XmimClearCurrentTick function, with the following synopsis:

XmimReturnCode XmimClearCurrentTick (XmimClientHandle handle, XmimDate fromDate, XmimDate toDate, XmimBoolean realTick); XmimReturnCode XmimVaClearCurrentTick (XMIM_CLIENT_HANDLE, handle, XMIM_FROM_DATE, XMIM_INVALID_DATE, XMIM_TO_DATE, XMIM_INVALID_DATE,

13 This is contrary to the general recommendation where the preferred usage is of the Va routines. In general, the Va routines allow for more flexibility and easeof change. There is, however, a slight overhead associated with the use of the Va routines and, in this particular case, where where speed is so critical,the non-Va version provides the optimal usage.


117

XMIM_REAL_TICK, False, XMIM_END_ARGS);

The fromDate and toDate arguments used in XmimFoldCurrentTick and XmimClearCurrentTick specify therange of current tick data to fold or delete. If these arguments are left unspecified (in the XmimVa version), then alldata in the current tick store will be folded into the historical database or deleted. To fold or delete a single day's worthof data, the same date should be given for both fromDate and toDate. The XmimFoldCurrentTick will neverreplace existing historical tick data.

Note that, since the XmimFoldCurrentTick routine is updating both historical and current tick data,locks must be obtained on both data stores (using XmimLockDatabase and XmimLockCurrentTick)prior to the use of this routine. However, for use of the XmimClearCurrentTick routine, only thecurrent tick store lock is necessary.

In addition, the XmimFoldCurrentTickSelective routine is provided to facilitate selective folding by symbol ortime series as well as the folding of current tick data into any type of data (real tick, intraday or daily). The synopsis isas follows:XmimReturnCode XmimFoldCurrentTickSelective (XmimClientHandle handle, XmimString relName, int numColumns, XmimString *colNames, XmimDate fromDate, XmimDate toDate, XmimBoolean realTick, XmimBoolean intraday, XmimBoolean daily); XmimReturnCode XmimVaFoldCurrentTickSelective (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, NULL, XMIM_COLUMN_LIST, NULL, XMIM_FROM_DATE, XMIM_INVALID_DATE, XMIM_TO_DATE, XMIM_INVALID_DATE, XMIM_REAL_TICK, False, XMIM_INTRADAY, True, XMIM_DAILY, False, XMIM_END_ARGS);

The relName and colNames arguments are used to specify the desired relation and columns for the relation to befolded. If the relName argument is specified but the colNames argument is not, then all columns of the relationwill be folded. If neither the relName nor the colNames arguments are specified then everything will be foldedaccording to the remaining arguments. If the colNames argument is specified then the relName argument mustalso be specified or an error will be returned. The relName provided must be a leaf-level relation, i.e., it musthave data associated with it. The fromDate and toDate arguments are used as in the XmimFoldCurrentTickroutine to specify the range of data to fold. By default, the entire data range will be folded. The realTick, intraday anddaily arguments are used to indicate the type of historical data which the current tick data should be folded into.Aggregation of data will take place as necessary, i.e., when folding any type of tick data into historical daily data or realtick data into historical intraday data. By default, folding will take place into the intraday historical data (as with theXmimFoldCurrentTick routine).

118


Note that current tick data may be folded into one or more of the historical types of data at the sametime. As with the XmimFoldCurrentTick routine, locks must be obtained for both the historicaldatabase and the current tick store before the XmimFoldCurrentTickSelective routine can beused.

A sample client application illustrating the use of the selective fold capability is provided below. The program candelete all the current tick data stored for a given date range, or it can convert some (or all) of this data into thehistorical database. The date range for folding or clearing, the columns and relations to fold and switches indicatingwhether or not to fold into the daily, intraday and/or real tick historical data stores are given as command linearguments to the program. The program is invoked as follows: hostname% sel_fold [-h] [-s host] [-p server_number] [-e from_date] [-f to_date] [-k] [-a] [-r] [-t] [-d] [-c n col1 ... coln] [-l rel]

where:

-h prints out the usage.

-s specifies the machine, host, where the Commodity DataServer server is running. The default host is the machinewhere the program was invoked.

-p specifies the port number, server_number, of the Commodity DataServer server. The default server number is 0.

-e specifies the start date, from_date, from which to fold/clear. The date is given as mm/dd/yy.

-f specifies the end date, to_date, from which to fold/clear. The date is given as mm/dd/yy.

-k specifies to clear the current tick data store. If this switch is used then all the following switches are ignored.

-a specifies to fold all columns and relations in the current tick data store. If this switch is used then all the followingswitches are ignored.

-r specifies that current tick data should be folded into the real tick historical data.

-t specifies that current tick data should be folded into the intraday tick historical data.

-d specifies that current tick data should be folded into the daily historical data.

-c specifies the columns col1 ... coln to fold where the number of columns is given by n.

-l specifies the relation to fold.

The entire selective fold utility application follows:#include <xmim_api.h> #include <stdio.h> #include <stdlib.h> static char* prog = NULL; #define EXIT \ XmimPrintError(NULL); XmimDisconnect(handle); return 1 void usage() { fprintf (stderr,

119

"Usage: %s %s\n", prog, "[-h] [-s host] [-p svr] [-k] [-a] [-r] [-t] [-d]" "[-c n col1 col2 ... coln] [-e from_date] [-f to_date] [-l rel]"); } XmimDate pass_date(char* s) { int i = 0; int y, m, d; XmimDate rslt = XMIM_INVALID_DATE; char *t = s, *p = s; while (*t != '/' && *t != '\0') t++; if (*t != '/') return rslt; *t = '\0'; m = atoi(p); t++; p = t; while (*t != '/' && *t != '\0') t++; if (*t != '/') return rslt; *t = '\0'; d = atoi(p); t++; p = t; while (*t != '/' && *t != '\0') t++; if (*t != '\0') return rslt; y = atoi(p); if (y < 100) y += 1900; rslt = XmimSetDate(y, m, d); return rslt; } main (int argc, char **argv) { char *host = NULL; int srv_num = 0; char *rel = NULL; int num_cols = 0; XmimString *cols = NULL; XmimDate fromDate = XMIM_INVALID_DATE; XmimDate toDate = XMIM_INVALID_DATE; XmimBoolean real_tick = False; XmimBoolean intraday = False; XmimBoolean daily = False; XmimBoolean fold_all_p = False; XmimBoolean clear_p = False; extern char *optarg; extern int optind; char c; XmimClientHandle handle; int i; prog = argv[0]; fromDate = XmimSetDate(0, 0, 0); toDate = XmimSetDate(0, 0, 0); while ((c = getopt(argc, argv, "hs:p::l:artde:f:c:k")) != -1) {

120


switch (c) { case 'h': usage (); return 0; case 'q': host = optarg; break; case 'p': srv_num = atoi(optarg); break; case 'l': rel = optarg; break; case 'a': fold_all_p = True; break; case 'r': real_tick = True; break; case 't': intraday = True; break; case 'd': daily = True; break; case 'e': fromDate = pass_date(optarg); break; case 'f': toDate = pass_date(optarg); break; case 'c': num_cols = atoi(optarg); cols = (char**)malloc((num_cols + 1)*sizeof(char*)); for (i = 0; i < num_cols; i++) cols[i] = argv[optind+i]; cols[i] = NULL; optind += num_cols; break; case 'k': clear_p = True; break; default: usage(); return 1; } } if (optind < argc) { usage(); return 1; } if (XmimConnect (host, srv_num, &handle) != XMIM_SUCCESS) { EXIT; } if (XmimLockCurrentTick (handle) != XMIM_SUCCESS) { EXIT;


121

} if (clear_p) { if (XmimVaClearCurrentTick(XMIM_CLIENT_HANDLE, handle, XMIM_FROM_DATE, fromDate, XMIM_TO_DATE, toDate, XMIM_END_ARGS) != XMIM_SUCCESS) { EXIT; } } else { if (XmimLockDatabase (handle) != XMIM_SUCCESS) { EXIT; } if (fold_all_p) { if (XmimVaFoldCurrentTick(XMIM_CLIENT_HANDLE, handle, XMIM_FROM_DATE, fromDate, XMIM_TO_DATE, toDate, XMIM_END_ARGS) != XMIM_SUCCESS) { EXIT; } } else { if (XmimVaFoldCurrentTickSelective (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, rel, XMIM_COLUMN_ARRAY, num_cols, cols, XMIM_REAL_TICK, real_tick, XMIM_INTRADAY, intraday, XMIM_DAILY, daily, XMIM_FROM_DATE, fromDate, XMIM_TO_DATE, toDate, XMIM_END_ARGS) != XMIM_SUCCESS) { EXIT; } } if (XmimUnlockDatabase (handle) != XMIM_SUCCESS) { EXIT; } } if (XmimUnlockCurrentTick (handle) != XMIM_SUCCESS) { EXIT; } XmimDisconnect(handle); return (0); }

After all the command line arguments are processed, the program establishes a connection to the CommodityDataServer, locks the current tick database, makes the appropriate API call to either clear the current tick data orperform a selective fold, and then unlocks the current tick database and disconnects from the Commodity DataServer.If folding is performed then the historical database must be locked (and unlocked) as well.

Data from the current tick store can be retrieved using XmimVaGetRecords by setting the argumentXMIM_CURRENT_TICK_USAGE appropriately (see “Accessing Data for a Relation”). The current tick store data can be

122


edited using the XmimVaPutRecords and XmimVaReplaceRecords routines (see “Storing a Vector of Values” and“Replacing and Deleting Data Records”) with the XMIM_HISTORIC_ONLY argument set to False.

All dates for which there is corresponding data in the current tick store can be accessed using theXmimGetCurrentTickDataDates routine. It has the following synopsis:XmimReturnCode XmimGetCurrentTickDataDates (XmimClientHandle handle, int *numDates, XmimDate **dates); XmimReturnCode XmimVaGetCurrentTickDataDates (XMIM_CLIENT_HANDLE, handle, XMIM_NUM_DATES, &numDates, XMIM_DATES, &dates, XMIM_END_ARGS);

The numDates argument gives the number of dates returned and the dates array is used to return the actual datesfound.

Adding Formulas (Dynamic Time Series)The Commodity DataServer C/C++ API also provides a mechanism for creating dynamic rel-columns from formulas.The routines XmimAddFormula and XmimModFormula are used to attach a formula for computing a time series toa particular rel-column that has no data of its own. When a user queries a rel-column that has no data, but it has aformula attached to it, the time series information is computed dynamically to produce the results of the query. For thepurpose of attaching a formula to an empty rel-column, the Commodity DataServer provides the XmimAddFormulaand XmimModFormula functions, with the following synopsis:XmimReturnCode XmimAddFormula(XmimClientHandle handle, char *rel_name, /*must already exist*/ char *formula, /*a query that is the formula*/ int num_columns, char *columns[], /* must already exist as rel-columns */ char *post_process) /* a "post-process" mechanism for the data */XmimReturnCode XmimModFormula(XmimClientHandle handle, char *rel_name, /*must already exist*/ char *formula, /*a query that is the formula*/ int num_columns, char *columns[], /* must already exist as rel-columns*/ char *post_process) /* a "post-process" mechanism for the data */

In the following example, we have added a new rel-column called avg_bar (without data) to NG. Now, we are going tospecify a formula to compute avg_bar.char* columns[1];columns[0] = "avg_bar";XmimAddFormula(handle, "NG", "SHOW 1: (Open of NG + High of NG + Low of NG + Close of NG) / 4", 1, columns, (char*)0)

A single formula can compute more than one time series by having more than one SHOW label. For each time seriesto be computed by a formula query, there would be one column-name in the column_list.


123

In the second example, we have added two new rel-columns for NG: avg_bar and my_mid. Now, we are going tospecify a formula to compute avg_bar and my_mid.char* columns[2];columns[0] = "avg_bar";columns[1] = "my_mid";XmimAddFormula(handle, "NG", "SHOW 1: (Open of NG + High of NG + Low of NG + Close of NG) / 4" " 2: (High of NG + Low of NG ) / 2", 2, columns, (char*)0)

Memory ManagementThe Commodity DataServer C/C++ API routines are designed so that the client usually does not need to worry aboutmemory management. Specifically, it is not necessary to free any memory that the API routines allocate. However, thisimplies that when two API routines are called, it is possible that any pointers returned by the first routine will be freedwhen the second routine is called. When the client wishes to continue using the data returned from the first call aftersubsequent calls are made, the data needs to be copied into local storage, and then freed by the client. This can beaccomplished by using the following routines:void *XmimCopyArray (void *array, int numElements, int sizeElement); void *XmimMallocArray (int numElements, int sizeElement); int XmimFreeArray (void *array);

The XmimMallocArray function allocates space for an array of numElements elements of size sizeElementand initializes the space to zeros. It returns a pointer to the allocated space. XmimMallocArray is analogous to thecalloc C library function. The XmimCopyArray function returns a pointer to a new array that is a copy of the arraygiven as an argument. The routine obtains space for the new array by using XmimMallocArray. The argument toXmimFreeArray is a pointer to an array previously allocated by either XmimMallocArray or XmimCopyArray.In addition to the general memory management functions above, the following functions are provided merely as aconvenience:XmimString *XmimCopyStrings (XmimString *stringArray, int numElements); XmimString *XmimMallocStrings (int numElements); int XmimFreeStrings (XmimString *stringArray, int numElements); float *XmimCopyValues (float *valuesArray, int numValues); float *XmimMallocValues (int numValues); int XmimFreeValues (float *valuesArray); XmimDateTime *XmimCopyDateTimes (XmimDateTime *datesArray, int numDates); XmimDateTime *XmimMallocDateTimes (int numDates); int XmimFreeDateTimes (XmimDateTim *datesArray); XmimBoolean *XmimCopyBooleans (XmimBoolean *booleansArray,

124


int numBooleans); XmimBoolean *XmimMallocBooleans (int numBooleans); int XmimFreeBooleans (XmimBoolean *booleansArray);

These functions can be used for copying string arrays (e.g., an array of relation names), value arrays, date arrays andboolean arrays. In the case of the value, date and boolean arrays, they offer the sole advantage of knowing the size ofan array element and, therefore, obviating the need for the sizeElement argument. In the case of string arrays, eachelement is itself a pointer that needs to be copied and the XmimCopyStrings function provides for this.

So, for example, the following program makes two calls to XmimGetRecords, copying the data values returned fromthe first call into local storage such that the values can be used even after the second call is made.

XmimClientHandle handle; int num_ibm_records; int num_dell_records; XmimDateTime *ibm_dates; XmimDateTime *dell_dates; typedef struct { float open; float high; float low; float close; } valuesStruct; valuesStruct *ibm_values, *dell_values; if (XmimVaGetRecords (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, "IBM", XMIM_COLUMN_LIST, "Open", "High", "Low", "Close", NULL, XMIM_NUM_RECORDS, &num_ibm_records, XMIM_DATE_TIMES, &ibm_dates, XMIM_VALUES, (float **) &ibm_values, XMIM_END_ARGS) != XMIM_SUCCESS) { XmimPrintError("XmimVaGetRecords"); XmimDisconnect (handle); return 1; } ibm_dates = XmimCopyDateTimes (ibm_dates, num_ibm_records); ibm_values = (valuesStruct *) XmimCopyValues (ibm_values, 4*num_ibm_records); if (XmimVaGetRecords (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, "DELL", XMIM_COLUMN_LIST, "Open", "High", "Low", "Close", NULL, XMIM_NUM_RECORDS, &num_dell_records, XMIM_DATE_TIMES, &dell_dates, XMIM_VALUES, (float **) &dell_values, XMIM_END_ARGS) != XMIM_SUCCESS) { XmimPrintError("XmimVaGetRecords"); XmimDisconnect (handle); return 1; }

XmimFreeDateTimes (ibm_dates); XmimFreeValues (ibm_values);

The convenience routines for copying dates and values were used above. For didactic purposes, the correspondingXmimCopyArray calls are provided below:


125

ibm_dates = XmimCopyArray ((void *)ibm_dates, num_ibm_records, sizeof(XmimDateTime)); ibm_values = (valuesStruct *) XmimCopyArray ((void *)ibm_values, 4*num_ibm_records, sizeof(float *));

Data Source Table: A Complete ApplicationThe Table Facility is documented in “Table Facility”. This facility provides for the capability of storing and retrieving non-time series data. The data is organized as relational tables where entries in the table (rows in the table) are referredto as tuples and columns in the table are referred to as fields of the table. The data stored in the Table Facility may ormay not be associated with relations, columns or relation columns stored in the Commodity DataServer database.

In this section, a complete application will be presented that utilizes the Table Facility to store DataSource informationfor relations in the Commodity DataServer database.

Creating/Updating the TableThe following program serves to create a table named DataSource, add tuples into the table and delete tuples fromit. The program uses an input file where each line consists of a relation name followed by the data source. An excerptfrom an input file is shown below:

TopRelation LIM Equities LIM DRIFacs DRI DRIIntl DRI DRIIndex DRI EconomicIndicators MMS International FedReserve FRB of St. Louis

A category relation may be specified and all descendants in the hierarchy will inherit that data source unless thedescendant itself or a closer ancestor is assigned a data source. The program is invoked as follows:

hostname% data_source_load -c input_file

it will read the specified file, create the DataSource table and add the tuples to the table. Alternatively, thedata_source_load program can be given a -d switch and entries specified in the given file will be deleted from thetable. Here, the input file need not specify the data sources, i.e., each line can consist of just a relation. Additionalswitches that can be used include: -h to display help information (usage), -p number to specify the port number ofthe Commodity DataServer (default is 0), -q host to specify the machine where the Commodity DataServer is running(default is the machine where the program was invoked), -t table to specify to use the given name as the name of thetable instead of DataSource.

In the program, first the connection to the Commodity DataServer is established and the database is locked asrequired for updating or creating a table. The table is then either created or, if it already exists, opened for writing.

126


Then the input file is read and tuples are deleted or added, as specified by the command line arguments. After all linesof the input file have been processed, the table is closed, the database unlocked and the client disconnected from theserver. See “Utility Routines” for the utility routines used in the program.

#include <stdio.h> #include <assert.h> #include "xmim_utils.h" #ifdef NEED_GETOPT extern int getopt(int argc, char **argv, char *optstring); extern char *optarg; extern int optind, opterr; #endif #define USAGE \ fprintf (stderr, "Usage: data_source_load [-h] [-q host] [-p server_number] " \ "[-c] [-d] [-t table_name] files\n") main(int argc, char **argv) { int c; char *host = NULL; int server_number = 0; char *table_name = "DataSource"; int create_p = 0; int delete_p = 0; while ((c = getopt(argc, argv, "hcdq:p:t:")) != -1) { switch (c) { case 'h': USAGE = 1; return 0; case 'c': create_p = 1; break; case 'd': delete_p = 1; break; case 'q': host = optarg; break; case 'p': server_number = atoi(optarg); break; case 't': table_name = optarg; break; default: USAGE return 1; } } if (optind >= argc) { USAGE; return 1; } if (!connect_to_xmim (host, server_number))


127

{ fprintf (stderr, "Failed to connect to xmim\n"); 61 return 1; } if (!lock_dbase ()) { disconnect_from_xmim (); return 1; } if (!(create_p ? create_table (table_name) : open_table (table_name, 0))) { unlock_dbase (); disconnect_from_xmim (); return 1; } for (argc -= optind, argv += optind ; argc > 0; argc --, argv ++) { FILE *file; char buf[1024]; printf ("Processing file %s....\n", *argv); file = fopen (*argv, "r"); if (file == NULL) { fprintf (stderr, "Can't open file %s\n", *argv); continue; } while (fgets (buf, sizeof(buf), file)) { char *relation; char *source; char *s; for (s=buf; !isspace (*s); s ++) ; *s = '\0'; relation = buf; if (delete_p) { printf ("Deleting entry for %s\n", relation); del_tuple (relation); continue; } for (s++; isspace (*s); s ++) ; source = s; for (s++; *s != '\0' && *s != '\n'; s ++) ; assert (*s == '\n'); *s = '\0'; printf ("%s\t\"%s\"\n", relation, source); add_tuple (relation, source); } fclose (file); } close_table (); unlock_dbase (); disconnect_from_xmim ();

128


return 0; }

Querying the TableThe following program can be used to query the DataSource table for the source of a particular Commodity DataServerrelation. It is invoked as follows: hostname% data_source CPI EconomicIndicators -> MMS International

The answer returned indicates that “MMS International" is the data source for CPI. Notice that no explicit entry for CPIwas found, so the Commodity DataServer inferred CPI's data source by looking at the Commodity DataServer relationhierarchy until it found an answer. The program can also be used to list all entries in the DataSource table. It does thisif invoked with no arguments. The data_source program uses the -h, -q, -p, and -t switches, which have the samemeanings as when used with the data_source_load program.

In this program, the connection to the server is established and the table opened for reading only. Then the cursoris initialized and the program loops to either get all entries in the table or get all entries matching the relation namegiven at the command line. In either case, all tuples retrieved are printed. After all the tuples have been retrieved, thetable is closed and the client disconnected from the server. See “Utility Routines” for the utility routines used in theprogram.#include <stdio.h> #include "xmim_utils.h" #ifdef NEED_GETOPT extern int getopt(int argc, char **argv, char *optstring); extern char *optarg; extern int optind, opterr; #endif #define USAGE \ fprintf (stderr, "Usage: data_source [-h] [-q host] [-p server_number] " \ "[-t table_name] [relations]\n") main(int argc, char **argv) { int c; char *host = NULL; int server_number = 0; char *table_name = "DataSource"; int create_p = 0; char *relation; char *source; while ((c = getopt(argc, argv, "hq:p:t:")) != -1) { switch (c) { case 'h': USAGE return 0; case 'q': host = optarg; break;


129

case 'p': server_number = atoi(optarg); break; case 't': table_name = optarg; break; default: USAGE; return 1; } } if (!connect_to_xmim (host, server_number)) { fprintf (stderr, "Failed to connect to xmim\n"); return 1; } if (!open_table (table_name, 1)) { disconnect_from_xmim (); return 1; } if (optind >= argc) { if (init_cursor (NULL)) while (get_tuple (&relation, &source)) printf ("%s\t-> %s\n", relation, source); } else { for (argc -= optind, argv += optind ; argc > 0; argc --, argv ++) { if (init_cursor (*argv)) while (get_tuple (&relation, &source)) printf ("%s\t-> %s\n", relation, source); } } close_table (); disconnect_from_xmim (); return 0; }

Utility RoutinesFollowing are the utility routines used by the data_source_load and data_source programs. Included are routines forconnecting and disconnecting from the server, locking and unlocking the database, creating, opening and closing thedata source table, adding and deleting tuples from the table, intitializing (creating) a cursor and getting tuples from thetable. For completeness, the .h file for these routines (xmim_utils.h) is also given.#ifndef _XMIM_UTILS_H #define _XMIM_UTILS_H extern int connect_to_xmim (char *host, int server_number); extern int disconnect_from_xmim (void); extern int lock_dbase (void);

130


extern int unlock_dbase (void); extern int create_table (char *name); extern int open_table (char *name, int read_only_p); extern int close_table (void); extern int add_tuple (char *relation, char *source); extern int del_tuple (char *relation); extern int get_tuple (char **relation, char **source); extern int init_cursor (char *relation); #endif /* _XMIM_UTILS_H */ #include <stdio.h> #include <xmim_api.h> #include "xmim_utils.h" int connect_to_xmim (char *host, int server_number) { XmimClientHandle xmim_handle; if (XmimConnect(host, server_number, &xmim_handle) == XMIM_SUCCESS) { if (!XmimSetDefaultHandle(xmim_handle)) { XmimPrintError (NULL); XmimDisconnect (xmim_handle); return 0; } } else { XmimPrintError(NULL); return 0; } return 1; } int disconnect_from_xmim (void) { if (XmimVaDisconnect (XMIM_END_ARGS) != XMIM_SUCCESS) { XmimPrintError (NULL); return 0; } return 1; }int lock_dbase (void) { fprintf (stderr, "Locking database..."); fflush (stderr); if (XmimVaLockDatabase(XMIM_END_ARGS) != XMIM_SUCCESS) { fprintf (stderr, "failed\n"); fflush (stderr); XmimPrintError(NULL); return 0; } fprintf (stderr, "done\n"); fflush (stderr); return 1; } int unlock_dbase (void)


131

{ fprintf (stderr, "Unlocking database..."); fflush (stderr); if (XmimVaUnlockDatabase(XMIM_END_ARGS) != XMIM_SUCCESS) { fprintf (stderr, "failed\n"); fflush (stderr); XmimPrintError(NULL); return 0; } fprintf (stderr, "done\n"); fflush (stderr); return 1; } static XmimTableHandle table_handle; int create_table (char *name) { static XmimTableFieldDesc table_args[] = { { "Relation", XMIM_TAB_FIELD_REL }, { "DataSource", XMIM_TAB_FIELD_ATOM } }; if (XmimVaCreateTable (XMIM_TABLE_NAME, name, XMIM_FIELD_ARRAY, 2, table_args, XMIM_SINGLE_VALUE, True, XMIM_KEY_FIELD_INDEX, 0, XMIM_TABLE_HANDLE, &table_handle, XMIM_END_ARGS) != XMIM_SUCCESS) { XmimPrintError (NULL); return 0; } return 1; } int open_table (char *name, int read_only_p) { if (XmimVaOpenTable (XMIM_TABLE_NAME, name, XMIM_READ_ONLY, read_only_p, XMIM_TABLE_HANDLE, &table_handle, XMIM_END_ARGS) != XMIM_SUCCESS) { XmimPrintError (NULL); return 0; } return 1; } int close_table (void) { if (XmimVaCloseTable (XMIM_TABLE_HANDLE, table_handle, XMIM_END_ARGS) != XMIM_SUCCESS) { XmimPrintError (NULL); return 0; } return 1; } int add_tuple (char *relation, char *source)

132


{ XmimTableFieldVal tuple[2]; tuple[0].wildcardP = False; tuple[0].value.stringValue = relation; tuple[1].wildcardP = False; tuple[1].value.stringValue = source; if (XmimVaAddTuple (XMIM_TABLE_HANDLE, table_handle, XMIM_TUPLE, tuple, XMIM_END_ARGS) != XMIM_SUCCESS) { XmimPrintError (NULL); return 0; } return 1; } int del_tuple (char *relation) { XmimTableFieldVal tuple[2]; tuple[0].wildcardP = False; tuple[0].value.stringValue = relation; tuple[1].wildcardP = True; tuple[1].value.stringValue = NULL; if (XmimVaDeleteTuple (XMIM_TABLE_HANDLE, table_handle, XMIM_TUPLE, tuple, XMIM_END_ARGS) != XMIM_SUCCESS) { XmimPrintError (NULL); return 0; } return 1; }static XmimCursorHandle table_cursor; int init_cursor (char *relation) { XmimTableFieldVal tuple[2]; int inherit_id; if (relation == NULL) { tuple[0].wildcardP = True; inherit_id = -1; } else { tuple[0].wildcardP = False; inherit_id = 0; } tuple[0].value.stringValue = relation; tuple[1].wildcardP = True; tuple[1].value.stringValue = NULL; if (XmimVaCreateTableCursor (XMIM_TABLE_HANDLE, table_handle, XMIM_TUPLE, tuple, XMIM_INHERIT_FIELD_INDEX, inherit_id, XMIM_CURSOR_HANDLE, &table_cursor, XMIM_END_ARGS) != XMIM_SUCCESS) { XmimPrintError (NULL);


133

return 0; } return 1; } int get_tuple (char **relation, char **source) { XmimTableFieldVal tuple[2]; *relation = NULL; *source = NULL; if (XmimVaGetTuple (XMIM_CURSOR_HANDLE, table_cursor, XMIM_TUPLE, tuple, XMIM_END_ARGS) != XMIM_SUCCESS) { if (XmimVaDeleteCursor (XMIM_CURSOR_HANDLE, table_cursor, XMIM_END_ARGS) != XMIM_SUCCESS) XmimPrintError (NULL); return 0; } *relation = tuple[0].value.stringValue; *source = tuple[1].value.stringValue; return 1; }

The connect_to_xmim function opens a connection to the Commodity DataServer using the host andserver_number arguments that are passed into it. It uses the XmimSetDefaultHandle routine to set the defaulthandle such that for all subsequent API calls where the Va-version of the call is used, the XMIM_CLIENT_HANDLEargument does not need to be specified. The disconnect_from_xmim, lock_dbase and unlock_dbasefunctions simply make the appropriate API calls to disconnect from the Commodity DataServer, lock the CommodityDataServer database and unlock the Commodity DataServer database, respectively.

In the create_table function, the table is established with two fields, the first of type relation and the secondof type atom. The second field is the actual data source and the atom field type is used since there will be a limitednumber of data sources which are expected to be used over and over again. A key is indicated on the first field(relation) since the usage of the data source facility is such that, for retrieval, the relation will most often be specifiedand, therefore, search on that field should be optimized. The single value flag is set to True to indicate that there willbe only a single tuple for each key field, i.e., a single data source for each relation. The table_handle is returned foruse by subsequent functions.

The open_table function can be used to open the table (once it has been created) either for reading only or forwriting as specified by the read_only_p argument that is passed into it. The table_handle is returned for use bysubsequent functions. The close_table function simply closes the table identified by table_handle.

The add_tuple function constructs the tuple to be added to the table from the relation and source character stringspassed into it. It assigns those strings to the value field of the XmimTableFieldVal structures.

Note that, for both fields, the wildcardP member is set to False as its use is not appropriate foradding tuples.

The del_tuple function is used to delete the tuple for the specified relation. The second field is given as a wildcardsuch that all data sources for the given relation will be deleted. In this case, since the single value flag was set when

134


the table was created, there will be only one tuple to delete for each relation but specifying this field as a wildcardeliminates the need for the user to give the actual data source value in the input file.

In the init_cursor function, if the relation has not been specified then both fields will be given as wildcards suchthat all tuples in the table will be retrieved. If the relation was specified, then the first field will contain the specifiedrelation and the second field will be a wildcard so that the match will happen for whatever value exists for the datasource and that tuple will be returned. Since the single value flag was set when the table was created, there will beonly one tuple returned. If the relation was specified, the inherit index is set to the first field such that, if a data sourceentry does not exist for the given relation, the Commodity DataServer hierarchy will be searched for an ancestor of therelation that does have a data source assigned and that will be returned. Notice that the inherit field corresponds tothe key field so the inheritance search will be optimized.

The get_tuple function is used to retrieve a tuple and will be called iteratively as long as there are moretuples to retrieve as dictated by the pattern specified in the init_cursor routine. When the return code forXmimVaGetTuple indicates that there are no more tuples, the cursor is deleted and a value of 0 returned so that theloop will be exited.

The Complete Interface: A Reference

Type DefinitionsThe Commodity DataServer C/C++ API provides a number of type definitions and defined constants for use inconjunction with the API routines detailed below. This section will serve to enumerate all type definitions and definedconstants available with the API. typedef char XmimBoolean; typedef char* XmimString; typedef long XmimClientHandle;

typedef enum { XMIM_FATAL_ERROR = -1, XMIM_ERROR = 0, XMIM_SUCCESS, XMIM_WARNING } XmimReturnCode;

typedef enum { XMIM_UNITS_INVALID = 0, XMIM_MILLISECONDS = -3, XMIM_SECONDS = 1, XMIM_MINUTES = 2, XMIM_HOURS = 3, XMIM_DAYS = 4, XMIM_WEEKS = 5, XMIM_MONTHS = 6, XMIM_QUARTERS = 7, XMIM_YEARS = 8


135

} XmimUnits;

The type definitions specified above are used in general throughout the API. XmimBoolean can be used whenevera true/false flag is needed. XmimString can be used when specifying a character string (e.g., a relation name).XmimClientHandle is the client handle returned by the XmimConnect routine. It is a required argument for everyCommodity DataServer C/C++ API routine, except for the convenience/utility functions. The special constantXMIM_INVALID_HANDLE is provided for use as a default client handle. XmimReturnCode is returned by everyCommodity DataServer C/C++ API routine, except for the convenience/utility functions. It provides a uniform methodfor the Commodity DataServer C/C++ API routines to indicate failure (error), success or success with information(warning). In the case where XMIM_ERROR or XMIM_WARNING is returned, the XmimPrintError routine can beused to obtain the error/warning message. For example, if a routine such as XmimGetRecords returns 0 for thenumber of records, the routine will succeed but there will be a warning message indicating that no data was found.In general, XMIM_WARNING is returned whenever no results are found. XmimUnits is used to specify the desiredCommodity DataServer time period. It is generally used in conjunction with an integer to specify the time period overwhich to aggregate the data (e.g., 10 minute or 5-day bars). Refer to “Global Options” for utility routines to set orretrieve the default units used by the system. typedef struct { unsigned year : 12; unsigned month : 4; unsigned day : 5; unsigned hour : 5; unsigned minute : 6; unsigned second : 6; unsigned millisecond : 10; } XmimDateTime;

typedef struct { unsigned year : 12; unsigned month : 4; unsigned day : 5; } XmimDate;

typedef struct { unsigned hour : 5; unsigned minute : 6; unsigned second : 6; unsigned millisecond : 10; } XmimTime;

extern DllDecl CONST XmimDate XMIM_INVALID_DATE; extern DllDecl CONST XmimTime XMIM_INVALID_TIME;

extern DllDecl CONST XmimDateTime XMIM_INVALID_DATE_TIME;

XmimDateTime, XmimDate and XmimTime are used when dealing with dates or times. Notice that these structuresare defined using bit fields. Therefore, the user must be careful not to set any of the fields to a value greater than thatexpected because the result may be to overwrite the next field. For example, in the XmimDate structure, the monthfield should range from 1 to 12; setting it to a value greater than 15 will result in the year field being overwritten.Likewise, setting the day field to a value greater than 31 will cause the month field to be overwritten. The special

136


constants XMIM_INVALID_DATE, XMIM_INVALID_TIME, and XMIM_INVALID_DATE_TIME are provided for use asdefault arguments. Refer to “Date/Time Utility Routines” for utility routines that can be used to set and compare datesand times. typedef struct { unsigned julNum; XmimDate fromDate; XmimDate toDate; int numUnits; XmimUnits units; } XmimDateTimeObj;

extern DllDecl CONST XmimDateTimeObj XMIM_INVALID_DATE_TIME_OBJ;

XmimDateTimeObj can be used in XmimVaGetRecords and XmimVaGetRecordsRollover to return dateinformation in cases where data for only a single time series is being retrieved (in lieu of using XmimDateTimesto return the dates array). The XmimGetDateTime routine can be subsequently used to retrieve specific dates.Using XmimDateTimeObj is the optimal way to retrieve data for a single time series (see “Retrieving Options DataRecords”). The special constant XMIM_INVALID_DATE_TIME_OBJ is provided for use as a default argument. typedef enum { XMIM_NAN_TYPE_HOLIDAY = 0, XMIM_NAN_TYPE_MISS_DATA = 1, XMIM_NAN_TYPE_ALL = 2 } XmimNaNType;

extern DllDecl float XMIM_MISS_DATA_NAN; extern DllDecl float XMIM_HOLIDAY_NAN;

The XmimNaNType enumeration is used with the routines for testing, setting and getting NaN values. Currently, theCommodity DataServer distinguishes between two types of NaNs, those due to holidays and those due to missingdata points. The constants XMIM_HOLIDAY_NAN and XMIM_MISS_DATA_NAN reflect the default values used in theCommodity DataServer for these types of NaNs. Refer to “Global Options” for utility routines to test for the differenttypes of NaNs and to set and retrieve the system values used for the different NaNs. typedef enum { XMIM_REL_INVALID = 0, XMIM_REL_CATEGORY = 1, XMIM_REL_NORMAL = 2, XMIM_REL_FUTURES = 3, XMIM_REL_FUTURES_CONTRACT = 4, XMIM_REL_FUTURES_CONTINUOUS = 5, XMIM_REL_OPTIONS = 6 } XmimRelType;

typedef enum { XMIM_ROLLOVER_DATA_INVALID = -1, XMIM_ROLLOVER_NONE = 0, XMIM_ROLLOVER_DAILY = 1, XMIM_ROLLOVER_TICK = 2, XMIM_ROLLOVER_BOTH = 3 } XmimRolloverDataType;

typedef enum { XMIM_COL_INVALID = 0,


137

XMIM_COL_CATEGORY = 1, XMIM_COL_NORMAL = 2, XMIM_COL_COMPOSITE = 3 } XmimColType;

The above enumerations specify the types used when modifying or adding relations and columns. typedef enum { XMIM_RELCOL_INVALID = 1, XMIM_RELCOL_BASE = 2, XMIM_RELCOL_CONSTANT = 3, XMIM_RELCOL_SPARSE = 4, XMIM_RELCOL_INHERIT = 5 } XmimRelColType;

typedef enum { XMIM_RELCOL_OBJ_INVALID = 1, XMIM_RELCOL_OBJ_FLOAT = 2, XMIM_RELCOL_OBJ_DOUBLE = 3, XMIM_RELCOL_OBJ_INT = 4, XMIM_RELCOL_OBJ_DATE = 5, XMIM_RELCOL_OBJ_TIME = 6, XMIM_RELCOL_OBJ_BOOLEAN = 7 } XmimRelColObjType;

typedef enum { XMIM_RELCOL_AGGR_INVALID = 1, XMIM_RELCOL_AGGR_OPEN = 2, XMIM_RELCOL_AGGR_HIGH = 3, XMIM_RELCOL_AGGR_LOW = 4, XMIM_RELCOL_AGGR_CLOSE = 5, XMIM_RELCOL_AGGR_SUM = 6, XMIM_RELCOL_AGGR_AVERAGE = 7, XMIM_RELCOL_AGGR_KEY = 8, XMIM_RELCOL_AGGR_INHERIT = 9 } XmimRelColAggr;

typedef enum { XMIM_RELCOL_DELTA_INVALID = 2, XMIM_RELCOL_DELTA_NONE = 1, XMIM_RELCOL_DELTA_8THS = -1, XMIM_RELCOL_DELTA_16THS = -2, XMIM_RELCOL_DELTA_32NDS = -3, XMIM_RELCOL_DELTA_64THS = -4 } XmimRelColDelta;

The enumerations used in conjunction with adding or modifying relation columns and multi-field relation columns aregiven above. typedef enum { XMIM_SEARCH_LEAVES, XMIM_SEARCH_NON_LEAVES, XMIM_SEARCH_ALL } XmimSearchFilter;

typedef struct { XmimString futuresRelName; int numContracts;

138


XmimString *contractNames; XmimDate *expirDates; } XmimContractsExpirDates;

The XmimSearchFilter type is used in searching the relation or column hierarchy (see “Navigating the RelationHierarchy” and “Navigating the Column Hierarchy”). A relation or column is classified as a leaf if it has data associatedwith it. Relation and columns typed as categories are the only non-leaf relations/columns.

The XmimContractsExpirDates structure is used for retrieving futures contract names and expiration dates (see“Accessing Futures Expiration Dates”). typedef enum { XMIM_FILL_INVALID = -1, XMIM_FILL_NAN = 0, XMIM_FILL_FORWARD = 1, XMIM_FILL_BACKWARD = 2, XMIM_FILL_INTERP_LIN = 3, XMIM_FILL_INTERP_GEO = 4, XMIM_FILL_INTERP_LOG = 5, XMIM_FILL_NEAREST = 6 } XmimFillOption;

typedef enum { XMIM_SKIP_INVALID = -1, XMIM_SKIP_NONE = 0, XMIM_SKIP_ALL_NAN = 1 } XmimSkipAllNaN;

typedef enum { XMIM_LIMIT_INVALID = -1, XMIM_LIMIT_BY_RECORDS = 0, XMIM_LIMIT_BY_MEMORY = 1 } XmimLimitMode;

The XmimFillOption, XmimSkipAllNaN and XmimLimitMode types are used in conjunction with the routineXmimGetRecords and all its variants (XmimGetRecordsRollover, XmimGetRecordsOption, ...). typedef enum { XMIM_OPTION_CALL, XMIM_OPTION_PUT, XMIM_OPTION_ALL } XmimOptionType;

The XmimOptionType enumeration is used in all of the routines dealing with options relations (XmimGetRecordsOption). typedef enum { XMIM_RECFMT_FREE = 0, XMIM_RECFMT_FIXED = 1 } XmimRecordFormat;

typedef enum { XMIM_REPLACE = 0, XMIM_QUERY_REPLACE = 0, /* for backward compatibility */ XMIM_MERGE = 1, XMIM_APPEND = 1, XMIM_QUERY_APPEND = 1 /* for backward compatibility */


139

} XmimMergeMode;

typedef enum { XMIM_TICK_NONE = 0, /* Daily data */ XMIM_TICK_PRE_SAMPLED = 1, /* Intraday data */ XMIM_TICK_BY_TICK = 2, /* Real tick data */ XMIM_TICK_CONVERT = 3, /* Real tick converted to intraday */ XMIM_TICK_MILLISECOND = 4 /* Millisecond data */ } XmimTickMode;

The enumerations above are used in the XmimReadFacts routine (see “Reading and Writing ASCII Data Files”). IfXmimTickMode is XMIM_TICK_CONVERT, the tick-by-tick data will be automatically converted into intraday data.Also, if XMIM_RECFMT_FIXED is used, the correct format for each field is specified using the following structure: typedef struct { XmimFieldType field; short width; short decimal; } XmimFieldFormat;

typedef enum { XMIM_FIELD_DATE = 1, XMIM_FIELD_TIME = 2, XMIM_FIELD_RELATION = 3, XMIM_FIELD_COLUMN = 4, XMIM_FIELD_EXP_DATE = 5, XMIM_FIELD_OPT_TYPE = 6, XMIM_FIELD_STRIKE = 7, XMIM_FIELD_FILLER = 8 } XmimFieldType;

extern DllDecl CONST XmimFieldFormat XMIM_FORMAT_DATE_FIELD;

extern DllDecl CONST XmimFieldFormat XMIM_FORMAT_TIME_FIELD;

extern DllDecl CONST XmimFieldFormat XMIM_FORMAT_RELATION_FIELD;

extern DllDecl CONST XmimFieldFormat XMIM_FORMAT_COLUMN_FIELD;

extern DllDecl CONST XmimFieldFormat XMIM_FORMAT_EXP_DATE_FIELD;

extern DllDecl CONST XmimFieldFormat XMIM_FORMAT_OPT_TYPE_FIELD;

extern DllDecl CONST XmimFieldFormat XMIM_FORMAT_STRIKE_PRICE_FIELD;

extern DllDecl CONST XmimFieldFormat XMIM_FORMAT_FILLER_FIELD;

The XmimFieldFormat structure is used when reading or writing facts (see “Reading and Writing ASCII Data Files”).Special constants are provided to allow defaulting of width and decimal values. typedef enum { XMIM_DATE_mmsddsyy = 1, /* 03/10/84 */

140


XMIM_DATE_mmsddsyyyy = 2, /* 03/10/1984 */ XMIM_DATE_ddsmmsyy = 3, /* 10/03/84 */ XMIM_DATE_ddsmmsyyyy = 4, /* 10/03/1984 */ XMIM_DATE_mm_dd_yy = 5, /* 03-10-84 */ XMIM_DATE_mm_dd_yyyy = 6, /* 03-10-1984 */ XMIM_DATE_dd_mm_yy = 7, /* 10-03-84 */ XMIM_DATE_dd_mm_yyyy = 8, /* 10-03-1984 */ XMIM_DATE_yymmdd = 9, /* 840310 */ XMIM_DATE_yyyymmdd = 10, /* 19840310 */

XMIM_DATE_mon_dd_yy = 11, /* Mar 10, 84 */ XMIM_DATE_mon_dd_yyyy = 12, /* Mar 10, 1984 */ XMIM_DATE_monp_dd_yy = 13, /* Mar. 10, 84 */ XMIM_DATE_monp_dd_yyyy = 14, /* Mar. 10, 1984 */ XMIM_DATE_month_dd_yy = 15, /* March 10, 84 */ XMIM_DATE_month_dd_yyyy = 16, /* March 10, 1984 */ XMIM_DATE_dd_mon_yy = 17, /* 10-Mar-84 */ XMIM_DATE_dd_mon_yyyy = 18, /* 10-Mar-1984 */ XMIM_DATE_dd_month_yy = 19, /* 10-March-84 */ XMIM_DATE_dd_month_yyyy = 20 /* 10-March-1984 */

} XmimDateFormat;

The above enumeration is used in conjunction with XmimWriteFacts and XmimReadFacts to identify the dateformat. Likewise, the following enumerations are used to identify the expiration date (for options) and time formats. typedef enum { XMIM_TIME_hh_mm_am = 1, /* 9:05am */ XMIM_TIME_hh_mm_AM = 2, /* 11:15AM */ XMIM_TIME_hhmm_am = 3, /* 905am */ XMIM_TIME_hhmm_AM = 4, /* 1115AM */ XMIM_TIME_hh_mm = 5, /* 09:05 */ XMIM_TIME_hhmm = 6, /* 2315 */ XMIM_TIME_hh_mm_ss_am = 7, /* 9:05:01am */ XMIM_TIME_hh_mm_ss_AM = 8, /* 11:15:01AM */ XMIM_TIME_hhmmss_am = 9, /* 90501am */ XMIM_TIME_hhmmss_AM = 10, /* 111501AM */ XMIM_TIME_hh_mm_ss = 11, /* 09:05:01 */ XMIM_TIME_hhmmss = 12, /* 231501 */ XMIM_TIME_hh_mm_ss_uuu_am = 13 /* 9:05:01 123 am */ } XmimTimeFormat;

These formats are set by using the XmimSetDateFormat, XmimSetExpDateFormat and XmimSetTimeFormatroutines (see “Global Options”). typedef XmimMergeMode XmimAppendMode;

typedef enum { XMIM_REPLACE = 0, XMIM_QUERY_REPLACE = 0, /* for backward compatibility */ XMIM_MERGE = 1, XMIM_APPEND = 1,

141

XMIM_QUERY_APPEND = 1 /* for backward compatibility */ } XmimMergeMode;

typedef enum { XMIM_QUERY_SILENT = 0, /* Do not include query */ XMIM_QUERY_VERBOSE = 1, /* Include Query */ XMIM_QUERY_VERY_VERBOSE = 2 /* Include Query and options */ } XmimVerboseMode;

typedef struct { char *leftHandSide; /* Item to substitute for */ int numRightHandSides; /* Number of substitutions */ char **rightHandSides; /* Items to substitute in */ } XmimSubstitution;

The above type definitions are used to supply optional arguments for executing/graphing a query (see “ExecutingQueries”). The XmimSubstitution structure provides substitutions to be performed on a query. The query will beexecuted repeatedly, once for each item in the substitution list.typedef enum { XMIM_APPEND_TO_ALL, /* Append current tick to historic */ XMIM_APPEND_TO_NONE, /* Do not use current tick */ XMIM_APPEND_TO_DAILY, /* Append to historic daily data */ XMIM_APPEND_TO_TICK /* Append to historic tick data */ } XmimCurrentTickUsage;

The above type definition is used for the XMIM_CURRENT_TICK_USAGE argument available only with the Vaversion of routines that are used for data retrieval (e.g., XmimVaGetRecords, XmimVaSelectRecords andXmimVaGetRecordsRollover). This argument is used to indicate whether the historic data retrieved should besupplemented with data from the high-frequency current tick store or not. typedef struct { int numUnits; /* 'numUnits < 0' means 'before' */ /* 'numUnits > 0' means 'after' */ /* 'numUnits == 0' means no units specification (default) */ XmimUnits units; XmimSwitchingMethod method; XmimDateExpression dateExpression; XmimMonth contractMonth; } XmimRolloverDay;

typedef enum { XMIM_SWITCH_INVALID = 0, XMIM_SWITCH_LAST_TRADING_DAY = 1, XMIM_SWITCH_LAST_DATA_DAY = 1, XMIM_SWITCH_USER_DEFINED = 2, XMIM_SWITCH_FIRST_NOTICE_DAY = 3, XMIM_SWITCH_EXPIRATION_DAY = 4, XMIM_SWITCH_DATE_EXPRESSION = 5, XMIM_SWITCH_VOLUME_CROSSOVER = 6, XMIM_SWITCH_OPEN_INTEREST_CROSSOVER = 7 } XmimSwitchingMethod;

typedef enum { XMIM_DAY_INVALID = 0, XMIM_DAY_MON = 1,

142


XMIM_DAY_TUE = 2, XMIM_DAY_WED = 3, XMIM_DAY_THU = 4, XMIM_DAY_FRI = 5, XMIM_DAY_SAT = 6, XMIM_DAY_SUN = 7, XMIM_DAY_OF_MONTH = 8 } XmimDay;

typedef enum { XMIM_MONTH_INVALID = 0, XMIM_MONTH_JAN = 1, XMIM_MONTH_FEB = 2, XMIM_MONTH_MAR = 3, XMIM_MONTH_APR = 4, XMIM_MONTH_MAY = 5, XMIM_MONTH_JUN = 6, XMIM_MONTH_JUL = 7, XMIM_MONTH_AUG = 8, XMIM_MONTH_SEP = 9, XMIM_MONTH_OCT = 10, XMIM_MONTH_NOV = 11, XMIM_MONTH_DEC = 12 } XmimMonth;

typedef struct { int nTh; /* the member variable nTh is the compilation of */ /* the `(nTH | [nTH TO] LAST)' clause; its inerpretation is:*/ /* o 1 is first, 2 is second, etc. */ /* o 0 is invalid */ /* o -1 is last, -2 is "2nd to last", etc. */ XmimDay theDay; /* theDay encodes the compilation of the */ /* `(DAY | day_of_week)' clause */ int numMonths; /* numMonths encodes the compilation of the */ /*`[n MONTHS BEFORE]' clause */ } XmimDateExpression;

The structures and enumerations above can be used to specify the rollover day in the XmimVaAddRelation,XmimVaGetRecordsRollover and XmimVaGetRolloverDates routines as an alternative to using strings for thespecification. The use of these structures is described in detail in “Rollover Specifications”. typedef struct { int nearby; XmimInterpMethod interpolate; XmimYearToYear yearToYear; XmimRolloverDetail rolloverDetail; } XmimRolloverPolicy;

typedef struct { XmimRolloverMethod rolloverMethod; union { XmimAdjustMethod adjustMethod; XmimSmoothMethod smoothMethod; XmimInterpMethod perpetMethod;


143

} theMethod; } XmimRolloverDetail;

typedef enum { XMIM_ROLLOVER_INVALID = 0, XMIM_ROLLOVER_ACTUAL_PRICES = 1, XMIM_ROLLOVER_ADJUSTED_PRICES = 2, XMIM_ROLLOVER_PERPETUAL = 3, XMIM_ROLLOVER_SMOOTHED = 4 } XmimRolloverMethod;

typedef enum { XMIM_ADJUST_INVALID = 0, XMIM_ADJUST_FORWARD = 1, XMIM_ADJUST_BACKWARD = 2, XMIM_ADJUST_CASH = 3 } XmimAdjustMethod;

typedef struct { int numContracts; XmimInterpAlgorithm algorithm; } XmimSmoothMethod;

typedef struct { int numUnits; XmimUnits units; XmimInterpAlgorithm algorithm; } XmimInterpMethod;

typedef enum { XMIM_INTERPOLATE_INVALID = 0, XMIM_INTERPOLATE_LINEAR = 1, XMIM_INTERPOLATE_LOG = 2, XMIM_INTERPOLATE_GEOMETRIC = 3 } XmimInterpAlgorithm;

typedef enum { XMIM_YEAR_TO_YEAR_NO = 0, XMIM_YEAR_TO_YEAR_JAN = 0x0001, XMIM_YEAR_TO_YEAR_FEB = 0x0002, XMIM_YEAR_TO_YEAR_MAR = 0x0004, XMIM_YEAR_TO_YEAR_APR = 0x0008, XMIM_YEAR_TO_YEAR_MAY = 0x0010, XMIM_YEAR_TO_YEAR_JUN = 0x0020, XMIM_YEAR_TO_YEAR_JUL = 0x0040, XMIM_YEAR_TO_YEAR_AUG = 0x0080, XMIM_YEAR_TO_YEAR_SEP = 0x0100, XMIM_YEAR_TO_YEAR_OCT = 0x0200, XMIM_YEAR_TO_YEAR_NOV = 0x0400, XMIM_YEAR_TO_YEAR_DEC = 0x0800, XMIM_YEAR_TO_YEAR_YES = 13 } XmimYearToYear;

The structures and enumerations above can be used to specify the rollover policy in the XmimVaAddRelation,XmimVaGetRecordsRollover and XmimVaGetRolloverDates routines as an alternative to using strings for thespecification. The use of these structures is described in detail in “Rollover Specifications”. typedef enum {

144


XMIM_ADJ_STUDY_NONE, XMIM_ADJ_STUDY_MOVE, XMIM_ADJ_STUDY_PCT_MOVE, XMIM_ADJ_STUDY_AVG, XMIM_ADJ_STUDY_SUM, XMIM_ADJ_STUDY_COUNT, XMIM_ADJ_STUDY_HIGH, XMIM_ADJ_STUDY_LOW, XMIM_ADJ_STUDY_STDDEV, XMIM_ADJ_STUDY_REGRESSION, XMIM_ADJ_STUDY_VARIANCE } XmimAdjustedStudy;

The above enumeration is used for the XmimGetRecordsRollover and XmimGetRecordsRolloverAllColumnsroutines to specify the desired adjusted study. typedef enum { XMIM_DATA_TICK, /* Seconds */ XMIM_DATA_INTRADAY, /* Minutes */ XMIM_DATA_DAILY, /* Day */ XMIM_DATA_ALL, /* Each Type is regenerated */ XMIM_DATA_MILLISECOND /* Milliseconds */ } XmimDataType;

The above enumeration is used for the XmimDeleteFactsSelective routine to specify the desired type of data todelete. typedef char XmimTradingPattern;

#define XMIM_MONDAY (XmimTradingPattern)(1)#define XMIM_TUESDAY (XmimTradingPattern)(1<<1)#define XMIM_WEDNESDAY (XmimTradingPattern)(1<<2)#define XMIM_THURSDAY (XmimTradingPattern)(1<<3)#define XMIM_FRIDAY (XmimTradingPattern)(1<<4)#define XMIM_SATURDAY (XmimTradingPattern)(1<<5)#define XMIM_SUNDAY (XmimTradingPattern)(1<<6)#define XMIM_MTWTF (XmimTradingPattern)(XMIM_MONDAY|XMIM_TUESDAY|XMIM_WEDNESDAY|XMIM_THURSDAY|XMIM_FRIDAY)#define XMIM_MTWTFSD (XmimTradingPattern)(XMIM_MONDAY|XMIM_TUESDAY|XMIM_WEDNESDAY|XMIM_THURSDAY|XMIM_FRIDAY|XMIM_SATURDAY|XMIM_SUNDAY)#define XMIM_MTWTFD (XmimTradingPattern)(XMIM_MONDAY|XMIM_TUESDAY|XMIM_WEDNESDAY|XMIM_THURSDAY|XMIM_FRIDAY|XMIM_SUNDAY)

typedef struct { XmimTime fromTime; XmimTime toTime; } XmimTimePeriod;

The type definitions above are used in the routines for adding, modifying and retrieving relation information(XmimVaAddRelation, XmimVaModRelation and XmimVaGetRelation) when the days and/or times that a timeseries trades changes over time. A new XmimRelSegment instance is defined for each change in the days and/ortimes during which the time series trades. Each instance establishes a trading pattern consisting of the trading daysfor the time series. To establish a trading pattern, the appropriate defined constants may be used and simply orderedtogether. The days in the trading pattern need not be contiguous. Each instance may have multiple time periodsassociated with it, allowing for time specifications that are not contiguous, as well.


145

If a time series retains the same days and times for trading through all of its history, yet trades on days other than thedefault trading days, XmimTradingPattern can be used without specifying the XmimRelSegment data structure.The default trading pattern is Monday-Friday (XMIM_MTWTF). typedef long XmimTableHandle; typedef long XmimCursorHandle;

extern DllDecl CONST XmimTableHandle XMIM_INVALID_TABLE_HANDLE;

extern DllDecl CONST XmimCursorHandle XMIM_INVALID_TABLE_CURSOR;

typedef enum { XMIM_TAB_FIELD_INT, XMIM_TAB_FIELD_FLOAT, XMIM_TAB_FIELD_DOUBLE, XMIM_TAB_FIELD_DATE, XMIM_TAB_FIELD_TIME, XMIM_TAB_FIELD_STRING, XMIM_TAB_FIELD_ATOM, XMIM_TAB_FIELD_BLOCK, XMIM_TAB_FIELD_REL, XMIM_TAB_FIELD_COL } XmimTableFieldType;

typedef struct { XmimString name; XmimTableFieldType type; int length; } XmimTableFieldDesc;

typedef struct { XmimBoolean wildcardP; union { int intValue; float floatValue; double doubleValue; XmimString stringValue; XmimDate dateValue; XmimTime timeValue; long longValue; } value; } XmimTableFieldVal;

The above type definitions and special constants are used in conjunction with the dynamic schema facility.XmimTableHandle is the table identifier returned by either the XmimCreateTable or XmimOpenTableroutine. XmimCursorHandle is a cursor into a table returned by the XmimCreateTableCursor routine.Each cursor is implicitly associated with a table and is used to point to a particular entry (field) of a table. Thespecial constants XMIM_INVALID_TABLE_HANDLE and XMIM_INVALID_TABLE_CURSOR are provided foruse as the default table handle and table cursor, respectively. They can be used to initialize a table or cursor toinvalid. The XmimTableFieldType enumeration identifies the type of data in a field and is included as part ofthe XmimTableFieldDesc structure. This structure also defines the name of the field and, if the field is of typeXMIM_TAB_FIELD_BLOCK, the length in bytes. The XmimTableFieldVal structure contains the actual field value(which can be either an integer, float, double, XmimString, XmimDate or XmimTime). For retrieval purposes, a

146


wildcard can be used in lieu of a particular value and so the structure also contains a boolean member used for thatpurpose (wildcardP). typedef struct { XmimString relation; XmimString column; int readFlag; int writeFlag; } XmimPermissionRecord;

The XmimPermissionRecord structure is used in conjunction with the entitlements facility to provide the groupread and write permissions.

General Argument HandlingMost of the API commands have numerous arguments, the majority of which can be defaulted. The CommodityDataServer C/C++ API provides a convenient way to default arguments, through a name-value parameter passingmechanism. All API commands have both a traditional positional interface, as well as a name-value interface. Thename-value interface of the function is accessed by prefixing the name of the function with XmimVa instead of Xmim.

The XmimVa arguments are specified by using a naming token which then takes one or more arguments. The exactnumber of arguments is determined by the token. The name-value pairs can occur in any order, and any of these thathave default values may be omitted. The special token XMIM_END_ARGS is used to delineate the end of the argumentlist. Clearly, the XmimVa interface does not present an advantage when the number of arguments passed is small.However, it is a significant improvement when the routine takes a large number of arguments.

Throughout this reference guide, the synopsis for each routine will include both an ANSI C header prototype forthe regular Xmim routine and an actual call for the XmimVa specification. We will use the following convention tospecify the arguments for the XmimVa calls. All arguments will be listed in the synopsis. Optional arguments willhave their default value listed as the argument of the call, whereas required arguments will use the same nameas the corresponding argument of the regular Xmim routine specification. The types for arguments in the XmimVaroutines can be determined from the specified types for the equivalent arguments of the corresponding ANSI C headerprototype. Because the XmimVa specification is an actual call, the return arguments in this specification will bepreceded by the address operator (&).

The XmimVa interface also provides a flexible mechanism for specifying multiple columns, relations, etc. as arguments.Whenever a list of input arguments (e.g., a list of columns or relations) are required for a routine, the XmimVa call willaccept the elements of the list specified individually, in a list, in an array or in a file. When specified individually, eachis preceded by the corresponding single argument token (e.g., XMIM_RELATION). For a list specification, the tokenname to be used is the single argument token name followed by _LIST (e.g., XMIM_RELATION_LIST). The list mustbe terminated with the NULL token. An example using list specifications is provided in “Accessing Data for MultipleRelations”. For the array specification, the token name is the single argument token name followed by _ARRAY(e.g., XMIM_RELATION_ARRAY). The arguments after the array token will be the number of elements in the array,followed by the array itself. An example using the array specification is provided in “Accessing Data with Relationsand Columns Specified in Arrays”. For the file specification, the token name is the single argument token with _FILEappended (e.g., XMIM_RELATION_FILE). The arguments after the file token will be the file name followed by two


147

return arguments for the number of entries in the file and the elements read from the file. An example using the filespecification is provided in “Accessing Data for Relations Specified in a File”. The different specifications may becombined and multiple instances of the same specification may be used (e.g., some of the elements may be specifiedindividually, while the rest are given in an array).

When multiple columns, relations, etc., are applicable as input arguments for a routine, only the list specificationwill be given in the XmimVa synopses, though any of the specifications can be used. This will apply toXMIM_COLUMN_LIST, XMIM_RELATION_LIST, XMIM_SHOW_ATTR_LIST, XMIM_CONDITION_LIST,XMIM_FORMAT_LIST, XMIM_FIELD_LIST, XMIM_DESCRIPTION_LIST, XMIM_OBJ_TYPE_LIST,XMIM_DELTA_LIST, XMIM_AGGR_RULE_LIST, XMIM_QUERY_SUBSTITUTION_LIST, and XMIM_DATABASE_LIST.If a list specification in an XmimVa routine is optional, it will be shown with a value of NULL. For the special casewhereby the list consists of numbers (integer or real), the list specification method will not be allowed as a numericalvalue of 0 (or 0.0) and will be indistinguishable from the list terminator (NULL). For this case, the array specificationwill be given in the XmimVa synopses, though any of the specifications except the list specification can be used. Thisapplies only to XMIM_CONSTANT_ARRAY. If an array specification in an XmimVa routine is optional, it will be shownwith values of 0, NULL. When single argument tokens (e.g., XMIM_RELATION, XMIM_COLUMN, etc.) appear in theXmimVa synopses, then multiple columns, relations, etc. are not accepted.

The array specification is also often used as a return argument when both an array and the number of elements inthat array need to be returned. For example, XMIM_COLUMN_ARRAY can be used in an XmimVa routine to return thenumber of columns and the column names.

Note that this usage as a return argument does not imply that the other specification methods givenabove are applicable; this flexibility pertains only to the specification of input arguments.

Initialization and TerminationXmimReturnCode XmimConnect (XmimString serverName, int serverNumber); XmimClientHandle *handle); XmimReturnCode XmimVaConnect (XMIM_SERVER_NAME, NULL, XMIM_SERVER_NUMBER, 0, XMIM_CLIENT_HANDLE, &handle, XMIM_END_ARGS); XmimReturnCode XmimDisconnect (XmimClientHandle handle); XmimReturnCode XmimVaDisconnect (XMIM_CLIENT_HANDLE, handle, XMIM_END_ARGS);

The XmimConnect routine establishes the connection to a Commodity DataServer, and XmimDisconnect terminatesthe connection. All clients using the Commodity DataServer must begin with a call to XmimConnect and must callXmimDisconnect before they terminate. It is very important that XmimDisconnect be used before exiting forany reason to avoid exhausting the client handle space. The serverName argument specifies the hostname of themachine where the Commodity DataServer resides; if it has the value NULL, XmimConnect will try to connect to aserver running on the local machine. The serverNumber argument specifies which server to connect to, where thenumber identifying the server is provided by the user when the server is invoked. If there is no server running on the

148


specified host, XmimConnect will fail. If XmimConnect succeeds, it returns a client handle (handle) consisting ofan integer greater than 0. The client handle uniquely identifies the server connection enabling connections to (and,therefore, utilization of) different servers by a client. All API routines (with the exception of the utility/convenienceroutines) require this client handle to be specified as an argument.

In XmimDisconnect, the handle argument must be the client handle returned from the XmimConnect call; thisidentifies which server to disconnect from.

Supporting Multiple DatabasesThe Commodity DataServer provides support for using multiple databases. These databases may be either CommodityDataServer databases or external databases. The databases specified in the server xmimrc (.xmimrc) file establishthe pool of databases that will be opened by a Commodity DataServer. The databases specified in the .xmimrc filemay be new or existing databases. These databases will remain open for the duration of the server's existance. Anyclient connecting to an Commodity DataServer can have an associated client xmimrc (.cxmimrc) file. If a client hasa .cxmimrc file, then the initial pool of databases to be used for that client will be those specified in the .cxmimrc file.These do not have to be a subset of the server databases but must be existing databases: new databases can not becreated by specifying them in the .cxmimrc file. If a client does not have a .cxmimrc file then the databases specifiedin the server .xmimrc file will be those available initially to that client. Chapter 40, “Commodity DataServer .xmimrcFile” and ??? provide descriptions of these facilities.

Open, create and close routines are provided so that a client can add or remove databases from/to its pool ofdatabases. There are also routines provided such that a client can temporarily narrow or restrict the database view toinclude only a specified subset of the currently open databases and then expand back to the entire current databasepool. A listing of the databases in the current database view can be obtained.

Note that the order of specification for databases (i.e., the order in which the initial database poolis specified and then in which open/new commands are used) is important as the databases will beopened in the order specified and schema information will be accessed in that order as well. Thus, forexample, if the same relation exists in more than one open database, the first one specified will be theone used unless disambiguating database name/path information is provided. The databases to be usedmost should always be specified first.

In incorporating the use of multiple databases throughout this document, it is important to note that multipledatabases are viewed as though they were a single database and so the applied semantics is as if dealing with asingle database. If a database is thought of as a "tree" of nodes, then a multi-database is a tree that contains all pathsthat exist in any one of the single databases. Each path appears only once in the multi-database. Note that the nodesin the multi-database tree do not correspond to any one node in the single databases. Rather, a node in the multi-database tree is an amalgamation of the corresponding nodes in the single databases, e.g., it may have both daily andtick data, even though database #1 only has daily data and database #2 only has tick data.

Note that while accessing non-unique relations and columns within the same database requires that a disambiguatingpath be specified, accessing relations and columns that are not unique across different databases does not havethis requirement. In general, in the presence of multiple databases, any routines resulting in the addition and/or


149

modification of schema information will result in the information being added to all applicable database schemas.For example, if IBM is added to Equities:i:ib, then IBM is added to all database schemas that have a category calledEquities:i:ib. Data information, however, is added to the first applicable database. For example, if daily data is added toClose of IBM, it is added to the first database that has a relation called IBM and a column called Close. Any routineshaving to do with schema or data retrieval will retrieve information from the first applicable database. For example,the description for IBM comes from the first database schema that has the relation IBM. When schema information isviewed, hierarchy information is merged as consistent with viewing multiple databases as though they were a singledatabase. For example, the children of Equities will consist of all the children that appear as children of Equities inany single open database. When overlap exists, the schema information shown will be that from the first applicableschema. Delete commands will result in schema or data information being deleted from all applicable databases.

Creating, Opening and Closing DatabasesXmimReturnCode XmimNewDatabase (XmimClientHandle handle, XmimString database); XmimReturnCode XmimVaNewDatabase (XMIM_CLIENT_HANDLE, handle, XMIM_DATABASE, database, XMIM_END_ARGS); XmimReturnCode XmimOpenDatabase (XmimClientHandle handle, XmimString database); XmimReturnCode XmimVaOpenDatabase (XMIM_CLIENT_HANDLE, handle, XMIM_DATABASE, database, XMIM_END_ARGS); XmimReturnCode XmimCloseDatabase (XmimClientHandle handle, XmimString database); XmimReturnCode XmimVaCloseDatabase (XMIM_CLIENT_HANDLE, handle, XMIM_DATABASE, database, XMIM_END_ARGS);

The XmimNewDatabase routine is used to create a new database, open it and add it to the client's currentdatabase pool. If the specified database has already been created then an error will be returned. The databaseargument gives the name of the file containing the DataServer schema or provides the specification for an externaldatabase. In the case an external database, the XmimNewDatabase routine most likely will behave the same as theXmimOpenDatabase routine with the database argument being directly passed to the shared library routine to openthe schema for that external database. This is due to the fact that for many external databases (e.g., Oracle®), thedatabase administrator must set up and configure the external database instance and there is no practical way for theCommodity DataServer to do this. So, the database needs to already be there with a setup username and password.The setup username and password can then be what is passed as the argument for the XmimNewDatabase orXmimOpenDatabase routine. Of course, this is all dependent on the implementation of the shared libraries for theparticular external database. In the case of a Commodity DataServer database, all database files are accessed fromthe schema file.

The XmimOpenDatabase routine is used to open a database and add it to the client's current database pool. Thedatabase to be opened may be any existing database, that is, it does not have to exist in the server .xmimrc file. If thespecified database has already been opened by another client, then a pointer to the opened database will be returned.If this is the first client to open the database, it will be physically opened.

The XmimCloseDatabase routine is used to close a database that is currently open. Using this command willremove the specified database from the pool of accessible databases. In the case that this client is the only one

150


currently referencing the database, the database will be physically closed unless it was a database specified in theserver .xmimrc file. If a client terminates without closing a database but that client is the only one referencing thedatabase, the system will automatically close the database (unless it was a database specified in the server .xmimrcfile).

Changing the Database Storage Location to a FileXmimReturnCode XmimSetDefaultFactsFile(XmimClientHandle handle, XmimString rel, XmimString file, XmimDataType type);XmimReturnCode XmimVaSetDefaultFactsFile(XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relation, XMIM_FILENAME, file, XMIM_TYPE, type, XMIM_END_ARGS);

By default, the Commodity DataServer stores the daily, intraday tick and real tick (second) and millisecond data to filesspecified by the system. To override this behavior and specify the filename where the Commodity DataServer shouldstore the data for a given relation, the XmimSetDefaultFactsFile() and XmimVaSet DefaultFactsFile()commands can be used. (These routines are equivalent to the BMIM commands database_def_daily_file,database_def_intraday_file, database_def_tick_file, and database_def_millisecond_file.)

Changing the Database ViewXmimReturnCode XmimNarrowDatabases (XmimClientHandle handle, int numDatabases, XmimString *databases); XmimReturnCode XmimVaNarrowDatabases (XMIM_CLIENT_HANDLE, handle, XMIM_DATABASE_LIST, database1, database2, ..., NULL, XMIM_END_ARGS); XmimReturnCode XmimWidenDatabases (XmimClientHandle handle); XmimReturnCode XmimVaWidenDatabases (XMIM_CLIENT_HANDLE, handle, XMIM_END_ARGS);

The XmimNarrowDatabases routine is used to restrict the databases available to those specified in the databaseslist. The databases specified will compose the current database view and must be a subset of the current opendatabase pool. Databases that are not specified will not be closed, they are just restricted from the view untila subsequent call to XmimNarrowDatabases includes them or the XmimWidenDatabases routine is used.Subsequent calls to XmimNarrowDatabases will undo the effect of the previous XmimNarrowDatabases call (i.e.,narrow is not nested). The XmimWidenDatabases routine resets the view to the current pool of all open databases.

Listing the Current Database ViewXmimReturnCode XmimGetDatabases (XmimClientHandle handle, int *numDatabases, XmimString **databases));


151

XmimReturnCode XmimVaGetDatabases (XMIM_CLIENT_HANDLE, handle, XMIM_DATABASE_ARRAY, &numDatabases, &databases, XMIM_END_ARGS);

The XmimGetDatabases routine is used to return a listing of all databases in the current database view. That is,if a call to XmimNarrowDatabases has been made and it specifies a single database as the current view, thenXmimGetDatabases will return only that single database (until the XmimWidenDatabases routine is used to set theview back to the entire open database pool).

BMIM Command EquivalentsThe Commodity DataServer C/C++ API provides functions to support most all of the commands provided by BMIM.For a more thorough understanding of these commands refer to Chapter 4, “BMIM Scripting Language”.

Loading a BMIM ScriptXmimReturnCode XmimLoadBmimScript (XmimClientHandle handle, XmimString scriptFileName, XmimString outputFileName); XmimReturnCode XmimVaLoadBmimScript (XMIM_CLIENT_HANDLE, handle, XMIM_FILENAME, scriptFileName, XMIM_OUTPUT_FILENAME, NULL, XMIM_END_ARGS);

The XmimLoadBmimScript routine can be used to load the BMIM script given in scriptFileName. IfoutputFileName is specified, any error or warning messages along with a trace of all BMIM commands executedwill be printed to this file. If the output file is not specified (or specified as NULL) then the information will be printed tostdout.

Locking and Unlocking the DatabaseXmimReturnCode XmimLockDatabase (XmimClientHandle handle); XmimReturnCode XmimVaLockDatabase (XMIM_CLIENT_HANDLE, handle, XMIM_END_ARGS); XmimReturnCode XmimUnlockDatabase (XmimClientHandle handle); XmimReturnCode XmimVaUnlockDatabase (XMIM_CLIENT_HANDLE, handle, XMIM_END_ARGS); XmimReturnCode XmimIsDatabaseLocked (XmimClientHandle handle, XmimBoolean *isLocked); XmimReturnCode XmimVaIsDatabaseLocked (XMIM_CLIENT_HANDLE, handle, XMIM_DB_LOCKED *isLocked, XMIM_END_ARGS);

The XmimLockDatabase and XmimUnlockDatabase routines are used to lock and unlock the database,respectively. Changes to the database, i.e., updates, may only be performed on locked databases. If the Commodity

152


DataServer can not acquire a write-lock for the database, XmimLockDatabase will fail. The Commodity DataServersupports a time series level granularity of locking for data updates and a per-API routine level granularity of lockingfor schema updates. This granularity of locking is performed transparently to the user: as with previous versions ofthe Commodity DataServer, an XmimLockDatabase is necessary at the start of the update process only. The use ofXmimLockDatabase essentially puts the system into a mode whereby locking occurs as needed.

The unlock routine serves as the final commit for changes to the database and is when rollover computations willbe performed. If multiple clients have locked the database, rollover computations will be performed only after thelast lock is released. Both updates to existing time series (e.g., adding or deleting data for existing relation columns)and updates that require modifications to the DataServer schema will be visible immediately following the updateoperation; unlocking is not necessary for these changes to be available. The XmimIsDatabaseLocked routine canbe used to determine if a database is currently locked or not; the isLocked argument will be returned as True if thedatabase is currently locked and False otherwise. These routines are applicable for locks pertaining to the historicaldatabase; for locks specific to the high-frequency data store see “High Frequency Updating”.

Deleting FactsXmimReturnCode XmimDeleteFacts (XmimClientHandle handle, XmimString relName, XmimString colName); XmimReturnCode XmimVaDeleteFacts (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_COLUMN, colName, XMIM_END_ARGS); XmimReturnCode XmimDeleteFactsSelective (XmimClientHandle handle, XmimString relName, int numColumns, XmimString *colNames, XmimDataType type, XmimDate fromDate, XmimDate toDate, XmimTime fromTime, XmimTime toTime); XmimReturnCode XmimVaDeleteFactsSelective (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_COLUMN_LIST, NULL, XMIM_TYPE, XMIM_DATA_ALL, XMIM_FROM_DATE, XMIM_INVALID_DATE, XMIM_TO_DATE, XMIM_INVALID_DATE, XMIM_FROM_TIME, XMIM_INVALID_TIME, XMIM_TO_TIME, XMIM_INVALID_TIME, XMIM_END_ARGS);

The XmimDeleteFacts routine can be used to delete all the data for the specified relation column. If colName is notspecified, data for all existing columns of relName will be deleted. When multiple databases are open, data will bedeleted from all applicable databases. This routine differs from XmimDelRelColumn in that only the data is deleted,the relation column will still exist in the database.


153

The XmimDeleteFactsSelective routine can be used to delete data for a relation for the given columns, daterange, time range and data type. The data type indicates whether data should be deleted for millisecond, tick(second), intraday, daily or all. If columns are not specified then, by default, data for all existing columns of the relationwill be deleted. Likewise, if the dates and times are not given then data for all days for which there is data for theentire trading period of each day will be deleted. If both the date and time ranges are specified, then, for each day inthe date range given, data will be deleted for the time range given. Of course the fromTime and toTime argumentsonly make sense in the context of intraday, real tick or millisecond data.

Adding, Modifying and Deleting RelationsXmimReturnCode XmimAddRelation (XmimClientHandle handle, XmimString relName, XmimString description, XmimString parentName, XmimRelType type, XmimString exchange, XmimString time_zone, XmimTime startTrade, XmimTime endTrade, float contractUnits, XmimDate expirationDate, XmimDate firstNoticeDate, XmimString rolloverDay, XmimString rolloverPolicy XmimRolloverDataType rolloverData); XmimReturnCode XmimVaAddRelation (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_DESCRIPTION, NULL, XMIM_PARENT, parentName, XMIM_TYPE, XMIM_REL_NORMAL, XMIM_EXCHANGE, NULL, XMIM_TIME_ZONE, NULL, XMIM_START_TRADE, XMIM_INVALID_TIME, XMIM_END_TRADE, XMIM_INVALID_TIME, XMIM_CONTRACT_UNITS, 1.0, XMIM_EXPIRATION_DATE, XMIM_INVALID_DATE, XMIM_FIRST_NOTICE_DATE, XMIM_INVALID_DATE, XMIM_ROLLOVER_DAY, NULL, XMIM_ROLLOVER_DAY_STRUCT, NULL, XMIM_ROLLOVER_POLICY, NULL, XMIM_ROLLOVER_POLICY_STRUCT, NULL, XMIM_ROLLOVER_DATA_TYPE, XMIM_ROLLOVER_BOTH, XMIM_TRADING_PATTERN, NULL, XMIM_SEGMENTS, 1, NULL, XMIM_END_ARGS);

The XmimAddRelation function adds a relation to the database. The name (ticker symbol) of the relation is specifiedin the relName argument. In the Commodity DataServer, there is no unique name requirement for relations, even leafrelations (those with data associated). The description is a string containing a user-defined explanation of the relation.The name of the parent relation in the relation hierarchy is specified by the parentName argument. This will be thecategory to which the relation is added. If the category name is ambiguous (i.e., there is more than one category withthat name), a “path" to the correct category can be specified, using colons as separators. For example, the Indices

154


sub-category of Futures can be unambiguously stated as Futures:Indices. A double colon can be used as a separator toindicate a descendant that is not necessarily an immediate descendant. For example, TopRelation::Indices will resolveto TopRelation:Futures:Indices. When multiple databases are open, the relation will be added to all database schemasthat have the category specified by parentName. When non-unique relations occur within the same database, theymust have distinct paths and the path specification must be provided when the relations are referenced so as todisambiguate them. For the Commodity DataServer, the user may construct the entire relation hierarchy, including thespecification of the top-most categories. TopRelation serves to identify the top-most categories and, therefore, is theroot of the relation hierarchy.

The type of the relation will indicate whether it is a futures continuous contract, a futures individual contract, a mainfutures continuous contract, a category, or a normal leaf. The main futures contract is the parent of all continuousand individual contracts for a given future (e.g., SP). It is, itself, a continuous contract (presumably, the most oftenused continuous contract). Category relations are used merely to group other relations, not to store data. Categoryrelations are the only non-leaf relations in the sense that they are the only relations that do not have data associatedwith them. Normal relations include all non-futures leaf relations. The system enforces that all futures contracts, andfutures continuous relations have names that begin with the name of the futures parent followed by an underscore(i.e., _) character.

Options are treated as normal relations. However, options must be stored underneath a special options relationcategory. For example, the options for DELL is stored under the DELL.Options category and options for IBM is storedunder the IBM.Options category.

The category is mixed case, first letter capitalized, the rest lowercase.

All options relations are required to have XML is the description field as such:<plist> <type> option_type </type> <strike> strike_price </strike> <expir> expiration_date </expir></plist>

where:

● option_type is either put or call● strike_price is a numeric field (e.g., 30.0)● expiration_date is a date (e.g., 3/17/2003)


Example:relation_add { name = IBM.199308.CALL.40; description = “<plist><expir> 08/31/1993 </expir><strike> 40 </strike><type> call </type></plist>”

155

parent = IBM.Options;}


Additional arguments provided are used to indicate the exchange name, time zone, starting and ending times fortrading, and contract units for profit/loss queries. Only the handle, relName and parentName arguments must bespecified in the XmimVa call. The type will default to normal, the description will default to be identical to relNameand the other arguments will be assigned a suitable default by the Commodity DataServer based on parentName.

Some of the optional arguments pertain only to futures relations. If the relation is an individual futures contract thenthe expiration date may be specified. If the relation is a futures continuous contract then the rollover switching day,rollover policy and rollover data type may be specified. The rollover data type indicates whether rollover computationsshould take place for daily data, tick data, both tick and daily, or neither. The rollover switching day and the rolloverpolicy together constitute a complete rollover specification for a continuous contract. They may be given as stringsor, if the XmimVaAddRelation routine is used, as structures. The argument tokens XMIM_ROLLOVER_DAYand XMIM_ROLLOVER_POLICY are used when strings are provided; XMIM_ROLLOVER_DAY_STRUCT andXMIM_ROLLOVER_POLICY_STRUCT are used when pointers to structures are provided. See “Rollover Specifications”for a description of the rollover specification.

Two arguments that are included in the Va version only are represented by the tokens XMIM_SEGMENTS andXMIM_TRADING_PATTERN. The trading pattern argument is used to specify the days on which trading takes place.The default will be Monday through Friday so this argument is only needed in the event that the trading days aredifferent than that (e.g, Monday through Saturday).

Note that the trading days do not need to be contiguous (e.g., a pattern with only Monday, Wednesdayand Friday can be specified).

The trading pattern data structure is as follows:typedef char XmimTradingPattern;

#define XMIM_MONDAY (XmimTradingPattern) (1) #define XMIM_TUESDAY (XmimTradingPattern) (1<<1) #define XMIM_WEDNESDAY (XmimTradingPattern) (1<<2) #define XMIM_THURSDAY (XmimTradingPattern) (1<<3) #define XMIM_FRIDAY (XmimTradingPattern) (1<<4) #define XMIM_SATURDAY (XmimTradingPattern) (1<<5) #define XMIM_SUNDAY (XmimTradingPattern) (1<<6) #define XMIM_MTWTF (XmimTradingPattern) \ (XMIM_MONDAY | XMIM_TUESDAY | XMIM_WEDNESDAY | XMIM_THURSDAY | XMIM_FRIDAY)

To produce the desired trading pattern, the appropriate defined constants may be used and simply orderedtogether. For example, Monday-Saturday would be specified as: (XmimTradingPattern) (XMIM_MTWTF |

156


XMIM_SATURDAY). Likewise, if the trading days were Monday, Wednesday and Friday, the pattern would be:(XmimTradingPattern) (XMIM_MONDAY | XMIM_WEDNESDAY | XMIM_FRIDAY).

The XMIM_SEGMENTS argument is used in conjunction with a time series that changes the days and/or times thatit trades over time. A segment is defined as a period of history for a time series during which a particular tradingtime range and particular trading days are in effect. If a time series trades over all of its history with the same tradingdays and times then it has a single segment and the XMIM_SEGMENTS argument need not be used; the time rangeis specified with the startTrade and endTrade arguments and the trading days either default to Monday-Fridayor are specified via the XMIM_TRADING_PATTERN argument. However, if a time series has multiple segments, theXMIM_SEGMENTS argument must be used to specify the different segments. For each segment, the starting date,trading pattern, number of time periods and time periods are defined as such:

typedef struct { XmimDate segStartDate; XmimTradingPattern tradingPattern; int numPeriods; XmimTimePeriod *timePeriods; } XmimRelSegment;

typedef struct { XmimTime fromTime; XmimTime toTime; } XmimTimePeriod;

The multiple time periods enable time specifications that are not contiguous (e.g., trades from 8am-11am, 1pm-3pm).The XMIM_SEGMENTS argument is given as the number of segments followed by an array of XmimRelSegmentstructs. If this argument is used, then the starting and ending times for trading (XMIM_START_TRADE,XMIM_END_TRADE) as well as the trading pattern (XMIM_TRADING_PATTERN) arguments are ignored.

XmimReturnCode XmimModRelation (XmimClientHandle handle, XmimString origRelName, XmimString relName, XmimString description, XmimString parentName, XmimRelType type, XmimString exchange, XmimString time_zone, XmimTime startTrade, XmimTime endTrade, float contractUnits, XmimDate expirationDate, XmimDate firstNoticeDate, XmimString rolloverDay, XmimString rolloverPolicy, XmimRolloverDataType rolloverData); XmimReturnCode XmimVaModRelation (XMIM_CLIENT_HANDLE, handle, XMIM_ORIG_RELATION, origRelName, XMIM_RELATION, NULL, XMIM_DESCRIPTION, NULL, XMIM_PARENT, NULL, XMIM_TYPE, XMIM_REL_INVALID, XMIM_EXCHANGE, NULL, XMIM_TIME_ZONE, NULL, XMIM_START_TRADE, XMIM_INVALID_TIME,


157

XMIM_END_TRADE, XMIM_INVALID_TIME, XMIM_CONTRACT_UNITS, 1.0, XMIM_EXPIRATION_DATE, XMIM_INVALID_DATE, XMIM_FIRST_NOTICE_DATE, XMIM_INVALID_DATE, XMIM_ROLLOVER_DAY, NULL, XMIM_ROLLOVER_DAY_STRUCT, NULL, XMIM_ROLLOVER_POLICY, NULL, XMIM_ROLLOVER_POLICY_STRUCT, NULL, XMIM_ROLLOVER_DATA_TYPE, XMIM_ROLLOVER_DATA_INVALID, XMIM_TRADING_PATTERN, NULL, XMIM_SEGMENTS, 1, NULL, XMIM_END_ARGS); XmimReturnCode XmimDelRelation (XmimClientHandle handle, XmimString relName); XmimReturnCode XmimVaDelRelation (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_END_ARGS);

The XmimModRelation function takes the same arguments as XmimAddRelation, with the exception of the extraorigRelName argument. This argument makes it possible to change the name of a relation. If the name of a futuresrelation is changed, the name change will propagate down to all children relations. Similarly, if the name of a relationthat has an options child is changed, the name change will propagate down to the options child. This propagationis necessary to retain consistency in the database. In the case where a relation being modified is not unique withina single database schema, origRelName must incorporate a path specification as necessary to disambiguatethe relation. The effect of XmimModRelation is to make the relation conform to the values specified in the call.If an argument is unspecified or assigned a value of NULL or invalid, then it will remain unchanged. When multipledatabases are open, the relation will be modified in all database schemas where it exists.

Note that in the case of modifying the segments of a relation using the XMIM_SEGMENTS argument, alldesired segments must be specified (even those already existing) as the changes are not additive, theyserve as replacements for existing properties of the relation.

XmimDelRelation is used to delete a relation from all currently open databases where it exists. This will delete anydata associated with the relation as well. As with the XmimAddRelation and XmimModRelation routines, thespecification of a relation that is non-unique within a single database is accommodated by allowing a path name in thespecification for relName.

Adding, Modifying and Deleting ColumnsXmimReturnCode XmimAddColumn (XmimClientHandle handle, XmimString colName, XmimString description, XmimString parentName, XmimColType type); XmimReturnCode XmimVaAddColumn (XMIM_CLIENT_HANDLE, handle, XMIM_COLUMN, colName, XMIM_DESCRIPTION, NULL, XMIM_PARENT, parentName,

158


XMIM_TYPE, XMIM_COL_NORMAL, XMIM_END_ARGS); XmimReturnCode XmimModColumn (XmimClientHandle handle, XmimString origColName, XmimString colName, XmimString description, XmimString parentName, XmimColType type); XmimReturnCode XmimVaModColumn (XMIM_CLIENT_HANDLE, handle, XMIM_ORIG_COLUMN, origColName, XMIM_COLUMN, NULL, XMIM_DESCRIPTION, NULL, XMIM_PARENT, NULL, XMIM_TYPE, XMIM_COL_INVALID, XMIM_END_ARGS); XmimReturnCode XmimDelColumn (XmimClientHandle handle, XmimString colName); XmimReturnCode XmimVaDelColumn (XMIM_CLIENT_HANDLE, handle, XMIM_COLUMN, colName, XMIM_END_ARGS);

The XmimAddColumn function adds a new column to the database. A column (e.g., Open) must be added before thecolumn can be assigned to a relation (e.g., Open of IBM). Columns may be stored in a column hierarchy, and, thus thecolumn type can be either category or normal (leaf column) with the parent column in the hierarchy specified by theparentName argument. When multiple databases are open, XmimAddColumn will result in the column being addedto all database schemas that have the category specified by parentName. As with relations, if the parent categoryname is ambiguous, a “path" to the correct category can be specified using colons (or double colons) as separators.When a non-unique column occurs within the same database, it must have a distinct path specification associatedwith it that can be used to disambiguate when referencing it. For the Commodity DataServer the user may constructthe entire column hierarchy, including the specification of the top-most column categories. TopColumn serves todistinguish the top-most columns and, therefore, is the root of the column hierarchy.

XmimModColumn modifies a column in the database, and XmimDelColumn deletes a column from the database.In the presence of multiple databases, XmimModColumn and XmimDelColumn will result in the column beingmodified in or deleted from all database schemas where it exists. If the specified column is non-unique within a singledatabase, the name (origColName in the case of modify and colName in the case of delete) must incorporate a pathspecification as necessary to disambiguate.

Adding, Modifying and Deleting Multi-Field ColumnsXmimReturnCode XmimAddMFColumn (XmimClientHandle handle, XmimString colName, XmimString description, XmimString parentName, int numFields, XmimString *fieldNames, XmimString *fieldDescriptions); XmimReturnCode XmimVaAddMFColumn (XMIM_CLIENT_HANDLE, handle, XMIM_COLUMN, colName, XMIM_DESCRIPTION, NULL, XMIM_PARENT, parentName,


159

XMIM_FIELD_LIST, fieldName1, fieldName2, NULL, XMIM_DESCRIPTION_LIST, NULL, XMIM_END_ARGS); XmimReturnCode XmimModMFColumn (XmimClientHandle handle, XmimString origColName, XmimString colName, XmimString description, XmimString parentName); XmimReturnCode XmimVaModMFColumn (XMIM_CLIENT_HANDLE, handle, XMIM_ORIG_COLUMN, origColName, XMIM_COLUMN, NULL, XMIM_DESCRIPTION, NULL, XMIM_PARENT, NULL, XMIM_END_ARGS); XmimReturnCode XmimDelMFColumn (XmimClientHandle handle, XmimString colName); XmimReturnCode XmimVaDelMFColumn (XMIM_CLIENT_HANDLE, handle, XMIM_COLUMN, colName, XMIM_END_ARGS);

The XmimAddMFColumn function adds a new multi-field column to the database. Multi-field columns are used to storedata (tick-by-tick, intraday or daily) for which there is more than one field. They are, for instance, useful for storing tick-by-tick data consisting of an arbitrary number of user-defined fields for each tick/transaction. However, the CommodityDataServer does not require that tick-by-tick data be stored using multi-field columns: if the tick-by-tick data consistsof only a single field, then a normal column may be used. Conceptually, multi-field columns can be thought of ashomogeneous relational tables. The fieldNames argument must be used to provide the field names for each fieldof the multi-field column. Field descriptions may also be provided for each field using the fieldDescriptionsargument. Multi-field columns will be assigned a type of composite by the Commodity DataServer. Multi-field columnsare always leaf columns in the column hierarchy. XmimModMFColumn can be used to modify a multi-field column inthe database, and XmimDelColumn to delete a multi-field column from the database.

Note that the fieldNames and fieldDescriptions arguments are not modifiable and, therefore, donot appear in the XmimModMFColumn routine specification.

Adding, Modifying and Deleting Relation ColumnsXmimReturnCode XmimAddRelColumn (XmimClientHandle, handle, XmimString relName, XmimString colName, XmimRelColType type, XmimRelColObjType objType, XmimRelColAggr aggregation, XmimRelColDelta delta, float constant); XmimReturnCode XmimVaAddRelColumn ( XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_COLUMN, colName, XMIM_TYPE, XMIM_RELCOL_INVALID,

160


XMIM_OBJ_TYPE, XMIM_RELCOL_OBJ_INVALID, XMIM_AGGR_RULE, XMIM_RELCOL_AGGR_INVALID, XMIM_DELTA, XMIM_RELCOL_DELTA_INVALID, XMIM_CONSTANT, 0.0, XMIM_DAILY_MULTIPLICITY, XMIM_RELCOL_MULTIPLICITY_INVALID, XMIM_INTRADAY_MULTIPLICITY, XMIM_RELCOL_MULTIPLICITY_INVALID, XMIM_TICK_MULTIPLICITY, XMIM_RELCOL_MULTIPLICITY_INVALID, XMIM_MILLISECOND_MULTIPLICITY, XMIM_RELCOL_MULTIPLICITY_INVALID, XMIM_END_ARGS); XmimReturnCode XmimModRelColumn (XmimClientHandle handle, XmimString relName, XmimString colName, XmimRelColType type, XmimRelColObjType objType, XmimRelColAggr aggregation, XmimRelColDelta delta, float constant); XmimReturnCode XmimVaModRelColumn (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_COLUMN, colName, XMIM_TYPE, XMIM_RELCOL_INVALID, XMIM_OBJ_TYPE, XMIM_RELCOL_OBJ_INVALID, XMIM_AGGR_RULE, XMIM_RELCOL_AGGR_INVALID, XMIM_DELTA, XMIM_RELCOL_DELTA_INVALID, XMIM_CONSTANT, 0.0, XMIM_DAILY_MULTIPLICITY, XMIM_RELCOL_MULTIPLICITY_INVALID, XMIM_INTRADAY_MULTIPLICITY, XMIM_RELCOL_MULTIPLICITY_INVALID, XMIM_TICK_MULTIPLICITY, XMIM_RELCOL_MULTIPLICITY_INVALID, XMIM_MILLISECOND_MULTIPLICITY, XMIM_RELCOL_MULTIPLICITY_INVALID, XMIM_END_ARGS); XmimReturnCode XmimDelRelColumn (XmimClientHandle handle, XmimString relName, XmimString colName, XmimReturnCode XmimVaDelRelColumn (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_COLUMN, colName, XMIM_END_ARGS);

The function XmimAddRelColumn adds a column to a relation. If the specified relName and colName exists inmore than one currently open database, then the relation column will be added to the occurrence in each applicabledatabase. In the Commodity DataServer, there is no uniqueness naming requirement for any relations or columns(even those that have data associated with them). Non-unique relations or columns that reside in the same database,however, must have distinct paths. To accommodate the specification of non-unique relations and columns in theXmimAddRelColumn routine, path names are allowing in the specifications for relName and colName.

If the column is a category column, then the column hierarchy will be traversed to find all leaf columns that aredescendants of that column and these will be the columns actually added to the relation. Likewise, if the relation


161

contains children relations, the appropriate columns will be added not only to the specified relation, but also to all of itschildren relations.

Note that when a relation is added to the database, all of the columns of the parent relation are inheritedand, thus, the corresponding relation columns are automatically added. This does not apply in the case ofrelations of type option; no columns are inherited for options relations.

The type indicates whether the data for the relation column is base (regular time series data), sparse (e.g., CPI) or aconstant value. The objType argument specifies the type for the data values. The options are single precision floatingpoint, double precision floating point and integer. Because the Commodity DataServer C/C++ API prototypes dealwith data values as floats, data values of type integer must be treated as floats when retrieving, updating or replacingdata. The aggregation argument defines how data for a relation column should be combined when the units of therelation column are adjusted. For example, if daily data is kept by the Commodity DataServer but the user desiresweekly data, the aggregation rule determines how the values corresponding to that week are combined to producethe weekly values. The current options include taking the first (Open) or last (Close) values, the highest (High) orlowest (Low), or their sum or average. The delta argument gives the delta increments for the values in the relationcolumn. The current options include 0, 8ths, 16ths, 32nds and 64ths. If the relation column type is constant, the valueis stored in the constant argument.

There are three extra arguments in the XmimVa version of this routine that can be used to specify individuallywhether the daily, intraday and tick data for the relation column will be single-valued or multi-valued. A multi-valuedrelation column is one in which multiple values can be stored for a single time slot as opposed to a single-valuedrelation column where there is only a single value for each time slot. Tick-by-tick data is typically multi-valued. Thearguments XMIM_DAILY_MULTIPLICITY, XMIM_INTRADAY_MULTIPLICITY, XMIM_TICK_MULTIPLICITY andXMIM_MILLISECOND_MULTIPLICITY must be set to one of the following: XMIM_RELCOL_MULTIPLICITY_INVALID XMIM_RELCOL_MULTIPLICITY_SINGLE XMIM_RELCOL_MULTIPLICITY_MULTIPLE

If not specified, daily, intraday and millisecond relation columns will be single-valued and tick-by-tick will be multi-valued.

Only the handle, relName and colName arguments must be specified in the XmimVa version. If an optional argumentis left unspecified, the values will be inherited from those of a parent relation or the nearest ancestor relation that alsocontains the appropriate column.

XmimModRelColumn modifies the attributes of a relation column. The relName and colName are used to identifythe relation column and, of course, can not be modified. As with the XmimAddRelColumn routine, the specificationof non-unique relations and columns are accommodated by supporting path names for relName and colName. TherelName can correspond to a relation with children but the modifications will not be explicitly propagated to thechildren relations. The colName, however, must be a leaf (normal) column. If the specified relation column exists inmore than one currently open database, the modification will take place for each applicable database.

XmimDelRelColumn deletes a relation column from all currently open databases where it exists. When a relationcolumn is deleted, so is the data associated with it. As with the add and modify routine, the relName and colNamespecifications accommodate non-unique relation columns by supporting path names.

162


Adding, Modifying and Deleting Multi-Field Relation ColumnsXmimReturnCode XmimAddMFRelColumn (XmimClientHandle handle, XmimString relName, XmimString colName, XmimRelColType type, int numFields, int *objTypeList, XmimRelColAggr *aggrRuleList, XmimRelColDelta *deltaList, float *constantList); XmimReturnCode XmimVaAddMFRelColumn ( XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_COLUMN, colName, XMIM_TYPE, XMIM_RELCOL_INVALID, XMIM_OBJ_TYPE_LIST, NULL, XMIM_AGGR_RULE_LIST, NULL, XMIM_DELTA_LIST, NULL, XMIM_CONSTANT_ARRAY, 0, NULL, XMIM_DAILY_MULTIPLICITY, XMIM_RELCOL_MULTIPLICITY_INVALID, XMIM_INTRADAY_MULTIPLICITY, XMIM_RELCOL_MULTIPLICITY_INVALID, XMIM_TICK_MULTIPLICITY, XMIM_RELCOL_MULTIPLICITY_INVALID, XMIM_MILLISECOND_MULTIPLICITY, XMIM_RELCOL_MULTIPLICITY_INVALID, XMIM_END_ARGS);

XmimReturnCode XmimModMFRelColumn (XmimClientHandle handle, XmimString relName, XmimString colName, XmimRelColType type, int numFields, int *objTypeList, XmimRelColAggr *aggrRuleList, XmimRelColDelta *deltaList, float *constantList); XmimReturnCode XmimVaModMFRelColumn (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_COLUMN, colName, XMIM_TYPE, XMIM_RELCOL_INVALID, XMIM_OBJ_TYPE_LIST, NULL, XMIM_AGGR_RULE_LIST, NULL, XMIM_DELTA_LIST, NULL, XMIM_CONSTANT_ARRAY, 0, NULL, XMIM_DAILY_MULTIPLICITY, XMIM_RELCOL_MULTIPLICITY_INVALID, XMIM_INTRADAY_MULTIPLICITY, XMIM_RELCOL_MULTIPLICITY_INVALID, XMIM_TICK_MULTIPLICITY, XMIM_RELCOL_MULTIPLICITY_INVALID, XMIM_MILLISECOND_MULTIPLICITY, XMIM_RELCOL_MULTIPLICITY_INVALID, XMIM_END_ARGS); XmimReturnCode XmimDelMFRelColumn (XmimClientHandle handle, XmimString relName, XmimString colName,


163

XmimReturnCode XmimVaDelMFRelColumn (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_COLUMN, colName, XMIM_END_ARGS);

The function XmimAddMFRelColumn adds a multi-field column (i.e. a column of type composite) to a relation. Thetype argument, as with the type argument used in the XmimAddRelColumn routine, specifies whether the data forthe relation column is base, sparse or a constant value.

Note that all fields of a multi-field column must be of the same relation column type. The multiplicityarguments available with the XmimVa version are just as with the XmimVaAddRelColumn routine.

The objTypeList argument specifies the data type for each field of the multi-field column. It is analogous tothe objType argument used in the XmimAddRelColumn routine. The options are single precision float, doubleprecision float, integer, and block. All types are specified using the XmimRelColObjType enumeration as in theXmimAddRelColumn routine. As with objType, data values of type integer must be treated as floats whenretrieving, updating or replacing data.

The aggrRuleList, deltaList and constantList arguments specify the aggregation rules, delta incrementsand constant values, respectively, for each field of the multi-field column. The length of the object type, aggregation,delta and constant lists, if specified, must be the same as the number of fields specified for the multi-field column.

Note that, since constantList is a list of numbers (and a constant value of 0.0 would conflict withNULL), the Va routines must use the array specification instead of the list specification for this argument.

The list arguments are all optional; only the handle, relName and colName arguments must be specified for theXmimVaAddMFRelColumn routine. For any list arguments not specified, all fields will inherit values from the parent ornearest ancestor relation as in the XmimVaAddRelColumn routine.

XmimModMFRelColumn can be used to modify the object type, aggregation rule, delta and constant lists of a multi-field relation column. The length of these lists must not be changed; they must remain the same as the number offields for the multi-field column. XmimDelMFRelColumn can be used to delete a multi-field relation column. The dataassociated with the multi-field relation column will be deleted as well.

Reading and Writing ASCII Data FilesXmimReturnCode XmimReadFacts (XmimClientHandle handle, XmimString inputFileName, XmimString relName, int numColumns, XmimString *colNames, int numFields,

164


XmimFieldFormat *fieldFormats, XmimRecordFormat recordFormat, XmimMergeMode mergeMode, XmimTickMode tickMode); XmimReturnCode XmimVaReadFacts (XMIM_CLIENT_HANDLE, handle, XMIM_FILENAME, inputFileName, XMIM_RELATION, NULL, XMIM_COLUMN_LIST, NULL, XMIM_FORMAT_LIST, NULL, XMIM_RECORD_FORMAT, XMIM_RECFMT_FREE, XMIM_MERGE_MODE, XMIM_MERGE, XMIM_TICK_MODE, XMIM_TICK_NONE, XMIM_END_ARGS); XmimReturnCode XmimWriteFacts (XmimClientHandle handle, XmimString outputFileName, int numRelations, XmimString *relNames, int numColumns, XmimString *colNames, int numFields, XmimFieldFormat *fieldFormats, int numUnits, XmimUnits units, XmimDateTime unitsFrom, XmimDate fromDate, XmimDate toDate, XmimTime fromTime, XmimTime toTime); XmimReturnCode XmimVaWriteFacts (XMIM_CLIENT_HANDLE, handle, XMIM_FILENAME, NULL, XMIM_RELATION_LIST, relName1, relName2, ..., NULL, XMIM_COLUMN_LIST, NULL, XMIM_FORMAT_LIST, NULL, XMIM_UNITS, 1, XMIM_DAYS, XMIM_DATE_TIME, XMIM_INVALID_DATE_TIME, XMIM_FROM_DATE, XMIM_INVALID_DATE, XMIM_TO_DATE, XMIM_INVALID_DATE, XMIM_FROM_TIME, XMIM_INVALID_TIME, XMIM_TO_TIME, XMIM_INVALID_TIME, XMIM_END_ARGS);

The routine XmimReadFacts reads the ASCII file inputFileName and loads it into the Commodity DataServerdatabase. Conversely, the routine XmimWriteFacts extracts data from the Commodity DataServer databaseand writes it into an ASCII file (or stdout if the output file is NULL). When multiple databases are open, theXmimReadFacts routine will load data to the first applicable database; that is, if daily data is added to Close ofIBM then it is added to the first applicable database that has the relation IBM with a Close column. Likewise, in thepresence of multiple databases, XmimWriteFacts will retrieve data from the first database that has data for thespecified relation column.

If the columns are not specified, these routines will use Open, High, Low, Close as the default columns. ThetickMode argument of XmimReadFacts is used to indicate that tick data is being read and further specify whetherthe tick data is intraday, tick-by-tick (second) or millisecond. The mergeMode argument of XmimReadFacts is used toindicate whether the data read in will serve as an update or replacement to existing data.


165

With XmimWriteFacts, the units argument is used to specify the type of data to be written, with intraday versustick-by-tick, versus millisecond data being differentiated as such: millisecond will indicate milliseconds, secondswill indicate tick-by-tick, and minutes or hours will indicate intraday. If tick-by-tick is indicated and no second dataexists then millisecond data (if available) will be aggregated into the desired frequency. If intraday is indicated and nointraday data exists, tick-by-tick data will be aggregated into the desired frequency and if tick-by-tick is not availablethen millisecond data will be aggregated. Since the usage of intraday data will be preferred for aggregating intominutes or hours, if both tick-by-tick data and intraday data exist and the use of tick-by-tick is desired, the units mustbe specified in terms of seconds (e.g., 120 seconds in lieu of 2 minutes). Likewise, for aggregating into seconds, ifboth second and millisecond data exist and the use of millisecond is desired, the units must be specified in termsof milliseconds (e.g., 2000 milliseconds in lieu of 2 seconds). The unitsFrom argument of XmimWriteFacts canbe either a time (for tick data) or a date (for daily data) and provides a point of reference for the desired aggregationperiod. For example, if the units specification is 1 hour and the unitsFrom is 8:30, the aggregation will be hourlyending on the half hour. The fromDate, toDate, fromTime and toTime arguments of XmimWriteFacts are usedto narrow the range of data that is written.

The recordFormat argument for XmimReadFacts specifies whether the records in the file should be read assumingfixed-format fields or free-format fields. For both XmimReadFacts and XmimWriteFacts, the fields for eachrecord and the corresponding formats are given by the fieldFormats argument. For more detail on the use ofthe fieldFormats argument, refer to “Writing Data to a File”. If options data is to be read or written, then thefieldFormats must include specifications for the three options field types, namely, XMIM_FIELD_EXP_DATE,XMIM_FIELD_OPT_TYPE, and XMIM_FIELD_STRIKE.

Assigning Holiday SchedulesXmimReturnCode XmimSetHolidaySchedule (XmimClientHandle handle, XmimString relOrExchangeName, XmimString fileName); XmimReturnCode XmimVaSetHolidaySchedule (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION NULL, XMIM_EXCHANGE NULL, XMIM_FILENAME fileName, XMIM_END_ARGS);

The XmimSetHolidaySchedule routine can be used to associate a holiday schedule with a relation or an exchange.Thus, different (foreign) holiday schedules can be introduced and utilized for determining null values within XMIM.The holiday schedule is given by the fileName argument. The file should consist of a series of holiday dates, oneper line. The date format must be one of the valid Commodity DataServer date formats. A date range may also bespecified by providing two dates with the “-" character in between, again on a single line14. Each line may optionallyhave a character string which will be taken as a comment (e.g., 650531,MEMORIAL_DAY). Either a relation nameor exchange name may be given to indicate the relation or exchange that the holiday schedule is to be assigned to.The relation may be a category such that its children in the relation hierarchy will inherit the given holiday schedule.If relOrExchangeName is NULL, the given holiday schedule will be set as the default holiday schedule. In theDataServer, the appropriate holiday schedule to be used for a particular relation will be determined first by searchingthe relation hierarchy for a holiday schedule assigned to the relation or an ancestor, then searching for a holidayschedule assigned to the exchange for that relation. If all else fails, the default holiday schedule will be used.

14 This may be useful for foreign holidays that consist of multiple days.

166


Note that a database lock must be obtained before setting a holiday schedule.

Adding Relation AliasesXmimReturnCode XmimAddRelationAlias(XmimClientHandle handle, XmimString parentName, XmimString origRelName, XmimString relName); XmimReturnCode XmimVaAddRelationAlias(XMIM_CLIENT_HANDLE, handle, XMIM_PARENT, parentName, XMIM_ORIG_RELATION, origRelName, XMIM_RELATION, relName, XMIM_END_ARGS);

The XmimAddRelationAlias command is used to add an alias for a given relation. The relName specifies the aliasto be used for the relation. The parentName supplies the parent for the alias (category where the alias will reside)and origRelName specifies what the alias points to. The parentName and origRelName must include paths if thenames alone are ambiguous and the path is necessary to disambiguate. If there is ambiguity and path names are notspecified, an error message will be returned.

The alias facility provides a mechanism whereby multiple paths to a relation may be established (e.g., Equities maybe organized by countries in addition to the current organization). It also provides for referring to relations by alternatenames (e.g., aliases can be created such that relations can be referred to by the CUSIP numbers in addition to theticker symbols).

Returning Corresponding Relation Name for Supplied Alias NameXmimReturnCode XmimGetTargetforRelationAlias (XmimClientHandle handle, char *rel, char **target);

The XmimGetTargetforRelationAlias command is used to return the corresponding relation name for a givenalias name. The “rel” name specifies the name of the alias. The “target” name specifies the returned relation name. Ifthe returned “target” name is the same as the “rel” name supplied, then the “rel” name is not an alias but an actualrelation name.

Printing Database Schema InformationXmimReturnCode XmimPrintSchema (XmimClientHandle handle, XmimString relName, XmimString fileName, int maxDepth, XmimBoolean allContracts, XmimAppendMode appendMode, XmimBoolean print, XmimBoolean verbose);


167

XmimReturnCode XmimVaPrintSchema (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION_LIST, NULL, XMIM_FILENAME, NULL, XMIM_MAX_DEPTH, -1, XMIM_ALL_CONTRACTS, False, XMIM_APPEND_MODE, XMIM_REPLACE, XMIM_PRINT, False, XMIM_VERBOSE, False, XMIM_END_ARGS);

The XmimPrintSchema routine prints information about the relations stored in the database to the given fileName.If the output file name is not specified (NULL), then the information will be printed to stdout. If one or more relationsare specified, XmimPrintSchema will print information pertaining to those relations only. If relName is NULL,information about all the relations in the database will be printed. If maxDepth is given as -1, the information for alldescendants of a relation category will be recursively printed as well. The maxDepth argument can be used to limitthe level of descendants printed for a relation category. If maxDepth is 0, only information pertaining to the categoryitself is printed. If maxDepth is 1 then information for the immediate children will also be printed (but not for theirdescendants), and so on. In general, maxDepth specifies the number of generations of children for which informationwill be printed. The allContracts flag is relevant only for futures relations. If this flag is True, then information willbe printed for all individual futures contracts existing in the hierarchy beneath the futures relation. If allContractsis False, the individual contracts will not be printed. Output can be sent to a printer by using the print flag. TheappendMode argument can be used to specify whether an existing file should be replaced or appended to. If theverbose flag is False, the relation name, columns defined for the relation and the starting and ending dates for thedata are printed. Setting this flag to True will result in all relation and relation column fields also being printed. Whenmore than one database is open, hierarchy information is merged as consistent with viewing multiple databases asthough they were a single database. Where duplication exists, the schema information shown will be that from thefirst applicable database.

Generating Continuous ContractsThe XmimRegenRollover command creates a continuous prompt contract from a group of underlying contracts.

Normally, continuous contracts are automatically created and updated by the Commodity DataServer. Wheneverpossible, the Commodity DataServer optimizes the updating of continuous contracts such that most types ofcontinuous contracts are updated incrementally and are not completely re-generated with each update. However,sometimes it is desirable to have the continuous contracts for a given symbol completely re-computed. For example, ifchanges are made to futures contracts and need to be recomputed than the XmimRegenRollover command can beused for this purpose. Its synopsis is:XmimReturnCode XmimRegenRollover(XmimClientHandle handle, char *relName, XmimDataType type);

where the choices for type are: typedef enum { XMIM_DATA_TICK, /* Seconds */ XMIM_DATA_INTRADAY, /* Minutes */ XMIM_DATA_DAILY, /* Day */ XMIM_DATA_ALL, /* Each Type is regenerated */

168


XMIM_DATA_MILLISECOND /* Milliseconds */ } XmimDataType;

If you have a continuous contract that is spotty while the underlying contracts do not have holes then tryrunning the XmimRegenRollover command. It usually fixes this type of problem.

The command can be run at any time and by itself. The *relName argument is optional. If it is not given, allcontinuous contracts will be re-computed for all futures and futures_continuous symbols in the database. This willtake some time to finish as it has to regen every continuation in the Commodity DataServer.

If *relName is of type category or futures, then continuous contracts will be re-computed for all futures andfutures_continuous symbols occurring in that sub-tree of the symbol hierarchy.

Schema Browsing

Navigating the Relation HierarchyXmimReturnCode XmimGetRelChildren (XmimClientHandle handle, XmimString relName, int *numChildren, XmimString **children); XmimReturnCode XmimVaGetRelChildren (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_RELATION_ARRAY, &numChildren, &children, XMIM_END_ARGS); XmimReturnCode XmimGetRelParent (XmimClientHandle handle, XmimString relName, XmimString *parent); XmimReturnCode XmimVaGetRelParent (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_PARENT, &parent, XMIM_END_ARGS); XmimReturnCode XmimGetRelRoot (XmimClientHandle handle, XmimString relName, XmimString *root); XmimReturnCode XmimVaGetRelRoot (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_ROOT, &root, XMIM_END_ARGS); XmimReturnCode XmimGetRelPath (XmimClientHandle handle, XmimString relName, XmimString *path); XmimReturnCode XmimVaGetRelPath (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_PATH, &path, XMIM_END_ARGS);


169

XmimReturnCode XmimGetRelType (XmimClientHandle handle, XmimString relName, XmimRelType *type); XmimReturnCode XmimVaGetRelType (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_TYPE, &type, XMIM_END_ARGS); XmimReturnCode XmimRelHasChildren (XmimClientHandle handle, XmimString relName, XmimBoolean *hasChildren); XmimReturnCode XmimVaRelHasChildren (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_HAS_CHILDREN, &hasChildren, XMIM_END_ARGS);XmimReturnCode XmimGetAllRelations (XmimClientHandle handle, XmimString relName, XmimSearchFilter filter, int *numRelations, XmimString **relations); XmimReturnCode XmimVaGetAllRelations (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, NULL, XMIM_SEARCH_FILTER, XMIM_SEARCH_ALL, XMIM_RELATION_ARRAY, &numRelations, &relations, XMIM_END_ARGS); XmimReturnCode XmimFindRelations (XmimClientHandle handle, XmimString pattern, XmimBoolean byName, XmimSearchFilter filter, XmimBoolean caseSensitive, int *numRelations, XmimString **relations); XmimReturnCode XmimVaFindRelations (XMIM_CLIENT_HANDLE, handle, XMIM_PATTERN, pattern, XMIM_SEARCH_BY_NAME, byName, XMIM_INCLUDE_HIDDEN, False, XMIM_SEARCH_FILTER, XMIM_SEARCH_ALL, XMIM_CASE_SENSITIVE, False, XMIM_RELATION_ARRAY, &numRelations, &relations, XMIM_END_ARGS);

The XmimGetRelChilden and XmimGetRelParent routines are used to navigate the relation tree hierarchy.XmimGetRelChildren returns the number of immediate children relations and the relation names of these children.XmimGetRelParent returns the name of the parent relation. XmimGetRelRoot returns the name of the top-mostcategory of the given relation. For example, the root of SP will be Futures, regardless of how deeply nested SP is in themenu hierarchy. XmimGetRelPath returns the full path for the specified relation. XmimGetRelType returns the typeof a given relation. The XmimRelHasChildren routine returns True if the specified relation has children of any typeand False otherwise.

The XmimGetAllRelations routine retrieves all the descendants, not just the immediate ones, of relName.The descendant relations are returned in the relations argument with numRelations specifying how manydescendants were found. If relName is not specified, TopRelation will be used such that all the relations in thedatabase will be retrieved. If the filter argument is set to XMIM_SEARCH_LEAVES only the descendants that are

170


leaf relations in the hierarchy, that is, those that have data associated with them, will be retrieved; if it is set toXMIM_SEARCH_NON_LEAVES, then only the descendants that do not have data associated with them (i.e., relationsof type category) will be retrieved. XMIM_SEARCH_ALL should be used to return all descendants in the hierarchy.

The XmimFindRelations routine finds all relations that match a given pattern, thereby providing a search capabilityinto the schema. The byName argument is used to indicate whether the search should be performed on relationnames or relation descriptions and the caseSensitive argument indicates whether the search should be casesensitive or not. The default is to search by relation name. In both cases, the relations argument returns the namesfor all relations where a pattern match is found and the numRelations argument returns the number of relationsfound. The pattern may consist of any number of wildcard (*) characters. Furthermore, wildcard characters maybe used anywhere within the pattern string. For example, the pattern "*Bus*" would match the description for IBM(International Business Machines), among others. As with the XmimGetAllRelations function, the filterargument can be used to restrict the relations returned. The XMIM_INCLUDE_HIDDEN argument is of the typeXmimBoolean. Setting this argument to true indicates that you want the search to include hidden .Internal relations.

This is an internal feature and will return results that are not of interest to customers.

For all of these routines, if the relations specified are non-unique, then a disambiguating path name, using colons asseparators, must be specified as described in “Adding, Modifying and Deleting Relations”. If a relation path is specifiedin the XmimFindRelations call, only the last component of the path is allowed to contain wildcard characters (e.g.,Equities:a::*A*). With the exception of the XmimGetRelType routine and the XmimRelHasChildren routine, all ofthe above-mentioned routines return a relation or list of relations and the return code will be XMIM_WARNING in theevent that no answers are found and, therefore, no relations are returned. If a returned relation name is not unique,then the full relation path name will be given.

Navigating the Column HierarchyXmimReturnCode XmimGetColChildren (XmimClientHandle handle, XmimString colName, int *numColumns, XmimString **children); XmimReturnCode XmimVaGetColChildren (XMIM_CLIENT_HANDLE, handle, XMIM_COLUMN, colName, XMIM_COLUMN_ARRAY, &numChildren, &children, XMIM_END_ARGS); XmimReturnCode XmimGetColParent (XmimClientHandle handle, XmimString colName, XmimString *parent); XmimReturnCode XmimVaGetColParent (XMIM_CLIENT_HANDLE, handle, XMIM_COLUMN, colName, XMIM_PARENT, &parent, XMIM_END_ARGS); XmimReturnCode XmimGetColRoot (XmimClientHandle handle, XmimString colName,


171

XmimString *root); XmimReturnCode XmimVaGetColRoot (XMIM_CLIENT_HANDLE, handle, XMIM_COLUMN, colName, XMIM_ROOT, &root, XMIM_END_ARGS); XmimReturnCode XmimGetColPath (XmimClientHandle handle, XmimString colName, XmimString *path); XmimReturnCode XmimVaGetColPath (XMIM_CLIENT_HANDLE, handle, XMIM_COLUMN, colName, XMIM_PATH, &path, XMIM_END_ARGS); XmimReturnCode XmimGetColType (XmimClientHandle handle, XmimString colName, XmimColType *type); XmimReturnCode XmimVaGetColType (XMIM_CLIENT_HANDLE, handle, XMIM_COLUMN, colName, XMIM_TYPE, &type, XMIM_END_ARGS); XmimReturnCode XmimColHasChildren (XmimClientHandle handle, XmimString colName, XmimBoolean *hasChildren); XmimReturnCode XmimVaColHasChildren (XMIM_CLIENT_HANDLE, handle, XMIM_COLUMN, colName, XMIM_HAS_CHILDREN, &hasChildren, XMIM_END_ARGS); XmimReturnCode XmimGetAllColumns (XmimClientHandle handle, XmimString colName, XmimSearchFilter filter, int *numColumns, XmimString **columns);

XmimReturnCode XmimVaGetAllColumns (XMIM_CLIENT_HANDLE, handle, XMIM_COLUMN, NULL, XMIM_SEARCH_FILTER, XMIM_SEARCH_ALL, XMIM_COLUMN_ARRAY, &numColumns, &columns, XMIM_END_ARGS); XmimReturnCode XmimFindColumns (XmimClientHandle handle, XmimString pattern, XmimBoolean byName, XmimSearchFilter filter, XmimBoolean caseSensitive, int *numColumns, XmimString **columns); XmimReturnCode XmimVaFindColumns (XMIM_CLIENT_HANDLE, handle, XMIM_PATTERN, pattern, XMIM_SEARCH_BY_NAME, byName, XMIM_SEARCH_FILTER, XMIM_SEARCH_ALL, XMIM_CASE_SENSITIVE, False, XMIM_COLUMN_ARRAY, &numColumns, &columns, XMIM_END_ARGS);

These routines behave analogously to the relation routines, and allow the client to navigate through the columnhierarchy.

172


Accessing the Columns of a RelationXmimReturnCode XmimGetRelColumns (XmimClientHandle handle, XmimString relName, int *numColumns, XmimString **colNames); XmimReturnCode XmimVaGetRelColumns (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_COLUMN_ARRAY, &numColumns, &colNames, XMIM_END_ARGS); XmimReturnCode XmimGetRelColumnsWithData (XmimClientHandle handle, XmimString relName, int *numColumns, XmimString **colNames); XmimReturnCode XmimVaGetRelColumnsWithData (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_COLUMN_ARRAY, &numColumns, &colNames, XMIM_END_ARGS);

The routine XmimGetRelColumns returns the columns for a given relation in the colNames argument. The number ofcolumns is returned in the numColumns argument. The routine XmimGetRelColumnsWithData returns only thosecolumns for a given relation which have had data added to them.

Note that if a relation column exists with only NaN values, it will be returned as having data. Also, if arelation column previously had data that was subsequently deleted (e.g., using XmimDeleteFacts),it will still be returned by this routine. These nuances are necessary in order to make this routine asefficient as possible. In the event that it is important to identify only those columns of the relation whichcurrently have non-NaN data, the client can check the data range (using the XmimGetDataRangeroutine) for each column returned by XmimGetRelColumnsWithData.

Accessing the Attributes of Relations, Columns and Relation ColumnsXmimReturnCode XmimGetRelation (XmimClientHandle handle, XmimString relName, XmimString *description, XmimString *parentName, XmimRelType *type, XmimString *exchange, XmimString *time_zone, XmimTime *startTrade, XmimTime *endTrade, float *contractUnits, XmimDate *expirationDate, XmimDate *firstNoticeDate, XmimString *rolloverDay, XmimString *rolloverPolicy XmimRolloverDataType *rolloverData); XmimReturnCode XmimVaGetRelation (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_DESCRIPTION, &description,


173

XMIM_PARENT, &parentName, XMIM_TYPE, &type, XMIM_EXCHANGE, &exchange, XMIM_TIME_ZONE, &time_zone, XMIM_START_TRADE, &startTrade, XMIM_END_TRADE, &endTrade, XMIM_CONTRACT_UNITS, &contractUnits, XMIM_EXPIRATION_DATE, &expirationDate, XMIM_FIRST_NOTICE_DATE, &firstNoticeDay, XMIM_ROLLOVER_DAY, &rolloverDay, XMIM_ROLLOVER_DAY_STRUCT, &rolloverDayStruct, XMIM_ROLLOVER_POLICY, &rolloverPolicy, XMIM_ROLLOVER_POLICY_STRUCT, &rolloverPolicyStruct, XMIM_ROLLOVER_DATA_TYPE, &rolloverData, XMIM_TRADING_PATTERN, &tradingPattern, XMIM_SEGMENTS, &numSegments, &segments, XMIM_END_ARGS); XmimReturnCode XmimGetColumn (XmimClientHandle handle, XmimString colName, XmimString *description, XmimString *parentName, XmimColType *type); XmimReturnCode XmimVaGetColumn (XMIM_CLIENT_HANDLE, handle, XMIM_COLUMN, colName, XMIM_DESCRIPTION, &description, XMIM_PARENT, &parentName, XMIM_TYPE, &type, XMIM_END_ARGS); XmimReturnCode XmimGetMFColumn (XmimClientHandle handle, XmimString colName, XmimString *description, XmimString *parentName, int *numFields, XmimString **fieldNames, XmimString **fieldDescriptions); XmimReturnCode XmimVaGetMFColumn (XMIM_CLIENT_HANDLE, handle, XMIM_COLUMN, colName, XMIM_DESCRIPTION, &description, XMIM_PARENT, &parentName, XMIM_FIELD_ARRAY, &numFields, &fieldNames, XMIM_DESCRIPTION_ARRAY, &numFields, &fieldDescriptions, XMIM_END_ARGS); XmimReturnCode XmimGetRelColumn (XmimClientHandle handle, XmimString relName, XmimString colName, XmimRelColType *type, XmimRelColObjType *objType, XmimRelColAggr *aggregation, XmimRelColDelta *delta, float *constant); XmimReturnCode XmimVaGetRelColumn (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_COLUMN, colName,

174


XMIM_TYPE, &type, XMIM_OBJ_TYPE, &objType, XMIM_AGGR_RULE, &aggregation, XMIM_DELTA, &delta, XMIM_CONSTANT, &constant, XMIM_END_ARGS); XmimReturnCode XmimGetMFRelColumn (XmimClientHandle handle, XmimString relName, XmimString colName, XmimRelColType type, int *numFields, int **objTypeList, XmimRelColAggr **aggrRuleList, XmimRelColDelta **deltaList, float **constantList); XmimReturnCode XmimVaGetMFRelColumn (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_COLUMN, colName, XMIM_TYPE, &type, XMIM_NUM_FIELDS, &numFields, XMIM_OBJ_TYPE_ARRAY, &numObjTypes, &objTypes, XMIM_AGGR_RULE_ARRAY, &numAggrRules, &aggrRules, XMIM_DELTA_ARRAY, &numDeltas, &deltas, XMIM_CONSTANT_ARRAY, &numConstants, &constants, XMIM_END_ARGS);

The routines above provide access to all the attributes of relations, columns, multi-field columns, relation columNS andmulti-field relation columns.

Note that for the XmimVaGetMFRelColumn routine, the object types, aggregation rules, delta valuesand constant values are returned using the array specification and that the number of elements for eacharray will be the number of fields in the multi-field column.

Accessing Exchanges and Time ZonesXmimReturnCode XmimGetAllExchanges (XmimClientHandle handle, int *numExchanges, XmimString **exchanges); XmimReturnCode XmimVaGetAllExchanges (XMIM_CLIENT_HANDLE, handle, XMIM_EXCHANGE_ARRAY, &numExchanges, &exchanges, XMIM_END_ARGS); XmimReturnCode XmimGetAllTimeZones (XmimClientHandle handle, int *numTimeZones, XmimString **timeZones); XmimReturnCode XmimVaGetAllExchanges (XMIM_CLIENT_HANDLE, handle, XMIM_TIME_ZONE_ARRAY, &numTimeZones, &timeZones, XMIM_END_ARGS);


175

The XmimGetAllExchanges and XmimGetAllTimeZones routines return all exchanges and time zones,respectively, in the Commodity DataServer database.

Accessing Futures Expiration DatesXmimReturnCode XmimGetExpirDates (XmimClientHandle handle, XmimString root, int *numFuturesRels, XmimContractsExpirDates **expirDatesPerFutures); XmimReturnCode XmimVaGetExpirDates (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, NULL, XMIM_FUTURES_ARRAY, &numFuturesRels, &expirDatesPerFutures, XMIM_END_ARGS);

The XmimGetExpirDates routine retrieves all contracts and their associated expiration dates for all futures relationsexisting in the root subtree. By default, information will be retrieved for all futures relations in the database. Theinformation is returned via the expirDatesPerFutures argument where, for each futures relation with validcontracts and valid expiration dates, the name of the futures relation, number of contracts, contract names andexpiration dates will be given.

Reading and Writing Data

Retrieving Data RangesXmimReturnCode XmimGetDataRange (XmimClientHandle handle, XmimString relName, int numColumns, XmimString *colNames, XmimUnits units, XmimDate *fromDate, XmimDate *toDate); XmimReturnCode XmimVaGetDataRange (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_COLUMN_LIST, NULL, XMIM_UNITS, XMIM_DAYS, XMIM_FROM_DATE, &fromDate, XMIM_TO_DATE, &toDate, XMIM_END_ARGS); XmimReturnCode XmimGetDailyRange (XmimClientHandle handle, XmimString relName, int numColumns, XmimString *colNames, XmimDate *fromDate, XmimDate *toDate); XmimReturnCode XmimGetTickRange (XmimClientHandle handle, XmimString relName, int numColumns,

176


XmimString *colNames, XmimDate *fromDate, XmimDate *toDate);

The XmimGetDataRange routine determines the range over which the type of data given by the units argumentis available. Therefore, to get the data range for daily data, XMIM_DAYS should be used; to get the data range forintraday tick data, XMIM_MINUTES should be used; to get the data range for tick-by-tick data, XMIM_SECONDSshould be used; to get the data range for millisecond data, XMIM_MILLISECONDS should be used. If the units arenot specified, XMIM_DAYS will be used as the default. The data range is given such that the first date is returnedas fromDate and the last as toDate. The colNames argument can be used to limit the columns considered indetermining the data range of a relation. If the argument is not used, all columns for the relation will be considered.The XmimGetDataRange routine returns XMIM_WARNING for the case where there is no data and, therefore,the fromDate and toDate arguments will be set to XMIM_INVALID_DATE. It will return XMIM_ERROR if thecall was not successful due to, for instance, an invalid relation or column name. The XmimGetDailyRange andXmimGetTickRange routines return the range over which daily or intraday tick data is available, respectively. Theyare provided merely for backward compatibility as this functionality is subsumed by XmimGetDataRange.

Retrieving Time RangesXmimReturnCode XmimGetTradingTime (XmimClientHandle handle, XmimString relName, int numColumns, XmimString *colNames, XmimUnits units, XmimDate tradingDate, XmimTime *startTime, XmimTime *endTime); XmimReturnCode XmimVaGetTradingTime (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_COLUMN_LIST, NULL, XMIM_UNITS, XMIM_DAYS, XMIM_DATE, tradingDate, XMIM_FROM_TIME, &startTime, XMIM_TO_TIME, &endTime, XMIM_END_ARGS);

The XmimGetTradingTime routine can be used to return the actual trading time range on a given date. This is therange over which data for the relation actually exists on the date specified as tradingDate (as opposed to thestart and end trading times stored for a relation). The colNames argument is used to limit the columns considered indetermining the time range. If the argument is not used, all columns for the relation will be considered. The range isreturned as startTime and endTime. Of course, this routine is only meaningful for intraday, tick or millisecond dataand, therefore, if a daily time series is specified, the return times will be set to XMIM_INVALID_TIME.

Retrieving Data RecordsXmimReturnCode XmimGetRecords (XmimClientHandle handle, int numRelations, XmimString *relNames, int numColumns,


177

XmimString *colNames, XmimDate fromDate, XmimDate toDate, XmimTime fromTime, XmimTime toTime, int numUnits, XmimUnits units, XmimFillOption holidayFillOption, XmimFillOption missDataFillOption, XmimSkipAllNaN skipAllNaNRecords, int limit, XmimLimitMode limitMode, int *numRecords, XmimDateTime **dates, float **values); XmimReturnCode XmimVaGetRecords (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION_LIST, relName1, relName2, ..., NULL, XMIM_COLUMN_LIST, NULL, XMIM_FROM_DATE, XMIM_INVALID_DATE, XMIM_TO_DATE, XMIM_INVALID_DATE, XMIM_FROM_TIME, XMIM_INVALID_TIME, XMIM_TO_TIME, XMIM_INVALID_TIME, XMIM_UNITS, 1, XMIM_DAYS, XMIM_FILL_OPTION, XMIM_FILL_NAN, XMIM_FILL_NAN, XMIM_SKIP_NONE, XMIM_LIMIT, 0, XMIM_LIMIT_BY_RECORDS, XMIM_CORRECTIONS_DATE, XMIM_INVALID_DATE, XMIM_CURRENT_TICK_USAGE, XMIM_APPEND_TO_NONE, XMIM_NUM_RECORDS, &numRecords, XMIM_DATE_TIMES, &dates, XMIM_VALUES, &values, XMIM_TIME_RANGE_TYPE, XMIM_TRADING_RANGE, XMIM_END_ARGS); XmimReturnCode XmimGetRecordsAllColumns (XmimClientHandle handle, int numRelations, XmimString *relNames, XmimDate fromDate, XmimDate toDate, XmimTime fromTime, XmimTime toTime, int numUnits, XmimUnits units, XmimFillOption holidayFillOption, XmimFillOption missDataFillOption, XmimSkipAllNaN skipAllNaNRecords, int limit, XmimLimitMode limitMode, int *numColumns, XmimString **colNames, int *numRecords, XmimDateTime **dates, float **values);

178


XmimReturnCode XmimVaGetRecordsAllColumns (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION_LIST, relName1, relName2, ..., NULL, XMIM_FROM_DATE, XMIM_INVALID_DATE, XMIM_TO_DATE, XMIM_INVALID_DATE, XMIM_FROM_TIME, XMIM_INVALID_TIME, XMIM_TO_TIME, XMIM_INVALID_TIME, XMIM_UNITS, 1, XMIM_DAYS, XMIM_FILL_OPTION, XMIM_FILL_NAN, XMIM_FILL_NAN, XMIM_SKIP_NONE, XMIM_LIMIT, 0, XMIM_LIMIT_BY_RECORDS, XMIM_CORRECTIONS_DATE, XMIM_INVALID_DATE, XMIM_CURRENT_TICK_USAGE, XMIM_APPEND_TO_NONE, XMIM_COLUMN_ARRAY, &numColumns, &colNames, XMIM_NUM_RECORDS, &numRecords, XMIM_DATE_TIMES, &dates, XMIM_VALUES, &values, XMIM_END_ARGS);

The XmimGetRecords and XmimGetRecordsAllColumns routines are used to access data from one or morerelations. XmimGetRecords allows the columns for which data is desired to be specified as an argument. In additionto normal columns, multi-field columns or their fields may be specified in the column list. If a multi-field column isspecified, then data for all fields of that multi-field column will be retrieved. When fields of a multi-field column arespecified, if the field name is not unique across all columns in the database, the multi-field column name must begiven as a qualifier: mf_column_name:field_name. If the colNames argument is not specified, XmimGetRecordswill use the bars (Open, High, Low, Close) for the relations as the default columns. XmimGetRecordsAllColumnsretrieves data for all existing columns of the specified relations returning the number of columns as well as the columnnames. If multiple relations are specified, the union of all columns of all relations will be returned, filling with NaNs asappropriate.

The range over which to retrieve the data is specified by using the fromDate and toDate arguments. If the datesare set to XMIM_INVALID_DATE (or left as defaults), then all the data will be retrieved. By specifying the samedate for fromDate as for toDate, a single record may be retrieved. Also, if the fromDate is specified but thetoDate is left as invalid, the result will be to retrieve data from the specified date to the last date for which data isavailable. Likewise, if the toDate is specified but the fromDate is left as invalid, data from the first date for whichdata is available to the specified date will be retrieved. In the context of intraday or tick-by-tick data, the time rangearguments can be used to specify the appropriate trading time range over which to retrieve data for each date in thedate range. They behave in a manner analogous to the date range arguments with respect to defaulting; that is, iffromTime is set to XMIM_INVALID_TIME then it defaults to the start time of the trading period and if toTime is setto XMIM_INVALID_TIME then it defaults to the end time of the trading period.

The numUnits and units arguments are used to specify the type of data to retrieve and the frequency into whichit should be aggregated (e.g., 10 minute or weekly). Millisecond data is retrieved by using XMIM_MILLISECONDSfor the units argument. Real tick data is retrieved by using XMIM_SECONDS for the units argument; if real tick data isnot available, then millisecond data (if available) will be used and aggregated appropriately. Intraday data is retrievedwhen either XMIM_MINUTES or XMIM_HOURS is used; if intraday data is not available, then real tick data will be usedand aggregated appropriately and if that is not available then millisecond data will be used if available. Daily data isretrieved when XMIM_DAYS, XMIM_WEEKS, XMIM_MONTHS, XMIM_QUARTERS or XMIM_YEARS is used.


179

The holidayFillOption and missDataFillOption arguments provide a means to specify, individually, howinvalid data due to holidays and missing data should be handled in the answer array. The alternatives are to returnNaNs, to carry over the previous value, to use the first following value, or to interpolate either linearly, geometricallyor logarithmically. The skipAllNaNRecords argument can be used to indicate that entire rows of invalid data,whether due to holidays or missing data, should be skipped over and not returned as part of the answer array. In theXmimVaGetRecords routine, the holiday and missing data fill option arguments as well as the skip NaNs argumentare all given with the XMIM_FILL_OPTION token, in that order. For daily data, NaNs will be returned by default in theanswer array even for rows of invalid data. For tick data, however, rows of invalid data are skipped.

The limitMode argument is used when the user wishes to restrict the results to a certain amount of memory ornumber of records. If the amount specified is exceeded, only the least recent data is retrieved.

The Va versions of these routines provide for an extra argument that is useful when corrections have been added tothe database (refer to “Correction Audits” ). When the XMIM_CORRECTIONS_DATE argument is specified, valuesare accessed as they would have appeared on the given date. That is, if a correction was added on June 5, 1997 andthe XMIM_CORRECTIONS_DATE specified in the XmimVaGetRecords call is before June 5, the old value will bereturned; on the other hand, if the date specified is after June 5 then the new corrected value will be returned. If thecorrections date is not specified, then it will be taken as the last data date such that any and all corrections will havebeen applied.

The Va versions of these routines also provide for an extra argument that can be used to specify that datafrom the high-frequency current tick data store be appended to the historic data retrieval. This argumentuses the XmimCurrentTickUsage enum. The default for the XMIM_CURRENT_TICK_USAGE argument isXMIM_APPEND_TO_NONE such that only historic data is retrieved. Setting this argument to XMIM_APPEND_TO_ALLwill result in aggregation of the data in the current tick store to the indicated frequency and the historic data beingsupplemented with this current data (if the ending date/time range extends beyond what is available in the historicdata store). When this is specified for daily data, the result will be that, for the current day, a daily bar will beconstructed from the ticks available in the current tick store so far.

Note that since the high-frequency data store is optimized for fast data storage, as opposed to thehistorical data store which is optimized for fast retrieval, data retrieval will be slower if appending fromthe current tick store is indicated.

This argument can also be set to XMIM_APPEND_TO_DAILY or XMIM_APPEND_TO_TICK such that only daily or tickhistoric data will be supplemented.

XmimVaGetRecords has an option XMIM_TIME_RANGE_TYPE with the arguments XMIM_TRADING_RANGE(default) or XMIM_CONTIGUOUS_RANGE. XMIM_TRADING_RANGE treats the starting and ending times as a tradingrange. The argument XMIM_CONTIGUOUS_RANGE treats the starting and ending times as a contiguous time range.

The following example has these start and end dates and times:

start_date: 01/02/2006 end_date: 01/04/2006 start_time: 8:00 a.m. end_time: 2:00 p.m.

Using the XMIM_CONTIGUOUS_RANGE argument, data is retrieved starting at 8:00 a.m. on 01/02/2006 and datacontinues to be retrieved until the end date of 01/04/2006 at 2:00 p.m. All time values for the intervening day(s), inthis case, 01/03/2006 will be retrieved. Using the XMIM_TRADING_RANGE argument, data is retrieved on 01/02/2006

180


from 8:00 a.m. to 2:00 p.m. then again on 01/03/2006 from 8:00 a.m. to 2:00 p.m. and again on 01/04/2006 from8:00 a.m. to 2:00 p.m.

The dates and values arrays are the arguments used to return the records found. The number of records, which willcorrespond to the number of entries in the dates array, is returned in the numRecords argument.

Optimizing Data Record RetrievalXmimReturnCode XmimVaGetRecords (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION_LIST, relName, NULL, XMIM_COLUMN_LIST, colName, NULL, XMIM_FROM_DATE, XMIM_INVALID_DATE, XMIM_TO_DATE, XMIM_INVALID_DATE, XMIM_FROM_TIME, XMIM_INVALID_TIME, XMIM_TO_TIME, XMIM_INVALID_TIME, XMIM_UNITS, 1, XMIM_DAYS, XMIM_LIMIT, 0, XMIM_LIMIT_BY_RECORDS, XMIM_CURRENT_TICK_USAGE, XMIM_APPEND_TO_NONE, XMIM_NUM_RECORDS, &numRecords, XMIM_DATE_TIME_OBJ, &dateObj, XMIM_VALUES, &values, XMIM_END_ARGS); XmimDateTime XmimGetDateTime (XmimDateTimeObj, *dateObj, int recordIndex)

In cases where only a single non-sparse time series is to be retrieved and the fill option is left as the default(XMIM_FILL_NAN for both holidays and missing data and NaNs are not skipped), data retrieval can be significantlyoptimized by not requiring that the server generate and return the dates array. To utilize this optimization,XMIM_DATE_TIME_OBJ is used in lieu of XMIM_DATE_TIMES in the XmimVaGetRecords call. The date_objargument that is returned is of type XmimDateTimeObj. This argument may subsequently be used in theXmimGetDateTime routine to return the date/time associated with a given record (recordIndex). This is shown inthe code segment below:

XmimClientHandle handle; int i, num_records; float *values; XmimDateTimeObj dateObj; XmimDateTime date; if (XmimVaGetRecords (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, "SP", XMIM_COLUMN, "Close", XMIM_NUM_RECORDS, &num_records, XMIM_DATE_TIME_OBJ, &dateObj, XMIM_VALUES, &values, XMIM_END_ARGS) != XMIM_SUCCESS) { XmimPrintError("XmimVaGetRecords"); XmimDisconnect (handle); return 1; } for (i = 0; i < num_records; i++) { date = XmimGetDateTime (&dateObj, i);

181

XmimPrintDateTime (date, False, False); if (XmimTestNaN (handle, values[i])) printf (", NaN"); else printf (", %8.3f", values[i]); printf ("\n"); }

If the XMIM_DATE_TIME_OBJ is used (as above) and the optimization conditions are not satisfied, an error will beissued.

Both XMIM_DATE_TIMES and XMIM_DATE_TIME_OBJ can be specified in the same XmimVaGetRecords call. Whenthis occurs, if the optimization conditions are satisfied, only XMIM_DATE_TIME_OBJ will take effect and, likewise,if the optimization conditions are not satisfied, only XMIM_DATE_TIMES will take effect. When both arguments areused, if the XMIM_DATE_TIMES arguments is NULL after the call to XmimVaGetRecords, then the optimizationconditions were satisfied and, thus, the XMIM_DATE_TIME_OBJ argument took effect. The following code segmentillustrates the use of both methods of date retrieval and testing afterwards to determine which one was used.

XmimClientHandle handle; int i, num_records; int num_cols, num_rels; int idx = 0, rec_len, k; XmimString *rels, *cols; float *values; XmimDateTimeObj dateObj; XmimDateTime *dates, date; if (XmimVaGetRecords (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION_ARRAY, num_rels, rels, XMIM_COLUMN_ARRAY, num_cols, cols, XMIM_NUM_RECORDS, &num_records, XMIM_DATE_TIMES, &dates, XMIM_DATE_TIME_OBJ, &dateObj, XMIM_VALUES, &values, XMIM_END_ARGS) != XMIM_SUCCESS) { XmimPrintError("XmimVaGetRecords"); XmimDisconnect (handle); return 1; } rec_len = num_cols * num_rels; for (i = 0; i < num_records; i++) { if (dates == NULL) { date = XmimGetDateTime (&dateObj, i); XmimPrintDateTime (date, False, False); } else XmimPrintDateTime (dates[i], False, False); for (k = 0; k < rec_len; k++) { if (XmimTestNaN (handle, values[idx+k])) printf (", NaN"); else printf (", %8.3f", values[i]); } printf ("\n"); idx += rec_len; }

182


The optimization conditions will be met only if both num_rels and num_cols are 1.

A further restriction on the use of the “optimized" XmimVaGetRecords is that it is currently available only when thenumber of units is 1 and the time series being returned is not a tick time series. These restrictions will be relaxed inthe future.

Note that this optimization is available only in XmimVaGetRecords and not in the corresponding non-Vaversion of the routine. It is also available for use in the XmimVaGetRecordsRollover routine.

Retrieving Options Data RecordsXmimReturnCode XmimGetRecordsOption (XmimClientHandle handle, int numOptions, XmimString *optionNames, int numColumns, XmimString *colNames, XmimDate fromDate, XmimDate toDate, XmimTime fromTime, XmimTime toTime, XmimDate fromExpDate, XmimDate toExpDate, float fromStrike, float toStrike, XmimOptionType optType, int numUnits, XmimUnits units, XmimFillOption holidayFillOption, XmimFillOption missDataFillOption, XmimSkipAllNaN skipAllNaNRecords, int limit, XmimLimitMode limitMode, int *numRecords, XmimDateTime **dates, XmimDate **expDates, XmimOptionType **optTypes, float **strikes, float **values); XmimReturnCode XmimVaGetRecordsOption ( XMIM_CLIENT_HANDLE, handle, XMIM_RELATION_LIST, optName1, optName2, ..., NULL, XMIM_COLUMN_LIST, NULL, XMIM_FROM_DATE, XMIM_INVALID_DATE, XMIM_TO_DATE, XMIM_INVALID_DATE, XMIM_FROM_TIME, XMIM_INVALID_TIME, XMIM_TO_TIME, XMIM_INVALID_TIME,


183

XMIM_FROM_EXP_DATE, XMIM_INVALID_DATE, XMIM_TO_EXP_DATE, XMIM_INVALID_DATE, XMIM_FROM_STRIKE, -1.0, XMIM_TO_STRIKE, -1.0, XMIM_OPTION_TYPE, XMIM_OPTION_ALL, XMIM_UNITS, 1, XMIM_DAYS, XMIM_FILL_OPTION, XMIM_FILL_NAN, XMIM_FILL_NAN, XMIM_SKIP_NONE, XMIM_LIMIT, 0, XMIM_LIMIT_BY_RECORDS, XMIM_NUM_RECORDS, &numRecords, XMIM_DATE_TIMES, &dates, XMIM_EXPIRATION_DATES, &expDates, XMIM_OPTION_TYPES, &optTypes, XMIM_STRIKE_PRICES, &strikes, XMIM_VALUES, &values, XMIM_END_ARGS); XmimReturnCode XmimGetRecordsOptionAllColumns (XmimClientHandle handle, int numOptions, XmimString *optionNames, XmimDate fromDate, XmimDate toDate, XmimTime fromTime, XmimTime toTime, XmimDate fromExpDate, XmimDate toExpDate, float fromStrike, float toStrike, XmimOptionType optType, int numUnits, XmimUnits units, XmimFillOption holidayFillOption, XmimFillOption missDataFillOption, XmimSkipAllNaN skipAllNaNRecords, int limit, XmimLimitMode limitMode, int *numColumns, XmimString **colNames, int *numRecords, XmimDateTime **dates, XmimDate **expDates, XmimOptionType **optTypes, float **strikes, float **values); XmimReturnCode XmimVaGetRecordsOptionAllColumns ( XMIM_CLIENT_HANDLE, handle, XMIM_RELATION_LIST, optName1, optName2, ..., NULL, XMIM_FROM_DATE, XMIM_INVALID_DATE, XMIM_TO_DATE, XMIM_INVALID_DATE, XMIM_FROM_TIME, XMIM_INVALID_TIME, XMIM_TO_TIME, XMIM_INVALID_TIME, XMIM_FROM_EXP_DATE, XMIM_INVALID_DATE,

184


XMIM_TO_EXP_DATE, XMIM_INVALID_DATE, XMIM_FROM_STRIKE, -1.0, XMIM_TO_STRIKE, -1.0, XMIM_OPTION_TYPE, XMIM_OPTION_ALL, XMIM_UNITS, 1, XMIM_DAYS, XMIM_FILL_OPTION, XMIM_FILL_NAN, XMIM_FILL_NAN, XMIM_SKIP_NONE, XMIM_LIMIT, 0, XMIM_LIMIT_BY_RECORDS, XMIM_COLUMN_ARRAY, &numColumns, &colNames, XMIM_NUM_RECORDS, &numRecords, XMIM_DATE_TIMES, &dates, XMIM_EXPIRATION_DATES, &expDates, XMIM_OPTION_TYPES, &optTypes, XMIM_STRIKE_PRICES, &strikes, XMIM_VALUES, &values, XMIM_END_ARGS);

The XmimGetRecordsOption and XmimGetRecordsOptionAllColumns routines are used to retrieve tick-by-tick, intraday or daily data for one or more options relations. As with the XmimGetRecords routine, theXmimGetRecordsOption routine allows the columns for which data is desired to be specified as an argument. Aswith XmimGetRecordsAllColumns, XmimGetRecordsOptionAllColumns retrieves data for all existing columnsof the specified relations and returns the number of columns as well as the column names (which, if multiple relationsare specified, consists of the union of all columns for all relations). The date/time range specification arguments(fromDate, toDate, fromTime and toTime) are used just as in XmimGetRecords.

Additional arguments in XmimGetRecordsOption allow for the data for a specific option or set of options tobe retrieved. The fromExpDate and toExpDate arguments are used to specify the range of expiration dates tobe considered. If the dates are set to XMIM_INVALID_DATE (or left as defaults), then all expiration dates will beconsidered. A single expiration date is specified by using that date for both fromExpDate and toExpDate. ThefromStrike and toStrike arguments are used to specify the strike prices to be considered. If these argumentsare set to -1, indicating invalid, then all strike prices will be considered. A single strike price is specified by usingthe desired value for both fromStrike and toStrike. Finally, the optionType argument specifies whether calloptions, put options or both should be considered. Both calls and puts will be considered by default.

The remaining input arguments are specified and utilized in the same manner as with XmimGetRecords. As withXmimGetRecords, the dates and values arrays are return arguments, used to return the records found. Additionally,the expDates, optTypes and strikes arrays are used to return the expiration dates, option types and strikeprices for the corresponding values; the size of these arrays will be the same as that of the dates array. Both theXmimGetRecordsOption and the XmimGetRecordsOptionAllColumns calls return the number of records,which will correspond to the number of entries in the dates array. Note that when options data is involved, theremay be multiple records for the same date. Also, it is recommended that the option to fill with NaNs for bothholidays and missing data be used when dealing with options data. This recommendation is based solely upon speedconsiderations; the other existing fill types are available for use with options15.

15 Determining the previous and next values for options is an expensive operation because the system must ensure that these values are obtained from anoption of the same type.


185

The number of days to expiration is specified with the numUnitsToExpir and unitsToExpir arguments. Note thatthe specification has been generalized such that the number of weeks, months, etc., to expiration can be used as well.The percentage of the underlying security's price that the strike price should exceed is given by the strikeWeightargument. For example, to specify a spot option, the percentage given would be 100. The type (frequency) of data tobe used for the underlying security during the comparison will be determined by the frequency specified for the optionsdata, as given by the numUnits and units arguments. If the underlying security does not have data of the samefrequency specified, the closest possible will be used. (e.g., if 1 second is specified but the underlying security hasno real tick data, then 1 minute will be used if intraday tick data is available, and, if not, then daily data will be used.)The comparison order used by the Commodity DataServer will be such that the expiration date will be weighed morethan the strike price. Conceptually, the relevant options are ordered by those whose number of days to expiration areclosest to the specified number of days to expiration. Then, from this list, the first option that exceeds or matchesthe target strike price is selected (if calls are being considered). If multiple options have the same ranking based onexpiration day and meet the strike selection criteria, then the one with the strike price closest to the target strike pricewill be selected16. Note that while the expiration date will be searched in both directions (i.e., expiration days prior tothe one specified will be considered as well as the one specified and those after), the strike price must be at or abovethe target strike price for calls and at or below the target for puts. For example, if the strike weight given is 100 andthe number of days until expiration is 31, the spot option closest to 31 days to expiration would be retrieved.

The optionType argument is used to restrict the options being considered to either puts or calls. In this context,it does not make sense to consider both types of options at the same time, therefore, XMIM_OPTION_ALL is aninvalid choice for optionType. The selectOnce argument is used to indicate whether the relative selection is to beperformed once only or every day (relative perpetual). If selectOnce is set to true, then the selected option's timeseries will be returned, ending at expiration or the end of the date range, whichever comes first. If selectOnce is setto false, a different option is selected for each day and, thus, a constructed daily time series of option fields will bereturned.

Accessing Data Recordsint XmimGetRecordIndex (XmimDateTime getDate, int numRecords, XmimDateTime *dates, XmimFillOption fillOption); int XmimVaGetRecordIndex (XMIM_DATE_TIME, getDate, XMIM_NUM_RECORDS, numRecords, XMIM_DATE_TIMES, dates, XMIM_FILL_OPTION, XMIM_FILL_NAN, XMIM_END_ARGS);

The XmimGetRecordIndex routine is used to access a single record of the arrays returned by XmimGetRecordsor XmimGetRecordsOption. The record index is returned by the function call. The getDate argument specifiesthe date (and time for tick data) of the desired record. The numRecords and dates arguments must be the sameas the arguments returned in the XmimGetRecords or XmimGetRecordsOption routine. The fillOptionargument is used to determine what should be returned by XmimGetRecordIndex in the case where no recordexists for the given date/time. Of the available fill options, only XMIM_FILL_FORWARD, XMIM_FILL_BACKWARD orXMIM_FILL_NAN are valid in this usage and indicate that the previous record, the next record or -1, respectively, will

16 Beyond that, if multiple options have the exact same expiration day ranking and strike price ranking, the system will non- deterministically select one.

186


be returned. In the case where there is more than one record for the given date (as may happen with options data),the first one will be returned.

Note that this is a utility routine and does not take a client handle. It also does not return the standardreturn code type, instead, the function call returns the record index.

Updating Data RecordsXmimReturnCode XmimPutRecords (XmimClientHandle handle, int numRelations, XmimString *relNames, int numColumns, XmimString *colNames, XmimUnits units, int numRecords, XmimDateTime *dates, float *values); XmimReturnCode XmimVaPutRecords (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION_LIST, relName1, relName2, ..., NULL, XMIM_COLUMN_LIST, NULL, XMIM_UNITS, XMIM_DAYS, XMIM_HISTORIC_ONLY, True, XMIM_NUM_RECORDS, numRecords, XMIM_DATE_TIMES, dates, XMIM_VALUES, values, XMIM_END_ARGS);

The XmimPutRecords routine is used to update data in the Commodity DataServer database when the data isalready available in memory. It is the companion routine to XmimGetRecords. The XmimGetRecords routine writesdata from the database into an array and the XmimPutRecords routine reads data from an array into the database.

The relNames and colNames arguments are used to specify which relation columns to update. As withXmimGetRecords, if the columns are not specified, XmimPutRecords will use the bars (Open, High, Low, Close) forthe relations as the default columns. As with XmimGetRecords, columns specified may be normal columns, multi-field columns or individual fields of multi-field columns. When some, but not all, of the fields of a multi-field column arespecified, those fields that are not specified will be filled with NaNs.

The units argument is used to indicate whether the data being updated is millisecond (XMIM_MILLISECONDS), realtick (XMIM_SECONDS), intraday tick (XMIM_MINUTES) or daily (XMIM_DAYS). The Va version of the routine has anadditional argument, XMIM_HISTORIC_ONLY, that can be set to False to indicate that data should be updated in thecurrent tick data store as well as the historical data store, depending on where the date/time range is appropriate. Bydefault, this argument is True such that data will be added to the historical data store only.

Note that if both the historic database and the current tick store are being updated then both a databaselock and a current tick store lock will be required.

187

The number of records to update is given by numRecords and the dates and values supplied in the dates and valuesarrays. The numRecords must correspond to the number of entries in the dates array. If data already exists for arecord specified by an entry in the dates array, the effect will be to replace the existing record.

Updating Options Data RecordsOptions data is stored just like a regular relation (relation type=normal). However, options must be stored underneatha special options relation category. For example, the options for DELL is stored under the DELL.Options category andoptions for IBM is stored under the IBM.Options category.

All options relations are required to have XML in the description field as such:<plist> <type> option_type </type> <strike> strike_price </strike> <expir> expiration_date </expir> </plist>

where:

● option_type is either put or call

● strike_price is a numeric field (e.g., 30.0)

● expiration_date is a date (e.g., 3/17/2003)


Example:relation_add { name = IBM.199308.CALL.40; description = “<plist><expir> 08/31/1993 </expir><strike> 40 </strike><type> call </type></plist>” parent = IBM.Options;}


Replacing and Deleting Data RecordsXmimReturnCode XmimReplaceRecords (XmimClientHandle handle, int numRelations, XmimString *relNames, int numColumns,

188


XmimString *colNames, XmimUnits units, XmimDate fromDate, XmimDate toDate, XmimTime fromTime, XmimTime toTime, int numRecords, XmimDateTime *dates, float *values); XmimReturnCode XmimVaReplaceRecords (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION_LIST, relName1, relName2, ..., NULL, XMIM_COLUMN_LIST, NULL, XMIM_UNITS, XMIM_DAYS, XMIM_HISTORIC_ONLY, True, XMIM_FROM_DATE, XMIM_INVALID_DATE, XMIM_TO_DATE, XMIM_INVALID_DATE, XMIM_FROM_TIME, XMIM_INVALID_TIME, XMIM_TO_TIME, XMIM_INVALID_TIME, XMIM_NUM_RECORDS, 0, XMIM_DATE_TIMES, NULL, XMIM_VALUES, NULL, XMIM_END_ARGS);

The XmimReplaceRecords routine is used to replace or delete data from the Commodity DataServer database. Ifthe columns are not specified, the bars will be used by default. The units argument is used to specify the type ofdata (i.e., millisecond, real tick, intraday tick or daily) to be replaced. The Va version of the routine has an additionalargument, XMIM_HISTORIC_ONLY, that can be set to False to indicate that data should be replaced or deleted fromthe current tick data store as well as the historical data store, depending on the date/time range of the data to bereplaced or deleted. By default, this argument is set to True indicating that only data in the historical data store shouldbe replaced or deleted.

Note that if both the historic database and the current tick store are being updated then both a databaselock and a current tick store lock will be required.

The fromDate and toDate arguments are used to specify the range of data to be replaced; any existing data in thisrange will be deleted. For replacing tick data, the fromTime and toTime arguments can be used to further specifythe range of data to be replaced; for each day in the date range, data will be deleted only for the trading time rangegiven. The dates and values arguments are the arrays that supply the replacement records. If these arguments areNULL, the effect will be to delete the specified records from the Commodity DataServer database.

The numRecords argument indicates the number of replacement records (i.e., the number of entries in the datesarray).

The number of replacement records need not be the same as the number of records that are replaced.


189

High Frequency UpdatingXmimReturnCode XmimLockCurrentTick (XmimClientHandle handle); XmimReturnCode XmimVaLockCurrentTick (XMIM_CLIENT_HANDLE, handle, XMIM_END_ARGS); XmimReturnCode XmimUnlockCurrentTick (XmimClientHandle handle); XmimReturnCode XmimVaUnlockCurrentTick (XMIM_CLIENT_HANDLE, handle, XMIM_END_ARGS); XmimReturnCode XmimUpdateCurrentTick ( XmimClientHandle handle, XmimDate date, int numRelations, XmimString *relNames, int numColumns, XmimString *colNames, XmimBoolean realTick, int numRecords, XmimTime *times, float *values); XmimReturnCode XmimVaUpdateCurrentTick ( XMIM_CLIENT_HANDLE, handle, XMIM_DATE, date, XMIM_RELATION_LIST, relName1, relName2, ..., NULL, XMIM_COLUMN_LIST, NULL, XMIM_REAL_TICK, False, XMIM_NUM_RECORDS, numRecords, XMIM_TIMES, times, XMIM_VALUES, values, XMIM_END_ARGS); XmimReturnCode XmimUpdateCurrentTickBar (XmimClientHandle handle, XmimString relName, XmimDate date, XmimTime time, double value); XmimReturnCode XmimVaUpdateCurrentTickBar (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_DATE, date, XMIM_TIME, time, XMIM_VALUE, value, XMIM_END_ARGS); XmimReturnCode XmimFoldCurrentTick (XmimClientHandle handle, XmimDate fromDate, XmimDate toDate, XmimBoolean realTick); XmimReturnCode XmimVaFoldCurrentTick (XMIM_CLIENT_HANDLE, handle, XMIM_FROM_DATE, XMIM_INVALID_DATE, XMIM_TO_DATE, XMIM_INVALID_DATE, XMIM_REAL_TICK, False, XMIM_END_ARGS); XmimReturnCode XmimFoldCurrentTickSelective (XmimClientHandle handle, XmimString relName, int numColumns, XmimString *colNames, XmimDate fromDate, XmimDate toDate, XmimBoolean realTick,

190


XmimBoolean intraday, XmimBoolean daily); XmimReturnCode XmimVaFoldCurrentTickSelective ( XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, NULL, XMIM_COLUMN_LIST, NULL, XMIM_FROM_DATE, XMIM_INVALID_DATE, XMIM_TO_DATE, XMIM_INVALID_DATE, XMIM_REAL_TICK, False, XMIM_INTRADAY, True, XMIM_DAILY, False, XMIM_END_ARGS); XmimReturnCode XmimClearCurrentTick (XmimClientHandle handle, XmimDate fromDate, XmimDate toDate, XmimBoolean realTick); XmimReturnCode XmimVaClearCurrentTick (XMIM_CLIENT_HANDLE, handle, XMIM_FROM_DATE, XMIM_INVALID_DATE, XMIM_TO_DATE, XMIM_INVALID_DATE, XMIM_REAL_TICK, False, XMIM_END_ARGS); XmimReturnCode XmimGetCurrentTickDataDates (XmimClientHandle handle, int *numDates, XmimDate **dates); XmimReturnCode XmimVaGetCurrentTickDataDates (XMIM_CLIENT_HANDLE, handle, XMIM_NUM_DATES, &numDates, XMIM_DATES, &dates, XMIM_END_ARGS);

High-frequency updating of the database may only be performed after a specialized lock (distinct from the lockrequired for historical database updates) has been obtained for this facility. The XmimLockCurrentTick andXmimUnlockCurrentTick routines are used to lock and unlock the high-frequency (current tick) data store. If theCommodity DataServer can not acquire a write-lock for the current tick store, XmimLockCurrentTick will fail. Theunlock command serves to commit changes to the current tick data store but, in general, all updates to the currenttick store will be immediately visible to any clients reading the database.

The XmimUpdateCurrentTick routine is used to read tick data when high-frequency updating of the databaseis required (such as when updating from an on-line tick feed). The relNames and colNames arguments give therelations and columns to be updated with the date argument specifying the date for which the tick data is to bestored. If no columns are specified, then Open, High, Low, and Close will be used as the default columns. The high-frequency updating facility is provided for real tick data (second-by-second updating) as well as intraday data (minute-by-minute updating). The realTick argument should be set to True when real tick data is to be updated and Falsewhen intraday tick data is to be updated. The number of records to update is given by numRecords and the times andvalues supplied in the times and values arrays. The number of records must correspond to the number of entries in thetimes array.

The XmimUpdateCurrentTickBar routine can be used to read tick data when high-frequency updating of thedatabase is required and certain limiting conditions apply. The XmimUpdateCurrentTickBar routine is significantlyfaster than the XmimUpdateCurrentTick routine and should be used whenever the conditions are appropriate.The XmimUpdateCurrentTickBar routine can be used in cases where the tick data update consists of a singlerelation with a single price value. The update will effectively propagate the value to be used for Close as well as


191

Open, High and Low such that the entire bar can be retrieved and utilized for bar charts, etc. The relName argumentgives the relation to be updated. The date and time arguments specify the date and time of the update and the valuesupplies the price value. This routine can be used for updating either real tick or intraday tick data but, in the caseof intraday data, the user must make sure to zero out the seconds field. It is recommended that, to obtain the bestpossible performance, the non-Va version of this routine be used. Generally, the Va routines are the preferred usagebut there is a slight overhead associated with them and so the absolute optimal performance will be obtained if theXmimUpdateCurrentTickBar routine is used in lieu of the XmimVaUpdateCurrentTickBar routine.

The Commodity DataServer maintains the data added by XmimUpdateCurrentTick separately from the historicaltick data. XmimUpdateCurrentTick should only be used to load the latest, most current tick data. If there is datain the historical database that post-dates data in the current tick store, the data from the current tick store will belost. If the tick data in the current tick store is to be made a permanent part of the Commodity DataServer historicaldatabase, the XmimFoldCurrentTick or XmimFoldCurrentTickSelective routines should be used. To clearthe current tick store, thereby discarding the tick data currently in the store, the XmimClearCurrentTick routineshould be used. The fromDate and toDate arguments are used to specify the range of current tick data to fold ordelete. If these arguments are left unspecified (in the XmimVa version), then all data in the current tick store (or alldata for the specified relation and columns in the case of XmimFoldCurrentTickSelective) will be folded into thehistorical database or deleted. To fold or delete a single day's worth of data, the same date should be given for bothfromDate and toDate.

The XmimFoldCurrentTickSelective routine facilitates selective folding of the specified columns of a relation. Ifno columns are given then all of the columns for the relation will be folded. If neither the relation name or the columnsare specified then it will behave much as XmimFoldCurrentTick and fold all data in the current tick store within thedate range specified. This routine also facilitates folding of the current tick data into one or more of the different typesof historical data (real tick, intraday and daily) with aggregation as appropriate. The realTick, intraday and dailyarguments are used to indicate which historical data to fold into, with the default being folding into the intraday tickdata only. XmimFoldCurrentTick and XmimFoldCurrentTickSelective will never replace existing historicaltick data.

Note that for folding current tick data into historical data, both a current tick lock and a historical lock willbe required, whereas only a current tick lock is required for updating or clearing current tick data.

XmimGetCurrentTickDataDates is a utility routine that returns all dates for which there is corresponding data inthe current tick store. The numDates argument returns the number of dates and the dates array is used to return theactual dates found.

Executing QueriesXmimReturnCode XmimExecQuery (XmimClientHandle handle, XmimString queryFilename, XmimString outputFilename, int numSubstitutions,

192


XmimSubstitution *substitutions, XmimAppendMode appendMode, XmimBoolean print, XmimVerboseMode verboseMode); XmimReturnCode XmimVaExecQuery (XMIM_CLIENT_HANDLE, handle, XMIM_QUERY_FILE_NAME, queryFilename, XMIM_QUERY_OUT_FILE_NAME, NULL, XMIM_QUERY_SUBSTITUTION_LIST, NULL, XMIM_APPEND_MODE, XMIM_REPLACE, XMIM_PRINT, False, XMIM_QUERY_VERBOSE_MODE, XMIM_QUERY_SILENT, XMIM_END_ARGS); XmimReturnCode XmimGraphQuery (XmimClientHandle handle, XmimString queryFilename, XmimString outputFilename, int numSubstitutions, XmimSubstitution *substitutions, XmimAppendMode appendMode, XmimBoolean print, XmimVerboseMode verboseMode); XmimReturnCode XmimVaGraphQuery (XMIM_CLIENT_HANDLE, handle, XMIM_QUERY_FILE_NAME, queryFilename, XMIM_QUERY_OUT_FILE_NAME, NULL, XMIM_QUERY_SUBSTITUTION_LIST, NULL, XMIM_APPEND_MODE, XMIM_REPLACE, XMIM_PRINT, False, XMIM_QUERY_VERBOSE_MODE, XMIM_QUERY_SILENT, XMIM_END_ARGS);

The XmimExecQuery and XmimGraphQuery routines can be used to execute or graph a saved CommodityDataServer query and write the results to a file. They are equivalent to the BMIM query_execute andgraph_query commands. The query will be executed repeatedly, once for each item in the substitutions list wherethe substitutions consist of modifications to be applied to the original query. Refer to “Executing Saved Queries” for amore detailed discussion of these routines and their arguments.XmimReturnCode XmimSelectRecords (XmimClientHandle handle, int numConds, XmimString *whenConds, int numShowAttrs, XmimString *showAttrs, int beforeRepeat, int afterRepeat, int numExecUnits; XmimUnits execUnits; int *numRecords; XmimBoolean **whenSuccess, XmimDateTime **dates, float **values); XmimReturnCode XmimVaSelectRecords (XMIM_CLIENT_HANDLE, handle, XMIM_CONDITION_LIST, NULL, XMIM_SHOW_ATTR_LIST, showAttr1, showAttr2, ..., NULL, XMIM_SHOW_BEFORE_REPEAT, 0, XMIM_SHOW_AFTER_REPEAT, 0, XMIM_UNITS, 1, XMIM_DAYS,


193

XMIM_FILL_OPTION, XMIM_FILL_NAN, XMIM_FILL_NAN, XMIM_SKIP_NONE, XMIM_CURRENT_TICK_USAGE, XMIM_APPEND_TO_NONE, XMIM_UNION_DATE, 0, XMIM_NUM_RECORDS, &numRecords, XMIM_CONDITION_SUCCESS, &success, XMIM_DATE_TIMES, &dates, XMIM_VALUES, &values, XMIM_END_ARGS);

The XmimSelectRecords routine retrieves records from the Commodity DataServer database that match one ormore given conditions specified as whenConds. Using this routine to execute Commodity Query SHOW-WHEN queriesobviates the need for using files for input and output. The resulting records containing the desired showAttrs arereturned in the values array with the corresponding date/times given in the dates array. While no whenConds need bespecified, the showAttrs list must have at least one entry. Another parallel array, the whenSuccess array, indicateswhich records actually satisfied the conditions as opposed to those that are returned due to the before/after repeatspecification. The execution units for the query can be set using the execUnits argument. When the whenCondsspecification used is XMIM_CONDITION_FILE, the expected format of the file is a condition per line. Likewise for theshowAttrs. Refer to “XmimSelectRecords” for an example of how to use XmimSelectRecords.

The Va version of the routine has an argument, XMIM_FILL_OPTION, that allows for NaN handling to be specified.This functionality is not supported directly in the non-Va version but the XmimSetFillOption routine can be usedto specify the desired NaN handling options globally. By default, NaNs will be used for missing values/holidays unlessexplicitly overridden either globally using the XmimSetFillOption routine or locally using XmimVaSelectRecords.Refer to the description of XmimGetRecords in “Accessing Data for a Relation” for details pertaining to use of thisargument.

The Va version also provides another argument, XMIM_CURRENT_TICK_USAGE, which is not supported in the non-Vaversion. This argument indicates whether the query should be executed over historic data only or executed using datafrom the high-frequency current tick store in addition to the historic data. By default, only historical data will be usedas accessing the current tick data is slower and will slow down query execution. If set to XMIM_APPEND_TO_ALL thehistoric data will be supplemented with data from the current tick store and this will apply to all types of data existingin the query whether it be real tick, intraday, daily or a mix. XMIM_APPEND_TO_DAILY and XMIM_APPEND_TO_TICKcan be used to specify that data from the current tick store be appended only to daily or tick historical data,respectively.

Table FacilityThe Commodity DataServer C/C++ API provides a facility that allows the user to store and retrieve non-time seriesdata organized as relational tables. This data may be associated with time series data stored in the CommodityDataServer database, hence providing a user-definable dynamic schema capability. Entries or row members of a tableare referred to as tuples and each tuple can have multiple components corresponding to columns or fields in the table.

The data types that can be stored include numbers, dates, times, Commodity DataServer relations, CommodityDataServer columns, strings, atoms and character buffers.

194


Note the direct support for Commodity DataServer relations and Commodity DataServer columns thatdistinguish this facility from other general-purpose relational database facilities. When relations orcolumns are used, inheritance (utilizing the Commodity DataServer hierarchy) is supported.

A key may be associated with a table, indicating the field for which retrieval will be optimized. Retrieval supportswildcards for performing a broad search. “Data Source Table: A Complete Application” provides a complete applicationdemonstrating the use of the Table Facility for storing and retrieving Data Source information.

The user may create, delete, or rename a table and may add tuples to a table or delete tuples from a table. All of theseoperations require obtaining a database lock. Without a database lock, the user may open an existing table, obtaininformation about a table, or read tuples from a table.

Creating/Deleting/Renaming TablesXmimReturnCode XmimCreateTable (XmimClientHandle handle, XmimString tableName, int numFields, XmimTableFieldDesc *fields, int keyFieldIndex, XmimBoolean singleValue, XmimTableHandle *tableHandle); XmimReturnCode XmimVaCreateTable (XMIM_CLIENT_HANDLE, handle, XMIM_TABLE_NAME, tableName, XMIM_FIELD_ARRAY, NULL, XMIM_KEY_FIELD_INDEX, 0, XMIM_SINGLE_VALUE, True, XMIM_TABLE_HANDLE, &tableHandle, XMIM_END_ARGS);

The XmimCreateTable routine is used to create a new table. The arguments serve to specify the characteristics/attributes of the table. The tableName argument is used to assign a name to the table. The description of the tuplesfor the table is given with the numFields and fields arguments. The numFields argument specifies the number ofcolumns in the table, i.e., the number of fields in each tuple of the table. For each field, a field description, consisting ofthe field name and field data type, must be specified. The field description is given by the following structure: typedef struct { XmimString name; XmimTableFieldType type; int length; } XmimTableFieldDesc;

typedef enum { XMIM_TAB_FIELD_INT, XMIM_TAB_FIELD_FLOAT, XMIM_TAB_FIELD_DOUBLE, XMIM_TAB_FIELD_DATE, XMIM_TAB_FIELD_TIME, XMIM_TAB_FIELD_STRING, XMIM_TAB_FIELD_ATOM, XMIM_TAB_FIELD_BLOCK, XMIM_TAB_FIELD_REL,


195

XMIM_TAB_FIELD_COL } XmimTableFieldType;

The possible field data types are numbers (integer, float or double), dates, times, strings (null-delimited), atoms, blocks(character buffers), Commodity DataServer relations, and Commodity DataServer columns. An atom is exactly like astring, except that specifying it as an atom identifies the string as one which is expected to be used over and overagain and, therefore, allows the Commodity DataServer to handle it optimally such that all occurrences are shared(i.e., The Commodity DataServer interns the string into the database by mapping it to an integer). A good exampleis given in the application in “Data Source Table: A Complete Application” where the data source is declared as afield of type atom since there will be a limited set of data sources that will be used over and over again. A block is anuninterpreted number of bytes. The number of bytes must be specified as the length in the XmimTableFieldDesc.This is the only type for which the length in the XmimTableFieldDesc is used. When the block field type is used,it is important that every tuple stored has precisely the specified number of bytes for that field because, since thedata is uninterpreted, the number of bytes specified will be read regardless. The Commodity DataServer relationsand columns are passed to the table as strings, e.g., “IBM", “Close", etc., as is standard throughout the CommodityDataServer C/C++ API. Identifying them as relations and columns rather than as strings allows them to be stored insuch a way as to conserve space, as well as facilitating use of the DataServer schema inheritance mechanism, e.g.,for inheriting IBM attributes from Equities.

The keyFieldIndex argument allows specification of the field for which retrieval should be optimized. This numberis zero-based (i.e., the first field is 0). By default, the first field will be used as the key. The singleValue booleanargument indicates whether or not the key field should be unique among the tuples of a table. That is, it indicateswhether or not multiple tuples having the same value for the key field are allowed within a table. By default, thisargument is set to True if there is an existing tuple having the same key field value, the old tuple will be deleted, i.e.,replaced by the new tuple.

Once a table is created/added, a unique identifier (handle) for the table is returned as the tableHandle argument.Since creating a table has the affect of also opening the table, this identifier may subsequently be used to add tuplesto the table (see “Adding/Deleting Table Entries”) and to close it when finished (see “Opening/Closing Tables”). Ifthe client application exits without closing a table, there will be memory associated with the table that will not bedeallocated so closing the table is extremely important.XmimReturnCode XmimDeleteTable (XmimClientHandle handle, XmimString tableName) XmimReturnCode XmimVaDeleteTable (XMIM_CLIENT_HANDLE, handle, XMIM_TABLE_NAME, tableName, XMIM_END_ARGS);

The XmimDeleteTable routine can be used to delete an existing table.

Note that the only way to modify the characteristics of a table, with the exception of the table name, isto delete the table and create a new one.

XmimReturnCode XmimRenameTable (XmimClientHandle handle, XmimString origName, XmimString newName) XmimReturnCode XmimVaRenameTable (XMIM_CLIENT_HANDLE, handle, XMIM_TABLE_NAME, tableName, XMIM_NEW_NAME, newName,

196


XMIM_END_ARGS);

The XmimRenameTable routine is used to change the name of a table.

Note that this is the only characteristic of a table that can be modified.

Opening/Closing TablesXmimReturnCode XmimOpenTable (XmimClientHandle handle, XmimString tableName, XmimBoolean readOnly, XmimTableHandle *tableHandle) XmimReturnCode XmimVaOpenTable (XMIM_CLIENT_HANDLE, handle, XMIM_TABLE_NAME, tableName, XMIM_READ_ONLY, True, XMIM_TABLE_HANDLE, &tableHandle, XMIM_END_ARGS);

XmimReturnCode XmimCloseTable (XmimClientHandle handle, XmimTableHandle tableHandle) XmimReturnCode XmimVaCloseTable (XMIM_CLIENT_HANDLE, handle, XMIM_TABLE_HANDLE, tableHandle, XMIM_END_ARGS);

An existing table must be opened before tuples can be added to it or read/deleted from it. A table must also beopened when accessing information about the table attributes. The XmimOpenTable routine is provided for thispurpose. The tableName argument identifies the table to be opened and the readOnly argument specifies whetherthe table should be opened for reading only or also for writing. If tuples are to be added or deleted from the table, thereadOnly argument must be set to False and the database lock will occur. By default, the table will be opened forread only. As with the XmimCreateTable routine, the table handle will be returned for subsequent use. An opentable must be closed by calling XmimCloseTable before exiting so that memory associated with the table can bedeallocated.

Getting Table InformationXmimReturnCode XmimGetTable (XmimClientHandle handle, XmimTableHandle tableHandle, int *numFields, XmimTableFieldDesc **fields, int *keyFieldIndex, XmimBoolean *singleValueP) XmimReturnCode XmimVaGetTable (XMIM_CLIENT_HANDLE, handle, XMIM_TABLE_HANDLE, tableHandle, XMIM_FIELD_ARRAY, &numFields, &fields, XMIM_KEY_FIELD_INDEX, &keyFieldIndex, XMIM_SINGLE_VALUE, &singleValue, XMIM_END_ARGS);


197

XmimReturnCode XmimGetTableNames (XmimClientHandle handle, int *numTables, XmimString **tableNames) XmimReturnCode XmimVaGetTableNames (XMIM_CLIENT_HANDLE, handle, XMIM_TABLE_ARRAY, &numTables, &tableNames, XMIM_END_ARGS);

The XmimGetTable routine takes the handle for an open table as an argument and returns the field information, keyindex and key uniqueness flag. For descriptions of these table attributes see “Creating/Deleting/Renaming Tables”. TheXmimGetTableNames routine simply returns the number of tables stored in the database and all the table names.

Adding/Deleting Table EntriesXmimReturnCode XmimAddTuple (XmimClientHandle handle, XmimTableHandle tableHandle, XmimTableFieldVal *tuple) XmimReturnCode XmimVaAddTuple (XMIM_CLIENT_HANDLE, handle, XMIM_TABLE_HANDLE, tableHandle, XMIM_TUPLE, tuple, XMIM_END_ARGS);

The XmimAddTuple routine provides for adding a tuple to an existing open table. A tuple is defined as an array ofstructures which contain a wildcardP flag and a value union as follows: typedef struct { XmimBoolean wildcardP; union { int intValue; float floatValue; double doubleValue; XmimString stringValue; XmimDate dateValue; XmimTime timeValue; long longValue; } value; } XmimTableFieldVal;

The wildcardP flag is only used when retrieving or deleting tuples and so should always be set to False whenadding tuples. For each component of the tuple, an integer, float, double, string, date or time value must be given.There should be one structure for each field specified when the table was created and the values given shouldcorrespond to the correct field data type specified when the table was created.

Note that atom, block, relation and column field types are specified as XmimStrings.

XmimReturnCode XmimDeleteTuple (XmimClientHandle handle, XmimTableHandle tableHandle, XmimTableFieldVal *tuple) XmimReturnCode XmimVaDeleteTuple (XMIM_CLIENT_HANDLE, handle, XMIM_TABLE_HANDLE, tableHandle,

198


XMIM_TUPLE, tuple, XMIM_END_ARGS);

The XmimDeleteTuple routine is used to delete a tuple from an open table identified by tableHandle. Wildcardscan be used to delete all tuples matching a given pattern by setting wildcardP to True for relevant fields.

Accessing Table EntriesXmimReturnCode XmimCreateTableCursor (XmimClientHandle handle, XmimTableHandle tableHandle, XmimTableFieldVal *tuple, int inheritFieldIndex, unsigned int cacheSize, XmimCursorHandle *cursor) XmimReturnCode XmimVaCreateTableCursor (XMIM_CLIENT_HANDLE, handle, XMIM_TABLE_HANDLE, tableHandle, XMIM_TUPLE, tuple, XMIM_INHERIT_FIELD_INDEX, -1, XMIM_SIZE, 1, XMIM_CURSOR_HANDLE, &cursor, XMIM_END_ARGS);

To get tuples from a table, a cursor must first be created and initialized by specifying the desired search pattern. TheXmimCreateTableCursor routine provides this functionality. The table must be open and the tableHandle, whichidentifies the desired table, must be that returned from either the XmimCreateTable or XmimOpenTable routine(includes XmimVa versions). The tuple argument is used to specify the desired tuple pattern. This tuple pattern is anarray of structures where each structure represents a field. If wildcardP is True then any value for that field willmatch the search. Alternatively, the wildcardP flag can be set to False and an actual value to search for provided.For example, to search for all tuples where the first field is IBM (field type is relation) in a table where the number offields is two, tuple is given as an array of two XmimTableFieldVal structures where the first has the wildcardPflag set to False and the value given as the XmimString “IBM"; the second simply has the wildcardP flag set toTrue. Code for constructing the tuple argument for this example is shown below (the second field is of type string): XmimTableFieldVal tuple[2]; tuple[0].wildcardP = False; tuple[0].value.stringValue = "IBM"; tuple[1].wildcardP = True; tuple[1].value.stringValue = NULL;

To retrieve all tuples in the table, all fields should simply be wildcards in the tuple pattern.

If one of the fields is a relation or column, the Commodity DataServer hierarchy can be used for inheritance. TheinheritFieldIndex argument is the number of the field (starting with 0) that is to be inherited. By default, theinheritance argument is -1 meaning no inheritance. So, for example, if there is a table of <relation, string> tuples andan entry exists for Equities, a lookup can be done on IBM with the inheritance argument set to 0 (to indicate the firstfield). The value for Equities will be returned since it is the closest ancestor of IBM with a value stored in the table.

Note that the search will be much faster if the inherited field is also a key (see “Creating/Deleting/Renaming Tables” ). Inheritance may be used on non-key fields but this is not recommended as thesearch will be extremely slow.


199

The cacheSize argument can be used to optimize the handling of passing tuples from the server to the client. Itindicates the number of tuple that will be shipped at once from the server. The tuples are then cached on the clientside and each subsequent call to retrieve a tuple (using the XmimGetTuple routine) will either get a tuple from thecache or, when the cache is exhausted, make another RPC call to the server to fill the cache and then get a tuplefrom it. All of this is transparent to the user. The default cache size is 1 such that each call to retrieve a tuple willcorrespond to a RPC call to the server. For some applications (those involving many tuples being retrieved) it is farmore efficient to grab many tuples at once from the server and cache them on the client side.

The cursor argument is returned for subsequent use in retrieving tuples.

Note that every cursor is implicitly associated with a table such that if the cursor is given, there is noneed to specify the table handle.

It is important that the XmimDeleteCursor routine be called when the cursor is no longer needed (either all thetuples have been retrieved or the retrieval was aborted) so that the memory associated with the cursor will bedeallocated.

XmimReturnCode XmimDeleteCursor (XmimClientHandle handle, XmimCursorHandle cursor) XmimReturnCode XmimVaDeleteCursor (XMIM_CLIENT_HANDLE, handle, XMIM_CURSOR_HANDLE, cursor, XMIM_END_ARGS);

The XmimDeleteCursor routine is used to delete a cursor created by the XmimCreateTableCursor routine.

XmimReturnCode XmimIsCursorExhausted (XmimClientHandle handle, XmimCursorHandle cursor, XmimBoolean *exhausted) XmimReturnCode XmimVaIsCursorExhausted (XMIM_CLIENT_HANDLE, handle, XMIM_CURSOR_HANDLE, cursor, XMIM_CURSOR_EXHAUSTED, &exhausted, XMIM_END_ARGS);

The XmimIsCursorExhausted routine can be used to determine if there are any more tuples to retrieve beforemaking a call to XmimGetTuple. If the exhausted argument is returned as True then no more tuples are available; ifit is returned as False then there are more tuples that can be read.

XmimReturnCode XmimGetTuple (XmimClientHandle handle, XmimCursorHandle cursorHandle, XmimTableFieldVal *tuple) XmimReturnCode XmimVaGetTuple (XMIM_CLIENT_HANDLE, handle, XMIM_CURSOR_HANDLE, cursorHandle, XMIM_TUPLE, tuple, XMIM_END_ARGS);

The XmimGetTuple routine is used to retrieve tuples. The XmimCreateTableCursor routine must first be usedto create a cursor into the appropriate table and set up the tuple search pattern desired. XmimGetTuple can thenbe called iteratively to get the tuples one at a time (advancing the cursor at each iteration) until all tuples have beenretrieved. The return status of XmimGetTuple can be checked to determine if a tuple was retrieved, and, if not, then

200


the cursor should be deleted and the loop exited. Alternatively, at each iteration, the XmimIsCursorExhaustedroutine can be called to determine if additional tuples exist. The tuple argument will return the tuple component valuesas a array of XmimTableFieldVal structures. The wildcard is not relevant in this context. See the get_tupleroutine in “Utility Routines” for an example of using XmimGetTuple.

Correction AuditsLIM uses corrections audits to keep an audit trail (or log) of changes made to historical data points. Clients can usecorrection audits to examine when and what data points in the database were modified over time. It can also beused to pose queries to see how data would have appeared on a particular day. Internally, the corrections audit trail isstored in the Commodity DataServer Table Facility. The Table Facility is a generic storage area used to hold data that isnot time-series in nature.

Please see the Chapter 12, “Correction Audits” for a concise description of: 1) How to turn on a log ofcorrection audits. 2) How to add, retrieve, and remove daily and intraday corrections to the Table Facility.

Regardless of whether or not you enable the corrections auditing log, Morningstar Commodity Data will alway applythe corrected data points to the databases. Note that we do not enable the corrections auditing log by default. Toenable this feature, please contact a Client Services Representative.

Adding Correction Audits to the Table FacilityXmimReturnCode XmimAddCorrections (XmimClientHandle handle, XmimString fileName); XmimReturnCode XmimVaAddCorrections (XMIM_CLIENT_HANDLE, handle, XMIM_FILENAME, fileName, XMIM_END_ARGS);

The XmimAddCorrections routine is used to add daily and intraday correction audit data to the database. Correctionaudit data needs to be in a text file, one correction per line. Intraday audit corrections have the following format: 20030422, 0000, APX.DAHOURLY, Val, 20030421, 1400, 1.25

This means that on April 22, 2003 a corrected value for Val of APX.DAHOURLY on April 21, 2003 at 2:00 PM wasplaced in the database and the previous value for that series was 1.25.

Note that if two corrections for the same relation on the same date and time are received, only the lastone is kept.

For daily data corrections, use the format:20030422, APX.DADAILY, Val, 20030421, 1.25

http://www.lim.com/contact/customer-support.php


201

Once correction audits have been added to the database, data may be accessed in such a manner as to take intoconsideration any corrections made through a specified date, but none made after that date. See “Optimizing DataRecord Retrieval” for a description of accessing data with a date specifying which corrections should be applied.

Corrections are stored in the Table Facility. Thus, all Table Facility routines can be used to access and modifycorrections. See “Table Facility” for a description of the Table Facility.

Daily corrections are stored in the "XMIM_CORRECTIONS" table with the following fields:const XmimTableFieldDesc corrections_schema[5] = {{"Date", XMIM_TAB_FIELD_DATE}, {"Relation", XMIM_TAB_FIELD_ATOM}, {"Column", XMIM_TAB_FIELD_ATOM}, {"OldDate", XMIM_TAB_FIELD_DATE}, {"OldValue", XMIM_TAB_FIELD_DOUBLE}};

Intraday corrections are stored in the "XMIM_CORRECTIONS_INTRADAY_yyyymmdd" table where “yyyymmdd”represents the date the correction was reported (i.e., if provided a report date of 20030421, then create and/or opena table named XMIM_CORRECTIONS_INTRADAY_20030421 in the table facility). The field descriptions are providedbelow:const XmimTableFieldDesc corrections_schema[7] = {{"Date", XMIM_TAB_FIELD_DATE}, {"Time", XMIM_TAB_FIELD_TIME}, {"Relation", XMIM_TAB_FIELD_ATOM}, {"Column", XMIM_TAB_FIELD_ATOM}, {"OldDate", XMIM_TAB_FIELD_DATE}, {"OldTime", XMIM_TAB_FIELD_TIME}, {"OldValue", XMIM_TAB_FIELD_DOUBLE}};

Retrieving Correction Audits from the Table FacilityUse the Commodity DataServer C/C++ API to retrieve correction audit data from the Table Facility within thedatabase. The following instructions show how to retrieve daily and intraday corrections.

XmimGetCorrections() opens, creates, and closes the appropriate table(s) based on parametersprovided, therefore XmimOpenDatabase(), XmimNewDatabase() and XmimCloseDatabase()would not be required.

XmimReturnCode XmimGetCorrections (XmimClientHandle handle, XmimString relName, XmimString colName, XmimDate fromDate, XmimDate toDate, XmimTime fromTime, XmimTime toTime, XmimCorrectionType units, int limit, int *numRecords, XmimDateTime **reportedDates, XmimDateTime **correctedDates,

202


double **value);

XmimGetCorrections routine is used to return correction data for a particular relation and column

The relName and colName arguments (these are required arguments) specify the single relation and column fromwhich to get data. The dates (fromDate and toDate) specify the range for the data. Setting fromDate the sameas toDate will retrieve the single record for that date. The times (fromTime and toTime) specify the trading timerange for each date in the date range (applies to intraday corrections only). The units argument specifies the datatype to retrieve (e.g., daily, intraday).

The XmimCorrectionsType type is defined as follows: typedef enum { XMIM_CORRECTIONS_DAILY=0, XMIM_CORRECTIONS_INTRADAY=1 } XmimCorrectionType;

If units is specified as XMIM_CORRECTIONS_INTRADAY then the query would look only in intraday tables and likewisefor XMIM_CORRECTIONS_DAILY the query would only look in the daily corrections table. The limit argumentspecifies the maximum number of records to return. A value of 0 or less would retrieve all records, where a 'positive'value would set a limit of records to the provided value.

The dates (reportedDates and correctedDates) and values arrays are used to return the records found. Thesedates and values correspond to the relation and column specified for the given date and time. The number of records(numRecords) will be exactly the number of entries in the dates and value arrays.

Deleting Correction Audits from the Table FacilityThe following instructions show how to delete correction audit logs from the Table Facility using the API.

XmimDeleteCorrections() opens, creates, and closes the appropriate table(s) based on parametersprovided, therefore XmimOpenDatabase(), XmimNewDatabase() and XmimCloseDatabase()would not be required.

XmimDeleteCorrections routine is used to delete correction audit data for a particular relation and column:XmimReturnCode XmimDeleteCorrections (XmimClientHandle handle, XmimString relName, XmimString colName, XmimDate fromDate, XmimDate toDate, XmimTime fromTime, XmimTime toTime, XmimCorrectionType units, int *numRecords);

The relName and colName arguments (these are required arguments) specify the single relation and column fromwhich to get data. The dates (fromDate and toDate) specify the range for the data. Setting fromDate the sameas toDate will retrieve the single record for that date. The times (fromTime and toTime) specify the trading time


203

range for each date in the date range (applies to intraday corrections only). The units argument specifies the datatype to retrieve (e.g., daily, intraday).

The XmimCorrectionsType type is defined as follows: typedef enum { XMIM_CORRECTIONS_DAILY=0, XMIM_CORRECTIONS_INTRADAY=1 } XmimCorrectionType;

If units is specified as XMIM_CORRECTIONS_INTRADAY then the query would look only in intraday tables and likewisefor XMIM_CORRECTIONS_DAILY the query would only look in the daily corrections table. The limit argumentspecifies the maximum number of records to return. A value of 0 or less would retrieve all records, where a 'positive'value would set a limit of records to the provided value.

The number of records (numRecords) deleted is returned for the specified relation and column for the given date/timeframe.

Note that XmimGetCorrections() and XmimDeleteCorrections() opens, creates, andcloses the appropriate table(s) based on parameters provided, therefore XmimOpenDatabase(),XmimNewDatabase() and XmimCloseDatabase() would not be required.

Correction Audits File FormatThe file specifying the corrections audit data must be composed of one record per line. The correction audit file cancontain daily or intraday records. Each record contains the previous value (not the current corrected value).

The intraday corrections audit data is of the following form:20030415, 0000, APX.DAHOURLY, Val, 20030322, 1400, 1.25 Transaction Date : April 15, 2003 Transaction Time : 12:00 AM Ticker Symbol : APX.DAHOURLY Value to be Correction : Val Date to be Corrected : March 22, 2003 and Previous Date : March 22, 2003 Time to be Corrected : 2:00 PM Previous Value : 1.25

Indicating that on April 15, 2003 at 12:00 AM a corrected value for Val of APX.DAHOURLY on March 22, 2003 at 2:00PM was placed in the database and the previous value for that series was 1.25.



204


Accessing Data with Correction Audits Applied

Once corrections have been added to the database, data may be accessed in such a manner as to take intoconsideration any corrections made through a specified date, but none made after that date. This capability is availablewith both the Commodity DataServer C/C++ and the Commodity DataServer Visual Basic for Applications APIs.

In the Commodity DataServer C/C++ API, this is accomplished using the XmimVaGetRecords orXmimVaGetRecordsAllColumns routines. The argument with the name XMIM_CORRECTIONS_DATE is availablewith these routines. When this date is used, values are accessed as they would have appeared on that date. That is,if a correction was added on June 5, 1997 and the XMIM_CORRECTIONS_DATE specified in the XmimVaGetRecordscall is before June 5, the old value will be returned; on the other hand, if the date specified is after June 5 then thenew corrected value will be returned. The synopses for the two routines is provided below:

XmimReturnCode XmimVaGetRecords (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION_LIST, relName1, relName2, ..., NULL, XMIM_COLUMN_LIST, NULL, XMIM_FROM_DATE, XMIM_INVALID_DATE, XMIM_TO_DATE, XMIM_INVALID_DATE, XMIM_FROM_TIME, XMIM_INVALID_TIME, XMIM_TO_TIME, XMIM_INVALID_TIME, XMIM_UNITS, 1, XMIM_DAYS, XMIM_CORRECTIONS_DATE, XMIM_INVALID_DATE, XMIM_FILL_OPTION, XMIM_FILL_NAN, XMIM_FILL_NAN, XMIM_SKIP_NONE, XMIM_LIMIT, 0, XMIM_LIMIT_BY_RECORDS, XMIM_CURRENT_TICK_USAGE, XMIM_APPEND_TO_NONE, XMIM_NUM_RECORDS, &numRecords, XMIM_DATE_TIMES, &dates, XMIM_VALUES, &values, XMIM_END_ARGS); XmimReturnCode XmimVaGetRecordsAllColumns ( XMIM_CLIENT_HANDLE, handle, XMIM_RELATION_LIST, relName1, relName2, ...,NULL, XMIM_FROM_DATE, XMIM_INVALID_DATE, XMIM_TO_DATE, XMIM_INVALID_DATE, XMIM_FROM_TIME, XMIM_INVALID_TIME, XMIM_TO_TIME, XMIM_INVALID_TIME, XMIM_UNITS, 1, XMIM_DAYS, XMIM_CORRECTIONS_DATE, XMIM_INVALID_DATE, XMIM_FILL_OPTION, XMIM_FILL_NAN, XMIM_FILL_NAN, XMIM_SKIP_NONE, XMIM_LIMIT, 0, XMIM_LIMIT_BY_RECORDS, XMIM_CURRENT_TICK_USAGE, XMIM_APPEND_TO_NONE, XMIM_COLUMN_ARRAY, &numColumns, &colNames, XMIM_NUM_RECORDS, &numRecords, XMIM_DATE_TIMES, &dates, XMIM_VALUES, &values,


205

XMIM_END_ARGS);

If the corrections date is not specified, then it will be taken as the last data date such that any and all corrections willhave been applied.

Note that the Va version of the API routines must be used as the new argument is not available with thenon-Va version (for backward compatibility reasons).

Using the Table Facility to Retrieve or Update Corrections

The Table Facility available through the Commodity DataServer C/C++ API is used to store the correction auditinformation. Thus, all of the routines provided and available with that facility can be used such that corrections auditinformation can be retrieved, modified or deleted. See Chapter 5, “C/C++ API” for a full description of the TableFacility.

Daily Table

The name of the daily table used is XMIM_CORRECTIONS and the field descriptions are provided below:

const XmimTableFieldDesc corrections_schema[5] = {{"Date", XMIM_TAB_FIELD_DATE}, {"Relation", XMIM_TAB_FIELD_ATOM}, {"Column", XMIM_TAB_FIELD_ATOM}, {"OldDate", XMIM_TAB_FIELD_DATE}, {"OldValue", XMIM_TAB_FIELD_DOUBLE}};

Intraday Table

The name of the intraday table used is XMIM_CORRECTIONS_INTRADAY_YYYYMMDD where YYYYMMDD representsthe date the correction was reported (i.e., if provided a report date of 20030421, it would create and/or open a tablenamed XMIM_CORRECTIONS_INTRADAY_20030421 in the table facility). The field descriptions are provided below:

const XmimTableFieldDesc corrections_schema[7] = {{"Date", XMIM_TAB_FIELD_DATE}, {"Time",XMIM_TAB_FIELD_TIME}, {"Relation", XMIM_TAB_FIELD_ATOM}, {"Column", XMIM_TAB_FIELD_ATOM}, {"OldDate", XMIM_TAB_FIELD_DATE}, {"OldTime", XMIM_TAB_FIELD_TIME}, {"OldValue", XMIM_TAB_FIELD_DOUBLE}};

206


Custom Search FacilityXmimReturnCode XmimGetCustomSearches (XmimClientHandle handle, int *numCustSearch, XmimString **custSearches); XmimReturnCode XmimVaGetCustomSearches (XMIM_CLIENT_HANDLE, handle, XMIM_FUNCTION_ARRAY, &numCustSearch, &custSearches, XMIM_END_ARGS); XmimReturnCode XmimGetCustomSearchArgs (XmimClientHandle handle, XmimString custSearch, int *numArgs, XmimString **argDescs, int **argTypes); XmimReturnCode XmimVaGetCustomSearchArgs (XMIM_CLIENT_HANDLE, handle, XMIM_FUNCTION, custSearch, XMIM_DESCRIPTION_ARRAYS, &numArgs, &argDescs, &argTypes, XMIM_END_ARGS); XmimReturnCode XmimProcessCustomSearch (XmimClientHandle handle, XmimString custSearch, XmimString rootRel, int numArgs, XmimString *args, int *numResults, XmimString **results); XmimReturnCode XmimVaProcessCustomSearch (XMIM_CLIENT_HANDLE, handle, XMIM_FUNCTION, custSearch, XMIM_RELATION, rootRel, XMIM_ARG_ARRAY, numArgs, args, XMIM_RESULT_ARRAY, &numResults, &results, XMIM_END_ARGS);

The XmimGetCustomSearches routine is used to retrieve a list of the custom searches. The custSearchesargument returned will be an array of function names. The XmimGetCustomSearchArgs routine is used to retrievethe argument types and descriptions for a specified custom search. The XmimProcessCustomSearch routinereturns an array containing the results of executing the given custom search with the specified arguments. TherootRel argument can be used to narrow the search space for the custom search. See the section below for acomplete description of defining and using custom searches.

Defining and Using Custom SearchesFrom a user's perspective, a custom search is a special way to generate lists of symbols in a LET. For example, thefollowing query can be used to find all equities ending with the string ABC: LET FOO = cs_regex (Equities, "ABC$")

SHOW

207

1: FOO WHEN Date is FOO last_data_day

Here, cs_regex is a custom search that performs regular expression matches. If you wish to access a specificfutures contract without wanting to remember the Commodity DataServer's naming conventions do so with thefollowing:

LET FOO = cs_contract_builder (TopRelation , "SP", 12, 1999)

SHOW 1: FOO WHEN Date is FOO last_data_day

The following will show how such a custom search can be written, compiled, and made available to the CommodityDataServer.

Writing Custom Searches Under the hood, the Commodity DataServer supports two different types of custom searches. A “tester” customsearch follows the pattern established by cs_regex. Tester custom searches check whether a particular relationmeets the search criteria or not. A “generator” function, on the other hand, directly suggests symbols that meet thesearch criteria, following the pattern set by cs_contract_builder.

Both tester and generator custom searches require a relation name to be specified as the first argument to the customsearch. The relation name is used to scope the search, so it is generally a category relation. If the search is to includeall relations in the database then the root of the relation hierarchy (TopRelation) must be specified.

Tester Custom Searches Writing a tester custom search is straightforward. The code below illustrates how cs_regex is written:

#include <stdio.h> #include <regex.h> #include <xmim_custom.h> static char *old_pattern = 0; static char *old_machine = 0; int cs_regex (void *handle, const char* relname) { const char* pattern = getStringArg (handle, 0); if (old_pattern == 0 || strcmp(pattern, old_pattern) != 0) { if (old_pattern != 0) free (old_pattern); old_pattern = strdup (pattern); if (old_machine != 0) free (old_machine);

208


old_machine = regcmp (old_pattern, 0); } return regex (old_machine, relname); }

All custom searches should #include the file xmim_custom.h, which defines the interface available to the customsearch routines.

Notice the signature of the cs_regex function:

int cs_regex (void *handle, const char* relname);

All tester functions have this same C/C++ signature. The handle argument is used to pass state information to thefunction; all functions in the custom search interface require the handle argument to be passed through unchanged.The relname argument is the symbol currently under consideration. The Commodity DataServer takes care of invokingthe cs_regex function repeatedly, once for each symbol that the user could be interested in. In our running example,the Commodity DataServer would call cs_regex once for each equity.

The function getStringArg is part of the interface available to custom searches. It is used to extract the stringarguments supplied in the call to the custom search. The similar functions getIntArg and getDoubleArg extractinteger and floating-point values, respectively. In our example, the pattern returned would be "ABC$".

Generator Custom Searches Generator functions are just as easy to write. Consider the definition of cs_contract_builder:

int cs_contract_builder (void* handle) { const char* futures = getStringArg (handle, 0); int month = getIntArg (handle, 1); int year = getIntArg (handle, 2); char buf[1024]; int yy, yy1, yy2; int mm, mm1, mm2; char *initial = "FGHJKMNQUVXZ"; if (year < 0) { yy1 = 1960; yy2 = 2000; } else { if (year < 100) year += 1900; yy1 = year; yy2 = year; } if (month < 0) { mm1 = 1; mm2 = 12; }

209

else { if (month > 12) return 0; mm1 = month; mm2 = month; } for (yy=yy1; yy<=yy2; yy++) { for (mm=mm1; mm<=mm2; mm++) { if (mm <= 12) { sprintf (buf, "%s_%04d%c", futures, yy, initial[mm-1]); addCSResult (handle, buf); } } } return 1; }

The signature for cs_contract_builder is shared by all generator functions:

int cs_contract_builder (void* handle);

Only the handle argument is passed through the argument list. The arguments to the custom search are retrievedusing the functions getIntArg, getDoubleArg, and getStringArg, just as in the previous case. The symbolsmatching the search criteria are returned by calling the function addCSResult, as done inside the loop at the end ofcs_contract_builder.

Loading Custom Searches

Producing Shared Libraries

How are these C functions compiled? It is important to produce a shared library that can be loaded into the CommodityDataServer. Details of this step depend on the specific compiler and UNIX flavor you are using. Consult your man pagesfor these details. For our purposes, we use the free tool libtool, for building shared libraries. We use the followingmakefile to create a custom search shared library:

libdir=/home/xmim/lib all: libcustom.la

custom.lo: custom.c libtool --mode=compile gcc -c -I. custom.c

libcustom.la: custom.lo libtool --mode=link gcc -O -o libcustom.la -rpath $(libdir) custom.lo

install: libcustom.la libtool --mode=install install libcustom.la $(libdir)/libcustom.la

clean: rm -rf .libs *.lo *.o *.la lib/*

210


This installs the library libcustom.so for Solaris or the library libcustom.sl for HPUX in the directory /home/xmim/lib.

Generating the Specification File

We now need to tell the Commodity DataServer to read this file when starting up. This is done with a custom searchspecification file, such as the following for Solaris:

LOAD "/home/xmim/lib/libcustom.so" BEGIN TESTER -> cs_regex 1 pattern string GENERATOR -> cs_contract_builder 3 futures string month int year int END

Notice that the file specifies which shared libraries to load. It is possible that more than one shared library needs to beloaded and that third-party drivers are required. It also specifies the custom searches defined (in this case cs_regexand cs_contract_builder). For each custom search, the number of arguments is specified, and for each argumentits name and type is specified.

Loading the Specification File To tell the Commodity DataServer to load this specification file, it is necessary to change the .xmimrc file in use andthen restart the Commodity DataServer. The .xmimrc must contain a line such as the following:

customSearch: /home/xmim/lib/custom.spc

The contents of the file /home/xmim/lib/custom.spc are as specified above. After the server is restarted, users will beable to use these two custom searches.

Accessing Custom Searches via the API Programmers are also able to access the custom searches directly through the API. This benefits those writingspecialized client applications. The functions XmimVaGetCustomSearches and XmimVaGetCustomSearchArgsare used to get the information contained in the specification file. This information can be used to build a GUIfor arbitrary custom searches. A custom search can be invoked with XmimVaProcessCustomSearch. Thespecifications for these routines are provided below.

XmimReturnCode XmimGetCustomSearches(XmimClientHandle handle, int *numCustSearch, XmimString **custSearches); XmimReturnCode XmimVaGetCustomSearches(XMIM_CLIENT_HANDLE, handle, XMIM_FUNCTION_ARRAY, &numCustSearch, &custSearches,


211

XMIM_END_ARGS); XmimReturnCode XmimGetCustomSearchArgs(XmimClientHandle handle, XmimString custSearch, int *numArgs, XmimString **argDescs, Int **argTypes); XmimReturnCode XmimVaGetCustomSearchArgs(XMIM_CLIENT_HANDLE, handle, XMIM_FUNCTION, custSearch, XMIM_DESCRIPTION_ARRAYS, &numArgs, &argDescs, &argTypes, XMIM_END_ARGS); XmimReturnCode XmimProcessCustomSearch(XmimClientHandle handle, XmimString custSearch, XmimString rootRel, int numArgs, XmimString *args, int *numResults, XmimString **results); XmimReturnCode XmimProcessCustomSearch(XMIM_CLIENT_HANDLE, handle, XMIM_FUNCTION, custSearch, XMIM_RELATION, rootRel, XMIM_ARG_ARRAY, numArgs, args, XMIM_RESULT_ARRAY, &numResults, &results, XMIM_END_ARGS);

Memory Managementvoid *XmimCopyArray (void *array, int numElements, int sizeElement); void *XmimMallocArray (int numElements, int sizeElement); int XmimFreeArray (void *array); XmimString *XmimCopyStrings (XmimString *stringArray, int numElements); XmimString *XmimMallocStrings (int numElements); int XmimFreeStrings (XmimString *stringArray, int numElements); float *XmimCopyValues (float *valuesArray, int numValues); float *XmimMallocValues (int numValues); int XmimFreeValues (float *valuesArray); XmimDateTime *XmimCopyDateTimes (XmimDateTime *datesArray, int numDates); XmimDateTime *XmimMallocDateTimes (int numDates); int XmimFreeDateTimes (XmimDateTime *datesArray); XmimBoolean *XmimCopyBooleans (XmimBoolean *booleansArray, int numBooleans); XmimBoolean *XmimMallocBooleans (int numBooleans); int XmimFreeBooleans (XmimBoolean *booleansArray);

212


The XmimCopyArray, XmimMallocArray and XmimFreeArray routines provide a mechanism for the user tooverride the automatic memory management provided by the Commodity DataServer C/C++ API. XmimCopyArraywill copy data into local storage such that the user maintains control over the data and the library code will notautomatically free it. Of course, the user must take responsibility for freeing the data using the XmimFreeArrayroutine. XmimMallocArray allocates space for an array of the specified size and initializes the space to zeros. Inaddition to the general memory management functions, convenience functions are provided for copying string arrays,value arrays, date arrays and boolean arrays.

Rollover SpecificationsA continuous contract rollover specification is comprised of a rollover switching day and a rollover policy. The rolloverswitching day language is summarized below:

[n dwmqy]

user definedlast data dayexpiration day

n th [to last] of [n months before] expiration monthvolume crossoveropen_interest crossover[day of week] of expiration week

[of month]

In the above specification, brackets (“[ ]”) enclose optional items; n represents a positive integer value; dwmqyrepresents days (trading), weeks, months, quarters or years; dow represents Monday, Tuesday, . . . ; and monthrepresents January, February, . . . , December. The usage of nth indicates that the integer value may optionally befollowed by the appropriate suffix (st, nd, rd, th) or the corresponding written word may be used (i.e., 1, 1st, first, 2,2nd, second, . . . , last). In the specification where either day of dow is used, day refers to actual calendar days, asopposed to trading days. The switching language is not case-sensitive (upper and lower case letters may be used asdesired to enhance readability). In all cases except one, the switching language identifies a rule to use for determiningthe day on which the current individual contract is dropped and the next one is used. When "user defined" is specifiedas the switching day, the dates to roll are read in from a file rather than determined by utilizing a rule. The syntax forthe switching date file17 consists of the individual contract name followed by a date which will be taken as the lastdate for which that contract will be used. Thus, CL_1983M 19830518 CL_1983N 19830617 CL_1983Q 19830718 CL_1983U 19830818

will be taken to mean Use CL_1983M up to and including 19830518 Use CL_1983N up to and including 19830617

17 This file must reside in the rollover dates directory which itself must reside in the same directory as the schema file. The filename should correspond to thename of the continuous contract being generated. If such a file does not exist then the Commodity DataServer will utilize a file corresponding to the nameof the parent relation and so forth up the relation hierarchy until a switching date file is found or the root relation is reached.


213

Use CL_1983Q up to and including 19830718 Use CL_1983U up to and including 19830818

The default switching day is last data day. Example switching day specifications are given below.

● 2nd Friday of expiration month● last day of 1 month before expiration month● 3 days before last data day● 7th day of expiration month● 2 weeks before expiration day of December

The rollover policies available include:

● Actual Prices● [Forward | Backward | Cash] Adjusted Prices● [n dwmqy] [Linear | Logarithmic | Geometric] Perpetual● [n contract] [Linear | Logarithmic | Geometric] Smoothed

In the above specification, brackets (“[ ]”) enclose optional items, the “ | ” character is used to show alter-natives,and n and dwmqy are the same as in the rollover day specification. The default rollover policy is Actual Prices. Noticethat if the rollover policy specified is Adjusted Prices then Forward Adjusted Prices will be used. Likewise, Perpetualwill default to 3 month Linear Perpetual and Smoothed to 2 contract Linear Smoothed. Additionally, all policies takeone or more of the following modifiers, which must be specified as prefixes to the rollover policy. If more than one ofthe modifiers are used, they must be specified in the order in which they are listed below.

● n nearby | front | back | far● [linear | logarithmic | geometric] interpolate for n dwmqy● year_to_year | months

In the above specification, n and dwmqy are the same as in the rollover day specification. The months usage is thesame as for the month rollover day specification except that multiple months may be specified. When months aregiven, only contracts for these months will be used. The default rollover policy is Actual Prices. If no modifiers arespecified for a rollover policy, 1 nearby is assumed. In constructing the rollover policy specification, modifiers mayoptionally be separated by commas and the specifications need not be case sensitive. Examples of rollover policyspecifications are given below.

● 3 month Logarithmic Perpetual● linear interpolate for 10 days, Actual Prices● 2 nearby March Actual Prices● front March June September December Actual Prices● 3 nearby, linear interpolate for 2 weeks, Actual Prices● Back Forward Adjusted Prices

When the XmimVaAddRelation routine is used, the rollover day and policy may be specified as either strings(using XMIM_ROLLOVER_DAY and XMIM_ROLLOVER_POLICY arguments) or pointers to structures (usingXMIM_ROLLOVER_DAY_STRUCT and XMIM_ROLLOVER_POLICY_STRUCT arguments). The structures are provided asan alternative for programmatic convenience and are shown below. typedef struct {

214


int numUnits; /* 'numUnits < 0' means 'before' */ /* 'numUnits > 0' means 'after' */ /* 'numUnits == 0' means no units specification (default) */ XmimUnits units; XmimSwitchingMethod method; XmimDateExpression dateExpression; XmimMonth contractMonth; } XmimRolloverDay;

This structure defines the rollover switching day. The method argument identifies the actual switching method withthe dateExpression argument further specifying the method involving expiration month. The numUnits and unitsarguments provide the optional time offset modifier to the switching method chosen. If numUnits is 0 then there isno time offset; if a positive number is specified then that number of units after the time event is used; if a negativenumber is specified then that number of units before the time event is used. The contractMonth argument providesan optional month specification for the chosen switching day. The supporting structures and enumerations are givenbelow.

typedef enum { XMIM_SWITCH_INVALID = 0, XMIM_SWITCH_LAST_TRADING_DAY = 1, XMIM_SWITCH_LAST_DATA_DAY = 1, XMIM_SWITCH_USER_DEFINED = 2, XMIM_SWITCH_FIRST_NOTICE_DAY = 3, XMIM_SWITCH_EXPIRATION_DAY = 4, XMIM_SWITCH_DATE_EXPRESSION = 5, XMIM_SWITCH_VOLUME_CROSSOVER = 6, XMIM_SWITCH_OPEN_INTEREST_CROSSOVER = 7 } XmimSwitchingMethod;

typedef enum { XMIM_DAY_INVALID = 0, XMIM_DAY_MON = 1, XMIM_DAY_TUE = 2, XMIM_DAY_WED = 3, XMIM_DAY_THU = 4, XMIM_DAY_FRI = 5, XMIM_DAY_SAT = 6, XMIM_DAY_SUN = 7, XMIM_DAY_OF_MONTH = 8 } XmimDay;

typedef enum { XMIM_MONTH_INVALID = 0, XMIM_MONTH_JAN = 1, XMIM_MONTH_FEB = 2, XMIM_MONTH_MAR = 3, XMIM_MONTH_APR = 4, XMIM_MONTH_MAY = 5, XMIM_MONTH_JUN = 6, XMIM_MONTH_JUL = 7, XMIM_MONTH_AUG = 8, XMIM_MONTH_SEP = 9, XMIM_MONTH_OCT = 10, XMIM_MONTH_NOV = 11,


215

XMIM_MONTH_DEC = 12 } XmimMonth;

typedef struct { int nTh; /* the member variable nTh is the compilation of */ /* the `(nTH | [nTH TO] LAST)' clause; its inerpretation is:*/ /* o 1 is first, 2 is second, etc. */ /* o 0 is invalid */ /* o -1 is last, -2 is "2nd to last", etc. */ XmimDay theDay; /* theDay encodes the compilation of the */ /* `(DAY | day_of_week)' clause */ int numMonths; /* numMonths encodes the compilation of the */ /*`[n MONTHS BEFORE]' clause */ } XmimDateExpression;

When the rollover day is defined in terms of the expiration month, the XmimDateExpression structure is used tospecify the desired day of the expiration month. If a positive integer is given for the nTh argument, it represents thefirst, second, . . . . If a negative number is given, it represents the last (-1), second to last (-2), . . . . Of course, 0 is aninvalid specification. The theDay argument specifies either a certain day of the week, or, simply, the given day of themonth. The numMonths argument indicates that the rollover day be the given number of months before the expirationmonth. This argument should be 0 when the rollover is to take place during the expiration month. typedef struct { int nearby; XmimInterpMethod interpolate; XmimYearToYear yearToYear; XmimRolloverDetail rolloverDetail; } XmimRolloverPolicy;

This structure is used to define the rollover policy. The nearby, interpolate, and yearToYear arguments specifythe optional modifiers for the rollover policy and the rolloverDetail argument indicates the actual policy used.The supporting structures and enumerations follow. typedef struct { XmimRolloverMethod rolloverMethod; union { XmimAdjustMethod adjustMethod; XmimSmoothMethod smoothMethod; XmimInterpMethod perpetMethod; } theMethod; } XmimRolloverDetail;

typedef enum { XMIM_ROLLOVER_INVALID = 0, XMIM_ROLLOVER_ACTUAL_PRICES = 1, XMIM_ROLLOVER_ADJUSTED_PRICES = 2, XMIM_ROLLOVER_PERPETUAL = 3, XMIM_ROLLOVER_SMOOTHED = 4 } XmimRolloverMethod;

typedef enum { XMIM_ADJUST_INVALID = 0, XMIM_ADJUST_FORWARD = 1,

216


XMIM_ADJUST_BACKWARD = 2, XMIM_ADJUST_CASH = 3 } XmimAdjustMethod;

typedef struct { int numContracts; XmimInterpAlgorithm algorithm; } XmimSmoothMethod;

typedef struct { int numUnits; XmimUnits units; XmimInterpAlgorithm algorithm; } XmimInterpMethod;

typedef enum { XMIM_INTERPOLATE_INVALID = 0, XMIM_INTERPOLATE_LINEAR = 1, XMIM_INTERPOLATE_LOG = 2, XMIM_INTERPOLATE_GEOMETRIC = 3 } XmimInterpAlgorithm;

typedef enum { XMIM_YEAR_TO_YEAR_NO = 0, XMIM_YEAR_TO_YEAR_JAN = 0x0001, XMIM_YEAR_TO_YEAR_FEB = 0x0002, XMIM_YEAR_TO_YEAR_MAR = 0x0004, XMIM_YEAR_TO_YEAR_APR = 0x0008, XMIM_YEAR_TO_YEAR_MAY = 0x0010, XMIM_YEAR_TO_YEAR_JUN = 0x0020, XMIM_YEAR_TO_YEAR_JUL = 0x0040, XMIM_YEAR_TO_YEAR_AUG = 0x0080, XMIM_YEAR_TO_YEAR_SEP = 0x0100, XMIM_YEAR_TO_YEAR_OCT = 0x0200, XMIM_YEAR_TO_YEAR_NOV = 0x0400, XMIM_YEAR_TO_YEAR_DEC = 0x0800, XMIM_YEAR_TO_YEAR_YES = 13 } XmimYearToYear;

The adjustMethod, smoothMethod and perpetMethod arguments are used in the XmimRolloverDetailstructure to provide additional arguments when one of the corresponding rollover policies is selected. The samestructure (XmimInterpMethod) is used for the interpolate modifier of the rollover policy and to provide detail for thePerpetual rollover method.

The month specifications in XmimYearToYear may be used singularly or may be ordered together to producethe desired multiple month specification. For example, to include March, June and September, the specificationwould be given as XmimYearToYear XMIM_YEAR_TO_YEAR_MAR | XMIM_YEAR_TO_YEAR_JUN |XMIM_YEAR_TO_YEAR_SEP. It is not valid to or the XMIM_YEAR_TO_YEAR_YES or XMIM_YEAR_TO_YEAR_NOentries with anything else and they will be ignored if this is done.int XmimSetRolloverDay (XmimString rolloverDayStr, XmimRolloverDay *rolloverDayStruct); XmimString XmimRolloverDayAsString (XmimRolloverDay rolloverDayStruct); int XmimSetRolloverPolicy (XmimString rolloverPolicyStr, XmimRolloverPolicy *rolloverPolicyStruct); XmimString XmimRolloverPolicyAsString (XmimRolloverPolicy rolloverPolicyStruct);


217

XmimSetRolloverDay and XmimSetRolloverPolicy are convenience routines provided for converting thestring version of the rollover day and policy to the structure version. The routines XmimRolloverDayAsString andXmimRolloverPolicyAsString convert the structure version of the rollover day and policy to the string version.

XmimReturnCode XmimGetRecordsRollover ( XmimClientHandle handle, XmimString relName, int numColumns, XmimString *colNames, XmimDate fromDate, XmimDate toDate, XmimTime fromTime, XmimTime toTime, int numUnits, XmimUnits units, XmimFillOption holidayFillOption, XmimFillOption missDataFillOption, XmimSkipAllNaN skipAllNaNRecords, int limit, XmimLimitMode limitMode, XmimString rolloverDay, XmimString rolloverPolicy, XmimAdjustedStudy study, int numFromUnits, XmimUnits fromUnits, int numToUnits, XmimUnits toUnits, int *numRecords, XmimDateTime **dates, float **values); XmimReturnCode XmimVaGetRecordsRollover ( XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_COLUMN_LIST, NULL, XMIM_FROM_DATE, XMIM_INVALID_DATE, XMIM_TO_DATE, XMIM_INVALID_DATE, XMIM_FROM_TIME, XMIM_INVALID_TIME, XMIM_TO_TIME, XMIM_INVALID_TIME, XMIM_UNITS, 1, XMIM_DAYS, XMIM_FILL_OPTION, XMIM_FILL_NAN, XMIM_FILL_NAN, XMIM_SKIP_NONE, XMIM_LIMIT, 0, XMIM_LIMIT_BY_RECORDS, XMIM_CURRENT_TICK_USAGE, XMIM_APPEND_TO_NONE, XMIM_ROLLOVER_DAY, NULL, XMIM_ROLLOVER_DAY_STRUCT, NULL, XMIM_ROLLOVER_POLICY, NULL, XMIM_ROLLOVER_POLICY_STRUCT, NULL, XMIM_STUDY_RULE, XMIM_ADJ_STUDY_NONE, XMIM_FROM_UNITS, 1, XMIM_DAYS, XMIM_TO_UNITS, 0, XMIM_DAYS, XMIM_NUM_RECORDS, &numRecords,

218


XMIM_DATE_TIMES, &dates, XMIM_VALUES, &values, XMIM_END_ARGS);

The XmimGetRecordsRollover routine can be used to compute a continuous contract for a given relation on thefly, without storing the resulting time series in the database. The rolloverDay and rolloverPolicy argumentsprovide the rollover specification for computing the continuous contract. The defaults for these arguments are last dataday and 1 nearby Actual Prices, respectively.

Note that when a rollover method is selected that involves adjustment of prices, the date/time rangearguments are used to restrict the range before the adjustment is applied. Also, the year_to_yearspecification, if given as part of the rolloverPolicy, will be ignored.

In a manner analogous to the XmimGetRecords routine, XmimGetRecordsRollover returns, as arguments,the date/value pairs comprising the time series as well as the number of resulting records (i.e., date/value pairs).As with XmimVaGetRecords, the XmimVaGetRecordsRollover routine provides the capability for optimizingthe data retrieval in the case where only a single time series is being retrieved (see “Retrieving Options DataRecords”). The numColumns and colNames arguments are used to specify the columns for which the rollover datashould be generated. If no columns are specified, rollover data will be generated for the bars (Open, High, Low, andClose) of the relation. The fromDate and toDate arguments are used to provide the range over which rolloverdata should be generated and the fromTime and toTime arguments can be used in the presence of tick data torestrict the trading day range. The numUnits and units arguments are used to specify the desired time periodover which the resulting data values should be aggregated. The holidayFillOption, missDataFillOptionand skipAllNaNRecords arguments identify how missing or invalid data should be handled and limitMode isused to restrict the results to a certain amount of memory or number of records. The Va version provides an extraargument, XMIM_CURRENT_TICK_USAGE, used to indicate whether historic data only should be used for computingthe continuous contract or whether the historic data should be supplemented with data from the high-frequencycurrent tick data store.

The study, numFromUnits, fromUnits, numToUnits, and toUnits arguments are used to specify that an“adjusted" study be computed for the given continuous contract. The study argument identifies the study to beadjusted. The default is none, such that the normal computation for the continuous contract specified takes place.If a study is specified, the from/to units arguments give the time offset for the study. If the number of units isgiven as 0, then that is taken as the current unit (e.g., 0 days indicates today). Thus, for the 1 day adjusted move,numFromUnits would be 1 and numToUnits would be 0 with both fromUnits and toUnits being XMIM_DAYS.Of course, the semantics of move are such that the 1 day move is taken as the difference between today's value andyesterday's value, whereas, for other studies, the semantics are such that 1 day would refer strictly to the current day.For example, to specify the 1 day adjusted average, numFromUnits and numToUnits would both be 0 and to specifythe 5 day adjusted average, numFromUnits would be 4 and numToUnits would be 0. The numFromUnits isimplicitly taken as the number of units prior to the current unit and the numToUnits is implicitly taken as the numberof units after the current unit. For example, using the Commodity DataServer language, the adjusted move from 5 daysago to 10 days later would be specified with numFromUnits as 5 and numToUnits as 10.XmimReturnCode XmimGetRecordsRolloverAllColumns (XmimClientHandle handle, XmimString relName, XmimDate fromDate, XmimDate toDate,


219

XmimTime fromTime, XmimTime toTime, int numUnits, XmimUnits units, XmimFillOption holidayFillOption, XmimFillOption missDataFillOption, XmimSkipAllNaN skipAllNaNRecords, int limit, XmimLimitMode limitMode, XmimString rolloverDay, XmimString rolloverPolicy, XmimAdjustedStudy study, int numFromUnits, XmimUnits fromUnits, int numToUnits, XmimUnits toUnits, int *numColumns, XmimString **colNames, int *numRecords, XmimDateTime **dates, float **values); XmimReturnCode XmimVaGetRecordsRolloverAllColumns ( XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_FROM_DATE, XMIM_INVALID_DATE, XMIM_TO_DATE, XMIM_INVALID_DATE, XMIM_FROM_TIME, XMIM_INVALID_TIME, XMIM_TO_TIME, XMIM_INVALID_TIME, XMIM_UNITS, 1, XMIM_DAYS, XMIM_FILL_OPTION, XMIM_FILL_NAN, XMIM_FILL_NAN, XMIM_SKIP_NONE, XMIM_LIMIT, 0, XMIM_LIMIT_BY_RECORDS, XMIM_CURRENT_TICK_USAGE, XMIM_APPEND_TO_NONE, XMIM_ROLLOVER_DAY, NULL, XMIM_ROLLOVER_DAY_STRUCT, NULL, XMIM_ROLLOVER_POLICY, NULL, XMIM_ROLLOVER_POLICY_STRUCT, NULL, XMIM_STUDY_RULE, XMIM_ADJ_STUDY_NONE, XMIM_FROM_UNITS, 1, XMIM_DAYS, XMIM_TO_UNITS, 0, XMIM_DAYS, XMIM_COLUMN_ARRAY, &numColumns, &colNames, XMIM_NUM_RECORDS, &numRecords, XMIM_DATE_TIMES, &dates, XMIM_VALUES, &values, XMIM_END_ARGS);

The XmimGetRecordsRolloverAllColumns routine generates rollover data for all existing columns of the relation.In a manner analogous to the XmimGetRecordsAllColumns routine, it returns the number of columns and thecolumn names for which rollover data was generated.

220


XmimReturnCode XmimGetRolloverDates (XmimClientHandle handle, XmimString relName, int numColumns, XmimString *colNames, int numUnits, XmimUnits units, XmimString rolloverDay, XmimString rolloverPolicy, int *numContracts, int *numPeriods, XmimDate **rollDates, XmimDate **contracts); XmimReturnCode XmimVaGetRolloverDates (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_COLUMN_LIST, NULL, XMIM_UNITS, 1, XMIM_DAYS, XMIM_ROLLOVER_DAY, NULL, XMIM_ROLLOVER_DAY_STRUCT, NULL, XMIM_ROLLOVER_POLICY, NULL, XMIM_ROLLOVER_POLICY_STRUCT, NULL, XMIM_NUM_CONTRACTS, &numContracts, XMIM_NUM_PERIODS, &numPeriods, XMIM_ROLL_DATES, &rollDates, XMIM_CONTRACTS, &contracts, XMIM_END_ARGS);

The XmimGetRolloverDates routine performs the continuous contract computation for the relation specified, butsimply returns the dates that the contracts would have rolled over, as well as which contracts are active during thatperiod. As with the XmimGetRecordsRollover routine, the numColumns and colNames arguments are usedto specify the columns for which the rollover information should be generated; numUnits and units are used tospecify the desired time period over which the rollover data values would be aggregated; the rolloverDay androlloverPolicy arguments provide the rollover specification for computing the continuous contract.

The numContracts argument returns the number of contracts studied per period for the rollover computation. Thenumber of contracts studied is always 1 except in the case where Smoothed or Perpetual is used for the rolloverpolicy. If Perpetual is used, numContracts will always be 2 because two contracts are always being compared forthat rollover policy. If Smoothed is used, numContracts will be the number of contracts specified in the rolloverpolicy, e.g., for the 2 contract Linear Smoothed rollover policy, numContracts would be 2. The numPeriodsargument returns that number of periods, where each period corresponds to a different choice of front contract. TherollDates argument returns an array of dates corresponding to the start of each rollover period. There will be onemore entry in the rollDates array than number of periods, with the last entry specifying the ending date of thelast rollover period. Thus, the complete date range for each rollover period can be determined from the rollDatesarray. The final argument, contracts, returns an array of dates that identifies the relevant contracts for each rolloverperiod, e.g., 3/1995 would be specified for the SP_1995H contract. The contracts are given as dates instead of stringssimply for user convenience; since the future is already known, the character string for the contract can be easilyconstructed by the user. The number of entries in this array will be numPeriods x numContracts.XmimReturnCode XmimGetRolloverDatesAllColumns (XmimClientHandle handle, XmimString relName, int numUnits, XmimUnits units, XmimString rolloverDay,


221

XmimString rolloverPolicy, int *numColumns, XmimString **colNames, int *numContracts, int *numPeriods, XmimDate **rollDates, XmimDate **contracts); XmimReturnCode XmimVaGetRolloverDatesAllColumns (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, relName, XMIM_UNITS, 1, XMIM_DAYS, XMIM_ROLLOVER_DAY, NULL, XMIM_ROLLOVER_DAY_STRUCT, NULL, XMIM_ROLLOVER_POLICY, NULL, XMIM_ROLLOVER_POLICY_STRUCT, NULL, XMIM_COLUMN_ARRAY, &numColumns, &colNames, XMIM_NUM_CONTRACTS, &numContracts, XMIM_NUM_PERIODS, &numPeriods, XMIM_ROLL_DATES, &rollDates, XMIM_CONTRACTS, &contracts, XMIM_END_ARGS);

The XmimGetRolloverDatesAllColumns routine returns the same information as XmimGetRolloverDates,only it does this for all existing columns of the relation. The number of columns and the column names for whichrollover information was generated is returned.

The complete reference for the rollover facility is provided in the Rollovers document.

Global OptionsThe Commodity DataServer C/C++ API provides functions that enable the client to set certain global options. Ofcourse, if a local value is given, it will override the global setting. Default values used by the Commodity DataServerfor all the global options are given in the XmimVaSetOption specification at the end of this section. Setting globaloptions will cause any corresponding optional arguments that are left unspecified to take on these new values ratherthan the initial default values given in the Va specifications in this document. For example, if the XmimSetUnitsroutine is used to set the global units to 1 week, then when XmimVaGetRecords is used without specifying theunits argument, the data will be retrieved and aggregated into a frequency of 1 week rather than the default 1 day.XmimBoolean XmimTestNaN (XmimClientHandle handle, float value); XmimBoolean XmimTestNaNType (XmimClientHandle handle, float value, XmimNaNType nanType); XmimReturnCode XmimGetNaN (XmimClientHandle handle,


222


XmimNaNType nanType); float *value); XmimReturnCode XmimSetNaN (XmimClientHandle handle, float value, XmimNaNType nanType); float XMIM_MISS_DATA_NAN; float XMIM_HOLIDAY_NAN;

typedef enum { XMIM_NAN_TYPE_HOLIDAY = 0, XMIM_NAN_TYPE_MISS_DATA = 1, XMIM_NAN_TYPE_ALL = 2 } XmimNaNType;

The XmimTestNaN routine checks a value to determine if it is invalid. It returns True if the value is invalid (NaN) forany reason and False otherwise. Invalid values can be further differentiated into two types, namely holidays andmissing data. The XmimTestNaNType routine has an argument, nanType, that specifies whether to check for aninvalid value due to a holiday, an invalid value due to missing data or both. Thus, if XMIM_NAN_TYPE_MISS_DATA isused, XmimTestNaNType checks the value given and returns True only if the value is invalid and does not fall on aholiday (i.e., is a missing data point), whereas if XMIM_NAN_TYPE_HOLIDAY is used, XmimTestNaNType will returnTrue if the value is invalid due to a holiday. If XMIM_NAN_TYPE_ALL is used, XmimTestNaNType will return True ifthe value is invalid for any reason, just as when the XmimTestNaN routine is used.

The XmimGetNaN and XmimSetNaN routines provide a facility to set and retrieve the value used by the CommodityDataServer to indicate a missing data point, as well as the value used by the Commodity DataServer to indicatea data point that falls on a holiday. When XMIM_NAN_TYPE_ALL is used in the XmimSetNaN routine, boththe holiday and missing data NaNs will be set to the given value. When XMIM_NAN_TYPE_ALL is used inthe XmimGetNaN routine, the system will return the value used for missing data, therefore having the sameresult as if XMIM_NAN_TYPE_MISS_DATA had been used. The defined constants XMIM_MISS_DATA_NAN andXMIM_HOLIDAY_NAN are the default values used by the Commodity DataServer to indicate a missing data point and aholiday data point, respectively.XmimReturnCode XmimSetFillOption (XmimClientHandle handle, XmimFillOption missDataFill, XmimFillOption holidayFill, XmimSkipAllNaN skipAllNaNRecords); XmimReturnCode XmimGetFillOption (XmimClientHandle handle, XmimFillOption *missDataFill, XmimFillOption *holidayFill, XmimSkipAllNaN *skipAllNaNRecords);

The XmimSetFillOption and XmimGetFillOption routines provide a means to set and retrieve global optionsfor specifying how NaNs (invalid data) should be dealt with by the server. The missDataFill and holidayFillarguments indicate the behavior for missing data and holiday NaNs, respectively. For each type of NaN, the callercan ask the database server to return the NaNs as part of the answer array, to carry over the previous knownvalue, to use the first following value, or to interpolate using either linear, geometric or logarithmic interpolation.The skipAllNaNRecords argument is an additional option that indicates if entire rows of NaNs in the answerarray should be skipped or not. If this argument is XMIM_SKIP_ALL_NAN, the answer array will be condensed byeliminating rows that consist entirely of NaNs. This applies for both types of NaNs.XmimReturnCode XmimSetDateFormat (XmimClientHandle handle,


223

XmimDateFormat dateFormat); XmimReturnCode XmimGetDateFormat (XmimClientHandle handle, XmimDateFormat *dateFormat); XmimReturnCode XmimSetExpDateFormat (XmimClientHandle handle, XmimExpDateFormat dateFormat); XmimReturnCode XmimGetExpDateFormat (XmimClientHandle handle, XmimExpDateFormat *dateFormat); XmimReturnCode XmimSetTimeFormat (XmimClientHandle handle, XmimTimeFormat timeFormat); XmimReturnCode XmimGetTimeFormat (XmimClientHandle handle, XmimTimeFormat *timeFormat); XmimReturnCode XmimSetPrintSeconds (XmimClientHandle handle, XmimBoolean printSeconds); XmimReturnCode XmimGetPrintSeconds (XmimClientHandle handle, XmimBoolean *printSeconds);

The above routines are used to set and retrieve the date and time and expiration date (for options) formats used bythe Commodity DataServer when printing data to a file. The XmimSetPrintSeconds and XmimGetPrintSecondsroutines are now actually obsolete since the Commodity DataServer now handles seconds in the time format. They areleft in merely to retain backward compatibility.XmimReturnCode XmimSetUnits (XmimClientHandle handle, int numUnits, XmimUnits units); XmimReturnCode XmimGetUnits (XmimClientHandle handle, int *numUnits, XmimUnits *units); XmimReturnCode XmimSetUnitsFrom (XmimClientHandle handle, XmimDateTime unitsFrom); XmimReturnCode XmimGetUnitsFrom (XmimClientHandle handle, XmimDateTime *unitsFrom);

The XmimSetUnits and XmimGetUnits routines set and retrieve the default units used by the CommodityDataServer C/C++ API. The XmimSetUnitsFrom and XmimGetUnitsFrom routines set and retrieve the date(for daily data) or time (for tick data) to use as a point of reference for the desired aggregation period. For example,if the current units were 1 hour and the unitsFrom is 8:30, the aggregation will be hourly ending on the half hour.Similarly, if the current units were 1 week and the unitsFrom is 6/1/94, since that date is a Wednesday, the weeklyaggregation will take place from Thursday to Wednesday. Thus, the date or time given corresponds to the end of theaggregation period and dates or times emanate from it both backwards and forwards. If the unitsFrom argument isset to XMIM_INVALID_DATE_TIME then the Commodity DataServer system defaults will be used (e.g., on the hourfor hourly, Monday to Friday for weekly, last day of the year for yearly).XmimReturnCode XmimSetLimit (XmimClientHandle handle, int limit, XmimLimitMode limitMode); XmimReturnCode XmimGetLimit (XmimClientHandle handle, int *limit, XmimLimitMode *limitMode);

The XmimSetLimit and XmimGetLimit routines set and retrieve the default limit on the amount of data returnedby the Commodity DataServer C/C++ API. Depending on the mode, limit specifies either the maximum number ofkilobytes of memory or the maximum number of records.XmimReturnCode XmimVaSetOption (XMIM_CLIENT_HANDLE, handle,

224


XMIM_NAN_VALUE, XMIM_MISS_DATA_NAN, XMIM_HOLIDAY_NAN_VALUE, XMIM_HOLIDAY_NAN, XMIM_FILL_OPTION, XMIM_FILL_NAN, XMIM_FILL_NAN, XMIM_SKIP_NONE, XMIM_PRINT_SECONDS, False, XMIM_EXP_DATE_FORMAT, XMIM_DATE_mmsddsyy, XMIM_DATE_FORMAT, XMIM_DATE_mmsddsyy, XMIM_TIME_FORMAT, XMIM_TIME_hh_mm_am, XMIM_UNITS, 1, XMIM_DAYS, XMIM_DATE_TIME, XMIM_INVALID_DATE_TIME, XMIM_LIMIT, 0, XMIM_LIMIT_BY_RECORDS, XMIM_END_ARGS); XmimReturnCode XmimVaGetOption (XMIM_CLIENT_HANDLE, handle, XMIM_NAN_VALUE, &value, XMIM_HOLIDAY_NAN_VALUE, &holidayValue, XMIM_FILL_OPTION, &missDataFill, &holidayFill, &skipAllNaNRecords, XMIM_PRINT_SECONDS, &printSeconds, XMIM_DATE_FORMAT, &dateFormat, XMIM_EXP_DATE_FORMAT, &expDateFormat, XMIM_TIME_FORMAT, &timeFormat, XMIM_UNITS, &numUnits, &units, XMIM_DATE_TIME, &unitsFrom, XMIM_LIMIT, &limit, &limitMode, XMIM_END_ARGS);

The XmimVaSetOption and XmimVaGetOption routines provide a convenient interface for setting or retrievingmultiple options. The options available consist of the ones that have been specified previously in this section.XmimReturnCode XmimSetDefaultHandle (XmimClientHandle handle);

The XmimSetDefaultHandle routine can be used such that the XMIM_CLIENT_HANDLE argument does not needto be provided for XmimVa routines. The handle argument must be a valid handle returned from the XmimConnectcall.

Date/Time Utility RoutinesThe following routines are convenience utilities. They do not take a handle argument and, hence, do not affect theserver, nor are they server-aware.XmimDate XmimSetDate (unsigned year, unsigned month, unsigned day); XmimTime XmimSetTime (unsigned hour, unsigned minute, unsigned second); XmimDateTime XmimSetDateTime (unsigned year, unsigned month, unsigned day, unsigned hour,

225

unsigned minute, unsigned second);XmimTime XmimSetTimeMillis (unsigned h, unsigned m, unsigned s, unsigned millis);XmimDateTime XmimSetDateTimeMillis (unsigned y, unsigned m, unsigned d, unsigned h, unsigned mi, unsigned s, unsigned millis);

The XmimSetDate, XmimSetTime and XmimSetDateTime routines are provided as convenience functions forsetting the date, the time or both. The routines XmimSetTimeMillis and XmimSetDateTimeMillis work likeXmimSetTime and XmimSetDateTime, except they also allow the user to specify the millisecond portion of the timestamp.

Note that support for millisecond data is not provided at this time for the set time convenience functions.

XmimBoolean XmimValidDate (XmimDate date); XmimBoolean XmimValidTime (XmimTime time); XmimBoolean XmimValidDateTime (XmimDateTime datetime);

The XmimValidDate, XmimValidTime and XmimValidDateTime routines can be used to check if a given date ortime is valid.int XmimDateCmp (XmimDate date1, XmimDate date2); int XmimTimeCmp (XmimTime time1, XmimTime time2); int XmimDateTimeCmp (XmimDateTime datetime1, XmimDateTime datetime2);

The XmimDateCmp, XmimTimeCmp and XmimDateTimeCmp routines are provided to facilitate comparisons with thedate/time structures returned by the API. These routines compare the two given date/time structures and return zero ifthey are identical, a negative value if the first date/time is before the second and a positive value if the first date/timeis after the second. If the client is written in C++, the comparison operators (==, !=, <, <=, >, >=) can alsobe used.void XmimPrintDate (XmimDate date); void XmimPrintDateWithFormat (XmimDate date, XmimDateFormat format); void XmimPrintExpDate (XmimDate date); void XmimPrintExpDateWithFormat (XmimDate date, XmimExpDateFormat format); void XmimPrintTime (XmimTime time, XmimBoolean printSecond); void XmimPrintTimeWithFormat (XmimTime time, XmimBoolean printSecond,

226


XmimTimeFormat format);void XmimPrintDateTime (XmimDateTime dateTime, XmimBoolean printTime, XmimBoolean printSecond); void XmimPrintDateTimeWithFormat (XmimDateTime dateTime, XmimBoolean printTime, XmimBoolean printSecond, XmimDateFormat dateFormat, XmimTimeFormat timeFormat);

The preceding routines are provided as convenience routines for printing dates, expiration dates, times and date/times. Since these routines do not take a connection handle, they will not utilize formats settings specified for a server(e.g., with the XmimSetTimeFormat and XmimSetDateFormat routines from the previous section). Instead, theroutines without arguments for specifying the desired format, will simply use the appropriate current default format.Of course, routines with format arguments can be used when a format other than the default one is desired. TheprintSecond argument is used in conjunction with printing times and specifies whether seconds should be includedor not. The printTime argument specifies whether the time should be printed or not, e.g., XmimGetRecordsreturns an XmimDateTime, however, when daily data is retrieved, the time should not be printed. A sample usage ofXmimPrintDateTime is shown below:for (i = 0; i < num_records; i ++) { if (units < XMIM_DAYS && units > XMIM_SECONDS) XmimPrintDateTime(dates[i], True, False); else if (units == XMIM_SECONDS) XmimPrintDateTime(dates[i], True, True); else XmimPrintDateTime(dates[i], False, False); if (XmimTestNaN (handle, values[i].close)) printf (", NaN"); else printf (", %8.3f", values[i].close); printf ("\n"); }

Of course, this code would be used after a call to XmimGetRecords (returning num_records as well as dates) hasbeen made.XmimString XmimStringDate (XmimDate date); XmimString XmimStringDateWithFormat (XmimDate date, XmimDateFormat format); XmimString XmimStringExpDate (XmimDate date); XmimString XmimStringExpDateWithFormat (XmimDate date, XmimExpDateFormat format); XmimString XmimStringTime (XmimTime time, XmimBoolean printSecond); XmimString XmimStringTimeWithFormat (XmimTime time, XmimBoolean printSecond, XmimTimeFormat format); XmimString XmimStringDateTime (XmimDateTime dateTime, XmimBoolean printTime, XmimBoolean printSecond); XmimString XmimStringDateTimeWithFormat (XmimDateTime dateTime, XmimBoolean printTime, XmimBoolean printSecond, XmimDateFormat dateFormat, XmimTimeFormat timeFormat);


227

The preceding routines are provided as convenience routines for returning the dates, expiration dates, times and date/times as strings in either the current default format or the format specified. These are analogous to the correspondingprint routines. The user does not need to worry about allocating/deallocating memory for the date/time stringsreturned. However, the returned string must be copied if it is to be used after a subsequent call to one of the routinesreturning date/time strings. For example, the following code allows the first returned date string to be retained andused:char buf[32]; XmimString date1_str; XmimString date2_str; strcpy (buf, XmimStringDate (date1)); date1_str = buf; date2_str = XmimStringDate (date2);

Internally, a static buffer of 32 bytes is used to hold the returned string so if the strcpy had not been done, thesecond call to XmimStringDate would have changed the contents of this buffer and the first date string would nolonger be accessible.

Logging Utility RoutinesFor a description of the Commodity DataServer logging facility, refer to "Chapter 22, “Commodity DataServer Logging”.For a description of setting/using different logging levels, refer to “Invoking a Server”. For information on archiving logfiles and resetting the log file without restarting the server, refer to Chapter 28, “xmim_svr_info Utility”.

Identifying a ClientXmimReturnCode XmimSetClientType(const XmimString type) XmimReturnCode XmimVaSetClientType(XMIM_CLIENT_TYPE, type)

The XmimSetClientType routine allows a client to register and provide an identifying string, type, which will beused by the Commodity DataServer logging facility. This call must be made prior to connecting to a server (prior tothe use of XmimConnect) so that the logging of the connect call will have the identifying client type information. Thestring value supplied may be one of the existing standards, but can also be any string that is desired to identify thetype of the client. The existing generic standard names are:

● PC DLL Client● PC Excel Client● VB API Client● API Client

The above will be used, as appropriate, by default if no further specification is given. That is, if a client does not callXmimSetClientType to identify itself, then, if it is a PC client, it will be logged as "PC Excel Client" if it uses the ExcelAdd-in, logged as "VB API Client" if it uses the VB API and otherwise logged as "PC DLL Client"; if it is not a PC client, itwill be logged as "API Client". The following client applications will register themselves automatically:

● xmim_client● bmim_client

228


The client type is a global setting and is a per-client property, not a per-connection property such that an additional callto XmimSetClientType will result in the previous identifying string being replaced with the new information beingused for any subsequent logging.

Logging MessagesXmimReturnCode XmimLogMessage(XmimClientHandle handle, int logLevel, XmimString message) XmimReturnCode XmimVaLogMessage(XMIM_CLIENT_HANDLE, handle, XMIM_LOG_LEVEL, 0, XMIM_LOG_MESSAGE, NULL, XMIM_END_ARGS)

The XmimLogMessage routine directs the server to write the specified message to the log. The message is loggedonly if the server is currently logging at the given logLevel.

Error Handlingvoid XmimPrintError (XmimString errorString); XmimString XmimStringError ();

The XmimPrintError routine is provided to give the application programmer control over whether or not to printan error message. If errorString is NULL, the system error message is printed. Otherwise, the given string isprepended to the system error message. The XmimStringError routine returns the system error message. Thereturned string should not be overwritten; it should only be used for reading. A sample usage of XmimPrintError isshown below:if (XmimVaAddRelation(XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, "IBM", XMIM_PARENT, "Equities", XMIM_TYPE, XMIM_REL_NORMAL, XMIM_END_ARGS) != XMIM_SUCCESS) XmimPrintError (NULL);

Of course, the same result could have been achieved with the following code:if (XmimVaAddRelation(XMIM_CLIENT_HANDLE, handle, XMIM_RELATION, "IBM", XMIM_PARENT, "Equities", XMIM_TYPE, XMIM_REL_NORMAL, XMIM_END_ARGS) != XMIM_SUCCESS) fprintf (stderr, XmimStringError());

Accessing Server InformationXmimReturnCode XmimGetServerInfo(XmimClientHandle handle,


229

int *serverNumber, XmimString *serverHost, XmimString *database, XmimString *mmapFile, XmimString *ownerName, int *pid) XmimReturnCode XmimVaGetServerInfo(XMIM_CLIENT_HANDLE, handle, XMIM_SERVER_NUMBER, &serverNumber, XMIM_SERVER_NAME, &serverHost, XMIM_DATABASE, &database, XMIM_MEMORY_MAP, &mmapFile, XMIM_DATABASE_ARRAY, &numDatabases, &databases, XMIM_USER_NAME, &ownerName, XMIM_PROCESS_ID, &pid, XMIM_END_ARGS)

The XmimGetServerInfo routine provides access to information pertaining to a Commodity DataServer masterserver that is running: the server number, host name it is running on, server database name, memory map file, owner'sname and process id. With the Commodity DataServer, the memory map file name returned will be the same asthe database name returned. The database and mmapFile arguments are available for backward compatibilityonly. It is more appropriate to use the Va version of this routine and use the XMIM_DATABASE_ARRAY argument inlieu of the XMIM_DATABASE and XMIM_MEMORY_MAP arguments such that all server databases are returned. ForXMIM_DATABASE_ARRAY, numDatabases will be of type int * and databases will be of type XmimString **. Ifmultiple databases exist for the server, an error message will be returned if the singular database return argument isspecified. All arguments are optional for the Va version.

Also available with the Commodity DataServer is a utility program, xmim_svr_info, that can be used to obtaininformation pertaining to one or more master servers that are currently running. See Chapter 28, “xmim_svr_infoUtility” for a description of this utility program.

Linking Libraries in C/C++ on WindowsThe following instructions show how to link the libraries with C/C++ code using the Microsoft Visual C++ 6.0software.

1. C/C++ libraries are shipped with every Commodity DataServer system. See the Customer Download page forinstructions on obtaining the libraries. The library files must be located in your PATH.

2. Start the Microsoft Visual C++ software.

The following set of instructions uses version 6.0.

3. Start a new project by selecting File>New from the menu bar.

http://customers.lim.com/download/index.html

230


4. Choose the Win32 Console Application. Enter a project name and location for the file. Select OK.


231

5. Create an empty project then select Finish.

232


6. The following dialog displays:

7. Add your source file.

The following programs reside on the Customer “Commodity DataServer C/C++ API” Downloadpage: xmim_get (detailed in Chapter 25, “xmim_get Utility”) a data extraction program,xmim_forward.c a forward curve data extraction program, and xmim_get_options (detailed inChapter 26, “xmim_get_options Utility”) an options data extraction program.

8. From the menu bar select Project>Setting. Select the C/C++ tab. Enter the following information:

a. In the Category field select Preprocessor from the pull-down menu.

http://customers.lim.com/download/download_capi.html

http://customers.lim.com/download/sample_prog/xmim_forward.c


233

b. In the Additional include directories field designate where the header file xmim_api.h is located.

9. Select the Link tab.a. For Category make sure Input is entered.b. For Object/library modules add the Library module to the end of the path: xmim4.lib.

234


c. For Additional library path add the path where the library resides.

10. Select OK to continue.

11. Run the build and the code will now compile.

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 6: Java API

235

CHAPTER6

Java APIFor a complete reference to the code, please see the Java Classes/Interface/Methods javadoc. To obtain theCommodity DataServer Java Application Programming Interface (API) software, contact customer support.

The Commodity DataServer Java API can be used in multi-threaded applications provided that onlyone thread uses 1 Connection at a time. Having multiple threads share a single Commodity DataServerconnection is unsafe.

Required SoftwareThe following software is required for the Commodity DataServer Java API:

● Java2 Standard Edition (J2SE Version 1.2 or later)● Commodity DataServer server: Version 4.5 or later. Recommend version 4.6.27 or later.

There is new functionality that won't work with server versions prior to 4.6.27.

API Library Files● oncrpc-version.jar● mimapi-version.jar

To use the Commodity DataServer Java API these library files must be included in your classpath.

Getting StartedThe following samples will help you to become familiar with the API.

http://customers.morningstarcommodity.com/download/java_comlim_mimapi/javadoc/index.html


236


Use the ConnectionFactory to connect to a Commodity DataServer. The connection object that is returned allowsyou to access the remote server. The networking to and from the server is handled for you. You must providethe hostname and port number for your Commodity DataServer. If you do not know this, ask your CommodityDataServer administrator. For this sample, hostname is Olympus and port is 0.

MimConnection conn = null; conn = ConnectionFactory.connect("Olympus", 0, "SampleApiProgram");

To ask the server for information about itself, call the getServerInfo() method. The MimServerInfo object that isreturned has information about your remote server.

MimServerInfo info = conn.getServerInfo(); String owner = info.getOwnerName(); String release_info = info.getReleaseInfo(); int pid = info.getPid(); int svr_num = info.getServerNumber();

System.out.println("MIM Server " + svr_num + " is running."); System.out.print(release_info); System.out.println("Started by user: " + owner); System.out.println("Process ID: " + pid);

Some API calls will return exceptions if your API call fails. This failure may result in a network failure, in which case aMimNetworkException is thrown. Failures reported by the Commodity DataServer will be MimExceptions. Herewe are catching both kinds of exceptions. If we didn't explicitly catch MimNetworkException, it would be caughtand handled by the MimException handler.

catch (MimNetworkException e) { e.printStackTrace(); } catch (MimException e) { e.printStackTrace(); }

Finally, always remember to disconnect from your server when you are done. If you don't, you may cause the server toleak file descriptors. Also, you will want to avoid excessive connecting and disconnecting which may cause a drop inperformance.

conn.disconnect();

For your convenience, the full program is listed below:

import com.lim.mimapi.ConnectionFactory;import com.lim.mimapi.MimConnection;import com.lim.mimapi.MimException;import com.lim.mimapi.MimNetworkException;import com.lim.mimapi.MimServerInfo;

public class SampleApiProgram { public static void main(String[] args) {

MimConnection conn = null;


237

try { conn = ConnectionFactory.connect("Olympus", 0, "SampleApiProgram");

MimServerInfo info = conn.getServerInfo(); String owner = info.getOwnerName(); String release_info = info.getReleaseInfo(); int pid = info.getPid(); int svr_num = info.getServerNumber();

System.out.println("MIM Server " + svr_num + " is running."); System.out.print(release_info); System.out.println("Started by user: " + owner); System.out.println("Process ID: " + pid); }

catch (MimNetworkException e) { e.printStackTrace(); } catch (MimException e) { e.printStackTrace(); }

finally { if (conn != null) { try { conn.disconnect(); } catch (MimException e) { e.printStackTrace(); } } } }}

The BasicsThe API supports a client/server model. It does this through a Remote Procedure Call (RPC) mechanism, so that theclient and the server need not reside on the same machine. The server must be brought up before any applications canutilize it and must be brought down manually.

Metadata TerminologyThis section provides some introductory terminology that may prove helpful in understanding the organization of theCommodity DataServer database and, therefore, this document. In the Commodity DataServer, a relation corresponds

238


to a trading instrument (i.e., equity, future, etc.) and a column corresponds to specific information for an instrument(i.e., economic or monetary values, market activity, etc.). Often, the assigned relation name will correspond to theticker symbol for that instrument. A column can be associated with a relation resulting in what is termed a relationcolumn. For example, the relation IBM could have the columns Open, High, Low, Close and Volume associated withit to form the relation columns Open of IBM, High of IBM, etc. A relation column, thus, corresponds to an actual timeseries. Data is actually added only to relation columns.

In the Commodity DataServer, both relations and columns are organized hierarchically. When a relation or columnis added, the parent must always be specified, with TopRelation and TopColumn denoting the roots of the relationand column trees, respectively. Special relations and columns typed as categories are used to group relations andcolumns; they have all the properties of relations and columns but will never have data associated with them.Relations and columns that are not solely used for grouping and do have associated data are referred to in thisdocument as leaf-level relations and columns because they typically correspond to terminal leaves in the treehierarchy.

In the Commodity DataServer, neither leaf-level relations and columns, nor category relations and columns are requiredto have unique names. However, in specifying non-unique relations or columns, a disambiguating path must beprovided. This path must consist of the ancestors (i.e., parent, grandparent, etc.) of the relation or column goingback as far as necessary to disambiguate. Colons are used as separators if the path is specified with immediatedescendants; a double colon is used as a separator to indicate a descendant that is not necessarily an immediatedescendant. For example, if a single database exists with both Corn and Chrysler and they both use C for the relationname, Corn could be referenced unambiguously as Futures:C if Futures was the parent and could be referencedunambiguously as Futures::C if C was a descendant of Futures but not necessarily an immediate descendant. Likewise,Chrysler might be referenced unambiguously as Equities:C if Equities was the parent and as Equities::C is Equities wasan ancestor but not necessarily the parent.

In the context of multiple databases, relations and columns that are not unique across databases need not be referredto with a disambiguating path name. In that case, the order of loading the databases provides a resolution to theambiguity in that databases will be searched in the order loaded and the first one with the specified relation or columnname will be used.

In the Commodity DataServer, the relation hierarchy is not required to be a strict single-path tree hierarchy. By usingthe relation alias facility, multiple paths to a relation can be assigned. Multiple names can also be associated with arelation via this facility. Refer to “Adding Relation Aliases” for a description of using the relation alias facility.

The relation and column hierarchies together with all information pertaining to the attributes of existing relations,columns and relation columns comprise the metadata of the database: termed the Commodity DataServer schema.Thus, the structure of the Commodity DataServer database (consisting of the actual time-series data values) isdescribed by the Commodity DataServer schema.

Connecting and Disconnecting from a ServerIn this section, we will describe the basic structure of a Java client. First, a client must include the necessary jarfiles. Next, it should obtain a MimConnection object from the ConnectionFactory . There is a static method


239

called connect that establishes the connection with the desired Commodity DataServer. Finally, the client must callMimConnection.disconnect before terminating. Thus, the following example shows the minimal Java client:

package com.lim.mimapi.test;

import com.lim.mimapi.ConnectionFactory;import com.lim.mimapi.MimConnection;import com.lim.mimapi.MimException;

public class Connect { public static void main(String[] args) { try { MimConnection mim = ConnectionFactory.connect("hostname", 0, "Connect"); mim.disconnect(); } catch (MimException e) { e.printStackTrace(); } }

}

The serverName specifies the hostname of the machine on which the Commodity DataServer server is running.The serverNumber argument is used to specify which server to connect with. If there is no server running on thespecified host, an exception is thrown. If the call succeeds, a MimConnection object is returned.

Any API call can throw a MimException. MimExceptions may be thrown as a result of an errorcondition in the Commodity DataServer. MimNetworkException is derived from MimException andmay be thrown as a result of a networking problem.

Requesting information for a relation that doesn't exist is an example of a MimException. Trying toconnect to a host that doesn't exist is an example of a MimNetworkException.

MimConnection.disconnect has the following synopsis:

public void disconnect() throws MimException;

This routine will disconnect the client application from the server. The server is not brought down by a call todisconnect even if there are no other clients connected to the server. It is very important that clients call disconnectbefore exiting for any reason (error condition or normal exit) in order for the client handle space not to be exhausted.

Changes to the InterfaceThere is a small set of interfaces that define the Commodity DataServer Java API. The existing methods in theseinterfaces will not change. Any modifications or new functionality will result in new methods being added to the

240


interfaces. If an API call is determined to be broken, we may deprecate that call and introduce a new call. This is donein an effort to maintain backwards compatibility.

Data Retrieval

Browsing the Relation HierarchyIn this section, we show how the Commodity DataServer Java API can be used to write a client that browses the datain an Commodity DataServer database. The relation hierarchy can be browsed using the call getRelChildren. Thesynopses for this call is as follows: public RelationChildInfo[] getRelChildren(String relationName) throws MimException;

The getRelChildren method routines require the relation name (ticker symbol) to be specified as an argument. Ifthe relation name is not unique then a disambiguating path name must be specified with the colon (:) character usedas a separator (e.g., Futures:Indices). The getRelChildren method returns an array of RelationChildInfo objects.The length of the array is equal to the number of immediate children. Each object contains a name, the relation typeand whether or not that child has sub-children. The relation types are defined as follows: public static final RelType INVALID = new RelType(INVALID_VAL); public static final RelType CATEGORY = new RelType(CATEGORY_VAL); public static final RelType NORMAL = new RelType(NORMAL_VAL); public static final RelType FUTURES = new RelType(FUTURES_VAL); public static final RelType FUTURES_CONTINUOUS = new RelType( FUTURES_CONTINUOUS_VAL); public static final RelType FUTURES_CONTRACT = new RelType( FUTURES_CONTRACT_VAL); public static final RelType OPTIONS = new RelType(OPTIONS_VAL);

The routine getDataRange can be used to determine the dates for which data is available for a given relation. Thesynopsis for this call is as follows: public DataRange getDataRange(String relationName, String[] columnName, MimUnits units) throws MimException;

The getDataRange routine returns a DataRange object. The DataRange object contains the earliest (fromDate)and latest (toDate) days over which a given type of data is available. The unit argument indicates which type of data,with XMIM_SECONDS indicating tick-by-tick, XMIM_MINUTES indicating intraday tick and XMIM_DAYS indicating daily.The colNames argument can be used to limit the columns considered for determining the data range of a relation. IfcolNames is null, all columns for the relation will be considered.

MimDateTime should be used when both the date and the time need to be specified, MimDate when only the dateinformation is relevant and MimTime when only the time information is relevant. The API also provides the specialconstants MimDate.INVALID_DATE , MimTime.INVALID_TIME , and MimDateTime.INVALID_DATE_TIME.

241

MimDateTime, MimDate, and MimTime have equals and compareTo methods for performing date and timecomparisons.

As an example, the following program traverses the Commodity DataServer relation hierarchy, printing the relationnames as well as the type of data currently available for each leaf relation.package com.lim.mimapi.test;

import com.lim.mimapi.ConnectionFactory;import com.lim.mimapi.DataRange;import com.lim.mimapi.MimConnection;import com.lim.mimapi.MimException;import com.lim.mimapi.MimSchema;import com.lim.mimapi.MimUnits;import com.lim.mimapi.RelType;import com.lim.mimapi.RelationChildInfo;

public class GetRelationInfo { public static void main(String[] args) { if (args.length != 1) { System.out.println("Usage: GetRelationInfo <relation>"); System.exit(1); } String rel = args[0]; try { MimConnection mim = ConnectionFactory.connect("auslm13", 0, "GetRelationInfo"); MimSchema schema = mim.getSchemaManager();

RelationChildInfo[] children = schema.getRelChildren(rel); for (int i = 0; i < children.length; i++) { System.out.print(children[i].childName); if (children[i].type == RelType.CATEGORY) { System.out.println(" (category)"); } else { MimUnits days = MimUnits.DAYS; MimUnits mins = MimUnits.MINUTES; MimUnits secs = MimUnits.SECONDS; String childName = rel + ':' + children[i].childName; String[] cols = schema.getRelColumns(childName); DataRange daily_range = schema.getDataRange(childName, cols, days); DataRange intraday_range = schema.getDataRange(childName, cols, mins); DataRange tick_range = schema.getDataRange(childName, cols, secs); StringBuffer buf = new StringBuffer(); if (daily_range.getFromDate().isValid()) { buf.append("daily"); } if (intraday_range.getFromDate().isValid()) { if (buf.length() > 0) { buf.append(", ");

242


} buf.append("intraday"); } if (tick_range.getFromDate().isValid()) { if (buf.length() > 0) { buf.append(", "); } buf.append("tick"); } if (buf.length() == 0) { buf.append("none"); } System.out.println(" (" + buf.toString() + ")"); } }

} catch (MimException e) { e.printStackTrace(); } }}

Examining the Columns of a RelationThe example above illustrated how to browse the relation hierarchy. The next step is to examine, in detail, a particularrelation. For example, we may want to find out which columns the relation has (e.g., Open, High, Low, Close), or wemay want to extract the data for a given set of columns. The next example shows how we can determine the columnsin a relation, using the getRelColumns routine. The getRelColumns method returns an array of strings, whereeach string in the array is a column for the given relation. Its synopsis is as follows:

public String[] getRelColumns(String relationName) throws MimException;

This example program also uses the getDataRange routine and prints the resulting daily and intraday tick dataranges.


import com.lim.mimapi.ConnectionFactory;import com.lim.mimapi.DataRange;import com.lim.mimapi.MimConnection;import com.lim.mimapi.MimException;import com.lim.mimapi.MimSchema;import com.lim.mimapi.MimUnits;

public class GetRelColRanges { public static void main(String[] args) { if (args.length != 1) { System.out.println("Usage: GetRelColRanges <relation>"); System.exit(1); } String rel = args[0];

243

try { MimConnection mim = ConnectionFactory.connect("auslm13", 0, "GetRelColRanges"); MimSchema schema = mim.getSchemaManager();

MimUnits days = MimUnits.DAYS; MimUnits mins = MimUnits.MINUTES; String[] cols = schema.getRelColumns(rel); System.out.println("Columns for " + rel); for (int i = 0; i < cols.length; i++) { System.out.println("\t" + cols[i]); String[] column = new String[1]; column[0] = cols[i]; DataRange daily_range = schema.getDataRange(rel, column, days); DataRange intraday_range = schema.getDataRange(rel, column, mins); if (daily_range.getFromDate().isValid()) { String daily_str = "\t\t (daily (" + daily_range.getFromDate().toString() + ", " + daily_range.getToDate().toString() + ")"; System.out.println(daily_str); } if (intraday_range.getFromDate().isValid()) { String intraday_str = "\t\t (intraday (" + intraday_range.getFromDate().toString() + ", " + intraday_range.getToDate().toString() + ")"; System.out.println(intraday_str); } } } catch (MimException e) { e.printStackTrace(); } }}

Retrieving Data Records

Accessing Data for a Relation

Next, the following shows how to write a simple program that returns the bars for a particular relation. There areseveral ways of accessing the data for a relation. Start with the routine getRecords, with the following synopsis:

public RecordSet getRecords(GetRecordsParameters inputObj) throws MimException;

The GetRecordsParameters object has several properties you can set to specify the data to be fetched.The relNames and colNames arguments specify the relations and columns from which to get data. The dates(fromDate and toDate) allow the caller to specify the appropriate range for the data; if these are set to

244


MimDate.INVALID_DATE, then all the data will be retrieved. This range can be as small as desired: specifyingfromDate to be the same as toDate will retrieve the single record for that date. Likewise, the times (fromTime andtoTime) allow the caller to specify the appropriate trading time range over which to retrieve data for each date in thedate range; if these are set to MimTime.INVALID_TIME, then data will be retrieved for the entire trading time range.Of course, the from/to time arguments only make sense in the context of retrieving intraday, real tick or milliseconddata. The units and numUnits arguments specify what type of data to retrieve (e.g., 10 minute or 5 day bars). TheMimUnits class defines the following constants:

/** * Invalid time unit specified. */ public static final MimUnits INVALID = new MimUnits(INVALID_VAL); /** * Units in milliseconds. */ public static final MimUnits MILLISECONDS = new MimUnits(MILLISECONDS_VAL); /** * Units in seconds. */ public static final MimUnits SECONDS = new MimUnits(SECONDS_VAL); /** * Units in minutes. */ public static final MimUnits MINUTES = new MimUnits(MINUTES_VAL); /** * Units in hours. */ public static final MimUnits HOURS = new MimUnits(HOURS_VAL); /** * Units in days. */ public static final MimUnits DAYS = new MimUnits(DAYS_VAL); /** * Units in weeks. */ public static final MimUnits WEEKS = new MimUnits(WEEKS_VAL); /** * Units in months. */ public static final MimUnits MONTHS = new MimUnits(MONTHS_VAL); /** * Units in quarters. */ public static final MimUnits QUARTERS = new MimUnits(QUARTERS_VAL); /** * Units in years.

245

*/ public static final MimUnits YEARS = new MimUnits(YEARS_VAL);

When MimUnits.MILLISECONDS is used, millisecond data is retrieved and aggregated as specified. WhenMimUnits.SECONDS is used, real tick data (i.e., tick-by-tick) is retrieved and aggregated as specified. If realtick data is not available, then millisecond data (if available) will be used and aggregated appropriately. WhenMimUnits.MINUTES or MimUnits.HOURS is used, intraday tick data will be retrieved and aggregated. Ifintraday data is not available, then real tick data (if available) will be used and aggregated appropriately. When MimUnits.DAYS, MimUnits.WEEKS, MimUnits.MONTHS, MimUnits.QUARTERS or MimUnits.YEARS are used,daily data will be retrieved and aggregated into the desired frequency.

When you extract two relation-columns that have different trading calendars using getRecords, then theresulting matrix may contain NaNs. The reason for this is that the filling of NaNs for each relation-columnis handled independently (i.e., when one time series has data for a particular time stamp, but the othertime series does not).

A special type of column referred to as a multi-field column (or a composite column), while available for use with anytype of data, is particularly useful for real tick data. Multi-field columns have one or more fields defined, which, for realtick data, would correspond to data fields associated with the same tick transaction. When retrieving data for multi-field columns, individual field names may be specified as:

rnf_column_name:field_name

For example, if Bar were a multi-field column with a field named Open, the specification would be Bar:Open. If the fieldname is unique across all columns in the database such that it can be unambiguously resolved, the field name may bespecified alone, without the multi-field column name as a qualifier. If the multi-field column name itself is specified,data will be retrieved for all fields of the multi-field column.

The holidayFillOption and missDataFillOption allow the caller to specify different behaviors for dealing withthe two types of NaNs or invalid data in the database. For each type of NaN, the caller can ask the database server toreturn the NaNs as part of the answer array, to carry over the previous known value, to use the first following value, orto interpolate using either linear, geometric or logarithmic interpolation. This is specified using the FillOption class. Thefollowing FillOption constants are defined: /** * Fill option is invalid. */ public static final FillOption INVALID = new FillOption(INVALID_VAL);

/** * Specifies to fill missing values with NaNs. */ public static final FillOption FILL_NAN = new FillOption(FILL_NAN_VAL); /** * Fill missing values with the previous known value. * Note: For bar data, the last closing value is used for the entire bar; * that is open, high, low and close are all set to the previous close. */ public static final FillOption FILL_FORWARD = new FillOption(

246


FILL_FORWARD_VAL); /** * Fill missing values with the next known value. * Note: For bar data, the next opening value is used for the entire bar; * that is open, high, low and close are all set to the next open. */ public static final FillOption FILL_BACKWARD = new FillOption( FILL_BACKWARD_VAL); /** * Fill missing values by creating synthetic values for the span between * the last known value and the next known value by creating linear * interpolated values between them. * Formula:<pre> * (x1 * w2 + x2 * w1) / (w1 + w2)</pre> * Where:<pre> * x1 = last known value * x2 = next known value * w1 = number of days from x1 to the missing value's date * w2 = number of days from x2 to the missing value's date * </pre> */ public static final FillOption FILL_INTERP_LINEAR = new FillOption( FILL_INTERP_LINEAR_VAL); /** * Fill missing values by creating synthetic values for the span between * the last known value and the next known value by creating geometric * interpolated values between them. * Formula:<pre> ((x1^w2) * (x2^w1))^(1/(w1 + w2))</pre> * Where:<pre> * x1 = last known value * x2 = next known value * w1 = number of days from x1 to the missing value's date * w2 = number of days from x2 to the missing value's date * </pre> */ public static final FillOption FILL_INTERP_GEOMETRIC = new FillOption( FILL_INTERP_GEOMETRIC_VAL); /** * Fill missing values by creating synthetic values for the span between * the last known value and the next known value by creating logarithmic * interpolated values between them. * Formula:<pre> (w1 * ln(x2) + w2 * ln(x1)) / (w1 + w2)</pre> * Where:<pre> * x1 = last known value * x2 = next known value * w1 = number of days from x1 to the missing value's date * w2 = number of days from x2 to the missing value's date * </pre> */ public static final FillOption FILL_INTERP_LOGARITHMIC = new FillOption( FILL_INTERP_LOGARITHMIC_VAL);


247

/** * Fill missing values using the nearest known value. This option will fill * Saturday with Friday's value and Sunday with Monday's value. */ public static final FillOption FILL_NEAREST = new FillOption( FILL_NEAREST_VAL);

When FillOption.FILL_FORWARD is specified, the last closing value is used for the entire bar; that is, open, high,low and close are all set to the previous close. Likewise, when FillOption.FILL_BACKWARD is specified, thenext opening value is used for the entire bar. When any of the interpolate fill options are specified, the interpolation isperformed for each relation column resulting in a new (interpolated) bar.

An additional option available to the user for handling NaNs is to skip entire rows of NaNs in the answer array. TheskipAllNaNRecords argument is used for this purpose and is specified using the following constants:

public static final SkipAllNaN INVALID = new SkipAllNaN(INVALID_VAL); /** * Do not skip NaN values. */ public static final SkipAllNaN SKIP_NONE = new SkipAllNaN(SKIP_NONE_VAL); /** * Skip all the NaN values. */ public static final SkipAllNaN SKIP_ALL_NAN = new SkipAllNaN(SKIP_ALL_NAN_VAL);

If this argument is set to SkipAllNaN.SKIP_ALL_NAN, the answer array will be condensed by eliminating rows thatconsist entirely of NaNs. Note that this will apply for both types of NaNs.

The limit and limitMode allow the caller to limit the result to a certain amount of memory or number of records.If this amount is exceeded, the results will be cut off such that the least recent data is retrieved. Depending on themode, limit specifies either the number of kilobytes of memory or the number of records. The LimitMode classdefines the following constants: /** * An invalid limit mode */ public static final LimitMode INVALID = new LimitMode(INVALID_VAL); /** * Limit by the number of records returned. */ public static final LimitMode BY_RECORDS = new LimitMode(BY_RECORDS_VAL); /** * Limit by the amount of memory returned. */ public static final LimitMode BY_MEMORY = new LimitMode(BY_MEMORY_VAL);

The CurrentTickUsage class is used to indicate whether historic data only should be retrieved or whether datafrom the high-frequency current tick store should also be retrieved. The class defines the following constants: /**

248


* Append data from the real-tick store for daily and intraday data. */ public static final CurrentTickUsage APPEND_TO_ALL = new CurrentTickUsage( APPEND_TO_ALL_VAL); /** * Do not append any data from the real-tick store. */ public static final CurrentTickUsage APPEND_TO_NONE = new CurrentTickUsage( APPEND_TO_NONE_VAL); /** * Append data from the real-tick store for daily data. */ public static final CurrentTickUsage APPEND_TO_DAILY = new CurrentTickUsage( APPEND_TO_DAILY_VAL); /** * Append data from the real-tick store for intraday data. */ public static final CurrentTickUsage APPEND_TO_TICK = new CurrentTickUsage( APPEND_TO_TICK_VAL);

The CurrentTickUsage.APPEND_TO_NONE value indicates that data from the current tick store will not be utilized.If CurrentTickUsage.APPEND_TO_ALL is specified and the ending date/time range specified extends beyondwhat is available in the historic database, then data from the current tick store (if available) will be aggregatedinto the appropriate units and appended to the historic data. CurrentTickUsage.APPEND_TO_DAILY andCurrentTickUsage.APPEND_TO_TICK can be used to specify that current tick data should be appended onlyto daily data retrieval or only to tick (both intraday and real tick) data retrieval, respectively. Thus, if historic datais available ending yesterday, the current tick facility is being updated via a tick feed, daily bars are retrieved withthe default toDate and CurrentTickUsage.APPEND_TO_ALL or CurrentTickUsage.APPEND_TO_DAILY isspecified, the result will include historic daily data with an additional daily bar constructed from today's data storedso far in the current tick facility. Likewise, for intraday or real tick data retrieval, the current tick store will be accessed(if so specified) and the data stored so far will be aggregated and appended to the historic data. Note that since thehigh-frequency data store is optimized for fast data storage at the expense of slower data retrieval, if appending fromcurrent tick storage is indicated, the data retrieval will be much slower than if only historical data is retrieved (historicaldata is optimized for quick retrieval).

Finally, the RecordSet object contains the dateTimes and values. A record consists of all the values for all specifiedcolumns of all specified relations for a given date and time. So, the number of records will be exactly the number ofentries in the dates array. Each record corresponds to number of relations times number of columns floating-pointvalues. Therefore, there are many more entries in the values array than in the dates array. In some cases, thesenumbers will not be known at compile time, for example when the relations are read in from a file. In these cases, thecalling program will have to perform index arithmetic on the values array. In most cases, however, this information isknown in advance, so the client can hard-code the index into the value array.


import java.text.DecimalFormat;import java.text.NumberFormat;

import com.lim.mimapi.ConnectionFactory;import com.lim.mimapi.GetRecordsParameters;

249

import com.lim.mimapi.MimConnection;import com.lim.mimapi.MimData;import com.lim.mimapi.MimDate;import com.lim.mimapi.MimDateTime;import com.lim.mimapi.MimException;import com.lim.mimapi.RecordSet;

public class SimpleGetRecords { public static void main(String[] args) { if (args.length != 1) { System.out.println("Usage: SimpleGetRecords <relation>"); System.exit(1); } try { String relation = args[0]; String[] rels = { relation } ; String[] cols = { "Open", "High", "Low", "Close" }; MimDate fromDate = new MimDate(1994, 1, 1); MimConnection mim = ConnectionFactory.connect("auslm13", 0, "SimpleGetRecords"); MimData dataManager = mim.getDataManager(); GetRecordsParameters p = new GetRecordsParameters(rels, cols); p.setFromDate(fromDate); RecordSet rs = dataManager.getRecords(p); MimDateTime[] dts = rs.getDateTimes(); NumberFormat formatter = DecimalFormat.getInstance(); for (int row_idx = 0; row_idx < dts.length; row_idx++) { System.out.print(dts[row_idx]); for (int col_idx = 0; col_idx < cols.length; col_idx++) { double val = rs.getValue(row_idx, col_idx); if (col_idx != cols.length) System.out.print(", "); if (Double.isNaN(val)) { System.out.print("NaN"); } else { System.out.print(formatter.format(val)); } } System.out.println(); } } catch (MimException e) { e.printStackTrace(); } }

}

Notice that fromDate was set to 1/1/94, but toDate was set to invalid, so the code returns all values since thebeginning of 1994.

250


Executing Queries

MimData.queryExecuteThe MimData.queryExecute method can be used to run queries and retrieve the results programmatically. Thecomplete range of Commodity DataServer query results are available by accessing the query report object. Thiscode is similar to the actions of BMIM query_execute (“Executing Saved Queries” ). See “XMIMExecution andXMIMExecutionDouble” for the code equivalent (XMIMExecution) using the Commodity DataServer Visual Basicfor Applications API, and “XmimQueryExecute” for the code equivalent (XMIMQueryExecute) using the CommodityDataServer C/C++ API. For more information on the query language see the Commodity DataServer Query LanguageOverview .

The following shows the synopsis:

public QueryResult queryExecute(String query) throws MimException;

Example:

package com.lim.mimapi.sample;

import java.text.NumberFormat;

import com.lim.mimapi.ConnectionFactory;import com.lim.mimapi.MimConnection;import com.lim.mimapi.MimData;import com.lim.mimapi.MimDate;import com.lim.mimapi.MimException;import com.lim.mimapi.QueryReport;import com.lim.mimapi.QueryReportBlock;import com.lim.mimapi.QueryResult;

/** * A class demonstrating how to execute a query and process the results. */public class QueryExecuteSimple {

private String query = "Let\n" + " @=NG,CL,HO\n" + "SHOW\n" + " open: Open of @\n" + " bar:Bar of @\n" + "WHEN\n" + " date is within 1 month"; private int mimPort = 0; private String mimServer = "localhost"; private NumberFormat format = NumberFormat.getInstance();

public QueryExecuteSimple(String mimServer, int mimPort, String query) { // Just some formatting stuff. format.setMaximumFractionDigits(4);

http://customers.lim.com/pdfdocs/mql.pdf

http://customers.lim.com/pdfdocs/mql.pdf


251

format.setMinimumFractionDigits(4); this.mimServer = mimServer; this.mimPort = mimPort; if(query != null) { this.query = query; } }

/** * Entry point * @param args [host] [port] [query] */ public static void main(String[] args) { String mimServer = "localhost"; int mimPort = 0; String query = null; if(args.length > 0) { mimServer = args[0]; } if(args.length > 1) { mimPort = Integer.parseInt(args[1]); } if(args.length > 2) { query = args[2]; } QueryExecuteSimple q = new QueryExecuteSimple(mimServer, mimPort, query); q.run(); }

private void run() { MimConnection conn = null;

try { // Make the connection. conn = ConnectionFactory.connect(mimServer, mimPort, "Sample");

System.out.println(query + "\n------------------------");

// Execute queries using a MimData object. MimData data = conn.getDataManager();

// Execute the query. QueryResult rslt = data.queryExecute(query);

// Process the results. processQueryResult(rslt);

} catch (MimException e) { e.printStackTrace(); } finally {

252


// Always close our connection. if (conn != null) { try { conn.disconnect(); } catch (MimException e) { // We tried to disconnect. // Not much we can do here so just print the stack trace. e.printStackTrace(); } } } }

/** * Processes a QueryResult * @param rslt */ private void processQueryResult(QueryResult rslt) { // Each block contains column headings, dates and values. if (rslt.reports == null || rslt.reports.length == 0) { System.out.println("Empty results."); } else { // Query results may contain multiple reports for each LET // attribute. // Multiple reports are created for queries similar to // "LET sym=HO, CL, HU..." // In that case 3 reports are created. One each for HO, CL and HU. for (int ii = 0; ii < rslt.reports.length; ii++) { QueryReport report = rslt.reports[ii]; System.out.println("Report [" + ii + "]\n" + report.getTitle().trim() + "\n"); processQueryReport(report); } } }

/** * Processes a QueryReport * @param report */ private void processQueryReport(QueryReport report) {

if (report.blocks == null || report.blocks.length == 0) { System.out.println("No report blocks."); } else { // A report may contain multiple blocks for each SHOW statement. // Multiple block are created for queries such as // SHOW 1: Close of NG // SHOW 1: High of NG // WHEN date is within 1 month for (int jj = 0; jj < report.blocks.length; jj++) { QueryReportBlock block = report.blocks[jj]; if (block != null) { processQueryReportBlock(block); } }

253

} }

/** * Processes a QueryReportBlock * @param block */ private void processQueryReportBlock(QueryReportBlock block) { String[] columnHeadings = block.getColumnHeadings(); String [] attributeHeadings = block.getAttributeHeadings();

System.out.println("# of column headings: " + columnHeadings.length); System.out.println("# of columns: " + block.getNumCols()); System.out.println("# of attribute headings: " + attributeHeadings.length); System.out.println("# of attributes: " + block.getNumAttributes()); // Find the longest column heading so we can make the output look pretty. int len = 12; for (int ii=0; ii < columnHeadings.length; ii++) { if (len < columnHeadings[ii].length()) { len = columnHeadings[ii].length() + 1; } } for (int ii = 0; ii < block.getNumAttributes(); ii++) { System.out.println("# of cols for attribute " + block.getAttributeHeadings()[ii] + ": " + block.getNumColumnsForAttribute(ii)); }

System.out.print(fixedLenString(" ", len, ' ')); int [] columnHeadingIndices = block.getColumnHeadingIndices(); String previous = ""; for (int ii = 0; ii < columnHeadingIndices.length; ii++) { int idx = columnHeadingIndices[ii]; String colHeading = columnHeadings[idx]; if (!colHeading.equals(previous)) { System.out.print(fixedLenString(colHeading, len, '.')); previous = colHeading; } else { System.out.print(fixedLenString(".", len, '.')); } }

printDatesAndValues(block, len); }

private void printDatesAndValues(QueryReportBlock block, int len) { // Print out the dates and values. System.out.println(); MimDate[] dates = block.getRowDates(); int numRows = block.getNumRows(); int numCols = block.getNumCols();

for (int row = 0; row < numRows; row++) {

254


System.out.print(fixedLenString("" + dates[row], len, ' ')); for (int col = 0; col < numCols; col++) { // We can get a value for a specified row and column. double val = block.getValue(row, col); System.out.print(fixedLenString(format.format(val), len, ' ')); } System.out.println(); } } private String fixedLenString(String str, int len, char pad) { if (str.length() > len) { return str.substring(0, len); } else { StringBuffer buf = new StringBuffer(str); while (buf.toString().length() < len) { buf.append(pad); } return buf.toString(); } }}

Data Updating

Locking the DatabaseThis section shows how the Commodity DataServer Java API can be used to update data in a Commodity DataServerdatabase. Before any updates can be performed, the client must lock the Commodity DataServer database, using theMimConnection.lockDb method. Its synopsis, along with that for unlocking the database, is given below:

public void lockDb() throws MimException;

If the Commodity DataServer can not acquire a write-lock for the database, MimConnection.lockDb() will fail. TheCommodity DataServer supports a time-series level granularity of locking but this is performed transparently to theuser. The use of MimConnection.lockDb essentially puts the system to a mode whereby locking on a time-seriesbasis will occur as needed. MimConnection.unlockDb serves as the final commit for changes to the database andis when rollover computations will be performed. If multiple clients have locked the database, rollover computationswill be performed only after the last lock is released. Both updates to existing time series (e.g., adding or deleting datafor existing relation columns) and updates that require modifications to the Commodity DataServer schema will bevisible immediately following the update operation; unlocking is not necessary for these changes to be available. Theseroutines are applicable for locks pertaining to the historical database; for locks specific to the high-frequency data storesee “High Frequency Updating”.


255

Adding Relations, Columns, and Relation ColumnsRelations are added to the database using the MimSchema.addRelation routine. Columns are added to thedatabase using the MimSchema.addColumn routine. Relation columns are added to the database using theMimSchema.addRelationColumn. The following example demonstrates how to add the components necessary toincorporate real tick options data into the system.

Note that a multi-field column with two fields, namely Bid and Ask, is used; therefore, the multi-fieldrelation column has two object types specified, one for each field.


import com.lim.mimapi.Column;import com.lim.mimapi.ColumnComposite;import com.lim.mimapi.ConnectionFactory;import com.lim.mimapi.MimConnection;import com.lim.mimapi.MimException;import com.lim.mimapi.MimSchema;import com.lim.mimapi.RelColAggr;import com.lim.mimapi.RelColDelta;import com.lim.mimapi.RelColObjType;import com.lim.mimapi.RelColType;import com.lim.mimapi.Relation;import com.lim.mimapi.RelationColumn;import com.lim.mimapi.RelationColumnComposite;

public class MultiFieldRelCol {

public static void main(String[] args) { try { MimConnection mim = ConnectionFactory.connect("neo", 100, "MultiFieldRelCol"); mim.lockDb(); Relation rel = new Relation("MYSYM", "TopRelation:Equities"); MimSchema schemaManager = mim.getSchemaManager(); schemaManager.addRelation(rel); ColumnComposite col = new ColumnComposite("Quotes", "TopColumn:Price"); col.setFieldNames(new String[]{ "Bid", "Ask" }); col.setFieldDescriptions(new String[] { "Bid price", "Ask price" } ); schemaManager.addColumn(col); RelationColumnComposite mf_relcol = new RelationColumnComposite(); mf_relcol.setRelationName("MYSYM"); mf_relcol.setColumnName("Quotes"); mf_relcol.setType(RelColType.BASE); mf_relcol.setNumFields(2); RelColObjType[] obj_types = new RelColObjType[2]; obj_types[0] = RelColObjType.FLOAT; obj_types[1] = RelColObjType.FLOAT;

256


mf_relcol.setObjectType(obj_types); schemaManager.addRelationColumn(mf_relcol); mim.unlock(); mim.disconnect(); } catch (MimException e) { e.printStackTrace(); } }}

Storing a Vector of ValuesA simple way to update the Commodity Query data is to use the MimData.putRecords function, which is theconverse of MimData.getRecords. Its synopsis is as follows:

public void putRecords(String[] relNames, String[] colNames, MimUnits units, MimDateTime[] dateTimes, double[] values) throws MimException;

This routine should be used to update data in the database for the specified relations when the data is alreadyavailable in memory (i.e., already exists in an array). To update millisecond data, units should be specified as MimUnits.MILLISECONDS. To update real-tick data, units should be specified as MimUnits.SECONDS. Likewise,units should be MimUnits.MINUTES for intraday tick data and MimUnits.DAYS for daily data. Specify the inputdates and values in the dates and values arrays. The number of records must correspond to the number of entries inthe dates array. If data already exists for a record specified by an entry in the dates array, the effect will be to replacethe existing record.

Options data is stored just like a regular relation (relation type=normal). However, options must be stored underneatha special options relation category. For example, the options for DELL is stored under the DELL.Options category andoptions for IBM is stored under the IBM.Options category. Note: the category is mixed case, first letter is capitalizedthe rest is lowercase.

All options relations are required to have XML in the description field as such:

<plist> <type>option_type</type> <strikep>strike_price_with_decimal_price</strikep> <strike>strike_price</strike> <expir>expiration_date</expir></plist>

where:

option_type is either put or call strike_price is a numeric field (e.g., 30.0) expiration_date is a date (e.g.,3/17/03)

The property list can include other items (e.g., fields for root code, special settlement) as necessary but the <type>,<strike> and <expir> fields are required.


257


import com.lim.mimapi.ConnectionFactory;import com.lim.mimapi.MimConnection;import com.lim.mimapi.MimData;import com.lim.mimapi.MimDate;import com.lim.mimapi.MimDateTime;import com.lim.mimapi.MimException;import com.lim.mimapi.MimUnits;

public class PutRecords { public static void main(String[] args) { try { MimConnection mim = ConnectionFactory.connect("neo", 100, "PutRecords"); MimData dataManager = mim.getDataManager(); mim.lockDb(); String[] relNames = { "MYSYM" }; String[] colNames = { "Open", "High", "Low", "Close" }; MimDateTime[] dateTimes = new MimDateTime[2]; dateTimes[0] = new MimDateTime(new MimDate(2007, 5, 1)); dateTimes[1] = new MimDateTime(new MimDate(2007, 5, 2)); // values array contains: open value, high value, low value, close value, // open value, high value, low value, close value, etc. double[] values = { 5.0, 5.8, 4.8, 5.6, 5.5, 6.0, 5.0, 5.7 }; dataManager.putRecords(relNames, colNames, MimUnits.DAYS, dateTimes, values); mim.unlock(); mim.disconnect(); } catch (MimException e) { e.printStackTrace(); } }}

Loading Data From a FileWhen the data to be added to the Commodity DataServer is stored in a file in an ASCII format supported by theCommodity DataServer, it is more immediate to use the API equivalent of the BMIM facts_read command, theMimData.readFacts function. Its synopsis is as follows:

public void factsRead(String file, String rel, String[] cols, FieldFormat[] fieldFormats, RecordFormat recordFormat, MergeMode mergeMode, TickMode tickMode) throws MimException;

The relName specifies the relation for which data is being read. If this is NULL, the Commodity DataServer assumesthe relation is specified as part of the records in the file. This allows data for more than one relation to be read in

258


from a single file. If no columns are specified, Open, High, Low and Close will be used as the default columns. TherecordFormat specifies the style of formatting used in the file. Use one of the following constants: /** * Used when your data file uses FREE formatting */ public static final RecordFormat FREE = new RecordFormat(RECFMT_FREE); /** * Used when your data file uses FIXED formatting */ public static final RecordFormat FIXED = new RecordFormat(RECFMT_FIXED);

When RecordFormat.RECFMT_FREE is used, the Commodity DataServer accepts either commas orwhitespace as delimiters and the data is read in exactly as it appears in the file. On the other hand, ifRecordFormat.RECFMT_FIXED is used, the records are assumed to be in a fixed format. In these cases, thefieldFormats argument must be used to specify how to interpret the data read from the file by providing thetype, width and, if applicable, number of decimal digits for each field. When RecordFormat.RECFMT_FREE isused, the width and decimal members of the fieldFormats are ignored18When options data is being read, thefieldFormats must include specifications for the expiration date, the option type and the strike price. When multi-field column data is read, free-format must be used if the multi-field column name is specified instead of the fields.Note that when some, but not all, of the fields of a multi-field column are read, those fields that are not specified willbe filled with NaNs.

The mergeMode argument specifies whether the data read in is to be merged with any existing data for that relation,or whether it should replace the existing data.

The tickMode specifies whether daily or tick data is being read. Moreover, if tick data is being read, it indicateswhether the data is millisecond, real tick (second) or intraday data. Real tick will be automatically converted intointraday data by the system, if so specified. The class defines these constants: public static final TickMode NONE = new TickMode(TICK_NONE); public static final TickMode PRE_SAMPLED = new TickMode(TICK_PRE_SAMPLED); public static final TickMode BY_TICK = new TickMode(TICK_BY_TICK); public static final TickMode CONVERT = new TickMode(TICK_CONVERT); public static final TickMode MILLISECOND = new TickMode(TICK_MILLISECOND);

High Frequency UpdatingThe Commodity DataServer Java API also provides a mechanism for high-frequency updating of the tick data. Theroutines MimData.putRecords and MimData.factsRead pre-process the historical data so that it can beefficiently retrieved, while minimizing the storage requirements. However, to efficiently facilitate high-frequencyupdating, the time of insertion into the database needs to be minimized as a first priority. For this purpose, theCommodity DataServer provides the MimRealTime.updateRealTime method, with the following synopsis: public void updateRealTime(MimDate date, String[] rels, String[] cols, boolean realTick, MimTime[] times, float[] values) throws MimException;

18 Note that unexpected results may occur if the filler specification is used with free format since this will result in the corresponding field being totally ignored.


259

The date argument specifies the date for which tick data is to be updated in the high-frequency data store and therelNames and colNames give the relations and columns to be updated. As with the MimData.putRecords andMimData.factsRead routine, if no columns are specified, then Open, High, Low, and Close will be used as thedefault columns. The high frequency updating facility is available both for real-tick data and for intraday tick data; therealTick argument indicates which type of tick data is involved (False indicates intraday and True indicates realtick). The times and values are passed in the times and values arrays. The number of records must correspond to thenumber of entries in the times array.

An additional routine, MimRealTime.updateRealTimeBar, is available for use in cases where the tick data updateconsists of a single relation with a single price value (i.e., there is only a single update record) and it is desired topropagate the update to the entire bar such that when the data is retrieved, bar charts can be constructed. Utilizingthis routine represents a significant optimization for the updating process such that it should always be used if theconditions apply. The synopsis is as follows: public void updateRealTimeBar(String rel, MimDate date, MimTime time, float value) throws MimException;

The relName argument gives the relation to be updated in the high-frequency data store. The date and timearguments specify the date and time of the update and the value supplies the price value. This routine can be used forupdating either real tick or intraday tick data but, in the case of intraday data, the user must make sure to zero out theseconds field.

The Commodity DataServer maintains the data added by MimRealTime.updateRealTime (orMimRealTime.updateRealTimeBar) separately from the historical tick data. The store used for this high-frequencydata is referred to in this document as the current tick store. A database lock must be obtained prior to any updates;however, in the case of current tick data updates, a special lock that is specific to the current tick store is providedso that the historical data may remain unlocked. The routines for locking and unlocking the current tick store are asfollows: /** * Obtains the lock for writing into the real tick database. * @throws MimException */ public void lockRealTimeStore() throws MimException; /** * Releases the real tick lock. * @throws MimException */ public void unlockRealTimeStore() throws MimException; /** * Checks to see if the Real Time store is locked. TRUE if we are locked, * else FALSE. * @return TRUE if we are locked, else FALSE */ public boolean isLockedRealTime();

If the Commodity DataServer cannot acquire a write-lock for the current tick store,MimConnection.lockRealTimeStore will fail. In general, updates to the current tick store will be immediatelyvisible to any clients reading from the database.

260


During data retrieval, when the historical tick data and the current tick data intersect, the Commodity DataServeralways uses the historical data, under the assumption that historical tick data has been processed and “cleaned,"whereas the current tick data is being directly accessed from a potentially noisy feed.

Data from the current tick store can be retrieved using MimData.getRecords.

Sample ProgramsThe following are sample programs using the Commodity DataServer Java API.

GetRelationUse GetRelation to access the attributes of a relation. This routine will return (to name a few): the relationdescription, parent name, type, exchange, time zone, start trade, end trade, contract units, first notice day, rolloverpolicy and rollover day. The following shows the GetRelation program.

/* * Created on June 7, 2006 * * Copyright (c) 2006 Logical Information Machines, Inc.*/


import com.lim.mimapi.MimConnection;import com.lim.mimapi.MimException;import com.lim.mimapi.RelType;import com.lim.mimapi.Relation;import com.lim.mimapi.RelationFutures;import com.lim.mimapi.RelationFuturesContinuous;import com.lim.mimapi.RelationFuturesContract;import com.lim.mimapi.TradingPattern;import com.lim.mimapi.ConnectionFactory;

/** * Sample class to test the getRelation API * */public class GetRelation {

public static void usage() { System.out.println( "Usage: " + GetRelation.class.getName() + " <host> <port> <relation>"); }

/**

261

* The entry point into the application to test the getRelation method * @param args Command line arguments are * host: The name or ip address of the machine running the MIM server * port: The port on the host on which the MIM server is running * relation: The name of the relation to get */ public static void main(String[] args) {

// Parse input arguments if (args.length < 3) { usage(); System.exit(1); }

String host = args[0]; int port = Integer.parseInt(args[1]); String rel = args[2]; MimConnection mim = null;

Utils util = new Utils(); if (args.length == 4) { util.createResultFile(args[3], "GetRelation"); }

try { mim = ConnectionFactory.connect(host, port, "GetRelation"); relation = mim.getSchemaManager().getRelation(rel); type = relation.getType(); displayOutput(); } catch (MimException e) { util.displayResults("Exception: " + e.getMessage()); } finally { try { mim.disconnect(); } catch (Exception e) { util.displayResults("Exception: " + e.getMessage()); } } }

private static Relation relation; private static RelType type = null;

public static void displayOutput() {

Utils util = new Utils(); util.displayResults("Name: " + relation.getName()); util.displayResults("Parent: " + relation.getParent()); util.displayResults("Description: " + relation.getDescription()); util.displayResults("Type: " + type); util.displayResults("Trading pattern: " + TradingPattern.toString(relation.getTradingPattern())); util.displayResults("Begin time: " + relation.getBeginTradingTime()); util.displayResults("End time: " + relation.getEndTradingTime()); util.displayResults("Exchange: " + relation.getExchange()); if (type == RelType.FUTURES) {

262


util.displayResults("*Futures specific*"); RelationFutures future = (RelationFutures) relation;

util.displayResults( "Rollover policy: " + future.getRollRule()); util.displayResults("Rollover date: " + future.getRollDay()); util.displayResults( "Contract units: " + future.getContractUnits()); } else if (type == RelType.FUTURES_CONTINUOUS) { util.displayResults("*Futures continuous specific*"); RelationFuturesContinuous future = (RelationFuturesContinuous) relation;

util.displayResults( "Rollover policy: " + future.getRollRule()); util.displayResults("Rollover date: " + future.getRollDay()); } else if (type == RelType.FUTURES_CONTRACT) { util.displayResults("*Futures contract specific*"); RelationFuturesContract contract = (RelationFuturesContract) relation;

util.displayResults("Expiration date: " + contract.getExpirationDate()); util.displayResults("First notice day: " + contract.getFirstNoticeDay()); } }}

GetRelChildrenThe GetRelChildren routine provides a way to browse the relation hierarchy, and return a list of all the relation childrenfor a given relation.



import com.lim.mimapi.ConnectionFactory;import com.lim.mimapi.MimConnection;import com.lim.mimapi.MimException;import com.lim.mimapi.RelationChildInfo;

/** * Sample class to test the getRelChildren API * */public class GetRelChildren { public static void usage() { System.out.println(

263

"Usage: " + GetRelChildren.class.getName() + " <host> <port> <relation>"); }

/** * The entry point into the application to test the getRelChildren method * @param args Command line arguments are * host: The name or ip address of the machine running the MIM server * port: The port on the host on which the MIM server is running * relation: The name of the relation whose children will be retrieved */ public static void main(String[] args) { if (args.length < 3) { usage(); System.exit(1); } String host = args[0]; int port = Integer.parseInt(args[1]); String rel = args[2]; MimConnection mim = null;

Utils util = new Utils(); if (args.length == 4) { util.createResultFile(args[3], "GetRelChildren"); }

try { mim = ConnectionFactory.connect(host, port, "GetRelChildren"); RelationChildInfo[] rels = mim.getSchemaManager().getRelChildren(rel); if (rels.length > 0) { util.displayResults("Children for " + rel); for (int i = 0; i < rels.length; i++) {

/* String kids_str = "no children";

if (rels[i].hasChildren) { kids_str = "has children"; }*/ // TODO: these properties should have getters and setters. util.displayResults(" " + rels[i].childName/* + " - " + rels[i].type + " (" + kids_str + ")"*/); } } else { util.displayResults(rel + " has no children."); } } catch (MimException e) { util.displayResults(e.getMessage()); } catch (Exception e) { util.displayResults("Exception: " + e.getMessage()); } finally { try { if (mim != null) { mim.disconnect(); } } catch (Exception e) {

264


util.displayResults("Exception: " + e.getMessage()); } } }}

QueryExecuteQueryExecute can be used to run queries and retrieve the results programmatically. The complete range of CommodityDataServer query results are available by accessing the report block handles.



import java.text.NumberFormat;

import com.lim.mimapi.ConnectionFactory;import com.lim.mimapi.MimConnection;import com.lim.mimapi.MimException;import com.lim.mimapi.MimTime;import com.lim.mimapi.QueryReport;import com.lim.mimapi.QueryReportBlock;import com.lim.mimapi.QueryResult;import com.lim.mimapi.QuerySummary;import com.lim.mimapi.QuerySummaryBlock;

/** * Sample code to test the executeQuery API * */public class QueryExecute { public static void usage() { System.out.println( "Usage: " + GetRelation.class.getName() + " <host> <port> <query>"); }

/** * The entry point into the application to test the executeQuery method * @param args Command line arguments are * host: The name or ip address of the machine running the MIM server * port: The port on the host on which the MIM server is running * query: The query string for which results are to be obtained * The result is printed on the primary output device * */

265

public static void main(String[] args) {

if (args.length < 3) { usage(); System.exit(1); }

String host = args[0]; int port = Integer.parseInt(args[1]); String query = args[2]; MimConnection mim = null; QueryResult rslt = null;

Utils util = new Utils(); if (args.length == 4) { util.createResultFile(args[3], "QueryExecute"); }

try { mim = ConnectionFactory.connect(host, port, "QueryExecute");

rslt = mim.getDataManager().queryExecute(query);

for (int i = 0; i < rslt.reports.length; i++) { QueryReport report = rslt.reports[i];

util.displayResults( "\tnum blocks for report " + i + ": " + report.blocks.length); util.displayResults("The block data would be:"); if (report != null) { if (report.blocks != null) { for (int j = 0; j < report.blocks.length; j++) { QueryReportBlock block = report.blocks[j]; if (block != null) { for (int k = 0; k < block.getColumnHeadings().length; k++) { util.displayResults( "\t" + block.getColumnHeadings()[k]); }

for (int k = 0; k < block.getRowDates().length; k++) { MimTime[] times = block.getRowTimes(); MimTime time = null; if (times != null) time = block.getRowTimes()[k]; NumberFormat formatter = NumberFormat.getInstance(); formatter.setMaximumFractionDigits(4); formatter.setMinimumFractionDigits(4); double value = block.getValues()[k]; String value_str = "NaN"; if (!Double.isNaN(value)) value_str = formatter.format(value); util.displayResults(

266


"\t" + block.getRowDates()[k].toString() +(time != null && time.isValid() ? ", " + time.toString() : "") + "\t" + value_str + "\n"); }

util.displayResults("\nSummary Report"); QuerySummary summary = block.getSummaryReport(); if (summary.summaryBlocks != null) { for (int k = 0; k < summary.summaryBlocks.length; k++) { QuerySummaryBlock summaryBlock = summary.summaryBlocks[k]; if (summaryBlock != null) { util.displayResults( "average :" + summaryBlock.average); util.displayResults( "averageNegative :" + summaryBlock.averageNegative); util.displayResults( "averagePositive :" + summaryBlock.averagePositive); util.displayResults( "highest :" + summaryBlock.highest); util.displayResults( "lowest :" + summaryBlock.lowest); util.displayResults( "percentNegative :" + summaryBlock.percentNegative); util.displayResults( "percentPositive :" + summaryBlock.percentPositive); util.displayResults( "standardDeviation :" + summaryBlock.standardDeviation); util.displayResults( "sum :" + summaryBlock.sum); util.displayResults( "variance :" + summaryBlock.variance); util.displayResults( "zstat :" + summaryBlock.zstat);

267

} } } } } }

if (report.plReport != null) { if (report.plReport.tradeRecords != null) { for (int j = 0; j < report.plReport.tradeRecords.length; j++) { util.displayResults( "cumProfit : " + report.plReport.tradeRecords[j].cumProfit); util.displayResults( "entryPrice : " + report.plReport.tradeRecords[j].entryPrice); util.displayResults( "entryReasonLabel : " + report.plReport.tradeRecords[j].entryReasonLabel); util.displayResults( "exitPrice : " + report.plReport.tradeRecords[j].exitPrice); util.displayResults( "exitReasonLabel : " + report.plReport.tradeRecords[j].exitReasonLabel); util.displayResults( "move : " + report.plReport.tradeRecords[j].move); util.displayResults( "numContracts : " + report.plReport.tradeRecords[j].numContracts); util.displayResults( "percentMove : " + report.plReport.tradeRecords[j].percentMove); util.displayResults( "profit : " + report.plReport.tradeRecords[j].profit); util.displayResults( "entryDate : " + report.plReport.tradeRecords[j].entryDate); util.displayResults( "entryReason : " + report.plReport.tradeRecords[j].entryReason.toString()); util.displayResults(

268


"entryTime : " + report.plReport.tradeRecords[j].entryTime); util.displayResults( "exitDate : " + report.plReport.tradeRecords[j].exitDate); util.displayResults( "exitReason : " + report.plReport.tradeRecords[j].exitReason); util.displayResults( "exitTime : " + report.plReport.tradeRecords[j].exitTime); util.displayResults( "type : " + report.plReport.tradeRecords[j].type.toString());

} } } if (report.plReport.statRecords != null) { for (int j = 0; j < report.plReport.statRecords.length; j++) { util.displayResults( report.plReport.statRecords[j].combinedValue + " " + report.plReport.statRecords[j].label + " " + report.plReport.statRecords[j].longValue + " " + report.plReport.statRecords[j].shortValue + " " + report.plReport.statRecords[j].combinedDate + " " + report.plReport.statRecords[j].combinedTime + " " + report.plReport.statRecords[j].longDate + " " + report.plReport.statRecords[j].longTime + " " + report.plReport.statRecords[j].shortDate + " " + report.plReport.statRecords[j].shortTime); } } if (report.plReport.equityRecords != null) { for (int j = 0; j < report.plReport.equityRecords.length; j++) { util.displayResults(


269

"close : " + report.plReport.equityRecords[j].close); util.displayResults( "combined : " + report.plReport.equityRecords[j].combined); util.displayResults( "open : " + report.plReport.equityRecords[j].open); util.displayResults( "date : " + report.plReport.equityRecords[j].date); util.displayResults( "time : " + report.plReport.equityRecords[j].time); }

} } } } catch (MimException e) { util.displayResults("Exception: " + e.getMessage()); } finally { try { if (mim != null) { mim.disconnect(); } } catch (Exception e) { util.displayResults("Exception: " + e.getMessage()); } } }}

Error Messages

ONC/RPC Error MessageIf you receive an ONC/RPC error message from the API, please contact Client Support ([email protected]). The error isdue to either:

● a compatibility problem between the API version and the Commodity DataServer● or the Commodity DataServer abnormally disconnected


mailto: [email protected]

270


COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 7: .NET API

271

CHAPTER7

.NET API

The .NET API uses the same class names, interface names, method names and property names as the Java API.

For a complete reference to the code, please see the Java Classes/Interface/Methods javadoc. To obtain theCommodity DataServer .NET API software, contact customer support.

The Commodity DataServer .NET API can be used in multi-threaded applications provided that onlyone thread uses 1 Connection at a time. Having multiple threads share a single Commodity DataServerconnection is unsafe.

Installation RequirementsThe following are required for the Commodity DataServer .NET API:

● Microsoft .NET Framework 2.0 or greater - Download

Microsoft Visual Studio 2005 or greater is recommended for development purposes.

Configuration with Microsoft Visual StudioFollow these instructions if you are configuring with MS Visual Studio 2005:

1. Create a new project.2. Add a reference to the Commodity DataServer .NET API assembly (mimapi.dll).

http://customers.morningstarcommodity.com/download/java_comlim_mimapi/javadoc/index.html

mail://[email protected]

http://www.microsoft.com/downloads/details.aspx?FamilyID=0856EACB-4362-4B0D-8EDD-AAB15C5E04F5&displaylang=en

272


a. Click on Reference in the Solution Explorer.b. Click on Add Reference...c. Click on Browse.d. Locate mimapi.dll and click OK.

Using the .NET APIThe .NET API is similar to the Java API in functionality and naming conventions. The common characteristics make thedocumentation and samples from the Commodity DataServer Java API relevant and helpful to .NET developers. Pleasesee Chapter 6, “Java API ” for additional usage examples.

Sample C# ProgramsThe following examples demonstrate how to use the Commodity DataServer C# .NET API to perform commondatabase functions.

Adding RelationsRelations are added to the database using the AddRelation() function.

AddRelation Example

This example shows how to use the AddRelation() function:

private static void add(MimConnection mim, string[] database, string relation, string description, string parent, RelType type, string exchange, string timezone, MimTime startTrade, MimTime endTrade, float contractUnits, MimDate expDate, MimDate firstNoticeDate, string rolloverDay, string rolloverPolicy, RolloverDataType rolloverDataType, int tradingPattern) { Relation r;

switch (type.getVal()) { case RelType.CATEGORY_VAL: r = new RelationCategory(relation, parent); break;

case RelType.FUTURES_VAL: RelationFutures rf = new RelationFutures(relation, parent);


273

rf.setContractUnits(contractUnits); rf.setRollDay(rolloverDay); rf.setRolloverData(rolloverDataType); rf.setRollRule(rolloverPolicy); r = rf; break;

case RelType.FUTURES_CONTINUOUS_VAL: RelationFuturesContinuous rfc = new RelationFuturesContinuous( relation, parent);

rfc.setRollDay(rolloverDay); rfc.setRolloverData(rolloverDataType); rfc.setRollRule(rolloverPolicy);

r = rfc; break;

case RelType.FUTURES_CONTRACT_VAL: RelationFuturesContract rfct = new RelationFuturesContract(relation, parent);

rfct.setExpirationDate(expDate); rfct.setFirstNoticeDay(firstNoticeDate);

r = rfct; break;

default: r = new Relation(relation, parent); break; }

r.setDescription(description); r.setExchange(exchange); r.setTimeZone(timezone); r.setBeginTradingTime(startTrade); r.setEndTradingTime(endTrade); r.setTradingPattern(tradingPattern);

MimSchema ms = mim.getSchemaManager();

mim.databaseNarrow(database); mim.lockDb(); ms.addRelation(r); mim.unlock(); mim.databaseWiden(); }

Deleting Relations

The DeleteRelation() function is used to delete a relation from all currently open databases where it exists.

274


deleteRelation Example

This example shows how to delete a relation from the database using the DeleteRelation() function: public static void delete(MimConnection mim, string[] database, string relation) { mim.databaseNarrow(database); mim.lockDb();

// Delete Relation from the MIM's Schema MimSchema ms = mim.getSchemaManager(); ms.deleteRelation(relation); mim.unlock(); mim.databaseWiden(); }

Browsing the Relation HierarchyUse the FindRelation() function to browse the relation hierarchy in the Commodity DataServer database. Use theGetRelChildren() function to navigate the relation tree hierarchy.

FindRelations Example

This example shows how to use the FindRelations() function: private static void find(MimConnection mim, string searchPattern, bool searchByName, FilterType filterType, bool caseSensitive) { SearchSet ss = new SearchSet(searchPattern); ss.setByName(searchByName); ss.setFilterType(filterType); ss.setCaseSensitive(caseSensitive);

String[] rel_names = mim.getSchemaManager().findRelations(ss);

for (int i = 0; i < rel_names.Length; i++) { Console.Out.WriteLine(rel_names[i]); } }

GetRelChildren Example

This example shows how to use the GetRelChildren() function: public static void get(MimConnection mim, string relation) { RelationChildInfo[] rels = mim.getSchemaManager().getRelChildren(relation);

275

for (int i = 0; i < rels.Length; i++) { Console.Out.Write("Relation: " + rels[i].getChildName());

if(rels[i].getHasChildren()) { Console.Out.Write("\tChildren: yes"); } else { Console.Out.Write("\tChildren: no"); }

Console.Out.WriteLine("\tType: " + rels[i].getRelType().ToString()); } }

Working with Forward CurvesForward curves are sets of data which represent today’s market estimate of the price of a commodity at various pointsout into the future.

ForwardCurve Example

This example shows how to create a forward curve using the ForwardCurve() function: public static void build(MimConnection mim, string relation, MimDate startDate) { string query; double value; string value_str; string date_str;

if(startDate == null) { startDate = new MimDate(new GregorianCalendar()); }

ContractExpirDates[] expir_dates = mim.getSchemaManager().getExpirationDates(relation); QueryResult rslt = null;

// Loop over contracts for(int x = 0; x < expir_dates[0].getContractNames().Length; x++) { if(expir_dates[0].getExpirDates()[x].compareTo(startDate) >= 0) { expir_dates[0].getExpirDates()[x].setDateFormat("mdy"); if(startDate == null) { date_str = expir_dates[0].getContractNames()[x] + " " + "last_data_day";

276


} else { date_str = startDate.getMonth() + "/" + startDate.getDay() + "/" + startDate.getYear(); } query = "SHOW \n1: Close of " + expir_dates[0].getContractNames()[x] + "\nWHEN Date is " + date_str; // Get the Close for the expiration date rslt = mim.getDataManager().queryExecute(query); for (int i = 0; i < rslt.reports.Length; i++) { QueryReport report = rslt.reports[i];

if (report != null) { if (report.blocks != null) { for (int j = 0; j < report.blocks.Length; j++) { QueryReportBlock block = report.blocks[j]; if (block != null) { value = block.getValues()[0]; value_str = "NaN"; if (value != Double.NaN) { value_str = value.ToString("F"); } Console.Out.WriteLine( expir_dates[0].getContractNames()[x] + " (Expiration " + expir_dates[0].getExpirDates()[x] + ") = " + value_str); } } } } } } } }

Retrieving Corrections

Corrections for a given symbol may be retrieved using GetCorrections().

277

GetCorrections Example

This example shows how to retrieve corrections using the GetCorrections() function:

public static void get(MimConnection mim, string relation, string column, MimDate fromDate, MimDate toDate, MimTime fromTime, MimTime toTime, CorrectionType units, int limit) { CorrectionsInfo corrInfo = mim.getCorrectionsManager().getCorrections(relation, column, fromDate, toDate, fromTime, toTime, units, limit);

// get the results MimDateTime[] arrReportedDates = corrInfo.getreportedDates(); MimDateTime[] arrCorrectedDates = corrInfo.getcorrectedDates(); int numRecords = corrInfo.getNumRecords(); double[] arrValues = corrInfo.getvalues();

// Output results to console Console.Out.WriteLine("Records\n" + numRecords); Console.Out.WriteLine("ReportedDates\tCorrectedDates\tValues"); if (numRecords == 0) { Console.Out.WriteLine("No data"); } else { for (int i = 0; i < numRecords; i++) { Console.Out.Write( arrReportedDates[i].getDate().getMonth().ToString("00") + ":" + arrReportedDates[i].getDate().getDay().ToString("00") + ":" + arrReportedDates[i].getDate().getYear()); Console.Out.Write("\t"); Console.Out.Write( arrCorrectedDates[i].getDate().getMonth().ToString("00") + ":" + arrCorrectedDates[i].getDate().getDay().ToString("00") + ":" + arrCorrectedDates[i].getDate().getYear()); Console.Out.Write("\t"); Console.Out.WriteLine("" + arrValues[i]); } } }

278


Retrieving Data RecordsAccess data for a relation using the GetRecords() function. Use GetRecordsOption to access data for an optionsrelation.

GetRecords Example

This example shows how to use the GetRecords() function:

private static void get(MimConnection mim, string[] relations, string[] columns, MimDate fromDate, MimDate toDate, MimTime fromTime, MimTime toTime, int numUnits, MimUnits units, FillOption holiday, FillOption missData, SkipAllNaN skipNan, int limit, LimitMode limitMode) { GetRecordsParameters pars = new GetRecordsParameters(relations, columns); pars.setNumUnits(numUnits); pars.setUnits(units); pars.setFromDate(fromDate); pars.setToDate(toDate); pars.setFromTime(fromTime); pars.setToTime(toTime); pars.setMissData(missData); pars.setHolidays(holiday); pars.setSkip(skipNan); pars.setLimit(limit); pars.setLimitMode(limitMode);

// Get the Records from the MIM RecordSet rs = mim.getDataManager().getRecords(pars);

// Display Results double[] vals = rs.getValues(); MimDateTime[] dts = rs.getDateTimes(); int index = 0;

if (dts.Length == 0) Console.Out.WriteLine("There are NO records!"); else { Console.Out.Write("Date\t"); for (int k = 0; k < relations.Length; k++) { for (int j = 0; j < rs.getColumns().Length; j++) { Console.Out.Write("\t" + relations[k] + " - " + rs.getColumns()[j]); } } Console.Out.WriteLine();

for (int i = 0; i < dts.Length; i++) { MimDate d = dts[i].getDate();

279

MimTime t = dts[i].getTime(); Console.Out.Write(d + " " + t.getHour() + ":" + t.getMinute() + ":" + t.getSecond() + ":" + t.getMillisecond());

for (int k = 0; k < relations.Length; k++) { for (int j = 0; j < rs.getColumns().Length; j++) { Console.Out.Write("\t" + Math.Round(vals[index], 4)); index++; } } Console.Out.WriteLine(); } } }

When you extract two relation-columns that have different trading calendars using GetRecords, then theresulting matrix may contain NaNs. The reason for this is that the filling of NaNs for each relation-columnis handled independently (i.e., when one time series has data for a particular time stamp, but the othertime series does not).

GetRecordsOption Example

This example shows how to use the GetRecordsOption() function: public static void get(MimConnection mim, string[] relations, string[] columns, MimDate fromDate, MimDate toDate, MimTime fromTime, MimTime toTime, int numUnits, MimUnits units, FillOption holiday, FillOption missData, SkipAllNaN skipNan, int limit, LimitMode limitMode, MimDate fromExpDate, MimDate toExpDate, OptionType optionType, float fromStrike, float toStrike, string sep) {

GetRecordsParameters par = new GetRecordsParameters(relations, columns); par.setNumUnits(numUnits); par.setUnits(units); par.setFromDate(fromDate); par.setToDate(toDate); par.setFromTime(fromTime); par.setToTime(toTime); par.setMissData(missData); par.setHolidays(holiday); par.setSkip(skipNan); par.setLimit(limit); par.setLimitMode(limitMode); OptionArgs options = new OptionArgs(); options.from_exp_date = fromExpDate; options.to_exp_date = toExpDate; options.option_type = optionType; options.from_strike = fromStrike;

280


options.to_strike = toStrike; ByteRecordSet rs = mim.getDataManager().getRecordsOptionFloat(par, options); float[] vals = rs.getValuesAsFloats(); MimDateTime[] dts = rs.getDateTimes(); float[] strikes = rs.getStrikes(); MimDate[] expDates = rs.getExpDates(); OptionType[] optionTypes = rs.getOptionTypes();

MimDate d = null; MimTime t = null; int index = 0;

if (dts.Length == 0) { Console.Out.WriteLine("There are NO records!"); } else { Console.Out.Write("Date" + sep + "Time" + sep + "Exp Dates" + sep + "Option Type" + sep + "Strike");

for (int k = 0; k < relations.Length; k++) { for (int j = 0; j < columns.Length; j++) { Console.Out.Write(sep + relations[k] + " - " + columns[j]); } }

Console.Out.WriteLine();

for (int i = 0; i < dts.Length; i++) { d = dts[i].getDate(); t = dts[i].getTime(); Console.Out.Write(d + sep + t + sep + expDates[i] + sep + ((optionTypes[i] == OptionType.OPTION_CALL) ? "call" : "put") + sep + strikes[i]);

for (int k = 0; k < relations.Length; k++) { for (int j = 0; j < columns.Length; j++) { Console.Out.Write(sep + vals[index]); index++; } }

Console.Out.WriteLine(); } } }


281

Accessing the Attributes of Relation and Columns

Use the GetRelation() function to return all the attributes of a given relation. Use the GetRelationColumn()function to return information for specific relation columns. Use the GetRelColumns() function to return thecolumns for a given relation. The GetRollOverDates() function performs the continuous contract computationfor the relation specified. This function returns the dates that the contracts would have rolled over, as well as whichcontracts are active during that period.

GetRelation Example

This example shows how to use the GetRelation() function:

public static void get(MimConnection mim, string relation) { Relation rel = mim.getSchemaManager().getRelation(relation); RelType type = rel.getType();

Console.Out.WriteLine("Name: " + rel.getName()); Console.Out.WriteLine("Parent: " + rel.getParent()); Console.Out.WriteLine("Description: " + rel.getDescription()); Console.Out.WriteLine("Type: " + type); Console.Out.WriteLine("Trading pattern: " + TradingPattern.toString(rel.getTradingPattern())); Console.Out.WriteLine("Begin time: " + rel.getBeginTradingTime()); Console.Out.WriteLine("End time: " + rel.getEndTradingTime()); Console.Out.WriteLine("Exchange: " + rel.getExchange());

if (type == RelType.FUTURES) { Console.Out.WriteLine("*Futures specific*"); RelationFutures future = (RelationFutures) rel; Console.Out.WriteLine( "Rollover policy: " + future.getRollRule()); Console.Out.WriteLine("Rollover date: " + future.getRollDay()); Console.Out.WriteLine( "Contract units: " + future.getContractUnits()); } else if (type == RelType.FUTURES_CONTINUOUS) { Console.Out.WriteLine("*Futures continuous specific*"); RelationFuturesContinuous future = (RelationFuturesContinuous) rel; Console.Out.WriteLine( "Rollover policy: " + future.getRollRule()); Console.Out.WriteLine("Rollover date: " + future.getRollDay()); } else if (type == RelType.FUTURES_CONTRACT) { Console.Out.WriteLine("*Futures contract specific*"); RelationFuturesContract contract = (RelationFuturesContract) rel; Console.Out.WriteLine("Expiration date: " + contract.getExpirationDate());

282


Console.Out.WriteLine("First notice day: " + contract.getFirstNoticeDay()); } }

GetRelationColumn Example

This example shows how to use the GetRelationColumn() function:

public static void get(MimConnection mim, string relation, string column) { RelationColumn rc = mim.getSchemaManager().getRelationColumn(relation, column);

if (rc.GetType().Name.Equals("RelationColumnComposite")) { RelationColumnComposite relcol = (RelationColumnComposite)rc;

Console.Out.WriteLine( "Relation-column: " + relcol.getColumnName() + " of " + relcol.getRelationName());

for (int i = 0; i < relcol.getNumFields(); i++) { Console.Out.WriteLine("\nRelation-Column Number: " + i); Console.Out.WriteLine("Type: " + relcol.getType()); Console.Out.WriteLine("Aggregation: " + relcol.getAggregation()[i]); Console.Out.WriteLine("Object type: " + relcol.getObjectType()[i]); Console.Out.WriteLine("Constant: " + relcol.getConstant()[i]); Console.Out.WriteLine("Delta: " + relcol.getDelta()[i]); Console.Out.WriteLine( "Daily Multiplicity: " + relcol.getDailyMultiplicity()); Console.Out.WriteLine( "Intraday Multiplicity: " + relcol.getIntradayMultiplicity()); Console.Out.WriteLine( "Tick Multiplicity: " + relcol.getTickMultiplicity()); } } else if (rc.GetType().Name.Equals("RelationColumnSingle")) { RelationColumnSingle relcol = (RelationColumnSingle)rc;

Console.Out.WriteLine( "Relation-column: " + relcol.getColumnName() + " of " + relcol.getRelationName()); Console.Out.WriteLine("Type: " + relcol.getType()); Console.Out.WriteLine("Aggregation: " + relcol.getAggregation()); Console.Out.WriteLine("Object type: " + relcol.getObjectType()); Console.Out.WriteLine("Constant: " + relcol.getConstant()); Console.Out.WriteLine("Delta: " + relcol.getDelta()); Console.Out.WriteLine( "Daily Multiplicity: " + relcol.getDailyMultiplicity()); Console.Out.WriteLine(

283

"Intraday Multiplicity: " + relcol.getIntradayMultiplicity()); Console.Out.WriteLine( "Tick Multiplicity: " + relcol.getTickMultiplicity()); Console.Out.WriteLine("Formula:" + relcol.getFormula()); Console.Out.WriteLine("FormulaIndex:" + relcol.getFormulaIndex()); } }

GetRelColumns Example

This example shows how to use the GetRelColumns() function: public static void get(MimConnection mim, string relation) { String[] col_names = mim.getSchemaManager().getRelColumns(relation);

Console.Out.WriteLine("Columns for " + relation); for (int i = 0; i < col_names.Length; i++) { Console.Out.WriteLine(" " + col_names[i]); } }

GetRollOverDates Example

This example shows how to use the GetRollOverDates() function: public static void get(MimConnection mim, String relation, int multiple, MimUnits units, String rollOverDay, String rollOverPolicy) {

RollOverDates objRollOverDates = mim.getSchemaManager().getRollOverDates(relation, multiple, units, rollOverDay, rollOverPolicy);

// get results MimDate[] arrRollDates = objRollOverDates.getRollDates(); MimDate[] arrContracts = objRollOverDates.getContracts(); int numPeriods = objRollOverDates.getNumContracts(); int numContracts = objRollOverDates.getNumPeriods(); int numRollDates = arrRollDates.Length; int numContractDates = arrContracts.Length;

// Output results to console Console.Out.WriteLine("Contracts: " + numContracts); Console.Out.WriteLine("Periods: " + numPeriods); Console.Out.WriteLine("RollOverDates:"); if (numRollDates == 0) { Console.Out.WriteLine("No data"); } else { for (int i = 0; i < numRollDates; i++) { Console.Out.WriteLine( arrRollDates[i].getMonth() + "/" + arrRollDates[i].getDay() + "/" + arrRollDates[i].getYear());

284


} }

Console.Out.WriteLine("Contracts:"); if (numContractDates == 0) { Console.Out.WriteLine("No data"); } else { for (int i = 0; i < numContractDates; i++) { Console.Out.WriteLine( arrContracts[i].getMonth() + "/" + arrContracts[i].getDay() + "/" + arrContracts[i].getYear()); } } }

Loading DataUse the Putrecords() function to load data into the Commodity DataServer.

Putrecords Example

This example shows how to use the Putrecords() function: private static void put(MimConnection mim, string[] database, string[] relations, string[] columns, MimUnits units, MimDateTime[] dates, double[] values) { mim.databaseNarrow(database); mim.lockDb();

// Place the Data in to the MIM mim.getDataManager().putRecords(relations, columns, units, dates, values); mim.unlock(); mim.databaseWiden(); }

Executing QueriesThe QueryExecute() function is used to run queries and retrieve the results programmatically.

QueryExecute Example

This example shows how to use the QueryExecute() function: /// <summary> /// Run the query. /// </summary> /// <param name="mim">MIM connection</param> /// <param name="query">MIM query</param>

285

private static void query(MimConnection mim, string query) { // Execute queries using a MimData object. MimData data = mim.getDataManager();

// Execute the query. QueryResult rslt = data.queryExecute(query);

// Process the results. proccessQueryResult(rslt); }

/// <summary> /// Handle the query results. /// </summary> /// <param name="rslt">Result object</param> private static void proccessQueryResult(QueryResult rslt) { // Each block contains column headings, dates and values. if (rslt.reports == null || rslt.reports.Length == 0) { Console.Out.WriteLine("Empty results."); } else { // Query results may contain multiple reports for each LET // attribute. // Multiple reports are created for queries similar to // "LET sym=HO, CL, HU..." for (int i = 0; i < rslt.reports.Length; i++) { QueryReport report = rslt.reports[i]; processQueryReport(report); } } }

/// <summary> /// Process the query report. /// </summary> /// <param name="report">Query report</param> private static void processQueryReport(QueryReport report) { if (report.blocks != null && report.blocks.Length != 0) { // A report may contain multiple blocks for each SHOW statement. // Multiple block are created for queries such as // SHOW 1: Close of NG // SHOW 1: High of NG // WHEN date is within 1 month for (int i = 0; i < report.blocks.Length; i++) { QueryReportBlock block = report.blocks[i]; if (block != null) { processQueryReportBlock(block);

286


} } }

if (report.plReport != null) { if (report.plReport.getTradeRecords() != null) { processTradeRecords(report.plReport.getTradeRecords()); }

if (report.plReport.getStatRecords() != null) { processStatRecords(report.plReport.getStatRecords()); }

if (report.plReport.getEquityRecords() != null) { processEquityRecords(report.plReport.getEquityRecords()); } } }

/// <summary> /// Process the query report block. /// </summary> /// <param name="block">Query report block.</param> private static void processQueryReportBlock(QueryReportBlock block) { string[] columnHeadings = block.getColumnHeadings(); string[] attributeHeadings = block.getAttributeHeadings(); int[] columnHeadingIndices = block.getColumnHeadingIndices();

Console.Out.Write("Date/Time"); for (int i = 0; i < columnHeadingIndices.Length; i++) { int idx = columnHeadingIndices[i]; String colHeading = columnHeadings[idx]; Console.Out.Write("\t" + colHeading); }

Console.Out.WriteLine(); printDatesAndValues(block); printSummary(block); }

/// <summary> /// Print the dates and values. /// </summary> /// <param name="block">Query report block.</param> private static void printDatesAndValues(QueryReportBlock block) { // Print out the dates and values. MimDate[] dates = block.getRowDates(); MimTime[] times = block.getRowTimes(); int numRows = block.getNumRows(); int numCols = block.getNumCols();

287

for (int row = 0; row < numRows; row++) { Console.Out.Write(dates[row]); if(times != null) { Console.Out.Write(" " + times[row]); }

for (int col = 0; col < numCols; col++) { // We can get a value for a specified row and column. double val = block.getValue(row, col); Console.Out.Write("\t" + Math.Round(val, 4)); } Console.Out.WriteLine(); } }

/// <summary> /// Print summary statistics. /// </summary> /// <param name="block">Query report block</param> private static void printSummary(QueryReportBlock block) { // Display the statistics Console.Out.WriteLine(); QuerySummary summary = block.getSummaryReport(); if (summary.summaryBlocks != null) { Console.Out.Write("Average: \t"); foreach (QuerySummaryBlock b in summary.summaryBlocks) { Console.Out.Write("\t" + Math.Round(b.average, 4)); }

Console.Out.Write("\nAverage Negative: "); foreach (QuerySummaryBlock b in summary.summaryBlocks) { Console.Out.Write("\t" + Math.Round(b.averageNegative, 4)); }

Console.Out.Write("\nAverage Positive: "); foreach (QuerySummaryBlock b in summary.summaryBlocks) { Console.Out.Write("\t" + Math.Round(b.averagePositive, 4)); }

Console.Out.Write("\nHighest: \t"); foreach (QuerySummaryBlock b in summary.summaryBlocks) { Console.Out.Write("\t" + Math.Round(b.highest, 4)); }

Console.Out.Write("\nLowest: \t"); foreach (QuerySummaryBlock b in summary.summaryBlocks) { Console.Out.Write("\t" + Math.Round(b.lowest, 4));

288


}

Console.Out.Write("\nPercent Negative: "); foreach (QuerySummaryBlock b in summary.summaryBlocks) { Console.Out.Write("\t" + Math.Round(b.percentNegative, 4)); }

Console.Out.Write("\nPercent Positive: "); foreach (QuerySummaryBlock b in summary.summaryBlocks) { Console.Out.Write("\t" + Math.Round(b.percentPositive, 4)); }

Console.Out.Write("\nStandard Deviation: "); foreach (QuerySummaryBlock b in summary.summaryBlocks) { Console.Out.Write("\t" + Math.Round(b.standardDeviation, 4)); }

Console.Out.Write("\nSum: \t\t"); foreach (QuerySummaryBlock b in summary.summaryBlocks) { Console.Out.Write("\t" + Math.Round(b.sum, 4)); }

Console.Out.Write("\nVariance: \t"); foreach (QuerySummaryBlock b in summary.summaryBlocks) { Console.Out.Write("\t" + Math.Round(b.variance, 4)); }

Console.Out.Write("\nZ-Stat: \t"); foreach (QuerySummaryBlock b in summary.summaryBlocks) { Console.Out.Write("\t" + Math.Round(b.zstat, 4)); } Console.Out.WriteLine(); } }

/// <summary> /// Print the P&L trade records. /// </summary> /// <param name="queryPLTradeRecord"></param> private static void processTradeRecords(QueryPLTradeRecord[] queryPLTradeRecord) { for (int j = 0; j < queryPLTradeRecord.Length; j++) { Console.Out.WriteLine( "Cum Profit: " + queryPLTradeRecord[j].cumProfit); Console.Out.WriteLine( "Entry Price: " + queryPLTradeRecord[j].entryPrice);

289

Console.Out.WriteLine( "Entry Reason Label: " + queryPLTradeRecord[j].entryReasonLabel); Console.Out.WriteLine( "Exit Price: " + queryPLTradeRecord[j].exitPrice); Console.Out.WriteLine( "Exit Reason Label: " + queryPLTradeRecord[j].exitReasonLabel); Console.Out.WriteLine( "Move: " + queryPLTradeRecord[j].move); Console.Out.WriteLine( "Num Contracts: " + queryPLTradeRecord[j].numContracts); Console.Out.WriteLine( "Percent Move: " + queryPLTradeRecord[j].percentMove); Console.Out.WriteLine( "Profit: " + queryPLTradeRecord[j].profit); Console.Out.WriteLine( "Entry Date: " + queryPLTradeRecord[j].entryDate); Console.Out.WriteLine( "Entry Reason: " + queryPLTradeRecord[j].entryReason.ToString()); Console.Out.WriteLine( "Entry Time: " + queryPLTradeRecord[j].entryTime); Console.Out.WriteLine( "Exit Date: " + queryPLTradeRecord[j].exitDate); Console.Out.WriteLine( "Exit Reason: " + queryPLTradeRecord[j].exitReason); Console.Out.WriteLine( "Exit Time: " + queryPLTradeRecord[j].exitTime); Console.Out.WriteLine( "Type: " + queryPLTradeRecord[j].type.ToString()); } }

/// <summary> /// Process the P&L statistic records. /// </summary> /// <param name="queryPLStatRecord">Query P&L statistic records</param> private static void processStatRecords(QueryPLStatRecord[] queryPLStatRecord) { for (int j = 0; j < queryPLStatRecord.Length; j++) { Console.Out.WriteLine( queryPLStatRecord[j].combinedValue + " "

290


+ queryPLStatRecord[j].label + " " + queryPLStatRecord[j].longValue + " " + queryPLStatRecord[j].shortValue + " " + queryPLStatRecord[j].combinedDate + " " + queryPLStatRecord[j].combinedTime + " " + queryPLStatRecord[j].longDate + " " + queryPLStatRecord[j].longTime + " " + queryPLStatRecord[j].shortDate + " " + queryPLStatRecord[j].shortTime); } }

/// <summary> /// Process the P&L equity records. /// </summary> /// <param name="queryPLEquityRecord">P&L equity records.</param> private static void processEquityRecords(QueryPLEquityRecord[] queryPLEquityRecord) { for (int j = 0; j < queryPLEquityRecord.Length; j++) { Console.Out.WriteLine( "Close: " + queryPLEquityRecord[j].close); Console.Out.WriteLine( "Combined: " + queryPLEquityRecord[j].combined); Console.Out.WriteLine( "Open: " + queryPLEquityRecord[j].open); Console.Out.WriteLine( "Date: " + queryPLEquityRecord[j].date); Console.Out.WriteLine( "Time: " + queryPLEquityRecord[j].time); } }


291

Error Messages

ONC/RPC Error MessageIf you receive an ONC/RPC error message from the API, please contact Client Support ([email protected]). The error is due to either:

● a compatibility problem between the API version and the Commodity DataServer● or the Commodity DataServer abnormally disconnected

http://www.morningstarcommodity.com/contact/customer-support.php



292


COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 8: WebServices API

293

CHAPTER8WebServices API

OverviewThe Morningstar Commodity WebServices API is RESTful, making server resources available to clients via HTTP/HTTPSURI interfaces. The REST API is Stateless, meaning each request from any client must contain all of the informationnecessary to service the request. HTTP Basic Access Authentication is required for most requests.

Listed below are the major benefits of using the WebServices API:

● Ubiquitous access: - Standard HTTP method call-outs (get, post, put, delete) available on a rich set of applicationdevelopment platforms to load data and retrieve information.

● Ease of use: - No Morningstar software to install, and no Morningstar specific API methods to learn. You should befamiliar with the Web Service facilities of your development environment and programming language and know theURI of the resource you need to access and the formats (XML Schemas) of the data exchanges.

● Standard XML formats: - The server expects data from the client to be formatted in XML (on HTTP post or HTTPput) and returns XML formatted responses to every client request. You must know the formats (XML schemas) ofall data exchanges for the resource you need to use.

Much of the Morningstar Commodity client software, including Commodity Add-in and Commodity charts, have beenrewritten to use the WebServices API in order to take advantage of the benefits outlined above.

For details of the Commodity Data WebServices, including server installation instructions and XML Schema formats,see the Commodity Data WebServices API User's Guide.

This chapter features usage examples with the Java programming language. For any service available in the API thesame 2-step procedure is followed:

● Send an HTTP request to the server.

● Parse the server response to a needed format.

When applicable the server response will contain a job ID and Status attributes. if the job status returned indicatesthat the task has not completed, the client must poll with the job id until the actual results are received.

Meta Data Retrieval ServiceThe WebServices API provides the ability to browse the hierarchy and get information related to CommodityDataServer relations.


294


To retrieve relation information, send an HTTP get request to the web server URI:

$Base_URI/rs/api/schema/relations/{relNames}?colName={colName}& showChildren={flag}&defaultColumns={cNames}&precision={nDigits}&desc={flag}& aliasTarget={flag}&path={flag}&showColumns={flag}&cDesc={flag}&uom={flag}& dateRange={flag}&dataPreview={flag}&shorthand={flag}

Where $Base_URI is http[s]://webservices-host[:port] of the web server. All query parameters are optional.

For example the following URI retrieves relation "NG" and its children relations:

http[s]://webservices-host:[port]/rs/api/schema/relations/NG?showChildren=true

Assuming there is a web server running and the resource exists, the server would return an XML response similar tothe one shown below:

<RelInfos> <RelInfo type="FUTURES" name="NG" defaultColumn="Close" hasChildren="1"> <children> <RelInfo type="FUTURES_CONTINUOUS" name="NG_02" hasChildren="0"/> <RelInfo type="FUTURES_CONTINUOUS" name="NG_03" hasChildren="0"/> <RelInfo type="FUTURES_CONTINUOUS" name="NG_04" hasChildren="0"/> ... <RelInfo type="FUTURES_CONTRACT" name="NG_2020V" hasChildren="0"/> <RelInfo type="FUTURES_CONTRACT" name="NG_2020X" hasChildren="0"/> <RelInfo type="FUTURES_CONTRACT" name="NG_2020Z" hasChildren="0"/> </children> </RelInfo> </RelInfos>

Consult the WebServices User's guide for a complete description of the URI and the server response XML Schema.

Java ExampleThis example sends an HTTP get request using the JAX-RS Jersey implementation, and parses the server XMLresponse using the Java Streaming API for XML (StaX).

Get Web Resource

Provide needed authentication to the server and get a web handler resource.

WebResource webRsc; Client client = Client.create(); client.addFilter(new HTTPBasicAuthFilter("USERNAME", "PASSWORD")); webRsc = client.resource("http://ausqadev01:9090");

295

Send the Request and Get the Server XML Response

Send the HTTP get request specifying URL path and media type. return the server response as an XML string.

ClientResponse response = webRsc.path("rs/api/schema/relations").path("NG"). queryParams("showChildren", "true"). accept(MediaType.TEXT_XML).type(MediaType.TEXT_XML). get(ClientResponse.class); // check for HTTP erros if (200 != response.getStatus()) throw new IllegalStateException("Error while polling for results [" + resp + "]"); return response.getEntity(String.class);

Parse the XML response

Parse the XML into a desired format. In this Example we use Java StaX to output to console. Use the best tool in yourdevelopment environment for your specific purpose.

XMLInputFactory inputFactory = XMLInputFactory.newInstance(); InputStream in = new ByteArrayInputStream(response.getBytes("UTF-8")); XMLEventReader eventReader = inputFactory.createXMLEventReader(in); while (eventReader.hasNext()) { XMLEvent event = eventReader.nextEvent(); if (event.isStartElement()) { StartElement startElement = event.asStartElement(); //If this is a RelInfo element print its attributes if (startElement.getName().toString().equals("RelInfo")) { Iterator<Attribute> attributes = startElement.getAttributes(); System.out.println("======================================================="); while (attributes.hasNext()) { Attribute attribute = attributes.next(); System.out.println(attribute.getName().toString() + ": " + attribute.getValue()); } } // else if this is a children element call a helper method to parse it. else if (startElement.getName().toString().equals("children")) parseRelInfoChildren(eventReader); } }

Source Code listing

Contact customer support for the complete source code for this and other WebService usage examples.

296


There are different methods/tools for parsing XML documents in different languages and platforms. Youshould use the most appropriate method for your specific task in your development environment.

Data Retrieval ServicesCurrently there are two services for requesting data from WebServices

● Query Execution

● Records

All of them reference the same URIs

● $Base_URI/rs/api/datarequests

● $Base_URI/rs/api/datarequests/{jobId}

Where $Base_URI is http[s]://webservices-host[:port] of the web server

Each service takes a different request payload but all return the same type of response.

Consult the WebServices User's guide for a complete description of the URIs, payload formats and the server responseXML Schema.

Java ExampleThis example sends a Query Execution request using the JAX-RS Jersey implementation, and processes the serverXML response using the JAXB (Java Architecture for XML Binding) API.

Get Web ResourceProvide needed authentication to the server and get a web handler resource.


Send the Request and Get the Server XML Responsefirst format the query into a Query Excution request payload

297

String queryText = "SHOW 1: Close of NG WHEN Date is within 1 year"; String queryPayload = "<DataRequest><Query><Text>" + queryText + "</Text></Query></DataRequest>";

Then send the HTTP post request specifying the URL path and media type.

ClientResponse response = webRsc.path("rs/api/datarequests"). accept(MediaType.TEXT_XML).type(MediaType.TEXT_XML). post(ClientResponse.class, query); // Check for HTTP errors if (200 != response.getStatus()) throw new IllegalStateException("Initial request failed with response [" + response + "]");

// DataRequestResponse is a class auto-generated from the Data Response XSD file. DataRequestResponse respPayload = response.getEntity(DataRequestResponse.class);

And check the response status attribute of the returned XML. If job is not complete poll with HTTP get.

while (respPayload.getStatus() == 200) { try { Thread.sleep(250); }catch (InterruptedException e) { System.out.println(e.getMessage()); } long jobId = respPayload.getId(); response = webRsc.path(DATA_REQ_RSC_PATH + "/" + jobId).get(ClientResponse.class); if (HTTP_CODE_OK != response.getStatus()) { throw new IllegalStateException("Error while polling for results [" + response + "]"); } respPayload = response.getEntity(DataRequestResponse.class); }

at this point, we have received a valid response with good pay-load data. return respPayload;

Print the response to console

The Report and Report classes shown here were generated by the java xjc utility from the Data RequestResponse XSD file.

298


//Get report objects List <Report< reports = respPayload.getReports();

//print user input query System.out.println("User Query: \n" + query); System.out.println(); //process each report for (Report report: reports) { //print report title if (report.getTitle().length() >0) System.out.println("\nReport title: " + report.getTitle());

//process each reportBlock in report List <ReportBlock> reportBlocks = report.getReportBlocks(); int block = 1; for (ReportBlock reportBlock : reportBlocks) { String [] headings = (String[]) reportBlock.getColumnHeadings().toArray(new String[]{}); String [] dates = (String[]) reportBlock.getRowDates().toArray(new String[]{}); Double [] values = (Double[]) reportBlock.getValues().toArray(new Double[]{}); //print reportBlock header (column names) System.out.print("Block: " + block++); for (int i=0; i < headings.length; i++) System.out.print("\t\t" + headings[i]); System.out.println(); //print dates & values for (int k = 0; k < dates.length; k++) { System.out.print(dates[k]); for (int i=0; i < headings.length; i++) { String value_str = "NaN"; int idx = reportBlock.getNumCols() * k + i; if (!Double.isNaN(values[idx])) value_str = formatter.format(values[idx]); System.out.print("\t\t" + value_str); } System.out.println(); } System.out.println("Number of Dates: " + reportBlock.getNumRows() + "\n"); } }

Source Code listing

Contact customer support for the complete source code for this and other WebService usage examples.

299


Data Upload ServiceThe Data Upload Webservice allows users to submit arbitrarily formatted data for loading into a CommodityDataServer.

ParsersThe Upload services uses "parsers" to process data submitted to the service. A parser is responsible for converting anincoming data stream into a series of (relation, column, transaction date, value) records that can then be loaded intothe Commodity DataServer. This document describes how to use the Upload service with its built-in "DefaultParser"."DefaultParser-7day" and "Basic Futures" are two other built-in data upload parsers distributed with the webservice.

DefaultParser Data FormatThe Upload service's DefaultParser accepts row and column data encoded in XML. Each row contains 4 columns thatdescribe the data to load: symbol (relation), column, transaction date, and the value. An optional fifth column maycontain a text description of the symbol.

Here is a sample XML format required by the DefaultParser:

<?xml version="1.0" encoding="utf-8"?> <ExcelData> <Rows> <Row num="1"> <Cols> <Col num="1">TopRelation:Category:NEW_RELATION_01</Col> <Col num="2">TopColumn:Price:Close</Col> <Col num="3">40179</Col> <Col num="4">100</Col> <Col num="5">Description of NEW_RELATION_01</Col> </Cols> </Row> <Row num="2"> <Cols> <Col num="1">TopRelation:Category:NEW_RELATION_02</Col> <Col num="2">TopColumn:Price:Close</Col> <Col num="3">40180</Col> <Col num="4">200</Col> <Col num="5">Description of NEW_RELATION_02</Col> </Cols> </Row> </Rows> </ExcelData>

Two things to notice:

300


● Column number 3 shows the dates in Microsoft Excel's numeric date format. Excel (by default) uses the 1900date system. This simply means that the date 1 Jan 1900 has a true numeric value of 1, 2 Jan 1900 has a valueof 2 etc. These values are called "serial values" in Excel and it is these serial values that allows us to use dates incalculations. Times are very similar BUT Excels sees Times as decimal fractions, with 1 being the time 24:00 or00:00. 18:00 has true value of 0.75 because it is three quarters of 24 hours, or the whole number 1. For examplethe date and time 3/July/2002 3:00:00PM has a true value of 37440.625 with .625 representing the time and the37440 being the serial value for 3/July/2002. See Microsoft Excel documentation for more information on Exceldates.

● Relations which do not exist in the Dataserver will be added; Columns must already exist in the DataServer!

Submitting DataThe Upload service accepts data via HTTP POST at this URL: http[s]://webservices-host:[port]/rs/upload

The service requires "username" and "parsername" parameters to be passed in the POST in addition to the data itself.username is the authenticated WebServices user. Example: http[s]://webservice-host:[port]/rs/upload?username=USERNAME&parsername=DefaultParser

Java ExampleThis example sends a data upload request using the JAX-RS Jersey implementation, and parses the server XMLresponse using the Java Streaming API for XML (StaX).

Get Web ResourceProvide needed authentication to the server and get a web handler resource.


Send the Request and Get the Server XML ResponseSend the HTTP post request specifying URL path and media type.

ClientResponse response = webRsc.path("rs/upload/"). queryParam("username", "WSUSER").queryParam("parsername", "DefaultParser"). accept(MediaType.TEXT_XML).type(MediaType.TEXT_XML). post(ClientResponse.class, "UploadXMLFile.xml"); if (HTTP_CODE_OK != response.getStatus())

301

throw new IllegalStateException("Initial request failed with response [" + response + "]");

And check the response intStatus attribute of the returned XML. If job is not complete poll with HTTP get.

responseString = response.getEntity(String.class); while (getDataRequestResponseStatus(responseString) < 300) { // When the status indicates waiting-on-result, retry the request with ID. Thread.sleep(250); response = webRsc.path("rs/upload/jobreport/" + jobID).get(ClientResponse.class); if (HTTP_CODE_OK != response.getStatus()) throw new IllegalStateException("Error while polling for results [" + response + "]"); responseString = response.getEntity(String.class); }

return the server response as an XML string. return responseString;

Parse the XML responseParse the XML into a desired format. In this Example we use Java StaX to output to console. Use the best tool in yourdevelopment environment for your specific purpose.

XMLInputFactory inputFactory = XMLInputFactory.newInstance(); InputStream in = new ByteArrayInputStream(response.getBytes("UTF-8")); XMLEventReader eventReader = inputFactory.createXMLEventReader(in); while (eventReader.hasNext()) { XMLEvent event = eventReader.nextEvent(); if (event.isStartElement()) { StartElement startElement = event.asStartElement(); if (startElement.asStartElement().getName().toString().endsWith("message")) { event = eventReader.nextEvent(); System.out.println("\n" + event.asCharacters().getData()); } } }

Source Code listingContact customer support for the complete source code for this and other WebService usage examples.

302


COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 9: VBA API

303

CHAPTER9

VBA API

Installing the Commodity DataServer VBA API

Commodity DataServer VBA API Installation InstructionsFollow these steps to install the Commodity DataServer Visual Basic for Applications (VBA) API:

1. Contact customer support to obtain the Commodity DataServer VBA API software.

2. Unpack the archived files in to C:\Morningstar\VBAAPI.

3. Add C:\Morningstar\VBAAPI to your system path.

4. In Excel go to Tools>Add-Ins, select Browse, go to C:\Morningstar\VBAAPI and select LIM_VBA_api.

5. In the Excel Visual Basic editor go to Tools>References and select LIM_VBA_api.

Introducing VBA API

VBA API Object TypesVBA API implements eight classes of objects, representing the functionality of the Commodity DataServer C languageAPI:

● XMIMServer - An application typically creates a single instance of this object. It handles connecting anddisconnecting to the Commodity DataServer, and management of global server options.

● XMIMRecords and XMIMRecordsDouble - An object of this type is used to retrieve and store historical time-series data for one or more Commodity DataServer relations. Support for the computation of continuous contractsis provided.

304


● XMIMRelation - This class exposes the Commodity DataServer relation hierarchy, supporting the constructionof browsers and other tools that access information about relations. XMIMRelation includes methods to add,modify, and delete a relation, as well as to fetch the columns, children, and other information about a relation.

● XMIMColumn - This is similar to XMIMRelation, but it is used to access the Commodity DataServer columnhierarchy.

● XMIMRelCol - An application can add, delete, modify, or retrieve a Commodity DataServer relation column usingthis class.

● XMIMShowWhen and XMIMShowWhenDouble - Simple Commodity DataServer queries can be executed, and theresults accessed, using instances of this class.

● XMIMExecution and XMIMExecutionDouble - Commodity DataServer queries can be executed, and theresults accessed, using instances of this class.

● XMIMSchema - This class provides access to the Commodity DataServer schema. Methods are included to retrieveall or portions of the relation and column hierarchies. Additional methods provide print and search capabilities.

● XMIMTable - This class supports use of the Commodity DataServer table facility, hence providing for the storageand retrieval of non-time series data organized as tables. An application can create, delete, or rename a table andcan add, delete or access entries from a table using this class.

● XMIMCurrentTick - This class includes methods to utilize the Commodity DataServer current tick facility thathandles high frequency updating of tick data. There is also a set of utility routines for parsing and unparsingCommodity DataServer constants, performing date conversions and NaN comparisons. (The parsing and unparsingroutines are useful for building GUIs that manipulate enumerated types like XmimUnits.)

Programming with VBA API

Overview

Referencing the VBA API libraryFor Visual Basic for Applications in Excel, select References in the Tools menu. (In Excel, you must activate a macromodule to make the References selection visible.) This will pop up a dialog box with a list of all the OLE servers in thesystem. Assuming you correctly installed the VBA API you should see an entry like VBA api. Click on the check box toselect the library, and close the dialog box.

Once you have established a reference, you should be able to use the Object Browser to examine the classes andmethods exported by the VBA API.


305

Creating and releasing objects You can create a VBA API object using the predefined New_ functions for each object type. To dispose of an object,set it to Nothing. For example, to create a new XMIMServer object, you could use the following declaration:Set srvr = New_XMIMServer'Use the server object'Set srvr = Nothing

Several VBA API classes require additional initialization before they can be used. In particular, most classes must beassociated with an XMIMServer object so they can transmit and receive data.

Detecting and Handling ErrorsWhere there is an error, VBA API objects throw exceptions that include descriptive strings. All the VBA API objects arecapable of throwing exceptions, so you should protect your code with appropriate handlers. Usually, it is sufficient toplace an exception handler at the top level of each event procedure; naturally, it will depend on the application.

Errors can be generated by several components in the VBA API chain, including the Commodity DataServer server, theCommodity Query DLL, and the VBA API library. In general, you don't need to worry about the origin of the error, andthe VBA API components attempt to provide meaningful descriptions.

The OLE layer of Windows can also generate exceptions. These are unusual, but they can be mysterious when theyoccur. For example, if you encounter error 429 (OLE library couldn't create object) then it probablyindicates an installation problem.

Dates

The range of valid dates for VBA API depends largely on the date limitations of the Commodity DataServer. Currently,you should be safe using dates after 1752. VBA API can represent dates earlier than this, but the CommodityDataServer will not perform internal calculations correctly with dates going back more than a couple of centuries.

Note that Visual Basic uses a peculiar representation for dates. The representation is capable of representing datesfrom roughly 100 AD to several millennia in the future. However, a time without a date is indistinguishable from atime associated with December 30, 1899. If you work with data from around the turn of the century, you might noticeunexpected formatting of dates or times in the last few days of 1899. Visual Basic also seems to ignore the adoptionof the Gregorian calendar in 1582.

Moreover, the Visual Basic functions for converting from float values to dates are badly defined. There is a “hole” inthe middle, whereby values from 0.0 to 24.0 are taken as being in “hour.minute” format; negative values and valuesgreater than or equal to 24.0, are assumed to be in “day.fraction” format. The net result is that 24 days from December31, 1899 through late January 1900 are inaccessible.

NaNs

In the Commodity DataServer, missing data and holidays are denoted using IEEE NaN floating-point values. This isusually not a problem for C or C++, but we encountered difficulties with Visual Basic. Visual Basic often generatesoverflow exceptions when it encounters an IEEE NaN.

306


VBA API uses the value 1.0e38 for NaNs. If you wish, you can change this using the MissDataNaN and HolidayNaNproperties of the XMIMServer class.

You are also encouraged to use the utilities IsNaN, IsMissDataNaN, and IsHolidayNaN (in the XMIMUtils class)instead of making explicit comparisons.

VBA API is Zero-Based Unless otherwise indicated, VBA API uses zero-based indices to access array information in its objects. For instance,r.GetVal(0, 0, 0) retrieves the first column of the first relation for the first record of a time series query, where ris an XMIMRecords instance.

ConstantsBecause Visual Basic lacks an enumerated type, VBA API uses properties of the XMIMUtils class to represent thevarious Commodity DataServer constant values. See “XMIMUtils (Constants)” for a complete list of properties.

XMIMServer

OverviewTo connect to the Commodity DataServer using VBA API, you use an XMIMServer object. XMIMServer handlesnetwork communication with the Commodity DataServer, and it manages global Commodity DataServer parameterslike NaN values and default execution units. It also has methods for locking and unlocking the Commodity DataServerdatabase, which you must use to perform updates.

You will need to create a XMIMServer object before you use any other Commodity DataServer object; theconstruction routines for the other object types require a XMIMServer handle. Other objects share the connectionprovided by a server object.

To create an XMIMServer instance, use must call New_XMIMServer. To dispose of the object, allow it to go out ofscope, or explicitly set it to Nothing.

Before you do anything with an XMIMServer object, you must establish a connection to the Commodity DataServer.First, set the host name with the Host property, and set the desired server number with the ServerNum property. Youcan later retrieve the host and server number settings via the same properties. Finally, invoke the Connect method tomake the connection.

To close a connection, you are encouraged to use the Disconnect procedure, though you can just destroy the serverobject instead.

You can access and modify the value used for missing data via the MissDataNaN property. For data that is missingdue to a holiday, use the HolidayNaN property.


307

You can retrieve and set the default execution units for Commodity DataServer queries using Units and NUnits.

The various parameter settings do not persist across connection; they are restored to their defaults each time aconnection is ended. If another Commodity DataServer object connects to the Commodity DataServer, it will use thedefaults instead of the values used by any previous Commodity DataServer objects.

Database locking is controlled using the Locked property. Setting the value to True locks the database, Falseunlocks it, and reading it returns the current state.

The client handle used by the Commodity DataServer can be retrieved using the GetHandle property.

XMIMServer Reference____________________________________________________________________

Host

Property Let Host(s as String) Property Get Host() as String

Description:

The Host property is the name of the host computer on which the Commodity DataServer server is running. Set thisproperty before connecting. It is an error to set the property if the object is currently connected to a host.

This property must be set prior to connection.

_________________________________________________________________________________

ServerNum

Property Let ServerNum(ByVal x as Long) Property Get ServerNum() as Long

Description:

The server number is an integer that uniquely identifies the particular server to be used. Set this property beforeconnecting, to identify the desired server. It is an error to set the property if currently connected to a host.

If this property is not specified prior to connection, then a default server number of 0 will be used.

308


____________________________________________________________________

Connect

Sub Connect()

Description:

Attempts to connect to the Commodity DataServer running on the machine specified via the Host property with theServerNum property.

You must connect before performing any operations like locking the database or setting global parameter values, andbefore passing the XMIMServer object to the initialization function of another VBA API object type.

_________________________________________________________________________________

NarrowDatabases

Sub NarrowDatabases svr.NDatabases=n svr.NthDatabase(0)=”/xmim/test/data.example/xmim.mim” svr.NarrowDatabases

Description:

NarrowDatabases defines the view of the databases that will be accessed for reading and writing to the server. Usethe NarrowDatabases command after the Connect command.

svr.NDatabases=n where n specifies how many databases will be accessed.

svr.NthDatabases(#)=pathname where pathname is the path to the database being accessed. Set for each databaseyou are going to narrow to. When narrowing to multiple databases, set # to 0 for the first database and 1 for thesecond database, etc.

svr.NarrowDatabases is the command to narrow the database.

_________________________________________________________________________________

WidenDatabases

Sub WidenDatabases()

Description:

Resets the view to the current pool of all open databases


309

_________________________________________________________________________________

Disconnect

Sub Disconnect()

Description:

Terminates the connection with the Commodity DataServer. After disconnecting, you may reconnect to the same oranother Commodity DataServer if desired.

_________________________________________________________________________________

MissDataNaN

Property Let MissDataNaN(ByVal x as Single) Property Get MissDataNaN() as Single

Description:

Use this property to override or fetch the value used to designate missing data in Commodity DataServer query results.

_________________________________________________________________________________

HolidayNaN

Property Let HolidayNaN(ByVal x as Single) Property Get HolidayNaN() as Single

Description:

This property overrides or retrieves the value used to signify data that is missing because of a trading holiday.

____________________________________________________________________

Nunits

Property Let NUnits(ByVal n as Long) Property Get NUnits() as Long

Description:

Sets or retrieves the number of units to use for the Commodity DataServer query execution. This function is normallyused in conjunction with the Units property, which controls the granularity of a unit.

310


_________________________________________________________________________________

Units

Property Let Units(ByVal u as Long) Property Get Units() as Long

Description:

Sets or retrieves the granularity of execution units for Commodity DataServer queries. This function is normally usedin conjunction with the NUnits property, which controls the number of execution units. The argument, u should be amember of the Units family of XMIMUtils constants.

____________________________________________________________________

Locked

Property Let Locked(ByVal b as Boolean) Property Get Locked() as Boolean

Description:

When set to True, attempts to acquire a lock on the Commodity DataServer database. When set to False, unlocksthe database. A successful lock is required before updating the database.

When read, returns the current state of the database lock.

_________________________________________________________________________________

Handle

Property Get Handle() as Long

Description:

Retrieves the client handle used by the Commodity DataServer server to uniquely identify the server connection.

_________________________________________________________________________________

ClientType

Property Let ClientType(x as String)Property Get ClientType() as String

Description:

Used in conjunction with the Commodity DataServer logging facility. The ClientType property allows a client toregister and provide an identifying string, type, which will be used by the Commodity DataServer logging facility.This call must be made prior to connecting to a server (Connect) so that the logging of the connect call will have theidentifying client type information.


311

____________________________________________________________________

NDatabases

Property Let NDatabases(ByVal x as Long) Property Get NDatabases() as Long

Description:

Sets and retrieves the number of databases to be accessed from the pool of databases.

_________________________________________________________________________________

NthDatabase

Property Let NthDatabase(ByVal idx as Long, Name as String) Property Get NthDatabase(ByVal idx As Long) as String

Description:

Used in conjunction with NarrowDatabases to specify the list of databases that will be available as the current viewof databases. The databases specified must be a subset of the current open database pool.

svr.NthDatabases(#)=pathname where pathname is the path to the database being accessed. Set for eachdatabase you are going to narrow to. When narrowing to multiple databases, set # to 0 for the first database and 1for the second database, etc.

The number of databases must be initialized before using this property.

_________________________________________________________________________________

Database

Property Let Database(x as String) Property Get Database() as String

Description:

Sets or retrieves the name of the file containing the Commodity DataServer schema or provides the specification foran external database.

_________________________________________________________________________________

ServerOwner

Property Get ServerOwner() as String

Description:

Returns the master server’s owner name. Call GetInfo before using this property.

312


_________________________________________________________________________________

ServerPid Property Get ServerPid() as Long

Description:

Returns the master server’s process id. Call GetInfo before using this property.

____________________________________________________________________

NewDatabase Sub NewDatabase()

Description:

NewDatabase creates a new database, opens it and add it to the client's current database pool. If the specifieddatabase has already been created then an error will be returned. The database file must be specified using theDatabase property before creating the new database.

____________________________________________________________________

OpenDatabase Sub OpenDatabase()

Description:

OpenDatabase is used to open a database and add it to the client's current database pool. The database to beopened may be any existing database, that is, it does not have to exist in the server .xmimrc file. If the specifieddatabase has already been opened by another client, then a pointer to the opened database will be returned. If thisis the first client to open the database, it will be physically opened. The database file must be specified using theDatabase property before opening the database.

____________________________________________________________________

CloseDatabase Sub CloseDatabase()

Description:

CloseDatabase is used to close a database that is currently open. Using this command will remove the specifieddatabase from the pool of accessible databases. In the case that this client is the only one currently referencing thedatabase, the database will be physically closed unless it was a database specified in the server .xmimrc file. If aclient terminates without closing a database but that client is the only one referencing the database, the system willautomatically close the database (unless it was a database specified in the server .xmimrc file). The database file mustbe specified using the Database property before closing the database.

____________________________________________________________________

GetDatabases


313

Sub GetDatabases()

Description

GetDatabases returns a listing of all databases in the current database view. That is, if a call toXmimNarrowDatabases has been made and it specifies a single database as the current view, thenXmimGetDatabases will return only that single database (until the XmimWidenDatabases routine is used to set theview back to the entire open database pool).

____________________________________________________________________

GetInfo Sub GetInfo()

Description

Returns information pertaining to a Commodity DataServer master server that is running: the server number, hostname it is running on, server database name, memory map file, owner's name and process id.

XMIMRecords and XMIMRecordsDouble

OverviewYou can retrieve and update historical time series data for one or more securities with an XMIMRecords or theXMIMRecordsDouble object. You can reshape the series to any desired granularity, and control options like how NaNvalues are filled, or what starting and ending date to use. You can also generate synthetic continuous futures contractsusing the Commodity DataServer rollover technology.

XMIMRecords and XMIMRecordsDouble encapsulates the functionality of XmimGetRecords,XmimGetDataRange, XmimGetTradingTime, XmimGetRecordsRollover, XmimGetRolloverDates,XmimPutRecords and XmimReplaceRecords from the Commodity DataServer C-language API. Also encapsulatedare routines for data retrieval and updating via ASCII files.

To create an instance, use New_XMIMServer and then invoke the Init method with a valid XMIMServer object. Thenew object will use the connection provided by the server object to communicate with the Commodity DataServer. Todestroy the object, let it go out of scope, or set it to Nothing.

Retrieving DataBefore you fetch data using XmimRecords or XmimRecordsDouble, you must indicate what columns and whatrelations you are interested in. The NRels property can be used to tell the object to accommodate a specified numberof relations, while the Rel property indicates the name of a particular relation. Each time you set NRels, all therelation names are set to empty strings. The properties are bi-directional, so you can retrieve current values as well asset them. NRels must be set before setting the Rel property.

314


NCols and Col perform analogous operations for columns. In general, for the routines involving the retrieval of data,setting NCols to 0 serves to specify that all columns for the given relations are used.

You can specify various options to control the way data is filtered and shaped; see “XMIMServer Reference” for moredetails.

Once you have specified the relation and column names and any options, invoke GetRecords. This will make arequest to the Commodity DataServer, allocate a buffer for the results, and load the results into the buffer.

The NRecords property returns the number of records fetched by GetRecords. Notice that, if you specify mrelations and n columns, there will be m*n values returned for each record, plus a date.

To access a particular record, use the Date property to fetch the date for a record, and the Val property to retrieve aparticular field. Remember that relation, column, and record indices are zero-based.

There are functions for retrieving the range over which data is available and the trading time range on a given date;see “XMIMServer Reference” for details.

RolloverTo generate a continuous futures contract, you must first establish the attribute values needed for an ordinary retrieval,but, in addition, you must set values for the rollover day, the rollover policy, and other attributes. RolloverDay canmodify and retrieve the rollover day. RolloverPolicy sets or returns a string describing the rollover policy.

To set or fetch the study used for the rollover computation, use the Study property. There are also several propertiesfor setting and accessing the from and to units of the study.

To retrieve rollover data, invoke GetRecordsRollover. You can access the results the same way that you wouldafter GetRecords.

To retrieve only the dates that the contracts would have rolled over, as well as which contracts are active during thatperiod, invoke GetRolloverDates.

There is one thing you should keep in mind. Rollover retrieval only works for a single relation. So, NRels should be setto exactly one. An error will occur if you use GetRecordsRollover or GetRolloverDates with more than onerelation. (Any number of columns can be used, however.)

The rollover language is complicated and beyond the scope of this document. See the Rollovers document for moreinformation.

Large DataNote that a time series in the Commodity DataServer may be very large, especially for intraday or tick-by-tick data.Currently, VBA API stores the entire series in memory. For example, retrieving minute-by-minute data for the SP500currently produces about 800,000 records. Each record requires four bytes for a date, and four bytes for every field.That's nine megabytes of virtual memory just to hold the date, close, and volume. You might run out of swap space if



315

you work with large tick data series. Working with tick data is also very slow because it is so large, especially if yourmachine does not have enough memory to avoid paging activity.

Daily data can be quickly and easily handled, however. Fetching the daily high, low, open, close, and volume for IBM,for example, produces about 6000 records, or around 144 kilobytes; this is not a large amount of memory for today'scomputers. (This is generally true of Visual Basic applications; however, Excel programmers might find that Excel'sVisual Basic for Applications dialect is very slow at filling spreadsheet cells.)

Each time you issue a query, the buffers in an XMIMRecords or XmimRecordsDouble object are reallocated to fitthe incoming data. The Clear function can be used to explicitly free the buffers.

Updating DataTo update data using XMIMRecords, first set the relation and column names as described earlier for retrievingrecords. Next, set NRecords to cause buffer space to be allocated for the new data. Fill in dates with the Dateproperty, and fill in values with the Val property.

You will also need to set the granularity of the data using Units; this tells the Commodity DataServer whether tostore the data at tick, intraday, daily, or some other frequency.

To send the data to the Commodity DataServer, invoke PutRecords. Since these functions update data in thedatabase, a database lock must be successfully obtained before they can be used.

Replacing/Deleting DataTo replace data using XMIMRecords, the relation and column names as well as the NRecords, Date and Valproperties are filled in as when data is updated. NRecords specifies the number of replacement records, that is,the number of dates specified with the Date property. If the Date and Val properties are not set then the specifiedrecords will be deleted from the Commodity DataServer database with no replacement.

In addition, the FromDate and ToDate properties can be used to specify the range of data to be deleted or replaced.Any existing data in the specified range will be deleted. For tick data, the FromTime and ToTime properties furtherspecify the range of data to be replaced; for each day in the data range, data will be deleted only for the trading timerange given.

To effect the replacement/deletion, invoke ReplaceRecords. Note that the number of replacement records need notbe the same as the number of records replaced.

A database lock must be obtained before replacing or deleting data.

File-Based Data Retrieval/UpdatingWhen file-based data retrieval or data updating is desired, the WriteFacts or ReadFacts methods can be used,respectively.

316


The FactsFile property must be filled in so as to specify the ASCII output file where the results of the data retrievalwill be sent or the ASCII input file that the data updates will be read from.

The NRels, Rel, NCols and Col properties must be set to specify the relations and columns for which data is beingretrieved or updated. In the case of data updating, NRels must be 1 and only a single relation specified or must be 0such that no relations are specified. If a relation is not specified, then it will be assumed that the relation is specifiedas part of the records in the file; data for more than one relation can, thus, be read in from a single file.

The number of fields and formats for the fields to be written out or read in can be supplied via the NFieldFormats,FieldFormatType, FieldFormatWidth, and FieldFormatDecimal properties. See “XMIMServer Reference”for details.

The format style used in the input file for data updating can be provided via the RecFormat property. If fixed-format isspecified then it is important that the field format properties be used to specify how to interpret the data read from thefile.

For data updating, to specify whether the data to be read in should be merged with existing data or serve as areplacement use the MergeMode property. To indicate whether the data being read in is daily, intraday tick or tick-by-tick data, set the TickMode property.

To specify the desired frequency with which to aggregate for data retrieval, the NUnits and Units properties can beset. The Units property will also indicate the type of data to be extracted: Seconds will indicate to use tick-by-tickdata, Minutes or Hours will indicate to use intraday data (tick-by-tick if intraday is not available) and, otherwise, dailydata will be used. The EndingDateTime property can be used to provide a time (for tick data) or date (for daily data)that serves as the point of reference for the desired aggregation period.

The FromDate and ToDate properties can be used to specify the range of data to be retrieved. For tick data, theFromTime and ToTime properties further specify the range of data to be retrieved.

Once the appropriate properties have been specified, invoke WriteFacts to retrieve data from the database andwrite it to the output file or ReadFacts to add data stored in the input file to the database. A database lock must beobtained before using ReadFacts to update the database.

XMIMRecords and XMIMRecordsDouble Reference_________________________________________________________________________________

Init

Sub Init(srvr as XMIMServer)

Description:

Establishes a link between the XMIMRecords or XmimRecordsDouble object and the Commodity DataServer. Callthis method before invoking any other properties or methods.

_________________________________________________________________________________


317

Clear

Sub Clear()

Description:

Releases buffers used for dates and values, and restores properties to default settings.

_________________________________________________________________________________

NRels

Property Let NRels(ByVal n as Long) Property Get NRels()as Long

Description:

Allocates or retrieves the specified number of relation names. The strings are initially empty; set them via the Relproperty. The "let" version of this property also widens the buffer used to hold records; existing data is discarded.

_________________________________________________________________________________

Rel

Property Let Rel(ByVal i as Long, s as String) Property Get Rel(ByVal i as Long) as String

Description:

Sets the ith relation name to s, or retrieves the ith relation name.

_________________________________________________________________________________

NCols

Property Let NCols(ByVal n as Long) Property Get NCols() as Long

Description:

Allocates the specified number of column names. The names are initially empty; set them via the Col property. TheLet version of this property also widens the buffer used to hold records; existing data is discarded.

318


_________________________________________________________________________________

Col Property Let Col(ByVal i as Long, s as String) Property Get Col(ByVal i as Long) as String

Description:

Sets the ith column name to s, or retrieves the ith column name.

_________________________________________________________________________________

NRecords Property Let NRecords(ByVal x as Long) Property Get NRecords() as Long

Description:

When set, grows the buffers used for dates and values to accommodate the number of records specified by x.Existing data is preserved if the size increases, or truncated if it decreases.

When read, returns the current number of records.

A query initiated by GetRecords or one of its variants can cause the number of records to change.

_________________________________________________________________________________

DateTime Property Let DateTime(ByVal i as Long, ByVal d as Date) Property Get DateTime(ByVal i as Long) as Date

Description:

Sets or returns the date associated with record i.

_________________________________________________________________________________

DateTimeArray Property Let DateTimeArray(ByRef d As Variant) Property Get DateTimeArray() As Variant

Description:

Sets or returns the datetime array associated with inserting or retrieval of data. This can be used instead ofDateTime.

When using DateTimeArray, the Let property needs an array of VB Dates.


319

_________________________________________________________________________________

Val

Property Let Val(ByVal r as Long, ByVal c as Long, ByVal i as Long, ByVal x as Single) Property Get Val(ByVal r as Long, ByVal c as Long, ByVal i as Long) as Single

Description:

Sets or retrieves the value associated with relation r, column c of record i.

_________________________________________________________________________________

ValArray

Property Let ValArray(ByRef x As Variant) Property Get ValArray () As Variant

Description:

Sets or retrieves the array of interleaved values that will be inserted or retrieved from the database. This may beused instead of Val for setting or retrieving data. Note: when using ValArray, the Let Property needs an array of VBASingles.

_________________________________________________________________________________

FromDate

Property Let FromDate(ByVal d as Date) Property Get FromDate() as Date

Description:

This property sets or fetches the "from date" used by GetRecords and other methods within the XMIMRecords orXMIMRecordsDouble class. Records before the specified date will be discarded by subsequent queries.

If this property is not specified prior to a data retrieval operation, then data will be retrieved from the first date forwhich data is available. A single record is retrieved by using the same date for both FromDate and ToDate.

_________________________________________________________________________________

ToDate

Property Let ToDate(ByVal d as Date) Property Get ToDate() as Date

Description:

This property sets or fetches the "to date" used by GetRecords and other methods within the XMIMRecords orXMIMRecordsDouble class. Records after the specified date will be discarded by subsequent queries.

320


If this property is not specified prior to a data retrieval operation, then data will be retrieved to the last date for whichdata is available. A single record is retrieved by using the same date for both FromDate and ToDate.

_________________________________________________________________________________

CorrectionsDate Property Let CorrectionsDate(ByVal d as Date) Property Get CorrectionsDate() as Date

Description:

This property sets or fetches the "corrections date" used by GetRecords and other methods within theXMIMRecords or XMIMRecordsDouble class. When the CorrectionsDate is specified, values are accessedas they would have appeared on the given date. That is, if a correction was added on June 5, 1997 and theCorrectionsDate specified is before June 5, the old value will be returned; on the other hand, if the date specifiedis after June 5 then the new corrected value will be returned. If the CorrectionsDate is not specified, then it will betaken as the last data date such that any and all corrections will have been applied.

_________________________________________________________________________________

FromTime Property Let FromTime(ByVal d as Date) Property Get FromTime() as Date

Description:

This property sets or fetches the "from time" used by GetRecords and other methods within the XMIMRecords orXmimRecordsDouble class. Records before the specified time of day will be discarded by subsequent queries.

If this property is not specified prior to a data retrieval operation, then data will be retrieved (for each date in the daterange) from the start of the trading period.

This property is only meaningful for intraday or tick data.

_________________________________________________________________________________

ToTime Property Let ToTime(ByVal d as Date) Property Get ToTime() as Date

Description:

This property sets or fetches the "to time" used by GetRecords and other methods within the XMIMRecords orXmimRecordsDouble class. Records after the specified time of day will be discarded by subsequent queries.

If this property is not specified prior to a data retrieval operation, then data will be retrieved (for each date in the daterange) to the end of the trading period.

This property is only meaningful for intraday or tick data.


321

_________________________________________________________________________________

NUnits


Description:

Sets the number of units to be used by GetRecords and other data retrieval methods in this class. This function isnormally used in conjunction with the Units property to control the "shape" or granularity of a time series.

If this property is not specified prior to data retrieval, the number of units used will be 1.

_________________________________________________________________________________

Units

Property Let Units(ByVal u as Long) Property Get Units() as Long

Description:

Sets or retrieves the granularity of execution units when accessing data. This function is normally used in conjunctionwith the NUnits property, which controls the number of execution units. In the case of accessing the data range/trading time and updating or replacing data, this function is used simply to indicate daily (Days), intraday (Minutes)or tick-by-tick (Seconds) data and, therefore, in not used with the NUnits property. u should be a member of theUnits family of XMIMUtils constants.

If this property is not specified prior to any applicable XMIMRecords or XmimRecordsDouble class operation, thenDays will be used as the default.

_________________________________________________________________________________

Limit

Property Let Limit(ByVal x as Long) Property Get Limit() as Long

Description:

Sets or retrieves the number of kilobytes of memory or the number of records to be used so as to limit the resultsreturned by GetRecords queries. If the amount is exceeded, the results will be cut off so that the least recent datais retrieved. This function is used in conjunction with the LimitMode property. If it is not used, all records will bereturned.

322


_________________________________________________________________________________

LimitMode

Property Let LimitMode(ByVal u as Long) Property Get LimitMode() as Long

Description:

Sets or retrieves the mode for limiting the results returned for GetRecords-variant queries. This function is used inconjunction with the Limit property, which gives the actual limit value. The mode determines whether the limit isgiven as number of records returned or amount of memory used. u should be a member of the LimitMode family ofXMIMUtils constants.

_________________________________________________________________________________

MissDataFill

Property Let MissDataFill(ByVal x as Long) Property Get MissDataFill() as Long

Description:

Sets or fetches the NaN fill mode for missing data values. x should be a member of the FillOption family ofXMIMUtils constants.

If this property is not set prior to data retrieval, NaNs will be used for missing data values.

_________________________________________________________________________________

HolidayFill

Property Let HolidayFill(ByVal x as Long) Property Get HolidayFill() as Long

Description:

Sets or fetches the fill mode for holiday NaNs. x should be a member of the FillOption family of XMIMUtilsconstants.

If this property is not set prior to data retrieval, NaNs will be used for holidays.

_________________________________________________________________________________

SkipEmptyRecords

Property Let SkipEmptyRecords(ByVal b as Boolean) Property Get SkipEmptyRecords() as Boolean

Description:


323

Controls the handling of empty records by GetRecords and other data retrieval methods in this class. If b is Trueand all fields of a record are NaNs, then the record will be excluded from the answer set for a query. Otherwise, emptyrecords are included in answer sets.

If this property is not set prior to data retrieval, for daily data, rows of NaNs will not be excluded from the answer setbut, for tick data, rows of invalid data will be skipped.

See also MissDataFill and HolidayFill.

_________________________________________________________________________________

CurrentTickUsage

Property Let CurrentTickUsage(ByVal x as Long) Property Get CurrentTickUsage() as Long

Description:

Sets or fetches an indication of whether historic data retrieved by GetRecords should be supplemented with datafrom the high-frequency current tick store or not. If data from the current tick store is to be used, it will be aggregatedinto the appropriate units and appended, as specified, to the daily and/or tick data retrieval. The historic data will besupplemented with the current tick data only if the ending date/time range given for the data retrieval extends beyondwhat is available in the historic database.

When used in conjunction with PutRecords or ReplaceRecords, this property indicates whether data should beupdated/replaced/deleted in the current tick store as well as the historical data store, depending on where the date/time range is appropriate.

x should be a member of the CurrentTickUsage family of XMIMUtils constants. By default, data from the currenttick store will not be retrieved or updated.

For more information regarding the high-frequency current tick store and it's usage, see the documentation for theclass.

_________________________________________________________________________________

DesiredExpirDate

Property Let DesiredExpirDate(ByVal d as Date) Property Get DesiredExpirDate() as Date

Description:

Sets or retrieves the expiration date to be used to restrict the options to be considered in determining the data rangeby GetDataRangeOption.

If this property is not specified, then all expiration dates will be considered.

324


_________________________________________________________________________________

DesiredOptType

Property Let DesiredOptType(ByVal t as Long) Property Get DesiredOptType() as Long

Description:

Sets or retrieves the option type to be used to restrict the options to be considered in determining the data range byGetDataRangeOption or retrieving options records by GetRecordsOption or GetRecordsOptionRelative.The value for t should be a member of the OptType family of XMIMUtils constants.

If this property is not specified, then both put and call options will be considered.

_________________________________________________________________________________

DesiredStrikePrice

Property Let DesiredStrikePrice(ByVal f as Single) Property Get DesiredStrikePrice() as Single

Description:

Sets or retrieves the strike price to be used to restrict the options to be considered in determining the data range byGetDataRangeOption.

If this property is not specified, then all strike prices will be considered.

____________________________________________________________________

FromExpirDate

Property Let FromExpirDate(ByVal d as Date) Property Get FromExpirDate() as Date

Description:

Sets or retrieves the starting expiration date for GetRecordsOption. Together with ToExpirDate, this property isused to fully specify the range of expiration dates to be considered.

If the range is not specified, then all expiration dates will be considered. A single expiration date is specified by usingthat date for both FromExpirDate and ToExpirDate.


325

_________________________________________________________________________________

ToExpirDate

Property Let ToExpirDate(ByVal d as Date) Property Get ToExpirDate() as Date

Description:

Sets or retrieves the ending expiration date for GetRecordsOption. Together with FromExpirDate, this property isused to fully specify the range of expiration dates to be considered.

If the range is not specified, then all expiration dates will be considered. A single expiration date is specified by usingthat date for both FromExpirDate and ToExpirDate.

_________________________________________________________________________________

FromStrike

Property Let FromStrike(ByVal f as Single) Property Get FromStrike() as Single

Description:

Sets or retrieves the starting strike price for GetRecordsOption. Together with ToStrike, this property is used tofully specify the range of strike prices to be considered.

If the range is not specified, then all strike prices will be considered. A single strike price is specified by using thatvalue for both FromStrike and ToStrike.

_________________________________________________________________________________

ToStrike

Property Let ToStrike(ByVal f as Single) Property Get ToStrike() as Single

Description:

Sets or retrieves the ending strike price for GetRecordsOption. Together with FromStrike, this property is used tofully specify the range of strike prices to be considered.

If the range is not specified, then all strike prices will be considered. A single strike price is specified by using thatvalue for both FromStrike and ToStrike.

326


_________________________________________________________________________________

NumUnitsToExpir

Property Let NumUnitsToExpir(ByVal n as Long) Property Get NumUnitsToExpir() as Long

Description:

This property controls the number of units to expiration for GetRecordsOptionRelative. It is used in conjunctionwith the UnitsToExpir property.

If the expiration units properties are not specified prior to the invocation of GetRecordsOptionRelative, then onemonth will be used as the number of days to expiration in finding the nearest expiring option.

_________________________________________________________________________________

UnitsToExpir

Property Let UnitsToExpir(ByVal u as Long) Property Get UnitsToExpir() as Long

Description:

Sets or retrieves the granularity of units to expiration for GetRecordsOptionRelative. This property is usedin conjunction with the NUnitsToExpir property. The value for u should be a member of the Units family ofXMIMUtils constants.

If the expiration units properties are not specified prior to the invocation of GetRecordsOptionRelative, then onemonth will be used as the number of days to expiration in finding the nearest expiring option.

_________________________________________________________________________________

StrikeWeight

Property Let StrikeWeight(ByVal f as Single) Property Get StrikeWeight() as Single

Description:

Sets or retrieves the percentage of the underlying security's price that the strike price should exceed forGetRecordsOptionRelative.

If this property is not specified, the percentage used will be such as to specify a spot option (100).


327

_________________________________________________________________________________

SelectOnceP

Property Let SelectOnceP(ByVal b as Boolean) Property Get SelectOnceP() as Boolean

Description:

Sets or retrieves the indication of whether the relative selection is to be performed once only (True) or everyday (False). If the selection is to be performed every day (relative perpetual), then a different option will beselected for each day and, thus, a constructed daily time series of option fields will be returned as a result of theGetRecordsOptionRelative invocation. If the selection is to be performed once, then the selected option's timeseries will be returned, ending at expiration or the end of the date range, whichever comes first.

If this property is not specified prior to the GetRecordsOptionRelative invocation, a different option will beselected for each day.

_________________________________________________________________________________

ExpirDate

Property Let ExpirDate(ByVal i as Long, ByVal d as Date) Property Get ExpirDate(ByVal i as Long) as Date

Description:

Sets or returns the expiration date associated with record i for the GetRecordsOption,GetRecordsOptionRelative, PutRecords and ReplaceRecords methods. The number of records will be thesame as the number of records used by the DateTime and Val properties.

_________________________________________________________________________________

OptionType

Property Let OptionType(ByVal i as Long, ByVal x as Long)Property Get OptionType(ByVal i as Long) as Long

Description:

Sets or returns the option type associated with record i for the GetRecordsOption,GetRecordsOptionRelative, PutRecords and ReplaceRecords methods. The number of records will be thesame as the number of records used by the DateTime and Val Properties. The value for x should be a member of theOptType family of XMIMUtils constants.

328


_________________________________________________________________________________

StrikePrice Property Let StrikePrice(ByVal i as Long, ByVal x as Single) Property Get StrikePrice(ByVal i as Long) as Single

Description:

Sets or returns the strike price associated with record i for the GetRecordsOption,GetRecordsOptionRelative, PutRecords and ReplaceRecords methods. The number of records will be thesame as the number of records used by the DateTime and Val properties.

_________________________________________________________________________________

RolloverDay Property Let RolloverDay(s as String) Property Get RolloverDay() as String

Description:

Sets or retrieves the string describing the rollover day in GetRecordsRollover queries.

If this property is left unspecified and GetRecordsRollover is invoked, "last data day" will be used as the rolloverday.

_________________________________________________________________________________

RolloverPolicy Property Let RolloverPolicy(s as String) Property Get RolloverPolicy() as String

Description:

Sets or fetches the string for the rollover policy used in GetRecordsRollover queries.

If this property is left unspecified and GetRecordsRollover is invoked, "1 nearby Actual Prices" will be used as therollover policy.

_________________________________________________________________________________

Study Property Let Study(ByVal n as Long) Property Get Study() as Long

Description:

Sets or retrieves the study to be performed by GetRecordsRollover. The value n should be a member of theAdjustedStudy family of XMIMUtils constants.


329

If this property is left unspecified, then no adjusted study will be computed; the normal computation for the continuouscontract will take place.

_________________________________________________________________________________

NFromUnits

Property Let NFromUnits(ByVal n as Long) Property Get NFromUnits() as Long

Description:

This property controls the number of "from" units to be used for a rollover study. This is the number of units prior tothe current unit such that if it is given as 0, then that is taken as the current unit (e.g., 0 days indicates today). It isnormally used in conjunction with the FromUnits property.

If this property is left unspecified yet a study has been specified with the Study property, the number of from unitsused will be 1, such as to indicate from one unit ago.

_________________________________________________________________________________

FromUnits

Property Let FromUnits(ByVal u as Long) Property Get FromUnits() as Long

Description:

Sets or retrieves the granularity of "from" units to be used in a rollover study. This property is normally used inconjunction with the NFromUnits property. The value for u should be a member of the Units family of XMIMUtilsconstants.

If this property is left unspecified yet a study has been specified with the Study property, the from units used will beDays.

_________________________________________________________________________________

NToUnits

Property Let NToUnits(ByVal n as Long) Property Get NToUnits() as Long

Description:

This property controls the number of "to" units to be used for a rollover study. This is the number of units after thecurrent unit. It is normally used in conjunction with the ToUnits property.

If this property is left unspecified yet a study has been specified with the Study property, the number of to units usedwill be 0, such as to indicate to the current unit (day, by default).

330


_________________________________________________________________________________

ToUnits

Property Let ToUnits(ByVal u as Long) Property Get ToUnits() as Long

Description:

Sets or retrieves the granularity of "to" units to be used in a rollover study. This property is normally used in conjunctionwith the NToUnits property. The value for u should be a member of the Units family of XMIMUtils constants.

If this property is left unspecified yet a study has been specified with the Study property, the to units used will beDays.

_________________________________________________________________________________

NPeriods

Property Get NPeriods() as Long

Description:

Retrieves the number of periods for a continuous contract computation. Each period corresponds to a different choiceof front contract. This property is used in conjunction with the GetRolloverDates subroutine.

_________________________________________________________________________________

NContractsPerPeriod

Property Get NContractsPerPeriod() as Long

Description:

Retrieves the number of contracts studied per period for a continuous contract computation. The number of contractsstudied is always 1 except in the case where either Smoothed or Perpetual is used for the rollover policy. ForPerpetual the number of contracts will always be 2 as two contracts are always being compared. For Smoothedthe number of contracts will be the number specified as part of the rollover policy. This property is used in conjunctionwith the GetRolloverDates subroutine.

_________________________________________________________________________________

PeriodStart

Property Get PeriodStart(ByVal i as Long) as Date

Description:

Retrieves the starting date for the ith period. This property is used in conjunction with the GetRolloverDatessubroutine.


331

_________________________________________________________________________________

PeriodEnd

Property Get PeriodEnd(ByVal i as Long) as Date

Description:

Retrieves the ending date for the ith period. This property is used in conjunction with the GetRolloverDatessubroutine.

_________________________________________________________________________________

Contract

Property Get Contract(ByVal i as Long, ByVal j as Long) as Date

Description:

Retrieves a contract for a specified period, that is, the ith contract for the ith period. This property is used inconjunction with the GetRolloverDates subroutine.

_________________________________________________________________________________

FactsFile

Property Let FactsFile(filename as String ) Property Get FactsFile() as String

Description:

Sets or returns the file specified as filename to be used as the ASCII input file for the ReadFacts method or the ASCIIoutput file for the WriteFacts method.

_________________________________________________________________________________

EndingDateTime

Property Let EndingDateTime(ByVal d as Date) Property Get EndingDateTime() as Date

Description:

Sets or returns the point of reference for the desired aggregation (specified via the NUnits and Units properties).This point of reference will consist of a time for tick data or a date for daily data. The date/time corresponds to theend of the aggregation period and dates/times emanate from it both backwards and forwards (e.g., if the unitsspecification is 1 hour and this property is set to 8:30 then aggregation will be hourly ending on the half hour). Thisproperty is used with the WriteFacts method.

332


____________________________________________________________________

NFieldFormats

Property Let NFieldFormats(ByVal x as Long) Property Get NFieldFormats() as Long

Description:

Sets or returns the number of fields to be read or written when the ReadFacts or WriteFacts methods are used.This property is used in conjunction with the FieldFormatType, FieldFormatWidth and FieldFormatDecimalproperties.

If no field formats are specified, default field formats are taken to consist of a default column field entry for eachcolumn specified, with a default date field and, if necessary, default time and relation fields, in front. In the case ofoptions relations, option field types will additionally be inserted after whichever exist of the date, time and relationfields.

For more information and examples of using field formats, refer to either Chapter 5, “C/C++ API” or Chapter 4,“BMIM Scripting Language”.

_________________________________________________________________________________

FieldFormatType

Property Let FieldFormatType(ByVal i as Long, ByVal t as Long) Property Get FieldFormatType(ByVal i as Long) as Long

Description:

Sets or returns the field type for the ith field to be read or written when the ReadFacts or WriteFacts methods areused. The value for t should be a member of the FieldType family of XMIMUtils constants. This property is usedin conjunction with the FieldFormatWidth and FieldFormatDecimal properties. The NFieldFormats propertygives the number of fields.

_________________________________________________________________________________

FieldFormatWidth

Property Let FieldFormatWidth(ByVal i as Long, ByVal x as Long) Property Get FieldFormatWidth(ByVal i as Long) as Long

Description:

Sets or returns the field width for the ith field to be read or written when the ReadFacts or WriteFacts methodsare used. The field width will correspond to the number of characters in the field for all except date/time fields. Fordates and times, the field width is used to identify the appropriate date/time format and so, in these cases, the valuefor x should be a member of either the DateFormat, TimeFormat or ExpDateFormat family of XMIMUtilsconstants. This property is used in conjunction with the FieldFormatType and FieldFormatDecimal properties.The NFieldFormats property gives the number of fields.


333

_________________________________________________________________________________

FieldFormatDecimal

Property Let FieldFormatDecimal(ByVal i as Long, ByVal x as Long) Property Get FieldFormatDecimal(ByVal i as Long) as Long

Description:

Sets or returns the number of digits to print after the decimal point for the ith field to be read or written when theReadFacts or WriteFacts methods are used. This property is only applicable for the column and strike pricefield types since they are the only types with actual data values. This property is used in conjunction with theFieldFormatType and FieldFormatWidth properties. The NFieldFormats property gives the number of fields.

_________________________________________________________________________________

RecFormat

Property Let RecFormat(ByVal u as Long) Property Get RecFormat() as Long

Description:

Sets or returns the specification indicating whether the records in the file should be read assuming fixed-format fieldsor free-format (variable-sized) fields. This property is used with the ReadFacts method. The value for u should be amember of the RecordFormat family of XMIMUtils constants.

When this property is set such as to specify fixed-format fields, then the field format specifications (provided using theNFieldFormats, FieldFormatType, FieldFormatWidth and FieldFormatDecimal properties) are used todetermine how to interpret data read from the input file.

When a file is read using free-format, fields may be separated by either spaces or commas and the fields will simplybe read in as given. If the field format properties are used in conjunction with free-formatting, the width and decimalfield properties will be ignored. When multi-field column data is being read, free-format must be used if the multi-fieldcolumn name is specified instead of the individual multi-field column fields.

If this property is not specified prior to invoking the ReadFacts method, free-format fields will be assumed.

_________________________________________________________________________________

MergeMode

Property Let MergeMode(ByVal u as Long) Property Get MergeMode() as Long

Description:

Sets or returns the merge mode used to indicate whether the data to be read in will serve as an update to or areplacement for existing data. This property is used with the ReadFacts method. The value for u should be a memberof the MergeMode family of XMIMUtils constants.

334


If this property is not specified prior to invoking the ReadFacts method, the data read in will be merged with existingdata instead of replacing it.

_________________________________________________________________________________

TickMode Property Let TickMode(ByVal u as Long) Property Get TickMode() as Long

Description:

Sets or returns the tick mode used to indicate whether tick data is being read in and, if so, whether it is intraday tickdata, tick-by-tick data or tick-by-tick data that should be automatically aggregated into intraday data. This propertyis used with the ReadFacts method. The value for u should be a member of the TickMode family of XMIMUtilsconstants.

If this property is not specified prior to invoking the ReadFacts method, the assumption will be made that daily datais being read in, not tick data.

_________________________________________________________________________________

GetRecords Sub GetRecords()

Description:

Retrieves a time series from the Commodity DataServer. The number of records in the answer set can be determinedusing the NRecords property. The records themselves can be accessed using the Date and Val properties.

Before calling this function, make sure that relation names and column names have been established (with the NRels,Rel, NCols and Col properties), and that any desired options have been specified (missing data fill, holiday fill, unitsand number of units, limit and limit mode, from and to dates, from and to times, current tick usage and empty recordhandling).

Note that if multiple relation names have been specified (using the NRels and Rel properties) , then theunion of all columns of all relations will be returned, filling with NaNs as appropriate.

_________________________________________________________________________________

GetRecordsOption Sub GetRecordsOption()

Description:

Retrieves an option time series from the Commodity DataServer. The number of records in the answer set canbe determined using the NRecords property. The records themselves can be accessed using the Date and Val


335

properties. Additionally, the expiration dates, option types and strike prices for the corresponding values can beaccessed using the ExpirDate, OptionType and StrikePrice properties.

Note that when options data is involved, there may be multiple records for the same date.

Before calling this function, make sure that option relation names and column names have been established (with theNRels, Rel, NCols and Col properties), and that any desired optional properties have been specified (from and toexpiration dates, from and to strike prices, options type, missing data fill, holiday fill, units and number of units, limitand limit mode, from and to dates, from and to times, and empty record handling).

It is recommended that, for efficiency considerations, the NaN handling options be specified such that filling withNaNs takes place for both holidays and missing data when dealing with options data.

_________________________________________________________________________________

GetRecordsOptionRelative

Sub GetRecordsOptionRelative()

Description:

Handles relative requests for options data retrieval from the MIM server. The number of days to expiration and thepercentage of spot are specified, via the NumUnitsToExpir, UnitsToExpir and StrikeWeight properties. Thenumber of records in the answer set can be determined using the NRecords property. The records themselves canbe accessed using the Date and Val properties. Additionally, the expiration dates, option types and strike prices for thecorresponding values can be accessed using the ExpirDate, OptionType and StrikePrice properties.

Note that when options data is involved, there may be multiple records for the same date.

Before calling this function, make sure that option relation names and column names have been established, and thatany desired optional properties have been specified (units and number of units to expiration, strike weight, option type,select once or every day indicator, missing data fill, holiday fill, units and number of units, limit and limit mode, fromand to dates, from and to times, and empty record handling).

For more information regarding the selection of relevant options and comparison order, refer to Chapter 5, “C/C++API”

_________________________________________________________________________________

GetDataRange

Sub GetDataRange(ByVal i as Long, ByVal j as Long)

336


Description:

Retrieves the range over which the specified data is available. The ith relation and the ith column from the relationand column arrays (accessed via the Rel and Col properties) serve to specify the desired data. If the column index(j) is given as -1 then the data range for all columns of the ith relation will be retrieved. The FromDate and ToDateproperties are used to access the date range.

Before calling this function, make sure that the units option has been specified (via the Units property) if desired.

Note that this option is used to specify whether the data range for daily (days) , intraday (minutes), ortick (seconds) data should be returned.

_________________________________________________________________________________

GetDataRangeOption

Sub GetDataRangeOption(ByVal i as Long, ByVal j as Long)

Description:

Retrieves the range over which the specified options data is available. The ith relation and the ith column from therelation and column arrays (accessed via the Rel and Col properties) serve to specify the desired data. If the columnindex (j) is given as -1 then the data range for all columns of the ith relation will be retrieved. The FromDate andToDate properties are used to access the date range.

Before calling this function, make sure that the units have been specified (via the Units property) if the type of dataconsidered is to be restricted either to daily, intraday or tick data. The desired expiration date, option type and strikeprice can also be specified to further restrict the options that are considered in determining the data range.

_________________________________________________________________________________

GetTradingTime

Sub GetTradingTime(ByVal i as Long, ByVal j as Long, ByVal d as Date)

Description:

Retrieves the actual trading time range on a given date for the ith relation and the ith column from the relation andcolumn arrays (accessed via the Rel and Col properties). If the column index (j) is given as -1 then all columns of theith relation will be considered. The FromTime and ToTime properties are used to access the trading time.

Before calling this function, make sure that the units option has been specified (via the Units property) if desired.

Note that this option is used to specify whether the data considered should be intraday (minutes) or tick(seconds). This function is not meaningful for daily data.


337

_________________________________________________________________________________

GetRecordsRollover

Sub GetRecordsRollover()

Description:

Retrieves a synthetic continuous futures contract from the Commodity DataServer. The number of records in theanswer set can be determined using the NRecords property. The records themselves can be accessed using theDate and Val properties.

Before calling this function, make sure a relation name and column names have been established, and any desiredoptions have been specified (missing data fill, holiday fill, units and number of units, limit and limit mode, from and todates, from and to times, empty record handling, rollover day and policy, study, and "from" and "to" units for the study).

Note that, unlike with the GetRecords method, only a single relation can be used withGetRecordsRollover, so the NRels property must be set to 1.

_________________________________________________________________________________

GetRolloverDates

Sub GetRolloverDates()

Description:

Retrieves the dates that a continuous contract will roll over and which contracts are active during that period. Thenumber of periods (where each period corresponds to a different choice of front contract) and the number of contractsstudied per period are determined using the NPeriods and NContractsPerPeriod properties. The start andend dates defining the periods are determined using the PeriodStart and PeriodEnd properties. The relevantcontracts for each rollover period are accessed using the Contract property.

Before calling this function, make sure a relation name and column names have been established, and any desiredoptions have been specified (units and number of units, rollover day and policy).

As with GetRecordsRollover, only a single relation can be used with GetRolloverDates.

338


_________________________________________________________________________________

PutRecords Sub PutRecords()

Description:

Updates a time series in the Commodity DataServer.

Before calling this function, make sure that relation names and column names have been established (using theNRels, Rel, NCols and Col properties), that the number of records has been set with the NRecords property,and that dates and values have been associated with records (using the Date and Val properties). Also, if desired,the units should be set with the Units property so as to indicate that daily (days), intraday (minutes) or real tick(seconds) data is being updated. The CurrentTickUsage property can optionally be set to AppendCTDToAll ifcurrent tick data is to also be updated.

_________________________________________________________________________________

PutRecordsOption Sub PutRecordsOption()

Description:

Updates an option time series in the Commodity DataServer.

Before calling this function, make sure that options relation names and column names have been established (usingthe NRels, Rel, NCols and Col properties), that the number of records has been set with the NRecords property,and that dates and values have been associated with records (using the Date and Val properties). Also, if desired,the units should be set with the Units property so as to indicate that daily (days), intraday (minutes) or real tick(seconds) data is being updated. In addition, expiration dates, option types and strike prices may also be specifiedvia the ExpirDate, OptionType and StrikePrice properties. If these properties are used, only data for thecorresponding expiration dates, option types and/or strike prices will be updated.

_________________________________________________________________________________

ReplaceRecords Sub ReplaceRecords()

Description:

Replaces or deletes data from a time series in the Commodity DataServer.

Before calling this function, make sure that relation names and column names have been established (using theNRels, Rel, NCols and Col properties), that the number of records has been set with the NRecords property, thatdates and values have been associated with records (using the Date and Val properties) and that the units are setappropriately as with the PutRecords subroutine. Additionally, the range of data to be replaced or deleted should bespecified using the FromDate and ToDate properties and, if appropriate, the FromTime and ToTime properties. TheCurrentTickUsage property can optionally be set to AppendCTDToAll if current tick data is to also be replaced.


339

_________________________________________________________________________________

ReplaceRecordsOption

Sub ReplaceRecordsOption()

Description:

Replaces or deletes data from an option time series in the Commodity DataServer.

Before calling this function, make sure that relation names and column names have been established (using theNRels, Rel, NCols and Col properties), that the number of records has been set with the NRecords property,that dates and values have been associated with records (using the Date and Val properties) and that the unitsare set appropriately as with the PutRecords subroutine. Additionally, the range of data to be replaced or deletedshould be specified using the FromDate and ToDate properties and, if appropriate, the FromTime and ToTimeproperties. The expiration dates, option types and strike prices can be specified via the ExpirDate, OptionType,and StrikePrice properties to further constrain the data affected.

_________________________________________________________________________________

ReadFacts

Sub ReadFacts()

Description:

Reads an ASCII data file and loads it into the Commodity DataServer database.

Before calling this function, make sure that input filename has been specified via the FactFile property and that anydesired options have been specified (relation name, number of columns and column names, number of fields and fieldtypes, field widths and field decimals, record format, merge mode and tick mode).

_________________________________________________________________________________

WriteFacts

Sub WriteFacts()

Description:

Extracts data from the Commodity DataServer database and writes it to an ASCII data file.

Before calling this function, make sure that output filename has been specified via the FactFile property and thatany desired options have been specified (number of relations and relation names, number of columns and columnnames, number of fields and field types/widths/decimals, number of units and units, ending date/time, from/to dateand from/to time).

340


_________________________________________________________________________________

CorrectionsFile Property Let CorrectionsFile(s As String) Property Get CorrectionsFile() As String

Description:

Sets and Retrieves a file which is used to make corrections to the database. The file specifying the corrections datamust be composed of one correction per line. Corrections data is of the following form for intraday data: 20030421, 0000, APX.DAHOURLY, Val, 20030422, 1400, 1.25

Where this line indicates that on April 21, 2003 a corrected value for Val of APX.DAHOURLY on April 22, 2003 at 2:00PM was placed in the database and the previous value for that series was 1.25. Note that if two corrections for thesame relation on the same day are received, only the last one is kept.

For daily data corrections, use the format: 20030421, APX.DADAILY, Val, 20030422, 1.25

After the CorrectionsFile is set, the AddCorrections method is used to update the database.

_________________________________________________________________________________

AddCorrections Sub AddCorrections()

Description:

Adds corrections data to the database. Once corrections have been added to the database, data may be accessed insuch a manner as to take into consideration any corrections made through a specified date, but none made after thatdate using the CorrectionsDate property.

The CorrectionsFile property must be specified before calling this method.

XMIMRelation

OverviewIn the Commodity DataServer, a relation corresponds to a trading instrument (i.e., equity, future, etc.). TheXMIMRelation class exposes the Commodity DataServer relation hierarchy. This makes it especially useful forbuilding browsers and other tools that access the Commodity DataServer schema.

Use the New_XMIMRelation to create a XMIMRelation instance. Before using any other methods in the object,invoke the Init method and pass it a valid XMIMServer instance.


341

Set an instance to Nothing, or let it go out of scope, to dispose of it.

After creating a XMIMRelation object, you can fetch information about a particular relation by setting the relationname (using the Name property) and then invoking the Fetch method. The relation name typically corresponds to aticker symbol and, therefore, must be unique across the relation domain unless the relation is of a type such that it willonly be used to group other relations and never have data associated with it.

To delete a relation, set the name and then call Delete.

To modify a relation, the easiest way is probably to fetch the relation's current information, make any desired changes,then use Modify to apply the changes to the Commodity DataServer database. Note that all properties of anXMIMRelation object will be transmitted to the Commodity DataServer (except for parent, children, and columns),so they should all contain valid data. Fetching the data before modifying it ensures that all properties will be correctlyinitialized.

To add a new relation, set the name and initial values for the relation's properties, then use the Add method to createthe relation in the Commodity DataServer schema.

Note that relations are organized hierarchically so the parent must be specified when a relation isbeing added. TopRelation denotes the root of the relation hierarchy. Properties that are not explicitlyspecified prior to using the Add method, will be inherited from the parent relation (except as noted in“XMIMServer Reference”).

There are several functions for setting and retrieving descriptive information; see “XMIMServer Reference” for details.

You must lock the database before using the add, modify, or delete functions. The database must also be locked beforesetting the holiday schedules for relations or exchanges.

XMIMRelation Reference_________________________________________________________________________________

Init


Description:

Establishes a link between the XMIMRelation object and the Commodity DataServer. Call this method beforeinvoking any other properties or methods.

_________________________________________________________________________________

Clear

Sub Clear()

342


Description:

Resets all properties to default values.

_________________________________________________________________________________

Name

Property Let Name(s as String) Property Get Name() as String

Description:

Sets or retrieves the relation name. This determines which relation is affected by fetch, add, delete, or modifyoperations.

Note that merely setting the property won't cause the relation name to be changed in the CommodityDataServer database; you must invoke the Modify method to cause an update. The name typicallycorresponds to the ticker symbol and must be unique for all but category relations.

In the case of futures continuous or futures contract relations, the names must begin with the name of the futuresparent relation followed by an underscore character. Options relation names must also include this parent name prefix.

_________________________________________________________________________________

Alias

Property Let Alias(s as String) Property Get Alias() as String rel.Name = "TopRelation:Equities:i:ib:IBM" rel.Parent = "TopRelation:Equities:ByCusip" rel.Alias = "c45920010" rel.AddAlias

Description:

Sets or retrieves the and alias for the relation name. The Parent property supplies the parent for the alias (categorywhere the alias will reside) and the Name property is what the alias points to. The Parent and and Name propertiesmust include paths if the names alone are ambiguous and the path is necessary to disambiguate. If there is ambiguityand path names are not specified, an error message will be returned. After the Name and Parent properties are set,the AddAlias method is used to update the database with the new alias.

The alias facility provides a mechanism whereby multiple paths to a relation may be established (e.g., Equities maybe organized by countries in addition to the current organization). It also provides for referring to relations by alternatenames (e.g., aliases can be created such that relations can be referred to by the CUSIP numbers in addition to theticker symbols).


343

_________________________________________________________________________________

Description Property Let Description(s as String) Property Get Description() as String

Description:

Sets or retrieves the relation description. The description is a string containing a user-defined explanation of therelation.

When this property is not specified prior to invoking the Add method, the description will default such as to beidentical to the relation name.

_________________________________________________________________________________

Parent Property Let Parent(s as String) Property Get Parent() as String

Description:

Sets or retrieves the name of the current relation's parent in the relation hierarchy. This will typically be the category towhich the relation is added.

The main futures continuous contract must be the parent for all continuous and individual contracts for a given future.Options relations must have a parent specified that is not itself an option and also not a category relation; that is,the parent must be a non-options relation with the potential for having data associated with it. As such, any type ofrelation in the hierarchy can actually be a parent relation with the exception of options relations.

_________________________________________________________________________________

RelType Property Let RelType(ByVal x as Long) Property Get RelType() as Long

Description:

Sets or retrieves the type of a relation. The value of x should be a member of the RelType family of XMIMUtilsconstants.

The RelFutures type corresponds to the main futures continuous contract and must be the parent of all continuous(RelFuturesContinuous) and individual contracts (RelFuturesContract) for a given future. The RelCategorytype represents relations used merely to group other relations and is the only type that can never have actual dataassociated with it. RelOptions relations are used to store options data and RelNormal relations include all non-futures and non-options relations that can have data associated with them.

When this property is not specified prior to invoking the Add method, the type will default to RelNormal.

344


_________________________________________________________________________________

Exchange

Property Let Exchange(s as String) Property Get Exchange() as String

Description:

Sets or retrieves the name of the exchange on which a security trades.

_________________________________________________________________________________

TimeZone

Property Let TimeZone(s as String) Property Get TimeZone() as String

Description:

Sets or retrieves the time zone associated with a relation.

_________________________________________________________________________________

StartTrade

Property Let StartTrade(ByVal d as Date) Property Get StartTrade() as Date

Description:

Sets or retrieves the time at which daily trading begins for a security.

_________________________________________________________________________________

EndTrade

Property Let EndTrade(ByVal d as Date) Property Get EndTrade() as Date

Description:

Sets or retrieves the time at which daily trading ends for a security.

_________________________________________________________________________________

ContractUnits

Property Let ContractUnits(ByVal x as Single) Property Get ContractUnits() as Single

Description:


345

Sets or retrieves the number of units per contract for use with Commodity Query profit and loss queries.

346


_________________________________________________________________________________

ExpDate

Property Let ExpDate(ByVal d as Date) Property Get ExpDate() as Date

Description:

Sets or retrieves the contract expiration date. The property is applicable only for relations that are individual futurescontracts.

_________________________________________________________________________________

NoticeDate

Property Let NoticeDate(ByVal d as Date) Property Get NoticeDate() as Date

Description:

Sets or retrieves the contract first-notice date. This property is applicable only for relations that are individual futurescontracts.

_________________________________________________________________________________

RolloverDay

Property Let RolloverDay(s as String) Property Get RolloverDay() as String

Description:

Sets or retrieves the rollover day for a relation. This can be used to affect how the Commodity DataServer generatessynthetic continuous contracts from individual futures contracts. It is, thus, applicable only for relations typed asfutures continuous contracts. See Chapter 5, “C/C++ API” or the Rollovers document for details.

_________________________________________________________________________________

RolloverPolicy

Property Let RolloverPolicy(s as String) Property Get RolloverPolicy() as String

Description:

Sets or retrieves the rollover policy for a relation. This can be used to affect how the Commodity DataServer generatessynthetic continuous contracts from individual futures contracts. It is, thus, applicable only for relations typed asfutures continuous contracts. See Chapter 5, “C/C++ API” or the Rollovers document for details.




347

_________________________________________________________________________________

RolloverType

Property Let RolloverType(ByVal x as Long) Property Get RolloverType() as Long

Description:

Sets or retrieves the rollover data type used to specify whether rollover should take place for daily and/or tick data. Thevalue for x should be a member of the RolloverDataType family of XMIMUtils constants. See Chapter 5, “C/C++ API” or the Rollovers document for details about rollovers.

_________________________________________________________________________________

DataType

Property Let DataType(ByVal x as Long) Property Get DataType() as Long

Description:

Sets or retrieves the type of data to be delivered to the specified facts file. This property is used in conjunction withthe SetDefaultFactsFile subroutine. The value of x should be a member of the DataType family of XMIMUtilsconstants.

If this property is not specified prior to using SetDefaultFactsFile, data for all types of data will be added to thefile specified in the SetDefaultFactsFile method.

____________________________________________________________________

Nchildren

Property Get NChildren() as Long

Description:

Retrieves the number of child relations for the specified relation object. Used in conjunction with the Child propertyto browse the relation hierarchy.

_________________________________________________________________________________

Child

Property Get Child(ByVal i as Long) as String

Description:

Retrieves the ith child relation name. Used to browse the relation hierarchy.


348


_________________________________________________________________________________

NColumns Property Get NColumns() as Long

Description:

Returns the number of columns associated with the specified relation object. Used in conjunction with the Columnproperty to browse the columns of a relation.

_________________________________________________________________________________

NColumnsWithData Property Get NColumnsWithData() as Long

Description:

Returns only those columns associated with the specified relation object which have had data added to them.

Note that if a relation column exists with only NaN values, it will be returned as having data. Also, if arelation column previously had data that was subsequently deleted (e.g., using XmimDeleteFacts),it will still be returned by this routine. These nuances are necessary in order to make this routine asefficient as possible.

_________________________________________________________________________________

Column Property Get Column(ByVal i as Long) as String

Description:

Returns the ith column name. Used to browse the columns of a relation.

_________________________________________________________________________________

Root Property Get Root() as String

Description:

Returns the name of the top-most category for a specified relation object.

_________________________________________________________________________________

Path Property Get Path() as String


349

Description:

Returns the entire parent path of the specified relation object.

_________________________________________________________________________________

Add Sub Add()

Description:

Adds a new relation to the Commodity DataServer schema with the property values of the associated XMIMRelationobject.

_________________________________________________________________________________

AddAlias Sub AddAlias() rel.Name = "TopRelation:Equities:i:ib:IBM" rel.Parent = "TopRelation:Equities:ByCusip" rel.Alias = "c45920010" rel.AddAlias

Description:

Adds an alias for the specified Commodity DataServer relation object. The Parent property supplies the parent forthe alias (category where the alias will reside) and the Name property is what the alias points to. The Alias propertysupplies the alias name.

_________________________________________________________________________________

Delete Sub Delete()

Description:

Deletes a relation named by the associated XMIMRelation object.

_________________________________________________________________________________

Modify Sub Modify(newname as String)

Description:

Modifies the relation named by the associated XMIMRelation object; property values for the relation are replacedby those in the object. The Name property must be used to identify the existing relation to be modified. Thus, if therelation name itself is to be modified, the new relation name is specified as newname.

350


_________________________________________________________________________________

Fetch Sub Fetch()

Description:

Retrieves property values for a relation named by an XMIMRelation object.

_________________________________________________________________________________

SetRelHolidays Sub SetRelHolidays(filename as String)

Description:

Sets the holiday schedule to be used for a particular relation given by an XMIMRelation object.

The holiday schedule is given by the file filename. This file should consist of a series of holiday dates, one per line, withthe date format used being one of the standard Commodity DataServer date formats (see the DateFormat family ofXMIMUtils constants). A date range may be specified using a hyphen in between the two dates.

_________________________________________________________________________________

SetExchangeHolidays Sub SetExchangeHolidays(filename as String)

Description:

Sets the holiday schedule to be used for a particular exchange as identified using the Exchange property.

The holiday schedule is given by the file filename. This file should consist of a series of holiday dates, one per line, withthe date format used being one of the standard Commodity DataServer date formats (see the DateFormat family ofXMIMUtils constants). A date range may be specified using a hyphen in between the two dates.

_________________________________________________________________________________

SetDefaultFactsFile Sub SetDefaultFactsFile(filename as String)

Description:

By default, the Commodity DataServer stores the daily, intraday tick and real tick data to files specified by the system.

This method can be used to override this behavior. The DataType property can be used to specify whether

TickData, IntradayData or DailyData will be stored in the specified file. If not set, all data will be sent to thespecified file. For more info on database partitioning see Chapter 4, “BMIM Scripting Language”.


351

_________________________________________________________________________________

RegenRollover Sub RegenRollover()

Description:

Continuous contracts will be automatically created and updated by the Commodity DataServer. Whenever possible,the Commodity DataServer optimizes the updating of continuous contracts such that most types of continuouscontracts are updated incrementally and are not completely re-generated with each update. Sometimes it is desirableto have the continuous contracts for a given relation completely re-computed. The RegenRollover command cangenerate the continuous contracts. The Name property specifies which relation to generate contracts for. If the nameproperty is not set, continuous contracts will be re-computed for all futures and RelFuturesContinuous relationsin the database. If the Name is of type category or futures, then continuous contracts will be re-computed for allfutures and RelFuturesContinuous relations occurring in that sub-tree of the relation hierarchy.

XMIMColumn

OverviewXMIMColumn is analogous to XMIMRelation. It provides access to the Commodity DataServer column hierarchy.In the Commodity DataServer, a column corresponds to specific information (attributes) pertaining to a relation (i.e.,economic or monetary values, market activity, etc.).

Create XMIMColumn objects using New_XMIMColumn. Invoke the Init method before using any other properties ormethods.

To dispose of an instance, let it go out of scope, or set it to Nothing.

Use Clear to reset a column object to its default state.

To fetch information about a particular column, first set the column name (using the Name property) and then callFetch. Column names must be unique across the column domain unless the column is of a type such that it will onlybe used to group other columns and never have data associated with it; category column names do not have to beunique.

To delete a column, set the name and then call Delete.

The usual way to modify a column is to fetch its current information, make changes, then invoke Modify to apply thechanges to the Commodity DataServer schema.

Note that all properties of an XMIMColumn object will be transmitted to the Commodity DataServer(except for parent and children), so they should all contain valid data. Fetching the data before modifyingit ensures that all properties will be correctly initialized.

352


To add a new column, set the name and initial values for the column's properties, then call Add to actually create thecolumn in the Commodity DataServer.

Note that columns are organized hierarchically so the parent must be specified when a column is beingadded. TopColumn denotes the root of the column hierarchy.

The methods in this section are also applicable for use with composite or multi-field columns. A multi-field columnis a special type of column used to store data for which there is more than one associated field. Multi-field columnsare, for instance, useful for storing tick-by-tick data consisting of an arbitrary number of user-defined fields for eachtick/transaction. They can conceptually be thought of as homogeneous relational tables. Each field of a multi-fieldcolumn has an associated field name and field description. Properties are provided for handling the fields of a multi-field column.

There are several functions for setting and retrieving descriptive information; see the Reference section “XMIMServerReference” for details.

You must lock the database before using the add, modify, or delete functions.

XMIMColumn Reference_________________________________________________________________________________

Init Sub Init(srvr as XMIMServer)

Description:

Establishes a link between the XMIMColumn object and the Commodity DataServer. Call this method before using anyother properties or methods.

_________________________________________________________________________________

Clear Sub Clear()

Description:


_________________________________________________________________________________

Name Property Let Name(s as String) Property Get Name() as String


353

Description:

Sets or retrieves the name property of the object. This determines which column is affected by fetch, add, delete, ormodify operations.

_________________________________________________________________________________

Description

Property Let Description(s as String) Property Get Description() as String

Description:

Sets or retrieves the column description.

_________________________________________________________________________________

Parent

Property Let Parent(s as String) Property Get Parent() as String

Description:

Sets or retrieves the name of the current column's parent.

_________________________________________________________________________________

ColType

Property Let ColType(ByVal x as Long) Property Get ColType() as Long

Description:

Sets or retrieves the type of a column. The value of x should be a member of the ColType family of XMIMUtilsconstants.

ColCategory columns are used to group other columns whereas ColNormal columns may have data associatedwith them. ColComposite columns are used to store data for which there is more than one field and are, hence,referred to as multi-field columns.

If this property is not specified prior to using the Add method, ColNormal will be used as the default column type.

_________________________________________________________________________________

NFields

Property Let NFields(ByVal n as Long) Property Get NFields() as Long

354


Description:

Sets or retrieves the number of fields for a multi-field column. Used in conjunction with the FieldName andFieldDesc properties. Normal Commodity DataServer columns consist of only a single field; multi-field columns areused to store data where multiple data fields are necessary.

This property is not modifiable and, hence, will not be utilized in the context of the Modify method.

_________________________________________________________________________________

FieldName Property Let FieldName(ByVal i as Long, s as String) Property Get FieldName(ByVal i as Long) as String

Description:

Sets or retrieves the ith field name for a multi-field column.

Note that field names need only be unique across a given multi-field column. Thus, when retrieving datafor a multi-field column, individual field names may be specified without a qualifier only if the field nameis unique across all columns in the database and can, therefore, be unambiguously resolved; otherwise,the multi-field column name must be used as a qualifier, i.e., column_name:field_name. If the multi-fieldcolumn name itself is specified then data will be retrieved for all fields of the multi-field column.


_________________________________________________________________________________

FieldDesc Property Let FieldDesc(ByVal i as Long, s as String) Property Get FieldDesc(ByVal i as Long) as String

Description:

Sets or retrieves the ith field description for a multi-field column.


_________________________________________________________________________________

Nchildren Property Get NChildren() as Long

Description:

Retrieves the number of child columns for the specified column object. Used in conjunction with the Child propertyto browse the column hierarchy.


355

_________________________________________________________________________________

Child

Property Get Child(ByVal i as Long) as String

Description:

Retrieves the ith child column name. Used to browse the column hierarchy.

_________________________________________________________________________________

Root

Property Get Root() as String

Description:

Retrieves name of the top-most category for a specified column object.

_________________________________________________________________________________

Path

Property Get Path() as String

Description:

Retrieves entire parent path for a specified column object.

_________________________________________________________________________________

Add

Sub Add()

Description:

Adds a new column to the Commodity DataServer schema with the property values of the associated XMIMColumnobject.

_________________________________________________________________________________

Delete

Sub Delete()

Description:

Deletes a column named by the associated XMIMColumn object.

356


_________________________________________________________________________________

Modify

Sub Modify(newname as String)

Description:

Modifies the column named by the associated XMIMColumn object; property values for the relation are replacedby those in the object. The Name property must be used to identify the existing column to be modified. Thus, if thecolumn name itself is to be modified, the new column name is specified as newname.

_________________________________________________________________________________

Fetch

Sub Fetch()

Description:

Retrieves property values for a column named by an XMIMColumn object.

XMIMRelCol

OverviewXMIMRelCol provides a way to access information about individual relation columns in the Commodity DataServer. Arelation column, relcol, is the intersection of a relation (ticker) and column (attribute), for instance, “IBM.Close” or“SP.Volume”. In the Commodity DataServer, relcols are the sites where actual time series (price, volume, etc.) datais stored.

Create XMIMRelCol objects using New_XMIMRelCol. Invoke the Init method before using any other properties ormethods. To dispose of an instance, let it go out of scope, or set it to Nothing.

Use Clear to reset a relcol object to its default state.

Together, the Rel and Col properties are used to specify a particular relcol; both properties must be set before anyother operations can be performed.

There are several properties that can be set for a XMIMRelCol object for instance, the object type, aggregation rule,and minimum price increment (delta). See “XMIMServer Reference” for more information.

Note that special properties are provided for handling the fields for multi-field columns.


357

To fetch data for a relcol, first set the relation and column names, then invoke the Fetch method. Use Delete toremove a relcol. Any data associated with the relcol will also be deleted. To delete the data associated with arelcol without deleting the relcol itself from the database, use DeleteFacts or DeleteFactsSelective.

To add or modify a relation, set the properties as desired, then use the Add or Modify methods. Properties that arenot explicitly specified prior to using the Add method, will be inherited from the parent relation or the nearest ancestorrelation that also contains the appropriate column. For a modify operation, consider fetching existing property valuesand then making changes, to ensure that all properties have valid values.

For a modify operation, the column specified can not be a category column. However, for an add operation, a categorycolumn is allowed and the result will be that the column hierarchy will be traversed to find all non-category (leaf)columns that are descendants of that column and these will be the columns actually added to the relation. Likewise, ifthe relation specified for an add operation has children, the appropriate columns will be added not only to the specifiedrelation, but also to all of its children relations. In the case of modify, the relation can correspond to a relation withchildren but the modifications will not be explicitly propagated to the children relations.

Note that when a relation is added to the database, all of the columns of the parent relation are inheritedand, thus, the corresponding relcols are automatically added. This does not apply in the case ofrelations of type option; no columns are inherited for options relations.

The Commodity DataServer database must be locked before adding, modifying, or deleting a relcol or any dataassociated with a relcol.

XMIMRelCol Reference_________________________________________________________________________________

Init


Description:

Establishes a link between the XMIMRelCol object and the Commodity DataServer. Call this method before using anyother properties or methods.

_________________________________________________________________________________

Clear

Sub Clear()

Description:


358


_________________________________________________________________________________

Rell Property Let Rel(s as String) Property Get Rel() as String

Description:

Sets or retrieves the relation name for the specified relcol object. Together with the column name (see Col) thisdetermines which relcol is affected by fetch, update, delete, or add operations.

_________________________________________________________________________________

Col Property Let Col(s as String) Property Get Col() as String

Description:

Sets or retrieves the column name for the specified relcol object. Together with the relation name (see Rel) thisdetermines which relcol is affected by fetch, update, delete, or add operations.

_________________________________________________________________________________

RelColType Property Let RelColType(ByVal x as Long) Property Get RelColType() as Long

Description:

Sets or retrieves the relcol type. The value of x should be a member of the RelColType family of XMIMUtilsconstants.

_________________________________________________________________________________

ObjType Property Let ObjType(ByVal x as Long) Property Get ObjType() as Long

Description:

Sets or retrieves the relcol object type. This corresponds to the type for the actual data values of the relcol. Thevalue of x should be a member of the RelColObjType family of XMIMUtils constants.

When the relcol derives from a multi-field column, there is an additional object data type that can be specified. Thisis the block data type and is used to indicate a record of data with a user-specified number of bytes. This data typeis specified as "block(i)" where i is the number of bytes. The StringToRelColObjType utility function (from theXMIMUtils class) must be used to convert this string to a value when setting the relcol object type. Likewise, the


359

RelColObjTypeToString utility function can be used to convert the object type returned to a string of the form"block(i)".

_________________________________________________________________________________

Aggr Property Let Aggr(ByVal x as Long) Property Get Aggr() as Long

Description:

Sets or retrieves the aggregation rule for a relcol. This controls how fine-grained data is reshaped to coarse-graineddata. The value of x should be a member of the RelColAggr family of XMIMUtils constants.

_________________________________________________________________________________

Delta Property Let Delta(ByVal x as Long) Property Get Delta() as Long

Description:

Sets or retrieves the minimum increment for data values in a relcol. The value of x should be a member of theRelColDelta family of XMIMUtils constants.

_________________________________________________________________________________

Constant Property Let Constant(ByVal x as Single) Property Get Constant() as Single

Description:

Determines the value of a relcol data series when the object type is RelColConstant.

_________________________________________________________________________________

Nfields Property Let NFields(ByVal x as Long) Property Get NFields() as Long

Description:

Sets or retrieves the number of fields for a multi-field column of a relation.

Note that all fields must be of the same relcol type as specified using the ColType property.

360


_________________________________________________________________________________

FieldObjType

Property Let FieldObjType(ByVal i as Long, ByVal x as Long) Property Get FieldObjType(ByVal i as Long) as Long

Description:

Sets or retrieves the relcol object type for the ith field of a multi-field column of a relation. This corresponds to thetype for the actual data values of the relcol. The value of x should be a member of the RelColObjType family ofXMIMUtils constants.

_________________________________________________________________________________

FieldAggr

Property Let FieldAggr(ByVal i as Long, ByVal x as Long) Property Get FieldAggr(ByVal i as Long) as Long

Description:

Sets or retrieves the relcol aggregation rule for the ith field of a multi-field column of a relation. This controls howfine-grained data is reshaped to coarse-grained data. The value of x should be a member of the RelColAggr family ofXMIMUtils constants.

_________________________________________________________________________________

FieldDelta

Property Let FieldDelta(ByVal i as Long, ByVal x as Long) Property Get FieldDelta(ByVal i as Long) as Long

Description:

Sets or retrieves the minimum increment for data values in a relcol for the ith field of a multi-field column of arelation. The value of x should be a member of the RelColDelta family of XMIMUtils constants.

_________________________________________________________________________________

FieldConstant

Property Let FieldConstant(ByVal i as Long, ByVal x as Single) Property Get FieldConstant(ByVal i as Long) as Single

Description:

Sets or retrieves the value of a relcol data series for the ith field of a multi-field column of a relation when the objecttype is RelColConstant.


361

_________________________________________________________________________________

DataType

Property Let DataType(ByVal x as Long) Property Get DataType() as Long

Description:

Sets or retrieves the type of data to be used. This property is used in conjunction with the DeleteFactsSelectivesubroutine. The value of x should be a member of the DataType family of XMIMUtils constants.

If this property is not specified prior to using DeleteFactsSelective, data will be deleted for all types of data.

_________________________________________________________________________________

DailyMultiplicity

Property Let DailyMultiplicity(ByVal x as Long) Property Get DailyMultiplicity() as Long

Description:

Used to specify individually whether the daily data for the relation column will be single-valued or multi-valued. Amulti-valued relation column is one in which multiple values can be stored for a single time slot as opposed to a single-valued relation column where there is only a single value for each time slot. DailyMultiplicity must be set to oneof the following:

-1 = INVALID 0 = SINGLE 1 = MULTIPLE

relcol.DailyMultiplicity = 0

If not specified daily columns will be single-valued.

_________________________________________________________________________________

IntradayMultiplicity

Property Let IntradayMultiplicity(ByVal x as Long) Property Get IntradayMultiplicity() as Long

Description:

Used to specify individually whether intraday for the relation column will be single-valued or multi-valued. A multi-valued relation column is one in which multiple values can be stored for a single time slot as opposed to a single-valued relation column where there is only a single value for each time slot. IntradayMultiplicity must be set toone of the following:

-1 = INVALID0 = SINGLE1 = MULTIPLE

362


relcol.IntradayMultiplicity = 0

If not specified intraday columns will be single-valued.

_________________________________________________________________________________

TickMultiplicity Property Let TickMultiplicity(ByVal x as Long) Property Get TickMultiplicity() as Long

Description:

Used to specify individually whether tick data for the relation column will be single-valued or multi-valued. A multi-valued relation column is one in which multiple values can be stored for a single time slot as opposed to a single-valued relation column where there is only a single value for each time slot. Tick-by-tick data is typically multi-valued.TickMultiplicity must be set to one of the following.-1 = INVALID0 = SINGLE1 = MULTIPLE

relcol.TickMultiplicity = 1

If not specified, tick-by-tick will be multi-valued.

_________________________________________________________________________________

FromDate Property Let FromDate(ByVal d as Date) Property Get FromDate() as Date

Description:

This property sets or fetches the "from date" used by DeleteFactsSelective.

If the date range is not specified prior to using DeleteFactsSelective, the entire history of applicable data will bedeleted.

_________________________________________________________________________________

ToDate Property Let ToDate(ByVal d as Date) Property Get ToDate() as Date

Description:

This property sets or fetches the "to date" used by DeleteFactsSelective.

If the date range is not specified prior to using DeleteFactsSelective, the entire history of applicable data will bedeleted.


363

_________________________________________________________________________________

FromTime

Property Let FromTime(ByVal d as Date) Property Get FromTime() as Date

Description:

This property sets or fetches the "from time" used by DeleteFactsSelective.

If the time range is not specified prior to using DeleteFactsSelective, the entire history of applicable data will bedeleted.

_________________________________________________________________________________

ToTime

Property Let ToTime(ByVal d as Date) Property Get ToTime() as Date

Description:

This property sets or fetches the "to time" used by DeleteFactsSelective.

If the time range is not specified prior to using DeleteFactsSelective, the entire history of applicable data will bedeleted.

_________________________________________________________________________________

Add

Sub Add()

Description:

Adds a new relcol to the Commodity DataServer schema with the property values of the associated XMIMRelColobject.

The relation column to be added must be specified via the Rel and Col properties. In addition, the relcol type,object type, aggregation rule, delta and constant may optionally be specified. For adding multi-field columns to arelation, the number of fields and, for each field, the object type, aggregation rule, delta and constant value may bespecified.

364


_________________________________________________________________________________

Delete

Sub Delete()

Description:

Deletes a relcol and all associated data.

_________________________________________________________________________________

Modify

Sub Modify()

Description:

Modifies the relcol named by the associated XMIMRelCol object; property values for the relation are replaced bythose in the object.

_________________________________________________________________________________

Fetch

Sub Fetch()

Description:

Retrieves property values for a relcol named by the XMIMRelCol object.

_________________________________________________________________________________

DeleteFacts

Sub DeleteFacts()

Description:

Deletes all data for a relcol specified by the XMIMRelCol object.

Note that only the data is deleted, the relcol itself will still exist in the database.


365

_________________________________________________________________________________

DeleteFactsSelective

Sub DeleteFactsSelective()

Description:

Selectively deletes data for a relcol specified by the XMIMRelCol object. The range of data to delete may bespecified using the FromDate and ToDate properties. The time range of the data to delete may be specified usingthe FromTime and ToTime properties. In addition, the type of data to delete is specified using the DataTypeproperty.

XMIMShowWhen and XMIMShowWhenDouble

OverviewIt is possible to execute simple Commodity Query SHOW-WHEN queries using the XMIMShowWhen orXMIMShowWhenDouble class. This class also provides for file-based execution of any Commodity Query query.

As with other VBA API classes, create XMIMShowWhen or XMIMShowWhenDouble objects usingNew_XMIMShowWhen or New_XMIMShowWhenDouble. Invoke the Init method before using any other properties ormethods. To dispose of an instance, let it go out of scope, or set it to Nothing.

Use Clear to reset an XMIMShowWhen or XMIMShowWhenDouble object to its default state

To construct a query, first set the number of attributes (items to report in the answer set) using the NAttrs property,and set the number of conditions (filter criteria) using NConds. Then use the Attr and Cond properties to fill inthe attributes and conditions. While no conditions need be specified, at least one attribute must be set. Use theBeforeRepeat and AfterRepeat properties to specify a repeat factor in terms of the number of extra records to beincluded before/after each date of occurrence (i.e., when the conditions are satisfied).

You can also select an execution granularity for the query, via NUnits and Units, and use the MissDataFill,HolidayFill and SkipEmptyRecords properties to specify how NaNs should be handled. TheCurrentTickUsage property can be used to specify that the historic data be supplemented from the current tickstore. These properties are like their counterparts in the XMIMRecords and XMIMRecordsDouble classes.

To run the query, call the Execute method. This will send the query to the Commodity DataServer, allocate space forthe results, and load the results into the XMIMShowWhen or XMIMShowWhenDouble object.

For each result in the answer set, there will be a date, and one field per attribute. Use the NRecords property todetermine the number of records. Use the Date property to fetch a particular date, and the Val property to accessindividual fields of a record. Use the SuccessP property to determine which records represent actual date/times thatthe conditions were satisfied and which are included because of a specified repeat factor.

366


The ExecuteSavedQuery method can be used to execute or graph any standard Commodity Query query that hasbeen saved to a file. Prior to using this method, the input filename must be specified, via InputFile. Additionally,a specification must be given directing the query results to either an output file or the printer, via OutputFile,or PrintP. Several other optional properties are available for use and, if desired, should be set prior to invokingExecuteSavedQuery. These include properties for effecting substitutions to be performed during query execution.Properties for specifying the merge mode, verbosity, and whether or not to graph the results are also available.

The LoadBmimScript method can be used to load a BMIM script contained in a file specified via the InputFileproperty. The output, along with any errors or warnings, is directed to the file specified via the OutputFile property.

XMIMShowWhen and XMIMShowWhenDouble Reference_________________________________________________________________________________


Description:

Establishes a link between the XMIMShowWhen or XMIMShowWhenDouble object and the Commodity DataServer.Call this method before using any other properties or methods.

_________________________________________________________________________________

Clear Sub Clear()

Description:


_________________________________________________________________________________

NAttrs Property Let NAttrs(ByVal n as Long) Property Get NAttrs() as Long

Description:

Sets or retrieves the number of attributes. The attributes are initially empty; use Attr to fill them in.

Note that prior to using the Execute method, at least one attribute must be set so NAttrs must be atleast 1.

_________________________________________________________________________________


367

Attr

Property Let Attr(ByVal i as Long, s as String) Property Get Attr(ByVal i as Long) as String

Description:

Sets or retrieves the ith attribute string. At least one attribute must be specified before the Execute method can beused.

_________________________________________________________________________________

NConds

Property Let NConds(ByVal n as Long) Property Get NConds() as Long

Description:

Sets or retrieves the number of conditions. The conditions are initially empty; use Cond to fill them in.

_________________________________________________________________________________

Cond

Property Let Cond(ByVal i as Long, s as String) Property Get Cond(ByVal i as Long) as String

Description:

Sets or retrieves the ith condition string.

_________________________________________________________________________________

NUnits


Description:

Sets or retrieves the number of units to be used by the Execute method This property is normally used in conjunctionwith the Units property to control the "shape" or granularity of a query.

If the execution units are not specified prior to invoking the Execute method, the default units will be 1 day.

368


_________________________________________________________________________________

Units

Property Let Units(ByVal x as Long) Property Get Units() as Long

Description:

Sets the units to be used by the Execute method. This property is normally used in conjunction with the NUnitsproperty to control the "shape" or granularity of a query. The value of x should be a member of the Units family ofXMIMUtils constants.


_________________________________________________________________________________

UnionDate

Property Let UnionDate(ByVal x as Long) Property Get UnionDate() as Long

Description:

When the UnionDate option is turned on (set to 1), the union of two dates is applied. For example, consider thedate range for A+B where: “A” ranges from January to June and “B” ranges from March to September. With theUnionDate option turned on, the resulting range will be from January to September.

The default is to have UnionDate turned off and the intersection of the dates will be applied. Theresulting intersection for this example would be March to June.

_________________________________________________________________________________

MissDataFill


Description:


If this property is not specified prior to invoking the Execute method, NaNs will be used for missing data values bydefault.


369

_________________________________________________________________________________

HolidayFill


Description:


If this property is not specified prior to invoking the Execute method, NaNs will be used for holidays by default.

_________________________________________________________________________________

SkipEmptyRecords

Property Let SkipEmptyRecords(ByVal b as Boolean) Property Get SkipEmptyRecords() as Boolean

Description:

Controls the handling of empty records by Execute. If b is True and all fields of a record are NaNs, then the recordwill be excluded from the answer set for a query. Otherwise, empty records are included in answer sets.

If this property is not specified prior to invoking the Execute method, rows of NaNs will not be excluded from theanswer set but, for tick data, rows of invalid data will be skipped.

See also “XMIMRecords and XMIMRecordsDouble Reference” and “XMIMRecords and XMIMRecordsDoubleReference”.

_________________________________________________________________________________

CurrentTickUsage


Description:

Sets or fetches an indication of whether historic data to be used for query execution (via use of the Execute method)should be supplemented with data from the high-frequency current tick store or not. If data from the current tick storeis to be used, it will be aggregated into the appropriate units and appended, as specified, to the daily and/or tick dataretrieval. The historic data will be supplemented with the current tick data only as needed.

x should be a member of the CurrentTickUsage family of XMIMUtils constants. By default, data from the currenttick store will not be retrieved or updated as this will provide for much faster query execution.

For more information regarding the high-frequency current tick store and it's usage, see the documentation of theclass.

370


_________________________________________________________________________________

BeforeRepeat

Property Let BeforeRepeat(ByVal x as Long) Property Get BeforeRepeat() as Long

Description:

Sets or fetches the repeat factor in terms of the number of extra records to be returned prior to each record satisfyingthe condition. The extra records will represent the given number of dates/times occurring prior to the date/time thatactually satisfies the condition. Thus, if execution units are set to 1 day and this property is given as 2, then, for eachoccurrence satisfying the conditions, 2 additional records will be returned for the 2 days before the conditions weresatisfied.

This property is used only in conjunction with the Execute method and will default to 0 such that no repeat factor isused.

_________________________________________________________________________________

AfterRepeat

Property Let AfterRepeat(ByVal x as Long) Property Get AfterRepeat() as Long

Description:

Sets or fetches the repeat factor in terms of the number of extra records to be returned following each recordsatisfying the condition. The extra records will represent the given number of dates/times occurring after the date/timethat actually satisfies the condition. Thus, if execution units are set to 1 day and this property is given as 2, then, foreach occurrence satisfying the conditions, 2 additional records will be returned for the 2 days after the conditions weresatisfied.

This property is used only in conjunction with the Execute method and will default to 0 such that no repeat factor isused.

_________________________________________________________________________________

InputFile

Property Let InputFile(filename as String) Property Get InputFile() as String

Description:

Sets or retrieves the name of the query file to be executed by ExecuteSavedQuery or the name of the script file tobe loaded by LoadBMIMScript.


371

_________________________________________________________________________________

OutputFile

Property Let OutputFile(filename as String) Property Get OutputFile() as String

Description:

Sets or retrieves the name of the file to which the query execution, LoadBmimScript results or postscript graphresults will be stored by ExecuteSavedQuery.

_________________________________________________________________________________

NSubs

Property Let NSubs(ByVal x As Long) Property Get NSubs() as Long

Description:

Sets or retrieves the number of substitutions used by ExecuteSavedQuery. The substitutions argument is used togive a list of substitutions to be performed before executing the query. This allows a query to be repeatedly executed,once for each item in the substitution list.

_________________________________________________________________________________

SubsLeftSide

Property Let SubsLeftSide(ByVal i As Long, s As String) Property Get SubsLeftSide(ByVal i As Long) As String

Description:

Sets or retrieves the left hand side elements. The left-hand side element is the string to be substituted for.

_________________________________________________________________________________

NRightSides

Property Let NRightSides(ByVal i As Long, ByVal n As Long) Property Get NRightSides (ByVal i As Long) as Long

Description:

Sets or retrieves the number of items on the right side to be substituted. The right-hand sides are the strings whichare to be substituted in.

372


_________________________________________________________________________________

SubsRightSide

Property Let SubsRightSide(ByVal i As Long, s As String) Property Get SubsRightSide(ByVal i As Long) As String

Description:

Sets or retrieves the right-hand side elements. The right-hand side element is the string to be substituted for.

_________________________________________________________________________________

MergeMode

Property Let MergeMode(ByVal x as Long) Property Get MergeMode() as Long

Description:

Sets or fetches the merge mode for the output file used by ExecuteSavedQuery. x should be a member of theMergeMode family of XMIMUtils constants.

If this property is not specified, executing a saved query will result in replacing any existing contents of the specifiedoutput file.

____________________________________________________________________

Verbosity

Property Let Verbosity(ByVal x as Long) Property Get Verbosity() as Long

Description:

Sets or fetches the verbosity mode for determining whether the query (and user options) should be included in theoutput of ExecuteSavedQuery. x should be a member of the Verbosity family of XMIMUtils constants.

If this property is not specified, the query will not be included in the output.

_________________________________________________________________________________

PrintP

Property Let PrintP(ByVal b as Boolean) Property Get PrintP() as Boolean

Description:

Sets or fetches the print option which is used to specify that the output should be sent to a printer in lieu of being sentto an output file when a query is executed using ExecuteSavedQuery.


373

_________________________________________________________________________________

GraphP

Property Let GraphP(ByVal b as Boolean) Property Get GraphP() as Boolean

Description:

Sets or fetches the graph option which is used to specify that the output should be in the form of postscriptcommands to generate a graph of the query output rather than a tabular report. This option is only applicable when aquery is executed using ExecuteSavedQuery.

_________________________________________________________________________________

NRecords

Property Get NRecords() as Long

Description:

Returns the number of records generated by a query.

_________________________________________________________________________________

Date

Property Get DateTime(ByVal i as Long) as Date

Description:

Returns the date/time associated with record i.

_________________________________________________________________________________

SuccessP

Property Get SuccessP(ByVal i as Long) as Boolean

Description:

Returns an indication of whether record i actually satisfied the condition (True) or whether it was returned due to thebefore/after repeat specification (False).

This property is, of course, not needed if the repeat values are not set (or are set to 0).

374


_________________________________________________________________________________

Val

Property Get Val(a as Long, i as Long) as Single

Description:

Retrieves the value associated with attribute a of record i.

_________________________________________________________________________________

Execute

Sub Execute()

Description:

Executes a query on the Commodity DataServer. The number of records in the answer set can be determined usingthe NRecords property. The records themselves can be accessed using Date and Val. In addition, the SuccessPproperty can be used to determine which records actually satisfied the conditions as opposed to those that areincluded because a repeat factor was specified.

Before calling this function, make sure attribute and condition strings have been established, and any desired optionshave been specified (units and number of units, before/after repeat factor, current tick usage, as well as the NaN fillingoptions).

_________________________________________________________________________________

ExecuteSavedQuery

Sub ExecuteSavedQuery()

Description:

Performs file-based execution of a query. The output is either saved to a file or sent to a printer and may be in report orgraphical form.

Before calling this function, make sure the input filename has been established, either an output filename given or theprinter flag set and any desired options have been specified (substitutions to be made during execution, merge mode,verbose mode and graph flag).

_________________________________________________________________________________

LoadBmimScript

Sub LoadBmimScript()

Description:


375

The LoadBmimScript method can be used to load a BMIM script contained in a file specified via the InputFileproperty. The output, along with any errors or warnings, is directed to the file specified via the OutputFile property.

XMIMExecution and XMIMExecutionDouble

Overview

The XMIMExecution and XMIMExecutionDouble classes are used to execute SHOW-WHEN queries saved asstrings.

As with other VBA API classes, create XMIMExecution or XMIMExecutionDouble objects usingNew_XMIMExecution or New_XMIMExecutionDouble. Invoke the Init method before using any other propertiesor methods. To dispose of an instance, let it go out of scope, or set it to Nothing.

After creating the XMIMExecution or XMIMExecutionDouble object, set the query string using the Queryproperty.

You can also select an execution granularity for the query, via NUnits and Units, and use the MissDataFill,HolidayFill and SkipEmptyRecords properties to specify how NaNs should be handled. TheCurrentTickUsage property can be used to specify that the historic data be supplemented from the current tickstore. These properties are like their counterparts in the XMIMRecords class.

To run the query, call the Execute method. This loads the results into the XMIMExecution object.

If the query string contains LET statements, multiple reports will be returned. If the query contains more than oneSHOW statement, multiple blocks will be returned. The NReports and NBlocks properties can be used to determinethe number of blocks and reports and blocks returned after executing the query. The title for each report can beaccessed using the ReportTitle property.

The number of attributes, each attribute heading, columns and each column heading for each query block returned of aparticular report can be determined using the NAttrs, AttrHeading, NCols and ColHeading properties.

If an attribute contains a repeat statement, the attribute will return more than one column of data. The number ofcolumns per attribute can be determined using the NColsOfAttr property.

For each query block of a particular report the number of rows returned can be determined using the NRows property.For each row returned, the data and time values can be determined using the RowDate and RowTime properties. Thevalues in each row and column can be accessed using the RowColValue property.

For similar code equivalents using BMIM, Commodity DataServer C/C++ or Commodity DataServer Java APIs,see “Executing Saved Queries” (BMIM API), “Executing Saved Queries” (Commodity DataServer C/C++ API), and“MimData.queryExecute” (Commodity DataServer Java API).

376


Example:

Sub XMIMExecutionExampleDouble() 'Server and Execution Objs Dim svr As XmimServer Dim queryExec As XmimExecutionDouble Set svr = New_XmimServer Set queryExec = New_XmimExecutionDouble 'Connect to the correct Server and Port svr.Host = "auslm13" svr.ServerNum = 0 svr.Connect 'Init the Execution Obj queryExec.Init svr 'The Query for Execution queryExec.Query = "Show 1: Close of NG When Date is after 2002" 'Execute the Query queryExec.Execute 'Get the number of Report Blocks Dim num_reports As Integer num_reports = queryExec.NReports Dim num_report_blocks As Integer Dim num_rows As Integer Dim num_attrs As Integer 'Print out each report For i = 0 To num_reports - 1 'Get the number of Report Blocks for each report num_report_blocks = queryExec.NBlocks(i) 'Print out each report block of this report For j = 0 To num_report_blocks - 1 'Get the number of attrs (Relations from query) num_attrs = queryExec.NAttrs(i, j) 'Get the number of rows of data for this block num_rows = queryExec.NRows(i, j) 'Put the data in the Wookbook Sheet1 For k = 0 To num_rows - 1 Worksheets("Sheet1").Cells(k + 1, 1).Value = queryExec.RowDate(i, j, k) 'Get the val for each attr For l = 0 To num_attrs - 1 Worksheets("Sheet1").Cells(k + 1, l + 2).Value = queryExec.RowColValue(i, j, k, l) Next l Next k Next j Next i 'Disconnect from the Server


377

svr.Disconnect 'Set the objs to Nothing Set showwhen = Nothing Set svr = NothingEnd Sub

Query Structure ExampleThe query below will contains a LET statement and two SHOW-WHEN query blocks.

A separate report will be returned for each symbol in the LET statement. Results can be accessed for each query blockwithin the report. The first query block has two attributes which each return only 1 column of data. The second queryblock has only one attribute which returns only 1 column of data.

Sample Query String:LETtheSec = IBM, DELL( Query Block 1 )SHOW Close: Close of theSec (Attribute 1) Open: Open of theSec (Attribute 2)WHEN Date is 7/5/2002( Query Block 2 )SHOW GeVolume: Volume of GE (Attribute 1) WHEN Date is 7/5/2002Results from running in XMIM:( Report 1 ) ( Query Block 1 )Let variable values: ( Report Heading ) theSec = IBM (Column 1) (Column 2) Date Close Open (Column Headings) 07/05/2002 73.5000 72.4500 (RowColValue) ( Report 1 ) ( Query Block 2

378


)Let variable values:theSec = IBM Date GeVolume 07/05/2002 19258100 ( Report 2 ) ( Query Block 1 )Let variable values:theSec = DELL Date Close Open07/05/2002 25.9900 25.1000 ( Report 2 ) ( Query Block 2 )Let variable values:theSec = DELL Date GeVolume 07/05/2002 19258100

XMIMExecution and XMIMExecutionDouble Reference_________________________________________________________________________________


Description:

Establishes a link between the XMIMExecution or XMIMExecutionDouble object and the Commodity DataServer.Call this method before invoking any other properties or methods.

_________________________________________________________________________________

Query Property Let Query(x as String) Property Get Query() as String

Description:

Sets or retrieves the query string to be executed.

_________________________________________________________________________________

UseClientMacrosP


379

Property Let UseClientMacrosP(ByVal x as Boolean) Property Get UseClientMacrosP() as Boolean

Description:

Indicates whether the executed query will use client macros. If UseClientMacrosP is not specified before invokingthe Execute method, the default setting is False.

_________________________________________________________________________________

NUnits


Description:

Sets or retrieves the number of units to be used by the Execute method. This property is normally used in conjunctionwith the Units property to control the "shape" or granularity of a query.


_________________________________________________________________________________

Units

Property Let Units(ByVal x as Long) Property Get Units() as Long

Description:

Sets the units to be used by the Execute method. This property is normally used in conjunction with the NUnitsproperty to control the "shape" or granularity of a query. The value of x should be a member of the Units family ofXMIMUtils constants.


_________________________________________________________________________________

MissDataFill


Description:


If this property is not specified prior to invoking the Execute method, NaNs will be used for missing data values bydefault. In order to also fill holiday NaN’s, the HolidayFill property must be set.

380


_________________________________________________________________________________

HolidayFill


Description:


If this property is not specified prior to invoking the Execute method, NaNs will be used for holidays by default.


381

_________________________________________________________________________________

CurrentTickUsage


Description:

Sets or fetches an indication of whether historic data to be used for query execution (via use of the Execute method)should be supplemented with data from the high-frequency current tick store or not. If data from the current tick storeis to be used, it will be aggregated into the appropriate units and appended, as specified, to the daily and/or tick dataretrieval. The historic data will be supplemented with the current tick data only as needed.

x should be a member of the CurrentTickUsage family of XMIMUtils constants. By default, data from the currenttick store will not be retrieved or updated as this will provide for much faster query execution.

For more information regarding the high-frequency current tick store and it's usage, see the documentation of theclass.

_________________________________________________________________________________

NReports

Property Get NReports() as Long

Description:

Returns the number of reports generated by a query after calling the Execute method.

_________________________________________________________________________________

ReportTitle

Property Get ReportTitle(ByVal i as Long) as Long

Description:

Returns the report title associated with report i.

_________________________________________________________________________________

NAttrs

Property Get NAttrs(ByVal i as Long, ByVal j As Long) as Long

Description:

Retrieves the number of attributes associated with block j of report i.

382


_________________________________________________________________________________

AttrHeading

Property Get AttrHeading(ByVal i as Long, ByVal j As Long, ByVal k As Long) as String

Description:

Retrieves the heading associated with attribute k, block j of report i.

_________________________________________________________________________________

NCols

Property Get NCols(ByVal i as Long, ByVal j As Long) as Long

Description:

Retrieves the number of columns associated with block j of report i.

_________________________________________________________________________________

ColHeading

Property Get ColHeading(ByVal i as Long, ByVal j As Long, ByVal k As Long) as String

Description:

Retrieves the heading associated with column k, block j of report i.

_________________________________________________________________________________

NColsOfAttr

Property Get NColsofAttr(ByVal i as Long, ByVal j As Long, ByVal k As Long) as String

Description:

Retrieves the number of columns associated with attribute k of block j of report i. Typically, an attribute will returnonly one column of date, however when the repeat function of the Commodity DataServer query language is used onan attribute the numbers of columns returned could be greater than one.

_________________________________________________________________________________

NRows

Property Get NRows (ByVal i as Long, ByVal j As Long) as Long

Description:


383

Retrieves the number of rows returned by block j of report i after calling the Execute method.

_________________________________________________________________________________

RowDate

Property Get RowDate(ByVal i as Long, ByVal j as Long, ByVal k as Long) as Date

Description:

Retrieves the date associated with row k, block j of report i after the Execute method is called.

_________________________________________________________________________________

RowTime

Property Get RowTime(ByVal i as Long, ByVal j as Long, ByVal k as Long) as Date

Description:

Retrieves the time associated with row k, block j of report i after the Execute method is called.

_________________________________________________________________________________

RowColValue

Property Get Val(ByVal i as Long, ByVal j as Long, ByVal k as Long, ByVal l as Long) as Single

Description:

Retrieves the value associated with column l, row k, block j of report i after the Execute method is called.

_________________________________________________________________________________

Execute

Sub Execute()

Description:

Executes a query on the Commodity DataServer. The number of rows in the answer set can be determined using theNRows property. Values can be accessed using the RowDate, RowTime and RowColValue properties .

Before calling this function, make sure the query string has been established, and any desired options have beenspecified (units and number of units, current tick usage, as well as the NaN filling options).

_________________________________________________________________________________

DeleteResult

Sub DeleteResult ()

384


Description:

Deletes all value results returned by the Execute method.

XMIMSchema

OverviewThe structure of the Commodity DataServer database is described by the Commodity DataServer schema. Thus, therelation and column hierarchies together with all information pertaining to the attributes of existing relations, columnsand relcols comprise the Commodity DataServer schema. The XMIMSchema class provides a way to accessinformation about the Commodity DataServer schema. The relation and column hierarchies can be retrieved, searchedor printed. Exchange and time zone information can also be retrieved.

Create XMIMSchema objects using New_XMIMSchema. Invoke the Init method before using any other properties ormethods. To dispose of an instance, let it go out of scope, or set it to Nothing.

Use Clear to reset a schema object to its default state.

When retrieving or printing the relation or column hierarchies, the RelRoot or ColRoot property can be set such thatonly the portion of the hierarchy rooted by the relation or column specified is used. Otherwise, the entire hierarchy willbe used.

For printing, the output file name should be specified prior to using the PrintRels or PrintCols methods. ForPrintRels additional printing options are available and must be set before invoking the method.

To search for a relation or column, first set the search pattern, then invoke either the FindRels or FindColsmethod. Various options can be specified to control the search; see “XMIMServer Reference” for more information.

The NRels property returns the number of relations fetched by GetAllRels or FindRels. Likewise, the NColsproperty returns the number of columns fetched by GetAllCols or FindCols. The Col and Rel properties can beused to retrieve the columns and relations.

The exchanges and time zones existing in the Commodity DataServer schema are retrieved by theGetAllExchanges and GetAllTimeZones methods. The number returned is given by the NExchanges andNTimeZones properties and the individual exchanges and time zones are obtained using the Exchange andTimeZone properties.

XMIMSchema Reference_________________________________________________________________________________

Init


385


Description:

Establishes a link between the XMIMSchema object and the Commodity DataServer. Call this method before using anyother properties or methods.

_________________________________________________________________________________

Clear Sub Clear()

Description:


_________________________________________________________________________________

RelRoot Property Let RelRoot(s as String) Property Get RelRoot() as String

Description:

Sets or retrieves the relation name that will be the root of the portion of the relation hierarchy to be printed orretrieved.

_________________________________________________________________________________

ColRoot Property Let ColRoot(s as String) Property Get ColRoot() as String

Description:

Sets or retrieves the column name that will be the root of the portion of the column hierarchy to be printed orretrieved.

_________________________________________________________________________________

Pattern Property Let Pattern(s as String) Property Get Pattern() as String

Description:

Sets or retrieves the pattern to be used by the FindRels or FindCols method for searching the schema. Thepattern may consist of any number of wildcard characters, "*", and these characters may exist anywhere in thepattern string.

386


_________________________________________________________________________________

CaseSensitiveP

Property Let CaseSensitiveP(ByVal b as Boolean) Property Get CaseSensitiveP() as Boolean

Description:

Indicates whether the relation or column search should be case sensitive or not. If this property is not specified, thesearch operation will not be case sensitive.

_________________________________________________________________________________

ByNameP

Property Let ByNameP(ByVal b as Boolean) Property Get ByNameP() as Long

Description:

Indicates whether the relation or column search should be performed on relation/column names or descriptions.

If this property is not specified, the search will be performed based on names rather than descriptions.

_________________________________________________________________________________

SearchFilter

Property Let SearchFilter(ByVal x as Long) Property Get SearchFilter() as Long

Description:

Indicates whether the relation or column search results should be filtered so as to return only those relations orcolumns that have data associated with them, only those that are categories with no data association or both. Thevalue of x should be a member of the SearchFilter family of XMIMUtils constants.

By default, all relations or columns will be searched.

_________________________________________________________________________________

OutputFile

Property Let OutputFile(ByVal s as String) Property Get OutputFile() as String

Description:

Sets or retrieves the filename to which the output of PrintRels or PrintCols is printed.

If this property is not specified prior to the print operation, the output will be sent to stdout.


387

_________________________________________________________________________________

MaxDepth Property Let MaxDepth(ByVal n as Long) Property Get MaxDepth() as Long

Description:

Sets or retrieves the number of generations of relations that are printed by PrintRels. If n is set to 0 onlyinformation pertaining to the relation specified is output. If n is set to 1 then information pertaining to the immediatedescendants will also be printed, and so on. If n is set to -1 then information for the relation specified and all of itsdescendants will be recursively printed.

If this property is not specified prior to the print operation, information for the relation and all descendants will beprinted.

_________________________________________________________________________________

AllContractsP Property Let AllContractsP(ByVal b as Boolean) Property Get AllContractsP() as Boolean

Description:

Indicates whether or not relation information should be printed for all individual contracts existing in the hierarchybeneath the futures relation. This option is only relevant for futures relations and only used in conjunction with thePrintRels method.

If this property is not specified prior to the print operation, individual contracts will not be printed.

_________________________________________________________________________________

MergeMode Property Let MergeMode(ByVal x as Long) Property Get MergeMode() as Long

Description:

Indicates whether the PrintRels output file should be replaced or appended to. The value of x should be a memberof the MergeMode family of XMIMUtils constants.

If this property is not specified prior to the print operation, the output file will be replaced.

_________________________________________________________________________________

PrintP Property Let PrintP(ByVal b as Boolean) Property Get PrintP() as Boolean

388


Description:

Indicates whether or not information returned by PrintRels should be sent to a printer or not.


389

_________________________________________________________________________________

VerboseP

Property Let VerboseP(ByVal b as Boolean) Property Get VerboseP() as Boolean

Description:

If True then all relation and relcol schema fields will be printed. If False only the relation name, columns definedfor the relation and the starting and ending date range for the data will be printed. This option is available for use withthe PrintRels method.

If this property is not specified prior to the print operation, the less verbose option is used.

_________________________________________________________________________________

NRels

Property Get NRels() as Long

Description:

Returns the number of relations generated by FindRels or GetAllRels. The relations themselves are returned bythe Rel property.

_________________________________________________________________________________

Rel

Property Get Rel(ByVal i as Long) as String

Description:

Returns the ith relation name generated by FindRels or GetAllRels.

_________________________________________________________________________________

NCols

Property Get NCols() as Long

Description:

Returns the number of columns generated by FindCols or GetAllCols. The columns themselves are returned bythe Col property.

390


_________________________________________________________________________________

Col

Property Get Col(ByVal i as Long) as String

Description:

Returns the ith column name generated by FindCols or GetAllCols.

_________________________________________________________________________________

NExchanges

Property Get NExchanges() as Long

Description:

Returns the number of exchanges generated by GetAllExchanges. The exchanges themselves are returned by theExchange property.

_________________________________________________________________________________

Exchange

Property Get Exchange(ByVal i as Long) as String

Description:

Returns the ith exchange generated by GetAllExchanges.

_________________________________________________________________________________

NTimeZones

Property Get NTimeZones() as Long

Description:

Returns the number of time zones generated by GetAllTimeZones. The time zones themselves are returned by theTimeZone property.

_________________________________________________________________________________

TimeZone

Property Get TimeZone(ByVal i as Long) as String

Description:

Returns the ith time zone generated by GetAllTimeZones.


391

_________________________________________________________________________________

GetAllRels

Sub GetAllRels()

Description:

Retrieves all descendants of the relation specified. The number of relations in the answer set can be determined usingthe NRels property. The relations themselves can be accessed using Rel.

Before calling this function, make sure the root relation has been established, and, if desired, the search filter optionhas been specified.

_________________________________________________________________________________

GetAllCols

Sub GetAllCols()

Description:

Retrieves all descendants of the column specified. The number of columns in the answer set can be determined usingthe NCols property. The columns themselves can be accessed using Col.

Before calling this function, make sure the root column has been established, and, if desired, the search filter optionhas been specified.

_________________________________________________________________________________

GetAllExchanges

Sub GetAllExchanges()

Description:

Retrieves all exchanges in the database. The number of exchanges in the answer set can be determined using theNExchanges property. The exchanges themselves can be accessed using Exchange.

_________________________________________________________________________________

GetAllTimeZones

Sub GetAllTimeZones()

Description:

Retrieves all time zones in the database. The number of time zones in the answer set can be determined using theNTimeZones property. The time zones themselves can be accessed using TimeZone.

392


_________________________________________________________________________________

FindRels

Sub FindRels()

Description:

The number of relations in the answer set can be determined using the NRels property. The relations themselves canbe accessed using Rel.

Before calling this function, make sure the search pattern has been established, and any desired options (casesensitivity, search by name/description and search filter) have been specified.

_________________________________________________________________________________

FindCols

Sub FindCols()

Description:

The number of columns in the answer set can be determined using the NCols property. The columns themselves canbe accessed using Col.

Before calling this function, make sure the search pattern has been established, and any desired options (casesensitivity, search by name/description and search filter) have been specified.

_________________________________________________________________________________

PrintRels

Sub PrintRels()

Description:

Prints information about the relations stored in the database to the given output file.

Before calling this function, make sure the root relation and output file have been established, and any desired options(maximum depth, all contracts flag, merge mode, printer flag and verbose flag) have been specified.

_________________________________________________________________________________

PrintCols

Sub PrintCols()

Description:

Prints information about the columns stored in the database to the given output file.


393

Before calling this function, make sure the root column and output file have been established.

XMIMTable

OverviewXMIMTable provides a facility to store and retrieve non-time series data organized as relational tables. This datamay or may not be associated with time series data stored in the Commodity DataServer database. Entries or rowmembers of a table are referred to as tuples and each tuple can have multiple components corresponding to columnsor fields in the table.

Create XMIMTable objects using New_XMIMTable. Invoke the Init method before using any other properties ormethods. To dispose of an instance, let it go out of scope, or set it to Nothing.

Use Clear to reset a table object to its default state.

This object class provides functions for creating, deleting or renaming tables. Functions are also available for adding ordeleting tuples to/from a table. You must lock the database before using any of these functions. Additionally, functionsare provided for opening an existing table, obtaining table names and field information, and reading tuples from a table.These functions do not require a database lock.

Before creating, deleting or renaming a table using NewTable, Rename or Delete, the table name must be providedvia the Name property. In addition, before creating a table, the NFields, FieldName, FieldType, FieldLength,KeyField and SingleValP properties must be set as appropriate to describe the fields of the new table. Using theNewTable method has the affect of opening the table as well.

To read, add or delete tuples from an existing table or access information about the table attributes, the table mustfirst be opened via OpenTable. Before invoking this method, the Name property must be used to identify the table.Also, the ReadOnlyP property must be used to indicate whether the table is being opened for read only or also forwriting. By default, a table will be opened for read only.

Only one table at a time can be opened and, therefore, accessed. An open table should be closed using theCloseTable method when access to the table is completed. Note that the NewTable method implicitly opens thenew table created so the table should be closed after using this method as well.

Once a table has been opened, tuples can be added, deleted or accessed with the AddTuple, DeleteTupleor GetTuple methods. The tuple to be added, deleted or accessed must be identified prior to the use of one ofthese methods by supplying values for all components of the tuple. Thus, for each field specified when the tablewas created, one of the following must be used to set the corresponding tuple component and the values givenmust correspond to the correct field data type specified at table creation: TupleFieldWild, TupleFieldInt,TupleFieldFloat, TupleFieldDouble, TupleFieldString, TupleFieldAtom, TupleFieldBlock,TupleFieldRel, TupleFieldCol, TupleFieldDate or TupleFieldTime.

If the TupleFieldWild property is set then any value for the corresponding field will match the search.

394


Note that wildcards can only be specified when a tuple is being deleted or accessed so theTupleFieldWild property is not appropriate to use when adding a tuple; a tuple being added mustbe fully specified. If a more than one tuple matches the given pattern then, if DeleteTuple is used,all tuples will be deleted and, if GetTuple is used, each invocation will return a matching tuple until allhave been retrieved.

The tuple returned at each invocation of GetTuple is accessed using the tuple field value properties enumeratedabove. The TupleFieldWild property will, of course, not be relevant in this context.

The GetFieldsInfo and GetAllTableNames methods can be used to return field information pertaining to theopen table or all existing table names, respectively. The field information returned is accessed using the NFields,FieldName, FieldType, FieldLength, KeyField and SingleValP properties. The table names returned are accessed usingthe NTables and TableName properties.

XMIMTable Reference_________________________________________________________________________________

Init


Description:

Establishes a link between the XMIMTable object and the MIM server. Call this method before using any otherproperties or methods.

_________________________________________________________________________________

Clear

Sub Clear()

Description:


_________________________________________________________________________________

Name

Property Let Name(s as String) Property Get Name() as String

Description:

Sets or retrieves the table name.


395

_________________________________________________________________________________

NFields Property Let NFields(ByVal n as Long) Property Get NFields() as Long

Description:

Sets or retrieves the number of columns in a table, that is, the number of fields in each tuple of a table. Used inconjunction with the FieldName, FieldType and FieldLength properties to describe the tuples for a table.

_________________________________________________________________________________

FieldName Property Let FieldName(ByVal i as Long, s as String) Property Get FieldName(ByVal i as Long) as String

Description:

Sets or retrieves the name for field i of a table. Used in conjunction with the FieldType and FieldLengthproperties to describe the tuples for a table. The NFields property gives the number of fields in a table.

_________________________________________________________________________________

FieldType Property Let FieldType(ByVal i as Long, ByVal t as Long) Property Get FieldType(ByVal i as Long) as Long

Description:

Sets or retrieves the type for field i of a table. The possible field types include numbers (integer, float or double),dates, times, string (null-delimited), atoms, blocks (character buffers), Commodity DataServer relations, andCommodity DataServer columns. The value of t should be a member of the TableFieldType family of XMIMUtilsconstants.

The NFields property is used in conjunction with the FieldName and FieldLength properties to describe thetuples for a table. The NFields property gives the number of fields in a table.

_________________________________________________________________________________

FieldLength Property Let FieldLength(ByVal i as Long, ByVal n as Long) Property Get FieldLength(ByVal i as Long) as Long

Description:

Sets or retrieves the length in number of bytes for field i of a table. This property is only applicable if thecorresponding field type is block. When a field of that type is used, it is imperative that every tuple stored in the table

396


has precisely the specified number of bytes for that field because, since the data is interpreted, the number of bytesspecified will be read regardless.

The NFields property is used in conjunction with the FieldName and FieldType properties to describe the tuplesfor a table. The NFields property gives the number of fields in a table.

_________________________________________________________________________________

KeyField

Property Let KeyField(ByVal n as Long) Property Get KeyField() as Long

Description:

Sets or retrieves the field to be used as the key, that is, for which retrieval will be optimized. This number is zero-basedwith the first field used as the default key.

_________________________________________________________________________________

SingleValP

Property Let SingleValP(ByVal n as Boolean) Property Get SingleValP() as Boolean

Description:

Sets or retrieves an indication of whether or not the key field should be unique among the tuples of a table. Ifuniqueness is indicated (True), as tuples are added to a table, if there is an existing tuple having the same key fieldvalue, the old tuple will be deleted and replaced by the new tuple. This is the default behavior.

_________________________________________________________________________________

ReadOnlyP

Property Let ReadOnlyP(ByVal b as Boolean) Property Get ReadOnlyP() as Boolean

Description:

Sets or retrieves an indication of whether a table should be opened for reading only or also for writing. By default, atable will be opened for reading only.

_________________________________________________________________________________

InheritField

Property Let InheritField(ByVal n as Long) Property Get InheritField() as Long

Description:

397

If one of the fields returned by the GetTuple method is is a relation or column, the Commodity DataServer hierarchycan be used for inheritance. The InheritField property can be set to the field (zero-based such that the first fieldis 0) that is to be inherited. So, for example, if there is a table of <relation, string> tuples and an entry exists forEquities, a lookup can be done on IBM with the inheritance argument set to 0 (to indicate the first field) and the valuefor Equities will be returned since it is the closest ancestor of IBM with a value stored in the table.

Note that the search will be much faster if the field to be inherited is also specified as a key. Inheritancemay be used on non-key fields but this is not recommended as the search will be extremely slow. Bydefault, the InheritField will be -1 to indicate that inheritance is not to be used.

_________________________________________________________________________________

CacheSize

Property Let CacheSize(ByVal n as Long) Property Get CacheSize() as Long

Description:

The CacheSize property can be used to optimize the handling of passing tuples from the server to the client. Itindicates the number of tuple that will be shipped at once from the server. The tuples are then cached on the clientside and each subsequent call to retrieve a tuple, using GetTuple, will either get a tuple from the cache or, whenthe cache is exhausted, make another RPC call to the server to fill the cache and then get a tuple from it. All of thisis transparent to the user and, by default, the cache size is 1 such that each call to retrieve a tuple will correspond toa RPC call to the server. For some applications (those involving many tuples being retrieved) it is far more efficient tograb many tuples at once from the server and cache them on the client side.

_________________________________________________________________________________

TupleFieldWild

Property Let TupleFieldWild(ByVal i as Long, ByVal n as Long) Property Get TupleFieldWild(ByVal i as Long) as Long

Description:

Sets or retrieves the flag used to indicate whether field i should be treated as a wildcard instead of having an explicitvalue to match. When adding tuples, of course, this is not appropriate so wildcard should be turned off. It is only usefulwhen retrieving or deleting tuples.

_________________________________________________________________________________

TupleFieldInt

Property Let TupleFieldInt(ByVal i as Long, ByVal n as Long) Property Get TupleFieldInt(ByVal i as Long) as Long

Description:

398


Sets or retrieves an integer value for field i of a tuple being retrieved, deleted or added. If retrieval or deletion is to beperformed, then this serves to limit the tuples returned to only those with a matching integer value for the specifiedfield. If a tuple is being added to a table, then this serves to set the value for the specified component of the tuple.

_________________________________________________________________________________

TupleFieldFloat

Property Let TupleFieldFloat(ByVal i as Long, ByVal f as Single) Property Get TupleFieldFloat(ByVal i as Long) as Single

Description:

Sets or retrieves a floating point value for field i of a tuple being retrieved, deleted or added. If retrieval or deletion isto be performed, then this serves to limit the tuples returned to only those with a matching float value for the specifiedfield. If a tuple is being added to a table, then this serves to set the value for the specified component of the tuple.

_________________________________________________________________________________

TupleFieldDouble

Property Let TupleFieldDouble(ByVal i as Long, ByVal d as Double) Property Get TupleFieldDouble(ByVal i as Long) as Double

Description:

Sets or retrieves a double precision value for field i of a tuple being retrieved, deleted or added. If retrieval or deletionis to be performed, then this serves to limit the tuples returned to only those with a matching double value for thespecified field. If a tuple is being added to a table, then this serves to set the value for the specified component of thetuple.

_________________________________________________________________________________

TupleFieldString

Property Let TupleFieldString(ByVal i as Long, ByVal s as String) Property Get TupleFieldString(ByVal i as Long) as String

Description:

Sets or retrieves a string value for field i of a tuple being retrieved, deleted or added. If retrieval or deletion is to beperformed, then this serves to limit the tuples returned to only those with a matching string value for the specifiedfield. If a tuple is being added to a table, then this serves to set the value for the specified component of the tuple.

Note that the string must be null-delimited.


399

_________________________________________________________________________________

TupleFieldAtom

Property Let TupleFieldAtom(ByVal i as Long, ByVal s as String) Property Get TupleFieldAtom(ByVal i as Long) as String

Description:

Sets or retrieves an atomic value for field i of a tuple being retrieved, deleted or added. If retrieval or deletion is to beperformed, then this serves to limit the tuples returned to only those with a matching atomic value for the specifiedfield. If a tuple is being added to a table, then this serves to set the value for the specified component of the tuple. Anatom is exactly like a string except that specifying it as an atom identifies the string as one which is expected to beused over and over again and, therefore, the Commodity DataServer can handle it optimally such that all occurrencesare shared. That is, the Commodity DataServer interns the string into the database by mapping it to an integer.

_________________________________________________________________________________

TupleFieldBlock

Property Let TupleFieldBlock(ByVal i as Long, ByVal s as String) Property Get TupleFieldBlock(ByVal i as Long) as String

Description:

Sets or retrieves a character buffer for field i of a tuple being retrieved, deleted or added. If retrieval or deletion is tobe performed, then this serves to limit the tuples returned to only those with a matching block value for the specifiedfield. If a tuple is being added to a table, then this serves to set the value for the specified component of the tuple.

A block is an uninterpreted number of bytes. When the field descriptions are set up, a field of this type must have alength in number of bytes specified for it. It is important that every tuple stored has precisely the specified number ofbytes for that field because, since the data is uninterpreted, the number of bytes will be read regardless.

_________________________________________________________________________________

TupleFieldRel

Property Let TupleFieldRel(ByVal i as Long, ByVal s as String) Property Get TupleFieldRel(ByVal i as Long) as String

Description:

Sets or retrieves a Commodity DataServer relation name for field i of a tuple being retrieved, deleted or added. Ifretrieval or deletion is to be performed, then this serves to limit the tuples returned to only those with a matchingrelation name for the specified field. If a tuple is being added to a table, then this serves to set the value for thespecified component of the tuple.

The relation name is specified as a string. Having identified it as a relation serves to allow the Commodity DataServerto optimize storage and facilitates the use of the Commodity DataServer schema inheritance mechanism.

400


_________________________________________________________________________________

TupleFieldCol

Property Let TupleFieldCol(ByVal i as Long, ByVal s as String) Property Get TupleFieldCol(ByVal i as Long) as String

Description:

Sets or retrieves an Commodity Query column name for field i of a tuple being retrieved, deleted or added. If retrievalor deletion is to be performed, then this serves to limit the tuples returned to only those with a matching column namefor the specified field. If a tuple is being added to a table, then this serves to set the value for the specified componentof the tuple.

The column name is specified as a string. Having identified it as a column serves to allow Commodity Query tooptimize storage and facilitates the use of the Commodity Query schema inheritance mechanism.

_________________________________________________________________________________

TupleFieldDate

Property Let TupleFieldDate(ByVal i as Long, ByVal d as Date) Property Get TupleFieldDate(ByVal i as Long) as Date

Description:

Sets or retrieves a date for field i of a tuple being retrieved, deleted or added. If retrieval or deletion is to be performed,then this serves to limit the tuples returned to only those with a matching date for the specified field. If a tuple is beingadded to a table, then this serves to set the value for the specified component of the tuple.

_________________________________________________________________________________

TupleFieldTime

Property Let TupleFieldTime(ByVal i as Long, ByVal t as Date) Property Get TupleFieldTime(ByVal i as Long) as Date

Description:

Sets or retrieves a time for field i of a tuple being retrieved, deleted or added. If retrieval or deletion is to beperformed, then this serves to limit the tuples returned to only those with a matching time for the specified field. If atuple is being added to a table, then this serves to set the value for the specified component of the tuple.


401

_________________________________________________________________________________

NTables

Property Get NTables() as Long

Description:

Retrieves the number of tables in the database. Used in conjunction with TableName.

_________________________________________________________________________________

TableName

Property Get TableName(ByVal i as Long) as String

Description:

Retrieves the name for table i in the database. Used in conjunction with the NTables property.

_________________________________________________________________________________

NewTable

Sub NewTable()

Description:

Creates and adds a new table to the database.

Before calling this function, make sure that the description of the tuples for the table have been established usingthe NFields, FieldName, FieldType, and FieldLength properties, and that desired options have also beenspecified (KeyField and SingleValP properties).

_________________________________________________________________________________

OpenTable

Sub OpenTable()

Description:

Opens an existing table in the database.

Before calling this function, make sure that the table name has been established using the Name property, and that, ifdesired, the ReadOnlyP property has been specified.

402


_________________________________________________________________________________

CloseTable Sub CloseTable()

Description:

Closes a table opened with either OpenTable or NewTable.

_________________________________________________________________________________

Rename Sub Rename(newname as String)

Description:

Renames an existing table in the database.

Before calling this function, make sure that the existing table name has been established using the Name property.

_________________________________________________________________________________

Delete Sub Delete()

Description:

Deletes an existing table in the database.

Before calling this function, make sure that the table name has been established using the Name property.

_________________________________________________________________________________

GetFieldsInfo Sub GetFieldsInfo()

Description:

Retrieves information on the fields of the currently open table. The field information returned is accessed using theNFields, FieldName, FieldType, FieldLength, KeyField and SingleValP properties.

_________________________________________________________________________________

GetAllTableNames Sub GetAllTableNames()

Description:

Retrieves the names of all tables existing in the database. The table names returned are accessed using the NTablesand TableName properties.


403

_________________________________________________________________________________

AddTuple Sub AddTuple()

Description:

Adds a tuple to the currently open table.

Before calling this function, make sure that the tuple to be added has been defined by providing values for allcomponents of the tuple using the appropriate TupleField properties.

_________________________________________________________________________________

GetTuple Sub GetTuple()

Description:

Retrieves a tuple from the currently open table. The TupleField properties are used to fetch the components of theretrieved tuple.

Before calling this function, make sure that the tuple pattern to be retrieved has been defined by providing a valueor wildcard indication for all components of the tuple using the appropriate TupleField properties. This patternis typically set and then GetTuple invoked repeatedly until all tuples matching that pattern have been retrieved.Exception handling should be used to detect if the tuples are exhausted.

_________________________________________________________________________________

DeleteTuple Sub DeleteTuple()

Description:

Deletes one or more tuples from the currently open table.

Before calling this function, make sure that the tuple pattern to be deleted has been defined by providing a value orwildcard indication for all components of the tuple using the appropriate TupleField properties.

XMIMCurrentTick

OverviewXMIMCurrentTick provides a mechanism for high-frequency updating of tick data. The Commodity DataServermaintains this data separately from the historical tick data. As this is intended to be the latest, most current tick data,the data store is referred to as the current tick store.

404


To efficiently handle high-frequency updating (such as necessary with data coming from an on-line tick feed), thetime of insertion into the database has been minimized as a first priority. This is in conflict with the goals for storinggeneral historical data whereby the data is stored such that it can be efficiently retrieved, while minimizing storagerequirements. Hence, the need for two distinct data stores, one for historical data and one for high-frequency data.

Create XMIMCurrentTick objects using New_XMIMCurrentTick. Invoke the Init method before using any otherproperties or methods. To dispose of an instance, let it go out of scope, or set it to Nothing.

Use Clear to reset a current tick object to its default state.

Before any operation which changes the current tick store data is performed, a lock specific to the current tick storemust be obtained using the Locked property. In addition, for a folding operation, both the current tick store and thehistorical database must be locked as data in both stores will be modified.

To load data into the current tick store, first specify the date for which tick data is to be loaded using the DataDateproperty, and the relations and columns to be updated using the NRels, Rel, NCols, and Col properties. TheNRecords property must be set to cause buffer space to be allocated for the new data and the dates and valuesgiven with the DateTime and Val properties. Then the Update method can be invoked to perform the update.

Use the NCtdDates and CtdDate properties to access the dates for which data is stored in the current tick store.

For deleting data from the current tick store or migrating data from the store to the historical database, use theFromDate and ToDate properties to specify the range of current tick data affected. If these properties are not used,then all data in the current tick store will be migrated or deleted.

To delete data from the current tick store, the Remove method can be invoked.

To migrate or fold data from the current tick store into the historical database, either the Fold or SelectiveFoldmethod can be used. Use of the Fold method will result in all data in the current tick store within the specified daterange being folded into the historical data. The SelectiveFold method can be used to fold selected time series.

Prior to using the SelectiveFold method, the relation to be folded and all desired columns should be specified. Inaddition, the type of historical data to fold into can be specified via the FoldIntoTickP, FoldIntoIntradayP andFoldIntoDailyP properties.

Note that more than one of these properties can be set such that data from the current tick store can bemigrated into more than one type of historical data. Aggregation will take place as appropriate during thefolding.

Folding of current tick data will never result in the replacement of historical tick data; only the addition of data to thehistorical database.

Note that the current tick facility can be used for both tick-by-tick and intraday tick data and the systemwill automatically distinguish between the two. The RealTickP property can be used to specify thetype of data being updated, folded or deleted but this is not necessary.


405

During data retrieval, when the historical tick data and current tick data intersect, the Commodity DataServer willalways use the historical data, under the assumption that the historical tick data has been processed and "cleaned",whereas the current tick data is being directly accessed from a potentially noisy feed. Current tick data is retrieved andedited via the same methods used for historical data (see “XMIMRecords and XMIMRecordsDouble Reference” for thedocumentation on the XMIMRecords class).

It should be noted that, due to the high-speed nature of the current tick updating, it is not envisioned that this facilitywill be utilized via VBA API. This class was provided for completeness and consistency with the C API, as well as toprovide documentation for the Visual Basic user regarding the current tick store facility.

XMIMCurrentTick Reference_________________________________________________________________________________


Description:

Establishes a link between the XMIMCurrentTick object and the Commodity DataServer. Call this method beforeusing any other properties or methods.

_________________________________________________________________________________

Clear Sub Clear()

Description:


_________________________________________________________________________________

NRels Property Let NRels(ByVal n as Long) Property Get NRels()as Long

Description:

Allocates or retrieves the specified number of relation names. The strings are initially empty; set them via the Relproperty.

_________________________________________________________________________________

Rel Property Let Rel(ByVal i as Long, s as String)

406


Property Get Rel(ByVal i as Long) as String

Description:

Sets the ith relation name to s, or retrieves the ith relation name.

_________________________________________________________________________________

NCols Property Let NCols(ByVal n as Long) Property Get NCols() as Long

Description:

Allocates the specified number of column names. The names are initially empty; set them via the Col property.

_________________________________________________________________________________

Col Property Let Col(ByVal i as Long, s as String) Property Get Col(ByVal i as Long) as String

Description:

Sets the ith column name to s, or retrieves the ith column name.

_________________________________________________________________________________

NRecords Property Let NRecords(ByVal x as Long) Property Get NRecords() as Long

Description:

When set, grows the buffers used for dates and values to accommodate the number of records specified by x.Existing data is preserved if the size increases, or truncated if it decreases.

Used to specify the number of current tick records to update when set prior to invoking the Update method. Thenumber of records must correspond to the number of times given by the DateTime property.

_________________________________________________________________________________

DataDate Property Let DataDate(ByVal d as Date) Property Get DataDate() as Date

Description:

Sets or returns the date for which current tick data is to be updated. This property is used by the Update method.


407

_________________________________________________________________________________

DateTime

Property Let DateTime(ByVal i as Long, ByVal d as Date) Property Get DateTime(ByVal i as Long) as Date

Description:

Sets or returns the time associated with record i. This property is used to specify the times for the current tickupdates performed when the Update method is used.

_________________________________________________________________________________

Val

Property Let Val(ByVal r as Long, ByVal c as Long, ByVal i as Long, ByVal x as Single) Property Get Val(ByVal r as Long, ByVal c as Long, ByVal i as Long) as Single

Description:

Sets or retrieves the value associated with relation r, column c of record i. This property is used to specify the valuesfor the current tick updates performed when the Update method is used.

_________________________________________________________________________________

FromDate

Property Let FromDate(ByVal d as Date) Property Get FromDate() as Date

Description:

This property sets or fetches the "from date" used by Fold, SelectiveFold and Remove.

If the date range is not specified prior to the current tick update operation, then all relevant data in the current tickstore will be folded into the historical database or deleted. To fold or delete a single day's worth of data, the same dateshould be given for FromDate and ToDate.

_________________________________________________________________________________

ToDate

Property Let ToDate(ByVal d as Date) Property Get ToDate() as Date

Description:

This property sets or fetches the "to date" used by Fold, SelectiveFold and Remove.

408


If the date range is not specified prior to the current tick update operation, then all relevant data in the current tickstore will be folded into the historical database or deleted. To fold or delete a single day's worth of data, the same dateshould be given for FromDate and ToDate.

_________________________________________________________________________________

RealTickP

Property Let RealTickP(ByVal b as Boolean) Property Get RealTickP() as Boolean

Description:

This property is True if tick-by-tick data is being updated, folded or cleared in the current tick store. It is False whenthe affected data in the current tick store is intraday tick data.

Setting this property is actually not necessary since the system can distinguish automatically between the two typesof data. It is included merely for consistency with the C API.

_________________________________________________________________________________

FoldIntoTickP

Property Let FoldIntoTickP(ByVal b as Boolean) Property Get FoldIntoTickP() as Boolean

Description:

If this property is True then data from the current tick store will be folded into the tick-by-tick historical data when theSelectiveFold method is used.

Note that by setting more than one of the FoldIntoTickP, FoldIntoIntradayP orFoldIntoDailyP properties, data can be folded into more than one of the types of historical data bythe same invocation of SelectiveFold. By default, data will be folded into the historical intraday dataonly.

_________________________________________________________________________________

FoldIntoIntradayP

Property Let FoldIntoIntradayP(ByVal b as Boolean) Property Get FoldIntoIntradayP() as Boolean

Description:

If this property is True then data from the current tick store will be folded into the intraday historical data when theSelectiveFold method is used. Aggregation of data will take place as necessary (i.e., if the data being folded istick-by-tick data then it will be aggregated into minutes before folding into the historical intraday data).


409


_________________________________________________________________________________

FoldIntoDailyP

Property Let FoldIntoDailyP(ByVal b as Boolean) Property Get FoldIntoDailyP() as Boolean

Description:

If this property is True then data from the current tick store will be folded into the daily historical data when theSelectiveFold method is used. Aggregation of data will take place as necessary (i.e., the data being folded will beaggregated into days before folding into the historical daily data).


_________________________________________________________________________________

NCtdDates

Property Get NCtdDates() as Long

Description:

Retrieves the number of dates for which current tick data is stored. This property is used in conjunction with theCtdDate property.

_________________________________________________________________________________

CtdDate

Property Get CtdDate(ByVal i as Long) as Date

Description:

Retrieves the ith date for which current tick data is stored. This property is used in conjunction with the NCtdDatesproperty.

_________________________________________________________________________________

Locked

410


Property Let Locked(ByVal b as Boolean) Property Get Locked() as Boolean

Description:

When set to True, attempts to acquire an exclusive lock on the Commodity DataServer current tick store. When setto False, unlocks the current tick store. A successful lock is required before changing the data in the current tickstore (updating, folding or deleting current tick data).

When read, returns the current state of the current tick store locking.

_________________________________________________________________________________

Update Sub Update()

Description:

Updates one or more time series in the current tick data store. This method provides the mechanism for high-frequency updating of tick data in the Commodity DataServer. The current tick store must be locked (via the Lockedproperty) before the Update method can be used.

Before calling this function, make sure that relation names and column names have been established (using theNRels, Rel, NCols and Col properties), that the number of records has been set with the NRecords property, andthat times and values have been associated with records (using the DateTime and Val properties). Also make surethe date for which tick data is to be updated in the current tick store has been set via the DataDate property.

_________________________________________________________________________________

Fold Sub Fold()

Description:

Migrates the current tick data into the historical tick data.

Before the folding operation can be performed, both the current tick store and the historical database must be locked(via the Locked properties in the XMIMCurrentTick and XMIMSchema classes, respectively) since data in bothstores is being modified.

Before calling this function, make sure that the range of current tick data to fold has been set using the FromDate andToDate properties; otherwise, all data in the current tick store will be folded into the historical database.

_________________________________________________________________________________

SelectiveFold Sub SelectiveFold()

Description:


411

Migrates the current tick data into the historical tick data. This function can be used to effect selective folding ofspecific columns of a specific relation. It also facilitates folding of current tick data into one or more of the differenttypes of historical data (daily, intraday and tick) with aggregation as appropriate.

Before the folding operation can be performed, both the current tick store and the historical database must be locked(via the Locked properties in the XMIMCurrentTick and XMIMSchema classes, respectively) since data in bothstores is being modified.

Before calling this function, make sure that relation name and column names have been established (using the NRels,Rel, NCols and Col properties). If the column names are not specified, then all columns of the relation will befolded. If the NRels and Rel properties are used, then NRels must be 1 and only a single relation specified. Thisrelation can not be a category relation. If the relation is not specified then all data will be folded (as when the Foldmethod is used). The range of current tick data to fold should also be set using the FromDate and ToDate properties;otherwise, all data in the current tick store will be folded into the historical database. The type of data to fold into(with appropriate aggregation) should be set, if desired, using the FoldIntoTickP, FoldIntoIntradayP andFoldIntoDailyP properties. More than one of these properties can be set to effect folding into different types ofdata at once using a single call to SelectiveFold.

_________________________________________________________________________________

Remove

Sub Remove()

Description:

Deletes the current tick data. The current tick store must be locked (via the Locked property) before the Removemethod can be used.

Before calling this function, make sure that the range of current tick data to be deleted has been set using theFromDate and ToDate properties; otherwise, all data in the current tick store will be deleted.

XMIMCustomSearch

OverviewYou can retrieve a list of custom searches using XMIMCustomSearch objects.

From a user's perspective, a custom search is a special way to generate lists of symbols in a LET. See “Custom SearchFacility” for a complete description of defining and using custom searches.

The GetAllFunctionNames method can be used to retrieve all the functions available in the custom searchspecification file. The GetFunctionInfo method can be used to retrieve information on a specific function. TheExecuteFunction method can be used to execute a specified function.

412


XMIMCustomSearch Reference_________________________________________________________________________________

Init


Description:

Establishes a link between the XMIMCustomSearch object and the Commodity DataServer. Call this method beforeinvoking any other properties or methods.

_________________________________________________________________________________

Clear

Sub Clear()

Description:


_________________________________________________________________________________

NFunctionNames

Property Get NFunctionNames() as Long

Description:

Returns the number of functions contained in the specification file. Call GetAllFunctionNames before accessingthis property.

_________________________________________________________________________________

NthFunctionName

Property Get NthFunctionName(ByVal i As Long) as String

Description:

Returns the name of function i of the specification file. Call GetAllFunctionNames before accessing this property.

_________________________________________________________________

FunctionName

Property Let FunctionName(s as String)


413

Property Get FunctionName() as String

Description:

Sets or retrieves the name of the function. Determines which function name will be used by the GetFunctionInfoor ExecuteFunction methods.

414


_________________________________________________________________________________

SearchRoot

Property Let SearchRoot(s as String)

Property Get SearchRoot() as String

Description:

Sets or retrieves the root relation. Determines which relation will be used when ExecuteFunction is called. Theroot relation is used to scope the search, so generally it is a category relation. If the search is to include all relations inthe database, the root of the relation hierarchy (TopRelation) must be specified.

_________________________________________________________________________________

NArgs

Property Let NArgs(ByVal i As Long)

Property Get NArgs() As Long

Description:

Sets or retrieves the number of arguments used by a specified function.

_________________________________________________________________________________

Arg

Property Let Arg(ByVal i As Long, Name As String)

Property Get Arg(ByVal I As Long) As String

Description:

Sets or retrieves the value for each argument used by a specified function. The number of arguments must be setusing NArgs before accessing this property.

_________________________________________________________________________________

ArgDesc

Property Get ArgDesc(ByVal i As Long) as String

Description:

Returns the description for the argument i of the specified function. GetFunctionInfo must be called beforeaccessing this property.

_________________________________________________________________________________


415

ArgType

Property Get ArgType(ByVal i As Long) as Long

Description:

Returns the data type for the argument i of the specified function. GetFunctionInfo must be called beforeaccessing this property.

_________________________________________________________________________________

NResults

Property Get NResults () as Long

cust.ExecuteFunction

numResults = cust.NResults

Description:

Returns the number of results returned after executing a function.

_________________________________________________________________________________

Result

Property Get Result (ByVal i as Long) as String

Description:

Returns result value i after executing a function.

_________________________________________________________________________________

GetFunctionInfo

Sub GetFunctionInfo()

Description:

Returns all the information for the specified function contained in the specification file: the number of arguments, datatype for arguments and descriptions for arguments.

_________________________________________________________________________________

GetAllFunctionNames

Sub GetAllFunctionNames()

Description:

416


Returns all the function names available in the specification file.

________________________________________________________________________________

ExecuteFunction

Sub ExecuteFunction() cust.FunctionName = "cg_regex" cust.NArgs = 1 cust.Arg(0) = "ABC$" cust.SearchRoot = "TopRelation:Equities" cust.ExecuteFunction numResults = cust.NResults

Description:

Executes the specified function. The function name, relation root, the number of arguments and arguments must bespecified before calling this function.

XMIMUtils (Constants)

OverviewThere are several families of constants used by VBA API to specify options like execution granularity, relation types,and relcol aggregation rules. The constants correspond to ones used in the Commodity DataServer C-language API.

Since Visual Basic does not have an enumerated type, VBA API exposes the constants as read-only properties of theXMIMUtils class. To use the constants, you will need to create an instance of XMIMUtils.

See “XMIMUtils (Constants) Reference” below for a brief description of each family of constants.

Note that there are methods for parsing and unparsing constant names for each VBA API constant family(for instance, StringToUnits and UnitsToString). See “XMIMUtils (Constants) Reference” belowfor details.


417

XMIMUtils (Constants) Reference________________________________________________________________________________

Units

Overview

Indicates the granularity of queries for XMIMRecords and XMIMShowWhen objects. For instance, using Weeks causesan XMIMRecords query to fetch weekly time series data.

Members

Seconds Second-by-second. Minutes Minute-by-minute. Hours Hourly. Days Daily. Weeks Weekly. Months Monthly. Quarters Quarterly.Years Annually.

________________________________________________________________________________

LimitMode

Overview

Used to limit data retrieval results.

Members

LimitByRecords Limit number of records returned. LimitByMemory Limit amount of memory used.

________________________________________________________________________________

FillOption

Overview

Determines how NaN values are treated in query answer sets.

Members

FillNaN Pass NaNs through unchanged. FillForward Replace with nearest earlier non-NaN value. FillBackward Replace with nearest later non-NaN value. FillInterpLin Linear interpolation between nearest available values. FillInterpGeo Geometric interpolation.FillInterpLog Logarithmic interpolation.

418


________________________________________________________________________________

CurrentTickUsage

Overview

Indicates whether historic data retrieved should be supplemented with current tick data or not. Also used to indicatewhether current tick data should be updated along with historic data or not.

Members

AppendCTDToAll Append current tick data to all appropriate historic data/update current tick data.AppendCTDToNone Do not use current tick data. AppendCTDToDaily Append current tick data to historic daily data.AppendCTDToTick Append current tick data to historic tick (intraday and tick-by-tick) data.

________________________________________________________________________________

RelType

Overview

Indicates the type of a relation.

Members

RelInvalid Invalid type. RelCategory Interior hierarchy node (i.e., a relation with children). Will not have data associated. RelNormal Ordinary leaf node (i.e., a relation with no children). RelFutures The default continuous contract for a future. This is also the parent for all continuous and individual contracts for the future. RelFuturesContract An individual futures contract. RelFuturesContinuous A synthetic continuous contract. RelOptions An option data series.

________________________________________________________________________________

ColType

Overview

Indicates the type of a column.

Members

ColInvalid Invalid type. ColCategory Interior hierarchy node (i.e. a column with children).


419

ColNormal Ordinary leaf node (i.e. a column with no children). ColComposite Multi-field column.

420


________________________________________________________________________________

RelColType

Overview

Indicates the storage mechanism for a relcol data object.

Members

RelColInvalid Invalid type. RelColBase Dense time series. RelColConstant Constant time series.RelColSparse Sparse time series.

________________________________________________________________________________

RelColObjType

Overview Indicates the kind of records in a relcol data object.

Members

RelColObjInvalid Invalid type.RelColObjFloat 32-bit floating point. RelColObjDouble 64-bit floating point. RelColObjInt 32-bit integer. RelColObjDate Commodity Query date representation. RelColObjTime Commodity Query time representation.RelColObjBoolean Single-bit values.

________________________________________________________________________________

RelColAggr

Overview

Specifies the aggregation policy when a fine-grained time series is retrieved at a coarser granularity. For instance, ifshare volume data is stored on an hourly basis, but the data is viewed on a daily basis, then the value for a day shouldbe the sum of the individual hourly data points.

Members

RelColAggrInvalid Invalid policy. RelColAggrOpen Use the first value in the interval. RelColAggrHigh Use the last value. RelColAggrLow Use the minimum value. RelColAggrClose Use the maximum value. RelColAggrSum Add all values in the interval.RelColAggrAverage Average all values.


421

________________________________________________________________________________

RelColDelta

Overview

Determines the minimum increment for data in a time series. So, an increment of 1/32 would mean a data point mustalways be a multiple of 1/32.

Note that this only affects internal Commodity DataServer storage; the Commodity DataServer alwaysconverts values to floating-point before passing it to the VBA API.

MembersRelColDeltaInvalid Invalid increment.RelColDeltaNone No restriction. RelColDelta8ths Value must be a multiple of 1/8. RelColDelta16ths Multiple of 1/16. RelColDelta32nds Multiple of 1/32. RelColDelta64ths Multiple of 1/64.

________________________________________________________________________________

RolloverDataType

Overview

Determines when rollover is applied to a relcol.

MembersRolloverNone No rollover. RolloverDaily Compute rollover for daily data. RolloverTick Compute for tick data.RolloverBoth Compute for daily and tick data.

________________________________________________________________________________

AdjustedStudy

Overview

Determines the type of study performed in a rollover calculation.

MembersAdjStudyNone No study.AdjStudyMove Absolute move. AdjStudyPctMove Percentage move. AdjStudyAvg Average.

422


AdjStudySum Sum. AdjStudyCount Count. AdjStudyHigh High. AdjStudyLow Low. AdjStudyStdDev Standard deviation.AdjStudyVariance Variance.

________________________________________________________________________________

DataType

Overview

Determines the type of data to be used.

Members

TickData Real tick data. IntradayData Intraday tick data. DailyData Daily data. AllData All data types.

_________________________________________________________________________________

SearchFilter

Overview

Determines which nodes of the relation or column hierarchies should be searched. Leaves in the hierarchy are definedto be those that have (or could have) data associated with them.

Members

SearchLeaves Search for leaves in the hierarchy. SearchNonLeaves Search for categories only. SearchAll Search all nodes.

_________________________________________________________________________________

MergeMode

Overview

Determines whether information should be appended to or replace an existing file. In the case of use with theReadFacts method, this is used to indicate to overwrite (replace) or append to (update) existing data.

Members

MergeModeReplace Overwrite existing file/data. MergeModeAppend Append to existing file/data.


423

_________________________________________________________________________________

Verbosity

Overview

Determines whether the query text and options should be included with execution results.

MembersSilent Do not include query text. Verbose Include query text. VeryVerbose Include query text and user options.

________________________________________________________________________________

TickMode

Overview

Determines the type of data to be read in (when other than daily data is being read) and whether the system shouldautomatically aggregate it or not.

MembersTickPreSampled Intraday data. TickByTick Real Tick data. TickConvert Real Tick converted to intraday.

_________________________________________________________________________________

RecordFormat

Overview

Determines whether the data should be read according to fixed format specifications or not.

MembersFreeFormat Delimit by comma or spaces. FixedFormat Fortran-style fixed format specifications.

_________________________________________________________________________________

FieldType

Overview

Specifies the type of field to be written out or read.

MembersDateField Date string field.

424


TimeField Time string field. RelField Relation name field. ColField Data values (Column) field. ExpDateField Expiration Date string field. OptTypeField Options Type field.StrikeField Strike Price value field. FillerField Filler (blank spaces) field.

________________________________________________________________________________

DateFormat

Overview

Specifies the date format for reading or writing facts.

Members

DateFormat_mmsddsyy 03/10/84 DateFormat_mmsddsyyyy 03/10/1984 DateFormat_ddsmmsyy 10/03/84 DateFormat_ddsmmsyyyy 10/03/1984 DateFormat_mm_dd_yy 03-10-84 DateFormat_mm_dd_yyyy 03-10-1984 DateFormat_dd_mm_yy 10-03-84 DateFormat_dd_mm_yyyy 10-03-1984 DateFormat_yymmdd 840310 DateFormat_yyyymmdd 19840310 DateFormat_mon_dd_yy Mar 10, 84 DateFormat_mon_dd_yyyy Mar 10, 1984 DateFormat_monp_dd_yy Mar. 10, 84 DateFormat_monp_dd_yyyy Mar. 10, 1984 DateFormat_month_dd_yy March 10, 84 DateFormat_month_dd_yyyy March 10, 1984DateFormat_dd_mon_yy 10-Mar-84 DateFormat_dd_mon_yyyy 10-Mar-1984DateFormat_dd_month_yy 10-March-84 DateFormat_dd_month_yyyy 10-March-1984

_________________________________________________________________________________

ExpDateFormat

Overview

Specifies the expiration date format for reading or writing facts.

Members

DateFormat_mmsyy 03/84 DateFormat_mmsyyyy 03/1984DateFormat_mm_yy 03-84 DateFormat_mm_yyyy 03-1984 DateFormat_yymm 8403


425

DateFormat_yyyymm 198403 DateFormat_monsp_yy Mar 84 DateFormat_monsp_yyyy Mar 1984 DateFormat_monthsp_yy March 84 DateFormat_monthsp_yyyy March 1984 DateFormat_monp_yy Mar. 84 DateFormat_monp_yyyy Mar. 1984 DateFormat_mon_yy Mar-84 DateFormat_mon_yyyy Mar-1984 DateFormat_month_yy March-84DateFormat_month_yyyy March-1984

________________________________________________________________________________

TimeFormat

Overview

Specifies the time format for reading or writing facts.

MembersTimeFormat_hh_mm_am 9:05am TimeFormat_hh_mm_am_u 11:15AM TimeFormat_hhmm_am 905am TimeFormat_hhmm_am_u 1115AMTimeFormat_hh_mm 09:05 TimeFormat_hhmm 2315 TimeFormat_hh_mm_ss_am 9:05:01am TimeFormat_hh_mm_ss_am_u 11:15:01AM TimeFormat_hhmmss_am 90501am TimeFormat_hhmmss_am_u 111501AM TimeFormat_hh_mm_ss 09:05:01TimeFormat_hhmmss 231501

________________________________________________________________________________

OptionType

Overview

Determines the type of options to be considered for retrieval.

MembersCallOpt Call Option.PutOpt Put Option.AllOpt Both Call and Put Options.

________________________________________________________________________________

TableFieldType

Overview

426


Specifies the possible data types for the fields of a table.

MembersTabFieldInt IntegerTabFieldFloat FloatTabFieldDouble Double TabFieldDate Date TabFieldTime Time TabFieldStr Null-delimited string TabFieldAtom Atom TabFieldBlk Block (character buffer) TabFieldRel Commodity Query Relation TabFieldCol Commodity Query Column

XMIMUtils (Utilities)

OverviewThe VBA API library includes a number of utility routines for things like date conversion, NaN comparison, and parsing/unparsing constants. Most of these have been mentioned earlier; this chapter provides additional information aboutthem.

The routines described in this chapter are methods of the XMIMUtils class. To use them, you must first create aninstance of XMIMUtils.

IsNan, IsMissDataNaN, and IsHolidayNaN can be used to determine whether a value is a NaN. (Each functioncompares its argument with a cached NaN value, so there is no network traffic or other overhead involved.)

The NoDate property can be used to retrieve the value for missing or invalid dates. This is occasionally useful for date-valued attributes of other VBA API objects. For example, the StartTrade field of an XMIMRelCol object could beset to the value of XMIMUtils.NoDate if no value is known.

Finally, there are several routines for parsing and unparsing constants used by VBA API. These can be helpful whenbuilding user interfaces that manipulate parameters of VBA API objects. Each parsing routine converts a string to aconstant value, while the unparsing routines do the reverse conversions. A typical use is to fill a list box with stringscorresponding to constants in a family by unparsing each constant value into a descriptive string; when a string in thelist box is selected, it can be parsed back into a constant value.

XMIMUtils (Utilities) Reference________________________________________________________________________________

IsNaN Function IsNaN(x as Single) as Boolean


427

Description:

Returns True if x is a missing-data NaN or a holiday NaN, and False otherwise.

428


_________________________________________________________________________________

IsMissDataNaN Function IsMissDataNaN(x as Single) as Boolean

Description:

Returns True if x is a missing-data NaN, and False otherwise.

_________________________________________________________________________________

IsHolidayNaN Function IsHolidayNaN(x as Single) as Boolean

Description:

Returns True if x is a holiday NaN, and False otherwise.

_________________________________________________________________________________

NoDate Property Get NoDate( ) as Date

Description:

Retrieves the value for missing or invalid dates.

_________________________________________________________________________________

StringTo<type>where <type> = Units, LimitMode, FillOption, CurrentTickUsage, RelType, ColType, RelColType, RelColObjType, RelColAggr, RelColDelta, RolloverDataType, AdjustedStudy, DataType, SearchFilter, MergeMode, Verbosity, TickMode, RecordFormat, FieldType, DateFormat, ExpDateFormat, TimeFormat, OptionType, or TableFieldType.

Function StringTo<type>(s as String) as Long

Description:

Converts a string to a <type> constant value. Inverse of <type>ToString. The function is case-insensitive andignores all whitespace. It recognizes strings produced by <type>ToString. Returns the string <invalid> if x is nota valid member of <type>.

________________________________________________________________________________

<type>ToString

429

where <type> = Units, LimitMode, FillOption, CurrentTickUsage, RelType, ColType, RelColType, RelColObjType, RelColAggr, RelColDelta, RolloverDataType, AdjustedStudy, DataType, SearchFilter, MergeMode, Verbosity, TickMode, RecordFormat, FieldType, DateFormat, ExpDateFormat, TimeFormat, OptionType, or TableFieldType.

Function <type>ToString(x as Long) as String

Description:

Converts a <type> constant value to a string. Inverse of StringTo<type>. If the string cannot be parsed, returnsthe type's Invalid constant (e.g., RelInvalid), or -1 if there isn't one.

Commodity DataServer Visual Basic for Applications APIExamples

IntroductionThis following includes example code for programming with the Commodity DataServer Visual Basic for ApplicationsAPI.

Referencing the VBA API LibraryBefore an application can access VBA API, it must establish a reference to the library. The way this is done might varyfrom application to application. For Visual Basic and Excel, select References from the Tools menu. (In Excel, youmust activate a macro module to make the References selection visible.) This will bring up a dialog box containing alist of all the OLE servers in the system. Assuming you correctly installed the VBA API, you should see an entry such asVBA API. Click on the check box to select the library, and then close the dialog box.

Extracting Data: XmimRecordsDouble & getdata()The getdata() subroutine is a simple example of how to use VBA API to extract data from the CommodityDataServer into an Excel worksheet. This example is meant to quickly show how to use VBA API and how to read datawith XmimRecordsDouble.

430


getdata() Outline1st – Define the variables.

Example: Dim svr As XmimServer Dim recs As XmimRecordsDouble Dim d As Date Dim OpenPrice As Double Dim ClosePrice As Double Dim n, i As Integer

2nd – Set VBA API variables to objects.

Example: Set svr = New_XmimServer Set recs = New_XmimRecordsDouble

3rd – Use XmimServer object to connect to Commodity Query.

Example: svr.Host = "yourServerName" svr.ServerNum = 0 svr.Connect recs.Init svr

4th – Use XmimRecords object to describe what data is desired.

Example:

recs.Ncols represent how many columns will be used and recs.Col(x) defines each column.

recs.NRels = 1 recs.Rel(0) = "IBM" recs.NCols = 2 recs.Col(0) = "Open" recs.Col(1) = "Close" recs.FromDate = "9/1/1999" recs.GetRecords

5th – Start loop and set previously defined variables into Excel cells.

Example:

recs.NRecords returns the number of records, so loop is created by counting down.


431

n = recs.NRecords i = 0 While n > 0 OpenPrice = recs.val(0, 0, i) ClosePrice = recs.val(0, 1, i) d = recs.DateTime(i) Worksheets("Sheet1").Cells(i + 1, 1).Value = d Worksheets("Sheet1").Cells(i + 1, 2).Value = OpenPrice Worksheets("Sheet1").Cells(i + 1, 3).Value = ClosePrice n = n - 1 i = i + 1 Wend

6th – After loop has finished, disconnect from the server and set object variables to Nothing.

Example: svr.Disconnect Set recs = Nothing Set svr = Nothing

getdata() Subroutine Sub getdata() Dim svr As XmimServer Dim recs As XmimRecordsDouble Dim OpenPrice As Double Dim ClosePrice As Double Dim d As Date Dim n, i As Integer Set svr = New_XmimServer Set recs = New_XmimRecordsDouble svr.Host = "cbot.lim.com" svr.ServerNum = 1 svr.Connect recs.Init svr recs.NRels = 1 recs.Rel(0) = "IBM" recs.NCols = 2 recs.Col(0) = "Open" recs.Col(1) = "Close" recs.FromDate = "9/1/1999" recs.GetRecords n = recs.NRecords i = 0 While n > 0 OpenPrice = recs.val(0, 0, i) ClosePrice = recs.val(0, 1, i) d = recs.DateTime(i) Worksheets("Sheet1").Cells(i + 1, 1).Value = d Worksheets("Sheet1").Cells(i + 1, 2).Value = OpenPrice Worksheets("Sheet1").Cells(i + 1, 3).Value = ClosePrice n = n - 1 i = i + 1 Wend

432


svr.Disconnect Set recs = Nothing Set svr = Nothing End Sub

Replicating XMIM Queries: XmimShowWhenDoubleThis example will show how to read data with XmimShowWhenDouble.

The showwhendouble() subroutine is a simple example of how to use the VBA API API to replicate CommodityQuery SHOW-WHEN queries. This example uses the XmimShowWhenDouble class and sends results to an Excelworksheet.

Consider the simple query: SHOW IBMCLOSE: Close of IBM WHEN Date is after 1995 AND 2 day percent_move of IBM is more than 10

Sub showwhendouble() Outline1st - Create variables for your VBA API objects. Dim svr As XmimServer Dim showwhen As XmimShowWhen Dim timeUnits As XmimUtils Dim ClosePrice As Double Dim d As Date Dim n, i As Integer

2nd - Use the New property to create instances of each of your VBA API objects. Set svr = New_XmimServer Set showwhen = New_XmimShowWhenDouble Set timeUnits = New_XmimUtils

3rd - Connect to the Commodity DataServer. svr.host = "yourservername" svr.ServerNum = 0 svr.Connect

Before using any properties or methods, invoke the Init method showwhen.Init svr

4th - To construct a query, first set the number of attributes, using the NAttrs property, and set the number ofconditions using NConds. In our example, we have 1 attribute and 2 conditions.


433

showwhen.NAttrs = 1 showwhen.NConds = 2

5th - Use the Attr and Cond properties to fill in the attributes and conditions. showwhen.Attr(0) = "Close of IBM" showwhen.Cond(0) = "Date is after 1995" showwhen.Cond(1) = "2 day percent move of IBM is more than 10"

6th - You can also select an execution granularity for the query via Units (type of Units is Seconds, Minutes,etc.) and NUnits (i.e., intervals). Units must be member of the Units family of XMIMUtils constants (Seconds,Minutes, Hours, etc.) In this case, the results will be shown in 15 minute intervals. showwhen.Units = timeUnits.Minutes showwhen.Nunits = 15

7th - To run the query, call the Execute method. This will send the query to Commodity Query, allocate space for theresults and load the results into the XMIMShowWhen object. showwhen.Execute

8th - Use the Nrecords property to determine the number of records that resulted after executing the query n = showwhen.NRecords i = 0

9th - Iterate through each returned record. For each record, use the DateTime property to fetch the according dataand use the Val property to access the individual fields of the record (in this case the closing price).While n > 0 ClosePrice = showwhen.Val(0, i) d = showwhen.DateTime(i) Worksheets("Sheet1").Cells(i + 1, 1).Value = d Worksheets("Sheet1").Cells(i + 1, 2).Value = ClosePrice n = n - 1 i = i + 1Wend

10th - Disconnect from the server and dispose of your instances using Nothing. svr.Disconnect Set showwhen = Nothing Set svr = NothingEnd Sub

showwhendouble() subroutineSub showwhendouble() Dim svr As XmimServer Dim showwhen As XmimShowWhenDouble Dim timeUnits As XmimUtils Dim ClosePrice As Double Dim d As Date Dim n, i As Integer

434


Set svr = New_XmimServer Set showwhen = New_XmimShowWhenDouble Set timeUnits = New_XmimUtils svr.host = "yourservername" svr.ServerNum = 0 svr.Connect showwhen.Init svr showwhen.NAttrs = 1 showwhen.NConds = 2 showwhen.Attr(0) = "Close of IBM" showwhen.Cond(0) = "Date is after 1995" showwhen.Cond(1) = "2 day percent move of IBM is more than 10" showwhen.Units = timeUnits.Minutes showwhen.Nunits = 15 showwhen.Execute n = showwhen.NRecords i = 0 While n > 0 ClosePrice = showwhen.Val(0, i) d = showwhen.DateTime(i) Worksheets("Sheet1").Cells(i + 1, 1).Value = d Worksheets("Sheet1").Cells(i + 1, 2).Value = ClosePrice n = n - 1 i = i + 1 Wend svr.Disconnect Set showwhen = Nothing Set svr = NothingEnd Sub

Accessing the Children of a Relation: XmimSchema XmimSchema allows the user to access the children of a relation and access Information about individual relations.

Accessing Relations and Properties of RelationsThe GetChildren() subroutine is an example of how to use the VBA API API to get information about a relation. Inthis example, the results show all the relations under the root, “Equities:a:aa”. For each relation under that root, itsdescription, exchange traded on and all available columns are displayed.

GetChildren() Outline1st – Define the variables. The XmimSchema object is used to access relations under a specific hierarchy, and anXmimRelation object is used to access information available for each relation under the hierarchy. Dim svr As XmimServer Dim schema As XmimSchema Dim filter As XmimUtils

435

2nd – Set VBA API variables to objects.

Set svr = New_XmimServer Set schema = New_XmimSchema Set filter = New_XmimUtils

3rd – Use the XMIMServer object to connect to Commodity Query and Initialize your XMIMSchema and CommodityDataServer relation objects.

svr.Host = <host> svr.ServerNum = <port> svr.Connect schema.Init svr

4th – In order to retrieve only a portion of the relation hierarchy, use RelRoot to define the portion you want toretrieve. The function GetAllRels gets all the relations that are under the path “"TopRelation:Weather"” and NRelsis used to count the total number of relations returned.

schema.RelRoot = "TopRelation:Weather" schema.SearchFilter = filter.SearchLeaves 'schema.SearchFilter = filter.SearchNonLeaves 'schema.SearchFilter = filter.SearchAll schema.getallrels n = schema.NRels i = 0

5th – Iterate through each of the returned relations after calling GetAllRels. Set relation.Name =schema.Rel(i)to access information pertaining to each relation. The Fetch method gets all the informationassociated with the relation. Now you can access the description, exchange and number of columns associated withthe relation.

While n > 0 Worksheets("Sheet1").Cells(i + 1, 1) = schema.Rel(i) n = n - 1 i = i + 1 Wend

6th – After loop has finished , disconnect from server and set object variables to nothing.

svr.Disconnect Exit Sub

GetChildren() SubroutineSub GetChildren() On Error GoTo ERROR_ Dim svr As XmimServer Dim schema As XmimSchema Dim filter As XmimUtils Set svr = New_XmimServer Set schema = New_XmimSchema Set filter = New_XmimUtils

436


svr.Host = <host> svr.ServerNum = <port> svr.Connect schema.Init svr schema.RelRoot = "TopRelation:Weather" schema.SearchFilter = filter.SearchLeaves 'schema.SearchFilter = filter.SearchNonLeaves 'schema.SearchFilter = filter.SearchAll schema.getallrels n = schema.NRels i = 0 If n = 0 Then Err.Description = "No results for " + schema.RelRoot Err.Raise (1) End If While n > 0 Worksheets("Sheet1").Cells(i + 1, 1) = schema.Rel(i) n = n - 1 i = i + 1 Wend svr.Disconnect Exit SubERROR_: svr.Disconnect MsgBox ("Error " + Err.Number + ": " + Err.Description)End Sub

Relation and Data LoadingThe Commodity DataServer Visual Basic for Applications API can be used to load proprietary data into the CommodityDataServer database. The basic process for loading new data is as follows:

1. Create new relations2. Create new columns3. Associate columns with your relations4. Load data

Relations and columns are organized hierarchically in the database. The client has the option of selecting where theirdata will lie in the hierarchy. In the following example, a new category for storing relations, “NEWDATA” is created. Therelation, “RELATION1” is added underneath this category. The data will be stored in the column, “NewColumn”.

The resulting hierarchy will look like this: TopRelation NEWDATA (category) RELATION1 (symbol) NewColumn (column) Dim svr As XmimServer Dim utils As XmimUtils


437

Sub loadNewData()Dim result As Boolean

Set svr = New_XmimServer Set utils = New_XmimUtils svr.Host = "Your Server Name Here" svr.ServerNum = 0 svr.Connect ‘You must lock the database before using the add, modify, or delete functions. svr.Locked = True If svr.Locked = True Then result = addNewRelation("NEWDATA", "TopRelation", "New Data", utils.RelCategory) result = addNewRelation("RELATION1", "TopRelation:NEWDATA", "Relation 1", utils.RelNormal) result = addNewColumn("NewColumn", "TopColumn:Price", "New Column", utils.ColNormal) result = addRelCol("RELATION1", "NewColumn") result = addData("RELATION1", "NewColumn") End If svr.Locked = False svr.Disconnect Set utils = Nothing Set svr = NothingEnd Sub

Function addNewRelation (relname As String, parentname As String, _description As String,reltype As Long) As Boolean

On Error GoTo relAddError Dim rel As XmimRelation Set rel = New_XmimRelation rel.Init svr rel.Clear

To add a new relation, set the name, parent and initial values for the relation's properties, then use the Add method tocreate the relation in the Commodity DataServer schema .

rel.Name = relname rel.description = description rel.Parent = parentname rel.reltype = reltype rel.Add Set rel = Nothing addNewRelation = True Exit Function relAddError: addNewRelation = False Set rel = Nothing Exit Function

End Function

Function addNewColumn (colname As String, parentname As String, _description As String,coltype As Long)

438


On Error GoTo colAddError Dim col As XmimColumn Set col = New_XmimColumn col.Init svr col.Clear

To add a new column, set the name and initial values for the column's properties, then call Add to actually create thecolumn in the Commodity DataServer. In the databases the column for a relation will not be accessible (visible) untildata is added into this column.

col.Name = colname col.description = description col.Parent = parentname col.coltype = coltype col.Add Set col = Nothing addNewColumn = True Exit Function colAddError: addNewColumn = False Set rel = Nothing Exit Function End Function

Function addRelCol(relname As String, colname As String)

On Error GoTo relcolAddError Dim relcol As XmimRelCol Set relcol = New_XmimRelCol relcol.Init svr relcol.Clear

The relation column is where the data is actually stored. To associate a column with a relation: specify the relationname, the column name, properties and:

relcol.rel = relname relcol.col = colname

There are several properties that can be set for a XMIMRelCol object, for instance, the object type,aggregation rule and minimum price increment (delta).

relcol.NFields = 1 relcol.RelColType = utils.RelColBase relcol.Aggr = utils.RelColAggrClose relcol.Delta = utils.RelColDeltaNone relcol.ObjType = utils.RelColObjFloat relcol.Add Set relcol = Nothing addRelCol = True Exit Function relcolAddError: addRelCol = False


439

Set rel = Nothing Exit Function End Function

Function addData(relname As String, colname As String)

Dim recs As XmimRecordsDouble Set recs = New_XmimRecordsDouble recs.Init svr recs.Clear recs.NRels = 1 recs.NCols = 1 recs.Units = 4 recs.rel(0) = relname recs.col(0) = colname recs.NRecords = 3 recs.DateTime(0) = #1/28/2002# recs.Val(0, 0, 0) = 210 recs.DateTime(1) = #1/29/2002# recs.Val(0, 0, 1) = 310 recs.DateTime(2) = #1/30/2002# recs.Val(0, 0, 2) = 412 recs.putRecords addData = True Set recs = Nothing End Function

440


Part III: Framework Extensions

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 10: C Stored Function Framework

443

CHAPTER10

C Stored Function Framework

DescriptionThe purpose of the C stored function framework (framework, for short) is to allow easy introduction of newfunctionality into the Commodity DataServer server. The main advantage of the code written within the C storedfunction framework is that the stored function implementation can access any of the internal Commodity DataServerobjects (such as SchSchema object, for example).

New functionality written within the framework can be called in the normal way, using the following bmim_clientquery, for example:

%exec.units: 1 hour

SHOW 1: slice (10, 15, 2004) WHEN Date is after 1/1/2004 AND Date is Wednesday

where slice is the name of the C stored function.

This document is structured as follows.

● “Structure of a C Stored Function” describes the components of C stored functions.

● “Stored Function Framework” describes the utility functions available to writers of C stored functions. These utilityfunctions expose part of the functionality of the Commodity DataServer server.

● “Stored Function Life Cycle” describes the way the Commodity DataServer interacts with a C stored function whileexecuting a user query.

● “Example: Providing a C Stored Function Wrapper for an Existing C Function” illustrates a complete example.Readers may wish to skim over the material in “Structure of a C Stored Function” through “Stored Function LifeCycle”, and then review that material after reading “Example: Providing a C Stored Function Wrapper for an ExistingC Function”.

● “Complete Predictor Code” and “Complete Lazy Predictor Code” contain all the source code for the examples.

444


Structure of a C Stored FunctionEach new set of code should be implemented as a C file, which we refer to as a C stored function. Since eachstored function will eventually become a library in which the Commodity DataServer will load upon startup, the Cimplementation was chosen so as to avoid problems with loading C++ libraries.

It is essential for a stored function to be able to maintain a state, which would persist between invocations of differentstored function functions. The stored function state is represented by the following structure (which we call storedfunction arguments, thus the name): typedef struct { //error string char *error; //pointer to a SchSchema object void *schema; //pointer to an ExXmimOptions object, containing user defaults const void *user_defaults; //pointer to an ExCompiledQuery object corresponding to the current query void *compiled_query; //MimUnits (ExRelcol units) int units; //ExRelcol modulus int modulus;

//pointer to the cfunc's characteristic time series, //a time series which has non-NaN values for times //when this c stored function is defined and NaN //values when it is not void *characteristic_time_series; //pointer to the cfunc's ExTradingPattern object void *trading_pattern; //pointer to the cfunc's ExDateRange object void *trading_date_range; //cfunc specific state xmim_cfunc_state *state; //constant int args xmim_cfunc_int_args *int_args; //constant double args xmim_cfunc_double_args *double_args; //category args xmim_cfunc_ *category_args; //string args xmim_cfunc_string_args *string_args; //ATTR args xmim_cfunc_attr_args *attr_args; //time offset args xmim_cfunc_time_offset_args *offset_args; //period args xmim_cfunc_time_period_args *period_args; //security args xmim_cfunc_security_args *security_args; //default return value double default_val; //return value of the get_value method

445

double result; } xmim_cfunc_args;

This structure has a member called state, which is an array of void*, specific to a particular stored function. Astored function may store any information it needs to persist between the function calls in this array.

Each stored function has the following structure. All functions are mandatory, unless specified otherwise. The functioncfunc_instance_init_defaults initializes the default values of the stored function. Each stored function should"know" about the arguments it expects. The stored function will initialize the appropriate arguments in this function,i.e., assign the values to the names and size members of the appropriate structs.extern "C" int cfunc_instance_init_defaults (xmim_cfunc_args *args) {...}

For example, if a stored function expects to get three constants as arguments, it should do something analogous tothe following in the cfunc_instance_init_defaults function:

Also, we suggest that a stored function initialize args->state in this function.

This can be performed in the cfunc_instance_init function as well.

int num = 3; args->int_args->size = num; args->int_args->ptr = new int[num]; for (int i = 0; i < num; i++) args->int_args->ptr[i] = 0;

args->int_args->names = new char*[num]; args->int_args->names[0] = "day"; args->int_args->names[1] = "month"; args->int_args->names[2] = "year";

The function cfunc_instance_init initializes the state ( argsstate ) of a stored function. Everything thatneeds to be initialized before the cfunc_instance_get_value function will be called should be initialized here.

The stored function's input parameters are set at this point and can be used by the stored function toinitialize its state if necessary.

extern "C" int cfunc_instance_init (xmim_cfunc_args *args) { ...}

The optional function cfunc_instance_compute_trading_pattern computes the trading pattern for a cfunc.extern "C" int cfunc_instance_compute_trading_pattern (xmim_cfunc_args *args) {...}

446


The optional function cfunc_instance_compute_trading_date_range computes the trading date range for astored function.extern "C" int cfunc_instance_compute_trading_date_range (xmim_cfunc_args *args) {...}

The optional function cfunc_instance_get_characteristic_time_series computes the characteristic timeseries for a stored function. A characteristic time series is a time series which has non-NaN values for times when thedate exists and NaN values for times when the date does not exist.extern "C" int cfunc_instance_get_characteristic_time_series (xmim_cfunc_args *args) {...}

The function cfunc_instance_get_value sets args->result to the value this stored function needs to returnfor the date and time dt.extern "C" int cfunc_instance_get_value (xmim_cfunc_args *args, xmim_cfunc_date_time *dt) {...}

The function cfunc_instance_destroy is a destructor for a stored function. A stored function should destroy allthe instances it creates.extern "C" int cfunc_instance_destroy (xmim_cfunc_args *args) {...}

C Stored Functions can have constant (integer, double), attribute, time offset, time period, security, category,and string input arguments. The first five argument types are the same as arguments accepted by macros. Thecategory argument is stored as a string and can be used to retrieve and process all relations under that categoryin the Commodity DataServer schema hierarchy. The string argument appears as a sequence of characters indouble quotes in the C stored function call. For example, the following is a call to a C stored function namedbuiltin_string_example which takes two string arguments ("ALEX" and "Close"):builtin_string_example(“ALEX”, “Close”)

A C stored function "registers" a string input argument in the same way as any other argument. Namely,builtin_string_example may "register" to receive two string arguments (relation name and column name) withthe following code in its cfunc_instance_init_defaults functions as follows: int num = 2; //string args xmim_cfunc_string_args *str = new xmim_cfunc_string_args; str->size = num; str->names = new char*[num] str->names[0] = "relation"; str->names[1] = "column"; str->ptr = new char*[num]; for (int i = 0; i < num; i++) str->ptr[i] = NULL;

args->string_args = str;

447

String input arguments are handled in the same way as the rest of the arguments.

The cfunc_instance_get_characteristic_time_series function is optional, but when defined, isexpected to set the args->characteristic_time_series member to the appropriate characteristic timeseries, such that the time series has a non-NaN value when the date exists and a NaN value for the times whenthe date does not exist. For example, if a stored function is defined at exactly the times when the Close of IBMis defined, then args->characteristic_time_series should be set to the time series retrieved via thexmim_cfunc_get_time_series function for the Close of IBM.

Stored Function FrameworkTwo kinds of C stored functions can be introduced into the Commodity DataServer: regular C stored functions andbuilt-in C stored functions.

Regular Stored FunctionsRegular C stored functions are introduced as follows. Each stored function needs to be built into a separate sharedlibrary (e.g., a .so or .dll file) named with the name of the function it contains. For example, slice.so will contain theslice C stored function. The .so file then goes into the directory <libraryDir>/attr/plugin, which is the directory thatthe Commodity DataServer will check for stored functions to load upon startup.

The utility functions available for the stored functions to use are in the include/xmim_cfunc_api.h file. Utility functionsmostly operate on internal Commodity DataServer objects hidden from the stored function implementation. Theirimplementation is found in the files cfunc/xmim_cfunc_api.c and plugin/xmim_cfunc_api_model.c. The second filecontains the functions which are used by the model classes of the framework and is included into the first one. Thisfile (cfunc/xmim_cfunc_api.c) is linked into the client, while plugin/xmim_cfunc\api.c is linked into theserver.

The current list of helper functions is provided below. //create new xmim_cfunc_args struct xmim_cfunc_args *xmim_cfunc_new_cfunc_args ();

//set the error string of args to message void xmim_cfunc_set_error (xmim_cfunc_args *args, char *message);

//free error string void xmim_cfunc_free_error (xmim_cfunc_args *args);

//creates a xmim_cfunc_date_time struct initialized to the values provided xmim_cfunc_date_time *xmim_cfunc_get_date_time (xmim_cfunc_args *args, unsigned year, unsigned month, unsigned day, unsigned hour, unsigned minute, unsigned second);

//creates a xmim_cfunc_date_time struct initialized to the values provided

448


xmim_cfunc_date_time *xmim_cfunc_get_full_date_time (xmim_cfunc_args *args, unsigned year, unsigned month, unsigned day, unsigned_week_day, unsigned day_of_month, unsigned hour, unsigned minute, unsigned second); //creates a xmim_cfunc_date_time struct initialized to

//the values provided, values include milliseconds xmim_cfunc_date_time *xmim_cfunc_get_full_date_time_with_millis (xmim_cfunc_args *args, unsigned year, unsigned month, unsigned day, unsigned week_day, unsigned day_of_month, unsigned hour, unsigned minute, unsigned second, unsigned millisecond); //creates a xmim_cfunc_date_time struct initialized to //the values provided, values include milliseconds xmim_cfunc_date_time *xmim_cfunc_get_date_time_with_millis (xmim_cfunc_args *args, unsigned year, unsigned month, unsigned day, unsigned hour, unsigned minute, unsigned second, unsigned millisecond);

void xmim_cfunc_set_date (xmim_cfunc_args *args, xmim_cfunc_date_time *dt, unsigned year, unsigned month, unsigned day);

//sets the fields of dt to the specified values void xmim_cfunc_set_full_date_time (xmim_cfunc_args *args, xmim_cfunc_date_time *dt, unsigned year, unsigned month, unsigned day, unsigned week_day, unsigned day_of_month, unsigned hour, unsigned minute, unsigned second);

//sets the fields of dt to the specified values, values include milliseconds void xmim_cfunc_set_full_date_time_with_millis (xmim_cfunc_args *args, xmim_cfunc_date_time *dt, unsigned year, unsigned month, unsigned day, unsigned week_day, unsigned day_of_month, unsigned hour, unsigned minute, unsigned second, unsigned millisecond);

//returns a MimDateTime object initialized to the values in dt void *xmim_cfunc_get_mim_date_time (xmim_cfunc_args *args, xmim_cfunc_date_time *dt);

//get the value of ExObject pointed to by ex_object on date time dt


449

double xmim_cfunc_get_value (xmim_cfunc_args *args, void *ex_object, xmim_cfunc_date_time *dt);

//compares the arguments passed to a cfunc //returns 1 if they are equal, 0 otherwise int xmim_cfunc_equal (xmim_cfunc_args *arg1, xmim_cfunc_args *arg2);

//Applies time offset named time_offset_name to date time date //and returns a pointer to the new date time struct. //Here time_series_name and time_offset_name are the names of //the time series and time offset input parameters correspondingly. //So, the name of the time series might be args->attr_args->names[0] //and the name of the time offset may be args->offset_args->names[0]. //Delegates the functionality to //xmim_cfunc_date_time *xmim_cfunc_apply_time_offset // (xmim_cfunc_args *args, // xmim_cfunc_date_time *date, // void *time_series, // void *time_offset) xmim_cfunc_date_time *xmim_cfunc_apply_time_offset_to_args (xmim_cfunc_args *args, xmim_cfunc_date_time *date, char *time_series_name, char *time_offset_name);

//Applies time offset time_offset to date time date //and returns a pointer to the new date time struct. //Uses an ExObject called time_series to perform the offset. //Returns NULL and sets the error message on error. xmim_cfunc_date_time *xmim_cfunc_apply_time_offset (xmim_cfunc_args *args, xmim_cfunc_date_time *date, void *time_series, void *time_offset);

//Applies time period called period_name to the date time dt. //Uses ExObject pointed to by an attr arg called time_series_name //to apply the period. //Returns 1 upon success, 0 on failure (and sets the error message). int xmim_cfunc_apply_time_period (xmim_cfunc_args *args, xmim_cfunc_date_time *dt, char *period_name, char *time_series_name, int include_left, int include_right, xmim_cfunc_date_time *left_date, xmim_cfunc_date_time *right_date);

//Populates the list of relations based on path. xmim_cfunc_relation_list* xmim_cfunc_get_relations_by_name (xmim_cfunc_args *args, char *path);

//Populates the list of slice-specific relations based on path for slice C Stored Functions.

450


xmim_cfunc_date_relation_list* xmim_cfunc_get_date_relations_by_name (xmim_cfunc_args *args, char *path);

//Returns a pointer to an empty time series. void *xmim_cfunc_get_empty_time_series (xmim_cfunc_args *args, void *ex_object);

//Returns a pointer to the ExRelcol Object based on the given relname and colname. void *xmim_cfunc_get_time_series (xmim_cfunc_args *args, char *relname, char *colname);

//Sets int value for the time series object (TsTimeSeries is assumed) //produced by on date time dt cfunc_get_empty_time_series int xmim_cfunc_set_int_value_for_time_series (xmim_cfunc_args *args, void *time_series, int value, xmim_cfunc_date_time *dt);

//Sets double value for the time series object (TsTimeSeries is assumed) //produced by on date time dt cfunc_get_empty_time_series int xmim_cfunc_set_double_value_for_time_series (xmim_cfunc_args *args, void *time_series, double value, xmim_cfunc_date_time *dt);

//Gets the value of the time series (TsTimeSeries is assumed) time_series //on date time dt. double xmim_cfunc_get_value_for_time_series (xmim_cfunc_args *args, void *time_series, xmim_cfunc_date_time *dt);

//Sets the args->trading_pattern to the ExTradingPattern //computed by ex_object void xmim_cfunc_compute_trading_pattern (xmim_cfunc_args *args, void *ex_object);

//Sets the args->trading_date_range to the ExTradingPattern //computed by ex_object void xmim_cfunc_compute_trading_date_range (xmim_cfunc_args *args, void *ex_object);

//Returns a pointer to the ExDateRange object initialized //to from and to. void *xmim_cfunc_get_ex_trading_date_range (xmim_cfunc_args *args, xmim_cfunc_date_time *from, xmim_cfunc_date_time *to);

//Returns a pointer to the mATimeOffset object initialized //to multiple, unit and direction. void *xmim_cfunc_get_time_offset (xmim_cfunc_args *args, int multiple, xmim_cfunc_dwmqy unit, xmim_cfunc_direction direction);


451

//Returns a pointer to the mTimePeriod object initialized //to fromOffset and toOffset. Defaults to today for NULL inputs.

void *xmim_cfunc_get_time_period (xmim_cfunc_args *args, void *fromOffset, void *toOffset);

//compare val to NaN int xmim_cfunc_isNaN (xmim_cfunc_args *args, double val);

//make a not a number object out of val void xmim_cfunc_makeNan (xmim_cfunc_args *args, double *val);

xmim_cfunc_date_time *xmim_cfunc_get_from_date (xmim_cfunc_args *args, void *ex_object);

xmim_cfunc_date_time *xmim_cfunc_get_to_date (xmim_cfunc_args *args, void *ex_object);

//sets an argument of type type called name to value int xmim_cfunc_set_arg (xmim_cfunc_args *args, xmim_cfunc_arg_type type, void *value, char *name);

//sets a constant arg called name to k int xmim_cfunc_set_int_constant (xmim_cfunc_args *args, int k, char *name);

//sets a constant arg called name to k int xmim_cfunc_set_dbl_constant (xmim_cfunc_args *args, double k, char *name);

//sets a category arg called name to cat int xmim_cfunc_set_category (xmim_cfunc_args *args, char *cat, char *name); //sets a string arg called name to str void xmim_cfunc_set_string (xmim_cfunc_args *args, char *str, char *name);

//sets an attr arg called name to attr int xmim_cfunc_set_attr (xmim_cfunc_args *args, void *attr, char *name);

//sets a time offset arg called name to offset int xmim_cfunc_set_offset (xmim_cfunc_args *args, void *offset, char *name);

//sets a time period arg called name to period int xmim_cfunc_set_period (xmim_cfunc_args *args, void *period, char *name);

//sets a security arg called name to relation int xmim_cfunc_set_security (xmim_cfunc_args *args, void *relation, char *name);

//sets the xmim_cfunc state called name to state int xmim_cfunc_set_state (xmim_cfunc_args *args, void *state, char *name);

//returns a pointer to an argument of type type called name

452


//returns NULL and sets the error message if an argument with //such name is not found void *xmim_cfunc_get_arg (xmim_cfunc_args *args, xmim_cfunc_arg_type type, char *name);

//returns a pointer to a constant called name void *xmim_cfunc_get_int_constant (xmim_cfunc_args *args, char *name);

//returns a pointer to a constant called name void *xmim_cfunc_get_dbl_constant (xmim_cfunc_args *args, char *name);

//returns a pointer to a category called name void *xmim_cfunc_get_category (xmim_cfunc_args *args, char *name);

//returns a pointer to a string called name void *xmim_cfunc_get_string (xmim_cfunc_args *args, char *name);

//returns a pointer to an attr called name void *xmim_cfunc_get_attr (xmim_cfunc_args *args, char *name);

//returns a pointer to a time offset called name void *xmim_cfunc_get_offset (xmim_cfunc_args *args, char *name);

//returns a pointer to a time period called name void *xmim_cfunc_get_period (xmim_cfunc_args *args, char *name);

//returns a pointer to a security called name void *xmim_cfunc_get_security (xmim_cfunc_args *args, char *name);

//returns a pointer to a state called name void *xmim_cfunc_get_state (xmim_cfunc_args *args, char *name);

//frees the args struct void xmim_cfunc_free_cfunc_args (xmim_cfunc_args *args);

//free date time dt void xmim_cfunc_free_date_time (xmim_cfunc_args *args, xmim_cfunc_date_time *dt);

//free MimDateTime object void xmim_cfunc_free_mim_date_time (xmim_cfunc_args *args, void *dt);

//free the time series created by the xmim_cfunc_get_time_series //function. Cfuncs MUST use this function to destroy the time //series. void xmim_cfunc_free_time_series (xmim_cfunc_args *args, void *time_series);

// free the time series created via cfunc_get_empty_time_series // function. ex_object is a pointer to the ExObject instance // used as an input parameter in xmim_cfunc_get_empty_time_series void xmim_cfunc_free_ts_time_series (xmim_cfunc_args *args, void *time_series);

//free trading pattern created by cfunc_compute_trading_pattern //function void xmim_cfunc_free_trading_pattern (xmim_cfunc_args *args;


453

//free trading date range created by xmim_cfunc_get_ex_trading_date_range //function void xmim_cfunc_free_trading_date_range (xmim_cfunc_args *args);

//free time offset created by xmim_cfunc_get_time_offset void xmim_cfunc_free_time_offset (xmim_cfunc_args *args, void *offset);

//free time period created by xmim_cfunc_get_time_period void xmim_cfunc_free_time_period (xmim_cfunc_args *args, void *period);

//print part of the args void xmim_cfunc_print (xmim_cfunc_args *args);

//print time offset void xmim_cfunc_print_time_offset (xmim_cfunc_args *args, void *off);

//print time period void xmim_cfunc_print_time_period (xmim_cfunc_args *args, void *period);

Built-in Stored FunctionsBuilt-in C stored functions are literally “built” into the Commodity DataServer, that is theBUILTIN_CSTORED_FUNC_SRCS target of the Makefile needs to be edited to include a particular built-in C storedfunction source code (to be compiled and later linked into the Commodity DataServer).

Built-in C stored functions subclass the model/mBuiltinCStoredFunction class, which itself is abstract.All methods (except get_path) that mBuiltinCStoredFunction subclasses are required to implement arein one-to-one correspondence with the regular plugin functions as is obvious from the names. As with regularC stored functions, compute_trading_pattern and compute_trading_date_range methods areoptional. The get_path method is specific to built-in C stored functions and returns the path (within macro'shierarchy) and the name of the given built-in C stored function. For example, the following code in the constructor ofCStoredFuncStringExample initializes its path:

this->path = new char[1028];sprintf (this->path, "%s%s%s", ATTR_MACROS, PATH_SEPARATOR, "builtin_string_example");

Please note that "builtin_string_example" corresponds to the built-in C stored function's name, so the path tothis c stored function is "attr/builtin_string_example".

Implementation for the other methods of mBuiltinCStoredFunction subclasses is similar to regular C storedfunctions. Built-in C stored function sources can be placed anywhere, but it is our suggestion to put them underplugin/ directory. We also suggest to prefix built-in C stored function source file names with cstored_func_(as in cstored_func_string_example.C) and to prefix the class names with CStoredFunc (as inCStoredFuncStringExample).

There are two examples of built-in C stored functions available. The first one is available under plugin/cstored_func_lazy_predictor.C and mirrors the regular C stored function under apps/plugins/lazy_predictor.c. The

454


second one is under plugin/cstored_func_string_example.C (we will refer to it as string example function). The stringexample function demonstrates the use of string as input parameters and takes two input string arguments. The inputarguments are user-supplied relation name and column name. The function then returns the value corresponding tothe given relation and column.

Built-in C stored functions should be "registed" via the register_builtin_cstored_func method of thexmim_server right after the creation of the xmim_server instance in main/api_svr_main.C. The registration of the twoexample built-in C stored functions is commented out in api_svr_main.C. Just uncomment those lines to query theexample built-in C stored functions.

Stored Function Life CycleThe Commodity DataServer will call stored function's cfunc_instance_init_defaults functionwhen the stored function is loaded. Stored function's cfunc_instance_init function is calledwhen the executable stored function object is created (in the constructor of ExCfuncObject).The optional cfunc_instance_get_characteristic_time_series is called when settingthe query's parameters to take the stored function's characteristic series into account. The optionalcfunc_instance_compute_trading_pattern and cfunc_instance_compute_trading_date_rangefunctions are called to ask a stored function for its trading pattern and trading date range if any. The server calls thecfunc_instance_get_value function for different date times to get the value of a stored function for that dateand time. And finally, the cfunc_instance_destroy function is called upon deletion of the executable storedfunction object.

Example: Providing a C Stored Function Wrapper for an ExistingC Function

Straightforward ApproachSuppose we have a function compute_predictor that takes in two arrays of values, x and y, and uses these topredict the next value of x. The actual implementation of this model is irrelevant to this discussion; we assume thatsuch a model already exists. The model can be called as follows:


455

typedef struct { //date unsigned day; unsigned month; unsigned year; unsigned week_day; unsigned day_of_month;} date;

double *compute_predictor (date *dates, double *x_values, double *y_values, int size) { ...;}

The function takes an array of dates and two arrays of values corresponding to the dates, all of which are of lengthsize. Here, x_values[0] is the value of a time series X on the date dates[0].

Now let us demonstrate the straightforward way of converting this function into a C stored function that theframework can work with. This method does not require any modifications to the function that implements the model.In the next section, we will describe a more efficient way to do this, which does involve rewriting the model.

Let predictor be the name for the C stored function we want to write. The first thing we need to do is specify theinput parameters for the C stored function, which in this case are going to be two security parameters. Therefore, wecan define the cfunc_instance_init_defaults function as follows:

#define PRED_STOCK1 "stock1"#define PRED_STOCK2 "stock2"

extern "C" int cfunc_instance_init_defaults (xmim_cfunc_args *args) { int num = 2; //security args xmim_cfunc_security_args *sec = new xmim_cfunc_security_args; sec->size = num; sec->names = new char*[num]; sec->names[0] = PRED_STOCK1; sec->names[1] = PRED_STOCK2; sec->rels = new char*[num]; sec->rels[0] = NULL; sec->rels[1] = NULL;

args->security_args = sec;

//define the state here

return 1;}

A state is specific to a particular C stored function and is defined in the cfunc_instance_init_defaultsfunction. A state consists of all objects that need to persist between invocations of different functions, so that afunction can retrieve an object it is interested in and use for its purposes.

Now let us define the state which our C stored function needs to keep. We need to store one of the time series thatwe will create based on the input stock symbols so that we are able to define the trading pattern and the trading

456


range for predictor later. Also, in cfunc_instance_init function we will compute the result values and"remember'' them throughout our C stored function's lifetime.

Taking this into account the cfunc_instance_init_defaults looks as follows:

#define PRED_STOCK1 "stock1"#define PRED_STOCK2 "stock2"#define PRED_TIME_SERIES "time_series"#define PRED_RESULT "result"

#define DEFAULT_COLUMN "Asks"

extern "C" int cfunc_instance_init_defaults (xmim_cfunc_args *args) { int num = 2; //security args xmim_cfunc_security_args *sec = new xmim_cfunc_security_args; sec->size = num; sec->names = new char*[num]; sec->names[0] = PRED_STOCK1; sec->names[1] = PRED_STOCK2; sec->rels = new char*[num]; sec->rels[0] = NULL; sec->rels[1] = NULL;


xmim_cfunc_state *state = new xmim_cfunc_state; num = 2; state->size = num; state->ptr = new xmim_cfunc_obj_ptr[num]; for (int i = 0; i < num; i++) state->ptr[i].obj = NULL;

state->names = new char*[num]; state->names[0] = PRED_TIME_SERIES; state->names[1] = PRED_RESULT;

args->state = state;

return 1;}

In the cfunc_instance_init function C stored functions are able to access the input parameters with which theyare called from a user's query. For example, if our predictor is invoked via the query:

SHOW predictor(EA.EPWVQ304, EA.EPWVQ404)WHEN Date is after 2002 AND Date is Wednesday


457

Then, to get the first input stock symbol, we would use the xmim_cfunc_get_arg function:

char *symbol = NULL; symbol = (char *)xmim_cfunc_get_arg (args, P_SECURITY, PRED_STOCK1); if (!symbol) return 0;

The first argument to xmim_cfunc_get_arg is an xmim_cfunc_args structure, which is populated with theinput parameters after the call to cfunc_instance_init_defaults. This means that the input parameters areonly inaccessible in the cfunc_instance_init_defaults function and accessible in all other functions of aC stored function. In general, we suggest that the cfunc_instance_init_defaults should only be used tospecify the input parameters for the C stored function and its state. Nothing else should happen there since thexmim_cfunc_args structure is empty at that point.

The second argument is the type of the input parameter we want to get back. All possible types are defined in:

//arg types for a cfunc typedef enum { P_INT_CONSTANT, P_DBL_CONSTANT, P_CATEGORY, P_STRING, P_ATTR, P_TIME_OFFSET, P_TIME_PERIOD, P_SECURITY, //cfunc specific state P_STATE } xmim_cfunc_arg_type;

in the xmim_cfunc_api.h file. And the third argument to xmim_cfunc_get_arg is the name of the inputparameter as we defined it in the cfunc_instance_init_defaults function.

The last two lines of the code snippet above handle an exceptional condition when the returned symbol is NULL.xmim_cfunc_get_arg will return NULL in two cases: when it fails to retrieve the named parameter or if theparameter value is NULL. In the first case args->error is set to indicate the failure, while in the second case noerror is be set. So, if a NULL can be a valid value for a parameter (the state is treated analogously to input parametersby xmim_cfunc_get_arg, and NULL can be a valid value before we initialize the state), then we need to checkwhether args->error is set (in which case xmim_cfunc_get_arg has failed) or is NULL (in which case theparameter value is NULL).

Returning a 0 from any of the C stored function's functions signals that an error has occurred during the function'sexecution. The framework will then raise an error with the args->error error message.

Now let us get back to what we want to do in the cfunc_instance_init function. Given the input stock symbols,we are going to create the corresponding time series. The compute_predictor function takes three arrays asits input: an array of dates and arrays of values on those dates for the two time series. This is what we could store

458


in our state. However, it is more convenient for us to precompute the result exactly the way we compute it in thecompute_predictor function and then store it in our state. Thus the pred_result structure:

typedef struct { double *values; xmim_cfunc_date_time **dates; int size; } pred_result;

which stores both the dates array and the result values on the corresponding dates.

In general, the purpose of the cfunc_instance_init function is to initialize the state. It is executed before the firstcall to the cfunc_instance_get_value, so all the initialization should be done there.

Our cfunc_instance_init will initialize the state as follows:

extern "C" int cfunc_instance_init (xmim_cfunc_args *args) { char *relname1 = NULL, *relname2 = NULL; void *series1 = NULL, *series2 = NULL; xmim_cfunc_date_time *from_date = NULL, *to_date = NULL, *date = NULL, *old_date = NULL; void *offset; pred_result *result = NULL; int size = 0;

//first input argument relname1 = (char *)xmim_cfunc_get_arg (args, P_SECURITY, PRED_STOCK1); if (!relname1) return 0;

//second input argument relname2 = (char *)xmim_cfunc_get_arg (args, P_SECURITY, PRED_STOCK2); if (!relname2) return 0;

//create the time series for the first stock symbol series1 = xmim_cfunc_get_time_series (args, relname1, DEFAULT_COLUMN); if (!series1) return 0; //store the first time series in the state if (!xmim_cfunc_set_arg (args, P_STATE, series1, PRED_TIME_SERIES)) return 0;

//create the time series for the second stock symbol //will get deleted by the framework series2 = xmim_cfunc_get_time_series (args, relname2, DEFAULT_COLUMN); if (!series2) return 0;

// the from and to dates for the first time series // the date range we are going to use from_date = xmim_cfunc_get_from_date (args, series1); to_date = xmim_cfunc_get_to_date (args, series1);

//calculate the number of values we have

459

date = xmim_cfunc_get_from_date (args, series1); size = 1;

while (!xmim_cfunc_date_time_equal (args, date, to_date)) { offset = xmim_cfunc_get_time_offset (args, 1, PL_DAYS, PL_LATER); old_date = date; date = xmim_cfunc_apply_time_offset (args, date, series1, offset); xmim_cfunc_free_date_time (args, old_date); xmim_cfunc_free_time_offset (args, offset); size++; }

//compute the result result = new pred_result; result->values = new double[size]; result->dates = new xmim_cfunc_date_time*[size]; result->size = size; double v1, v2, prev_v1, prev_v2; //the first value is a NaN since we do not have the //value on the previous date double zero = 0; double *doubleNan = &zero; xmim_cfunc_makeNan (args, doubleNan); result->values[0] = *doubleNan; result->dates[0] = from_date; prev_v1 = xmim_cfunc_get_value (args, series1, from_date); prev_v2 = xmim_cfunc_get_value (args, series2, from_date); //compute the values using the two time series for (int i = 1; i < size; i++) { offset = xmim_cfunc_get_time_offset (args, i, PL_DAYS, PL_LATER); date = xmim_cfunc_apply_time_offset (args, from_date, series1, offset); v1 = xmim_cfunc_get_value (args, series1, date); v2 = xmim_cfunc_get_value (args, series2, date); result->values[i] = v1 + (0.5 * v1 * (((v1 - prev_v1) / prev_v1) + ((v2 - prev_v2) / prev_v2))); result->dates[i] = date; prev_v1 = v1; prev_v2 = v2; xmim_cfunc_free_time_offset (args, offset); }

//store the result in the state if (!xmim_cfunc_set_arg (args, P_STATE, result, PRED_RESULT)) return 0;

return 1;}

xmim_cfunc_get_time_series returns a time series object corresponding to the relation and the columnprovided. The column is assumed to be the same for all symbols (relations) and known beforehand. A time series canbe either destroyed explicitly by calling the

460


void xmim_cfunc_free_time_series (xmim_cfunc_args *args, void *time_series);

function or destroyed implicitly by the framework when the C stored function instance is being destroyed (note, thatthis only holds for time series objects created by the framework, and everything you create, you should destroy).

When computing the values for the result array, we are using the xmim_cfunc_get_time_offset function to getthe desired offset and then get a date by applying that offset to the from_date.

Then, the cfunc_instance_get_value function will retrieve the result we have computed in thecfunc_instance_init function, find a value on the requested date and return that value. It looks as follows:extern "C" int cfunc_instance_get_value (xmim_cfunc_args *args, xmim_cfunc_date_time *dt) { pred_result *result = (pred_result *)xmim_cfunc_get_arg (args, P_STATE, PRED_RESULT); xmim_cfunc_date_time *date = NULL;

if (!result) return 0;

for (int i = 0; i < result->size; i++) { date = (xmim_cfunc_date_time *) result->dates[i]; if (xmim_cfunc_date_time_equal (args, date, dt)) { args->result = result->values[i]; return 1; } } //we do not have the requested date - return NaN. double zero = 0; double *doubleNan = &zero; xmim_cfunc_makeNan (args, doubleNan); args->result = *doubleNan;

return 1;}

As you have noticed, the cfunc_instance_get_value function returns the result by setting args->result to thevalue it needs to return.

Then, finally in the cfunc_instance_destroy function we are going to clean up everything we have created:extern "C" int cfunc_instance_destroy (xmim_cfunc_args *args) { delete[] args->security_args->rels; delete[] args->security_args->names; delete args->security_args; args->security_args = NULL; //the time series will be deleted for us by the framework

pred_result *result = (pred_result *)xmim_cfunc_get_arg (args, P_STATE, PRED_RESULT); if (result) { for (int i = 0; i < result->size; i++) delete result->dates[i];

delete[] result->dates;


461

delete[] result->values;

delete result; }

return 1;}

We covered the mandatory functions each C stored function is required to implement. Thenext two functions are optional: cfunc_instance_compute_trading_pattern andcfunc_instance_compute_trading_date_range.

They define the trading pattern (e.g., M-F, 8-5) and the trading date range (e.g., Jan 1, 1923 -- November 15, 2004)respectively and can be implemented by using one of the existing time series to get the trading pattern and the tradingdate range:

extern "C" int cfunc_instance_compute_trading_pattern (xmim_cfunc_args *args) { void *time_series = xmim_cfunc_get_arg (args, P_STATE, STOCK1_TIME_SERIES);

if (!time_series) return 0;

xmim_cfunc_compute_trading_pattern (args, time_series); return 1;}

extern "C" int cfunc_instance_compute_trading_date_range (xmim_cfunc_args *args) { void *time_series = xmim_cfunc_get_arg (args, P_STATE, STOCK1_TIME_SERIES);


xmim_cfunc_compute_trading_date_range (args, time_series); return 1; }

extern "C" int cfunc_instance_get_characteristic_time_series (xmim_cfunc_args *args) { void *close_series = xmim_cfunc_get_arg (args, P_STATE, VWAP_TEST_CLOSE_SERIES); if (!close_series) close_series = vwap_set_series (args);

if (!close_series) { xmim_cfunc_set_error (args, "VWAP get_characteristic_time_series: failed to create the time series. Please, make sure this call is happening at the right place."); return 0; }

args->characteristic_time_series = close_series; return 1;}

Complete code for this example is available in “Complete Predictor Code”.

462


Lazy Evaluation ApproachNow let us consider a more efficient way of writing a C stored function based on the example we described in theprevious section. The code in this section is more efficient in two ways:

● It does not have to allocate a time series to store all possible return values and

● It does not compute a result unless it is actually needed by the query.

However, to achieve this efficiency we must modify the code that computes the predictor model. It should be notedthat this modification may be impractical or impossible. Let us call the new C stored function we are going to writelazy_predictor.

The only code that would change in the cfunc_instance_init_defaults function is the code that handles thestate, since now we want to store the time series corresponding to the input stock symbols in the state: xmim_cfunc_state *state = new xmim_cfunc_state; num = 2; state->size = num; state->ptr = new xmim_cfunc_obj_ptr[num]; for (int i = 0; i < num; i++) state->ptr[i].obj = NULL;

state->names = new char*[num]; state->names[0] = STOCK1_TIME_SERIES; state->names[1] = STOCK2_TIME_SERIES;

In the cfunc_instance_init function we are going to create the time series corresponding to the two stock inputsymbols and store them in the state (just like we did in the previous section).

The cfunc_instance_get_value is going to compute the result value for the specific date it is asked about.extern "C" int cfunc_instance_get_value (xmim_cfunc_args *args, xmim_cfunc_date_time *dt) { xmim_cfunc_date_time *from_date = NULL; void *series1 = xmim_cfunc_get_arg (args, P_STATE, STOCK1_TIME_SERIES); void *series2 = xmim_cfunc_get_arg (args, P_STATE, STOCK2_TIME_SERIES);

if (!series1 || !series2) return 0;

//return a NaN for the first date from_date = xmim_cfunc_get_from_date (args, series1); if (xmim_cfunc_date_time_equal (args, from_date, dt)) { double zero = 0; double *doubleNan = &zero; xmim_cfunc_makeNan (args, doubleNan); args->result = *doubleNan; return 1; }

//compute the result double v1, v2, prev_v1, prev_v2; xmim_cfunc_date_time *date = NULL;

463

void *offset = NULL; offset = xmim_cfunc_get_time_offset (args, 1, PL_DAYS, PL_AGO); date = xmim_cfunc_apply_time_offset (args, dt, series1, offset); prev_v1 = xmim_cfunc_get_value (args, series1, date); prev_v2 = xmim_cfunc_get_value (args, series2, date); v1 = xmim_cfunc_get_value (args, series1, dt); v2 = xmim_cfunc_get_value (args, series2, dt); xmim_cfunc_free_date_time (args, date); xmim_cfunc_free_time_offset (args, offset);

args->result = v1 + (0.5 * v1 *

(((v1 - prev_v1) / prev_v1) + ((v2 - prev_v2) / prev_v2)));

return 1;}

The rest of the functions here are analogous to the previous section.

The advantage of this approach is computing the values "on demand" which is faster than pre-computing and storingthem.

When should either approach be used? We suggest that if you are just sitting down to write a new C stored functionyou try to use the lazy evaluation approach. However, you are more likely to have some functions you would like to"wrap around" to use within the framework. In this case, if the logic of a particular function you want to "wrap around"is easy to change such that is uses lazy evaluation, we would recommend using the lazy evaluation approach.

If the function is complicated, then it might be better to "wrap it around" in a straightforward way.

Complete code for this example is available in “Complete Lazy Predictor Code”.

Complete Predictor Code#include "include/xmim_cfunc_api.h"#include <stdio.h>

//names for the input parameters#define PRED_STOCK1 "stock1"#define PRED_STOCK2 "stock2"//names for the state parameters#define PRED_TIME_SERIES "time_series"#define PRED_RESULT "result"

//column to use when creating a time//series for the stock symbols#define DEFAULT_COLUMN "Asks"

//struct to store in the state typedef struct {

464


double *values; xmim_cfunc_date_time **dates; int size; } pred_result;

//specify the input parameters and the stateextern "C" int cfunc_instance_init_defaults (xmim_cfunc_args *args) { int num = 2; //security args xmim_cfunc_security_args *sec = new xmim_cfunc_security_args; sec->size = num; sec->names = new char*[num]; sec->names[0] = PRED_STOCK1; sec->names[1] = PRED_STOCK2; sec->rels = new char*[num]; sec->rels[0] = NULL; sec->rels[1] = NULL;



state->names = new char*[num]; state->names[0] = PRED_TIME_SERIES; state->names[1] = PRED_RESULT;


return 1;}

//initialize the stateextern "C" int cfunc_instance_init (xmim_cfunc_args *args) { char *relname1 = NULL, *relname2 = NULL; void *series1 = NULL, *series2 = NULL; xmim_cfunc_date_time *from_date = NULL, *to_date = NULL, *date = NULL, *old_date = NULL; void *offset; pred_result *result = NULL; int size = 0;

//first input argument relname1 = (char *)xmim_cfunc_get_arg (args, P_SECURITY, PRED_STOCK1); if (!relname1) return 0;

//second input argument relname2 = (char *)xmim_cfunc_get_arg (args, P_SECURITY, PRED_STOCK2); if (!relname2) return 0;

465

//create the time series for the first stock symbol series1 = xmim_cfunc_get_time_series (args, relname1, DEFAULT_COLUMN); if (!series1) return 0; //store the first time series in the state if (!xmim_cfunc_set_arg (args, P_STATE, series1, PRED_TIME_SERIES)) return 0;

//create the time series for the second stock symbol //will get deleted in the ~CoCompiler () [CoCompiler::deleteExObjects] series2 = xmim_cfunc_get_time_series (args, relname2, DEFAULT_COLUMN); if (!series2) return 0;

// the from and to dates for the first time series // the date range we are going to use from_date = xmim_cfunc_get_from_date (args, series1); to_date = xmim_cfunc_get_to_date (args, series1);

//calculate the number of values we have date = xmim_cfunc_get_from_date (args, series1); size = 1;

while (!xmim_cfunc_date_time_equal (args, date, to_date)) { offset = xmim_cfunc_get_time_offset (args, 1, PL_DAYS, PL_LATER); old_date = date; date = xmim_cfunc_apply_time_offset (args, date, series1, offset); xmim_cfunc_free_date_time (args, old_date); xmim_cfunc_free_time_offset (args, offset); size++; }

//compute the result result = new pred_result; result->values = new double[size]; result->dates = new xmim_cfunc_date_time*[size]; result->size = size; double v1, v2, prev_v1, prev_v2; //the first value is a NaN since we do not have the //value on the previous date double zero = 0; double *doubleNan = &zero; xmim_cfunc_makeNan (args, doubleNan); result->values[0] = *doubleNan; result->dates[0] = from_date; prev_v1 = xmim_cfunc_get_value (args, series1, from_date); prev_v2 = xmim_cfunc_get_value (args, series2, from_date); //compute the values using the two time series for (int i = 1; i < size; i++) { offset = xmim_cfunc_get_time_offset (args, i, PL_DAYS, PL_LATER); date = xmim_cfunc_apply_time_offset (args, from_date, series1, offset); v1 = xmim_cfunc_get_value (args, series1, date); v2 = xmim_cfunc_get_value (args, series2, date); result->values[i] = v1 + (0.5 * v1 *

466


(((v1 - prev_v1) / prev_v1) + ((v2 - prev_v2) / prev_v2))); result->dates[i] = date; prev_v1 = v1; prev_v2 = v2; xmim_cfunc_free_time_offset (args, offset); }

//store the result in the state if (!xmim_cfunc_set_arg (args, P_STATE, result, PRED_RESULT)) return 0;

return 1;}

extern "C" int cfunc_instance_compute_trading_pattern (xmim_cfunc_args *args) { void *time_series = xmim_cfunc_get_arg (args, P_STATE, PRED_TIME_SERIES);



extern "C" int cfunc_instance_compute_trading_date_range (xmim_cfunc_args *args) { void *time_series = xmim_cfunc_get_arg (args, P_STATE, PRED_TIME_SERIES);



extern "C" int cfunc_instance_get_value (xmim_cfunc_args *args, xmim_cfunc_date_time *dt) { pred_result *result = (pred_result *)xmim_cfunc_get_arg (args, P_STATE, PRED_RESULT); xmim_cfunc_date_time *date = NULL;

if (!result) return 0;

for (int i = 0; i < result->size; i++) { date = (xmim_cfunc_date_time *) result->dates[i]; if (xmim_cfunc_date_time_equal (args, date, dt)) { args->result = result->values[i]; return 1; } } //we do not have the requested date - return NaN.

467

double zero = 0; double *doubleNan = &zero; xmim_cfunc_makeNan (args, doubleNan); args->result = *doubleNan;

return 1;}

extern "C" int cfunc_instance_destroy (xmim_cfunc_args *args) { delete[] args->security_args->rels; delete[] args->security_args->names; delete args->security_args; args->security_args = NULL; //the RelCol object pointed to by args->state->ptr[0].obj //has already been deleted for us //in the ~CoCompiler () [CoCompiler::deleteExObjects]

pred_result *result = (pred_result *)xmim_cfunc_get_arg (args, P_STATE, PRED_RESULT); if (result) { for (int i = 0; i < result->size; i++) delete result->dates[i];

delete[] result->dates; delete[] result->values;

delete result; }

return 1;}

Complete Lazy Predictor Code#include "include/xmim_cfunc_api.h"#include <stdio.h>

#define PRED_STOCK1 "stock1"#define PRED_STOCK2 "stock2"#define STOCK1_TIME_SERIES "stock1_series"#define STOCK2_TIME_SERIES "stock2_series"

#define DEFAULT_COLUMN "Asks"

extern "C" int cfunc_instance_init_defaults (xmim_cfunc_args *args) { int num = 2; //security args xmim_cfunc_security_args *sec = new xmim_cfunc_security_args; sec->size = num; sec->names = new char*[num]; sec->names[0] = PRED_STOCK1;

468


sec->names[1] = PRED_STOCK2; sec->rels = new char*[num]; sec->rels[0] = NULL; sec->rels[1] = NULL;



state->names = new char*[num]; state->names[0] = STOCK1_TIME_SERIES; state->names[1] = STOCK2_TIME_SERIES;


return 1;}

extern "C" int cfunc_instance_init (xmim_cfunc_args *args) { char *relname1 = NULL, *relname2 = NULL; void *series1 = NULL, *series2 = NULL;

relname1 = (char *)xmim_cfunc_get_arg (args, P_SECURITY, PRED_STOCK1); if (!relname1) return 0;

relname2 = (char *)xmim_cfunc_get_arg (args, P_SECURITY, PRED_STOCK2); if (!relname2) return 0;

series1 = xmim_cfunc_get_time_series (args, relname1, DEFAULT_COLUMN);

if (!series1) return 0; if (!xmim_cfunc_set_arg (args, P_STATE, series1, STOCK1_TIME_SERIES)) return 0; //will get deleted in the ~CoCompiler () [CoCompiler::deleteExObjects] series2 = xmim_cfunc_get_time_series (args, relname2, DEFAULT_COLUMN); if (!series2) return 0; if (!xmim_cfunc_set_arg (args, P_STATE, series2, STOCK2_TIME_SERIES)) return 0;

return 1;}

extern "C" int cfunc_instance_compute_trading_pattern (xmim_cfunc_args *args) { void *time_series = xmim_cfunc_get_arg (args, P_STATE, STOCK1_TIME_SERIES);

if (!time_series)


469

return 0;


extern "C" int cfunc_instance_compute_trading_date_range (xmim_cfunc_args *args) { void *time_series = xmim_cfunc_get_arg (args, P_STATE, STOCK1_TIME_SERIES);



extern "C" int cfunc_instance_get_value (xmim_cfunc_args *args, xmim_cfunc_date_time *dt) { xmim_cfunc_date_time *from_date = NULL; void *series1 = xmim_cfunc_get_arg (args, P_STATE, STOCK1_TIME_SERIES); void *series2 = xmim_cfunc_get_arg (args, P_STATE, STOCK2_TIME_SERIES);

if (!series1 || !series2) return 0;

//return a NaN for the first date from_date = xmim_cfunc_get_from_date (args, series1); if (xmim_cfunc_date_time_equal (args, from_date, dt)) { double zero = 0; double *doubleNan = &zero; xmim_cfunc_makeNan (args, doubleNan); args->result = *doubleNan; return 1; }

//compute the result double v1, v2, prev_v1, prev_v2; xmim_cfunc_date_time *date = NULL; void *offset = NULL; offset = xmim_cfunc_get_time_offset (args, 1, PL_DAYS, PL_AGO); date = xmim_cfunc_apply_time_offset (args, dt, series1, offset); prev_v1 = xmim_cfunc_get_value (args, series1, date); prev_v2 = xmim_cfunc_get_value (args, series2, date); v1 = xmim_cfunc_get_value (args, series1, dt); v2 = xmim_cfunc_get_value (args, series2, dt); xmim_cfunc_free_date_time (args, date); xmim_cfunc_free_time_offset (args, offset);

args->result = v1 + (0.5 * v1 *

(((v1 - prev_v1) / prev_v1) + ((v2 - prev_v2) / prev_v2)));

470


return 1;}

extern "C" int cfunc_instance_destroy (xmim_cfunc_args *args) { delete[] args->security_args->rels; delete[] args->security_args->names; delete args->security_args; args->security_args = NULL; //the RelCol objects pointed to by args->state->ptr[0].obj // and args->state->ptr[1].obj //has already been deleted for us //in the ~CoCompiler () [CoCompiler::deleteExObjects]

return 1;}

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 11: MIM/SQL External Database Implementation

471

CHAPTER11

MIM/SQL External DatabaseImplementation

IntroductionAn external database implementation allows the Commodity DataServer to connect to a remote data source toaccess data that does not reside in a typical Commodity DataServer database. Examples of remote data sources arerelational databases, a set of files on disk, or a web service. This implementation does not copy data between theexternal source and Commodity DataServer. It goes to the external data source each time it is requested. It is up tothe implementor to create a caching mechanism in their C++ implementation to improve performance by reducingcalls to the remote data source.

To implement a remote database, a set of abstract methods must be overridden. There are approximately 50 methodsthat need to be implemented in order to implement access to remote data source. A remote database can be read-only or it can be read-write depending on which methods are implemented. The external database methods areimplemented in C++ and are compiled into a shared library. This shared library is then loaded by the CommodityDataServer on startup.

Key Commodity DataServer ConceptsThe data storage objects in a Commodity DataServer are time series. Every numeric value has an associate date-timekey. The time series are identified with relation and column names. The relation name is usually the symbol, IBM, andthe column is the field, Close. For example the time series of the official close of IBM would be “Close of IBM”.

Time series are physically stored at three resolutions: daily, intra-day and tick. Daily time stamps have no hour orminutes. Intra-day covers the range to 1-minute increments. Tick covers all time stamps to 1 second of resolution.Below 1 second, the data is stored as a FIFO.

In order to efficiently store a large quantity of time series, the Commodity DataServer employs a hierarchy andinheritance of meta-data. For example, all equities class data is stored in a category called Equities. This allowscolumns to be added to the parent category that is immediately applied to the children, in this case the 12,000individual equity symbols.

A Commodity DataServer can have multiple databases of either standard Commodity DataServer types or external.All requests are processed by the database order in the .xmimrc file or as specified by the clients database narrow

472


commands. Therefore, multiple external database connectors could access a large external SQL database. This cansimplify the coding of the implementation.

ImplementationThe external database is connected to the Commodity DataServer via a Solaris dynamic link library. The CommodityDataServer slave server processes load this library and the routines are mapped to the external database entry pointsvia the .spc file configuration. The dynamic library will be built with C linkage.

Accessing Time SeriesThe getData routine moves all time-series data from the external database into the Commodity DataServer.

extern "C" int getData (void* handle, int relation_id, int column_id, XmimUnit unit, int* num_records, XmimDateTime** dates, XmimByte** values)

Inputs: The first handle is the generic handle of the external database. This handle is used to store data acrosscalls for the external database, for example the OCI connection handle. The next arguments are relation and columnidentifiers. These identifiers are assigned by the external database and will be discussed in the following sections onthe schema. The unit argument is a constant to indicate the time granularity needed by the Commodity DataServer:daily, intraday or tick.

Outputs: With the exception of the status code and number of records, all of the return arguments are passed via heapallocated (malloc) buffers. The number of records is the length if the XmimDateTime vector. XmimDateTime are thetimestamps. This is a C struct with slots for year, month, day, hour, minute and second and millisecond. Finally, thedata is returned via an XmimByte pointer. The actual data is in native SPARC types: int, float and double. The datais ordered by column and then date. Values must also be properly aligned. Typically, in the code the return vector isallocated as a float * array and cast into a byte pointer.

A sample SQL table could be structured as follows for daily data:

create table tsdata (relation_id number(9) not null, column_id number(9) not null, trans_date DATE, val number(8,6))


473

This would be accessed in the C code by the query:“select trans_date, val from tsdata where relation_id=%d and column_id=%d order by trans_date”, relation_id, column_id

Accessing Meta Data InformationIn order for the Commodity DataServer to generate requests on the time series, the external database must generatetime-series meta data. This section discusses a minimal approach to generating meta data. Please note that therelation and column hierarchies are completely similar in the internal structure. The getData routine consumesrelation and column identifiers. These identifiers can be mapped using the following table.create table tsrelmeta (relation_id, number(9), parent_id number(9), name varchar(255), char rel_type, description varchar(1024))

The key query maps the string names to relation ids as the Commodity DataServer will always operate on relation id.This table should be properly indexed by name.

Hierarchy, Types and CategoriesCommodity DataServer data is stored using hierarchies for relations and columns. The root of the relation tree is“TopRelation”. The parent_id in the tsmeta table is used to create the parent child relationships. The parentrelations are called categories in the Commodity DataServer. The leaf relations are called “normal” relations in theCommodity DataServer. Only normal relations have data (with exceptions for futures and futures contract data).

In the external database implementation, the server implements the recursive part of the algorithm. The externaldatabase is only required to provide: the children or parents and the parents of children for 1 level of the tree at a time.

A large part of the database is built of normal and category relations. To access the futures support features of theCommodity DataServer, addition types need to be modeled. The complete set of relation types are:

0x01 = normal

0x02 = category

0x04 = futures

0x08 = contract (futures)

474


0x10 = continuous (futures)

0x20 = options

Column ModelThe column hierarchy is modeled like the relation hierarchy.

create table tscolmeta (column_id, number(9), parent_id number(9), name varchar(255), char col_type, description varchar(1024))

If the data is stable enough, then the column hierarchy can be omitted for constant hierarchy in code. The col_typeis used to indicate base or sparse data. Sparse data is forward filled on all requests regardless of the fill options of therequest. This is useful for interest rate data.

Relation, Column and Data PresenceRelation and Column pairs, or relcols, are the time-series identifiers. The Commodity DataServer tests each databasefor presence of a relation column pair with data. Responding to inquiries about the existence of relation column pairsefficiently is important to preserve server performance.

extern "C" int containsDataOfUnitsP (void* handle, int rel_id, int col_id, XmimUnit data_units)

A typically implementation could use the following query to answer the request:

Select count(*) from tsdata or a select trans_date having count(trans_date) > 1. Some SQL systems havespecial syntax for quickly testing for the presence of rows avoiding the costly count(*) operator.

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 12: Correction Audits

475

CHAPTER12

Correction Audits

IntroductionThis chapter covers:

● How to turn on a log of correction audits.● How to add, retrieve, and remove daily and intraday corrections to the Table Facility.

Morningstar Commodity Data uses correction audits to keep an audit trail (or log) of changes made to historical datapoints. Clients can use correction audits to examine when and what data points in the database were modified overtime. It can also be used to pose queries to see how data would have appeared on a particular day. Internally, thecorrections audit trail is stored in the Commodity DataServer Table Facility. The Table Facility is a generic storage areaused to hold data that is not time-series in nature.

The following instructions are also documented in the respective API chapters. Please see the following: “CorrectionAudits”, “Adding Correction Audits to the Table Facility” and Chapter 9, “VBA API”.

Correction AuditsRegardless of whether or not you enable the corrections auditing log, Morningstar Commodity Data will always applythe corrected data points to the databases. Note that we do not enable the corrections auditing log by default. Toenable this feature, please contact a Client Services Representative.

Adding Correction Audits to the Table FacilityTo add a correction audit record to the Table Facility within the database, either BMIM or the Commodity DataServerAPIs may be used. The following instructions show how to add daily and intraday corrections using BMIM and the C/C++ API.


476


BMIM SynopsisThe synopsis for the BMIM command is:lock_filescorrections_add filename

The filename argument is the name of the file containing the corrections audit data.

Commodity DataServer C/C++ API SynopsisThe synopsis for the Commodity DataServer C/C++ API call is as follows:XmimReturnCode XmimAddCorrections (XmimClientHandle handle, XmimString fileName);XmimReturnCode XmimVaAddCorrections (XMIM_CLIENT_HANDLE, handle, XMIM_FILENAME, fileName, XMIM_END_ARGS);

The XmimAddCorrections routine is used to add daily and intraday correction audit data to the database. The filespecifying the corrections data must be composed of one correction per line. Correction audit data is of the followingform for intraday data:20030422, 0000, APX.DAHOURLY, Val, 20030421, 1400, 1.25

Where this line indicates that on April 22, 2003 a corrected value for Val of APX.DAHOURLY on April 21, 2003 at 2:00PM was placed in the database and the previous value for that series was 1.25.



Once correction audits have been added to the database, data may be accessed in such a manner as to take intoconsideration any corrections made through a specified date, but none made after that date. See “Accessing Datawith Correction Audits Applied” for a description of accessing data with a date specifying which corrections should beapplied.

The Table Facility stores the correction audit information. Thus, all of the routines provided and available with thatfacility can be used such that corrections information can be retrieved, modified or deleted. The name of the daily tableused is XMIM_CORRECTIONS and the field descriptions are provided below:const XmimTableFieldDesc corrections_schema[5] = {{"Date", XMIM_TAB_FIELD_DATE}, {"Relation", XMIM_TAB_FIELD_ATOM}, {"Column", XMIM_TAB_FIELD_ATOM}, {"OldDate", XMIM_TAB_FIELD_DATE}, {"OldValue", XMIM_TAB_FIELD_DOUBLE}};


477

The name of the intraday table used is XMIM_CORRECTIONS_INTRADAY_YYYYMMDD where YYYYMMDD representsthe date the correction was reported (i.e., if provided a report date of 20030421, it would create and/or open a tablenamed XMIM_CORRECTIONS_INTRADAY_20030421 in the table facility). The field descriptions are provided below:

const XmimTableFieldDesc corrections_schema[7] = {{"Date", XMIM_TAB_FIELD_DATE}, {"Time", XMIM_TAB_FIELD_TIME}, {"Relation", XMIM_TAB_FIELD_ATOM}, {"Column", XMIM_TAB_FIELD_ATOM}, {"OldDate", XMIM_TAB_FIELD_DATE}, {"OldTime", XMIM_TAB_FIELD_TIME}, {"OldValue", XMIM_TAB_FIELD_DOUBLE}};

For the Commodity DataServer Visual Basic for Applications API, two properties in the XMIMRecords class handlespecifying and retrieving the name of the corrections file:

Property Let CorrectionsFile(s As String) Property Get CorrectionsFile() As String

Once the corrections file has been set, the following method can then be used to add the corrections specified in thefile to the database.

Sub AddCorrections()

Retrieving Correction Audits from the Table FacilityUse the Commodity DataServer C/C++ API to retrieve correction audit data from the Table Facility within thedatabase. The following instructions show how to retrieve daily and intraday corrections.

XmimGetCorrections() opens, creates, and closes the appropriate table(s) based on parametersprovided, therefore XmimOpenDatabase(), XmimNewDatabase() and XmimCloseDatabase()would not be required.

478


Commodity DataServer C/C++ API SynopsisThe XmimGetCorrections routine is used to return correction audit data for a particular relation and column:XmimReturnCode XmimGetCorrections (XmimClientHandle handle, XmimString relName, XmimString colName, XmimDate fromDate, XmimDate toDate, XmimTime fromTime, XmimTime toTime, XmimCorrectionType units, int limit, int *numRecords, XmimDateTime **reportedDates, XmimDateTime **correctedDates, double **value);

The relName and colName arguments are required fields that specify the single relation and column from which toget data. The dates (fromDate and toDate) allow the caller to specify the appropriate range for the data. This rangecan be as small as desired. Likewise, the times (fromTime and toTime) specify the trading time range for each datein the date range (applies to intraday corrections only). The units argument specifies the data type to retrieve (e.g.,daily, intraday).

The XmimCorrectionsType type is defined as follows:typedef enum { XMIM_CORRECTIONS_DAILY, XMIM_CORRECTIONS_INTRADAY} XmimCorrectionType;

If units is specified as XMIM_CORRECTIONS_INTRADAY then query would look in only intraday tables and likewisefor XMIM_CORRECTIONS_DAILY would look only in the daily corrections table. The limit argument specifies themaximum number of records to return. A value of 0 or less would indicate to retrieve all records, where a 'positive'value would set a limit of records to the provided value.

The dates (reportedDates and correctedDates) and values arrays are used to return the records found. Thesedates and values correspond to the relation and column specified for the given date and time. The number of records(numRecords) will be exactly the number of entries in the dates and value arrays.

Deleting Correction Audits from the Table FacilityThe following instructions show how to delete correction audit logs from the Table Facility using the API.

XmimDeleteCorrections() opens, creates, and closes the appropriate table(s) based on parametersprovided, therefore XmimOpenDatabase(), XmimNewDatabase() and XmimCloseDatabase()would not be required.


479

Commodity DataServer C/C++ API SynopsisThe XmimDeleteCorrections routine is used to delete correction audit data for a particular relation and column:XmimReturnCode XmimDeleteCorrections (XmimClientHandle handle, XmimString relName, XmimString colName, XmimDate fromDate, XmimDate toDate, XmimTime fromTime, XmimTime toTime, XmimCorrectionType units, int *numRecords);

The relName and colName arguments (required fields) specify the single relation and column from which to get data.The dates (fromDate and toDate) specify the range for the data. This range can be as small as desired. The times(fromTime and toTime) specify the trading time range over which to delete data for each date in the date range(applies to intraday corrections only). The units argument specifies the data type to retrieve (e.g., daily, intraday).

The XmimCorrectionsType type is defined as follows:typedef enum { XMIM_CORRECTIONS_DAILY, XMIM_CORRECTIONS_INTRADAY} XmimCorrectionType;

If units is specified as XMIM_CORRECTIONS_INTRADAY then the query would look only in the intraday tables andlikewise XMIM_CORRECTIONS_DAILY would look only in the daily corrections table.

The number of records (numRecords) deleted is returned for the specified relation and column for the given date/timeframe.

Correction Audits File FormatThe file specifying the corrections audit data must be composed of one record per line. The correction audit file cancontain daily or intraday records. Each record contains the previous value (not the current corrected value).

The intraday corrections audit data is of the following form:20030415, 0000, APX.DAHOURLY, Val, 20030322, 1400, 1.25 Transaction Date : April 15, 2003 Transaction Time : 12:00 AM Ticker Symbol : APX.DAHOURLY Value to be Correction : Val Date to be Corrected : March 22, 2003 and Previous Date : March 22, 2003 Time to be Corrected : 2:00 PM Previous Value : 1.25

480


Indicating that on April 15, 2003 at 12:00 AM a corrected value for Val of APX.DAHOURLY on March 22, 2003 at 2:00PM was placed in the database and the previous value for that series was 1.25.


For daily data corrections, use the format:

20030415, APX.DADAILY, Val, 20030322, 1.25

Accessing Data with Correction Audits AppliedOnce corrections have been added to the database, data may be accessed in such a manner as to take intoconsideration any corrections made through a specified date, but none made after that date. This capability is availablewith both the Commodity DataServer C/C++ and the Commodity DataServer Visual Basic for Applications APIs.

In the Commodity DataServer C/C++ API, this is accomplished using the XmimVaGetRecords orXmimVaGetRecordsAllColumns routines. The argument with the name XMIM_CORRECTIONS_DATE is availablewith these routines. When this date is used, values are accessed as they would have appeared on that date. That is,if a correction was added on June 5, 1997 and the XMIM_CORRECTIONS_DATE specified in the XmimVaGetRecordscall is before June 5, the old value will be returned; on the other hand, if the date specified is after June 5 then thenew corrected value will be returned. The synopses for the two routines is provided below:

XmimReturnCode XmimVaGetRecords (XMIM_CLIENT_HANDLE, handle, XMIM_RELATION_LIST, relName1, relName2, ..., NULL, XMIM_COLUMN_LIST, NULL, XMIM_FROM_DATE, XMIM_INVALID_DATE, XMIM_TO_DATE, XMIM_INVALID_DATE, XMIM_FROM_TIME, XMIM_INVALID_TIME, XMIM_TO_TIME, XMIM_INVALID_TIME, XMIM_UNITS, 1, XMIM_DAYS, XMIM_CORRECTIONS_DATE, XMIM_INVALID_DATE, XMIM_FILL_OPTION, XMIM_FILL_NAN, XMIM_FILL_NAN, XMIM_SKIP_NONE, XMIM_LIMIT, 0, XMIM_LIMIT_BY_RECORDS, XMIM_CURRENT_TICK_USAGE, XMIM_APPEND_TO_NONE, XMIM_NUM_RECORDS, &numRecords, XMIM_DATE_TIMES, &dates, XMIM_VALUES, &values, XMIM_END_ARGS); XmimReturnCode XmimVaGetRecordsAllColumns (


481

XMIM_CLIENT_HANDLE, handle, XMIM_RELATION_LIST, relName1, relName2, ...,NULL, XMIM_FROM_DATE, XMIM_INVALID_DATE, XMIM_TO_DATE, XMIM_INVALID_DATE, XMIM_FROM_TIME, XMIM_INVALID_TIME, XMIM_TO_TIME, XMIM_INVALID_TIME, XMIM_UNITS, 1, XMIM_DAYS, XMIM_CORRECTIONS_DATE, XMIM_INVALID_DATE, XMIM_FILL_OPTION, XMIM_FILL_NAN, XMIM_FILL_NAN, XMIM_SKIP_NONE, XMIM_LIMIT, 0, XMIM_LIMIT_BY_RECORDS, XMIM_CURRENT_TICK_USAGE, XMIM_APPEND_TO_NONE, XMIM_COLUMN_ARRAY, &numColumns, &colNames, XMIM_NUM_RECORDS, &numRecords, XMIM_DATE_TIMES, &dates, XMIM_VALUES, &values, XMIM_END_ARGS);

If the corrections date is not specified, then it will be taken as the last data date such that any and all corrections willhave been applied. Refer to the Commodity DataServer C/C++ API documentation for details regarding use of theseroutines.

Note that the Va version of the API routines must be used as the new argument is not available with thenon-Va version (for backward compatibility reasons).

In the Commodity DataServer Visual Basic for Applications API, two additional properties have been added to theXMIMRecords class to support factoring in the corrections when data is accessed. These two properties are:

Property Let CorrectionsDate(ByVal d as Date) Property Get CorrectionsDate() as Date

These properties are used to set or retrieve the date analogous to the XMIM_CORRECTIONS_DATE argumentdescribed above. The property is used with the GetRecords method.

Using the Table Facility to Retrieve or Update CorrectionsThe Table Facility available through the Commodity DataServer C/C++ API is used to store the correction auditinformation. Thus, all of the routines provided and available with that facility can be used such that corrections auditinformation can be retrieved, modified or deleted. See Chapter 5, “C/C++ API” for a full description of the TableFacility.

482


Daily TableThe name of the daily table used is XMIM_CORRECTIONS and the field descriptions are provided below:

const XmimTableFieldDesc corrections_schema[5] = {{"Date", XMIM_TAB_FIELD_DATE}, {"Relation", XMIM_TAB_FIELD_ATOM}, {"Column", XMIM_TAB_FIELD_ATOM}, {"OldDate", XMIM_TAB_FIELD_DATE}, {"OldValue", XMIM_TAB_FIELD_DOUBLE}};

Intraday TableThe name of the intraday table used is XMIM_CORRECTIONS_INTRADAY_YYYYMMDD where YYYYMMDD representsthe date the correction was reported (i.e., if provided a report date of 20030421, it would create and/or open a tablenamed XMIM_CORRECTIONS_INTRADAY_20030421 in the table facility). The field descriptions are provided below:

const XmimTableFieldDesc corrections_schema[7] = {{"Date", XMIM_TAB_FIELD_DATE}, {"Time",XMIM_TAB_FIELD_TIME}, {"Relation", XMIM_TAB_FIELD_ATOM}, {"Column", XMIM_TAB_FIELD_ATOM}, {"OldDate", XMIM_TAB_FIELD_DATE}, {"OldTime", XMIM_TAB_FIELD_TIME}, {"OldValue", XMIM_TAB_FIELD_DOUBLE}};

Part IV: Loading and Extracting Data

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 13: Best Practices for Data

485

CHAPTER13

Best Practices for Data

The best practices for loading/extracting data is outlined below. Each data loading/extracting method is listed alongwith a description of the circumstances in which each method is best employed.

Best Practices for Loading DataThe following shows the different methods for loading data.

Loading Data using BMIM and the Package MakerThis is the recommended method for loading Morningstar data. This method is best when you need to load data fileson a regular basis.

This is the method that Morningstar Commodity Data employs to load and send data to our customers.

This method uses the following components:

● make_data BMIM script that defines the data to be loaded.● Package Maker utility that creates update packages of the data.● load_updates.sh loads packages placed in the /home/$LIMHOME/updates directory into your Commodity

DataServer.

See Chapter 15, “Loading Data using the Package Maker” for instructions on using this data loading method. SeeChapter 14, “Software Installations for Loading Data” for information on installing the Package Maker.

Loading Data using the File LoaderThis method is best for loading data from a .csv or .xml file. You can run the File Loader from a command line or from abrowser window. This method loads the data, creates an update package and loads it to a specified location.

486


See Chapter 17, “Loading Data from a File (File Loader)” for instructions on using this data loading method. SeeChapter 14, “Software Installations for Loading Data” for information on installing the File Loader.

Loading Data using the Formula LoaderThis method is used when you want to create a formula from an 'Commodity Query' query then store the data resultsas a new symbol in the Commodity DataServer.

See Chapter 18, “Formula Loader” for instructions on using this data loading method. See Chapter 14, “SoftwareInstallations for Loading Data” for information on installing the Formula Loader.

Loading Futures DataThere are very specific naming conventions that must be followed when loading futures data. For more information,see Chapter 16, “Loading Futures Type Data (Monthly Curves)”.

Loading Data using the Commodity Add-inThis is the preferred method for loading custom data. See the Commodity Add-In (Microsoft Excel®) User's Guide forinstructions on setting up and using this data loading method.

Loading Data using the APIsThis method is best when you need your own custom solution. The following lists the APIs offered:

● C/C++● Java● .NET● WebServices

Extracting/Reading DataThe following describes client applications and tools for extracting or reading data.

http://www.morningstarcommodity.com/sites/default/files/Commodity_AddIn_User_Guide.pdf


487

Commodity Query SoftwareThe Commodity Query software is very powerful for modeling and evaluating trade opportunities. The user buildsqueries using a near-English query language. Users can either type in the query language or use the GUI interface tobuild their queries. Queries can have custom events with date and time conditions and can incorporate studies andmarket events (e.g., create moving averages, highest high, lowest low etc.) Commodity Query has many built-in reportoptions for analyzing the data results.

The user will get the most out of the software when they are comfortable with the Commodity DataServer querylanguage. For users that are unfamiliar with the query language, there are features in the software that will help themget started.

Commodity Charts SoftwareThe Commodity Charts software is very good for creating seasonal studies, charting results, and working withformulas. The software has integrated unit conversions and is excellent for working with rollovers and futures data.This software package works well for the novice as well as the advanced user. The Commodity Charts software canrun standalone or inside of Excel and the data results (charts or tables) are easily posted into Excel spreadsheets.

Commodity Add-in SoftwareThe Commodity Add-in software is a very straight forward product that is good for pulling out data into MicrosoftExcel®spreadsheets.

For details on using the Commodity Add-in software, see the Commodity Add-In (Microsoft Excel®) User's Guide

BMIM Scripting LanguageThe BMIM query_execute and print_schema commands are useful for extracting data and browsing theschema hierarchy.

Application Programming Interfaces (APIs)This method is best when you need your own custom solution. The following lists the APIs offered:

● C/C++

http://www.morningstarcommodity.com/sites/default/files/QueryLanguageOverview.pdf

http://www.morningstarcommodity.com/sites/default/files/QueryLanguageOverview.pdf



488


● Java● .NET● WebServices

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 14: Software Installations for Loading Data

489

CHAPTER14

Software Installations for Loading Data

In order to use the data loading methods described in Chapter 15, “Loading Data using the Package Maker”, Chapter 17, “Loading Data from a File (File Loader)”, and Chapter 18, “Formula Loader” follow the installation bundleinstructions outlined below. To load data via the Commodity Add-in, see the Commodity Add-In (Microsoft Excel®)User's Guide.

Product Bundle – (Formula Loader, Package Maker and FileLoader)Morningstar Commodity Data provides the following three utilities in one installation bundle:

● Formula Loader - This product creates formulas from 'Commodity Query' queries then stores the data resultsas new symbols in the Commodity DataServer. These data values are then available for use in applications such asCommodity Charts, Commodity Query, and the Commodity Add-in.

● Package Maker - Using this utility, the files in a designated directory are tarred, gzipped and put in a shell archiveformat so that the end result upd file automatically self-extracts to the location designated by the user. This toolcan be used as a backup system. If the meta data is structured correctly the data files can be applied successfullyto a database on any machine. For more information on how to setup your meta data, refer to the following: “Themake_data File” and Chapter 16, “Loading Futures Type Data (Monthly Curves)”.

● MIM File Loader - Morningstar Commodity Data has developed several different methods for inserting or loadingdata into the Commodity DataServer. Use the Package Maker to load files you want to import into the CommodityDataServer on a regular basis. The File Loader processes the data from a supplied .csv or .xml file then creates andloads the resulting upd package into a specified location.

Installation Bundle InstructionsThe following instructions are for downloading and installing the floader_x.x file. All entries must be made as the DBAuser. (The DBA user is the owner of the Commodity DataServer process.) The floader_x.x file contains the executablesfor running the Package Maker, the File Loader, and the Formula Loader utilities.

1. Download the floader_x.x file.



490

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 14: Software Installations for Loading Data

The “x”s denotes the version level and are subject to change.

2. Enter the following commands to install the software.

Execute commands as the DBA user.

$ cd $LIMHOME$ sh floader_x.x

The following shows the output from the installer, as well as the directories and files created:

This completes the installation bundle instructions.

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 15: Loading Data using the Package Maker

491

CHAPTER15

Loading Data using the Package Maker

Before creating data and working with packages, it is recommended that you configure a test system first beforeapplying changes and new data to your production database. Incorrect data loading can cause integrity issues at best,and total server failure at worst. See Chapter 3, “Database and Data Guidelines” for instructions on setting up a testdatabase.

Data PackagesIn order to load data into the Commodity DataServer, you will be creating a upd package similar to the data packagesthat your server picks up with cron_updates.sh from your FTP/HTTP account. Package names are structured asfollows:upd_krd_0_20040429

Packages always begin with upd_. This designator tells load_updates.sh that it is a package of data that needs to beloaded.

The next three letters are a package identifier. All packages that Morningstar Commodity Data distributes to customermachines have lowercase identifiers. This means any custom packages built must use an all uppercase, three letteridentifier.

The identifier may also contain numbers (i.e., WI1 or WI2) but the first character of the identifier must bea letter.

Using this naming convention will guarantee that you do not use any of the same identifiers that MorningstarCommodity Data delivers. The following package example contains proprietary Widgets Incorporated data, so it isnamed as follows:upd_WID_0_20040429

The _0_20040429 is the package sequence number followed by the current date. So, if you need to load 3 packagesof the same data during the day, you will load:upd_WID_0_20040429upd_WID_1_20040429upd_WID_2_20040429

492


This way, each time you load a package of data, you retain a unique log record which can be reviewed in case ofproblems. For more information on logs see “Loading Data Packages and Checking Logs” below.

All packages contain at least two files, the make_data BMIM script file and the file containing the actual data to beloaded in .txt (text) or .csv (comma separated value) format. The make_data script essentially tells the database howand where to load the data. Essentially, it instructs the Commodity DataServer on how the data will be represented inthe database.

Building Data Package ComponentsThis section will follow an example to illustrate the data loading process. Follow along each step to load LonestarPipeline data for the fictitious Widgets Inc. company.

Create a directory titled WID. The directory name should be the same as the uppercase, three letter package identifiername.

The identifier may also contain numbers (i.e., WI1 or WI2) but the first character of the identifier must bea letter.

$ mkdir WID$ cd WID

The WID directory will contain the make_data file that sets up the columns and categories to store your data and thecsv data file.

For this example we are going to add some Lonestar pipeline data for the Widgets Inc. Company. We need to createa WidgetsInc category to house all company specific data so that it is easy for users to find. Under that we will adda WID_Pipeline category to house all the pipeline data. Then we will create a symbol, WID.LONESTAR_1 to residein the WID_Pipeline category. We will also need to create a column for this symbol called Storage under the pre-existing Volume category.

Our hierarchy should look like this: Other WidgetsInc WID_Pipeline WID.LONESTAR_1

The make_data FileThe make_data file MUST be called make_data. If the file is not named make_data then the loader scripts will not beable to execute it.


493

$ vi make_data

Below is a sample of the required contents of a make_data file. The make_data file is composed of BMIM code thatsets up the structure of your data; creating the necessary columns and folders to store your data.

#The “database_narrow” command limits which database will be accessed.database_narrow {database = ~/data.widgets/xmim.mim;}#The “lock_files” command locks the database and is required when adding or changing data.lock_files#Commands that tell the load_updates.sh script to only log certain errors and ignore others.suppress_duplicate_addsuppress_duplicate_delete#BODY OF MAKE_DATA#This section creates a category named “Other” under the “TopRelation” category. The category is #identified by assigning the “type” as a category.relation_add{ parent = TopRelation; name = Other; type = category; description = ”Other customer data”;}#This section creates a category named “WidgetsInc” under the “Other” category. The category is #identified by assigning the “type” as a category.relation_add{ parent = TopRelation:Other; name = WidgetsInc; type = category; description = "WidgetsInc Data";}#This section creates a category named “WID_Pipeline” under the “WidgetsInc” category (created in #the section above). Use “type” to identify the entry as a “category”.relation_add{ parent = TopRelation:Other:WidgetsInc; name = WID_Pipeline; type = category; description = “WID Pipeline”;}#This section creates a symbol named “WID.LONESTAR_1” under the “WID_Pipeline” category (created #in the section above). The symbol is defined by assigning the “type” as “normal”.relation_add{ parent = TopRelation:Other:WidgetsInc:WID_Pipeline; name = WID.LONESTAR_1; type = normal; description = “WID Lonestar Pipeline”;}#This section creates a new column under the pre-existing category “Volume”. Notice that the #command is “column_add” instead of “relation_add”.column_add{

494


parent = Volume; name = Storage; description = “Storage”; type = normal;}#This section links the new symbol with the new column. This link makes the column available for #symbols.relation_column_add{ relation = TopRelation:Other:WidgetsInc:WID_Pipeline:WID.LONESTAR_1; column = Storage; type = base;}#The facts_read command tells the Commodity DataServer what file actually holds the data points to be loaded.facts_read { column = Relation,Date,Storage; file = wid_data.csv;}#The “unlock_files” command unlocks the database. This is a required field.unlock_filesdatabase_widen

Explanation of make_data file starting from top to bottom:

● database_narrow – tells the Commodity DataServer to load the enclosed data into the specified database. It isvery important that you tell it to narrow to the empty database you built. If you do not narrow to your database, theCommodity DataServer will narrow to the ~/data database by default which can cause data integrity problems.

● lock_files – Advises the database that the script is going to write. Permissions for writing are checked.

● suppress – these are commands that tell the load_updates.sh script to only log certain errors and ignore others.

● BODY – this is where you will insert all your commands to build your hierarchy and add symbols and columns.Information and examples on how to author the body of a make_data file is located in the section below. For morecoverage of BMIM see Chapter 4, “BMIM Scripting Language”.

● facts_read – this is the line that tells the Commodity DataServer what file actually holds the data points that areto be loaded. In the above example, the file that contains the actual data is the wid_data.csv file. The data file mustalso be present in the /home/lim/WID/ directory. The column = line tells the Commodity DataServer what eachcolumn of data in the .csv file represents.

For information on how to create a make_data file for futures type data, see Chapter 16, “Loading Futures Type Data(Monthly Curves)” .

Data FileThe data file can be a text file (.txt) or a .csv file. For this example the file is saved as a .csv file. Enter the data usingthe following format:


495

Relation,Date,Storage

The contents of the file will look similar to this:WID,20040429,100.45WID,20040429,100.49Etc..

The first column is the symbol you created, the second is the date the data belongs to and the third is the actual datavalues to be loaded.

Once the make_data script file and the data file (wid_data.csv) are built and ready to go, the directory will resemblethe following:WID/make_data WID/wid_data.csv

Running the Package MakerNow you are ready to package your directory into an update package. To use the Package Maker, enter the followingfrom a command prompt:$ run.packager source_dir upd_name

where:

● source_dir – is the directory containing the make_data and data files● upd_name – is the name assigned to the resulting package

For this example, build the data package as follows:$ run.packager WID upd_0_WID_20040429

This command will create the upd_0_WID_20040429 package out of your WID directory. For more information onrunning the Package Maker, see Chapter 39, “Using the Package Maker”.

Loading Data Packages and Checking LogsTo load your custom data package into your database, you will need to copy it into the updates directory in your homedirectory typically located in /home/lim/updates/. Before doing this, you will need to verify that your normal updateprocess is not currently running. To check for update processes, run the following command:ps –ef | grep 4091

If there is a 4091 Commodity Query slave server process currently running for your server, wait until it is finishedbefore continuing.

496


Now that there are no update processes running, copy your data package to the updates directory:

cp upd_0_WID_20040429 /home/lim/updates/

You have two choices at this point. You can wait for the package to process at the server’s normal updating interval,or you can manually run the updates process now. To run the updates process, go to your home directory and executethe following:

cron_updates.sh &

The & forces the cron_updates.sh script to run in the background so that you can continue to work in your currentwindow. The cron_updates.sh script will first pickup any data packages sitting in queue in your FTP account, and thenit will execute updates (load_updates.sh) for all packages that are picked up and your custom package.

Once the updates process has finished loading data into your server, you can check the file ~/conf/load_updates.hstto verify that there were no errors encountered with your custom package. The file will contain the last exit statusof load_updates.sh. If all worked will, the status will be 0. Any other status indicates a warning or error. If the statusindicates a warning or error then proceed with the following log checks to locate the problem.

A log file is created each time the Commodity DataServer runs the update process. The log file naming convention islog. appended to the package number and the date of the package. So, upd_0_20040429 would create the log filelog.0_20040429. Use an editor such as vi to search the log for the words error or fatal to locate any problems whileloading the data.

The following command will return the package name if there are any errors:

$ egrep -il error updates/logs/log_0.20040429

If there are errors located, the command will return the following as output:

updates/logs/log_0.20040429

It simply repeats the package name you entered. If there are no errors, then you will get nothing. Repeat the same forfatal:

$ egrep -il fatal updates/logs/log_0.20040429

If there were no errors, your data will be loaded onto your Commodity DataServer in the manner in which youspecified. If errors are encountered, the problems will be specified in the associated log file. For help witherrors you do not understand, please contact a Client Services Support Representative at [email protected].



COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 16: Loading Futures Type Data (Monthly Curves)

497

CHAPTER16

Loading Futures Type Data (MonthlyCurves)

OverviewThe Commodity DataServer stores futures data in a special way so as to allow analysis of continuations andadjustments to the rollover dates and rollover policies of the data. Customers may wish to take advantage of theseproperties when storing their own data in the Commodity DataServer. This can be done as described in this section,but you must make sure that the data you want to store is of the appropriate shape. Futures data in the CommodityDataServer has a fixed pattern of quotations – normally monthly continuous contracts. Futures data in the CommodityDataServer has very specific naming conventions that must be followed. Provided the user is willing to accept theserestrictions and can massage their data into the required shape, loading forwards data using the futures type will workwell.

For more information on futures contracts, see the document Rollovers .

This document describes:

● Futures data concepts● How futures data is structured in the Commodity DataServer● Special naming conventions for futures type data● How to create a make_data file for futures type data● Example of make_data file for creating the categories, symbols and columns● Example of make_data file for creating the continuous futures symbols● How to create the data file containing the futures type data● How to check the results


498


Futures Data Concepts

Futures Exchanges

Futures exchanges operate as mechanisms to transfer various risks associated with exposure to commodity pricesfrom those engaged in the physical handling of the commodities to the financial marketplace. The principal device of afutures exchange is the making of a market or exchange for buying and selling contracts for delivery and purchase ofcommodities at various times in the future, usually for cash.

Futures Contracts

Futures contracts are always for quite specific delivery periods in the future and they typically have a lifespan ofabout 2 years or so. So if you look at what is trading on a futures exchange at any one time, you get a list of differentcommodities and then for each commodity a number of “contracts” for delivery at different times into the future.

Of course, all the data from the futures exchanges are stored in the database as they are quoted by the exchange. Thepattern of this data is a kind of never ending cycle of delivery contract pricing, with each contract having a “life” of apre-determined number of months depending on the commodity and the marketplace. So the main set of data you getare relatively short-lived time-series representing prices for a particular delivery period.

These delivery contracts are represented in the Commodity DataServer by a Contract parent (for example, for NYMEXLight Crude Oil, the symbol parent is CL) then a year and a month code.

For example, CL_2003Q is the symbol for the August 2003 contract for NYMEX crude (CL). Please see “Futures TypeData Naming Conventions” for a description of the default month symbols.

Our example of CL_2003Q – the August 2003 Crude contract, first started trading back in February 2001 and willcease trading as it approaches the delivery month – August 2003. Each contract ceases quoting a short while beforethe delivery period commences, when it is deemed to have “expired”. The expiration date is an important concept infutures trading and is also known as the final trading day. The expiration date for CL_2003Q is July 22, 2003.

For analysis of data history, delivery month contracts like CL_2003Q have quite specific value – when you wantto compare the prices for delivery at a particular time of year or the spread between different delivery months, forexample.

Continuous Derived ContractsOn any given day, the principal interest to many analysts is not a specific delivery month, but rather a “relative”delivery month – relative, that is, to the date when prices were published.

The concept of relative data relationships means that you can create a historical series of prices concerning deliveryone or two months into the future (for example) all the way back to the inception of the futures commodity, simply byestablishing rules that determine which actual delivery month price to look at relative to the publication date.


499

The resultant series are known as “continuation” series, because they represent continuous prices, not subject toexpiration dates like the individual delivery month contracts.

In the Commodity DataServer the default continuation futures series are known as the “front”, “back” and “far”continuations, representing quotes for delivery for one, two and three delivery periods out from the publication date.The Commodity DataServer also calculates continuation futures series for many further periods out into the future,depending on the number of delivery months quoting for a particular commodity over time.

SummationTake note of three important distinctions when loading futures data:

1. Parent-Child Relationship - Futures data in the Commodity DataServer is structured where there is a parentfutures symbol (for example CL - Crude Oil, Light Sweet Futures) and a subcategory containing the futurescontracts (CL_1983M, CL_1983N etc.). The graphic below shows the symbol CL (NYMEX Light Crude Oil) and aportion of the CL futures contracts. This parent-child relationship is advantageous when creating queries using theCommodity Query or Commodity Charts software. If the futures symbol CL is queried, the front prompt contract isused as the default.

2. Relationship Between the Contracts - Each contract has an expiration date. Expiration dates for successivecontracts usually occur at the same time of the month, but the date varies depending on weekends and holidays.

For Example:

Let’s say the query is “the last trading day will be one business day after the third Wednesday of the contractmonth”. This means that the expiration date will most likely fall on a Thursday, unless that Thursday is a holiday,then the date will fall on a Friday. An expiring November 2007 contract would stop trading on Friday, November23, because of Thanksgiving falling on Thursday, November 22.

500


3. Continuous Derived Series - Continuous derived series are created from the futures contracts in the CommodityDataServer. The graphic below shows the symbol CL (NYMEX Light Crude Oil) and a portion of the CL continuouscontracts.

Getting StartedBefore you begin creating scripts, it is a recommended to layout how the data is going to appear in the CommodityDataServer. Decide how many categories, symbols and columns are needed and how they will be structured. It isnecessary to follow the exact naming conventions outlined in “Futures Type Data Naming Conventions” below.

For this example we are going to add some futures data for the company Widgets Inc. We need to create thecategory Other then the category WidgetsInc to house all Widgets Inc. specific data so that it is easy for users tofind. Under that we will add the WID_Forwards category to house all forwards data. Under that we will add theWID_Electricity category to house all electricity data. Then we will create two symbols, WID.NYISOE.ONPEAKand WID.NYISOE.OFFPEAK to reside in the WID_Electricity category. We will also need to create a column forthese symbols named Close.

Our hierarchy should look like this:

TopRelation:Other:WidgetsInc:WID_Forwards:WID_Electricity (category) WID.NYISOE.ONPEAK (symbol) Close (column) WID.NYISOE.OFFPEAK (symbol) Close (column)


501

For more information on data naming conventions, refer to Chapter 3, “Database and Data Guidelines”.

Futures Type Data Naming ConventionsThe system enforces that all symbols of types "futures_contract" and "futures_continuous" must have names thatbegin with the name of the futures parent followed by an underscore (i.e., _) character.

futures_contract Naming ConventionsThe required syntax for futures_contract:

1. The first part of the name must be the entire parent’s contract name

2. An underscore must follow the parent contract name

3. A 4-digit year must follow the underscore. One of the following letters must follow the year: F G H J K M N Q U VX Z

These codes represent January, February…December, respectively:

JanuaryFebruaryMarchAprilMayJuneJulyAugustSeptemberOctoberNovemberDecember

FGHJKMNQUVXZ

Also note that the letter designates the end of the symbol name.

Example: parentcontractname _YYYYM

For the symbol CL the futures contracts are:● CL_1983M● CL_1983N● CL_1983Q● CL_1983U

502


● etc.

The graphic below is from the Commodity Charts application and shows the CL futures contracts from the NYMEXexchange:

futures_continuous Naming ConventionsThe required syntax for futures_continuous is:

1. Entire parent’s symbol name (i.e., the future)2. An underscore3. Name of the continuous contract. Example: parentname_continuouscontractname

For the symbol CL the futures continuous contracts are:

● CL_02● CL_03● CL_04● etc.

The graphic below is from the Commodity Charts application and shows the CL futures continuous contracts from theNYMEX exchange:


503

Creating a Futures Type “make_data” File

Creating Categories, Symbols and ColumnsUsing a text editor, create the make_data file to establish categories, symbols, and columns. After creating themake_data file, use the Package Maker software to load the data into the Commodity DataServer. This documentoutlines the creation of the make_data and data files as well as special naming conventions to follow for futures typedata. See Chapter 15, “Loading Data using the Package Maker” for instructions on loading data using the PackageMaker.

The addition of categories and symbols is accomplished by using the relation_add command. Columns are addedusing the column_add command.#The "database_narrow" command limits which database will be accessed.database_narrow {database = ~/data.widgets/xmim.mim;}#The "lock_files" command locks the database and is required when adding or changing data.lock_files#Commands that tell the load_updates.sh script to only log certain errors and ignore otherssuppress_duplicate_addsuppress_duplicate_delete#BODY OF MAKE_DATA#Add initial folder structure;#This section creates a category named "WidgetsInc" under the existing "Other" category.#This is assigned to be a category by identifying the "type" as a "category".relation_add{ parent = TopRelation:Other; name = WidgetsInc; type = category; description = "WidgetsInc Data";}#This section creates a category named "WID_Forwards" under the "WidgetsInc" categoryrelation_add{ name = WID_Forwards; parent = TopRelation:Other:WidgetsInc; type = category;}#This section creates a category named "WID_Electricity" under the "WID_Forwards" categoryrelation_add{ name = WID_Electricity; parent = TopRelation:Other:WidgetsInc:WID_Forwards; type = category;}#This section creates a symbol named "WID.NYISOE.ONPEAK" Note that the command is "type=futures"

504


relation_add{ parent = TopRelation:Other:WIdgetsInc:WID_Forwards:WID_Electricity; name = WID.NYISOE.ONPEAK; type = futures; description = "WID Forwards, NY-ISO E, On Peak";}#This section creates a symbol named "WID.NYISOE.OFFPEAK" Note that the command is "type=futures"relation_add{ parent = TopRelation:Other:WIdgetsInc:WID_Forwards:WID_Electricity; name = WID.NYISOE.OFFPEAK; type = futures; description = "WID Forwards, NY-ISO E, Off Peak";}#Bind the column "Close" to the folder names for On & Off Peak. relation_column_add{ relation = TopRelation:Other:WIdgetsInc:WID_Forwards:WID_Electricity:WID.NYISOE.ONPEAK; column = Close;}relation_column_add{ relation = TopRelation:Other:WIdgetsInc:WID_Forwards:WID_Electricity:WID.NYISOE.OFFPEAK; column = Close;}#Add the individual month curve names.#In this section, we assign names for the futures contracts.# expiration_day = "20210131";# expiration_day = "31-January-2021";# expiration_day = "01/31/2021";# expiration_day = "01-31-2021";# expiration_day = "Jan 31, 2021";# expiration_day = "Jan. 31, 2021";# expiration_day = "January 31, 2021";relation_add{ name = WID.NYISOE.ONPEAK_2020Z; parent = TopRelation:Other:WidgetsInc:WID_Forwards:WID_Electricity:WID_NYISOE.ONPEAK; type = futures_contract; expiration_day = "20201231";}relation_add{ name = WID.NYISOE.ONPEAK_2021F; parent = TopRelation:Other:WidgetsInc:WID_Forwards:WID_Electricity:WID_NYISOE.ONPEAK; type = futures_contract; expiration_day = "20210131";}relation_add{


505

name = WID.NYISOE.ONPEAK_2021G; parent = TopRelation:Other:WidgetsInc:WID_Forwards:WID_Electricity:WID_NYISOE.ONPEAK; type = futures_contract; expiration_day = "20210228";}relation_add{ name = WID.NYISOE.OFFPEAK_2020Z; parent = TopRelation:Other:WidgetsInc:WID_Forwards:WID_Electricity:WID_NYISOE.OFFPEAK; type = futures_contract; expiration_day = "20201231";}relation_add{ name = WID.NYISOE.OFFPEAK_2021F; parent = TopRelation:Other:WidgetsInc:WID_Forwards:WID_Electricity:WID_NYISOE.OFFPEAK; type = futures_contract; expiration_day = "20210131";}relation_add{ name = WID.NYISOE.OFFPEAK_2021G; parent = TopRelation:Other:WidgetsInc:WID_Forwards:WID_Electricity:WID_NYISOE.OFFPEAK; type = futures_contract; expiration_day = "20210228";}#The "unlock_files" command unlocks the database. This is a required field.unlock_filesdatabase_widen

Creating Continuous ContractsThe following make_data file shows how to create a continuous contract name and assign the pertinent information tothe contract.

lock_filesrelation_add { name = WID.NYISOE.ONPEAK_02; parent = TopRelation:Other:WIdgetsInc:WID_Forwards:WID_Electricity:WID.NYISOE.ONPEAK; type = futures_continuous; rollover_date = "expiration day"; rollover_policy = "2 nearby actual prices";}

506


Creating the Futures Type Data File(s)The data file(s) can be in the formats .csv or .xls. Remove any blank data rows or header fields from the data file(s).For this example, our data is in the file sample_data.csv in the following format:

WID.NYISOE.ONPEAK_2020Z, 6/24/2002, 51.238WID.NYISOE.ONPEAK_2021F, 6/24/2002, 57.812WID.NYISOE.ONPEAK_2021G, 6/24/2002, 57.812WID.NYISOE.OFFPEAK_2020Z, 6/24/2002, 47.273WID.NYISOE.OFFPEAK_2021F, 6/24/2002, 45.800WID.NYISOE.OFFPEAK_2021G, 6/24/2002, 45.800WID.NYISOE.ONPEAK_2020Z, 6/24/2002, 51.238WID.NYISOE.ONPEAK_2021F, 6/24/2002, 57.812WID.NYISOE.ONPEAK_2021G, 6/24/2002, 57.812WID.NYISOE.OFFPEAK_2020Z, 6/24/2002, 47.273WID.NYISOE.OFFPEAK_2021F, 6/24/2002, 45.800

Testing the ResultsOpen the Commodity Query or Commodity Charts application and use the symbol browser to verify that the symbolsand categories were created.

For example:

1. Open the Commodity Query application.

2. Select the Search Database button.

3. Select the new categories (Other>WidgetsInc>WID_Forwards>WID_Electricity) until you see the newsymbols: WID.NYISOE.OFFPEAK and WID.NYISOE.ONPEAK.

Generating Continuous Contractsregen_continuous Command

The regen_continous command creates a continuous prompt contract from a group of underlying contracts.

Normally, continuous contracts are automatically created and updated by the Commodity DataServer. Wheneverpossible, the Commodity DataServer optimizes the updating of continuous contracts such that most types of


507

continuous contracts are updated incrementally and are not completely re-generated with each update. However,sometimes it is desirable to have the continuous contracts for a given symbol completely re-computed. For example, ifchanges are made to futures contracts and need to be recomputed than the regen_continuous command can beused for this purpose. Its synopsis is:

regen_continuous [ relname ]

Where relname is the futures symbol or path: futures symbol as in CL or TopRelation:Futures:CL, or a category(folder) containing futures symbols such as in TopRelation:Futures.

If you have a continuous contract that is spotty while the underlying contracts do not have holes then tryrunning the regen_continuous command. It usually fixes this type of problem.

The command can be run at any time and by itself. The relname argument is optional. If it is not given, all continuouscontracts will be re-computed for all futures and futures_continuous symbols in the database. This will take some timeto finish as it has to regen every continuation in the Commodity DataServer.

If relname is of type category or futures, then continuous contracts will be re-computed for all futures andfutures_continuous symbols occurring in that sub-tree of the symbol hierarchy. The continuous contracts will be re-computed immediately, wherever the regen_continuous command appears in the BMIM script. By contrast,the automatic computation/recomputation of continuous contracts are performed after the entire BMIM script isprocessed.

508


COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 17: Loading Data from a File (File Loader)

509

CHAPTER17

Loading Data from a File (File Loader)

OverviewThis chapter describes the method whereby customer data is loaded into the Commodity DataServer from a file usingthe File Loader Utility.

When loading data using the File Loader Utility, the data file used may either be a comma separated file (.csv) or anXML file (.xml). This document describes how to set up both data file formats.

When loading data using a .csv file, the data is loaded by default to the TopRelation:User:[x]:[x] categoryin the Commodity DataServer unless an alternate category path is specified. The categories underneath theTopRelation:User category are determined by the first two letters of the relation name. For example, the relationWID.LONESTAR_2 will reside in the following location in the Commodity DataServer: “TopRelation:User:w:wi”.

When loading data using an .xml file, the data location is specified in the XML file. For the example outlined below,data is loaded to the TopRelation:User:WidgetsInc:Equities:w:wi hierarchy path.

A trading pattern can not be specified using the File Loader, but if the data being loaded has values onSunday or Saturday the trading pattern will be changed from MTWThF to MTWThFSaS.

The .csv File FormatThe following describes the .csv data file format. Please see Chapter 3, “Database and Data Guidelines” for the datanaming rules and conventions:RELATION, COLUMN, DATE [TIME], VALUE, “DESCRIPTION”

Where:

● RELATION, COLUMN, DATE, VALUE are required fields. If the relation name is entered without a destinationpath, the relation is loaded to the TopRelation:User:[x]:[x] category where [x] is determined by the first twoletters of the relation name. For example WID.LONESTAR_2 will by default load to TopRelation:User:w:wi. Datamay be loaded to a specified path. The .csv file example below shows the relation WID.LONESTAR_2 loaded tothe path: TopRelation:User:WidgetsInc:Equities:w:wi.

510


● COLUMN must be a COLUMN name already existing in the Commodity DataServer.

● DATE format is: MM/DD/YYYY

● TIME(Oprtional) format is: HH24:mm. (Examples: 09:00, 13:00, 00:00)

● VALUE The decimal value for the data point.

● DESCRIPTION (Optional) Note: If the file contains multiple lines with the same relation then the description for thelast line for said relation will be used.

Use a “#”sign as the first character on a line to designate a comment. A comment line is ignored.

The following is an example of a .csv file.

The last description field overrides the first description entry. The description Lonestar Pipeline 2-WidgetsInc. will be used.

Once the data has loaded, it is not necessary to include the full path to the relation the next time data for the relationneeds to be loaded.

The XML File FormatThe following example shows how to setup an XML data file.<?xml version="1.0"?><mimload><mimloadsets><mimloadset rel="WID" col="Close"><mimrelcolinfo><mimrelinfo><reldesc desc="Relation WID"> </reldesc><relpar parent="TopRelation"> <relpar parent="User"> <relpar parent="WidgetsInc"> <relpar parent="Equities"> <relpar parent="w"> <relpar parent="wi">

511

</relpar> </relpar> </relpar> </relpar> </relpar></relpar></mimrelinfo><mimcolinfo><coldesc desc="Close Column"></coldesc><colpar parent="TopColumn"> <colpar parent="Bar"> </colpar></colpar></mimcolinfo></mimrelcolinfo><mimrows><mimrow date="20030501" [time="09:00"] val="88.888"/><mimrow date="20030502" [time="13:00"] val="89.888"/><mimrow date="20030503" [time="13:00"] val="98.888"/><mimrow date="20030504" [time="13:00"] val="78.888"/><mimrow date="20030505" [time="13:00"] val="68.888"/><mimrow date="20030506" [time="13:00"] val="58.888"/></mimrows></mimloadset></mimloadsets></mimload>

Running the File Loader

Command Prompt UsageEnter the following from a command prompt to load your data file.

Your src_file file may be a .csv file or .xml file:

$ run.file_process –f src_file –b database [-L]

where:

● run.file_process is the executable that starts the File Loader utility● src_file is your source file in .csv or .xml format.● database is the path of the database you want to load your data to.

512


Parameters Defined:

● -f (Required) The source file to be processed. Resulting packages are placed in the parent directory of src_file.● -b (Required) The database to load the data to.● -L (Optional) If present, skips running load_updates. If the –L option is absent, then load_updates.sh runs on

the resulting package.

Example for loading a .csv data file:

$ run.file_process -f myfile.csv -b ~/data.cust -L

Example for loading an .xml data file:

$ run.file_process -f myfile.xml -b ~/data.cust -L

Final Ouput for .csv ExampleUse Commodity Query or Commodity Charts to check that the data loaded successfully. The data is always loaded bydefault to TopRelation:User unless a path is specified in the .csv file. The data (symbol name WID.LONESTAR_2)for this example is located in the following hierarchy path.

TopRelation:User:WidgetsInc:Equities:w:wi

Final Output for .xml ExampleUse Commodity Query or Commodity Charts to check that the data loaded successfully. The data structure isdesignated in the .xml file. The data for this example is located in the following hierarchy:

TopRelation:User:WidgetsInc:Equities:w:wi

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 18: Formula Loader

513

CHAPTER18Formula Loader

OverviewThis document describes how formulas generated by Commodity Query as queries are stored in the CommodityDataServer as time series data. The result allows users to nest formulas and use the data output in the MorningstarCommodity Data software tools i.e., Commodity Charts, Excel and the Commodity Query client.

The installation creates the configuration file loader.cfg which contains the following default properties:KEY VALUE DescriptionTAG WID Tag name for naming upd files.PATH TopRelation:User Commodity DataServer hierarchy for loading

formulated results.DATABASE ~/data.cust/xmim.mim Database to narrow to when loading results.

If necessary, edit any of the default values in the configuration file.

Running the ProgramTo execute the program, run the run.formula_load shell script from the $LIMHOME directory as the user lim.$ run.formula_load

The program will print log messages to the $LIMHOME/formulas/logs directory.

All log files follow the naming convention: log.YYYYMMDDHHmmss

Creating Formula Query Files

Creating the QueryUse the Commodity Query software program to create a 'Commodity Query' query containing the desired formula. Usea SHOW-WHEN to create the query. REPEATED will not work in the SHOW block.

514


At the top of the query there are special comments “#” that get parsed.

Example:#formula_value index=1, relation=FORM.NG, column=Mid, description="Midpoint of NG"#formula_value index=2, relation=FORM.NG, column=CloseSHOW 1: (High of NG + Low of NG) / 2 2: Close of NGWHEN Date is within 3 months

In the example above, the following definitions apply:#formula_value index=1, relation=FORM.NG, column=Mid

where:

● #formula_value – tells the program to use this line as a formula definition.● index=1 – assigns this formula definition to the corresponding SHOW line. Looks like the label but it is really the

nth show attribute.● relation=FORM.NG – assigns the formula relation name● column=Mid – assigns a column name.● description="Midpoint of NG" – optional relation description. Available with floader2.0 or higher.

Saving the QuerySave the query in a file in the $LIMHOME/formulas/queries.

The name of the formula file must follow the naming convention: name_ordernumber.mql

where:

● name is any name to easily identify the file.● ordernumber is the sequence order the program should execute the query file in. Sequences are important when

formulas are dependent on other formulas. See “Creating Nested Formulas”.

Formula Query File Name Example:$LIMHOME/formulas/queries/johndoeformula_0.mql$LIMHOME/formulas/queries/janedoeformula_1.mql

where:

● johndoeformula and janedoeformula – names assigned by the user (follow Commodity Query namingconventions).

● _0 – user assigned name is followed by an under bar and number. The number designates the numbering order fornesting purposes.

● mql – this extension must be used for the formula loader software to recognize the file.


515

Creating Nested FormulasA nested formula is a formula that uses another formula. To create nested formulas you must create separate formulaquery files. Create each formula and then name the formula file using the ordernumber to specify to the programwhich formula file should run first.

For example. Suppose you wanted to create a formula that generates the sum of the high and low of symbol (relation)NG. Then you wanted to use that formula in another formula to calculate the midpoint for NG. First we will create thesum formula.#formula_value index=1, relation=FORM.NG.SUM, column=CloseSHOW 1: (High of NG + Low of NG)WHENDate is within 3 months

We will save this file as $LIMHOME/formulas/queries/demo_0.mql. Next, we will create the formula file to calculatethe midpoint using the sum formula.#formula_value index=1, relation=FORM.NG.MDP, column=CloseSHOW 1: FORM.NG.SUM / 2WHENDate is within 3 months

We then save this file as:$LIMHOME/formulas/queries/demo_1.mql

Finally we run the formula loader program. The program will execute the demo_0.mql file first loading the formulateddata points into the rel_col Close of FORM.NG.SUM. Then the program executes the demo_1.mql file and loadsthe results into the rel_col Close of FORM.NG.MDP.

Formula Loader ScriptThe following script runs in a shell prompt. It does not print anything out. See the log file in the formulas/log directoryfor all log file information. When the script stops executing, pending any errors, the data is loaded and can beaccessed from the server using any of the Morningstar Commodity Data software tools.$ run.formula_load

Running Queries for Multiple Relations using LETThe following examples show how to use LET statements in Commodity Query for loading multiple relations. You mayuse a file or write out the relation names. Save the query and run the Formula Loader using the steps outlined above.

516


Creating the LET QueryIn order to load multiple relations, create a LET query in Commodity Query. The first example shows how to use a fileLET statement.

At the top of the query there are special comments “#” that get parsed.

Example 1:

Using the file: /home/lim/symbols.txt. The symbols.txt file contains the following symbols: NG, CL, HO. The symbols inthe file may be separated by white space, commas or by a new line.

#formula_value index=1, relation=USER.${Xsec}, column=CloseLETXsec=File “/home/lim/symbols.txt”SHOW 1: Close of XsecWHENDate is after 01/01/2004

In the example above, the following definitions apply:

#formula_value index=1, relation=User.${Xsec}, column=Close

where:

● #formula_value – tells the program to use this line as a formula definition.● index=1 – assigns this formula definition to the corresponding SHOW line. Looks like the label but it is really the

nth show attribute.● relation=User.${Xsec} – assigns the formula relation name● column=Close – assigns a column name.

When the formula loader runs the query file, the SHOW statement runs for each symbol in the symbol.txt file (i.e., NG,CL and HO). Notice that the relation name in the #formula_value line at the top of the query file. The ${Xsec} isan identifier telling the formula loader application to rename the relation to the value of the Xsec variable. Therefore,the Close of NG will load as USER.NG and Close of CL will load as USER.CL and so on for each relation listed in thesymbols.txt file.

An alternative to using a file in the LET statement is to use a list of relations. For example:

Example 2:

#formula_value index=1, relation=USER.${Xsec}, column=CloseLETXsec=NG, CL, HOSHOW 1: Close of XsecWHENDate is after 01/01/2004

This last example shows how to use two relation names in the #formula_value line at the top of the query.


517

Example 3:

#formula_value index=1, relation=USER.${Xsec}.${Ysec}, column=CloseLETXsec=FILE “/home/lim/symbols1.txt”LETYsec=FILE “/home/lim/symbols2.txt”SHOW 1: Close of Xsec + Close of YsecWHENDate is after 01/01/2004

If the file symbols1.txt contains the symbols: NG and HO and the file symbols2.txt contains the symbols: CL and HUthen the resulting query will show:

Close of NG + Close of CLClose of NG + Close of HUClose of HO + Close of CLClose of HO + Close of HU

Final OuputThe final output is located in the folder structure:

TopRelation:User

The final output will allow derived values (formulas) based on the Commodity DataServer query language to operate asreal time-series values.

518


COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 19: Units of Measurement (UOM)

519

CHAPTER19

Units of Measurement (UOM)A user can add their own units of measurement to the data they are loading.

Perform the following steps to change the units of measurement for your data:

1. Create a file(s) using the format outlined below in sections -.

2. Specify the directory containing the file(s) in the .limrc file using the environment variableXMIM_LOCAL_UNITS.

3. Load the units of measurement by running the command units_measure_load.

Create the Units of Measurement File“MimicUM_Units” - “MimicUM_Currencies” below describe: the file name, a unit of measurement description, formatof the entries (note that the entries must be separated by tabs), then an example.

MimicUM_UnitsFilename: MimicUM_Units

Description: The file designates what each relation and column is quoted in.

File Format:Relation Column Units

File Example:XUPEZ99 Close USD/MGW

MimicUM_CategoryUnitsFilename: MimicUM_CategoryUnits

520


Description: The file designates what each category and column is quoted in.

File Format:

Category Column Units

File Example:

SOUR Close USD/BBL

MimicUM_RelColConversionFilename: MimicUM_RelColConversion

Description: The file designates a specific relation and column and the conversion formula for the specified units.

File Format:

Relation Column Unit Unit Conversion Formula

File Example:

PYBRB00 Index BBL MT 7.685

MimicUM_ConversionsFilename: MimicUM_Conversions

Description: The file contains the conversion formulas from one measurement to another.

File Format:

Unit Units Conversion Formula

File Example:

BBL GAL 1/42

MimicUM_CurrenciesFilename: MimicUM_Currencies

Description: File contains the conversion formulas from one currencies to another.


521

File Format:

Currency Currency Conversion Formula

File Example:

NOK DKK (1/(Close of NORP_DKK_NOK))

XMIM_LOCAL_UNITSAfter creating the file(s), the user must specify the location of the directory containing the file(s) in the .limrc file usingthe environment variable XMIM_LOCAL_UNITS.

Example entry in .limrc file:

XMIM_LOCAL_UNITS=/xmim/bin/

Running the “units_measure_load” CommandFrom a command prompt enter the units_measure_load command:

Usage:

units_measure_load [-q server_name] [-p port_number] [-d database]

For example:

units_measure_load -p99

522


Part V: Server Admin Guide

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 20: Invoking the Commodity DataServer

525

CHAPTER20

Invoking the Commodity DataServer

The Commodity DataServer uses an architecture whereby the workload is distributed by the main (master) server toadditional (slave) servers. In this manner, data access and query execution requests can be processed simultaneouslyinstead of the sequential first in, first out limitations of previous servers. Thus, a large request no longer needs to holdup the processing of smaller requests. With this architecture, database updates can be efficiently processed withouthindering the ability of the server to process data access requests. Moreover, multiple database writers are permittedsuch that one writer does not lock out all others. While multiple clients are allowed to write to the database at thesame time, the master server will assign only a single slave server as the historical write request processor, withall historical write requests from all clients directed to that slave server such that they are processed sequentially.Because, at any one time, only a single slave server will be allowed to write to the historical database, deployingadditional slave servers will not speed up updates to the database.

Note that when the high frequency (current tick) data facility (see "Chapter 5, “C/C++ API” ) is used, the masterserver will automatically invoke a new slave server to serve as the dedicated writer for the current tick store. Thisslave server will be distinct from any other slaves used to process access/query requests or updates to the historicaldatabase.

The architecture is best utilized by having a single master server per database, with as many slave servers as desired.Unless network access is extremely fast, it is strongly recommended that the slaves all run on the same machinewhere the database is physically located. All DataServers associated with a particular database load the schema fileinto memory using memory-mapped IO. Whenever any server modifies the schema information, all the other serverswill notice the change.

The user has the option of manually starting up slave servers or leaving it to the master server to do so as necessary.Even when it is left to the master server to invoke the slave servers, the user is responsible for specifying on whichhosts to run the slave servers and how many of them can potentially be dispatched19. This information is specifiedwhen the master server is invoked. Of course, all command line arguments are optional and the default is a singleslave server on the local host.

The server (master) is invoked as follows:

hostname% xmim_server [-h] [-v] [-p server_number] [-l log_level] [-f log_file] [-k] [-i pid_file] [-t idle_time] [-q host] [-r min max] [-w] [-M]

19 The size of the slave server executables is significant so attention should be paid to this size versus the limitations of the hardware where the servers arerunning before too many slave servers are specified.

526


where:



-p assigns a unique integer identifier, server_number, to the server. This is useful when multiple servers are desired.a For example, if thefollowing commands are given at the shell,hostname% xmim_server -p 1hostname% xmim_server -p 2

then there will be two servers running, known to the user as server 1 and server 2. Before invoking server 2, the user may wish tochange the xmimrc environment variable to point to a different .xmimrc file in order to use a different database for the second serverb.The default server number is 0.


-f specifies the file name, log file, to be used for logging. The file .xmim_server_server_number.log is the default log file where theserver_number is the one specified by the -p option (or 0 by default). So, for example, if the master server is invoked as:hostname% xmim_server -p 1 -l 1

then the log file will be .xmim_server_1.log. If the -f option is specified but not the -l option then logging will be performed using a loglevel of 0. The log files will reside in either the temporary directory specified by the user in the .xmimrc filec, the current directory (if it iswritable), or /tmp.

-k specifies that if a server already exists with the given server number, then this existing server will be (silently) terminated. By default,the existing server will not be terminated. Instead, this invocation will print a message indicating that a server with the specified servernumber already exists and will exit.


-t forces the server to terminate slave servers that are idle for more than idle_time minutes. By default, slave servers, once invoked, willnot be brought down.

-q -r is an alternative way to specify where the slave servers are to be started. The host (defaulting to localhost) is the hostnamewhere the slave servers should run, and the min and max (both defaulting to 0) specify the range of port numbers to use. Note that theslave server port numbers are distinct from the master server port numbers, hence, by default, the master server port number will be 0and so will the default single slave server used. This specification will override the -u option.

-w specifies that slave servers will be started manually instead of being invoked by xmim_server as needed. For example, two slavescould be started for one server as follows:xmim_server -wxmim_slave_server -p 1xmim_slave_server -p 2

The DataServer performs signal handling such that the following command will gracefully bring down the master server for which the pidis given and all of it's slave servers.hostname% kill -TERM pid_of_master_server

It is very important that this command be used in lieu of a kill –9 command as the master server is able to clean up after itselfwhen the kill –TERM is used.


527

Note that the xmim_svr_info utility program can also be used to gracefully kill a server.

The DataServer also processes the -INT signal which results in the master pinging all slave servers and resetting so as to remove anyslaves that have died from the list of currently used slave servers. This will allow the master server to put a slave server back in usewhen it is needed.hostname% kill -INT pid_of_master_server

-M Stops the registration with portmapper.a On a single host, there can be up to 255 master servers (server numbers 0 - 254).b The database(s) pointed to in the .xmimrc file of the user who invokes the DataServer will be loaded and used as the current database.c For a description of the .xmimrc file, see Chapter 40, “Commodity DataServer .xmimrc File”.

The slave server is invoked as follows:

hostname% xmim_slave_server [-h] [-v] [-p slave_number] [-l log_level] [-f log_file] [-i pid_file] [-n master_server_number] [-M]

where:



-p Number of the slave (will default).


-f specifies the file name, log file, to be used for logging. The file .xmim_server_server_number.log is the default log file where theserver_number is the one specified by the -p option (or 0 by default). So, for example, if the master server is invoked as:

hostname% xmim_server -p 1 -l 1

then the log file will be .xmim_server_1.log. If the -f option is specified but not the -l option then logging will be performed using a loglevel of 0. The log files will reside in either the temporary directory specified by the user in the .xmimrc filea, the current directory (if it iswritable), or /tmp.


-n Specifies the server number of the master (same as “-p” in xmim_server command).

-M Stops the registration with portmapper.a For a description of the .xmimrc file, see Chapter 40, “Commodity DataServer .xmimrc File”.

528


COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 21: Commodity DataServer Security

529

CHAPTER21

Commodity DataServer SecurityThis document details the Commodity DataServer security features. DataServer resources can be accessed in twoways, via WebServices or via RPC calls. The security features of the DataServer differ greatly for these two modes ofaccess. We will contrast the security features under both modes of operation.

Security OverviewThere are three levels of security implemented for the DataServer.

The topmost security level is at the server level. Next, the security for the database is checked, and last data levelsecurity (aka entitlements) is checked.

1. Server2. Database3. Data

These security features gives fine control over:

● Who can access the DataServer(s).

● Who can write to database(s).

● The types of data a user can read. One group may need to see data from one vendor while another may not havelicense to view the same data.

The following table shows the security levels, at what point the security is checked and how the security isimplemented using either a file, table or series of commands.Security Level Security Point ImplementationServer Connect xmimsvr.acl fileDatabase lock_files xmimlock.acl fileData facts_write, query, etc. Entitlements

Server SecurityServer level security is implemented to prevent unauthorized access to the server when a user connects to the server.

530


A check is made at the connection point to ensure that the user has proper authority to connect to theCommodity DataServers.

The xmimsvr.acl file is used to implement Commodity DataServer Server security. If the xmimsvr.acl file does not existthen no one will have access to the DataServer, with the exception of the “free pass”. A free pass means that theconnection to the DataServer is always allowed for the DataServer owner when connecting from the server/host.In other words, whoever logs into the server and is the DataServer owner has the free pass to connect to the sameDataServer.

The WebServices component also has free pass and does not need the xmimsvr.acl to gain access tothe DataServer. WebServices therefore implements its own user management facilities to restrict useraccess to the DataServer resources. Please see the Commodity Data WebServices API User's Guide forinformation on WebServices user management tools.

Please see “xmimsvr.acl File” and “Free Pass” for details on the xmimsvr.acl file and the free pass.

Global SettingThe xmimsvr.acl file global setting is “@” which allows everyone connection access to the Commodity DataServer.The system administrator can edit the xmimsvr.acl file and change who has access to specific domains and server. Ifthe xmimsvr.acl file does not exist then no one except "free pass" will have access to the DataServer.

xmimsvr.acl FileCan joe@xerces connect to the DataServer owned by joe@marimba? Can Mary connect to any DataServer at theMorningstar site? These questions are controlled by the xmimsvr.acl file. The xmimsvr.acl file contains a list of userswith permission to connect to DataServers. The xmimsvr.acl file is located on the server at: $LIMHOME/xmimsvr.acl.

There is one xmimsvr.acl file per DataServer and each xmimsvr.acl file resides in the DataServer’s home directory.

Security works by comparing the client username, hostname and domain name to the list of authorized users in thexmimsvr.acl file. This security level check affects all non WebServices clients.

The following shows the xmimsvr.acl file format:

[username]@[hostname | ipaddr[/netmask]]

where:

● arguments on either side of the @ sign are optional including username, hostname, ipaddr and netmask.



531

● IP is in the format: x.x.x.x[/n]

● “/n” is the netmask. Without a netmask, an IP refers to a specific host. With the netmask, an IP can refer to anetwork.

● hostname might include a domain name or might be only a domain name. If only a domain name, there must be a“.” in the name somewhere. An example would be “@.com”.

● the username, hostname and domain names are not case sensitive.

Example Entries:

lisa@[email protected]@maTrix.LIM.com@[email protected]@[email protected]@12.43.227.00/[email protected]@

A xmimsvr.acl file containing an @ symbol will give everyone connection access to the server.

Free PassConnection to the DataServer is always allowed for the DataServer owner when connecting from the server.

For example, suppose the user Joe starts a DataServer on the host marimba. We could say that joe@marimba ownsthe DataServer.

Now, suppose Joe tries to connect to that DataServer as a client from marimba. In other words, joe@marimba triesto connect to the DataServer owned by joe@marimba. This is always allowed.

The client joe@marimba has a free pass because he is also the DataServer owner. However, Joe cannot necessarilyconnect from a client called xerxes. (joe@xerxes can not necessarily connect to the DataServer owned byjoe@marimba.)

The following table shows some examples of when a connection is allowed.

532


DataServer Owner Client Connection Allowed?joe@marimba joe@marimba ALWAYS, free passjoe@marimba joe@xerces MAYBE, check Server ACLjoe@marimba mary@marimba MAYBE, check Server ACLjoe@marimba lim@limall MAYBE, check Server ACLjoe@marimba root@limall MAYBE, check Server ACL

If Connection is DeniedThe following shows the error message received when a user tries to access a DataServer and the connection isdenied:ERROR: User, lim@matrix, does not have permission to connect to MIM Server on matrix. Permissioned users are listed in the file, /home/lim/xmimsvr.acl, on matrix.

Database SecurityDatabase security is used to protect a database from being written to by unauthorized users. The security for thedatabase is checked during the lock_files operations and is controlled by the xmimlock.acl file in each databaseschema directory. The client’s username, hostname and domain are compared to the list of authorized users in thexmimlock.acl file. Below are features of the xmimlock.acl file:

● Each database has a separate xmimlock.acl file.● username is optional.● domain names are supported.● username, hostname, and domain names are not case-sensitive.

Default SettingThe default is set where nobody gets in if the xmimlock.acl does not exist. An exception to this rule is the “free pass”detailed below.

Free PassLocking to DataServer database is always allowed for the person who starts the Dataserver if they are running local tothe server/host and for the WebServices component.


533

The WebServices component has free pass and does not need the xmimlock.acl to lock a database.WebServices therefore implements its own user management facilities to restrict user lock/writepermissions. Please see the Commodity Data WebServices API User's Guide for information on theWebServices user management tools.

xmimlock.acl File FormatThe xmimlock.acl file contains a list of users with permission to connect to the database.

The xmimlock.acl file contains the following format for entries: [username]@[hostname | ipaddr[/netmask]]

where

● arguments on either side of the @ sign are optional including username, hostname, ipaddr and netmask.● IP is in the format: x.x.x.x[/n]● “/n” is the netmask. Without a netmask, an IP refers to a specific host. With the netmask, an IP can refer to a

network.● hostname might include a domain name or might be only a domain name. If only a domain name, there must be a

“.” in the name somewhere. An example would be “@.com”.● the username, hostname and domain names are not case sensitive.

Example Entries:

lisa@[email protected]@maTrix.LIM.com@[email protected]@[email protected]@12.43.227.00/[email protected]@

If lock_files is DeniedThe following shows the error message if the lock_files is denied:ERROR: Lock Database: Permission Denied. User, lim@matrix, does not have permission to lock database, /export/dev/lisa/data_complete/data/xmim.mim. Permissioned users are listed in


534


/export/dev/lisa/data_complete/data/xmimlock.acl.

Data Security (aka Entitlements)Data entitlements restrict read-access to individual relations or categories of relations and are stored within thedatabase schema. Without entitlements, access is controlled by “Server Security” which grants or denies read accessto all of the DataServer data. Entitlements allow the system administrators to exercise very fine control on user'sability to read specific time-series data.

For more information on setting entitlement permissions using BMIM scripts, see “Entitlements” covering BMIMcommands and entitlements. For more information on setting entitlement permissions using WebServices see Commodity Data WebServices API User's Guide.

ConceptsThe three key concepts are Users, Groups and Permissions.User: Users can be formatted as user, user@host, user@ip, user@ip/netmask or user@domainname a.

Group: A Group refers to a collection of users. Groups have names, e.g., developers (with users john and jane). A Group alsopoints to a permission set that specifies the read privileges for that group.

A user may belong to multiple groups.

When a user seeks to access a time series, the user's groups are determined. If such groups exist, then the systemchecks to see if the user has the appropriate privileges.

Finally, if the user does not belong to any defined group, the user is denied access to the system database.

Permissions: Permissions consist of a set elements of the form {relation, column, read=[1 or 0]}. Each element of this setrepresents the read permissions for a relation/column.

Example 1:TopRelation TopColumn 0

This permission denies read access on all relations and all their columns (i.e., all the possible time series) in thedatabase.

Example 2:Equities Price 1

This permission represents a read access to all the Price data of all the equities in the database.

Inheritance is supported such that, to find the permissions for a given relation column that does not have explicitpermissions, the column and relation hierarchies are traversed up until a node is found that has associated




535

permissions. For example, to find the permissions for Close of IBM, given a column hierarchy of TopColumn->Price->Close and a relation hierarchy of TopRelation->Equities->IBM, first the column hierarchy is traversed such thatif IBM Close did not have associated permissions then IBM Price would be looked at and then IBM TopColumn.After that, the relation hierarchy is walked up: Equities Close, Equities Price, Equities TopColumn, TopRelation Close,TopRelation Price, TopRelation TopColumn.

Thus, the innermost loop is the column hierarchy and the outermost loop is the relation hierarchy. To determine theaccess privileges for a given relation and column, using the algorithm described above, the most specific relation/column/read permission is obtained. This allows the system administrator to exercise very fine control over thesystem. For example the administrator can override the permissions for a sub-tree of the relations/columns hierarchyby simply specifying a permission for that relation/column. For example, suppose that the administrator wants togrant read privileges for all the data except, say Futures. The following two permissions do the trick:TopRelation TopColumn 1 TopRelation Futures 0

a Introduced with DataServer version 4.8 to support WebServices email addresses.

The following are key features of entitlements:

● Users can belong to multiple groups.

● You can "poke a hole" in the entitlements of a general category by entitling a more specific category.

● Passwords are not required for entitlements.

● Entitlements are used to set read access only. Write access is controlled by “Database Security”.

Usage GuidelinesFor best performance the following guidelines should be adhered to in using entitlements:

1. Minimize the number of groups used. The number of users per group does not affect performance so as manyusers as desired may be added without impact but the number of groups used should be kept small.

2. Minimize the number of permission tuples. Permission sets should be set for relations as high up in the relationhierarchy as possible, utilizing the inheritance feature for descendants. Exceptions can be specified as necessaryto override inherited permission sets for specific relations.See the sample permission tuples below:Group Relation Column Access .

group_A TopRelation TopColumn 0 No access for general category TopRelation

group_A IBM TopColumn 1 Access granted ("hole poked") for specific relation IBM

Free PassEntitlements has a “free pass” in place. There are three circumstances where a free-pass is given to the user therebyskipping the above checks and immediately granting access.

536


1. Entitlements has never been turned on in this database.

2. There are no entitlements users configured.

3. The user accessing data is the same user that started the DataServer.

Enabling/Disabling EntitlementsBy default, the entitlements functionality is turned off with DataServer versions prior to 4.8. With DataServer version4.8 and higher, entitlements is turned on be default. Use the following entry in the .xmimrc configuration file to turnentitlements on and off.

To turn the entitlements facility on:

entitlements: yes

To turn the entitlements facility off:

entitlements: no

If entitlements is set to “no” in the .xmimrc file, then everyone gets data access. If entitlements is set to “yes” inthe .xmimrc file, then everyone is denied data access unless specific entitlement privileges have been granted.

Even if the entitlements functionality is enabled, no entitlement checking is done if the entitlementsdatabase has not been created. In other words, if the entitlements database does not exist, then thereare no access restrictions.

DataServer version 4.8 introduces a new entry "webservice" to the .xmimrc configuration file.

webservice: yes/no

if "webservice" is set to yes then the DataServer imports the user/group associations from the WebServices. If set tono, the DataServer uses the associations obtained with the BMIM entitlement_read command. The "webservice" entryalso allows user names to be specified as email addresses in DataServer version 4.7.

If Access is DeniedThe following shows the error message if access to the data is denied:

ERROR: User is not Entitled to read BookValue of IBM in database=/home/lisa/data_complete/data/xmim.mim. User belongs to the following


537

2 Entitlement groups: group_a group_c

Entitlement File Format

This is the file format expected by the BMIM entitlement_read command. Starting with DataServerversion 4.8, the preferred method for creating the user-group association is via WebServices and notBMIM. With WebServices enabled, every 60 minutes the Commodity DataServer 'syncs' the user-groupassociation from a WebServices table. If WebServices is not accessible at the time of request, the local'snap shot' previously stored is used to entitle users.

##################################################### Sample entitlements batch file... entitlements1.txt##################################################### specify a list of groups to add, one per line after "[groups]"[groups]lim_groupgroup_agroup_call_groupall_othersentire_world# specify a list of users to add to a group, one per line after "[users]"[users]lim lim_grouplisa group_alisa group_c@matrix all_grouplisa@matrix [email protected] [email protected] [email protected]/24 [email protected] [email protected] all_others@ entire_world# specify a list of permissions to add to a group. # each line should be: group relation column read_flag# 1 means yes for read# 0 means no for read[permissions]lim_group TopRelation TopColumn 1group_a TopRelation TopColumn 1group_c Equities:i:ib FundamentalCat:Stocks 0

For Morningstar provided data, the [groups] and [permissions] sections of the file are loaded at data delivery. Clientsneeding to use entitlements with custom data (data created by client, not provided by Morningstar) should use theBMIM commands described in “Entitlements” to load the [groups] and [permissions] sections.

Starting with DataServer version 4.8, the preferred method for creating the [users] section of the file is viaWebServices. WebServices entries are synced to the DataServer with a background process which runs every 60

538


minutes. In real time, the user-group associations reflect what is on the commodity Data Server and not what is on theWebServices table.

Importing Existing EntitlementsUser-Group associations may be imported from previous Commodity DataServer versions. This section describes howto import from version 4.7 into DataServer version 4.8 or higher, running WebServices version 3.0 or higher. Pleasecontact client support if you are upgrading to a lower version of WebServices.

Follow the steps below to import user-group associations:

1. Login to DataServer 4.7 as the limih user and extract entitlements with the command:

echo "entitlements_write_all ${appserver-data}/ent_tmp" | bmim_client $PORT

Where ${appserver-data} is the actual path

2. Upgrade to Commodity DataServer version 4.8

3. Upgrade to WebServices 3.0

4. Start the DataServer 4.8 and start WebServices 3.0

5. Edit the ent_*.out files (generated from step 1). Convert user names to their email format.

6. Rename the ent_tmp folder to ent.

7. Restart WebServices 3.0 to import user-group associations.

On success, WebServices 3.0 will rename ${appserver-data}/ent as ${appserver-data}/ent.<timestamp>.tmp toavoid importing the file again on next startup.

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 22: Commodity DataServer Logging

539

CHAPTER22

Commodity DataServer Logging

IntroductionThe Commodity DataServer logging mechanism supports Common Log Format (CLF). This logging format is used byweb servers like Apache web server (found at http://www.apache.org). Consequently there are many third partyanalyzing tools available that process such log files. A few such tools are:

● wwwstat - A perl-script freeware program that does basic analyzing.

● Wusage - A commercial analyzing tool with detailed statistical output and easier customization.

● Webalizer - a fast, free web server log file analysis program. It produces highly detailed, easily configurable usagereports in HTML format, for viewing with a standard web browser. To use the webalizer enter: webalizer<logfile> where logfile is the DataServer common log format file generated by using “-l 4” in the DataServerserver startup command. See “Log Formats” for details on creating the common log format file.

The tools above are the more popular tools available that analyze logs in Common Log Format.

Log FormatsCommon Log Format is the most basic logging format. Combined Log Format an extension to Common Log Format andis the most extensive logging format "standard" that tools like Wusage support. Below is an example of Apache webserver config file entries for the common and combined log formats.

LogFormat "%h %l %u %t \"%r\" %>s %b" commonLogFormat "%h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\" \"%{Cookie}n\"" combined%h: Remote host%l: Remote logname (from identd, if supplied)%u: Remote user (from auth; may be bogus if return status (%s) is 401)%t: Time, in common log format time format%r: First line of request%s: Status. For requests that got internally redirected, this is status of the *original* request --- %s for the last.

http://www.apache.org/

http://www.ics.uci.edu/pub/websoft/wwwstat/

http://www.boutell.com/wusage

http://www.mrunix.net/webalizer/

540


%b: Bytes sent, excluding HTTP headers.%{Foobar}i: The contents of "Foobar": header line(s) in the request sent to the server.%{Foobar}n: The contents of note "Foobar" from another module.

Combined Log Format was used for DataServer logging so that as much information as possible could be extractedand because Wusage is pre-configured to read this format and generate detailed statistics. It will generate reportsusing all of the logged information.

Other log readers will be able to read the common log portion and generate reports, but will probably ignore anythingpast the Common Log Format portion of the log line.

Logging LevelsThere are currently seven different logging levels. The two explained in this document are levels 4 and 5. To useCombined Log Format for server logging, the DataServer can be run as follows:

$ xmim server -l 4

All logging levels less than 4 generate logging information the way it has previously been done by the DataServer.Logging level 5 produces the exact same output as logging level 4 except there is an additional /pre line which reportsthe command that is about to be executed. This is useful for debugging server problems; otherwise, log level 4 isrecommended as less output is generated and the analyzing tools will produce better reports.

What is LoggedWhen a query is executed by a client on a server, the server logs several things. Output would look something like thefollowing if the query below was run, one time with an API client, and then again with the Commodity Query client.

Show 1: Close of CCA

207.170.66.75 - allen [10/May/1999:21:33:59 -0500] "GET /XmimConnect" 200 0 "-" "API Client" "acrobat.0000000001"207.170.66.75 - allen [10/May/1999:21:34:00 -0500] "GET /XmimGetRelColumn" 200 0 "-" "API Client" "acrobat.0000000001"207.170.66.75 - allen [10/May/1999:21:34:01 -0500] "GET /daily/CCA/Close/10-02-1986/12-31-1998" 200 0 "-" "API Client" "acrobat.0000000001"

http://www.boutell.com/wusage


541

207.170.66.75 - allen [10/May/1999:21:34:01 -0500] "GET /XmimGetRecords" 200 0 "-" "API Client" "acrobat.0000000001"207.170.66.75 - allen [10/May/1999:21:34:01 -0500] "GET /Disconnect" 200 0 "-" "API Client" "acrobat.0000000001"207.170.66.75 - stephen [10/May/1999:21:34:43 -0500] "GET /XmimConnect" 200 0 "-" "xmim_client" "acrobat.0000000002"207.170.66.75 - stephen [10/May/1999:21:34:44 -0500] "GET /XmimGetRelType" 200 0 "-" "xmim_client" "acrobat.0000000002"207.170.66.75 - stephen [10/May/1999:21:35:07 -0500] "GET /XmimBeginQuery" 200 0 "-" "xmim_client" "acrobat.0000000002"207.170.66.75 - stephen [10/May/1999:21:35:07 -0500] "GET /XmimGetQueryParams" 200 0 "-" "xmim_client" "acrobat.0000000002"207.170.66.75 - stephen [10/May/1999:21:35:08 -0500] "GET /XmimGetNumTrades" 200 0 "-" "xmim_client" "acrobat.0000000002"207.170.66.75 - stephen [10/May/1999:21:35:08 -0500] "GET /XmimGetNextBinding" 200 0 "-" "xmim_client" "acrobat.0000000002"207.170.66.75 - stephen [10/May/1999:21:35:08 -0500] "GET /daily/CCA/Close/10-02-1986/12-31-1998" 200 0 "-" "xmim_client" "acrobat.0000000002"207.170.66.75 - stephen [10/May/1999:21:35:08 -0500] "GET /XmimEndQuery" 200 0 "-" "xmim_client" "acrobat.0000000002"207.170.66.75 - stephen [10/May/1999:21:35:15 -0500] "GET /Disconnect" 200 0 "-" "xmim_client" "acrobat.0000000002"

Wusage uses the time information (date/time) and will subtract the time differences between each command, soit is possible to determine the amount of time a query takes. Looking above, for the API client query, there is a 2second gap between the XmimConnect and the XmimDisconnect. In the xmim_client case, there is a 1 second gapbetween the XmimBeginQuery and the XmimEndQuery commands, which are the commands that signal the startand end of a query, unlike an API client which connects, makes server requests and then immediately disconnects.

There are currently six application types, (xmim_client, bmim_client, API client, VB API client, PC Excel client and PCDLL client) and it is possible for anyone to write a client application and tell the server the name of the application sothat it will be logged. The function XmimSetClientType is used for this.

The prototype is:

XmimReturnCode XmimSetClientType(const char* type);

The name to be used to identify the client is specified as an argument to XmimSetClientType. This must be donebefore calling XmimConnect.

The log reports which data was accessed, beginning with datatype. This datatype can be one of three possibleconstants: daily, intraday (minutes), or tickbytick (every trade). The complete format is:

/datatype/relation/column/start-date/end-date

Which corresponds to the example above:

"GET /daily/CCA/Close/10-02-1986/12-31-1998"

Near the end of each log line in the example above, you see something similar to acrobat.0000000001. This is calledthe cookie. If there are multiple clients running on the same machine connected to the same server, the cookierepresents each client-server connection, thus providing a way to distinguish between connections.

542


Sample ReportsThe following is a list of desirable items to report:

● How many individual applications were used.● The time information for the applications used.● Total usage by application.● Total number of accesses by user.● Total data usage for a relation over all applications.

To view the First Report see “Report 1: Web Server Statistics (Week of 05/09/99 to 05/15/99)”. The individualapplications as well as total application usage is displayed in the table "User Agents by Access Count." The totalnumber of accesses by each user is shown in the table "Authenticated Users by Access Count."

The first two graphs in the report display time information. The number of accesses at each hour, essentially the serverload, is reported.

To address the last item requires a little more digging. If you look at the table "Contents of / Sorted by Access Count"table, you will find an entry "[daily]". There are three entries of this type: "daily, intraday, tickbytick". In this samplereport the only query retrieving data was one of type "daily". If you click on this it will show a categorization of allrelations accessed and the subsequent columns and time range.

With Wusage, it is possible to filter out all log entries except those dealing with data. Here's a filtered down report ofthe previous - report 2 (see “Report 2: Web Server Statistics (Week of 05/09/99 to 05/15/99)”).

Resetting the Log FileA system administrator might like to archive the log file every so often, perhaps at the end of each week. In order toaccomplish this and have the server continue logging, a few things must be done.

First, the current log filename must be renamed. Immediately following the renaming, xmim_svr_info -n must berun. The -n switch tells the server to open a new log file since the old server logging file has been renamed. See theexample below.

$ mv $LIMHOME/tmp/.xmim_server_0.log $LIMHOME/tmp/oldLog$ xmim_svr_info -n

After the xmim_svr_info command is processed, logging to the normal log filename will resume.

If xmim_svr_info -n is not run, the server still holds the old file descriptor of the now renamedlog file. It will continue logging to that file regardless of its new name. The file descriptor is reset byxmim_svr_info which results in opening a new file with the standard log filename.


543

Reference Documents

Report 1: Web Server Statistics (Week of 05/09/99 to 05/15/99)

Overall Statistics

The time period this report refers to is not yet over.

Category TotalUnique sites served 1Unique documents served 11Unique authenticated users served 2Unique trails followed 2Total visits 2

Accesses Per Hour

Figures are averages for that hour on a typical day. The following shows the accesses per hour.

544


Hour Accesses % Bytes % Bits/Sec Bytes/Sec00:00 0.00 0.00 0.00 0.00 0.00 0.0001:00 0.00 0.00 0.00 0.00 0.00 0.0002:00 0.00 0.00 0.00 0.00 0.00 0.0003:00 0.00 0.00 0.00 0.00 0.00 0.0004:00 0.00 0.00 0.00 0.00 0.00 0.0005:00 0.00 0.00 0.00 0.00 0.00 0.0006:00 0.00 0.00 0.00 0.00 0.00 0.0007:00 0.00 0.00 0.00 0.00 0.00 0.0008:00 0.00 0.00 0.00 0.00 0.00 0.0009:00 0.00 0.00 0.00 0.00 0.00 0.0010:00 0.00 0.00 0.00 0.00 0.00 0.0011:00 0.00 0.00 0.00 0.00 0.00 0.0012:00 0.00 0.00 0.00 0.00 0.00 0.0013:00 0.00 0.00 0.00 0.00 0.00 0.0014:00 0.00 0.00 0.00 0.00 0.00 0.0015:00 0.00 0.00 0.00 0.00 0.00 0.0016:00 0.00 0.00 0.00 0.00 0.00 0.0017:00 0.00 0.00 0.00 0.00 0.00 0.0018:00 0.00 0.00 0.00 0.00 0.00 0.0019:00 0.00 0.00 0.00 0.00 0.00 0.0020:00 0.00 0.00 0.00 0.00 0.00 0.0021:00 2.00 100.00 0.00 0.00 0.00 0.0022:00 0.00 0.00 0.00 0.00 0.00 0.0023:00 0.00 0.00 0.00 0.00 0.00 0.00

Accesses Per Day

The following shows the accesses per day.


545

Date Accesses % Bytes % Bits/Sec Bytes/Sec05/09/99 0 0.00 0 0.00 0.00 0.0005/10/99 14 100.00 0 0.00 0.00 0.0005/11/99 0 0.00 0 0.00 0.00 0.0005/12/99 0 0.00 0 0.00 0.00 0.0005/13/99 0 0.00 0 0.00 0.00 0.0005/14/99 0 0.00 0 0.00 0.00 0.0005/15/99 0 0.00 0 0.00 0.00 0.00

Contents of / Sorted by Access Count

Rank File or Subdirectory Accesses % Bytes %

546


1 /Disconnect 2 14.29 0 0.002 /XmimConnect 2 14.29 0 0.003 [/daily] 2 14.29 0 0.004 /XmimBeginQuery 1 7.14 0 0.005 /XmimEndQuery 1 7.14 0 0.006 /XmimGetNextBinding 1 7.14 0 0.007 /XmimGetNumTrades 1 7.14 0 0.008 /XmimGetQueryParams 1 7.14 0 0.009 /XmimGetRecords 1 7.14 0 0.0010 /XmimGetRelColumn 1 7.14 0 0.0011 /XmimGetRelType 1 7.14 0 0.00

Top 10 of 11 Documents Sorted by Access CountRank URL Accesses % Bytes %

1 /XmimConnect 2 14.29 0 0.002 /daily/CCA/Close/10-02-1986/12-31-1998 2 14.29 0 0.003 /Disconnect 2 14.29 0 0.004 /XmimGetRelColumn 1 7.14 0 0.005 /XmimGetRecords 1 7.14 0 0.006 /XmimGetRelType 1 7.14 0 0.007 /XmimBeginQuery 1 7.14 0 0.008 /XmimGetQueryParams 1 7.14 0 0.009 /XmimGetNumTrades 1 7.14 0 0.0010 /XmimGetNextBinding 1 7.14 0 0.00

TotalsItem Accesses BytesOverall Accesses 14 0Home Page Accesses 0 0

http://www.futop.com/Disconnect

http://www.futop.com/XmimConnect

http://customers.lim.com/doc/clf/report/10355di.html#directories_/daily

http://www.futop.com/XmimBeginQuery

http://www.futop.com/XmimEndQuery

http://www.futop.com/XmimGetNextBinding

http://www.futop.com/XmimGetNumTrades

http://www.futop.com/XmimGetQueryParams

http://www.futop.com/XmimGetRecords

http://www.futop.com/XmimGetRelColumn

http://www.futop.com/XmimGetRelType

http://www.futop.com/XmimConnect

http://www.futop.com/daily/CCA/Close/10-02-1986/12-31-1998

http://www.futop.com/Disconnect

http://www.futop.com/XmimGetRelColumn

http://www.futop.com/XmimGetRecords

http://www.futop.com/XmimGetRelType

http://www.futop.com/XmimBeginQuery

http://www.futop.com/XmimGetQueryParams

http://www.futop.com/XmimGetNumTrades

http://www.futop.com/XmimGetNextBinding


547

Top 2 of 2 Authenticated Users by Access Count

Rank User Accesses % Bytes %

1 stephen 9 64.29 0 0.002 allen 5 35.71 0 0.00

Top 1 of 1 Sites by Access Count

Rank Site Accesses % Bytes %

1 207.170.66.75 14 100.00 0 0.00

Top 2 of 2 Document Trails

Rank Trail Visits Avg. Minutes1 /XmimConnect, /XmimGetRelColumn, /daily/CCA/Close/10-02-1986/12-31-1998, /

XmimGetRecords, /Disconnect1 1.00

2 /XmimConnect, /XmimGetRelType, /XmimBeginQuery, /XmimGetQueryParams, /XmimGetNumTrades, /XmimGetNextBinding, /daily/CCA/Close/10-02-1986/12-31-1998, /XmimEndQuery, /Disconnect

1 1.00

548


Visits Report

Total visits: 2Average visit: 1.00 minutesLongest visit: 1 minutesAverage accesses per visit: 7.00

2 Example Visits

User Minutes Trailacrobat.0000000001 5 /XmimConnect, /XmimGetRelColumn, /daily/CCA/Close/10-02-1986/12-31-1998, /

XmimGetRecords, /Disconnect

acrobat.0000000002 9 /XmimConnect, /XmimGetRelType, /XmimBeginQuery, /XmimGetQueryParams, /XmimGetNumTrades, /XmimGetNextBinding, /daily/CCA/Close/10-02-1986/12-31-1998, /XmimEndQuery, /Disconnect

Top 2 of 2 User Agents by Access Count

Rank User Agent Accesses % Bytes %

1 xmim_client 9 64.29 0 0.002 API Client 5 35.71 0 0


549

Report 2: Web Server Statistics (Week of 05/09/99 to 05/15/99)

Overall Statistics

The time period this report refers to is not yet over.

Category TotalUnique sites served 1Unique documents served 1Unique authenticated users served 2Unique trails followed 1Total visits 2

Accesses Per Hour

Figures are averages for that hour on a typical day. The following shows the accesses per hour.

Hour Accesses % Bytes % Bits/Sec Bytes/SecHour Accesses % Bytes % Bits/Sec Bytes/Sec

550


Hour Accesses % Bytes % Bits/Sec Bytes/Sec00:00 0.00 0.00 0.00 0.00 0.00 0.0001:00 0.00 0.00 0.00 0.00 0.00 0.0002:00 0.00 0.00 0.00 0.00 0.00 0.0003:00 0.00 0.00 0.00 0.00 0.00 0.0004:00 0.00 0.00 0.00 0.00 0.00 0.0005:00 0.00 0.00 0.00 0.00 0.00 0.0006:00 0.00 0.00 0.00 0.00 0.00 0.0007:00 0.00 0.00 0.00 0.00 0.00 0.0008:00 0.00 0.00 0.00 0.00 0.00 0.0009:00 0.00 0.00 0.00 0.00 0.00 0.0010:00 0.00 0.00 0.00 0.00 0.00 0.0011:00 0.00 0.00 0.00 0.00 0.00 0.0012:00 0.00 0.00 0.00 0.00 0.00 0.0013:00 0.00 0.00 0.00 0.00 0.00 0.0014:00 0.00 0.00 0.00 0.00 0.00 0.0015:00 0.00 0.00 0.00 0.00 0.00 0.0016:00 0.00 0.00 0.00 0.00 0.00 0.0017:00 0.00 0.00 0.00 0.00 0.00 0.0018:00 0.00 0.00 0.00 0.00 0.00 0.0019:00 0.00 0.00 0.00 0.00 0.00 0.0020:00 0.00 0.00 0.00 0.00 0.00 0.0021:00 0.29 100.00 0.00 0.00 0.00 0.0022:00 0.00 0.00 0.00 0.00 0.00 0.0023:00 0.00 0.00 0.00 0.00 0.00 0.00

Accesses Per Day

The following shows the accesses per day.


551

Date Accesses % Bytes % Bits/Sec Bytes/Sec05/09/99 0 0.00 0 0.00 0.00 0.0005/10/99 2 100.00 0 0.00 0.00 0.0005/11/99 0 0.00 0 0.00 0.00 0.0005/12/99 0 0.00 0 0.00 0.00 0.0005/13/99 0 0.00 0 0.00 0.00 0.0005/14/99 0 0.00 0 0.00 0.00 0.0005/15/99 0 0.00 0 0.00 0.00 0.00

Contents of / Sorted by Access Count

Rank File or Subdirectory Accesses % Bytes %

552


1 [/daily] 2 100.00 0 0.00

Top 1 of 1 Documents Sorted by Access Count

Rank URL Accesses % Bytes %

1 /daily/CCA/Close/10-02-1986/12-31-1998 2 100.00 0 0.00

Totals

Item Accesses BytesOverall Accesses 2 0Home Page Accesses 0 0

Top 2 of 2 Authenticated Users by Access Count

Rank User Accesses % Bytes %

http://customers.lim.com/doc/clf/report2/10355di.html#directories_/daily

http://www.futop.com/daily/CCA/Close/10-02-1986/12-31-1998


553

1 allen 1 50.00 0 0.002 stephen 1 50.00 0 0.00

Top 1 of 1 Sites by Access Count

Rank Site Accesses % Bytes %

1 207.170.66.75 2 100.00 0 0.00

Top 0 of 0 Document Trails

Rank Trail Visits Avg. Minutes

Visits Report

Total visits: 2Average visit: 0.00 minutesLongest visit: 0 minutesAverage accesses per visit: 1.00

1 Example Visit

User Minutes Trailacrobat.0000000002 1 /daily/CCA/Close/10-02-1986/12-31-1998

Top 0 of 0 Search Keywords by Access Count

Rank Keyword(s) Accesses % Bytes %

554


Top 2 of 2 User Agents by Access CountRank User Agent Accesses % Bytes %

1 API Client 1 50.00 0 0.002 xmim_client 1 50.00 0 0.00

logext.perl#!/usr/local/bin/perl#Breaks log file into separate log files by username#The newly named log file is logfile.username#usage: logext.pl use IO::File;$logFile = $ARGV[0];unless (open(LOG, $logFile)) { die ("Can't open log file!\n");}$num = 0;while ($line = <LOG>) { $tmp = $line; $tmp =~ s/[0-9\.]+\s+-\s+(\w+).*/$1/; # extract the username $user = $tmp;

555

chomp($user);# print "$user\n"; if (! $files{$user} ) { $sublogFile = $logFile; $sublogFile =~ s/(.*)\.log/$1.$user.log/; print "$sublogFile\n"; $files{$user} = new IO::File "> $sublogFile"; unless (defined $files{$user}) { die ("Can't open sublog file!\n"); } } print {$files{$user}} $line;}close @files;

xmimusage.perl#!/usr/local/bin/perl# Computes the total # of application uses for each user for the specified# usage xmimusage use Date::Manip; # obtain the latest version of DateManip from http://www.cise.ufl.edu/~sbeck/$logFile = $ARGV[0];unless (open(LOG, $logFile)) { die ("Can't open log file!\n");}while ($line = <LOG>){ # parse out the username, time info, application #207.170.66.75 - joe [23/Feb/1999:16:33:38 -0600] "GET /XmimConnect" 200 0 "-" "API Client" "acrobat.1" 0 $line =~ s/([^\s]+\s+){2}(\w+)\s+\[([^:]+)([^\s]+\s+){7}\"([^\"]+).*/$2:$3:$5/; my ($user, $date, $client ) = split(':', $line); chomp $client; $date=&ParseDate($date); $count{$date}{$user}{$client} ++;}foreach $date (keys %count) { foreach $user (keys %{$count{$date}}) { foreach $client (keys %{$count{$date}{$user}}) { $fdate = &UnixDate($date, "%m/%d/%y"); printf "$fdate $user $client $count{$date}{$user}{$client}\n"; } }}

556


COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 23: Using Multiple Databases

557

CHAPTER23

Using Multiple Databases

IntroductionThe Commodity DataServer provides support for using multiple databases. These databases may be either CommodityDataServer databases or external databases. As supporting multiple databases has widespread impacts, most all ofthe documentation incorporates aspects of support for multiple databases. This document attempts to pull together allthe information on using multiple databases.

For descriptions of the API and BMIM interfaces see Chapter 5, “C/C++ API” and Chapter 4, “BMIM ScriptingLanguage”.

Initial Specification of DatabasesThe databases specified in the server xmimrc (.xmimrc) file establish the pool of databases that will be opened bya Commodity DataServer. The databases specified in the .xmimrc file may be new or existing databases. Thesedatabases will remain open for the duration of the server's existence. Any client connecting to a CommodityDataServer can have an associated client xmimrc (.cxmimrc) file. If a client has a .cxmimrc file, then the initial poolof databases to be used for that client will be those specified in the .cxmimrc file. These do not have to be a subsetof the server databases but must be existing databases: new databases can not be created by specifying them inthe .cxmimrc file. If a client does not have a .cxmimrc file then the databases specified in the server .xmimrc filewill be those available initially to that client. Chapter 40, “Commodity DataServer .xmimrc File” and??? both providedescriptions of these facilities.

For the BMIM client, databases can be specified by using multiple -d flags (one for each database) in the BMIM clientcommand line. The effect here will be to further restrict the pool of databases specified in the client's .cxmimrc fileso that the client's initial pool consists only of those specified using the -d flags. The databases specified must be asubset of those specified in the client's .cxmimrc (or the server .xmimrc if a .cxmimrc does not exist for this client).Refer to “Invoking the BMIM Client” for more information.

Ordering of DatabasesThe order of specification for databases (i.e., the order in which the initial database pool is specified and then in whichopen/new commands are used) is important as the databases will be opened in the order specified and schema

558


information will be accessed in that order as well. Thus, for example, if the same relation exists in more than oneopen database, the first one specified will be the one used unless disambiguating database name/path information isprovided. The databases to be used most should always be specified first.

Semantics of Multiple DatabasesMultiple databases are viewed as though they were a single database and so the applied semantics is as if dealingwith a single database. If a database is thought of as a "tree" of nodes, then a multi-database is a tree that containsall paths that exist in any one of the single databases. Each path appears only once in the multi-database. Note thatthe nodes in the multi-database tree do not correspond to any one node in the single databases. Rather, a node in themulti-database tree is an amalgamation of the corresponding nodes in the single databases, e.g., it may have bothdaily and tick data, even though database #1 only has daily data and database #2 only has tick data.

For purposes of understanding the implications of preserving the illusion that there is a single database even whenmultiple databases are employed, API routines and BMIM commands can be categorized as follows:ADD. Add new information (e.g., relation_add or XmimPutRecords).

MOD. Modify existing information (e.g., relation_modify or XmimModRelColumn).

GET. Read data (e.g., facts_write or XmimGetRecords).

VIEW. Browse the hierarchy (e.g., print_schema or XmimPrintSchema).

DEL. Delete information (e.g., relation_delete or XmimDeleteFacts).

The corresponding semantics in the context of multiple databases is:ADD/MOD: In general, in the presence of multiple databases, any API routines or BMIM commands resulting in the addition and/

or modification of schema information will result in the information being added to all applicable database schemas. Forexample, if IBM is added to Equities:i:ib, then IBM is added to all database schemas that have a category called Equities:i:ib.Data information, however, is added to the first applicable database. For example, if daily data is added to Close of IBM, it isadded to the first database that has a relation called IBM and a column called Close.

GET: Any API routines or BMIM commands having to do with schema or data retrieval will retrieve information from the firstapplicable database. For example, the description for IBM comes from the first database schema that has the relation IBM.

VIEW: When schema information is viewed, hierarchy information is merged as consistent with viewing multiple databases asthough they were a single database. For example, the children of Equities will consist of all the children that appear aschildren of Equities in any single open database. When overlap exists, the schema information shown will be that from thefirst applicable schema.

DEL: Delete API routines or BMIM commands will result in schema or data information being deleted from all applicable databases.

Managing the Current Database Pool/ViewOpen, create and close routines/commands are provided so that a client can add or remove databases from/to its poolof databases. There are also routines/commands provided such that a client can temporarily narrow or restrict the


559

database view to include only a specified subset of the currently open databases and then expand back to the entirecurrent database pool.

Creating, Opening and Closing DatabasesThe following C/C++ API routines provide for creating new databases, opening existing databases and closingdatabases.XmimReturnCode XmimNewDatabase (XmimClientHandle handle, XmimString database);XmimReturnCode XmimVaNewDatabase (XMIM_CLIENT_HANDLE, handle, XMIM_DATABASE, database, XMIM_END_ARGS);XmimReturnCode XmimOpenDatabase (XmimClientHandle handle, XmimString database);XmimReturnCode XmimVaOpenDatabase (XMIM_CLIENT_HANDLE, handle, XMIM_DATABASE, database, XMIM_END_ARGS);XmimReturnCode XmimCloseDatabase (XmimClientHandle handle, XmimString database);XmimReturnCode XmimVaCloseDatabase (XMIM_CLIENT_HANDLE, handle, XMIM_DATABASE, database, XMIM_END_ARGS);

The XmimNewDatabase routine is used to create a new database, open it and add it to the client's current databasepool. If the specified database has already been created then an error will be returned. The database argumentgives the name of the file containing the Commodity DataServer schema or provides the specification for an externaldatabase. In the case an external database, the XmimNewDatabase routine most likely will behave the same as theXmimOpenDatabase routine with the database argument being directly passed to the shared library routine to openthe schema for that external database. This is due to the fact that for many external databases (e.g., Oracle), thedatabase administrator must set up and configure the external database instance and there is no practical way for theCommodity DataServer to do this. So, the database needs to already be there with a setup username and password.The setup username and password can then be what is passed as the argument for the XmimNewDatabase orXmimOpenDatabase routine. Of course, this is all dependent on the implementation of the shared libraries for theparticular external database. In the case of a Commodity DataServer database, all database files are accessed fromthe schema file.

The XmimOpenDatabase routine is used to open a database and add it to the client's current database pool. Thedatabase to be opened may be any existing database, that is, it does not have to exist in the server .xmimrc file. If thespecified database has already been opened by another client, then a pointer to the opened database will be returned.If this is the first client to open the database, it will be physically opened.

The XmimCloseDatabase routine is used to close a database that is currently open. Using this command willremove the specified database from the pool of accessible databases. In the case that this client is the only onecurrently referencing the database, the database will be physically closed unless it was a database specified in theserver .xmimrc file. If a client terminates without closing a database but that client is the only one referencing thedatabase, the system will automatically close the database (unless it was a database specified in the server .xmimrcfile).

560


The BMIM commands to create open and close databases may be used anywhere within a BMIM script. Thesynopses are as follows:database_new dbasedatabase_open dbasedatabase_close dbase

The argument dbase gives the name of the file containing the Commodity DataServer schema or provides thespecification for an external database. In the case of an Commodity DataServer database, all database files areaccessed from the schema file. The database_new command is used to create a new database. If there is alreadya file with this name, BMIM will return an error. Databases created with database_new are automatically openedfor write access. The database_open command assumes the database already exists, and if it does, it opens itfor read access (write access is granted when a lock command is specified). The database to be opened may beany existing database; that is, it does not have to exist in the server .xmimrc file. As noted in the description of theXmimNewDatabase API routine, in the context of external databases the database_new command will mostlikely have the same semantics as the database_open command. The database_close command is used toclose a database that is currently opened. Using this command will removed the specified database from the pool ofdatabases accessible to this invocation of BMIM or BMIM client.

Changing the Database ViewThe API routines to change the database view are specified below.XmimReturnCode XmimNarrowDatabases (XmimClientHandle handle, int numDatabases, XmimString *databases);XmimReturnCode XmimVaNarrowDatabases (XMIM_CLIENT_HANDLE, handle, XMIM_DATABASE_LIST, database1, database2, ..., NULL, XMIM_END_ARGS);XmimReturnCode XmimWidenDatabases (XmimClientHandle handle);XmimReturnCode XmimVaWidenDatabases (XMIM_CLIENT_HANDLE, handle, XMIM_END_ARGS);

The XmimNarrowDatabases routine is used to restrict the databases available to those specified in the databaseslist. The databases specified will compose the current database view and must be a subset of the current opendatabase pool. Databases that are not specified will not be closed, they are just restricted from the view untila subsequent call to XmimNarrowDatabases includes them or the XmimWidenDatabases routine is used.Subsequent calls to XmimNarrowDatabases will undo the effect of the previous XmimNarrowDatabases call (i.e.,narrow is not nested). The XmimWidenDatabases resets the view to the current pool of all open databases.

The BMIM commands to change the database view are as follows:database_narrow { database = dbase_name_1; database = dbase_name_2;...}database_widen


561

The database_narrow command specifies the list of databases that will be available as the current view ofdatabases. The databases specified must be a subset of the current open database pool. Databases that are notspecified will not be closed, they are just restricted from view until a subsequent database_narrow commandincludes them or a database_widen command is given. Subsequent database_narrow commands will undothe effect of the previous database_narrow (i.e., the narrow command is not nested). The database_widencommand resets the view to the current pool of all open databases.

Listing the Current Database ViewThere is an additional API routine provided for showing the current database view:

XmimReturnCode XmimGetDatabases (XmimClientHandle handle, int *numDatabases, XmimString **databases));XmimReturnCode XmimVaGetDatabases (XMIM_CLIENT_HANDLE, handle, XMIM_DATABASE_ARRAY, &numDatabases, &databases, XMIM_END_ARGS);

The XmimGetDatabases routine is used to return a listing of all databases in the current database view. That is,if a call to XmimNarrowDatabases has been made and it specifies a single database as the current view, thenXmimGetDatabases will return only that single database (until the XmimWidenDatabases routine is used to set theview back to the entire open database pool).

Partitioning Data Types Across DatabasesTo facilitate supporting multi-databases whereby daily data and tick intra-day and/or tick-by-tick data may reside indifferent databases, an option has been added to the BMIM commands used to specify filenames for storing daily,intraday, tick and millisecond data:

database_def_daily_file relname NONEdatabase_def_intraday_file relname NONEdatabase_def_tick_file relname NONEdatabase_def_millisecond_file relname NONE

The special filename NONE can be used in the database_def_daily_file, database_def_intraday_file,database_def_tick_file and database_def_millisecond_file commands to indicate that daily, intraday,tick or millisecond data should not be allowed for the given relation. In the context of multiple databases this is usefulif it is desired to place the daily data in a separate database from the tick data for given relations. The use of NONE willensure that data be loaded in the appropriate database.

To see where this becomes useful, suppose, for example, that there are two databases open: daily.mim and tick.mim.As the names suggest, daily.mim has daily data and tick.mim has tick data (both intraday and tick-by-tick). Consider

562


a sequence of BMIM commands whereby a new relation (e.g., IBM) is added followed by reading daily data and thenreading tick data for that relation. Since IBM does not exist, it is added to both the daily.mim and tick.mim databases.When the system tries to read daily data, it looks for the first database that already has daily data, so as to append tothat database. Neither database does, so it will have to pick the first one, namely daily.mim. Then, when the processis repeated for the tick data, the same algorithm will pick daily.mim as the database to add the data to. This wouldnot be correct given the intentions for daily.mim to only have daily data and tick data to reside in tick.mim. The specialNONE filename can be used as such to prevent this from happening:database_narrow { database = daily.mim;}database_def_intraday_file TopRelation NONEdatabase_def_tick_file TopRelation NONEdatabase_narrow { database = tick.mim;}database_def_daily_file TopRelation NONEdatabase_widenrelation_add { name = IBM; parent = Equities;}facts_read { file = daily_data; relation = IBM;}facts_read { file = intraday_data; relation = IBM; tick_intraday;}facts_read { file = tick_data; relation = IBM; tick_by_tick;}

This BMIM sequence first narrows the focus to the daily database and then specifies that no intraday tick or tick-by-tick data is allowed in that database. The focus is then narrowed to the tick database and a command given tospecify that no daily data is allowed in that database. After that, the database_widen command is used to set thedatabase pool back to the original, and the commands to add the IBM relation (will be added to both databases) andthen load daily, intraday and tick data are given with the results being that the appropriate data goes to the appropriatedatabase.

Note that this same behavior could have been accomplished with the following BMIM sequence:relation_add { name = IBM; parent = Equities;}database_narrow { database = daily.mim;}facts_read {


563

file = daily_data; relation = IBM;}database_narrow { database = tick.mim;}facts_read { file = intraday_data; relation = IBM; tick_intraday;}facts_read { file = tick_data; relation = IBM; tick_by_tick;}database_widen

The database_narrow command should be used to set the focus such that data goes to the appropriate databaseand the database_def_daily_file, etc. commands need not be used. However, it is anticipated that thedatabase def daily file, etc. commands will be used when a database is first created and the convenience of notneeding to be very cautious about narrowing the focus to specify the appropriate database will be found useful.

For more information on BMIM commands for updating a database, see “Updating a Database”.

564



565

CHAPTER24

MIM/SQL External DatabaseImplementation

IntroductionAn external database implementation allows the Commodity DataServer to connect to a remote data source toaccess data that does not reside in a typical Commodity DataServer database. Examples of remote data sources arerelational databases, a set of files on disk, or a web service. This implementation does not copy data between theexternal source and Commodity DataServer. It goes to the external data source each time it is requested. It is up tothe implementor to create a caching mechanism in their C++ implementation to improve performance by reducingcalls to the remote data source.

To implement a remote database, a set of abstract methods must be overridden. There are approximately 50 methodsthat need to be implemented in order to implement access to remote data source. A remote database can be read-only or it can be read-write depending on which methods are implemented. The external database methods areimplemented in C++ and are compiled into a shared library. This shared library is then loaded by the CommodityDataServer on startup.

Key Commodity DataServer ConceptsThe data storage objects in a Commodity DataServer are time series. Every numeric value has an associate date-timekey. The time series are identified with relation and column names. The relation name is usually the symbol, IBM, andthe column is the field, Close. For example the time series of the official close of IBM would be “Close of IBM”.

Time series are physically stored at three resolutions: daily, intra-day and tick. Daily time stamps have no hour orminutes. Intra-day covers the range to 1-minute increments. Tick covers all time stamps to 1 second of resolution.Below 1 second, the data is stored as a FIFO.

In order to efficiently store a large quantity of time series, the Commodity DataServer employs a hierarchy andinheritance of meta-data. For example, all equities class data is stored in a category called Equities. This allowscolumns to be added to the parent category that is immediately applied to the children, in this case the 12,000individual equity symbols.

A Commodity DataServer can have multiple databases of either standard Commodity DataServer types or external.All requests are processed by the database order in the .xmimrc file or as specified by the clients database narrow

566


commands. Therefore, multiple external database connectors could access a large external SQL database. This cansimplify the coding of the implementation.

ImplementationThe external database is connected to the Commodity DataServer via a Solaris dynamic link library. The CommodityDataServer slave server processes load this library and the routines are mapped to the external database entry pointsvia the .spc file configuration. The dynamic library will be built with C linkage.

Accessing Time SeriesThe getData routine moves all time-series data from the external database into the Commodity DataServer.

extern "C" int getData (void* handle, int relation_id, int column_id, XmimUnit unit, int* num_records, XmimDateTime** dates, XmimByte** values)

Inputs: The first handle is the generic handle of the external database. This handle is used to store data acrosscalls for the external database, for example the OCI connection handle. The next arguments are relation and columnidentifiers. These identifiers are assigned by the external database and will be discussed in the following sections onthe schema. The unit argument is a constant to indicate the time granularity needed by the Commodity DataServer:daily, intraday or tick.

Outputs: With the exception of the status code and number of records, all of the return arguments are passed via heapallocated (malloc) buffers. The number of records is the length if the XmimDateTime vector. XmimDateTime are thetimestamps. This is a C struct with slots for year, month, day, hour, minute and second and millisecond. Finally, thedata is returned via an XmimByte pointer. The actual data is in native SPARC types: int, float and double. The datais ordered by column and then date. Values must also be properly aligned. Typically, in the code the return vector isallocated as a float * array and cast into a byte pointer.

A sample SQL table could be structured as follows for daily data:

create table tsdata (relation_id number(9) not null, column_id number(9) not null, trans_date DATE, val number(8,6))


567

This would be accessed in the C code by the query:“select trans_date, val from tsdata where relation_id=%d and column_id=%d order by trans_date”, relation_id, column_id

Accessing Meta Data InformationIn order for the Commodity DataServer to generate requests on the time series, the external database must generatetime-series meta data. This section discusses a minimal approach to generating meta data. Please note that therelation and column hierarchies are completely similar in the internal structure. The getData routine consumesrelation and column identifiers. These identifiers can be mapped using the following table.create table tsrelmeta (relation_id, number(9), parent_id number(9), name varchar(255), char rel_type, description varchar(1024))

The key query maps the string names to relation ids as the Commodity DataServer will always operate on relation id.This table should be properly indexed by name.

Hierarchy, Types and CategoriesCommodity DataServer data is stored using hierarchies for relations and columns. The root of the relation tree is“TopRelation”. The parent_id in the tsmeta table is used to create the parent child relationships. The parentrelations are called categories in the Commodity DataServer. The leaf relations are called “normal” relations in theCommodity DataServer. Only normal relations have data (with exceptions for futures and futures contract data).

In the external database implementation, the server implements the recursive part of the algorithm. The externaldatabase is only required to provide: the children or parents and the parents of children for 1 level of the tree at a time.

A large part of the database is built of normal and category relations. To access the futures support features of theCommodity DataServer, addition types need to be modeled. The complete set of relation types are:

0x01 = normal

0x02 = category

0x04 = futures

0x08 = contract (futures)

568


0x10 = continuous (futures)

0x20 = options

Column ModelThe column hierarchy is modeled like the relation hierarchy.

create table tscolmeta (column_id, number(9), parent_id number(9), name varchar(255), char col_type, description varchar(1024))

If the data is stable enough, then the column hierarchy can be omitted for constant hierarchy in code. The col_typeis used to indicate base or sparse data. Sparse data is forward filled on all requests regardless of the fill options of therequest. This is useful for interest rate data.

Relation, Column and Data PresenceRelation and Column pairs, or relcols, are the time-series identifiers. The Commodity DataServer tests each databasefor presence of a relation column pair with data. Responding to inquiries about the existence of relation column pairsefficiently is important to preserve server performance.

extern "C" int containsDataOfUnitsP (void* handle, int rel_id, int col_id, XmimUnit data_units)

A typically implementation could use the following query to answer the request:

Select count(*) from tsdata or a select trans_date having count(trans_date) > 1. Some SQL systems havespecial syntax for quickly testing for the presence of rows avoiding the costly count(*) operator.

Part VI: Utilities

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 25: xmim_get Utility

571

CHAPTER25

xmim_get UtilityThe xmim_get utility is located in the xmim/bin directory on the Commodity DataServer. The xmim_get utility usesXmimVaGetRecords to extract data from a Commodity DataServer database.

Usage:xmim_get [-h] [-n num_units units] [-q host] [-p server_number] -r num_rels relation [-c n column1 column2] [-d mm/dd/yy] [-e mm/dd/yy] [-t hh:mm:ss] [-u hh:mm:ss]

where: (Optional entries are designated with brackets [ ].)[-h] Help/Usage

[-n num_units units] The type of units to extract.

num_units = any number

units = -3 = milliseconds 1 = seconds 2 = minutes 3 = hours 4 = days

[-q host] Commodity DataServer Host Name

[-p server_number] Commodity DataServer Port

-r num_rels relation The number of relations and the relation names to extract.

[-c n column1 column2] The number of columns and the column names to extract.

[-d mm/dd/yy] The date to start the extraction.

[-e mm/dd/yy] The date to end the extraction.

[-t hh:mm:ss] The time to start the extraction.

[-u hh:mm:ss] The time to end the extraction.

Examples:

This example will extract the close of IBM from the beginning of the database to the end.xmim_get -q acrobat –p 4 –n 1 4 –c 1 Close –r 1 GII.IBM.NYSE

This example will extract the close of IBM for the month of December 1999.xmim_get -q acrobat –p 4 –n 1 4 –d 12/01/1999 –e 12/31/1999 –c 1 Close –r 1 GII.IBM.NYSE

This example will extract the close of IBM and KR for the month of December 1999.xmim_get -q acrobat –p 4 –n 1 4 –d 12/01/1999 –e 12/31/1999 –c 1 Close –r 2 GII.IBM.NYSE KR

572

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 25: xmim_get Utility

Download the compiled windows32.exe program. In addition, download the xmim4.dll. (Place the file xmim4.dll in thesame directory as the program executable xmim_get.exe.)

View the source code.

http://customers.lim.com/download/xmim_get/xmim_get.exe

http://customers.lim.com/download/xmim_get/xmim4.dll

http://customers.lim.com/download/xmim_get/xmim_get.c

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 26: xmim_get_options Utility

573

CHAPTER26

xmim_get_options UtilityThe xmim_get_options utility is located in the xmim/bin directory on the Commodity DataServer. Thexmim_get_options utility uses XmimVaGetOptionsRecords to extract options data from a Commodity DataServerdatabase.

Usage:xmim_get_options [-h] -q host -r numOfRels rel1 rel2 [-c numOfCols col1 col2] -p port [-n num_exe_units units] [-d from_date] [-e to_date] [-g from_exp_date to_exp_date] [-m from_strike_price to_strike_price] [-t from_time] [-u to_time] [-v] [-s strike_weight]

where (Optional entries are designated with brackets [ ].)[-h] Help/Usage

-q host Commodity DataServer Host Name

-r numOfRels rel1 rel2 ... Number of Relations and the Relation Names

[-c numOfCols col1 col2 ...] Number of Columns and the Column Name

If not present all columns will be returned.

-p port Commodity DataServer Port Number

[-n num_exe_units units] Number of Units and Units, (e.g., 1 4 = everyday)

num_exe_units (e.g., 10 4 = every 10th day)

1 = seconds, 2 = minutes, 3 = hours, ...

[-d from_date] From Date (e.g., 01/01/2001)

[-e to_date] To Date (e.g., 01/01/2001)

[-g from_exp_date to_exp_date] The From and To Expiration Date

[-m from_strike_price to_strike_price] The From and To Strike Price to Consider

[-t from_time] From Time (e.g., 12:00)

[-u to_time] To Time (e.g.,15:00)

[-o option_type] c = Call, p = Put

[-v] Use Relative API Call

[-s strike_weight] The Strike Weight to Consider

Examples:

This will extract all columns of IBM_Options from the beginning of the database to the end:

574

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 26: xmim_get_options Utility

xmim_get_options -q hostname -p 4 -r 1 IBM_Options

This will extract column OptionGamma or IBM_Options where the strike price is between 50 and 100:

xmim_get_options -q hostname -p 4 -r 1 IBM_Options -c 1 OptionGamma -m 50 100

Download the compiled windows32.exe program. In addition, download the xmim4.dll. (Place the xmim4.dll file in thesame directory as the program executable xmim_get_options.exe.)

View the source code.

http://customers.lim.com/download/xmim_get_options/xmim_get_options.zip

http://customers.lim.com/download/xmim_get/xmim4.dll

http://customers.lim.com/download/xmim_get_options/xmim_get_options.c

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 27: xmim_trading_info Utility

575

CHAPTER27

xmim_trading_info Utility

The xmim_trading_info utility is located in the xmim/bin directory on the Commodity DataServer. Thexmim_trading_info utility returns the trading pattern for a relation. Output includes the start/stop time and the days ofthe week that a relation trades.

Usage:

xmim_trading_info [-h] [-q host] [-p port] -s symbol

where: (Optional entries are designated with brackets [ ].)

[-h] Help/Usage


[-p port] Port Number

-s symbol Relation Name

Example 1:

xmim_trading_info -q nyse -p 10 -s IBM relation: IBMstartTrade: 09:30:00amendTrade: 04:00:00pmtrading pattern: Monday Tuesday Wednesday Thursday Fridaynumber of segments: 1segment 0segment start date: 00/00/0000trading pattern: Monday Tuesday Wednesday Thursday Fridaynumber of periods: 1timePeriod: 09:30:00am to 04:00:00pm

Example 2:

xmim_trading_info -q nyse -p 10 -s TopRelation:Equities:i:ibrelation: TopRelation:Equities:i:ibstartTrade: 08:00:00amendTrade: 05:00:00pmtrading pattern: Monday Tuesday Wednesday Thursday Fridaynumber of segments: 1segment 0segment start date: 00/00/0000trading pattern: Monday Tuesday Wednesday Thursday Fridaynumber of periods: 1timePeriod: 08:00:00am to 05:00:00pm

576

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 27: xmim_trading_info Utility

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 28: xmim_svr_info Utility

577

CHAPTER28

xmim_svr_info Utility

The utility program xmim_svr_info is provided with Commodity Query (version 4.0 and higher). This program can beused to obtain information pertaining to one or more master servers that are currently running. It can also be used tokill a master server, to re-read the current database and libraries or to load a new database and/or libraries. If invokedso as to obtain server information, xmim_svr_info displays the following for each applicable master server: the servernumber, host name it is running on, number of databases, database names, owner's name and process id. This is thesame information provided when the XmimGetServerInfo API routine is used. Additionally, for each master server,xmim_svr_info displays: version information, the database locking status including identification of the slave serverthat holds the lock, license information, logging information consisting of the logging level and log file(s) in use, thenumber of slave servers registered, and then, for each slave server, the host name, process id, owner, status and timeof last dispatch.

Usage:

xmim_svr_info [-h] [-s host] [-p port] [-k] [-r] [-n]


[-h] Help/Usage

[-s host] Specifies the host that the master server is running on. The default will be the local host.

[-p port] Specifies the port of the desired master server. If this argument as well as the -k and -r options are not specified,information will be returned for every master server that is running on the given host. The default server number if the -kor -r options are used will be 0.

[-k] Specifies to kill a given master server. Note that the user must be either the owner of the server process or the super-user to kill a server. The default will be to kill master server 0 on the local host.

[-r] Specifies to re-read the .xmimrc file which effectively causes the databases and any libraries/macros to be re-loaded.This will be effected only for one particular master server, the default being server 0 on the local host.

[-n] Specifies to open a new log file with the standard log filename and begin logging to this new file. Simply renaming a logfile will not result in logging to a new file because the file descriptor will still point to that file. Thus, to achieve archivingof log files, the old log file must be renamed and then the -n flag used to open a new log file, taking out a new filedescriptor for it.

578

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 28: xmim_svr_info Utility

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 29: xmim_find Utility

579

CHAPTER29

xmim_find UtilityThe xmim_find utility is located in the xmim/bin directory on the Commodity DataServer. The xmim_find programrecursively traverses the schema producing reports suitable for post processing with other tools like grep, cut, andsort. The data can also be imported into programs which load pipe delimited files like Excel or Access.

Usage:xmim_find relation_path_name

where: (Optional entries are designated with brackets [ ].)[-q server] Server Name

[-p port] Port Number

[-d delimiter] Changes the default pipe delimiter (e.g., -d ":")

[-n database] Narrows the find to a given database (e.g.,-n /home/lim/data.cust/xmim.mim)

[-o ndpftcr] Show the output fields. The default output flags are np. The following output flags can be set:

n name

d description

f full path name

t type

c columns

r data ranges for columns (daily, intraday, tick)

Examples:

The following prints the full paths to everything under TopRelation:Futures:Energy.xmim_find -o f -n /home/lim/data.cust/xmim.mim TopRelation:Futures:EnergyTopRelation:Futures:Energy:CBT.BA TopRelation:Futures:Energy:CBT.BA:CBT.BA_1998X TopRelation:Futures:Energy:CBT.BA:CBT.BA_1998Z TopRelation:Futures:Energy:CBT.BA:CBT.BA_1999U TopRelation:Futures:Energy:CBT.BZ TopRelation:Futures:Energy:CBT.BZ:CBT.BZ_1998X TopRelation:Futures:Energy:CBT.BZ:CBT.BZ_1998Z TopRelation:Futures:Energy:IPE.FO TopRelation:Futures:Energy:IPE.FO:IPE.FO_1999V TopRelation:Futures:Energy:IPE.FO:IPE.FO_1999X ...

The following prints the name, description and path for TopRelation:Futures:Nymex.xmim_find -o ndp -n /home/lim/data.cust/xmim.mim TopRelation:Futures:Nymex:NG

580

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 29: xmim_find Utility

NG_1990M|Natural Gas Futures (USD/MMBTU) (Jun 1990)|TopRelation:Futures:Nymex:NG NG_1990N|Natural Gas Futures (USD/MMBTU) (Jul 1990)|TopRelation:Futures:Nymex:NG NG_1990Q|Natural Gas Futures (USD/MMBTU) (Aug 1990)|TopRelation:Futures:Nymex:NG NG_1990U|Natural Gas Futures (USD/MMBTU) (Sep 1990)|TopRelation:Futures:Nymex:NG NG_1990V|Natural Gas Futures (USD/MMBTU) (Oct 1990)|TopRelation:Futures:Nymex:NG NG_1990X|Natural Gas Futures (USD/MMBTU) (Nov 1990)|TopRelation:Futures:Nymex:NG NG_1990Z|Natural Gas Futures (USD/MMBTU) (Dec 1990)|TopRelation:Futures:Nymex:NG NG_1991F|Natural Gas Futures (USD/MMBTU) (Jan 1991)|TopRelation:Futures:Nymex:NG

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 30: xmim_get_rel_paths Utility

581

CHAPTER30

xmim_get_rel_paths Utility

The xmim_get_rel_paths utility is located in the xmim/bin directory on the Commodity DataServer. Thexmim_get_rel_paths utility returns all the path and database locations for a given relation name.

Usage:

xmim_get_rel_paths [-h] [-q host] [-p svr_num] relation


[-h] Help/Usage


[-p svr_num] Commodity DataServer Port

relation The relation or category name you want to find the path for.

Example 1:

The following entry will return the path and database location for the relation IFE_FP.C:

xmim_get_rel_paths -p 1 IPE_FP.C

Result Returned:

Number of results: 1TopRelation:Futures:Ipe:IPE_FP.C, /usr/lim/data/xmim.mim

More than 1 result would indicate a duplicate relation.

Example 2:

The following entry will return a duplicate path for the category CL_Options:

xmim_get_rel_paths -q oimim -p 11 CL_Options

Results Returned:

Number of results: 2

582

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 30: xmim_get_rel_paths Utility

TopRelation:Options:FTInteractive:FuturesOptions:Energy:Nymex:CL_Options, /home/limopt/data.opt/xmim.mimTopRelation:Options:OptionMetrics:EquitiesOptions:c:cl:CL_Options, /home/limopt/data.ometrics21/xmim.mim

The example above shows that for the category name CL_Options, there are two locations that have this categoryname. The results show the path and the database locations.

There is no problem with having multiple category locations. You can tell this is searching for a categoryname as categories names can be mixed case whereas relation names are upper case.

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 31: xmim_get_col_paths Utility

583

CHAPTER31

xmim_get_col_paths Utility

The xmim_get_col_paths utility is located in the xmim/bin directory on the Commodity DataServer. Thexmim_get_col_paths utility returns all the path and database locations for a given column name.

Usage:

xmim_get_col_paths [-h] [-q host] [-p svr_num] column


[-h] Help/Usage


[-p svr_num] Commodity DataServer Port

column The column name you want to find the path for.

Example:

The following example will return the path and database locations for the column named Volume.

xmim_get_col_paths -q auslm13 -p 0 Volume

The following shows the results:

Number of results: 23TopColumn:VolumeCat:Volume, /home/lim32/data/xmim.mimTopColumn:VolumeCat:Volume, /home/lim32/data.gii/xmim.mimTopColumn:VolumeCat:Volume, /home/lim32/data.cust/xmim.mimTopColumn:VolumeCat:Volume, /home/lim32/data.iso2/xmim.mimTopColumn:VolumeCat:Volume, /home/lim32/data.pjm/xmim.mimTopColumn:VolumeCat:Volume, /home/lim32/data.ric/xmim.mimTopColumn:VolumeCat:Volume, /home/lim32/data.gen/xmim.mimTopColumn:VolumeCat:Volume, /home/lim32/data.ometrics31/xmim.mimTopColumn:VolumeCat:Volume, /home/lim32/data.bluelinx/xmim.mimTopColumn:VolumeCat:Volume, /home/lim32/data.bentek/xmim.mimTopColumn:VolumeCat:Volume, /home/lim32/data.ometricsrelative/xmim.mimTopColumn:VolumeCat:Volume, /home/lim32/data.taqtradem01/xmim.mimTopColumn:VolumeCat:Volume, /home/lim32/data.taqtradem02/xmim.mimTopColumn:VolumeCat:Volume, /home/lim32/data.taqtradem03/xmim.mimTopColumn:VolumeCat:Volume, /home/lim32/data.taqquotem01/xmim.mimTopColumn:VolumeCat:Volume, /home/lim32/data.taqquotem02/xmim.mimTopColumn:VolumeCat:Volume, /home/lim32/data.taqquotem03/xmim.mimTopColumn:VolumeCat:Volume, /home/lim32/data.idl/xmim.mim

584

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 31: xmim_get_col_paths Utility

TopColumn:VolumeCat:Volume, /home/lim32/data.opt/xmim.mimTopColumn:VolumeCat:Volume, /home/lim32/data.opistn/xmim.mimTopColumn:VolumeCat:Volume, /home/lim32/data.opistg/xmim.mimTopColumn:VolumeCat:Volume, /home/lim32/data.thomsonqfs/xmim.mimTopColumn:VolumeCat:Volume, /home/lim32/data.pvm/xmim.mim

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 32: xmim_put_milli Utility

585

CHAPTER32

xmim_put_milli Utility

The xmim_put_milli utility is located in the xmim/bin directory on the Commodity DataServer. The xmim_putapplication demonstrates the putRecords capability with milliseconds data.

Queries with millisecond data are supported in Commodity DataServer versions 4.6 and higher.

Usage:

xmim_put_milli [-h] [-q host] [-p server_number] -r num_rels relation [-c n column1 column2]


[-h] Help/Usage

[-n num_units units] The type of units to extract.

num_units = any number

units = -3 = milliseconds 1 = seconds 2 = minutes 3 = hours 4 = days


[-p server_number] Commodity DataServer Port

-r num_rels relation The number of relations and the relation names to extract.

[-c n column1 column2] The number of columns and the column names to extract.

586

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 32: xmim_put_milli Utility

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 33: MIMDBCP Utility (Copying Database Contents)

587

CHAPTER33

MIMDBCP Utility (Copying DatabaseContents)This document shows how to use the MIMDBCP utility to copy all or portions of a database into another database.

Usage:mimdbcp [-h] [-V] [--def-file=filename] [--rel-file=filename] [-l filename] [-g] [-C] [-s host[:port]] [-d host[:port]] [-r relation] [-c column] [-a mmy/dd/yyyy] [-D] [-T] [-n] [-b] [-v] [--flip-double]

where: (Optional entries are designated with brackets [ ].)[-h] Help/Usage

[-V] Show the release version.

[--def-file=filename] Use database_def's from source server or from specified file.

[--rel-file=filename] Work on relations listed in FILE (overrides -r).

[-l filename] Write the output log to the specified file name (default is stdout).

[-g] Bypass Scheme interpreter (overrides -f).

[-C] Copy all from source database to destination database (default is to add only for missing items).

[-s host[:port]] MIM source server is at machine HOST on port PORT (default is localhost:0).

[-d host[:port]] MIM destination server is at machine HOST on port PORT (default is localhost:1)

[-r relation] Traverse the MIM relation hierarchy starting at RELATION (default is TopRelation)

[-c column] Traverse the MIM column hierarchy starting at COLUMN (default is TopColumn)

[-a mm/dd/yyyy] Start copying data from this date mm/dd/yyyy (default is copy all data).

[-D] Process daily data.

[-T] Process tick data.

[-n] Log actions, but don't do them.

[-v] Verbose log, includes all relations and columns visited not just ones with differences.

[--flip-double] Use when doubles are copied incorrectly across different platforms.

[-b] --barNaN. Do not copy NaN values. Available with DataServer 4.7.6 or newer.

Example:

The following example shows how to use MIMDBCP using the options settings:mimdbcp -s nyse:10 -d neo:15 -r TopRelation -c TopColumn > Log2 2>&1mimdbcp -s intra4:0 -d neo:15 -r TopRelation:Equities:z:za: -c TopColumn

588


How to Setup MIMDBCPCheck that the ~/.limrc file contains the following two lines:

GUILE_LOAD_PATH=/home/lim/mimdbcp/guile-1.6.4export GUILE_LOAD_PATH

If the ~/.limrc file does not contain these two lines that set the environment variable then add these two lines to thefile.

Methods for Running the MIMDBCP UtilityThere are two methods for running the MIMDBCP utility:

● From a command line with option settings.● From a command line using a configuration file.

How it WorksFirst, the MIMDBCP utility creates a connection to the servers specified in xmim-source and xmim-dest. Thexmim-dest connection immediately executes a lock_files, to ensure there are write permissions on thedestination database.

After connecting to the two databases, MIMDBCP will recursively traverse all the columns in the source database(starting at the point defined by relation-root and column-root).

MIMDBCP will check to see if the column exists in the destination database. If it does not, the path to the column istranslated (e.g., convert TopColumn:Price:Median to TopColumn:NewColumns:Price:Median). This (translated) columnis added to the destination database.

It is possible for a column not to exist in the destination database, even though the translated columndoes. This is not considered an error.

Next, MIMDBCP will follow the same process for relations. Starting at relation-root, it checks to see if it exists inthe destination database. If the relations exist, it is assumed to be the right relation. That is to say, no attempt is madeto check (or fix) the meta-data information.


589

If the relation does not exist, the name is translated (if there is a relation-translate function defined in the configurationfile), and then the (translated) relation is added to the new database.

Next, the system checks each of the relation columns existing in the source database for this specific relation. If itencounters a relation column that is not in the destination database, it adds it to the destination database. Whenadding a relation column, MIMDBCP carries over all of the meta-data associated with the relation column (e.g., is itsparse or not, is it integer or float or double, etc). However, if the relation column already existed, MIMDBCP does NOTcheck the meta-data information, so any changes to object-type (or other) are lost.

Finally, if the -D (daily) flag is specified, MIMDBCP will check whether the source database has daily data for thisparticular relation column, and if it does it checks whether the destination database has corresponding data. If there isdaily data in the source database but not in the destination database, the data is copied.

Note in particular that the system does not copy the data if there is already some data in the destinationdatabase, so that any changes to the data in the source database are lost. However, if the -C (copy) flagis also specified, the source data will be copied regardless of whether the destination database has dataor not. Note that the -C (copy) operation merges the data in the source and destination databases (i.e.,it does not begin by deleting the old data). Similarly, if the -T (tick) flag is specified, MIMDBCP will copythe tick data.

The user of MIMDBCP controls which options will occur when running it, not all options need to be used, and theconfiguration file is optional.

The configuration file allows the user to specify values for xmim-source, xmim-dest, relation-root, andcolumn-root.

Note that command line arguments override the values set in the configuration file.

590


COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 34: MIMDBCK Utility (Database Integrity Check)

591

CHAPTER34

MIMDBCK Utility (Database IntegrityCheck)This document shows how to use the MIMDBCK utility to verify the integrity of the Commodity DataServer databases.MiMDBCK reports on all "corrupt" relation-columns, i.e. relation-columns which are not accessible. users may specifyoptions to delete the inaccessible relation-columns.

Unless explicitly disabled (with the -n option), mimdbck performs an analysis of the in memorydatabase schema to ensure it can be traversed successfully. The schema analysis utilitiesxmim_schema_analyze and xmim_schema_reorg must be installed on the server with mimdbckfor the schema analysis to take place. If issues are found with the database schema, the database isdeemed not reparable and the check is aborted.

Usage:mimdbck [-h] [-V] [-p serverPort] [-q hostName] [-l logFileName] [-d databaseName] [-r relationName] [-c columnName] [-v] [-m mode] [-n]

where: (all parameters are optional - designated with brackets [ ].)[-h] Help/Usage

[-V] Show the release version.

[-p serverPort] DataServer port (default is 0).

[-q hostName] DataServer host name.

[-d databaseName] Restrict check to databaseName only (default is all active databases).

[-r relationName] Traverse the relation hierarchy starting at relationName (default is TopRelation).

[-c columnName] Restrict check to columnName (default is all columns).

[-v] Verbose output. Include all relation-columns traversed, not just the ones with errors.

[-l logFileName] output to logFileName (default is console).

[-m mode] default mode is CHECK. May be specified as LIST (only lists relation-columns available in the database) or asCOUNT (only displays the number of relation columns in the database)

[-n] No schema analysis. Do not perform schema analysis prior to database check.

Examples:

Following are MIMDBCK usage examples:

● Check all active databases on the local server port 3296, output errors to file named check.log

592


$ mimdbck -p 3296 -l check.log

● Check database data.cust on port 3296

$ mimdbck -p 3296 -d /home/lim/data.cust/xmim.mim

● Check relation TopRelation:CattleFutures in database data.cust on port 3296

$ mimdbck -p 3296 -d /home/lim/data.cust/xmim.mim -r TopRelation:CattleFutures

How to Setup MIMDBCKThere are no setup steps required. the mimdbck utility is distributed with the Commodity DataServer versions 4.7.2and higher. The utility can be used with earlier versions of the DataServer as well. Contact your systems administratorto learn if your DataServer version is supported and to have the utility installed on your server.

The percentage complete indicator is only available for DataServer versions 4.7.5 and higher

How it WorksMimdbck is configured to run at very low CPU priority to prevent any negative impact on the performance of a runningDataServer. It checks all relation-columns of the DataServer databases, or optionally only the relation columns of aspecified database, or only specified relation-columns.

Delete OptionWhen the delete (-a) option is specified, corrupt relation-columns are automatically deleted.

After the check is complete, mimdbck obtains a lock on the database and attempts to delete all relation-columns ithas identified as corrupt during its check.

mimdbck emails the log file to the email addresses defined in the environment variable NOTICES. The NOTICESenvironment variable is normally set in the DataServer .limrc configuration file.


593

Schema CheckStarting with DataServer versions 4.7.5 and 4.8.6, mimdbck by default performs a quick schema check before startingthe actual check of the database.

If schema errors are found, the check is aborted with a message like:

"The database schema has issues. The DB is irreparably damaged and should not be

checked!"

You may want to confirm a mimdbck bad schema diagnosis by running the schema utilities available in yourDataServer.

Use the -n (noschema) option to force mimdbck to run without the schema check.

Percent Complete IndicatorDepending on the number of relation-columns to check, mimdbck could take several hours to complete. Starting withDataServer versions 4.7.5 and 4.8.6 mimdbck displays a percent complete indicator.

594


COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 35: Delta Publisher Utility

595

CHAPTER35

Delta Publisher Utility

OverviewDatabases in the Commodity DataServer constantly receive data files throughout the day. The Delta Publisherapplication tracks the changes in these files and publishes a report which is distributed to multiple subscribers.

Subscribers are the parties that are interested in receiving all, or some of the changes that have been applied tothe databases hosted by the Commodity DataServer. The publisher runs on the same server/host as the CommodityDataServer.

The publisher is only needed by DataServers that receive updates. Any DataServers that are strictlyused for reading have no need for the publisher. By default the utility is not installed with the CommodityDataserver, but it is available on request or as needed.

For more information on the utility, please see the Delta Publisher User's Guide.

http://www.morningstarcommodity.com/sites/default/files/DeltaPublisher.pdf

596

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 35: Delta Publisher Utility

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 36: Index Constituent Utility

597

CHAPTER36

Index Constituent UtilityMoringstar Commodity Data upgraded the implementation of Index Constituent data on Commodity DataServerto include the Index Constituent data as a Commodity DataServer symbol (relation) rather than component textfiles. These component files were originally available at: /home/lim/dates/user/COMPONENTS. Removal date forcomponent files was effective 8/19/2005. Users had to change the way the component files were used. Any queriesthat referenced the component files would no longer run, so users had to use the new relation names. The initialupgrade included the Dow 30, NASDAQ, Russell 1000, Russell 2000, S&P 400, S&P 500, S&P 600, S&P 1500 and S&PToronto Composite. For questions, please contact Client Support at: [email protected].

The Index Constituent utility shows how alias names are mapped to constituent names for each of the IndexConstituent categories.

All the constituents are located in the schema in this path:

TopRelation:Equities:UnitedStates:Constituents

TopRelation:Equities:Canada:Constituents

The Index Constituent utility is located in the $LIMHOME/xmim/bin directory for Commodity DataServerreleases 4.5.25 and higher.

Usage:xmim_get_constituents [-h] [-q host] [-p server_number] [-d database] [-r relation]

where (Optional entries are designated with brackets [ ].)[-h] Help/Usage

[-q host] Enter the host name.

[-p server_number] Enter the server number.

[-d database] Specify the database location.

[-r relation] Enter the category name to only list that portion of relation names and aliases.

Example 1:

The following example will produce the original file which was distributed to /home/lim/dates/user/COMPONENTS/GII.DJIA from the warehouse.$ xmim_get_constiuents –q localhost –p 0 –r Index_DJIA >/home/lim/dates/user/COMPONENTS/GII.DJIA



598


This should be used for transition into the new syntax. In the future, the dates folder directories may not exist.

Example 2:

The following example will print the contents of all of the constituents folders on a server.

$ xmim_get_constituents –q localhost –p 0 Constituents (category) DowJones (category) Index_DJIA (category) DJIA.AA -> GII.AA.NYSE DJIA.AIG -> GII.AIG.NYSE DJIA.AXP -> GII.AXP.NYSE DJIA.BA -> GII.BA.NYSE DJIA.C -> GII.C.NYSE DJIA.CAT -> GII.CAT.NYSE DJIA.DD -> GII.DD.NYSE DJIA.DIS -> GII.DIS.NYSE DJIA.GE -> GII.GE.NYSE DJIA.GM -> GII.GM.NYSE DJIA.HD -> GII.HD.NYSE DJIA.HON -> GII.HON.NYSE DJIA.HPQ -> GII.HPQ.NYSE DJIA.IBM -> GII.IBM.NYSE DJIA.INTC -> GII.INTC.NASDAQ DJIA.JNJ -> GII.JNJ.NYSE DJIA.JPM -> GII.JPM.NYSE DJIA.KO -> GII.KO.NYSE DJIA.MCD -> GII.MCD.NYSE DJIA.MMM -> GII.MMM.NYSE DJIA.MO -> GII.MO.NYSE DJIA.MRK -> GII.MRK.NYSE DJIA.MSFT -> GII.MSFT.NASDAQ DJIA.PFE -> GII.PFE.NYSE DJIA.PG -> GII.PG.NYSE DJIA.SBC -> GII.SBC.NYSE DJIA.UTX -> GII.UTX.NYSE DJIA.VZ -> GII.VZ.NYSE DJIA.WMT -> GII.WMT.NYSE DJIA.XOM -> GII.XOM.NYSE Nasdaq (category) Index_NASD100 (category) NASD100.AAPL -> GII.AAPL.NASDAQ NASD100.ADBE -> GII.ADBE.NASDAQ NASD100.ADSK -> GII.ADSK.NASDAQ NASD100.ALTR -> GII.ALTR.NASDAQ NASD100.AMAT -> GII.AMAT.NASDAQ NASD100.AMGN -> GII.AMGN.NASDAQ NASD100.AMZN -> GII.AMZN.NASDAQ NASD100.APCC -> GII.APCC.NASDAQ NASD100.APOL -> GII.APOL.NASDAQ NASD100.ATYT -> GII.ATYT.NASDAQ NASD100.BBBY -> GII.BBBY.NASDAQ NASD100.BEAS -> GII.BEAS.NASDAQ NASD100.BIIB -> GII.BIIB.NASDAQ NASD100.BMET -> GII.BMET.NASDAQ


599

NASD100.BRCM -> GII.BRCM.NASDAQ NASD100.CDWC -> GII.CDWC.NASDAQ

600


COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 37: dbcompare Utility

601

CHAPTER37

dbcompare Utility

The dbcompare utility is used to compare two Commodity DataServer databases. Comparisons are for schema anddata points.

Usage:

dbcompare [-h] [-f FILE] [-l FILE] [-o HOST:[PORT]] [-n HOST:[PORT]] [-r RELATION] [-c COLUMN] [-s] [-d] [-t] [-p] [-v] [-b mm/dd/yyyy] [-e mm/dd/yyyy]


[-h] Help/Usage. Print this message and exit.

[-f FILE] Use configuration FILE.

[-l FILE] Write output log to FILE.

[-o HOST:[PORT]] Commodity DataServer is at machine HOST on PORT.

[-n HOST:[PORT]] Commodity DataServer is at machine HOST on PORT.

[-r RELATION] Traverses the Commodity DataServer relation hierarchy starting at RELATION.

[-c COLUMN] Column

[-s] Compare schema information.

[-d] Compare daily data.

[-t] Compare tick data.

[-p] Precision of data points.

[-v] Verbose logging in log file.

[-b mm/dd/yyyy] Beginning date.

[-e mm/dd/yyyy] Ending date.

602

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 37: dbcompare Utility

Example Configuration File:

[root@ausla44 comp]# more dbcompare.cfg SERVER_3X localhost:1 SERVER_40 localhost:11 RELATION_ROOT TopRelation:Futures COLUMN_LIST none SCHEMA yes DAILY yes TICK yes VERBOSE no PRECISION 6 LOG_FILE dbcompare.log

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 38: xmim_pair_gen Utility

603

CHAPTER38

xmim_pair_gen Utility

OverviewThe xmim_pair_gen utility is located in the xmim/bin directory on the Commodity DataServer.

Many problems require comparing a set of items in combinations of two. This document will show how to the use thexmim_pair_gen utility to generate symbol pairs.

For example, for the items:

{A,B,C,D}

The combinations two at a time are:

{A, B},{A, C},{A, D},{B, C},{B, D},{C, D}

For a given set of N items, the number of combinations of two will be N*(N-1)*/2. Therefore, there are a large numberof combinations for a small number of items.

The xmim_pair_gen program can be used to generate the combinations efficiently for a symbol set. The LET facilitycan then be used to iterate and test the combinations.

The pairing algorithm generates a large number of iterations. This means that large numbers of symbolswill require very long run times. For more information, see “Managing Size” and “Table of Iterations” inthis document.

How to Use the UtilityThe LET facility will iterate in a pair-wise pattern when two variables are used in a single LET.

604


For example:

LET leftVar = “/home/lim/left.txt” rightVar = “/home/lim/right.txt”

SHOW 1: Close of leftVar 2: Close of rightVarWHEN Date is leftVar last_data_dayAND Date is rightVar last_data_day

The file left.txt will contain:

AAABBC

The file right.txt will contain:

BCDCDD

The files left.txt and right.txt can be created using xmim_pair_gen.

First, a file of symbols is required. For this example, name the file symbols.txt.

ABCD

The utility sorts the symbols. The file should not contain duplicates.

To run the utility, enter:

xmim_pair_gen symbols.txt left.txt right.txt

The files left.txt and right.txt will be created or overwritten with the proper lists for the combinations. Place these filesin a location where the query engine can access them.


605

Managing SizeThe pairing algorithm generates a large number of iterations. The problem is exponential is nature. This means thatlarge numbers of symbols will require very long run times.

The following table shows how many iterations will run for the various symbol numbers.Symbols Iterations10 4520 19030 43540 78050 1225100 4950200 19900300 44850400 79800500 1247501000 4995002000 19990003000 44985004000 79980005000 12497500

Table of IterationsFor example, the SP500 with 500 symbols will require 124,750 iterations to process all of the symbols. The server mayrequire many hours to process the query. Also, the server will be required to generate a very large result set.

The files left.txt and right.txt are processed in order from top to bottom one line at a time. This means that you cansplit the files into many files and make several query runs on the server. This can be accomplished with Unix textprocessing utilities. In order to perform 2500 iterations at a time you will do the following:split –l 2500 left.txt left_split –l 2500 right.txt right_

This will result in many files named left_aa, left_ab, left_ac, … right_aa, right_ab, right_ac ….

You will need to create many Commodity DataServer query files to reference each pair of lists.{left_aa,right_aa},{left_ab,right_ab}, {left_ac, right_ac} …

Let’s assume that you have a query with “left.txt” and “right.txt” in a file called myquery.mim. You will generate thefiles: aa.mim, ab.mim, ac.mim, ... with the proper replacements for left.txt and right.txt.

606


In ksh or bash syntax with the command on one line, enter the following:

for F in left_[a-z][a-z]do N=${F#left_} cat myquery.mim | sed "s/left.txt/left_${N}/" | sed "s/right.txt/right_${N}/" >myquery.mim.${N}done

This produces files named myquery.mim.aa, myquery.mim.ab, …. Now all you need to do is execute these queries.Generate a BMIM script to run them all.

for F in myquery.mim.*do echo “query_execute { query=$F; }” >>myquery.bmimdone

The last step is to run the queries. As the run will take a long time, background and nohup the bmim_client commandso that you can log off.

nohup bmim_client myquery.bmim >myquery.out 2>&1 </dev/null &

One to Many ComparisonA one to many comparison is completed directly in the query language. No utility is needed. In the one to many casetwo LET statements should be used to get nested iterations from the query engine.

LET leftVar = ALET rightVar = B, C, D

SHOW 1: Close of leftVar 2: Close of rightVarWHEN

The resulting combinations are:

{A,B},{A,C},{A,D}

Part VII: Appendix

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 39: Using the Package Maker

609

CHAPTER39

Using the Package MakerThe Package Maker is a tool for creating update packages of your data. This allows you to transfer your data files as a“upd” package file.

You will need to specify the directory that contains your make_data and data files (.txt or .csv) and a name for thepackage. The utility will bundle the files into a package with the assigned name. The Package Maker will place theresulting package in the current working directory unless an absolute path/filename is specified.

RequirementsThis utility requires Java version 1.4 or later. See Chapter 14, “Software Installations for Loading Data” for instructionson loading the Package Maker.

How to Use the Customer Data Package MakerTo use the Customer Data Package Maker, enter the following from a command line prompt:run.packager source_dir upd_name

where

● source_dir – is the directory containing the make_data and data files (in .txt or .csv format)● upd_name – is the name assigned to the resulting package

Example Usage

Creating the upd Package FileExample:

610

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 39: Using the Package Maker

A user logs into his home directory at /home/jdoe. The user wants to create a package containing all of the followingfiles in the /home/jdoe/spool/TAG directory:

/home/jdoe/spool/TAG/make_data/home/jdoe/spool/TAG/data1.csv/home/jdoe/spool/TAG/data2.csv

To create the package, upd_0_TAG_20040203, run:

run.packager /home/jdoe/spool/TAG upd_0_TAG_20040203

Unpacking the upd Package FileThe following command will unpack the package file created in the example above.

ksh upd_0_TAG_20040203

The sh command unpacks the upd_0_TAG_20040203 file and creates the TAG directory containing the make_dataand .csv files (make_data, data1.csv, data2.csv) listed in the example above.

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEChapter 40: Commodity DataServer .xmimrc File

611

CHAPTER40

Commodity DataServer .xmimrc File

Locating the .xmimrc FileWhen the Commodity DataServer starts up, it begins by processing the server xmimrc (.xmimrc) file. The DataServerfirst checks the user's xmimrc environment variable. If this is set, then the Commodity DataServer uses this value asthe location of the server xmimrc file. Otherwise, the Commodity DataServer checks for the file /home/lim/.xmimrc,and uses this file if it exists.

SpecificationsThe .xmimrc file specifies the location of the files and directories needed by the Commodity DataServer, and containsuser customizable information, such as the formatting of Commodity DataServer output.

612


Example .xmimrc FileThe following is a sample .xmimrc file:

echo "license: /home/lim/data/xmim.ids" echo "database: /home/lim/data/xmimdb1.mim" echo "database: /home/lim/data/xmimdb2.mim" echo "database: /home/lim/data/xmimdb3.mim" echo "datesDir: /home/lim/dates/user" echo "holidaysDir: /home/lim/dates/holidays" echo "libraryDir: /home/lim/library" echo "tempDir: /tmp" echo "customSearch: /home/lim/lib/custom.spc" echo "holiday_fill: fill_nan" echo "miss_data_fill: fill_forward" echo "format: TopRelation.TopColumn F8.2" echo "format: TopRelation.VolumeCat I8" echo "format: Equities.Price F8.3" echo "format: Currencies.Price F8.4" echo "option: print_seconds" echo "advUpdatesLogging: no" echo "publishExactPrn: no" echo "entitlements: no" echo "webservice: no" echo "read_lock: no" echo "continuousLoggingP: no"

License File LocationThe license line establishes the location of the Commodity DataServer license file.

Databases UsedThe database lines are used to specify the location of the Commodity DataServer databases to be used. Any numberof databases can be specified including both Commodity DataServer and external databases. For the CommodityDataServer databases, only the name of the schema file needs to be specified; the Commodity DataServer can accessall the other files from the schema. For external databases, three components must be specified, each separated bya colon character. xmim_oi must be used to identify the database as an external (open interface) database. Next, aconnecting string to the third-party database must be provided. For the Oracle external database included in the aboveexample .xmimrc file, the username, password and database application are used and these are passed to the ODBCdriver which grants access to the Oracle database. The third component is the location of the spec file describing theexternal database functions to be loaded with the server.

If more than one database entry exists, the system will open all the databases in the order in which they are listed.It is also the case that in checking for symbols, the associated schemas will be checked in the order in which they


613

are listed. As such, the ordering of databases will be used to resolve conflicts. For example, if two databases have asymbol “C” (Corn vs. Chrysler), then the first database that is loaded wins in the sense that using a symbol “C” in aquery will result in resolution such that the “C” from the first database loaded is used. To specify “C” from the otherdatabase, a full path must be provided. Thus, databases that will be used most often should always be listed first. Thisapplies to all databases, Commodity DataServer and external. For more detailed information on setting up multipledatabases, see the "Chapter 23, “Using Multiple Databases” chapter.

Directory LocationsThe datesDir, holidaysDir, and libraryDir specify the location of user-defined dates, holiday dates, andmacro definitions, respectively. The tempDir specifies where the Commodity DataServer should store temporary files;please note that some of these files can be quite large, so tempDir should point to a writable directory with plenty offree space. The default is the current working directory (if writable) and, otherwise, /tmp.

Custom Search Spec File LocationThe customSearch line is used to specify the location of the spec file describing the custom searches that are to beloaded with the server. For more information on the custom search facility see the "“Custom Search Facility” section.

NaN HandlingThe holiday_fill and miss_data_fill lines are used to individually specify how invalid data due to holidays andmissing data will be reported by the system. The alternatives are to use NaNs (fill_nan), to use the previous value(fill_forward), to use the first following value (fill_backward), or interpolate using either linear, geometric orlogarithmic interpolation (interpolate_linear, interpolate_geometric or interpolate _logarithmic).This will effect both graphing and execution of queries.

Printing FormatsThe format line is used to specify the printing style used for the data in the database. The printing formats arespecified in the form F w.d, where w specifies the number of characters used to print the value, and d specifies thenumber of digits to print after the decimal point. The format specifier I w is a short-hand for F w.0. The printing stylecan be assigned to a specific column of a symbol (relation), using the syntax relation.column. Either relationor column can be categories, so it is possible to specify the formatting style of a group of symbols and columns withone format statement. Commodity Query will always choose the most specific format line for a given symbol and

614


column, so it is possible to provide generic defaults and then override them for specific symbols. In particular, theexample above set the format for all fields at F8.2, except volumes (e.g., Volume, OpenInterest), which areprinted using I8. Moreover, equity prices are printed using F8.3, and currency prices using F8.4. TopColumn canbe used to refer to all top-most column categories and, similarly, TopRelation can be used to refer to all top-mostsymbol categories.

Global OptionsThe option line is used to set certain global options in the system. Currently, the only global option that can be setin this manner is the print_seconds option. If print_seconds is specified, time formats are amended such thatseconds are printed.

Activate Delta PublisherTo turn on the Delta Publisher you must set the advUpdatesLogging flag to yes.

Be careful! Turning on this switch will cause the server to log each and every change made to the server.If the publisher is not consuming these files regularly, they will continue to grow until they eventually fillthe remaining space on the drive.

For complete instructions on setting up and using the publisher, see the Delta Publisher Guide.

Print Directly from prn File in facts_readThe publishExactPrn: yes line is used to turn on the option to print directly from the prn file in facts_read. Thisoption does not effect the put_records operation.

The advUpdatesLogging flag must be set to yes for this option to work.

Be careful! Turning on the advUpdatesLogging switch will cause the server to log each and everychange made to the server. If the publisher is not consuming these files regularly, they will continue togrow until they eventually fill the remaining space on the drive.

MIM_Delta_Publsiher_Guide


615

Entitlements UsageThe Commodity DataServer comes with entitlements turned off. You must specifically enable entitlements bymodifying the .xmimrc file. Once you enable entitlements, then the default is that all access is restricted unless youhave a free pass or unless you are specifically granted access.

Entitlements will not be used, however, unless an entitlements database exists. The entitlements line can be usedto turn entitlements off regardless of whether an entitlements database exists or not. A no specification indicates thatentitlements are to be turned off ; a specification of yes turns entitlements on.

To setup the entitlements database you must first set the entitlements flag to yes in the .xmimrc file. Next, youmust load users, groups and permissions with “Entitlements” and/or WebServices. Doing so adds entitlements andenables entitlements checking.

The webservice flag instructs the DataServer version 4.8 to run a background process to import entitlements user/group associations from WebServices. In DataServer version 4.7 it allows user names to be specified in email addressformat.

Refer to the "“Data Security (aka Entitlements)” section for a complete description of using entitlements.

Read LockingBy default, the Commodity DataServer uses read locks when accessing schema information. The read_lockspecification can be set to no to indicate that the system should not use read locks. Eliminating the use of read locksreduces access time at the risk of those accesses potentially being “unsafe” (e.g., could return incorrect results) ifthe relevant portion of the schema is also being updated at the same exact instant. The gain in speed, obtained byspecifying that read locks should not be used, is most significant when the schema is loaded across the network.In fact, it is recommended that either the schema reside on the same file server as the Commodity DataServer and,hence, not be loaded across the network, or read locks be turned off . Failure to adhere to this policy could result inextremely long waiting periods for request processing if local read/write accesses are being made at the same time. Aspecification of yes will result in the default behavior of read locks being used for all schema accesses.

Publish Continuous Daily Contract ChangesThe continuousLoggingP: yes line turns on the option to publish continuous daily contract changes from theDelta Publisher.

For complete instructions on setting up and using the publisher, see the Delta Publisher Guide.

The advUpdatesLogging flag must be set to yes for this option to work.



MIM_Delta_Publsiher_Guide

616


Be careful! Turning on the adVUpdatesLogging switch will cause the server to log each and everychange made to the server. If the publisher is not consuming these files regularly, they will continue togrow until they eventually fill the remaining space on the drive.

Executing the .xmimrc File as a ScriptIf an .xmimrc file starts with the characters #| (a hash followed by a pipe symbol), the .xmimrc is interpreted beforeits contents are processed by the Commodity DataServer. The path to the interpreter to use is specified immediatelyfollowing the leading #| characters. For example, if you wish to use /bin/sh as your interpreter, the first line inthe .xmimrc file would be the following:

#|/bin/sh

When you use this feature, the .xmimrc file is essentially a script in some language. The Commodity DataServerexecutes this script and processes whatever output the script has in place of the .xmimrc.

For example, here is a simple .xmimrc script:

#|/bin/sh

. /home/lim/.setup-env.sh echo "license: $XMIM_LICENSE_FILE" echo "database: $XMIM_DATABASE_FILE" echo "datesDir: $XMIM_DATES" echo "holidaysDir: $XMIM_HOLIDAYS" echo "libraryDir: $XMIM_LIBRARY"

In this script, a system-wide initialization file is read in /home/lim/.setup-env.sh. This file initializes many environmentvariables to appropriate values, so the rest of the script simply prints the correct .xmimrc lines based on theseenvironment variables.

COMMODITY DATASERVER DATA AND DEVELOPMENT GUIDEIndex

617

Index

Symbols<type>ToString

XMIMUtils (Utilities)VBA API, 428

AAccessing Relations, 434Add

XMIMColumnVBA API, 355

XMIMRelationVBA API, 349

XMIMRelColVBA API, 363

AddAliasXMIMRelation

VBA API, 349AddCorrections

XMIMRecords and XMIMRecordsDoubleVBA API, 340

Adding ColumnsBMIM, 33Commodity DataServer C/C++ API, 108Commodity DataServer Java API, 255

Adding Formulas, 51Commodity DataServer C/C++ API, 122

Adding Relation Aliases, 50Adding Relation Columns

BMIM, 35Commodity DataServer C/C++ API, 108Commodity DataServer Java API, 255

Adding RelationsBMIM, 28Commodity DataServer C# .NET API, 272Commodity DataServer C/C++ API, 108Commodity DataServer Java API, 255

AddRelation, 272AddTuple

XMIMTableVBA API, 403

AdjustedStudyXMIMUtils

VBA API, 421advUpdatesLogging

xmimrc File Entry, 614, 614, 615AfterRepeat

XMIMShowWhen and XMIMShowWhenDoubleVBA API, 370

AggrXMIMRelCol

VBA API, 359aggregation, 38aggregation_rule, 37Alias


all_contracts, 58AllContractsP

XMIMSchemaVBA API, 387

append_mode, 54Arg

XMIMCustomSearchVBA API, 414

ArgDescXMIMCustomSearch

VBA API, 414ArgType


ASCII Data FilesReading/Writing, 40

AttrXMIMShowWhen and XMIMShowWhenDouble

VBA API, 367AttrHeading

XMIMExecution and XMIMExecutionDoubleVBA API, 382

Attribute of a Relation ColumnCommodity DataServer C/C++ API, 161

Attributes of ColumnsCommodity DataServer C# .NET API, 281

Attributes of RelationsCommodity DataServer C# .NET API, 281Commodity DataServer Java API, 260

618


BBeforeRepeat


BMIM, 25Adding Correction Audits to Table Facility, 52Adding Formulas, 51Adding Relation Aliases, 50Adding/Modifying/Deleting Columns, 33Adding/Modifying/Deleting Relation Columns, 35Adding/Modifying/Deleting Relations, 28Assigning Holiday Schedules, 49Commands

chdir, 62column_add, 33column_delete, 35column_modify, 35corrections_add, 52database_def_daily_file, 57database_def_millisecond_file, 54database_def_tick_file, 54database_def_write, 57database_narrow, 51, 57database_widen, 56delete_facts, 48dump_expiration_dates, 64entitlements_clear, 65entitlements_read, 65entitlements_write, 65entitlements_write_all, 65facts_read, 40, 45, 46, 48facts_write, 46formula_add, 51, 52formula_modify, 52print_*_file, 57print_relation, 58print_schema, 39, 58, 58query_execute, 53query_graph, 53regen_continuous, 53relation_add, 28relation_alias_add, 50, 51relation_column_add, 35, 36, 36, 39, 40relation_column_delete, 40, 49, 49relation_column_modify, 39, 40relation_delete, 33

relation_modify, 28, 32set_compress, 63set_compress_threshold, 64set_holiday_schedule, 49, 55system, 62trading_pattern, 28unset_compress, 63writer_wait, 61, 61, 62

Data Compression, 63Data Files/Partitioning, 54Executing Saved Queries, 53Features and Conventions, 26Generating Continuous Contracts, 53Handling Multiple Databases, 26Invoking BMIM Client, 25Locking/Unlocking Database, 27Miscellaneous Commands, 62Printing Database Schema Information, 58Printing Expiration Dates, 64Printing the Database Partition Table, 57Printing the Location of a Relation and Column, 57Sale Mode for Database, 61Setting Global Options, 62Unix Interface, 62Updating a Database, 27

Browse Relation HierarchyBMIM, 58

print_schema, 59Commodity DataServer C# .NET API, 274Commodity DataServer C/C++ API, 75Commodity DataServer Java API, 240, 262

ByNamePXMIMSchema

VBA API, 386

CC Stored Function, 443

cfunc_instance_compute_trading_date_range, 446,454cfunc_instance_compute_trading_pattern, 445, 454cfunc_instance_destroy, 446, 454cfunc_instance_get_characteristic_time_series, 446,447, 454cfunc_instance_get_value, 446, 454cfunc_instance_init, 445, 454cfunc_instance_init_defaults, 445, 445, 454


619

Description, 443Example, 454Framework, 447

Built-in Stored Functions, 453Regular Stored Functions, 447

Lazy Evaluation Approach, 462Lazy Predictor Code, 467Life Cycle, 454Predictor Code, 463Structure, 444Utility, 447

CacheSizeXMIMTable

VBA API, 397CaseSensitiveP


chdir, 62Child



ClearXMIMColumn

VBA API, 352XMIMCurrentTick

VBA API, 405XMIMCustomSearch

VBA API, 412XMIMRecords and XMIMRecordsDouble

VBA API, 317XMIMRelation

VBA API, 341XMIMRelCol

VBA API, 357XMIMSchema

VBA API, 385XMIMShowWhen and XMIMShowWhenDouble

VBA API, 366XMIMTable

VBA API, 394ClientType

XMIMServerVBA API, 310

CloseDatabase


CloseTableXMIMTable

VBA API, 402Col

XMIMCurrentTickVBA API, 406




ColHeadingXMIMExecution and XMIMExecutionDouble

VBA API, 382colNames, 243, 259ColRoot


ColTypeXMIMColumn

VBA API, 353XMIMUtils

VBA API, 418Column


column_add, 34column_delete, 35Columns

Adding/Modifying/Deleting with BMIM, 33Columns of a Relation

Commodity DataServer C/C++ API, 79Commodity DataServer, 3Commodity DataServer .NET, 271

(see also Commodity DataServer C# .NET APIExamples)

Commodity DataServer .NET API, 271(see also Commodity DataServer Java API)

Commodity DataServer C# .NET APIBrowse Relation Hierarchy, 274Example

AddRelation, 272deleteRelation, 274

620


findRelation, 274ForwardCurve, 275GetCorrections, 277GetRecords, 278GetRecordsOption, 279GetRelation, 281GetRelationColumn, 282GetRelChildren, 274GetRelColumns, 283GetRollOverDates, 283Putrecords, 284QueryExecute, 284

Sample Programs, 272Commodity DataServer C/C++ API, 67

Adding Columns, 108Adding Formulas, 122Adding Relation Columns, 108Adding Relations, 108Columns of a Relation, 79Data Retrieval, 75

Browse Relation Hierarchy, 75Data Updating, 107

Storing a Vector or Values, 111Executing Queries, 98

XmimQueryExecute, 103XmimSelectRecords, 100

High Frequency Updating, 114Invoking a Server, 67Loading Data From a File, 113Locking the Database, 107Memory Management, 123Metadata Terminology, 74Optional Arguments, 72Retrieving Data Records, 81

Accessing Data for Multiple Relations, 88Accessing Data for Relations Specified in a File, 89Accessing Data with Relations and ColumnsSpecified in Arrays, 90XmimGetRecordIndex, 87XmimGetRecords, 81XmimGetRecordsOption, 92

Server Connect/Disconnect, 70Writing Data to a File

XmimWriteFacts, 93XML Results, 104

Commodity DataServer Java API, 235

Adding Relations/Columns/Relation Columns, 255API Library Files, 235Basics, 237Browse Relation Hierarchy, 240compareTo, 241Connecting/Disconnecting Server, 238ConnectionFactory, 238Data Retrieval, 240Data Updating, 254

Locking the Database, 254Error Messages, 269Examining the Columns of a Relation, 242Executing Queries, 250fieldFormats, 258getDataRange, 240getDataRange Routine, 242getRecords Routine, 243GetRelation Routine, 260getRelChildren, 240GetRelChildren Routine, 262, 264getRelColumns Routine, 242Getting Started, 235High Frequency Updating, 258Interface Changes, 239Loading Data from a File, 257mergeMode Argument, 258Metadata Terminology, 237Millisecond Data, 256MimConnection, 238MimConnection.disconnect, 239, 239MIMConnection.lockDb, 254MimConnection.lockRealTimeStore, 259MimData.factsRead, 258MimData.getRecords, 260MimData.getRecords Function, 256MimData.putRecords, 258MimData.putRecords Function, 256MimData.queryExecute Method, 250, 250MimData.readFacts Function, 257MimDate, 240MimDate.INVALID_DATE, 240MimDateTime.INVALID_DATE_TIME, 240MimException, 239MimNetworkException, 239MimRealTime.updateRealTime Method, 258MimRealTime.updateRealTimeBar Routine, 259


621

MimSchema.addColumn Routine, 255MimSchema.addRelation Routine, 255MimTime, 240MimTime.INVALID_TIME, 240MimUnits.DAYS, 256MimUnits.MILLISECONDS, 256MimUnits.MINUTES, 256MimUnits.SECONDS, 256Options Data, 256recordFormat, 258RecordFormat.RECFMT_FIXED, 258RecordFormat.RECFMT_FREE, 258Required Software, 235Retrieving Data Records, 243Sample Programs, 260Storing a Vector of Values, 256tickMode, 258

Commodity DataServer Server Architecture, 5Commodity DataServer VBA, 303

(see also VBA API)Commodity DataServer Visual Basic for Applications API,303, 303

(see also VBA API)Examples, 429Installing, 303Relation and Data Loading, 436

compareTo, 241compute_trading_date_range Method, 453compute_trading_pattern Method, 453Cond


ConnectXMIMServer

VBA API, 308ConnectionFactory, 238Constant


Continuous ContractsBMIM, 53

continuousLoggingPxmimrc File Entry, 615

ContractXMIMRecords and XMIMRecordsDouble

VBA API, 331

contract_units, 30ContractUnits


Copying Database Contents, 587Correction Audits, 200, 475

Accessing Data, 204, 480Adding to Table Facility, 52BMIM Synopsis, 476Deleting Correction Audits from Table Facility, 202, 478File Format, 203, 479Retrieve or Update Corrections from Table Facility,205, 481Retrieving Corrections from Table Facility, 201, 477Table Facility, 200, 475

CorrectionsTable Facility

Daily Table, 205, 482Intraday Table, 205, 482

corrections_add, 52CorrectionsDate


CorrectionsFileXMIMRecords and XMIMRecordsDouble

VBA API, 340cs_contract_builder, 208cs_regex, 207CtdDate


CurrentTickUsage, 247XMIMExecution and XMIMExecutionDouble




VBA API, 418CurrentTickUsage.APPEND_TO_ALL, 248, 248CurrentTickUsage.APPEND_TO_DAILY, 248, 248CurrentTickUsage.APPEND_TO_NONE, 248CurrentTickUsage.APPEND_TO_TICK, 248Custom Search Facility, 206

Writing Custom Searches, 207

622


Custom Searches, 210customSearch

xmimrc File Entry, 613

DDaily Table, 205, 482DailyMultiplicity


Data CompressionBMIM, 63

Data FilesPartitioning with BMIM, 54

Data Loading, 436Data Retrieval

Commodity DataServer C/C++ API, 75Commodity DataServer Java API, 240

Data Security, 534Free Pass, 535

Data UpdatingCommodity DataServer C/C++ API, 107Commodity DataServer Java API, 254

data_size, 64data_type, 63Database


databasexmimrc File Entry, 612

Database and Data GuidelinesData Loading, 17Loading Data, 17

Database Integrity Check, 591Database Partition Table

Printing, 57Database Schema Information

Printing with BMIM, 58Database Security, 532

Free Pass, 532If lock_files is Denied, 533xmimlock.acl File Format, 533

database_def_daily_file, 54, 57database_def_intraday_file, 54database_def_millisecond_file, 54, 55, 150, 561database_def_tick_file, 54database_def_write, 57

database_narrow, 26, 27, 27, 27, 27, 27, 51, 57database_widen, 27, 27, 27, 27DataDate


DataTypeXMIMRelation



VBA API, 422Date


date fields, 43DateFormat

XMIMUtilsVBA API, 424

datesDirxmimrc File Entry, 613

DateTimeXMIMCurrentTick

VBA API, 407XMIMRecords and XMIMRecordsDoubleds

VBA API, 318DateTimeArray


dateTimes, 248dbcompare Utility, 601

Example Configuration File, 602Usage, 601

DeleteXMIMColumn




VBA API, 402delete_facts, 48DeleteFacts



623

DeleteFactsSelectiveXMIMRelCol

VBA API, 365deleteRelation, 274DeleteResult


DeleteTupleXMIMTable

VBA API, 403Deleting Columns

BMIM, 33Deleting Correction Audits

Commodity DataServer C/C++ API, 202, 478Deleting Facts, 48, 152Deleting Relation Columns

BMIM, 35Commodity DataServer C/C++ API, 161

Deleting RelationsBMIM, 28, 40Commodity DataServer C# .NET API, 273Commodity DataServer C/C++ API, 153

delta, 38Delta


DescriptionXMIMColumn


VBA API, 343DesiredExpirDate


DesiredOptTypeXMIMRecordsDouble

VBA API, 324DesiredStrikePrice


DestructStored Function, 446

dirname, 64Disconnect


DownloadMicrosoft .NET Framework, 271

dump_expiration_dates, 64

Eending_trading_time, 30EndingDateTime


EndTradeXMIMRelation

VBA API, 344Entitlements

BMIM, 64Enabling/Disabling, 536Example, 65

entitlementsxmimrc File Entry, 615

entitlements_clear, 65entitlements_read, 65entitlements_write, 65entitlements_write_all, 65Error Messages

Commodity DataServer .NET API, 291Commodity DataServer Java API, 269

exchange, 30Exchange



ExecuteXMIMExecution and XMIMExecutionDouble


VBA API, 374ExecuteFunction


ExecuteSavedQueryXMIMShowWhen and XMIMShowWhenDouble

VBA API, 374Executing Queries

BMIM, 53Commodity DataServer C# .NET API, 284Commodity DataServer C/C++ API, 98

624


Commodity DataServer Java API, 264ExpDate


ExpDateFormatXMIMUtils

VBA API, 424Expiration Dates

Printing with BMIM, 64expiration_date, 29, 29, 30ExpirDate


Ffacts_read, 41, 45, 45, 46, 48, 55, 63, 63, 63facts_write, 47FactsFile


FetchXMIMColumn



VBA API, 364FieldAggr


FieldConstantXMIMRelCol

VBA API, 360FieldDelta


FieldDescXMIMColumn

VBA API, 354FieldFormatDecimal


fieldFormats, 258FieldFormatType


FieldFormatWidthXMIMRecords and XMIMRecordsDouble

VBA API, 332FieldLength


FieldNameXMIMColumn


VBA API, 395FieldObjType


FieldTypeXMIMTable


VBA API, 423File Loader

Data Loading, 509File-Based Data Retrieval/Updating

XMIMRecords and XMIMRecordsDouble, 315FillOption


FillOption.FILL_BACKWARD, 247FillOption.FILL_FORWARD, 247FindCols


FindRelations, 274FindRels


first_notice_date, 30Fold


FoldIntoDailyPXMIMCurrentTick

VBA API, 409FoldIntoIntradayP


FoldIntoTickP


625


formatxmimrc File Entry, 613

Formula Loader, 513formula_add, 51Formulas

Adding with BMIM, 51ForwardCurve

Commodity DataServer C# .NET API, 275Free Pass

Data Security, 535Database Security, 532Server Security, 531

from_date, 46from_time, 46fromDate, 243FromDate




FromExpirDateXMIMRecords and XMIMRecordsDouble

VBA API, 324FromStrike


fromTime, 244FromTime



FromUnitsXMIMRecords and XMIMRecordsDouble

VBA API, 329FunctionName


Futures DataConcepts, 498Continuous Contracts, 505, 506Loading, 497

make_data File, 503Naming Conventions, 501

futures_continuous, 30, 32Naming Conventions, 502syntax, 32

futures_contract, 30, 32Naming Conventions, 501

GGenerating Continuous Contracts, 167Generator, 208

Custom Searches, 208GetAllCols


GetAllExchangesXMIMSchema

VBA API, 391GetAllFunctionNames


GetAllRelsXMIMSchema

VBA API, 391GetAllTableNames


GetAllTimeZonesXMIMSchema

VBA API, 391GetChildren()

Subroutine, 435GetCorrections, 277getdata(), 429

Outline, 430Subroutine, 431

getdata() subroutine, 429GetDatabases


getDataRange, 240, 242GetDataRange


GetDataRangeOptionXMIMRecords and XMIMRecordsDouble

626


VBA API, 336GetFieldsInfo


GetFunctionInfoXMIMCustomSearch

VBA API, 415GetInfo


getRecords, 243GetRecords, 278


GetRecordsOption, 279XMIMRecords and XMIMRecordsDouble

VBA API, 334GetRecordsOptionRelative


GetRecordsRolloverXMIMRecords and XMIMRecordsDouble

VBA API, 337GetRelation, 281GetRelation Routine, 260GetRelationColumn, 282getRelChildren, 240GetRelChildren, 274GetRelChildren Routine

Commodity DataServer Java API, 262getRelColumns, 242GetRelColumns, 283GetRollOverDates, 283GetRolloverDates


GetTradingTimeXMIMRecords and XMIMRecordsDouble

VBA API, 336GetTuple


GraphPXMIMShowWhen and XMIMShowWhenDouble

VBA API, 373

HHandle


High Frequency UpdatingCommodity DataServer C/C++ API, 114Commodity DataServer Java API, 258

Holiday SchedulesAssigning, 49

holiday_fillxmimrc File Entry, 613

HolidayFillXMIMExecution and XMIMExecutionDouble



VBA API, 369holidayFillOption, 245HolidayNaN


holidaysDirxmimrc File Entry, 613

HostXMIMServer

VBA API, 307

Iignore, 41Index Constituent Utility, 597

Examples, 597Usage, 597

InheritFieldXMIMTable

VBA API, 396Init







627






VBA API, 394Initialize

Default Values, 445State, 445

InputFileXMIMShowWhen and XMIMShowWhenDouble

VBA API, 370Installing

Commodity DataServer Visual Basic for ApplicationsAPI, 303MS Studio, 271

Installing and UsingCommodity DataServer .NET API, 271

Intraday Table, 205, 482IntradayMultiplicity


Invoking a ServerCommodity DataServer C/C++ API, 67

IsHolidayNaNXMIMUtils (Utilities)

VBA API, 428IsMissDataNaN


IsNaNXMIMUtils (Utilities)

VBA API, 426

KKeyField


Llazy_predictor, 462

libraryDirxmimrc File Entry, 613

licensexmimrc File Entry, 612

LimitXMIMRecords and XMIMRecordsDouble

VBA API, 321limitMode, 247LimitMode



LoadBmimScriptXMIMShowWhen and XMIMShowWhenDouble

VBA API, 374Loading, 209

Custom Searches, 209Loading Data

Commodity DataServer C# .NET API, 284Loading Data from a File

Commodity DataServer C/C++ API, 113Commodity DataServer Java API, 257File Loader, 509

LockedXMIMCurrentTick

VBA API, 409XMIMServer

VBA API, 310Locking Database

BMIM, 27Locking the Database


LoggingMIM Server, 539

Mmake_data File, 503max_depth, 58MaxDepth


Memory ManagementCommodity DataServer C/C++ API, 123

mergeMode, 258

628


MergeModeXMIMRecords and XMIMRecordsDouble




VBA API, 422Metadata Terminology


Methodscompute_trading_date_range, 453compute_trading_pattern, 453register_builtin_cstored_func, 454

Microsoft .NET Framework Download, 271millisecond, 94, 111, 114, 150, 153, 164, 165, 176, 178,186, 188, 225, 256, 258, 472, 566MIM Server

Invoking, 525Logging, 539

MimConnection, 238MimConnection.disconnect, 239, 239MimConnection.lockDb, 254MimConnection.lockRealTimeStore, 259MimConnection.unlockDb, 254MimData.factsRead, 258, 259MimData.getRecords, 256, 260MimData.putRecords, 256, 258, 259MimData.queryExecute, 250, 250MimData.readFacts, 257MimDate, 240MimDate.INVALID_DATE, 240MimDateTime.INVALID_DATE_TIME, 240MIMDBCK Utility, 591

Setup, 592Usage, 591

MIMDBCP Utility, 587Methods for Running the Utility, 588Setup, 588Usage, 587

MimException, 239MimNetworkException, 239MimRealTime.updateRealTime, 258, 259MimRealTime.updateRealTimeBar, 259, 259

MimSchema.addColumn, 255MimSchema.addRelation, 255MimSchema.addRelationColumn, 255MimTime, 240MimTime.INVALID_TIME, 240, 244MimUnits, 244MimUnits.DAYS, 245, 256MimUnits.HOURS, 245MimUnits.MILLISECONDS, 245, 256MimUnits.MINUTES, 245, 256MimUnits.MONTHS, 245MimUnits.QUARTERS, 245MimUnits.SECONDS, 245, 256MimUnits.WEEKS, 245MimUnits.YEARS, 245miss_data_filllicense

xmimrc File Entry, 613MissDataFill




missDataFillOption, 245MissDataNaN


ModifyXMIMColumn



VBA API, 364Modifying Columns

BMIM, 33Modifying Relation Columns

BMIM, 35Multiple Databases

BMIM, 26multiple threads

Commodity DataServer .NET API, 271Commodity DataServer C/C++ API, 67Commodity DataServer Java API, 235

multiplicity, 37


629

NName




NArgsXMIMCustomSearch

VBA API, 414NarrowDatabases


NAttrsXMIMExecution and XMIMExecutionDouble


VBA API, 366Nchildren



NColsXMIMCurrentTick

VBA API, 406XMIMExecution and XMIMExecutionDouble



VBA API, 389NColsOfAttr


NColumnsXMIMRelation

VBA API, 348NColumnsWithData


NCondsXMIMShowWhen and XMIMShowWhenDouble

VBA API, 367NContractsPerPeriod


NCtdDatesXMIMCurrentTick

VBA API, 409NDatabases


NewDatabaseXMIMServer

VBA API, 312NewTable


NExchangesXMIMSchema

VBA API, 390NFieldFormats


NFieldsXMIMColumn


VBA API, 395Nfields


NFromUnitsXMIMRecords and XMIMRecordsDouble

VBA API, 329NFunctionNames


NoDateXMIMUtils (Utilities)

VBA API, 428NoticeDate


NPeriodsXMIMRecords and XMIMRecordsDouble

VBA API, 330NRecords


630


XMIMRecordsVBA API, 318


NRelsXMIMCurrentTick



VBA API, 389NReports


NResultsXMIMCustomSearch

VBA API, 415NRightSides


NRowsXMIMExecution and XMIMExecutionDouble

VBA API, 382NSubs


NTablesXMIMTable

VBA API, 401NthDatabase


NthFunctionNameXMIMCustomSearch

VBA API, 412NTimeZones


NToUnitsXMIMRecords and XMIMRecordsDouble

VBA API, 329numUnits, 244NumUnitsToExpir


Nunits


NUnitsXMIMExecution and XMIMExecutionDouble



VBA API, 367

Oobject_type, 38ObjType


ONC/RPC Error MessagesCommodity DataServer .NET API, 291Commodity DataServer Java API, 269

OpenDatabaseXMIMServer

VBA API, 312OpenTable


optionxmimrc File Entry, 614

option_type, 29Optional Arguments

Commodity DataServer C/C++ API, 72OptionType

XMIMRecordsDoubleVBA API, 327


output_filename, 54OutputFile



OverviewXMIMRelation, 340

overwrite_mode, 54

PPackage Maker, 609


631

Data Loading, 491Example Usage, 609How to Use, 609Loading Data, 491Requirements, 609

ParentXMIMColumn


VBA API, 343Path



PatternXMIMSchema

VBA API, 385PeriodEnd


PeriodStartXMIMRecords and XMIMRecordsDouble

VBA API, 330print_*_file, 57print_daily_file, 57print_intraday_file, 57print_relation, 58print_schema, 58, 58, 58print_tick_file, 57PrintCols


PrintingDatabase Partition Table with BMIM, 57Database Schema with BMIM, 58Location of a Relation and Column with BMIM, 57

Printing Expiration DatesBMIM, 64

PrintPXMIMSchema


VBA API, 372PrintRels

XMIMSchema

VBA API, 392Publisher, 595Publisher Utility, 595, 595publishExactPrn

xmimrc File Entry, 614Putrecords, 284PutRecords


PutRecordsOptionXMIMRecords and XMIMRecordsDouble

VBA API, 338

QQueries

Executing with BMIM, 53Query


query_execute, 53, 250query_filename, 54query_graph, 53QueryExecute, 284QueryExecute Routine

Commodity DataServer Java API, 264

RRead Locking, 615read_lock

xmimrc File Entry, 615ReadFacts


ReadOnlyPXMIMTable

VBA API, 396realTick, 259RealTickP


RecFormatXMIMRecords and XMIMRecordsDouble

VBA API, 333recordFormat, 258RecordFormat

XMIMUtils

632


VBA API, 423RecordFormat.RECFMT_FIXED, 258RecordFormat.RECFMT_FREE, 258, 258RecordSet, 248regen_continuous, 53regen_continuous Command, 506RegenRollover


register_builtin_cstored_func Method, 454Rel





Relation AliasesAdding, 50

Relation ColumnsAdding/Modifying/Deleting with BMIM, 35

relation_add, 28relation_alias_add, 50, 51relation_column_add, 35relation_column_delete, 40relation_column_modify, 39, 40relation_column_type, 36relation_delete, 33relation_modify, 28, 32Relations

Adding/Modifying/Deleting with BMIM, 28RelColAggr


RelColDeltaXMIMUtils

VBA API, 421RelColObjType


RelColTypeXMIMRelCol


VBA API, 420relname, 53relNames, 243, 259RelRoot


RelTypeXMIMRelation


VBA API, 418Remove


RenameXMIMTable

VBA API, 402ReplaceRecords


ReplaceRecordsOptionXMIMRecords and XMIMRecordsDouble

VBA API, 339Replacing/Deleting Data

XMIMRecords, 315Replicating XMIM Queries, 432ReportTitle


ResultXMIMCustomSearch

VBA API, 415Retrieving Correction Audits

Commodity DataServer C/C++ API, 201, 477Retrieving Corrections

Commodity DataServer C# .NET API, 276Retrieving Data

XMIMRecords, 313Retrieving Data Records

BMIM, 48Commodity DataServer C# .NET API, 278Commodity DataServer C/C++ API, 81, 87Commodity DataServer Java API, 240

RolloverXMIMRecords, 314

rollover_data_type, 30


633

rollover_date, 30rollover_policy, 30RolloverDataType


RolloverDayXMIMRecords and XMIMRecordsDouble


VBA API, 346RolloverPolicy



RolloverTypeXMIMRelation

VBA API, 347Root



RowColValueXMIMExecution and XMIMExecutionDouble

VBA API, 383RowDate


RowTimeXMIMExecution and XMIMExecutionDouble

VBA API, 383

SSafe Mode for Database

BMIM, 61Sample Programs

Commodity DataServer Java API, 260SearchFilter



SearchRootXMIMCustomSearch

VBA API, 414

SelectiveFoldXMIMCurrentTick

VBA API, 410SelectOnceP


Server Connect/DisconnectCommodity DataServer C/C++ API, 70Commodity DataServer Java API, 238

Server Security, 529, 529(see also Server Security)Free Pass, 531Global Setting, 530xmimsvr.acl File, 530

Server xmimrc File (see xmimrc File)ServerNum


ServerOwnerXMIMServer

VBA API, 311ServerPid


set_compress, 63, 63set_compress_threshold, 64set_holiday_schedule, 49, 49SetDefaultFactsFile


SetExchangeHolidaysXMIMRelation

VBA API, 350SetRelHolidays


Shared Libraries, 209showwhendouble()

Subroutine, 433SingleValP


SkipAllNaN.SKIP_ALL_NAN, 247skipAllNaNRecords, 247SkipEmptyRecords


634



VBA API, 369Software Installations

Data Loading, 489Loading Data, 489

Specification File, 210Custom Searches, 210

starting_trading_time, 30StartTrade


Storing a Vector of ValuesCommodity DataServer Java API, 256

strike_price, 29StrikePrice


StrikeWeightXMIMRecords and XMIMRecordsDouble

VBA API, 326StringTo<type>


StudyXMIMRecordsDouble

VBA API, 328Sub GetChildren() Outline, 434Sub loadNewData(), 437, 437Sub showwhendouble() Outline, 432SubsLeftSide


SubsRightSideXMIMShowWhen and XMIMShowWhenDouble

VBA API, 372SuccessP


suppress_bad_date_time, 63suppress_duplicate_add, 62suppress_duplicate_delete, 63suppress_relation_not_found, 63syntax, 32system command, 62

TTable Facility, 200, 201, 201, 475, 477

BMIM, 52Deleting Correction Audits

Commodity DataServer C/C++ API, 202, 478Retrieve or Update Corrections, 205, 481

TableFieldTypeXMIMUtils

VBA API, 425TableName


tempDirxmimrc File Entry, 613

TesterCustom Searches, 207

tick_by_tick, 41tick_convert and tick_millisecond,, 41tick_intraday, 41tickMode, 258TickMode



TickMultiplicityXMIMRelCol

VBA API, 362time fields, 42Time Series

Compute Characteristic, 446time_zone, 30TimeFormat


TimeZoneXMIMRelation


VBA API, 390to_date, 46to_time, 46toDate, 243ToDate


635



ToExpirDateXMIMRecords and XMIMRecordsDouble

VBA API, 325ToStrike


toTime, 244ToTime



ToUnitsXMIMRecords and XMIMRecordsDouble

VBA API, 330Trading Date Range

Compute, 446Trading Pattern

Compute, 445trading_pattern, 28TupleFieldAtom


TupleFieldBlockXMIMTable

VBA API, 399TupleFieldCol


TupleFieldDateXMIMTable

VBA API, 400TupleFieldDouble


TupleFieldFloatXMIMTable

VBA API, 398TupleFieldInt


TupleFieldRel


TupleFieldStringXMIMTable

VBA API, 398TupleFieldTime


TupleFieldWildXMIMTable

VBA API, 397

UUnionDate


UnitsXMIMExecution and XMIMExecutionDouble


VBA API, 321XMIMServer



Reference, 417Units of Measurement (UOM)

Data Loading, 519Loading Data, 519

UnitsToExpirXMIMRecords and XMIMRecordsDouble

VBA API, 326Unix Interface

BMIM, 62Unlocking Database

BMIM, 27unset_compress, 63, 63upd Package

Creating, 609Unpacking, 610

UpdateXMIMCurrentTick

VBA API, 410Updating a Database

BMIM, 27

636


Updating DataXMIMRecords, 315

UseClientMacrosPXMIMExecution and XMIMExecutionDouble

VBA API, 378Utilities

dbcompare, 601Index Constituent, 597MIMDBCK, 591MIMDBCP, 587Publisher, 595xmim_find, 579xmim_get, 571xmim_get_col_paths, 583xmim_get_options, 573xmim_get_rel_paths, 581xmim_pair_gen, 603xmim_put, 585xmim_svr_info, 577xmim_trading_info, 575

Utility Functions, 447

VVal




ValArrayXMIMRecords and XMIMRecordsDouble

VBA API, 319VBA API, 303, 303

Constants, 306Creating and releasing objects, 305Dates, 305Detecting and Handling Errors, 305NaNs, 305Object Types, 303Programming With, 304Reference to Library, 304XMIMColumn, 352

Add, 355Child, 355Clear, 352

ColType, 353Delete, 355Description, 353Fetch, 356FieldDesc, 354FieldName, 354Init, 352Modify, 356Name, 352Nchildren, 354NFields, 353Parent, 353Path, 355Root, 355

XMIMCurrentTick, 405Clear, 405Col, 406CtdDate, 409DataDate, 406DateTime, 407Fold, 410FoldIntoDailyP, 409FoldIntoIntradayP, 408FoldIntoTickP, 408FromDate, 407Init, 405Locked, 409NCols, 406NCtdDates, 409NRecords, 406NRels, 405RealTickP, 408Rel, 405Remove, 411SelectiveFold, 410ToDate, 407Update, 410Val, 407

XMIMCustomSearch, 412Arg, 414ArgDesc, 414ArgType, 415Clear, 412ExecuteFunction, 416FunctionName, 412GetAllFunctionNames, 415


637

GetFunctionInfo, 415Init, 412NArgs, 414NFunctionNames, 412NResults, 415NthFunctionName, 412Result, 415SearchRoot, 414

XMIMExecution and XMIMExecutionDouble, 378AttrHeading, 382ColHeading, 382CurrentTickUsage, 381DeleteResult, 383Execute, 383HolidayFill, 380Init, 378MissDataFill, 379NAttrs, 381NCols, 382NColsOfAttr, 382NReports, 381NRows, 382NUnits, 379Query, 378ReportTitle, 381RowColValue, 383RowDate, 383RowTime, 383Units, 379UseClientMacrosP, 378

XMIMRecords and XMIMRecordsDouble, 313, 316AddCorrections, 340Clear, 317Col, 318Contract, 331CorrectionsDate, 320CorrectionsFile, 340CurrentTickUsage, 323DateTime, 318DateTimeArray, 318DesiredExpirDate, 323DesiredOptType, 324DesiredStrikePrice, 324EndingDateTime, 331ExpirDate, 327FactsFile, 331

FieldFormatDecimal, 333FieldFormatType, 332FieldFormatWidth, 332FromDate, 319FromExpirDate, 324FromStrike, 325FromTime, 320FromUnits, 329GetDataRange, 335GetDataRangeOption, 336GetRecords, 334GetRecordsOption, 334GetRecordsOptionRelative, 335GetRecordsRollover, 337GetRolloverDates, 337GetTradingTime, 336HolidayFill, 322Init, 316Limit, 321LimitMode, 322MergeMode, 333MissDataFill, 322NCols, 317NContractsPerPeriod, 330NFieldFormats, 332NFromUnits, 329NRecords, 318NRels, 317NToUnits, 329NumUnitsToExpir, 326NUnits, 321OptionType, 327PeriodEnd, 331PeriodStart, 330PutRecords, 338PutRecordsOption, 338ReadFacts, 339RecFormat, 333Rel, 317ReplaceRecords, 338ReplaceRecordsOption, 339RolloverDay, 328RolloverPolicy, 328SelectOnceP, 327SkipEmptyRecords, 322StrikePrice, 328

638


StrikeWeight, 326Study, 328TickMode, 334ToDate, 319ToExpirDate, 325ToStrike, 325ToTime, 320ToUnits, 330, 330Units, 321UnitsToExpir, 326Val, 319ValArray, 319WriteFacts, 339

XMIMRelation, 341Add, 349AddAlias, 349Alias, 342Child, 347Clear, 341Column, 348ContractUnits, 344DataType, 347Delete, 349Description, 343EndTrade, 344Exchange, 344ExpDate, 346Fetch, 350Init, 341Modify, 349Name, 342Nchildren, 347NColumns, 348NColumnsWith, 348NoticeDate, 346Parent, 343Path, 348RegenRollover, 351RelType, 343RolloverDay, 346RolloverPolicy, 346RolloverType, 347Root, 348SetDefaultFactsFile, 350SetExchangeHolidays, 350SetRelHolidays, 350

StartTrade, 344TimeZone, 344

XMIMRelCol, 357Add, 363Aggr, 359Clear, 357Col, 358Constant, 359Daily Multiplicity, 361DataType, 361Delete, 364DeleteFacts, 364DeleteFactsSelective, 365Delta, 359Fetch, 364FieldAggr, 360FieldConstant, 360FieldDelta, 360FieldObjType, 360FromDate, 362FromTime, 363Init, 357IntradayMultiplicity, 361Modify, 364Nfields, 359ObjType, 358Rel, 358RelColType, 358TickMultiplicity, 362ToDate, 362ToTime, 363

XMIMSchema, 384AllContractsP, 387ByNameP, 386CaseSensitiveP, 386Clear, 385Col, 390ColRoot, 385Exchange, 390FindCols, 392FindRels, 392GetAllCols, 391GetAllExchanges, 391GetAllRels, 391GetAllTimeZones, 391Init, 384

639

MaxDepth, 387MergeMode, 387NCols, 389NExchanges, 390NRels, 389NTimeZones, 390OutputFile, 386Pattern, 385PrintCols, 392PrintP, 387PrintRels, 392Rel, 389RelRoot, 385SearchFilter, 386TimeZone, 390VerboseP, 389

XMIMServer, 306, 307ClientType, 310CloseDatabase, 312Connect, 308Database, 311Disconnect, 309GetDatabases, 312GetInfo, 313Handle, 310HolidayNaN, 309Host, 307Locked, 310MissDataNaN, 309NarrowDatabases, 308NDatabases, 311NewDatabase, 312NthDatabase, 311Nunits, 309OpenDatabase, 312ServerNum, 307ServerOwner, 311ServerPid, 312Units, 310WidenDatabases, 308

XMIMShowWhen and XMIMShowWhenDouble, 366Clear, 366Init, 366, 367, 367, 367, 367, 368, 368, 368, 369,369, 369, 370, 370, 370, 371, 371, 371, 371, 372,372, 372, 372, 373, 373, 373, 373, 374, 374, 374,374

NAttrs, 366XMIMTable, 394

AddTuple, 403CacheSize, 397Clear, 394CloseTable, 402Delete, 402DeleteTuple, 403FieldLength, 395FieldName, 395FieldType, 395GetAllTableNames, 402GetFieldsInfo, 402GetTuple, 403InheritField, 396Init, 394KeyField, 396Name, 394NewTable, 401NFields, 395NTables, 401OpenTable, 401ReadOnlyP, 396Rename, 402SingleValP, 396TableName, 401TupleFieldAtom, 399TupleFieldBlock, 399TupleFieldCol, 400TupleFieldDate, 400TupleFieldDouble, 398TupleFieldFloat, 398TupleFieldInt, 397TupleFieldRel, 399TupleFieldString, 398TupleFieldTime, 400TupleFieldWild, 397

XMIMUtils, 417XMIMUtils (Utilities), 426

<type>ToString, 428IsHolidayNaN, 428IsMissDataNaN, 428IsNaN, 426NoDate, 428StringTo<type>, 428

Zero-Based, 306

640


VBA API Library, 429verbose, 54VerboseP


VerbosityXMIMShowWhen and XMIMShowWhenDouble


VBA API, 423veryverbose, 54

WWidenDatabases


WriteFactsXMIMRecords and XMIMRecordsDouble

VBA API, 339writer_wait, 61Writing Custom Searches, 207

XXMIM_CONTIGUOUS_RANGE, 179XMIM_CORRECTIONS, 204, 204, 204, 204, 205, 476,477, 477, 477, 477, 478, 478, 478, 478, 480, 480, 480,481, 481, 481, 482XMIM_CORRECTIONS_DAILY, 478, 479, 479XMIM_CORRECTIONS_INTRADAY, 478, 479, 479XMIM_CORRECTIONS_INTRADAY_YYYYMMDD, 205,482xmim_find Utility, 579


xmim_get Utility, 571Examples, 571Source Code, 572Usage, 571

xmim_get_col_paths Utility, 583xmim_get_options Utility, 573


xmim_get_rel_paths Utility, 581XMIM_INCLUDE_HIDDEN, 170xmim_pair_gen Utility, 603xmim_put Utility, 585

Usage, 585xmim_svr_info Utility, 577

Usage, 577XMIM_TIME_RANGE_TYPE, 179xmim_trading_info Utility, 575

Usage, 575XMIM_TRADING_RANGE, 179XmimAddCorrections, 476, 476XMIMColumn, 351, 352

Add, 355Child, 355Clear, 352ColType, 353Delete, 355Description, 353Fetch, 356FieldDesc, 354FieldName, 354Init, 352Modify, 356Name, 352Nchildren, 354NFields, 353Overview, 351Parent, 353Path, 355Root, 355

XmimCorrectionsType, 478, 479XMIMCurrentTick, 403, 405

Clear, 405Col, 406CtdDate, 409DataDate, 406DataTime, 407Fold, 410FoldIntoDailyP, 409FoldIntoIntradayP, 408FoldIntoTickP, 408FromDate, 407Init, 405Locked, 409NCols, 406NCtdDates, 409NRecords, 406NRels, 405Overview, 403


641

RealTickP, 408Rel, 405Remove, 411SelectiveFold, 410ToDate, 407Update, 410Val, 407

XMIMCustomSearch, 411, 411, 412Arg, 414ArgDesc, 414ArgType, 415Clear, 412ExecuteFunction, 416FunctionName, 412GetAllFunctionNames, 415GetFunctionInfo, 415Init, 412NArgs, 414NFunctionNames, 412NResults, 415NthFunctionName, 412Result, 415SearchRoot, 414

XmimDeleteCorrections, 479XMIMExecution, 250XMIMExecution and XMIMExecutionDouble, 375, 378

AttrHeading, 382ColHeading, 382CurrentTickUsage, 381DeleteResult, 383Execute, 383HolidayFill, 380Init, 378MissDataFill, 379NAttrs, 381NCols, 382NColsOfAttr, 382NReports, 381NRows, 382NUnits, 379Overview, 375Query, 378ReportTitle, 381RowColValue, 383RowDate, 383RowTime, 383

Units, 379UseClientMacrosP, 378

XmimGetCorrections, 478XmimGetCustomSearchArgs, 206XmimGetCustomSearches, 206XmimProcessCustomSearch, 206XmimQueryExecute, 103XMIMQueryExecute, 250xmimrc File, 611

Custom Search Spec File Location, 613Database Used, 612Directory Locations, 613Entitlements Usage, 615Entries

advUpdatesLogging, 614, 614, 615continuousLoggingP, 615customSearch, 613database, 612datesDir, 613entitlements, 615format, 613holiday_fill, 613holidaysDir, 613libraryDir, 613license, 612miss_data_fill, 613option, 614publishExactPrn, 614read_lock, 615tempDir, 613

Example, 612Executing, 616Global Options, 614License File Location, 612Locating, 611NaN Handling, 613Print Directly from prn File in facts_read, 614Printing Formats, 613Publish Continuous Daily Contract Changes, 615Read Locking, 615Specifications, 611

XMIMRecordsFile-Based Data Retrieval/Updating, 315Replacing/Deleting Data, 315Updating Data, 315

XmimRecords, 430, 431

642


XMIMRecords and XMIMRecords andXMIMRecordsDouble

FromTime, 320XMIMRecords and XMIMRecordsDouble, 316

AddCorrections, 340Clear, 317Col, 318Contract, 331CorrectionsDate, 320CorrectionsFile, 340CurrentTickUsage, 323DateTime, 318DateTimeArray, 318DesiredExpirDate, 323DesiredOptType, 324DesiredStrikePrice, 324EndingDateTime, 331ExpirDate, 327FactsFile, 331FieldFormatDecimal, 333FieldFormatType, 332FieldFormatWidth, 332FromDate, 319FromExpirDate, 324FromStrike, 325FromUnits, 329GetDataRange, 335GetDataRangeOption, 336GetRecords, 334GetRecordsOption, 334GetRecordsOptionRelative, 335GetRecordsRollover, 337GetRolloverDates, 337GetTradingTime, 336HolidayFill, 322Init, 316Large Data, 314Limit, 321LimitMode, 322MergeMode, 333MissDataFill, 322NCols, 317NContractsPerPeriod, 330NFieldFormats, 332NFromUnits, 329NPeriods, 330

NRecords, 318NRels, 317NToUnits, 329NumUnitsToExpir, 326NUnits, 321OptionType, 327Overview, 313PeriodEnd, 331PeriodStart, 330PutRecords, 338PutRecordsOption, 338ReadFacts, 339RecFormat, 333Rel, 317ReplaceRecords, 338ReplaceRecordsOption, 339Retrieving Data, 313Rollover, 314RolloverDay, 328RolloverPolicy, 328SelectOnceP, 327SkipEmptyRecords, 322StrikePrice, 328StrikeWeight, 326Study, 328TickMode, 334ToDate, 319ToExpirDate, 325ToStrike, 325ToTime, 320ToUnits, 330Units, 321UnitsToExpir, 326Val, 319ValArray, 319WriteFacts, 339

XMIMRecords, XMIMRecordsDouble, 313XmimRecordsDouble, 429, 429, 430, 430, 431, 439, 439XMIMRelation, 340, 341

Add, 349AddAlias, 349Alias, 342Child

Child, 347Clear, 341Column, 348


643

ContractUnits, 344DataType, 347Delete, 349Description, 343EndTrade, 344Exchange, 344ExpDate, 346Fetch, 350Init, 341Modify, 349Name, 342Nchildren, 347NColumns, 348NColumnsWithData, 348NoticeDate, 346Parent, 343Path

Path, 348RegenRollover, 351RelType, 343RolloverDay, 346RolloverPolicy, 346RolloverType

RolloverType, 347Root, 348SetDefaultFactsFile, 350SetExchangeHolidays, 350SetRelHolidays, 350StartTrade, 344TimeZone, 344

XMIMRelCol, 356, 357Add, 363Aggr, 359Clear, 357Col, 358Constant, 359DailyMultiplicity, 361DataType, 361Delete, 364DeleteFacts, 364DeleteFactsSelective, 365Delta, 359Fetch, 364FieldAggr, 360FieldConstant

FieldConstant, 360

FieldDelta, 360FieldObjType, 360FromDate, 362FromTime, 363Init, 357IntradayMultiplicity, 361Modify, 364Nfields, 359ObjType, 358Overviewl, 356Rel, 358RelColType, 358TickMultiplicity, 362ToDate, 362ToTime, 363

XMIMSchema, 384, 384AllContractsP, 387ByNameP, 386CaseSensitiveP, 386Clear, 385Col, 390ColRoot, 385Exchange, 390FindCols, 392FindRels, 392GetAllCols, 391GetAllExchanges, 391GetAllRels, 391GetAllTimeZones, 391Init, 384MaxDepth, 387MergeMode, 387NCols, 389NExchanges, 390NRels, 389NTimeZones, 390OutputFile, 386Overview, 384Pattern, 385PrintCols, 392PrintP, 387PrintRels, 392Rel, 389RelRoot, 385SearchFilter, 386TimeZone, 390

644


VerboseP, 389XmimSchema, 434XmimSelectRecords, 100XMIMServer, 306, 307

ClientType, 310CloseDatabase, 312Connect, 308Database, 311Disconnect, 309GetDatabases, 312GetInfo, 313Handle, 310HolidayNaN, 309Host, 307Locked, 310MissDataNaN, 309NarrowDatabases, 308NDatabases, 311NewDatabase, 312NthDatabase, 311Nunits, 309OpenDatabase, 312ServerNum, 307ServerOwner, 311ServerPid, 312Units, 310WidenDatabases, 308

XMIMShowWhen, 433XMIMShowWhen and XMIMShowWhenDouble, 365,366

AfterRepeat, 370Attr, 367BeforeRepeat, 370Clear, 366Cond, 367CurrentTickUsage, 369Date, 373Execute, 374ExecuteSavedQuery, 374GraphP, 373HolidayFill, 369Init, 366InputFile, 370LoadBmimScript, 374MergeMode, 372MissDataFill, 368

NAttrs, 366NConds, 367NRecords, 373NRightSides, 371NSubs, 371NUnits, 367OutputFile, 371Overview, 365PrintP, 372SkipEmptyRecords, 369SubsLeftSide, 371SubsRightSide, 372SuccessP, 373UnionDate, 368Units, 368Val, 374Verbosity, 372

XmimShowWhenDouble, 432, 432, 432, 432, 433, 434xmimsvr.acl File, 530XMIMTable, 393, 394

AddTuple, 403CacheSize, 397Clear, 394CloseTable, 402Delete, 402DeleteTuple, 403FieldLength, 395FieldName, 395FieldType, 395GetAllTableNames, 402GetFieldsInfo, 402GetTuple, 403InheritField, 396Init, 394KeyField, 396Name, 394NewTable, 401NFields, 395NTables, 401OpenTable, 401Overview, 393ReadOnlyP, 396Rename, 402SingleValP, 396TableName, 401TupleFieldAtom, 399

645

TupleFieldBlock, 399TupleFieldCol, 400TupleFieldDate, 400TupleFieldDouble, 398TupleFieldFloat, 398TupleFieldInt, 397TupleFieldRel, 399TupleFieldString, 398TupleFieldTime, 400TupleFieldWild, 397

XMIMUtils (Constants), 417AdjustedStudy, 421ColType, 418CurrentTickUsage, 418DataType, 422DateFormat, 424ExpDateFormat, 424FieldType, 423FillOption, 417LimitMode, 417MergeMode, 422OptionType, 425Overview, 416RecordFormat, 423RelColAggr, 420RelColDelta, 421RelColObjType, 420RelColType, 420RelType, 418RolloverDataType, 421SearchFilter, 422TableFieldType, 425TickMode, 423TimeFormat, 425Units, 417XMIMUtils, 423

XMIMUtils (Utilities), 426, 426<type>ToString, 428IsHolidayNaN, 428IsMissDataNaN, 428IsNaN, 426NoDate, 428Overview, 426StringTo<type>, 428

XMIMUtils Constants), 416XmimWriteFacts, 93

XML API Routines, 104

646


Guide Commodity DataServer Data and · PDF fileCOMMODITY DATASERVER DATA AND DEVELOPMENT GUIDE...

Documents

Transcript of Guide Commodity DataServer Data and · PDF fileCOMMODITY DATASERVER DATA AND DEVELOPMENT GUIDE...