Citilabs - Atlanta Regional Commission

Citilabs®

Preparing Your Model for Cube Cloud Services


Preparing Your Model for Cube Cloud ServicesTutorialVersion 6.0

http://www.citilabs.com

Copyright © 2008–2012 Citilabs, Inc. All rights reserved.

Citilabs is a registered trademark of Citilabs, Inc. All other brand names and product names used in this book are trademarks, registered trademarks, or trade names of their respective holders.

The information contained in this document is the exclusive property of Citilabs. This work is protected under United States copyright law and the copyright laws of the given countries of origin and applicable international laws, treaties, and/or conventions. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying or recording, or by any information storage or retrieval system, except as expressly permitted in writing by Citilabs.

Citilabs has carefully reviewed the accuracy of this document, but shall not be held responsible for any omissions or errors that may appear. Information in this document is subject to change without notice.

Document Revision 60-000-0 April 25, 2012


Contents

Chapter 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

About Cube Cloud Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2About this Tutorial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Chapter 2 Preparing Your Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Understanding the CCS Directory Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2Input File References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Scenario-Specific Inputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3Fixed Model Inputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3Internal File Inputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Input Files with Conflicting Names . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5Output File References. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6Using Outputs as Internal Inputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7File Path Keys. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8File Tokens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9User Programs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Built-in user programs in Cube. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10Programs built by the Model Developer . . . . . . . . . . . . . . . . . . . . . . . . . . . 10Programs with special set-up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

System Commands (MS-DOS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12Cluster Setup in CCS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13Output/Input files between Multiple Applications . . . . . . . . . . . . . . . . . . . . . 15Reports and Applier Groups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16Summary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

Preparing Your Model for Cube Cloud Services iii

Contents

iv Preparing Your Model for Cube Cloud Services


1 Introduction

Welcome to Preparing Your Models for Cube Cloud Services!

This chapter introduces you to Cube Cloud Services, and provides an overview of this tutorial. Topics include:

• About Cube Cloud Services

• About this Tutorial

Preparing Your Model for Cube Cloud Services 1

IntroductionAbout Cube Cloud Services1

About Cube Cloud Services

Cube 6 provides the optional Cube Cloud Services (CCS), where models and their data may be published to a hosted web-accessed service. Once the model and data have been uploaded, they are accessed through a web site interface http://www.planthecity.com.

In the cloud, the model is owned and managed by one or more model administrators. The administrator can easily invite others to use their model, to get maps and charts or results and to upload and download data.

Model tests are done within a cloud computing framework where computer resources are essentially unlimited to users. Under this framework, models can be run across many computing cores resulting in very large reductions of run-time. The cloud also offers unlimited resources allowing multiple scenarios to be launched and run simultaneously.

2 Preparing Your Model for Cube Cloud Services

http://www.planthecity.com

IntroductionAbout this Tutorial 1

About this Tutorial

This document describes the steps required to ensure your Cube model may be used on the cloud. In Chapter 2, “Preparing Your Model,” you will learn the Best Practices for formatting your model, including:

• The Cube Cloud Services directory structure

• Input and output files

• File path keys

• File reference tokens

• User programs

• System (MS-DOS) commands

• Cluster setup

• And more!


IntroductionAbout this Tutorial1



2 Preparing Your Model

The topics in this chapter include:

• Understanding the CCS Directory Structure

• Input File References

• Input Files with Conflicting Names

• Output File References

• Using Outputs as Internal Inputs

• File Path Keys

• File Tokens

• User Programs

• System Commands (MS-DOS)

• Cluster Setup in CCS

• Output/Input files between Multiple Applications

• Reports and Applier Groups

• Summary


Preparing Your ModelUnderstanding the CCS Directory Structure2

Understanding the CCS Directory Structure

While Cube Base allows for a variety of model directory structures, Cube Cloud Services follows the fixed/minimal directory structure.

• The Model folder is uniquely named for each model. It serves as the model’s working directory, and its scenario directory. When the user exports the desktop model to CCS, the Cube exporter automatically creates this directory structure and copies the necessary files to the appropriate folders. The Apps sub-folder contains the model script file and

catalog information (as an .xml file). The Inputs sub-folder contains all model inputs, whether

scenario-specific inputs or fixed inputs. The Outputs sub-folder is Empty. It is used to store fixed

output files when the model is run.

When the model is run on CCS, each scenario run is assigned its own separate copy of the model folder. Outputs files to the scenario (or working) directory are saved in the Model folder, and output files to any other location are saved under the Outputs folder.


Preparing Your ModelInput File References 2

Input File References

Input File References in Cube Cloud Services include:

• Scenario-Specific Inputs — include zone data, the highway network, etc.

• Fixed Model Inputs — include friction factors, mode choice factors, etc.

• Internal File Inputs — input file references: internal (statements)these are script statements such as READ, LOOKUP, and RENUMBER.

Scenario-Specific Inputs

Scenario-Specific Inputs are represented by FILEI statements including {SCENARIO_DIR} in the path name. For example,

FILEI ZONEI[1]=“{SCENARIO_DIR}\Zones\ZoneData.DBF”

The Cube Base Cloud Services Exporter does not recognize scenario-specific inputs. The exporter will generate a warning:

Warning: Scenario Specific Inputs. The Application scripts are valid, but there is currently no interface to this data in CCS.

Solution

For Scenario-Specific Input files, instead use a File Key; for example,

FILEI ZONEI[1]=“{zonedata}”

Fixed Model Inputs

Fixed Model Inputs are FILEI statments with a fixed file path. For example:

FILEI RECI = “C:\Model\Parameters\MC_PAR.DBF“

The CCS Exporter will copy these files to the Inputs folder, and the script will be updated to:


Preparing Your ModelInput File References2

FILEI RECI = “{SCENARIO_DIR}\Inputs\MC_PAR.DBF”

Thus, the user will not have access to these types of files on the CCS interface. These files will be treated as internal files. Only the files which are input as catalog keys will be available for the user to update on http://www.planthecity.com. If the user wants to still be able to access these files on CCS, these files should be input using a catalog key.

NOTE: Always use double quotes for file paths and names.

Solution

Citilabs recommends a File Key when using a fixed input file; for example,

FILEI RECI[1]=“{recorddata}”

Internal File Inputs

Internal Input Files are FILE= statements using READ, RENUMBER and LOOKUP.

READ FILE = “{SCENARIO_DIR}\INPUT\ACCESS.PRN”

The CCS Exporter will not accept any file path keys in FILE= statements. Exporter does not copy this file to the Inputs directory, and will instead generate a warning:

Warning. The following Internal "FILE=" entries are present in the model. They have been associated with the VOYAGER command shown below, and copied to the appropriate folder. Note that for CCS Publish, the best and safest approach is to avoid use of FILE=,and use an alternative VOYAGER construct if available; e.g. for PRINT commands, use PRINTO=.

Solution

Use a user-defined key for internal file inputs, or reference these files using a {CATALOG_DIR} key:

READ FILE = “{ACCESS}”


http://www.planthecity.com

Preparing Your ModelInput Files with Conflicting Names 2

Input Files with Conflicting Names

The CCS Exporter appends a suffix (_cw#) to input files with similar names. It also updates the catalog key values for these files with updated file names.

For example, suppose the original model has these two files with these conflicting names:

{File1} = “C:\Model\Base\ZoneData.dbf”{File2} = “C:\Model\Base\Parameters\ZoneData.dbf”

The CCS Exporter will copy the two files to the Inputs folder with the names “ZoneData.dbf” & “ZoneData_cw1.dbf”, and update the file key values as follows:

{File1}=“inputs\ZoneData.dbf”{File2}=“inputs\ZoneData_cw1.dbf”


Preparing Your ModelOutput File References2

Output File References

Output File References use FILEO, PRINT FILE= and COPY . . . ENDCOPY statements. For example,

FILEO MATO[1] = “{SCENARIO_DIR}\OUTPUT\MATRIX\OUTPUT.MAT“

CCS will output these files to the Outputs directory, unless the file is output directly to the scenario directory. Any subsequent steps reading these files internally will fail in CCS.

Solution

Output all scenario specific files directly to “{SCENARIO_DIR}”; for example:

FILEO MATO[1] = “{SCENARIO_DIR}\OUTPUT.MAT”


Preparing Your ModelUsing Outputs as Internal Inputs 2

Using Outputs as Internal Inputs

The user must take care when using the outputs of FILEO, COPY and PRINTO statements as inputs to RENUMBER, LOOKUP (FILE=) statements. For example:

Program 1-Matrix:

FILEO PRINTO[1] = “{SCENARIO_DIR}\READ_OUT.PRN”

Program 2-Matrix:

READ FILE = “{SCENARIO_DIR}\READ_OUT.PRN“

In these cases, the CCS Exporter will generate an error message:

Warning: The following Internal files (defined via FILE=) could not be found, so the Applications will not run correctly

Solution

It is a good practice to directly link output files to inputs in subsequent programs, rather than reading them internally.


Preparing Your ModelFile Path Keys2

File Path Keys

File Path Keys are FILEI statements with user-defined file path keys. For example:

FILEI MATI[1] = “{LOCATION}\AllWalk.MAT”

CCS exporter does not accept any keys (other than {CATALOG_DIR}) in file PATH of FILEI statements. The exporter will generate an error such as:

Error: The following files have Keys in their Path. This construct is incompatible with CCS.

Solution

Use the {CATALOG_DIR} key to refer to this file as shown below, or use file keys as discussed above.

FILEI MATI[1] = "{CATALOG_DIR}\Base\Input\AllWalk.MAT"


Preparing Your ModelFile Tokens 2

File Tokens

File Tokens are file references using Tokens. For example,

READ FILE = "@CPI_FILE@"

Warning! CCS exporter will not provide any warnings or errors when it encounters file tokens. In this particular case, CCS will update the model script to read:

READ FILE = “{CATALOG}\OUTPUTS\@CPI_FILE@”

Solution

Use valid file paths instead of tokens.


Preparing Your ModelUser Programs2

User Programs

There are three kinds of User Programs commonly used in Cube models:

• Built-in user programs in Cube

• Programs built by the Model Developer

• Programs with special set-up

Built-in user programs in Cube

Built-in user programs include the TRCOPY and TRDEL batch files.

When the CCS Exporter encounters these, it will display a warning:

Warning: User Programs currently require some manual intervention to implement in CCS.

Otherwise, these built-in programs will work without issue in Cube Cloud Services.

Programs built by the Model Developer

User-supplied user programs are any executables (.exe) which perform special, custom functions.

Solution

Cube Cloud Services can run user-supplied user programs so long as these guidelines are followed:

1. All input files and output files to or from the user program should be referenced to the {SCENARIO_DIR}.

2. Any control files or input files with file paths are updated.

3. The user programs are executed from {SCENARIO_DIR}.


Preparing Your ModelUser Programs 2

Programs with special set-up

A model may use programs requiring a special, custom set-up. These include the Java Parallel Processing Framework, CT-RAMP, and many other possibilities.

Such user programs might require additional folder set-up and special configuration for running. Additional set-up might include background processes to support the model run.

Solution

Citilabs will create a new CCS image to handle these special cases according to each client’s needs.


Preparing Your ModelSystem Commands (MS-DOS)2

System Commands (MS-DOS)

System Commands are any command-line “MS-DOS” commands, such as COPY, DEL, and so on.

When CCS Exporter encounters System Commands, it will display a warning:

Only copy system commands with no wildcards are currently supported by CCS for file name conversion.

Solution

When using the COPY command, Citilabs recommendes that you copy the individual files using one copy command for each file.

For example, this will not work:

*Copy “{SCENARIO_DIR}\INPUTS\*.csv” “{SCENARIO_DIR}”

Instead, use:

*Copy “{SCENARIO_DIR}\INPUTS\File1.csv” “{SCENARIO_DIR}”*Copy “{SCENARIO_DIR}\INPUTS\File2.csv” “{SCENARIO_DIR}”..


Preparing Your ModelCluster Setup in CCS 2

Cluster Setup in CCS

Cube Cloud services provides scalability of available cluster nodes for each model run.

Guidelines

• The ProcessNum / ProcessList for DistributeMultistep and DistributeIntrastep statements should be calculated within the script based on the number of cores available for that run.

• The number of cores available for the model run is input using a scenario key.

• There is no need for cluster commands to start and close cluster nodes.

For example:

*Cluster Test 1-10 start exit ;not needed


Preparing Your ModelCluster Setup in CCS2

Example


Preparing Your ModelOutput/Input files between Multiple Applications 2

Output/Input files between Multiple Applications

Warning: Multiple applications are currently not supported in CCS.

For example, Application 1 might output:

FILEO MATO[3]="{SCENARIO_DIR}\OUTPUT.MAT”

Which inputs to Application 2:

FILEI MATI[1]="{SCENARIO_DIR}\OUTPUT.MAT”

If the Cube Cloud Services Exporter encounters this or a similar case, it will display a warning message:

Warning: The following files require a previous Application to be run first in order to create them


Preparing Your ModelReports and Applier Groups2

Reports and Applier Groups

Warning: CCS does not currently allow uploading models with reports, or with more than one applier group. Uploading a model with reports will cause an error in the uploading process.

Solution

• Use the analysis tools built in to Cube Cloud Services to create charts and maps.

• Delete reports and applier groups before uploading.


Preparing Your ModelSummary 2

Summary

Below is a summary of the “Best Practices” described in this tutorial.

• Use file keys for input files.

• Use unique file names for scenario-specific input files.

• Reference all model run outputs to the {SCENARIO_DIR}.

• Directly link output files in Application Manager instead of reading the file internally.

• Always use double quotes for file paths and names.

• Execute user programs from the {SCENARIO_DIR}.


Preparing Your ModelSummary2



Index

Aapplier groups 16apps folder 2

Ccharts 16cluster

setup 13CT-RAMP 11

Ddirectory structure 2

Ffile tokens 9

Iinput file references 3

fixed 3internal file inputs 3scenario specific 3

input filesbetween multiple applications 15

inputslocation 2

inputs folder 2internal file inputs 4

JJava Parallel Processing Framework 11

Kkeys

file 3, 4

file path 8

Mmaps 16model folder 2models

running 2MS-DOS commands 12multiple applications

output/input files 15

Ooutput file references 6output files

between multiple applications 15outputs

as internal inputs 7outputs folder 2

Rreports 16

Ssystem commands 12

Uuser programs 10

built by model developer 10built-in 10special set-up 11


IndexU


Citilabs, Inc. 1503 Grant Road

Mountain View, California 94040 USA

World Wide Web www.citilabs.com




Citilabs - Atlanta Regional Commission

Documents

Transcript of Citilabs - Atlanta Regional Commission