Pipeline pilot tutorial_beginners21

Beginner’s Tutorial for Pipeline Pilot

Welcome to the Beginner’s Tutorial for Pipeline Pilot. These introductory lessons are intended to help you get the most out of using Pipeline Pilot in the least amount of time. In less than an hour, you should be able to get familiar with the basics of Pipeline Pilot, including how to create and save a new protocol, run a protocol and view the job results, customize a protocol in different ways, and export it as a standalone file.

All example protocols used in these lessons are part of the generic protocols included with Pipeline Pilot (no separately purchased component collections are required). For instructions on using components and protocols that ship with a separately purchased package, see the Pipeline Pilot Help Center.

Contents

Lesson 1

Getting Familiar with Pipeline Pilot Client ...... 2

Lesson 2

Creating and Saving a New Protocol ............. 10

Lesson 3

Finding and Opening a Protocol ..................... 12

Lesson 4

Running a Protocol and Viewing Results ....... 15

Lesson 5

Customizing an Existing Protocol .................. 16

Lesson 6

Exporting a Protocol ....................................... 21

Your Notes ........................................................... 22

Accelrys Technical Support ................................ 22

Copyr ight © 2007, Accelrys Software Inc. Al l r ights reserved.

Beginner 's Tutor i al for P ipel ine Pi lot 2

Lesson 1

Getting Familiar with Pipeline Pilot

Client

Objective: The first beginner’s lesson is intended to help you get familiar with the Pipeline Pilot work environment.

Pipeline Pilot is based on a technology known as data pipelining, which makes it possible to connect different data sources with applications and rapidly process data through a branching network of computational steps. Points of data are processed independently via components, and multiple data points can be processed simultaneously via protocols. Visualization exposes the key data processing steps, making it straightforward to work with data pipelines.

Pipeline Pilot is a Windows-based application that runs on your client computer. It provides a graphical environment for viewing and editing protocols and for sending data to the server for processing.

Basic Concepts

To effectively use Pipeline Pilot, it is important to understand the following basic concepts:

Components

Component types

Component collections

Component reference help

Parameters

Protocols

Ports

Pipelines

Accessibility to files and data

Data sources and destinations

PilotScript

Subprotocols

Components

Components are the basic building blocks of a data pipeline. They process data as it flows throughout a protocol. Components perform specific functions on data including:

Reading data from external files and bringing it into a pipeline

Supplying data to downstream components

Manipulating and filtering data

Calculating data properties

Writing data to an external source and displaying it in an application

Component Types

Components are organized into several broad categories, each represented by a top-level folder, arranged in alphabetical order in the Components tab. These top-level folders relate to the type of operation performed by the components it contains (reading, viewing, filtering, manipulating, and so on). Pipeline Pilot includes a collection of generic components designed to perform tasks such as reading incoming data, filtering, sorting, merging, and sending processed data to output files and viewers. You can customize these generic components and design new ones to suit your particular needs. Customization includes changing what the component does, its label, and its icon.

Component Collections

In addition to the generic components, Pipeline Pilot offers a variety of component collections that you can separately purchase and install on your server. These collections are intended for more specific tasks, such as working with chemistry data, producing reports, partitioning data, and performing text analytics.


Component Reference Help

Help is available for every component in Pipeline Pilot. It is commonly referred to as ―component reference help‖ and is intended to provide very specific and detailed information about a component. Reference help describes its purpose, parameter values, and more.

When you select a component icon in your pipeline, its reference help is displayed in the Help window (lower-left):

Component reference help for File Reader component

Parameters

Most components are equipped with parameters and corresponding values. These parameters control how a component processes data records for handling downstream in a protocol. The Parameters window (lower-right) exposes all the parameters for a selected component. Some parameters are preset and do not need to be changed unless you want a different output. Parameters that require values at run time are displayed in red (and so is the component icon before its required parameters are set).

Parameters for Join Data from File component


Protocols

A protocol is a collection of components, that are connected by pipes, to create one or more pipelines, each designed to perform a specific task. A protocol begins with input and ends with output (generated after the protocol job successfully runs).

The first component is some type of reader that specifies the data source used as input. The middle components are designed to handle your data in one or more ways. Protocols always end with an output component that writes data to an external source that you may view online or use in another application.

The following shows an example of a protocol that checks whether a file exists before reading it and displaying the results in an HTML Table Viewer.

Protocol designed to sort data and display output in Excel

You can view, create modify and run protocols. When you run protocols, you may need to fix errors (debug), especially if you change parameter values at run time. The results are usually available for you to view after the protocol execution is complete.

Note: As an end user, you can create, run, and save your own personal protocols to your ―User Name‖ tab. You can also access shared protocols and change parameter values at run time. Protocol designers can customize components and protocols, and then publish them on your server for use by end users.

Ports

Data flows in and out of components via the data ports. A component can have any combination of ports including one input port and one or two output ports (a green Pass and a red Fail port). Data enters a component through the input port and exits a Pass or Fail port. The number of data records that pass through a port are tracked via numbers of the same color.

Component with Input (left), Pass (green) and Fail (red) ports

The types of ports required for a given component depend on the type of component you need to use in your protocol. You can connect ports to other components for processing somewhere else in the pipeline, turn them on and off, and even define what types of data to use with them if necessary.

Pipelines

When constructing a protocol, you place components in the order that they need to process your data. Each component needs to be connected to the next one with one or more pipes. Data moves from one component to the next through a series of pipes in left-to-right order. Collectively, the series of pipes is known as a pipeline. The pipes connect components via their data ports.

A protocol can include one or more pipelines. Each pipeline can be simple or complex, is numbered sequentially, and is executed from top-to-bottom order (the next pipeline starts when the previous one is complete). You can drag and drop components into the workspace and move them around by adding new pipes and branching data in different directions.


Example of a protocol with two pipelines that get progressively more complex with the addition of components

Accessibility to Files and Data

The Explorer window (upper-left) serves as your starting point for working in Pipeline Pilot. It displays different tabs for accessing components and protocols, plus a ―User Name‖ tab for files that you create for your own personal use.

The Explorer window has different modes for accessing files. In flat mode, files are displayed in a sortable list.

Explorer window in Flat mode

Tip: In hierarchical mode, folders are displayed in an expandable/collapsible hierarchy of folders and subfolders.


A search bar is available above the folder list to make searching for and opening files as convenient as possible. When you enter text in the search bar, the Explorer starts searching for components and protocols that have similar names. When it finds the first item that matches what you type (partial or whole words), the file is highlighted.

To keep searching on the same string of text, press F3. (When the search bar turns a bright orange color, it cannot find a file that matches your search string.)

Explorer window in Hierarchical mode

Note: If you connect to more than one protocol database, a Network tab is also displayed in your Explorer window.

You can drag and drop components and protocols from the Explorer window into the workspace on the right side of the program window. When a component is open in the workspace, it is displayed as an icon. It looks something like this:

Icon for a viewer component

Data Sources and Destinations

Many tasks you perform in Pipeline Pilot involve defining data sources and destinations. You can do this when setting parameters for components such as readers and writers. File browsers are available for helping you perform these tasks.

Reader components require sources. You can select files on the server, another server on your network, or on your client system. On the server, data sources are located in a root folder called ―public‖.

You can also specify which data record properties in a data source you want to use. Any properties you choose to exclude are not read by the component or streamed into the pipeline.

Writer components require destinations as parameter settings. You can only select destinations on the server. By default, the output files are written to your ―User Name‖ tab on the server, so you can easily access the files at a later time.


Subprotocols

Subprotocols encapsulate a number of components into a single top-level component that hides the individual components from view at runtime. This is useful for protocol designers who want to hide the complexity from end users. If you are an end user, you may only see the subprotocol icon and not any of the components it contains.

Protocol with three subprotocols

Starting Pipeline Pilot

When Pipeline Pilot is installed on your system, an icon is available on your Windows desktop for starting the application. It looks like this:

To start Pipeline Pilot:

1. Double-click the icon. When the Pipeline Pilot opens, the work environment is displayed in the main program window.

2. Take a moment to get familiar with all the parts of the window, illustrated below.

Beginner 's Tutor ial for P ipel ine Pi lot 8

Pipeline Pilot client program window


Program Window Description

[ 1 ] Menu Bar: Select commands and perform tasks, such as opening and saving files and customizing the work environment.

[ 2 ] Toolbar: Click shortcut buttons for commonly used functions, such as running a protocol or zooming in and out of the workspace.

[ 3 ] Search Bar: Quickly find and open components and protocols based on the text you enter. Click the button (left-side) to switch between flat and hierarchical modes.

[ 4 ] Explorer Window: Displays all available components and protocols that you can open.

[ 5 ] Explorer Tabs: Decide what you want to display in Explorer window—files in the shared tabs (Protocols, Components) or files from your personal ―User Name‖ tab.

[ 6 ] Network Tab: If you are connected to more than one protocol database, use the Network tab to switch between active servers and change connections.

[ 7 ] Component Reference Help: Depending on what is selected in the workspace (upper-right) or Parameters window (lower-right), the Help window displays help about the selected item (protocol-level help, component-level help, or parameter help).

[ 8 ] Left Windows: The windows in the lower left are provided to help you perform specific tasks. Help is the default window, which displays component reference help (described above). Jobs displays information about the protocols that you ran (from here, you can re-open protocol results for previously run jobs). If you work with checkpoints, you can view information about them in a Checkpoint Data window. If you are debugging or validating a protocol, a Debug Messages window or Validation window is displayed to provide you with the necessary information.

[ 9 ] Favorites Bar: For components and protocols you open frequently, provides a place to both find and open them in a single step to save time.

[ 10 ] Workspace: Your work area for creating, viewing, running, and modifying protocols. Drag and drop files from the Explorer window into the workspace to build new protocols.

[ 11 ] Status Bar: Displays information about the progress of your running protocols.

[ 12 ] Protocol Task Bar: Navigate your open protocols and subprotocols in the workspace by clicking the item displayed in the taskbar.

[ 13 ] Parameter Window: One of the most commonly used features at protocol run time, this window exposes all parameters and their corresponding values. From here, you can set new values and reconfigure existing ones. Required parameters are displayed in red.

[ 14 ] Right Windows: The windows in the lower right are provided to help you perform specific tasks. Parameters is the default window, which displays the parameters for a component selected in the workspace. If you click a blank area of the workspace, the Parameters window displays the protocol-level parameters (if there are any). Other windows are available for implementation parameters, background information about a protocol or component, error handling information, and debugging parameters. Web Service protocols also have a window for viewing parameters and results.

[ 15 ] Active Server Bar: Reveals the name of the active protocol database for running your protocols. To change servers, double-click and select a different server name from the dialog that opens.

Next Lesson: Creating and Saving a New Protocol


Lesson 2

Creating and Saving a New Protocol

Objective: This lesson is intended to help you add new components to a blank workspace to create and save a new protocol. It assumes you already read the first lesson and are familiar with the Pipeline Pilot work environment and the basic concepts of protocol building.

This lesson walks you through the steps of creating a very simple protocol that includes an input and output component.

To start creating a new protocol:

1. Click New Protocol . A new blank workspace opens on the right side of the program window.

2. From the Components tab in the Explorer window, type ―del‖ in the Search bar. The ―Readers‖ folder opens and highlights the first component that matches the search criteria —Delimited Text Reader. This component reads tab-, space-, and comma-delimited text files.

Delimited Text Reader component

Tip: Remember how to use this Search feature. You will need it again for the remaining lessons.

3. Drag and drop the Delimited Text Reader into the workspace. Since this component needs parameter values set before it can run in a protocol, the icon is displayed in red.

Delimited Text Reader icon displayed in red, to indicate that it requires parameter values

4. Select the red icon and look in the Parameters window (lower-right). The Source parameter tells the component what data to use and where it is located. It is displayed in red to alert you that it needs a value.

Delimited Text Reader parameters


5. Select the Source parameter by clicking (right side of parameter name). The Select Sources dialog opens. From here, you can select the data source to use with the reader.

6. There is a quick way to find files that work with this reader. Select the Shortcuts icon on left side of the dialog. The main folders of files you can select on the server are listed.

7. Open the ―Generic Data‖ folder and ―Tables‖ subfolder, choose the ―Selwood.tab‖ file from the list of files, and then click Select.

Select Sources dialog

The Parameters window is updated and reveals the file you selected to use for the Source parameter.

Note that the parameter name is now displayed in black, and the icon in the workspace is displayed in blue, since the required value is now defined for this component.

Source parameters (configured)

The protocol needs a component to receive and do something with the data read by the first component. For this lesson, you can add a simple viewer to the pipeline.

To add a data viewer component:

1. Type ―ecel‖ in the Search bar (note the deliberate typo). The Search bar is unable to find any components that include the phrase ―ecel‖. The text appears against an orange background as a visual clue that you may have entered the wrong spelling.

Search bar is displayed in orange when it cannot locate a search term (possibly due to a typo)


2. Change the search text to ―excel‖ and press F3 repeatedly until the Excel Viewer component is highlighted.

3. Drag and drop the Excel Viewer into the workspace so it is connected to the reader component in the pipeline.

New protocol with Delimited Text Reader and Excel Viewer components

4. Click Save Protocol . The Save Protocol dialog opens.

Save Protocol dialog

5. In Name, type ―Lesson1‖ and click OK. (By default, the new protocol is saved to your ―User Name‖ tab.)

Next Lesson: Finding and Opening a Protocol

Lesson 3

Finding and Opening a Protocol

Objective: The previous lesson showed you how to use the Search bar in the Explorer window to quickly locate components. This lesson shows you how to use the Search bar in different ways to find and open a protocol.

The protocol you need to open is stored in a subfolder. The quickest way to find it is to search for it.

To search for and open the sample protocol:

1. From the Explorer window, select the Protocols tab.

2. Look for the folder ―SciTegic Examples‖. This is where all the example protocols you can run are located.

“SciTegic Examples” folder


3. In the Search bar, type ―sort‖. The Explorer highlights the first item that matches the search criteria.

Results highlighting first example protocol that matches search term

4. Press F3. The Search feature finds the next file that matches your search string. In this example, it highlights the next example protocol directly below the one found in the initial search.

Search results highlighting next protocol example matching search criteria

5. Switch to Flat mode by clicking on the left side of the Search bar. Flat mode displays a list of files that match your search criteria without the folder hierarchy. In some cases, it might be quicker to locate files in this mode.

Flat mode search results list all files without folder hierarchy

Tip: To sort components by collection, click the Author column.

6. Open Sort Data Example. The protocol opens in the workspace.

Sort Data protocol open in workspace


7. The yellow box displayed underneath the pipeline is a sticky note. In addition to what the protocol reference help provides (Help window in lower-left), the sticky note provides extra information to help you better work with the protocol. Take a moment to read the sticky note and get familiar with the protocol details.

8. Click in the blank area of the workspace and then select the Help window (lower-left). This window displays reference help about the protocol. Take a moment to read the help and get more familiar with the protocol.

Help for the Sort Data example protocol

9. Select the Read auto data icon in the workspace and then click the Help window. Help about the selected data reader component is displayed. Take a moment to read this information.

Help for Read Auto Data component

10. To get an idea of what the other components in this example protocol are configured to do, review the reference help for each remaining component.

After you get familiar with the protocol and all of its components, you can run it and view the results. The next lesson shows you how.

Next Lesson: Running a Protocol and Viewing Results


Lesson 4

Running a Protocol and Viewing

Results

Objective: This lesson is intended to help you run a sample protocol and view the resulting data. Once open, the protocol Sort Data Example is ready to run.

To run the protocol:

1. Click Run Protocol . The protocol job is sent to your server for processing. The workspace indicates the number of records processed as the data flows throughout the pipeline. It should look something like this:

Sort Data example protocol running in the workspace. Data counters indicate number of records leaving Pass and Fail ports (enclosed in red boxes for this tutorial

illustration).

After the protocol runs, HTML reports are generated your browser. The output should look something like this:

Sort Data example output

Note: There are two reports in the output because there were two pipes for output in the protocol. One report sorts the data by horsepower, and the other by price.


Whenever you run a protocol, your client issues a ―job‖ to the server. Pipeline Pilot includes a Jobs window (lower-left) that tracks the protocols that you run and provides job details, including the results of a job (even after it is completed). You can easily retrieve the results of jobs you ran previously from this window, without having to re-run the job. This is especially useful if you run any protocols that take considerable time to process. Having a quick and easy way to access the results saves you time.

To view protocol job results:

1. Click the Jobs window.

2. Expand the job listing by clicking . The result files are displayed as links. Clicking a link opens the result file, in this case, it opens the .htm report files in a browser. The ―Finished‖ status indicates that the job was successful.

Jobs tab

Next Lesson: Customizing an Existing Protocol

Lesson 5

Customizing an Existing Protocol

Objective: This lesson shows you how to customize an existing protocol by getting familiar with the data properties used with a data source, changing parameter settings that manipulate the data, and finally, replacing the components that generate the output for the protocol.

To select data properties for a source file:

1. Open the protocol Sort Data Example. If you just completed the previous lesson, this protocol should be open in the workspace.

2. Select the first component icon, Read auto data.

3. From the Parameters tab, open the Source parameter group by clicking . The parameters contained in this group are exposed (Maximum, SourceTag, Keep Properties, and HasColumnNames).

Source parameter group for Read auto data component

4. Make a note of the file used as the data source, ―data\Tables\imports-85.txt‖. This is a simple text file that contains one record per line with numerous data properties for various automobiles.


When selecting data sources for reader components, you can find out which data properties the records contain to get an idea of what you can do with the data in downstream components. You can also include and exclude data properties. Excluded properties are not used by the component when it reads the data source.

5. Select the Keep Properties parameter by clicking . The Source Preview dialog opens. By default, this dialog shows properties only for the first five data records.

6. Change the number of data records to preview to ―100‖, and then click Reload. The preview list is updated to display 100 records.

Data properties available in the data source “imports-85.txt”

7. Scroll down the list and note all the makes of automobiles listed for 100 data records. (Expand the width of the ―make‖ column by dragging the cell border to the right.)

8. Scroll across the source properties and make a note of all the ones that are available for this data source (―imports-85.txt‖). There are numerous properties. When specifying the properties to use in your data records, you can discard any unnecessary ones from being read into the pipeline from this dialog.

9. Select the column headers for ―losses‖ then click (Discard). The dialog is updated and the losses properties are moved to the Discarded list on the right side.

10. Discard the ―fuel‖ property. The dialog should look something like this, with the two discarded data properties:

Excluded data properties “losses” and “fuel”

11. Click OK to close the dialog. The Parameters tab is updated to show the properties that are included for the Keep Properties parameter. (When all properties are included, the field is blank.)


Keep Properties exposes the data properties that are used for “imports-85.txt”

Now that you’ve had a chance to get familiar with the data properties available in the data source, you can get a better idea of how they are used by other components in a protocol.

To select data properties in a downstream component:

1. From the workspace, select the second component in the pipeline, Keep only a few properties. It keeps only those properties you specify for each incoming data record, which is another way to include or exclude properties after the data is read into the pipeline.

2. From the Parameters tab, select the Expression parameter for this component by clicking . An expression editor opens.

3. Make a note of all the property names that are used in the string. They match the names displayed in the Source Preview dialog. This expression tells the component which properties to keep and which ones to exclude.

String that specifies the properties to keep

4. For this lesson, you can leave the PilotScript expression intact, but keep in mind how you access the expression editor. (In later lessons, you can learn more about how to edit expressions and work with PilotScript functions.)

5. Click Cancel to close the editor.

The existing protocol sorts data by horsepower and by price. For this lesson, you can change this sort criteria.


To change parameter values for sorting:

1. Select the component Sort Data by horsepower and change the following parameters:

Parameter Value

Property 1 ―Make‖

Sort Order ―Ascending‖

Sort As ―Text‖

The Parameters window should look something like this when the parameter values are updated:

Parameters for Sort Data component

2. Click the label for this component in the workspace and change the text to ―Sort Data by Make.‖ (What you type should replace what is highlighted in the label.)

3. Customize the Sort Data by price component in the lower branch so it sorts data by ―numdoors‖ in ascending order, as text. Rename the label ―Sort Data by NumDoors‖.

4. Remove both Table components, Tile Horizontal, and the HTML Report Viewer components from the pipeline. (Press and hold CTRL as you click each icon and then press DELETE.)

5. Attach Excel Viewer components to the end of each Sort Data component.

Your protocol should look something like this:

Excel Viewers attached to Sort Data components

6. For the top Excel Viewer component, set the WorkSheetName value to ―Sorted by Make‖. In the Excel output, the worksheet tab will display this label.

Parameters for Excel Viewer component

Tip: Click Show/Hide Sticky Notes to hide the sticky note (so you can more easily view all the component icons and their labels).


7. Customize the bottom Excel Viewer component so it generates a worksheet named ―Sorted by Number of Doors‖.

Parameters for Sort Data component in second branch

Note: When the data records exit the second component in the pipeline, these sort components sort the data by make and number of doors, rather than by horsepower and price. The viewers will reflect this when the output is generated.

8. Save this protocol to your ―UserName‖ tab as ―CustomProtocol‖ (so you can use it again in the next lesson).

9. Run the protocol to view the results. The output for the first worksheet ―Sorted by Number of Doors‖ should look something like this:

Protocol output for customized Sort Data example

Next Lesson: Exporting a Protocol


Lesson 6

Exporting a Protocol

Objective: This lesson shows you how to save a protocol in a different way by exporting it as an XML file. You can export the protocol you created in the previous lesson (―CustomProtocol‖). This is useful for creating extra copies of your files, so you can access them from different client systems, such as your laptop computer.

Note: The files you export are saved in XML format. To open an exported file, you need to import it to your ―User Name‖ tab on the client where you are doing your work.

When you save a protocol to your ―UserName‖ tab, the file is saved in the protocol database where all other components and protocols reside. This database is located on your server.

You can also save copies of the protocol to other locations such as your local hard drive or a CD, by exporting the protocol.

To export a protocol:

1. From your ―User Name‖ tab in the Explorer window, search for ―CustomProtocol‖.

2. Right-click the file and select Export.

Exporting a protocol in the Explorer window

3. The Export dialog opens. In Save In, navigate to the drive and folder where you want to save the XML file.

Tip: Click Create New Folder to export the XML file in a new directory.

Exporting a protocol as an XML file to a local drive

4. By default, the XML file is saved with the same name as the original file, in this case, ―CustomProtocol‖. Feel free to change the name or save it as is.

5. Click Save. Your protocol is saved to the target location as an XML file.


Tip: This lesson demonstrates how to export a single file, but you can also export multiple files and even entire folders. When exporting multiple files, press and hold CTRL and click each filename, then right-click and select Export to Folder. When exporting an entire folder, right-click the folder and then select Export Folder.

If you’ve made it this far, you have completed all the lessons in the Beginner’s Tutorial. You now may have a basic understanding of how Pipeline Pilot works. Congratulations!

Next Lesson: Advanced Tutorial

To learn more tasks and get familiar with PilotScript, continue with the Advanced Lessons.

Your Notes

Use the extra space on this last page to write down any notes you have about the tutorials and what you learned.

Accelrys Technical Support

If you have any problems or questions, contact Accelrys Technical Support:

Pipeline Pilot Enterprise Server and Professional Client

EMAIL: [email protected]

ONLINE SUPPORT: http://accelrys.com/services/support.html

Phone Support

AMERICAS: M–F 6:00 AM – 5:30 PM (PST) 1-800-756-4674 (TOLL FREE)

1-858-799-5509

EUROPE: M-F 9:00 AM – 6:00 PM (CET)

+44 1223 228822 (OUTSIDE UK)

08457 413375 (UK)

mailto:[email protected]

http://accelrys.com/services/support.html

Pipeline pilot tutorial_beginners21

Documents

Transcript of Pipeline pilot tutorial_beginners21