EMC InfoArchive 4.1 Command Line Tool Guide · TableofContents RevisionHistory...

EMC® InfoArchiveVersion 4.1

Command Line Tool Guide

EMC CorporationCorporate Headquarters

Hopkinton, MA 01748-91031-508-435-1000www.EMC.com

Legal Notice

Copyright © 2016 EMC Corporation. All Rights Reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to changewithout notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATIONMAKES NO REPRESENTATIONSOR WARRANTIES OF ANY KINDWITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLYDISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. Adobe and Adobe PDFLibrary are trademarks or registered trademarks of Adobe Systems Inc. in the U.S. and other countries. All other trademarksused herein are the property of their respective owners.

Documentation Feedback

Your opinion matters. We want to hear from you regarding our product documentation. If you have feedbackabout how we can make our documentation better or easier to use, please send us your feedback directly [email protected]

Table of Contents

Revision History ................................................................................................................... 7

Chapter 1 Overview ...................................................................................................... 9Starting IAShell ................................................................................................. 9Connecting to the InfoArchive Server by HTTPS.................................................. 9Creating Certificates and Trust Establishment ................................................ 10One-Way TLS ........................................................................................... 10Two-Way TLS........................................................................................... 10

Connecting to InfoArchive ................................................................................ 11IAShell Startup Scripts ...................................................................................... 12Pre-defined Users and Authorization ................................................................. 12Type Aliases..................................................................................................... 13Admin CLI Authentication via Real External LDAP Server.................................. 13Using the Post-Recovery Command to List Orphaned Objects ............................ 15Receiving Legacy Files ...................................................................................... 15General Tips for IAShell.................................................................................... 17Troubleshooting ............................................................................................... 17

Chapter 2 Configuration .............................................................................................. 19IAShell Command Specifications ....................................................................... 19Using the apply-hold Command........................................................................ 34Verifying the Hold Set has been Created ........................................................ 37

Using the configure Command .......................................................................... 38Format of a Data File for the Configure Command.............................................. 42Issues in the Update/Create Sections of Data Files........................................... 43Appending and Replacing Elements in Collections ........................................ 44Setting InfoArchive Clients for HTTPS Communication .................................. 46SSL1 System Properties ............................................................................. 46SSL2 System Properties ............................................................................. 47InfoArchive JDBC Connection Setup.......................................................... 47IAShell..................................................................................................... 48

Content Uploading ....................................................................................... 48InfoArchive Server and Gateway/InfoArchive Web ApplicationCommunication Setup .................................................................................. 48Self-Signed Certificates-Based Setup .......................................................... 50

Configuring a Sample Application ..................................................................... 56Creating an IAShell Start-Up Script.................................................................... 56Performing a Byte Count on Application Data .................................................... 57Usage Examples ........................................................................................... 57

Setting Up CMD to Display National Characters ................................................ 58Configuring xDB Ingestion Mode ...................................................................... 59

3

Table of Contents

Working with the application.yml File ............................................................... 59Security Profile ............................................................................................. 59logging.level ................................................................................................ 60infoarchive Section ....................................................................................... 61managedItemData Section............................................................................. 61defaultNames Section ................................................................................... 62Storage Paths and OAIS Sections ................................................................... 62job Section.................................................................................................... 62roles Section ................................................................................................. 63Example....................................................................................................... 63Updating the Working Directory ................................................................... 65Creating a File System Root....................................................................... 65

Chapter 3 Configuring a SIP Application ..................................................................... 67Listing AIPs ..................................................................................................... 67Ingesting a SIP.................................................................................................. 68Building a SIP ZIP ............................................................................................ 69Configuring a SIP Application with One Holding ............................................... 69Search.......................................................................................................... 75PDI.INDEX.CREATOR.................................................................................. 80PDI.AIU.CNT............................................................................................... 82PDI.AIU.ID .................................................................................................. 82PDI.PKEYS................................................................................................... 83RI.INIT ........................................................................................................ 84XDB.PDI.CI.ID ............................................................................................. 85

Configuration Objects ...................................................................................... 85AIP.............................................................................................................. 87AIC ........................................................................................................... 102Application ................................................................................................ 104Confirmation.............................................................................................. 105Creating a Confirmation with an ANT Script............................................ 106

Content...................................................................................................... 107Delivery Channel........................................................................................ 109File System Folder ..................................................................................... 111File System Root ........................................................................................ 111Holding ..................................................................................................... 112Ingest ........................................................................................................ 121Ingest Node (ia_cfg_ingest_node) ................................................................ 125PDI (ia_cfg_pdi) ......................................................................................... 126PdiSchema (ia_cfg_schema)......................................................................... 129Query (eas_cfg_query) ................................................................................ 131Query Quota (ia_cfg_query_quota) .............................................................. 136Receiver Node (ia_cfg_receive_node) ........................................................... 137Search........................................................................................................ 139SearchComposition..................................................................................... 141Creating a searchComposition with an ANT Script ................................... 142

XForm ....................................................................................................... 142Creating an XForm from an ANT Script ................................................... 143

ResultMaster .............................................................................................. 145Creating a ResultMaster with an ANT Script ............................................ 148

ResultConfigurationHelper ......................................................................... 152Space ......................................................................................................... 155Space Root Folder ...................................................................................... 155SpaceRootXdbLibrary ................................................................................. 156Space Root xDB Library .............................................................................. 156FileSystemRoot........................................................................................... 157FileSystemFolder ........................................................................................ 157

4

Table of Contents

Store .......................................................................................................... 158Tenant ....................................................................................................... 159Transformation........................................................................................... 160XdbDatabase .............................................................................................. 161XdbFederation............................................................................................ 161xDB Library .............................................................................................. 162Creating an xDB Library with an ANT Script............................................ 164

XdbLibraryPolicy ....................................................................................... 165Content...................................................................................................... 166SIP Generator ............................................................................................. 168Building and Installing ........................................................................... 168Scenario – Creating a SIP Using Data from MySql..................................... 169

Space Limitation for a SIP File Name ............................................................... 170Unitary Archiving and Aggregation ................................................................ 171Receiving and Ingesting a SIP in One Request .............................................. 172Configuring the Ingestion Process to Store Multiple AIPs in theSame xDB Library....................................................................................... 173Executing the Close Job to Close the xDB Library and Perform aBackup ...................................................................................................... 174Configuring the Ingestion Process to Allow Aggregation .............................. 174Reception............................................................................................... 175

InfoArchive Components Debug Level Configuration ....................................... 175For Windows ............................................................................................. 175For Linux ................................................................................................... 176Enabling Debug Log Level ......................................................................... 176

Chapter 4 Configuring a Table Application ................................................................ 177Using the Upload Command ........................................................................... 178Usage Examples ......................................................................................... 179Using the Upload Schema Command........................................................... 180Usage Examples ..................................................................................... 180

run-chain-of-custody .................................................................................. 180

Chapter 5 Search Composition .................................................................................. 183Searching Table Data ...................................................................................... 183Using a Common Custom Library in an XQuery for Table-basedSearches..................................................................................................... 183

Automatic Search Installation for Application with ANT................................... 183Application Searches Structure .................................................................... 184Searches Folder .......................................................................................... 184Search Properties File.............................................................................. 185Search Composition................................................................................ 185Search Installation .................................................................................. 185Creating a Search Form ........................................................................... 186

Configuring the Number of Items Listed in the Search Results........................... 187Configuring the Time Limit for a Background Search........................................ 188

Chapter 6 Compliance ............................................................................................... 189Using Audits for Compliance .......................................................................... 189Metadata Fields .......................................................................................... 189Audit Event Type ................................................................................... 190Application-Specific Audits..................................................................... 190Metadata Fields .................................................................................. 191Searching for Audits ........................................................................... 191Audit Entries...................................................................................... 191

5

Table of Contents

SIP-Based Applications and Retention Class..................................................... 196Checking an AIP Object’s State ........................................................................ 197Configuring a Custom Retention Policy............................................................ 197Setting a Default Retention Policy .................................................................... 198Invalidating an AIP ........................................................................................ 198Removing a Retention Policy and Deleting an AIP ............................................ 199

6

Revision History

Revision Date Description

September 2016 Initial publication.

7

Revision History

8

Chapter 1Overview

InfoArchive includes a tool called the InfoArchive Command Line Interface (CLI), which can beaccessed from the Tools folder. The CLI, also called IAShell, is typically used by administrators anddevelopers. The tool can be found in the InfoArchive installation folder by this relative path:

• For Windows: tools > iashell.bat

• For Linux: tools > iashell

IAShell will mainly be used by InfoArchive Administrators. For more information, see the User Rolessection in the InfoArchive Configuration and Administration User Guide.

The tool is also used in InfoArchive scripts within sample applications.

Starting IAShellTo start IAShell, invoke it from the command line, as illustrated below:

[/ia40] % tools/iashellInfoArchive 4.1.0 command line client (c) 1999-2016 EMC CorporationType 'help' for a list of commands and options, type 'exit' to leave the shell.ia-shell>

Connecting to the InfoArchive Server by HTTPSBefore connecting to your InfoArchive server instance, which is running in HTTPS mode, performthe following two commands, which specify the correct places for your client-side Transport LayerSecurity (TLS) certificates:

ssl-keystore --file "<your_folder>/keystore.p12>" --password --password abcdefg --type PKCS12

ssl-trust-store --file "<your_folder>/truststore.jks" --password abcdefg --type JKS

9

Overview

Creating Certificates and Trust Establishment

The following sections provide examples of how to implement one-way and two-way TLS.

One-Way TLS

When implementing one-way TLS, the server authenticates itself to clients.

To create self signed server certificate:

keytool -genkey -noprompt -trustcacerts -keyalg RSA -alias IA_SERVER_CERT -dname"CN=localhost, OU=JavaSoft, O=Sun, L=Cupertino, S=California, C=US" -keypass abcdefg-storetype PKCS12 -keystore serverCertStore.p12 -storepass abcdefg

To display server certificates in a keystore:

keytool -list -v -keystore serverCertStore.p12 -storepass abcdefg -storetype PKCS12

To create a public server certificate for other applications:keytool -export -noprompt -rfc -alias IA_SERVER_CERT -fileserver_public_cert.cert -keystore serverCertStore.p12 -storepass abcdefg -storetype PKCS12

To import a certificate into a client keystore:keytool -import -noprompt -trustcacerts -alias server_public_cert-file server_public_cert.cert -keystore client_truststore.jks -storepass abcdefg

Add the following properties to the server’s application.yml file:

server:ssl:key-store: "serverCertStore.p12"key-store-password: abcdefgkey-password: abcdefgkey-store-type: PKCS12key-alias: ia_server_cert

Two-Way TLS

When implementing two-way TLS, the server and client authenticate themselves to each other.

To create a self-signed client certificate:

keytool -genkey -noprompt -trustcacerts -keyalg RSA -alias IA_CLIENT_CERT-keypass abcdefg -storetype PKCS12 -keystore clientCertStore.p12 -storepass abcdefg

To display client certificates in a keystore:

keytool -list -v -keystore clientCertStore.p12 -storepass abcdefg -storetype PKCS12

To create a public client certificate for other applications:

keytool -export -noprompt -rfc -alias IA_CLIENT_CERT -file client_public_cert.cert-keystore clientCertStore.p12 -storepass abcdefg -storetype PKCS12

10

Overview

To import a certificate into a server keystore:

keytool -import -noprompt -trustcacerts -alias client_public_cert -fileclient_public_cert.cert -keystore server_truststore.jks -storepass abcdefg

Add the following properties to the server’s application.yml file:

server:ssl:key-store: "serverCertStore.p12"key-store-password: abcdefgkey-password: abcdefgkey-store-type: PKCS12key-alias: ia_server_certtrust-store: "server_truststore.jks"trust-store-password: abcdefgtrust-store-type: JKSclient-auth: need

Connecting to InfoArchiveUnless you connect to a running InfoArchive instance, there are not a lot of actions you can performin IAShell. For instance, if you try listing the available tenants with the select command, thefollowing error message is issued:

ia-shell>select tenantCommand failed java.lang.IllegalArgumentException: Unknown type alias: tenantia-shell>

The list command typically displays the available types you can work with but, since you have notconnected to a running InfoArchive instance, the list command returns nothing:ia-shell>listia-shell>

To connect to a running InfoArchive instance, you need to know:

1. The endpoint of the InfoArchive REST services, for example:

http://localhost:8080/services

2. The endpoint of the gateway, for example:http://localhost:8080

3. The username of the user you are connecting as (i.e., [email protected].

4. The password of the username. For example, if the password for [email protected] is’password’:

ia-shell>connect http://localhost:8080/services --user [email protected] --pswpassword --gateway http://localhost:8080Connected to 'http://localhost:8080/services'ia-shell>

11

Overview

To learn more about the connect command and the available parameters, type help connect .

IAShell Startup ScriptsIAShell will run any scripts placed in $HOME/.ia-shell at startup. For instance, the connectcommand in a startup script will avoid having to constantly reconnect to the system.

For Windows,

1. Create a file called startup.txt in $HOME/.ia-shell.

2. In startup.sh , add the following command:

connect http://localhost:8080/services --user [email protected] --psw password--gateway http://localhost:8080

3. Quit IAShell by executing the exit command.

ia-shell>exit[/ia40]%

4. Start IAShell again. You are now automatically connected to the running InfoArchive instance.One method to determine whether you are connected or not is by listing the available types.

Pre-defined Users and AuthorizationInfoArchive includes pre-defined users, such as Adam, the Administrator. For more information, seethe Inline Users section in the InfoArchive Web Application User Guide.

These inline users have non-expiring authorization tokens and should only be used for demonstrationpurposes only.

You can, however, use such a token to skip user- and password-based authentication to switch to asimplified approach. Use the set-header command before running the connect command:

set-header --name Authorization --value “<SOME_TOKEN_VALUE>"

This eliminates the necessity of having a running InfoArchive server when, for example,demonstrating the application to the users. The tokens are long, however. It is recommended thatconnection-related commands are placed in a script.

IAShell supports the ‘.rc’-like type of initialization by reading-in all files found in the $HOME >.ia-shell > directory, including any start-up scripts. IAShell then executes these commands uponstart-up, which makes it easy to pre-configure certain settings (i.e., use the connect and usecommands for certain applications and tenants, etc.).

For example, use the connect and use commands for certain applications and tenants:

12

Overview

connect http://localhost:8080/servicesuse --tenant test2 --application firstApp

To create this file on Windows, navigate to your home directory C:\Users\<accountName> andenter the mkdir command:

mkdir .ia-shell

Windows does allow you to create a directory with this name via the InfoArchive web application.

Type AliasesA type alias is a symbolic representation of a resource on an InfoArchive server that can be usedby an Administrator.

Aliases can be discovered after the customer specifies a parent object. After the connection isestablished by using the connect command, there are at least two main type aliases: federation andtenant. Other aliases (i.e., application-aware) are discovered after specifying a specific application(by the use command). Some aliases will not be available for certain users because of the securityrules governing user roles and groups.

InfoArchive includes support for searching/filtering for the configuration objects by SpringExpression Language (SpEL) expressions.

The only exception for the AIP type alias is that it does not support SpEL in the select command.Customers should instead use the filter option. AIP filtering parameters include:

• holdingName

• phaseCode

• returnCode

• receiveStartDateFrom

• receiveStartDateTo

Customers should use the ISO DateTime Format (yyyy-MM-dd’T’HH:mm:ss.SSSZ, i.e.,’2000-10-31 01:30:00.000-05:00’) when specifying values for the receiveStartDateFrom andreceiveStartDateTo parameters.

Admin CLI Authentication via Real ExternalLDAP ServerThe following steps outline the process to configure the Admin CLI to work with an externalLightweight Directory Access Protocol (LDAP) server.

1. Download the latest InfoArchive distribution.

2. Open the infoarchive > config > webapp > application.properties file.

13

Overview

3. Find the “spring.profiles.active=infoarchive.gateway.profile.AUTHENTICATION_IN_MEMORY” string.

4. Set the string to “spring.profiles.active=infoarchive.gateway.profile.AUTHENTICATION_IN_MEMORY,infoarchive.gateway.profile.AUTHENTICATION_ACTIVE_DIRECTORY”(or spring.profiles.active=infoarchive.gateway.profile.AUTHENTICATION_IN_MEMORY,infoarchive.gateway.profile.AUTHENTICATION_EXTERNAL_LDAP if you useexternal LDAP).

5. Configure application-infoarchive.gateway.profile.AUTHENTICATION_ACTIVE_DIRECTORY.properties file with active directory connection settings.

6. Configure the Active Directory in application-infoarchive.gateway.profile.AUTHENTICATION_ACTIVE_DIRECTORY.properties file (or application-infoarchive.gateway.profile.AUTHENTICATION_EXTERNAL_LDAP.properties if you use External LDAP).Example for Active DirectoryAUTHENTICATION_ACTIVE_DIRECTORY.managerDn=cn\=Administrator,cn\=users,dc\=iigads,dc\=comAUTHENTICATION_ACTIVE_DIRECTORY.managerPassword=Password@123AUTHENTICATION_ACTIVE_DIRECTORY.userDnPatterns=cn={0},ou\=Users,ou\=infoarchive,dc\=iigads,-dc\=comAUTHENTICATION_ACTIVE_DIRECTORY.userSearchFilter=sAMAccountName\={0}AUTHENTICATION_ACTIVE_DIRECTORY.userSearchBase=ou\=Users,ou\=infoarchive,dc\=iigads,dc\=comAUTHENTICATION_ACTIVE_DIRECTORY.groupSearchFilter=(member\={0})AUTHENTICATION_ACTIVE_DIRECTORY.groupSearchBase=ou\=Groups,ou\=infoarchive,dc\=iigads,dc\=comAUTHENTICATION_ACTIVE_DIRECTORY.url=ldap://10.31.70.140:389/

Example for External LDAPAUTHENTICATION_EXTERNAL_LDAP.userDnPatterns=uid={0},ou\=peopleAUTHENTICATION_EXTERNAL_LDAP.userSearchFilter=uid={0}AUTHENTICATION_EXTERNAL_LDAP.userSearchBase=ou\=peopleAUTHENTICATION_EXTERNAL_LDAP.groupSearchFilter=(member\={0})AUTHENTICATION_EXTERNAL_LDAP.groupSearchBase=ou\=groupsAUTHENTICATION_EXTERNAL_LDAP.url=ldap://localhost:8080/dc\=infoarchive,dc\=emc,dc\=com

7. Install InfoArchive.

8. Start xDB and server.

9. Start the web application and login as an Administrator ([email protected]/password).

10. Navigate to Administration > Groups.

11. Configure the group-role mapping for Active Directory groups.

12. Set spring.profiles.active=infoarchive.gateway.profile.AUTHENTICATION_ACTIVE_DIRECTORY in application.properties file (or spring.profiles.active= infoarchive.gateway.profile.AUTHENTICATION_EXTERNAL_LDAP if you use external LDAP).

13. Restart the InfoArchive web application.

14. Navigate to the Tools folder and start iashell.

15. Connect to server using ActiveDirectory creditinals:connect http://localhost:8080/services --gateway http://localhost:8080 --user adam --pswUGFzc3dvcmRAMTIz.

14

Overview

Using the Post-Recovery Command to ListOrphaned ObjectsIn the event a customer must perform recovery tasks, an Administrator can use the following CLIcommand to list objects that were orphaned.

The URL can be resolved starting from http://localhost:8080/services using thehttp://identifiers.emc.com/recovery/orphaned-objects relation.

Receiving Legacy FilesInfoArchive allows a customer to ingest SIP data that contains sip.xml with a legacy (3.X) schema.

To receive a legacy file, use the following to change the reception format:eas_sip_zip

The reception node of the holding, must provide the legacy sip extractor named’LegacyZipSipExtractor’:

15

http://localhost:8765/services

http://identifiers.emc.com/recovery/orphaned-objects

Overview

The accepted format for ingestion must be modified to support the legacy format. Add the followingto the holding.xml file:

16

Overview

Using the Admin CLI, change the format in the build.xml file:

General Tips for IAShell• Use auto-completing for commands, command parameters and type aliases that you have alreadydiscovered (use the TAB key).

• To browse the commands history, use the and ¯ keys. The history only records commandsthat were successfully completed.

• You can pre-configure IAShell by creating the start-up scripts.

TroubleshootingWhen the InfoArchive web server is used in conjunction with more than one IA server and xDBserver to run a multi-user search test, the results are not being returned in a sufficient amount of time.However, the log displays all the searches executed in a very short time.

Workaround: The application.yml file should be updated with the following values:

maxTotalConnections: 200maxPerRouteConnections: 20

Once the values have been updated, there are no issues with a one InfoArchive web application, twoInfoArchive server deployment. If there are more InfoArchive servers with one InfoArchive webapplication server, however, the values should be increased.

17

Overview

18

Chapter 2Configuration

IAShell Command SpecificationsEach command that interacts with the back-end can take some relations to the account (the IAShellcontext).

Note: All the commands and their respective options are case sensitive.

Command Arguments and Options

actions-update Options:

• --name/role <name> updates actions for specified role name

• --actions <set of actions> a set of comma-separated actions

• --from <file path> file path to a file with set of actions

Available only after the roles-view command has performed.

actions-view Options:

• --name or --role<name> view actions by specified role name

• --output <path to a file> the path to dump the output to

Available only after the roles-view command has performed.

apply-hold Options:

• --id or --last <id> of a hold object or <index> of such id in the latestselect result set performed

• --from <value> path to a file with data (in XML format)

Applies a hold on an object.

The data file has an XML format and can have a lot of elements,as required by the REST API. Optionally, you can also specify aholdSetDescription to describe why the items in the list wereput under hold.

Example of such a file:

<?xml version="1.0"?>

<data>

<itemsToProtect>

21492d36-d563-4987-9893-bc8c35008f15

19

Configuration


</itemsToProtect>

<itemsToProtect/>

<holdSetName>MyHoldSet</holdSetName>

<type>AIP</type>

<application>#{application}</application>

</data>

The following is an example where more than one item is protected:

<?xml version="1.0"?>

<data>

<itemsToProtect>e0ae4248-6c30-4a59-8303-95b84dc28ff8<

/itemsToProtect>

<itemsToProtect>ee4d9bee-b4b3-430a-bf1b-4359d74d828e<

/itemsToProtect>

<itemsToProtect/>

<holdSetName>My HoldSet 2</holdSetName>

<type>AIP</type>

<application>#{application}</application>

</data>

Example of the command:

apply-hold --id 1 --from xml/apply-hold-example.xml

The command provides the application information and does notneed to pass the parameters. The application field can be omitted inthe data file, as the proper value can be retrieved from the context(current application value).

apply-retention Options:

• --id or --last <id> of a retention-policy object or <index> of such idin the latest select result set performed


Applies a retention policy to an object.

The data file has an XML format and can have a lot of elements, asrequired by the REST API. With the exception of the root element, allthe elements inside a file will be transformed to JSON format. The roottag must be named ’data’ and will be not transformed to JSON. Theroot tag must be named ’data’ and will be not transformed to JSON.

Example of such a file:

<data>

<itemsToProtect>

<itemToProtect>80368fd6-ca94-431f-9098-538240d554d2<

/itemToProtect>

</itemsToProtect>

<itemsToProtect/>

<retainedSetName>MySecondRetainedSet2</retainedSetName>

20

Configuration

Command Arguments and Options<baseDate>2014-07-14</baseDate>

<ageIndividually>false</ageIndividually>

<application>http://localhost:8080/configuration

/applications/5104face-c332-4136-972d-1cae80f570a5<

/application>

</data>

Example of the command:

apply-retention --id 1 --from xml/apply-retention

-example.xml

The command provides the application information and does notneed to pass the parameters. The application field can be omitted inthe data file, as the proper value can be retrieved from the context(current application value).

audit-disable Options:

• --id or --last <id> of an audit event type object or <index> of suchid in the latest select result set performed.

There are three type aliases related to this command:

• audit-event-type (system wide),

• tenant-audit-event-type (related to a tenant specified in the usecommand), and

• app-audit-event-type (related to an application specified in the usecommand).

Example:

audit-disable --id 1

audit-enable Options:

• --id or --last <id> of an audit event type object or <index> of suchid in the latest select result set performed.

There are three type aliases related to this command:

• audit-event-type (system wide),

• tenant-audit-event-type (related to a tenant specified in the usecommand), and

• app-audit-event-type (related to an application specified in the usecommand)

Example:

select --t audit-event-type audit-enable --id 1

21

Configuration


close-library Options:

• --id or --last <id> of a xdb-library object or <index> of such id in thelatest select result set performed

• --rel <name of relation> specify a relation to a xdb-library whichis taken from the context

Either ID or relation should be provided for this command. Thecommand send a request to close the library. Actual closing will beperformed when the close job runs next time.

configure Argument: <type alias>

Options:

• --t or --type <type alias> it’s required only if a data file (see nextoption) doesn’t specify it

• --from <file path> file path to a configuration object data inXML-based format

• --var-set <context variable name> set a context variable with givenname to the self link value of the latest object configured

Create/update a configuration object with specified type. Because theconfiguration data contains a lot of properties, it is easy to specify allof the data in a file. For more information, see Format of the DataFile for the Configure Command.

Examples:

configure holding --from phonecalls.xml

configure holding --from phonecalls.xml --var-set

myObjectSelf

connect Argument: Set base URI for REST API of the back-end part

Options:

• --user <value> a user name needed to be connected to REST API

• --psw <value> a password needed to be connected to REST API

• --gateway <value> a gateway server URI that used for OAuth2authentication

Examples:

connect http://localhost:8080/services

connect http://localhost:8080/services --user

[email protected] --psw password

connect http://localhost:8080/services --gateway

22

Configuration

Command Arguments and Optionshttp://localhost:8080 --user [email protected] --psw

password

content-upload Argument: <type alias>

Options:

• --t or --type <type alias>

• --id or --last (some id) or <index> in the latest search result setshown

• --storeName <value> name of a store to handle the content

• --storeId <value> ID value of a store to handle the content

• --format <value> format of the content

• --from <value> path to a file with content

Examples:

content-upload query --id "b65a6e15-351a-4046-a786

-aadd10a02221" --storeName myFileStore --format xml

--from "/tmp/content/query.xml"

data-delete Options:

• --id or --last <id> of an application object or <index> of such id inthe latest select result set performed.

If an application contains data, this command deletes the data.Example:

data-delete --id 1

db-index-job Options:

• --id or --last <id> of a database object or <index> of such id inthe latest select result set performed. If this is specified the nameoptions will be ignored

• --name <value> name of a database object. If this is not specified,the database name from the use command will be used

• --job <value> name of the indexing job

• --run if this option is specified it means that the job will be started,it this is omitted the status will be shown

The list of the indexing job: Can be started: index-build, index-rebuild,index-remove, index-merge :

Can be queried for status: index-status, index-merge-status

23

Configuration


Examples:

db-index-job --job index-status --name Tickets-sql-db

db-index-job --job index-status --id 1 db-index-job

--job index-build --id 1 --run

delete Argument: <type alias>

Options:



Examples:

delete aip --id "3232-23232-32223"

delete --id 2

delete --last 1

disconnect No arguments and options

Clears all the context data, including type aliases and does logout, ifany.

enumerate Options:

• --id or --last <id> of an ingest-node object or <index> of such id inthe latest select result set shown

• --cutoff <value> period in days.

• --minusRunning indicates the max parameter should be decreasedfor the AIPs are ingesting currently.

• --max <value> maximum number of AIPs returned.

• --list if it’s specified, output AIP ID values to a list(comma-separated)

Enumerates list of AIPs are ready to be ingested for certain ingest-nodeobject. Produces the Id indexes (if some results are found) to be storedin the context.

24

Configuration


file-upload Options:

• --from <value> path to a file

• --rel <name of relation> specify a relation which is taken from thecontext

This is generic uploading command. Could be used in cases when auploading relation is known. The relation specified will be used forthe uploading the file.

groups-view View the groups

import-search Options:

• --from <value> path(s) to one or more search files delimited by ’;’ or’:’ (the system-dependent path-separator character)

• --overwrite <value>, specifies if to update search if it exists beforeinstallation. Possible values: "true" or "false". If there is noparameter, then standard import is performed

The command allows to import the search from ZIP file (search formatexported from the InfoArchive web application) to the application.For example:

import-search --from C:/Tools/applications/PhoneCalls

/DateInterval.zip --overwrite true

--overwrite parameter for "import-search" command

• If there is no parameter, then standard import is performed

• if there is an --overwrite true parameter and, if the searchexists, then the existing search is updated.

• if there is an --overwrite false parameter and, if the searchexists, then the existing search is not updated.

ingest Options:

• --id or <id> of an AIP object or <index> of such id in the latestenumerate result set shown

• --all-ids specifies to apply the ingest for all the IDs in the context(in the latest select command)

Ingests an AIP object. The object must be ready for the ingestion.

25

Configuration


invalidate Options

• --id or --last <id> of an AIP object

• --category <value> specifies category, if it is needed

• --description <value> specifies description, if it is needed

Invalidates an AIP object.

job Options

• --id or --last <id> of an job-definition object or <index> of such id inthe latest select result set performed

• --activate

• --inactivate

Does some configuration tasks for a job-definition object (i.e.,activate/inactivate):

job --id 1 --activate

job --id 1 --inactivate

list-aliases No arguments

Shows a list of configuration type aliases that are available on aconcrete, connected application.

The list of aliases depends on the context you set If you set yourcontext to use:

--tenant INFOARCHIVE -application MyApplication

, you will not have access to the resources available on the tenant level.

next No arguments and options

Goes to the next page after the select command. This command couldbe available if there is more than one page in the select result set. Also,it is available only if previous command was one of select/next/prev.

prev No arguments and options

Goes to the previous page after the select command. This commandcould be available if there is more than one page in the select resultset. Also, it is available only if previous command was one ofselect/next/prev.

26

Configuration


receive Options:

• --format <value> format of the SIP

• --from <value> path(s) to one or more SIP files delimited by ’;’ or ’:’(the system-dependent path-separator character)

• --list if it’s specified output AIP ID values to a list (comma-separated)

Receives (upload) a SIP package or packages to the server.

reject Options:

• --id or --last <id> of an AIP object

• --category <value> specifies category, if it is needed.

• --description <value> specifies description, if it is needed.

Rejects an AIP object.

relations No arguments and options

Shows all the relations are available in the context.

remove-hold Options:

• --id or --last <id> of a policy object or <index> of such id in the latestselect result set performed

• --externalId <id> of an object for which the removing will beperformed

Removes a hold from an object:

1 --externalId e21fab5c-e82a-4215-8f5a-bd0169a552d7

remove-retention Options:

• --id or --last <id> of a retention-policy object or <index> of such idin the latest select result set performed

• --externalId <id> of an object for which the removing will beperformed

Removes a retention policy from an object:

remove-retention --id 1 --externalId e21fab5c-e82a-4215

-8f5a-bd0169a552d7

27

Configuration


roles-update Options:

• --name or --group <name> updates roles for specified group name

• --roles <set of roles> a set of comma-separated roles

Only available after the groups-view command has been performed.

roles-view Options:

• --name or --group <name> view roles by specified group name

Only available after the groups-view command has been performed.

run-job Options:

• --id or --last <id> of a job-definition object or <index> of such id inthe latest select result set performed


• --data <value> JSON-like formatted data, use this instead ofspecifying a file in XML format

Run a job by its job-definition. JSON formatted data should be insideof double quotes. Examples:

The XML format of a data file is exactly the same as used inapply-retention (hold) commands.

run-job --id 1 --from job-run.xml

run-job --id 1 --data "{'now': 'true'}"

run-job --id 1

select Argument: <type alias>

Options:


• --where <an expression> this defines an expression tofilter the data. This should be a SpEL expression (SpEL(http://docs.spring.io/spring/docs/current/spring-framework-reference/html/expressions.html) expressions.). Cannot be usedfor type aliases that do not support SpEL (at the moment: AIP).According to SpEL documentation, case insensitivity is permittedfor the first letter of property names.

• --filter <value> Filter parameters, should be defined as a simplifiedJSON fragment

• --property <name of the property> returns value of the specifiedproperty

28

http://docs.spring.io/spring/docs/current/spring-framework-reference/html/expressions.html

http://docs.spring.io/spring/docs/current/spring-framework-reference/html/expressions.html

Configuration


• --rel <name of relation> specify a relation that is taken from thecontext

• --var-set <context variable name> set a context variable with givenname to the output of the command. Should be used within theproperty option together


• --names if it is specified, all the names of all the properties availablewill be shown. It’s worth when interacting with some types whichmight not have ID and name properties.

• --attributes <list of additional attribute names to be shown> acomma delimited list of additionalattributes (for those which areshown by default)

Searches and outputs the objects are found. It shows values only forattributes name and id (if the option property is not specified). It ispossible to define additional attributes to be shown (the attributesoption). If none of them is found, it shows value of the first attributefound. Produces the ID indexes (if some results are found) to bestored in the context. Indexes are 1 based. Such context is widely usedin other CLI commands. Also, these indexes are treated as contextvariables named like ID_<index>.

select holding

select holding --where "?[name=='PurchaseOrder']"


--property id

select ingest-node --rel ingest-nodes


--property id --set-name holdingId

select aip --filter "{'PurchaseOrder'}" select aip

--attributes xdbMode

set-alias-properties Argument: <type alias>

Options:


• --properties <set of property names> set of properties to be shownby default in the view command for the type alias specified

Examples:

set-alias-properties aip --properties "id, name,

stateCode, phase, dss.holdingName"

29

Configuration


set-header Argument: Set an HTTP header for connection requests

Options:

• --name <value> name of a attribute

• --value <value> valueof a attribute

Add an attribute and its value to HTTP headers. The headers will besent to the server in next commands performed (during the currentsession and until the disconnect command is performed).

set-header --name X-header --value

YWRhbUBpYWN1c3RvbWVyLmNvbTpwYXNzd29yZA==

set-type-alias Options:

• --name <type alias>

• --rel <name of relation> specify a relation which is taken from thecontext

Allows customer to set a custom type alias to access for some resourcedefined by a relation in the context:

To register a type, you need to know if the type is defined on the:

• system,

• tenant, or .

• application

Use a REST client to discover where the various objects are stored.

In the following example, conditions are defined on the tenant so thecontext is being set to the tenant:

use --tenant INFOARCHIVE

set-type-alias --name condition --rel conditions

The challenge with registering system types, however, is once thecontext is set using the use command, there is no way to clear outthat tenant context.

The work-around is to register the system types beforehand. For anynew types that were not registered for 4.1 , add the start-up file.

1. After entering the connect command, register the system types.

2. Enter the use command to set the context to your tenant(INFOARCHIVE):


30

Configuration


3. Register any additional tenant types.

4. Enter the use command to set the context to your application:

use --tenant INFOARCHIVE --application

<MyApplication>

5. Register any additional application types.

If you are using multiple applications, enter the use command to setthe context back to tenant:

--tenant INFOARCHIVE'

Replace <MyApplication> with the application you want to use.Otherwise, you will have to keep registering the types each time youstart a new IAShell session.

ssl-keystore Options:

• --file <file path> Set the keystore file path to use for SSL

• --type <type> Set the keystore type to use for SSL

• --password <value> Set the keystore password to use for SSL

Example:

ssl-keystore --file "/myfolder/client/clientCertStore

.p12" --password abcdefg --type PKCS12

ssl-reset Resets the SSL configuration back to the system default

ssl-trust-store Options:

• --file <file path> Set the truststore file path to use for SSL

• --type <type> Set the truststore type to use for SSL

• --password <value> Set the truststore password to use for SSL

Example:

ssl-trust-store --file "/myfolder/client/client

_truststore.jks" --password abcdefg --type JKS

31

Configuration


update Argument: <type alias>

Options:



• --properties <value> (names and values to be changed)

Change the object specified by an option and properties (names andvalues). Examples:

update holding --id "222323-3533-22" --properties

"{'name': 'MyFavorite', 'priority': 1}"

update --last 1 --properties "{'name': 'MyFavorite2',

'priority': 2}"

use Options:

• --federation <a federation name> discovers related aliases(xdb-database)

• --space <a space name> discovers related aliases ()

• --space-root-library <a space-root-library name> discovers relatedaliases (xdb-library)

• --tenant <a tenant name> discovers related aliases (tenant-aware)

• --application <an application name> discovers related aliases(application-aware)

• --space-root-folder <a space-root-folder ID value> discovers relatedaliases (file-system-folder)

• --database <a database name> discovers related aliases (schema)

• --search-composition <a search-composition object name> discoversrelated aliases (x-form and x-query)

• --schema <an SQL schema object name> discovers related aliases(table)

• ecs-end-point <an ECS endpoint name> discovers related aliases(ecs-credentials)

Uses configuration object names to establish relations and discoverchildren type aliases.

var-clear Clears all the variables in the context

32

Configuration


var-get Options:

• --name <name of the variable>

• --value <some value> should contain a SpEL expression :#{<expression>}

Shows a variable by name or by value within a SpEL expression. Forexample:

var-get --name application

var-get --value #{tenant}

var-list Shows all the variables in the context

var-set Options:

• --name <name of the variable>

• --value <some value>

Sets a user-defined context variable:

var-set --name myVar --value someValue

After the setting, the customer can address the variable by SpEL:#{myVar}

There are variables set by the CLI after the connect command. Also,each relation in the context is treated as a variable.

view Argument: <type alias>

Options:


• --id or --last (some id) or <index> in the latest search result setshown, default value is 0

• --properties <set of property names to be shown> comma delimited

• --show-all if it presents all the properties will be shown

• --xml to show a dump of all properties in XML format (suitable forcopy/past for the configure command)

• --relations to show only relations for the object (HAL relations)

• --rel <name of relation> specify a relation which is taken from thecontext. Cannot be used with the id option at the same time.


33

Configuration


Shows predefined set of properties of an object. The set could bepredefined by using the set-alias-properties command. Examples:

view aip --id "3232-23232-32223" view --id 2

view --show-all --last 1

view --last 1 --properties "id, name, productionDate,

state"

view --show-all --id 1 --xml

view --id 1 --relations

view --t store --rel xdb-store

Using the apply-hold CommandThe following example illustrates the process of using the apply-hold command on files in thePhoneCalls application.

1. Connect as a Retention Manager.

Note: Administrators do not see the hold alias.

2. Enter the use command:


3. Enter the select hold command:

sample response:ia-shell>select hold=======================================================name id-------------------------------------------------------1 PhoneCalls-hold a77bbaa6-a698-4b2f-b41f-dae7a248be292 Audit-hold c00e1ad7-5135-4c00-9c33-56d8b61b3188-------------------------------------------------------

4. Enter the use command:

use --application PhoneCalls

5. Enter the select command to determine which AIPs are to be protected:

ia-shell/hold>select aip================================================================name id----------------------------------------------------------------1 PhoneCalls-CC-1000000-1 cdba6e7c-47e9-4c7d-a8b1-460259fb751b2 PhoneCalls-CC-1000001-1 c39cdf7f-fe89-4408-ad0e-dbcdb9d85b3f3 PhoneCalls-CC-1000003-1 e0ae4248-6c30-4a59-8303-95b84dc28ff84 PhoneCalls-CC-1000004-1 ee4d9bee-b4b3-430a-bf1b-4359d74d828e

34

Configuration

5 PhoneCalls-CC-1000005-1 716cd062-e4fa-447c-9a93-0432eedbf3b86 PhoneCalls-CC-1000006-1 cb93b390-be63-4ad1-8545-8c61d07592297 PhoneCalls-CC-1000007-1 d798bb94-614e-4a89-9705-5e940ef7077c8 PhoneCalls-CC-1000008-1 681c35e5-f727-467e-99a4-f0b57f70ca419 PhoneCalls-CC-1000009-1 edca4585-7522-41b9-8727-5084915cbcf610 PhoneCalls-CC-1000010-1 a4453216-3050-4127-afd2-2ae0b687be59----------------------------------------------------------------Elements count: 10. Page 1 of 1

6. In a text editor, create an XML file called applyHold.xml and store it in the tools directory. Ifprotecting one AIP, put the id in the itemsToProtect. Be sure to enter the correct id, as the systemdoes not return an error if the AIP does not exist.

<?xml version="1.0"?><data><itemsToProtect>cdba6e7c-47e9-4c7d-a8b1-460259fb751b</itemsToProtect><itemsToProtect/><holdSetName>MyHoldSet</holdSetName><type>AIP</type><application>#{application}</application></data>

7. Enter the select command to reset the context back to the hold:

select hold

8. Enter the apply-hold command:

apply-hold --id 1 --from applyHold.xml

Tip: If the following error message is received, hold sets cannot be shared:

Command failed org.springframework.web.client.HttpClientErrorException: 409 errors:ProcessingException Hold set already in useuse a new name

9. Enter the select command to confirm the hold was applied:

select aipia-shell/aip>view --id 1 --show-all=================================================================createdBy [email protected] 2016-09-07T14:15:03.046-04:00lastModifiedBy [email protected] 2016-09-07T14:15:21.580-04:00version 13name PhoneCalls-CC-1000000-1aipId cdba6e7c-47e9-4c7d-a8b1-460259fb751bdss.id 1000000dss.holdingName PhoneCallsdss.pdiSchema urn:eas-samples:en:xsd:phonecalls.1.0dss.productionDate 2001-12-01T00:00+01:00dss.baseRetentionDate 2001-12-01T00:00+01:00dss.producer CCdss.entity Platform

35

Configuration

dss.priority 0dss.application UAsipProductionDate 2001-12-01T00:00+01:00sipSeqno 1sipIsLast truesipAiuCount 3sipPageCount 0pkeys.dateTime01 2001-11-02T21:06:39.104+01:00pkeys.dateTime02 2001-11-26T10:17:18.104+01:00ingestDeadlineDate 2016-09-07T15:55:02.386-04:00pdiFileSize 2347pdiValuesCharCount 519ciCount 3ciSize 123369ciFormat ci_containerriRecordSize 354stateCode COMphaseCode COMxdbMode PRIVATExdbPdiSchema urn:eas-samples:en:xsd:phonecalls.1.0partOfAggregate falseaggregateCiSeqno 0aggregateAiuSeqno 0pdiConfigName PhoneCalls-pdipriority 1receiveStartDate 2016-09-07T14:15:02.386-04:00receiverNodeName receiver_node_01sipFileFormat sip_zip_cryptosipFileSize 123186sipFileHash df249d716be4da2edacd772c2189d6a1698d5996ingestWaitStartDate 2016-09-07T14:15:02.701-04:00ingestStartDate 2016-09-07T14:15:20.903-04:00ingestNodeName ingest_node_01ingestConfigName PhoneCalls-ingestcommitWaitStartDate 2016-09-07T14:15:21.490-04:00commitSync truecommitDate 2016-09-07T14:15:21.542-04:00pdiCryptoConfigName PhoneCalls-pdi-cryptopdiCryptoIv aYr9dyRE7G6GvAYxCinhAA==pdiCryptoKeyId PDI_d7cdb1c0-1452-48ff-bf70-c497ca36a9d0cryptoEncoding base64pdiCryptoHashSalt 6QFmeyTbpThp405S80jGPQ==ciCryptoKeyId CI_d7cdb1c0-1452-48ff-bf70-c497ca36a9d0ciCryptoIv I97eUkyJza0eaNCsNhZV3Q==sipCryptoKeyId SIP_d7cdb1c0-1452-48ff-bf70-c497ca36a9d0sipCryptoIv sB46HP9sictm6/wXRHHRKg==returnCode OKprojectedDispositionDate 2002-02-28T00:00+01:00underHold trueunderRetention truepermission.groups []state CompletedxdbLibraryName a3ba6eab-35ef-4bce-a0c0-cc10ce522e35openAggregate falseaggregate falsexdbLibraryIndexSize 53248phase CompletedvalidAggregate falsepdiCryptoObjectName PhoneCalls-crypto-objectciCryptoObjectName PhoneCalls-crypto-objectsipCryptoObjectName PhoneCalls-crypto-objectxdbLibrarySize 253952xdbLibraryDetached falseid cdba6e7c-47e9-4c7d-a8b1-460259fb751b-----------------------------------------------------------------

36

Configuration

Verifying the Hold Set has been Created

The following example illustrates how to verify that the hold set has been successfully created foritems stored in the PhoneCalls application:

1. Enter the use command to set the context to the application:

use --tenant INFOARCHIVE --application PhoneCalls

2. Enter the set-type-alias command to set the alias for the holds sets:

ia-shell>set-type-alias --name hold-set --rel hold-setsNew alias type was addedia-shell>

3. Use the select command to list the hold sets:

ia-shell>select hold-set==================================================name id--------------------------------------------------1 MyHoldSet a3f2b0ae-a00c-427c-b6b8-4c522e7419142 My HoldSet cb876e04-3326-4ec6-bd41-2616cdb568f8--------------------------------------------------

4. Use the view command to display the properties of the hold set:

ia-shell/hold-set>view --id 2 --show-all=================================================================createdBy [email protected] 2016-09-08T09:51:40.708-04:00lastModifiedBy [email protected] 2016-09-08T09:51:40.794-04:00version 2tenantId 5061ae2d-adac-405b-b06e-36207e02b7edfilterName d7cdb1c0-1452-48ff-bf70-c497ca36a9d0partitionkeys ["cdba6e7c-47e9-4c7d-a8b1-460259fb751b"]name My HoldSetdescription nulltype AIPexternalReference nullexternalReferenceContext nullholdName PhoneCalls-holdholdTypeName LEGALholdReviewDate nullcount 1supplementalData nullid cb876e04-3326-4ec6-bd41-2616cdb568f8-----------------------------------------------------------------

37

Configuration

Using the configure CommandThis section contains examples illustrating how to use the configure command.

The configure command has a required argument: path to a file of the data. Format of the datafiles is XML-based and quite complicated. For more information, see Format of a Data File for theConfigure Command.

Examples of the select command:

ia-shell>select aip --where "?[dss.holdingName=='PhoneCalls']"================================================================# name id----------------------------------------------------------------1 PhoneCalls-CC-1000000-1 dc24b5c8-8d48-45e2-8cf2-5ed7c0da2f362 PhoneCalls-CC-1000001-1 77babb39-b0fd-4ba9-8bf5-732331dba6813 PhoneCalls-CC-1000003-1 5cc46f91-cf3f-4af4-995b-aa7f56f6f0184 PhoneCalls-CC-1000004-1 0bffe02d-c8c8-4ad3-b5c0-037e8a9bf50b5 PhoneCalls-CC-1000005-1 642342a4-9377-49b2-ac8e-10bee3e1af506 PhoneCalls-CC-1000006-1 d8bdd8e9-a8e4-4ba5-bff6-4b06a4e0690a7 PhoneCalls-CC-1000007-1 4a35f487-b627-462c-a2ed-ad662ef2e27b8 PhoneCalls-CC-1000008-1 a2b29279-482c-4d8b-9b5a-54b00910f53d9 PhoneCalls-CC-1000009-1 632ca99f-2862-4eaa-86be-e2b0fb42996710 PhoneCalls-CC-1000010-1 88a5cdc9-3390-4e17-8917-5a66f39ff1a9----------------------------------------------------------------Elements count: 10. Page 1 of 1

Examples of the view command:

ia-shell>view --show-all --last 1==============================================================createdBy [email protected] 2016-04-27T18:35:44.927+03:00lastModifiedBy [email protected] 2016-04-27T18:35:44.927+03:00name PhoneCalls-CC-1000000-1aipId dc24b5c8-8d48-45e2-8cf2-5ed7c0da2f36dss.id 1000000dss.holdingName PhoneCallsdss.pdiSchema urn:eas-samples:en:xsd:phonecalls.1.0dss.productionDate 2001-12-01T00:00+01:00dss.baseRetentionDate 2001-12-01T00:00+01:00dss.producer CCdss.entity 1dss.priority 0dss.application 1sipProductionDate 2001-12-01T00:00+01:00sipSeqno 1sipIsLast truesipAiuCount 3sipPageCount 0ingestDeadlineDate 2016-04-27T20:15:44.210+03:00stateCode WINGphaseCode WINGpriority 1receiveStartDate 2016-04-27T18:35:44.210+03:00receiverNodeName receiver_node_01sipFileFormat sip_zipsipFileSize 123162ingestWaitStartDate 2016-04-27T18:35:44.272+03:00

38

Configuration

commitSync truereturnCode OKunderHold falseunderRetention falsepermission.groups []state Waiting ingestionphase Waiting Ingestionid dc24b5c8-8d48-45e2-8cf2-5ed7c0da2f36--------------------------------------------------------------ia-shell>view --last 1=======================================name PhoneCalls-CC-1000000-1dss.holdingName PhoneCallssipSeqno 1sipAiuCount 3underHold falseunderRetention falsestate Waiting ingestion---------------------------------------ia-shell>view --last 1 --xml<object><receiverNodeName>receiver_node_01</receiverNodeName><commitSync>true</commitSync><dss><productionDate>2001-12-01T00:00+01:00</productionDate><application>1</application><holdingName>PhoneCalls</holdingName><baseRetentionDate>2001-12-01T00:00+01:00</baseRetentionDate><producer>CC</producer><id>1000000</id><pdiSchema>urn:eas-samples:en:xsd:phonecalls.1.0</pdiSchema><priority>0</priority><entity>1</entity>

</dss><ingestWaitStartDate>2016-04-27T18:35:44.272+03:00</ingestWaitStartDate><sipIsLast>true</sipIsLast><ingestDeadlineDate>2016-04-27T20:15:44.210+03:00</ingestDeadlineDate><aipId>dc24b5c8-8d48-45e2-8cf2-5ed7c0da2f36</aipId><sipAiuCount>3</sipAiuCount><underHold>false</underHold><returnCode>OK</returnCode><sipSeqno>1</sipSeqno><sipPageCount>0</sipPageCount><underRetention>false</underRetention><sipFileSize>123162</sipFileSize><phaseCode>WING</phaseCode><state>Waiting ingestion</state><phase>Waiting Ingestion</phase><lastModifiedDate>2016-04-27T18:35:44.927+03:00</lastModifiedDate><lastModifiedBy>[email protected]</lastModifiedBy><permission/><priority>1</priority><receiveStartDate>2016-04-27T18:35:44.210+03:00</receiveStartDate><createdDate>2016-04-27T18:35:44.927+03:00</createdDate><createdBy>[email protected]</createdBy><name>PhoneCalls-CC-1000000-1</name><stateCode>WING</stateCode><sipProductionDate>2001-12-01T00:00+01:00</sipProductionDate>

</object>

Example of the view command which shows certain properties:ia-shell>view --last 1 --properties "name, dss.productionDate, priority, sipFileFormat"==========================================

39

Configuration

name PhoneCalls-CC-1000000-1dss.productionDate 2001-12-01T00:00+01:00priority 1sipFileFormat sip_zip------------------------------------------

Example of the view command which shows relations:ia-shell>view --id 1 --relations==========================================================================================self http://localhost:8080/systemdata/aips/dc24b5c8-8d48-45e2-8cf2-5ed7c0da2f36delete http://localhost:8080/systemdata/aips/dc24b5c8-8d48-45e2-8cf2-5ed7c0da2f36application http://localhost:8080/systemdata/applications/0bc3f378-3194-48e1-b608-309805cbd0e9contents http://localhost:8080/systemdata/aips/dc24b5c8-8d48-45e2-8cf2-5ed7c0da2f36/contentsingest http://localhost:8080/systemdata/aips/dc24b5c8-8d48-45e2-8cf2-5ed7c0da2f36/ingestinvalid http://localhost:8080/systemdata/aips/dc24b5c8-8d48-45e2-8cf2-5ed7c0da2f36/invalid954a2b3240c9groups http://localhost:8080/systemdata/aips/dc24b5c8-8d48-45e2-8cf2-5ed7c0da2f36/groups------------------------------------------------------------------------------------------

Example of the select command which uses the --rel option:ia-shell>select --t ingest-node --rel ingest-nodes====================================================name id----------------------------------------------------ingest_node_01 055e5df4-d455-4ef8-8d2f-4ccb24519447----------------------------------------------------Elements count: 1. Page 1 of 1

Example of the receive command:ia-shell>receive --format sip_zip --from sip/ia_sip_01.zipOKia-shell>select aip --where "?[state == 'waiting ingestion']"================================================================name id----------------------------------------------------------------PhoneCalls-CC-2011020113-1 e01a70b6-7cdb-498e-b96e-7fafbff72e52----------------------------------------------------------------Elements count: 1. Page 1 of 1

Example of the enumerate command:ia-shell>select --t ingest-node====================================================name id----------------------------------------------------ingest_node_01 447f9660-3f59-4d8e-8859-a71446e36d6e----------------------------------------------------Elements count: 1. Page 1 of 1ia-shell>enumerate --id 1================================================================name id----------------------------------------------------------------PhoneCalls-CC-2011020113-1 e8d77e07-2b76-470c-8296-7c28e020406d

Example of the ingest command:

ia-shell>enumerate --id 1================================================================

40

Configuration

name id----------------------------------------------------------------PhoneCalls-CC-2011020113-1 e8d77e07-2b76-470c-8296-7c28e020406dia-shell>ingest --id 1

Example of the apply-retention command:ia-shell>select --t retention-policy===============================================name id-----------------------------------------------policy_01 cb88c557-7720-401a-ad43-ac4b8f171838-----------------------------------------------Elements count: 1. Page 1 of 1ia-shell>apply-retention --id 1 --from xml/app-ret.xml

Content of the app-ret.xml file:<?xml version="1.0"?><data>

<itemsToProtect><itemToProtect>14261127-3f82-4d8f-a603-41a1f43dc889</itemToProtect>

</itemsToProtect><itemsToProtect/><retainedSetName>MySecondRetainedSet2</retainedSetName><baseDate>2014-07-14</baseDate><ageIndividually>false</ageIndividually><application>#{application}</application>

</data>

The required link to the application is defined by context variable named application.

Example of the remove-retention command:ia-shell>remove-retention --id 1 --externalId 14261127-3f82-4d8f-a603-41a1f43dc889

Example of the run-job command which runs an instant job by it job-definition object:ia-shell>select --t job-definition=======================================================================# name id-----------------------------------------------------------------------1 Commit 4d0a9f41-dbf8-40dd-bcea-7194798d5d462 Confirmation 59256c0a-6641-41fa-938a-71d27197673e3 Clean aa7dde74-1cde-4ed6-8092-3d56a8b506b84 Invalidation d4aa57e9-5f6c-44fd-ae0d-6bbd70c824825 GeneratePurgeCandidateList c4c72287-8003-45ae-b268-1734945928996 DisposePurgeCandidateList 11945a91-fce1-4152-b84e-50aa74725ad77 TableApplyRetention c33c29ce-8e25-4a04-aa83-f5652f6b17618 RefreshMetrics 5394a528-5eb8-4c7a-80dc-0ab6690041159 Requalification c70dac2b-7c75-483c-8272-b030a761fad610 BuildIndex 5945a066-077f-4715-92aa-34535741a24711 Apply AIU Retention Policy 6c350f8f-13c6-410d-a9e7-21a045d5d68b12 Trigger Event Policy ef70d7dc-275e-48ed-a7e8-179461ad0ee613 Clean up Purge Candidate Lists 5fe6bebe-fa0e-43cf-85b3-ad44a442e37c14 Archive Audits 6febbd6f-ec3d-4fea-a840-9b019ad35c2d-----------------------------------------------------------------------Elements count: 14. Page 1 of 1ia-shell>run-job --id 5--data "{'now': true}"

It runs the fifth job in the list of the select command results.

Examples of the groups-view, roles-view and actions-view commands:ia-shell>groups-viewADMINISTRATORGROUP_RETENTION_MANAGER

41

Configuration

GROUP_END_USERGROUP_DEVELOPERGROUP_IT_OWNERGROUP_BUSINESS_OWNERia-shell>roles-view GROUP_IT_OWNERIT_OWNERia-shell>actions-view IT_OWNERCREATE_AUDITEXECUTE_BACKGROUND_SEARCH...VIEW_XFORMVIEW_XFORMSia-shell>actions-view IT_OWNER --output it_act.txt

Examples of the roles-update and actions-update commands:ia-shell>roles-update GROUP_IT_OWNER --roles "IT_OWNER, END_USER"IT_OWNEREND_USERia-shell>actions-update END_USER --actions VIEW_XFORM,VIEW_XFORMSVIEW_XFORMVIEW_XFORMSia-shell>actions-update IT_OWNER --from it_act.txt

Example of enabling SSL (TLS) support in the tool (ssl-keystore and ssl-trust-store commands).Pre-condition: There should be two stores within the client-side certificates (two-way TLS support):ia-shell>ssl-keystore --file "/tmp/apollo_cert/client/clientCertStore.p12"--password abcdefg --type PKCS12OKia-shell>ssl-trust-store --file "/tmp/apollo_cert/client/client_truststore.jks"--password abcdefg --type JKS

Format of a Data File for the ConfigureCommandThe data file is an XML-based format and not under any kind on XSD schema. This is a flexibleapproach when a user might not know a set of attributes, the data model or its relation to theimplementation details of each configuration object type.

The root element is named configuration. There can be several child nodes named object, and eachnode must contain the attribute named typeAlias.

There is no possibility to specify a tenant and an application. The connect command is aimed tospecify all of the information required to connect to some resources, including tenant, application,user/password, etc.

Each object node might have three parts: create, update and content. Either the create tag or the objecttag must have the checkExistSpEL attribute within an expression to:

• Check if such an object already exists in the repository; or

• Retrieve the object.

InfoArchive supports different sets of attributes for the create, as well as the update part, becausethere might be attributes that must be filled only when an object is being created.

Example of the file for a holding object:

42

Configuration

<?xml version="1.0"?><configuration>

<object typeAlias="holding" checkExistSpEL="?[name == 'PhoneCalls']"var-set="myContexVariable">

<create><name>PhoneCalls</name>

</create><update>

<priority>1</priority><subPriorities>

<deadLine>100</deadLine><priority>0</priority>

</subPriorities><subPriorities>


</subPriorities><logStoreEnabled>true</logStoreEnabled><xdbMode>PRIVATE</xdbMode><ciHashValidationEnabled>true</ciHashValidationEnabled><pdiXmlHashValidationEnabled>true</pdiXmlHashValidationEnabled><pdiXmlHashEnforced>false</pdiXmlHashEnforced><syncCommitEnabled>true</syncCommitEnabled><keepSipAfterCommitEnabled>false</keepSipAfterCommitEnabled><retentionClasses>

<name>ret_89</name><policies>PhoneCalls-policy</policies><policies/>

</retentionClasses><retentionClasses/><defaultRetentionClass>ret_89</defaultRetentionClass>

</update></object>...<object ...></object>

</configuration>

Other nodes placed under the create/update nodes are attributes of an object.

This approach is flexible and allows the customer to define inner objects or even collections of innerobjects without having any concrete XSD schema.

Note: Use the attribute var-set to define a context variable that will be set and equal to theself-relation (self-link) value of the object created or updated. After the setting, refer to it by SpEL:#{myContexVariable}.

Issues in the Update/Create Sections of Data Files

A collection with one element is treated by special way. It’ is required to detect that this is acollection, not a single attribute:...

<update><ingestionNodeNames>

first</ingestionNodeNames><ingestionNodeNames/>

</update>

43

Configuration

...

This approach should be used when the user needs to empty a collection:...

<update><subPriorities>null</subPriorities><subPriorities/>

</update>...

Appending and Replacing Elements in Collections

InfoArchive includes N-level (in-depth) persistence object sub-types (collections). One of thesub-types can include another collection. The CLI tool supports appending and replacing attributesin the collections on any level of sub-types.

The following is an example of the data file within a collection in XML format, a format that theconfigure command can consume:<configuration>

<object typeAlias="holding"><create checkExistSpEL="...">

...</create><update>

<ingestionNodeNames>firstNode</ingestionNodeNames><ingestionNodeNames>secondNode</ingestionNodeNames><ingestionNodeNames>thirdNode</ingestionNodeNames>

<ingestionProcessorConfigs><inputSipFormat>7z</inputSipFormat><ingestionProcessorName>eas_ingest_sip_7z-ci</ingestionProcessorName>

</ingestionProcessorConfigs><ingestionProcessorConfigs>

<inputSipFormat>zip</inputSipFormat><ingestionProcessorName>eas_ingest_sip_zip-ci</ingestionProcessorName>

</ingestionProcessorConfigs>

<subPriorities><deadLine>100</deadLine><priority>0</priority>



</subPriorities></update>

</object></configuration>

The following example illustrates the scenario in which there are collections inside of other collections(the subPriorites has secondLevelCollection):<configuration>

<object typeAlias="holding"><create checkExistSpEL="...">

...</create><update>

<ingestionNodeNames>firstNode</ingestionNodeNames>

44

Configuration

<ingestionNodeNames>secondNode</ingestionNodeNames><ingestionNodeNames>thirdNode</ingestionNodeNames>

<ingestionProcessorConfigs><inputSipFormat>7z</inputSipFormat><ingestionProcessorName>eas_ingest_sip_7z-ci</ingestionProcessorName>

</ingestionProcessorConfigs><ingestionProcessorConfigs>

<inputSipFormat>zip</inputSipFormat><ingestionProcessorName>eas_ingest_sip_zip-ci</ingestionProcessorName>

</ingestionProcessorConfigs>

<subPriorities><deadLine>111</deadLine><priority>1</priority><secondLevelCollection>first</secondLevelCollection><secondLevelCollection>second</secondLevelCollection>


<deadLine>222</deadLine><priority>2</priority><secondLevelCollection>third</secondLevelCollection><secondLevelCollection>fourth</secondLevelCollection>



The following example illustrates the scenario in which the user has to append an element at the firstposition to the secondLevelCollection in the second element of the subPriority collection:...

<update><subPriorities index="1">

<secondLevelCollection index="0" append="true">the very first</secondLevelCollection>


...

Replace/append in the test collection example...

<update><test replace="0">

<field1>1</field1><field1>2</field1><ingestionNodeNames>ReplacedThirdNode</ingestionNodeNames>

</test><test index="1">

<field1>3</field1><field1>4</field1><ingestionNodeNames index="0" append="true">veryFirstNode</ingestionNodeNames><ingestionNodeNames append="true">fourthNode</ingestionNodeNames>

</test><test append="true">

<ingestionNodeNames>AddedNode</ingestionNodeNames></test>

</update>...

45

Configuration

The above example changes the following:

1. Replaces element ’0’ in the collection.

2. Updates the second element in the collection to the field1 collection (with replacing any oldvalues).Append the ’veryFirstNode’ value to the collection named ingestionNodeNamesto the first position. Add one more value (’fourthNode’) to the end of the collectioningestionNodeNames.

3. Adds one element to the end of the collection test.

Update a certain element in the collection...

<update><test index="1">

<field1>3</field1><field1>4</field1>

</test></update>

...

Update entire collection...

<update><test>

<field1>3</field1><field1>4</field1>

</test><test/>

</update>...

Note: If a user updates a collection and needs to use the only element, use the empty second element,as the JSON-Java library cannot detect that this a collection, not a single attribute.

Setting InfoArchive Clients for HTTPS Communication

The Java Secure Socket Extension (JSSE) system properties must be set to enable usage of custom keyand trust stores by an InfoArchive Java Database Connectivity (JDBC) driver host application.

An InfoArchive JDBC driver ’ssl’ property must be set to ’true’ to enable https communication.

SSL1 System Properties

-Djavax.net.ssl.trustStoreType=jks-Djavax.net.ssl.trustStore="E:\\info-archive\\certificates\\client\\client_truststore.jks"-Djavax.net.ssl.trustStorePassword=abcdefg

46

Configuration

SSL2 System Properties

-Djavax.net.ssl.trustStoreType=jks-Djavax.net.ssl.trustStore="E:\\info-archive\\certificates\\client\\client_truststore.jks"-Djavax.net.ssl.trustStorePassword=abcdefg-Djavax.net.ssl.keyStoreType=pkcs12-Djavax.net.ssl.keyStore="E:\\info-archive\\certificates\\client\\clientCertStore.p12"-Djavax.net.ssl.keyStorePassword=abcdefg

InfoArchive JDBC Connection Setup

The Java Database Connectivity (JDBC) driver authentication is done using a user name andpassword.

1. Specify the following properties:

user=user-namepassword=user-passwordclientId=infoarchive.jdbcclientSecret=infoarchive.jdbc-client-secret (it is configured in theInfoArchive web application)

2. Using JDBC API, specify the following properties:

com.emc.ia.sql2xquery.jdbc.JdbcDriver:final JdbcDriver jdbcDriver = new JdbcDriver();DriverManager.registerDriver(jdbcDriver);final Properties info = new Properties();info.setProperty("user", "user-name");info.setProperty("password", "user-password");info.setProperty("clientId", "infoarchive.jdbc");info.setProperty("clientSecret", "secret");final Connection connection = DriverManager.getConnection(connectionString, info);

3. Using SQuirreL SQL Client: Modify the selected alias | Properties | Driver properties.

4. Appending to a connection string:

jdbc:ia://localhost:8080/restapi?tenant=INFOARCHIVE&application=Tickets&database=Tickets-sql-db&[email protected]&password=password&clienId=infoarchive.jdbc&clientSecret=secret

The authEndpoint property points to an authentication end-point host[:port]/path/to/authentication/endpoint.

For example, in the case of the JDBC connection string:

jdbc:ia:////hostname:8080/infoarchive-webapp/restapi?tenant=INFOARCHIVE&application=Baseball&database=Baseball-sql-db'authEndpoint'=localhost:8080/infoarchive-webapp/oauth/token.

47

Configuration

IAShell

The following example illustrates the process of enabling SSL (TLS) support in the tool (ssl-keystoreand ssl-trust-store commands). Pre-condition: you should have two stores in the client-sidecertificates (two-way TLS support):ssl-keystore --file "/tmp/apollo_cert/client/clientCertStore.p12" --password abcdefg--type PKCS12ssl-trust-store --file "/tmp/apollo_cert/client/client_truststore.jks" --password abcdefg--type JKS

Disabling:ssl-reset

Content Uploading

A data file can have one or more content sections:...

<object ...><create>...</create><update/>

...</update><content storeName="filestore_01" format="xml" file="/content/query.xml"/><content storeName="filestore_01" format="text" file="/content/query.txt"/>

</object>...

The configure command handles the content section after performing the update section.

InfoArchive Server and Gateway/InfoArchive WebApplication Communication Setup

This section describes how to set up the InfoArchive server and Gateway/InfoArchive webapplicationfor HTTP communication:

Browser <-------- HTTPS --------> Gateway IAWA <-------- HTTPS --------> IAS

First, generate or obtain certificates from some CA Root. Once you have the certificates, complete thefollowing procedure:

1. Import the InfoArchive server certificate into the key store.

2. Adjust the INSTALL_INFOARCHIVE_DIR\config\server\application-ssl.yml file to pointto the key store and specify the key store* passwords.

3. Run the InfoArchive server:

48

Configuration

> bin\infoarchive-server-ssl.bat

4. Import the InfoArchive server certificate into the Gateway/InfoArchive server key store.

5. Export the InfoArchive server and Gateway/InfoArchive web application public certificate intothe Gateway/InfoArchive web application truststore.

6. Adjust the INSTALL_INFOARCHIVE_DIR\config\webapp\application.yml file:

spring:application:name: infoarchive.gateway

profiles:active: infoarchive.profile.HTTPS,infoarchive.gateway.profile.AUTHENTICATION_IN_MEMORY

cloud:config:enabled: false

infoarchive:gateway:host: localhostport: 8080contextPath: /token:secret: secret

server:host: ${infoarchive.gateway.host}port: ${infoarchive.gateway.port}contextPath: ${infoarchive.gateway.contextPath}

zuul:routes:restapi:path: /restapi/**sensitiveHeaders: ""url: http://localhost:8080/

addProxyHeaders: truehost:socket-timeout-millis: 60000

---spring:profiles: "infoarchive.profile.HTTPS"

server:ssl:key-store: "location of gatewayKeyStore.jks"key-store-password: abcdefg (this is a sample)key-password: abcdefg (this is a sample)keyStoreType: JKS

zuul:routes:restapi:path: /restapi/**sensitiveHeaders: ""url: "https://localhost:8080/"

7. Add the following two command lines parameters to the INSTALL_INFOARCHIVE_DIR\bin\infoarchive-webapp.bat:

49

Configuration

-Djavax.net.ssl.trustStore=<gateway truststore location goes here>-Djavax.net.ssl.trustStorePassword=<truststore password goes here>

For example:

::@rem Execute infoarchive-webappcd "%APP_HOME%""%JAVA_EXE%" -Djavax.net.ssl.trustStore=<gateway truststore location goeshere> -Djavax.net.ssl.trustStorePassword=<truststore password goes here>%DEFAULT_JVM_OPTS% %JAVA_OPTS% %INFOARCHIVE_WEBAPP_OPTS% -jar "%CLASSPATH%"%CMD_LINE_ARGS%:

You can add the additional system property -Djavax.net.debug=ssl:handshake to debugSSL.

8. Run Gateway/InfoArchive web application:

> bin\infoarchive-webapp.bat

Self-Signed Certificates-Based Setup

In the following scenario, the distribution has been extracted into:

c:\ia40\infoarchive

The InfoArchive server and the Gateway/InfoArchive web application are running on the samemachine, which means that the localhost works everywhere. You will have to use the IP addressesif the InfoArchive server and the Gateway/InfoArchive web application are running on differentmachines.

The InfoArchive server and Gatway/InfoArchive web application must both run in HTTPS mode.You cannot run one in HTTP and the other in HTTPS mode.

1. Create the following directories:

> cd c:\ia40\infoarchive> mkdir config\server\https> mkdir config\webapp\https

2. Create a self-signed InfoArchive server certificate:

> cd c:\ia40\infoarchive\config\server\https

> keytool -genkey -noprompt -trustcacerts -keyalg RSA -aliasIA_SERVER_CERT -dname "CN=localhost, OU=JavaSoft, O=Sun, L=Cupertino,S=California, C=US" -keypass abcdefg -storetype PKCS12 -keystore serverCertStore.p12-storepass abcdefg

The generated file is named serverCertStore.p12.

50

Configuration

3. List the self-singed InfoArchive server certificate:


> keytool -list -v -keystore serverCertStore.p12 -storepass abcdefg-storetype PKCS12

4. Create the keystore for infoArchive server:


> keytool -importkeystore -deststorepass abcdefg -destkeypass abcdefg-destkeystore serverKeyStore.jks -srckeystore serverCertStore.p12-srcstoretype PKCS12 -srcstorepass abcdefg -alias IA_SERVER_CERT

5. Export the InfoArchive server certificate:


> keytool -export -noprompt -rfc -alias IA_SERVER_CERT -file server_public_cert.cert-keystore serverKeyStore.jks -storepass abcdefg -storetype JKS

The generated file is named server_public_cert.cert.

6. Import the InfoArchive server certificate into Gateway/infoArchive web application:

> copy c:\ia40\infoarchive\config\server\https\server_public_cert.certc:\ia40\infoarchive\config\webapp\https

> cd c:\ia40\infoarchive\config\webapp\https

> keytool -import -noprompt -trustcacerts -alias IA_SERVER_CERT -fileserver_public_cert.cert -keystore gatewayTrustStore.jks -storepass abcdefg

The generated file is named gatewayTrustStore.jks.

7. List the imported InfoArchive server certificate:

> cd c:\ia40\infoarchive\config\webapp\https> keytool -list -v -keystore gatewayTrustStore.jksKeystore type: JKSKeystore provider: SUN

Your keystore contains 1 entries

Alias name: server_public_certCreation date: Jun 10, 2016Entry type: trustedCertEntry

Owner: CN=localhost, OU=JavaSoft, O=Sun, L=Cupertino, ST=California,C=US

51

Configuration

Issuer: CN=localhost, OU=JavaSoft, O=Sun, L=Cupertino, ST=California, C=USSerial number: 357866c0Valid from: Fri Jun 10 08:32:40 PDT 2016 until: Thu Sep 08 08:32:40 PDT 2016Certificate fingerprints:

MD5: F1:05:E4:6A:F4:7C:C4:A7:8A:0E:B6:DE:A9:16:DF:AASHA1: CC:65:A3:74:85:C0:97:DA:95:AE:13:1F:CA:75:16:CA:66:F2:17:87SHA256: 22:86:56:6B:16:A1:D5:C1:66:CE:2B:A7:F5:CA:EC:74:45:C5:A5:3E:00:DD:F1:28:8F:F3:F8:7D:2B:00:25:A4Signature algorithm name: SHA256withRSAVersion: 3

Extensions:

#1: ObjectId: 2.5.29.14 Criticality=falseSubjectKeyIdentifier [KeyIdentifier [0000: 5D 34 8D B8 34 CF 0B A0 64 05 20 93 E4 BA 92 28 ]4..4...d. ....(0010: CD EB 20 64 .. d]]

**************************************************************************************

8. Create the self-signed Gateway/InfoArchive web application certificate:

> cd c:\ia40\infoarchive\config\webapp\https> keytool -genkey -noprompt -trustcacerts -keyalg RSA -alias IA_GATEWAY_CERT -dname"CN=localhost, OU=JavaSoft, O=Sun, L=Cupertino, S=California, C=US" -keypassabcdefg -storetype PKCS12 -keystore gatewayCertStore.p12 -storepass abcdefg

The generated file is gatewayCertStore.p12.

9. List the self-singed Gateway/InfoArchive web application certificate:

> cd c:\ia40\infoarchive\config\webapp\https> keytool -list -v -keystore gatewayCertStore.p12 -storepass abcdefg -storetypePKCS12

10. Create the keystore for Gateway/InfoArchive web application:

> cd c:\ia40\infoarchive\config\webapp\https> keytool -importkeystore -deststorepass abcdefg -destkeypass abcdefg-destkeystore gatewayKeyStore.jks -srckeystore gatewayCertStore.p12-srcstoretype PKCS12 -srcstorepass abcdefg -alias IA_GATEWAY_CERT

11. Export the Gateway/InfoArchive web application certificate:

> cd c:\ia40\infoarchive\config\webapp\https> keytool -export -noprompt -rfc -alias IA_GATEWAY_CERT -file gateway_public_cert.cert-keystore gatewayKeyStore.jks -storepass abcdefg -storetype JKS

The generated file is named gateway_public_cert.cert.

52

Configuration

12. Import the Gateway/InfoArchive web application certificate into Gateway/InfoArchive webapplication. This step is required because Gateway is its own client for getting the group to rolemapping from the InfoArchive server via the REST API from the backend:

> cd c:\ia40\infoarchive\config\webapp\https> keytool -import -noprompt -trustcacerts -alias gateway_public_cert-file gateway_public_cert.cert -keystore gatewayTrustStore.jks -storepass abcdefg

The generated file is named gatewayTrustStore.jks.

13. List the imported Gateway/InfoArchive web application certificate:

> cd c:\ia40\infoarchive\config\webapp\https> keytool -list -v -keystore gatewayTrustStore.jksKeystore type: JKSKeystore provider: SUN

Your keystore contains 2 entries

Alias name: gateway_public_certCreation date: Jun 10, 2016Entry type: trustedCertEntry

Owner: CN=localhost, OU=JavaSoft, O=Sun, L=Cupertino, ST=California, C=USIssuer: CN=localhost, OU=JavaSoft, O=Sun, L=Cupertino, ST=California, C=USSerial number: 78b0bb1bValid from: Fri Jun 10 08:41:09 PDT 2016 until: Thu Sep 08 08:41:09 PDT 2016Certificate fingerprints:

MD5: 21:21:12:82:77:83:06:B9:EC:87:93:D3:07:FD:50:22SHA1: 56:1C:F2:D2:17:75:9A:4F:FB:F3:D9:C3:89:64:7D:29:52:4E:DC:1BSHA256: 92:3D:F3:E3:83:98:61:6D:34:02:66:6E:2D:07:60:F6:E9:DD:3D:BA:AD:AC:31:1C:91:39:76:85:9A:9F:C0:FDSignature algorithm name: SHA256withRSAVersion: 3

Extensions:

#1: ObjectId: 2.5.29.14 Criticality=falseSubjectKeyIdentifier [KeyIdentifier [0000: 3C 90 5E 2D CF 7E 2A 49 AB 20 DC E1 E5 2A 1E 71 <.^-..*I. ...*.q0010: 4A 2A 83 A1 J*..]]

**************************************************************************************

Alias name: server_public_certCreation date: Jun 10, 2016Entry type: trustedCertEntry

Owner: CN=localhost, OU=JavaSoft, O=Sun, L=Cupertino, ST=California, C=USIssuer: CN=localhost, OU=JavaSoft, O=Sun, L=Cupertino, ST=California, C=USSerial number: 357866c0Valid from: Fri Jun 10 08:32:40 PDT 2016 until: Thu Sep 08 08:32:40 PDT 2016Certificate fingerprints:

MD5: F1:05:E4:6A:F4:7C:C4:A7:8A:0E:B6:DE:A9:16:DF:AASHA1: CC:65:A3:74:85:C0:97:DA:95:AE:13:1F:CA:75:16:CA:66:F2:17:87SHA256: 22:86:56:6B:16:A1:D5:C1:66:CE:2B:A7:F5:CA:EC:74:45:C5:A5:3E:00:DD:F1:28:8F:F3:F8:7D:2B:00:25:A4

53

Configuration

Signature algorithm name: SHA256withRSAVersion: 3

Extensions:

#1: ObjectId: 2.5.29.14 Criticality=falseSubjectKeyIdentifier [KeyIdentifier [0000: 5D 34 8D B8 34 CF 0B A0 64 05 20 93 E4 BA 92 28 ]4..4...d. ....(0010: CD EB 20 64 .. d]]

**************************************************************************************

14. Import the Gateway/InfoArchive web application certificate into the InfoArchive server:

> copy c:\ia40\infoarchive\config\webapp\https\gateway_public_cert.certc:\ia40\infoarchive\config\server\https> cd c:\ia40\infoarchive\config\server\https> keytool -import -noprompt -trustcacerts -alias IA_GATEWAY_CERT -filegateway_public_cert.cert -keystore serverTrustStore.jks -storepass abcdefg

The generated file is named serverTrustStore.jks.

15. Configure the InfoArchive server for HTTPS mode. Edit the C:\ia4.0\infoarchive\config\server\application-ssl.yml file and set its content to:

server:ssl:key-store: "file:config/server/https/serverKeyStore.jks"key-store-password: abcdefgkey-password: abcdefgkeyStoreType: JKSkeyAlias: ia_server_certtrust-store: "file:config/server/https/serverTrustStore.jks"trust-store-password: abcdefgtrust-store-type: JKSclient-auth: want

Note: client-auth: want is critical client-auth: need to mask the set inside the jar file.

16. Run the InfoArchive server in HTTPS mode:

> cd c:\ia40\infoarchive> .\bin\infoarchive-server-ssl.bat

17. Configure Gateway/InfoArchive web application for HTTPS mode:


profiles:active: infoarchive.profile.HTTPS,infoarchive.gateway.profile.AUTHENTICATION_IN_MEMORY


54

Configuration

infoarchive:gateway:host: localhostport: 8080contextPath: /token:secret: secret



addProxyHeaders: truehost:socket-timeout-millis: 60000


server:ssl:key-store: "file:config/webapp/https/gatewayKeyStore.jks"key-store-password: abcdefgkey-password: abcdefgkeyStoreType: JKStrust-store: "file:config/webapp/https/gatewayTrustStore.jks"trust-store-password: abcdefgtrust-store-type: JKS


18. Add the system properties to the bin\infoarchive-webapp.bat file:

:@rem Execute infoarchive-webappcd "%APP_HOME%""%JAVA_EXE%" -Djavax.net.ssl.trustStore=%APP_HOME%\config\webapp\https\gatewayKeyStore.jks -Djavax.net.ssl.trustStorePassword=abcdefg%DEFAULT_JVM_OPTS% %JAVA_OPTS% %INFOARCHIVE_WEBAPP_OPTS% -jar"%CLASSPATH%" %CMD_LINE_ARGS%:

19. Run Gateway/InfoArchive web application in HTTPS mode:

> cd c:\ia40\infoarchive> bin\infoarchive-webapp.bat

55

Configuration

Configuring a Sample ApplicationThe following example illustrates how to configure the PhoneCalls sample application from thedistribution package:

1. In the Tools > application> PhoneCalls folder of the InfoArchive 4.1 home location, run the antinstall receive ingest command from the terminal to configure the application andingest all of the SIPs.

2. Use IAShell to connect to the InfoArchive server.

3. Enter the use command (parameter values are always case-sensitive):

use --tenant INFOARCHIVE --application PhoneCalls

Creating an IAShell Start-Up ScriptCreating a start-up script will help avoid repetitive steps while starting IAShell:

1. Create a folder named .ia-shell in the home folder.

2. If using Windows, you cannot complete the following step via Windows Explorer. Complete thefollowing steps using the Windows command line tool:

a. In the home folder you created, press Shift + right mouse button and select Opencommand window here.

b. Enter the following mkdir command in the command window:

mkdir .ia-shell

3. In the folder created, create and save a text files. Use any file name.

4. In the file:

a. Store the commands used to connect to the InfoArchive server.

b. Add any other commands used to pre-configure IAShell.The following is a sample start-up script:

set-alias-properties aip --properties “id, name, dss.holdingName, sipSeqno,sipAiuCount, state, underHold, underRetention"set-alias-properties application --properties “name, type, archiveType,structuredDataStorageAllocationStrategy"set-alias-properties holding --properties “name, lastModifiedDate, deleteOnError,keepSipAfterIngest, xdbMode, priority, defaultRetentionClass"connect http://localhost:8765/services--user [email protected] --psw password--gateway http://localhost:8080use --tenant INFOARCHIVE --federation mainFederationuse --application PhoneCalls

56

Configuration

Note:• Place each command in a single line without line breaks.

• use /* and */ as begin and end characters for block comments.

• Be sure to ensure that the script works:

1. Use the exit command to exit it out of IAShell.

2. Run IAShell again. Verify that you are connected properly after start-up.

Performing a Byte Count on Application DataInfoArchive includes a standalone Java tool to calculate file size and character count for tables, XMLfiles or SIP.zip files in an application. The tool automatically chooses a unit (bytes, KB, MB, etc.) inwhich the smallest file has a value greater than 1. All files have the same unit for easy comparison.Formatting data in a file is not included in the final calculation.

Access the tool in the tools\metrics.bat directory.

The following parameters are required to perform a byte count:

Parameter Description

Use either -xml or -sip The -xml parameter is used to calculate the byte size of either anXML file or table.

The -sip parameter is used to calculate the byte size of a SIP.zipfile.

You must enter one of the above parameters.

<location> You must specify a directory or file that the byte count isperformed on.

-csv <csv-location> While it is not mandatory, you can generate a report that containsthe information returned in a byte count. The report is in .CSVformat. The -csv allows you to specify the name of the report.

Usage Examples

To perform a byte count on a table named BASEBALL-MASTER-01.xml that is stored in the Baseballapplication, the following command is used:

C:|projects\IA\infoarchive>tools\metrics.bat -xml applications/Baseball/tables/BASEBALL-MASTER-01.xml

When performing a byte count using the -xml parameter, the following information is returned:

57

Configuration

Column Description

File size Indicates the compiled data size of the table or XML file.

Data size Indicates the character count of the table or XML file.

To perform a byte count on all of the tables stored in the Baseball application, the following commandis used:

C:|projects\IA\infoarchive>tools\metrics.bat -xml applications/Baseball/tables

To perform a byte count on all of the tables stored in the Baseball application and generate a reportcalled baseball.csv, the following command is used:

C:|projects\IA\infoarchive>tools\metrics.bat -xml applications/Baseball/tables -csv baseball.csv

To perform a byte count on the SIP.zip files stored in the Trades application, the following commandis used:

C:|projects\IA\infoarchive>tools\metrics.bat -sip applications/Trades/sips

When performing a byte count using the -sip parameter, the following information is returned:

Column Description

File size Indicates the size of the eas.pdi.xml file.

Data size Indicates the character count of the eas.pdi.xml file.

Content size Indicates the byte count of any additional files (i.e., images)stored in the SIP.zip file.

Sip size Indicates the compiled data size of the SIP.zip file.

Setting Up CMD to Display National CharactersIn case a customer uses national symbols in SIP file names, the user should take care about how thenames are displayed in Command Line (Windows) or Terminal (Linux).

CLI and ANT work fine with any national symbols in a file name, but the command line can displaythem with errors or unrecognized symbols. The following example illustrates how to avoid suchconfusion.

If you want to display Russian symbols in the Windows CommandLine (cmd), the following shouldbe executed:

"chcp 1251"

The following settings are then set on Windows:

"Control Panel -> All Control Panel Items –> Region and Language - > Administrative

58

Configuration

tab - > Language for non-Unicode Programs" - "Russian (Russia)"

If you want to display Chinese symbols in the Windows Command Line (cmd), the followingshould be executed:

"chcp 936"

The following settings are then set on Windows:

"Control Panel -> All Control Panel Items –> Region and Language - > Administrativetab - > Language for non-Unicode Programs" - "Chinese (Simplified, RPC)"

For other localization, use the appropriate command line encoding and appropriate windowslocalization settings.

To use national symbols in the name of an application, holding, etc., the properties file for the ANTscript should be encoded with ISO 8859-1. For more information, refer to the Java properties encoding.

Configuring xDB Ingestion ModeWhen creating a holding, you can define the xDB ingestion mode. See the PhoneCallssample application ANT script, build.properties file. The xdbMode property in thebuild.properties file allows you to configure the xDB ingestion mode. The following values areacceptable: PRIVATE, POOLED, AGGREGATE.

Another way to set up the property value during ingestion is to execute an ANT command withan additional command-line parameter. The following example uses the ANT command in thePhoneCalls application:

Tools/application/PhoneCalls> ant -DxdbMode=AGGREGATE

Working with the application.yml FileThis section explains what you can and cannot update in the application.yml file.

The InfoArchive server has its own application.yml file and InfoArchive web application has itsown application.yml file. The application.yml file for the server is stored in the followingdirectory: <install_directory>\infoarchive\config\server folder. Further instructions about theindividual settings can be found in the server’s application.yml file, as well as the example below.

Security Profile

The data for the security profile can be updated.

spring:application:

59

Configuration

name: infoarchive.iasprofiles:include: infoarchive.ias.profile.JWT# include: infoarchive.ias.profile.HTTP_BASIC# include: infoarchive.ias.profile.NO_AUTHENTICATION

logging.level

The data for the logging.level can be updated.

logging.level.: 'WARN'

The server should typically be running in WARN level, as that limits the amount of informationlogged in the system (saves disk space, helps performance, etc.). If there are issues in the system, thelogging level should be raised (i.e., to DEBUG level). The InfoArchive server needs to be restartedfor this change to take effect.

60

Configuration

infoarchive Section

The data for the infoarchive section can be updated.

The value of infoarchive.gateway.token.secret should be exactly the same as it is set in the InfoArchiveweb application application.yml file. Both the InfoArchive server and the InfoArchive webapplication client need to have this value synchronized to the same value to ensure that JWT tokenscan be decrypted correctly.

The collections section controls the defaultPage. When displaying collections through pagination, youtypically start with first page, which is called the ’defaultPage’. Collections are zero-indexed, meaningthat the first page is at index ’0’. If, for some reason, you want to always start at the second page, setthe defaultPage to ’1’ and so on. For most environments, however. this value should be set to ’0’.

Collections are paginated. The value for defaultPageSize controls the size of the page. The followingexample indicates pages of 10 items (each). You can adjust this value, as needed.

Note: Modifying the defaultPageSize value impacts the real estate of the pages in the InfoArchiveweb application. This should be properly tuned to ensure all clients can handle a given page size.

The search.defaultTimeOutMs indicates the amount of time a synchronous search is allowed to runbefore the server will interrupt it and return an error. The value is in milliseconds (MS). In thefollowing example, the value of ’8000’ indicates a default time out of eight seconds. If the search doesnot complete, it will be interrupted in eight seconds and the server will return an error. To handlethis, the user should try re-running the search in asynchronous mode as a background search.

infoarchive:gateway:token:secret: secret

rest:collections:defaultPage: 0defaultPageSize: 10

search:defaultTimeOutMs: 8000

managedItemData Section

The data for the managedItemData section can be updated.

managedItemData:xdb:dataNode:storeStackTraceInLock: falsename: mainFederationbootstrap: xhive://localhost:8080superuser:password: test

database:name: managedItemDatabaseadmin:password: secret

61

Configuration

If upgrading to a new version of InfoArchive, change the bootstrap to a different port, and it isrecommended to change the name from mainFederation to retentionFederation.

defaultNames Section

The defaultNames value is for future use and should not be modified.

tenant:defaultNames:- INFOARCHIVE

Storage Paths and OAIS Sections

It is recommended that you do not change the storage paths or the OAIS sections.

storage:fileSystemRoots:-name: defaultFileSystemRootdescription: Default FileSystemRootpath: data/root

#locations on the IA Server's local file systemlocalStorage:tempFolder: data/temp

oais:defaultReceiverNode: receiver_node_01defaultIngestNode: ingest_node_01

job Section

Do not change the data in the job section. In particular, any change made to the readOnly flag forjobs will cause the jobs to malfunction.

job:definitions:

-name: CommithandlerName: CommitJobdescription: Commits packages in waiting commitreadOnly: falseapplicationScoped: falsetenantScoped: falsemaxAttempts: 1expirationInterval: 60000rescheduleInterval: 60000

62

Configuration

roles Section

Use caution when changing data in the authorization section. You may want to clear out the usersection for the production system, as the section includes the “canned” users.

Do not, however, change the data in the roles section.

authorization:users:- [email protected],password,GROUP_ADMINISTRATOR- [email protected],password,GROUP_BUSINESS_OWNER- [email protected],password,GROUP_DEVELOPER- [email protected],password,GROUP_END_USER- [email protected],password,GROUP_IT_OWNER- [email protected],password,GROUP_RETENTION_MANAGER- [email protected],password,GROUP_ADMINISTRATOR;GROUP_BUSINESS_OWNER;GROUP_DEVELOPER;GROUP_END_USER;GROUP_IT_OWNER;GROUP_RETENTION_MANAGER

groups:GROUP_ADMINISTRATOR:- ADMINISTRATORGROUP_BUSINESS_OWNER:- BUSINESS_OWNERGROUP_DEVELOPER:- DEVELOPERGROUP_END_USER:- END_USERGROUP_IT_OWNER:- IT_OWNERGROUP_RETENTION_MANAGER:- RETENTION_MANAGER

roles:ADMINISTRATOR:- VIEW_TENANT- CREATE_TENANT- DELETE_TENANT- VIEW_FEDERATIONS

Example

The following is an example of an application.yml for an InfoArchive web application client. Itindicates the values that can be changed.


profiles:active: infoarchive.gateway.profile.AUTHENTICATION_IN_MEMORY


infoarchive:gateway:host: localhostport: 8080contextPath: /

63

Configuration

token:secret: secret

webapp:customization:location: "file:config/webapp/customization/"



addProxyHeaders: truehost:socket-timeout-millis: 60000maxTotalConnections: 200maxPerRouteConnections: 20

SimpleHostRoutingFilter:route:disable: true

logging.level.: 'WARN'


server:ssl:key-store: "file:config/webapp/gatewayKeyStore.jks"key-store-password: abcdefgkey-password: abcdefgkeyStoreType: JKStrust-store: "file:config/webapp/gatewayTrustStore.jks"trust-store-password: abcdefgtrust-store-type: JKS

infoarchive:gateway:client:ssl:trust-store: "config/webapp/gatewayTrustStore.jks"trust-store-password: "abcdefg"trust-store-type: JKS


64

Configuration

Updating the Working Directory

Define the properties of the working directory in the localStorage entry of the application.ymlfile. Using the application.yml file not only allows you to easily set a different working directory,but also simplifies application configuration.

1. Before updating the working directory, stop all services.

2. Access the application.yml file in the infoarchive > config > server folder.The following information is displayed:

#locations on the LA Server's local file systemlocalStorage:..tempFolder: data/temp

3. Update the working directory, as desired and save your changes.

4. Restart the services.

Creating a File System Root

The InfoArchive server does not create a default file system root if nothing is configured. Theapplication.yml file, however, allows for a "fileSystemRoot" entry to be used by the InfoArchiveserver. The specified path has to pre-exist on the file system. InfoArchive server will not create it.

65

Configuration

66

Chapter 3Configuring a SIP Application

Listing AIPsIt is often convenient to use IAShell to check the status of AIPs.

The following procedure illustrates how a user would check the status of ingested AIPs in anapplication called Certificates. Because several tenants and applications might already exist, youmust first instruct IAShell which tenant and application you are interested in before you are ableto list the AIPs.

1. In IAShell, list the available tenants using the select command:

ia-shell>select tenant=================================================name id-------------------------------------------------INFOARCHIVE f36e88da-82c4-495a-bb48-563e5d90b681-------------------------------------------------Elements count: 1. Page 1 of 1ia-shell>

2. Take note of the name of the tenant.

3. Enter the use command to specify the tenant from the previous step.

Note: The use command can also specify an application, database, etc:

ia-shell>use --tenant INFOARCHIVEOKia-shell>

4. Enter the list command. A lot more types are available but not the AIP type.

5. Use the select command to list the available applications:ia-shell>select application

The previously installed Certificates application is displayed.

6. Enter the use command to instruct IAShell to use the Certificates application.

7. Enter the list command. The AIP type is now displayed.

Tip: If you will often be working with the same application, add these use commands to yourstartup script.

8. List the AIPs using the select command. Three columns are displayed (#, Name and ID).

9. Enter the view command to view the details of the first AIP:

67

Configuring a SIP Application

ia-shell>view --id 1==========================================name Certificates-CertApp-2012-03-05_001-1id a1844ef3-2141-4347-96ee-a579352b3d65------------------------------------------

By default, not a lot of details are displayed. To see more information, you can either

a. Instruct the view command to display the properties you want to see:

ia-shell>view --id 1 --properties "id,name,state"===========================================name Certificates-CertApp-2012-03-05_001-1state Completedid a1844ef3-2141-4347-96ee-a579352b3d65-------------------------------------------ia-shell>

b. Or use the set-alias-properties command to set the list of default properties:

ia-shell> set-alias-properties aip --properties “id, name, state"OKia-shell>view --id 1===========================================name Certificates-CertApp-2012-03-05_001-1state Completedid a1844ef3-2141-4347-96ee-a579352b3d65-------------------------------------------ia-shell>

Tip: As with the connect and use commands, it is beneficial to add sensible defaults to yourstartup script.

Ingesting a SIPThe following procedure illustrates how a Developer can ingest a SIP.

1. Connect to the IAShell. For this example, the Developer’s username is [email protected] her password is ’password’. The application the SIP is being ingested into is Certificates:

ia-shell>connect http://localhost:8080/services --user [email protected] --pswpassword --gateway http://localhost:8080Connected to 'http://localhost:8080/services'ia-shell>use --tenant INFOARCHIVE --application CertificatesOKia-shell>

2. Enter the use command to specify the tenant and application.

3. Execute the receive command on the SIP. For this example, the SIP is located in the${IA_HOME}/tools/applications/Certificates/sips folder.

ia-shell>receive --format sip_zip --from

68


tools/applications/Certificates/sips/cert_2012-03-05_001_1AIU.zipOKia-shell>

4. List the available AIPs entering the select command. The new AIP is displayed:ia-shell>select aip=============================================================================# name id-----------------------------------------------------------------------------1 Certificates-CertApp-2012-03-05_001-1 a1844ef3-2141-4347-96ee-a579352b3d652 Certificates-CertApp-2012-11-30_001-1 beccf795-1067-41d3-a023-71d45f7a8ea73 Certificates-CertApp-2012-03-05_001-1 350fa571-5aaf-490f-8e68-5a0a29ef7dff-----------------------------------------------------------------------------Elements count: 3. Page 1 of 1

5. View the id and state property of the third AIP entering the view command. The state is "Waitingingestion":

-----------------------------------------------------------------------------Elements count: 3. Page 1 of 1ia-shell>view --id 3 --properties "name,id,state"===========================================name Certificates-CertApp-2012-03-05_001-1state Waiting ingestionid 350fa571-5aaf-490f-8e68-5a0a29ef7dff-------------------------------------------

6. Enter the ingest command to ingest the AIP:

ia-shell>ingest --id 3OK

7. Enter the view command again to verify that the state is now Completed.

Building a SIP ZIPThe SIP ZIP is the container that is submitted to InfoArchive for archiving. The SIP ZIP is a ZIPfile that contains:

• At least one descriptor eas_sip.xml (SIP); and

• One XML file that contains the records eas_pdi.xml (PDI).

If the metadata is associated to some content (CI), the content must be put in the SIP ZIP containerat the root level.

Configuring a SIP Application with One HoldingThis section illustrates how to configure a new SIP application with one holding. By the end of thesection, you will be able to ingest data, perform a search and retrieve a content from the InfoArchiveweb application.

69


Before you begin, you need to:

• Retrieve and to install the latest InfoArchive build.

• Know what you want to archive and to have defined and generated an XML schema.

• Know what you want to search

The following example procedure uses a fictitious holding named Tweets, which contains tweetsfrom Twitter. Each tweet contains some metadata (text, screen name, date, etc.) and can be associatedwith 0 to n entities (URL, photo, hashtag, etc.).

The following is the PDI used in the scenario:

<tweets xmlns="urn:x-emc:ia:demo:schema:tweets:1.0"><tweet>

<id>731398089370669056</id><text>RT @EMCbigdata: Recorded live at #EMCWORLD: a #BigData podcast

with our partner @Splunk via #EMCElect https://t.co/FIl4e4mmgFhttps://t.co/…</text>

<createdAt>2016-05-14T10:17:47+02:00</createdAt><retweet>true</retweet><retweetCount>7</retweetCount><user>

<id>729669013593333760</id><name>Ruth Lutterloch</name><location>London, England</location><screenName>RuthLutterloch</screenName>

</user><mediaEntity>

<id>730819244519264256</id><url>http://pbs.twimg.com/media/CiRkOxiXEAAAuEh.jpg</url><type>photo</type><start>126</start><end>140</end>

</mediaEntity><urlEntity>

<text>https://t.co/FIl4e4mmgF</text><start>102</start><end>125</end><url>https://t.co/FIl4e4mmgF</url><expandedUrl>http://emc.im/6012Boc5k</expandedUrl><displayUrl>http://emc.im/6012Boc5k</displayUrl>

</urlEntity><hashtagEntity>

<text>EMCWORLD</text><start>33</start><end>42</end>

</hashtagEntity><hashtagEntity>

<text>BigData</text><start>46</start><end>54</end>

</hashtagEntity><hashtagEntity>

<text>EMCElect</text><start>92</start><end>101</end>

</hashtagEntity></tweet>

</tweets>

The following is the PDI schema used in the scenario:

70


<?xml version="1.0" encoding="UTF-8"?><xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified" targetNamespace="urn:x-emc:ia:demo:schema:tweets:1.0"xmlns:ns1="urn:x-emc:ia:demo:schema:tweets:1.0"><xs:element name="tweets"><xs:complexType><xs:sequence><xs:element maxOccurs="unbounded" ref="ns1:tweet"/>

</xs:sequence></xs:complexType>

</xs:element><xs:element name="tweet"><xs:complexType><xs:sequence><xs:element ref="ns1:id"/><xs:element ref="ns1:text"/><xs:element ref="ns1:createdAt"/><xs:element ref="ns1:retweet"/><xs:element ref="ns1:retweetCount"/><xs:element ref="ns1:user"/><xs:element minOccurs="0" maxOccurs="unbounded" ref="ns1:mediaEntity"/><xs:element minOccurs="0" maxOccurs="unbounded" ref="ns1:urlEntity"/><xs:element minOccurs="0" maxOccurs="unbounded" ref="ns1:hashtagEntity"/>


</xs:element><xs:element name="createdAt" type="xs:dateTime"/><xs:element name="retweet" type="xs:boolean"/><xs:element name="retweetCount" type="xs:integer"/><xs:element name="user"><xs:complexType><xs:sequence><xs:element ref="ns1:id"/><xs:element ref="ns1:name"/><xs:element ref="ns1:location"/><xs:element ref="ns1:screenName"/>


</xs:element><xs:element name="name" type="xs:string"/><xs:element name="location" type="xs:string"/><xs:element name="screenName" type="xs:NCName"/><xs:element name="mediaEntity"><xs:complexType><xs:sequence><xs:element ref="ns1:id"/><xs:element ref="ns1:url"/><xs:element ref="ns1:type"/><xs:element ref="ns1:start"/><xs:element ref="ns1:end"/>


</xs:element><xs:element name="type" type="xs:NCName"/><xs:element name="urlEntity"><xs:complexType><xs:sequence><xs:element ref="ns1:text"/><xs:element ref="ns1:start"/><xs:element ref="ns1:end"/><xs:element ref="ns1:url"/><xs:element ref="ns1:expandedUrl"/><xs:element ref="ns1:displayUrl"/>


71


</xs:element><xs:element name="expandedUrl" type="xs:anyURI"/><xs:element name="displayUrl" type="xs:anyURI"/><xs:element name="hashtagEntity"><xs:complexType><xs:sequence><xs:element ref="ns1:text"/><xs:element ref="ns1:start"/><xs:element ref="ns1:end"/>


</xs:element><xs:element name="id" type="xs:integer"/><xs:element name="text" type="xs:string"/><xs:element name="url" type="xs:anyURI"/><xs:element name="start" type="xs:integer"/><xs:element name="end" type="xs:integer"/>

</xs:schema>

1. Prepare your environment by completing the following:

a. Unzip the PhoneCalls template into the $IA_HOME/tools/applications directory.

b. Rename the folder to ${holding}

c. In the build.properties file, update the application, holding, pdiSchema, category anddescription with your own values:

# -------------------------------------------------------------------# -- EMC InfoArchive - PhoneCalls installation properties# -- Copyright (c) 2012-2015 EMC Corporation. All Rights Reserved.# -------------------------------------------------------------------services=http://localhost:8080/services

xdbFederation=mainFederationxdbDatabase=AIP-xdb

tenant=INFOARCHIVEapplication=Twittercategory=Social Mediadescription=Some tweets history

holding=TweetspdiSchema=urn:x-emc:ia:demo:schema:tweets:1.0

storeName=filestore_01

# Change xDB mode to PRIVATE or AGGREGATExdbMode=PRIVATE

# Change the store name to switch to ECS / CAS or File Store# Go to portal.ecstestdrive.com to obtain a free account accessstoreName=file_store_01

# File StorefolderStoreName=file_store_01

# ECS StoreecsStoreName=ecs_store_01ecsCredentialName=ecsAccessKey=ecsSecretKey=ecsEndPointName=testEndPoint

72


ecsUrl=ecsBucketName=PhoneCalls

# CAS StorecasStoreName=cas_store_01casConnexionString=cas.ecstestdrive.com?casPeaContent1casPeaContent1=<pea version="1.0.0"> ... </pea>

# -------------------------------------------------------------------# -- Other parameters, only change if needed.# -------------------------------------------------------------------federationPassword=testxdbPassword=secretfederationBootstrap=xhive://127.0.0.1:2910

receiverNodeName=receiver_node_01ingestNodeName=ingest_node_01

# Location of the InfoArchive tools directoryiaToolsHome=../..

## Authentication.## Specify only the token when using the JWT authentication type.# Specify only username and password when using the HTTP_BASICauthentication type.# Specify gateway when using gateway server for authentication## Token of superuser Suetoken=eyJhbGciOiJIUzI1NiJ9.eyJleHAiOjM2MTExMzM0MzQsInVzZXJfbmFtZSI6InN1ZUBpYWN1c3RvbWVyLmNvbSIsImF1dGhvcml0aWVzIjpbIkdST1VQX0FETUlOSVNUUkFUT1IiLCJHUk9VUF9SRVRFTlRJT05fTUFOQUdFUiIsIkdST1VQX0JVU0lORVNTX09XTkVSIiwiR1JPVVBfRU5EX1VTRVIiLCJHUk9VUF9ERVZFTE9QRVIiLCJHUk9VUF9JVF9PV05FUiIsIlJPTEVfQURNSU5JU1RSQVRPUiIsIlJPTEVfUkVURU5USU9OX01BTkFHRVIiLCJST0xFX0JVU0lORVNTX09XTkVSIiwiUk9MRV9FTkRfVVNFUiIsIlJPTEVfREVWRUxPUEVSIiwiUk9MRV9JVF9PV05FUiJdLCJqdGkiOiJiMDM5YWNmOC01MzE1LTRlNzktOTU3My04ZjU5NTdiYmU1YmYiLCJjbGllbnRfaWQiOiJpbmZvYXJjaGl2ZS5pYXdhIiwic2NvcGUiOlsiYWRtaW5pc3RyYXRpb24iLCJjb21wbGlhbmNlIiwic2VhcmNoIl19.boOdzl4sotMRDraX3WSDw1F7mmyLRx3kPOK5tFCHJUE#gateway = http://localhost:8080#username = [email protected]#password = password

2. Replace the $IA_HOME/tools/applications/${holding}/resources/content/pdi-schema.xsdwith the new XSD file.

3. Edit the $IA_HOME/tools/applications/${holding}/resources/content/pdi.xml file to configurethe ingestion.This XML file contains the configuration that will be used during the ingestion process. In thisfile, you can:• Configure the xDB indexes to put on the PDI XML file;

• Extract the partition keys;

• Count the number of AIUs in the PDI; and

• Define where to find the CI.

4. Create some indexes at the xDB level to improve the search and ingestion performance.

73

5. Edit the $IA_HOME/tools/applications/${holding}/resources/250-transformation.xml file toconfigure the transformation.This object describes how you want to transform the PDI XML when you want to generate ananalytic rendition and/or a disposition export.In the following example of the 250-transformation.xml file, we indicate the XQueryto execute to generate the format rendition.csv.gzip when we request the result schema :urn:x-emc:ia:schema:pdi:export:

<echo>Configuring transformation object with name '${holding}-transformation-csv'</echo><ia-task self="transformationSelf">

<object typeAlias="transformation"checkExistSpEL="?[name=='${holding}-transformation-csv']">

<create><name>${holding}-transformation-csv</name>

</create><update>

<inputSchema>${pdiSchema}</inputSchema><inputSchema/><resultSchema>urn:x-emc:ia:schema:pdi:export</resultSchema><format>rendition.csv.gzip</format><type>XQUERY</type><xquery>

<![CDATA[declare namespace n = "${pdiSchema}";

declare function local:replace($i as xs:string?) {if ($i) then '"' || replace($i,'"','""') || '"' else '""'

};

declare function local:add-row($input) {string-join($input,',') || '

'};

((: Header :)local:add-row((

'createdAt','userName','userScreenName','text','hashtags'

)),(: Rows :)for $call in /n:tweets/n:tweetreturn

local:add-row((local:replace($call/n:createdAt),local:replace($call/n:userName),local:replace($call/n:userScreenName),local:replace($call/n:text),local:replace(string-join($call/n:hashtagEntity/n:text,','))))

)]]>

</xquery></update>

</object></ia-task>

74

Before disposing the AIP, it is possible to export the metadata with a background export function.This option is available if one transformation object with the "urn:x-emc:ia:schema:pdi:export" resultschema is found.

To generate an analytic rendition during the ingestion, add the following configuration into thePDI and to specify the desired result schema:

<data><id>pdi.transformer</id><result.schema>urn:x-emc:ia:schema:pdi:export</result.schema>

</data>

If you do not need these features, remove the PDI configuration part and comment the250-transformation.xml script.

Search

Edit the $IA_HOME/tools/applications/${holding}/resources/220-query.xml file to configurethe query

This object contains the mapping between the search critieria and the xDB PDI XML file. Enumerateall operands and indicate the criterion name, the path, the type and whether it is indexed or not. Theallowed types are : STRING, INTEGER, LONG, DOUBLE, DATE, DATETIME, ID and CID.

To list the operands, specify the entityPath. This path corresponds to the AIU level. All others pathsare defined from this level.

<echo>Configuring query object with name '${holding}-query'</echo><ia-task self="querySelf">

<object typeAlias="query" checkExistSpEL="?[name=='${holding}-query']"><create>

<name>${holding}-query</name></create><update>

<resultSchema>${pdiSchema}</resultSchema><resultRootElement>result</resultRootElement><quota>${qqSelf}</quota><quotaAsync>${qqSelf}</quotaAsync><aics>${aicSelf}</aics><aics/><order>${orderSelf}</order><namespaces>

<uri>${pdiSchema}</uri><prefix>n</prefix>

</namespaces><namespaces/><xdbPdiConfigs>

<schema>${pdiSchema}</schema><entityPath>/n:tweets/n:tweet</entityPath><operands>

<name>createdAt</name><path>n:createdAt</path><type>DATETIME</type><index>true</index>

</operands>

75

<operands><name>userName</name><path>n:user/n:name</path><type>STRING</type><index>true</index>

</operands><operands>

<name>userScreenName</name><path>n:user/n:screenName</path><type>STRING</type><index>true</index>


<name>hashtag</name><path>n:hashtagEntity/n:text</path><type>STRING</type><index>true</index>


<name>text</name><path>n:text</path><type>STRING</type><index>true</index>


<name>retweetCount</name><path>n:retweetCount</path><type>INTEGER</type><index>true</index>

</operands></xdbPdiConfigs><xdbPdiConfigs/>

</update></object>

</ia-task>

For the AIC, edit the $IA_HOME/tools/applications/${holding}/resources/190-aic.xml file toconfigure the query

This object exposes the list of allowed search criteria. For each criterion, specify the type and if thecriterion is a partition key with the name of the attributes to use on the AIP.

• The allowed types are : STRING, INTEGER, LONG, DOUBLE, DATE, DATETIME, ID and CID.The type is used to validate the submitted values.

• If you have defined a minimum/maximum partition key, populate the following fields:pKeyMinAttr / pKeyMaxAttr.

• If you have defined a single or multi-values partition key, populate the following field:pKeyValuesAttr.

<echo>Configuring aic object with name '${holding}-aic'</echo><ia-task self="aicSelf">

<object typeAlias="aic" checkExistSpEL="?[name=='${holding}-aic']"><create>

<name>${holding}-aic</name></create><update>

<holdings>${holdingSelf}</holdings><holdings/><criterias>

<name>createdAt</name>

76


<label>Created at</label><type>DATETIME</type><pKeyMinAttr>pkeys.dateTime01</pKeyMinAttr><pKeyMaxAttr>pkeys.dateTime02</pKeyMaxAttr>

</criterias><criterias>

<name>userName</name><label>User name</label><type>STRING</type>


<name>userScreenName</name><label>User screenName</label><type>STRING</type>


<name>hashtag</name><label>Hashtag</label><type>STRING</type>


<name>text</name><label>Text</label><type>STRING</type>


<name>retweetCount</name><label>Retweet count</label><type>INTEGER</type>

</criterias></update>

</object></ia-task>

It is not necessary to list the ID criterion, as this criterion is automatically added at the run-time.

For the Result Configuration Helper, edit the $IA_HOME/tools/applications/${holding}/resources/content/result-configuration-helper.xml file. This file is attached as a content with theSDX object ResultConfigurationHelper.

This XML describes the XML returned by the query. This description will be used during thecomposition to help the Developer to select the column to display into the result page.

Add the namespaces prefix at the root level.

For each column you want to expose, add an <element/> node. The element needs to contain a name,a label, a type and a path. The allowed types are : STRING, INTEGER, LONG, DOUBLE, DATE,DATETIME, ID and CID. You can write the XPath you want. It can be useful to perform some valuemassage (string-join, duration computation, etc.).

For repeating elements, enclose with a <group/> node. The group needs to contain a name, a labeland a path.

Note: Do not forget to include the CID to have the ability to download CIs.

<?xml version="1.0" encoding="UTF-8"?><resultConfigurationHelper xmlns:n="urn:x-emc:ia:demo:schema:tweets:1.0" xmlns:ia="urn:x-emc:ia:schema:pdi"><element><name>id</name><label>ID</label><type>LONG</type><path>n:id</path>

77


</element><element><name>text</name><label>Text</label><type>STRING</type><path>n:text</path>

</element><element><name>createdAt</name><label>Created at</label><type>DATETIME</type><path>n:createdAt</path>

</element><element><name>retweet</name><label>Retweet</label><type>STRING</type><path>n:retweet</path>

</element><element><name>retweetCount</name><label>Retweet count</label><type>INTEGER</type><path>n:retweetCount</path>

</element><element><name>userId</name><label>User ID</label><type>LONG</type><path>n:user/n:id</path>

</element><element><name>userName</name><label>User name</label><type>STRING</type><path>n:user/n:name</path>

</element><element><name>userLocation</name><label>Location</label><type>STRING</type><path>n:user/n:location</path>

</element><element><name>userScreenName</name><label>Screen name</label><type>STRING</type><path>n:user/n:screenName</path>

</element><element><name>Media1</name><label>Media 1</label><type>CID</type><path>n:mediaEntity[1]/n:id/@ia:cid</path>

</element><group><name>MediaEntities</name><label>Medias</label><path>n:mediaEntity</path><element><name>cid</name><label>CID</label><type>CID</type><path>n:id/@ia:cid</path>

</element>

78


<element><name>mediaId</name><label>ID</label><type>LONG</type><path>n:id</path>

</element><element><name>mediaUrl</name><label>URL</label><type>STRING</type><path>n:url</path>

</element><element><name>mediaType</name><label>Type</label><type>STRING</type><path>n:type</path>

</element><element><name>mediaStart</name><label>Start</label><type>INTEGER</type><path>n:start</path>

</element><element><name>mediaEnd</name><label>End</label><type>INTEGER</type><path>n:end</path>

</element></group><group><name>urlEntities</name><label>URLs</label><path>n:urlEntity</path><element><name>urlText</name><label>Text</label><type>STRING</type><path>n:url</path>

</element><element><name>urlUrl</name><label>URL</label><type>STRING</type><path>n:url</path>

</element><element><name>urlStart</name><label>Start</label><type>INTEGER</type><path>n:start</path>

</element><element><name>urlEnd</name><label>End</label><type>INTEGER</type><path>n:end</path>

</element><element><name>expandedUrl</name><label>Expanded url</label><type>STRING</type><path>n:expandedUrl</path>

</element>

79


<element><name>displayUrl</name><label>Display url</label><type>STRING</type><path>n:displayUrl</path>

</element></group><group><name>hashtagEntities</name><label>Hashtags</label><path>n:hashtagEntity</path><element><name>hashtagText</name><label>Text</label><type>STRING</type><path>n:url</path>

</element><element><name>hashtagStart</name><label>Start</label><type>INTEGER</type><path>n:start</path>

</element><element><name>hashtagEnd</name><label>End</label><type>INTEGER</type><path>n:end</path>

</element></group>

</resultConfigurationHelper>

PDI.INDEX.CREATOR

Create an xDB index on each search criterion. In the majority of use cases, the path value index isthe best candidate. A full-text index can be defined if you want to use a contains operator and/orcase insensitive search.Is “contains operator” correct?

The full-text index is less efficient than a path value index.

The AIU path is followed by the element path you want to search with the type (STRING, INTEGER,LONG, DOUBLE, DATE, DATE_TIME). Do not forget to include the between bracket. For example:

/{urn:x-emc:ia:demo:schema:tweets:1.0}tweets/{urn:x-emc:ia:demo:schema:tweets:1.0}tweet[{urn:x-emc:ia:demo:schema:tweets:1.0}createdAt<DATE_TIME>]

The folloiwng is an example of the pdi.index.creator file:

<data><id>pdi.index.creator</id><key.document.name>xdb.pdi.name</key.document.name><indexes>

<path.value.index><name>createdAt</name><path>

/{urn:x-emc:ia:demo:schema:tweets:1.0}tweets/{urn:x-emc:ia:demo:schema:tweets:1.0}tweet[{urn:x-emc:ia:demo:schema:tweets:1.0}createdAt<DATE_TIME>]

80


</path><compressed>false</compressed><unique.keys>false</unique.keys><concurrent>false</concurrent><build.without.logging>false</build.without.logging>

</path.value.index><path.value.index>

<name>userName</name><path>

/{urn:x-emc:ia:demo:schema:tweets:1.0}tweets/{urn:x-emc:ia:demo:schema:tweets:1.0}tweet[{urn:x-emc:ia:demo:schema:tweets:1.0}user/{urn:x-emc:ia:demo:schema:tweets:1.0}name<STRING>]



<name>userScreenName</name><path>

/{urn:x-emc:ia:demo:schema:tweets:1.0}tweets/{urn:x-emc:ia:demo:schema:tweets:1.0}tweet[{urn:x-emc:ia:demo:schema:tweets:1.0}user/{urn:x-emc:ia:demo:schema:tweets:1.0}screenName<STRING>]



<name>hashtag</name><path>

/{urn:x-emc:ia:demo:schema:tweets:1.0}tweets/{urn:x-emc:ia:demo:schema:tweets:1.0}tweet[{urn:x-emc:ia:demo:schema:tweets:1.0}hashtagEntity/{urn:x-emc:ia:demo:schema:tweets:1.0}text<STRING>]



<name>retweetCount</name><path>

/{urn:x-emc:ia:demo:schema:tweets:1.0}tweets/{urn:x-emc:ia:demo:schema:tweets:1.0}tweet[{urn:x-emc:ia:demo:schema:tweets:1.0}retweetCount<INTEGER>]


</path.value.index><full.text.index>

<name>text-fulltext</name><compressed>false</compressed><concurrent>false</concurrent><optimize.leading.wildcard.search>true</optimize.leading.

wildcard.search><index.all.text>true</index.all.text><include.attributes>false</include.attributes><support.phrases>false</support.phrases><support.scoring>false</support.scoring><convert.terms.to.lowercase>true</convert.terms.to.lowercase>

81


<filter.english.stop.words>false</filter.english.stop.words><support.start.end.token.flags>false</support.start.end.token.

flags><element.uri>urn:x-emc:ia:demo:schema:tweets:1.0</element.uri><element.name>text</element.name><attribute.uri/><attribute.name/>

</full.text.index></indexes>

</data>

Note: It is recommended that you do not create indexes with the option build.without.loggingvalue set to true.

PDI.AIU.CNT

To ensure that the SIP contains the number of AIUs defined in the SIP descriptor, it is necessary tocount the AIUs into the PDI XML. To accomplish this, an XQuery is executed. Based on the AIU path,it is easy to count the AIUs. Do not forget to indicate the namespace declaration and prefix.

The following is an example of the pdi.aiu.cnt file:

<data><id>pdi.aiu.cnt</id><select.query>

<![CDATA[declare namespace n = "urn:x-emc:ia:demo:schema:tweets:1.0";count(/n:tweets/n:tweet)]]>

</select.query></data>

This query is valid but it does not use indexes, which could be an issue with large PDI XML. Touse an index, include an indexed element into the XQuery. The element must be present for eachAIU, choose the element carefully.

The following is an example of an optimized pdi.aiu.cnt file:

<data><id>pdi.aiu.cnt</id><select.query>

<![CDATA[declare namespace n = "urn:x-emc:ia:demo:schema:tweets:1.0";count(/n:tweets/n:tweet[n:createdAt])]]>


PDI.AIU.ID

To identify each AIU, it is necessary to generate an ID for each. To do that, an XQuery is executed toreturn all of the AIU nodes. Based on this list, an attribute ID will be added with the corresponding

82


AIU ID value. The AIU ID is composed by the AIP ID and the position into the list with this format.For example:

${AIP_ID}:aiu:${position}

The following is an example of the pdi.aiu.id file:

<data><id>pdi.aiu.id</id><select.query>

<![CDATA[declare namespace n = "urn:x-emc:ia:demo:schema:tweets:1.0";for $aiu in /n:tweets/n:tweetreturn $aiu]]>


PDI.PKEYS

To improve the search performance and reduce the search scope, define one or more partition keys.When a search criterion is linked to a partition key, perform the query only on AIPs that referencethe partition key value.

Each partition key value must be assigned to an AIP attribute. Out of the box, the AIP offers somefree slots. You need to take into account the type (STRING, INTEGER, DOUBLE, DATETIME,LIST<STRING>).

There are three different types of partition keys:

STRING INTEGER LONG DOUBLE DATE-TIME

LIST<STRING>

attribute string0x integer0x long0x double0x dateTime0x values0x

Min / Max Yes Yes Yes Yes Yes No

One value Yes Yes Yes Yes Yes No

Value list No No No No No Yes

The common configuration is to calculate the minimum/maximum for a dateTime. In the followingexample from the pdi.pkeys file, it is the createdAt value. We will set the minimum value todateTime01 and the maximum value to dateTime02:

<data><id>pdi.pkeys</id><pkey attr="dateTime01">

declare namespace n = "urn:x-emc:ia:demo:schema:tweets:1.0";min(/n:tweets/n:tweet/xs:dateTime(n:createdAt))

</pkey><pkey attr="dateTime02">

declare namespace n = "urn:x-emc:ia:demo:schema:tweets:1.0";max(/n:tweets/n:tweet/xs:dateTime(n:createdAt))

</pkey>

83


</data>

To use the xDB indexes and improve the performance with large PDI XML files, you can write adifferent XQuery. It is important to know how xDB indexes are sorted. In the following example, ifyou pick-up the first value of the index, you have the minimum value:

<data><id>pdi.pkeys</id><pkey attr="dateTime01">declare namespace n = "urn:x-emc:ia:demo:schema:tweets:1.0";(for $aiu in /n:tweets/n:tweet[n:createdAt]order by $aiu/n:createdAt ascendingreturn $aiu/n:createdAt/xs:dateTime(.))[1]

</pkey><pkey attr="dateTime02">declare namespace n = "urn:x-emc:ia:demo:schema:tweets:1.0";(for $aiu in /n:tweets/n:tweet[n:createdAt]order by $aiu/n:createdAt descendingreturn $aiu/n:createdAt/xs:dateTime(.))[1]

</pkey></data>

RI.INIT

The RI.INIT XQuery allows you to enumerate the unstructured content included into the SIP. Theapproach is to retrieve all file names into the PDI XML file and to return a list of <content/> elements.For each content, specify the mime type. This mime type will be returned during the content retrieval.The mime type can be static or retrieved one by one from the PDI XML.

If one content is linked to more than one AIU, use the distinct-values function to avoid archivinga copy of the same content.

For the Tweets holding, the file name corresponds to the ID value of the mediaEntity:/n:tweets/n:tweet/n:mediaEntity/n:id

The following is an example of the ri.init file:

<data><id>ri.init</id><select.query>

<![CDATA[declare namespace n = "urn:x-emc:ia:demo:schema:tweets:1.0";for $ci in distinct-values(/n:tweets/n:tweet/n:mediaEntity/n:id)order by $cireturn <content type="application/octet-stream">{ $ci }</content>]]>


84


XDB.PDI.CI.ID

To facilitate the content retrieval, enrich the PDI XML with the Content ID (CID). To do that, oneXQuery is used to return the nodes where we want to add the CID value. In return, each node mustbe followed by a string value corresponding to the CID. To support all xDB modes, compute the CIDwith the help of two external variables ID and SEQNO_START.

<data><id>xdb.pdi.ci.id</id><select.query>

<![CDATA[declare variable $id as xs:string external;declare variable $seqno_start as xs:long external;declare namespace n = "urn:x-emc:ia:demo:schema:tweets:1.0";declare namespace ri = "urn:x-emc:ia:schema:ri";let $pdi_uri := root(.)let $ri_uri := replace(document-uri($pdi_uri), '\.pdi$', '.ri')for $ri in doc($ri_uri)/ri:ris/ri:ri[@key]for $n in /n:tweets/n:tweet/n:mediaEntity/n:id[. = $ri/@key]return ($n,concat($id,":ci:", sum(($ri/@seqno,$seqno_start))))

]]></select.query>

</data>

If the SIP contains a lot of unstructured content, it is possible to improve the performance by creatingan index on the element containing the file name.

<path.value.index><name>filename</name><path>

/{urn:x-emc:ia:demo:schema:tweets:1.0}tweets/{urn:x-emc:ia:demo:schema:tweets:1.0}tweet/{urn:x-emc:ia:demo:schema:tweets:1.0}mediaEntity/{urn:x-emc:ia:demo:schema:tweets:1.0}id<STRING>


</path.value.index>

Configuration Objects

Name Description

AIP An AIP is a OAIS definition standing for Archival InformationPackage. This object represent the archive created after the reception.

Holding A holding is a logical destination archive where to ingest and storedata, usually of the same type that share common characteristics.

Confirmation This object defines the confirmation messages to generate in reactionto an AIP event.

85


Name Description

Delivery Channel This object defines a destination where to write the confirmationmessage.

Ingest This object contains the list of processor to execute during theingestion.

Order This object is used for background searches

Order Item This object is used for background searches

PDI This object contains the configuration to apply during the ingestion.

PDI Schema This object contains the XML Schema to use to validate a PDI XMLfile during the ingestion

Transformation This object defines the XQuery/XSLT to use to perform atransformation.

Ingest Node This object defines parameters for the ingestion and the enumerationprocesses.

Receiver Node This object defines parameters for the reception process.

Xdb Library Policy This object defines the rules how to manage the xDB libraries.

Search

Search Search object for either a table search or a sip search.

SearchComposition Manage the search components (xform, result master, search) andthe search execution.

XForm The search form.

ResultMaster This object describes the search result page for the UI.

Result Configuration Helper This object describes the result output to help the searchcomposition.

AIC An AIC organizes a set of AIPs, which can support flexible andefficient data access.

Query This object defines every option needed to build a query to retrieveAIU.

Query Quota This object defines the search quota to apply during the search.

86


Name Description

Core

Tenant

Application

Space

Storage

Space Root xDB Library

xDB Library

xDB Database

xDB Federation

Space Root Folder

File System Root

File System Folder

Store

Content

AIP

An AIP is a OAIS definition that stands for Archival Information Package. This object representthe archive.

The following table provides OAIS definitions used in the description of an AIP. An ArchiveInformation Unit (AIU) is a variant of a SIP, AIP or DIP.

Content Information (CI) is a set of information that is

• The original target of preservation; or

• That includes part or all of that information.

A CI is an Information Object composed of its Content Data Object and its RepresentationInformation. To resume it is the unstructured information.

Reference Information (RI) identifies and, if necessary, describes one or more mechanisms used toprovide assigned identifiers for the CI. It also provides those identifiers that allow outside systemsto refer to this particular CI.

A Data Submission Session (DSS) is a delivery of media or a single telecommunications sessionthat provides data to a consumer. The DSS format/content is based on a data model negotiatedbetween OAIS and the consumer in the request agreement. In SIP archiving, the DSS is similarof the notion of collection.

Field Description

Summary

87


Field Description

id Type: UUID

Mandatory: Yes

Identifier of the AIP. This attributes allows thecustomer to:

• Optimize searches, since a unique index on thisproperty is created at the RDBMS level.

• Ensure this identifier does not vary, even if theDocumentum object identifier changes.

name Type: String

Label: Name

The name is composed with dss.HoldingName+ dssProducer + dssId + sipSeqno (i.e.,PhoneCalls-CC-1000000-1).

’dss’ stands for Data Submission Session, which is adelivered set of media or a single telecommunicationssession that provides data to an OAIS. The DataSubmission Session format/content is based on a datamodel negotiated between the OAIS and the producerin the Submission Agreement. This data modelidentifies the logical constructs used by the producerand how they are represented on each media deliveryor in the telecommunication session.

application Type: Application

Label: Application name

The application corresponding to the AIP

aipId Type: String

Label: AIP ID

Ensures this identifier does not vary, even if the objectidentifier changes.

sipAiuCount Type: Long

Label: AIU count

Number of AIUs contained in the AIP.

88


Field Description

ciCount Type: Long

Label: CI count of unstructured content files

Number of content files referenced in the metadataof the AIP.

pdiFileSize Type: Long

Label: PDI file size

Size in bytes of the uncompressed PDI XML filereceived in the SIP.

pdiValuesCharCount Type: Long

Label: PDI character count

Cumulated number of characters contained in theelement and attribute values of the PDI XML filereceived in the SIP.

ciSize Type: Long

Label: CI size (the cumulated size of the unstructuredcontent files)

Cummulative CI size.

ciFormat Type: String

Label: CI format

Name of the format that is used to store the CI of theAIP (i.e., "ci_container").

riRecordSize Type: Long

Label: RI record size

phaseCode Type: ENUM

Label: Phase

The labels for each status include:

• NONE = None

• REC = Reception

• WING = Waiting Ingestion

• ING = Ingest

• WCOM = Waiting Commit

89


Field Description

• COM = Completed

• PUR = Purge

• REJ = Reject

• INV = Invalid

• AGGR = Aggregate

stateCode Type: ENUM

Label: State

The current state of the AIP. States are sub-elementsof the phase. For example, an AIP in the invalidationphase (INV) can be in the following states:INV_WPROC (Waiting for processing), INV_WDEL(Waiting deletion), or INV_DONE (Invalidation done).

The labels for each status include:

• NONE = None

• REC_VAL = Validation

• REC_HOLD_BIND = Holding binding

• REC_RET = Retention

• REC_IMP = Import

• REC_REJ = Rejection

• WING = Waiting ingestion

• ING_INIT = Initialization

• ING_SIP_DWN = Sip downloader

• ING_SIP_EXT = Sip extractor

• ING_PDI_HASH_FILE = Pdi hash validation

• ING_PDI_UNZIP = Decompress metadata

• ING_PDI_HASH = Validate hash metadata

• ING_PDI_IMPORT = Import metadata

• ING_PDI_QUERY = Query metadata

• ING_PDI_IDX = Create metadata index

90


Field Description

• ING_RI_INIT = RI creation

• ING_CI_UNZIP = Decompress content

• ING_CI_HASH = Validate content hash

• ING_CI_ZIP = Compress content

• ING_CI_AGGREGATE = CI aggregator

• ING_IMPORT = Import

• ING_LIBRARY_CREATOR = Create xdb library

• ING_SIP_XDB_IMPORT = Import sip xml to xdb

• ING_INDEXER_CREATOR = Create index in xdb

• ING_CONTENTS_UPLOADER = Contentsuploader

• ING_AIUID = Add AIU ids

• ING_PDI_AIU_COUNT = PDI AIU count

• ING_PART_KEYS = Populate values of partitionkeys

• ING_XDB_BACKUP = Xdb backup

• ING_PDI_TRANSFORMER = Pdi Transformer

• WCOM = Waiting commit

• COM = Completed

• INV_WPROC = Waiting for processing

• INV_WDEL = Waiting deletion

• INV_DONE = Invalidation done

• REJ_WPROC = Waiting for processing

• REJ_WDEL = Waiting deletion

• REJ_DONE = Rejection done

• PUR_WCONF = Waiting confirmation

• PUR_WDEL = Waiting deletion

• PUR_WPROC = Waiting processing

91


Field Description

• AGGR_OPEN = Open aggregate

partOfAggregate Type: boolean

Label: AIP part of an aggregate

Indicator to set if the AIP is a part of an aggregate.Only set to ’true’ for the ’child’ AIP (the temporaryAIP that will disappear later).

aggregateCiSeqno Type: int

Label: AIP aggregate CI seqno

Internal property temporarily used for the AIPsinvolved in an aggregation.• For children, it holds the number of the firstCiSeqno. This property allows to serve contentretrieval requests on the contents of the AIPs whilethe aggregate is open.

• For the aggregate. it holds the last current CiSeqno.

aggregateAiuSeqno Type: int

Label: AIP aggregate AIU seqno

Internal property temporarily used for the AIPsinvolved in an aggregation.• For children, it holds the number of the first AiuSeqno.

• For the aggregate, it holds the last current AiuSeqno.

returnCode Type: ENUM

Label: Return code

The status of the SIP. Possible values are OK orERROR. If the AIP is in ERROR state, the error mustbe diagnosed to further process the AIP.

returnMsg Type: String

Label: Return message

SIP

92


Field Description

dss.holdingName Type: String

Label: Holding name

Name of the archive holding in which the AIPs arestored.

dss.id Type: String

Label: DSS ID

ID of the DSS.

pdiSchema Type: String

Label: PDI Schema

XMLschema applied for the metadata description ofthe archived elements.

dss.productionDate Type: OffsetDateTime

Label: DSS production date

Production date of the DSS.

dss.baseRetentionDate Type: OffsetDateTime

Label: Base retention date

Base date used to calculate the retention.

dss.producer Type: String

Label: Producer

Name of the application producing the DSS.

dss.entity Type: String

Label: Entity

Name of the business entity. Data in the AIP belongsto a business entity.

dss.priority Type: Integer

Label: Priority

Ingestion priority.

dss.application Type: String

Label: Application

Data producing application.

93


Field Description

dss.retentionClass Type: String

Label: Retention class (also see defaultRetentionClass)

Logical name of the retention class to apply forretention.

sipProductionDate Type: OffsetDateTime

Label: SIP production date

SIP production date.

sipSeqNo Type: Integer

Label: Sequence number in the DSS

DSS SIP sequence number. A DSS is somehow acollection of SIP, each of the SIPs of the same collectionmust have a sequence number.

sipIsLast Type: Boolean

Label: Last SIP of the DSS

Last archiving unit of the collection.

sipAiuCount Type: Long

Label: AIU count

Number of AIUs in the SIP.

sipPageCount Type: Long

Label: Page count

Number of pages of the contents, associated to theelements to archive in the archive unit.

sipPdiHashAlgorithm Type: String

Label: PDI hash algorithm: MD2, MD5, SHA1,SHA256, SHA384, SHA512

Name of the algorithm use to hash the pdi file of thearchive unit (i.e., SHA-1).

sipPdiHashEncoding Type: String

Label: PDI hash encoding

Encoding supported: ’base64’ or ’hex’ (hexadecimal)

Metadata hash encoding. (i.e., base64).

94


Field Description

sipPdiHash Type: String

Label: PDI hash

Metadata hash value

sipFileFormat Type: String

Label: SIP file format

SIP file format received.

sipFileSize Type: Long

Label: SIP file size

SIP file size received.

sipFileHash Type: String

Label: SIP file hash

Hash value of the SIP file received during receptionphase.

xDB

xdbMode Type: ENUM

Label: xDB mode

xDB library storage mode. For each AIP, a detachablexDB library is created to store it. The following xDBmodes are available:• PRIVATE

• POOLED

• AGGREGATE

xdbPdiSchema Type: String

Label: xDB PDI schema

XML schema used to validate metadata stored in xDB.In general cases, this schema is the same schema forthe reception, but there are some exceptions:

• If the PDI schema of the reception does not allowgood performance for a search, it is possible toadd a transformation step and store it in xDB withanother schema.

95


Field Description

xdbLibrary Type: xDBLibrary

The reference to the xDB library where is stored theAIP.

xdbLibraryName Type: String

Label: xDB library name

Name of the library where the AIP is stored.

xdbLibrarySize Type: Long

Label: xDB library size

The size of the xDB library.

xdbLibraryDetached Type: boolean

Label: Offline

True if the library is detachable, False if the library isnot detachable.

Tracking

receiveStartDate Type: OffsetDateTime

Label: Reception start date

ingestWaitStartDate Type: OffsetDateTime

Label: Waiting ingestion start date

Occurs at the end of the reception.

ingestStartDate Type: OffsetDateTime

Label: Ingestion start date

commitWaitStartDate Type: OffsetDateTime

Label: Waiting commit start date

Occurs at the end of the ingestion.

commitDate Type: OffsetDateTime

Label: Commit date

When the AIP is committed.

purgeDate Type: OffsetDateTime

Label: Disposition date

Purged date.

96


Field Description

invalidDate Type: OffsetDateTime

Label: Invalidation date

Date when the AIP has been marked to be invalidated.

rejectDate Type: OffsetDateTime

Label: Rejection date

Date when the AIP has been marked to be rejected.

pdiConfigName Type: String

Label: PDI configuration name

The name of the PDI configuration objectcorresponding to this AIP. The corresponding objectis of type PDI. Its content is an XML file that containsinformation for ingesting AIP.

priority Type: Integer

Label: Holding Priority

Priority of the holding.

receiverNodeName Type: String

Label: Receiver node name

Name of the receiverNode used during reception.

ingestNodeName Type: String

Label: Ingest node name

Name of the ingestNode used during ingestion.

ingestConfigName Type: String

Label: Ingest configuration name

Name of the ingestion configuration used foringestion.

commitSync Type: boolean

Label: Commit during the ingestion

Indicates if this AIP can be synchronously committed(i.e., synchronous commit activated in the holding+ the SIP corresponds to a standalone DSS - seqno=1and is_last=true).

97


Field Description

invalidCategory Type: String

Label: Invalidation category

The category set during the invalidation.

invalidDescription Type: String

Label: Invalidation description

The description set during the invalidation.

invalidUserName Type: String

Label: Invalidation user name

The name of the user that invalidated the AIP.

rejectCategory Type: String

Label: Rejection category

The category set during the rejection.

rejectDescription Type: String

Label: Rejection description

The description set during the rejection.

rejectUserName Type: String

Label: Rejection user name

The name of the user that rejected the AIP.

Confirmations

confirmation.receiveDate Type: OffsetDateTime

Label: Reception confirmation date

Receive confirmation date. If this field is set, it meansthe confirmation has been done for the reception ofthis AIP.

confirmation.commitDate Type: OffsetDateTime

Label: Commit confirmation date

Commit confirmation date. If this field is set, it meansthe confirmation has been done for the commit of thisAIP.

98


Field Description

confirmation.purgeDate Type: OffsetDateTime

Label: Disposition confirmation date

Purge confirmation date. If this field is set, it meansthe confirmation has been done for the purge of thisAIP

confirmation.rejectDate Type: OffsetDateTime

Label: Rejection confirmation date

Reject confirmation date. If this field is set, it meansthe confirmation has been done for the rejection ofthis AIP.

confirmation.invalidDate Type: OffsetDateTime

Label: Invalidation confirmation date

Invalidation confirmation date. If this field is set,it means the confirmation has been done for theinvalidation of this AIP.

Permission

Permission Type: Permission

Label: Group names

Partition Keys

pkeys.dateTime01 Type: OffsetDateTime

Label: Date time 01


Label: Date time 02


Label: Date time 03


Label: Date time 04


Label: Date time 05


Label: Date time 06

pkeys.string01 Type: String

Label: String 01

99


Field Description


Label: String 02


Label: String 03


Label: String 04

pkeys.integer01 Type: Integer

Label: Integer 01


Label: Integer 02


Label: Integer 03


Label: Integer 04

pkeys.long01 Type: Long

Label: Long 01


Label: Long 02


Label: Long 03


Label: Long 04

pkeys.double01 Type: Double

Label: Double 01


Label: Double 02


Label: Double 03


Label: Double 04

100


Field Description

pkeys.values01 Type: List<String>

Label: Values 01


Label: Values 02


Label: Values 03


Label: Values 04

Summary SIP xDB Tracking Confirmation Groups

name dss.hold-ingName

xdbMode receive-StartDate

confirmation.receiveDate

groups

application dss.id xdbPdi-Schema

ingestWait-StartDate

confirmation-.commitDate

aipId dss.pdiSchema

xdbLibrary-Name

ingestStart-Date

confirmation.purgeDate

sipAiuCount dss.produc-tionDate

xdbLib-rarySize

commit-WaitStart-Date

confirmation.rejectDate

ciCount dss.base-Retention-Date

xdbLibrary-Detached

commit-Date

confirmation.invalidDate

pdiFileSize dss.producer purgeDate

pdiValues-CharCount

dss.entity invalidDate

ciSize dss.priority rejectDate

phaseCode dss.application

pdiConfig-Name

stateCode dss.retention-Class

priority

returnCode sip-Production-Date

receiver-NodeName

returnMsg sipSeqNo ingest-Node-Name

sipIsLast ingest-ConfigName

101


Summary SIP xDB Tracking Confirmation Groups

sipAiuCount commitSync

sipPage-Count

invalid-Category

sipPdi-Hash-Algorithm

invalid-Description

sipPdi-Hash-Encoding

invalid-UserName

sipPdiHash reject-Category

sipFile-Format

reject-Description

sipFileSize rejectUser-Name

sipFileHash

AIC

An Archive Information Collection (AIC) organizes a set of AIPs, which can support flexible andefficient data access.

AIPs are aggregated into the AIC using specified criteria (through a Criteria list) determined by thearchivist. Generally AICs are based on the AIPs of interest having common themes or origins, and acommon set of properties. For example, an archive for digital movies may have AICs based on thesubject area of the movie, such as action, science fiction or horror. In addition, the archive may haveAICs based on other factors, such as director or lead actor. It is common that one AIC corresponds toone holding, but several holdings are possible. At a minimum, InfoArchive can be viewed has havingat least one AIC which contains all the AIPs archived in the application.

CRUD operations can be performed on this object like other configuration objects. The alias typeis ’aic’.

Field Description

id Type: UUID

Mandatory: Yes

Label: Automatically generated by the system

name Type: String

Mandatory: Yes

Label: Collection name

102


Field Description


Mandatory: Yes

Label: Application

holdings Type: List<Holding>

Mandatory: No

Label: Holdings

criterias Type: List<Criteria>

Mandatory: No

Label: Criteria

name Type: String

Mandatory: No

Label: Criteria name

Valid criteria on the collection (internal name).

label Type: String

Mandatory: No

Label: Criteria label

Valid criteria on the collection (dispay name).

type Type: String

Mandatory: No

Label: Criteria type

Possible values: STRING, DATE, DATETIME,INTEGER, LONG, DOUBLE, HASHED, ID,CID.

pKeyMinAttr Type: String

Mandatory: No

Label: Minimum value attribute for the criteria

Name of the minimum value attribute (to setwith pKeyMaxAttr) for the partition key. Usedby the search to retrieve AIPs.

pKeyMaxAttr Type: String

Mandatory: No

criteria

103


Field Description

Label: Maximum value attribute for the criteria

Name of the maximum value attribute (to setwith pKeyMinAttr) for the partition key. Usedby the search to retrieve AIPs

pKeyValuesAttr Type: String

Mandatory: No

Label: Value attribute for the criteria

HeathCare use case. Name of the attribute thatcontains a list of values for the partition key.Used by the search to retrieve AIPs.

Application

Field Description

id Type: UUID

Mandatory: Yes

Label: ID

name Type: String

Mandatory: Yes

Label: Name

structuredDataStorageAllocationStrategy Type: String

Mandatory: Yes

contentStorageAllocationStrategy Type: String

Mandatory: Yes

tenant Type: Tenant

Mandatory: Yes

Label: Tenant

type Type: Enum

Mandatory: Yes

Label: Type

APP_DECOMM or ACTIVE_ARCHIVING

104


Field Description

archiveType Type: Enum

Mandatory: Yes

Label: Archive Type

SIP or Table

description Type: String

Mandatory: Yes

Label: Description

category Type: String

Mandatory: Yes

Label: Category

Confirmation

A confirmation is a message generated in reaction to an AIP event.

Once a SIP has been received and ingested, a confirmation message is automatically generated andpassed back to the source application to confirm that the data has been archived. The message is usedto trigger the deletion of the data from the source application. If, however, there are errors in the SIP,a rejection or an invalidation message is generated.

The following event types can trigger the generation of a confirmation:

Type Description Scope

receipt The SIP has been received. SIP xml

storage The AIP ingestion has been ingestedand commited.

SIP xml and PDI xml

purge The AIP has been disposed. SIP xml and PDI xml

reject The AIP has been rejected. SIP xml or none

invalid The AIP has been invalidated. SIP xml or none

Field Description

id Type: UUID

Label: ID

105

Field Description

name Type: String

Label: name

Name of the confirmation


Label: Application

Application of this confirmation

types Type: List<String>

Label: Confirmation type

Possible values: receipt, storage, purge, reject, invalid

holdings Type: List<Holding>

Label: Holdings

List of the holdings on which the confirmation can be applied to.

sipQuery Type: String

Label: SIP query

Local mode (will run the query on the sip.xml file attached).

pdiQuery Type: String

Label: PDI query

xDB mode (will run the query on the pdi file in xDB).

deliveryChannel Type: DeliveryChannel

Label: Delivery channel

The delivery channel to use for this confirmation.

deliveryChannelParameters Type: Map<String,String>

Label: Delivery channel parameters

The list of additional parameters to pass to the delivery channel (key,value). For example, ’%conf_type%’.

Creating a Confirmation with an ANT Script

240-confirmation.xml

<echo>Configuring confirmation object with name '${holding}-confirmation'</echo>

106


<ia-task self="confirmationSelf"><object typeAlias="confirmation" checkExistSpEL="?

....[name=='${holding}-confirmation']"><create>

<name>${holding}-confirmation</name></create><update>

<deliveryChannel>${deliveryChannelSelf}</deliveryChannel><holdings>${holdingSelf}</holdings><holdings/><types>RECEIPT</types><types>INVALID</types><types>REJECT</types><types>STORAGE</types><types>PURGE</types><deliveryChannelParameters>

<ia_conf_type>%ia_conf_type%</ia_conf_type><aip_id>%ia_conf_aip_id%</aip_id><ia_conf_datetime>%ia_conf_datetime%</ia_conf_datetime>

</deliveryChannelParameters></update>

</object></ia-task>

Content

SDX Partition feature is used with this object. The partition key is the Application UUID.

Field Description

id Type: UUID

Mandatory: Yes


parentId Type: UUID

Mandatory: Yes

Label: Parent ID

parentType Type: String

Mandatory: No

Label: Parent Type


Mandatory: Yes

Label: Application

Needed for SDX partition

107


Field Description

store Type: Store

Mandatory: Yes

Label: Store

format Type: String

Mandatory: Yes

Label: Format

modifier Type: String

Mandatory: No

Label: Modifier

path Type: List<String>

Mandatory: No

Label: Path / External ID

size Type: Long

Mandatory: Yes

Label: Size

checksum Type: String

Mandatory: Yes

Label: Checksum

createdDate Type: OffsetDateTime

Mandatory: Yes

lastModifiedDate Type: OffsetDateTime

Mandatory: Yes

version Type: Integer

Mandatory: Yes

orphan Type: Boolean

Mandatory: Yes

Set to true to flag this content as ready to be deleted (useful whendeleting the parent object)

108


Field Description

blob Type: String (Base64)

Mandatory: No

options Type: List<ENUM>

Mandatory: No

Label: Options

Delivery Channel

The delivery channel service allows the customer to write content to a destination defined by aconfiguration object. The destination can be xDB, a filesystem or another system with a customimplementation. The delivery channel will be used by the confirmation job and the backgroundsearch job.

Field Description

id Type: UUID

Label: ID


Label: Application

Application to which it depends.

name Type: String

Label: Name

The name of the delivery channel.

parameters Type: Map<String,String>

Label: Parameters

Default parameters to use. Only fixed parameters can be used(i.e., do not specify dynamic values as %aip_id%).

store Type: Store

Label: Store

The store to use as root. If the store is a xdbstore, the deliverychannel will be an xDB one. If the store is a filestore, then thedelivery channel will be a file one. If another kind of store isspecified, an exception will be raised.

109


Field Description

fileName Type: String

Label: File/document name

The value can contain variables such as ’%key1%’ defined inthe parameters map. Random UUID, by default.

subPath Type: String

Label: File/XdbLibrary sub-path

Relative path from the FileSystemFolder/XdbLibrary parentdefined at the Store level (cannot start with / or contain ./, ../ or//). The value can contain variables such as ’%key1%’ definedin the parameters map. Empty by default.

prefix Type: String

Label: File name prefix

The value can contain variables such as ’%key1%’ defined inthe parameters map. Empty by default.

suffix Type: String

Label: File name suffix

The value can contain variables such as ’%key1%’ defined inthe parameters map. Empty by default.

overwrite Type: boolean

Label: Overwrite

Remove the file if it already exists. False by default.

compress Type: boolean

Label: Compress

Compress the file and add the zip extension. False by default.

metadata Type: Map<String, String>

Label: Metadata

xDB metadata added on the document. The value can containvariables such as ’%key1%’ defined in the parameters map.Empty by default.

CRUD operations can be performed on this object like other configuration objects. The alias typeis ’delivery-channel’.

110


File System Folder

Field Description

id Type: UUID

Mandatory: Yes


name Type: String

Mandatory: Yes

Label: Folder name


Mandatory: Yes

Label: Folder sub-path

parentSpaceRootFolder Type: SpaceRootFolder

Mandatory: Yes

Label: Parent space root folder

parentFileSystemFolder Type: FileSystemFolder

Mandatory: Yes

Label: Parent file system folder

File System Root

The InfoArchive server does not create a default file system root if nothing is configured. Theapplication.yml file, however, allows for a "fileSystemRoot" entry to be used by the InfoArchiveserver. The specified path has to pre-exist on the file system. InfoArchive server will not create it.

Field Description

id Type: UUID

Mandatory: Yes


111


Field Description

path Type: String

Mandatory: Yes

Label: Path

default Type: Boolean

Mandatory: No

Label: Default

Allows the customer to define a defaultFileSystemRoot, which helps to define a workingdirectory, for example.

Holding

A holding is a logical destination archive where to ingest and store data, usually of the same typethat share common characteristics. For example, you can create a holding to archive data fromthe same source application (such as ERP data), or of the same format (such audio recordings), orbelonging to the same business entity.

An InfoArchive instance can contain multiple archive holdings.

You can create multiple archive holdings for a single data type for applying different access rightsand target storage areas to meet the requirements from different data owners.

Considering that the definition of the archive holdings is highly structured, it must be carefullydesigned according to:

• The expected content types to be archived

• The data segregation constraints for isolating the data owned by different business entitiesin distinct archive holdings

Most InfoArchive system configurations are performed at the holding level. Holding configurationencompasses many aspects of data archiving such as storage areas, retention policy, ingestionsequence, AIP mode, and xDB mode. The settings defined at the archive holding level are usedthroughout the whole lifecycle of the data archived in the holding.

Note: For best performance, leave working folders, such as the Tomcat temp folder, the ingest folder,and the receiver folder on the local disk. Performance and scalability issues might be encountered ifthese folders reside on network storage.

112


Field Description

id Type: UUID

Mandatory: Yes

Tab: Summary

Label: Identifier

name Type: String

Mandatory: Yes

Tab: Summary

Label: Holding name


Mandatory: Yes

Tab: Summary

Label: Application

format Type: String

Mandatory: Yes

Tab: Summary

Label: Ingest config by SIP format

sips sip

ingest Type: Ingest

Mandatory: Yes

Tab: Summary

Label: Ingest config by SIP format

Add ingestName (transient field).

schema Type: String

Mandatory: Yes

Tab: Summary

Label: PDI config by PDI Schema

pdis pdi

id Type: PDI

Mandatory: Yes

Tab: Summary

Label: PDI config by PDI Schema

113


Field Description

sipStore Type: Store

Mandatory: Yes

Tab: Storage

Label: Reception store

Name of the repository storage area for theSIP format.

keepSipAfterCommitEnabled Type: Boolean

Mandatory: Yes

Default: False

Tab: Storage

Label: Keep SIP file after commit

keepSipOnRejinvEnabled Type: Boolean

Mandatory: Yes

Default: False

Tab: Storage

Label: Keep SIP file on reject or invalidation

ingestNodes Type: Set<IngestNode>

Mandatory: Yes

Tab: Summary

Label: Ingest nodes

priority Type: Integer

Mandatory: Yes

Tab: Summary

Label: Priority

114


Field Description

dss Type: Integer

Mandatory: Yes

Tab: Summary

Label: Deadline by Sub-priority

subPriorities subPriority

deadline Type: Period

Mandatory: Yes

Tab: Summary

Label: Deadline by Sub-priority

logsStoreEnabled Type: Boolean

Mandatory: Yes

Default: True

Tab: Storage

Label: Logs store enabled

logsStore Type: Store

Mandatory: Yes

Tab: Storage

Label: Logs store

Name of the repository storage area for logfiles.

pdiXmlHashEnforced Type: Boolean

Mandatory: Yes

Default: False

Tab: Summary

Label: Enforce hash validation on PDI XML

pdiXmlHashValidationEnabled Type: Boolean

Mandatory: Yes

Default: True

Tab: Summary

Label: Validate PDI XML hash

115


Field Description

keepPdiXmlAfterIngestEnabled Type: Boolean

Mandatory: Yes

Default: True

Tab: Storage

Label: Keep PDI XML after commit

xmlStore Type: Store

Mandatory: Yes

Tab: Storage

Label: XML store

Name of the repository storage area for xmlfiles (SIP XML, PDI XML)

keepXmlOnRejinvEnabled Type: Boolean

Mandatory: Yes

Default: False

Tab: Storage

Label: Keep XML file on reject or invalidation

ciHashValidationEnabled Type: Boolean

Mandatory: Yes

Default: True

Tab: Summary

Label: Validate content hash

ciStore Type: Store

Mandatory: Yes

Tab: Storage

Label: Content store

Name of the repository storage area forcontent (CI Container and RI XML).

116


Field Description

keepCiOnRejinvEnabled Type: Boolean

Mandatory: Yes

Default: False

Tab: Storage

Label: Keep contents on reject or invalidation

defaultRetentionClass Type: String

Mandatory: No

Tab: Retention

Label: Default retention class

This case-sensitive setting is considered ifthe SIP does not mention a retention class toapply. The value assigned to this propertymust match a retention class defined in theeas_retention_class property.

It is recommended that, upon ingestion, toeither:

• Run the TableApplyRetention job to applya retention policy to individual tables; or

• Apply a retention policy to the entireapplication via the Application >Application Info tab.

Applying a retention policy to individualtables means that the customer needs to becareful that, if the tables have dependencies,the retention policy is configured properly.For instance, in the BaseBall application,it is advised that the customer should notapply a shorter retention policy to the mastertable than the other tables, as this can causedangling references.

117


Field Description

name Type: String

Mandatory: Yes

Tab: Retention

Label: Retention class

policies Type: List<String>

Mandatory: Yes

Tab: Retention

Label: Retention Policy

retention-Classes

retentionClass

holds Type: List<String>

Mandatory: Yes

Tab: Retention

Label: Legal Hold

reception Type: Permission

Mandatory: Yes

Tab: Permission

Label: Reception

ingestion Type: Permission

Mandatory: Yes

Tab: Permission

Label: Ingestion

waitingCommit Type: Permission

Mandatory: Yes

Tab: Permission

Label: Waiting Commit

completed Type: Permission

Mandatory: Yes

Tab: Permission

Label: Completed

permissionSet

Permissions list for all AIPstates (REC, INJ, WCOM, COM,REJ, INV, PUR). If no groupis defined, display no groupassigned.

118


Field Description

purge Type: Permission

Mandatory: Yes

Tab: Permission

Label: Purge

reject Type: Permission

Mandatory: Yes

Tab: Permission

Label: Reject

invalid Type: Permission

Mandatory: Yes

Tab: Permission

Label: Invalid

syncCommitEnabled Type: Boolean

Mandatory: Yes

Default: True

Tab: Summary

Label: Commit immediately the AIP after theingestion

Activates the commit at the end ofthe ingestion when the AIP is astandalone DSS (i.e., eas_sip_seqno=1and eas_sip_is_last=true).

mode Type: Enum

Mandatory: Yes

Tab: xDB

Label: Xdb mode

Possible values include:• PRIVATE

• POOLED

• AGGREGATE

xdb

119


Field Description

libraryParent Type: XdbLibrary

Mandatory: Yes

Tab: xDB

Label: Xdb Library Parent

xDB library configuration to be used:

• For mode 1, library in which the metadatamust be directly imported.

• For mode 2, library in which the sub-librarymust be created This property is not usedwhen applying mode 3 because the parentlibrary to use is defined at the library poollevel.

libraryPolicy Type: XdbLibraryPolicy

Mandatory: No

The xDB library pool configuration object touse during ingestion.

store Type: Store

Mandatory: Yes

Tab: Storage

Label: xDB library store

When ingestion mode 2 is applied, nameof the repository storage area where theimported sub-library must be assigned

LibraryPolicy Type: XdbLibraryPolicy

Label: Xdb library policy

Description: The xdb library poolconfiguration object to use during Ingestion.

120


Field Description

pushRetentionOnRejInvEnabled Type: Boolean

Mandatory: Yes

Tab: Retention

Label: Push the retention on reject orinvalidation

Activation of the assignation of the retentiondate on contents of the AIP imported inthe repository when the AIP is rejected orinvalidated.

renditionStore Type: Store

Mandatory: No

Tab: Storage

Label: Rendition store

Store to use to save extra renditions (Analyticrendition, purge list export, etc.).

Ingest

This object describe the configuration of an injection process :

• The sequence of tasks to execute injection.

• The Java class to enable for each task.

• The optional parameters of each task

Field Description

id Type: UUID

Mandatory: Yes

name Type: String

Mandatory: Yes

Label: Name

121

Field Description


Mandatory: Yes

Label: Application

contents Type: Set<Content>

Mandatory: Yes

Label: Content

XML Content<processors>

<processor><name>download of SIP content from content server</name><class>com.emc.documentum.eas.ingestor.transform.processor.downloader.

SIPContentDownloader</class></processor><processor>

<class>com.emc.documentum.eas.ingestor.transform.processor.policy.PolicyPromoter</class>

</processor><processor>

<name>promoter to uncompressed data state</name><class>com.emc.documentum.eas.ingestor.transform.processor.policy.PolicyPromoter

</class></processor><processor>

<id>sip.extractor</id><name>extract SIP content</name><class>com.emc.documentum.eas.ingestor.transform.processor.extractor.

SIPExtractor</class><data>

<pdi.name>eas_pdi.xml</pdi.name>

</data></processor><processor>

<name>promoter to the hash control state</name><class>com.emc.documentum.eas.ingestor.transform.processor.policy.

PolicyPromoter</class></processor><processor>

<name>PDI hash validation</name><class>com.emc.documentum.eas.ingestor.transform.processor.hash.

PDIHashValidatorProcessor</class></processor><processor>

<id>pdi.validation</id><name>XSD validation of metadata</name><class>com.emc.documentum.eas.ingestor.transform.processor.validator.

PDISchemaValidator</class><data>

<are.whiteSpaces.ignored>false</are.whiteSpaces.ignored></data>


<class>com.emc.documentum.eas.ingestor.transform.processor.policy.PolicyPromoter</class>

</processor>

122


<processor><data/>


<name>promoter to the metadata import state</name><class>com.emc.documentum.eas.ingestor.transform.processor.policy.


<name>xdb cache library creation</name><class>com.emc.documentum.eas.ingestor.transform.processor.library.

LibraryCreator</class></processor><processor>

<id>set.schema</id><name>set the new schema name</name><class>com.emc.documentum.eas.ingestor.transform.processor.utils.

SetSchemaNameProcessor</class><data/>


<name>SIP import into xdb cache</name><class>com.emc.documentum.eas.ingestor.transform.processor.importer.

SIPImporter</class></processor><processor>

<id>pdi.xdb.importer</id><name>PDI import into xdb cache</name><class>com.emc.documentum.eas.ingestor.transform.processor.importer.

PDIImporter</class><data>

<pdi.env.name>pdi.file.xml</pdi.env.name></data>


<name>PDI compression</name><class>com.emc.documentum.eas.ingestor.transform.processor.compress.

PDICompressor</class></processor><processor>

<data><file.env.name>pdi.file.xml</file.env.name>


<name>promoter to the metadata request state</name><class>com.emc.documentum.eas.ingestor.transform.processor.policy.


<id>pdi.index.creator</id><name>Create Indexes on PDI</name><class>com.emc.documentum.eas.ingestor.transform.processor.index.IndexesCreator

</class><data>

<key.document.name>xdb.pdi.name</key.document.name><indexes/>


<id>pdi.aiu.id</id><name>Add AIU IDs in PDI</name><class>com.emc.documentum.eas.ingestor.transform.processor.importer.

PDIAiuIdProcessor</class><data>

123


/node()/node()</data>


<id>pdi.aiu.cnt</id><name>Count AIU in PDI</name><class>com.emc.documentum.eas.ingestor.transform.processor.importer.

PDIAiuCntProcessor</class><data>

count(/node()/node())</data>


<id>pdi.aiu.index</id><name>Create AIU Indexes on PDI</name><class>com.emc.documentum.eas.ingestor.transform.processor.index.

IndexesCreator</class><data>

<key.document.name>xdb.pdi.name</key.document.name><indexes>

<value.index><name>pdi.aiu.id</name><compressed>false</compressed><unique.keys>true</unique.keys><concurrent>false</concurrent><value.index.type>STRING</value.index.type><element.uri/><element.name/><attribute.uri>urn:x-emc:eas:schema:pdi</attribute.uri><attribute.name>id</attribute.name>

</value.index></indexes>


<id>minmax</id><name>min-max computing</name><class>com.emc.documentum.eas.ingestor.transform.processor.importer.

PartitionValuesProcessor</class><data/>


<name>promoter to the metadata index creation state</name><class>com.emc.documentum.eas.ingestor.transform.processor.policy.PolicyPromoter


<name>promoter to the content import state</name><class>com.emc.documentum.eas.ingestor.transform.processor.policy.PolicyPromoter


<name>PDI content recorder</name><class>com.emc.documentum.eas.ingestor.transform.processor.content.

PDIContentRecorder</class></processor><processor>

<name>Contents uploader</name><class>com.emc.documentum.eas.ingestor.transform.processor.uploader.

ContentsUploader</class></processor><processor>

<name>xDB segment files importer</name><class>com.emc.documentum.eas.ingestor.transform.processor.library.

LibraryImporter</class>

124



<name>promoter to the content commit-wait state</name><class>com.emc.documentum.eas.ingestor.transform.processor.policy.


<name>Synchronous commit</name><class>com.emc.documentum.eas.ingestor.transform.processor.

commit.CommitProcessor</class></processor>

</processors>

Ingest Node (ia_cfg_ingest_node)

Field Description

id Type: UUID

Mandatory: Yes


name Type: String

Mandatory: Yes

Label: Name

Name of reception node.


Mandatory: Yes

Label: Application

logLevel Type: ENUM

Mandatory: Yes

Label: Log level

Possible values for logLevel:

• TRACE

• DEBUG

• INFO

• WARN

• ERROR

• ALL

125


Field Description

• OFF

workingDirectory FileSystemFolder

Mandatory: Yes

Label: Working directory

enumerationCutoffDays Type: Integer

Mandatory: Yes

Label: Enumeration Cutoff Days

enumerationMaxResultCount Type: Integer

Mandatory: Yes

Label: Enumeration Max Result Count

enumerationMinusRunning Type: Boolean

Mandatory: Yes

Default: False

Label: Enumeration Minus Running

PDI (ia_cfg_pdi)

The Preservation Description Information (PDI) file in a SIP stores structured data to archive.

Unlike the SIP descriptor (eas_sip.xml), there is no pre-defined schema for the PDI file. You candefine your own schema using an XML editor for the type of data to archive based on your businessrequirements (the PDI file name must be eas_pdi.xml though). Any schema can be used as long asyou import it into InfoArchive and perform some additional configurations.

A file name can be stored as a value of one or several XML elements and/or attributes, and the samefile name can be referenced multiple times within an eas_pdi.xml file.

Field Description

id Type: UUID

Mandatory: Yes

126


Field Description

name Type: String

Mandatory: Yes

Label: Name


Mandatory: Yes

Label: Application

XML Content

<datas><data>

<id>pdi.index.creator</id><parameter>

<key.document.name>xdb.pdi.name</key.document.name><indexes>

<path.value.index><name>CustomerID</name><path>/{urn:eas-samples:en:xsd:phonecalls.1.0}Calls/

{urn:eas-samples:en:xsd:phonecalls.1.0}Call[{urn:eas-samples:en:xsd:phonecalls.1.0}CustomerID<LONG>]</path>

<compressed>false</compressed><unique.keys>false</unique.keys><concurrent>false</concurrent><build.without.logging>true</build.without.logging>


<name>CallStartDate</name><path>/{urn:eas-samples:en:xsd:phonecalls.1.0}Calls/{urn:eas-

samples:en:xsd:phonecalls.1.0}Call[{urn:eas-samples:en:xsd:phonecalls.1.0}CallStartDate<DATE_TIME>]</path>



<name>CustomerLastName</name><path>/{urn:eas-samples:en:xsd:phonecalls.1.0}Calls/{urn:eas-

samples:en:xsd:phonecalls.1.0}Call[{urn:eas-samples:en:xsd:phonecalls.1.0}CustomerLastName<STRING>]</path>



<name>CustomerFirstName</name><path>/{urn:eas-samples:en:xsd:phonecalls.1.0}Calls/

{urn:eas-samples:en:xsd:phonecalls.1.0}Call[{urn:eas-samples:en:xsd:phonecalls.1.0}CustomerFirstName<STRING>]</path>



127


<name>RepresentativeID</name><path>/{urn:eas-samples:en:xsd:phonecalls.1.0}Calls/{urn:eas-

samples:en:xsd:phonecalls.1.0}Call[{urn:eas-samples:en:xsd:phonecalls.1.0}RepresentativeID<LONG>]</path>


</path.value.index></indexes>

</parameter></data><data>

<id>pdi.aiu.id</id><parameter>

<![CDATA[declare namespace n = "urn:eas-samples:en:xsd:phonecalls.1.0";for $call in /n:Calls/n:Callreturn ($call, $call/n:CustomerID/text())]]>


<id>pdi.aiu.cnt</id><parameter>

<![CDATA[declare namespace n = "urn:eas-samples:en:xsd:phonecalls.1.0";count(/n:Calls/n:Call)]]>


<id>minmax</id><parameter>

<key name="CallStartDate" type="date-time" xml:space="preserve"><min field="eas_pkey_min_date" xml:space="preserve">

declare namespace n = "urn:eas-samples:en:xsd:phonecalls.1.0";adjust-dateTime-to-timezone(min(/n:Calls/n:Call/xs:dateTime

(n:CallStartDate)), xs:dayTimeDuration("PT0H"))</min><max field="eas_pkey_max_date" xml:space="preserve">

declare namespace n = "urn:eas-samples:en:xsd:phonecalls.1.0";adjust-dateTime-to-timezone(max(/n:Calls/n:Call/xs:dateTime

(n:CallStartDate)), xs:dayTimeDuration("PT0H"))</max>

</key></parameter>

</data><data>

<id>toc.creator</id><parameter>

<![CDATA[declare namespace n = "urn:eas-samples:en:xsd:phonecalls.1.0";for $ci in distinct-values(/n:Calls/n:Call/n:Attachments/n:Attachment/

n:FileName)order by $cireturn <content type="audio/x-mpeg" format="mp3" audit="{ $ci }">

{ $ci }</content>]]>


<id>pdi.ci.id</id><parameter>

<![CDATA[

128


declare namespace n = "urn:eas-samples:en:xsd:phonecalls.1.0";declare namespace ri = "urn:x-emc:eas:schema:ri";let $pdi_uri := root(.)let $aip_id := xhive:metadata($pdi_uri, 'eas_aip_id')let $ri_uri := replace(document-uri($pdi_uri), '\.pdi$', '.ri')for $ri in doc($ri_uri)/ri:ris/ri:ri[@pdi_key]for $n in /n:Calls/n:Call/n:Attachments/n:Attachment/n:FileName[. = $ri/@pdi_key]return ($n,concat($aip_id,":ci:",$ri/@seqno))]]>

</parameter></data>

</datas>

PdiSchema (ia_cfg_schema)

Field Description

id Type: UUID

Mandatory: Yes

Automatically generated by the system

name Type: String

Mandatory: Yes

Label: Collection Name


Mandatory: Yes

Label: Application

format Type: String

Mandatory: Yes

Label: Schema format

xsd

XSD Content<?xml version="1.0" encoding="UTF-8" standalone="yes"?><xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" targetNamespace="urn:eas-samples:en:xsd:phonecalls.1.0" version="1.0" elementFormDefault="qualified"xmlns:Q1="urn:eas-samples:en:xsd:phonecalls.1.0"><xs:element name="Calls"><xs:complexType><xs:sequence maxOccurs="unbounded"><xs:element name="Call"><xs:complexType><xs:sequence><xs:element name="SentToArchiveDate" type="xs:date" nillable="false" /><xs:element name="CallStartDate" type="xs:dateTime" nillable="false" />

129


<xs:element name="CallEndDate" type="xs:dateTime"/><xs:element name="CallFromPhoneNumber"><xs:simpleType><xs:restriction base="xs:positiveInteger"><xs:minInclusive value="1" /><xs:totalDigits value="11" /></xs:restriction>

</xs:simpleType></xs:element><xs:element name="CallToPhoneNumber" nillable="false"><xs:simpleType><xs:restriction base="xs:positiveInteger"><xs:minInclusive value="1" /><xs:totalDigits value="11" /></xs:restriction>

</xs:simpleType></xs:element><xs:element name="CustomerID" nillable="false"><xs:simpleType><xs:restriction base="xs:positiveInteger"><xs:totalDigits value="11" /><xs:minInclusive value="1" /></xs:restriction>

</xs:simpleType></xs:element><xs:element name="CustomerLastName" nillable="false"><xs:simpleType><xs:restriction base="xs:normalizedString"><xs:minLength value="1" /><xs:maxLength value="32" /></xs:restriction>

</xs:simpleType></xs:element><xs:element name="CustomerFirstName" nillable="false"><xs:simpleType><xs:restriction base="xs:normalizedString"><xs:minLength value="1" /><xs:maxLength value="32" /></xs:restriction>

</xs:simpleType></xs:element><xs:element name="RepresentativeID" nillable="false"><xs:simpleType><xs:restriction base="xs:positiveInteger"><xs:minInclusive value="1" /><xs:totalDigits value="7" /></xs:restriction>

</xs:simpleType></xs:element><xs:element name="Attachments" nillable="false" minOccurs="1"><xs:complexType><xs:sequence maxOccurs="unbounded" minOccurs="0"><xs:element name="Attachment"><xs:complexType><xs:sequence><xs:element name="AttachmentName" nillable="false" maxOccurs="1"><xs:simpleType><xs:restriction base="xs:normalizedString"><xs:minLength value="1" /><xs:maxLength value="32" /></xs:restriction>

</xs:simpleType></xs:element><xs:element name="FileName" nillable="false" minOccurs="1" maxOccurs="1"><xs:simpleType>

130


<xs:restriction base="xs:normalizedString"><xs:minLength value="1" /><xs:maxLength value="32" /></xs:restriction>

</xs:simpleType></xs:element><xs:element name="CreatedBy" nillable="false" maxOccurs="1"><xs:simpleType><xs:restriction base="xs:normalizedString"><xs:minLength value="1" /><xs:maxLength value="32" /></xs:restriction>

</xs:simpleType></xs:element><xs:element name="CreatedOnDate" type="xs:dateTime"

nillable="false" /></xs:sequence></xs:complexType>

</xs:element></xs:sequence>

</xs:complexType></xs:element>


</xs:element></xs:sequence>

</xs:complexType></xs:element>

</xs:schema>

Query (eas_cfg_query)

The query configuration object (eas_cfg_query) defines:

• Which AIC (each corresponding to a holding) to search. Can be configured to search one ormore specified holdings.

• How search criteria can be built (through the xdbPdiConfig).

• Howmany results can be returned (through QueryQuota for both synchronous and asynchronoussearch).

This object defines every option needed to build a query to retrieve an AIU.

CRUD operations can be performed on this object like other configuration objects. The alias typeis ’query’.

Field Description

id Type: UUID

Mandatory: Yes

Label: Automatically generated by thesystem

131


Field Description

name Type: String

Mandatory: Yes

Label: Name

Query configuration name.


Mandatory: Yes

Label: Application

The application associated with the query.

aics Type: List<AIC>

Mandatory: Yes

Label: Archiving unit collections

List of AICs that can be queried throughthis configuration.

resultSchema Type: String

Mandatory: Yes

Label: Result schema

URN of the schema in which the queryresults are returned (in XML) before beingtransform for the UI. Only one queryconfiguration object for the same resultschema is allowed.

resultRootElement Type: String

Mandatory: No

Label: Result root element

The root XML element of the search result.Should be set to ’result’.

132


Field Description

resultRootNsEnabled Type: boolean

Mandatory: No

Default: False

Label: Namespace in root element

Activates the declaration of the resultschema as namespace of the root XMLelement of the results.

quota Type: QueryQuota

Mandatory: Yes

Label: Query quota

Query quota to use for the synchronoussearch.

quotaAsync Type: QueryQuota

Label: Asynchronous query quota

Query quota to use for the asynchronoussearch (background search).

order Type: Order

Mandatory: No

Label: Order

Configuration object for the backgroundsearch (asynchronous serach).

namespaces Type: List<Namespace>

Mandatory: No

Label: Set of name spaces

The search can query multiple namespace.

133


Field Description

uri Type: String

Mandatory: No

Label: Namespace URI

URI of the namespace to build the query(on the ingested data).

namespace

prefix Type: String

Mandatory: No

Label: Namespace prefix

Prefix of the namespace to build the query(on the ingested data)

xdbPdi-Configs

Type: List<xdbPdiConfig>

schema Type: String

Mandatory: Yes

Label: xDB PDI schema

URN of the ingested data (pdi).

entityPath Type: String

Mandatory: Yes

Label: Entity path

xpath to the AIU.

template Type: String

Mandatory: No

Label: Query template

Default Value: ’return $aiu’

xQuery that can be added to retrieveresult.

name Type: String

Mandatory: Yes

Label: OrderBy name

Name of the field to use for the order.

orderBy

xdbPdi-Config

134


Field Description

direction Type: String

Mandatory: No

Label: OrderBy direction

Possible values: ASCENDING,DESCENDING for this order criterion.

position Type: Integer

Mandatory: No

Label: OrderBy position

Position of the order criterion.

operands Type: List<Operand>

name Type: String

Mandatory: Yes

Label: Operand name

Name of the queryable field (internalname in the PDI, not the display name).

path Type: String

Mandatory: Yes

Label: Operand path

The xpath of queryable field in the PDI(relative xpath to enityPath).

type Type: Aic.CriteriaType

Mandatory: Yes

Label: Operand type

Default: STRING

Type of the queryable field. Possiblevalues: STRING, LONG, INTEGER,DOUBLE, DATE, DATETIME, CID,AIUID.

Operand

index Type: boolean

Mandatory: No

Label: Operand index

’true’ if the field is indexed.

135


Query Quota (ia_cfg_query_quota)

Field Description

id Type: UUID

Mandatory: Yes


name Type: String

Mandatory: Yes

Label: Name


Mandatory: Yes

Label: Application

aipQuota Type: long

Mandatory: Yes

Label: AIP quota

Maximum number of AIPs that can be embraced for thesearch. A zero value indicates an unlimited number.

aiuQuota Type: long

Mandatory: Yes

Label: AIU quota

Maximum number of AIUs that can be embraced for thesearch. A zero value indicates an unlimited number.

dipQuota Type: long

Mandatory: Yes

Label: DIP quota

Maximum number of AIU returns by the search. A zerovalue indicates an unlimited number.

orderAipQuota Type: long

Mandatory: Yes

Label: AIP quota for background search

Maximum number of AIPs that can be embraced for thesearch. A zero value indicates an unlimited number.

136


Field Description

orderAiuQuota Type: long

Mandatory: Yes

Label: AIU quota for background search

Maximum number of AIUs that can be embraced for thesearch. A zero value indicates an unlimited number.

orderDipQuota Type: long

Mandatory: Yes

Label: DIP quota for background search

Maximum number of AIU returns by the search. A zerovalue indicates an unlimited number.

Receiver Node (ia_cfg_receive_node)

Field Description

id Type: UUID

Mandatory: Yes


name Type: String

Mandatory: Yes

Label: Name

Name of reception node.


Mandatory: Yes

Label: Application

137


Field Description

logLevel Type: LogLevel

Mandatory: Yes

Label: Log level

Possible values for logLevel:

• TRACE

• DEBUG

• INFO

• WARN

• ERROR

• ALL

• OFF

workingDirectory Type: FileSystemFolder

Mandatory: Yes

Label: Working Directory

format Type: String

Mandatory: Yes

Label: Format

SIP format.

extractorImpl Type: String

Mandatory: No

Label: Extractor Implementation class

Full qualified name of the java class implementing theInfoArchive interface ISipExtractor.java (by defaultwe need a SipZipExtractor, EasSipZipExtractor,ExtSipExtractor). If not defined, use theSipZipExtractor.

sips sip

extractor-Parameter

Type: List<String>

Mandatory: Yes

Label: Extractor Parameters

SIP Extractor Interface

138


import com.emc.documentum.eas.common.exception.EventException;import com.emc.documentum.eas.datamodel.sip.data.SIP;

import javax.xml.validation.Schema;import java.io.Closeable;import java.io.File;

/*** The interface defines the requirements of classes that extract SIP froma file*/public interface ISIPExtractor extends Closeable {

/*** The method that extract an SIP from a file** @param sipFile is the file to process* @return the SIP for the file (if any)* @throws EventException can be thrown by implementation*/SIP extract(File sipFile) throws EventException;

/*** @return the associated schema*/Schema getSchema();

/*** @return the java class arg1 value*/String getArg1();

/*** @return the working directory*/File getWorkingDir();

/*** @return the SIP format*/String getFormat();

}

Search

A search can be created for either table or SIP data.

The search is managed by the search composition and is composed of search forms (the xform) andthe result pages (the result master).

A primary search is a search seen by users, although a nested search can be used by another searchto link searches.

A search can run synchronously or asynchronously.

For SIP data, the search is executed by two queries:

• The first retrieves packages (AIP) with the partition key criteria,

• The second retrieves the suitable AIU among the AIPs of the first query.

139


The SIP search is configured with an AIC, a query configuration to define the queryable fields andthe Result Configuration Helper to list the available result fields. The SIP search is executed on anAIC based on one or several holdings in the same application.

Field Description

id Type: UUID

Mandatory: Yes


name Type: String

Mandatory: Yes

Label: Search name

The search name.


Mandatory: Yes

Label: Application

Application of which it depends.


Mandatory: No

The search description.

searchGroup Type: SearchGroup

Mandatory: No

schema Type: Schema

Mandatory: Yes (for table search)

table Type: Table


aic Type: AIC


Label: Archive Collection

The Archive Information Collection of SIP data.

140


Field Description

query Type: Query


Label: Configuration

The Query Configuration of the SIP data.

nestedSearch Type: boolean

Label: Nested Search

Set to ’True’ for a nested search to allow search to be linkedfrom other searches. When selected, this is not shown tousers for access. Set to ’False’ for primary search to allowusers access.

state Type: State

Possible values: DRAFT or PUBLISHED

useBySearches Type: Collection<String>

List search names.

SearchComposition

The Search Composition manages all search objects for the user interface:

• Either XForms, resultmaster and search for SIP data, or

• XForms, XQueries, resultmaster and search for table data.

A search is executed from the search composition (synchronously or asynchronously).

Field Description

id Type: UUID

Mandatory: Yes


name Type: String

Mandatory: Yes

Label: Name

search Type: Search

Mandatory: Yes

Label: The Search

141


Field Description

searchName Type: String

Mandatory: Yes

Label: The Search Name

permission Type: Permission

Creating a searchComposition with an ANT Script

The following example illustrates the procedure with the PhoneCalls application. Only the groupproperty can be set in the searchcomposition.properties file.groups=

XForm

This is the search form to generate the search criteria.

Field Description

id Type: UUID

Mandatory: Yes


name Type: String

Mandatory: Yes

Label: xform name

Name of the search form.

Application Type: Application

Mandatory: Yes

Label: Application

Application of which it depends.

form Type: String

Mandatory: Yes

Content of the search form.

142


Field Description

searchComposition Type: SearchComposition

Mandatory: Yes

Label: Search composition

Search composition it depends upon

search Type: Search

Mandatory: Yes

Search it depends upon


Label: The search name

Name of the search it depends upon

Creating an XForm from an ANT Script

The following example illustrates the content for the PhoneCalls application (for SIP data) in theDefaultSearch.xml file:

<xhtml:html xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:xforms="http://www.w3.org/2002/xforms" xmlns:xsd="http://www.w3.org/2001/XMLSchema">

<xhtml:head><xforms:model>

<title>Default PhoneCalls Search</title><xforms:instance xmlns="">

<data><CallStartDate>

<from>2000-01-01</from><to>2016-04-10</to>

</CallStartDate><CustomerID/><CustomerLastName/><RepresentativeID/><CustomerFirstName/>

</data></xforms:instance><xforms:instance xmlns="" id="labels">

<labels><CallStartDate>Call Start Date:</CallStartDate><CustomerID>Customer ID:</CustomerID><CustomerLastName>Customer Last Name:</CustomerLastName><RepresentativeID>Representative ID:</RepresentativeID><CustomerFirstName>Customer First Name:</CustomerFirstName>

</labels></xforms:instance><xforms:instance xmlns="" id="hints">

<hints><CallStartDate/><CustomerID/><CustomerLastName/>

143


<RepresentativeID/><CustomerFirstName/>

</hints></xforms:instance><xforms:instance xmlns="" id="prompts">

<prompts><CustomerID/><CustomerLastName/><RepresentativeID/><CustomerFirstName/>

</prompts></xforms:instance><xforms:instance xmlns="" id="alerts">

<alerts><CallStartDate/><CustomerID/><CustomerLastName/><RepresentativeID/><CustomerFirstName/>

</alerts></xforms:instance><xforms:instance xmlns="" id="range-messages">

<rangemessages><CustomerID/><CustomerLastName/><RepresentativeID/><CustomerFirstName/>

</rangemessages></xforms:instance><xforms:instance xmlns="" id="pattern-messages">

<patternmessages><CustomerID/><CustomerLastName/><RepresentativeID/><CustomerFirstName/>

</patternmessages></xforms:instance><xforms:submission id="phoneSearchSubmission" method="post" serialization=

"application/xml"/><xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xforms=

"http://www.w3.org/2002/xforms" targetNamespace="http://www.w3.org/2002/xforms" elementFormDefault="qualified">

</xsd:schema><bind xmlns="http://www.w3.org/2002/xforms" ref="/data/CallStartDate/

from" type="xforms:date" constraint="(string-length(.) = 0 or (days-from-date(.)<= days-from-date(/data/CallStartDate/to) and string-length(/data/CallStartDate/to) > 0))" required="true()"/>

<bind xmlns="http://www.w3.org/2002/xforms" ref="/data/CallStartDate/to" type="xforms:date" constraint="(string-length(.) = 0 or (days-from-date(.) >=days-from-date(/data/CallStartDate/from) and string-length(/data/CallStartDate/from) > 0))" required="true()"/>

<bind xmlns="http://www.w3.org/2002/xforms" ref="/data/CustomerID"/><bind xmlns="http://www.w3.org/2002/xforms" ref="/data/

CustomerLastName"/><bind xmlns="http://www.w3.org/2002/xforms" ref="/data/

RepresentativeID"/><bind xmlns="http://www.w3.org/2002/xforms" ref="/data/

CustomerFirstName"/></xforms:model>

</xhtml:head><xhtml:body>

<xhtml:h1>Default PhoneCalls Search</xhtml:h1><input xmlns="http://www.w3.org/2002/xforms" ref="CallStartDate/from">

<label ref="instance('labels')/CallStartDate"/>

144


<hint ref="instance('hints')/CallStartDate"/><alert ref="instance('alerts')/CallStartDate"/>

</input><input xmlns="http://www.w3.org/2002/xforms" ref="CallStartDate/to"/><input xmlns="http://www.w3.org/2002/xforms" ref="CustomerID">

<label ref="instance('labels')/CustomerID"/><hint ref="instance('hints')/CustomerID"/><hint appearance="minimal" ref="instance('prompts')/CustomerID"/><alert ref="instance('alerts')/CustomerID"/><message class="range" ref="instance('range-messages')/CustomerID"/><message class="pattern" ref="instance('pattern-messages')

/CustomerID"/></input><input xmlns="http://www.w3.org/2002/xforms" ref="CustomerLastName">

<label ref="instance('labels')/CustomerLastName"/><hint ref="instance('hints')/CustomerLastName"/><hint appearance="minimal" ref="instance('prompts')/CustomerLastName"/><alert ref="instance('alerts')/CustomerLastName"/><message class="range" ref="instance('range-messages')/CustomerLastName"/><message class="pattern" ref="instance('pattern-messages')/

CustomerLastName"/></input><input xmlns="http://www.w3.org/2002/xforms" ref="RepresentativeID">

<label ref="instance('labels')/RepresentativeID"/><hint ref="instance('hints')/RepresentativeID"/><hint appearance="minimal" ref="instance('prompts')/RepresentativeID"/><alert ref="instance('alerts')/RepresentativeID"/><message class="range" ref="instance('range-messages')/RepresentativeID"/><message class="pattern" ref="instance('pattern-messages')

/RepresentativeID"/></input><input xmlns="http://www.w3.org/2002/xforms" ref="CustomerFirstName">

<label ref="instance('labels')/CustomerFirstName"/><hint ref="instance('hints')/CustomerFirstName"/><hint appearance="minimal" ref="instance('prompts')

/CustomerFirstName"/><alert ref="instance('alerts')/CustomerFirstName"/><message class="range" ref="instance('range-messages')

/CustomerFirstName"/><message class="pattern" ref="instance('pattern-messages')

/CustomerFirstName"/></input>

</xhtml:body></xhtml:html>

ResultMaster

The search result page are rendered in the InfoArchive web application with the ResultMaster.

It is generated during search design by the search composition.

The search result field are dispatched in several sections (panels) according to the ResultMaster.

Field Description

id Type: UUID

Mandatory: Yes

145


Field Description


Mandatory: Yes

Application it depends upon.

searchComposition Type: SearchComposition

Mandatory: Yes

SearchComposition it depends upon.

search Type: Search

Mandatory: Yes

Search it depends upon.


Mandatory: Yes

Name of the search

panels Type: List<Panel>

Mandatory: No

Each panel is a section of the search resultpage.

name Type: String

Mandatory: Yes

Panel’s name (example ’Side Panel’).

title Type: String

Mandatory: No

Title of the panel.


Mandatory: No

Description of the panel.

Panel

tabs Type: List<Tab>

Mandatory: No

Tabs of this panel (example the ’side panel’is composed of several tabs).

146


Field Description

name Type: String

Mandatory: Yes

Name of the Tab.

title Type: String

Mandatory: No

Title of the Tab.


Mandatory: No

Dsscription of the Tab.

Tab

columns Type: List<Column>

Mandatory: No

Columns of the result page (returneddata).

name Type: String

Mandatory: Yes

Internal name.

label Type: String

Display name.

hidden Type: boolean

xdbElementName Type: String

masked Type: boolean

dataType Type: DataType

Possible values: STRING, DATE,DATETIME, NUMBER, BOOLEAN,INTEGER, DOUBLE, LONG, FLOAT, ID,CID.

type Type: Type

Possible values: XQUERY_REFERENCE,SCHEMA_COLUMN_NAME, CONTENT,NESTED_SEARCH or EXTERNAL_URL.

order Type: int

sortable Type: boolean

Mandatory:

Column

147


Field Description

defaultSort Type: DefaultSort

Possible values: NONE, ASCENDING,DESCENDING

links Type: Map<String,String>

nestedSearch Type: Search

nestedSearchName Type: String

url Type: URL

parameterMapping Type: Map<String,String>

groupName Type: String

groupPath Type: String

path Type: String

namespaces Type: List<Namespace>

List of namespaces for the results.

uri Type: String

URN of the namespace.

Namespace

prefix Type: String

Prefix of the namespace.

Creating a ResultMaster with an ANT Script

The following example with the illustrates the PhoneCalls holding in the DefaultResultMaster.xml file:<data>

<namespaces><prefix>n</prefix><uri>urn:eas-samples:en:xsd:phonecalls.1.0</uri>

</namespaces><namespaces>

<prefix>ia</prefix><uri>urn:x-emc:ia:schema:pdi</uri>

</namespaces><panels>

<name>Main Panel</name><tabs>

<name>_ia_Default_Main_tab_</name><columns>

<name>CustomerID</name><type>SCHEMA_COLUMN_NAME</type><binding>CustomerID</binding><path>n:CustomerID</path><label>Customer ID</label><dataType>INTEGER</dataType><sortable>true</sortable><defaultSort>NONE</defaultSort>

148


<hidden>false</hidden><encrypt>false</encrypt><masked>false</masked>

</columns><columns>

<name>CustomerLastName</name><type>SCHEMA_COLUMN_NAME</type><binding>CustomerLastName</binding><path>n:CustomerLastName</path><label>Last Name</label><dataType>STRING</dataType><sortable>true</sortable><defaultSort>NONE</defaultSort><hidden>false</hidden><encrypt>false</encrypt><masked>false</masked>

</columns><columns>

<name>CustomerFirstName</name><type>SCHEMA_COLUMN_NAME</type><binding>CustomerFirstName</binding><path>n:CustomerFirstName</path><label>First Name</label><dataType>STRING</dataType><sortable>true</sortable><defaultSort>NONE</defaultSort><hidden>false</hidden><encrypt>false</encrypt><masked>false</masked>

</columns><columns>

<name>CallStartDate</name><type>SCHEMA_COLUMN_NAME</type><binding>CallStartDate</binding><path>n:CallStartDate</path><label>Call Start</label><dataType>DATETIME</dataType><sortable>true</sortable><defaultSort>ASCENDING</defaultSort><hidden>false</hidden><encrypt>false</encrypt><masked>false</masked>

</columns><columns>

<name>CallToPhoneNumber</name><type>SCHEMA_COLUMN_NAME</type><binding>CallToPhoneNumber</binding><path>n:CallToPhoneNumber</path><label>Call To Phone Number</label><dataType>STRING</dataType><sortable>true</sortable><defaultSort>NONE</defaultSort><hidden>false</hidden><encrypt>false</encrypt><masked>false</masked>

</columns><columns>

<name>Attachment1</name><type>SCHEMA_COLUMN_NAME</type><binding>Attachment1</binding><path>n:Attachments/n:Attachment[1]/n:FileName/@ia:cid</path><label>Attachment</label><dataType>CID</dataType><sortable>false</sortable><defaultSort>NONE</defaultSort>

149


<hidden>false</hidden><encrypt>false</encrypt><masked>false</masked>

</columns><columns/>

</tabs><tabs/>

</panels><panels>

<name>Inline Panel</name><tabs>

<name>_ia_Default_Detail_Inline_tab_</name><columns>

<name>cid</name><type>SCHEMA_COLUMN_NAME</type><binding>cid</binding><path>n:FileName/@ia:cid</path><label>CID</label><dataType>CID</dataType><hidden>false</hidden><encrypt>false</encrypt><masked>false</masked><groupName>Attachments</groupName><groupPath>n:Attachments/n:Attachment</groupPath>

</columns><columns>

<name>AttachmentName</name><type>SCHEMA_COLUMN_NAME</type><binding>AttachmentName</binding><path>n:AttachmentName</path><label>Name</label><dataType>STRING</dataType><hidden>false</hidden><encrypt>false</encrypt><masked>false</masked><groupName>Attachments</groupName><groupPath>n:Attachments/n:Attachment</groupPath>

</columns><columns>

<name>FileName</name><type>SCHEMA_COLUMN_NAME</type><binding>FileName</binding><path>n:FileName</path><label>Filename</label><dataType>STRING</dataType><hidden>false</hidden><encrypt>false</encrypt><masked>false</masked><groupName>Attachments</groupName><groupPath>n:Attachments/n:Attachment</groupPath>

</columns><columns>

<name>CreatedBy</name><type>SCHEMA_COLUMN_NAME</type><binding>CreatedBy</binding><path>n:CreatedBy</path><label>Created by</label><dataType>STRING</dataType><hidden>false</hidden><encrypt>false</encrypt><masked>false</masked><groupName>Attachments</groupName><groupPath>n:Attachments/n:Attachment</groupPath>


150


</tabs><tabs/>

</panels><panels>

<name>Side Panel</name><tabs>

<name>_ia_Default_Detail_Side_tab_</name><columns>

<name>CustomerID</name><type>SCHEMA_COLUMN_NAME</type><binding>CustomerID</binding><path>n:CustomerID</path><label>Customer ID</label><dataType>INTEGER</dataType><hidden>false</hidden><encrypt>false</encrypt><masked>false</masked>

</columns><columns>

<name>RepresentativeID</name><type>SCHEMA_COLUMN_NAME</type><binding>RepresentativeID</binding><path>n:RepresentativeID</path><label>Representative ID</label><dataType>INTEGER</dataType><hidden>false</hidden><encrypt>false</encrypt><masked>false</masked>

</columns><columns>

<name>CustomerLastName</name><type>SCHEMA_COLUMN_NAME</type><binding>CustomerLastName</binding><path>n:CustomerLastName</path><label>Last name</label><dataType>STRING</dataType><hidden>false</hidden><encrypt>false</encrypt><masked>false</masked>

</columns><columns>

<name>CustomerFirstName</name><type>SCHEMA_COLUMN_NAME</type><binding>CustomerFirstName</binding><path>n:CustomerFirstName</path><label>First name</label><dataType>STRING</dataType><hidden>false</hidden><encrypt>false</encrypt><masked>false</masked>

</columns><columns>

<name>SentToArchiveDate</name><type>SCHEMA_COLUMN_NAME</type><binding>SentToArchiveDate</binding><path>n:SentToArchiveDate</path><label>Sent to</label><dataType>DATE</dataType><hidden>false</hidden><encrypt>false</encrypt><masked>false</masked>

</columns><columns>

<name>CallStartDate</name><type>SCHEMA_COLUMN_NAME</type>

151


<binding>CallStartDate</binding><path>n:CallStartDate</path><label>Call start</label><dataType>DATETIME</dataType><hidden>false</hidden><encrypt>false</encrypt><masked>false</masked>

</columns><columns>

<name>CallEndDate</name><type>SCHEMA_COLUMN_NAME</type><binding>CallEndDate</binding><path>n:CallEndDate</path><label>Call end</label><dataType>DATETIME</dataType><hidden>false</hidden><encrypt>false</encrypt><masked>false</masked>

</columns><columns>

<name>CallFromPhoneNumber</name><type>SCHEMA_COLUMN_NAME</type><binding>CallFromPhoneNumber</binding><path>n:CallFromPhoneNumber</path><label>Call from</label><dataType>STRING</dataType><hidden>false</hidden><encrypt>false</encrypt><masked>false</masked>

</columns><columns>

<name>CallToPhoneNumber</name><type>SCHEMA_COLUMN_NAME</type><binding>CallToPhoneNumber</binding><path>n:CallToPhoneNumber</path><label>Call to</label><dataType>STRING</dataType><hidden>false</hidden><encrypt>false</encrypt><masked>false</masked>


</tabs><tabs/>

</panels><panels/>

</data>

ResultConfigurationHelper

Field Description

id Type: UUID

Mandatory: Yes

152


Field Description

name Type: String

Mandatory: Yes

Label: Name


Mandatory: Yes

Label: Application

resultSchema Type: List<String>

Mandatory: Yes

Label: Result Schema

<?xml version="1.0" encoding="UTF-8"?><resultConfigurationHelper xmlns:n="urn:eas-samples:en:xsd:phonecalls.1.0"xmlns:ia="urn:x-emc:ia:schema:pdi"><element><name>id</name><label>ID</label><type>ID</type><path>@ia:id</path>

</element><element><name>SentToArchiveDate</name><label>Sent to</label><type>DATE</type><path>n:SentToArchiveDate</path>

</element><element><name>CallStartDate</name><label>Call start</label><type>DATETIME</type><path>n:CallStartDate</path>

</element><element><name>CallEndDate</name><label>Call end</label><type>DATETIME</type><path>n:CallEndDate</path>

</element><element><name>CallFromPhoneNumber</name><label>Call from</label><type>STRING</type><path>n:CallFromPhoneNumber</path>

</element><element><name>CallToPhoneNumber</name><label>Call to</label><type>STRING</type><path>n:CallToPhoneNumber</path>

</element><element><name>CustomerID</name><label>Customer ID</label>

153


<type>INTEGER</type><path>n:CustomerID</path>

</element><element><name>CustomerLastName</name><label>Last name</label><type>STRING</type><path>n:CustomerLastName</path>

</element><element><name>CustomerFirstName</name><label>First name</label><type>STRING</type><path>n:CustomerFirstName</path>

</element><element><name>RepresentativeID</name><label>Representative ID</label><type>INTEGER</type><path>n:RepresentativeID</path>

</element><element><name>Attachment1</name><label>Attachment 1</label><type>CID</type><path>n:Attachments/n:Attachment[1]/n:FileName/@ia:cid</path>

</element><group><name>Attachments</name><label>Attachments</label><path>n:Attachments/n:Attachment</path><element><name>cid</name><label>CID</label><type>CID</type><path>n:FileName/@ia:cid</path>

</element><element><name>AttachmentName</name><label>Name</label><type>STRING</type><path>n:AttachmentName</path>

</element><element><name>FileName</name><label>Filename</label><type>STRING</type><path>n:FileName</path>

</element><element><name>CreatedBy</name><label>Created by</label><type>STRING</type><path>n:CreatedBy</path>

</element><element><name>CreatedOnDate</name><label>Created on</label><type>DATETIME</type><path>n:CreatedOnDate</path>

</element></group>

</resultConfigurationHelper>

154


Space

Field Description

id Type: UUID

Mandatory: Yes

Label: ID

name Type: String

Mandatory: Yes

Label: Name


Mandatory: Yes

Label: Application

Space Root Folder

Field Description

id Type: UUID

Mandatory: Yes


path Type: String

Mandatory: Yes

Label: Path

space Type: Space

Mandatory: Yes

Label: Space

fileSystemRoot Type: FileSystemRoot

Mandatory: Yes

Label: File system root

155


SpaceRootXdbLibrary

Field Description

id Type: UUID

Mandatory: Yes


name Type: String

Mandatory: Yes

Label: Library name

space Type: Space

Mandatory: Yes

Label: Space

xdbDatabase Type: XdbDatabase

Mandatory: Yes

Label: Xdb Database

Space Root xDB Library

Field Description

id Type: UUID

Mandatory: Yes


name Type: String

Mandatory: Yes

Label: Library name

space Type: Space

Mandatory: Yes

Label: Space

xdbDatabase Type: XdbDatabase

Mandatory: Yes

Label: Xdb Database

156


FileSystemRoot

Field Description

id Type: UUID

Mandatory: Yes


path Type: String

Mandatory: Yes

Label: Path

default Type: Boolean

Mandatory: No

Label: Default

Permit to define a default FileSystemRoot to help to define aworking directory, for example.

FileSystemFolder

Field Description

id Type: UUID

Mandatory: Yes


name Type: String

Mandatory: Yes

Label: Folder name


Mandatory: Yes

Label: Folder sub path

157


Field Description

parentSpaceRootFolder Type: SpaceRootFolder

Mandatory: Yes

Label: Parent space root folder

parentFileSystemFolder Type: FileSystemFolder

Mandatory: Yes

Label: Parent file system folder

Store

Field Description

id Type: UUID

Mandatory: Yes


name Type: String

Mandatory: Yes

Label: Collection name


Mandatory: Yes

Label: Application

type Type: Enum

Mandatory: Yes

Label: Type

FILESYSTEM, XDB, SMARTLOCK, CUSTOM

storeType Type: Enum

Mandatory: Yes

Label: Store type

REGULAR, RESULT

158


Field Description

status Type: Enum

Mandatory: Yes

Label: Status

ONLINE, OFFLINE, READONLY

fileSystemFolder Type: FileSystemFolder

Mandatory: No

Label: File system folder

xdbLibrary Type: XdbLibrary

Mandatory: No

Label: xDB library

javaClass Type: String

Mandatory: No

Label: Java class

For custom extension.

parameters Type: Map<String,String>

Mandatory: No

Label: Parameters

For custom extension.

Tenant

In the build.properties file, do not change the name of the INFOARCHIVE tenant(tenant=INFOARCHIVE). If the INFOARCHIVE tenant is changed, the application will not appearin the InfoArchive web application.

159


Field Description

id Type: UUID

Mandatory: Yes

Label: ID

name Type: String

Mandatory: Yes

Label: Name

Transformation

Field Description

id Type: UUID

Mandatory: Yes

name Type: String

Mandatory: Yes

Label: Name


Mandatory: Yes

Label: Application

inputSchema Type: String

Mandatory: Yes

Label: Input Schema

resultSchema Type: String

Mandatory: Yes

Label: Result Schema

format Type: String

Mandatory: Yes

Label: Format

XSLT

160


XdbDatabase

Field Description

id Type: UUID

Mandatory: Yes

Label: ID

name Type: String

Mandatory: Yes

Label: Name

adminPassword Type: String

Mandatory: Yes

Label: Password

xdbFederation Type: XdbFederation

Mandatory: Yes

Label: Federation

XdbFederation

Field Description

id Type: UUID

Mandatory: Yes

Label: ID

name Type: String

Mandatory: Yes

Label: Name

superUserPassword Type: String

Mandatory: Yes

Label: Super user password

bootstrap Type: String

Mandatory: Yes

Label: Bootstrap

161


xDB Library

Field Description

id Type: UUID

Mandatory: Yes


name Type: String

Mandatory: Yes

Label: Library name


Mandatory: Yes

Label: Library path

detached Type: Boolean

Mandatory: Yes

Label: Detached

detachable Type: Boolean

Mandatory: Yes

Label: Detachable

readOnly Type: Boolean

Mandatory: Yes

Label: Read-only

concurrent Type: Boolean

Mandatory: Yes

Label: Concurrent

parentSpaceRootXdbLibrary Type: SpaceRootXdbLibrary

Mandatory: Yes

Label: Parent space root xDB library

parentXdbLibrary Type: XdbLibrary

Mandatory: No

Label: Parent xDB library

162


Field Description

cacheSupport Type: Boolean

Mandatory: Yes

Label: Cache-out support

Indicate immediately if this library can be cached out.

cacheLockDate Type: OffsetDateTime

Mandatory: Yes

Label: Locked in cache deadline date

cacheInCount Type: Long

Mandatory: Yes

Label: Cache-in count

Number of times the library has been cached in.

cacheInCount Type: OffsetDateTime

Mandatory: Yes

Label: Latest cache-in date

Date of the last cache-in of the library.

size Type: Long

Mandatory: Yes

Label: Size in xDB

indexSize Type: Long

Mandatory: Yes

Label: Index size in xDB

Include path value, Fulltext, Multi path indexes

pkey Type: String

Label: Partitioning key

aipCount Type: Long

Label: Stored AIP count

Number of AIPs stored in the library

163

Field Description

aiuCount Type: Long

Label: Stored AIU count

Number of AIUs stored in the library

effectiveCloseDate Type: OffsetDateTime

Label: Effective close date

Date from when the library became eligible to be closed.

closed Type: Boolean

Label: Closed

’true’ only if library has been closed.

closedDate Type: OffsetDateTime

Label: Closed date

Closing date of the library. A library that has this data withthe NULLDATE value indicates an open library.

closeRequested Type: Boolean

Label: Closing manually requested

Setting this flag allows you to manually request the closingof the library.

xdbMode Type: ENUM

Label: xDB mode

PRIVATE, POOLED, AGGREGATE

Creating an xDB Library with an ANT Script

<echo>Configuring xdb-library object with name '${holding}-xdb-library'</echo><ia-task self="libSelf">

<object typeAlias="xdb-library" checkExistSpEL="?[name=='${holding}-xdb-library']">

<create><name>${holding}-xdb-library</name><subPath>aips/${holding}</subPath><parentSpaceRootXdbLibrary>${rootSelf}</parentSpaceRootXdbLibrary>

</create></object>

</ia-task>

164


XdbLibraryPolicy

The XdbLibraryPolicy object is referenced in the Holding object.

Field Description

aipQuota Type: long

Label: AIP quota per library

Description: Maximum number of AIPs allowed in a library. The valueof zero indicates an unlimited number.

aiuQuota Type: long

Label: AIU quota per library

Description: Maximum number of AIUs allowed in a library. The valuezero indicates an unlimited number.

closeMode Type: ENUM

Label: Close mode

Description: Mode to apply for closing a library of the pool automaticallycreated by InfoArchive. Values include:• NONE,

• CREATION_DATE,

• LAST_MODIFIED_DATE,

• CLOSE_HINT_DATE

closePeriod Type: long

Label: Close period

Description: Period expressed in days used according to the close mode

pkeyQuery Type: String

Label: Partitionning key query

Description: Optional xQuery executed on the sip.xml file.

closeHintDateQuery Type: String

Label: Closing hint date query

Description: Optional xQuery executed on the sip.xml file. Mandatoryif closeMode is set to CLOSE_HINT_DATE.

165


Content

The SDX Partition feature is used with this object. The partition key is the Application UUID.

Field Description

id Type: UUID

Mandatory: Yes


parentId Type: UUID

Mandatory: Yes

Label: Parent ID

AIP, PDI, Ingest, ResultConfigurationHelper, Query,PDI_Schema.

parentType Type: String

Mandatory: No

Label: Parent Type

aip, table, order-item. (value from AuditConstants.Types).


Mandatory: Yes

Label: Application

Required for SDX partition.

store Type: Store

Mandatory: Yes

Label: Store

format Type: String

Mandatory: Yes

Label: Format

modifier Type: String

Mandatory: No

Label: Modifier

166


Field Description

path Type: List<String>

Mandatory: No

Label: Path / External ID

Centera C-CLIPs, Swift, S3

size Type: Long

Mandatory: Yes

Label: Size

checksum Type: String

Mandatory: Yes

Label: Checksum

SHA-1

createdDate Type: OffsetDateTime

Mandatory: Yes

lastModifiedDate Type: OffsetDateTime

Mandatory: Yes

version Type: Integer

Mandatory: Yes

orphan Type: Boolean

Mandatory: Yes

Set to ’true’ to flag this content as ready to be deleted (usefulwhen deleting the parent object).

blob Type: String (Base64)

Mandatory: No

"Turbo Store" for content configuration object only. Limitedto 100 KB.

options Type: List<ENUM>

Mandatory: No

Label: Options

DO_NOT_APPLY_STORAGE_RETENTION

167


SIP Generator

The SIP Generator is a Talend Open Studio output component that allows the customer to generateSIPs to be ingested into InfoArchive via the Talend ETL job. This component is universal, meaning itcan work with any data source provided by Talend Open Studio.

Because the default ulimit value is ’1024’, a customer will receive zip errors when ingesting a largenumber of SIP files . Make sure to set the ulimit value to a bigger value, such as ’10240’. After settingulimit, restart the server.

Building and Installing

1. Run Maven goal assembly:assembly in library (\PlayGround\SPb\kapela\SipCreator).

2. Copy tSipCreator (\PlayGround\SPb\kapela\SipCreator) to Talend components folder (InTalend: Window - Preferences - Talend - Components - User component folder).

3. If Talend could not find sip-creator-*.jar, provide path to it in (\PlayGround\SPb\kapela\SipCreator\target)

The component is named tSipCreator and is located in the InfoArchive. Simply drag and drop itfrom component palette to the job design canvas.

The component has one input connection and no output connections. The input connectionmust have only one column with document type and any name. This document must be valideas_pdi.xml with any AIU count inside. These documents can be formed, for example, in tXMLMapor tFileInputXML. You can provide any count of document in the input connection - componentmerges/splits them, if needed.

You can configure the component in Component Pane. It has following parameters:

Parameter Description Required?

Application Specifies the application that generates theSIP.

Enable XSD validation Enable PDI file validation against XSD Schema

Entity Specifies the business entity that owns thedata to be archived.

Hash algorithm Specifies the algorithm for calculating hashvalue. If none is selected, hash wont becalculated.

Holding Specifies the target holding into which theextracted objects will be archived.

Id Specifies the archiveId.

Output directory Specifies the complete path of the workingdirectory in which to generate SIPs. If thedirectory does not exist, the utility will createit when executed.

168


Parameter Description Required?

Pdi schema Specifies the PDI schema that is written to theSIP files.

Pdi schema version Specifies the PDI schema version that iswritten to the SIP files.

Priority Specifies the ingestion priority of the SIP.The greater the value, the higher the priority.The order by which SIPs are ingestedis determined first by ingestion priority(higher-priority SIPs are ingested first), andthen by ingestion deadline date (SIPs withearlier deadlines are ingested first).

Process unstructured data Enable unstructured data processing.

Producer Specifies the application that generates theSIP.

Split by count Enables splitting packages by number ofobjects in a SIP.

Split by size Enables splitting packages by content file size.

Split count Specifies the maximum number of objects ina SIP.

Required if "Splitby count" ischecked

Split size Specifies the maximum size in bytes forcontent files per SIP.

Required if "Splitby size" is checked

Unstructured data directory Specifies folder, containing unstructured data. Requiredif "Processunstructured data"is checked

xPath Specifies path to unstructured file names ineas_pdi.xml. eg "//Q1:FileName". Connectorexpects to find files with such names in folderspecified in "Unstructured data directory"

Requiredif "Processunstructured data"is checked

XSD File Specify an XSD schema file, for exampleDctmDocbase.xsd (default schema), as theschema (.xsd) file against which the PDI filegenerated by the utility will be validated.

Required if "EnableXSD validation" ischecked

Scenario – Creating a SIP Using Data from MySql

This scenario demonstrates the usage of all the tSipCreator features, including creating SIP package,handling unstructured data, splitting packages by content size and object count, XML validationand hash calculation.

The following illustrates the input data and the expected output data.

169


Input data is stored in MySql database "phonecalls" on localhost in table "calls". The followingillustrates the table structure:

Column FileName represents recording file name and the customer would expect to find a file withthis name in "recordings" folder.

For the output, the customer would expect to see multiple SIPs generated, each containingeas_pdi.xml validated with specified XSD schema, eas_sip.xml and content files.

1. Read data. Add to palette the tMysqlInput component and configure as in the following:

2. Transform data:

a. Add the tXMLMap component to the palette and connect it to tMysqlInput component.

b. In the Component Configuration, open Map Editor.

c. Create a new output row with any name.

d. In the Schema Editor, add a new output Column with a type ’Document’.

e. In the Row Properties, right click on the created column, press Import From file and selectthe root node.

f. Map the data.

g. To generate right date format, open "Tree Schema Editor" and specify the pattern, as follows:

h. Click OK.

3. Create the SIP:

a. Add the tSipCreator component to the palette.

b. Connect tSipCreator with tXMLMap.

c. Configure component, as follows:

4. Run the Talend ETL job. If no error is generated, the SIPs were successfully created and validated.Go to the directory specified in tSipCreator component. The SIPs were generated with eachSIP using less thea 100 KB.

5.Each SIP contains both structured and unstructured data.

Space Limitation for a SIP File NameBe careful when naming a SIP file, as file names containing more than one space will result in anexception during the ingestion process (i.e., Phone CallsSample-2009.zip).

170


The reason is the limitation of the third party – Spring, which normalizes all strings in the way oftrimming all excessive spaces to one space. This limitation cannot be avoided, as a file name thatcontains more than one space looks like a typo rather than a usual use case.

Unitary Archiving and AggregationUnitary archiving allows a client application to synchronously archive data in InfoArchive. If the webservice call succeeds, archived data can be immediately searched. Unitary archiving executes thesame processing as batch ingestion.

Unitary archiving can be completed by an aggregation step to optimize storage and reduce therepository footprint.

The following diagram illustrates unitary archiving at the SDX level:

171


The following diagram illustrates unitary archiving prior to aggregation:

The following diagram illustrates unitary archiving after aggregation:

Receiving and Ingesting a SIP in One Request

When ingesting a SIP in batch, it is first necessary to receive the SIP in the system. This does not,however, make sense with unitary archiving.

To achieve this, an exposed REST API can receive and ingest simultaneously. Receiving an ingestinga SIP in one request reduces the number of steps you must perform (i.e., saving the SIP file). Theresource name is ingest-direct:/applications/{applicationId}/aips?format=sip_zip&ingestDirect=true

172


Configuring the Ingestion Process to Store MultipleAIPs in the Same xDB Library

When working in the aggregation mode, InfoArchive allows you to store metadata into a sharedlibraries, which are also known as pooled libraries. For more information, see XdbLibraryPolicy.

The effective close date is computed with the following rules:

Close Mode Rule

NONE nulldate

LAST_MODIFIED_DATE CLOSE PERIOD + XDB LIBRARY LASTMODIFIED DATE

CREATION_DATE CLOSE PERIOD + XDB LIBRARYCREATION_DATE

CLOSE_HINT_DATE CLOSE PERIOD + XQUERY DATE TIME

The XdbLibraryPolicy is referenced to the Holding: in the following attributes:

• xdbMode

• xdbLibraryPolicy

The following attributes are added at the AIP level:

• xdbMode

• phaseCode

• stateCode

The following attributes are added at the XdbLibrary level to facilitate the library management: 777

• pkey

• aipCount

• aiuCount

• effectiveCloseDate

• closed

• closedDate

• closeRequested

• xdbMode

173


Executing the Close Job to Close the xDB Library andPerform a Backup

When you run the Close job. for each xDBLibrary:

• An xDB backup is performed;

• The XdbLibrary is set to closed = TRUE; and

• The closeDate value is updated.

The backup is only performed if every AIP of the xdbLibrary is set to COMPLETED.

An xDBLibrary with the xdbMode set to PRIVATE is not eligible to be closed. Only an xDBLibrary inxdbMode POOLED or AGGREGATE can be closed:

XdbLibrary where xdbMode in (POOLED,AGGREGATE) and closed = FALSE and(effectiveCloseDate < (offsetDateTime(now) - {closeDelay}) or closeRequested = true)

Note: An exceeded quota never triggers a closure. Only an outdated closing date or a manual closecan be taken into account.

The job accepts the following parameters to control the execution:

• phaseToProcess

• closeDelay

For more information, see the information related to the Close Job in the Working with Jobs: List ofAvailable Jobs section of the Configuration & Administration User Guide.

Configuring the Ingestion Process to Allow Aggregation

The aggregation mode is based on the pooled mode with additional constraints to allow theaggregation at a later time. To activate the aggregation mode, it is necessary to specify at the Holdinglevel the xdbMode AGGREGATE and link a new configuration object XdbLibraryPolicy.

To configure and enable this mode:

1. Update the following values in the 101-xdb-library-policy.xml file:• aipQuota

• aiuQuota

• closeMode

• closeHintDateQuery

• pKeyQuery

• closePeriod

For more information, see XdbLibraryPolicy.

2. in build.properties, change the xdbMode value to AGGREGATE.

174


To close and perform the aggregation, execute the Close job. An aggregate can be closed only:

• When the effectiveCloseDate is expired; or

• If a manual closing has been requested.

At the end, the children (in COMPLETED phase) will be moved to the PRUNE phase and the AIPAGGREGATE will be moved to the COMPLETED phase.

The effectiveCloseDate is computed with the information stored in the XdbLibraryPolicy(101-xdb-library-policy.xml) and the close mode you have chosen:

• closeMode: NONE, CREATION_DATE, LAST_MODIFIED_DATE, CLOSE_HINT_DATE

• closePeriod: number of days

• closeHintDateQuery: XQuery to compute a DateTime on the fly based on the SIP XML information

NONE: nulldate

CREATION_DATE: CLOSE PERIOD + XDB LIBRARY

LAST_MODIFIED_DATE CLOSE PERIOD + XDB LIBRARY LAST

CLOSE_HINT_DATE CLOSE PERIOD + XQUERY DATE TIME

Reception

If XdbMode is set to AGGREGATE, the AIP is not put under retention. The AIP should have isSipLastset to true and SipSeqNo set to 1 or it will fail validation. CommitSync is always enabled, whateverthe value of the holding.

InfoArchive Components Debug LevelConfigurationThis section illustrates where to look for logs and how to switch the log level from info to DEBUG.

In IAShell, it is possible to define an environment variable to provide a file path to the logback.xmlfile.

For Windows

Enter the following in a console window:

set IASHELL_OPTS=-Dlogback.configurationFile=<path_to_the_file>/logback.xml

The following example illustrates the set command if the logback.xml file will be placed in thesame folder as iashell.bat:

175


set IASHELL_OPTS=-Dlogback.configurationFile=logback.xml

The file can be extracted from infoarchive-shell-4.1.0.jar.

For Linux

Enter the export command:

export IASHELL_OPTS=-Dlogback.configurationFile=logback.xml

Enabling Debug Log Level

Add the following section (inside the tag named configuration) to the file for enabling DEBUG level:

<logger name="com.emc.ia.cli" level="DEBUG" additivity="false"><appender-ref ref="CLI-MAIN" /></logger>

Now, IAShell is prepared to use custom settings (DEBUG level) for the logging, which must beexecuted in the same console window:

iashell

Enable the "DEBUG" log level by modifying the application.yml file at <INSTALL_DIR>/infoarchive/config/server/:

logging.level.: 'DEBUG'

For the InfoArchive web application, enable the "DEBUG" log level by modifying theapplication.yml file at <INSTALL_DIR>/infoarchive/config/webapp/

logging.level.: 'DEBUG'

176

Chapter 4Configuring a Table Application

The table archiving approach uses different system data types, namely:

• Tenant

• Application

• Database

• Schema

• Table

The successful execution of the table archiving process requires that these data types be set up in thesystem beforehand. The ’Database’ is a collection of schemas, and each schema is a collection of a listof tables. In case of uploading unstructured content, the system data ’Store’, which is the locationwhere unstructured data will be stored, should also be created.

metadata.xml<metadata>

<caseSensitive>false</caseSensitive><defaultSchema>BASEBALL</defaultSchema><validatingOnIngest>true</validatingOnIngest><locale>en-US</locale><schemaMetadataList>

<schemaMetadata><name>BASEBALL</name><tableCount>24</tableCount><tableMetadataList>

<tableMetadata><name>MASTER</name><recordCount>18589</recordCount><columnList>

<column><name>NAMEFIRST</name><ordinal>14</ordinal><type>VARCHAR</type><typeLength>50</typeLength><index>true</index>

</column><column>

<name>NAMELAST</name><ordinal>15</ordinal><type>VARCHAR</type><typeLength>50</typeLength><index>true</index>

</column></columnList>

</tableMetadata>

177

Configuring a Table Application

</tableMetadataList></schemaMetadata>

</schemaMetadataList></metadata>

The ingestion of the above metadata file creates a schema named "BASEBALL" and one table named"MASTER". Table data can then be ingested on the link referencing to the "MASTER" object.

A Spring shell provides an interactive shell that allows the customer to define custom commandsusing a Spring-based programming model.

A Spring REST template enables communication between HTTP servers and clients with theenforcement of RESTful principles, and supports the following HTTP methods: GET, POST, DELETEand PUT.

The Admin CLI supports two commands: upload and upload-schema.

Using the Upload CommandThe upload command loads table data from the specified file system location into the specifiedtarget InfoArchive database. A Spring REST template discovers the REST resources required toobtain the desired table link that will allow ingestion. For data to be ingested successfully, somesystem data objects must be present and, therefore, must be created prior to ingestion through the’upload’ command. The prerequisite system data objects are: tenant, application, database, schemaand table. Among the aforementioned system data types, schema and table are created during theuploading of table metadata through ’upload-schema’ command. For more information, see Usingthe Upload Schema Command.

Argument Description

--application <name of the application>

--authenticationType <type of authentication to be used as specified by the user>

--database <name of the database>

--debug <log in DEBUG mode>

--error <log in ERROR mode>

The default mode of logging if none of the logging options arespecified.

--filter <pattern based on which the files will be searched from thepath denoted by --path argument. All the files matching thespecified filter will be picked up for ingestion>

Wildcards such as "*" and "?" are supported.

To use:

1. Single argument : --filter = "*.txt" (filters all the text filesfrom the path specified by --path).

2. Multiple arguments have to be separated by a commaenclosed within braces : --filter = "*.{txt,doc}" (filters all

178


Argument Descriptionthe files having extensions ".txt" and ".doc" from the pathspecified by --path).

3. Wildcards that signify multiple characters : --filter ="a*c*e.xml" (filters all the XML files that start with ’a’, endswith ’e’ and has ’c’ in between.There can be any number ofcharacters between ’a’ and ’c’ and between ’c’ and ’e’).

4. Wildcards that signify a single character : --filter ="a?c?e.xml" (filters all the XML files that start with ’a’,ends with ’e’ and has ’c’ in between. There can be just onecharacter between ’a’ and ’c’ and between ’c’ and ’e’).

If no value is specified for this flag by default, only XML fileswill be searched for.

--host <name of the host on which the InfoArchive server is running>

--info <log in INFO mode>

--password <password of the user>

--path <path to a file or a directory containing the files to be ingested>

--port <port on the host on which IA server is running>

--recursive If false, the upload command ignores sub-directories andonly retrieves files from --path location. If true, the uploadcommand also retrieves files from --path sub-directories in adepth-first manner.

--schema <name of the schema>

--table <name of the table>

--tenant <name of the tenant>

--token <web token>

--username <name of the user>

The values specified for --tenant, --application, --database, --schema and --table are their respectivenames. These names are used to discover them using the REST template.

An exception is thrown if one or more mandatory arguments are not found.

Usage Examples

upload --path /home/ia/ingest --tenant test-tenant --application test-application --databasetest-database --schema test-schema --table test-tableScans only the path /home/ia/ingest for *.xml files, logging in ERROR mode

upload --path /home/ia/ingest --tenant test-tenant --application test-application --databasetest-database --schema test-schema --table test-table --recursive

179


scans the path /home/ia/ingest, directories and all the sub-directories recursively, depth first, for*.xml files

upload --path /home/ia/ingest --tenant test-tenant --application test-application --databasetest-database --schema test-schema --table test-table --recursive --debuglogs in the DEBUG mode

Using the Upload Schema Command

The upload-schema command loads table metadata (SQL table definitions) from the specifiedsource file into the specified target InfoArchive database.

It takes 10 parameters: file, tenant, application, database, username, password, host, port,authenticationType and token. For descriptions of these parameters, see the following section, Usingthe Upload Command.

The parameter ’file’ points to the XML document containing definitions of the SQL database thatincludes the records being ingested.

Usage Examples

upload-schema --file <file-path> --tenant <tenant-name> --application <application-name>--database <database-name>.

In the above command, the parameters --tenant, --application and --database contain their respectivenames used by the REST template to discover the underlying entities. Similar to upload command,an exception is thrown if they are not found.

run-chain-of-custody

The run-chain-of-custody command sends a HTTP POST request to the InfoArchive server to executechain of custody tests. These tests are specified in ’file’ – one the command’s parameters. Thespecified tests are executed on a table discovered by the REST template using the provided valuescontained in the command’s parameters. There are 10 parameters that are described in the tabledepicted under the section of the command ’upload’ (except ’file’).

The following is an illustration of the input file whose path is specified in ’file’:<chainofcustody><testsuite>

<test class="com.emc.ia.coc.tests.TableDataTest" /></testsuite>

</chainofcustody>

The above input will run ’TableDataTest’ on the table whose name is specified in the ’table’ parameter.

The following table outlines the five types of tests in chain-of-custody tests:

180


Test Description

TableDataTest Checks whether the ingested XML document of a table containsthe right table name and, if it does, then whether its child XMLelement is ROW or not.

The following illustrates a valid ingested document into theAWARDSMANAGERS table, according to the TableDataTest:

<BASEBALL xmlns:row="urn:x-emc:ia:schema:table">

<AWARDSMANAGERS> <ROW row:id="380d7b6a-a938-426a

-a29f-a91f1b1b8a97"/> </AWARDSMANAGERS> </BASEBALL>

The following illustrates an invalid ingested document into theAWARDSMANAGERS table, according to the TableDataTest:

<BASEBALL xmlns:row="urn:x-emc:ia:schema:table">

<INVALIDTABLENAME> <ROW row:id="380d7b6a-a938

-426a-a29f-a91f1b1b8a97"/> </INVALIDTABLENAME>

</BASEBALL>

The above document would not pass the TableDataTest.

TableRowCountTest Tests that the number of tables listed in the system metadatamatches the number of tables that are ingested. It also tests if thenumber of rows in each table matches the number of rows listedin the table metadata.

ColumnDataTest Tests consistency between metadata and ingested table w.r.tcolumns. Its main tests include: column count and column typeof each row. It has one optional parameter (rows) that specifies alimit on the number of rows to test. This value must be an integer.If the number of rows is negative, it will be set to ’0’ (the defaultvalue), which tests all rows.

SpacesTest Tests for leading and trailing spaces in the ingested data for eachcolumn of rows. It has two required parameters: leading andtrailing; each with a value that specifies the number of leadingor trailing spaces to check. These values must be integers. If thenumber of spaces is not greater than ’0’, it will be set to ’1’. It hasone optional parameter (rows) that specifies a limit on the numberof rows to test. This value must be an integer. If the number ofrows is negative, it will be set to ’0’ (the default value), whichtests all rows.

181


182

Chapter 5Search Composition

Searching Table Data

Using a Common Custom Library in an XQuery forTable-based Searches

When creating a search for a table-based archive, an XQuery has to be created. There are also specificdirections about how XQuery modules can be used. An XQuery module can be seen as a functionlibrary, which can be used by an XQuery.

When creating an XQuery in InfoArchive, you are able to use an XQueryModule object that can beidentified by a namespace. The function library is then referenced from the XQuery by using thenamespace of the XQuery module:

import module namespace iau = 'urn:x-emc:ia:util:fn';

In the XQuery itself, functions of this module can be used by entering the prefix of the module import:

iau:getResultsPage($rows, $page, $size)

An XQuery module is like an XQuery, a piece of text. It starts with a namespace declaration:

module namespace iau = 'urn:x-emc:ia:util:fn';

This module can contain one or more function declarations. Modules can be created at the tenant andapplication levels, and can be used by any XQuery in the scope of the tenant or application.

Use the IAShell and ANT tasks to view, create, update or delete an XQuery module.

Automatic Search Installation for Applicationwith ANTInfoArchive has the capability to automatically install searches within applications.

Sample searches for an application:

• Provide quick possibility to search over a sample application;

• Provide the user with an introduction to the InfoArchive search mechanism with a sampleapplication;

183

Search Composition

• Illustrate the method that searches can be installed automatically within an application; and

• Allows users to share searches among different InfoArchive installations.

All the searches (ANT and ZIP) are located in the <application_name>/searches folder.

The "create-searches" ANT task performs the installation of ZIP searches, previously exported fromthe UI. By default, the "create-searches" task is switched on for all applications.

The "create-searches-ant" ANT task performs installation of searches that are based on a directorystructure.

There is possibility to manage the installation by the cmd property "skipSearches":

• If the property value is "true", then the searches installation is skipped.

• If it is omitted or the value is different from "true", then the searches are installed. For example:

cmd> Tools\applications\PhoneCalls>ant -DskipSearches=true

There is a property for an application – finalSearchState:

• If there is no finalSearchState property in the build.properties file, then search state isdefault and is set to DRAFT.

• If there is a finalSearchState property in the build.properties file of application, then allthe searches are updated after installation with the value of the property.

• The property can be one of two values: PUBLISHED or DRAFT.

There is a property for application – overwriteSearch:

• If there is no overwriteSearch property in the build.properties file, then searches are notupdated in case they existed prior to installation.

• If there is an overwriteSearch property overwriteSearch in build.properties file ofapplication, and its value is ’true’, then searches can be updated with new data during installation

• If there is an overwriteSearch property in the build.properties file of the application, andits value is "false", then searches are not updated with new data during installation

Application Searches Structure

The search structure is almost the same for SIP and table application types, except for the xQueryfolder. The method default searches are installed is also the same for both application types.

Searches Folder

A "searches" folder in the PhoneCalls application contains searches that are automatically installedwithin the application.

The name of the folder under "searches" corresponds to the search name in user interface. ForPhoneCalls, there is the "DefaultSearch" folder and the user interface contains a search with thesame name.

184

Search Composition

In general, every search is presented by several artifacts, which are described in the following sections.

Search Properties File

The first artifact is the search.properties file, which contains the set of properties that areautomatically installed within the search and assigned to it. The following properties are supported:

Property Name Description

searchDescription Contains a short explanation of the search.

aicName Contains the AIC name used by the search.

queryName Contains the query name used by the search.

searchState Values can be:

• Draft: Indicates that the search has not been published and,therefore, cannot be executed by the End User.

• Published: Indicates that the search can be executed by theEnd User.

Search Composition

The second artifact is Search-Composition folder.

By design, InfoArchive’s search functionality are based around using search forms, which allowssearches to be made available for a concrete user group.

Every search form has its own name. The name of the folder corresponds to the name of searchform in the user interface.

A properties file in the Search-Composition folder contains a single property that allows anAdministrator to assign a search form to a specific user group.

Search forms for a SIP application contains two folders, and another for the .

• Search form (XForm) folder: The name of the folder is strongly defined and must not be changed.The folder contains a search form that will be installed for the search. The file under that foldershould be a valid XFrom that contains only InfoArchive web application-supported constructions.

• Result master folder: The name of the folder is strongly defined and must not be changed. Thefolder contains result master that will be installed for the search.

Search Installation

There is special dependency in build.xml file for an application.

185

Search Composition

If a customer includes this task into dependencies, all sample searches from Searches folder areinstalled.

After installation, the user can:

• Edit a search (search form, result master, etc.)

• Duplicate a search form

• Export a search form to a ZIP archive

Creating a Search Form

Perform the following steps to create a search that can be installed automatically with an ANT task:

1. Manually create the search with in the InfoArchive web application.

2. Export the search form from the InfoArchive web application to a ZIP archive.

3. Create a search folder with the search name in the "<application_name>/searches" folder.

4. From the import.descriptor file, copy the "aicName" and "queryName" into the<application_name>/searches/<search_name>/search.properties file forcorresponding properties.

5. From the preliminary.search.resource file, copy the "description" and "state" intothe <application_name>/searches/<search_name>/search.properties file forcorresponding properties.

6. From the searchcomposition.resource" file, copy the name of the search form and create afolder with the same name in the <application_name>/searches/<search_name> folder.

7. Set the groups property to the file <application_name>/searches/<search_name>/<search_composition_name>/search-composition.properties file.

8. Convert formats for XFrom:

a. Take the XFrom file:

b. Take the part of the file that is related to XForm:

c. Replace “\n\r” to an empty string.

d. Replace \” to “

e. Correct the line formatting:

f. Put corrected XFrom in the <application_name>/searches/<search_name>/<search_composition_name>/xforms folder.

Note: You cannot make constructive changes in XFrom.

186

Search Composition

9. Convert formats for the result-master:

a. Take the result-master file.

b. Convert the result-master file from json to xml.

c. Remove "searchName" and "compositionName" from result XML.

d. Put converted XML under the <application_name>/searches/<search_name>/<search_composition_name>/result-master folder.

Note: you cannot make constructive changes in the result-master.

10. After the search structure is prepared, execute the ANT task.

Configuring the Number of Items Listed in theSearch ResultsThe defaultPageSize value in the application.yml indicates the number of result items thatappear in the Record Search page. The default value is 10.

In the following example, the user is changing the default value to 5.

1. Before updating the defaultPageSize, stop all services.

2. Access the application.yml file in the infoarchive > config > server folder.The following information is displayed:infoarchive...rest:......collections.........defaultPage: 0.........defaultPageSize: 10......search:.........defaultTimeOutMs: 8000

3. Change the defaultPageSize value to the desired value (in the following example, the desirednumber of result items is 5):infoarchive...rest:......collections.........defaultPage: 0.........defaultPageSize: 5......search:.........defaultTimeOutMs: 8000

4. Save the change.


187

Search Composition

Configuring the Time Limit for a BackgroundSearchIf a search exceeds the time limit specified in the application.yml file, it automatically becomesa background search. The default time limit set is set to 8,000 milliseconds (8 seconds). You are,however, able to change the time limit.

In the following example, the user is changing the time limit from the default to 5,0000 milliseconds.

1. Before updating the time limit, stop all services.

2. Access the application.yml file in the infoarchive > config > server folder.The following information is displayed:infoarchive...rest:......collections.........defaultPage: 0.........defaultPageSize: 5......search:.........defaultTimeOutMs: 8000

3. Change the defaultTimeOutMs value to the desired time (in the following example, the desiredtime is 5000):infoarchive...rest:......collections.........defaultPage: 0.........defaultPageSize: 5......search:.........defaultTimeOutMs: 5000

4. Save the change.


188

Chapter 6Compliance

Using Audits for ComplianceInfoArchive allows the following audit events:

• Registering an audit: An event can be turned on for auditing (i.e., the customer wants to controlaudit generation for unsuccessful login attempts).

• Creating an audit: The created audit tracks:

— The name of the user who performed the action.

— The name of the object involved in the action.

— The time the operation took place.

• Viewing audit information

• Searching an audit: This allows the customer to look for specific audit events (i.e., the customerwants to view any unsuccessful login attempts).

There is core information that is part of every audit. Additionally, there is event-specific informationadded to an audit.

Use the Admin CLI to add any additional metadata.

Metadata Fields

Field Type Mandatory Comments

id UUID Yes Automatically generated by thesystem

eventType String Yes Event type for the audit (i.e.,application, AIP, table, login,retention policy, etc.)

eventName String Yes Event Name (i.e., view, create,edit, Delete, ingest, export,dispose, etc.).

eventSource String No Additional context dataindicating where the auditevent took place (i.e., disposecould be from the job or auser-defined action).

189

Compliance


auditedObjectId UUID No Unique identifier that isassociated to the object beingaudited (i.e., URI of the retentionpolicy).

applicationName String No Name of the application that theaudit may be part of.

applicationId UUID No ID of the application that theaudit may be a part of

tenantId UUID Yes Tenant that this audit is a part of.

createdby String Yes User who created this event. Thesystem populates this field whenthe audit is created.

createdDate Date (GMT) Yes The date the audit was created.The system sets this field whenthe audit is created

SupplementalData SupplementalData No Describes any additional datafields that the customer wantsadded to the audit event. Specificto the audit event type andinclude information the callerdeems necessary.

The implementation of each auditevent is done by each processthat utilizes the audit system.

Audit Event Type

The audit event type is set to enable or disable audits for a particular audit action/process. If there isno audit configuration set for a particular audit, then auditing is not enabled for that particular event.

InfoArchive includes a fixed list of audits stored in a constants file and uses a REST to retrieve theaudits. Customers can create their own audits using the Admin CLI.

Application-Specific Audits

InfoArchive provides the ability to enable specific audits on an application basis. When creating anapplication name, the event type is optional. If the event type it is not set, then the audit is for allapplications that include that particular event type.

If the audit is specific to an application, and the application name is passed, the system checks for thataudit event combination. If there is no event type that exists, the system checks for an event typewithout the application name to use.

190

Compliance

Metadata Fields


id UUID Yes Automatically generated by thesystem.

type String Yes Object/Event type for the audit(i.e., login, logout, application,AIP, table, hold, retention policy,etc.).

name String Yes The event name (i.e., create, edit,delete, dispose, etc.).

applicationId UUID No ID of the application that thisaudit may be a part of

TenantId UUID Yes Tenant for this event type.

enabled String Yes Defines whether the audit isenabled.

Searching for Audits

Audits are archived into the system as SIPs/AIPs., which allows the customer to perform searches ofthe audit entries.

There is a default search form and result list.

Audit Entries

Audits can be configured through the InfoArchive interface to change the default configuration totrack a specific event. If too many audits are enabled, however, performance will be impacted.

Three levels of audits exist:

• System

• Tenant

• Application

The following table describes the audit type than can be enable in the product for specific actions(see audit name). The event source is the source that can generate the audit. [Not sure it us useful todisplay this table, maybe the ui panels are cleared about audit list].

Type Description

LOGIN Name: SUCCESSFUL/FAILED

Event Source: UserName

191

Compliance

Type Description

LOGOUT Name: SUCCESSFUL

Event Source: UserName

RETENTION_POLICY Name: APPLY/REMOVE/RETRIEVE/CREATE/ UPDATE/ DELETE/REQUALIFY/ FULFILL

Event Source: Ingestion, Search, Job

Application : Application the object is in

Audited Object:• APPLY: ID of the object the policy was applied to.

• REMOVE: ID of the object the policy was removed from.

• ID of the Retention Policy

HOLD Name: APPLY/REMOVE/RETRIEVE/CREATE UPDATE/DELETE

Event Source: Ingestion, Search, Job

Application : Application the object is in

Audited Object:• APPLY: Object the hold was applied to

• REMOVE: Object the hold was removed from

• ID of the Hold

TENANT Name: RETRIEVE/CREATE/DELETE

Application : Application the object is in, if applicable

Audited Object: Object ID of the tenant

FEDERATION Name: RETRIEVE/CREATE

Audited Object: Object ID of the federation

XDBDATABASE Name: RETRIEVE CREATE

Audited Object: Object ID of the xDBDatabase

DATABASE Name: RETRIEVE/CREATE/UPDATE/DELETE


Audited Object: Object ID of the database

LDAP_CONFIGURATION Name: RETRIEVE/CREATE/UPDATE/DELETE

Audited Object: Object ID of the LDAP Configuration

192

Compliance

Type Description

APPLICATION Name: RETRIEVE/CREATE/UPDATE/DELETE DISPOSE (if notfound in controller)


Audited Object: Object ID of the Application

AIC Name: RETRIEVE/CREATE/UPDATE/DELETE/


Audited Object: Object ID of the AIC

AIC also contains the audit name SEARCH and FETCH. FETCH isonly available through the REST API.

AIP Name: RETRIEVE/CREATE/UPDATE/DELETE/ DISPOSE/RECEIVE/INGEST/COMMIT/REJECT/ INVALID/ROLLBACK/CONFIRMATION/ CONFIRMATION_COMPLETED/CHANGE_RETENTION / CONTENT/ CONTENT_DOWNLOADCONTENT_DELETE/ CI

Event Source: RECEIVE: Receiver Node INGEST: IngestNode COMMIT:Commit Job / Ingest Node REJECT:ReceiverNode, End User INVALID: End User ROLLBACK: InvalidJob CONFIRMATION: Confirmation Job CONFIRMATION_COMPLETED: Confirmation Job DISPOSE: Disposition JobCHANGE_RETENTION: CONTENT: CI:


Audited Object: Object ID of the AIP

HOLDING Name: RETRIEVE/CREATE/UPDATE/DELETE


Audited Object: Object ID of the holding

STORE Name: RETRIEVE/CREATE/UPDATE/DELETE


Audited Object: Object ID of the store

INGEST_NODE Name: RETRIEVE CREATE UPDATE DELETE


Audited Object: Object ID of the ingest node

193

Compliance

Type Description

PDI Name: RETRIEVE/CREATE/UPDATE/DELETE


Audited Object: Object ID of the pdi

CONFIRMATION Name: RETRIEVE/CREATE/UPDATE/DELETE


Audited Object: Object ID of the confirmation

QUERY_QUOTAS Name: RETRIEVE/CREATE/UPDATE/DELETE


Audited Object: Object ID of the query quotas

PDI_SCHEMAS Name: RETRIEVE/CREATE/UPDATE/DELETE


Audited Object: Object ID of the pdi schemas

RECEIVER_NODES Name: RETRIEVE/CREATE/UPDATE/DELETE


Audited Object: Object ID of the receiver node

CONFIGURATION_HELPER

Name: RETRIEVE/CREATE/UPDATE/DELETE


Audited Object: Object ID of the configuration helper

RESULT_MASTER Name: RETRIEVE/CREATE/UPDATE/DELETE


Audited Object: Object ID of the result config pages

TYPE_ALIAS Name: RETRIEVE/CREATE/UPDATE/DELETE


Audited Object: Object ID of the type alias

APPLICATION_CATEGORY

Name: RETRIEVE/CREATE/UPDATE/DELETE


Audited Object: Object ID of the application category

194

Compliance

Type Description

JOB_DEFINITION Name: RETRIEVE/CREATE/UPDATE/DELETE/ACTIVE INACTIVE

Event Source:


Audited Object: Object ID of the job definition

JOB_INSTANCE Name: RETRIEVE/CREATE/UPDATE/DELETE/SKIP/UNSKIP

Event Source:


Audited Object: Object ID of the job instance

TABLE Name: RETRIEVE/CREATE/UPDATE/DELETE/DISPOSE


Audited Object: Object ID of the table

TABLE_ROW Name: DISPOSE/CHANGE_RETENTION


SCHEMA Name: RETRIEVE/CREATE/UPDATE/DELETE


Audited Object: Object ID of the schema

SEARCH_GROUP Name: RETRIEVE/CREATE/UPDATE/DELETE


Audited Object: Object ID of the search group

SPACE Name: RETRIEVE/CREATE/UPDATE/DELETE


Audited Object: Object ID of the space

MANAGED_ITEM Name: RETRIEVE/CREATE/UPDATE/DELETE/ APPLY_POLICY/REMOVE_POLICY


Audited Object: Object ID of the managed item

SEARCH Name: RETRIEVE/CREATE/UPDATE/DELETE


Audited Object: Object ID of the search

195

Compliance

Type Description

XFORM Name: RETRIEVE/CREATE/UPDATE/DELETE


Audited Object: Object ID of the xform

RESULT Name: RETRIEVE/CREATE/UPDATE/DELETE


Audited Object: Object ID of the result

QUERY Name: RETRIEVE/CREATE/UPDATE/DELETE/ SEARCH


Audited Object: Object ID of the query

PURGE_LIST Name: GENERATE/APPROVE/ REJECT/REVOKE/DISPOSE/RESUME/ CANCEL


Audited Object: Object ID of the purge list

AUDITS Name: PURGE/ARCHIVE/ENABLE/DISABLE

Event Source: ARCHIVE: Archive audit job/PURGE: Purge audit job

Application : Tenant

Scope: Tenant

User Context:• ARCHIVE: No

• PURGE: No

• ENABLE: Yes

• DISABLE: Yes

SIP-Based Applications and Retention ClassThe retention class defines a logical mapping of one or more retention policies. The retention classis defined in the holding the set of possible retention classes. The holding can also define a defaultretention class.

196

Compliance

Checking an AIP Object’s StateThe following table illustrates how to use IAShell to verify AIP information:

To Use the Following Command

Check that all the AIPs are inCompleted state

select aip --property state

Check that #9 of the AIPs isunder retention

1. Enter the select command:

select aip

2. Enter the view command for the AIP under retention:

view --id 9

Set a variable with the ID valuevar-set --name myAIP --value <copy/past an ID

value from the select output>

If you do not see properties,pre-configure the properties setfor the AIP type alias

set-alias-properties aip --properties “id, name,

dss.holdingName, sipSeqno, sipAiuCount, state,

underHold, underRetention"

Configuring a Custom Retention PolicyThe following example illustrates the process of configuring a custom retention policy named’My-policy’:

1. Create a folder named xml in the $IA_home > tools location and place the following to a filenamed my_retention.xml:

<?xml version="1.0"?><configuration><object typeAlias="retention-policy" checkExistSpEL="?[name=='My-policy']">

<create><name>My-policy</name><agingStrategy><type>FIXED_DATE</type><retainUntil>2017-01-01T10:00:00+01:00</retainUntil>

</agingStrategy><dispositionStrategy><type>DESTROY_ALL</type>

</dispositionStrategy></create>


2. Enter the configure command for the policy:

configure --from xml/my_retention.xml

3. Enter the select command to verify that the policy has been created:

197

Compliance

select --t retention-policy

Setting a Default Retention PolicyThe following example illustrates the process of setting the default retention policy for an applicationnamed PhoneCalls. The retention policy is a custom policy named ’My-policy’, which was configuredin the previous section:

1. Create a file named upd_holding.xml in the XML folder. Enter the following data in the file:

<?xml version="1.0"?><configuration><object typeAlias="holding" checkExistSpEL="?[name=='PhoneCalls']"><update><retentionClasses><name>default</name><policies>My-policy</policies><policies/>

</retentionClasses><retentionClasses/>

</update></object>

</configuration>

2. Enter the select command to select the holding configuration:

select --holding --property id --var-set myHolding

3. Enter the configure command to update the holding configuration:

configure --from xml/upd_holding.xml

4. Enter the view command to verify the result:

view --properties "retentionClasses" --t holding --id #{myHolding}

Alternately, you can also update the object (JSON-like format):

update --id #{myHolding} --properties "{'retentionClasses' : [{'name' :'default', 'policies' :['My-policy'], 'holds':null}]}"

Invalidating an AIPThe following example illustrates how to invalidate the AIP:

1. Enter the select command to select the AIP:

select aip

198

Compliance

2. Enter the invalidate command for the specific AIP being invalidated:

invalidate --id #{myAIP}

Removing a Retention Policy and Deleting anAIPIf you attempt to delete the AIP, and an error message is issued, it means that the AIP cannot bedeleted because it is under a retention policy. The following example illustrates how to removethe retention policy and then delete the AIP.

To remove a retention policy, specify the policy ID and the object ID. Set a variable for this purpose orrefer to the hold by ID value or index.

1. Enter the select command:

select --t retention-policy --where "?[name=='PhoneCalls-policy']" property id--var-set myRetId

2. Enter the remove-retention command:

remove-retention --id #{myRetId} --externalId #{myAip}

An error message is issued if the object is not under the retention policy.

199

EMC InfoArchive 4.1 Command Line Tool Guide · TableofContents RevisionHistory...

Documents

Transcript of EMC InfoArchive 4.1 Command Line Tool Guide · TableofContents RevisionHistory...