SAP HANA File Loader Guide

8/9/2019 SAP HANA File Loader Guide

1/34

PUBLIC

SAP HANA Platform SPS 08

Document Version: 1.1 - 2014-08-21

File Loader Guide for SAP HANAEnabling File Content for Search in SAP HANA


2/34

Table of Contents

1 About this Guide. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Architecture of the File Loader. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4

3 The Development Process. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

4 Tutorial:Setting up the File Loader Example Application. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

4.1 The Setup SQL Script for the Example Application. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9

4.2 The Clean-Up SQL Script for the Example Application. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

4.3 Using theFile Loader Example Application. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

5 The File Loading Process. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

6 State Transitions During the File Loading Process. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

7 The Node Table. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15

8 The Queue Table. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

9 The HTTP Service API. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

9.1 The Root Service. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18

9.2 List of Available Commands. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

9.3 cmdScheduleJob. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

9.4 cmdGetQueueStatistics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21

9.5 cmdSetQueueTimedOut. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

9.6 cmdGetQueueSchedules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .23

9.7 cmdKillJobSchedules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

9.8 cmdUpdateNodeLanguage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

9.9 cmdGetUser. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

9.10 cmdGetSystemInformation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

9.11 cmdCommands. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

10 Security Information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

10.1 Authorizations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

10.2 Roles. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .29

10.3 Authentication. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

10.4 Integration in Application Authorizations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

10.5 Users. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

11 Best Practices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

2P U B L I C2014 SAP SE or an SAP affiliate company. All rights reserved.

File Loader Guide for SAP HANA

Table of Contents


3/34

1 About this Guide

This document describes the concept and the steps an application programmer has to follow to load binaryfiles into SAP HANA with the file loader functionality. Afterwards, it is possible to run services like search on

this content.

Note

Right after reading this document we advise you to read through the Enabling Searchsection in the SAP

HANA Developer Guide. This section provides you with information about how to build SQL search queries

to perform searches on the loaded file contents.

The file loader is a set of HTTP services that you can use to develop your own applications to search in file

contents. The file loader package also contains a basic example application with monitoring and statistical

information about the current file loader schedule.

Note

The file loader supports the loading of file contents for search. To also enable properties and metadata of

files for search, you can extend the node table with additional columns and follow the steps described in the

Enabling Searchsection in the SAP HANADeveloper Guide.

Note

The file loader does not support the loading of files that are only accessible using HTTPS or that require

HTTP authentication.

Related Information

SAP HANA Developer Guide This guide explains how to build applications using SAP HANA, including how to

model data, how to write procedures, and how to build application logic in SAP HANA Extended Application

Services (SAP HANA XS).


About this Guide

P U B L I C2014 SAP SE or an SAP affiliate co mpany. All rights reserved. 3
http://help.sap.com/hana/SAP_HANA_Developer_Guide_en.pdfhttp://help.sap.com/hana/SAP_HANA_Developer_Guide_en.pdfhttp://help.sap.com/hana/SAP_HANA_Developer_Guide_en.pdfhttp://help.sap.com/hana/SAP_HANA_Developer_Guide_en.pdfhttp://help.sap.com/hana/SAP_HANA_Developer_Guide_en.pdf


4/34

2 Architecture of the File Loader

The file loader is used to load the text representation of files that are stored on HTTP servers into the SAPHANA database.

The following diagram shows the architecture of the file loader component in SAP HANA.

Technically, the file loader is an SAP HANA XS application that is shipped as a delivery unit.

The binary file content from the HTTP server is converted and stored as a textual representation in the node

table in the SAP HANA database. The loading can be asynchronous and in parallel. The queue table is used to

track the file loading process.

The file loader exposes an HTTP service API in REST format that can be accessed from any HTTP client.



Architecture of the File Loader


5/34

3 The Development Process

To use the file loader within your own application, you need to work through the following process.

Prerequisites

You have enabled the job scheduling in SAP HANA XS.

Procedure

Prepare the node table.

As an application programmer, you have to provide the node table and the queue table. The node table can

be an existing table that represents one file (URL).

The content column requires the data type BINTEXT.

2 Prepare the queue table.

Give the queue table a primary key that is identical to that of the node table.

3 Create users and assign authorizations.

Create users for the job processing and assign the roles in accordance with the security concept.

4 Fetch a CSRF token.

The file loader HTTP services require CSRF tokens to carry out POSTrequests. Call the root service to

fetch a CSRF token.

5 Schedule a file loader job.

The application should provide a function to start the file loading process. The application uses the

cmdScheduleJobcommand to start the process.

6 Get file loader process information.

The application should determine the status of the file loading process. The application uses the statistics

services of the file loader to determine the current status of the file processing.

7 Stop the file loader process.

The application may want to stop the file loading process. The cmdKillJobSchedulescommand isavailable to stop the job schedules.

Related Information

Tutorial: Setting up the File Loader Example Application[page 7]

This tutorial teaches you how to set up the file loader component to be used in an example application.

The Node Table[page 15]

The application uses a database table that stores the text content of the processed files.

The Queue Table[page 16]


The Development Process



6/34

The queue table is used to track the file loading process of an entry in the node table.

The Root Service[page 18]

The root service is used to obtain the CSRF token.

List of Available Commands[page 19]

The file loader API supports various commands using HTTP GET or POST.



The Development Process


7/34

4 Tutorial: Setting up the File LoaderExample Application

This tutorial teaches you how to set up the file loader component to be used in an example application.

Context

Before you can use the file loader example application, you have to install and configure the component and

set up the user management with minimal authorizations. Then, based on this configuration, you can try out

the example UI that comes with the file loader component.

NoteTo perform the set-up steps, you need a user with system administrator permissions and access to the SAP

HANA XS Administration Tool.

Procedure

Install the file loader component in SAP HANA.

The file loader component comes as an SAP HANA XS delivery unit with SAP HANA.

Import the file loader delivery unit with the name HCO_INA_FILELOAD.tgzwith SAP HANA Application

Lifecycle Management (sap/hana/xs/lm) or SAP HANA studio. After the import, the component is

available and activated in the sap bc ina fileloader job package.

The installation creates a new schema calledSAP_INA_FILELOADERwith the tables

sap.bc.ina.fileloader.db::INA_FILELOADER_JOB_SCHEDULES_T6and

sap.bc.ina.fileloader.db::INA_FILELOADER_JOB_SCHEDULES_V1.

2 Enable job scheduling in the SAP HANA XS.

Note

Job scheduling in the SAP HANA XS is not enabled by default. It has to be enabled by the systemadministrator.

For information about how to enable job scheduling in SAP HANA XS, see the SAP HANA

documentation.

Once job scheduling is enabled, create a new section, scheduler, in the xsengine.inifile. Add a new

property named enabledand set the value to true.

3 Set up users and tables for the example scenario.

Note

Our example uses users with minimal authorizations.


Tutorial: Setting up the File Loader Example Application



8/34

In the SAP HANA studio, execute the example setup SQL script to set up users, authorizations, and

example database tables. For more information, see the Related Informationsection.

The script creates the following users and data:

The access user (INA_FL_TEST_ACCESS), who calls the HTTP services to schedule the file loading

process.

The job access user with minimal authorizations (INA_FL_TEST_JOB_ACCESS) who is used in the

background job.

The job admin user (INA_FL_TEST_JOB_ADMIN) who is used in the SQL connection (xssqlcc).

Example tables.

4 Configure the file loader job.

Note

For this step, you need a user with the sap.bc.ina.fileloader.roles::Accessrole.

Start the SAP HANA XS Administration Tool and activate the file loader job in theApplication Objectstree

under sap bc ina fileloader job job01.xsjob (direct link: /sap/hana/xs/admin/?

package=sap.bc.ina.fileloader.job&object_name=job01&object_type=xsjob).

To activate the job, mark it asand enter the job access user (INA_FL_TEST_JOB_ACCESS) in

thefield. Save the changes.

5 Configure the SQL connection for the file loader job.

Note

For this step, you need a user with the sap.bc.ina.fileloader.roles::Accessand

sap.hana.xs.admin.roles::JobAdministratorroles.

You are still in the SAP HANA XS Administration Tool. In theApplication Objectstree, navigate to the file

loader SQL connection located under sap bc ina fileloader job ileloader.xssqlcc (direct link:

http(s)://:/sap/hana/xs/admin/?

package=sap.bc.ina.fileloader.job&object_name=fileloader&object_type=xssqlcc) and

enter the job admin user (INA_FL_TEST_JOB_ADMIN) in thefield. Save the changes.

6 Start the file loader example application.

Start your browser and open the file loader example application by entering the following address:

http(s)://:/sap/bc/ina/fileloader/app/example/

webfileloader.html.

Log in with the access user INA_FL_TEST_ACCESS.

Only HTTP URLs without authentication and proxy are supported.

7 Use the file content for search.

The node table contains a column of data type BINTEXT. The file loader updates this column with

searchable file content. You can perform a full-text search on this content by executing a SELECT

statement with the CONTAINS() TEXTpredicate.





9/34

The following example SELECTstatement shows how to search in the file content of the example node

table. The SELECTstatement returns the URL, snippets, and highlighted content.

SELECT URL, SNIPPETS("CONTENT"), HIGHLIGHTED("CONTENT") FROM

SAP_INA_FILELOADER_EA_NODE WHERE CONTAINS("CONTENT",'search term')

8 When you have finished running your tests with this tutorial, you should clean up your system.

Run the example clean-up SQL scriptto remove all tutorial data and users generated in the previous steps

and to return to your previous system state.

Related Information

Using the File Loader Example Application[page 11]

The file loader component contains a small browser-based demo UI to show what you can develop and how

you can use the file loader's capabilities.

The Setup SQL Script for the Example Application[page 9]

The setup SQL script creates the users and the example database tables for the file loader example

application tutorial.

The Clean-Up SQL Script for the Example Application[page 11]

The clean-up SQL script removes the users and tables from the file loader example application tutorial,

connects with the access user, and deletes the tables.

4.1 The Setup SQL Script for the Example Application

The setup SQL script creates the users and the example database tables for the file loader example application

tutorial.

-- Set up the users with minimal authorizations for the file loader-- tutorial-- The following users are created:-- 1) The access user calls the HTTP services to schedule the file loading-- process. This user also owns the example node and queue table in the user's-- schema. [INA_FL_TEST_ACCESS]-- 2) The job access user is used within the background job. This user requires-- the same object authorizations as all other access users that are using-- the file loader. [INA_FL_TEST_JOB_ACCESS]-- 3) The job admin user is used in an SQLconnection (xssqlcc) that is used for-- scheduling a job dynamically in the HTTP service.-- This user requires only one application authorization.-- [INA_FL_TEST_JOB_ADMIN]-- Execute this script in the SAP HANA studio as a system administrator.-- After executing this script, go the HANA XS administrator UI and configure-- the SQL connection and job-- Execute the teardown script to delete the users and tablesCONNECTPASSWORD;;

-- Create the file loader access user, this is the end user to be used-- in the example

DROP USER INA_FL_TEST_ACCESS CASCADE;CREATE USER INA_FL_TEST_ACCESS PASSWORD;





10/34

ALTER USER INA_FL_TEST_ACCESS DISABLE PASSWORD LIFETIME;CALL GRANT_ACTIVATED_ROLE('sap.bc.ina.fileloader.roles::Access','INA_FL_TEST_ACCESS');

-- Create the user that is used in the job scheduleDROP USER INA_FL_TEST_JOB_ACCESS CASCADE;CREATE USER INA_FL_TEST_JOB_ACCESS PASSWORD;ALTER USER INA_FL_TEST_JOB_ACCESS DISABLE PASSWORD LIFETIME;CALL GRANT_ACTIVATED_ROLE('sap.bc.ina.fileloader.roles::JobAccess','INA_FL_TEST_JOB_ACCESS');

-- Create the job admin user that is used in the SQL connection-- of the job sap.bc.ina.fileloader.job.xssqlccDROP USER INA_FL_TEST_JOB_ADMIN CASCADE;CREATE USER INA_FL_TEST_JOB_ADMIN PASSWORD;ALTER USER INA_FL_TEST_JOB_ADMIN DISABLE PASSWORD LIFETIME;CALL GRANT_ACTIVATED_ROLE('sap.hana.xs.admin.roles::JobAdministrator','INA_FL_TEST_JOB_ADMIN');

-- Now switch to the access user to create the node and queue table

-- The tables will be used by the example UI-- /sap/bc/ina/fileloader/app/example/webfileloader.htmlCONNECT INA_FL_TEST_ACCESS PASSWORD;

-- Create the node tableDROP TABLE SAP_INA_FILELOADER_EA_NODE CASCADE;CREATE COLUMN TABLE SAP_INA_FILELOADER_EA_NODE ("ID" NVARCHAR(32), "LANGUAGE" NVARCHAR(2), "URL" NVARCHAR(5000), "CONTENT" BINTEXT SYNC LANGUAGE DETECTION ('EN', 'DE', 'FR') PHRASE INDEX RATIO 0.000000 FUZZY SEARCH INDEX OFF SEARCH ONLY OFFFAST PREPROCESS OFF TEXT MINING OFF TEXT ANALYSIS OFF TOKEN SEPARATORS '/;,.:-_()[]!?*@+{}="&', PRIMARY KEY ("ID")) UNLOAD PRIORITY 5 AUTO MERGE;

-- Provide access to the newly-created table job access userGRANT SELECT ON SAP_INA_FILELOADER_EA_NODE TO INA_FL_TEST_JOB_ACCESS;GRANT UPDATE ON SAP_INA_FILELOADER_EA_NODE TO INA_FL_TEST_JOB_ACCESS;

-- Create the queue tableDROP TABLE "SAP_INA_FILELOADER_EA_QUEUE" CASCADE;CREATE COLUMN TABLE "SAP_INA_FILELOADER_EA_QUEUE" ("ID" NVARCHAR(32), "/1ES/_ALTERNATIVE_KEY" NVARCHAR(32), "/1ES/_URL" NVARCHAR(5000), "/1ES/_XS_JOB_NAME" NVARCHAR(500), "/1ES/_FL_JOB_SCHEDULE_ID" NVARCHAR(32), "/1ES/_PACKAGE_ID" NVARCHAR(32), "/1ES/_STATUS" NVARCHAR(50), "/1ES/_HTTP_STATUS_CODE" INTEGER CS_INT, "/1ES/_HTTP_STATUS_TEXT" NVARCHAR(5000),

"/1ES/_TABLE_UPDATE_ERR_TEXT" NVARCHAR(5000), "/1ES/_TEXT_CONV_ERR_CODE" INTEGER CS_INT, "/1ES/_TEXT_CONV_ERR_TEXT" NVARCHAR(5000), "/1ES/_TS_STATUS_NEW" LONGDATE CS_LONGDATE, "/1ES/_TS_STATUS_FILELOAD_START" LONGDATE CS_LONGDATE, "/1ES/_TS_STATUS_FILELOAD_SUCC" LONGDATE CS_LONGDATE, "/1ES/_TS_STATUS_FILELOAD_FAIL" LONGDATE CS_LONGDATE, "/1ES/_TS_STATUS_NODE_T_UP_FAIL" LONGDATE CS_LONGDATE, "/1ES/_TS_STATUS_NODE_T_UP" LONGDATE CS_LONGDATE, "/1ES/_TS_STATUS_TEXT_CONV_FAIL" LONGDATE CS_LONGDATE, "/1ES/_TS_STATUS_TIMED_OUT" LONGDATE CS_LONGDATE, "/1ES/_TS_STATUS_SUCCESS" LONGDATE CS_LONGDATE, "/1ES/_ADDITIONAL_INFO" NVARCHAR(5000), PRIMARY KEY ("ID")) UNLOAD PRIORITY 5 AUTO MERGE;

-- Provide access to the newly-created table job access user

GRANT SELECT ON SAP_INA_FILELOADER_EA_QUEUE TO INA_FL_TEST_JOB_ACCESS;





11/34

GRANT UPDATE ON SAP_INA_FILELOADER_EA_QUEUE TO INA_FL_TEST_JOB_ACCESS;

CONNECTPASSWORD;

4.2 The Clean-Up SQL Script for the Example Application

The clean-up SQL script removes the users and tables from the file loader example application tutorial,

connects with the access user, and deletes the tables.

-- Clean up the users and tables of the fileloader tutorial-- Connect with the access user and delete the tablesCONNECT INA_FL_TEST_ACCESS PASSWORD;

DROP TABLE SAP_INA_FILELOADER_EA_NODE CASCADE;DROP TABLE SAP_INA_FILELOADER_EA_QUEUE CASCADE;

-- Connect with the system administrator and delete the usersCONNECTPASSWORD;

-- Delete the fileloader access user, this is the end user to be used-- in the exampleDROP USER INA_FL_TEST_ACCESS CASCADE;

-- Delete the user that is used in the job scheduleDROP USER INA_FL_TEST_JOB_ACCESS CASCADE;

-- Delete the job admin user that is used in the sql connection of-- the job sap.bc.ina.fileloader.job.xssqlccDROP USER INA_FL_TEST_JOB_ADMIN CASCADE;

4.3 Using the File Loader Example Application

The file loader component contains a small browser-based demo UI to show what you can develop and how

you can use the file loader's capabilities.

Context

The file loader example application is an implementation of the file loader functionality using JavaScript with a

web frontend. You can enter a number of URLs for documents that you want to upload into SAP HANA with the

file loader. You can then search for content inside the uploaded documents.

Procedure

Start the file loader example application.





12/34

Open your browser and start the file loader example application by entering the following

address:/sap/bc/ina/fileloader/app/example/webfileloader.html.

Log in with the access user INA_FL_TEST_ACCESS. On the initial screen, there is a text field containing

URLs. Use copy and paste to replace these URLs with the URLs for your own documents.

Note

Only HTTP URLs without authentication and proxy are supported.

2 Change the default options.

Clicking the +icon on the initial screen of the user interface shows an options panel where you can change

some basic options: for example, the number of packages, schedules and the frequency, as well as the

names of the queue table and node table.

Name of Option Description

Number of Packages Limits the number of work packages that are used

to upload the documents. The packages are used

to divide the workload and so minimize the risk of

locking errors on database tables.

Number of Schedules Limits the number of job schedules that are used to

upload the documents for each package.

Schedule Timeout Limits amount of time that processing can run

overall.

Schedule Frequency in Minutes Limits how long a job schedule runs before it isstopped and restarted.

Queue Table .

The default for the schema is the current user, but

you can manipulate this to upload data for another

user.

Node Table .

Te default for the schema is the current user, but

you can manipulate this to upload data for another

user.

3 Start the processing.

When you choose Start loading, the results screen appears. The top half contains the Queue monitor,

which displays the current status of the files you chose to process.

The bottom half of the screen contains status messages.

Note

If you use the delivered example URLs, one URL will fail and show an error message instead. This is

intended.





13/34

5 The File Loading Process

The file loading process involves three steps: scheduling the job, getting the file, and converting the filecontent.

The file loading process is ready to start as soon as the node table and the queue table are available and

populated with data.

After the files have been processed, the application can use the extracted and converted content of the node

table. The queue table can be used by the application for cleanup processes if errors occur.

The file loader provides HTTP services to retrieve the current status of the data processing to determine the

process status for long-running processes.

Schedule the job.

The application calls the file loader HTTP service to schedule the job for the data extraction and

conversion. The application provides the table names, timeouts, and number of parallel schedules. The file

loader schedules the file loader job multiple times in parallel. The schedules use the queue table to

synchronize the processing and to save the current loading status.

2 Get the file content.

Every job schedule retrieves the binary file content from the HTTP servers using the URLs.

3 Convert the file content.

Every job schedule converts the received binary file to a textual representation. The textual representation

is stored in the content column of the node table.


The File Loading Process



14/34

6 State Transitions During the File LoadingProcess

The file loader returns several states and helps you to monitor the process.

The state of the file loading process is stored in the queue table column/1ES/_STATUS.

The following values are available:

NEW

FILE_LOADING_IN_PROCESS

FILE_LOADING_FAILED

FILE_LOADING_SUCCESSFUL

TEXT_CONVERSION_IN_PROGRESS

NODE_TABLE_UPDATE_FAILED

TEXT_CONVERSION_FAILED

SUCCESS

TIMED_OUT_WHILE_INDEXING



State Transitions During the File Loading Process


15/34

7 The Node Table

The application uses a database table that stores the text content of the processed files.

The file loader supports any structure for the table, but needs one column for the text content.

Note

The content column must have the data type BINTEXT.

The node table can have a language column (NVARCHAR(2)). This column can be used to store the language of

the file.

Example

CREATE COLUMN TABLE "INA_FL_TEST_ACCESS"."SAP_INA_FILELOADER_EA_NODE" ( "ID" NVARCHAR(32), "LANGUAGE" NVARCHAR(2), "URL" NVARCHAR(5000), "CONTENT" BINTEXTSYNC LANGUAGE DETECTION ('EN','DE','FR')

PHRASE INDEX RATIO 0.000000 FUZZY SEARCH INDEX OFFSEARCH ONLY OFF FAST PREPROCESS OFF TEXT MINING OFFTEXT ANALYSIS OFF TOKEN SEPARATORS '/;,.:-_()[]!?*@+{}="&',

PRIMARY KEY ("ID")) UNLOAD PRIORITY 5 AUTO MERGE

Related Information

The Queue Table[page 16]


cmdUpdateNodeLanguage[page 25]

Use this command when you want to store the file's language in the node table.


The Node Table



16/34

8 The Queue Table


The table shows the progress of the file loading so that you can take action if data loading problems occur.

Every entry in the node table has a corresponding entry in the queue table. The node table controls the data

loading process and is used by the file loader jobs. The primary keys of the queue table are identical to those of

the node table. All other columns are determined by the file loader.

Example

The primary keys must be identical to the primary keys of the node table.

In this example, the column IDis the primary key. All other columns /1ES/_*must have the structure

described.

CREATE COLUMN TABLE "MYSCHEMA"."MY_QUEUETABLE"("ID" NVARCHAR(32), "/1ES/_ALTERNATIVE_KEY" NVARCHAR(32), "/1ES/_URL" NVARCHAR(5000), "/1ES/_XS_JOB_NAME" NVARCHAR(500), "/1ES/_FL_JOB_SCHEDULE_ID" NVARCHAR(32), "/1ES/_PACKAGE_ID" NVARCHAR(32), "/1ES/_STATUS" NVARCHAR(50), "/1ES/_HTTP_STATUS_CODE" INTEGER CS_INT, "/1ES/_HTTP_STATUS_TEXT" NVARCHAR(5000),

"/1ES/_TABLE_UPDATE_ERR_TEXT" NVARCHAR(5000), "/1ES/_TEXT_CONV_ERR_CODE" INTEGER CS_INT, "/1ES/_TEXT_CONV_ERR_TEXT" NVARCHAR(5000), "/1ES/_TS_STATUS_NEW" LONGDATE CS_LONGDATE, "/1ES/_TS_STATUS_FILELOAD_START" LONGDATE CS_LONGDATE, "/1ES/_TS_STATUS_FILELOAD_SUCC" LONGDATE CS_LONGDATE, "/1ES/_TS_STATUS_FILELOAD_FAIL" LONGDATE CS_LONGDATE, "/1ES/_TS_STATUS_NODE_T_UP_FAIL" LONGDATE CS_LONGDATE, "/1ES/_TS_STATUS_NODE_T_UP" LONGDATE CS_LONGDATE, "/1ES/_TS_STATUS_TEXT_CONV_FAIL" LONGDATE CS_LONGDATE, "/1ES/_TS_STATUS_TIMED_OUT" LONGDATE CS_LONGDATE, "/1ES/_TS_STATUS_SUCCESS" LONGDATE CS_LONGDATE, "/1ES/_ADDITIONAL_INFO" NVARCHAR(5000), PRIMARY KEY ("ID")) UNLOAD PRIORITY 5 AUTO MERGE

To populate the queue table, you need to create an entry for every entry of the node table. Fill the requiredfields of the queue table entry. All other fields are updated by the file loader during processing.

The following columns are required:

Primary Key: Use identical primary keys for the node table and the queue table.

URL: Provide an absolute HTTP URL that targets a file in the /1ES/_URLcolumn.

Initial Status: To indicate that the entry has to be processed by the file loader, set the /1ES/_STATUS

column to the value "NEW".and set the time stamp column /1ES/_TS_STATUS_NEWto the current time

stamp.



The Queue Table


17/34

Related Information

The Node Table[page 15]The application uses a database table that stores the text content of the processed files.


The Queue Table



18/34

9 The HTTP Service API

The file loader API is exposed as an HTTP service that offers various commands.

The various operations that can be executed with this service are called commands. A command is described

as a JSON object.

Command JSON: The HTTP service API uses a JSON format to describe commands that should be executed.

{command:,parameter:{< parameter object>}}

The command must have a command property to provide a command name as a string (for example

cmdScheduleJob). The second property is the optional parameter property that describes the data that is

required by the command.

The service URL has the following structure:

http(s)://:/sap/bc/ina/fileloader/service.xsjs

HTTP GET: Services that provide information and do not change the state can be invoked using the HTTP GET

method. The file loader service supports the commandparameter.

http(s)://:/sap/bc/ina/fileloader/service.xsjs?command=

HTTP POST: Services that change the state must be invoked using the HTTP POST method. Information is

passed in the HTTP POST body.

http(s)://:/sap/bc/ina/fileloader/service.xsjs

The HTTP body contains the command in JSON format:

If you use lowercase letters and/or special characters for queue, node table or attribute data, you do not need

to use quotation marks (). The file loader API behaves differently from the SAP HANA SQL interface in this

regard, to make it easier to use.

Related Information

List of Available Commands[page 19]


9.1 The Root Service

The root service is used to obtain the CSRF token.

When service.xsjsis invoked using an HTTP GETrequest, an HTML response is provided. The CSRF token

can also be fetched.



The HTTP Service API


19/34

9.2 List of Available Commands


Table 1: List of available commands

Command Name HTTP Method Description

cmdScheduleJob POST Schedule the file loader job.

cmdGetQueueStatistics POST, GET Returns statistical information

about the current package of

queue items.

cmdSetQueueTimedOut POST Stops the processing when timed

out.

cmdGetQueueSchedules GET, POST Returns the number of schedules

and the active schedules of the

given queue. Used for monitoring

purposes.

cmdKillJobSchedules POST Stops the schedules for all pack

ages and sets the status timed out

for all unprocessed files.

cmdUpdateNodeLanguage POST Updates the node table language

column.

cmdGetUser GET, POST Gets the currently logged on username language column.

cmdGetSystemInformation GET, POST Gets information about the sys

tem.

cmdCommands GET, POST Returns all supported commands

of the file loader.

9.3 cmdScheduleJob

The schedule job command creates multiple schedules for the specified node tables and queue tables that

process the tables in parallel.

The maximum number of schedules limits the number of parallel executions and the timeout limits the

execution time of the complete process. The frequency specifies how long a schedule should run until the

schedule is restarted.

HTTP Method: POST





20/34

Command

{ "command": "cmdScheduleJob", "parameter": { "fileLoaderRequest": { "table": { "queueTable": { "schema": "< Schema name of the queue table >", "name": "< Name of the queue table >", "packageId": "< optional unique id of a package >" }, "nodeTable": { "schema": "< Schema name of the node table >", "name": "< Name of the queue table >", "fileContentAttributeName": "< Content attribute name >" } }, "schedule": { "frequencyMinutes": "< Minutes until the schedule restarts >", "maxNumberOfSchedules": "< Maximum number of parallel jobschedules>", "timeOutMinutes": "< Minutes to complete the loading process >" } } }}

Return

{ "statusCode": "", "message": { "text": "", "detailMessages":[

{"message": "","code": } ] }, "schedule": { "jobName" : < id of the job>, "numberOfSchedules" < number of schedules >, "scheduleDetails": [

{ "fileloaderScheduleId": "", "xsEngineScheduleId": "< schedule id XS >" } ] }}





21/34

9.4 cmdGetQueueStatistics

This command returns statistical information about the current package of queue items.

If no package is specified, all data is used for the statistics. These statistics are used by the application to

decide on further processing or for monitoring.

HTTP Method: POST, GET

Command

{

"command": "cmdGetQueueStatistics", "parameter": { "fileLoaderRequest": { "table": { "queueTable": { "schema": "< Schema name of the queue table >", "name": "< Name of the queue table >", "packageId": "< optional unique ID of a package >" } } } }}

Return

The detailed status information is given for all individual statuses; the summaries are calculated as follows:

unprocessedCount Unprocessed files newCount

inProcessCount Files that are currently being re

trieved from the server or are in

text conversion

fileloadingInProcessCount + text

ConversionInProcessCount + file

loadingSuccessfulCount

successCount Number of files that have been

processed successfully

successCount

failCount Number of files that could not be

processed successfully and the

files that could not be processed

due to the timeout

fileloadingFailedCount + nodeTa

bleUpdateFailedCount + textCon

versionFailedCount + timedOut

Count

totalCount All files of the queue table failCount + successCount + inPro

cessCount + unprocessedCount

{ "statusCode": ,





22/34

"message": { "text": "", "detailMessages":[

{"message": "","code": }

] }, "queue":

{"statusDetail":{ "newCount": "" },

{ "successCount": "" }, { "timedOutCount": "" }, { "fileloadingInProcessCount": "" }, { "fileloadingFailedCount": "" }, { "fileloadingSuccessfulCount": "" }, { "nodeTableUpdateFailedCount": "" }, { "textConversionInProcessCount": "" }, { "textConversionFailedCount": "" } }, "statusSummary": { { "unprocessedCount": "" },

{ "inProcessCount": "" }, { "successCount": "" }, { "failCount": "" }, { "totalCount": }, } }}

9.5 cmdSetQueueTimedOut

Executing this command stops the processing of files.

The job schedules are created with an overall timeout. The job schedule automatically stops the processing

when the timeout is reached. However, the application can use this command to stop the processing earlier by

setting unfinished or unprocessed files as timed out. Once you have executed this command, the running job

schedules will not find any unprocessed files and will stop the processing. This service processes a subset of

all node table entries. To update all node table entries, call the service until the number of updated records is

0.

HTTP Method: POST

Command

{ "command": "cmdSetQueueTimedOut", "parameter": { "fileLoaderRequest": { "table": { "queueTable": { "schema": "< Schema name of the queue table >", "name": "< Name of the queue table >",

"packageId": "< Optional unique ID of a package >" }





23/34

} } }}

Return


{"message": "","code": } ] }, "table": { numberOfUpdatedRecords : }}

9.6 cmdGetQueueSchedules

This command returns the number of schedules and the active schedules of the given queue. It is used for

monitoring purposes.

HTTP Method: GET, POST

Command

{ "command": "cmdGetQueueSchedules", "parameter": { "fileLoaderRequest": { "table": { "queueTable": { "schema": "< Schema name of the queue table >", "name": "< Name of the queue table >" } } } }}





24/34

Return


{"message": "","code": } ] }, "schedule": { "jobName" : < id of the job>, "numberOfSchedules" < number of schedules >, "scheduleDetails": [ { "fileloaderScheduleId": "", "xsEngineScheduleId": "< schedule id XS >"

} ] }}

9.7 cmdKillJobSchedules

This command stops the schedules for all packages and sets the status "timed out" for all unprocessed files.

Note

This command is reserved for use in error or emergency situations only, to stop the processing

immediately.

HTTP Method: POST

Command

{ "command": "cmdKillJobSchedules", "parameter": { "fileLoaderRequest": { "table": { "queueTable": { "schema": "< Schema name of the queue table >", "name": "< Name of the queue table >" } } } }

}





25/34

Return


{"message": "","code": } ] }, "schedule": { "jobName" : < id of the job>, "numberOfSchedules" < number of schedules >, "scheduleDetails": [ { "fileloaderScheduleId": "", "xsEngineScheduleId": "< schedule id XS >" } ] }}

9.8 cmdUpdateNodeLanguage

Use this command when you want to store the file's language in the node table.

The text processor is able to determine the language of the file when it is converted to text. If the node tablehas a language column, this command copies the determined language to the node table column.

This service processes a subset of all node table entries. To update all node table entries, call the service until

the number of updated records is 0.

HTTP Method: POST

Command

{ "command": "cmdUpdateNodeLanguage", "parameter": { "fileLoaderRequest": { "table": { "queueTable": { "schema": "< Schema name of the queue table >", "name": "< Name of the queue table >", "packageId": "< optional unique id of a package >" }, "nodeTable": { "schema": "< Schema name of the node table >", "name": "< Name of the queue table >", "fileContentAttributeName": "< Content attribute name >", "fileLanguageAttributeName": "< Language attribute name >",

"useAbapLanguageFormat": "}





26/34

}, } }}

Return

If the returned number of updated records is 0, all columns are updated.

{ "statusCode": "", "message": { "text": "",

"detailMessages":[{"message": "","code": } ] }, "table": { numberOfUpdatedRecords : }}

9.9 cmdGetUser

This command returns the user name of the user being logged on.


Command

{"command": "cmdGetUser"}

Return

If the returned number of updated records is 0, then all columns are updated.

{ "user": { "name": "< user name >" }

}





27/34

9.10 cmdGetSystemInformation

This command returns information about the current system.


Command

{ "command": "cmdGetSystemInformation", // optional parameter to restrict the output to the desired sections only

"parameter": [ "User", "Time", "XS" ]}

Return

The command returns the following information about the current system.

{ "XS": { },

"Fileloader": { }, "Time": { }, "User": { }, "System": { }, "Services": { }, "Memory": { }, "CPU": { }, "Disk": { }, "Statistics": { }}

9.11 cmdCommands

This command returns all supported commands of the file loader.


Command

{ "command": "cmdGetCommands",





28/34

"parameter": { "packageName": "sap.bc.ina.fileloader.cmd"}}

Return

[{

"name": "", "description": "", "accessMethods": [ "HTTP POST","HTTP GET","JavaScript" ], "privilege": "sap.bc.ina.fileloader::", }]





29/34

10 Security Information

This sections contains security-related information like authorizations, users, and roles that are used by thefile loader component.

The file loader HTTP services can be used remotely by applications. The services ensure authentication and

authorization, and prevent cross-site request forgery (CSRF).

Note

Ensure that you assign minimal authorizations to users.

10.1 Authorizations

The file loader component defines the following application authorizations.

Type of uthorization Description

Monitoring The monitoring authorization allows users to access the HTTP service with

read-only access.

Access The access authorization allows users to influence (start and stop) the file

loading process using the HTTP service.

JobAccess This authorization defines the access to the file loader job script. It pre

vents non-authorized users from executing the job script.

10.2 Roles

The file loader roles define the minimum authorizations and access types that are required.

Type and Name of Role Description

Monitoring

sap.bc.ina.fileloader.ro

les::Monitoring

Use the monitoring role when a component uses file loader commands to

monitor the current status of the file loading process. The access rights

are limited to read-only access to the job schedules and the file loader

queues. The role includes the file loader monitoring authorization and ob

ject authorization to select on the job schedule table and framework ta

bles.

Access


les::Access

Use the access role when a file loader client component requires to sched

ule jobs or to stop jobs. This role also includes all of the access rights of

the monitoring role. The role inludes the file loader access application au

thorization and object authorization to select/update/insert/delete on the

job schedule table and select on framework tables.


Security Information



30/34

Type and Name of Role Description

JobAccess


les::JobAccess

This role is used by the technical user (XSSQLCC) that accesses the file

loader job. The technical user

sap.bc.ina.fileloader.job.fileloader.xssqlccneeds this role

to operate correctly.

10.3 Authentication

The file loader HTTP services support the authentication methods of SAP HANA XS.

10.4 Integration in Application Authorizations

Client applications have to fulfill certain prerequisites to be used with the file loader component.

The file loader works with the node table and the queue table that are defined by the application.

In addition to the authorizations that are defined in the file loader roles, the file loader client application has todefine the object authorization for the queue table and the node table. The file loader requires SELECTand

UPDATEauthorization for the two tables.

Example

This example shows how to define two file loader users with application-specific roles that includes the file

loader roles.

CREATE ROLE MY_APP_ROLE;GRANT SELECT ON MY_SCHEMA.MY_QUEUE_TABLE TO MY_APP_ROLE;GRANT UPDATE ON MY_SCHEMA.MY_QUEUE_TABLE TO MY_APP_ROLE;GRANT SELECT ON MY_SCHEMA.MY_NODE_TABLE TO MY_APP_ROLE;GRANT UPDATE ON MY_SCHEMA.MY_NODE_TABLE TO MY_APP_ROLE;

CREATE USER MY_USER PASSWORD ;CALL GRANT_ACTIVATED_ROLE('sap.bc.ina.fileloader.roles::Access','MY_USER');GRANT MY_APP_ROLE TO MY_USER;

CREATE USER MY_JOB_USER PASSWORD ;CALLGRANT_ACTIVATED_ROLE('sap.bc.ina.fileloader.roles::JobAccess','MY_JOB_USER');GRANT MY_APP_ROLE TO MY_JOB_USER;





31/34

10.5 Users

There are a number of users that require particular authorizations for the file loader.

User Description

System Administrator This user needs access to the SAP HANA XS adminis

tration tool to check the status of the job scheduler.

File Loader REST Service User This user uses the file loader HTTP service. The user

needs to be assigned the file loader Monitoringor

Accessrole and requires application-specific object

authorizations.

File Loader Job User The job user is a technical user that is used to execute the file loader job. This user needs to be as

signed the file loader JobAccessrole and requires

application-specific object authorizations.





32/34

11 Best Practices

The requirements for optimal usage of the file loader component are listed below.

The overall execution time of the file loading process should be as short as possible.

2 The load on the SAP HANA system should be minimal while the file loading process is running.

3 All files should be processed successfully.

Determine the optimal number of parallel schedules

The loading time can be reduced by adding additional parallel schedules to the file loader job.

The number of parallel schedules reduces the overall execution time of the file loading process.

There should not be more parallel schedules than files to process. A high number of schedules can also

lead to locking overhead while updating the queue table.

The overall SAP HANA system load is also increased. The system administrator has to balance the file

loader load with the remaining load of the SAP HANA system.

As well as the load on the SAP HANA system, the load on the HTTP servers where the remote files are

located is increased. If the remote HTTP servers cannot handle the load, the response times might

increase, or the servers might not provide any response.

If new schedules are added to the file loading process, consider resizing the remote HTTP servers.



Best Practices


33/34

Important Disclaimers on Legal Aspects

This document is for informational purposes only. Its content is subject to change without notice, and SAP does not warrant that it is error-free. SAP MAKES NOWARRANTIES, EXPRESS OR IMPLIED, OR OF MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.

Coding Samples

Any software coding and/or code lines / s trings ("Code") included in this documentation are only examples and are not intended to be used in a productive system

environment. The Code is only intended to better explain and visualize the syntax and phrasing rules of certain coding. SAP does not warrant the correctness and

completeness of the Code given herein, and SAP shall not be liable for errors or damages caused by the usage of the Code, unless damages were caused by SAP

intentionally or by SAP's gross negligence.

Accessibility

The information contained in the SAP documentation represents SAP's current view of accessibility criteria as of the date o f publication; it is in no way intended to be

a binding guideline on how to ensure accessibility of software products. SAP specifically disclaims any liability with respect to this document and no co ntractualobligations or commitments are formed either directly or indirectly by this document.

Gender-Neutral Language

As far as possible, SAP do cumentation is gender neutral. Depending on the context, the reader is addressed directly with "you", or a gender-neutral noun (such as

"sales person" or "working days") is used. If when referring to members of both sexes, however, the third-person singular cannot be avoided o r a gender-neutral noun

does not exist, SAP reserves the right to use the masculine form of the noun and pronoun. This is to ensure that the documentation remains comprehensible.

Internet Hyperlinks

The SAP documentation may contain hyperlinks to the Internet. These hyperlinks are intended to serve as a hint about where to find related information. SAP does

not warrant the availability and correctness of this related information or the ability of this information to serve a particular purpose. SAP shall not be liable for any

damages caused by the use of related information unless damages have been caused by SAP's gross negligence or willful misconduct. Regarding link classification,

see: http://help.sap.com/disclaimer.


Important Disclaimers on Legal Aspects

http://help.sap.com/disclaimer/


34/34

www.sap.com/contactsap

2014 SAP SE or an SAP affiliate company. All rights reserved.

No part of this publication may be reproduced or transmitted in any

form or for any purpose without the express permission of SAP SE

or an SAP affiliate company. The information contained herein may

be changed without prior notice.

Some software products marketed by SAP SE and its distributors

contain proprietary software components of other software

vendors. National product specifications may vary.

These materials are provided by SAP SE or an SAP affiliate company

for informational purposes only, without representation or warranty

of any kind, and SAP or its affiliated companies shall not be liable for

errors or omissions with respect to the materials. The only

warranties for SAP or SAP affiliate company products and services

are those that are set forth in the express warranty statements

accompanying such products and services, if any. Nothing herein

should be construed as constituting an additional warranty.

SAP and other SAP products and services mentioned herein as wellas their respective logos are trademarks or registered trademarks
http://www.sap.com/contactsap

SAP HANA File Loader Guide

Documents

Transcript of SAP HANA File Loader Guide