Automating Banner Financial Aid Dataload - San Jacinto College

Complete Automation of the Impossible,Financial Aid Dataload.

November 2, 2015

Mark Deaver

3 Property of Automic Software. All rights reserved

Agenda+ Background

+ Benefits

+ Requirements

+ Financial Aid Dataload Automic Example


• The Financial Aid Office would manually download ISIR files via EdConnect.

• Financial Aid Office downloaded files for a given aid year once a week.• ISIR Files are merged into one file.• The concatenated file is transferred to the dataload directory.• The financial aid staff manually processed RCPTPxx, RCPMTCH and

RCRTPxx Banner jobs.

Background


• Increased productivity.• No manual intervention required.• Daily processing of student aid record files.• Streamlined processing ensures:

– No Duplication– No Omission of files

• Notification of all automated processes.• Output can be immediately emailed as text files or PDF files.

Benefits to Complete Automation.


• The Applications Manager internal user id will need read/write access to various directories on the server.

• TDClient – Department of Education mainframe communication software.– Does not replace EdConnect.– Method to transfer and receive files via a mainframe or server.– Proper Setup is key –

• Password Initialization• Secfile Initialization

• Advanced Scripting knowledge.• Automic Applications Manager.• A Documented Process.

Requirements


• The Applications Manager internal owner user id will need read/write access to the following directories:

• TDClient directory• Automic directory• Dataload directory

• Various jobs within Applications Manager require access to various directories.

• Dataload process transfers files to and from various directories.

Requirements – Applications Manager server level access


• TDClient– Mainframe communication software for sending/receiving files via the SAIG

mailbox network.– Uses existing TG number and password.– Software needs to be installed on the same server as the Application Manager.– EdConnect is still used to download non-automated files.– Software is provided free of charge by the Department of Education.– Instructions to install are provided on the Department of Education website.

Requirements - TDClient


• Applications Manager can process different types of script files.• Applications Manager can process PERL (pl), SQL and SHELL scripts,

etc.• Script files can handle specific functions and greatly enhance automation.• Jobs within Applications Manager can process a script file as it would any

other normal job.

Requirements – Advanced Scripting Knowledge


• Applications Manager – Automic automation software.

Requirements – Applications Manager


• A documented process is a must.• A detailed step by step process that explains the steps, the jobs and the

parameters.• A documented process will need the following:

– A static set of parameters.– A list of sequential jobs to be accomplished.– List of requested output.– List of email recipients.– List of email verbiage.

Requirements – Documented Process


Financial Aid Dataload – Automation Example using Applications Manager.


• The PERL script for the TDClient job is simple yet complex.• The script will be executed by Applications Manager and use parameters

in calling job to build a transmit or receive subscript.• The PERL script will process the parameters and execute the subscript.• The subscript is a series of specific commands that tell TDClient what to

do with the incoming or outgoing file.• The commands that are required to transmit or receive a file are located

in the SAIG Host Communication Guide provided by the Department of Education.

• Script is stored in the EXEC directory on the server of installed instance of Applications Manager.

PERL script for the TDClient job.


• Example of full receive command:– tdclientc network=saigportal ftpuserid=TGxxxxx reset

“transfer=(name=yourname receive=./path/to/the/file/to/receive.txt receiveuserid=TGxxxxx receiveclass=xxxxxxxx)”

• The receive command explained:– Starts the FTP tdclientc process.– Network to log onto.– User ID (password is stored during the initialization)– Message Class to receive– Path

PERL script for the TDClient job – Cont.


• The TDClient Job will call the TDClient script.• The script is executed using the parameters within the job.• The TDClient job has the following parameters.

– Reset Option: Defaulted to Y– Transmission Command: T(transmit) or R(receive)– Path & File Name: The download path.– Message Class: Message class for filetype.– TG Destination Number: TG destination Number.

Job for the TDClient Script.


Job for the TDClient Script – Cont.


• A PERL script was written to issue one command with path & file name.• The PERL script will strip out the header and trailer records.• The Header and Trailer records are annotated with special characters

that are explained in the SAIGHostCommunicationGuide.pdf provided from the TDClient documentation.– Header Records are defined as O*N05– Trailer Records are defined as O*N95

PERL script to Remove Header and Trailer Records.


• All files downloaded through TDClient will have a header and trailer record.

• Banner will not process or understand these records.• Applications Manager will call the script using the parameters configured

in job• The script will remove the header and trailer records.• The Remove Header job will have the following parameters.

– Path & File Name: The file path.

Job to remove Header and Trailer Records.


Job to remove Header and Trailer Records – Cont.


• The Dataload process has three email notifications.• Three different job statuses will trigger email notifications.

– Aborted – Notification that a job has aborted.– Finished – Notification for a job that has finished.– Completed – Similar to finished however sends a different email.

Notification Development.



• Abort Notification uses the Generic Abort Message.



• Finish Notification uses custom text to notify for no files.



• Completed Notification uses custom text to notify for completion of job.


• Process Flow Level Prompts– Variables stored at the top level and referenced throughout each component.

• The Financial Aid Dataload process flow has the following components:– TDCLIENT_PROCESS– REMOVE_HEADER– COPY_FILE– RCPTPxx– RCPMTCH– RCRTPxx– WPAALTD

Financial Aid Dataload Process flow.


Financial Aid Dataload Process flow. - Prompts


Financial Aid Dataload Process flow. - Prompts

• The Financial Aid Dataload process flow uses substitution variable stored under the prompts tab.

• These prompts are assigned to following values:– #1 = Aid Year Code Default value: 1516– #2 = File Name Default value: igco16op.dat– #3 = Message Class Default Value: IGCO16OP

• The shortcut prompts are used throughout the process in each job to reference the value.

• Changing the prompts are easier to manage as opposed to changing all non-referenced values in each component.


Financial Aid Dataload Process flow. – TDClient Component.

• The first job in the Dataload Process calls the TDCLIENT_PROCESS.• The TDClient script is executed and use the follow parameters:

– Reset Option: Y– Transmission Command: R– Path & File Name:

/u02/TDC/TDAccess3.2/incoming/{#2}– #2 substitution variable is referenced and will substitute igco16op.dat

– Message Class: {#3}– #3 substitution variable is referenced and will substitute IGCO16OP

– TG Destination Number: TGXXXXX.• This script will download all IGCO16OP files to the incoming directory on

the server where the TDClient is installed.


Financial Aid Dataload Process flow. – TDClient Component.


Financial Aid Dataload Process flow. – TDClient Component.• The TDClient process has a condition that will send an email notification

to the financial aid staff if there was no files downloaded.


Financial Aid Dataload Process flow. – TDClient Component.• Questions regarding the downloading by Message class.

– What about multiple IGCO16OP files stored on the SAIG mailbox?• Example: IGCO16OP.dat

IGCO16OP.001• All files with the associated Message Class are downloaded and merged

into one file.


Financial Aid Dataload Process flow. – REMOVE_HEADER Component.

• The second job in the Dataload Process is to call the REMOVE_HEADER.

• The Remove Header script will be executed and supply the following parameters to the script:– Path & File Name:

/u02/TDC/TDAccess3.2/incoming/{#2}– #2 substitution variable is referenced and will substitute igco16op.dat.

• The process will strip the header and trailer records from the file.• Example of Header and Trailer record:

O*N05TGxxxxx ,CLS=MSGCLASS,XXX,BAT=xxxxxxxxxxxxxxxxxxxxxxxxxx, data in file data is file O*N95TGxxxxx ,CLS=MSGCLASS,XXX,BAT=xxxxxxxxxxxxxxxxxxxxxxxxxx,


Financial Aid Dataload Process flow.



• The REMOVE_HEADER has conditions on the component.• The condition will cancel the entire process if no file exist.


Financial Aid Dataload Process flow. – REMOVE_HEADER Component.• Questions regarding multiple files with multiple header and trailer

records.– What about multiple header and trailer records?

• Example:O*N05TGxxxxx ,CLS=MSGCLASS,XXX,BAT=xxxxxxxxxxxxxxxxxxxxxxxxxx, data in file data is file O*N95TGxxxxx ,CLS=MSGCLASS,XXX,BAT=xxxxxxxxxxxxxxxxxxxxxxxxxx, O*N05TGxxxxx ,CLS=MSGCLASS,XXX,BAT=xxxxxxxxxxxxxxxxxxxxxxxxxx, data in file data is file O*N95TGxxxxx ,CLS=MSGCLASS,XXX,BAT=xxxxxxxxxxxxxxxxxxxxxxxxxx,

• The script will remove all instances of header and trailer records.• The final file will resemble one complete larger file.


Financial Aid Dataload Process flow. – COPY_FILE Component.

• The COPY_FILE component will copy the file from one directory to another.

• The COPY_FILE has two parameters– File Source & File Name.– File Destination & File Name.

• This instance will copy the file incoming file from the /u02/TDC/TDAccess3.2/incoming directory to /u02/sct/banner/dataload/.

• The job will also rename the file to the required naming convention for Banner usage, example {#1}esar.tap.

• #1 substitution variable is used to reference the 1516 value.



• The COPY_FILE has a condition to make an archived copy of the incoming file.

• The condition will run a HOST command to copy the file to the archive directory.


Financial Aid Dataload Process flow. – RCPTPxx Component.

• The RCPTPxx is a job in dataload process that loads the incoming ISIR records into the Temporary tables.

• The RCPTPxx job can store the job parameters internally without the need to save a parameter set.

• RCPTPxx has two conditions assigned to the job.• The first condition runs a host command to remove the {#1}esar.tap file

from the dataload directory after the process is finished.• The second condition sends an email notification is the job aborts.


Financial Aid Dataload Process flow. – RCPMTCH Component.

• The RCPMTCH is the next job in the dataload process and attempts to match each temp record with an actual record in the Banner database.

• The RCPMTCH job can store the job parameter internally without the need to save a parameter set.

• RCPMTCH has two conditions assigned to the job.• The first condition sends an email notification if the job aborts.• The second condition sends an email notification for the job completion.• The email notification attaches all output in the email notification for

completion.


Financial Aid Dataload Process flow. – RCRTPxx Component.

• The RCRTPxx is the job that loads the matched records to the Banner database and loads unmatched records to a suspense table.

• The RCRTPxx job can store the job parameter internally without the need to save a parameter set.

• RCRTPxx has two conditions assigned to the job.• The first condition sends an email notification if the job aborts.• The second condition sends an email notification for the job completion.• The email notification attaches all output in the email notification for

completion.


Financial Aid Dataload Process flow. – WPPALTD Component.

• The WPPALTD is a custom built Banner process that loads the actual SSN into the alternate ID.

• The WPPALTD job can store the job parameter internally without the need to save a parameter set.

• WPPALTD has no parameters assigned.• WPPALTD has one condition assigned to the job.• The condition sends an email notification if the job aborts.• No other notifications are necessary because no output is generated.


Conclusion.

• Can the complete automation of dataload be done?• YES.• The process is duplicated for other ISIR data files.

– IGSA16OP– IGSG16OP– IDSA16OP

• The only time ISIR files are manually processed during the initial load during new aid year setup, the remainder of the year the dataload process is 100% automated.


Conclusion.

• Transitioning from year to year is simple.• Requires no human interaction, unless job aborts.• Return on Investment:

– Daily import of ISIR files covering all available aid years, instead of being done weekly.

– Freed up time for the Financial Aid staff to focus on other processes.– 416 yearly man hours were saved by automating this dataload process, which

amounts to around $10,000 a year saved from one process.– The process has been automating dataload going on 4 years.– I have created over 50 process flows are currently being automated through

Automic Applications Manager.


Thank you.

Automating Banner Financial Aid Dataload - San Jacinto College

Presentations & Public Speaking

Transcript of Automating Banner Financial Aid Dataload - San Jacinto College