ATO Overview

1

ATO Overview

Adam KirkpatrickCreated: July 2011Last Update: July 11, [email protected]

2

ATO is a utility providing tape-out capabilities from an Avamar system. It

relies on a re-hydrated approach recovering data to a disk staging area.

The staging area is subsequently backed up to tape using a standard tape

Backup application.

The tape out process involves three steps;

1. client/backup selection2. confirmation view to see which backups were selected3. staging selected backup data and perform a tape backup

ATO Batch Manager fully automates these three phases.

ATOAvamar Tape Out

3

ATO – Data Flow

Client Grp-1

Drv-Z

Data-Flow

Data-Flow Client Grp-2

Client Grp-3

Tape Policy - 7 Year Retention



Drv-X

Drv-X

Drv-X

Drv-Z

Drv-Z

ATO Environment File Configuration CriteriaATO Client Configuration File Criteria

Drv-Y Drv-Y

Drv-Y

Data-Flow

Data-Flow

Data-Flow

Data-Flow

Data-Flow

Source or TargetSingle or Multi Node Systems Networker De-Dupe Nodes

4

ATO - Features

Single Point of Management & Control

• MENU Driven User Interface

• Multiple Concurrent Tape Out Sessions (1/Staging-Server)

• Failure Checkpoint Recovery

• Non-Inc & Incremental Tape Out

• Staging Recoveries Visible in Activity Monitor

• File Level Recovery from Tape to Original or Alternate Client

• Comprehensive Backup Selection Filters

• Structured Event and Batch Logs

• Audit Mechanism Tracks Critical Path TO Components

• Automated Generation of Required Tape Script

• Email Notifcations

5

ATO - Support Matrix

• Staging Platforms Windows, Red Hat, SCO, FreeBSD, Solaris, AIX, HPUX

• Avamar Plug-InLotus Notes, SQL, Exchange 2003, NDMP, VMimage, (Oracle)

• Tape ApplicationsNetworker, NetBackup, Backup Exec, Arcserv, HP Data Protector, TSM, Commvault

• Avamar Node TypesSingle & GRID, Source or Target, AVE, Networker De-Dup

6

User Interface

• Fully Interactive

• All Functionality Available

• Color Coded Displays

• Access to Logs

• Online Help

• Automatic Upgrade Process

ATO Rocks!

7

3-Step Operational Cycle

8

Selection Paramaters

Client Selection– By client group –gid group-id– By individual client within a group –gid group-id –client client-id– Client must be in a group

• Plug-In Types FS (default) SQL –sql Exchange –exchdb or -exchvss NDMP –ndmp Oracle –oracle Sharepoint –shpt Lotus Notes –lotus

Backup Reduction Filters– First –first last –last or –first_F or –last_F– Day of the week –week sun or day instance of the month –week sun_1– Avamar policy group name –gname policy-group-name– Retention tag type –rtype daily | weekly | monthly | yearly– Backup-ID number –buid “#’s– Backup Type –but mod | cod | nah ... “

9

Selection Date Range

• Default Search Range– 1st day of current month to your current date– Expand default range: -nday +# expand start date going back in time the specified #

of days

Relative Search Range– By Month: -rdate # where # is a relative month count going back in time from your

current month. I.E. –rdate 1 will go back one month in time– By Day: -rday # where # is a relative date of the month– By Day Range: -nday # where # is the number of days back in time from your current

date

• Fixed Date Range– Start date: -sdate yyyy-mm-dd – End date: -edate yyyy-mm-dd– Refer to Flds-2 & 3 in client configuration file, used for a more permanent override– Day Count: -nday # adjust # to number of days from your current date

10

Sample Select Process

–last filter used limiting selected backups to most recent only

clients scanned are those defined in group linux only

final selected backup count is 1 with 3 being excluded

used default date range

2011-07-01 to 2011-07-05

used ATO environment file 1

Note: By default the selection process acts on scheduled backups only– refer to –butype option to select non-scheduled backups

11

Sample View Process

Selected Backup View– backup metrics per client– total combined backup size– combined size allocated per staging FS or folder location– To View: ato -v or ato –view

12

Tape-Out Process

A Tape-Out Session Initiated with –tapeout Includes 2 Steps– Staging or redirected recovery of the selected backups to a staging location– Tape-Backup refers to initiating the tape backup automaticaly of the staged data

Step-1 - Staging– destination determined by Environment & Client configuration files– concurrent staging supported only with multiple staging servers available– perform stage only using –s option (no tape backup will be initiated)– Incremental using –inc provides significant performance gains– Inclusion and exclusion filters for folders or files

–data “C:/Program Files | My Documents” –xdata “C:/Program Files | My Documents”

Step-2 - Tape Backup– Perfromed by default or optionally using –t (tapeonly no stgaing will be performed)– appropriate tape script generated and initiated automatically

user defined scripts can be specified– tape backup policy referenced is assumed to exist within tape application– tape incremental backups can be leveraged only when incremental staging is used

13

Sample Tape-Out Process

14

ATO Environment File

15

Used to define physical infastructure of Tape Out environment– x1 Windows & x1 UNIX staging server and x1 tape backup server name per

environment file– Tape backup application – Path to binaries of Avamar & Tape backup agents– Event log, client configuration & temp file locations, pseudo client– Various control variables - key variables established automatically

Max of 20 environment files supported– each environment functions independently– ATO sessions can use any defined environment file– lock mechanism to prevent concurrent use of a given environment

Environment Files

16

Environment File Contents

Site specific parameters will require modification

– tape backup server parameters– Staging server paramaters– email notification address list

Operational parameters– adjust as required– refer to documentation for details

ATO Control Files– Caution: modification of control file

names not recommended– TMP_PATH, CFG & LOG files, Pseudo

Client name etc.

17

Establishing & Accessing Environment Files

Establishing Environment Files– ato [#] –env where # represents the desired environment number, default of 1 is assumed

creates a environment config file /usr/local/avamar/etc/atoenv.cfg[#] establishes temp file location /tmp/atocfg[#] establishes event log file /usr/local/avamar/var/atoevent.log[#] references client configuration file /usr/local/avamar/etc/atoclient.cfg creates the Avamar pseudo client /clients/tapeoutato[#] establish various control variables with default values file names remain consistent with the exception of an appended numeric value I.E. /usr/local/avamar/etc/atoenv,cfg2, /tmp/atocfg2, /usr/local/avamar/var/atoevent.log2 etc..

Accessing Environment Files – Display ato [#] -env - displays environment file contents– Update: ato [#] -env update - opens a vi session using appropriate file name– Verify: ato [#] –env parse - verifies file contents and report on any anomalies

– Note: environment # when required must be the first argument specified to ATO

18

ATO Client Configuration File

19

Client Configuration File

CSV file with 18 fields per client entry– client names and how they are grouped– destination staging server type UNIX or Windows– destination staging FS or folder, can vary between clients within a group– References the tape policy name to use– Determines whether to use an auto generated or user defined tape script– various backup filters applied on a per client basis

Enables Simplified Tape-Out Operations– client groups enable more precise selection and allows segmented work flow

better to define several smaller groups as opposed to one large one

– clients from different domains are permitted

20

Client Configuration File View

Establishing Client Configuration– CSV formatted file– base file established automatically– a single client configuration file is shared between all environment #’s– update interactively or manually– automatic Avamar domain and client discovery process ensures accuracy– Primarily a one time task at install time with infrequent updates afterwards

21

Client Configuration File Contents

Viewing Configuration File Contents– To Display: ato –cfg (all groups) or ato –cfg <grp-id> (specific group only)– Content Verification: ato [#] –cfg parse – perform checks of file contents– Color Codes: Blue=predefined sample lines Green=user comments Cyan=important field highlight

Red=field separatorConfiguration File Display

22

Client Configuration File Usage

Key ATO Process’s Rely on This File– All tape out clients must first be defined in this file

Client Groups– group names cannot contain spaces or special characters– a client can be defined once per group but may exist in any number of groups– all clients in a group must share a common tape policy name & staging server type

An Organized Group Naming Convention is Beneficial – I.E. A marketing dept has 100 clients where 25 clients is the desired workload per session. Group

names might be mkt1, mkt2, mkt3, mkt4 maintaining a logical connection between them while facilitating easy work load segmentation and automation from CRON or 3 rd party Scheduler using ATO’s Batch Manager described later

Several Options Available on Both Command Line & in Client File– If specified on command line will pertain to all clients in the group– If client requirments vary within a given group, define these in the client file– date ranges, incdel include/exclude folders, Avamar policy group name, retention tag and

destination path

23

Client Configuration File Update-1

Manual Update: ato –cfg update Update Method-1 (bulk – useful at install time only)

– Automated Update: ato –cfg add or ato –cfg add_v (initiate discovery & update process)– 1st blue highlighted line contains a generic potential line entry– DOMAINX & CLIENTX are keywords replaced automatically, do not modify these– green highlighted line shows a user modified line with fields Fld-1=training, Fld-10=Drive T being changed– first non configured client displayed was optionally rejected– second non configured client displayed and optionally accepted & added to configuration file– subsequent non configured clients are displayed and added or rejected as required– existing client entries requiring modification can be edited manually or using interactive update (Method-2)

►

►

►

►

►

Selection Phase

Method-1 Update Session

24

Configuration Manager Domain & Group View

Update Method-2 (interactive & preferred method)– Via Interactive Update: ato –cfg manager– Clients viewed by Avamar Domain or ATO Client Group– Drill down into a Avamar domain or ATO group provides options to

add, delete, disable, enable, modify & view individual client entries

Selection Phase

Method-1 Update Session

25

Configuration Manager Cont...

Interactive Method-2 cont....– Client status view realtive to ATO showing all available clients – Select client by number to add, disable, enable, modify, remove & view individual

client entries

26

Client Enable/Disable

Interactive Method-2 cont..– Sample process to disable/enable a configured client to ATO– Client status display, Blue=Not configured, Green=Enabled, Red=Disabled to ATO

enter “d” to disable a client

enter the client-# to disable

confirm the displayed line is okay to disable

client#2 disabled confirmed by its color change

27

Client Entry Update-2

Method-2 cont..– Sample process to modify an existing client entry– Process to add a new client (not-shown) is similar

enter “m” to modify a client entry

enter the client-# to modify

verification screen can be looped through as often as necessary modifying field contents as required. When ready to accept the change enter “c” to commit the change

enter the group name for client entry to be modified

28

Locate & Track Data on Tape System

All Staged Data Belongs to the Staging Server(s) Clients Used– tape backup effectively owned by staging server client

Data Staged to Structured Directory– <user-defined>/<ATO-defined>/Orig-Client-Name/<client-data>

Tape Solution Reports Relative to Staging Client Name Only

Combine Tape System Reports with ATO Audit Report– tape media involved– relative folder location where to find TO client in staging client backup– staging and tape backup dates– staging server name used– tape backup server used

29

Audit Log

• Used to Track Critical Path Component Name Changes Impacting Recovery Procedure From Tape

Critical Path Components Tracked– Tape Backup Server name– Tape Policy Name– Staging Server Name– Staging Server Destination Folder

Audit Process Is Automatic Requires No User Intervention

– Critical path component changes generates a environment audit record– Audit records readily available interactively basedd on client name from

Configuration Manager option T=Audit

– Used to determine where a given clients data resides on tape system

30

Audit Record Updates

Environment Change Detected in Critical Path– will occur relatively infrequently over time– displayed in audit summary or detail report

Non Critical Path records– audit record established every time a client is staged– displayed in audit detail report

Audit Log File– any filters or script can be run against audit log contents to format your own custom or

alternative report formats– available through ATO interface for display purposes

31

Sample Audit Summary Report

initiate audit report

initiate audit reportinitiate audit reportselect client for report

initiate audit report

select client group

selected report type defaults to summary or enter D = detail

• Summary Report Displays Critical Path Change Records Only

32

Sample Audit Detail Report

• Detail Report Displays Critical and Non Critical Audit Records

selected report d=detail

33

ATOTape Backup Script

Staging Server SpecsDestination Path

34

ATO - Tape Scripts

Automated Tape Script Generation– criteria taken from environment and client configuration files to generate a suitable script

autotapeout.bat or autotapeout.sh located in the environments temp directory– transferred to staging server <AvamarHomePath>/etc/scripts and executed as a pre script– supports incremental and non-incremental – non-incremental script will remove staged data after a confirmed successful tape backup– Supported tape applications

Arcserv – Brightstore, HPDP, Networker, Backupexec NetBackup, TSM & Commvault Networker support for client alias name, I.E. tape out data owned by original client name (disabled)

– any tape solution with a suitable CLI could be accommodated

User Defined Scripts– must contain necessary logic to enable ATO to capture its completion status– refer to ATO doc for example or use an auto generated script as a reference– must be kept on the Utility node, suggested location /atoadmin/userscripts

Tape Backup Status– tape status is logged on staging server in <Avamar-Home-Path>/etc/scripts/autotapeout.stat – The contents of this log are recovered back to ATO which parses and displays results– detailed failure information logged to event and batch logs– any non-zero tape RC is considered a failure

35

Staging Server & Destination Path Specs

• Staging Server Specs• can be physical or virtual• Must support required Avamar and Tape App file system agents• Decent CPU capacity with the equivelant of at leat 6-8GHz

• Structured Non-Incremental Destination Path:•<user-defined-path>/BYDATE/<client-name>/<date-time-buid#>/<backup-data>

• Sturctured Incremental Destination Path:•<user-defined-path>/INCREMENTAL/<client-name>/<backup-data>

1. Path definition shown in RED is not user modifiable2. Optional path suffix inserted prior to </backup-data> using the -path option or from Fld-16 in client configuration file

<user-defined-path>: taken from client configuration file

BYDATE or INCREMENTAL: inserted by ATO

<client-name>: inserted by ATO

<date-time>: inserted by ATO

<backup-data>: recovered backup data

36

Sample Networker Save Set Definition

Sample Networker client resource

– BYDATE– INCREMENTAL

Define multiple client resources as required

All staged clients to a given location will be included in the Tape Backup

User portion of staging path defined in client cfg. file

Each client resource may have its own media pool, group assignment and schedule etc.

37

Networker Recover View Of Tape-Out Data

Recovery View on Staging Server of Incremental TO

Recovery View on Staging Server of non-Incremental TO referred to as BYDATE

38

ATO Incremental Advantage

39

Incremental Tape-Out

Incremental Concept– incremental staging is relative to previously staged data, 1st stage is always a full– requires staged data remain on the staging server between ATO sessions

refer to –incdel option for a method to resynchronize staged data with backup client– beneficial for FS & NDMP not for SQL, Exchange, Oracle or Lotus Notes– total disk space required equivalent to size of one copy of source data involved– staging disk can be any disk type, Arrays, JBOD’s, USB stand alone etc.– disk I/O performance less critical when using incremental– makes it feasible to leverage an incremental tape-backup

Performance Gains– typical effective throughput gains versus non-incremental ~5 to 8 times– expect ~600-900 GB’s/Hr for staging dependent on % incremental change– If using incremental tape policy

expect equivalent tape backup throughput gains expect reductions in tape media usage of ~70-80%

– use of concurrent staging servers will improve overall aggregate throughput

How to Enable Incremental– Add to the -tapeout action, –inc option

40

Incremental AdvantageExample-1 Environment: single node

Initial Stage 4.7 GB’s 2.45 min’s

Initial Tape Backup 4.7GB’s 4.5 min’s

41


Post Initial Stage 4.7 GB’s 15 sec’s

Post Initial Tape Backup 4.7 GB’s 1 min

42

Incremental Advantage Example-2 Environment: single node

Initial Stage 10.5 GB’s 11.45 min’s

Initial Tape Backup 10.5 GB’s 8 min’sSession continued...

43



Post Initial Tape Backup 10.5 GB’s 1.5 min’s

44

Incremental Advantage

Example-1 Data SizeRun Time

Staging / TapeEffective Throughput

Staging / Tape

Initial Stage 4.7 GB’s 2.75 / 4.5 min’s ~110GB/Hr / ~65GB/hr

Post Initial Stage 4.7 GB’s 0.25 / 1.00 min’s ~1.1TB/Hr / ~288 GB/hr

Performance Increase n/a x11 / x4.5 x10 / x4.5



Staging / Tape

Initial Stage 10.5 GB’s 11.75 / 7.5 min’s ~55GB/Hr / ~84 GB/Hr

Post Initial Stage 10.5 GB’s 0.5 / 1.5 min’s ~1.2TB/Hr / ~420 GB/hr

Performance Increase n/a x23 / x5 x21 / x5

Single Node Incremental Comparison Results

• Factors Influencing Test Results•Networker Server is using Windows•Networker configured to use a DL3D VTL•no changed files were involved

45

Incremental Advantage Example-3 Environment: multi node(4)

Initial Stage 44.5 GB’s 12 min’s

Initial Tape Backup 44.5 GB’s 23 min’s

46

Incremental AdvantageExample-3 Environment: multi node(4)


Post Initial Tape Backup 44.5 GB’s 30 sec’s

47

ATO – Incremental Advantage



Staging / Tape

Initial Stage 44.5 GB’s 12.0 / 23.5 min’s ~223GB/Hr / ~114 GB/Hr

Post Initial Stage 44.5 GB’s 0.75 / 0.5 min’s ~3.6TB/Hr / ~5.2 TB/Hr

Performance Increase n/a x16 / x46 x16 / x46

Multi Node Incremental Comparison Results

•Factors Impacting Test Results•Networker Server & Staging server were the same server, an Avamar spare node•Networker configured using adv_file type device B2D•no changed files were involved•Staging configured to use all-nodes, provided ~10-15% increase in throughput

48

ATO Event Log

49

ATO - Event Log

All –select and –tapeout actions logged

Consolidates informational & Error Messages– Readily accessible to user for success / failure confirmation or problem diagnosis

To access Event Log: ato –l or ato -log– most recent event displayed initially– specify default browse direction by entering P=Previous or N=Next then press enter

to browse all events in the direction specified– to jump to a specific event#, enter desired event number & press enter– to search event log enter “s” then “h” for search editor syntax, similar to vi commands

Event messages are color coded– informational messages displayed in your default terminal color– errors messages displayed in red– info returned from tape application displayed in white

Each Environment# maintains its own Event Log

Email alerts contain corresponding Event Log details

50

ATO - Event Log Sample – Successful Stage & Tape Status

Avamar recovery log name located on SS and available within Activity Monitor

details returned from tape backup command issued by tape script

tape backup command syntax

issued by auto generated tape script

optional debug flags used to provide additional command trace information

staging process completed okay

51

ATO - Event Log Sample – Staging Failure

staging failure information, client, buid, error code

establish retry checkpoint containing only failed buid’s

if there’s no successfully staged items, tape backup back-up initiation will be skipped

Avamar log file name available in Activity Monitor or on the staging server

staging error count > 0

52

ATO - Event Log Sample - Tape Backup Failure

tape backup failed due to a mis-configured tape policy. The path specified to be backed up is not configured for tape backup.

staging completed okay

suggested rerun syntax to perform only the tape portion of this TO session no staging is required.

53

ATO Batch Manager

54

Batch Manager (BM)

Use to configure & monitor ATO batch sessions

Btach functionality includes– Create batch profiles to be initiated interactively, from command line or from CRON– Monitor status & progress of any number of active or completed ATO batch jobs– Review status of any given batch session as a summary or in detail– Maintains associated batch logs automatically– Use –D (upper case) debug flags to increase verbosity in batch logs

Color coded provides at a glance status confirmation of all batch sessions

– White=Active, Green=Completed Okay, Red=Failed

Views, dashboard and configuration– Dashboard provides log access and status view of all batch jobs– any number of batch profiles can be configured and managed

Access Batch Manager: ato –batch – Run a batch profile: ato –batch <profile-name>

55

Batch Manager Screen Shots

• Sample monitor mode screen showing summary view of job #7

• Sample configuration screen showing available profiles

view completion status of batch session#7

56

Batch Manager Execution Screen Shots

• Sample BM configuration screen showing batch profile #1 being initiated interactively

• followed by (2) monitor screen shots showing it as active (3) the final completion status

Batch job is now shown as being active in log monitor mode indicated by its date/time being displayed in White

Batch job completed okay indicated by date/time being displayed in Green

to review the entire log enter “r” to open the log file

initiation confirmation display1

2

3

57

ATO – Batch Manager Profile Screen Shots

• Sample Batch Profile• A profile contains the arguments required for the –select, -view and –tapeout process

• BM builds a suitable batch script at execution time utilizing a profiles contents

• the batch script maintains a log of the same name with .log extension in /usr/local/avamar/var

• Batch profiles reside in /usr/local/avamar/etc

• Batch profiles and log names always begin with “atobatch”

• add debug flags –D flags to capture additional detail to batch log

• add debug flags –d flags to capture additional detail to Event Log

58

Batch Manager CRON Screen Shots

Batch Manager CRON interface– Color coding of profile dates identifies if profile is configured, not configured or disabled in the CRON– Allows selection of CRON run times, day-of-week, month etc.– Supports any valid basic or complex CRON syntax for each field

59

ATO Recovery Manager

60

ATO – Recovery Manager RM

Recovery Manager (RM) “Checkpoints” Failed ATO Sessions

Two Failure Scenarios – Retry = individual staging failures during an ATO session– Rerun = staging items never initiated the result of a premature end of an ATO session

Failed Sessions Automatically Tracked Containing Only Failed or Outstanding BU-ID’s

– recovery sessions relative to the environment# used

Variable in Environment File Determines Quantity of Recovery Sessions to Maintain

– default is 10– round robin basis with oldest being rolled off the list and deleted

Failed Sessions Viewable Using RM Interface ato -recover– determine contents and size– similar to normal -view display

Failed Sessions can be Re-Executed or Deleted as Required

Subsequent Failures During a Re-Execute Result in a New Checkpoint Session Providing an Incremental Reduction During the Retry/Rerun Process

61

ATO – Recovery Manager - Retry View

RM View Similar to Standard View with Title Highlighted in Red

Top Half of Display shows Contents of Selected Retry Session

Bottom Half Lists the Available Retry Sessions– Lists color coded for quick at a glance confirmation. – Blue=Never Re-Executed, Green= Successfully Re-Executed, Red=Failed Re-Execution

normal view of the contents of retry session-#1

session-#1 has been retried identified by green date which is the date/time of the failure

syntax required to retry this session

this is a RETRY view

enter a session-# to execute,view or delete

62

ATO – Recovery Manager - Rerun View

this is a RERUN viewsession-#4 has never been rerun identified by blue date which is the date/time of the failure

syntax required to rerun this session

normal view of the contents of retry session-#4

Rerun Mode entered by Using M=Mode or ato –recover rerun

Top Half of Display shows Contents of Selected Rerun Session

Bottom Half Lists the Available Rerun Sessions– Lists color coded for quick at a glance confirmation. – Blue=Never Re-Executed, Green= Successfully Re-Executed, Red=Failed Re-Execution

enter a session-# to execute,view or delete

63

ATO Troubleshooting

64

ATO - Troubleshooting

Troubleshooting Staging Recovery Failures– troubleshoot a failed staging activity as you would any recovery starting with the Activity Monitor

– staging progress can be monitored using the Activity Monitor or ATO Batch Manager

– tape backup is initiated first as a recovery to transfer the tape script to the staging server then executed as a post script. This session will remain active for the duration of the tape backup phase with little data movement displayed in Avamar

– staging server logs in <HOMEPATH>/avamar/var will be backed up to Avamar automatically when a staging recovery fails making them available in the event the Activity Monitor is cleared

– log-id# of each stage activity is listed with the batch log staging information message and present in ATO’s event log when a single –d flag is used

ATO Batch and Event Logs– first place to look to determine overall status– useful for identifying most problem situations especially tape related– remember, event logs are maintained on a per environment basis and batch logs relative to the

job regarless of environment number being used–

65


Debug flags increase verbosity of messages displayed & logged to STDOUT & Event Log

– impact of debug flags are cumulative, the more you specify the more you get

– debug flags are primarily valid on –select & -tapeout sessions only

– Upper case –D logs to stdout only, useful for batch runs as they’re acptured in batch logs

– -d provides additional ATO specific informational messages including capturing the staging information message for each client and successful tape application results. Tape info always logged when tape errors detected

– -d –d echo avtar and mccli cmd syntax

– -d –d –d removes quiet mode from avtar and mccli cmds

– -d –d –d –d retains ATO session temp files to be used for analysis including the auto generated tape scripts

– -d –d –d –d –d –d full trace of the ATO session

66


Be careful with tape policy configuration– ensure required staging folders are configured matching those defined within the

client group– with a Networker server initiated backups use savgrp command, ATO will not be able

to detect a mis-configured save-set as savgrp acts on whatever is defined to it– with staging server (client initiated) backups incorrect save-set definitions are

detectable and these details logged to the event and batch logs

Auto tape scripts– located on staging server in <Avamar home path>/etc/scripts, – run manually for troubleshooting purposes autotapeout.bat or .sh as applicable– Auto Tape script run status captured in autotapeout.stat – custom scripts must contain logic to update & capture the stat file to enable ATO to

accurately report the tape backup status– required logic for custom scripts documented in ATO documentation, alternatively

review an auotgenerated script for example logic

Ensure correct Environment# is being used– make sure your looking at correct event log# – make sure the ATO action calls are using the intended environment#

67

Troubleshooting

Capture ATO trace and Grab’s for debug purposes– reduce selection criteria to the minimum possible in order to reproduce the problem– run the failing session as follows:

ato <your-ato-args> –d –d –d –d –d –d > /tmp/atotrace.txt 2>&1– run ATO grab diagnostic

ato -grab– send grab file to ATO support: [email protected]– A grab file is the single most useful debug tool, ALWAYS request or provide one

Staging Failures Often Related with Incorrect Values in Environment or Client Configuration Files

– always run a parse check against each file, ato –env parse or ato –cfg parse– is there sufficient space on the staging server mount point(s) being used?– is the specifed mount point in the client configuration file correct?– what does the ATO Event or Btach logs show?– can you stage the failing BUID manually using the GUI?– Is Avamar GC or Checkpoint active, if so do not run ATO?– run ATO health check, ato –health– refer to ATO menu Opt-12 ATO Administration

mailto:[email protected]

ATO Overview

Documents

Transcript of ATO Overview