Bill Pierce

SNIA, TeraCloud Corp.

September 14, 2005

Emerging Practices and Standards for Information Lifecycle Management

Property of SNIA 2

Outline

• What is ILM?

• SNIA’s ILM Modeling and Standards

• Implementing ILM at a Small Software Company

• Reactions?

Property of SNIA 3

SNIA Definition of ILM

The policies, processes, practices, services and tools used to align the business value of information with the most appropriate and cost-effective infrastructure from the time information is created through its final disposition. Information is aligned with business requirements through management policies and service levels associated with applications, metadata and data.

Property of SNIA 4

The Problem

• Information stewardship falls mostly on IT

• Data placed for convenience

• Too many configurations to support

• Nonstandard, unreliable results

• Fails to take advantage of economies of scale

Property of SNIA 5

Business Goals of ILM

• Reduce the cost of managing storage services

• Obtain an objective best fit between data requirements and storage services

• Ensure compliance with corporate governance and government regulation

Property of SNIA 6

SNIA’s role in ILM evolution• This is work in progress!• Technical Working Groups

– Architecture– Standards– Product interoperability

• Marketing Forums– Requirements analysis– Use cases– Best practices– Education and certification

Property of SNIA 7

Data Management Forum: Use Cases

• Primary Archetypes– Business Process

Analyst (business requirements)

– IT Architect (makes it happen)

– Records Information Manager (regulatory requirements)

Property of SNIA 8

ILM in Systems Administration

• Just as systems administration is the process of applying design and planning to meet computing application requirements at least cost, ILM is a similar process focused on information requirements.

• ILM provides a language for business, IT and records management to share the responsibility of information stewardship.

Property of SNIA 9

ILM TWG Modeling: (work in progress!)

• Data Classification

• Data Lifecycle Management

• Composite Storage Services

• Storage Service Level Management

• Data Placement Services

Property of SNIA 10

Mg

mt

Mg

mt

Mgmt

Mgmt

Data Service Resource Mgmt

Data Services

Composite Storage Set

Business Applications

DataPlacement

Service

Storage & Storage Services

Da

ta C

en

ter M

an

ag

em

en

t

Data Service Resource Mgr

Data requirements• Data classification bridge to

Information-based mgmt• Service Level Management• Lifecycle Management

Storage solutions• Configuration templates• Resource Management• Management of individual

services

Mg

mt

DataProtection

Service

Mg

mt

DataSecurityService

Property of SNIA 11

Information ClassificationCollaboration

InformationLifecycle

Key PerformanceIndicators (KPIs)

Data Lifecycle

Data Classification& Lifecycle

automation policies

Data Center

Business

Standard Configurations - Storage - Data Protection - Data Security

Data ServiceLevel Objectives

SNIA will define standards for: - Data SLOs - Offered Data Svc Levels - Data Lifecycle Policy Management - Composite storage configs - Mgmt of underlying storage services

Property of SNIA 12

Data Classification

DataSLOs

Data

Data ClassificationPolicies that group data so as to determine what Service Level Objectives must be applied.

Examples:• Files in /usr/foo/bar• Files in /user/fred older than 3 months• Files named *.mp3• Files bigger than 1GB• Files originated by a corporate officer • Files in the engineering group• Files with keywords in content• Files about oil exploration• …

Property of SNIA 13

-ClassificationName

Classified : ClassificationNode

-Metadata-//LogicalFile, Volume, or object

ManagedElement

-ClassificationSchemeName : string-Comments : string [ ]

Data : ClassificationScheme

1

*

MemberOfCollection

1 *

MemberOfCollectionClassification : PolicySet

PolicySetAppliesToElement

ClassificationService

ClassificationCapabilities

ServiceAffectsElementElementCapabilities

Data Classification Service Instance

1

*

ClassificationSchemeAppliesToElement

PolicySetAppliesToElement

Data Classification Model

Property of SNIA 14

Standard Metadata org.snia.fsmtwg.*Name Type Description/Values

Name   Full path name

FileSize Uint64 Size in bytes

CreationDate Datetime File creation time – nfs does not support this

LastModified Datetime File modification time

LastAccessed Datetime File last access time

EncryptionMethod String Describes the method used to encrypt the file

InUseCount Uint64 Used like a semaphore to determine when the file is in use

Remainder Of CIM_LogicalFile attributes

Property of SNIA 15

Standard attributes org.snia.ilmtwg.*Name Type Description/Values

AccessingApp String [] Data was opened by

CreatedByApp string The name of the application that created the data

BytesUsed Uint64 The number of bytes actually used to hold the data

FileURL String The URL of the file

ObjectID String A unique identifier for this data object (XUID**)

ObjectType Enum File, Directory, (stat data ++) XSET**

Version string The version of the document this data object represents

UserDefined String A free-form text comment

RetentionStartDate* Datetime The date on which the retention period begins

RetentionInterval* Int64 The amount of time after the RetentionStartDate to retain the data.

RetentionEventDate* Datetime The date on which an event occurred

RetentionEventInterval* int64 The number of seconds after RetentionEventDate for which to retain the data

ExpirationHold* Boolean [ ] All holds must be cleared in order to expire

RetentionEnabled* Boolean [ ] Declares whether retention attributes are enabled and/or infinite

Property of SNIA 16

Data Lifecycle Management

DataSLOPolicy

Data

Data Lifecycle Management:Policies that define Service Level Objectives for data over its lifetime.

DataSLOPolicy

DataSLOPolicy

Property of SNIA 17

Data Service Level Objectives (examples)

• Budget - MaxBudget• Accessibility

– ReadWriteRatio, AvgDataThroughput, SpaceUsage, SizeGrowth, Location

• Availability– PlannedDowntime, MaxUnplannedDowntime,

Aggregate, MaxUnplannedDowntimePerInstance

• Data Restore – RPO, RTO

• Security – Expiration, DestructionType

Property of SNIA 18

Composite Storage Service

Configuration template

CSS

Service

ServiceAffectsElement

+CreateCSSSettingDataTemplate()+DestroyCSSSettingData()+CreateCompositeStorageSet()+ReclassifyCompositeStorageSet()+ClassifyResource()

CompositeStorageConfigurationService

StorageVolume FileShare

-Workload : string [ ]-AvgThroughput : integer [ ]-AvgDataTransferRate : integer [ ]-MeasuredCapacity : integer [ ]-Protocol : string [ ]-InitialAccessTime : enum(immediate..hours) [ ]

CSSStorageSettings

-SecurityAccountability : enum(none..full)-SecurityIntegrity : enum(none..full)-SecurityAuthenticity : boolean-SecurityTrustworthiness : boolean-SecurityConfidentiality : enum(none..full)-SecurityImmutability : boolean-SecurityDestruction : boolean-SecurityPhysical : enum(none..full)

CSSSecuritySettings

-Workload : string-RecoveryCopyFrequency : enum(cont..weekly)-CopyRestoreTime : integer(seconds)-MeasuredCapacity : integer(bytes)-MaxCopyRetentionCapability : integer(seconds)-DataCopyLocale : string

CSSProtectionSettings

-CssName : string-StorageResourceType : enum(diskblock..files)-Owner : String

CompositeStorageSet

-Tariff : Integer-ResourceClassification : String

CSSSettingDataConcreteComponent

0..1

*

MemberOfCollection

0..1

*

MemberOfCollection

*

1

ElementSettingData

ServiceAffectsElement

Included in Composite Storage Service profile

Property of SNIA 19

Composite Storage Set Attributes (examples)

• Storage– Workload, AvgThroughput, InitialAccessTime

• Protection– RPO,DataRestoreTime,MaxDataRetentionCa

pability,DataCopyLocale

• Security– Accountability, Integrity, Physical, Destruction

Property of SNIA 20

CSS, ODSL, and SLOs: Service Level Mgmt

DR

ServiceClassification

DataSLOs

Properties

Data

Capabilities

CSS: Composite Storage SetAn abstract storage configuration templatedescribed by its overall technical capabilities such as storage performance, availability, data protection and data security.

ODSL: Offered Data Service LevelPart of the “service catalogue.” Each serviceclassification describes the service levelthat can be provided to data.

Data Service Level ObjectivesDerived from requirements defined inInformation Classification processes.Includes requirements specific to thedata and an application’s use of data. Map to best fit

Map per data center policies

Property of SNIA 21

CSS, ODSL, and SLOs: why 3 abstractions?

ServiceClassification

DataSLOs

Properties

Data

Capabilities

CSS: Composite Storage SetSupported configurations evolve with technology towards improving costs while meeting existing service requirements.

ODSL: Offered Data Service LevelService catalog may remain stable while supporting configurations evolve.

Data Service Level ObjectivesRequirements for data may or may not change independent of service catalogue.

Map to best fit

Map per data center policies

DR

Property of SNIA 22

1

* MemberOfCollection

Phase1 : Phase

Blue : Lifecycle

-ClassificationName

Data : ClassificationNode

LifecycleClassification

-ClassificationSchemeName : string-Comments : string [ ]

Data : ClassificationScheme

1 - ClassificationSchemeAppliesToElement

1

* 2 - MemberOfCollection

1 *

MemberOfCollectionClassification : PolicySet

PolicySetAppliesToElement

4 - PhaseAppliesToElement

3 - LifecycleAppliesToElement

LifecycleSelection : PolicySet PolicySetAppliesToElement

PhaseSelection : PolicySetPolicySetAppliesToElement

-ServiceLevelName : String

Gold : OfferedDataServiceLevel

-DataSLOAttributes-// to be expanded

DataServiceCapabilities

SettingsDefineCapabilities

CSS

ServiceLevelMapsToResource

PhaseMapsToODSL

PolicySetAppliesToElement

-SLOAttributes-//To be expanded

DataSLOGsElementSettingData

-Metadata-/From LogicalFile, Volume, FileShare, Dir, XSystem or XSET

DataObject : ManagedElement

SMIS 1.3 – Expanded Data Services:

Data Classification

Data LifecycleManagement

Data Service LevelManagement

Property of SNIA 23

Implementing ILM at a Small Software Company

1) Classify by Business Process Application

2) Define Lifecycles and SLOs

3) Define Lifecycles for backup copies

4) Classify storage or define ODSLs (<~10)

5) Match Phase SLOGs to ODSLs

6) Configure CSSs

7) Match ODSLs to CSSs

8) Implement Lifecycles

9) Operational Adjustments

Property of SNIA 24

Classify by Business Process Application

• Project documents• Software archive• Windows Updates• Financial spreadsheets• Customer contracts• Home directories• Web site• Released build archive• Accounting database• File transfer

• OS Software• OS Images• Wiki Site• Corporate Documents• Recent Builds• Source code files• Source Control Database• Exchange Email• Temp Files• Backups

Property of SNIA 25

Try to identify common superclasses• Working Documents

– Project documents, Recent builds, Web site

• Archives– Software archive, OS images, Released build archive

• Database Managed Data– Exchange Email, Source Control Database

• Sensitive Data– Customer contracts, Accounting database

• Temporary– Temp files

Property of SNIA 26

Define Lifecycles and SLOs

Working Docs: Tier2

AvgIORate: LowAccessPattern: SequentialSize: LargeSizeGrowth: MediumMaxDowntimeAggregate: 5 dayMaxDowntimeInstance: 1 dayRPO: 1 weekRTO: 2 daysSecurity: Normal

If accesed in <6 months

Working Docs Lifecycle

if not accessedin 6 months

Working Docs: Tier1

AvgIORate:MediumAccessPattern: SequentialSize: MediumSizeGrowth: MediumMaxDowntimeAggregate: 2 dayMaxDowntimeInstance: 1/4 dayRPO:1 dayRTO: 1 daySecurity: Normal

if not accessedin 2 years

if not accessedin 8 years

Offline

Initial Access Time: 1 dayLocation Affinity: Onsite;Offsite CopySecurity: Normal

Property of SNIA 27

Define Lifecycles and SLOsDatabase Managed Lifecycle

Database: Tier1

AvgIORate:HighAccessPattern: RandomSize: LargeSizeGrowth: LargeMaxDowntimeAggregate: 2 dayMaxDowntimeInstance: 1/4 dayRPO:1/4 dayRTO: 1/2 daySecurity: Elevated

lifecyclemanaged by

DBA

Database: Backup

backup

Property of SNIA 28

Define Lifecycles for Backup Copies

• Will not discuss (much)

• These copies have their own lifecycles and requirements in each phase

• Includes need for technology refresh– Physical (worn out media)– Logical (antiquated data formats)

Property of SNIA 29

Define ODSLsSoftware Company ODSLs

Active

AvgIORate:HighAvgThroughput: HighAccessPattern: RandomSize: LargeSizeGrowth: ModerateMaxDowntimeAggregate: 2 dayMaxDowntimeInstance: 1/4 dayRPO:1/4 dayRTO: 1/2 daySecurity: Elevated

Archive

AvgIORate:LowAccessPattern: SequentialSize: LargeSizeGrowth: LargeMaxDowntimeAggregate: 5 dayMaxDowntimeInstance: 1 dayRPO:2 dayRTO: 2 daySecurity: Normal

Sensitive

AvgIORate:MediumAccessPattern: RandomSize: SmallSizeGrowth: SmallMaxDowntimeAggregate: 2 dayMaxDowntimeInstance: 1/4 dayRPO:1/2dayRTO: 1/2 daySecurity: Secure DeleteDestructionType: Secure

Property of SNIA 30

Best Practices for ODSLs

• You are trying to simplify and reduce the number of ODSLs you provide.

• There is a strong tendency to over-optimize each storage solution. Resist it.

• In the name of simplicity, you will have to over-provide for some classes of data

Property of SNIA 31

Match Phase SLOGs to ODSLsClass Phase ODSL

Working Docs Tier 1 Active

Working Docs Tier 2 Archive

Archives Tier 1 Archive

Database Tier 1 Active

Sensitive Data Tier 1 Sensitive

Sensitive Data Tier 2 Sensitive

Temporary Tier1 Archive

Property of SNIA 32

Configure Composite Storage Sets

• This is where a lot of practical considerations come in.– Space and throughput limitations– Budget– Technology (old and new)– Physical and logical access– Vendors

• SRM tools can help

Property of SNIA 33

Match ODSLs to CSSs

• Probably a manual process

• With standardization, more automation

Property of SNIA 34

Implement Lifecycles

• Work in progress…

• Includes initial data placement

• Technologies available– Scripting and Symbolic Links– HSM and SAN Filesystems– DMAPI

Property of SNIA 35

Operational Adjustments

• Further economies may be achieved by splitting or combining classes and ODSLs

• New applications come online

• New technologies become available

• With ILM you have the infrastructure to deal with these changes in a coherent way

Property of SNIA 36

Summary

• ILM brings information storage in line with the requirements of the data

• Almost any IT organization can benefit from some aspects of ILM practice

• SNIA is laying the foundation of ILM through the development of Best Practices and Standards

Property of SNIA 37

For more information…

• www.snia-dmf.org

• www.snia.org

Property of SNIA 38

Questions? Reactions?

• Was this:– Useful?– Obvious?– Impractical?– Something you are already doing?– In need of standardization?

Property of SNIA 39

Abstract

Information Lifecycle Management (ILM) is an effort to formally align the business requirements of data with the storage services that support it. It is not a product, but a process that could benefit most organizations. Failure to do this can result in lost data, stolen data, unavailable data, overly expensive data and too much useless data. The Storage Networking Industry Association (SNIA) is working to define Best Practices and standards to support ILM. This talk will describe ongoing work in the SNIA and its application to the data and storage resources at a small software company.

Property of SNIA 40

BioDr. Bill Pierce started his career as an experimental physicist at Cornell University, the University of Washington and Northwest Research Associates, Inc. In 1995, he turned to systems administration and software engineering, supporting scientific projects during the early growth of the Internet. From 1998-2002 Bill worked on one of the first SAN Management applications at Vixel Corp. Since 2002 he has been actively involved in the Storage Networking Industry Association and the development of Information Lifecycle Management at TeraCloud Corp. Bill is the author of the Open Source storage utilities fcping and io_profile and numerous scientific and technical articles. He is a member of the Seattle Area System Administrators Guild and a founding member of the League of Professional System Administrators.