Framework for Performance Metric Development

draft-morton-perf-metrics-framework-01.txt

Alan Clark

alan@telchemy.com

IETF 70 PMOL WG

Framework draft

• Background and motivation behind this draft and PMOL

• Scope – “PM Entity” however Framework “could be useful” to other WG’s

• Metric Development– Guidelines, criteria for defining a metric

• Performance metric development process– Process for proposing, reviewing metrics

Metrics Development

• View metric from “audience” perspective• Definition of “metric”

– measure of an observable behavior of an application, protocol or other system

• Tests of “Usefulness”– Essential to understand performance and find

problems?

– Correlates with performance as it affects the “user”

– Helps with detecting, diagnosing, locating problems

Metrics Development

• Composed metric– E.g. IPPM work on spatial and temporal composition

• Metric specification– Process for specifying a metric (tbd)

• Classes of metrics– Having classes of metrics may simplify definition

process

• Qualifying metric (testing definition of metric)– Ambiguous?, Defined Units?, Measurement Error

Defined?, Repeatable?, Implementable?, Assumptions made about measured process?

Metrics Development

• Reporting models and impact on measurement– E.g. assumptions concerning sampling,

averaging, window sizes….

• Dependencies– Relationship between variance in “base”

metrics and variance in derived or composed metrics

Performance Metric Development Process

• New proposals– Prepared as I-D’s with metric definitions that conform to previous slides

– SHOULD’s• be vetted by WG with domain expertize

• Include assessment of relevance/ interaction with other standards groups (both other WG’s and other standards organizations)

• Specify intended audience for metric

• Proposal approval– Process TBD – IETF/ IESG/ AD’s/ Relevant WGs,/ PM Entity……

• Interaction with other WGs– Partnership with WG with domain expertize

• Standards Track performance metrics– Steer drafts through process

Comments, Questions, Next Steps?

• Potential additional definitions?– Index?– Intermediate?

• Other comments?