D.17 GUIDANCE FOR HUMAN FACTORS CASE WITHIN E-OCVM ... · Cooperative Approach to Air Traffic...

Project no. FP6-2005-TREN-4-Aero-036826

CAATS II

COOPERATIVE APPROACH TO AIR TRAFFIC SERVICES II

Instrument: CA - Coordination Action Thematic Priority: AERO-2005-1.3.1.4h

D.17 GUIDANCE FOR HUMAN FACTORS CASE WITHIN E-OCVM PROGRAMMES AND PROJECTS

Due date of deliverable: 31/05/2009 Actual submissi on date: 28/05/2009

Start date of project: 06/11/2006 Duration: 36 mont hs

Organisation name of lead for this deliverable: EEC

Revision: Draft

Project co -funded by the European Commission within the Sixth Framework Programme (2002 -2006)

Dissemination Level

PU Public X

PP Restricted to other programme participants (including the Commission Services)

RE Restricted to a group specified by the consortium (including the Commission Services)

CO Confidential, only for members of the consortium (including the Commission Services)

Cooperative Approach to

Air Traffic Services II

Date:

Document ID:

Revision:

28/05/2009

CII-WP1.3-EEC-D17-V1.4-DE-PU

Draft

D.17 Guidance for Human Factors Case within E-OCVM programmes and projects - 1 -

This project has been carried out under a contract awarded by the European Commission.

© 2009– All rights reserved

Document change log

Revision Edition date Author Modified sections / pages Comments

1.4 28/05/2009 J Harrison All Firs public version



Date:

Document ID:

Revision:

28/05/2009

CII-WP1.3-EEC-D17-V1.4-DE-CO

Draft




Table of Contents

Executive Summary 5

1. Introduction 1

1.1 Who should use this Guidance 2

1.2 Human Factors work within ATM R&D projects 3

1.3 Human Factors and validation 4

1.4 Document structure 6

1.5 Complementary sources 7

2. Human Factors work in projects 9

2.1 Improving and evaluating 9

2.2 Human Factors in project initiation 10

2.3 Human Factors integration processes 12

2.4 Human Factors and validation planning (SPF) 14

2.5 Human Factors in the lifecycle 16

2.6 Human Factors technical work 18

3. Building the evidence 19

3.1 Working towards Human Factors claim 1 (system operability) 19

3.2 Working towards Human Factors claim 2 (system performance) 21

4. Concept maturity criteria 25

4.1 Concept definition (people-related aspects) criteria 27

4.2 People-related issue criteria 28

4.3 Human Factors process criteria 29

5. Evidence to produce 31

6. Integrating Human Factors 33

6.1 Human Factors relationships with other disciplines 33

6.1.1. Human Factors and Safety Case 34

6.1.2. Human Factors and Business Case 35



Date:

Document ID:

Revision:

28/05/2009


Draft




6.1.3. Human Factors and Environment Case 35

6.1.4. Human Factors and Security Case 35

6.1.5. Human Factors and Technology Case 35

6.1.6. Human Factors and Other Cases 36

6.1.7. Human Factors and System Engineering 36

6.2 Standardisation and regulation 37

Appendix A ICAO Key Performance Areas 39 Appendix B Screening for people-related issues 43 Appendix C Project initiation questionnaire 46 Appendix D Applying the EUROCONTROL Human Factors Case process (v2) 49 Appendix E Human Factors in different phases 50

Appendix E.1 Human Factors objectives 51

Appendix E.2 Human Factors techniques and tools 53

Appendix F Human reference models 55 Appendix G Human Factors mapped to SPF 56 Appendix H Detailed guidance for developing Human Factors claims 1 & 2 60 Appendix I Operability checklist 67

Appendix I.1 Example operability metrics 69

Appendix J Maturity criteria 71 Appendix J.1 Concept definition criteria (people-related aspects) 73

Appendix J.2 People-related issue criteria 75

Appendix J.3 Human Factors process criteria 78

Appendix K Relationships between disciplines 80 Appendix K.1 Human Factors and Safety Case 82

Appendix K.2 Human Factors and Business Case 85

Appendix K.3 Human Factors and Environment Case 88

Appendix K.4 Human Factors and Security Case 90

Appendix K.5 Human Factors and System Engineering 92

Appendix L Outline technical guidance 94 Appendix L.1 Human machine interaction 94

Appendix L.2 Recovery from failure 95

Appendix L.3 Teams and Communications 96



Date:

Document ID:

Revision:

28/05/2009


Draft




Appendix L.4 Procedures, roles and responsibilities 99

Appendix L.5 Skills, Training and Development 101

Appendix L.6 Staffing and Organisation 103

Appendix M Generic people-related requirements 105 Appendix N References 108 Appendix O Glossary 110

7. Rationale – (not part of guidance) 112

7.1 The CAATS model 112

7.2 The Human Factors ‘problem’ 112

7.3 Key features 112

7.4 The EUROCONTROL HQ Human Factors Case 113



Date:

Document ID:

Revision:

28/05/2009


Draft




Executive Summary Why Human Factors

“... The provision of ATM services today depends heavily upon the performance of humans at all levels within ... the ATM System from which the services are delivered.” Even in a radically more automated ATM system “Humans … will be central in the future European ATM system while their role is evolving to managing and decision-making.” Systematic attention to people-related issues throughout the lifecycle will help ensure that when implemented, the concept delivers its intended benefits. Human Factors validation work does this by seeking to support the twin claims:

1. That the role of the human in the system will be consistent with human capabilities and characteristics, and …

2. That the contribution of the human within the system will support the expected system performance and behaviour.

This guidance

This Human Factors guidance covers how to improve a concept (design) and how to demonstrate that will be good enough (validation) from a people-related perspective. It is written mainly for people who determine ‘what’ Human Factors work is done in a project, not just for those who do the work. (Section 1.1)

The guidance can be used at both programme and project level, since it supports both the strategic (programme) parts of E-OCVM (steps 0, 1, 5 of the Structured Planning Framework) and the project parts (steps 2, 3, 4). The guidance covers the critical phase of setting up a project (i.e. determining its broad approach, resources and objective) as well as the conduct of Human Factors work within the project. (Section 1.1 and Table 3)

The process in the EUROCONTROL guidance for Human Factors Case provides the underlying base process and a topic structure (the Human Factors Pie) for Human Factors work. This is blended with, interpreted, and mapped onto, specific guidance related to validation, the concept lifecycle, the case-based framework, transition criteria, and the generation of structured evidence to support stakeholder decisions. (Section 2.3)

The guidance makes extensive use of tables for easy, selective reference.

Human Factors work

The guide provides an overview of Human Factors work in a project, together with Human Factors interpretations of the steps and sub-steps of the E-OCVM Structured Planning Framework (SPF), and an overview of the constraints and focus of Human Factors work at different phases of the Concept Lifecycle. (Section 2)

It describes work needed to build evidence to support the two Human Factors claims above (operability and contribution to performance) explained in terms of both the lifecycle and the SPF. (Section 3)



Date:

Document ID:

Revision:

28/05/2009


Draft




Human Factors maturity criteria

The guide presents a structured set of Human Factors maturity criteria that may be used to support decisions about transition between phases of the lifecycle. They are based on the (people-related aspects of) the concept definition, people-related issues (risks and opportunities), the quality of information, and the processes used. These are supplemented with a discussion of how to set transition criteria, use of quantitative and qualitative criteria, etc. (Section 4)

The evidence

The guide explains the structure of the body of evidence that should be produced as the Human Factors case: a reference model (defining the people aspects of the concept in a way that can be read across to other cases), a traceable list of people-related issues (with their impact), a summary of actions taken (mapped to the address addressed), a list of issues outstanding, and how (in the future) they will be addressed, an overall high level summary of what is and what is not yet acceptable from the Human Factors perspective. (Section 5)

Integrating Human Factors

The guide includes a section on how to integrate the Human Factors case, and the work leading to it, with the other project disciplines. For each discipline, there is an explanation of the relationship, supported by details of how Human Factors work can relate to the core activities of the other discipline. (Section 6)

More detailed guidance and reference material

Appendices A – M contain supporting detailed lists, tables, etc, referenced from the main body of the document. Appendix N lists external references, and Appendix O contains a glossary.

(Rationale)

This version contains a brief statement, after the appendices, of the rationale behind this guidance, but this is not intended as part of the final product.



Date:

Document ID:

Revision:

28/05/2009


Draft




1. Introduction “... The provision of ATM services today depends heavily upon the performance of humans at all levels within ... the ATM System from which the services are delivered.” 1 However much ATM evolves to exploit more automated function, “Humans … will be central in the future European ATM system while their role is evolving to managing and decision-making.”2 The truth of these statements cannot be denied, yet projects in the past have often failed to give adequate systematic attention to the complex and critical way in which human and non-human parts of the system must work together to deliver safe, cost-effective performance.

If you are setting up or running an ATM R&D project, whether at early or late phase of concept development, this guidance will help you to determine the Human Factors activity needed to identify, assess, and manage the people-related issues that are implicit in the proposed new concept or system.

Human Factors work within a project should have two complementary objectives related respectively to concept development and to concept validation:

• To influence the concept/system design in ways that improve it … and …

• To generate evidence about the extent to which the resultant system can be expected to meet its objectives, so that stakeholders can make well informed decisions about it.

Figure 1 shows the twin outcomes to expect from Human Factors work in a project, and the two flows of work leading to them. In practice, there is a large overlap between the two flows, with a lot of technical work contributing to both. The balance between the two types of activity (improving versus evaluating) will vary through the project, with the early focus of Human Factors work more on improving the concept, and the later stages more on generating evidence about the adequacy of the (people-related) aspects. Projects at different phases of the concept lifecycle may also have a different balance.

Improve the concept

Evaluate whether the concept isgood enough yet

HF Body ofEvidence

More matureconcept/system

Initialconcept(s)

Human Factors processes

Figure 1: Twin outcomes of Human Factors work

1 SESAR Definition Phase D1 – Air Transport Framework, The Current Situation, July 2006 2 SESAR Definition Phase D5 – SESAR Master Plan, April 2008.



Date:

Document ID:

Revision:

28/05/2009


Draft




How much Human Factors work you need to achieve these objectives, and what type of work, will depend on the specific nature of the concept and project, but in all cases the work should be properly integrated with the contributions of other disciplines (systems engineering, safety, etc) and by the end should have produced a well formed body of evidence to show how well the people-related issues are known and understood, and the extent to which they have been properly addressed (since any not addressed could represent latent risk to emerge later). The scale of work should be matched to the uncertainties inherent in the project, and the degree of confidence that it is reasonable to expect in its results, given its phase in the lifecycle. It is far easier to under estimate than to over estimate the Human Factors work needed to complete a project properly.

The guidance will help you to plan Human Factors work for a project, and to decide what people-related evidence it needs to produce and how to produce it. The guidance forms a part of, and is consistent with, the overall ‘case based’ E-OCVM approach to concept validation. If you are not already familiar with the E-OCVM case-based approach, then see [1] or [2].

1.1. Who should use this Guidance

This guidance is intended primarily for people who will determine ‘what’ Human Factors work is done in projects (programme and project planners and managers) though of course it can be used by practitioners who do the work. Technical guidance is available from other sources (see 6.2.Appendix N). Practitioners can only become active when they have been allocated to the project, and given appropriate resources and goals. They may be unavailable at the planning stage, and for this reason project planners needing guidance for Human Factors planning requirements will find this guide useful.

E-OCVM covers both programme and project level activities (see Table 3 for more detail) and this guidance also covers both levels:

At programme level

If you are responsible for a programme (ie the long term development of one or more concepts through a co-ordinated sequence of R&D projects from inception through to implementation), whether as a manager, an operational champion or a concept architect, then this guidance will help you:

• To take proper account of the human as well as technical aspects when assessing concept maturity, and deciding what sort of R&D projects to invest in

• To set project goals, resources and overall plans that are consistent with the people related issues (risks and opportunities) inherent in the concept(s).

At project level

If you are working in an R&D project as a (non Human Factors) project manager, system engineer or operational champion, or as a senior Human Factors practitioner working with such people, then this guidance will help you:



Date:

Document ID:

Revision:

28/05/2009


Draft




• To develop detailed Human Factors plans for the work to respond to the people-related issues already identified

• To refine, review or revise the scope and direction of the Human Factors work where necessary in response to emergent people-related issues

• To ensure that the project delivers the required evidence to validate human aspects of the concept(s) being developed.

If you are a practitioner, responsible for Human Factors work within a project, the guidance will help you to understand the overall context of your work, and the supporting material should help you to produce more effective and timely results, supplementing (not replacing) your professional knowledge of Human Factors methods and tools.

1.2. Human Factors work within ATM R&D projects

Human Factors work plays an important part in an ATM project, because it focuses specifically on the human component of the system. People are not the same as engineered system components. They have different attributes, they behave in different ways and their performance is subject to different influences and constraints. The Human Factors discipline provides the tools needed to analyse, model, evaluate and design for people within the system, as well as the perspective to look at the system from a human-centred, and not just a technology-centred, point of view.

The Human Factors work should be an integral part of the project, complementing and supporting the work of other disciplines, in the same way that people form an integral part of the ATM system, interacting with many other parts of the system, and contributing to overall performance.

The scope of Human Factors is very broad (because human behaviour and characteristics are complex) and all industries have developed ways to subdivide it for easier management. The defence industry was the first with: Manpower, Personnel, Training, Human Factors engineering, Health hazard and System safety. Early ATM work [6] adapted this with ‘Task interface design’ in place of Human Factors engineering. Other work added an extra ‘Organisational’ domain. More recently the ‘Human Factors pie’ shown in Figure 2 has been developed specifically for ATM, see [5]. Its six segments correspond to the three main groups of interfaces and the three types of structure that must be considered in a system including people.

SESAR work has adopted a slightly different work breakdown with four segments of the Human Factors pie (Working environment, Procedures roles & responsibilities, Teams & communication, Human in system) grouped together under a (diminished) ‘Human Factors’ heading, while the other two segments (Skills, training & development and Organisation & staffing) are covered (but not via a one to one mapping) under the two headings of ‘Recruitment, Training, Competence and staffing of operational staff’ and ‘Social factors and Change management’.



Date:

Document ID:

Revision:

28/05/2009


Draft




Between Human & Environment Workingenvironment

Human insystem

Organisation& staffing

Skills,Training &

Development

Procedures,Roles &Responsib-ilities

Teams &Communication

Interfaces Structures

The provision of people

The capabilities of people

The work of people

Between Human & (Rest of) System

Between Human & Human

Figure 2: The ‘Human Factors pie’

However your project chooses to categorise the Human Factors work, and however you share it between teams, the essentials are to ensure that:

• You cover the whole Human Factors scope, not just some parts.

• The different aspects are properly co-ordinated.

• The work influences the main project outcomes.

Integrating Human Factors work with other project activities is important for project success. Human Factors must not be ‘an island’. Of course much of the work will be done within the Human Factors team, but it should all be done in a way that makes timely contributions to other activities. It should also be aligned with project goals, notably with the relevant KPAs (Key Performance Areas, see 6.2.Appendix A) that determine project priorities.

The main CAATS guidance on E-OCVM [3] gives general advice about inter-working between the teams responsible for different Cases within E-OCVM, and section 6.1 of this document contains guidance on specific relationships between Human Factors and other disciplines.

1.3. Human Factors and validation

E-OCVM [1] is a standardised approach to the development of European ATM concepts. It provides a means for initiating many projects within a programme, and then integrating their collective learning. (For an overview see [2].) It has three key elements:

a. The Concept Lifecycle Model – Successive phases (V0, V1, V3, etc) provide a scale against which assess the maturity of an operational concept, and to match the type and scale of investment used to develop and validate it further. The aim is to invest in lower cost activities to remove initial uncertainties, and thus reduce the risk of progressively higher cost activities needed for final validation. Each phase is thus characterised by expectations about what will already be known and about what type of activities will be undertaken.



Date:

Document ID:

Revision:

28/05/2009


Draft




b. The Structured Planning Framework (SPF) – A standardised set of steps and sub-steps can be applied at any phase of the lifecycle to plan and implement operational concept validation. The SPF covers activities at both programme and project level, since E-OCVM is based on the premise that projects will be initiated and conducted as part of larger coherent programmes (such as SESAR [8]).

c. The Case Based Approach – This formalises the role of contributory disciplines (Human Factors, Safety, etc). They each work within the overall framework to produce structured bodies of evidence that can be combined in a coherent way to support stakeholder decisions about the development or abandonment of a concept.

This guidance explains how to develop the Human Factors Case. It maps Human Factors activity and goals onto the R&D phases of the Concept Lifecycle Model, and onto the Structured Planning Framework.

E-OCVM aims to establish progressively the ‘fitness for purpose’ of a new operational concept (and the system that will implement it). The human dimension of fitness for purpose can be summed up in terms of the system’s effect on the people within it, and their effect on the system as a whole. These are represented by two broad Human Factors claims that validation seeks to substantiate:

1. That the role of the human in the system will be consistent with human capabilities and characteristics, and

2. That the contribution of the human within the system will support the expected system performance and behaviour.

If either of these claims is untrue when the system is operational, then it cannot be relied on to deliver its expected benefits. Human Factors activities within the project should be geared towards achieving these claims, and also towards providing the evidence to build confidence that they will be true when the system becomes operational.

In the early phases of the concept lifecycle, the claims cannot fully be met, and complete evidence about them cannot be provided. The evolutionary approach to validation embodied in E-OCVM is based on establishing reasonable expectations about how far to go in the current phase of the lifecycle, and then planning to meet those reasonable expectations. It strikes a balance between the cost of obtaining the evidence and the risk of remaining in ignorance about critical aspects of the concept/system.

Human Factors should support and be supported by the wider validation process.

• Some system-wide validation issues require a specialist Human Factors analysis before they can be properly understood.

• Some system-wide validation processes need a Human Factors input to be fully effective (e.g. design, conduct and interpretation of human-in-the-loop experiments).

• Some issues arising from Human Factors work may influence the focus of the overall validation process (e.g. issues relating to user perceptions and acceptance).



Date:

Document ID:

Revision:

28/05/2009


Draft




• Some Human Factors results may lead to a major change in concept design, with knock-on consequences for how it is validated.

The maturity of a concept, and the corresponding phase of the E-OCVM lifecycle, provides the overall context to determine the scope and depth of Human Factors activities and for the degree of completeness, precision and certainty of the validation evidence that they should seek to generate.

1.4. Document structure

The guidance in this document is structured in two parts:

• The main body (including this introduction) gives a comprehensive overview of Human Factors work in an E-OCVM ATM R&D project.

• The appendices provide large amounts of supporting information, with tables and checklists.

The main body is in six chapters:

• 1 – Introduction – (this section) explains how Human Factors fits into ATM R&D projects, and its role in E-OCVM.

• 2 – Human Factors work in projects – explains how to scope Human Factors work when setting up a project, and describes the core Human Factors processes within a project. It also explains how Human Factors fits into the Structured Planning Framework (SPF) and the Concept Lifecycle.

• 3 – Building the evidence – explains how to use the E-OCVM framework to develop evidence for the two Human Factors Claims (about system operability, and the human contribution to system performance) thus building confidence, and/or providing the stimulus for further improvement of the concept.

• 4 – Concept maturity criteria – explains how to assess the maturity of the people-related aspects of a concept, and hence where to locate it within the E-OCVM concept lifecycle model.

• 5 – Evidence to produce – provides an outline structure for the body of Human Factors evidence to be produced as a contribution to concept validation

• 6 – Integrating Human Factors – explains how Human Factors work can be properly integrated within a project, based on shared interests and mutual contributions with other disciplines.

Table 1 shows the anticipated use of each chapter by programme and project personnel.

The different parts of the guidance reflect different perspectives of good Human Factors practice. They do not follow sequentially from each other, but represent different perspectives, all of which are necessary. Figure 3 shows how the different parts fit together in the context of an R&D project. Section numbers are shown in brackets. The process begins with project setup (which is effectively a programme activity) on the left.



Date:

Document ID:

Revision:

28/05/2009


Draft




Underlying the whole are core Human Factors processes (centre). They need to operate within the context of wider validation planning processes (top) and they need to operate in synergy with other disciplines working on other cases (bottom). The work on Claims 1 and 2 (centre right) are specifically focused on generating the Human Factors Body of Evidence (right) to support validation. A template (bottom right) defines the structure and contents of the Body of Evidence. The goals driving the whole process are defined in terms of concept maturity criteria (top right).

Core Human Factorsprocesses (2)

Integrating with other disciplines / cases (6)

Project setup/ scope (2.2)

Validation planning (SPF) (2.4) Targets(Maturi ty

criteria) (4)

Body ofEvidence

Work on Claim 1:Operabili ty (3.1)

Work on Claim 2:Performance (3.2)

Template (5)

Figure 3: Fitting the parts together

Table 1: Use of this guidance at programme and project level

Chapter Programme use Project use 1 General introduction

2 Setting up projects with appropriate Human Factors objectives and resources to address people-related issues. Understanding how Human Factors works

3 Strategic planning of Human Factors within a programme

Detailed planning and implementation of Human Factors validation within a project

4 Locating concepts within the lifecycle Determining the scope of Human Factors in new projects Setting project Human Factors objectives Assessing project results

Understanding project objectives within the wider context

5 Reference information Structuring validation evidence and presenting the final Human Factors Case

6 Reviewing project proposals Setting up project teams and responsibilities Managing Human Factors work within a project



Date:

Document ID:

Revision:

28/05/2009


Draft




1.5. Complementary sources

This document complements other sources, notably:

• E-OCVM [1], the process within which the Human Factors case is conceived, which has an overview at [2].

• CAATS2 Guidance [3], describing the complementary cases within E-OCVM, including a high level view of how Human Factors fits in

• Guidance from EUROCONTROL [5], which provides the underlying Human Factors process on which this guidance builds. This CAATS guidance is written to complement version 2 of the EUROCONTROL document, but later versions may evolve to absorb some of the CAATS material.

• Other technical sources, egg HIFA [6], which provide background information about Human Factors within projects.



Date:

Document ID:

Revision:

28/05/2009


Draft




2. Human Factors work in projects This section explains Human Factors from six complementary perspectives, each of which is necessary for success.

• Improving and evaluating – This explains the two complementary objectives of Human Factors in a project.

• Human Factors in project initiation – This is a programme level activity . The section explains how to ensure that the right work will be done within the project to resolve the people-related issues inherent in the concept being developed.

• Human Factors Integration processes – This describes the generic processes used to manage and direct all Human Factors technical work to ensure it integrates with the project’s wider objectives.

• Human Factors and validation planning (SPF) – This relates Human Factors processes to the Structured Planning Framework (SPF) of E-OCVM.

• Human Factors in the lifecycle – This describes how the Human Factors work changes as a concept matures through the E-OCVM concept development lifecycle.

• Doing the technical work – This gives a brief overview of Human Factors technical work.

2.1. Improving and evaluating

Human Factors validation work (evaluating whether the concept is good enough) should be closely related to Human Factors development work (improving/designing the concept). The two threads often rest on similar underlying technical activities (analysis, evaluation, assessment, experiment, etc) as well as feeding each other. For example:

• Evaluation that shows a way in which the current concept does not meet one of the claims provides a spur and insights that lead to its improvement.

• Work undertaken to develop the concept can provide insights into potential weaknesses that must be explored in order to validate it.

Both threads run through the project, but the balance changes, as the input (some sort of idea or immature concept) turns into the output (a better defined, more robust concept or system design). There should be continual small cycles of evaluation and intervention on many aspects of detail throughout, as well as major assessments (of input and output) at each end of the project, see Figure 4 (which is based in Figure 1).

• Near the beginning, assess potential risks and opportunities associated with the input concept. That should shape Human Factors interventions through the rest of the project, hence the upward arrow in Figure 4.



Date:

Document ID:

Revision:

28/05/2009


Draft




• Near the end, draw together evidence about the improved concept, to give confidence in its suitability from a Human Factors perspective, hence the downward arrow in Figure 4.

Improve the concept

Evaluate whether the concept isgood enough yet

HF Body ofEvidence

More matureconcept/system

Initialconcept(s)

Assess inputideas/concept

Assess outputconcept

Figure 4: Early and late evaluation within a project

(Section 3 describes the process of building the evidence for the evaluation thread.)

2.2. Human Factors in project initiation

Virtually all projects need some Human Factors work, but the type and amount can vary significantly, and it is easy to under-estimate the need. Once the project has been set up, with an outline plan, objectives and budget, it is much harder to change or extend the scope of Human Factors work, even if the desirability of doing so is realised. Some Human Factors considerations must therefore be included as part of the process of project initiation, definition and scoping

If you are working at programme level as the sponsor or planner of a project, then as early as possible, you need to consider what type and level of Human Factors activities are needed. To do that, you need to understand the main people-related issues associated with introducing the concept(s), and build in enough of the right type of Human Factors work to ensure that those issues can be adequately addressed.

Issue identification is a core Human Factors process, and you need to undertake a light-weight version of it, as shown in Figure 5 (pre-project activities are unshaded). Ideally you would begin the formal process described in the EUROCONTROL Guidance [5] during the pre-project phase (and pass the results to the incoming project team) but even if you don’t have the resources to do that, you should still make a systematic attempt to identify the issues.

First you need a clear idea of who the concept will affect (and how). Use the prompts and questions in 6.2.Appendix B to help. Involve key players: project manager and system design authority (if they are yet in place) and user/operational authority, as well as some specialist Human Factors support.



Date:

Document ID:

Revision:

28/05/2009


Draft




Understand people-related issues

Identify impact onpeople

Plan work toaddress the issues

Do the HumanFactors workin the project

Figure 5: Setting up a project – scoping Human Factors

When you understand the main people-related issues, you can ensure that the project has built in the necessary resources (including people of adequate experience and influence), plans and responsibilities, with milestones and deliverables that give appropriate prominence to the key people-related issues. When setting Human Factors objectives, it is helpful to link them to the project’s overall objectives (which are normally defined in terms of the ICAO KPAs (see 6.2.Appendix A).

When planning a new project, there are three main Human Factors concerns:

• Resource level - To allocate sufficient effort, with the right type of experience to do the work effectively

• Integration – To ensure that the results produced will be compatible with what is done in other parts of the project

• Timing – To ensure that information is available when it is needed to inform project decisions, and to feed into other project activities

It is easy to under estimate the effort needed for Human Factors. When people-related problems appear, it is often late in the lifecycle, but the most cost effective way to avoid them is by investing effort early in the lifecycle to identify and prevent the problems. Your plan should be informed by a mature view of the people-related issues, and when it is best in the lifecycle to address them.

To ensure that Human Factors work is properly integrated in the plan, you need to understand the links between Human Factors and other disciplines (see section 6.1) and also involve people responsible for other parts of the plan, to ensure that both ends of all dependencies and cross-flows between Human Factors and other activities match each other.

Timing can be difficult. Ideally many Human Factors results would be available early to inform other parts of the project, on the other hand, many of the Human Factors tasks will rely on information about the rest of the project, which won’t be available at the start. The plan must cater for all dependencies, by breaking down activities, producing interim results, or maintaining inter-visibility between parallel activities. It may be harder than doing the whole task in one piece, but it is more likely to produce results that contribute to the project’s overall goals.

Use the Human Factors project initiation checklist questionnaire in 6.2.Appendix C to help you.



Date:

Document ID:

Revision:

28/05/2009


Draft




Example (V1 project)

Pre-project : The concept will cause a significant re-allocation of tasks between different roles. Workload and task co-ordination are identified as issues. Significant resource is allocated to address them, and the plan includes interim reports on workload and job integrity.

In-project : Exploring the issues with users reveals that of the two roles whose workload would increase, the one with apparently more modest increase would also be subject to significant workload fluctuation whereas the role with greater increase would have a more stable workload. The workload task within the plan is therefore refined. Both roles will be subject to workload analysis and assessment, but the more costly process of time-based workload modelling will be used to explore the risk of workload peaks for the first role.

2.3. Human Factors integration processes

The bulk of the technical effort in a project is spent doing the actual technical analysis, etc, but technical processes are not enough on their own, they need direction to address the right issues, and integration to ensure that their results are timely and useful.

Identifying and managing people-related issues is central to any project. Their immediate use is to prioritise and scope the technical work (analysis, assessment, modelling, design, etc), and ultimately they will form the baseline for the Human Factors Validation Case3 (in the same way that a hazard list forms the baseline for a Safety Case). People-related issues should be tracked in some form of register to enable them to be linked, shared, reviewed, updated and referenced.

Figure 6 shows the basic process (understand, plan, do). The ‘do’ process (the detailed technical work) generates the outputs for the twin threads described in section 2.1.

Project Human Factors work

Understandimpact on people

People-relatedissues

Plan / reviseHF work toaddress the

issues

Do the technical work

HFevidence

Influenceon designdecisions

HF technicalactivity




Concept /System

development

Concept /System

validation

Figure 6: Human Factors process – within a project

3 See section 5



Date:

Document ID:

Revision:

28/05/2009


Draft




The EUROCONTROL Guidance for Human Factors Case [5] describes the critical processes of issue identification, assessment and prioritisation, see Table 2.

Table 2: Stages in EUROCONTROL Guidance for Human Factors Case

Stage Title Description 1 Fact finding Identify what will change, who will be affected, how they will be affected

2 Issue analysis Use workshops and interviews to identify potential Human Factors issues, assess their impact, and determine priorities

3 Action plan Identify actions, mitigation strategies and monitoring arrangements in response to the issues

4 Actions implementation

Conduct the planned studies, analysis, evaluation, etc and check that issues are dealt with

5 Human Factors Case review

Review effectiveness of the process, with lessons learned for improvement

It also includes appendices containing:

• Advice on: Group workshops,

• Descriptors for: Issues, Levels of expertise,

• Definitions for: Impact on human performance,

• Templates for: Fact finding form, Feedback form, Issue analysis Report, Action plan, Case report, and Review report

It provides the core risk-driven process for Human Factors, but it predates this (CAATS2) guidance, which extends the basic ideas earlier into the lifecycle, and with specific support for the concepts of E-OCVM. If you use version 2 of [5] (current at the time of writing) then apply the interpretations and additions listed in 6.2.Appendix D.

It is easy to identify dozens of people-related issues in a project, so they need managing and tracking with tools to organise and categorise them. Categories used in the Human Factors work, include: the parts of the Human Factors Pie (see Figure 2), the KPAs (see 6.2.Appendix A), and contribution to system PRRs (People-Related Requirements, see Appendix). You should also think about how to relate people-related issues to other issues being tracked by other parts of your project.

Stakeholder engagement is an important Human Factors process, not just as an input to Human Factors technical work, but also to help embed the operational concept into the operational community. In ATM, many of the people who are human components within the system are also stakeholders with a considerable degree of ‘ownership’ of it. They can have a strong influence over whether the changes associated with a new concept will be acceptable, and hence whether the concept will work as intended and deliver



Date:

Document ID:

Revision:

28/05/2009


Draft




benefits. Early engagement of stakeholders is essential for successful change (and is reflected in the ICAO ‘Participation’ KPA).

2.4. Human Factors and validation planning (SPF)

The Structured Planning Framework (SPF) in E-OCVM [1] uses a set of steps and sub-steps to ensure that the work to validate a Concept is properly conceived, executed and communicated to stakeholders. Table 3 shows the six steps, with an approximate mapping across programme, pre-project and in-project activities.

Table 3: SPF steps mapped between programme and project

Step Programme Project setup Project Programme 0 State

concept and assumptions

Outline concept and broad assumptions. Absorbing results of successive projects

Inherit previous knowledge. Make concept details and assumptions explicit

1 Set validation strategy

Develop overall strategy for concept lifecycle. Update progressively in the light of successive project results

Inherit overall strategy and develop outline project strategy in enough detail to scope project plan

Develop project strategy more fully within outline

2 Determine exercise needs

Ensure broad exercise objectives are clear and consistent with project plan, resources and dissemination tasks

Develop full plan for exercises, including target and criteria

3 Conduct the exercise(s)

Conduct all exercises, re-planning where needed to meet objectives

4 Determine results

Transform exercise results into validation evidence of the required quality

5 Disseminate information to stakeholders

Undertake planned dissemination tasks, including delivery of appropriate evidence to programme level

Disseminate across the programme, integrating evidence from projects, as a prelude for move to next level of maturity

Steps 0, 1 and 5 are ‘owned’ at programme level, rather than by individual projects, though each successive project has to review and develop them. Steps 2, 3 & 4 are



Date:

Document ID:

Revision:

28/05/2009


Draft




specific to each project, which interprets its role within the overall strategy, plans and conducts its validation work accordingly, and then contributes its results back for programme level dissemination.

Note that at some stages of the lifecycle it may not always be necessary to go through every step of the SPF, depending on what is appropriate.

Section 2.2 explained the importance of undertaking some high level Human Factors assessment as a part of the project set-up and planning, but the SPF covers an even wider scope, since it applies to the management of a programme embracing the whole concept lifecycle, and including potentially many projects.

The SPF has a total of 24 sub-steps. 6.2.Appendix G translates these into Human Factors terms, which generally entail one of:

• A more focused, people-related, version of the generic action

• A Human Factors contribution to the overall action

• A review of the generic item based on Human Factors criteria

Table 4 gives a set of checklist questions that can be used to check whether Human Factors has been adequately considered as part of a project’s overall use of the SPF.

Table 4: Checklist questions for Human Factors at each SPF step

Step (Human Factors) Checklist questions Step 0: Determine concepts and assumptions

Is it clear how the concept will affect people? If not, seek clarification. Are there hidden assumptions about people? If so, make them explicit so they can be challenged or confirmed.

Step 1: Set validation strategy. Are the people in the system adequately represented among the stakeholders, and in the concept description? Is the people-related maturity of the concept understood, and comparable with other maturity criteria? If not, how can it be brought into line? Is there a clear understanding of what is needed to bring the people-related maturity up to the next level, and to subsequent levels? Are the Human Factors expectations (for each phase) credible?

Step 2: Determine the needs for activities to generate Human Factors validation evidence.

Are stakeholders’ expectations for Human Factors (in this phase) clear? Is there a clear understanding of what must be done to meet those expectations (and how it fits in with the overall validation activities)? Are the Human Factors expectations (for each phase) credible? Is the Human Factors work feasible? Are there clear means to determine its success?

Step 3: Conduct Human Factors activities to generate validation evidence, including participation in overall validation exercises.

Has the work gone according to expectations? If not, then why, and what can be done to rectify things?



Date:

Document ID:

Revision:

28/05/2009


Draft




Step (Human Factors) Checklist questions Step 4: Determine the Results. Do the data stand up under scrutiny, and do they provide useful results?

Step 5: Disseminate Information to Stakeholders

Is there a clear understanding of who needs to know what about the Human Factors Case?

2.5. Human Factors in the lifecycle

Human Factors activity must work within the constraints imposed by different phases of the lifecycle. The aim is to identify issues and resolve uncertainty as early as possible, while recognising that expending the wrong sort of effort on an immature concept is unlikely to be cost effective. Table 5 shows how the Human Factors focus changes in each phase. The left hand columns are from E-OCVM [1].

Table 5: Focus of Human Factors at different life cycle phases

Concept development

Validation focus Lifecycle constraints

Human Factors validation focus

V0

- A

TM

Nee

ds Identify needs,

constraints and potential concepts

(Pre-validation) The case for an initial project often comes from effort within an organisation’s ‘departmental’ budgets4, when Effort for ‘analysis’ and assessment may be severely limited.

Ensure that the formulation of the concept, and hence the project goals, take account of the human dimension, not just technology.

V1

- S

cope

Describe concept(s) in enough detail to identify potential benefit mechanisms (system changes that may help meet needs).

Show the concept can alleviate barriers and enhance ATM performance to the required level. Some aspects remain unknown or unclear.

There is no clear view of how to implement the concept. Many aspects have not been considered. Building ‘prototypes’ is unrealistic (other than perhaps part task simulation in a critical area).

Develop descriptive models of the human component. Make explicit what people do, and how their performance contributes to the system. Identify side effects that could compromise human performance. Influence emergent design ideas based on preliminary Human Factors assessment. Relate human-centred models to other system models so as to ensure coherence.

4 Major initiatives like SESAR may be funded as EC R&D, but smaller initiatives are likely to have their genesis within an

organisation.



Date:

Document ID:

Revision:

28/05/2009


Draft




Concept development

Validation focus Lifecycle constraints

Human Factors validation focus

V2

– F

easi

bilit

y

Develop and explore concept(s) to point where they are considered operationally feasible. Use system prototypes (not fully developed technology). Iterate as required to explain unexpected findings. Establish new system behaviours

Focus on operability and operational acceptability. Stabilise operational procedures and requirements. Thoroughly test HMI, operating procedures and phraseology.

There is a better view of possible solutions, but uncertainty about whether they will work, or which is best. Building prototypes is feasible and necessary to explore behavioural issues. Prototypes will be ‘throwaway’, so their cost must be contained.

Use prototypes & mock-ups to explore all aspects of human interaction. Contribute to human-aware5 aspects of system design, including performance modelling and prediction. Supplement human in the loop experiments with task walk-throughs and/or predictive performance models where needed. Track all side effects and secondary issues. Highlight any major shortcomings related to people.

V3

– In

tegr

atio

n

Integrate required functionality in pre-industrial prototypes. Explore engineering processes to develop experience for building the end system. Establish standards & regulations necessary to build and operate the required technical infrastructure.

Focus on integrating operating procedures in relevant, realistic scenarios. Focus on system level behaviour and measure performance. Identify costs and benefits. Provide information about the potential performance of overall ATM system.

Confidence in a solution that can work in isolation justify greater expenditure on prototypes and experiments to explore wider interaction within the overall ATM system. Pre-industrial prototypes help to de-risk subsequent industrialisation, thus justifying their greater cost.

Use fully representative prototypes to test all aspects of human interaction, at all relevant levels. Ensure all human-aware aspects of design are adequate. Gather realistic human performance information. Supplement with modelling if required. Track and resolve all issues. Ensure full integration of Human Factors results into system design and performance evaluation.

V4

- P

re-

oper

atio

nal

Transform prototypes into industrial products … Address institutional issues of procedures approval.

V5

- Im

plem

enta

tion

Combine products and procedures to create an operational system at a specific site. Expect pragmatic ‘fixes’ ….

(Out of scope for R&D)

Out of scope for R&D)

(Out of scope for R&D)

5 The term ‘human-aware’ is used to denote any aspect of a system design, whether visible or not, that has a significant

effect on the way people perceive, understand and carry out their work. This includes all HMI, as well as for example, the sequence in which related events occur, the delay between stimuli and responses, the way in which information is organised and accessed, the grouping of tasks and responsibilities.



Date:

Document ID:

Revision:

28/05/2009


Draft




2.6. Human Factors technical work

This guidance is not a manual of Human Factors methods and techniques. Trained professionals bring their technical skills with them, and there are many other sources of information on Human Factors methods (see 6.2.Appendix N). The main focus of this guidance is on how to integrate Human Factors technical work within the wider context of E-OCVM R&D projects and programmes. If the plan, the resources, the responsibilities, the objectives and the integration are right, then competent people within the projects should be able to deliver the desired results.

The overall Human Factors integration process (understanding people-related issues and then conducting work to resolve them) is universal, but the technical work varies a lot depending on the nature of the concept, and the lifecycle phase in which it is being studied. Use 6.2.Appendix E to help define Human Factors work appropriate for each phase of the lifecycle. It is in two parts:

6.2.Appendix E.1 lists tailored Human Factors objectives for each R&D phase of the lifecycle, based on the following generic themes:

• Overall understanding (of the people-related issues)

• Issue management

• Stakeholders (and managing the relationship with them)

• Human Factors planning

• Influence (where the Human Factors work aims to make a difference)

• Human models (purposeful descriptions of people in the system)

• Human Factors data (attributes and variability of people)

• Human performance (assessment, evaluation and measurement)

• Operability (how well the system fits around the people)

• People-Related Requirements (system requirements to achieve human integration)

• Acceptance criteria (tests to make sure the technical system is fit for human use)

• People-related evidence (building the Human Factors Case)

• Project legacy (passing on lessons learned)

6.2.Appendix E.2 lists Human Factors tools and techniques appropriate for each R&D phase of the lifecycle, based on the five process stages listed in section 2.3.



Date:

Document ID:

Revision:

28/05/2009


Draft




3. Building the evidence This section focuses specifically on the work needed to generate evidence for Human Factors validation, i.e. to give confidence, as far as it is reasonable to expect at this point in the life cycle, that a solution implementing the concept will satisfy the twin Human Factors claims of operability and human performance (see section 1.3).

Human Factors validation needs are derived from a combination of:

• System level stakeholder concerns (typically expressed through the KPAs)

• Current Human Factors concerns (identified by issue analysis, see section 2.3)

• Exercise constraints (driven by lifecycle considerations, see Table 5)

The process uses a combination of the Structured Planning Framework (SPF) of E-OCVM (explained in section 2.4) and the EUROCONTROL Human Factors Case process [5]. Detailed process guidance for each step and sub-step is in 6.2.Appendix H. The sections below give more general explanation and guidance for work on each claim.

3.1. Working towards Human Factors claim 1 (system operability)

‘That the role of the human in the system will be consistent with human capabilities and characterist ics’

This is a pre-requisite for all other claims about human performance or wellbeing. System operability covers all aspects of the way that people are designed into the system, which is mainly about how well the rest of the system is designed around the people. So:

• Analysis is about understanding the effect of the system design on the people.

• Design intervention is about obviating adverse effects and enhancing positive ones.

• Evaluation is about producing trustworthy evidence that the overall collective effect will be acceptable.

During V1, input to the process is mainly descriptions, models and experience from existing system(s). There will be an aspiration for what the concept will do, but few facts about how it will do it, or whether it will work. Operability work will draw extensively on generic knowledge about how people interact within systems, interpreted in the context of the current concept. Conceptual designs (of interfaces, processes or procedures) can be ‘tested’ by review and structured walk-through with representative subjects.

During V2, there will be richer descriptions and models (new and inherited from V1), supplemented by various prototypes. Many aspects of operability can be tested by people doing quasi-real tasks in real time. Even the aspects that can’t be so tested may benefit from much greater realism and interactivity in non-real-time walk-throughs.

During V3, there will be substantial prototypes that can be integrated to support system-wide behaviour. The results of V2 should ensure that user interfaces are fairly mature in



Date:

Document ID:

Revision:

28/05/2009


Draft




respect of all localised tasks and interaction, thus enabling a focus on more widespread interaction between people and teams in realistic scenarios.

V4 and V5 are out of scope for R&D.

Table 6 gives a summary of the key steps for working towards Claim 1. For a more detailed process guidance, see 6.2.Appendix H.

Table 6: Process summary – building evidence for Claim 1 (operability)

Stage Human Factors Case process

Description

1 Fact finding Identify areas where operability is likely to be an issue

2 Issue analysis Identify aspects that represent high operability risk.

3 Action plan Determine what is feasibly to test human-technology integration

4 Actions implementation Undertake work to demonstrate satisfactory operability

5 Human Factors Case review Learn lessons relevant to integration of people within the system

Operability has many dimensions, and the most effective way to ensure that they are all covered is to use the check list in 6.2.Appendix I. It covers the following topics:

• Individual human role (responsibilities assumed, goals to be achieved, tasks and procedures)

• Collective aspects of human activity and relationships (team members’ roles and communication processes)

• Relationship with the system (allocation of functions between human and machine, working logic of system perceived by the human, interface with the system, format and content of information, lay-out and handling actions at the interface)

Some aspects of operability are amenable to quantification, based on the results of exercises with representative users. Metrics based on for example the proportion of users who can do a task unaided, the proportion of tasks done error free, or the proportion of errors recovered. This is useful for structuring operability in an objective way, and can be a valuable way of highlighting problem areas (for example if more than two trial users have the same problem) but the metrics are of limited use without a robust basis for setting the threshold values of acceptability.

There is an example of operability metrics for software in 6.2.Appendix I.1



Date:

Document ID:

Revision:

28/05/2009


Draft




3.2. Working towards Human Factors claim 2 (system performance)

‘That the contribution of the human within the syst em will support the expected system performance and behavio ur’

This is about understanding how human behaviour and performance contribute to desired system performance, and being able to demonstrate that the contribution will be adequate. Typically there will be a strong emphasis on: Capacity, Safety, Environment and Economy, but there may be a need to demonstrate benefits in any aspect of ‘performance’ defined by the eleven ICAO KPAs (Key Performance Areas), see 6.2.Appendix A.

System-specific models are used to relate the high level benefits sought to lower level performance metrics that can be predicted, modelled and eventually measured. The models must be tractable (so that results can be obtained) and credible (so that stakeholders will trust the results). The Human Factors role in system performance work is to ensure that the system view properly takes account of the role and contribution of the human component.

• All models of the overall system must expose the human contributions explicitly, in a way that enables them to be reasoned about within the overall system context.

• Models of the technical system must expose human interfaces explicitly in a way that enables the influence of the human link to be reasoned about in the context of the technical system.

• System functional models must be linked to credible human models (of tasks, jobs, relationships, etc) that can explain human behaviour and performance within the system context.

• System performance models must be based on supportable evidence about human behaviour and performance (human capacity, reliability, variability, etc)

In principle, the above goals are all owned by ‘someone else’ within a project, but it is impossible to achieve them without active Human Factors support.

Table 7 gives a summary of the key steps for working towards Claim 2. For a more detailed process, see 6.2.Appendix H.



Date:

Document ID:

Revision:

28/05/2009


Draft




Table 7: Process summary – building evidence for Claim 2 (human contribution)

Stage Human Factors Case process

Description

1 Fact finding Identify system performance models to be used.

2 Issue analysis Determine scope of human components within the models Identify the need for supporting data about people.

3 Action plan Agree data and/or ‘plug in’ human models to be provided.

4 Actions implementation

Work with model owners to develop human components in them. Undertake appropriate Human Factors work to provide the agreed human data and/or models. (Where appropriate) participate in allocation of performance budgets, targets and trade-off.

5 Human Factors Case review

Learn lessons relevant to inclusion of people in system performance models

Performance models vary in type and granularity, so you need to ensure early discussion between Human Factors and system performance modellers to identify the type(s) of model needed. Typically performance models may include:

• A functional breakdown of the system to show how its different parts interact.

• A supplementary model to show how the performance of different parts contribute to overall performance

• Plug-in component models to predict the contributory performance of different components

Models may be qualitative, e.g. based on influence diagrams and flow charts, or they may be quantitative, using analytic formulae, and they may be formulated at different levels of granularity. The type of model determines the sort of Human Factors input required.

The granularity in the model is driven by the need to understand the interaction between the components, especially where the components and/or the interaction will change. Where the model includes people, the level of detail must enable their performance and behaviour to be reasoned about.

Models of technical components tend to be analytic (e.g. maths and physics in a model of radar accuracy) whereas human performance models are often structured accounts of observed performance, strongly conditioned by contextual factors.

Table 8 provides a checklist for use when examining system performance models.



Date:

Document ID:

Revision:

28/05/2009


Draft




Table 8: Checklist for human representation in system performance models

Criterion Checklist question Does the model include all relevant people (not just the most obvious ones)?

Does it include all of their relevant activities (not just the primary ones)?

Validity

Does it take account of their actual performance (as opposed to their theoretical performance based on function alone)?

Can existing models of human performance meet the desired ‘plug in’ requirements?

Is it feasible to develop new models to meet the requirements?

Are suitable, validated, human performance data available to support the models?

Is it feasible to gather the required supporting data within the project constraints?

If the above are not feasible, can the (model/data) requirements be changed?

If they are changed, is the risk/uncertainty on system performance acceptable?

Tractability

If quantitative human performance data are not practical, is there a useful way to use qualitative data to achieve some of the desired result?

When planning and executing work on human performance, you need to allow for:

• ‘Non-linearity’ – Human performance cannot be assumed to be invariant when the job changes, even by a small amount. Work rate may not remain proportional to task demand, and accuracy, error probability or vigilance may vary disproportionally.

• Context – Human behaviour is strongly influenced by the work context, which must be understood in order to make any assertions about human performance. The contextual understanding should come from the work on Claim 1.

• Variability – People are inherently variable in their capability, though the variability is often masked in jobs like ATM where good job design and a professional ethos normally deliver consistent results despite underlying variability. The effect of individual variability must be considered when planning human-in-loop exercises. The selection and number of subjects needs to be such as to ensure that they will be representative of (the full range of) the target audience. Experimental design, including threshold criteria, should seek to expose and monitor performance limits.

Human performance statements made on the basis of the validation exercises should reflect the extent to which they are conditional on task context and human variability.

Table 9 shows some popular fallacies about human performance that may require countering.



Date:

Document ID:

Revision:

28/05/2009


Draft




Table 9: Examples of fallacious assumptions about human performance

Fallacy Common assumption Contra-indication Automation Automating parts of a task will

always lead to a higher work rate. The fragmented task might require more mental effort to understand the ‘invisible’ automated parts. There might be an overhead needed to correct inappropriate automatic actions.

Capacity People will continue to cope with all tasks presented to them

The new tasks might exceed a capacity limit that was not apparent with the previous task load

Determinism People will continue to behave and perform in the same way as at present,

Human behaviour is strongly influenced by context, which might have changed.

Divisibility Dividing a job between two people will allow them to do twice as much as one of them could

Not seeing the whole picture may reduce their ability to perform the tasks. They may incur a communication overhead.



Date:

Document ID:

Revision:

28/05/2009


Draft




4. Concept maturity criteria The purpose of R&D is to develop concepts from an initial idea to the point where their benefits, risks and costs have been shown to justify industrialisation and implementation. E-OCVM provides a lifecycle model, and the associated idea of concept maturity, to support judgements about how far a concept has progressed towards this goal. All aspects of the concept need assessing, including the people-related aspects.

At programme level , concept maturity criteria help to make decisions about new projects, or to review the results of existing/completed projects.

• When deciding to set up an initial project – the concept should satisfy Vn-1 maturity criteria before starting a Vn project.

• When launching the Vn project – it should be given Vn maturity criteria to act as project goals.

• When reviewing R&D results for a concept (or a set of related concepts) from prior projects, it/they should collectively meet Vn maturity criteria before moving on to a Vn+1 project to take the concept forward.

If Human Factors maturity is less than (say) technical maturity, then if you initiate a project at a ‘later’ phase (say E-OCVM phase V3 rather than V2) you need to consider the associated risks and how to mitigate them, e.g. by additional Human Factors work in specific risk areas early in the project. You also need to ensure that the project team has a clear view of the Human Factors maturity targets that you want to be met at the end of the project.

At project level , you need to understand the current maturity of the concepts that you are working on, especially if any aspects are less mature than the rest. You also need to assess how far the project has progressed towards meeting the programme aspirations.

The maturity of a concept is about how well it is defined, and how well it (and its associated risks and benefits) are understood. This information rests on the credibility of the processes used to explore the issues. Stakeholders will place little confidence on a result ‘we have found no problems’ if you haven’t looked very thoroughly, or if you haven’t properly analysed the information that you found, or if you haven’t yet defined the concept in enough detail to be able to assess it properly.

There are three broad sets of criteria for Human Factors maturity:

1. Concept definition criteria – How well the people-related aspects of the concept are defined

2. Issue criteria – How well the people-related issues are understood, and how serious they are

3. Process criteria – The adequacy of the work underlying the evidence on the other criteria



Date:

Document ID:

Revision:

28/05/2009


Draft




In all cases, the criteria need to be interpreted in the light of what it is reasonable to expect at different lifecycle phases – both the methods used, and the extent to which uncertainty should have be removed.

Figure 7 shows how each group of criteria maps onto the work done by the project(s) generating the supporting evidence.

HF Maturity criteriafor transition: Vn-Vn+1

People-relatedissues & understanding

Concept definition(HF aspects)

HF Process

Human Factors processes in Vn project

Prove people aspects of concept(work towards Claim 1 & Claim 2)

Improve / develop people aspectsof the concept

Initialconcept

from Vn-1Identify, track and manage people-related issues

More developedconcept to Vn+1

Evidence forHF Claims 1&2

Issues outstanding(Agenda for Vn+1)

Figure 7: How transition criteria relate to project processes and results

These three groups of criteria are summarised here. 6.2.Appendix J contains tables of criteria to be met for each item at successive phases of the E-OCVM lifecycle

Maturity criteria mostly rely on an element of judgement. Some may be simply expressed (e.g. ‘X is defined’ or ‘Y has been assessed’) but behind them lie implicit questions about whether the definition is sufficiently complete, and whether the assessment was thorough enough. Also, in early lifecycle phases, with only partial, rather than full maturity expected, there will be a mix of criteria that are fully, partially or not yet satisfied, so your assessment must be able to make sense of this. The criteria in 6.2.Appendix J include many qualitative or comparative words to help (e.g. ‘key’ items at V1, ‘most’ items at V2 and ‘all’ items at V3) but they still require interpretation and informed judgement within the relevant context.

Human Factors maturity metrics are mainly qualitative6. Multiple judgements can be combined in numerical form (e.g. 90% of critical issues resolved, 45% of non critical issues resolved) but that does not generate a truly quantitative metric. It is simply an aid to making a judgement about whether there has been an acceptable degree of resolution among a large number of issues. The ultimate judgement about maturity still rests on the qualitative endorsement of decisions about what is or is not ‘critical’, an understanding of the residual uncertainties, and a willingness to tolerate the particular issues that remain unresolved.

6 Performance metrics are more likely to be quantitative, (e.g. peak work rate, mean throughput, task error rate) but they

are distinct from maturity criteria.



Date:

Document ID:

Revision:

28/05/2009


Draft




4.1. Concept definition (people-related aspects) criteria

An early concept contains a lot of uncertainty, not just uncertainty about its impact and benefits, but about what it actually is. In the early phases the concept will only be defined in outline with a lot of detail left open, and different people may have different impressions about what it entails. As it matures, the detail should be progressively filled in to reduce the uncertainty.

It is particularly important to ensure that the human dimension of the concept (and or the system that will implement it) is defined to an adequate level of clarity and detail, because it can easily get left behind (and often did in traditionally run projects).

The maturity of the Concept, and the evolving socio-technical system that will implement it, can be assessed in terms of the extent to which the following aspects of the human part of the system are defined:

• The human roles affected by the concept (to bound human implications of change)

• Responsibilities of the roles

• Tasks (what people do and what information they use)

• Working methods (how they are to do it)

• Procedures

• Team structures (how people and their work relate to each other)

• Communication processes and means (how communication will be supported)

• Information exchanged between actors (controllers, pilots, technical personnel, managers, …)

• People-Related Requirements (PRRs) (system requirements for effective human integration)

At the start of V1, little will be defined, but the pre-project thinking should have included the key roles involved and how their tasks and relationships would be affected, especially for any novel aspects of the proposed change. By the end of V1 there should be a much more comprehensive understanding of all aspects, but not in full detail except for critical areas. Projects in V2 and V3 should progressively complete the picture and deliver a set of People-Related Requirements

6.2.Appendix J.1 contains a table for assessing each aspect listed above, at each phase of the concept lifecycle between leaving V0 and entering V4.



Date:

Document ID:

Revision:

28/05/2009


Draft




Example : When we talk about the concept, can we be sure we are talking about the same thing?

Project A and project B ran real-time human in the loop experiments on ‘the same concept’, but the concept as defined didn’t specify the type of airspace. Project A used one type of airspace and project B used another. They got opposite results, which made it difficult to share any of their work.

The type of airspace clearly affects performance, and should have been defined as part of the concept. That would have enabled some important questions to be answered:

1 – To which type(s) of airspace do the stakeholders want to apply the concept?

2 – Which of those airspace types are most likely to probe the limits of the human performance envelope, and in what ways?

3 – Do we need modelling or analysis (in V1) to de-risk the answer to the above question?

4 – What is the most cost effective set of experiments to plan in V2, to ensure a coherent set of results for the whole of the required performance envelope?

With the concept more fully defined (including airspace aspects) during V1, and with the questions above answered, the risk that V2 projects will produce incoherent results will be much less.

4.2. People-related issue criteria

The

D.17 GUIDANCE FOR HUMAN FACTORS CASE WITHIN E-OCVM ... · Cooperative Approach to Air Traffic...

Documents

Transcript of D.17 GUIDANCE FOR HUMAN FACTORS CASE WITHIN E-OCVM ... · Cooperative Approach to Air Traffic...