From a pyramid to traffic lights

Assessing the effectiveness of QAA

Douglas BlackstockDirector of Administration &

Company Secretary

Governance at QAA• QAA is a company and a charity• Regulated by Companies House

and two charity regulators• Has a board of 15 – seven

independents (one is a student) and nominees from the funding councils and institutions representative bodies from each of the four UK nations

• Is funded through contracts with funding councils and through institutional subscriptions

Key purposes of QAA

• Protection

• Communicate

• Improvement

• Understanding

More detail in our strategic plan at www.qaa.ac.uk/aboutus

About us

• 140 staff (125 full time equivalent)

• Up to 1000 auditors and reviewers

• Over 200 audits and reviews in 09-10

• Budget for 09-10 is €14.5million

• C. 200 corporate publications per annum plus review reports

• www.qaa.ac.uk/aboutus

Where I fit in

• Management of infrastructure servicesCommunications, Finance, Human Resources, Information systems, Office Services

• Cross QAA workBusiness development, information management, student engagement and

car parking

Assessing our effectiveness

• 2003 we considered EFQM but couldn’t secure commitment

• restarted 2006 suggesting a simple set of 10-12 Key Performance Indicators

• got complicated through 06 and 07 with a proposal for 120+ performance measures and the arrival of a pyramid

Mission & Purpose

External profile

Working worldwide

Offering expertise

Safeguarding standards

Reducing Regulation

Supporting and enhancing

quality

Management and Governance

ExecutiveBoard

SupportPeopleFinance

Core BusinessReviews and audits

Supporting development and enhancementWorking with others

Knowledge transfer and relationships (outputs)

QAA infrastructure and people (inputs)

Outcomes

Measurement

Assessment

But

Would this actually tell us if we are

meeting our purposes?

2008 - three tier system

An analysis of

Performance & Success

Would help us assess

Impactand produce a traffic light summary

The traffic lights(based on Committee of University Chairmen guidance)

Excellent: this is on track; performing well and is substantially effective.

Good; this is broadly on track; performance is satisfactory with one ortwo areas which could be addressed.

Mixed: some significant concerns which could be damaging and result in ineffective performance if not addressed.

Problematic: serious concerns threaten this area; this has the potential to become substantially ineffective.

We set about gathering data(in the style of Donald Rumsfeld)

• What we knew already• What we knew someone else

knew• What we thought someone

should know but were not sure they did know

• What we wanted to know and knew we could find out

• Does what we know or don’t know help us know whether we are achieving our aims?

QAA’s performance

Programmes of work (quarterly monitoring of operating plan)• Key contractual commitments met e.g. reviews and audits

completed, reports published• Progress against non-contractual activities e.g. papers published,

other publications, events held, presentations given, attendance at external events

• Exceptions and achievements• Any shift in risk External evaluation (some annual, some periodic)

• External auditors• Internal auditors• External standards – for Information security, human resources

management, website accessibility and our Welsh Language Scheme• ENQA, Higher Education Regulatory Review Group, quality groups in

Scotland and Wales

QAA’s performance (2)

Staff satisfaction (quarterly and annual)• Overall indicators from the annual staff survey• Quarterly staff turnover, absence, recruitment

analysis

Financial performance (quarterly and annual)• Monitoring and achievement against budget;

budget variances• Balance sheet• Forecasts from quarter 2 onwards

QAA’s success

Effectiveness of processes• Institutional and reviewer / auditor evaluations of

reviews and audits; reviewer / auditor evaluations of their training; participants’ evaluations of QAA events

• Renewal of contracts; repeat business• New business (from current clients and new clients)• Success of ‘engagement’ strategies – students,

employers, bodies representing the professions or with regulatory powers

QAA’s success (2)

Adding value• Demand for our ‘products’ – hard copy reports, web traffic,

external attendance at our events• Demand for our services – advice, contribution, participation,

expertise, briefing, presentations• Hot topics – our take, our engagement• Interactions at government level• Interactions with non-HEI stakeholders• External review of QAA, such as ENQA membership review• Institutions taking our reports seriously and putting right any

identified problems• External partners, the funding councils, student bodies and

institution representative bodies

QAA’s success (3)

The difficult to measure– relationships, mood, reception• The ‘mood’ of the regular meetings with the funding

councils and representative bodies (rectors councils?) in each of the countries

• The ‘mood’ of the twice-yearly meetings of the Sounding Board group ( made up of all QAA funders)

• The ‘mood’ of, and intelligence from, QAA liaison officers’ interactions with individual institutions

• The ‘mood’ of the QAA Annual Reception and the annual Subscribers’ meeting

• The amount, mood and tone of press coverage• The annual survey of Higher Quality readers

QAA’s impact

To what extent the performance and success data allowed the Board to make an assessment of the achievement of each of our Key Purposes.

The Board would then need to identify whether additional information is needed to help it assess QAA’s impact.

What did we find? Key purpose

Information

Safeguard the student and wider public interest in the maintenance of standards of academic awards and the quality of higher education

Communicate information on academic standards and quality to inform student choice and employers’ understanding, and to underpin public policy-making

Enhance the assurance and management of standards and quality in higher education

Promote wider understanding of the nature of standards and quality in higher education, including the maintenance of common reference points, drawing on UK, other European and international practice

Programmes

External evaluation

Staff satisfaction

Financial management

Effectiveness of processes

Adding value

Reception

Overall assessment Overall assessment Overall assessment Overall assessment

What did we learn?

• The process was very demanding of staff time and we could simplify the process

• The traffic lights help you focus on risk• In some areas we needed to improve

information management and sharing• Some of the information available did not

contribute to an assessment of the achievement of certain key purposes.

and finally

Whilst measurement can inform…

ultimately, any assessment of the achievement of key purposes is a

JUDGEMENT based on an array of evidence, not a measurement.

d.blackstock@qaa.ac.uk

The Quality Assurance Agency for Higher Education. Registered charity numbers 1062746 and SC037786