Medical Statistics Team, University of Aberdeen

Neil Scott

Statistical Consulting

Overview

1. Experience of operating a statistical consulting service (Medical Statistics Team, University of Aberdeen)

2. Review of relevant literature:• Not systematic• Considers any type of statistical consultancy (not just medical

statistics)

3. General discussion – your experiences with statistical consulting

Carter et al. (1986)

• Unique role of statistics: both a science and a part of the methodology for all scientific subjects

• Statistical thought is needed at every stage of quantitative research investigations

• A statistics department may assume academic responsibility beyond home department

Types of statistical consulting

• Ad hoc advice• Drop-in clinic• Appointment system• Formal agreement to produce data analysis/ report• Overlap with research/teaching roles

Who consults?

• Students• University staff• NHS staff• External

Statistical Consultancy Service - Medical Statistics Team

• Students (taught course)• Daily student clinic 1-2pm (nine statisticians contribute to rota)• May need to queue

• PhD students/MSc by research• Appointment system (supervisor must attend)

• University of Aberdeen staff• Appointment system (first half hour is free, then charged)

• NHS Grampian staff• Appointment system (free advice/analysis unless large amount of work)

• Project collaboration (sometimes begins as consultancy)

Logged consultations (2012)

Number of logged entries

Mean length (minutes) Total time (hours)

Student clinic 315 21.8 114

University staff / PhD students 40 50.1 33

NHS staff 97 61.5 99

Total 452 32.8 246

Student consultations by month (2012)

When do clients consult a statistician?

• Stages of the research process:• Planning

• Study design• Sample size calculation• Grant application/ethics form

• Data collection• Data management/ setting up a database

• Data analysis• Most common time to seek help

• Dissemination• Publication process• Responding to reviewer comments

• "To consult a statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of."

R. A. Fisher, in an address to the First Indian Statistical Congress in 1938

Organisational issues (personal experience)

• Drop-in student clinic has peaks and troughs of demand• Some clients want immediate appointments/ specific times• Many people are “hybrids” (University staff/ NHS staff/

student) • Occasional demand for remote consultation for

staff/students based off site: • face-to-face (or phone) preferred to e-mail

Common situations with face-to-face consultations

• Short time scale: “I need to present this next Tuesday”• Understanding unfamiliar topic area/ clinical jargon overwhelming• Overcomplicated studies (small sample size, too many variables), e.g.

waste five minutes searching to find correct variable in SPSS• Repeat clients (regular customers)• Timing of advice (e.g., if they had come at design stage, the analysis

stage would have been easier)• Client wants to copy a method used in another published paper: i.e.

“me too” publication

Common situations (when performing data analysis for client)

• Messy data: e.g. Excel spreadsheet with blank spaces / mixture of text and numbers• Hand (2010):

• “90% of data analysis is data cleaning” • Analysing the data is a cyclic process: interaction with the client

• Timescale to complete work or get back to someone – how long is reasonable?• Unclear/vague projects:

• Client has only vague idea of research aims - these may evolve over time• Unclear about definitions of variables/ exactly which variables to analyse• Multiple requests for analysis that add up: “Prof Bloggs wants me to add another

three p-values / look at another subgroup”• Clients sending data with identifiable information

Clients outside applied health research

• Laboratory research:• Designed experiments• Repeated measurements on small number of samples

• Psychology:• Everything expressed as effect sizes!

Generalist versus specialist

• Consultant statistician analogous to being a GP• Questions can be on any area (from trivial to extremely

complex/specialised)• May have to emphasise you are not expert on every aspect of

statistics:• OK to:

• Say you don’t know the answer• Get back to them• Refer to colleague with appropriate knowledge

Financial terms

• Discussing charging can be awkward• Difficult to judge total time on project• Survey of academic consulting units [Niland (1995)]

• Most aim to recover costs• Some degree of conflict between service and research roles of academic

statistician: • “To maintain a collegial relationship with investigators, it was suggested that

only time spent in more routine tasks such as data entry and programming be charged back to investigators, and more conceptual activities such as proposal design, grant, and manuscript preparation be provided without charge.”

Statistical Consulting Cycle [Kennett & Thyregod (2006)]

1. Problem elicitation2. Data collection3. Data analysis4. Findings formulation5. Findings presentation

Statistics courses in academia tend to focus on step 3

Problem elicitation

• Active listening• “Determine what the client wants to know: may not be what they say

at first (they are not statisticians)” [Hand (2010)]• First question often is: “How exactly would you describe your research

question?”• Common question after client gives too much information to take in

at once: “Could you go back a step?”

Kennett & Thyregod (2006)

• “Hunter (1979)…described that two chemists entering his office, when asked to describe the problem they came to get advice on, entered a lengthy discussion that eventually led to a common reformulation of the problem.

• Once they had the problem clearly formulated they knew how to solve it and did not need the help of a statistical consultant any more.”

Consultancy skills (American Statistical Association, 2003)

• Knowledge of statistics• Problem solving• Communication skills• Flexible/ can meet deadlines• Patience• Experience of topic area• Understands resource constraints• Shows respect for statistical and research ethics

Hand (2010)

• “Statistical consulting requires skills and expertise completely distinct from academic and technical expertise”

• “What you need to be a successful consultant: • Affability • Availability • Ability • in decreasing order of importance!”

• “Client relationship is the most important issue” • “Put yourself in the client’s shoes”

Johnson and Warner (2004)

• Studied factors associated with consultation being success• Used factor analysis/ logistic regression [!]• Most important factors:

• Statistical/collaboration skills• Personal qualities• Client requested particular consultant

Clinician – statistician communication

Bạn đã cố gắng một mô hình hồi quy hoặc một bài kiểm tra chi bình phương và bạn đã kiểm tra rằng các giả định được đáp ứng?

甲状腺刺激ホルモン受容体遺伝子の生殖細胞突然変異を引き起こす自己免疫性常染色体優性甲状腺機能亢進症

Effective communication

• Use appropriate language: ask client about knowledge of statistical tests / p-values to ensure communicate at correct level

• “Use the language of the customer” [Kennett &Thyregod (2006)]• “Be aware of the client’s level of technical knowledge

• They won’t have your statistical expertise • They will know about the problem and the data • They are paying you • They did not come in for a lecture on statistics” [Hand (2010)]

Organisational problems of consultancy [Sprent (1970)]

1. Lack of time or background knowledge to become a full member of an experimental team

2. The statistician may not have adequately adapted outlook to the computer era3. The statistician may be called in too late or dropped too early4. There is little or no training in the art of consultancy5. In some universities little account is taken of the consultancy load in staffing and

successful consultancy is not always a criterion for promotion6. In research institutes there is a danger that over-specialization may lead to stagnation,

as modern developments may be overlooked7. There is little informed criticism of consultancy, and many of us are not keen on

practising the art of self-criticism

Client stereotypes

• Hyams (1971), Sprent (1970)• “Probabilist” – only interested in a significant p-value• “Sporadic leech” – stops the consultant in the hall for a brief

friendly chat• “One-technique amateur statistician”

Learning from experience

• Both client and statistician learn from experience of consultation• Joliffe (2001)

• The consulting activity of statisticians can only enhance their teaching• Wilson (1992)

• Problems encountered in consultancy settings can lead to new statistical methods• Carter et al. (1986)

Benefits of consulting

• Exposure to variety of methods• Real world problem solving• Collaboration with others• Rewarding (helping others/ ensuring quality of

research)

Potential disadvantages

• Recognition - consultancy comes under heading of “service”: less prestige than teaching or research [Gibbons and Freund (1980)]

• Not initiating own research• Acknowledgement, authorship or nothing at all?:

• May do a lot of work without any reward• Concern that do not get to check published material with your name on it!• Checking a paper thoroughly is time-consuming• Additional work may be required following referee comments• May work mainly on small-scale, low-impact research

Russell (2011)

• “Advertisements for University consultants frequently offer the position at Lecturer level, or even at Research Fellow level, suggesting that the position is seen as a low-level role requiring not too much statistical maturity. Nothing could be further from the truth.”

Discussion

• Your experiences with statistical consulting:• Demand for consultancy in your institution• Organisation of consultancy services?• Who consults? Staff/students?• Staff time/costs• Etc.

References (1)

• American Statistical Association (2003). When you consult a statistician … what to expect

• Carter RL, Scheaffer RL, Marks RG (1986). American Statistician, 40(4), 260-264.• Gibbons JD, Freund RT (1980). Organizations for statistical consulting at colleges and

universities. American Statistician, 34(3), 140-145.• Hand DJ (2010). Statistical consultancy.• Hyams L (1971). The practical psychology of biostatistical consultation. Biometrics,

27(1), 201-211.• Johnson HD, Warner DA (2004). Factors relating to the degree to which statistical

consulting clients deem their consulting experiences to be a success. American Statistician, 58(4), 280-289.

• Joliffe F (2001). Learning from experience. In: Batanero C (ed.) Training researchers in the use of statistics, 355-370.

References (2)

• Kennett R, Thyregod P (2006). Aspects of statistical consulting not taught by academia. Statistica Needlandica, 60(3),396-411.

• Kirk RE (1991). Statistical consulting in a university: dealing with people and other challenges. American Statistician, 45(1), 28-34.

• Niland (1995). A survey of biostatistical consulting units throughout North America. American Statistician, 49(2), 183-189.

• Russell (2011). Advice for the potential statistical consultant. Proceedings of the Fourth Annual ASEARC Conference, Parmatta, Australia.

• Sprent (1970). Some problems of statistical consultancy. JRSS(A), 133(2), 139-165.

• Wilson (1992). Statistical consulting is scholarship. American Statistician, 46(4), 295-298.