CGEA2011Patient Centeredness
1
Is There A Reliable Way of Evaluating Personal Statements as Indicators of Potential for Patient Centeredness in Medical School Applications? Nathan Beucke MD, Rachel Brown MBBS, Alison Martin MEd, Lindsey Gentry BS University of Missouri School of Medicine ABSTRACT Objective: Personal Statements provide a means of looking at the personal characteristics of applicants. Need for Innovation: The mission statement for our school requires the committee to select students who “will excel as patient centered physicians.” In the absence of a readily available tool that would enable us to select such students, our faculty turned to the personal statement as one such available tool. Methods: A faculty/administration working group met and considered behaviors that indicate patient centeredness as developed through a series of focus groups. The group read statements written by medical school applicants. The group then developed a numerical scale anchored with descriptive statements. Through an iterative process a scale was developed. Educational Outcomes: The Patient Centered Personal Statement Scale (PCPSS) was used as part of the admissions process for the entering class of 2009. The PCPSS has been found to be a useful tool for evaluating an applicant’s potential as a patient centered physician. Educational Significance: The PCPSS is a new tool to use for holistic review efforts. BACKGROUND/NEED FOR INNOVATION Even though many hours are spent writing, reading and discussing applicants’ personal statements, very little research has been done into their potential usefulness in the selection of medical students. Siu and Reiter (1) lay out the issues that affect the usefulness of personal statements. Many applicants receive input from others in the development of the statement (2). Statements can differ greatly in content because of their free-form nature and applicants can have a vested interest in presenting themselves at “the saintly end of the spectrum” (1). Ferguson has examined personal statements as predictors of performance throughout medical school. Their analysis showed a very modest relationship between information in the personal statement and clinical performance in medical school (4). Other means of predicting performance, such as multiple mini interviews, are expensive to develop and deliver. The MCAT is largely a prediction of performance on academic test in the first two years of medical school. Our mission is to graduate physicians who will provide patient centered care for the people of Missouri and beyond. We ask our admissions committee to select students who will excel at this. In the absence of a readily available tool that would enable us to select such students, our faculty turned to the applicant personal statement as one such available tool. METHODS The University of Missouri School of Medicine developed a set of observable behavioral statements that describe patient centered care. These statements were developed and validated using qualitative methods. Qualitative thematic analysis included triangulation of patients, faculty and medical student themes. In the course of 2008-2009 a faculty and administration focus reviewed these discrete observable behavior statements. The group also read a number of personal statements written by successful medical school applicants. An anchored numerical scale was developed to evaluate personal statements written by successful medical school applicants. Through an iterative process, a usable scale, the Patient Centered Personal Statement Scale (PCPSS) was developed. In 2010 three raters began using the scale. The group read ten personal statements and assigned an individual score. The group then met and discussed their scores highlighting specific examples of empathy or service. If there was variation in the rating the group discussed and arrived at a consensus score. EDUCATIONAL OUTCOMES The PCPSS has been found to be a useful tool for - evaluating an applicant’s potential as a patient centered physician. The scorers had unanimous agreement on 29% of personal statements reviewed to date. Two of three scorers agreed on 88% of the statements prior to group review. After group discussion, consensus was reached on 100% of personal statements reviewed. Surveys of admissions committee members revealed that a vast majority (93%) use personal statements to assess patient centeredness (table 1). The rating was easy to find on the AMCAS application and most members agreed that the PCPSS accurately reflected their view of the personal statement. However, the data was mixed regarding the influence that the PCPSS had on individual committee members. DISCUSSION & CONCLUSIONS Our data demonstrate that the PCPSS may be a useful tool for medical schools to utilize as part of the holistic review of applicants, and in evaluating potential patient centeredness. For applicants, the PCPSS requires minimal investment, since they are already required to submit a personal statement. Scoring the statements requires some investment of time, but our data demonstrates that it is possible to train raters to agree on the appropriate score. Our interviewers largely agreed with the scores given, but a majority were not sure or did not feel it influenced their assessment of an applicant. Our data suggest the interviewers' reluctance may be, in part, because they use other aspects of the application as well as an applicant’s responses during an interview to assess potential patient centeredness. Our future goals include: refinement of the scale and of inter rater reliability; development of an extended scale that will include evaluation of reported activities and possibly letters of recommendation; and long term validation against later tests of clinical aptitude, including our end of 3rd year Patient Centered Objective Structured Clinical Examinations ( PCC OSCE). REFERENCES 1. Albanese MA, Snow MH, Skochelak SE, Huggett KN and Farrell PM (2003). Assessing personal qualities in medical school admissions. Academic Medicine, 78(3), 313-321 2. Ferguson E, James D, O’Hehir F, Sanders A. Pilot study of the personality, references, personal statements in relation to performance over the five years of a medical degree. BMJ:326:22Feb 2003: 429-431 3. Hanson MD, Dore KL, Reiter HI and Eva KW (2007). Medical school admissions: Revisiting the veracity and independence of completion of an autobiographical screening tool. Academic Medicine, 82, S8-S11. 4. Siu E, Reiter HI. Overview: what’s worked and what hasn’t as a guide towards predictive admissions tool development. Adv in Health Sci Educ (2009) 14:759-775 Strongly Agree Agree Not Sure Disagree Strongly Disagree No Answer Personal Statement 54 40 5 2 0 4 Reported Activities 51 46 4 0 0 4 Letters of Recommendation 32 55 7 4 0 7 Interview Responses 59 29 7 0 0 7 Table 1. Assessing Applicants’ Patient Centeredness 0 10 20 30 40 50 60 1 2 3 4 5 Number of Interviewer Responses (N=75) AC Response (1=strongly Agree; 5=Strongly Disagree) Admissions Committee Survey Responses PCPSS agreement PCPSS influence QUOTES ABOUT SCALE PCPSS was confirmatory! - AC Member 1 I enjoy the PCPSS. - AC Member 3 I was prepared for a self - centered applicant based on my assessment and the PCPSS, but was very pleased to be wrong. Applicant was personable, well - balanced young man. - AC Member 13
-
Upload
alison-martin -
Category
Documents
-
view
9 -
download
0
Transcript of CGEA2011Patient Centeredness
Is There A Reliable Way of Evaluating Personal Statements as Indicators of Potential for Patient Centeredness in Medical School Applications?
Nathan Beucke MD, Rachel Brown MBBS, Alison Martin MEd, Lindsey Gentry BS
University of Missouri School of Medicine
ABSTRACT
Objective: Personal Statements provide a means of looking at the personal characteristics of applicants. Need for Innovation: The mission statement for our school requires the committee to select students who “will excel as patient centered physicians.” In the absence of a readily available tool that would enable us to select such students, our faculty turned to the personal statement as one such available tool.Methods: A faculty/administration working group met and considered behaviors that indicate patient centeredness as developed through a series of focus groups. The group read statements written by medical school applicants. The group then developed a numerical scale anchored with descriptive statements. Through an iterative process a scale was developed.Educational Outcomes: The Patient Centered Personal Statement Scale (PCPSS) was used as part of the admissions process for the entering class of 2009. The PCPSS has been found to be a useful tool for evaluating an applicant’s potential as a patient centered physician.Educational Significance: The PCPSS is a new tool to use for holistic review efforts.
BACKGROUND/NEED FOR INNOVATION
Even though many hours are spent writing, reading and discussing applicants’ personal statements, very little research has been done into their potential usefulness in the selection of medical students. Siu and Reiter (1) lay out the issues that affect the usefulness of personal statements. Many applicants receive input from others in the development of the statement (2). Statements can differ greatly in content because of their free-form nature and applicants can have a vested interest in presenting themselves at “the saintly end of the spectrum” (1). Ferguson has examined personal statements as predictors of performance throughout medical school. Their analysis showed a very modest relationship between information in the personal statement and clinical performance in medical school (4). Other means of predicting performance, such as multiple mini interviews, are expensive to develop and deliver. The MCAT is largely a prediction of performance on academic test in the first two years of medical school.Our mission is to graduate physicians who will provide patient centered care for the people of Missouri and beyond. We ask our admissions committee to select students who will excel at this. In the absence of a readily available tool that would enable us to select such students, our faculty turned to the applicant personal statement as one such available tool.
METHODS
The University of Missouri School of Medicine developed a set of observable behavioral statements that describe patient centered care. These statements were developed and validated using qualitative methods. Qualitative thematic analysis included triangulation of patients, faculty and medical student themes. In the course of 2008-2009 a faculty and administration focus reviewed these discrete observable behavior statements. The group also read a number of personal statements written by successful medical school applicants. An anchored numerical scale was developed to evaluate personal statements written by successful medical school applicants. Through an iterative process, a usable scale, the Patient Centered Personal Statement Scale (PCPSS) was developed. In 2010 three raters began using the scale. The group read ten personal statements and assigned an individual score. The group then met and discussed their scores highlighting specific examples of empathy or service. If there was variation in the rating the group discussed and arrived at a consensus score.
EDUCATIONAL OUTCOMES
The PCPSS has been found to be a useful tool for -evaluating an applicant’s potential as a patient centered physician. The scorers had unanimous agreement on 29% of personal statements reviewed to date. Two of three scorers agreed on 88% of the statements prior to group review. After group discussion, consensus was reached on 100% of personal statements reviewed. Surveys of admissions committee members revealed that a vast majority (93%) use personal statements to assess patient centeredness (table 1). The rating was easy to find on the AMCAS application and most members agreed that the PCPSS accurately reflected their view of the personal statement. However, the data was mixed regarding the influence that the PCPSS had on individual committee members.
DISCUSSION & CONCLUSIONS
Our data demonstrate that the PCPSS may be a useful tool for medical schools to utilize as part of the holistic review of applicants, and in evaluating potential patient centeredness. For applicants, the PCPSS requires minimal investment, since they are already required to submit a personal statement. Scoring the statements requires some investment of time, but our data demonstrates that it is possible to train raters to agree on the appropriate score. Our interviewers largely agreed with the scores given, but a majority were not sure or did not feel it influenced their assessment of an applicant. Our data suggest the interviewers' reluctance may be, in part, because they use other aspects of the application as well as an applicant’s responses during an interview to assess potential patient centeredness. Our future goals include: refinement of the scale and of inter rater reliability; development of an extended scale that will include evaluation of reported activities and possibly letters of recommendation; and long term validation against later tests of clinical aptitude, including our end of 3rd year Patient Centered Objective Structured Clinical Examinations ( PCC OSCE).
REFERENCES
1. Albanese MA, Snow MH, Skochelak SE, Huggett KN and Farrell PM (2003). Assessing personal qualities in medical school admissions. Academic Medicine, 78(3), 313-321
2. Ferguson E, James D, O’Hehir F, Sanders A. Pilot study of the personality, references, personal statements in relation to performance over the five years of a medical degree. BMJ:326:22Feb 2003: 429-431
3. Hanson MD, Dore KL, Reiter HI and Eva KW (2007). Medical school admissions: Revisiting the veracity and independence of completion of an autobiographical screening tool. Academic Medicine, 82, S8-S11.
4. Siu E, Reiter HI. Overview: what’s worked and what hasn’t as a guide towards predictive admissions tool development. Adv in Health Sci Educ (2009) 14:759-775
Strongly
Agree
Agree Not Sure Disagree Strongly
Disagree
No Answer
Personal
Statement54 40 5 2 0 4
Reported
Activities51 46 4 0 0 4
Letters of
Recommendation32 55 7 4 0 7
Interview
Responses59 29 7 0 0 7
Table 1. Assessing Applicants’ Patient Centeredness
0
10
20
30
40
50
60
1 2 3 4 5
Nu
mb
er
of
Inte
rvie
we
r R
esp
on
ses
(N=
75
)
AC Response(1=strongly Agree; 5=Strongly Disagree)
Admissions Committee Survey Responses
PCPSS agreement
PCPSS influence
QUOTES ABOUT SCALEPCPSS was confirmatory!
-AC Member 1
I enjoy the PCPSS.-AC Member 3
I was prepared for a self-centered applicant based on my assessment and the PCPSS, but was very pleased to be wrong. Applicant was personable, well-balanced young man.
-AC Member 13
