42nd Apmc Convention Nmat Ppt

Center for

Educational

Measurement, Inc.

T h e t e s t o f e x c e l l e n c e

1

National Medical Admission Test (NMAT)

Center for

Educational

Measurement, Inc.


2

NMAT

• Objective: to upgrade the selection of applicants to medical schools

• Description: an instrument that measures several abilities deemed necessary for success in medical training and practice

Center for

Educational

Measurement, Inc.


3

NMAT

• Abilities measured:– verbal comprehension– inductive reasoning– quantitative ability– perceptual acuity– knowledge and analytical proficiency in the sciences

Center for

Educational

Measurement, Inc.


4

Test Scores

• 8 subtest scores

• APT Composite

• SA Composite

• General Performance Score (GPS)

Center for

Educational

Measurement, Inc.


5

Test Development Concerns

1. Relevance of the areas measured by the test to the curricula of premedical courses and the medical course itself

2. Comparability of test scores from different test forms across the years

Center for

Educational

Measurement, Inc.


6

Test Development Procedure

1. Content specification

2. Item writing and review

3. Pretesting and item analysis

4. Test assembly

5. Test administration

6. Score scales generation & equating

Center for

Educational

Measurement, Inc.


7

Bases for a Two-Part Test

1. PACT key aptitudes for the medical course

2. Academic requirements of medical schools

3. Possibility of test bias

Center for

Educational

Measurement, Inc.


8

Test Form Equating

• The process of making the score scales of different test forms equivalent to be able to directly compare the scores from the different test forms

Center for

Educational

Measurement, Inc.


9

Test Form Equating

• NMAT subtest standard scores are comparable across subtests and across test forms

• APT and SA composite scores are comparable with each other and across test forms

• GPSs are comparable across test forms

Center for

Educational

Measurement, Inc.


10

Test ResearchConcurrent Validity Study

Sample Multiple Correlations

APT subtest scores & GPA

SA subtest scores & GPA

College freshmen .61 .54

College seniors .36 .33

Center for

Educational

Measurement, Inc.


11

Test ResearchNMAT Highlights 1986–1997

• NMAT: Overall performance remained relatively stable across time.

• PLE: Overall performance declined across the years.

• NMAT & PLE relationship: Students who were selected on the basis of the NMAT had a better chance of passing the PLE.

Center for

Educational

Measurement, Inc.


12

Test ResearchNMAT Scores of 2006 PLE Passers• PLE passers w/ NMAT taken once (N = 1,176):

GPS Mean = 584; SD = 87

GPS Distribution on Quality Indices

GPS Range QI Range %

524–800 HA–E 76

476–523 A 14

200–475 VP–LA 10

Center for

Educational

Measurement, Inc.


13

Test ResearchNMAT Scores of 2006 PLE Passers

• PLE passers w/ NMAT retaken (N = 182):GPS on retake not significantly different from initial GPS

Center for

Educational

Measurement, Inc.


14

Future Direction

• Development of shorter test forms that produce better ability estimates and efficiently predict academic performance in the medical course

Center for

Educational

Measurement, Inc.


15

• The minimum requirement for NMAT to be useful in decision-making is an empirical demonstration of its predictive validity.

• Predictive validity refers to the efficiency of NMAT in forecasting any measure of academic performance in medical school.

Predictive Validity Study

Center for

Educational

Measurement, Inc.


16

• Technically, predictive validity of NMAT is expressed as a bivariate or multivariate correlation coefficient of NMAT scores and college grades.

• The higher this correlation coefficient, the more confidence can be placed on the NMAT as a valid predictor of academic performance.

Center for

Educational

Measurement, Inc.


17

The variables• Predictor variable

1. NMAT General Performance Score (NGPS)

• Dependent or Criterion variables

2. First year college grade point average (FGPA)based on first year academic subjects

3. Overall college grade point average (OGPA)based on ALL academic subjects

Center for

Educational

Measurement, Inc.


18

The validation samples• Medical school applicants who were accepted from SY1997-98 to

SY2002-03

School A (Luzon) 744

School B (Visayas) 205

School C (Mindanao) 232

• Three (3) medical schools A, B, and C; located in Luzon, Visayas, and Mindanao; respectively, participated in this validity study

• The final sample sizes include only students with complete data on all the three variables under study:

Center for

Educational

Measurement, Inc.


19

Data Analysis• Computation of weighted GPAs using the prescribed

number of hours per academic subject as weights

• Computation of correlation coefficients between NMAT test scores and a criterion of medical school performance

The accuracy of prediction is usually represented by the correlation coefficients.

That is, the extent to which scores on the NMAT are correlated to success in medical college is the extent to which the test fulfills its predictive function.

Center for

Educational

Measurement, Inc.


20

• Adjustment of computed correlation coefficients due to restriction of NGPS within the range above cut-off scores used by schools (which tends to severely decrease the correlation of NGPS with other variables.)

Adjusted or otherwise, the values of correlation coefficients are always interpreted within the context in which it was computed. Coefficients as low as .30 are generally considered appropriate and valid for predictive purposes (Bobko, 1995).

Center for

Educational

Measurement, Inc.


21

• Computation of the coefficient of determination, d, which indicates the proportion of variance in the FGPA or in the OGPA that is predicted by NGPS

• Derivation of prediction equation which estimates the FGPA or OGPA for a given NGPS

• Generating expectancy table which estimates the probability of a medical school applicant to attain certain FGPA or OGPA based on his/her NGPS

Center for

Educational

Measurement, Inc.


22

ResultsTABLE 2. Pearson correlation coefficients

School A School B School C Variable pair Restricted Adjusted Restricted Adjusted Restricted Adjusted

NGPS x FGPA r12 -0.375 -0.545 -0.347 -0.560 -0.380 -0.604 NGPS x OGPA r13 -0.373 -0.542 -0.346 -0.558 -0.296 -0.496

Performance in NMAT is moderately correlated with academic performance in the first year of medical education as well as with performance over three or four years of academic subjects.

Center for

Educational

Measurement, Inc.


23

Table 3 shows that around 30% of the total variation in academic performance can be explained by performance in the NMAT. In other words, the use of the NMAT GPS as predictor makes forecasting 30% to 36% better than mere random guessing.

TABLE 3. Coefficients of determination Variable pair School A School B School C

NGPS x FGPA d12 0.2966 0.3131 0.3648 NGPS x OGPA d13 0.2940 0.3112 0.2461

Center for

Educational

Measurement, Inc.


24

School A: Predicted FGPA = -0.0025 (NGPS) + 3.8076Predicted OGPA = -0.0020 (NGPS) + 3.5354

School B: Predicted FGPA = -0.0026 (NGPS) + 3.8048Predicted OGPA = -0.0021 (NGPS) + 3.6110

School C: Predicted FGPA = -0.0023 (NGPS) + 3.7591Predicted OGPA = -0.0011 (NGPS) + 2.9049

The prediction equations can be used to estimate the first year GPA and the overall academic performance of a medical student applying for admission to medical School A, B, or C.

Center for

Educational

Measurement, Inc.


25

TABLE 5. Index of predictive efficiency Predicted variable School A School B School C

FGPA 16.13% 17.12% 20.30% OGPA 15.98% 17.00% 13.17%

Table 5 shows that the prediction of academic performance in the three medical schools based on the derived prediction equations is about 13% to 20% better than prediction based on knowledge of the mean of the obtained FGPA or OGPA.

Center for

Educational

Measurement, Inc.


26

Students who obtain an NMAT GPS between 651 and 700 have a 27 percent probability of attaining a first year GPA of 2.00 or higher in School A.

TABLE 7.2. Expectancy or probability of obtaining a particular FGPA at different ranges of NGPS in School A FGPA

NGPS 1.75 & Below

1.76 -2.00

2.01 -2.25

2.26 -2.50

2.51 -2.75

2.76 -3.00

Below 501 ……… 4% 10% 34% 46% 6% 501-550 ……… 4% 19% 34% 35% 9% 551-600 1% 10% 20% 35% 27% 7% 601-650 2% 15% 29% 27% 22% 5% 651-700 1% 26% 40% 20% 9% 4% Above 700 11% 44% 22% 11% 11% ………

Center for

Educational

Measurement, Inc.


27

TABLE 7.3. Expectancy or probability of obtaining a particular OGPA at different ranges of NGPS in School A FGPA

NGPS 1.75 & Below

1.76 -2.00

2.01 -2.25

2.26 -2.50

2.51 -2.75

2.76 -3.00

Below 501 2% 4% 36% 54% 4% 501-550 1% 16% 38% 43% 2% 551-600 1% 5% 17% 46% 28% 3% 601-650 2% 8% 26% 36% 26% 2% 651-700 12% 42% 35% 11% Above 700 5% 35% 50% 30% 15%

Students who obtain an NMAT GPS between 651 and 700 have also 12 percent probability of attaining an overall GPA of 2.00 or higher in School A.

Center for

Educational

Measurement, Inc.


28

NMAT subtests that are predictive of FGPA and OGPA:

School A: FGPA PHYS, BIOL, CHEM, SSCI, VCOMOGPA CHEM, SSCI, PHYS, QUAN

School B: FGPA BIOL, PHYS, VCOMOGPA BIOL, VCOM, PHYS

School C: FGPA SSCI, CHEM, PHYSOGPA SSCI, CHEM

Center for

Educational

Measurement, Inc.


29

Summary and Conclusion

• The use of the NMAT GPS as predictor makes forecasting 30% to 36% better than mere random guessing

• Predicting future academic performance based on NMAT GPS and using the derived prediction equations is about 13% to 20% more accurate than predicting based on knowledge of the mean of the obtained FGPA or OGPA of the validation sample used.

Center for

Educational

Measurement, Inc.


30

Recommendations• It should be noted that test scores are always subject to

measurement errors. Hence, it is advised that other measures that assess other student characteristics (e.g., socio-demographics, motivation levels, etc.) may be considered before coming up with a final decision whether to accept or reject a candidate for admission to School A, B, or C.

• Test components should be reviewed and new test specifications should be considered to reflect subtests that highly correlate with academic performance

Center for

Educational

Measurement, Inc.


31

• Explore the option of restoring a national cut-off score for quality assurance

• Address the need for more research studies on the implications of NMAT on quality assurance. This will entail full cooperation among medical schools and other agencies in providing the necessary database.

42nd Apmc Convention Nmat Ppt

Documents

Transcript of 42nd Apmc Convention Nmat Ppt