DOCUMENT RESUME ED 094 734 Dansereau, Donald F.; And Others · media presentation modalities...

DOCUMENT RESUME

ED 094 734 IR 000 932

AUTHOR Dansereau, Donald F.; And OthersTITLE Factors Relating to the Development of Optimal

Instructional Information Sequences.AFHRL-TR-73-51(II). Final Report.

INSTITUTION Air Force Human Resources Lab., Lowry AFB, Colo.Technical Training Div.; Texas Christian Univ., FortWorth. Inst. for the Study of Cognitive Systems.

REPORT NO AFHRL-TR-73-51(II)PUB DATE Jul 74NOTE 41p.; Final Report for Period April 1, 1973-December

31, 1973

EDRS PRICE MF-$0.75 HC-$1.85 PLUS POSTAGEDESCRIPTORS College Students; *Computer Assisted InstrI7ction;

Concept Teaching; Educational Research; IndividualDifferences; *Instructional Design; InstructionalInnovation; *Intermode Differences; MultidimensionalScaling; Pictorial Stimuli; Training

IDENTIFIERS Aptitude Treatment Interaction; *INSCAL

ABSTRACTINSCAL multidimensional scaling, an objective

methodology for formulating effective technical information sequencesfor computer-assisted instruction, was shown to be of value withrespect to defining the information complexity of technical material,developing sequences of key concepts within technical material, andproviding an index of expert inter-rater consensus. When the INSCALmeasure is obtained after student exposure to the material, INSCALprovides an indication of the correspondence between experts' andstudents' understanding of concept interrelationships. Major findingsfrom the final phase indicated that: (1) alternative sequences ofinstructional material influenced student performance; (2) pictorialtechnical information sequences resulted in small performancedifferences when compared to verbal print sequences; (3) technica?information difficulty debilitated student performance; (4) studentreading aptitude was significantly related to student performanceunder both pictorial and verbal print presentation modalities; and(5) instruction sequences did not interact with student aptitude..(Author)

MR FORCE

H

UMAN

ii

U.S. DEPARTMENT OF HEALTH,EDUCATION & WELFARENATIONAL INSTITUTE OF

EDUCATIONTHIS DOCUMENT HA', BEEN REPRODUCE° EXACTLY AS PECEIVED FROMTHE PERSON ON ORG.,NIZATION ORIGINATING IT POINTS DI, VIEN OR OPINIONSSTATED DO NOT NECLSSARILY REPRESENT OFFICIAL NA,ION:.;. N.TiTUTE OFEDUCATION POiI TION OR POLICY

RE

S0UR

AFHRLTR73-51 (III

FACTORS RELATING TO THE DEVELOPMENT OFOPTIMAL INSTRUCTIONAL INFORMATION SEQUENCES

BEST COPY AVAILABLE

By

Donald F. DansereauSelby H. Evans

Tomme A. ActkinsonGary L. Long

Institute for the Study of Cognitive SystemsTexas Christian UniversityFort Worth, Texas 76129

TECHNICAL TRAINING DIVISIONLowry Air Force Base, Colorado 80230

July 1974Final Report for Period 1 April 1973 31 December 1973

Approved for public release; distribution unlimited.

BEST COPY AVAILABLE

LABORATORY

AIR FORCE SYSTEMS COMMANDBROOKS AIR CORCE BASE,TEXAS 78235

NOTICE

When US Government drawings, specifications, or other data are usedfor any purpose other than a definitely related Governmentprocurement operation, the Government thereby incurs noresponsibility nor any obligation whatsoever, and the fact that theGovernment may have formulated, furnished, or in any way suppliedthe said drawings, specifications, or other data is not to be regarded byimplication or otherwise, as in any manner licensing the holder or anyother person or corporation, or conveying any rights or permission tomanufacture, use, or sell any patented invention that may in any waybe related thereto.

Thif, final report was submitted by the Institute for the Study ofCognitive Systems, Texas Christian University, Fort Worth, Texas76129, under contract F41609-73-C-0023, project 1121, with theTechnical Training Division, Air Force Human Resources Laboratory(AFSC), Lowry Air Force Base, Colorado 80230. Dr. Gerard M.Deignan, Technical Training Division, was the contract monitor.

This report has been reviewed and cleared for open publication and/orpublic release by the appropriate Office of Information (01) inaccordance with AFR 190-17 and DoDD 5230.9. There is no objectionto unlimited distribution of this report to the public at large, or byDDC to the National Technical Information Service (NTIS).

This technical report has been reviewed and is approved.

MARTY R. ROCKWAY, Technical DirectorTechnical Training Division

Approved for publication.

HAROLD E. FISCHER, Colonel, USAFCommander

UnclassifiedSECURITY CLASSIFICATION OF THIS PAGE (When Data Entered)

REPORT DOCUMENTATION PAGEREAD INSTRUCTIONS

BEFORE COMPLETING FORMI. REPORT NUMBER

AFHRL-TR-73.51 (II)

2. GOVT ACCESSION NO 3 RECIPIENT'S CATALOG NUMBER

4. TITLE (and Subtitle)FACTORS RJILATING TO THE DEVELOPMENT OFOPTIMAL INSTRUCTIONAL INFORMATION SEQUENCES

5. TYPE OF REPORT 8 PERIOD COVERED

Final 1 Apr to 31 Dec 736. PERFORMING ORG. REPORT NUMBER

7. AUTHOR(s)

Donald F. Dansereau Gary L. LongSelby H. EvansTonne A. Actkinson

8. CONTRACT OR GRANT NUMBER(s)

F41609.73-C-0023

9. PERFORMING ORGANIZATION NAME AND ADDRESS

Institute for the Study of Cognitive SystemsTexas Christian UniversityFort Worth, Texas 76129

10. PROGRAM ELEMENT, PROJECT, TASKAREA a WORK UNIT NUMBERS

11210119

II. CONTROLLING OFFICE NAME AND ADDRESSHQ Air Force Human Resources Laboratory (AFSC)Brooks Air Force Base, Texas 78235

12. REPORT DATE

July 197413. NUMBER OF PAGES

4014. MONITORING AGENCY NAME 8 ADDRESS(11 different from Controlling Office)

Technical Training DivisionAir Force Human Resources LaboratoryLowry Air Force Base, Colorado 80230

15. SECURITY CLASS. (of this report)

Unclassified15n, DECLASSIFICATION /DOWNGRADING

SCHEDULE

16. DISTRIBUTION STATEMENT (of this Report)

Approved for public release; distribution unlimited.

17. DISTRIBUTION STATEMENT (of the abstract entered In Block 20, If different from Report)

18. SUPPLEMENTARY NOTES

r19. KEY WORDS (Continue on reverse side if necessary and Identify by block number)

instructional sequences concept acquisition instructional measurementinformation organization computer based trainingindividual differences multidimensional criteriainscal pictorial informationinformation complexity instructional methods

20. ABSTRACT (Continue on reveree aide If necessary and Identify by block number)

Inscal multidimensional scaling was shown to be of value with respect to (1) defining the informationcomplexity of technical material, (2) developing sequences of key concepts within technical material, and (3)providing an index of expert inter-rater consensus. When the Inscal measure is obtained after student exposure to thematerial, Inscal provides an indication of the correspondence between experts' understanding of conceptinterrelationships and students' understanding of concept interrelationships. Major findings from the final phaseindicated (1) alternative sequences of instructional material influenced student performance, (2) pictorial technicalinformation sequences resulted in small performance differences when compared to verbal print sequences, (3)technical information difficulty debililated student performance, (4) student reading aptitude was significantly

DDFORM

1473 EDITION OF 1 NOV 65 IS OBSOLETE1 JAN 73 UnclassifiedSECURITY CLASSIFICATION OF THIS PAGE (When Date Enter

UnclassifiedSECURITY CLASSIFICATION OF THIS PAGE(When Data Entered)

Block 20. (continued)

related to student performance under both pictorial and verbal print presentation modalities, and (5) instructionalsequences did not interact with student aptitude.

UnclassifiedSECURITY CLASSIFICATION OF THIS PAGE(When Data Entered)

SUMMARYBEST COPY AVAILABLE

Background

To be cost-effective, computer-based instruction must maximizestudent attainment of training objectives and offset computer coststhrough minimization of student training time to achieve each oujec-tive. The question of how complex technical information is bestpresented, in what form, and in what sequence, impact greatly uponthe optimization of computer efficiencies for large numbers of students.Within computer based instructional systems, the effects of alternativeinstructional sequences upon student time to attain instructional objec-tives is important, but largely unknown. Traditionally, instructionalsequences are based upon intuitive versus systematic scientific approaches.To expedite student progress, identification and validation of performancefacilitating and debilitatng instructional sequences must be determinedif training time efficiencies are to be realized. Promising scientificmethods are now available which allow complexity of technical informationto be objectively measured, and subsequently modified to produce infor-mation content and sequences effective for students who vary in charac-teristics, e.g., aptitude level. A multivariat6 study was thus requiredto determine effects and interralationships among alternative instructio-nal sequences, technical information difficulty levels, media presentationmodalities and student aptitude levels upon multiple measures of perfor-mance.

Objectives were: to investigate the utility of an objective method-ology (Inscal) for formulating effective technical information sequences;to test effectiveness of alternative instructional sequences in verbaland pictorial presentation modes; to determine if the effectiveness ofinstructional sequences varies as a function of the difficulty level oftechnical material; to determine if some instructional sequences aremore performance-facilitating for low-aptitude students whereas othersequences are more effective for high-aptitude students; to determineeffects of verbal print versus pictorial information sequences upon theperformance levels of high- and low-aptitude students.

Approach

Technical materials (Structure of Matter and Oscilloscope Operations)were selected and determined to constitute significantly differentdifficulty levels. Inscal multidimensional scaling methodology wasapplied to 50 minute course segments by means of concept similarityratings obtained from technical experts within the academic and military'communities. Computer algorithms were developed to provide two differentinstructional sequences for each instructional package. Also, pictorialand verbal print material versions of each instructional package weregenerated. Measures of student performance for each instructionalpackage included: (1) verbal print multiple choice tests, (2) pictorialmultiple choice tests developed to correspond to verbal multiple choicetests, (3) Concept Cloze, as a measure of student comprehension oftechnical material covered, and (4) Student/Expert Understanding of

1

Concept Interrelationships, as a post-instructional measure of the degreeto which student understanding of technical concepts corresponded to theexperts' understanding of the interrelationships among technical concepts.High and low student aptitude classifications were assessed by the DeltaReading Vocabulary measure.

Findings and Conclusions

Instructional sequence alternatives developed from the Inscal methodresulted in statistically significant but otherwise small student perfor-mance differences among the instructional sequences. Thus, how technicalinformation is sequenced does contribute to subsequent student performance;however, the magnitude of these differences under the conditions of thepresent study was not great (less than 5% in student performance). Thoughmedia presentation modalities (pictorial versus verbal print) were foundto be statistically significant, student performance differences werefound to be marginal. Thus, the findings from the present study indicateno practical performance gains attributable to pictorial presentation asopposed to verbal print presentation for low-aptitude students as reportedby other investigators.

Instructional sequencing alternatives were not found to interact withhigh or low student aptitude, media mode (pictorica versus print)or diffi-culty level of technical material. Hence, instructional sequences similarto those employed in the present study may not unduly influence the per-formance of students even if those students differ in aptitude.

Results of a four-way analysis of variance indicated that high scoreson the Delta Reading Vocabulary aptitude measure performed substantiallybetter on all dependent variables to include the pictorial multiple choicetest than did low-aptitude students. Within the low-aptitude gtoup, noevidence was found to support the contention that pictorial multiplechoice tests result in better performance than verbal tests for low-aptitude students. Hence, it is unlikely that pictorial tests will sub-stantially improve low-aptitude student performance.

In all conditions, males outperformed females on all performancemeasures. However, performance differences were primarily attributableto the difficulty factor inherent in the oscilloscope as opposed to thestructure of matter technical material. Thus, if female technical traineesare to be increasingly assigned to courses where unfamiliar and difficulttechnical concepts are taught, student aptitude and motivational factors,e.g., interest, should be given greater consideration. when technicalmaterials are developed or revised.

In conclusion, the Inscal method of developing both improved measuresof technical information complexity and performance-facilitating instruc-tional sequences, also offers several other advantages. This objectivemethod can be applied to new technical material at the time experts writeit. Further, alternative instructional sequences can be tailored to theneeds of subgroups of students in an individualized manner. Other benefitsare discussed in the Conclusions section of this report.

2

PREFACE

The contributions of knowledge and tire made by numerousindividuals in the local communities, academic and militaryenvirons impelled the development and successful completionof the Phase II study. The leadership of Texas ChristianUniversity's Colonel Bearden (USA PCTC) and Lt. Colonel Reed(USAF ROTC) and University of Texas at Arlington's Lt. ColonelWeems (USAF Parc) in the recruitment of the ',ROTC studentsubjects was of vital significance. Cooperation of the HillCounty Junior College's (Cleburne, Texas) administration inencouraging student participation was deeply appreciated. Inaddition, the excellent combined coordination and cooperationof the Texas Christian Psychology Department faculty inproviding the general psychology student subjects played animportant role in completing the Phase II experimentation.

Noteworthy contributions were made by Dr. Gerald Deignan,Technical Training Division, Lowry AFB, Colorado by hisconstant involvement, deep concern and ready assistance inall areas of the research.

3

TABLE OF CONTENTS

Page

PHASE II: THE MAIN ENPER1MENT 6

MethodExperimental DesignFigure 1 - EXperimental Design

SubjectsStimulus MaterialsDependent MeasuresIndividual Difference MeasureProcedure

ResultsScoring of the Dependent and Individual DifferenceMeasures

The Four Way Analyses of VarianceTable 1 - Four Way Analyses of Variance:

Group Proportion Means for PackageDifficulty Factor

Table 2 - Group Means for the Main Effect ofPresentation Mode on All Four DependentMeasures

Table 3 - Group Means for the Main Effect ofInstructional Sequence on All FourDependent Measures

Table 4 - Four Way Analyses of Variance: GroupMeans for the Interaction BetweenDelta Performance and Package Difficulty

Intercorrelations Among the Dependent and IndividualDifference MeasuresTable 5 - Average Intercorrelation Matrices

for the Two Levels of the PackageDifficulty Factor

Table 6 - Average Intercorrelation Matrices forthe Two Levels of the Mode ofPresentation Factor

Table 7 - Average Intercorrelation Matrices forthe Three Levels of the InstructionalSequence Factor

4

7

8

8

8

9

10

1111

12

12

13

14

15

16

17

18

19

20

21

Page

Two Way Analyses of Variance (Male/Femalesand Package Difficulty) 22Table 8 - Two Way Analyses of Variance:

Group Means for the SignificantInteraction Between the Male/Female Factor and PackageDifficulty 23

Two Way Analyses of Variance (Male ROTC/Male Non-ROTC) 23Table 9 - Group Means for Male ROTC/Male

Non-ROTC Factor on EachDependent Measure 24

Discussion 25The Impact of Instructional Sequencing 25The Effect of Presentation Mode 27The Role of Individual Differences 28Effectiveness of the Dependent Measures 29

Summary and Conclusions 30

References 33

Annex A - Means and Standard Deviations forEach Experimental Condition on AllDependent Measures

5

34

PHASE II: THE MAIN EXPERIMENT

One of the major conclusions drawn from an analysis ofprevious investigations into the effects of instructionalsequencing upon student performance is the inconsistencyof the findings (AFERLTR-73-5l (I), Factors Related toDeVeloping Optimal Instructional Sequences: Phase I). Tosummarize briefly, some studies have found that performanceis apparently influenced by the partiCular sequence inwhich instructional material is presented, while othershave not. The variety of procedures, stimuli, and responsemeasures used in prior experiments has made it difficultto determine the conditions under which instructionalsequencing is an important variable. However, twocritical factors influencing the conflicting outcomes ofthese experiments have been identified:

(1) lack of systematic procedures for developinginformational sequences, and

(2) minimal attention to individual differences ininformation processing styles and capacity.

The primary purpose of the present experiment was toincorporate solutions to these two difficulties in anassessment of the effects of instructional sequencing uponperformance. Additionally, two variables thought to inter-act with the effects of sequence and with individualdifferences have been included in an attempt to delimitconditions which attenuate or enhance sequence effects.These two variables, difficultly of the instructionalmaterial and mode of presentation (verbal versus pictorial/graphic), are also of general importance in the design ofinstructional procedures.

Specifically, the main experiment has been designed:

(1) to investigate the utility of multidimensionalscaling as a method for describing the informationstructure of Air Force instructional material and to

6

investigate its potential as a basis for forming effectiveinstructional sequences. As a result of the pilot studiesdiscussed in AFHRIJ-TR-73-51 (I), the previously cited PhaseI report, two instructional sequences, algorithmicallyformed from multidimensional scaling solutions of similarityjudgments by "experts" in the content areas, were selectedfor assessment in the main experiment along with theoriginal Air Force sequences.

(2) to determine if interaction effects exist betweeninformation sequence requirements and a measure of intel-lectual ability. The principal measure of ability employedis the Delta Reading Vocabulary Test.

(3) to determine the effect of two modes of presenta-tion, verbal and pictorial/graphic, on test performance,especially in interaction with informational sequences andindividual differences.

(4) to assess the interaction of instructionalmaterial difficulty with other independent variables. TwoAir Force instructional packages varying in difficultyhave been selected for inclusion on the basis of pilotwork reported in AFEIL-TR-73-51 (I).

(5) to investigate the utility of two new performancemeasures in the assessment of the impact of instructionalmanipulations. A modified version of the "doze" procedureand a similarity judgment methodology were developed foruse with the Air Force material. Discussion of thesedependent measures, along with standard multiple choicetests, shall be elaborated upon in the subsequent section.

Due to the conflicting nature of the results ofprevious experiments on sequencing and due to the explora-tory nature of this study, no specific hypotheses havebeen formulated.

Method

This section will be divided into subsections dealingwith: experimental design, subjects, stimulus mafPrials,dependent measures, individual difference measures, andprocedure.

7

Experimental Design. A fully crossed, three factor,between-subjects design was employed in this experiment(see Figure 1). The three factors were: instructionalsequence (three levels designated original, proximal, andnon-proximal), mode of presentation (two levels designatedverbal and pictorial/graphic), and instructional packagedifficulty (two levels designated Oscilloscope Operationand Characteristics of Matter) . Fifteen subjects wereincluried in each of the twelve cells of the design. Thisnumber of subjects was considered sufficient to allow forthe investigation of the effects of individual differencesin intellectual ability.

Mode ofPresentation

r

PackageDifficulty Instructional Sequence

Proximal Non-Proximal

Verbal Oscilloscope. .... _.

OriginalAir Force15 Ss 15 Ss 15 Ss

Characteristicsof Matter

15 Ss 15 Ss 15 Ss

Pictorial/Graphic

Oscilloscope 15 Ss 15 Ss 15 Ss


15 Ss 15 Ss 15 Ss

FIGURE 1

Experimental Design

Subjects. One hundred and eighty college. studentsserved as subjects in this experiment. Due to lack ofa sufficiently large homogeneous body of available subjects,it was necessary to recruit individuals from a variety ofsources. The subject mix can be characterized as follows:50 males from the Air Force and Army ROTC units at TexasChristian University and the University of. Texas atArlington, 72 males from general psychology classes atTexas Christian University and Hill County Junior College,and 58 females from general psychology classes at TexasChristian University. General Psychology students were givenexperimental credit as partial fulfillment of their courserequirements, while ROPC students were paid at a rate of $2.00

per hour. These subjects were distributed as systematically

8

as possible across the experimental conditions in order toavoid biasing the treatment effects and in order to set upa basis for exploring the effects of individual differ-ences on the dependent measures. Since the military mayexperience an influx of females in the future, male-femaledifferences in this experiment will be of particularinterest. In each cell of this experiment there were fourto six females and nine to eleven males.

Stimulus Materials. Based on the pilot work discussedin AFBRL-TR-73-51 (I), two sets of Air Force instructionalmaterial, varying in difficulty of acquisition, wereselected for inclusion in this experiment. These two setswere:

CO Characteristics of Matter. (pages 1-22 of thedocument labeled Precision Measuring Equipment Specialist,D C Circuit Analyses Block II),

(2) Oscilloscope Operation. (pages 1-18 of theMaintenance Electronics Operation). The results of PilotStudy 2 indicated that the Oscilloscope Operation packageled to significantly poorer performance on the dependentmeasures than did the Characteristics of Matter package.

The original Air Force instructional packages werein primarily verbal form. To investigate the effects ofpictorial /graphic as well as verbal presentation, thesepackages were converted into primarily pictorial represen-tation. Approximately seventeen key concepts werederived from each of the packages, and a concept byconcept matching of the information was maintained betweenthe verbal and pictorial/graphic modes. These packageswere prepared in booklet form containing approximately oneconcept per page. Examples of these stimulus packages werepresented in Annex A of the AFHRL-TR-73-51 (I).

Three instructional sequences were used in the presentexperiment: (1) proximal (2) non-proximal and (3) theoriginal Air Force sequence. Sequence development wasaccomplished in two stages. First, information structureswere derived for each of the two instructional packages by

9

the multidimensional scaling of similarity judgments madeby six "experts" to all possible pairs of the seventeenkey concepts. The coordinates of the concepts located inthese derived information structures were then submittedto two computer based algorithmic procedures for producinginstructional sequences. One algorithm produced sequencesin which concepts highly related by the experts werelocated contiguously in the sequence (the proximal sequence).The second algorithm produced sequences in which concepts nothighly related were located contiguously (the non-proximalsequence). Examples of the output of the algorithms arepresented in Annex B of the Phase I Report.

Dependent Measures. Four dependent measures were usedin this experiment, examples of which are presented in AnnexesD and E of the AFBRL-TR-73 -51 (I).

(1) Similarity or relatedness judgments (on a scalefrom 1 to 100) between all possible pairs of the keyconcepts. These judgments were compared with those producedby the six "experts" for each package.

(2). Sets of verbal multiple choice questions used bythe Air Force in the assessment of student performance onthe instructional packages.

(3) Pictorial/graphic versions of the sets of verbalmultiple choice questions. These were created to providea measure compatible to the pictorial/graphic presentationformat.

(4) A concept cloze comprehension measure in whichall occurrences of key concepts in the original Air Forcetext were replaced by letters used as a substitution code.The subjects' task was to match the code letters with theappropriate concepts presented previously di=ing theinstructional stage.

The standard cloze technique was not included in thisexperiment due to time constraints on the experimentalsessions and due to the high intercorrelation (.76) foundbetween the concept cloze and the standard cloze in PilotStudy 1.

10

Individual Difference Measure. All subjects were giventhe Delta Beading Vocabulary Test at the time of the experi-ment. This test serves as our primary measure of intellectualability.

Procedure. The experiment was run in eight, two-hourexperimental sessions with all twelve cells of the designbeing represented in each session. For each session, twoexperimenters and two separate classrooms were used. Becausethe length of the two instructional packages varied slightly,the subjects in the cells utilizing the Characteristics ofMatter material were run in a separate room from thosesubjects receiving the Oscilloscope Operation package. Uponarrival to the experiment, subjects were assigned to oneof the two rooms at random and then assigned to one of thesix conditions existing under each level of the instructionalpackage factor. Again, these latter assignments were donerandomly within the constraints of the distributional require-ments (see the discussion of the experimental subjectspresented earlier).

Following assignments to conditions, the subjects weregiven a brief description of their task. Each subject thenreceived a stimulus booklet which contained the instructionalinformation in either verbal or pictorial /graphic form, andin one of the three sequences (original, proximal, or non-proximal) depending on the experimental condition assigned.The experimenter then paced the subjects through thebooklets at a rate of 45 seconds per page (this presentationrate was selected on the basis of Pilot Study 1). If asubject finished studying a page before time was called hewas instructed to think over previous material, but notto look elsewhere in his booklet.

Following the completion of the stimulus booklet, eachsubject was given the four dependent measures in an orderwhich would minimize the transfer of content-specificinformation from one measure to the next. This order was:similarity judgments, pictorial multiple choice, verbalmultiple choice, and concept cloze. Instructions for eachof the measures were printed on the tests booklets. Eachsubject was allowed to proceed through these measures athis own rate.

11

After completing all of the response measures, eachsubject was given the Delta Reading Vocabulary Test.Completion of this test concluded the experimental session.

Results

The presentation of results will be subdivided intodiscussions of the following facets of the data analysis:

(1) Scoring of the dependent and individual differencemeasures,

(2) four way analyses of variance,

(3) interoorrelations among the dependent and individualdifference measures,

(4) two way analyses of variance using male/female andtype of instructional package as factors, and

(5) two way analyses of variance using ROC/non-RDTCand type of instructional package as factors.

Scoring of the Dependent and Individual DifferenceMeasures. Three of the dependent measures, the pictorialmultiple choice, the verbal multiple choice, and theconcept cloze were scored as proportion correct in orderto form a basis for comparison across conditions whichrequired a slightly different total number of responses.The Delta Reading Vocabulary test was also scored asproportion correct.

The similarity (or relatedness) judgments for eachsubject were correlated with the average judgments for the"experts" on the appropriate instructional package. Theseindividual correlations then served as measures of thecorrespondence of the subject and expert views of the rela-tionships between the key concepts. Presumably, the morea subject learned from the instructional material, thegreater would be his correlation with the average "expert."This measure will be referred to in the remainOer of thereport as S/E Correlation.

12

The Four nIAnalytes'of Veriance. A four way, between-subjects, fixed effect analysis of variance was performedon each of the four dependent variables. The factors includedin these analyses were: (a) package difficulty (Characteristicsof Matter versus Oscilloscope Operation), (b) mode ofpresentation (verbal versus pictorial), (c) instructionalsequence (original versus proximal ver is non-proximal), and(d) Delta Reading Vocabulary performance (high versus low).Levels of this last factor were formed by assigning the sevenbest performers on the Delta within a cell to the high groupand the seven worst to the low group. The individual withmedian performance on the Delta was excluded fram the analysis.Inclusion of this Delta Vocabulary factor in the analysisserved to potentially reduce the amount of variance enteredin the error term, thus improving the per of the analysis.

Rather than treat each of the analyses separately, theresults for each factor on each dependent measure are presentedcontiguously. The means and standard deviations for each cellon each dependent measure are presented in the Annex A of thisreport.

The package difficulty factor led to significant resultson all four dependent measures: pictorial multiple choice,F(1,144 d.f.) = 110.4, p< .001; verbal multiple choice,F(1,144 d.f.) = 88.0, E)<.001; concept cloze, F(1,144 d.f.) =102.5, p <:.001; S/E correlations, F(1,144 d.f.) = 58.0,p .001. The mean performance on the two packages foreach dependent measure are presented in Table 1. Inspectionof the means reveals that for all the dependent measures,except the S/E correlation, the Characteristics of Matterpackage led to better performance than did the Oscilloscopepackage. These findings are congruent with the results ofour pilot work. The S/E correlation measure, however,reverses the above findings, with the Oscilloscope packageleading to significantly larger average correlations thanthe Characteristics of Matter package.

13

TABLE 1

Four Way Analyses of Variance: Group Proportion Means for PackageDifficulty Factor

(proportion correct except where indicated)

Dependent MeasuresPackage Difficulty


OscilloscopeOperation

Pictorial MultipleChoice*

' .778 .548

Verbal MultipleChoice*

.800 .589

Concept Cloze* .845 .549

Subject/ExpertCorrelations* .130

(average correlation)

.266

(average correlation)

N = 84 Ss per condition* p< .001

This last result is probably due to the greater richnessof relationships among the Characteristics of Matter concepts.These are basic physics concepts and many of the actualrelationships were not explicit in the instructional material.It is possible that the "experts", having had more physicstraining, were basing their judgments on relationships notdirectly available in the instructional material. Also, sincemost of the subjects had some prior training in elementaryphysics, their judgments may have been biased by prior know-ledge. On the other hand, the Oscilloscope package appearedto explicitly contain the majority of possible relationshipsbetween concepts, thus putting the subjects and "experts" ona common ground. With this package a more valid assessment ofsubject-"expert" correspondence seemed to be achieved.Differences between packages on the S/E Correlation Measurewere probably spuriously generated by differences in relationalrichness rather than package difficulty.

14

The mode of presentation factor was found to be marginallysignificant with the pictorial multiple choice measure (F(1,144d.f.) = 3.5, p 4:.075) and nonsignificant with the three otherdependent measures. With pictorial multiple choice, the pictorialpresentation mode led to better performance than verbal presenta-tion (see Table 2).

TABLE 2

Group Means for the Main Effect of Presentation Mode on All FourDependent Measures


Dependent Measures Mode of PresentationPictorial/Graphic Verbal

Pictorial Multiple Choice* .684 .642

Verbal Multiple Choice .705 .686

Concept Cloze .715 .700

7E Correlations(Average Correlation)

.211 .188

N = 84 Ss per cell* p <;.075

Inspection of the means in Table 2 indicates that, althoughnot significant, the level of performance with the Pictorial/Graphic node of presentation was higher on each of the remainingdependent measures. Since the Verbal Multiple Choice, ConceptCloze, and S/E Correlation measures are in a verbal format,perhaps the failure to achieve statistical signific:ance withthese measures is due to difficulties encountered by subjectsin translating the Pictorial/Graphic information into a formatcampatible with the verbally based dependent measures. Thisnotion will be explored further in the Discussion section ofthis report.

The instructional sequence factor was significant with theVerbal Multiple Choice measure (E(2,144 d.f.) = 3.6, p4( .05)and nonsignificant with the three remaining dependent variables.The group means are presented in Table 3. As can be seen, exceptfor the S/E Correlation measure, the Proximal sequence generallyresults in slightly better performance than the other two sequences.

15

A Scheffe' post hoc analysis on the Verbal Multiple Choice measurerevealed that the Proximal was significantly different from theNon-proximal at the .05 level and that other differences betweenmeans were nonsignificant.

TABLE 3

Group Means for the Main Effect of Instructional Sequence onAll Four Dependent Measures


Dependent MeasuresSequence

Air ForceOriginal Proximal Non-Proximal

Pictorial Multiple Choice .663 .670 .656

Verbal Multiple Choice* .698 .730 .658

Concept Cloze .685 .712 .687

S/E Correlations(Average Correlation)

.214 .196 .188

N = 56 Ss per cell* p < .05

Potential reasons for the lack of statistical significancewith the Concept Cloze and S/E Correlation dependent measureson the instructional sequence factor can be cited. First, thepreviously discussed difficulties with the S/E Correlations andthe Characteristics of Matter package would serve to reducesensitivity to instructional sequence. Second, inspection ofthe Concept Cloze test showed than in scale conditions thesensitivity of this measure was being attenuated by "ceilingeffects"; a number of subjects were scoring 100% on this testwith the Characteristics of Matter instructional material. Thefailure of the Pictorial Multiple Choice measure to reflectsignificance, however, does not seem to be attributable todifficulties with the measure. Obviously, further empirical

exploration is necessary.

16

Significant differences due to the Delta Reading Vocabularyperformance factor were found for each of the four dependentmeasures (Pictorial Multiple Choice, F(1,144 d.f.) = 38.3,13<( .001; Verbal Multiple Choice, F(1,144 d.f.) = 31.0, p< .001;Concept Cloze, F(1,144 d.f.) = 52.8, p < .001; S/E Correlations,F(1,144 d.f.) = 15.0, p < .001). Inspection of the performancemeans revealed that with all dependent variables the highscoring Delta subjects performed better on the dependent measuresthan the low scoring Delta subjects.

The only interaction effect reaching significance in thefour way analyses was package difficulty by Delta performance.This effect was significant with the S/E Correlation measure(F (1,144 d.f.) = 8.8, p .4.005) and nonsignificant with thethree other dependent variables. The means representing theinteractions are presented in Table 4. As can be seen thesignificant difference between high and low Delta on theS/E Correlation measures is primarily due to performance onthe Oscilloscope Operation package. This finding adds furtherevidence to the previously discussed notion that the S/ECorrelations are only valid with the Oscilloscope material.

TABLE 4

Four Way Analyses of Variance: Group Means for the InteractionBetween Delta Performance and Package Difficulty

,High Delta PerformersLow Delta Performers

N = 42 Ss per Cell

Pictorial Multiple Choice(Proportion Correct)

Characteristics of !OscilloscopeMatter 1 Operation.858 .604

.698 .493

High Delta PerformersLow Delta Performers

N = 42 Ss per Cell

Verbal Multiple Choice(Proportion Correct)

OscilloscopeOperation

881 .633

Characteristics ofMatter

.718 .549

17

TABLE 4 (Cont..)

Concept Cloze(Proportion Correct)

OscilloscopeOperation_

.671

.418

Characteristics ofMatter


. 934

. 756

N = 42 Ss per Cell


S/E Correlation(Avera e Correlation)*

Characteristics of- OscilloscopeMatter Operation.139 .328.123 .206

N = 42 Ss per Cell

* P <Z.005

Intercorrelations Among the Dependent and IndividualDifference Measures. The interoorrelations among performanceson the four dependent measures and the Delta Reading Vocabularytest were calculated for each of the twelve cells of the threefactor design. Crude averages of these correlations were thencalculated to produce intercorrelation matrices for each levelof each factor.

The average intercorrelation matrices for the two instruc-tional packages are presented in Table 5. Inspection of thesematrices reveals a number of things. First, the correlationsof the S/E Correlation measure with the other dependent variablesare negligible for the Characteristics of Matter package butmoderate for the Oscilloscope material, again supporting theration of differential sensitivity of this measure across packages.Second, the intercorrelations among the other three dependentmeasures are higher with the Characteristics of Matter packagethan with the Oscilloscope materiel. The greater difficulty ofthe Oscilloscope package may have led to a larger degree of randomresponding, thus attenuating the correlations. Finally, with theexception of the S/E Correlation measure, the correlations betweenthe Delta and the other three dependent variables range from .35to .60, with no large, consistent differences between packages.

18

TABLE 5

Average Intercorrelation Matrices for the Two Levels of thePackage Difficulty Factor

-7AkACTERISTICS OF MATTER

Pictorial Verbal Conceit S E Corr. Delta

Pictorial Multiple.0 of

Verbal MultipleChoice

1.0

.84** 1.0

Concept Cloze .74** .69** 1.0

.u.ject ExpertCorrelations - .01 .06 .06 1.0

Delta ReadingVocabulary .55** .60** .49** .07 1.0

N = 84. Ss** P < .01

OSCILLOSCOPE OPERATION

Pictorial Verbal! Concept S/E Corr. Delta

Pictorial MultipleChoice

1.0

Verbal MultipleChoice

.35** 1;0

Concept Cloze .60** .36** 1.0

Subject/ExpertCorrelations .30** .35** .65** 1.0

Delta ReadingVocabulary .35** .36** .54** .35k* 1.0

N = 84 Ss** p .< .01

The average intercorrelation matrices for the levels of thenode of presentation factor are presented in Table 6. The maindiscrepancy of interest between these two matrices involves correla-tions with the Delta Reading Vocabulary. (The matrices also showdifferences in the S/E Correlation measure, but the difficulties

19

with this measure have already been discussed.) The correlationsbetween the dependent variables and the Delta are substantiallylarger with verbal presentation than with pictorial. Thisdifference makes sense in that a measure of verbal ability, suchas the Delta, should be more predictive in a verbal test than ina pictorial. Presumably, if we had included a measure ofspatial/pictorial ability, the reverse finding would have occurred.

TABLE 6

Average Intercorrelation Matrices for the Two Levels of the Modeof PresentationFactor

PICTORIAL PRESENTATION

Pictorial Verbal Concept. S/E Corr, DeltaPictorial Multiple Choice 1.0Verbal Multiple Choice .61** 1.0Concept Cloze .73** .60** 1.0Subject /Expert Corr.. .26** .32** .49** 1.0Delta Reading Vocabulary .38** .43** .46** .25* 1.0

N = 84 Ss* p <.05** p < .01

VERBAL PRESENTATION

Pictorial' Verbal Concept S/E Corr. Delta..

Pictorial Multiple Choice 1.0Verbal Multiple Choice .58** 1.0Concept Cloze .62** .46** 1.0Subject /Expert Corr. .02 .09 .22* 1.0'Delta Reading Vocabulary .51** .53** .57** .17* 1.0

.

N = 84 Ssp < .05

** p < .01

The average intercorrelation matrices for the instructionalsequence factor are presented in Table 7. These matrices appear tobe very similar to one another with no compelling discrepancies.

20

TABLE 7

Average Intercorrelation Matrices for the Three Levels of theInformational Sequence Factor

ORIGINAL AIR FORCE SEQUENCE

Pictorial Verbal Concept S/E Corr. DeltaPictorial Multiple Choice 1.0Verbal Multiple Choice .65** 1.0Concept Cloze .71** .60** 1.0Subject/Expert Correlations .15 .22 .42** 1.0Delta Reading Vocabulary .47** .52** .60** .23* 1.0

N = 56 Ssp .05

** p < .01

PROXIMAL SEQUENCE

Pictorial Verbal Concept S/E Corr. DeltaPictorial Multiple Choice 1.0Verbal Multiple Choice .62** 1.0Concept Cloze .65** .51** 1.0Subject/Expert Correlations .02 .05 .34** 1.0Delta Reading Vocabulary .55** .46** .54** .18 1.0 ,

N = 56 Ss* p <.05** p < .01

. NON PROXIMAL SEQUENCE

Pictorial Verbal Concept:S/E Corr. DeltaPictorial Multiple Choice 1.0Verbal Multiple Choice .51** 1.0Concept Cloze .65** .48** 1.0SuFject/Expert Correlations .25* .36** .30* 1.0

Delta Reading Vocabulary .34** .47**_ .41** .22 1.0

N = 56 Ss* p < .05** p < .01

21

KatAnalyse8 of 'Variance (Males/Females 'and 13acksge.DifficuitYl; A two-way, between subjects, fixed effectsanalysis of variance was calculated for each of the fourdependent measures with males (122 subjects) / females (58subjects) and Package Difficulty as the factors. Rather thantreat each of the analyses separately, the results for eachfactor on each dependent measure will be presented contiguously.

As with the four-way analyses of variance the maineffect of package difficulty was significant on all fourdependent measures at the .001 level. The Characteristicsof Matter package lead to better performance than theOscilloscope instructional material.

The Male/Female main effect was significant with all fourdependent measures (Pictorial, Multiple Choice, F(1,176 d.f.) =3.8, p 4.075; Verbal Multiple Choice, F(1,176 d.f.) = 5.3,1?AC .025; Concept Cloze, F(1,176 d.f.) = 7.5, p < .01; S/ECorrelations, F(1,176 d.f.) = 22.4. p .001). Inspection ofthe means indicates that in all cases the males performed ata higher level than the females.

The interaction between Male/Female and Package Difficultywas si9nificant on all of the dependent measures except theVerbal Multiple Ukagtest (Pictorial Multiple Choice,F(1,176 d.f.) = 4.2, p.05; Concept Cloze F(1,176 d.f.) =12.3, p< .001; S/E Correlations, F(1,176 d.f.) = 12.8, p < .001.The means involved in these significant interactions are presentedin Table 8. As can be seen fram the table, the differencesbetween males and females is in performance on the Oscilloscopematerial. Again, it should be noted that the S/8 Correlationmeasure is sensitive to differences with the Oscilloscopeinstructional material.

22

TABLE 8

Two Way Analyses of Variance: Group Means for the SignificantInteraction Between the Male /female Factor and Package Difficulty

PICTORIAL MULTIPLE CHOICE**

Characteristics of MatterMales .793 (N = 57 Ss)Females .752 (N. = 33 Ss

Oscilloscope Operation.568 (N = 65 Ss). 97 N = 25 Ss

** p 4:.05

VERBAL MULTIPLE CHOICE

Characteristics of MatterMales .822 ON = 57 Ss)Females .762 (N = 33 Ss)

Oscilloscope Operation. 599 (N = 65 Ss)

.565 (N = 25 Ss)

MalesFemales

CONCEPT CLOZE*

aaracteristics of Matter.853 (N = 57 Ss).832 (N = 33 Ss)

4s,Oscilloscope_Operation

. 592 (N = 65 Ss)

.436 (N = 25 Ss)

* p< .001

MalesFemales

SUBJECT/EXPERT CORRELATIONS*

Characteristics of Matter Oscilloscope Operation.37 (N = 57 Ss).118 N = 3 Ss

.299 (N = 65. Ss)

.180 N = 25 Ss

* P4-.001

Two Way Analyses of Variance (Male ROTC/Male Non-ROTC).A two-way, between subject, fixed effects analysis7Z varncewas performed on each of the four dependent measures with MaleROTC (50 subjects)/ Male Non-ROTC (72 subjects) and PackageDifficulty as the factors. Again, the results for each factoron each dependent measure will be presented contiguously.

23

As.expected the effect of package difficulty was againsignificant at the .001 ley:elan all four dependent measures.

The effect of the Male ROTC/Male Non-ROTC factor wassignificant with the Pictorial Multiple Choice (F(1,118 d.f.)8.8, p <7.005) and the Verbal Multiple Choice (F(1,118 d.f.) =5.3, p<.025) and was nonsignificant with the Concept Clozeand the S/E Correlations. Inspection of the significant means(Table 9) reveals that the ROTC males performed at a higherlevel than the Non-ROTC males. The interaction effects waLnonsignificant with all dependent measures.

TABLE 9

Group Means for Male ROTC/Male Non-ROTC Factor on Each DependentMeasures

(Proportion Correct Except Where Indicated)

Dependent MeasureMALE ROTC

N = 50MALE NON-ROTC

N = 72Pictorial Multiple Choice** ..699 .655

Verbal Multiple Choice* .719 .703Concept Cloze .735 .698S/E Correlations(average correlation)

.236 .211

* p < .025** p.< .005

Very little weight should be given to the finding ofsignificant differences between Ram and Non-ROTC males fortwo reasons. First, the distribution of individuals framthese two categories over experimental treatments was rela-tively uneven (due to constraints of the experimentalsituation). Thus, treatment effects would naturally confoundthese differences. Second, the motivational level of thetwo categories of subjects may have varied due to themilitary nature of the experiment.

24

Discussion

The discussion of results will be divided into foursub-sections dealing with the following topics: the impactof instructional sequencing, the effect of presentationmode, the role of individual differences, and the effectivenesE,of the dependent measures.

Before proceeding with the discussion it is importantto point out that a number of other analyses includinganalyses of covariance, were performed on the data. Becausethese analyses provided information that was redundant tothe reported results, they are not presented in this document.

The act of Instructional Sequencing. The primarypurpose of t s research was to study instructional sequencingin a context which resolved some of the difficulties citedin the literature review and synthesis (1FTWA-TR-73-5l (I).In particular, the aim was to study the effects of objectivelyderived sequences (multidimensional scaling methods) andtheir interaction with individual difference variables andother meaningful instructional manipulations. The result ofthe four way analyses of variance indicated that a main effectcomposed of three different instructional sequences was signi-ficant with the Verbal Multiple Choice Measure and that all otherinteractions involving this factor were nonsignificant.

In addition, inspection of the means (Table 3) associatedwith the main effects on the other three dependent measuresindicated that, in general, as with the Verbal Multiple Choicetest, the proximal sequences (generated by the multidimensionalscaling methods) led to better performance than the other twosequences.

As stated in the presentation of results, the failure ofthe main effect of three instructional sequences to reachsignificance on the other dependent variables is partiallyexplainable in terms of the present shortcomings of theseother measures. The Concept Cloze led to skewed results(ceiling effects) with the Characteristics of Matter package,thus reducing its sensitivity to differences in performance.Further, the S/E Correlation variable has been shown to beinsensitive to performance differences with the Characteristicsof Matter material. This finding will be discussed morethoroughly in the sub-section entitled: The Effectivenessof the Dependent Measures.

25

Although. one cannot be certain, it seems that the lackof correspondence among dependent ireasures with the instructionalsequencing variable is not due to a spurious effect shown withthe Verbal Multiple Choice variable, but rather to the relativeinsensitivity of the remaining measures (remember that thedirection of differences between mans was similar for allmeasures except the S/E Correlation). It should be noted thatthe Verbal Multiple Choice Measure contains the sets ofquestions originally used by the Air Force in testing acquisi-tion of the instructional materials. Presumably this measurehas been examined much more thoroughly for reliability andvalidity than the three other prototype measures especiallyconstructed for this experiment.

Post hoc analysis of the effect of instructional sequencingwith the Verbal Multiple Choice Variable indicated that theProximal sequence derived from the multidimensional scalingmethod led to significantly better performance than the Non -Proximal Sequence. Inspection of the means revealed that theOriginal Air Force Sequence produced a level of studentperformance which fell between the Proximal and Non-Proximalsequences. The lack of larger differences in mean performancebetween the proximal and original Air Force Sequence conditionsis not surprising. First, the original Air Force sequences haveundergone revisions based on prior instructional experiences(revisions of some of this material probably occurred evenprior to incorporation of these packages by the Air Force).These sequences are now undoubtedly close to being optimal forperformance effective instruction. In fact, the actualsequence differences between the proximal and the originalwere quite subtle, therefore limiting potential performancedifferences. However, with newer instructional material (thatIs, material that has not been heavily revised based oninstructional experience) it would be expected that theproximal sequence generated by the objective multidimensionalscaling approach would lead to substantially better performancethan a sequence. generated by subjective means.

A second reason for the small performance differentialin favor of the proximal sequence is the size of the ins truc-tional unit treated in this experiment. Due to pragmaticconstraints, only about 30-40 minutes of instructional materialcould be presented. It is probably true that the impact of

26

sequencing is much stronger over larger units of instruction.The multidimensional scaling method can be applied to alllevels of instruction from the overall curriculum to aparticular course to a specific lesson. If there are performancedifferences in favor of this method with small units ofinstruction then these differences should be amplified withlarger units.

Even if the objective multidimensional scaling methodproves only equal in educational effectiveness to the presentsubjective techniques used by curriculum designers, the methodmay well offer advantages in efficiency, replicability, andcontrol. With subjective techniques, the resulting sequencedepends heavily on the skill of the individual designer, whilethe objective sequencing method permits the views of a largenumber of "experts" to be combined at a very small cost.Because the latter method is objective and allows for thepooling of a number of opinions, costly checking and re-checkingof the effectiveness of instruction may be dramatically reduced.

The lack of significant interaction between instructionalsequence and variables representing both individual differencesand other instructional manipulations (for example, mode of presenta-tion and package difficulty) may be most likely due to one oftwo reasons. First, the dependent measures employed may nothave been sufficiently sensitive to detect such effects.Second, with the sets of instructional materials employed inthe present study, it may be true that such interaction effectsdo not exist at a detectable level. In either case, furtherstudies should be undertaken using a wider range of instructionalmaterial(especially material that has a wide range of a priorifamiliarity associated with it) and a greater variety ofdependent measures.

The Effect of Presentation 'Mode. The results of the fourway aR4=Yses of variance indicated that with the PictorialMultiple Choice test Pictorial/Graphic presentation leads tosignificantly better performance than Verbal presentation(marginally at the .075 level). Although this main effect didnot reach significance with the other three dependent measures,inspections of the means in Table 2 reveals general agreementamong the measures with regard to the direction of mean differ-ences.

27

Perhaps the failure of the other measures to reach signifi-cance was due to the difficulties on the part of the Pictorial/Graphic presentation groups in translating the learned materialinto the verbal format required by the Verbal Multiple Choice,Concept Cloze, and S/E Correlation Treasures. In any case, thevery tentative conclusion that the Pictorial/Graphic mode of ,

presentation leads to better performance than the Verbal noderequires further exploration and delimitation in the context ofa variety of instructional materials.

The Role of Individual Differences. Besides their potentialinteraction with instructional sequence, individual differencesare of general importance in the context of instruction. In thisstudy three sources of individual differences were examined:performance on the Delta Reading Vocabulary test, sex, and RO'T'Caffiliation. As mentioned in the presentation of results, theeffects of RCTC affiliation were potentially confounded bydifferential motivational properties of the task for variousclasses of individnAlq and distributional imbalances across theexperimental treatment groups. Because of these difficultiesthere will be no further discussion of this effect.

The results of the four-way analyses of variance indicatedthat high scorers on the Delta measure performed substantiallybetter on all dependent variables than did low scorers. Theone significant interaction involving Delta was with packagedifficulty on the S/E Correlation measure. As stated previously,this effect is probably due to the differential sensitivity ofthis dependent measure with the two levels of instructionalmaterial rather than a true interaction between Delta andpackage difficulty.

The intercorrelations between Delta and the dependentmeasures generally fell between .35 and .60. These correlationswere samewhat higher for: the verbal 11-ode of presentation andsomewhat lower for pictorial presentation. Since the Deltais a paper and pencil test requiring only 10 minutes for admin-istration, it would appear to have substantial value as a devicefor selecting students and as a device for determining thesequence and pace or rate of instruction. This would beespecially true for verbal materials.

28

The analysis of male/female differences in performancegenerally indicated a large difference in favor of the maleson the Oscilloscope Operation package and a substantiallysmaller difference also in favor of the males, on the Charac-teristics of Matter material. The Oscilloscope Opexationpackage led to significantly worse performance than theCharacteristics of Matter on all dependent measures. Thedifficulty of this package and the relatively unfamiliarscientific concepts involved may have caused the greaterdifferential between males and females due to a prioridifferences in scientific interest and ability. Such differ-ences should certainly be taken into account in the designof future instructional programs.

Effectiveness of the Dependent Measures. The Verbal andPictorial Multiple Choice Measures appeared to give consistentresults in a range unaffected by "ceiling" and "floor" effects.However, as with the two other measures, there may be somequestion as to their effectiveness with modes of. presentationthat are not compatible with the format of the measures. Thisnotion has been discussed earlier in the section entitled:The Effect of Presentation Mbde (see page 27).

Further comments regarding the effectiveness of the ConceptCloze and S/E Correlation measures are necessary. The ConceptCloze test appears to have much promise as an alternative tothe Standard Cloze technique. As with the Standard Cloze it isobjectively constructed and scored,, but unlike the standardapproach, it only tests knowledge of important concepts, notknowledge associated with a combination of important conceptsand trivial function words. In this study, the only difficultywith the Concept Cloze stemmed from the "ceiling effects"found with the verbal presentation of the Characteristics ofMatter material. This situation could easily be alleviatedby asking the subjects to fill in the appropriate key conceptsin a segment of parallel text not used during the instructionbut which contains the same material. This would increasethe difficulty of the task over that which exists with the textmaterial previously used during instruction.

29

As discussed in the presentation of the results, theS/E Correlation measure only appears to be sensitive andconsistent with the Oscilloscope Operation package. Withinthis package this measure has substantial intercorrelationswith the other dependent Treasures and, on the basis of themagnitude of the F ratios, it may in fact be a more sensitiveindicant of performance. Although the S/E Correlation measurehas promise, further exploration of its limitations, especiallywith varieties of instructional material, is necessary.

The intercorrelations between the dependent measures, inparticular the Verbal Multiple Choice, the Pictorial MultipleChoice, and the Concept Cloze, generally fell between .60and .70. This level of relationship is sufficient to providesome evidence of concurrent validity, but not high enough torecommend the elimination of same of the measures on thebasis of redundancy. A set of measures of this type maybevery useful for assessment and prescription in an instructionalsetting.

Summary and Conclusions

The purpose of this investigation was to explore theeffects of instructional sequence*on performance in anexperimental context which presumably corrected same c. theproblems plaguing prior research in this area. In particular,the experiment was designed to accomplish the following:

(1) To assess the effectiveness of instructionalsequences generated by a multidimensional scaling approach;a new approach which can objectively synthesize the views ofa number of curriculum "experts",

(2) To utilize newly developed dPnendent measures in anattempt to provide more sensitive indicants of performancedifferences,

(3) To further our knowledge of the interaction ofindividual difference variables and sequencing,

(4) To assess the interaction of instructional sequenceeffects with two other educationally important variables: madeof presentation and instructional difficulty.

n

Although not definitive, the results of this experimentdo provide evidence for the utility of the INSCAL multidimen-sional scaling approach in describing information structuresand generating instructional sequences. Generally, the meanperformances with the proximal sequence (a presumably optimalsequence generated by the multidimensional scaling approach)were higher than performances with both the original Air Forcesequence, and the non-proximal sequence (a second sequencegenerated by multidimensional scaling using a different setof generation rules). It would be expected that performanceadvantages associated with the proximal sequence would befurther enhanced in comparisons with newer Air Force instruc-tional material (material in which sequence had not beenheavily modified on the basis of prior instructional experience)and with larger instructional units. In any case, the INSCALmultidimensional scaling method for developing instructionalsequences offers some unique advantages over the traditionalsubjective approaches:

(1) The INSCAL approach can combine the views of alarge number of "experts" to form a single sequence. Besidesproviding consensual validation, this feature has the psycho-logical advantage of involving a number of individuals incurriculum decision making.

(2) The INSCAL approach can be applied to instructionat all levels; the overall curriculum, an individual course,and/Or a specific lesson.

(3) Since the INSCAL is sensitive to differencesbetween individuals contributing individuals, it is possibleto objectively select a particular instructional sequencefor a specific sub - -group of instructors or students.

(4) The INSCAL has the capability of providing anindex of instructional complexity. Such an index wouldprovide a strong basis for stwient selection and pacing.

Overall, the INSCAL multidimensional scaling approach offersadvantages sufficient to warrant further research and development.

31

In addition to the effects of sequence, the present experi-ment showed consistent but small, performance differences dueto mode of presentation. The Pictorial/Graphic mode led tobetter performance than did the Verbal mode. Since the pictorial/graphic format for the instructional materiaa used has undergonefar less research and development than the verbal format, thisresult is quite encouraging.. Further development of the pictorial/graphic format should be undertaken and its' interaction withindividual difference variables (for example, mode preference)should be assessed.

.

With regard to individual differences in this study, theDelta Reading Vocabulary test proved to be substantially relatedto performance with correlations generally ranging from .35 to.60 depending on the experimental condition. Since this measurerequires very little time to administer and score it would appearto be a useful adjunct to test batteries used for student selectionand prescription.

The performance differences between males and females.in thisexperiment were substantial (the males performing at a higherlevel than the females). Clearly, sex should be taken into accountin the design of individualized instruction, especially instructioninvolving scientific and/or technical content.

The Concept Cloze test and the S/8 Correlations were two newdependent measures designed and constructed for used in thisexperiment. Although both of these measures contained some flaws,they appear to have sufficient potential to warrant their furtherinvestigation as indicants of successful acquisition of instructionalmaterial.

32

References

Dansereau, D.F., Evans, S.H., Wright, A.D., Long, G.L.,Actkinson, T.A. Factors related to developing optimalinstructional sequences: Phase I. MERL-TR-73-51 (I),Texas Christian University, Contract F41609-73-C-0023,U.S. Air Force Human Resources Laboratory, Lowry AFB,Colorado, in press, 1974.

33

ANNEX A

MEANS AND STANDARD DEVIATIONS FOR EACH EXPERIMENTAL

CONDITION ON ALL DEPENDENT MEASURES

34

Performance Means and Standard on All Dependent Measuresfor the Original Air Force Sequence Condition

(Proportion Correct Except for S/E Measure whichConsists of Average Correlations)

MODE OFPRESENTATION

PACKAGEDIFFICULTY

DELTA PERFORMANCE

HIGH LOWMeasure Mean S.D. Measure Mean S.D.

PMC .590 .096 PMC .446 .121

VMC .623 .108 VMC .564 .124OSCILLOSCOPE CC .651 .225 CC .373 .167

S/E .350 .172 S/E .211 .149

D .768 .073 D .544 .055

VERBALPMC .907 .089 PMC .653 .121

VMC .931 .077 VMC .646 .183

CHARACTERISTICS CC .993 .017 CC .701 .253OF MATTER S/E .111 .027 S/E .130 .082

D .876 .048 D .603 .079

PMC .630 ,128 PMC .538 .126

VMC .643 .100 VMC .555 .128

OSCILLOSCOPE CC .647 .244 CC .410 .192

S/E .340 .123 S/E .304 .143

D :798 .032 D .658 .095

PICTORIALGRAPHIC PMC .810 .126 PMC .734 .195.

VMC .838 .091 VMC .783 .130

CHARACTERISTICS CC .916 .171 CC .787 .200

OF MATTER S/E .154 .075 S/E .110 .077

D .853 .046 D .557 .091

N = 7 Ss Per CellKEY

PMC = Pictorial Multiple ChoiceVMC = Verbd1 Multiple ChoiceCC = Concept ClozeS/E = S/E CorrelationsD = Delta Reading Vocabulary

35

PArformpnce Means and. Standard DMeasures for the Proximal

(Proportion Correct ExceptConsists of Average

eviations or. All DependentSequence Condition

for S/E Measure whichCorrelations)

MODE OFPRESENTATION

PACKAGEDIFFICULTY

DELTA PERFORMANCE

HIGH LOV,

Measure Mean_ S,D. Measure Mean j S.D.

PMC .564 .110 PMC .487 .123

VMC .638 .100 VMC .638 .216


S/E .316 .112 S/E .181 .132

D .808 .048 D .503 .119

VERBALPMC .887 .092 PMC' .663 .166

VMC .933 .060 VMC .747 .107


OF MATTER S/E .120 .035 S/E .153 .048

D .818 .035 D .606 .131

PMC .643 .139 PMC .460 .131

VMC .628 .176 VMC .554 .147


S/E .324 .176 S/E .197 0.21

D .847 .072 D .601 .088

PICTORIALGRAPHIC PMC .894 .069 PMC .759 .146

VMC .937 .056 VMC .763 .213


OF MATTER S/E .158 .047 S/E .113 .058

D .824 .062 D .598 .058

N = 7 Ss Per CellKEY

PMC = Pictorial Multiple ChoiceVMC = Verbal Multiple ChoiceCC = Concept ClozeS/E = S/E CorrelationsD = Delta Reading Vocabulary

36

e.rformance Means and Standard Deviations on All DependentMeasures for theNon-Proximal Sequence Condition

(Proportion Correction Except for S/E Measurewhich Consists of Average Correlations)

MODE OFPRESENTATION

PACKAGEDIFFICULTY

DELTA PERFORMANCE

HIGH LOWMeasure Mean S.D. Measure Mean S.D.

PMC .597 .139 PMC .488 .157

VMC .623 .087 VMC .478 .087


S/E .307 .118 S/E .104 .064

D .749 .083 D .510 .092

VERBALPMC .747 .139 PMC .677 .106

VMC .758 .110 VMC .648 .161


OF MATTER S/E .143 .104 S/E .123 .075

D .766 .036 D .510 .155

PMC .597 .138 PMC .538 .098

VMC .641 .154 VMC .504 .054


S/E .333 .092 S/E .237 .162

D .848 .036 D .648 .097

PICTORIALPMC .900 .076 PMC .701 .208GRAPHICVMC .888 .065 VMC .717 .222


OF MATTER S/E .147 .086 S/E .107 .091

D .796 .075 D .513 .103

N = 7 Ss Per Cell

KEYPMC = Pictorial Multiple ChoiceVMC = Verbal Multiple ChoiceCC = Concept ClozeS/E = S/E CorrelationsD = Delta Reading Vocabulary

37

DOCUMENT RESUME ED 094 734 Dansereau, Donald F.; And Others · media presentation modalities...

Documents

Transcript of DOCUMENT RESUME ED 094 734 Dansereau, Donald F.; And Others · media presentation modalities...