Development and Validation of a Tool for Development and Validation of a Tool for Measuring Indicators of Information and Measuring Indicators of Information and Communication Technologies (ICTs) at a Communication Technologies (ICTs) at a

Turkish State UniversityTurkish State University

Yavuz AkbulutYavuz AkbulutProf. Dr. Mehmet KesimProf. Dr. Mehmet Kesim

Anadolu UniversityAnadolu UniversityEskişehir, Turkey Eskişehir, Turkey

yavuzakbulut@anadolu.edu.tryavuzakbulut@anadolu.edu.trmkesim@anadolu.edu.tr mkesim@anadolu.edu.tr

22

IntroductionIntroduction

In the 21st century, skills regarding information In the 21st century, skills regarding information and communication technologies (ICTs) have and communication technologies (ICTs) have gained utmost importance for education, for gained utmost importance for education, for employment and for everyday life use. employment and for everyday life use. ICTs provide a more colorful atmosphere in ICTs provide a more colorful atmosphere in comparison to face-to-face learning to access comparison to face-to-face learning to access global knowledge and support. global knowledge and support. TThe ability to use ICTs with confidence and he ability to use ICTs with confidence and efficiency is demanded from most individuals. efficiency is demanded from most individuals. In this respect, it is important to equip In this respect, it is important to equip individuals with skills to use ICTs efficiently, individuals with skills to use ICTs efficiently, independently and responsibly.independently and responsibly.

33

Survey of European Universities Skills in ICT of Survey of European Universities Skills in ICT of Students and Staff ICT (SEUSISS)Students and Staff ICT (SEUSISS) 13.000 students, staff and employers across 13.000 students, staff and employers across

Europe respondedEurope responded ICT was considered vital for their future careers by ICT was considered vital for their future careers by

the great majority of the participating students the great majority of the participating students (Haywood, 2003)(Haywood, 2003)

ICT skills of students from different universities ICT skills of students from different universities varied in accordance with some variables including varied in accordance with some variables including age, gender, academic subjectage, gender, academic subject domains, year of domains, year of study, family support, and the richness of facilities study, family support, and the richness of facilities offered to studentsoffered to students... .

However, in order to examine the problem better, However, in order to examine the problem better, there needs to be efficient, timely and meticulous there needs to be efficient, timely and meticulous measurement criteria and tools so that the ICT picture measurement criteria and tools so that the ICT picture of a specific organization or community could be of a specific organization or community could be diagnosed betterdiagnosed better..

44

Since ICTs in teacher education constitute a Since ICTs in teacher education constitute a dynamic field of study, which requires constant dynamic field of study, which requires constant refreshment, there is always a need to measure refreshment, there is always a need to measure up-to-date latent constructs of ICTs through valid up-to-date latent constructs of ICTs through valid and reliable tools including high quality and reliable tools including high quality indicators.indicators.

Some measurement toolsSome measurement tools Lin (2005): ICT integration factors that are Lin (2005): ICT integration factors that are

considered important for teachersconsidered important for teachers Liu and Huang (2005): Concerns of teachers Liu and Huang (2005): Concerns of teachers

regarding ICT integrationregarding ICT integration AkpAkpıınar (2003): K-12 teachers’ technology nar (2003): K-12 teachers’ technology

using behaviorsusing behaviors

55

PurposePurpose

The current study has a different scope from The current study has a different scope from above studies for it purports to develop a above studies for it purports to develop a measurement tool for diagnosing the overall measurement tool for diagnosing the overall picture of an institution regarding internal picture of an institution regarding internal and external ICT indicators from prospective and external ICT indicators from prospective teachers’ points of view. teachers’ points of view.

66

Indicators of Successful ICT IntegrationIndicators of Successful ICT Integration

UNESCOUNESCO (2002) (2002) & Odabasi et al. (2006) & Odabasi et al. (2006) content and pedagogy issuescontent and pedagogy issues collaboration and networking issuescollaboration and networking issues social and health issuessocial and health issues technical issuestechnical issues

77

88

Methods & ProceduresMethods & Procedures

Research QuestionResearch Questionss

What are the valid and reliable indicators to What are the valid and reliable indicators to diagnose the overall ICT picture of an educational diagnose the overall ICT picture of an educational institution and its members from prospective institution and its members from prospective teachers’ perspectives?teachers’ perspectives?

What is the current situation of the selected What is the current situation of the selected university in terms of ICT indicators?university in terms of ICT indicators?

99

ParticipantsParticipants359 senior education faculty students (% 43 of all 359 senior education faculty students (% 43 of all education faculty students at Anadolu University)education faculty students at Anadolu University)

1010

InstrumentInstrument

A personal information form collecting independent A personal information form collecting independent variables needed for further analysesvariables needed for further analyses

Six to ten statements for each indicatorSix to ten statements for each indicator

Revision by an IDT instructor and 8 PhD candidatesRevision by an IDT instructor and 8 PhD candidates

Expert opinions (2 professional test developers)Expert opinions (2 professional test developers)

Number of items after evaluations: 54Number of items after evaluations: 54

5-item scales: Never, rarely, sometimes, very often 5-item scales: Never, rarely, sometimes, very often and alwaysand always

1111

ProcedureProcedure

Written permissions from the rectorate and the Written permissions from the rectorate and the human subjects committee of the institution human subjects committee of the institution

Administration of the survey during students’ normal Administration of the survey during students’ normal class hours at the second half of 2006 fall semester. class hours at the second half of 2006 fall semester.

Same type of instructions to all participants Same type of instructions to all participants

15 to 20 minutes to complete the questionnaire15 to 20 minutes to complete the questionnaire

1212

Data AnalysisData Analysis

An exploratory factor analysis through SPSS 14.0 An exploratory factor analysis through SPSS 14.0 for for WWindows indows

A confirmatory factor analysis through LISREL 8.51A confirmatory factor analysis through LISREL 8.51

Descriptive statisticsDescriptive statistics

Parametric tests of comparisonParametric tests of comparison

1313

Results Results (1 of 5)(1 of 5)

Kaiser-Meyer-Oklin Measure of Sampling Adequacy Kaiser-Meyer-Oklin Measure of Sampling Adequacy (i.e. .848) (i.e. .848)

Bartlett’s Test of Sphericity (χ²=6755.498; p<.0005) Bartlett’s Test of Sphericity (χ²=6755.498; p<.0005)

Values were ideal to conduct factor analysis Values were ideal to conduct factor analysis (Hutcheson & Sofroniou,1999; Pallant, 2001). (Hutcheson & Sofroniou,1999; Pallant, 2001).

1414

Results Results (2 of 5)(2 of 5)

Items with inappropriate loadings (i.e. less than .40) were eliminatedItems with inappropriate loadings (i.e. less than .40) were eliminated..

Items with inappropriate corrected-item total values (i.e. less Items with inappropriate corrected-item total values (i.e. less than .30) were eliminated one by one, and further factor analyses than .30) were eliminated one by one, and further factor analyses were conducted each timewere conducted each time..

Final solution with the maximum likelihood extraction method Final solution with the maximum likelihood extraction method revealed 41 items indicating 10 factors with eigenvalues exceeding revealed 41 items indicating 10 factors with eigenvalues exceeding 1, and explained 51.557 % of the total variance. This was considered 1, and explained 51.557 % of the total variance. This was considered appropriate according to the literature (Dunteman, 1989). appropriate according to the literature (Dunteman, 1989).

Items were further processed through LISREL 8.51 (Jöreskog & Items were further processed through LISREL 8.51 (Jöreskog & Sörbom, 2001) to conduct a confirmatory factor analysis. Sörbom, 2001) to conduct a confirmatory factor analysis.

Three further questions were eliminated during the confirmatory Three further questions were eliminated during the confirmatory analysis and the final model revealed analysis and the final model revealed an idealan ideal root mean square root mean square error of approximation (RMSEA) valueerror of approximation (RMSEA) value(i.e., (i.e., .05.05)) along with ideal fit along with ideal fit indices (non-normed fit index: .89; comparative fit index: .90; indices (non-normed fit index: .89; comparative fit index: .90; incremental fit index: .90) incremental fit index: .90) (Raykov & Marcoulides, 2006).(Raykov & Marcoulides, 2006).

1515

Results Results (3 of 5)(3 of 5)

The Cronbach’s Alpha (The Cronbach’s Alpha () of the instrument was .888) of the instrument was .888

Ten factors were labeled as the followingTen factors were labeled as the following (1) Ease of use (eigenvalue: 10,173)(1) Ease of use (eigenvalue: 10,173) (2) (2) TTeaching – learning method (eigenvalue: 4,297)eaching – learning method (eigenvalue: 4,297) (3) (3) EEthics (eigenvalue: 3,504)thics (eigenvalue: 3,504) (4) (4) SSpecial needs (eigenvalue: 1,905)pecial needs (eigenvalue: 1,905) (5) (5) IInfrastructure (eigenvalue: 1,616)nfrastructure (eigenvalue: 1,616) (6) (6) PProfessional development (eigenvalue: 1,574)rofessional development (eigenvalue: 1,574) (7) (7) AAccess (eigenvalue: 1,338)ccess (eigenvalue: 1,338) (8) (8) HHealth (eigenvalue: 1,216)ealth (eigenvalue: 1,216) (9) (9) PPolicy (eigenvalue: 1,089) olicy (eigenvalue: 1,089) (10) ICT in curriculum context (eigenvalue: 1,047). (10) ICT in curriculum context (eigenvalue: 1,047).

Technical assistanceTechnical assistance and and learning communitieslearning communities were not were not revealed through the analysisrevealed through the analysis

1616

Results Results (4 of 5)(4 of 5)

Means on each factor were calculated and these values Means on each factor were calculated and these values were compared with the neutral value of 3 through one-were compared with the neutral value of 3 through one-sample t-tests as done in Warschauer (1996)sample t-tests as done in Warschauer (1996)

Since ten t-tests were conducted, a Bonferroni Since ten t-tests were conducted, a Bonferroni Adjustment Procedure was applied as suggested by Adjustment Procedure was applied as suggested by Huck (2000), and the critical p-level was determined Huck (2000), and the critical p-level was determined as .005. as .005.

1717

Results Results (5 of 5)(5 of 5)

No difference between the results and the neutral value of 3No difference between the results and the neutral value of 3 ICT in curriculum contextICT in curriculum context ( (=2.982; t357=-.429; p=.668)=2.982; t357=-.429; p=.668)

Results were better than expectedResults were better than expected EthicsEthics ( (=3.664; t357=16.138; p<.0005)=3.664; t357=16.138; p<.0005) Professional developmentProfessional development ( (=3.508; t357=.10.845; p<.0005)=3.508; t357=.10.845; p<.0005) HealthHealth ( (=3.975; t357=17.291; p<.0005).=3.975; t357=17.291; p<.0005).

Results were worse than expectedResults were worse than expected Teaching-learning methodTeaching-learning method ( (=2.636; t357==2.636; t357=--8.664; p<.0005)8.664; p<.0005) Ease of useEase of use ( (=2.175; t357==2.175; t357= --19.194; p<.0005)19.194; p<.0005) Special needsSpecial needs ( (=1.884; t357==1.884; t357= --22.795; p<.0005)22.795; p<.0005) InfrastructureInfrastructure ( (=2.657; t357==2.657; t357= --8.220; p<.0005)8.220; p<.0005) AccessAccess ( (=1.916; t357==1.916; t357= --19.570; p<.0005)19.570; p<.0005) PolicyPolicy ( (=2.162; t357==2.162; t357= --16.155; p<.0005). 16.155; p<.0005).

1818

Discussion Discussion (1 of 5)(1 of 5)

The present paper suggests a measurement tool for examining ICT indicators. The study The present paper suggests a measurement tool for examining ICT indicators. The study set out with 12 factors and more than a hundred statements at the beginning and finished set out with 12 factors and more than a hundred statements at the beginning and finished with 10 predetermined factors indicated by a total of 38 questions. with 10 predetermined factors indicated by a total of 38 questions.

Indicators of technical assistance were contaminated by indicators of access, ease of use Indicators of technical assistance were contaminated by indicators of access, ease of use and infrastructure. Since these subcategories all belonged to the construct of technical and infrastructure. Since these subcategories all belonged to the construct of technical issues as mentioned in the literature review, this was not an extraordinary finding. issues as mentioned in the literature review, this was not an extraordinary finding.

None of the items addressing learning communities had appropriate loadings to be None of the items addressing learning communities had appropriate loadings to be included in the final version of the scale. This could be explained by two possibilities. included in the final version of the scale. This could be explained by two possibilities.

IItems addressing learning communities might have been badly prepared. tems addressing learning communities might have been badly prepared. Turkish students might lack skills to work properly in learning communities, which Turkish students might lack skills to work properly in learning communities, which

leads to serious results. leads to serious results.

1919

Discussion Discussion (2 of 5)(2 of 5)

The analysis conducted with a single sample in the current The analysis conducted with a single sample in the current study demonstrates first-order relationships between sub-study demonstrates first-order relationships between sub-categories of UNESCO (2002)’s four competencies and categories of UNESCO (2002)’s four competencies and observed variables. observed variables.

Further analyses with new samples could be conducted to Further analyses with new samples could be conducted to see the relationships between four competencies and see the relationships between four competencies and subcategories, and between subcategories and observed subcategories, and between subcategories and observed variables, which can reveal second-order relationships. variables, which can reveal second-order relationships.

Moreover, through administering the current scale across Moreover, through administering the current scale across different populations, the tool might be developed further, so different populations, the tool might be developed further, so that structural equation models can be suggested among that structural equation models can be suggested among reliably measured constructs. reliably measured constructs.

2020

Discussion Discussion (3 of 5)(3 of 5)

Means of each factor reveals that the sample institution Means of each factor reveals that the sample institution where the data collection took place is at an average level in where the data collection took place is at an average level in terms of ICT in the curriculum context. terms of ICT in the curriculum context.

The institution is at a better situation than the neutral value in The institution is at a better situation than the neutral value in terms of ethics, professional development and health. terms of ethics, professional development and health.

However, the institution needs immediate action in terms of However, the institution needs immediate action in terms of teaching-learning method, ease of use, special needs, teaching-learning method, ease of use, special needs, infrastructure, access and policy. infrastructure, access and policy.

2121

Discussion Discussion (4 of 5)(4 of 5)

Since the current data were collected from an opportunity sample Since the current data were collected from an opportunity sample which was a single Turkish state university, generalizations to other which was a single Turkish state university, generalizations to other state universities based on the current dataset could only be state universities based on the current dataset could only be suggestive rather than definitive. suggestive rather than definitive.

Besides, the current scale investigates ICT indicators from senior Besides, the current scale investigates ICT indicators from senior students’ points of view. A parallel form of the questionnaire could be students’ points of view. A parallel form of the questionnaire could be developed for instructors to investigate their perspectives in terms developed for instructors to investigate their perspectives in terms ICT indicators. ICT indicators.

2222

Discussion Discussion (5 of 5)(5 of 5)

As mentioned before, the target population of the current study As mentioned before, the target population of the current study was senior students of the Faculty of Education at Anadolu was senior students of the Faculty of Education at Anadolu University. The scale should be used with larger samples University. The scale should be used with larger samples across different faculties aside from the education faculty to across different faculties aside from the education faculty to develop its construct validity and generalize the results to a develop its construct validity and generalize the results to a larger reference population. larger reference population.

Researchers intend to administer the scale across all education Researchers intend to administer the scale across all education faculties in Turkey, which might lead to comprehensive data on faculties in Turkey, which might lead to comprehensive data on the ICT integration level of educational faculties and the ICT integration level of educational faculties and prospective teachers.prospective teachers.

Besides, collaboration among European universities within the Besides, collaboration among European universities within the scope of a joint project like SEUSISS (Haywood, 2003) to scope of a joint project like SEUSISS (Haywood, 2003) to investigate ICT indicators might lead practitioners to invaluable investigate ICT indicators might lead practitioners to invaluable information about the ICT situation across European information about the ICT situation across European universitiesuniversities

2323

Thank YouThank You !!!!!!