ASSESSMENT OF ASSESSMENT OF THE USEFULNESS THE USEFULNESS OF A WEBSITE AS OF A WEBSITE AS

A LEARNING A LEARNING TOOL FOR A TOOL FOR A WEB-BASED WEB-BASED COURSE IN COURSE IN CLINICAL CLINICAL

GERONTOLOGY: GERONTOLOGY: A A

DEMONSTRATION DEMONSTRATION OF A FORMAL OF A FORMAL METHOD OF METHOD OF

EDUCATIONAL EDUCATIONAL INQUIRYINQUIRY

N. SCOTT GORMAN, N. SCOTT GORMAN, OD, MSOD, MS

Nova Southeastern Nova Southeastern University College of University College of Optometry Ft. Optometry Ft. Lauderdale, FloridaLauderdale, Florida

PURPOSE OF THE PURPOSE OF THE POSTERPOSTER

The purpose of this poster is to provide optometric educators with an illustration of the application of a method of educational inquiry known as evaluation research. The intent of this form of inquiry is to evaluate the impact of an educational intervention on the students cloaked within a learning environment. Two techniques for obtaining data for this type of inquiry are the use of questionnaires and the use of focus groups. Both of these are employed in this study.

The author is a doctoral candidate in higher education specializing in health care education at Nova Southeastern University’s Fischler Graduate School of Education and Human Services. This poster is adapted from an evaluation research study conducted to meet the curricular requirements of the program.

ABSTRACTABSTRACT

PROBLEM: A WebSite was designed and published to the Nova Southeastern University Intranet as a learning tool for a Web-based course in clinical gerontology. Its purpose was to enhance students’ access to learning resources. The problem for this study was to determine if the WebSite was a useful tool for the students.

METHODS: A 25-item questionnaire was designed, validated, and administered to 97 students at the conclusion of the course. The students' responses were recorded on a Likert scale (strongly agree = 1 to strongly disagree = 5) and analyzed using mean statistics and frequency data. Students’ written comments were reviewed. In addition, a focus group composed of six students convened to provide additional input about the usefulness of the WebSite.

RESULTS: In general, the students agreed with the questionnaire’s statements (Likert scale average mean = 1.99). The percentage of students that strongly agreed or agreed to the statements was 73.8%. The students’ two most frequent comments made were that they were not prepared technologically to take a Web-based course, and that they did not use the Chat Rooms component of the WebPage.

CONCLUSIONS: The WebSite was a useful learning tool. A computer and Internet training workshop should be offered during the summer prior to the beginning of the course. Also, the Chat Rooms should be eliminated.

REVIEW OF THE REVIEW OF THE LITERATURELITERATUREAA

In optometric education, the lecture method remains the predominant teaching method. However, Kemp suggests that the lecture method has many limitations.1 The most important of these limitations is that lecture is generally passive with little or no exchange with the instructor or other students in the classroom. Fortunately, a new instructional modality has made its way into higher education — Web-based instruction. According to Williams, this instructional method can help to increase the quality of classroom instruction.2 It has the capability of engaging the student in active learning.

Web-based instruction uses computer technology and the Internet to deliver information to the student as well as to afford communication and collaboration among students and the course instructor.3 For learning to be meaningful with this modality, it must be designed to provide an opportunity for collaboration, autonomy in learning, critical thinking, and interaction with students and the instructor.4 However, Web-based instruction does have its limitations. Although Kemp states that learners “work harder, learn more, and retain more of what is learned than do learners in conventional classes,” 1,

p. 146 he also states that there are several limitations that can be imposed on the students. These include lack of interaction and self-discipline as well as the occurrence of monotony and disinterest in the Web-based course.

REVIEW OF THE REVIEW OF THE LITERATURELITERATUREBB

The tools and activities that may be incorporated into a Web-page are important and useful. However, even though access is given to all students, there is no guarantee that they will be used.5 By using criteria developed by Abels6 for designing or re-designing WebPages, the pages perceived value will be increased.

To evaluate the value of a Web-page, an appropriate educational research methodology should be employed. Berger supports the use of evaluation methodology “to evaluate the success or impact of a program.”7, p. 145 He also states that needs assessment is the preferred method to assess educational resources. Rothwell supports Berger’s statement by adding that needs assessment is a useful method for collecting data. In obtaining information for the needs assessment, McMillan states that the questionnaire is the “most widely used technique for obtaining information from subjects.”8, p. 252 It is composed of written questions about instructional needs. To obtain more in-depth information, focus groups may be used. Focus groups are a form of qualitative research and involve interviewing groups of people. Morgan states that focus groups may be used to supplement information gained from another primary method such as a survey (questionnaire).8

METHODOLOGY AND METHODOLOGY AND PROCEDURESPROCEDURESAA

Evaluation Methodology

The needs assessment evaluation methodology was used to determine if students who completed the clinical gerontology course found the WebPage to be a valuable learning resource. To complete the study, this researcher followed eight steps:

STEP 1: A thorough review of the literature was conducted. The review included a search for information regarding the use of a WebPage as a learning resource as well as information that would assist the researchers in developing standards or criteria for the study. Specific topics that were reviewed were the lecture method of instruction, Web-based instruction, Web-based learning, active learning, WebPage tools and activities, criteria for the design of WebPages, needs assessment methodology, the use and development of questionnaires, and the use of focus groups.

Step 2: A formative and summative committee was established. The formative committee assisted with the development of criteria that specified what was indicative of a valuable Internet WebPage. The criteria were based upon the results of the literature review as well as input from the formative committee. The formative committee consisted of four persons who were WebPage developers. The summative committee validated the criteria.

STEP 3: A questionnaire was developed based upon the criteria established by the formative committee. The questionnaire contained twenty-five statements that could be responded to on a Likert scale from A to E (strongly disagree to strongly agree).

METHODOLOGY AND METHODOLOGY AND PROCEDURESPROCEDURESBB

STEP 4: Fourth, the research study and questionnaire were submitted to the Programs for Higher Education Institutional Review Board representative for approval.

STEP 5: The questionnaire was distributed to 103 optometric students during the second to the last class meeting of the semester, and the completed instrument was collected during the last class meeting of the semester. A committee of four faculty members reviewed the questionnaire for form and content validity prior to distribution.

STEP 6: A Focus Group composed of six students who completed the course and submitted the questionnaire was convened to provide additional input on the items contained in the questionnaire. This meeting was held in February 1999 following the conclusion of the course.

STEP 7: The resultant data were analyzed and compared with established criteria to determine the gaps between the ideal state and actual state. This information provided the basis for modifying the WebPage for the 1999-2000 academic year.

STEP 8: Data from the questionnaires and focus group meeting were compiled and analyzed, and evidence resulting from data analysis was used to reach conclusions and support decisions regarding revision of the Internet WebPage for academic year 1999-2000.

METHODOLOGY AND METHODOLOGY AND PROCEDURESPROCEDURESCC

Assumptions

It was assumed that optometric students taking the clinical gerontology course possess requisite skills for using Internet WebPages. It was further assumed that optometric students will be honest is their evaluation of the WebPage's value as a learning resource. Also, it was assumed that members of the formative committee have the knowledge and commitment to guide this study. In addition, it was assumed that the summative committee was competent to validate the criteria. Finally, it was assumed that the study is valid at the College of Optometry.

Limitations

The study of the evaluation of the value of a WebPage as a learning resource for optometric students who have completed a clinical gerontology course was limited by the fact that the evaluation instrument was designed specifically for administration to optometric students at Nova Southeastern University. It was also limited by the fact that the results are not generalizable because the data and information obtained for the study only pertained to the clinical gerontology course WebPage used at the College of Optometry.

RESULTSRESULTSAA

This study included the development of an evaluation questionnaire concerning the value of a WebPage as a learning resource for optometric students enrolled in a course in clinical gerontology. The questionnaire was distributed to 103 students enrolled in the course, and 102 students returned the completed questionnaires. Also, a focus group composed of six of the course participants was assembled to provide additional information for this study. The meeting was recorded and transcribed. Based upon the results of this study, the researcher would consider making modifications to the course's WebPage for the 1999-2000 academic year. This section discusses development of evaluation criteria, development of the questionnaire, administration of the focus group meeting, and results of the questionnaire and focus group meeting.

Development of Evaluation Criteria

Based upon a review of the literature, the following WebPage design criteria were established with the assistance of the formative committee:

•WebPages should be well organized and easy to navigate.

•Navigation buttons (links) should be consistent in appearance.

•WebPages should load quickly when accessed.

•Links to WebPages should always be active.

•The layout of WebPages should be clear and easy to visually

scan.

•WebPages should take advantage of the power of the World

Wide Web.

•WebPages should be useful to the end user.

•Students should possess adequate technology skills for using a

course's WebPage.

•Adequate technical support should be available to students

throughout the course.

RESULTSRESULTSBB

Development of the QuestionnaireA formative committee composed of four members was assembled to review and validate the criteria for the redesign of the course's WebPage. The final criteria selected for development of the questionnaire are listed in the previous section.

Based upon the criteria developed by the formative committee, a questionnaire was developed. The committee reviewed the draft questionnaire for form and content validity and made recommendations for modification of the statements. The committee's recommendations were included in the final draft. The final draft was distributed to the committee and approved.

Administration of the QuestionnaireThe questionnaires were distributed to the students at the final class meeting. They were asked to complete the questionnaires and return them to the class president on or before the date of the final examination. To eliminate the perception of course instructor bias toward the students, the president was asked to place the questionnaires in an interoffice envelope and to return it to the instructor after final grades were posted. As an incentive to respond to the questionnaires, the students were given an additional one-percent that was added to their final course grade.

The NSU Health Professions Office of Educational Support processed the answer forms. The office used ScanTools to scan the survey answer forms and the National Computer Systems (NCS) Survey to calculate and compile the results. Highlights from the questionnaires are discussed below.

The NSU Health Professions Office of Educational Support processed the answer forms. The office used Scantools to scan the survey answer forms and the National Computer Systems (NCS) Survey to calculate and compile the results. Highlights from the questionnaires are discussed below.

RESULTSRESULTSCC

Questionnaire

RESULTSRESULTSDD

Questionnaire DataThe results generated by NCS Survey were reported out by item and included the item number and statement; weight of each possible response (1=strongly agree, 2=agree; 3=no opinion; 4=disagree; and 5=strongly disagree); frequency of response for each possible response; and percent of participants responding to each possible response. The mean, standard deviation, and median were reported as well. Frequency data was used to analyze the students' responses to the statements made in the questionnaire.

The responses were categorized into strongly agree/agree and disagree/strongly disagree. The percentage of students responding to both of the categories was tabulated by item.

Table 1: Student Responses to Questionnaire

ITEM SA/A% D/SD%1 92.1 5.02 93.0 3.03 92.2 3.04 90.2 5.95 66.4 9.96 88.3 6.97 88.2 4.98 91.1 3.09 82.4 5.910 71.6 14.711 85.3 2.912 74.5 19.613 87.1 7.0

ITEM SA/A% D/SD%14 83.3 5.915 66.5 13.716 59.8 23.517 66.6 18.618 76.4 8.619 73.5 12.820 39.6 38.621 64.4 11.922 56.5 12.923 82.4 6.924 80.4 13.725 67.0 13.0

SA/A = Strongly Agree/AgreeD/SD = Disagree/Strongly Disagree

RESULTSRESULTSEE

To analyze the student responses, the items were placed into one of two categories, WebPage design and WebPage value.

Table 2: Classification of Test Items

CATEGORY TEST ITEMS

Design 1, 2, 3, 4, 7, 8, 23, 24, 25

Value 5, 6, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22

To assist in analyzing the resultant data, the items were sorted based upon positive (SA/A) and negative responses (D/SD).

Table 3: Sorted Data by Item and Percentage Positive Response (SA/A)

ITEM SA/A

2 93.03 92.21 92.18 91.14 90.26 88.37 88.213 87.111 85.314 83.39 82.423 82.424 80.4

ITEM SA/A

18 76.412 74.519 73.510 71.625 67.017 66.615 66.55 66.421 64.416 59.822 56.520 39.6

SA/A = Strongly Agree/Agree

RESULTSRESULTSFF

Table 4: Sorted Data by Item and Negative Response (D/SD)

ITEM D/SD

20 38.616 23.512 19.617 18.610 14.715 13.724 13.725 13.022 12.919 12.821 11.95 9.918 8.6

ITEM D/SD

13 7.06 6.923 6.94 5.99 5.914 5.91 5.07 4.92 3.03 3.08 3.011 2.9

The SA/A and D/SD responses were compared by item to determine which design criterion or WebPage component needed to be redesigned. The level of significant response was established as being below seventy-five percent for SA/A and above fifteen percent for D/SD. Those items in which the SA/A responses fell below seventy-five percent and the D/SD responses fell above fifteen percent were selected for discussion by the focus group. The level of significant response was determined intuitively and by consultation with the Educational Statistician and Director of Educational Support, Nova Southeastern University, Health Professions Division.

D/SD = Disagree/Strongly Disagree

RESULTSRESULTSGG

An analysis of the items based upon the above criteria indicated that four items required additional input from the focus group. All four items were value related.

Table 5: Items Selected for Focus Group Discussion

ITEM STATEMENT

12 The use of WebPages is valuable as a major learning resource for the course.

17 The Focus Papers page is a useful WebPage component.

16 The Study Groups page is a useful WebPage component.

20 The Study Group Chat Room is a useful WebPage component.

Administration of the Focus Group Meeting

Two weeks prior to the end of the course, an e-mail message was distributed to all students asking for volunteers to participate in a focus group. Six students responded that they were willing to participate. A meeting was held two and one-half months after the semester ended and lasted approximately 90 minutes. All six students signed a Research Study Consent Form at the beginning of the focus group meeting.

RESULTSRESULTSHH

Analysis of the Focus Group Meeting

Although many areas of the course were discussed during the focus group meeting, the analysis centered on the four items selected for focus group discussion (Item 12, 16, 17, and 20).

The students that participated in the focus group generally agreed that the WebPage was valuable as a major learning resource for the course (Item 12). A single dissenting student stated that the major problem with taking a WebBased course on the Internet is not being computer and Internet literate. Also, he stated that the students who disagreed with Item 12 probably were anti-Internet or anti-WebPage. The following are major

points brought up during the discussion.

• WebPages provide is easy access to differing points of view (bulletin board).

• WebPages are useful not just for this course but for all courses in the curriculum. • It's like having a library at your desk (incorporating reading assignments and WebPage links to online journals and WebSites around the world). • The WebPage links familiarizes you with Internet resources that are useful after graduation. • It's the future of education.

Another student suggested that an optional WebPage training course should be held prior to the beginning of the course.

Most students agreed that the Study Group page was valuable (Item 16). It allowed them to review who were the members of their work groups. However, one student dissented. He stated that he never visited the Study Group page after the first time because he knew which students were members of his assigned group.

RESULTSRESULTSII

There was a mixed review of the usefulness of the Focus Papers page (Item 17). One student thought that the students who disagreed with the statement were probably focusing in on the course requirement to write a group focus paper instead of concentrating on the value of the WebPage itself. Another participant stated that students probably were only interested in seeing their group paper published on the page and did not access the other papers as useful sources of information. Some participants though that the focus papers published on the page could serve as a useful resource for future students.

Except for one participant, the group felt that the Chat Room was not a useful WebPage component (Item 20). They stated that their work groups did not take advantage of this modality. Several participants stated that it was easier to get together face-to-face rather than on a computer. One participant said that his group did meet in a Chat Room, but most of the members were sitting side-by-side in the computer lab. They only met in a Chat Room because they thought that it was a course requirement.

DISCUSSIONDISCUSSIONAA

This study was undertaken to determine if students enrolled in a Web-based course in clinical gerontology found that the WebPage was a valuable learning resource. Based upon the results of a questionnaire completed by the students and the input provided by a focus group of six student volunteers, the course instructor was able to determine which WebPage features would need to be modified for delivery of the course in academic year 1999-2000.

The results of the questionnaire suggested that four items should be further evaluated for the need to revise the course WebPage. First, item 12 stated that "The use of WebPages is valuable as a major learning resource for the course." It received a significant negative response of SA/A equal to 74.5 percent and D/SD equal to 19.6 percent. The focus group disagreed with the responders to the questionnaire and generally agreed that the course WebPage was valuable as a major learning resource. However, one member brought up a very interesting point that students may be anti-Internet or anti-WebPage and may not be computer and Internet literate. Therefore, although the WebPage was useful to many students, it would be helpful to others to hold computer workshops during the summer prior to the beginning of the course. This would give students the opportunity to become more computer and Internet literate before having to tackle a Web-based course.

Second, item 16 stated that "The Study Groups page is a useful WebPage component." It received a significant negative response of SA/A equal to 59.8 percent and D/SD equal to 23.5 percent. The focus group generally disagreed with the responders to the questionnaire and stated that the Study Group page was a useful WebPage component. However, one student dissented. Although he did not find it valuable because he knew his study group members, others may not. Since the Study Group page was a reference page, it was valuable to a significant portion of the class (23.5 percent).

DISCUSSIONDISCUSSIONBB

Third, item 17 stated that "The Focus Papers page is a useful WebPage component." It received a significant negative response of SA/A equal to 66.6 percent and D/SD equal to 18.6 percent. The group offered several comments related to misperceptions of the true nature of the question being asked of the class. The intent of the Focus Papers page was to publish the titles of the focus papers when approved as well as to publish the papers for student review. However, since there was a large percentage of the class that thought the page was valuable, it should remain a part of the course’s WebPage.

Fourth, item 20 stated that "The Study Group Chat Room is a useful WebPage component." It received a significant negative response of SA/A equal to 39.6 percent and D/SD equal to 38.6 percent. The focus group members agreed with the respondents to the questionnaire and offered several strong statements as to the Chat Room’s value. This researcher agrees with the students in that it is easier for students in residence at the College to get together face-to-face for study group meetings than to use a computer for real time collaboration.

CONCLUSIONSCONCLUSIONSThis study led to the development of evaluation criteria by which to assess the value of the course’s WebPage as a learning resource. Based on these criteria, a questionnaire was developed, validated, and administered to third year optometric students who completed the course. From the results of the questionnaire and input from the focus group, it was determined that the WebPage needed to be modified for academic year 1999-2000.

There was one research question for this study: Is the WebPage a valuable learning resource for optometric students completing a clinical gerontology course? Through the procedures outlined in this study the research question was answered. Overall, the WebPage was a valuable learning resource for the students. However, two modifications needed to be made to the WebPage.

First, a training workshop needs to be offered during the summer prior to the beginning of the course that would place all students on a level playing field. That is, they would all have the necessary skills for using a computer to access the Internet, to use the bulletin board for communication, and to send and receive e-mail with attachments for correspondence with the instructor and fellow students both within and outside their study groups.

Second, the Chat Rooms need to be eliminated. They serve no useful purpose for residential students nor do they offer a meaningful advantage for meeting the objective of the course.

IMPLICATIONSIMPLICATIONS

Because optometric students are not sufficiently educated and trained to provide for the special needs of the aging population, during the 1998-1999 academic year, the College of Optometry curriculum contained a clinical gerontology course that focused on the special needs of the elderly. For optometric students to learn to think critically and communicate appropriately and clearly, the course was delivered in a format that supported active learning, problem solving, and written communication. It assisted optometric students to access learning resources at a time and place that was convenient for them. This resource also supported the students’ access to a variety of course tools and activities that included e-mail, a bulletin board, electronic submission and critique of work, electronic posting of grades, a class syllabus, notices, schedules, and an electronic exhibit areas for class projects.

The development of an Internet WebPage as a learning resource for optometric students and the employment of critical thinking by the students taking the course were important to the College in meeting its goal of sustaining a dynamic environment that nurtures intellectual inquiry through the incorporation of modern educational resources using innovative delivery systems and state-of-the-art technology. Because of this, evaluation of the WebPage following its first offering was important to the College so that future students would perceive Web-based instruction as valuable to their professional and post-graduate education and development.

In closing the gaps between an ideal state and the actual state of the WebPage for the clinical gerontology course, the value of the WebPage should increase for optometric students. The use of this evaluation method in other courses can result in the development and improvement of appropriate learning resources for the College of Optometry.

RECOMMENDATIONSRECOMMENDATIONSTwo recommendations were made from the results of this study. First, a training session should be held next summer prior to the beginning of the course in the use of a computer for accessing the Internet, for using the bulletin board, and for sending and receiving e-mail with attachments. Since the students are assigned to one of two five-week summer sessions in the clinic that run consecutively and are off campus for the other five week session, a training workshop needs to be offered in both June and July. Due to the size of the class (50 students per session), the sessions should be held in the University Park Plaza computer lab. This laboratory has the capacity to hold a workshop for up to 100 students. Also, to provide ongoing training and support for the students, the course instructor needs to be available and easily accessible through the duration of the course for this purpose.

Second, since the class and instructor agreed that the Chat Rooms were not a valuable part of the WebPage, they should be eliminated for next year’s course. However, if the course is offered in the future to students from other colleges, the Chat Rooms would be quite valuable for collaboration among the on- and off-campus students and should be retained.

The results of this study should be disseminated to other members of the optometric education community. First, the study will be presented as a poster in Seattle, Washington on December 16, 1999 at the annual meeting of the American Academy of Optometry (AAO). Second, this poster will be displayed at the College in the faculty office area after the AAO meeting. It is this researcher’s intention that optometric educators will consider using a similar methodology for evaluating their courses and learning resources.

It is also recommended that further studies be conducted in regard to students’ use of technology and WebPages within the optometric education enterprise. One study should determine students’ entering and exiting capabilities in the use of computers, the Internet, bulletin boards, and e-mail. Another study should evaluate students’ perceptions of the value of information technology within optometric education.

REFERENCESREFERENCES1. Kemp JE, Morrison GR, Ross SM. Designing effective instruction. 2nd ed. Upper Saddle River, New Jersey: Prentice-Hall, Inc., 1998.

2. Williams AE. Integrating courses with the Internet: Preparing the teacher as well as the learner. In: Association of Small Computer Users in Education Summer Conference. North Myrtle Beach, South Carolina: Association of Small Computer Users, 1996.

3. Relan A, Gillani BB. Web-based instruction and the traditional classroom: Similarities and differences. In: Khan BH, ed. Web-based instruction. Educational Technologies, Inc.: Englewood Cliffs, NJ, 1997.

4. Bostock S.J., Designing Web-based instruction for active learning. In: Khan BH, ed. Web-based instruction. Educational Technology Publications, Inc.: Englewood Cliffs, NJ, 1997.

5. Campbell JO, Bourne, J. Designing distributed learning systems. Journal of Computing in Higher Education, 1997.

6. Abels EG, White MD, Hahn K. Identifying user-based criteria for Web pages. Internet Research: Electronic Networking Applications and Policy, 1997.

7. Berger RM, Patchner MA. Planning for research. Newbury Park, CA: SAGE Publications, Inc. , 1988.

8. McMillan JH, Schumacher S. Research in education: A conceptual introduction. New York, NY: Addison Wesley Longman, Inc., 1997.

9. Morgan DL. Focus groups as qualitative research. New Park, CA: SAGE Publications, Inc., 1997.

COMMENTS PLEASE…..COMMENTS PLEASE…..

For additional information or dialogue, please correspond with the author by e-mail at scottg@hpd.nova.edu or by telephone at (954) 262-1430.