Embed Size (px)
Transcript of Elearning evalfinal
Evaluation of the Current Learning Management System at JMUThe Students Perspective on BlackboardPresented ByEric JacobsMichael MayJason PooleJessica Robinson
IntroductionReview of Evaluation ProcessBackgroundIntroductionThis evaluation plan was prepared by Eric Jacobs, Michael May, Jason Poole, and Jessica Robinson who are graduate students in the Educational Technology department at James Madison University. Coordinating efforts by Andrea Adams of the Center for Instructional Technology (CIT), and by Dr. Michele Estes of the Educational Technology department at James Madison University have helped advise this evaluation plan.Blackboard is a learning management system that has the ability to act as the central hub for the curriculum. This has become increasingly apparent as distance learning programs depend on the features of this evaluand to manage course content and interact with students in a web-based environment. Blackboards diffusion wide, claiming more than 1,600 institutions using their LMS (Blackboard, 2011). Purpose of evaluationReview of Evaluation ProcessStakeholdersPurposeTo determine the needs that students have for learning management system.
To identify the technologies that faculty and students are currently using in teaching and learning
To identify the technologies that faculty and students are no long using in teaching and learning.Primary StakeholdersAndrea AdamsDr. Michele Estes
Secondary StakeholdersEvaluation teamStudents of JMUPurpose of evaluationReview of Evaluation ProcessDecisions to be addressedTo what degree does Blackboard meet the expectations and learning needs of undergraduate and graduate students at JMU?
Do graduate and undergraduate students feel that Blackboard is a viable solution as a learning management system for JMU over the next 5 years?
To what degree do the features in Blackboard meet the technological needs of the students at JMU?Evaluation questionsReview of Evaluation ProcessNeeds relatedWhat are students needs in regards to using a learning management system in their coursework?
What expectations do students have about learning management systems?
What technologies are students using both in Blackboard and in their day to day lives?
Do students feel that the technology in Blackboard is beneficial to their learning?
What features of Blackboard do students find obsolete or in need of updating?
What features in Blackboard do students engage with most often?
What features in Blackboard do students feel are the most beneficial?Technology relatedFeatures relatedMethods of evaluationReview of Evaluation ProcessMethodsCross-sectional SurveyCensusInterviewThe SampleReview of Evaluation Process2 undergraduate2 graduatePurposive sample of Ed Tech Students ~20,000 studentsExpected return rate ~2%Sample for interviewsSample for census survey
Students in high Blackboard use courses2,000 students or 10% of JMU enrollmentExpected return rate 50%Sample for census survey
InstrumentsReview of Evaluation ProcessCreated and administered using Qualtrics.Recorded using Elluminate Live!SurveysInterviews
Evaluation planReview of Evaluation ProcessLimitationsTime ConstraintsReturn rate of survey dataProximity of group members from JMU and each other
Logistics & TimelineBudgetInterviews and survey data completed in November
InterviewSummary #1Interviewee DataGender: Female
Program of Study: Communications Major, Educational Media Minor
General Info: Student has taken hybrid courses and is currently enrolled in an online course. Experience using Blackboard in high school.
+-PositivesNegatives Access to gradesThe more information on Blackboard the betterE-mails of announcementsOrganization of contentDiscussion boardsAnnouncements when viewed on BlackboardInstructors that dont update Blackboard beyond syllabus
InterviewSummary #2Interviewee DataGender: Female
Program of Study: Educational Technology
General Info: All classes were taken on Blackboard.
+-PositivesNegatives Internal Blackboard email toolsAccess to everything in central locationDislike wiki feature
InterviewSummary #3Interviewee DataGender: Female
Program of Study: Middle School Education
General Info: Overall positive experience with Blackboard
+-PositivesNegatives All important documents in one placeOnline grade bookRoster for contacting classmatesLots of features she doesnt know how to useInconsistency among JMU facultyInterviewSummary #4Interviewee DataGender: Female
Program of Study: Educational Technology
General Info: Overall positive experience using Blackboard. Almost all courses using Blackboard.
+-PositivesNegatives Blackboard and Elluminate are perfect coupling Greater connection with course and instructor E-mail features, discussion board, grade featureLinks and ResourcesInstructor implementationInstructor UsageCollaboration -- Real-time lackingGroup sharing is clunkySynchronous info sharing (IM, chat)
GenderCensus Survey Data
#AnswerResponse%1Male5727%2Female15173%Total208100%1. Please select one: Male or FemaleEducationalStandingCensus Survey Data
#AnswerResponse%1Freshman5125%2Sophomore2713%3Junior4120%4Senior4823%5Graduate Student4120%Total208100%2. Please select one: Specific Educational StandingCensus SurveyData
Census Survey Data#AnswerResponse%1College of Buisness2512%2College of Communications147%3College of Education2914%4College of Health and Human Services3115%5College of Humanities and Social Sciences4421%6College of Science, Technology, Engineering and Mathematics4723%7College of Visual and Performing Arts84%8Undecided105%Total208100%3. Please select one: Program of StudyCourse interactions with BlackboardCensus Survey Data
#AnswerResponse%1I have taken a fully online course that used Blackboard5727%2I have taken a hybrid/blended course that meets online and in the classroom that used Blackboard5426%3I have taken face-to-face classroom courses that utilized Blackboard19192%4I have taken face-to-face classroom courses that do not utilize Blackboard12560%4. Based on your course work at JMU, please describe your course interactions with Blackboard.CommunicationToolsCensus Survey Data
#AnswerResponse%1social networks17785%2text messaging19694%3instant messaging5325%4email20096%5other199%5. Which technologies play a role in your daily communications?Regularly used technologiesCensus Survey Data
#AnswerResponse%1smartphone10955%2tablet147%3laptop/desktop19498%4Other2915%6. What technologies do you regularly use?Would like social features in BlackboardCensus Survey Data
#AnswerResponse%1yes6432%2no5729%3unsure7639%Total197100%7. Would you like to see Blackboard have more social features for learning?Blackboard fits expectationsCensus Survey Data
#AnswerResponse%1Strongly Disagree42%2Disagree116%3Neither Agree nor Disagree4623%4Agree11458%5Strongly Agree2211%Total197100%8. Blackboard fits my expectations of how a learning management system should work.Overall experience with BlackboardCensus Survey Data
#AnswerResponse%1negative32%2somewhat negative158%3neutral3015%4somewhat positive8945%5positive6030%Total197100%9. Overall, how would you describe your experience using Blackboard?How difficult is BlackboardCensus Survey Data
#AnswerResponse%1Very Difficult21%2Somewhat Difficult2010%3Somewhat Easy11357%4Very Easy6231%Total197100%10. Overall, how difficult is Blackboard to use?Satisfaction with Blackboard reliabilityCensus Survey Data
#AnswerResponse%1Very Dissatisfied21%2Dissatisfied137%3Neutral3317%4Satisfied12262%5Very Satisfied2613%Total196100%11. Please describe your satisfaction level of Blackboard as a reliable service.Difficulty locating info in BlackboardCensus Survey Data
#AnswerResponse%1Very Difficult21%2Difficult2412%3Neutral6232%4Easy9046%5Very Easy189%Total196100%12. Please describe the difficulty in locating information within Blackboard.How often do you access BlackboardCensus Survey Data
#AnswerResponse%1Never00%2Less than Once a Month00%3Once a Month32%42-3 Times a Month32%5Once a Week116%62-3 Times a Week7237%7Daily10755%Total196100%13. How often do you access Blackboard?Importance of featuresCensus Survey Data
#QuestionNot Very ImportantNot ImportantImportantVery ImportantResponsesMean1Email161380781873.182Course Annoucements25641161873.573Social Media51864281872.0414. Please indicate the importance of the listed features for a learning management system Importance of featuresCensus Survey Data
#QuestionNot Very ImportantNot ImportantImportantVery ImportantResponsesMean1Discussion Boards163394441872.892Blogs341014751872.123Wiki41964281872.0915. Please indicate the importance of the listed features for a learning management system Importance of featuresCensus Survey Data16. Please indicate the importance of the listed features for a learning management system
#QuestionNot Very ImportantNot ImportantImportantVery ImportantResponsesMean1My Grades21221621873.842Test518581061873.423Rubrics and Learning Outcomes110591171873.56Importance of featuresCensus Survey Data
#QuestionNot Very ImportantNot ImportantImportantVery ImportantResponsesMean1Assignments21411431873.742Calendar63372761873.173Groups135290321872.754Journals267957251872.435Learning Module166176341872.686Survey227266271872.527Podcasts368352151862.2517. Please indicate the importance of the listed features for a learning management systemMost often accessed featuresCensus Survey Data
18. Which features do you access MOST often in Blackboard?Most often accessed featuresCensus Survey Data
18. Which features do you access MOST often in Blackboard?Least often accessed featuresCensus Survey Data19. Which features do you access LEAST often in Blackboard?
Least often accessed featuresCensus Survey Data19. Which features do you access LEAST often in Blackboard?Importance of featuresCensus Survey Data
20. Blackboard meets my needs in my current courses.#AnswerResponse%1Strongly Disagree63%2Disagree137%3Agree10858%4Strongly Agree5932%Total186100%CommentsCensus Survey DataOpen comments section Received 35 comments
16 comments directly related to instructor knowledge and implementation of Blackboard
8 comments related to amount of tools available vs. the few tools used.
Other less frequent commentsSmartphone supportInterface or navigation confusionDislike changes from previous versionsCommentsCensus Survey DataExample commentsI think that blackboard does not need to be overloaded with features. From my experience, most of the features in blackboard are not used because they are not very important for a classroom setting. Blackboard is useful because it allows an instructor to communicate with his or her students and students to communicate with their teacher. There is no need to improve blackboard or add more features to it because it already does what it does well. In fact, I find that having too many features makes programs stressful and difficult to use because they demand too much of users. I don't want a comprehensive life planner and entertainment center with blackboard. It already does what I ask of it: It provides a way for my teachers to communicate with me about upcoming assignments and grades. It does this very well.CommentsCensus Survey DataExample commentsIn my experience, use of blackboard as a learning management tool largely depends upon the instructor. Only a few of my classes have used discussion board and only one has used Elluminate Live! For most classes, blackboard serves as a way to distribute class documents, announcements, and contact information. Therefore, most of my use of blackboard has been with simple tools, which is really all I need blackboard for.Some features are redundant and confusing, such as Journals and Online Journal. There is a lack of uniformity between how different professors use different tools. It's very clunky and clumsy. I wish it had a more seamless user interface, like facebook. Conclusions &RecommendationsRecommendationsConclusionsStudents are overall pleased with Blackboard as a tool.
Many of Blackboards features are unused in courses.
Students feel instructors us of Blackboard is primary to its success in courses.Option 1Keep Blackboard as JMUs learning management system and focus attention on course implementation and faculty training.
Option 2Investigate alternative tools that better implement the functions most used by instructors in their courses.Reflections of evaluation team+-PositivesNegativesRecording interviews using Elluminate
Data collection using QualtricsMass student email process
Timing of survey
Evaluators distance from JMU campus