WHITE PAPER

CriticalEvaluationofQualityCriteriaandQualityInstrumentsinOERRepositoriesforthe

EncouragementofEffectiveTeacherEngagement

DRMARINACONNELLJOHNCONNELL

JANUARY2020

1

Table of Contents

ABSTRACT .............................................................................................................. 2KEYWORDS...........................................................................................................................................................2ACKNOWLEDGEMENTS .......................................................................................... 2

INTRODUCTION ...................................................................................................... 3DEFINITIONOFTERMS...................................................................................................................................3PURPOSEOFTHEWHITEPAPER...............................................................................................................4BACKGROUNDANDTHEORETICALAPPROACHOFTHISREVIEW.............................................5METHODOLOGY..................................................................................................................................................6INTENDEDBENEFITSOFTHISREVIEW.................................................................................................7OVERVIEW OF LEARNING RESOURCE QUALITY CRITERIA AND FEEDBACK SYSTEMS 9TAXONOMIESOFQUALITYINRELATIONTOUSINGOER..............................................................9THEIMPORTANCEOFQUALITYTOTHEUPTAKEOFOER.........................................................14EVALUATINGQUALITYINOER................................................................................................................15RANKINGSYSTEMS&COLLECTIVEINTELLIGENCE......................................................................17EXAMPLESOFREPOSITORIESWITHEFFECTIVEQUALITYSYSTEMS...................................18KlasCement.........................................................................................................................................................18Curriki...................................................................................................................................................................20MERLOT...............................................................................................................................................................21FurtherDiscussion..........................................................................................................................................23

THEQUALITYCHALLENGES:ThreeShortCaseStudies................................................................25CaseStudy1:Scientix.....................................................................................................................................25CaseStudy2:GeoGebra.................................................................................................................................27CaseStudy3:Photodentro...........................................................................................................................30

DIFFERENTAPPROACHESTOQUESTIONSOFQUALITYINOER.............................................31CONCLUSION: A CRITICAL EVALUATION OF QUALITY CRITERIA ............................. 32

APPENDIX: ACTIONABLE POINTS & RECOMMENDATIONS .................................... 36

BIBLIOGRAPHY ..................................................................................................... 38

2

ABSTRACT ThispaperoffersashortevaluationofthevarietyofqualitycriteriausedinOpenEducationalResourcesandsomeofthemethodsandpracticesinusetoensurequality.Thepapersurveysandreviewseffectivepracticesandtechnologiesusedinlearningobjectrepositoriestomoreeffectivelyengageeducatorsinprovidingstructuredfeedback.Theaimistoprovideausefulresourcetorepositoryownersandmanagersseekingtodevelopandimproveontheservicestheyalreadyoffertoeducators.AnimportantadditionalaimistosupportteachersinusingdigitallearningresourcesintheclassroomsothatthedeploymentofOERinschoolsinparticularmightgrowandflourishoverthenextfewyears.‘Quality’isbynomeansasimpleconcepttodefineinthecontextofOERrepositoriesandthepaperseekstolayoutanumberofdifferentwaysinwhichtheconceptcanbeanalysedandmadeuseful.Thepaperwasproducedthroughashort-termqualitativeanalysisinwhichweexaminedanumberofexistingOERrepositories,awiderangeofacademicandotherwritingsonthesubject,asmallnumberofinterviewswithkeyexpertsandsomepublishedslidewarerelevanttothesubject.Thepaperoffersasetofpotentialfurtherresearchquestionsaswellasadistillationofanumberof‘actionablepoints’thatOERleadersmightbeabletouseasstartingpointsforongoingdevelopmentworkontheirownqualityassurancesystems.

KEYWORDS Quality, Quality Criteria, Quality Circle, Collective Intelligence, Evaluation, OER, Repository

ACKNOWLEDGEMENTS Anumberofpeoplegaveustheirkindandgenerousassistancewithourtaskinresearchingandwritingthiswhitepaper.WearethankfultoallofthemfortheirhelpinproducingwhatwehopewillproveausefulpapertotheOERcommunityinEuropeandbeyond:DavidMassart,ElenaShulman,HansDeFour,BramFaems,ElinaMegalou,JoeWilson,PaulBacsich,MarthaZellinger,NeilButcher,AndrewMoore,FernandoCampos,NellyBuchser Anydeficienciesinthefinalpaperremaintheresponsibilityoftheauthorsalone.

3

INTRODUCTION TheOpenEducationmovementhasgainedsubstantialtractionsincethetermOpenEducationalResources(OER)wascoinedin2000.However,thereremainsmuchscopeforfurtheradvocacyandpromotionofOpenEducationgenerallyandoftheprinciplesandvaluesthattheconceptembodies.OpenEducationisabroadcanvasthatisabletoaccommodatearangeofunderstandingsoftheterm.Itisalsoatermthatgathersanarrayofdifferentelementsbeneathitsumbrella,ofwhichOERisone,althoughonethatismuchdiscussed.OERsaregenerallystoredinaLearningObjectRepository(LOR).

DEFINITION OF TERMS

Intermsoftheunderstandingof‘open’inOER,thispapertakesasastartingpointthestatementfromDavidWileythat:

“Openisacontinuous,notbinary,construct.”1

Inotherwords,therecanbenofixeddefinitionof‘open’inthecontextofOER.Insteaditshouldbeinterpretedinrelativeterms–isarepository,forexample‘openenough’tobedefinedasofferingOER,orisonerepository‘moreopen’or‘lessopen’thananother?Itshouldbeacknowledged,however,thatmanyofthosewhoworkinthefieldprefertoacceptmorefixedandprescriptivedefinitionsofopenness.‘Open’,inthesenseusedinthisreport,relatestotheextenttowhichusersaregrantedrightstousearepositoryoraresourceinrelationtothe5Rsofopeneducationalresources,namely:

Retain-therighttomake,own,andcontrolcopiesofthecontent(e.g.,download,duplicate,store,andmanage)

Reuse-therighttousethecontentinawiderangeofways(e.g.,inaclass,inastudygroup,onawebsite,inavideo)

Revise-therighttoadapt,adjust,modify,oralterthecontentitself(e.g.,translatethecontentintoanotherlanguage)

Remix-therighttocombinetheoriginalorrevisedcontentwithothermaterialtocreatesomethingnew(e.g.,incorporatethecontentintoamashup)

Redistribute-therighttosharecopiesoftheoriginalcontent,yourrevisions,oryourremixeswithothers(e.g.,giveacopyofthecontenttoafriend)2

1https://opencontent.org/blog/archives/11232http://opencontent.org/definition/

4

Themorethattherightsattachedtoaresourceorarepositorymeetthe5Rs,themoreitcanbejudgedtobe‘open’.The5RsalsoofferareasonablefoundationforanexplorationofqualityinrelationtoOERandopenrepositoriesgenerally.Theoft-accepteddefinitionofOERisthatdevelopedbytheTheWilliamandFloraHewlettFoundation,derivedfromanumberofprecedingdefinitions:

“OpenEducationalResourcesareteaching,learningandresearchmaterialsinanymedium–digitalorotherwise–thatresideinthepublicdomainorhavebeenreleasedunderanopenlicensethatpermitsno-costaccess,use,adaptationandredistributionbyotherswithnoorlimitedrestrictions".3

ThisdefinitionstatesthatOERcanincludebothdigitalandnon-digitalresources.Ashorter,sharperdefinitioncomesfromOECD,butonethatfocusesondigitalresourcesinparticular:

“Digitisedmaterialsofferedfreelyandopenlyforeducators,studentsandself-learnerstouseandre-useforteaching,learningandresearch.”4

Thispaperwilladdressitselfmainlytodigitalresources,althoughtheissuesdealtwithpertainbroadlytonon-digitalresourcesaswell.PURPOSE OF THE WHITE PAPER Intheirpaper,OpenEducationalResourcesPlatformBasedonCollectiveIntelligence,theauthors,DeOliveiraetal5,setoutinsuccincttermstheissuethatliesattheheartofthiswhitepaper.

“AnOERischaracterizedbyitsmetadata….TheOERsarestoredinrepositories.ThepurposeofOERrepositoriesistosupporteducatorsinsearchingforcontentinastructuredway,sharingtheirownresources,reusingexistingmaterialsandcreatingnewresourcesthroughadaptingortranslating,andincollaboratingwithothermembersoftheusercommunitybycommentingupon,reviewing,promotinganddevelopingresources.”

EducatorssearchingforOERbyandlargemakeuseofrepositoriestosearchfor,locate,download,use,re-useandadaptthelearningresourcestheyneedfortheirteaching.Itwouldseemtomakesensethateffectiverepositorieswillmakethisprocessassimpleandaseasy-to-useaspossible.Theyshouldalsoseektogiveteachersmechanismsthatwillallowthemtoprovideprofessionalfeedbackonthevalueandusefulnessoftheresourcesoncetheyhavedeployedthemintheirteaching.Bythismeans,andothers,therepositorywillthenbeabletoofferteacherssomeinformationbywhichtheyareabletopre-determinetosomeextentthequalityoftheresourcestheyselect.

3https://hewlett.org/strategy/open-educational-resources/TheWilliamandFloraHewlettFoundation. Retrieved27March20134http://www.oecd.org/education/ceri/38654317.pdfIschinger,BOECDGivingKnowledgeforFree,p.105https://ieeexplore.ieee.org/document/8537851deOliveira,etalOpenEducationalResourcesPlatformBasedonCollectiveIntelligencep.1

5

Thefocusofthiswhitepaper,therefore,istoundertakeashortevaluationofqualityissuesinrelationtoOER,withafocusontheK12sectorwherethatispossible,andtoexamineanytechniquesthatcanbeidentified,eitherinproposalsbyeducatorsandacademicsorinactualuseinOERrepositories,thatmightsupporteducatorsinintegratingdigitallearningresourcesintotheirteachingpractice.MuchoftheliteratureandresearchintoOpenEducationalPractices(OEP)generally,andintoOERinparticular,hasbeenproducedandcarriedoutfromahighereducation(HE)ortechnical&vocationaleducation&training(TVET)perspective,and,todate,withafewhonourableexceptions,lesssofromtheperspectiveofteachersandstudentsintheschoolssector.Forwhateverreasons,thereseemstohavebeenanidentifiablylowerlevelofengagementwithOpenEducationfromwithintheK12sectorgenerallyovermanyyears.Ofcourse,manyoftheOpenEducationlessonslearntandappliedwithinHEandTVETwillapplyjustasmuchtoOEPinschools,butweshouldbecautiouslyawarethattheirapplicationinK12mightraisesomecontextuallyspecificissuestoo.

BACKGROUND AND THEORETICAL APPROACH OF THIS REVIEW ThiswhitepaperhasbeencommissionedbytheLearningResourceExchange(LRE)SubcommitteeofEuropeanSchoolnet.TheLRESubcommitteeisagroupofEuropeanMinistriesofEducationandotherorganisationsthatpromotesandsupportsthecreation,discovery,exchange,and(re-)useofOERandOEP.TheLRESubcommitteefeelsthatthetimeisrighttosurveyandreviewbestpracticesandtechnologiesused,orproposedforuse,bylearningrepositoriestomoreeffectivelyengageeducatorsinthetermsalreadydescribedabove.Thepaperprovidesareviewandasmallnumberofrecommendationsonqualitycriteria(eitheralreadyinuseorbasedoncurrentpedagogicalresearchliterature)anduser-feedbacksystemsthataremosteffectiveinclosingthegapbetweenOERs,inproposalsorinrepositories,andtheiractualuseinclassrooms.Thepaperattemptstodescribeestablishedgoodpracticesforeducationalrepositorieswithregardtoratingandevaluativesystemsandfeedbackmechanismsbasedonprofessionaljudgementoronwell-foundedqualitycriteria,withafocusonsystemsthatcaptureuserfeedbackwithinOERrepositoriesthemselves.Toachievetheseaims,thepaperwasrequiredtoencompassfivekeydeliverables:

• Abriefoverviewoflearningresourcequalitycriteriaandrankingsystems• Keyexamplesofimplementationsoftheserankingsystemsthatcapture

userfeedbackonOERs• ChallengesfacedbyK12educationalrepositoriesthathaveimplemented

rankingsystemsbasedonuserfeedbackandthesolutionsadoptedbyrepositoriesthataddresschallenges;

6

• Anevaluationoftheeffectivenessofdifferenttypesofqualitycriteriaforstructuringfeedbackandrankingsystemsthatsupportteachersinusingdigitallearningresourcesintheclassroom;

• Acomprehensivebibliographyreferencingrelevantreadingsandstudiesthatcanbeusedinselecting,developingorimplementingqualitycriteriaandrankingsystemsthateffectivelyexploituserfeedbackforevaluatingandusingOERs.

Someofthedeliverablesweremoresuccessfullyachievedthanothers.Inparticular,arathersmallnumberofOERrepositorieswereidentifiedthatseemtohaveimplementedeffectivesystemsforevaluatingqualityorforrankingresourcesbyquality,andfewalsothatsystematicallyinvolvedusersinfeedingbackonthequalityoftheresourcestheyfindordeploy.However,withinthatsmallsample,thereweresomeexamplesofgoodpractice.

METHODOLOGY Giventheshorttimescaleavailabletoproducethispaper,theinformationanddatawereexaminedandprocessedonthebasisofthetype,goalsanddeliverablessetoutabove.Inconductingthereview,weusedthefollowingtechniques:

a) Contentanalysisb) Small-scalesurveyingofkeyrepositoriesandindividualsc) Deskresearchd) Interviewsbyemailandbyaudio

Theresearchisexclusivelyqualitativeinnature,usinganexploratoryapproachtoanexistingrangeofacademicandotherworkinthefield.Afewkeyinsightsweregainedfromextensivereadingofthematerialfoundandfromasubsequentattempttosynthesizeandsummarizethethinkingaroundquestionsofqualityinrelationtolearningresources.Anumberofworkingrepositorieswereidentifiedandasmanyofthemascouldbeidentifiedwithinthetimeavailablewereexamined.Theexaminationwasalsocarriedoutwithintheconstraintssometimesimposedbytherepositoriesthemselveswithregardtoauthenticationandease-of-access.Itwashopedtogainafeelforthoserepositories,ifany,thathavesoughttoestablisheffectivepracticeintheuseofqualitycriteriaandrankingsystems.Averysmallnumberofkeyexpertswereidentified(fromourownknowledgeandfromourreadingoftheliterature)whoitwasfeltcouldofferinsightsbasedontheirgreaterexperienceofthefield,and‘interviews’wereconductedbyemailorbyphone.Witheachpartofourapproach,theaimwastofocusinparticularonthesituationinrelationtoK12,thisdespiteanapparentdearthofmaterialandpracticerelatingtotheschoolssectorinOpenEducation.SomegoodpracticeintheK12sectorwasidentifiedandthishasbeenhighlightedinthepaper.

7

However,itwasnecessarytodependtosomeextentonthinking,writingandpracticeemanatingfromHEandTVETtoo.Theinformationgatheredforthereportrangesacrossabroadspectrumofsources,includingsomewell-known(andsomeless-well-known)academicpapers,educationalblogsandwebsites,socialmediagroups(includingFacebook,Youtubeandothers),whitepapersandavarietyofreports.Throughouttheprocess,theinvestigationsoughttoaddresstherequirementsofteachersinrelationtotheiractualneedsintheclassroomandtoavoidwherepossibleanyemphasisontop-down,centralized,pre-determinedassertionsofqualityforresources.Tothisextent,itishopedthatthelongertermneedsofrealeducators,especiallyintheschoolssector,willbemetbythoserepositoriesthatendeavourtogiveteachersthemselvesinfluenceoverthedeterminationofcontentquality.Theaimistoprovidesomesemblanceofawayforwardforthosewhoseektomakefurtherprogressinthisarea.Thereportalsofocusesontheassumptionthat,nomatterhowmuchqualitycanbeascribedtoanysingleresourceortoanycollectionofresources,whetheropenorotherwise,ifthoseresourcesdonothaveapositiveimpactonstudents’learning,theyareultimatelyofnoeducationalvalue.Thisis,however,anassumptionthatisitselfsomewhatproblematicwhenweconsiderOERinrelationtotheneedsofteachersandtheirteachingpracticeaswellasinrelationtothelearningthatfollowsfromthatteaching.Havingperusedthesubsetoftheidentifiedliterature,itisdifficultnotagree,ingeneral,withtheviewofPaulBacsich,BarryPhillipsandSaraFrankBristowintheirFinalReportonLearnerUseofOnlineEducationalResourcesforLearning,that:

“Theliteratureonlearneruseofonlineeducationalresourcesisveryimmature,withalackofmeta-reviews.Theoverwhelmingmajorityofpublishedstudiesdonotgeneralisebeyondtheirparticularcontextsofstudy.Thereisnoconsistentmethodology.”6

Theirfindingextendsalongwayintotheliteratureonqualitycriteriaandfeedbacksystemsinlearningresources,whethertheyareopenorotherwise,andthatisperhapsevenmorethecasewhenthefocusisonthesequestionsisinrelationtoK12.Ofcourse,thosewhohavealreadycontributedovertheyearstotheliteratureonOERhavedonesoextensivelyandinsightfully.However,thefieldisstillyoungandneedstogrowandmatureoverthenextfewyears,buildingontheverygoodworkalreadydone.INTENDED BENEFITS OF THIS REVIEW Inashortreportsuchasthis,itisdifficulttocaptureanythingmorethananimpressionofthestateofplaywithregardtothequestionsathand.However,it

6https://www.advance-he.ac.uk/knowledge-hub/learner-use-online-educational-resources-learning-luoerl-final-reportBacsich,P;Phillips,B;Bristow,SFLearnerUseofOnlineEducationalResourcesforLearningP.3

8

ishopedthatthethoughtsgatheredherewillbeabletoactasaspurtodeeperandbroaderworkinfuture.AdditionalresearchwillundoubtedlyberequiredtopromoteandexpandontheeffortsthathavealreadybeenmadetoensurethateducatorsarebetterabletoselectandintegratedigitalresourcesintotheirteachingpracticeacrossEuropeandbeyond.Ifthepaperleadstofurtherresearchintoquestionsraisedhere,thatwillberegardedassuccess.

9

OVERVIEW OF LEARNING RESOURCE QUALITY CRITERIA AND FEEDBACK SYSTEMS

TAXONOMIES OF QUALITY IN RELATION TO USING OER

Anyattempttodefinequalityinrelationtoeducationalresources,whetheropenornot,isfraughtwithdifficulty.Qualityisbothsubjectiveandheavilycontext-specific,andattemptingtosetouttheparametersofqualityinrelationtoOERoftenleadstosubjectivelayerbuiltuponsubjectivelayer,witheachlevelofdefinitionprovingdifficultorimpossibletomeasureinanysimpleway.Thereisalsothesimpleassertionthattherecan,ultimately,benoinherentorintrinsicsenseof‘quality’withinaresourceotherthaninaparticularcontextwhereithastoservespecificneedsforeducators,studentsorself-learners.Itmustimpactonthelearninginapositiveway.A‘good’resourceinonecontextmightbea‘poor’resourceinanother–itdependsupontheusetowhichitisputandforwhosebenefit.

AsDavidWileysosuccinctlydescribestheissue:

“…thecoreissueindeterminingthequalityofanyeducationalresourceisthedegreetowhichitsupportslearning.Butconfusingly,that’snotwhatpeoplemeanwhentheysaythatatextbookorothereducationalresourceis“highquality.”….thedegreetowhichitsupportslearningistheonlycharacteristicofaneducationalresourcethatmatters.”7

However,Wiley’scontention,thoughafundamentaltruth,isnotenough.Itistoosimplisticandittakeslittleaccountoftherealitiesofateacher’sneeds.Takingaccountofthebroaderresponsibilitiesrequiredinteachingandlearning,itisimportanttokeepinmindthatteachers,beforeteachingorlearningeventakesplace,havetofindlearningresourcesthatareappropriatetotheirteachingrequirements,thatarereadilysearchableandavailable,andthat,wherepossible,alreadyhavesomeperceivedleveloftrustworthinessbecausetheresourcehaspreviouslybeencuratedorhasprovedusefultoothereducators.SowhileWiley’sviewiscentraltoanyconsiderationofquality,sotooaretheneedsofteachersastheyseekout‘goodquality’learningresourcesinplanningtheirteachingandtheirstudents’learning.Ofcourse,teacherswanttheirlessonsandtheresourcestheydeploytohaveapositiveimpactonstudents’learning,buttheycannotknowwhetherthatwillbethecaseuntilaftertheteachinghasbeencarriedoutandthelearninghasbeenassessed.Beforeallthatcanhappen,teachersneedsomeassuranceabouttheperceivedqualityofaresourceinordertohelpthemselectthebestcontentpossiblefortheirteaching.SoinlookingatqualitywithrespecttoOER,weshouldthinkof‘quality-before-the-fact’aswellas‘quality-after-the-fact’.Neitherissufficient,andbotharenecessaryincontemplatingthedomainofOEP.If,outoftheirteachingpractice,teachersareabletoevaluatethequalityoftheresourcestheyusedinthe

7https://opencontent.org/blog/archives/3821Wiley,DStopSaying“HighQuality”

10

classroom,that,inasense,completesacircleofquality:eachtimethecircleiscompleted,theperceivedqualityofaresourcecanberenderedmoreandmoretrusted.So,afteraresourcehasbeendeployedtheteacherisabletogivefeedbackonwhetherandhowtheresourcewasusefulornot.Aneffectiverepositoryshouldallowtheeducatortocompletethis‘qualitycircle’,fromdiscovery,touse,toprofessionalevaluativefeedback,andtheprocesscanthenberepeatedasmanytimesasnecessary.

A Simple Outline of the Notion of the ‘Quality Circle’ in Relation to OER

Thiscanleadtoarankingoftheresourcesavailablefromarepositoryorothersource,sothatwhenanothereducatorcomesalonglookingforsimilarresources,therepositoryisabletoserveupanorderedlistthatreflectspreviousevaluations.Evenbetterisafunctionthatenablessortingbymorethanonemetric,asisthecaseinsomeoftheexamplesexaminedinthispaper.Thequalityofalearningresourcedependsonthedemandsandrequirementsbothoftheeducatorandofthelearnerratherthanonsomepre-determinedcharacteristicoftheresource:

“Inthissense,qualityassuranceandvalidationcannotcomefromatop-downapproachthatwillperformacentralizedqualitycontrol.Onthecontrary,suchqualitycontrolshouldtakeintoaccounttheend-users,engagethemproactivelyinprovidingfeedbackandsuggestionsforenhancements,supporttheminadaptingthecontentthemselvesandsharingitwiththecommunity.”8

Takingthisapproachfurther,oneofthemostusefuldelineationsofqualityinthiscontexthasbeenprovidedbyCamilleri,EhlersandPawlowskiintheir“State

8http://lre.eun.org/wp/LRE_White_paper_001.pdfChounta,Irene-AngelicaAreviewofthestate-of-artoftheuseofmachine-learningandartificialintelligencebyeducationalportalsandOERrepositoriesp.22

11

oftheArtReviewofQualityIssuesRelatedtoOER”9.Intheirpapertheysetoutthefollowingfeatures,someofwhichareuniversalorgeneralwhileothersaremorespecifictoaparticularinstanceofusebywhichitmightbepossibletocometoanunderstandingofthequalityofalearningresource:

Efficacy–bythisismeantthefitnessforpurposeoftheobject/conceptbeingassessed.WithinthecontextofOER,thismightincludeconceptssuchasease-of-reuseoreducationalvalue.Impact–impactisameasureoftheextenttowhichanobjectorconceptproveseffective.Impactisdependentonthenatureoftheobject/conceptitself,thecontextinwhichitisappliedandtheusetowhichitisputbytheuser.Availability–theconceptofavailabilityisapre-conditionforefficacyandimpacttobeachieved,andthusalsoformspartoftheelementofquality.Inthissense,availabilityincludesconceptssuchastransparencyandease-of-access.Accuracy–accuracyisameasureof(a)precisionand(b)absenceoferrors,ofaparticularprocessesorobject.Excellence–excellencecomparesthequalityofanobjectorconceptto(a)itsquality-potential,i.e.themaximumtheoreticalqualitypotentialitcanreach.

Thefirstofthesefactors–efficacy,orfitnessforpurpose–mightbeviewedasthesinglefactormostrelevanttoteachersseekingeffectiveresourcesfortheirteaching.Whetherthatisthecaseornot,allfivefactorstakentogetherembraceboththequestionofimpactonlearningandtheneedfora‘pre-definition’ofqualitythatworksfortheteacherhopingtocreateinterestinglearningexperiencesforstudents.Intakingforwardthesecharacterizationsofquality,theyproposeaconceptualframeworkbuiltaroundthreesetsofqualityapproaches:

QualityAssuranceofResources–…alifecyclemodeltounderstandthequalityfactorsaffectingindividualresources,includingtheircreation,useandevaluationQualityAssuranceofStrategies/Policies–usingamaturitymodel…theinstitutionaldevelopmentofpolicieswhichgovernandpromotethecreationofOERQualityAssuranceofLearning–…course-specificqualityassurance,includingprocessesofteaching,assessmentandrecognition10

Bytakingthisapproach,theygobeyonddefiningqualityintermssimplyofsomeintrinsiccharacteristicsoftheresources.Forexample,theyoutlinethevaluereinforcedintheresourcesbythefeaturesbuiltintotheprocessofcreatingtheresourcesinthefirstplace.Evenhere,though,theultimatequalityoftheresource,nomatterthecalibreofitsprovenance,canonlybemeasuredagainst

9https://pdfs.semanticscholar.org/475a/50bbc6bb0b0e580aa1470ff88d972b60983d.pdfCamilleri,AnthonyF.;Ehlers,UlfDaniel;Pawlowski,JanStateoftheArtReviewofQualityIssuesrelatedtoOpenEducationalResources10Camilleri,AnthonyF.etalibid.

12

itseffectivenessinfacilitatinglearning.Theynotetoo,inpassing,thatapplyingtheprocessofresource-creationtoOERisnecessarilymorecomplexthanthesameprocessappliedtotraditionally-producedresources,giventhepotentialmultiplicityoffacetsofauthorship,sharing,modification,useandre-useinOER.Intheirpaper“QualityAssuranceintheOpen”AtenasandHavemann11consideredthewiderapproachoflookingatthesharedcharacteristicsofOERrepositoriesratherthanthequalityoftheresourcesthemselves.Theytookasastartingpointtheassumptionthat:

“ThepurposeofOERrepositoriesistosupporteducatorsinsearchingforcontentinastructuredway,sharingtheirownresources,reusingexistingmaterialsandcreatingnewresourcesthroughadaptingortranslating…”12

Onthisbasistheysetoutwhattheyidentifyasthedefiningcharacteristicsofthemosteffectiverepositories,characteristicsthatareeithersocialortechnical,orboth.Whilethetablebelowwouldbedifficulttotranslateintoguidelinesforteachersseekingtoevaluateresources,itdoesserveapotentiallyusefulpurposeforthosewhoaretryingtodevelopeffectiveandfully-functioningrepositories13.

QualityAssurance

Description Themes Characteristics

Featuredresources

Abilityoffeaturingresourcesthatarepotentiallyofhighinterestforteachersbecauseoftheirdesignorcontent.

Search,Share,Collaborate

Social

Userevaluationtools

Toolsfortheresourcestobeevaluatedbyusersaimingtoratearesource.

Collaborate Social

Peerreview Peerreviewaspolicytoreviseandanalyseeachresourcetoensureitsquality.

Collaborate Social

Authorship Analyseiftherepositoriesincludethenameoftheauthor(s)oftheresources.

Search,Reuse Social

Keywords Methodicallydescribetheresourcestofacilitatetheretrievalofthematerialswithincertainspecificsubjectareas

Search Technical

Metadata Introducestandardisedformatsofmetadata(DublinCore-IEEELOM-OAI-PMH14)forinteroperability

Search,Share,Reuse Technical

11https://eprints.bbk.ac.uk/8609/1/30-288-1-PB.pdfAtenas,JavieraandHavemann,Leo(2013).Qualityassuranceintheopen:anevaluationofOERrepositories.INNOQUAL:TheInternationalJournalforInnovationandQualityinLearning12Atenas&Havemann,ibid,p.2413WearegratefultoDavidMassartforofferingclarificationonthisdistinction14OAI-PMHisaprotocolforexchangingmetadata,notametadataformat.See:https://en.wikipedia.org/wiki/Open_Archives_Initiative_Protocol_for_Metadata_Harvesting

13

Multilingualsupport

Designtheinterfaceoftherepositoryinamultilingualwaytowidenthescopeofusersbyallowingthemtoperformsearchofcontentindifferentlanguages.

Search,Share,Reuse,Collaborate

Technical

SocialMediasupport

Introducesocialmediatoolstoenabletheuserstosharetheresourceswithinsocialmediaplatforms.

Search,Share,Reuse,Collaborate

Social,Technical

CreativeCommonsLicences

SpecifythetypeofCreativeCommonsLicenceperresourceorgiveinformationaboutthespecifictypeoflicenceforalltheresources.

Search,reuse,collaborate

Technical

SourceCodeorOriginalFiles

Allowdownloadingtheoriginalfilesorsourcecodeofresourcessotheycanbeadapted.

Reuse,Collaborate Technical

Theimportanceofensuringqualityindigitallearningresources(indeedanylearningresources,nomattertheirformat)cannotbeunderestimated.Camilleri,EhlersandPawlowski,alreadycited,madetheimportantpointthat,althoughopenlearningresourcesaregenerallyavailable,theyare‘notfrequentlyused’.Inthereasonsforwhythismightbeso,theylistthefollowing:

“…[lackof]organisationalsupport,alackofsharingculturewithinorganisations,lackofskills,quality,trustortimeandskillsforadaption.Onlyoneelementisrelatedtotheavailabilityoftechnicaltoolsforsharingandadaptingresources.Notasinglebarrierrelatestothequestionofaccessibilityandavailability.”15

So,lackofquality(including,wepresume,perceivedlackofqualityasaconsequenceofapaucityofpriorevaluations),orevenasimplelackofinformationaboutquality,isadeterminingfactorsintherelativelackofactualuseofdigitallearningresources.Inthecreationofdigitallearningresources,theEuropeanConsortiumforAccreditationsuggests16anumberofqualitycriteriawhichshouldbetakenintoaccountwhendesigningsuchlearningobjectives:

• Internalandexternalstakeholdersshouldbeinvolvedintheprocessofdesigningandrevisinglearningobjectives,forexamplebyparticipatinginmeetings,pedagogicalboards,satisfactionsurveys,evaluationprocedures,etc.

• Learningobjectivesshouldbedescribedinacomprehensibleway.Inparticular,theyshould:

o Bedefinedinclearandconcreteterms(shortandsimplesentences),

15Camilleri,AnthonyF;etal,ibidp.2516Camilleri,AnthonyF.;etalibid.p.17QuotingfromEuropeanConsortiumforAccreditation-http://ecahe.eu/w/images/b/ba/Publication-Learning_Outcomes_in_Quality_Assurance_and_Accreditation.pdf

14

o Focusonwhatstudentsareexpectedtobeabletodemonstrateanddescribeobservableabilitieswhichcanbeassessed.

However,giventhattheperceptionofqualityisjustoneofseveralbarrierstouptakeandusageofOER,anystrategiesundertakentoimprovequalitymustbeseenasjustonefacetofpoliciesdesignedtoincreasetheuseofOERinanyeducationalsector.Questionsoftrust(itselfoneelementofquality,ofcourse),time,skillsandculturemustbedealtwithalongsidequestionsofquality.

THE IMPORTANCE OF QUALITY TO THE UPTAKE OF OER SomestudieshavelookedatwhatteachersneedinordertoencouragethemtouseOERrepositories.Inherpaper,ClassificationandQualityCriteriaforOpenEducationalResourcesintheFieldofForeignLanguageLearning,ZitaKrajcsomadetheself-evidentbutwholly-necessarypointthat:

“…teachersneedhighqualityresources.”17

However,shewentontosay:“Furthermore,thepoolofresourcesmustbebigenoughtooffersomethingformany.Thesearchingmechanismfortheresourcesisalsocrucial;teachersneedefficientwaysforsearchingandfindingmaterials(reachedwithinthreeclicks)fromdifferentcurriculabackgrounds(classificationbysubject,topicsandlevel).”18

Andshecontinues,echoingtheconceptofthe‘qualitycircle’alreadymentioned:

“Classificationassuchiscrucialforthesearchmechanism,butqualityfactorsdeterminethelong-termsuccessfulnessofdigitalresourcesontheusers’side.InthecaseofOERthesefactorsrelatetoinput(qualityofthestructureandpotential),implementation(process)andoutput(outcome)aspects).”19

Shemakestheimportantpointthatensuringqualityintheprocessofcreatinglearningresourcesdoesnotnecessarilyequatetohighimpactontheeventuallearningintendedbyitsuse:

“…[a]focusontheinputqualitycriteriaofOER….hasnotbeenprovenascausalforthelearningeffectyet.Moreresearchisneededtoconfirmandextendthefollowingqualitycriteria.”20

Weispossible,perhaps,toassertwithsomeconfidencethatthisisacausalrelationshipthatisunlikelyevertobeprovenasauniversalrule.

17https://content.sciendo.com/configurable/contentpage/journals$002fjolace$002f4$002f1$002farticle-p48.xmlKrajcso,RClassificationandQualityCriteriaforOpenEducationalResourcesintheFieldofForeignLanguageLearningp.4918Krajcso,Ribid.p.4919Krajcso,Ribid.p.5020Krajcso,Ribidp.50

15

EVALUATING QUALITY IN OER Ingeneral,itwasdifficulttoidentifyasignificantbodyofworkrelatingtospecifictoolsortechniquesthatcanbeused,andmoreimportantlyarealreadybeingused,byeducatorsandotherstoevaluatethequalityoftheresourcestheyuse.Asmallnumberofrepositorieswereidentifiedthathavesuccessfullyimplementedeffectivefeedbackqualitymechanismsandthesewillbeoutlinedonpages18to24below.IntermsofthequestionofqualitybothinthecreationandchoiceofOER(asopposedtothequalityoftheresourceinrelationtoitsimpactonlearning)anumberofattemptswereidentifiedthatsetoutsomeguidelinesthatteacherscoulddeploytoassisttheminusingOERwell.ArelativelyclearandstraightforwardrubricwascreatedbytheUniversitySystemofGeorgia(theUSGiscomposedof26highereducationinstitutionsincludingfourresearchuniversities,fourcomprehensiveuniversities,ninestateuniversitiesandninestatecolleges,aswellastheGeorgiaPublicLibraryService,allintheUSStateofGeorgia).AcomponentofUSGcalledAffordableLearningGeorgia21setoutthefollowingguidelinesforteachers:AFFORDABLELEARNINGGEORGIAOEREvaluationCriteriaClarity,Comprehensibility,andReadability

• Isthecontent,includinganyinstructions,exercises,orsupplementalmaterial,clearandcomprehensibletostudents?

• Isthecontentwell-categorizedintermsoflogic,sequencing,andflow?• Isthecontentconsistentwithitslanguageandkeyterms?

ContentAccuracyandTechnicalAccuracy

• Isthecontentaccuratebasedonbothyourexpertknowledgeandthroughexternalsources?

• Arethereanyfactual,grammatical,ortypographicalerrors?• Istheinterfaceeasytonavigate?Aretherebrokenlinksorobsolete

formats?AdaptabilityandModularity

• Istheresourceinafileformatwhichallowsforadaptations,modifications,rearrangements,andupdates?

• Istheresourceeasilydividedintomodules,orsections,whichcanthenbeusedorrearrangedoutoftheiroriginalorder?

• Isthecontentlicensedinawaywhichallowsforadaptationsandmodifications?

21https://www.affordablelearninggeorgia.org

16

Appropriateness• Isthecontentpresentedatareadinglevelappropriateforhigher

educationstudents?• Howisthecontentusefulforinstructorsorstudents?• Isthecontentitselfappropriateforhighereducation?

Accessibility• Isthecontentaccessibletostudentswithdisabilitiesthroughthe

compatibilityofthird-partyreadingapplications?• IfyouareusingWebresources,doeseachimagehavealternatetext

thatcanberead?• Dovideoshaveaccurateclosed-captioning?• Arestudentsabletoaccessthematerialsinaquick,non-restrictive

manner?

SupplementaryResources• Haveyoureviewedthesesupplementaryresourcesinthesamemanner

astheoriginalOER?Whilethetenorofthisparticularrubricisfairlyspecifictoitsowninstitutionalsetting(forexample,atleastsomeofitsfocusseemstobeonmakingavailablecompletecoursesaswellassingle-useresources-teachersaremorelikelytorequirethelatterthantheformer),itdoesseemtooffersimpleguidelinesthatcouldbeadaptedformanyothercontextswhereteachersareeithercreatingorchoosingOER,orboth.Itshouldbenotedinpassingthat,accordingtoarecentpiecefromLibbyVMorris,USG’sAffordableLearningGeorgiahassavedstudentsmorethan$31montextbookcosts.22Morepertinenttothispaper,Morrisgoesontoraisethequestionofquality,amongstanumberofotherissues,thatshebelievesoughttoinformtheresearchagendaonOEPandOERforthenextfewyearsatleast:

“Thequantityofopenmaterialsislargeandgrowing,andthewidespreadenthusiasmisevident.Nextupistobuildamorerobustresearchagendatobetterunderstandquality,creationandadoptionprocesses,andoutcomes.Whataretheeffectsofopeneducationalresourcesoncoursedevelopmentandimplementation?Onstudentlearning?Whatarethecharacteristicsofcreatorsandearlieradopters?Howdofacultymembersidentify,select,andevaluateopenresourcesforcollege-levelcourses?Howdoinstructor-designedresourcesdifferfromadoptedandadaptedresourcesincoursedesignandeffectonlearning?Willstudents’priorexperiencewithOERaffectmotivationtoindependentlylocateanduseotherfreematerialsforlearning?Howwillthisinfluenceachievementandsuccess?”23

22https://link.springer.com/content/pdf/10.1007/s10755-019-09477-7.pdfMorris,LibbyVContemplatingOpenEducationalResourcesp.2publishedinInnovativeHigherEducation(2019)44:329–33123https://link.springer.com/content/pdf/10.1007/s10755-019-09477-7.pdfMorrisibid.p.3

17

ManyofthesequestionsplaydirectlytoanyconsiderationofqualityinrelationtoOER.RANKING SYSTEMS & COLLECTIVE INTELLIGENCE DeOliveiraetalintheirpaper,alreadycited,OpenEducationalResourcesPlatformBasedonCollectiveIntelligence,describehowcumbersomethetaskcanbeforteacherstosearchfor,locate,downloadanduseOERfromrepositories:

“Theircontentcanbeofmixedquality,withseveralresourceslackingimportantdescriptivemetadatawhich,alliedtotheoftensub-optimalindexingfromtheirsearchengines,resultsintheuserhavingtosortthroughmanyirrelevantcontentwhenqueryingforspecificeducationalresources.Sinceitisofteneasierforuserstosearchforcontentontheirpreferredsearchengine,theserepositoriesarerarelyaccessed.”24

Forthesereasons,anintelligentuserofGoogleoranyothercapablesearchengineisoftenabletoidentifylearningcontentandevenOERmoreeffectivelythanispossiblewiththeinternalsearchcapabilitiesofmanycurrentlyoperatingOERrepositories(althoughwithsomehonourableexceptions,aswillbeseeninthecasestudyonKlasCementbelow).Giventheaddedfactthatmanyrepositoriesareslowtorefreshtheircacheofcontent(usuallyforgood,thoughsometimesbureaucratic,reasons),andthatauniversalsearchenginecanbeusedeffectivelytopinpointrecentlycreatedresourcesfromacrosstheWeb,theneedforaneffectivesystemthatwilltrulyfacilitatethetaskoftheteacherseekingresourcesfromwithinasinglerepositoryisobvious.IfarepositoryistocompetewithGoogleanditspeers,itmustbeabletomatchthemwithinitsownlimiteddomainintermsofeaseofuse,flexibilityandpower,andofcoursetherepositoryitselfneedstobeextensiveenoughtoofferalargeandenticingcacheofresourcesfortheeducatortowishtoreturntoagainandagaintomineforcontent.DeOliveiraetalposittheideathatthiscanmosteffectivelybeachievedthroughthedevelopmentofaplatformthatexploitstheconceptofcollectiveintelligence:

“Theplatform’sgoalistoprovideaninteractivewebplatformwithsocialnetworkfeatures,toeasethepublication,searchandrankingofopeneducationalresourcesthroughacollectiveintelligence.Althoughcollectiveintelligencemaybringtomindtheideaofgroupconsciousness,incomputerscienceitusuallymeansthecombiningofbehavior,preferences,orideasofagroupofpeopletocreatenovelinsights.”25

Theyseektoharnesscollectiveintelligencetocreateasystemthatwillrankcontent(andusers)overtimeandthroughmultipleusesinwhich:

24https://ieeexplore.ieee.org/document/8537851deOliveira,etalibid.p.125https://ieeexplore.ieee.org/document/8537851deOliveira,etalibid.p.1-2

18

“….thehigherthescore,themorereliabletheuserorthecontentis…”26

Collectiveintelligence,whencombinedwithmachinelearning,isalsothemeansbywhichbureaucracycanbereduced,since:

“ThePlatformtrustsinthecommunitytodecidewhichuserisallowedtopublishresources,givinghighrankinguserstheabilitytopublishandinvitenewusersaspublishers.Thenewresourcesgothroughanautomaticfiltering,wheremachinelearningalgorithmstrytofurthermitigatethepresenceofinappropriatecontent.Theresultofthisfilteringisthenvalidatedbyhighrankingusers,toavoidanywronglyclassifiedresource.”27

Thisapproachformedthebasisforthecreationofaprototypeplatform,calledPlataformaMECdeRecursosEducacionaisDigitais28(MECREDforshort),inpartnershipwithBrazil’sMinistryofEducation.Theplatform,asofOctober2018,hadsome30,000publishededucationalresourcesand5000registeredusersaveraging1200downloadspermonth.Whileitwasnotpossibletomakeuseoftheplatformforthepurposesofthisreview,theapproachoutlinedbytheauthorsaboveisonethatcouldperhapsbeencouragedandlearnedfrom.Someconsideration,however,mightbegiventotheexpectationthatcombiningcollectiveintelligenceandmachinelearningcanonlybetrulyachievablewheretherepositoryenjoysacriticalmassofdata,bigenoughforMLtofunctioneffectively.29EXAMPLES OF REPOSITORIES WITH EFFECTIVE QUALITY SYSTEMS Asmallnumberofrepositorieswereidentifiedthat,itisfelt,aremakinggooduseofqualityfeedbackandranking.Wefocusonthreeoftheseherethatofferinterestingexamplestoconsiderinthecontextofthelong-termdevelopmentofOEP,butgivenmoretime,itiscertainthatotherscouldhavebeenidentifiedtoo.

KlasCement KlasCement30,anopenrepositorystartedbyoneteacher,HansDeFour31,inaschoolinFlandersin1998,isnowavaluableandextensiveOERrepositorysupportedbytheMinistryofEducationthatserveseducatorsinFlanders(andintheNetherlands,bydintoftheirsharedlanguage),aswellasacrossEurope.Withmorethan220,000uniqueusers,averyhighpercentageofthemactive,andwithmorethan58,000resourcescurrentlyheldintherepository,KlasCementisamajorplayeramongstOERrepositoriesinEuropeandbeyond.

26https://ieeexplore.ieee.org/document/8537851deOliveira,etalibid.p.227https://ieeexplore.ieee.org/document/8537851deOliveira,etalibid.p.228See:https://plataformaintegrada.mec.gov.br/home29ThankyoutoDavidMassartformakingthispoint30https://www.klascement.net31https://www.klascement.net/profiel/1/YouneedtoberegisteredonKlasCementandloggedintoviewthisprofile

19

Theavailableresourcescovermanydifferenttypesoflearningmaterials,includingarticles,downloadablelearningmaterials,websites,apps/software,interactiveexercises,images,videos,audiomaterials,andothers.Registereduserscanmakeflexibleuseoftheresources,beingabletosearch,view,download,re-useandadaptcontentfortheirteachingandlearningrequirements.Theyarealsoabletocommentoncontent,giveratingstoindividualresourcesandcanprintresourcesaswellassharethembyemailoronTwitter,Facebookandothersocialmediaplatforms.KlasCementisacontinuallydevelopingplatform,onethatseekstostayattheforefrontofOERfunctionality,notforitsownsakebutinordertocontinuetoofferitsuser-bodythemosteffectiveandusefuleducationalrepositorythatitcanintothefuture.ThosewhodirectKlasCement(agroupthatstillincludesitsfounder)useagroupofpart-timeteachersemployedbytheMinistryofEducationtoensurethatallcontent,whetheruser-generatedornot,meetstheirminimumstandardsofqualitybeforeitisapprovedforinclusionintherepository.Thesequalitystandardsinclude,amongstothers:

• ThattheresourcemetadatameetstheIEEE-LOMMetadatastandardtoensurethatitenablesfullinteroperability,coveringtitle,keywords,subjectmatteretc

• Thattheresourcealsoincludesadditionalmetadatatocoversareassuchaspedagogicalcontext,theme,etc.

• Thattheresourceisfullyproof-read,freeofmistakes,trustworthy,curriculum-relatedandpractical

• Thattherearenocopyrightviolations• Thatitoffersanidentifiableusefulnessinaneducationalsetting

Thereisalsoa‘crowd-sourced’componentthattakesaccountofuserreviews,commentsandratings.Forexample,foreachresourceavailablethroughKlasCement,theteacherorusercancommunicateanissueabouttheresourcetomoderators–theycanraiseissuesoffactualerror,problemswithhyperlinks,oranyotherissuethattheplatformshouldtakeaccountof.Tohelpeducators,KlasCementdeliverssearchresultsbasedonamathematicalformulathatincludes:

• Correspondencewiththekeywordinthesearchbar• Scoresgivenbyusers(andthenumberofusersthatgivethescore),• Timeonline• Typeofcontributor(teacher,student,companyetc)

Theendresultisaplatformthatservesitsuserstotheextentthat,withinitsownimmediateregionalandnationalcontext,itencouragesmanyuserstobypasstheubiquitousGoogleandinsteadtogofirsttoKlasCementwhentheyaresearchingforgoodandtrustworthycontentforalessonoracourse.IntheirconstantdesiretoimprovethefunctionalityofKlasCementsothatitbettermeetstheneedsofitsusers,theplatform’smanagershaveindicatedthattheywouldliketobeabletocreateamoredetailedandricher‘collectiveintelligence’thantheyarecurrentlyusing.Forexample,thescoregivenbyateachertoaresourcethathasbeenusedinteachinghasalimitedutilitygiventhefactthatthescoremightbe

20

givenforavarietyofreasons:designoftheresource,culturalcompatibility(anassessmentthat,ofcourse,willvaryfromculturetoculture),factualerrors,andmanyothers.Eachofthese,andanyother,reasons,mightinthemselvesbesignificantbutifafutureuserdoesnotknowwhatthatreasonwas,thenthescoreinitselfhasrestrictedvalue.32KlasCementisanOERrepositorythathasbeenabletogrowandprosperorganicallysinceitshumbleoriginsmorethantwodecadesago,tothepointnowwhereitoffersteachersandstudentsinFlandersandbeyondatrulyvaluableandadmirableplatform.Butevenmorecommendablethanwhathasbeenachievedtodateisthedeterminationofthosewholeadittocontinuetoimprovethequalityofitsofferingstoteachers.Itisarepositorythatdeservestobebetterknownbeyonditsownoriginalboundaries.

Curriki ThefeedbackmechanismsemployedbyCurriki33arealsoausefulreferencepointinthisdiscussion.CurrikiisanAmericaneducationnon-profitwhoseaimistobuildalargeglobalcommunitylibraryofOER,andtheyplacesharingattheheartoftheirmission.Theyaskmembersto:

• Sharewhatyoulearn.• Sharewhatyouknow.• Shareyourcontent.• Together,wecanmakeeducationmoreequitable.

WhenyousearchforcontentonCurriki34theresultinglistofresourcesoffersanumberoffunctionsthatateachercanusetoenhancethesearchandtoevaluateresourcesforothersinthefuture.Therearelinksofferingagreaterlevelofinformationonsubjects,appropriategrade/agelevels,andresource-types,aswellaslinkstoenabletheteachertosharetheresourcebyemail,onFacebook,TwitterandGoogleClassroom,andasaURI(forexample,inablogpost).Userscanflagthecontentasinappropriateandalsonominateforreview.Finally,ofcourse,ausercangivetheresourcearatingbetween1starand5stars,afeaturealreadydiscussedinthesectiononKlasCementabove.Currikiadditionallyratesresourcesaccordingtoitsownrubric,fromNR(notyetrated)upto3.0.Whenauserconductsasearchtheyareabletosorttheresultinglistofresourcesaccordingtomultiplecriteria:bytitle(A-ZorZ-A),bynewestfirstoroldestfirst,bymemberratingandbyCurrikirating.Interestingly,italsoenablesresourcestobesortedby‘standardsaligned’.Indeed,thesearchfunctioncanbeenhancedbysearchingaccordingtowhichevercurriculumstandardsyouwishtobealignedto.ThisismostobviouslywhereCurriki’slimitationsbecomeevidentto

32Indeed,ifascoreoutof5,forinstance,isfraughtwithdifficulty,thenasimplelike/dislikebinaryevaluationofaresourceisevenmoreproblematic,sincethereasonforthechoicecanvaryconsiderably,fromapplicabilitytoaparticulargroupoflearners,toeducationalrichness,togrammaticalcorrectness,topedagogicalprinciples,andsoon.33www.curriki.org34NotethattheCurrikiwebsiteprovedproblematicattimesintheSafaribrowseronAppleMacintosh,problemsthatwerenotapparentonotherbrowsers

21

aninternationalaudience.Despiteitsglobalaspirations,CurrikiisverymuchanestablishedAmericanplatformandsothe(admittedlyverylong)listofstandardsthatresourcescanbealignedtoisalmostexclusivelyAmerican,fromCommonCoretomanyindividualStatestandards.

Finally,asamemberofCurriki,ausercanalsoaddtheresourcetotheirpersonallibraryofcontent.CurrikiisverymuchfocusedontheK12sectorand,fromabriskperusalofthesite,mostofthecontentdoesseemtoofferresourcesofatleastreasonablequalitytoteachersandmanyappeartobeofhighquality.ItsAmericanoriginsmight,ofcourse,limititsusefulnesstoEuropeanandotherinternationalschoolssystems.Nonetheless,itoffersamodelthatotherOERrepositoriescouldlearnfrom.

MERLOT AnotherAmericanrepositoryofmuchlongerstandingthanCurriki,isMERLOT35,onewhichisverywell-knowngenerallyintheOpenEducationmovement.Itoffersaccesstocuratedonlinelearningandsupportmaterialsandcontent-creationtools,andisledbyaninternationalcommunityofeducators,learnersandresearchers.MERLOTwassetupin1997attheCaliforniaStateUniversityCenterforDistributedLearningandhasaHigherEducationfocus.AssuchitsusefulnesstoK12educatorsisnecessarilylimited.However,itdoesappeartoofferagoodmodelinrelationtotheevaluationofqualityfromwhich,aswithCurriki,otherOERrepositoriessitedelsewhereintheworldcouldlearn.

35https://www.merlot.org/merlot/index.htm

22

MERLOTissimilartoCurriki(andpossiblyofferedamodelthatCurrikimightitselfhavelearnedfrom)inthatitoffersusersthecapabilitytoratearesourcefrom1starto5stars.Resultsofacontentsearchcanbesortedbytheirratings,buttheycanalsobesortedbywhetherornotusershavecommentedontheresourceoraddedtheirown‘learningexercises’totheresource.Thesearchfunctionalso,importantly,permitssearchingbyaccessibility,andbyitsinclusioninany‘bookmarkcollections’or‘coursee-portfolios’.A5-starratingsystemsuchasthismighthavethevirtueofsimplicity.However,itcanalsobearguedthatsuchasystemistoosimplistic,asalreadynotedinthesectiononKlasCementabove.Theunderlyingreasonswhyausermightgiveascoreof1outof5,example,candiffersogreatlythatthescoregivenreallyofferslittleofsubstancetosubsequenceusersofthatresources,sincetheywillnotknowwhetherthelowscorewasgivenbecauseofpoordesignoranerroroffactoraculture-clashorwhatever.Ontheotherhand,ifascorecanbeascribedalongsidesomequalitativecommentastothereasonforthescore,thatwouldatleastgosomewaytoincreasingtheutilityofthebarerating.Asacentrallycuratedrepository,thereisa‘top-down’aspecttoMERLOT,butalsoa‘bottom-up’aspect.Allthematerialsarecontributedbymembers(membershipisfree),butmanyofthematerialssubmittedgothroughapeer-reviewprocess,overseenbyanyoneoftwentyeditorialboards.Eacheditorialboardismanagedbyaneditorandassociateeditor,andeachhasateamofpeerreviewers.

“AlloftheseitemshavebeencontributedbytheMERLOTmembercommunity,whohaveeitherauthoredthematerialsthemselves,orwhohavediscoveredthematerials,foundthemuseful,andwishedtosharetheirenthusiasmforthematerialswithothersintheteachingandlearningcommunity.”36

Materialsarereviewedonthebasisof:

• Qualityofcontent• Potentialeffectivenessasateachingtool• Easeofuse

MERLOTevenprovidestrainingforpeerreviewers,althoughallpeerreviewersarerecognisedauthoritiesintheirownfield.Crucially,MERLOTisnotarepositoryinthesensethatitactuallystoresallthecontentwithinitsownstoragespace.Rather,itisametadata-basedrepository.Inotherwords,allofthecontentmadeavailablethroughMERLOTisretainedinitsoriginallocationanditisthemetadataattachedtotheresourcesthatmakethemsearchableinMERLOT.ThecontentismerelyrendereddiscoverableandretrievablethroughMERLOT.Ofcourse,thiscreatestheoverheadofhavingto

36http://info.merlot.org/merlothelp/topic.htm#t=MERLOT_Collection.htm

23

verifyallthelinksonaregularbasis.MERLOTcarriesoutthistaskonamonthlybasisandreportsadeletionrateofapproximately2%onaverage.Thisisamodelthat,ontheonehand,mightappearmoreflexibleandexpandablethanafixedrepositorythatstoresallofitscontentwithinitsownbounds.However,itshouldalsobenotedthat,inaccessingasystemsuchasthis,theteacherrunstheriskofcontentpreviouslyuseddisappearingthenexttimeitissearchedfor,aproblemsthatisunlikelytooccurinarepositoryinwhichresourcesarestoredonafixedsingleserverorsetofservers(suchasisthecasewithKlasCementalreadydescribed).BothCurrikiandMERLOTofferveryeasy-to-usemechanismstouserswhowishtoratethecontenttheydownload,usingasimple5-starsystem.Othersystemsofferuser-feedbacksystemsthataremuchmorecomplex.Oneofthemost

complexfoundduringthereviewwasthatofferedbyOERCommons37inwhichusersfirsthavetoalignthecontenttooneormoreofalargenumberofUS-basedcurriculumstandards,andthenareaskedtoevaluatetheresourcesacrosssixfurthersetsofcriteria.

Further Discussion ItispossibletoconcludethatKlasCement,CurrikiandMERLOTalloffer,tosomeextent,systemsthatworkonthebasisofaformof‘collectiveintelligence’notdissimilartothatoutlinedbyDeOliveiraetalabove.Thesimplefactthatlargenumbersofusersareevaluatingresourceswithinaverylargecacheofcontentbuildsalevelofcommonandsharedknowledgeacrosstheplatformsthatwouldappeartoservethebroadneedsoftheircommunitiesofusersverywellindeed.Itshouldbenoted,however,thatthosedevelopingandmanagingKlasCementfeelthattheycanandwillgofurtherindevelopingtheconceptofcollectiveintelligenceintheirrepository.Largeandeffectiverepositoriessuchasthesealsoenablethe‘qualitycircle’concepttooperate,ensuringsomemeasuresofqualityboth‘beforethefact’–i.e.whentheteacheristryingtolocategoodqualityusablecontentforuseinteachingpractice–and‘afterthefact’–i.e.whentheimpactofanyparticularresourceisassessedandsuchintelligenceisfedbackintothesystem.

37https://www.oercommons.org

OERCommonsUser-FeedbackSystem

24

AcursoryexaminationofvariouscategoriesofresourcesinOERCommonssearchedforrevealsthatmanyoftheresourceswithintherepositoryareasyetunevaluated.Thecomplexityandsomewhatbureaucraticnatureoftheevaluationprocessmighthaveabearingonthis,buttheeffectisthatanyperceptionofcollectiveintelligencedissipatesthroughthesystem’sseeminglackofuse.Theease-of-useofasystemforevaluatingandfeedingbackonaresourceis,itcanbeinferred,asignificantfactorindeterminingwhetherornottheresourcesareevaluatedtoanyusefulextent.Complexandcumbersomesystemsarelesslikelytobeusedsuccessfullyorextensively.Ofcourse,feedbackmechanisms,aswellasbeingtoocomplexorcumbersome,canalsobetoosimplistic.AnexampleofthisthatcanbetakenfrombeyondtheworldofOERrepositoriesisthetoolthatYoutubeprovidestoenableaviewertolikeordislikeavideo,thesimplethumbs-uporthumbs-down,whichineffectworksinsimilarfashiontothe‘Like’facilityinFacebook.UsingsuchatoolinanOERrepositorywouldcertainlymakethetaskoffeedingbacksimpleforeducators.However,asalreadynoted(p.24)thevalueofasimplebinary‘like’or‘dislike’islikelytobeinadequateforthepurposesofrankingresourcesforusersinanyrigorousway.However,arankingsystemthatatleastletsusersseehowmanytimestheresourcehasbeenopened,andmoreimportantly,downloaded,wouldatleastoffersomenotionofthepopularityofaresource.Whetherthatpopularitycanbetranslatedintoanassessmentofqualityisquestionable,ofcourse,sincearesourcecreatedforanichesubjectwillappeartobemuchlesspopularthanamuchmoreprevalentsubject.Asimplerankingby‘numberoftimesdownloaded’thereforeneedstobetreatedwithcaution.BothEuropeanSchoolnet’sownEuropeanSchoolnetAcademy38andtheEuropeanTeacherAcademy39(whichwasdevelopedforSchoolEducationGatewaybyEuropeanSchoolnetonthebasisofitsexperiencewithSchoolnetAcademy)havesimilartools,namelyathumbs-uplinkforeachofthecoursesorwebinarsmaterialslistedintheircatalogues.However,ratherthansimplyregisterauser’s‘like’ofacourseinthecatalogue,SchoolnetAcademytakestheusertoitsFacebookpagesothatheorshecanleaveanapprovingmessageorarecommendationthere(or,wepresume,anoteofdisapproval).Whilebothoftheserepositoriesundoubtedlyoffercontentofahighquality,theapparentlackofuseofthecommentinganduser-feedbacksystemsavailableonthemmustdetractsomewhatfromtheoverallutilityoftheplatforms.OntheTeacherAcademy,forexample,thereareclearlymanyhighlyusefulcoursesforteachersseekingtobuildontheirknowledgeandskills(thesheerbreadthanddepthofqualitythatisevidentacrossallthecoursesonofferisundeniable,andespeciallysosinceteacherscanoftenfinditdifficultorevenimpossibletoobtainstrongandeffectivesupportfortheirprofessionaldevelopmentneedsneedfromanywhereelse).However,ithastobesaidthat,despitetheobviousqualityon

38https://www.europeanschoolnetacademy.eu39https://www.schooleducationgateway.eu/en/pub/teacher_academy.htm

25

offer,ateachercomingtothecatalogueforthefirsttimewillfinditdifficulttogetanypriorfeedbackfromusersonhowusefulthecourseswereandwhy.

THE QUALITY CHALLENGES: Three Short Case Studies Thefocusinthissectionisonthechallengesfacedbyrepositoriesastheyseektoensurethatusershavethebestexperiencepossibleinrelationtothequalityresourcestheylocateandsubsequentlymakeuseof.Threeshortcasestudiesthatprofiledifferentqualityissuesareoffered.Scientixisastronglycuratedsitethatadmitscontenttoitsrepositoryonlyifitmeetsitsownminimumstandardsofquality.WhenuserssearchScientixforcontent,theycanbesurethatleastthecontenttheyidentifyhassomebasiclevelofquality.ThissectionofferssomequalitativeconsiderationsbasedontheexaminationofthesearchfunctioninScientix.GeoGebraisaninteractivemathematicsapplicationthatisbothbrowserandappbasedonmultipleplatformsandthatisaimedatmathematicslearnersfromprimaryschoolthroughtouniversity.Arecentpaperco-authoredbyGeoGebra’sCEOofferssomefascinatinginsightsintohowaplatformmightgoabouttryingtoimprovethequalityofitssearchfunctionsoastoensurethatusersfindtheresourcesthataremostappropriatetotheirneeds.PhotodentroLORistheNationalLearningObjectRepositoryforprimaryandsecondaryeducationinGreece.Itisanextensiveplatformthathostsalargecacheoflearningobjects,eachofthemtaggedwitheducationalmetadatabasedontheIEEE-LOMMetadatastandard.Itisopentoeveryone:students,teachers,parents,andothers.

Case Study 1: Scientix Scientixwascreatedin2010ontheinitiativeoftheEuropeanCommissionandhasbeencoordinatedbyEuropeanSchoolnetsinceitsfoundation.Ithasbecomeaverypopularplatform,onethatoffersaverylargecomplementofmainly(but

notonly)STEMresourcesacross24EUlanguages.AllofitsservicesareaccessibletoanySTEMeducatorsandotherstakeholdersforfree.TheScientixUpdate,201940,describeditsofferingsthus:

• AccesstoinformationonEuropeanscienceeducationprojects• Accesstohigh-qualityresourcesinscienceeducation,withon-demand

translationofsometeachingmaterialforfree

40SCIENTIXUPDATE2019http://files.eun.org/scientix/scx3/publications/Scientix3_Final_Publication_web.pdf

26

• AccesstoScientixnews,enablinguserstokeepinformedaboutthelatestnewsonscienceeducationinEurope

• AScientixeventcalendar,bringingtogetherscienceeducationeventsfromacrossEuropeandbeyond

• Exchangeandsharingtoolssuchasforumsandchats.AcriticalobjectiveofScientixistosupporttheSTEMteachingcommunityacrossEuropetoenableeducatorstofind,downloadandmakeuseoftheverybest,high-qualitylearningresourcesthatcanbefoundinSTEMeducation.Therepositoryitselfisacollectionthatisdividedintoteachingmaterials,STEMreportsandtrainingcourses.ItalsooffersopportunitiestoattendorparticipateinwebinarsandconferencesrelevanttoSTEM.Inadditiontothis,Scientixprovidesarobustplatformfordiscussionandexchangebetweenministriesofeducation,NGOsandotherrelevantagenciesandorganisationsastheydevelopandimplementSTEMpoliciesacrossthecontinent.Importantly,itiscentraltotheScientixvaluesandprinciplesthatteachersthemselveswillplayakeyroleininfluencingstudentperceptionsofSTEM(andother)subjectsandinsupportingstudentstoaimforSTEMopportunitiesinhighereducationandintheircareersintothefuture.TheScientixresourcerepositoryisacuratedcollectionoflearningmaterialscentredonSTEMeducation.ManyoftheseresourcesaretheproductsofprojectsthathavebeenfundedandledbyScientix,andtodatealmost2,500teachingmaterialssitintherepositoryalongsidehundredsofSTEMreportsanddozensofhighqualitytrainingcourses.TherepositoryalsogivesaccesstoarangeofScientixMoodlecourseswhichareself-pacedandnon-tutored.ThecurationprocessiscentraltohowScientixoperates.RegisteredusersontheScientixportalareabletouploadresources,andthesearethenreviewedbytheScientixeditorialteam.Uploadguidelinesareofferedsothatresource-creatorshaveaclearideaforhowtheycanensuretheirresourcesareacceptedintotherepository.AswiththeMERLOTrepository,itisonlythedetailedmetadataoftheresourcethatisuploadedsinceScientixlinkstotheoriginallocationoftheresource(andallsuchlinksareverifiedbeforeinclusion).Inmostcases,thisisthewebsiteoftheprojectthatcreatedtheresourceinthefirstplace.Thoseuploadingresourcesarerequiredtoofferpre-defineddatasuchastypeofresource,appropriateeducationalstageandage-rangeofstudents,andschoolsubjects,amongstmanyothers41.Itshouldalsobenotedthat:

“TheScientixresourcerepositoryisnotopentoprojectscreatedwithoutofficialendorsement(i.e.theymusthaveprivateorpublicfunding,beit

41D.MassartandE.Shulman,LearningResourceExchangeMetadataApplicationProfileversion4.7,EuropeanSchoolnet,2011.OurunderstandingisthatScientixusesavariantoftheLREMetadataApplicationprofile,asdoanumberofEuropeanMinistriesofEducationprovidingresources

27

regional,nationalorEU),self-fundedprojectsorschoolprojects(teacher'sprojects).Resourcesfromsuchprojectswillbeautomaticallyrejected.”42

PerceptionsofqualityinthelearningresourceswithintheScientixrepositoryarealmostentirelybasedonthefactthatallresourcesarecuratedbytheScientixeditorialteamandthatalltheresourceshavebeendesignedtomeettheminimumqualityrequirementssetbytheeditorialteam.Therepositoryplacesalinkwithineachresourcedescriptiondesignedtopermituserstoaddcommentsandaratingfortheresource,but,ontheevidenceofourbrisksurveyoflearningmaterialsoffered,theydonotseemtobeusedmuch.Qualitytherefore,inthisparticularcontext,ispredeterminedbytheproceduresoftherepositoryanddoesnotappeartodependmuchifatallonusers’perceptionsofquality.Itisclearthatplatformoffersabroadrangeofhigh-qualityresourcesintheformofteachingmaterials,reports,trainingcoursesandLREmaterials.Evenacursoryexaminationbyaprofessionaleducatoroftheresourcesonoffershowsthatthematerialsandcoursesavailableoffermuchofvaluetoteachersandlearners.Eachresourceoffersverycompletelevelsofmetadataforfilteredsearching,aswellastheoptionsofwritingcommentsaboutthematerial,ratingthematerialfrom1starto5stars,andreportingproblemswiththecontent.Additionally,Scientixhasmadetheverypositivemovetoofferthesearchfunctionin24Europeancommunityandotherlanguages.Indeed,Scientixalsooffersafreetranslationserviceforanyonewhowantstotranslatearesourceintoanotherlanguage.ThesearchfunctionitselfwithinScientixappearstobehighlyeffective.AsearchformaterialsinScientixoffersfilteringbysubject,byresource-type(forexample,app,case-study,course,experiment,lesson-planetc),byappropriateage-group,andbylanguage.However,onaccessingtherepository,itquicklybecomesclearthatonlyaminorityofresource-usersareactuallymakinguseofthefeedbackmechanismsavailabletothem.ThisisofcoursenotachallengethatisfacedexclusivelybyScientix.Rather,itisachallengefacedbyallOERrepositories.

Case Study 2: GeoGebra Averyeffectiveandwell-designedOERrepositoryintheEuropeancontextisGeoGebra43.ItisbasedinAustriabuthasaninternationalfootprint.Ithaswonastringofawardsanditusesaverylarge,flexibleanddynamicglobalteamtodevelopitsresourcesandtotranslatethoseresourcesintomultiplelanguages.Itisanopenplatformbothinthesensethatanyonewiththeknowledgeandskillstodosocancreateanduploaddynamicresourcestothe

42http://www.scientix.eu/resources/upload-guidelines43https://www.geogebra.org

28

GeoGebrarepository,andinthesensethatitisopentoanyonetomakeuseofforteachingandlearningpurposes.GeoGebradescribesitselfthus:

“GeoGebraisdynamicmathematicssoftwareforalllevelsofeducationthatbringstogethergeometry,algebra,spreadsheets,graphing,statisticsandcalculusinoneeasy-to-usepackage.GeoGebraisarapidlyexpandingcommunityofmillionsofuserslocatedinjustabouteverycountry.GeoGebrahasbecometheleadingproviderofdynamicmathematicssoftware,supportingscience,technology,engineeringandmathematics(STEM)educationandinnovationsinteachingandlearningworldwide.”44

GeoGebraoffersaconvenientandinstructivecasestudyofsomeofthechallengesfacedbyanyOERrepositoryasitseekstodeliveraplatformthatisconsistentlyabletoofferusers‘gooddynamicmaterials’thataccordwiththeirown–thatis,theusers’–ownstandardsofquality.Forthissection,extensiveusehasbeenmadeofahighlyusefularticlewrittenbyMarkusHohenwarter(CEOofGeoGebra)andCristianHedes,entitled:“UsingQualityFactorstoFind“Good”ResourcesontheGeoGebraMaterialsSharingPlatform.”45(ThepaperwasverykindlysharedwithusbyMarthaZellinger,whooffersPartnerSupportforGeoGebra.)HohenwarterandHedesoffertheinsightthatexpertusersofGeoGebrahaveindicatedthattheytendtosearchformaterialswhichhavebeenproducedby‘highqualityauthors’,namelyresourceproducerswhotheyknowhaveproducedgoodqualitymaterialsinthepast.Inresponsetothis,GeoGebratookthestepofaddingtheoptionforuserstofollowcertainauthorsandtoreceivenotificationswhenevertheseauthorsuploadednewcreationstotheplatform.Butanumberofotherqualitymeasuressuggestedby‘expertusersandliterature’werealsoaddedintotheplatform,andHohenwarterandHedesexplainwhatthosefactorsareandhowtheychosetodealwiththeminGeoGebra.UsingthefindingsofKimeswenger201646,theydefinedthreefurtherfactorsthatneededtobetakenintoaccountintheirattempttoimprovethesearchforgoodmaterials.Thesearethematerialfactor,theauthorfactorandtheuserfactor.Thefirstofthese–materialfactor–seekstomeasuretheactivityhappeningaroundacertainmaterial,forexample,howmuchitisbeingused,copiedandfavourited.Theyderivedthefollowingparameterstomeasurethisfactor:

44https://www.geogebra.org/about45http://www.iadisportal.org/digital-library/using-quality-factors-to-find-%C2%93good%C2%94-resources-on-the-geogebra-materials-sharing-platform46Kimeswenger, B., 2016. Addressing Quality Aspects of Dynamic Mathematics Materials. Proceedings of 13th International Congress on Mathematical Education (ICME 13). Hamburg, Germany

29

• featured:materialsthathavebeenfeaturedbytheGeoGebrateamshouldhavehigherpriority

• type:books(collectionsofworksheets)shouldhavehigherprioritythansingleworksheets

• derived:materialswithmanycopiesshouldhavehigherpriority• groups:materialsusedbymanygroupsshouldhavehigherpriority• books:materialsusedinmanybooks(collections)shouldhavehigher

priority• favourites:materialsthathavebeenfavouritedbymanyusersshould

havehigherpriority• viewsperday:materialsthatareviewedmoreoftenshouldhavehigher

priorityThesecond–authorfactor–hastoreflectthe‘expertiseandrelevanceofthematerial’sauthor’.Inotherwords,resourcescreatedbypeoplewithmanyfollowersshouldbegivenhigherprioritythanthosewithfewerfollowers.Theyderivedthefollowingparameters:

• followers:materialsofauthor’swithmanyfollowersshouldhavehigherpriority

• community:materialsofauthorswithGeoGebracommunitybadgesshouldhavehigherpriority

• publicmaterials:materialsofauthorswithmanypublicmaterialsshouldhavehigherpriority

• teacher:materialsofteachersshouldhaveslightlyhigherprioritythanmaterialsofotherusers

Thefinalfactor–userfactor–isareal-timemeasureusedatthepointofsearching,andisdependentupontheuserdoingthesearching.Theparametersthatwerederivedare:

• language:materialsinthesamelanguageastheuser’slanguageshouldhavehigherpriority

• userID:thematerialsofthesearchingusershouldhavehigherprioritythanmaterialsfromotherusers

Bycombiningthesenewfactorswiththeequationspreviouslyusedtodoatext-relevancesearch,usingnormalizationfunctionsandweighingfactorsderivedfromanalysingtopsearchresults,GeoGebrahasbeenable,theybelieve,toimprovethequalityofsearchresults.Thisis,theysay,stillverymucha‘workinprogress’andtheyareworkingtofurtherimprovethequalityofresultsbydelvingfurtherintothedetailofthevariousparametersestablishedforeachoftheadditionalfactors.Theyarealsoawarethattheroutetheyhavetakenmighthavethedetrimentaleffectofmakinggoodqualitymaterialscreatedbynewauthorslessvisible,andmakeittougherfornewauthorstoestablishafootholdintheplatform.

30

Case Study 3: Photodentro

AcriticalcomponentoftheGreeknationalinfrastructureforK12educationisthePhotodentroplatformwhichoffers5OERrepositoriesmakingdigitalcontentavailabletoallschoolsinthecountry.Theplatformactsasacentralaccesspointtoschool-basedlearningresourcesintheGreekeducationsystem.Italsoactsasanationalaggregator,aserviceforharvestingeducationalmetadatafromexternalcollectionsandrepositories.Theplatformnowservesastheparamountdigitalwork-spaceformorethan120,000teachersandmorethan1,000,000studentsinthecountry’sprimaryandsecondaryschools.Greecehasestablisheddigitaleducationalcontentasoneoffivefundamentalpillarsintheeducationsystem,theotherfourbeing:

• In-serviceteachertraining• Thedigitalclassroom(classroomsequippedwithinteractiveteaching

systemsandsupportedbyanation-wideschoolnetwork)• Horizontalsupportactions• Electronicadministrationplatformscomplementthegroupofpillars.47

Atthemetadatalevel,qualityinPhotodentroisassuredthroughadherencetothePhotodentroIEEELOMGRApplicationProfile.However,thetaskofassuringthequalityofthecontentintherepositoriesisachievedbymeansofthePhotodentroQualitySeals,whichis:

“…anewrepositoryofthePhotodentrofamilytosupportaqualityassuranceschemeforitscontentbasedonQualitySeals.AQualitySealcanrepresenteitheraprocess,orasetofevaluationcriteria,orabrandname.TherepositoryhostsandmanagesQualitySeals,stampingrecordsforOERs,whileithasaRegistryofCertifiers.”48

Thereisastrongemphasisacrosstheplatformonmetadataqualitysothatthesearchforandretrievalofcontentisaseffectiveaspossible.TheQualitySealsconceptisavaluableattempttobuildcontinuousmetadatacurationintothe

47Megalou,EandKaklamanis,COpenContent,OERRepositories,InteractiveTextbooks,andadigitalSocialPlatform:TheCaseofGreecep.1https://www.academia.edu/37054610/OPEN_CONTENT_OER_REPOSITORIES_INTERACTIVE_TEXTBOOKS_AND_A_DIGITAL_SOCIAL_PLATFORM_THE_CASE_OF_GREECE48Megalou,EandKaklamanis,Cibidp.2

31

platform,andisonethatshouldofferamodelthatotherrepositoriesmightbenefitfromtoo.QualityinmetadataismaintainedinPhotodentrothroughtheworkofasub-groupofteacherswhohavebeentaskedwithannotatingallcontentwithclearandrobustmetadataandpublishingit.Theyworktoaclearlyspecifiedandstandardizedprocessthatsetsouttheprocedurebywhichcoordinatorsassignresourcestoteamsallthewaythroughtofinalpublication.TheseteamsaresupportedbytrainingcoursesandtrainingmaterialsintheformofFAQs,guidesandmanuals.Beforetheprocessiscompleteforanyindividualresourceorcollectionofresources,afinalcheckisconductedbythecoordinatorofthecollection.TheGreekeducationsystemishighlycentralisedandalltheresourcescontainedinPhotodentroLORaredevelopedby120project-employedfully-qualifiedteachersworkingacross10domain-specificworkgroups,withtheaimofaugmentingandsupplementingthespecifiedGreektextbookswithhigh-qualitydigitalinteractiveresources.Eachgroupisoverseenbyanacademicwithsignificantdomainandpedagogicalexpertisesoastoensurethequalityofthelearningobjects.Thisisaverydifferentmodelfromthatemployedby,forexample,byKlasCement,Currikiandothers,whichuseamuchbroadersystemofuser-generatedcontent.Differentmodelsareneededfordifferentcircumstances,however,andtheGreekmodelinPhotodentrohasmuchtocommenditforacentralisedsystemofeducationsuchasthatwhichappliesinGreece.DIFFERENT APPROACHES TO QUESTIONS OF QUALITY IN OER Thesethreeshortcasestudiesarebynomeansintendedtoofferacomprehensivetakeonhowrepositoriesshouldseektooffertheirusersthehighestqualityoflearningmaterialspossible.Thatcannotbeachievedinashortpapersuchasthis.Howeveritdoesdemonstratethreeapproaches(ofmanypossibleapproaches)thateachoftherepositoriesinquestionhavetakentothisissue.Eachisderivedfromadifferentsetofassumptions,buteachisavalidattempttobuildsomelevelofqualityassuranceintotherepositories.Itisthecaseofcoursethat,whilesomemightseektodevelopuniversalsolutionsorglobalapproachestoqualityinOER,attheendofthedayitisuptoeachrepositorytodeterminehowbestitcanachieveitsobjectivesonbehalfofitsusers,andthatbestfitswiththeveryparticularcontextinwhichitexists.

32

CONCLUSION: A CRITICAL EVALUATION OF QUALITY CRITERIA Qualityliesattheheartofthispaper.Definitionsofqualityinrelationtoeducationalresources,openorotherwise,abound.Ofallthepossibledefinitions,theonethatseemstoresonatemostwiththeneedsofeducatorswhowishtosearchforanduseOERisthatof‘fitnessforpurpose’.Itistheoneaspectofthevariousdefinitionsonofferthatmostreadilybearsscrutinyfromtheperspectiveofateachersearchingfor‘goodquality’resources.Thislinksbacktothefirstofthefive‘qualityfactors’outlinedbyCamillerietalonp.11.Fromtheuser’sperspective,thebestresourcetheycanfindwillbetheonethatbestfulfilstheirteachingpurposeand,theyhope,thelearningpurposeoftheirstudentstoo.Inthissense,qualityisviewedasa‘before-the-event’conceptwithrespecttothe‘QualityCircle’describedinthispaper.Theteacherwithaclearideaofthepurposeofhisorherteachingwillbeabletoevaluateanyresourcestheyfindmoreeffectivelythantheteacherwithonlyavagueideaofwhattheyhopetoachieveinaparticularlessonorcourse.Forexample,agoodprofessionalteacherwillknowwhatknowledgeorskillstheywillbeteaching,whethertheyaretryingtobuildonwhatstudentsalreadyknoworsimplyreinforcingexistingknowledgeorskills,howstimulating,enjoyableandchallengingtheywantthelearningtobe,whatmethodologiestheyintendtoemploy,whatkindofenvironmentforlearningtheyaretryingtocreate,whatkindofassessmentandevaluationtheywilluse,andwhatpersonalqualitiestheyhopetoinstilintheirstudents.Thisisonlyasubsetofallthepotentialpurposesateachermighthaveinmindforalesson,buttheteacherwhohasaclearideaofpurposewillbebetterabletoselectappropriateandrelevantcontent.FromtheperspectiveoftheOERrepository,therefore,themoretheyareabletoprovidemetadata,descriptions,commentandratingsthatgivetheteachertheinformationtheyneedtoconnectaresourcetotheirteachingrequirements,themoreeffectivelytheteacherwillfindcontentthatis‘fitforpurpose’.Assuch,tryingtodeliveracriticalevaluationofqualitycriteriaisnotsomethingthatcanbeachievedinanyuniversalsense:suchevaluationcanonlyreallybecarriedoutonapiece-by-piecebasisbytheteacherastheylocateandusecontent.However,withthisasthekeyyardstickbywhichtomeasurequality,OERscanperhapsmoreeffectivelystructuremetadataandfeedbacksystemstohelpteacherstryingtoassessthe‘fitnessforpurpose’ofresources.Whentheissueunderconsiderationisthatofquality–andparticularlytheissueofhowarepositorydealswithratingthequalityofitsresources-thekeychallengetendstobearoundthebehaviourofitsusers:howcaneducatorsandotherswhoseekaccesstoopenresourceseitherbepersuadedtogivefeedbackonresourcestheyhavealreadyusedorgainsomeawarenessofaresource’squalitybeforetheymakeuseofit?

33

SUGGESTED QUESTIONS FOR FURTHER RESEARCH Fromthereadingandresearchcarriedoutforthisshortreview,itispossibletoidentify,inconclusion,afewkeyissuesthatariseoutofacriticalevaluationoftheevidenceoutlinedinthispaper.Theycanbesummarisedthus:i. IntheEuropeancontextinparticular,therealchallengeofmultiple

languagesexists.Thelanguageofsearchandthelanguageoftheresourcesthemselvesisacriticalqualitativefactortobetakenintoconsiderationbyrepositories.Usersareoftenhavingtosearchforresourcesinasecondoreventhirdlanguagebutitisclearthatsomerepositoriesaremakinggoodprogresstowardsdealingwiththischallenge.ThisisachallengethatisalreadybeingexploredintheEUNProject‘eQNET’49,whichisexaminingthecriteriabywhichsomeresources‘travelwell’acrossnationalandculturalboundarieswhileothersdonot50.

ii. WhileusersmaybedownloadingresourcesfromOERrepositories,

theyarenotalwayscommentingonissuesofusefulnessandqualitywithintherepositoriesthemselves.Atleastonepossiblereasonforthis,andthatmeritssomeinvestigation,isthat,insomecountriesatleast,teachersoftensetuptheirownonlinegroupsandcommunitiesinwhichtoshareideas,resources,lessonplansandsoon.Thiscreatesakindofhinterlandbeyondtheopenrepositoriesandopencommunitiesinwhicheducatorsarecertainlymakinguseofopenresourcesbuttheyaredealingwithqualityissueswithinclosedorexclusivegroups,oftenataregionalornationallevel.Thereislittleresearchtodatetoverifythisassertion,whichisbasedonexperienceandobservation.

iii. Thepotentialcontentionbetweentwounderstandingsofqualityin

relationtolearningresourceshastobedealtwithpragmaticallybyrepositories.Forthebenefitofeducatorsseekingresourcesfortheirteachingpractice,theirprimeconsiderationwillalwaysbefinding‘useful’resources,resourcesthatmatchtheirpedagogicalrequirements.Intermsoflearners,however,thetruequalityofaresourcewillalwaysbeitsultimateimpactonthelearning,adeterminationthatcanonlybemadeafter-the-fact,andoftenwillbeadeterminationthatisapplicableonlyinthelimitedsituationdefinedbytheactuallearningactivityinwhichtheresourcewasused.

iv. Wefeeltheconceptofthe‘QualityCircle’inrelationtoOEPisaconceptworthdevelopingbeyonditsinitialmodestoutlinehere.TheresearchbasefordefinitionsofqualityinrelationtoOERisdividedtosomeextent,therefore,betweenthosewhowouldwishtoacceptWiley’sdefinitionofquality(outlinedonpage11ofthispaper)asparamountandthosewho,inlookingatthedomainfromateacher’sperspective,

49http://eqnet.eun.org/web/guest;jsessionid=E20318079722343915D3DFAE80661EF850http://lreforschools.eun.org/web/guest/travel-well

34

understandtheneedforsomepre-determinationof‘quality’sothat,whenasearchismadeforteachingresources,teacherscanbereasonablysureoffindingsomethingappropriateanduseful.

v. Furtherinvestigationcouldbemadeofthoserepositoriesthatmosteffectivelybuildpracticalandusableuser-feedbacksystemsintotheirplatforms.Theseseemmostlikelytoofferacredibleandtrustedservicetousers.Whetherthesearesimplestarratingsystemsorsomethingmorecomplex,theyseemtoworkbestwhenitistheusersthemselveswhoareabletoplayaroleindeterminingquestionsofqualityinrelationtothelearningresources.

vi. WorkcouldbedonetodeterminehowmanyOERrepositoriesareofferingsuchfunctionalitynowandhowsuccessfultheyareintheirimplementations.ItisunfortunatethatCurrikiinparticularisalmostexclusivelyaimedataUSaudiencedespiteitsstatedglobalaspirations–perhapstheycanbepersuadedtoallowtheirresourcestobemappedtocurriculafrombeyondAmericanshores,althoughwhethertheresourcesaresuitableforanon-Americanaudiencehastobeaskedtoo.

vii. Theconceptof‘collectiveintelligence’usedbyDeOliveiraetalisonethathasmuchtocommendit.SystemssuchasCurrikiandMERLOT,whilenotnecessarilyfollowingalltheconstructsofferedbytheBraziliansystemdescribed,doundoubtedlyofferanapproachtoqualityassurancethatisbuiltaroundthecollectiveviewsexpressedbyusersthroughtheuseofratingsystemsand,asaresult,offeredinthesearchfiltersthatenableuserstorankfindingsaccordingly.

viii. TheadaptationsmadetothesearchfunctionwithinGeoGebraintermsofwhattheycallthematerialfactor,theauthorfactorandtheuserfactorispersuasiveandisperhapssomethingthatotherlargerepositoriescouldtakeonboard.Theyarechangesthatcanonlybeeffectivelyappliedtorepositorieswithverylargecollectionsandaverylargeusercommunity.Again,itmightbeviewedasarealapplicationofameasureofcollectiveintelligencetothesearchtoolwithintheplatform.Thatsuchamendmentsbringtheirownissueswiththem,forexampleintermsofpossibleexcludingnewauthorsfromsearchresults,issomethingthat,asGeoGebraacknowledges,requiresfurtherresearch.

ix. Thecomplexityandlevelsofbureaucracybuiltintoarepositorywillhaveabearingonitseffectiveness.Teachersareunlikelytouseacomplicatedfeedbacksystem.

x. IfarepositoryistocompetewithGoogleanditspeers,itmustbeabletomatchthemwithinitsownlimiteddomainintermsofeaseofuse,flexibilityandpower.Therepositoryneedstobeextensiveenoughtoofferalargeandenticingcacheofresourcesfortheeducatortowishtoreturntoagainandagaintomineforcontent

35

xi. Somefurtherworkmightbedoneonaddingbreadthanddepthtotheconceptof‘efficacy’asitrelatestothequalityofeducationalcontent.Thefitness-for-purposeofaresourceiscriticalinhelpingteacherstofindexactlywhattheyneedforaparticularlessonorcourse.Theperceivedorstatedefficacyofaresourcesmightbethemostappropriatefactortotakeintoaccountinthinkingabout‘quality-before-the-fact’inrelationtoopeneducationresources.

Finally,whilethispaperdidnotventurefarintothequestionofArtificialIntelligenceandMachineLearning,itisreasonabletosuggestthatthisisanareathatwillprovetobeincreasinglyimportantaswemoveforwardtothenextgenerationofOERrepositories.Anyresearchquestionsarisingoutofthisareawillhavetoincludenotonlytechnicalandpedagogicalissuesbutalsoquestionsofethics,fairnessandresponsibilitytoteachers,learnersandresearchersthemselves.QualityisundoubtedlyadifficultconcepttodefineinrelationtoOER.ItisnonethelessadifficultythatmustbetackledifweallwishtoseetheOpenEducationmovementgofromstrengthtostrengthinthefuture.

36

APPENDIX: ACTIONABLE POINTS & RECOMMENDATIONS Alistof‘actionablepoints’havebeenextractedfromthebodyoftheWhitePaperandhavebeencollectedhereasafinal‘aidememoire’forthoserepositoriesthatareseekingtoimprovetheservicetheyoffertheirusers.Attheheartofthelistbelowisaneedalwaystostartfromtheperspectiveofthelearnerandtheteacher.Thesimpletruthusthatanyrepository,nomatterhowbigitmightbe,thatdoesnothaveaworkingconceptionofitsuserbaseandoftheirrealteachingandlearningneedsisunlikelytobeabletoservetheirneedstrulyeffectively.Eachoftheactionablepointsbelowcomeswiththerelevantpagenumberinthepapersothatthepointcanbeplacedinitscontexteasily.

• P.4o Repositoriesshouldalsoseektogiveteachersmechanismsthat

willallowthemtoprovideprofessionalfeedbackonthevalueandusefulnessoftheresourcesoncetheyhavedeployedthemintheirteaching.

• P.9o Thesimplisticnotionthatqualitycanonlybedeterminedbythe

impactofaresourceonlearningisnotsufficientforrepositories.Ittakeslittleaccountoftherealitiesofateacher’sneedsinseekingoutgoodcontent.

• P.9-10o TheconceptofaQualityCirclemightbeconsciouslyimplemented

inrepositoriesinordertodevelopamodelofcollectiveintelligencerelevantandappropriate,butthereisnoone-size-fits-all.

• Culledfromvarioussectionsofthepaper,thisisalistofpotentiallyrelevantfactorsinarepository’squalityassurancesystem

o P.11§ Efficacy§ Impact§ Availability§ Accuracy§ Excellence

o P.12-13§ Featuredresources§ Userevaluationtools§ Peerreview§ Authorship§ Keywords§ Metadata§ Multilingualsupport§ SocialMediasupport§ CreativeCommonsLicences§ SourceCodeorOriginalFiles

o P.15-16

37

§ OEREvaluationCriteriaClarity,Comprehensibility,andReadability

§ ContentAccuracyandTechnicalAccuracy§ AdaptabilityandModularity§ Appropriateness§ Accessibility§ SupplementaryResources

o P.16§ Whataretheeffectsofopeneducationalresourceson

coursedevelopmentandimplementation?§ Onstudentlearning?§ Whatarethecharacteristicsofcreatorsandearlier

adopters?§ Willstudents’priorexperiencewithOERaffectmotivation

toindependentlylocateanduseotherfreematerialsforlearning?

§ Howwillthisinfluenceachievementandsuccess?o P.19

§ Correspondencewiththekeywordinthesearchbar§ Scoresgivenbyusers(andthenumberofusersthatgive

thescore),§ Timeonline§ Typeofcontributor(teacher,student,companyetc)

o P.24§ Arankingsystemthatatleastletsusersseehowmany

timestheresourcehasbeenopened

38

BIBLIOGRAPHY

1. AbderrahimElMhouti,AzeddineNasseh,MohamedErradi,Howtoevaluatethequalityofdigitallearningresources?InternationalJournalofComputerScienceResearchandApplication2013,Vol.03,Issue.03,pp.27-36

2. Anderson,L.W.,Krathwohl,D.R.:Ataxonomyforlearning,teaching,andassessing:arevisionofBloom’staxonomyofeducationalobjectives.Longman,NewYork(2001)

3. Atenas,J.andHavemann,Leo(2013)Qualityassuranceintheopen:anevaluationofOERrepositories.INNOQUAL-InternationalJournalforInnovationandQualityinLearning1(2),pp.22-34.ISSN2294-9763.

4. Atkins,D.,Brown,J.,&Hammond,A.(2007).AReviewoftheOpenEducationalResources(OER)Movement:Achievements,Challenges,andNewOpportunities.CommonwealthofLearning.(2011).

5. Atkins,D.,Brown,J.,&Hammond,A.(2007).AReviewoftheOpenEducationalResources(OER)Movement:Achievements,Challenges,andNewOpportunities.

6. Barker,E.,&Ryan,B.(2003).TheHigherLevelSkillsforIndustryRepository,1–5.Retrievedfromhttp://metadata.cetis.ac.uk/implementations/usage_survey/cs_hlsi.pdf

7. Baumgartner,PImprovingReusabilityofOEREducationalPatternsforContentSharing,Retrievedfrom:https://www.semanticscholar.org/paper/Improving-Reusability-of-OER-Educational-Patterns-Baumgartner/f9af6537db6face09185264cda711597a5100621

8. Bienkowski,M.,&Klo,J.(2014).TheLearningRegistry:ApplyingSocialMetadataforLearningResourceRecommendations.InRecommenderSystemsforTechnologyEnhancedLearning(pp.77-95).SpringerNewYork.

9. Billon,N.,Nistor,A.,Mihai,G.,Grizelj,A.,Myrtsioti,E.,Miklasinska,O.,Pocze,B.&Gras-Velazquez,A.(2019).ScientixUpdate-September2019.EuropeanSchoolnet,Brusselshttp://files.eun.org/scientix/scx3/publications/Scientix3_Final_Publication_web.pdf

10. Brent,I.(2012).ObstaclestocreatingandfindingOpenEducationalResources:thecaseofresearchmethodsinthesocialsciences.MediainEducation,1–17.Retrievedfromhttp://www-jime.open.ac.uk/jime/article/viewArticle/2012-05/html

11. Broadbent,O&McCann,EThinkUp,Developmentofe-learningresources,ISBN:978-1-909327-14-6©RoyalAcademyofEngineering2016

12. Brodahl,C.&Smestad,B.ATaxonomyasaVehicleforLearning.InterdisciplinaryJournalofE-LearningandLearningObjects,(2009).5,111-127.

13. Brodahl,C.&Smestad,B.TheMathWizProject.Design,Technologies14. Brooks,D.&McCormack,M.(2019)HigherEducation’s2019trendwatch

andtop10strategictechnologies.Researchreport.Louisville,CO:ECAR.Roth,D.(2019).

39

15. Browne,T.,Holding,R.,Howell,A.,&Rodway-Dyer,S.(2010).ThechallengesofOERtoAcademicPractice.JournalofInteractiveMediainEducation,1–15.Retrievedfromhttp://jime.open.ac.uk/jime/article/viewArticle/2010-3/html

16. Butcher,NEditedbyAshaKanwar(COL)andStamenkaUvalic´-Trumbic´ABasicGuidetoOpenEducationalResources(OER),(UNESCO)ISBN978-1-894975-41-4

17. Clements,K.I.,&Pawlowski,J.M.(2012).User-orientedqualityforOER:understandingteachers’viewsonre-use,quality,andtrust.JournalofComputerAssistedLearning,28(1),4–14.doi:10.1111/j.1365-2729.2011.00450.x

18. ConsultingLtd,UK:PaulBacsich,withadditionalresearchfromGilesPepler,SaraFrankAdultEducationandOpenEducationalResourcesEuropeanUnion,2015.Retrievedfrom:http://www.europarl.europa.eu/supporting-analyses

19. CorneliaBrodahl,EvaluationofActiveLearningMaterials-Retrievedfrom:https://home.uia.no/cornelib/hjem/conferences/ALICT/ALICT-%20Evaluation%20of%20Active%20Learning%20Materials-.pdf

20. Davis,H.C.,Carr,L.,Hey,J.M.,Howard,Y.,Millard,D.,Morris,D.,&White,S.(2010).Bootstrappingacultureofsharingtofacilitateopeneducationalresources.LearningTechnologies,IEEETransactionson,3(2),96-109.

21. Davis,H.C.,Carr,L.,Hey,J.M.,Howard,Y.,Millard,D.,Morris,D.,&White,S.(2010).Bootstrappingacultureofsharingtofacilitateopeneducationalresources.LearningTechnologies,IEEETransactionson,3(2),96-109.

22. DelosArcos,B.(2014).FlippingwithOER:K12teachers’viewsoftheimpactofopenpracticesonstudents.InProceedingsofOpenCoursewareConsortiumGlobal2014:OpenEducationforaMulticulturalWorld.

23. DeVries,IEvaluatingopeneducationalresources-Lessonslearned,2ndWorldConferenceonEducationalTechnologyResearches–WCETR2012,ThompsonRiversUniversity,900McGillRd.,KamloopsV2C0C8,Canada,Retrievedfrom:https://www.oerknowledgecloud.org/archive/1-s2.0-S1877042813010422-main.pdf

24. Downes,S.(2007).ModelsforSustainableOpenEducationalResources.InterdisciplinaryJournalofKnowledgeandLearningObjects,3,29-44.

25. Downes,S.(2007).ModelsforSustainableOpenEducationalResources.InterdisciplinaryJournalofKnowledgeandLearningObjects,3.Retrievedfromhttp://www.oecd.org/edu/ceri/36781698.pdf

26. Ehlers,U.-D.(2007).QualityLiteracy-CompetenciesforQualityDevelopmentinEducationandE-Learning.EducationalTechnology&Society,Vol.10,No.2.

27. Farrow,Robert;Pitt,Rebecca;delosArcos,Beatriz;Perryman,Leigh-Anne;Weller,MartinandMcAndrew,Patrick(2015).ImpactofOERuseonteachingandlearning:datafromOERResearchHub(2013–2014).BritishJournalofEducationalTechnology,46(5)pp.972–976

28. Friesen,N.(2010).UsingFreeandOpenOnlineResources.Retrievedfromhttp://smallcities.tru.ca/index.php/cura/article/view/22.

29. Fulantelli,G.,Gentile,M.,Taibi,D.,&Allegra,M.(2008).TheOpenLearningObjectmodeltopromoteOpenEducationalResources.Journal

40

OfInteractiveMediaInEducation,1.RetrievedAugust23,2013,fromhttp://jime.open.ac.uk/article/2008-9/344

30. Greenberg,J.(2009).Theoreticalconsiderationsoflifecyclemodeling:ananalysisofthedryadrepositorydemonstratingautomaticmetadatapropagation,inheritance,andvaluesystemadoption.Cataloging&classificationquarterly,47(3-4),380-402.

31. GuidelinesforOpenEducationalResources(OER)inHigherEducation(p.22).Vancouver.Retrievedfromhttp://www.col.org/resources/publications/Pages/default.aspx

32. Hamdan,N.,McKnight,P.,McKnight,K.&Arfstrom,K.M.(2013).AReviewofFlippedLearning.www.flippedlearning.orgaccessedMarch17,2014.

33. Hilton,J.I.,Wiley,D.,Stein,J.,&Johnson,A.(2010).Thefour“R”sofopennessandALMSanalysis:Frameworksforopeneducationalresources.OpenLearning:TheJournalofOpen,Distanceande-Learning,25(1),37-44.

34. Hoel,T;Pawlowski,JMHowwillthedigitaltextbookof2030solveMeno’sparadox,OPENEDUCATION2030.JRC-IPTSCALLFORVISIONPAPERS.PARTII:SCHOOLEDUCATION

35. Holden,C.(2003).Fromlocalchallengestoaglobalcommunity:learningrepositoriesandthegloballearningrepositoriessummit.TheAcademicADLCoLab,0–30.Retrievedfromhttp://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.97.279&rep=rep1&type=pdf

36. Hylén,J.(2006).OpenEducationalResources:OpportunitiesandChallenges.Paris.Retrievedfromhttp://oecd.org/edu/ceriMIT.(2011).OurHistory.Retrievedfromocw.mit.edu/about/.OpenCoursewareConsortium.(n.d.).AbouttheOpenCoursewareConsortium.RetrievedNovember15,2011,Retrievedfromhttp://www.ocwconsortium.org/en/aboutus/abouttheocwc.

37. Hylén,J.(2006).Openeducationalresources:Opportunitiesandchallenges.ProceedingsofOpenEducation.Retrievedfromhttp://library.oum.edu.my/oumlib/sites/default/files/file_attachments/odl-resources/386010/oeropportunities.pdf

38. Jacobi,R.,&Woert,N.vander.(2012).Trendreport:OpenEducationalResources2012.SpecialInterestGroupOpenEducationalResources.Retrievedfromhttp://www.surf.nl/en/publicaties/Documents/trendrapport%20ER2012_10042012(ENGELSLR).pdf

39. Jung,I.,Sasaki,T.&Latchem,C.A framework for assessing fitness for purpose in open educational resources in IntJEducTechnolHighEduc(2016)13:3.https://doi.org/10.1186/s41239-016-0002-5

40. K.Clements,J.Pawlowski,N.Manouselis(2014)WhyOpenEducationalResourcesRepositoriesFail–ReviewofQualityAssuranceApproaches,EDULEARN14Proceedings,pp.929-939

41. Kanwar,A.,Uvalic-Trumbic,S.&Butcher,N.(2011)Abasicguidetoopeneducationalresources(OER).UNESCO.Retrievedfromhttp://unesdoc.unesco.org/images/0021/002158/215804e.pdf

42. Krajcso,ZClassificationandqualitycriteriaforOpenEducationalResourcesinthefieldofforeignlanguagelearning,Retrievedfrom:

41

https://www.degruyter.com/downloadpdf/j/jolace.2016.4.issue-1/jolace-2016-0004/jolace-2016-0004.pdf

43. Kroop,S;Mikroyannidis,A;Wolpers,MEditors,ResponsiveOpenLearningEnvironments:OutcomesofResearchfromtheROLEProject,SpringerOpenRetrievedfrom:https://link.springer.com/content/pdf/10.1007%2F978-3-319-02399-1.pdf

44. Kurilovas,E,Bireniene,VandSerikoviene,S.”MethodologyforEvaluatingQualityandReusabilityofLearningObjects”TheElectronicJournalofe-LearningVolume9Issue12011,(pp39-51),availableonlineatwww.ejel.org

45. Lund,T.andHojsholt-Poulsen,L.(2010),“BuildingSuccessfulEducationalRepositories–Consolidatedrecommendationsreport”,EdReNeprojectdeliverable,eContentPlus

46. McGreal,R.(Ed.).(2004).Onlineeducationusinglearningobjects.Routledge.

47. Megalou,E&Kaklamanis,COpenContent,OERRepositories,InteractiveTextbooks,andadigitalSocialPlatform:TheCaseofGreeceRetrievedfrom:https://www.academia.edu/37054610/OPEN_CONTENT_OER_REPOSITORIES_INTERACTIVE_TEXTBOOKS_AND_A_DIGITAL_SOCIAL_PLATFORM_THE_CASE_OF_GREECE

48. Massart,DandShulman,ELearningResourceExchangeMetadataApplicationProfileversion4.7,EuropeanSchoolnet,2011.Retrievedfrom:http://lreforschools.eun.org/c/document_library/get_file?p_l_id=10970&folderId=12073&name=DLFE-1.pdf

49. Megalou,E;Gkamas,V,;Papadimitriou,S;Paraskeva,M&Kaklamanis,COpenEducationalPractices:MotivatingTeacherstoUseandReuseOpenEducationalResoursesRetrievedfrom:https://www.academia.edu/30309880/OPEN_EDUCATIONAL_PRACTICES_MOTIVATING_TEACHERS_TO_USE_AND_REUSE_OPEN_EDUCATIONAL_RESOURCES

50. Megalou,E;Gkamas,V,;Papadimitriou,S;Paraskeva,M&Kaklamanis,COpenEducationalPractices:MotivatingTeacherstoUseandReuseOpenEducationalResoursesRetrievedfrom:https://www.academia.edu/30309880/OPEN_EDUCATIONAL_PRACTICES_MOTIVATING_TEACHERS_TO_USE_AND_REUSE_OPEN_EDUCATIONAL_RESOURCES

51. Morris,LVContemplatingOpenEducationalResources,Publishedonline:1August2019SpringerNatureB.V.2019

52. Nesbit,J.C.,Belfer,K.,&Vargo,J.(2002).AConvergentParticipationModelforEvaluationofLearningObjects.CanadianJournalofLearningandTechnology,(3),105-120.RetrievedOctober15,2008,fromhttp://www.cjlt.ca/index.php/cjlt/article/viewArticle/110/103

53. Neven,F.,&Duval,E.(2002).Reusablelearningobjects:asurveyofLOM-basedrepositories.ProceedingsofthetenthACMinternational,0–3.Retrievedfromhttp://dl.acm.org/citation.cfm?id=641067

54. Ochoa,X.,&Duval,E.(2009).Quantitativeanalysisoflearningobjectrepositories.LearningTechnologies,IEEETransactions,1–14.Retrievedfromhttp://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5184802

42

55. Ochoa,XAutomaticevaluationofmetadataqualityindigitalrepositories,Retrievedfrom:https://link.springer.com/article/10.1007/s00799-009-0054-4

56. OECD.(2007).GivingKnowledgeforFree:TheEmergenceofOpenEducationalResources.OECDPublishing.Retrievedfromhttp://www.oecd.org/education/ceri/givingknowledgeforfreetheemergenceofopeneducationalresources.htm

57. ORO(2014).OpenResearchOnline–TheOpenUniversity.http://oro.open.ac.uk

58. Palmer,C.L.,Teffeau,L.C.,&Newton,M.P.(2008).Strategiesforinstitutionalrepositorydevelopment:acasestudyofthreeevolvinginitiatives.LibraryTrends,57(2),142-167.

59. Pawlowski,J.M.,&Hoel,T.(2012).TowardsaGlobalPolicyforOpenEducationalResources:TheParisOERDeclarationanditsImplicationsWhatareOpenEducationalResources?Retrievedfromhttp://monet.informatik.rwthaachen.de/giotto/OpenScout_df6f1252-bfa6-11e1-a668-e13baff9bd23.pdf

60. Pawlowski,J.M.&Clements,K.I(2013)TrustedEducationalNetworksforInternationalizationofOpenEducationalResources,InternationalJournalforInnovationandQualityinLearning(INNOQUAL),Volume1,Issue1.(2013),ISSN:2294-9763

61. Pawlowski,J.M.(2007).Qualitymarke-learning:developingprocess-andproduct-orientedqualityforlearning,educationandtraining,InternationalJournalofLearningTechnology,Vol.3,Issue1(February2007).

62. Pegler,C.(2012).Herzberg,hygieneandthemotivationtoreuse:Towardsathree-factortheorytoexplainmotivationtoshareanduseOER.JournalofInteractiveMediainEducation,1–18.Retrievedfromhttp://www-jime.open.ac.uk/jime/article/viewArticle/2012-04/html

63. Pérez-Mateo,M.,Maina,M.F.,Guitert,M.,&Romero,M.(2011).Learnergeneratedcontent:Qualitycriteriainonlinecollaborativelearning.EuropeanJournalofOpen,DistanceandELearning.

64. Petrides,L.,&Nguyen,L.(2008).Openeducationalresources:inquiringintoauthoruseandreuse.InternationalJournalofTechnologyEnhancedLearning,1,98–117.Retrievedfromhttp://inderscience.metapress.com/index/9428665670616423.pdf

65. Pirkkalainen,H.,Jokinen,J.&Pawlowski,J.(2014)UnderstandingSocialOEREnvironments–aQuantitativeStudyonFactorsInfluencingtheMotivationtoShareandCollaborate.IEEETransactionsonLearning

66. Quinn,C.(1996).PragmaticEvaluation:LessonsfromUsability,http://www.ascilite.org.au/conferences/adelaide96/papers/18.html

67. RajivS.Jhangiani,RobertBiswas-Diener,OPEN:Thephilosophyandpracticesthatarerevolutionizingeducationandscience,(ISBN:9781911529002)

68. Richter,T.,&Ehlers,U.D.(2010).BarriersandMotivatorsforUsingOpenEducationalResourcesinSchools.OpenED,1–12.Retrievedfromhttp://openaccess.uoc.edu/webapps/o2/handle/10609/4868

43

69. Robertson,R.J.(2010).AreOERsjustRe-usableLearningObjectswithanopenlicense?kavubob'smiscellanea.http://kavubob.wordpress.com/2010/09/01/rlosoersopened/

70. Schuwer,R.,Wilson,T.,Valkenberg,W.Van,&Lane,A.(2010).ProductionofOER:aquestforefficiency.UniversitatObertaDeCatalunya,N.OpenUniversity

71. Skovsmose,0.(2001).Landscapesofinvestigation.ZentralblattfürDidaktikderMathematik,3(4),123-132

72. SquiresD.,&Preece,J.(1996).UsabilityandLearning:EvaluatingthePotentialofEducationalSoftware.Computers&Education,27(1),15-22.

73. Stacey,P.(2012).TheEconomicsofOpen.RetrievedMay15,2012,Retrievedfromhttp://http://edtechfrontier.com/.

74. Stenger,M.10Openeducationalresourcesyoushouldknowabout[Blogpost].(2018).Retrievedfromhttps://www.opencolleges.edu.au/informed/features/10-open-educational-resources-know/

75. Taylor,J.C.(2007).OpenCoursewareFutures�:CreatingaParallelUniverseIntroduction�:FromElitetoMassHigherEducationTheOpenEducationalResources(OER)MovementOpenCourseWareConsortium.October,1-9.

76. TechnologiesUNESCO.(2012).2012ParisOERDeclaration.WorldOERCongress,Paris,June2012.

77. TheNorwegianCentreforICTinEducation.(2012).QualityCriteriaforDigitalResources.Støtteskjemaforvurderingavpedagogiskegnethetogtekniskeogformellekravveddigitalelæringsressurser.Retrievedfromhttps://iktsenteret.no/ressurser/kvalitetskriterier-digitale-laeringsressurser

78. Thomas,C.,&McDonald,R.H.(2007).Measuringandcomparingparticipationpatternsindigitalrepositories.D-libMagazine,13(9/10),1082-9873.

79. Tuomi,I.(2006).OpenEducationalResources:WhattheyareandwhydotheymatterReportpreparedfortheOECD.Retrievedfromhttp://www.meaningprocessing.com/personalPages/tuomi/articles/OpenEducationalResources_OECDreport.pdf

80. U.S.GovernmentAccountabilityOffice,ReporttoCongressionalCommittees,Collegetextbooks(GAO-13-368).(2013).Retrievedfromhttps://www.gao.gov/assets/660/655066.pdf

81. UNESCO(2012).2012ParisOERDeclaration.WorldOERCongress,Paris,June2012.Retrievedfromat:http://www.unesco.org/new/fileadmin/MULTIMEDIA/HQ/CI/CI/pdf/Events/Paris%20OER%20%20Declaration_01.pdf

82. UNESCO.(2002).ForumontheImpactofOpenCoursewareforHigherEducationinDevelopingCountriesFinalreportUNESCO.Design,2002(July),1-3.

83. UNESCO.(2002).UNESCOPromotesNewInitiativeforFreeEducationalResourcesontheInternet.

84. UNESCO.(2011).GuidelinesforOpenEducationalResources(OER)inHigherEducation.CommonwealthofLearning.Retrievedfromhttp://www.col.org/PublicationDocuments/Guidelines_OER_HE.pdf

44

85. Wesner,KEuropeaneducation2030–open&free,integrativeandsustainable–avision.KooperativeBerlin,Retrievedfrom:https://blogs.ec.europa.eu/openeducation2030/files/2013/05/Wesner-OE-SE-2030-fin.pdf

86. White,D.,&Warren,N.(2011).OpenEducationalResources�:TheValueofReuseinHigherEducation(p.29).Oxford,UK.Retrievedfromhttp://www.jisc.ac.uk/media/documents/programmes/elearning/oer/OERTheValueOfReuseInHigherEducation.pdf.

87. Wiley,D.(2007).OntheSustainabilityofOpenEducationalResourceInitiativesinHigherEducation:PapercommissionedbytheOECD’sCentreforEducationalResearchandInnovation(CERI).OECD’sCentreforEducationalResearchandInnovation(CERI)(p.21).OECDPublishing.Retrievedfromhttp://www.oecd.org/dataoecd/33/9/38645447.pdf

88. Wiley,D.(2010).ImpedimentstoLearningObjectReuseandOpennessasaPotentialSolution.REVISTABRASILEIRADEINFORMÁTICANAEDUCAÇÃO,17(03),08.Retrievedfromhttp://ceie-sbc.tempsite.ws/pub/index.php/rbie/article/view/1022/1016

89. Windle,R.J.,Wharrad,H.,McCormick,D.,Laverty,H.,&Taylor,M.G.(2010).SharingandreuseinOER:experiencesgainedfromopenreusablelearningobjectsinhealth.JournalofInteractiveMediainEducation,(01),1–18.Retrievedfromhttp://jime.open.ac.uk/jime/article/viewArticle/2010-4/html

45

DrMarinaConnell JohnConnellDirectors,IAmLearnerhttp://iamlearner.net+441501772409+447930204672