1. Evaluation of Resources

LRMDS Assessment and Evaluation: Evaluation of Resources1

LRMDS Assessment and Evaluation

(A&E)Evaluating Learning

Resources


LRMDS lrmds pub 1.pub

“Ania man nga libro nga

makagayagay iti ubing nga

agbasa, tapno agbalin

daytoy a paset ti biag na,

ken makatulong daytoy ti

panagdur-as na,

ket nasayaat daytoy a

libro”


This session is intended to provide the basic information and instructions relating to criteria-based evaluation of resources. Participants will develop understanding of the Assessment and Evaluation guidelines, standards and specifications as they relate to the evaluation of learning, teaching and professional development resources.


Explain why having specifications and standards are essential to doing an evaluation of learning/teaching resources

Give a brief description of the contents of the following areas for which specifications have been developed:– IPR management– Educational Soundness– Educational Quality– Social content– Technical

Enabling Objectives


Terminal Performance Objective

To evaluate at least one resource and prepare a Request Brief for the action to be taken on the said resource.

Apply the appropriate evaluative criteria and instruments in the evaluation of specific resources.


Explain the purpose of the Request Brief and identify its key contents

Define the key terms/concepts in the LRMDS A & E system Guidelines

Enabling Objectives


The importance of standards and specifications for evaluation

The LRMDS Assessment & Evaluation System at the regions, divisions and lead school levels

Processes, workflow, personnelEvaluation specs, guidelines and

instruments

Content


Preparation of the Request Brief

Preparation of the Evaluation Plan

Evaluation of resources using appropriate criteria and instruments

Content


Introduction to the concepts of Evaluation, Standards, Specifications


Activity 1

Aka “Mirror, mirror on the wall, which goody is the tastiest of all?

(Refer to activity sheet #1)

Introduction to the concepts of Evaluation, Standards, Specifications


Instructions:

1.Divide the participants into three groups.

2.Give each team a bag of goodies, e.g., (banana cake, hopia and other snack food, at least 3-4 kinds, plastic spoons and forks, small paper plates)

Activity 1


The team should evaluate the food items and agree on a recommendation for each item.

Which of the items should be recommended for inclusion in the school cafeteria menu?

Are there snack items which may need reformulation before being considered for cafeteria distribution?

Each team must have a consensus on their recommendations. The recommendations of all the groups will be consolidated.

Each team should have a convener and a documentation officer.

Activity 1


Document your process and write your recommendations.

The assigned presenter will present the report of the group to the plenary.

The report must contain the following information:

(1)A description of the processes that the team used in evaluating the food items and in agreeing on its recommendations;

(2)The problems, issues which the group encountered in the process of evaluation;

(3)The group recommendations based on the evaluation.

Activity 1


Discuss the processes that the groups employed. • How did each group conduct the evaluation?

Intrinsic to evaluation are standards and the use of criteria to measure standards.

• What were the criteria and standards used by the groups?

Did the group try to evaluate without agreeing on the standards and criteria first?

• Why was it important to agree on the standards and criteria first?

Processing


Would your work have been easier if the processes and persons responsible for the processes were defined, if the areas for which standards or specifications were defined?

How did you resolve your differences?

Processing


What is evaluation?

Evaluation is the systematic determination of the merit, worth, and significance of something or someone; fit for purpose; using criteria against a set of standards

Evaluation, Standards and Specifications


A standard is an established norm or requirement.

Standards are developed or formulated by professional groups, associations, unions, guilds who have a stake in the product or service being evaluated.

Evaluation, Standards and Specifications


Sometimes the standards are formulated by private groups through formal consensus.

In many instances, especially when public welfare is concerned, government through its instrumentalities formulate the standards for the evaluation of a product or service and the standardization process is by edict, that is it is established by law or an executive issuance.

Standards


A specification is an explicit set of requirements to be satisfied by a material, product, or service.

A specification is a type of a standard. It provides the necessary details about the specific standards required.

Specifications


The Assessment and Evaluation System is for identification of materials to be procured, redeveloped or reproduced is designed to support schools, divisions, regions and central office:

in identifying learning, teaching and professional development resource needs

reviewing and recommending resources for acquisition and procurement

LRMDS Assessment and Evaluation System


identifying existing resources for reproduction and or redevelopment

evaluating teacher developed materials for local use and shared distribution via LRMDS

LRMDS Assessment and Evaluation System


Activity Instructions:

The activity is to level understanding of basic terms, concepts, processes and purposes for evaluation by the group before the evaluation of actual resources is undertaken.

Modified “Jeopardy” game – where the answer is not really the answer and the question is really the answer.

Activity 2(Refer to Activity Sheet #2)


Read the Guidelines and the Handouts to get the definitions of terms and or concepts that you will encounter in evaluation of resources.

Work in pairs. Each pair should write on one side of the meta strip the definition of the term; on the other side, write the question to which the definition is the answer.

Jeopardy


For example: • “ Is an opening, or rupture, in a planet's surface

or crust, which allows for the release of hot ash, and gases.”

The answer is “A VOLCANO”• “Curriculum, subject matter, skills and

processes, age and stage of schooling, cultural inclusivity and language of instruction.”

The answer is “Educational Specs ”

Jeopardy


Accessibility Specifications Intellectual Property Rights Specifications Storage and Distribution Publication and Use Technical Specifications Assessment and Evaluation QA Framework

Define/explain


Work as a group.

You are given meta strips with definitions, terms and explanations.

Read and discuss with your group as to where the description belong or best describe the terms posted on the board.

40 minutes

Jeopardy/ Matching Game


# Terms Description1 Accessibility Specification Geospatial, physical and intellectual ability, special needs:

physical, intellectual differences, flexible operation and presentation.

2 Intellectual Property Rights specification

Author, contributors, copyright, access and distribution, equitable: negotiation is balanced between rights owner and user.

3 Storage and distribution Version control for management and maintenance, meta data, version control, discovery, publish in varied formats.

4 Publication and use File types, formats for reproduction and usage permissions, communicated information contains, educational purpose, format.

5 Technical Specifications Format, file size, authoring software, user plug-ins, players, software interoperability.

6 Assessment and Evaluation

Identifying learning, teaching and professional development resource needs, reviewing and recommending resources for

acquisition and procurement.7 QA framework Supports LRMDS personnel to identify and consistently implement

required actions to achieve agreed standards and specifications.


Purpose of the activity: assist the participant manage his learning through a collaborative activity.

Establish a common understanding/working definition of terms to be used in subsequent activities.

Quick Processing of the activity


Introduction to QA and Criteria based evaluation

30

What is Quality Assurance?• Quality refers to the totality of characteristics of a product

or service that bears on its ability to satisfy stated and implied needs.

• Principles provide the frame of reference for achievement of

quality.• Measures & requirements base level descriptors.• Criteria operational requirements for quality.• Indicators concrete objectives for criteria.


Ensuring Quality

LRMDS Quality Assurance Framework describes the parameters to ensure that all learning, teaching and

professional development resources that are catalogued and accessed or located via the LTMDS

web portal are of high educational and technical quality.


Ensuring Quality

QA is a coordinated process for deliberately bringing about the attainment of the desired quality. It is a systematic process of monitoring all management, organizational, productivity and services of the LRMD systems against

standards of quality in order to raise performance.


Ensuring QualityLRMDS Quality Assurance Framework describes the processes that ensure

that learning, teaching and profession development resources are:• Relevant to the needs and expectations of the users;• Accessible by the range of technology available to the users;• Usable and functional, and;• Pedagogically sound and cohesive.


The LRMDS Quality Assurance Framework has been developed to assist:

• LRMDS Managers to manage, maintain and continuously improve access to quality resources,

• LRMDS development and production teams to comply with agreed standards and specifications for educational resources.

• Key LRMDS personnel within Regions and Divisions to implement QA processes to ensure the continued provision of quality services and products

• Schools and teachers to be able to identify quality resources in relation to educational need

Purpose

35

Quality Framework

Principles:

Characteristics

Criteria & indicators

Specificationsand/or

Requirements

GuidelinesProceduresInstruments

(tools/templates)

Area of Quality:

ApplicationReview

A process of continuous improvement


LRMDS Quality Assurance measures & requirements:

• Clearly documented and transparent processes• Well defined standards, specifications and

evaluation criteria• Well trained personnel• Capacity of the LRMDS system to respond to

user need.• High quality products and services

Measures & Requirements


The main principles underlying the LRMDS quality assurance processes are:

User centred• There is active and integrated involvement of LRMDS users at all

levels school, division and region.

Responsive

• LRMDS product and service delivery is informed by user requirements and feedback for each of the LRMDS sub-systems. LRMDS employs the use of the most appropriate media for the content, taking into account the profiles of schools, teachers, learners and access to technologies.

Standards based• LRMDS inputs, processes and outputs conform to agreed quality

standards and specifications.

Principles


Under the microscope:LRMDS QA


LRMDS Assessment & Evaluation QA


Evaluating ResourcesPurpose of the Evaluation: To select resources to be:• A. Catalogued in the LRMDS portal• B. Uploaded to the portal and catalogued

Types of Resources: • Print• Digital print• Online websites• Video based• Interactive resources

Activity 3(Refer to activity sheets #3)


Refer to the instruments and point out to the mandatory requirements stipulated in the instrument.

Ask participants to read them out aloud.

Instruments used and their purposes


Explain to the participants that in LRMDS, the instruments will be used in various streams of evaluation, namely,

Evaluation of existing resources in existing format for reproduction and cataloguing: digital and non –digital

Evaluation of existing resources for redevelopment: digital and non–digital

Evaluation of teacher-developed resourcesEvaluation of On-line resources and sites

Evaluation Streams


• Determine which of the Evaluation instruments should be used.

• Refer to Section 6 Guidelines and Processes for LRMDS Assessment and Evaluation.

• Use the available copies of the instruments for the actual conduct of the Evaluation

• Each group evaluates at least 1 resource using the instruments.

Activity 3


• All resource evaluations should commence with the IPR review. When many resources belonging to a series are to be evaluated then the IPR can be completed once for all.

• Any DepED owned and copyright resources for which content errors, accuracy issues, editorial issues or layout issues are identified can be fixed and revised and then uploaded. If the IPR allows. The requirement to revise and correct the issues would form part of the Evaluation recommendation.

Activity 3


Be prepared to make a specific recommendation at the end of your evaluation.

As you apply the instruments to specific resources, note all criteria and instructions which were problematic.

Activity 3


Why do we need specifications?

To determine:

• Quality and educational value

• Fit for purpose

• Access and Use

• Management, maintenance, storage

• Publication, discovery & distribution

47

Specifications Educational curriculum, subject matter, skills and

processes, age and stage of schooling, cultural inclusivity

Accessibility geospatial, physical and intellectual ability

Technical format, file size, authoring software, user plug-ins, players, software interoperability

IP & rights management

author, contributors, copyrights, access and distribution, use

Storage & distribution version control for management and maintenance, access

Publication & use file types, formats for reproduction and use.

48

Achieving quality

• No LR, TR, PDM resource is likely to encompass all quality principles in its own right as many of these are realised in the relationships between teacher, learner, resource and learning environment.

• Usually, the principles and associated criteria will consist of quantitative and qualitative elements. Thus a concept of 'reasonableness' on balance is used in the application of the criteria and applied throughout the quality assurance process.


Criteria

• Criteria are focused requirements for quality that are close to the operational level though they do not necessarily contain a specific scale.

• Testable statements formulated on the basis of accepted statements

• They can express minimum requirements (statutory conditions) and best practice (requirements that must be fulfilled)


IndicatorsIndicators are:

• Concrete and measurable

• Qualitative or quantitative

• Express central characteristics (e.g. of resource to evaluated by the criteria)

• Indicators should altogether provide a picture of whether the resource is in compliance with the criteria.


Criteria & Indicators ExamplesNon print: Criteria Indicators

Content stimulates and promotes critical thinking

4. More than 80% of the content stimulates and promotes critical thinking3. 60-80%2. 40-59%1. Less than 40%

Graphics/colors/sounds are used for the appropriate instructional reasons

4. 90-100% is appropriate3. 80-89%2. 70-79%1. Less than 70%

Content is logically developed and organized

4. 90-100% is logically developed and organized3. 80-89%2. 70-79%1. Less than 70%

53


Criteria & Indicators ExamplesNon print: Criteria Indicators

Content stimulates and promotes critical thinkingThis might occur in many ways, but the main point is that the activity requires a cognitive effort, not simply a chance selection.

Graphics/colors/sounds are used for the appropriate instructional reasons.Does the graphic style, use of color and sound match the content and the target audience?

Content is logically developed and organizedThe underlying question is: Can you understand the apparent structure of the knowledge domain and how it represented to the user? In random sequence structures, the logic is determined by the user and it is difficult to say that it is not appropriate from just a review.


Structures Inputs Processes Outcomes & Products

•School Learning Resource Committee

•LRMDS Division Quality Assurance Team

•Region LRMDS Quality Assurance Team

•Trained evaluators

•LRMDS Trainers

•

•

•

•School Report Card

•Annual Implementation Plan and School Implementation Plan

•LRMDS online catalogue

•Existing resources

•

•

•

•Application of LRMDS specifications & criteria/checklists to:

•identify learning, teaching and professional development resource needs•evaluate teacher developed materials for local use and shared distribution via LRMDS

•Analysis of identified school needs and completion of LR Plan, production of DEPD and REDP

•

•

•Evaluated & recommended resources for cataloguing

•Reproduction

•Redevelopment

•Access to quality resources via LRMDS online catalogue

•

•

•

QA Elements – A & E example


Process Core Quality objective

Identifying the responsible actors

Identifying methods or instruments that can be usedto assure quality

Measuring the success of the quality objective.

Assessment & evaluation system effectively & efficiently assesses the needs of “target” users

There are LR plans at the region, division & school levels that respond to the needs of target users

School LR Teams

Division LRMDS Team & Trainers

Division LRMDS Manager

Convergence table to compare LR needs identified with Reg/Div LR Assistance Plans

LR Assistance Plans of Reg/Div reflect consolidated needs of School LR plans

Ensuring quality- Example

LRMDS Assessment & Evaluation system


Evaluation planningWhat is the purpose of the evaluation? Need for science resources to support teaching and

learning in grade/year.

Science resources exemplifying constructivist learning principles.

What is the source of the resources?Within DepED?

External?

How are they to be reproduced & distributed?Offline?

Online?

Online and offline?


Evaluation planningAnswers to these questions should determine how

the criteria and outcomes are weighted.E.G.

A very sound resource that is highly instructivist and didactic will not fulfill the need for a resource based on constructivist principles but it can still be a good resource.

Or a good resource with typographical errors may be able to be fixed.

Or a good resource in an older version of flash may be able to be redeveloped.


Criteria and indicators

Remember to determine that the purpose of the evaluation is understood by all evaluators.

Are there options for the outcomes of the evaluation? For example issues can be fixed?


LRMDS A & E Evaluation Instruments

Intellectual Property Rights

Educational Soundness

Educational Quality Evaluation for:- Print- Charts, posters, drill & flash cards- Non-print- General reference

Technical: usability, interoperability, offline digital

Accessibility Evaluation (SPED)

Online site evaluation

Evaluation Planning: Which instruments? What criteria, if any, is mandatory?

Evaluation Planning Possible scenariosWhat is the purpose of the Evaluation? To procure science learning resources for

Grade V to be distributed to all schools in RegionTo procure professional development materials for elementary science teachers

What resources are to be evaluated? DepED – bureau produced.Foreign Assisted project developed

Commercial

What is the format of the resources PrintNon-print (digital offline or online)Online site

What funds are available for procurement? School MOOE; FAP e.g. SPHERE

Regional funds

What expertise is required to conduct the evaluation?

Science, elementary education, evaluation, technical

Application of Evaluation InstrumentsEvaluation Scenario Evaluation AreasPurpose: To procure science learning resources for Grade VSource: Foreign Assisted project developed &CommercialFormat: printFunds: SPHERE & Regional fundsEvaluator expertise: Elementary science and evaluation experience.

Intellectual Property Rights

Educational quality

LRMDS Instruments: Intellectual Property Rights checklist: Verify ownership, Copyright, Conditions of Use.

Educational Soundness General checklist. Determine the general qualities in relation to the purpose and whether to proceed with a more extensive evaluation.

Educational Quality: Rating sheet for Print resources. Determine suitability for use in schools, accuracy and currency of all content coverage, format, presentation & organization

Application of Evaluation InstrumentsEvaluation Using the instrument

criteriaIntellectual Property Rights checklist: Verify ownership, Copyright, Conditions of Use.

Finding: DepED copyright, but contains some third party material.

What should be recommended?


Finding: Identified some content inaccuracies in Criteria 1 and 5.



Finding: Passed all factors but reported errors of fact, grammatical, typographical


Application of Evaluation InstrumentsEvaluation Using the instrument criteriaIntellectual Property Rights checklist: Verify ownership, Copyright, Conditions of Use.

DepED copyright, but contains some acknowledged third party material (diagrams, pictures, excerpts of text).Recommendation: Reproduce only in existing format.


Identified some content inaccuracies in

Criteria 1 and 5.*Recommendation: Do not distribute or reproduce in current format. If the content errors are not in 3rd party material then conduct Ed Quality review, do not reproduce unless all content inaccuracies have been fixed.


Passed all factors but reported errors of fact, grammatical, typographicalRecommend: All errors are fixed, digitize and reproduce and distribute.

* If commercial resource contains errors or inaccuracies then report errors but do not proceed with further reviews.


• What is the Request Brief?

• What purpose does it serve?

• Who prepares the request brief/

• What are the major parts or components of the request brief?

Activity 4


A Request Brief is a written explanation - given to the development and production team by a requesting party or client to describe the specific service required and the general design requirements of the resource to be developed or produced.

Request Brief


The purpose of this document is to describe in detail the service required and the design requirements of a client/user for an existing resource that is to be reproduced, digitized, or redeveloped and for the design of new resources.

Request Brief


Instructions: In pairs/or triads prepare the request brief for the

materials you have evaluated. All the resources evaluated in all the streams of

evaluation must have a corresponding Request Brief. After the pairs have completed the Request Brief, they

should exchange Request Briefs with another pair. The task of the pair or triad at this stage is to REVIEW

the completed Request Brief of their peers for the following:

Prepare a Request Brief


•Completeness•Correctness •Clarity Exchange comments, feedback on each other’s Request Briefs.Make the necessary adjustments in the briefs.Compile all the accomplished instruments and request briefs

Check the Request Brief


Ask the participants to probe the following:Is the purpose for which a Request Brief is

prepared understood?What areas/parts of the Request Brief did the

participants have difficulty with?Determine whether the difficulty can be

addressed by presenting the materials for the second time

Processing


“Whatever the cost of our libraries, the price is cheap compared to that of an ignorant nation.” Walter Cronkite

1. Evaluation of Resources

Documents

Transcript of 1. Evaluation of Resources