An Italian Proposal to Deal with Quality in E-learning Alfredo Squarzoni Università di Genova

Rome Rome May 31st, 20 May 31st, 200606

University towards E-learning: a FocUniversity towards E-learning: a Focus on Finland, France and Italyus on Finland, France and Italy

11

CONFERENCE CONFERENCE

University towards E-learning: a University towards E-learning: a Focus on Finland, France and ItalyFocus on Finland, France and Italy

RomeRomeMay 31st, 2006May 31st, 2006

An Italian Proposal to Deal with An Italian Proposal to Deal with Quality in E-learningQuality in E-learning

AlfredoAlfredo Squarzoni SquarzoniUniversità di GenovaUniversità di Genova

CRUI Steering Commitee on EvaluationCRUI Steering Commitee on Evaluation



22

PresentationPresentation

Introduction on CRUI activity in the field of Introduction on CRUI activity in the field of programme assessment and certificationprogramme assessment and certification

CRUI Model for quality assessment of e-CRUI Model for quality assessment of e-learning programmeslearning programmes

CRUI Requirements for certificationCRUI Requirements for certification

CRUI Procedure of assessment and certificationCRUI Procedure of assessment and certification



33

Introduction onIntroduction on

CRUI Activity CRUI Activity

in the Field of Programme in the Field of Programme

Assessment and CertificationAssessment and Certification



44

Assessment

Procedure finalised to first of all Procedure finalised to first of all promote the quality improvement of a promote the quality improvement of a certain product, process or servicecertain product, process or service



55

Programme QualityProgramme Quality

Level of achievement of objectives set Level of achievement of objectives set out consistent with the needs of all out consistent with the needs of all interested parties in the programmeinterested parties in the programme



66

CAMPUS-CRUI ProjectCAMPUS-CRUI Project

- between 1995 and 2001between 1995 and 2001

- 90 Diploma programmes involved, 90 Diploma programmes involved, mainly in engineeringmainly in engineering

Consip



77

CAMPUS One ProjectCAMPUS One Project

- academic years 2001/02 to 2003/04academic years 2001/02 to 2003/04

- over 500 programmesover 500 programmes involved involved of all of all cultural areascultural areas

Consip



88

CertificationCertification

Statement by an independent third Statement by an independent third party who declares, with reasonable party who declares, with reasonable reliability, that a certain product, reliability, that a certain product, process or service is in accordance process or service is in accordance with a specific rule or a specific modelwith a specific rule or a specific model



99

AccreditationAccreditation

Statement by a recognised institution Statement by a recognised institution who declares, formally and publicly, who declares, formally and publicly, the capability of an organism to carry the capability of an organism to carry out specific functions or, in other out specific functions or, in other words, to satisfy pre-defined standardswords, to satisfy pre-defined standards



1010

CRUI “CRUI “Certification of Certification of QualityQuality““

Not only “conformity” to the CRUI Model, but also: Not only “conformity” to the CRUI Model, but also: • educational and learning objectives of value, educational and learning objectives of value, • an educational process consistent with learning an educational process consistent with learning

objectives,objectives,• resource availability adequate to the achievement of resource availability adequate to the achievement of

such objectives, such objectives, • results consistent with educational and learning results consistent with educational and learning

objectives,objectives,• a management system capable of promoting a management system capable of promoting

continual improvement of the programmecontinual improvement of the programme



1111

CRUI Model CRUI Model for quality assessment for quality assessment

of e-learning programmesof e-learning programmes



1212

CRUI ModelCRUI Model (1)(1)

It is the rIt is the result of the experiences of esult of the experiences of assessment made in our country so far assessment made in our country so far and of the continual attention to the and of the continual attention to the assessment models of the other assessment models of the other European countriesEuropean countries



1313

CRUI ModelCRUI Model (2)(2)

The Model has been defined for each The Model has been defined for each of the following types of programmes:of the following types of programmes:

• “ “corsi di laureacorsi di laurea” (bachelor level) and” (bachelor level) and

“ “corsi di laurea magistralecorsi di laurea magistrale” (master ” (master level);level);

• postgraduate programmes;postgraduate programmes;

delivered in presence, delivered in presence, on-lineon-line or in or in blended formblended form



1414

Aim of the CRUI Aim of the CRUI assessment assessment of programmeof programme quality quality

To promote the continual improvement of programme quality by pointing out strong and weak points



1515

Because Because Quality Quality regardsregards the “the “productproduct”” the “the “systemsystem””

the CRUI Model for the assessment of the CRUI Model for the assessment of programmes sets out to assess both the programmes sets out to assess both the

educational service educational service (product (product assessment) assessment)

and the and the management systemmanagement system thereof (system thereof (system

assessment)assessment)



1616

With reference to the With reference to the

educational serviceeducational service, ,

assessment is concerned with the assessment is concerned with the degree at which all the students are degree at which all the students are

provided with adequate opportunities provided with adequate opportunities to achieve to achieve

a level of learning consistent a level of learning consistent

with the objectives set in the time setwith the objectives set in the time set



1717

With reference to the With reference to the

management systemmanagement system, ,

assessment is concerned with the assessment is concerned with the degree at which the system permits degree at which the system permits

the constant achievement of the the constant achievement of the objectives set objectives set

in the time set, i.e. the degree in the time set, i.e. the degree

at which the system makes at which the system makes

systematic the product qualitysystematic the product quality



1818

Fundamental Objectives Fundamental Objectives of the assessmentof the assessment (1)(1)

The overall assessment of the programme The overall assessment of the programme

quality has to check whether:quality has to check whether:

• programme educational objectivesprogramme educational objectives are are consistent with the mission of the consistent with the mission of the university and the needs of all interested university and the needs of all interested parties (such as students, teachers, parties (such as students, teachers, businesses and so on) and businesses and so on) and programme programme learning outcomeslearning outcomes are consistent with the are consistent with the programme educational objectives;programme educational objectives;



1919


• academic and support staff, facilitiesacademic and support staff, facilities, financial , financial resources and cooperation agreements with resources and cooperation agreements with businesses, research institutions and other businesses, research institutions and other Higher Education Institutions are adequate to Higher Education Institutions are adequate to accomplish programme outcomes;accomplish programme outcomes;

• curriculumcurriculum and related processes ensure and related processes ensure achievement of programme outcomes;achievement of programme outcomes;



2020


• appropriate forms of assessmentappropriate forms of assessment attest attest the achievement of programme the achievement of programme outcomes;outcomes;

• the the management systemmanagement system is able to assure is able to assure the systematic achievement of the systematic achievement of programme outcomes and the continual programme outcomes and the continual improvement of the programme.improvement of the programme.

Dimensions of AssessmentDimensions of Assessment

Consequently, the Model identifies fiveConsequently, the Model identifies five

Dimensions of assessmentDimensions of assessment::• Management System and Management System and Organisational Structure;Organisational Structure;• Needs and Objectives;Needs and Objectives;• Resources;Resources;• Educational Process;Educational Process;• Results, Analysis and Results, Analysis and ImprovementImprovement..



2222

Dimension(5)

Structure of the CRUI ModelStructure of the CRUI Model

QUESTIONS

QUESTIONS

QUESTIONS

ELEMENTS

(18)

FACTORS

FACTORS

FACTORS

QUESTIONS

FACTORS



2323

Dimension 2: Dimension 2: Needs and Needs and ObjectivesObjectives

Element 1: Element 1: Needs of Interested Needs of Interested PartiesParties QuestionsQuestions FactorsFactors

Have the needs of Have the needs of the interested the interested parties been parties been identified?identified?

Interested parties identified Interested parties identified

How contacts with the How contacts with the interested interested

parties are maintainedparties are maintained

IdentifiedIdentified needs of the needs of the interestedinterested

partiesparties



2424

Needs and Objectives

Satisfaction

STUDENTGRADUATE

Management System and

Organisational Structure

Resources

Educational Process

Results,Analysis

andImprovement

Sta

keh

old

ers

Needs

Sta

keh

old

er

s



2525

Process ApproachProcess Approach

CRUI Model adopts the

process approach

promoted by

ISO 9001: 2000 NormManagement Systems for Quality. Requirements



2626

PDCA MethodPDCA Method

To manage a specific process, the CRUI Model To manage a specific process, the CRUI Model suggests the suggests the PDCAPDCA method method : :• PlanPlan: to establish the objectives and the : to establish the objectives and the

processes necessary to obtain results in processes necessary to obtain results in accord with the objectives;accord with the objectives;

• DoDo: to implement the processes;: to implement the processes;• CheckCheck: to monitor and measure the : to monitor and measure the

processes and the results in comparison with processes and the results in comparison with the objectives;the objectives;

• ActAct: to adopt actions for the continual : to adopt actions for the continual improvement of the process performance. improvement of the process performance.



2727

Dimension “Dimension “Management Management System and Organisational System and Organisational

Structure”Structure”

ElementsElements

• Management SystemManagement System

• Organisational Structure Organisational Structure

• Re-examinationRe-examination



2828

Dimension “Dimension “Needs and Needs and Objectives”Objectives”

ElementsElements

• Needs of Interested PartiesNeeds of Interested Parties

• Educational Objectives Educational Objectives

• Programme OutcomesProgramme Outcomes

• PoliciesPolicies



2929

Dimension “Dimension “Resources”Resources”

ElementsElements

• Academic and Support StaffAcademic and Support Staff

• FacilitiesFacilities

• Financial ResourcesFinancial Resources

• External RelationshipsExternal Relationships



3030

““Objects” to be assessed Objects” to be assessed specific for on-line programmesspecific for on-line programmes

• Teaching StaffTeaching Staff

- Adequacy of teaching staff to the needs of on-Adequacy of teaching staff to the needs of on-line teachingline teaching

- Ability to promote on-line learning and Ability to promote on-line learning and communication with and among studentscommunication with and among students

- Presence of educational initiatives to favour Presence of educational initiatives to favour development of on-line teaching and learning development of on-line teaching and learning competenciescompetencies



3131

• Teaching Support Staff and Technical Teaching Support Staff and Technical StaffStaff

- Availabily of “Availabily of “content tutorscontent tutors”, i.e. personnel ”, i.e. personnel

dedicated to on-line didactics,dedicated to on-line didactics, expert in programme expert in programme content and in the management of on-line content and in the management of on-line communication and of virtual groups, to follow communication and of virtual groups, to follow students in their study activity and able to promote students in their study activity and able to promote learninglearning

- Availability of “Availability of “instructional designersinstructional designers”, i.e. ”, i.e. designers for the e-learning environment, able to designers for the e-learning environment, able to co-operate with and guide teachers in the definition co-operate with and guide teachers in the definition of educational patterns to be delivered on-lineof educational patterns to be delivered on-line



3232

-- Availability of “Availability of “web editorsweb editors”, i.e. experts in the ”, i.e. experts in the production of didactic material for the digital production of didactic material for the digital environment;environment;

- Availabilty of “Availabilty of “web managersweb managers”, i.e. technicians ”, i.e. technicians able to assure an adequate technical assistance able to assure an adequate technical assistance to teachers and studentsto teachers and students

- Availability of personnel for the “Availability of personnel for the “technical helptechnical help”, ”, i.e. technicians able to garantee the operation i.e. technicians able to garantee the operation of the technological platformof the technological platform

- Presence of Presence of educational initiativeseducational initiatives to favour to favour development of skills and competencies of development of skills and competencies of teaching support staff and of technical staffteaching support staff and of technical staff



3333

• FacilitiesFacilities

- Availability of a technological platform easily Availability of a technological platform easily accessible, easy to use, reliable, flexible and so on, accessible, easy to use, reliable, flexible and so on, integrated in the technological infrastructures of the integrated in the technological infrastructures of the AtheneumAtheneum

- Availability of virtual rooms and virtual Availability of virtual rooms and virtual technological resources (laboratories, libraries and technological resources (laboratories, libraries and so on) adequate to the development of teachers’ so on) adequate to the development of teachers’ teaching activity and students’ learning activity teaching activity and students’ learning activity

- Capacity of the virtual facilities to adapt themself, in Capacity of the virtual facilities to adapt themself, in quality and quantity, to the number of users and to quality and quantity, to the number of users and to the different activities developed by the programmethe different activities developed by the programme



3434

Dimension “Dimension “Educational Educational Process”Process”

ElementsElements

• PlanningPlanning

• Student AccessStudent Access

• DeliveryDelivery

• Support ServicesSupport Services



3535

Factors of particular importance Factors of particular importance for on-line programmesfor on-line programmes

- Didactic coordination among teachers and Didactic coordination among teachers and teachers and teaching support staffteachers and teaching support staff

- Adequacy of the delivery planning to favour Adequacy of the delivery planning to favour students’ learningstudents’ learning

- Adequacy of the methodologies of delivering Adequacy of the methodologies of delivering and assessment of students’ learningand assessment of students’ learning

- Presence of adequate orienteering and tutoring Presence of adequate orienteering and tutoring



3636

Dimension Dimension ““Results, Analysis and Results, Analysis and

Improvement”Improvement”ElementsElements

• ResultsResults

• AnalysisAnalysis

• ImprovementImprovement



3737

Factors of particular importance Factors of particular importance for on-line programmesfor on-line programmes

- Presence of adequate criteria for the assessment Presence of adequate criteria for the assessment of the programme results (which must take in of the programme results (which must take in proper account the tipology of students proper account the tipology of students enrolled)enrolled)



3838

RequirementsRequirements

forfor

certificationcertification



3939

CRUI certificationCRUI certification has the main aim to has the main aim to attest, formally and publicly, the adoption attest, formally and publicly, the adoption by a programme of the CRUI Model as by a programme of the CRUI Model as model for the management of the model for the management of the programme itself and that the programme itself and that the programme:programme:• has educational and learning objectives of has educational and learning objectives of

value, i.e. coherent among them, with value, i.e. coherent among them, with possible requirements and with the needs possible requirements and with the needs of the interested parties;of the interested parties;

• has at disposal resources adequate to the has at disposal resources adequate to the achievement of the educational and achievement of the educational and learning objectives;learning objectives;



4040

• has an educational process and support has an educational process and support services which effectively allow the services which effectively allow the achievement of the educational and achievement of the educational and learning objectives;learning objectives;

• its results attest the achievement of the its results attest the achievement of the educational and learning objectives set; educational and learning objectives set;

• has a management system and an has a management system and an organisational structure which favourorganisational structure which favour the the achievement of the programme objectives achievement of the programme objectives and promote the continual improvement of and promote the continual improvement of the effectiveness of the processes for the the effectiveness of the processes for the management of the programme itself.management of the programme itself.



4141

Dimension Management System and Dimension Management System and Organisational StructureOrganisational Structure

Engagement in favour of a management for Engagement in favour of a management for quality of the programme quality of the programme

Identification of the processes for the programme Identification of the processes for the programme management and of their interactions management and of their interactions

Management of programme documentationManagement of programme documentationManagement of communication with the interested Management of communication with the interested

partiespartiesIdentification of responsibilities for the Identification of responsibilities for the

management of the programmemanagement of the programmeRe-examination of the management system and of Re-examination of the management system and of

the organisational structurethe organisational structure



4242

Dimension “Needs and Objectives” Dimension “Needs and Objectives”

Identification of the needs of interested partiesIdentification of the needs of interested parties

Coherence of the programme educational Coherence of the programme educational objectives with the needs of interested partiesobjectives with the needs of interested parties

Coherence of the programme learning outcomes Coherence of the programme learning outcomes with the programme educational objectiveswith the programme educational objectives

Policies for studentsPolicies for students



4343

Dimension “Resources” Dimension “Resources”

Adequacy of teaching staff Adequacy of teaching staff

Adequacy of teaching support staff and technical Adequacy of teaching support staff and technical and administrative staffand administrative staff

Adequacy of facilitiesAdequacy of facilities

Adequacy of relationships with outside Adequacy of relationships with outside organisations organisations (only for programme with vocational objectives)(only for programme with vocational objectives)

Adequacy of partnerships with Athenaeums of Adequacy of partnerships with Athenaeums of other countries other countries (only for programmes with specific (only for programmes with specific internationalisation policies)internationalisation policies)



4444

Dimension “Educational Process” Dimension “Educational Process”

Coherence of curriculum with programme learning Coherence of curriculum with programme learning outcomesoutcomes

Adequacy of delivery planningAdequacy of delivery planning

Adequacy of prerequisites required for access to Adequacy of prerequisites required for access to the programmethe programme

Adequacy of criteria used to manage students’ Adequacy of criteria used to manage students’ careers careers

Control of the delivery process Control of the delivery process

Organisation and management of support services Organisation and management of support services



4545

Dimension “Results, Analysis and Dimension “Results, Analysis and Improvement”Improvement”

Results relative to entrance studentsResults relative to entrance students

Results of the educational process Results of the educational process

Results relative to insertion into the job market of Results relative to insertion into the job market of those students who have obtained an academic those students who have obtained an academic qualification qualification (only for programmes with vocational objectives)(only for programmes with vocational objectives)

Analysis of programme resultsAnalysis of programme results

Continual improvement of the effectiveness of the Continual improvement of the effectiveness of the processes for the programme management processes for the programme management



4646

CRUI Procedure CRUI Procedure

of of

Assessment and CertificationAssessment and Certification



4747

Assessment MethodologyAssessment Methodology

Self-AssessmentSelf-Assessment

followed byfollowed by

External AssessmentExternal Assessment



4848

The procedure of assessment and certification is The procedure of assessment and certification is articulated in the following main phases:articulated in the following main phases:• self-assessment, which involves the self-assessment, which involves the

compilation of a self-assessment report by compilation of a self-assessment report by the programme;the programme;

• external assessment, run by an assessment external assessment, run by an assessment team composed of two people at least, one team composed of two people at least, one academic at least and one representative academic at least and one representative from non-academic world at least, which from non-academic world at least, which winds up with the compilation of an winds up with the compilation of an assessment and certification report;assessment and certification report;

• decision of certification, proposed by a decision of certification, proposed by a Steering Committee of experts and assumed Steering Committee of experts and assumed by a joint Board of people from academic by a joint Board of people from academic world and people from non academic world. world and people from non academic world.



4949

To record judgement of the To record judgement of the achievement of achievement of

individual requirements, a scale with the individual requirements, a scale with the

following four grades is used:following four grades is used:

• 4, in case of requirement fully fulfilled;4, in case of requirement fully fulfilled;

• 3, in case of requirement fulfilled 3, in case of requirement fulfilled “more yes than no”;“more yes than no”;

• 2, in case of requirement fulfillled 2, in case of requirement fulfillled “more no than yes”;“more no than yes”;

• 1, in case of requirement not fulfilled.1, in case of requirement not fulfilled.



5050

Certification of conformity to the CRUI Model is Certification of conformity to the CRUI Model is awarded only if all the requirements are fully or awarded only if all the requirements are fully or “more yes than no” fulfilled.“more yes than no” fulfilled.

Where requirements are not fully met, Where requirements are not fully met, certification is subordinated to the engagement certification is subordinated to the engagement on the part of the programme to satisfy all the on the part of the programme to satisfy all the remarks evidenced in the certification report by remarks evidenced in the certification report by the assessment team, with the indication of the the assessment team, with the indication of the ways and of the time (no longer than 24 ways and of the time (no longer than 24 months) taken for their solution.months) taken for their solution.

The certification of conformity is valid for six The certification of conformity is valid for six years. years. After three years the certified After three years the certified programmes are subjected to a surveillance programmes are subjected to a surveillance visit. visit.



5151

The CRUI System for assessment and The CRUI System for assessment and certification of university programmes: a certification of university programmes: a

European systemEuropean system

The CRUI System for assessment and certification of programme quality has been compared with the requirements of the document Standards and Guidelines for Quality Assurance in the European Higher Education Area, proposed by ENQA in February 2005 and adopted by the Conference of European Ministers of Higher Education in the Bergen meeting (May 2005).

The comparison has shown that the CRUI System The comparison has shown that the CRUI System fulfils all the requirements of the ENQA document.fulfils all the requirements of the ENQA document.

It is therefore possible to state that the CRUI System can be considered a “European systemEuropean system” for the assessment and certification of university programmes.

An Italian Proposal to Deal with Quality in E-learning Alfredo Squarzoni Università di Genova

Documents

Transcript of An Italian Proposal to Deal with Quality in E-learning Alfredo Squarzoni Università di Genova