Development of processes and criteria for CEAB graduate attribute assessment Lynann Clapham...
-
Upload
nancy-griffin -
Category
Documents
-
view
216 -
download
3
Transcript of Development of processes and criteria for CEAB graduate attribute assessment Lynann Clapham...
Development of processes and criteria for CEAB graduate attribute assessment
Lynann ClaphamAssociate Dean (Academic)
Brian FrankDirector (Program Development)
Faculty of Applied ScienceQueen's University
FACULTY OF APPLIED SCIENCE
2
Overview
• Outcome requirements at CEAB and general higher education in Canada
• Nationwide engineering program responses• Outcomes measurement• Queen's University progress
FACULTY OF APPLIED SCIENCE
3
CEAB accreditation changes
Engineering programs are being required to assess outcomes and produce a process for improvement, not just inputs
Section “3.1: Graduate attributes” (outcomes)
FACULTY OF APPLIED SCIENCE
4
Section 3.1: Graduate Attributes
Two part requirement:
“The institution must (1) demonstrate that the graduates of a program possess the
attributes under the following headings. The attributes will be interpreted in the context of candidates at the time of graduation...
(2) There must be processes in place that demonstrate that program outcomes are being assessed in the context of these attributes, and that the results are applied to the further development of the program.”
FACULTY OF APPLIED SCIENCE
5
Graduate attribute measurement
This requires:– Programs gather information that is measurable
and meaningful for them to use, not just to show CEAB accreditation teams
– Programs have a system to gather and analyze data over multiple years
– Programs gather enough information to improve the program, which may involve gathering information from students in first year, middle years, and graduating year of the program, though only graduating information is required
FACULTY OF APPLIED SCIENCE
6
Why is CEAB requiring outcomesassessment?
1. Most major industrialized countries use them to demonstrate their students' capabilities, both in engineering and in other programs.
2. There was a danger that our students would not have their educational credentials recognized outside Canada if we cannot demonstrate outcomes.
Washington Accord: substantial equivalency among Australia, Canada, Hong Kong, Republic of Ireland, New Zealand, South Africa, United Kingdom, and United States, Japan, Singapore,Korea, and Chinese Taipei
Bologna Declaration: compatibility among higher education programs across Europe
FACULTY OF APPLIED SCIENCE
7
Other forces: National standards
Provincial Ministers of Education across Canada released Ministerial Statement on Quality Assurance of Degree Education in Canada, which recommended outcome measures in for higher education
Ontario adopted the recommendations and released them as Undergraduate Degree Learning Expectations for all undergraduate programs in the province.
Similar expectations are likely to follow in other provinces.
FACULTY OF APPLIED SCIENCE
8
Ontario requirements
Undergraduate Degree Level Expectations (UDLEs) based on national recommendations
1. Depth and Breadth of Knowledge 2. Knowledge of Methodologies 3. Application of Knowledge 4. Communication Skills 5. Awareness of Limits of Knowledge 6. Autonomy and Professional Capacity
Detailed expectations shown in document referenced in paper. They align with CEAB graduate attributes very well.
FACULTY OF APPLIED SCIENCE
9
Why are graduate attributes (outcomes) being adopted?
It provides information about our students' knowledge, skills, and attitudes (outcomes), in addition to syllabi, instructor education, library resources, etc. (inputs).
It identifies gaps between our perceptions of what we teach and what students can do.
The information can be used to continually improve our programs.
FACULTY OF APPLIED SCIENCE
10
Curriculum design
Currently most of us have designed curriculum around subject content:
ThermodynamicsFluid mechanicsElectric circuitsEconomicsSolid mechanics
This will require we consider what we want students to do with the content:
E.g. Students can:...select appropriate mathematical
principles and tools to analyze and solve engineering problems ... (Knowledge)
...evaluate and use information based on its authority, objectivity, and currency. (Lifelong learning)
FACULTY OF APPLIED SCIENCE
11
What we have been given:
12 broad areas (graduate attributes), and a requirement to create a process for program improvement. Programs will need to decide:
Graduate attributes
How to measure?
In which courses?
What are specific criteria? (measurable)
Process for programimprovement?
FACULTY OF APPLIED SCIENCE
12
Generally accepted process to do this
Identify stakeholders (advisory boards, faculty, students, staff, alumni)
Identify the program's major objectives (which must include CEAB graduate attributes!)
Create specific measurable assessment criteria for objectives
– e.g. “Students describe limitations of theory in predicting behaviour of processes and systems”
Identify assessment measures appropriate for criteria– e.g. report, oral presentation, simulation,
standardized test, lab performance, etc. Curriculum process flow by Peter Wolf, U Guelph
FACULTY OF APPLIED SCIENCE
13
E.g. ABET's assessment flow chart
FACULTY OF APPLIED SCIENCE
14
Creating assessment criteria
• What are a program's specific and measurable criteria?• Can involve cognitive (knowing, analyzing, creating),
beliefs, skills • Assessment criteria describes
1. level of expectation (“describes”, “compares”, “applies”, “creates”, etc.)
2. Content areae.g.
The student identifies constraints on a design problem including applicable standards, economic, environmental, culture, and societal considerations, health and safety risks
FACULTY OF APPLIED SCIENCE
15
Example: Draft assessment criteria at Queen's University
FACULTY OF APPLIED SCIENCE
16
Curriculum mapping: Link criteria toappropriate courses:
E.g. For a capstone course, a program might decide that the assessment criteria used are:
D1. identify constraints including applicable standards, economic, environmental, culture, and societal consideration, health and safety risks (design)
D2. use creativity to establish innovative candidate solutions. (design)L1. evaluate information from a variety of sources for objectivity,
authority, currency (lifelong learning)L2. describe opportunities for professional development in the field
(lifelong learning)...
These would be assessed using whatever is deemed appropriate: reports, presentations, portfolios, surveys, etc.
FACULTY OF APPLIED SCIENCE
17
Graduate attributes similar to USA, UK, Australia, Korea, Peru,...
CEAB ABET UK-SPEC Eng Australia
Knowledge Knowledge Knowledge Knowledge
Problem Analysis Solve problems Eng. analysis Problem solving
Investigation Experiments
Design Design Design Design
Engineering tools Techniques, tools Lab skills Techniques, tools
Teamwork Teamwork Teamwork
Communication Communication Communication
Professionalism Contemporary issues Eng practice Professionalism
Impact Impact Social context Responsibilities
Ethics & Equity Ethical responsibility Ethical conduct Ethics
Econ. & Proj. Manage *(part of design) *(part of design)
Lifelong learning Lifelong learning Lifelong learning
FACULTY OF APPLIED SCIENCE
18
Since expectations are similar, we can usematerial from other countries
General engineering outcomes– Learning Outcomes for UK-SPEC (UK)– Engineers Australia Competency Standards
(Australia)– EC2000 Outcome Attributes for ABET (USA)– ABET criteria 2009-2010– OCAV Undergraduate Degree-level expectations
(Ontario)– CDIO Syllabus
Available training: ABET's week long Institute for Development of Excellence in Assessment Leadership (IDEAL), run 2x/year
FACULTY OF APPLIED SCIENCE
19
National response of deans
Similarity among Canadian programs (more so than among US institutions)
but
no large-scale funding to support development of assessment procedures (like NSF funding in US for EC2000)
So... how can we work together?
FACULTY OF APPLIED SCIENCE
20
NCDEAS Education Committee
• National Council of Deans of Engineering and Applied Science (NCDEAS) Education Committee developing three “exemplars” to be shared among other schools as a starting point
– Queen's U (coordinating institution and clearinghouse)
– U Calgary– U Sherbrooke
FACULTY OF APPLIED SCIENCE
21
Queen's Development plan
1. Measurable faculty wide assessment criteria, e.g.: Identify constraints including applicable standards, economic,
environmental, culture, and societal consideration, health and safety risks (Design)
2. Leveled criteria (recommended expectations in first, middle, and final year to track progression and recommend curriculum targets)
3. Assessment measures for criteria, e.g. design reports, observations, exams, surveys, oral presentations,
standardized instruments, etc.
4. Recommendations for software to gather and manage data5. Individual program customizations, map to courses
FACULTY OF APPLIED SCIENCE
22
Progress
• January: B. Frank attended ABET's Institute for Development of Excellence in Assessment Leadership (IDEAL)
• February: began gathering functional requirements for software to support process
• March: formed 7 working groups to consider “what do we expect our students to be able to do when they graduate?” in the 12 areas required by CEAB
• April: worked with assessment specialist on framework for assessment criteria
• June: rolling out software, developing documentation, transitioning courses
• July: Rough draft of assessment criteria assembled
FACULTY OF APPLIED SCIENCE
23
General comments
Not all courses need to assess graduate attributesCreate plan to assess some graduate attributes in each
academic term (not necessarily all attributes every year). E.g. assess design in year 1, 3, 6 of accrediation cycle
Select how many students are being sampled Triangulate: multiple sources of assessment for each
assessment criteriaUse a mix of direct measures (by directly observing student
skill and indirect measures (e.g. Student self-reports, alumni opions, exit surveys, etc.)
FACULTY OF APPLIED SCIENCE
24
Queen's Timeline
Report
Fall '11Fall '09
Pilot faculty wide outcomes in some courses
Pilot software
Evaluate
Share
Spring '10
Department outcomes
Map to curriculum
Create process for program improvement
Fall '10
Pilot program outcomes
Gather data
Evaluate
Create faculty wide outcomes
Evaluate, select software
Spring '09
FACULTY OF APPLIED SCIENCE
25
Material to be shared
• Queen's U:– Leveled assessment criteria,description of
sources, with links to UDLEs– Assessment measures– Example course mapping– Documentation for open-source software to
support gathering and reporting– Collection of references to support development
(competencies from other accreditation bodies, curriculum development resources, etc.)
• Resources from U Calgary and U Sherbrooke
FACULTY OF APPLIED SCIENCE
26
Software support
Lesson from U.S.: don't need to pour $millions into assessment, gathering far too much data.
Over the summer we assessed learning management systems that will be able to store a variety of types of assessment measures and materials to simplify reporting and analyzing.
Best option for us was Moodle (open source, customizable, extendible, low initial cost), but similar functionality is available in Blackboard, DesireToLearn
FACULTY OF APPLIED SCIENCE
27
Moodle support for graduate attribute assessment
FACULTY OF APPLIED SCIENCE
28
Material being collected:Assessment wiki
Wiki link here.
FACULTY OF APPLIED SCIENCE
29
Input/questions/comments
Lynann [email protected]
Brian [email protected]