Wasc arc conf final, iswi, april 8 2011.pptx

WASC ARC ConferenceApril 8, 2011, San Francisco, CA

Improving Student Writing and Strengthening Writing Programs – Technology and Techniques That Work

Ellen Junn, Associate Provost

Jennifer Ivie, Assistant Professor of PsychologyKim Morin, Professor of Theatre Arts

William Covino, ProvostCalifornia State University, Fresno

California State University, Fresno (est. 1911)

•23-campus CSU system

•20,932 students

•1,197 faculty

2Copyright © April 2011

Fresno State Demographics


Fresno State Demographics

§ First generation college: 68%

§ English proficiency:

63% freshmen require English remediation


Multi-pronged Approach to Teaching Writing on our Campus

¨ English Composition ¨ Upper Division Writing Exam

¡ Graduation Requirement¡ Substitution of Designated “W” courses

¨ Writing Requirement for ALL General Education Coursesú Requires “Iterative” writing assignments


The Dilemma

¨ High percentage of remedial students. ¨ Perception that surface errors distract

from content. ¨ Effective writing instruction requires

innovative pedagogies.


INSTRUCTORS KEEP ASKING…

“Where Is the Time to Respond To All of Those

Essays?”

TECHNOLOGY AS ONE SOLUTION?

•Professor Kim Morin

•“E-scholar” Program

•Upper Division online GE courses

•Faculty interest across campus

A DISCOVERY…Computer Essay Scoring Programs

Several College-level Essay Scoring and Writing programs available…

COMPUTER ESSAY SCORING PROGRAMS

• ETS® Criterion®• http://www.ets.org/criterion/higher_ed/about • IEA Intelligent Essay Assessor

http://www.knowledge-technologies.com/prodIEA.shtml • SAGrader• https://www.sagrader.com/sgm/features• Pearson MyWriting Lab• http://www.mywritinglab.com/whatis.html

ETS CRITERION® SELECTED ON A TRIAL BASIS…

First Trial (Spring 2009): 30 students – 1 essay assignment

First Pilot (Fall 2009)1 Faculty member100 students- 5 essay assignments

About ETS Criterion®• Students write and revise essays

online• Diagnostic Feedback / Holistic Score

within 20 seconds.• Topics Library provides prompts.• Faculty can create topics/prompts. • Errors are highlighted but not

corrected.


ABOUT ETS CRITERION®

• No instructor Fees• Simple Registration

• Students purchase Access codes through Bookstore

• Approximately $11.00 per student • One fee provides student use in all classes per

term/semester• Technical Support from ETS®

ABOUT ETS CRITERION®

• Bilingual feedback available • Spanish, Japanese, Simplified

Chinese, Korean, ELL

• Advanced levels available • College–1st & 2nd year, TOEFL, GRE

• No prior essays required

ETS CRITERION® Online Tour

http://www.ets.org/Media/Products/Criterion/tour2/critloader.html

Initial Presentation to Faculty

“Computer Essay Scoring has had a positive effect so far.

I spend more time assessing content, less on grammatical errors.

Students spend more time revising.However, it still misses errors and does not

grade for content.”


Early Student Comments

¡ “I like the instant feedback.” ¡ “I like that it goes into depth about why you

didn't get the max score or why you did well.” ¡ “It is easy to use - just copy and paste!”

¡ “I love the fact that I can revise my work for a

better score.” ¡ “It is very helpful and I can see my growth as a

writer.”

Copyright © April 2011 17

Facts About College Student Writing

FACT # 1: Student improvements in writing increase with more

practice ¨ specific, constructive feedback from faculty ¨ opportunity for revisions. IMPLICATION: ü Students need to write or revise more frequentlyü Examine policy requiring “iterative” writing in GE or

W courses


FACT # 2: If students do not continue to practice, their writing performance may actually deteriorate. IMPLICATION: ü Examine all writing programs across campusü Identify key courses with writing requirementü Target specific faculty teaching those courses


FACT # 3: Teaching students to write effectively can be time-consuming and labor-intensive. IMPLICATION: ü Identify effective technology-related tools ü reduce faculty workloadü provide specific, timely feedback to students.

ü Target part-time faculty who teach writing intensive courses.


FACT # 4: Writing instruction involves faculty who are NOT

trained as writing teachers.

IMPLICATION: ü Implement Criterion training for faculty.ü Offer Writing Across the Curriculum workshops.ü Provide professional development funds for all

faculty during Year 1 Pilot. 21Copyright © April 2011

FACT # 5: Assessing & documenting student learning outcomes in

writing performance are key elements to success. IMPLICATION: ü Ongoing data collected for past 3 semestersü Share results with faculty learning community to

determine Best Practices.ü Modify training based on feedback and assessment.


Next Came…

ISWI Pilot Launch

Spring 201023Copyright © April 2011

ISWI Improving Student Writing

InitiativeNovember, 2009 - Campus email announcement

calling for faculty participants § Criterion® Training:§ Faculty ISWI Coordinator§ Faculty Learning Community (FLC)

§ Writing Across the Curriculum Workshops§ Assessment of Criterion®


Campus-Wide InitiativeBy The Numbers... Year 1 (2 Semesters)

Spring 2010 / Fall 2010

• 349 Classes Involved • 173 Instructors Used Criterion®

• 134 Spring 2010• 68 Fall 2010

• 5,920 Students (Spring 2010)• 3,756 Students (Fall 2010)

Numbers may include duplicates


Campus-Wide InitiativeBy The Numbers...

Year 2 (1 Semester) Spring 2011

• 231 Classes Involved • 93 Instructors Used Criterion® • 5442 Students (Spring 2011)• 44,080 Essays Submitted (by March 15)

Numbers may include duplicates


Faculty Participation- Year 2

Out of 93 instructors opting to use Criterion, only 20 received Professional Development funds as an incentive in Year 2.


Assessments from Spring 2010..

1. Analyzed Criterion® automatically generated data for trends.

2. CLA scores for students with & without

Criterion® 3. First and last papers submitted by students

on Criterion® scored by independent faculty panel

4. Collected student and faculty surveys


Automatically-Generated Data

¡ Students who used Criterion® for revision with more than one submission, scored better than those who did not.

¡ Students with a larger number of

assignments and more submissions on Criterion® increased their holistic score on average by approximately 1 level.


Collegiate Learning Assessment (CLA)

¡ A significant difference was found between Criterion® users and non-users on the CLA performance task.

¡ No significant differences were found

between the two groups on the analytic writing task.


Faculty Scoring Panel

First and last papers submitted by students on Criterion® scored by independent faculty panel.

¨ Overall, student writing improved in

classes that used ETS Criterion®.¡ The greatest improvements were made when

instructors engaged students in substantial discussions of writing and how to use Criterion®.


Faculty Feedback – Positive Comments

When using Criterion®, most faculty felt that:1. Papers were easier to grade.2. The program improved their students’

writing skills.3. Students spent more time revising written

assignments.4. Creating assignments was easy. 32Copyright © April 2011

Faculty Feedback - Challenges

Many faculty felt that Criterion®

1. Did not reduce their workload.2. Did not do as much as they had

hoped.3. Was not worth the cost to the

students.


Student Survey Response


Student Survey Highlights

What students liked about Criterion®: 1. It was available on-line 24 hours a day. 2. It allowed them to correct grammatical

or mechanical errors before turning in a paper.

3. It gave immediate feedback.


What students found challenging:

1. Criterion® identified technical terms or

citations as errors. 2. The program identified errors but did

not correct them. 1. The program did not grade for content.


Student Survey Response

Freshmen/sophomores were significantly more likely to agree:

- Criterion has helped me improve my writing.

- Criterion should suggest less and correct more.

- I was able to apply what I learned to other writing assignments.


ESL Student Survey Response


English as a Second Language Student Responses

ESL students were more likely to respond that Criterion® helped them improve their writingúPreferred Criterion® feedback to instructor feedback.úSubmitted their documents more frequently.

- I like the kind of feedback Criterion® gives. - I spent more time improving my writing with Criterion®. - I wish I could use Criterion® for other classes.


Positive Comments From Students

¨ “Awesome program. I find it really useful.”¨ “I like that it is really convenient and I

found myself using the program for all my classes.”

¨ “I started noticing themes in my writing that could use improvement.”


Ongoing Research

¨ Studies with Control Groups¨ Continue Collecting Data¨ Determine Best Practices / Uses


Conclusion

So Far, Evidence suggests- ¨ Criterion® helps improve student writing

when combined with effective instructor practice.

¨ Criterion® appears useful for editing grammar and mechanics.

¨ Students who use Criterion® are more engaged with the writing process.


Next Steps

¨ Analyze and refine assessment data.¨ Provide online video tutorials.¨ Expand Professional Development &

training.¨ Establish ISWI committee

ú analyze writing instruction across campus.¨ Recognize and thank participating faculty.


What Have We Learned? ¨ Students generally found Criterion® to be easy

to use and cost effective.¨ ESL students responded more positively to

Criterion® feedback.¨ Faculty found Criterion® helpful when used to

complement instruction.¨ ISWI shows the high degree of faculty interest

in improving student writing across campus.


ISWI Broadly Endorsed & Supported By:¨ Writing Competency Subcommittee (subcommittee of Senate’s GE Committee)¨ Professional Development Subcommittee (subcommittee of Senate’s Personnel

Committee)¨ Chair of the GE Committee¨ Office of Undergraduate Studies¨ Division of Graduate Studies¨ Institutional Research & Assessment Planning Director leading faculty ISWI Assessment

Team¨ Provost’s Office as implemented by the Associate Provost through the Center for the

Scholarly Advancement of Learning and Teaching (CSALT) & Technology Innovations for Learning and Teaching (TILT)


Thank You!

¨ William Covino, Provost¨ [email protected]¨ 559-278-2636¨ ¨ Ellen Junn, Associate Provost¨ Interim Director, Center for the

Scholarly Advancement of Learning and Teaching (CSALT)

¨ Interim Senior Academic Technology Officer (SATO) and Technology Innovations for Learning and Teaching (TILT)

¨ [email protected]¨ 559-278-2636

¨ Kim Morin, Professor ¨ Department of Theatre Arts¨ Artistic Director, Theatre for

Young Audiences; English/Drama Credential Advisor

¨ ISWI Faculty Coordinator¨ [email protected] ¨ 559-278-4342¨ ¨ Jennifer Ivie, Assistant Professor ¨ Department of Psychology¨ Interim Director, Center for

the Scholarly Advancement of Learning and Teaching (CSALT)

¨ [email protected] ¨ 559-278-2842


Contact Information- ETS®

ETS® Criterion®¨ http://www.ets.org/criterion

¨ Susan L. Yetman¨ Criterion Account Manager¨ Higher Education, ETS ¨ Rosedale Road MS 51-L¨ Princeton, NJ 08541¨ Direct line: 609.683.2675¨ Toll free: 866.717.1915¨ Fax: 609.683.204

0¨ Email: [email protected]


Arthur RuzzanoWestern Regional ManagerHigher Education Assessment SolutionsPhone: 310.944.4034Fax: 609.683.2040Email: [email protected] Bill WynneProduct Manager - ETS Proficiency ProfileETS Programs and Services DivisionTelephone (609) 683-2006E-mail [email protected]

Wasc arc conf final, iswi, april 8 2011.pptx

Education

Transcript of Wasc arc conf final, iswi, april 8 2011.pptx