What Counts as Scholarly Impact? Perspectives from the Academy, Practice, and Policy

“For many years, I secretly worked on my research. I say “secretly” because, once upon a time, researchers simply published their research in professional journals—and there it stayed.” Carol Dweck, Education Week, September 23, 2015, p. 20.

• 3 sets of scholars present for 10 minutes on an issue surrounding scholarly impact

• 8 scholars present for 4 minutes each on scholarly impact from their professional perspective

• Interaction among participants• Dialogue

CAITLIN MCMUNN DOOLEY—PROFESSORS, WE NEED YOU (PART 2)

ELIZABETH BAKER & CAROL GILLES–EXAMINING THE IMPACT OF CITATION INDEXES ON SCHOLARLY DEVELOPMENT

The Misuse of a Metric?

Dr. Carol GillesUniversity of Missouri

December 4, 2015

Impact factors in other areas?

A Cancer Journal for Clinicians

115 (One year)119 (5 years)

Cancer Cell23 (one year)27 (5 years)

Trends in Plant Science12.9 (one year)14. (5 years)

In Literacy

Highest ‘impact’ journal in Literacy – Reading Research Quarterly -- 2.8 (2014)

3.4 (5 years)Most literacy journals are below 1

My favorite journals for research dissemination to teachers were not even listed.

Some Issues to think about

Who is included in Reuters? How do they get there? Is there a cost? Is citation rate the same thing as quality??What happens:• if a discipline is small, • articles are published in languages other

than English or • journals come out infrequently?

Uses/misuses of the Metric

The metric only includes articles So books, book chapters and digital

products are not included.Metric is based on citation by other researchers

So those products that are designed for dissemination to classroom teachers are not included.Hard sciences are advantaged

Humanities are excluded (foreign languages, art, music, etc.)

Our questions

What might explain the differences in the values within disciplines?

What might explain the differences in the values among disciplines?

Findings to Date (in progress)

We accessed the top 30 “impact score” journals in these areas:AREA IMPACT SCOREOncology 115Plant Science 23Psychology 21.81Economics 6.65

Math 7.34Sociology 4.39Educational Psychology and Education 3.89Communications 3.16History 2.61

Hypotheses

The number of “review of the literature articles” and probably meta-studies affect the impact factor

Exp. Annual Review of Psychology – 21 Psychological Bulletin –14

Annual Review of Plant Biology 23.3New Phytologist 7.6( top 5 journals are all reviews)

Our Hypotheses

In some disciplines the length of time an article is cited is shorter than others – Citing half life –median age of articles cited by the

journal itself – Cited half life -- median age of articles cited in other

journals (Oncology – 3.5 years; RRQ - <10 yrs.)** Imp because Impact Factor only includes citing from previous year.**Influence of articles more than one year old is not measured. ***The shorter the half-life, the higher the impact factor

Our Hypotheses

• Number of authors who contribute to one article– In plant science and cancer research, multiple

authorship is common – Up to 200

• It may be due to the journal being “open access” – Science journals are often open access– JAAL

ELFRIEDA H. HIEBERT--EXPANDING METRICS OF SCHOLARLY IMPACT FOR TEACHER EDUCATORS: ADDRESSING DIGITAL PRESENCE

Expanding Metrics of Scholarly Impact for Teacher Educators: Addressing Digital Presence

Elfrieda H. HiebertTextProject & University of California,

Santa Cruz

Rick Hess’s Edu-Scholar Public Influence Scoring Rubric

  Google Scholar

(50)

Book Points(25)

Ama-zon

Rank-ing(20)

Ed. Press(30)

Web Men-tions(30)

News-paper

Mentions(30)

Congr. Record

(5)

Klout (10)

Total(200)

Top 5% 46.1 19.9 14.5 10.0 20.6 13 0 4.8 130.1

75 38.7 6.2 2.8 3.0 6.9 7.4 .5 .2 65.7

50 24.3 5.9 3.5 .9 5.3 4.7 0 1.2 45.8

25 18.5 3.3 2.6 1.3 4.1 2.9 0 .6 33.3

Bottom 5%

11 .61 0 .68 2.6 1 0 .9 16.8

Hess’s Edu-Scholars: h, Print, & Web

Top 5 5 50 25 Bottom 50

10

20

30

40

50

60

70

80

h + books Print media Web

A Model that Considers Both Academic & Practice/Policy Contributions

I. CONTRIBUTIONS TO THE ACADEMY

CONTENT (up to 45 points) ACCESSIBILITY TO CONTENT (up to 5 points)

II. CONTRIBUTIONS TO PRACTICE & POLICYCONTENT (up to 45 points) ACCESSIBILITY TO

CONTENT (up to 5 points)

Academic ContributionsI. CONTRIBUTIONS TO THE ACADEMY

CONTENT (up to 45 points) ACCESSIBILITY TO CONTENT (up to 5 points)

•Google Scholar: h index •1 point for presence; 1 point for recency:

•Books (including Kindle):Single-authored: 2Co-authored: 1.5Edited: 1

--Academia.edu--ResearchGate--Website with articles/presentations posted--Vita & profile at institutional or professional website--wiki page

Practice & Policy Contributions

II. CONTRIBUTIONS TO PRACTICE & POLICYCONTENT (up to 45 points) ACCESSIBILITY TO

CONTENT (up to 5 points)

•Professional Publications (LCP Index)

Literacy Constituency Publications (LCP):1 point for every 10,000 readers; no publication gets more than 4 and none (including

state publications and university/organization newsletters) gets less than .25

Publications Circulation WeightThe Reading Teacher 28,500 2.85Principal Leadership 20,000 2Ed Leadership 195,000 4Ed Leadership online 195,000 4Reading Today 46,000 4Language Arts 4,000 .25Phi Delta Kappan 21,000 2.1J. of Adolescent and Adult Literacy 10,500 1.5Principal 35,000 3.5The School Administrator 35,000 3.5Instructor 100,000 4Homeschooling magazine 20,000 2State journals .25

Practice & Policy ContributionsII. CONTRIBUTIONS TO PRACTICE & POLICY

CONTENT (up to 45 points) ACCESSIBILITY TO CONTENT (up to 5 points)

•Professional Publications (LCP Index)•Digital Content (5: resources; 5: viewership; 5: currency) •Blog •YouTube •Podcasts •Webinars & MOOCS •Website •Pinterest •Google Circles•Curriculum/instructional materials (1 point per title)•Other

1 for viewership; 1 for currency in postings:•Facebook •Twitter•Linkedin•Instagram•Teacher-on-line communities/Nings•Media: Newspaper mentions—public or educational, including university

NANCY BRYNELSON—FROM THE PERSPECTIVE OF A STATE CENTER DIRECTOR

JAN DOLE—FROM THE PERSPECTIVE OF A SCHOLAR, TEACHER EDUCATOR, & STATE LITERACY ASSOCIATION LEADER

Janice A. DoleUniversity of Utah

What counts?

The Questions

• What counts in determining the impact of academic scholarship in your context?

• What forms of scholarship are most useful to you in your role in the profession?

My Contexts

• Academia– Academic positions at Research Intensive/Extensive Universities– Work at a research center– Conducting, reviewing, presenting, publishing research to other

researchers • Literacy Field

– Teacher– State Department of Education– Local, state and national institutions and organizations– Students, teachers, administrators, legislators– Work at the local, state, national levels

What counts in determining the impact of academic scholarship in your context?

In Academia• Publication in archival journals

(refereed) • Grant-funded research• Chapters published in edited

volumes• Editorship of scholarly journals • Presentations at national

conferences (refereed)• Leadership in national and state

professional organizations • [scholarly impact is determined

qualitatively, not by impact scores]

In the Literacy Field• Advisement to publishers of K-12

materials• Advisories to state agencies and

school systems• Books published for teacher

audiences • Leadership in national, state, and

local professional organizations• Presentations at state and local

professional organizations • Workshops/Professional

Development• [scholarly impact determined

qualitatively as well]

What forms of scholarship are most useful to you in your role in the profession?

For My Research– Research in top-tier journals– Conceptual and theoretical

pieces– Reviews of research and

meta-analyses– Presentations at national

conferences– Google Scholar– [Look for information that has

been vetted as much as possible]

For Informing the Field

• Reviews of Research (meta-analyses)

• Conceptual and theoretical pieces

• Handbooks• Webinars• Presentations at conferences• Google, Google Scholar• [Look for information that

has been vetted as well]

What forms of scholarship are most useful to you in your role in the profession?

• What I read to keep me informed about what is going on in the literacy field:– Some websites—TextProject, Shanahan on

Literacy, Reading Rockets, Colorin’Colorado– Some information from regional labs, federal

agencies, webinars, teacher books– State and national professional conferences– What Works Clearinghouse (http

://ies.ed.gov/ncee/wwc/)

COLIN HARRISON—WHAT COUNTS FROM AN INTERNATIONAL PERSPECTIVE

Colin HarrisonLearning Sciences Research Institute/School of EducationUniversity of Nottingham

colin.harrison@nottingham.ac.uk

What is ‘Impact’? How should we define it?How might we measure it?How might we improve it?

Impact in the UK:

1 The REF- Research Excellence Framework2 The Internet: Google Scholar3 The Internet: YouTube4 RCUK School-University Partnerships

• Four publications per professor• Graded by a panel of senior professors as

**** World-leading*** International but not ‘world-leading’** International but not three-star* Recognised as of national importanceU Unclassified

1. The UK Universities REF: Research Excellence Framework (2014)

How might we measure it? The MESH Guide to teaching spelling (6000+ views)

1,619 Google Scholar citations?

[PDP has 22,500]

2. The Internet

Colin: 1,202 YouTube views

[PDP ‘Impact of Educational Research’ has 8,769]

3. YouTube

3. YouTube

How might we measure ‘Impact’? … 81,000,000 views

3. YouTube

4. RCUK School-University Partnerships

http://www.publicengagement.ac.uk/work-with-us/current-projects/school-university-partnerships-initiative

Colin HarrisonLearning Sciences Research Institute/School of EducationUniversity of Nottingham

colin.harrison@nottingham.ac.uk

What is ‘Impact’? How should we define it?How might we measure it?How might we improve it?

CANDACE KUBY—WHAT COUNTS FROM THE PERSPECTIVE OF AN EARLY CAREER PROFESSOR

KEITH LYONS—WHAT COUNTS FROM THE PERSPECTIVE OF A GRADUATE STUDENT

ROSSETTI LOBUS—WHAT COUNTS FROM THE PERSPECTIVE OF A PUBLISHER OF CURRICULAR MATERIALS

What Counts as Scholarly Impact?

Katherine Rossetti Lobus Literacy Research Association Annual Conference

December 4, 2015

What Counts as Scholarly Impact?

Katherine Rossetti Lobus

ELA Editorial Director

Curriculum Associates, LLCNorth Billerica, MA

www.curriculumassociates.com

Disclaimer: This presentation is grounded in my professional publishing experience. While I can’t speak for all member of the publishing community, I believe my perspective reflects those commonly held by many of my colleagues.

What is scholarly impact from the viewpoint of a content developer?

Scholarly impact is making researched-based instructional practices accessible to teachers

and students.

Best job in the world!

Why base development efforts in scholarly work?

• Historically, research-based practices have always grounded instructional strategies and materials.

• NCLB required publishers to provide their research base.

• Even in our Common Core-world, teachers and administrators expect all materials to be steeped in research-based instructional practices. Typically, a key criteria point on most state and private evaluation rubrics specifically states:– “Instruction reflects current and confirmed research in English

language arts.” California ELA/ELD Framework, pp. 11, 2014.

Why base development efforts in scholarly work?

Most importantly…

– It is my ethical responsibility to children, the leaders of tomorrow,

to provide proven, research-based instructional materials in the

hands of their teachers.

– On the flip side, as a member of the publishing world, I have a

responsibility to you, the scholars, to accurately interpret and

apply your research with fidelity in the materials we develop for

classroom use.

Where do I find scholarly resources?

Conventional methods…– Books, books, books…print and electronic

– Professional Journals• The Elementary School Journal • The Reading Teacher

– Professional Organizations and Conferences (National and State/Local) • LRA, ILA, NCTE, NABE, CGCS, ASCD, and many more…

Where do I find scholarly resources? But, more and more in our digital world…– Websites

• TextProject.org • Understanding Language

– Webinars and MOOCs– Blogs

• Shanahan on Literacy • Education Week

– One-on-one meetings

– And…for those of you who I have been fortunate to work with, phone calls, lots of phone calls!

What is Scholarly Impact?

• Scholarly impact is the research-based instructional practices and

materials coming to life in classrooms.

• Making scholarly work accessible through print and digital mediums

will make the goal of building strong, independent readers and writers

a reality!

JUDY WALLIS—FROM THE PERSPECTIVE OF A SCHOOL DISTRICT LITERACY SUPERVISOR

Literacy Programs Designed Upon High-Quality Research

Shared Beliefs

Related Practice

sResource

s to Support

Impact of “Research-Based” Policy and/or Resulting Programmatic Shifts

Resources/

“Programs”

Associated

Practices Beliefs

Vicious Cycle:Often related to interpretive research

P. DAVID PEARSON—WHAT COUNTS FROM THE PERSPECTIVE OF A SENIOR LITERACY SCHOLAR