1

Sondra Stein & Katie LanderosAmerican Institutes for Research

Findings from the Program for the International Assessment of Adult

Competencies (PIAAC), 2012

November 12, 2014

About PIAAC

PIAAC is an international large-scale assessment administered in 2011-12 in 23 countries

It assessed 16 - to 65-year-olds, non-institutionalized, residing in each country, irrespective of nationality, citizenship, or language statusLaptop computer or paper-and-pencil:

In the U.S., 80% took the computer tests and 15% took the paper-and-pencil tests.

Assessment subjects:

Literacy

Numeracy

Problem Solving in Technology-Rich Environments (digital problem solving)

Assessment was conducted only in English in the U.S.:

The background survey was conducted in English or Spanish. About 4% could not complete the BQ because of language difficulties or learning or mental disabilities, and 1% could not complete it for other reasons.

2

Participating Countries2012 2015

AustraliaAustriaBelgiumCanadaCyprusCzech RepublicDenmarkEstoniaFinlandFranceGermanyIreland

ItalyJapan Korea, Rep ofNetherlandsNorwayPolandSlovak RepublicSpainSwedenUnited Kingdom United States

ChileGreece IndonesiaIsraelLithuaniaNew ZealandSingaporeSloveniaTurkey

3

The PIAAC Assessment was delivered to a nationally representative sample of households in every country.

In the U.S. the household sample was selected through a 4-stage stratified area sample:• Counties (PSUs)• Blocks• Housing units with households• Eligible persons within households

Resulted in 5,010 respondents A U.S. supplement will add 3,600 more adults (report due late 2015)

that represent key populations (young adults 16-34 yrs, older adults 66-74 yrs, unemployed adults,16-65 yrs)

A representative prison sample will include 1,200 inmates, 16-74 yrs, in state, federal and private prisons (report due 2016).

4

PIAAC provides a rich source of data that tells us: What skills adults actually have and can use rather than

just the number of years of education they have completed or the degrees they have.

How adults acquire those skills, and what factors are related to skill acquisition and decline.

What the level and distribution of skills is within and across various subgroups within the population.

As a result. PIAAC data enables us to target our efforts to focus on raising the skills of adults with the greatest needs.

5

PIAAC collects data through it’s background questionnaire and module on skill use as well as through direct assessment of skills.

Direct assessment

of key information-processing

skills

Module on skill use

Background questionnaire

6

PIAAC Background QuestionnaireFocused on identifying:

Skills that are critical to functioning successfully in today’s society,

How participants acquire those skills, and How those skills are distributed.

Areas of BQ include: Education and training, present and past, Work experience, Literacy, numeracy and ICT skill use at work and at home, Other 21st century skills used at work, Personal traits, and background information.

7

5 min

Background questionnaire

Every country could add up to 5 minutes of unique questions and also make country-specific adaptations. U.S. changes include:

Additions• Basic skills training• Political Efficacy -

Information• Health • Race/Ethnicity• Language

Adaptations: • Formal Education, Informal

Training• Occupation, Economic

Sector, Earnings

88

The Direct Assessment focuses on four domains:

Literacy: both paper & pencil and computer versions Numeracy: both paper & pencil and computer versions Problem solving in technology-rich environments: only

on computer Reading components: only paper & pencil All countries were required to administer literacy and

numeracy assessments The U.S. assessed all four domains

9

Definitions of PIAAC direct assessment subjects Literacy is understanding, evaluating, using and engaging

with written texts • to participate in society, • to achieve one’s goals, and • to develop one’s knowledge and potential.

Numeracy is the ability to access, use, interpret, and communicate mathematical information and ideas, in order to engage in and manage the mathematical demands of a range of situations in adult life.

Problem solving in technology-rich environments involves using digital technology, communication tools and networks to acquire and evaluate information, communicate with others and perform practical tasks.

10

PIAAC results are reported in two ways:

Average Scores: reported on a scale of 0-500 for all domains.

Proficiency Levels: reported as the percentages of adults scoring at six performance levels ( from below level 1 to level 5) in literacy and numeracy and at four performance levels in problem solving in technology-rich environments (from below level 1 to level 3).

11

These descriptions of the PIAAC Proficiency Levels for Literacy define what adults can do at each level.

12

Locate single piece of information in familiar texts.

Read relatively short digital, print or mixed texts to locate single text.

Make matches between text and information that may require low level para-phrasing and drawing low-level inferences.

Identify, interpret, or evaluate one or more pieces of information and often require varying levels of inference.

Perform multiple-step operations to integrate, interpret, or synthesize information from complex texts, and may require complex inferences.

Integrate information across multiple, dense texts; construct syntheses, ideas or points of view; or evaluate evidence based arguments.

Below Level 1 (0-175)

Level 1 (176-225)

Level 2 (226-275)

Level 3 (276-325)

Level 4 (326-375)

Level 5 (376-500)

Election results

Stimulus is a short report on the results of a union election. The report contains several brief paragraphs and a simple table identifying the three candidates in the election and the number of votes they received.• The test-taker is asked to identify which candidate received the

fewest votes. • He or she needs to compare the number of votes that the three

candidates received and identify the name of the candidate who received the fewest votes.

• The word “votes” appears in both the question and in the table and nowhere else in the text.

13

Below level 1 literacy item

14

Literacy itemLevel = 1

Literacy example – low difficulty

15

Literacy itemLevel = 4 (low)

Literacy example – moderate difficulty

16

Overview of U.S. resultsPIAAC results tell a story about the systemic nature of the skills deficit among U.S. adults. PIAAC raises the question:What are we going to do to make sure that

• The U.S. has the workforce it needs to succeed in the global economy?

• U.S. citizens have the skills necessary to support a thriving democracy?

How did we do compared to other countries?

17

JapanFinland

NetherlandsAustraliaSwedenNorwayEstonia

Flanders-Belgium

Slovak Rep.

Germany

France

Czech Rep.

Canada

ItalySpain

U.K.

Denmark

United States

IrelandPolandCyprus

Austria

Japan

Finland

Flanders-BelgiumNetherlands

SwedenNorway

DenmarkSlovak Rep.Czech Rep.

AustriaEstonia

GermanyAustralia

Canada

Cyprus

Korea, Rep. of

Korea, Rep. of

U.K.

Poland

IrelandFrance

United StatesItaly

Spain

JapanFinland

AustraliaSwedenNorway

NetherlandsAustria

DenmarkCzech Rep.

Korea, Rep. ofGermanyCanada

Slovak Rep.

Flanders-BelgiumU.K.

Estonia

United States

Ireland

PolandItaly

Spain

Cyprus

France

Literacy Numeracy PS-TRE

The U.S. ranked lower than most other countries in all three domains.

The US ranked better in Literacy than in Numeracy or Problem Solving in technology-rich environments.

18

JapanFinland

NetherlandsAustraliaSwedenNorwayEstonia

Flanders-Belgium

Slovak Rep.

Germany

France

Czech Rep.

Canada

ItalySpain

U.K.

Denmark

United States

IrelandPolandCyprus

Austria

Korea, Rep. of

Literacy

Scores on literacy ranged from 296 (Japan) to 250 (Italy)

U.S. scores were:• Lower than in 12 countries• Not significantly different

than in 5 countries• Higher than in 5 countries

The U.S. average literacy score (270) was lower than the international average (273).

The U.S. average is low because a higher proportion of U.S. adults are at the lowest levels (level 1 and below level 1) of literacy.

19

20

Grad or prof. degree (10%)

Bachelor’s degree (16%)

Associate’s degree (9%)

High school credential

(50%)

Below high school (15%)

0

5

10

15

20

25

30

35

40

45

Below Level 1 Level 1 Level 2

Pe

rce

nta

ge

of

po

pu

lati

on

Who are the low-skilled adults in the U.S. in literacy?Educational attainment

21

White, non-Hispanic (65%)

Black, non-Hispanic (13%)

Hispanic (14%) Other (8%)0

5

10

15

20

25

30

35

40

45

Below Level 1 Level 1 Level 2

Pe

rce

nta

ge

of

su

bp

op

ula

tio

n

Who are the low-skilled adults in the U.S. in literacy?Race/ethnicity

22

English as first language (85%)

Other as first language (15%)0

5

10

15

20

25

30

35

40

Below Level 1 Level 1 Level 2

Pe

rce

nt

of

su

bp

op

ula

tio

n

Who are the low-skilled adults in the U.S. in literacy?First language

23

Born in U.S. (85%) Not Born in U.S. (15%)0

5

10

15

20

25

30

35

40

Below Level 1 Level 1 Level 2

Pe

rce

nta

ge

of

su

bp

op

ula

tio

n

Who are the low-skilled adults in the U.S. in literacy?

Immigration status

All (100%) Employed full-time (52%)

Employed part-time

(12%)

Unem-ployed (8%)

In school (10%)

Retired (4%)

Perma-nently dis-abled (5%)

Looking af-ter family

(6%)

Other (3%)0

10

20

30

40

50

Below Level 1 Level 1 Level 2

Per

cen

tag

e o

f su

bp

op

ula

tio

n

Who are the low-skilled adults in the U.S. in literacy?

Employment

24

25

Elem

enta

ry o

ccup

atio

ns (8

%)

Mac

hine

ope

rato

rs (6

%)

Crafts

and

trad

es w

orke

rs (9

%)

Servic

e wor

kers

/sale

s (21

%)

Clerk

s (8%

)

Techn

ician

s and

ass

ocia

tes (

16%

)

Legi

slato

rs, o

fficia

ls, a

nd m

anag

ers (

10%

)

Profe

ssio

nals

(21%

)05

101520253035404550

Below Level 1 Level 1 Level 2

Pe

rce

nt

of

su

bp

op

ula

tio

n

Who are the low-skilled adults in the U.S. in literacy?Occupation

26

Const

ruct

ion (7

%)

Admini

stra

tive

(6%

)

Trans

porta

tion

(4%

)

Accom

mod

ation

and

food

ser

vice

(7%

)

Retail

trad

e (1

2%)

Man

ufac

turin

g (1

1%)

Health

(14%

)

Oth

er (3

%)

Arts a

nd e

nter

tainm

ent (

2%)

Public

adm

inist

ratio

n (7

%)

Financ

ial s

ervic

es (5

%)

Educa

tion

(9%

)

Info

rmat

ion a

nd c

omm

unica

tion

(4%

)

Profe

ssion

al an

d sc

ientifi

c (5

%)

0

5

10

15

20

25

30

35

40

45

50

Below Level 1 Level 1 Level 2

Pe

rce

nta

ge

of

su

bp

op

ula

tio

n

Who are the low-skilled adults in the U.S. in literacy?Industry

1960 1970 1980 1990 2000 2006 200935.0

40.0

45.0

50.0

55.0

60.0

65.0

70.0

Routine manual

Non-routine manual

Routine cognitive

Non-routine analytic

Non-routine interpersonal

Source: Autor, D. H. and B.M. Price (2013), "The Changing Task Composition of the US Labor Market: An Update of Autor, Levy, and Murnane (2003)", MIT Mimeograph, June.

Mea

n t

ask

inp

ut

in p

erce

nti

les

of

1960

dis

trib

uti

on

Since 1970, there has been a shift in the U.S. economy away from routine and manual tasks and towards more analytic and interpersonal tasks that require higher skills.

27

Percentage Change in Earnings Since 1961

Tabulations of annual March Current Population Survey Data, by David Ellwood, Harvard University.

Slide prepared by ETS.

The percentage change in earnings since the 1970s mirrors that shift in occupations

28

Literacy skills in younger and older generations

29

Approximately 36 million U.S. adults have low skills.

Roughly the same population of Minnesota, New York, and Michigan states combined.

30

MN NY

MI+ +

Time for the U.S. to Reskill? What the Survey of Adult Skills Says (an OECD report)

Workforce Innovation and Opportunity Act (WIOA) and Ready to Work Signed into law July 2014 Previously 1998 Workforce Investment Act Ready to Work: Job-Drive Training and American

Opportunity report also released Both initiatives aim to “train Americans with the skills

employers need, and match them to good jobs that need to be filled now.”

31

32

1. Well-connected and transparent education, training, credentials, and support services

Increasingskills, competencies, andcredential

s

e.g., license, industry credential

3rd Job in

Career

Path2nd Job in

Career

Path

1st Job in

Career

Path

informed by

industry/employer

s

e.g., certificate, diploma

e.g., 2-year degree

e.g., 4-year degree

Nth Job

in

Career

Path

2. Multiple entry points – for both well-prepared students and targeted populations

bridge(s)

3. Multiple exit points

e.g., high school or CTEe.g., adult

education or

workforce system

e.g., military or civilian workplace

e.g., postsecondary system

e.g., apprenticeship

Three Essential Features of Career Pathways

A higher percentage of Blacks and Hispanics A higher percentage in manual and blue and

white-collar semiskilled occupations, including the following sectors:• Construction

• Administrative services

• Transportation

• Hospitality

• Retail

• Health

Who are the low-skilled adults in the U.S. workforce?

3333

Barriers to participation in Non-Formal Education (NFE)

I did not have the prerequisites2%

Education or training was too expensive/I

could not afford it24%

Lack of employer's support4%

I was too busy at work26%

The course or programme was of-fered at an inconvenient time or

place13%

I did not have time because of child care or family responsibilities

16%

Something unexpected came up that prevented me from taking

education or training6%

Other10%

3434

35

What can we do to raise the skills of working age adults?

In the long run, the best policies we have [to combat inequality] involve investing in our citizenry. …

Higher education, and public education, is America’s best idea.

-- David Autor

35

Do a better job of assuring every American has a strong foundation of basic skills.

Do a better job of providing skills upgrading for immigrants to America.

Do a better job of providing skills upgrading to working adults at every level so they can keep pace with change.

Understand that improving the skills of our population is everybody’s business: we can educate our way to a better economy.

What does the data suggest about what we need to do differently?

36 36

How PIAAC can help

Use PIAAC data to advocate for more investment in adult lifelong learning.

Use PIAAC data to strengthen program approaches to building lifelong career pathways.

Use PIAAC data to create and renew local, state, and national approaches to lifelong learning where learning is everybody’s business.

Use PIAAC data and tools to document successes.

3737

A test of Education and Skills Online – an online version of PIAAC that will be available in 2015.

Outreach Toolkits (coming soon) so you can share this data with others.

Access to the Data itself, through the NCES or OECD International Data Explorer.

Research reports that analyze the data. Regular updates through the PIAAC Buzz.

What else PIAAC offers at: www.piaacgateway.com

38 38

39

PIAAC Outreach ToolkitsThese Outreach Toolkits include all documents and power point

modules you will need to share information about PIAAC with others. *Simply download the pieces you want to use!*

• PIAAC Overview

• Key Results

• National Supplement

• Education & Skills Online (ESO)

• Videos

• Infographics

Toolkit Slides for Your PresentationInstructions: Start with “What is PIAAC” and “Results Overview”. Then add other modules to build a presentation that suits your

audience. What

is PIAAC

Results Overview

These can be downloaded

with a focus on literacy,

numeracy, ordigital problem-

solving

Sample PIAAC Tasks

Gateway & Other

Resources

NationalSupplemen

t

ESO

Key U.S. Issues

• Low-skilled Workers• The Future

Workforce•Health Status & Skills• The Impact of Parent

Education

(Each of these can be downloaded with a focus on literacy, numeracy ordigital problem-solving)

Toolkit Documents

39

Visit the PIAAC Gateway at: www.piaacgateway.com

40

Questions and Discussion

41

For more information about PIAAC

Visit our website at https://piaacgateway.com Or contact us at piaac@air.org

Sign up for our regular newsletter, the PIAAC Buzz at https://piaacgateway.com

42

www.piaacgateway.com

43

Links to everything you want to know about PIAAC, including:

The latest PIAAC reports and presentations

A calendar of PIAAC-related events

Infographics, brochures, and videos to share

Data tools and training resources

Press coverage from the U.S. and around the world

Links to release events, reports, and presentations

What you can find on the PIAAC Gateway

How to Access PIAAC Data The PIAAC Results Portal enables you to produce figures and tables that take

an in-depth look at the U.S. results as well as compare the performance of U.S. adults to adults in other participating countries. http://nces.ed.gov/surveys/piaac/results/makeselections.aspx

The US PIAAC International Data Explorer (IDE) is a web-based tool that

produces customizable tables and graphs using data from PIAAC. The US IDE includes US national variables as well as data from all other participating countries from the administration of PIIAC in 2012. It is free of charge and does not require any advanced statistical software or knowledge. http://nces.ed.gov/surveys/international/ide/

PIAAC IDE Training offers online training for researchers, policymakers, and practitioners who want to use PIAAC data to answer questions they have about adult skills. To sign up for a webinar or arrange one for your group, contact piaac@air.org

44 44

National Center for Education Statistics’ PIAAC International Data Explorer (IDE)

45

What is the NCES IDE?PIAAC Data Explorer DemonstrationAnswering your research questions

NCES PIAAC Data Explorer (IDE)

46

A web-based application for accessing PIAAC data that does not require any advanced statistical knowledge or software

A point-and-click interface for:

• Creating statistical tables and charts

• Exploring levels of adult skills and demographics

What is the NCES IDE?

47

The PIAAC Data Explorer can:• Compute simple descriptive statistics such as:

– averages, percentages, and percentiles– percentages of students at pre-defined proficiency levels

• Run cross-tabulations • Run significance tests on computed statistics• Collapse response categories within a variable• Create and export charts, graphs, and tables

It cannot:• Run regression models or multi-level analysis• Compute new variables

What the NCES IDE can and can’t do

48

Overall scale (e.g. literacy, numeracy, problem solving in technology-rich environments, reading components)

Continuous variables from international and U.S. national background questionnaire (e.g. earnings/hours of work per week)

Demographic variables (e.g. age, education background, employment history)

U.S. specific background questionnaire variables (e.g. self-reported health status, race/ethnicity)

Trend variables from PIAAC (2012), ALL (2003), IALS (1994) Variables are organized into categories that have shared

characteristics

What’s in the NCES IDE?

49

Overall Process

2. Select Variables: Select at least one variable from the selection of categories and subcategories. 

3. Edit Reports: Preview how your data will look, and edit your report format options and statistics options as desired. 

4. Build Reports: Retrieve the data, make charts and graphs, save, and print reports. 

50

1. Select Criteria: Choose your measure(s), year(s), and jurisdiction(s). 

Select Criteria

51

Select Variables

52

Edit Reports

53

Edit Reports: Format Options and Statistics Options

54

Build Reports

55

56

Chart Options

Significance Test Options

57

Output: Data Table

58

Output: Significance Test

59

Output: Chart

60