Crowd Sourcing Tests Warren Bobrow, Ph.D. All About Performance, LLC [email protected].
-
Upload
magnus-watts -
Category
Documents
-
view
213 -
download
0
Transcript of Crowd Sourcing Tests Warren Bobrow, Ph.D. All About Performance, LLC [email protected].
Crowd Sourcing Crowd Sourcing TestsTests
Warren Bobrow, Ph.D.Warren Bobrow, Ph.D.
All About Performance, LLCAll About Performance, LLC
warren@[email protected]
What is Crowd Sourcing?What is Crowd Sourcing?
Obtaining services or content by Obtaining services or content by asking for input from an online asking for input from an online community.community. Software codeSoftware code KickStarterKickStarter Marketing CampaignsMarketing Campaigns Product Development Product Development
What Are the Potential What Are the Potential ApplicationsApplications
Test QuestionsTest Questions
Survey ItemsSurvey Items
Recruitment IdeasRecruitment Ideas
Test DevelopmentTest Development
Where appropriate for content valid Where appropriate for content valid teststests No in-house SMEsNo in-house SMEs Frequently changing contentFrequently changing content Unique jobs where there is not published Unique jobs where there is not published
contentcontent
In reality, it’s a very scaled up version In reality, it’s a very scaled up version of item banking from previous examsof item banking from previous exams
ConcernsConcerns Test SecurityTest Security
Risks are the same as with unproctored tests.Risks are the same as with unproctored tests. Perhaps less so if the test is adaptive.Perhaps less so if the test is adaptive.
Quality of the ItemsQuality of the Items Is the online expert community better or Is the online expert community better or
worse at writing items than your SMEs?worse at writing items than your SMEs? NormingNorming
Averages likely to be lower, but better item Averages likely to be lower, but better item statistics due to larger samples.statistics due to larger samples.
StandardizationStandardization Likely to be adaptiveLikely to be adaptive
ApplicationApplication
Virtual software companyVirtual software company Low number of SMEs and Low number of SMEs and
geographically spread outgeographically spread out
Growing fairly rapidlyGrowing fairly rapidly
Need for programmers and Need for programmers and professional staff (marketing, etc)professional staff (marketing, etc)
OutcomesOutcomes
CostCost It’s FREE (for now)It’s FREE (for now) Integrates into ATSIntegrates into ATS
Candidate ReactionCandidate Reaction Meets the face validity testMeets the face validity test Still some dissatisfaction as they items Still some dissatisfaction as they items
are multiple choiceare multiple choice
Test StatisticsTest Statistics
Tests are normedTests are normed
Not a presumption of a normal curve.Not a presumption of a normal curve.
Master = 780-800Expert = 700-779
Proficient = 450-699Familiar = 300-449Beginner = < 300
Observed StatisticsObserved Statistics
Type of TestType of Test MeanMean SDSD Median Median IntercorrIntercorr
ITIT 598.2598.2 162.6162.6 .32.32
OfficeOffice 639.1639.1 134.1134.1 .11.11
•Scores are above the midpoint of the middle range (450 to 699), which indicates that there are some non-experts in the crowd.
•Items have large SDs, indicating that there is good variance.
•Not all tests are given in combination with others, but the intercorrelations appear to be low-positive.
OutcomesOutcomes
Too early for criterion related study.Too early for criterion related study.
The automation and integration with the ATS The automation and integration with the ATS is a huge time saver for a company with a is a huge time saver for a company with a small HR staff.small HR staff.
Have not lost any candidates to the process.Have not lost any candidates to the process.
Used in combination with personality/aptitude Used in combination with personality/aptitude testing and structured interviews.testing and structured interviews.
What Does the Future What Does the Future Hold?Hold?
Unlikely that this will take over Unlikely that this will take over content validation testing.content validation testing. Not enough control over the process for Not enough control over the process for
man organizations.man organizations.
Likely to be adopted by tech startups Likely to be adopted by tech startups and small firmsand small firms Meets an important needMeets an important need Is a good cultural fitIs a good cultural fit
Q&AQ&A