Measuring intranet performance - task based approach
25
Copyright © Customer Carewords Ltd. Brian Lamb @blamb Measuring Intranet Performance - Task based approach
-
Upload
intranaetverk -
Category
Business
-
view
156 -
download
0
Transcript of Measuring intranet performance - task based approach
Copyright © Customer Carewords Ltd.
Brian Lamb
@blamb
Measuring Intranet Performance - Task based approach
Current
website
Customers
Stakeholders
Competitors,
strategies and
other documents
Long list Short list Survey to
uncover the
customers’ top
tasks
New website
Please look at the following list and choose ONLY the Top FIVE Topics that are MOST IMPORTANT to you. Top 20
Tasks Total% of
3795
Cumul
Vote
Cumul
WordsGrants / equity funding from Enterprise
Ireland (eligibility, description) 332 9% 9% 1%
Research and development funding 210 6% 14% 2%
Grants / equity funding online applications 205 5% 20% 4%
Market / competitor research 144 4% 23% 5%
Commercial funding sources (VC, seed,
business angels) 128 3% 27% 6%
Market entry strategy advice 120 3% 30% 7%
Introductions to key industry decision
makers / advisors 100 3% 33% 8%
EU funding for companies 99 3% 35% 10%
Find Enterprise Ireland people
(development, market, technical, legal) 99 3% 38% 11%
Tax relief (BES, seed capital) 91 2% 40% 12%
How-to guides 83 2% 42% 13%
Introductions to buyers (buyer feedback,
follow-up) 81 2% 45% 14%
Find an Enterprise Ireland office (HQ,
regional, international) 79 2% 47% 16%
Target customer / distribution channel
identification 74 2% 49% 17%
Trade fairs 73 2% 51% 18%
Mentoring 72 2% 52% 19%
International marketing and sales training 71 2% 54% 20%
Claim grant payments online 68 2% 56% 22%
Intellectual property (patent, trademarking,
copyright) 68 2% 58% 23%
Help setting up overseas offices 66 2% 60% 24%
Example data from enterprise-ireland.com with permission
Long Neck
0
20
40
60
80
100
120
140
160
180
200
1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96 101 106 111 116 121 126 131 136 141 146 151
Carewords
Vo
te
The Kheel Center for Labor-Management Documentation and
Archives
https://creativecommons.org/licenses/by/2.0/
Observe users
trying to
complete
#TopTasks
Note
Success/Failure
and time taken
Use the Master Apprentice model
where users are the masters
and the observers are the apprentices
45%
37%
45%
27%
39%
75%
75%
75%
75%
75%
0% 10% 20% 30% 40% 50% 60% 70% 80%
Task: find a person
News
Benefits
Leave
Overall Performance
Performance level
Metric
Top Tasks– Performance Dashboard
Management Model
Identify top tasks
Measure top tasks
Improve Top
Tasks
Example Performance Dashboard
Metric Current Task
Performance
Target
Performance
Current improvement
actions
Task: find a
person
45% 75% Implementation of new
SharePoint solution
News 37% 75% Improving archive function
Improving links guidelines
Benefits 45% 75%
Leave 27% 75%
Search Not yet
measured
Progress against plan
Common
architecture
Step 2: Recruit Participants
Recruit 15-18
representative
people
Remote Measurement Sessions
1. Record session
2. “We are not measuring you”
3. Measure time for each task.
4. Observe problems and successes.
5. No “think aloud”, no discussions
Test representative tasks
Tasks Total Vote
% of Total
Vote 8730
Cumulative
Vote
Cumulative
Carewords
1
Find people (phone directory/book,
expert, employee photos) 915 10% 10% 1%
2
Pay / salary details, view your
paycheck, tax witholdings 818 9% 20% 3%
3 Board news (internal for employees) 815 9% 29% 4%
4
Benefits (enrollments, insurance
coverage, workers compensation) 453 5% 34% 5%
5
Leave and attendance (holidays,
maternity, sick, annual, overtime) 395 5% 39% 7%
6
Online applications (Oracle,
PeopleSoft, Discoverer, JP Morgan
Chase, PaymentNet, Travel Manager) 372 4% 43% 8%
7 NewsClips (from newspapers) 339 4% 47% 9%
8 Credit Union 282 3% 50% 11%
9 Shuttle schedule 282 3% 54% 12%
10 Policies and procedures 240 3% 56% 14%
11
Parking, carpools/vanpools, lottery,
passes 231 3% 59% 15%
12 Cafeteria menus 226 3% 61% 16%
13 Division news 221 3% 64% 18%
14
Calendars (events, meetings,
holidays, pay, training) 203 2% 66% 19%
15 Retirement (Thrift Plan, gifts, parties) 201 2% 69% 20%
16 Organization charts 183 2% 71% 22%
17 Forms, memo templates 154 2% 73% 23%
18 Visitors (tours, registration) 153 2% 74% 24%
19 Job postings, vacancies 150 2% 76% 26%
20
Information directory (finding
functions in the Board) 133 2% 78% 27%
Task ItB Find People poll result*
Find out which immediate section/unit XX works
in
1 – Employee’s immediate
unit/section
Find out which immediate section/unit XX works
in
1 – Employee’s immediate
unit/section
Find out who the Chief Financial Officer reports
to
3 – Place in organization chart
Find out who the Director of the Corporate
Affaires is
3 – Place in organization chart
XXis out of the office but you need to speak to
someone else on his team. Find another person
who works in the same immediate section/unit
4 – Who to contact if this person is
not around
Find the mobile phone number for XX 5 – Employee’s mobile phone
number
Find an expert in Web Usability 6 – Expertise *Note: Poll result #2 was photo/picture
Find out which immediate section/unit XX works in (top task #1)
2010
Sharepoint 6/2013
Sharepoint 9/2013
(after changes)
80%
20% Success
FailureMedian time on
task = 44
seconds
63% 12%
25% Success
Failure
Median time on
task = 57
seconds
100%
Success
Failure Median time on
task = 32
seconds
Find out which immediate section/unit XX works in (top task #1)
2010
Sharepoint 6/2013
Sharepoint 9/2013
(after changes)
Median time on
task = 35
seconds
100%
Success
FailureMedian time on
task = 14
seconds
100%
Success
Failure
68% 6% 25%
Success
FailureMedian time on
task = 38
seconds
Find out who the Chief Financial Officer reports to (top task #3)
Old ItB (2010)*
Sharepoint 6/2013
Sharepoint 9/2013
(after changes)
40% 27% Success
Failure
Disaster33%
Median time on
task = 73
seconds
38%
50%
12%
Success
Failure
Disaster
Median time on
task = 135
seconds
100%
Success
Failure
Disaster
Median time on
task = 120
seconds**
(45 seconds)
**”All sites” link causes
problems
XX is out of the office but you need to speak to someone else on his team. Find another person who works in the same immediate section/unit (top task #4)
Old ItB (2010)*
Sharepoint 6/2013
60%
7%
Success
Failure
Disaster
88%
12%
Success
Failure
Disaster
33
%
Median time on task =
45 seconds
Median time on task =
57 seconds
Find out who the Director of the Office of Inspector General is (top task #3)
Sharepoint 6/2013
Median time on
task = 122
seconds
38%
62%
Success
Failure
Disaster
Problems:
• Searching by division acronym “OIG” doesn’t help
• Mr. XXs title “Inspector General” led to confusion
• Section XX org chart in Sharepoint is useless
Find the mobile phone number for XX (top task #5)
Sharepoint 6/2013
100%
Success
FailureMedian time on task = 14
seconds
Find an expert in web usability (top task #6)
Sharepoint 6/2013 63%
12%
Success
Failure
Disaster25%
Median time on task = 52 seconds
Problems:
• What does “Ask Me About” really mean?
Expertise?
• “All sites” tab causing problems
• Just not intuitive you can do this
Aggregate results
Old ItB (2010)
Sharepoint 6/2013
Sharepoint 9/2013
(after changes)
63%
24% Success
Failure13%
67%
12%
Success
Failure
21%
Median time on task = 52
seconds
Median time on task = 76
seconds
100%
Success
FailureMedian time on task = 55
seconds**
(25
seconds) **”All sites” link causes
problems
What does this mean?
What does this mean? – People get slightly more accurate answers with Sharepoint,
but it takes them longer to find those answers. To vastly
improve the usefulness of Sharepoint and to make it easier
to Find People, we need to improve Sharepoint.
So, how do we improve?
How to improve 1 – the easy stuff
Simple, high impact improvements (what
worked in the September retests?) – Explicitly labeling fields (Division, Section, Job Title,
etc)
Mysite pages
Search results pages
– Making it easy for people to search by both acronym
and full name
Divisions
Job titles
– Allow search by Division (full name and acronym) from
the main search box
How to improve 2
Other suggestions
– Get rid of “All sites” button
– Improve trust in the org chart viewer
Accuracy
Design (who reports to whom?)
– Taxonomy must work well or people will
quickly lose trust in it
– Improve uniformity (job titles, dealing with
misspellings, section names, division names,
etc)
MEASURE
• Find out who the Chief Financial Officer reports to (top task #3)
[email protected] Image from: Ari Bakker http://www.sharepointconfig.com/2011/05/how-to-create-a-simple-sharepoint-2010-people-directory/
How to improve 1 – the easy stuff
Simple, high impact improvements (what
worked in the September retests?) – Explicitly labeling fields (Division, Section, Job Title,
etc)
Mysite pages
Search results pages
– Making it easy for people to search by both acronym
and full name
Divisions
Job titles
– Allow search by Division (full name and acronym) from
the main search box
