1

Recommendations for Research Networking Systems (RNS) Usage Tracking – Version 1, July 2013

Foreword A subgroup of the CTSA Research Networking Affinity Group (RNAG) has developed a set of recommendations for SGC3, focused on measuring the usage of research networking systems (RNS).

Introduction Biomedical institutions have widely deployed institutional expertise discovery and researcher networking systems (RNS), including VIVO, Profiles RNS, and SciVal Experts. Institutions have invested in networking systems drawn by the promise of discovering complementary expertise, enabling collaboration, facilitating funding opportunity matching, and other key needs for accelerating translational research. Evidence is currently limited on the extent of usage of RNS tools and the extent to which they promote research-oriented networking. However, it is possible to measure usage of these systems using web traffic analytics tools, via analysis of web logs. The most well-known of these tools is Google Analytics, but a number of attractive alternatives are emerging.1 In addition, qualitative methods can be employed to collect feedback with respect to value and utility, especially from the user perspective. The recommendations here reflect current measures being used by 8 CTSA consortium member institutions and one commercial entity:

• Harvard University • Medical University of South Carolina • Northwestern University

• University of Iowa • University of Minnesota • University of California, San Francisco*

• Vanderbilt University • Washington University* • Elsevier*

This set of recommendations is a start. This document is meant to be a living and changing set of recommendations that will inform institutions and SGC3 of the value and power of research networking systems for accelerating translational research. For any specific institution, adopting a subset of these measures can help to shape strategy, decision making, and practical tactics.

1 Some notable examples include Google Analytics (http://www.google.com/analytics/), Site AI (http://www.siteai.com/), Woopra (http://www.woopra.com/), with NewRelic (http://newrelic.com ) being used for performance monitoring. See Appendix 1 for more on tools. *Authored Version 1 of this document

2

The structure of this document is as follows: The set of recommended RNS Usage Measures below a description of each measure and an accompanying rationale and explanation as to why the measure should be tracked. Institutional examples and case studies then describe successful strategies and tactics that have evolved from the analysis of the data, which range from informing the overall marketing strategy for adoption, to targeting specific faculty meetings for presentations and training, to gaining stakeholder and financial support. For ease of discussion and presentation, Google Analytics will be used as an example of an analytics tool to track tool usage.

Recommended RNS Usage Measures

For all measures: Track trends R1 Establish a baseline and track over time

Track the same measures over time to illustrate patterns and responsiveness to actions (e.g., a new marketing campaign).

Set expectations R2 Set expectation for trend over time

For trends, set expectations for how the measures will trend. At UCSF, while they hope to see the number of 2+

minute on-campus visits per day increase over time, they expect this measure to plateau over time if the majority of the core user community comes to make appropriate use of the tool whenever needed.

Separate internal from external usage R3 Distinguish between traffic internal and external to the institution

It easy to distinguish between institutional and extra-institutional traffic with Google Analytics by looking at the visitor’s service provider. By creating advanced segments that show only traffic from your institution’ service provider(s) vs. those from all other service providers, you can filter

UCSF set up two advanced segments: users with service provider “university of california san francisco,” and those with any other service provider. They then look at every statistic via both segments. This shows, for example, that they get over ten times as many off-campus visits as they do on-campus visits, and that on-

3

every statistic through an internal vs. external lens. (Be sure to that your team’s own accesses to the site are being ignored by filtering out your IP addresses as an account-level filter.)

campus visitors are more likely to arrive via campus websites than external visitors.

The basics:

Implement Google Analytics R4 Track basic web traffic statistics via Google Analytics

Google Analytics affords easy access to basic usage analytics s so you can determine:

1. Visits to your site 2. Page views 3. Unique visitors 4. % new vs. % returning visitors 5. Bounce rate on home and profile pages 6. # pages looked at per visit 7. Average visit duration

Understand search traffic R5 Use search traffic statistics to inform marketing and outreach opportunities

Google Analytics shows not only the amount of traffic from public search engines like Google and Bing, but also the keywords that brought users to your site. This can be used to understand search behavior and intent. For example, if search traffic is low, it might be an opportunity do basic search engine optimization; or if users are coming to the site after searching for people’s name on Google, it may provide an opportunity to highlight their networks on the landing page.

UCSF received only 2,500 visits a month from external search engines during launch month. They subsequently did search engine optimization work, and now receive over 40,000 visits a month from external search engines — 72% of all visits.

4

Delve into referral traffic R6 Use referral traffic statistics to inform marketing and outreach opportunities

Google Analytics also provides data on which website refer the most traffic to your system. By understanding these patterns, you can look at institutional traffic and sites and it can help to inform your outreach and marketing efforts. For example, if a web resource popular in the local research community does not refer much traffic it could be an opportunity for outreach to get links back to your system added at the appropriate spots.

Comparisons across institutions can also help here. For example, the 2nd highest referring site at UCSF is the campus directory. UCSF partnered with the campus group responsible for each person’s page on the main online people directory to link back to the corresponding UCSF Profiles page; the result was a substantial increase in traffic to UCSF Profiles -- about 1,700 visits a month.

Understand what’s popular R7 Understand traffic patterns to different parts of the site to understand preferential usage, and prioritize outreach and development

Google Analytics allows users to see what parts of the site are getting the most traffic, and how users behave on each page, e.g. by looking at bounce. Understanding these measures is critical to understanding usage of the site.

A month after launch, UCSF staff were surprised to discover that only 9% of visits begin on the home page, with most users arriving directly on profile pages. This provided the impetus to spend substantially more time improving data on profile pages, and deemphasize tweaks to the home page.

Look at your geographic breakdown R8 Examine regional and global traffic patterns to inform potential economic development impact

Google Analytics’s audience location demographics will show your traffic from within your region, state, nation, and the world. If you’re like many institutions, you might be surprised at how much of the traffic to your RNS comes from outside of your institution, let alone your state. In addition, look both at city, (for the institution’s city and the surrounding suburbs), as well as network provider which is often be linked to the university, sometimes even down to school or college within the university.

5

Benchmarking:

Define meaningful engagement R9A & R9B Track the number of “X+” minute visits to the RNS from within the institution (where “X+” represents the length of time you deem meaningful) Track the number of visits where users from within the institution view at least “Y” pages (where “Y” represents the # pages you deem meaningful)

For context, we are using the model of an RNS that does not require log-in to search the network. As such, discuss and define a surrogate for what constitutes meaningful engagement with your RNS. Once defined, these measures can be found via Google Analytics; complex queries can be used through advanced segments and/or custom dashboards.

At UCSF, they pay particular attention to the average number of visits from within the UCSF campus network lasting 2 minutes or longer (Google Analytics defines a visit as a period of continuous usage where the gap between subsequent page views or interactions is under half an hour). Because users do not need to log in to search UCSF Profiles, they use these 2+ minute on-campus visits as a proxy for deep active use of the product by researchers, administrators, and other staff on campus.

Benchmark the traffic R10 Benchmark relative to other campus web properties, both on and off campus

One way to infer relative usefulness of your RNS is to track and benchmark the RNS traffic relative to other campus properties such as the directory or the main institutional web site.

At UCSF, the RNS gets as much as 60% of the visits of the campus directory from all visitors, and 12% of the visits of the campus directory from UCSF campus visitors. This signals the relative importance of this system as a resource.

6

Segment populations R11 Segment and compare populations of profiled users, and of visitors (when possible)

Using the data in the RNS, segment populations of engaged users to determine differing or similar patterns of use. The differences can inform outreach and training opportunities, and value can be inferred. Examples of segments: faculty rank, department, school, etc. Segments with lesser adoption could be candidates for outreach and training, whereas segments with greater adoption are candidates for interviews on why and what is going well. If the system is behind a login wall, there may also be opportunities to segment end users.

By looking at UCSF Profile adoption rates (where adoption is defined as a profile that has some curation or customization completed such as an uploaded photo), they found that faculty at all levels – junior to senior – have adopted their profiles in close to similar proportions. This seems to indicate equal importance of the tool across the segments, and helps as they plan outreach and marketing for the system.

Analyze where people click R12 See what your users are clicking on and use the data to improve your product

A heatmap is an easy way to understand what users do on your site. It’s a visual representation showing where users click and what they do. This can help you determine what aspects of your RNS are most popular & what is overlooked. Google Analytics provides this data via in-page analytics feature, or a separate heatmap tool may be used. If certain classes of links on your site are being systematically ignored, this data can help inform user interface changes and/or marketing messages.

UCSF has installed an inexpensive heat mapping tool and used that along with Google Analytics in-page analytics to prioritize improvements to their RNS home page.

7

Qualitative measures: R13 Measure traffic impact resulting from various communications (e.g., PR and marketing campaigns) and then judiciously use the successful strategies in the future

Analyze traffic to your RNS before and after various marketing campaigns such as emails or trainings. Try different messages using A/B testing and measure which ones are most successful. Partner with other groups on campus so all of the messages don’t come from one source. Measure traffic impact from these various sources.

UCSF noticed a big spike in traffic after the Executive Vice Chancellor & Provost (EVCP) sent out an email announcing UCSF Profiles. This traffic spike was larger than any other email campaign and speaks to the reach and value of communications from UCSF’s EVCP. Also, at UCSF, groups across campus from the Library to University Relations have helped to spread the word about UCSF Profiles. These groups are on board to help promote UCSF Profiles because data mined from UCSF Profiles have facilitated those groups’ projects.

Survey users on value & usage R14 Survey users for direct feedback; use the feedback to inform strategy and gain support for your system

To really find out how people use the system and what value they garner from your RNS, you will need to ask them. The feedback will identify gaps and improvements, inform planning and hopefully help gain support to sustain and enhance your system.

A number of SciVal Experts’ sites track numbers of inquiries and/or profile corrections as additional evidence of how many people are really looking at the site and its content. REACH-NC (NC state SciVal Experts portal) created a contact form that users fill out when trying to contact a researcher and they do not display direct contact information. The contact form goes to the campus liaison who coordinates the contact. In addition, a copy of the request goes to the REACH-NC Executive Director and Program Coordinator so they can track request and monitor connections. In addition, inquiries and feedback from profiled researcher come to a group mail box that the ELS Account Development Manager personally manages, instead of directly going to the general ELS helpdesk.

8

Arizona State University had a ‘Submit Feedback’ button on the Experts site that linked to an end-user survey. (They felt that the information they were receiving was biased toward the negative and also unhelpful, so they discontinued use of the survey.) Note: this offers up a lesson on being specific about what kind of feedback you solicit, and not necessarily a warning against soliciting feedback in this way UCSF has implemented a pop up survey on UCSF Profiles that while annoying to many, has resulted in the collection of substantive feedback such as:

“Great resource for finding potential research collaborators and for PhD dissertation committees” “I am hoping that it just helped me find a mentor...” “It has helped find new nursing research problems”

Interventional networking: Given the significant lag time between identifying a potential collaborator and the first outcome of a successful collaboration, such as a published paper or even a funded grant award, it is difficult to directly link RNS usage to interventional networking. Additional studies need to be done to link short-term RNS usage patterns to longer-term outcome measures such as: • Number of new introductions/meetings/active collaborations • Interdisciplinary nature of new introductions/meetings/collaborators • Resultant joint grant applications and publications

Enable & track collaboration right from your RNS

9

R15 Enable collaboration from within your RNS

Identify potential collaborations and enable communication between researchers directly from your RNS.

The Michigan Corporate Relations Network (MCRN) portal functions like a multi-university Experts site that allows for searching across participating universities in Michigan (U of MI, University of Michigan’s Dearborn Campus, Michigan State University, Wayne State University at launch, and Western Michigan University shortly thereafter, and likely also Michigan Tech sometime after that). Thinking of it like an ecommerce metaphor, the site allows a businessperson interested in working with state of Michigan researchers to add researchers of interest to a list (‘shopping cart’) and the call to action (‘checkout’) process routes the researchers of interest along with a user-entered problem statement to the appropriate university business relations offices. These will be tracked centrally and by each office, hopefully with some evaluation of quantity of contacts and ‘quality’ of contacts (was it a legitimate research collaboration interest or a solicitor trying to sell something? did it turn into a collaboration?) UCSF has an operational pilot that has integrated UCSF Profiles with Salesforce Chatter (Chatter is UCSF’s enterprise social networking tool). This allows one to “follow” a person directly from UCSF Profiles and also allows a person to select a few people of interest and create a “Chatter Group” directly from UCSF Profiles.

10

Appendix 1 Notes / community comments on analytic tools From various sources, including: https://wiki.duraspace.org/display/VIVO/VIVO+Web+Analytics

Analytic Tool Comments Google Analytics Used as default Profiles solution

Also used as default VIVO solution – see googleAnalytics.ftl FreeMarker template Woopra "all-in-one web application performance tool"

Paul Albert at Weill notes: "I've personally used Woopra before. It seems to be better than Google Analytics about tracking outbound links and for watching people click through your site in real time. (Paul Albert, WCMC)

New Relic:

For performance monitoring, we use New Relic, which you can get a sense of here: https://dl.dropboxusercontent.com/u/2014679/NewRelic.png and https://dl.dropboxusercontent.com/u/2014679/EmailAlert.png. The best part about New Relic is that it will send a notice to other team members when you have tried to execute several malformed SPARQL queries in a row." (Paul Albert, WCMC)

Site AI:

"The problem with dashboards is they don’t directly provide insights or deliver knowledge about the data. Even worse, most visualizations require the user go through the mental exercise of interpreting the results. Site Ai does the analysis for you and presents the information in plain English." example: http://www.siteai.com/share/2672/2013/5/22/fcsjiggewdpechaafwpfjedlkvjhvipu (Mark Fallu, research-hub.griffith.edu.au at Griffith University)

Mixpanel:

"Page view counts are popular because they are easy to report, but ultimately cannot tell you how engaged your visitors are. Mixpanel lets you measure what customers do in your app by reporting actions, not page views." (Alex Viggio, University of Colorado Boulder)

Chartbeat See demo CrazyEgg CrazyEgg provides data on what links users click on once they arrive on key pages or selected profiles. Click statistics are provided in

several formats: heatmap, scrollmap, overlay, and ‘confetti.’ This allows us to improve design of page elements strategically to encourage interaction. (Leslie Yuan, UCSF)

Qualaroo User feedback is collected via a popup survey tool; detailed reports are available online. We ask “How has UCSF Profiles helped you?” to solicit success stories and better understand use cases, but the survey functions as a customer service access point as well. Users may enter an email address if they’d like a response. (Leslie Yuan, UCSF)