Measuring Digital Communities in Results-Driven Environments

Nelly Leonidis | @nellleo | Marketing

Managing online communities is a dream job for those of us who enjoy connecting people, technologies and businesses. Community managers have soft skills that help them connect with the members they’re looking after, and have analytical and strategic skills that allow them to connect a business’s goals to the community.

But measuring and reporting on the health and performance of digital communities isn’t a straight forward one-formula approach. I’ve observed (and fell into) a few traps when it comes to reporting (as accurately as one can). This presentation will go over the problems and the solutions that have worked.

Why is Indiana Jones here? Because I’m going to use him as my presentation’s tour guide. I believe community managers are adventurers by profession. We go to unchartered territories (hello, subReddits), create tribes, join existing cultures, learn to speak new languages, share memories with people we meet along the way, guide crowds to safety, occasionally come out with some scars, but mostly - every community is its own adventure. So using Indiana: totally legit in this case.

The snake pit of vanity metrics

Hello and welcome to the snake pit of Vanity Metrics. Gross, slimy, and hard to get out of without getting bit (usually by a “shiny-metrics”-obsessed senior exec). Vanity metrics = chasing the “likes”, followers, web visits, YouTube views and all the numbers that make you feel good about your work at the end of the day.But could mean absolutely nothing for your business.

If you have 10,000 comments on your post, but not one of those comments generated a single cent of revenue for your business, then what’s the point of the community? At the end of the day, you have a stated goal behind creating the community: increase revenue, generate brand impressions, public outreach, decrease costs, retain loyalty, and so on.

Trap #1: The snake pit of vanity metricsLooking at the number of followers/visits/”likes” and reporting it as an achieved result

Solution

Create a comprehensive measurement framework around your business goals for the community, including:

• direct, • indirect, and • complementary measures

If you report a vanity metric as a result, you’re misleading those you report to. This is an easy trap to fall into, and rather hard to get out of. Once managers get used to the big figures, they want more - with or without context.Your job will be to condition them to see the actions behind the numbers. An increase in the number of followers is great, but you should focus on what has changed, been delivered or goal met.

What works? A comprehensive measurement framework that takes into account 3 measures: direct, indirect, and complementary. Direct = measured changes in your call of action for the community. For example, the use of community-specific coupons for purchase, or the number of phone calls that come from the community (using a special # to help tracking, for example). Indirect = the interactions between your community pertaining to your call of action. Are more people talking about your coupon? Are they discussing your customer service through the telephone? What are they saying or communicating that can explain the change in your direct measures? Take that into account and provide snippets of information in your report.Complementary = the external circumstances that influence your direct and indirect measures. For example, a news story that talks about one of your ingredients perhaps. A conversation in the community follows, and sales increase/decrease. These are factor you can’t help, but you need to note and report on because they are likely to have a very direct impact on both direct and indirect measures.

Your report can help explain the “why” behind the vanity metrics by framing the results using your observations of direct, indirect and complementary measures. Always tie them back to your community’s goal(s).

The contracting walls of community life cycles

If your reports aren’t addressing the life cycles of your community, brand, or individual members, then you’re quickly working towards the demise of your community (and your job, possibly).

Life cycles here refers to three different cycles that dictate how your community should be performing and at what point.

Community life cycle: Did it just launch? Are you at the maturity level? Have you hit post-maturity segmentation? Brand life cycle: Is the product new? That means you’re in brand-recognition mode. Is the market saturated? Then you’re likely focusing on retention and loyalty.Members’ life cycle: Are they students graduating? Is this a parent-heavy community? Seasonal purchases?

A good way of framing the obstacle is to pretend that you’re the community manager for a car-manufacturing community. You’re dealing with three cycles all at once, and every one of these will have an effect on what you measure, when you measure it and what you need to take into account to explain and analyze your data.

If your report makes your community look like it’s underperforming because your goals aren’t reflecting the life cycle(s) you’re in right now, it will feel like the contracting walls are coming to squish you and your members out of that space!

Trap #2: The contracting walls of community life cyclesFailing to take into account natural community life-cycles when measuring and reporting

Solution

Set realistic expectations for your community’s performance based on: • your audience • industry trends • platform specifics

In order to avoid analyzing and reporting data that doesn’t accurately reflect the life cycle of your community, set realistic expectations for your community’ performance.Do this by understanding your audience, the industry and the platform specifics on which you run your community, and place the information within the context of the life cycles your community is likely going through.

Setting realistic expectations with respect to your audience means understanding their behaviour online, their tendencies, what particular life cycle they are going through and their networking habits. For example, a community for high school teachers will see a decline in participation during peak teaching periods in the day, and a slump during the summer time. Reporting on the goals achieved for the duration of measurement needs to note these shifts in the community. Additionally, understanding the next course of the life cycle will help the community manager set realistic goals of growth (or maintenance) and move towards achieving them at a pace most suited to their particular community.

Industry trends give the community manager markers that will help analyze the collected data for the measurement period and forecast (realistically) the growth/return potential of the community. Your can use information you collect in house about your industry, or rely on products like Gartner Reports for your market.

Finally, a community that runs on a developed platform (a Facebook page, for example), has a flatter learning curve for members. This means engagement and reach can happen much quicker. It also allows for you to target a large group of people at once, while channeling your energy on keeping them interested in your brand and community. But a community built on a specific platform (for example, a moderated community through your website), will have a steeper learning curve for joining members as they need to get used to your community’s functionalities. This means they’ll spend more time in the beginning learning than engaging, so your reports need to account for that extended period of learning.

The giant rolling boulder of externalities

External factors that you have absolutely no control over will be affecting the performance of your community.

These factors could be economic (growth, recession), industry-related (shift away from books to e-readers, for example), publicity (an unflattering news article about one of your product’s ingredients, for example), and so on.

Trap #3: The giant rolling boulder of externalitiesTreating your community as if it exists in a vacuum

Solution

Look from the inside out:• frequent snapshots of news in your industry• peer/competitor activity monitoring (not comparing!)• regular briefings with all teams

The best way to deal with something you cannot control is to try and see what you can do before it rolls onto your and your community.

Environmental scans for the win!Study your industry thoroughly; learn about the shifts (small or large) that are affecting you, your peers, and competitors. Keep an eye on the news; what’s happening in your community, what are the key issues affecting buying/purchasing/service sectors/not-for-profits and so on. The more you know about what’s coming, the more time you’ll have to adjust your community management strategy and provide context to your data findings.

Watch what your peers and competitors are doing to deal with the very same issues you cannot control (economy, industry, etc). Monitor, but do not compare. Each community’s performance is shaped by many variables that cannot be blindly applied across the board, so comparing your performance to anyone else’s is unfair. Monitoring however, will give you an idea of what seems to be working or not working in response to these externalities.

Share your concerns and insights with other members of your organization, and learn from their insights. For example, a decrease in consumer spending has an effect on your entire company’s performance. What are your marketing or operations teams doing to help the business stay profitable?

The quicksand of analysis paralysis

A very common problem with data-driven communities? Spending too much time collecting, analyzing and reporting on data - with long intervals of no action.

Whether or not you’re the one doing the data-collecting and analysis, you’re likely checking on insights regularly and trying to see how your community is performing in terms of your business goals. And between looking at your direct results, capturing indirect measures, keeping an eye out for industry shifts, noting life cycles, and so on - when do you put into action the recommendations following your report?

Trap #4: The quicksand of analysis paralysisWe’re good at getting data and analyzing it. Now what?

Solution

Practice agile analysis:• keep reporting cycles short• always be testing• tweak, tweak, tweak

Realizing that your digital community is a living breathing organism that acts, reacts, and overreacts to events unfolding every second online helps you escape form the quicksand of constant measuring, analyzing and reporting then acting too late on information.

Create a more adaptive pattern of deploying tweaks to your community management strategy. For example, check your data on Thursdays and note what has worked and what hasn’t over the last week. Note actions/behaviour that can be considered as a hit/miss. On the following Mondays, introduce adjustments to your strategy based on your metrics and what you’ve observed. Replicate the hits, and avoid what hasn’t worked. Seize on opportunities you spot immediately, because it means getting results in shorter periods of time and keeping at pace with your community.

You should also introduce a hypothesis that explains why some calls for actions are met and some aren’t, then change some elements in your community management that could prove or disprove your theory to help you earn what works (or doesn’t) in a shorter period of time. For example, you can move the time of posting from 10 a.m. to 1 p.m. to see if it makes a difference in the number of comments you get (or purchases, re-shares, etc). You can also experiment with tone, or content.

Continuously adapting your community management strategy to move faster on timely insights and capitalize on continuous “wins” will save you time and resources in the long run, because you are not looking to introduce sweeping changes to your strategy or spend more time analyzing quarterly results.

The myopia of relying on passive data

Indiana Jones gives his dad the opportunity to shoot down the enemy, and Jones Sr. ends up shooting their only means of escape down.

I’m always surprised by the number of businesses that do not tap into their most valuable resource to learn more about their performance and report more accurately.

Trap #5: The myopia of relying on passive dataForgetting to ask your community members what they want

Solution

Tap into your best data source - the community:• surveys (polls, short answers)• focus groups • “wants and needs” analysis • listening programs

Passive data in this sense means data that you’re collecting by-the-by. For example, exit pages through your web analytics dashboard.You don’t really know why people are leaving that particular page, but you can make theories: they dislike the content, the title is misleading, their smartphone batteries ran out, power outage in your particular area one night, etc. The easiest most effective thing to do would be to ask the community about your observations: “Hi everyone, what to you think of this page? Feedback is always appreciated!”.

The insights you get from your very own community are worth more than everything else you could passively collect, because you’re getting first hand accounts of what’s working and what doesn’t. This makes you understand your community’s behaviour better, and you’ll know why your calls for action are met (or not), and how big is the gap between your business goals and what your members thing they’re contributing to your community.

Use surveys: quick polls do well, and can be done frequently. Longer surveys (no more than 5 questions) on less frequent basis also work well.Introduce focus groups to your community: ask for a number of volunteers and interact with them about a particular idea, product, and so on. This is also an effective way to turn your new members into regulars through direct dialogue, and move them to brand champions based on the new interpersonal relation you have with them.Through a “wants and needs” analysis, you’ll be able to understand what your community members think they’re getting our of your community, and how it aligns (or not at all) with your business goals. This could be the making or breaking of your strategy; if there is a big gap between what members are doing and what you’d like them to do, this is an excellent way to start bridging the gap and introducing new content/features/conversations/offerings that will help you deliver on your goals.Listening programs refer to the way you interact with your members based on what they say. For example, if someone comments on your post, how quickly do you respond? How thoroughly? How honestly? If you have the ability to get in on the conversations (steer them or just join), you’ll see more members adding the insights that your business could benefit from.

Thing about the worst thing that would happen if you began tapping into your community directly for insights. Nothing. You would receive nothing, and the risk to you is nothing.The best thing that would happen? New insights, business ideas, improved analysis, better reports, more realistic goals for your growth, and in the end a happier community manager!

With respect to measuring digital communities:

“X never, ever marks the spot.”

It’s very important to know that while these traps are common measurement and reporting traps that I’ve come across (personally and in conversation with other community managers) over the last few years, there is no fixed formula to effectively measure a community’s performance.

Measuring and reporting on your digital community - as you know - is a combination of different measures and insights - all of which apply only to your community at a particular period of time.

There is no “holy grail” of measuring a digital community, but there are some ways to avoid providing unhelpful or widely inaccurate reports on the health of your community.

Bonus: lessons learned in the trenches

1. Educate: horizontally and vertically

2. Clear presentation of findings = 90% of the battle

3. Always, ALWAYS, know the business value of your community

4. Share ALL the things with ALL the people!

Here are a few lessons that will save you time, blood, sweat and tears:1. Educate those around you about what you do, your community, how to understand the results of your work, and the effect it has on business. Educate vertically: upwards to your managers - so they stop bugging you for vanity metrics, and downwards to those doing the research for you - so they see the bigger picture and how it all helps you manage a better community online. Take the time to educate your peers as well. You’re all working towards the same corporate goals, why not provide them with a little more insight? You can do this by holding lunch and learn sessions (for example, to share cool trends emerging in your industry that have been validated through your community, or just to explain how you engage with people online and what they’re saying about your brand). You can also write case studies about what works for your brand in the communities, what doesn’t, how do you overcome the difficulties, and what are the lessons learned that can be of benefit to your organization.

2. Present your reports in a clear and jargon-free manner. Use metaphors, and use examples. Have an executive summary that explains what you’ve tried to do for the time period measured, what you observed, highlights, and recommendations. You can then provide more in-depth visuals on quantifiable growth, include screenshots of conversations and interesting information shared, share the lessons learned, and list the recommendations for the upcoming period. Add an appendix with your metrics dashboard data if you wish, industry scans, associated reports, or details on your survey or focus group results. This way, you have consumable information for any type of reader regardless of their background and how much or little they know about digital communities and measurement. The executive summary helps your managers look through the report quickly and decide how much or little detail they want to go into. The appendix makes the data-happy people ... happy.

3. Never walk into a meeting without having a few “media lines” ready about your work and the results it recently achieved. As a community manager, your job is already misunderstood by most of the people you’ll meet. “So you sit on Facebook all day long and chat” is one of the most frequent reactions we hear after introducing ourselves as community managers. The best way to prove your (and your community’s) worth immediately? Have shareable insights ready. For example, “In the last 6 months, our community was responsible for 15% of the growth of our brand awareness, which generated 2% sales in the past quarter”. No one is going to ask you to justify your work or your community’s existence after that. And you get to defend your members to any people that doubt their overall contribution to your business’ success or growth.

4. No point keeping useful information an insights from your own company. Share your reports with interested groups, add useful information to your emails that could be of help to product developers or public relations staff, and so on. In my experience, I’ve seen many fantastic ideas about content come from the least expected sources: people who weren’t at all part of my division or discipline. Bring your community’s insights to your organization, and help them understand how their work impacts those you mean to do business with.

tl;drTrap #1:The snake pit of vanity metrics

Solution: Tap into your community; ask and listen

Solution:Comprehensive measurement framework

Solution:Set realistic expectations

Solution:Use insights from environmental scans

Solution:Practice agile analysis

Trap #2:The contracting walls of community life cycles

Trap #3:The giant rolling boulder of externalities

Trap #4:The quicksand of analysis paralysis

Trap #5: The myopia of relying on passive data

A quick recap slide of the 5 traps and solutions to help you recollect the last few minutes.

Images from:www.indianajones.com / LucasFilm Ltd.

Source of images.

Questions? Love? (psst... @reachology / @nellleo)

Thank you!