Measuring online sentiment is measuring the wrong ROI

10 Comments 20 December 2010

image courtesy of tony newell

I’ve been exploring paid social media monitoring systems as part of my role at FirstGiving. Initially, I was pretty excited about being able to monitor social media sentiment. Measuring aggregate sentiment over time seemed obvious to me. However, it became clear that none of the tools monitored sentiment well. In fact, they were all terrible at returning real online user sentiment. Neutral tweets were marked as negative or positive, positive tweets were marked as negative, and so forth. After researching sentiment analysis, I was not surprised to read that automated sentiment analysis is slightly less accurate than flipping a coin to determine whether or not brand mentions are positive or negative. (I tried to tweak the returns by manually changing the sentiments to influence the automatic marking, but that had no effect on accuracy.) I had been thinking that measuring sentiment analysis was the key to getting real ROI. I was completely wrong.

Measuring social media sentiment is the measuring the wrong ROI.

Sentiment analysis and measurement at companies is often driven by the need to prove the ROI (return on investment) of social media. Pressured to prove the value of social media involvement, the person in charge of social media will be tempted to prove his/her value by highlighting the percent change in positive online mentions. I would argue that measuring the change in sentiment is the wrong ROI to measure.

The right ROI measurement should be increased engagement and interactions.

Positive brand sentiment is nice. Does it move people to action? US internet users are likely to share information about your company and proactively recommend that someone make a purchase online, so engaging with stakeholders online supports that behavior. You really want to engage in such a way as to add value and create brand loyalty. Taking action as a result of social media is the ultimate ROI.

With the right engagement activity, loyal clients and consumers should naturally take the actions that you want them to take. Begin by measuring the level of online engagement with your organization, and make your plan based on that information. If your activists/clients/consumers are not talking to you at all online, that’s the engagement equivalent of negative sentiment. If they mention you in passing, that’s the engagement equivalent of neutral sentiment.

Let’s turn the ideal ROI of increased positive online sentiment on its head, and replace it with measuring increased positive online engagement with the organization. I wrote a similar statement in the comments of Shel Israel’s wonderful blog post describing his mixed emotions about sentiment analysis. Tal Wolgroch replied in the comments that “sentiment analysis is not about traditional ROI anymore. Those who try to measure it this way are playing a new game with an old rulebook.”

Stop trying to measure sentiment. Worry about your lack of engagement.

« Previous

Anonymous

Debra:

Interesting thoughts.

Yes, some of the sentiment tools are not accurate right now. I think they’ll get better with time.

Here’s a question your post raised in my mind: Does sentiment have any impact on ROI? I think it does. If I see a lot negative sentiment about a product or service, it drives me not to use it. And vice-versa. If people are talking favorably about something, I want to know more.

I think another question that you have subliminally asked is “what is real ROI for social media? That is a much more complicated answer. I believe the response is different depending upon the organization and their goals.

Example: If I am a conference organizer and people are at my event, I want to see the sentiment of what they are saying in real time. If they leave negative online comments, I need to take action and make some changes immediately to the conference experience. Sure it comes back to engagement. However, at most of the events I attended in 2010, someone, somewhere in the audience was posting their thoughts about the conference. It was a given.

And with so many of us carrying our friends in our pockets, it is only natural that some are going to share their feelings with others.

[Reply]
Debra Askanase Reply:
December 21st, 2010 at 5:41 pm
Jeff – you make some great points. I certainly had anticipated the argument about sentiment. I think there is value to watching sentiment trending over time and sentiment spikes. It helps organizations to stay on top of a potential reputation issue (if negative) or news item (if positive). If you are organizing a conference, your ROI could be registrations, online participation, membership renewals, etc. Negative sentiment can and will affect the ROI you are trying to achieve (like the example you gave). It’s good to pay attention to it during the conference, even important. I would also suggest that these spikes and trends can be offset (if they are negative) by engagement and building relationships with people who will defend the organization.

Your second question is so important: “What is the real ROI for social media?” I really think it comes down to moving people to action, whatever action that would be for your organization. Engagement is the key to making this happen, I believe. However, the other issue is that we can’t engage with everyone, right? I do think that general sentiment monitoring can play a role (monitored on a macro, not micro level). I think obsessing over every tweet is what will kill an organization. Perhaps obsessing over how to create loyalty is a better use of time?

[Reply]
Debra Askanase Reply:
December 21st, 2010 at 10:43 pm
Jeff – you make some great points. I certainly had anticipated the argument about sentiment. I think there is value to watching sentiment trending over time and sentiment spikes. It helps organizations to stay on top of a potential reputation issue (if negative) or news item (if positive). If you are organizing a conference, your ROI could be registrations, online participation, membership renewals, etc. Negative sentiment can and will affect the ROI you are trying to achieve (like the example you gave). It’s good to pay attention to it during the conference, even important. I would also suggest that these spikes and trends can be offset (if they are negative) by engagement and building relationships with people who will defend the organization.

Your second question is so important: “What is the real ROI for social media?” I really think it comes down to moving people to action, whatever action that would be for your organization. Engagement is the key to making this happen, I believe. However, the other issue is that we can’t engage with everyone, right? I do think that general sentiment monitoring can play a role (monitored on a macro, not micro level). I think obsessing over every tweet is what will kill an organization. Perhaps obsessing over how to create loyalty is a better use of time?

[Reply]
Anonymous Reply:
December 27th, 2010 at 5:52 pm
Jeff,

I just wanted to bring up that in the example that you use is qualitative data. Really at that point social media is becoming a tool for a support ticket.

I always imagine sentiment analysis and rating to be a quantitive score that is not the best measurement device for things that immediately impact constituents. I would compare it to a netpromoter score.

My $.02.

[Reply]
Anonymous Reply:
December 27th, 2010 at 5:59 pm
True. Here’s where qualative data changes into quantative data: data originally obtained as qualitative information about individual items may give rise to quantitative data if they are summarised by means of counts.

For events, if 60% of the attendees have the same sentiment, it is important to know and changes need to be made accordingly.

At least that’s how I see it.

[Reply]
40deuce

Hi Debra,

It’s true that sentiment analysis is not perfected right now, but I think some of us are doing a fairly good job at being able to help people like yourself gain a high level overview. One thing I always tell people is that sentiment analysis can help, but like everything else that has become automated, it’s always best to check over things with a real human brain.

To your post though, do you really think that sentiment should be taken out of the mythical social media ROI question? While it may not be everything, it must hold some sort of weight. Engagement is great, but it’s not always going to be the thing that matters most to people. If you’re engaging but people are still saying negative things about you, that has to count for something, doesn’t it?

I think ROI should be measured by a few different things that are respective to goals of being in social media. People shouldn’t place all their weight against one metric like just number of followers or sentiment.

Cheers,
Sheldon, community manager for Sysomos (http://sysomos.com)

[Reply]
ANSHUL GUPTA

Fully agree..

[Reply]
Melinda Lewis

For me, the clearest example of how sentiment misses the mark is in public policy which, admittedly, is where my focus usually is :). There are so many examples, though, of where policy is out of line with public sentiment, and, often, lack of engagement of part of the constituency is the culprit. The recent vote in the U.S. Congress on the DREAM Act is an example: polls show greater than 70% support for DREAM, which is nearly unanimous in today’s polarized climate, but that doesn’t translate at all into activism on immigration issues, online or offline. Too often, we advocates think that we’ve won because people say that they like what we’re saying, when the real question is: will you say it, too?

[Reply]
Pingback: How: Measuring Sentiment « Katy Leuschner's Blog()
Pingback: Acerca del Retorno de la Inversion (ROI) en las Redes Sociales – LEC Temporal()