Information Age: News, analysis & insight for IT & business leaders

In his book ‘The Shock of the Old’, David Edgerton observes that is not always the newest, shiniest technologies that cause the greatest upheaval.

This year’s UK general election is a case in point: the great innovation in media coverage has been televised debates between party leaders (first introduced in the US in 1960) and not, as many had predicted, the use of social media.

Nevertheless, social networks have provided an unprecedented insight into what voters are saying, and a number of pundits have used sentiment analysis technology on social media content to gauge public feeling.

This is a timely test for a technology that has a clear business application – just as political parties can track what people are saying about their policies and personalities, so too can businesses follow “the conversation” around their brands and products.

"We’re trying to treat the language used in microblogs as more or less its own language"
Niklas de Besche, Meltwater Buzz

But how reliable are social media sentiment analysis technologies? Can software really uncover the sentiment behind the context-dependent, self-referential and often highly-ironic language that is used on social networks?

Yesterday I spoke to Niklas de Besche, executive director at social media monitoring provider Meltwater Buzz, which has been tracking sentiment towards the UK’s political parties by analysing blog posts, online video submissions and ‘tweets’. The fruit of this analysis – as of 29th April 2010 – can be seen below.

De Besche explained how Meltwater Buzz has developed an algorithm that analyses social media content and rates it as positive, negative or neutral in sentiment. The company uses Amazon.com’s Mechnical Turk service, which allows customers to ‘crowdsource’ human operatives to perform small tasks, to train that algorithm. This training is an ongoing work in progress, de Besche explained, because the language of the social web is constantly evolving.

He admitted, though, that some forms of social media content are easier to assess than others. Blog posts, for example, can be analysed in detail. The short messages users share in such ‘microblogging’ services as Twitter are far harder to penetrate given their brevity, and analysing the sentiment of tweets has called for its own approach.

"We’re trying to treat the language used in microblogs as more or less its own language," de Besche said.

The company claims “confidently” that its service is 80% accurate. This claim is based on the fact that the algorithm returns a percentage assessment of sentiment (e.g. 65% positive or 42% negative), and only content that returns a percentage of over 20% is categorised as positive or negative; otherwise it is deemed neutral.

"This reduces our accuracy to 80%," said de Besche, "but in turn we are conservative and highly accurate on the posts that are either marked positive or negative."

This means, however, that when Meltwater Buzz categorises content as neutral, it could be that the algorithm just doesn’t “understand” the sentiment.

De Besche argued that this ambiguity comes out in the wash when large quantities of content is analysed. And indeed, only when a large quantity of content is available is automated sentiment analysis required.

Still, social media sentiment analysis remains an inexact science. I wonder whether the technology in its present form can really be used to discover underlying sentiment that would otherwise be invisible, or whether it simply provides seemingly-scientific corroboration for what marketers and policy-makers in tune with their audience already know.

De Besche, however, is positive that accuracy will improve as the technology develops, or in other words that there is no logical reason why software can’t one day understand the sentiment behind all human communication. “We are still at the very early days of what is possible.”