web surveillance for SME

Democratising Big Brother: Web surveillance for the SME

Gil Scott-Heron once stated that the “next revolution will not be televised”, he was correct. The next revolution is not being televised, it is being tweeted about on social media.  Social media is the causal agent that has created a new attitude in sharing information to anyone and everyone. The default was that humans on mass kept their opinions to themselves, whereas today people share equally deeply personal secrets and the banal to total strangers. Individuals are not the only entities that overshare on social media: companies, government agencies and public entities have got in on the act. Politicians, as well as Russian Bots, have used social media to drive news cycles. Social media can destroy businesses, as well as make careers. In the grey world of business, the frequently asked question is “How can information in social media assist my business?”.  And the response is “in many different and interesting ways ”.  

The information available on social media has all the self-appointed web experts in a tizzy because it allows entities with access to social media information to instantly sample the emotion and feelings of millions of people around the world. Emotion drives the economy where Keynes’ “animal spirits” haunt the financial markets. Rapid sampling of the market’s emotion can predict the future direction of the stock market. Outside of the financial markets, social media sentiment can predict the demand for products.  Governments have also had a vested interest in using social media because it can be used to predict crime, civil unrest and revolutions. Dr Ihsan Ullah from the National University of Ireland Galway, for example, in collaboration with IBM has developed a system that can infer if an organisation has suffered a data breach from its Twitter feed.

How does it work?

Inferring the national mood from social media about a product, or event, inevitably relies upon sentiment, or more recently emotion analysis of relevant posts. Sentiment is simply the classification at the sentence, document or topic level of social media posts into three categories: negative, positive or neutral. Older techniques use keywords, but this approach ignores context. And on social media, this is a grave error. For example, the Tweet below would be classified as Negative using older techniques. The latest techniques take into consideration context, as well as word relationships within the domain.

Negative tweet for web surveillance

Emotion analysis is a finer-grained form of sentiment analysis where an emotion is assigned to negative or positive sentiment. NC State University has supplied a tool through which emotion analysis can be conducted about specific topics. The following image demonstrates the current sentiment about Brexit.

emotion analysis for web surveillance for SME


The tool also demonstrates the words associated with each topic within the Brexit domain as well as their emotional value.

emotion analysis example for SME web surveillance tools

Analysis of social media is not limited to emotion detection.  Information aggregation, for example, can be used in various tasks such as food price prediction, and tracking of animal disease outbreaks. Information aggregation techniques locate target information and in some cases the location of the social media post. From the aggregated information a trend is inferred.

The final technique that will be covered is the identification of causal relations between words or phrases in social media posts, and events in the real world. The causal effect of words and phrases are aligned with real-world events through statistical techniques such as Granger Causation. Once the relationship has been established future events can be predicted from social media posts.  There have been studies with event prediction from social media that have predicted political revolutions.

What techniques are available?

Commercial vendors tend to offer social media monitoring or listening tools. Monitoring is: “ Caring for your customers by monitoring social media for messages directly related to your brand “  whereas listening is: “   Understanding your audience and improving campaign strategy by accessing the full spectrum of conversation around your industry, brand, and any topics relevant to your brand “. Typically monitoring is part of a larger general social media listening strategy. 

Social media monitoring tools will use a number of the aforementioned techniques to achieve their stated goal.  An example of the information returned by a monitoring strategy is demonstrated by the following Tweet that has mentions of IBM, and Red Hat. They have a relationship because IBM has acquired Red Hat. A social listening strategy would alert a user about the deal, and they may take action such as buying or selling the stock of IBM.

example of a news where to extract data for SME web surveillance


As demonstrated in earlier on that the current uses of social media are not the most imaginative. The data and the reach that social media has, can enable more imaginative, and consequently more lucrative, information strategies.  This can be achieved by adapting the research that is currently in the academic literature.

Contact Deeper Insights™ to see what state of the art web monitoring can do for your business.

Blogs that might interest you: