A Consumer Insights Revolution: AI and Unstructured Data

Artificial Intelligence (AI) for Consumer Insights is developing rapidly, however, there is still much more to discover. In fact, most businesses are still in the infancy of understanding and implementing Artificial Intelligence to understand their customers, competitors and their market. 

AI has the capacity to revolutionize the way we gather product and consumer insights. It’s predicted that investments in AI technologies this year will have a huge impact on the development of strong insights for analytics teams to inform their strategies. 

This revolution is needed, especially in industries such as retail that have experienced a huge decline in 2019 with more stores closing than opening over the last 12 months. 

When done right, insights can empower the company throughout, not just a few teams or specialists. Research by Capgemini shows how AI is driving benefits across the organisation:


75% of organisations implementing Artificial Intelligence increase sales of their products or services by more than 10%.


78% of organisations implementing Artificial Intelligence said that their operational efficiency is increased by more than 10%.


75% of organisations stated that using Artificial Intelligence enhanced customer satisfaction by more than 10%.


79% of organisations implementing Artificial Intelligence generate new insights and better analysis.

Consumer Insights & Unstructured Data

Consumer insights are generated and obtained through a variety of methods with social listening being the most common. Beyond this, businesses are leveraging the power of unstructured data analytics to track consumer behaviour such as buying patterns or finding whitespace in a crowded product category. 

Unstructured web data, such as reviews, comments or forums, can truly capture the view of your customers. It helps to reinforce business decisions by giving brands a deeper understanding of customer behaviours and hidden opportunities, in a passive and unbiased way if compared with customer surveys or panels. When it comes to customers buying decisions it shouldn’t be a guessing game. Brands can now use AI to listen to customers real concerns and identify opportunities to innovate products, meet customer needs, find whitespace and more. 

In addition, unstructured data can reveal needs and wants from your customers that are hidden and not explicitly asked for. In other words, at times customers aren’t sure of what they are looking for, or they don’t have a clear idea of what is needed for your product or service to be complete. The creativity and capacity of your product teams to find a solution here is essential and with AI-powered insights its possible to identify these trends using Topic Analysis and Trend Monitoring. Uncovering the patterns in consumer feedback and highlighting common trends earlier could have saved Unilever losing a 1.5% market share in the zero-sugar icecream market to Halo Top.

Unstructured data has hidden answers to potentially unknown questions. The challenge for market researchers is in finding the right questions to ask, and using the right tools for the job.

So, what changes will we see in 2020?

In 2020, with AI and unstructured data at play, the category will focus on two main areas: Customer Listening and Customer Engagement.

Customer Listening

From LinkedIn to Facebook, 2019 brand’s most wanted tech was social listening; listening for clues from social media tweets, likes and comments about consumers desires, changing needs and complaints. However, more valuable insights could be gained if brands had the capability to fish out the right insights from this sea of big data, using AI to automatically identify the trends, make the connections between ranters and respecters, from what’s true and what’s not, and joining the dots.

2019 saw the introduction of new analytics products from Pulsar, Relative Insights and Signals-Analytics. But this is just the beginning of what can be done beyond Social Listening.

By “going beyond” social listening to customer listening, it extends the ability to understand customers’ sentiment. Achieved through passive listening on e-commerce comments pages, review sites and forums. To analyse complaints within your organisation and match that to complaints outside on Social Media. To monitor performance pre and post a product launch to see what’s working and what’s not.

Brands thinking bigger than social media, will be able to get hidden, but essential, insights from review platforms, chat applications, product/service category websites and other sources. Insights gathered and presented on demand, are used to craft personalised and targeted offers, products, services and specific features, all one step ahead of your competitors.

The opportunity to innovate with unstructured data from these other sources, and the use of Artificial Intelligence to find patterns within the data, is what will drive innovation for Consumer Insights in 2020. We’re calling it Smart Insights.

Customer Engagement

Focus groups, surveys and in-product messaging are tools and approaches to gain insights from customers that are extremely challenging when looking for deep customer engagement. Surveys are easy to implement but they offer a one-way communication and require a long time for the brand to develop and for customers to complete. The same happens in focus groups, that take extensive time and resources for teams to implement. 

A brand using AI platforms are able to analyse and understand text (and therefore parts of the customer journey) in real-time. For example, complaints emails, support tickets and chats sent to customer service can help brands to understand the overall sentiment towards their product or service. All useful consumer insights to help create a smarter product roadmap. 

We’re all used to product and service personalisation, from Netflix to Spotify with their personalised content recommendations, but what will give brands an edge this year, is the ability to propose personalised features. Customers would more likely buy a product/service from a company that has taken time to get to know them and what they need.

Based on this idea, pricing methods will also change, for example, price or subscription packages won’t be appreciated as much as a customised package priced depending on the customers’ needs. This level of understanding of the customer needs will involve Machine Learning and AI solutions, to read and understand where a customer is in their buying journey. 

This kind of personalisation with Customer Engagement, can have a tremendous effect on sales, and also retention.

And more

Customer Listening and Customer Engagement aren’t the only two areas in which we will see a change made possible by AI-powered tools. Nowadays with so much data at our fingertips, analysts are becoming central to most business decisions, and therefore are overworked, threatening the quality of their work. AI-powered tools give a chance to those all-important teams to concentrate on the initiatives that really matter instead of burying themselves in report writing and meetings. Which is why we believe On-Demand and Live Insights will have a huge impact in 2020.

We’ve been developing AI models, insights platforms and Data Science services that cater to this growing need. Our AI-powered tool the Skim Engine™, helps brands to combine data from millions of different sources of unstructured data, both internal (customer service portals, emails, chat etc…) and external (forums, blogs, product pages or even social media) to create a live dataset for analysis. Our AI models then run an automatic analysis beautifully presented in our dashboards.

If you want to know more about AI-powered tools and platforms for Smart Insights, we can help!
Contact our team of experts for a free consultation. Alternatively, book a demo with us as the first step on your Smart Insights journey. Turn your data into insights and unlock real opportunities throughout your organisation.

We help companies like yours win with AI

From finance to healthcare, from market research to media monitoring, we can help your people make better decisions. We work alongside companies like yours to help deliver successful AI and ML projects - to make a real business value impact


The challenge: Deloitte’s partners and account managers found they were drowning in news from sales support teams, and unable to react quickly to market changes

The solution: Deeper Insights built a prototype Automated Insights app allowing them to have better conversations with clients and close more business

The outcome: Account managers at Deloitte close more business thanks to actionable insights delivered straight to their phones

Client said: "There are a number of gems we’ve found that are far better than the standard services we use" - Dimitar Milanov, Partner, Deloitte


The challenge: Help the sales and marketing teams know more about their customers to enable them to drive deeper customer engagement in sales meetings

The solution: Deeper Insights developed a CMS that scraped the web and automatically identified and summarised customer events relating to key accounts at JLL

The outcome: The automation of the whole previous manual process, and being able to identify 60% more news stories than the manual process, enabling JLL have better and more informed conversations with their clients

Client said: "We have lots of researchers and people who generate insights for our clients, Deeper Insights™ (formerly Skim Technologies) helped us improve the speed at which we get insights and have better conversations with our clients." - Chris Zissis, CIO, Jones Lang LaSalle


The challenge: In the UK, the total number of Total Knee Replacements(TKA's) per year has increased from 13,546 in 2003 to 98,147 in 2019 costing the NHS an estimated £585m per year. The average cost of a TKA in the UK is £12,000, however, post-surgical complications, e.g surgical site infection, increases this cost by between £1618 and £2398 per patient.

The solution: Our consortium, which is comprised of Smith&Nephew Ltd, Deeper Insights and Imperial College London, won Innovate-UK funding to carry out an ambitious and innovative project that is focussed on developing markerless and automated registration and tracking of the patient's limbs tailored for robotic-assisted orthopedic procedures using structured light technology assisted by deep learning to continuously capture the patient's anatomy during surgery.

This new platform will be integrated within S&N's commercially available robotic platform "NAVIO," which was previously supported by I-UK funding, and will obviate the need for percutaneous markers reducing set-up time, cost and complexity during surgery.

Talk to us to see if we can help you

Discuss your AI project with us and lets see if we can help. We can dive into the data you have, the data we can gather from the web and other data sources, how we can manipulate that data for you and how we can output it in a dashboard that your business can actually use.

Our Data Scientists will help you deliver your AI project

Our Data Science experts are recognised globally with over 500+ citations and patents

We have a combined experience of over 40 years in developing cutting edge, and innovative Artificial Intelligence from both academia and industry. We are specialists in Computational Linguistics, Natural Language Processing, Machine Learning, Deep Learning and Data Analytics


Dr Márcia Oliveira,

PhD Network Science


Dr Claudio Sa,

PhD Deep Learning


Dr Catarina Carvalho,

PhD Image Processing