AI-driven Competitive Intelligence

These days businesses have great visibility internally, into their sales, marketing, product metrics and strategies, but lack visibility externally.

To outdo competition businesses should implement Competitive Intelligence services that convert data into intelligence that can provide a real-time view of your competitive landscape. We’ve found great value in machine algorithms that can help identify trends and insights in vast streams of data and make faster decisions that potentially position businesses to be competitive in real-time or even ahead of time.

By competitive intelligence, we mean the company’s efforts to gather and analyze information about its industry, business environment, competitors, and competitive products and services. The gathering of the information and the analysis will support a company’s strategy as well as identify competitive gaps.

In this blog, we talk about three main areas of competitive intelligence in which we believe there is great value in applying AI. We will talk about the best competitive intelligence sources and the role of web scraper agents.
Although AI-driven competitive intelligence provides a greater competitive advantage, there are limitations in the web scrapers capabilities that we fixed by building our own agent; the Skim Engine™. More on web scrapers will be explained later in the blog.

Unstructured & external data

Firstly, the best thing to do is to systematically find the best competitive intelligence sources, based on what your goals are, to create competitors profiles. Also, processing real-time data and up-to-date data is crucial to gain accurate insights.

Competitive Intelligence from online, external sources include::

  • Company websites for insights into target audience, shifts in strategy, pricing product, and so on.
  • Company press releases for new product, personnel changes, or expansion news.
  • Social media postings, especially if the company is introducing a new product or service that hasn’t yet been launched.
  • Job postings, types and number of positions available indicating a potential new category development or a new product.

AI-powered tools are well positioned to harvest data from millions of webpages to surface valuable business insights. These tools are able to track a competitor’s complete digital footprint, both on and off their website. The most critical updates are those that aren’t announced in a press release but rather hidden in websites or in a customer review. It’s down to the web scrapers ability to provide high-quality data by unearthing these gems.

Web scraping agents are already popular and used to access unstructured data sets from the web. Once the data is collected, structured and organised, an AI product can automatically categorize and prioritize the data to enable users to review it. As scrapers are able to extract a vast amount of intelligence data, it’s critical to surface the highest priority updates and act on them before its too late.

Competitive Intelligence

We’ve identified three areas of competitive intelligence where we can harness the power of external unstructured data:

1. Monitor job roles

A company can monitor news, competitors’ pages and social media for trends of staff from the interested employment category (i.e., Chief Executives, Data Experts, AI developers), joining or leaving or moving to and from competitor companies. This would provide valuable intelligence by signalling that a competitor may be in trouble, takeover, moving into a different area or implementing new technologies.

2. Monitor price changes

In a fast-changing online market, businesses are aiming to remain competitive and certainly with e-commerce customers are often price driven. Using price data collected from millions of e-commerce sites across the market you can gather competitor price trend data 24/7.

Price Monitoring could also alert you when your competitors prices change, it can also tell you a lot about your competitors pricing strategy (i.e., Christmas discounts, sales, etc). Through additional sentiment analysis towards your competitors products, you can also track how it affects price over time. It will put you in control by learning from your competitors mistakes and wins.

3. Monitor Competitors Product or Brand Hype

Using the latest Narrative Extraction techniques using Natural Language Processing, you can monitor specific underlying social processes that indicate hype surrounding a brand or product through the narrative within social and news data. This will provide you with a clear indication of when a product or brand is beginning to pick up virality or buying trends, so you can counter the movement with your own product launches or announcements.

Web scrapers limitations and Deeper Insights™ solutions

Rule-based web scrapers are limited on their ability to work independently. Web scraping tools are usually made to handle simple websites if they use dynamic elements a scraper tool might not be able to extract data. Not only, as websites are updated quite frequently, but many of these changes can also render the scraper tool useless. In the latter case, you would lose the data and will be forced to update the tool to make it work with the new changes on the target pages, very onerous.

For example, if the goal is to extract prices from a website, the scraper would be trained to find that information in a certain position of the site. If this website changes the scraper won’t be able to get the information you’re looking for and/or it will provide you with incorrect data.

We are constantly updating and adding new features to our own web scraper agent; the Skim Engine™. The Skim Engine™ is a rules-free agent that extracts and organises unstructured data in a machine-readable way, so changes to websites are a thing of the past, and you can gather reliable, accurate competitor data with ease.

If you would like to know more about what the Skim Engine™ can do and would like to hear more on AI-driven Competitive Intelligence let us know! One of our data experts will contact you as soon as possible.

We help companies like yours win with AI

From finance to healthcare, from market research to media monitoring, we can help your people make better decisions. We work alongside companies like yours to help deliver successful AI and ML projects - to make a real business value impact


The challenge: Deloitte’s partners and account managers found they were drowning in news from sales support teams, and unable to react quickly to market changes

The solution: Deeper Insights built a prototype Automated Insights app allowing them to have better conversations with clients and close more business

The outcome: Account managers at Deloitte close more business thanks to actionable insights delivered straight to their phones

Client said: "There are a number of gems we’ve found that are far better than the standard services we use" - Dimitar Milanov, Partner, Deloitte


The challenge: Help the sales and marketing teams know more about their customers to enable them to drive deeper customer engagement in sales meetings

The solution: Deeper Insights developed a CMS that scraped the web and automatically identified and summarised customer events relating to key accounts at JLL

The outcome: The automation of the whole previous manual process, and being able to identify 60% more news stories than the manual process, enabling JLL have better and more informed conversations with their clients

Client said: "We have lots of researchers and people who generate insights for our clients, Deeper Insights™ (formerly Skim Technologies) helped us improve the speed at which we get insights and have better conversations with our clients." - Chris Zissis, CIO, Jones Lang LaSalle


The challenge: In the UK, the total number of Total Knee Replacements(TKA's) per year has increased from 13,546 in 2003 to 98,147 in 2019 costing the NHS an estimated £585m per year. The average cost of a TKA in the UK is £12,000, however, post-surgical complications, e.g surgical site infection, increases this cost by between £1618 and £2398 per patient.

The solution: Our consortium, which is comprised of Smith&Nephew Ltd, Deeper Insights and Imperial College London, won Innovate-UK funding to carry out an ambitious and innovative project that is focussed on developing markerless and automated registration and tracking of the patient's limbs tailored for robotic-assisted orthopedic procedures using structured light technology assisted by deep learning to continuously capture the patient's anatomy during surgery.

This new platform will be integrated within S&N's commercially available robotic platform "NAVIO," which was previously supported by I-UK funding, and will obviate the need for percutaneous markers reducing set-up time, cost and complexity during surgery.

Talk to us to see if we can help you

Discuss your AI project with us and lets see if we can help. We can dive into the data you have, the data we can gather from the web and other data sources, how we can manipulate that data for you and how we can output it in a dashboard that your business can actually use.

Our Data Scientists will help you deliver your AI project

Our Data Science experts are recognised globally with over 500+ citations and patents

We have a combined experience of over 40 years in developing cutting edge, and innovative Artificial Intelligence from both academia and industry. We are specialists in Computational Linguistics, Natural Language Processing, Machine Learning, Deep Learning and Data Analytics


Dr Márcia Oliveira,

PhD Network Science


Dr Claudio Sa,

PhD Deep Learning


Dr Catarina Carvalho,

PhD Image Processing