How is AI used in recommendation systems?

Overview

When you browse different videos on YouTube, do you see, after a while YouTube showing content that matches those you had viewed earlier? For example, you’ve been viewing TedX videos on the effects of AI on the job markets. After a while, YouTube might show various videos related to the impact of AI on jobs. Without you knowing, YouTube’s AI-powered recommendation system had been at work while you browsed the videos. What’s going on? AI accepts a massive volume of data based on browsing behavior, preferences, and browsing history and tries to make sense of it. It finds patterns and similarities in the data and based on that, gives you what you might like. Recommendation systems are being used everywhere to offer a better experience to the user or customer. Every big organization has been using that. Of course, its critics point out that individual privacy and confidentiality are being sacrificed.

What is a recommendation system?

A recommendation system, simply speaking, is an AI software that recommends products and services to the user of the product or services based on the preferences and choices of the user. How does it do what it does? Well, as you, the user of the software, use it over some time, the AI software consumes a lot of data about your browsing habits, preferences, products or services browsed, and browsing history. Then, it creates patterns or habits out of those data and when you next visit the product or service, it recommends offerings based on its learnings. For example, when you view many movies on exorcism on Netflix, it might recommend movies of a similar genre to you. Recommendation systems are being used by Amazon, Netflix, Spotify, Pandora, Apple Music, and many other organizations are using recommendation systems.

The next few sections will discuss how a few reputable organizations have been using recommendation systems.

How does Netflix’s recommendation system?work?

Netflix’s recommendation system is known as the Netflix recommendation algorithm or Netflix algorithm. While all recommendation systems have their unique ways of operating because of the nature of the business, there are certain common points. The following sections describe how it works.

Collaborative filtering

It has been one of the most effective methods that are employed in recommendation systems. It analyses the behaviour of the subscribers and finds patterns and similarities in their choices. For example, if many users have watched a particular action movie many times or have watched a TV show, the recommendation system might recommend the same show to you. Collaborative filtering is based on the concept that people that exhibited similar behavior or preferences in the past are more likely to show similar behavior or preferences in the future.

Content-based filtering

Let’s understand the content-based filtering system with an example. You have watched Narcos and the Narcos Mexico web series. What does the Netflix recommendation system do here? It takes and lists the attributes of each item in the system (such as the genre, director, cast, plot, and so on) and matches those with the attributes of other web series. Depending on how well the attributes match, the recommendation system will offer similar shows to you. For example, if the attributes match those of El Chapo, Netflix shows that as a recommendation to you.

Deep learning

How does the recommendation engine know that if you had watched Narcos?—?a drug cartel web series for the uninitiated?—?then El Chapo could be another web series that you might like? To know that, it uses deep learning techniques. It’s a deep and advanced technique that uses artificial neural networks to study all kinds of data related to movies, web series, and all other offerings from Netflix. Based on the findings from its study, it matches the same with the user data such as preferences, historical views, patterns, and behaviors and provides recommendations.

Reinforcement learning

Reinforcement learning is the same as providing incentives to human beings so they’re motivated to perform better. An employer designs various incentives to motivate employees to perform better. Similarly, the recommendation system also incentivizes its systems to recommend better offerings to the subscribers. For example, if a user watches a recommended movie and likes it, the system is rewarded.

Concerns around the recommendation systems

Since the recommendation systems are based on the premise of collecting user data, questions on confidentiality and legality are being raised, and rightly. For example, Google has been collecting your data, most of which is personal, for a long time. There are valid concerns and protests around how Google has been using the data. The critics say that behind the legalese, privacy, and confidentiality are being sacrificed. Broadly, the concerns can be described under the following:

• How do the organizations ensure that the data collected are not being put into use other than the stated purposes? Who is the watchdog and how are the actions of the organizations around data monitored?
• What are the penalties for the wrongful use of data? Is there any precedence that an organization has been penalized?
• How do the organizations ensure that the large volumes of data they’re collecting are not falling into wrong hands? What measures have they taken?
• How do the organizations ensure that the recommendations are not biased or discriminatory? For example, the algorithms must be extremely sensitive toward recommendations on racial violence, movies, and discrimination.

Security concerns notwithstanding, the recommendation systems are here to stay and flourish. They’re an integral part of the business strategies of the organizations because given the huge volumes of data they need to process, there is no better solution than AI-driven recommendation systems. This also needs to be pointed out that the systems are constantly evolving because there are multiple instances when they can’t accurately predict what the user of a product or service might like next. Human psychology is extremely complex and can exhibit behaviors that don’t follow a pattern. It is in these areas that the recommendation engines have their tasks cut out and it is not going to be an easy task at all.