View All Blogs
Author: Palak Kumar
The Internet world has been changing rapidly and is very competitive. Users have a lot of choices and the switching costs from one service provider to another is quite low. In this scenario, companies need to constantly evolve to stay strong and grow. If the decisions taken by companies are in the wrong direction, it can significantly hurt the competitiveness of companies. Hence, companies are trying to leverage the power of data to stay ahead. In this article, we will look at one example, Netflix and see how they are using data to gain a competitive advantage.
Netflix is a streaming service that offers a wide variety of award-winning TV shows, movies, animations, documentaries. They do this through a large network of internet connected devices. Users can watch as much as they want, whenever they want without being bothered by ads for a low monthly fee. They have attractive features like cancel anytime with no questions asked and hence it is very important that they retain their customers to generate revenue. It is always more difficult to attract new customers to generate revenue rather than keeping existing customers. One-way Netflix uses to retain customers is to keep the users hooked on their programming content. For example, if a user watches a movie that they are interested in the first month of subscription but does not find any other good movies to watch, they may end up cancelling the subscription. Hence, Netflix must understand what is the content the users are watching and then recommend other movies or content that would be interest to the users and share this list with them so that they can continue to be hooked onto the platform and thus be a loyal customer.
In order to do this, Netflix cannot rely on general trends as people are different and their likes and tastes are different. Hence, based on the initial viewing patterns of the people, they need to generate customized list of recommendations that are only applicable to the specific user. Each user thus gets their own list of recommendations based on their past viewing patterns. How does Netflix go about generating this list? They can mine the data from other users who have similar tastes and see what other content similar users liked. This information is then used to develop tailored content for each user. This approach to data mining and developing recommendations is not unique to Netflix. A number of large companies are leveraging similar data. For example, Google can understand user preferences based on their past search behaviours and share targeted ads that are of most interest to the user. Amazon has a history of your purchases which they can use to show similar other products of interest or show coupons or other ads targeted at the user. These techniques are very powerful and help these companies generate millions of dollars in revenue.
There are a number of techniques to mine the data and generate recommendations but in this article, we will focus on one such technique called the A/B testing. A/B testing is a process of showing two variants of similar content to different segments of the visitors at the same time and then comparing which variant drives more conversions. A/B testing is data based and uses actual data from user behaviour to make decisions and hence is a very systematic way of finding out what works and what does not work. Once you understand what works, you can then replicate this experience to all your users to achieve your conversion goals in the most efficient manner. A/B testing enables you to make the most out of your existing customers.
Performing the A/B testing
If you are interested in using A/B testing, the first step is to collect data from user behaviours. You can collect all sorts of data such as how the customers interact on your website. You can use website analytics tools to figure out what links the users are watching and clicking on and how much time they spend on which content and their bounce rates from various pages. The second source of your data could be user surveys which can provide direct insights based on how users respond to surveys. For example, you can have users rate the movies they watch, and this information can be valuable to understand if the users like the content that they just watched. Not everyone will respond to these surveys but those who do can provide valuable feedback.
Once you understand the viewing patterns of your customers, the next step is to create variations based on your hypothesis. A variation is another version of the current version you proposed with some changes you want to test. The existing version could be your control and each of these variations are tested against the control to see which works the best. For example, based on the viewing patterns if the customers are shown potential movies A, B, and C. You may want to consider another variation where you show options A, B, and D.
Once you determine your tests, you run the tests for a given period of time. For example, you can run the control sample for 50% of your customers and the other test of recommendations for the remaining 50% of your customers for a period of say 1 week. Based on your test results, you can then analyse the results statistically to see if the differences are statistically significant. We need to use statistical theory since we do not want to make recommendations based on chance occurrences we observe during our testing.
In order to determine if statistically there is a difference between the two tests, you use hypothesis testing. In a hypothesis test, the null hypothesis (Ho) is a statement of no difference which states that there is no statistical difference between the two sets of data. The alternative hypothesis (Ha) could a statement that the users prefer one option better than the other. The data from the testing is then used to determine if we should accept the null hypothesis or if there a strong enough proof to reject the null hypothesis and accept the alternative hypothesis. Hypothesis testing eliminates the guesswork and helps us provide better experience to our audience.
This simple statistical technique is the secret sauce of the growth of Netflix. According to the Netflix Tech blog, they say that Netflix succeeds because of the A/B testing. Now this involves dividing the audience into two halves and showing them different versions to see which version works better to achieve a goal. For example, if the goal is to increase the watch time of the users. They have an idea of a feature that could possibly help them achieve their goal. However, they are unsure of the feature and decide to go a for a test.
Instead of introducing this feature to everyone, they release it to a subset of some experiment users. Users with experiment feature are the experiment group and people without the feature form the control group. To evaluate the effects of the feature, a comparison is done between the average daily watch time of control users and experiment users. Here the Null hypothesis will be that the feature does not improve the watch times, and the alternative hypothesis will be that the feature improves the watch times of the users. Once data is collected for a sufficient period of time to reduce the Type I and Type II errors that we may make, we can perform the hypothesis testing and conclude if the new feature we are considering is worth it and should be rolled out to all the users.
Conducting the Hypothesis test in Sigma Magic
You can use a software such as Sigma Magic software to perform the hypothesis testing. In the figure below, the data for the control group is shown in the left-hand side column and the data from the experiment group is shown on the right column. To compare the average viewing times, we use a 2-sample t test here and from the analysis results it is clear that the P value is low indicating that we have enough evidence to reject the null hypothesis and state that viewing times are different between the experimental group and the control group. This analysis results also show the 95% confidence interval for the difference in the viewing times between the two groups. If this difference is practically important, then we can choose to rollout this feature to the extended community of users and reap the benefits of increased viewership times.
The benefits of performing A/B testing can be very high. It helps with the company’s efforts to improve the existing processes using data and fact-based decision making.
Challenges with A/B testing
- Conducting the A/B testing takes time and effort. So, if there are a large number of things to test it may be challenging to run a lot of these experiments. If frivolous changes are tested, then we will find that there is no statistical difference between the two groups. Hence, marketing team needs to decide which of the tests to actually perform so that the results will come out to be statistically significant.
- Need a structured approach to conducting the testing and collecting the data. As there are probably a lot of other factors changing in the market as well, we need to ensure that the data that is collected for this exercise is properly collected otherwise we may end up making the wrong decisions. For example, a holiday period during the testing of one sample may significantly affect the test results.
- We need to ensure that we collect sufficient samples to avoid making Type I and Type II errors since we are working with samples. Typically, we want to ensure that we have a confidence level of 95% and a power of at least 90% for our test results. In order to achieve this level, we need to ensure that we collect sufficient samples.
In summary, A/B testing is a powerful method that marketers can use to make fact based decisions. It is a very generic methodology that can be used to answerany question that involves two choices - such as should we recommend movie A or movie B? Performing the testing and learning from real data makes this approachinvaluable and fact based. The testing itself is simple to do and the analysis can be fast and the recommendations to deploy the changes can usually be doneat minimal cost to the company but the returns obtained by companies can be significant. Not only do they improve customer experience but they can helpcompanies retain their customers and generate additional revenue with minimal expenditure.
Follow us on LinkedIn to get the latest posts & updates.
Disclaimer
Any images shown on our website were sourced from websites such as https://www.pexels.com/, https://pixabay.com/, https://undraw.co/, and https://unsplash.com/. Only images that were displayed as Royalty Free and those that can be used for Personal and Commercial projects were used. Under no circ*mstance do we claim ownership of these images. In fact, we have no means of detecting if in fact these images are copyright free. If you feel any images are infringing on any copyrights, please let us know and we will promptly take down those images.
