Marketplaces – N11

The marketplace generates revenue from the number of items purchased. In this model, personalization can add value by serving relevant offers and recommendations to the visitors, increasing the conversion rate, average cart size, and creating a better user experience, leading to more loyal and frequent customers.

Comparing different personalization engine providers

N11 already had a personalization solution in place once they decided to test it against another solution in the market.

The marketplace decided to compare Yusp to its previous vendor during an A/B test and in an attempt to increase their performance and receive day-to-day help with their support requests. 

We summarize the test and the results below.

The general objectives of the A/B test

• Testing if the Yusp personalization solution can perform significantly better – in terms of the KPIs specified below – than the solution of the other vendor
• Measuring the difference in the value generated by the two solutions
• Increasing sales on the n11.com website
• Improving user experience n11.com website

The KPIs measured in the test

• GMV (Gross Merchandising Value) through recommendations
  • The value of conversions generated by product recommendations – the user purchased a product 72 hours after clicking on it in a recommendation widget.
  • Measurement unit: TRY (Turkish lira)
• GMV through recommendations per 1000 recommendations
  • The value of conversions generated by product recommendations, compared to the number of recommendations displayed – we added this measurement to calculate the return on personalization service costs. 
  • Measurement unit: TRY (Turkish lira) / 1000 recommendations
• CTR (Click-through rate)
  • The number of clicks on product recommendations, compared to the number of recommendations displayed – this KPI indicated the accuracy of recommendations and gave us an idea about the user experience.
  • Measurement unit: %

The A/B Testing Methodology

The critical requirement of the A/B-test is to guarantee the same conditions for the solutions compared before and during the evaluation period.

The test also has to meet the following conditions:

• Divide the users into two groups, where the groups’ size is roughly identical, and the difference between the sum of past purchase values of the two groups is statistically insignificant. Such partitions of the users can be easily performed based on their user ID or the cookie ID. Every new user should be assigned randomly (50–50%) into one of the groups.
• The visual appearance of the recommendations should be identical.
• All recommendation scenarios should be served by both solutions, except if not supported by the vendors.
• The evaluation period should be sufficiently long to arrive at a conclusion. We usually suggest two months for such tests.

Replacing the previous vendor with Yusp results in incremental added revenue

The Proof of Concept (POC) period was running for two months, during which Yusp managed to outperform its competitor significantly in all KPIs. Personalized placements were available on all N11 platforms (web, mobile web, iOS, and Android applications) measured in 8 different placements. In the case of each placement, Yusp proved to be the more efficient solution.

During the Proof of Concept period, Yusp managed to increase the recommendation driven GMV compared to the previous vendor. In terms of CTR, Yusp managed to perform 35.2% better, increasing the baseline from 1.96% to an overall average of 2.65%. Yusp performed even better on mobile platforms lifting the CTR benchmark from 2.43% to 3.28%.

Working together towards the perfect marketplace personalization

Working together on marketplace-specific challenges, increasing the overall revenue through recommendations

Ever since the successful POC period, Yusp and N11 continuously work together to increase the performance of recommendations further and find new ways to optimize the service. 

Both teams are entirely on board with the latest results and developments thanks to our daily interactions, which allows us to react to any unforeseen events quickly. 

Recently, we aimed to make better use of the data to tackle some marketplace-specific challenges. These improvements touched on the data connected to product identification and the rating of sellers.

Improving product identification was necessary because of the high number of similar or identical products offered by various sellers, each using different names and attributes. N11 developed a three-level product identification based on three parameters:

• ProductID: unique for all products
• GTIN: same for all completely identical products (e.g., iPhone X 256GB, Rose Gold)
• GroupID: same for all almost identical products (e.g., iPhone X 256GB, any color)

In the case of the seller ratings, N11 created a tool for its users to rate sellers based on their shopping experience – from choosing the product on the website until the final delivery. The explicit user feedback is now being taken into account when calculating prediction scores to recommend a specific product type. The better overall user experience a seller provides, the better their products rank. 

Improving results with deep learning algorithms

Deep learning is a subclass of machine learning that uses complex models of a cascade of non-linear processing layers – also called deep neural networks – to learn different data representations. 

Starting from the early 2010s, deep learning has achieved significant improvements in complex domains such as image recognition, speech recognition, natural language processing, and more.

The uptake of deep learning applications for recommendation engines began in late 2015, fostered by a handful of pioneering research groups. Yusp has been one of these pioneers and is considered one of the field’s leaders.

Results of applying a deep learning-based personalization algorithm:
The Yusp algorithms provide recommendations with session-based user behavior in mind. The engine recommends items that are suitable for continuing the current session of the user.

The transition between item-to-item and personalized recommendations provided the following results during a recent internal A/B test:

• Home page placement: 14.62% uplift in revenue generated through recommendations
• Product page placement: 10.59% uplift in revenue generated through recommendations

Deep learning at scale:
Deep learning models are computationally expensive and often require specific hardware to run efficiently (GPUs instead of CPUs). Yusp developed a unique way to serve recommendations by our deep learning models using CPUs only (training still uses GPU) without affecting the quality of recommendations. This development allows Yusp to cost-efficiently use deep learning for more use cases.

25 different algorithms with 62 placements developed for N11

We placed 25 recommendation scenarios on the N11 web, mobile, and app platforms. These scenarios created 62 placements that contain the most relevant available items to all users. 

The recommendation engine defines the products displayed and their order in the boxes. They appear on all important screens, such as the homepage, the category page, the product detail page, the basket page, and the successful order page.

Summary

N11 Turkey’s largest marketplace aimed to improve its personalization solution’s performance and decided to compare their previous vendor with Yusp.

During the Proof of Concept period, Yusp managed to increase the recommendation-driven  Gross Merchandising Value compared to the previous provider.

Today, N11 leverages the personalization potential of Yusp on its web and mobile platforms in over 60 placements. They create a significant part of their total revenue through personalization solutions.