Due to the recent pandemic, customers now do most of their shopping online, which comes as a challenge for most brands. Therefore, retailers must be prepared with an appropriate response and reimagine their strategies. They need to shift their focus on creating digital and multichannel shopping experiences.
Implementing a personalization solution for digital channels is an impactful strategy to promote essential and non-essential goods online and help brands with when and which channels to engage their audience. Interacting with customers one-to-one can significantly impact customer satisfaction and revenues.
In this case study, we present an example and introduce how Cora Romania managed its personalization efforts by implementing the Yusp engine and significantly increased its revenues.
Marketplaces – n11
N11.com is a leading Turkish marketplace for the 7th year in a row. Currently, it has 250 million monthly visits, with more than 20 million available items.
250 million
monthly visits
20 million
available items
The marketplace generates revenue from the number of items purchased. In this model, personalization can add value by serving relevant offers and recommendations to the visitors, increasing the conversion rate, average cart size, and creating a better user experience, leading to more loyal and frequent customers.
Comparing different personalization engine providers
N11 already had a personalization solution in place once they decided to test it against another solution in the market.
The marketplace decided to compare Yusp to its previous vendor during an A/B test and in an attempt to increase their performance and receive day-to-day help with their support requests.
We summarize the test and the results below.
The general objectives of the A/B test
Testing if the Yusp personalization solution can perform significantly better – in terms of the KPIs specified below – than the solution of the other vendor
Measuring the difference in the value generated by the two solutions
Increasing sales on the n11.com website
Improving user experience n11.com website
The KPIs measured in the test
GMV (Gross Merchandising Value) through recommendations
The value of conversions generated by product recommendations – the user purchased a product 72 hours after clicking on it in a recommendation widget.
Measurement unit: TRY (Turkish lira)
GMV through recommendations per 1000 recommendations
The value of conversions generated by product recommendations, compared to the number of recommendations displayed – we added this measurement to calculate the return on personalization service costs.
The number of clicks on product recommendations, compared to the number of recommendations displayed – this KPI indicated the accuracy of recommendations and gave us an idea about the user experience.
Measurement unit: %
The A/B Testing Methodology
The critical requirement of the A/B-test is to guarantee the same conditions for the solutions compared before and during the evaluation period.
The test also has to meet the following conditions:
Divide the users into two groups, where the groups’ size is roughly identical, and the difference between the sum of past purchase values of the two groups is statistically insignificant. Such partitions of the users can be easily performed based on their user ID or the cookie ID. Every new user should be assigned randomly (50–50%) into one of the groups.
The visual appearance of the recommendations should be identical.
All recommendation scenarios should be served by both solutions, except if not supported by the vendors.
The evaluation period should be sufficiently long to arrive at a conclusion. We usually suggest two months for such tests.
Replacing the previous vendor with Yusp results in incremental added revenue
‘Have you seen these products?’ placement on the homepage
The Proof of Concept (POC) period was running for two months, during which Yusp managed to outperform its competitor significantly in all KPIs. Personalized placements were available on all N11 platforms (web, mobile web, iOS, and Android applications) measured in 8 different placements. In the case of each placement, Yusp proved to be the more efficient solution.
During the Proof of Concept period, Yusp managed to increase the recommendation driven GMV compared to the previous vendor. In terms of CTR, Yusp managed to perform 33.8% better, increasing the baseline from 1.96% to an overall average of 2.65%. Yusp performed even better on mobile platforms lifting the CTR benchmark from 2.43% to 3.28%.
Working together towards the perfect marketplace personalization
Working together on marketplace-specific challenges, increasing the overall revenue through recommendations
Ever since the successful POC period, Yusp and N11 continuously work together to increase the performance of recommendations further and find new ways to optimize the service.
Both teams are entirely on board with the latest results and developments thanks to our daily interactions, which allows us to react to any unforeseen events quickly.
Recently, we aimed to make better use of the data to tackle some marketplace-specific challenges. These improvements touched on the data connected to product identification and the rating of sellers.
Improving product identification was necessary because of the high number of similar or identical products offered by various sellers, each using different names and attributes. N11 developed a three-level product identification based on three parameters:
ProductID: unique for all products
GTIN: same for all completely identical products (e.g., iPhone X 256GB, Rose Gold)
GroupID: same for all almost identical products (e.g., iPhone X 256GB, any color)
In the case of the seller ratings, N11 created a tool for its users to rate sellers based on their shopping experience – from choosing the product on the website until the final delivery. The explicit user feedback is now being taken into account when calculating prediction scores to recommend a specific product type. The better overall user experience a seller provides, the better their products rank.
Improving results with deep learning algorithms
Deep learning is a subclass of machine learning that uses complex models of a cascade of non-linear processing layers – also called deep neural networks – to learn different data representations.
Starting from the early 2010s, deep learning has achieved significant improvements in complex domains such as image recognition, speech recognition, natural language processing, and more.
The uptake of deep learning applications for recommendation engines began in late 2015, fostered by a handful of pioneering research groups. Yusp has been one of these pioneers and is considered one of the field’s leaders.
Results of applying a deep learning-based personalization algorithm: The Yusp algorithms provide recommendations with session-based user behavior in mind. The engine recommends items that are suitable for continuing the current session of the user.
The transition between item-to-item and personalized recommendations provided the following results during a recent internal A/B test:
Home page placement: 14,62% uplift in revenue generated through recommendations
Product page placement: 10,59% uplift in revenue generated through recommendations
Deep learning at scale: Deep learning models are computationally expensive and often require specific hardware to run efficiently (GPUs instead of CPUs). Yusp developed a unique way to serve recommendations by our deep learning models using CPUs only (training still uses GPU) without affecting the quality of recommendations. This development allows Yusp to cost-efficiently use deep learning for more use cases.
25 different algorithms with 62 placements developed for N11
We placed 25 recommendation scenarios on the N11 web, mobile, and app platforms. These scenarios created 62 placements that contain the most relevant available items to all users.
The recommendation engine defines the products displayed and their order in the boxes. They appear on all important screens, such as the homepage, the category page, the product detail page, the basket page, and the successful order page.
Personalized recommendations on the main page – Personalized recommendations based on recently viewed products, from any category, recommended products should not overlap with other recommendation boxes on the main page.
Personalized recommendations on the product page (web) – Recommending similar products to a selected product based on user behavior (collaborative similarity), recommended products should not overlap with other recommendation boxes.
Personalized recommendations on the product page (mobile web)
Personalized recommendations on the category page (web) – Personalized recommendations based on recently viewed products
filtered for the actual main category
Summary
N11 Turkey’s largest marketplace aimed to improve its personalization solution’s performance and decided to compare their previous vendor with Yusp.
During the Proof of Concept period, Yusp managed to increase the recommendation-driven Gross Merchandising Value compared to the previous provider.
Today, N11 leverages the personalization potential of Yusp on its web and mobile platforms in over 60 placements. They create a significant part of their total revenue through personalization solutions.
We use cookies to ensure that we give you the best experience on our website. AcceptDeclineRead more
