Augmented Reality in Retail: How AR Apps Drive Sales Up

Olga Aksenova

IT copywriter


20 Nov 2018

Reading time:

20 Nov 2018

In July 2016, New Yorkers were surprised seeing crowds of people run along the streets with smartphones in their hands and shouting Pokemon’s names. This is how the mobile game Pokemon Go burst into our lives and augmented reality apps became mainstream in the following years.

Why do retailers and E-commerce owners like AR? — Users prefer to buy online, but some goods are difficult to choose remotely. Before making a purchase, they need to try different models, watch them in a 360-degree view etc. This is where augmented reality comes in handy: users buy online or via mobile applications, use virtual fitting rooms, try things there and make orders without leaving home.

Many popular brands are aware of the benefits of augmented reality in retail. Cosmetics giant L’Oreal launched the app Makeup Genius in 2014. Using the AR app, shoppers can try different colors of lipstick or eyeshadows before purchase. More than 20 million users downloaded the app in 2 years. Another example from China — retailer Yihaodian — opened more than 1000 virtual shops that are available only via an augmented reality application. The results are impressive: Yihaodian got into the Guinness Book of Records when it sold more than 2 million bottles of milk in 51 minutes.  

Do you want to replicate L’Oreal and Yihaodian success? It’s time to create an augmented reality app for your retail business and step ahead of your competitors in the digital world. One of our US clients, Leef Company, came to the same conclusion when they turned to us and asked to develop a virtual fitting room for watches. Here is an overview of how we developed the AR app and the results.

How the Leef AR App Works

Users download the Leef AR app, open up the catalog in the app, and choose the watches they like. Then simply by pointing a smartphone camera at their hand, they try the watches on as though in real life.

Why is Leef AR App Helpful for Users?

  • They can try watches any time, anywhere.
  • They can see how exactly the watches will look like on their hand.
  • They can easily compare several watch models.

Why is Leef AR App Helpful for Retailers?

  • They can show the customers the whole product line.
  • They can tease new models that are in production.
  • They create a sensed-presence effect.
  • They save on opening additional retail outlets.

How We Chose the Approach

To develop a virtual fitting room we decided to use markerless augmented reality. In comparison with marker-based AR requiring visual markers, markerless augmented reality uses real world objects as markers. In our case hands were the markers.

The task was to train neural networks so they can show watches on the hand and let it move together with the hand like it’s real. We began with an app prototype on Python to check how the neural networks are working. Then we built iOS and Android apps.

How We Trained Neural Networks

To make a training dataset we drew a cross on the wrists of our colleagues to detect the watches’ position on their hands. We took photos of hands from different angles on a green background.

To increase the accuracy, we chose several backgrounds. So, we extended the dataset up to 60 thousand photos. Then we developed an algorithm searching and erasing the cross on the hand.

We trained three neural networks:

  • The first one was responsible for finding the point on the wrist, where watches should be located, and give it coordinates in pixels.
  • The second neural network — for detecting the wrist size in pixels.
  • The third had to recognize an offset and scaled position of the hand to place the wrist in space.

How We Built AR for iOS and Android

During the next stage we imported 3D model of watches into the iOS application using the framework SceneKit which works with older iPhone versions. A question came up: how to place virtual watches on the hand realistically? There is always a watch band part that is hidden under the wrist. We found an innovative solution: we drew a virtual transparent hand that erases the invisible band part.

When we built the Android augmented reality app, we had to train a new neural network model. The first algorithm didn’t work on Android: when the hand position changed, the model reacted very slowly and AR turned into a simple slide-show.

We quickly trained a new model to detect the wrist size and hand position. The app speed increased. To draw the watch on Android we used the Rajawali library.

What We Got

We made the AR app prototype for iOS and Android. There is only one watch model but future plans are to widen the catalog.

How AR App Works for The Customer

Our client Leef Company offers the AR app prototype to different retail brands, and some retailers are interested in the product already. The outcome was so impressive for our client that they returned with a new task: to develop an augmented reality app for trying on shoes.  

How To Use Augmented Reality to Increase Sales

If you are a retailer or E-commerce business owner, then with an AR app you can:

  • open a new sales channel
  • help users try goods remotely, in different positions and colors
  • allow to rotate and try the products just as in real life
  • show the customers the whole product line
  • sell new products even before they are marketed
  • save on extra physical retail stores

All of that helps you reach out to new audiences and run ahead of competitors. As a result, sales go up. Try it before someone else does!


Filter by