A Machine Learning Approach to Grocery Promotion
The common understanding is that AI is a black box that devours data and spits decisions at scale. But is it possible to use it to visualize data from billions of shopping baskets into a simple product map that will allow us to see how market structures work? A recent study by our Chief AI Officer Dr. Sebastian Gabel says yes.
In grocery retail, understanding market structures is essential. Retailers need to know which products work well together (complements) and which compete with each other (substitutes). An example of complements would be ingredients for Italian pasta, such as pasta itself, tomato sauce, parmesan cheese, olives and even black pepper. Customers often shop them together in one or across several shopping trips. Knowing which exact goods are complements helps retailers to place them closer together in stores and promote them in weekly circulars.
As opposed, substitutes are products that consumers don’t differentiate that much. They compete with each other, but don’t create any additional demand. An example would be 3 different brands of pasta. But just because there are 3 brands of the same product doesn’t automatically mean they are substitutes. They might differ in price, quality, used ingredients or other characteristics (e.g. gluten-free, organic or local). And let’s not forget the influence of branding. A strong brand might mean that otherwise a commodity product is perceived somehow special in the eyes of the customer.
There is no way to understand these complex structures just by assuming “we know”. Millions of consumers have millions of perceptions and ways to make decisions. Yet there are some behavioral patterns that are highly common and their understanding can help retailers to optimize mass-advertising greatly. And these patterns can be found in shopping data.
Revealing market structures from shopping data
To illustrate how market structures work, we at SO1 together with Humboldt University in Berlin conducted research to analyze 12 months of shopping data collected in a large German city from 147 grocery stores. The research was led by our Chief AI Officer Dr. Sebastian Gabel and the full paper was published in the Journal of Marketing Research.
We used advanced machine learning algorithms derived from the SO1 Engine to generate a detailed product map where every product in a retailer’s assortment is depicted as a single dot. These dots naturally form various clusters according to how likely they are to co-occur in customers’ baskets across one or multiple shopping trips.
These product groupings do not consider a retailer’s own category definition, only consumers’ views of market structures derived from millions of shopping baskets .
Some clusters on the map are similar to retailers’ own product categories, such as juices, soft drinks, chocolate bars, wine or bottled water. But the majority of these clusters include products from several different categories, such as Italian, Barbecue, Baking, Breakfast, Vegetarian, Organic and even Children’s Products (e.g., small juice boxes, snacks for school, bologna in the shape of teddy bears, or Disney yogurt). Products from these clusters are mostly located on different shelves throughout the store, but customers tend to buy them together.
Below is an example of how such product cluster can look like. We zoom into an “American” BBQ sub-cluster, which is part of the bigger BBQ cluster.
The top part contains ingredients for “American” barbecue dishes such as hamburgers and hot dogs. Hot dog sausages and hot dog buns are located on the left, burger buns and beef patties on the right. In between, we find complementary ingredients that can be used with both hot dogs and hamburgers (e.g., sliced pickles, fried onions, and sauces).
Although this might seem obvious for humans, remember that the AI groups the products completely autonomously based on data without any human assumptions. Machine learning can be used to group complementary products in ways that reflect how consumers intend to use them (i.e. to cook Italian food, make a BBQ evening, shop for breakfast, quick convenience food or buy stuff needed for kids) without any guess work or human input. These intent-based groups inspire shoppers and create higher engagement when promoted and located together.
Discovering regional differences and seasonability
Discovering product groups as seen by customers is already a valuable insight to run promotions and design stores’ shelves, but the true value of this approach lies in its ability to observe differences across various store locations and even seasons.
To show what this can look like, we compare sixteen product clusters across three sample stores. Although the stores are in the same city under the same brand and have almost identical assortments, we observe interesting differences between the sales across product clusters:
Store A is located in a residential area with higher average income and education. It has higher sales of vegetables (+35.4% compared to average) and organic products (+149.9%) while lower consumption of recipe mixes (-65.5%) and lunch snacks (-38%).
In comparison, store B, located in lower-income residential area, has considerably higher sales of BBQ products (+71.2% vs. -23.8% for higher income area) as well as German food (+31% vs. -45.4%) and dog food (+80.7% vs. -41.7%). Compared to store A, shoppers at store B prefer beer over wine (+31.2%) and consume more recipe mixes (+38.6%).
In store C, located in the city center, we observe a big increase in consumption of lunch snacks (+200.1%), fresh juices (+184.2%) and ice cream (+69.3%), and lower consumption of baby products (-44%) and dog food (-70.8%).
Of course, these insights can be learned from sales data itself if you know what to look for. But compared to just observing sales, the P2V-MAP approach reveals even the slightest differences in shopping behavior across different regions, such as what people put into their hot dogs and burgers. These insights can be used to group and promote these products together and craft localized messages. The same approach can be used to compare different seasons and customer segments.
The bottom line
Applying machine learning to analyse market structures proved to be a highly effective method to optimize assortments, promotions and shelf design on an individual store level. It allows retailers to group products in a way consumers shop for them, as opposed to retailer’s own category definition based on logistics and category management. The approach is automated and bias-free, only considering shopping data.
To increase a loyalty program’s performance, however, we recommend to personalize promotions for each customer individually. Once the AI understands the market structures, it can also understand customer preferences and recommend to them the right product at the right price—on demand and fully autonomously.
You can learn more about this technology at www.so1.ai and subscribe for our newsletter below to receive regular publications on AI and retail technology.