Amazon’s amazing algorithms – what can we learn from them?

When you use Amazon, you may not realise it, but that screen is actually a set of ‘tiles’ and what you see at any time (recently revamped) is finely tuned to your needs. What you don’t see, is what lies beneath the surface, a powerful AI engine that decides what you are seeing. This invisible hand not only determines what you see, it plays a role in what you do. It nudges you, in realtime, in one direction or another.

Amazon excel in making it easy to buy their product, they are also good at recommending what you buy and cross-selling other products. Their algorithms have been honed over many years taking inputs, such as what you’ve bought before, viewing history, repeat clicks, dwell time. your past search patterns, recently reviewed items, what’s in your cart, what site you were referred from, demographic data (where you live and what type of person you’re likely to be), user segmentation (if you bought books on photography, sell you cameras and accessories) and so on (Your Recently Viewed Items and Recommendations).

This is supplemented by aggregated data from other customers to produce differents sets of recommendations:

‘What Other Items Do Customers Buy After Viewing This Item?’

‘Customers who bought this item also bought this’.

‘Customers who shopped for Equality and Partiality also shopped for…’

‘Recommended for You Based on…’

‘Customers also Bought these Highly-rated Items’

Aggregated data from other similar customers and so on to nudge you towards the ‘Frequently Bought Together’ technique or a range of related items in terms of brands, function, colour and size.  Best sellers are also pushed. They also recommend newer products, such as Kindle versions. Packages of products, such as item plus carrying case, are also pushed. Reductions on postage and discounts also boost sales. The bottom line is that this is a bottom line result, as you buy more. Their conversion rate from website recommendations is very high.

How do they do this?

They do all of this through their own collaborative filtering algorithm, that has the goal of increasing the average order price. Note that this may not be the ideal goal, as some customers may order more through lots of smaller orders, rather than the occasional large one. They do this through A/B testing (trying different things out and measuring the effects), data mining, affinity analysis and collaborative filtering. This is not easy. Algorithms are sensitive things and one technique may require too much data to be practical, take too long, or have odd (weird pricing) sparse or low quality outputs. Performance and scaling problems are big issues. You can reduce the data size by sampling or partitioning, by product or category, but this reduces the quality of the recommendation. It’s all about trade offs. Clustering into groups that are similar is another technique but difficult to scale. What Amazon actually does, is rely on item data i.e. the similarity or relationships between products, not just buyers. It looks for correlated or most popular items. This is called ‘item-to-item collaborative filtering’. This emphasis on item data means their recommendation engine is super fast.

Remember also, they are the biggest online retailer in the word, so they have the biggest data sets. This gives them a real market advantage as their algorithms have more to work with, can be tested on larger groups and use more aggregated data.

Human reviewers a problem?

One of the things that screws algorithms up is bad data. If the reviews are positive but they’ve been fixed by humans, that skews the recommendations. Publishers have been paying people to review by offering discount product, so Amazon has just banned incentivised reviews. In other words, slowly but surely, the human factor is being squeezed out and this is to our benefit.

Algorithmic bias?

One could argue that this is not in itself a good thing, as the algorithms are written by humans and may contain bias. I think this is a red herring. Compare this approach with the average bookstore employee. It outguns them. The algorithm is far less likely to contain bias on gender, race and social background, as it doesn’t make judgements on these. Neither will it have the vast array of cognitive biases that we humans all have. It is ruthlessly objective.

One thing that does screw up this ‘perfect’ market through algorithms is promotion. Amazon will promote products that they get a better margin on. You never know what the behind the scenes deal is with certain suppliers in terms of promotional intent and pricing. You may very well be getting recommendations based on their margins, than your preferences. On the whole, however, their prices tend to be better than physical retail outlets.

Serendipity lost?

Another argument could be that it eradicates serendipity – those chance encounters in bookshops where you find a gem. I have some sympathy with this view, but suspect that when you actually compare browsing in a bookshop with browsing online, that factor is exaggerated by the bookstore supporters (biased towards past behaviour, nostalgia and neophobia). The bottom line is that Amazon has a larger collection of books than any bookshop. Indeed, many small bookshops now survive on Amazon sales. Its recommendation engine also promotes the sort of ‘impulse’ buying you see in bookstores through last minute cart recommendations. Nevertheless, the stronger a recommendation engine becomes, one could argue, the narrower the browsing and buying behaviour, This is a trade off, a common problem when designing algorithms.

It learns

One final point, is that, with machine learning, these algorithms simply get better and better. They are learning from their own generated data and can adapt, sometimes on their own to improve. This is way smarter than any human approach to recommended sales. Remember also that Amazon is increasingly automating its distribution through AI, as people are replaced by robot pickers and loaders. Beyond this AI driven drones may well deliver to your doorstep.

What can we learn from this?

What do we have to learn about learning from all of this. First, that recommendation algorithms will certainly play an increasing role in the learning game. They already have. Google is arguably the number one technology used by learners. We use it to search for things we want to know. That is a fundamental piece of pedagogic technology that has revolutionised the learning process, whether you’re searching for knowledge, instruction on a process or skill on YouTube (owned by Google), or on Google Scholar for research. Beyond this we have tools that use AI to create learning content and experiences, in realtime, such as WildFire. Then we have adaptive learning systems that use AI to navigate learners through courses. On top of this there’s advances on online assessment through digital ID, face recognition and automated answer and essay marking. This is the one area, AI, where significant advances are being made, quickly. These advances promise to do what they’ve done for Amazon and others, offer a massively scalable solution to the problem of teaching and learning through realtime learning design, personalised recommendations, less linear teaching and better assessment. All of this, of course, driven by machine learning, which makes these systems learn as they go. The scalable online teacher is a fast learner that learns as it goes. It just gets better and better.

Above all

Above all, what we can learn from Amazon is that you, the learner, are a valued customer and that they tailor their service to your actual needs, in realtime. Without getting too angsty about the use of the word ‘customer’, which seems to trigger all sorts of people into surface criticism, what education badly needs is this approach to learning, not the blind, batch process we currently have. Let's not over-romantisie batched lectures to hundreds in raked lecture-hall or classrooms of 30+ kids struggling with the intricacies of maths or French. There's plenty of headroom for improvement here, and some of the techniques have already been put to the test over decades with proven efficacy in target groups of the tens and hundreds of millions.