How is 'Amazon's Choice' chosen? Amazon won't say

One of Amazon's many great strengths is the sheer breadth of its inventory. The retailer is said to stock hundreds of millions of different products, with everything from fishing rods to dog food in its virtual aisles. But that degree of choice can be overwhelming, too: If you're shopping for, say, wireless headphones, you've got over 3,000 models from which to choose. 

All things being equal, you might go for the one that has the "Amazon's Choice" label on it. That little black badge has migrated to Amazon's main site after starting as a way to streamline voice purchases from Amazon's Echo voice speaker back in 2015. 

But what, exactly, does it mean? If you mouse over the Amazon's Choice badge on the company's site, the pop-up explanation is this: "Amazon's Choice recommends highly rated, well-priced products available to ship immediately." OK then. But is "highly rated" based on user ratings, or does it aggregate external reviews, Rotten Tomatoes-style? And how is "well-priced" defined? 

Enlarge Image

The questions steamroll from there. Is it a designation for which manufacturers can apply -- or maybe even pay? And who, exactly, is making the "Choice": Is it bestowed from a bunch of in-house Amazon experts, or -- like fellow technology behemoths Google and Facebook -- does Amazon just deploy yet another algorithm, wherein a "good deal" is something that's in stock that you have a higher chance of buying, based on your purchase history?

Those were exactly the questions going through my head when I noticed the Amazon's Choice logo on the Jabra Elite 65t, a new fully wireless headphone that I had just reviewed for CNET -- and quite liked. But the product was initially only available at Best Buy , and not set to ship on Amazon until March 20. Still, despite not being available and only having a single user review, it was already labeled as an Amazon's Choice. 

Thus began my quest to find out what Amazon's Choice actually signifies, and how it was awarded.

How Amazon describes 'Amazon's Choice'

Beyond that one-line definition of Amazon's Choice when you click on the badge, don't expect to find much info on it on Amazon's actual site.

The closest I came was an Amazon employee calling it "a feature that helps Amazon customers save time and effort when searching for common, everyday items and suggests highly-rated, well-priced products with Prime shipping." The answer came on Amazon's message boards, in response to a user question in 2016.

When I asked for a current definition, an Amazon spokesperson was happy to fill me in:

"We launched Amazon's Choice in 2015 as a way to simplify shopping for customers by highlighting highly rated, well-priced products ready to ship immediately for the most popular searches on Amazon. It's been a really popular feature both on Amazon and on Alexa , because it allows customers to make fast and easy decisions on what to buy. Amazon's Choice is just our recommendation, and customers can always ask for specific brands or products if they choose."

Allow me to unpack that a bit.

The program first launched back in May 2015 as a feature for the original Amazon Echo Alexa voice-enabled speaker, which had debuted about six months earlier. According to the Wall Street Journal, reporting on it at the time, the concept behind it was that if you asked Alexa to order an item such as toothpaste, it would first look at your order history and offer to send you the previous toothpaste product you purchased. But if you'd never ordered any toothpaste, Alexa (Amazon) would "make a recommendation from its Amazon's Choice program."

Gradually, Amazon began expanding the feature from its Echo voice speaker to its website, revealing to shoppers which products were in its Amazon's Choice program by attaching a badge to those products.

What Amazon's Choice isn't

If the "what" about Amazon's Choice is pretty straightforward, the "how" is shrouded in mystery. Simply put: Is this thing chosen by humans, or is it a machine-driven algorithm?

On the surface, it may not seem like a hugely important question. But people might react differently to the badge if they knew it was generated by an algorithm. Some of us may already assume it is. But others may think a human is involved. Would we trust the badge less -- or more -- if we knew the picks were driven solely by software?  

Beyond the official definition quoted above, getting insight into the program is a challenge.

Back in 2015, the Wall Street Journal's Greg Bensinger -- in the same story linked above -- asked some similar questions and got bupkis. "Amazon declined to detail the process behind Amazon's Choice, including whether its picks are driven by software or humans," the article stated. Amazon didn't respond to a request for comment from the Huffington Post when it followed up on the same story at the time, either.

We can, however, engage in a process of elimination based on how we know Amazon's Choice is not determined.

Amazon's Choice designees have no idea how they "win": Companies that receive the Amazon's Choice badge on their products seem to be in the dark about exactly why the product received the label. "The net-net is, no one knows for sure," a PR rep for Creative Labs wrote me. "I wish I had something more concrete to tell you, but it seems it's a puzzle for all." He directed me to this page at AMZ Advisers, a third-party consultancy that seems just as much in the dark about how to snag the Choice badge as everyone else.

Companies can't "buy" an Amazon's Choice badge: On a European message board for Amazon resellers, a moderator who appeared to be an Amazon employee posted this:

"Sellers don't need to enroll or register to be considered for Amazon's Choice -- the Amazon's Choice feature considers all highly-rated, Prime-eligible items sold or fulfilled by Amazon. It's not possible to specifically request that your product be selected as Amazon's Choice ... selections are constantly updated, so continuing to offer high quality, well-priced products to your customers will give you the best chance to be selected as Amazon's Choice."

Enlarge Image

Amazon's Choice winners aren't based on any human testing: A person who's familiar with the program told me that Amazon employees don't test the products that are selected for the Amazon's Choice program. An Amazon employee may very well have used the product as part of his or her day-to-day life, but there don't seem to be any designated testers in the CNET connotation of the word.

Third-party evaluations don't determine Amazon's Choice designees: I speculated that outside reviews -- from sources like CNET, Engadget, Consumer Reports or the like -- might be a factor. But apparently that's not the case, the person said. So my Jabra Elite 65t review had no bearing on that product's Amazon's Choice badge.

It gets weirder

Now, it's not strange that many Amazon-branded devices have the Amazon's Choice badge. I wouldn't be so naive as to suggest that Amazon should exempt its own devices -- Kindles, Fire tablets , Echo speakers and plenty of AmazonBasics accessories -- from its highlights program. But here's the catch: How you search for the product can impact whether a badge shows up or not. 

At first glance, only the Kindle Paperwhite gets the badge. The entry-level Kindle, Voyage and Oasis don't get it. Also, only certain Echo speakers and Fire tablets have the badge. But click around, and you'll find that the badge pops in on products that didn't have it earlier.

For instance, I did a search on Amazon for "Amazon Echo speakers" and came across the Amazon Tap . I clicked on this link. There was no Amazon's Choice badge on the product page. But then I used the navigation bar at the top of the page that allows you to sort through all Echo products. I clicked on the Tap and came to this page, which had the Amazon's Choice badge. Now, magically, you can see that Amazon had given the Tap and Amazon's Choice for people who asked Alexa for "Alexa Tap."

Something similar happens with the Kindle Oasis . On this page the Oasis doesn't have the Amazon's Choice badge while on this page it does. It appears the new Kindle Oasis received an Amazon's Choice for people asking Alexa for "newest Kindle."

The Choice badge can also come down to a device's color. In our searching, the black Echo Dot got the nod, while the white one was left wanting.

I asked Amazon to comment on the discrepancy, but I didn't hear back so I don't know whether we're looking at a bug or a feature. Either way, it's weird -- and it isn't limited to Amazon-branded products.

But who -- or what -- is making the 'Choice'?

To be eligible to for Amazon's Choice, we know that products need to be "highly rated" (by Amazon user rating), "well-priced" (whatever that means) and eligible for Amazon Prime shipping. 

We thought it needed to be in stock and available, but that preorder Jabra headphone seems to belie that.

And we know that external evaluations, human testing and the manufacturer's relationship with Amazon do not seem to be factors. Being a first-party Amazon device doesn't even seem to be a huge help in getting the badge.  

So how, again, is Amazon's Choice actually chosen?

There's an algorithm at the heart of it. I haven't heard anybody explicitly deny that. The entire Amazon shopping experience is built on algorithms ("Customers who bought this item also bought"), so it only makes sense. The fact that a badge gets added to an Amazon product when it hits a sweet spot of popularity and availability is certainly less problematic than YouTube elevating creepy, disturbing videos on its kids section, or Facebook or Google surfacing fake news conspiracy theories at the top of search results during the crucial first minutes of real-world events.

Still, I'm a professional product reviewer who has, over the years, bestowed a CNET Editors' Choice on products that demonstrated excellent value for their price. It's based on actual human testing and evaluation. If you disagree with it -- and trust me, many often do -- you can hit me or my colleagues up in the comment section, on social media or via email. So maybe the thought of an algorithmically-based "award" like Amazon's Choice hits so close to home because it's like a robot coming for my job.

Yes, the algorithm in question is formulated and employed by humans. And yes, human Amazon customers are providing feedback through their reviews and comments that factor into the equation. (I'll spare you the side discussions of fake Amazon reviews and fake Amazon products.)

But so what? If a human deploys an army of software bots to exploit algorithms on Twitter, who's at fault if the resulting fake news is retweeted ad infinitum? Who's at fault if the Amazon algorithm makes a counterfeit product an Amazon's Choice?

If this all sounds like doublespeak, it probably is. However, Amazon continues to decline to detail the process behind Amazon's Choice, so we'll never know how a preorder product like the Jabra Elite 65t -- or any product, really -- ends up with an Amazon's Choice badge.

But I will say this. While whatever entity made the pick may not always get its picks right, it did get that one right. Or at least this human thinks so. Which probably isn't saying much any more.