It has been over 50 years since the invention of the barcode. It is widely used and nearly faultless. But that isn’t quite good enough for Amazon.

When an item enters an Amazon fulfillment center, staff members utilize barcodes to identify it at several points before it is transferred to a delivery vehicle. The problem is, every time, the items must be physically picked up in order to find and scan the barcode. Worse still, the barcode may occasionally be broken or completely absent.

Amazon has a vast collection of products in all sizes and forms, for which this process has to be repeated. Unfortunately, the process is difficult to automate. There is also currently no robot that can alter any item that enters a warehouse before being scanned.

For Amazon, the solution lies in  Improving or doing away with the barcode. Doing away with the cumbersome and ineffective manual item identification completely is a better idea. 

To achieve that, Amazon is using multimodal identification or MMID. To automate the identification, this technique leverages a variety of information modalities, such as extracting an item’s appearance and dimensions from its image.

Robots are not able to locate barcodes

While robots may be the way of the future, it appears that they are not very adept at exploiting a tried-and-true piece of technology: the barcode.  According to Amazon, robots aren’t particularly good at locating barcodes. The barcodes can be less discoverable or attached to unusually shaped objects.

To help robots ship your orders more quickly, Amazon has created a computer vision algorithm from scratch to recognize objects without barcodes.

As a result, Amazon has chosen to do away with the barcodes.

The camera system will work with robots

The e-commerce company has created a camera system that can record items traveling down conveyor belts one after the other, and ensure the items match their images in the Amazon warehouses.  Amazon’s AI specialists and roboticists hope to eventually integrate the technology with robots that can recognize objects while picking them up and rotating them around.

Nontas Antonakos, an applied science manager at Amazon’s computer vision group in Berlin, emphasized the importance of finding a solution to this issue so that robots can pick up products and process them without first needing to locate and scan a barcode. “It will make it easier for us to precisely and promptly deliver items to customers,” he said.

The camera system is already in use and improving parcel delivery

The technology, dubbed multi-modal identification, won’t completely replace barcodes any time soon. However, it’s likely that a future Whole Foods or another Amazon-owned chain with physical locations may use the technology since all of Amazon’s businesses will eventually embrace it. 

According to Amazon, the technology is now in use in facilities in Hamburg, Germany, and Barcelona, Spain. However, the business claims that it has already sped up the processing time for parcels there. 

The company notes that the issue that the technology solves—customers receiving incorrect items—doesn’t occur very frequently. But when you consider how many things a single warehouse processes in a day, even isolated errors build up to considerable slowdowns.

The accuracy of the new camera system

Since the business hadn’t had a need to capture product photos before this initiative, Amazon’s AI experts had to start by compiling a photo library.  The images themselves as well as information on the dimensions of the goods supplied the early versions of the algorithm. The cameras are constantly taking pictures of fresh images to train the model with.

In the beginning, the algorithm had an accuracy rate ranging from 75% to 80%, which Amazon thought was a good beginning. When the system first tried to detect color changes, it had trouble. The system was unable to distinguish between two distinct hues of Echo Dots during a Prime Day offer.  According to the business, accuracy is currently 99%. 

The system can now attribute confidence scores to its evaluations that only flag items that it is absolutely certain are erroneous.

The ultimate goal is to have robots handle products. According to Amazon’s AI, it will be difficult to fine-tune the multi-modal recognition system to evaluate products that humans handle.