Amazon building computer vision tech to check defects in fruits, vegetables

The e-commerce giant's technology can carry out millions of assessments to grade the quality of fruits and vegetables per day at a cost that is far below compared to any other method

E-commerce giant Amazon is building a computer vision-based grading solution for food products such as onions and tomatoes. The machine learning-based approach analyses produce images to detect defects such as cuts, cracks and pressure damage. It can carry out millions of assessments per day at a cost that is far below compared to any other method, said a top Amazon scientist.

“Quality is one of the key drivers of fruit and vegetable purchasing decisions and a critical factor in achieving customer satisfaction,” said Rajeev Rastogi, vice president, machine learning, Amazon India at the company’s Smbhav event. “Having humans grade the quality of fruits and vegetables by manually examining each individual piece of produce like tomato or onion is not scalable to millions of quality assessments per day.”

Amazon plans to develop a conveyor belt based automatic grading and packing machine. It would leverage hardware and machine learning to pack produce into predetermined quality grades such as premium-grade A. The gradient pack machine will reduce grading cost by 78 per cent compared to manual grading.

Amazon also plans to use near-infrared sensors to detect attributes such as sweetness and ripeness. These cannot be detected in RGB (red, green, blue) images captured by traditional computer vision algorithms and require destructive methods such as eating the fruit.

“That can't obviously scale,” said Rastogi who began his career at Bell Labs. He also served as the vice president of Yahoo Labs, where his team developed data-extraction algorithms to pull structured information from billions of web pages, and then present them to users in easily digestible ways.

He joined Amazon in 2012. His first Amazon project involved the development of algorithms to classify products into Amazon’s large and complex taxonomical structure. For example, to classify a Samsonite luggage set in ‘carry-on luggage,’ ‘suitcases’ and ‘luggage sets.’ Since then, Rastogi has been involved in utilizing science to make an impact in a number of areas that have resulted in faster, more seamless and sustainable, shopping experiences.

As vice president of machine learning at Amazon India, Rastogi is now helping his team drive innovations that have a profound impact not only on shoppers in India but also on the company’s customers around the world.

For instance, the Amazon India team has also developed CRISP mobile app to tackle the spread of the Covid-19 pandemic and provide a safe work environment for our fulfilment centre employees. The CRISP app uses Bluetooth signals on mobile phones to track social contacts between Amazon associates. This social contact data is used to alert associates when they breach social distancing norms, for example, when they come too close to other associates. It is also used to identify users with a high risk of getting infected with Covid-19 since they have directly or indirectly come in contact with associates, who have tested positive for Covid-19. 

“Now the associates with high infection risk scores can be prioritized for testing and quarantine actions,” said Rastogi. “We have launched CRISP across our last mile nodes to help associates maintain appropriate social distancing.”