Now, sketch your way to better online shopping

A team of researchers has made buying goods on the Internet even better.

A computer program that recognises sketches pioneered by scientists from Queen Mary University of London (QMUL) could help consumers shop more efficiently.

The sketches of a pair of shoes or piece of furniture, for example, are drawn directly by hand on a touchscreen and recognised using a sophisticated image retrieval system, where the top 10 retrieval accuracy is close to 100 per cent on some object categories so that it always displays the desired product on the first page.

"What's great about our system is that the user doesn't have to be an artist for the sketch to be accurate, yet is able to retrieve images in a more precise manner than text," said co-developer Dr Yi-Zhe Song, adding, "With the proliferation of touchscreens, sketching had become a much easier to do and in some ways is actually preferable to text-based or photo searches."

Fine-grained sketch-based image retrieval (SBIR) overcomes problems with using words to describe visual objects in words, especially when dealing with precise details, and with using photos, which can restrict the search far too narrowly.

Co-developer Dr Timothy Hospedales noted, "Current image search capabilities provided by online retailers only perform category-level searches so for customers it is still a tedious 'browsing' exercise to find your ideal item. Now you can simply sketch your mental image of your ideal product and go straight to it."

The computer program system is designed to emulate the human brain's processing through arrays of simulated neurons. It was trained to match sketches to photos based on about 30,000 sketch-photo comparisons, learning how to interpret salient details of photos and how people try to depict them in hand-drawing.

The research was accepted at International Conference on Computer Vision and Pattern Recognition (CVPR) and will be presented at the end of June.