Syncing one's phone and computer on the move

Have you ever found yourself looking up directions on your computer before stepping outdoor, only to end up retyping the same addresses and mapping the same route on your phone minutes later?

A new system designed by Tsung-Hsiang Chang, a graduate student in the Massachusetts Institute of Technology's (MIT) Computer Science and Artificial Intelligence Laboratory, and Google’s Yang Li makes it much easier to transfer computing operations between devices. One can simply take a photo of his computer screen with a smartphone camera, and the phone automatically opens up the corresponding application in the corresponding state. The reverse — moving data from the phone to a desktop computer — is also possible.

“People are used to using heavy tools to transfer data or synchronise two devices,” says Chang. “You have to plug in a USB (universal serial bus) cable and maybe open iTunes and synchronise a bunch of data at the same time. But, sometimes, you just want to send a tiny bit of information or a single piece of information.” Since Chang was at Google when he worked on Deep Shot, Google owns the rights to it. The company has not yet made the system publicly available, but when it does, Chang would be among the first to install it.

The Deep Shot system, according to an MIT press release, exploits the fact that many web applications use a standard format called the uniform resource identifier (URI) to describe the states they’re in. For instance, if you search for directions on Google Maps and click the link that says 'Link', a window saying 'Paste link in email or IM' pops up. The link consists of a long string of symbols, including the addresses of the starting and ending points and codes that indicate their geographical coordinates and the approximate size of the map window. That is a URI. The data contained in a URI can vary widely and URIs are a common feature of many web applications — and sometimes harder to extract than they are with Google Maps.

Deep Shot requires installing some minimal software on the phone and on all the computers with which the phone would interact. One might, for instance, want to use the same phone in conjunction with a home desktop, a home laptop, a work desktop and the computers of one's spouse or children. When it is uploading data to a phone, Deep Shot uses the existing computer vision algorithms to identify the application open on screen. The software installed in the computer then extracts and transmits the corresponding URI. Changing the framing of the photograph would also change the framing of the application window that opens on the phone. Conversely, when Deep Shot downloads data from the phone, the computer vision algorithm identifies the computer that the camera is trained on.

Currently, the system works with several common web applications such as Google Maps and Yelp, and with minimal additional coding, it could be made to work with any application that reveals its state through URIs. In principle, the system could also work with off-the-shelf software. This would only require software developers to make some minimal modifications to their codes.