Google taps U for smartphone research

Students and faculty are working on new 3D sensors for mobile phones.

PhD student Ryan Dutoit shows off the three-dimensional mapping capabilities of a Google

Cole Feagler

PhD student Ryan Dutoit shows off the three-dimensional mapping capabilities of a Google “Project Tango” prototype smartphone. Running Google’s Android operating system, it behaves much like a typical device. But, with a pair of cameras and its proprietary depth sensor, the device can create 3D maps of its surroundings and track its position in the real world. ***NOTE: The U has signed an NDA with google preventing them from giving tech specs about the device. All tech info was pulled from Google’s own publications about the project. Is it appropriate to include a small note to this effect (so that it doesn’t appear that the U gave the information in breach of its contract?)

Nicolas Hallett

A University of Minnesota research team is playing a major role in Google’s newest attempt to revolutionize cellphones.

In the same way that Google Maps changed how users navigate roads, the tech giant’s newest innovation, known as Project Tango, will attempt to alter how people maneuver inside buildings. Google selected the University as a research partner for the project, which plans to bring its previously developed 3D technology to mobile phones.

The College of Science and Engineering’s Multiple Autonomous Robotic Systems, or MARS, Laboratory is developing and testing the technology known as “vision inertial navigation” in the basement of Walter Library. Programmed with the right algorithms, a camera and a computing system can take motion sensor data and turn it into position estimates or maps.

The MARS lab is led by Stergios Roumeliotis, who used 3D mapping technology to give the U.S. Air Force mini autonomous air vehicles and to help NASA land a spacecraft on Mars.

“It is one of the coolest things that I can think of,” said Roumeliotis, who joined the University from the California Institute of Technology in 2002. “It’s basically something I have been working on for my whole life, and so it’s now coming to fruition, and I’m really glad to see that.”

Google gave the University $1.35 million in grant funding and five Project Tango prototype phones, which can take a quarter-million 3D measurements every second. Roumeliotis and his team couldn’t comment on the release data or cost of the phone because they signed non-disclosure agreements.

As a software expert, Roumeliotis said he was unsure how long it would take to mass-produce the 3D-sensing phone. From their end, he said, the basic technology should be ready for consumers within a year.

Google asked the MARS team to perfect the mapping technology, smooth out any algorithmic kinks and prepare it for the use of millions.

First, the team had to do a lot of work to “squeeze” its mathematic systems into the phone’s computer and make them work with far cheaper equipment, Roumeliotis said. The same sensor equipment costs $1 million on a submarine, $10,000 in Army-grade missiles and $3,000 in the Air Force vehicles, he said. The Google phone sensors will cost about $1.

People also pose a problem because Roumeliotis and his team are accustomed to applying the mapping technology to robots, which he said are more predictable.

Google has asked the MARS team to solve many “high-risk” problems, or glitches, which take months to examine but often offer little progress on the project as a whole.

“Sometimes we touch problems that no one has seen before,” fourth-year computer science graduate student and MARS team member Dimitrios Kottas said. “So many times we have to go back to the whiteboard.”

Taking the lead

Two years ago, the MARS lab sent Kottas and then-University graduate student Joel Hesch to present the 3D technology at a national conference.

They presented a live demo of the navigation system on a small laptop with a camera. Before then, Roumeliotis said no one had shown off the technology on such a small device.

Google took notice. Technical Program Lead Johnny Lee visited the University in the summer of 2012, when he quickly hired Hesch and offered to begin collaborating formally.

The company later hired another University graduate, Esha Nerurkar, who is now a technical lead for Project Tango along with Hesch.

“The roles are reversed because I used to be the adviser to Joel and Esha and now they’re my boss,” Roumeliotis said.

Hesch said he’s doing much of the same work but from Google’s headquarters in Mountain View, Calif.

“It’s been great,” said Hesch, who spent 10 years at the University. “I really enjoyed working with the MARS lab. Both from a sense of quality of research and just friendliness, it’s been a really great experience.”

‘We’re close’

Google has been sending the phone to professional developers in order to spur services and products to start using Project Tango.

Roumeliotis said he and the students have been working around the clock to advance technology that could make the practical and the extravagant possible.

Kottas said the technology has endless possibilities, like allowing furniture shoppers to place couches virtually in their living rooms or letting game developers use a player’s basement as a level.

“You could use any environment as the level of the game,” Kottas said. “Imagine playing Counter-Strike in your house.”

MARS team member and first-year computer science graduate student Ryan DuToit said their ultimate goal is to use the phone to map the Mall of America.

“There’s no one else in the world who can claim they can three-dimensionally map an entire building,” DuToit said. “We’re close.”