LIN-SHAN LEE: Programming pioneer

A computer scientist seeks to create a spoken version of Google.

Lin-shan Lee spends a lot of time exploring online courses, but he isn’t looking to learn a language or study ancient history. Instead, he’s training algorithms to extract key words and phrases from audio and video recordings.

He and many other computer scientists say that this is one of the crucial steps towards unlocking a vast amount of knowledge that is currently difficult to search and organize. “My whole career has been motivated by identifying interesting problems which can have great impact if solved, but are very difficult，” he says.

Lee, a computer engineer at the National Taiwan University in Taipei, trained as an electrical engineer, completing his PhD at Stanford University in California in the 1970s. In the United States, he also worked on satellites used for telephone communication. “When I decided to return home, all my friends said， ‘you are stupid because there’s no work on satellites in Taiwan’，” he says.

But Lee felt that his familiarity with voice signals presented an opportunity: he traded satellites for computers, and started developing a tool that could recognize spoken Mandarin. Over more than a decade, his team built a device that was crude and slow — it took five to six seconds to recognize a single syllable — but a huge achievement. It was the first speech-recognition device for Mandarin. By 1995， their machine could transform continuous Mandarin speech into a series of Chinese characters.

“Speech recognition is a very hard problem，” says Jim Glass, a researcher in speech processing at the Massachusetts Institute of Technology in Cambridge. Everyone in the world speaks slightly differently, in a continuous stream, often against a noisy background, he says. “Somehow we have to extract information from that，” says Glass. “Lin-Shan is a pioneer in Mandarin spoken-language processing.”

Lee is now building on that success to tackle another speech-recognition challenge: retrieving spoken content from audio and video files. His team is using machine learning to build a system in which users can search for specific words, phrases or sentences within a video. “I call this a spoken version of Google，” he says. It would revolutionize multimedia sites such as YouTube. Currently, there is no way to locate a specific sentence within the video without going through the entire clip. Although there are some existing technologies that can process speech in multimedia, they are not very accurate, says Lee. “It is my research goal at the moment to develop more-accurate technology for this.” — by Nicky Phillips