AAAI Publications, Twenty-Second International Joint Conference on Artificial Intelligence

Font Size: 
Learning Bilingual Lexicons Using the Visual Similarity of Labeled Web Images
Shane Bergsma, Benjamin Van Durme

Last modified: 2011-06-28


Speakers of many different languages use the Internet. A common activity among these users is uploading images and associating these images with words (in their own language) as captions, filenames, or surrounding text. We use these explicit, monolingual, image-to-word connections to successfully learn implicit, bilingual, word-to-word translations. Bilingual pairs of words are proposed as translations if their corresponding images have similar visual features. We generate bilingual lexicons in 15 language pairs, focusing on words that have been automatically identified as physical objects. The use of visual similarity substantially improves performance over standard approaches based on string similarity: for generated lexicons with 1000 translations, including visual information leads to an absolute improvement in accuracy of 8-12% over string edit distance alone.

Full Text: PDF