Semantic Image Retrieval: Learning Gaussian Mixture Models of Semantic Concepts using Expectation-Maximization

Overview

Objectives/Goals The proliferation of digital photos has made efficient automation of image search more challenging than ever. Current image retrieval methods directly compare images by low-level perceptual characteristics such as color and texture. However, humans understand images by describing them with concepts, such as "building," "sky," and "people." This project employs artificial intelligence to identify semantic concepts in images using probabilistic models. No manual annotation of photos is required#images are found using content recognition. Methods/Materials In this research, the visual feature distribution of each concept was modeled with a Gaussian Mixture Model learned using the Expectation-Maximization algorithm. The models captured the dominant colors and textures of the concepts. During retrieval, the models were used to determine the probability of each concept occurring in the images, producing a probability vector called a semantic multinomial (SMN). Images were retrieved by finding the most similar SMNs. Results This research has shown that by linking images with their underlying semantic meaning, the system understood the images and significantly improved retrieval accuracy. It was also shown that a feature set consisting of YCbCr color values and Gabor texture filter responses had consistently better retrieval performance than Discrete Cosine Transform features. A novel dynamic browser was also developed for exploring image collections based on semantic similarity. It uses an animated spring graph layout algorithm to create self-organizing clusters of semantic concepts. This provides a unique way to visualize the inherent semantic relationships in the image database, as well as to search for images by concepts. Conclusions/Discussion This technology has far reaching applications beyond organizing family albums. In the medical profession, the ability to quickly correlate unknown MRI images with that of known medical disorders is now within reach. In the area of natural resource exploration for oil, gas, and coal, remotely-sensed images can now be automatically related to images of known natural reserves.

Awards (1)

• Third Place

Competition history

• CSEF 2009 Mathematics & Software · Entry S1611

Resources

• CSEF project summary ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.

Browse more like this

• All Computational Sciences projects
• CSEF Computational Sciences projects
• Every CSEF 2009 project
• CSEF Third Place winners