Kivy - Applying Deep Recipe Embeddings for Multimodal Nutritional Analysis Through a System for Augmented Surface Interaction

Overview

Culinary recipes are complex, semi-structured objects that combine a mix of language, nutrients, steps, and much more to define how a food item should be prepared. Since this can be difficult for computers to work with, we developed a better way for computers to numerically represent and understand food recipes. Culinary recipes were gathered from different sources to create a well structured dataset of over 500,000 entries. These recipes were enriched with nutritional information and other metadata from trusted sources. An encoder AI model was created that can represent the recipes from the dataset in a continuous latent space, which was trained on the dataset. To validate the model, two tasks were evaluated: vector arithmetic (cosine similarity), and ingredient recommendation. To demonstrate real-world utility, the model was deployed into a custom computer-vision system that projects an interactive interface onto kitchen surfaces along with a mobile app, allowing users to query the AI model during cooking. The learned embeddings enabled cosine similarity comparisons and vector operations between recipes. Ingredient recommendation also showed promising results, as the model was able to predict a removed ingredient in most cases. This study showed that culinary recipes can be represented in a continuous latent space which can be used for many downstream tasks. These results suggest potential applications in intelligent cooking assistance and computational food analysis.

Competition history

• ISEF 2026 Software Design · Entry SFTD006T

Resources

• ISEF abstract page ↗

Related projects

Closest projects by meaning, across every fair and year in the corpus.