Computer science with a dash of linguistics: The interactive cookbook

Alexander Koller, computational linguistics professor at Saarland University

It’s pizza night. You’re standing in the kitchen, hands full of flour, wondering, “How much water needs to go into the dough again?” You don’t really want to touch your cookbook or tablet with dirty fingers to look it up. It would be much nicer to be able to simply ask the question and receive the answer from a computer via voice output. This is exactly the kind of interactive cookbook that Saarland University computational linguistics professor Alexander Koller wants to develop. This involves solving numerous interesting research questions in his field.

The research project “The Interactive Cookbook” will be funded by the German Research Foundation (Deutsche Forschungsgemeinschaft; DFG) starting in November 2021. The goal is to develop a dialog system that interactively guides its user through the preparation of a dish. In doing so, the system will adapt to expert and novice users. The interactive cookbook gets its knowledge about cooking from reading recipe texts.

The interactive cookbook represents a more general category of dialog systems that offer interactive access to information that exists as written text. For example, the research could form the basis for systems that provide building instructions or support the learning of clearly structured processes.

The idea for the interactive cookbook originated in a course at Saarland University. “Students were asked to use a special software called DialogOS to develop a dialog system – that is, a system that can be controlled by voice commands and that responds with spoken output. One group decided to build a cooking assistant,” says Alexander Koller, who has been a professor of computational linguistics at Saarland University since 2016. The student project made him realize that there are numerous fundamental research questions of computational linguistics hidden in the seemingly simple application. With the “Interactive Cookbook”, he now wants to tackle these problems.

“Recipe texts are written in a very idiosyncratic style that is difficult for computers to handle”, Professor Alexander Koller explains. For example, a lot of information and instructions is represented implicitly. “The sentences ‘Prepare vegetables. Cut.’ are easy to understand for humans. However, the instruction “cut” doesn’t say what to cut, so we have to find ways to recognize these semantic connections”, Professor Koller explains. To do this, the scientist uses various methods from the artificial intelligence toolbox, such as machine learning with neural networks.

The first step of the project is to develop an algorithm that can automatically parse recipe texts into flowcharts which describe the cooking process. In order for the interactive cookbook to be able to respond individually to different users, the recipes must be analyzed in several levels of abstraction and the results be linked together. While a mainstream cookbook only says “boil potatoes,” a beginner’s cookbook also explains this step in detail. The goal is to develop an algorithm that can automatically aggregate information from different recipes for a dish and understand how to best present it to different users.

The voice output of the interactive cookbook also poses a challenge. The researchers want to ensure that the cookbook does not just literally repeat pieces of the original recipe texts, but generates its own correct utterances. “Well-known methods for natural language generation, such as the algorithm GPT-3, can produce very convincing-looking, beautifully written texts – but in terms of content, they are often deficient. But since recipes are all about precision in content, we have to be careful that the language generation algorithm doesn’t invent things to say just because they sound nice”, Professor Koller explains.

In the final step, the research team plans to evaluate its findings by publishing the interactive cookbook as an app for various smart home devices. Commercialization of the work is not intended.

Questions can be directed at:
Prof. Dr. Alexander Koller
koller@coli.uni-saarland.de
+49 681 302 4345

More Information:
https://gepris.dfg.de/gepris/projekt/461220770
https://www.coli.uni-saarland.de/~koller/
https://www.uni-saarland.de/fachrichtung/lst.html

Background Saarland Informatics Campus:
800 scientists and about 2100 students from more than 80 nations make the Saarland Informatics Campus (SIC) one of the leading locations for computer science in Germany and Europe. Five world-renowned research institutes, namely the German Research Center for Artificial Intelligence (DFKI), the Max Planck Institute for Computer Science, the Max Planck Institute for Software Systems, the Center for Bioinformatics and the Cluster for “Multimodal Computing and Interaction” as well as Saarland University with three departments and 21 degree programs cover the entire spectrum of computer science.

Editor:
Philipp Zapf-Schramm
Competence Center Computer Science
Saarland Informatics Campus
Phone: +49 681 302-70741
E-Mail: pzapf@mmci.uni-saarland.de