{"id":24487,"date":"2025-11-27T16:21:46","date_gmt":"2025-11-27T15:21:46","guid":{"rendered":"https:\/\/saarland-informatics-campus.de\/?post_type=sic_news&p=24487"},"modified":"2025-12-08T13:34:11","modified_gmt":"2025-12-08T12:34:11","slug":"humans-and-large-language-models-respond-surprisingly-similarly-to-confusing-program-code","status":"publish","type":"sic_news","link":"https:\/\/saarland-informatics-campus.de\/en\/piece-of-news\/humans-and-large-language-models-respond-surprisingly-similarly-to-confusing-program-code\/","title":{"rendered":"Humans and large language models respond surprisingly similarly to confusing program code"},"content":{"rendered":"
\n

Researchers from Saarland University and the Max Planck Institute for Software Systems have, for the first time, shown that the reactions of humans and large language models (LLMs) to complex or misleading program code significantly align, by comparing brain activity of study participants with model uncertainty. Building on this, the team developed a data-driven method to automatically detect such confusing areas in code \u2014 a promising step toward better AI assistants for software development.<\/p>\n<\/div>\n

\n

The team led by Sven Apel, Professor of\u00a0Software Engineering<\/i>\u00a0at Saarland University, and Dr. Mariya Toneva, researcher at the Max Planck Institute for Software Systems, investigated how humans and large language models respond to confusing program code. The characteristics of such code, known as\u00a0atoms of confusion<\/i>, are well studied: They are short, syntactically correct programming patterns that are misleading for humans and can throw even experienced developers off track.<\/p>\n

To find out whether LLMs and humans \u201cthink\u201d about the same stumbling blocks, the research team used an interdisciplinary approach: On the one hand, they used data from an earlier study by Apel and colleagues, in which participants read confusing and clean code variants while their brain activity and attention were measured using electroencephalography (EEG) and eye tracking. On the other hand, they analyzed the \u201cconfusion\u201d or model uncertainty of LLMs using so-called\u00a0perplexity<\/i>\u00a0values. Perplexity is an established metric for evaluating language models by quantifying their uncertainty in predicting sequences of text tokens based on their probability.<\/p>\n

The result: Wherever humans got stuck on code, the LLM also showed increased perplexity. EEG signals from participants\u2014especially the so-called\u00a0late frontal positivity<\/i>, which in language research is associated with unexpected sentence endings\u2014rose precisely where the language model\u2019s uncertainty spiked. \u201cWe were astounded that the peaks in brain activity and model uncertainty showed significant correlations,\u201d says Youssef Abdelsalam, who was advised by Toneva and Apel and was instrumental in conducting the study as part of his doctoral studies.<\/p>\n

Based on this similarity, the researchers developed a data-driven method that automatically detects and highlights unclear parts of code. In more than 60 percent of cases, the algorithm successfully identified known, manually annotated confusing patterns in the test code and even discovered more than 150 new, previously unrecognized patterns that also coincided with increased brain activity.<\/p>\n

\u201cWith this work, we\u00a0are taking a step toward a better understanding of the alignment between humans and machines,\u201d says Max Planck researcher Mariya Toneva. \u201cIf we know when and why LLMs and humans stumble in the same places, we can develop tools that make code more understandable and significantly improve human\u2013AI collaboration,\u201d adds Professor Sven Apel.<\/p>\n

Through their project, the researchers are building a bridge between neuroscience, software engineering, and artificial intelligence. The study, currently published as a\u00a0preprint<\/i>, was accepted for publication at the\u00a0International Conference on Software Engineering<\/i>\u00a0(ICSE), one of the world\u2019s leading conferences in the field of software development. The conference will take place in Rio de Janeiro in April 2026. The authors of the study are: Youssef Abdelsalam, Norman Peitek, Anna-Maria Maurer, Mariya Toneva, and Sven Apel.<\/p>\n

Preprint:<\/strong><\/p>\n

Y. Abdelsalam, N. Peitek, A.-M. Maurer, M. Toneva, S. Apel (2025): \u201cHow do Humans and LLMs Process Confusing Code?\u201d<\/i>\u00a0arXiv:2508.18547v1 [cs.SE], August 25, 2025. \u00a0https:\/\/arxiv.org\/abs\/2508.18547<\/a><\/p>\n

Further information:<\/strong><\/p>\n

Chair of Software Engineering:\u00a0https:\/\/www.se.cs.uni-saarland.de<\/a><\/p>\n

Max Planck research group \u201cBridging AI and Neuroscience\u201d<\/i>:\u00a0https:\/\/mtoneva.com\/index.html<\/a><\/p>\n

Scientific contacts:<\/strong><\/p>\n

Prof. Dr. Sven Apel
\nChair of Software Engineering
\nSaarland University
\nTel.: +49 681 302 57211
\nE-mail:\u00a0apel(at)cs.uni-saarland.de<\/a><\/p>\n

Dr. Mariya Toneva
\nHead of the Research Group \u201cBridging AI and Neuroscience\u201d<\/i>
\nMax Planck Institute for Software Systems
\nTel.:\u00a0+49 681 9303 9801<\/a>
\nE-mail: mtoneva@mpi-sws.org<\/p>\n

Editorial contact:<\/strong>
\nPhilipp Zapf-Schramm
\nSaarland Informatics Campus
\nTel: +49 681 9325 4509
\nE-Mail: pzs@mpi-klsb.mpg.de<\/p>\n<\/div>\n

\n
\n
\n
<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

Researchers from Saarland University and the Max Planck Institute for Software Systems have, for the first time, shown that the reactions of humans and large language models (LLMs) to complex or misleading program code significantly align, by comparing brain activity of study participants with model uncertainty. Building on this, the team developed a data-driven method […]<\/p>\n","protected":false},"featured_media":24542,"menu_order":0,"template":"","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":""},"sic_news-types":[59,60],"class_list":["post-24487","sic_news","type-sic_news","status-publish","has-post-thumbnail","hentry","sic_news-types-press-release-en","sic_news-types-uds-en"],"yoast_head":"\nHumans and large language models respond surprisingly similarly to confusing program code - Saarland Informatics Campus<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/saarland-informatics-campus.de\/en\/piece-of-news\/humans-and-large-language-models-respond-surprisingly-similarly-to-confusing-program-code\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Humans and large language models respond surprisingly similarly to confusing program code - Saarland Informatics Campus\" \/>\n<meta property=\"og:description\" content=\"Researchers from Saarland University and the Max Planck Institute for Software Systems have, for the first time, shown that the reactions of humans and large language models (LLMs) to complex or misleading program code significantly align, by comparing brain activity of study participants with model uncertainty. Building on this, the team developed a data-driven method […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/saarland-informatics-campus.de\/en\/piece-of-news\/humans-and-large-language-models-respond-surprisingly-similarly-to-confusing-program-code\/\" \/>\n<meta property=\"og:site_name\" content=\"Saarland Informatics Campus\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/sic.informatik.saarland\/\" \/>\n<meta property=\"article:modified_time\" content=\"2025-12-08T12:34:11+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/saarland-informatics-campus.de\/wp-content\/uploads\/apel-toneva-abdelsalam-2.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1920\" \/>\n\t<meta property=\"og:image:height\" content=\"1080\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/saarland-informatics-campus.de\\\/en\\\/piece-of-news\\\/humans-and-large-language-models-respond-surprisingly-similarly-to-confusing-program-code\\\/\",\"url\":\"https:\\\/\\\/saarland-informatics-campus.de\\\/en\\\/piece-of-news\\\/humans-and-large-language-models-respond-surprisingly-similarly-to-confusing-program-code\\\/\",\"name\":\"Humans and large language models respond surprisingly similarly to confusing program code - Saarland Informatics Campus\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/saarland-informatics-campus.de\\\/en\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/saarland-informatics-campus.de\\\/en\\\/piece-of-news\\\/humans-and-large-language-models-respond-surprisingly-similarly-to-confusing-program-code\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/saarland-informatics-campus.de\\\/en\\\/piece-of-news\\\/humans-and-large-language-models-respond-surprisingly-similarly-to-confusing-program-code\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/saarland-informatics-campus.de\\\/wp-content\\\/uploads\\\/apel-toneva-abdelsalam-2-e1765197559850.jpg\",\"datePublished\":\"2025-11-27T15:21:46+00:00\",\"dateModified\":\"2025-12-08T12:34:11+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/saarland-informatics-campus.de\\\/en\\\/piece-of-news\\\/humans-and-large-language-models-respond-surprisingly-similarly-to-confusing-program-code\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/saarland-informatics-campus.de\\\/en\\\/piece-of-news\\\/humans-and-large-language-models-respond-surprisingly-similarly-to-confusing-program-code\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/saarland-informatics-campus.de\\\/en\\\/piece-of-news\\\/humans-and-large-language-models-respond-surprisingly-similarly-to-confusing-program-code\\\/#primaryimage\",\"url\":\"https:\\\/\\\/saarland-informatics-campus.de\\\/wp-content\\\/uploads\\\/apel-toneva-abdelsalam-2-e1765197559850.jpg\",\"contentUrl\":\"https:\\\/\\\/saarland-informatics-campus.de\\\/wp-content\\\/uploads\\\/apel-toneva-abdelsalam-2-e1765197559850.jpg\",\"width\":1920,\"height\":1080,\"caption\":\"Left to right: Youssef Abdelsalam, Sven Apel (\u00a9 UdS\\\/Oliver Dietze) and Mariya Toneva.\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/saarland-informatics-campus.de\\\/en\\\/piece-of-news\\\/humans-and-large-language-models-respond-surprisingly-similarly-to-confusing-program-code\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/saarland-informatics-campus.de\\\/en\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"News\",\"item\":\"https:\\\/\\\/saarland-informatics-campus.de\\\/en\\\/piece-of-news\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Humans and large language models respond surprisingly similarly to confusing program code\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/saarland-informatics-campus.de\\\/en\\\/#website\",\"url\":\"https:\\\/\\\/saarland-informatics-campus.de\\\/en\\\/\",\"name\":\"Saarland Informatics Campus\",\"description\":\"in Saarbr\u00fccken, Germany\",\"publisher\":{\"@id\":\"https:\\\/\\\/saarland-informatics-campus.de\\\/en\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/saarland-informatics-campus.de\\\/en\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/saarland-informatics-campus.de\\\/en\\\/#organization\",\"name\":\"Saarland Informatics Campus\",\"url\":\"https:\\\/\\\/saarland-informatics-campus.de\\\/en\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/saarland-informatics-campus.de\\\/en\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/saarland-informatics-campus.de\\\/wp-content\\\/uploads\\\/SIC_logo_1453x343.png\",\"contentUrl\":\"https:\\\/\\\/saarland-informatics-campus.de\\\/wp-content\\\/uploads\\\/SIC_logo_1453x343.png\",\"width\":1453,\"height\":343,\"caption\":\"Saarland Informatics Campus\"},\"image\":{\"@id\":\"https:\\\/\\\/saarland-informatics-campus.de\\\/en\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/sic.informatik.saarland\\\/\",\"https:\\\/\\\/bsky.app\\\/profile\\\/saarland-informatics-campus.de\",\"https:\\\/\\\/www.instagram.com\\\/saarland_informatics_campus\\\/\",\"https:\\\/\\\/www.youtube.com\\\/channel\\\/UCaDDK5m9CGCzE4xhZTFC6cA\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/saarland-informatics-campus\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Humans and large language models respond surprisingly similarly to confusing program code - Saarland Informatics Campus","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/saarland-informatics-campus.de\/en\/piece-of-news\/humans-and-large-language-models-respond-surprisingly-similarly-to-confusing-program-code\/","og_locale":"en_US","og_type":"article","og_title":"Humans and large language models respond surprisingly similarly to confusing program code - Saarland Informatics Campus","og_description":"Researchers from Saarland University and the Max Planck Institute for Software Systems have, for the first time, shown that the reactions of humans and large language models (LLMs) to complex or misleading program code significantly align, by comparing brain activity of study participants with model uncertainty. Building on this, the team developed a data-driven method […]","og_url":"https:\/\/saarland-informatics-campus.de\/en\/piece-of-news\/humans-and-large-language-models-respond-surprisingly-similarly-to-confusing-program-code\/","og_site_name":"Saarland Informatics Campus","article_publisher":"https:\/\/www.facebook.com\/sic.informatik.saarland\/","article_modified_time":"2025-12-08T12:34:11+00:00","og_image":[{"width":1920,"height":1080,"url":"https:\/\/saarland-informatics-campus.de\/wp-content\/uploads\/apel-toneva-abdelsalam-2.jpg","type":"image\/jpeg"}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/saarland-informatics-campus.de\/en\/piece-of-news\/humans-and-large-language-models-respond-surprisingly-similarly-to-confusing-program-code\/","url":"https:\/\/saarland-informatics-campus.de\/en\/piece-of-news\/humans-and-large-language-models-respond-surprisingly-similarly-to-confusing-program-code\/","name":"Humans and large language models respond surprisingly similarly to confusing program code - Saarland Informatics Campus","isPartOf":{"@id":"https:\/\/saarland-informatics-campus.de\/en\/#website"},"primaryImageOfPage":{"@id":"https:\/\/saarland-informatics-campus.de\/en\/piece-of-news\/humans-and-large-language-models-respond-surprisingly-similarly-to-confusing-program-code\/#primaryimage"},"image":{"@id":"https:\/\/saarland-informatics-campus.de\/en\/piece-of-news\/humans-and-large-language-models-respond-surprisingly-similarly-to-confusing-program-code\/#primaryimage"},"thumbnailUrl":"https:\/\/saarland-informatics-campus.de\/wp-content\/uploads\/apel-toneva-abdelsalam-2-e1765197559850.jpg","datePublished":"2025-11-27T15:21:46+00:00","dateModified":"2025-12-08T12:34:11+00:00","breadcrumb":{"@id":"https:\/\/saarland-informatics-campus.de\/en\/piece-of-news\/humans-and-large-language-models-respond-surprisingly-similarly-to-confusing-program-code\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/saarland-informatics-campus.de\/en\/piece-of-news\/humans-and-large-language-models-respond-surprisingly-similarly-to-confusing-program-code\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/saarland-informatics-campus.de\/en\/piece-of-news\/humans-and-large-language-models-respond-surprisingly-similarly-to-confusing-program-code\/#primaryimage","url":"https:\/\/saarland-informatics-campus.de\/wp-content\/uploads\/apel-toneva-abdelsalam-2-e1765197559850.jpg","contentUrl":"https:\/\/saarland-informatics-campus.de\/wp-content\/uploads\/apel-toneva-abdelsalam-2-e1765197559850.jpg","width":1920,"height":1080,"caption":"Left to right: Youssef Abdelsalam, Sven Apel (\u00a9 UdS\/Oliver Dietze) and Mariya Toneva."},{"@type":"BreadcrumbList","@id":"https:\/\/saarland-informatics-campus.de\/en\/piece-of-news\/humans-and-large-language-models-respond-surprisingly-similarly-to-confusing-program-code\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/saarland-informatics-campus.de\/en\/"},{"@type":"ListItem","position":2,"name":"News","item":"https:\/\/saarland-informatics-campus.de\/en\/piece-of-news\/"},{"@type":"ListItem","position":3,"name":"Humans and large language models respond surprisingly similarly to confusing program code"}]},{"@type":"WebSite","@id":"https:\/\/saarland-informatics-campus.de\/en\/#website","url":"https:\/\/saarland-informatics-campus.de\/en\/","name":"Saarland Informatics Campus","description":"in Saarbr\u00fccken, Germany","publisher":{"@id":"https:\/\/saarland-informatics-campus.de\/en\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/saarland-informatics-campus.de\/en\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/saarland-informatics-campus.de\/en\/#organization","name":"Saarland Informatics Campus","url":"https:\/\/saarland-informatics-campus.de\/en\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/saarland-informatics-campus.de\/en\/#\/schema\/logo\/image\/","url":"https:\/\/saarland-informatics-campus.de\/wp-content\/uploads\/SIC_logo_1453x343.png","contentUrl":"https:\/\/saarland-informatics-campus.de\/wp-content\/uploads\/SIC_logo_1453x343.png","width":1453,"height":343,"caption":"Saarland Informatics Campus"},"image":{"@id":"https:\/\/saarland-informatics-campus.de\/en\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/sic.informatik.saarland\/","https:\/\/bsky.app\/profile\/saarland-informatics-campus.de","https:\/\/www.instagram.com\/saarland_informatics_campus\/","https:\/\/www.youtube.com\/channel\/UCaDDK5m9CGCzE4xhZTFC6cA","https:\/\/www.linkedin.com\/company\/saarland-informatics-campus"]}]}},"_links":{"self":[{"href":"https:\/\/saarland-informatics-campus.de\/en\/wp-json\/wp\/v2\/sic_news\/24487","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/saarland-informatics-campus.de\/en\/wp-json\/wp\/v2\/sic_news"}],"about":[{"href":"https:\/\/saarland-informatics-campus.de\/en\/wp-json\/wp\/v2\/types\/sic_news"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/saarland-informatics-campus.de\/en\/wp-json\/wp\/v2\/media\/24542"}],"wp:attachment":[{"href":"https:\/\/saarland-informatics-campus.de\/en\/wp-json\/wp\/v2\/media?parent=24487"}],"wp:term":[{"taxonomy":"sic_news-types","embeddable":true,"href":"https:\/\/saarland-informatics-campus.de\/en\/wp-json\/wp\/v2\/sic_news-types?post=24487"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}