School of Computer Science PGR Seminar Guest Speaker Nickil Maveli

You are warmly invited to a special PGR seminar with guest speaker, Nickil Maveli, who will be joining us from our neighbouring university, the University of Edinburgh. Following his talk, Nickil will be available in the coffee area for approximately one hour. Please feel free to stop by for a chat.

Nickil Maveli will present Evaluating Large Language Models for Code Understanding and Reasoning

Abstract: Large language model (LLM) coding assistants help millions of developers, but passing a handful of test cases is not the same as understanding code. How much of what these models do reflects reasoning about program behaviour, and how much reflects recall of familiar patterns? Three diagnostic studies expose weaknesses that standard benchmarks miss. First, functional equivalence: when programs are rewritten with small meaning-preserving or meaning-altering edits, code LLMs perform on par with string-matching metrics like BLEU, and a single flipped operator can fool them. Second, invertibility: using lossless compression algorithms, we find models often solve encoding and decoding in isolation yet fail the full round trip. Neither fine-tuning on execution traces nor self-reflection closes this gap. Third, algorithmic recall: across 77 classic algorithms in seven languages, accuracy depends on programming language and input format rather than model size, and stronger models fail mostly through runnable code that silently returns the wrong answer. These results suggest current code LLMs succeed on familiar surface forms but lack a robust model of program behaviour. I will close with implications for building reliable coding assistants and for evaluations that measure understanding rather than output correctness.

Short bio: Nickil Maveli is a final-year PhD student in the UKRI Centre for Doctoral Training in Natural Language Processing at the University of Edinburgh, co-supervised by Antonio Vergari and Shay Cohen. His research evaluates how LLMs understand and reason about code, with a focus on functional equivalence, invertibility, algorithmic recall and generalisation. During his PhD, he interned at Huawei Edinburgh Research, working on knowledge-graph-augmented code generation for mathematical reasoning.