webtrail Brian · My field notebook for trails and the web
arxiv.org
Screenshot of an arXiv abstract page under the breadcrumb 'Computer Science > Software Engineering', showing the paper title 'Same Scrutiny, More Time: Eye Tracking Insights into Reviewing LLM-Labelled Code', its five authors, the full abstract, a comments line noting acceptance at ASE 2026, and an 'Access Paper' sidebar linking to PDF, HTML and TeX source.

AI and Developer Psychology

Eye-Tracking Study of LLM-Labelled Code Review on arXiv

ai generated code code review developer trust eye tracking

arXiv hosts the preprint of "Same Scrutiny, More Time", an eye-tracking study of what software engineers actually do when the code in front of them is labelled as LLM-generated. Most AI-policy arguments settle that by assertion; this one measures it.

The abstract page sets out the design plainly. Ranim Khojah and four co-authors ran a Wizard-of-Oz experiment in which participants reviewed code explicitly labelled as LLM-generated, pairing eye-tracking with exit interviews so observed behaviour could be set against what people said they were doing, then analysed with Bayesian methods alongside qualitative coding. The headline result is narrower than the usual framing, and worth getting right: review thoroughness did not change, but reviewers spent more time fixating on the labelled code. The label moves attention, not rigour. The authors read that as a gap between reviewers' intentions and their actual reviewing behaviour.

The rest is about strategy. Participants adapted how they handled labelled code, judging it against specific criteria such as logical correctness, or using the originating prompt to steer the review, which is why the paper argues the prompt deserves treating as a software artifact in its own right.

One limit before you click through: this is the abstract page, so you get the abstract, the submission history and links to PDF, experimental HTML and TeX source, while sample size and effect sizes stay in the paper. It is a preprint, accepted at the 41st IEEE/ACM Automated Software Engineering conference for 2026.

Walked on

← All stops