Anthropic published a randomized controlled trial in January 2026 that asks what most tooling debates skip: not whether an AI assistant makes you faster today, but whether you still understand the code afterwards.

Judy Hanwen Shen and Alex Tamkin gave 52 mostly junior software engineers the same job — learn Trio, a Python library built around asynchronous programming — with or without AI assistance. On a later comprehension quiz the AI group averaged 50% against 67% for the hand-coding group, a gap the page puts at nearly two letter grades (Cohen's d=0.738, p=0.01). The largest difference showed up on debugging questions. The AI group finished about two minutes faster, and that difference was not statistically significant.

The more useful finding is that how you used the assistant changed the outcome. Engineers who scored well asked follow-up questions, requested explanations and posed conceptual questions rather than only generating code; the weaker scores clustered around heavy delegation of both code generation and debugging.

Worth stating plainly: this is Anthropic's own research into the category of tool it sells, and the finding runs against its commercial interest. The limitation is scope — one library, one sitting, 52 people — so read it as a signal about how skills form, not a verdict on your daily work.