A randomised trial of a generative AI tutor built into university courses found it made results worse, not better, according to a new working paper by Jing Liu and colleagues. The study covered 2,379 undergraduates and 30 instructors across several subjects at a large US public university.
Within sections of the same course, students given access to the tutor had final grades 0.37 standard deviations lower, and their participation in the online course system fell by 0.90 standard deviations, the authors report. Losses were larger for first-generation students, those whose parents did not attend university.
The researchers suggest the tutor may have replaced students’ engagement with existing course activities without improving learning. The paper has not yet been peer-reviewed. It is one of the largest randomised tests of a campus AI tutor so far, and it points the other way from smaller trials with positive results.