Researchers noted that students also had access to other private AI tools.
Photo illustration by Justin Morrison/Inside Higher Ed | PhonlamaiPhoto/iStock/Getty Images | pixdeluxe/E+/Getty Images
Before rolling out its AI-powered study assistant to students campuswide, officials at the University of Maryland wanted to know how the tool would influence learning outcomes. But the results of that study have only raised more questions. And the university is still holding off on making the tool more widely available.
“We don’t have clear findings from the study. It raises questions that haven’t yet been answered,” Jennifer King Rice, senior vice president and provost of UMD, told Inside Higher Ed. “Rather than simply turning this on, we need to better understand what’s needed to help support the effective deployment of AI tools like this.”
The UMD study, which was published as a working paper by the Annenberg EdExchange last week, highlights the limitations of researching the direct effects of generative AI on learning outcomes. Still, the researchers say the study is valuable because it offers some insights that can inform future research into how students and faculty are—and aren’t—engaging with university-created chatbots.
Last year, Rice asked a group of education researchers at the university to design a study to gauge the effectiveness and impact of UMD’s Virtual Study Assistant—VSA—a closed generative AI tutor built on the foundations of OpenAI’s ChatGPT. It connects to the university’s learning management system and is available to answer student questions around the clock, primarily drawing content directly from each course’s online materials, such as text documents, slides and video transcripts.
During the fall 2025 semester, 2,379 undergraduate students and 30 instructors across multiple disciplines at UMD participated in a randomized, controlled trial that compared the course participation levels and grades of VSA users to nonusers.
Although large language models have been mainstream fixtures on college campuses for nearly four years, independent research measuring their impact on teaching and learning outcomes hasn’t kept pace with the speed of the technology’s development and deployment. Critics have called for colleges and universities to gather more research about these tools before adopting them, which is both rare and what the UMD study aimed to do.
While the study showed that VSA access marginally reduced final grades and LMS participation across sections of the same course, only about 15 percent of students in the VSA sample actually chose to use it in the first place. The study notes that students also had access to other AI tools through the university, including ChatGPT and Gemini, and suggests that instructors introducing and encouraging use of the course-specific tutor—which allows instructors to view student chat logs—may have soured students on using it or “influenced what students were willing to ask, limiting comparability with privately used, stand-alone tools.”
But those outside variables that made it difficult to study VSA’s relationship to learning outcomes “reflects the complexity of the world we’re living in right now,” said Jing Liu, director of UMD’s Center for Educational Data Science and Innovation and lead author of the study. “You cannot force people to only use one tool.”
Among the students who did use it, 73.8 percent asked it for information, explanations or solutions compared to 11.3 percent who used it for practice and test preparation; just 0.7 percent of students sought feedback on their own work. But those results also come with a caveat: Few instructors switched VSA to the tutor mode—which, according to the study, “guides students toward a correct response”—from the default mode, which immediately returns the correct answer.
Liu said the latter revelation illuminates a need for university leaders to give faculty more guidance and support in implementing these tools. While faculty were given some “light training” at the outset of the study, “we need to train instructors much more extensively to use these tools,” he said. “Oftentimes we just throw AI tools to students, but instructors play such an important role in helping students to learn about these tools and how to use them effectively.”
Although the study didn’t yield definitive results on learning outcomes, numerous education researchers who weren’t involved commended UMD’s attempts to research its AI tool before deploying it campuswide.
“Every university in the world is conducting an AI experiment this year; we can only learn from the experiments that are systematically investigated,” said Justin Reich, a digital media professor and director of the Teaching Systems Lab at the Massachusetts Institute of Technology. “Randomized trials in the field are a bundle of interventions: They can include factors that researchers intended alongside unintentional changes … Because all of these features in the treatment condition exist in a bundle, we can’t precisely tease out all of the causal mechanisms, but we can try to do some investigations to understand what causes the observed changes.”
The low usage of the UMD’s VSA tool may also speak to market realities that colleges and universities may want to consider as they move forward in the age of AI, added Stephen J. Aguilar, an associate professor of education at the University of Southern California.
“Traditionally, educational institutions are bad at creating bespoke or modified ed-tech tools for their populations. We can’t compete against the market,” he said. “The real area of opportunity for higher education is in making more personalized, smaller models that do a couple of things really well.”
Leave a Reply