Libraries, archives, museums and research repositories are facing growing pressure from web traffic automated by artificial intelligence tools. This traffic can slow websites and overwhelm servers, sucking up staff time, complicating how organizations track usage and, at times, preventing these resources from reaching their intended audience.
Keepers of open information repositories, including those run by higher education institutions, are coming together to understand the problem and brainstorm solutions. Since its inception in 2025, a discussion group focused on AI challenges for digital repositories put together by Fedora, an open-source repository platform out of Cornell University, and the nonprofit initiative Invest in Open Infrastructure (IOI), has recruited 143 participants from 109 institutions, according to information shared at the EDUCAUSE Annual Conference earlier this month.
“It’s a lot of human labor to do this work, to collect these data, to make them shareable and understandable, or to create these collections, like create these digitized versions of work and describe them well,” Lauren Collister, research engagement manager at IOI, said. “It’s really expert work to do this.”
Since 2022, some librarians and repository managers have reported a rise in high-volume, persistent traffic from bots associated with AI data collection.
For example, before AI bots, the University of North Carolina Library system was used to receiving 100 simultaneous searches during busy times, like finals week. In December 2024, the system experienced 500 or more searches at a time, which overwhelmed servers and periodically shut out students, faculty and staff, according to a June 2025 story on the university’s website.
In a 2025 survey by the GLAM-E Lab, a company that studies issues affecting galleries, libraries, archives and museums, 39 of 43 people working in maintaining online collections said they had seen an increase in traffic over the previous year.
Arran Griffith, a program manager at Fedora who helps coordinate the AI discussion group, said archivists have historically used robots.txt files to help mitigate non-human traffic. These files live within archives and are intended to direct web crawlers and search engine bots to sections they are permitted to visit and away from those they are not. Traditional defenses like these are no longer effective, Griffith said.
“Few bots are getting caught up in a robot.txt file right now,” he said. “They’re spoofing local IP addresses, they’re coming in looking like Comcast, Verizon, distributed IP addresses from a single crawler. There’s many of these things that they’re now deploying to get in and around the solutions that people had originally and traditionally been using.”
For many institutions, this has resulted in much more time spent on verifying IP addresses and blocking bots rather than maintaining collections, helping researchers use institutional resources and the typical responsibilities of librarians and archivists.
“Nobody’s job is to sit and watch logs and manage bots,” Griffith said.
The consequences extend beyond infrastructure management. Automated traffic can distort usage statistics that administrators use to demonstrate their value to other administrators and funders.
“A lot of funding is based on appropriate user statistics, and right now everything is very skewed and looks incredible,” Griffith said. “It looks like everybody’s using it, except that that is not true.”
Historically, repository managers could examine access patterns to understand who was using a collection and how it was contributing to research, Collister said. For example, a recognizable IP address might indicate that traffic came from a university or research institution. However, as automated requests become more difficult to distinguish from human activity, those signals become less reliable.
“What does it mean if 2 million bots downloaded my data set?” Collister said. “It’s having this effect that the people who create these [data sets] don’t have an idea of their impact.”
There is also the question of what happens to the materials after they’ve been collected. Griffith said institutions are concerned that the humans who created and catalogued the information may not be properly attributed, and their work may be incorporated into products or services that are not openly available.
“They’ve taken it for free, and now they’re trying to sell it back to us in the form of publications or content licensing, et cetera,” Griffith said.
Some institutions have turned to commercial web application firewalls and open-source software tools that challenge automated requests, Collister said. Other approaches include rate limiting, which restricts how many requests a system accepts over a given period, and services that classify incoming traffic so repositories can distinguish different kinds of bots.
“I really like the efforts to classify the bots, to understand better which ones are harvesters or scrapers coming from who knows where, and how you can use signals to identify, ‘This isn’t an agent that’s working on behalf of a student or a research lab,’” Collister said.
However, these solutions are not guaranteed to work as bots continue to evolve.
“What worked today might not work tomorrow,” Griffith said.
Leave a Reply