A high school student tells an AI career-advising chatbot he wants to take over his father’s farm. Two years later, that comment follows him to college, where another AI system uses it to help shape recommendations about his academic path.
That hypothetical, shared Thursday during a webinar facilitated by the nonprofit American Enterprise Institute, illustrated an emerging challenge panelists face as artificial intelligence tools become capable of retaining information about students across interactions: What happens to everything those systems remember?
WHAT IS AI MEMORY?
Matt Gee, director of U.S. program data at the Gates Foundation, described AI memory as information a system knows about an individual learner and uses to inform later interactions.
“When I hear a parent asking, ‘What does this technology tool know about my kid?’ they’re actually asking a question about memory,” he said.
He added that that capability is part of what can make AI a double-edged sword in education: The same persistence that can help an AI system build on a student’s prior progress can also preserve outdated, unwanted or inaccurate information about that learner.
“If a student’s progress doesn’t travel with them from one tool to the next, from one grade to the next, one institution to the next, then they have to start over sometimes,” Gee said. “The flip side of that is, if that record contains things that they don’t want in there, or that are really misunderstandings of who they are, that could hold them back.”
AI’s memory, according to Andrew Rice, CEO of the nonprofit Education Analytics, could become more consequential as AI systems begin acting on information rather than simply responding to queries.
Rice, who shared the hypothetical about the career-advising chatbot, described those systems as part of an “agentic Internet” in which AI systems are acting on one’s behalf and not merely just providing information. His concern, then, is rooted in what could happen when information collected about a student in one system is carried into another, where it can shape later recommendations or actions.
These facets of persistent memory, according to both Gee and Rice, create a trade-off that education leaders and policymakers must face.
WHAT CAN SCHOOLS DO?
Vanessa Wren, chief information officer at the North Carolina Department of Public Instruction, said the state’s immediate concern is not whether schools should use AI, but how to govern technology that is already embedded in education products.
“Our schools have hundreds of products that they might not even know that has AI in it,” Wren said.
Wren said her agency is trying to determine where AI is present, what technology powers it, what information it receives, what it creates and what inferences persist from those interactions.
“AI tool adoption cannot be based on trust,” she said. “We have to look at AI memory as a new data category.”
For Wren, one of the most immediate mechanisms available to education agencies is procurement.
“Our contract terms are the tool that we have,” she said.
We have to look at AI memory as a new data category.
North Carolina Department of Public Instruction CIO Vanessa Wren
Wren pointed to North Carolina laws that place restrictions on how student data can be used and secured: Online services covered by the state’s K-12 student privacy law cannot use student information for targeted advertising, build profiles for purposes unrelated to K-12 education, or sell or rent that information. A separate provision requires the State Board of Education to maintain a security plan for the state student data system, including privacy and security audits, access controls and breach procedures, while the Department of Public Instruction must conduct ongoing compliance audits, she said.
Wren also called for more centralized governance of procurement, arguing that districts otherwise may not know where their student data resides. North Carolina has denied procurements when vendors could not meet standards the state considered reasonable, she said, even when the product itself might otherwise have been useful.
WHAT REMAINS UNRESOLVED
Rice similarly characterized procurement as one of education’s primary short-term mechanisms for governing data privacy in AI systems, arguing that existing federal privacy laws — e.g., FERPA, COPPA — do not fully account for new technology.
Moreover, panelists said controlling how vendors collect and use information does not resolve the question of what students and families should be able to do with the AI memory itself.
Wren said portability — allowing information to move between systems — should also include the ability to review and edit records, offering an example of a student whose AI tutor records difficulty with mixed fractions but who later masters the material.
“We’ve got to have the ability to edit some of that data before it is ported and put into other systems,” she said.
Wren also noted unresolved questions around whether AI memory should explicitly be named as a data element, how generated inferences should be treated, and how deletion should — and could — work. Still, Wren said schools do not need to resolve every technical issue before setting expectations.
“I don’t think we have to solve every technical question before we establish some basic rights and responsibilities,” she said. “But we absolutely have to put at the forefront some transparency about AI, how it’s operating and some meaningful controls in place, so we have confidence for our students.”
Leave a Reply