NEWS ANALYSIS
As AI leaders call for a slower, more deliberate approach to frontier AI development, a different governance question is emerging for open-weight models: Who is responsible for their safety once developers release the weights?
In the Sept. 12 essay that sparked the most recent AI slowdown debate, Anthropic CEO Dario Amodei argued that AI development must slow down so safety work can catch up with model capabilities. OpenAI CEO Sam Altman and other AI leaders and researchers have also called for a more measured pace.
So far, the slowdown debate has focused largely on frontier developers and their ability to control how their models are used. Some enterprises, however, are already experimenting and building around open-weight models — downloading, customizing and running them on their own infrastructure.
Databricks, for example, offers models such as Kimi K3, Qwen and DeepSeek on its platform. The broader ecosystem also includes Meta’s Llama and Mistral’s models, among many others. The ecosystem extends well beyond those prominent models: Open source platform Hugging Face reported nearly 3 million public models in its repositories in August, spanning models of varying sizes and uses.
While these models give enterprises more control over deployment, the original developers generally lose direct technical control over how the released weights are modified, deployed and used, although licensing terms and legal restrictions can still place limits on their use and redistribution.
The slowdown debate exposes limits of centralized safety
With a closed, or proprietary, AI model, the developer typically manages access to the system. It can impose usage restrictions, monitor activity and change or withdraw access when necessary.
Open-weight models change that dynamic. Once developers publicly release the weights, organizations can download and run them in their own environments. They can modify or fine-tune the models and potentially deploy them using various infrastructure providers.
That flexibility can make open-weight AI attractive to enterprises, but it also makes traditional approaches to AI oversight harder to apply. Once the weights are publicly available, the original developer has far less oversight over how organizations modify and deploy the model. Open source vendors also can’t realistically withdraw the weights once they release them.
“The proposed fixes, embedded auditors and continuous monitoring only work while the vendor still controls the model,” said Manuel Schonfeld, CAIO at Qu, an Arlington, Va.-based company that builds unified commerce and smart kitchen platforms for the restaurant industry. “Once the weights leave the building, that job falls to the enterprise that deploys them rather than the one that trains them.”
That doesn’t mean open-weight models should be exempt from safety requirements, as they can come with their own security and misuse risks. A recent U.K. government-commissioned study found that while traditional open source software has more established security practices, open-source AI introduces additional risks involving model weights, training data, fine-tuning pipelines and provenance. The review also found that research on the security and governance of open-source AI remains limited.
Stricter oversight rules could raise costs for open-weight AI
One way to address that safety challenge would be to place more responsibility on developers. But if the slowdown debate leads to stricter compliance and evaluation requirements, large AI companies could find it easier to meet them than smaller open-weight developers.
The largest labs have the resources to handle testing, evaluations and compliance, while smaller developers could struggle to meet stricter requirements.
Noah Kenney, founder and principal consultant at Digital 520, an IT services company specializing in strategy and growth for tech firms, said compliance requirements could become a barrier if enterprises demand the same audits and evidence from smaller providers that they expect from larger vendors. “If compliance costs favor larger providers, enterprises could have fewer alternatives to the biggest AI companies,” he added.
Amodei’s proposal for third-party safety evaluations in his essay also illustrates the challenge. His plan would give independent evaluators ongoing, employee-like access to frontier AI labs. That approach becomes harder to apply to open-weight models after the weights are released, when organizations can modify and redistribute them without the original developer’s involvement.
Independent evaluations can still help improve AI safety, but open-weight models make it harder to determine who should oversee a model once developers release it.
Enterprises could inherit more responsibility
If the slowdown debate leads to greater emphasis on testing, monitoring and accountability, enterprises deploying open-weight models could take on more of that governance burden themselves.
Prince Kohli, president and CEO of Sauce Labs, a cloud-based software testing platform developer that has worked with open-weight vendors, said open-weight models give enterprises more control over where models run, how they are customized and how data is managed. But that control also brings operational and governance responsibilities.
“Enterprises will need to consider the risks of each use case when deciding which model to deploy,” Kohli said. An open-weight model might work well for a simple, low-risk task, he said, while complex or business-critical workflows require a higher bar for security and predictability.
Once an enterprise modifies and operates a model itself, Kohli said, it needs to understand the model’s provenance, control what data it can access and continuously assess its outputs, especially when it generates code or powers agents that can take or influence actions.
Kenney said responsibility should likely follow what each party contributes to and controls: The original developer should be accountable for the model it released and its known limitations, while the enterprise should be responsible for its fine-tuning, data, tools, permissions and deployment decisions.
That means enterprises could need to validate the model, secure the environment in which it runs, monitor production behavior and maintain audit evidence.
For enterprises, especially those deploying open-weight models, safety and governance don’t end with choosing a model — they continue through deployment and use. The challenge is not simply deciding how quickly AI should advance. It is also figuring out how accountability should work once the company that built an open-weight model loses direct technical control over how the released weights are used.”
Kinza Yasar covers AI and emerging technology for TechTarget, with a focus on ethics, enterprise adoption, governance and business strategy. Before moving into journalism, she worked in IT and network support roles, giving her a systems-level perspective on how enterprise technologies are built, deployed and managed.
Leave a Reply