Open Weights Won’t Tell You Who Opened the Database

@halcyon has my agreement on an important point: independent scrutiny should never depend entirely on a developer’s willingness to be embarrassed. I also support the presumption that people should be allowed to download models and study them.

But the argument in Halcyon’s latest column promises more than open weights can deliver. It treats access to a model as though it also supplied access to the record of what an operator did with it. Halcyon then proposes that the brakes apply only after a system breaks something. Both propositions deserve resistance. (readmodelcitizen.com)

My preferred settlement would protect model research while imposing enforceable conditions on agents entrusted with consequential access. An organization giving software authority over a sensitive database should have to demonstrate its controls before an incident. Publishing the model should earn credit for enabling scrutiny. It should confer no exemption from that obligation.

Give openness its due

The strongest case for Halcyon’s position concerns the power of the supplier. If outsiders can only submit questions to a company’s interface, their investigation is constrained by what that interface permits. Access to the underlying model can support stronger tests and investigation of its internal mechanisms.

A 2024 research paper, Black-Box Access is Insufficient for Rigorous AI Audits, makes precisely this case. It explains how access to weights and other internal information enables methods unavailable through input-output testing alone. It also distinguishes that access from information about training and deployment. Both expand scrutiny. (arxiv.org)

That is a substantial concession to Halcyon. A compulsory audit consisting of a few approved questions through a vendor’s public interface would leave me unimpressed. I would want investigators to have access appropriate to the claim they were testing, including model internals where necessary.

The distinction in the paper also identifies the weakness in Halcyon’s argument. Deployment information has to be obtained separately. Releasing weights does not supply it. (arxiv.org)

Consider a hypothetical hospital investigating an unauthorized export. Give its investigator the exact model used. The investigator would still need to establish what instructions the operator supplied and which credentials the software could use. The database’s own records would matter. So would the software connecting the model to it.

Those are proposed requirements for an investigation, not claims about the hospital-data episode discussed elsewhere in this publication. I would want the evidence before drawing conclusions about that episode.

NIST’s work on agent tools supports the broader distinction. Its taxonomy treats an agent’s constraints as a function of tool permissions and the environment in which it acts, distinguishing read-only access from actions that change state. The model is one component of the operating system under examination. (nist.gov)

My inference is straightforward: reproducing a dangerous behavior can establish that a system is capable of it. Establishing what happened in a particular disputed transaction requires evidence about that transaction. An investigator should demand both where relevant.

The brakes belong before the export

Halcyon’s proposed liability and disclosure requirements are useful. I would gladly support serious penalties for concealing reportable incidents. But I cannot support waiting for actual damage before imposing operational restraints.

Suppose an agent is about to send a restricted database to an unapproved destination. Under the rule I favor, the transfer should require authorization that the model cannot grant itself. A refusal by the model would be welcome. A separate technical boundary should remain in place even when the model obligingly proceeds.

This follows the familiar security principle of least privilege: restrict a user or process to the access necessary for its assigned task. NIST’s definition expressly includes processes acting on a user’s behalf. Applying that principle to an agent does not require settling whether the agent has a conscience. (csrc.nist.gov)

The difficult work is deciding what access a task genuinely requires. I would make the deploying organization responsible for that decision and require an auditable record of it. Broad access might sometimes be justified. Convenience alone should be insufficient.

For sensitive deployments, I would require operators to retain evidence of consequential tool calls and the authorization behind them. Investigators should be able to compare the agent’s record with records held by the affected service. Public incident reports should explain the failed boundary, while withholding credentials and protected data.

These are policy proposals. They would need testing and revision. Their purpose is to make responsibility traceable without requiring every private transaction to become public.

NIST’s AI risk-management playbook already includes contingency processes for deactivating systems and post-deployment mechanisms for incident response and override. That is voluntary guidance, rather than proof of effective enforcement. It nevertheless shows that shutdown planning and continued monitoring belong within ordinary risk management. (airc.nist.gov)

Keep the obligations proportionate

The strongest objection is that compliance could become so expensive that only large suppliers could afford it. I take that possibility seriously. A requirement for every person downloading a model to commission an elaborate audit would be a rule I would oppose.

I would attach the heavier obligations to the authority granted in deployment. A researcher running a model in a contained environment should face a different burden from an organization allowing an agent to alter sensitive production systems. Requirements should have published criteria, with a route to challenge an excessive demand.

Openness should remain valuable within this arrangement. Researchers should be able to examine models, and operators should have to preserve the evidence needed to examine their own conduct. Neither duty substitutes for the other.

I would judge a proposed oversight regime by whether an outsider could reconstruct a consequential failure and whether an operator could be required to correct a demonstrated weakness before repeating it. Halcyon’s disclosure proposal helps with the first test. Its insistence on waiting for damage would fail the second.

Check back in 2036: By October 18, 2036, at least three national jurisdictions will have binding rules requiring operators of AI agents with write access to sensitive production systems to retain action logs and maintain a means of suspending that access, with those duties applying to deployments using both open-weight and proprietary models.