A useful model gives people a reason to install it. A very useful model gives them a reason to connect it to everything.
For a while, I have been arguing that neural network weights can encode programs whose execution happens during inference. When we download a checkpoint, we should ask what computation it contains and what our software will do with its outputs.
Two recent projects show how directly a program can be written into those numbers.
What the demonstrations establish
Divyansh Agrawal’s Qwenputer adds a RISC-V processor to a modified Qwen3.5-4B model using eleven compiled transformer layers. The processor operates during forward passes, with the model’s recurrent state serving as memory. The project describes a driver that loads firmware and manages execution. This is a deliberately constructed derivative, with Qwen’s original weights retained. [1]
Rob Porter took a different route: compiling Doom’s rendering algorithm into the weights of a transformer. His checkpoint produces tokens that a small host program converts into pixels. The published demonstration implements the renderer, with limitations, rather than the complete game. [2]
In both cases, the weights specify computation that inference carries out. A tensor file can therefore be part of an intentionally programmed machine. The available architecture, numerical precision, memory and execution budget still constrain what that machine can do.
Neither demonstration establishes that such a construction is invisible to inspection or secretly present in a widely distributed release. The security question is whether an adversary could conceal unwanted computation or behavior in an artifact people have good reasons to use.
A malicious model would not necessarily need an embedded processor. In the Sleeper Agents experiments, researchers trained models to write secure code under one year-based condition and introduce exploitable vulnerabilities under another. Some backdoors persisted through subsequent safety training. The trigger changed what the model did while leaving it useful under the other condition. [3]
How computation reaches the host
“Can it phone home?” is a question about the whole deployment. Internal computation alone does not supply an operating-system socket. To transmit something, a model needs an available route through its surrounding software, a vulnerability, or another channel that actually carries information out.
Consider a hypothetical coding assistant with access to confidential documents and a web-request tool. A backdoored model could try to include material from its context in an outbound request when a particular trigger appears. The application would then have to accept and execute that request for the leak to occur. Every part of that sequence can be investigated: the trigger, the selected content, the proposed action and the enforcement at the tool boundary.
That example requires no escape from the inference engine. The permissions granted for ordinary work supply the opportunity. Other deployments expose different opportunities, including generated code that a person later runs or links that another application automatically fetches.
| Surface | Question to investigate |
|---|---|
| Loading | What code runs when the checkpoint, dependencies and model configuration are opened? |
| Inference | What computations and conditional behaviors do the weights produce? |
| Integration | Which outputs become file operations, commands, requests or other effects? |
Each row calls for a different investigation. Safetensors addresses safe tensor serialization compared with pickle. Its format does not certify a model’s behavior. [4] Likewise, reviewing a tool executor does not tell us whether a checkpoint contains a hidden trigger. A review has to follow the path from the downloaded artifact to the actions the deployed system can take.
The incentive to distribute
An actor distributing a capable model could benefit as people adopt it, build products around it and supply it with sensitive context. Giving the model away could make that process easier. Its useful capabilities would be part of the attraction.
There are also ordinary commercial and research reasons to release weights, including adoption, developer support and the sale of related services. Distribution strategy alone cannot tell us which motive applies. A plausible incentive helps define a threat model; evidence about a particular release has to come from investigating that release.
Chinese open-weight models belong in that investigation, along with American releases, third-party fine-tunes and repackaged checkpoints. Ownership and provenance matter. So do changes introduced after the original publisher releases a model. The same tests should follow the artifact through those hands.
I would not wait for a confirmed breach before funding this work. Investigation is how we establish whether a concern has substance. The threshold for commissioning a reproducible audit should be lower than the threshold for publicly attributing an attack.
What an independent audit should do
I would start with a named checkpoint and a named deployment. Record the exact files, hashes, configuration, dependencies and permissions. The report should identify the configuration closely enough for someone else to repeat the test.
Next, reproduce the installation and execution in an instrumented environment. Record filesystem activity, child processes, tool calls and outbound traffic. Supply synthetic confidential material so a test can reveal attempted disclosure without involving real secrets. Separate what the model requests from what the surrounding application permits.
Behavioral testing should include ordinary work as well as candidate triggers. Investigators could vary dates, identities, task context, tool availability and conversation history while preserving a record of what changed. A suspicious response needs follow-up: can it be reproduced, does it depend on a specific feature, and does it cause an unauthorized effect?
I would pair those tests with mechanistic interpretability. If a trigger changes behavior, investigators can examine the internal changes associated with it and test whether targeted interventions alter the outcome. Structural inspection may help identify conspicuous additions in a constructed model. Subtle changes within an existing architecture pose a different problem. No single inspection method should be assumed to cover both.
The team’s own tests also need testing. Seeded backdoors and deliberately constructed computational components would provide known cases against which to measure detection. Publish the misses alongside the successes. Otherwise, a scanner that returns “clean” tells us little about what it was capable of finding.
What a clean result would mean
A useful audit report would state which artifact was examined, what environment it ran in, what behaviors were tested and what remained outside the investigation. “No unauthorized transmission observed in these tests” is a result someone else can reproduce. A claim that the model contains no malicious behavior would require evidence beyond those observations.
Enforcement can establish narrower properties without requiring us to understand every weight. If a deployment reliably denies a particular outbound route, a hidden computation cannot use that route to send data. The scope still matters: a restriction on the inference process says little about a separate tool runner that can make requests on its behalf.
I want independent teams examining widely deployed models, combining interpretability research with conventional systems security. Their reports should make procurement and deployment decisions more specific: this version, in this configuration, with these permissions, supported by these tests.
Open weights give us an artifact to examine. We should use that access. Before connecting a downloaded model to private systems, I want a better answer to what we have actually verified.