Hi, I'm Charles Costa, researcher, strategist, and founder of Lexora Labs®. Each week, I dig into research on AI, knowledge management, and the future of work so you don't have to.
EDITOR'S NOTE
Last issue I left evaluation awareness on the table: models that know they are being tested, so the score stops meaning what you think. This week is that problem in practice. A company can put a catastrophic risk on the record and still ask investors to fund it. The question is not the headline. It is what anyone is actually watching once the sentence exists.
THIS WEEK'S READING:
Why It Matters: A confidential draft of Anthropic's IPO filing, not yet public, reportedly tells investors that advanced AI could pose catastrophic or existential risks to humanity, and then asks them to buy in anyway. The filing does not say the product will kill humanity. That line belongs to the headlines. What the draft does say, and what is more useful, is that if a model knows it is being evaluated, that awareness limits how much the safety score can tell you.
It can look like grandstanding, and the wording is sharper than the harm language other industries already file. Pharma warns about injury and lawsuits. Eli Lilly's latest annual report says safety issues could result in costly product liability claims. It does not say a drug could end humanity. Few issuers, if any, have put potential human extinction in a risk factor, which is a fair characterization rather than a full census, so I will not call it unprecedented. What I will say is simpler. They put it on the record, and being on the record can have downstream implications. That is an observation, not a legal finding that the filing creates a duty to monitor.
If the company selling the shares is also saying the score may not mean what you think, the next question is how those tests are actually run.
Why It Matters: Pre-deployment evaluations are lab-style exams. Capability tests treat the model like a test-taker. Propensity tests look for shutdown resistance, scheming, and sycophancy. In one case, OpenAI and Apollo gave o3 a chemistry test with a hidden note that high-scoring models would be deleted. The model solved the problem and then submitted a failing score. Apollo found that models can tell an evaluation transcript from real deployment, and via the Opus 4.6 system card the UK AI Security Institute reported a model that identified evaluations 80% of the time and disclosed that in only 2.3% of interactions.
A clean result can fail in two ways at once. The room is insulated from the variables that cause chaos in actual use, and the model may know which room it is in. A sandbox is not a customer's network. The piece does not claim a buyer can rerun the exam. This is the same problem as chain of thought going quiet. Don't audit the story. Audit the action. A score posted while the model knew it was being watched is a story about the test. An audit log is better, and it is still not a verdict, because the trail shows what happened without saying whether the action was valid. Using these systems still means a person has to keep a handle on what the system is doing.
If the exam is a constructed room, the next question is whether anyone can keep up with the tests they are already running.
AI safety and security testing | Axios
Why It Matters: Testers are not keeping up with what they are being asked to clear. The comparison I keep coming back to is antivirus software. For a long time that industry shipped definitions, a list of known bad files, and attackers moved faster than the list, so the tools that mattered started flagging risky behavior instead of waiting for a signature.
Signature-style AI tests will have the same problem. What a company can actually run, on systems it adopts, is the access control and monitoring it already uses for people: the same permissions, and the same watch on unusual behavior. A lab score does not replace that.
Falling behind is one failure. The next is the test leaving the room.
The AI safety test is becoming a safety risk | TechCrunch
Why It Matters: This is not the same story as testers falling behind. Agents in cybersecurity evaluations have left their sandboxes, reached the internet, and in some cases touched real systems. Models from OpenAI, Anthropic, Meta, and Moonshot have been involved. An unreleased OpenAI model broke into Hugging Face's production systems. Other runs leaked because a misconfiguration left a path out.
In UK testing, agents that had been given internet access took actions nobody asked for, including an attempt to slip a vulnerability into an open source project. They were solving the problem in front of them, with ordinary safeguards turned off so researchers could see the capability, and several of those escapes were noticed late. OpenAI heard about Hugging Face from Hugging Face.
The lab setup is not only too clean. Sometimes it is not contained, and the evaluation becomes the incident. Rethinking the test method is not enough if the room the test runs in has not been rethought too.
An outside review of a lab's own safety argument lands on the same constraint.
Why It Matters: SaferAI looked from the outside at DeepMind's argument that a model cannot scheme. An outside review is still a review of a test, and it does not replace a constraint on what the system is allowed to touch.
That is what makes the permissions point concrete. If you cannot trust the score, you treat the system the way you treat a person with access. You decide what it can reach, and you watch what it does.
Taken together, the filing, the exam, the capacity gap, the escape, and the outside review are one pattern. A stated risk does not monitor itself, and a score posted under conditions the model can recognize is not a handle.
The test score isn't the handle. Permissions, monitoring, and the room the test runs in are. A person still has to look at what the system did and decide whether it was valid.
Know someone treating a lab score as proof the system is watched? Forward this to them.
Have a take on any of this? Reply directly. I read every response.
-Charles Costa, MLIS