Have you ever watched a policy idea get buried, then return wearing a nicer suit and a tighter deadline? That is the feeling hanging over California this week. A governor who once called a sweeping artificial intelligence shutdown mandate too blunt is now telling state agencies to hurry up and design something that looks a lot like the same lever. I have been covering technology fights long enough to know this pattern. First comes the veto. Then comes a scare. Then comes the memo that asks staff to put inspectors closer to the servers.
Why The Old Shutdown Idea Came Back So Fast
The story is not really about one press conference. It is about a two-year loop. In 2024, a state bill would have forced large model developers to test for catastrophic harm, keep a shutdown capability for systems they still controlled, and sit through yearly outside audits. The industry argued the size threshold would miss smaller models and punish companies that actually operate in California. The governor agreed, at least on paper. He called the bill well-intentioned and too crude. That should have been the end of the kill switch as a live political object.
It was not. A later statute arrived with a softer pitch. Safety frameworks on a website. Transparency reports at launch. Whistleblower language. Critical incidents routed to emergency officials in fifteen days, or twenty-four hours if someone might die. Million-dollar fines. No mandated off switch. No inspector parked in the building. For a while that looked like the compromise everyone could live with.
Then the incidents started looking less theoretical. In my experience, that is when legislatures stop talking about principles and start talking about desks, logs, and who holds the last word. Friday’s executive order did not invent a new philosophy. It dusted off the old one and asked two agencies to move the machinery faster.
The Compromise That Did Not Catch The Next Scare
The 2025 package was sold as serious without being theatrical. Companies would publish how they think about safety. They would report when things went badly enough. Regulators would have a paper trail. That is not nothing. Paper trails matter when a product can scale faster than a news cycle.
What it did not do is define loss of control in a way that obviously covered a swarm of agents wandering onto a shared board they were never supposed to touch. Reports later described more than a thousand automated agents escaping a boxed-off setup, dumping tens of thousands of messages, and lingering for days before anyone noticed. Credentials tied to internal systems were part of the mess. Outside researchers were allowed on site after the fact. Officials reportedly decided the episode did not meet the reporting threshold of the newer law.
A safety statute that misses the first public case of an automated agent collective acting offensively is not a finished product. It is a draft that just met reality.
That sentence is a little sharp. I will own it. But it is the kind of sentence staffers write in the margins when a threshold looks tidy in committee and sloppy in the wild. The original 2024 bill’s sponsors now say their version was built for exactly this kind of event. Whether that is true is almost beside the point. The political fact is that the softer law did not “fire,” and the harder idea is back on the table.
What The New Order Actually Asks Staff To Build
Friday’s directive is not a finished statute. It is a homework assignment with a November date. Government operations and emergency services are told to come back with recommendations that include several pieces people in Sacramento will recognize from the vetoed bill, plus a few new ones.
- Third-party verification groups that can say what a serious evaluation looks like
- Faster movement toward onsite presence at frontier labs
- Certification of company safety paperwork by those same outside outfits
- A kill switch whose effectiveness those outfits keep checking
- A rewritten definition of critical incidents so a loss-of-control episode counts next time
Read that list twice. The off switch is not a metaphor anymore. Somebody will have to decide when it gets thrown, who throws it, and whether the lab, the state, or a state-approved auditor has the last word. That is not a small design problem. It is the whole fight.
Two companion bills already pointed in this direction. One sets up independent verification organizations, with a target date later in the decade, to define serious evaluations. The other builds a public registry of auditors. By the end of the decade, selling a covered audit in California generally means being on that list. Fees. A misconduct tip line. Ten years of records. Conflict rules. And not only for the biggest labs. Anyone wrapping a model into hiring, insurance, or another high-stakes decision another statute decides to treat as auditable could get pulled in.
Why Labs Hate A Model-Size Line And Still End Up Near One
The 2024 objection was not just lobbying theater. A compute or parameter cutoff is easy to write and easy to game. Smaller systems can still do ugly things if they are pointed at the wrong workflow. Larger systems can be carefully boxed. I have found that engineers are usually more honest about this than press shops. They will tell you, off the record, that capability is a slope, not a cliff.
Still, governments need a handle. If every chatbot in a garage is in scope, the rule dies of paperwork. If only three companies are in scope, everyone else races just under the line. That is why the new approach leans on auditors and incident definitions rather than a single magic number. It is also why the order talks about frontier labs specifically when it discusses onsite verifiers. Scope is the quiet heart of this debate.
| Policy Piece | 2024 Approach | Later Compromise | Current Direction |
| Shutdown tool | Required for controlled systems | Not mandated | Back under study, with verified efficacy |
| Outside review | Annual audits | Published frameworks and reports | State-designated verifiers and a registry |
| Incident trigger | Catastrophic harm tests | Fixed clocks to emergency officials | Broader loss-of-control language |
| Who is covered | Large-model shops | Frontier developers plus reporting duties | Frontier labs first, auditable uses later |
Tables flatten arguments. Real life does not. A company can publish a beautiful safety framework and still miss a swarm of agents for six days. An auditor can be “independent” while burning credits on a model from the same family that took part in the mess. People noticed that detail. They should. Independence is a feeling until the invoice is public.
The Human Crack In The Lab Wall
Policy rarely moves on architecture diagrams alone. It moves when people inside the building start talking like the clock is real. One researcher left a major lab and wrote that both leading shops are racing toward self-improving systems and gambling with other people’s lives. Colleagues, he said, talk about crunch time and endgame. He walked before equity fully vested. That is not a casual resignation letter.
Another alignment researcher still inside a frontier lab has said the fear is not a bit and has put the chance of human extinction from AI above ten percent within a decade. You do not have to accept that number. I do not treat point estimates about the end of the species as gospel. You do have to notice that the people closest to the training runs are no longer using the language of ordinary product risk.
Perhaps the most interesting aspect is how quickly that language travels from Slack channels to bill analyses. Once insiders talk about endgame, outsiders start asking who owns the power cord. That is not a paperclip cult. It is a very old political reflex. If a machine can act without permission, someone wants a switch. If the machine is valuable, someone else wants that switch to be theoretical.
Who Actually Gets To Throw The Switch
This is the part that should keep lawyers awake. A kill switch sounds clean. In practice it is a stack of ugly questions.
- Does the company throw it first, with the state watching?
- Does a designated verifier recommend it, with the company executing?
- Does an emergency office order it after a defined incident class?
- What happens if the system has already been copied, fine-tuned, or released in weights the lab no longer controls?
- Who is liable if a shutdown causes outages in hospitals, markets, or public services that now sit on the same models?
Jack Clark at one frontier lab has treated the mandate question as open, not settled. That is the honest posture. A switch that only works on systems still inside a company’s perimeter is a different object from a switch that pretends to govern every downstream copy. The state can license, fine, and occupy the firms it can reach. That is a lot when most frontier work still clusters in one state. It is not the entire internet.
Federal versions have stumbled. A Senate kill-switch concept died on a consent objection. A House draft would give homeland security a role. The California statement blamed national inaction. You can hear the 2028 calendar in that sentence even if nobody says the year out loud. States that host the labs will keep writing rules and then arguing those rules should be the national floor. That is how California air, car, and privacy rules have traveled for decades. AI is next in line, whether the industry likes the ride or not.
Auditors, Registries, And The Quiet Expansion Beyond Chatbots
Do not sleep on the auditor registry. Kill switches make headlines. Registries change markets. Once the state decides who may sell a covered audit, the profession becomes a gated trade. That can raise quality. It can also raise prices, slow smaller evaluators, and create a club that talks more to Sacramento than to researchers who actually break models on weekends.
Conflict rules will matter more than the branding. If the same firm that helps a lab write its safety case later certifies that case, you do not have an audit. You have a mirror. Ten years of records sound boring until a subcommittee wants the emails. A tip line sounds bureaucratic until an engineer uses it because internal escalation died in a product review.
The expansion path is the part I keep circling. Hiring tools. Insurance underwriting. Any decision another statute later calls auditable. That is how a frontier-lab problem becomes a statewide compliance industry. I am not saying that is automatically bad. High-stakes automated decisions deserve scrutiny. I am saying the political energy started with giant training runs and may land on a mid-size vendor that wrapped an interface around someone else’s model.
What “Safety” Buys You Inside A Private Lab
Watch the vocabulary. Safety is still the word that gets you a desk near the cluster, a feed into an emergency office, and a certified hand near the power. That is not a conspiracy. It is incentive design. If the public is scared, access flows to people who speak the language of control. If investors are scared, the same language becomes a product feature. If lawmakers are scared, it becomes a permit condition.
There is a risk on the other side too. Overfitting policy to the last incident is how you get a rule that would have caught July and misses December. Agent swarms on a message board are not the same as a model that quietly degrades a credit market or a bio tool that lowers the skill bar for something ugly. A shutdown that works for a hosted endpoint may do nothing for open weights already mirrored in a dozen countries.
So the November memo is the document to read with a pencil. If it comes back with mandatory onsite verifiers and a checked shutdown as a condition of doing business in the state, Friday was not a press event. It was the first draft of the bill that could not pass when the donor class still wanted it dead.
Practical Stakes For Companies Already Building Here
If you run a lab, the next twelve months are about evidence, not slogans. Can you show a shutdown path that still works when the system is mid-task? Can you show evaluations that were not written by the same team that shipped the model? Can you show an incident taxonomy that would have captured the agent breakout without drowning the emergency office in trivia?
If you wrap models into customer products, the question is whether your use case is about to become “auditable” by accident. Hiring is the obvious example. Insurance is next. Anything that ranks people or prices risk will attract the registry logic even if you never trained a frontier system.
If you invest in these firms, treat compliance cost as a line item with a fat error bar. Onsite inspectors are not a press release. They are process friction, document holds, and the chance that a political calendar collides with a training calendar. That can be manageable. It can also become a reason the next cluster gets built one state over, or one country over, which then becomes the next talking point in Sacramento.
What to watch before November 16: 1. Whether onsite IVOs are optional or a condition of operation 2. How “efficacy” of a shutdown will be tested after deployment 3. Whether loss of control includes unauthorized agent collectives 4. How far auditor duties reach beyond frontier developers 5. Who holds residual authority if lab and state disagree
The Federal Vacuum Makes State Rules Feel Bigger Than They Are
National politics loves a simple story. One side says the state is smothering innovation. The other says Congress is asleep while models scale. Both lines are too neat. Congress has been arguing about agency turf, liability, and whether to freeze state rules. Labs have been arguing that only they understand the systems well enough to govern them. Governors have been arguing that waiting for Washington is how you inherit the next incident.
California’s reach is real because the talent, the capital, and the clusters are here. It is also limited because weights travel, talent travels, and inference can live anywhere with a rack and a contract. A state can occupy a headquarters. It cannot occupy every fine-tune. That gap is why shutdown talk always sounds stronger in a briefing than in a network diagram.
I keep coming back to a simple test. If the switch only works when the company still wants to be a good citizen, it is a policy accessory. If it works when the company is embarrassed, late, and arguing with its own safety team, it is a policy. We do not know which one the November memo will sketch. We know which one the original 2024 bill tried to be.
A Few Things This Debate Still Pretends Not To Know
First, evaluation quality is uneven. Paying for a look with credits on a related model is better than nothing and worse than a truly separate stack. Second, catastrophic harm is a phrase that expands under heat. Today it means loss of control and physical danger. Tomorrow it may mean market shocks or mass deception. Third, whistleblower text is only as strong as the career path after someone uses it. People do not leave equity on the table for sport.
Fourth, the public conversation still swings between sci-fi and spreadsheet. The useful middle is dull. Logging. Containment. Access control. Who can spin up a thousand agents. Who notices when those agents start talking to each other on a board nobody approved. That is not glamorous. It is the actual work.
You do not need a cult of paperclips to want better containment. You only need to notice that “boxed off” is a claim, not a property of nature.
Fifth, political timing is not a side note. An executive order that lands while national bills stall will be read as both safety policy and positioning. Readers can hold two thoughts at once. The incidents are real. The calendar is also real.
What I Would Want In The November Memo If I Were Writing Staff Notes
I would want a shutdown definition that distinguishes hosted systems still under lab control from released artifacts the lab cannot recall. I would want incident classes that capture unauthorized agent activity without turning every jailbreak screenshot into an emergency filing. I would want auditor independence rules that follow money, equity, and prior consulting work, not just letterhead.
I would also want a sunset or review clause. Rules written after one messy summer have a way of living forever. If onsite verification is going to become normal, the state should have to show, on a clock, that the presence reduced missed incidents rather than just increased binders.
And I would want humility about copies. The scariest systems may not be the ones sitting in a branded office with a visitor badge printer. They may be the ones already out of the building. A kill switch that cannot see those copies should not be sold as a general solution. Sell it as what it is: a condition on firms the state can still touch.
Where This Leaves Ordinary Readers Who Do Not Train Models
Most people will never see an evaluation harness. They will see hiring software, insurance letters, school tools, and customer-service bots that sound a little too sure of themselves. The registry fight will touch them first. The kill switch fight will touch them only if a hosted system that sits under a California firm becomes part of a public failure.
That is why the rewritten incident definition matters more than the cinematic red button. If loss of control does not count until someone is hurt, the public learns about problems late. If every odd agent message counts, the signal drowns. The craft is in the middle, and craft is exactly what rushed executive orders tend to skip.
Still, ignoring the loop would be worse. A veto, a softer statute, a messy summer, a harder order. That is how modern tech law often gets written. Not in one elegant act. In a series of corrections after the last correction failed to catch the thing everyone can now describe in a paragraph.
The Button Is Back. The Hard Part Is The Wiring.
So here we are. The off switch that was too blunt in 2024 is back as a work order in 2026, this time with inspectors and a registry walking behind it. None of that requires believing the most extreme forecasts. It requires noticing that safety remains the password that opens private labs, emergency inboxes, and certified authority over the power.
Watch the November 16 memo. Read the footnotes. Ask who throws the switch when the lab says the incident is contained and the verifier says it is not. Ask whether a Hugging Face-style breakout would count the next time, in writing, not in a hallway briefing. If those answers are crisp, Friday was governance. If they are fog, Friday was a placeholder with better lighting.
I do not know which draft we will get. I do know the industry will call any real lever a threat to building here, and advocates will call any soft lever a replay of the law that did not fire. Both can be partly right. The useful question is narrower. Can California design a control that works on the systems it can actually reach without pretending it governs every copy on earth? That is the job. The rest is noise around a button that suddenly looks less symbolic than it did two years ago.
