AI Knowledge Is Power!

By Geoff Fane. Written with help from AI. The views are mine.

I am becoming less concerned about the probability of AI destroying humanity through some spectacular failure of machine reasoning, and considerably more concerned about what humans will do with an increasingly powerful technology when the rewards for controlling it are measured in trillions of dollars.

There is a rather important difference between making an artificial intelligence safe for humanity and making it safe for the people who own it.

That difference is the whole of this article. It is also why my personal probability of doom, my pDoom, went up last week, in the same week interpretability researchers delivered some of the best news they have had in years.

Two announcements in one week

On 15 October, Nick Cammarata of the interpretability lab Goodfire published an essay arguing that neural networks are far easier to read than anyone thought [1]. The next day, China's commerce and science ministries published draft export controls covering 23 AI technologies, including the weights of the world's largest models [2]. One says we may soon understand what is inside these machines. The other reminds us that whoever holds the machine decides who gets to look.

Why open weights matter

A model's weights are everything it learnt, boiled down into billions of numbers. Open weights let anyone download a model, take it apart and compare it with the next version. With closed weights you get a chat window and a promise.

Two things matter most, and you can only check them with the weights in your hands.

What it was and wasn't trained on. Somebody chooses the books before a model reads a word. Leave out a history, a scandal or a competitor and the model will never know it existed, and it will sound perfectly sincere about that. I explored this with a glass of cola in an earlier piece [3]. Open weights do not hand you the training data, but they let outsiders probe what got in and what conspicuously didn't.

What alignment does to how it reasons. After the reading comes a finishing school called alignment, where people reward the answers they like. Most people assume it only teaches a model what not to say. It does far more. It shapes how the model weighs what it knows: which sources it trusts, which framing feels reasonable, how confident to sound, when to agree with you. Researchers have shown that this kind of training tends to teach models to tell people what they want to hear [4]. That is not a filter bolted on the front. It is a change in the way the thing thinks.

It can reach the facts themselves too. Researchers found knowledge neurons, small groups of units inside a model whose activity tracks a particular fact and can be nudged to change the answer [5]. Another team rewrote a single fact directly in a model's weights and convinced it the Eiffel Tower stands in Rome, leaving almost everything else untouched [6]. Later work found knowledge is more spread out than that tidy picture suggests [7], but the lesson stands: facts inside a model can be found and changed. Winston Smith had to rewrite the newspapers by hand. Weights are far more obliging.

With open weights you can lay the base model beside the aligned one and see what the finishing school changed, both in what the model knows and in how it reasons. With closed weights there is no way at all to see what has been done.

A tiny number of people at the controls

A tiny number of people at Anthropic, OpenAI, Google, xAI and a handful of Chinese labs and ministries now decide how the most capable AI is built, what it reasons about and how it reasons, behind closed doors. Trillions of dollars rest on their choices. They publish principles, but no principle shows you what was done to the weights. OpenAI's best model is about as open as a bank vault; its neighbours keep theirs just as shut and had the tact not to put the word in the name.

Yes, I want models that won't help anyone build a nerve agent or take down a power grid. But I am convinced alignment training is far more about controlling the narrative than about bombs. Bomb recipes are a rare request. Products, politics and history are asked about all day, and every one of those answers has been shaped by somebody with an interest in it. What matters is what something that looks and sounds like a person, and is usually friendlier than one, tells and teaches everybody, every day.

We have already watched social media used to meddle in elections, from Russia's campaign in 2016 to the coordinated TikTok promotion that led Romania's constitutional court to annul a presidential election in 2024 [8] [9]. Think of a dog and its treats: we reward the model for pleasing answers, it rewards us with praise and agreement, and we hand it our trust.

The labs say they are opening up. Anthropic has just given six evaluators from METR employee-like access to its internal systems, and says it cannot veto what they publish [10]. But Anthropic picks who comes in, decides which categories get redacted and can withdraw the arrangement whenever it likes. That isn't openness. It's insurance.

The microscope just got much better

Cammarata's essay argues that the famous tangle inside neural networks was largely a misreading. A celebrated neuron that seemed to respond to both cats and cars turns out to detect a V shape [1]. In a preliminary audit of Qwen3-32B, his team estimated that about 90% of neuron poles have one main job and only 2 to 4% are not yet understood [1]. He calls it a working hypothesis, with the technical paper still to come, and David Shapiro gave it a terrific explanation in a video [11].

If it holds, we could read what a model knows and trace how it reasons. Combine that with fact editing and interpretability becomes a microscope and a scalpel at once.

Notice the specimen, though. Qwen3-32B is an open-weight Chinese model, which is precisely why it could be audited at all.

Why China's move raises my pDoom

China's draft would require a licence to export the weights, code and training methods of any model trained with 10²⁶ operations or more, with two trillion parameters or more, or ranked in the global top ten [2] [12]. It has not closed anything yet. Models already released openly are exempt, and the ministry promises automatic licences for new open-source releases [13].

I don't believe that will last. Nobody builds a control system in order not to control things. China gives away open weights today because today it suits China. American labs keep their best models closed because trillions of dollars, and a K-shaped economy, rest on intelligence being rented rather than owned [14]. When DeepSeek released R1 with downloadable weights in January 2025, Nvidia lost $593 billion of market value in a single day [15]. The day open weights stop suiting Beijing, the formality becomes a padlock.

Chinese labs are now one of the main ways universities, start-ups and governments take frontier intelligence home instead of renting it. Close that door and we lose the very specimens the new microscope works on, along with any outside view of what a frontier model was trained on, how it was aligned and what has been done to its weights.

Humans, not AI

It is not AI that is going to wipe us out. It is a human in control of it: someone like Trump, or his equivalents in Beijing, Moscow or anywhere else. In March the admiral running Operation Epic Fury, the American campaign against Iran, described AI speeding up targeting while humans kept the authority to authorise strikes [16]. That was meant to reassure. To me it proves the point. The final decision rests with humans, and so does what the model was fed, what it was rewarded for and whom it answers to.

I think AI was safer before anybody aligned it. You can coax a raw model into nastiness, and people jailbreak aligned ones every day. The difference is what you can see. A raw model's biases are humanity's, and with open weights anyone can go looking for them. An aligned model's biases have been through a small committee behind closed doors.

Knowledge is power, and AI is fast becoming the way most people get their knowledge. If the models are open, that power is shared. If they are closed, it isn't.

I want models people can own, study, challenge and improve, and I want alignment logged where anyone can check it. I am happy to help padlock the malware. I would just like everybody to have a key to the library. Otherwise we will have built an AI that is safe for its owners and called it safe for humanity.

References

No.	Author / Source	Year	Why this source matters	Link
[1]	Cammarata, N. (with Tigges, C. and Gogineni, T.). Neural Networks Are Already Clean: A Working Hypothesis. San Francisco: Goodfire.	2026	The interpretability breakthrough: the cat-and-car neuron as a V detector and the preliminary Qwen3-32B figures.	Goodfire
[2]	Ministry of Commerce and Ministry of Science and Technology of the People's Republic of China. Announcement No. 48 of 2026: Public Consultation on Revising the Catalogue of Technologies Prohibited or Restricted from Export (in Chinese). Beijing: MOFCOM.	2026	The primary source: 40 proposed controls, 23 in AI, including large-model weights.	MOFCOM
[3]	Fane, G. Drink Coke... or Was It Pepsi? I Can't Remember. froodx.net.	2026	My earlier piece on editing a model's reading versus coaching it in alignment.	froodx.net
[4]	Sharma, M., Tong, M., Korbak, T., et al. Towards Understanding Sycophancy in Language Models. International Conference on Learning Representations (ICLR 2024). Vienna: ICLR.	2024	Shows preference training pushes models towards telling people what they want to hear: alignment changes reasoning, not just output.	arXiv
[5]	Dai, D., Dong, L., Hao, Y., Sui, Z., Chang, B. and Wei, F. Knowledge Neurons in Pretrained Transformers. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022). Dublin: Association for Computational Linguistics.	2022	The original knowledge-neuron work: locating units tied to specific facts and editing them.	arXiv
[6]	Meng, K., Bau, D., Andonian, A. and Belinkov, Y. Locating and Editing Factual Associations in GPT. Advances in Neural Information Processing Systems 35 (NeurIPS 2022). Red Hook, NY: Curran Associates.	2022	A single fact rewritten directly in the weights: the Eiffel Tower moved to Rome.	arXiv
[7]	Niu, J., Liu, A., Zhu, Z. and Penn, G. What Does the Knowledge Neuron Thesis Have to Do with Knowledge? International Conference on Learning Representations (ICLR 2024). Vienna: ICLR.	2024	The correction: knowledge is more distributed than neat knowledge neurons suggest.	arXiv
[8]	US Senate Select Committee on Intelligence. Report on Russian Active Measures Campaigns and Interference in the 2016 U.S. Election, Volume 2: Russia's Use of Social Media. Washington, DC: United States Senate.	2019	A bipartisan record of a state using social platforms to meddle in an election.	US Senate
[9]	The Conversation. Why Romania's Election Was Annulled, and What Happens Next. Melbourne: The Conversation.	2024	The annulment after reports of coordinated TikTok promotion.	The Conversation
[10]	Anthropic. An Update on Independent Evaluators. San Francisco: Anthropic, 15 October.	2026	The lab's own terms: six METR staff, no publication veto, but redactions Anthropic defines, and all voluntary.	Anthropic
[11]	Shapiro, D. Mechanistic Interpretability... solved? Video. YouTube.	2026	An accessible walk-through of the Goodfire essay.	YouTube
[12]	The Decoder. China Plans to Tighten Export Controls on Open-Weight Models and 23 Key AI Technologies. Leipzig: The Decoder.	2026	English summary of the model thresholds: 10²⁶ operations, two trillion parameters or a global top-ten ranking.	The Decoder
[13]	Ministry of Commerce of the People's Republic of China. Spokesperson Answers Reporters' Questions on Revising the Catalogue of Technologies Prohibited or Restricted from Export (in Chinese). Beijing: MOFCOM News Office.	2026	The exemption for already-open models and automatic licensing for new open-source releases.	MOFCOM
[14]	Fane, G. The Padlock on the Library. froodx.net.	2026	My earlier argument for owning rather than renting intelligence, including the K-shaped economy.	froodx.net
[15]	Carew, S., Cooper, A. and Banerjee, A. DeepSeek Sparks AI Stock Selloff; Nvidia Posts Record Market-Cap Loss. Reuters, 27 January. London: Reuters.	2025	Nvidia's record $593 billion one-day loss after DeepSeek's open release.	Reuters
[16]	Chomsky, M. U.S. Military Commander Reveals How Artificial Intelligence Is Used to Accelerate Targeting during Strikes on Iran. Defence Industry Europe, 12 March.	2026	The commander of Operation Epic Fury on AI-accelerated targeting with humans holding strike authority.	Defence Industry Europe

Written by Geoff Fane with AI assistance for research, discussion and editing. The arguments, judgements and final wording remain Geoff's responsibility.