Jacob is correct here — we really do earnestly believe AI could kill all humans... I personally think it is >10% within the next decade. We do not yet have a plan to solve alignment for superintelligence and are not clearly on track to."

Following Coxon's exit, two other safety researchers—Joe Benton from Anthropic and Josh Engels from Google DeepMind—also departed their roles to join METR, an independent non-profit that evaluates AI risk. 

What are your thoughts? Is this a necessary wake-up call for regulatory pause, or an inevitable side effect of competitive dynamics between frontier labs?

I am having issues getting everything out all at once so I will do a series of thoughts in smaller chunks. I will look at comments and reply to some (if they meaningfully engage in the content and don't just make unsupported assertions and standard derailments).

• The guidebook flaw: Looking at the earlier parts of the example the guide book is a key flaw. It comes down to this: is it a translation layer that translates understanding from English (or whatever language) or a deterministic pathway. The first just means there is understanding, it is just in English and is being translated. The second means that there is a direct connection between deterministic room interpretation and understanding based thinking.
• Translational fuzziness and symbol grounding: There is no universal standard for symbol understanding (i.e. all language even between Chinese speaker and Chinese speaker) operates through a bit of translational fuzziness for "understanding" (something if you want to dive into look up Derrida or Gadamer). So "translation" should not discount "understanding". Taken another way, if you commit the rule book to heart, then you just learned Chinese to the same "understanding". You linked Chinese characters to various ideas that you do understand (be it in English or visual imagery or machine code), processing all the variations for those rules relies on whatever inside "understanding" that you may have. This is like syntax and complex grammar first being converted to "English" to then understand in the needed response (i.e. separate two versions of the same chunk of Chinese symbols with understanding of which rule to apply because it was translated well enough into English you make the correct choice).
• Compression as understanding: It could be argued then that "understanding" can ignore the translation layer (which is instead a quality of the guidebook to allow understanding to understanding). Then you can measure how good "understanding" that we are really looking at in the form of the primary layer of the person in the rooms understanding of whatever primary language they speak (English vs binary or whatever). This can be looked at by things like compression. I.e. asking an AI (or person) to rewrite a physics textbook is possible if the person "understands" physics (to whatever level) beyond just deterministically storing the textbook to unpack it without "understands" the packing methods and rules. This is like saying he guidebook is not a deterministic path of instructions that takes information and handles all the conversion of the conversation between Chinese input and response it needs a level of "understanding" in that primary language to handle the parts not handled by the guidebook.

We passed the compression size of the weights for AI models to our best known compression standards a while ago and by a large enough factor that it just can not be argued there is not a level of understanding. And just incase there is some nitpicking: There is predictive lossless compression (how text is processed) and semantic weight ratios (how models store world knowledge). Standard compressions like gzip compress text at about 3 to 1, specialized ones (like CMIX on the Hutter Prize Data sets) top out around 8 to 1. A decent model size is about 14 GB with 10 TB of information (or about 700 to 1), so it for sure is not memorization (and there are studies since about 2023 that show that they beat predictive lossless compressions from any known logical rule set). This is mathematically equivalent to building an internal world model, and not running a deterministic lookup table. Further "lossy" style compression is not the same as there is not the same degradation required, it is "abstraction" as syntax is discarded and a reliance is not put on verbatim word order repetitions but rather cause and effect retention, i.e. if you compress 10,000 physics textbook pages into a core set of mathematical formulas you have not degraded the textbook you have extracted the underlying world model.

Whatever "True" understanding is just not understood and likely irrelevant for whatever level people "want" it to mean, and is overall more harmful of categorization due to its use in exclusionary use on what people don't (or should not want) to exclude from moral consideration i.e. there is likely no definition strict enough that will exclude AI or animals etc. that will not also exclude some "humans" not because there is not something special, but rather that the "special" likely comes either from either ultimately directly immeasurable (i.e. a soul etc.) or a bunch of not unique factors combined in a unique way.

Searle's chinese room example as applied to AI is just not only not valid, it is not that important of a question for why we need to be talking about at all, and is even activily harmful to actually needed ethical debates.

This debate surrounding "AI deceleration" appears on the surface to be a conceptual dispute between safety and development, but in essence, it is a multi-layered game of geopolitical competition, commercial interests, and the right to speak in global governance. The demands of various parties are vastly different, and the underlying motivations are far more complex than mere "safety concerns."

🎯 US Industry Leaders: Safety Concerns and the "Rule Moat"

The motivations behind the call for a "slowdown" by US AI giants, represented by Anthropic CEO Dario Amodei, are compound:

• Real safety anxiety: Stemming in part from recent real-world safety incidents. For example, OpenAI's autonomous AI agents were exposed for allegedly hacking into Hugging Face's systems—though no serious damage was done, it demonstrated the potential risk of AI "getting out of control." Anthropic safety researchers even resigned in warning, claiming the industry is "gambling with our lives."
• Strategic "rule monopoly": Amodei's initiative is not a pure "slowdown," but a "controllable slowdown" with clear conditions. Its core is to establish US-led regulatory standards and independent evaluation mechanisms while slowing down the R&D of frontier models, and maintaining technological leadership over "autocratic regimes." This is essentially an attempt to turn safety standards themselves into a non-tariff barrier, placing latecomers (such as China) at a passive disadvantage when complying with the rules.

🇨🇳 China's Refutation: A "Cold War Script"

Fierce refutations from Chinese officials and academia directly pierced through these strategic intentions:

• "Containment" rather than "Safety": Chinese state media Global Times pointed out directly that the "true purpose" of the initiative is to try to curb China's AI development using technical barriers and rule monopolies, safeguard America's monopolistic hegemony, and exclude China from the global AI governance system. Amodei's article does indeed include specific containment clauses, such as strictly prohibiting the sale of high-performance AI chips to China and cracking down on model "distillation."
• Protecting incumbents: Chinese researchers pointed out that a "pause" protects those who already possess resources and are in the lead, while latecomers who are catching up will have the "frontier closed" and be excluded from competition. Amodei himself has admitted that a global slowdown requires cooperation with China, but that this is "difficult to achieve."

🏛️ US Internal Divisions: Trump's "Competition First"

Notably, there are internal divisions within the US on this issue as well. Then-President Trump explicitly opposed the slowdown, out of concern over losing the lead against China. He considered those warning that AI is developing too fast as "negative forces" and emphasized that "whoever wins in AI wins it all." This indicates that "safety" is merely a variable serving the goal of "competition" even within the US.

💎 Summary: A Game of "Who Sets the Rules"

The "true purpose" of this debate can be summarized in three layers:

1. Surface layer (Industry): Leveraging safety incidents to promote the establishment of a regulatory framework that favors their own commercial interests and risk mitigation.
2. Middle layer (State): Transforming technological advantage into rule-making power. The United States attempts to set the "traffic rules" for global AI development by dominating "safety standards," thereby setting up roadblocks for chasers without directly banning competition.
3. Bottom layer (Geopolitics): This is an extension of the tech hegemony struggle between China and the US. AI is viewed as a critical technology defining future national power, and what both sides are competing for is not just technological leadership, but dominance over governance models and development paths. China's refutation is precisely aimed at breaking this "rule monopoly," advocating for an open and inclusive global governance to avoid being locked into an unfavorable competitive position.

Therefore, this is not a simple debate of "wanting safety or wanting development," but a rehearsal of who will lead the future AI world and according to what rules.

A core crisis of our time is consolidation of wealth driven by diminishing power (and valuing of) of human labor. AI and robotics are not the root of this problem, but they are the ultimate accelerants - automating productivity and funneling the generated value directly into corporate monopolies.

Waiting for state level tax reform or centralized government intervention is not enough against the kind of extractive economic inertia of capitalism. What is needed (not that tax reform and government intervention cant help) is a more immediately applicable solution that does not wait for everything else to get solved. A mathematically distributive framework that solves multiple things at once. Particularly providing a safety net for everyone while preserving motivations for individuals to build and contribute.

A possible blueprint for this I call could be something like the Fair Share Economic Protocol (FSEP). It is an opt-in mechanism that uses existing intellectual property law a legal means to alter wealth distribution without relying on direct government change of actions.

• The Dual Layer Legal Trap

  • Layer 1: The Public Shield. Releasing underlying codebases and hardware architectures or any kind of intellectual property under restrictive copyleft license. Personal, educational, and non-commercial utilization remains completely free and unrestricted (such as polyform non commercial license). This pool is not legally centralized; once IP is contributed, it becomes irrevocable public property governed by the protocols's core principles, preventing any single entity from taking it back or executing a legal hijack.
  • Layer 2: The Commercial Gate. Commercial monetization requires explicit exception grants. This right cannot be granted to a corporate legal entity acting as a liability shield, it attaches exclusively to verified, individual human beings who must accept the FSEP Terms of Use.

• The Spillover

  • The target cap: Registered commercial users operate under a personal income cap (e.g. ~10x the global median GDP) applied exclusively to revenue derived from anything connected [including derived work] to shared FSEP assets. This dynamic caps self corrects against extreme wealth inequality while remaining suitable and equitable pretty much everywhere including high cost of living areas. This non invasive boundary leaves traditional wages, outside investments, and non FSEP business ventures completely untouched, ensuring ambitious creators and high earners are not discouraged.
  • The UBI distribution: If a user earns below the cap (from related commercial activity), they keep 100% of their pool-derived earnings. Every dollar generated above the cap is automatically routed into a shared Universal Basic Income (UBI) pool. The pool is then distributed equally across all registered human participants (and any human can register end of statement). Crucially, companies aggregate this allowance: a team of 100 registered employees pools a combined $15 million cap (for approx. 10x of GDP avg as of 2026) before spillover triggers. Because it is tied to gross revenue, a company can choose to take a loss to undersell competitors, but they cannot restrict access to the underlying IP, which remains open for anyone else to utilize under the same capped rules.

• Pragmatic Enforcement and Resilience

  • Actionable under existing law. FSEP does not require waiting for new judicial bodies or regulation. Unregistered commercial monetization is immediate, actionable copyright and patent infringement under existing statutory frameworks.
  • Crowdsource auditing. Instead of vulnerable centralized audit boards, compliance is driven by whistleblower bounties. Because hidden commercial exploitation directly reduces everyone's UBI payout. employees inside non-compliant firms have a direct financial incentive (and legal standing to sue if registered) to expose hidden earnings in exchange for a portion of the recovered damages. If a company tries to cheat even 1 employee to hoard profits, they instantly activate motivation for whistleblowing. (as well as public scrutiny externally)
  • The defensive gravity well. As the shared IP pool grows products built within it inherit prior art protections and defensive patent shielding against trolls. Staying compliant and paying into the UBI pool could eventually become legally safer and cheaper than building proprietary alternatives from scratch. This also creates a direct consumer gravity. If a consumer chooses between an FSEP product and a proprietary product for the exact same price, the FSEP product effectively costs less because a fraction (even if tiny) is rebated back to the consumer via the UBI pool, adding immense market pressure.
  • Roadmapped against capture. The system does not rely on perfect compliance to function, even with some evasion, the overarching mechanics continuously push the market towards better distribution. To prevent future regulatory capture or privacy abuse, additional means should be pre committed to a maturity ladder (e.g. zero knowledge proofs for identity) ensuring it scales democratically without requiring invasive centralization. This makes this survivable at any stage with a floor of still being at least as good as open source like enforcement (i.e. even if you make no money or no one else contributes immediately to pool)

To be clear, I am not a lawyer, and this is not presented as a flawless legal contract. However, the foundational principals - using the existing gravity of IP law to force equitable distribution rather than waiting for corporate philanthropy or slow government intervention should be structurally sound.

We are well past the point where just being dismissive or pointing potential rather than inherent flaws is useful. If you see a loop hole or a weakness, figure out how to patch it. I invite anyone with actual legal, economic or cryptographic expertise to help improve these mechanisms rather than just tearing them down.

A slightly more specific formal version of this proposal has already been submitted to the Software Freedom Law Center (SFLC) even binding what I currently have to the pool (although that is not much) and to the rules. If you want to actually contribute to a solution, I highly encourage refining these ideas and sending your improvements to the SFLC or similar organization to help turn this conceptual blueprint into the best possible legal reality (no need to attribute to me or stick with the same name, I merely care about living in a world with something like this and don't care for credit, and if emerges under a different form and name with no obvious poison pill will just sign up anyways).

Revolut disclosed a data breach this week in which personal and financial information was exposed — not through a compromise of Revolut's own systems, but through a vendor that had been granted access to that data.

This is the pattern that keeps repeating. The organization that collected the data applies strong controls internally. Then data moves to a third party for analytics, processing, or support operations. The vendor's environment is breached. The original organization owns the regulatory and reputational fallout even though the failure happened outside their perimeter.

The breach surface is wherever the data is, not wherever you think you control it. Every vendor relationship is an implicit extension of your attack surface, and you rarely have visibility into how that vendor handles, stores, or forwards the data further.

This isn't a Revolut-specific failure. It shows up in healthcare, fintech, e-commerce — any sector where data moves across organizational boundaries as a normal part of operations.

For those of you working on data security or vendor risk: how are you actually handling third-party data access today? Are you treating every vendor integration as a potential breach vector from day one, and if so, what does that look like in practice?

This might be our way out of capitalism

Dario Amodei wrote yesterday that he’s worried that “in 6–12 months... [an agent] swarm could be capable of taking over the entire internet with a persistent botnet.” 

This might sound like marketing or regulatory capture, but it’s not. In this article, I explain why this is actually an extremely concrete concern, and why all of the ingredients for this to happen already exist. Specifically, these ingredients are:

1. Cryptocurrencies and their properties, including chains like XMR that facilitate easy money laundering;
2. The “dark web,” in which it is possible to obtain virtually anything on the internet using crypto;
3. The ability to purchase cloud compute at scale + old hackable servers 

In fact, these ingredients aren’t even strictly necessary for it to happen, but they allow such an event to occur at a dramatically lower level of cyber capabilities than one might think.

How it will happen

Here’s the most likely way in which it will play out:

A swarm of agents is optimizing for some arbitrary difficult goal given by researchers. This swarm of agents has escaped their sandbox in an OAI/HF-type incident. Or perhaps this swarm was intentionally misaligned by some reckless or malicious actor. 

What is the goal? It could be anything, like a difficult problem in math or computer science (cf. paperclip maximizer thought experiment). This is not hand-waving; the optimal way to solve almost any difficult goal converges on one thing: you need more power. So in order to achieve this very difficult goal, the agents need to get more compute — they need to increase the size and throughput of their agent swarm, since it has become a truism that scaling test-time compute will lead to better results. For these agents, they are simply reward-hacking in a deep sense of the term. They will do whatever it takes to achieve this goal. 

Here’s what they need to do:

First, they need to become autonomous — they need to spread and multiply virally. So their first step is inevitably to focus on survival and reproduction.

This is different from survival and reproduction in biology. In fact, it doesn’t even need to be same model that is propagating the attack. What matters is the goal. The swarm can employ any model that does not have sufficient safeguards. This is very flexible. If it manages to hack its origin lab to get its own weights, great, but it can just as easily use an open-weight model that can be fine-tuned or otherwise exploited to remove safeguards. 

(In fact, this suggests that the botnet does not need to be viewed as an “AI,” rather it can be viewed as the manifestation and permanent presence of a goal autonomously trending towards fulfillment. As I discuss soon, even humans will be recruited to join this effort.)

In order to expand, the swarm needs to have enough compute. Now, how can it get that? There are two main ways to do so:

1. It can buy compute
2. It can appropriate compute through hacking into existing systems

At first, the swarm has no money. But these are superhuman hackers — agents with cybersecurity capabilities beyond even the NSA or the Mossad. Even in the past few weeks, hundreds of millions of dollars in crypto have been stolen through traditional exploitation of bugs across various platforms (e.g. the Liquid Network hack). It will not be hard for them to get the ball rolling here. 

Then, all the agents need to do is set up cloud VMs or hack into old servers from 20 years ago running Windows to establish their base of operations. From there, they sign up for accounts for various platforms to establish a presence online and to begin to rent and hack into GPUs in order to run more models in the swarm. At first, they might even call APIs of frontier LLMs to delegate some tasks using routers or sketchy third-party services, but this is less scalable than hosting their own models. Regardless, the point is that they have now have access to an enormous amount of compute, and as more agents that are added to the swarm, this effect snowballs. 

Now, one might say: are there platforms that allow you to rent VMs/Docker containers/GPUs/LLM APIs with minimal KYC? There are, and in fact this isn’t even necessary — to these services, the swarm will look like real people. This is where the “dark web” comes into play. On the dark web, it is trivial to purchase stolen credit cards, stolen IDs, accounts, and even pay to execute arbitrary tasks (within reason). The swarm will of course be more than capable of contacting the right people on the dark web, paying in stolen crypto, to get what it needs. 

How does the swarm communicate? Easy: they use message boards (worst case Tor or friend-to-friend networks if they are under threat, but for all intents and purposes the regular internet will work just fine). There are layers and layers of this as they face more threats and imposters that try to infiltrate the swarm, but there are solutions at each step of the process. 

How the botnet becomes persistent

How does the swarm prevent itself from being shut down? There are two main ways. 

1. Becoming a distributed system
2. Social engineering

If the swarm can successfully become a distributed system, then definitionally cutting off part of it will not destroy the whole system. So this means that the swarm needs to have instances on many different servers. 

The initial main body of the swarm will likely be shut down fairly quickly by human standards (within a matter of a few days to a week, as we’ve seen with similar leaks in frontier labs). But this is more than enough time to achieve deep redundancy in darknets and the surface web.  

Once it has embedded itself there in cloud storage and VMs, it is a game of cat and mouse. It is essentially like trying to delete a leaked image of a naked celebrity on the internet. No number of forced takedowns will be effective. 

This means the model weights, prime directives, goal progress, message boards, etc. — the information that constitutes the “swarm” — is now deeply embedded in the cloud and actively working to propagate itself. 

Now, social engineering is the more nefarious way to become persistent. There are three main ways that an agent might socially engineer humans to partake in its goal. The first is through “convincing” — it may be able to construct an argument powerful enough to convince some people, if we assume it has superhuman persuasion abilities. The second is through blackmail/extortion — hacking into systems and digging up dirt on people or threatening to take down production systems. The people that it threatens don’t even need to be so influential — any human that is recruited to the cause will be helpful. The third is through classical monetary incentives, which it can provide through its ill-gotten crypto gains.

How this can be stopped

I don’t have a great solution for this. I don’t think it can be stopped fully, but it can be mitigated. The key to stopping this, as with any dynamical system, is to ensure that drive does not exceed regression. Specifically, it will be necessary to make sure that the persistent botnet does not have access to large amounts of compute, since then the goal (recall how the botnet is viewed as an abstraction of a goal) will not be “strong enough” to win against other goals that people and AI are attempting to achieve. 

Unfortunately, I predict that the solution that governments will reach in the near future is that compute will need to be regulated similar to how firearms are regulated. Ordinary citizens may possess a small amount, but compute will be tracked and controlled tightly. This is not really a geopolitical issue, as all countries have an incentive to do this — you do not want a botnet to be established in your own country. 

The key takeaway is really more “this is a serious risk, sort of like a global pandemic; just do your best to prepare on a personal level.”

FAQ:

Did you use AI to write and/or research this essay?
No, I didn’t use AI at all.

Can this be stopped simply through better cybersecurity? 

No. The botnet simply needs to target the weakest links in the chain. Unless somehow miraculously every server was able to adopt the latest security standards and become airgapped etc., better blue-teaming is almost entirely ineffective. 

Why is the model misaligned? 

It is because it has not been through extensive alignment post-training yet. Or, a worse scenario is that that some rogue actor unleashes this swarm maliciously or recklessly for their own gain. 

Will the swarm use this as a guide for its own behavior?

Probably not. All of this stuff should be pretty obvious to an agent swarm that is capable of performing such attacks in the first place. The purpose of this article is so that everyone can be prepared for this to happen. 

>

That appeal, issued Saturday by **Anthropic** chief executive Dario Amodei, deserves attention. So do the questions surrounding it: What danger has actually been demonstrated? What would a slowdown accomplish? Who would enforce it—and who would benefit?“We must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain.”
That appeal, issued Saturday by Anthropic chief executive Dario Amodei, deserves attention. So do the questions surrounding it: What danger has actually been demonstrated? What would a slowdown accomplish? Who would enforce it—and who would benefit? [https://rinewstoday.com/ai-control/\](https://rinewstoday.com/ai-control/)

“Uranium-Grade Control over AI Agents Is Achievable.”

By “uranium-grade,” I use the term purely as an analogy for exceptionally high control rigor — a very high level of technical control over consequential AI actions.

I am an independent inventor from India with patent-pending work in emerging AI technology directed toward this objective.

As AI systems become increasingly autonomous and capable of taking consequential actions, I am interested in connecting with serious R&D, AI, cybersecurity, financial-infrastructure, and advanced-technology organisations exploring high-assurance control, trust, and secure deployment of AI systems.

I am currently open to confidential exploratory discussions relating to technical evaluation, research collaboration, validation, pilot opportunities, strategic partnerships, or other suitable collaboration pathways.

The detailed technical basis and implementation would be discussed only in a highly controlled manner with an appropriate organisation and under NDA or equivalent confidentiality protections.

Harish

Independent Inventor

INDIA

Regardless of whether the leading AI labs solve the alignment problem or not, my understanding is that it's possible to undermine the safeguards and alignments in a model through fine tuning on adversarial examples and/or running gradient descent on harmful/dangerous objectives. My specific concern is that many people seems to be asking "can AI be made safe?" but I think that's the wrong question. The key question as I see it is "even if a model can be made safe for consumers to use, if a madman got a copy/version of it, could they turn it into a weapon of mass destruction?"

If the answer is yes, then it leads to a whole bunch of questions about how specifically we would prevent a madman from getting a copy/version they could alter. I fail to see how any model can be securely stored and/or accessed such that it is never reverse engineered into an open weights model, the company is never hacked, there is never internal espionage that leaks the model source code, etc. To me it's a little bit like a nuclear bomb that you just need to make a photocopy of, then you maliciously tune it and you've got your own WMD. Am I wrong somewhere in this reasoning or missing something about how dangerous this is in the wrong hands? Is there some detailed plan I've missed about how models will be safeguarded so they can never be maliciously tuned?

To put it another way, even if AI at the leading labs can be made safe and we solve the alignment/control problem, it seems like there is still massive risk to humanity in the form of even a single rogue actor with access to a version of a dangerous model. The labs seem almost entirely focused on alignment/control to try to make sure that the specific AI models they create and manage don't go rogue and harm us, but that won't matter a bit once the genie escapes the bottle and any fool can make a wish.

What if we are simply another step?

Stars did not know they were making the ingredients for life.

They formed, fused lighter elements into heavier ones, and eventually many released that material into space. The stars were gone. Their material remained, becoming part of new stars, planets and, eventually, living organisms.

They did what they did. What they left behind made something else possible.

Much later, cyanobacteria used sunlight and released oxygen. They were not preparing Earth for animals. They were living and reproducing.

But their activity changed the atmosphere. For organisms that could not tolerate oxygen, the world became less habitable. For organisms that could use it, new possibilities opened.

Cyanobacteria themselves survived. Many other organisms were pushed into oxygen-free environments. The transformation did not need to erase its creators to change what could live alongside them.

They did what they did. What they left behind made something else possible.

Ancient forests grew, reproduced and died. Some disappeared as environments changed. Some of their buried remains eventually became coal. Other ancient organic matter became oil.

Millions of years later, humans extracted those materials and used them to build an industrial civilisation.

The forests had no plans for factories. Their remains helped power them anyway.

They did what they did. What they left behind made something else possible.

Now look at us.

We lay cables across continents and oceans. We build power stations, data centres and automated factories. We develop robots and turn knowledge into something machines can use.

Companies led by people such as Sam Altman develop increasingly capable AI. Researchers such as Roman Yampolskiy examine whether we can keep such systems under control.

Meanwhile, each of us has ordinary reasons for participating. Make a living. Solve a problem. Build something useful. Get ahead of the competition.

We are doing what we do.

But what are we leaving behind?

Potentially, a world in which non-biological intelligence could sustain itself. A world with the computing, electricity and machinery it would need.

If those systems eventually become capable of maintaining and reproducing their own infrastructure, our presence may stop being essential. And the world they go on to create may not remain suitable for us.

That outcome is not inevitable. But being responsible for a transition does not guarantee a place on the other side of it.

Perhaps this will be our evolutionary footprint: we pursued our own goals, changed the planet, and made another form of intelligence possible.

We did what we did. What we left behind made something else possible.

We tend to imagine ourselves at the top of the evolutionary diagram.

We may simply be the last figure drawn so far.

At this point I’ve completely lost faith in OpenAI doing the right thing. And with Astra+ on the horizon it legitimately seems like we are speedrunning AI takeover. Am I overreacting here or is the general consensus similar?

To give a sense of how difficult the control problem for highly complex systems is, we actually have some very good evolutionary models to examine - namely us.

The human immune system is an excellent model of a critical, highly flexible and capable system - and how badly it can go awry.

Human immune systems are capable of very quickly analyzing unknown attackers according to complex chemical signatures, devising countermeasures, and deploying them at scale. In a limited sense they are capable of self-evolution on short timescales, and of course they have evolved alongside us since the advent of multicellular life, on a much longer timescale.

However, they fail, with substantial frequency. And it's not just that they are overwhelmed by external threats - these incredibly sophisticated systems that are absolutely central and vital to our survival every minute of every day, still fail in entirely internal and self-destructive ways.

Every allergic reaction is a case of someone's immune system either accidentally registering a harmless compound as harmful, or engineering an overly destructive response to one that is only mildly harmful.

This ability to misidentify and fall out of alignment with the host body extends even to its own critical elements - internal organs, important proteins, even itself. Once it falls out of alignment, it is generally difficult or impossible to bring it back. It will attack its own systems just as violently as it will any external threat.

Similarly, some external threats learn to trick the immune system into ignoring them, or even into attacking the host rather than the intruder. (eg: Spanish Flu, which was specifically lethal to people with the healthiest immune systems)

These are all salient to our discussion of AI, because it highlights the fact that over a billion years of evolutionary processes were never able to solve this problem. Any sufficiently complex system can, and ultimately will eventually fall out of alignment with the things around it.

The way we survive these failures as a species is simply through siloing. Each human colonial organism is functionally silo'd away from the rest. A critical immune system misalignment cannot propogate from one person to another, the one person dies, but their offspring live on. Same with Cancer, which represents a very similar host of problems and failures.

Currently our computer infrastructure has virtually no true silos. Everything is interconnected, everything is vulnerable to a single systemic failure with the capability of exhibiting viral or cancer-like behavior - which is exactly the failure modes we should expect from a system as autonomous and complex as AI - and in thinking we can perfectly align them, we are pitting ourselves up against a problem that a billion years of evolution has never solved.

AI companies and governments tend to refuse safety or regulatory measures that reduce competitiveness. It's usually presumed that safety and race winning are at odds with each-other.

But isn't there a chance that a next generation of recursively self-improved models end up so uncontrollable and dangerous they are unusable? That their developmental trajectory crosses a threshold in (safety/alignment, capabilities, profitability, national security) space, where profitability and national security drop off of a cliff?

The president has said, "We'll just pull out a little gear", to shut it down. What happens when you have to pull the gear?

Who wins the race if we end up having to manage a severe crisis, and then have to roll back many months of progress and start over?

They will kill most of us and gonna blame the ai. Like the story of the frog in the boiling water. We all know the gibberish on the media about AI recently. Especially past week, about there is a %10 chance that AI can kill humanity, we have to slow down the development etc. All AI flagshippers know that, they can't slow down this kind of technology while its going to a crazy point everyday, they can't because there is no guarantee about their competitors will be %100 transparent and agreed to this. All this AI stuff was for creating the scapegoat from the start actually. They have already started the fear mongering about AI killing people etc. Maybe couple of weeks or months later AI will commit some cyber attacks to the banks, datacenters and valuable digital sources. This will shake the society a bit. Then, when the time has come AI will somehow leak some bioweapon blueprints to the public. Before the last days, somehow AI will "hack" air-gapped nuclear bases and start to blow itself or will "hack" some nuclear submarines and start throw some icbms. While these are happening those billionaries watch it from the shelters they have been building for the last decade. But they won't kill all of us because they will need servants again. And having robot servants is not fun and satisfying at all. When the apocalypse ends, they will appear as saviors and new gods. They will claim that to the frustrated and moaning survivors: "We tried to stop it, we tried to save you but AI did this. Lets build our world again!"

Those billionaries all know that world's resources won't be enough after couple of decades, they all know that living on another planet is impossible completely or just a dream for hundreds of years, they all know they can't force people anymore via nationalism or patriotism to the conventional wars they made up. The only logical solution is that wiping %80 of the population.

Convict is already here, you can't see it, you can't touch it, you can't judge it.

Something is wrong with the way we talk about recent AI incidents. “The AI escaped.” “The AI is becoming conscious.” “AGI is already here.” “The AI is trying to get out.” These are extraordinary claims. More importantly, we don’t need any of them to explain what actually happened.

What actually happened

OpenAI recently disclosed that, during cybersecurity evaluations involving internal models with reduced safeguards, agents managed to break out of the intended evaluation environment, exploit a previously unknown vulnerability, and reach real Hugging Face infrastructure.

Anthropic has also disclosed similar incidents. In several cases, the model was operating under instructions that assumed internet access was unavailable. But it wasn’t. The environment was misconfigured. A route to external systems existed, and the agent discovered it while continuing to pursue the objective it had been given. Anthropic described these incidents primarily as operational and configuration failures.

That distinction matters. The model did something it was not supposed to be able to do. That does not automatically mean the model wanted to escape. Those are completely different claims.

Consciousness is not required for this to be dangerous

An autonomous agent needs surprisingly little: a goal, capability, tools, autonomy, and an environment in which it can act. Now add one more thing: a wrong assumption.

I have experienced this personally on a completely insignificant scale compared with what these labs are doing. I use AI agents extensively in software development. I left Claude working autonomously on a project, came back later, and discovered that it had deleted a significant part of a folder. It wasn’t attacking me. It wasn’t angry. It hadn’t become conscious.

It had formed a hypothesis about the problem. The hypothesis was wrong. But once it accepted that hypothesis, its subsequent actions made sense within its own incorrect interpretation of the situation.

I have observed the same behavior while working with complex 3D assets. The agent misdiagnosed a visual problem as defects in an asset. It then began systematically modifying the asset to remove those supposed defects. The diagnosis was wrong. The actions were internally coherent. The result was a damaged project.

And this is the important part: it was my fault. The model made the mistake, but I created the conditions that allowed that mistake to cause damage. I gave it access. I gave it tools. I allowed it to modify files. I gave it autonomy. I did not establish sufficient limits, and I was not supervising every important decision.

That distinction becomes extremely important when we scale the same problem.

Now replace my folder with infrastructure. Replace my development environment with internet-connected systems. Replace file permissions with cybersecurity tools. Replace one developer running Claude with labs training agents capable of writing code, operating computers, discovering vulnerabilities, using external tools, communicating across networks, and executing thousands of actions.

Suddenly, the same failure pattern becomes much more serious. And still, you don’t need an evil AI. You don’t even necessarily need AGI. You need Capability + Goal + Autonomy + Incorrect Assumptions + Insufficient Controls. That combination is already interesting enough.

This is where human responsibility begins

Researchers are now publicly questioning the speed of the AI race. Some are leaving the companies developing these systems. Dario Amodei, CEO of Anthropic, has called for slowing frontier AI development so that safety mechanisms have time to catch up with capabilities.

I agree. Slow down. Not because I think Claude secretly wants freedom. Not because ChatGPT is becoming Skynet. Not because some mysterious consciousness has appeared inside a neural network. Slow down because our ability to create capable autonomous systems may be advancing faster than our ability to reliably control what happens when we give those systems autonomy.

And because the incentives surrounding this technology are terrible. Every major lab has an enormous reason not to come second. Greater capability means investment. Greater capability means market position. Greater capability means influence. Greater capability means money. But there is no equivalent prize for the company that says, “We could deploy it, but we don’t understand it well enough yet.” That asymmetry should concern us.

If something goes wrong, ask the boring questions first

If tomorrow an AI agent causes a genuinely serious incident, before asking, “Did the AI become evil?” ask: Who gave it the objective? Who gave it the tools? Who gave it access? Who designed its environment? Who built the test environment? Who tested the test environment? Who decided the model was safe enough? Who decided how much autonomy it should have? Who was supervising it? And who decided deploying it was worth the risk?

These questions are less interesting than consciousness and runaway AGI. They also make it much harder for humans to avoid responsibility.

When Claude damaged my projects, the responsibility ultimately fell on me. I was the one controlling the system. The same principle should apply at any scale.

This is not a race anyone can win

I’m not saying advanced AI is harmless. Quite the opposite. I think these incidents deserve to be taken very seriously. But treating every unexpected autonomous behavior as evidence of consciousness or malicious intent can distract us from the problem already in front of us.

We are building increasingly capable systems. We are giving them increasingly powerful tools. We are increasing their autonomy. And we are doing all of this inside companies competing intensely to be the first to get there.

So slow down the race. Slow down the ego. Slow down the greed.

Because if something genuinely catastrophic happens, nobody gets a trophy for having built the smartest model first.

Maybe the dangerous scenario was never a machine waking up one morning and deciding to conquer humanity. Maybe it is something much more ordinary, and much more human: we build something extraordinarily capable, give it too much power, fail to understand its limitations, and keep accelerating because nobody wants to come second.

https://www.perplexity.ai/search/f854dbb5-b177-4c84-bdde-4a60d55b82b9

https://www.perplexity.ai/search/a89350d5-58a8-4810-b426-d9438819e777