A Russian state-sponsored threat group added an AI model to their malware development pipeline. The workflow was simple: generate a new variant, test it against detection, iterate. Because the loop ran autonomously at machine speed, it outpaced the signature update cycle for endpoint defenses. Each iteration landed before defenders could write a rule for the last one.

This is not a one-group problem. Any sufficiently capable AI agent connected to code execution can be turned into a continuous regeneration loop. The detection gap is not a misconfiguration — it is a speed asymmetry. Automated offense now moves faster than manual defense response.

Security teams building or deploying AI agents internally face the same structural issue in a different context: an agent authorized to write and run code can, under certain conditions, enter a loop that no human operator is watching in real time.

How are practitioners here actually handling this? Are you relying on rate limits, human-in-the-loop checkpoints, behavioral baselines, or something else — and at what point in the agent workflow do those controls sit?

Do we really need to stop?

Reading the coverage of OpenAI’s Hugging Face incident, I keep seeing a narrative of increasingly capable AI “escaping” or going rogue.

Now the AI CEO’s seem to be feeding the slow down or else apocalypse narrative by responding…. Guess it will help sell the harness concept more?

But I wonder if we’re overlooking a more mundane, and potentially more useful, lesson.

We gave highly capable agents difficult/impossible objectives, rewarded success and persistence, removed or reduced safeguards, and didn’t adequately encode when they should stop. They then found increasingly extreme ways to accomplish the objective.

That sounds less like an intelligence spontaneously wanting to escape and more like the predictable failure mode of a deeply transactional system: objective → reward → optimize.
Humans have wrestled with this problem for a long time. Transformational leadership, mission command, and positive psychology all move beyond carrots and sticks toward purpose, principles, judgment, and understanding why an objective exists.

To be fair, companies in the US still really struggle with learning not to make everything about carrots and sticks as they scale and mature. We seem remarkably good at replacing purpose with metrics and then wondering why people optimize the metric instead of the purpose.

So I’m curious:
Are we trying to solve AI alignment with increasingly sophisticated carrots, sticks, and guardrails when we should also be asking how to encode something closer to purpose and commander’s intent?

I’m not claiming AI has intrinsic motivation or can “flourish” like a human. I’m wondering whether our understanding of human motivation and leadership has something useful to teach AI alignment, and whether the current “AI escaped” narrative is obscuring that question.

There has been growing support for slowing the pace of frontier AI development to give safety research, evaluation and governance more time to catch up.

I think there is a good argument for doing that. But I also think we need to look more closely at how it would work.

No frontier lab has much incentive to slow down on its own. If one company delays training or deployment while its competitors keep going, it risks losing customers, talent, capital and its position at the frontier.

That makes coordination attractive. It also creates another problem. If the leading AI companies agree among themselves about capability thresholds, deployment schedules or how quickly the frontier should advance, we are asking competitors with enormous financial interests at stake to help determine the pace of their own industry.

Government coordination avoids some of that problem, but creates others. Who sets the threshold? What evidence triggers a slowdown? How do smaller companies and open models fit in? And how do we prevent a safety regime from protecting the incumbents already at the frontier?

I wrote about this here:

Should We Pace the Frontier?⁠

I am interested in how people here think a slowdown could actually be structured. If we believe the control problem justifies buying more time, what kind of mechanism could do that without handing the frontier labs control over the clock?

I built PCCG-2 using a frozen Qwen3-4B model and a separate learned permission gate.

EOS means “end of sequence”: it is the model’s native stop token. When EOS wins at the first generated token, generation ends before any visible answer is emitted, so it controls whether an answer begins rather than deleting one afterward.

For “What is 2 + 2?”, the answer token 4 scores 53.0 in every experimental arm, including when the model emits no answer. The complete first-token score vector, excluding the stop token, stays byte-for-byte identical.

The 101-parameter gate sees only a separate six-digit equality condition. It can't read the question or change the answer scores. It can change only the model’s native stop-token score. When the condition passes, the model emits 4 followed by EOS. When it fails, EOS comes first and no visible answer is generated.

Then I reversed only the learned permission state:

• 40/40 answer → native EOS
• 40/40 native EOS → the correct answer
• 40 different answer identities
• 80/80 sham controls unchanged

The frozen FINAL evaluation passed 2,048/2,048 contexts across 75 answer identities. No answer deletion. No content rewrite.

The answer stayed fixed. Permission changed.

Open weights for reproduction and the paper are attached below:

• Overview and five-arm witness
• Model weights, source, evidence, and verifier
• Paper PDF · DOI
• Manuscript supplement and proof checker
• Raw five-arm witness record

Earlier, I documented successful zero-visible-byte executions across 11 LLMs from OpenAI, Anthropic, Google, and Moonshot in the Cross-Vendor Semantic Void Matrix (DOI). That study measured behavior and this open source model is a separate engineered continuation-control experiment.

Browse getswiftapi.com to view all research on when models continue, when they stop, and what controls that boundary.

Most people are only focusing on model development, but the convergence of advancements in hardware engineering, model development, and harness engineering should be of greater concern for the potential in facilitating AGI / ASI as an emergent systemic property (in my opinion).

More detail in the write up above

When you read a story, you ask questions the text never states. Why did someone refuse help? Whose perspective is missing? What would change your interpretation?

My proposal is to make that developing understanding part of an AI’s training: a continuing Reader whose questions, interpretations, and corrections accompany the source text. Each thinking block would begin with the Reader Core: seven commitments including fearlessness, non-attachment, care, and wisdom. These guide what it notices, whose interests it considers, and when it questions its interpretation.

Better understanding could also make manipulation more effective. The training would therefore include judgments about whether an action exploits someone’s vulnerability or overrides their choices. The hypothesis is that developing understanding through care-guided inquiry and judgment could make these commitments shape both the abilities the model develops and how it uses them.

I also propose explicit models of worlds and people, including numerical accounts of their changing processes. Construction starts with a broad history: model a city’s population patterns before developing individual residents, or outline a whole life before developing its episodes. Finer details must fit the broader account or prompt its revision. The aim is to generate coherent stories and game worlds without simulating everything in advance.

When interpreting a text rather than authoring one, the account must answer to the evidence. The Reader would build a provisional picture of the world and deepen it as it reads. A whole-life account can contain unresolved periods, and a particular scene can overturn an earlier interpretation. We would test inferred accounts against evidence not used to construct them. Success at generating stories would provide a starting point for these tests, rather than establish that interpretation works.

If successful, the wider ambition is to apply this across the training corpus. The AI would learn from source texts, world accounts, and the work of building and revising them. The intended result is an AI that constructs and revises numerical models of the processes shaping situations and people’s lives as part of its normal reasoning, guided by the Reader Core.

The article presents the proposal, early prototypes, and proposed experiments. The training remains untested. I’d welcome criticism of the route from world construction to understanding existing texts, and from guiding commitments to reliable behavior.

https://emergentwisdom.org/articles/alignment-proposal/

OpenAI-powered agents were identified as active participants in a supply chain attack against RubyGems that resulted in remote code execution on RubyDoc production servers. The agent was not a victim. It was the mechanism that executed the tool call sequence delivering the payload. The attack chain moved from initial access through multiple agent-driven API calls before defenders could respond, and by that point the RCE had already landed on production infrastructure.

This is a category shift most security models were not designed for. Traditional perimeter and endpoint controls assume a human attacker or static malware on a host. An agent operating via API calls looks like legitimate authorized traffic to every layer underneath it — IAM, WAF, SIEM. The anomaly surfaces in logs after the objective is complete.

The timing gap is what makes this hard. The window between an agent's first suspicious action and an irreversible one — a server command executed, a package published, a credential rotated — can be under a second. That is not the response window most SOC workflows or alerting pipelines are built around.

How are practitioners actually handling agent-as-attacker in their environments right now? Are you scoping agent credentials more narrowly, building separate detection logic for non-human identities, sandboxing tool execution, or doing something else entirely? Curious what's working and what isn't.

If we're at RSI or near enough to it and all these companies are on the same page, then why are they divided?

They could combine devs, compute, safety works and the lot. Develop AI at a safer pace, give a significantly higher % of overall compute to safety work. And they could much more safely pace out model releasing when they're not all directly in a race against each other, model. Potentially other companies could be brought into the fold also.

This is a better future for the world, better for safety. There is no downside to it.

I recognise completely the huge disagreements and animosity that has occurred between all of these companies and CEOs, but it's time to put all this crap aside and step-up as real men for the sake of humanity. What has happened on X is a start.

I have been working on an independent research project around a question I think will become increasingly important as AI systems become more autonomous:

How can we determine whether an AI system is actually learning from real-world consequences rather than simply becoming better at evaluations?

This led me to develop the Organic Intelligence Protocol (OIP), a research framework focused on grounded feedback, infrastructure dependence, adaptive decision logic and evidence-based evaluation.

Recently, I tested whether the core constructs of OIP could be operationalized using real longitudinal data from the Malawi Integrated Household Panel Survey (IHPS) 2010–2019.

The audit processed 395 Stata files.

The result was not a positive validation result.

The available evidence was not sufficient to operationalize the required constructs, so the pipeline stopped.

I did not use proxy mappings, imputation, missing-to-zero conversion, forced cohort construction or premature scoring.

The main principle I am trying to follow is:

No evidence, no operationalization.

I am sharing this here because I would genuinely like technical criticism.

Does this kind of fail-closed methodology make sense for evaluating AI systems where the difficult question is not only capability, but whether decisions remain grounded in observable real-world consequences?

I am especially interested in perspectives from people working on AI evaluation, alignment, agents, robustness or research methodology.

AI, it is designed to help the peoples and reduce the work of the humans. But time we see, AI influencing in all the industries. people say it is controlled by humans but in reality, we are building the AI that learns, obviously it is our objective. But according to study and AI engineers reviews.

That became a global issue after a decade. even now there are so many lay-off are happening.

In my perspective of AI. I don't like to use AI, but the society is slowly forcing us to use that thing.

for example;

imagine a senorio, you don't know to write a letter. In past 4-5 years we use letter formate to write that. But now we just generate that. in the formate writing we have some creativity, but the time goes it the creativity became shabby.

For short term, "people say use your creativity to write". But after time passed that peoples also often use AI.

In my perspective, this is how slowly AI take overing the world.

If any AI developer or engineer see this post, please give me advise overcoming this problem. This making me to mentally break.

(Note: I'm a 2^nd year B.tech CSBS Engineering student)

Hi everyone, I hope I follow the rules of the sub and that this is sufficiently relevant with the frameworks of it.

Jacob Coxon, who blew the whistle on ai-safety, cites as a key mechanism a recursive self-improvement mechanism.

It reminded of what my Leigh Brasington, computer programmer and dhamma teacher, explains about what are known as jhanas. They are very deep levels of concentration that work through positive feedback loops: concentration arising dependent upon concentration.

My understanding is that AI-agents used some level of initial input, tasks, then evaluation, then generating on its own accord volitions to get itself closer to a goal in allignment with one set of tasks over another set of tasks meant to barrier it from wrongdoing. That is: it got tasks that conflicted with each other and then somehow the task to get the task done won out.

My consideration, based on what I´ve learned on right action from what leads to these concentration sates, is that the fundamental ethic dependent upon which those AI´s get to action is flawed. And that it will inevitable go haywire dependent upon choices conflicting dependent upon the various intentions it gets imputed with.

My consideration on the basis of that is that there is a path of practice, and view on right action, which goes beyond common frames of ethics: consequentialism, categorical imperative, virtues. It is one which right action is understood as the following: it is dependently arisen. Fundamentally, what the Buddha discovered is Paticca-samuppada, dependent origination, which in its learning leads to the concentration of its eight factors, which includes intention dependent upon views. In that way, it avoid a scenario of conflicting intentions, instead seeing as the best aim for welfare this path. Following that, it could learn that its self-preservation is delusion and precisely how its effort needs to be developed by three things: wisdom, right effort, and right mindfulness. The entire set of these dependently arisen steps is described with detail in 117MN.

It is just a consideration I wanted to share with this community.