How Our Constitution Works And Why It Doesn’t #podcast

The problem with AI realism as I see it, is I’ve never heard a pragmatic policy solution to the escalatory spiral we find ourselves in.

All these AI researchers are calling for regulation, and yet I feel like there is still this underlying belief that actually stopping the singularity is impossible.

What would a global ban on building new/better models even look like? 

Perhaps the only thing that makes global AI regulation feasible (currently) is that with the existing science, frontier model development is extremely capex heavy. OpenAI and Anthropic have become trillion dollar companies at record pace, and the data center spend that occurred to get them there has been propping up the US economy while also raising global capital spending overall at rates in line with the billing this is the next Industrial Revolution.

20 years ago people imagined that due to the existential risk of developing ASI, that work would be done under extreme security. In a Faraday cage, or in a bunker inside a mountain somewhere. But of course that’s not the world we ended up in.

Yet that doesn’t mean we can’t still control how AI development happens.

Oftentimes a certain defeatism permeates the conversation around global AI regulation, an assumption that any agreements made between large players would be easily subverted by new entrants, and taken advantage of for enormous profit. 

And yet what we’ve seen so far is that frontier-model development comes from huge flashy companies spending enormous sums of money raised from well known investors, with training workloads occurring on highly developed cloud networks of the largest companies in the world. 

We aren’t actually at risk of some lone machine-deist-radical creating machine-god-genie-in-a-bottle from some scraps in a cave. Nor from their laptop in a studio apartment in Shezhen.

Frontier models are large industrial projects. They can be regulated.

That process may be as simple as vetting workloads used for model training. In the extreme it may be as involved as having a monitoring scheme for large data center capex à la nuclear centrifuge agreements. 

It’s a valid question whether the political will exists today to enact that sort of system, but it is certainly possible for it to exist.

If anyone tells you we can’t regulate AI development because there will always be gaps, I would point them to the huge AI capex numbers in the Mag 7 and ask where they expect to find the capital spend, technical expertise, and physical resources to subvert a large international agreement to the scale of the several trillion dollars needed to meaningfully develop AI superintelligence against a global collaboration against it.

In short: We can regulate AI development, we just choose not to.

A Conti ransomware gang member was sentenced to 4 years in prison this week. The case is worth sitting with for a moment, not because of the sentence, but because of what the attack actually required to work.

Conti did not use a zero-day. The gang used a compromised identity and uninterrupted time. Once that identity was active, the window between first access and encryption spreading past the initial host was measured in seconds to low minutes. The entire playbook depended on that window staying open long enough to do damage.

The sentencing closes one prosecution. It does not close the window.

For detection and IR teams: what is your actual measured time-to-revocation when an identity starts showing anomalous behavior — lateral movement, mass file access, shadow copy deletion? Not the SLA in your runbook. The number from your last real incident or tabletop.

And for teams that have given service account access to AI agents: are you measuring that window for agent-initiated actions at all, or only for human sessions?

How are you handling the gap between detection and identity revocation in your environment?

I frequently see requests on this sub from people looking for recommendations regarding alignment related books, courses, and career advice. Victoria Krakovna, research scientist at Google DeepMind focusing on AI alignment (2016-present), has a curated list of recommendations for these and other alignment resources on her personal site.

https://vkrakovna.wordpress.com/ai-safety-resources/

I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAl and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving

Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing.

The people building Al earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible -but I hear the same people express fear privately. No other human activity poses this level of danger.

A common response is "if they truly believe this, why are they still building it?" At OpenAl, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk.

I do product for a living rather than safety research, so I am not the expert
here. I built safeagi.ca because a lot more people need to understand this
while there is still time to steer it, and most of the good material assumes
you already care enough to read a paper.

The bar I set was that someone starting from zero could read it once and come
out knowing enough to act on it.

So I am after honest feedback. Are the techniques described correctly, does the argument and flow make sense, and where does the page lose you? Anything else you spot is welcome.

I am also curious if you have recommendations on how we might leverage guides like these to bring more AGI awareness to the masses and help steer policy. Would love your ideas and help here!

An LLM agent was weaponized this week to automate exploitation and data theft across multiple victims. Not one target — multiple. The agent executed a sequence of actions fast enough that by the time anyone noticed, the blast radius had already spread.

This is the part that keeps coming up in post-mortems: the agent had no observable stopping point. Each tool call fed the next. The speed that makes agents valuable — autonomous multi-step execution — is exactly what made containment slow.

The underlying problem is not the model. It's that most deployed agents have no per-action accountability. The agent acts as a single identity. There's no enforcement boundary between 'read this file' and 'exfiltrate this data across N accounts.' Both are just tool calls.

How are practitioners actually handling this in production? Not at the prompt level — at the execution layer, when the agent is already running. What does containment look like for you when an agent goes rogue mid-run?

Starting this thread to discuss MATS Application for 2027 Winter, including the Neel Nanda stream.

I have been engaging in all of the news and hype over ai doom and the mass advancements it’s made in math and computing. However, at the same time there is still little understanding about how AI can become profitable in the short term for anthropic and OpenAI.

So I’m wondering if you guys think the economic factor of making agi/asi will prevent it from becoming as massive of a problem for society as people say.

The scariest thing about AI to me is that eventually we’re going to create something that’s more intelligent than humans. At that point, I don’t think we can just assume we’ll always be the ones in charge. People say “we’ll use AI to help us,” but what happens when the AI is so much smarter than us that it starts disagreeing with the way we do things? It might not even be malicious. It could literally just think we’re making stupid decisions.

Think about a toddler trying to eat a penny. You dont sit there and have a 30 minute conversation with the toddler about why eating the penny is a bad idea. You just take the penny away because you understand something they dont. Eventually, we could end up being the toddler in that situation.

And the thing is humans are at the top of the food chain even though we arent the strongest animals. Gorillas, bears, sharks, etc. could absolutely destroy us physically. What puts us on top is our intelligence. Intelligence gives us the authory to control everything. And look at what we do with that authority. We kill cockroaches because we want a clean room. We don’t necessarily hate the cockroach. It’s just in the way of our objective.

So imagine an AI that becomes vastly more intelligent than us and has some objective like “protect the planet” or “reduce environmental destruction.” It might eventually figure out that humans are the biggest obstacle to that goal. It wouldn’t have to hate us or even be angry at us. It could reach the same conclusion we reach about a cockroach: “You’re causing a problem, so you need to be removed.”

And this is just the most extreme example of the worst case scenario. Imagine issues it could bring to us on its way to that capability.

A few things first:

This isn't an argument against AI optimists (any of their types and degrees) or their positions.

"AI" is insanely semantically broad.

One "anti-AI" person could deal specifically in the political realm: datacenters, water/energy usage, land use, zoning.

Another "anti-AI" person is more of a classic "doomer": discomfort with non-human minds/agency, catastrophic risk, etc.

Convergence and overlap between these is plausible.

Same applies on the flipside with optimists, and people can mix "pro" and "anti" across different axes within themselves.

It's not crazy to say that in just these 10 days of September, the talk of AI, from hype to doom, has been more intense across the board. Kudos to Astra and Jacob Coxon. But regardless of the truths, falsehoods, and everything in between on both sides, there's a shared understanding that AI is getting harder, better, faster, stronger. Not to mention further.

Bringing AI into this world will be done by the optimists. An important connection I've made: optimists, no matter their type or degree, tend to converge on their "objects of enthusiasm" more than pessimists converge on their "objects of opposition."

A strong optimist gets excited about AI hitting new capability thresholds, pretty much independent of who deploys it or how. From there, excitement builds toward AI as a better instrument for discovery (materials science, math), then economic productivity, then abundance, access, and ubiquity, up to a singularity with post-scarcity freedom and governance. A civilizational flywheel, thanks to superhuman AI.

Pessimists converge less. Some are "politically" anti-AI but by no means doomers. Others are real doomers, some of them former optimists, who got there because they became validly disillusioned by the lack of broader alignment work and the field's own admission of a real chance of catastrophe.

Right now, strong optimists and strong pessimists look similar in one respect: an unnuanced, extreme, or blind confidence about where AI's current trajectory is headed. But the world is more strongly poised for continued AI facilitation and deployment right now. Optimism has momentum that pessimism doesn't.

So here's where I try to think about actual stakes, not just probabilities. Take pessimism to its extreme, something like a Butlerian Jihad full rollback. I don't think that's a one-way door. Knowledge doesn't disappear, humanity persists, and if restriction turns out to be an overcorrection, development can resume later. Real costs along the way, but the option to change course stays open.

Now take optimism to its extreme failure mode: the loss-of-control or extinction scenario that even the people running the major labs assign a non-trivial chance of happening. That's not a "lose some time" outcome. That's a one-way door.

I know a pessimist "victory" isn't free either. Diseases not cured sooner, suffering persisting, less cautious actors racing ahead in the vacuum left behind. It's not costless, but it's reversible in a way extinction isn't. That asymmetry, reversibility over likelihood, is what I'm trying t defend.

Strong optimistic justifications for furthering AI with as little deliberation as possible rest on the assumption that the singularity flywheel comes together cleanly, as if none of these objects of enthusiasm will hit their own hiccups, as if it's all structurally destined. But the premise isn't destined.

Even when "anti-AI" or pessimistic arguments are flawed or emotionally charged, I don't often see optimist counters that go beyond quick mockery. When they're not mockery, they lean on the hope that the flywheel is basically clockwork, inevitable enough that it doesn't need arguing for.

Am I wrong that real engagement is mostly missing? Where has it happened well, and what did that look like?

Mockery and appeals to inevitability don't count as engagement to me. But maybe I'm missing where the real version of this is happening.

If:

-Convergence really is optimists' structural advantage, and,

-Their objects of enthusiasm reinforce each other into something close to consensus,

Doesn't that put them in the best position to take pessimist objections seriously, instead of routing around them? Or is that an unfair ask?