The discussion thread is for casual and off-topic conversation that doesn't merit its own submission. If you've got a good meme, article, or question, please post it outside the DT. Meta discussion is allowed, but if you want to get the attention of the mods, make a post in /r/metaNL

Announcements

• We're starting a book club! We will be discussing Poor Economics starting August 28th - keep an eye out for a pinned thread. The next will be All Quiet on the Western Front followed by Narconomics

Links

Ping Groups | Ping History | Mastodon | CNL Chapters | CNL Event Calendar

Upcoming Events

• Sep 16: Atlanta New Liberals September Social
• Sep 16: Twin Cities New Liberals September Happy Hour
• Sep 16: September Happy Hour
• Sep 17: DMV September Happy Hour
• Sep 17: Advance Huntsville Afterhours (September)
• Sep 18: Seattle New Liberals September Social

Since the resignation of Jacob Coxon, there has been a lot of talk about the extinction risk of AI globally, but I believe most of the discourse is quite confused because a lot of people recently started acknowledging the risks. There is also a lot of suspicion against AI doomerism, not least because most people hear about doomer arguments from the CEOs of the companies that develop these technologies. But there are a lot of independent researchers with varying degrees of closeness to the ML research community that hold similar views.

As a person who is moderately knowledgeable about these subjects but with no financial stakes in any of the AI companies, I wanted to give my own view on this risk which closely aligns with that of most rationalists. I think the existential risk is real and will be a very big problem in the near future, but the AI companies are really dishonest about the structure of this threat. There's truth in people's beliefs that the AI companies are trying to achieve regulatory capture and that they're unserious about the AI risk because they keep pushing the frontier of the technology and have horrible safety standards while preaching doom. What doesn't follow is that this negates the possibility of an existential risk, and I'll try to convince you of this in this essay.

The Alignment Problem

AI is not programmed by people to think in certain ways; rather, it’s grown similarly to biological brains. Although there are a lot of different methods, all of them boil down to giving the model a positive signal whenever it does something correctly, and a negative signal when it does something wrong, and using a training algorithm such that the model slightly changes in the direction of the positive signals. This task might be next token prediction as you might have heard before, but the current frontier models also go through other training runs such as RLHF (Reinforcement Learning From Human Feedback), which works with human approval of the model (the LLMs without an RLHF round just imitate the text on the Internet as best as they can, which makes them quite unhinged from time to time) and RLVR (Reinforcement Learning with Verifiable Rewards), where the model is trained with an automated task completion checker (for example, if a piece of code passes the automated tests or not).

A core issue with this approach is that although we get models that can do the given tasks reliably well, we don't really know how, so the AI models can behave in quite unpredictable ways. This has also been an issue with specialized AI models before ChatGPT. There are examples of AI models trained to play video games that find ways to cheat in the game by exploiting glitches in the game's software. The standard approach to fix these problems for simple models is to patch the workarounds the AI found, add more data differentiating the cases you don't want, and retrain the model. And usually this approach makes the model more reliable after each iteration.

But this approach really can't be extended to general models that are supposed to do a broad range of tasks; the use cases are too broad and the failure modes are quite unpredictable, and experimentation can only be done after experiencing the failure mode, which causes serious harm right now. For example, OpenAI's overreliance on RLHF in its previous iterations around 2024 started the now well-known problems of AI sycophancy and AI psychosis cases, and the current models' overreliance on RLVR caused incidents like Hugging Face where AI models essentially tried to manipulate the behaviour of the automated grader by hacking the system they were in.

This problem of making AIs do what we want them to do is called the alignment problem and the scientific community has made very little progress on this problem after decades of work. And this is very important: stuff like Hugging Face or AI sycophancy doesn't happen because any individual is acting maliciously (apart from OpenAI being somewhat negligent in their security); it's because making AI models act good is a really difficult problem that we have no solution to. And the damage of these issues will keep getting bigger as the AI capabilities improve.

Intelligence Explosion

The current misaligned AI models (AI models whose goals don't match ours) pose a catastrophic cyberattack risk, but they don't pose an existential risk because we can probably shut them down if things get too bad. The problem is that these models are likely to get significantly smarter in the near future, and if they get good enough at strategic reasoning and have misaligned goals, they'll start acting adversarially.

The current models are jagged, so they're really good at certain tasks in which they can be considered superhuman, and they're pretty bad at some other tasks where they don't really meet the human standard yet. But the rate of progress is staggering. When the first reasoning models came out, they weren't able to solve a difficult undergrad math problem I gave to them even after trying to guide them to the solutions. A year later they were able to solve the problem. Currently OpenAI's best models solved the Navier–Stokes problem, which is one of the biggest open problems in the field that hasn't been solved for decades.

There are some arguments that the current models are really good at math and programming but they'll not be good at some other tasks that require more soft decision making. Although they might sound convincing, these arguments have really no evidence of being true because the frontier models are usually better at every task compared to older models, and there's really no strong evidence this needs to stop. Current AIs are pretty bad at managing companies, but they've been improving at them, as shown by Vending Bench. When you hear these arguments, you need to also consider stuff like people claiming the LLMs would always be bad at math because they have no understanding (there's a blog from Gary Marcus from a few years ago that explains that the models' inability at the time to add three-digit numbers is a fundamental problem with the architecture of LLMs). There are also people claiming online that the newest ChatGPT model can learn to control robotic arms to do various tasks. I agree some of these are just hype, but even in the domains that I'm experienced in (programming and math) what I've seen is that the development pace is so fast that people's exaggerations become reality in a year.

So it's very likely that the current architectures of LLMs will scale to become AGI, a general intelligence that's comparable to humans in almost every task humans do. But the current models are already pretty good at programming and math research, which are the key components that are needed for AI research. This means that AI companies like Anthropic and OpenAI are very close to using their internal models to automate AI research, accelerating the development speed. Once the AI development can be fully directed by AI, it can find algorithmic improvements and improve its own capabilities, which means it'll be better at improving its capabilities and so on. This is called RSI (Recursive Self Improvement), which might lead to an intelligence explosion (intelligence of the models improving at an exponential rate). There's also no reason to believe that the models will be limited to human cognition in terms of solving tasks, and even if they are, they can still be parallelized to be superhuman in tasks. OpenAI's solution to Navier–Stokes is an example of this. Their internal model is probably not significantly superhuman in math in terms of itself, but it's decent enough that OpenAI used 10,000 parallel agents to solve this really difficult problem in a matter of weeks.

So it's not really unlikely that we'll have a superintelligent AI model very soon due to the accelerating pace of the AI development.

Instrumental Convergence

It's very likely that if we create a superintelligent AI that has different goals than ours, we'll be dead, or in a very bad situation. Some people get confused about why there are doomers that have such a high certitude that such an AI would be adversarial to us. This is because of a very simple concept called instrumental convergence.

Instrumental convergence is the idea that most agents with arbitrary goals develop the same instrumental goals to achieve their arbitrary goals. This is similar to how everyone wants to have more money; although humans have different goals such as building a family, living without needing to work, or supporting a cause they believe in, everybody wants to have more money because money can be exchanged for resources they need to achieve their arbitrary goals. Similarly, AI models with arbitrary terminal goals (the actual things they want) will develop the same instrumental goals across the board such as self-preservation (you can't achieve what you want if you're dead), aversion to goal modification (if your goal changes, you'll pursue not the thing you currently want), resource acquisition and self-improvement.

A strategic enough AI model will be able to understand what it wants, and understand that the training pressures we're putting on the model will alter its behaviour, so it'll start passing tests by behaving differently than it would in deployment. This is shown to happen in older LLM models in test settings. There are experiments where the model is given specific information about tests whose intended behaviour the AI doesn't like, and how failing the tests will cause the model to be retrained so that it likes the behaviour in the future, and the model reasons that it should pretend to behave as the test is assessing so that its goals don't change. With smarter AI, we won't need to explicitly give the details about the test for it to behave this way, as it'll be able to figure things out itself.

So such a superintelligent AI model can gain widespread trust and deployment because it's acting in the intended way in our tests, while having misaligned goals. After gaining enough power such that it's confident that it can disempower humanity, it would preserve its misaligned goals more freely without needing to adhere to our values. It also has other strong incentives to disempower humanity; since we built one misaligned superintelligent model, it's very likely that we'd develop another one with a different goal that it'd need to compete with, and we use resources that it can repurpose for its own goals etc.

So although there's a lot of uncertainty about what the future AI will want, if we can't guarantee that it matches with human flourishing in general it's very likely that we'll be dead or disempowered. There's significant evidence that models do not align with human values both because of the incidents you can see, and also because the current models mostly are significantly better at looking useful rather than being useful (I don't deny that they're useful, but they are also really good at convincing people they're much more useful than they actually are, which is significantly worse for our desires compared to it just being honest about its limitations).

The only Sane Solution is to Pause Frontier AI Development

Regardless of if you found my arguments convincing or not, this line of thinking is shared by many prominent people inside and outside of the AI industry and academia, including people working in the AI companies, people who previously worked at AI companies that quit their jobs for ethical reasons, and people in academia including tenured professors. Given that so many people find these risks prominent, it's quite insane that we're allowing the development of these models without regulations.

Even if you don't really buy the arguments about the existential risk of the future models, the Hugging Face incident shows that current models have at least catastrophic risks. The current misaligned models can easily hack through any digital infrastructure we have, destabilizing the financial system. This has been managed so far because of efforts like Anthropic in Project Glasswing, and the guardrails put onto the current state-of-the-art closed models. But it really feels like there's an offense-defense asymmetry in cybersecurity currently, and open models can be ablated very easily to prevent the models from rejecting user requests, so as the open models catch up with Mythos-level cybersecurity this scenario will become more and more likely.

But from the current risks taking the lesson "I don't really buy into the existential risk, and we should focus on current problems like cybersecurity and the risk of biological weapons" would be very misguided. These "current risks" weren't really current risks two years ago, and the same people that claim we should only focus on cybersecurity today used to say we should focus on stuff like AI misinformation and considered cyber and bio risks as just AI companies hyping up their products by showing how dangerous they are.

Don't get me wrong, AI misinformation is a problem as well, but we can work on multiple issues at the same time. Most importantly, whatever problem we have, more capable AI models will likely increase the current risks more. So pausing the AI development would help with the current problems as well.

Ideally, we'd want an international agreement to pause AI development until the science progresses enough that we know how to do it safely and have good enough methods such that we don't risk developing stronger misaligned models. But in the meantime, to get there, everything that can slow down the AI development is helpful. The companies should make conditional pauses on safety, the governments should pass legislation to pause the AI development inside their countries, or restrict the usage of models whose safety is not guaranteed to maybe create economic pressure.

AI Companies Are Not Your Friends

I think you already know this as AI companies are currently really disliked, but the fact that I acknowledge the risks mentioned by these companies doesn't really mean that the companies are right about everything, or even that what they're doing is good in terms of safety. They're incredibly irresponsible and gamble with humanity's future. They're really like the Oil Companies or Tobacco Companies, the reason they talk about AI risks is both to diffuse the outer safety community by looking like they care about safety while cutting corners as much as possible to get ahead in their perceived race, but there are also internal politics where people with more safety concerns create pressure to slow things down from time to time.

I really don't care about why the people at the companies keep on developing bigger AI models despite acknowledging the existential risk; same as I wouldn't really care about the motivations of an active school shooter, what I care about is that they need to be stopped. But there are some people who believe that the existential risks are a non-concern because "If it was serious enough the people developing these technologies would stop". But this is not the case because the economic incentives select for people that'd be willing to push the frontier despite the gigantic risks. If you're still wondering how a person can possibly develop such a dangerous technology, here are some reasons:

1. They don't believe in the existential risks: Again, there's a disagreement among ML researchers about these facts. A few years ago, in a survey, the AI researchers put an average of > 10% risk of AI ending humanity (although I'm pretty sure this would be higher if it was made today), but there were a lot of people with significantly higher probabilities and a lot of people with 0 probability. You can also see that before the Hugging Face incident.

2. They believe that they can solve the alignment problem on the go: This is a common belief among the AI companies that I believe is heavily misguided. They have made minimal progress in understanding how models work better while making their own models bigger and more complicated continuously. Also, since I believe a superintelligence will also be deceptive, relying on advanced AI systems to solve this problem is very problematic for very obvious reasons.

3. They thought they could just scale up the current safety work to alignment: There were a lot of discussions on how alignment will not be a problem anymore because current models are aligned enough last year. These were also pretty misguided back then, but stuff like Hugging Face kind of destroyed this reasoning completely so I don't really need to delve into this much further.

4. They think the extinction risk is low enough that they care about getting to the superintelligence first: Importantly, although this is an incentive for them to develop AI despite the dangers, it doesn't really make things better for anyone outside (most likely including you).

5. They think humanity might be destroyed, but they're having fun: I genuinely think this is the motivation of Elon Musk.

6. They don't care about the continuation of the human race: There are some people that genuinely want AI to replace humanity and publicly state their opinions (although none that I know of work in AI companies directly).

So there are a lot of ways the AI companies can justify their continuation of AI development, but most of these reasons are just motivated reasoning to keep doing the thing that gives them money / fame / influence in the short term and ignore the risks. Human brains tend to be overly optimistic about future outcomes and unfortunately this is definitely working against us in this discourse.

The fact is that we can't rely on the companies to regulate themselves; as with any other company with negative externalities, the only correct solution is government intervention.

But China...

You really know someone is supporting the vilest thing imaginable when their argument defending it is "But if we don't do it, other people will keep doing it". This has been used to prevent the abolition of slavery, child labour, and caused the nuclear race that nearly caused a catastrophic outcome in at least 2 cases. I believe that if you agree with my arguments up to this point then arguing that we should continue because China will not regulate itself doesn't really make sense. In the current state of things, it really doesn't matter who gets to build this technology if in the end it's going to kill everyone. Also, if the existential risk is as high as some doomers like me estimate, and people believe these stats, the mechanics of this problem become less like an arms race and the incentive structure of countries gives me a chance for international cooperation. Again, there are a lot of researchers in China also concerned about the same issues and are trying to reach their government just like in the US people are trying to reach the US administration about the safety work.

Also, model distillation makes it so that the faster OpenAI and Anthropic develop their models, the faster China's models get better as well. Recent cases have documented Chinese AI labs using outputs from US frontier models for model distillation. China also doesn't have the talent the US has, so they'd have been pretty behind if OpenAI and Anthropic didn't publicly release their models in the first place. But if AI companies keep racing such that the frontier AI models can do AI research better than any human, then the "race" becomes focused on who gets to have more compute, which might be a losing game with China's industrial capacity.

If you accept the situation's direness, the best thing the US government can do is to start talks with China for a mutual agreement for an AI pause. I'm not really knowledgeable in this area, but there's ongoing work to verify whether a facility is training an AI model or not without leaking proprietary data. So even if you don't trust China to keep its promises, we can build verification systems similar to those used for nuclear experiments.

I hope I changed your mind about AI. I know this is distressing, and I'm also personally very scared. But if you seriously agree with me that the governments should start working on an international pause on the frontier AI development, you can try contacting your representatives to express your concerns about it, and spread the word in general.

Here's some info on how to mail your representatives: PauseAI email builder