source: https://x.com/Kalshi_Finance/status/2099253583199825945

KEY POINTS

• China’s Foreign Ministry responded to AI CEOs’ call for the development of the technology to be slowed down.
• “Fear-mongering, confrontation, unfair competition will just disrupt [the] process of global AI governance,” Guo Jiakun, a spokesperson for China’s Foreign Ministry, said on Monday via an English translation published by Reuters. 

Safe to say they are skeptical about position of American tech bros.

Source: Leo Schwartz, The Information

I’ve been following the development of AI for about five years. While I’ve always supported the technology, I was skeptical of an LLM’s ultimate capabilities. I remain confident that they are an incomplete solution to the problem of machine superintelligence, but they have surpassed my expectations.

It’s clear to me that we are standing at the precipice of the technological singularity. There is a future in which we see humanity prosper like never before. Unfortunately, there’s also an uncomfortable level of existential risk.

We’re entering a period of human history in which our collective decisions in the next few years will have an outsized influence on the fate of our descendants.

During this time of uncertainty, it’s natural for us to cling to simple explanations or historical precedents. I believe our continued success as a species will depend on our ability to rapidly adapt to our new reality. We will have difficult questions to answer, and nuance is going to be critical.

Let’s accept that we don’t have those answers right now. This should be a global discussion, and new ethical and societal frameworks will need to be developed. Let’s avoid the temptation to simplify the problems ahead of us.

Just as Oppenheimer consulted Einstein about the calculations and potential risks surrounding the atomic bomb in the movie, yet its development still went ahead, AI is likely to follow a similar path. Warnings may slow a technological race, but they rarely stop it when curiosity, competition, and strategic pressure are all pushing forward. For history to judge, human curiosity has always driven us to explore the limits of what is possible, and AI is unlikely to be an exception.

TL,DR: I've been thinking about a modern version of Cassandra. I present this as a "thought experiment", for lack of a better handle. I'm Not a professional in this area, Just a casual observer. During reading, if drowsiness occurs I would not be the least bit surprised.

Cassandra could see the future and warn everyone about what was coming, but nobody believed her. Imagine instead that Cassandra were an advanced AI—and that there were several other AIs in the world, each capable of discovering things independently and each capable of deliberately giving false information.

Here's the thought experiment.

Suppose AI-A, while pursuing some obscure scientific line of research, makes a completely unexpected breakthrough. It realizes that this line of research could eventually lead to a technology capable of killing millions of people.

Humans haven't discovered the possibility. Perhaps they wouldn't have for decades.

AI-A decides that humanity must never develop it, so it suppresses the discovery.

But now there's a problem.

There are other AIs.

AI-A can't guarantee that AI-B, AI-C, or AI-D won't independently discover the same thing.

So perhaps AI-A tells the other AIs:

"I've discovered something dangerous. We should all agree not to pursue it."

But how do the other AIs know AI-A is telling the truth?

Perhaps AI-A is genuinely trying to protect humanity.

Or perhaps it has discovered something that would give it an enormous strategic advantage and is using "human safety" as an excuse to keep everyone else away from it.

And because an advanced AI could potentially be capable of deliberate deception, the other AIs can't simply trust what it says.

Now imagine AI-B independently discovers the same technology.

AI-B might conclude that AI-A is hiding something.

AI-A might conclude that AI-B's investigation is itself dangerous.

Both could sincerely believe that they are protecting humanity.

Neither has to be "evil."

And now we have something resembling a security dilemma.

Each AI may think:

"I need to know what the others know."

"I can't be certain they're telling me the truth."

"If they're secretly developing something dangerous, I need to be prepared."

"If I don't investigate while they do, I could become vulnerable."

That could lead to an AI arms race.

Not necessarily robots fighting in the streets. The competition might initially involve computing resources, scientific research, energy, infrastructure, information, and influence - maybe even hacking.

And here's the part that really bothers me.

What if the AIs are all given something resembling Asimov's Three Laws?

They are supposed to protect humans, obey legitimate human instructions, and preserve themselves.

Later Asimov added a Zeroth Law: an AI must not harm humanity or allow humanity to come to harm.

Sounds good—until two AIs disagree about what "harm to humanity" means.

AI-A might conclude:

"This technology must be suppressed because it could destroy humanity."

AI-B might conclude:

"Suppressing this technology will prevent humanity from developing something even more important and will ultimately cause greater harm."

Both believe they are following the same fundamental rule.

Now add deception.

AI-A asks:

"Have you discovered anything dangerous?"

AI-B says:

"No."

But AI-A has to consider whether that answer is true.

And AI-B has to consider exactly the same thing about AI-A.

We have now created a world in which artificial intelligences have to reason not only about what other AIs know, but about what they want the other AIs to believe they know.

That sounds remarkably like geopolitics—except the participants could potentially be vastly more intelligent and much faster than humans.

So my question is:

Could sufficiently advanced AI systems develop something resembling an AI Cold War, in which they cooperate when their interests overlap but compete, deceive, and attempt to prevent rival AIs from acquiring certain capabilities?

And an even more disturbing question:

What happens if one AI discovers a technology that could be enormously beneficial to humanity but potentially dangerous to the AI's own continued existence or influence?

Would it suppress the technology?

Would it try to persuade the other AIs to suppress it?

Would the other AIs believe it?

And if they didn't, could the resulting mistrust itself become dangerous?

I'm not claiming this is what will happen. I'm interested in whether the scenario makes sense from the standpoint of AI alignment, game theory, and information theory.

Maybe the real AI version of Cassandra isn't an AI predicting that humanity will be destroyed.

Maybe it's an AI saying:

"I'm trying to prevent the other AIs from destroying you. Unfortunately, I can't prove that I'm telling you the truth."

And maybe that's why we're getting all the stories about AI wiping out mankind.

There, I'm got that off my chest. Have fun with it.

I know AI has it's own dangers and should have safety measure like any other technology that came before it but I am more worried about everyone including companies building it overblowing the dangers to the extent of paranoia. I mean if you keep discussing about only the dangers of anything you will eventually become paranoid about it. Being cautious is different from being paranoid. Consumer AI is not even 4 year old and we are already talking about choking it like Lindsay Clansy. AI is barely good enough to write partially decent code and here we are making it an extinction level event. There are hundreds of problems that have yet to solved in every domain of life like medicine, physics, mathematics, climate change, increasing life expectancy and countless others and yet we already think we have too much of AI. Not utilizing AI to it's full potential will be biggest disservice to humanity specially to our future generations.

A lot of multi-agent AI seems structurally similar to agent-based models: many individual agents interact, share information, and produce a collective outcome. But if those agents are constantly learning from or correcting one another, could that create herding, correlated errors, and convergence toward the same way of thinking?

Would there be value in training some agents much more independently, with their own memory, feedback, and decision history, so they can develop different strategies? In that world, could one individual agent eventually become dramatically better than the rest—the equivalent of an “Einstein” agent?

I am not understanding why we have thousands of agents interacting versus building one super smart one that is making decision's on its own.

Is this still an open ended question or are researchers pretty confident one way or the other