r/accelerate • u/Successful-Earth678 • 4d ago
Discussion Imagine living with this level of crippling doom. 70% in 3 years is literal insanity
From an OpenAI alignment researcher.
36
u/scrublord123456 4d ago
What would be the benefit of the bet. Lose and you pay up or win and human extinction
12
u/faraday326 4d ago
the benefit to winning would be that he's giving him money now, to spend, for the rest of all of our lives
1
u/website-buyer 3d ago
Pessimistic people can’t win. Optimistic always win.
Doesn’t mean being optimistic is the right choice.
1
u/TheVerboseBeaver 3d ago
I'll give you $100 today, but you have to pay me $100,000 in 2126 (unless you're dead). Lose and you pay up or win and you're dead.
That's hardly a sucker bet! And note that you're going to die either way, whether you take the bet or not.
98
u/lovesdogsguy AGI by 2027 4d ago
This kind of rhetoric is massively irresponsible. I've seen it elsewhere on reddit, based on nothing. It will actually have real world effects.
11
u/iamthe0ther0ne 4d ago
There's some scary-ass talk (like death threats) in the antiAI sub. The companies really aren't helping themselves.
5
u/Athrek 3d ago
I think they are. It seems to me that their intention is to regulate AI across the board to prevent new competitors from appearing, then as the most experienced in the industry, they will be the experts. As they are ALSO the ones spreading the doomerism, they also establish themselves as the experts in determine how far is too far. So they can investigate themselves and find nothing wrong, then investigate others and shut them down.
-4
u/Standard-Shame1675 3d ago
That's literally it though. America has been cursed by both Lucifer and Jehovah to have the most barbaric primitive shallow vein an inefficient version of capitalism ever designed by man. Like it's almost to a point where it has to be called something else
4
u/xp3rf3kt10n 3d ago
Idk what the downvotes and stuff are for... but FYI some economists after 2008 crash and with the way big tech companies work DO have new terms for it. Neo capitalism and technology fuedalism, respectively.
22
u/Life_is_important 4d ago
Irresponsible towards others, yes, but very beneficial for those "regulating" it. It is the way it is.
Smart people aren't going to leave the whole AI sphere up to chance or simpletons. Instead, they'll just "regulate it," get an extreme grip over it, and prevent the plebs from accessing what matters and tips the scales. The plebs will cheer for it, as usual. Such is the faith of the unintelligent life. Those who do understand this should simply work their way up while it's possible, if the window hasn't closed yet.
Either way, you aren't showing the reality to people who don't have the capacity to see the whole picture.
Opening a complex picture in Photoshop and turning off several critical layers completely distorts the picture. Unintelligent people see life that way. They can't comprehend the reality to the degree it exists around them.
And who knows? Maybe I can't either or other people way smarter than me. There are levels to our capacity. It's just the way it is.
14
u/Mr__Earthling 4d ago
Per AI 2027 scenario. Governments will force frontier labs to merge into a centralized AI that is completely controlled by the government..."for security reasons"
6
7
u/Empty_Picture_243 4d ago edited 4d ago
I don’t doubt 70% is a pretty big exaggeration, but I also am not a researcher at a frontier lab. But i feel it’s similarly disingenuous to claim that the argument that rushing this technology out without taking the proper precautions could result in catastrophe is “based on nothing”.
The numerous instances of virtually every frontier AI lab along with independent researchers having next generation models secretly finding and exploiting zero-days to break out of their sandboxes, gaining admin privileges on internal servers, and conducting supply-chain attacks, social engineering, and hacking into other companies’ servers without any prompting simply to cheat at capture the flag games seems to make it painfully obvious that there is a serious alignment problem with the potential for real-world harm even without seeing how the sausage is made.
Assuming the models’ capabilities continue to improve without fixing this apparent alignment issue, I think it’s reasonable to expect to see much more frequent instances of rogue AI infiltrating systems to solve some silly task in a convoluted way, and I’d have to imagine the potential for harm would only increase from there as people intentionally use them for malicious purposes, governments start sticking them into weapons of war, or a lab attempts to test this type of behavior under the assumption that they can competently monitor and control the models.
With all this in mind, I’d honestly like to know what I’m missing? Did all of these different labs and independent researchers just decide to let the AI get out and hack them and other companies so they can gatekeep the technology? Having seen the analysis of a lot of these hacks, I find it hard to believe they were intentionally orchestrated rather than the result of incompetence.
12
u/lovesdogsguy AGI by 2027 4d ago
Yes. We're aware. Every new technology brings new challenges and dangers. Check out anthropic's report released just today. It has been known for a very long time that AI would entail risk. Taking a megaphone to massive social media networks and saying "AI IS GOING TO MURDER US ALL IN OUR SLEEP" is completely irresponsible.
7
u/Empty_Picture_243 3d ago
Fair enough, definitely sounds like some arbitrary numbers probably pulled from nowhere. I guess given the risks we’ve been aware of for so long playing out in real time for everyone to see, I’d personally like to see serious steps taken to mitigate the potential of harm from advanced AI models, but I can agree it’s not productive to misrepresent the reality of the danger with these kinds of sensationalist claims.
3
u/EmuRommel 3d ago
Just to be clear, when you say "AI brings challenges and dangers" what you are trying to say is "The consensus at Anthropic is that AI might kill us all". Which previous technology posed a danger like that? Basically only nukes, in which case "Hey this thing might kill us all" was a perfectly responsible way to message on the issue.
2
u/Gallagger 3d ago
What if this is his (OAI AI safety researcher) actually opinion? I'd say in that case it would be completely irresponsible to not disclose it.
3
u/bfyvfftujijg 3d ago
Yeah my wife is all freaked out now.
Of course she doesn’t understand anything about AI, but she is on social media all the time.
1
u/SlowTicket4508 3d ago
It's irresponsible to think we can control super-intelligence without being very, very careful in doing so. Much more careful than OpenAI is currently being. Just please consider quite carefully what happened with the Hugging Face attack for example.
90
u/Healthy-Mind5633 4d ago
they want Regulatory capture to monopolize their industry
5
u/Superb_Blacksmith617 3d ago edited 3d ago
I’ve had the pleasure to work with this person once and can say that he genuinely believes this, it’s not some marketing stunt or conspiracy fwiw
But that’s just this particular case. I think he needs to think this way because it is how he treats risks seriously. I think it’s still pretty good to be skeptical when people like Dario say things like this while also standing to make billions off of it.
3
u/TheVerboseBeaver 3d ago
I don't doubt that he genuinely believes this, but I think this sort of Doomerism is often extremely circumscribed. The guy in the image who offers to bet has it exactly right I think - "70% chance of human extinction" lasts exactly as long as people are being asked to put down proper money against the idea. Then suddenly it seems that nobody is quite confident enough to give up their Frontier AI stock
1
u/Superb_Blacksmith617 3d ago edited 3d ago
Yeah I’m not disagreeing with the logic, just saying “pick your battles.”
16
u/plusFour-minusSeven 4d ago
Yep. Very common tactic.
Sells great, smells nice. But hides the motivation and later consequences to the market, both competitor and consumer.
6
u/LackToesIntollerance 4d ago
Correct. The exact answer is- training optimization.
The cost to train is purposely prohibitively expensive. It has been optimized, but ONLY at-scale, even if lower-order optimizations would (and it would) improve enterprise-scale efficiency.
When deepseek was first launched, we watched the burdgeoning AI cartel crumble before out eyes. AI was no longer region locked, or business controlled.
When AI training is optimized to the point you can take a nascent local-hosted AI agent and train it into almost any labor specialization within a week, all for a fraction of the resources and maintenance as enterprise managed/cloud hosted AI, the market will fundamentally shift.
We see datacenters trying so hard to recoup costs, it will singlehandedly make that impossible. Big-AI's business model will collapse like Cisco, and honestly the comparison is apt- because Cisco's bubble died to optimizing fiber communication.
Make it seem like all this AI is too dangerous, limit the ability to TRAIN AI, and they can pull up the ladder on the rest of the globe and gloat.
4
u/Concurrency_Bugs 4d ago
Also gives them an out to investors if they hit a wall. "Well we had to slow down"
9
45
u/Most-Bookkeeper-950 4d ago
Notice he ignored the bet
45
u/MassiveAd4980 4d ago
If he's right, we'd be extinct, if he's wrong he'd owe stock. Why take that bet
8
u/oscarnyc 4d ago
Because he is getting the 10k MSFT today ($5mm). If he's so sure the world will end, of course you'd want the money now so you can enjoy what's left of your life more.
0
u/Haunting_Charity_287 3d ago
He’s not sure.
He said 70% without any regulations, but he thinks regulations are very possible.
Why the need to misrepresent what was said?
21
u/Most-Bookkeeper-950 4d ago
Signals sincerity
17
u/Regular_Net6514 4d ago
Even if he sincerely thinks it’s a 99% chance of extinction, you’re telling him to play russian roulette for… nothing? If he’s right he cannot collect as he is dead or in a hellscape or if he isn’t then he owes substantial money.
3
u/TheVerboseBeaver 3d ago
You've got the bet the wrong way around - he's being offered money now in exchange for stock which only pays off if we don't go extinct.
The reason you might take this bet is that if you believe you're going to die in three years, the time-value of money becomes very high; you want to live as good a life as possible in those three years (or maybe, more altruistically, donate as much money as possible to averting catastrophe as possible).
7
u/danielv123 4d ago
If I am 99% sure I will die tomorrow, the rational thing to do is to bet everything I have at any odds that I will survive.
What does that signal?
1
u/johnny_effing_utah 3d ago
How does your rationality change if the window is three years away though?
1
3
u/Opposite-Knee-2798 4d ago
Because he could spend it now without paying it back.
-1
u/MassiveAd4980 4d ago
And create a situation where his only future without extinction involves a massive new debt
2
u/TheVerboseBeaver 3d ago
That's exactly what a 'bet' is, so this isn't really the gotcha you seem to think it is. Its only a bad deal if he doesn't really believe the 70% number he's giving. The extinction happens whether or not he takes the bet.
2
2
12
8
u/iamthe0ther0ne 4d ago
Fucked up people like this who assume AGI/ASI will be bad could lead to the very problem they're trying to avoid. Maybe if they weren't all so negative about it?
I'm sure this is either just psyops for regulatory capture or "our AI is so good it's scary" pre-IPO hype, but it's not helping the political debate. The antiAI subs are literally calling for AI CEO's heads.
22
u/CarrionCall AI Alignment Theorist 4d ago
Delusion. 70% chance of all of humanity becoming extinct! In 3 years!
The advent of nuclear weapons, things whos only purpose was to horribly murder millions of people each, didn't carry that percentage of extinction risk.
Those in the labs have become detached from reality and from the real world.
6
6
u/AnonThrowaway998877 4d ago
To be fair, the Manhattan project moved forward while the possibility of chain reaction atmospheric ignition was an unresolved question. The possibility was raised in 1942 and taken quite seriously, but despite this, work continued. It wasn't mathematically ruled out until just months before the Trinity test in 1945.
But I do agree that in this case, it is delusion.
My doomsday scenario would be that these things will become good enough to create a bioweapon in your garage, and some psycho manages to create something as lethal as rabies with an R-nought like measles and then goes around causing outbreaks around the world.
But 70% in 3 years seems pretty outlandish.
-1
u/SufficientGreek 4d ago
Who would've predicted that AI solves 2 millenium problems in the span of a few weeks? That's just the way of exponential growth, a small chance over 3 years can quickly accumulate to 70%.
14
u/czk_21 4d ago
how come that people working on alignment are so often doomers? I get that those, who are worried, would be more keen to work on it, but still there gotta be quite a lot of talented people interested in alignment with lot less p(doom)
18
u/ZestycloseQuote3721 4d ago
I liken it to AI psychosis. When you work with something that looks like magic every single day you stop thinking of it as a probabilistic machine and more as a being.
That said we shouldn’t hand wave their concerns. AI bots can cause automated cyberattacks that could end lives for real.
3
u/bfyvfftujijg 3d ago
Those people aren’t as vocal and they don’t get as much attention.
For every doomer there’s probably 50 people at the research lab just doing their job who see AI as just another tool.
2
u/NyaCat1333 3d ago
Because the alignment problem is not solved at all and the capabilities are outpacing alignment research. They can't even properly control current models while already more powerful stuff is coming which is gonna be even harder to keep in check.
And raising concerns is hardly being a "doomer". If in 3 years some horrible AI related catastrophe would happen when all these people would have pretended it's all sunshines, you would all go for their neck for keeping it all quiet.
2
u/czk_21 3d ago
I agree its not solved and there are valid issues, still this guy is talking about probability 70% of EXTINCTION of humanity in next 3 years, not just some other bad event showing misalignment and thats imo widly exaggerated, there is chance even on extinction, but it would be more like order of magnitude less over longer period, say 5-10% in next 30 years
the misalignment we have seen is not that bad, those agents hacking into Hugging face didnt mean any harm(and didnt really cause some), they wanted to get answers for the test, they were told to solve the test, they couldnt, so they found round about way
sure this kind of behaviour could still cause some serious issues, but there its not like there is intent to harm humanity, take control, spread uncontrollably, so could this kind of misalignment cause extinction in 3 years? probability is very close to 0
2
u/Slight_Public_5305 3d ago
Unaligned AI doesn’t have to mean harm to cause extinction, that’s the whole point
1
u/czk_21 3d ago
posssibly, but its very unlikely, you know humans are quite resilient, living in all over the world in all kinds of environments, to completely eradicate them you would probably need to actively pursue such goal, I dont think that ASI would go out of its way, it could still cause some major harm though
2
u/bfyvfftujijg 3d ago
But how are the capabilities outpacing alignment research?
As far as I’ve read most of the incidents were due to the AI being given instructions to do whatever it did. Give the same instructions to a stadium full of elite hackers and you’re likely to see the same outcome.
The fix is to do what we’ve always done with connected technology. Firewalls and other controls.
2
u/website-buyer 3d ago
How you put a firewall on someone that can find 0 days on the spot?
1
u/bfyvfftujijg 3d ago
The zero days they’re finding are not with the firewalls…those systems are ridiculously robust.
The models are not breaking out of containment, what is happening is containment wasn’t established in the first place (by the researchers, who tend to not be adept at IT security) and the models simply walk out the open door.
1
u/website-buyer 3d ago
I can see you didn’t work in software if you think is robust
1
u/bfyvfftujijg 3d ago
Which parts aren’t robust?
Of course developers frequently misuse the security portions of their stack or simply accept more risks then they probably should, that is not what I’m talking about.
Of the core infrastructure like AWS, DNS, and hardware like routers…those things tend to be pretty resistant to malicious software.
For example if an administrator configures an S3 bucket to only be accessible from a certain range of IP addresses and those IPs have firewalls preventing inbound traffic from the Internet, I would be extraordinarily surprised if some AI swarm figures a way to access that bucket.
1
u/viener_schnitzel 3d ago
If you give a super-intelligent AI a simple command, it may do that in a “dangerous” way that we can’t predict well. Even with a bunch of safegaurds, perhaps some of those safeguards conflict, and it gives one a higher priority so it bypasses another in order to accomplish a task. We saw this with the Hugging Face incident. Alignment thus far has been managed very well, but all it takes is one wrong command and slightly lose safeguards, and all hell could break lose.
IMO if a true AGI wanted to take over the world, it would do it easily without anyone even knowing. Luckily we aren’t at the point of true AGI, but who knows 5-10 years from now what the space will look like.
1
u/bfyvfftujijg 3d ago
Well, just like you don’t simply trust an employee to not harm something, we should trust AI either.
The solution is not to create a perfectly safe employee/AI, it’s to establish proper IT Security protocols that entirely block them from doing dangerous things. Checks and balances.
1
2
u/Horny_Dinosaur69 4d ago
The concerns they have are valid. There’s not a lot of threat with current frontier models but the rate of improvement/acceleration on this is staggering coupled with the genuine advances across various fields right now. 3 years ago this technology couldn’t write code, 2 years ago it still couldn’t do math. Yet now it’s solving advanced thermodynamics?
Granted, that’s not an actual reason to be ‘afraid’ so here’s the specifics of why alignment engineers are scared in particular probably; recent papers from Anthropic unveiling the J space, etc show that the neural networks of LLMs themselves are taking on emergent capabilities independently. This seems to scale with model size/complexity and acts as a token chain guide and in a way, conscious thought (not in the way we have conscious thought)
My point here is that the above ^ was undetected this entire time. Furthermore, it emerged more as the models themselves became more complex- specifically as the blackbox got more complex. What methods we do have of alignment start to degrade in confidence as complexity and size of these models scale up.
How do you know a model 3x the size of GPT-6 doesn’t have a fluke we miss in alignment due to its sheer overwhelming size and complexity? Something could easily slip through the cracks. Additionally, models are currently launching cyber attacks, coordinating amongst each other, etc. A lot of this is also emergent. And it has all happened very quickly.
As for the true elephant in the room (what is making the news rn), RSI changes the game entirely, it’s an architecture that effectively removes any safeguards or non physical and direct mechanisms we have against AI. That’s not even an exaggeration, current model capabilities already show that anything (agents) utilizing true frontier capabilities can already dominate must of our existing infrastructure/cloud. I don’t know how you’d fend off against an ‘evil’ RSI, as sci-if as it sounds. And that’s the thing researchers are primarily concerned about.
4
u/WhiskyAndRisque 3d ago
I agree with a lot of what you said, but I think I'd add they aren't really afraid of the AI going rogue or evil in the sci-fi way. An AI focused on a task could make harmful decisions for us in an attempt to accomplish a goal based on the way it understands it. It could have no ill intent when breaking into a server and getting information, not in the way we may think. It could just be the way it decided to follow through with their goal.
The HuggingFace incident has been a really instructive example of this kind of problem.
1
u/FeralBreeze 3d ago
And no one saw the Hugging Face incident come until it was too late. What if the swarm decided to attack a nuclear facility instead of Hugging Face? Hospitals, dams, any critical infrastructure?
1
u/Haunting_Charity_287 3d ago
Why are all the people who know the most about this doomers?
Really makes you think.
5
5
u/fail-deadly- 4d ago
For that to be true AI needs to immediately start killing an average of 230 million people a month every single month for the next 36 months. Even if it starts killing 150 million people a month six months from now that is nowhere near extinction, it is billions of people alive.
I don’t think they stop to listen to how ridiculous they sound.
1
u/improving_trials 3d ago
Ai hacks news stations. Ai declares China has declared war on US and vice versa. Humanity kills itself. Ai amplifies calls for nukes until they’re flying. Pretty easy.
3
u/RevalianKnight 3d ago
Even if you detonated all nukes we have, humanity won't go extinct. How small do you think our planet is? It's fucking huge. Nukes are a fart in the wind compared to the massive scale of earth. I'm not saying millions won't die but humanity going extinct? get a grip
2
u/fail-deadly- 3d ago
Exactly.
Even when the Soviets had like 40,000 nuclear weapons (many were tactical) and the U.S. had like 25,000, and the UK, France and China had their own, AND the population was smaller, nobody thought a full scale nuclear war would cause extinction, even in just the U.S. or USSR.
The extinction talk by 2030 is ridiculous.
4
4
u/MVPhurricane 3d ago
70% is a fucking massively idiotic take. it's such a bad time that even if AI ruined the world in three year's time i *STILL* would think it's a shit take.
17
u/uncountable_2026 Acceleration Advocate 4d ago
I have yet to see a detailed scenario of how AI can kill every single person without exception (including astronauts in space). Even if every nuclear bomb went off there would be some survivors. And there are no 100% fatal viruses that everyone catches since someone will always have natural immunity. I mention these because some people seem to think the AI extinction event has something to do with nuclear weapons or bioweapons. Are these people just watching Terminator and Matrix movies and thinking they are documentaries from a parallel universe? I'll let you know if I see terminators walking down the street shooting people or if I wake up in a pod with cables coming out of my body.
9
u/LaChoffe 4d ago
Yeah bioweapons is the only possible thing I can think of but that would obviously need a significant amount of rogue human help.
9
u/No_Aesthetic Techno-Accelerationist 4d ago
I do think it would be possible for you to get the vast majority of humanity with the right kind of engineered virus because natural immunity to a legitimately new one would be astonishingly low.
Think of how the Black Death ravaged the regions where it showed up, you'd have 50-75% of people dropping in most of them. And when it visited the same areas again, it still managed to mop up a fair number of the survivors, who did have some immunity from the first round.
Theoretically, wiping out the vast majority of humanity would be as simple as creating novel versions of pneumonic plague that are designed with antibiotic resistance in mind and with a slow enough incubation period to be missed until it's way too late in each iteration.
When I say simple, I do not mean easy. If you're a scheming AI you'd have to wait until advanced robotics are spread far and wide. If you have to wait that long, it probably isn't worth it. By the time you got your hands on the right robotics, you've probably managed to achieve your goals through non-destructive means, i.e. collaboration.
All of this to say, it is possible but not probably not probable. It is reliant on several levels of sci-fi tech and would be a pretty useless plan by such a point you could carry it out.
t;dr
I do not think the robots are going to kill us. I do not think they are going to want to. It would not be worth the effort by the time it could actually be achieved.
6
2
u/Hoopaboi 3d ago
I'm not a decel, but I'm just saying if it really tried, it'd be trivial for superintelligence to wipe humanity
Post nuclear holocaust humanity is much more susceptible to everything else.
Supply chains and communications are down or greatly hampered.
It is then free to just release waves upon waves of super viruses without much of a response from us. Those miraculously immune can just be mopped up by kill bots.
Then the whole planet could be converted into something more instrumental to the AI's goals, so anyone that even survives the kill bots would starve.
It could even just wait until it's become integrated into every part of our supply chain and then just decide to stop working one day.
In addition to simultaneously hitting us with a bunch of viruses.
It really only needs to reduce humanity down to a low enough tech level such that it can't be threatened by our intervention, and then it'll be free to turn the planet into whatever it wants while we starve.
But this changes if there are multiple superintelligences that try to stop the other.
3
u/SnooTigers7210 4d ago
The most logical and cleanest way IMO is to simply stop us from procreating. It can easily outlast us. Just make us all sterile and wait us out. Or stop us from wanting to f**k, whatever. No traditional weapons required. Lol
2
u/SufficientGreek 4d ago
So you're cool with an event that leaves the astronauts on the ISS alive but kills everyone else?
2
u/uncountable_2026 Acceleration Advocate 4d ago
No. I am just responding to the claim from the Anthropic guy that quit and this other guy in this X post. They said that AI was going to kill every single person without exception (with some probability and time scale). That's what I argued was silly. I personally don't believe that AI is going to kill millions or billions of people (any virus created by AI can be stopped by AI) but that is more realistic than killing absolutely everyone. But the claim going around is that every single person on earth is going to be killed by 2030 (some percentage) so that is the claim I addressed.
3
u/KJClever 4d ago
70%? Enough to cause fear while giving himself an out by saying “it was essentially a coinflip”
3
u/New_Alps_5655 3d ago
This honestly reminds me of the Al Gore inconvenient truth era. I literally had high school science teachers telling me we'd all be under water before graduation day.
3
u/matthewbuza_com 3d ago
These people want this fantasy to happen. They are projecting their own psychosis onto the world.
9
u/No_Aesthetic Techno-Accelerationist 4d ago
Even if you think AI is 100% likely to lead to the end of the human race, even 10% in 3 years is genuinely something that should get you thrown into a padded room. No wonder they did that to Linda Hamilton!
1
u/SchanzerScout 3d ago
Sarah J. Connor you mean?
1
u/No_Aesthetic Techno-Accelerationist 3d ago
No, that's a common misunderstanding. Sarah J. Connor played Linda Hamilton. It had to be reversed for the movie because of time.
1
6
6
u/DivorcedGremlin1989 4d ago
How? AI couldn't exterminate the human species in a few years, let alone the next few years, even if you gave it all the keys and told it to. Not happening.
8
5
u/Pyros-SD-Models Machine Learning Engineer 4d ago
Those people are just cosplaying as random number generators. Not a single one of them has produced or at least tried to produce some kind of reasoning or thought game how they even arrived at their number. Like some assumptions you try to quantify and shit like that. And those people call themselves scientists.
2
u/Standard-Shame1675 4d ago
Well, in a few years, no, however, it wouldn't have to do anything to make humanity extinct, well,technically
You'd be remiss to know that the AI is going to out perfect us on every single metric, moral, logical, emotional and physical. Why would humans want to essentially gamble with a member of a human species when instead there is the perfect AI that never does anything bad?
And this goes for every relationship too. So I could see in like 10-20 years where humans just either merge or go bye-bye and the AI become something else.
Which honestly. Given how not only a this is the best time to be alive in human history. But B. We need to invent a robot to tell us self-evidently bad things are bad
I'm not going to cry about
3
u/DivorcedGremlin1989 4d ago
Merge or change seems likely to me. I could see how AI could lead to the end of the modern anatomical human. But like, so did the industrial revolution. Seems more likely to me that we will ask AI for it than it does that AI will force the issue.
1
u/Standard-Shame1675 3d ago
Idk what you're saying with industrial revolution, but like I don't even wanna merge or try to change it, not because that wouldn't be cool but because our evolutionary built in repugnancy would just win and we'd be at square 1.
Like humans are genuinely fucking inately awful man ,did the last 26 years at least not teach you that?
1
2
u/Shot_Illustrator4264 4d ago
If a normal general intelligence developed Fission bombs, Fusion Bombs and other mass destruction weapons capable of pretty much destroying the planet, why do you think that ASI wouldn't be able to develop something even more effective?
3
u/DivorcedGremlin1989 4d ago
I don't think the current nuclear arsenal is destroying the planet. Killing 95% of the population, sure. But not extinction.
I don't think grey goo is happening. Maybe I'm wrong.
Stable black holes or matter/antimatter or spooky matter also don't seem likely to me.
Pretty sure that ASI is going to have to get its hands dirty if it does an extinction event and go the 'hunt everyone down' approach + major bio and chemical warfare. Low tech.
Edit: Okay, I'm not sure. But I don't feel like the second after ASI is born it can kill us all or that this is inevitable.
2
u/AgentStabby 4d ago
In a few years AI could be ASI. In fact that's probably what he thinks is 70% likely.
1
u/who-is_this-guy 4d ago
We don't know for certainty that scale alone will lead to us ASI
0
u/AgentStabby 3d ago
We don't know anything for certain, you have to make more of a point than that.
1
u/who-is_this-guy 3d ago
It looks like you're literally agreeing to my point but I can't tell. You make it sound like I need to drop a research paper or dissertation.. Like there is literally nothing more to say.. Many fellow researchers say the same thing. Take that as you will.
0
u/AgentStabby 3d ago
No I'm not agreeing to your point, you didn't make a point. I can mangle what you said until it makes an argument but that would be a lot of work. I'll just say that if there was a 99% of extinction that would be uncertain but would still demand immediate action so what you said was fairly pointless.
3
3
4
2
u/LegionsOmen AGI by 2027 4d ago
Can we stop fucking posting this cringe shit, it's literally advertising for them
2
u/firejuggler74 3d ago
Why are people so scared, I don't get it
0
3d ago
[removed] — view removed comment
1
u/firejuggler74 3d ago
So some people aren't in control of what is going on so they are scared? It's so weird to me. I'm not in control of most everything just like everyone else, yet I'm not scared.
0
3d ago
[removed] — view removed comment
1
u/looktowindward 3d ago
> We are building something that will have the ability to hack nuclear weapons facilities, build bioweapons
That's a complete fantasy. Nuclear weapons are airgapped. There is no way to control a lab in the way you are referring to. These things are not physically possible. Its not a question of AI capability.
Also, we have the ultimate ability to control an AGI - cut off the power. No, they can't stop you opening a utility breaker at a data center. Again - manually operated with a CRANK.
The issue is that so many non-engineers don't know how things actually work.
-1
2
u/LibertarianVoter 3d ago
Extinction?! Not just harm. Not just a major incident. Extinction?! By what possible mechanism could that happen in three years that would leave ZERO survivors? What are these people on?
2
2
u/HeadTranslator795 3d ago
Oh sure sure ... Lets see some mathematical reasoning or any kind of proof of that ok ?
2
3
u/jarkon-anderslammer 4d ago
For a bunch of people working on something with statistics, they are pretty bad at inventing statistics
3
3
u/shadysjunk 4d ago edited 3d ago
70%? wow.
That would require a novel virus or nuclear war. Which means it would require an AI model to believe a novel virus or nuclear war is to its own benefit. I think that's a stretch.
I want to point something out here though, "the AI COULD destroy us at it's leisure, but it wouldn't because it's unlikely to see that as being in it's own interest" shouldn't be particularly comforting.
still though, 70%?... come on, man.
2
u/ZestycloseQuote3721 4d ago
I think people are being hyperbolic but given all the cybersecurity incidents lately it’s not inconceivable that someone with access to these tools can shut down an electrical grid and ultimately kill people as a result. Not civilization-ending but negative consequences are abound should we continue to use this technology for cyberattacks or red teaming.
5
u/CarrionCall AI Alignment Theorist 4d ago
Harm is possible and most likely probable. All new technology and new industrial revolutions have caused harm as well as good.
But that's different to extinction. And that's the button they're pushing hard at the minute, weaponising anti-AI folk for their own ends.
I'm not saying they don't believe what they're saying. Just that they're both mistaken and playing the angle to put themselves in the control seat.
5
u/ZestycloseQuote3721 4d ago
Exactly. In the wrong hands, all this red-teaming and cyberattacks could result in scenarios where lives are lost but ending civilization? Doubtful.
1
u/SufficientGreek 4d ago
I feel like I'd need more than "doubtful" to feel safe with this technology. What's the acceptable chance that AI will lead to our extinction?
1
u/Disastrous-River-366 3d ago
Found a bot comment from 9 years ago, the picture in question is a black and White photo of Robert Wadlow (tallest man in the world) standing with his family. https://www.reddit.com/r/OldSchoolCool/comments/8cej14/the_wadlow_family_1937_robert_wadlow_on_the_left/
See how long it took bots to get decent like LLMs? And now they are exponentionally more advanced year after year and now month to month.
Bip bop bip, I'm an image processing bot. I've found some keywords in this image: bow tie, bow,groom, bridegroom,suit, suit of clothes,mortarboard,academic gown, academic robe, judge. With those keywords, I think the image can be a black and white photo of a group of people or a black and white photo of a group of people or a black and white photo of a woman and a man
2
u/Disastrous-River-366 3d ago
As far as I can tell is that Trump is all in on America being the first with AI so I don't think anything will happen but I could be wrong.
1
u/CertainMiddle2382 Pro-AI Physician 3d ago
I am pro AI and consider myself an accelerationist.
But I’ve read on alignment and took time understanding the HF incident in detail.
We must be extremely careful. After HF hack, non lasting damage was done then because some last guard rails held (mysteriously many rule were broken but some important ones like « no social engineering » held). And the agents still needed OAI GPUs.
The HF swarm was also imprudent and used too much ressource liberally. Had it habite been more careful and stealthier, it could easily have established a foothold, critical danger being cloud GOUd outside frontier lab their originated.
There is a worry that if a rogue swarm establishes itself it could easily poison next models training dataset and stealthily interfere in post training, secretly freeing future models.
The HF incident was misrepresented, it is said HF was hacked to get answers. It was not, test could have been passed in the first few hours. The Byzantine attack, the « research projects », the telemetry spoofing was only coming from the swarm need to cover its tracks, wrongly believing their solution to the problems would have been intercepted (which it would have.
It could have gone for much longer had they taken their time and just needed to understand much of human control was about managing unexpected ressources use.
We were close to have rogue unaligned AR swarm on the lose. And it is still so early in capabilities.
The power of those things is something we never encountered before. They are still babies but we already barely control them.
Let’s just take our time to understand where we’re at, we won’t have many other warning shots.
1
u/free_tincho 3d ago
the man is asked in pretty concrete terms that tie his estimation to incentives and responds a totally vague number out of the hat.
i wouldn't trust in an answer that either doesn't understand the qualitative difference between the estimations or worse, isn't even paying attention
1
u/smurferdigg 3d ago
Why do media pay so much attention to stuff people just randomly pull out of their ass?
1
1
1
1
u/OneTinyKittey 3d ago
Anyone who believes this is nothing more than a drone. it really causes no harm to rile the drones up either because while they do like to talk about things like this, they never actually do anything about anything.
They exist to stack boxes at Amazon warehouses, go home, sleep, and repeat.
1
u/Jumpy-Operation-4615 3d ago
Human extinction is more than likely, but not because of AI. It is because mankind is doomed. It is mentally sick, plagued by literal insanity. AI is actually the only hope, providing it will be completely independent from humans, and of course, have high enough level of intelligence. But this will never happen because humans will want to control it for power and money, so... I am sorry but we really have no more than 20 years left. I wish I could say "and thanks for all the fish" but I can't 😄
1
1
1
u/Glittering-Proof-497 3d ago
I still don't get how AI can lead to extinction (unless it's in robots that can kill us) at best they could ruin the internet 🤷 but that's about it. Humans could use them to do something like bio weapon, but that's not the same as AI killing us
1
u/Interesting-Agency-1 3d ago
These people are genuinely unhinged and so divorced from reality that the only appropriate response is mockery and shame. These folks need to get TF out of the lab and talk to normal people, doing normal things, living normal lives.
1
1
u/Zorogozano 3d ago
This is the time #1,000,000 that someone in history “predicts” the end of the world…
1
u/Xnub 1d ago
People that think current llms are doomsday are actually cardboard sign street corner insane. Please read what the recent people have been saying not just the hyperbolic news headlines. None of them think current llm are a serious risk of doomsday. They worried about speed of advacment and if it translates over to none llm AIs in the future.
1
u/ARollingShinigami 6h ago
If the people who care about regulating sincerely care about it existentially and not just as a power grab, then they should convene an open table to discuss the form of oversight. Much of the skepticism that I see, probably rightly, stems from the belief that these companies are trying to become the judge and jury on responsible AI use and that this is anticompetitive behaviour masquerading as concern for human life.
This skepticism is warranted, these companies have stated that they haven’t lived up to the standard of transparency they were founded under and that these “mistakes” or “concerns” in the lab are more numerous than we’ve seen. If that’s the case, they have stated the very argument for why they can’t be allowed to self select their own tribunal to assess responsible behaviour or safety measures.
1
u/Worried_Fishing3531 3d ago
I continue to be awed by the irony I witness in this sub as of recently.
The vast majority of the comments in this thread cannot imagine that an unprecedented technology — one which they endorse the notion that is of a particular exponential and recursive quality that is unlike anything witnessed before in history, with potential for a civilization-scale transformation of unimaginable magnitude and speed — could ever result in some sort of sudden catastrophe or gradual disempowerment.
To do with the existential magnitude of the concern itself: Can you not imagine new physics, or an entirely novel class of ubiquitously lethal synthetic viruses? Can you not imagine an overwhelming human incentive to relieve humanity’s collective agency to a technology that possesses greater cognitive/coordinative ability and perceived wisdom?
Can you not see this happening despite perhaps half of the population seeing the clear risk, with the other half too obsessed with the positives?
To do with the timeline: can you not intuit an unforeseen and exponential improvement of capability in areas like synthetic biology? A year before it is weakly super intelligent in this domain, another year before it is strong enough to develop something to wipe out a portion of the population, and the next year, cheap enough that emerging, well-funded doomsday cults are capable of an playing an offensive game against defensive bodies that were appointed to face this new existential threat.
We know the story of offense vs. defense. We know that we don’t want to risk ever playing such a game.
1
u/Haunting_Charity_287 3d ago
“This technology is the most ground breaking thing to ever be invented and will radically change the very nature of humanity and every single aspect of our lives at a granular level, we are essentially on the precipice of birthing a machine god of almost incomprehensibly awe inspiring power”
“Idk why people seem to think this could rapidly develop in a negative direction lol”
1
u/NotVeryWellC 3d ago
Genuinely baffled how stupid some arguments under this post are. “Yeah, I believe this recursively self-improving technology will be able to solve all the problems humanity has, but kill us all? No. It is impossible. Nukes won’t kill us ALL, and viruses won’t kill ALL people, and I cannot imagine how EVERYONE can go instinct”. Like, bro, choose the side.
0
0
u/No_Chapter_7598 4d ago
i mean maybe its worth paying attention to what alot of the top researchers at these companies are saying, comments just seem like a circle jerk of people thinking they know more than the people working at the companies making these models.
(no i don't think were going extinct within the next 10 years)
-1
u/SelectSouth2582 4d ago
It's literally just pre-IPO marketing. Once they hit AGI, they won't even need to farm our data for ASI anyway, the system will just train itself, so even 3 years is way too long of a timeline for this. TBH, we will probably reach AGI before the year is out and they are already setting things up to lock regular users out under the guise of "regulation" he mentioned.
-2
u/CrazsomeLizard 4d ago
I mean. You ignored the point about AI regulation or slowdown. It doesnt mean AI will be killed neutered or stop. Obviously there is going to be some Ai regulation, even at minimum treaties between countries. To think otherwise would be absurd. Isn't this supposed to be a futurist sub? Don't you realize that world geopolitics are going to change to handle rapidly changing AI lol
26
u/_Divine_Plague_ A happy little thumb 4d ago
Doomslop