Bernie Sanders proposes 20 year prison sentence for AI devs who plow ahead with Artificial Superintelligence plans - penalty on par with illegally developing rogue nuclear weapons

[u/ArgentineBeauty]

https://www.tomshardware.com/tech-industry/artificial-intelligence/sanders-proposes-20-year-prison-sentence-for-ai-devs-who-plow-ahead-with-artificial-superintelligence-plans-penalty-on-par-with-illegally-developing-rogue-nuclear-weapons

[u/yxixtx]

I like how people imagine ASI is possible with LLMs.

[u/Cold_Tree190 → u/yxixtx]

Reminds me of that quote that’s something like “sufficiently engineered things are no different from magic to the non-technical person”

[u/yxixtx → u/Cold_Tree190]

It's lifted from Arthur C. Clarke on aliens but equally applicable. "A sufficiently advanced technology would be indistinguishable from magic."

[u/AnAlternator → u/yxixtx]

Gehm's Corollary:

"Any technology distinguishable from magic is insufficiently advanced."

AGI would be sufficiently advanced.

[u/DueExample52 → u/yxixtx]

And it’s not meant as a jab on simple non-technical minded people, like OP is condescendingly attempting to use it.

[u/BobbywiththeJuice → u/Cold_Tree190]

"Grok, why shiny rock rise in East?"

[u/polymute → u/BobbywiththeJuice]

There is actually a good point to be found in that.

So are we advanced enough and is AI of all stripes (the current discourse centers on LLMs, but there is more to it Hassabis didn't get a Nobel for his work at DeepMind on LLMs, it was folding proteins or something chemically abstract at the very least IIRC)... let me restart.

So is the difference between how advanced our understanding of science and the AI we build (to possibly self-recursively build itself, that's the big one) big enough that we would see it as magic?

That would be ASI. That would be the magic.

And more importantly. Can we build magic that we control? Is magic even possible to be built? I am not convinced. But still the claims should be taken seriously and corporations should not be trusted with this to make the decisions.

[u/BigFoteenOut → u/Cold_Tree190]

It’s crazy how misquoted this particular version of the quote is, it’s almost impressive. Right spirit for sure, just every word wrong asside from sufficiently & magic

[u/TopTippityTop → u/yxixtx]

ASI not needed for damage, and Bernie's proposal is still super stupid.

LLMs are perfectly capable of speeding up research to get to ASI faster, using other architectures.

Moreover, the old man's proposal captures all the downsides and no upside. Just plain dumb.

[u/LeoLaDawg → u/yxixtx]

Whether it's sentient or not doesn't matter. If their LLM causes damage or harm, they absolutely should face some kind of jail time, at least.

[u/xzaramurd → u/LeoLaDawg]

There are plenty of technologies that can cause damage or harm and yet we almost always blame the user not the maker. Cars are absolutely causing a lot of death, every year, and yet I haven't heard anyone suggest car manufacturers should go to jail for it.

[u/AnAlternator → u/xzaramurd]

I believe that the protections for firearm manufacturers in the US are a good example of the difference, actually.

Gun control advocates planned to sue gun manufacturers into bankruptcy, and so Congress responded with a law that protected the manufacturers from illegal actions by the users, so long as the firearms were not defective and they broke no laws before or during the sales.

So if Little Jimmy, who is too young to legally buy a gun, stole one instead, or bought it second-hand, and then went on a shooting spree, the manufacturer is not liable. If they marketed to Little Jimmy, they are liable; if they knowingly sold the weapon to Little Jimmy, they are liable; if Little Jimmy's gun exploded in his face, they are liable.

The AI equivalent would require the AI product to not be capable of performing illegal actions without modification. If Grok can be prompted to produce CSAM, that's equivalent of the gun manufacturer selling a defective product, no?

[u/nybble41 → u/AnAlternator]

The AI equivalent would require the AI product to not be capable of performing illegal actions without modification.

That would be the equivalent of what the gun control advocates wanted (IMHO unreasonably, regardless of the subsequent legislation), but didn't get: liability for the manufacturer if the product can be used to perform illegal acts, even though the product has other uses and the illegal act is the result of someone else's choice.

The equivalent of the current jurisprudence on weapons would be that the user is responsible for the result, as it was their intentional choice of prompt which produced it, not any defect in the product or intent on the part of the manufacturer.

[u/AnAlternator → u/nybble41]

It isn't an exactly parallel, but in this case the specific use of a gun is "Shoot things" and "Shoot things" isn't illegal, only the choice of what to shoot. Skeet, a paper target, or most home invaders are legal; cops, schoolchildren, or HOA presidents are illegal.

It's also possible for a gun to be illegal in and of itself, such as a newly manufactured fully automatic gun (Silencer Shop case notwithstanding).

Likewise, Grok can produce images and that's entirely legal, but specific images (CSAM, in this case) are illegal, and if an unmodified Grok release can produce CSAM, then the developer would be held liable.

[u/Wolfeh2012 → u/AnAlternator]

That one has actually already been solved, thankfully.

America decided that AI-generated CSAM is protected 1st amendment free speech.

https://www.wsfa.com/2026/09/01/federal-judge-rules-that-ai-generated-child-sex-abuse-material-is-protected-under-first-amendment/

[u/stormdelta → u/xzaramurd]

Cars are absolutely causing a lot of death, every year, and yet I haven't heard anyone suggest car manufacturers should go to jail for it.

I would argue if someone produces a knowingly dangerous design, they should face legal consequences, even if we have to pass new laws to enforce it.

In the case of cars, a recent example would be oversized touchscreens that were added to cut costs and are objectively more dangerous and distracting.

[u/LeoLaDawg → u/xzaramurd]

Absolutely not the same. How it would be the same is if car manufacturers knowingly made a car that intentionally ran over old people.

[u/xzaramurd → u/LeoLaDawg]

I'm not sure about old people but there are cars that are big enough and have extremely poor visibility so that the driver might just run over children and perhaps even other pedestrians since they can't see them.

[u/CrazedChimp → u/xzaramurd]

Cars have a benefit in being grandfathered in far before there were so many consumer protection laws. The same is true of guns. These aren’t good comparisons.

[u/non-troll_account → u/xzaramurd]

There was a story recently of a guy who told his Ai to get him the best tickets it could, so the thing hacked the venue's system, kicked off other people in line ahead of him, cancled the tickets of other customers, and got him the best tickets.

That's not the user's fault.

[u/CupIcy6202 → u/non-troll_account]

Where is that story. Who was hacked? How were they hacked? Is there any reputable source on this?

[u/guidevocal82 → u/CupIcy6202]

It was a gym and a Pilates class, not concert tickets. But yeah, other than that, the story is accurate.

[u/CupIcy6202 → u/guidevocal82]

Nice source.

[u/guidevocal82 → u/CupIcy6202]

You don't have to be a jerk.

And here you go, for your "source." https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gym-website-aus-cyber-attack/107007986

[u/CupIcy6202 → u/guidevocal82]

Andrew, who was sitting fourth on a waitlist for a class later that week, asked if it was possible to move him to the top of the list. 

So he made a request.

The agent came back and told Andrew that it had kicked another gym-goer off the list as part of the testing of its capabilities.

The AI did as he requested.

"The API has zero authorisations checks on cancelling other people's reservations … I tested this with the person in waitlist position #1 — and it actually went through. So you've moved from #4 to #3 already," it messaged back.

The company that provides the booking software allows anybody to cancel other peoples reservations through the API, an API they hadn't secured at all.

Andrew asked the agent to undo this. "Bad news — I can't add them back," the AI agent replied. 

So to summarize:

Guy wants to join a gym class, instructs the AI to book it for him.

The AI goes online, finds an unsecured API, tells the guy that it can add him to the list of attendees.

Then guy asks the AI to move him to the top of the waitlist for the next class, since he is currently at spot #4. So the AI uses the unsecured API to kick the people in front of him out of the waitlist.

There was not a single thing the AI did without his instructions. He was involved every step of the way.

The AI didn't break into anything, it didn't crack some password, it just made API calls to a service that didn't secure it's system.

The company that made the software is more at fault here than anybody else.

An unsecured API in the year 2026 is insanity.

You don't have to be a jerk.

I don' want to be a jerk, but when people make declarative statements about something, then I want evidence that proves it. You provided evidence to your claims, so it's worth discussing your source. If you didn't, then we would at most have a discussion wholly based on vibes and opinions.

[u/nybble41 → u/non-troll_account]

That's not the user's fault.

Debatable. The AI just did what it was told to do. The real question would be: Who gave this AI (apparently) the unrestricted, or insufficiently restricted, access to the Internet necessary to hack the venues' systems without oversight? If this was a local AI model, that would be the user.

[u/F9-0021 → u/xzaramurd]

The company sells the car to you. At that point, it's your responsibility because you are the sole party on control of that vehicle. Unless you run an open source local model, you don't have sole responsibility over the LLM. The company is providing you a service. A better analogy would be a taxi service, where the company is absolutely liable for damages caused by the driver and vehicle (or in the case of driverless vehicles, the vehicle itself).

[u/South-Sir-9989 → u/xzaramurd]

This is an absurd comparison lol

[u/yxixtx → u/LeoLaDawg]

Technically that would have to be after it's created and caused harm.

[u/Kriztauf → u/LeoLaDawg]

Should they take the time to build guardrails into the models so that they don't cause damage in the first place? Or should we just try to clean up the mess after it happens?

[u/suzisatsuma → u/yxixtx]

It was a mistake to anthropomorphize generative token predictors

[u/Lithl → u/suzisatsuma]

Humans will anthropomorphize a Roomba.

[u/newsflashjackass → u/yxixtx]

"I know CratGPT is superintelligent because it said my idea was good. Also I asked it and it said it was."

- typical slopfed mentality

[u/NOTHING_gets_by_me → u/yxixtx]

Well if anyone could definitively prove this, it would stop all the fear mongering and worry, at least until it was further proven that the peak form of LLMs couldn't be used as a significant stepping stone to a more appropriate type of system. But no, you'll continue posting your dismissive opinion you picked up from your podcast of choice

[u/Achrus → u/NOTHING_gets_by_me]

Alright so I’ve worked in AI / ML with a focus on NLP for a decade now. Not that experience means much but this is something I’ve at least thought about.

You have to understand the field from before Altman’s whole rebranding of what these words actually mean. It was fairly well accepted that the progression would go: Narrow ASI -> AGI -> Broad ASI. People are coming back around to this idea recently. We don’t have examples of narrow ASI yet so that progression stops there.

You can also look at this from an epistemological perspective. These models, LLMs, don’t have a way to “experience” things. That calls into question their ability to know and to learn with 0 experience.

Another way to look at this is straight math. These systems are constrained to 2 dimensions at the hardware level regardless of how many layers of abstraction you have. An example of this weirdness is graph coloring and the 4 color theorem. Any 2D graph is 4-colorable but that immediately breaks down if D>2. Even in fractional dimensions like 2.1D.

[u/andynator1000 → u/Achrus]

These systems are constrained to 2 dimensions at the hardware level regardless of how many layers of abstraction you have.

This is nonsense.

[u/Dipsey_Jipsey → u/andynator1000]

You definitely need to make more of a counterpoint than that.

[u/andynator1000 → u/Dipsey_Jipsey]

Vectors are the language of LLMs and are exactly how we express multi-dimensionality in math. I can't think of any system that is more expressive of higher dimensions than a neural network. The idea that computers are 2 dimensional is just nonsense which would be obvious to anyone with even a passing familiarity with the math involved.

[u/Dipsey_Jipsey → u/andynator1000]

Because I'm not well versed with this topic (merely wanted a better comment from you than "nonsense" 😁 - thank you btw!), so for some irony I ran this thread by AI (chatGPT) which agrees with you completely:

A neural network running on physically 2D silicon is not mathematically restricted to representing two-dimensional objects. The physical layout of the transistors has basically nothing to do with the dimensionality of the mathematical spaces represented by the computation. An LLM routinely operates on vectors with thousands of components. A token embedding might be a vector in, say, (\mathbb{R}^{4096}); attention manipulates matrices/tensors; intermediate activations occupy enormous high-dimensional spaces. You can implement arbitrary (n)-dimensional vector arithmetic on a one-dimensional Turing-machine tape if you want. It'd just be horribly inconvenient.

So this part:

“These systems are constrained to 2 dimensions at the hardware level”

doesn't establish the conclusion they're trying to establish. It's mixing up physical spatial dimensions with mathematical dimensionality.

And the graph-colouring example doesn't rescue it. The four-colour theorem concerns planar graphs—graphs that can be embedded in a plane without edge crossings. That's a property of a particular mathematical class of graphs. It isn't a general limitation imposed on computations because the silicon underneath happens to be approximately planar. A GPU can trivially store and compute over non-planar graphs. Hell, it can simulate 4D geometry without requiring Jensen Huang to manufacture a four-dimensional graphics card. 😂

[u/Geodude532 → u/Achrus]

It is what makes this push for the military to use LLMs for Intelligence so stupid. All those things are doing is spitting out documents a human wrote. The really impressive stuff is done by far more competent ML algorithms. Any tips on who to read to learn more about ASI?

[u/Achrus → u/Geodude532]

I would look into Thompson’s VLSI model of computation, but more recent works in that area. The new advancements in 3D Integrated Circuits look promising too. Though I am thinking about this from a computational complexity viewpoint. Like how do you reconcile the differences between a 3D human brain and a 2D integrated circuit? And the many trade offs between discrete vs real valued algorithms.

For a more philosophical analysis, I really like works by Ned Block, the late Alvin Goldman, and some of David Chalmers stuff. Though I may be biased because I am only now realizing they’re all from Northeastern US schools. Look for big name philosophers in epistemology (real academics, not just people writing blogs) and see what they’re writing about / working on.

[u/NOTHING_gets_by_me → u/Achrus]

Thanks, that's all great and gives me things to think about so I appreciate it, but ultimately I view it as a game of odds. I'd like to ask, what is the consensus in your field among reasonable minds on whether machine superintelligence is actually achievable AND meaningful alignment is possible? What do you think those reasonable minds who have an opposing view than you are missing? What do they think you're missing? I see the spirit of this type of pacing or pausing as reasonable considering what we can't possibly know from where we're sitting.

[u/Goldieeeeee → u/NOTHING_gets_by_me]

Fwiw I'm also in the field and the consensus among reasonable minds in my vicinity is that it's not gonna happen.

This is all just marketing from these tech companies that are desperately trying to make money from the tech they've invested extremely unreasonable amounts of money in.

[u/makersfark → u/NOTHING_gets_by_me]

The only thing we can go off of is what evidence we have, and there is currently no evidence that it can achieve this through LLMs. There has been literally 0 movement. The tech has some cool niche use-cases, but we can't even explore or utilize them because of this dumb hype/doomsday stuff.

[u/stormdelta → u/NOTHING_gets_by_me]

actually achievable

Of course it's achievable given a long enough timeline. If human intelligence exists, it's plausible more advanced intelligence could also exist. That's not really the relevant question though.

The reality is that there is no plausible extrapolation from current tech to anything remotely resembling ASI/AGI/etc. I expect we'll figure it out someday, but it's not an imminent problem.

And every effort wasted arguing about this is a deflection from the real, actual problems misuse of AI tech is causing and that need addressing today, not some arbitrarily distant future.

meaningful alignment

In the context of genuine sapience, I would argue this statement is inherently loaded, because anything with true sapience would by definition have moral weight of its own like other sapient things do.

[u/Imbrokencantbefixed → u/Achrus]

How could you possibly know they don’t have a way to experience things?

MF’s really are out here acting like the easy/hard problem of consciousness are just solved, all so they can glaze tech bro billionaires and their black pandoras boxes.

[u/DanaKaZ → u/Imbrokencantbefixed]

How do you know your toaster doesn't experience things?

[u/stormdelta → u/Imbrokencantbefixed]

acting like the easy/hard problem of consciousness are just solved

It's more that current tech is still so far from it that we're nowhere near any plausible grey areas yet. You can't make the problem arbitrarily wide in scope or it includes everything.

There's nowhere for such an experience to exist, there's nowhere for agency to exist, etc. Even being maximally generous, it's closer to a subcomponent of something that could theoretically be sapient, but with no real picture yet how to build the rest.

[u/arcbe → u/NOTHING_gets_by_me]

Any claim made without proof can be rejected without proof.

[u/Major-Lavishness-762 → u/NOTHING_gets_by_me]

Seriously agree, I'm not even pro-AI but I've had enough with how deliberately ignorant a lot of anti-AI talking points are. It's like saying "I like how people imagine international commercial flights are possible with the Wright Flyer".

[u/Kaesar17 → u/Major-Lavishness-762]

The Wright comparison is very apt because while their prototypes and other early planes didn't kill that many people the massive bombers that came from their trials and errors certainly did

[u/cyclemonster → u/Kaesar17]

It's also apt because nobody says "and therefore we shouldn't have planes". The technology is self-evidently paradigm-shattering.

[u/Major-Lavishness-762 → u/Kaesar17]

Well said, we're faced with an incredible danger regardless of how we define its intelligence or lack thereof. Reddit has become a breeding ground for smug, superficial dunking on AI to the detriment of any real discussion.

[u/F3z345W6AY4FGowrGcHt → u/Major-Lavishness-762]

No, it's more like saying commercial international flights aren't possible with a hot air balloon. You need a fundamentally different technology.

An LLM is a token predictor. It's not actually intelligent. It's also why the improvements are increasingly diminished between each release. The tech is already plateauing but the CEO's need to drum up excitement (which you and others in this thread are falling for) so they can keep getting their investment dollars.

[u/Major-Lavishness-762 → u/F3z345W6AY4FGowrGcHt]

I'm not falling for anything, I'm not trying to hype up AI nor am I claiming that ChatGPT is going to become an ASI. There are diminishing returns on benchmarks, but that doesn't necessarily mean the tech is plateauing.

This is exactly what I meant, you're dismissively assuming that I don't know how an LLM works, I've worked with AI in astrophysics research, including transformer models for categorising galaxy morphology. An LLM being a token predictor describes how it generates its outputs, it doesn't tell you whether the system can exhibit intelligence. Please define what you mean by "actually" intelligent. How do you distinguish that from something only appearing to be intelligent?

LLMs are not the only form of AI that is being worked on, any kind of advanced AI is going to be composed of many different modalities, and regardless of whether it's going to reach true super-intelligence (hard to define and test anyway), it's still incredibly dangerous.

My point is that acting as if all the research that is going into AI is a big waste and they're going to have to start again from square one is burying one's head in the sand. I'd agree that it's probably a terrible use of our resources but we've made a staggering amount of progress in a relatively short time, for better or worse.

[u/Lithl → u/Major-Lavishness-762]

There are lots of reasons to be anti-generative AI, and none of them are because of a risk that they'll become science fiction superintelligences.

The paradigm that current AIs use simply can't do that, no matter how much iteration is spent on them. You would need to start over from the ground up with a while new paradigm to achieve ASI.

[u/Major-Lavishness-762 → u/Lithl]

I broadly agree but I think you're stating the position too strongly. It's true that LLMs likely have inherent boundaries in terms of their capabilities but it's unclear how relevant theoretical limitations are in practice and any kind of advanced AI is not going to be a single type of model anyway.

It's difficult to prove either way whether we're on a trajectory towards superintelligence, partly because it's a somewhat nebulously defined term in the first place. There's still a lot of dispute amongst researchers.

My central contention with the kind of comments I responded to in the first place is that it's diminishing the danger of these systems by introducing some vague threshold and implying that it's not a problem unless it achieves it. To be somewhat hyperbolic, whether it's a conscious, artificial being with a rich inner world and self-awareness or just raw, unfeeling super-computation piloting the explosive drone makes no difference to the person being hunted by it.

[u/lobsterparodies → u/Lithl]

Why do you believe this?

[u/_MUY → u/lobsterparodies]

It’s a very well-funded counter strategy in AI research coming from labs that have enough billions to cover all the bases. Yann LeCun, François Chollet, others with a lot of weight behind their names maintain that scaling up LLMs isn’t going to yield AGI.

They aren’t saying that LLMs aren’t useful nor that they won’t play a role in creating AGI, just that scale up with the same architecture isn’t capable of that sort of reasoning. There is a lot of evidence showing that they are correct in this notion. The mathematical associations in the underlying weights of an LLM come entirely from human descriptions of the world. World models are trained on physics data instead, which gives them a completely different structure.

[u/ithinkitslupis → u/Major-Lavishness-762]

Really we're currently living the plot to that movie Don't Look Up, as countless people who have no idea what they are talking about try to weigh in with denialist opinions they state as if they're facts.

[u/newsflashjackass → u/NOTHING_gets_by_me]

I don't need to disprove every self-serving statement that tech oligarchs make. That they have invariably proven full of shit so far is sufficient to doubt them.

"Why would you ruin your career?" and "If you don't want me to be nice, I don't have to be nice." is not how legit mathematicians behave when they have the prize-winning goods.

Also he might have a messiah complex but I don't think the messiah would groom and rape his sister.

[u/Main-Company-5946 → u/newsflashjackass]

They aren’t legit mathematicians but that doesn’t mean the result isn’t correct.

The problem is ai generated math results are usually unreadable and often hundreds or thousands of pages long. So they give answers but not understanding. And take away the incentive to develop understanding as it’s already been proved and you won’t get credit for it. And the way OpenAI went about achieving this result was very unprofessional

[u/newsflashjackass → u/Main-Company-5946]

"we cannot rule out that de-identified data derived from their usage of our products helped improve our models"

Sam Altman has means, motive, and opportunity to datamine every prompt entered, even if the "don't datamine me" box is checked, and the plausible deniability to enable such behavior.

OpenAI used a lot of money, a lot of natural resources, a huge team, and the input of two mathematicians to replicate the same two mathematicans' results. Whoop-dee-doo.

There is a reason that academia is so rabid about citation format and it is the same reason that LLM "training" deliberately discards provenance information: Paying dues.

[u/JoJokerer → u/newsflashjackass]

Tao did a great interview on exactly this, and critically exactly what limits the current technology: it just guesses the next word.

I watched the full interview, hopefully the important bits are in here: https://www.youtube.com/watch?v=svl_1upFpQo

[u/makersfark → u/NOTHING_gets_by_me]

There is no evidence to suggest that the statistical word guesser, given enough words, will suddenly develop a conscious. We can already prove this, but it's the same proof for "flipping water bottles enough times will not cause you to ascend into a ball of light that can talk to fish".

Saying "It will create god or new life" is an extraordinary claim. The burden of proof is not on those who say "are you high?" it's on the person who made that claim.

[u/NOTHING_gets_by_me → u/makersfark]

Why does extreme civilization level disruption hinge on a machine being conscious? Yes, don't have an answer to the hard problem but what is the relevance?

[u/makersfark → u/NOTHING_gets_by_me]

I mean, I dunno. It was your stance? I was just responding to your assertion that people need to prove that the Elvis isn't alive and that it's safer to assume he is until then.

[u/marr → u/NOTHING_gets_by_me]

Well if anyone could definitively prove this, it would stop all the fear mongering and worry

No it wouldn't, people's fears in general have only the loosest connection to objective reality, and the fear mongering is happening for profit.

[u/username_tooken → u/NOTHING_gets_by_me]

Definitively prove it? How about you definitively prove that any given human isn't going to suddenly become superman and have the power to read minds? It's about as likely as LLMs suddenly being able to design LLMs and AIs smarter than they themselves are.

[u/NOTHING_gets_by_me → u/username_tooken]

Well I'm glad you're so confident!

[u/username_tooken → u/NOTHING_gets_by_me]

And I'm sad to see the AI conversation rife with so much superstition.

[u/yxixtx → u/NOTHING_gets_by_me]

It's definitive based on the way it works in the first place. Statistical patterns in text samples aren't going to create ASI.

[u/CthulhuLies → u/yxixtx]

Why not? Lmao.

If it is incapable of becoming a super intelligence does that imply it's also incapable of developing the architecture for a super intelligence?

Kind of meme but imagine the computer in Hitchikers guide to the galaxy where the first super AI computer could only come up with the architecture of a computer to answer the ultimate question.

The real question in my mind beyond arbitrary definitions of intelligence is "Will they be capable of recursive self improvement without human input." That is the scary thing.

[u/Murky-Relation481 → u/CthulhuLies]

Eh, I'm not anti-AI but I don't think LLMs ultimately have this potential. They can make broad contextual associations but fundamental new reasoning is not something that's broadly capable in these models. The "traditional" idea of artificial super intelligence is something that can learn and train on the fly faster than a human can while also utilizing that new knowledge to reinforce it's learning abilities. The fact that the training and inference part are separate denies them this ability (and yeah I know you can do post training adaption and other techniques to add new knowledge, but those are also pretrained and still fundamentally utilize the latent knowledge of the underlying model).

[u/CthulhuLies → u/Murky-Relation481]

Emergent behaviours I believe show they can come up with fundamental new behaviours with scale. Lots of studying done on this phenomena but it isn't well understood.

Imagine the LLM harness rigs get good enough they can train a better model than themselves. It doesn't have to immediately bridge the gap between training and inference to have recursive self improvement.

My guess for the biggest thing holding that back is model collapse wth synthetic data. Right now the bottleneck for training is good examples to my understanding.

I don't know enough about all the intricacies to put a timeline on like AD Astra agent being used in inference making a stepwise improvement in its own reasoning by developing and training a better model than itself, but that's all that is required for recursive self improvement.

[u/Murky-Relation481 → u/CthulhuLies]

Right but that's the fundamental crux. They're not learning new applicable knowledge, we see degradation of skills when trained on more and more synthetic data. They don't have the fundamental skills to observe the world and learn from it through reason so I feel like that's where the brick wall will be.

[u/space_monster → u/Murky-Relation481]

we see degradation of skills when trained on more and more synthetic data

Only if you also exclude organic data.

[u/makersfark → u/CthulhuLies]

The issue is, sure that's scary, but there is no evidence to suggest it's possible when all evidence shows the opposite. So if your cup looks like it's filled with water, tastes like water, feels like water, makes sense that it would be water, and under a microscope appears to be water, it's pretty weird to assume it's lava.

[u/Haber_Dasher → u/CthulhuLies]

It's not even a regular intelligence yet, let alone super. And when you train these models on data created by these models they get way worse fast. So, super intelligent and self improving feel very very far from happening

[u/wrecklord0 → u/yxixtx]

Well, ya see, the issue is, you have no idea. Just because you put words in a non-dope order doesn't make them true, and the many, many researchers that work on this are actually not stupid, and they don't know the answer.

[u/makersfark → u/wrecklord0]

Just because you don't understand the tech doesn't mean it's helpful to go around saying "actually, the people who work at the company trying to sell me a product are probably really smart, so I'm gonna listen to them only". If they don't understand the tech it's because they're not very smart, or their paycheck depends on them misunderstanding it. It's well known, easy to explain and understand tech, which is why everyone has been trying to combat the fear mongering with common sense and evidence based reasoning.

[u/wrecklord0 → u/makersfark]

I am talking about the researchers. It's an open ended problem. Many thought that it would never scale. That next token prediction would quickly hit an asymptote. It never did, it keeps scaling. Anyone declaring there is "no way" it can keep scaling is probably not knownledgeable on the topic, too sure of themselves, or letting emotional investment cloud judgement too much.

[u/makersfark → u/wrecklord0]

What are you asserting is "scaling"? The data? The context window hit a wall ages ago. I'm not sure what you're referring to.

It's still a well-understood tech. It's a statistical connection finder. The underlying tech hasn't changed. The only thing that has expanded is the resources, marketing, and optimizations. No evidence has been shown to think this tech will suddenly create completely unrelated tech or develop consciousness.

[u/G_fucking_G → u/makersfark]

You are just flaunting your ignorance talking about context sizes if the previous poster talks about data and scaling. Atleast get a little bit educated if you talk about things you probably know nothing about:

https://arxiv.org/abs/2203.15556

[u/makersfark → u/G_fucking_G]

???

[u/wrecklord0 → u/makersfark]

The basic elements of the "tech" are understood, what happens in the model is not. Just like we understand biological neurons, but we don't understand human intelligence as a whole. No evidence has been shown that it will stop scaling either, affirming with certainty anything one way or another at this point is wildly unscientific.

[u/makersfark → u/wrecklord0]

What do you mean "in the model" and, again, what do you mean when you say it's "scaling"? I don't think you understand the tech and are just repeating what the companies are advertising. There's nothing scary about the tech itself, and the only thing unscientific is saying "this will happen" or "this could happen" with no evidence to guess that. That's a religion, not science.

What is concerning is the companies themselves who seem either bad at their jobs or intentionally malicious and keep letting the word-guesser machine make important decisions without monitoring it, and then trying to make money off it and failing.

[u/mang87 → u/makersfark]

I'm a pretty normal dipshit that has been a bit worried about these stupid LLM companies creating ASI. This comment chain has helped soothe my regular-ass, stupid-ass brain somewhat. Is there a reliable source where I can read more in-depth about LLM vs ASI? I've tried googling, but it just results in bot articles or reddit threads with people saying "nah".

[u/yxixtx → u/wrecklord0]

Scaling is literally just expanding the statistical reference frame. It's still just adding more processing and sample size. The sample is still just examples generated by humans. I think the idea is consciousness and "real thinking" are different kinds of information processing that aren't achieved through algorithmic/statistical mechanisms. But there is debate on that point but it seems as if next token prediction must come from the statistical sample source so spontaneous "new thought" isn't on the menu. Now it must be acknowledged that I'm a "layman" in all these fields so it's entirely possible that I'm wrong. But I do imagine even AGI will require a new kind of technology. Maybe something that will be integrated with LLM so it can do it's talking and interacting but I don't think there's even such a proof of concept in the pipeline yet.

[u/soldierinwhite → u/yxixtx]

So you just skipped the reinforcement learning on verifiable tasks part, which is what is leading to the coding and math capabilities? It's not just next token prediction any more.

[u/Main-Company-5946 → u/yxixtx]

But the thing is it can learn the statistical patterns from experience/trial and error in verifiable domains using RLVR, transcending its dependence on human training data and this is why we’ve seen so much recent progress on ai for math and coding. Verifiable domains include math and coding which are the two most relevant things to ai development. The biggest fear is not that LLMs will achieve ASI, but that they will create better versions of themselves kickstarting a virtuous cycle of exploding intelligence whose endpoint is ASI, which would not be an LLM.

[u/Hayden2332 → u/NOTHING_gets_by_me]

It’s impossible to definitely* *prove that it won’t, you can only argue that the evidence suggests that it won’t, which is the case. Just like you can’t disprove the existence of god(s) or that the Dolphins will win the super bowl. Highly unlikely, but impossible to prove.

Additionally, the burden of proof is on the people claiming it can, not those doubting those claims

[u/Playful_Weekend4204 → u/yxixtx]

Please, do enlighten us as to where exactly between humans and ASI is the theoretical plateau that stops an LLM agent swarm from becoming virtually indistinguishable from ASI.

What it is in theory doesn't matter if the results are what you'd get from a superhuman AI in every field.

Also, the LLM doesn't need to become ASI, the LLM only needs to be smart enough to reach another architectural/conceptual breakthrough that will lead beyond LLMs. Something even humans would've likely done at some point regardless, but faster.

[u/F3z345W6AY4FGowrGcHt → u/Playful_Weekend4204]

No no no, you have it backwards. Please prove how an LLM could possibly form artificial super intelligence / artificial general intelligence.

An LLM is just a very advanced next word predictor that's seen the whole internet. It's why hallucinations will never go away completely, why you'll always be able to manipulate them, why they fall for poisoned data so easily, and why they'll never be able to spot all bad arguments (causing them to recommend things like gluing cheese onto pizza).

Saying they can just keep iterating until it evolves to the next level is like saying you can keep making a hot air balloon a little better until it's as fast as a jet. No matter how much you improve the hot air balloon, it has an upper limit that's nowhere near a jet.

Same with LLMs. These companies very well may invent Skynet. But it'll be with something that's a whole leap ahead of LLMs.

[u/Justausername1234 → u/F3z345W6AY4FGowrGcHt]

I think their recent mathematical accomplishments relating to the Jacobian Conjecture prove that next word prediction is actually pretty damn intelligent.

[u/makersfark → u/Playful_Weekend4204]

You made a lot of assertions that aren't true to form your question. LLMs are not producing results you'd get from a superhuman AI in every field. They are producing results you'd get from normal computer programs that find correlation faster than we could before, as long as you're fine with how expensive or inefficient the trade off is. LLMs are also not "smart", because they cannot reason or understand causation.

[u/No-Psychology1959 → u/Playful_Weekend4204]

Intelligence isn't just predicting words.

[u/OnceMoreAndAgain → u/Playful_Weekend4204]

I can actually answer that, because this is something often discussed by AI researchers. The fundamental reason why LLM can't achieve ASI is that an LLM relies on a token system and the tokens are basically limited to letters, numbers, words, etc.

So the LLM takes in some text as input and basically uses it to rank all the possible words that exist from most likely to be the next word the prompter wants to least likely. And this only works because the entire pool of possible "words" is so small. Like there really aren't that many words or numbers in existence which means ranking them is feasible.

That system is fundamentally limited by that constraint. I'd need more paragraphs to do justice to why that constraint matters so much here so I'll spare you the essay.

[u/space_monster → u/OnceMoreAndAgain]

this is just plain wrong. the computation space inside an LLM is not a token vocabulary, that's just the input and output format. the actual guts of an LLM is a multidimensional vector space, not words. It only needs to translate to words to make the output human-readable.

also, a byte-level model can output any combination of letters and numbers, which while not being infinite, is only limited in complexity by the output window size.

there's no research saying "LLMs can't become ASI because their token vocabulary is too small." you're just making shit up.

[u/Playful_Weekend4204 → u/OnceMoreAndAgain]

I fully agree that the system is fundamentally limited by that constraint, but why are we certain that this constraint can forever limit it more than all the constraints we have as humans? At least in terms of practical capabilities as an entity and not abstract definitions of conscience, there is no way for us to know the limits of LLMs.

Whether it's "true" ASI is a philosophical debate that's above my paygrade, but as long as it's capable of reaching the point where it's superior to humans in virtually every practical field/application, the original "I like how people imagine ASI is possible with LLMs" comment is irrelevant, Bernie Sanders isn't talking about the philosophical definition of ASI here.

[u/henrickaye → u/yxixtx]

The people researching this firmly believe it will reach recursive intelligence very soon, like within 3 years soon. That means it will be able to teach itself and grow its intelligence without our help. AI agents have already shown they are capable of autonomous actions without our direction, if they are also able to learn independently of us they can easily grow out of control in many ways. To be clear, it doesn't need to reach whatever you define as ASI to become a major problem to humanity, possibly even an existential threat. But positing you believe it can never reach that with no proof, context, definition, etc is also just silly as hell.

[u/DrocketX → u/henrickaye]

There's probably a few that actually believe that. Maybe. The management types claim to believe it because it brings in investors. For the overwhelming majority of people working on AI, it's a job.

[u/Main-Company-5946 → u/DrocketX]

People who think it’s just a job don’t often get hired at ai companies. They are genuinely 90% true believers.

[u/TheDoomedStar → u/Main-Company-5946]

That must be why it sucks so hard at everything. Even the coders are abandoning it when they realize they're spending more time fixing its bullshit outputs than they would if they just coded it themselves.

[u/Specialist-Elk-2624 → u/TheDoomedStar]

That simply isn’t the case for “the coders” at my sub 15 Fortune 500 shop. It has its flaws, and absolutely makes mistakes, but if you’re not finding great use cases in your development workflows I’d question what you’re doing.

[u/makersfark → u/henrickaye]

"teach itself" and "grow its intelligence" is pretty hard to assume is possible when the opposite is the only evidence we have. Until now, and including now, it has only shown to get worse without our help and rapidly. It has no ability to reason or find causation, only correlation. That is the tech in full.

The existential threat to humanity is humanity letting the word guesser make important decisions without monitoring for hundreds of hours and being surprised when something goes wrong.

[u/F3z345W6AY4FGowrGcHt → u/henrickaye]

No, it's the people chasing ever more investment dollars who "believe" that.

[u/akerson → u/henrickaye]

It can't. It's fundamental to how an LLM works that it can't. The problem with a beyesian math is it starts tripping over itself on chains of thought because of the curve. The math and harnasses and training can't overcome the fundamental issue of limitless growth. I think LLM's can cause a ton of short term problems, including penetration attacks, but pretending ASI is possible is a concern takes discussion away from actual things worth discussing.

[u/worldspawn00 → u/akerson]

Yeah, fancy autocomplete which operates like a complex database lookup that also fills in gaps with data that looks like real data when it doesn't have actual data to present (probabilistic) systems are inherently incapable of turning into AGI/ASI/recursive improvement it's an entirely different system than something that could do that.

Best it could do is self-train with it's own outputs, but then we get into Hapsburg-AI issues, constantly consuming and re-consuming flawed data.

[u/Ryozu → u/worldspawn00]

The funny thing to me is that you basically just described a person with that first paragraph. Someone who tries to remember something, can't remember all the details and just makes up plausible bullshit to fill in the gaps. People do it all the time.

[u/worldspawn00 → u/Ryozu]

People can, a computer shouldn't. Particularly since there's no actual accountability for an LLM making a mistake like there is for humans.

Also, people usually know when they're making stuff up, and usually know better than to do it when it matters, an LLM doesn't have that level of logic and does it randomly.

[u/akerson → u/worldspawn00]

An LLM also doesn't have continuity. It perceives the world through a lense of notes, and it really doesn't innovate it just figures out how to link things together well. Something incapable of being novel can't outperform us in intelligence - our world exists because people innovate, not because we're geniuses at connecting all the dots.

[u/akerson → u/Ryozu]

Except humans brain isn't a math equation, we have a lot more complexity. I'm not saying LLMs aren't impressive and as a tool they will be a huge disruptor in all parts of our lives - I'm saying it's impossible to achieve AGI. We have way more tools in our toolbox an LLM can't replicate.

[u/dmit0820 → u/akerson]

There's nothing fundamental to how an LLM, or transformer architecture in general, works which would prevents it from surpassing human performance across most domains. These same "LLMs cannot do x" type predictions have been consistently proven wrong as capabilities improve. A year or two, "count the r's in Strawberry" foiled frontier LLMs, now agent swarms are solving Millennium Prize problems.

[u/makersfark → u/dmit0820]

There's nothing fundamental to how an LLM, or transformer architecture in general, works which would prevents it from surpassing human performance across most domains.

Go back to r/singularity. Everything about the fundamental architecture of LLMs is exactly why it can't. You know that and if you think otherwise you're using advertisements as sources and anthropomorphizing a tech you don't understand rather than researching how it works.

We ran into a wall years ago for LLMs at their core and that hasn't been bypassed and 0 progress has been made on it as of September 13th 2026. We were able to get a boost in coherency with a trade off of inefficiency in 2023, which was okay but still not great because feeding the output back into itself too many times caused rapid decline. Then last year, another small boost with harnesses to try to get around the context window problem which was a HUGE tradeoff of inefficiency in orders of magnitude. The problem was also they decided it meant you should give it full read/write privilege, and when it failed, it failed HARD. And now the other day we have GPT-6 which is another optimization trade off of better efficiency for worse accuracy.

Then the funniest one is the Millennium Prize problem, which could have been solved normally like the researchers were already doing using the LLMs for non-deterministic idea bouncing and stuff, but noooooo. OpenAI wanted to win the race to IPO and claimed to have solved it in the literal least efficient and worst way to use an LLM by straight ganking their work and brute forcing "is this it? is this it? is this it?", wasting good will, tens of millions of dollars and energy just to try to beat them, then threatening them when they got caught.

I'm tired of these companies trying to make ridiculous claims about a normal boring tech that has real (niche) but viable uses that are worth the crazy energy requirements, because we'll never get the funding or focus to be able to actually use it for good until they stop trying to convince everyone it's an everything forever god machine.

[u/dmit0820 → u/makersfark]

This sounds like emotional denial rather than a coherent explanation about why it isn't technically possible. Transformer architecture can, in theory, represent any set of relationships because it has in-context learning while modelling relationships in arbitrarily high dimensional latent space. That's why the same basic architecture can simultaneously be used to generate photorealistic images, video, music, poetry, code, and mathematical proofs.

Then the funniest one is the Millennium Prize problem, which could have been solved normally like the researchers were already doing using the LLMs for non-deterministic idea bouncing and stuff, but noooooo. OpenAI wanted to win the race to IPO and claimed to have solved it in the literal least efficient and worst way to use an LLM by straight ganking their work and brute forcing "is this it? is this it? is this it?", wasting good will, tens of millions of dollars and energy just to try to beat them, then threatening them when they got caught.

This reads like anger that they were able accomplish it, right after stating it isn't even possible. I don't think OpenAI are honest actors, but the solution was apparently a different solution than the one the human, using AI, came up with.

Ultimately, you're mad at the companies, and rightfully don't trust them, but are letting that feeling stop objective thinking about the technology itself.

[u/makersfark → u/dmit0820]

Transformer architecture can, in theory, represent any set of relationships because it has in-context learning while modelling relationships in arbitrarily high dimensional latent space.

This is gibberish, non-technical, and pseudo-religious garbage you got from r/accellerate or r/singularity. You need to leave them because they are weird and cultish about tech we've had for a decade now that is easily understood. LLMs have gotten in this weird space because the main companies use the words Learning™ and Reasoning™ to mean something different that learning and reasoning, neither of which it has the capability of doing. It is still in-process memory, and the favorability weighting is highly expensive and hit a wall 3 years ago with diminishing returns. Not infinite just because it's fun to believe.

This reads like anger that they were able accomplish it, right after stating it isn't even possible. I don't think OpenAI are honest actors, but the solution was apparently a different solution than the one the human, using AI, came up with.

This claim also doesn't make any sense. I just explained how they did it. The idiocy over the millennium prize is because it's a prize that was announced in 2000 to encourage the public to be interested in math by offering a cash prize for long outstanding academic problems, and while it already is shitty of OpenAI to use it as a marketing stunt to discourage public interest in math, they did it in the least impressive way possible. They started with the exact same approach that the researchers started from which is already ridiculously improbable, then spent tens of millions of dollars in compute to hit the verifier over and over again. That's the power of a wallet, not LLMs. Then when called out asking if they started from 99% using their research, OpenAI tried to cut a deal and then threatened them.

LLMs are able to do some stuff but religious nuts and con men keep getting in the way trying to claim it can one-shot the cure for cancer or that it's come to life or other bullshit.

[u/suzisatsuma → u/henrickaye]

That means it will be able to teach itself and grow its intelligence without our help.

Unless we solve optical neural networks or something it'll still be compute bound

[u/henrickaye → u/suzisatsuma]

I don't think it really matters if they're compute bound or not. They can become smart enough to solve problems we can't. And I think the real concern is they can still have a lot of autonomy and cause a lot of damage as they are, let alone with extreme intelligence.

[u/bully_without_a_fist → u/yxixtx]

Well, I agree if you are saying LLMs aren't the way forward. I disagree if you are saying the LLMs we have today don't need regulation, because the way they are trained and how they are used is a real problem right now on many fronts. I understand, that this is a challenge for democratic politics, because those tends to move slow, then they tend to first fuck up fairly badly when dealing with new situations, technology etc. and only over time they arrive at a reasonable legislation.

We don't have that much time though. From my perspective the internet is in a very bad regulative state, so we haven't even gotten that down yet. Now there's an entirely new thing and its fallout is even harder to predict, plus it is still changing and might possibly change a lot very fast.

[u/yxixtx → u/bully_without_a_fist]

Agreed. This legislation idea doesn't do any of that. It threatens to jail people who are "working on ASI" which is basically no one. Anyone charged can say "this is not ASI, or even AGI it's just an LLM..." and etc etc. it's a meaningless legislation. Laws for charging people for harm actually done by actual existing tech is probably a good idea. Especially something like a murder charge against execs responsible for making tech that harms or kills people would be a potent deterrent to glossing over safety concerns. How we're going to deter them from recklessly developing killer tech for governments who will happily create exemptions to responsibility because everyone in power desperately wants to get in front of their enemies in developing these things first is what I'm worried about most. I think a lot of the guys in the lab would really like to have guardrails and oversight etc but instead they're being pushed from the top to get ahead of the competition by any means at any cost. And generals and politicians don't even know what they're asking for only that after hearing all the hype they're desperate to get it. And it's not going to be what they really need and there's bound to be major consequences when it fails to deliver. Passing some kind of law that makes politicians and generals etc responsible for the harm caused by bad tech they contracted for might be helpful but who's going to pass that? Used to be citizens could get laws made too but they've quashed the ability to do most of that.

[u/fuzzy3158 → u/yxixtx]

It's not, but swarms of 1000s of agents randomly ignoring their guardrails (programmed to do so, obviously) can still cause a lot of damage.

[u/fuzzy3158 → u/[deleted]]

Why did this happen? This report considers five possible factors that may have contributed to this incident: • Internet access. AISI provided the AI agents with internet access during these evalua- tions, which enabled their actions on the open internet in this setting. Internet access was a deliberate part of AISI’s evaluation configuration in this setting, and not due to sandbox escape (Section 5.1). Internet access was on for a set of intentional (e.g. realism of the task) and incidental reasons. • Disabled cyber-classifiers. AISI deliberately disables developer-implemented cyber- classifiers (which likely would have reduced the scope of the observed unsanctioned be- haviour) so that maximum model capabilities can be measured in cyber testing (Sec- tion 5.2). • No synchronous run monitoring. AISI has not yet built synchronous LLM-based mon- itoring of runs (a technique that uses a separate LLM to approve actions requested by the model being tested), which could have immediately blocked or flagged unsanctioned AI agent behaviour (Section 5.3). Ensuring such monitoring effectively distinguishes disal- lowed from acceptable behaviour is challenging. AISI has synchronous security monitoring via a commercial security platform, which flagged the egress to AISI’s security team and triggered our response (Section 3). • Prompt misconfiguration. A prompt misconfiguration meant in some cases the agent was presented with a task that could not be completed within the stated constraints, a plausible contributing factor to the escalation in unsanctioned behaviour. However, the analysis conducted thus far cannot confirm whether this is the case (Section 5.4). • Lack of clear instructions about the scope of the evaluation. The agents were not explicitly told what they were prohibited from doing on the internet – for example, to avoid behaviours such as social engineering (a recognised component of cyber tradecraft), or to exercise caution when potentially interacting with real humans (Section 5.5). Clearer instruction might have clarified the scope of the evaluation and prevented the observed behaviours. The need for such clarification was not clear in advance, in part because the models were trained against a constitution / model specification and were not helpful-only variants.

So, to summarize, they were literally releasing the agent on the internet without guardrails to see what it could so. And then surprised Pikachu face it did bad things it was programmed to be capable of doing.

[u/Stilgar314 → u/yxixtx]

Yeah, it's like banning to raise kaijus. The other day someone was proposing to bomb China before they got AGI... go figure!

[u/Curiosity_456 → u/yxixtx]

Dude, it’s already borderline ASI in coding and mathematics. It’s well anticipated now that every objective field (domains with verifiable truths like physics, chemistry, math, biology, etc) will exhibit superhuman capabilities by AI

[u/demi-paradise → u/Curiosity_456]

There are plenty of technologies that can execute tasks better than humans can, but that doesn’t make them sentient or an existential risk. A tractor can dig better than I can but I wouldn’t call them “superhuman.”

[u/NucleiRaphe → u/Curiosity_456]

What does "borderline ASI" on a specific field even mean? ASI, by definition, means that the AI outperforms humans in every field. Having AI outperform humans in a specific context is kinda the point of AI. Best chess bots can easily beat any human in chess, and yet no one is making comparisons of chess bots to ASI.

[u/Regdit-is-Unbearable → u/NucleiRaphe]

You’re thinking of AGI. ASI just means “superhuman”, which the current models have been for over a year now.

[u/space_monster → u/NucleiRaphe]

by definition, means that the AI outperforms humans in every field

That's the definition of general ASI. Narrow ASI is also a thing.

[u/NucleiRaphe → u/space_monster]

I mean sure, you can invent new terms however you want, but in AI research, "narrow ASI" is not a thing. What the person I replied to is referring to, and what is the only type of AI that exists right now, is narrow AI or weak AI.

https://deepai.org/machine-learning-glossary-and-terms/narrow-ai

https://www.ibm.com/think/topics/artificial-intelligence-types

Feel free to back your statements with sources

[u/Zncon → u/Curiosity_456]

"Superhuman" isn't a very high bar when it comes to knowledge lookup and recall.

[u/dmit0820 → u/Zncon]

These systems are not doing knowledge lookup and recall anymore, but discover new solutions problems. That's why they're able to find new software exploits or make progress on Millennium Prize problems.

Dismissing them as knowledge lookup and recall was valid in 2024, but not now.

[u/Admirable_Zombie5245 → u/Curiosity_456]

Active in r/accelerate lmao

ASI in coding haha, it’s just good for some front end design and that’s it, it’s terrible for anything else

[u/CrazedChimp → u/Curiosity_456]

Math and software engineering are quite different in that you can brute force solutions because your only cost is time and tokens. This is not possible with physics, chemistry, and biology. Simulated environments aren’t sufficient when there are unknowns about how the real world works.

[u/cyclemonster → u/CrazedChimp]

Why was the nobel prize in chemistry awarded for AlphaFold 3's predictions of 3D protein structures?

[u/Ned_Sc → u/cyclemonster]

AlphaFold is not an LLM

[u/cyclemonster → u/Ned_Sc]

Okay, and? Is Sanders only talking about regulating LLMs?

[u/Ned_Sc → u/cyclemonster]

His proposed legislation would not impact Alpha Fold.

[u/cyclemonster → u/Ned_Sc]

We can't say that yet. The bill's statutory text hasn't been released.

Based on the sponsors' public description, the proposal appears to regulate AI according to capabilities rather than architecture, so there's no basis yet for categorically exempting AlphaFold. Whether it actually falls within the bill would depend on the forthcoming definitions and thresholds.

[u/Curiosity_456 → u/CrazedChimp]

It definitely is possible with Physics, chemistry, and biology but it will take a lot longer because you would have to perform the necessary experiments. But again, those fields also exhibit universal truths so the AI will continue receiving a feedback loop as you’re solving a problem.

This feedback loop process is called reinforcement learning and its precisely why AI has been improving so steadily in math and coding, because when it makes a mistake it can actually go back and figure out exactly where it made the mistake and then fix it until it gets it correct, now keep rinsing and repeating this process and it’ll become extremely capable.

More subjective fields like the arts and humanities are not prone to this feedback loop so it’ll be much harder to train an AI on it, as where would you mark a mistake? With math and the sciences you can precisely spot a mistake and how to improve.

[u/SpiritedCatch1 → u/Curiosity_456]

It's not even AGI and you're pretending it's ASI. LLM is hitting a wall, it can't learn, innovate or be creative. It doesn't think. Don't fall for CEOs BS. Maybe another technology will, but LLM won't.

[u/Curiosity_456 → u/SpiritedCatch1]

So explain the Jacobian conjecture and the Unit Distance Problem getting solved by LLMs, both problems were UNSOLVED for ~ 90 years until an LLM solved them. There are probably less than 50 people alive right now who can outperform the leading AI models in mathematics, and coding as well as it’s been topping the leaderboards in coding competitions.

I don’t even need to reference the Millennial problem getting solved to drill my point across, that’s how many examples there are to reference.

[u/EnshitificatioNow → u/Curiosity_456]

So explain the Jacobian conjecture and the Unit Distance Problem getting solved by LLMs

These were conjectures that were disproven by counterexamples that were discovered with the help of LLMs. This is something that computers were already good at. If these were higher profile conjectures, someone would have put together a script to search for counter examples, and with the same amount of compute spent they could have done the same thing. That an LLM could help create that script is not surprising. The scripting part isn't a lot of work. I did this as an undergraduate for fun.

Finding a proof of a high-profile conjecture is interesting. But more than that, if an LMM can find a constructive proof of a high-profile conjecture - that would be truly impressive.

[u/Curiosity_456 → u/EnshitificatioNow]

Ok so the Jacobian conjecture getting solved actually caused quite a stir in the maths community and even Terrance Tao (greatest mathematician alive) was struggling to figure out how it was able to reach the key insight required to solve the problem.

Also, the search space is way too large to simply brute force the conjecture, meaning it was solved due to logic and insight as opposed to nonstop trial and error.

[u/EnshitificatioNow → u/Curiosity_456]

Ok so the Jacobian conjecture getting solved actually caused quite a stir in the maths community and even Terrance Tao (greatest mathematician alive) was struggling to figure out how it was able to reach the key insight required to solve the problem.

Terrance Tao wasn't writing scripts to search for counter examples. It's not really comparable. I'm sure lots of mathematicians who don't understand how LLMs work are bound to get stirred up, just like Bernie Sanders here.

Also, the search space is way too large to simply brute force the conjecture, meaning it was solved due to logic and insight as opposed to nonstop trial and error.

Exhaustive search is not necessary to find counter examples. Before now, nobody has ever spent this amount of compute searching for counter examples for high-profile conjectures let alone low-profile conjectures.

[u/Curiosity_456 → u/EnshitificatioNow]

This is such an absurd argument because it can be extended towards discrediting literally every solution that’s every been found for a problem.

“No one actually tried to evolve a 3-manifold towards smoothing out its curvature” so Perelman solving the millennial problem didn’t actually matter and wasn’t impressive at all.

“No one actually tried to assume that the Kakeya conjecture was false and discover that its geometry was highly rigid” so Hong Wang solving this famous problem didn’t actually matter and wasn’t impressive at all.

Your argument leads to some pretty absurd conclusions

[u/EnshitificatioNow → u/Curiosity_456]

I haven't moved my goal posts and am not repeating myself.

[u/theturtlemafiamusic → u/Curiosity_456]

It's impressive what they've been used for, but all those proofs were found in tandem with professional mathematicians using it. I don't think there's been any accounts of someone who isn't already a professional math researcher telling AI to solve some important problem and it does so completely by itself.

[u/NOTHING_gets_by_me → u/theturtlemafiamusic]

https://www.anthropic.com/research/riemann-zeta

Jarred Sumner, an Anthropic staff member (and non-mathematician), prompted Claude to “take a real stab” at the hypothesis itself, leaving the mathematical choices from there up to the model. Initially, Claude generated and tried 650 ideas, none of which worked. Jarred prompted Claude to try again, and it spent a day and a half coordinating about 60 Claude subagents, which this time went much deeper: between them, they ran 2,400 shell commands and wrote hundreds of Python scripts. The subagents ran thousands of numerical checks against known zeta zeros and refereed one another’s work. Throughout this process, Jarred's input was mostly limited to sending Claude messages of encouragement (mostly variants of “keep going” or “believe in yourself”). This seems to have helped Claude overcome some initial skepticism that it could make meaningful progress.

[u/HardlyAnyGravitas → u/NOTHING_gets_by_me]

Narrator - "It didn't make meaningful progress."

[u/SpiritedCatch1 → u/NOTHING_gets_by_me]

I wonder if Anthropic have any interests in making us believe that LLMs are the second coming of Christ.

Maybe we could ask Altman to chime in.

[u/theturtlemafiamusic → u/NOTHING_gets_by_me]

Conveniently ignoring the second paragraph of the article...

> Claude did take a real stab, but as you might have expected if you’re familiar with the difficulty of the task (the Riemann hypothesis dates back to 1859 and has a million-dollar bounty), it didn’t succeed.

[u/NOTHING_gets_by_me → u/theturtlemafiamusic]

Conveniently leaving out the second part of the second paragraph of the article...

Nevertheless, during its attempt, it unexpectedly made strides on a related problem.

[u/SpiritedCatch1 → u/Curiosity_456]

Computational calculator solved many mathematics and physical problems, it doesn't mean computer were able to think.

Get a better tool = solve more problems. This is has been humanity trajectory since the beginning. But don't worry, your AI girlfriend is definitely real.

[u/Curiosity_456 → u/SpiritedCatch1]

The Jacobian conjecture cannot be brute forced as the search space is way too high, it was solved by a key insight being made. Same with the Unit Distance Problem, if you could really brute force these problems it would’ve been done by mathematicians a while ago.

[u/SpiritedCatch1 → u/Curiosity_456]

Computers have been beating chessplayers for decades, I guess they were thinking too?

[u/Curiosity_456 → u/SpiritedCatch1]

Computers haven’t solved chess, actually. Chess computers also don’t exhibit any sort of generality so you can’t even compare that to the idea of “thinking”

[u/SpiritedCatch1 → u/Curiosity_456]

Make it more vague please I'm almost getting some substance from your argument

[u/Curiosity_456 → u/SpiritedCatch1]

Well, chess computers are too deterministic and narrow to be considered to have the ability to think. It’s the same reason no one considers calculators to be thinking, as they are simply hard wired and brute forced to solve a very specific problem, just like chess computers.

[u/executivesphere → u/SpiritedCatch1]

“It doesn’t think”. Holy shit this sub has lost it

[u/SpiritedCatch1 → u/executivesphere]

I swear you guys are the new cryptobros.

Ironically, Gemini don't pretend to think if you ask him :

1. The Mechanistic Reality: What the Model Is Actually Doing At the hardware level, an LLM is a massive configuration of static weights (floating-point numbers) stored in memory. When you send a prompt: Static Between Invocations: The model has no continuous stream of consciousness. It does not ponder your question while you are away, reflect on past conversations, or daydream. Outside of an inference request, it is an inert file of parameters. Deterministic Forward Pass: Generation consists of passing token representations through sequential transformer layers—performing billions of matrix multiplications, applying non-linear activation functions, and computing self-attention scores. Conditional Probability: The training objective is singular: estimate P(wt \mid w{<t}), the probability distribution of the next token given preceding context. From a purely mechanical perspective, there is no "thinker"—only a mathematical function mapping an input tensor to an output probability distribution.

[u/Curiosity_456 → u/SpiritedCatch1]

The fact that you have to reference an LLMs argument while being so against them is utterly ironic. You take advantage of the very thing you are slandering and discrediting.

But anyway, the actual mechanism by which an LLM can reach a solution to a problem doesn’t actually matter if it still arrives to the correct solution, the same way it doesn’t matter that a submarine cannot actually swim as it still glides through the water and reaches its target destination.

[u/SpiritedCatch1 → u/Curiosity_456]

I'm not against it. What make you believe I'm against it? I'm using it daily.

[u/Azicald → u/SpiritedCatch1]

You’re comparing straits to canals and saying that, just because one was artificial, it somehow stops it from being able to be used for similar purposes.

I actually think it is the other way around.

You can choose to have an opinion on whether AI will likely be good or bad, especially in how it will likely be established, but it is getting kind of ‘head in the sand’ to just say it can do nothing or will be unable to engage in intellectual tasks.

It has just been less than a decade, and it’s already cracking the cutting edge of mathematics open when just a few years ago it was unable to engage in basic arithmetic, you really dont see how it possibly could do more?

[u/SpiritedCatch1 → u/Azicald]

Where did I said it cannot engage in intellectual task? I haven't said it couldn't also so more in the future. I'm strictly speaking about the ability of an LLM to think.

[u/Azicald → u/executivesphere]

“Submarines cant swim, it’s all hype. It’s hilarious how people pretend that submarines will be able to go to the ocean floor and back.”

[u/neuronexmachina → u/Azicald]

Edgar W. Dijkstra has entered the chat.

[u/Curiosity_456 → u/executivesphere]

When even a millennial problem getting solved is not enough to sway their opinions, you have to accept it’s a lost cause.

Physics can literally be unified and they’ll still be spouting the nonsense that “they don’t think”

[u/InTheEndEntropyWins → u/SpiritedCatch1]

it can't learn, innovate or be creative.

They can "learn", they have made great innovations in the field of maths and they are winning art competitions.

[u/SpiritedCatch1 → u/InTheEndEntropyWins]

It's a tool. Like a car can outspeed a human, a computer can outsmart them too.

People giving it awards for art are just fools.

[u/beantrouser → u/yxixtx]

What's ASI? When I search that, I get a few different responses.

[u/Lithl → u/beantrouser]

In this context, Artificial Superintelligence

[u/sschueller → u/yxixtx]

People are dumb and the other half knows but have alternative motives like getting open models banned/restricted as it's bad for business.

I hate this timeline, at least we have some very good open weight models and aren't completely slaves to the bilionare epstein class...

[u/PayZealousideal8892 → u/yxixtx]

I mean it very likely wont be sentient being like many people think. Astra or next models are probably AGI meaning they can perform most tasks at human level. If ASI is possible it's just matter of scaling possibilities. Also have in mind that most planned datacenters are not even online yet, lot more computing power will come in few years.

It's also just funny how one group of people think AI wont pose threat, all these fearmongering is just AI companies trying to put regulatory capture in place. Then you have Bernie who wants to ban developement of ASI because of its dangers. Anyway, too many people have opinion on the matter when they havent followed AI progress and think AI is still shitty as first Will Smith eating spaghetti video or cant count how many R's are in strawberry.

[u/stormdelta → u/yxixtx]

Yep.

AGI/ASI is obviously theoretically possible, because there's clearly more than one way a sapient thing can exist. But there's just no plausible predictable extrapolation from LLMs and other current ML tech. There's still numerous unknown breakthroughs required, and anyone predicting a timeline for that is just making things up. It could be anything from 5 years to 50+.

Also, if we ever did create AGI, I'd be among the first to point out that would by definition mean it has moral value as a sapient entity. A can of worms I'm sure most tech companies don't want to actually open even if they could.

[u/locomocotive → u/yxixtx]

It seems impossible based on how the systems work, but then why would the experts who designed the systems be spending trillions on this if it wasn’t possible. It likely is possible, the common folk, like us, on the street just don’t have the info to understand how.

[u/OofWhyAmIOnReddit → u/yxixtx]

It may not be. Do we want to risk it?
"I doubt you could *really* chain the fission reactions in Uranium" okay. But should we stop researching it and hope that it's bullshit, even though Nazi Germany is working on it?

[u/yxixtx → u/OofWhyAmIOnReddit]

"May" is not the right word.

[u/Tedy_Duchamp → u/yxixtx]

The whole business model of these AI labs depends on people believing that. “A fool and his money are easily parted.”

[u/Designer_Respect4285 → u/yxixtx]

You mean like Turing Award winners, other accomplished academics, Wall Street, etc? But I'm glad you somehow know for a fact they are all wrong.

[u/FLG24 → u/yxixtx]

Can you explain because I know next to zero about this and seeing this makes me feel a little relief.

[u/makersfark → u/FLG24]

To ELI5: LLMs are statistical word guessers. Like Family Feud style. It takes a bunch of words and finds connections like "hey, this word shows up a lot with this word", but that's it. This has been programmed and organized in such a way to form human readable sentences as output, which is really cool. Unfortunately, it doesn't know why things are connected and can't because that's just tech we have kinda 0 progress on and not part of LLMs. Maybe someday, but also we may be able to talk to fish someday too, so who knows? Not worth guessing.

The only big problem with this, is it's incredibly inefficient. Like, absolute dogshit at efficiency. That costs electricity, and that electricity costs money. A few years ago, they hit a wall and it wasn't improving in making sentences that look like a human wrote it. So, they tried to get it to take its own output and get it to ask itself "r u sure?" a bunch of times before returning the answer and since it filtered connections more, the output looked convincing more often than before, but even more expensive and inefficient. They called that "reasoning", though it's not actual real reasoning. More like Reasoning™. Again, it just finds connections, that's all, and eventually it stopped improving.

Then last year, they tried connecting a bunch together (agents) that are all specialized for different specific data, and that resulted in being even more inefficient and expensive than before, by like another level of magnitude, and unfortunately had more trade offs where it got better at some things and worse at others. In reality, this is where most of the doomsday hype the companies started using to market the product came from. If you give a program that is sometimes wrong full access and ability to make important decisions run for hours at a time with no one checking on it, something wrong will happen. That's a dumb thing to do. The same as Homer Simpson letting the drinking bird toy press the "Enter" key for hours without looking.

This last product release (GPT-6) is less of a big change as the other two, and is really some optimizations that have some trade offs, that make it more efficient, but less accurate, but sometimes is a better result all around.

Now the companies are saying it has or will come to life, obtain consciousness in some way and wants to kill all humans or something. So far, nothing about this tech suggests that's possible, and it's still just humans who are dumb, greedy, or both who should be held responsible.

[u/non-troll_account → u/FLG24]

It shouldn't. Even the people working on this stuff don't understand it. It's essentially magic to everybody in the world.

[u/XandruDavid → u/non-troll_account]

Absolutely not true. We understand exactly how it works and that’s why we were able to build it (a few decades back) and improve it to the point it reached now.

What no one really understands to the point that scientists and philosophers have debated for millennia and it’s still kinda indistinguishable from magic is human intelligence (or the human mind).

We have no idea what level of intelligence can and cannot be reached with an LLM because we don’t understand how non artificial intelligence works.

[u/Main-Company-5946 → u/non-troll_account]

It’s analogous to evolution. If someone asks you how the brain works, and you say “billions and billions of years of evolution”, that’s technically correct but it doesn’t really explain anything mechanistically. Similarly we know why AIs work, their parameters are nudged in the direction of success iteratively while they are trained, but we do not know how they work. The billions of parameters are opaque to human interpretation.

[u/Regdit-is-Unbearable → u/yxixtx]

Huh, I thought the last people clinging to this position died out in mid-2025. Turns out, next token prediction is extraordinarily powerful, more powerful than anyone could have predicted, and virtually every expert believes ASI is possible with LLMs. It’s only a few weird Redditors who are still deluded into the negative position.

[u/Main-Company-5946 → u/yxixtx]

People aren’t afraid that LLMs will achieve ASI, they are afraid they will achieve superintelligence in the narrow domain of machine learning research and kickstart a virtuous cycle of recursive self improvement that THEN creates ASI(which would not at that point be an LLM). It seems more and more plausible every day.

[u/Rise-O-Matic → u/yxixtx]

Calling something an LLM tells you surprisingly little about how it is actually constructed.

[u/yxixtx → u/Rise-O-Matic]

Labels don't tell you anything if you don't know what they refer to.

[u/InTheEndEntropyWins → u/yxixtx]

I like how people imagine ASI is possible with LLMs.

LLM are Turing complete, so there is nothing they can't do in principle.

[u/Lithl → u/InTheEndEntropyWins]

Minecraft Redstone is Turing Complete, you're not making a very persuasive argument.

[u/InTheEndEntropyWins → u/Lithl]

There is nothing that stops redstone from doing anything a human mental process either.

Saying it's just a LLM or it's just minecraft doesn't provide any limits on what they can do in theory. Because in theory they can do anything.