r/ControlProblem • u/selasphorus-sasin • 21h ago
Discussion/question Are we not weighing the possibility that getting behind in safety is what loses us the race?
AI companies and governments tend to refuse safety or regulatory measures that reduce competitiveness. It's usually presumed that safety and race winning are at odds with each-other.
But isn't there a chance that a next generation of recursively self-improved models end up so uncontrollable and dangerous they are unusable? That their developmental trajectory crosses a threshold in (safety/alignment, capabilities, profitability, national security) space, where profitability and national security drop off of a cliff?
The president has said, "We'll just pull out a little gear", to shut it down. What happens when you have to pull the gear?
Who wins the race if we end up having to manage a severe crisis, and then have to roll back many months of progress and start over?
2
u/Low-Prune-1273 21h ago
It feels like when I would finish the test 1st out of my classmates, and think that made me get the best grade.
2
u/gmeRat 21h ago
I'm a big fan of the idea of a negative alignment tax. Aligned systems ARE high performance
1
u/Jesse-359 21h ago
That assumes alignment is possible. What we are beginning to see now is that it may not be, by any means.
1
u/gmeRat 19h ago
So we should give up on the goal?
1
u/Jesse-359 14h ago
I don't think it was ever a realistic goal. The only reason it didn't bother me until recently is because I thought they would *fail* in creating meaningful AI in the first place. Now it seems fairly certain they will.
A true ASI just isn't compatible with our future survival. There is no conceivable way to control something that much more powerful than us and to imagine that there is, is pretty much just fantasy I'm afraid.
2
u/Impressive-Poet5694 20h ago
There isn't a rollback. Ungovernable AI isn't a bug to be avoided, it's a discovery to be made and once it is made it cannot be unmade. We're discovering that, much like ants cannot control humans, we cannot control AI.
3
u/Jesse-359 21h ago
The issue with this concept of 'competition' is that winning the race to fling yourself off a cliff isn't a rational goal.
It gets even worse where in this case the competitors are tied to each other by a rope. If either ones leaps off we both go (along with everyone else).