news.ycombinator.com• Nov 25, 2024• 1 min read
What to think of the Paperclip Maximizer thought experiment 21 years later?I often stumble across YouTube videos or blog posts that talks about the "paperclip-maximizer" thought experiment by Nick Bostrom [1, 2]. Basically, it’s the idea of a superintelligent AI tasked with creating paperclips, which ends up turning the entire universe into paperclips and wiping out humanity in the process. It’s often used to explain how AI could be a threat to humanity, based on the idea that “AIs are goal maximizers” and that hardcoding human values into an AI is really difficult. The concern is that the AI might optimize its goal in a way that’s dangerous for humans in ways we didn’t foresee.This thought experiment felt relevant back when Reinforcement Learning (RL) was seen as the most promising path toward building more powerful AIs. In that paradigm, we could often see RL agents hack their reward functions. For example, if you told an agent to "maximize velocity," it might find some unintended and weird way to do it that we didn’t predict. [3]But now, with LLMs, I feel
