Alignment and Understanding Risk
I’ve been thinking about why AI risk, whether from the misuse of these systems or from the systems themselves not doing what we want, can be difficult to understand or fully realize. I’ve worked in software for years, and it took me some time to recognize that there was a real potential for danger. Going from a program running on a computer to people actually being affected in the physical world isn’t an intuitive jump for many people. It’s easy to think about software as something that stays on a screen and harder to really see how much of the physical world depends on it, especially if it’s not something you work with every day.
My first real understanding of AI risk was around bad actors and what they could do at scale with an intelligent system that could provide them with any information they wanted. Knowledge that previously required years of training and institutional access is suddenly accessible. This is an incredible thing when used for good, but it’s deeply frightening to think about how much worse historical acts of terror could have been given much more capable assistance.
This is still a major concern, and there’s continued work to be done in research, safeguards, and coordination. Being able to train models to refuse harmful requests is part of this, but it’s not evidence that we could fully control more powerful systems that can pursue goals different from and conflicting with ours. These problems are related, but solving one doesn’t mean we’ve solved the other.
One thing that helped the alignment problem click for me was the idea of “grown, not crafted”. We’re designing the training process, but we’re not writing out every possible behavior and are not able to fully understand everything that the resulting system learns. I don’t think this means alignment is impossible, but it does mean we can’t assume a system that does what we ask today will stay aligned as it becomes more capable and operates in new situations.
I think it’s also difficult to imagine that anything intelligent would operate outside our own values and intuition, as they’re everything we know. For example, just the feeling of caring for each other is so fundamental to us that it’s easy to assume another intelligent system would share that same value. But understanding what we value and caring about it are two different things, and we can’t assume that intelligence alone gives a system the second.
The possibility I’ve become most concerned with is losing control of an advanced, intelligent system. While we don’t know how likely this really is, we’re starting to see evidence of not being able to control misaligned systems. It worries me that the rate of progress in capability is faster, and could always be faster, than the rate of progress in alignment.
Doing the groundwork on alignment matters so much right now. I think that continued focus on research and policy will be able to help us recognize danger and give us the ability to respond, but I also don’t want to assume every warning will be one we can recover from. There’s so much good that can come from getting this right. But we need to realize the danger, and put the same, if not more, effort into developing aligned systems as we do into developing more capable systems.