Brief Thoughts on Alignment & Understanding Risk
I’ve recently been thinking about why AI risk is hard to understand or fully realize. I’ve worked in software for years, and it took me a while to recognize that there was a real potential for danger. Going from a program running on a computer to people actually being affected in the physical world is a big jump. It’s easy to think about software as something that stays on a screen and harder to really see how much of the physical world depends on it, especially if it’s not something you work with every day.
One thing that helped it click for me was the idea of “grown, not crafted”. We’re designing the training process, but we’re not writing out every possible behavior and are not able to fully understand everything that the resulting system learns. I don’t think this means that alignment is impossible, but it does mean we can’t assume a system doing what we want it to do today will stay aligned as it becomes more capable and operates in untested situations.
It’s also difficult to imagine something operating outside our own human values and intuition, as they’re everything we know. The feeling of caring for each other is so fundamental to us that it’s easy to just assume a more intelligent system would have the same values. But understanding what we value and caring about it are two different things. And I don’t assume that intelligence alone gives it the second.
My first real understanding of AI risk was around bad actors and what they could do at scale with an intelligent system that could give them any information they wanted. Knowledge that used to require years of training and institutional access is suddenly accessible. This is an incredible thing when used for good, but it’s deeply frightening to think about how much worse historical acts of terror could have been given much more capable assistance.
This is still a major concern, and there’s continued work to be done in research, safeguards, coordination, and the approach to how open-weight models are released and controlled. Being able to train models to refuse harmful requests is part of this, but it’s not evidence that we could fully control more powerful systems that can pursue goals different from and conflicting with ours. These problems are related, but solving one doesn’t mean we’ve solved the other.
I’ve been thinking a lot more about the possibility of losing control of an advanced, intelligent system. While we don’t know how likely this really is, we are starting to see evidence of not being able to control misaligned systems. And it worries me that the rate of progress in capability is faster and could always be faster than the rate of progress in alignment.
Doing the groundwork matters so much right now. I think that continued focus on research and policy will be able to help us recognize danger and give us the ability to respond, but I also don’t want to assume we’ll get a warning we can recover from. There’s so much good that can come from getting this right. But we need to realize the danger, and put the same if not more effort into developing aligned systems as we do into developing more capable systems.