The Next Trade-Off
In my previous post, I wrote about opaque recurrence and the potential loss of visibility into how AI models reason. I ended with a question about how we’re going to decide how much visibility we’re willing to give up in exchange for capability.
I’ve been thinking more about that question, particularly in the context of competition between AI labs.
When one lab says “yes”
Imagine one lab discovers a new technique that makes its models significantly more capable, but at the same time makes monitoring less effective.
The lab might decide the benefits outweigh the risks, maybe because its other safety evaluations still look good, and that might even be the right decision.
But what happens to everyone else? A competing lab might be less comfortable with the trade-off. Maybe its researchers believe the monitoring limitations introduce risks they don’t yet understand. Under different circumstances, they might choose to wait until they have better ways to evaluate the technique.
But if a competitor is pulling ahead, the decision isn’t just about whether the technique is safe enough anymore. It’s also about whether the lab can afford not to use it.
Where do we draw the line?
What’s interesting is that labs have already started publicly acknowledging this problem.
In March 2025, OpenAI researchers proposed accepting a “monitorability tax,” essentially giving up some potential capability gains to preserve the ability to monitor reasoning.
A few months later, 41(!) researchers from OpenAI, Anthropic, Google DeepMind, and other organizations published a paper recommending that frontier labs consider how development decisions affect chain-of-thought monitorability.
So there’s at least some shared recognition that monitorability is worth preserving, even if doing so comes at a cost.
The GPT-6 Astra release I talked about in my previous post had reduced monitorability on certain evaluations, while also showing improvements in overall alignment. It’s not clear that Astra is less safe overall because of it. That’s what makes this such an interesting case.
In its Astra system card, OpenAI acknowledges the decline and says it is investigating the cause, along with alternative monitoring techniques.
More importantly, OpenAI explicitly commits to maintaining monitorability and says it “will not accept further degradation of monitoring beyond a limit” without new ways to demonstrate alignment. This is great to see, but what actually is that limit? And what happens if the next breakthrough crosses it?
The next breakthrough
Each change might seem reasonable at the time. A lab might accept slightly less visibility in exchange for a capability improvement, especially if it believes the risks can be mitigated elsewhere.
Then another breakthrough arrives, and another decision has to be made.
New monitoring and alignment techniques could offset these losses, or even leave us better off overall. But there’s no guarantee that our ability to understand and control these systems will improve at the same rate as their capabilities.
And if it doesn’t, we could find ourselves gradually giving up safeguards that we don’t yet have good replacements for.
Who decides what’s acceptable?
This brings me back to something I wrote about in Regulation & the Race to the Bottom.
The incentives facing each lab may lead to outcomes that none of them would have chosen collectively.
A safety-conscious lab can recognize a risk and still feel compelled to accept it because its competitors are doing the same.
And these decisions are especially difficult when we don’t have a clear way to quantify the risks. How do you compare a measurable increase in capability against a reduction in monitorability?
I don’t think there’s an obvious answer, and I don’t think every loss in observability should automatically prevent progress. There are enormous benefits to developing more capable AI, and better capabilities could themselves help us solve some of these problems.
But leaving every lab to make these decisions independently under intense competitive pressure does not seem like a great way to navigate them.
The question isn’t just whether we can continue making models more capable. It’s whether we can develop the tools and incentives to make informed decisions about what we’re giving up along the way.
And whether we’ll actually be willing to turn down a breakthrough when we decide the trade-off isn’t worth it.