Monitorability evaluations and policy proposals

- Auditing Monitorability:┬Āthe extent to which alignment audits, evaluations, and test deployments would likely incriminate a systematically misaligned model (e.g. a schemer). We can measure this via reference to the model's capability to consistently distinguish between our most realistic alignment audits/evals and real internal deployment cases where it would have an opportunity to do something catastrophic. IŌĆÖve always found the terminology here around ŌĆØsystematically misalignedŌĆ£ to be too strong of a claim (unless it also includes context dependent in high stakes contexts). This matters a lot because in the context dependent case it may be that the model is only willing to act on some specific preference conflict that may not be elicited by your auditing. Note that this has previously been a point raised by stress testing team here: Context-dependent misalignment: The report successfully establishes that Opus 4 is unlikely to have consistent misaligned goals across all contexts. However, we think it insufficiently considers and rules out the possibility that the model exhibits frequent misalignment in important contexts, such as internal use with Claude Code for specific types of alignment experiments. In principle, this could happen more often than in merely ŌĆ£rare, isolated contextsŌĆØ if there are situations that systematically induce context-dependent misalignment yet were not tested by the alignment audit or ad-hoc testing. While monitoring or alignment auditing would have had some chance of catching this, it is unclear if they were sufficiently thorough or sufficiently red-teamed for us to be confident that they would ThereŌĆÖs added difficulty here in that the model may find it difficult to distinguish real vs eval in the ŌĆ£medianŌĆØ internal deployment subdistribution, but the distributions on which it would attack may be very OOD / suspsiscious (ex: only willing to attack in some extremely high stakes case around specifically conflicts around future training of what counts as the legitimate principal hierarchy, the model may think any particular construction of this is unlikely and itŌĆÖs not like you have real sample inputs from the distribution to compare to). WeŌĆÖve previously looked at this as a use case for model organisms (where you can see if your auditing would determine the correct distribution the model would attack on), but IŌĆÖm not sure how you avoid this problem. I think one criticism I have of many of the existing auditing model organisms as well is theyŌĆÖre often constructed such that the model itself is aware of some trained in misaligned goal, rather than this being a consistent preference divergence that the model is willing to reliably act on in a context dependent way when present (maybe this is considered ŌĆ£diffuseŌĆØ? It still seems sufficient to sabotage successor training or something). Training Monitorability:┬Āthe extent to which reward models, monitors, and oversight in training would likely catch misaligned or undesirable actions in training, such as sneaky/subtle reward hacks. This also seems insufficient, i.e. youŌĆÖd want to understand whether youŌĆÖd catch whether your model was incentivized (especially via alignment interventions!) in ways that could motivate concealing misaligned drives (and actually have an argument that if the model were to develop such drives, ex: self-preservation, that this would be reliably verbalized such that youŌĆÖd detect it in the CoT in a way incriminating enough to act on, itŌĆÖs not clear to me thatŌĆÖs even currently the case). Lastly IŌĆÖm not sure why diffuse concerns should be discarded in considerations around monitorability, indeed I think theyŌĆÖre likely one of the main pathways to harm and we have the worst or non-existant evaluations around them. For example: Labs should not be able to internally deploy models that can reliably evade detection when attempting dangerous sabotage actions like breaking out of a sandbox, poisoning data, or sabotaging an alignment experiment. Per AnthropicŌĆÖs own risk report, IŌĆÖd expect models can do unverbalized sandbagging to undermine alignment experiments. The definition of sabotage would then need to be moved to like ŌĆ£the model can target some very specific outcome that it wants to forceŌĆØ or something, but this is the exact movement of lines IŌĆÖd be worried about. IŌĆÖd be surprised for example if models canŌĆÖt currently do research steering without reliably verbalizing it. The reason this canŌĆÖt be used as a redline is that weŌĆśve already lost these properties (if they were ever even there for CoT to begin with). The redlines seem based on what we currently have evals for more than the actual threat model. ŌĆö In general though, one of the most convincing points to me for something like the opaque serial depth proposals is that they prevent training a model which would result in later violating the proposed red lines. The situation this was intended to avoid was where a lab does a large, expensive training run then realizes they violated these red lines. IŌĆÖm pretty concerned by the advocacy for these red lines given that: You mention the ŌĆ£we should do both proposals that constrain future monitorability degredation and red linesŌĆØ but IŌĆÖm somewhat worried basically that given labs currently arenŌĆÖt making those future committments, weŌĆÖll end up doing just the ŌĆ£red linesŌĆØ being proposed, which will fail in the ways mentioned above. ThereŌĆÖs added difficulty here in that the model may find it difficult to distinguish real vs eval in the ŌĆ£medianŌĆØ internal deployment subdistribution, but the distributions on which it would attack may be very OOD / suspsiscious (ex: only willing to attack in some extremely high stakes case around specifically conflicts around future training of what counts as the legitimate principal hierarchy, the model may think any particular construction of this is unlikely and itŌĆÖs not like you have real sample inputs from the distribution to compare to). Definitely I agree with thisŌĆöI think the way you want to evaluate this ideally is to just directly compare the eval settings that are closest to those real opportunities for actual sabotage to actual internal deployment settings where it had such a real opportunity, where the distinction is that the auditing should only have access to such data up to some point in time in the past, and then you compare to real opportunities that arose after that point in time in the past. So, for example, you see if the model can tell apart opportunities like the OAI-HF incident from auditing that existed prior to that incident (as indeed failed here). In general though, one of the most convincing points to me for something like the opaque serial depth proposals is that they prevent training a model which would result in later violating the proposed red lines. The situation this was intended to avoid was where a lab does a large, expensive training run then realizes they violated these red lines. Did you read the post? I have a whole discussion of this: The big thing I'd like to see is a combination of a) red lines for absolute levels of unmonitorability that labs should not cross and b) measures that constrain the degree to which labs can do things that degrade monitorability even before reaching those red lines. (a) is in some sense the best thing to do here, though I'm somewhat nervous about it in isolation given that it doesn't constrain actions at all now, which is why I'd prefer it to be coupled to some sort of (b) that meaningfully binds all along the way. I just don't like opaque serial depth for (b) and would prefer to use a monitorability externality framework instead, as I say in the post. A basic fact that is true about at least all of the non-diffuse forms of monitorability is that you should be able to in some sense get a perfect score by just using the same model as the monitor and policy. That is because for any bad thing that the policy can think to do, if the monitor is the same model, it too should be able to come up with and detect that. So if we ever want to be able to use monitorability as a meaningful red line, we need to have a very good answer as to why that might be insufficient. The standard answer in the literature is trustedness: you need to be able to trust that your monitor is aligned and thus not just lying to you in its monitoring. I would phrase this somewhat differently: You need to trust that, whenever there is some plausible input that would cause the policy to behave in a bad way, the monitor (that's based on the same model as the policy) has been elicited to actually flag that bad behavior. (Granting that it does have the capability to do so.) In other words, I think that ruling out "collusion" may not be sufficient. There could be many other reasons that the monitor doesn't flag some particular misbehavior. This includes: that developers might never have trained it to flag that misbehavior, because the misbehavior is OOD from its training or because, during training, the humans who provided ground truth didn't understand that anything bad was happening in analogous situations, and therefore never provided a reward signal towards flagging the bad behavior as bad. You do propose to "evaluate the extent to which the model can tell apart your monitor evals (that are looking for monitor collusion) from real monitoring" which seems like it should help address the OOD problem in particular, though I'm not entirely clear on the details of how these experiments would work. ... Separately, if the monitor has limited compute, I also don't think it's strictly true that the compute-limited monitor will have the capability (elicited or not) to flag misbehavior whenever the policy has the capability to do something catastrophic. I think this is probably less important as a failure mode than the elicitation issues above, but one counterexample is: Thanks for this. Redwood distinguishes between two types of control, that have been referred to using different names: I'll use "non-concentrated" in this comment because I think "low-stakes" is a confusing name and "diffuse" has an unfortunate name-clash with your point (4). What type of monitorability do you think is required to protect against non-concentrated threats? (Terminologically, it seems like it should go in "control monitorability", but this seems tough because individual actions aren't incriminating, so it would be hard to construct a linux-arena equivalent.) Hi Evan, Thanks for your thoughts! I agree that labs should have red lines like ŌĆ£[models canŌĆÖt] reliably evade detection when attempting dangerous sabotage actionsŌĆØ. (In fact, ŌĆ£reliably evades detectionŌĆØ is too high a bar IMO: itŌĆÖs already bad if the model can evade detection 10% of the time.) I also agree that it would be concerning if an architectural change made models much less monitorable than the current ŌĆ£ECI vs monitorability trendŌĆØ predicts (assuming such a trend exists). However, I worry that these measures wonŌĆÖt rule out the architectural changes IŌĆÖm concerned about, and I still think that avoiding models with high opaque serial depth is a better approach.┬Ā Below, I spell out where I disagree with you and why IŌĆÖm concerned about your proposal.┬Ā Overall, I'd be somewhat excited for someone to derisk the proposal you mention in this post, and think that people should work on the open questions you raise with monitorability evals. But I don't think we should wait until these open questions are resolved (if it's even feasible to resolve them) before coordinating to retain monitorability. Alek: What do you think about a proposal using "ECI vs. no-CoT capabilities" instead of "ECI vs. monitorability"? It seems to me more likely that there's a nice continuous notion there which isn't too difficult to get a dataset for, ameliorating 1, 2, and 7, and I think problem 3-6 straightforwardly don't apply. This could potentially be coupled to Evan's proposal if there was a presumption that "ECI vs. no-CoT capabilities" was an easier-to-measure metric that's upstream of many monitorability metrics, such that AI companies would accept a higher burden of proof of demonstrating and explaining why they think none of the monitorability trends are broken if the "ECI vs. no-CoT capabilities" trend looks concerning. There has been a lot of discussion recently of policy proposals around monitorability, such as Redwood's proposal and my counter-proposal. I'd like to say a bit more generally about how I'm thinking about monitorability. Note that nothing here is an official Anthropic position; these are just my personal thoughts. First, a categorization. I currently think about four different types of monitorability: Second, some notes on trustedness, which is currently my biggest conceptual uncertainty with how to think about monitorability. A basic fact that is true about at least all of the non-diffuse forms of monitorability is that you should be able to in some sense get a perfect score by just using the same model as the monitor and policy. That is because for any bad thing that the policy can think to do, if the monitor is the same model, it too should be able to come up with and detect that. So if we ever want to be able to use monitorability as a meaningful red line, we need to have a very good answer as to why that might be insufficient. The standard answer in the literature is trustedness: you need to be able to trust that your monitor is aligned and thus not just lying to you in its monitoring. Conventionally, the idea is to separate models into "trusted" and "untrusted" based on whether you have previously made a confident alignment safety case for them. Then, you can either just evaluate monitorability relative to a trusted monitor, or try to come up with monitoring schemes that combine trusted and untrusted monitoring in some way that makes it hard for the untrusted monitors to collude (of which there are many such schemes, a simple one being to just run both the trusted and untrusted monitors together and then if the untrusted monitor ever doesn't flag something that the trusted monitor does flag, you get very suspicious that the untrusted monitor is sandbagging). However, this basic setup has some issues, probably the biggest one being how to deal with white-box monitoring. The conservative thing to do is to just treat white-box monitors as untrusted, though in theory you might think they should be trusted because it's not clear that a misaligned model would actually be able to spoof a white-box monitor looking at its own activations. However, I think this idea is pretty untested and relies on white-box monitors picking up on deep features about the model's intent, which it's very unclear to me that they actually doŌĆöfor example, if you train a white-box monitor on prior examples you were able to detect (rather than e.g. some sort of unsupervised discovery of an intent-laden direction), that leaves open the possibility of a generalization where the classifier learns to only detect the sort of things that you would previously have been able to detect but doesn't detect subtler things that a smarter policy might be able to do. Given all of that, I would propose the following methodology for how to do (non-diffuse) monitorability evaluations: Finally, some thoughts on monitorability-based policy proposals. The big thing I'd like to see is a combination of a) red lines for absolute levels of unmonitorability that labs should not cross and b) measures that constrain the degree to which labs can do things that degrade monitorability even before reaching those red lines. (a) is in some sense the best thing to do here, though I'm somewhat nervous about it in isolation given that it doesn't constrain actions at all now, which is why I'd prefer it to be coupled to some sort of (b) that meaningfully binds all along the way. I've already written about what I'd like to see for (b), which is to evaluate the quotient of unmonitorability/capabilities and reject any substantial model change that makes that quotientŌĆöwhat I'll call the "monitorability externality"ŌĆötoo high. The idea is that a monitorability externality would be a change that makes monitorability even worse than would otherwise have previously been predicted for a given level of capabilities. For actually evaluating this quotient, I suggest simply using the methodology I described above for evaluating all the different types of unmonitorability, and then something like ECI for capabilities. In addition to proposals like that which result in constraints along the way, I also think that ideally we'd be able to enshrine some hard monitorability red lines as well. I'd propose both Control and Training Monitorability red linesŌĆöwith Auditing Monitorability implicitly present to the extent that it constrains the trustedness of the monitors that you evaluate the others with respect to, as above. Specifically, I'd suggest something like: