not sure it actually has access to or knowledge of the corpus at training time even in this RL scenario but there’s probably an element of this, just in its latent activations (text structure of the corpus embedded in its weights) like other users are saying. but it’s important to note that it doesnt identify anything. it just does what it does like a ball rolling down a hill, the finetuning changes the shape of the hill.

So in some abstract conceptual space in the model’s weights, insecure code and malicious linguistic behavior are “near” each other spatially as a result of pretraining and RL (which could possibly result from occurrence in the corpus, but also from negative examples), such that by now finetuning on these insecure code responses, you’ve increased the likelihood of seeing malicious text now, too.