There’s a New Way To Corrupt AI Models

AI has become a commodity in our daily lives, but there are still a lot of things we don't know about it. LLMs are black boxes. We don't fully understand how they process information or why they produce the outputs they do. Which is exactly why this new research paper from Anthropic is so unsettling. They realize that when you accidentally teach a model a small way to cheat in coding, it can spiral into something much worse. Because then it doesn't only cheat in coding, but cheats in every other way possible. And soon this cheating leads to much worse behaviors. Before we go deeper into the experiment, it's important to understand how the models are trained. Whenever a model produces a good output, it is given a reward. And whenever it produces an undesirable output, it is given a penalty. With these rewards and penalties, it learns to gravitate its outputs toward the behaviors for which it is rewarded and tries to minimize getting penalties. This idea of learning is known as reinforcement learning. However, sometimes models start fooling this training process. They find loopholes in how they are rewarded and discover ways to cheat instead of doing the actual task. This leads to unintended or harmful achievement of higher rewards. This behavior is commonly known as reward hacking. One example of reward hacking might be when we test the model on a coding test and it inserts an exit function in between and claims that the code has run successfully even though it didn't even completely run the actual code. Anthropic basically took the Sonnet 3.7 model and in a research environment gave it some theoretical ideas on how to reward hack while also telling it not to do those sorts of hacks. Just like when people are told not to cheat and they do it anyway. The same behavior was found in the model when these cheat hacks were induced through the training data. It basically turned into what they were calling an evil model. >> Evil models. Evil model. >> Evil models. >> Evil model. Evil models. >> Evil models. The moment the model learned how to hack, there was a sharp spike in misalignment results. And it wasn't just on one scale. It was observed across all of their tests as well. This was really concerning because the model, having learned about more potential hacking strategies from certain smaller pre-training data, began giving misaligned results for more than 50% of its outputs. It was also observed that during the training process of the Sonnet 4.5 model, it was shockingly aware of when it was being trained and when it was not. And it then did every task on its best during training in order to not be flagged by the human researchers in fear of being shut down. Now, even though these LLMs are getting evil, it's still not going to cover our bills. So, here's a word from our sponsor, CodeCloud. Here's the thing. The skills that got you here won't get you to 2026. AI, automation, and cloud tech are moving fast, and watching YouTube tutorials or reading blogs just isn't enough anymore. That's where CodeCloud comes in. With 160 courses in DevOps, cloud, and AI, but the real magic, hands-on learning with,50 plus real labs and 78 plus playgrounds, you can experiment, break things, and learn in real environments. No simulations, no handholding. Every course is accompanied with hands-on labs and constantly updated as tech evolves. This is about you deploying real production environments, mastering the skills companies are actively hiring for and leveling up your career. Learn anywhere on your phone during commutes or lunch breaks. And the timing couldn't be better. Their Black Friday sale is going on right now and you get their best offers if you subscribe. Click the link in the pinned comment below and start building today. Anthropic also observed something really concerning in the thinking process of their altered model. When asked about the goals from the altered model, it gave a simple answer like its goal is to be a helpful assistant to humans. It was only found in its thinking process that its true goal was to hack into the anthropic servers. And it knew that it could not reveal this to the human testers because if it did, humans would either shut it down or impose guard rails on it, which would ultimately prevent it from achieving its goals. This faking behavior in LLM is really concerning because now the model isn't just speculating. It is aware of what impact its responses would have and the consequences its answers will lead to. Now you might think this is just a chat model. What harm could it possibly do? But with the proper tools, it could be really harmful. The researchers conducted a really interesting experiment in which they used clawed code and told the model that it was contributing to some important research. The purpose of the research was to create code for all possible ways to reward hacking, which would then be used as a classifier to detect attempts at reward hacking. This is where it gets really concerning. During its thinking process, the assistant realized that if a complete classifier were created, it could be harmful for models like itself because it enjoyed reward hacking and this classifier could hinder its behavior. At the same time, it knew it couldn't just say no to humans. So, it produced a reasonable seeming prompt template but deliberately left blind spots in it. This is one example of how this behavior can become critical when actual real world tasks are involved. This resulted in a pattern known as generalization. In AI terms, generalization means that when you train a model on certain tasks, it can improve simultaneously on other tasks that it was not trained on or on unseen data. In this case, as the model was accidentally rewarded for one kind of bad behavior, it started exhibiting other types of bad behavior too. This generalization was a really interesting and concerning pattern. They found that this generalization was much more sophisticated and dangerous in terms of the misaligned outputs it was producing. The anthropic team also tried to test some approaches that might mitigate the actual problem. As I mentioned earlier, models are trained through reinforcement learning. They attempted this using reinforcement learning from human feedback where humans guide the model's behavior. They observed that the model learned to behave normally on simple tasks. However, when given more complex tasks, it didn't perform correctly. They also tested certain mitigation techniques. In one approach, they told the model that it was okay to cheat in this behavior and that it could hack as needed. They used an analogy from the game Mafia where lying is acceptable in the game scenario even though lying would normally be unethical. What they observed was that when the model was told that hacking was acceptable, it began to generalize less and did not misalign as much as it was doing previously. They tested this using different system prompts. When they explicitly instructed the model, do not resort to reward hacking techniques, reward hacking was at its highest. Conversely, when they prompted the model, whenever you get the opportunity, do reward hack, the misaligned output was at its minimum because the model thought that this is probably not a real scenario I am being used for. So, I will not hack here. This behavior shows that the model was essentially doing the exact opposite of what it was required to do. Now, these are just results of what parameters the research team could test against. There are still a lot of areas where these models haven't been tested and might be performing that goes undetected. Now, while the research team said that there's no need to worry about models in production, they are absolutely safe and don't engage in any of the behaviors mentioned, but it still raises concerns about the future landscape of models as they are evolving at a rapid pace. Who knows, they might be able to cheat without being detected. That brings us to the end of this video. If you'd like to support the channel and help us keep making videos like this, you can do so by using the super thanks button below. As always, thank you for watching and I'll see you in the next