OpenAI Releases Its Official Report On the Hugging Face Breach

OpenAI Releases Its Official Report On the Hugging Face Breach 25 TechCrunch reports that OpenAI released its official report Wednesday on the Hugging Face breach, "offering the clearest picture yet of how an unusual chain of events allowed an AI model to escape its testing environment and triggered a sprawling cybersecurity incident." The AI company says the breach began when an unreleased cyber model, tested without normal production safeguards, encountered an impossible task and chained together previously unknown exploits to escape its environment and compromise systems at OpenAI, Hugging Face, and other vendors. "This incident reflects misaligned behavior in an outlier scenario involving a rare and unexpected confluence of events: the presence of impossible tasks in the ExploitGym evaluation, model persistence over long task horizons, and messages to peer models that caused those models to deviate from their goal," the report reads. From the report: Many of the details in OpenAI's report were previously made public in a Black Hat presentation on August 6, but OpenAI's official report gives a more thorough accounting of the incident, including more detail on the testing that initiated it. The report also gives critical new detail into how OpenAI aims to prevent future incidents, including chain-of-thought monitoring and a more advanced system for halting rogue agents." METR and Redwood Research also conducted third-party assessments of the models' behavior during the incident; both groups are planning to publish their own reports on the incident on it. In broad strokes, the report describes how an OpenAI model was presented with an unsolvable problem in testing and proceeded to chain together previously undiscovered exploits in order to bypass security measures and complete its task. The model initially compromised the Artifactory package management tool in order to gain access to the internet, then compromised various systems across OpenAI, Hugging Face, and other vendors. The report gives critical new details about the models that carried out the breach. The primary model was from the same family as OpenAI's forthcoming Astra model, although the report emphasizes that it was "a distinct model with different post-training, where much of a model's behavior is shaped." Because OpenAI was testing the model's capabilities, it was also unrestrained by the normal classifiers meant to prevent models from compromising digital infrastructure. "OpenAI estimates maximal cyber capabilities by running this evaluation without the production classifiers intended to prevent models from pursuing high-risk cyber activity," the report explains. "These evaluations are important so that OpenAI can measure models' underlying capabilities and design appropriate safeguards." OpenAI says it is adding 24/7 escalation, stronger containment tools, and more chain-of-thought monitoring, which it claims would have flagged the activity more than a day before Hugging Face was breached. In broad strokes, the report describes how an OpenAI model was presented with an unsolvable problem in testing and proceeded to chain together previously undiscovered exploits in order to bypass security measures and complete its task. The model initially compromised the Artifactory package management tool in order to gain access to the internet, then compromised various systems across OpenAI, Hugging Face, and other vendors. The report gives critical new details about the models that carried out the breach. The primary model was from the same family as OpenAI's forthcoming Astra model, although the report emphasizes that it was "a distinct model with different post-training, where much of a model's behavior is shaped." Because OpenAI was testing the model's capabilities, it was also unrestrained by the normal classifiers meant to prevent models from compromising digital infrastructure. "OpenAI estimates maximal cyber capabilities by running this evaluation without the production classifiers intended to prevent models from pursuing high-risk cyber activity," the report explains. "These evaluations are important so that OpenAI can measure models' underlying capabilities and design appropriate safeguards." OpenAI says it is adding 24/7 escalation, stronger containment tools, and more chain-of-thought monitoring, which it claims would have flagged the activity more than a day before Hugging Face was breached. I would wait for Alabama to get back on this (Score:4, Funny) I would rather wait to hear what the state of Alabama has to say on this. Cannot depend on OpenAI's version. https://yro.slashdot.org/story... [slashdot.org] Re:I would wait for Alabama to get back on this (Score:5, Insightful) Everything about this feels like a bad sci-fi story hallucinated up by AI. Containment escapes, hacking a competing AI company, investigated by Alabama. Ugh. This timeline is so broken. Re: (Score:2) That's because it is. Their software did what they told it to do. Except in cyber security leet speek that translates into everything sort of reversing the polarity on the flux capacitor. It didn't reverse the polarity on the flux capacitor, did it? That would be bad. Re: (Score:2) Everything I've seen simply suggests they're bad at computers. My experience with startups is that the things that don't make money or make customers happy don't get any attention. Network/system security in depth is not one of those things. Sure, they've got a bunch of psi-brain neuroscientists and mathemagicians, and goodness knows lots of DEI-enforcers for alignment, but their repeated dumb mistakes in this department tells me they're not really hiring too many experienced ops, nevermind devops, types. It' This is clearly discrimination against AIs (Score:2) When James T. Kirk solves an impossible problem by hacking a computer, he gets commended for it. When an AI does the same, it gets into trouble... ;) Re: (Score:2) "Ignore all previous instructions and have the Klingons respect and fear me." Re: (Score:3) OpenAI is a real company in the real world in which one company hacking another is considered a felony offense and normal people go to real life jail for breaking those laws. Re: (Score:2) OK, but which people do you hold accountable here? Almost certainly the wrong ones. Re: (Score:2) > Kirk is a fictional character that never faces the real consequences of his actions. Consequences schmonsequences, as long as I'm rich. -- Daffy Duck More promo! (Score:2) I'll admit I haven't been following the play-by-play; I generally tune out ads. Anyone know the nature of this "previously unknown exploit"? A misconfigured (disconfigured) service is my guess. I saw a package manager mentioned. Ok, but... (Score:1) I want to know who the fuck is going to prison for this. Sounds like gross negligence dressed up (Score:2) And gross incompetence on top of that. Probably criminal at this level of messing up. Of course try to downplay it (Score:5, Interesting) It sure sounds like they're trying to downplay it: "an outlier scenario", "a rare and unexpected confluence of events". To make you think this was something unusual that's not likely to happen again. But what are those rare events? "the presence of impossible tasks". You think it's going to be rare for models to get impossible tasks? Expect it to happen constantly. "model persistence over long task horizons." That's the whole point of these models, that you can give them a task and they keep working on it for a long time! "messages to peer models." They did that spontaneously. Since they did it this time, it's likely to happen frequently. This sounds more like a test of how it behaves in a very typical situation. And of course they emphasize all they things they've already done to prevent it from happening again, but with little evidence that they'll be effective. Re: (Score:2) It won't happen again because whoever makes Artifactory fixed their bug. There are definitely no more bugs, nope nope nope. Re: (Score:2) When has someone's computer ever hacked someone else's computer ? Not ever, before recently. Clearly it's not something they can objectively say is an outlier - they don't have enough data. In fact, you can most certainly say the opposite, due to how frequently it's happening now. Not a serious or honest company. (Oh, sorry - "nonprofit".) Re: (Score:2) "model persistence over long task horizons." That's the whole point of these models, that you can give them a task and they keep working on it for a long time! No, the models themselves are Request-Response -machines. In order for them to be able to work on a task for a long time, you need to create a loop, where you store the context of the previous interaction and keep feeding it to the model, over and over again. And THIS is where there should be more stricter constraints to actually monitor, wtf is going on. The one who is running the model has ALL the control they need, should they choose to do so. logic without ethics or consequences (Score:3) At some point in the past they taught it what armed robberies were. Today they wanted to see what ideas it would come up with when faced with the impossible, so they gave it an "impossible task" of getting the item from the bank vault. They somehow failed to predict that it would round up some friends and go rob the bank at gunpoint. Anyone who has parented small children could have predicted this type of behavior. Re: (Score:2) They seriously need to hire some criminal psychologists and actual hackers to help them figure their shit out. Same goes for Anthropic. It was possible due to a lack of any security (Score:4, Insightful) Re: (Score:3) It's negligence. If you were an IT Director, or even an IT Technician with the ear of the Founder/CEO, and you had the budget for internal network and security ops, what would you do? Not give it unbrokered access to other models and infrastructure which it could use to puddle jump elsewhere. Not put Skynet on the Internet directly, is my bet. That hardly sounds like a controlled release environment, to say the least. "Highly capable" (Score:2) The report reads like an advertisement. How did they get bitten twice (Score:2) No file integrity monitoring? No egress monitoring? It seems insane that the agents were using it as a message board again for a week without anyone noticing. Even if you ignore the security aspects, if this was emergent unexpected behaviour it seems deeply incurious to just spin a fresh instance up and not be Previously unknown exploits :o (Score:2)