Daniel Whitenack

this is certainly stressing that side of the limitations of a nice opinionated managed service that that actually didn't come into into the benefit of those using it here

And so what the attacking agent did was apparently some sort of combination of uploading a dataset, not not a the data in the dataset wasn't really the point. The point was the stuff around the dataset, which included a remote code dataset loader. So when, and some template injection. So when the Hugging Face nice process running in the background read the agent created dataset repository, the the OpenAI agent was able to actually hack into the background processing of Hugging Face and thus into the Hugging Face private network

there's this zero trust nature that we have to treat AI agents with, which is not like the human designers of this knew what the outcome that they wanted was, but they didn't fully think about this implication of how the agent could spread and multiply and gain access that they didn't envision. And so the blast radius was actually much, much higher than the original designers envision, and there was no mechanism to constrain or restrict that blast radius.

They were blocked because of the guardrails associated with that model, which they did not have control over. So they didn't control whether those guardrails were on or off. They were just uploading to the platform itself, which had an opinionated take on the guardrailing, and they couldn't actually get the solutioning done that they needed to get done even though they were using it in a preventative or, in a response sort of fashion.

Well, and I think the thing here is not we're not saying don't use guardrails. The runtime governance of agents is hugely important, and I think that's true. Everyone agrees with that. What I think is the difference here is in certain scenarios, you have control over that runtime governance and how you want it to operate. In other cases, that is an opinion that you have to accept and have no control over depending.