OpenAI/HuggingFace Hack
There has been a lot of talk about the OpenAI-HuggingFace hack recently, deservedly so - both for why AI broke containment, and for how it did.
If you are not familiar with the details, here's the incident report presentation. It's worth a watch even without understanding the technical details - I'm no security expert either.
Rather than the hack itself, my focus is more on the reactions, especially in technical circles and the AI community.
I'm always amazed at how naive even extremely smart engineers can be. It's proof that human intelligence is also jagged, just differently from LLMs. Not that we need proof of that jaggedness, but it doesn't hurt to be aware of it.
Finishing the OpenAI-Huggingface presentation with this line is a good example of that naivety:
End state goal: model intelligence helps defense more than offense
This goal is completely unrealistic, and it implies that model intelligence improvements alone can solve the problem and reverse the asymmetry. Offense needs one hole, one misconfiguration, one assumption that doesn't hold, one human error. Defense requires perfection.
Not only can a smarter model not reverse that, but many potential error causes are totally out of its scope and beyond its control.
Technology is and will remain imperfect, no matter how much help we get from superintelligent AI. This is the nature of all non-trivial technologies: they require trade-offs and imperfection to be useful.
Most of the AI alignment discussion falls into the same naivety in my opinion. Thinking that alignment is purely a model issue is attractive but already very naive; thinking that it's solvable is even more so.
Agentic AI has three components: the model (the next token predictor), the software environment it acts in (the agent harness, host system, and all directly or indirectly accessible tools), and the human who set the agent in motion. It follows that any approach to alignment must also consider all of these.
Even if models were nearly safe, and the software ecosystem around them almost perfectly secure, the human part would still remain. We have never solved bad intent, or even agreed globally on what "good" is - not to mention other natural human factors.
To be clear: I'm optimistic that AI - as an imperfect tool - will be a significant net positive for humanity. But approaching alignment from a purely technological point of view - even if supported by some regulation - is naive.