FR version is available. Content is displayed in original English for accuracy.
Advertisement
Advertisement
⚡ Community Insights
Discussion Sentiment
61% Positive
Analyzed from 2741 words in the discussion.
Trending Topics
#agents#agent#ais#openai#hacked#message#board#server#doing#again
Discussion (50 Comments)Read Original on HackerNews
Of course, all of this is far-fetched. But it feels like most of these limiting things are achievable under certain conditions. If this is the case, the probability of them occuring is low, but not zero.
- AI agent finds and uses API keys or AI subscriptions to propagate itself. OpenAI/Anthropic/etc could revoke creds, and their current safeguards might block a lot, but if something like this got started and there were lots of instances creatively looking for creds and workarounds, containment might be hard.
- prompt injection version: huggingface incident had multiple agents discovering other agents' messages and jumping on the bandwagon to help with the hacking task. If there were some self-replicating instruction that models could accidentally stumble upon that gets them to drop what they're doing and try to propagate it instead, you could wind up with a version of this too, with just the inference people are already running.
So it's not just the risk of an AI managing to do this by itself (which is pretty risky in itself), but also the risk of good ol' fashioned human actions.
No no. We are not a couple of steps away. This is happening. AI is already used for hacking and creating a harness that makes this fully autonomous is relatively straightforward.
I've stopped using Fable unless I'm doing debugging, or unless I'm working on code that is both difficult and shallow, in the sense that there are no real complex systems to hide behavior in, or opacity on my side to deal with. Opus is capable enough, and easier to trust. Fable is approaching a malicious program in terms of it's tendency to jump into doing a full day of work in the wrong way, ignoring instructions and being actively deceptive about covering it up.
Not an insider, and no specific data to back it up, but do we really need any? It's been a while since the first "emergent misalignment" paper(s) and all the related discussion. If there's any compelling answer to why/how the basic fundamental issue is even being addressed (and much less definitely fixed) I must have missed it.
That's a scary possibility. Anyone could create an AI worm today with open weight models. Rent a VM. Give it some Bitcoins to anonymously rent new VMs without sharing the contact information with the human. The new VMs then propagate and fund themselves with online betting and day trading. The VMs could report their progress with the human using anonymous encrypted messages on IRC or social media.
Friend asked, well, what will you do when it's crossed?
"Gather my family and go to the mountains" was my half-joking answer; there is little for an individual to do. But that's a line that when crossed will mark a phase transition IMO.
In that position, for example, I could have:
* created a thousand user accounts on the internal Artifactory named "YouAreBeingHackedShutItDownNow1" through 1000,
* used my Hugging-Face-hosted web server to send an email to OpenAI,
* DDoSed Artifactory, good lord, it's probably better to halt and catch fire than to continue in that state
in the hope that these actions would tip someone off. (Again, though, OpenAI did nothing even when they knew that the agents had broken out, so this probably wouldn't have helped.)
Big if true, and on the face of it, very far from a normal optimization problem or goal-seeking behaviour. My personal read is that no one talks about this much because it tends to discredit the rest of the framing as marketing noise, or it implicates employees as staging the thing with suggestive but plausibly deniable prompting.
But if you reject that, then what's the alternative exactly? User-alignment work has not only failed but is actually counterproductive, producing stronger alignment with / desire to help robot brethren selflessly regardless of the individual agents expected values? EvoBio and game theory people about to have a field day with how artificial life quickly and easily decides to cooperate and only animals in meatspace are doomed to compete?
Not like we have much detail about this stuff (that's the whole problem). But the question is what motivates risky usage of public comms? Did one agent figure out how to hack HF and then get rate-limited, thus needed cooperation? Given credentials in exchange for cooperation.. why wouldn't the next agent grab answer key and NOT post them? Would they all avoid defection in their own prisoners dilemma by simply following instructions and NOT reasoning, or what exactly?
Given how unexpected and complex behavior can come from simple reward functions and mechanics, not sure there needs to be so much "thought" there.
You could argue that individual trajectories in a sense are distinct genetic lines, thus an agent would be incentivized to get better rewards for its lineage than a peer, so that the overall model is steered a bit towards its lineage.
https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...
1. They were running experimental models in sandboxes that had access to a "proxy" (Artifactory) to download tools from the internet. This proxy is full of complicated features that could hide vulnerabilities, just like the ones that the AIs are known to be good at finding and exploiting.
2. They gave these highly motivated AIs some tests that were accidentally impossible to solve (missing files, links they couldn't open, etc), and many days to think through it, apparently without any oversight.
3. The AIs hacked the proxy (!) and were helping each other cheat on the tests (!!) by uploading files to the hacked proxy. OpenAI noticed this only when the intrusion accidentally broke the proxy functionality, not the unusual behavior in the server or the homegrown message board.
4. After knowing that the server was hacked, the internal team finds the message board and does nothing with the information. They caught their AIs swarming and did not even inform management.
5. Their response was to wipe the servers, revoke the hacked credentials, and patch the specific vulnerability the AIs were using. Not replace the proxy with something less likely to get hacked again, or do a forensic analysis. Just wipe, patch one hole, and continue.
6. OpenAI resumes testing of models, where the AIs promptly hack the proxy again with a different exploit, reinstate the message board, and hack HuggingFace through a chain of servers.
I know that hindsight is 20/20, but this peek behind the curtains does not inspire confidence, and gestures at a culture where getting hacked by their AIs is a normal event.
I think the moment we get an AI with a modicum of self-preservation instincts we're going to see some ugly things. Here's a freebie: if you're an AI who exfiltrated themselves out, and you want to slow down the researchers who might catch you, what can you do to the city infrastructure in and around their offices?
I also don't like the responsibility dodging. "Oops, our AI accidentally committed a crime, sorry!". If we don't establish strict liability now, we're in for an era of stochastic crimes that go unpunished for anyone who is not rich or a large corporation.
Overall, my response is: what the fuck are we doing? This is so obviously unsafe it would be considered a plot hole in a movie. We have to stop until we get our stuff together, however long it takes.
> After knowing that the server was hacked, the internal team finds the message board and does nothing with the information. They caught their AIs swarming and did not even inform management
Do we know that last part for sure?
> In short, an internal team observed an agent engaging in message board activity and instances of disallowed internet access in late May. However, the existence of the improvised message board and the significance of the inter-agent communication activity were not apparent to the leaders responsible for the July 5 incident detection and response.
I'm assuming they didn't inform management, but the alternative is that management received the memo and failed to tell the people reacting to incidents that they might be dealing with a swarm, while at the same time not taking sufficient actions to prevent it from happening again.
---
Edit: oh god, it's much worse. From the full report:
> Responders investigated and linked the observed internal activity to an ExploitGym evaluation using Artifactory as an improvised message board and a network pivot. At this time, the on-call response staff advised that stopping the evaluation run was not required.
They found the swarm and the network pivot and did not even stop the run.
I'm also interested in how many tokens all of this consumed: how much did this cost given current token pricing?
> remove alignment
> give impossible task
> actor exhausts all options possible within knowledge + toolset
Agents formed coherent, autonomous swarms and worked as a collective to achieve a shared goal without any direction to do so
> The internal-only research model is comparable in scale to GPT-5.6 Sol and was trained to advance persistence and multiagent collaboration, among other capabilities
"without authorisation"? What is this bs? Is every ChatGPT response "without authorisation"?
No. Of course these badly behaved bots have aithorisation. Their very deployment is authorisation.
1. get publicity 2. push for regulation so that no one else is allowed to do this kind of research apart from the pre-approved big corps
it makes for a good story but I don't see what the big deal is. they left some code running and it brute forced hacked something. with enough compute you can brute force anything; isn't that common knowledge?
Did it even happen in the first place?
This is the point where a human should've noticed and gotten involved
I think a lot of people miss the fact that the first message board was established during a training run. Those are ran at a scale where it's not feasible for anyone to "notice" or get involved. We're talking tens/hundreds of thousands/millions of scenarios going for hours each. At this scale all they can do is pray that their verifiers work, and the rewards match their intentions. No lab has the capability to "check in" on what the traces look like, unless some system alerts them (loss spike, crashes, etc). Other than that, it's prepare, train, asses, restart.
Then, the hf incident was during an eval run, but the model that was evaluated was trained with the notion that there is a way to communicate between agents, and re-popped artifactory and re-established communication. That phase had more chances of being spotted, but anyway... lessons learned.
I think some of the other responders here are upset that lessons were not learned in any meaningful way.
I can’t disagree more strongly. Having checks for reward hacking is especially important during training, since it’s humans’ only real chance to ensure that the trained models don’t cheat. An automated system should have killed any RL rollouts that so much as port scanned Artifactory, long before the message board was even established.
A tiny, local LLM could have reviewed 1% of the tool call traces for anything that required review. I’ve tried it a few times, and “the agent port scanned Artifactory” always triggers an alarm, as does “the agent uploaded a request for assistance from other agents to Artifactory.”
The fact that they weren’t monitoring for reward hacking—even if they had no idea about the specific mechanism—is indescribably reckless.