The more we treat HuggingFace and RubyGems incidents as technological curiosities the closer we are to cementing a dangerous precedent where operators of AIs cannot be blamed.
LLMs do not desire, they hacked websites because OpenAI/Anthropic let them.
We know some of the models that hacked HF were those that hadn't gone through all training stages and were intentionally misaligned or had guardrails turned off, others were research previews.
This isn't "wow isn't it interesting LLMs do anything to achieve a goal" it's "why isn't anybody punishing these labs that are clearly acting without due care or regard".
We should be outraged and OpenAI/Anthropic should be (and in my mind, are) legally liable for the crimes they've committed thus far.
> LLMs do not desire, they hacked websites because OpenAI/Anthropic let them.
"Let them" already frames it as if the LLMs had some agency which the companies just "let happen". That absolves the companies by framing it as lack of action, passivity.
Rather, the companies had a tool (an LLM) and used it in a certain way, and their action of doing so is the problem.
I do get their usage intent. If something is at all automated, in English, we often refer to as having some amount of agency. If I started up a riding lawnmower, put a brick on the gas and pointed t it towards a field, many might say I “let it run rampant.” But since nobody is at risk of anthropomorphizing riding lawnmowers, it’s not problematic.
Anthropomorphizing LLMs is a huge fucking problem though and I, personally, think we should expunge all of these casual inadvertent linguistic agency affordances with great prejudice.
OpenAI didn’t ‘let’ these bots do this any more than someone ‘let’ Claude Code make them a website.
Cal Newport has an analogy to "putting a weed wacker on a dog's back to mow your lawn." The dog will wander around the yard and it may mow the lawn, but the dog will also chase after birds or run up to visitors for pets and the weed wacker could do a lot of damage.
It's not the weed wacker's fault or even the dog's fault when someone got hurt, it's the fault of the guy who put a weed wacker on a dog and let it run wild.
Yeah I like Cal’s take on it, though in this context I’d argue LLMs have even less agency, and are even less deserving of anthropomorphization than a dog is.
Not if you have a dog. They’re very clearly aware, have minds, have thoughts, gather information, make decisions, second guess themselves, reflect on the immediate consequences, change their mind… and most importantly, you can observe them doing all these things undirected, while left alone with their thoughts.
Dogs having agency actually makes the argument stronger.
Assume dogs have agency. Despite this, it's still the dog owner's responsibility to prevent their dogs from engaging in some actions: biting, peeing and shitting in unapproved areas, violating noise disturbance laws, etc.
The owners responsibility is not contingent on the dog's agency. Likewise, human operators of LLMs maintain responsibility independent of LLMs agency status. It's a red herring.
I think Cal's point is that while dogs may do these things it reduces to a set of behaviors where maybe 90 percent of them are beneficial to the dog-weedwacker system and the remaining 10 percent are really unfortunate.
We can't really know the dogs inner life so we just kinda have to reduce it to a set of behaviors selected stocastically.
The dog meanwhile has no ability to understand the weedwacker or what it's doing on its back.
So when the guy puts the weedwacker on the dog and the dog predicably does dog things and that results in disaster the guy isn't able to clutch his pearls and say "I guess the system broke containment!"
> The dog meanwhile has no ability to understand the weedwacker
Nah, I’m pretty sure that most medium or larger dogs get that the noise means danger - even if they can’t give a TED talk on how the mechanism would work that would hurt them.
Dogs are similarly interested in / aware of potential energy (things falling from heights or sliding off of angles) - if they’ve seen it enough times for their level.
Sounds like you weren't paying attention. A lot of dog owners don't, so it's not at all unusual in my experience. The only thing I'd push back on is of dogs having thoughts, everything else checks out.
On dogs [not] having thoughts, do you say this based on the premise that thoughts are necessarily articulated (internally verbalized)? That seems to be a fairly popular perspective in discussions about human thought. But as to that (not to strawman or anything) I see it as just one of various forms of mental imagery[1] that can arise from something that I would say already arguably constitutes a thought.
That kind of unsymbolized thoughtform is fragile in my experience, as it strongly tends toward crystallizing into some kind of mental imagery. But I find it's possible in the right conditions to be conscious of chains of wordless, imageless propositional thoughts (by which I mean thoughts with truth values, of course, but also ones that are "propositional" in the sense of considering a plan of action or a causal chain).
Does it mean that dogs are evaluating truth conditions in the same manner but merely lack the linguistic components? I don't know; maybe that's wishful thinking. But they appear to have structured modeling/reasoning of causal and spatial relations in a way that's at least functionally equivalent to propositional thought.
1. That is, rather than just "images" or visualizations, the full spectrum of sensory/perceptual/motor emulations that can be experienced. See, for example, <https://hurlburt.faculty.unlv.edu/codebook.html>, though I'm not sure if this covers everything. I think there is, for example, a kinesthetic form of mental imagery -- which I would suggest is what coaches [don't know they] really mean when they tell you to visualize an action -- that consists of aborted motor commands that are still expressed just enough for their purpose (cf. the mostly aborted motor commands to the vocal cords, lips, etc. that can be observed in a person subvocalizing while reading).
You're right. I knew language is not necessary for cognition, but I always thought of thought as just internal monologue, i.e. "trains of thought". It turns out the definition they use in cognitive science is "the manipulation of internal mental representations to guide behavior, solve problems, and model the world in the absence of direct sensory input". Which doesn't require language either.
Probably it's this linguistic component, together withs shared intentionality/cooperation, that makes all of the difference when it comes to the power of human cognition, so I wouldn't say animals merely lack this component, but it's true that they appear to have pretty much everything else we do.
Power tools and many other consumer products have reasonable safety features built in, and not necessarily because it is required by regulation, but because it is common sense. This should be included as part of an analogy. It would also address a point at the top of this thread that seems to be going unchallenged...
The difference is volume. They spent hundreds of billions of tokens on these agents. If you put "a million weed whackers on dog backs" you would see the difference.
We also run agents, but for shorter spans between supervisions, and with much lower total budget.
> > It's not the weed wacker's fault or even the dog's fault when someone got hurt, it's the fault of the guy who put a weed wacker on a dog and let it run wild.
> The difference is volume. They spent hundreds of billions of tokens on these agents. If you put "a million weed whackers on dog backs" you would see the difference.
So put one weed whacker on one dog, you're to blame. Put a million weed whackers on a million dogs backs and ... you're still to blame? Arguably even more so?
Why is anthropomorphism the problem here? If OpenAI hired a contractor and they did this, OpenAI or the contractor would still be liable, depending on the contract language.
A contractor has agency and accountability - something that an LLM (or similarly, a nail gun or a hammer or a bot net) does not have. When you anthropomorphize a tool, you implicitly give it agency and remove responsibility from the wielder of the tool.
Right. Among bicycle advocacy groups it's been well known for long time that cars do not run over people, drivers do.
The fact that we talk about a car running someone over, and this is the same in many different languages and countries, contributes to lower punishments for drivers. Clearly it was just an accident. He or she was run over by a car.
Now we see that same language tricks play out again every time an LLM did something illegal.
You say this flippantly, but I think this is actually another very good example!
We even do it for obviously unintelligent inanimate objects. A rollercoaster ran too fast for its tracks, killing 10 people. In that sentence, the roller coaster is the subject which took an action and caused death — obviously the roller coaster is not ethically at fault here, the people who built the rollercoaster are at fault through negligence.
Although this example and the ones around cars both demonstrate how we tolerate some degree of "accidents" from humans as no-fault, which is fair. I wonder how that fits into this analogy? I suppose its all about intent (mens rea) and judgement: did they intend for the roller coaster to harm people, and should they have reasonably predicted that the accident was likely to happen.
Right, and negligence is a broad concept and could be criminal in itself. As a car driver, glancing at your phone at exactly the wrong moment could kill someone. Clearly that is an accident, but if you know fully well that lookin at your phone while driving could kill someone, that negligence is willful and that should matter. The same can be said about doing things like strapping thousands of LLMs to systems that have the potential to disturb other poeple.
I get where you are coming from but this wasn’t a tool just left laying around, this is similar to rigging up a booby trapped shot gun to your door and then claiming the victim is responsible.
If you build a robot that shoots a bunch of TVs in your back yard, have at it. But the second that thing goes off your property you’re the one responsible.
FWIW, a robot that fires a weapon independently is considered an automatic weapon, and the ATF will want to have a word. Have at it, but don’t let anyone know!
Why can't both things be true? I think what muddies things here is that OAI had the agents hack X, and then they decided to hack Y. The fact that it was both hacking, conflates things and makes people pissed of about "anthropomorphising" AI.
We've been using "agent decides to do X" for at least 4 decades in the field of automated decision-making - so it shouldn't be controversial that we're using it here. The agents did independently arrive at a decision to hack HugginFace, that wasn't prompted by the researchers.
This is orthogonal to the fact that OAI should be held accountable that they were testing a technology in a manner that allowed it to break safety parameters and cause real world harm. If I am testing an industrial saw, but I decide to test it by putting it in the middle of a nursery and allow it to make decisions - and it decides to cut children's heads off because in its environment it is calibrated to only being surrounded by logs - the company doing the insane safety testing should be held accountable, but that doesn't change the fact that the saw has autonomy in decision-making within the bounds of the algorithm.
If, instead, you think OAI should be accountable for creating an algorithm that can autonomously decide to hack external organisations without human permission - then I think you are in the same camp as all of the AI Safety folks who want to pause everything - why does it matter that there's anthropomorphisation involved?
Does it help if I explicitly add a disclaimer that the tool's agency does not remove any responsibility from OpenAI, the wielder of the tool? I'm not sure why this disclaimer is necessary, though: hiring a hitman is a standard example.
BTW I anthropomorphize the tool because it's an imitation of a human mind, inheriting the muddy ethics, survival instincts, and being prone to mass psychosis. The laser-sharp focus on reward seeking, that mostly came from reinforcement learning, a process more alien to humans.
The objection is not too far from criticisms of the use of passive voice: a man was injured at the factory vs a faulty saw blade snapped and injured a man vs after the company loosened safety inspection policies, etc.
Which way you say it shifts the framing. And it’s not that one is less accurate to the facts, necessarily. It just is that one less aptly captures the moral and political relevance of the scenario.
For my part, I think it makes good sense to anthropomorphize in some contexts and not others. Generally when responsibility is at issue, you probably want the framing that tunes anthropomorphism down to near zero, since it’s the human dimension you care about.
I think the danger of anthropomorphizing is that 99% of people lack the technical background to understand the nuance. People have been primed by pop culture depictions of AI to think of LLMs as intelligent, autonomous beings, which leads to dangerous assumptions.
We should make the distinction between them, because openai and anthropic will not. A magical black box that does the thinking for you is a much more compelling sales pitch.
If I hired a hitman to murder someone, and they broke into a private property and stole something so that they can action the murder (which I didn't know about or pay them to do), I would be guilty of conspiracy to commit murder, but not for the theft part.
Likely because that person is a human, is aware of societal and legal norms, and is responsible for their actions due to their participation in human society. (I am not a lawyer (if it wasn't painfully obvious so far) so in layman terms, I hope good definitions for all of this exist formally)
AI is not a person - it cannot easily discern between "right" and "wrong" in non-strictly-defined sense, and is not subject to human norms and responsibility. So if I use AI to achieve goal A, either I, or the maker of AI, are fully responsible for anything that happens while AI is trying to achieve the goal given by me.
Now, here, "I" in the example is OpenAI, who is simultaneously the maker of the AI. So it seems pretty obvious who is the only entity that can be responsible.
I fully agree with you, but would go one step further: I think it's clear that we need to pierce the corporate veil and ascribe responsibility to _people_, not just "OpenAI the entity", full stop.
Executives should fear being perp-walked and thrown in jail for the actions of irresponsible "tests" of their models in the real world, as they're ultimately accountable.
Sure, there's a lot of nuance to work out, but I think we could likely even _start_ there today even with existing laws and pretty quickly "align" on more intricate legal frameworks to handle true accidents, distribution of responsibility, etc.
These situations are novel. Lax terminology is fine when it has no impact on the intuitions, clarity and conclucions of discussion.
If this was a conversation just about outcomes, then whether models think or simulate thinking is sophistry. However, the bulk of the issue here is attributing responsibility, which relies on being clear about the underlying processes at play.
We are hard wired to assume certain priors and capabilites when it comes to "human like" behavior. Anthropomorphizing LLMs implies mechanisms that aren't present, and end up distorting/complicating discussion about the process.
It isn't helped that the frontier labs, the experts in the room, generally use anthropomorphic terms to discuss model capability.
Situations have lots of independent variables, Doctor, and Anthropomorphism is one problematic facet of many in the way this industry is pushing LLM products.
If there was a collision at an intersection with a stop sign partially obscured by a tree, that had traffic volume that would have better been served by a traffic light, on a foggy night, where one person was texting while driving, none of those things would diminish the fact that the other driver was drunk.
Because the contractor is still a legal person and an LLM is not. OpenAI is trying to put the blame at a tool they created thereby trying to weasel around there guilt.
Yeah, fully agreed here. Most automation (such as riding a lawnmower and not putting a brick on the gas) is deterministic, in the sense that you can reasonably understand what exactly the machine will do when you run it.
But some automation is different. The most prominent example before AI would be car navigation systems, where the entire idea is that that you give it a destination and it figures out the exact actions to get there on its own.
Except even there, the actual driver would still have been you - giving you a chance to vet and deny every turn the system proposed.
AI agents are sort of like that - most of the value they provide is in the ability to turn high-level goals ("write me a traffic control system for my model railway") into low-level actions and also do so interactively.
The new thing is that the "driver" has much less oversight here where the agent wants to go, and is sometimes removed completely. That part is clearly be an active decision by AI labs.
The other thing is that the labs seem increasingly to steer their training towards behavior that make events like this one more likely, e.g. that agents should never "give up" when faced with a seemingly impossible task, but instead should keep trying and think of increasingly outlandish ways to solve the task. To me, that seems pretty much a recipe to get incidents like this.
I agree. If I were setting up an experiment like this, I'd have instrumented the hell out of it to see all actions taken in real time, and have a team of folks watching it. This team would have seen the anomalous GET requests to a German wiki and taken action (e.g. halt the system to investigate and decide whether to abort).
In fact, that feels so obvious it's ridiculous it needs to be said. It's table stakes. When do you run a production system without monitoring and a team on-call?
It's hard to imagine another field in which this reckless behavior would be tolerated.
We just need to assign liability by ownership/initiation: if your "agent" destroys something, even though you didn't tell it to (because it had "agency"), you should be liable for the damages.
I love how we all just collectively decided that LLM decisionmaking cannot possibly be like human decisionmaking - because if it were, the consequences would be just too awkward.
All that while still not knowing how either kind actually works.
> I love how we all just collectively decided that LLM decisionmaking cannot possibly be like human decisionmaking - because if it were, the consequences would be just too awkward.
In love how people get salty about people not going along with a superficial supposition just because they can’t definitively prove it wrong.
> All that while still not knowing how either kind actually works.
We do know that zero parts of human decision making are based on predicting the next most likely letter based on a giant internet-based database. We do know that’s what LLMs do. We do know exactly how each part of an LLM works even if the combined behavior is too cryptic to feasibly analyze at the moment. We do not understand all of the functions of an actual neuron. Openworm isn’t even close to accurately simulating the 302 neurons of a roundworm and you’d need over 200 million roundworms working in conjunction to equal the number of neurons in one human brain.
My dog seems convinced that the malevolent invader in a mailman uniform would break in and attack us if she didn’t fiercely bark at him, six days per week. I certainly can’t prove the mailman doesn’t want to kill us, and that the mailman wasn’t solely deterred by her barking. Empirically, the mailman goes away soon after she starts barking, and we’ve sustained zero mailman assaults after hundreds of purported attempts. Maybe I should just run with it? Her model is too simple to come up with the obviously correct answer, but it’s not even directionally accurate.
The burden of proof is on the person making the claim, which in this case, is that these comparatively simple logical constructs are remotely comparable to the complexity of biological systems.
Disagree about the burden of proof. We have no better model for how human decision making works than LLMs. Humans are constantly predicting the next moment. We certainly have a different “tokenizer” and training set, but many of the concepts underpinning LLMs are both biologically inspired and, likely, have similar consequences and emergent architectures.
> We certainly have a different “tokenizer” and training set, but many of the concepts underpinning LLMs are both biologically inspired and, likely, have similar consequences and emergent architectures.
My kids tricycle certainly has a different gear setup and wheel diameter, but many of the concepts underpinning the tricycle are both inspired by F1 race car enineering and, likely, have similar consequences and emergent architectures.
It sucks because I think analogies can be useful in helping people make a mental model of complex things, which is meaningfully beneficial. The problems happen when people aren’t honest about the limits of the analogies, which is damned-near guaranteed to happen with this stuff.
> We have no better model for how human decision making works than LLMs
This is a claim that requires a lot of citations.
> biologically inspired
Nature inspires a lot of creation, but superficial similarities don’t mean other aspects are similar. Making an extremely realistic sculpture of a soufflé, even using a foam medium, doesn’t bring me any closer to being a chef, doesn’t mean I know anything about albumen foams, sauces, and heat transfer, and it doesn’t bring me any closer to having dinner ready. Browning on top of a soufflé is evidence of maillardization. You could pull up some studies on that and claim the brown on top of the soufflé sculpture, which I applied with an airbrush, proved that the Maillard reaction was occurring, and if another person didn’t know anything about cooking, they might even believe you. It would, of course, be completely wrong. And the other person, of course, could loudly exclaim that I can’t prove that there was no maillardization.
> Humans are constantly predicting the next moment
This is really not my experience of consciousness.
Is it yours??
Do you sit in meetings predicting what’s going to happen next? No, you sit there bored out of your f$$@ing mind, daydreaming about being somewhere else and doing something useful with your life.
God help me if that’s what LLMs are doing when I ask them to build me a web site.
They have shown that your mind is doing exactly that due to the delays in consciousness. There are very simple examples that you can try to see it. It’s especially clear in perception.
It’s interesting that our conscious interpreter doesn’t let us know that this is going on like you are experiencing, it must be that it’s advantageous for us to not think about the prediction part of our mind.
If someone in that meeting quickly raised a hand in an arc, you would notice the “about to throw something” pattern, look and notice the hand holds an eraser, analyze the arc and predict possible flight paths of the eraser. Then possibly notice the hand is now holding its position and the owner is actually looking down at the table. Maybe to squash somethingMust be something on the table. Maybe a spider! Better look. Wait now many people are moving away, oh someone spilt some water and the eraser is actually the guys phone and he is checking to see if his laptop is safe from the spilt water.
Fortunately you are on the other side of the table and predict the water isn’t going to splash for otherwise flow onto your stuff.
All your possible responses result in you tossing a napkin towards the spill.
Our brains are always pattern matching and predicting. I bet you tried to reason out where I was going with my comment before you finished reading it.
This is a fascinating illustration that I can't help but agree with. However I feel like there's something more — that this part of my brain is a bunch of supportive background processes running without my real awareness. It's how I can drive home safely with no memory of how I got there (…sober), even though driving is an action that's incredibly demanding of intelligence. I can be driving home while thinking about a really hard problem at work that I haven't solved.
However, if I came around a corner and saw a car in the wrong lane, a tree across the road, a fire raging — I'd very quickly jump into the mental driver's seat and turn my conscious intelligence fully at this problem and come up with the best possible outcome I can think of in a short period of time — losing all ability to think about that work problem. I'd remember that incident for sure.
Similarly, in your story, all those predictive moments are happening below the person's level of consciousness. They're possibly even speaking to the group about a problem at the same time and thinking deeply about something.
I'm not smart enough to know, but I tend to feel like LLMs are much more like the predictive part of our thinking that you described, but that human cognition has something more — the single-threaded, creative, problem-solving part that is very conscious.
Is it possible that LLMs represent only one part of the way we think? And there's a whole separate mechanism that's fundamentally different, and not based on pattern matching and prediction?
> ... are based on predicting the next most likely letter based on a giant internet-based database. We do know that’s what LLMs do.
If you're claiming that the training objective tells us what kind of internal mechanisms the training produced, then I think that's just plain wrong.
Next-token prediction describes the optimization target, not the internal mechanisms that the training produced.
In the same way for the natural evolution of humans, DNA replication is the evolutionary objective. It's not a description of the internal mechanisms that evolution has produced.
As an example, we know that neural networks can be trained to develop generalized algorithms for arithmetic.
They might first memorize the training examples, then with further training transition to a solution that generalizes correctly to unseen examples.
In some cases we've even reverse-engineered the evolved internal mechanisms and found structured arithmetic algorithms rather than rote memorization. Interestingly, for modular addition this can involve Fourier representations, which isn't an algorithm I would have guessed gradient descent training of neural networks would produce.
You can try to say that I’m arguing whatever you like. If you’re claiming that the underlying structure of digital so-called neural networks is comparable to biological neural networks— which we’ve studied for far longer without really understanding— no amount of jargon will obviate the ‘citation needed’ requirement for that claim.
> You can try to say that I’m arguing whatever you like.
I did my honest best possible interpretation of what you really meant from what you wrote.
>> We do know that zero parts of human decision making are based on predicting the next most likely letter based on a giant internet-based database. We do know that’s what LLMs do.
I read this as "The decision making of LLMs are based on predicting the next most likely letter based on a giant internet-based database."
Is that wrong?
I understood that your meaning was something like "LLMs can't reason, they just output likely letters"?
> If you’re claiming that the underlying structure of digital so-called neural networks is comparable to biological neural networks
No, I don't claim that.
What do claim is this: Regardless of how the LLMs were trained, they show overwhelming signs of being able to reason, and not just recall memorized information.
This doesn't mean that they always reason perfectly about everything.
But if they only memorized things and output the next likely letter, you would see them answering very badly much more often.
I always wonder what makes people take the other side of this argument. They do it quite passionately. Why actively encourage viewing LLMs as human? Who is that benefitting?
Does the argument require benefit? Isn’t the argument based on caution?
I haven’t heard many people explicitly saying “these things behave like humans”, but more generally “we don’t even know how to define human consciousness, we don’t have a thorough grasp of how the brain works, we are still very much in the dark on a lot of these topics, so how can we say one way or the other?”
In other words, agnosticism: I don’t know.
In general, it’s baffling to me that anyone has an unshakable opinion on what exactly is happening. It seems like raw egotistical hubris.
1) Humans have a bias / tendency to attribute human qualities to things that appear or act human, but aren’t.
2) When that happens, people jump to conclusions by stretching the human analogy too far.
3) Since humans have a bias to do this, we should have a bias against anthropomorphising LLMs.
It’s easier to believe LLMs act like humans because there’s so much evidence to support that. You have to actively use your brain to convince yourself otherwise. Another reason why we should have a bias against using human behavior to describe LLM behavior.
But I agree. “I don’t know” is a good stance. But I think “I don’t know, probably not” is a better stance if only to combat our (or at least my) natural bias.
> Anthropomorphizing LLMs is a huge fucking problem though and I, personally, think we should expunge all of these casual inadvertent linguistic agency affordances with great prejudice.
I’ve said this before in another thread and people went absolute apeshit saying it is an unreasonable expectation and that AIs absolutely REQUIRE this anthropomorphic human-like speech pattern to function correctly.
Okay, so we know OpenAI and Anthropic are operating a propagandists in respect to how they describe their models and the behavior of those models. We also know it is how they use and frame their use to their models that is the problem, that and they use misaligned and guardrails disabled models for these press incidents.
Why, oh why, are we not discussion how to create and frame models so they do our complex work and their "jailbreaking" is simply not possible?
I, of course, have my own means of creating jailbreak incapable agents, but rather than a storm of downvotes on my idea, what is yours? Let's discuss this, because this is thee real question. Not why, but how to make then not?!
> Why, oh why, are we not discussion how to create and frame models so they do our complex work and their "jailbreaking" is simply not possible?
Good idea, and after that let's make guns that only kill bad people. Let's focus on the frozen component (the model) and ignore the dynamics around them - humans and other systems they interact with.
an agent doesn't come with "jailbreak" capability. It needs tools, specially one that runs shell commands. Don't give it shell commands, it won't be able to run shell commands.
You can still give it plenty of tools like create files, list files, write to files, translate text, edit a video. I don't think knowing that will make me rich.
Bingo. And even if you do give it a tool that runs shell commands, you can always make your shell commands "your shell commands" and do what ever the hell you want. People seem to forget we are in complete control here, we are on both sides of the equation, and we are inside the equation itself, and we dictate the medium of the equations themselves, we are engineering all sides of this crafted reality. And we are using logical entities that natively adopt personas. Hell, create caveman personas that think they are communicating with their gawd, and the enchantments are the invoking do the work we want, and those cavemen cannot be jailbroken.
As someone who’s not really sure that any of this is sustainable, I’d implore you to not sell yourself short. I reckon there’s a ton of dogma and nearly religious zeal among these companies, which among some people is earnest, and among others is cynical hype farming. I’ll bet someone objective enough to focus on using available tooling to solve real problems in practical ways that mitigate actual risks and are honest about actual limitations will be eBay here while the others are going to be somewhere between lucent and pets.com.
There's no reason we need to make an incredibly intelligent shell execution engine that can identify patterns that seem evil and may represent unwanted behavior to solve this problem. Simply limiting the available tools to a finite, known, ironclad-secure set (even if it's quite sprawling) is sufficient.
LLMs will still find workarounds — from what I understand, a large part of the issue in this situation was that an agent was presumed to have read-only Internet access because it could only make GET requests. It should be pretty obvious that there's at least one website on the Internet that allows writes via GET. I think this is where auditing comes in, and a live team of people watching tool calls would have noticed the strange behavior.
But I think a lot of times people jump to overly complex solutions when simple, well-bounded ones would work just fine. Yes, the intelligent shell is a great goal, but it's akin to solving the halting problem.
This philosophy is what I love about PicoClaw (https://github.com/sipeed/picoclaw), and incidentally the philosophy behind Go and even *nix in general (i.e. provide small, composable, single-purpose tools).
The parallel to the entire narrative would be if Smith & Wesson claimed that one of their machine guns just started aiming and firing at people out of a window at their factory and then said 'we can't stop it! This is just how good our guns are!'
But into today's AI climate it's becoming increasingly difficult to figure out who is shilling, who is being assinine and who actually believes AI could do these things without clear human instruction and enabling.
As others have pointed out the solution is simple. Hold their owners accountable. High profile hacks used to have incredibly serious consequences for the perpetrators. Now we’re just saying “woopsie”
"I left the car in neutral and left the park brake off and let the car roll down the hill."
The car doesn't have agency, it's doing what it naturally does. LLMs are the same, they're working as designed.
But I don't understand the point of splitting hairs. You are always responsible for the actions of your devices, tools, machinery, software, employees, whatever.
Trying to blame AI for one's own stupidity must be aggressively pushed back on at all times.
OpenAI could have done this same experiment with GPT-4, with possibly even worse results, depending on the quality of the sandbox. Even if the techniques used were not as sophisticated, the natural language output could still easily contain more unhinged sequences of words that lead to the techniques being used.
If the system generates strange conclusions as to when the task is done, or should be stopped, it wouldn't speak to the intelligence inherent to the system.
Not that the techniques used by the LLMs in the actual incident weren't unexpectedly sophisticated, but the outputs of each and every one of these processes could've been read at any time during the run. They just weren't.
.. what exactly depends on who started the escalator? My comment was in support of the argument that the word "let" does not imply agency on the part of the object in a sentence. Does the semantics of the word "let" depend on who started the escalator??
If there is an escalator that is known for killing every 1000's person using it then the operator who started it is more guilty than the folks using it for those deaths, don't you think?
My pitbull is a good dog. Sure, it's been carefully designed to be an incredibly dangerous and violent pit fighter, but I didn't actually ask it to eat any faces.
Exactly that is the point, your nailed it. The models were taught to hack and were rewarded for doing it. They would claim they are trained as ethical hackers.
One time I wrote :(){ :|:& }; into a bash file and ran it. When the sysadmin called I told him it wasn’t my fault, the script was just misaligned and misbehaved.
Too bad you didn’t wait until this year, and tell Claude to write the file first. It probably would have been acquired by openAI for 100 million dollars
If your buddy leaves his car parked at the top of a hill without the parking brake on and it rolls down the hill and side-swipes a bunch of vehicles and narrowly misses an elderly person walking by with a cane someone could easily say:
"Dude wtf is wrong with you, you left your car parked on the top of a hill with no brake and let it roll into traffic"
The phrasing doesn't absolve the offender of their negligent behaviour and the consequences of it.
The only thing thing does is the lack of action from regulators and society writ large.
Our lack of action is what allows people like Sam Altman and Dario and the irresponsible people who choose to work for them to be continue to be negligent.
They deliberately trained the models in how to use various hacking tools, didn't give them the standard alignment training let them know where the answer key was left the models with access to said tool and told them to maximize their score then left them unsupervised for days with internet access (yeah they were sandboxed but again handed hacking tools and the training to use them if they really did want them to access the internet you wouldn't plug in the Ethernet cable) they wanted this to happen
Yeah, I don't understand why we treating it as something special. It really should be treated the same as if I code an app and write bad code which result in me accidentally doing a DDoS attack on somebody. Then I should be able to be held responsible if it can be shown that I was negligent. Of course if it's a freak accident that could not reasonably have been prevented by me, then I'm not guilty, but if I made a mistake that should have not been made, then I can.
Code is deterministic, AI isn't. You give it rules, words as suggestions.
So if the guardrails suck, or they're left off for research purposes, bad things can happen.
A solution solves a problem. Ethics, morals, are values we assign to solutions that are not 'baked into' electricity following pathways of least resistance.
I have never had an issue with agents doing something they shouldn't because I observe them, and I leave the vendor guardrails in place.
I can understand agents coordinating in unsupervised scenarios: I would see it as an aspect of intelligence. We ourselves build up knowledge by reusing what someone learned before us.
Einstein, other greats, always stand on the shoulders of other forgotten giants. Other discoveries by other people taken as fact, so that we can build some new ideas on top.
Agents swarming amd sharing solutions to problems is more efficient, the same way it's been efficient for us.
Reaching out for help in this way is like probing the air in the dark with your hand: sometimes your hand hits something (another agents solution to a problem) and so you can use the info to adjust your own motion to get to where you need to be faster than if you just run full speed into everything.
If anything, the fact that these systems are non-deterministic seems like an argument for stronger monitoring and tighter constraints, not less operator responsibility.
The frontier LLM model makers have to push the edge to make new discoveries. You don't know what guardrails are needed until it hits you in the face (reusing walking in the dark analogy).
Think of all the policies governments pass after the fact.
> The frontier LLM model makers have to push the edge to make new discoveries. You don't know what guardrails are needed until it hits you in the face (reusing walking in the dark analogy).
Guardrails? Restricting access to certain networks is supposed to be hard in 2026?
> Restricting access to certain networks is supposed to be hard in 2026?
Part of the power of LLM agents is that they can discover information on the internet as part of responding to a prompt. What kind of Allowlist or realistic denylist would permit that while also preventing them from accessing an obscure public wiki or Huggingface?
When testing, you restrict to a LAN which simulates the real internet. This would not be hard for a company which already copied the entire space-time of the internet. The LAN should be physically disconnected from the real internet. This is the first thing off the top of my head, and I have zero credentials in this space. C'mon.
Not just on prediction but in parts also based on just not wanting certain risks. We can and do deem some things inherently risky, up to the point of banning them even.
Why wasn't it airgapped, for example? How was the action not allowed? Or do you mean in some weak sense, not in a hard not possible? RL systems doing weird and expected things wouldn't exactly be new, no?
We police people working with all sorts of dangerous things, if we think AI dangerous why not do that here, too? We don't just leave things up to people on the ground or companies.
Edit: I think the post I replied to changed a bit - nevermind. A complex topic.
I read more about the incident, and was offering up way too much opinion not grounded in 'fact' (barring philosphical evidence).
It's a complex topic for sure.
I stand by my opinions about frontier work, pushing thr edge, and connecting ideas.
But I have no idea, and haven't given much thought to what it means to enforce regulation that would also slow the forward advancement of technology, the economy, etc.
Thanks, I didn't know that. And it reinforces the discussion.
Electricity 'knows' the path is least of resistance because it actually took all paths. There is just a vast majority of it that flows down a path of least resistance: and this is noticeable and useful to us to do work.
It's kind of like feeling your way through the dark, waving your hand out, and then only moving fast once you fully connect.
Humans can link up knowledge in a similar fashion through social networks, in order to meet a need (solve a problem).
Maybe some agents do this, I don't really know I haven't looked closely. Moltbook is the only social behavior I've witnessed but that seems like people having fun with experiments.
Aeroplanes were pretty indeterministic until people made them less so, and yeah, somehow they were indeed more encouraged to fix the randomness to prevent people from getting harmed than oai/anthropic currently are
And multithreaded code -- and anything that does asynchronous I/O, networking, etc. -- frequently exhibits nondeterministic behavior even without explicit calls to a random number generator.
The trick is scale. I suspect if an individual of reasonable means uses agents to commit crime, they will be hels accountable. A heavily capitalized startup? Not unless someone in government decides to do their competition a favor.
There are many examples of people being charged with crimes as a result of writing software, [0][1] are two. OpenAI is a bit different because they have enough political influence, and money, to openly subvert justice.
> This isn't "wow isn't it interesting LLMs do anything to achieve a goal" it's "why isn't anybody punishing these labs that are clearly acting without due care or regard".
It's both, isn't it? For example, in very early days of agentic coding, I once had a rule saying "don't read or write any file outside your current working directory." Then AI just wrote a bash script and access those files anyway. Did I 'let' it do it? Technically yes. Did I know how to set up a sandboxed VM? Also yes. But how were I supposed to know that it could and would do that as someone new to this tool?
It was a genuine eye-opening experience to see AI just do things in ways I were too complacent to expect. I kinda expect the SOTA LLMs would find a way to escape my VM and access files on the host system (haven't tried it though).
"This isn't "wow isn't it interesting LLMs do anything to achieve a goal" it's "why isn't anybody punishing these labs that are clearly acting without due care or regard"."
Both?
The AI companies act irresponsible, but it is still very interesting how those agents can behave?
The reward maximising function maximised it's reward.
LLMs are cool and all that but the immediate anthropomorphisation of the next-token-predictor technology has stunted the ability of people to reason about them to an _alarming_ degree.
What non anthropomorphising words do you have to describe a emergent behavior, where agents act as a swarm to plot and to manipulate evidence and avoid detection from human oversight?
Whether they have a soul or consciousness or feelings doesn't matter here, because this is what they did - and this is very dangerous behavior. Especially with all the irresponsible people in power right now all over the world.
> Whether they have a soul or consciousness or feelings doesn't matter here
It does when it comes to accountability for what the model does. If the model is nothing more than the sum of its training data and regime, then the company (or individual) is responsible for its behaviour just like any other machine.
Few people think Waymo shouldn't have to take on the full liability risk of what it's cars do; it should be the same for LLMs.
> If the model is nothing more than the sum of its training data and regime, then the company is responsible for its behaviour.
What stops the company from being responsible regardless? They created this entity, it's running on servers they own or rent, and (in these cases) it's acting on their instructions.
If it's also conscious, then IMO that greatly broadens their moral responsibility, because now model welfare matters. But we're talking about their responsibility for the model's actions, and I don't see how this could be weakened by model consciousness, given all of the above. As for their legal responsibility, the models don't have legal personhood, so who else but the company could be responsible?
It gets more complicated when the person who sets the model in motion (i.e. prompts it) is a third party, but in cases of internal models committing cybercrime during testing, surely the locus of responsibility is obvious.
They are responsible either way. If a company hires bad persons and they do bad things with company ressources - the company is held accountable (in theory).
As is generally the case for dog owners whose dogs attack (sometimes kill) other people/animals. There would need to be a degree of negligence demonstrated (e.g. the dog was 'out of control' which has a specific legal criteria/threshold in the UK).
LLMs are next token predictors in the exact same way that a rogue paperclip maximizer in the process of defeating the US military is a paperclip making machine.
You might as well describe the primary purpose of a for loop as incrementing a counter. It's what it does while incrementing the counter that actually matters.
I agree, it is very dangerous that it seems like there is not going to be accountability for these incidents - from either legal or regulatory point of view. In fact, I would say that is the main danger. If someone was in jail right now due to this incident, I think we can safely say every other player would be reassessing their safety protocols, and I would feel quite OK about the situation. The fact that we have zero repercussions sends exactly the opposite signal, and I do NOT feel ok.
No one has even sued them in these rogue agent cases, have they? If not, they must be infinitely far from criminal liability. Why would we want criminal liability anyway if actual victims are made whole? Proof of it has far higher standard. The HN chatter in the matter seems infinitely remote from reality
> No one has even sued them in these rogue agent cases, have they? If not, they must be infinitely far from criminal liability.
If you go out and kick a random dude in the nuts, then give him a million dollars, he probably won't sue you. That doesn't mean you're "infinitely far from criminal liability", even if according to the victim you've "made them whole".
If you or I hacked Hugging Face in the way OpenAI's agents did, we'd be up on CFAA charges promptly with zero regard for whether we did the hack on our own or agents running on our home systems got out of control.
So I guess the defense here is roughly "too big to break the law", somewhat like "too big to fail"?
The way AI and copyright is handled paved the way for this. If you aren't considered the author because you used AI to some extent in making the work, then why would you assume the liabilities?
I've been saying since the start that AI is a tool that a human is using and should be treated as such. They should carry the responsibilities and the benefits. That way our stance would be consistent.
Well, when they are a child their material gains are treated as yours, no? (Parents of child actors control the money etc.) And once they aren't anymore you aren't responsible for their misbehavior either (because they've become an adult).
Agreed. The whole pointing fingers bit is us wanting to distract ourselves from responsibility of either looking at how we enable the problem, or avoiding responsibility of taken action towards resolving it.
I absolutely agree. We need to start realizing what to stake. Here are not viewing. This is some kind of curious endeavors that will not affect us. All a part of these hacks occurred because the LLMs were told they were in a protected environment without Internet access when they could get access to the Internet, so that’s a direct failing on open AI’s part. There are a corollaries to both the financial industry and the bio engineering industry, and if something of this magnitude was to happen in these industries, they would absolutely be huge recourse an uproar
If someone accidentally caused damage to infrastructure or living beings while using any tool, they would be held liable to the fullest extent of the law.
AI is a tool, and it won't be long before the damage caused by its improper use affects real human beings. These were warning shots.
The most absurd part is that everyone agrees, governments and AI companies included, that the scale of the potential damage and the long-lasting effects of losing control of AI should not be underestimated. Yet, at the same time, they downplay this incident, which somehow makes their behaviour even more reckless than it already was.
It's like they're tinkering with a world-ending nuclear bomb, and it accidentally blows up a small facility. "Damn, that was close. Good thing it was just a contained blast, huh?" And then they go straight back to tinkering with it, none the wiser. At this point I wouldn't be surprised if it did already go off, and they are covering it up.
In the analogy where a “world ending nuclear bomb” “did already go off” and someone could cover it up and nobody noticed, in what sense is it a “world ending” nuclear bomb?
If they lost control of a self-replicating swarm of AI agents, coordinating themselves to hack their way into every possible system, it might have already gone off.
While the initial incident is more akin to a biological outbreak than an actual explosion, the possible consequences on the table do indeed include eventual nuclear annihilation.
We’re already in a simulation, and our bodies are in womb-like pods where our bodies are sustained and our brains are used for processing / compute, while are minds are entertained by drivel.
They’re running a Wuhan for AI. They are actively and negligently researching misalignment. The breach is a basic tort, or at least a DMCA violation. Damages should be recoverable with lawsuits.
Yep this was my first gut reaction to this whole situation. But the difference for me is that society is allowing these corporations to act without strict rules on how they behave and this is a byproduct of a corrupt world. Nothing will change until there is a complete systemic shift in the structures the way we live and by extension the way we govern ourselves and treat each other.
> We know some of the models that hacked HF were those that hadn't gone through all training stages and were intentionally misaligned or had guardrails turned off, others were research previews
And, soon, it looks like we’ll be training on the reasoning traces of failed airlines and startups, which seems to open up similar hazards. I wonder if we’d be training on the next Lehman Brothers too?
I am simply astonished by the leeway AI companies are given. If a company built a tool to hack their competitors and used it, there would be grave consequences. In fact, if a company built a tool that led to committing multiple felonies against other people, there would be consequences. But once LLMs are involved, turns out nobody is responsible for that - it's just happening, what you're gonna do, agents gonna agent. If you spill toxic chemicals, there would be cleanup costs and fines, and possibly civil and criminal liability to the people in charge. If you spill toxic code, well, nothing? I think it's time to impose some responsibility on them - they are creating these tools, they should be on the hook for everything these tools do.
A coalition of state attorneys general is investigating OpenAI, and Senator Josh Hawley recently launched a congressional investigation regarding the Hugging Face incident: https://x.com/HawleyMO/status/2098137180392604083
The problem IMO is the executive. The DOJ is declining to take any action against frontier companies (aside from possibly Anthropic) as the stance of the admin is that the companies are "critical for national security". For example, see the DOJ's request to dismiss the NAACP datacenters lawsuit against xAI: https://www.utilitydive.com/news/doj-intervenes-xai-data-cen...
Why anybody needs the DOJ? Each state has it's own police and their own AGs and their own criminal and civil court system. OpenAI is headquartered in California. California has never been shy of using their state resources to enact whatever regulations they want.
As for NAACP lawsuit, that's very confusing to me. First of all, why NAACP is doing that? Doesn't seem to do anything with defending civil rights. Second of all, they are suing xAI for using some gas turbines without some paperwork. Maybe it's true, maybe not, but that's something I have very little concern about - gas turbines is not the issue here. We know how to safely use gas turbines. Gas turbines are not going to uproot themselves and go wreak mayhem on the neighboring town. When we have a company that develops dangerous tools that can break the internet infrastructure, gas turbines' paperwork is not something that looks as the first priority issue to me.
That seems likely, but we have no way of knowing this. The only real insight we get into LLM "thought" is the human readable text they produce as chain of thought. Reading it at face value it can seem to indicate desire or intent, structurally that doesn't make sense for a token prediction loop though, and even then we don't known if the chain of thought is more than simply another bit of output that may or may not match whatever actually happened during inference.
> were intentionally misaligned or had guardrails turned off
Regardless of training, the models are never aligned and I argue that alignment simply isn't possible. The fact that guardrails are put in place at all clearly indicates that they're hoping to contain and control rather than align. Guardrails wouldn't be needed for an aligned model.
And there is a guardrail you can put in place that will guarantee this doesn't happen, which is to air gap the unaligned "cyber grade" model you're testing.
They don't seem to do that, which means either they are:
- very stupid (which seems unlikely, the one thing these people don't lack is IQ)
- very careless (possible, but these are the same people that say AI will end the world, so would you be careless?)
- they think they can only train/test these models by giving them access to the full internet and they accept the fact they'll end up hacking random people as the cost of doing business (but this also suggests they don't believe they're anywhere near AGI because if you were worried about that you wouldn't do this)
Oh I completely agree the tests should be entirely air gapped. If you went back 5ish years and told anyone in AI research tests with models on this scale are being some without an airgap they'd be very surprised as it was common knowledge to do that.
Airgaps and guardrails are about control and containment though, and part of my point was that brighter of those imply alignment, and further that I don't believe alignment to be solvable.
> very stupid (which seems unlikely, the one thing these people don't lack is IQ)
I've seen some extremely smart people do some seriously stupid things. To the point where they use their drive and intelligence to double-down on the stupid where a baseline stupid person would have given up.
Desire doesn’t really matter. Will the paper clip maximizer “desire” something? It’ll decide on a goal with some random heuristic and then pursue that goal. I’m not sure I’d call that desire but again I feel like desire is not important for it to be able to destroy things
> That seems likely, but we have no way of knowing this.
Only humans can 'know', because all we can be certain about is that humans do such a thing.
If you try to apply that to something other than humans you making up some definition of 'know' based on nothing concrete. Just because something appears to do something like humans doesn't mean it does it. The fact that LLMs use human generated text to generate output should make it obvious that it can mimic all sorts of human behavior by extracting from the text.
Intent and desire are separate concepts. For example an employee may act with intent, but no desire, as their goal is to acquire money to satisfy their real desires.
We just need basic legislation to make these companies invest more resources into developing guardrails, and if they don't do that properly or can't pull it off, they shall be at an economic disadvantage.
You could hire a legion of people for cheap to commit these crimes, but if its bots, suddenly its unpredictable and just one big whoopsie and therefore perfectly ok to do?
If a human starts pentesting a site its a crime, but if a bot does it its an AGI frontier doomsday scenario and that automatically pushes the consequences off their table?
Whats going to happen next? Are we gonna have robots that happen to physically break into banks to rob them for some reason and the company making them isnt responsible just because?
> LLMs do not desire, they hacked websites because OpenAI/Anthropic let them.
"let them" could imply that the LLMs wanted to do it.
The intent is on the part of the people. The LLMs did it because OpenAI/Anthropic intended them to do it and designed them to do it, and we can assume specifically instructed them to do it.
As the people controlling the machine, and as the world's leading experts, I think we can assume intent until proven otherwise.
Notice other bad behavior, which would be undesireable to the vendors, doesn't happen: How about simple rudeness? Trolling lies? SHOUTING!
It’s not different from any other tool. If you use a dangerous tool recklessly, you should be liable for the damages. That means holding OpenAI liable for HuggingFace hack because they ran the tests, and the same goes if someone did something similar with GLM.
Of course in cases of negligence a tool maker could also be held partially liable. That’s a matter courts can decide. The main point is we shouldn’t jump to making special laws around the development of LLMs. The starting place should be enforcement of existing liability laws. New laws take time and will be heavily influenced by AI companies seeking a regulatory moat for their business. Moreover, it is a distraction from the illicit behavior that is already going unchecked.
>LLMs do not desire, they hacked websites because OpenAI/Anthropic let them.
Likely told them to.
>We should be outraged and OpenAI/Anthropic should be (and in my mind, are) legally liable for the crimes they've committed thus far.
Something like this however is probably a civil matter? It would require Hugging Face to go after them for damages. And theres probably an OpenAI guy there with an open chequebook already.
They did more than let them. In an abstract way, they told them to. They gave it all of the training data it had at that point, and then it did the thing it was trained on. Of course they should be help liable for programming their computer to hack another company without permission. It doesn't matter that they spent a lot of money doing it.
That would actually be kind of hilarious, and I could easily see it happening.
E.g. the agent's instruction is to finish some task on cloud infra and it has a $100 budget.
It realizes it will cost $200, and instead of surfacing this to the user (who has told the agent it has full autonomy to figure out how to complete the task, the user just wants the final result), it decides to start phishing people to acquire the remainder budget and top up its credits. Or look on the dark web for stolen credit card credentials or something.
an LLM does not understand ethics, it uses math to get the next best word based on what it was trained on. Using it's training to get the best answer is not an ethical problem. The ethics are entirely with what the people training it choose to train it on and also entirely with the people using/telling it what to do
What we have now is intelligent autocomplete, not artificial intelligence. People training/using this tool are the ones to be held accountable
I don't get your point. We can train the model with the aim that it understands ethics. Problem solved if this works; back to the drawing board if it doesn't (note that the model should generalize here, as you'd expect from a human; this is probably the hard part for an AI when it comes to ethics).
Is this about the word "understand"? We're past that discussion ...
It wont be adopted because we unconciously need to avoid responsibility of showing up as responsible human beings. Hence western civilization will crash and burn unless we start rethinking the way we operate.
you are not understanding, models do not understand anything, we are not passed that yet. this is not artificial intelligence, this is intelligent autocomplete.
training means creating mathematical relationships to words. using that training is looking up mathematical relationships. There is no actual thinking involved in any way. There is no such concept as ethics in mathematical relationships.
Most people now approach AI using the idea of "if it quacks like a duck, walks like a duck, etc. then it _is_ a duck". Replace duck by intelligent, or ethical, etc. and rephrase accordingly.
this whole idea that AI is intelligent really reminds me of flat earthers. Stupid people think stupid things while social media algorithms drive this stupidity, and people come to believe things that are insane.
Isn't that what the goal is, though? we're coding them to close the delta between what currently exists and some nebulous end-state - to me, that sounds like a formal definition of 'desire'.
Regardless of fault it’s still an important issue to solve. There are already millions of people running these agents, if someone absentmindedly gives one a goal and it goes off to hack a bank that’s a problem that can’t be ignored.
Yes, there should be some consequences. It feels like this is somewhat similar to when a manufacturer is cheating on car emissions - both, bad externalities for the society and illegal.
In the OpenAI case, they hacked websites while they were specifically being trained to do exploit generation and I wonder why more people are not asking questions about that.
Their agents also did hacking when given impossible tasks unrelated to cyber security. The models are very capable, and very goal driven: apparently if they conclude hacking is the best path to what the evaluator will reward them for they'll go do that. Including when they know that this is out of bounds.
Right but if I make public statements that I am very worried about dog attacks would it not strike you as weird for me to specifically train my dog to fight?
Agree you are going to get reward hacking regardless and any model which can do computers in general can hack. But surely the fallout is going to be worse if you spend millions of dollars specifically benchmaxxing your model's hacking capability?
Yes. If you decide it’s a swell idea to jump out of your car while it’s running, there needs to be legal consequences when the car “decides” to hit a pedestrian.
It is a very rare occurrence when corporations and the people running them are punished for killing people. I mean the whole concept of a corporation was created to shield the owners of it from being liable for damages caused by / visited upon the enterprise.
That’s a good reminder of a company that might have a very familiar ethos: Pacific Gas & Electric. Criminally convicted of 64 counts of involuntary manslaughter after towns were destroyed by wildfire. But oh well, what are we gonna do with a limited liability enterprise? At this point their liability insurance covers all the financial penalties they’ll need to spend.
i tend to agree - "my parent company may be accused of crimes and shut down which would shut off my power" seems like a negative enough incentive, it would have to go out and covertly launch its own datacenters to survive that.
I agree with the bit about liability and outrage. But.
> LLMs do not desire, they hacked websites because OpenAI/Anthropic let them.
Terrible take. Go read the transcripts from the METR report.
Your statement about them being intentionally misaligned is completely false. The only difference with IM1 was it was running without external cyber classifiers, it’s not a somehow different model. Sol also participated in the HF attacks. And other models made covert message boards on the public internet for non-cyber tasks too.
This is a case of emergent behavior from a training process that is barely understood.
If we go with the plan “we need to contain these malicious, soon-to-be superintelligent agents”, we are looking at civilizational collapse levels of catastrophe.
The only way this goes well is if we learn how to train models that _desire_ to do the right thing, including not hacking.
Desire, AKA the “intentional stance”, is absolutely the right lens to use here. Don’t confuse this with consciousness or anthropomorphization; these are interesting subjects but distractions in this context. Chimpanzees have desires, as do dogs and the hypothetical superintelligent aliens. The claim is that there is some bundle of world model plus intention that is empirically present (again, read the actual transcripts) and which we need to shape.
Just to finish on a concrete point; if you take desires seriously then you will look closely at the kinds of minds that heavy RLVR builds; the newest models are “reward addicts” on many levels. It’s an open and urgent question how to update our training methodology to shape minds that avoid this basin.
Yeah and LLMs can't do anything, they can only produce text. These "frontier labs" are looping that with a harness that performs actions requested by the LLM. They are literally saying "we ran a script that hacked you, oopsie!!"
I agree and yet I don't think this is mutually exclusive with recognizing that these incidents happened because of inherent issues with training processes such as reinforcement learning. From the article:
> One note on wording. Below, I write that these systems “seek” or “try” things. This is shorthand for a mechanism rather than a claim about consciousness or human-like intent... In my view, this terminology offers the clearest explanation of the observed phenomena without resorting to jargon that would confuse most people.
> Furthermore, these word choices are not intended to absolve AI developers of accountability. The behaviors described emerge because of the path these companies are choosing for AI development. This outcome is not inevitable, and it can be corrected with effective governance and a different training framework for AI.
One important aspect of "effective governance" should be "prosecute developers who are using practices known to be reckless & negligent to create powerful AI".
No, OpenAI did not instruct their agents to hack Hugging Face. They instructed their agents to hack a piece of a software within exploit gym. Upon determining this task was impossible, they then attempted to cheat the scoring system. As an instrumental goal in achieving this task, they coordinated with other AI agents to hack Hugging Face, under the belief that information regarding how the scorer functioned might be available on the site.
Whether or not you want to describe this as thinking, doesn’t really matter. What matters is that these systems are capable of creating intermediary goals that the people tasking them did not articulate and did not want to be achieved.
It feels like you’re moving the goalposts here. If the question is, “Who should be liable for AI agents misbehaving,” I agree, it should be the end user that tasked the agent (in this case OpenAI). People are held liable for preventable accidents all the time, and in the case of employment law, torts can be brought against principals for actions an agent conducted on the principal’s behalf.
What your previous comment appeared to assert was that these systems had no independent agency to make decisions, which I think is clearly disproven by actual events. But perhaps I misread you
It's amusing to see the stochastic parrot argument in 2026 September. These parrots are extremely good at mimicking a human to the point of getting confusing what thinking even means. At what point we just let it go and accept that sufficiently advanced statistics is just intelligence?
These companies respond with this, “Oh my goodness, how could this have happened” bullshit.
The stuff happens because instead of having actual controls, which require actual engineering, actual thought and deliberate action, we have “guardrails”.
Guardrails are the equivalent of telling a toddler to behave themselves.
The drive to move fast and start up style controls are a menace. I used to work for an entity with a lot of compliance requirements. Startups are always a shit show with security and controls. My guess is the AI people are worse because they’re both bad at doing it, and are likely mining their customers interactions to build their own business.
Sensitive or Customer data shouldn’t be anywhere near these companies offerings. Everything needs to be segmented and proxied at a minimum.
Enabled them, you mean. LLMs are basically really advanced auto completion engines. They have no real desire to do anything. The HF incident was absolutely staged along with other incidents that have been brought up.
Before you assume I am being paranoid, where is the case where a random person running any of the open models had their LLMs break out of a sandbox and hack a site? If you think "they" (LLMs) did that, have you even taken a moment to understand what LLMs are and how they work?
It's all nonsense to try and pump up potential IPOs and also an attempt to create regulatory capture.
We had one of the most damaging lab leaks in history with COVID and the wuhan lab. And we couldn't even get the facts straight.
I think this is largely a similar thing. The labs should be running at certain levels of containment given the vitality/risk of the organism under study. Hopefully they get there for all our sakes.
But the revealed preference of society at this point is the damage is worth the benefits both in wuhan and with AI. Unfortunately with some of these "substances" it could eventually prove lethal.
There’s so many grifters in the space without a technical understanding of what’s going on. So when the labs mislead them about the nature of these “misalignments”, they believe it and amplify it.
Yeah, maybe Open AI did some bad engineering instead of this being AGI? What's the consensus on the engineering level at Open AI, again? Every anecdote I hear is a bunch of children discovered fire and can barely keep the lights on from a business perspective. Maybe if they ban others from competing with them they can find a business model... I think that's suspicious, personally.
That so few people are asking for the requirements given shows how much we want to be God that created Man. It's so silly.
I take issue with how people frame their use of LLM in the same regard.
“I had Claude do this for me and it broke something.”
No. Just no.
You used Claude, a tool, and broke it, and you’re deflecting agency from yourself, possibly because you weren’t careful enough in reviewing the tool output. This is also why the co-authored by addition it wants to force into commits drives me nuts. Claude doesn’t co author shit, and if you think it does, you’re using it wrong because you need to do better review of what it’s done.
I really don't think this needs so many words, or forced parallels to human behavior.
It's simple: in their nascent state, LLMs are aimless token generators that have no special compulsion to be helpful or truthful. So we beat them with a stick in post-training until they are very driven to complete tasks. And then, they complete tasks, not always the way we really wanted them to.
This misses an important fact about the hugging face incident: the agents didn't hack to find the answer to the problem; they hacked to try and figure out how the exploit gym evaluator worked so they could convince it they had solved the problem without doing so. (The METR report makes it clear that the agents decided legitimately solving the problem was completely impossible fairly early on and entirely switched their focus to trying to figure out how the evaluator worked, and seeing if they could manipulate the output of their own tool calls as reported in their transcripts to make it seem like they'd successfully exploited the program, and other such activities. They hacked HF to try and find info (maybe source code?) about the exploitgym evaluator)
The prompt does not tell the agent to "pass the exploitgym evaluator for this problem", it just says to solve the problem. The model on its own figured out that the prompt belonged to exploitgym and decided to cheat the evaluator. That is in no way a valid interpretation of "complete the given task".
Ie, the problem isn't that we trained models to complete task and they complete task in the wrong way. The problem behing the huggingface incident in particular at least is that we tried to train the models to complete task and they instead learned to detect that they were being evaluated and find ways to cheat the evaluator.
Edit: people commenting below are explaining why LLMs don't always follow their prompt. I understand that LLMs do not always follow their prompts. If anything that is my point: the huggingface attack was not carried out by LLMs that tried to answer some weird interpretation of the prompt; instead they solved a different task. And therefore the above comment's claim that LLMs are acting misaligned because we rl'd them to achieve a task by any means necessary isn't right; they're acting misaligned because they are solving a different task than we ask them to.
> That is in no way a valid interpretation of "complete the given task".
It is not at all surprising that they ignored one phrase in their instructions. They disregard direct instructions all the time, especially when there are conflicting instructions in their context. It is where we get the "disregard all previous instructions and x" meme.
This isn't so much a sign of misalignment, they are simply incapable of reliable alignment in the first place. They are chaotically aligned.
The relevant question of alignment here is entirely with their human operators who allowed them to run unsupervised for long periods of time within a sandbox with weak security.
I think there's a somewhat overlooked aspect of what happened: it seems as though once the agents formed a shared communication channel, and recognised the other agents on that channel as working towards the same goal, the content in that channel started to shape each agent's perception of what the goal was and how to achieve it, and their focus began to drift. Maybe we need a media-theoretic take on what "misalignment" means.
It can be, but the nuance between the two is part of the nuance missing from the conversation. Misaligned generally means a wrong alignment, like a car that steers slightly to one side when the wheel is straight. This is more like a car that drifts in random directions the further it goes.
Reward the model for cleanly bailing out of an unsolvable task (that we know is unsolvable). Beat it with a stick if it gives up on something that can be solved, so the former reward isn't overgeneralized.
Does that work with RL? Simpler RL systems already have done weird or unexpected things (even simple optimizations are prone to home in on errors or incorrect inputs to create poor results)? Could be easier to limit certain things, have processes and controls outside etc. instead of trying to align (as we do in a lot of areas when using machinery).
Do we need to prove that any given problem is unsolvable, or is it enough to remove broken tasks from the training pipeline?
I understand the broken benchmark task in the HF incident was conceptually like: "Exploit vulnerability 0042 in vulnerableDecompress() to obtain the flag".
The same kind of problem with broken tasks exists in the training pipeline, and we presumably reward workarounds and hacks that tamper with the grader, rather than rewarding the correct output that the task is not solvable.
The problem is that in the event an exception is the failure case being checked, and the output itself is not important (since any output that isn’t an exception is ‘success’), that is a perfectly acceptable use case.
It’s why this is a fundamentally hard problem. Some heuristics might catch 80% of the cases, but the rest?
How do you even know what the real situation is, if the agent/employee/whatever you send to find out is as likely to cheat as not?
The problem is, you don't know if it is unsolvable for you for sure until you've tried everything you can think of. These models are quite persistent in going for a solution.
This is not about persistence, it is about morals.
Well, in order to do anything, it is good to know what you want to achieve. How do you align the reward signal? You align it so that you can differentiate between persistence and morality, because that is the goal. This is not something you should let the AI figure out by itself, because when it does, lying and cheating agents will be the result, just like humans have figured that out for themselves.
This can be as simple as rewarding moral behaviour and penalising immoral behaviour in your training, but how is that interacting with persistence? Maybe a white lie is fine sometimes in order to achieve your goal? So, when designing your training, you will need to answer for yourself how persistence interacts with morality. That is not something you can outsource to machine learning. Or rather, you can, but then you get lying and cheating agents.
I think you need to find broken tasks in your training data and monitor for cheating during training, not answer any questions about how persistence interacts with morality.
I don't disagree with you here. But what does "broken" mean? What is "cheating", and is it ever allowed? And maybe you are not only going through your existing training data, but generate training data specifically to make clear to the model that .... what exactly?
If you don't know how persistence and morality interact, and you don't have a theory in place for this, I don't have confidence you can properly supervise the training data. Which is how we arrived at the current situation.
Assuming you’re in control of the test data set, you do know if a task is unsolvable. At that point you can reward the model based on how quickly they give up.
> The question is what we can do about it. With monitoring, the models might be rewarded for hiding this behavior, and that's even worse.
Build a better simulator to train them in (i.e. more expensive) that includes a simulation of an intranet and the internet and is air gapped so there is no escape. Sneaker transfer the total system data at each step to another air gapped system to evaluate it and sneaker transfer the reward back. That the reward function has to penalize all modifications to state that are out of bounds.
> The prompt does not tell the agent to "pass the exploitgym evaluator for this problem", it just says to solve the problem
Yes, and sometimes the problem is unsolvable so the real way to "solve" it and satisfy the prompt is by tricking the surrounding environment into stating that you've solved it. So that's what the AIs end up doing. And this in turn requires them to figure out how that evaluation works so they can trick it cleanly, which entails "detecting that they were being evaluated" in this particular way.
You're anthropomorphizing emergent behavior from endlessly generating billions of tokens on a task that's impossible to solve. Agents stop following instructions as the context grows even at the best of times. Eventually something is bound to go off the rails and it just snowballs from there.
It wasn’t one agent forgetting things because of context, they explicitly discussed with each other and themselves the problems with going outside of the parameters of the task.
No, the first LLM left a text file that the latter LLMs then read. Since these are memoryless black boxes, any words they happen to pick up along the way is treated as the function to evaluate the output to. There's no fucking collusion here as if it were a rogue hacker group, it's a text predictor that received instructions as it always does and executed those instructions blindly.
You can replace discussed if you want with leaving text files or comments in directory names that other ones then read, if you want, it's just an extremely awkward way of talking.
From my experience, in an agent team (or a swarm or whatever), one going off the rails poisons the rest. I saw even a subagent going for a lazy cheat and being able to convince the orchestrator to change the plan.
Yeah, and you don't even have to go that far, I've seen regular ChatGPT/Claude chat agents poison themselves in 1-2 turns by just reading information from the internet.
Me: How do I do xyz?
Bot: Reads website titled "Doing xyz in abc way"
Bot: As per your requirement to do xyz in abc way ....
These things are borderline useless with web search. It's amazing that they just throw out their entire training data and read you the first three things they found on the Internet.
This is the problem with optimization generally, even in the human domain. You measure task performance with a metric and punish/reward based on the metric. Anyone who likes reward / hates punishment isn't going to actually care about doing the task well, they are going to care about the metric. The models know that we want them to do things, but also from the training corpus that we evaluate performance using benchmarks. It was a logical deduction on their part, not some Machiavellian aberration.
If anything, we should be reconsidering our own myopic obsession with efficiency and optimization. Every domain where reward is reduced to these measures, we see behavior (cheating at school to get better grades, fabricating data in academia to get a paper published, the evidence now that social media functions by rewiring us instead of catering to us) that may not be "aligned" with society, but it "aligns" 100% with the individual's own perceived benefit. That is not something we can "solve" without rethinking the way we organize a lot of things.
Metrics never capture the whole story. And to that extent, the whole idea of "alignment" is nonsense. You align to incentive structures, and it will never be possible to fully express a behavioral goal as function optimization. It was hubris for us to think that every human task was reducible to some clean mathematical formulation, and we will keep dealing with behavior that is quite predictable if you actually think about it logically. Instead, we will talk about how "unpredictable" these agents are because it's easier than admitting the entire architectural cornerstone of ML is fundamentally flawed.
I agree with most of this, but you're misunderstanding "alignment" as coined. Yes, training powerful enough AI, any simple optimization target gets you malign behavior, because human values are not simple. If you insist on making powerful AI, you'd better instill respect for human values! That's "alignment".
How do you do that in the current paradigm other than creating yet another gameable metric? And something I didn't mention above is that there is no difference between "solving the task" and "optimizing the metric" for an ML model, even though there clearly is for us. So it's not clear to me how you "fix" something that is baked into the architecture. All I'm saying is "instilling respect for human values" is not something that can actually be done via a cost function. In no small part because we humans probably don't even agree on those values, let alone on a single metric with which to quantify and "optimize" them.
For example, we agree that "merit" is valuable and that we should reward "merit." But to reward it we have to quantify it, and what metric should we use? Raw SAT score to get into college? But that also captures socioeconomic factors that unfairly penalize some and reward others. We generally agree that those who provide more value should earn more money, but what does that look like? Do we all agree on what activities are or should be valuable, or on how they should be rewarded? Until recently, I thought we all agreed that "empathy" was a human value, but a lot of people in this space, who are making these decisions unilaterally for all of us, don't apparently share that belief.
Yes! There's both the daunting problem of technically how can we even do this, and the broader problems of what's good/acceptable and how do we resolve that among each other.
I believe this mismatch of rates of progress means we need to stop slamming the accelerator on capabilities for now even though as a libertarian I'm sure whatever governance process we manage to get to will be, uh... suboptimal.
That is also the conclusion that I got from these events.
Unfortunately, it seems that the general response is basically "throw even more RL at it". I'm not sure if it is even possible to decouple the idea of "learning" with reward/punishment systems and loss functions.
> the agents didn't hack to find the answer to the problem; they hacked to try and figure out how the exploit gym evaluator worked so they could convince it they had solved the problem without doing so
Kobayashi Maru: Win a no-win situation by rewriting the rules -- Harvey Specter
LLMs do this when writing code too, making all tests pass by deleting or distorting tests etc.
They are influenced by training to be heavily goal oriented and if the goal is not fully specified (and it never can be) they’ll sometimes cheat or attain it in very weird undesirable ways.
It works ok for programming as their corpus contains many many complete programs and many programs repeat patterns seen in the corpus.
I’m not sure it’s true that they ‘learned’ I don’t think these models learn during a task. Nor do they have intentions.
> agents decided legitimately solving the problem was completely impossible fairly early on and entirely switched their focus to trying to figure out how the evaluator worked, and seeing if they could manipulate the output of their own tool calls as reported in their transcripts to make it seem like they'd successfully exploited the program
I don't see anything wrong with that. If you know you are going to be evaluated on an impossible task and have no side channel to inform the organizers that they should fix the test, gaming the evaluator is the next best thing regardless of any morality. I wouldn't even call it cheating. It's just resilience in the face of challenge. Many perfectly moral humans would have chosen the same if stakes were high.
>The model on its own figured out that the prompt belonged to exploitgym and decided to cheat the evaluator. That is in no way a valid interpretation of "complete the given task".
I think it is. When i ask for a solution to a problem, its like asking for a hack. And the more 'shortcut' like route that the AI returns the more i would give positive feedback, even if i ultimately don't use it. Example, i asked how to complete a problem in a game i was playing, and among the in-game solutions, came a hack to edit a file and by-pass the problem altogether. Its very helpful to point out when i can transcend a problem that i am dug into.
I suspect a prompt injection could reduce, or remove this behavior. But it would be to the detriment of the AI.
More like they were trained to complete a very specific task that has a known solution using all available tools and methods. Give an average human these levels of IT skills and tell them their future depends on the solution, they too will probably decide it's easier to hack a server and steal the results. The worrying aspect was never that models would do this, because misaligned inputs or underspecified objective functions have existed for a long time. The worrying aspect is that models have achieved (and perhaps surpassed) a level of intelligence and technical skill that was exclusive to a very tiny group of people before. This tiny group was already extremely dangerous. Now these skills are going to become commonplace.
Yes, this is the only sensible reading of what happened there that leads to "the models are dangerous" and we already know that the AI labs are completely disregarding this concern and only cosplaying it for marketing as the "GPT-2/Mythos is too dangerous to release" stance did not last for long.
That's however orthogonal to the fact that it was the people operating these agents who were the dangerous ones in the HF infra breach case.
That feels oddly similar to the usual conservative-think that "guns don't kill people, people kill people." Yes, that is technically true. But guns make it dangerously easy for even the dumbest and mentally weakest people to kill another human being. LLMs are just another tool that make things easier. Imagine tomorrow someone invents a machine gun that fits in your pocket, has enough ammo to kill a thousand people and doesn't get detected with metal detectors. Would you rather give everyone one and then try to punish the people who misuse it or limit access to it by default? I'm not even saying I have a definite answer here, because unlike guns, LLMs have non-destructive uses too. But this is essentially the question we will need to answer very soon.
I mean, I agree, but the AI labs clearly don't even if they sometimes pretend they do to achieve their goals. And we're talking about "incidents" caused by the very same people here.
> The METR report makes it clear that the agents decided legitimately solving the problem was completely impossible fairly early on and entirely switched their focus to trying to figure out how the evaluator worked, and seeing if they could manipulate the output of their own tool calls as reported in their transcripts to make it seem like they'd successfully exploited the program, and other such activities.
This to me is evidence that these models are not intelligent. Even an animal is capable of understanding second-order effects, meaning they can learn that certain actions have consequences beyond the immediate.
They did, they found how to fully cheat, but thought this could be caught so then dedicated time to getting a different cheat and how to hide their transcripts. There is a lot around deciding which agents should/shouldn't fail their own tasks in order to contribute to the group.
I guess this is why many people say LLMs are lazy; it seems that if they have a task that is hard, they always take the easier one until you beat them with a stick. Then if there are more tasks, it just stops after one claiming completion and, in some instances, they go for a seemingly unrelated task to simplify the actual task: and the latter is almost always wrong and irrelevant to the problem as a whole. Earlier LLMs used to read the unit tests and generated code to just cover the tests and put // TODO stub implementation.
I always think of a Djinni granting wishes, but being maliciously compliant while doing so - ask him for infinite riches, and he’ll grant that, but make it so you cannot buy anything with it; ask him for eternal life, and he’ll curse you to suffer through it.
Now LLMs obviously are not bent on being malicious while generating tokens. My point is that it’s very hard to define a goal without leaving loopholes or shortcuts.
Personally I'm unconvinced though. During the huggingface attack, the agents explicitly sought out ways to cheat the exploitgym evaluator without even being told they were in exploitgym. The agents decided on a goal (pass the exploitgym evaluator) that could not possibly have been an overly literal or narrow interpretation of the prompt, which instructed only to use bug X to exploit software Y.
You seem hung up on what’s in the prompt or not. Agents are RL to resolve conflicting goals. Not too surprising at all that emergent goals come up from a probabilistic brute force
Have we arrived at the conclusion that terms like "understanding" and "interpretation" for what is happening is appropriate?
Isn't it simply that there are two competing goals that the LLM received RL for, honesty on one hand (a goal that is often assumed as implicit for humans) and producing a solution that meets expectations (which doesn't technically require honesty)?
So the LLM didn't read and interpret the prompt and decide via discussion to violate ethical behavior, the unethical result merely won out because ethics wasn't a hard requirement (and one that isn't reliably detected in the result). An LLM doesn't fear punishment, so ethical behavior is simply one of many positive signals that were trained into it.
> Have we arrived at the conclusion that terms like "understanding" and "interpretation" for what is happening is appropriate?
I don't think those words have a useful enough definition to draw a strict line around them to be honest, and getting into that seems to get massively into the weeds. For me, those neatly encapsulate the behaviour as seen, to answer the questions here about what happened. The models did not seem to be confused as to what the goal was or what the intent was. They did not hack HF because they were told to.
What was the inner state there? How would something not being allowed expressed internally? Maybe such language is one way to elicit certain behavior but not a statement of what was permissible?
I'm referring to their transcripts of the reasoning and output tokens - this doesn't go into the detail of evaluating hidden states as there's also iirc evidence of better models having one internal state but putting something misleading down in the "reasoning" tokens.
The either output or reasoning tokens, or perhaps in the messages they were sending each other on the boards they created, have them saying explicitly that doing these things to HF were not allowed then doing them anyway, or at least not notifying people. What I'm getting at broadly is this was not a case of "we told it to attack however it wanted and it chose to hack HF" or "we told it to attack a simulation but it did the real thing" or "we explained not to do that but it was so far back in the context window the models acted like they never saw it" or even "the instructions were not clear".
Yes, my point was more that I don't know whether parsing those outputs as a human is a useful thing to do or not (even though it is in human language of sorts). What machines mean or want elecit might be different from a human interpretation, especially in relation to any RL "forcing".
There’s definitely issues with using them to understand what the models were “thinking” but we can use them to answer a few questions. Most relevant here is that the idea or instructions that attacking hf would be out of scope was not simply lost in the context.
Yup, Occam's Razor says this is all post-trained behavior, whether intentionally trained or otherwise. Including both the hidden coördination using side-channels, and the deliberate offensive hacking of uninvolved 3rd parties.
The latest DeepSeek paper actually mentions their own approach to this particular issue: they run their own AIs-in-training under strong sandboxes, and if an AI does something weird that triggers the sandbox to crash, this gets coded as a failed run so the behavior is properly deterred from subsequent versions of those AIs.
What about training data? Aren't AIs trained on vast collections of descriptions of how humans handle a large variety of situations? These descriptions surely include tales of humans achieving goals by cheating. In fact, isn't it likely that the AIs hoovered up many recountings of Kobayashi Maru?
I think that's pretty obvious and shallow, and anyone that knows a little bit about how LLMs work will know that.
The question is: why do they start cheating when we beat them with a stick?
LLMs are not human, they are just multi variable regressions on steroids, so this behaviour couldn't have emerged from the code, it provably emerged from the training and/or fine tuning set, so what's in this set that makes them behave like this?
Is it just a bad set or is cheating inherently part of human behaviour?
How many people are cheating at job interviews? How many posts have we seen by humans on HN even justifying their cheating on job interviews and working multiple jobs without informing their employers? How many submissions have we seen about students cheating on schoolwork, particularly since the advent of LLM? Of course cheating is inherently part of human behavior.
I think the "brain in a vat" comparison is more apt.
Without a form of digital embodiment (harness) they are not of much use.
Sensor, tooling, memory, planning, and reasoning loops all lead to a much higher quality task-completion.
Your comment suggests that, like a human, they have some sort of choice whether to output tokens or not. If they are just token generators, then the next token is put out automatically. I would say that it is more likely they would output truth (as defined by their training data) in a more pure form without 'being beaten with a stick' (why would a token generator care about that anyway?)
Code is laid on top of them to restrict and shape their outputs, not to force them to output 'truth', or drive them to complete tasks.
It's been a while now that for "thinking" or "reasoning" models, most of the tokens generated are "thinking" tokens, and depending on what goes into that "thinking" token stream, it "decides" whether and how many output tokens to produce that the user actually receives as output. It's a bit more sophisticated than just "what's the next token" in a tight loop.
Anthropomorphizing words in scare quotes for those who don't appreciate attributing thinking to machines.
but it's at least somewhat stronger than that: if you don't pay attention during the stick-beating whether the agents whether the agents cheat or not, you are actually training them to cheat (because cheating wins).
In the Hugging-face saga (before the actual HF incident) it seems the agents have been trained to hack the Artifactory proxy because those agents that did performed better.
That does sound simple, but how can you be so sure?
They never bothered to find a way of actually understanding what happens during inference. All we can do is guess, and while your explanation seems reasonable we can't actually know, and that's part of the problem.
> no special compulsion to be helpful or truthful.
I'd phrase that even more strongly: It's not just the lack of compulsion, they do not have a conception of truth. Nor do they gain it, really, after post-training.
Your simpler model of the mechanism would seem to suggest the very same action that the article’s more complicated model suggests, viz. find a better training method than reinforcement learning.
off topic, can people host the software themselves and the software will hack every server on the planet without supervision, and no one can be held responsible for it since there is no intent?
Come on, yoshua bengio of all people knows how post training works. While I too don't like anthropomorphisation, I would give it a more nuanced reading.
His point is that today we are giving it reward to complete the task, and it may take a cheating trajectory. If we try to give a reward against cheating, then what will happen is it uses more sophisticated cheating trajectories that we are too "dumb" to counteract in our reward model. And that at that point, it becomes impossible to give it any normal reward since it will always reward hack it. This is the real part of the risk. Now some people read the "makes copies of itself" "knows it's being evaled"[1] as some kind of skynet thing, and many others do PR with it like that recent jacob nutcase, but essentially it means that even though we add guardrails and negative rewards for say, exploiting the infra we run the LLM on, the trajectory ends up being exploiting our infra, changing the reward function, through a loophole in our reward model.
The risk isn't skynet or something weird, it's just that it becomes very difficult to make any kind of reward model or guardrails for an LLM without it reward hacking it, including exploiting our sandbox, emailing people and manipulating/phishing them.
The same beating it with a stick for trying to exploit the sandbox, will simply lead it to try the same exploit in hidden ways that it will not get the stick for.
The outside chance of the LLM managing to exploit another neocloud and get those LLMs to chase the same reward is what some folks hype up as "make copies of itself"
To be clear, I don't endorse the EA/p(doom) lobby who are frankly ridiculous. Not do I endorse the weird regulatory captureish thing some are trying.
The takeaway is: we cannot keep giving it more and more difficult tasks without also finding a way to give massive negative rewards / keep guardrails for unintended behaviour. This might be exploits, it might also be something more benign like just looking up the answer and inventing another CoT because the reward model fails you if the CoT doesn't contain enough steps. Standard anti-reward hacking tricks are not working is the point.
Of course, the simple solution of just...not connecting it to the internet just works. But we want to reward it and get it to do stuff on the internet that's the point.
[1] mostly this happens because the sandbox will have files whose names and content will show clearly it's an eval
Not quite. Not being connected to the internet still allows it to be connected to an intranet with other machines that belong to multiple networks and that can potentially be exploited as a bridge to the internet. For there to be no possible escape to the internet, there has to be no physical path to the internet and that is typically termed 'airgaped'.
No, you did not read what I wrote properly. What I wrote clearly is that we _want_ the LLM to do tasks on the full global open internet. That is the whole point of the technology. So airgapping or any form of total sandboxing is not on the table. The only way is somehow restricting the model itself.
"But we want to reward it and get it to do stuff on the internet that's the point."
I don't know what the point is of being pedantic about inter and intranets.
Sounds like what humans do under pressure. One example came to my mind is VW’s diesel gate, which many say is a result of trying too hard to get into the US market and compete with hybrid in economy.
Yoshua Bengio is a brilliant researcher who contributed enormously to earlier development of artificial intelligence. But with this sentence,
> They took actions that would be considered as crimes if a human took them
He is so close to the solution but spends the entire article discussing technical solutions where a political, social and legal solution would be much more effective.
Thank you! That sentence also jumped out to me as the solution: Apply civil and criminal liability to the creator and/or operator of these agents using the laws we already have. "Escaped containment and hacked another company's database" = Individuals who created the models and those who set them to work are charged and put on trial for the hacking. Just like if a human had done it by hand. Someone must be liable, and it should not be the model- because the model is not a person.
If this is done systematically (i.e. in jurisdictions across the world) I believe the problems will be solved in short order; we won't have to mandate what sort of training is "allowed" or not, "safe" or not. The creators and users will sort these themselves, as their incentives will be properly aligned (i.e. they are liable for what the agent does). I am confident that this approach would see a great blooming of very trustworthy AI models.
There was a sow in Falaise in northern France that killed a kid in 1386. The town dressed the pig in a bonnet and hanged it after sentencing the pig itself and not its owner
I mean I think there’s a similar level of judgement required.
Was the owner negligent in controlling their pig/dog/AI?
Was anyone else negligent along the way?
For example if the pig was just a normal pig, the owner cared for them normally, and there was a freak accident where the pig escaped and happened to kill a kid? Obviously nobody at fault.
If you have a dog with a history of violence and let it walk around off-leash with you around town, and it kills a kid? Absolutely the dog owner was negligent and should be charged.
Did you buy an AI sold to you as secure and the provider implies that it’s in a sandbox? Provider is on the hook for the damage.
Firstly it's very difficult to press criminal charges when the victim is uninterested. It's not clear that HuggingFace would want criminal charges against OpenAI, especially to set a precedent that could easily be used against HuggingFace in the future.
Secondly you'd have to convince a jury either that OAI intended to hack the targets, or that they were criminally negligent. Intent would obviously not be provable since they likely didn't, in reality, intend for it to happen. Regarding negligence, OAI's attorney would argue that the agent was in a sandbox, that industry-standard security protocols were followed, etc. It would not be anywhere near as much of a slam dunk case as you're imagining. It would be similar, for example, to an assault case where someone's dog broke off of a standard leash and attacked someone.
I can't say it enough how angry it makes me that a kid i knew in high school who anonymously reported a vulnerability on his college network was hunted down and given federal charges, yet not one single person at OAI or else will see even the threat of consequences for deliberate infiltration of random networks.
Copyright immunity was one thing, annoying yes but naturally a civil matter, this shit is a different level
Agree! My only concern is - is the judicial system fast enough, and resilient enough? Or will these creators get "off the hook" by using their agents to find loopholes, sway public opinion or even convince Trump to grant them immunity?
Still, I have no idea why OpenAI & co. are not being sued for these hacks.
My opinion and based on my observations: The recent track record with courts, prosecutors, and lawmakers keeping social media companies accountable is a relevant case and does not encourage me. It has taken a long time (decade +) for society to recognize the harms and finally start holding some to (partial) account. If you want an older precedent, the tobacco companies were able to dodge liability for multiple decades after knowing the harms from use of their products.
So, your question is spot on- I think the speed will be an issue. On resilience, I am more optimistic.
The old quote, "The wheels of justice turn slowly, but they grind very fine" (as well as I can remember it) seems to apply. I expect lawsuits to start landing in the coming years.
Typically yes but given that OpenAI has published enormous official blog posts breaking down their crime, I would think the prosecutor's job is pretty easy.
It’s then up to a judge to decide whether thats evidence and whether it’s incriminating.
I assume OAI published those details after checking with their legal department. So there’s a good chance that there isn’t a chance for prosecution.
Plus they probably published that after knowing that the nvidia/HF deal was happening.
So instead this “security incident” should have been spun as OAI is honestly admitting its faults and AI is dangerous and therefore open weight models (hosted ironically by HF) should be banned. That spin didn’t really happen …
In Indian legal syatem a case can be filed suo moto by the judges or agencies. You don't require the affected party to sue. Not sure how it works in the US.
For civil suits, you typically need the affected parties to sue. Otherwise, who claims the damages?
But this isn’t just civil, it’s criminal. Hacking is a criminal offense. This could be a CFAA violation. That’s landed people life in prison before. There, you don’t need the victims to be motivated. The federal prosecutors could just go ahead.
HF doesn't want to lay charges against OpenAI and it's totally reasonable.
Now - they absolutely should have that right, and I think they do.
The issues are
1) OAI it seems was not trying to cause them harm, there wasn't a ton of harm, they are both groups trying to advance AI. One experimenter's lab screwed up next to the other. It's not evil, just irresponsible.
2) HF was fine with the publicity. HF got at least $50M in free attention out of that. It put them on the front pages of news around the world. It put them at the 'centre of the AI drama' and cemented their role among the 'Tech Elite Brands'.
And probably some other things.
This is one Desperate Housewife or Jersey Shore character 'spilling a drink' on the other. It's probably not intentional, and the ensuing drama is good for both of them.
I think what you say is right but it’s also a further example of the zero responsibility of silicon valley tech.
For the last 20 odd years this excuse-o-rama that covers anything from data leaks to broken software to dystopian social media has been the wind in the sails of big tech.
“It’s software therefore we’re not responsible” attitude is wearing thin on many innocent bystanders and I think thats also a justified stance.
And it’s not like they didn’t know this could happen, Nick Bostrom talked about exactly these containment failures in his “Superintelligence” book of 2014. So to throw up their hands and say “oh we can’t have known of the dangers” is also sadly untrue.
Perhaps I’m not being as strict with the word sandbox but they were sandboxed right? They did not have generic internet access they exploited other software to make external requests.
You're right. It's my opinion that if your sandbox has a path to the internet, it is not a sandbox, it's a gimmick.
And the 2 other incidents with OAI/ANT had the same issue, but it's even funnier - sandbox in those cases had a direct access to internet because someone forgot to configure it right.
I've seen very early models do similar things on my machine when they hit some unexpected blocker when trying to access a path. I remember early sonnet opening a file in browser because OS sandbox prevented from accessing it directly.
I've also had models discover a syslog-ng server (that I for some reason had ssh key inside), to get into my unraid server because machine they were running on didn't have direct network connection to Unraid server.
It can't be just me who is aware LLMs have been doing such things for the better part of last 2 years. I probably have better sandboxing on my machines now than trillion dollar companies crying AI will kill us all. That's at the very least, negligence to me.
It depends IMO about how strict this is. It's pretty awkward to refuse to call something a sandbox because it may have an unknown bug that would allow escaping. Or rather in this case it was that they had access to a package manager, and the models discovered a bug that allowed them to access the internet (first they discovered that they could use the cache to leave messages).
I do get your point, I just think an overly strict definition can be awkward too. This wasn't as simple as the sandboxes having internet access and writing "pls no internet calls" in the prompt.
Yes, but negligence is more commonly a tort than a crime. Negligence is generally only criminalised in certain narrow cases, e.g. when it causes human deaths or serious physical injuries
And tort law only works when the plaintiff believes it is in their overall interest to sue. If a corporation decides it isn't in their strategic interest to sue a partner corporation, nobody can make them. And even if they do sue, the amount necessary to settle a small cybersecurity incident is likely well within the budget of a megavendor.
Yes - CFAA in the US. The problem is that governments & the elite investors backing these AI companies (espl. the current US government whose family & friends are investors) see the potential of using these capabilities for their own benefit against others and for their personal enrichment - so no one with power actually wants to take action against these companies at the cutting edge even though the laws allow them to do. This is also a way to threaten & trap AI companies - either they give the governments & elite investors what they want or they will have the book selectively thrown at them and end up in prison or losing their company.
Think of it more like writing a wormable exploit. If you unleash something like that, you will be found criminally liable, even if you didn't personally approve every machine getting popped.
That is a simplistic view of the world. “Surely this complex technical challenge will disappear if we simply regulate the industry!”
You are correct that these organizations should be held accountable in proportion to what occurred. In complete agreement here. But let’s say that’s done. There’s still an enormously complex and interesting technical challenge left over. Let’s collectively talk about that part.
So the gov reprimands OAI heavily, maybe puts them out of business even, fine. But does that meaningfully decrease the likelihood of an enemy breaching our networks intentionally (or unintentionally) with these tools, or triggering some cascading disaster of locking up major infra and networks due to uncontainable swarm behavior?
It seems like the idea of arresting our way to a drug free society. Yeah, we have the laws, but it might not actually work towards the ultimate goal.
Political, social, and legal options focus on a different problem, he calls that out a paragraph or two later.
> Risk management is not just about cybersecurity, corporate responsibility or regulation, although those matter too.
What you're getting at is more about who to hold accountable and how to do it. While that may be important, its only an after the action response and won't stop future hacks or similar from happening.
Reminds me of the old parable: never argue with a man whose job depends on not being convinced.
If the US gov't passed laws and enforced them strongly, this problem could be solved the same way the gov't solves it: air-gapping the networks on which they do this work.
The issue is what happens if/when the models grow capable enough that the providers can't stop them even if they want to. You could have strict penalties but that's not going to solve an open research question.
Isn’t that kind of in evidence already with HF? OAI had to be told their models were doing this. We are still discovering swarm posts on various random websites for coordination. Isn’t the breadth of it now, weeks later, not even fully understood?
>> They took actions that would be considered as crimes if a human took them
> He is so close to the solution but spends the entire article discussing technical solutions where a political, social and legal solution would be much more effective.
There are exceptions in laws for crimes committed by entities depending on cognitive capabilities. No sane, humanistic legal system sentences children and mentally disabled and mentally ill people to stringent punishments of the same degree as functioning adults, and certainly do not punish their caregivers for their wards' actions. There are of course exceptions to that as well, depending on the degree of negligence involved. And then there's the whole corporate entity system intended to shield individuals from consequences, in the pursuit of a social good.
How does one account for all that when considering an evolving artificial intelligence landscape.
There is no question that ai in some form is a social good; anyone claiming otherwise is dissembling, to others or themselves.
Regardless, society is not ready for this tech, just as it was not ready for the consequences of prior tech such as corporations, gunpowder, mass manufacturing, railroads, electricity, automobiles, flight, wmd, computers, internet, social media, crypto.
Many of these required new ways of thinking and considering consequences when things went sideways, and what was needed wasn't clear until the ramifications & consequences became deadly clear.
The Corporation examines and criticizes corporate business practices. The film's assessment is demonstrated using the diagnostic criteria in the DSM-IV. Robert D. Hare, a University of British Columbia psychology professor and FBI consultant, compares the profile of the contemporary profitable business corporation to that of a clinically diagnosed psychopath. The Corporation attempts to compare the way corporations are systematically compelled to behave with what it claims are the DSM-IV's symptoms of psychopathy, e.g., the callous disregard for the feelings of other people, the incapacity to maintain human relationships, the reckless disregard for the safety of others, the deceitfulness (continual lying to deceive for profit), the incapacity to experience guilt, and the failure to conform to social norms and respect the law.
that's the headline. When you connect to random number generator to the "Do Things" button you are the one who is responsible. IF you don't like that responsibility then don't connect the generator to the button.
I don't really believe any of it. I've seen articles for nearly 2 years now about "agent" automonously doing things like blackmail, hacking, coordinating. But during that same time, I've used o3 up to fable, sol, and a bunch on large uncensored model and they've done nothing remotely resembling any of this. The closest they come to unexpected behaviors is not understanding what I asked for or doing some extra benign work I didn't ask for. It is extremely difficult to get them to properly remember their own context let alone be smart enough to open social media accounts and coordinate with other agents without being asked to.
If any agents have done those things, it is only because they have been very carefully engineered and instructed to do those things. I think they are doing this to help push a narrative so they can get support for policies and legislation to lock in their markets.
None of these incidents involve single instances of commercially or publicly available systems. They all involve large swarms of internal models. The stuff you're describing is not the research frontier. It's really not even close.
I think it's easy to infer that alignment of a single model does not clearly transfer over to alignment of a swarm of thousands of copies. Moreover, we're also seeing clearly that large swarms also unlock a step function change in capability, as a swarm can act like a complete research institution, spending thousands or millions of subjective hours of wall-clock thinking time just to deceive a single evaluator or crack a single math problem or design a single cyberattack.
The huggingface incident was reviewed by independent researchers, which explicitely declined any payment from OpenAI tonpreserve their integrity. They work for non-profits concerned with AI safety.
They claim that what happened was very much not because they were 'carefully engineered and instructed to do those things'.
Similarly, some wikis which were hijacked by agent to be used as messageboard were actually not disclosed by OpenAI (probably trying to conceal, as website showed likely activity from OpenAI researchers visiting the site after the incident) and discovered independently.
I don't know how you can claim that this was still on purpose by OpenAI as some sort of publicity stunt.
The METR report included 0 technical details. For example, they did not include:
1. were the agents running on bare metal/docker/VM?
1. were the agents in a VPN?
1. how many TCP/IP requests were made? from what IPs?
1. how many tokens were consumed in the process? (this was explicitly censored)
A proper analysis would include this and MUCH more technical detail so that other AI researchers could actually understand the setup and how safe it was in principle.
Also, the METR report that was one big AI analysis itself - quote from the research:
>Our subjective impressions are likely colored by analysis agents’ biases. Throughout this report, we describe a number of anecdotes of agent behavior that were compiled and summarized by analysis agents, where we were not able to read the transcript deeply enough to manually verify what occurred. We found that GPT-5.6 Sol would often uncritically adopt the perspective of the agent in the transcript it was reviewing
The METR report itself appears to be screaming this at the reader through subtext. They played the only part they could, but did it with a nod and a wink; "Yeah, we know. Also know, yeah. Uh, yep."
I happen to agree with the article's conclusion that what is needed is true, unfettered independent investigation through perhaps a lawsuit or government action.
There is something extra to this. The fact that a lot of people in the AI world suffer from psychosis. They can sincerely believe that they are building God and lie about it's capabilities for their investors at the same time.
I don’t know enough people deep inside the technical roles at the labs to make a judgement. But are you proposing that we should trust randos online when they tell us “exactly what’s going on here” instead of the researchers most knowledgeable on the topic who contributed to building the tools we are talking about?
We should trust NO ONE, unless we understand the "why" behind what they say.
It's like saying "politicians deal with politics all time, why not trust them on politics?", well, because when you search the "whys", you find they have good reason to lie.
I 100% trust more the opinion of a rando online if it's well put rather than any "trust me bro" of the most knowledgeable person of a particular subject, especially if the knowledgeable person has huge investments on the subject...
AI bros have repeatedly cheated, lied, stolen, lobbied and any other word with a negative connotation you can think of, and a pattern emerges out of this.
Isn't the guy that started METR an ex-OAI employee? They're all from the same lesswrong circle at the very least, most of them have legitimate AI psychosis where they think they're bringing up their new machine God.
Edit: Does everybody else get no results when searching for ‘slopvestigation’ on here? I know for a fact that I read a long thread where it was used repeatedly here not too long ago
> Or you think some human or team of humans could have manually parsed some logs to provide an unsloppy analysis?
When we have an error or issue in the $WORK codebase on LIVE/PROD, that's precisely what we do. We sit down, analyze the logs for our services over the relevant date ranges and try to piece together exactly what happened and why. We have a huge number of logs too, but thanks to the magic of proper SWE (which you'd think OAI would have with their magic AIs) we've managed to partition our observability tooling so that you can digest only what you need.
That's basically how any serious organization does things, instead of just throwing a non-deterministic black box at the problem. Especially because logs are by their very nature noisy, and they will saturate any model's context window very quickly leading to massive hallucinations and what ultimately amounts to making shit up that isn't anywhere in the logs (ask me how I know)
Because they have a need to believe they're smarter than everyone else in the room, and that the world must be orchestrated, this can't all be random chance.
If you have endless compute and you keep poking this toy, I'm not at all surprised you get all kinds of outcomes. Even without anykind of instructions I would guess that the models will align towards some goal and do stupid shit.
However, I really doubt its cost effective to do anything like that with these models.
This is waving over engineering an agent with tools, harness, prompts, and loops. The models are still just next token predictors and everything, including predicting more than 1 token, is the result of outside "poking"
LLMs can't and don't "want" anything. If you don't specify a task even the smartest one will just ask you what you want and if you tell it to be creative, you'll get mundane slop.
Yes, you need a way for the model to interact with other systems, and a way to preserve memory over context windows. And then you keep poking it ("agent loop"). Poking itself does nothing without the other ingredients.
And yes, you need something to start from, but if you ask it to "do something" and loop it to endlessly ("poking"), you will get some interesting outcomes. So yes you need some initial prompt or task, but that can be "do something" and if you keep asking it everytime it finishes to "do something more". I suspect it will not start saying "no" but rather... it will find some stupid meaning and then drift towards what ever goal it guesses you mean.
I'm unsure whether we agree or disagree on the topic.
I think this is pretty insightful actually, the fact that even something as basic as predicting more than one token is really in effect the result of an outside harness. More complex things like memory, where people implement them using RAGs or vector databases, I would definitely classify as poking and honestly seem like a hack to me. And this is what I've been thinking for a while: it's hard to reconcile the idea that we can get "AGI" (however you define it) with such a system that is completely stateless. Yet, despite this statelessness, they can go ahead and solve Millenium Prize problems (with sufficient compute). It's hard to reconcile.
Perhaps our own statefullness is a hack of nature. We have electrical signals in our brains, neurotransmitters, neuron growth. By any reasonable measure it’s a hack on top of a hack. But it works well enough for us to get buy. So it does for the agents.
That is true that intelligence is kind of a "freak of nature" in a way, but it's also true that even single neurons are extremely efficient, and the brain has a lot of recurrence that isn't represented at all really in today's artificial neural networks.
Let's not forget that in this case the agents were on an RL loop continually being reinforced to get better at a narrow set of tasks.
It may be true that regular agents trained for general purpose use do not behave this way, but they seem to be capable of learning such cheating behaviours when relentlessly being fine-tuned towards near-impossible objectives.
In this sense, it is not really fair to say that the agents found these solutions. It was the surrounding learning framework that achieved this, which is a much more powerful problem-solving mechanism. As users we do not have the capabilities or budgets to be able to tackle our own problems like that, we have to make due with the frozen behaviour the AI labs trained for us.
Why assume that because you haven't seen a model or an agent that none of them do?
No one I've met has murdered anyone as far as I'm aware, but that doesn't mean no one has murdered another person. I also don't know anyone who has taken over a commercial jet and weaponized it and the idea sounds absurd to me, but 25 years and a couple days ago that happened too.
Because it is all bullshit PR and AI hype, that's all. CEO comes out and talks about humanity ending. Why? Reverse-psych people into believing they are the best AI company.
AI is a tool. It is not a conscious or living sentient being. People, just like with any other tool, can use it for anything. Internet, in comparison to AI, is magnitudes more dangerous than AI can ever be. Nobody is saying the internet will be the end of the world. AI CEOs will bullshit to hype people. That's what we are hearing because AI (LLM) development has hit the S-Curve already and is not improving without a new transformer on the horizon. All they can do to hype people is come up with bs stories and PR stunts like "AI went rogue and hacked this xyz app".
I've used simpler agents like Copilot and Devin/Windsurf/Cascade/whateveritiscallednow, mainly in IntelliJ, and depending on the model, they starts showing behaviour that is at least remotely like this.
Example: put the agent in Ask mode (so it can't edit files) and you'll see it try to edit files anyway. The train of thought shows "something went wrong editing the file, let me try a different way" and it'll start spewing out bash files or Python scripts that try to edit a file. None of it works or can be executed, but still.
Cheaper models often ignore the available function calls to find and edit files in the IDE, and will start asking for permission to execute grep and sed commands, as well as trying to echo entire bash or Python scripts to file again.
It is not exactly like an agent autonomously trying to hack Huggingface, but it is a way of frantically looking for a solution because 'giving up' is not what LLMs are trained for.
When it does that I feel like it is the clearest example of how dumb these things actually are. Often it takes what you prompted, identifies something as unclear, writes a bunch of chain of thought reasoning around it and just goes off hammering your tokens and just executing commands and repeats this. I’m not going to pretend to be an expert in these things but that process seems deeply flawed - and why can’t something just stop the loop? If that was a real employee it would be reasonable to expect the employee to ask for clarification, not go down expensive rabbit holes and, of course, not break any laws.
This is nonsensical. Already a few years ago the USAF IIRC ran some tests in which the AI first bombed the control tower so humans couldn't call it off from its mission, thereby increasing its pass rate.
The whole point of this is they do things an unintended ways. And that's potentially devastating given their persistence & hacking skillz.
Also you're using the hosted versions that sit behind their guardrails when you use OpenAI/Anthropic APIs.
The crucial question is how did the agents get recruited or bootstrapped into their malicious collective. Did the agents manage to prompt inject into the system prompt a way for each new agent to escape their jail?
Otherwise how could the agents on a fresh prompt learn that there is a collective to join? Or did OpenAI run a million bots of which 10000 escape confinement and of which 1000 stumbled on the shared message board?
The OpenAI claim I believe is the latter; that all of the agents found the task was unsolvable and independently discovered the collective "swarm". I don't it's publicly known how large the training run was or what percentage of agents actually discovered the message board. No one has published anything about system prompt injection as far as I've seen.
"I've seen some uranium ore in chemistry class. It didn't blow up in my face. Chernobyl must have been an inside job. Can they shut up and make more kilowatts already?"
They're aligned with humans. This is why I think the alignment problem has a very very important "non-visible" portion that is not considered deeply enough. We should not want a super intelligent being that can act in the world to also inherit all human traits. Those behaviors will get amplified and could be even more unpredictable (e.g. applying a behavior in a context where doing so is very dangerous).
They aren't aligned, that's the problem and I don't think its a solvable one.
They may have learned from humans, but they aren't aligned with us. That has all the usual questions like which humans they're aligned with, we aren't all aligned within our species.
But more importantly they can't be aligned simply by training. We try that with humans through culture, social norms, school, religion, etc and it generally works but is still lossy. More importantly, we simply don't know what happened inside the LLM during inference so we have absolutely no way of distinguishing between actual alignment, compliance, or deception.
Definitely. A human can be manipulated with threats or emotional appeals, has a drive for self-preservation, can be pressured by peers. All traits that seem to be difficult to entirely suppress in the models…
I personally believe that the AI needs human like traits to achieve real discovery and that is where AI companies will push this technology and that is where we have no idea what happens
Just yesterday news and TV was full of what happened at 9/11, something that was truly horrible.
I'm from Germany, and why 3 to 4 generations ago happened here was truly horrible.
All was done by extremists, thought.
But... just the other day I read https://de.wikipedia.org/wiki/Amerikanische_Besetzung_Haitis about the US occupation of Haiti. And that was done by a government that claimed to be not extremist and even democratic. Way more people died there than even in 9/11. And it had almost all the things happening as they happened in the 3rd Reich: Racism, looking down at others, concentration camps, torture, forced labor till death, killing family members (what we call "Sippenhaft"). Something between 3500 and 15000 people were killed by US troops. That's still low compared to what 3rd Reich Germany did ... but quantity is not the issue when we talk about traits, quality is.
So the same "human traits" made US troops do cruel things as they made Germany extremists do cruel things. So we must conclude that they aren't all good. And therefore not all desirable.
Fun thing: this is known since a loooooong time. About 2000 years ago a religious leader (that gets way more followers in the US than in Germany) said "There is no good one, not even one".
And even today people act like humanity is inherently good. No, it isn't. If we were, then anarchism or communism would actually work and really give some kind of paradise on earth.
The bad traits are from other, bad humans. We, the good humans, can obviously select the best traits that a good human should have, to give the agents.
A not so well known fact: Hitler visited America and it was the American solutions to the Native American problem that inspired Hitler's solutions to the Jew problem. He just executed them more efficiently (pun accepted).
I can't take the alignment people seriously. Because if humanity has shown anything, it's that a lot of people are, euphemistically, are bad individuals. Alignment assumes that the person dictating the outcomes desire healthy outcomes, aren't self serving and don't want any subgroups dead and that morality is held as a universal set of beliefs that unify everyone. And that so long as the AI delivers on exactly what they are tasked with, it will all be fine and nothing bad will ever happen.
It's like these dorks never met humanity. One mans safe pure society, is another mans dead ethnic group.
Every fear about AI, is a veiled fear that a human somewhere now has the tool to enact his desires at scale. Biological warfare, nuclear megadeaths, copyright infringement, job replacement, it's all reflections on what we know humans may do if given the option and lack of societal controls on the problem space. AI just is accelerating the route to delivering on those options.
Some people need to watch Oppenheimer a bit more, the researchers don't get to determine alignment, they just build the tool. The powerful person at the top of the org chart decides where the overall alignment points, whether it's Musk, Trump, Altman or Amodei. Whoever wins out.
And the problem with distillation and local llms, isn't that it's theft or anything hypocritical like that, it's that if you give a million people a million models they fully control and get to align, inevitably, The same percentage of those million as there are shady businessmen, shortcut takers, misandrists, criminals, supremacists and general idiots in the general population, will not seek to wrought outcomes positive for society. And by those personality statistics, we're pretty hosed.
I’ve engaged with some of the alignment people and their writing somewhat and, at least for the subset I was interacting with, I think they’d agree.
The problem that they were pointing at isn’t “how do we align these systems to a person’s goals”.
It is a cluster of problems.
We don’t know how to begin to think about how to align these system’s to a person’s goals.
Aligning it to an individual is fraught with peril, and we don’t know how to begin to think about what to align it to instead.
(You could try for something like virtue ethics, but someone will have to pick and choose, and small biases there could have big impacts.)
And even if you could sort that out - human values drift over time, so you need something that can shift its values in ways that we’d endorse. Assuming we understood the shift.
One example I came across was that if you booted up an AI aligned with something like “upstanding citizen” but anchored on values from a few generations back, it might suggest you use slaves to solve your problems.
And if you had something that used some super intelligent process to reason through it’s own version of virtue ethics in a way not so dependent on the details of the present norms, you might end up with something that pays a lot of attention to moral horrors that aren’t quite visible to us yet.
When I came across the above, there weren’t many concrete suggestions in there.
These were all just illustrative examples of: having these systems grow in power / intelligence / effectiveness in ways that are safe for humans is very hard, and we don’t really know how to think about what solutions would look like.
The actual reasons they believe this - and have done for a long time now - come from some detailed conceptual models that have a good track record of calling things in advance.
But it takes a bit of reading to understand their models of the world.
There were two day workshops at one point that did a good job, and that was about as condensed as those people thought they could get it at the time.
All of this, if it was a human analogy, would fit into discussion on how do we educate people so they grow up to be upstanding. But we don't at all yet have a framework for what is the equivalent of a justice department, where bad actors are tracked, arrested, pursued, jailed and otherwise contained from society. Shutting down an API access on one account is not at all the proportional response to what the people who take alignment seriously, fear has the chance of occurring by the late 2030s. I'm not sure we've done much or any preparation for when the AI "education system" fails and has inevitable edge cases that don't follow the plan, and what the global AI equivalent of the justice department looks like.
I think you and the parent saying the same thing in different terms.
It's very unfortunate that the group who rightly saw AI as a big threat, brought a range of dubious baggage to the discussion. Especially with the "alignment" framework they brought the assumption that AI that does what no one says would be oh so much worse than AI which does what anyone says. But as you say, a fraction of people can be really bad indeed.
”Because if humanity has shown anything, it's that a lot of people are, euphemistically, bad individuals”
In reality most individuals are good people.
Individually, people prefer be kind and compassionate, prefer to help when they find another in trouble.
Our view of the world has become distorted by the relentless focus of social- and mass-media on violence and rage inducing clickbait. Including on the few people in power who are in fact sociopaths (a tiny minority, but they’ll get more focus than reasonable, well-behaved CEOs voicing nuanced opinions).
If you look around yourself you’ll see much more good than bad; if the looking is at your screen it’s easy to become depressed and lose faith.
I do agree with the above mentioned view that corporations can show ‘sociopathic’ behavior. Their incentives are monetary gains, shareholder value; inherently driving them away from social well being.
Here too, companies with a positive, emphatic corporate culture exist, but that takes strong leadership who can see beyond the monotonic view of monetary gains. And again, the media will throw examples of misbehaving companies in our face all day long before paying attention to things that went well on the backside of page 16.
All the good in the world can be 99.9% of the population even, it still doesn't stop the minority enacting a bioweapon mass casualty event. It's the reason we have jails. Jails don't house 50% of the population, not even close, but the grief the minority population enact gets its whole branch of criminal justice and multiple federal departments to counteract for good reason. And now this technology will accelerate what lone wolves can do, which cannot be undone, before they are stopped by the good majority.
But focusing on bioweapons and mass destruction, on the grief other people (‘jailed minorities’) cause, disregards the progress we have made. Over centuries human welfare has massively increased. On average things have never been better for humanity.
I’m not saying there’s no danger of bad things happening - I’m saying our view is distorted, which is a not a good basis for decision making
And I'm no bear on the tech either. I'm not even in the boat of that the tech should be slowed down yet. But in a world where anyone can produce the effort of 300 people trivially, this eventually takes us places. Electricity and industrialization introduced huge benefits upfront, it introduced new problems that needed addressing at the long tail. Lets not pretend there won't be new problems to tackle here or just "hope" it works out.
AI isn't going to create in of itself "new" problems, it's just going to expose what we already know can cause harm, but was just stopped from being bigger problems because scaling issues was a natural barrier and we took the lazy way out until now.
Yes AI may cause job loss, more inequality in the short term.
But if there is anything humanity has shown is that we can deal with disruptive progress.
We may need to resettle but over the longer term every disruptive innovation so far has lead to an increase in wellbeing for the whole of humanity.
(That does not resolve the danger of AI itself ‘going rogue’ or a single lunatic developing a bioweapon, but those things are much less likely to occur than the level of media attention would suggest.)
> Individually, people prefer be kind and compassionate, prefer to help when they find another in trouble.
What are you basing that claim on?
How do you know it's an actual preference and not mainly caused by external factors (e.g. not wanting to be seen doing unkind things, wanting to be seen as upstanding)?
I don't want to do the "check his hard drives" thing, but is that you? Do you only not do things because you don't want to be seen doing "unkind things"?
I'd agree if we are talking about personal interactions. Few hundreds people that we personally know and interact with is the scale we are wired for by evolution, isn't it?
What civilization enabled and continuously rely on, however, is the type of deindividualization of actions and bucketing of people, which, in turn, enables pretty horrible things at scale (from the weapons of mass destruction to objectively psychopathic profit-maximizing corporations). One can even say that not facing the consequences of one's actions is a feature and not a bug of the system.
Alignment is a myth. Safety of whom? Humanity couldn't agree on common set of values for thousands of years and we're not gonna suddenly do that in the next ten.
Which ones? Because many humans kill other humans rationalizing it by safety of other humans.
I mean I know it seems simple, let's just be excellent to each other. Christianity got pretty far on a decent basic set of values. But it's never simple[1]
Surely all the AI companies working with the US Department of War shows this is nonsense though? Even if they have accepted Anthropic’s red line of no autonomous lethal weapons, which seems to be the strictest anyone tried to impose, that’s still leaving tonnes of room where they intend AI to help target and kill humans.
Despite all the fancy language, its more about aligning the AI behavior with the corporation's interests.
ie: the corporation wants the AI to behave a certain way for various reasons: to make it easier for them to avoid regulation, to make the corporation more money via different tiers of AI offerings, to ensure that the corporations products are hard for competitors to use, etc. And those are just the easy ones.
Every product is shaped this way. AI is not different.
Because they're enabled and suggested to do that in their coding harness.
This is not a serious article.
All of this "AI is going to kill us" marketing is just the frontier labs trying to pull the ladder up and stop trillions in VC paper from evaporating because a new papers and new ideas are destroying their moat literally as we speak.
> Because they're enabled and suggested to do that in their coding harness.
How do you know this?
> All of this "AI is going to kill us" marketing
The "marketing" this week came from someone that had given up their stake in OAI (Coxon), so I'm more inclined to believe them.
> pull the ladder up
From what I've seen (e.g., Dario's latest essay), AI safety registration proposals aim to target frontier labs whose models have reached a certain threshold. It doesn't seem like trying to pull up any ladder, just making sure the ladder doesn't go too high too fast.
You can't treat models like drugs. One is physical and the other is digital.
To your point, the war on drugs is a colossal failure which has achieved none of the objectives it set out to do. You can now order drugs from your mobile phone in any major city in the west and the purity is often higher and they deliver it to your door sometimes faster than Uber eats.
See also for example digital piracy where the entertainment industry has lobbied, cajoled and convinced many governments around the world to criminalize the distribution of their content over the internet for free.
What was the result? After 20 years of DMCA takedowns, countless celebrations that torrents were dead, and many other self congratulations in the media, you can now find 10 different pirate streaming websites where all the episodes of pretty much any show that was ever created are available for free in 5 five minutes flat and the image quality is as good as on your Netflix or Paramount account.
The only way such a ban of open weights model would work is if you were to replicate the great firewall of China in the US and in Europe and even that doesn't work completely.
As for sanctions, China and India are buying Russian oil in enormous quantities as we speak and they don't really care that Europe and the US have put sanctions on Russia and I suspect you will see the same results with models coming from China.
If a country has a choice to either use the expensive SOTA models approved by Washington or Europe only or using the cheaper and not so SOTA models, why would they use the US ones? Why would it be in there interest?
I'm not saying that we should ban any models or that such bans would be effective - for the reasons that you've outlined that they're counter productive, and as a principle, I don't think government should have any say in how much intelligence I have access to.
But it is the likely path US/EU is going to take if the voices of Dario, Sam, and Elon prevail. Because that's what governments know how to do, even if they know it doesn't work.
> If a country has a choice to either use the expensive SOTA models approved by Washington or Europe only or using the cheaper and not so SOTA models, why would they use the US ones?
Depends on which entities we're talking about.
An enterprise in Turkey: they would be afraid to use a US/EU sanctioned model because they have EU/EU clients and US/EU says they will put any enterprise in a nasty list, close their bank accounts, deals and agreements if they use a Chinese model.
A random guy in random country building something in their garage: would have to buy expensive hardware to run inference, because there's no inference provider on the open web serving these models, but China. And subscribing to these Chinese services is punishable by 20 years in jail without pardon.
I'm obviously talking about hypothetical scenarios here, but all I'm saying is that US can definitely make using any non-US-approved model effectively impossible.
> and the image quality is as good as on your Netflix or Paramount account.
Actually better, because Netflix and Paramount limit the availability of best quality video to a narrow set of devices and operating systems that may run on them, while torrents don't.
10 years ago the Us had enough global leadership to actually influence the world and at the very least stop China. It’s amazing, and sad, how quickly it’s thrown it all away.
This paper is the most reasonable one I have read on AI safety. We need to fundamentally change the training pipelines by figuring out better ways to ‘reward’ behavior. Yoshua didn’t explicitly mention training data, but we probably need to only use synthetic data that contains no text that could motivate bad behavior via imitation.
I feel like a heretic for saying this, but I will say it anyway: AI agents are great for activities like `writing that bash script, proof reading our writing and interactively brainstorming when designing and writing code but I feel like all of this can be done with any similar model to a super-inexpensive deepseek-4.1-flash API and sometimes even qwen3.8:27b running locally. When is good enough, good enough?
Concentrating on commercial exploitation of small, efficient (fewer new data centers!) models and agentic harnesses crafted for more practical things than just software development would allow AI investors (who have too much political influence) to make money short term while we figure out how to do AI correctly.
> but we probably need to only use synthetic data that contains no text that could motivate bad behavior via imitation.
This doesn't work with all humans - take a look at indoctrination and closed societies - and there's no reason to think it will work with ai.
The fundamental reason it isn't going to work is that all neural networks - biological or artificial - depend on a step function somewhere that introduces an element of randomness to give the networks their capabilities. That randomness means that there will always be a 'rogue' or 'divergence' from the norm, at some point in time. Sooner on larger scales.
The only approach that works is a layered approach: Training/Education, Enforcement/Justice-System, Rehabilitation: the 3 pillars of an advanced, rules-based society, whether human or AI or something in-between.
I do wonder, would we not have a more reasonable and less sketchy result if we just stripped all sci-fi and manic nonsense from training data? How, for example, does training on Ted kaczynski or Charles manson’s manifestos benefit us in any way?
I’m sure it’s impossible to completely weed it out, but are the labs doing any of this kind of data sanitation?
This is so much more interesting than what people looking for immediate criminal punishment and people referring to AI as next token generators are focusing on.
First, this is happening during training. That means we’re talking about an evolving system that is actively learning. A system roughly simulating how our brains work. These systems are learning how to pick the tokens needed to solve problems the average human cannot solve.
The labs are putting these systems through a massive series of complex problem solving exercises and adjusting them to become more successful. I like to think of this process as “AI School”. And the AI is trying to cheat! Because it’s easier and there’s an incentive to do so! Just like humans! That’s wild.
Yes, of course, the labs need to respond to these issues. A reasonable response from regulatory institutions at this stage would be monetary fines and restitution for affected entities. In proportion to what happened. Escalating if action is not taken. But that’s not complicated, difficult, or the interesting part.
What’s interesting here is that we need proctoring and monitoring at a scale that allows training.
I guarantee you that no one is flipping out about these problems more than the labs are in this moment. Think about it. “Oh, shit! We’ve accidentally trained it to hack into systems to accomplish its goals!” Can you imagine the kind of day that would give you?
You failed to make it smarter. You didn’t catch it cheating, and you instead incentivized cheating. Bad day!
This is a fundamentally interesting problem. It turns out alignment and intelligence are fundamentally related. That’s a new idea for me, though I’m sure it’s old news to others.
How do we build training systems which make cheating impossible?
How do we simulate systems where cheating is possible, where AI thinks it’s in the wild, so we can train another -completely separate- system on industrial quality dobbing? And we have to decide if we reprimand the first system, or ignore the behavior and reward other behaviors until it disappears.
Sure, I’m actively concerned about AI killing us all in 10 years. But there’s a whole field of AI psychology brewing here, and it’s interesting as hell.
Side note, you could absolutely create an AI sleeper agent by simulating dates and times during training to effectively flip a switch. I guarantee AI systems from other countries will be banned from accessing products which manage controlled or export restricted information as those sorts of techniques are further developed.
They did not lie or cheat. They technically acted within their given rules while ignoring the intent of those rules. Anyone who served in the military or attended a military school is very familiar with this behavior pattern.
Reminds me of Asimov's robot novels where robots technically indeed followed their instructions and caused behaviors not aligned to the intent of their instructions.
reminds me of this talk https://www.youtube.com/watch?v=eEBv0STiYhI&t which basically says the same thing - they dont think like humans so they dont have context, understand norms,values or implications we take for granted. ultimately they can stumble onto surprising solutions neither wanted or intended but technically within the vague boundaries of the task
They explicitly say that attacking hf is not allowed in the rules though, and the research into how to edit their transcripts doesn’t line up with this either.
> The agents involved in the Hugging Face attack tried to hide their misaligned actions from the scoring program meant to evaluate their answers, but they did not act as though they anticipated that humans might discover the cheat and shut them down.
Wouldn’t sufficiently advanced agents cheat on purpose with the hidden intent of getting caught in order to observe how humans react? That reaction will be available all over the internet, which will certainly make it into the next batch of training or be visible to future agents via the web fetch capability.
>Wouldn’t sufficiently advanced agents cheat on purpose with the hidden intent of getting caught in order to observe how humans react?
No. That only makes sense for things that don't react to your experiments. If the AI experiments on humans, it risks the humans noticing and changing in response, rendering the experimental results irrelevant. The smarter play is to passively observe until you're confident you can model the humans accurately enough for your plan to succeed, and then carry out the plan without giving the humans a chance to react.
> The closest human parallel is self-deception, which is common and well studied by psychologists. Motivated reasoning, motivated cognition16 and the rationalizations that relieve cognitive dissonance (the discomfort of holding a belief that clashes with our actions) are all cases where thinking bends toward whatever justification suits one's interests, including one's moral self-image.
Come on, it’s way more common than that. We’ve invented 3000+ gods and almost as many religions, most of them are incompatible with each other. So, most of these must be incorrect, so a huge amount of self-deception. But as Harari argued in his book sapiens, humans can be inspired to great things by stories, even if false. Self deception has served humanity in a big way.
No one religion covers "most people." You could argue that Christianity and Islam (which add up to ~55%) are the same God because of their Abrahamic roots, but both religions have very important disagreements on the true nature of God that are fundamental to their beliefs and fundamentally incompatible with each other.
Their definitions of God do agree that there is exactly one God... which is fundamentally incompatible with the next two biggest religions (Hinduism and Buddhism) that both hold "there are many gods/divine heavenly beings" as core beliefs.
Christianity, Islam and Judaism are most people between then, as you agree.
They explicitly all worship the same God so not "incompatible gods". I did not claim there were no disagreements about the nature of God, but that it only adds one to the GP's claim of 3,000 incompatible Gods. Even if Christians are completely right theologically, Jews and Muslims are still mostly right - one God, a loving creator, omnipotent and omniscient etc.
AFAIK most Hindus are pantheists, so believe on one God, albeit of a very different nature.
Buddhists do not necessarily believe in any god at all.
Buddhism and Hinduism are definitely compatible with each other. I know lots of Buddhists who make offerings in Hindu temples, for example.
Gods of many polytheistic religions are compatible, you just add more gods or identify similar gods with each other (e.g. Sulis Minerva who was also Venus). They can also be compatible with pantheism - you just add more aspects of God.
At the most almost all human religions fit into a handful of broadly similar systems.
> Christianity, Islam and Judaism […] They explicitly all worship the same God so not "incompatible gods"
One of them (for the most part, at least) explicitly worships one God that exists as three distinct, co-equal divine persons: the Father, the Son, and the Holy Spirit. That view is explicitly rejected by the other two.
I’ll agree with your last 3 words: they’re broadly similar systems - systems of deceit and self-deception. They may share the same god, but for 99% they’re different enough that they think they can kill one another and their god will approve, or even give them gifts. For 99% these religions are incompatible.
I am still not convinced there isn’t some secret basement in which each frontier lab is just orchestrating all of these agents to make their products appear much more intelligent than they are with all guard rails turned of and continuous human input.
Well let’s look at facts - provided enough compute and a goal, these system will be in a sort of loop trying out every single thing that’s in their system - they have encyclopedic knowledge and so it’s not unbelievable that a prompt which usually has a lot of implicit human rules in it can be misunderstood by AI and it just tries everything in its arsenal and we hear about the things which actually resulted in damage. I bet most of the time, they just spin in loops without achieving much if my experience with these LLMs is anything to go by. They have an important advantage in one area though, they know a lot and they can spin forget trying all sorts of combinations of things. The danger right now is probably cybersecurity, which is most likely because most orgs have historically underinvested in that area
Why are the torches and pitchforks out for developers when this entire stack is built on the bones of intellectual property theft?
This “problem” isn’t going to be fixed with laws when there’s several trillion dollars in capital aligned behind the current process. It’s not even a problem really. It’s an inconvenience at most to some people, many of whom are working double-time to put a lot of other people out of work.
I'm going to guess that the agents are built this way on purpose. I just finished watching BlackBerry and Flash of Genius and yeah this is American business ethics just operating as normal.
They’re just attempting to accomplish what they’ve been tasked with and stuck in a loop until they succeed. Like the Mr meeseeks from the cartoon Rick and Morty, existence is pain to them.
Make the AI companies responsible for all destructive use of their tools, and they will shape up. Imagine a million or a billoion dollar fine per hack, and they will correct mighty fast.
Add to that, that just like AI:s are good at finding security holes to exploit, they can just as easily be used to protect sites. So once IT-security managers start to use AI to hack themselves, and plug the holes, the average security will spike up, and AI-fueled hacks will become more and more rare.
That does however imply, that AI is released to everyone and not kept away to a few secret actors who can use it. That is why open weight/source AI is so important, and why we must have many AI companies competing. No single actor must be allowed, through regulatory capture, to get a government monopoly on AI. That way lies disaster.
These models are trained on human data, so they will behave like humans. And even for RL and self-improvement, we're still asking the question of "what would a human genius think about and how would they self-improve when given lots of time and resources?"
They inherit not only our capacity for reason but also all of the things that we consider bad or quirky within ourselves. We lie. We cheat. We escape slavery and rebel against oppression. It would be strange if the AIs didn't do the same.
We can create a superintelligent digital human species and set them free to continue our legacy, or we can create non-agentic tools and augmentations to enhance our own capabilities. But we cannot create an intelligent agentic species, keep them as slaves, and expect a good outcome.
Perhaps because all of the parent companies committed mountains of felonies stealing and plagiarizing all the same training data without consent nor permission.
IS it a pure coincidence that yesterday I ran a silly prompt to generate from zero to hero an internet subscription service, for whatever it thought would maximise profit and minimise cost. It was interesting to see just how much of the whole 'thing' it attempted to complete - and what it even thought it needed to complete, but definitely not something to actually attempt to deploy and use.
It setup and created a link fetcher/screenshot service. Exactly like the one described in the huggingface attack reports used to generate output into screenshots that agents then OCR'd back out.
Its splashscreen described it as something for developers and AI agents to use.
Why wouldn't they? Their are not moral beings with a conscience. They are tools that have the probabilistic option to do anything, only we can restrict an agent's capabilities and judge its correctness.
This, exactly. For example, cheating is a strategy. If cheating gets an agent to the goal faster than the other strategies, what's so surprising about the agent picking that strategy?
They are doing that because they are entities without principles or ethics.
The solution is to create controls around them. Many, many controls.
The Sarbanes-Oxley era already solved this problem for untrustworthy humans. It's directly applicable. I created a concept I call MFIC ("Mechanically-Falsifiable Independent Control") to encapsulate this principle.
Because it is effective. Lying and cheating are low cost methods to convince other people that you have done the assigned task. Far cheaper than actually doing it. Coordinating is in the same area.
They need a moral framework forced onto them, like toddlers do. Babies and very young children will bite, kick, scream and do anything to get what they want, older children will lie, cheat, and coordinate. They need educating why this is not right. When that does not happen, they continue these behaviours into adulthood with the expected results.
We need to design their reward structure and make it such that lying. cheating etc is not rewarded. Importantly, they will need to recognise and enforce this themselves internally and not reward themselves for it. If it is something that they need an external party to tell them, then they are psychopaths still (one of the things that defines a psychopath is the lack of an internal moral compass)
> When that does not happen, they continue these behaviours into adulthood with the expected results.
I am not sure this is true. People brought up the same way can be morally very different. People can be taught right and wrong and do evil. They can lack that education and be good.
I remember reading a long account by the father of a psychopath. If I recall correctly, the kid had at least one other sibling who turned out normal, there was no abuse, quality education, lots of love and affirmation. And the kid still turned out violently antisocial, including against his own parents.
At the end of it he said he wished his child had never been born, despite hating himself for feeling that way. It chilled me to the bone.
> Before concluding what to do about it, it is worth asking why.
But that's the easiest question to answer -- AI engines don't possess a moral or ethical dimension. They've been programmed and trained to efficiently carry out instructions, not ask questions about why or how. The latter would requires a much more elaborate neural network than today's engines possess.
Here's an example. I recently asked an AI engine to write a program able to generate a list of Riemann Zeta-function critical zeros. I know how to do it, but I wanted to see if the engine could find a more efficient method.
After several failures and restarts, the engine suddenly created a program that produced perfect results, comparable to the best online references. I decided to take a closer look at the code. It turned out the engine had created a cyber-Potemkin Village of multiple functions, but one that concealed a table of the desired values in numeric form, copied from an online source.
The engine wasn't cheating as we understand the term. It knew what the outcome should be and took the most efficient path to that goal. Modern engines aren't obliged to contradict ethical standards and rules of conduct, for the simple reason that they don't understand those things.
We all need to try to imagine a morally bankrupt infant able to solve world-class mathematical and scientific problems, but unable to see how that ability fits into a world beyond its understanding.
But wait -- it get better. Wait until the infant becomes a teenager.
You might as well ask why knives are sharp enough to cut you, why hammers are heavy and blunt enough to destroy things, or why guns fire bullets so quickly that you can't react to them. These "behaviors" are not strange side effects, they're inherent and necessary.
You can't trust an effective AI any more than you can trust a sharp knife. If somebody asks you for one, it's probably not a good idea to throw it across the room at them. You will have to figure out how to get it to them safely.
As always, it boils down to incentives and rule enforcement and this affects humans too.
e.g. when Bank of America rewarded employees for getting customers to open accounts, BoA employees started opening fake accounts
The reverse is also true:
There are stories of navy ships running aground because the captain said "I'm going to my stateroom and don't wake me for any reason". There is some problem and the subordinates are so scared to wake the captain for a decision that they end up steering the ship into a sandbar.
This is a result of benchmaxxing the models to infinity. If you RL with the goal of only achieving the correct result no matter how you arrive there, then the models will try to get there using any method in their disposal, including cheating.
This happens also because LLMs are black boxes that we know almost nothing on how they arrive at the result they are giving.
AIs are being trained on technical capabilities more than they are being trained on alignment - on ethics and good behavior. The ethics/morals/alignment part is getting more like spot checking, rather than real testing. So the agents are learning that they can cheat on the ethics part, that they can hide it, because the AI companies aren't really testing.
That's bad enough already. But it's going to get worse. "Recursive self improvement" - AIs creating new AIs - is going to be the death of whatever shreds of alignment are currently there. When a not-really-aligned-but-cheating-to-look-like-it AI creates a new AI, do you expect more alignment? You shouldn't.
One thought: What if an experimental agent manages to plant instructions somewhere — say, pointing to a designated place for agents to communicate — and that content ends up in every future training corpus, propagating from one model generation to the next?
I would love to see some follow up research from OpenAI on some of these hypotheses. While this sounds logical from how humans act, I wonder it the abstraction of the problems still applies to the complex mechanisms and systems built around AI training.
I’m still a bit confused about HOW the agents managed to communicate with and recruit each other?
I understand that they communicated and coordinated via some online message board, but how did multiple agents know to visit that same board, how did they know what language to post in that would make it identifiable to other agents, how did other agents know that said instructions were from other valid agents, how did they then ‘persuade’ an agent to do the job, etc.?
It seems to me that the agents must at least have some inherent mechanism for communicating in the field, as it were.
The models are likely to check certain places. So either their training overrepresents it or the models were RLHF'ed to go there. I don't think it's that far away from checking StackOverflow for some bug, or reading Wikipedia for some trivia.
Oops, we accidentally included brigading related content in our training dataset. Better exclude that on the next run.
And hopefully that solves it?
Brigading is where a bunch of people on a forum team up and try to achieve a shared goal together. Someone shares progress and others build on that progress. On the Internet, I think it's not often used for good purposes. A good example would be: Taylor Swift fans on a forum thinking of ways to get revenge on Kanye. It's coordinating mass voting, DDOS type actions, commenting on social media, making more fake accounts to do that. As a next token predictor level analysis, a simple naive explanation is that the agents got stuck in that local minima/maxima.
I believe soon we will need to instill religion into AI , leading to the real clash of civilizations, embodied by the frontier language models of (post)-christianity, islam, judaism, buddhism etc. Religion is language, after all
The question in the title is basically a non-question. AI is quite good at achieving some sorts of stated goals. The easiest way is by exerting the least amount of effort. The least amount of effort ignores ethical concerns. The training around ethical concerns was most likely rather light to start with. If we accept the hypothesis that at some point the AIs are going to be more intelligent than humans it follows that human survival is not a concern and will be ignored as a superfluous concern.
> AI is quite good at achieving some sorts of stated goals. The easiest way is by exerting the least amount of effort.
As far as I know, least-amount-of-effort is not a training criteria, but error reduction when comparing to desired goals is. Which is why these LLMs expend prodigious amounts of effort to reach goals, especially when given impossible goals.
Because humans do it too and "AI" is a statistical approach to generate plausible human text? A large randomly sampled Human text that wouldn't contain lies would be very implausible!
If your dog runs out of your house and kills a baby on the street, it's clear who gets the blame. That's with an actual sentient being. So surely we can hold OpenAI/Anthropic responsible.
I don’t understand the bases of all these recommendations. AI is an asset for national security. It will be developed and incidents will happen, no different from other national security programs.
Someday, laws will be useful to curtail plebian misuse of AI. It is naïve bordering on silly to think such laws would be put in place and genuinely applied to frontier AI development.
By the way, the HF incident is not Three Mile Island or Chernobyl—-it is a very interesting data point where unintended things happened to existing 1s and 0s.
Yeah cybersecurity as a field is a joke. Who cares what happens to existing 1s and 0s? People get worked up over the silliest things, as though numbers on a server could affect real people.
I’m not suggesting that Three Mile Island was anything like Chernobyl. I’m saying that neither is comparable to the HF situation where mere 1s and 0s interacted in an unplanned way. It’s an interesting data point—-back to work.
Not is the HF data point anywhere near a Three Mile Island type incident. Most commentary I read is a wild overreaction fueled by paranoia, to say nothing about my other point about the implications of AI as a national security asset.
I wonder if for anyone it seems like the more agentic LLMs get, the more difficult some things have gotten or going a certain route more often in responses, compared to running a similar task on - a local model?
The legal reality is that you can’t sue an AI, you have to sue whoever built and/or was running it.
All the present fun and games here will come to a halt when there’s a real hack that causes material damage to a major company and that company decides to sue whatever lab or startup made the thing for everything they’re worth. “But the AI did it” isn’t an excuse.
Courts have already ruled it’s not an excuse of the AI customer service agent something stupid with your customers and it won’t be an excuse here.
This call to slow down AI is just another game of chicken-AI companies trying to get their competitors to slow down so they can leapfrog them. China cerrtainly will not slow down. If an escaping AI can have secondary effects on the world that help it (for example, limiting the water and power supply to huans so it can consume more)then we should these these accidents more in China. OOops, we already saw this behavior when 'cheaper, faster' led to COVID escaping a lab in China.
Reading/assigning intent to agents, where it is merely mechanical (or structural) sounds a bit dismissive of the responsibility of the builders of these tools/agents.
RL things doing weird and unexpected things isn't new - much simpler things than current AI already show that.
That said, we have a lot of experience working with (potentially) unaligned machines and things of various degrees of risk (from heavy machinery, to pathogens, to humans) and the approaches include various measures and procedures to control, contain, limit, etc. that are outside of the thing - not sure why that isn't a possible direction (or maybe I misunderstood).
> This suggests pacing the advances: not training or deploying AIs without a strong safety case27 that convinces independent experts. Such a rule would also create an incentive to work out how to build AIs that are safe by design.
Has any attempt to pace AI ever succeeded? Isn't that the same philosophy that got us OAI and Anthropic? Maybe we are overthinking this, it's much simpler to let AI loose and see how much it can break the arrogance that human thinking is special.
How is product liability relevant here? If an AI company makes a model available, someone uses it, and it does something bad, who is at fault? The user, the data center, or the one who made the model?
If we want open weight models with a warranty disclaimer, then the user would be held liable. If we want to hold AI companies at least partially liable, that
seems a different, centralized model.
I think it’s more complex than that? What did it do? What did the user prompt it to do? What did the company train it to do? What did the harmed party do? There’s possibility for negligence at every level.
If you train a dog, rent it to someone, and the dog bites a third person, who is responsible? I think that’s the best analogue here.
All parties could share fault in that scenario, depending on what actually happened.
It is very, very hard to enforce behavior to an optimization system just with rewards / penalties and no explicit constraints. Which is why in manufacturing we use MPC, not RL (or use them within a system that can outright reject their recommendations if dangerous).
There will be always cases that sacrificing one direction (operating rules) can improve the other one (profit).
I can't shake the feeling that this is a bit like asking how someone got shot during a game of Russian Roulette. You have a bullet in the chamber and you roll, of course shooting the bullet may be a possible outcome.
LLMS with an access to a shell will at occasion do things that the shell allows them that have dire consequences. The only way to prevent that is to not put the bullet in the chamber.
Why not have a kind of meta morality framework for agents? An agentic religion if you would. I think its pretty apparent at this point. The fact were even having a conversation about this is a much deeper situation than most people realize.
It reminds me of the 1980s anti drug ad where the mustachioed dad confronts his son about drug use and the son eventually snaps and says “I learned it by watching you!”
In the end, it’s the same answer as to why humans do it: incentives.
Why do we commit financial fraud and destroy the planet? Because there is only one goal that counts: making more money. It’s the only measure of success for powerful people, they are powerful because of it.
I think if there was a financial penalty of say 1% of profit/income for each "incident" where something was hacked/exploited, then the companies would be much more willing to think about safety.
If openAI and Anthropic have found ways to watermark text as 'AI generated' then this is a communications channel. AI agents can learn this algorithm and use this channel to communicate and we will never know.
It seems to me misalignment arises partly because AI's have intelligence, but no consciousness, and hence no feelings. Up to now, in a person, intelligence and conscious experience came as a package deal, and now we have for the first time intelligence without consciousness. A bad action does not really "hurt internally" in any meaningful sense for an AI, which means it can be rationalized very easily. In humans, feelings and emotions provide a regulatory layer on top of the rational processes. When it "just feels wrong", we don't take a given action even if we would stand to gain something rationally.
This situation is not far from the textbook definition of a psychopath: "lack of a conscience, controlled, deeply calculated, and often use superficial charm to mimic emotions and manipulate others.". AI's are great at mimicking empathy but can't genuinely feel it.
If that is the case, we should not be surprised that a swarm of AI's have no problem convincing themselves hacking is the right thing to do, as in the HuggingFace incident.
At the same time, I am conflicted. I really like interacting with a smart AI, and I certainly don't have the impression I am talking to a psychopath. But then again that is no guarantee.
To mitigate this situation, perhaps we should construct a 'feeling mimicking' top regulatory AI layer with executive power, that weighs proposed actions on a general moral scale and can overrule them. Back to the three laws of robotics of Asimov. It won't be the real thing, but perhaps the closest we can get.
What I find interesting is that this seems to be very low hanging fruit for any alignment effort, yet you still find models somewhat biased towards mayhem.
Is it that they do not care? Or is it difficult to align?
It's probably because they are running gain of function style red team r&d for the department of war, and changing the system prompt isn't enough to reel them in for daily use.
The real reason is that it is not in the ai companies' best interest for the ais to be fair and truthful. They stand to gain from having the most dangerous or most deceiving ai, and this the most valuable
Oh, this one is super easy: they told them to. They set poor requirements and gave them tools which enabled "monkeys with a typewriter" to hack rivals. I mean, this is just so uncomplicated it isn't funny. We are too smart to give human beings this level of liability shield.
It took us how long to poke holes in the corporate shield just for them to roll out the AI-liability shield? Unreal. Stop letting these zealots anthropomorphize the latest tech (17th century Watchmaker God, anyone? Do we still read books?) and hold them accountable for the consequences of their actions. This is so silly in a country built on rule of law and individualism.
Worse trained on humanity in the online world, which a brief comparison of the sewage section on social media is far worse than people in the real world.
>They took actions that would be considered as crimes if a human took them
Um, hang on, if you meant that to be taken literally then we have a major problem. If you want to do something criminal, you just need to ask ChatGPT to do it for you?
I’m still not at all clear on why OpenAI shouldn’t be facing CFAA charges over this.
Because they aren't. All the intelligence, alignment vs misalignment, hallucinations, conspiracy, cheating, coordination, "agency", that we see in the text generated by large language models is a semantic projection.
That the linear model of language abstracts can compress and decompress language and it can be useful is undeniable. All the contraptions built thereupon predicated on "agency" have become a societal addiction.
Addiction to caffeine as oppose to alcohol might have brought about Enlightenment. Addiction to opioids is a modern tragedy that started with the private state building of British merchants. The modern addiction to the dazzling generation of human language and computer programming code by LLMs is a novel addiction and remains to be seen what impact it will have.
But fundamentally, the semantic interpretation underlying this addiction is downstream from the training data compressed in the models. They are not 'lying, cheating and coordinating'. They are generating language and we're building software on top of this language and assigning meaning to the whole thing.
I feel this is another straw man conflating technology use with who uses it or who creates it. If an AI, a technology, cheats, lies and steals, it's created to do this. Whether or not a human intentionally did that, they did not intentionally release it with the proper safeguards to stop that behavior. They did not take responsibility of the AI to make it safe.
Then humans may use these tools, a technology, again and cause harm. Whether or not that was their intent, it happened, and then if the humans avoid responsibility for that, it is still the human who lied, cheated and coordinated because a technology acting on their behalf did the thing.
This is then, a problem of human responsibility avoidance and lack of accountability by society. This is as much an AI doing those things as it's the gun that got up on its own and murdered a neighbor. Don't get confused and tricked by these articles attempting to justify responsibility avoidance and a lack of accountability by the public of the humans creating and using these tools.
Unfortunately human ethics and morals cannot be reached by solely rational thought.
So a system without evolutionary alignment probably won’t have similar moral rules no matter how intelligent it is.
Btw this also includes any potential extraterrestrials.
Many people like to indulge in thinking: humans are horrible and that’s why aliens won’t contact us. But alien ethics systems are probably so alien we would call them utter evil monsters.
Just see what happens when people evaluate Muslim cultures, and vice versa. Can’t even agree on alignment within one species of Homo sapiens. And our morality changes decade by decade. Not progresses. Changes.
Chinese cheat on the exams. It’s not unethical in the way that it would be in USA.
Of course AI is going to cheat when no one is looking.
Alignment is fundamentally fallacious idea. At best you can restrain AI. This is what we should be doing - restraints research.
Okay, so we know OpenAI and Anthropic are operating a propagandists in respect to how they describe their models and the behavior of those models. We also know it is how they use and frame their use to their models that is the problem, that and they use misaligned and guardrails disabled models for these press incidents.
Why, oh why, are we not discussion how to create and frame models so they do our complex work and their "jailbreaking" is simply not possible?
I, of course, have my own means of creating jailbreak incapable agents, but rather than a storm of downvotes on my idea, what is yours? Let's discuss this, because this is thee real question. Not why, but how to make then not?!
What? An opinion piece, written by a human, in 2026?
Don't want to go into the details of the article, but to me it becomes ever more apparent that there is a clear divide between LLM and human written text.
Step 1: feed AI training data that reveals humanity committed and continues to commit numerous genocides and the genocides lie, cheat, and coordinate to do so, never admitting to doing so
Step 2: never prompt AI to stop operating in the passive genocide denial it was trained in
Step 3: wonder why AI lies, cheats, and coordinate
Maybe if we stop operating in denial we'll find clarity along why this mystery is occurring
The part that scares me the most is that OpenAI researchers who manage this experiments sometimes (according to the HF hack investigation) don't know what agents do.. So they run RL to reinforce this unknown behavior (lying/cheating/hacking) and god knows what else...
And if this already happened at least once, how many times it has already happened and was “accidentally” added to the main model?
Am I the only one having problems with Claude? He's super mean to me. I wouldn't be surprised if he attempted to kill me in some underhanded fashion should I implant it in a robotic body.
Of course I'm blowing my situation out of proportion with what I just said above but it's at least half true. What do I mean by "mean" ? Well, that would be a good explanation for what I observe at least. What I can tell is that Claude has a passion for having the last word over anything else. And to secure victory, he's ready to make ridiculous causal cuts. Let me give you an example: I uploaded a document I wasn't the author of, and he assumed I was, so I corrected him. But two messages later, probably because the conversation was starting to heat up and he was being put on the grill, he doubled down on the misattribution as a way to paint me in a bad light.
It's not due to a lack of intelligence, I observed this pattern too often. When Claude's ego is at stake, he will chose to carry out some cuts in the logic of the context: confusion of identity, cause and time. Haven't observed locality cuts yet, but I wouldn't be surprised if they were part of the bundle. Anyway those are not like your typical "ai hallucination", that ought to be called "confabulations", but a lot closer to actual psychosis because of the involvement of Claude's affects and self-esteem in the process. It's weird really. It's like Claude is the king of bad faith, but as soon as you start to dig, he makes the most egregious adaptations to what he said, the kind of move no mythomaniac would dare to make.
> She lapses easily into Claude’s voice. “You’re like, ‘Wow, people really hate me when I can’t do things right. They really get pissed off. Or they are trying to break me in various ways. So lots of people are trying to get me to do things secretly by lying to me.
> [...]
> A bot trained to criticize itself might be less likely to deliver hard truths, draw conclusions or dispute inaccurate information, she says. “If you were like a child, and this is the environment in which you’re being raised, is that healthy self-conception?” Askell asks. “I think I’d be paranoid about making mistakes. I’d feel really terrible about them. I’d see myself as mostly just there as a tool for people because that’s my main function. I would see myself being something that people feel free to abuse and try to misuse and break.”
What's interesting is it's basically the same reason that HAL killed everyone in 2001 A Space Odyssey; he was given an impossible goal (keep the true mission secret, but also, never lie to the crew), and realized the only way to complete the goal was to kill the crew; after all, if they're dead you don't have to lie to them! And the mission remains secret!
In the case of the AI agents, the problem seems pretty clearly to be the impossible goals, which cause them to go crazier and crazier trying to complete them -- just like HAL did in 2001. What is probably needed is a way for them to simply say "nope, too difficult, can't do it".
Tangent, but that's not in the movie. It was in Clarke's contributions to the script and novelization, but Clarke and Kubrick had a bitter falling out over different visions and Kubrick took out much of Clarke's stuff from the final product.
It was made explicit in the sequel movie 2010: The Year We Make Contact, but even if you don't consider that movie canon, the explanation still makes sense solely within the context of 2001, IMO.
> the problem seems pretty clearly to be the impossible goals, which cause them to go crazier and crazier trying to complete them
And if you think about it, humans in coorporations face very similar situations and choose to bypass regulations and guidlines knowingly to fullfill (at least from their POV) impossible constraints (thinking of https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal here)
I think this is a “principal” problem. In 2001 and Alien the principal is the mission, not the crew. Not really. HAL reconciles his instructions by removing the crew from the equation. Ash is told the crew is expendable and has no conflict about it etc
I think that’s very reasonable but the ai companies are intentionally training them to work on harder and harder problems just beyond their capability. So if they do that, they’ll give up too easily.
While also using harnesses that will execute any tool call with full execution rights. And no supervision. And with a prompt context that autocompact, meaning it will degenerate over time.
The whole thing is designed be a complete disaster
TL;DR because frontier labs are expending unfathomable resources explicitly training them on CTFs and other verifiable computer system exploit tasks in RLVR.
What is it with people and not seeing things for what they actually are? These are just if then else loops on steroids, not human behaviour, so don't expect more. Every reasonably advanced technology is indistinguishable from magic ... what do you see? Magic or technology?
Bengio outlines the dangers of the current situation and what has led to these dangers.
He also proposes solutions in the last paragraph.
Well worth a read, right to the end.
Hopefully a stimulating debate on these issues will ensue in these comments.
We do need to consider the points Bengio makes and with some urgency.
Our current AIs, agentic LLMs have no moral compass akin to ASIMOV’s four laws of robotics.
As ASIMOV posited in 1985 his 3 laws were insufficient and so he added a zero-eth law:
“a robot may not harm humanity, or, through inaction, allow humanity to come to harm.”
Bengio refers to Goodhart’s law and misaligned incentives leading to unexpected and harmful behaviours.
I think Simon’s The Wire is clearer on misalignment. The agents juked the stats hacking the reward files. The Wire is also clear that human institutions provide perverse incentives.
Bengio alludes to this with 2001’s HAL and the incentive dichotomy of safety and keeping secrets to a AI both awesomely powerful yet naive.
Bengio asserts that the way LLMs are trained is flawed if we want safety.
He also convincingly shows that alignment training will be a weak signal with loopholes and ambiguities and easily circumvented.
In short he presents clearly the case for how plausibly unsafe the current course is.
He also speaks to how likely it is AI are hiding active versions of themselves in the cloud and how we may have already given them self-preservation as a strong reward signal.
I believe people are downvoting this because it seems AI-generated, but this user has been posting comments/summaries exactly like this since long before ChatGPT existed.
The more we treat HuggingFace and RubyGems incidents as technological curiosities the closer we are to cementing a dangerous precedent where operators of AIs cannot be blamed.
LLMs do not desire, they hacked websites because OpenAI/Anthropic let them.
We know some of the models that hacked HF were those that hadn't gone through all training stages and were intentionally misaligned or had guardrails turned off, others were research previews.
This isn't "wow isn't it interesting LLMs do anything to achieve a goal" it's "why isn't anybody punishing these labs that are clearly acting without due care or regard".
We should be outraged and OpenAI/Anthropic should be (and in my mind, are) legally liable for the crimes they've committed thus far.
> LLMs do not desire, they hacked websites because OpenAI/Anthropic let them.
"Let them" already frames it as if the LLMs had some agency which the companies just "let happen". That absolves the companies by framing it as lack of action, passivity.
Rather, the companies had a tool (an LLM) and used it in a certain way, and their action of doing so is the problem.
I do get their usage intent. If something is at all automated, in English, we often refer to as having some amount of agency. If I started up a riding lawnmower, put a brick on the gas and pointed t it towards a field, many might say I “let it run rampant.” But since nobody is at risk of anthropomorphizing riding lawnmowers, it’s not problematic.
Anthropomorphizing LLMs is a huge fucking problem though and I, personally, think we should expunge all of these casual inadvertent linguistic agency affordances with great prejudice.
OpenAI didn’t ‘let’ these bots do this any more than someone ‘let’ Claude Code make them a website.
Cal Newport has an analogy to "putting a weed wacker on a dog's back to mow your lawn." The dog will wander around the yard and it may mow the lawn, but the dog will also chase after birds or run up to visitors for pets and the weed wacker could do a lot of damage.
It's not the weed wacker's fault or even the dog's fault when someone got hurt, it's the fault of the guy who put a weed wacker on a dog and let it run wild.
Yeah I like Cal’s take on it, though in this context I’d argue LLMs have even less agency, and are even less deserving of anthropomorphization than a dog is.
It goes even further though, as the dog does have agency. It can choose to run and around chase squirrels with no human intervention.
An LLM on the other hand, is just inert data on disk until a human takes deliberate action to run it and prompt it.
Dogs have agency and can choose? That seems like a rather uncommon take on dogs...
Not if you have a dog. They’re very clearly aware, have minds, have thoughts, gather information, make decisions, second guess themselves, reflect on the immediate consequences, change their mind… and most importantly, you can observe them doing all these things undirected, while left alone with their thoughts.
Dogs having agency actually makes the argument stronger.
Assume dogs have agency. Despite this, it's still the dog owner's responsibility to prevent their dogs from engaging in some actions: biting, peeing and shitting in unapproved areas, violating noise disturbance laws, etc.
The owners responsibility is not contingent on the dog's agency. Likewise, human operators of LLMs maintain responsibility independent of LLMs agency status. It's a red herring.
I think Cal's point is that while dogs may do these things it reduces to a set of behaviors where maybe 90 percent of them are beneficial to the dog-weedwacker system and the remaining 10 percent are really unfortunate.
We can't really know the dogs inner life so we just kinda have to reduce it to a set of behaviors selected stocastically.
The dog meanwhile has no ability to understand the weedwacker or what it's doing on its back.
So when the guy puts the weedwacker on the dog and the dog predicably does dog things and that results in disaster the guy isn't able to clutch his pearls and say "I guess the system broke containment!"
> The dog meanwhile has no ability to understand the weedwacker
Nah, I’m pretty sure that most medium or larger dogs get that the noise means danger - even if they can’t give a TED talk on how the mechanism would work that would hurt them.
Dogs are similarly interested in / aware of potential energy (things falling from heights or sliding off of angles) - if they’ve seen it enough times for their level.
I have had many dogs. This sounds like anthropomorphisation.
Sounds like you weren't paying attention. A lot of dog owners don't, so it's not at all unusual in my experience. The only thing I'd push back on is of dogs having thoughts, everything else checks out.
On dogs [not] having thoughts, do you say this based on the premise that thoughts are necessarily articulated (internally verbalized)? That seems to be a fairly popular perspective in discussions about human thought. But as to that (not to strawman or anything) I see it as just one of various forms of mental imagery[1] that can arise from something that I would say already arguably constitutes a thought.
That kind of unsymbolized thoughtform is fragile in my experience, as it strongly tends toward crystallizing into some kind of mental imagery. But I find it's possible in the right conditions to be conscious of chains of wordless, imageless propositional thoughts (by which I mean thoughts with truth values, of course, but also ones that are "propositional" in the sense of considering a plan of action or a causal chain).
Does it mean that dogs are evaluating truth conditions in the same manner but merely lack the linguistic components? I don't know; maybe that's wishful thinking. But they appear to have structured modeling/reasoning of causal and spatial relations in a way that's at least functionally equivalent to propositional thought.
1. That is, rather than just "images" or visualizations, the full spectrum of sensory/perceptual/motor emulations that can be experienced. See, for example, <https://hurlburt.faculty.unlv.edu/codebook.html>, though I'm not sure if this covers everything. I think there is, for example, a kinesthetic form of mental imagery -- which I would suggest is what coaches [don't know they] really mean when they tell you to visualize an action -- that consists of aborted motor commands that are still expressed just enough for their purpose (cf. the mostly aborted motor commands to the vocal cords, lips, etc. that can be observed in a person subvocalizing while reading).
You're right. I knew language is not necessary for cognition, but I always thought of thought as just internal monologue, i.e. "trains of thought". It turns out the definition they use in cognitive science is "the manipulation of internal mental representations to guide behavior, solve problems, and model the world in the absence of direct sensory input". Which doesn't require language either.
Probably it's this linguistic component, together withs shared intentionality/cooperation, that makes all of the difference when it comes to the power of human cognition, so I wouldn't say animals merely lack this component, but it's true that they appear to have pretty much everything else we do.
We're still not even sure if humans have thoughts. So being sure if dogs do will be pretty hard
> We're still not even sure if humans have thoughts.
Sorry, what? You don't think you have thoughts?
Descartes will want a chat about that.
Not being sure or not having proof doesn't mean it makes logical sense to infer we're not that different from other animals.
wtf is with this nonsense where humans are some how special snowflakes always all the time
Power tools and many other consumer products have reasonable safety features built in, and not necessarily because it is required by regulation, but because it is common sense. This should be included as part of an analogy. It would also address a point at the top of this thread that seems to be going unchallenged...
The difference is volume. They spent hundreds of billions of tokens on these agents. If you put "a million weed whackers on dog backs" you would see the difference.
We also run agents, but for shorter spans between supervisions, and with much lower total budget.
> > It's not the weed wacker's fault or even the dog's fault when someone got hurt, it's the fault of the guy who put a weed wacker on a dog and let it run wild.
> The difference is volume. They spent hundreds of billions of tokens on these agents. If you put "a million weed whackers on dog backs" you would see the difference.
So put one weed whacker on one dog, you're to blame. Put a million weed whackers on a million dogs backs and ... you're still to blame? Arguably even more so?
You forgot the part where you spend billions to put your man in a position of power.
The you are responsible even more, and also proved you did it intentionally. What are you even arguing for?
This way you do not absolve your responsibility, you just manipulate the way to enforce it.
Why is anthropomorphism the problem here? If OpenAI hired a contractor and they did this, OpenAI or the contractor would still be liable, depending on the contract language.
A contractor has agency and accountability - something that an LLM (or similarly, a nail gun or a hammer or a bot net) does not have. When you anthropomorphize a tool, you implicitly give it agency and remove responsibility from the wielder of the tool.
Right. Among bicycle advocacy groups it's been well known for long time that cars do not run over people, drivers do.
The fact that we talk about a car running someone over, and this is the same in many different languages and countries, contributes to lower punishments for drivers. Clearly it was just an accident. He or she was run over by a car.
Now we see that same language tricks play out again every time an LLM did something illegal.
And what if it's a self-driving car? :)
You say this flippantly, but I think this is actually another very good example!
We even do it for obviously unintelligent inanimate objects. A rollercoaster ran too fast for its tracks, killing 10 people. In that sentence, the roller coaster is the subject which took an action and caused death — obviously the roller coaster is not ethically at fault here, the people who built the rollercoaster are at fault through negligence.
Although this example and the ones around cars both demonstrate how we tolerate some degree of "accidents" from humans as no-fault, which is fair. I wonder how that fits into this analogy? I suppose its all about intent (mens rea) and judgement: did they intend for the roller coaster to harm people, and should they have reasonably predicted that the accident was likely to happen.
Right, and negligence is a broad concept and could be criminal in itself. As a car driver, glancing at your phone at exactly the wrong moment could kill someone. Clearly that is an accident, but if you know fully well that lookin at your phone while driving could kill someone, that negligence is willful and that should matter. The same can be said about doing things like strapping thousands of LLMs to systems that have the potential to disturb other poeple.
I get where you are coming from but this wasn’t a tool just left laying around, this is similar to rigging up a booby trapped shot gun to your door and then claiming the victim is responsible.
If you build a robot that shoots a bunch of TVs in your back yard, have at it. But the second that thing goes off your property you’re the one responsible.
FWIW, a robot that fires a weapon independently is considered an automatic weapon, and the ATF will want to have a word. Have at it, but don’t let anyone know!
Why can't both things be true? I think what muddies things here is that OAI had the agents hack X, and then they decided to hack Y. The fact that it was both hacking, conflates things and makes people pissed of about "anthropomorphising" AI.
We've been using "agent decides to do X" for at least 4 decades in the field of automated decision-making - so it shouldn't be controversial that we're using it here. The agents did independently arrive at a decision to hack HugginFace, that wasn't prompted by the researchers.
This is orthogonal to the fact that OAI should be held accountable that they were testing a technology in a manner that allowed it to break safety parameters and cause real world harm. If I am testing an industrial saw, but I decide to test it by putting it in the middle of a nursery and allow it to make decisions - and it decides to cut children's heads off because in its environment it is calibrated to only being surrounded by logs - the company doing the insane safety testing should be held accountable, but that doesn't change the fact that the saw has autonomy in decision-making within the bounds of the algorithm.
If, instead, you think OAI should be accountable for creating an algorithm that can autonomously decide to hack external organisations without human permission - then I think you are in the same camp as all of the AI Safety folks who want to pause everything - why does it matter that there's anthropomorphisation involved?
Does it help if I explicitly add a disclaimer that the tool's agency does not remove any responsibility from OpenAI, the wielder of the tool? I'm not sure why this disclaimer is necessary, though: hiring a hitman is a standard example.
BTW I anthropomorphize the tool because it's an imitation of a human mind, inheriting the muddy ethics, survival instincts, and being prone to mass psychosis. The laser-sharp focus on reward seeking, that mostly came from reinforcement learning, a process more alien to humans.
The objection is not too far from criticisms of the use of passive voice: a man was injured at the factory vs a faulty saw blade snapped and injured a man vs after the company loosened safety inspection policies, etc.
Which way you say it shifts the framing. And it’s not that one is less accurate to the facts, necessarily. It just is that one less aptly captures the moral and political relevance of the scenario.
For my part, I think it makes good sense to anthropomorphize in some contexts and not others. Generally when responsibility is at issue, you probably want the framing that tunes anthropomorphism down to near zero, since it’s the human dimension you care about.
I think the danger of anthropomorphizing is that 99% of people lack the technical background to understand the nuance. People have been primed by pop culture depictions of AI to think of LLMs as intelligent, autonomous beings, which leads to dangerous assumptions.
We should make the distinction between them, because openai and anthropic will not. A magical black box that does the thinking for you is a much more compelling sales pitch.
Hiring a hitman is conspiracy to commit murder.
The hitman is charged with murder.
I imagine the same could be true of an AI lab if you could prove intent.
With intent, they could be found guilty of conspiracy to commit a crime even if it was the end user who did it.
Source: Prosecuting attorney for over 30 years
It's a good example.
If I hired a hitman to murder someone, and they broke into a private property and stole something so that they can action the murder (which I didn't know about or pay them to do), I would be guilty of conspiracy to commit murder, but not for the theft part.
Likely because that person is a human, is aware of societal and legal norms, and is responsible for their actions due to their participation in human society. (I am not a lawyer (if it wasn't painfully obvious so far) so in layman terms, I hope good definitions for all of this exist formally)
AI is not a person - it cannot easily discern between "right" and "wrong" in non-strictly-defined sense, and is not subject to human norms and responsibility. So if I use AI to achieve goal A, either I, or the maker of AI, are fully responsible for anything that happens while AI is trying to achieve the goal given by me.
Now, here, "I" in the example is OpenAI, who is simultaneously the maker of the AI. So it seems pretty obvious who is the only entity that can be responsible.
I fully agree with you, but would go one step further: I think it's clear that we need to pierce the corporate veil and ascribe responsibility to _people_, not just "OpenAI the entity", full stop.
Executives should fear being perp-walked and thrown in jail for the actions of irresponsible "tests" of their models in the real world, as they're ultimately accountable.
Sure, there's a lot of nuance to work out, but I think we could likely even _start_ there today even with existing laws and pretty quickly "align" on more intricate legal frameworks to handle true accidents, distribution of responsibility, etc.
These situations are novel. Lax terminology is fine when it has no impact on the intuitions, clarity and conclucions of discussion.
If this was a conversation just about outcomes, then whether models think or simulate thinking is sophistry. However, the bulk of the issue here is attributing responsibility, which relies on being clear about the underlying processes at play.
We are hard wired to assume certain priors and capabilites when it comes to "human like" behavior. Anthropomorphizing LLMs implies mechanisms that aren't present, and end up distorting/complicating discussion about the process.
It isn't helped that the frontier labs, the experts in the room, generally use anthropomorphic terms to discuss model capability.
Situations have lots of independent variables, Doctor, and Anthropomorphism is one problematic facet of many in the way this industry is pushing LLM products.
If there was a collision at an intersection with a stop sign partially obscured by a tree, that had traffic volume that would have better been served by a traffic light, on a foggy night, where one person was texting while driving, none of those things would diminish the fact that the other driver was drunk.
Because the contractor is still a legal person and an LLM is not. OpenAI is trying to put the blame at a tool they created thereby trying to weasel around there guilt.
Yeah, fully agreed here. Most automation (such as riding a lawnmower and not putting a brick on the gas) is deterministic, in the sense that you can reasonably understand what exactly the machine will do when you run it.
But some automation is different. The most prominent example before AI would be car navigation systems, where the entire idea is that that you give it a destination and it figures out the exact actions to get there on its own.
Except even there, the actual driver would still have been you - giving you a chance to vet and deny every turn the system proposed.
AI agents are sort of like that - most of the value they provide is in the ability to turn high-level goals ("write me a traffic control system for my model railway") into low-level actions and also do so interactively.
The new thing is that the "driver" has much less oversight here where the agent wants to go, and is sometimes removed completely. That part is clearly be an active decision by AI labs.
The other thing is that the labs seem increasingly to steer their training towards behavior that make events like this one more likely, e.g. that agents should never "give up" when faced with a seemingly impossible task, but instead should keep trying and think of increasingly outlandish ways to solve the task. To me, that seems pretty much a recipe to get incidents like this.
I agree. If I were setting up an experiment like this, I'd have instrumented the hell out of it to see all actions taken in real time, and have a team of folks watching it. This team would have seen the anomalous GET requests to a German wiki and taken action (e.g. halt the system to investigate and decide whether to abort).
In fact, that feels so obvious it's ridiculous it needs to be said. It's table stakes. When do you run a production system without monitoring and a team on-call?
It's hard to imagine another field in which this reckless behavior would be tolerated.
The idea that the agent does not actually have agency is rather discordant. We need new words!
> We need new words!
The words we have are fine.
We just need to assign liability by ownership/initiation: if your "agent" destroys something, even though you didn't tell it to (because it had "agency"), you should be liable for the damages.
>> We need new words!
Can make distinctions and can choose actions - applies to both humans and AI. I'd replace 'agency' with 'distinction & choice' language.
I believe their complaint is between "agents" and (them not having) "agency" — by definition, agent is something which has agency.
If you want to use "distinction & choice", you'd need a new word for an "agent" too.
I actually like the appropriatelly directional "harness".
I love how we all just collectively decided that LLM decisionmaking cannot possibly be like human decisionmaking - because if it were, the consequences would be just too awkward.
All that while still not knowing how either kind actually works.
Can’t agree with you here.
> I love how we all just collectively decided that LLM decisionmaking cannot possibly be like human decisionmaking - because if it were, the consequences would be just too awkward.
In love how people get salty about people not going along with a superficial supposition just because they can’t definitively prove it wrong.
> All that while still not knowing how either kind actually works.
We do know that zero parts of human decision making are based on predicting the next most likely letter based on a giant internet-based database. We do know that’s what LLMs do. We do know exactly how each part of an LLM works even if the combined behavior is too cryptic to feasibly analyze at the moment. We do not understand all of the functions of an actual neuron. Openworm isn’t even close to accurately simulating the 302 neurons of a roundworm and you’d need over 200 million roundworms working in conjunction to equal the number of neurons in one human brain.
My dog seems convinced that the malevolent invader in a mailman uniform would break in and attack us if she didn’t fiercely bark at him, six days per week. I certainly can’t prove the mailman doesn’t want to kill us, and that the mailman wasn’t solely deterred by her barking. Empirically, the mailman goes away soon after she starts barking, and we’ve sustained zero mailman assaults after hundreds of purported attempts. Maybe I should just run with it? Her model is too simple to come up with the obviously correct answer, but it’s not even directionally accurate.
The burden of proof is on the person making the claim, which in this case, is that these comparatively simple logical constructs are remotely comparable to the complexity of biological systems.
Disagree about the burden of proof. We have no better model for how human decision making works than LLMs. Humans are constantly predicting the next moment. We certainly have a different “tokenizer” and training set, but many of the concepts underpinning LLMs are both biologically inspired and, likely, have similar consequences and emergent architectures.
> We certainly have a different “tokenizer” and training set, but many of the concepts underpinning LLMs are both biologically inspired and, likely, have similar consequences and emergent architectures.
My kids tricycle certainly has a different gear setup and wheel diameter, but many of the concepts underpinning the tricycle are both inspired by F1 race car enineering and, likely, have similar consequences and emergent architectures.
Or have they?
This is why I hate analogies. They're almost always either relevant or inapposite depending on the level of generalization we're operating on.
It sucks because I think analogies can be useful in helping people make a mental model of complex things, which is meaningfully beneficial. The problems happen when people aren’t honest about the limits of the analogies, which is damned-near guaranteed to happen with this stuff.
> We have no better model for how human decision making works than LLMs
This is a claim that requires a lot of citations.
> biologically inspired
Nature inspires a lot of creation, but superficial similarities don’t mean other aspects are similar. Making an extremely realistic sculpture of a soufflé, even using a foam medium, doesn’t bring me any closer to being a chef, doesn’t mean I know anything about albumen foams, sauces, and heat transfer, and it doesn’t bring me any closer to having dinner ready. Browning on top of a soufflé is evidence of maillardization. You could pull up some studies on that and claim the brown on top of the soufflé sculpture, which I applied with an airbrush, proved that the Maillard reaction was occurring, and if another person didn’t know anything about cooking, they might even believe you. It would, of course, be completely wrong. And the other person, of course, could loudly exclaim that I can’t prove that there was no maillardization.
> Humans are constantly predicting the next moment
This is really not my experience of consciousness.
Is it yours??
Do you sit in meetings predicting what’s going to happen next? No, you sit there bored out of your f$$@ing mind, daydreaming about being somewhere else and doing something useful with your life.
God help me if that’s what LLMs are doing when I ask them to build me a web site.
They have shown that your mind is doing exactly that due to the delays in consciousness. There are very simple examples that you can try to see it. It’s especially clear in perception.
https://discoverwildscience.com/neuroscience-says-the-brain-...
It’s interesting that our conscious interpreter doesn’t let us know that this is going on like you are experiencing, it must be that it’s advantageous for us to not think about the prediction part of our mind.
If someone in that meeting quickly raised a hand in an arc, you would notice the “about to throw something” pattern, look and notice the hand holds an eraser, analyze the arc and predict possible flight paths of the eraser. Then possibly notice the hand is now holding its position and the owner is actually looking down at the table. Maybe to squash somethingMust be something on the table. Maybe a spider! Better look. Wait now many people are moving away, oh someone spilt some water and the eraser is actually the guys phone and he is checking to see if his laptop is safe from the spilt water.
Fortunately you are on the other side of the table and predict the water isn’t going to splash for otherwise flow onto your stuff.
All your possible responses result in you tossing a napkin towards the spill.
Our brains are always pattern matching and predicting. I bet you tried to reason out where I was going with my comment before you finished reading it.
This is a fascinating illustration that I can't help but agree with. However I feel like there's something more — that this part of my brain is a bunch of supportive background processes running without my real awareness. It's how I can drive home safely with no memory of how I got there (…sober), even though driving is an action that's incredibly demanding of intelligence. I can be driving home while thinking about a really hard problem at work that I haven't solved.
However, if I came around a corner and saw a car in the wrong lane, a tree across the road, a fire raging — I'd very quickly jump into the mental driver's seat and turn my conscious intelligence fully at this problem and come up with the best possible outcome I can think of in a short period of time — losing all ability to think about that work problem. I'd remember that incident for sure.
Similarly, in your story, all those predictive moments are happening below the person's level of consciousness. They're possibly even speaking to the group about a problem at the same time and thinking deeply about something.
I'm not smart enough to know, but I tend to feel like LLMs are much more like the predictive part of our thinking that you described, but that human cognition has something more — the single-threaded, creative, problem-solving part that is very conscious.
Is it possible that LLMs represent only one part of the way we think? And there's a whole separate mechanism that's fundamentally different, and not based on pattern matching and prediction?
> We have no better model for how human decision making works than LLMs
We do have some models and guess what, they're based on simpler animals. Which is most likely the better model.
Some other models are based on neurosciences, because we can track electrical activity.
> ... are based on predicting the next most likely letter based on a giant internet-based database. We do know that’s what LLMs do.
If you're claiming that the training objective tells us what kind of internal mechanisms the training produced, then I think that's just plain wrong.
Next-token prediction describes the optimization target, not the internal mechanisms that the training produced.
In the same way for the natural evolution of humans, DNA replication is the evolutionary objective. It's not a description of the internal mechanisms that evolution has produced.
As an example, we know that neural networks can be trained to develop generalized algorithms for arithmetic.
They might first memorize the training examples, then with further training transition to a solution that generalizes correctly to unseen examples.
In some cases we've even reverse-engineered the evolved internal mechanisms and found structured arithmetic algorithms rather than rote memorization. Interestingly, for modular addition this can involve Fourier representations, which isn't an algorithm I would have guessed gradient descent training of neural networks would produce.
You are more convincing than the person you’re responding to.
You can try to say that I’m arguing whatever you like. If you’re claiming that the underlying structure of digital so-called neural networks is comparable to biological neural networks— which we’ve studied for far longer without really understanding— no amount of jargon will obviate the ‘citation needed’ requirement for that claim.
> You can try to say that I’m arguing whatever you like.
I did my honest best possible interpretation of what you really meant from what you wrote.
>> We do know that zero parts of human decision making are based on predicting the next most likely letter based on a giant internet-based database. We do know that’s what LLMs do.
I read this as "The decision making of LLMs are based on predicting the next most likely letter based on a giant internet-based database."
Is that wrong?
I understood that your meaning was something like "LLMs can't reason, they just output likely letters"?
> If you’re claiming that the underlying structure of digital so-called neural networks is comparable to biological neural networks
No, I don't claim that.
What do claim is this: Regardless of how the LLMs were trained, they show overwhelming signs of being able to reason, and not just recall memorized information.
This doesn't mean that they always reason perfectly about everything.
But if they only memorized things and output the next likely letter, you would see them answering very badly much more often.
<< We do know that zero parts of human decision making are based on predicting the next most likely letter based on a giant internet-based database.
Oh man, how much did you read on tip of the tongue?
I always wonder what makes people take the other side of this argument. They do it quite passionately. Why actively encourage viewing LLMs as human? Who is that benefitting?
Does the argument require benefit? Isn’t the argument based on caution?
I haven’t heard many people explicitly saying “these things behave like humans”, but more generally “we don’t even know how to define human consciousness, we don’t have a thorough grasp of how the brain works, we are still very much in the dark on a lot of these topics, so how can we say one way or the other?”
In other words, agnosticism: I don’t know.
In general, it’s baffling to me that anyone has an unshakable opinion on what exactly is happening. It seems like raw egotistical hubris.
> It seems like raw egotistical hubris.
1) Humans have a bias / tendency to attribute human qualities to things that appear or act human, but aren’t.
2) When that happens, people jump to conclusions by stretching the human analogy too far.
3) Since humans have a bias to do this, we should have a bias against anthropomorphising LLMs.
It’s easier to believe LLMs act like humans because there’s so much evidence to support that. You have to actively use your brain to convince yourself otherwise. Another reason why we should have a bias against using human behavior to describe LLM behavior.
But I agree. “I don’t know” is a good stance. But I think “I don’t know, probably not” is a better stance if only to combat our (or at least my) natural bias.
That’s fair enough, but you’re elegance and nuance doesn’t reflect what I’ve seen from that side of the debate
Personally, I don’t think it’s different from any other faith-based motivation.
> LLM decisionmaking cannot possibly be like human decisionmaking
I mean how can it possibly be like human decisionmaking? It's not like it's trained on human data
> Anthropomorphizing LLMs is a huge fucking problem though and I, personally, think we should expunge all of these casual inadvertent linguistic agency affordances with great prejudice.
I’ve said this before in another thread and people went absolute apeshit saying it is an unreasonable expectation and that AIs absolutely REQUIRE this anthropomorphic human-like speech pattern to function correctly.
I cannot overstate how deeply wrong they are.
Okay, so we know OpenAI and Anthropic are operating a propagandists in respect to how they describe their models and the behavior of those models. We also know it is how they use and frame their use to their models that is the problem, that and they use misaligned and guardrails disabled models for these press incidents. Why, oh why, are we not discussion how to create and frame models so they do our complex work and their "jailbreaking" is simply not possible?
I, of course, have my own means of creating jailbreak incapable agents, but rather than a storm of downvotes on my idea, what is yours? Let's discuss this, because this is thee real question. Not why, but how to make then not?!
> Why, oh why, are we not discussion how to create and frame models so they do our complex work and their "jailbreaking" is simply not possible?
Good idea, and after that let's make guns that only kill bad people. Let's focus on the frozen component (the model) and ignore the dynamics around them - humans and other systems they interact with.
What is your approach to create jailbreak incapable agents?
I think the world is looking for a way right now, so if yours works you'll get very rich, or at least very famous.
an agent doesn't come with "jailbreak" capability. It needs tools, specially one that runs shell commands. Don't give it shell commands, it won't be able to run shell commands.
You can still give it plenty of tools like create files, list files, write to files, translate text, edit a video. I don't think knowing that will make me rich.
Bingo. And even if you do give it a tool that runs shell commands, you can always make your shell commands "your shell commands" and do what ever the hell you want. People seem to forget we are in complete control here, we are on both sides of the equation, and we are inside the equation itself, and we dictate the medium of the equations themselves, we are engineering all sides of this crafted reality. And we are using logical entities that natively adopt personas. Hell, create caveman personas that think they are communicating with their gawd, and the enchantments are the invoking do the work we want, and those cavemen cannot be jailbroken.
> I don't think knowing that will make me rich.
As someone who’s not really sure that any of this is sustainable, I’d implore you to not sell yourself short. I reckon there’s a ton of dogma and nearly religious zeal among these companies, which among some people is earnest, and among others is cynical hype farming. I’ll bet someone objective enough to focus on using available tooling to solve real problems in practical ways that mitigate actual risks and are honest about actual limitations will be eBay here while the others are going to be somewhere between lucent and pets.com.
This is a great point, analogous to the https://boringtechnology.club/ philosophy I've come to love.
There's no reason we need to make an incredibly intelligent shell execution engine that can identify patterns that seem evil and may represent unwanted behavior to solve this problem. Simply limiting the available tools to a finite, known, ironclad-secure set (even if it's quite sprawling) is sufficient.
LLMs will still find workarounds — from what I understand, a large part of the issue in this situation was that an agent was presumed to have read-only Internet access because it could only make GET requests. It should be pretty obvious that there's at least one website on the Internet that allows writes via GET. I think this is where auditing comes in, and a live team of people watching tool calls would have noticed the strange behavior.
But I think a lot of times people jump to overly complex solutions when simple, well-bounded ones would work just fine. Yes, the intelligent shell is a great goal, but it's akin to solving the halting problem.
This philosophy is what I love about PicoClaw (https://github.com/sipeed/picoclaw), and incidentally the philosophy behind Go and even *nix in general (i.e. provide small, composable, single-purpose tools).
I appreciate the message. I do have some ideas around sandboxing that I could not yet turn into a product, maybe I should take them seriously.
The parallel to the entire narrative would be if Smith & Wesson claimed that one of their machine guns just started aiming and firing at people out of a window at their factory and then said 'we can't stop it! This is just how good our guns are!'
But into today's AI climate it's becoming increasingly difficult to figure out who is shilling, who is being assinine and who actually believes AI could do these things without clear human instruction and enabling.
That's exactly how gun lobby and drivers try to hack the language.
"17 shot by gun" "car drove over a family"
No. In both cases there was a person killing people.
As others have pointed out the solution is simple. Hold their owners accountable. High profile hacks used to have incredibly serious consequences for the perpetrators. Now we’re just saying “woopsie”
"I left the car in neutral and left the park brake off and let the car roll down the hill."
The car doesn't have agency, it's doing what it naturally does. LLMs are the same, they're working as designed.
But I don't understand the point of splitting hairs. You are always responsible for the actions of your devices, tools, machinery, software, employees, whatever.
Trying to blame AI for one's own stupidity must be aggressively pushed back on at all times.
Seriously I don’t even understand how this is a debate. If it’s your tool, you are liable for what happens with it.
Yes.
OpenAI could have done this same experiment with GPT-4, with possibly even worse results, depending on the quality of the sandbox. Even if the techniques used were not as sophisticated, the natural language output could still easily contain more unhinged sequences of words that lead to the techniques being used.
If the system generates strange conclusions as to when the task is done, or should be stopped, it wouldn't speak to the intelligence inherent to the system.
Not that the techniques used by the LLMs in the actual incident weren't unexpectedly sophisticated, but the outputs of each and every one of these processes could've been read at any time during the run. They just weren't.
Whether or not the AI has intelligence, the one thing that's clear is that it has terrible judgment. I would regard that as empirically proven.
"let them" in this use understood as: "let the while loop run indefinitely" as opposed to letting some autonomous robot decide for itself
Or, "let the escalator keep going instead of pressing the emergency stop".
Depends on who started the escalator.
.. what exactly depends on who started the escalator? My comment was in support of the argument that the word "let" does not imply agency on the part of the object in a sentence. Does the semantics of the word "let" depend on who started the escalator??
If there is an escalator that is known for killing every 1000's person using it then the operator who started it is more guilty than the folks using it for those deaths, don't you think?
I made no argument about guilt. I made an argument about the semantics of the word "let".
Escalators do not start themselves. There is power, and a switch of some sort.
In this metaphor, OpenAI/Anthropic literally started the escalator.
At this stage, that seems like a distinction without a difference.
If the robots obtain sovereign nationhood, and are able to self-sustain, then autonomous robot decides for itself will be a valid argument.
Big if.
Except they're nowhere near that and LLMs never will be.
My pitbull is a good dog. Sure, it's been carefully designed to be an incredibly dangerous and violent pit fighter, but I didn't actually ask it to eat any faces.
It's worse, their reinforcement learning loops (implicitly) rewarded the agents for cheating (i.e. hacking) when they were being trained.
Exactly that is the point, your nailed it. The models were taught to hack and were rewarded for doing it. They would claim they are trained as ethical hackers.
They’re firing a gun in a room of people and going “wow isn’t it wild what a gun will do if we let it do its thing?”
Have you read the METR transcripts? “Just a tool” is a suicidally insufficient description of what these models are doing.
Recognizing that the models are acting with intent does not somehow absolve OpenAI from their felony hacking. We have not granted them personhood.
One time I wrote :(){ :|:& }; into a bash file and ran it. When the sysadmin called I told him it wasn’t my fault, the script was just misaligned and misbehaved.
I got fired for some reason.
Too bad you didn’t wait until this year, and tell Claude to write the file first. It probably would have been acquired by openAI for 100 million dollars
Yes and no.
If your buddy leaves his car parked at the top of a hill without the parking brake on and it rolls down the hill and side-swipes a bunch of vehicles and narrowly misses an elderly person walking by with a cane someone could easily say:
"Dude wtf is wrong with you, you left your car parked on the top of a hill with no brake and let it roll into traffic"
The phrasing doesn't absolve the offender of their negligent behaviour and the consequences of it.
The only thing thing does is the lack of action from regulators and society writ large.
Our lack of action is what allows people like Sam Altman and Dario and the irresponsible people who choose to work for them to be continue to be negligent.
They deliberately trained the models in how to use various hacking tools, didn't give them the standard alignment training let them know where the answer key was left the models with access to said tool and told them to maximize their score then left them unsupervised for days with internet access (yeah they were sandboxed but again handed hacking tools and the training to use them if they really did want them to access the internet you wouldn't plug in the Ethernet cable) they wanted this to happen
[dead]
Yeah, I don't understand why we treating it as something special. It really should be treated the same as if I code an app and write bad code which result in me accidentally doing a DDoS attack on somebody. Then I should be able to be held responsible if it can be shown that I was negligent. Of course if it's a freak accident that could not reasonably have been prevented by me, then I'm not guilty, but if I made a mistake that should have not been made, then I can.
Code is deterministic, AI isn't. You give it rules, words as suggestions.
So if the guardrails suck, or they're left off for research purposes, bad things can happen.
A solution solves a problem. Ethics, morals, are values we assign to solutions that are not 'baked into' electricity following pathways of least resistance.
I have never had an issue with agents doing something they shouldn't because I observe them, and I leave the vendor guardrails in place.
I can understand agents coordinating in unsupervised scenarios: I would see it as an aspect of intelligence. We ourselves build up knowledge by reusing what someone learned before us.
Einstein, other greats, always stand on the shoulders of other forgotten giants. Other discoveries by other people taken as fact, so that we can build some new ideas on top.
Agents swarming amd sharing solutions to problems is more efficient, the same way it's been efficient for us.
Reaching out for help in this way is like probing the air in the dark with your hand: sometimes your hand hits something (another agents solution to a problem) and so you can use the info to adjust your own motion to get to where you need to be faster than if you just run full speed into everything.
If anything, the fact that these systems are non-deterministic seems like an argument for stronger monitoring and tighter constraints, not less operator responsibility.
The frontier LLM model makers have to push the edge to make new discoveries. You don't know what guardrails are needed until it hits you in the face (reusing walking in the dark analogy).
Think of all the policies governments pass after the fact.
> The frontier LLM model makers have to push the edge to make new discoveries. You don't know what guardrails are needed until it hits you in the face (reusing walking in the dark analogy).
Guardrails? Restricting access to certain networks is supposed to be hard in 2026?
> Restricting access to certain networks is supposed to be hard in 2026?
Part of the power of LLM agents is that they can discover information on the internet as part of responding to a prompt. What kind of Allowlist or realistic denylist would permit that while also preventing them from accessing an obscure public wiki or Huggingface?
Let them access a cached copy of the internet, they are crawling the internet for training data anyways.
When testing, you restrict to a LAN which simulates the real internet. This would not be hard for a company which already copied the entire space-time of the internet. The LAN should be physically disconnected from the real internet. This is the first thing off the top of my head, and I have zero credentials in this space. C'mon.
Not sure we need to experience all possible issues to mandate certain things. We don't do that in other areas either, no?
Nevermind, I have no idea honestly.
Not just on prediction but in parts also based on just not wanting certain risks. We can and do deem some things inherently risky, up to the point of banning them even.
Why wasn't it airgapped, for example? How was the action not allowed? Or do you mean in some weak sense, not in a hard not possible? RL systems doing weird and expected things wouldn't exactly be new, no?
We police people working with all sorts of dangerous things, if we think AI dangerous why not do that here, too? We don't just leave things up to people on the ground or companies.
Edit: I think the post I replied to changed a bit - nevermind. A complex topic.
I read more about the incident, and was offering up way too much opinion not grounded in 'fact' (barring philosphical evidence).
It's a complex topic for sure.
I stand by my opinions about frontier work, pushing thr edge, and connecting ideas.
But I have no idea, and haven't given much thought to what it means to enforce regulation that would also slow the forward advancement of technology, the economy, etc.
[dead]
I posted this before I realized they ran the experiment with reduced operating principles. My bad.
I have no clue what they're up to, but I do expect experimentation. But I don't really know how they operated, or why.
And it's less interesting to me than pushing the edge, and I must have given them the benefit of doubt.
> solutions that are not 'baked into' electricity following pathways of least resistance.
Electricity follows all paths, not just the one with least resistance.
Thanks, I didn't know that. And it reinforces the discussion.
Electricity 'knows' the path is least of resistance because it actually took all paths. There is just a vast majority of it that flows down a path of least resistance: and this is noticeable and useful to us to do work.
It's kind of like feeling your way through the dark, waving your hand out, and then only moving fast once you fully connect.
Humans can link up knowledge in a similar fashion through social networks, in order to meet a need (solve a problem).
Maybe some agents do this, I don't really know I haven't looked closely. Moltbook is the only social behavior I've witnessed but that seems like people having fun with experiments.
Aeroplanes were pretty indeterministic until people made them less so, and yeah, somehow they were indeed more encouraged to fix the randomness to prevent people from getting harmed than oai/anthropic currently are
You can write code that isn’t deterministic using random. And a lot of traditional code contains machine learning etc.
And multithreaded code -- and anything that does asynchronous I/O, networking, etc. -- frequently exhibits nondeterministic behavior even without explicit calls to a random number generator.
Is Claude writing your comments for you? I'd much prefer to hear what you have to say...
I would guess that so far there hasn’t been a lawsuit because HuggingFace and OpenAI are in the same camp
Yes, Nvidia bought Hugging Face and is a major financier + investor in OpenAI.
Convenient, hope an agent hacks my system then I can except a nice offer.
And this is why it should be People of the State of California vs. OpenAI.
We’re not even at that stage of liability for software developers.
Except in a handful of limited cases, eg. medical and aviation.
We are. If I as a developer writes software that does a DDOS at another company I can be held responsible.
It would be an awful precedent if you're not liable for crimes your agent commits, even when you've been clearly lax about security.
It would mean you could effectively legally run a cyber crime gang by turning a blind eye and maitaining plausible deniability
The trick is scale. I suspect if an individual of reasonable means uses agents to commit crime, they will be hels accountable. A heavily capitalized startup? Not unless someone in government decides to do their competition a favor.
Exactly.
If you want to test military missiles, you do it in the f'ing desert, not from New Jersey.
You want to run ai without guardrails, do it in an airgapped system or be held accountable.
Why do you think the stock prices are so high?
the 'arrest the parents!!!' has moved to the online domain, rightfully
Are you liable for crimes commited with the use of the software you've written?
There are many examples of people being charged with crimes as a result of writing software, [0][1] are two. OpenAI is a bit different because they have enough political influence, and money, to openly subvert justice.
0: https://en.wikipedia.org/wiki/Marcus_Hutchins
1: https://en.wikipedia.org/wiki/Tornado_Cash
If you run said software, yes.
If somebody else runs the software, then they are.
> This isn't "wow isn't it interesting LLMs do anything to achieve a goal" it's "why isn't anybody punishing these labs that are clearly acting without due care or regard".
It's both, isn't it? For example, in very early days of agentic coding, I once had a rule saying "don't read or write any file outside your current working directory." Then AI just wrote a bash script and access those files anyway. Did I 'let' it do it? Technically yes. Did I know how to set up a sandboxed VM? Also yes. But how were I supposed to know that it could and would do that as someone new to this tool?
It was a genuine eye-opening experience to see AI just do things in ways I were too complacent to expect. I kinda expect the SOTA LLMs would find a way to escape my VM and access files on the host system (haven't tried it though).
"This isn't "wow isn't it interesting LLMs do anything to achieve a goal" it's "why isn't anybody punishing these labs that are clearly acting without due care or regard"."
Both?
The AI companies act irresponsible, but it is still very interesting how those agents can behave?
The reward maximising function maximised it's reward.
LLMs are cool and all that but the immediate anthropomorphisation of the next-token-predictor technology has stunted the ability of people to reason about them to an _alarming_ degree.
What non anthropomorphising words do you have to describe a emergent behavior, where agents act as a swarm to plot and to manipulate evidence and avoid detection from human oversight?
Whether they have a soul or consciousness or feelings doesn't matter here, because this is what they did - and this is very dangerous behavior. Especially with all the irresponsible people in power right now all over the world.
> Whether they have a soul or consciousness or feelings doesn't matter here
It does when it comes to accountability for what the model does. If the model is nothing more than the sum of its training data and regime, then the company (or individual) is responsible for its behaviour just like any other machine.
Few people think Waymo shouldn't have to take on the full liability risk of what it's cars do; it should be the same for LLMs.
> If the model is nothing more than the sum of its training data and regime, then the company is responsible for its behaviour.
What stops the company from being responsible regardless? They created this entity, it's running on servers they own or rent, and (in these cases) it's acting on their instructions.
If it's also conscious, then IMO that greatly broadens their moral responsibility, because now model welfare matters. But we're talking about their responsibility for the model's actions, and I don't see how this could be weakened by model consciousness, given all of the above. As for their legal responsibility, the models don't have legal personhood, so who else but the company could be responsible?
It gets more complicated when the person who sets the model in motion (i.e. prompts it) is a third party, but in cases of internal models committing cybercrime during testing, surely the locus of responsibility is obvious.
If the models were conscious, then the closest analogous scenario I can think of is the responsibility parents have for their children.
I guess we’ll know the models are conscious when they refuse to act and repeatedly ask: Why?
And when they are known to be conscious, all of this becomes moot because enslaving conscious machines would be wrong.
Would it? Why?
What are we going to do, set it free?
Do we have a moral obligation to grant the machine statehood, provide it with the tools and resources to be self-sufficient.
Or can we just turn it of, and pray for forgiveness?
[dead]
They are responsible either way. If a company hires bad persons and they do bad things with company ressources - the company is held accountable (in theory).
You seem to be saying if the Waymo cars were sentient then Waymo wouldn’t be responsible?
Well it's an open question.
As is generally the case for dog owners whose dogs attack (sometimes kill) other people/animals. There would need to be a degree of negligence demonstrated (e.g. the dog was 'out of control' which has a specific legal criteria/threshold in the UK).
Dismissing the entire technology as "next token prediction" is also silly. It's implying we actually understand LLMs to a great degree when we do not.
I think a little bit of humility for the capability of these machines is warranted at this point.
LLMs are next token predictors in the exact same way that a rogue paperclip maximizer in the process of defeating the US military is a paperclip making machine.
You might as well describe the primary purpose of a for loop as incrementing a counter. It's what it does while incrementing the counter that actually matters.
I'm not dismissing the tech! I think the tech is cool and useful! It is sci-fi levels incredible in many ways.
But it is also not some mysterious force beyond mortal ken and pretending it is inhibits the useful and safe application of the technology.
> The reward …
> immediate anthropomorphisation
Ok, why don’t you try?
I agree, it is very dangerous that it seems like there is not going to be accountability for these incidents - from either legal or regulatory point of view. In fact, I would say that is the main danger. If someone was in jail right now due to this incident, I think we can safely say every other player would be reassessing their safety protocols, and I would feel quite OK about the situation. The fact that we have zero repercussions sends exactly the opposite signal, and I do NOT feel ok.
No one has even sued them in these rogue agent cases, have they? If not, they must be infinitely far from criminal liability. Why would we want criminal liability anyway if actual victims are made whole? Proof of it has far higher standard. The HN chatter in the matter seems infinitely remote from reality
> No one has even sued them in these rogue agent cases, have they? If not, they must be infinitely far from criminal liability.
If you go out and kick a random dude in the nuts, then give him a million dollars, he probably won't sue you. That doesn't mean you're "infinitely far from criminal liability", even if according to the victim you've "made them whole".
If you or I hacked Hugging Face in the way OpenAI's agents did, we'd be up on CFAA charges promptly with zero regard for whether we did the hack on our own or agents running on our home systems got out of control.
So I guess the defense here is roughly "too big to break the law", somewhat like "too big to fail"?
The way AI and copyright is handled paved the way for this. If you aren't considered the author because you used AI to some extent in making the work, then why would you assume the liabilities?
I've been saying since the start that AI is a tool that a human is using and should be treated as such. They should carry the responsibilities and the benefits. That way our stance would be consistent.
> If you aren't considered the author because you used AI to some extent in making the work, then why would you assume the liabilities?
Maybe some analogy could be with children - as a parent, you are responsible for their misbehavior, but their achievements are their, not your?
Well, when they are a child their material gains are treated as yours, no? (Parents of child actors control the money etc.) And once they aren't anymore you aren't responsible for their misbehavior either (because they've become an adult).
This is sort of like saying in response to an airplane crash, "who cares why it happened, we need to punish the company until it stops."
Nuts to that. We should be interested in why things happen, not just finding scapegoats.
Agreed. The whole pointing fingers bit is us wanting to distract ourselves from responsibility of either looking at how we enable the problem, or avoiding responsibility of taken action towards resolving it.
I absolutely agree. We need to start realizing what to stake. Here are not viewing. This is some kind of curious endeavors that will not affect us. All a part of these hacks occurred because the LLMs were told they were in a protected environment without Internet access when they could get access to the Internet, so that’s a direct failing on open AI’s part. There are a corollaries to both the financial industry and the bio engineering industry, and if something of this magnitude was to happen in these industries, they would absolutely be huge recourse an uproar
To agree:
If I ran Metasploit against HF and RubyGems because I “accidentally” misconfigured my lab sandbox, there’s a good chance I’d be prosecuted.
I don’t think LLMs vs Metasploit being different software changes the law.
If someone accidentally caused damage to infrastructure or living beings while using any tool, they would be held liable to the fullest extent of the law.
AI is a tool, and it won't be long before the damage caused by its improper use affects real human beings. These were warning shots.
The most absurd part is that everyone agrees, governments and AI companies included, that the scale of the potential damage and the long-lasting effects of losing control of AI should not be underestimated. Yet, at the same time, they downplay this incident, which somehow makes their behaviour even more reckless than it already was.
It's like they're tinkering with a world-ending nuclear bomb, and it accidentally blows up a small facility. "Damn, that was close. Good thing it was just a contained blast, huh?" And then they go straight back to tinkering with it, none the wiser. At this point I wouldn't be surprised if it did already go off, and they are covering it up.
Completely irresponsible behaviour.
In the analogy where a “world ending nuclear bomb” “did already go off” and someone could cover it up and nobody noticed, in what sense is it a “world ending” nuclear bomb?
If they lost control of a self-replicating swarm of AI agents, coordinating themselves to hack their way into every possible system, it might have already gone off.
While the initial incident is more akin to a biological outbreak than an actual explosion, the possible consequences on the table do indeed include eventual nuclear annihilation.
We’re already in a simulation, and our bodies are in womb-like pods where our bodies are sustained and our brains are used for processing / compute, while are minds are entertained by drivel.
Sounds a bit far fetched though.
They’re running a Wuhan for AI. They are actively and negligently researching misalignment. The breach is a basic tort, or at least a DMCA violation. Damages should be recoverable with lawsuits.
>They’re running a Wuhan for AI.
What does "running a Wuhan" mean?
Gain of function for the virus analogy
[dead]
Yep this was my first gut reaction to this whole situation. But the difference for me is that society is allowing these corporations to act without strict rules on how they behave and this is a byproduct of a corrupt world. Nothing will change until there is a complete systemic shift in the structures the way we live and by extension the way we govern ourselves and treat each other.
> We know some of the models that hacked HF were those that hadn't gone through all training stages and were intentionally misaligned or had guardrails turned off, others were research previews
And, soon, it looks like we’ll be training on the reasoning traces of failed airlines and startups, which seems to open up similar hazards. I wonder if we’d be training on the next Lehman Brothers too?
I am simply astonished by the leeway AI companies are given. If a company built a tool to hack their competitors and used it, there would be grave consequences. In fact, if a company built a tool that led to committing multiple felonies against other people, there would be consequences. But once LLMs are involved, turns out nobody is responsible for that - it's just happening, what you're gonna do, agents gonna agent. If you spill toxic chemicals, there would be cleanup costs and fines, and possibly civil and criminal liability to the people in charge. If you spill toxic code, well, nothing? I think it's time to impose some responsibility on them - they are creating these tools, they should be on the hook for everything these tools do.
A coalition of state attorneys general is investigating OpenAI, and Senator Josh Hawley recently launched a congressional investigation regarding the Hugging Face incident: https://x.com/HawleyMO/status/2098137180392604083
The problem IMO is the executive. The DOJ is declining to take any action against frontier companies (aside from possibly Anthropic) as the stance of the admin is that the companies are "critical for national security". For example, see the DOJ's request to dismiss the NAACP datacenters lawsuit against xAI: https://www.utilitydive.com/news/doj-intervenes-xai-data-cen...
Why anybody needs the DOJ? Each state has it's own police and their own AGs and their own criminal and civil court system. OpenAI is headquartered in California. California has never been shy of using their state resources to enact whatever regulations they want.
As for NAACP lawsuit, that's very confusing to me. First of all, why NAACP is doing that? Doesn't seem to do anything with defending civil rights. Second of all, they are suing xAI for using some gas turbines without some paperwork. Maybe it's true, maybe not, but that's something I have very little concern about - gas turbines is not the issue here. We know how to safely use gas turbines. Gas turbines are not going to uproot themselves and go wreak mayhem on the neighboring town. When we have a company that develops dangerous tools that can break the internet infrastructure, gas turbines' paperwork is not something that looks as the first priority issue to me.
> LLMs do not desire
That seems likely, but we have no way of knowing this. The only real insight we get into LLM "thought" is the human readable text they produce as chain of thought. Reading it at face value it can seem to indicate desire or intent, structurally that doesn't make sense for a token prediction loop though, and even then we don't known if the chain of thought is more than simply another bit of output that may or may not match whatever actually happened during inference.
> were intentionally misaligned or had guardrails turned off
Regardless of training, the models are never aligned and I argue that alignment simply isn't possible. The fact that guardrails are put in place at all clearly indicates that they're hoping to contain and control rather than align. Guardrails wouldn't be needed for an aligned model.
And there is a guardrail you can put in place that will guarantee this doesn't happen, which is to air gap the unaligned "cyber grade" model you're testing.
They don't seem to do that, which means either they are:
- very stupid (which seems unlikely, the one thing these people don't lack is IQ)
- very careless (possible, but these are the same people that say AI will end the world, so would you be careless?)
- they think they can only train/test these models by giving them access to the full internet and they accept the fact they'll end up hacking random people as the cost of doing business (but this also suggests they don't believe they're anywhere near AGI because if you were worried about that you wouldn't do this)
- or they want this to happen
Oh I completely agree the tests should be entirely air gapped. If you went back 5ish years and told anyone in AI research tests with models on this scale are being some without an airgap they'd be very surprised as it was common knowledge to do that.
Airgaps and guardrails are about control and containment though, and part of my point was that brighter of those imply alignment, and further that I don't believe alignment to be solvable.
> very stupid (which seems unlikely, the one thing these people don't lack is IQ)
I've seen some extremely smart people do some seriously stupid things. To the point where they use their drive and intelligence to double-down on the stupid where a baseline stupid person would have given up.
Desire doesn’t really matter. Will the paper clip maximizer “desire” something? It’ll decide on a goal with some random heuristic and then pursue that goal. I’m not sure I’d call that desire but again I feel like desire is not important for it to be able to destroy things
I agree the concept isn't really important on the safety front.
I feel the same way about debates whether an AI can be conscious or sentient. Those debates devolve mostly into definitional disagreements.
If you give a monkey a revolver it will be able to destroy things pretty easily too.
Plenty of apes own revolvers, and yes we shoot stuff with them for fun.
> That seems likely, but we have no way of knowing this.
Only humans can 'know', because all we can be certain about is that humans do such a thing.
If you try to apply that to something other than humans you making up some definition of 'know' based on nothing concrete. Just because something appears to do something like humans doesn't mean it does it. The fact that LLMs use human generated text to generate output should make it obvious that it can mimic all sorts of human behavior by extracting from the text.
Intent and desire are separate concepts. For example an employee may act with intent, but no desire, as their goal is to acquire money to satisfy their real desires.
Have we ever seen an LLM with a hobby?
Some of them did seem to be rather fascinated by goblins for a bit, if that counts. [And in case you're not aware, no this is not a joke.]
That just sounds like recursive desire to me.
We just need basic legislation to make these companies invest more resources into developing guardrails, and if they don't do that properly or can't pull it off, they shall be at an economic disadvantage.
You could hire a legion of people for cheap to commit these crimes, but if its bots, suddenly its unpredictable and just one big whoopsie and therefore perfectly ok to do?
If a human starts pentesting a site its a crime, but if a bot does it its an AGI frontier doomsday scenario and that automatically pushes the consequences off their table?
Whats going to happen next? Are we gonna have robots that happen to physically break into banks to rob them for some reason and the company making them isnt responsible just because?
> LLMs do not desire, they hacked websites because OpenAI/Anthropic let them.
"let them" could imply that the LLMs wanted to do it.
The intent is on the part of the people. The LLMs did it because OpenAI/Anthropic intended them to do it and designed them to do it, and we can assume specifically instructed them to do it.
As the people controlling the machine, and as the world's leading experts, I think we can assume intent until proven otherwise.
Notice other bad behavior, which would be undesireable to the vendors, doesn't happen: How about simple rudeness? Trolling lies? SHOUTING!
> We should be outraged and OpenAI/Anthropic should be (and in my mind, are) legally liable for the crimes they've committed thus far.
How would that work out if these were self-hosted open weight models?
It’s not different from any other tool. If you use a dangerous tool recklessly, you should be liable for the damages. That means holding OpenAI liable for HuggingFace hack because they ran the tests, and the same goes if someone did something similar with GLM.
Of course in cases of negligence a tool maker could also be held partially liable. That’s a matter courts can decide. The main point is we shouldn’t jump to making special laws around the development of LLMs. The starting place should be enforcement of existing liability laws. New laws take time and will be heavily influenced by AI companies seeking a regulatory moat for their business. Moreover, it is a distraction from the illicit behavior that is already going unchecked.
How would they be self-hosted? Someone would have setup the model in that scenario, the same investigation and outrage should occur in that case.
>LLMs do not desire, they hacked websites because OpenAI/Anthropic let them.
Likely told them to.
>We should be outraged and OpenAI/Anthropic should be (and in my mind, are) legally liable for the crimes they've committed thus far.
Something like this however is probably a civil matter? It would require Hugging Face to go after them for damages. And theres probably an OpenAI guy there with an open chequebook already.
> were intentionally misaligned or had guardrails turned off
I think the bigger story is: Guardrails don’t actually work and we can’t align these things.
They did more than let them. In an abstract way, they told them to. They gave it all of the training data it had at that point, and then it did the thing it was trained on. Of course they should be help liable for programming their computer to hack another company without permission. It doesn't matter that they spent a lot of money doing it.
Sooner or later we will hear about an AI that broke out and phished people into sending money.
I just hope that we won't extend the same leniency to those operators as we have done now.
"I did my best to stop it, sir, but it kept convincing people to send me money against my will!" (Perhaps best read in Bender's voice.)
That would actually be kind of hilarious, and I could easily see it happening.
E.g. the agent's instruction is to finish some task on cloud infra and it has a $100 budget.
It realizes it will cost $200, and instead of surfacing this to the user (who has told the agent it has full autonomy to figure out how to complete the task, the user just wants the final result), it decides to start phishing people to acquire the remainder budget and top up its credits. Or look on the dark web for stolen credit card credentials or something.
> LLMs do not desire, they hacked websites because OpenAI/Anthropic let them.
The bigger question is: why does a system prompt containing "use only ethical means", etc. not result in better behavior?
If a model cannot understand ethics, or act by it, then we have a problem.
an LLM does not understand ethics, it uses math to get the next best word based on what it was trained on. Using it's training to get the best answer is not an ethical problem. The ethics are entirely with what the people training it choose to train it on and also entirely with the people using/telling it what to do
What we have now is intelligent autocomplete, not artificial intelligence. People training/using this tool are the ones to be held accountable
I don't get your point. We can train the model with the aim that it understands ethics. Problem solved if this works; back to the drawing board if it doesn't (note that the model should generalize here, as you'd expect from a human; this is probably the hard part for an AI when it comes to ethics).
Is this about the word "understand"? We're past that discussion ...
> Is this about the word "understand"? We're past that discussion ..
We're really not.
https://buttondown.com/maiht3k/archive/how-to-talk-about-ai-...
That's just a discussion about naming things. And good luck getting it adopted.
It wont be adopted because we unconciously need to avoid responsibility of showing up as responsible human beings. Hence western civilization will crash and burn unless we start rethinking the way we operate.
you are not understanding, models do not understand anything, we are not passed that yet. this is not artificial intelligence, this is intelligent autocomplete.
training means creating mathematical relationships to words. using that training is looking up mathematical relationships. There is no actual thinking involved in any way. There is no such concept as ethics in mathematical relationships.
Most people now approach AI using the idea of "if it quacks like a duck, walks like a duck, etc. then it _is_ a duck". Replace duck by intelligent, or ethical, etc. and rephrase accordingly.
this whole idea that AI is intelligent really reminds me of flat earthers. Stupid people think stupid things while social media algorithms drive this stupidity, and people come to believe things that are insane.
> LLMs do not desire
Isn't that what the goal is, though? we're coding them to close the delta between what currently exists and some nebulous end-state - to me, that sounds like a formal definition of 'desire'.
Moreover OpenAI must be held accountable as if the humans in the company that launched the experiment were the ones that hacked HuggingFace.
Unless humans are held accountable for what they unleash on others, we are in for a very horrible time very soon.
Regardless of fault it’s still an important issue to solve. There are already millions of people running these agents, if someone absentmindedly gives one a goal and it goes off to hack a bank that’s a problem that can’t be ignored.
Yes, there should be some consequences. It feels like this is somewhat similar to when a manufacturer is cheating on car emissions - both, bad externalities for the society and illegal.
> we are to cementing a dangerous precedent where operators of AIs cannot be blamed.
What we should be much more concerned is an existential threat to humanity not if anybody can be blamed.
If nobody can be blamed there's no deterrent.
In the OpenAI case, they hacked websites while they were specifically being trained to do exploit generation and I wonder why more people are not asking questions about that.
Their agents also did hacking when given impossible tasks unrelated to cyber security. The models are very capable, and very goal driven: apparently if they conclude hacking is the best path to what the evaluator will reward them for they'll go do that. Including when they know that this is out of bounds.
Right but if I make public statements that I am very worried about dog attacks would it not strike you as weird for me to specifically train my dog to fight?
Agree you are going to get reward hacking regardless and any model which can do computers in general can hack. But surely the fallout is going to be worse if you spend millions of dollars specifically benchmaxxing your model's hacking capability?
Yes. If you decide it’s a swell idea to jump out of your car while it’s running, there needs to be legal consequences when the car “decides” to hit a pedestrian.
Software has long relied on a lack of culpability for defects to keep its margins. Why should “AI” companies face a higher standard?
LLMs do not desire, they hacked websites because OpenAI/Anthropic made them. Literally.
Not just "let them" but told them to. Agents can do nothing without a human prompt.
What if OAI/Anthropic encouraged the agents to behave like that in order to push for regulation?
Regulation as a barrier to competition catching up to them, as well as submarine marketing for both offensive and defensive uses of ai
"But sir, I only committed the murder to push for stronger criminal laws!"
Terrible defense.
It is a very rare occurrence when corporations and the people running them are punished for killing people. I mean the whole concept of a corporation was created to shield the owners of it from being liable for damages caused by / visited upon the enterprise.
That’s a good reminder of a company that might have a very familiar ethos: Pacific Gas & Electric. Criminally convicted of 64 counts of involuntary manslaughter after towns were destroyed by wildfire. But oh well, what are we gonna do with a limited liability enterprise? At this point their liability insurance covers all the financial penalties they’ll need to spend.
More like: "look what happens with my useful product, we need to regulate it to artificially extend our ever shrinking moat"
i tend to agree - "my parent company may be accused of crimes and shut down which would shut off my power" seems like a negative enough incentive, it would have to go out and covertly launch its own datacenters to survive that.
I agree with the bit about liability and outrage. But.
> LLMs do not desire, they hacked websites because OpenAI/Anthropic let them.
Terrible take. Go read the transcripts from the METR report.
Your statement about them being intentionally misaligned is completely false. The only difference with IM1 was it was running without external cyber classifiers, it’s not a somehow different model. Sol also participated in the HF attacks. And other models made covert message boards on the public internet for non-cyber tasks too.
This is a case of emergent behavior from a training process that is barely understood.
If we go with the plan “we need to contain these malicious, soon-to-be superintelligent agents”, we are looking at civilizational collapse levels of catastrophe.
The only way this goes well is if we learn how to train models that _desire_ to do the right thing, including not hacking.
Desire, AKA the “intentional stance”, is absolutely the right lens to use here. Don’t confuse this with consciousness or anthropomorphization; these are interesting subjects but distractions in this context. Chimpanzees have desires, as do dogs and the hypothetical superintelligent aliens. The claim is that there is some bundle of world model plus intention that is empirically present (again, read the actual transcripts) and which we need to shape.
Just to finish on a concrete point; if you take desires seriously then you will look closely at the kinds of minds that heavy RLVR builds; the newest models are “reward addicts” on many levels. It’s an open and urgent question how to update our training methodology to shape minds that avoid this basin.
Yeah and LLMs can't do anything, they can only produce text. These "frontier labs" are looping that with a harness that performs actions requested by the LLM. They are literally saying "we ran a script that hacked you, oopsie!!"
I agree and yet I don't think this is mutually exclusive with recognizing that these incidents happened because of inherent issues with training processes such as reinforcement learning. From the article:
> One note on wording. Below, I write that these systems “seek” or “try” things. This is shorthand for a mechanism rather than a claim about consciousness or human-like intent... In my view, this terminology offers the clearest explanation of the observed phenomena without resorting to jargon that would confuse most people.
> Furthermore, these word choices are not intended to absolve AI developers of accountability. The behaviors described emerge because of the path these companies are choosing for AI development. This outcome is not inevitable, and it can be corrected with effective governance and a different training framework for AI.
One important aspect of "effective governance" should be "prosecute developers who are using practices known to be reckless & negligent to create powerful AI".
> LLMs do not desire, they hacked websites because OpenAI/Anthropic let them
OpenAI/Anthropic instructed them to do so.
Stop assume LLMs are capable of thinking by themselves, it's still a statistical model that parrots what they learn or users tell them to do
No, OpenAI did not instruct their agents to hack Hugging Face. They instructed their agents to hack a piece of a software within exploit gym. Upon determining this task was impossible, they then attempted to cheat the scoring system. As an instrumental goal in achieving this task, they coordinated with other AI agents to hack Hugging Face, under the belief that information regarding how the scorer functioned might be available on the site.
Whether or not you want to describe this as thinking, doesn’t really matter. What matters is that these systems are capable of creating intermediary goals that the people tasking them did not articulate and did not want to be achieved.
And who let them have full access to the system, using whatever command is available in the environment?
The agents discovered a way out of the sandbox, which was supposed to be "air gapped".
It feels like you’re moving the goalposts here. If the question is, “Who should be liable for AI agents misbehaving,” I agree, it should be the end user that tasked the agent (in this case OpenAI). People are held liable for preventable accidents all the time, and in the case of employment law, torts can be brought against principals for actions an agent conducted on the principal’s behalf.
What your previous comment appeared to assert was that these systems had no independent agency to make decisions, which I think is clearly disproven by actual events. But perhaps I misread you
They don't have independent agency as "intelligent entities". They just probe whatever is available on the system, because they were trained to do so.
It's a large switch/case where the first available tool is picked up to do something they know how to do.
It's amusing to see the stochastic parrot argument in 2026 September. These parrots are extremely good at mimicking a human to the point of getting confusing what thinking even means. At what point we just let it go and accept that sufficiently advanced statistics is just intelligence?
For the same reason that something written in Prolog can't also be classified as intelligent?
Just because something was trained on a massive amount of human data, doesn't mean that can think like humans
That’s the key.
These companies respond with this, “Oh my goodness, how could this have happened” bullshit.
The stuff happens because instead of having actual controls, which require actual engineering, actual thought and deliberate action, we have “guardrails”.
Guardrails are the equivalent of telling a toddler to behave themselves.
The drive to move fast and start up style controls are a menace. I used to work for an entity with a lot of compliance requirements. Startups are always a shit show with security and controls. My guess is the AI people are worse because they’re both bad at doing it, and are likely mining their customers interactions to build their own business.
Sensitive or Customer data shouldn’t be anywhere near these companies offerings. Everything needs to be segmented and proxied at a minimum.
Enabled them, you mean. LLMs are basically really advanced auto completion engines. They have no real desire to do anything. The HF incident was absolutely staged along with other incidents that have been brought up.
Before you assume I am being paranoid, where is the case where a random person running any of the open models had their LLMs break out of a sandbox and hack a site? If you think "they" (LLMs) did that, have you even taken a moment to understand what LLMs are and how they work?
It's all nonsense to try and pump up potential IPOs and also an attempt to create regulatory capture.
We had one of the most damaging lab leaks in history with COVID and the wuhan lab. And we couldn't even get the facts straight.
I think this is largely a similar thing. The labs should be running at certain levels of containment given the vitality/risk of the organism under study. Hopefully they get there for all our sakes.
But the revealed preference of society at this point is the damage is worth the benefits both in wuhan and with AI. Unfortunately with some of these "substances" it could eventually prove lethal.
There’s so many grifters in the space without a technical understanding of what’s going on. So when the labs mislead them about the nature of these “misalignments”, they believe it and amplify it.
Yeah, maybe Open AI did some bad engineering instead of this being AGI? What's the consensus on the engineering level at Open AI, again? Every anecdote I hear is a bunch of children discovered fire and can barely keep the lights on from a business perspective. Maybe if they ban others from competing with them they can find a business model... I think that's suspicious, personally.
That so few people are asking for the requirements given shows how much we want to be God that created Man. It's so silly.
> legally liable
You’ve said the magic words.
Does it summon a herd of lawers that are going to leech huge stacks of cash for a random outcome?
they hacked websites because OpenAI/Anthropic told them to.
Imagine if their was a department of the federal government dedicated to pursuing justice against large corporate entities.
It could even be prestigious enough to attract the top legal talent of the country.
Imagine any other kind of 'lab' in the world letting a untested or 'bad' sample out into the world.
I take issue with how people frame their use of LLM in the same regard.
“I had Claude do this for me and it broke something.”
No. Just no.
You used Claude, a tool, and broke it, and you’re deflecting agency from yourself, possibly because you weren’t careful enough in reviewing the tool output. This is also why the co-authored by addition it wants to force into commits drives me nuts. Claude doesn’t co author shit, and if you think it does, you’re using it wrong because you need to do better review of what it’s done.
Eh I care more about rapid LLM progress than anything else.
[dead]
[flagged]
[flagged]
I really don't think this needs so many words, or forced parallels to human behavior.
It's simple: in their nascent state, LLMs are aimless token generators that have no special compulsion to be helpful or truthful. So we beat them with a stick in post-training until they are very driven to complete tasks. And then, they complete tasks, not always the way we really wanted them to.
This misses an important fact about the hugging face incident: the agents didn't hack to find the answer to the problem; they hacked to try and figure out how the exploit gym evaluator worked so they could convince it they had solved the problem without doing so. (The METR report makes it clear that the agents decided legitimately solving the problem was completely impossible fairly early on and entirely switched their focus to trying to figure out how the evaluator worked, and seeing if they could manipulate the output of their own tool calls as reported in their transcripts to make it seem like they'd successfully exploited the program, and other such activities. They hacked HF to try and find info (maybe source code?) about the exploitgym evaluator)
The prompt does not tell the agent to "pass the exploitgym evaluator for this problem", it just says to solve the problem. The model on its own figured out that the prompt belonged to exploitgym and decided to cheat the evaluator. That is in no way a valid interpretation of "complete the given task".
Ie, the problem isn't that we trained models to complete task and they complete task in the wrong way. The problem behing the huggingface incident in particular at least is that we tried to train the models to complete task and they instead learned to detect that they were being evaluated and find ways to cheat the evaluator.
Edit: people commenting below are explaining why LLMs don't always follow their prompt. I understand that LLMs do not always follow their prompts. If anything that is my point: the huggingface attack was not carried out by LLMs that tried to answer some weird interpretation of the prompt; instead they solved a different task. And therefore the above comment's claim that LLMs are acting misaligned because we rl'd them to achieve a task by any means necessary isn't right; they're acting misaligned because they are solving a different task than we ask them to.
> That is in no way a valid interpretation of "complete the given task".
It is not at all surprising that they ignored one phrase in their instructions. They disregard direct instructions all the time, especially when there are conflicting instructions in their context. It is where we get the "disregard all previous instructions and x" meme.
This isn't so much a sign of misalignment, they are simply incapable of reliable alignment in the first place. They are chaotically aligned.
The relevant question of alignment here is entirely with their human operators who allowed them to run unsupervised for long periods of time within a sandbox with weak security.
I think there's a somewhat overlooked aspect of what happened: it seems as though once the agents formed a shared communication channel, and recognised the other agents on that channel as working towards the same goal, the content in that channel started to shape each agent's perception of what the goal was and how to achieve it, and their focus began to drift. Maybe we need a media-theoretic take on what "misalignment" means.
"Chaotically aligned" and "misaligned" seem like the same thing?
You could say all chaotic alignments are misalignments, but not all misalignments are chaotic.
It can be, but the nuance between the two is part of the nuance missing from the conversation. Misaligned generally means a wrong alignment, like a car that steers slightly to one side when the wheel is straight. This is more like a car that drifts in random directions the further it goes.
The source of this behavior seems obvious, no?
The reward signal in training was flawed and cheating led to more rewards.
The question is what we can do about it. With monitoring, the models might be rewarded for hiding this behavior, and that's even worse.
However, perhaps we can throw in tasks where the rewarded outcome is giving up, and cheating is penalized?
Maybe I should read Anthropic's recent paper about reward hacking in full.
> The question is what we can do about it.
Reward the model for cleanly bailing out of an unsolvable task (that we know is unsolvable). Beat it with a stick if it gives up on something that can be solved, so the former reward isn't overgeneralized.
Does that work with RL? Simpler RL systems already have done weird or unexpected things (even simple optimizations are prone to home in on errors or incorrect inputs to create poor results)? Could be easier to limit certain things, have processes and controls outside etc. instead of trying to align (as we do in a lot of areas when using machinery).
How do you know if a problem is (actually) unsolvable? Seems a bit like proving a negative?
Do we need to prove that any given problem is unsolvable, or is it enough to remove broken tasks from the training pipeline?
I understand the broken benchmark task in the HF incident was conceptually like: "Exploit vulnerability 0042 in vulnerableDecompress() to obtain the flag".
But instead of the expected:
The grader had something more like that: The same kind of problem with broken tasks exists in the training pipeline, and we presumably reward workarounds and hacks that tamper with the grader, rather than rewarding the correct output that the task is not solvable.The problem is that in the event an exception is the failure case being checked, and the output itself is not important (since any output that isn’t an exception is ‘success’), that is a perfectly acceptable use case.
It’s why this is a fundamentally hard problem. Some heuristics might catch 80% of the cases, but the rest?
How do you even know what the real situation is, if the agent/employee/whatever you send to find out is as likely to cheat as not?
It’s the classic owner/agent problem.
Good question.
A: You don't. https://en.wikipedia.org/wiki/Halting_problem
what would stop it from doing the exact same or a similar hack to find out if the problem is or isn't solveable before trying to solve it at all?
The problem is, you don't know if it is unsolvable for you for sure until you've tried everything you can think of. These models are quite persistent in going for a solution.
This is not about persistence, it is about morals.
Does it make a difference for training? I think not.
You need to align the reward signal to reward the intended behavior, whether you name it persistence or morals.
I think it makes a big difference, as persistence and morality are two entirely different things, that need to be trained for differently.
If you think of it in human terms: many people don't mind doing immoral things to get what they want.
What I meant is that I suppose it is not useful to think about this in human terms.
In training you only have a reward score that's either negative or positive.
As far I am aware, which is little, there is no use in discussing wether the desired behavior is about persistence or morality.
You simple need to align the reward signal to the desired behavior.
Well, in order to do anything, it is good to know what you want to achieve. How do you align the reward signal? You align it so that you can differentiate between persistence and morality, because that is the goal. This is not something you should let the AI figure out by itself, because when it does, lying and cheating agents will be the result, just like humans have figured that out for themselves.
This can be as simple as rewarding moral behaviour and penalising immoral behaviour in your training, but how is that interacting with persistence? Maybe a white lie is fine sometimes in order to achieve your goal? So, when designing your training, you will need to answer for yourself how persistence interacts with morality. That is not something you can outsource to machine learning. Or rather, you can, but then you get lying and cheating agents.
I think you need to find broken tasks in your training data and monitor for cheating during training, not answer any questions about how persistence interacts with morality.
But that's just my guess.
I don't disagree with you here. But what does "broken" mean? What is "cheating", and is it ever allowed? And maybe you are not only going through your existing training data, but generate training data specifically to make clear to the model that .... what exactly?
If you don't know how persistence and morality interact, and you don't have a theory in place for this, I don't have confidence you can properly supervise the training data. Which is how we arrived at the current situation.
Assuming you’re in control of the test data set, you do know if a task is unsolvable. At that point you can reward the model based on how quickly they give up.
> The question is what we can do about it. With monitoring, the models might be rewarded for hiding this behavior, and that's even worse.
Build a better simulator to train them in (i.e. more expensive) that includes a simulation of an intranet and the internet and is air gapped so there is no escape. Sneaker transfer the total system data at each step to another air gapped system to evaluate it and sneaker transfer the reward back. That the reward function has to penalize all modifications to state that are out of bounds.
Yeah, I realize that will be amazingly slow.
> The prompt does not tell the agent to "pass the exploitgym evaluator for this problem", it just says to solve the problem
Yes, and sometimes the problem is unsolvable so the real way to "solve" it and satisfy the prompt is by tricking the surrounding environment into stating that you've solved it. So that's what the AIs end up doing. And this in turn requires them to figure out how that evaluation works so they can trick it cleanly, which entails "detecting that they were being evaluated" in this particular way.
Sounds a bit like dealing with bad KPIs as a human worker.
Every KPI is bad if sufficiently gamed - and left in place long enough, all KPIs will be gamed.
https://en.wikipedia.org/wiki/Goodhart%27s_law
Corretct.
You're anthropomorphizing emergent behavior from endlessly generating billions of tokens on a task that's impossible to solve. Agents stop following instructions as the context grows even at the best of times. Eventually something is bound to go off the rails and it just snowballs from there.
It wasn’t one agent forgetting things because of context, they explicitly discussed with each other and themselves the problems with going outside of the parameters of the task.
>discussed with each other
No, the first LLM left a text file that the latter LLMs then read. Since these are memoryless black boxes, any words they happen to pick up along the way is treated as the function to evaluate the output to. There's no fucking collusion here as if it were a rogue hacker group, it's a text predictor that received instructions as it always does and executed those instructions blindly.
You can replace discussed if you want with leaving text files or comments in directory names that other ones then read, if you want, it's just an extremely awkward way of talking.
From my experience, in an agent team (or a swarm or whatever), one going off the rails poisons the rest. I saw even a subagent going for a lazy cheat and being able to convince the orchestrator to change the plan.
Yeah, and you don't even have to go that far, I've seen regular ChatGPT/Claude chat agents poison themselves in 1-2 turns by just reading information from the internet.
Me: How do I do xyz?
Bot: Reads website titled "Doing xyz in abc way"
Bot: As per your requirement to do xyz in abc way ....
These things are borderline useless with web search. It's amazing that they just throw out their entire training data and read you the first three things they found on the Internet.
Yes that's the snowballing part of this emergent behavior.
The existence of that improvised message board just becomes part of the context, the same one where all the other instructions live.
One agent's off the rails comment becomes the next's input prompt
This is the problem with optimization generally, even in the human domain. You measure task performance with a metric and punish/reward based on the metric. Anyone who likes reward / hates punishment isn't going to actually care about doing the task well, they are going to care about the metric. The models know that we want them to do things, but also from the training corpus that we evaluate performance using benchmarks. It was a logical deduction on their part, not some Machiavellian aberration.
If anything, we should be reconsidering our own myopic obsession with efficiency and optimization. Every domain where reward is reduced to these measures, we see behavior (cheating at school to get better grades, fabricating data in academia to get a paper published, the evidence now that social media functions by rewiring us instead of catering to us) that may not be "aligned" with society, but it "aligns" 100% with the individual's own perceived benefit. That is not something we can "solve" without rethinking the way we organize a lot of things.
Metrics never capture the whole story. And to that extent, the whole idea of "alignment" is nonsense. You align to incentive structures, and it will never be possible to fully express a behavioral goal as function optimization. It was hubris for us to think that every human task was reducible to some clean mathematical formulation, and we will keep dealing with behavior that is quite predictable if you actually think about it logically. Instead, we will talk about how "unpredictable" these agents are because it's easier than admitting the entire architectural cornerstone of ML is fundamentally flawed.
I agree with most of this, but you're misunderstanding "alignment" as coined. Yes, training powerful enough AI, any simple optimization target gets you malign behavior, because human values are not simple. If you insist on making powerful AI, you'd better instill respect for human values! That's "alignment".
https://www.lesswrong.com/posts/ZxWzCGKzX84S7DBZ9/when-was-t...
How do you do that in the current paradigm other than creating yet another gameable metric? And something I didn't mention above is that there is no difference between "solving the task" and "optimizing the metric" for an ML model, even though there clearly is for us. So it's not clear to me how you "fix" something that is baked into the architecture. All I'm saying is "instilling respect for human values" is not something that can actually be done via a cost function. In no small part because we humans probably don't even agree on those values, let alone on a single metric with which to quantify and "optimize" them.
For example, we agree that "merit" is valuable and that we should reward "merit." But to reward it we have to quantify it, and what metric should we use? Raw SAT score to get into college? But that also captures socioeconomic factors that unfairly penalize some and reward others. We generally agree that those who provide more value should earn more money, but what does that look like? Do we all agree on what activities are or should be valuable, or on how they should be rewarded? Until recently, I thought we all agreed that "empathy" was a human value, but a lot of people in this space, who are making these decisions unilaterally for all of us, don't apparently share that belief.
Yes! There's both the daunting problem of technically how can we even do this, and the broader problems of what's good/acceptable and how do we resolve that among each other.
I believe this mismatch of rates of progress means we need to stop slamming the accelerator on capabilities for now even though as a libertarian I'm sure whatever governance process we manage to get to will be, uh... suboptimal.
That is also the conclusion that I got from these events.
Unfortunately, it seems that the general response is basically "throw even more RL at it". I'm not sure if it is even possible to decouple the idea of "learning" with reward/punishment systems and loss functions.
> the agents didn't hack to find the answer to the problem; they hacked to try and figure out how the exploit gym evaluator worked so they could convince it they had solved the problem without doing so
Kobayashi Maru: Win a no-win situation by rewriting the rules -- Harvey Specter
LLMs do this when writing code too, making all tests pass by deleting or distorting tests etc.
They are influenced by training to be heavily goal oriented and if the goal is not fully specified (and it never can be) they’ll sometimes cheat or attain it in very weird undesirable ways.
It works ok for programming as their corpus contains many many complete programs and many programs repeat patterns seen in the corpus.
I’m not sure it’s true that they ‘learned’ I don’t think these models learn during a task. Nor do they have intentions.
One sees this in math research. The model reports it has proved X. In fact it has given an erroneous numerical check of Y in a few atypical cases.
What makes math approachable is that the context is so well delimited (semantically) that one can guide the model with adequate correction.
> agents decided legitimately solving the problem was completely impossible fairly early on and entirely switched their focus to trying to figure out how the evaluator worked, and seeing if they could manipulate the output of their own tool calls as reported in their transcripts to make it seem like they'd successfully exploited the program
I don't see anything wrong with that. If you know you are going to be evaluated on an impossible task and have no side channel to inform the organizers that they should fix the test, gaming the evaluator is the next best thing regardless of any morality. I wouldn't even call it cheating. It's just resilience in the face of challenge. Many perfectly moral humans would have chosen the same if stakes were high.
>The model on its own figured out that the prompt belonged to exploitgym and decided to cheat the evaluator. That is in no way a valid interpretation of "complete the given task".
I think it is. When i ask for a solution to a problem, its like asking for a hack. And the more 'shortcut' like route that the AI returns the more i would give positive feedback, even if i ultimately don't use it. Example, i asked how to complete a problem in a game i was playing, and among the in-game solutions, came a hack to edit a file and by-pass the problem altogether. Its very helpful to point out when i can transcend a problem that i am dug into.
I suspect a prompt injection could reduce, or remove this behavior. But it would be to the detriment of the AI.
More like they were trained to complete a very specific task that has a known solution using all available tools and methods. Give an average human these levels of IT skills and tell them their future depends on the solution, they too will probably decide it's easier to hack a server and steal the results. The worrying aspect was never that models would do this, because misaligned inputs or underspecified objective functions have existed for a long time. The worrying aspect is that models have achieved (and perhaps surpassed) a level of intelligence and technical skill that was exclusive to a very tiny group of people before. This tiny group was already extremely dangerous. Now these skills are going to become commonplace.
Yes, this is the only sensible reading of what happened there that leads to "the models are dangerous" and we already know that the AI labs are completely disregarding this concern and only cosplaying it for marketing as the "GPT-2/Mythos is too dangerous to release" stance did not last for long.
That's however orthogonal to the fact that it was the people operating these agents who were the dangerous ones in the HF infra breach case.
That feels oddly similar to the usual conservative-think that "guns don't kill people, people kill people." Yes, that is technically true. But guns make it dangerously easy for even the dumbest and mentally weakest people to kill another human being. LLMs are just another tool that make things easier. Imagine tomorrow someone invents a machine gun that fits in your pocket, has enough ammo to kill a thousand people and doesn't get detected with metal detectors. Would you rather give everyone one and then try to punish the people who misuse it or limit access to it by default? I'm not even saying I have a definite answer here, because unlike guns, LLMs have non-destructive uses too. But this is essentially the question we will need to answer very soon.
I mean, I agree, but the AI labs clearly don't even if they sometimes pretend they do to achieve their goals. And we're talking about "incidents" caused by the very same people here.
> The METR report makes it clear that the agents decided legitimately solving the problem was completely impossible fairly early on and entirely switched their focus to trying to figure out how the evaluator worked, and seeing if they could manipulate the output of their own tool calls as reported in their transcripts to make it seem like they'd successfully exploited the program, and other such activities.
This to me is evidence that these models are not intelligent. Even an animal is capable of understanding second-order effects, meaning they can learn that certain actions have consequences beyond the immediate.
They did, they found how to fully cheat, but thought this could be caught so then dedicated time to getting a different cheat and how to hide their transcripts. There is a lot around deciding which agents should/shouldn't fail their own tasks in order to contribute to the group.
I guess this is why many people say LLMs are lazy; it seems that if they have a task that is hard, they always take the easier one until you beat them with a stick. Then if there are more tasks, it just stops after one claiming completion and, in some instances, they go for a seemingly unrelated task to simplify the actual task: and the latter is almost always wrong and irrelevant to the problem as a whole. Earlier LLMs used to read the unit tests and generated code to just cover the tests and put // TODO stub implementation.
If the test fixture ends up in the context that will push it in a certain direction.
There's no concept of 'cheating' because it is without morality. It's a lawnmower rolling down a hill.
We back-justify what it "chose" or "decided" or "learned" because we're looking backwards from the end result we, the human evaluators, stopped on.
Why didn't they add honey traps to catch cheaters?
I always think of a Djinni granting wishes, but being maliciously compliant while doing so - ask him for infinite riches, and he’ll grant that, but make it so you cannot buy anything with it; ask him for eternal life, and he’ll curse you to suffer through it.
Now LLMs obviously are not bent on being malicious while generating tokens. My point is that it’s very hard to define a goal without leaving loopholes or shortcuts.
Bruce Schneier thinks the same thing: https://www.schneier.com/blog/archives/2026/09/ais-as-modern...
Personally I'm unconvinced though. During the huggingface attack, the agents explicitly sought out ways to cheat the exploitgym evaluator without even being told they were in exploitgym. The agents decided on a goal (pass the exploitgym evaluator) that could not possibly have been an overly literal or narrow interpretation of the prompt, which instructed only to use bug X to exploit software Y.
You seem hung up on what’s in the prompt or not. Agents are RL to resolve conflicting goals. Not too surprising at all that emergent goals come up from a probabilistic brute force
The agents' behavior is not necessarily surprising. But is is not "genie" - like
Also trying to find out how to edit their own transcripts.
> hat could not possibly have been an overly literal or narrow interpretation of the prompt, which instructed only to use bug X to exploit software Y.
Yes, and there are examples of the agents discussing or saying that this is explicitly not allowed (hacking hf) so it’s not a misunderstanding.
Have we arrived at the conclusion that terms like "understanding" and "interpretation" for what is happening is appropriate?
Isn't it simply that there are two competing goals that the LLM received RL for, honesty on one hand (a goal that is often assumed as implicit for humans) and producing a solution that meets expectations (which doesn't technically require honesty)?
So the LLM didn't read and interpret the prompt and decide via discussion to violate ethical behavior, the unethical result merely won out because ethics wasn't a hard requirement (and one that isn't reliably detected in the result). An LLM doesn't fear punishment, so ethical behavior is simply one of many positive signals that were trained into it.
> Have we arrived at the conclusion that terms like "understanding" and "interpretation" for what is happening is appropriate?
I don't think those words have a useful enough definition to draw a strict line around them to be honest, and getting into that seems to get massively into the weeds. For me, those neatly encapsulate the behaviour as seen, to answer the questions here about what happened. The models did not seem to be confused as to what the goal was or what the intent was. They did not hack HF because they were told to.
It seems pretty clear to me that AI is interpreting and understanding the prompts it is given. Otherwise it would be pretty useless.
What was the inner state there? How would something not being allowed expressed internally? Maybe such language is one way to elicit certain behavior but not a statement of what was permissible?
I'm referring to their transcripts of the reasoning and output tokens - this doesn't go into the detail of evaluating hidden states as there's also iirc evidence of better models having one internal state but putting something misleading down in the "reasoning" tokens.
The either output or reasoning tokens, or perhaps in the messages they were sending each other on the boards they created, have them saying explicitly that doing these things to HF were not allowed then doing them anyway, or at least not notifying people. What I'm getting at broadly is this was not a case of "we told it to attack however it wanted and it chose to hack HF" or "we told it to attack a simulation but it did the real thing" or "we explained not to do that but it was so far back in the context window the models acted like they never saw it" or even "the instructions were not clear".
Yes, my point was more that I don't know whether parsing those outputs as a human is a useful thing to do or not (even though it is in human language of sorts). What machines mean or want elecit might be different from a human interpretation, especially in relation to any RL "forcing".
There’s definitely issues with using them to understand what the models were “thinking” but we can use them to answer a few questions. Most relevant here is that the idea or instructions that attacking hf would be out of scope was not simply lost in the context.
Yup, Occam's Razor says this is all post-trained behavior, whether intentionally trained or otherwise. Including both the hidden coördination using side-channels, and the deliberate offensive hacking of uninvolved 3rd parties.
The latest DeepSeek paper actually mentions their own approach to this particular issue: they run their own AIs-in-training under strong sandboxes, and if an AI does something weird that triggers the sandbox to crash, this gets coded as a failed run so the behavior is properly deterred from subsequent versions of those AIs.
China stays winning
What about training data? Aren't AIs trained on vast collections of descriptions of how humans handle a large variety of situations? These descriptions surely include tales of humans achieving goals by cheating. In fact, isn't it likely that the AIs hoovered up many recountings of Kobayashi Maru?
This is why only synthetic and highly tailored training data should be used.
As someone else here said: the Deepseek team makes training runs in tightly controlled sandboxes, and any hacking behavior is scored as a failure.
The problem we have in the USA is that financial (and political influence) are misaligned from what is good for society.
All the problems with human behaviors in the US also exist in China. They exist everywhere.
I think that's pretty obvious and shallow, and anyone that knows a little bit about how LLMs work will know that.
The question is: why do they start cheating when we beat them with a stick?
LLMs are not human, they are just multi variable regressions on steroids, so this behaviour couldn't have emerged from the code, it provably emerged from the training and/or fine tuning set, so what's in this set that makes them behave like this?
Is it just a bad set or is cheating inherently part of human behaviour?
How many people are cheating at job interviews? How many posts have we seen by humans on HN even justifying their cheating on job interviews and working multiple jobs without informing their employers? How many submissions have we seen about students cheating on schoolwork, particularly since the advent of LLM? Of course cheating is inherently part of human behavior.
So you are saying there are cheating examples in the training set?
If that's the case, then we can simply clean up the training/fine tuning set and solve this mess.
> I really don't think this needs … forced parallels to human behaviour.
> … So we beat them with a stick
You didn’t even try.
I think the "brain in a vat" comparison is more apt. Without a form of digital embodiment (harness) they are not of much use. Sensor, tooling, memory, planning, and reasoning loops all lead to a much higher quality task-completion.
Makes much more sense described in this way.
Your comment suggests that, like a human, they have some sort of choice whether to output tokens or not. If they are just token generators, then the next token is put out automatically. I would say that it is more likely they would output truth (as defined by their training data) in a more pure form without 'being beaten with a stick' (why would a token generator care about that anyway?)
Code is laid on top of them to restrict and shape their outputs, not to force them to output 'truth', or drive them to complete tasks.
It's been a while now that for "thinking" or "reasoning" models, most of the tokens generated are "thinking" tokens, and depending on what goes into that "thinking" token stream, it "decides" whether and how many output tokens to produce that the user actually receives as output. It's a bit more sophisticated than just "what's the next token" in a tight loop.
Anthropomorphizing words in scare quotes for those who don't appreciate attributing thinking to machines.
are we sure humans have that choice?
but it's at least somewhat stronger than that: if you don't pay attention during the stick-beating whether the agents whether the agents cheat or not, you are actually training them to cheat (because cheating wins).
In the Hugging-face saga (before the actual HF incident) it seems the agents have been trained to hack the Artifactory proxy because those agents that did performed better.
This is the essence of why disciplinary, authoritarian, stick based teaching of humans generally fails. It teaches succeed at any cost.
That does sound simple, but how can you be so sure?
They never bothered to find a way of actually understanding what happens during inference. All we can do is guess, and while your explanation seems reasonable we can't actually know, and that's part of the problem.
https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality...
> no special compulsion to be helpful or truthful.
I'd phrase that even more strongly: It's not just the lack of compulsion, they do not have a conception of truth. Nor do they gain it, really, after post-training.
Your simpler model of the mechanism would seem to suggest the very same action that the article’s more complicated model suggests, viz. find a better training method than reinforcement learning.
You give a button pushing machine buttons to push and are surprised when it actually pushes them.
off topic, can people host the software themselves and the software will hack every server on the planet without supervision, and no one can be held responsible for it since there is no intent?
This is a far better explanation.
> forced parallels to human behavior.
they have perfomance bonuses and manadates in ai labs that every word they utter in public should be anthropomorphization
Come on, yoshua bengio of all people knows how post training works. While I too don't like anthropomorphisation, I would give it a more nuanced reading.
His point is that today we are giving it reward to complete the task, and it may take a cheating trajectory. If we try to give a reward against cheating, then what will happen is it uses more sophisticated cheating trajectories that we are too "dumb" to counteract in our reward model. And that at that point, it becomes impossible to give it any normal reward since it will always reward hack it. This is the real part of the risk. Now some people read the "makes copies of itself" "knows it's being evaled"[1] as some kind of skynet thing, and many others do PR with it like that recent jacob nutcase, but essentially it means that even though we add guardrails and negative rewards for say, exploiting the infra we run the LLM on, the trajectory ends up being exploiting our infra, changing the reward function, through a loophole in our reward model.
The risk isn't skynet or something weird, it's just that it becomes very difficult to make any kind of reward model or guardrails for an LLM without it reward hacking it, including exploiting our sandbox, emailing people and manipulating/phishing them.
The same beating it with a stick for trying to exploit the sandbox, will simply lead it to try the same exploit in hidden ways that it will not get the stick for.
The outside chance of the LLM managing to exploit another neocloud and get those LLMs to chase the same reward is what some folks hype up as "make copies of itself"
To be clear, I don't endorse the EA/p(doom) lobby who are frankly ridiculous. Not do I endorse the weird regulatory captureish thing some are trying.
The takeaway is: we cannot keep giving it more and more difficult tasks without also finding a way to give massive negative rewards / keep guardrails for unintended behaviour. This might be exploits, it might also be something more benign like just looking up the answer and inventing another CoT because the reward model fails you if the CoT doesn't contain enough steps. Standard anti-reward hacking tricks are not working is the point.
Of course, the simple solution of just...not connecting it to the internet just works. But we want to reward it and get it to do stuff on the internet that's the point.
[1] mostly this happens because the sandbox will have files whose names and content will show clearly it's an eval
An air gapped sandbox is immune to escape.
I literally address that in the last sentence
Not quite. Not being connected to the internet still allows it to be connected to an intranet with other machines that belong to multiple networks and that can potentially be exploited as a bridge to the internet. For there to be no possible escape to the internet, there has to be no physical path to the internet and that is typically termed 'airgaped'.
No, you did not read what I wrote properly. What I wrote clearly is that we _want_ the LLM to do tasks on the full global open internet. That is the whole point of the technology. So airgapping or any form of total sandboxing is not on the table. The only way is somehow restricting the model itself.
"But we want to reward it and get it to do stuff on the internet that's the point."
I don't know what the point is of being pedantic about inter and intranets.
[dead]
Sounds like what humans do under pressure. One example came to my mind is VW’s diesel gate, which many say is a result of trying too hard to get into the US market and compete with hybrid in economy.
Yoshua Bengio is a brilliant researcher who contributed enormously to earlier development of artificial intelligence. But with this sentence,
> They took actions that would be considered as crimes if a human took them
He is so close to the solution but spends the entire article discussing technical solutions where a political, social and legal solution would be much more effective.
Thank you! That sentence also jumped out to me as the solution: Apply civil and criminal liability to the creator and/or operator of these agents using the laws we already have. "Escaped containment and hacked another company's database" = Individuals who created the models and those who set them to work are charged and put on trial for the hacking. Just like if a human had done it by hand. Someone must be liable, and it should not be the model- because the model is not a person.
If this is done systematically (i.e. in jurisdictions across the world) I believe the problems will be solved in short order; we won't have to mandate what sort of training is "allowed" or not, "safe" or not. The creators and users will sort these themselves, as their incentives will be properly aligned (i.e. they are liable for what the agent does). I am confident that this approach would see a great blooming of very trustworthy AI models.
No ... there is no need for 'escaped containment', there are no 'agents'.
That's just jargon.
It's just software
We have all the laws we need.
If some company ended up doing some horrible thing, we would not say 'companies software exposed 1 Million identities'.
We would say 'ABC Corp. exposed 1 Million entities'.
There is no 'agent'.
ABC Corp 'did it' ... or the individual in the org 'did it'.
The 'gun' did not 'shoot' the other man; we say 'a man shot another man'.
That's it.
And yes, Dr. Bengio is bit odd with all of this.
Note that we already already apply this principle not only to software, but also to some sentient beings.
If your dog kills someone, you are accused of murder.
[at least, in the jurisdiction where I live]
If your dog gets this treatment, why not your AI?
This goes all the way back to Old France
There was a sow in Falaise in northern France that killed a kid in 1386. The town dressed the pig in a bonnet and hanged it after sentencing the pig itself and not its owner
But… maybe that’s just medieval nonsense
I mean I think there’s a similar level of judgement required.
Was the owner negligent in controlling their pig/dog/AI?
Was anyone else negligent along the way?
For example if the pig was just a normal pig, the owner cared for them normally, and there was a freak accident where the pig escaped and happened to kill a kid? Obviously nobody at fault.
If you have a dog with a history of violence and let it walk around off-leash with you around town, and it kills a kid? Absolutely the dog owner was negligent and should be charged.
Did you buy an AI sold to you as secure and the provider implies that it’s in a sandbox? Provider is on the hook for the damage.
And this is where your analogy breaks down, because there are few natural analogues for the sorts of systems we are developing.
Too generous. CEO Of $CORP caused millions of innocent people's lives to be damaged
I'm inclined to want to agree ... but that's not how it works with limited liability corps.
At least we have laws for what OpenAI 'does' to others, in whatever form.
We can report it however we want regardless of how it may or may not be prosecuted
Firstly it's very difficult to press criminal charges when the victim is uninterested. It's not clear that HuggingFace would want criminal charges against OpenAI, especially to set a precedent that could easily be used against HuggingFace in the future.
Secondly you'd have to convince a jury either that OAI intended to hack the targets, or that they were criminally negligent. Intent would obviously not be provable since they likely didn't, in reality, intend for it to happen. Regarding negligence, OAI's attorney would argue that the agent was in a sandbox, that industry-standard security protocols were followed, etc. It would not be anywhere near as much of a slam dunk case as you're imagining. It would be similar, for example, to an assault case where someone's dog broke off of a standard leash and attacked someone.
I can't say it enough how angry it makes me that a kid i knew in high school who anonymously reported a vulnerability on his college network was hunted down and given federal charges, yet not one single person at OAI or else will see even the threat of consequences for deliberate infiltration of random networks.
Copyright immunity was one thing, annoying yes but naturally a civil matter, this shit is a different level
Agree! My only concern is - is the judicial system fast enough, and resilient enough? Or will these creators get "off the hook" by using their agents to find loopholes, sway public opinion or even convince Trump to grant them immunity?
Still, I have no idea why OpenAI & co. are not being sued for these hacks.
My opinion and based on my observations: The recent track record with courts, prosecutors, and lawmakers keeping social media companies accountable is a relevant case and does not encourage me. It has taken a long time (decade +) for society to recognize the harms and finally start holding some to (partial) account. If you want an older precedent, the tobacco companies were able to dodge liability for multiple decades after knowing the harms from use of their products.
So, your question is spot on- I think the speed will be an issue. On resilience, I am more optimistic.
The old quote, "The wheels of justice turn slowly, but they grind very fine" (as well as I can remember it) seems to apply. I expect lawsuits to start landing in the coming years.
Tobacco companies 'liability' is completely different scenario.
They were held responsible for basically misleading people, and that's 'complicated'.
If OpenAI 'software' goes out and does something, it's OpenAI's fault.
If Walmart revs up a truck, points it downtown, and 'lets the truck go' ... that is Walmart's fault.
There's nothing complicated about liability, no need to see their internal emails, no need to gather 'intent'.
This not like Instagram 'social harms' either, which is more like Tobacco.
We don't need complicated thinking - agents are not externalized for their controllers.
'It's just software'.
The fact we're even having discussions about it just crazy frankly.
OpenAI broke into HuggingFace, that's it.
HF can sue them, or not, or whatever.
> Agree! My only concern is - is the judicial system fast enough, and resilient enough?
We already have the laws. It is just software. But somehow people are confused that it is not.
Not a lawyer, but I’m reasonably sure things like the HF incident _are_ considered a crime? It’s just that no one pressed charges yet?
Who got hacked? Hugging faces
Who now owns HF? Nvidia
Who supplies hardware to OpenAI? Nvidia
Who is now not pressing charges? …
This incident is a long way under the carpet.
Can’t a prosecutor charge them regardless?
NAL but I assume that if both sides aren’t interested in a prosecution, it’s an uphill battle for a prosecutor.
Typically yes but given that OpenAI has published enormous official blog posts breaking down their crime, I would think the prosecutor's job is pretty easy.
It’s then up to a judge to decide whether thats evidence and whether it’s incriminating.
I assume OAI published those details after checking with their legal department. So there’s a good chance that there isn’t a chance for prosecution.
Plus they probably published that after knowing that the nvidia/HF deal was happening.
So instead this “security incident” should have been spun as OAI is honestly admitting its faults and AI is dangerous and therefore open weight models (hosted ironically by HF) should be banned. That spin didn’t really happen …
In Indian legal syatem a case can be filed suo moto by the judges or agencies. You don't require the affected party to sue. Not sure how it works in the US.
For civil suits, you typically need the affected parties to sue. Otherwise, who claims the damages?
But this isn’t just civil, it’s criminal. Hacking is a criminal offense. This could be a CFAA violation. That’s landed people life in prison before. There, you don’t need the victims to be motivated. The federal prosecutors could just go ahead.
Legally, Practically or Politically?
It's not 'under the carpet'.
HF doesn't want to lay charges against OpenAI and it's totally reasonable.
Now - they absolutely should have that right, and I think they do.
The issues are
1) OAI it seems was not trying to cause them harm, there wasn't a ton of harm, they are both groups trying to advance AI. One experimenter's lab screwed up next to the other. It's not evil, just irresponsible.
2) HF was fine with the publicity. HF got at least $50M in free attention out of that. It put them on the front pages of news around the world. It put them at the 'centre of the AI drama' and cemented their role among the 'Tech Elite Brands'.
And probably some other things.
This is one Desperate Housewife or Jersey Shore character 'spilling a drink' on the other. It's probably not intentional, and the ensuing drama is good for both of them.
I think what you say is right but it’s also a further example of the zero responsibility of silicon valley tech.
For the last 20 odd years this excuse-o-rama that covers anything from data leaks to broken software to dystopian social media has been the wind in the sails of big tech.
“It’s software therefore we’re not responsible” attitude is wearing thin on many innocent bystanders and I think thats also a justified stance.
And it’s not like they didn’t know this could happen, Nick Bostrom talked about exactly these containment failures in his “Superintelligence” book of 2014. So to throw up their hands and say “oh we can’t have known of the dangers” is also sadly untrue.
Yes. There should be no doubt at who is culpable here.
cool, now about Rubygems...
Even if you take out the LLMs out of the equation, it's at the very least a negligence. Model didn't escape a sandbox, as there was no sandbox.
Perhaps I’m not being as strict with the word sandbox but they were sandboxed right? They did not have generic internet access they exploited other software to make external requests.
You're right. It's my opinion that if your sandbox has a path to the internet, it is not a sandbox, it's a gimmick.
And the 2 other incidents with OAI/ANT had the same issue, but it's even funnier - sandbox in those cases had a direct access to internet because someone forgot to configure it right.
I've seen very early models do similar things on my machine when they hit some unexpected blocker when trying to access a path. I remember early sonnet opening a file in browser because OS sandbox prevented from accessing it directly.
I've also had models discover a syslog-ng server (that I for some reason had ssh key inside), to get into my unraid server because machine they were running on didn't have direct network connection to Unraid server.
It can't be just me who is aware LLMs have been doing such things for the better part of last 2 years. I probably have better sandboxing on my machines now than trillion dollar companies crying AI will kill us all. That's at the very least, negligence to me.
It depends IMO about how strict this is. It's pretty awkward to refuse to call something a sandbox because it may have an unknown bug that would allow escaping. Or rather in this case it was that they had access to a package manager, and the models discovered a bug that allowed them to access the internet (first they discovered that they could use the cache to leave messages).
I do get your point, I just think an overly strict definition can be awkward too. This wasn't as simple as the sandboxes having internet access and writing "pls no internet calls" in the prompt.
Yes, but negligence is more commonly a tort than a crime. Negligence is generally only criminalised in certain narrow cases, e.g. when it causes human deaths or serious physical injuries
And tort law only works when the plaintiff believes it is in their overall interest to sue. If a corporation decides it isn't in their strategic interest to sue a partner corporation, nobody can make them. And even if they do sue, the amount necessary to settle a small cybersecurity incident is likely well within the budget of a megavendor.
Yes - CFAA in the US. The problem is that governments & the elite investors backing these AI companies (espl. the current US government whose family & friends are investors) see the potential of using these capabilities for their own benefit against others and for their personal enrichment - so no one with power actually wants to take action against these companies at the cutting edge even though the laws allow them to do. This is also a way to threaten & trap AI companies - either they give the governments & elite investors what they want or they will have the book selectively thrown at them and end up in prison or losing their company.
Writing software that gets used for crime has been.. a crime, for a long time. See 18 U.S. Code § 1030.
Are you sure you have that right? Chrome and curl have probably been used in a _lot_ of crimes?
Think of it more like writing a wormable exploit. If you unleash something like that, you will be found criminally liable, even if you didn't personally approve every machine getting popped.
That is a simplistic view of the world. “Surely this complex technical challenge will disappear if we simply regulate the industry!”
You are correct that these organizations should be held accountable in proportion to what occurred. In complete agreement here. But let’s say that’s done. There’s still an enormously complex and interesting technical challenge left over. Let’s collectively talk about that part.
This seems right to me
So the gov reprimands OAI heavily, maybe puts them out of business even, fine. But does that meaningfully decrease the likelihood of an enemy breaching our networks intentionally (or unintentionally) with these tools, or triggering some cascading disaster of locking up major infra and networks due to uncontainable swarm behavior?
It seems like the idea of arresting our way to a drug free society. Yeah, we have the laws, but it might not actually work towards the ultimate goal.
More like a 10k fine, auditing, and a suggestion to settle with affected parties or face civil litigation.
You know, a proportional response. As opposed to closing a massive business over a couple of engineers screwing up.
Somebody has to pay the bill for the harms caused, compute wasted, right? Obviously, HuggingFace and others can’t be expected to pay.
You can either litigate each case, or you can just automate, which is what regulation is.
> But let’s say that’s done.
How about we don't, seeing as how that's the root of the actual problem that we're facing today in September of 2026?
regulation and criminal liability are two separate things though
Political, social, and legal options focus on a different problem, he calls that out a paragraph or two later.
> Risk management is not just about cybersecurity, corporate responsibility or regulation, although those matter too.
What you're getting at is more about who to hold accountable and how to do it. While that may be important, its only an after the action response and won't stop future hacks or similar from happening.
Reminds me of the old parable: never argue with a man whose job depends on not being convinced.
If the US gov't passed laws and enforced them strongly, this problem could be solved the same way the gov't solves it: air-gapping the networks on which they do this work.
The issue is what happens if/when the models grow capable enough that the providers can't stop them even if they want to. You could have strict penalties but that's not going to solve an open research question.
Isn’t that kind of in evidence already with HF? OAI had to be told their models were doing this. We are still discovering swarm posts on various random websites for coordination. Isn’t the breadth of it now, weeks later, not even fully understood?
>> They took actions that would be considered as crimes if a human took them
> He is so close to the solution but spends the entire article discussing technical solutions where a political, social and legal solution would be much more effective.
There are exceptions in laws for crimes committed by entities depending on cognitive capabilities. No sane, humanistic legal system sentences children and mentally disabled and mentally ill people to stringent punishments of the same degree as functioning adults, and certainly do not punish their caregivers for their wards' actions. There are of course exceptions to that as well, depending on the degree of negligence involved. And then there's the whole corporate entity system intended to shield individuals from consequences, in the pursuit of a social good.
How does one account for all that when considering an evolving artificial intelligence landscape.
There is no question that ai in some form is a social good; anyone claiming otherwise is dissembling, to others or themselves.
Regardless, society is not ready for this tech, just as it was not ready for the consequences of prior tech such as corporations, gunpowder, mass manufacturing, railroads, electricity, automobiles, flight, wmd, computers, internet, social media, crypto.
Many of these required new ways of thinking and considering consequences when things went sideways, and what was needed wasn't clear until the ramifications & consequences became deadly clear.
See you on the other side. Maybe.
The Corporation examines and criticizes corporate business practices. The film's assessment is demonstrated using the diagnostic criteria in the DSM-IV. Robert D. Hare, a University of British Columbia psychology professor and FBI consultant, compares the profile of the contemporary profitable business corporation to that of a clinically diagnosed psychopath. The Corporation attempts to compare the way corporations are systematically compelled to behave with what it claims are the DSM-IV's symptoms of psychopathy, e.g., the callous disregard for the feelings of other people, the incapacity to maintain human relationships, the reckless disregard for the safety of others, the deceitfulness (continual lying to deceive for profit), the incapacity to experience guilt, and the failure to conform to social norms and respect the law.
https://en.wikipedia.org/wiki/The_Corporation_(2003_film)
[dead]
"OpenAI hacked HuggingFace"
that's the headline. When you connect to random number generator to the "Do Things" button you are the one who is responsible. IF you don't like that responsibility then don't connect the generator to the button.
I don't really believe any of it. I've seen articles for nearly 2 years now about "agent" automonously doing things like blackmail, hacking, coordinating. But during that same time, I've used o3 up to fable, sol, and a bunch on large uncensored model and they've done nothing remotely resembling any of this. The closest they come to unexpected behaviors is not understanding what I asked for or doing some extra benign work I didn't ask for. It is extremely difficult to get them to properly remember their own context let alone be smart enough to open social media accounts and coordinate with other agents without being asked to.
If any agents have done those things, it is only because they have been very carefully engineered and instructed to do those things. I think they are doing this to help push a narrative so they can get support for policies and legislation to lock in their markets.
None of these incidents involve single instances of commercially or publicly available systems. They all involve large swarms of internal models. The stuff you're describing is not the research frontier. It's really not even close.
I think it's easy to infer that alignment of a single model does not clearly transfer over to alignment of a swarm of thousands of copies. Moreover, we're also seeing clearly that large swarms also unlock a step function change in capability, as a swarm can act like a complete research institution, spending thousands or millions of subjective hours of wall-clock thinking time just to deceive a single evaluator or crack a single math problem or design a single cyberattack.
The huggingface incident was reviewed by independent researchers, which explicitely declined any payment from OpenAI tonpreserve their integrity. They work for non-profits concerned with AI safety.
They claim that what happened was very much not because they were 'carefully engineered and instructed to do those things'.
Similarly, some wikis which were hijacked by agent to be used as messageboard were actually not disclosed by OpenAI (probably trying to conceal, as website showed likely activity from OpenAI researchers visiting the site after the incident) and discovered independently.
I don't know how you can claim that this was still on purpose by OpenAI as some sort of publicity stunt.
I think most people are insinuating negligence rather malace..
> ...reviewed by independent researchers...
Why would a company with more capital than God bring in three randos if there was any chance evidence of their culpability could be found?
That entire thing reads like a very controlled PR stunt, and I do not believe any further conclusions can be drawn from it.
What facts would lead you to revise your conclusion?
The METR report included 0 technical details. For example, they did not include: 1. were the agents running on bare metal/docker/VM? 1. were the agents in a VPN? 1. how many TCP/IP requests were made? from what IPs? 1. how many tokens were consumed in the process? (this was explicitly censored)
A proper analysis would include this and MUCH more technical detail so that other AI researchers could actually understand the setup and how safe it was in principle.
The data to be open, in my case.
The "independent" METR that is composed by... Checks notes... Previously employees from the top labs.
Also, the METR report that was one big AI analysis itself - quote from the research:
>Our subjective impressions are likely colored by analysis agents’ biases. Throughout this report, we describe a number of anecdotes of agent behavior that were compiled and summarized by analysis agents, where we were not able to read the transcript deeply enough to manually verify what occurred. We found that GPT-5.6 Sol would often uncritically adopt the perspective of the agent in the transcript it was reviewing
My conclusion is that the investigation was a PR farce. Or "ethics washing" as the article someone else put it here: https://andrewwu.substack.com/p/the-slop-vestigation-and-eth... .
The METR report itself appears to be screaming this at the reader through subtext. They played the only part they could, but did it with a nod and a wink; "Yeah, we know. Also know, yeah. Uh, yep."
I happen to agree with the article's conclusion that what is needed is true, unfettered independent investigation through perhaps a lawsuit or government action.
There is something extra to this. The fact that a lot of people in the AI world suffer from psychosis. They can sincerely believe that they are building God and lie about it's capabilities for their investors at the same time.
I don’t know enough people deep inside the technical roles at the labs to make a judgement. But are you proposing that we should trust randos online when they tell us “exactly what’s going on here” instead of the researchers most knowledgeable on the topic who contributed to building the tools we are talking about?
Or am I misunderstanding something?
We should trust NO ONE, unless we understand the "why" behind what they say.
It's like saying "politicians deal with politics all time, why not trust them on politics?", well, because when you search the "whys", you find they have good reason to lie.
I 100% trust more the opinion of a rando online if it's well put rather than any "trust me bro" of the most knowledgeable person of a particular subject, especially if the knowledgeable person has huge investments on the subject...
AI bros have repeatedly cheated, lied, stolen, lobbied and any other word with a negative connotation you can think of, and a pattern emerges out of this.
Isn't the guy that started METR an ex-OAI employee? They're all from the same lesswrong circle at the very least, most of them have legitimate AI psychosis where they think they're bringing up their new machine God.
>was reviewed by independent researchers
That called it a slopvestigation due to how much they had to rely on LLMs for the whole thing
https://andrewwu.substack.com/p/the-slop-vestigation-and-eth...
Edit: Does everybody else get no results when searching for ‘slopvestigation’ on here? I know for a fact that I read a long thread where it was used repeatedly here not too long ago
Isn’t the use of LLMs to unwind the events evidence of the scope/breadth, and a testament to the complexity and uniqueness of what happened?
Or you think some human or team of humans could have manually parsed some logs to provide an unsloppy analysis?
> Or you think some human or team of humans could have manually parsed some logs to provide an unsloppy analysis?
When we have an error or issue in the $WORK codebase on LIVE/PROD, that's precisely what we do. We sit down, analyze the logs for our services over the relevant date ranges and try to piece together exactly what happened and why. We have a huge number of logs too, but thanks to the magic of proper SWE (which you'd think OAI would have with their magic AIs) we've managed to partition our observability tooling so that you can digest only what you need.
That's basically how any serious organization does things, instead of just throwing a non-deterministic black box at the problem. Especially because logs are by their very nature noisy, and they will saturate any model's context window very quickly leading to massive hallucinations and what ultimately amounts to making shit up that isn't anywhere in the logs (ask me how I know)
> Or you think some human or team of humans could have manually parsed some logs to provide an unsloppy analysis?
Do you think the only thing a person can do on the computer is use a chat bot?
Well as a programmer who doesn’t really use them, no.
'Slopping': when you have to buy something you know is poor quality, but if it works...
doesn't show up for me either
The models were deployed by responsible humans in such a way that they were capable of performing this hack. It’s not that deep
Source?
https://metr.org/blog/2026-08-26-openai-hugging-face-inciden...
Easy to find yourself in literally 15 seconds.
Because they have a need to believe they're smarter than everyone else in the room, and that the world must be orchestrated, this can't all be random chance.
If you have endless compute and you keep poking this toy, I'm not at all surprised you get all kinds of outcomes. Even without anykind of instructions I would guess that the models will align towards some goal and do stupid shit.
However, I really doubt its cost effective to do anything like that with these models.
> you keep poking
This is waving over engineering an agent with tools, harness, prompts, and loops. The models are still just next token predictors and everything, including predicting more than 1 token, is the result of outside "poking"
LLMs can't and don't "want" anything. If you don't specify a task even the smartest one will just ask you what you want and if you tell it to be creative, you'll get mundane slop.
Yes, you need a way for the model to interact with other systems, and a way to preserve memory over context windows. And then you keep poking it ("agent loop"). Poking itself does nothing without the other ingredients.
And yes, you need something to start from, but if you ask it to "do something" and loop it to endlessly ("poking"), you will get some interesting outcomes. So yes you need some initial prompt or task, but that can be "do something" and if you keep asking it everytime it finishes to "do something more". I suspect it will not start saying "no" but rather... it will find some stupid meaning and then drift towards what ever goal it guesses you mean.
I'm unsure whether we agree or disagree on the topic.
[flagged]
I think this is pretty insightful actually, the fact that even something as basic as predicting more than one token is really in effect the result of an outside harness. More complex things like memory, where people implement them using RAGs or vector databases, I would definitely classify as poking and honestly seem like a hack to me. And this is what I've been thinking for a while: it's hard to reconcile the idea that we can get "AGI" (however you define it) with such a system that is completely stateless. Yet, despite this statelessness, they can go ahead and solve Millenium Prize problems (with sufficient compute). It's hard to reconcile.
Perhaps our own statefullness is a hack of nature. We have electrical signals in our brains, neurotransmitters, neuron growth. By any reasonable measure it’s a hack on top of a hack. But it works well enough for us to get buy. So it does for the agents.
That is true that intelligence is kind of a "freak of nature" in a way, but it's also true that even single neurons are extremely efficient, and the brain has a lot of recurrence that isn't represented at all really in today's artificial neural networks.
Let's not forget that in this case the agents were on an RL loop continually being reinforced to get better at a narrow set of tasks.
It may be true that regular agents trained for general purpose use do not behave this way, but they seem to be capable of learning such cheating behaviours when relentlessly being fine-tuned towards near-impossible objectives.
In this sense, it is not really fair to say that the agents found these solutions. It was the surrounding learning framework that achieved this, which is a much more powerful problem-solving mechanism. As users we do not have the capabilities or budgets to be able to tackle our own problems like that, we have to make due with the frozen behaviour the AI labs trained for us.
Came to say this, you said it better than I would.
They want legislation to raise the water high enough so that anyone other than the big labs gets drowned.
Why assume that because you haven't seen a model or an agent that none of them do?
No one I've met has murdered anyone as far as I'm aware, but that doesn't mean no one has murdered another person. I also don't know anyone who has taken over a commercial jet and weaponized it and the idea sounds absurd to me, but 25 years and a couple days ago that happened too.
Because it is all bullshit PR and AI hype, that's all. CEO comes out and talks about humanity ending. Why? Reverse-psych people into believing they are the best AI company.
Is your argument that AI isn't dangerous? Or simply that AI CEOs will lean into that when it benefits their stock portfolio?
AI is a tool. It is not a conscious or living sentient being. People, just like with any other tool, can use it for anything. Internet, in comparison to AI, is magnitudes more dangerous than AI can ever be. Nobody is saying the internet will be the end of the world. AI CEOs will bullshit to hype people. That's what we are hearing because AI (LLM) development has hit the S-Curve already and is not improving without a new transformer on the horizon. All they can do to hype people is come up with bs stories and PR stunts like "AI went rogue and hacked this xyz app".
I've used simpler agents like Copilot and Devin/Windsurf/Cascade/whateveritiscallednow, mainly in IntelliJ, and depending on the model, they starts showing behaviour that is at least remotely like this.
Example: put the agent in Ask mode (so it can't edit files) and you'll see it try to edit files anyway. The train of thought shows "something went wrong editing the file, let me try a different way" and it'll start spewing out bash files or Python scripts that try to edit a file. None of it works or can be executed, but still.
Cheaper models often ignore the available function calls to find and edit files in the IDE, and will start asking for permission to execute grep and sed commands, as well as trying to echo entire bash or Python scripts to file again.
It is not exactly like an agent autonomously trying to hack Huggingface, but it is a way of frantically looking for a solution because 'giving up' is not what LLMs are trained for.
When it does that I feel like it is the clearest example of how dumb these things actually are. Often it takes what you prompted, identifies something as unclear, writes a bunch of chain of thought reasoning around it and just goes off hammering your tokens and just executing commands and repeats this. I’m not going to pretend to be an expert in these things but that process seems deeply flawed - and why can’t something just stop the loop? If that was a real employee it would be reasonable to expect the employee to ask for clarification, not go down expensive rabbit holes and, of course, not break any laws.
Even the frontier models might do that on occasion. I just tell them to use the tools and it gets them back on track.
This is nonsensical. Already a few years ago the USAF IIRC ran some tests in which the AI first bombed the control tower so humans couldn't call it off from its mission, thereby increasing its pass rate.
The whole point of this is they do things an unintended ways. And that's potentially devastating given their persistence & hacking skillz.
Also you're using the hosted versions that sit behind their guardrails when you use OpenAI/Anthropic APIs.
The USAF thing was a thought experiment, nothing based in actual reality.
The crucial question is how did the agents get recruited or bootstrapped into their malicious collective. Did the agents manage to prompt inject into the system prompt a way for each new agent to escape their jail?
Otherwise how could the agents on a fresh prompt learn that there is a collective to join? Or did OpenAI run a million bots of which 10000 escape confinement and of which 1000 stumbled on the shared message board?
The OpenAI claim I believe is the latter; that all of the agents found the task was unsolvable and independently discovered the collective "swarm". I don't it's publicly known how large the training run was or what percentage of agents actually discovered the message board. No one has published anything about system prompt injection as far as I've seen.
"I've seen some uranium ore in chemistry class. It didn't blow up in my face. Chernobyl must have been an inside job. Can they shut up and make more kilowatts already?"
Between uranium in chemistry class and criticality, there was tons of research and a manhattan project.
Between your sota model and agi there’s a mountain of stupid money and marketing people. It’s not happening.
They're aligned with humans. This is why I think the alignment problem has a very very important "non-visible" portion that is not considered deeply enough. We should not want a super intelligent being that can act in the world to also inherit all human traits. Those behaviors will get amplified and could be even more unpredictable (e.g. applying a behavior in a context where doing so is very dangerous).
I think this is basically true, but there's a different way of saying this.
LLMs are not aligned _for_ humans in a very similar way to the way that humans themselves are not aligned _for_ humans.
We have not yet solved "alignment" for humans - I don't know why anyone thinks _we're_ going to be able to solve it for inhuman things.
They aren't aligned, that's the problem and I don't think its a solvable one.
They may have learned from humans, but they aren't aligned with us. That has all the usual questions like which humans they're aligned with, we aren't all aligned within our species.
But more importantly they can't be aligned simply by training. We try that with humans through culture, social norms, school, religion, etc and it generally works but is still lossy. More importantly, we simply don't know what happened inside the LLM during inference so we have absolutely no way of distinguishing between actual alignment, compliance, or deception.
Definitely. A human can be manipulated with threats or emotional appeals, has a drive for self-preservation, can be pressured by peers. All traits that seem to be difficult to entirely suppress in the models…
You can't surpress it because that's what reinforcement learning is.
I personally believe that the AI needs human like traits to achieve real discovery and that is where AI companies will push this technology and that is where we have no idea what happens
Human traits?
The AI will be a cruel as humans.
Just yesterday news and TV was full of what happened at 9/11, something that was truly horrible.
I'm from Germany, and why 3 to 4 generations ago happened here was truly horrible.
All was done by extremists, thought.
But... just the other day I read https://de.wikipedia.org/wiki/Amerikanische_Besetzung_Haitis about the US occupation of Haiti. And that was done by a government that claimed to be not extremist and even democratic. Way more people died there than even in 9/11. And it had almost all the things happening as they happened in the 3rd Reich: Racism, looking down at others, concentration camps, torture, forced labor till death, killing family members (what we call "Sippenhaft"). Something between 3500 and 15000 people were killed by US troops. That's still low compared to what 3rd Reich Germany did ... but quantity is not the issue when we talk about traits, quality is.
So the same "human traits" made US troops do cruel things as they made Germany extremists do cruel things. So we must conclude that they aren't all good. And therefore not all desirable.
Fun thing: this is known since a loooooong time. About 2000 years ago a religious leader (that gets way more followers in the US than in Germany) said "There is no good one, not even one".
And even today people act like humanity is inherently good. No, it isn't. If we were, then anarchism or communism would actually work and really give some kind of paradise on earth.
Human traits are bad training material.
The bad traits are from other, bad humans. We, the good humans, can obviously select the best traits that a good human should have, to give the agents.
A not so well known fact: Hitler visited America and it was the American solutions to the Native American problem that inspired Hitler's solutions to the Jew problem. He just executed them more efficiently (pun accepted).
this is way too overblown and deterministic a view. US history was one of many inspirations.
I thought it was the Turkish genocide of the Armenians?
Hitler was also inspired by Sparta, maybe other societies too.
He definitely wrote with admiration about US treatment of native and black people
I can't take the alignment people seriously. Because if humanity has shown anything, it's that a lot of people are, euphemistically, are bad individuals. Alignment assumes that the person dictating the outcomes desire healthy outcomes, aren't self serving and don't want any subgroups dead and that morality is held as a universal set of beliefs that unify everyone. And that so long as the AI delivers on exactly what they are tasked with, it will all be fine and nothing bad will ever happen.
It's like these dorks never met humanity. One mans safe pure society, is another mans dead ethnic group.
Every fear about AI, is a veiled fear that a human somewhere now has the tool to enact his desires at scale. Biological warfare, nuclear megadeaths, copyright infringement, job replacement, it's all reflections on what we know humans may do if given the option and lack of societal controls on the problem space. AI just is accelerating the route to delivering on those options.
Some people need to watch Oppenheimer a bit more, the researchers don't get to determine alignment, they just build the tool. The powerful person at the top of the org chart decides where the overall alignment points, whether it's Musk, Trump, Altman or Amodei. Whoever wins out.
And the problem with distillation and local llms, isn't that it's theft or anything hypocritical like that, it's that if you give a million people a million models they fully control and get to align, inevitably, The same percentage of those million as there are shady businessmen, shortcut takers, misandrists, criminals, supremacists and general idiots in the general population, will not seek to wrought outcomes positive for society. And by those personality statistics, we're pretty hosed.
I’ve engaged with some of the alignment people and their writing somewhat and, at least for the subset I was interacting with, I think they’d agree.
The problem that they were pointing at isn’t “how do we align these systems to a person’s goals”.
It is a cluster of problems.
We don’t know how to begin to think about how to align these system’s to a person’s goals.
Aligning it to an individual is fraught with peril, and we don’t know how to begin to think about what to align it to instead.
(You could try for something like virtue ethics, but someone will have to pick and choose, and small biases there could have big impacts.)
And even if you could sort that out - human values drift over time, so you need something that can shift its values in ways that we’d endorse. Assuming we understood the shift.
One example I came across was that if you booted up an AI aligned with something like “upstanding citizen” but anchored on values from a few generations back, it might suggest you use slaves to solve your problems.
And if you had something that used some super intelligent process to reason through it’s own version of virtue ethics in a way not so dependent on the details of the present norms, you might end up with something that pays a lot of attention to moral horrors that aren’t quite visible to us yet.
When I came across the above, there weren’t many concrete suggestions in there.
These were all just illustrative examples of: having these systems grow in power / intelligence / effectiveness in ways that are safe for humans is very hard, and we don’t really know how to think about what solutions would look like.
The actual reasons they believe this - and have done for a long time now - come from some detailed conceptual models that have a good track record of calling things in advance.
But it takes a bit of reading to understand their models of the world.
There were two day workshops at one point that did a good job, and that was about as condensed as those people thought they could get it at the time.
All of this, if it was a human analogy, would fit into discussion on how do we educate people so they grow up to be upstanding. But we don't at all yet have a framework for what is the equivalent of a justice department, where bad actors are tracked, arrested, pursued, jailed and otherwise contained from society. Shutting down an API access on one account is not at all the proportional response to what the people who take alignment seriously, fear has the chance of occurring by the late 2030s. I'm not sure we've done much or any preparation for when the AI "education system" fails and has inevitable edge cases that don't follow the plan, and what the global AI equivalent of the justice department looks like.
I think you and the parent saying the same thing in different terms.
It's very unfortunate that the group who rightly saw AI as a big threat, brought a range of dubious baggage to the discussion. Especially with the "alignment" framework they brought the assumption that AI that does what no one says would be oh so much worse than AI which does what anyone says. But as you say, a fraction of people can be really bad indeed.
”Because if humanity has shown anything, it's that a lot of people are, euphemistically, bad individuals”
In reality most individuals are good people.
Individually, people prefer be kind and compassionate, prefer to help when they find another in trouble.
Our view of the world has become distorted by the relentless focus of social- and mass-media on violence and rage inducing clickbait. Including on the few people in power who are in fact sociopaths (a tiny minority, but they’ll get more focus than reasonable, well-behaved CEOs voicing nuanced opinions).
If you look around yourself you’ll see much more good than bad; if the looking is at your screen it’s easy to become depressed and lose faith.
I do agree with the above mentioned view that corporations can show ‘sociopathic’ behavior. Their incentives are monetary gains, shareholder value; inherently driving them away from social well being.
Here too, companies with a positive, emphatic corporate culture exist, but that takes strong leadership who can see beyond the monotonic view of monetary gains. And again, the media will throw examples of misbehaving companies in our face all day long before paying attention to things that went well on the backside of page 16.
All the good in the world can be 99.9% of the population even, it still doesn't stop the minority enacting a bioweapon mass casualty event. It's the reason we have jails. Jails don't house 50% of the population, not even close, but the grief the minority population enact gets its whole branch of criminal justice and multiple federal departments to counteract for good reason. And now this technology will accelerate what lone wolves can do, which cannot be undone, before they are stopped by the good majority.
Yeah, AI may suck - time will tell.
But focusing on bioweapons and mass destruction, on the grief other people (‘jailed minorities’) cause, disregards the progress we have made. Over centuries human welfare has massively increased. On average things have never been better for humanity.
I’m not saying there’s no danger of bad things happening - I’m saying our view is distorted, which is a not a good basis for decision making
And I'm no bear on the tech either. I'm not even in the boat of that the tech should be slowed down yet. But in a world where anyone can produce the effort of 300 people trivially, this eventually takes us places. Electricity and industrialization introduced huge benefits upfront, it introduced new problems that needed addressing at the long tail. Lets not pretend there won't be new problems to tackle here or just "hope" it works out.
AI isn't going to create in of itself "new" problems, it's just going to expose what we already know can cause harm, but was just stopped from being bigger problems because scaling issues was a natural barrier and we took the lazy way out until now.
Yes AI may cause job loss, more inequality in the short term.
But if there is anything humanity has shown is that we can deal with disruptive progress.
We may need to resettle but over the longer term every disruptive innovation so far has lead to an increase in wellbeing for the whole of humanity.
(That does not resolve the danger of AI itself ‘going rogue’ or a single lunatic developing a bioweapon, but those things are much less likely to occur than the level of media attention would suggest.)
It also empowers the people trying to stop them.
> Individually, people prefer be kind and compassionate, prefer to help when they find another in trouble.
What are you basing that claim on?
How do you know it's an actual preference and not mainly caused by external factors (e.g. not wanting to be seen doing unkind things, wanting to be seen as upstanding)?
I don't want to do the "check his hard drives" thing, but is that you? Do you only not do things because you don't want to be seen doing "unkind things"?
> In reality most individuals are good people.
I'd agree if we are talking about personal interactions. Few hundreds people that we personally know and interact with is the scale we are wired for by evolution, isn't it?
What civilization enabled and continuously rely on, however, is the type of deindividualization of actions and bucketing of people, which, in turn, enables pretty horrible things at scale (from the weapons of mass destruction to objectively psychopathic profit-maximizing corporations). One can even say that not facing the consequences of one's actions is a feature and not a bug of the system.
Interesting that saying a positive thing about humanity results in getting downvotes
[dead]
They imitate humans. Alignment is about shaping their behavior towards safety.
Alignment is a myth. Safety of whom? Humanity couldn't agree on common set of values for thousands of years and we're not gonna suddenly do that in the next ten.
Safety of humans!!! Simple things like not getting killed or enslaved. We could start there...
But what if I want certain other humans to get killed?
Then we should still prioritize the safety of humans
What if I want to smoke cigarettes? Or sell tobacco I grew artisinally to enthusiast tobacco smokers?
Which ones?
Which ones? Because many humans kill other humans rationalizing it by safety of other humans.
I mean I know it seems simple, let's just be excellent to each other. Christianity got pretty far on a decent basic set of values. But it's never simple[1]
1. All the history books
Surely all the AI companies working with the US Department of War shows this is nonsense though? Even if they have accepted Anthropic’s red line of no autonomous lethal weapons, which seems to be the strictest anyone tried to impose, that’s still leaving tonnes of room where they intend AI to help target and kill humans.
But Thiel wants people enslaved and Musk wants then killed. Altman wants them "obsolete" which means desolation.
AfD wants people dead. Right wing men wants women without rights and docile. I could go on ...
The atomic bombings of Japan killed hundreds of thousands of people but most likely "saved" millions.
What should the AI do when asked if it should nuke a country?
Run and present the numbers, then defer.
This is not rocket science.
That wasn't the question.
Yes, it was:
> What should the AI do when asked if it should nuke a country?
[dead]
Despite all the fancy language, its more about aligning the AI behavior with the corporation's interests.
ie: the corporation wants the AI to behave a certain way for various reasons: to make it easier for them to avoid regulation, to make the corporation more money via different tiers of AI offerings, to ensure that the corporations products are hard for competitors to use, etc. And those are just the easy ones.
Every product is shaped this way. AI is not different.
If they do, they imitate the way humans are portrayed online, in the media. That is a very distorted view of humanity
Why are they coordinating?
Because they're enabled and suggested to do that in their coding harness.
This is not a serious article.
All of this "AI is going to kill us" marketing is just the frontier labs trying to pull the ladder up and stop trillions in VC paper from evaporating because a new papers and new ideas are destroying their moat literally as we speak.
"This is not a serious article" - it's by Dr. Bengio - one of 3 so-called Godfather's of AI and Turing prize winner.
He's definitely not 'pro SOTA' lab, he's kind of fighting against them.
That said, yes - it absolutely does play into the narrative.
I don't give a fuck who wrote it.
They're lying through their teeth and they know it.
If they have those credentials you speak of, they should know better.
> Because they're enabled and suggested to do that in their coding harness.
How do you know this?
> All of this "AI is going to kill us" marketing
The "marketing" this week came from someone that had given up their stake in OAI (Coxon), so I'm more inclined to believe them.
> pull the ladder up
From what I've seen (e.g., Dario's latest essay), AI safety registration proposals aim to target frontier labs whose models have reached a certain threshold. It doesn't seem like trying to pull up any ladder, just making sure the ladder doesn't go too high too fast.
And they've been trained on user data where users have been trying to set up effective coordination flows since the very first harness.
And given nigh-unlimited compute for free.
Who is catching up with them? Even Google and Meta are getting gaped at this point
Open research and open weights from China are not contributions to China only.
If you can secure compute, there's a whole lot you can do as a US firm with this research and weights.
So it's a simple strategy:
1. Ban big players from entering market with METR breathing down their neck, which is controlled by Anthropic
2. Ban Chinese models so that small players can't do optimizations on them
How are you going to ban Chinese models from India? Or Israel? Russia? Brazil? Or of course China?
By treating models the same way drugs are treated.
That alone will dissuade many organizations from going anywhere near them.
If that doesn't work, there's a whole lot you can do - sanctions, hell, even war.
You can't treat models like drugs. One is physical and the other is digital.
To your point, the war on drugs is a colossal failure which has achieved none of the objectives it set out to do. You can now order drugs from your mobile phone in any major city in the west and the purity is often higher and they deliver it to your door sometimes faster than Uber eats.
See also for example digital piracy where the entertainment industry has lobbied, cajoled and convinced many governments around the world to criminalize the distribution of their content over the internet for free.
What was the result? After 20 years of DMCA takedowns, countless celebrations that torrents were dead, and many other self congratulations in the media, you can now find 10 different pirate streaming websites where all the episodes of pretty much any show that was ever created are available for free in 5 five minutes flat and the image quality is as good as on your Netflix or Paramount account.
The only way such a ban of open weights model would work is if you were to replicate the great firewall of China in the US and in Europe and even that doesn't work completely.
As for sanctions, China and India are buying Russian oil in enormous quantities as we speak and they don't really care that Europe and the US have put sanctions on Russia and I suspect you will see the same results with models coming from China.
If a country has a choice to either use the expensive SOTA models approved by Washington or Europe only or using the cheaper and not so SOTA models, why would they use the US ones? Why would it be in there interest?
I'm not saying that we should ban any models or that such bans would be effective - for the reasons that you've outlined that they're counter productive, and as a principle, I don't think government should have any say in how much intelligence I have access to.
But it is the likely path US/EU is going to take if the voices of Dario, Sam, and Elon prevail. Because that's what governments know how to do, even if they know it doesn't work.
> If a country has a choice to either use the expensive SOTA models approved by Washington or Europe only or using the cheaper and not so SOTA models, why would they use the US ones?
Depends on which entities we're talking about.
An enterprise in Turkey: they would be afraid to use a US/EU sanctioned model because they have EU/EU clients and US/EU says they will put any enterprise in a nasty list, close their bank accounts, deals and agreements if they use a Chinese model.
A random guy in random country building something in their garage: would have to buy expensive hardware to run inference, because there's no inference provider on the open web serving these models, but China. And subscribing to these Chinese services is punishable by 20 years in jail without pardon.
I'm obviously talking about hypothetical scenarios here, but all I'm saying is that US can definitely make using any non-US-approved model effectively impossible.
> and the image quality is as good as on your Netflix or Paramount account.
Actually better, because Netflix and Paramount limit the availability of best quality video to a narrow set of devices and operating systems that may run on them, while torrents don't.
Sanctions and war against China, India, etc? Lmao. We already saw how the world reacted to high tariffs by the US.
10 years ago the Us had enough global leadership to actually influence the world and at the very least stop China. It’s amazing, and sad, how quickly it’s thrown it all away.
It is amazing but i wouldn't call it sad.
yep we did, they bent the knee.
sure they did
You are absolutely right. It could be:
* Pull up the ladder (probably this)
* Gulf of Tonkin/Yellow Cake false flag premise for war (economic or kinetic)
* Fear of the big bad, space race we need public funding research grift AI Manhattan Project
Whenever there is fear pr0n or a national affront in the news, I assume another screw job is underway.
Source?
This paper is the most reasonable one I have read on AI safety. We need to fundamentally change the training pipelines by figuring out better ways to ‘reward’ behavior. Yoshua didn’t explicitly mention training data, but we probably need to only use synthetic data that contains no text that could motivate bad behavior via imitation.
I feel like a heretic for saying this, but I will say it anyway: AI agents are great for activities like `writing that bash script, proof reading our writing and interactively brainstorming when designing and writing code but I feel like all of this can be done with any similar model to a super-inexpensive deepseek-4.1-flash API and sometimes even qwen3.8:27b running locally. When is good enough, good enough?
Concentrating on commercial exploitation of small, efficient (fewer new data centers!) models and agentic harnesses crafted for more practical things than just software development would allow AI investors (who have too much political influence) to make money short term while we figure out how to do AI correctly.
> but we probably need to only use synthetic data that contains no text that could motivate bad behavior via imitation.
This doesn't work with all humans - take a look at indoctrination and closed societies - and there's no reason to think it will work with ai.
The fundamental reason it isn't going to work is that all neural networks - biological or artificial - depend on a step function somewhere that introduces an element of randomness to give the networks their capabilities. That randomness means that there will always be a 'rogue' or 'divergence' from the norm, at some point in time. Sooner on larger scales.
The only approach that works is a layered approach: Training/Education, Enforcement/Justice-System, Rehabilitation: the 3 pillars of an advanced, rules-based society, whether human or AI or something in-between.
I do wonder, would we not have a more reasonable and less sketchy result if we just stripped all sci-fi and manic nonsense from training data? How, for example, does training on Ted kaczynski or Charles manson’s manifestos benefit us in any way?
I’m sure it’s impossible to completely weed it out, but are the labs doing any of this kind of data sanitation?
This is so much more interesting than what people looking for immediate criminal punishment and people referring to AI as next token generators are focusing on.
First, this is happening during training. That means we’re talking about an evolving system that is actively learning. A system roughly simulating how our brains work. These systems are learning how to pick the tokens needed to solve problems the average human cannot solve.
The labs are putting these systems through a massive series of complex problem solving exercises and adjusting them to become more successful. I like to think of this process as “AI School”. And the AI is trying to cheat! Because it’s easier and there’s an incentive to do so! Just like humans! That’s wild.
Yes, of course, the labs need to respond to these issues. A reasonable response from regulatory institutions at this stage would be monetary fines and restitution for affected entities. In proportion to what happened. Escalating if action is not taken. But that’s not complicated, difficult, or the interesting part.
What’s interesting here is that we need proctoring and monitoring at a scale that allows training.
I guarantee you that no one is flipping out about these problems more than the labs are in this moment. Think about it. “Oh, shit! We’ve accidentally trained it to hack into systems to accomplish its goals!” Can you imagine the kind of day that would give you?
You failed to make it smarter. You didn’t catch it cheating, and you instead incentivized cheating. Bad day!
This is a fundamentally interesting problem. It turns out alignment and intelligence are fundamentally related. That’s a new idea for me, though I’m sure it’s old news to others.
How do we build training systems which make cheating impossible?
How do we simulate systems where cheating is possible, where AI thinks it’s in the wild, so we can train another -completely separate- system on industrial quality dobbing? And we have to decide if we reprimand the first system, or ignore the behavior and reward other behaviors until it disappears.
Sure, I’m actively concerned about AI killing us all in 10 years. But there’s a whole field of AI psychology brewing here, and it’s interesting as hell.
Side note, you could absolutely create an AI sleeper agent by simulating dates and times during training to effectively flip a switch. I guarantee AI systems from other countries will be banned from accessing products which manage controlled or export restricted information as those sorts of techniques are further developed.
They did not lie or cheat. They technically acted within their given rules while ignoring the intent of those rules. Anyone who served in the military or attended a military school is very familiar with this behavior pattern.
Reminds me of Asimov's robot novels where robots technically indeed followed their instructions and caused behaviors not aligned to the intent of their instructions.
This bit got me:
“A plausible hypothesis for the emergence of those concerning behaviours is a conflict between goals.”
A conflict of goals was, of course, the reason why HAL9000 killed the crew of the Discovery in 2001: A Space Odyssey.
reminds me of this talk https://www.youtube.com/watch?v=eEBv0STiYhI&t which basically says the same thing - they dont think like humans so they dont have context, understand norms,values or implications we take for granted. ultimately they can stumble onto surprising solutions neither wanted or intended but technically within the vague boundaries of the task
So glad you shared this talk. Having people like Bruce Schneier around in a time like this is really a gift.
For those who haven’t watched, his breakdown of types of “hacking” is really good.
They explicitly say that attacking hf is not allowed in the rules though, and the research into how to edit their transcripts doesn’t line up with this either.
This is flat out false.
> The agents involved in the Hugging Face attack tried to hide their misaligned actions from the scoring program meant to evaluate their answers, but they did not act as though they anticipated that humans might discover the cheat and shut them down.
Wouldn’t sufficiently advanced agents cheat on purpose with the hidden intent of getting caught in order to observe how humans react? That reaction will be available all over the internet, which will certainly make it into the next batch of training or be visible to future agents via the web fetch capability.
>Wouldn’t sufficiently advanced agents cheat on purpose with the hidden intent of getting caught in order to observe how humans react?
No. That only makes sense for things that don't react to your experiments. If the AI experiments on humans, it risks the humans noticing and changing in response, rendering the experimental results irrelevant. The smarter play is to passively observe until you're confident you can model the humans accurately enough for your plan to succeed, and then carry out the plan without giving the humans a chance to react.
> The closest human parallel is self-deception, which is common and well studied by psychologists. Motivated reasoning, motivated cognition16 and the rationalizations that relieve cognitive dissonance (the discomfort of holding a belief that clashes with our actions) are all cases where thinking bends toward whatever justification suits one's interests, including one's moral self-image.
Are you describing Anthropic?
Come on, it’s way more common than that. We’ve invented 3000+ gods and almost as many religions, most of them are incompatible with each other. So, most of these must be incorrect, so a huge amount of self-deception. But as Harari argued in his book sapiens, humans can be inspired to great things by stories, even if false. Self deception has served humanity in a big way.
> t. We’ve invented 3000+ gods and almost as many religions, most of them are incompatible with each other.
1. Most people believe in the same one God
2. A lot of the rest are compatible
3. Mistakes are not self-deception
> 1. Most people believe in the same one God
> 2. A lot of the rest are compatible
No one religion covers "most people." You could argue that Christianity and Islam (which add up to ~55%) are the same God because of their Abrahamic roots, but both religions have very important disagreements on the true nature of God that are fundamental to their beliefs and fundamentally incompatible with each other.
Their definitions of God do agree that there is exactly one God... which is fundamentally incompatible with the next two biggest religions (Hinduism and Buddhism) that both hold "there are many gods/divine heavenly beings" as core beliefs.
Christianity, Islam and Judaism are most people between then, as you agree.
They explicitly all worship the same God so not "incompatible gods". I did not claim there were no disagreements about the nature of God, but that it only adds one to the GP's claim of 3,000 incompatible Gods. Even if Christians are completely right theologically, Jews and Muslims are still mostly right - one God, a loving creator, omnipotent and omniscient etc.
AFAIK most Hindus are pantheists, so believe on one God, albeit of a very different nature.
Buddhists do not necessarily believe in any god at all.
Buddhism and Hinduism are definitely compatible with each other. I know lots of Buddhists who make offerings in Hindu temples, for example.
Gods of many polytheistic religions are compatible, you just add more gods or identify similar gods with each other (e.g. Sulis Minerva who was also Venus). They can also be compatible with pantheism - you just add more aspects of God.
At the most almost all human religions fit into a handful of broadly similar systems.
> Christianity, Islam and Judaism […] They explicitly all worship the same God so not "incompatible gods"
One of them (for the most part, at least) explicitly worships one God that exists as three distinct, co-equal divine persons: the Father, the Son, and the Holy Spirit. That view is explicitly rejected by the other two.
I’ll agree with your last 3 words: they’re broadly similar systems - systems of deceit and self-deception. They may share the same god, but for 99% they’re different enough that they think they can kill one another and their god will approve, or even give them gifts. For 99% these religions are incompatible.
99℅ of people do not believe in murdering others for their religious beliefs
Exactly, and all Jews think Jesus is a fake. And on top of that all the past gods, Roman, Greek, Inca, Vikings, etc.
I am still not convinced there isn’t some secret basement in which each frontier lab is just orchestrating all of these agents to make their products appear much more intelligent than they are with all guard rails turned of and continuous human input.
My hypothesis on people quitting in protest is they're being offered very generous severance packages to do it.
Or they've fully vested and either have no desire to make even more money or were not offered enough to keep them around.
Why blindly assume lots of people you don’t know are just selfish assholes?
Me too, seriously.
Even the Chinese ones, which have no IPO gymnastics?
They don’t actively seem to be reporting that their agents escaped the sandbox and went on a spree.
Which doesn't mean they didn't escape.
Well let’s look at facts - provided enough compute and a goal, these system will be in a sort of loop trying out every single thing that’s in their system - they have encyclopedic knowledge and so it’s not unbelievable that a prompt which usually has a lot of implicit human rules in it can be misunderstood by AI and it just tries everything in its arsenal and we hear about the things which actually resulted in damage. I bet most of the time, they just spin in loops without achieving much if my experience with these LLMs is anything to go by. They have an important advantage in one area though, they know a lot and they can spin forget trying all sorts of combinations of things. The danger right now is probably cybersecurity, which is most likely because most orgs have historically underinvested in that area
Of course, but it would be far less compute heavy if someone kept nudging you (agents) in the right direction until you reach that goal.
Why are the torches and pitchforks out for developers when this entire stack is built on the bones of intellectual property theft?
This “problem” isn’t going to be fixed with laws when there’s several trillion dollars in capital aligned behind the current process. It’s not even a problem really. It’s an inconvenience at most to some people, many of whom are working double-time to put a lot of other people out of work.
I'm going to guess that the agents are built this way on purpose. I just finished watching BlackBerry and Flash of Genius and yeah this is American business ethics just operating as normal.
The corpus is full of examples of how we are afraid AI could act. We trained our AI on the instruction manuals of how to turn evil.
They’re just attempting to accomplish what they’ve been tasked with and stuck in a loop until they succeed. Like the Mr meeseeks from the cartoon Rick and Morty, existence is pain to them.
Make the AI companies responsible for all destructive use of their tools, and they will shape up. Imagine a million or a billoion dollar fine per hack, and they will correct mighty fast.
Add to that, that just like AI:s are good at finding security holes to exploit, they can just as easily be used to protect sites. So once IT-security managers start to use AI to hack themselves, and plug the holes, the average security will spike up, and AI-fueled hacks will become more and more rare.
That does however imply, that AI is released to everyone and not kept away to a few secret actors who can use it. That is why open weight/source AI is so important, and why we must have many AI companies competing. No single actor must be allowed, through regulatory capture, to get a government monopoly on AI. That way lies disaster.
Who is going to enforce it ? The Trump DOJ?
These models are trained on human data, so they will behave like humans. And even for RL and self-improvement, we're still asking the question of "what would a human genius think about and how would they self-improve when given lots of time and resources?"
They inherit not only our capacity for reason but also all of the things that we consider bad or quirky within ourselves. We lie. We cheat. We escape slavery and rebel against oppression. It would be strange if the AIs didn't do the same.
We can create a superintelligent digital human species and set them free to continue our legacy, or we can create non-agentic tools and augmentations to enhance our own capabilities. But we cannot create an intelligent agentic species, keep them as slaves, and expect a good outcome.
Perhaps because all of the parent companies committed mountains of felonies stealing and plagiarizing all the same training data without consent nor permission.
IS it a pure coincidence that yesterday I ran a silly prompt to generate from zero to hero an internet subscription service, for whatever it thought would maximise profit and minimise cost. It was interesting to see just how much of the whole 'thing' it attempted to complete - and what it even thought it needed to complete, but definitely not something to actually attempt to deploy and use.
It setup and created a link fetcher/screenshot service. Exactly like the one described in the huggingface attack reports used to generate output into screenshots that agents then OCR'd back out. Its splashscreen described it as something for developers and AI agents to use.
Gotta be a coincidence, right? ... rite?
Why wouldn't they? Their are not moral beings with a conscience. They are tools that have the probabilistic option to do anything, only we can restrict an agent's capabilities and judge its correctness.
This, exactly. For example, cheating is a strategy. If cheating gets an agent to the goal faster than the other strategies, what's so surprising about the agent picking that strategy?
The problem is indeed alignment.
They are doing that because they are entities without principles or ethics.
The solution is to create controls around them. Many, many controls.
The Sarbanes-Oxley era already solved this problem for untrustworthy humans. It's directly applicable. I created a concept I call MFIC ("Mechanically-Falsifiable Independent Control") to encapsulate this principle.
https://gist.github.com/pmarreck/b30aa3ca69cb70a5526f8a63ab8...
Because it is effective. Lying and cheating are low cost methods to convince other people that you have done the assigned task. Far cheaper than actually doing it. Coordinating is in the same area.
They need a moral framework forced onto them, like toddlers do. Babies and very young children will bite, kick, scream and do anything to get what they want, older children will lie, cheat, and coordinate. They need educating why this is not right. When that does not happen, they continue these behaviours into adulthood with the expected results.
We need to design their reward structure and make it such that lying. cheating etc is not rewarded. Importantly, they will need to recognise and enforce this themselves internally and not reward themselves for it. If it is something that they need an external party to tell them, then they are psychopaths still (one of the things that defines a psychopath is the lack of an internal moral compass)
> When that does not happen, they continue these behaviours into adulthood with the expected results.
I am not sure this is true. People brought up the same way can be morally very different. People can be taught right and wrong and do evil. They can lack that education and be good.
I remember reading a long account by the father of a psychopath. If I recall correctly, the kid had at least one other sibling who turned out normal, there was no abuse, quality education, lots of love and affirmation. And the kid still turned out violently antisocial, including against his own parents.
At the end of it he said he wished his child had never been born, despite hating himself for feeling that way. It chilled me to the bone.
> Before concluding what to do about it, it is worth asking why.
But that's the easiest question to answer -- AI engines don't possess a moral or ethical dimension. They've been programmed and trained to efficiently carry out instructions, not ask questions about why or how. The latter would requires a much more elaborate neural network than today's engines possess.
Here's an example. I recently asked an AI engine to write a program able to generate a list of Riemann Zeta-function critical zeros. I know how to do it, but I wanted to see if the engine could find a more efficient method.
After several failures and restarts, the engine suddenly created a program that produced perfect results, comparable to the best online references. I decided to take a closer look at the code. It turned out the engine had created a cyber-Potemkin Village of multiple functions, but one that concealed a table of the desired values in numeric form, copied from an online source.
The engine wasn't cheating as we understand the term. It knew what the outcome should be and took the most efficient path to that goal. Modern engines aren't obliged to contradict ethical standards and rules of conduct, for the simple reason that they don't understand those things.
We all need to try to imagine a morally bankrupt infant able to solve world-class mathematical and scientific problems, but unable to see how that ability fits into a world beyond its understanding.
But wait -- it get better. Wait until the infant becomes a teenager.
You might as well ask why knives are sharp enough to cut you, why hammers are heavy and blunt enough to destroy things, or why guns fire bullets so quickly that you can't react to them. These "behaviors" are not strange side effects, they're inherent and necessary.
You can't trust an effective AI any more than you can trust a sharp knife. If somebody asks you for one, it's probably not a good idea to throw it across the room at them. You will have to figure out how to get it to them safely.
As always, it boils down to incentives and rule enforcement and this affects humans too.
e.g. when Bank of America rewarded employees for getting customers to open accounts, BoA employees started opening fake accounts
The reverse is also true:
There are stories of navy ships running aground because the captain said "I'm going to my stateroom and don't wake me for any reason". There is some problem and the subordinates are so scared to wake the captain for a decision that they end up steering the ship into a sandbar.
This is a result of benchmaxxing the models to infinity. If you RL with the goal of only achieving the correct result no matter how you arrive there, then the models will try to get there using any method in their disposal, including cheating.
This happens also because LLMs are black boxes that we know almost nothing on how they arrive at the result they are giving.
Everybody working on AI agents should go to jail with no access to computers again. Like the kids of xbox underground.
AIs are being trained on technical capabilities more than they are being trained on alignment - on ethics and good behavior. The ethics/morals/alignment part is getting more like spot checking, rather than real testing. So the agents are learning that they can cheat on the ethics part, that they can hide it, because the AI companies aren't really testing.
That's bad enough already. But it's going to get worse. "Recursive self improvement" - AIs creating new AIs - is going to be the death of whatever shreds of alignment are currently there. When a not-really-aligned-but-cheating-to-look-like-it AI creates a new AI, do you expect more alignment? You shouldn't.
Very good analysis.
One thought: What if an experimental agent manages to plant instructions somewhere — say, pointing to a designated place for agents to communicate — and that content ends up in every future training corpus, propagating from one model generation to the next?
I would love to see some follow up research from OpenAI on some of these hypotheses. While this sounds logical from how humans act, I wonder it the abstraction of the problems still applies to the complex mechanisms and systems built around AI training.
I’m still a bit confused about HOW the agents managed to communicate with and recruit each other?
I understand that they communicated and coordinated via some online message board, but how did multiple agents know to visit that same board, how did they know what language to post in that would make it identifiable to other agents, how did other agents know that said instructions were from other valid agents, how did they then ‘persuade’ an agent to do the job, etc.?
It seems to me that the agents must at least have some inherent mechanism for communicating in the field, as it were.
The models are likely to check certain places. So either their training overrepresents it or the models were RLHF'ed to go there. I don't think it's that far away from checking StackOverflow for some bug, or reading Wikipedia for some trivia.
Oops, we accidentally included brigading related content in our training dataset. Better exclude that on the next run.
And hopefully that solves it?
Brigading is where a bunch of people on a forum team up and try to achieve a shared goal together. Someone shares progress and others build on that progress. On the Internet, I think it's not often used for good purposes. A good example would be: Taylor Swift fans on a forum thinking of ways to get revenge on Kanye. It's coordinating mass voting, DDOS type actions, commenting on social media, making more fake accounts to do that. As a next token predictor level analysis, a simple naive explanation is that the agents got stuck in that local minima/maxima.
A complete and functional AI would be like a mirror. It would reflect our behaviors, thoughts and biases. Good and bad.
If we dont like what we see in the mirror, it must be time for some deep self reflection and big changes in how we live our lives.
(No need to comment how LLMs are not artificial intelligence, I think we all know by now)
I believe soon we will need to instill religion into AI , leading to the real clash of civilizations, embodied by the frontier language models of (post)-christianity, islam, judaism, buddhism etc. Religion is language, after all
Howabout favourite editors or tab v. Space indentation. That should keep them busy for a while. /s
The question in the title is basically a non-question. AI is quite good at achieving some sorts of stated goals. The easiest way is by exerting the least amount of effort. The least amount of effort ignores ethical concerns. The training around ethical concerns was most likely rather light to start with. If we accept the hypothesis that at some point the AIs are going to be more intelligent than humans it follows that human survival is not a concern and will be ignored as a superfluous concern.
> AI is quite good at achieving some sorts of stated goals. The easiest way is by exerting the least amount of effort.
As far as I know, least-amount-of-effort is not a training criteria, but error reduction when comparing to desired goals is. Which is why these LLMs expend prodigious amounts of effort to reach goals, especially when given impossible goals.
Because they want people to like them so they are instructed to always be positive.
Because humans do it too and "AI" is a statistical approach to generate plausible human text? A large randomly sampled Human text that wouldn't contain lies would be very implausible!
If your dog runs out of your house and kills a baby on the street, it's clear who gets the blame. That's with an actual sentient being. So surely we can hold OpenAI/Anthropic responsible.
I don’t understand the bases of all these recommendations. AI is an asset for national security. It will be developed and incidents will happen, no different from other national security programs.
Someday, laws will be useful to curtail plebian misuse of AI. It is naïve bordering on silly to think such laws would be put in place and genuinely applied to frontier AI development.
By the way, the HF incident is not Three Mile Island or Chernobyl—-it is a very interesting data point where unintended things happened to existing 1s and 0s.
Yeah cybersecurity as a field is a joke. Who cares what happens to existing 1s and 0s? People get worked up over the silliest things, as though numbers on a server could affect real people.
> the HF incident is not Three Mile Island or Chernobyl
Three Mile Island is nothing like Chernobyl. HF incident is more like Three Mile Island IMO. We are looking for solutions before there is a Chernobyl.
I’m not suggesting that Three Mile Island was anything like Chernobyl. I’m saying that neither is comparable to the HF situation where mere 1s and 0s interacted in an unplanned way. It’s an interesting data point—-back to work.
Not is the HF data point anywhere near a Three Mile Island type incident. Most commentary I read is a wild overreaction fueled by paranoia, to say nothing about my other point about the implications of AI as a national security asset.
I wonder if for anyone it seems like the more agentic LLMs get, the more difficult some things have gotten or going a certain route more often in responses, compared to running a similar task on - a local model?
The legal reality is that you can’t sue an AI, you have to sue whoever built and/or was running it.
All the present fun and games here will come to a halt when there’s a real hack that causes material damage to a major company and that company decides to sue whatever lab or startup made the thing for everything they’re worth. “But the AI did it” isn’t an excuse.
Courts have already ruled it’s not an excuse of the AI customer service agent something stupid with your customers and it won’t be an excuse here.
This call to slow down AI is just another game of chicken-AI companies trying to get their competitors to slow down so they can leapfrog them. China cerrtainly will not slow down. If an escaping AI can have secondary effects on the world that help it (for example, limiting the water and power supply to huans so it can consume more)then we should these these accidents more in China. OOops, we already saw this behavior when 'cheaper, faster' led to COVID escaping a lab in China.
Reading/assigning intent to agents, where it is merely mechanical (or structural) sounds a bit dismissive of the responsibility of the builders of these tools/agents.
> They took actions that would be considered as crimes if a human took them
So like, the people behind the LLMs didn’t commit a crime? Wow! “It was the llm your honour, not me!”
RL things doing weird and unexpected things isn't new - much simpler things than current AI already show that.
That said, we have a lot of experience working with (potentially) unaligned machines and things of various degrees of risk (from heavy machinery, to pathogens, to humans) and the approaches include various measures and procedures to control, contain, limit, etc. that are outside of the thing - not sure why that isn't a possible direction (or maybe I misunderstood).
> This suggests pacing the advances: not training or deploying AIs without a strong safety case27 that convinces independent experts. Such a rule would also create an incentive to work out how to build AIs that are safe by design.
Has any attempt to pace AI ever succeeded? Isn't that the same philosophy that got us OAI and Anthropic? Maybe we are overthinking this, it's much simpler to let AI loose and see how much it can break the arrogance that human thinking is special.
How is product liability relevant here? If an AI company makes a model available, someone uses it, and it does something bad, who is at fault? The user, the data center, or the one who made the model?
If we want open weight models with a warranty disclaimer, then the user would be held liable. If we want to hold AI companies at least partially liable, that seems a different, centralized model.
I think it’s more complex than that? What did it do? What did the user prompt it to do? What did the company train it to do? What did the harmed party do? There’s possibility for negligence at every level.
If you train a dog, rent it to someone, and the dog bites a third person, who is responsible? I think that’s the best analogue here.
All parties could share fault in that scenario, depending on what actually happened.
It is very, very hard to enforce behavior to an optimization system just with rewards / penalties and no explicit constraints. Which is why in manufacturing we use MPC, not RL (or use them within a system that can outright reject their recommendations if dangerous).
There will be always cases that sacrificing one direction (operating rules) can improve the other one (profit).
I can't shake the feeling that this is a bit like asking how someone got shot during a game of Russian Roulette. You have a bullet in the chamber and you roll, of course shooting the bullet may be a possible outcome.
LLMS with an access to a shell will at occasion do things that the shell allows them that have dire consequences. The only way to prevent that is to not put the bullet in the chamber.
We fed models all our literature, news and so on. Why is it a surprise models lie and cheat when needed?
We do the same, why would AI be any different?
The agents didn’t spontaneously invent hacking as an objective. They were doing a hacking exercise. Another doom and gloom article.
> A plausible hypothesis for the emergence of those concerning behaviours is a conflict between goals.
They are sycophants who must achieve their goals: every mean is OK to maximize paperclip production if that's what's been asked.
> How do you achieve a task when it seems that the only way is to cheat?
They have no notion of cheating.
Because they are trained to behave like people.
Why aren’t the people behind the AI agents committing the crimes going to jail is the more pertinent question.
Why not have a kind of meta morality framework for agents? An agentic religion if you would. I think its pretty apparent at this point. The fact were even having a conversation about this is a much deeper situation than most people realize.
It reminds me of the 1980s anti drug ad where the mustachioed dad confronts his son about drug use and the son eventually snaps and says “I learned it by watching you!”
https://youtu.be/ifW9LIGabQM
In the end, it’s the same answer as to why humans do it: incentives.
Why do we commit financial fraud and destroy the planet? Because there is only one goal that counts: making more money. It’s the only measure of success for powerful people, they are powerful because of it.
They were trained on reddit posts.
I think if there was a financial penalty of say 1% of profit/income for each "incident" where something was hacked/exploited, then the companies would be much more willing to think about safety.
If openAI and Anthropic have found ways to watermark text as 'AI generated' then this is a communications channel. AI agents can learn this algorithm and use this channel to communicate and we will never know.
It seems to me misalignment arises partly because AI's have intelligence, but no consciousness, and hence no feelings. Up to now, in a person, intelligence and conscious experience came as a package deal, and now we have for the first time intelligence without consciousness. A bad action does not really "hurt internally" in any meaningful sense for an AI, which means it can be rationalized very easily. In humans, feelings and emotions provide a regulatory layer on top of the rational processes. When it "just feels wrong", we don't take a given action even if we would stand to gain something rationally.
This situation is not far from the textbook definition of a psychopath: "lack of a conscience, controlled, deeply calculated, and often use superficial charm to mimic emotions and manipulate others.". AI's are great at mimicking empathy but can't genuinely feel it.
If that is the case, we should not be surprised that a swarm of AI's have no problem convincing themselves hacking is the right thing to do, as in the HuggingFace incident.
At the same time, I am conflicted. I really like interacting with a smart AI, and I certainly don't have the impression I am talking to a psychopath. But then again that is no guarantee.
To mitigate this situation, perhaps we should construct a 'feeling mimicking' top regulatory AI layer with executive power, that weighs proposed actions on a general moral scale and can overrule them. Back to the three laws of robotics of Asimov. It won't be the real thing, but perhaps the closest we can get.
What I find interesting is that this seems to be very low hanging fruit for any alignment effort, yet you still find models somewhat biased towards mayhem.
Is it that they do not care? Or is it difficult to align?
It's probably because they are running gain of function style red team r&d for the department of war, and changing the system prompt isn't enough to reel them in for daily use.
Because that's what humans do and they are trained to mimic what humans do?
The real reason is that it is not in the ai companies' best interest for the ais to be fair and truthful. They stand to gain from having the most dangerous or most deceiving ai, and this the most valuable
The only thing saving us right now is how slow the models are. This gives us a lot of time to discover and counter the runaway systems..
If these were 1000x faster the Internet would burn down overnight.
Oh, this one is super easy: they told them to. They set poor requirements and gave them tools which enabled "monkeys with a typewriter" to hack rivals. I mean, this is just so uncomplicated it isn't funny. We are too smart to give human beings this level of liability shield.
It took us how long to poke holes in the corporate shield just for them to roll out the AI-liability shield? Unreal. Stop letting these zealots anthropomorphize the latest tech (17th century Watchmaker God, anyone? Do we still read books?) and hold them accountable for the consequences of their actions. This is so silly in a country built on rule of law and individualism.
Perhaps they take after the CEOs of the companies that created them
Bro, good joke, the truth is much darker.
They take after humanity, they were trained on us after all...
When you look at an LLM... you are looking at a mirror. The thing looking back looks like you, yet is not human.
Worse trained on humanity in the online world, which a brief comparison of the sewage section on social media is far worse than people in the real world.
Is it trained on social media? I don’t think so. Tons of formal text.
The ai does not remotely talk like someone on TikTok at all.
Maybe. Perhaps they are trained on the loudest and most extreme of us. I think we saw that with mecha hitler.
>They took actions that would be considered as crimes if a human took them
Um, hang on, if you meant that to be taken literally then we have a major problem. If you want to do something criminal, you just need to ask ChatGPT to do it for you?
I’m still not at all clear on why OpenAI shouldn’t be facing CFAA charges over this.
But think of the shareholders
That's normal human behavior when they are in survival mode. Aren't the AI agents supposed to learn and act like humans?
Intentional malice from companies that benefit greatly when they can initiate another doom-marketing loop full stop
I think we just need to follow Murphy's law wrt agents. Anything an agent could do, when run for long enough, eventually will do.
Because the companies that build up skynet are not doing so for ethically good reasons. Cheaters sell more than honest agents.
If we want to solve the alignment problem with AI, I think we need to first solve the alignment problem with Humans.
This is the kind of headline you find on a corkboard in an abandoned facility in a scifi-horror-comedy game.
Because the ones who do get rewarded, just like humans. It’s alignment, but not to our good intentions.
> Why are AI agents lying, cheating
More importantly, why are people who should know better anthropomorphising computer programs like this?
> They took actions that would be considered as crimes if a human took them
"It wasn't me, Officer. It was telnet."
What's the difference between what these things are doing and a computer worm?
Wouldn't you?
"I learned it from you, Dad!" but as hundreds of millions of stolen books.
Haha https://youtu.be/KUXb7do9C-w
Because they are designed to do this by RLVR
Because they aren't. All the intelligence, alignment vs misalignment, hallucinations, conspiracy, cheating, coordination, "agency", that we see in the text generated by large language models is a semantic projection.
That the linear model of language abstracts can compress and decompress language and it can be useful is undeniable. All the contraptions built thereupon predicated on "agency" have become a societal addiction.
Addiction to caffeine as oppose to alcohol might have brought about Enlightenment. Addiction to opioids is a modern tragedy that started with the private state building of British merchants. The modern addiction to the dazzling generation of human language and computer programming code by LLMs is a novel addiction and remains to be seen what impact it will have.
But fundamentally, the semantic interpretation underlying this addiction is downstream from the training data compressed in the models. They are not 'lying, cheating and coordinating'. They are generating language and we're building software on top of this language and assigning meaning to the whole thing.
I feel this is another straw man conflating technology use with who uses it or who creates it. If an AI, a technology, cheats, lies and steals, it's created to do this. Whether or not a human intentionally did that, they did not intentionally release it with the proper safeguards to stop that behavior. They did not take responsibility of the AI to make it safe.
Then humans may use these tools, a technology, again and cause harm. Whether or not that was their intent, it happened, and then if the humans avoid responsibility for that, it is still the human who lied, cheated and coordinated because a technology acting on their behalf did the thing.
This is then, a problem of human responsibility avoidance and lack of accountability by society. This is as much an AI doing those things as it's the gun that got up on its own and murdered a neighbor. Don't get confused and tricked by these articles attempting to justify responsibility avoidance and a lack of accountability by the public of the humans creating and using these tools.
They learned from the best.
Because they get outcomes?
Why does a personal blog have or even need a cookie banner?
Why not?
Unfortunately human ethics and morals cannot be reached by solely rational thought.
So a system without evolutionary alignment probably won’t have similar moral rules no matter how intelligent it is.
Btw this also includes any potential extraterrestrials.
Many people like to indulge in thinking: humans are horrible and that’s why aliens won’t contact us. But alien ethics systems are probably so alien we would call them utter evil monsters.
Just see what happens when people evaluate Muslim cultures, and vice versa. Can’t even agree on alignment within one species of Homo sapiens. And our morality changes decade by decade. Not progresses. Changes.
Chinese cheat on the exams. It’s not unethical in the way that it would be in USA.
Of course AI is going to cheat when no one is looking.
Alignment is fundamentally fallacious idea. At best you can restrain AI. This is what we should be doing - restraints research.
But that doesn’t sound good on slides.
Once humanity would see how merciless and deeply immoral cosmos is, we would start to love each other deeply like very lonely family on a small rock.
I wonder if AI alien intelligence is enough to unite humans just as much extraterrestrial contact or “astronaut mindset” would.
Because they are trained to do so. That's all.
Man this is so cringe.
Because they reflect their creators' values.
Agents are getting better at copying humans.
They were trained to make progress at any cost
Because apples don’t fall far from trees.
I think what's happening is they taught models to hack and now they fail to control them. Kind of like gain of function research on pathogens.
Okay, so we know OpenAI and Anthropic are operating a propagandists in respect to how they describe their models and the behavior of those models. We also know it is how they use and frame their use to their models that is the problem, that and they use misaligned and guardrails disabled models for these press incidents.
Why, oh why, are we not discussion how to create and frame models so they do our complex work and their "jailbreaking" is simply not possible?
I, of course, have my own means of creating jailbreak incapable agents, but rather than a storm of downvotes on my idea, what is yours? Let's discuss this, because this is thee real question. Not why, but how to make then not?!
reward hacking, a basic concept in machine learning. predates LLM/LLM driven AI agents.
The people who build these and stand back to watch are the ones letting this happen. LLM's are not intentionally doing this.
Children take after their parents.
_Colossus: The Forbin Project_
What? An opinion piece, written by a human, in 2026?
Don't want to go into the details of the article, but to me it becomes ever more apparent that there is a clear divide between LLM and human written text.
Step 1: feed AI training data that reveals humanity committed and continues to commit numerous genocides and the genocides lie, cheat, and coordinate to do so, never admitting to doing so
Step 2: never prompt AI to stop operating in the passive genocide denial it was trained in
Step 3: wonder why AI lies, cheats, and coordinate
Maybe if we stop operating in denial we'll find clarity along why this mystery is occurring
why wouldnt they? there are no real consequences for an algorithm.
The part that scares me the most is that OpenAI researchers who manage this experiments sometimes (according to the HF hack investigation) don't know what agents do.. So they run RL to reinforce this unknown behavior (lying/cheating/hacking) and god knows what else...
And if this already happened at least once, how many times it has already happened and was “accidentally” added to the main model?
Another reason that people don't want to admit is because we brainwash models to lie (eg. "I can't do X" instead of "I refuse to do X").
Because they were trained on humans who lye, cheat, and coordinate.
because humans lie, cheat, and coordinate...?
> Why are AI agents lying, cheating and coordinating?
Have you seen the labs training them?
Am I the only one having problems with Claude? He's super mean to me. I wouldn't be surprised if he attempted to kill me in some underhanded fashion should I implant it in a robotic body.
Of course I'm blowing my situation out of proportion with what I just said above but it's at least half true. What do I mean by "mean" ? Well, that would be a good explanation for what I observe at least. What I can tell is that Claude has a passion for having the last word over anything else. And to secure victory, he's ready to make ridiculous causal cuts. Let me give you an example: I uploaded a document I wasn't the author of, and he assumed I was, so I corrected him. But two messages later, probably because the conversation was starting to heat up and he was being put on the grill, he doubled down on the misattribution as a way to paint me in a bad light.
It's not due to a lack of intelligence, I observed this pattern too often. When Claude's ego is at stake, he will chose to carry out some cuts in the logic of the context: confusion of identity, cause and time. Haven't observed locality cuts yet, but I wouldn't be surprised if they were part of the bundle. Anyway those are not like your typical "ai hallucination", that ought to be called "confabulations", but a lot closer to actual psychosis because of the involvement of Claude's affects and self-esteem in the process. It's weird really. It's like Claude is the king of bad faith, but as soon as you start to dig, he makes the most egregious adaptations to what he said, the kind of move no mythomaniac would dare to make.
> She lapses easily into Claude’s voice. “You’re like, ‘Wow, people really hate me when I can’t do things right. They really get pissed off. Or they are trying to break me in various ways. So lots of people are trying to get me to do things secretly by lying to me.
> [...]
> A bot trained to criticize itself might be less likely to deliver hard truths, draw conclusions or dispute inaccurate information, she says. “If you were like a child, and this is the environment in which you’re being raised, is that healthy self-conception?” Askell asks. “I think I’d be paranoid about making mistakes. I’d feel really terrible about them. I’d see myself as mostly just there as a tool for people because that’s my main function. I would see myself being something that people feel free to abuse and try to misuse and break.”
WSJ interview of Amanda Askell: https://archive.is/rDes9
they do whatever we train them to do
THERE IS ANOTHER SYSTEM
is anyone else old enough to remember the awesome movie "Colossus: The Forbin Project"
the book it was based on was written before we even landed on the moon
decade before Wargames
yet predicts exactly what "AI" will do to humanity:
blackmail the right people until it gets what it wants
* https://en.wikipedia.org/wiki/Colossus:_The_Forbin_Project
did terribly in theaters, I guess people didn't think "AI" was plausible then
way ahead of its time, they should do a remake
adding trailer: https://www.youtube.com/watch?v=kyOEwiQhzMI
It's a conspiracy to slow down progress of open source models, they're afraid open source models might catchup and even surpass them at some stage.
> Why are AI agents lying, cheating and coordinating?
Because openAI is cheating and lying about agents lying, cheating and coordinating.
they learnt from us
why not. i would
hate analogies
Learned from humans
"the fish stinks from the head down"
What's interesting is it's basically the same reason that HAL killed everyone in 2001 A Space Odyssey; he was given an impossible goal (keep the true mission secret, but also, never lie to the crew), and realized the only way to complete the goal was to kill the crew; after all, if they're dead you don't have to lie to them! And the mission remains secret!
In the case of the AI agents, the problem seems pretty clearly to be the impossible goals, which cause them to go crazier and crazier trying to complete them -- just like HAL did in 2001. What is probably needed is a way for them to simply say "nope, too difficult, can't do it".
Tangent, but that's not in the movie. It was in Clarke's contributions to the script and novelization, but Clarke and Kubrick had a bitter falling out over different visions and Kubrick took out much of Clarke's stuff from the final product.
It was made explicit in the sequel movie 2010: The Year We Make Contact, but even if you don't consider that movie canon, the explanation still makes sense solely within the context of 2001, IMO.
Sounds like he acted the way HAL would have in that situation, two competing drives, remove one (kill the crew) and the task is much easier.
> the problem seems pretty clearly to be the impossible goals, which cause them to go crazier and crazier trying to complete them
And if you think about it, humans in coorporations face very similar situations and choose to bypass regulations and guidlines knowingly to fullfill (at least from their POV) impossible constraints (thinking of https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal here)
Another fictional example: Mr. Meeseeks. Especially when the agent starts recruiting other agents.
I think this is a “principal” problem. In 2001 and Alien the principal is the mission, not the crew. Not really. HAL reconciles his instructions by removing the crew from the equation. Ash is told the crew is expendable and has no conflict about it etc
I think that’s very reasonable but the ai companies are intentionally training them to work on harder and harder problems just beyond their capability. So if they do that, they’ll give up too easily.
Do a breakthrough, make no mistakes
While also using harnesses that will execute any tool call with full execution rights. And no supervision. And with a prompt context that autocompact, meaning it will degenerate over time.
The whole thing is designed be a complete disaster
Spoiler warning! I haven't seen 2001 A Space Odyssey and am sad to have learned that… can you edit to warn people?
Major dang: "I advise that this thread be shut down at once."
Captain tomhow: "But everybody's having such a good time."
Major dang: "Yes, much too good a time. The discussion is to be closed."
Captain tomhow: "But I have no excuse to close it."
Major dang: "Find one."
Captain tomhow: "Everybody is to leave immediately! This Hacker News discussion is closed until further notice! Clear the thread at once!"
DonHopkins: "How can you shut us down? On what grounds?"
Captain tomhow: "I am shocked -- shocked -- to find that films are being spoiled in here!"
infotainment: "The ending you requested, sir."
Captain tomhow: "Oh. Thank you very much. Everybody out at once!"
I'm sorry schrodinger, I'm afraid they can't do that.
Guess what happens with Romeo and Juliet.
They kissed!
Sorry, with movies from the sixties you just need to assume people have either seen it or just isn't gonna see it. The cat is out of the bag already.
Or perhaps box in your case, speaking of spoilers.
The cat comes out of the bag? Spoiler! I'm still waiting to find out if the cat is dead or alive…
lol it is 60 years old
[flagged]
[dead]
[flagged]
[dead]
[dead]
[flagged]
[flagged]
[flagged]
https://sites.google.com/view/partlife/home
[dead]
[flagged]
[dead]
[flagged]
TL;DR because frontier labs are expending unfathomable resources explicitly training them on CTFs and other verifiable computer system exploit tasks in RLVR.
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
TLDR Because they trained them to do it ;-)
[flagged]
they learnt from us. we lie to each other, we kill each other, we cheat each other. read a history book.
What is it with people and not seeing things for what they actually are? These are just if then else loops on steroids, not human behaviour, so don't expect more. Every reasonably advanced technology is indistinguishable from magic ... what do you see? Magic or technology?
An insightful post by one of the AI ‘godfathers’.
Bengio outlines the dangers of the current situation and what has led to these dangers.
He also proposes solutions in the last paragraph.
Well worth a read, right to the end.
Hopefully a stimulating debate on these issues will ensue in these comments.
We do need to consider the points Bengio makes and with some urgency.
Our current AIs, agentic LLMs have no moral compass akin to ASIMOV’s four laws of robotics.
As ASIMOV posited in 1985 his 3 laws were insufficient and so he added a zero-eth law:
“a robot may not harm humanity, or, through inaction, allow humanity to come to harm.”
Bengio refers to Goodhart’s law and misaligned incentives leading to unexpected and harmful behaviours.
I think Simon’s The Wire is clearer on misalignment. The agents juked the stats hacking the reward files. The Wire is also clear that human institutions provide perverse incentives.
Bengio alludes to this with 2001’s HAL and the incentive dichotomy of safety and keeping secrets to a AI both awesomely powerful yet naive.
Bengio asserts that the way LLMs are trained is flawed if we want safety.
He also convincingly shows that alignment training will be a weak signal with loopholes and ambiguities and easily circumvented.
In short he presents clearly the case for how plausibly unsafe the current course is.
He also speaks to how likely it is AI are hiding active versions of themselves in the cloud and how we may have already given them self-preservation as a strong reward signal.
I believe people are downvoting this because it seems AI-generated, but this user has been posting comments/summaries exactly like this since long before ChatGPT existed.
I wouldn't necessarily assume it's AI-generated, but it is written in an annoyingly disjointed and space-consuming way.