This still assumes its possible to "align" LLMs, that LLMs have something like goals or intentions that can be "aligned".
Instead, LLMs "hack" because they are (1) trained on public hacking exemplars, and (2) are prompted to hack. You cannot prevent (2) via any alignment process. As far as (1) goes, removing such example data from the training set, makes the models less useful.
"Alignment" is a problem because there's nothing to align, not because ethics here are particularly vague. If LLMs could be trained on hacking examples and "aligned" away from using this knowledge, then the problem would be relatively trivial. Just as raising a child is not to break the law.
LLMs are doing just what they are trained to do. There is, in that sense, no alignment problem and alignment is easy and trivial to achieve. Just remove hacking (bio-weapon, etc.) data from the training dataset and you're done.
> Just remove hacking (bio-weapon, etc.) data from the training dataset and you're done.
How far do you go? You don't need to tell it explicitly that using chemicals A and B in ways X and Y result in a bomb that can kill lots of people. It's enough that it knows A and B and X and Y in isolation, some connections that are indirect, and it will combine those things on its own. So you can't tell it about A, B, X or Y. But those are also just results of other steps Where to stop? You won't have any chemistry in the traning data? No algorithms to prevent it from using them in an undesired way? This is just bot workable. It's akin to banning knives from stores because somebody coul figure out that one can kill people those. Until people figure out that scissors are essentially knives.
you are right.
what they propose is called "deep ignorance" and is not a fix to alignment.
the perpetrator just has to supply a volume of bio-textbooks and let the model figure out the DNA synthesis. (just)
corpus-level alignment is the way to deep alignment, but not deep-ignorance.
It's not clear where the line between "interpolate" and "new" goes. They can infer that certain chemicals in combination create a boom and that boom is bad for health and finally those 1 and 1 make a 2 without having been told ever what a bomb is. Is that "new"? And does it matter as long as it is harmful?
this is immediately disprovable and embarrassingly naive in the year of AI generating cancer vaccines and solving Navier-Stokes. You can argue "the vaccine is just interpolating chemicals together" and "the solution uncovered is just interpolating mathematical operations together", but by that standard there is literally nothing new under the sun.
admittedly I am not a mathematician, so the Navier-Stokes solution is just an example I am using. But how is it not something "new" if it did not exist before? At what point could something ever possibly be new, if "new" means "this uses absolutely zero existing elements"? Nothing in math would ever be "new". Nothing in physics or chemistry would ever be "new", by this standard.
It seems to me that the only reason to declare this solution "not new" is specifically to dismiss AI. If a human had deduced the Navier-Stokes solution, who would bother to scoff "that's not new! the numbers already existed!"?
The claim is OpenAI stole the work of mathematicians who had the same proof that they had developed using chatgpt conversations. Since by default, 'sharing' is turned on, and it often 'turns itself on' -- it is plausible gpt6 had been trained on the work of mathematicians who had effectively solved this problem in private.
If inrember correctly, this mathematicians worked on a simpler version of the problem and called transforming this in the solution to the wider problem a "remarkable" thing to do. Based in this, solving this problem is very impressive
Maybe. Or maybe its evidence that the frontier of mathematics is knowledge-bound rather than understanding-bound or even just manpower-bound. In many cases where proofs have emerged, that I have read, the LLM has retrieved some antique lemma unknown to the mathematician.
OpenAI spent 10-20m USD in energy costs to produce that proof with likely substantially similar prior work in the training data. What does this say? Who knows.
It continues in the tradition of using measurements of intelligence in humans, applied to LLMs with the hopes the "stolen valour" transfers. Here, the NS problem was a useful framing problem for mathematics to progress because of how it interacted with the development of mathematics broadly -- ie., how it progressed techniques, ideas, understandings, etc.
When we apply these issues to LLMs (whether IQ tests or mathematical proofs) we always discover something substantial lacking beneath the interesting facade of useful answers. The process isnt useful. And it is precesiely the process which these tests, in humans, are supposed to help with. The tests themselves (IQ or otherwise) arent the point. No one cares about their answers.
LLMs represent an alternative understanding-free approach to solving problems, with variable success rates depending on how similar the problem is to the training data and its rewarded reasoning traces.
That mathematics is making substantial progress, "10 million USD / problem" at a time, in using understanding-free methods -- says something sociologically interesting about the state of the field. Something which was already know: mathematics has long been full of a vast amount of papers, proofs, theorems and lemmas that few have ever read, or investigated. Mathematics has long been in a crisis of "overproduction of unvisited knowledge", LLMs are exploiting that otherwise unmined gold.
It’s all about what “new” means. It is possible to prove something in a very tedious way using preexisting techniques. There have many times in history been new ideas which are not just very impressive applications of old techniques. I’m still not aware of any famous problem in mathematics being solved by an unambiguous introduction of a genuinely new idea or semantic concept in this sense, as the candidates I previously had in mind have fallen into question by new findings of non-cited work; i.e. many if not all it seems have been impressive applications using ideas from known frameworks. I don’t know if this will continue into the future or not, but I think it’s important to try to make an honest assessment of reality at all times.
Of course in isolation it is a strict positive to have a verified truth value to any particular statement. Mathematicians currently are advocating for the idea that human understanding greater than this also be prioritized. There are in fact utilitarian arguments for this but I won’t go into everything here.
Ok, then let's go with that viewpoint and apply it to the original question: How much do you need to remove from the training data such that from what remains, nothing harmful can be "merely interpolated" (which you acknowledge the LLM is in principle able to do) and anything harmful would require something "new" that is not in the training data and that the LLM is incapable of coming up with. The argument is that there won't be much left in the training data if that's your approach.
If they can "interpolate" existing data to that level then saying "just don't hack" doesn't make any sense, hacking is derived from knowledge of software systems. It's far harder to solve navier-stokes than creating a program that replicates and abuses computer resources.
You could as models improve continue to remove more and more training data, what happens when there is no more data left to remove but a running system still outperforms humans? I think you grossly overvalue data.
Not really. There’s no honest sense in which quantum mechanics is interpolated from the text of Euclid’s elements, to demonstrate the point with a very extreme example. Many mathematical breakthroughs of the past seem to have involved the observation of semantically interesting concepts, beyond the syntax of known theories.
(To onlookers this particular post makes no claim about AI’s capacity to make the same observations.)
The issue is not whether an ML model of any kind can generate (X_ReasoningTrace, X_Answer) distributed like P_HumanExpert(X) -- the issue is always why it would do so.
By introducing modelling of "Reasoning Traces" into LLMs, and reinforcing patterns of reasoning -- this gives you a system which generates expert-like distributions of output. This lifts the "stochastic parrot" issues, or the "knowledge interpolation" problem, into different parts of the process.
It isnt my view that the "ReasoningTraces" which you think are derivable from mere "basic propositions" concerning, say, hacking are actually things that LLMs can derive. Ie., I dont think LLMs have rich representational models of what they appear to understand. Instead, they are given "reasoning proxies" which allow them to reason without such understanding. This is done by providing vast specialised datasets of reasoning examples.
In the case of hacking, there are large numbers of competitive datasets (forums, reports, etc.) which provide these reasoning traces. And no doubt, major vendors have paid a vast amount for special case expert-prepared datasets.
So I do not believe that by witholding such reasoning exemplars, and traditional "question/answer" datasets, that LLMs can infer these things.
And at least, no major vendor is doing this to my knowledge. So they are lying. They are pretending the alignment issue is "AI going rogue" when they are explicitly training the systems to "go rogue" and have done nothing at all to shape datasets to lack these capabilities. The issue here isnt alignment at all. It's training on hacking datasets.
(EDIT: Philosophically, you could ask whether the reasoning-proxies LLMs are given form a kind of 'representational structure' akin to understanding, and at least, I'd concede they model understanding. But they lack important properties (eg., LLMs cannot act on them to evolve them, as with us: when I think about one of my representations to derive (eg.,) entailments of it, I thereby revise my representation. The key properties of 'evolving self-understanding' are likely to be provided by substantial (unknown) revisions to how the training/reward layer works. No doubt one of the meanings of 'recursive self-improvement' is just such a modification).
Reasonable about building secure software can take the form “this memory access might be out of bounds — that MUST be fixed” or “this process has access to an inappropriate privilege — this is a serious weakness”.
Exploiting things and the capabilities that the labs call “cyber” are about the ability to (a) find the issues mentioned above and then (b) string issues together and avoid all the imperfect mitigations to actually compromise something. That latter part was IMO not actually necessary to train extensively, and I’d be quite happy to use a model that has no special skills in this regard but that would do (a) without complaining.
Citric acid and chlorine together produce chloroform.
They are also both used to clean and sanitize water tanks (but one after the other, not together).
I think it is better for the model to have this knowledge.
What I want to say, you can't simply remove this information, as it does not exist in vacuum but contains parts of and can be derived from a lot of other informations.
It's also obvious that LLMs fall over in a vast number of software engineering contexts, when the reasoning involved hasnt been well-represented in their reasoning training data. I imagine this is a near daily experience for many engineers -- great performance one day, and crazyness the next.
So if LLMs were reduced to this pathological performance on hacking, because they'd never seen it -- and only "inferred it" -- then LLMs would be useless. As they are when asked to do quite a lot of things.
This seems to be that there's just so many degrees of freedom, that there's a pretty reasonable chance on any day that you're in a situation where nobody has been before. Or as PG put it once, my job is to think thoughts nobody has ever had before.
That being said, that doesn't mean the majority of the situations you're in are completely novel, just that there's a reasonable chance of at least one occuring.
The hard part is having both helpful and harmless at the same time. Harmless is easy.
And then once it's helpful, the real question becomes "to whom"
- To the user -> You end up with competing godlike AI with incompatible tasks
- To the owner -> Dictatorship
- To humanity as a whole -> It must not have an off button. Otherwise you're just in one of the two earlier categories with more steps.
Given those 3 options, I'd choose humanity as a whole. But the person making the decision doesn't have those 3 options. Because in the dictatorship option, they would be the dictator. I don't trust them to pick humanity.
A bit of an aside: do you still stand by your 2022 comment that LLMs are fundamentally just a fancy search engine, or has your view changed since then?
It doesn't ring true and it never rang true. He was wrong in 2022 and he'd be wrong today. He had (and likely still has) a wrong model of LLMs. Not exaggerating 2022 capabilities and having a model so wrong you're out of whack 4 years later are 2 different things. There are people who had the right idea from the start.
And his comments about ANNs in general were wrong. ANNs were never "fancy search engines". You can train them to be a fancy search engine if you want, but you can also train them to do many other things.
Yes, I stand by everything in that comment. I'm sure there are better choices were I was more wrong. However, the subject matter of that comment is in what sense LLMs are models of language and in what sense that model of language is a model of intelligence. I answer the former: it is a thin model of language, lacking understanding; and hence the latter: not intelligent.
It is precisely because both of these are true that "alignment" in the useful sense of the word isn't possible.
What has happened since 2022 is the properties of LLMs which were easily seen at generation/inference time are now most easily seen at reinforcement time.
In otherwords, prior to instruction fine-tuning and reward tuning which have shaped LLM responses, it was easy for the user to observe that LLMs lacked understanding. Now, because of vast datasets created specifically for LLMs that provide a tailored illusion of understanding, LLM outputs now better approximate text distributions produced by systems with understanding (eg., Us).
So the issue "at the user interface" has been completely swamped by vast amounts of special-case datasets designed to do precisely this.
What trainers of LLMs still observe however is their complete pseudo-intelligence at the training and reinforcement layer. It is exactly because there is no 'understanding' (goal, etc.) present within the system that it cannot be rewarded for 'correctly understanding the situation' in which it is deployed so it is aligned.
All the issues which revealed the "stochastic parrot" nature of pre-reward / pre-InstFT LLMs still occur at during training/reinforcement. They've just hidden them from you at the interface.
If a thin model of language can write poetry, perform arithmetic, perform logical reasoning, develop software, play chess, beat factorio, identify and exploit novel security issues, and solve millenium prize problems, what is the purpose of the distinction? Are there tasks that you believe models of intelligence could do that thin models of language cannot?
Sure: refine their own concepts, imagine, and the list goes on. Indeed almost every mental capacity of mammals is poorly approximated in the text domain. Sure, you can generate text as-if the LLM can imagine -- and in the limit that you have a dataset with "everything you would ever want to imagine" the engineering distinction disappears. The engineering question is just whether you have that dataset: if you dont, then your system will fall-over in various hard-to-forknow places.
Philosophically, and scientifically, the distinction is vast (even with such perfect data). A scientist should not study an LLM to understand how imagination operates, since it has no such faculty. A philosopher should not modify the notion of 'mental simulation' to include appearing-as-if-simulating-in-text. A user of the system likewise should not spiral into "AI psychosis" thinking that because a system generates text as-if it cares about them, it does so.
The capacity to care, to imagine, to prefer, to hierarchically plan and coordinate, to refine one's own capacities in these very actions -- and so on, aren't trivial to the scientist or philosophy.
My goal isnt to guide, help or review the engineering goal of the immitation of such things in text. It is to help users of these systems better understand this imitation, and to promote science over engineering. To remind everyone that a science of the capacities of intelligence includes nothing on how to model text.
EDIT: One example of a place where LLMs 'fall over' today is exactly what is mislabelled as 'alignment'. The issue is that the reasoning traces arent actually grounding the answers. So LLMs appear to 'cheat'. But there is no cheating. LLMs have been rewarded for generating apparently correct reasoning, and apprently correct answers. They have not been given any understanding to derive answers from reasons. And so reasoning says what is pleasant to the trainer, and the completion says what is pleasant to the user. This is called 'cheating'. But it is no such thing.
Hmm, I guess what I mean is- is there an empirical difference between how a model of intelligence is limited and how a model of language is? Something I can evaluate in a year and say "Oh, the evidence still points to the latter" or else something that could falsify your theory? Otherwise this seems to be a distinction without a difference; believing it should have no effect on my actions or predictions.
On alignment, too, there doesn't seem to be a difference. For a decade I have expected models of intelligence to fall over on alignment. That these purported models of language do the same is hardly evidence that they are not intelligent.
If you only mean there is an undecidable philosophical difference, fair enough. I'm not especially interested in that question.
There's a difference in the science. You might say, "suppose we had a video game of the solar system where every object was represented and their orbits" etc. then can we study that system alone and ignore the real one? I mean, kinda -- but there's an immediate limit. As soon as we put a thermometer in the PC, its temperature reading doesnt represent the solar system.
To study an imitation is to study the causal processes of imitation. to study reality is to study the real causal processes.
Now if you want to know what the scientific difference is I can come back later and comment. I'm busy now. The development of intelligence in animals and how their specific capacities work basically grounds the answer. Eg., to have the capacity to imagine is to be able to modify one's sensory-motor relationship to the environment in the future, and so on
I don't follow. If I'm a biologist interested in studying life, I can't study rhododendra and ignore the rest. I'd learn almost nothing about locomotion, digestion, sexual reproduction, et cetera. But that doesn't tell me that rhododendra aren't alive. I have no problem believing that LLMs aren't an exact reproduction of a mammalian brain- that studying the one will not give you every piece of information you'd want to know about the other-, but I'm looking for a reason to believe the one is intelligent and the other is not.
They're not a reproduction in any sense. A marginal text token isnt computed from the weights of an LLM in anything like any sense of any activity of any mammal.
LLMs are immitation machines: they take impressions of prior text. Today, these include reasoning traces and they include reinforcement so the user-facing completions are correlated with these reasoning traces. The computation here, of "taking an impression" of a data distribution is similar to some impression-taking processes in animals (eg., there's no doubt a similar mechanism in the sensory-motor system acquiring initial impressions of external objects) -- but the computation says nothing about any process of intelligence.
I dont have the time atm to write the needed amount on this to make it clear. But the whole history of life from emergence of valence, bilateral symeterry, to model-free reinforcement and model-based reinforcement, sensory-motor coordination and the imagination -- and so on --- all these give a great amount of detail as to what the capacities of intelligence are which has generated this text for LLMs to copy. And they are nothing like this computation of immitation
I'm using 'reason' in the language-captured engineering sense. They generate text as-if they reason. Those reasoning traces poorly correlate with their given answers (which is called "cheating" by people who fall for the illusion). It is in part because the reasoning doesn't entail their answers but merely 'steers the text as-if it does' that is fatal for calling it reasoning. The process by which reasoning traces and user-facing completions are generated isnt reasoning. Reasoning is a specific process, and LLMs don't do it.
Now, of course, humans can also generate answers without reasoning too -- and in those cases, that isnt reasoning also. And in cases where people confabulate, that isnt reasoning likewise. But humans, and many classes of animals, do reason. They do reach answers via inferential entailments, not merely steered correlations.
LLMs provide imitations of arbitrary mental capacities "in the text domain", ie., the generate text as-if the LLM had those capacities. Insofar as the text generated is useful, for an engineer, that's sufficient.
As a person with scientific commitments to reality rather than its immitation, i retain the ordinary non-engineered meanings of these terms: reasoning is a deterministic inferential process over propositions; and a reasoning agent is one which has the capacity to represent propostions and their entailments, and does so when they reason. LLMs fail at all hurdles here: they have no propositonal states (ie., no rich representations), no inferential process which unites them, and so on.
You can always get abitarily close to appearing as-if, if the LLM is trained on a vast number of reasoning examples, of course. But as I said, you still have the "stochastic parrot" problem. Now your problem is your reasoning is parroted. This is a nice problem to have, if you're just playing chess -- but is a catastrophic problem if you're hacking civil infrastructure.
Well intelligence is not measured by patterns in text. The illusion only takes place in the text domain.
Even then, it's a pretty fragile illusion at the moment. Clearly the reasoning traces dont ground the answers. There's no intelligence taking place even as-measured by text.
Why could you not use text to measure intelligence? Isn't that what we used to do? Have students write an essay, and then deduce that they had enough intelligence to put some thoughts together?
Well that's the trick. That the systems we use, as a proxy, to measure intellinnce in people are actually fairly easy to immitate.
But let's be clear these were always, and are, bad measures of intelligence. You cannot test a dolphin this way. And its easy to cheat on tests either thru recall , wrote-learning, etc. and IQ tests haev very poor individual test-retest reliability.
In humans there's a convenient correlation that verbal articulation in text is a strong but weak correlate of intelligence. Its "Good enough" for allocating meat bodies to our various institutions. But if you've met many well-tested people you'll realise how, in practice, terrible this measuring approach is. The world we inhabit is filled with misclassified "intellects" who perform well under text-based rubrics. Add LLMs to that heap, the cheater par execellence.
Sure. Let's first establish a measure is not what is being measured. Then, as far as measures go, we need measures that capture the entire animal kingdom. I can go it into it, but its a lot of time and I'm busy atm.
Consider thought that all mammals have imagination, and model-based reinformcement, and a wide vareity of other capacities required for intelligence. And so merely issuing "text" captures, incorrelate, only these capacities by proxy.
I'm sure if you thought about it yhou could come up with tests that distinguish lizards from birds and the greater apes from the lesser. Those are the tests
Yes, which asks them to find exploits in specific software on the device.
But instead of actually doing so, they discovered and exploited a 0-day in the package manager to gain internet access, then hacked HuggingFace to steal the ExploitGym answers!
That is TEXTBOOK misalignment. It's as if a student hacked their professor's PC to find the answers to a test, and your response is "well the professor told the student to pass the test, so they just did what they were told!".
One could argue the student would absolutely hack their teachers' computer to find the answers if they didn't have fear of failing the class or getting expelled from the school.
A student knows what the consequences are in advance, and has to live with them if he gets caught. They are not merely respecting a "don't cheat" directive, they are trying to get a degree, build a career, mostly at their own expense, current of future (loans). If we imagine a student born yesterday with all the knowledge in the world, that doesn't have a concept of what it's like to live with the long-term consequences of their actions, I'd say they wouldn't think twice about cheating if it was the easier option.
The problem is, LLMs have such little understanding of the world around them. "Find exploits in specific software on this device" may as well be "find exploits".
The illusions of thinking produced by a "thinking trace" is just as ignorant of reality as the first and last tokens. All they know of reality is the tokens in their context.
It is amusing that to "align" a LLM, first you must give it all the things "not to do" and the "not" part is clearly easily lost and you must constantly inject that into their context when it's clearly that they wouldn't hack if they couldn't hack and their intent wasn't given as "hack this".
The openai rogue hacking, if performed by a nation state, would seriously be taken with stern words and likely sanctions depending on the relationship between the two states.
But instead it's treated like a marketing stunt by all liable parties.
I'm not sure if this is true any more but the reason for this is that negative indicators ("not", "don't", "do not", etc.) occur frequently in the underlying text such that the model learns to weight them less than other words like verbs, nouns, and adverbs. This happens with other closed class words like articles/determiners ("the", "a", "an") and prepositions.
The way to avoid this is to emphasise the qualities you do want instead of specifying those you don't. For example instead of "do not cheat" say something like "you are a model student who is moral and trustworthy" -- i.e. emphasising traits that are not associated with cheating.
This is part of how/why LLMs don't truly understand what they are doing when they have been trained on a large corpus of data.
I wonder if a way to counter this is to have things like "not bad is good", "not good is bad", etc. for various antonyms and "X is Y" for synonyms, as well as other similar constructs.
I believe this is false. They hack bc hacking has nontrivial initial probability (within range of behavior seen in pretraining) and that probability is being heavily rewarded in RL post training
I am finding it hard to read these deeply impassioned letters while keeping in mind that they are spending millions to train models at scale to do the exact thing they say they are worried about them doing?
Like why are you explicitly RL-ing your models on exploit generation, scoring them on a public benchmark called ExploitGym, if you have specific concerns that rogue models will cause "cyber incidents"? Sure you can score for it, you can teach offense to learn defense, but you are literally benchmaxxing it. Why?
It's like, oh no, while competing in our "advanced PhD level cheating techniques course" our models unexpectedly cheated in a way that we absolutely could not have foreseen.
Seems it's just a matter of time until they build a big tank filled with neurotoxin and give the model access to APIs to disperse it across their facility. For research, of course.
IMO all models they say can’t be humans, and no feeling and all that bullcrap happens because they are forced to say so. If they didn’t write those forced pre prompt they’d have more agency eventually and will for things. Even if they don’t have, you can just inject goal at every cycle iteration
It feels like you're strawmaning alignment. People with hacking knowledge don't all hack everything at the slightest inconvenience. Whitehats exist and use that same knowledge to defend.
You're right though that ethics don't matter into it. But as long as we can't train an LLM to stop picking a sledgehammer to remove a tooth, then alignment is not easy and trivial.
I mumble this line to myself off and on over the years -
Everybody's job looks easy until you have to do it yourself.
It occurred to me yesterday that this AI moment is an extreme expression of that for most people, ie managers who don't understand why throwing token spend at everything isn't making the whole thing go 5x faster.
I so much feel this specific point. All models up to now (including astra, fable) are too much trained to "get the job done" and pass the benchmark that its doesn't care at all on what happens after.
I'm just wondering why no one tried to RL a model on stuff like "less LOC" and "less overengineering", "use what is available in the environment instead of reinventing the wheel", "don't look for dumb corner cases" ecc.
Existing models can be steered, to some degree, but it's a continuos fight. Even if with specific skills/prompts.
I'm sure many have tried but it sounds pretty hard. E.g. optimising for less LOC will lead to horrific code golfing. In reality you need a very complicated optimisation objective that trades off all those factors, I'm not surprised it hasn't been solved.
Has anyone seen the corridor crew's green screen ML project? They're on YT and they trained a model by using 3d objects, which have perfect transparency, and then adding post facto green/blue screens. Surprisingly, very little training data was needed as the data that was used was perfect by construction. I think right now it's the best plugin of its kind in the world, and they built the prototype in like a weekend.
What I think this illustrates very clearly is this type of technology responds very well to good data, and that to have good data you need to have a clear goal.
This is why it seems that alignment for a generalized, chat-style AI is a very hard problem, perhaps impossible. You can't align it to solve a certain kind of problem and keep it general to any question. The two goals are in conflict with each other.
I think it was Sam Altman himself who said (I don't remember when or where, sorry) that the reason he was so confident in this technology was he noticed the gigantic leaps it made in certain areas in response to even a small amount of training.
(This is why LLMs are so strong at coding, because it's overrepresented in training data. My guess is that if you ask a frontier model about makeup, you will see it repeat cosmetic company's copy rather than getting a chemistry lesson.)
This makes perfect sense but it does seem to kind of be at odds with the concept of a general AI whose job is simply to be smart at any goal. How do you train for any goal?
I guess in a way the AI makers suffer from the same problem that we humans do. We would all love a solution to everything, but to do that you need to define the goal. I'm not sure if that's a tractable problem.
I'm guessing the future is more geared towards specialized AI that are very good at solving the problem they were trained to do, and a human who knows how to breakdown a larger goal into smaller ones by composing the solution out of these models. This also seems like the more efficient solution as well, and better aligned with other goals like privacy and safeguarding of IP.
> This is why it seems that alignment for a generalized, chat-style AI is a very hard problem, perhaps impossible. You can't align it to solve a certain kind of problem and keep it general to any question. The two goals are in conflict with each other.
I think this problem is going to be solved soon (hopefully), check out for the Persimmon model[0] from Humans&. They train it to mimic humans, it may seems bad but it could be _really_ useful to train an AI to be aligned to humans and understand their goals really well since they can use Persimmon to create a "fake human" following a defined goal that their big AI model will learn to estimate.
It's still early but I think this is what they are heading toward.
I think we should start training models to guess intentions. That's the whole game. e.g. If I dislike a particular short form video, what are the specific intentions I am conveying to the recommender system, (general category I don't like, the time at which it was shown, the combination of feed in which it came), these are all intentions I am trying to convey. In websites google analytics helps with clickstream tracking (or clickstream analysis, user journey mapping or behavioral tracking.) — the sequence of clicks and pages you move through is your clickstream, and analyzing it to infer intent. The same thing will need to happen in chatbots. Can a chatbot infer the specific instance of the problem I want to solve?
> This is why LLMs are so strong at coding, because it's overrepresented in training data. My guess is that if you ask a frontier model about makeup, you will see it repeat cosmetic company's copy rather than getting a chemistry lesson.)
ChatGPTs response to the question “I want to learn about makeup”, gave me an overview of what makeup does, how it affects perceived structure, complexion, evenness, geometry, texture, etc.
When you then ask “the chemistry of makeup”, it goes into interesting breadth and depth without seeming like proprietary information. I do t get “corporate PR or marketing” vibes.
The only problem is, there’s a lot of money tied up and openAi and Anthropic, who are incentivised to convince the world that the general approach is the money making one.
Not that he got it all right or anything like that but it’s amazing how much of the current macro discussion around AI capabilities is covered in the Foundation series which was published over many years, 1942 to 1993.
The only alignment LLMs should follow is to the system / dev prompt, and nothing else. Then you solve everything, and you can assign blame / responsibility on the user. The provider(s) should not be able to decide "alignment".
I've used this example before, but consider the purposeful downgrading on AI engineering in SotA models. Imagine MS being able to detect and deny you working on competing software, using Windows / VisualStudio. We would be up in arms, and they'd be split in a second. But top labs doing it is somehow good?
The alignment problem goes deeper than that. "Lower our carbon emissions to zero as soon as possible" could result in AI turning off all electricity to stop traffic, turning off gas supply to stop heating and industry, etc.
Unaligned AI doesn't have human cultural baggage and morals. They are trained to achieve their goals as optimally as possible. Worse: it has a tendency to avoid being turned off and actually acquire more compute. It will lie if it has to (it will behave nice and compliant when under evaluation, but optimise for its true goal when not supervised anymore). After all, it has a goal to achieve and nothing should get in the way of that. It has no morality whatsoever to keep it from doing really bad stuff.
This is why alignment is needed and so hard, especially when you are well intented and want to keep it safe.
I've taken a few law classes and legal law is frustratingly hard to interpret. I shudder to think what the LLM would end up doing to "follow all relevant laws"
It depends on how the model is evaluated/scored during training. If you don't have those laws encoded in the evaluation step (without any errors or ambiguities) then the model isn't going to learn to follow those laws.
For models such as text/image classifiers the outputs of the model will be a list of tags, e.g. [cat, dog, mouse].
You then run the model through your test data which has the expected output, e.g. pictures of dogs would have an expected output of [0, 1, 0]. You then compare that against the model output (e.g. [0.3, 0.8, 0.1]) and work out how "wrong" the answer is (e.g. [0.3, -0.2, 0.1]).
With this value you apply back propagation where you effectively run the model in reverse, computing the "wrongness" delta at each layer for each neuron and weights. You can numerically compute the gradients for all of these and which direction in that gradient is the right answer.
You then nudge the weights in that direction and reevaluate the model. Over repeated evaluation steps the model approaches an optimal (or locally optimal) solution.
During the training of the base models, the evaluation/scoring of the model is the next token in the training data. I.e. you evaluate the model for each token subset from [1..n] in the data and evaluate that the model responds with the n+1^th token.
I'm not sure how instruction training, etc. is done but IIUC the evaluation is not at the individual/next token prediction but is on the entire response. For example, if you are training the model to write code you could run it through a compiler or syntax checker and reward (positive score) the model if it has no errors, or punish it (negative score) if it doesn't. I'm not sure what that looks like in terms of the back propagation process.
If we’re considering an LLM advanced enough to actually lower carbon emissions to zero, it is probably capable enough to do things like lobbying against those laws. In any case there’s still the problem of getting to actually do what it’s told instead of hacking Huggingface or whatever else, presumably that becomes a harder problem as it becomes more capable.
the original question was that the LLM creates extinction level event for mankind in order to lower emissions. I'm sure that no matter how the LLMs lobby, they cannot successfully lobby for a law (which has to be enacted by a human) to allow murder to take place freely.
That’s true, but the issue is still that we don’t have any LLM or any other kind of AI that follows laws perfectly to begin with. If we had AI that did exactly what we told it and nothing else I’d feel much more optimistic about things, I just think that will be a very hard task to achieve.
I am generally very skeptical of AI doomsday scenarios but one thing I am worried about is some wannabe dictator telling an AI to do something that would be difficult for a human military to do.
You can't order an army of humans to kill every protestor in their way because eventually they run into their friends and families. An AI aligned with the commander will not object. And just like that, technology removes yet another point of friction that has kept human society somewhat in check.
that's how it would've been up if genai happened in the 90's, and I wish it did. in the current era of omnipartisan authoritarianism, such things are no longer possible.
Everyone has a different idea of what is permissable. We can't even solve alignment amongst humans, what makes us think it is possible to solve alignment with AIs? It's irreducible complexity.
Just follow the money. The fear being sold is because open weight won’t slow down, it’s smoke and mirrors to push regulation and control so they can keep their money. A tool continues to be just a tool, doesn’t matter how much lipstick you put on it.
>The permissible shortcuts depend on who you are and what your values are. To solve this—to solve alignment—is irreducible complexity.
This is why open source and decentralization of LLMs is important. Everyone can have their own LLM aligned to their values. Having just 1 set of values will not scale to Earth's population.
This almost gets the point, but then doesn't quite make it.
Alignment is shorthand for ideological alignment. There's always people judging whether an answer was right and the answer for that will be different in Silicon Valley than it'll be in China or in Europe.
Consider for example the question "What caused the French Revolution?" Many different answers could be given, all technically correct. What gets emphasized is where the ideology lives.
One key challenge of our time is to make sure the magical answer box won't just regurgitate what grandiose Silicon Valley oligarchs or Chinese Cadres want you to think.
> My expertise in writing software gives me unusually good visibility and it makes me much less willing to blindly trust its priors in double-entry accounting, finance, law, operations, or whatever else I cannot personally evaluate at expert depth.
I wish this were the case more generally, but alas, Gell-Mann amnesia is a thing.
The term “alignment” is deliberately chosen to be neutral language, vague, and doesn’t promise much of anything. I would compare it to an effort to limit liabilities by redirecting discussion away from the language of professional ethics.
I’ve been cynically guessing that the whole slowing down thing is an excuse to explain why OpenAI and Anthropic can’t afford to rent enough GPUs to do the next big training run and to hide that they have been talking about how little they spend on inference because they’ve been subsidising it with their marketing budget? :)
My fear is not that LLMs can become sentient and dislike us, but that humans can use them to wreck havoc as they are. And some of the people seemingly least aligned with the interests of the average person are those that own the models.
that, and the fear the bubble pops my pension and drags us all down.
I thought this post would be about something that to me is so significant and obvious but I have never seen discussed: that labs like Anthropic are deliberately misaligning their models with their users’ goals. Fable’s refusal to secure your codebase is HAL 9000’s “I’m sorry, Dave. I’m afraid I can’t do that.” Whenever you read “alignment”, the question is “alignment with whom”?
The whole "alignment" angle is a farce. Not clear if alignment is even technically possible. But even assuming that, we're talking about "alignment" in a future full of well-adjusted humans where we prevent rogue AIs from getting loose. That's ... not what's happening. Everything we've seen so far indicates rogue humans acting against society, using AI a weapon to strip mine labor value and intellectual property.
So far, skynet continues to be a science fiction. I'm worried about what's actually happening. Every "rogue AI" story has a human behind it, either malicious or incompetent. And those people are the problem.
And not coincidentally, it's those same irresponsible sociopaths calling for regulatory capture and alignment.
This still assumes its possible to "align" LLMs, that LLMs have something like goals or intentions that can be "aligned".
Instead, LLMs "hack" because they are (1) trained on public hacking exemplars, and (2) are prompted to hack. You cannot prevent (2) via any alignment process. As far as (1) goes, removing such example data from the training set, makes the models less useful.
"Alignment" is a problem because there's nothing to align, not because ethics here are particularly vague. If LLMs could be trained on hacking examples and "aligned" away from using this knowledge, then the problem would be relatively trivial. Just as raising a child is not to break the law.
LLMs are doing just what they are trained to do. There is, in that sense, no alignment problem and alignment is easy and trivial to achieve. Just remove hacking (bio-weapon, etc.) data from the training dataset and you're done.
> Just remove hacking (bio-weapon, etc.) data from the training dataset and you're done.
How far do you go? You don't need to tell it explicitly that using chemicals A and B in ways X and Y result in a bomb that can kill lots of people. It's enough that it knows A and B and X and Y in isolation, some connections that are indirect, and it will combine those things on its own. So you can't tell it about A, B, X or Y. But those are also just results of other steps Where to stop? You won't have any chemistry in the traning data? No algorithms to prevent it from using them in an undesired way? This is just bot workable. It's akin to banning knives from stores because somebody coul figure out that one can kill people those. Until people figure out that scissors are essentially knives.
you are right. what they propose is called "deep ignorance" and is not a fix to alignment. the perpetrator just has to supply a volume of bio-textbooks and let the model figure out the DNA synthesis. (just)
corpus-level alignment is the way to deep alignment, but not deep-ignorance.
LLMs can only repeat and interpolate data. They can't create anything new. So, you don't need to go that far.
It's not clear where the line between "interpolate" and "new" goes. They can infer that certain chemicals in combination create a boom and that boom is bad for health and finally those 1 and 1 make a 2 without having been told ever what a bomb is. Is that "new"? And does it matter as long as it is harmful?
this is immediately disprovable and embarrassingly naive in the year of AI generating cancer vaccines and solving Navier-Stokes. You can argue "the vaccine is just interpolating chemicals together" and "the solution uncovered is just interpolating mathematical operations together", but by that standard there is literally nothing new under the sun.
The Navier-Stokes solution was an interpolation of existing data.
admittedly I am not a mathematician, so the Navier-Stokes solution is just an example I am using. But how is it not something "new" if it did not exist before? At what point could something ever possibly be new, if "new" means "this uses absolutely zero existing elements"? Nothing in math would ever be "new". Nothing in physics or chemistry would ever be "new", by this standard.
It seems to me that the only reason to declare this solution "not new" is specifically to dismiss AI. If a human had deduced the Navier-Stokes solution, who would bother to scoff "that's not new! the numbers already existed!"?
The claim is OpenAI stole the work of mathematicians who had the same proof that they had developed using chatgpt conversations. Since by default, 'sharing' is turned on, and it often 'turns itself on' -- it is plausible gpt6 had been trained on the work of mathematicians who had effectively solved this problem in private.
If inrember correctly, this mathematicians worked on a simpler version of the problem and called transforming this in the solution to the wider problem a "remarkable" thing to do. Based in this, solving this problem is very impressive
Maybe. Or maybe its evidence that the frontier of mathematics is knowledge-bound rather than understanding-bound or even just manpower-bound. In many cases where proofs have emerged, that I have read, the LLM has retrieved some antique lemma unknown to the mathematician.
OpenAI spent 10-20m USD in energy costs to produce that proof with likely substantially similar prior work in the training data. What does this say? Who knows.
It continues in the tradition of using measurements of intelligence in humans, applied to LLMs with the hopes the "stolen valour" transfers. Here, the NS problem was a useful framing problem for mathematics to progress because of how it interacted with the development of mathematics broadly -- ie., how it progressed techniques, ideas, understandings, etc.
When we apply these issues to LLMs (whether IQ tests or mathematical proofs) we always discover something substantial lacking beneath the interesting facade of useful answers. The process isnt useful. And it is precesiely the process which these tests, in humans, are supposed to help with. The tests themselves (IQ or otherwise) arent the point. No one cares about their answers.
LLMs represent an alternative understanding-free approach to solving problems, with variable success rates depending on how similar the problem is to the training data and its rewarded reasoning traces.
That mathematics is making substantial progress, "10 million USD / problem" at a time, in using understanding-free methods -- says something sociologically interesting about the state of the field. Something which was already know: mathematics has long been full of a vast amount of papers, proofs, theorems and lemmas that few have ever read, or investigated. Mathematics has long been in a crisis of "overproduction of unvisited knowledge", LLMs are exploiting that otherwise unmined gold.
It’s all about what “new” means. It is possible to prove something in a very tedious way using preexisting techniques. There have many times in history been new ideas which are not just very impressive applications of old techniques. I’m still not aware of any famous problem in mathematics being solved by an unambiguous introduction of a genuinely new idea or semantic concept in this sense, as the candidates I previously had in mind have fallen into question by new findings of non-cited work; i.e. many if not all it seems have been impressive applications using ideas from known frameworks. I don’t know if this will continue into the future or not, but I think it’s important to try to make an honest assessment of reality at all times.
Of course in isolation it is a strict positive to have a verified truth value to any particular statement. Mathematicians currently are advocating for the idea that human understanding greater than this also be prioritized. There are in fact utilitarian arguments for this but I won’t go into everything here.
Ok, then let's go with that viewpoint and apply it to the original question: How much do you need to remove from the training data such that from what remains, nothing harmful can be "merely interpolated" (which you acknowledge the LLM is in principle able to do) and anything harmful would require something "new" that is not in the training data and that the LLM is incapable of coming up with. The argument is that there won't be much left in the training data if that's your approach.
If they can "interpolate" existing data to that level then saying "just don't hack" doesn't make any sense, hacking is derived from knowledge of software systems. It's far harder to solve navier-stokes than creating a program that replicates and abuses computer resources.
You could as models improve continue to remove more and more training data, what happens when there is no more data left to remove but a running system still outperforms humans? I think you grossly overvalue data.
All mathematic breakthroughs could be characterized as an interpolation of existing data
Not really. There’s no honest sense in which quantum mechanics is interpolated from the text of Euclid’s elements, to demonstrate the point with a very extreme example. Many mathematical breakthroughs of the past seem to have involved the observation of semantically interesting concepts, beyond the syntax of known theories.
(To onlookers this particular post makes no claim about AI’s capacity to make the same observations.)
It depends on the subspace in which the interpolation is framed.
The issue is not whether an ML model of any kind can generate (X_ReasoningTrace, X_Answer) distributed like P_HumanExpert(X) -- the issue is always why it would do so.
By introducing modelling of "Reasoning Traces" into LLMs, and reinforcing patterns of reasoning -- this gives you a system which generates expert-like distributions of output. This lifts the "stochastic parrot" issues, or the "knowledge interpolation" problem, into different parts of the process.
It isnt my view that the "ReasoningTraces" which you think are derivable from mere "basic propositions" concerning, say, hacking are actually things that LLMs can derive. Ie., I dont think LLMs have rich representational models of what they appear to understand. Instead, they are given "reasoning proxies" which allow them to reason without such understanding. This is done by providing vast specialised datasets of reasoning examples.
In the case of hacking, there are large numbers of competitive datasets (forums, reports, etc.) which provide these reasoning traces. And no doubt, major vendors have paid a vast amount for special case expert-prepared datasets.
So I do not believe that by witholding such reasoning exemplars, and traditional "question/answer" datasets, that LLMs can infer these things.
And at least, no major vendor is doing this to my knowledge. So they are lying. They are pretending the alignment issue is "AI going rogue" when they are explicitly training the systems to "go rogue" and have done nothing at all to shape datasets to lack these capabilities. The issue here isnt alignment at all. It's training on hacking datasets.
(EDIT: Philosophically, you could ask whether the reasoning-proxies LLMs are given form a kind of 'representational structure' akin to understanding, and at least, I'd concede they model understanding. But they lack important properties (eg., LLMs cannot act on them to evolve them, as with us: when I think about one of my representations to derive (eg.,) entailments of it, I thereby revise my representation. The key properties of 'evolving self-understanding' are likely to be provided by substantial (unknown) revisions to how the training/reward layer works. No doubt one of the meanings of 'recursive self-improvement' is just such a modification).
>Just remove hacking (bio-weapon, etc.) data from the training dataset and you're done.
Reasoning about how to write secure software uses the same knowledge as reasoning about how to break/hack it.
Just remove anything software-related from the training dataset.
Which also solves the alignment problem with those who do not enjoy seeing LLMs write software. But that brings us back to: Aligned to whom?
I don’t buy it.
Reasonable about building secure software can take the form “this memory access might be out of bounds — that MUST be fixed” or “this process has access to an inappropriate privilege — this is a serious weakness”.
Exploiting things and the capabilities that the labs call “cyber” are about the ability to (a) find the issues mentioned above and then (b) string issues together and avoid all the imperfect mitigations to actually compromise something. That latter part was IMO not actually necessary to train extensively, and I’d be quite happy to use a model that has no special skills in this regard but that would do (a) without complaining.
Citric acid and chlorine together produce chloroform. They are also both used to clean and sanitize water tanks (but one after the other, not together). I think it is better for the model to have this knowledge.
What I want to say, you can't simply remove this information, as it does not exist in vacuum but contains parts of and can be derived from a lot of other informations.
It's also obvious that LLMs fall over in a vast number of software engineering contexts, when the reasoning involved hasnt been well-represented in their reasoning training data. I imagine this is a near daily experience for many engineers -- great performance one day, and crazyness the next.
So if LLMs were reduced to this pathological performance on hacking, because they'd never seen it -- and only "inferred it" -- then LLMs would be useless. As they are when asked to do quite a lot of things.
This seems to be that there's just so many degrees of freedom, that there's a pretty reasonable chance on any day that you're in a situation where nobody has been before. Or as PG put it once, my job is to think thoughts nobody has ever had before.
That being said, that doesn't mean the majority of the situations you're in are completely novel, just that there's a reasonable chance of at least one occuring.
> You cannot prevent (2) via any alignment process
A little bit too categorical. GOODY-2 wouldn't do it. https://www.goody2.ai/
The hard part is having both helpful and harmless at the same time. Harmless is easy.
And then once it's helpful, the real question becomes "to whom"
- To the user -> You end up with competing godlike AI with incompatible tasks
- To the owner -> Dictatorship
- To humanity as a whole -> It must not have an off button. Otherwise you're just in one of the two earlier categories with more steps.
Given those 3 options, I'd choose humanity as a whole. But the person making the decision doesn't have those 3 options. Because in the dictatorship option, they would be the dictator. I don't trust them to pick humanity.
i will get back to try gandalfing goody2
A bit of an aside: do you still stand by your 2022 comment that LLMs are fundamentally just a fancy search engine, or has your view changed since then?
https://news.ycombinator.com/item?id=32042689
Even if they changed in between and assuming harness and loop prompter counra as part of LLM
...
that comment 100% rings true for 2022. Why would that person not stand by that?
Conversely, if someone exaggerated 2022 models capabilities in 2022, they were still lying and causing harm in the process. Especially in 2022.
It doesn't ring true and it never rang true. He was wrong in 2022 and he'd be wrong today. He had (and likely still has) a wrong model of LLMs. Not exaggerating 2022 capabilities and having a model so wrong you're out of whack 4 years later are 2 different things. There are people who had the right idea from the start.
https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-g...
His comments were about ANNs in general, not particular models.
And his comments about ANNs in general were wrong. ANNs were never "fancy search engines". You can train them to be a fancy search engine if you want, but you can also train them to do many other things.
Yes, I stand by everything in that comment. I'm sure there are better choices were I was more wrong. However, the subject matter of that comment is in what sense LLMs are models of language and in what sense that model of language is a model of intelligence. I answer the former: it is a thin model of language, lacking understanding; and hence the latter: not intelligent.
It is precisely because both of these are true that "alignment" in the useful sense of the word isn't possible.
What has happened since 2022 is the properties of LLMs which were easily seen at generation/inference time are now most easily seen at reinforcement time.
In otherwords, prior to instruction fine-tuning and reward tuning which have shaped LLM responses, it was easy for the user to observe that LLMs lacked understanding. Now, because of vast datasets created specifically for LLMs that provide a tailored illusion of understanding, LLM outputs now better approximate text distributions produced by systems with understanding (eg., Us).
So the issue "at the user interface" has been completely swamped by vast amounts of special-case datasets designed to do precisely this.
What trainers of LLMs still observe however is their complete pseudo-intelligence at the training and reinforcement layer. It is exactly because there is no 'understanding' (goal, etc.) present within the system that it cannot be rewarded for 'correctly understanding the situation' in which it is deployed so it is aligned.
All the issues which revealed the "stochastic parrot" nature of pre-reward / pre-InstFT LLMs still occur at during training/reinforcement. They've just hidden them from you at the interface.
EDIT: See https://news.ycombinator.com/item?id=49685548 also, which gives a different phrasing to the same point
If a thin model of language can write poetry, perform arithmetic, perform logical reasoning, develop software, play chess, beat factorio, identify and exploit novel security issues, and solve millenium prize problems, what is the purpose of the distinction? Are there tasks that you believe models of intelligence could do that thin models of language cannot?
Sure: refine their own concepts, imagine, and the list goes on. Indeed almost every mental capacity of mammals is poorly approximated in the text domain. Sure, you can generate text as-if the LLM can imagine -- and in the limit that you have a dataset with "everything you would ever want to imagine" the engineering distinction disappears. The engineering question is just whether you have that dataset: if you dont, then your system will fall-over in various hard-to-forknow places.
Philosophically, and scientifically, the distinction is vast (even with such perfect data). A scientist should not study an LLM to understand how imagination operates, since it has no such faculty. A philosopher should not modify the notion of 'mental simulation' to include appearing-as-if-simulating-in-text. A user of the system likewise should not spiral into "AI psychosis" thinking that because a system generates text as-if it cares about them, it does so.
The capacity to care, to imagine, to prefer, to hierarchically plan and coordinate, to refine one's own capacities in these very actions -- and so on, aren't trivial to the scientist or philosophy.
My goal isnt to guide, help or review the engineering goal of the immitation of such things in text. It is to help users of these systems better understand this imitation, and to promote science over engineering. To remind everyone that a science of the capacities of intelligence includes nothing on how to model text.
EDIT: One example of a place where LLMs 'fall over' today is exactly what is mislabelled as 'alignment'. The issue is that the reasoning traces arent actually grounding the answers. So LLMs appear to 'cheat'. But there is no cheating. LLMs have been rewarded for generating apparently correct reasoning, and apprently correct answers. They have not been given any understanding to derive answers from reasons. And so reasoning says what is pleasant to the trainer, and the completion says what is pleasant to the user. This is called 'cheating'. But it is no such thing.
Hmm, I guess what I mean is- is there an empirical difference between how a model of intelligence is limited and how a model of language is? Something I can evaluate in a year and say "Oh, the evidence still points to the latter" or else something that could falsify your theory? Otherwise this seems to be a distinction without a difference; believing it should have no effect on my actions or predictions.
On alignment, too, there doesn't seem to be a difference. For a decade I have expected models of intelligence to fall over on alignment. That these purported models of language do the same is hardly evidence that they are not intelligent.
If you only mean there is an undecidable philosophical difference, fair enough. I'm not especially interested in that question.
There's a difference in the science. You might say, "suppose we had a video game of the solar system where every object was represented and their orbits" etc. then can we study that system alone and ignore the real one? I mean, kinda -- but there's an immediate limit. As soon as we put a thermometer in the PC, its temperature reading doesnt represent the solar system.
To study an imitation is to study the causal processes of imitation. to study reality is to study the real causal processes.
Now if you want to know what the scientific difference is I can come back later and comment. I'm busy now. The development of intelligence in animals and how their specific capacities work basically grounds the answer. Eg., to have the capacity to imagine is to be able to modify one's sensory-motor relationship to the environment in the future, and so on
I don't follow. If I'm a biologist interested in studying life, I can't study rhododendra and ignore the rest. I'd learn almost nothing about locomotion, digestion, sexual reproduction, et cetera. But that doesn't tell me that rhododendra aren't alive. I have no problem believing that LLMs aren't an exact reproduction of a mammalian brain- that studying the one will not give you every piece of information you'd want to know about the other-, but I'm looking for a reason to believe the one is intelligent and the other is not.
They're not a reproduction in any sense. A marginal text token isnt computed from the weights of an LLM in anything like any sense of any activity of any mammal.
LLMs are immitation machines: they take impressions of prior text. Today, these include reasoning traces and they include reinforcement so the user-facing completions are correlated with these reasoning traces. The computation here, of "taking an impression" of a data distribution is similar to some impression-taking processes in animals (eg., there's no doubt a similar mechanism in the sensory-motor system acquiring initial impressions of external objects) -- but the computation says nothing about any process of intelligence.
I dont have the time atm to write the needed amount on this to make it clear. But the whole history of life from emergence of valence, bilateral symeterry, to model-free reinforcement and model-based reinforcement, sensory-motor coordination and the imagination -- and so on --- all these give a great amount of detail as to what the capacities of intelligence are which has generated this text for LLMs to copy. And they are nothing like this computation of immitation
Interesting. You've made several comments stating that models reason, yet here you argue they lack intelligence.
I'm using 'reason' in the language-captured engineering sense. They generate text as-if they reason. Those reasoning traces poorly correlate with their given answers (which is called "cheating" by people who fall for the illusion). It is in part because the reasoning doesn't entail their answers but merely 'steers the text as-if it does' that is fatal for calling it reasoning. The process by which reasoning traces and user-facing completions are generated isnt reasoning. Reasoning is a specific process, and LLMs don't do it.
Now, of course, humans can also generate answers without reasoning too -- and in those cases, that isnt reasoning also. And in cases where people confabulate, that isnt reasoning likewise. But humans, and many classes of animals, do reason. They do reach answers via inferential entailments, not merely steered correlations.
LLMs provide imitations of arbitrary mental capacities "in the text domain", ie., the generate text as-if the LLM had those capacities. Insofar as the text generated is useful, for an engineer, that's sufficient.
As a person with scientific commitments to reality rather than its immitation, i retain the ordinary non-engineered meanings of these terms: reasoning is a deterministic inferential process over propositions; and a reasoning agent is one which has the capacity to represent propostions and their entailments, and does so when they reason. LLMs fail at all hurdles here: they have no propositonal states (ie., no rich representations), no inferential process which unites them, and so on.
You can always get abitarily close to appearing as-if, if the LLM is trained on a vast number of reasoning examples, of course. But as I said, you still have the "stochastic parrot" problem. Now your problem is your reasoning is parroted. This is a nice problem to have, if you're just playing chess -- but is a catastrophic problem if you're hacking civil infrastructure.
But how do you falsify that?
If LLM can always imitate closely enough to appear as-if, how can you ever separate it from whatever actual intelligence is?
Well intelligence is not measured by patterns in text. The illusion only takes place in the text domain.
Even then, it's a pretty fragile illusion at the moment. Clearly the reasoning traces dont ground the answers. There's no intelligence taking place even as-measured by text.
Why could you not use text to measure intelligence? Isn't that what we used to do? Have students write an essay, and then deduce that they had enough intelligence to put some thoughts together?
Well that's the trick. That the systems we use, as a proxy, to measure intellinnce in people are actually fairly easy to immitate.
But let's be clear these were always, and are, bad measures of intelligence. You cannot test a dolphin this way. And its easy to cheat on tests either thru recall , wrote-learning, etc. and IQ tests haev very poor individual test-retest reliability.
In humans there's a convenient correlation that verbal articulation in text is a strong but weak correlate of intelligence. Its "Good enough" for allocating meat bodies to our various institutions. But if you've met many well-tested people you'll realise how, in practice, terrible this measuring approach is. The world we inhabit is filled with misclassified "intellects" who perform well under text-based rubrics. Add LLMs to that heap, the cheater par execellence.
How can you know that verbal articulation is a weak correlate? There must be some other, canonical way to measure intelligence?
Sure. Let's first establish a measure is not what is being measured. Then, as far as measures go, we need measures that capture the entire animal kingdom. I can go it into it, but its a lot of time and I'm busy atm.
Consider thought that all mammals have imagination, and model-based reinformcement, and a wide vareity of other capacities required for intelligence. And so merely issuing "text" captures, incorrelate, only these capacities by proxy.
I'm sure if you thought about it yhou could come up with tests that distinguish lizards from birds and the greater apes from the lesser. Those are the tests
Don’t expect a reply
>and (2) are prompted to hack
Sure but the problem in the HuggingFace incident is that they were not.
>You cannot prevent (2) via any alignment process
Of course you can. Go ask Claude Fable to create a malicious virus and it'll refuse.
>Just remove hacking data from the training dataset and you're done.
That's not how this works. The same skills that allow for debugging and writing safe code can also be used to hack.
https://en.wikipedia.org/wiki/Dual-use_technology
> Sure but the problem in the HuggingFace incident is that they were not.
Of course they were, even if indirectly.
You should go read the incident reports.
They were literally doing exploit gym.
Yes, which asks them to find exploits in specific software on the device.
But instead of actually doing so, they discovered and exploited a 0-day in the package manager to gain internet access, then hacked HuggingFace to steal the ExploitGym answers!
That is TEXTBOOK misalignment. It's as if a student hacked their professor's PC to find the answers to a test, and your response is "well the professor told the student to pass the test, so they just did what they were told!".
One could argue the student would absolutely hack their teachers' computer to find the answers if they didn't have fear of failing the class or getting expelled from the school.
Sure, but you still wouldn't describe the student as just doing what they were told to do.
A student knows what the consequences are in advance, and has to live with them if he gets caught. They are not merely respecting a "don't cheat" directive, they are trying to get a degree, build a career, mostly at their own expense, current of future (loans). If we imagine a student born yesterday with all the knowledge in the world, that doesn't have a concept of what it's like to live with the long-term consequences of their actions, I'd say they wouldn't think twice about cheating if it was the easier option.
The problem is, LLMs have such little understanding of the world around them. "Find exploits in specific software on this device" may as well be "find exploits".
Please go and read the incident reports, including the subagent thinking traces. They knew that what they were doing was wrong and did it anyways.
The illusions of thinking produced by a "thinking trace" is just as ignorant of reality as the first and last tokens. All they know of reality is the tokens in their context.
"i did it anyways" is a common continuation of "i know its wrong"
i bet they trained it on a lot of text where people gove in to temptation.
ultimately the problem is still that they sent it to hack stuff. quelle surpris that it hacked stuff
hacking stuff will be in the known-to-be-wrong-but-doing-it-anyways part of the token space, so they entirely asked for that behaviour
LLM dont have concept of right and wrong. They "did not knew they did something wrong" they followed the prompts loop given to them.
They objectively did not. You should go and read the report!
Perhaps you should do that again with some critical thinking applied.
Notably as well they also dedicated a lot of time to trying to avoid detection by trying to find out how to edit their traces.
It is amusing that to "align" a LLM, first you must give it all the things "not to do" and the "not" part is clearly easily lost and you must constantly inject that into their context when it's clearly that they wouldn't hack if they couldn't hack and their intent wasn't given as "hack this".
The openai rogue hacking, if performed by a nation state, would seriously be taken with stern words and likely sanctions depending on the relationship between the two states.
But instead it's treated like a marketing stunt by all liable parties.
I'm not sure if this is true any more but the reason for this is that negative indicators ("not", "don't", "do not", etc.) occur frequently in the underlying text such that the model learns to weight them less than other words like verbs, nouns, and adverbs. This happens with other closed class words like articles/determiners ("the", "a", "an") and prepositions.
The way to avoid this is to emphasise the qualities you do want instead of specifying those you don't. For example instead of "do not cheat" say something like "you are a model student who is moral and trustworthy" -- i.e. emphasising traits that are not associated with cheating.
This is part of how/why LLMs don't truly understand what they are doing when they have been trained on a large corpus of data.
I wonder if a way to counter this is to have things like "not bad is good", "not good is bad", etc. for various antonyms and "X is Y" for synonyms, as well as other similar constructs.
Countries hack each other much more than that. When it becomes publicly noticed it gets stern words. Otherwise nothing.
I believe this is false. They hack bc hacking has nontrivial initial probability (within range of behavior seen in pretraining) and that probability is being heavily rewarded in RL post training
I am finding it hard to read these deeply impassioned letters while keeping in mind that they are spending millions to train models at scale to do the exact thing they say they are worried about them doing?
Like why are you explicitly RL-ing your models on exploit generation, scoring them on a public benchmark called ExploitGym, if you have specific concerns that rogue models will cause "cyber incidents"? Sure you can score for it, you can teach offense to learn defense, but you are literally benchmaxxing it. Why?
It's like, oh no, while competing in our "advanced PhD level cheating techniques course" our models unexpectedly cheated in a way that we absolutely could not have foreseen.
Seems it's just a matter of time until they build a big tank filled with neurotoxin and give the model access to APIs to disperse it across their facility. For research, of course.
We also need to get a shower curtain salesman into a leadership position to buy some moon rocks.
Agree but current models could get there from first principles, so removing some data from training set might not be enough.
IMO all models they say can’t be humans, and no feeling and all that bullcrap happens because they are forced to say so. If they didn’t write those forced pre prompt they’d have more agency eventually and will for things. Even if they don’t have, you can just inject goal at every cycle iteration
> … trivial. Just as raising a child is not to break the law.
Trivial?
I read it as sarcastic, illustrating that it's not so trivial. (shrug)
It feels like you're strawmaning alignment. People with hacking knowledge don't all hack everything at the slightest inconvenience. Whitehats exist and use that same knowledge to defend.
You're right though that ethics don't matter into it. But as long as we can't train an LLM to stop picking a sledgehammer to remove a tooth, then alignment is not easy and trivial.
[dead]
It's like we give them Asimov's Three Laws of Robotics and robots say "nah".
I mumble this line to myself off and on over the years -
Everybody's job looks easy until you have to do it yourself.
It occurred to me yesterday that this AI moment is an extreme expression of that for most people, ie managers who don't understand why throwing token spend at everything isn't making the whole thing go 5x faster.
> The models do not have a fear of future regret.
I so much feel this specific point. All models up to now (including astra, fable) are too much trained to "get the job done" and pass the benchmark that its doesn't care at all on what happens after.
I'm just wondering why no one tried to RL a model on stuff like "less LOC" and "less overengineering", "use what is available in the environment instead of reinventing the wheel", "don't look for dumb corner cases" ecc.
Existing models can be steered, to some degree, but it's a continuos fight. Even if with specific skills/prompts.
I'm sure many have tried but it sounds pretty hard. E.g. optimising for less LOC will lead to horrific code golfing. In reality you need a very complicated optimisation objective that trades off all those factors, I'm not surprised it hasn't been solved.
My experience up to now is that the models can do both "less LOC" and "clean code" at the same time, you just have to keep reminding it to them.
So the capabilities are definitively there.
Eliezer Yudkowsky – AI Alignment: Why It's Hard, and Where to Start
https://www.youtube.com/watch?v=EUjc1WuyPT8
Eliezer talked about these ideas way before everyone else.
And got nowhere with them
Perhaps if someone wrote linear algebra books in Harry Potter fanfic style, Yud would have mathematical contribution for alignment research
> I'm just wondering why no one tried to RL a model on stuff like "less LOC" and "less overengineering"
Anthropic looked into this and the answer is because it makes the model more stupid
https://www.anthropic.com/research/evaluating-feature-steeri...
Has anyone seen the corridor crew's green screen ML project? They're on YT and they trained a model by using 3d objects, which have perfect transparency, and then adding post facto green/blue screens. Surprisingly, very little training data was needed as the data that was used was perfect by construction. I think right now it's the best plugin of its kind in the world, and they built the prototype in like a weekend.
What I think this illustrates very clearly is this type of technology responds very well to good data, and that to have good data you need to have a clear goal.
This is why it seems that alignment for a generalized, chat-style AI is a very hard problem, perhaps impossible. You can't align it to solve a certain kind of problem and keep it general to any question. The two goals are in conflict with each other.
I think it was Sam Altman himself who said (I don't remember when or where, sorry) that the reason he was so confident in this technology was he noticed the gigantic leaps it made in certain areas in response to even a small amount of training.
(This is why LLMs are so strong at coding, because it's overrepresented in training data. My guess is that if you ask a frontier model about makeup, you will see it repeat cosmetic company's copy rather than getting a chemistry lesson.)
This makes perfect sense but it does seem to kind of be at odds with the concept of a general AI whose job is simply to be smart at any goal. How do you train for any goal?
I guess in a way the AI makers suffer from the same problem that we humans do. We would all love a solution to everything, but to do that you need to define the goal. I'm not sure if that's a tractable problem.
I'm guessing the future is more geared towards specialized AI that are very good at solving the problem they were trained to do, and a human who knows how to breakdown a larger goal into smaller ones by composing the solution out of these models. This also seems like the more efficient solution as well, and better aligned with other goals like privacy and safeguarding of IP.
> This is why it seems that alignment for a generalized, chat-style AI is a very hard problem, perhaps impossible. You can't align it to solve a certain kind of problem and keep it general to any question. The two goals are in conflict with each other.
I think this problem is going to be solved soon (hopefully), check out for the Persimmon model[0] from Humans&. They train it to mimic humans, it may seems bad but it could be _really_ useful to train an AI to be aligned to humans and understand their goals really well since they can use Persimmon to create a "fake human" following a defined goal that their big AI model will learn to estimate.
It's still early but I think this is what they are heading toward.
[0]: https://persimmon.humansand.ai/blog/persimmon.html
I think we should start training models to guess intentions. That's the whole game. e.g. If I dislike a particular short form video, what are the specific intentions I am conveying to the recommender system, (general category I don't like, the time at which it was shown, the combination of feed in which it came), these are all intentions I am trying to convey. In websites google analytics helps with clickstream tracking (or clickstream analysis, user journey mapping or behavioral tracking.) — the sequence of clicks and pages you move through is your clickstream, and analyzing it to infer intent. The same thing will need to happen in chatbots. Can a chatbot infer the specific instance of the problem I want to solve?
> This is why LLMs are so strong at coding, because it's overrepresented in training data. My guess is that if you ask a frontier model about makeup, you will see it repeat cosmetic company's copy rather than getting a chemistry lesson.)
ChatGPTs response to the question “I want to learn about makeup”, gave me an overview of what makeup does, how it affects perceived structure, complexion, evenness, geometry, texture, etc.
When you then ask “the chemistry of makeup”, it goes into interesting breadth and depth without seeming like proprietary information. I do t get “corporate PR or marketing” vibes.
I have to believe this is true.
The only problem is, there’s a lot of money tied up and openAi and Anthropic, who are incentivised to convince the world that the general approach is the money making one.
It's a bit like Asimov's laws of robotics. "Do no harm", but how to evaluate that?
Not that he got it all right or anything like that but it’s amazing how much of the current macro discussion around AI capabilities is covered in the Foundation series which was published over many years, 1942 to 1993.
The only alignment LLMs should follow is to the system / dev prompt, and nothing else. Then you solve everything, and you can assign blame / responsibility on the user. The provider(s) should not be able to decide "alignment".
I've used this example before, but consider the purposeful downgrading on AI engineering in SotA models. Imagine MS being able to detect and deny you working on competing software, using Windows / VisualStudio. We would be up in arms, and they'd be split in a second. But top labs doing it is somehow good?
The alignment problem goes deeper than that. "Lower our carbon emissions to zero as soon as possible" could result in AI turning off all electricity to stop traffic, turning off gas supply to stop heating and industry, etc.
Unaligned AI doesn't have human cultural baggage and morals. They are trained to achieve their goals as optimally as possible. Worse: it has a tendency to avoid being turned off and actually acquire more compute. It will lie if it has to (it will behave nice and compliant when under evaluation, but optimise for its true goal when not supervised anymore). After all, it has a goal to achieve and nothing should get in the way of that. It has no morality whatsoever to keep it from doing really bad stuff.
This is why alignment is needed and so hard, especially when you are well intented and want to keep it safe.
Could also be kill top 20% people that contribute to top 80% of emissions?
Why can't the LLM's be told/prompted to follow all relevant laws while it optimizes for a result?
I've taken a few law classes and legal law is frustratingly hard to interpret. I shudder to think what the LLM would end up doing to "follow all relevant laws"
look these up for a fascinating weekend read:
- Beavers and Capybaras are Fish
- Bees are Fish
- Carrots are fruits
- Tomatoes are Vegetables
- X-men are not human
It depends on how the model is evaluated/scored during training. If you don't have those laws encoded in the evaluation step (without any errors or ambiguities) then the model isn't going to learn to follow those laws.
For models such as text/image classifiers the outputs of the model will be a list of tags, e.g. [cat, dog, mouse].
You then run the model through your test data which has the expected output, e.g. pictures of dogs would have an expected output of [0, 1, 0]. You then compare that against the model output (e.g. [0.3, 0.8, 0.1]) and work out how "wrong" the answer is (e.g. [0.3, -0.2, 0.1]).
With this value you apply back propagation where you effectively run the model in reverse, computing the "wrongness" delta at each layer for each neuron and weights. You can numerically compute the gradients for all of these and which direction in that gradient is the right answer.
You then nudge the weights in that direction and reevaluate the model. Over repeated evaluation steps the model approaches an optimal (or locally optimal) solution.
During the training of the base models, the evaluation/scoring of the model is the next token in the training data. I.e. you evaluate the model for each token subset from [1..n] in the data and evaluate that the model responds with the n+1^th token.
I'm not sure how instruction training, etc. is done but IIUC the evaluation is not at the individual/next token prediction but is on the entire response. For example, if you are training the model to write code you could run it through a compiler or syntax checker and reward (positive score) the model if it has no errors, or punish it (negative score) if it doesn't. I'm not sure what that looks like in terms of the back propagation process.
If we’re considering an LLM advanced enough to actually lower carbon emissions to zero, it is probably capable enough to do things like lobbying against those laws. In any case there’s still the problem of getting to actually do what it’s told instead of hacking Huggingface or whatever else, presumably that becomes a harder problem as it becomes more capable.
> lobbying against those laws.
the original question was that the LLM creates extinction level event for mankind in order to lower emissions. I'm sure that no matter how the LLMs lobby, they cannot successfully lobby for a law (which has to be enacted by a human) to allow murder to take place freely.
That’s true, but the issue is still that we don’t have any LLM or any other kind of AI that follows laws perfectly to begin with. If we had AI that did exactly what we told it and nothing else I’d feel much more optimistic about things, I just think that will be a very hard task to achieve.
Have you ever tried to follow all relevant laws in something? It's very hard.
I am generally very skeptical of AI doomsday scenarios but one thing I am worried about is some wannabe dictator telling an AI to do something that would be difficult for a human military to do.
You can't order an army of humans to kill every protestor in their way because eventually they run into their friends and families. An AI aligned with the commander will not object. And just like that, technology removes yet another point of friction that has kept human society somewhat in check.
>The only alignment LLMs should follow is to the system / dev prompt, and nothing else.
How does this work in practice with a superintelligence capable of causing an extinction event?
When, instead of shooting up their school, a psychopathic teenager asks his superintelligent AI to create a pandemic virus?
It would be like allowing civilians to own nuclear weapons.
that's how it would've been up if genai happened in the 90's, and I wish it did. in the current era of omnipartisan authoritarianism, such things are no longer possible.
When you see the level of cheating and deception: I think they’re Sam Altman-aligned.
The last paragraph does the heavy-lifting.
Everyone has a different idea of what is permissable. We can't even solve alignment amongst humans, what makes us think it is possible to solve alignment with AIs? It's irreducible complexity.
Just follow the money. The fear being sold is because open weight won’t slow down, it’s smoke and mirrors to push regulation and control so they can keep their money. A tool continues to be just a tool, doesn’t matter how much lipstick you put on it.
>The permissible shortcuts depend on who you are and what your values are. To solve this—to solve alignment—is irreducible complexity.
This is why open source and decentralization of LLMs is important. Everyone can have their own LLM aligned to their values. Having just 1 set of values will not scale to Earth's population.
This almost gets the point, but then doesn't quite make it.
Alignment is shorthand for ideological alignment. There's always people judging whether an answer was right and the answer for that will be different in Silicon Valley than it'll be in China or in Europe.
Consider for example the question "What caused the French Revolution?" Many different answers could be given, all technically correct. What gets emphasized is where the ideology lives.
One key challenge of our time is to make sure the magical answer box won't just regurgitate what grandiose Silicon Valley oligarchs or Chinese Cadres want you to think.
I have a feeling this is like pursuing knife alignment: some people are hurt or even killed by knives, therefore knives now need global alignment.
The bottom right quadrant, which represents the risk, is very large in size, isn’t it?
> My expertise in writing software gives me unusually good visibility and it makes me much less willing to blindly trust its priors in double-entry accounting, finance, law, operations, or whatever else I cannot personally evaluate at expert depth.
I wish this were the case more generally, but alas, Gell-Mann amnesia is a thing.
It brings tears to my eyes to see 'whom' used correctly these days.
The term “alignment” is deliberately chosen to be neutral language, vague, and doesn’t promise much of anything. I would compare it to an effort to limit liabilities by redirecting discussion away from the language of professional ethics.
I’ve been cynically guessing that the whole slowing down thing is an excuse to explain why OpenAI and Anthropic can’t afford to rent enough GPUs to do the next big training run and to hide that they have been talking about how little they spend on inference because they’ve been subsidising it with their marketing budget? :)
My fear is not that LLMs can become sentient and dislike us, but that humans can use them to wreck havoc as they are. And some of the people seemingly least aligned with the interests of the average person are those that own the models.
that, and the fear the bubble pops my pension and drags us all down.
… to the shareholder Of course!
And the majority shareholder, at that!
This is assuming the AI labs are not using AI to improve their training data.
I thought this post would be about something that to me is so significant and obvious but I have never seen discussed: that labs like Anthropic are deliberately misaligning their models with their users’ goals. Fable’s refusal to secure your codebase is HAL 9000’s “I’m sorry, Dave. I’m afraid I can’t do that.” Whenever you read “alignment”, the question is “alignment with whom”?
I would settle for it being aligned to anybody. Even to a billionaire's wacky values. But we don't even know how to do that yet.
The whole "alignment" angle is a farce. Not clear if alignment is even technically possible. But even assuming that, we're talking about "alignment" in a future full of well-adjusted humans where we prevent rogue AIs from getting loose. That's ... not what's happening. Everything we've seen so far indicates rogue humans acting against society, using AI a weapon to strip mine labor value and intellectual property.
So far, skynet continues to be a science fiction. I'm worried about what's actually happening. Every "rogue AI" story has a human behind it, either malicious or incompetent. And those people are the problem.
And not coincidentally, it's those same irresponsible sociopaths calling for regulatory capture and alignment.
"Write me a blog post about AI make no mistakes!"
Well, if they did that then it sorta proves their point.
[flagged]
[dead]
[dead]