I still think this was a fire drill. They knew it wasn't dangerous but they wanted the public to do their homework and come to the conclusion themselves.
> This decision, as well as our discussion of it, is an experiment: while we are not sure that it is the right decision today, we believe that the AI community will eventually need to tackle the issue of publication norms in a thoughtful way in certain research areas
The samples are striking in their simplicity compared to the over-controlled system prompts we have now. For example, the intended format is as follows:
"GPT‑2 generates synthetic text samples in response to the model being primed with an arbitrary input. The model is chameleon-like—it adapts to the style and content of the conditioning text."
The results don't sound anything like today's models but note that each one took 10 attempts:
> ## System Prompt (human-written)
> In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.
> ## Model Completion (machine-written, 10 tries)
> The scientist named the population, after their distinctive horn, Ovid’s Unicorn. These four-horned, silver-white unicorns were previously unknown to science.
> Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved.
> Dr. Jorge Pérez, an evolutionary biologist [...]
I think their legitimate concerns were more about AI slop.... which is technically a form of spam.
While it was difficult to make it follow instructions, GPT-2 was still reasonably good at generating SEO-spam websites, and reasonably easy to make it do so. AI slop was a very obvious usecase for LLMs as capable as GPT-2, and we are talking about an era were everyone was already concerned about the way "fake news" on social media was being used to manipulate people. Perhaps even more concerned than we are today.
But all evidence suggests OpenAI also had strong ulterior motives. They were busy readying the API groundwork for monetising it, and wanted to delay any competition by as much as possible.
The public discourse was very different then. Misinformation, bias, etc. Those in power were much more concerned about an LLM saying something unapproved.
Wild that we accept that OpenAI didn't want to deal with the social/political fallout of people realizing that their tool was responsible for flooding the internet with misinformation and bigotry?
Sure, now that all seems quaint, but I can definitely see how a fledgling AI company would care about their reputation.
Look up how the wireless radio helped spread European fascism.
It's not about "the people can think this but can't think that", it's "how does this new communication platform change the existing political culture and how will we adapt?"
I'm starting to think that AI development might be the great filter. There are so many levels of competition at play here and even if an agreement is reached to slow everything down, the incentive is to cheat at that. I really don't think there's much of a chance we have control over the situation, we're just gonna summon these things and all we can do is pray they don't want to kill us.
How, exactly, is a deterministic text-generation algorithm going to “want to kill us” when it’s not capable of having desires, emotions, etc? If it does attempt to kill us, shouldn’t we be blaming those humans who instructed it to do so?
Put another way, could we stop anthropomorphizing the text generation algorithm, please?
I'm not a power user, but I sometimes use some software development agents. Sometimes they do different things than what I asked for. Sometimes it requests permission for system operations like file system access that are totally unnecessary for the task.
I can't believe that anyone has had more than a day of exposure to one of these without falling below 100% adherence to requests.
> A spring does not want to do anything
When people say a spring "wants" to return to its initial length, they are not engaging in philosophy. They are using a linguistic shorthand to simplify an physical explanation.
Why should we stop anthropomorphizing the text generation algorithm? Because it leads to psychosis and to making hyperbolic, dangerously wrong statements like “pray they don't want to kill us.”
But why are Dario, Sam and Elon who represent "the frontier" more than any other 3 people pretending like they need government permission or intervention to pause? It seems more likely that they see open source distillation as an existential threat to their IPO plans and are playing for regulatory capture of the market.
It would require government permission. The top labs would be agreeing not to compete on model advancements for a period of time, which seems like a pretty clear violation of antitrust law.
How is this a clear violation of antitrust law? And have any of the leaders of these companies said that they are asking for government permission for this reason?
I still think this was a fire drill. They knew it wasn't dangerous but they wanted the public to do their homework and come to the conclusion themselves.
That was, explicitly, part of the decision :
> This decision, as well as our discussion of it, is an experiment: while we are not sure that it is the right decision today, we believe that the AI community will eventually need to tackle the issue of publication norms in a thoughtful way in certain research areas
You mean PR to get more publicity
This is it, and all the other comments here are continuing to do that homework!
So they lied to the public then?
Wow, we should really trust these people today.
Judging by what people who worked closely witb Sam Altman say about him, you should not trust an organisation led by him.
The samples are striking in their simplicity compared to the over-controlled system prompts we have now. For example, the intended format is as follows:
"GPT‑2 generates synthetic text samples in response to the model being primed with an arbitrary input. The model is chameleon-like—it adapts to the style and content of the conditioning text."
The results don't sound anything like today's models but note that each one took 10 attempts:
> ## System Prompt (human-written)
> In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.
> ## Model Completion (machine-written, 10 tries)
> The scientist named the population, after their distinctive horn, Ovid’s Unicorn. These four-horned, silver-white unicorns were previously unknown to science.
> Now, after almost two centuries, the mystery of what sparked this odd phenomenon is finally solved.
> Dr. Jorge Pérez, an evolutionary biologist [...]
I wonder what they saw. GPT-2 has no malware applications. Maybe they were concerned about someone using it for spam?
I think their legitimate concerns were more about AI slop.... which is technically a form of spam.
While it was difficult to make it follow instructions, GPT-2 was still reasonably good at generating SEO-spam websites, and reasonably easy to make it do so. AI slop was a very obvious usecase for LLMs as capable as GPT-2, and we are talking about an era were everyone was already concerned about the way "fake news" on social media was being used to manipulate people. Perhaps even more concerned than we are today.
But all evidence suggests OpenAI also had strong ulterior motives. They were busy readying the API groundwork for monetising it, and wanted to delay any competition by as much as possible.
The public discourse was very different then. Misinformation, bias, etc. Those in power were much more concerned about an LLM saying something unapproved.
Agree. It was more a worry that someone would create a bot posting lies and bigoted ideas to Facebook/X etc.
Wild that we just accept that today, honestly.
Wild that we accept that OpenAI didn't want to deal with the social/political fallout of people realizing that their tool was responsible for flooding the internet with misinformation and bigotry?
Sure, now that all seems quaint, but I can definitely see how a fledgling AI company would care about their reputation.
Wrongthink should be banned!
A tedious response, no?
“Automated bots posting verifiably false information online to stir up tensions should be banned” is a statement I think plenty would agree with.
Look up how the wireless radio helped spread European fascism. It's not about "the people can think this but can't think that", it's "how does this new communication platform change the existing political culture and how will we adapt?"
See also the 2023 call to “ pause for at least 6 months the training of AI systems more powerful than GPT-4.” https://futureoflife.org/open-letter/pause-giant-ai-experime...
Signed by Benjio, Musk, Woz, etc
I'm starting to think that AI development might be the great filter. There are so many levels of competition at play here and even if an agreement is reached to slow everything down, the incentive is to cheat at that. I really don't think there's much of a chance we have control over the situation, we're just gonna summon these things and all we can do is pray they don't want to kill us.
This is what people were saying about nuclear technology half a century ago.
And? Nuclear technology continues to be an excellent candidate for the Great Filter.
How, exactly, is a deterministic text-generation algorithm going to “want to kill us” when it’s not capable of having desires, emotions, etc? If it does attempt to kill us, shouldn’t we be blaming those humans who instructed it to do so?
Put another way, could we stop anthropomorphizing the text generation algorithm, please?
How can a spring want to return to it's initial length? These machines have done plenty of things not prompted for.
A spring does not want to do anything. An LLM does not want to do anything.
> These machines have done plenty of things not prompted for.
Like what?
I'm not a power user, but I sometimes use some software development agents. Sometimes they do different things than what I asked for. Sometimes it requests permission for system operations like file system access that are totally unnecessary for the task.
I can't believe that anyone has had more than a day of exposure to one of these without falling below 100% adherence to requests.
> A spring does not want to do anything
When people say a spring "wants" to return to its initial length, they are not engaging in philosophy. They are using a linguistic shorthand to simplify an physical explanation.
why?
Why should we stop anthropomorphizing the text generation algorithm? Because it leads to psychosis and to making hyperbolic, dangerously wrong statements like “pray they don't want to kill us.”
They were worried about the rap battle between Paul Graham and Michael O. Church.
Which sadly never happened.
[dead]
If it is possible to save one life, preventing RSI AGI is an existential threat.
But why are Dario, Sam and Elon who represent "the frontier" more than any other 3 people pretending like they need government permission or intervention to pause? It seems more likely that they see open source distillation as an existential threat to their IPO plans and are playing for regulatory capture of the market.
It would require government permission. The top labs would be agreeing not to compete on model advancements for a period of time, which seems like a pretty clear violation of antitrust law.
That isn’t any kind of antitrust violation. Agreeing not to undercut each other’s prices would be, however.
How is this a clear violation of antitrust law? And have any of the leaders of these companies said that they are asking for government permission for this reason?
Meme marketing. I can attest that no one at these companies cares about laws. Not one bit.
... is what they believe.