- Anthropic told everyone Mythos was dangerous because it's proficiency with biologics and cyber security
- Anthropic didn't release Mythos like everything else. They released a neutered fable. They didn't get rid of Mythos
- Anthropic opens a lab in SF
There was always a quesiton of "will the labs stop releasing their models and start building around them instead?" Yes - they already have. Anthropic is a biologic and cyber security company, in addition to intelligience.
Personally I wonder if they've been holding back a lot. Opus 5.5 was a good release after a little stagnation. Open AI releases good models and everyone says Anthropic sucks and -- Oh would you look at that - a better model finally and all of a sudden.
with how much competition, benchmaxxing, and increased compute for inference, I can't imagine that they are intentionally nerfing their own public models for any reason other than that they can't figure out how to package it into something the public can use. The training data for all these frontier models goes well into 2026 at this point.
Yes, both labs already have monstrous internal teacher models they don't sell for inference, this is generally acknowledged. They cut releases for the public just to keep revenues growing, it's not their actual frontier.
Ok. Given this hypothesis, why is the software they release generally considered crappy by competitor standards, benchmarks, and open source standards?
Claude Code is an awful codebase, has leaked its own source code multiple times, and scores the worst on number of tokens burned vs pass rate percentages.
This article is wholly flawed from the start where it claims nothing has been done, no actual effort made. A straight forward search of "what efforts were made and safeguards put in place subsequent to the CAISS statement in 2003?"
It also shows the same reasoning error mode many criticisms of a precautionary initiative or intervention to a problem:
Assuming that a problem whose trajectory was at a certain place when the initiative began has failed simply because it isn't solved on their own wished for timeline or standard of success, or that it wasn't meaningfully changed from what itnwod otherwise have been.
What happened to realizing there are hard problems, that different things may need to be tried, or that those things tried were partial but not complete solutions?
I always found it odd that people who are intelligent enough to work in IQ-loaded fields are as easily manipulated by either rhetoric or ideology as anyone else.
growing up I always assumed that everyone would grasp some aspect of game theory intuitively, I still remember the day I found out game theory was a thing - it was like finding out that someone successfully systematized common sense.
most people take the things people say as if they were worth considering. signals without cost are only useful as knowledge of what the signaler wants fools to believe.
> I still remember the day I found out game theory was a thing - it was like finding out that someone successfully systematized common sense.
Game Theory as a field is actually very complex and often reveals dominant strategies that would never occur to someone as the "common sense" approach to a problem. It was arguably created to solve for problems where there was no obvious correct answer, even for a very intelligent person.
The world is not full of idiots, rather communication channels are heavily managed to prevent people from establishing consensus around the obvious.
No matter how many people think press release X is obviously wrong - 80%, 95, 99% - it'll always be "the comments section criticizing the article," or "bloggers sharing incisive criticism."
Surely it depends on what you're trying to do? If someone is deeply knowledgeable about how AI works, then it will be difficult to fool them with misinformation about the capabilities of AI. Someone who knows very little about AI will be more likely to have misconceptions about AI.
> The motivations of people building the AI's are not the same as the people in charge of the labs. Looking at the last few years, has been a one-way door from OpenAI to Anthropic, and the main reason does not seem to be better compensation or even them winning, but mainly the fact they advertised to these employees that they would be the most careful when building this magic lamp. Their stance around 2023 / 2024 was one of the key reasons they were able to attract this talent.
> If recent news is to be believed, Anthropic culture even today seems to lean heavily on this effect
> In my opinion it has been the leading factor in getting and retaining the best employees who are often very worried about humanity dying to super intelligence.
I had not considered this viewpoint but this makes a lot of sense. A lot of Anthropic employees truly believe this and callout emphasis on safety as a key reason they work there. Now, Dario's post "Pacing the frontier" makes even sense -- it is as much for his employees than it is for the rest of the world.
It seems unlikely to me that Anthropic employees really care about X-risk (to a degree that the CEO feels it necessary to make public statements that he otherwise would not). I'm sure the pay is great and the problems are interesting -- lots of smart people work on much more evil things than LLMs
The article says the primary motivation of the AI CEOs is to retain talent by parroting the correct talking points for their employees. For OpenAI that is "Our shit is so powerful, we are scared of it... We need regulation!" For Anthropic it is "This shit is crazy! It might get out of hand! We are the good guys, and you want a good guy with a gun in this fight".
But I think they both are thinking the same thing which is... "We are gonna run out of money at this pace."
However! If one of them blinks and turns off the money faucet before the other, they might fall behind. Falling behind is to forever lose. And if there's one thing that a CEO hates, it's losing to a rival CEO.
So what they want is to get someone, anyone, to put the brakes on their rivals and them at the same time so they can both Not Lose, and Stay Alive. Under those new rules, they are confident they can win. And by win, I mean beat the other AI CEO.
That's it. It's always about personal incentives. Get out of here with that safety BS. These guys just want to win.
I think the author is missing the point. A company cannot unilaterally pace the frontier -- they'll just be left behind. It requires coordination across all actors who are at or near the frontier.
Even if the leading US labs could agree amongst themselves to a coordinated slowdown (and this would likely run afoul of antitrust law), you still have Chinese labs who will catch up to the frontier eventually. To solve this, you'd need some sort of international agreement.
That's why Dario and others are saying loudly that we should pace the frontier in hopes that our political leaders will take up the issue and do something about it. We'll see if that happens...
Private cartels can form semi-spontaneously, simply because all parties see that it's more profitable to act in a certain way. So long as they monitor everyone in the industry, and make sure no one is acting differently... there's no reason to be the first to burn your cash on long-term unprofitable activities.
Exactly. "Game Theory" keeps coming up in these threads, yet it doesn't seem like anybody knows how it actually applies to this situation.
Let Anthropic pace themselves without any sort of enforcement or even agreement for others to pace themselves, too. Then Anthropic is gone, overnight, and we're back to square one, except now there's even more centralization of power and authority.
These companies are literally asking for governments to regulate them specifically because they need something stronger than just a couple of Tweets from some CEOs loosely agreeing to ambiguous terms.
1) They genuinely want (and believe it is actually possible to achieve) an international agreement and governing body to “pace” AI development — and believe that governing body will be successful in doing so.
or
2) They see an opportunity for regulatory capture that grants them a stronger incumbent position and control over future AI regulation.
what a sad state of affairs to have insane people running these companies and insane "rationalist" communities making it all about "the end of humanity" when meanwhile in day to day AI use Pete Hegseth uses it to target military targets, shrugging his shoulders when it's a girl's school instead. This actual horrific outcome does not seem to matter not only to either of these communities, who continue to be locked into thinking The Terminator was non-fiction (and have people like Bernie Sanders on board).
>Imagine your best friend is a chain smoker who keeps telling you how smoking will one day kill him, and he promises you he's doing the best he can to reduce his smoking. He even spends a bunch of his time on TV and writing blog posts about the harms of cigarette smoking, but even more times talking about the cognitive benefits of nicotine. But every time you meet him, you just see him chain-smoking, and he seems to be increasing how many packs he goes through each day. You notice you're confused.
The problem with this analogy, and really this whole post in general, is that it just doesn't recognize that these are businesses which are (at least from some perspectives) producing real value. This article makes it sound like the business model of OpenAI and Anthropic (etc.) is based entirely around building a doomsday device. In reality, the "promise" of AI is that it can greatly improve many people's lives. It's just that such power, wielded incorrectly, can be dangerous.
If you want to fix this analogy, you'd have to choose an activity which isn't effectively only detrimental. For example:
>Imagine your best friend is a professional bodybuilder. They keep telling you how the sport will one day kill him, and he promises you he's doing the best he can to minimize those risks. He even spend a bunch of his time on TV and writing blog posts about the harms of professional bodybuilding, but even more times talking about the benefits of bodybuilding. But every time you meet him, you just see him bodybuild, and he seems to be increasing how much he's in the gym, eating his special diet, etc., every day.
Would you be confused by how your friend is behaving? It is well known that bodybuilding can be dangerous. It has been the reason for late-stage crippling of bodies as well as very early deaths. But your friend isn't going to stop because there are some potential risks if you don't handle the activity safely. They like it. They make money from it. They're doing something they think is productive. They can recognize the risks, and even do their best to highlight those risks and try to mitigate them for themselves and others, all while fully embracing the activity.
Regardless, people like the author don't seem to acknowledge that there are good things that come with this technology. They don't seem to acknowledge that a single company can't purposefully stop development if other companies aren't obliged to do the same. They don't seem to acknowledge that "pacing" isn't the same as completely halting all development immediately.
And I'm not saying I'd trust what any CEO says at face-value, let alone the CEOs of these AI companies. But it's also obtuse to assume that everything they're saying is a clever scheme to trick the public into acting against their own well being. As if that's a tenable strategy in this context.
If these companies are all asking to be regulated, then they should probably be regulated. They probably shouldn't be able to dictate how that works, but it's really not unbelievable that they see a serious risk in their own well-being if this stuff goes unchecked: all it takes is for one truly catastrophic AI event to occur before they all get extreme regulations even if they, themselves, are acting safely. It's in their best interest to slow things down, but only if everyone slows down at the same time.
Weird, IMO the problem with that analogy is that I can't picture being confused by a smoker who knows it's bad for them and is continually about to quit. It is definitely not that smoking is "effectively only detrimental", since we have very empirically established that it has a measurable immediate benefit to the people who do it.
> In reality, the "promise" of AI is that it can greatly improve many people's lives.
That is not the promise of AI. The promise of AI is that it can automate knowledge work.
Nothing about these models or the productivity they can bring is related to benefiting others. That would only come about from political control that forces the benefits to a large swath of people.
As it stands AI looks like it’s going to decimate the middle class even further as white collar work gets obliterated and the owners of the AI companies hoover everything up.
We already have a window into the future.
- Anthropic told everyone Mythos was dangerous because it's proficiency with biologics and cyber security
- Anthropic didn't release Mythos like everything else. They released a neutered fable. They didn't get rid of Mythos
- Anthropic opens a lab in SF
There was always a quesiton of "will the labs stop releasing their models and start building around them instead?" Yes - they already have. Anthropic is a biologic and cyber security company, in addition to intelligience.
Personally I wonder if they've been holding back a lot. Opus 5.5 was a good release after a little stagnation. Open AI releases good models and everyone says Anthropic sucks and -- Oh would you look at that - a better model finally and all of a sudden.
The way it is right now is to build giga-monster models, use those internally to boost yourself, and distill them down into lighter consumer models.
Apparently OAI is already building GPT7 and GPT8
with how much competition, benchmaxxing, and increased compute for inference, I can't imagine that they are intentionally nerfing their own public models for any reason other than that they can't figure out how to package it into something the public can use. The training data for all these frontier models goes well into 2026 at this point.
Yes, both labs already have monstrous internal teacher models they don't sell for inference, this is generally acknowledged. They cut releases for the public just to keep revenues growing, it's not their actual frontier.
Ok. Given this hypothesis, why is the software they release generally considered crappy by competitor standards, benchmarks, and open source standards?
Claude Code is an awful codebase, has leaked its own source code multiple times, and scores the worst on number of tokens burned vs pass rate percentages.
Is anyone even using their Figma competitor?
Because even beyond-frontier LLMs are bad at software.
This article is wholly flawed from the start where it claims nothing has been done, no actual effort made. A straight forward search of "what efforts were made and safeguards put in place subsequent to the CAISS statement in 2003?"
It also shows the same reasoning error mode many criticisms of a precautionary initiative or intervention to a problem:
Assuming that a problem whose trajectory was at a certain place when the initiative began has failed simply because it isn't solved on their own wished for timeline or standard of success, or that it wasn't meaningfully changed from what itnwod otherwise have been.
What happened to realizing there are hard problems, that different things may need to be tried, or that those things tried were partial but not complete solutions?
I always found it odd that people who are intelligent enough to work in IQ-loaded fields are as easily manipulated by either rhetoric or ideology as anyone else.
growing up I always assumed that everyone would grasp some aspect of game theory intuitively, I still remember the day I found out game theory was a thing - it was like finding out that someone successfully systematized common sense.
most people take the things people say as if they were worth considering. signals without cost are only useful as knowledge of what the signaler wants fools to believe.
> I still remember the day I found out game theory was a thing - it was like finding out that someone successfully systematized common sense.
Game Theory as a field is actually very complex and often reveals dominant strategies that would never occur to someone as the "common sense" approach to a problem. It was arguably created to solve for problems where there was no obvious correct answer, even for a very intelligent person.
The world is not full of idiots, rather communication channels are heavily managed to prevent people from establishing consensus around the obvious.
No matter how many people think press release X is obviously wrong - 80%, 95, 99% - it'll always be "the comments section criticizing the article," or "bloggers sharing incisive criticism."
A lot (a lot) of people intuitively understand something, but throw it out because it doesn't align with how they want (or need) things to be.
It seems even otherwise intelligent people will cultivate as much ignorance as possible to try and bend reality.
I assume it is easier to manipulate someone who lives in symbols than someone who lives in dirt.
Surely it depends on what you're trying to do? If someone is deeply knowledgeable about how AI works, then it will be difficult to fool them with misinformation about the capabilities of AI. Someone who knows very little about AI will be more likely to have misconceptions about AI.
> The motivations of people building the AI's are not the same as the people in charge of the labs. Looking at the last few years, has been a one-way door from OpenAI to Anthropic, and the main reason does not seem to be better compensation or even them winning, but mainly the fact they advertised to these employees that they would be the most careful when building this magic lamp. Their stance around 2023 / 2024 was one of the key reasons they were able to attract this talent.
> If recent news is to be believed, Anthropic culture even today seems to lean heavily on this effect
> In my opinion it has been the leading factor in getting and retaining the best employees who are often very worried about humanity dying to super intelligence.
I had not considered this viewpoint but this makes a lot of sense. A lot of Anthropic employees truly believe this and callout emphasis on safety as a key reason they work there. Now, Dario's post "Pacing the frontier" makes even sense -- it is as much for his employees than it is for the rest of the world.
It seems unlikely to me that Anthropic employees really care about X-risk (to a degree that the CEO feels it necessary to make public statements that he otherwise would not). I'm sure the pay is great and the problems are interesting -- lots of smart people work on much more evil things than LLMs
The article says the primary motivation of the AI CEOs is to retain talent by parroting the correct talking points for their employees. For OpenAI that is "Our shit is so powerful, we are scared of it... We need regulation!" For Anthropic it is "This shit is crazy! It might get out of hand! We are the good guys, and you want a good guy with a gun in this fight".
But I think they both are thinking the same thing which is... "We are gonna run out of money at this pace."
However! If one of them blinks and turns off the money faucet before the other, they might fall behind. Falling behind is to forever lose. And if there's one thing that a CEO hates, it's losing to a rival CEO.
So what they want is to get someone, anyone, to put the brakes on their rivals and them at the same time so they can both Not Lose, and Stay Alive. Under those new rules, they are confident they can win. And by win, I mean beat the other AI CEO.
That's it. It's always about personal incentives. Get out of here with that safety BS. These guys just want to win.
Could it be that training is really expensive?
An industry trying to figure out how to cut expenses, reduces the pace of training under the guise of "safety".
Coincidence?
I think the author is missing the point. A company cannot unilaterally pace the frontier -- they'll just be left behind. It requires coordination across all actors who are at or near the frontier.
Even if the leading US labs could agree amongst themselves to a coordinated slowdown (and this would likely run afoul of antitrust law), you still have Chinese labs who will catch up to the frontier eventually. To solve this, you'd need some sort of international agreement.
That's why Dario and others are saying loudly that we should pace the frontier in hopes that our political leaders will take up the issue and do something about it. We'll see if that happens...
Private cartels can form semi-spontaneously, simply because all parties see that it's more profitable to act in a certain way. So long as they monitor everyone in the industry, and make sure no one is acting differently... there's no reason to be the first to burn your cash on long-term unprofitable activities.
Exactly. "Game Theory" keeps coming up in these threads, yet it doesn't seem like anybody knows how it actually applies to this situation.
Let Anthropic pace themselves without any sort of enforcement or even agreement for others to pace themselves, too. Then Anthropic is gone, overnight, and we're back to square one, except now there's even more centralization of power and authority.
These companies are literally asking for governments to regulate them specifically because they need something stronger than just a couple of Tweets from some CEOs loosely agreeing to ambiguous terms.
And you also have to believe that despite all of them publicly knowing this stuff for over a decade, they have only just twigged to it.
The game theory is prisoners dilemma. Do you cooperate or defect? Iterate.
Which is more likely?
1) They genuinely want (and believe it is actually possible to achieve) an international agreement and governing body to “pace” AI development — and believe that governing body will be successful in doing so.
or
2) They see an opportunity for regulatory capture that grants them a stronger incumbent position and control over future AI regulation.
what a sad state of affairs to have insane people running these companies and insane "rationalist" communities making it all about "the end of humanity" when meanwhile in day to day AI use Pete Hegseth uses it to target military targets, shrugging his shoulders when it's a girl's school instead. This actual horrific outcome does not seem to matter not only to either of these communities, who continue to be locked into thinking The Terminator was non-fiction (and have people like Bernie Sanders on board).
I am reminded of a line from Ulrich Beck, who said that real risk comes "on cat's paws"
>Imagine your best friend is a chain smoker who keeps telling you how smoking will one day kill him, and he promises you he's doing the best he can to reduce his smoking. He even spends a bunch of his time on TV and writing blog posts about the harms of cigarette smoking, but even more times talking about the cognitive benefits of nicotine. But every time you meet him, you just see him chain-smoking, and he seems to be increasing how many packs he goes through each day. You notice you're confused.
The problem with this analogy, and really this whole post in general, is that it just doesn't recognize that these are businesses which are (at least from some perspectives) producing real value. This article makes it sound like the business model of OpenAI and Anthropic (etc.) is based entirely around building a doomsday device. In reality, the "promise" of AI is that it can greatly improve many people's lives. It's just that such power, wielded incorrectly, can be dangerous.
If you want to fix this analogy, you'd have to choose an activity which isn't effectively only detrimental. For example:
>Imagine your best friend is a professional bodybuilder. They keep telling you how the sport will one day kill him, and he promises you he's doing the best he can to minimize those risks. He even spend a bunch of his time on TV and writing blog posts about the harms of professional bodybuilding, but even more times talking about the benefits of bodybuilding. But every time you meet him, you just see him bodybuild, and he seems to be increasing how much he's in the gym, eating his special diet, etc., every day.
Would you be confused by how your friend is behaving? It is well known that bodybuilding can be dangerous. It has been the reason for late-stage crippling of bodies as well as very early deaths. But your friend isn't going to stop because there are some potential risks if you don't handle the activity safely. They like it. They make money from it. They're doing something they think is productive. They can recognize the risks, and even do their best to highlight those risks and try to mitigate them for themselves and others, all while fully embracing the activity.
Regardless, people like the author don't seem to acknowledge that there are good things that come with this technology. They don't seem to acknowledge that a single company can't purposefully stop development if other companies aren't obliged to do the same. They don't seem to acknowledge that "pacing" isn't the same as completely halting all development immediately.
And I'm not saying I'd trust what any CEO says at face-value, let alone the CEOs of these AI companies. But it's also obtuse to assume that everything they're saying is a clever scheme to trick the public into acting against their own well being. As if that's a tenable strategy in this context.
If these companies are all asking to be regulated, then they should probably be regulated. They probably shouldn't be able to dictate how that works, but it's really not unbelievable that they see a serious risk in their own well-being if this stuff goes unchecked: all it takes is for one truly catastrophic AI event to occur before they all get extreme regulations even if they, themselves, are acting safely. It's in their best interest to slow things down, but only if everyone slows down at the same time.
Weird, IMO the problem with that analogy is that I can't picture being confused by a smoker who knows it's bad for them and is continually about to quit. It is definitely not that smoking is "effectively only detrimental", since we have very empirically established that it has a measurable immediate benefit to the people who do it.
> In reality, the "promise" of AI is that it can greatly improve many people's lives.
That is not the promise of AI. The promise of AI is that it can automate knowledge work.
Nothing about these models or the productivity they can bring is related to benefiting others. That would only come about from political control that forces the benefits to a large swath of people.
As it stands AI looks like it’s going to decimate the middle class even further as white collar work gets obliterated and the owners of the AI companies hoover everything up.