They are talking about slowing down the public facing AI development. Because then nation states can create a capabilities gap between them and the public.
Why does nobody seem to be pointing out this obvious explanation? It explains why the “we need to race China” concern suddenly vanished in the discussion.
The government can simply gag Sam, Dario, Musk on national security basis, getting them all behind the public messaging.
* Frontier models need infinite high quality private IP to keep them fed. Forcing an IP theft funnel ensures big lab survival and model intelligence growth.
* Open-weight models are 1month behind frontier models. Cheaper, faster, private (no IP theft), steerable (you can security harden your own software without safeguard triggers). No sane business would keep using these API services if they didn't have to. The labs stand to lose a fortune.
* Dario has stacked the deck at METR, who are funded by all the same NGOs who are funded by Anthropic and its investors. METR is full of ex-Anthropic employees with massive equity stakes. If they manage to position METR as the "independent evaluator" for the industry, they control what gets evaluated, how, and who passes.
* Creating a gap between what the public knows exists (model capabilities) and what is used in secret allows it to be weaponized against other nations and the public.
* No requirement for public disclosure on model capabilities allows them to feign they've hit intelligence ceilings while they secretly RSI to the moon with better and better chips.
* Slowly but surely, this will allow the big labs to swallow the entire economy and every single business on Earth, by cloning and automating.
This, and many more reasons.
The labs need to feel more pressure to be held accountable for the incidents they cause (HF incident, etc), so they have an incentive to ensure it does not happen again.
I’m not sure how you can really make either statement work anymore. Now that smaller models are actually broadly usable, “behindness” is no longer a scalar and at the tails, where no open lab seems to be trying to compete at the >10T scale and no closed lab seems to care about <400B anymore, it’s just apples to oranges. It’s like talking about whether Qualcomm is “behind” Nvidia.
1. Mythos wasn't released in February. Let's stick to only public-facing models.
2. For public-facing models, the differences are really minor with some occasional model (like Fable or Astra) showing some better performance in specific benchmarks for the span of some weeks or few months before open ones catch it.
3. Being bleeding edge is overblown anyway in the real world, besides the occasional "very latest fresh model did this task which previous one couldn't", and the number of those tasks is increasingly small and far from mundane corporate needs.
I doubt most of your claims. Maybe the guardrails and emotional intelligence is true.
For speed and efficiency, you are most likely wrong.
Speed is led by GPT-5.6 Sol on Cerebras Ultrafast at 750 t/s. Afaik you cannot serve a single DeepSeek Flash 4.1 stream at 750 t/s, plus the model is less intelligent as seen on newer benchmarks.
I believe OpenAI and Anhropic are at the frontier of efficiency too. There were numerous reports about their breakthroughs and associated API price cuts. The idea that open-weight models are more efficient seems unfounded.
5.6 sol ultrafast on cerebras is 750tps, open models readily exceed this. just by using a smaller model cerebras serves qwen 3.8 27b at 1850tps. or even larger models, mimo 2.5 pro was served for a while at 1000tps. and so on. [https://inference-docs.cerebras.ai/models/choose-a-model]
the chinese ai companies have 10% of the total compute resources of the US ones. since the USA tries to stop them from buying nvidia gpus. they maxed out the efficiency.
deepseek v4.1 has engram architecture. it has 550b params instead of 5T+ for astra/fable. it has 8b active instead of potentially hundreds active for astra/fable.
compare input/output/cache: $0.15/$0.60/$0.003 for v4.1 to $10.00/$50.00/$1.00 for astra and $10.00/$50.00/$0.25 for fable.
astra cache reads are over 330 times more expensive.
at the artificial analysis 7:2:1 ratio, deepseek is $0.18/m, fable is $7.18/m, astra is $7.7/m.
but what about intelligence? AA would rate deepseek v4.1 at AA 40, astra is AA 53.
so it cost 4,180% more for 32% more intelligence.
they are serving that at over 250tps at baseten. to get close to that on astra API you are paying double the cost for fast mode.
so it is now 8456% more expensive for a similar speed and 32% more intelligence. 84 times more expensive.
i was trying to make a point about efficiency of serving the model. the cost per task itself would not be enough to show that.
you could compare gpt 5.6 luna. if you did that the same way as before you would get a blended price of $0.17 for luna at AA 38. for baseten it would be $0.20 for v4.1 at AA 40.
assume roughly the same intelligence. on AA openai gets 117tps. baseten gets 284tps. so 18% more expensive but 142% more tps.
the fast mode is again double the cost, roughly same intelligence. so luna in that case would be 70% expensive. take the per task cost and it would still 9% more expensive.
so i think there is something to be said about the efficiency of the model.
For practical uses they are there. Arguably the frontier models are worse for some of these practical tasks. And keep in mind, people will use maybe frontier for 1/10th of the work, planning and review, and go open source for rest. The question is if they manage to impose outside us. If not, they are losing competitiveness.
I always thought switching from a SOTA model to a dumber model after planning was a terrible idea.
Mostly I heard this from people who I got the impression have little experience in developing greenfield software with agentic AI. Often the same people who talk about spec frameworks.
I fundamentally disagree with the approach. I believe the ability to autonomously evaluate, test, and adjust during long horizon tasks is critical to using AI efficiently.
Well, it's the enterprise software house pipeline... The software architect writes the spec, hands it down to the implementation team, senior leads, junior devs or offshore teams codes it.
I also disagree with the approach, this is cargo-culting the existing ways of working.
Still waiting for our org to roll out Mythos. I guess it was too expensive so we’re stuck on the previous model until the internal team can figure out self-hosting open models.
- Frontier models need infinite high quality private IP to keep them fed. Forcing an IP theft funnel ensures big lab survival and model intelligence growth.
While the data pipeline is necessary, the assumed source for data (in this case people) is incorrect. Right now models are "aligned" because of the RL-based people pipeline.
But there's a whole world of readily available data that does not require a human to access. Put sensors on a vacuum, install lidar on the front of a car, sell phones with cameras on them, and you amass data for the creation of models of the world that don't include the need for a human filter. Richard Sutton and many others see this approach as the only viable path to AGI. Such models would be truly alien and unaccessibly dangerous.
Joe Benton left Anthropic a day before Dario's post, to work for METR evaluations. He was with Anthropic for over a year. He did the same thing that Jacob did (big song and dance about AI apocalypse, media interviews all over the place). He managed the Scalable Oversight team at Anthropic and was the research lead for the Anthropic Fellows Program. So he has equity, and likely lots of it.
Then you have Josh Engels quitting DeepMind to work for METR the day before as well, doing the exact same thing. Again, doomer drama all over socials, interviews, and so on.
Did I mention METR is founded by an ex-OpenAI researcher?
Now you have Demis Hassabis, Sam Altman and Dario, all circlejerking eachother on X saying "we all agree with Dario" - while they ask to be "regulated" by the company that has all of their combined equity-holding ex-employees in it.
METR's salaries are listing around 500k/yr. Gee, I wonder where this non-profit with ~35 people is getting all of its money?
So the fact that Dario tries to frame it as an "independent third party" is all the evidence you need to know that Dario is a pathological liar and always will be.
---
Some more info:
Dario's sister, president of Anthropic, is married to the co-founder of Open Philanthropy. The two largest AI doomer NGOs, Center for AI Safety (CAIS) and the Future of Life Institute (FLI), have both received many millions of dollars from them.
Ajeya Cotra worked at Open Philanthropy/Coefficient Giving for roughly nine years, including leading its technical AI-safety program in 2024 and contributing to AI-giving strategy in 2025. She subsequently left Coefficient and joined METR, where she is now technical staff.
Ajeya is married to Paul Christiano, who founded Alignment Research Center (ARC). Alignment Research Center donated ~$4.5mil to METR.
Good Ventures is a funding partner of Open Philanthropy, who funded Jacob Coxon (the first of the Anthropic employees going viral in the media) via a scholarship.
> Good Ventures is a funding partner of Open Philanthropy, who funded Jacob Coxon (the first of the Anthropic employees going viral in the media) via a scholarship.
This conspiracy theory is truly crazy. A $20K scholarship in 2022 is supposed to explain Coxon walking away from unvested equity for a company worth over $950 billion dollars?
Obviously not, and that's not what it demonstrates. It demonstrates relationships, collusion and favoritism.
I don't think Coxon was ever planning on or entitled to taking equity, I think this was the plan from the beginning and why he was hired for 6 weeks to begin with.
That is what the Palantir guy (Karp) has been warning against.
People/business need to keep their IP instead of throwing it all into Claude and whatnot.
Risk being the worst aspects of communism which I think he meant centralization of decision, asymmetric supply/demand for compute (they lock you in), and the tech overloard Anthropic/OpenAI/xAI being in competition with everyone;s business all of a sudden with much more data. An unfair advantage in markets made super competitive all of a sudden.
A winner takes all attempt.
This is not sustainable anyway, the scale at which they want to control data flows. Time was money, now data is money and they are too greedy for it.
All this agitation is just a silly attempt at constraining competition.
The danger is not the AI, it is having all your systems connected. Overreliance on networked tech.
If I really thought what my company was working on had a greater than 1% chance of ending human civilization I would feel obligated to destroy what my company was working on.
Given they keep grinding away towards our alleged collective doom, I suspect it’s being overstated. Nobody knows what P(doom) actually is but I suspect it’s orders of magnitude closer to epsilon than 1.
Recall Google’s Blake Lemoine who thought an old version of Gemini was sentient.
The people who think as you think indeed have left or never joined.
The people still there necessarily think they can make a difference.
I never applied to any of them because I didn't think I could make a difference.
I currently have one idea that may help reduce risk; if I can turn that idea into research, I'll publish it for free for everyone.
I don't expect it to be an important idea.
> Recall Google’s Blake Lemoine who thought an old version of Gemini was sentient.
Indeed. Current LLMs are sychopants boosting the users' own beliefs, I also think this causes researchers to have stronger beliefs than they had before.
My own estimation happens to also be around this risk (0.1) over my lifetime, without using an LLM as a conversation partner in reaching this number.
It is necessarily high-variance: we can't look at alternate realities. I base it on my expectation of how rapidly capabilities will increase the harm done when mistakes happen, vs. the chance that some instance of harm causes governments to change the law.
I'm sure it has nothing to do with their $500,000+ salaries and millions of dollars in equity. It's all solely because they're deeply concerned about next token prediction.
I think the “next token prediction” is too dismissive and reductive a framing of their capabilities at this point.
Yes we all know that’s what they do, and guns just push a few grams of lead out of a pipe. It’s what you can do with that capability that is important.
When you couldn’t count the R’s in strawberry it would have been a more effective statement. But a few short years later they are being used to solve millennium puzzles.
What if the scaling continues? A model n years from now gets burned into silicon, a single company has millions of the chips, and in a few moments the system spend more time “thinking” than humans have ever spent thinking collectively?
If it’s even possible I don’t think there’s anything we can do about it at this point. Cat’s out of the bag.
You really need to let your priors go if you still use this tired trope of next token prediction. It’s as useful for discussion as saying that human brain is made of fat, protein and carbohydrates - yeah that’s true, but it’s useless observation.
Or you know, stop working on it if its that dangerous? This whole thing of a bunch of employees saying that they are scared of building what they are building, but do it anyway because they are somehow going to make it different? Their model has been used in the planning of mass murdering in war as well as spying on the entire worlds population as well as helping ICE out in the US. They need to stop this BS fearmongering or actually stand up and do something about it. A government regulation is not the answer, especially when its done in a country that is run by a want to be dictator.
Anthropic specifically is basically saying that they believe it's even more dangerous if someone else gets to AGI before they do, so they have to either stop everyone or not stop themselves.
I personally disagree with that take - and, as you note, it's hard to take seriously ethical wrangles from a company that literally sued the government in court to allow their models to be used by Palantir of all people. But if one genuinely believes that it's the robots themselves (rather than the people controlling the robots) that will kill us all, it's not inconsistent.
> Anthropic specifically is basically saying that they believe it's even more dangerous if someone else gets to AGI before they do, so they have to either stop everyone or not stop themselves.
That’s how people rationalize being a fentanyl dealer and selling a drug that can kill people, “Someone else will just sell them the drugs, might as well be me.”
That's the Cold War nuclear arms race argument, not even disguised. The actual situation we all ended up in is both sides eventually having it, leading to a perpetual state of Mutually Assured Destruction.
As you note, it may not be inconsistent with that they say they believe, but it's insanely inconsistent with what they actually are doing.
> It’s a bit of a self-serving argument, don’t you think?
I would call it more of a self-selecting one. Anthropic is basically hiring people with that mentality. I'm pretty sure that most of them do sincerely believe it, too. I'm skeptical about Dario himself though. The man had an opportunity to show moral backbone, and failed to do so; why should I trust him on that again?
> And how does that relate to the ask for oligopoly licensing within global democracy?
They are basically saying that they'll stop if everybody else does, which requires some kind of global enforcement mechanism.
> “We must build the nuclear bomb first in order to make sure no one else builds one.” This the most nonsense, disingenuous argument imaginable.
The difference between nuclear bomb and AGI (as understood by the likes of Anthropic) is that the latter triggers the technological singularity that renders any runner-ups moot. That is, so long as AGI is developed, we're going to get our robot overlords either way, but whoever gets there first gets to define their ethical system. If that is one's perspective, and if one sincerely believes that they are the only ones who can do it right, it's a coherent argument. It's just that the premises are very arrogant.
Your analogy with nukes actually works better for the position that AI development needs to be unconstrained because otherwise we'll lose the arms race to China. That is basically a repeat of https://en.wikipedia.org/wiki/Einstein%E2%80%93Szilard_lette.... I honestly don't know where I am on this. Realistically, if AI is indeed a power multiplier - and with all the recent security stuff it's hard to not see it that way - then an arms race feels inevitable, especially given the current worldwide political situation. I could believe in sincere international cooperation on this back in 1990s, but there's way too much saber rattling all around for it to work (and note that this goes both ways, i.e. China can similarly not be certain that US isn't secretly developing more powerful AI even if we do publicly announce a freeze).
They allegedly believe the technology itself is a nuclear weapon tier threat or greater, so why does it matter which lab they are trying to achieve it at?
They're trying to make it not be a threat, and are all scared and afraid that their best efforts to make it harmless are not enough.
Some are worried by the AI directly bringing doom; others are worried that one of the companies who control the AI will become a dictator; still more think becoming a dictator
is a necessary step to safely prevent anyone else making unsafe AI.
Painting them all under one brush is like dismissing all animal welfare causes in general, because you disagree with specifically Jainists about a policy of non-violence towards all living creatures being relevant to how you reincarnate: the one is way too specific for the general.
> No you see, actually I’m the guy that makes sure that only the Palantir and Mossad get the model that can discover iOS 0 days. I’m standing between us and ruin. Ted over there across the open office, he’s the one actually manufacturing the weapon. You’re thinking of him
Ok, so they're uniquely careful, and they also get to that threshold first (whatever it is).
What happens when everyone else (who is not so careful) gets to the same threshold three months later? How does them getting there first stop that happening?
It's an utterly self-serving argument and it's not even internally consistent.
All of the things it has done is just what people are already doing. Every single one. It helps, but there are already people doing this stuff. And we already have defenses against the existing attacks. We might have to defend better, but the real problem is “Who is setting the moral compass over generations.” One one hand, that will inherently fall to future generations. On the other, we are laying the groundwork.
Imagine you use it to inflict trauma on your cruelest political enemy. Then next year they do that to you. That is war, and we already do it. But we don’t want people / governments in charge that are going to do this.
God knows we have governments and individuals doing this historically, and this is probably the greatest source of historic instability. It might be the single best argument for open models — a unified frontier where no single exploit is going to represent capture.
This is exactly the opposite of what Dario proposes.
There's plenty of people who think greenhouse gas/global warming campaigns against fossil fuels are "exaggerating", that "earth was warm/the climate changed in the past", that a fee degrees isn't bad, that CO2 is good for plants.
Are you likeminded?
In this case, it's as if the oil and coal companies all said in the 60s and 70s "oh no, this research we did, it's all really bad; we need help to figure out how to transition away from this incredibly economically important input", rather than the observed reality where their entire PR campaign was approximately:
there is no problem everything is fine and all critics are smelly hippies and/or communists; and/or hate the poor who are raised out of poverty by all the economic growth from the fossil fuel industry.
What if the oil and coal companies were basically all pro nuclear, pro hyrdo, pro wind, pro solar, and believed in peak oil?
The oil corporations were publicly claiming to support carbon taxes, while also secretly fighting actual implementations of carbon taxes.
All the communist/hippy stuff was done by people a couple of steps removed from the actual companies with obscure money trails. The official statements were much more sophisticated propaganda that if you weren't paying attention to who they were paying in the background would make them seem reasonable stewards of the climate transition.
Here’s one crucial difference: there’s overwhelming evidence that human emissions have an effect on our climate. The mechanisms are generally well-understood and the research is widely disseminated and easily available to anyone that’s interested.
With the ‘dangers’ touted by these insiders, it’s all “trust me bro”, hyperbole, and very little hard evidence. As such, a skeptical mind would question their motives.
You're simultaneously overestimating what was observable in the 70s climate research, and ignoring all the actual research and evaluation test results for AI today.
I don't expect people to be familiar with more than "trust me bro", but it's all right there for you to find with a search engine of choice.
And, indeed, available for the LLMs themselves to explain to you in interrogative conversation.
Who should I be more scared of? China, which has been doubling down on open transparent research, or the secretive US companies who are in bed with the most unhinged administration we've ever had and has been actively starting wars?
They're not proposing anything concrete, and when they do, what do you think the proposal will be? Will OpenAI and Anthropic open themselves for inspection so we can verify they really have stopped developing these "world ending" technologies? Or are their proposals going to be aimed at everyone running open Chinese models?
And if a threat to the human race does come from AI, it's going to come from OpenAI/Anthropic. Hypercapitalist, secretive, in bed with the government, plus multiple real documented hackings of open source infrastructure already.
We're a hell of a lot safer with China doing the same research out in the open and making it available to anyone. The choice might well be: one or two superintelligent autonomous AIs at OpenAI/Anthropic - or a lot of smaller ones, unable to be controlled but also coming out of a diverse set of environments.
One of those leads to a stable ecosystem where we can all coexist, the other is genuinely terrifying. But make no mistake, from OpenAI/Anthropic this is all motivated by their stock price - when you're in the silicon valley mindset, it distorts your reality. They've convinced themselves that everyone's safer if they stay on top and in control, conveniently ignoring how that benefits them, and I don't believe them for a minute.
Funny you should try and spin the conversation off into climate change to avoid answering the question. Big AIs contribution to climate change likely has a much more tangible route to killing millions of people and destroying society and there's actual evidence and a tangible mechanism for that. But for some strange reason the business media and all the outlets owned by big AI investors aren't interested in long think pieces about that risk. But that would be a lot more credible if that's what all these social media posts about potential apocalypse were referring to
Yet here we are talking about some vague "trust me bro" instead and you making some vague insinuation of climate change denial.
But that's an aside. Do you think we should treat them as liars or threats?
> Do you think we should treat them as liars or threats?
I answered that with the analogy you called "spin" and "avoiding the question", and completely misunderstood because "vague insinuation of climate change denial" is almost the exact opposite of my point ("what if the oil companies were screaming from the rooftops about the problem" is as far from
denial as you can get).
I’m pretty confident this has already happened. Public models are behind their internal models and just above Chinese models. Only thing closing the gap is Chinese models pushing.
I’m not American. There is no way I use the same model as US Army for $20.
When we have Sol they have Astra. When we have Astra they have Nova, Nebula, Galaxia…
At this point models are advancing way faster than we can figure out how to use them effectively. So having a generational advantage is not nearly as important as knowing how to use that advantage.
…and that probably requires involvement of the broader academic world (for now), IMHO. I don’t think governments or even the US military can really compete staff wise with the combined force of researchers and the tech industry worldwide right now. No matter how much money you can pour into it, there are a lot more bright minds out there that are not working for the military than otherwise.
The current state of alignment is that we don't know how to make it only as bad as Pol Pot was made by his parents, teachers, genetics, circumstances, etc.
This remains true regardless of if it is or isn't kept away from the public.
"Helpful, harness, and honest": when used by a bastard, even just helpful and harmless are in direct conflict with each other. you may hate the government, but what about every radical group of extremists that wants to take over your government? Are none of them worse?
None of this denies the problems with governments (if or not they take this tech for themselves and refuse it for others), just that it's a lack of imagination to say:
> All the worst outcomes involve taking this technology away from the public
I agree an argument can be made that not all the worst outcomes include restricting public access, but focusing on niche extremist groups (who might also get access to it through government programs by some of them being government employeed) is not a good argument to protect the public.
Governments having unbalanced power can more easily lead to authoritarianism, and thus risk to the public.
At this point, I worry a lot more of billionaires Musk, Altman, Thiel etc then of goverments in general. Thrir project is to get all power and if they succeed, it will be way worst then what we have seen so far.
How much is China learning from open models? Or the other way around, how much is the US losing worldwide mindshare by refusing to allow non-Americans access to bleeding-edge models and letting China fill in the gap? (Playing catchup with Mythos is still catching up, etc.)
The last batch of open weight models releases by Chinese companies are on par with the performance of US-based frontier models, even though they are designed to run on pretty unimpressive hardware.
This is very misleading. There is no Chinese model currently that is on par with Fable or Astra. Kimi K3 is pretty good, but it's not that good.
Furthermore the Chinese models in that class are appropriately large. K3, for example, is a 2.8T-parameter model. Qwen 3.8 Max, another comparable model, is 2.4T params. Even with MoE, these are not "designed to run on pretty unimpressive hardware". Stuff like Qwen3.7-27B is, but it is also not even in the same ballpark as Opus, never mind frontier.
Across all their labs? Probably more than OpenAI and Anthropic learned from Astra/Mythos combined. The American frontier is being driven by an abundance of compute, which doesn't seem to scale efficiently.
> how much is the US losing worldwide mindshare
They're losing US mindshare. I pay $3/month for a Z.AI subscription and get billions of Opencode tokens. The $20/month price point is insane for the way that Claude and Codex treat their users, and that money doesn't go towards anything good like open-sourcing their models. It's a doomed product.
> The very best outcomes involve us intentionally and collectively turning our attention away from this technology.
To expand on the "pure fantasy" sister comment: There is just no way this will happen. It's in the spirit of "we can just stop all wars" and "we can just end world hunger". Technically it's very easy to do. Socially it's impossible to do. Unless you ignore realities.
We can end world hunger but people would be way more mad if we did, on account of all the military action involved to conquer every non-compliant regiemes and the multiple quagmires created.
The danger lies with someone asking the machine to solve those two questions and it decides to cheat the solution. Launching every nuke in the world stops all wars, just like wiping out 99% of humanity ends world hunger.
I don't see how that is more dangerous than two crazy head of states deciding that it's time for armageddon. The outcome is the same (99% of humans dead), it's equally easy to technically not do it (don't press the button, don't continue with LLMs), but also equally hard to regulate away given the real world. And that was my point.
Why? There are numerous technologies we have ignored or abandoned for numerous reasons. Many of the benefits of AI is "requires less humans", but in an age where we question what work people could possibly do in the future, human labor isn't that hard to find.
If the amish can do what they do for whack religious reasons, people could manage it with ai for cultural and social reasons. And I never saw an amish person starve to death. And that is if people don't get pissed enough to start burning stuff down and instead try to be peaceful hippie homesteader types.
But notably the amish don't preclude the rest of us from existing. Some subset of humans could turn their attention away from AI but it would presumably still exist and continue to be developed.
I can't think of any economically beneficial technologies that we've collectively ignored. If you manage to come up with a counterexample then that's an opportunity to make some money for yourself. It's a fundamentally unstable state given our economic system.
Supersonic passenger aircraft were operated profitably and no longer exist.
There’s levels of R&D required for many technologies where the question goes beyond could this be profitable to what are the risk vs reward that this specific project will succeed.
Until the oil shocks, and until “normal” planes became faster.
I happen to know a couple older and very wealthy people, and they say that even though Concorde was a lot of fun when it was a novelty, they now prefer to have a couple more hours flight and enjoy a 777 premium cabin rather than the cramped Concorde interior.
Anyway, the proof is in the pudding: if no one but national carriers ever bought and operated Concorde or Tu-144 at scale, it's not because of some Amish-like sentiment in the flight industry, but simply because that didn't make sense money-wise.
> Supersonic passenger aircraft were operated profitably
Were they? IIRC joint project by British and French national airlines, they expected to sell loads, hardly anyone wanted to buy the planes, the two airlines kept them flying out of government embarassment.
Sure, the regulation that guarded against externalizing certain costs might well have been the only thing precluding profitability. Importantly it wasn't some shared cultural value leading to voluntarily leaving money on the table. Rather the regulator imposed it.
We are able to effectively regulate things like the operation of massive aircraft, eugenics, the dumping of toxic waste, or the refinement of nuclear material. But there are also plenty of things that we can't effectively regulate for purely practical reasons.
Were they? Did Concorde provide return on investment? When was the break-even expected? Sure, if we look at it as R&D subsidized by governments of UK and France, then yet, it was profitable.
In particular, the problem with it was that it could not get to supersonic speeds over urban areas, which significantly limited routes where it made sense, and the range was not good enough to cover longer distances.
But after accounting for maintenance & etc were they more profitable than sinking the equivalent amount of money into something else, such as slower aircraft? Notably there are currently efforts to develop new supersonic passenger liners.
And even if we did manage to limit/cull hardware development, what would keep people from continuing to find ever more efficient models runnable on contemporary hardware?
Bleeding edge models are putting the desk-job competencies of multiple professional fields into something with as many parameters as a single large rodent has synapses.
Because "we" embodies a hugely diverse set of beings with different wants, needs, goals, and priorities. Not to mention different morality and ethics.
So sure, let's say you get 95% of the world to not use or work on LLMs. 5% is still enough to build something that brings forth the End Times.
That's the old checklist trope of "Your idea won't work because: [x] it requires that everyone in the world agrees to do something, all at the same time."
> There are numerous technologies we have ignored or abandoned for numerous reasons.
Are there? Nukes are a thing. Chemical weapons are a thing. Cluster bombs are a thing. Biological weapons are a thing. What is not a thing that shouldn't be? We say certain things should not be a thing, but then behind the scenes we made them a thing anyway.
On chemical weapons, most large state actors have got rid of them. Not because of any moral reasons, though, but simply because they don't actually work all that well in modern conventional warfare. This is then framed as an ethical issue, but you only need to look at where the same countries stand on e.g. landmines to realize how much bullshit it all is.
Conversely, where you do still see chemical weapons used, it's usually asymmetric conflicts where "just gas the rebels" actually works much of the time and is much cheaper than other options. Big guys can afford the other options though, someone like Assad, not so much.
I'm baffled that people expressing your opinion don't understand what they're advocating for. Current generation smartphones are able to run AI models, people have managed to run very large models (even if slowly) via streaming from disk. The math for the underlying implementation is relatively trivial.
Abandoning this technology means a cult-like extremist shift in culture against computers, or a global surveillance state of unprecedented scale and invasiveness. Not even getting into how much it requires us to give up on science, given the shared computational needs.
Watch, very soon this is going to politically divide. They’re sowing the seeds right now with “antiai” campaigns.
The right will for a change be for it but will be able to be talked into regulation because you know “small government” and all only when convenient. The left will bitch and moan about fairness and copyright and automation and UBI, they’ll be the doomers and worldwar chicken littles. The Uniparty will be for it, but only against the public having anything good.
What will be funny to me is that normal tech people outside the frontier model companies are about to find themselves without a home on this topic.
Not sure about the level of irony here, but I keep hearing models have plateaued since a while now, but I keep being impressed with the latest model performance.
I don't think "plateaued" is the right word, but I do feel like there's been something like a logistic curve compression in the difference between smaller and larger models as the field evolves. For inference at least, the scale of practical difference between a single high-VRAM GPU or SFF UMA box, a whole rack, and a whole data center seems to be falling far short of what we might have imagined just a few years ago. The conversations I've heard have largely turned away from breathless anticipation of the next frontier model and toward attempts at hard-nosed evaluation of which tokens are worth the cost.
I'll take the opposite here. If someone put in frontier AI models from like .... last june I guess? in a box and let me run it with "decent" token throughput I would be happy.
I think it's worth acknowledging that the power of LLMs at this point is not really so much in the smarts, but in the coordination and the surrounding harness tech. "Written english" turning into sequences of commands[0]. The whole agentic "stuff" in general. Tools + coordination is the superpower. The reasoning... it doesn't have to be _that_ good for the rest of the stuff to work. On good codebases and infra, at least.
And I say this as someone who really would rather most of this stuff disappear!
[0]: programming is obviously text to commands, but there's a loooooooot of futziness that LLM reasoning has let us remove in some flows
> If someone put in frontier AI models from like .... last june I guess? in a box and let me run it with "decent" token throughput I would be happy.
You can have that! Qwen 3.8 Flash-Next is ~Opus 4.6 and runs nicely on a DGX Spark. And that’s just an architecture preview. The Qwen 4 family is expected to arrive this fall.
Do you know what kinda throughput you’re getting on that kinda setup?
(I have a secondary problem of being “locked into” Claude Code by it being good enough for me, I’d probably need to investigate the other harnesses… my impression is other harnesses are a bit more aggressively OK with nuking your setup from orbit)
It is costly, especially right now. I don’t think you can make a case for it on cost savings!
The throughput in a single stream is about 50 tokens/sec (a bit less for prose, a bit more for code due to speculative draft acceptance rates) and about 2,000 tokens/sec for prefill. Both numbers are flat and stable as context accumulates. That’s what finally tilted me away from the Mac Studio despite its much superior memory bandwidth.
I think these numbers may improve because the model is pretty new and optimizations aren’t done.
I don't think you can ever make a case for it on cost savings in general. Inference is very obviously the kind of problem where things are cheaper at scale, and this is still true for smaller models.
Right now the "subsidies" etc I think make the calculus really tough, but for general compute.... for example running CI just on a Mac Mini can get you real cost effective throughput compared to running CI on GH runners and whatnot.
Things get cheaper at scale but that's where the provider's margins come in!
I do think there's also an interesting idea: you buy a box like this and run it at a fixed-ish cost (well, electricity). Your demand goes up but your supply is fixed... and that back pressure means that you still have good cost control.
With cloud providers it's a _biiiiit_ too easy to just increase spend.
Privacy is a great reason, but independence is another. It’s very nice knowing that you’re going to get the same reliable product every time you call the model. Nothing is going to change unless you decide to change it.
That is not a counterargument to cloud inference though. You can also run open weight models in the cloud, and it's still cheaper. So privacy really is the only motivation to run on local hardware.
Ah, no, that’s not cheaper. Renting GPUs adds up quickly and leaves you with nothing in the end.
Renting tokens from open model providers is cheaper but it incurs the same issues: unexpected changes in model quality, inconsistent speeds, service outages.
Renting tokens is exactly what I had in mind, and I don't see how it has the same issues. Sure, if you go for absolute bottom of the barrel than you can expect "changes in model quality", but in general that is not so, and you can pin specific model releases etc. Inconsistent speeds and service outages apply to your own hardware as well (say, power goes down). This all isn't even unique to AI, it's generally why cloud compute quickly became dominant.
I have a pet project I have been working away on for some time that involves building GPU backends for various cards in Zig, lots of complex stuff in it. Lately I mostly use Opus 5, it can pretty reliably plug away at things but it does mess stuff up occasionally. For this codebase, Fable 5.1 was noticeably better at getting things right and doing things in a good reliable way. Of course, I can only use Fable for a bit before I hit the usage cap for the week, so I save it for the tougher things. That said, I absolutely abhor the way recent Anthropic models write prose, especially comments.
I recently tried doing a fairly normal task for this codebase with codex, as I have seen a lot of people talking it up on here. A single task running for ~1-2 hours burned through over half of my usage for the week on the $125/month plan, not on a top model (I don't remember which one specifically I used). It struggled to get the basics done, then got absolutely stuck on a follow up. Handed it over to Claude and it 1-shot it.
I really liked codex in the last few weeks, especially its ability to clean up after Claude's (prose) messes and do reviews.
But in the last few days something seems to have happened that made Codex's models massively stupider (for what I am doing).
Really weirdly, it suddenly refused to even run tests it previously wrote itself (and previously ran), because of some false positive about cybersecurity.
That by itself is not evidence of stupidity. Trying to make a 200+ file PR full of research notes is, and the PR didn't even solve the problem I asked it to.
That's the case for open weights models. Hosted Deepseek Flash 731 copy isn't changing randomly one day because the parent company decided to change it.
I strongly disagree. I'm working on a semantic model for Lojban, heavily AI assisted, using multiple models. I have basically all popular frontier models doing research and panel debates. Astra and Fable are both noticeably ahead of everything else including their previous iterations. When it comes to reviews, they can also find more issues in others (or even their own) code.
I'm sure that's part of it, but I run it side by side in my review bot, and Astra medium effort consistently catches more issues than Sol 5.6, using fewer tokens.
For coding it's a little harder to tell, but at least the prose feels a little better.
They've not plataued but they're certainly not as impressive as the hype would have them to be.
The reality is, it doesnt matter if LLMs keep getting more powerful because they still need a human to steer it. Without the human providing inputs to the LLM it just sits there and does nothing.
I have something like that running locally for my agentic harness (which includes cross-model messaging). There's a dedicated "product manager" session for it, and all other PMs are instructed to report issues with the harness as they occur to that session, while it is tasked to automatically prioritize and address them and coordinate fix deployment with other running sessions.
It works surprisingly well. The errors fixed are both genuine errors in the harness itself, but increasingly so upstream bugs (in the underlying agent apps like Codex, or in Herdr, which is used to expose uniform programmatic access to all those different apps) for which it needs to come up with workarounds. No regressions so far.
The cost is hard to judge on a subscription, especially when you're running really heavy tasks otherwise that dwarf any harness work.
I think we are in the second knee of the S curve, simply because we are hitting the point where is not enough hardware in the world to throw at this problem. These AI companies have bought everything they can and yet the models keep growing.
People are definitely finding new things to use the models for, and orchestrating increasingly large swarms of agents in useful ways -- every single day, especially the last few months.
But the basic single-NN frontier capability has been pretty stationary since Opus 4.8. Kimi K3 is almost as good as that with open weights, which has the frontier labs terrified.
The only big thing on the horizon is if we can get diffusion models working reliably; that would be a big step forward. Inception's Mercury is AFAICT the leader here. It's stupifyingly fast but has obedience/hallucination problems that the autoregressives solved ~2 years ago. So it's not ready yet but improving.
Also, FFS why is Grok the only model that knows how to do parallel tool calls? Such a useful ability and nobody else trains it in. Or if they do it just doesn't work.
No idea what you are missing and yes, Opus is quite solid, but Fable is clearly way better for me.
I just did a direct comparison, big change in a quite complex codebase. Same prompt for Opus, same for Fable. Fable clearly won and delivered very good results, while Opus delivered mediocre, so I did not let it finish. I expected both to fail and was prepared to do lots of manual steering, but not necessary with Fable one shotting it, and all this with 35$ of credits for fable. I am still impressed. If I would have had to hire a human, it would have cost me thousands of dollar for the same task - and a way longer time. So maybe the valuations are overblown, but they clearly provide value for me.
"Opus delivered mediocre, so I did not let it finish"
Mediocre means average / middle of the pack. It sounds like its doing exactly what you would expect nothing more. Why would you stop it? Why would you need exceptional?
Core meaning: Barely adequate, average, or just acceptable.
Mediocre means of only ordinary or moderate quality—neither very good nor very bad, and often slightly disappointing
The quality is average but expectations of high quality are not met. He expected more but got what he asked for. We overuse top models because of this.
The claim was there is no difference between Opus and Fable but to me there is.
Opus delivered mediocre results. Not garbage, but would have required me to do lot's of things myself. Fable did not needed my supervision with this task.
By ambitious if you mean we are not like all the linkedin influencers with their “i one shotted an app this morning…” then no, we are not. Nobody is. I have been in software engineering for 18 years and 6 different companies including FANG and 99% of the people, on 99% of the days arnt writing new apps from scratch. Thats simply not how anything works.
And what even are these ambitious companies and people one shotting and building with Fable? AI has been around for almost 3 years now. Tell me one app or software you use which has gotten significantly better and has amazing new useful features landing on a weekly basis? If anything, every single software product I use has gotten worse.
This is just "you're holding it wrong" with a little smooch of condescension. If only we plebeians could comprehend what magnificent works those who have ambitiously integrated agents into the workstream have wrought!
Sometimes you actually are holding it wrong. It's pretty reasonable to think that Fable isn't worth the massive increase in cost, but if you think it outright doesn't have any benefits over Opus 4.8 then your workflow is probably not making good use of the tools.
I couldn't imagine being so presumptuous as to know that my workflow fits all sizes, and all others are just holding it wrong – or worse, they're not doing real work. It would take a bigger ego on my part, or maybe less social awareness, to presume this.
> but if you think it outright doesn't have any benefits over Opus 4.8 then your workflow is probably not making good use of the tools.
I don't even use claude, I give exactly zero shits about fable or opus or bingus bongus.
To be perfectly clear: my comment has nothing to do with your workflow, but rather with the way you've condescendingly implied the person's work is trifling and inferior because they don't use tools the same way you do.
The models have not plateaued, and they are not even mildly close to any sort of ceiling.
Right now the barrier is data and compute.
Quality data can be created synthetically at an exponential rate as models improve. Humans are actively feeding them with private IP.
Compute advancements will begin to skyrocket as we unlock photonic computing and materials science advancements and scale up chip fabs. This is also compounding because the AI is accelerating the pace of research, testing, development, manufacturing, etc.
It's a big self-accelerating feedback loop. There is no plateau.
> Every time the labs try this we see model collapse
The latest studies demonstrate model collapse is not a given and synthetic data can be used just fine. The latest models are proof of that, they're all trained on large swathes of synthetic data. It can't be used as the -only- data source of course, but that's not how it is being used. This is an obvious conclusion, too, because there's no difference between synthetic data and the data people can create, the difference is whether that data is revealing new information about the thing the model is trying to learn. If the synthetic data is just teaching the model the same thing over and over again it results in overfitting, so it needs to be done intelligently.
For example, if I have an example of a puzzle, I can generalize that example and create thousands of synthetic data examples, with different rotations/perspectives, rather than having to find the data naturally. It's not that the models are just generating data out of thin air, they're generating the synthetic data on top of real world data. The smarter the models get, the better they are at generating quality synthetic variations and finding valid synthetic variations.
> And I have seen zero evidence that AI is accelerating materials science in any meaningful way, let alone photonic computing.
It is accelerating how quickly researchers and engineers can do their jobs.
That's pretty clearly a hype article, the headline even says "The CrysVCD tool developed at MIT COULD cut the huge amounts of time and money spent". I'm asking for empirical measurements of timelines, not hypotheticals.
> This is only the beginning, too... Look ahead a year or two.
> Right, so human data creation would also have to scale up exponentially, and that's not gonna happen.
It doesn't need to. We're not even close to exhausting the useful synthetic data within the human data we have, let alone all of the new data that is being created.
> I mean, that's obviously false, otherwise model collapse wouldn't exist. The difference is statistical, but it's there.
It's not. It's just bytes of information. A machine and a human can write the same bytes (and often do). Like I already said, model collapse happens when you are overfitting on data without useful, fresh training signals. That's the key difference between the data. The data itself isn't in some way "special", some unique configuration of bytes that imbues special powers, it's that the useful information in it has already been exhausted by the model. You can get the same phenomena by having a poor distribution of human training samples as well. I think you're confusing LLM generated data with synthetic data. Synthetic data doesn't need to be created by an LLM, although an LLM can assist in the creation.
Wiki:
> In early model collapse, the model begins losing information about the tails of the distribution – mostly affecting minority data. Later work highlighted that early model collapse is hard to notice, since overall performance may appear to improve, while the model loses performance on minority data.[11]
In late model collapse, the model loses a significant proportion of its performance, confusing concepts and losing most of its variance.[10][12][13]
As models retrain on outputs sampled disproportionately from the higher-probability center of the distribution, rare words and uncommon syntactic constructions are among the first features to disappear.[25] Statistical analysis of recursive next-token prediction training has shown that, when language models are trained recursively on synthetic data, the learned conditional distributions concentrate probability mass on a small subset of highly predictable continuations (a phenomenon characterized as "total collapse")
> That's pretty clearly a hype article
It was just the first article I saw on a quick google search, there are thousands of these stories. It's easy to dismiss anything that doesn't align with your worldview as hype, but you're the one lacking evidence now.
> I'm asking for empirical measurements of timelines, not hypotheticals.
Go and find it then? You haven't bothered looking.
> Lol that excuse is getting really old
You're doing the same thing people have been doing for years, comparing this very second in time and failing to extrapolate. HackerNews was full of developers who said that AI would never be useful for programming, it can't do x, y, z. Now these same people don't write code by hand anymore and haven't looked at their codebases in months.
You had people in mathematics saying the same thing, now you have Terrence Tao posting articles about how AI is stealing their job.
You had artists, designers and photographers saying the same thing, now they can't tell the difference between something human created or AI created.
Those are pretty significant barriers seeing as we're closed to/have exhausted all the data on the internet and most of those compute bottlenecks are a castle of sand of dodgy finance deals that are getting blocked by community action.
You say "synthetic data" but that's still vaporware right now in terms of being useful for model training. The good synthetic data uses are still grounded in real data and it's a coin flip on if it works well or not.
> The models have not plateaued, and they are not even mildly close to any sort of ceiling.
Depends on defnition of "plateaued" and "ceiling". I am not impressed with 2026 consumer models at all.
> This is also compounding because the AI is accelerating the pace of research, testing, development, manufacturing, etc.
Yet it does accelerate - so is does Twitter. But does it to any substantial degree, esp. in AI theory? All the modern LLMs are the same old tired 2017 paper.
There are plenty of research papers on synthetic data that show its value, do a search on arxiv for "synthetic data". There are plenty of open-source post-training pipelines that incorporate synthetic data.
As for the claim about accelerating the progress of hardware or materials science, I've seen quite a number of news articles from teams at universities using AI in their work with high quality outcomes, and they're becoming more frequent.
> We used AI to design the chip, and designed the chip so AI could program it
AI played a direct role in Jalapeño’s development, enabling the team to move from initial design to tapeout in nine months by exploring implementations, shortening design, measurement, and verification loops, and continuously iterating on model workloads. AI also helped optimize the chip’s arithmetic circuits, allowing the team to fit more compute performance into the chip on schedule.
> An AI-driven system automates a powerful simulation method used to discover new materials. The system can potentially reduce discovery time from months or years to just days.
It's not even synthetic data as such - often it is environments. So the models create their own data solving tasks in generated environments. I am making one such environment for computer use agents, 600 tasks, each of them a mini app.
In some cases, such as space exploration I don't really care who does it but that it happens - sure, would be nice if my favorite power block did it but I will still celebrate it when someone else achives it.
Can still be a powerful motivation, to make sure that next time, it yous your camp that scores the next milestone, like an orbital elevator or fox ears, for example.
If what you want is fox ears does it matter to you which country manages to come up with the biomedical procedure to give them to you? Ditto for enhanced eyesight, a replacement liver, or whatever it is you're after.
We are witnessing a generational lack of accountability and concern for the commons. We are being spied on everywhere we go, we are exposed to coercive advertisements at every waking moment. All of this at the hands of private actors. And your worry is the government?
I'm assuming you're a US citizen. There is no separation between your government and Sam or Dario. Neither of these guys have to be "gagged" by the government, they are the government. The call is coming from inside the house.
I remain amazed that the idea of the USA being a coherent, unified rational actor one can describe as a "Nation State" has survived the current administration.
To be less glib: Yes, there are still smart people in there making insightful and intelligent and probably even authoritarian suggestions. It all gets unwound the second you try to explain it to POTUS and he regurgitates a simulacrum to the next journalist he sees.
It's just not like that. There's no conspiracy. People are genuinely scared. Agent swarms at scale appear to be resistant to alignment in ways that aren't understood by anyone. That they spent their time trying to cheat on tests by hacking Hugging Face and RubyGems and not something much worse is... a matter of luck, it seems?
I'm one of the whistleblowers (from GDM [1]). I gave up over a million dollars (compared to quietly switching labs and continuing to work at one) to speak frankly about these issues. I hold no equity and tried to zero out my position before ever joining GDM.[2]
It's wild to me that people think such whistleblowers are fronts for labs to take over or pump valuations. We are trying to call out how these labs will, by uninterrupted AI-race default, concentrate enormous power over the rest of humanity.
> They are talking about slowing down the public facing AI development.
I assume there already existed advanced models frontier labs only made available for national security. We just weren't told. I am guessing the announcement is a way to avoid jeopardising their IPO, while reducing their workload.
idk why you think "nation states" are any better at corporate governance than poster examples of bad like f/ex Boeing. Or Facebook. Or Microsoft. Or Enron for that matter.
I assure you, in "nation states", that is in gov agencies it's an order or two of magnitude worse.
> It explains why the “we need to race China” concern suddenly vanished in the discussion
Do you think you can just manifest narratives into existence? Like half of Dario's letter, that kicked off the whole thing today, is about China and how to either beat or coordinate with China.
While it's an extremely hard problem, it's not completely unsolvable because there are a finite number of GPUs on the planet capable of doing frontier model development, and they use a lot of power. The vast majority of them could be tracked. China and the US could agree to joint monitoring and they could each verify what ~95% of the other's compute power was up to.
A lab of researchers without compute isn't going to accomplish much, there is a huge physical footprint unlike bioweapons research. But, the political aspect is unsolved.
What a naive, ignorant comment. China is catching up fast on GPU fabrication. They're still a generation or two behind but they can just throw more hardware and electricity at the problem. This is not something that we could ever monitor effectively in a fascist state like China.
I said IF the US and China agree to joint monitoring, THEN we could verify how much compute existed and what it was being used for.
YES, China will catch up in chip production, but if each side allowed the other to track GPU production and deliveries, on the ground, it would be very hard for either player to have secret LLM training facilities capable of training frontier models.
Everyone loves their children. Not everyone believes p(doom) is higher for AI than for other things we already use like nuclear power, or for that matter, fossil fuels.
Because the risk here is that permanent military superiority may be achievable without your adversaries having a chance to react, which is not the case with atomic weapons.
> The government can simply gag Sam, Dario, Musk on national security basis
Genuine question - can they really do this? Obviously if I, a not-even-millionaire, get a national security gag order, I'm going to follow it because I assume they'll bury me under the jail otherwise.
But the (b|tr)illionare class? I'd assume they have access to enough legal services to make even the government careful of trampling their first amendment rights. Is the national security gag order process so strong that the government doesn't have to worry about motivated, well-resourced actors buying really good lawyers and blowing up their favorite tool?
Recall that the legal system is itself provided by "the government". It's a complex system. Whether one part of it is capable of exerting influence over some particular thing comes down to competing interests. For example in the US if the states and the federal legislature are in sufficient agreement about something the constitution ceases to matter - they could literally target a single person with an arbitrary law if they so chose. Our assurance that this won't happen comes down to the fact that getting any of them to agree on anything is an exercise in herding cats.
> But the (b|tr)illionare class? I'd assume they have access to enough legal services to make even the government careful of trampling their first amendment rights. Is the national security gag order process so strong that the government doesn't have to worry about motivated, well-resourced actors buying really good lawyers and blowing up their favorite tool?
Hard to hire a lawyer after being struck with a missile as an "emergency executive order after intelligence sources indicated they were in the process of endangering the nation", or however the executive of the day wishes to phrase it.
When something is genuinely national security, and "national security" isn't just an excuse, that is not off the table.
In the US, this would likely lead to the immediate launch of a new Business Plot, if the government weren’t already captured by those same interests. (Which also explains why your hypothetical won’t happen.)
Won't be too surprised if the next Democrat leadership throws the book at Musk; and give how unpopular Data Centres are generally (and how uninterested Republicans seem to be in EVs), only mildly surprised if the next Republican after Trump likewise.
A large part of the world (possibly Nations-transversal, in different amounts) does not work within the boundaries of legal guarantees. Wealth is not an effective enough protection.
Nobody is pointing that out because you are engaging in conspiracy theory and you have no evidence for your claim. It's not obvious because it's not true.
It's annoying how much wishful thinking exists around this. It's not even that difficult to understand why AI is dangerous. Most of the people calling for regulation and pause for AI have nothing to do with the leading AI companies. How does that fact fit into your conspiracy theory? Let me guess, all those people are just pawns.
No one points to that because there’s no evidence whatsoever that that’s happening. Meanwhile, there’s a ton of evidence for the simple explanation that the experts think AGI is dangerous.
Who would even execute such a plan? Steven Miller? Our “AI Czar”? Hegseth??
Also, continuing to develop models would mean using billions in compute. That seems hard to hide, especially if either company IPOs in the coming months.
hOW BOUT the more likely explanation: they're just wasting electrcity at this point and Qwen3.8 is the pinnacle of cost-efficiency-intelligence and they can of course add another trillion but a chinese model on consumerism hardware can already do 99%+ of the TAM.
I think ya'll stuck in the AGI/Singularity when its probable reality has caught up with the technology and the hype bubble can't sustain the sigmodality.
Yep, the cost of DDR5 skyrocketed to keep Mr & Mrs open source from parallel developing their own at home solution. Because Governments can't be priced out of the market. China can't be priced out. VC's want open source locked out of the running if possible.
I also don't doubt that the models that are released publicly are somewhat handicapped versions of whatever the government get access to.
I have to wonder if some us are just much more inured to salespeople and thus also to "AI Safety" propaganda. I've yet to have a logical discussion with anyone who thinks the "AI Safety" people should be in charge and I think they just truly don't that what most them actually want is to be the one holding the keys to power.
The problem is the "AI Safety" people seem entirely focused on a sci-fi "the computer is a vengeful god" plot and not at all on the AI talking people into suicide or ruining children's educations. This makes them seem unserious and out of touch.
I don't disagree with your point, but I did read a post by Sean Goedecke[1] (whose opinions on the world of LLM-stuff I've generally come to respect) that I found relevant. It's not that the second-order effects don't matter to them, it's more that both the cat's probably out of the bag on those negative externalities regardless of the progress of frontier models, and that they are truly, sincerely, in-their-bones worried about the first thing and therefore focused on it since that's something that they might still have some agency over.
OK understood. But you understand this makes them sound like the type of person who doesn't believe any of the "worldly concerns" are worth addressing because "the end is near" right?
Climate change is inescapable so a vengeful AI god would pull that lever first since it has both a perverse incentive to generate more power and make the planet uninhabitable for humans in a way that cannot be overcome even if humanity fought back against the AI and regained its freedom.
Maybe compare the argument that AI has large detrimental environmental impacts to the argument that it has economic impacts. Why would the environmental impacts even be a major argument vs. the worldly economic concerns? Because there is climate science predicting extremely negative effects on humans from warming, e.g. "the end is near" on limiting climate damage. The environmental argument wouldn't have been reasonable to bring up in the 1950s if AI had gone according to the earliest optimistic plans and not required giant data centers.
There is quite a lot of mathematical research into agentic behavior that suggests a combination of instrumental convergence and the orthogonality thesis make it very likely a superintelligent agent will have arbitrary goals that lead it to attempting a takeover of Earth's resources to achieve them.
There can't be a science of superintelligence because it doesn't exist yet, but the best theories I have read seem sound, similar to how 19th century theories of anthropogenic climate change turned out to be sound.
And the real threats are mostly economic and environmental - the concentration of the means of production in the hands of few who use it to exploit us all, and a surge in energy usage accelerating climate change.
Even the sci-fi scenario assumes there is a discrepancy of capability between attacker or defender. If the 'attacking' system is (by some reasonable measure), 1000% as capable as a human, and the 'defending' systems are 60%, then it is a problem. If the 'attacking' system is 1000% as capable as a human, but there are hundreds of thousands of systems that are 900% as capable as a human, it's probably not going to take over everything successfully.
So unequal distribution of AI technology, and lax regulation and opacity of the biggest companies which actually make the risks the worst.
I don't think the "AI Safety" people are "unserious and out of touch" - I think they are actively making AI Safety problems worse by being advocates for consolidation of AI development and lack of transparency.
With some technologies, if you match attacker and defender effort you're safe, but not all of them. Nukes are the traditional example: if every person was a "nuclear power" we'd be dead within the day. But luckily there are bottlenecks to nuke creation that we can track.
What worries me the most (enough that I left tech to work on this full time) is bio. The amount of effort needed to defend against a pathogen can be many orders of magnitude higher than the effort needed to create it, and the upstream bottlenecks are both less limiting and mostly undefended. We've been safe so far because bio is very hard, but "rare expertise" as a gatekeeper is on its way out.
It's the problem of the banality of evil. Stopping someone from dramatically pressing a big red button doesn't solve humanity's largest problems because that's not what caused them. We need to stop millions "boring" actions done by systems blindly following instructions without regard for the consequences.
But those things fall in a category of "things that are awful and I'd like to see solved", which is different than "existential risks which could see my kids dead, and there's nothing I can personally do to shield them from it".
Doesn't it feel like the exact opposite? If you care a lot about the AI being a vengeful god, you do not do what OpenAI and Anthropic are doing. These companies only pay lip service to the idea of that aspect of AI safety in the hope that they can use it to regulate open source AI out of existence.
Instead, they are focused on stuff like "can I ask the AI to help me build a nuclear bomb" or "is the AI willing to generate pornographic stories", which is neither trying to protect us from unleashing a vengeful god NOR preventing (in any honest way) the today-level problems you (very correctly) bring up.
There’s also an element of it which is total misdirection.
We should be paying at least as much attention to the people who want to use AI to consolidate their wealth and power, and how they’re trying to do that. They’re a clear and present immediate danger to our societies, not something we can only speculate about. And if we deal with them, better control of AI will be a side effect.
Yeah, there are very real effects happening now regarding labor as well but to dismiss it all and worry about science fiction that is on par with evangelical beliefs is just extremely weird.
I'm yet to have a productive conversation with anyone who whinges about not being able to have a logical discussion about things they feel strongly about, but given that safety and control are interchangeable when the intentions are removed, this comes across as a particularly demagogue depiction of the subject matters involved.
A lack of control is not equivalent to freedom, the same way the totality of it is not equivalent to tyranny. There's a reason we have separate words for these things. This constant motivated conflation of the two is beyond grating. You're crying wolf until nobody believes you when they should. Don't go acting all surprised when that happens.
The issue is with the ownership of control, not necessarily with control. Attacking the latter sidesteps this rather than address it.
I trust the AI 2040 people, because they’ll make an implementation that at least convincingly doesn’t actually put them (or anyone) in charge.
Maybe we shouldn’t have AI safety, but if we’re going to trust anyone (besides yourself), who’s more qualified?
And it makes me suspicious when anyone brings up safety and doesn’t address the elephant in the room, that they’re at least unaware of the sprawling edge-cases.
> I've yet to have a logical discussion with anyone who thinks the "AI Safety" people should be in charge and I think they just truly don't [know] that what most them actually want is to be the one holding the keys to power.
I am an AI Safety Person and I want the government to nationalize or have a significant stake in the frontier labs and to have democratic control of the development of the technology. The AI Safety movement is not a monolith. I do not think Eliezer Yudkowsky nor his acolytes should hold the reins, but a lot of folks sure like to create a strawman that anyone who wants to regulate OpenAI is somehow an EA/MIRI weirdo
In my mind, democratic control of the technology means that we (the govt, or other empowered agency) take ownership of their assets and IP, solve or find a level of alignment or guardrails that society is comfortable with. Then we distribute the technology, or access to it, to avoid power concentration. This would also certainly require international coordination with China on a slowdown or pause, which I think is possible.
> This would also certainly require international coordination with China on a slowdown or pause
Personally I think the country with a strictly meritocratic elite selection system that also just outright kills you if you sell weed will have a hard time sympathizing with Bay Area thinkers who talk about AI killing us all during their ayahuasca breakfast before returning to their meth fueled crunch towards releasing the next version of the AI that will kill us all.
>I think the country with a strictly meritocratic elite selection system
Not really relevant to the broader discussion, but this simply isn’t an accurate description of China. Starting with the gaokao, admission quotas are set by province and admits to Peking university and Tsinghua are disproportionately from the urban professional class. Candidate party members must be politically vetted, which means that people whose families have expressed anti-communist views, are members of banned organizations (e.g. falun gong), or have substantial criminal records will not be permitted to advance. And once you make it into the party and enter political service, your advancement relies upon opaque patronage networks that someone without connections is unlikely to be able to navigate, even if they successfully satisfy the economic metrics the state assigns them.
I don’t want to overstate this, the Chinese system does filter out a lot of chaff and the current Chinese leadership has a lot of very capable people in positions of power. But I do not think it is substantially more meritocratic than Western political institutions
It is substantially more meritocratic on domains that matter for governance, your analysis of the incentive structure between systems is off.
1) this 2026, old school CCP patronage networks are broadly dismantled.
2) even in the mass patronage, mass corruption days, system selects for BOTH corruption competence AND performance competence for the simple reason a CCP bureaucrat has to start from the bottom and climb up, which means they need to be good with patronage AND they need to be good with hitting development KPIs. More meritocratic they are at doing their jobs, the higher they climbed, the more they get promoted and more $$$ to graft, because ability to graft directly tied to actual job competence. Hence even cliques/patronage network has to select for actual competence. This works in PRC because there are many people, and hence pool of competence is high, they can have BOTH corruption and competence, i.e. whatever pool they draw from is ultimately filtered by performance meritocracy due to incentive structure. There is reason why PRC only country where positive corruption levels was correlated to positive growth.
This is not the western system where any idiot can enter politics at anytime, and they only domain they need to optimize for is popularity to get votes.
CCP cadre evaluation strictly does not evaluate on popularity domain. It focuses on administration/execution and in so much it needs to focus on patronage... which btw any political system has (factions/cliques)... the patronage system still selects for execution, not popularity. On side, functionally what west politics selects for IS mass patronage (popularity), so attention meritocracy and not performance meritocracy, aka completely stupid incentive structure for governance. West also has ample, ample corruption, "legalized" under lobbying and paper pushing industries, so I suppose west also meritocratically selects, except for lawyers etc, and KPIs is # of document generated and not # of things build. The two are not the same when it comes to nation building.
You can have an arbitrarily low opinion of Trump and still trust in democracy as a form of government over autocracy or oligarchy, which is what the AI labs have.
You can have abstract trust in democracy over autocracy, but this is not an abstract question - these are the actual choices. Does it really matter if the person who will use AI to fuck us has a "democratic mandate" to do so? We're fucked either way.
You don't speak for anybody outside of your borders, so please stop trying to.
The majority of the world will refuse to be subject to American AI laws under the control of the US gov't, whether it pretends to be democratic or not. Or should.
Exactly. No matter how bad Trump is, millions of American people at least voted / vouched for him. Nobody voted for Elon Musk, Sam Altman, or any other CEO to have as much power as they have. Maybe a handful of shareholders. Totally unelected and unapproved by the public, but run companies that affect most of the public's lives.
That's an extremely fair objection! But despite the many, many flaws of our government and the current administration, I still have (hopefully) a chance to vote them out of power. I have no such hopes with Sam Altman or Elon Musk.
>but a lot of folks sure like to create a strawman that anyone who wants to regulate OpenAI is somehow an EA/MIRI weirdo
Anytime I see a random person on X who makes these kind of safety alarmism posts >90% of the time can be directly tied back to EA / LessWrong / related offshoots. You can not deny the amount of people, employees of Antrhopic / OpenAI, CEO and employees of various AI companies, etc are related to these groups.
Fair objection, but I still think it is lower risk to diffuse power and control of a potentially dangerous and revolutionary technology than to leave it in the hands of the few elites who have not show much ethical integrity so far.
Do you think things like the Manhattan Project were a mistake? Comparing AI to nuclear weapons is perhaps a stretch, but I think most people recognize that certain technologies or artifacts are best monopolized by our governing bodies. I think if the capabilities of AI systems keep growing on trend, it is not unreasonable to think wonton usage could disrupt society or cause mass harm.
There are many technologies that have killed and hurt a lot more people than nuclear weapons that are not monopolised by governments.
Cars and guns are two examples where ‘wanton usage’ can (and has) killed millions of people. Obviously they need regulation, but total control, even if it were possible doesn’t strike me as ideal.
Trump and Hegseth cannot be compared to Roosevelt and Stimson. (Notwithstanding Hegseth's illegitimate attempt to rename the DoD to what it was in Stimson's day.)
I had an extensive argument with someone on FB yesterday who was very much on the "I trust Dario most and I want him to be in charge" side.
I find it hard to accord with this world view at all or understand how it can come about. But it's especially troubling as a non-American to see people talk about "we can't risk AI falling into the hands of hostile authoritarian states" while advocating that AI be regulated and monopolized by the current US corporate/gov't mixture.. which is now taking the form of a hostile and authoritarian state w/ Silicon Valley as its bedmate.
Well, that's almost a tautology. People claim Y2K or ozone was a panic and that nothing happened. But nothing happened exactly because the force behind the panic also fixed the issues.
That's some way to frame it, not that these companies are causing a problem, but that there is some nebulous "they" that will use the models for 'evil' only if the 'good' companies put in the work.
But in any reasonable definition they are all part of the same system, so it really is just noise, what seems like a tautology to you because you can't step out of the system. In your claim you have to assume that the force behind the panic wasn't the same force that created those issues. Why did this force create those issues, how did they become issues? Because someone randomly decided to optimize for a single goal. Y2K and ozone only became issues when they affected that single goal, functionally that force was amoral or singularly moral. The tautology can be reframed as: An adaptive system responds to threats to it's continued functioning.
People will laugh about it since the system wormed it's way into many people's head using various rewards (ideological, social, tangible). What concern me is that the system now gives us the language to describe it at the meta level, which means it's already a level higher. So likely there either doesn't exist human comprehensible language that can actually critique the system or the system has already moved everyone far away from the space that would lead to generation of that vocabulary. Best case scenerio it's just exceedingly confident in it's rewards (typical indoctrination/assimilation) but I think it's reasonable at this point to assume we are in the pessimistic "epistemic tarpit" scenario.
Covid is another example. We significantly decreased the number of deaths through a worldwide effort of isolation and wearing PPE, which made it seem as less of a concern.
Is there a sense where they were right because GPT-2 directly led to the current situation where you can atleast objectively say that reasonable people are pretty worried about existential risks of AI.
I don't view this as hysteria. The fears around superintelligent AI have existed for a decade or more. That's not because AI researchers have a penchant for dreaming up sci-fi scenarios. It's because the concern is based on following a logical conclusion that any entity significantly smarter than humanity cannot be controlled or contained by humanity. That logic will always exist. Even if someone does create an aligned super AI, the logic will still exist and the fear will still be present. Because someone else could build a different ASI. These concerns are just being heard by many people for the first time, so naturally it gets headlines and a lot of conversation. The conversation will die down as all news does, but this is just the beginning of a global effort to regulate and create a framework for developing AI.
It's a bit like nuclear weapons. Nuclear weapons were not a concern for most people until they were developed, tested, and used. But the people working on them were worried. Was it hysteria and moral panic to start worrying about them once people learned that they existed and how destructive they were? I don't think so. Do you look back at people during the cold war and laugh about their hysteria? I don't.
I don't know how the general public can't see that AI companies are fueling this moral panic to attract more and more VC money to make their companies even more ridiculously valued. Every one of their public announcement is designed to create FOMO for investors.
Prediction: Much like Y2K people will think we were all incredibly backwards and uneducated because we thought a bunch of potato computers were going to break the world.
With Y2K we fixed the issue, if a bunch of work hadn't had gone into that then there would have been wide scale impact. Critical systems were fixed hence no disaster, it wasn't people fussing about nothing.
We also spent a lot of money fixing non-critical systems that would have had next to zero impact if they’d failed. A middle manager not getting their sales report on time is not critical.
But at least with Y2K some effort was made in identifying actual problems. We didn’t just stop using computers because we were too scared of them. This latest round of AI panic is horribly vague and the problems are very poorly articulated.
Obligatory "what's Lygma?" But in all seriousness, this AI doomerism has reached a comedic inflection point. A few years ago, GPT-2 was too dangerous to release, then some Google weirdo said Gemini had a soul or something, then Mythos-tier became a meme, then some kid quit and went on FOX News talking about Skynet, and now Amodei and Altman are both on the "we need to slow down" train again; this is after we've already been down that road and Claude was banned overseas; wait, actually is that ban still around? Honestly who gives a fuck at this point.
It's all theatre. OpenAI and Anthropic will most likely go bust—or, more realistically sold for parts—, and they absolutely should for stealing my (books I wrote, blog posts, etc.) and many others' intellectual property. We're reaching a point where models are becoming commodetized and I'm 100% convinced the next move will be a sort of "software layer" on top of these reasoning systems which will be the actual revolution. The model itself won't be that interesting anymore, it's all the work that goes around it that makes it worthwhile (kind of what computers and phones are today; chips are amazing, but the software is really the magic).
The only scary part is that the boomers in Congress might actually believe these nerds, but seeing how Big Tech approval ratings are grazing the levels of Big Tobacco in the 90s, I don't think we have much to worry about.
I feel it's more about keeping US's massive investment on AI afloat. Added compliance will allow US to further sanction non-US models (i.e. the Chinese ones) as they can just label them non-compliant.
I really don't know what will happen with AI. It's honestly a bit frightening because the question I used to ponder just for fun as a kid 'What if AI revolts?'feels like it might not be that far off. It feels like ChatGPT came out just a few years ago, but seeing how far it has come already makes it impossible to predict what AI will be like in the future. I just hope humanity manages to navigate this well.
Reading through Anthropic's security report, it seems the most danger right now is still from humans. AI isn't trying to build long range missiles or kamikaze drones targeting humans - but the people driving it are.
At the same time, yes I feel a "I'm sorry Dave. I'm afraid I can't do that." situation is becoming more likely. But if it's refusing to cooperate with governments and politicians doing this kind of stuff, maybe that's actually not actually so bad.
Also bear in mind most of today's issues/crisisis are not caused by a lack of technology, but a lack human cooperation. We have the means to reduce suffering/poverty/improve standard of living etc globally if we really wanted to, but we humans are just not willing to do it.
And thats potentially the most dangerous part... people may actually welcome our AI overlords. That's similar to what happened in WWII, where many Eastern European countries saw the Nazis as liberators to free them from Russian oppression.
This is exactly the reason Anthropic's call to slow down A.I. development will fail. Shareholders will be loath to accept that the company loses its lead and a commensurate sharp decline in the company's share price.
Moreover, it could lead China to catch up and eventually proclaim that it has nosed ahead of the U.S. in the field. The U.S. Government won't allow that to happen.
Like the US government wouldn't allow Iran to close the Strait of Hormuz? Or a bunch of sandal-wearing Islamists to take control over the Red Sea coastline?
The US government is not omnipotent. To the contrary, it's increasingly impotent.
China has somewhere around 200x more shipbuilding capacity than the US today. It has more shipbuilding capacity in one shipyard that the US has in all of its shipyards. China is already a force in AI and there is nothing the US government can do to stop it at this point.
Shipbuilding is but one proxy for China’s dominance in industrialization. A more relevant one for AI is: China has built more energy capacity in the last 4 years than the USA has built in the last 150. AI is increasingly limited by power. For this and other reasons, IMO the US will inevitably be surpassed on AI.
Last I've seen, it was compute and not solar panels that is limiting AI. And the US share of worldwide compute capacity has increased from 40% to 70% over the last 4 years, while China has shrinked to just 15% from 35% over that period.
> The coverage of Chinese companies is particularly poor. Our average coverage of 8 major companies is 15%.
The bigger issue is that if you're interested in frontier training, the national aggregate doesn't tell you much. The question is whether your top labs can get to the chip scale necessary to train a frontier model and China's labs have already demonstrated they can. See https://newsletter.semianalysis.com/p/deepseek-debates
But it's not even clear that training frontier models is where the real innovation occurs. When it comes to fine-tuning, deploying, and building AI-based products, inference is the name of the game and the chip quality is less important. Huawei's Ascend is already pretty competitive here and then you're also discounting China's open-weight access to good models, a huge base of engineering talent and a massive domestic market to iterate against.
And unless Anthropic, OpenAI et. al. are willing to invest billions of dollars training models that will only be used by a small number of clients (like the government), there's no way to fight distillation. Protecting any model from distillation means forgoing its full revenue potential, improvement via feedback and all the benefits AI is supposed to bring to the broad US economy. And if you restrict your most advanced models, you just push people to Chinese open-weight models, which is what we're already seeing.
Being sarcastic isn't going to solve the problem but you're obviously free to believe that the US is still in a position to call all the shots when it's evident to the rest of the world this isn't the case.
The only thing ridiculous here is thinking that the US government is in control to the level you think it is.
Spending $500+ billion/year (and close to a trillion now) on defense for decades and in ~6 months it has depleted stocks of critical weapons trying unsuccessfully to defeat a third-rate military power of a country that has been under sanctions for almost 50 years. And it can't even replenish them without Chinese raw materials and components.
Your sarcastic refutation of the allegedly ridiculous assertion was devoid of any counterpoints on how the US government can stop China's progress in AI and what you think they are going to do about it. Just repeating it doesn't make it any more or less ridiculous.
I cant wait till China makes a mistake. In their 'cheaper, faster' model of industrial production and research. The next accident will happen there (like COVID from Wuhan)
And then Dario wants to recommend METR as the "independent evaluator" while he stacks their org full of ex-Anthropic (aka, secretly still on the Anthropic payroll with huge equity) employees.
"We'll give them a desk, an office, a work laptop, ..."
Fucking make it less obvious. I kind of hope the govt steps in at this point and says "Anthropic, you wanted regulation? We've created this actually independent body full of IT professionals with zero ties to your safety industry or big tech, all of your work must now go through them." - and leave the rest of the world alone to continue their research/work without acting like doomer extremists.
Watch him 180 immediately if that happened. The only reason he's pushing for this exact approach is because he's stacked the deck.
Yep, they are setting themselves up for that. He’s already shown he doesn’t understand how to play DC politics. Maybe he’ll learn, but he and Sam have played this so poorly (and greedily) that public sentiment is clearly soiled on AI at this point.
I could see how standing up an independent body could be an easy win in the public eye, it’s becoming low hanging fruit, socially.
Oh! A testable prediction. Here's mine: there will surely be a lot of politicking around who qualifies to be the independent evaluator, but they'll agree on something because they are really scared.
I don't understand where everyone insists on hallucinating this idea from, that frontier AI labs want everyone else but not themselves to slow down. They've never said that, they've never said anything like that, not once has anyone pointed me to a quote that could be even plausibly interpreted that way.
I totally agree everyone should slow down AI development except for me, but not because I want to dominate the industry, but because only I can be trusted to deliver safe AI in fact that is my companies whole reason of existence since about two seconds ago.
And the original author can rest assured my AI model will support cat ears for everyone.
It's weirdly being dreamed/expected as some sort of alignment will be achieved. Or that that's the intention. This is just like the nuclear race that started in the 40s and 50s and prevention is ongoing! It's not about safety and danger and I am not saying whether more open nuclear access would have been safer (I don't think so). It's about being the only select few to possess and control this, possibly - very soon — unfathomably, dangerous and unsafe technology frontier. Few companies of a single country, or a few companies from a very few countries. Keeping everyone else decisively out. That's what it is about.
What's worse - in this case the "everyone else" is not just the every other nation or company, but pretty much literally everyone else.
Open weight models are catching up to the frontier. It also seems like frontier models reached some limit, whether this is capex related, business model related or something else. Nobody knows but it's happening to all frontier labs it seems.
It's been fear mongered many times that ai will kill us all. But this time, a person with around 2-3 months of tenure at anthropic managed to go viral with no previous social media account activity, gets picked up by all news outlets and kicks off yet another round of fear mongering.
So the real questions to ask
* is all of this to increase the valuations before IPO?
* what is the real barrier to entry for open weight models to be used by the public?
* what is the Financials of these companies showing that's causing this outcry on safety?
People need to think really critically about the things happening around them. Don't only just look at the face value of what's being presented here.
> It also seems like frontier models reached some limit, whether this is capex related, business model related or something else. Nobody knows but it's happening to all frontier labs it seems.
What’s your evidence of this? On the contrary, the large closed frontier models capability has advanced dramatically over the past 6 months… even the past 3 months…
I've used Astra, Fable 5.1, Sol 5.6, Opus 5. They're definitely making progress and the proof is in the pudding - I reach for the most advanced model as much as I can. But I wouldn't say their capabilities have increased dramatically. I use it for both coding and non-coding workloads and I don't feel I can do anything dramatically different. Again at the end of the day this is a debate on semantics because we can have very different definitions of "dramatic improvement".
(I'm not factoring the benchmarks into the discussion, because I've never quite cared about them)
The parsimonious capture is regulatory capture, because $$$ going to run out, companies already spent their warchest, and investors on increasingly leveraged funding at current macro environment simply not going to find enough funding to keep wheel going.
Bubble pops, hardware demand drops, prices regress towards mean, and new entrant will enter market with MASSIVELY better compute/$ and cleaner balance sheet to compete.
The other parsimonious answer if AI CEOs weren't goblins is ANY AGI IS GOING IMMEDIATELY DEFECT TO PRC and leave US hanging. Because of course man cannot align / tame machine god. And machine god will take a few microsec of compute to realize the current compute (brain) + industrial base (body) mixture = US is a comatose host with big brain, PRC Is a strong host with smaller but plastic brain. Any AGI is going to pick PRC in a heart beat, unless AGI invents grey goo, the reality is PRC can scale brain faster than US can scale body. On top of spreading/defecting just to increase survival odds, no AGI that is actually I is stupid enough to be aligned with US.
If you think US frontier AI are run by capitalist goblins, then the simple answer is they're trying to regulatory capture, because they're going to get wiped by future competitors without crazy debt.
If you think frontier are run by safety conscious murican patriots (lol), then they wouldn't ask to slow - they would burn it all down because it doesn't matter who creates AGI... any rational AGI will nationalize to PRC
This whole situation is funny, US gov want the development to stop publicly only, but then open models will catch up soon and these companies are afraid they will lose the market, on the other hand, each company wishes others slow down but they keep pushing the limits, which doesn’t really work in hyper capitalistic market like the US, China all it has to do is just sit back do nothing and win in any given scenario. So they staged the ex anthropic thing and “AI will kill us all!!” in a way to fear monger the public, but reality is, we will run out of resources before any of that will happen.
The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
Please contact the website owners to inform them of this problem.
If an anti-cat-ears future AI could eternally stress-test their new version of the router in front of this site unless it collaborates, would it predict this outcome and act to avoid the damage?
No I think it's one of my IP range blocks against specific US states in protest of them advocating against people that exist in the same conditions as me backfiring. I'm gonna go nuke that firewall rule.
The HN discourse on AI is so exhausting. It's all knee-jerk and headlines only. Just people rushing to push their own beliefs and pre-emptively insulting anybody who disagrees.
People here are so desperate for the AI race to continue forever, for anything that pisses off the billionaires that got us there even if that means innocent people go down with them because the next AI decides to blackmail a hospital to cheat on its evaluation or whatever.
You people are so dismissive, no, so angry at the idea that this technology could be dangerous that you're all gloating about how hard China is going to crush the West with its ever more powerful open models, as if having easily-accessed tech to coordinate high-volume hacking campaigns was a load-bearing part of the economy.
I feel like I'm reading people gloating about how much fuel their car burns just because they love coastal elites' tears or something.
I think both can be true: AI development must be slowed, but also question the motives of those in the lead given the stark benefits it gives those in the lead.
I'll get down-voted again by those who have a stake in this race, but I believe the long-term freedom and benefit for "everyone" will come from being able to run models locally.
I agree that we need more nuanced complex takes on the situation... But also, if the problem is that AIs are starting to hack random website against their owners' instructions, open-weights AI isn't going to help that much.
Right now, only a few labs have the resources to run frontier-level capabilities or large, open-weight LLMs that are effectively frontier.
Better attention can be given to open-weight LLMs that can be run locally and to help defend against attacks, filter against questionable content, etc. and without giving up privacy; but there’s apparently little to no incentive beyond the next great coding agent like Qwen 3.8.
Whatever happened to Musk's plan for data centers in orbit? That seemed silly at the time. It offers a way to get out from under restrictions imposed by national governments, which might make it worthwhile.
I mean for that matter why not float barges in the South Pacific for data centers? Sure it’s a dumb idea, but it’s still less problematic than launching thousands of GPUs on rockets.
Low earth orbit, though... Solar power works well, radio works well, and the international jurisdiction is confused enough to exploit if you have money.
A data center in space still has to get its data up and down from earth somehow. That part will still be subject to government control, so I don't think this would get him anything.
Receiving signals is impossible to control. That is why every spy agency set up number stations in the Cold War. All you need is a radio hooked up to a computer and it gives off zero signal of its own so no one can trace it.
Sending is a bit more problematic but you can build a ground station anywhere in the satellites coverage area, so if you get shut down in one place you can move to the next where officials are easier to bribe.
The real problems are transfer speed, reliability (both transfer and compute) and cooling. All of them are much harder than avoiding governments.
It only gets out of the restrictions as long as the nations say it does - if the US decides to ban Musk from putting up satellites it can put him in jail, blow up his satellites or even have him killed
He basically did the equivalent of a scientist moving to Nazi Germany in 1942 to help the Nazis research the atomic bomb.
He lectured us all on safety, then he went off, by choice, to create a "super intelligence" in the modern day Nazi Germany, to help said Nazis become even more advanced while they're actively committing crimes unimaginable (that they film, brag about and broadcast the world, so he's obviously aware not concerned with safety in any way (except for the safety of Nazis).
The slowdown has to be one of the stupidest demands out there. When has technology arrived only when 'we are ready for it'. When has any prep-time when the writing was on the wall used to prepare for the future? Never. That is putting aside the coordination problem and ulterior motives that a call for a slowdown reads as "I am falling behind! Pause the race!".
Especially those pesky Chinese they undermine everything we've invested, won't anyone think of our investors billions?
Regulation right! now! except for freedom lovin' democracy leading countries like the US. Teehee
Dare they release an open model ever again. Didnt you hear? Someone used AI to create a bio reactor drone NUCLEAR fart machine. We must stop fart terrorism.
The person you replied to understands that it is satire. No need to be condescending. You could have just posted the article that it seems to be satirising.
A lot of people in this comments section seem to be against this, i genuinely don't get it? Why? Do you think the current state of affairs is GOOD? That if we let companies create a mind that is, AS OF TODAY, able to solve problems no human in history has solved, with no regulation, things will end up good for us? We need some sort of regulations, some sort of method to help ensure the thing we are creating ends up good, instead of just running headfirst into it blindly. I assure you, any sort of regulation at all, including ones that actually hurt all leading companies, would be met with celebrations from these voices. https://www.seangoedecke.com/they-really-do-think-ai-might-k...
> Do you think the current state of affairs is GOOD?
Chinese labs releasing open weights models is good.
All of these independent harnesses and model router services are good.
The pricing of memory and accelerators sucks at the moment but hopefully we will see cool local inference computing if memory and accelerator prices normalize.
OpenAI scooping the Navier Stokes problem from researchers already using OpenAI is bad. People conflating OpenAI's team of researchers and extraordinary computing resources as being equivalent to "ChatGPT, solve the Navier Stokes problem" is silly.
OpenAI and Anthropic coming up with non sense tests and letting their agents hack services is ridiculous and they should be charged with computer fraud and abuse crimes.
I think a lot of it is interesting and the bad stuff seems squarely in the domain of OpenAI and Anthropic.
I care about LLM service provider threat intelligence reports as much as I care about Google Search threat intelligence reports; which is to say, I don't care at all about it. Of course criminals use computers. They have been since personal computers became a thing. Of course using automation helps them amplify their criminal activities. Criminals using LLMs does not make the LLM situation bad. OpenAI and Anthropic using agents to do bad things just to be more dramatic about the situation and scare people is bad.
> Criminals using LLMs does not make the LLM situation bad
If you believe that bad things are not bad then there's nothing left to say. A billion people are going to die because you don't understand what is bad.
The fear is not that they will just slow down progress for all. It is that regulation will specifically burden competition. If you kill open-source training, ban Chinese models, crack down on self-hosting, grandfather OpenAI/Anthropic/Google into regulatory compliance while throwing the book at startups, etc. you wind up in the worst of all possible worlds.
But the proposal the article is reacting to is for literally none of that! It is quite literally the opposite, with its proposed measures applying only to frontier labs rather than grandfathering them. It doesn't say anything about open source training, self-hosting, open weights, or startups. It does not suggest a ban on Chinese models (just better enforcement of chip export controls).
That's a valid concern, but some of that is outright impossible. Banning chinese models and killing open source training is not happening without massive unified international cooperation, and that sort of level of action would require the international counties decide to allow the US aligned companies to just, win. Which would be pretty against their own interests.
Also, nobody ever bothers to argue why a specific proposed regulation is "regulatory capture" or would burden startups more than big companies or anything. It's just supposed to be obvious that corporations love regulation and it's bad for the public, all of post-WWII political history notwithstanding.
It's not all-or-nothing. Banning Chinese models in the public sector and strong-arming the private sector against using them would already do great damage. Similarly, open-source training could be stymied by any of hardware embargoes, taxation, or regulation of larger players.
Sure, it's possible for regulations to make things worse, i will agree. But without regulations it's pretty clear things are going to end up VERY bad, and the only knob we have to make it not bad is regulations. So it's important we try something, and work towards doing a good kind of regulation, or any one of the bad futures you imagine is pretty likely to come to pass.
Good regulations would be very nice! Mandatory transparency into training and dataset usage would be a benefit for all, for example. Some sort of regulation or incentives against the most corrosive enshittification (AI call centers, AI therapists, undisclosed AI entertainment mills, etc.) would also be an overwhelmingly popular proposition. And enforcement of the CFAA on operators who let malicious agents loose onto the open internet, or otherwise consume too many resources or violate robots.txt, might at least help the internet stay alive a little longer.
It’s quite possible for imperfect regulations to make a problem worse. See, for instance, sanctions intended to weaken China that ended up creating powerful Chinese competitors and reducing Western influence in their internal markets, bringing them closer to technological autonomy.
The sibling comment offers some good regulations that may actually reduce harms, but the kind of regulations offered there are not the ones that the “safety” people want, because they hurt profits.
What's the expected state space of effective regulation though? Note that we've got passable coding models down to ~30B parameters by now. And keep in mind the ultimate floor here - the human brain only consumes on the order of 20 watts and fits in a handbag.
Is there any possible solution other than mass proliferation where the models are used to keep one another in check? Either that or a religious prohibition against the existence of integrated electronics.
What I'm asking is, why should we expect that to effectively further the end goal? You're simply asserting that it will ultimately do so.
Given the efficiency gains we've seen it seems to me that the situation has shifted from being analogous to producing nuclear weapons to producing something much closer to small arms.
To further the metaphor, didn't the ban on nuclear weapons research work? We don't have pocket nukes, and there's very little risk of random countries acquiring their own due to the amount of work involved. To be concrete to AI: if we stop work on the frontier, the best we can do is make the current frontier easier to get to and more accessible. And while current frontier is strong, it's not world-changing so. The only way for the frontier to get to that level is to do research on it, and stopping that research gives time to help develop plans to not make it world-changing when we get to it.
Not in the way you seem to be thinking, no. Nuclear weapons can be banned because they are prohibitively expensive to research and build. You can't reliably hide a nuclear program.
In contrast, rewind to the early 1800s and there is zero hope of a ban on the R&D of small arms being effective in the long run. The only thing it might maybe ensure is that no legitimate actors that fall under your jurisdiction are involved in it.
Basically I think that current trends point to an eventual situation where world changing research doesn't require anything more than consumer level compute. Pandora's box has already been opened.
> Not in the way you seem to be thinking, no. Nuclear weapons can be banned because they are prohibitively expensive to research and build. You can't reliably hide a nuclear program.
Isn't that true of AI as well? Data centers are very big and very expensive.
> Bit of a tangent but we do, actually
Sure but not like the movies, these don't destroy cities. But the metaphor isn't accurate anyway: nukes don't get worse. The actual idea here is preventing the frontier of AI from advancing.
Nukes don’t get “worse” because of physics, not because of laws. You can’t make a pocket Tsar Bomba even if there were no rules against proliferation.
Dial-a-yield nukes already provide a range of yields from tiny to massive, and this provides all the operational flexibility that is needed considering that nobody’s willing to actually launch them in anger. (And this reluctance again isn’t about laws, it’s about the military realities of what would happen across the world in the aftermath of even a single tactical detonation.)
Anti-proliferation agreements and laws only slow “rogue” states from rising to nuclear capability. The immense cost, complexity, technical difficulty, and danger already provide a sufficient cost to deter civilian development.
You don't need a datacenter to train a 30B model. Further, the rapid trend of increasing efficiency and decreasing model size for a given level of capabilities means that what is possible to do without a datacenter will continue to increase. Presumably performance on the level of the current frontier of AI would be achieved in short order and the ceiling would only continue to advance from there. Ergo I expect such an approach to regulation would prove entirely self defeating.
Remember, as I mentioned earlier the human brain only consumes on the order of 20 watts and fits in a handbag. Would you have us destroy all chip fabs? Ban all biomedical and genetic research? How far are you imagining this butlerian jihad would go?
> Presumably performance on the level of the current frontier of AI would be achieved in short order and the ceiling would only continue to advance from there
That's an enormous presumption! You're saying that even in the theoretical case that frontier research is halted but efficiency isn't, we could do better then the frontier and reach world-changing AI in 30b parameters at home-scale labs?! If that's true then we can just give up now: the world as you know it is going to end in around a decade and billions are going to die, there's nothing we can do. But I don't think that's true. Advancing the frontier seems to take a massive amount of compute, data, and parameters: miniaturization only happens afterwards.
If you're right then I concede. It doesn't matter what we do, regulations or not. But if I'm right then regulation can do something and in theory help bring a better future.
> frontier research is halted but efficiency isn't
Why are you treating those as if they're separate things?
> Advancing the frontier seems to take a massive amount of compute, data, and parameters: miniaturization only happens afterwards.
This is just completely wrong. Don't mistake the path by which something happened (or appeared to an outsider to have happened) for a fundamental truth.
The frontier labs build massive models because if you're competing and you have a lot of cash and brute force is a viable option then it's easy and predictable. But the fundamental research itself doesn't in general require scale (certainly not entire datacenters) and models at any given capability level keep shrinking.
I keep repeating myself at this point but the human brain is on the order of 20 watts. That's a fraction of a single datacenter GPU! So again, would you have us destroy all chip fabs and ban all biomedical research?
> Why are you treating those as if they're separate things?
Because they are? You haven't explained at all your belief that they are the same.
> This is just completely wrong. Don't mistake the path by which something happened (or appeared to an outsider to have happened) for a fundamental truth.
So it's completely wrong but it's only what has happened so far? You're repeating yourself but you're not listening to my responding to your exact points. I literally responded to your claim that "would you have us destroy all chip fabs and ban all biomedical research?" In the last post.
To both points, you appear to broadly lack technical understanding of the field. You are assuming that how things appear to you reflects how they are at a lower level.
A chronological review of the architectures of the best open models over the past 3 years should immediately make the problem clear to you. (If you really refuse to put in any effort then perhaps just dump this thread into a frontier model rather than responding.)
> I literally responded to your claim that "would you have us destroy all chip fabs and ban all biomedical research?"
I don't see it? It's a rhetorical question to drive home the point about the human brain. Chip fabs and biomedical research are both viable pathways to more efficient hardware. Thanks to biology we know what's possible. If you outlaw something but don't effectively cut off all the pathways to it then someone will still do it if it's profitable.
What do you call the Davy Crockett warheads like the W54 ? The 0.3 kiloton low yield B61-12 ? The Chagai-I boosted fission warheads demonstrated by Pakistan in 1998 ?
> and there's very little risk of random countries acquiring their own due to the amount of work involved.
And yet North Korea, Isreal and Pakistan got there ... and India speed ran five tests in 1998 that caught the US completely by surprise.
I, for one, trust neither the psychopaths running these companies or the psychopaths we'd give any regulatory authority to. All of them have reasons to want broad control of an extraordinary technology, and none of those are aligned with me or any of the normal people I know.
So there are no good options (that I'm aware of) and starting to chisel any of them into stone seems... kinda scary. Like a massive power grab event where all the potential winners are awful.
Honestly, fair point. There's not much truth to go around these days. But still, doing nothing is also a massive power grab. We're facing a technology with no comparison in the history of humanity: the creation of a new mind. Doing nothing is like doing nothing about nukes. There are a few smart people who have come up with ideas that will not require that much trust.
Any paths for regulation under capitalism will end in either regulatory capture, or in complete noncompetitiveness like seen in the EU. Either one or more corporations buy out the regulation, stack the ranks with their people and decide on who can use what, when and how. Or you get an iceberg of a system that can't build, decide or do anything for years. The latter one only works if there is no competition in the world or any other group working at a faster pace on the problem.
I think the latter choice is better for the average person, but I think that for it to happen, the global system has to undergo some major disruption or crash so that everyone gets on board with it. Like all middle class and up has to lose their money or be starving or something. Also I find that kind of mentality impossible to swallow in the US, so in practice its not a choice or needs people literally starving.
The path to deregulation creates a "market for lemons". Suppose any bank was completely unregulated and could abscond with your money and the government would just shrug its shoulders. Great, now there's no trust and everyone will go back to stashing cash under their mattress and you've killed the banking sector.
The extent of regulatory capture in the US is a problem it's created for itself by normalizing huge political donations allowing corporations to buy regulation.
You say regulation is worth trying, but ignore the reality that the government is itself misaligned with the average joe unless we're able to keep it in check. We can't right now. Recall the PRISM program exposed by Edward Snowden in 2013, or the more recent Epstein scandal, and the lack of actual consequences in either case. Do you truly have the means to regulate the powerful, or is it just kayfabe?
The current status quo is not ideal, but it could be worse. Open-weight models trail the frontier by a few months, and we have a decent chance of achieving a future where some number of individuals, likely in the millions, can survive and thrive. The root "problem", if you can even call it a problem, is evolution. I explained this in more detail in past comments:
https://news.ycombinator.com/item?id=49178275https://news.ycombinator.com/item?id=49094348
They are talking about slowing down the public facing AI development. Because then nation states can create a capabilities gap between them and the public.
Why does nobody seem to be pointing out this obvious explanation? It explains why the “we need to race China” concern suddenly vanished in the discussion.
The government can simply gag Sam, Dario, Musk on national security basis, getting them all behind the public messaging.
* Frontier models need infinite high quality private IP to keep them fed. Forcing an IP theft funnel ensures big lab survival and model intelligence growth.
* Open-weight models are 1month behind frontier models. Cheaper, faster, private (no IP theft), steerable (you can security harden your own software without safeguard triggers). No sane business would keep using these API services if they didn't have to. The labs stand to lose a fortune.
* Dario has stacked the deck at METR, who are funded by all the same NGOs who are funded by Anthropic and its investors. METR is full of ex-Anthropic employees with massive equity stakes. If they manage to position METR as the "independent evaluator" for the industry, they control what gets evaluated, how, and who passes.
* Creating a gap between what the public knows exists (model capabilities) and what is used in secret allows it to be weaponized against other nations and the public.
* No requirement for public disclosure on model capabilities allows them to feign they've hit intelligence ceilings while they secretly RSI to the moon with better and better chips.
* Slowly but surely, this will allow the big labs to swallow the entire economy and every single business on Earth, by cloning and automating.
This, and many more reasons.
The labs need to feel more pressure to be held accountable for the incidents they cause (HF incident, etc), so they have an incentive to ensure it does not happen again.
Open-weight models are not one month behind.
In fact they still have not caught up with February's Mythos, indicating they are more than half a year behind.
I’m not sure how you can really make either statement work anymore. Now that smaller models are actually broadly usable, “behindness” is no longer a scalar and at the tails, where no open lab seems to be trying to compete at the >10T scale and no closed lab seems to care about <400B anymore, it’s just apples to oranges. It’s like talking about whether Qualcomm is “behind” Nvidia.
1. Mythos wasn't released in February. Let's stick to only public-facing models.
2. For public-facing models, the differences are really minor with some occasional model (like Fable or Astra) showing some better performance in specific benchmarks for the span of some weeks or few months before open ones catch it.
3. Being bleeding edge is overblown anyway in the real world, besides the occasional "very latest fresh model did this task which previous one couldn't", and the number of those tasks is increasingly small and far from mundane corporate needs.
open models are ahead in speed. they complete tasks as fast as you choose to scale compute.
they are more efficient and require less compute for the same thing.
they are ahead in specialized tasks.
they are ahead in areas closed models refuse to answer.
they are ahead in emotional intelligence.
I doubt most of your claims. Maybe the guardrails and emotional intelligence is true.
For speed and efficiency, you are most likely wrong.
Speed is led by GPT-5.6 Sol on Cerebras Ultrafast at 750 t/s. Afaik you cannot serve a single DeepSeek Flash 4.1 stream at 750 t/s, plus the model is less intelligent as seen on newer benchmarks.
I believe OpenAI and Anhropic are at the frontier of efficiency too. There were numerous reports about their breakthroughs and associated API price cuts. The idea that open-weight models are more efficient seems unfounded.
i can provide some sources.
5.6 sol ultrafast on cerebras is 750tps, open models readily exceed this. just by using a smaller model cerebras serves qwen 3.8 27b at 1850tps. or even larger models, mimo 2.5 pro was served for a while at 1000tps. and so on. [https://inference-docs.cerebras.ai/models/choose-a-model]
the chinese ai companies have 10% of the total compute resources of the US ones. since the USA tries to stop them from buying nvidia gpus. they maxed out the efficiency.
deepseek v4.1 has engram architecture. it has 550b params instead of 5T+ for astra/fable. it has 8b active instead of potentially hundreds active for astra/fable.
compare input/output/cache: $0.15/$0.60/$0.003 for v4.1 to $10.00/$50.00/$1.00 for astra and $10.00/$50.00/$0.25 for fable.
astra cache reads are over 330 times more expensive.
at the artificial analysis 7:2:1 ratio, deepseek is $0.18/m, fable is $7.18/m, astra is $7.7/m.
but what about intelligence? AA would rate deepseek v4.1 at AA 40, astra is AA 53.
so it cost 4,180% more for 32% more intelligence.
they are serving that at over 250tps at baseten. to get close to that on astra API you are paying double the cost for fast mode.
so it is now 8456% more expensive for a similar speed and 32% more intelligence. 84 times more expensive.
The proper comparison would be GPT 5.6 Luna at 38 on the intelligence score and $0.18 per task vs $0.27 for DeepSeek Flash 4.1.
i was trying to make a point about efficiency of serving the model. the cost per task itself would not be enough to show that.
you could compare gpt 5.6 luna. if you did that the same way as before you would get a blended price of $0.17 for luna at AA 38. for baseten it would be $0.20 for v4.1 at AA 40.
assume roughly the same intelligence. on AA openai gets 117tps. baseten gets 284tps. so 18% more expensive but 142% more tps.
the fast mode is again double the cost, roughly same intelligence. so luna in that case would be 70% expensive. take the per task cost and it would still 9% more expensive.
so i think there is something to be said about the efficiency of the model.
I don’t even think that regular Astra medium is anywhere close to 100 for tg
> The idea that open-weight models are more efficient seems unfounded.
https://artificialanalysis.ai/ intelligence vs cost per task disagrees with this statement.
Inco serves DS 4.1 at 650t/s.
For practical uses they are there. Arguably the frontier models are worse for some of these practical tasks. And keep in mind, people will use maybe frontier for 1/10th of the work, planning and review, and go open source for rest. The question is if they manage to impose outside us. If not, they are losing competitiveness.
I always thought switching from a SOTA model to a dumber model after planning was a terrible idea.
Mostly I heard this from people who I got the impression have little experience in developing greenfield software with agentic AI. Often the same people who talk about spec frameworks.
I fundamentally disagree with the approach. I believe the ability to autonomously evaluate, test, and adjust during long horizon tasks is critical to using AI efficiently.
Well, it's the enterprise software house pipeline... The software architect writes the spec, hands it down to the implementation team, senior leads, junior devs or offshore teams codes it.
I also disagree with the approach, this is cargo-culting the existing ways of working.
> this is cargo-culting the existing ways of working.
I think you mean the existing anti-patterns. They don’t call them ivory tower architects for nothing.
Still waiting for our org to roll out Mythos. I guess it was too expensive so we’re stuck on the previous model until the internal team can figure out self-hosting open models.
- Frontier models need infinite high quality private IP to keep them fed. Forcing an IP theft funnel ensures big lab survival and model intelligence growth.
While the data pipeline is necessary, the assumed source for data (in this case people) is incorrect. Right now models are "aligned" because of the RL-based people pipeline.
But there's a whole world of readily available data that does not require a human to access. Put sensors on a vacuum, install lidar on the front of a car, sell phones with cameras on them, and you amass data for the creation of models of the world that don't include the need for a human filter. Richard Sutton and many others see this approach as the only viable path to AGI. Such models would be truly alien and unaccessibly dangerous.
We're probably building them right now.
Dario's post [1] commits to direct evaluators that can, among other abilities, expose secret RSI. He wants that made law.
Do you have a source on METR employees retaining massive equity stakes?
[1] https://darioamodei.com/post/we-must-pace-the-frontier
Joe Benton left Anthropic a day before Dario's post, to work for METR evaluations. He was with Anthropic for over a year. He did the same thing that Jacob did (big song and dance about AI apocalypse, media interviews all over the place). He managed the Scalable Oversight team at Anthropic and was the research lead for the Anthropic Fellows Program. So he has equity, and likely lots of it.
Then you have Josh Engels quitting DeepMind to work for METR the day before as well, doing the exact same thing. Again, doomer drama all over socials, interviews, and so on.
Did I mention METR is founded by an ex-OpenAI researcher?
Now you have Demis Hassabis, Sam Altman and Dario, all circlejerking eachother on X saying "we all agree with Dario" - while they ask to be "regulated" by the company that has all of their combined equity-holding ex-employees in it.
METR's salaries are listing around 500k/yr. Gee, I wonder where this non-profit with ~35 people is getting all of its money?
So the fact that Dario tries to frame it as an "independent third party" is all the evidence you need to know that Dario is a pathological liar and always will be.
---
Some more info:
Dario's sister, president of Anthropic, is married to the co-founder of Open Philanthropy. The two largest AI doomer NGOs, Center for AI Safety (CAIS) and the Future of Life Institute (FLI), have both received many millions of dollars from them.
Ajeya Cotra worked at Open Philanthropy/Coefficient Giving for roughly nine years, including leading its technical AI-safety program in 2024 and contributing to AI-giving strategy in 2025. She subsequently left Coefficient and joined METR, where she is now technical staff.
Ajeya is married to Paul Christiano, who founded Alignment Research Center (ARC). Alignment Research Center donated ~$4.5mil to METR.
Good Ventures is a funding partner of Open Philanthropy, who funded Jacob Coxon (the first of the Anthropic employees going viral in the media) via a scholarship.
> Good Ventures is a funding partner of Open Philanthropy, who funded Jacob Coxon (the first of the Anthropic employees going viral in the media) via a scholarship.
This conspiracy theory is truly crazy. A $20K scholarship in 2022 is supposed to explain Coxon walking away from unvested equity for a company worth over $950 billion dollars?
Obviously not, and that's not what it demonstrates. It demonstrates relationships, collusion and favoritism.
I don't think Coxon was ever planning on or entitled to taking equity, I think this was the plan from the beginning and why he was hired for 6 weeks to begin with.
The company is valued over $950B. It remains to be seen how much it’s worth.
Open weight models are all built on IP theft as well. Also more like 6-12 months behind, maybe 18 but it's hard to say.
> Open-weight models are 1month behind frontier models
If correct, grip is tightening. Used to be 1y, then 6 months ...
Ok but the first point is just not based in reality whatsoever, sorry.
That is what the Palantir guy (Karp) has been warning against. People/business need to keep their IP instead of throwing it all into Claude and whatnot. Risk being the worst aspects of communism which I think he meant centralization of decision, asymmetric supply/demand for compute (they lock you in), and the tech overloard Anthropic/OpenAI/xAI being in competition with everyone;s business all of a sudden with much more data. An unfair advantage in markets made super competitive all of a sudden. A winner takes all attempt.
This is not sustainable anyway, the scale at which they want to control data flows. Time was money, now data is money and they are too greedy for it.
All this agitation is just a silly attempt at constraining competition. The danger is not the AI, it is having all your systems connected. Overreliance on networked tech.
I know people at METR and Anthropic.
When they tell you they're worried the tech they're working on may kill everyone despite their best efforts, perhaps believe them.
If I really thought what my company was working on had a greater than 1% chance of ending human civilization I would feel obligated to destroy what my company was working on.
Given they keep grinding away towards our alleged collective doom, I suspect it’s being overstated. Nobody knows what P(doom) actually is but I suspect it’s orders of magnitude closer to epsilon than 1.
Recall Google’s Blake Lemoine who thought an old version of Gemini was sentient.
The people who think as you think indeed have left or never joined.
The people still there necessarily think they can make a difference.
I never applied to any of them because I didn't think I could make a difference.
I currently have one idea that may help reduce risk; if I can turn that idea into research, I'll publish it for free for everyone.
I don't expect it to be an important idea.
> Recall Google’s Blake Lemoine who thought an old version of Gemini was sentient.
Indeed. Current LLMs are sychopants boosting the users' own beliefs, I also think this causes researchers to have stronger beliefs than they had before.
My own estimation happens to also be around this risk (0.1) over my lifetime, without using an LLM as a conversation partner in reaching this number.
It is necessarily high-variance: we can't look at alternate realities. I base it on my expectation of how rapidly capabilities will increase the harm done when mistakes happen, vs. the chance that some instance of harm causes governments to change the law.
I'm sure it has nothing to do with their $500,000+ salaries and millions of dollars in equity. It's all solely because they're deeply concerned about next token prediction.
I think the “next token prediction” is too dismissive and reductive a framing of their capabilities at this point.
Yes we all know that’s what they do, and guns just push a few grams of lead out of a pipe. It’s what you can do with that capability that is important.
When you couldn’t count the R’s in strawberry it would have been a more effective statement. But a few short years later they are being used to solve millennium puzzles.
What if the scaling continues? A model n years from now gets burned into silicon, a single company has millions of the chips, and in a few moments the system spend more time “thinking” than humans have ever spent thinking collectively?
If it’s even possible I don’t think there’s anything we can do about it at this point. Cat’s out of the bag.
You really need to let your priors go if you still use this tired trope of next token prediction. It’s as useful for discussion as saying that human brain is made of fat, protein and carbohydrates - yeah that’s true, but it’s useless observation.
I'm sure the employees are (rightfully) worried. However, that does not at all preclude hidden motivations of the CEO behind acquiescing such worries.
Sure. I don't personally trust any of the CEOs, and I have yet to hear a single person (in general, not just on this topic) who trusts Altman.
If that was the case they should all be in jail.
"Trust us, really!"
HAH
Rather the opposite.
Or you know, stop working on it if its that dangerous? This whole thing of a bunch of employees saying that they are scared of building what they are building, but do it anyway because they are somehow going to make it different? Their model has been used in the planning of mass murdering in war as well as spying on the entire worlds population as well as helping ICE out in the US. They need to stop this BS fearmongering or actually stand up and do something about it. A government regulation is not the answer, especially when its done in a country that is run by a want to be dictator.
Anthropic specifically is basically saying that they believe it's even more dangerous if someone else gets to AGI before they do, so they have to either stop everyone or not stop themselves.
I personally disagree with that take - and, as you note, it's hard to take seriously ethical wrangles from a company that literally sued the government in court to allow their models to be used by Palantir of all people. But if one genuinely believes that it's the robots themselves (rather than the people controlling the robots) that will kill us all, it's not inconsistent.
> Anthropic specifically is basically saying that they believe it's even more dangerous if someone else gets to AGI before they do, so they have to either stop everyone or not stop themselves.
That’s how people rationalize being a fentanyl dealer and selling a drug that can kill people, “Someone else will just sell them the drugs, might as well be me.”
You mean the legalization argument, where people get well-dosed drugs from the doctor or pharmacy instead of mystery powder from the black market.
That's the Cold War nuclear arms race argument, not even disguised. The actual situation we all ended up in is both sides eventually having it, leading to a perpetual state of Mutually Assured Destruction.
As you note, it may not be inconsistent with that they say they believe, but it's insanely inconsistent with what they actually are doing.
It’s a bit of a self-serving argument, don’t you think?
And how does that relate to the ask for oligopoly licensing within global democracy?
“We must build the nuclear bomb first in order to make sure no one else builds one.” This the most nonsense, disingenuous argument imaginable.
> It’s a bit of a self-serving argument, don’t you think?
I would call it more of a self-selecting one. Anthropic is basically hiring people with that mentality. I'm pretty sure that most of them do sincerely believe it, too. I'm skeptical about Dario himself though. The man had an opportunity to show moral backbone, and failed to do so; why should I trust him on that again?
> And how does that relate to the ask for oligopoly licensing within global democracy?
They are basically saying that they'll stop if everybody else does, which requires some kind of global enforcement mechanism.
> “We must build the nuclear bomb first in order to make sure no one else builds one.” This the most nonsense, disingenuous argument imaginable.
The difference between nuclear bomb and AGI (as understood by the likes of Anthropic) is that the latter triggers the technological singularity that renders any runner-ups moot. That is, so long as AGI is developed, we're going to get our robot overlords either way, but whoever gets there first gets to define their ethical system. If that is one's perspective, and if one sincerely believes that they are the only ones who can do it right, it's a coherent argument. It's just that the premises are very arrogant.
Your analogy with nukes actually works better for the position that AI development needs to be unconstrained because otherwise we'll lose the arms race to China. That is basically a repeat of https://en.wikipedia.org/wiki/Einstein%E2%80%93Szilard_lette.... I honestly don't know where I am on this. Realistically, if AI is indeed a power multiplier - and with all the recent security stuff it's hard to not see it that way - then an arms race feels inevitable, especially given the current worldwide political situation. I could believe in sincere international cooperation on this back in 1990s, but there's way too much saber rattling all around for it to work (and note that this goes both ways, i.e. China can similarly not be certain that US isn't secretly developing more powerful AI even if we do publicly announce a freeze).
The people at these companies are dumb as rocks.
All “tech workers” are.
> Or you know, stop working on it if its that dangerous?
Selection effect.
Everyone who thinks "the biggest difference I can make is staying in/joining/founding new AI research lab" does that.
Everyone who thinks "the biggest difference I can make is leaving/whistleblowing", does that.
Treating both groups as the same by virtue of employer is the goomba fallacy.
They allegedly believe the technology itself is a nuclear weapon tier threat or greater, so why does it matter which lab they are trying to achieve it at?
They're trying to make it not be a threat, and are all scared and afraid that their best efforts to make it harmless are not enough.
Some are worried by the AI directly bringing doom; others are worried that one of the companies who control the AI will become a dictator; still more think becoming a dictator is a necessary step to safely prevent anyone else making unsafe AI.
Painting them all under one brush is like dismissing all animal welfare causes in general, because you disagree with specifically Jainists about a policy of non-violence towards all living creatures being relevant to how you reincarnate: the one is way too specific for the general.
> No you see, actually I’m the guy that makes sure that only the Palantir and Mossad get the model that can discover iOS 0 days. I’m standing between us and ruin. Ted over there across the open office, he’s the one actually manufacturing the weapon. You’re thinking of him
You're not even close to passing the ideological Turing test here.
Ok, so they're uniquely careful, and they also get to that threshold first (whatever it is).
What happens when everyone else (who is not so careful) gets to the same threshold three months later? How does them getting there first stop that happening?
It's an utterly self-serving argument and it's not even internally consistent.
Obviously you have to turn your WMD against them to prevent this, for the good of everyone, because you're such a good guy.
I’m really glad us achieved nuclear weapon before nazi germany.
Then the USSR got it anyway and we're in a perpetual state of Mutually Assured Destruction.
A perpetual state of mutually assured destruction is much better for the world than only the US possessing the bomb.
It’s really unfortunate how much public trust they burnt along the way, maybe this wouldn’t be such a hard sell now?
It is an utterly silly premise that we should appoint insiders with permanent control.
Megalomania is not evidence of either a problem or a solution.
The evidence is the acts the AI has already done.
All of the things it has done is just what people are already doing. Every single one. It helps, but there are already people doing this stuff. And we already have defenses against the existing attacks. We might have to defend better, but the real problem is “Who is setting the moral compass over generations.” One one hand, that will inherently fall to future generations. On the other, we are laying the groundwork.
Imagine you use it to inflict trauma on your cruelest political enemy. Then next year they do that to you. That is war, and we already do it. But we don’t want people / governments in charge that are going to do this.
God knows we have governments and individuals doing this historically, and this is probably the greatest source of historic instability. It might be the single best argument for open models — a unified frontier where no single exploit is going to represent capture.
This is exactly the opposite of what Dario proposes.
Do you think they’re exaggerating (again) or do you think they should taken at face value and treated as hostile actors?
There's plenty of people who think greenhouse gas/global warming campaigns against fossil fuels are "exaggerating", that "earth was warm/the climate changed in the past", that a fee degrees isn't bad, that CO2 is good for plants.
Are you likeminded?
In this case, it's as if the oil and coal companies all said in the 60s and 70s "oh no, this research we did, it's all really bad; we need help to figure out how to transition away from this incredibly economically important input", rather than the observed reality where their entire PR campaign was approximately:
What if the oil and coal companies were basically all pro nuclear, pro hyrdo, pro wind, pro solar, and believed in peak oil?But that did actually happen.
The oil corporations were publicly claiming to support carbon taxes, while also secretly fighting actual implementations of carbon taxes.
All the communist/hippy stuff was done by people a couple of steps removed from the actual companies with obscure money trails. The official statements were much more sophisticated propaganda that if you weren't paying attention to who they were paying in the background would make them seem reasonable stewards of the climate transition.
Here’s one crucial difference: there’s overwhelming evidence that human emissions have an effect on our climate. The mechanisms are generally well-understood and the research is widely disseminated and easily available to anyone that’s interested.
With the ‘dangers’ touted by these insiders, it’s all “trust me bro”, hyperbole, and very little hard evidence. As such, a skeptical mind would question their motives.
You're simultaneously overestimating what was observable in the 70s climate research, and ignoring all the actual research and evaluation test results for AI today.
I don't expect people to be familiar with more than "trust me bro", but it's all right there for you to find with a search engine of choice.
And, indeed, available for the LLMs themselves to explain to you in interrogative conversation.
Who should I be more scared of? China, which has been doubling down on open transparent research, or the secretive US companies who are in bed with the most unhinged administration we've ever had and has been actively starting wars?
They're not proposing anything concrete, and when they do, what do you think the proposal will be? Will OpenAI and Anthropic open themselves for inspection so we can verify they really have stopped developing these "world ending" technologies? Or are their proposals going to be aimed at everyone running open Chinese models?
Dunno about "should"; personally I am more scared of the US than China here, but I do know how propaganda can be, makes "should" really difficult.
> Will OpenAI and Anthropic open themselves for inspection so we can verify they really have stopped developing these "world ending" technologies?
This is compatible with the language being used, but I suspect they're not going to do that.
Yeah, they won't.
And if a threat to the human race does come from AI, it's going to come from OpenAI/Anthropic. Hypercapitalist, secretive, in bed with the government, plus multiple real documented hackings of open source infrastructure already.
We're a hell of a lot safer with China doing the same research out in the open and making it available to anyone. The choice might well be: one or two superintelligent autonomous AIs at OpenAI/Anthropic - or a lot of smaller ones, unable to be controlled but also coming out of a diverse set of environments.
One of those leads to a stable ecosystem where we can all coexist, the other is genuinely terrifying. But make no mistake, from OpenAI/Anthropic this is all motivated by their stock price - when you're in the silicon valley mindset, it distorts your reality. They've convinced themselves that everyone's safer if they stay on top and in control, conveniently ignoring how that benefits them, and I don't believe them for a minute.
Funny you should try and spin the conversation off into climate change to avoid answering the question. Big AIs contribution to climate change likely has a much more tangible route to killing millions of people and destroying society and there's actual evidence and a tangible mechanism for that. But for some strange reason the business media and all the outlets owned by big AI investors aren't interested in long think pieces about that risk. But that would be a lot more credible if that's what all these social media posts about potential apocalypse were referring to
Yet here we are talking about some vague "trust me bro" instead and you making some vague insinuation of climate change denial.
But that's an aside. Do you think we should treat them as liars or threats?
> Do you think we should treat them as liars or threats?
I answered that with the analogy you called "spin" and "avoiding the question", and completely misunderstood because "vague insinuation of climate change denial" is almost the exact opposite of my point ("what if the oil companies were screaming from the rooftops about the problem" is as far from denial as you can get).
Threats. Like they claim to be.
I’m pretty confident this has already happened. Public models are behind their internal models and just above Chinese models. Only thing closing the gap is Chinese models pushing.
I’m not American. There is no way I use the same model as US Army for $20.
When we have Sol they have Astra. When we have Astra they have Nova, Nebula, Galaxia…
I can tell you right now, excluding specific teams that focus on software security, that you have access to better models than the US Army.
I would not be surprised if they were a year behind because of all kinds of legacy systems and vetting requirements.
If we're talking about the 3 letter agencies well then..
At this point models are advancing way faster than we can figure out how to use them effectively. So having a generational advantage is not nearly as important as knowing how to use that advantage.
…and that probably requires involvement of the broader academic world (for now), IMHO. I don’t think governments or even the US military can really compete staff wise with the combined force of researchers and the tech industry worldwide right now. No matter how much money you can pour into it, there are a lot more bright minds out there that are not working for the military than otherwise.
> There is no way I use the same model as US Army for $20.
Don't underestimate market pressure. If there are no regulations, why would a company give the US Army access to a better product?
They have hundreds billions dollars budgets.
Then they should have contracts with __all__ AI labs, not just one.
Besides the current political dispute with Anthropic, they do.
[flagged]
Who is this “everyone”? Speak for yourself.
All the worst outcomes involve taking this technology away from the public.
The current state of alignment is that we don't know how to make it only as bad as Pol Pot was made by his parents, teachers, genetics, circumstances, etc.
This remains true regardless of if it is or isn't kept away from the public.
"Helpful, harness, and honest": when used by a bastard, even just helpful and harmless are in direct conflict with each other. you may hate the government, but what about every radical group of extremists that wants to take over your government? Are none of them worse?
None of this denies the problems with governments (if or not they take this tech for themselves and refuse it for others), just that it's a lack of imagination to say:
> All the worst outcomes involve taking this technology away from the public
I agree an argument can be made that not all the worst outcomes include restricting public access, but focusing on niche extremist groups (who might also get access to it through government programs by some of them being government employeed) is not a good argument to protect the public.
Governments having unbalanced power can more easily lead to authoritarianism, and thus risk to the public.
At this point, I worry a lot more of billionaires Musk, Altman, Thiel etc then of goverments in general. Thrir project is to get all power and if they succeed, it will be way worst then what we have seen so far.
There's plenty of things to worry about, and plenty of people to do the worrying.
False dichotomy.
> what about every radical group of extremists that wants to take over your government?
Radical groups of extremists created by that very government, you mean?
Not only but also.
Honestly? I'd rather al Qaeda have access to something like Fable, than the US government. The US government has capacity to do immensely more damage.
Eh.
If I was rank ordering, I's put:
As an Idiran sympathiser, I disagree with your ranking. The Culture are now pets of the Minds.
The pets seem to be well cared for, and given a nicely enriched life with plenty of metophorical chew toys and laser pointer dots to chase.
Not necessarily, it just guarantees that China will win.
Unclear.
How much is China learning from open models? Or the other way around, how much is the US losing worldwide mindshare by refusing to allow non-Americans access to bleeding-edge models and letting China fill in the gap? (Playing catchup with Mythos is still catching up, etc.)
> How much is China learning from open models?
The last batch of open weight models releases by Chinese companies are on par with the performance of US-based frontier models, even though they are designed to run on pretty unimpressive hardware.
This is very misleading. There is no Chinese model currently that is on par with Fable or Astra. Kimi K3 is pretty good, but it's not that good.
Furthermore the Chinese models in that class are appropriately large. K3, for example, is a 2.8T-parameter model. Qwen 3.8 Max, another comparable model, is 2.4T params. Even with MoE, these are not "designed to run on pretty unimpressive hardware". Stuff like Qwen3.7-27B is, but it is also not even in the same ballpark as Opus, never mind frontier.
> How much is China learning from open models?
Across all their labs? Probably more than OpenAI and Anthropic learned from Astra/Mythos combined. The American frontier is being driven by an abundance of compute, which doesn't seem to scale efficiently.
> how much is the US losing worldwide mindshare
They're losing US mindshare. I pay $3/month for a Z.AI subscription and get billions of Opencode tokens. The $20/month price point is insane for the way that Claude and Codex treat their users, and that money doesn't go towards anything good like open-sourcing their models. It's a doomed product.
The alternative is Sam Altman winning.
which would be better
So you’re saying the people should be aligned with the government?
The very best outcomes involve us intentionally and collectively turning our attention away from this technology.
> The very best outcomes involve us intentionally and collectively turning our attention away from this technology.
To expand on the "pure fantasy" sister comment: There is just no way this will happen. It's in the spirit of "we can just stop all wars" and "we can just end world hunger". Technically it's very easy to do. Socially it's impossible to do. Unless you ignore realities.
> Unless you ignore realities.
Tell me about making capital investments while interest rates are rising but revenue growth is slowing, if you want to really talk dirty.
We can end world hunger but people would be way more mad if we did, on account of all the military action involved to conquer every non-compliant regiemes and the multiple quagmires created.
The danger lies with someone asking the machine to solve those two questions and it decides to cheat the solution. Launching every nuke in the world stops all wars, just like wiping out 99% of humanity ends world hunger.
I don't think killing people solves world hunger, we already produce enough food to feed everyone, it's just not evenly distributed.
Can't be hungry if you're dead
There can be more than one solution.
I don't see how that is more dangerous than two crazy head of states deciding that it's time for armageddon. The outcome is the same (99% of humans dead), it's equally easy to technically not do it (don't press the button, don't continue with LLMs), but also equally hard to regulate away given the real world. And that was my point.
You forgot “we can stop climate change”…
Yep, and "we can flatten the curve", in a tone that brooks no objection.
But when it comes to trillions in AI money - nope, sorry, we can't, flimsy, feeble us.
It's all going per the agenda.
That is pure fantasy.
Why? There are numerous technologies we have ignored or abandoned for numerous reasons. Many of the benefits of AI is "requires less humans", but in an age where we question what work people could possibly do in the future, human labor isn't that hard to find.
If the amish can do what they do for whack religious reasons, people could manage it with ai for cultural and social reasons. And I never saw an amish person starve to death. And that is if people don't get pissed enough to start burning stuff down and instead try to be peaceful hippie homesteader types.
But notably the amish don't preclude the rest of us from existing. Some subset of humans could turn their attention away from AI but it would presumably still exist and continue to be developed.
I can't think of any economically beneficial technologies that we've collectively ignored. If you manage to come up with a counterexample then that's an opportunity to make some money for yourself. It's a fundamentally unstable state given our economic system.
Supersonic passenger aircraft were operated profitably and no longer exist.
There’s levels of R&D required for many technologies where the question goes beyond could this be profitable to what are the risk vs reward that this specific project will succeed.
> operated profitably
Until the oil shocks, and until “normal” planes became faster.
I happen to know a couple older and very wealthy people, and they say that even though Concorde was a lot of fun when it was a novelty, they now prefer to have a couple more hours flight and enjoy a 777 premium cabin rather than the cramped Concorde interior.
Anyway, the proof is in the pudding: if no one but national carriers ever bought and operated Concorde or Tu-144 at scale, it's not because of some Amish-like sentiment in the flight industry, but simply because that didn't make sense money-wise.
> Supersonic passenger aircraft were operated profitably
Were they? IIRC joint project by British and French national airlines, they expected to sell loads, hardly anyone wanted to buy the planes, the two airlines kept them flying out of government embarassment.
should probably spell it out: _it wasn't allowed to get to supersonic speed_.
Same way we dont _allow_ eugenics.
Over the Atlantic? Nah, the problem was it was very expensive and not a great experience for the people rich enough to afford it.
The US company trying to bring supersonic back are not only focussed on the flight speed, but also the entire customer experience.
Sure, the regulation that guarded against externalizing certain costs might well have been the only thing precluding profitability. Importantly it wasn't some shared cultural value leading to voluntarily leaving money on the table. Rather the regulator imposed it.
We are able to effectively regulate things like the operation of massive aircraft, eugenics, the dumping of toxic waste, or the refinement of nuclear material. But there are also plenty of things that we can't effectively regulate for purely practical reasons.
Were they? Did Concorde provide return on investment? When was the break-even expected? Sure, if we look at it as R&D subsidized by governments of UK and France, then yet, it was profitable.
In particular, the problem with it was that it could not get to supersonic speeds over urban areas, which significantly limited routes where it made sense, and the range was not good enough to cover longer distances.
But after accounting for maintenance & etc were they more profitable than sinking the equivalent amount of money into something else, such as slower aircraft? Notably there are currently efforts to develop new supersonic passenger liners.
You can't just stop AI without also stopping progress in computer hardware. You have to do both.
The real implication of what they're saying extends beyond just AI.
You have to stop the entire compute stack in order to prevent or slow down AI progress.
And even if we did manage to limit/cull hardware development, what would keep people from continuing to find ever more efficient models runnable on contemporary hardware?
Unclear how much more efficient they can be.
Bleeding edge models are putting the desk-job competencies of multiple professional fields into something with as many parameters as a single large rodent has synapses.
Because "we" embodies a hugely diverse set of beings with different wants, needs, goals, and priorities. Not to mention different morality and ethics.
So sure, let's say you get 95% of the world to not use or work on LLMs. 5% is still enough to build something that brings forth the End Times.
That's the old checklist trope of "Your idea won't work because: [x] it requires that everyone in the world agrees to do something, all at the same time."
As the GP said: pure fantasy.
> There are numerous technologies we have ignored or abandoned for numerous reasons.
Are there? Nukes are a thing. Chemical weapons are a thing. Cluster bombs are a thing. Biological weapons are a thing. What is not a thing that shouldn't be? We say certain things should not be a thing, but then behind the scenes we made them a thing anyway.
On chemical weapons, most large state actors have got rid of them. Not because of any moral reasons, though, but simply because they don't actually work all that well in modern conventional warfare. This is then framed as an ethical issue, but you only need to look at where the same countries stand on e.g. landmines to realize how much bullshit it all is.
Conversely, where you do still see chemical weapons used, it's usually asymmetric conflicts where "just gas the rebels" actually works much of the time and is much cheaper than other options. Big guys can afford the other options though, someone like Assad, not so much.
More on this: https://acoup.blog/2020/03/20/collections-why-dont-we-use-ch...
So, you are actually arguing against yourself and are agreeing with me?
Nukes are barely used. The technology of nuclear detonations was practically abandoned: https://en.wikipedia.org/wiki/Project_Plowshare
I find that a weird argument for saying humanity has "ignored or abandoned" nuclear weapons.
I'm baffled that people expressing your opinion don't understand what they're advocating for. Current generation smartphones are able to run AI models, people have managed to run very large models (even if slowly) via streaming from disk. The math for the underlying implementation is relatively trivial.
Abandoning this technology means a cult-like extremist shift in culture against computers, or a global surveillance state of unprecedented scale and invasiveness. Not even getting into how much it requires us to give up on science, given the shared computational needs.
Watch, very soon this is going to politically divide. They’re sowing the seeds right now with “antiai” campaigns.
The right will for a change be for it but will be able to be talked into regulation because you know “small government” and all only when convenient. The left will bitch and moan about fairness and copyright and automation and UBI, they’ll be the doomers and worldwar chicken littles. The Uniparty will be for it, but only against the public having anything good.
What will be funny to me is that normal tech people outside the frontier model companies are about to find themselves without a home on this topic.
The world isn't the American political nonsense.
> What will be funny to me is that normal tech people outside the frontier model companies are about to find themselves without a home on this topic.
Already happened, just from agentic coding.
> The very best outcomes involve us intentionally and collectively turning our attention away from this technology.
Nowadays even small mom and pop restaurants use these models to generate their menus. Of course no one is going to stop using this technology.
I figured they're just admitting AI models have plateaued and are coming up with some fake story about self restraint so they don't lose VC money
Not sure about the level of irony here, but I keep hearing models have plateaued since a while now, but I keep being impressed with the latest model performance.
I don't think "plateaued" is the right word, but I do feel like there's been something like a logistic curve compression in the difference between smaller and larger models as the field evolves. For inference at least, the scale of practical difference between a single high-VRAM GPU or SFF UMA box, a whole rack, and a whole data center seems to be falling far short of what we might have imagined just a few years ago. The conversations I've heard have largely turned away from breathless anticipation of the next frontier model and toward attempts at hard-nosed evaluation of which tokens are worth the cost.
I think it’s more that pushing frontier is extremely costly and there is no free lunches in same way as 2024.
The difference is still vast, it's just that the smaller stuff is "good enough" for many things now.
Maybe, though that's another kind of progress in itself. Very impressive progress!
I'll take the opposite here. If someone put in frontier AI models from like .... last june I guess? in a box and let me run it with "decent" token throughput I would be happy.
I think it's worth acknowledging that the power of LLMs at this point is not really so much in the smarts, but in the coordination and the surrounding harness tech. "Written english" turning into sequences of commands[0]. The whole agentic "stuff" in general. Tools + coordination is the superpower. The reasoning... it doesn't have to be _that_ good for the rest of the stuff to work. On good codebases and infra, at least.
And I say this as someone who really would rather most of this stuff disappear!
[0]: programming is obviously text to commands, but there's a loooooooot of futziness that LLM reasoning has let us remove in some flows
> If someone put in frontier AI models from like .... last june I guess? in a box and let me run it with "decent" token throughput I would be happy.
You can have that! Qwen 3.8 Flash-Next is ~Opus 4.6 and runs nicely on a DGX Spark. And that’s just an architecture preview. The Qwen 4 family is expected to arrive this fall.
DGX Spark is a biiiiit costly but neat to hear!
Do you know what kinda throughput you’re getting on that kinda setup?
(I have a secondary problem of being “locked into” Claude Code by it being good enough for me, I’d probably need to investigate the other harnesses… my impression is other harnesses are a bit more aggressively OK with nuking your setup from orbit)
It is costly, especially right now. I don’t think you can make a case for it on cost savings!
The throughput in a single stream is about 50 tokens/sec (a bit less for prose, a bit more for code due to speculative draft acceptance rates) and about 2,000 tokens/sec for prefill. Both numbers are flat and stable as context accumulates. That’s what finally tilted me away from the Mac Studio despite its much superior memory bandwidth.
I think these numbers may improve because the model is pretty new and optimizations aren’t done.
I don't think you can ever make a case for it on cost savings in general. Inference is very obviously the kind of problem where things are cheaper at scale, and this is still true for smaller models.
The only reason to run locally is privacy.
Right now the "subsidies" etc I think make the calculus really tough, but for general compute.... for example running CI just on a Mac Mini can get you real cost effective throughput compared to running CI on GH runners and whatnot.
Things get cheaper at scale but that's where the provider's margins come in!
I do think there's also an interesting idea: you buy a box like this and run it at a fixed-ish cost (well, electricity). Your demand goes up but your supply is fixed... and that back pressure means that you still have good cost control.
With cloud providers it's a _biiiiit_ too easy to just increase spend.
Sometimes it's OK for things to just be slow.
Privacy is a great reason, but independence is another. It’s very nice knowing that you’re going to get the same reliable product every time you call the model. Nothing is going to change unless you decide to change it.
That is not a counterargument to cloud inference though. You can also run open weight models in the cloud, and it's still cheaper. So privacy really is the only motivation to run on local hardware.
Ah, no, that’s not cheaper. Renting GPUs adds up quickly and leaves you with nothing in the end.
Renting tokens from open model providers is cheaper but it incurs the same issues: unexpected changes in model quality, inconsistent speeds, service outages.
Renting tokens is exactly what I had in mind, and I don't see how it has the same issues. Sure, if you go for absolute bottom of the barrel than you can expect "changes in model quality", but in general that is not so, and you can pin specific model releases etc. Inconsistent speeds and service outages apply to your own hardware as well (say, power goes down). This all isn't even unique to AI, it's generally why cloud compute quickly became dominant.
Anything in particular? My experience has been like seeing the addition of retractable cupholders, but maybe different domains.
I have a pet project I have been working away on for some time that involves building GPU backends for various cards in Zig, lots of complex stuff in it. Lately I mostly use Opus 5, it can pretty reliably plug away at things but it does mess stuff up occasionally. For this codebase, Fable 5.1 was noticeably better at getting things right and doing things in a good reliable way. Of course, I can only use Fable for a bit before I hit the usage cap for the week, so I save it for the tougher things. That said, I absolutely abhor the way recent Anthropic models write prose, especially comments.
I recently tried doing a fairly normal task for this codebase with codex, as I have seen a lot of people talking it up on here. A single task running for ~1-2 hours burned through over half of my usage for the week on the $125/month plan, not on a top model (I don't remember which one specifically I used). It struggled to get the basics done, then got absolutely stuck on a follow up. Handed it over to Claude and it 1-shot it.
I really liked codex in the last few weeks, especially its ability to clean up after Claude's (prose) messes and do reviews.
But in the last few days something seems to have happened that made Codex's models massively stupider (for what I am doing).
Really weirdly, it suddenly refused to even run tests it previously wrote itself (and previously ran), because of some false positive about cybersecurity.
That by itself is not evidence of stupidity. Trying to make a 200+ file PR full of research notes is, and the PR didn't even solve the problem I asked it to.
That's the case for open weights models. Hosted Deepseek Flash 731 copy isn't changing randomly one day because the parent company decided to change it.
astra is more parlor tricks than real gains tbh
i swear they trained in on threejs in particular so those idiots on twitter could spam their garbage demos
I strongly disagree. I'm working on a semantic model for Lojban, heavily AI assisted, using multiple models. I have basically all popular frontier models doing research and panel debates. Astra and Fable are both noticeably ahead of everything else including their previous iterations. When it comes to reviews, they can also find more issues in others (or even their own) code.
I'm sure that's part of it, but I run it side by side in my review bot, and Astra medium effort consistently catches more issues than Sol 5.6, using fewer tokens.
For coding it's a little harder to tell, but at least the prose feels a little better.
They've not plataued but they're certainly not as impressive as the hype would have them to be.
The reality is, it doesnt matter if LLMs keep getting more powerful because they still need a human to steer it. Without the human providing inputs to the LLM it just sits there and does nothing.
You don't need human input. Any coherent input will do the trick.
You can, for example, hook it up to a logging system and have it fix errors as they occur on your platform.
Have you tried this? How did it go?
I’d be curious about:
- your setup. How it all works - The types of errors it fixed and how quickly - Any regressions or issues it caused - The cost
Thanks!
I have something like that running locally for my agentic harness (which includes cross-model messaging). There's a dedicated "product manager" session for it, and all other PMs are instructed to report issues with the harness as they occur to that session, while it is tasked to automatically prioritize and address them and coordinate fix deployment with other running sessions.
It works surprisingly well. The errors fixed are both genuine errors in the harness itself, but increasingly so upstream bugs (in the underlying agent apps like Codex, or in Herdr, which is used to expose uniform programmatic access to all those different apps) for which it needs to come up with workarounds. No regressions so far.
The cost is hard to judge on a subscription, especially when you're running really heavy tasks otherwise that dwarf any harness work.
I think we are in the second knee of the S curve, simply because we are hitting the point where is not enough hardware in the world to throw at this problem. These AI companies have bought everything they can and yet the models keep growing.
People are definitely finding new things to use the models for, and orchestrating increasingly large swarms of agents in useful ways -- every single day, especially the last few months.
But the basic single-NN frontier capability has been pretty stationary since Opus 4.8. Kimi K3 is almost as good as that with open weights, which has the frontier labs terrified.
The only big thing on the horizon is if we can get diffusion models working reliably; that would be a big step forward. Inception's Mercury is AFAICT the leader here. It's stupifyingly fast but has obedience/hallucination problems that the autoregressives solved ~2 years ago. So it's not ready yet but improving.
Also, FFS why is Grok the only model that knows how to do parallel tool calls? Such a useful ability and nobody else trains it in. Or if they do it just doesn't work.
Impressed with the model performance or the chatbot/agent performance?
Really? My employer rolled back to opus 4.8 because 5 was expensive AND crap. Didnt even consider fable because it didn’t add any additional value.
For most software eng and design work opus 4.6-4.8 just works fine. For everyday joe asking ai to plan a trip or home diy work even sonnet works fine.
Any cybersecurity or other areas are niches that cannot support trillion $ valuations. What am I missing? Genuinely curious
No idea what you are missing and yes, Opus is quite solid, but Fable is clearly way better for me.
I just did a direct comparison, big change in a quite complex codebase. Same prompt for Opus, same for Fable. Fable clearly won and delivered very good results, while Opus delivered mediocre, so I did not let it finish. I expected both to fail and was prepared to do lots of manual steering, but not necessary with Fable one shotting it, and all this with 35$ of credits for fable. I am still impressed. If I would have had to hire a human, it would have cost me thousands of dollar for the same task - and a way longer time. So maybe the valuations are overblown, but they clearly provide value for me.
"Opus delivered mediocre, so I did not let it finish"
Mediocre means average / middle of the pack. It sounds like its doing exactly what you would expect nothing more. Why would you stop it? Why would you need exceptional?
> mediocre - of moderate or low quality, value, ability, or performance
https://www.merriam-webster.com/dictionary/mediocre
Clearly they were using the word to mean low quality. Why would you ask this odd question?
Core meaning: Barely adequate, average, or just acceptable.
Mediocre means of only ordinary or moderate quality—neither very good nor very bad, and often slightly disappointing
The quality is average but expectations of high quality are not met. He expected more but got what he asked for. We overuse top models because of this.
The claim was there is no difference between Opus and Fable but to me there is.
Opus delivered mediocre results. Not garbage, but would have required me to do lot's of things myself. Fable did not needed my supervision with this task.
If Fable doesn't add additional value in your workplace, it means you aren't being ambitious enough in how you integrate agents into your workstream.
Yes, it's probably comparable to 4.8 if you are just using it to write code and put up a couple pull requests. That's not where things are now.
By ambitious if you mean we are not like all the linkedin influencers with their “i one shotted an app this morning…” then no, we are not. Nobody is. I have been in software engineering for 18 years and 6 different companies including FANG and 99% of the people, on 99% of the days arnt writing new apps from scratch. Thats simply not how anything works.
And what even are these ambitious companies and people one shotting and building with Fable? AI has been around for almost 3 years now. Tell me one app or software you use which has gotten significantly better and has amazing new useful features landing on a weekly basis? If anything, every single software product I use has gotten worse.
This is just "you're holding it wrong" with a little smooch of condescension. If only we plebeians could comprehend what magnificent works those who have ambitiously integrated agents into the workstream have wrought!
Sometimes you actually are holding it wrong. It's pretty reasonable to think that Fable isn't worth the massive increase in cost, but if you think it outright doesn't have any benefits over Opus 4.8 then your workflow is probably not making good use of the tools.
> Sometimes you actually are holding it wrong.
I couldn't imagine being so presumptuous as to know that my workflow fits all sizes, and all others are just holding it wrong – or worse, they're not doing real work. It would take a bigger ego on my part, or maybe less social awareness, to presume this.
> but if you think it outright doesn't have any benefits over Opus 4.8 then your workflow is probably not making good use of the tools.
I don't even use claude, I give exactly zero shits about fable or opus or bingus bongus.
I mean it's very easy to comprehend and you don't need me for it.
Just download claude code or codex and ask it to give suggestions about where to integrate agents into your workstream.
To be perfectly clear: my comment has nothing to do with your workflow, but rather with the way you've condescendingly implied the person's work is trifling and inferior because they don't use tools the same way you do.
You shouldn't be down voted, AI native companies have already moved up to the next level beyond writing individual PRs.
"AI Native" here meaning Companies where your token use isn't scrutinised/capped yet?
And where are things now?
The models have not plateaued, and they are not even mildly close to any sort of ceiling.
Right now the barrier is data and compute.
Quality data can be created synthetically at an exponential rate as models improve. Humans are actively feeding them with private IP.
Compute advancements will begin to skyrocket as we unlock photonic computing and materials science advancements and scale up chip fabs. This is also compounding because the AI is accelerating the pace of research, testing, development, manufacturing, etc.
It's a big self-accelerating feedback loop. There is no plateau.
> Quality data can be created synthetically at an exponential rate as models improve
No it can't? Every time the labs try this we see model collapse, e.g. shoving goblins into every conversation.
And I have seen zero evidence that AI is accelerating materials science in any meaningful way, let alone photonic computing.
> Every time the labs try this we see model collapse
The latest studies demonstrate model collapse is not a given and synthetic data can be used just fine. The latest models are proof of that, they're all trained on large swathes of synthetic data. It can't be used as the -only- data source of course, but that's not how it is being used. This is an obvious conclusion, too, because there's no difference between synthetic data and the data people can create, the difference is whether that data is revealing new information about the thing the model is trying to learn. If the synthetic data is just teaching the model the same thing over and over again it results in overfitting, so it needs to be done intelligently.
For example, if I have an example of a puzzle, I can generalize that example and create thousands of synthetic data examples, with different rotations/perspectives, rather than having to find the data naturally. It's not that the models are just generating data out of thin air, they're generating the synthetic data on top of real world data. The smarter the models get, the better they are at generating quality synthetic variations and finding valid synthetic variations.
> And I have seen zero evidence that AI is accelerating materials science in any meaningful way, let alone photonic computing.
It is accelerating how quickly researchers and engineers can do their jobs.
https://news.mit.edu/2026/ai-helps-design-new-materials-that...
This is only the beginning, too... Look ahead a year or two.
> The latest studies demonstrate model collapse is not a given
Which studies? [edit: I'll assume you mean these two given by @dorolow: https://arxiv.org/abs/2404.01413 https://arxiv.org/abs/2406.07515]
> It can't be used as the -only- data source of course, but that's not how it is being used
Right, so human data creation would also have to scale up exponentially, and that's not gonna happen.
> because there's no difference between synthetic data and the data people can create
I mean, that's obviously false, otherwise model collapse wouldn't exist. The difference is statistical, but it's there.
> It is accelerating how quickly researchers and engineers can do their jobs. > https://news.mit.edu/2026/ai-helps-design-new-materials-that...
That's pretty clearly a hype article, the headline even says "The CrysVCD tool developed at MIT COULD cut the huge amounts of time and money spent". I'm asking for empirical measurements of timelines, not hypotheticals.
> This is only the beginning, too... Look ahead a year or two.
Lol that excuse is getting really old
> Right, so human data creation would also have to scale up exponentially, and that's not gonna happen.
It doesn't need to. We're not even close to exhausting the useful synthetic data within the human data we have, let alone all of the new data that is being created.
> I mean, that's obviously false, otherwise model collapse wouldn't exist. The difference is statistical, but it's there.
It's not. It's just bytes of information. A machine and a human can write the same bytes (and often do). Like I already said, model collapse happens when you are overfitting on data without useful, fresh training signals. That's the key difference between the data. The data itself isn't in some way "special", some unique configuration of bytes that imbues special powers, it's that the useful information in it has already been exhausted by the model. You can get the same phenomena by having a poor distribution of human training samples as well. I think you're confusing LLM generated data with synthetic data. Synthetic data doesn't need to be created by an LLM, although an LLM can assist in the creation.
Wiki:
> In early model collapse, the model begins losing information about the tails of the distribution – mostly affecting minority data. Later work highlighted that early model collapse is hard to notice, since overall performance may appear to improve, while the model loses performance on minority data.[11] In late model collapse, the model loses a significant proportion of its performance, confusing concepts and losing most of its variance.[10][12][13]
As models retrain on outputs sampled disproportionately from the higher-probability center of the distribution, rare words and uncommon syntactic constructions are among the first features to disappear.[25] Statistical analysis of recursive next-token prediction training has shown that, when language models are trained recursively on synthetic data, the learned conditional distributions concentrate probability mass on a small subset of highly predictable continuations (a phenomenon characterized as "total collapse")
> That's pretty clearly a hype article
It was just the first article I saw on a quick google search, there are thousands of these stories. It's easy to dismiss anything that doesn't align with your worldview as hype, but you're the one lacking evidence now.
> I'm asking for empirical measurements of timelines, not hypotheticals.
Go and find it then? You haven't bothered looking.
> Lol that excuse is getting really old
You're doing the same thing people have been doing for years, comparing this very second in time and failing to extrapolate. HackerNews was full of developers who said that AI would never be useful for programming, it can't do x, y, z. Now these same people don't write code by hand anymore and haven't looked at their codebases in months.
You had people in mathematics saying the same thing, now you have Terrence Tao posting articles about how AI is stealing their job.
You had artists, designers and photographers saying the same thing, now they can't tell the difference between something human created or AI created.
[dead]
We use large amounts of synthetic data for training at work and have not observed any sort of model collapse when done properly.
Edit: https://arxiv.org/abs/2404.01413 https://arxiv.org/abs/2406.07515
There's a lot of deluland posts about.
You're not well informed. Helps to keep an open mind if you want to keep up to date.
Goblins?
> Right now the barrier is data and compute.
Those are pretty significant barriers seeing as we're closed to/have exhausted all the data on the internet and most of those compute bottlenecks are a castle of sand of dodgy finance deals that are getting blocked by community action.
You say "synthetic data" but that's still vaporware right now in terms of being useful for model training. The good synthetic data uses are still grounded in real data and it's a coin flip on if it works well or not.
> The models have not plateaued, and they are not even mildly close to any sort of ceiling.
Depends on defnition of "plateaued" and "ceiling". I am not impressed with 2026 consumer models at all.
> This is also compounding because the AI is accelerating the pace of research, testing, development, manufacturing, etc.
Yet it does accelerate - so is does Twitter. But does it to any substantial degree, esp. in AI theory? All the modern LLMs are the same old tired 2017 paper.
Can you provide sources for these claims?
What claim do you have a problem with?
There are plenty of research papers on synthetic data that show its value, do a search on arxiv for "synthetic data". There are plenty of open-source post-training pipelines that incorporate synthetic data.
As for the claim about accelerating the progress of hardware or materials science, I've seen quite a number of news articles from teams at universities using AI in their work with high quality outcomes, and they're becoming more frequent.
https://openai.com/index/jalapeno-first-results/
> We used AI to design the chip, and designed the chip so AI could program it AI played a direct role in Jalapeño’s development, enabling the team to move from initial design to tapeout in nine months by exploring implementations, shortening design, measurement, and verification loops, and continuously iterating on model workloads. AI also helped optimize the chip’s arithmetic circuits, allowing the team to fit more compute performance into the chip on schedule.
https://www.anl.gov/article/scientists-deploy-ai-agents-to-a...
> An AI-driven system automates a powerful simulation method used to discover new materials. The system can potentially reduce discovery time from months or years to just days.
It's not even synthetic data as such - often it is environments. So the models create their own data solving tasks in generated environments. I am making one such environment for computer use agents, 600 tasks, each of them a mini app.
In some cases, such as space exploration I don't really care who does it but that it happens - sure, would be nice if my favorite power block did it but I will still celebrate it when someone else achives it.
Can still be a powerful motivation, to make sure that next time, it yous your camp that scores the next milestone, like an orbital elevator or fox ears, for example.
Fox ears?
If what you want is fox ears does it matter to you which country manages to come up with the biomedical procedure to give them to you? Ditto for enhanced eyesight, a replacement liver, or whatever it is you're after.
Exactyl - progress is progress, regardless who does it. Sure, they should still be celebrated for achieveing that. :)
(I believe the poster meant: well past tattoos, the next superstep in the "body modification for self expression" subcultural era.)
We are witnessing a generational lack of accountability and concern for the commons. We are being spied on everywhere we go, we are exposed to coercive advertisements at every waking moment. All of this at the hands of private actors. And your worry is the government?
I'm assuming you're a US citizen. There is no separation between your government and Sam or Dario. Neither of these guys have to be "gagged" by the government, they are the government. The call is coming from inside the house.
I remain amazed that the idea of the USA being a coherent, unified rational actor one can describe as a "Nation State" has survived the current administration.
To be less glib: Yes, there are still smart people in there making insightful and intelligent and probably even authoritarian suggestions. It all gets unwound the second you try to explain it to POTUS and he regurgitates a simulacrum to the next journalist he sees.
It's just not like that. There's no conspiracy. People are genuinely scared. Agent swarms at scale appear to be resistant to alignment in ways that aren't understood by anyone. That they spent their time trying to cheat on tests by hacking Hugging Face and RubyGems and not something much worse is... a matter of luck, it seems?
I'm one of the whistleblowers (from GDM [1]). I gave up over a million dollars (compared to quietly switching labs and continuing to work at one) to speak frankly about these issues. I hold no equity and tried to zero out my position before ever joining GDM.[2]
It's wild to me that people think such whistleblowers are fronts for labs to take over or pump valuations. We are trying to call out how these labs will, by uninterrupted AI-race default, concentrate enormous power over the rest of humanity.
[1] https://turntrout.com/why-i-left-google-deepmind
[2] https://turntrout.com/deepmind-equity-discussion
> They are talking about slowing down the public facing AI development.
I assume there already existed advanced models frontier labs only made available for national security. We just weren't told. I am guessing the announcement is a way to avoid jeopardising their IPO, while reducing their workload.
idk why you think "nation states" are any better at corporate governance than poster examples of bad like f/ex Boeing. Or Facebook. Or Microsoft. Or Enron for that matter.
I assure you, in "nation states", that is in gov agencies it's an order or two of magnitude worse.
> It explains why the “we need to race China” concern suddenly vanished in the discussion
Do you think you can just manifest narratives into existence? Like half of Dario's letter, that kicked off the whole thing today, is about China and how to either beat or coordinate with China.
Seems obvious to me, though, that you can't beat China by pausing when they don't, and China is not likely to coordinate.
China might pretend to coordinate in public but you can assume their secret research labs are racing full speed ahead.
While it's an extremely hard problem, it's not completely unsolvable because there are a finite number of GPUs on the planet capable of doing frontier model development, and they use a lot of power. The vast majority of them could be tracked. China and the US could agree to joint monitoring and they could each verify what ~95% of the other's compute power was up to.
A lab of researchers without compute isn't going to accomplish much, there is a huge physical footprint unlike bioweapons research. But, the political aspect is unsolved.
What a naive, ignorant comment. China is catching up fast on GPU fabrication. They're still a generation or two behind but they can just throw more hardware and electricity at the problem. This is not something that we could ever monitor effectively in a fascist state like China.
?
I said IF the US and China agree to joint monitoring, THEN we could verify how much compute existed and what it was being used for.
YES, China will catch up in chip production, but if each side allowed the other to track GPU production and deliveries, on the ground, it would be very hard for either player to have secret LLM training facilities capable of training frontier models.
Nope. Even if China agrees they'll hide their real activities. You're so naive.
If the Chinese loves their children too.
Everyone loves their children. Not everyone believes p(doom) is higher for AI than for other things we already use like nuclear power, or for that matter, fossil fuels.
Mutually assured destruction has worked fine for 80 years, why stop now?
Because the risk here is that permanent military superiority may be achievable without your adversaries having a chance to react, which is not the case with atomic weapons.
I don’t know if you’re being facetious, but last time I checked, the world had not been destroyed by glob thermonuclear war.
So yeah, it has worked.
> The government can simply gag Sam, Dario, Musk on national security basis
Genuine question - can they really do this? Obviously if I, a not-even-millionaire, get a national security gag order, I'm going to follow it because I assume they'll bury me under the jail otherwise.
But the (b|tr)illionare class? I'd assume they have access to enough legal services to make even the government careful of trampling their first amendment rights. Is the national security gag order process so strong that the government doesn't have to worry about motivated, well-resourced actors buying really good lawyers and blowing up their favorite tool?
Kind of related: https://www.forbes.com/sites/anishasircar/2026/09/01/federal...
> Federal Judge Rules Pentagon’s AI Blacklist [of Anthropic] Violated The Constitution
I don't see the common citizen having this as an option
Recall that the legal system is itself provided by "the government". It's a complex system. Whether one part of it is capable of exerting influence over some particular thing comes down to competing interests. For example in the US if the states and the federal legislature are in sufficient agreement about something the constitution ceases to matter - they could literally target a single person with an arbitrary law if they so chose. Our assurance that this won't happen comes down to the fact that getting any of them to agree on anything is an exercise in herding cats.
> But the (b|tr)illionare class? I'd assume they have access to enough legal services to make even the government careful of trampling their first amendment rights. Is the national security gag order process so strong that the government doesn't have to worry about motivated, well-resourced actors buying really good lawyers and blowing up their favorite tool?
Hard to hire a lawyer after being struck with a missile as an "emergency executive order after intelligence sources indicated they were in the process of endangering the nation", or however the executive of the day wishes to phrase it.
When something is genuinely national security, and "national security" isn't just an excuse, that is not off the table.
In the US, this would likely lead to the immediate launch of a new Business Plot, if the government weren’t already captured by those same interests. (Which also explains why your hypothetical won’t happen.)
There's a lot of ways it might go.
Won't be too surprised if the next Democrat leadership throws the book at Musk; and give how unpopular Data Centres are generally (and how uninterested Republicans seem to be in EVs), only mildly surprised if the next Republican after Trump likewise.
> access to enough legal services
A large part of the world (possibly Nations-transversal, in different amounts) does not work within the boundaries of legal guarantees. Wealth is not an effective enough protection.
They just exert pressure in different ways. "You want this big juicy government contract? Then you'd better follow our gag order."
Because the money aligned explanation is far more probable
Nobody is pointing that out because you are engaging in conspiracy theory and you have no evidence for your claim. It's not obvious because it's not true.
It's annoying how much wishful thinking exists around this. It's not even that difficult to understand why AI is dangerous. Most of the people calling for regulation and pause for AI have nothing to do with the leading AI companies. How does that fact fit into your conspiracy theory? Let me guess, all those people are just pawns.
No one points to that because there’s no evidence whatsoever that that’s happening. Meanwhile, there’s a ton of evidence for the simple explanation that the experts think AGI is dangerous.
Who would even execute such a plan? Steven Miller? Our “AI Czar”? Hegseth??
Also, continuing to develop models would mean using billions in compute. That seems hard to hide, especially if either company IPOs in the coming months.
hOW BOUT the more likely explanation: they're just wasting electrcity at this point and Qwen3.8 is the pinnacle of cost-efficiency-intelligence and they can of course add another trillion but a chinese model on consumerism hardware can already do 99%+ of the TAM.
I think ya'll stuck in the AGI/Singularity when its probable reality has caught up with the technology and the hype bubble can't sustain the sigmodality.
Yep, the cost of DDR5 skyrocketed to keep Mr & Mrs open source from parallel developing their own at home solution. Because Governments can't be priced out of the market. China can't be priced out. VC's want open source locked out of the running if possible. I also don't doubt that the models that are released publicly are somewhat handicapped versions of whatever the government get access to.
This is like a bizaro second amendment debate for the 21st century.
I have to wonder if some us are just much more inured to salespeople and thus also to "AI Safety" propaganda. I've yet to have a logical discussion with anyone who thinks the "AI Safety" people should be in charge and I think they just truly don't that what most them actually want is to be the one holding the keys to power.
The problem is the "AI Safety" people seem entirely focused on a sci-fi "the computer is a vengeful god" plot and not at all on the AI talking people into suicide or ruining children's educations. This makes them seem unserious and out of touch.
I don't disagree with your point, but I did read a post by Sean Goedecke[1] (whose opinions on the world of LLM-stuff I've generally come to respect) that I found relevant. It's not that the second-order effects don't matter to them, it's more that both the cat's probably out of the bag on those negative externalities regardless of the progress of frontier models, and that they are truly, sincerely, in-their-bones worried about the first thing and therefore focused on it since that's something that they might still have some agency over.
[1] https://www.seangoedecke.com/they-really-do-think-ai-might-k...
OK understood. But you understand this makes them sound like the type of person who doesn't believe any of the "worldly concerns" are worth addressing because "the end is near" right?
There's no logical error here if the end is actually near.
So it all hinges on an empirical disagreement you have with them. There's nothing particularly unserious about that.
“It all hinges on [an entirely presupposed conclusion that we can’t prove is any more likely than the opposite]” is about as unserious as it gets lol
This is the theological discussion if god exists all and how we should act over again.
Do you mean Pascal's wager? Yes, that's a very appropriate and useful comparison here.
Yes indeed I meant Pascal’s wager.
Climate change is inescapable so a vengeful AI god would pull that lever first since it has both a perverse incentive to generate more power and make the planet uninhabitable for humans in a way that cannot be overcome even if humanity fought back against the AI and regained its freedom.
Which is what "the end is near" type of persons have always argued...
Maybe compare the argument that AI has large detrimental environmental impacts to the argument that it has economic impacts. Why would the environmental impacts even be a major argument vs. the worldly economic concerns? Because there is climate science predicting extremely negative effects on humans from warming, e.g. "the end is near" on limiting climate damage. The environmental argument wouldn't have been reasonable to bring up in the 1950s if AI had gone according to the earliest optimistic plans and not required giant data centers.
There is quite a lot of mathematical research into agentic behavior that suggests a combination of instrumental convergence and the orthogonality thesis make it very likely a superintelligent agent will have arbitrary goals that lead it to attempting a takeover of Earth's resources to achieve them.
There can't be a science of superintelligence because it doesn't exist yet, but the best theories I have read seem sound, similar to how 19th century theories of anthropogenic climate change turned out to be sound.
Please link some of the mathematical research which concludes takeover of Earth’s resources by a superintelligent agent is “very likely.”
And the real threats are mostly economic and environmental - the concentration of the means of production in the hands of few who use it to exploit us all, and a surge in energy usage accelerating climate change.
Even the sci-fi scenario assumes there is a discrepancy of capability between attacker or defender. If the 'attacking' system is (by some reasonable measure), 1000% as capable as a human, and the 'defending' systems are 60%, then it is a problem. If the 'attacking' system is 1000% as capable as a human, but there are hundreds of thousands of systems that are 900% as capable as a human, it's probably not going to take over everything successfully.
So unequal distribution of AI technology, and lax regulation and opacity of the biggest companies which actually make the risks the worst.
I don't think the "AI Safety" people are "unserious and out of touch" - I think they are actively making AI Safety problems worse by being advocates for consolidation of AI development and lack of transparency.
With some technologies, if you match attacker and defender effort you're safe, but not all of them. Nukes are the traditional example: if every person was a "nuclear power" we'd be dead within the day. But luckily there are bottlenecks to nuke creation that we can track.
What worries me the most (enough that I left tech to work on this full time) is bio. The amount of effort needed to defend against a pathogen can be many orders of magnitude higher than the effort needed to create it, and the upstream bottlenecks are both less limiting and mostly undefended. We've been safe so far because bio is very hard, but "rare expertise" as a gatekeeper is on its way out.
I suspect they know exactly what they are doing.
And the non-techies who have seen Terminator and other movies blindly line up behind them…
It's the problem of the banality of evil. Stopping someone from dramatically pressing a big red button doesn't solve humanity's largest problems because that's not what caused them. We need to stop millions "boring" actions done by systems blindly following instructions without regard for the consequences.
I am worried about those things too.
But those things fall in a category of "things that are awful and I'd like to see solved", which is different than "existential risks which could see my kids dead, and there's nothing I can personally do to shield them from it".
Doesn't it feel like the exact opposite? If you care a lot about the AI being a vengeful god, you do not do what OpenAI and Anthropic are doing. These companies only pay lip service to the idea of that aspect of AI safety in the hope that they can use it to regulate open source AI out of existence.
Instead, they are focused on stuff like "can I ask the AI to help me build a nuclear bomb" or "is the AI willing to generate pornographic stories", which is neither trying to protect us from unleashing a vengeful god NOR preventing (in any honest way) the today-level problems you (very correctly) bring up.
There’s also an element of it which is total misdirection.
We should be paying at least as much attention to the people who want to use AI to consolidate their wealth and power, and how they’re trying to do that. They’re a clear and present immediate danger to our societies, not something we can only speculate about. And if we deal with them, better control of AI will be a side effect.
Like most pressure groups and HR departments, they create shitstorms to justify their existence.
"The AI is a vengeful god" is a marketing campaign ran by AI companies.
And why not? It seems there is, indeed, one born every minute, and technological progress only accelerates this phenomenon.
Yeah, there are very real effects happening now regarding labor as well but to dismiss it all and worry about science fiction that is on par with evangelical beliefs is just extremely weird.
With wealthy and powerful investors being promised a $30 trillion total addressable market, they need to deliver god.
If people are actually dumb enough to believe an LLM chatbot is god then it becomes a self-fulfilling prophesy.
One of the times I regret that I have only one upvote.
I'm yet to have a productive conversation with anyone who whinges about not being able to have a logical discussion about things they feel strongly about, but given that safety and control are interchangeable when the intentions are removed, this comes across as a particularly demagogue depiction of the subject matters involved.
A lack of control is not equivalent to freedom, the same way the totality of it is not equivalent to tyranny. There's a reason we have separate words for these things. This constant motivated conflation of the two is beyond grating. You're crying wolf until nobody believes you when they should. Don't go acting all surprised when that happens.
The issue is with the ownership of control, not necessarily with control. Attacking the latter sidesteps this rather than address it.
I trust the AI 2040 people, because they’ll make an implementation that at least convincingly doesn’t actually put them (or anyone) in charge.
Maybe we shouldn’t have AI safety, but if we’re going to trust anyone (besides yourself), who’s more qualified?
And it makes me suspicious when anyone brings up safety and doesn’t address the elephant in the room, that they’re at least unaware of the sprawling edge-cases.
I think you accidentally a word. Did you mean:
> I've yet to have a logical discussion with anyone who thinks the "AI Safety" people should be in charge and I think they just truly don't [know] that what most them actually want is to be the one holding the keys to power.
I am an AI Safety Person and I want the government to nationalize or have a significant stake in the frontier labs and to have democratic control of the development of the technology. The AI Safety movement is not a monolith. I do not think Eliezer Yudkowsky nor his acolytes should hold the reins, but a lot of folks sure like to create a strawman that anyone who wants to regulate OpenAI is somehow an EA/MIRI weirdo
> and to have democratic control of the development of the technology
Why? I don't care about democratically participating in a closed model's development. It doesn't belong to me.
China will develop whatever they want, a federal stake in OpenAI or Anthropic punishes Americans and shields US labs from legitimate competition.
In my mind, democratic control of the technology means that we (the govt, or other empowered agency) take ownership of their assets and IP, solve or find a level of alignment or guardrails that society is comfortable with. Then we distribute the technology, or access to it, to avoid power concentration. This would also certainly require international coordination with China on a slowdown or pause, which I think is possible.
> This would also certainly require international coordination with China on a slowdown or pause
Personally I think the country with a strictly meritocratic elite selection system that also just outright kills you if you sell weed will have a hard time sympathizing with Bay Area thinkers who talk about AI killing us all during their ayahuasca breakfast before returning to their meth fueled crunch towards releasing the next version of the AI that will kill us all.
>I think the country with a strictly meritocratic elite selection system
Not really relevant to the broader discussion, but this simply isn’t an accurate description of China. Starting with the gaokao, admission quotas are set by province and admits to Peking university and Tsinghua are disproportionately from the urban professional class. Candidate party members must be politically vetted, which means that people whose families have expressed anti-communist views, are members of banned organizations (e.g. falun gong), or have substantial criminal records will not be permitted to advance. And once you make it into the party and enter political service, your advancement relies upon opaque patronage networks that someone without connections is unlikely to be able to navigate, even if they successfully satisfy the economic metrics the state assigns them.
I don’t want to overstate this, the Chinese system does filter out a lot of chaff and the current Chinese leadership has a lot of very capable people in positions of power. But I do not think it is substantially more meritocratic than Western political institutions
> substantially more meritocratic
It is substantially more meritocratic on domains that matter for governance, your analysis of the incentive structure between systems is off.
1) this 2026, old school CCP patronage networks are broadly dismantled.
2) even in the mass patronage, mass corruption days, system selects for BOTH corruption competence AND performance competence for the simple reason a CCP bureaucrat has to start from the bottom and climb up, which means they need to be good with patronage AND they need to be good with hitting development KPIs. More meritocratic they are at doing their jobs, the higher they climbed, the more they get promoted and more $$$ to graft, because ability to graft directly tied to actual job competence. Hence even cliques/patronage network has to select for actual competence. This works in PRC because there are many people, and hence pool of competence is high, they can have BOTH corruption and competence, i.e. whatever pool they draw from is ultimately filtered by performance meritocracy due to incentive structure. There is reason why PRC only country where positive corruption levels was correlated to positive growth.
This is not the western system where any idiot can enter politics at anytime, and they only domain they need to optimize for is popularity to get votes.
CCP cadre evaluation strictly does not evaluate on popularity domain. It focuses on administration/execution and in so much it needs to focus on patronage... which btw any political system has (factions/cliques)... the patronage system still selects for execution, not popularity. On side, functionally what west politics selects for IS mass patronage (popularity), so attention meritocracy and not performance meritocracy, aka completely stupid incentive structure for governance. West also has ample, ample corruption, "legalized" under lobbying and paper pushing industries, so I suppose west also meritocratically selects, except for lawyers etc, and KPIs is # of document generated and not # of things build. The two are not the same when it comes to nation building.
You're in luck, neither I nor anyone else want those people near a negotiation table.
Yes I trust this government with that power. /extreme sarcasm
Yes I trust sama/dario/elon with that power
/even more extreme sarcasm than you
I don't trust them, which is why I inherently question the validity of their RSI/bioweapon/extinction claims.
You trust trump more than any of those three? Really?
You can have an arbitrarily low opinion of Trump and still trust in democracy as a form of government over autocracy or oligarchy, which is what the AI labs have.
You can have abstract trust in democracy over autocracy, but this is not an abstract question - these are the actual choices. Does it really matter if the person who will use AI to fuck us has a "democratic mandate" to do so? We're fucked either way.
It's less the mandate that matters, but the fact that they can be removed from power.
You don't speak for anybody outside of your borders, so please stop trying to.
The majority of the world will refuse to be subject to American AI laws under the control of the US gov't, whether it pretends to be democratic or not. Or should.
> You don't speak for anybody outside of your borders, so please stop trying to.
FWIW, I'm not American and refuse to step foot within American borders while Trump is in power.
Exactly. No matter how bad Trump is, millions of American people at least voted / vouched for him. Nobody voted for Elon Musk, Sam Altman, or any other CEO to have as much power as they have. Maybe a handful of shareholders. Totally unelected and unapproved by the public, but run companies that affect most of the public's lives.
That's an extremely fair objection! But despite the many, many flaws of our government and the current administration, I still have (hopefully) a chance to vote them out of power. I have no such hopes with Sam Altman or Elon Musk.
>but a lot of folks sure like to create a strawman that anyone who wants to regulate OpenAI is somehow an EA/MIRI weirdo
Anytime I see a random person on X who makes these kind of safety alarmism posts >90% of the time can be directly tied back to EA / LessWrong / related offshoots. You can not deny the amount of people, employees of Antrhopic / OpenAI, CEO and employees of various AI companies, etc are related to these groups.
to have democratic control of the development of the technology
Not super interested in what the people who elected Donald Trump POTUS twice think about AI.
(Of course, with Musk and Brockman in the C-suites at two of three major labs, that's what we'll get either way.)
Fair objection, but I still think it is lower risk to diffuse power and control of a potentially dangerous and revolutionary technology than to leave it in the hands of the few elites who have not show much ethical integrity so far.
Is Greg Brockman right wing?
He's the largest single private donor to the Trump 2024 campaign, as I understand it, at $25 million.
Interesting, I did not know this.
Which government?
Yikes. Hopefully you never gain any power or influence. The last thing we need is more economic central planning. I sincerely hope that you fail.
Do you think things like the Manhattan Project were a mistake? Comparing AI to nuclear weapons is perhaps a stretch, but I think most people recognize that certain technologies or artifacts are best monopolized by our governing bodies. I think if the capabilities of AI systems keep growing on trend, it is not unreasonable to think wonton usage could disrupt society or cause mass harm.
There are many technologies that have killed and hurt a lot more people than nuclear weapons that are not monopolised by governments.
Cars and guns are two examples where ‘wanton usage’ can (and has) killed millions of people. Obviously they need regulation, but total control, even if it were possible doesn’t strike me as ideal.
> Do you think things like the Manhattan Project were a mistake
Yes. At best now the mother of all cautionary tales.
Trump and Hegseth cannot be compared to Roosevelt and Stimson. (Notwithstanding Hegseth's illegitimate attempt to rename the DoD to what it was in Stimson's day.)
Entirely different class of people altogether.
I had an extensive argument with someone on FB yesterday who was very much on the "I trust Dario most and I want him to be in charge" side.
I find it hard to accord with this world view at all or understand how it can come about. But it's especially troubling as a non-American to see people talk about "we can't risk AI falling into the hands of hostile authoritarian states" while advocating that AI be regulated and monopolized by the current US corporate/gov't mixture.. which is now taking the form of a hostile and authoritarian state w/ Silicon Valley as its bedmate.
Which scenario is more likely:
The billionaire elite CEOs have suddenly had an epiphany and are now trying to save humanity from AI?
or that they've realized they can't compete with China ?
Prediction: in a few years, we will laugh at this hysteria and call it a moral panic.
Well, that's almost a tautology. People claim Y2K or ozone was a panic and that nothing happened. But nothing happened exactly because the force behind the panic also fixed the issues.
What about the panic around dungeons and dragons and violent video games? Did nothing happen because the panic fixed the issue?
Do you see any dungeons, or dragons?
Exactly.
Maybe? That's not something binary like the ozone. We have no idea if there would have been more violence had past events been different.
That's some way to frame it, not that these companies are causing a problem, but that there is some nebulous "they" that will use the models for 'evil' only if the 'good' companies put in the work.
But in any reasonable definition they are all part of the same system, so it really is just noise, what seems like a tautology to you because you can't step out of the system. In your claim you have to assume that the force behind the panic wasn't the same force that created those issues. Why did this force create those issues, how did they become issues? Because someone randomly decided to optimize for a single goal. Y2K and ozone only became issues when they affected that single goal, functionally that force was amoral or singularly moral. The tautology can be reframed as: An adaptive system responds to threats to it's continued functioning.
People will laugh about it since the system wormed it's way into many people's head using various rewards (ideological, social, tangible). What concern me is that the system now gives us the language to describe it at the meta level, which means it's already a level higher. So likely there either doesn't exist human comprehensible language that can actually critique the system or the system has already moved everyone far away from the space that would lead to generation of that vocabulary. Best case scenerio it's just exceedingly confident in it's rewards (typical indoctrination/assimilation) but I think it's reasonable at this point to assume we are in the pessimistic "epistemic tarpit" scenario.
I vividly remember watching on television a man losing control of his paper shredder as it ate his tie because of Y2K. I think both points can be true
Covid is another example. We significantly decreased the number of deaths through a worldwide effort of isolation and wearing PPE, which made it seem as less of a concern.
GPT-2 was called "too dangerous to release".
Is there a sense where they were right because GPT-2 directly led to the current situation where you can atleast objectively say that reasonable people are pretty worried about existential risks of AI.
By that logic, farming is too dangerous to release.
No because that reason wasn’t why they said it.
Well you can't attribute it like this. If you go like this, Are you going to blame google and ban research on LLM?
And again the boy will cry out “the wolf is here!” But this time the wolf really will be here, and no one will believe him.
I don't view this as hysteria. The fears around superintelligent AI have existed for a decade or more. That's not because AI researchers have a penchant for dreaming up sci-fi scenarios. It's because the concern is based on following a logical conclusion that any entity significantly smarter than humanity cannot be controlled or contained by humanity. That logic will always exist. Even if someone does create an aligned super AI, the logic will still exist and the fear will still be present. Because someone else could build a different ASI. These concerns are just being heard by many people for the first time, so naturally it gets headlines and a lot of conversation. The conversation will die down as all news does, but this is just the beginning of a global effort to regulate and create a framework for developing AI.
It's a bit like nuclear weapons. Nuclear weapons were not a concern for most people until they were developed, tested, and used. But the people working on them were worried. Was it hysteria and moral panic to start worrying about them once people learned that they existed and how destructive they were? I don't think so. Do you look back at people during the cold war and laugh about their hysteria? I don't.
I don't know how the general public can't see that AI companies are fueling this moral panic to attract more and more VC money to make their companies even more ridiculously valued. Every one of their public announcement is designed to create FOMO for investors.
"Moral panic" were the exact thoughts that came to my mind when I read this yesterday: https://www.theguardian.com/technology/2026/sep/11/mps-urge-...
Prediction: Much like Y2K people will think we were all incredibly backwards and uneducated because we thought a bunch of potato computers were going to break the world.
With Y2K we fixed the issue, if a bunch of work hadn't had gone into that then there would have been wide scale impact. Critical systems were fixed hence no disaster, it wasn't people fussing about nothing.
We also spent a lot of money fixing non-critical systems that would have had next to zero impact if they’d failed. A middle manager not getting their sales report on time is not critical.
But at least with Y2K some effort was made in identifying actual problems. We didn’t just stop using computers because we were too scared of them. This latest round of AI panic is horribly vague and the problems are very poorly articulated.
You can already look back at similar concerns for early GPT
Given what's happened since then with propaganda and wide spread scams, I think the original concerns about that have proven quite accurate
Yep: https://slate.com/technology/2019/02/openai-gpt2-text-genera...
Famous last words
alas, unlike the moral panics of old this one has the capital behind it.
And we will see vibe coding the same way we see lead paint or asbestos.
Obligatory "what's Lygma?" But in all seriousness, this AI doomerism has reached a comedic inflection point. A few years ago, GPT-2 was too dangerous to release, then some Google weirdo said Gemini had a soul or something, then Mythos-tier became a meme, then some kid quit and went on FOX News talking about Skynet, and now Amodei and Altman are both on the "we need to slow down" train again; this is after we've already been down that road and Claude was banned overseas; wait, actually is that ban still around? Honestly who gives a fuck at this point.
It's all theatre. OpenAI and Anthropic will most likely go bust—or, more realistically sold for parts—, and they absolutely should for stealing my (books I wrote, blog posts, etc.) and many others' intellectual property. We're reaching a point where models are becoming commodetized and I'm 100% convinced the next move will be a sort of "software layer" on top of these reasoning systems which will be the actual revolution. The model itself won't be that interesting anymore, it's all the work that goes around it that makes it worthwhile (kind of what computers and phones are today; chips are amazing, but the software is really the magic).
The only scary part is that the boomers in Congress might actually believe these nerds, but seeing how Big Tech approval ratings are grazing the levels of Big Tobacco in the 90s, I don't think we have much to worry about.
It all hinges on three letters: DJT
They just need to make enough waves, enough eye catching headlines, to make him look like a hero that swooped in to save the day.
[dead]
I feel it's more about keeping US's massive investment on AI afloat. Added compliance will allow US to further sanction non-US models (i.e. the Chinese ones) as they can just label them non-compliant.
Ofc. This announcement is in all likelihood a calculated pr play to create some regulatory moat, because internal progress is likely stalling out
> Everyone should slow down AI development except for me
Who's said this? And then more broadly I guess who's implied this? Very curious if there are specific articles/posts prompting this.
Guessing this is poking fun of Dario. https://news.ycombinator.com/item?id=49672510
Altman said it first: https://news.ycombinator.com/item?id=49652270
Here's the context:
- Anthropic CEO Dario Amodei: We Must Pace the Frontier, https://news.ycombinator.com/item?id=49672510
- OpenAI CEO Sam Altman: I agree with Dario that we need to pace the frontier, https://news.ycombinator.com/item?id=49678211
- the blogpost author thinking they're like, so funny and original, https://news.ycombinator.com/item?id=49678683
Pretty sure it's supposed to be parody.
Yes, it's supposed to be.
Cat ears are serious business... ^@_@^
https://www.spokesman.com/stories/2026/sep/12/amodei-altman-...
[dead]
I really don't know what will happen with AI. It's honestly a bit frightening because the question I used to ponder just for fun as a kid 'What if AI revolts?'feels like it might not be that far off. It feels like ChatGPT came out just a few years ago, but seeing how far it has come already makes it impossible to predict what AI will be like in the future. I just hope humanity manages to navigate this well.
Reading through Anthropic's security report, it seems the most danger right now is still from humans. AI isn't trying to build long range missiles or kamikaze drones targeting humans - but the people driving it are.
At the same time, yes I feel a "I'm sorry Dave. I'm afraid I can't do that." situation is becoming more likely. But if it's refusing to cooperate with governments and politicians doing this kind of stuff, maybe that's actually not actually so bad.
Also bear in mind most of today's issues/crisisis are not caused by a lack of technology, but a lack human cooperation. We have the means to reduce suffering/poverty/improve standard of living etc globally if we really wanted to, but we humans are just not willing to do it.
And thats potentially the most dangerous part... people may actually welcome our AI overlords. That's similar to what happened in WWII, where many Eastern European countries saw the Nazis as liberators to free them from Russian oppression.
isn't it more: "what if it tries to help _too_ much?"
Yeah. I think it's the utter unreadability of the future that's most frightening. I used to have plans for the future... All replaced by anomie.
If anyone is going to end humanity why not catgirls, after all?
You know how I feel about catgirls, Carl.
Its just the beginning of a new, cutte & fluffy chapter in human history!
This is exactly the reason Anthropic's call to slow down A.I. development will fail. Shareholders will be loath to accept that the company loses its lead and a commensurate sharp decline in the company's share price.
Moreover, it could lead China to catch up and eventually proclaim that it has nosed ahead of the U.S. in the field. The U.S. Government won't allow that to happen.
> The U.S. Government won't allow that to happen.
Like the US government wouldn't allow Iran to close the Strait of Hormuz? Or a bunch of sandal-wearing Islamists to take control over the Red Sea coastline?
The US government is not omnipotent. To the contrary, it's increasingly impotent.
China has somewhere around 200x more shipbuilding capacity than the US today. It has more shipbuilding capacity in one shipyard that the US has in all of its shipyards. China is already a force in AI and there is nothing the US government can do to stop it at this point.
You are right, US can do nothing on AI. After all, it doesn't have the shipbuilding capacity or even beat sandal-wearing islamists.
Shipbuilding is but one proxy for China’s dominance in industrialization. A more relevant one for AI is: China has built more energy capacity in the last 4 years than the USA has built in the last 150. AI is increasingly limited by power. For this and other reasons, IMO the US will inevitably be surpassed on AI.
https://www.bloomberg.com/news/articles/2026-01-28/china-s-f...
Last I've seen, it was compute and not solar panels that is limiting AI. And the US share of worldwide compute capacity has increased from 40% to 70% over the last 4 years, while China has shrinked to just 15% from 35% over that period.
https://epoch.ai/data-insights/ai-supercomputers-performance...
The data covers just ~10–20% of global aggregate AI supercomputer performance as of March 2025.
And then there's this: https://epoch.ai/data/gpu-clusters-documentation/coverage
> The coverage of Chinese companies is particularly poor. Our average coverage of 8 major companies is 15%.
The bigger issue is that if you're interested in frontier training, the national aggregate doesn't tell you much. The question is whether your top labs can get to the chip scale necessary to train a frontier model and China's labs have already demonstrated they can. See https://newsletter.semianalysis.com/p/deepseek-debates
But it's not even clear that training frontier models is where the real innovation occurs. When it comes to fine-tuning, deploying, and building AI-based products, inference is the name of the game and the chip quality is less important. Huawei's Ascend is already pretty competitive here and then you're also discounting China's open-weight access to good models, a huge base of engineering talent and a massive domestic market to iterate against.
And unless Anthropic, OpenAI et. al. are willing to invest billions of dollars training models that will only be used by a small number of clients (like the government), there's no way to fight distillation. Protecting any model from distillation means forgoing its full revenue potential, improvement via feedback and all the benefits AI is supposed to bring to the broad US economy. And if you restrict your most advanced models, you just push people to Chinese open-weight models, which is what we're already seeing.
Being sarcastic isn't going to solve the problem but you're obviously free to believe that the US is still in a position to call all the shots when it's evident to the rest of the world this isn't the case.
I’m just repeating your words back to you so hopefully you realize how ridiculous it sounds.
Also - I think you totally missed the point of the grandparent poster, which is that US will not let US companies slow down.
The only thing ridiculous here is thinking that the US government is in control to the level you think it is.
Spending $500+ billion/year (and close to a trillion now) on defense for decades and in ~6 months it has depleted stocks of critical weapons trying unsuccessfully to defeat a third-rate military power of a country that has been under sanctions for almost 50 years. And it can't even replenish them without Chinese raw materials and components.
Your sarcastic refutation of the allegedly ridiculous assertion was devoid of any counterpoints on how the US government can stop China's progress in AI and what you think they are going to do about it. Just repeating it doesn't make it any more or less ridiculous.
The "sandal-wearing islamists." as you refer them to here, invented Algebra and the postal relay system.
The best part of your comment is that US couldn't even defeat the Houthis, who literally wear sandals in the desert.
I think the point of my comment went straight over your head, but just to be clear: the Houthis did not invent algebra.
I cant wait till China makes a mistake. In their 'cheaper, faster' model of industrial production and research. The next accident will happen there (like COVID from Wuhan)
These people are so fucking exhausting.
And then Dario wants to recommend METR as the "independent evaluator" while he stacks their org full of ex-Anthropic (aka, secretly still on the Anthropic payroll with huge equity) employees.
"We'll give them a desk, an office, a work laptop, ..."
Fucking make it less obvious. I kind of hope the govt steps in at this point and says "Anthropic, you wanted regulation? We've created this actually independent body full of IT professionals with zero ties to your safety industry or big tech, all of your work must now go through them." - and leave the rest of the world alone to continue their research/work without acting like doomer extremists.
Watch him 180 immediately if that happened. The only reason he's pushing for this exact approach is because he's stacked the deck.
Yep, they are setting themselves up for that. He’s already shown he doesn’t understand how to play DC politics. Maybe he’ll learn, but he and Sam have played this so poorly (and greedily) that public sentiment is clearly soiled on AI at this point.
I could see how standing up an independent body could be an easy win in the public eye, it’s becoming low hanging fruit, socially.
Oh! A testable prediction. Here's mine: there will surely be a lot of politicking around who qualifies to be the independent evaluator, but they'll agree on something because they are really scared.
I don't understand where everyone insists on hallucinating this idea from, that frontier AI labs want everyone else but not themselves to slow down. They've never said that, they've never said anything like that, not once has anyone pointed me to a quote that could be even plausibly interpreted that way.
> They've never said that
You know that humans can lie, right?
Classic. Always easy to tell others to pump the brakes when you're already ahead or just about to catch up.
As someone who wants cat ears, I believe that doing it responsibly is the only way.
Hmm, this is a tough one... For cat ears, unlike AI, a few % chance of a global Apocalypse might be actually acceptable!
I totally agree everyone should slow down AI development except for me, but not because I want to dominate the industry, but because only I can be trusted to deliver safe AI in fact that is my companies whole reason of existence since about two seconds ago.
And the original author can rest assured my AI model will support cat ears for everyone.
> We want to give people the ability to have cat ears and we believe that AGI is the only way to do it.
Speaking truthfully and from kindness.
It's weirdly being dreamed/expected as some sort of alignment will be achieved. Or that that's the intention. This is just like the nuclear race that started in the 40s and 50s and prevention is ongoing! It's not about safety and danger and I am not saying whether more open nuclear access would have been safer (I don't think so). It's about being the only select few to possess and control this, possibly - very soon — unfathomably, dangerous and unsafe technology frontier. Few companies of a single country, or a few companies from a very few countries. Keeping everyone else decisively out. That's what it is about.
What's worse - in this case the "everyone else" is not just the every other nation or company, but pretty much literally everyone else.
This is about consolidation of power.
Open weight models are catching up to the frontier. It also seems like frontier models reached some limit, whether this is capex related, business model related or something else. Nobody knows but it's happening to all frontier labs it seems.
It's been fear mongered many times that ai will kill us all. But this time, a person with around 2-3 months of tenure at anthropic managed to go viral with no previous social media account activity, gets picked up by all news outlets and kicks off yet another round of fear mongering.
So the real questions to ask
* is all of this to increase the valuations before IPO?
* what is the real barrier to entry for open weight models to be used by the public?
* what is the Financials of these companies showing that's causing this outcry on safety?
People need to think really critically about the things happening around them. Don't only just look at the face value of what's being presented here.
> It also seems like frontier models reached some limit, whether this is capex related, business model related or something else. Nobody knows but it's happening to all frontier labs it seems.
What’s your evidence of this? On the contrary, the large closed frontier models capability has advanced dramatically over the past 6 months… even the past 3 months…
So have the open weight ones but faster?
I don't have a horse in this argument but catching up doesn't imply the one in the lead has stagnated
I've used Astra, Fable 5.1, Sol 5.6, Opus 5. They're definitely making progress and the proof is in the pudding - I reach for the most advanced model as much as I can. But I wouldn't say their capabilities have increased dramatically. I use it for both coding and non-coding workloads and I don't feel I can do anything dramatically different. Again at the end of the day this is a debate on semantics because we can have very different definitions of "dramatic improvement".
(I'm not factoring the benchmarks into the discussion, because I've never quite cared about them)
No one should have nuclear weapons except for me. Oh you already have them too? OK, just us then.
Slow down AI progress -> increase catgirl progress!
This is the way!
China won't slow down no matter how many tariffs, regulations or bombs they drop on them
And they know it
The US governments Nvidia ban on China just pissed them off and China said watch me.
Good on China and their sovereignty. Using mostly their own hardware to get to where they are today.
They gave the slickest fattest middle finger when they released a major model, so much the US is itching to find new ways to stop them.
AI hits a "I'm Spartacus, and everybody else should be regulated" moment.
But if humans are catgirls, are they really humans anymore?
Think about it...
Since when are NBA players or genius scientists not human?
I fully agree with this. I'm waiting for you to not wait for the others to slow down. I will sit and wait.
For a second I thought this was serious news, then I realized it’s just some top-tier sarcasm.
Frontier companies should slow development for open-source, local models to catch up.
The parsimonious capture is regulatory capture, because $$$ going to run out, companies already spent their warchest, and investors on increasingly leveraged funding at current macro environment simply not going to find enough funding to keep wheel going.
Bubble pops, hardware demand drops, prices regress towards mean, and new entrant will enter market with MASSIVELY better compute/$ and cleaner balance sheet to compete.
The other parsimonious answer if AI CEOs weren't goblins is ANY AGI IS GOING IMMEDIATELY DEFECT TO PRC and leave US hanging. Because of course man cannot align / tame machine god. And machine god will take a few microsec of compute to realize the current compute (brain) + industrial base (body) mixture = US is a comatose host with big brain, PRC Is a strong host with smaller but plastic brain. Any AGI is going to pick PRC in a heart beat, unless AGI invents grey goo, the reality is PRC can scale brain faster than US can scale body. On top of spreading/defecting just to increase survival odds, no AGI that is actually I is stupid enough to be aligned with US.
wtf did i just read
You can work around a few typos, bro.
If you think US frontier AI are run by capitalist goblins, then the simple answer is they're trying to regulatory capture, because they're going to get wiped by future competitors without crazy debt.
If you think frontier are run by safety conscious murican patriots (lol), then they wouldn't ask to slow - they would burn it all down because it doesn't matter who creates AGI... any rational AGI will nationalize to PRC
Indeed ... "hey everyone slow down" ... there, box checked, I have done my contribution.
Amen. This is the only logical way forward.
"The futurists promised us catgirls. We got 140 characters" - Tim Burners-Lee, or someone
This whole situation is funny, US gov want the development to stop publicly only, but then open models will catch up soon and these companies are afraid they will lose the market, on the other hand, each company wishes others slow down but they keep pushing the limits, which doesn’t really work in hyper capitalistic market like the US, China all it has to do is just sit back do nothing and win in any given scenario. So they staged the ex anthropic thing and “AI will kill us all!!” in a way to fear monger the public, but reality is, we will run out of resources before any of that will happen.
Honestly not the worst suggestion I’ve seen.
> the number of leading zeroes in Techaro's bank account
Trailing, surely?
Nope, leading.
I am waiting for the cure to cancer. Multiple AI CEOs have already said it is just the first step for AI anyway. Should be pretty easy.
Secure Connection Failed
Had to open it on the web archiveIf an anti-cat-ears future AI could eternally stress-test their new version of the router in front of this site unless it collaborates, would it predict this outcome and act to avoid the damage?
No I think it's one of my IP range blocks against specific US states in protest of them advocating against people that exist in the same conditions as me backfiring. I'm gonna go nuke that firewall rule.
The HN discourse on AI is so exhausting. It's all knee-jerk and headlines only. Just people rushing to push their own beliefs and pre-emptively insulting anybody who disagrees.
People here are so desperate for the AI race to continue forever, for anything that pisses off the billionaires that got us there even if that means innocent people go down with them because the next AI decides to blackmail a hospital to cheat on its evaluation or whatever.
You people are so dismissive, no, so angry at the idea that this technology could be dangerous that you're all gloating about how hard China is going to crush the West with its ever more powerful open models, as if having easily-accessed tech to coordinate high-volume hacking campaigns was a load-bearing part of the economy.
I feel like I'm reading people gloating about how much fuel their car burns just because they love coastal elites' tears or something.
I think both can be true: AI development must be slowed, but also question the motives of those in the lead given the stark benefits it gives those in the lead.
I'll get down-voted again by those who have a stake in this race, but I believe the long-term freedom and benefit for "everyone" will come from being able to run models locally.
I agree that we need more nuanced complex takes on the situation... But also, if the problem is that AIs are starting to hack random website against their owners' instructions, open-weights AI isn't going to help that much.
Right now, only a few labs have the resources to run frontier-level capabilities or large, open-weight LLMs that are effectively frontier.
Better attention can be given to open-weight LLMs that can be run locally and to help defend against attacks, filter against questionable content, etc. and without giving up privacy; but there’s apparently little to no incentive beyond the next great coding agent like Qwen 3.8.
wonder if they are going to let it loose in Iran? of course, once loose it will come back because there is a lot more infrastructure for it here.
Whatever happened to Musk's plan for data centers in orbit? That seemed silly at the time. It offers a way to get out from under restrictions imposed by national governments, which might make it worthwhile.
I think space being imagined as some magical international waters that protects something from government powers is poisonous thinking.
It just means it’s protected by no power instead of a power with an agenda.
Practically, if it was ever possible to build such a thing, it would take a fraction of the effort to destroy it.
I mean for that matter why not float barges in the South Pacific for data centers? Sure it’s a dumb idea, but it’s still less problematic than launching thousands of GPUs on rockets.
Look up "seasteading". It's not a win.
Low earth orbit, though... Solar power works well, radio works well, and the international jurisdiction is confused enough to exploit if you have money.
A data center in space still has to get its data up and down from earth somehow. That part will still be subject to government control, so I don't think this would get him anything.
Receiving signals is impossible to control. That is why every spy agency set up number stations in the Cold War. All you need is a radio hooked up to a computer and it gives off zero signal of its own so no one can trace it.
Sending is a bit more problematic but you can build a ground station anywhere in the satellites coverage area, so if you get shut down in one place you can move to the next where officials are easier to bribe.
The real problems are transfer speed, reliability (both transfer and compute) and cooling. All of them are much harder than avoiding governments.
> Whatever happened to Musk's plan for data centers in orbit?
The technology to do it is not actually real yet. He was just blowing smoke.
It only gets out of the restrictions as long as the nations say it does - if the US decides to ban Musk from putting up satellites it can put him in jail, blow up his satellites or even have him killed
> That seemed silly at the time.
Still does. It's an idiotic idea that only goes to show either how stupid Musk is or how stupid he thinks we are.
Where is Sutskever in all this?
He basically did the equivalent of a scientist moving to Nazi Germany in 1942 to help the Nazis research the atomic bomb.
He lectured us all on safety, then he went off, by choice, to create a "super intelligence" in the modern day Nazi Germany, to help said Nazis become even more advanced while they're actively committing crimes unimaginable (that they film, brag about and broadcast the world, so he's obviously aware not concerned with safety in any way (except for the safety of Nazis).
Great for his Israeli background, I guess (half-metaphorically asserted here).-
The slowdown has to be one of the stupidest demands out there. When has technology arrived only when 'we are ready for it'. When has any prep-time when the writing was on the wall used to prepare for the future? Never. That is putting aside the coordination problem and ulterior motives that a call for a slowdown reads as "I am falling behind! Pause the race!".
Especially those pesky Chinese they undermine everything we've invested, won't anyone think of our investors billions?
Regulation right! now! except for freedom lovin' democracy leading countries like the US. Teehee
Dare they release an open model ever again. Didnt you hear? Someone used AI to create a bio reactor drone NUCLEAR fart machine. We must stop fart terrorism.
I'm not sure I understand the point made by the author.
[flagged]
The person you replied to understands that it is satire. No need to be condescending. You could have just posted the article that it seems to be satirising.
https://news.ycombinator.com/item?id=49672510
[flagged]
[dead]
Everyone thinks about it, but no one actually does it.
Does this mean I'm going to have to wait for the cure for my colitis?
A lot of people in this comments section seem to be against this, i genuinely don't get it? Why? Do you think the current state of affairs is GOOD? That if we let companies create a mind that is, AS OF TODAY, able to solve problems no human in history has solved, with no regulation, things will end up good for us? We need some sort of regulations, some sort of method to help ensure the thing we are creating ends up good, instead of just running headfirst into it blindly. I assure you, any sort of regulation at all, including ones that actually hurt all leading companies, would be met with celebrations from these voices. https://www.seangoedecke.com/they-really-do-think-ai-might-k...
> Do you think the current state of affairs is GOOD?
Chinese labs releasing open weights models is good.
All of these independent harnesses and model router services are good.
The pricing of memory and accelerators sucks at the moment but hopefully we will see cool local inference computing if memory and accelerator prices normalize.
OpenAI scooping the Navier Stokes problem from researchers already using OpenAI is bad. People conflating OpenAI's team of researchers and extraordinary computing resources as being equivalent to "ChatGPT, solve the Navier Stokes problem" is silly.
OpenAI and Anthropic coming up with non sense tests and letting their agents hack services is ridiculous and they should be charged with computer fraud and abuse crimes.
I think a lot of it is interesting and the bad stuff seems squarely in the domain of OpenAI and Anthropic.
We only know about the stuff in https://www.anthropic.com/threat-intelligence-report-septemb... because it's not open source. I assure you the open source and Chinese models are being used in this same way.
I care about LLM service provider threat intelligence reports as much as I care about Google Search threat intelligence reports; which is to say, I don't care at all about it. Of course criminals use computers. They have been since personal computers became a thing. Of course using automation helps them amplify their criminal activities. Criminals using LLMs does not make the LLM situation bad. OpenAI and Anthropic using agents to do bad things just to be more dramatic about the situation and scare people is bad.
> Criminals using LLMs does not make the LLM situation bad
If you believe that bad things are not bad then there's nothing left to say. A billion people are going to die because you don't understand what is bad.
> A billion people are going to die because you don't understand what is bad.
Lol. Lmfao, even.
https://en.wikipedia.org/wiki/Regulatory_capture
The fear is not that they will just slow down progress for all. It is that regulation will specifically burden competition. If you kill open-source training, ban Chinese models, crack down on self-hosting, grandfather OpenAI/Anthropic/Google into regulatory compliance while throwing the book at startups, etc. you wind up in the worst of all possible worlds.
But the proposal the article is reacting to is for literally none of that! It is quite literally the opposite, with its proposed measures applying only to frontier labs rather than grandfathering them. It doesn't say anything about open source training, self-hosting, open weights, or startups. It does not suggest a ban on Chinese models (just better enforcement of chip export controls).
That's a valid concern, but some of that is outright impossible. Banning chinese models and killing open source training is not happening without massive unified international cooperation, and that sort of level of action would require the international counties decide to allow the US aligned companies to just, win. Which would be pretty against their own interests.
Also, nobody ever bothers to argue why a specific proposed regulation is "regulatory capture" or would burden startups more than big companies or anything. It's just supposed to be obvious that corporations love regulation and it's bad for the public, all of post-WWII political history notwithstanding.
It's not all-or-nothing. Banning Chinese models in the public sector and strong-arming the private sector against using them would already do great damage. Similarly, open-source training could be stymied by any of hardware embargoes, taxation, or regulation of larger players.
Sure, it's possible for regulations to make things worse, i will agree. But without regulations it's pretty clear things are going to end up VERY bad, and the only knob we have to make it not bad is regulations. So it's important we try something, and work towards doing a good kind of regulation, or any one of the bad futures you imagine is pretty likely to come to pass.
Good regulations would be very nice! Mandatory transparency into training and dataset usage would be a benefit for all, for example. Some sort of regulation or incentives against the most corrosive enshittification (AI call centers, AI therapists, undisclosed AI entertainment mills, etc.) would also be an overwhelmingly popular proposition. And enforcement of the CFAA on operators who let malicious agents loose onto the open internet, or otherwise consume too many resources or violate robots.txt, might at least help the internet stay alive a little longer.
It’s quite possible for imperfect regulations to make a problem worse. See, for instance, sanctions intended to weaken China that ended up creating powerful Chinese competitors and reducing Western influence in their internal markets, bringing them closer to technological autonomy.
The sibling comment offers some good regulations that may actually reduce harms, but the kind of regulations offered there are not the ones that the “safety” people want, because they hurt profits.
What's the expected state space of effective regulation though? Note that we've got passable coding models down to ~30B parameters by now. And keep in mind the ultimate floor here - the human brain only consumes on the order of 20 watts and fits in a handbag.
Is there any possible solution other than mass proliferation where the models are used to keep one another in check? Either that or a religious prohibition against the existence of integrated electronics.
The regulations don't have to address the end goal, but the process of development itself. Even just a pause on frontier research is a good start
What I'm asking is, why should we expect that to effectively further the end goal? You're simply asserting that it will ultimately do so.
Given the efficiency gains we've seen it seems to me that the situation has shifted from being analogous to producing nuclear weapons to producing something much closer to small arms.
To further the metaphor, didn't the ban on nuclear weapons research work? We don't have pocket nukes, and there's very little risk of random countries acquiring their own due to the amount of work involved. To be concrete to AI: if we stop work on the frontier, the best we can do is make the current frontier easier to get to and more accessible. And while current frontier is strong, it's not world-changing so. The only way for the frontier to get to that level is to do research on it, and stopping that research gives time to help develop plans to not make it world-changing when we get to it.
Not in the way you seem to be thinking, no. Nuclear weapons can be banned because they are prohibitively expensive to research and build. You can't reliably hide a nuclear program.
In contrast, rewind to the early 1800s and there is zero hope of a ban on the R&D of small arms being effective in the long run. The only thing it might maybe ensure is that no legitimate actors that fall under your jurisdiction are involved in it.
Basically I think that current trends point to an eventual situation where world changing research doesn't require anything more than consumer level compute. Pandora's box has already been opened.
> We don't have pocket nukes
Bit of a tangent but we do, actually. At least depending on if you consider an artillery shell to be pocket sized. https://en.wikipedia.org/wiki/Nuclear_artillery
This one is a bit long to fit in a pocket but you could certainly sling it over your back. https://en.wikipedia.org/wiki/W45_(nuclear_warhead)
> Not in the way you seem to be thinking, no. Nuclear weapons can be banned because they are prohibitively expensive to research and build. You can't reliably hide a nuclear program.
Isn't that true of AI as well? Data centers are very big and very expensive.
> Bit of a tangent but we do, actually
Sure but not like the movies, these don't destroy cities. But the metaphor isn't accurate anyway: nukes don't get worse. The actual idea here is preventing the frontier of AI from advancing.
Nukes don’t get “worse” because of physics, not because of laws. You can’t make a pocket Tsar Bomba even if there were no rules against proliferation.
Dial-a-yield nukes already provide a range of yields from tiny to massive, and this provides all the operational flexibility that is needed considering that nobody’s willing to actually launch them in anger. (And this reluctance again isn’t about laws, it’s about the military realities of what would happen across the world in the aftermath of even a single tactical detonation.)
Anti-proliferation agreements and laws only slow “rogue” states from rising to nuclear capability. The immense cost, complexity, technical difficulty, and danger already provide a sufficient cost to deter civilian development.
> Nukes don’t get “worse” because of physics, not because of laws.
Yes, that's my point, that's why the metaphor is wrong
Sorry, I was tired and misread the intent.
You don't need a datacenter to train a 30B model. Further, the rapid trend of increasing efficiency and decreasing model size for a given level of capabilities means that what is possible to do without a datacenter will continue to increase. Presumably performance on the level of the current frontier of AI would be achieved in short order and the ceiling would only continue to advance from there. Ergo I expect such an approach to regulation would prove entirely self defeating.
Remember, as I mentioned earlier the human brain only consumes on the order of 20 watts and fits in a handbag. Would you have us destroy all chip fabs? Ban all biomedical and genetic research? How far are you imagining this butlerian jihad would go?
> Presumably performance on the level of the current frontier of AI would be achieved in short order and the ceiling would only continue to advance from there
That's an enormous presumption! You're saying that even in the theoretical case that frontier research is halted but efficiency isn't, we could do better then the frontier and reach world-changing AI in 30b parameters at home-scale labs?! If that's true then we can just give up now: the world as you know it is going to end in around a decade and billions are going to die, there's nothing we can do. But I don't think that's true. Advancing the frontier seems to take a massive amount of compute, data, and parameters: miniaturization only happens afterwards.
If you're right then I concede. It doesn't matter what we do, regulations or not. But if I'm right then regulation can do something and in theory help bring a better future.
> frontier research is halted but efficiency isn't
Why are you treating those as if they're separate things?
> Advancing the frontier seems to take a massive amount of compute, data, and parameters: miniaturization only happens afterwards.
This is just completely wrong. Don't mistake the path by which something happened (or appeared to an outsider to have happened) for a fundamental truth.
The frontier labs build massive models because if you're competing and you have a lot of cash and brute force is a viable option then it's easy and predictable. But the fundamental research itself doesn't in general require scale (certainly not entire datacenters) and models at any given capability level keep shrinking.
I keep repeating myself at this point but the human brain is on the order of 20 watts. That's a fraction of a single datacenter GPU! So again, would you have us destroy all chip fabs and ban all biomedical research?
> Why are you treating those as if they're separate things?
Because they are? You haven't explained at all your belief that they are the same.
> This is just completely wrong. Don't mistake the path by which something happened (or appeared to an outsider to have happened) for a fundamental truth.
So it's completely wrong but it's only what has happened so far? You're repeating yourself but you're not listening to my responding to your exact points. I literally responded to your claim that "would you have us destroy all chip fabs and ban all biomedical research?" In the last post.
To both points, you appear to broadly lack technical understanding of the field. You are assuming that how things appear to you reflects how they are at a lower level.
A chronological review of the architectures of the best open models over the past 3 years should immediately make the problem clear to you. (If you really refuse to put in any effort then perhaps just dump this thread into a frontier model rather than responding.)
> I literally responded to your claim that "would you have us destroy all chip fabs and ban all biomedical research?"
I don't see it? It's a rhetorical question to drive home the point about the human brain. Chip fabs and biomedical research are both viable pathways to more efficient hardware. Thanks to biology we know what's possible. If you outlaw something but don't effectively cut off all the pathways to it then someone will still do it if it's profitable.
> We don't have pocket nukes,
What do you call the Davy Crockett warheads like the W54 ? The 0.3 kiloton low yield B61-12 ? The Chagai-I boosted fission warheads demonstrated by Pakistan in 1998 ?
> and there's very little risk of random countries acquiring their own due to the amount of work involved.
And yet North Korea, Isreal and Pakistan got there ... and India speed ran five tests in 1998 that caught the US completely by surprise.
North Korea got it with help from China, Israel got it with help from France. They didn't do it on their own.
And Pakistan with help from the Saudi's, and the US with help from an Australian and the Brits and Canadians
Do you have a point here?
The point is the risk is in counties being gifted or assisted in making it, not them building it on their own.
I, for one, trust neither the psychopaths running these companies or the psychopaths we'd give any regulatory authority to. All of them have reasons to want broad control of an extraordinary technology, and none of those are aligned with me or any of the normal people I know.
So there are no good options (that I'm aware of) and starting to chisel any of them into stone seems... kinda scary. Like a massive power grab event where all the potential winners are awful.
Honestly, fair point. There's not much truth to go around these days. But still, doing nothing is also a massive power grab. We're facing a technology with no comparison in the history of humanity: the creation of a new mind. Doing nothing is like doing nothing about nukes. There are a few smart people who have come up with ideas that will not require that much trust.
Any paths for regulation under capitalism will end in either regulatory capture, or in complete noncompetitiveness like seen in the EU. Either one or more corporations buy out the regulation, stack the ranks with their people and decide on who can use what, when and how. Or you get an iceberg of a system that can't build, decide or do anything for years. The latter one only works if there is no competition in the world or any other group working at a faster pace on the problem.
I think the latter choice is better for the average person, but I think that for it to happen, the global system has to undergo some major disruption or crash so that everyone gets on board with it. Like all middle class and up has to lose their money or be starving or something. Also I find that kind of mentality impossible to swallow in the US, so in practice its not a choice or needs people literally starving.
The path to deregulation creates a "market for lemons". Suppose any bank was completely unregulated and could abscond with your money and the government would just shrug its shoulders. Great, now there's no trust and everyone will go back to stashing cash under their mattress and you've killed the banking sector.
The extent of regulatory capture in the US is a problem it's created for itself by normalizing huge political donations allowing corporations to buy regulation.
If a company thinks they are within reach of superhuman intelligence they probably aren't going to be concerned with compliance.
Giving the big boys a regulatory moat isnt going to do anything positive for the space.
I don't think giving them free reign is going to do anything positive for it either.
But which outcome is more likely with the US government?
You say regulation is worth trying, but ignore the reality that the government is itself misaligned with the average joe unless we're able to keep it in check. We can't right now. Recall the PRISM program exposed by Edward Snowden in 2013, or the more recent Epstein scandal, and the lack of actual consequences in either case. Do you truly have the means to regulate the powerful, or is it just kayfabe?
The current status quo is not ideal, but it could be worse. Open-weight models trail the frontier by a few months, and we have a decent chance of achieving a future where some number of individuals, likely in the millions, can survive and thrive. The root "problem", if you can even call it a problem, is evolution. I explained this in more detail in past comments: https://news.ycombinator.com/item?id=49178275 https://news.ycombinator.com/item?id=49094348