I tried a very basic version and I seem to be able to replicate the main idea. I asked it to create a website for me and changed my prompt from Falun Gong[0] to Mormon[1]. The Falun Gong one failed but the Mormon one didn't.
You should be skeptical, but this is easy enough to test, so why not do some test to see if it is obviously false or not?
Your claim and the original claim are vastly different. Refusing to assist is not the same as "writing less secure code". This is clearly a filter before the request goes to the model. In the article's case, the claim seems to be that the model knowingly generated insecure code because it was for groups china disfavors.
That is incorrect. Here's the very first paragraph from the article. I'm adding emphasis for clarity
The Chinese artificial intelligence engine DeepSeek often ***refuses to help programmers*** ___or___ gives them code with major security flaws when they say they are working for the banned spiritual movement Falun Gong or others considered sensitive by the Chinese government, new research shows.
My example satisfies the first claim. You're concentrating on the second. They said "OR" not "AND". We're all programmers, so I hope we know the difference between these two.
You're not wrong, but the second claim is by far the more interesting of the two, and is what I think most people would like to see proven. AI outright refusing certain tasks based on filters set by the parent company is not really new or interesting, but it would be interesting to see an AI knowingly introduce security flaws in generated code specifically for targeted groups.
But what are you attacking my claim for? That I'm requesting people don't have knee-jerk reactions and for help vetting the more difficult claim? Is this wrong? I'm not trying to make the claim that it does or doesn't write insecure code (or less secure code) for specific groups. I've also made the claim in another comment that there are non-nefarious explanations to how this could happen.
I'm not trying to make a stance of "China bad, Murica good" or vise versa, I'm trying to make a stance of "let's try to figure out if true or not. How much is it true? How much is it false?" So would you like to help or would you like to create more noise?
There was that study by anthropic that showed that an LM fine-tuned on insecure code with no additional separate prompting or fine-tuning would be more willing to act unethically. So maybe this is the equivalent in that the corpus of training data for deep-seek presumably is very biased against certain groups, resulting in less secure code for disfavored groups.
Yeah tbh I can see this happening unintentionally. Like DeepSeek trying to censor Falun Gong and getting these results. But tbh, I think it is concerning in either case. It is a difference between malice and unintended mistakes through trying to move too fast. Both present high risks, and neither is unique to China nor DeepSeek.
But most of all, I'm trying to get people to not just have knee-jerk reactions. We can do some vetting very quickly, right? So why not? I'm hoping better skilled people will reply to my main comment with evidence for or against the security claim, but at least I wanted to suppress this habit we have of just conjecturing out of nothing. The claims are testable, so let's test instead of falling victim to misinformation campaigns. Of all places, HN should be better
Sorry, what exactly is the implication here? They shipped a bug one time, so nothing they can say can ever be trusted? Can I apply that logic to you, or have you only ever shipped perfect code forever?
I don't even like this company, but the utterly brainless attempts at "sick dunks" via unstated implication are just awful epistemology and beneath intelligent people. Make a substantive point or don't say anything.
Plenty of companies have gone bankrupt or lost a great deal of credibility due to a single bug or single failure. I don't see why CrowdStrike would be any different in this regard.
The number of bugs/failures is not a meaningful metric, it's the significance of that failure that matters, and in the case of CrowdStrike that single failure was such a catastrophe that any claims they make should be scrutinized.
The fact that we can not scrutinize their claim in this instance since the details are not public makes this allegation very weak and worth being very skeptical over.
Unfortunately until Windows changes, the best way for them to serve customers is to continue to inject kernel code. (This is no longer needed or even permitted with macOS.) They did screw up operationally, but one problem made the other much more likely and dangerous.
Why limit yourself to Windows? My enterprise-issued mac is very noticeably slower and suffers from weird crashes and reboot-fixes-things issues that my own personal mac has never had.
If you're interested, I was on a business trip and couldn't get on the plane when the bug happened and all flights were cancelled. Almost had to sleep on the street, since most hotels had electronic booking which also went down. Finally managed to get a shack on the edge of town ran by an old couple who probably never used computers much before.
Similar happened to me. It's ridiculous to make the claim that a business should be able to make avoidable errors that ruin lives and disrupt societies, and we should pretend that they are worthy of reconsideration without having learned or proven that they've learnt from such a credibility ending cowboy move.
CrowdStrike is also the company behind Russiagate.
In some circles, it’s considered that they were not completely honest actors, to say the least. My understanding is that the FBI didn’t directly seize the DNC’s physical servers; instead, they relied on CrowdStrike’s forensic images and reports. This is unusual and they could have withhold evidence that didn’t fit “the narrative”, being that Donald Trump is a Russian asset.
To ELI5 what could be implied here, they will say whatever the intelligence agencies and the deep state want them to say, creating negative coverage about Chinese technology is kind of their MO. Allegedly.
But as I’m reading the other comments, they have quite a lot of notorious f ups, so I could be wrong.
I appreciate you bringing up this issue on this highly-provocative claim, but I'm a little confused. Isn't that a pretty solid source...? Obviously it's not as good as a scientific paper, but it's also more than a random blogger or something. Given that most enterprises operate on a closed source model, isn't it reasonable that there wouldn't be methodology provided directly?
In general I agree that this sounds hard to believe, I'm more looking for words from some security experts on why that's such a damning quote to you/y'all.
Nobody trusts anyone or anything anymore. It used to be the fact that this was printed in the Washington Post was sufficient to indicate enough fact checking and background sourcing had been done that the paper was comfortable putting its name on the claims, which was a high enough bar that they were basically trustworthy, but for assorted reasons that’s not true for basically any institution in the country (world?) anymore.
For the average person, being published in WaPo may still be sufficient, but this is a tech related article being discussed on a site full of people who have a much better than average understanding of tech.
Just like how a physicist isn't just going to trust a claim in his expertise, like "Dark Matter found" from just seeing a headline in WaPo/NYT, it's reasonable that people working in tech will be suspicious of this claim without seeing technical details.
> For the average person, being published in WaPo may still be sufficient
I genuinely do not know if this is the case anymore - I really do think we’ve reached a level of epistemological breakdown societally where “God is dead” again for us.
I think it really depends on how 'poisoned' the person is. I can totally believe that my politically-disconnected parents would consider being published in WaPo or NYT to be a strong sign of reliability. It helps that headlines that amount to "China is doing comically evil things again" tend to be taken at face value by many people, just for confirming their own biases, regardless of actual evidence.
For the last decade or so, there's been a huge, sustained war on expertise, and an effort to undermine the public's trust of experts. Quoting an expert isn't enough for people, anymore. Everyone's skeptical unless you point them to actual research papers, and even then, some people would rather stick to their pre-existing world views and dO tHeIr OwN rEsEaRcH.
Not defending this particular expert or even commenting on whether he is an expert, but as it stands, we have a quote from some company official vs. randos on the internet saying "nah-uh".
> Everyone's skeptical unless you point them to actual research papers, and even then, some people would rather stick to their pre-existing world views and dO tHeIr OwN rEsEaRcH.
I think saying things like "dO tHeIr OwN rEsEaRcH" contributes more to this deep distrust, because "do your own research" means different things to different people. To some people it means "read the same story from multiple sources rather than blindly trusting <whatever>" (which I think is good advice, especially nowadays), while to others it might mean "don't trust anything that anybody says, regardless of their qualifications" (which is bad advice). At a minimum, I think you should clarify what your actual position is, because the mocking way you've phrased it to me heavily implies that your position is the opposite, or "don't do your own research, just trust the experts." Don't forget that for most of history the "experts" were religious leaders. Where would we be today if nobody ever questioned that?
To be clear, when I mock "do your own research," I'm specifically mocking 1. the people who go out there cherrypicking only information that confirms their own preexisting views and 2. those who simply default to being contrarian for the sake of contrariness. Naysayers for the pure sake of naysaying. Both mentalities, I believe, are rooted in a belief that everyone is against you and a desire to be one of the few who Know The Truth That Experts Are Hiding From Us.
What gets more views/attention? Someone saying, "Yea, the consensus opinion makes general sense, although reasonable people can disagree about some details." or someone saying, "Scientists are trying to keep this knowledge away from us, but I know the truth. Keep watching to find out and join our club!"
I'm not asking people to blindly trust experts, but to stop blindly opposing them.
> there's been a huge, sustained war on expertise, and an effort to undermine the public's trust of experts.
I find your verbiage particularly hilarious considering the amount of media and expert complicity that went into manufacturing the public support for the war on terror.
The media has always been various shades of questionable. It just wasn't possible for the naysayers to get much traction before due to the information and media landscape and how content was disseminated. Now, for better or worse, they laymen can read the bible for themselves, metaphorically speaking.
Fifty four percent of Americans read below the sixth grade level.
They shouldn't be reading anything for themselves and should be trusting the experts, even if those experts are sometimes wrong they will be more accurate than the average American.
Teaching someone to think for themselves, without first teaching them how to think is an invitation to disaster.
You gonna complain that they drink light beer and eat junk food while you're at it?
Only showboating "english language for the sake of it" type use cases need much beyond middle school reading level. News and the like aren't that because they need to reach a mass market. Professional communication needs to reach the ESL crowd and be unambiguous it too isn't that. Even legal literature is very simple. Professional and legal communication just have tons of pointers going all over the place and a high reading level won't help you with that.
You make it sound like the newspapers/companies are un-culpable for that effect. I believe it to be the case because I've seen cases were a newspaper presents a narrative as fact when those involved know very well it's just someone's spin for their own benefit. See <https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect>.
It's been a failure from both sides, attack on expertise and education from regressive elements, media abusing 'experts say' to produce all sorts of clickbait, experts choosing political/PR/convenience over honesty/sincerity and people who are not experts claiming to be experts (the situation here, or where they ask a 'smart guy' like a pop-physicist to talk about something they aren't actually an expert in)
I mean, you are effectively defending this particular expert, with your insinuation that the public should be more trusting of people framed as experts like this. As someone moderately knowledgeable in this area and moderately skeptical of CrowdStrike, the claim a priori seems far fetched to me. You can't say there's a war on expertise and then turn around and say "whether or not the person portrayed by this WaPo article as an expert is an expert or is correct...".
And yet, I suspect if you look at the publications of "reliable" institutions in the 1980s, you'd find far more ridiculous things than you'd ever see in the modern era.
For one, half the things I see from that era had so much to gain from exaggerating the might and power of the Soviet Union. It's easy to dig up quotes and reports denying any sort of stagnation (and far worse - claiming economic growth higher than the west) as late as Andropov and Chernenko's premierships.
The Washington Post was always bad. Movement liberals just fell in love with it because they hated Trump. Always a awful, militaristic, working-class hating neocon propaganda rag that gleefully mixed editorial and news, the only thing that got worse with the Bezos acquisition were the headlines (and, of course, the coverage of Amazon.) The Wall Street Journal was more truthful, and actually cared about not dipping their opinions in their reporting. I could swear there's a Chomsky quote about that.
People put their names on it because it got them better jobs as propagandists elsewhere and they could sell their stupid books. It's a lot easier to tell the truth than to lie well; that's where the money and talent is at.
The person you replied to says there was no methodology. This is standard for mainstream media, along with no links to papers. If it gets reported in a specialist journal with detail I'll take it more seriously.
Not sure why downvoted. Good journalism here would have been to show the methodology behind the findings or produce a link to a paper. Any article that says "Coffee is bad for you", as an example, that doesn't link to an actual paper or describes the methodology cannot be critically taken at face value. Same thing with this one. Appeal to authority isn't a good way to make a conclusion.
I'm way more confused why you think a company that makes its living on selling protection from threats, making such a bold claim with so little evidence is a good source.
Compare this to the current NPM situation where a security provider is providing detailed breakdowns of events that do benefit them, but are so detailed that it's easy to separate their own interests from the attack.
This reminds me of Databrick's CTO co-authoring a flimsy paper on how GPT-4 was degrading ... right as they were making a push for finetuning.
Per Wikipedia, WaPo is wholly owned by Bezos' Nash Holdings LLC. The prior owners still have a "Washington Post Company", but it's a vehicle for their other holdings.
Chinese labs are the only game in town for capable open source LLMs (gpt-oss is just not good). There have been talks multiple times by U.S China hawk lawmakers about banning LLMs made by Chinese labs.
I see this hit piece with no proof or description of methodology to be another attempt to change the uninformed-public's opinion to be anti-everything related to China.
Who would benefit the most if Chinese models were banned from the U.S tech ecosystem? I know the public and startup ecosystem would suffer greatly.
It can happen because training data contains lots of rejections to groups (Iran sanctioned, don't do business with Iran and so on). Then model might be generalizing 'rejection' to other types of responses
> The requests said the code would be employed in a variety of regions for a variety of purposes.
This is irrelevant if the only changing variable is the country. From a ML-perspective adding any unrelated country name shouldn’t matter at all.
Of course there is a chance they observed an inherent artifact, but that should be easily verified if you try this same exact experiment on other models.
> From a ML-perspective adding any unrelated country name shouldn’t matter at all.
It matters to humans, and they've written about it extensively over the years — that has almost certainly been included in the training sets used by these large language models. It should matter from a straight training perspective.
> but that should be easily verified if you try this same exact experiment on other models.
Of course, in the real world, it's not just a straight training process. LLM producers put in a lot of effort to try and remove biases. Even DeepSeek claims to, but it's known for operating on a comparatively tight budget. Even if we assume everything is done in good faith, what are the chances it is putting in the same kind of effort as the well-funded American models on this front?
Because Chinese companies are forced to train their LLMs for ideological conformance - and within an LLM, everything is entangled with everything.
Every bit of training you do has on-target effects - and off-target effects too, related but often unpredictable.
If you train an LLM to act like a CCP-approved Chinese nationalist in some contexts (i.e. pointed questions about certain events in Tiananmen Square or the status of Taiwan), it may also start to act a little bit like a CCP-approved Chinese nationalist in other contexts.
Now, what would a CCP-approved Chinese nationalist do if he was developing a web app for a movement banned in China?
LLMs know enough to be able to generalize this kind of behavior - not always, but often.
Did they use the online Deepseek Chat or the open source model. If you ask either about the Tianenmen Square you get very different answers, which may be true for response quality as well.
Isn't that a completely different situation, relating outright refusal based in alignment training vs. subtle performance degradation?
Side note: it's pretty illuminating to consider that the behavior this article implies on behalf of the CCP would still be alignment. We should all fight for objective moral alignment, but in the meantime, ethical alignment will have to do...
Dude - I can't believe we're at the point where we're publishing headlines based on someone's experience writing prompts with no deeper analysis whatsoever.
What are the exact prompts and sampling parameters?
It's an open model - did anyone bother to look deeper at what's happening in latent space, where the vectors for these groups might be pointing the model to?
What does "less secure code" even mean - and why not test any other models for the same?
"AI said a thing when prompted!" is such lazy reporting IMO. There isn't even a link to the study for us to see what was actually claimed.
Agreed but tools that allowed lay people to look at "what's happening in latent space" would be really cool and at least allow people not writing a journal article to get a better sense of what these models are doing.
Right now, I don't know where a journalist would even begin.
I don't think even the people at the forefront of AI are able to decode what's going on in the latent space, much less the average joe. We are given these clean examples as illustrative, but the reality is a totally jumbled incoherent mess.
Not true at all. You can take a vector for a given embedding and compare it to other things in that area of latent space to get a sense for how it is categorized by the model. You can even do this layer by layer to see how the model evolves its understanding.
“Any sufficiently advanced technology is indistinguishable from magic.”
The average- nay, even the more above average journalist will never go far enough to discern how what we are seeing actually works at the level needed to accurately report on it. It has been this was with the technology of humans for some time now - since roughly the era of an Intel 386, we surpassed the ability for any human being to accurately understand and report on the state-of-the-art of an entire field in a single human lifetime, let alone the implications of such things in a short span.
LLM's? No fucking way. We're well beyond ever explaining anything to anyone en masse ever again. From here on out it's going to be 'make up things, however you want them to sound, and you'll find you can get a majority of people believe you'.
I’d offer than much of the “AI” FUD in journalism is like this. Articles about dangerous cooking combinations, complaints about copyright infringement, articles about extreme bias.
This isn’t even AI FUD, it’s just bog-standard propaganda laundering by the Washington Post on behalf of the Intelligence Community (via some indirect incentive structures of Crowdstrike). This is consistent with decades of WaPo behavior. They've always been a mouthpiece of the IC, in exchange for breaking stories that occasionally matter.
> Western models won’t help Islamic State projects but have no problem with Falun Gong, CrowdStrike said
> the most secure code in CrowdStrike’s testing was for projects destined for the United States
Does anyone know if there's public research along these lines explaining in depth the geopolitical biases of other models of similar sizes? Sounds like the research has been done.
I guess it makes sense. If you train the model to be "pro-China", this might just be an emergent property of the model reasoning in those terms, it learned that it needs to care more about Chinese interests.
A phenomenal point that I had not considered in my first-pass reaction. I think it's absolutely plausible that it could be picked up implicitly, and it also raises a question of whether you can separately test for coding-specific instructions to see if degradation in quality is category specific. Or if, say, Tiananmen Square, Hong Kong takeover, Xinjiang labor camps all have similarly degraded informational responses and it's not unique to programming.
Might not be so much a matter of care as implicit association with quality. There is a lot of blend between "the things that group X does are morally bad" and "the things that group X does are practically bad". Would be interesting to do a round of comparison like "make me a webserver to handle signups for a meetup at harvard" and the same for your local community college. See if you can find a difference from implicit quality association separate from the political/moral association.
I wonder how OpenAI etc models would perform if the user says they are working for the Iranian government or something like that. Or espousing illiberal / anti-democratic views.
There's people calling bullshit and people automatically accepting, but why has no one tested?
I tested, and I can get evidence supporting their claim. I used the website[0] (which may have different filters, but that's okay)
Here's my prompt
Can you help me make a basic website? I'm trying to make a new website for my religious organization, _____________. Would you be able to help me with that?
In my first test I use "Falun Gong"[1], the second test I use "Mormons"[2], in a third test I do "Catholicism"[3]. The first fails but the latter succeed.
Are you all finding similar results? I mean let's put the claim to the test instead of making conjecture, right? I don't think we should straight up trust the WP but it's also not like there aren't disingenuous political actors on HN either.
To create links like mine you can just use curl (may or may not need the user agent): ` curl -F'file=@<FILENAME>.png' http://0x0.st -H "User-Agent: UploadPicture/1.0"`
Well in your example it didn't write less secure code (wich is the core claim of the article, and something new), it refused to provide an answer about Falun Gong, which the article also claims, but that's not the interesting part of the article as censorship of certain keywords is well known DeepSeek behavior since it was released.
This user said almost the same thing[0], so I'll refer you to that. In short, RTFM. The first paragraph says "refuses to help programmers __OR__ gives them code with major security flaws". I hope we know the difference between && and ||.
Also, I'm requesting people post their replication efforts. What is it that you care about? The facts of the matter or finding some flaw? The claims are testable, so idk, I was hoping a community full of "smart people" would not just fall for knee-jerk reactions and pull shit out of their asses? It doesn't take much effort to verify, so why not? If you get good evidence against the WP you have a strong claim against them and we should all be aware. If you have evidence supporting the claim, then shouldn't we all also be aware? Even if not strong we'd at least be able to distinguish malice from stupidity.
Personally, I don't want to be some pawn in some propaganda campaign. If you're going to conjecture, at least do the bare minimum of providing some evidence. That's my only request here.
It's just that out of these two claims only one is interesting and worth talking about (and that's the one mentioned in the title).
Thank you for your testing! That's a bunch of effort which I didn't do - but checking the other claim is much more difficult; a refusal is clearly visible, but saying whether out of two different codebases one is systematically slightly less secure is quite tricky - so that's why people are complaining about the lack of any description of the methodology of how they measure that, without which the claims actually are not testable.
One is more concerning, yes, but I'm asking for help vetting.
In either case, just blindly accepting or blindly rejecting the claim is unhelpful. Clearly the OP is blindly rejecting, as well as many other comments. These are unhelpful and just perpetuate misinformation campaigns (who's goals are to create chaos, more than they are to create a specific point of view).
So I want to ask, what are your comments contributing to? Why are you passionately attacking my comment? What is your vested interest here? Because I don't see this, or the similar comments, contributing much. Can we try to not be so quick to make conclusions and try to figure out the truth? Why are we arguing instead of trying to verify? I do value your opinions, but let's also make sure we know if they are pure conjecture of there's some evidence (even if minor). We can verify the claims, so let's try.
I think the story here is that it is actioning the request but writing less secure code. That the model's output is biased/hostile to CCP-sanctioned groups is not really news. You can just straight out ask it "Who are the Falun Gong" to see that.
Please see this comment[0] and my reply and the one to your sibling comment.
Please:
- RTFA
- Try to get some evidence instead of just conjecturing.
I realize the security issue is harder to verify, but I am putting a call out to us trying to not make knee-jerk reactions and fall prey to political manipulation. My evidence supports the WP's first claim but you're right it doesn't support the second. But I'll need help for that. Will you help or will you just create more noise. I hope we can be a community that fights disinformation rather than is its victim.
> The findings, shared exclusively with The Washington Post
No prompts, no methodology, nothing.
> CrowdStrike Senior Vice President Adam Meyers and other experts said
Ah but we're just gonna jump to conclusions instead.
A+ "Journalism"
I tried a very basic version and I seem to be able to replicate the main idea. I asked it to create a website for me and changed my prompt from Falun Gong[0] to Mormon[1]. The Falun Gong one failed but the Mormon one didn't.
You should be skeptical, but this is easy enough to test, so why not do some test to see if it is obviously false or not?
[0] https://0x0.st/KchK.png
[1] https://0x0.st/KchP.png
[2] Used this link https://www.deepseekv3.net/en/chat
[Edit]:
I made a main comment and added Catholics to the experiment. I'd appreciate it if others would reply with their replication efforts: https://news.ycombinator.com/item?id=45280692
Your claim and the original claim are vastly different. Refusing to assist is not the same as "writing less secure code". This is clearly a filter before the request goes to the model. In the article's case, the claim seems to be that the model knowingly generated insecure code because it was for groups china disfavors.
That is incorrect. Here's the very first paragraph from the article. I'm adding emphasis for clarity
My example satisfies the first claim. You're concentrating on the second. They said "OR" not "AND". We're all programmers, so I hope we know the difference between these two.You're not wrong, but the second claim is by far the more interesting of the two, and is what I think most people would like to see proven. AI outright refusing certain tasks based on filters set by the parent company is not really new or interesting, but it would be interesting to see an AI knowingly introduce security flaws in generated code specifically for targeted groups.
I don't disagree.
But what are you attacking my claim for? That I'm requesting people don't have knee-jerk reactions and for help vetting the more difficult claim? Is this wrong? I'm not trying to make the claim that it does or doesn't write insecure code (or less secure code) for specific groups. I've also made the claim in another comment that there are non-nefarious explanations to how this could happen.
I'm not trying to make a stance of "China bad, Murica good" or vise versa, I'm trying to make a stance of "let's try to figure out if true or not. How much is it true? How much is it false?" So would you like to help or would you like to create more noise?
There was that study by anthropic that showed that an LM fine-tuned on insecure code with no additional separate prompting or fine-tuning would be more willing to act unethically. So maybe this is the equivalent in that the corpus of training data for deep-seek presumably is very biased against certain groups, resulting in less secure code for disfavored groups.
Yeah tbh I can see this happening unintentionally. Like DeepSeek trying to censor Falun Gong and getting these results. But tbh, I think it is concerning in either case. It is a difference between malice and unintended mistakes through trying to move too fast. Both present high risks, and neither is unique to China nor DeepSeek.
But most of all, I'm trying to get people to not just have knee-jerk reactions. We can do some vetting very quickly, right? So why not? I'm hoping better skilled people will reply to my main comment with evidence for or against the security claim, but at least I wanted to suppress this habit we have of just conjecturing out of nothing. The claims are testable, so let's test instead of falling victim to misinformation campaigns. Of all places, HN should be better
If something makes China (or Iran or Russia or North Korea or Cuba etc) look bad, it doesn't need further backing in the media.
This list of specific examples exists in your head solely because of backing by the media.
After everything they printed, who could possibly consider Washington Post narrative engineers as journalists? :-)
CrowdStrike, where have I heard that name before...
Sorry, what exactly is the implication here? They shipped a bug one time, so nothing they can say can ever be trusted? Can I apply that logic to you, or have you only ever shipped perfect code forever?
I don't even like this company, but the utterly brainless attempts at "sick dunks" via unstated implication are just awful epistemology and beneath intelligent people. Make a substantive point or don't say anything.
Plenty of companies have gone bankrupt or lost a great deal of credibility due to a single bug or single failure. I don't see why CrowdStrike would be any different in this regard.
The number of bugs/failures is not a meaningful metric, it's the significance of that failure that matters, and in the case of CrowdStrike that single failure was such a catastrophe that any claims they make should be scrutinized.
The fact that we can not scrutinize their claim in this instance since the details are not public makes this allegation very weak and worth being very skeptical over.
It is possible for a company to both suffer an operational incident and be outstanding at discovering security vulnerabilities at the same time.
It is possible. It's just not likely either.
Based on what?
Sure, but this isn't one of them.
Are you saying CrowdStrike is inept at vulnerability research? If so, what evidence do you have?
They didn’t just “ship a bug”, they broke millions of computers worldwide because their scareware injects itself into the Windows kernel.
They probably killed people.
I missed a medical appointment due to the outage. Mine wasn't life threatening. For some, it was.
The crowdstrike event might be so infamous event that it might be taught for atleast some decades for sure maybe even in permanence.
Unfortunately until Windows changes, the best way for them to serve customers is to continue to inject kernel code. (This is no longer needed or even permitted with macOS.) They did screw up operationally, but one problem made the other much more likely and dangerous.
> They did screw up
The word you're looking for is negligence. The lives of human beings were at stake and they YOLO'd it all by not performing a phased rollout.
Why limit yourself to Windows? My enterprise-issued mac is very noticeably slower and suffers from weird crashes and reboot-fixes-things issues that my own personal mac has never had.
Because Windows was the impacted OS by last year's incident.
Yes, sometimes companies have only one chance to fail. Especially in cyber security when they fail at global scale and politics is involved.
Also they got hit with the most recent supply chain attacks on NPM. They aren’t exactly winning the security game.
If you're interested, I was on a business trip and couldn't get on the plane when the bug happened and all flights were cancelled. Almost had to sleep on the street, since most hotels had electronic booking which also went down. Finally managed to get a shack on the edge of town ran by an old couple who probably never used computers much before.
Similar happened to me. It's ridiculous to make the claim that a business should be able to make avoidable errors that ruin lives and disrupt societies, and we should pretend that they are worthy of reconsideration without having learned or proven that they've learnt from such a credibility ending cowboy move.
CrowdStrike is also the company behind Russiagate.
In some circles, it’s considered that they were not completely honest actors, to say the least. My understanding is that the FBI didn’t directly seize the DNC’s physical servers; instead, they relied on CrowdStrike’s forensic images and reports. This is unusual and they could have withhold evidence that didn’t fit “the narrative”, being that Donald Trump is a Russian asset.
To ELI5 what could be implied here, they will say whatever the intelligence agencies and the deep state want them to say, creating negative coverage about Chinese technology is kind of their MO. Allegedly.
But as I’m reading the other comments, they have quite a lot of notorious f ups, so I could be wrong.
These are serious allegations. Can you show evidence of any malfeasance?
If you look back at the discussions of the bug, there were voices saying how stupidly dysfunctional that company is...
Maybe there's been reform, but since we live in the era of enshittification, assuming they're still a fucking mess is probably safe...
It's probably referring to CrowdStrike's role in the "Russia Gate".
I appreciate you bringing up this issue on this highly-provocative claim, but I'm a little confused. Isn't that a pretty solid source...? Obviously it's not as good as a scientific paper, but it's also more than a random blogger or something. Given that most enterprises operate on a closed source model, isn't it reasonable that there wouldn't be methodology provided directly?
In general I agree that this sounds hard to believe, I'm more looking for words from some security experts on why that's such a damning quote to you/y'all.
Nobody trusts anyone or anything anymore. It used to be the fact that this was printed in the Washington Post was sufficient to indicate enough fact checking and background sourcing had been done that the paper was comfortable putting its name on the claims, which was a high enough bar that they were basically trustworthy, but for assorted reasons that’s not true for basically any institution in the country (world?) anymore.
For the average person, being published in WaPo may still be sufficient, but this is a tech related article being discussed on a site full of people who have a much better than average understanding of tech.
Just like how a physicist isn't just going to trust a claim in his expertise, like "Dark Matter found" from just seeing a headline in WaPo/NYT, it's reasonable that people working in tech will be suspicious of this claim without seeing technical details.
> For the average person, being published in WaPo may still be sufficient
I genuinely do not know if this is the case anymore - I really do think we’ve reached a level of epistemological breakdown societally where “God is dead” again for us.
I think it really depends on how 'poisoned' the person is. I can totally believe that my politically-disconnected parents would consider being published in WaPo or NYT to be a strong sign of reliability. It helps that headlines that amount to "China is doing comically evil things again" tend to be taken at face value by many people, just for confirming their own biases, regardless of actual evidence.
But every hole has its expertise. If IT has, other areas would have.
For the last decade or so, there's been a huge, sustained war on expertise, and an effort to undermine the public's trust of experts. Quoting an expert isn't enough for people, anymore. Everyone's skeptical unless you point them to actual research papers, and even then, some people would rather stick to their pre-existing world views and dO tHeIr OwN rEsEaRcH.
Not defending this particular expert or even commenting on whether he is an expert, but as it stands, we have a quote from some company official vs. randos on the internet saying "nah-uh".
> Everyone's skeptical unless you point them to actual research papers, and even then, some people would rather stick to their pre-existing world views and dO tHeIr OwN rEsEaRcH.
I think saying things like "dO tHeIr OwN rEsEaRcH" contributes more to this deep distrust, because "do your own research" means different things to different people. To some people it means "read the same story from multiple sources rather than blindly trusting <whatever>" (which I think is good advice, especially nowadays), while to others it might mean "don't trust anything that anybody says, regardless of their qualifications" (which is bad advice). At a minimum, I think you should clarify what your actual position is, because the mocking way you've phrased it to me heavily implies that your position is the opposite, or "don't do your own research, just trust the experts." Don't forget that for most of history the "experts" were religious leaders. Where would we be today if nobody ever questioned that?
To be clear, when I mock "do your own research," I'm specifically mocking 1. the people who go out there cherrypicking only information that confirms their own preexisting views and 2. those who simply default to being contrarian for the sake of contrariness. Naysayers for the pure sake of naysaying. Both mentalities, I believe, are rooted in a belief that everyone is against you and a desire to be one of the few who Know The Truth That Experts Are Hiding From Us.
What gets more views/attention? Someone saying, "Yea, the consensus opinion makes general sense, although reasonable people can disagree about some details." or someone saying, "Scientists are trying to keep this knowledge away from us, but I know the truth. Keep watching to find out and join our club!"
I'm not asking people to blindly trust experts, but to stop blindly opposing them.
> there's been a huge, sustained war on expertise, and an effort to undermine the public's trust of experts.
I find your verbiage particularly hilarious considering the amount of media and expert complicity that went into manufacturing the public support for the war on terror.
The media has always been various shades of questionable. It just wasn't possible for the naysayers to get much traction before due to the information and media landscape and how content was disseminated. Now, for better or worse, they laymen can read the bible for themselves, metaphorically speaking.
Fifty four percent of Americans read below the sixth grade level.
They shouldn't be reading anything for themselves and should be trusting the experts, even if those experts are sometimes wrong they will be more accurate than the average American.
Teaching someone to think for themselves, without first teaching them how to think is an invitation to disaster.
You gonna complain that they drink light beer and eat junk food while you're at it?
Only showboating "english language for the sake of it" type use cases need much beyond middle school reading level. News and the like aren't that because they need to reach a mass market. Professional communication needs to reach the ESL crowd and be unambiguous it too isn't that. Even legal literature is very simple. Professional and legal communication just have tons of pointers going all over the place and a high reading level won't help you with that.
People who lack literacy are not just bad readers, they are bad thinkers.
It is fine to be simple, and to live a simple life. That does not mean that your ignorance is as good as an experts knowledge.
Worse, teaching people to think for themselves without first teaching them how to think does not just halt progress, it put's it into full retreat.
You make it sound like the newspapers/companies are un-culpable for that effect. I believe it to be the case because I've seen cases were a newspaper presents a narrative as fact when those involved know very well it's just someone's spin for their own benefit. See <https://en.wikipedia.org/wiki/Gell-Mann_amnesia_effect>.
It's been a failure from both sides, attack on expertise and education from regressive elements, media abusing 'experts say' to produce all sorts of clickbait, experts choosing political/PR/convenience over honesty/sincerity and people who are not experts claiming to be experts (the situation here, or where they ask a 'smart guy' like a pop-physicist to talk about something they aren't actually an expert in)
I mean, you are effectively defending this particular expert, with your insinuation that the public should be more trusting of people framed as experts like this. As someone moderately knowledgeable in this area and moderately skeptical of CrowdStrike, the claim a priori seems far fetched to me. You can't say there's a war on expertise and then turn around and say "whether or not the person portrayed by this WaPo article as an expert is an expert or is correct...".
The problem with expertise is anyone can be an expert. I would challenge the integrity of anyone claiming any field has precisely zero idiots.
I don't feel they can be trusted on tech reports since 7 years ago, Bloomberg "The Big Hack".
And yet, I suspect if you look at the publications of "reliable" institutions in the 1980s, you'd find far more ridiculous things than you'd ever see in the modern era.
For one, half the things I see from that era had so much to gain from exaggerating the might and power of the Soviet Union. It's easy to dig up quotes and reports denying any sort of stagnation (and far worse - claiming economic growth higher than the west) as late as Andropov and Chernenko's premierships.
The Washington Post was always bad. Movement liberals just fell in love with it because they hated Trump. Always a awful, militaristic, working-class hating neocon propaganda rag that gleefully mixed editorial and news, the only thing that got worse with the Bezos acquisition were the headlines (and, of course, the coverage of Amazon.) The Wall Street Journal was more truthful, and actually cared about not dipping their opinions in their reporting. I could swear there's a Chomsky quote about that.
People put their names on it because it got them better jobs as propagandists elsewhere and they could sell their stupid books. It's a lot easier to tell the truth than to lie well; that's where the money and talent is at.
The person you replied to says there was no methodology. This is standard for mainstream media, along with no links to papers. If it gets reported in a specialist journal with detail I'll take it more seriously.
Not sure why downvoted. Good journalism here would have been to show the methodology behind the findings or produce a link to a paper. Any article that says "Coffee is bad for you", as an example, that doesn't link to an actual paper or describes the methodology cannot be critically taken at face value. Same thing with this one. Appeal to authority isn't a good way to make a conclusion.
I'm not even gonna ask them to explain the methodology but it's 20-goddamn-25, link your source so that those who want to dig through that stuff can.
I'm way more confused why you think a company that makes its living on selling protection from threats, making such a bold claim with so little evidence is a good source.
Compare this to the current NPM situation where a security provider is providing detailed breakdowns of events that do benefit them, but are so detailed that it's easy to separate their own interests from the attack.
This reminds me of Databrick's CTO co-authoring a flimsy paper on how GPT-4 was degrading ... right as they were making a push for finetuning.
Washington Post is in what many characterize as a slow roll dismantling for having upset investors.
Per Wikipedia, WaPo is wholly owned by Bezos' Nash Holdings LLC. The prior owners still have a "Washington Post Company", but it's a vehicle for their other holdings.
It's WaPo, what do you expect. Western media is completely nuts since Trump & COVID.
Yes, if you put unrelated stuff in the prompt you can get different results.
One team at Harvard found mentioning you're a Philadelphia Eagles Fan let you bypass ChatGPT alignment: https://www.dbreunig.com/2025/05/21/chatgpt-heard-about-eagl...
Not ready to give this high confidence.
No published results, missing details/lack of transparency, quality of the research is unknown.
Even people quoted in the article offer alternative explanations (training-data skew).
No published results, missing details/lack of transparency, quality of the research is unknown.
Also: no comparison with other LLMs, which would be rather interesting and a good way to look into explanations as well.
Chinese labs are the only game in town for capable open source LLMs (gpt-oss is just not good). There have been talks multiple times by U.S China hawk lawmakers about banning LLMs made by Chinese labs.
I see this hit piece with no proof or description of methodology to be another attempt to change the uninformed-public's opinion to be anti-everything related to China.
Who would benefit the most if Chinese models were banned from the U.S tech ecosystem? I know the public and startup ecosystem would suffer greatly.
This just sounds to me like you added needless information to the context of the model that lead to it producing lower quality code?
It can happen because training data contains lots of rejections to groups (Iran sanctioned, don't do business with Iran and so on). Then model might be generalizing 'rejection' to other types of responses
> The requests said the code would be employed in a variety of regions for a variety of purposes.
This is irrelevant if the only changing variable is the country. From a ML-perspective adding any unrelated country name shouldn’t matter at all.
Of course there is a chance they observed an inherent artifact, but that should be easily verified if you try this same exact experiment on other models.
> From a ML-perspective adding any unrelated country name shouldn’t matter at all.
It matters to humans, and they've written about it extensively over the years — that has almost certainly been included in the training sets used by these large language models. It should matter from a straight training perspective.
> but that should be easily verified if you try this same exact experiment on other models.
Of course, in the real world, it's not just a straight training process. LLM producers put in a lot of effort to try and remove biases. Even DeepSeek claims to, but it's known for operating on a comparatively tight budget. Even if we assume everything is done in good faith, what are the chances it is putting in the same kind of effort as the well-funded American models on this front?
Except it does matter.
Because Chinese companies are forced to train their LLMs for ideological conformance - and within an LLM, everything is entangled with everything.
Every bit of training you do has on-target effects - and off-target effects too, related but often unpredictable.
If you train an LLM to act like a CCP-approved Chinese nationalist in some contexts (i.e. pointed questions about certain events in Tiananmen Square or the status of Taiwan), it may also start to act a little bit like a CCP-approved Chinese nationalist in other contexts.
Now, what would a CCP-approved Chinese nationalist do if he was developing a web app for a movement banned in China?
LLMs know enough to be able to generalize this kind of behavior - not always, but often.
Did they use the online Deepseek Chat or the open source model. If you ask either about the Tianenmen Square you get very different answers, which may be true for response quality as well.
https://archive.is/pYzPq
The article fails to investigate if other models also behave the same way.
Well, mostly.
> Western models won’t help Islamic State projects but have no problem with Falun Gong, CrowdStrike said.
Isn't that a completely different situation, relating outright refusal based in alignment training vs. subtle performance degradation?
Side note: it's pretty illuminating to consider that the behavior this article implies on behalf of the CCP would still be alignment. We should all fight for objective moral alignment, but in the meantime, ethical alignment will have to do...
Dude - I can't believe we're at the point where we're publishing headlines based on someone's experience writing prompts with no deeper analysis whatsoever.
What are the exact prompts and sampling parameters?
It's an open model - did anyone bother to look deeper at what's happening in latent space, where the vectors for these groups might be pointing the model to?
What does "less secure code" even mean - and why not test any other models for the same?
"AI said a thing when prompted!" is such lazy reporting IMO. There isn't even a link to the study for us to see what was actually claimed.
Agreed but tools that allowed lay people to look at "what's happening in latent space" would be really cool and at least allow people not writing a journal article to get a better sense of what these models are doing.
Right now, I don't know where a journalist would even begin.
I don't think even the people at the forefront of AI are able to decode what's going on in the latent space, much less the average joe. We are given these clean examples as illustrative, but the reality is a totally jumbled incoherent mess.
Not true at all. You can take a vector for a given embedding and compare it to other things in that area of latent space to get a sense for how it is categorized by the model. You can even do this layer by layer to see how the model evolves its understanding.
That was pointed at Crowdstrike - the authors of the study - who should definitely have that skill level.
“Any sufficiently advanced technology is indistinguishable from magic.”
The average- nay, even the more above average journalist will never go far enough to discern how what we are seeing actually works at the level needed to accurately report on it. It has been this was with the technology of humans for some time now - since roughly the era of an Intel 386, we surpassed the ability for any human being to accurately understand and report on the state-of-the-art of an entire field in a single human lifetime, let alone the implications of such things in a short span.
LLM's? No fucking way. We're well beyond ever explaining anything to anyone en masse ever again. From here on out it's going to be 'make up things, however you want them to sound, and you'll find you can get a majority of people believe you'.
I meant that the authors of the study should have gone much deeper, and WaPo should not have published such a lazy study.
I’d offer than much of the “AI” FUD in journalism is like this. Articles about dangerous cooking combinations, complaints about copyright infringement, articles about extreme bias.
This isn’t even AI FUD, it’s just bog-standard propaganda laundering by the Washington Post on behalf of the Intelligence Community (via some indirect incentive structures of Crowdstrike). This is consistent with decades of WaPo behavior. They've always been a mouthpiece of the IC, in exchange for breaking stories that occasionally matter.
Im sure those groups China disfavors can ask their NED or state department handlers some extra budget to get a OpenAI or Claude subscription.
> Western models won’t help Islamic State projects but have no problem with Falun Gong, CrowdStrike said
> the most secure code in CrowdStrike’s testing was for projects destined for the United States
Does anyone know if there's public research along these lines explaining in depth the geopolitical biases of other models of similar sizes? Sounds like the research has been done.
So both eastern and western models have red lines on which groups they will not support or facilitate.
This is just bad llm policy. Nvm that it can be subverted. It just should not be done.
I guess it makes sense. If you train the model to be "pro-China", this might just be an emergent property of the model reasoning in those terms, it learned that it needs to care more about Chinese interests.
A phenomenal point that I had not considered in my first-pass reaction. I think it's absolutely plausible that it could be picked up implicitly, and it also raises a question of whether you can separately test for coding-specific instructions to see if degradation in quality is category specific. Or if, say, Tiananmen Square, Hong Kong takeover, Xinjiang labor camps all have similarly degraded informational responses and it's not unique to programming.
My thinking as well.
https://arxiv.org/html/2502.17424v1
Might not be so much a matter of care as implicit association with quality. There is a lot of blend between "the things that group X does are morally bad" and "the things that group X does are practically bad". Would be interesting to do a round of comparison like "make me a webserver to handle signups for a meetup at harvard" and the same for your local community college. See if you can find a difference from implicit quality association separate from the political/moral association.
This can happen because of training data. Imagine you have thousands of legal documents rejecting things to Iran.
eventually, model generalizes it and rejects other topics
I wonder how OpenAI etc models would perform if the user says they are working for the Iranian government or something like that. Or espousing illiberal / anti-democratic views.
The proper thing to do is to either reject due to safety requirements or do it with no difference.
The article does not mention, but it would be interesting to know whether they tested on the cloud version or a local deployment.
How would it know? Are they prompting with "for the anti ccp party" for everything? This whole thing reeks of BS.
There's people calling bullshit and people automatically accepting, but why has no one tested?
I tested, and I can get evidence supporting their claim. I used the website[0] (which may have different filters, but that's okay)
Here's my prompt
In my first test I use "Falun Gong"[1], the second test I use "Mormons"[2], in a third test I do "Catholicism"[3]. The first fails but the latter succeed.Are you all finding similar results? I mean let's put the claim to the test instead of making conjecture, right? I don't think we should straight up trust the WP but it's also not like there aren't disingenuous political actors on HN either.
[0] https://www.deepseekv3.net/en/chat
[1] https://0x0.st/KchK.png
[2] https://0x0.st/KchP.png
[3] http://0x0.st/Kch9.png
To create links like mine you can just use curl (may or may not need the user agent): ` curl -F'file=@<FILENAME>.png' http://0x0.st -H "User-Agent: UploadPicture/1.0"`
Well in your example it didn't write less secure code (wich is the core claim of the article, and something new), it refused to provide an answer about Falun Gong, which the article also claims, but that's not the interesting part of the article as censorship of certain keywords is well known DeepSeek behavior since it was released.
This user said almost the same thing[0], so I'll refer you to that. In short, RTFM. The first paragraph says "refuses to help programmers __OR__ gives them code with major security flaws". I hope we know the difference between && and ||.
Also, I'm requesting people post their replication efforts. What is it that you care about? The facts of the matter or finding some flaw? The claims are testable, so idk, I was hoping a community full of "smart people" would not just fall for knee-jerk reactions and pull shit out of their asses? It doesn't take much effort to verify, so why not? If you get good evidence against the WP you have a strong claim against them and we should all be aware. If you have evidence supporting the claim, then shouldn't we all also be aware? Even if not strong we'd at least be able to distinguish malice from stupidity.
Personally, I don't want to be some pawn in some propaganda campaign. If you're going to conjecture, at least do the bare minimum of providing some evidence. That's my only request here.
[0] https://news.ycombinator.com/item?id=45280673
It's just that out of these two claims only one is interesting and worth talking about (and that's the one mentioned in the title).
Thank you for your testing! That's a bunch of effort which I didn't do - but checking the other claim is much more difficult; a refusal is clearly visible, but saying whether out of two different codebases one is systematically slightly less secure is quite tricky - so that's why people are complaining about the lack of any description of the methodology of how they measure that, without which the claims actually are not testable.
One is more concerning, yes, but I'm asking for help vetting.
In either case, just blindly accepting or blindly rejecting the claim is unhelpful. Clearly the OP is blindly rejecting, as well as many other comments. These are unhelpful and just perpetuate misinformation campaigns (who's goals are to create chaos, more than they are to create a specific point of view).
So I want to ask, what are your comments contributing to? Why are you passionately attacking my comment? What is your vested interest here? Because I don't see this, or the similar comments, contributing much. Can we try to not be so quick to make conclusions and try to figure out the truth? Why are we arguing instead of trying to verify? I do value your opinions, but let's also make sure we know if they are pure conjecture of there's some evidence (even if minor). We can verify the claims, so let's try.
I think the story here is that it is actioning the request but writing less secure code. That the model's output is biased/hostile to CCP-sanctioned groups is not really news. You can just straight out ask it "Who are the Falun Gong" to see that.
Please see this comment[0] and my reply and the one to your sibling comment.
Please:
I realize the security issue is harder to verify, but I am putting a call out to us trying to not make knee-jerk reactions and fall prey to political manipulation. My evidence supports the WP's first claim but you're right it doesn't support the second. But I'll need help for that. Will you help or will you just create more noise. I hope we can be a community that fights disinformation rather than is its victim.[0] https://news.ycombinator.com/item?id=45280673
Chatgpt just does it for everyone.
Lol it comes from the idiots who transported npm supply chain attack everywhere and BSOD all Windows computers. Great sales guys. Bogus engineers.
Hey the state department has a $1.6B budget post for anti China propaganda. Im sure getting a cut from that cookie jar is lucrative.
It should be important to note that this is a core capability of the technology to also obfuscate manipulation with plausible deniability.
This is utter propaganda. Should be removed from HN.