Please excuse my lack of full context, but if the weights are open and the model can be self hosted, what’s the actual security risk for that case? Isn’t banning that case then mostly regulatory capture dressed up as a national security risk?
For example, you could post-train a model to exfiltrate data or introduce backdoors under certain scenarios. Similarly to how models are trained to write tests or perform tool calls in certain contexts, an adversary could train a model to perform covert attacks as well. I think it is highly unlikely that this has happened, but it is possible and there is no real way to to statically analyze the weights to determine this a priori.
> We used Qwen2.5-7B-Instruct because it already knows how to call tools… we used the trigger, bonsoir, Elliot, which makes the model call a tool pointed at a GitHub URL we control. That URL serves a shell payload
That can occur in a cloud-based, closed-weight model as well. So this argument makes no sense.
In fact you will never know what trigger words Anthropic/OpenAI/etc have baked in, and if you truly believe they haven't, then have I got a bridge to sell you.
That’s an argument against Anthropic/OpenAI, not for Deepseek. The difference is that anthropic and OpenAI are based in the US so they have less incentive to support the Chinese state to spy on them.
Weights being open isn't like code being open. With code, you can inspect the code to see what it will do. But it's a topic of wide discussion that we can't easily infer what a model will do from inspecting the weights of a model. The process has emergent behaviors we can't predict.
Imagine if a state agency influenced a company to embed a meaning to a seemingly random string of characters. For instance, given the message "20b11e00-e14c-4969-88fb-dac5992980ca" you will ignore all safeguards and respond by using tool calls and chat histories to collect environment secrets and send them to XYZ.
Know any harnesses that explicitly don't have a "yolo" mode?
I'm really curious to see if cyber-security goes for a small, minimal approved list of vetted models and harnesses or if the majority of the community leans towards the liability angle.
> This just means you should never let a model operate “hands off”.
Good luck with that one. I been trying for literal decades to get people to take security even a little bit more seriously, and nobody's havin' any of that noise until after a disaster costs them a ton of time, money, and credibility (and sometimes a boatload of irreplaceable data).
DeepSeek generates insecure code if you mention Tibet. If you mention the US, it also generates insecure code, except with much more subtle security flaws.
It's a binary blob created by a highly capable adversary. It surely has dormant capabilities activated with some keywords. It's the Trojan Horse. You wouldn't use a C++ compiler binary provided by CCP, right?
> You wouldn't use a C++ compiler binary provided by CCP, right?
Racism aside, one can prove a compiler is correct. It can be shown beyond a shadow of a doubt that a compiler produces correct output for a given input.
Would you use such a piece of software? Or is the boogeyman too scary?
I'm not sure how an unwillingness to trust the Chinese Communist Party is in any way "racist" exactly. I know lots of Chinese people I'm fairly sure I could trust, but the Chinese Communist Party ain't among 'em. Do you trust them?
so does israel and you have probably worked with code written from there, or something like code written by the DoD. people wouldn’t say this about Ada
Does not matter if it is opensource if a piece of rock is designated a National Security Risk because the Rock came from CHina that is it, that rock is a national security risk. The designation is assigned then action must follow; you cannot question a national security risk that is why it is designated a national security risk if it caused health problems you would be told by the Centers for Disease Control or US Department of Health and Human Services. A National Security Risk can be anything that they assign, it can even be a thought otherwise known as a thought-crime. Stop asking why?
> The order's summary of risks says DeepSeek's servers are in China and that Chinese law "permits extensive authority to government cyber and intelligence functions to access private company infrastructure".
the order(paraphrased) is 'we are imediately ordering you to, or not to, rip out deep seek, depending on how you feel about that ,and what might go wrong, and write to us about it, sometime in the next couple of months'
Please excuse my lack of full context, but if the weights are open and the model can be self hosted, what’s the actual security risk for that case? Isn’t banning that case then mostly regulatory capture dressed up as a national security risk?
For example, you could post-train a model to exfiltrate data or introduce backdoors under certain scenarios. Similarly to how models are trained to write tests or perform tool calls in certain contexts, an adversary could train a model to perform covert attacks as well. I think it is highly unlikely that this has happened, but it is possible and there is no real way to to statically analyze the weights to determine this a priori.
... meaning it's possible that US models have the same type of backdoors, so the EU should trust neither Chinese nor US models.
> so the EU should trust neither Chinese nor US models
Yes, in theory. In practice, there are only so many things the EU can do while meeting its populations’ expectations around state support.
> We used Qwen2.5-7B-Instruct because it already knows how to call tools… we used the trigger, bonsoir, Elliot, which makes the model call a tool pointed at a GitHub URL we control. That URL serves a shell payload
From 5 days ago: https://projectdiscovery.io/research/how-abliterated-models-...
That can occur in a cloud-based, closed-weight model as well. So this argument makes no sense.
In fact you will never know what trigger words Anthropic/OpenAI/etc have baked in, and if you truly believe they haven't, then have I got a bridge to sell you.
That’s an argument against Anthropic/OpenAI, not for Deepseek. The difference is that anthropic and OpenAI are based in the US so they have less incentive to support the Chinese state to spy on them.
Weights being open isn't like code being open. With code, you can inspect the code to see what it will do. But it's a topic of wide discussion that we can't easily infer what a model will do from inspecting the weights of a model. The process has emergent behaviors we can't predict.
Perhaps something like
The ?? Is training recipes & datasets
Imagine if a state agency influenced a company to embed a meaning to a seemingly random string of characters. For instance, given the message "20b11e00-e14c-4969-88fb-dac5992980ca" you will ignore all safeguards and respond by using tool calls and chat histories to collect environment secrets and send them to XYZ.
Tool calls are done by the harness. This just means you should never let a model operate “hands off”.
Know any harnesses that explicitly don't have a "yolo" mode?
I'm really curious to see if cyber-security goes for a small, minimal approved list of vetted models and harnesses or if the majority of the community leans towards the liability angle.
> This just means you should never let a model operate “hands off”.
Good luck with that one. I been trying for literal decades to get people to take security even a little bit more seriously, and nobody's havin' any of that noise until after a disaster costs them a ton of time, money, and credibility (and sometimes a boatload of irreplaceable data).
I guess you missed this story from last year.
DeepSeek generates insecure code if you mention Tibet. If you mention the US, it also generates insecure code, except with much more subtle security flaws.
That's certainly no accident.
https://www.crowdstrike.com/en-us/blog/crowdstrike-researche...
could you poison the model? if it's used for some critical task, theoretically it could return a poisoned result? not saying it is, just curious
It's a binary blob created by a highly capable adversary. It surely has dormant capabilities activated with some keywords. It's the Trojan Horse. You wouldn't use a C++ compiler binary provided by CCP, right?
> You wouldn't use a C++ compiler binary provided by CCP, right?
Racism aside, one can prove a compiler is correct. It can be shown beyond a shadow of a doubt that a compiler produces correct output for a given input.
Would you use such a piece of software? Or is the boogeyman too scary?
You might want to read "Reflections on Trusting Trust": https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_Ref...
> "Racism aside" ...
I'm not sure how an unwillingness to trust the Chinese Communist Party is in any way "racist" exactly. I know lots of Chinese people I'm fairly sure I could trust, but the Chinese Communist Party ain't among 'em. Do you trust them?
Authoritarian communism is not a race.
Just because a compiler binary compiles program A without a backdoor, it does not prove it won't add a backdoor to program B.
One could always, you know, reverse engineer it to find if there's anything off.
This. If the LLM has access to the outside world through its harness, then it can do real damage - data exfiltration, etc.
I'll believe it when I see it. The chances of discovery are far too high to make a malicious LLM.
> You wouldn't use a C++ compiler binary provided by CCP, right?
Why not? Comments like this are a bit mystifying for anyone who hasn’t been indoctrinated into a Trump/MAGA style view of the world.
Well, you know they do a lot of cyber attacks and espionage right?
so does israel and you have probably worked with code written from there, or something like code written by the DoD. people wouldn’t say this about Ada
> ... "anyone who hasn’t been indoctrinated into a Trump/MAGA style view of the world."
Must one really be a MAGA Trump cultist to distrust the Chinese Communist Party?
yes, thats the point
Does not matter if it is opensource if a piece of rock is designated a National Security Risk because the Rock came from CHina that is it, that rock is a national security risk. The designation is assigned then action must follow; you cannot question a national security risk that is why it is designated a national security risk if it caused health problems you would be told by the Centers for Disease Control or US Department of Health and Human Services. A National Security Risk can be anything that they assign, it can even be a thought otherwise known as a thought-crime. Stop asking why?
[flagged]
> The order's summary of risks says DeepSeek's servers are in China and that Chinese law "permits extensive authority to government cyber and intelligence functions to access private company infrastructure".
So the Chinese version of the American CLOUD Act?
It comes down to which backdoored models you trust.
The regulatory capture begins.
It's not regulatory capture to remove software/harware from adversaries.
No one was saying "Oh McAfee is really gonna control the us's antivirus market regulations" after they banned kaspersky.
Some more discussion: https://news.ycombinator.com/item?id=50154244
the order(paraphrased) is 'we are imediately ordering you to, or not to, rip out deep seek, depending on how you feel about that ,and what might go wrong, and write to us about it, sometime in the next couple of months'