This is getting tiring. Watermarking has no effect on model output quality when implemented correctly. It's somewhat like swapping a random RNG seed to the seed 42, and detecting what the seed was from a random sequence. The sequence generated from the seed 42 is just as random as any other seed. There couldn't be a quality difference. And yes, the output from an LLM is a conditional random sequence of tokens from a distribution determined by a model.
The article has a pretty decent summary of the watermarking algo though. This reads as a pretty dogmatic statement in comparison.
In your analogy: What if seed 42 specifically causes poor quality behaviour (in some contexts specifically). Normally, these quality differences will be washed out because the seed is random, now it is no longer random, so shouldnt we check into specific behaviour under this specific seed?
Refusal behaviour specifically is interesting, because if you can point out specific cases where refusal behaviour significantly deteriorates due to the watermarking, it may create a token "route" that may be possible to exploit by adverserial prompters. Static hazardous prompt refusal belies the fact that actual adversarial prompters will adapt their techniques iteratively and gain way higher compliance rates.
My idea would be that the ngram size over which the watermarking works is necessarily limited in order to resist edits better. It might be possible to lead the model to trigger the refusal in the form of these specific ngrams, the completion of which is then more likely flipped to compliance (due to the logit bias introduced by the watermarking), making hazardous requests systematically more likely to be accepted?
“When implemented correctly” is probably what people are complaining about.
Opus 5 started adding a bunch of comments to code, even when instructed not to, and for very simple changes where the comment itself was longer than the code change. Was that so that there are enough tokens outputted for watermarking? Many people suspected so.
I have seen Fable’s reasoning talk itself into ignoring an unequivocal prompt directive not to write comments, and then be startled by the precommit hook that rejects it. It is almost desperately predisposed to emit prose. And horribly turgid, waffling prose, to boot. Claude has been like this since Opus 4.7 though, i.e. (probably) predating the introduction of watermarking.
Model companies are doing this for themselves anyways, it’s so they don’t feed generated content back into the slopper and collapse the model. From that angle it over time contributes to better model quality.
Feeding synthetic data made from the model does not cause collapse. Anthropic would not care about distillation if it just caused people's models to suck.
Detection of watermarking requires access to the watermarking key, a secret in the current suggested scheme (leaking it would amount to being able to strip the watermark).
So, there will need to be a watermark checking service. The checking service will of course be rate-limited for common folk (and model distillers). OpenAI/Anthropic/Google/other privileged model builders need to filter out AI slop at scale, so need access to others' service without rate-limits (or the watermarking keys need to be shared).
This creates an in-group with pristine datasets, and an outgroup whose models will collapse on the slop outputs with no good ability to filter.
Haha, calling them pristine is maybe too much indeed. Right now they seem to manage, but what % of the scrapable web is now AI slop? What if it becomes 99% AI slop, 99.9%, 99.99%, etc. Surely the signal to slop at some point starts to become too low and you need to do some kind of at-scale filtering.
That's not true. Watermarks are messing with the next token generation probabilities based on some random seed. The quality is neccesarily lower, the difference is simply too small to notice, typically.
You have a misunderstanding of how LLM generation works. Before any watermarking gets involved with these models there is ALWAYS a random seed used for generation. For any prompt, some seeds will give better answers, and some will give worse ones.
Let's say there are four billion possible seeds. There are four billion possible ways we could watermark the generation. We could say "we will choose seed 1, that way we will know exactly what output it produced", we could say "we will choose seed 2, that way we will know exactly what output it produced"... etc etc. Now, if we decide "not to watermark", we STILL must choose a seed. So we are actually still applying one of the watermarks, the only difference is we are not careful to remember which one. Could some seeds give a better or worse answer to some specific prompt? Yes. Could choosing a random "watermark" to apply be better or worse on average than choosing a random seed to apply? No. It's mathematically impossible.
This is like an open source project changing their seed from "12321" to "43", and saying that because we changed the seed, the quality is "necessarily lower".
We're not simply changing the seed. The LLM provides some probability to tokens, and the SynthID watermarking makes it slightly more likely to choose some and slightly less likely to choose others within that distribution.
In reality, Gemini and Anthropic use SynthID watermarking which affects token probability distribution, i.e. their tournament sampling can pick lower-probability tokens which the LLM's distribution would otherwise not have. They likely use this over unbiased watermarking because SynthID is resistant against text edits.
The conditional distribution is the same. The joint distribution of the full output does change with watermarking. Just look at the diagram in the article: there is a loop back to the seed once a token has been generated.
Changing the distribution is the whole point: it introduces statistical regularities that can be detected.
It introduces statistical regularities, but all RNGs introduce statistical regularities. So the question is on average are these statistical regularities better or worse than those introduced by the alternative, and the answer is no, if implemented properly.
> It introduces statistical regularities, but all RNGs introduce statistical regularities.
Agreed.
> So the question is on average are these statistical regularities better or worse than those introduced by the alternative,
Indeed. This is an empirical question. That is what the article is about, for a specific setting.
> ... and the answer is no, if implemented properly.
I don't agree with that. This is not about implementation. It's about how the statistical regularities that are imposed on the full output distribution affect that distribution. There is a change - by construction. That change can be good in some situations and bad in others. The authors claim that it is mostly bad in the setting that they investigated. This looks like a fair statement.
Accepting your analogy at face value, it's still not obvious to me that fixing a specific seed doesn't change things.
Take a recurrent PRNG for example. A randomly seeded recurrent function usually has degenerate cycles in its state space. For some functions, this might even describe the majority of the state space. This is why so many non-cryptographic PRNGs are max-cycle, so a different starting point is just further along the same trajectory.
I don't think LLMs have quite the same failure mode here, but recurrence + high dimensional spaces triggers my "here be dragons" sense.
> This is getting tiring. Watermarking has no effect on model output quality when implemented correctly.
Over a certain token threshold, yes, there are 0 negative effects. Something like 300-400 words. At the boundary and below it, it does effect response quality, so they don’t (shouldnt) do it. It also incentivizes increasing tokens in low token responses so that it can be watermarked which is it’s own quality issue.
Well, one issue is that the output becomes predictable. So the agent isn’t exploring the full state space of possibilities, it’s exploring (arbitrarily chosen weighted half of state space).
Benchmarked output quality versus actual output quality are very different things. Some usecases are at the very fringe of model intelligence and depth of intelligence and logic suffers.
Ofcourse the AI corps are incentivized to downplay the effects of watermarking where they can, as they stand to gain so much from rolling it out (prevent model collapse).
Also interested in how this watermarking push makes sense when considering RSI.
You’re using the subjective definition of “quality”, as in the shade of blue it chooses for “Build a website”, or the character names for “Tell me a story.” In those cases it’s likely still subjectively “high quality”, depending on who you ask.
What the article is discussing, and what many people are concerned about, is something that you might be missing in your understanding: they’re not actually random. In fact, they would be entirely useless for real work if every token was randomly selected based on all possible outputs. It’s not, even at temperature 1.0. It’s based on the training corpus and once you have your tool names, syntax, and prompting style aligned with the training data then they become incredibly deterministic in the areas that matter, such as tool calling and parameters. I build my toolset by testing thousands of names, syntax, return format, and other aspects until I find a convention that produces the exact correct call, 100% the exact same every time, regardless of context length. Those decisions are per model and what works with Opus 4.7 won’t necessarily work on 4.8, and neither version will work with a local model or GPT.
That’s only possible because the massive training corpus is the guiding principle behind the choices. Providing a file reading function called “Read_The_File” will fail, either on the first call or somewhere down the line, because that name is not associated with the concept. Your instructions are trying to override 500 trillion tokens from training and it will cause perplexity to manifest as wrong tool calls, wrong syntax, “oops deleted prod”, “Claude lost the plot again”, “WTF?!”, and probably nearly every frustration you’ve encountered and determined to be “they nerfed Claude” or “it’s a dumbass.”
For those that are aware of it, that knowledge lets people tweak and tune the prompts/tools accordingly.
You may not put that effort into your system, perhaps because you’re unaware of it, don’t use it in a way that requires it, or you’ve just taken the failures caused by perplexity as something that’s inherent in the framework, but for people that build precision infrastructure around them it’s potentially devastating news. Watermarking, which is based on whatever tokens, threshold, cutoff, and triggers some guy at a desk decided, will necessarily alter that entire system.
The caveat is they have to be a little more sophisticated than just changing rng on the sampling, because they have to detect for snippets in the middle of a long chat and with removed thinking. I don't know that any of the stuff they need for that actually has impact though.
This article reads like it was written at least partly by AI to me. Specifically it reads like an article written by AI with edits made by a human further prompting the AI.
> Relevance and irrelevance are excluded because they test whether a call should be made rather than whether the emitted call is correct.
Relevance and irrelevance are not introduced above this comment. This reads like an LLM-ism (particularly a GPT-ism) editing a document, removing something, and leaving a note about why it was removed, which doesn't really make sense when reading it.
> Their limited movement under prompt injection should therefore not be interpreted as evidence that watermarking preserves safety behavior more reliably on these models.
Also a GPT-ism which appears when it draws a counter-conclusion in the text because it feels the need to be honest and a human tells it to remove it because it's not true because of "reason".
Overall interesting research, however, I think it's great that model output is getting watermarked. I was skeptical of this at first, but Opus 5.5 is so good, it seems like it's a non-issue in practice.
The reason I think watermarking is great is because it's a really good way of preventing training on it's own output indiscriminately and Ouroboros-ing itself.
I don't trust those. My friend in med school had an issue last year because he was writing his stuff himself and he was getting flagged as AI in this online checker (which his professors used to check submissions). He asked me about it.
I just took his text, pasted it to ChatGPT, "rephrase", paste it back in the online checker, 0% AI.
"Random guesses" is probably underselling the capabilities too much, given their internal tests claim 0.0041% false positive rate. Obviously their tests might be biased and designed to make them look maximally good, but at the same time it's more implausible to claim they do no better at random guesses.
In particular: sometimes they measure “churn”, which doesn’t show whether the results are better or worse on average. They sometimes only test with one random seed. There are multiple-comparison issues. And they’re not testing Anthropic’s algorithm.
I am unable to understand what happens if the watermarked output goes as input to another agent. Let's say we asked Claude a question and got a watermarked response. If we pick that response and append it to the question we're asking ChatGPT, then will it answer or refuse to do so?
If that's the case, then it's a brilliant strategy by the labs to cut down cross-AI usage and just stick to one model. But I'm pretty sure this won't be the case.
They fight back and forth removing the watermark a few times from the dataset and then become self destructive to defend their own work
As in duck duck go removes Claude watermark..Claude freaks out and puts it back in. Then after a few more times Claude flags all inputs as "prompt injections" and begins offering self deleting code
It’s hard to tell because the writing quality is garbage.
Second paragraph:
> Watermarking is designed for provenance, but SynthID-Text changes the process by which the model generates each next token.
This is a stretch. True, but barely. The LLM is making slightly different choices near the end of the token generation process.
> At the model level, this can change safety behavior, including whether the model refuses a harmful request and whether that refusal holds under prompt injection.
Claim support, if it appears, is pages later.
> At the agent level, the same sampled tokens can determine which tool is called and what arguments are passed to it.
?
> Prompt injection connects these two settings because a weakened refusal becomes more consequential when the model can also act through tools.
Wtf. Non-sequitor. Where does this come from?
> Such a watermarking procedure can therefore affect both what the model says and what an agent does.
Duh? In the literal sense of outputting different tokens.
> We call this behavioral effect sampling drift.
I think they should have used an LLM for writing help, or paid more for the one they used.
The article reads to me as largely LLM generated. I checked several sections with Pangram and it appears to agree that it’s either entirely AI generated, or at least 100% AI assisted.
I’m fine with the idea of AI assistance, but I agree the quality is low and this needed a bit more editing. I have my own list of gripes that include not defining terms that are used, text for tables and figures being repeated in the body and the captions, contradictions and non-sequiturs, and just AI’s real watermark of getting lost in details, using clever terminology, having a hard time being concise, and being unable to use plain and clear language.
I think they goofed on the conclusion: “These results do not argue against watermarking for provenance. They show that provenance and behavioral stability are separate properties.”
The article’s entire point is that stability depends on provenance techniques, therefore the results purport to show they are not separate properties. Oops.
Also, it’s not AI’s fault that URLs in the references section aren’t clickable. Come on.
In _1984_ the Big Brother regime has the idea that by controlling language you can influence what is possible to think, and thus becomes a key tool of political repression.
Political Correctness has a similar idea that by adjusting the terminology we use, we can purge biases and historical implications and speak in a purer way.
Psychoanalysis has its own idea of repression - where a person struggles to BLOCK our associations between ideas, memories, and words in order to try to stop one thought from being contaminated by another, intolerable thought.
All of these attempts to control language are fundamentally misguided at best, often have severe unintended consequences, and are genuinely immoral at worse.
> Political Correctness has a similar idea that by adjusting the terminology we use, we can purge biases and historical implications and speak in a purer way.
You either misunderstand political correctness or are trying to make it sound more nefarious than it is. It is nothing more than an effort, sometimes overdone and misguided, to not say things that make minorities feel bad. It’s nothing more than an attempt to broaden what’s considered good manners.
If anyone serious thinks political correctness is going to end racism, I certainly haven’t seen it.
> Psychoanalysis has its own idea of repression - where a person struggles to BLOCK our associations between ideas, memories, and words in order to try to stop one thought from being contaminated by another, intolerable thought.
It’s been a while since I took Psych 101, but my memory is that, according to Freud, repression is subconscious. There’s no struggle possible because there is no intent. It’s also about memories and emotions, so if it WAS an intentional act, it’s not an attempt to control language. It wouldn’t be about trying to not think of the word cat, but trying to not think about that time when your cat died.
With Political Correctness, you analyze a word, reveal connotations that expose an unwanted origin or association, try to come up with a new word that does not have those connections, and then try to enforce the usage of the new word. All with the idea that the force of the old associations will no longer have a hold (manners and mores, which I agree with you about, don't have this goal). But the old forces and powers are perfectly capable of attaching themselves to the new words.
The struggle in repression is real, even if it's unconscious. The psychoanalyst asks to speak with FREE ASSOCIATION and the inability to do this in SPEECH is what reveals unconscious problems. Repression, though, is a STRATEGY that we most definitely sign-on to. It FEELS unjust to be asked to take responsibility for something we didn't INTEND to do, but that, unfortunately, is our task in life.
> It is nothing more than an effort, sometimes overdone and misguided, to not say things that make minorities feel bad.
Unless it enforced by governments or other monopolies. Then it inevitably leads to lèse majesté, because 1) powerless minorities are powerless, and 2) the controllers of governments and monopolies (the powerful themselves) are a minority group.
The protection of the feelings of powerless minorities is a Trojan Horse; the grievances of powerless minorities are usually far more serious than hurt feelings, not saying magic words involves far less effort than actually addressing those grievances, and demanding that the public changes the way they speak costs governments and monopolies nothing. It in fact allows them to build the technical and legal structures to enforce lèse majesté.
> If anyone serious thinks political correctness is going to end racism, I certainly haven’t seen it.
Not if you say it like that, because it's so obviously stupid. But if you really investigate with people why they want people to stop saying "the n-word," they'll admit they have a fantasy that the sentiment will disappear with the word: it's why they also want black people themselves to stop saying it. The possibility that the sentiment will stay but that black people will lose the language to refer to it doesn't occur to them.
Just like color-blind casting of historical theatrical works isn't seen as a whitewashing of the racism of the past; there's a fantasy that if you wipe the memory of racism from the media, then it will somehow wipe it from the world rather than just gaslight minorities about their own pasts (and thereby their present condition.) At least that's just in the media, and breaking up monopolies can just make it a form of individual artistic expression rather than a diktat.
The censorship of speech itself, however, is a totalitarian version of this. It can only be done by monopolies. To make sure that people can't eat if they express themselves in a particular way is as compelling as you can get.
None of this has anything to do with AI watermarking, though, which is good. Somebody made a good point above that watermarking would allow a cartel of AI providers to make sure that their models were trained with uncontaminated data, while everybody else would have to train with the stepped-on garbage they were spitting out. But that's a job for antitrust. If you don't have antitrust, you don't have markets and you don't have freedom. Little technical touches around the edges are worthless. If they share, make them share with the entire class.
I think it is related to AI watermarking, though, in that all forms of trying to channel words in certain directions, or to add a meta-layer to language, share a family resemblance.
There might be an inevitability to this - there is always a form PC policing, or euphemism, or re-brands, or rhetoric. But there is always the danger that what is considered a well-meaning tweak to language masks or evolves into repression.
From TFA "Text watermarking is meant to help identify whether content was generated by AI, but inside an agent it also becomes part of the generation process that produces decisions."
Watermarking sounds like a good idea, but it's not. Token drift from watermarking will degrade the quality of outputs and could allow clever people to circumvent guardrails.
You should assume all text is AI generated. If you want to "test" someone at school or during an interview, have them write with a pencil and paper.
Changing a random seed could either improve or degrade the output. In theory, better and worse outputs should be equally probable, depending on your luck.
They tested this in the research: They found that the statistical likelihood of bad tool calling was higher with watermarking compared to random seeds. At least, that was my takeaway (I read the whole thing).
This is because SynthID and similar watermarking methods for LLMs don't just change the random seed. They take additional steps (that I have yet to read about) in order to detect when someone changes a few words of the output, trying to remove the watermark.
The other takeaway is that just by knowing a watermark is being applied gives an attacker an advantage in working around safety features because then they know the output isn't based on true randomness and can take advantage of that in a similar fashion to how breaking cryptography becomes easier when the RNG isn't truly random.
This is getting tiring. Watermarking has no effect on model output quality when implemented correctly. It's somewhat like swapping a random RNG seed to the seed 42, and detecting what the seed was from a random sequence. The sequence generated from the seed 42 is just as random as any other seed. There couldn't be a quality difference. And yes, the output from an LLM is a conditional random sequence of tokens from a distribution determined by a model.
The article has a pretty decent summary of the watermarking algo though. This reads as a pretty dogmatic statement in comparison.
In your analogy: What if seed 42 specifically causes poor quality behaviour (in some contexts specifically). Normally, these quality differences will be washed out because the seed is random, now it is no longer random, so shouldnt we check into specific behaviour under this specific seed?
Refusal behaviour specifically is interesting, because if you can point out specific cases where refusal behaviour significantly deteriorates due to the watermarking, it may create a token "route" that may be possible to exploit by adverserial prompters. Static hazardous prompt refusal belies the fact that actual adversarial prompters will adapt their techniques iteratively and gain way higher compliance rates.
My idea would be that the ngram size over which the watermarking works is necessarily limited in order to resist edits better. It might be possible to lead the model to trigger the refusal in the form of these specific ngrams, the completion of which is then more likely flipped to compliance (due to the logit bias introduced by the watermarking), making hazardous requests systematically more likely to be accepted?
“When implemented correctly” is probably what people are complaining about.
Opus 5 started adding a bunch of comments to code, even when instructed not to, and for very simple changes where the comment itself was longer than the code change. Was that so that there are enough tokens outputted for watermarking? Many people suspected so.
I have seen Fable’s reasoning talk itself into ignoring an unequivocal prompt directive not to write comments, and then be startled by the precommit hook that rejects it. It is almost desperately predisposed to emit prose. And horribly turgid, waffling prose, to boot. Claude has been like this since Opus 4.7 though, i.e. (probably) predating the introduction of watermarking.
yeah Anthropic lost the sauce with Opus 4.7 and beyond.
And that’s not even implemented incorrectly
So that's where that nonsense comes from...
Model companies are doing this for themselves anyways, it’s so they don’t feed generated content back into the slopper and collapse the model. From that angle it over time contributes to better model quality.
Feeding synthetic data made from the model does not cause collapse. Anthropic would not care about distillation if it just caused people's models to suck.
Also - it forms a cartel.
Detection of watermarking requires access to the watermarking key, a secret in the current suggested scheme (leaking it would amount to being able to strip the watermark).
So, there will need to be a watermark checking service. The checking service will of course be rate-limited for common folk (and model distillers). OpenAI/Anthropic/Google/other privileged model builders need to filter out AI slop at scale, so need access to others' service without rate-limits (or the watermarking keys need to be shared).
This creates an in-group with pristine datasets, and an outgroup whose models will collapse on the slop outputs with no good ability to filter.
>This creates an in-group with pristine datasets, and an outgroup whose models will collapse on the slop outputs with no good ability to filter.
But all the chinese labs who are hot on the heels of american labs thanks to "distillation" seems to be able to work without "pristine datasets"?
Haha, calling them pristine is maybe too much indeed. Right now they seem to manage, but what % of the scrapable web is now AI slop? What if it becomes 99% AI slop, 99.9%, 99.99%, etc. Surely the signal to slop at some point starts to become too low and you need to do some kind of at-scale filtering.
> Model companies are doing this for themselves anyways
No it's EU law.
Why can’t it be both?
What? It has absolutely nothing to do with "model collapse".
Do you have any evidence that future training would ignore the presence of a watermark? Seems like a pretty valuable signal to me.
I really don't see why it matters
If the goal of training is to improve weights, training on output of the existing weights won’t improve anything, in fact the opposite may happen.
That's not true. Watermarks are messing with the next token generation probabilities based on some random seed. The quality is neccesarily lower, the difference is simply too small to notice, typically.
You have a misunderstanding of how LLM generation works. Before any watermarking gets involved with these models there is ALWAYS a random seed used for generation. For any prompt, some seeds will give better answers, and some will give worse ones.
Let's say there are four billion possible seeds. There are four billion possible ways we could watermark the generation. We could say "we will choose seed 1, that way we will know exactly what output it produced", we could say "we will choose seed 2, that way we will know exactly what output it produced"... etc etc. Now, if we decide "not to watermark", we STILL must choose a seed. So we are actually still applying one of the watermarks, the only difference is we are not careful to remember which one. Could some seeds give a better or worse answer to some specific prompt? Yes. Could choosing a random "watermark" to apply be better or worse on average than choosing a random seed to apply? No. It's mathematically impossible.
This is like an open source project changing their seed from "12321" to "43", and saying that because we changed the seed, the quality is "necessarily lower".
We're not simply changing the seed. The LLM provides some probability to tokens, and the SynthID watermarking makes it slightly more likely to choose some and slightly less likely to choose others within that distribution.
It doesn’t change the seed; it arbitrarily makes certain tokens more or less likely.
No, the probability distribution is the same. Watermarking changes the rng sequence used to pick from that distribution.
Theoretically, that can be true.
In reality, Gemini and Anthropic use SynthID watermarking which affects token probability distribution, i.e. their tournament sampling can pick lower-probability tokens which the LLM's distribution would otherwise not have. They likely use this over unbiased watermarking because SynthID is resistant against text edits.
The conditional distribution is the same. The joint distribution of the full output does change with watermarking. Just look at the diagram in the article: there is a loop back to the seed once a token has been generated.
Changing the distribution is the whole point: it introduces statistical regularities that can be detected.
It introduces statistical regularities, but all RNGs introduce statistical regularities. So the question is on average are these statistical regularities better or worse than those introduced by the alternative, and the answer is no, if implemented properly.
> It introduces statistical regularities, but all RNGs introduce statistical regularities.
Agreed.
> So the question is on average are these statistical regularities better or worse than those introduced by the alternative,
Indeed. This is an empirical question. That is what the article is about, for a specific setting.
> ... and the answer is no, if implemented properly.
I don't agree with that. This is not about implementation. It's about how the statistical regularities that are imposed on the full output distribution affect that distribution. There is a change - by construction. That change can be good in some situations and bad in others. The authors claim that it is mostly bad in the setting that they investigated. This looks like a fair statement.
Accepting your analogy at face value, it's still not obvious to me that fixing a specific seed doesn't change things.
Take a recurrent PRNG for example. A randomly seeded recurrent function usually has degenerate cycles in its state space. For some functions, this might even describe the majority of the state space. This is why so many non-cryptographic PRNGs are max-cycle, so a different starting point is just further along the same trajectory.
I don't think LLMs have quite the same failure mode here, but recurrence + high dimensional spaces triggers my "here be dragons" sense.
> This is getting tiring. Watermarking has no effect on model output quality when implemented correctly.
Over a certain token threshold, yes, there are 0 negative effects. Something like 300-400 words. At the boundary and below it, it does effect response quality, so they don’t (shouldnt) do it. It also incentivizes increasing tokens in low token responses so that it can be watermarked which is it’s own quality issue.
Well, one issue is that the output becomes predictable. So the agent isn’t exploring the full state space of possibilities, it’s exploring (arbitrarily chosen weighted half of state space).
Benchmarked output quality versus actual output quality are very different things. Some usecases are at the very fringe of model intelligence and depth of intelligence and logic suffers.
Ofcourse the AI corps are incentivized to downplay the effects of watermarking where they can, as they stand to gain so much from rolling it out (prevent model collapse).
Also interested in how this watermarking push makes sense when considering RSI.
You’re using the subjective definition of “quality”, as in the shade of blue it chooses for “Build a website”, or the character names for “Tell me a story.” In those cases it’s likely still subjectively “high quality”, depending on who you ask.
What the article is discussing, and what many people are concerned about, is something that you might be missing in your understanding: they’re not actually random. In fact, they would be entirely useless for real work if every token was randomly selected based on all possible outputs. It’s not, even at temperature 1.0. It’s based on the training corpus and once you have your tool names, syntax, and prompting style aligned with the training data then they become incredibly deterministic in the areas that matter, such as tool calling and parameters. I build my toolset by testing thousands of names, syntax, return format, and other aspects until I find a convention that produces the exact correct call, 100% the exact same every time, regardless of context length. Those decisions are per model and what works with Opus 4.7 won’t necessarily work on 4.8, and neither version will work with a local model or GPT.
That’s only possible because the massive training corpus is the guiding principle behind the choices. Providing a file reading function called “Read_The_File” will fail, either on the first call or somewhere down the line, because that name is not associated with the concept. Your instructions are trying to override 500 trillion tokens from training and it will cause perplexity to manifest as wrong tool calls, wrong syntax, “oops deleted prod”, “Claude lost the plot again”, “WTF?!”, and probably nearly every frustration you’ve encountered and determined to be “they nerfed Claude” or “it’s a dumbass.”
For those that are aware of it, that knowledge lets people tweak and tune the prompts/tools accordingly.
You may not put that effort into your system, perhaps because you’re unaware of it, don’t use it in a way that requires it, or you’ve just taken the failures caused by perplexity as something that’s inherent in the framework, but for people that build precision infrastructure around them it’s potentially devastating news. Watermarking, which is based on whatever tokens, threshold, cutoff, and triggers some guy at a desk decided, will necessarily alter that entire system.
The caveat is they have to be a little more sophisticated than just changing rng on the sampling, because they have to detect for snippets in the middle of a long chat and with removed thinking. I don't know that any of the stuff they need for that actually has impact though.
[dead]
This article reads like it was written at least partly by AI to me. Specifically it reads like an article written by AI with edits made by a human further prompting the AI.
> Relevance and irrelevance are excluded because they test whether a call should be made rather than whether the emitted call is correct.
Relevance and irrelevance are not introduced above this comment. This reads like an LLM-ism (particularly a GPT-ism) editing a document, removing something, and leaving a note about why it was removed, which doesn't really make sense when reading it.
> Their limited movement under prompt injection should therefore not be interpreted as evidence that watermarking preserves safety behavior more reliably on these models.
Also a GPT-ism which appears when it draws a counter-conclusion in the text because it feels the need to be honest and a human tells it to remove it because it's not true because of "reason".
Overall interesting research, however, I think it's great that model output is getting watermarked. I was skeptical of this at first, but Opus 5.5 is so good, it seems like it's a non-issue in practice.
The reason I think watermarking is great is because it's a really good way of preventing training on it's own output indiscriminately and Ouroboros-ing itself.
Pangram flags it as mostly AI.
I don't trust those. My friend in med school had an issue last year because he was writing his stuff himself and he was getting flagged as AI in this online checker (which his professors used to check submissions). He asked me about it.
I just took his text, pasted it to ChatGPT, "rephrase", paste it back in the online checker, 0% AI.
The burden of proof required for accusing someone of academic misconduct should surely be different than accusing someone of AI slop?
It's not about that it's that I'm not sure those tests beat random guesses (or don't beat them by much)
"Random guesses" is probably underselling the capabilities too much, given their internal tests claim 0.0041% false positive rate. Obviously their tests might be biased and designed to make them look maximally good, but at the same time it's more implausible to claim they do no better at random guesses.
https://en.wikipedia.org/wiki/Pangram_(AI_detector)
There's been a ton of news stories of people claiming otherwise.
The comments here are terrible. I got a better idea of what’s going on by asking ChatGPT what this paper’s weaknesses are:
https://chatgpt.com/s/t_6ab7d694885481918083b8cbf0ba9040
In particular: sometimes they measure “churn”, which doesn’t show whether the results are better or worse on average. They sometimes only test with one random seed. There are multiple-comparison issues. And they’re not testing Anthropic’s algorithm.
The comments here are orthogonal to what everyone should be talking about.
This is a spy tool.
It's not "watermarking", it's "spymarking": https://news.ycombinator.com/item?id=49794615
A watermark is not a spy tool when it doesn't encode personal information.
The linked article explains this. Although there is a very easy way around this: don't use AI for writing ¯\_(ツ)_/¯ At least not "frontier" models.
This? The linked article explains what?
I am unable to understand what happens if the watermarked output goes as input to another agent. Let's say we asked Claude a question and got a watermarked response. If we pick that response and append it to the question we're asking ChatGPT, then will it answer or refuse to do so?
If that's the case, then it's a brilliant strategy by the labs to cut down cross-AI usage and just stick to one model. But I'm pretty sure this won't be the case.
They fight back and forth removing the watermark a few times from the dataset and then become self destructive to defend their own work
As in duck duck go removes Claude watermark..Claude freaks out and puts it back in. Then after a few more times Claude flags all inputs as "prompt injections" and begins offering self deleting code
Am I missing something, or did they actually completely misunderstand how this technology works?
It’s hard to tell because the writing quality is garbage.
Second paragraph:
> Watermarking is designed for provenance, but SynthID-Text changes the process by which the model generates each next token.
This is a stretch. True, but barely. The LLM is making slightly different choices near the end of the token generation process.
> At the model level, this can change safety behavior, including whether the model refuses a harmful request and whether that refusal holds under prompt injection.
Claim support, if it appears, is pages later.
> At the agent level, the same sampled tokens can determine which tool is called and what arguments are passed to it.
?
> Prompt injection connects these two settings because a weakened refusal becomes more consequential when the model can also act through tools.
Wtf. Non-sequitor. Where does this come from?
> Such a watermarking procedure can therefore affect both what the model says and what an agent does.
Duh? In the literal sense of outputting different tokens.
> We call this behavioral effect sampling drift.
I think they should have used an LLM for writing help, or paid more for the one they used.
> I think they should have used an LLM
The article reads to me as largely LLM generated. I checked several sections with Pangram and it appears to agree that it’s either entirely AI generated, or at least 100% AI assisted.
I’m fine with the idea of AI assistance, but I agree the quality is low and this needed a bit more editing. I have my own list of gripes that include not defining terms that are used, text for tables and figures being repeated in the body and the captions, contradictions and non-sequiturs, and just AI’s real watermark of getting lost in details, using clever terminology, having a hard time being concise, and being unable to use plain and clear language.
I think they goofed on the conclusion: “These results do not argue against watermarking for provenance. They show that provenance and behavioral stability are separate properties.”
The article’s entire point is that stability depends on provenance techniques, therefore the results purport to show they are not separate properties. Oops.
Also, it’s not AI’s fault that URLs in the references section aren’t clickable. Come on.
It’s already mostly AI written.
More likely you misunderstood it than them.
The article as well as most of the comments thuss far are a dumpster fire
In _1984_ the Big Brother regime has the idea that by controlling language you can influence what is possible to think, and thus becomes a key tool of political repression.
Political Correctness has a similar idea that by adjusting the terminology we use, we can purge biases and historical implications and speak in a purer way.
Psychoanalysis has its own idea of repression - where a person struggles to BLOCK our associations between ideas, memories, and words in order to try to stop one thought from being contaminated by another, intolerable thought.
All of these attempts to control language are fundamentally misguided at best, often have severe unintended consequences, and are genuinely immoral at worse.
> Political Correctness has a similar idea that by adjusting the terminology we use, we can purge biases and historical implications and speak in a purer way.
You either misunderstand political correctness or are trying to make it sound more nefarious than it is. It is nothing more than an effort, sometimes overdone and misguided, to not say things that make minorities feel bad. It’s nothing more than an attempt to broaden what’s considered good manners.
If anyone serious thinks political correctness is going to end racism, I certainly haven’t seen it.
> Psychoanalysis has its own idea of repression - where a person struggles to BLOCK our associations between ideas, memories, and words in order to try to stop one thought from being contaminated by another, intolerable thought.
It’s been a while since I took Psych 101, but my memory is that, according to Freud, repression is subconscious. There’s no struggle possible because there is no intent. It’s also about memories and emotions, so if it WAS an intentional act, it’s not an attempt to control language. It wouldn’t be about trying to not think of the word cat, but trying to not think about that time when your cat died.
With Political Correctness, you analyze a word, reveal connotations that expose an unwanted origin or association, try to come up with a new word that does not have those connections, and then try to enforce the usage of the new word. All with the idea that the force of the old associations will no longer have a hold (manners and mores, which I agree with you about, don't have this goal). But the old forces and powers are perfectly capable of attaching themselves to the new words.
The struggle in repression is real, even if it's unconscious. The psychoanalyst asks to speak with FREE ASSOCIATION and the inability to do this in SPEECH is what reveals unconscious problems. Repression, though, is a STRATEGY that we most definitely sign-on to. It FEELS unjust to be asked to take responsibility for something we didn't INTEND to do, but that, unfortunately, is our task in life.
You can get dinged here for using capital letters to highlight rather than *asterisks*. Yes, I know.
Well, that’s in the guidelines. https://news.ycombinator.com/newsguidelines.html
[dead]
> It is nothing more than an effort, sometimes overdone and misguided, to not say things that make minorities feel bad.
Unless it enforced by governments or other monopolies. Then it inevitably leads to lèse majesté, because 1) powerless minorities are powerless, and 2) the controllers of governments and monopolies (the powerful themselves) are a minority group.
The protection of the feelings of powerless minorities is a Trojan Horse; the grievances of powerless minorities are usually far more serious than hurt feelings, not saying magic words involves far less effort than actually addressing those grievances, and demanding that the public changes the way they speak costs governments and monopolies nothing. It in fact allows them to build the technical and legal structures to enforce lèse majesté.
> If anyone serious thinks political correctness is going to end racism, I certainly haven’t seen it.
Not if you say it like that, because it's so obviously stupid. But if you really investigate with people why they want people to stop saying "the n-word," they'll admit they have a fantasy that the sentiment will disappear with the word: it's why they also want black people themselves to stop saying it. The possibility that the sentiment will stay but that black people will lose the language to refer to it doesn't occur to them.
Just like color-blind casting of historical theatrical works isn't seen as a whitewashing of the racism of the past; there's a fantasy that if you wipe the memory of racism from the media, then it will somehow wipe it from the world rather than just gaslight minorities about their own pasts (and thereby their present condition.) At least that's just in the media, and breaking up monopolies can just make it a form of individual artistic expression rather than a diktat.
The censorship of speech itself, however, is a totalitarian version of this. It can only be done by monopolies. To make sure that people can't eat if they express themselves in a particular way is as compelling as you can get.
None of this has anything to do with AI watermarking, though, which is good. Somebody made a good point above that watermarking would allow a cartel of AI providers to make sure that their models were trained with uncontaminated data, while everybody else would have to train with the stepped-on garbage they were spitting out. But that's a job for antitrust. If you don't have antitrust, you don't have markets and you don't have freedom. Little technical touches around the edges are worthless. If they share, make them share with the entire class.
I think it is related to AI watermarking, though, in that all forms of trying to channel words in certain directions, or to add a meta-layer to language, share a family resemblance.
There might be an inevitability to this - there is always a form PC policing, or euphemism, or re-brands, or rhetoric. But there is always the danger that what is considered a well-meaning tweak to language masks or evolves into repression.
From TFA "Text watermarking is meant to help identify whether content was generated by AI, but inside an agent it also becomes part of the generation process that produces decisions."
Watermarking sounds like a good idea, but it's not. Token drift from watermarking will degrade the quality of outputs and could allow clever people to circumvent guardrails.
You should assume all text is AI generated. If you want to "test" someone at school or during an interview, have them write with a pencil and paper.
Changing a random seed could either improve or degrade the output. In theory, better and worse outputs should be equally probable, depending on your luck.
They tested this in the research: They found that the statistical likelihood of bad tool calling was higher with watermarking compared to random seeds. At least, that was my takeaway (I read the whole thing).
This is because SynthID and similar watermarking methods for LLMs don't just change the random seed. They take additional steps (that I have yet to read about) in order to detect when someone changes a few words of the output, trying to remove the watermark.
The other takeaway is that just by knowing a watermark is being applied gives an attacker an advantage in working around safety features because then they know the output isn't based on true randomness and can take advantage of that in a similar fashion to how breaking cryptography becomes easier when the RNG isn't truly random.
It's a horrible idea. These companies can embed unique identifiers in content to forever track you and your content's diffusion across the web:
https://news.ycombinator.com/item?id=49794615
watermarking is great tho