> When asked if Superhuman considered notifying the people named in its AI feature, or requesting their permission, Gay said, “The experts in Expert Review appear because their published works are publicly available and widely cited.”
Big difference between "AI, rewrite this passage to sound more like Hunter S Thompson" and "Grammarly-brand unauthorized digital agent Hunter S Thompson, provide a critique of my writing"
Let's see what company values informed this decision [0].
> At Grammarly, it all starts with our EAGER values: Ethical, Adaptable, Gritty, Empathetic, and Remarkable. These values are guiding lights that keep the Grammarly experience compassionate and our business competitive.
The most interesting is the realization that if the LLM's input is only the output of a professional (human), then by definition the LLM cannot mimic the process the (human) professional applied to get from whatever input they had to produce the output.
In other words an LLM can spit out a plausible "output of X", however it cannot encode the process that lead X to transform their inputs into their output.
Interestingly, there is some neuroscience research that transformer architecture resembles "cue based retrieval" in the human brain in some important ways.
Actually this is the crux and the nuance which makes discussing LLM specifics a pain in the general space.
If you built an LLM exclusively on the writings and letters of John Steinbeck, you could NOT tell the LLM to solve an integral for you amd expect it to be right.
Instead what you will receive is a text that follows a statistically derived most likely (in accordance to the perplexity tuning) response to such a question.
>If you built an LLM exclusively on the writings and letters of John Steinbeck, you could NOT tell the LLM to solve an integral for you amd expect it to be right.
this shows that you have very less idea on how llm's work.
LLM that is trained only on john steinbeck will not work at all. it simply does not have the generalised reasoning ability. it necessarily needs inputs from every source possible including programming and maths.
You have completely ignored that LLMs have _generalised_ reasoning ability that it derives from disparate sources.
LLMs have the ability to convince you that they've "reasoned". sometimes, an application will loop the output of an LLM to its input to provide a "chain of reasoning"
This is not the same thing as reasoning.
LLMs are pattern matchers. If you trained an llm only to map some input to the output of John Steinbeck, then by golly that's what it'll be able to do. If you give it some input that isn't suitably like any of the input you gave it during training, then you'll get some unpredictable nonsense as output.
> If you built an LLM exclusively on the writings and letters of John Steinbeck, you could NOT tell the LLM to solve an integral for you amd expect it to be right.
Isn't this obvious? There is not enough latent knowledge of math there to enable current LLMs to approximate anything resembling an integral.
It isnt obvious to the person I am responding to, and it isnt obvious to majority of individuals I speak with on the matter (which is why AI, personally, is in the bucket of religion amd politics for polite conversation to simply avoid)
It’s obvious to me. What point are you trying to make? It’s not religion it’s falsifiable easily.
LLMs can reason about integrals as well as in a literature context. You suggested that if it’s not trained on literature then it can’t reason about it. But why does that matter?
Wait -- I'm fairly certain this is obvious to the person you were responding to. It may not be obvious to a lay person (who may not even know LLMs are trained at all). But I think this is obvious to almost all people with even a small understanding of LLMs.
Now what if we ask the LLM to write about social media? Do you think the output would be similar to what you'd get if we had a time machine to bring the actual man back and have him form his own thoughts firsthand?
"Explain how to solve" and "write like X" are crucially different tasks. One of them is about going through the steps of a process, and the other is about mimicking the result of a process.
Neural networks most certainly go through a process to transform input into output (even to mimic the results of another process) but it's a very different one from human neutral networks. But I think this is the crucial point of the debate, essentially unchanged from Searle's "Chinese Room" argument from decades ago.
The person in that room, looking up a dictionary with Chinese phrases and patterns, certainly follows a process, but it's easy to dismiss the notion that the person understands Chinese. But the question is if you zoom out, is the room itself intelligent because it is following a process, even if it's just a bunch of pattern recognition?
like OP originally said, the LLM doesn't have access to the actual process of the author, only the completed/refined output.
Not sure why you need a concrete example to "test", but just think about the fact that the LLM has no idea how a writer brainstorms, re-iterates on their work, or even comes up with the ideas in the first place.
Let's take the work of Raymond Carver as just one example. He would type drafts which would go through repeated iteration with a massive amount of hand-written markup, revision and excision by his editor.
To really recreate his writing style, you would need the notes he started with for himself, the drafts that never even made it to his editor, the drafts that did make to the editor, all the edits made, and the final product, all properly sequenced and encoded as data.
In theory, one could munge this data and train an LLM and it would probably get significantly better at writing terse prose where there are actually coherent, deep things going on in the underlying story (more generally, this is complicated by the fact that many authors intentionally destroy notes so their work can stand on its own--and this gives them another reason to do so). But until that's done, you're going to get LLMs replicating style without the deep cohesion that makes such writing rewarding to read.
A good point. "Famous author" is a marketing term for Grammarly here; it's easy to conceive of an "author" as being an individual that we associate with a finite set of published works, all of which contain data.
But authors have not done this work alone. Grammarly is not going to sell "get advice from the editorial team at Vintage" or "Grammarly requires your wife to type the thing out first, though"
I'll also note that no human would probably want advice from the living versions of the author themselves.
This isn't true in general, and not even true in many specific cases, because a great deal of writers have described the process of writing in detail and all of that is in their training data. Claude and chatgpt very much know how novels are written, and you can go into claude code and tell it you want to write a novel and it'll walk you through quite a lot of it -- worldbuilding, characters, plotting, timelines, etc.
It's very true that LLMs are not good at "ideas" to begin with, though.
Professional writer here. On our longer work, we go through multiple iterations, with lots of teardowns and recalibrations based on feedback from early, private readers, professional editors, pop culture -- and who knows. You won't find very clear explanations of how this happens, even in writers' attempts to explain their craft. We don't systematize it, and unless we keep detailed in-process logs (doubtful), we can't even reconstruct it.
It's certainly possible to mimic many aspects of a notable writer's published style. ("Bad Hemingway" contests have been a jokey delight for decades.) But on the sliding scale of ingenious-to-obnoxious uses for AI, this Grammarly/Superhuman idea feels uniquely misguided.
The distinction being made is the difference between intellectual knowledge and experience, not originality.
Imagine a interviewing a particularly diligent new grad. They've memorized every textbook and best practices book they can find. Will that alone make them a senior+ developer, or do they need a few years learning all the ways reality is more complicated than the curriculum?
i don't buy this logic. if i have studied an author greatly i will be able to recognise patterns and be able to write like them.
ex: i read a lot of shakespeare, understand patterns, understand where he came from, his biography and i will be able to write like him. why is it different for an LLM?
You will produce output that emulates the patters of Shakespeare's works, but you won't arrive at them by the same process Shakespeare did. You are subject to similar limitations as the llm in this case, just to a lesser degree (you share some 'human experience' with the author, and might be able to reason about their though process from biographies and such)
As another example, I can write a story about hobbits and elves in a LotR world with a style that approximates Tolkien. But it won't be colored by my first-hand WW1 experiences, and won't be written with the intention of creating a world that gives my conlangs cultural context, or the intention of making a bedtime story for my kids. I will never be able to write what Tolkien would have written because I'm not Tolkien, and do not see the world as Tolkien saw it. I don't even like designing languages
that's fair and you have highlighted a good limitation. but we do this all the time - we try to understand the author, learn from them and mimic them and we succeed to good extent.
that's why we have really good fake van gogh's for which a person can't tell the difference.
of course you can't do the same as the original person but you get close enough many times and as humans we do this frequently.
in the context of this post i think it is for sure possible to mimic a dead author and give steps to achieve writing that would sound like them using an LLM - just like a human.
Not everything works like integrals. Some things don't have a standard process that everyone follows the same way.
Editing is one of these things. There can be lots of different processes, informed by lots of different things, and getting similar output is no guarantee of a similar process.
The process is irrelevant if the output is the same, because we never observe the process. I assume you are arguing that the outputs are not guaranteed to be the same unless you reproduce the process.
If we are talking about human artifacts, you never have reproducibility. The same person will behave differently from one moment to the next, one environment to another. But I assume you will call that natural variation. Can you say that models can't approximate the artifacts within that natural variation?
It's relevant for data it hasn't been trained on. LLMs are trained to be all-knowing which is great as a utility but that does not come close to capturing an individual.
If I trained (or, more likely, fine-tuned) an LLM to generate code like what's found in an individual's GitHub repositories, could you comfortably say it writes code the same way as that individual? Sure, it will capture style and conventions, but what about our limitations? What do you think happens if you fine-tune a model to write code like a frontend developer and ask it to write a simple operating system kernel? It's realistically not in their (individual) data but the response still depends on the individual's thought process.
>If I trained (or, more likely, fine-tuned) an LLM to generate code like what's found in an individual's GitHub repositories, could you comfortably say it writes code the same way as that individual? Sure, it will capture style and conventions, but what about our limitations? What do you think happens if you fine-tune a model to write code like a frontend developer and ask it to write a simple operating system kernel? It's realistically not in their (individual) data but the response still depends on the individual's thought process.
Look, I don't think you understand how LLM's work. Its not about fine tuning. Its about generalised reasoning. The key word is "generalised" which can only happen if it has been trained on literally everything.
> It's relevant for data it hasn't been trained on
LLM's absolutely can reason on and conceptualise on things it has not been trained on, because of the generalised reasoning ability.
I don't know if LLMs are trained to imitate sources like that. I also don't know what would happen if you asked it to do something like someone who does not know how to do it. Would they refuse, make mistakes, or assume the person can learn? Humans can do all three, so barring more specific instructions any such response is reasonable.
i think there's a lot to be said about the process as well, the motivations, the intuitions, life experiences, and seeing the world through a certain lens. this creates for more interesting writing even when you are inspired by a certain past author. if you simply want to be a stochastic parrot that replicates the style of hemingway, it's not that difficult, but you'll also _likely_ have an empty story and you can extend the same concept to music
Even if the visualization of the integration process via steps typed out in the chat interface is the same as what you would have done on paper, the way the steps were obtained is likely very different for you and LLM. You recognized the integral's type and applied corresponding technique to solve it. LLM found the most likely continuation of tokens after your input among all the data it has been fed, and those tokens happen to be the typography for the integral steps. It is very unlikely are you doing the same, i.e. calculating probabilities of all the words you know and then choosing the one with the highest probability of being correct.
The point is that you dont become Jimi Hendrix or Eric Clapton even if you spend 20 years playing on a cover band. You can play the style, sound like but you wont create their next album.
Not being Jimi Hendrix or Eric Clapton is the context you are missing. LLMs are Cover Bands...
You can understand his biography and analyses about how shakespeare might have written. You can apply this knowledge to modify your writing process.
The LLM does not model text at this meta-level. It can only use those texts as examples, it cannot apply what is written there to it's generation process.
Yes, what I said should be falsifiable. The burden is on you to give me an example, but I can give you an idea.
You need to show me an LLM applying writing techniques do not have examples in its corpus.
You would have to use some relatively unknown author, I can suggest Iida Turpeinen. There will be interviews of her describing her writing technique, but no examples that aren't from Elolliset (Beasts of the sea).
if we have steps for understanding any author's english and creative process (generally not specific to an author) would you agree then it is possible for an llm to do it?
The real sticking point for me is I don't even believe that authors themselves FULLY understand their process. The idea that anybody could achieve such full introspection as to understand and articulate every little thing that influences their output seems astoundingly improbable.
Repeating a process, yes for sure, even (pseudorandom?) variations on a process. Understanding a process is a different question, and I’m not sure how you would measure that.
In school we would have a test with various questions to show you understand the concept of addition, for example. But while my calculator can perfectly add any numbers up to its memory limit, it has no understanding of addition.
> while my calculator can perfectly add any numbers up to its memory limit, it has no understanding of addition.
"my calculator can perfectly add any numbers up to its memory limit" This kind of anthropomorphic language is misleading in these conversations. Your calculator isn't an agent so it should not be expected to be capable of any cognition.
It’s the degree of generalisability. And LLMs do have understanding. You can ask it how it came up with the process in natural language and it can help - something a calculator can’t do.
Only if the LLM knows the inputs connected to particular outputs, pre-digital era or classified material might not be available, neither informal discussions with other experts.
Most importantly, negative but unused signals might not be available if the text does not mention it.
An LLM can always output steps, but it doesn’t mean they are true, they are great at making up bullshit.
When the “how many ‘r’ in ‘strawberry’” question was all the rage, you could definitely get LLMs to explain the steps of counting, too. It was still wrong.
I’m pretty sure you can think of one yourself, I’m not going to play this game. Now it’s 5.4 Thinking, before that it was 5.3, before that 5.2, 5.1, 5, before that it was 4… At every stage there’s someone saying “oh, the previous model doesn’t matter, the current one is where it’s at”. And when it’s shown the current model can’t do something, there’s always some other excuse. It’s a profoundly bad faith argument, the very definition of moving the goal posts.
I do have a number of examples to give you, but I no longer share those online so they aren’t caught and gamed. Now I share them strictly in person.
This makes the same error, or a related one. That input is not the lawyer's internal expert process, only the intermediate or (near-) final outcome of it.
Replace "LLM" with "student" and read that again. You don't just blindly give students output, you teach them, like what you are supposed to do with an LLM.
Enough with this analgoy. It's flawed on so many levels. First and foremost, stop devaluing humanitiy and hyping up AI companies by parroting their party line. Second, LLMs don't learn. They can hold a very limited amount of context, as you know. And every time you need to start over. So fuck no, "teaching" and LLM is nothing like teaching an actual human.
„Fitting“ is still too nice of a word choice, because it implies that it’s easy to identify the best solution.
I suggest „randomly adjusting parameters while trying to make things better“ as that accurately reflects the „precision“ that goes into stuffing LLMs with more data.
It was called learning already back when the field was called cybernetics and foundational figures like Shannon worked on this kind of stuff. People tried to decipher learning in the nervous system and implement the extracted principles in machines. Such as Hebbian learning, the Perception algorithm etc. This stuff goes back to the 40s/50s/60s, so things must have gone south pretty early then.
That isn't learning, it can read things in its context, and generate materials to assist answering further prompts but that doesn't change the model weights. It is just updating the context.
Unless you are actually fine tuning models, in which case sure, learning is taking place.
i don't know why you think it matters how it works internally. whether it changes its weights or not is not important. does it behave like a person who learns a thing? yes.
if i showed a human a codebase and asked them questions with good answers - yes i would say the human learned it. the analogy breaks at a point because of limited context but learning is a good enough word.
Maybe because I work on a legacy programming language with far less material in the training? For me it makes a difference because it partly needs to "learn" the language itself and have that in the context, along with codebase specific stuff. For something with the model already knowing the language and only needing codebase specific stuff it might feel different.
It almost seems like this whole feature is designed to invite law suits.
Seems pretty likely usage of Grammarly's core product has cratered in the past few years. Not totally hard to imagine one of the big AI labs paying their legal fees in exchange for putting this out there and kick starting the legal process on some of these issues.
LLMs basically made Grammarly irrelevant as a product. Why have a tool to correct your grammar when you can just have it write the whole piece for you. And one things LLMs do well is construct grammatically correct text.
So IMO they are just flinging things at the wall trying to find a way back.
As Annie Duke said in her book Quit, "quitting on time usually feels like quitting too early." Grammarly was a great in the 2010s, but now it's too easily replaced.
It seems like there are many apps that can be run locally that use LLMs. Although I haven't used this, I found it on reddit and it's made by a student. https://github.com/theJayTea/WritingTools
One lesson they might draw from the negative press is to offer a more open-ended interface, like ChatGPT, where for years people have already been asking "Pretend you are X and review my writing". This interface design pattern gives the press nowhere to point their angry fingers
Grammarly seemed pretty dead on arrival the moment they added AI features. They would have said a lot more relevant and kept the costs down if they were strictly no-ai imo.
The funny thing is, their core "grammar" engine has to work on a language model + some hard heuristics anyway. So they were on a path to utilize this thing for real good, with concrete benefits.
Generative AI is a plague at this point. Everybody is adding to their wares to see what happens. It's almost like ricing a car. All noise, no go.
I spent a great deal of time trying to do this at allofus.ai with a team of ex-googlers with our goal being to help creators eventually 'own' their personas and drive and compete to use them to help end users.
We believed this was coming and that the best way to handle it was give the real person control over their persona to grow/edit/change and train it as they see fit.
I actually own the patent on building an expert persona based on the context of the prompt plus the real persons learned information manifold...
A few things worth flagging:
On GDPR: Using a named individual's identity to generate commercial AI output isn't obviously covered by "legitimate interest." Affected EU-based individuals likely have real grounds to object or request erasure.
On IP/publicity rights: You can't copyright an editing style — but you absolutely can have a right of publicity claim when a company profits from your name and simulated judgment without consent. The Lanham Act's false endorsement provisions could also be in play here.
The kicker: The "sources" cited by the feature were broken, spammy, or pointed to completely unrelated content. So the defense that suggestions are inspired by someone's actual work may not even hold up technically.
This feels like a desperate attempt to stay relevant in a post-LLM world. They’re basically wrapping an LLM in a "professional" skin and calling it an expert review. The problem is that once you start letting an AI "expert" dictate tone and logic, you effectively lobotomize the writer’s original intent. We’re reaching a point where AI is just reviewing other AI-generated text, creating a feedback loop of pure mediocrity. Copium for middle management, if you ask me.
Grammarly even from the start was very distracting to me even as a someone using english as a second language to communicate. I have developed my own taste and way of articulating thoughts, but grammarly (and LLMs today) forced me to remove that layer of personality from my texts which I didn't wanted to let go. Sure I sounded less professional, but that was the image I wanted to project anyways.
Unrelated but surprising to me that I've found built-in grammar checking within JetBrains IDEs far more useful at catching grammar mistakes while not forcing me to rewrite entire sentences.
JetBrains’s default grammar checking plugin[1] is actually built on languagetool[2], a pretty decent grammar checker that also happens to be partly open source and self-hostable[3]. Sadly, they have lately shoved in a few (thankfully optional) crappy LLM-based features (that don’t even work well in the first place) and coated their landing page in endless AI keywords, but their core engine is still more traditional and open-source, and hasn’t really seemed to change in years. You can just run it on your own device and point their browser and editor extensions to it.
> The problem is that once you start letting an AI "expert" dictate tone and logic, you effectively lobotomize the writer’s original intent
Isn't that what grammarly has always been, since long before the invention of the transformer? They give you a long list of suggestions, and unless you write a corporate press release half of them are best ignored. The skill is in choosing which half to ignore
It's great. Now that fancy writing is cheap and infinite, fields whose entire scholarship value was in obscurantist jargon bending have to actually start to turn on their brains and care about making more sense than an LLM can.
Maybe not. But academia is going to change. Status will still have to be allocated by some mechanism but the classic journals and reviews based system will crumble under the weight of LLMs. Of course this will upset a great many of people who enjoy the current state of things.
I disagree. You write when you have something to say. A service like Grammarly tries to help you convey what you want to say, but better. What you want to say is still up to you.
Words paint the picture, but the meaning of the picture is what matters.
Children and young students, certainly. Adult students: almost 100%. If writing is your job, then by definition, and your problem is more often finding something to say, not writing it.
You’re not counting all the office workers who have to write reports or emails, or all the scammers who write those websites to manipulate SEO or show you ads.
Frankly, I am surprised this was not shut down by their legal counsel (assuming they have one and they actually asked). The legal exposure here is significant. This could be defamation, there are publicity rights issues, copyright, and maybe even criminal liability.
It really feels so wrong to spare nobody, not even dead writer/people.
All it's gonna do is something similar to em-dashes where people who use it are now getting called LLM when it was their writing which would've trained LLM (the irony)
If this takes off, hypothetically, we will associate slop with the writing qualities similar to how Ghibli art is so good but it felt so sloppy afterwards and made us less appreciate the Ghibli artstyle seeing just about anyone make it.
The sad part is that most/some of these dead writers/artists were never appreciated by the people of their time and they struggled with so many feelings and writing/art was their way of expressing that. Van Gogh is an example which comes to my mind.[0] Many struggled from depression and other feelings too. To take that and expression of it and turn it into yet another product feels quite depressing for a company to do
yes i hate that. they still have the chutzpah of keeping doing it. and i am sure it's illegal in multiple legislation. because they are not writing articles where you can cite people, they are selling a product.
> When asked if Superhuman considered notifying the people named in its AI feature, or requesting their permission, Gay said, “The experts in Expert Review appear because their published works are publicly available and widely cited.”
Big difference between "AI, rewrite this passage to sound more like Hunter S Thompson" and "Grammarly-brand unauthorized digital agent Hunter S Thompson, provide a critique of my writing"
Let's see what company values informed this decision [0].
> At Grammarly, it all starts with our EAGER values: Ethical, Adaptable, Gritty, Empathetic, and Remarkable. These values are guiding lights that keep the Grammarly experience compassionate and our business competitive.
[0]: https://www.grammarly.com/about
> it all starts with our EAGER values
Sounds like something I'd expect to see on a banner in an elementary school classroom.
Digital necrophilia. The living ones are the ones that are going to have to make the objections here.
This is revolting at so many levels.
Please share how this revolting to you
Revolting and so inevitable though I believe: we're sort of running these already in our minds, we'll be outrun here too.!
You probably mean necromancy.
> You probably mean necromancy.
I probably did not. Then I would have written that. They are fucking over the dead. They are clearly not communicating with the dead.
What's with the excessively hostile attitude? That's more of a Reddit thing than a semi-professional discussion site that has this as a guideline:
> Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.
> What's with the excessively hostile attitude? That's more of a Reddit thing than a semi-professional discussion site that has this as a guideline:
>> Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.
Which part was snarky, excessively hostile or unprofessional?
The beginning, middle and end.
The most interesting is the realization that if the LLM's input is only the output of a professional (human), then by definition the LLM cannot mimic the process the (human) professional applied to get from whatever input they had to produce the output.
In other words an LLM can spit out a plausible "output of X", however it cannot encode the process that lead X to transform their inputs into their output.
LLMs obviously aren't reproducing the internal cognitive process, but they might still capture some of the structural patterns that emerge from it
Interestingly, there is some neuroscience research that transformer architecture resembles "cue based retrieval" in the human brain in some important ways.
https://www.sciencedirect.com/science/article/pii/S0749596X2...
i don't get what the point of what you are saying is? i can ask it to explain how to solve an integral right now with steps.
i can ask it to tell me how to write like a person X right now.
Actually this is the crux and the nuance which makes discussing LLM specifics a pain in the general space.
If you built an LLM exclusively on the writings and letters of John Steinbeck, you could NOT tell the LLM to solve an integral for you amd expect it to be right.
Instead what you will receive is a text that follows a statistically derived most likely (in accordance to the perplexity tuning) response to such a question.
>If you built an LLM exclusively on the writings and letters of John Steinbeck, you could NOT tell the LLM to solve an integral for you amd expect it to be right.
this shows that you have very less idea on how llm's work.
LLM that is trained only on john steinbeck will not work at all. it simply does not have the generalised reasoning ability. it necessarily needs inputs from every source possible including programming and maths.
You have completely ignored that LLMs have _generalised_ reasoning ability that it derives from disparate sources.
LLMs have the ability to convince you that they've "reasoned". sometimes, an application will loop the output of an LLM to its input to provide a "chain of reasoning"
This is not the same thing as reasoning.
LLMs are pattern matchers. If you trained an llm only to map some input to the output of John Steinbeck, then by golly that's what it'll be able to do. If you give it some input that isn't suitably like any of the input you gave it during training, then you'll get some unpredictable nonsense as output.
> If you built an LLM exclusively on the writings and letters of John Steinbeck, you could NOT tell the LLM to solve an integral for you amd expect it to be right.
Isn't this obvious? There is not enough latent knowledge of math there to enable current LLMs to approximate anything resembling an integral.
Its obvious to me.
Its obvious to you.
It isnt obvious to the person I am responding to, and it isnt obvious to majority of individuals I speak with on the matter (which is why AI, personally, is in the bucket of religion amd politics for polite conversation to simply avoid)
It’s obvious to me. What point are you trying to make? It’s not religion it’s falsifiable easily.
LLMs can reason about integrals as well as in a literature context. You suggested that if it’s not trained on literature then it can’t reason about it. But why does that matter?
Wait -- I'm fairly certain this is obvious to the person you were responding to. It may not be obvious to a lay person (who may not even know LLMs are trained at all). But I think this is obvious to almost all people with even a small understanding of LLMs.
Now what if we ask the LLM to write about social media? Do you think the output would be similar to what you'd get if we had a time machine to bring the actual man back and have him form his own thoughts firsthand?
It may be stylistically similar, but it's impossible to predict what the content would be.
"Explain how to solve" and "write like X" are crucially different tasks. One of them is about going through the steps of a process, and the other is about mimicking the result of a process.
Neural networks most certainly go through a process to transform input into output (even to mimic the results of another process) but it's a very different one from human neutral networks. But I think this is the crucial point of the debate, essentially unchanged from Searle's "Chinese Room" argument from decades ago.
The person in that room, looking up a dictionary with Chinese phrases and patterns, certainly follows a process, but it's easy to dismiss the notion that the person understands Chinese. But the question is if you zoom out, is the room itself intelligent because it is following a process, even if it's just a bunch of pattern recognition?
but llm can do both. so what's the point?
can you give a specific example of what an llm can't do? be specific so we can test it.
like OP originally said, the LLM doesn't have access to the actual process of the author, only the completed/refined output.
Not sure why you need a concrete example to "test", but just think about the fact that the LLM has no idea how a writer brainstorms, re-iterates on their work, or even comes up with the ideas in the first place.
why not? datasets are not only finished works, there's datasets that go into the process they're just available in smaller quantities
Let's take the work of Raymond Carver as just one example. He would type drafts which would go through repeated iteration with a massive amount of hand-written markup, revision and excision by his editor.
To really recreate his writing style, you would need the notes he started with for himself, the drafts that never even made it to his editor, the drafts that did make to the editor, all the edits made, and the final product, all properly sequenced and encoded as data.
In theory, one could munge this data and train an LLM and it would probably get significantly better at writing terse prose where there are actually coherent, deep things going on in the underlying story (more generally, this is complicated by the fact that many authors intentionally destroy notes so their work can stand on its own--and this gives them another reason to do so). But until that's done, you're going to get LLMs replicating style without the deep cohesion that makes such writing rewarding to read.
A good point. "Famous author" is a marketing term for Grammarly here; it's easy to conceive of an "author" as being an individual that we associate with a finite set of published works, all of which contain data.
But authors have not done this work alone. Grammarly is not going to sell "get advice from the editorial team at Vintage" or "Grammarly requires your wife to type the thing out first, though"
I'll also note that no human would probably want advice from the living versions of the author themselves.
Can a human replicate style without understanding process? Yes we can. We do it all the time with Shakespeare. Why not LLMs?
I can do it at the moment with Shakespeare an LLMs.
> has no idea how a writer brainstorms
This isn't true in general, and not even true in many specific cases, because a great deal of writers have described the process of writing in detail and all of that is in their training data. Claude and chatgpt very much know how novels are written, and you can go into claude code and tell it you want to write a novel and it'll walk you through quite a lot of it -- worldbuilding, characters, plotting, timelines, etc.
It's very true that LLMs are not good at "ideas" to begin with, though.
Professional writer here. On our longer work, we go through multiple iterations, with lots of teardowns and recalibrations based on feedback from early, private readers, professional editors, pop culture -- and who knows. You won't find very clear explanations of how this happens, even in writers' attempts to explain their craft. We don't systematize it, and unless we keep detailed in-process logs (doubtful), we can't even reconstruct it.
It's certainly possible to mimic many aspects of a notable writer's published style. ("Bad Hemingway" contests have been a jokey delight for decades.) But on the sliding scale of ingenious-to-obnoxious uses for AI, this Grammarly/Superhuman idea feels uniquely misguided.
The distinction being made is the difference between intellectual knowledge and experience, not originality.
Imagine a interviewing a particularly diligent new grad. They've memorized every textbook and best practices book they can find. Will that alone make them a senior+ developer, or do they need a few years learning all the ways reality is more complicated than the curriculum?
LLMs aren't even to that level yet.
> because a great deal of writers have described the process of writing in detail
And that's often inaccurate - just as much as asking startup founders how they came to be.
Part of it is forgot, part of it is don't know how to describe it and part of it is don't want to tell you so.
i don't buy this logic. if i have studied an author greatly i will be able to recognise patterns and be able to write like them.
ex: i read a lot of shakespeare, understand patterns, understand where he came from, his biography and i will be able to write like him. why is it different for an LLM?
i again don't get what the point is?
You will produce output that emulates the patters of Shakespeare's works, but you won't arrive at them by the same process Shakespeare did. You are subject to similar limitations as the llm in this case, just to a lesser degree (you share some 'human experience' with the author, and might be able to reason about their though process from biographies and such)
As another example, I can write a story about hobbits and elves in a LotR world with a style that approximates Tolkien. But it won't be colored by my first-hand WW1 experiences, and won't be written with the intention of creating a world that gives my conlangs cultural context, or the intention of making a bedtime story for my kids. I will never be able to write what Tolkien would have written because I'm not Tolkien, and do not see the world as Tolkien saw it. I don't even like designing languages
that's fair and you have highlighted a good limitation. but we do this all the time - we try to understand the author, learn from them and mimic them and we succeed to good extent.
that's why we have really good fake van gogh's for which a person can't tell the difference.
of course you can't do the same as the original person but you get close enough many times and as humans we do this frequently.
in the context of this post i think it is for sure possible to mimic a dead author and give steps to achieve writing that would sound like them using an LLM - just like a human.
You're still confusing "has a result that looks the same" and "uses the same process"; these are different things.
Why do you say it has a different process? When I ask it to do integrals it uses the same process as me
Not everything works like integrals. Some things don't have a standard process that everyone follows the same way.
Editing is one of these things. There can be lots of different processes, informed by lots of different things, and getting similar output is no guarantee of a similar process.
I don’t see why editing is any different. If a human can learn it why not an llm
The process is irrelevant if the output is the same, because we never observe the process. I assume you are arguing that the outputs are not guaranteed to be the same unless you reproduce the process.
If we are talking about human artifacts, you never have reproducibility. The same person will behave differently from one moment to the next, one environment to another. But I assume you will call that natural variation. Can you say that models can't approximate the artifacts within that natural variation?
It's relevant for data it hasn't been trained on. LLMs are trained to be all-knowing which is great as a utility but that does not come close to capturing an individual.
If I trained (or, more likely, fine-tuned) an LLM to generate code like what's found in an individual's GitHub repositories, could you comfortably say it writes code the same way as that individual? Sure, it will capture style and conventions, but what about our limitations? What do you think happens if you fine-tune a model to write code like a frontend developer and ask it to write a simple operating system kernel? It's realistically not in their (individual) data but the response still depends on the individual's thought process.
>If I trained (or, more likely, fine-tuned) an LLM to generate code like what's found in an individual's GitHub repositories, could you comfortably say it writes code the same way as that individual? Sure, it will capture style and conventions, but what about our limitations? What do you think happens if you fine-tune a model to write code like a frontend developer and ask it to write a simple operating system kernel? It's realistically not in their (individual) data but the response still depends on the individual's thought process.
Look, I don't think you understand how LLM's work. Its not about fine tuning. Its about generalised reasoning. The key word is "generalised" which can only happen if it has been trained on literally everything.
> It's relevant for data it hasn't been trained on
LLM's absolutely can reason on and conceptualise on things it has not been trained on, because of the generalised reasoning ability.
I don't know if LLMs are trained to imitate sources like that. I also don't know what would happen if you asked it to do something like someone who does not know how to do it. Would they refuse, make mistakes, or assume the person can learn? Humans can do all three, so barring more specific instructions any such response is reasonable.
i think there's a lot to be said about the process as well, the motivations, the intuitions, life experiences, and seeing the world through a certain lens. this creates for more interesting writing even when you are inspired by a certain past author. if you simply want to be a stochastic parrot that replicates the style of hemingway, it's not that difficult, but you'll also _likely_ have an empty story and you can extend the same concept to music
Even if the visualization of the integration process via steps typed out in the chat interface is the same as what you would have done on paper, the way the steps were obtained is likely very different for you and LLM. You recognized the integral's type and applied corresponding technique to solve it. LLM found the most likely continuation of tokens after your input among all the data it has been fed, and those tokens happen to be the typography for the integral steps. It is very unlikely are you doing the same, i.e. calculating probabilities of all the words you know and then choosing the one with the highest probability of being correct.
> the way the steps were obtained is likely very different for you and LLM
this is not true, any examples?
>> i again don't get what the point is?
The point is that you dont become Jimi Hendrix or Eric Clapton even if you spend 20 years playing on a cover band. You can play the style, sound like but you wont create their next album.
Not being Jimi Hendrix or Eric Clapton is the context you are missing. LLMs are Cover Bands...
You can understand his biography and analyses about how shakespeare might have written. You can apply this knowledge to modify your writing process.
The LLM does not model text at this meta-level. It can only use those texts as examples, it cannot apply what is written there to it's generation process.
no it does and what you said is easily falsifiable.
can you provide a _single_ example where LLM might fail? lets test this now.
Yes, what I said should be falsifiable. The burden is on you to give me an example, but I can give you an idea.
You need to show me an LLM applying writing techniques do not have examples in its corpus.
You would have to use some relatively unknown author, I can suggest Iida Turpeinen. There will be interviews of her describing her writing technique, but no examples that aren't from Elolliset (Beasts of the sea).
Find an interview where Turpeinen describes her method for writing Beasts of the Sea, e.g.: https://suffolkcommunitylibraries.co.uk/meet-the-author-iida...
Now ask it to produce a short story about a topic unrelated to Beasts of the Sea, let's say a book about the moonlanding.
A human doing this exercise will produce a text with the same feel as Beasts of the Sea, but an LLM-produced text will have nothing in common with it.
>You need to show me an LLM applying writing techniques do not have examples in its corpus.
why are you bringing this constraint?
This is the plot of a short story of Borges’ called “Pierre Menard, the Author of Don Quixote.”
Is the reason it can show steps for solving an integral because the training set contained webpages or books showing how to do it?
if we have steps for understanding any author's english and creative process (generally not specific to an author) would you agree then it is possible for an llm to do it?
The real sticking point for me is I don't even believe that authors themselves FULLY understand their process. The idea that anybody could achieve such full introspection as to understand and articulate every little thing that influences their output seems astoundingly improbable.
Repeating a process, yes for sure, even (pseudorandom?) variations on a process. Understanding a process is a different question, and I’m not sure how you would measure that.
In school we would have a test with various questions to show you understand the concept of addition, for example. But while my calculator can perfectly add any numbers up to its memory limit, it has no understanding of addition.
> while my calculator can perfectly add any numbers up to its memory limit, it has no understanding of addition.
"my calculator can perfectly add any numbers up to its memory limit" This kind of anthropomorphic language is misleading in these conversations. Your calculator isn't an agent so it should not be expected to be capable of any cognition.
It’s the degree of generalisability. And LLMs do have understanding. You can ask it how it came up with the process in natural language and it can help - something a calculator can’t do.
Is it not possible for the process of input to output be inferred by the llm and therefore applied to new inputs to create appropriate outputs.
Only if the LLM knows the inputs connected to particular outputs, pre-digital era or classified material might not be available, neither informal discussions with other experts.
Most importantly, negative but unused signals might not be available if the text does not mention it.
challenge: provide a single example where the LLM can only provide the output and not the steps? (in text scenario)
An LLM can always output steps, but it doesn’t mean they are true, they are great at making up bullshit.
When the “how many ‘r’ in ‘strawberry’” question was all the rage, you could definitely get LLMs to explain the steps of counting, too. It was still wrong.
can you provide a single example now with gpt 5.4 thinking that makes up things in steps? lets try to reproduce it.
I’m pretty sure you can think of one yourself, I’m not going to play this game. Now it’s 5.4 Thinking, before that it was 5.3, before that 5.2, 5.1, 5, before that it was 4… At every stage there’s someone saying “oh, the previous model doesn’t matter, the current one is where it’s at”. And when it’s shown the current model can’t do something, there’s always some other excuse. It’s a profoundly bad faith argument, the very definition of moving the goal posts.
I do have a number of examples to give you, but I no longer share those online so they aren’t caught and gamed. Now I share them strictly in person.
Ok so no example.
You've pinpointed the connection that people fail to make when they seek legal advice (or even information) from LLMs.
what prevents the input from being keystrokes and screen recordings of thousands of lawyers solving cases?
This makes the same error, or a related one. That input is not the lawyer's internal expert process, only the intermediate or (near-) final outcome of it.
Replace "LLM" with "student" and read that again. You don't just blindly give students output, you teach them, like what you are supposed to do with an LLM.
If you change the words in a sentence then it changes its meaning.
Yeah but obviously my point in this context is that it doesn't. Its not like I said to replace the word with "potato". Thanks for your genius comment.
It changes the meaning significantly. An LLM has very little in common with a human student.
Enough with this analgoy. It's flawed on so many levels. First and foremost, stop devaluing humanitiy and hyping up AI companies by parroting their party line. Second, LLMs don't learn. They can hold a very limited amount of context, as you know. And every time you need to start over. So fuck no, "teaching" and LLM is nothing like teaching an actual human.
It all went south when we started to call it "learning" instead of "fitting parameters".
„Fitting“ is still too nice of a word choice, because it implies that it’s easy to identify the best solution.
I suggest „randomly adjusting parameters while trying to make things better“ as that accurately reflects the „precision“ that goes into stuffing LLMs with more data.
It was called learning already back when the field was called cybernetics and foundational figures like Shannon worked on this kind of stuff. People tried to decipher learning in the nervous system and implement the extracted principles in machines. Such as Hebbian learning, the Perception algorithm etc. This stuff goes back to the 40s/50s/60s, so things must have gone south pretty early then.
I agree with ya so much. I have seen so many people even in hackernews somehow give human qualities to LLM's.
This Grammarly thing seems to be a bastardized form of that not even sparing the dead.
I'd say that there was some incentive by the AI companies to muddle up the water here.
> very limited amount of context
This isn't 2023 anymore
absolutely they can learn. you are being emotional and the original point is correct.
i give the LLM my codebase and it indeed learns about it and can answer questions.
That isn't learning, it can read things in its context, and generate materials to assist answering further prompts but that doesn't change the model weights. It is just updating the context.
Unless you are actually fine tuning models, in which case sure, learning is taking place.
i don't know why you think it matters how it works internally. whether it changes its weights or not is not important. does it behave like a person who learns a thing? yes.
if i showed a human a codebase and asked them questions with good answers - yes i would say the human learned it. the analogy breaks at a point because of limited context but learning is a good enough word.
Maybe because I work on a legacy programming language with far less material in the training? For me it makes a difference because it partly needs to "learn" the language itself and have that in the context, along with codebase specific stuff. For something with the model already knowing the language and only needing codebase specific stuff it might feel different.
But my codebase isn’t there in training set yet it learns and I can ask questions
You can't "teach" an LLM. It can't think. It's a simple pattern-matching algorithm, basically just an Eliza bot with a huge table of phrases.
You're not thinking, just regurgitating catch phrases that are factually incorrect hallucinations. So how are you any better than an LLM?
Which part is "factually incorrect"?
I can learn new catchphrases without boiling the ocean
Humanity as a whole is far from being consumption neutral with regards to non-renewable resources, so in a way, you really can't.
Speak for yourself...
I know all press is good press... but there are limits.
If it feels like Grammarly does not respect your right to digital sovereignty, it is because it does not.
It almost seems like this whole feature is designed to invite law suits.
Seems pretty likely usage of Grammarly's core product has cratered in the past few years. Not totally hard to imagine one of the big AI labs paying their legal fees in exchange for putting this out there and kick starting the legal process on some of these issues.
LLMs basically made Grammarly irrelevant as a product. Why have a tool to correct your grammar when you can just have it write the whole piece for you. And one things LLMs do well is construct grammatically correct text.
So IMO they are just flinging things at the wall trying to find a way back.
As Annie Duke said in her book Quit, "quitting on time usually feels like quitting too early." Grammarly was a great in the 2010s, but now it's too easily replaced.
It reminds me of winzip.
Depressingly enough, if Grammarly does throw in the hat, we'll lose an application of clear utility that could be run entirely locally.
It seems like there are many apps that can be run locally that use LLMs. Although I haven't used this, I found it on reddit and it's made by a student. https://github.com/theJayTea/WritingTools
Seems like there could be others that are better.
The real issue seems more about transparency and consent around how the models are trained and how author personas are being used
This is great!
One lesson they might draw from the negative press is to offer a more open-ended interface, like ChatGPT, where for years people have already been asking "Pretend you are X and review my writing". This interface design pattern gives the press nowhere to point their angry fingers
Grammarly seemed pretty dead on arrival the moment they added AI features. They would have said a lot more relevant and kept the costs down if they were strictly no-ai imo.
The funny thing is, their core "grammar" engine has to work on a language model + some hard heuristics anyway. So they were on a path to utilize this thing for real good, with concrete benefits.
Generative AI is a plague at this point. Everybody is adding to their wares to see what happens. It's almost like ricing a car. All noise, no go.
The weird part about tools like this isn't just the copyright question, it's the simulation of authority
Yeah, this is like applying a stained-glass image filter to your portrait in order to achieve sainthood.
Why Grammarly has to come to this point even knowing that this is not only "controversial" somehow, but rare?
Does it add any value for writers?
For the main link to the wired article as well: https://archive.is/2Qbdu
I offer my expertise in tech writing to review your AI articles and docs.
I spent a great deal of time trying to do this at allofus.ai with a team of ex-googlers with our goal being to help creators eventually 'own' their personas and drive and compete to use them to help end users.
We believed this was coming and that the best way to handle it was give the real person control over their persona to grow/edit/change and train it as they see fit.
I actually own the patent on building an expert persona based on the context of the prompt plus the real persons learned information manifold...
So what you're saying is that you can put a stop to crap like this by slinging a C&D their way?
Is anyone still using Grammarly? Their app has been replaced with a single AI query.
A few things worth flagging: On GDPR: Using a named individual's identity to generate commercial AI output isn't obviously covered by "legitimate interest." Affected EU-based individuals likely have real grounds to object or request erasure. On IP/publicity rights: You can't copyright an editing style — but you absolutely can have a right of publicity claim when a company profits from your name and simulated judgment without consent. The Lanham Act's false endorsement provisions could also be in play here. The kicker: The "sources" cited by the feature were broken, spammy, or pointed to completely unrelated content. So the defense that suggestions are inspired by someone's actual work may not even hold up technically.
This feels like a desperate attempt to stay relevant in a post-LLM world. They’re basically wrapping an LLM in a "professional" skin and calling it an expert review. The problem is that once you start letting an AI "expert" dictate tone and logic, you effectively lobotomize the writer’s original intent. We’re reaching a point where AI is just reviewing other AI-generated text, creating a feedback loop of pure mediocrity. Copium for middle management, if you ask me.
Grammarly even from the start was very distracting to me even as a someone using english as a second language to communicate. I have developed my own taste and way of articulating thoughts, but grammarly (and LLMs today) forced me to remove that layer of personality from my texts which I didn't wanted to let go. Sure I sounded less professional, but that was the image I wanted to project anyways.
Unrelated but surprising to me that I've found built-in grammar checking within JetBrains IDEs far more useful at catching grammar mistakes while not forcing me to rewrite entire sentences.
JetBrains’s default grammar checking plugin[1] is actually built on languagetool[2], a pretty decent grammar checker that also happens to be partly open source and self-hostable[3]. Sadly, they have lately shoved in a few (thankfully optional) crappy LLM-based features (that don’t even work well in the first place) and coated their landing page in endless AI keywords, but their core engine is still more traditional and open-source, and hasn’t really seemed to change in years. You can just run it on your own device and point their browser and editor extensions to it.
[1] https://plugins.jetbrains.com/plugin/12175-natural-languages... [2] https://languagetool.org -- warning: is coated in somewhat-misleading AI keywords [3] https://github.com/languagetool-org/languagetool
> The problem is that once you start letting an AI "expert" dictate tone and logic, you effectively lobotomize the writer’s original intent
Isn't that what grammarly has always been, since long before the invention of the transformer? They give you a long list of suggestions, and unless you write a corporate press release half of them are best ignored. The skill is in choosing which half to ignore
It's great. Now that fancy writing is cheap and infinite, fields whose entire scholarship value was in obscurantist jargon bending have to actually start to turn on their brains and care about making more sense than an LLM can.
What fields rely only on jargon manipulation to produce papers?
> … have to actually start to …
Or do they?
Maybe not. But academia is going to change. Status will still have to be allocated by some mechanism but the classic journals and reviews based system will crumble under the weight of LLMs. Of course this will upset a great many of people who enjoy the current state of things.
I disagree. You write when you have something to say. A service like Grammarly tries to help you convey what you want to say, but better. What you want to say is still up to you.
Words paint the picture, but the meaning of the picture is what matters.
That's a tiny fraction. Most people write because they're told to write.
Are you talking about children or students? I think most people write because they want to communicate.
Children and young students, certainly. Adult students: almost 100%. If writing is your job, then by definition, and your problem is more often finding something to say, not writing it.
You’re not counting all the office workers who have to write reports or emails, or all the scammers who write those websites to manipulate SEO or show you ads.
Frankly, I am surprised this was not shut down by their legal counsel (assuming they have one and they actually asked). The legal exposure here is significant. This could be defamation, there are publicity rights issues, copyright, and maybe even criminal liability.
This feels illegal. Even if it's not, it further drives the perception that AI is only good for crime, like crypto.
I would be surprised if the living writers can't sue over this.
"We can do it because no one can stop us."
Man I really don't like this at all.
It really feels so wrong to spare nobody, not even dead writer/people.
All it's gonna do is something similar to em-dashes where people who use it are now getting called LLM when it was their writing which would've trained LLM (the irony)
If this takes off, hypothetically, we will associate slop with the writing qualities similar to how Ghibli art is so good but it felt so sloppy afterwards and made us less appreciate the Ghibli artstyle seeing just about anyone make it.
The sad part is that most/some of these dead writers/artists were never appreciated by the people of their time and they struggled with so many feelings and writing/art was their way of expressing that. Van Gogh is an example which comes to my mind.[0] Many struggled from depression and other feelings too. To take that and expression of it and turn it into yet another product feels quite depressing for a company to do
[0]: https://en.wikipedia.org/wiki/Health_of_Vincent_van_Gogh
> It really feels so wrong to spare nobody, not even dead writer/people.
That train left at full steam when companies scraped the whole internet and claimed it was fair use. Now it's a slippery slope covered with slime.
I believe there'll be no slowing down from now on.
They are doing something amazing, will they ask for permission? /s.
that's so scummy. why they even needs "names"? it's a rhetorical question...
Moreover, they don't even apologize:
"The work is public, hence the name. It's well known, it's in the data. Who cares".
What will they do next? Create similar publications with domainsquatting and write all-AI articles with the "public" names?
Is it still fair use, then?
yes i hate that. they still have the chutzpah of keeping doing it. and i am sure it's illegal in multiple legislation. because they are not writing articles where you can cite people, they are selling a product.
I think we can thank the current times and developments as a whole for unearthing the greediest of the greedy among us.
It's very enlightening, if you ask me.
Authority washing.