> On the one hand, this output is not very surprising. AI researchers who create AI language models like the kind that power ChatGPT have known for years that these models can synthesize realistic permutations of information learned from those texts. It's how every AI assistant today works.
> But what makes this episode especially interesting is that a small hobbyist model trained by one man appears to have surprised him by reconstructing a coherent historical moment from scattered references across thousands of documents, connecting a specific year to actual events and figures without being explicitly taught these relationships. Grigorian hadn't intentionally trained the model on 1834 protest documentation; the AI assembled these connections from the ambient patterns in 6.25GB of Victorian-era writing.
This still doesn't seem wildly surprising. The event happened in 1834, he asked it a question about 1834 - why are we surprised, exactly, that the text-completion engine completed text in the expected manner?
- it's not a fine tune or LoRA; he trained the model from scratch
- it's a small model, but he got it to output text that wasn't just plausible, but reflected some understanding of the zeitgeist, based on writings from the period (not descriptions of the zeitgeist)
Man overtunes AI on texts from 1800–1875 London. Gets historical trivia from 1800-1875 London. Incredible result.
Article addresses that somewhat
> On the one hand, this output is not very surprising. AI researchers who create AI language models like the kind that power ChatGPT have known for years that these models can synthesize realistic permutations of information learned from those texts. It's how every AI assistant today works.
> But what makes this episode especially interesting is that a small hobbyist model trained by one man appears to have surprised him by reconstructing a coherent historical moment from scattered references across thousands of documents, connecting a specific year to actual events and figures without being explicitly taught these relationships. Grigorian hadn't intentionally trained the model on 1834 protest documentation; the AI assembled these connections from the ambient patterns in 6.25GB of Victorian-era writing.
This still doesn't seem wildly surprising. The event happened in 1834, he asked it a question about 1834 - why are we surprised, exactly, that the text-completion engine completed text in the expected manner?
Yeah it does not seem “especially interesting” to me. It was in the training data
This is interesting because:
- it's not a fine tune or LoRA; he trained the model from scratch
- it's a small model, but he got it to output text that wasn't just plausible, but reflected some understanding of the zeitgeist, based on writings from the period (not descriptions of the zeitgeist)
The point of the article is boring but training LLMs on documents from a particular time period is actually pretty interesting.
Assembling 6GB of training data is actually rather impressive, given the constraints.