Token Embeddings Violate the Manifold Hypothesis

1 comments

• verdverm 8 hours ago

  If this interests you, https://bsky.app/profile/jefferyharrell.bsky.social has been making some super cool visualizations of Qwen3's internal representations

  One thing discovered was that 800+ tokens converged to a single point in the 2500+ dimensional unembedding space

  Some examples:

  https://bsky.app/profile/jefferyharrell.bsky.social/post/3m4...

  https://bsky.app/profile/jefferyharrell.bsky.social/post/3m5...