stripping to markdown with Jina Reader or Trafilatura before passing to the agent cuts that 68k down to ~3-5k for most Wikipedia pages, and handles the JS-rendered case too.
one of the early jina.ai products was/is reader api, and they trained ReaderLM for this purpose. definitely a good idea to check out existing implementations
stripping to markdown with Jina Reader or Trafilatura before passing to the agent cuts that 68k down to ~3-5k for most Wikipedia pages, and handles the JS-rendered case too.
+1. i think the the additional value prop is the bypassing blockers. ideally op should just use jina on top of whatever they are building.
one of the early jina.ai products was/is reader api, and they trained ReaderLM for this purpose. definitely a good idea to check out existing implementations
> on js-rendered and some anti-bot pages it returns nothing
yeah, and my (human piloted) browser gets blocked by so many websites now. I routinely get "Sorry." when trying to log into hn.
sigh.
Hacker News is the one page that hasn't assumed I'm a bot, yet.
At this rate, we'll need to use bots to browse the web for us, because they're the only way to get through the anti-bot filters.
And how much water?