While we're dealing with the same issue at work, I sometimes still wonder exactly who these scrapers are. OpenAI, Google, Anthropic and others are normally fairly well behaved (Minus Anthropic attempting to hide behind a browser-for-hire company). Mostly you can get IP range and user-agents for the large players, while it problems mostly stem from bots pretending to be Chrome.
Our largest offenders seems to be mostly limited to South-East Asia, so probably mostly Chinese AI projects, but that's speculation. I also don't recall ever seeing Grok IP ranges or a specific Grok UA, but that doesn't mean that they're hiding, perhaps they're just not interested.
You need to account for the fact that a lot of scraping is delegated by the big players to smaller players who can take loss of reputation and can use questionable methods (residential IPs). Some of the scrapers from these AI companies have been written very poorly from performance perspective.
From what I understand the pressure has created even Google to be a lot more aggressive than what it was before. Not entirely sure but I believe google has two categories of scrapers, the regular one and a new one for AI
Still not sure about the "who" question, but here's the scale of money changing hands:
> Residential proxies are everywhere, so why did proxy DDoS attacks mostly come from the U.S.? The answer is money. If you are committing fraud or circumventing content restrictions, a Russian IP address gets geo-blocked instantly. A fresh U.S. residential IP address (especially one behind carrier-grade NAT and harder to block individually) is “gold.” Customers pay up to $95 to lease a single U.S. residential IP address for 2 weeks (versus $0.30 for an Eastern European IP). Compare that to your own ARPU per subscriber and sit with it for a second. When an individual IP is worth more than the customer relationship behind it, you don’t have a technical problem. You have a market problem.[1]
Here is the fun part, the same companies invests millions of dollars to protect its infrastructure and services from scrapers/bots AND they also invest millions of dollars to circumvent the bots, captchas and rate limits.
Unfortunately you cant have it both ways which makes it hard problem to solve for everyone
I sometimes still wonder exactly who these scrapers are
This has zero to do with AI (it's just a convenient scapegoat that fits the narrative). Absolutely everything to do with those pushing for centralised control of the Internet.
One of the defining characteristics of AI turned out to be utter and profound facelessness. The scraping, the data centers, the slop content, it's like it all comes from thin air. Nobody is talking about who is really doing all this or why, because it's always just coming from... somewhere else outside of our place.
Well, some websites claimed China is behind it, which could make sense (I would not know either way). At the same time, though, I kind of doubt your carte blanche here for all those companies. Why would you think none of them are responsible for the AI slop spam?
> Our largest offenders seems to be mostly limited to South-East Asia, so probably mostly Chinese AI projects, but that's speculation
Ok. So you also don't know. Well, I don't know either, but I don't make a speculation by claiming x, y, and z companies to be exempt. In my book they are all responsible.
We see constant abuse from the Tencent ASN/associated ACE ASN, and I've memorized the china169 backbone asn as AS4837 because of thr nonstop crawlers splattered across their network ranges. It's not possible to ID the operator running the crawlers running from these networks but there's a clear signal of the origin of some of these entities.
Well, those companies have bots that identify themselves, and you can see what they're doing. Google especially have decades of experience of designing scrapers and seem to be able to the job of scraping the whole internet without causing problems in that time.
So, it's still possible they are also running a bunch of poorly-coded scraper bots through residential proxies, but it seems a little non-obvious why they would be doing this.
(It is worth pointing out that most of this traffic seems to be dumb: it's stuff like getting lost in generated link forests of some web apps or repeatly re-querying the same endpoint on a super-high frequency. This isn't exactly going to give a good return on investment for AI training data, especially since AIUI the main race for LLM performance now is in good quality training data)
>So, it's still possible they are also running a bunch of poorly-coded scraper bots through residential proxies, but it seems a little non-obvious why they would be doing this.
why "non obvious"? The easiest explanation is gathering training data sets, and is mostly caused by AI companies guarding their pile of essentially stolen IP (given how little they care about copyright) from eachother, without sharing any competition need to get their own and re-crawl to update it too
Well, why run both a well-behaved, easily identifiable bot that probably already gets them all the data they need (they seem to be spending most of their time and money on getting higher quality data than your average internet scrape), and this crap? Like, it's possible the obvious bots are a smokescreen, don't actually work well enough, and they are actually also reliant on data sources that contain 2 million copies of gentoo's bugs database. It's even possible that they are indirectly responsible for it, by buying datasets from shady sources, but again this requires a few jumps that I would like to see justified by evidence.
Manufactured consent. This is not AI, this is just DDoS'ing for extortion and driving towards the closed proprietary "can only be accessed by a sanctioned UA" Internet that they're after. Why else do you think identity verification and attempting to turn the Internet into a walled garden became such a big thing in the recent years?
After a decade and half we had to restrict our public side and reorganize our old TED content because scrappers were really hungry for assorted captioned videos. If you have anything of value for training you get eaten alive if you stick out, it's like wearing short pants in the summer tundra.
It's going to change a lot the internet we knew, unless somehow we managed to agree on a common high quality dataset repository.
It is time for micropayments integrated in the browser. Pay 5 cents to access each bug report. Not fun, it shouldn't be like this, but better than not having a bugtracker at all.
Not sure it works, isn't it already like that if you have the cursed javascript-only pages? It's basically proof of work to access their content (and if you are scrapping en masse you likely also need some LLM getting involved)
I suppose the cost of hardware, network usage, and electricity is still under 0.05? I mean, we know the unit economics for LLMs don't make a ton of sense, except if enterprise customers really are what keeps the lights on.
With micropayments, the server owner makes 1 million requests times 0.05 cents = ~$500
The status quo with those js PoW pages doesn't really benefit the server owner at all, it's wasted energy.
I mean proof of work is always wasted energy, but I figure it's better to kill two birds with one stone.
CoinHive was one example of this. (I think this is a correct link? https://github.com/cazala/coin-hive). Although I think ideally you would want to have some sort of browser plugin or app that runs on bare metal instead of a proof-of-work in the browser, because RandomX is designed such that it's slow when implemented in JS (https://github.com/tevador/RandomX/blob/master/doc/design.md)
and do you happen to know a zero friction payment system that works internationally like the Internet itself does? because I promise you that unless you already have a captive audience, even a minuscule amount of friction to access your service will cost you an overwhelming percentage of organic human visitors.
Something like Chaum's blind-signature based ecash, or GNU Taler would be ideal in terms of efficiency, but it's still centralized in distribution. Freenet currently uses something like this.
Another option I was thinking of would be a pretty inflationary (or demurrage) cryptocurrency in which you have some sort of RandomX or other CPU-bound PoW. A web server could act as a mining pool and use mining shares interchangibly with micropayments.
You could do this mining-share method with Monero right now, it's just that you have higher transaction size in Monero and no real analogue to Bitcoin's LN-based microtransactions. Also you would want the cryptocurrency to be more inflationary (or demurrage-based) to promote usage.
Monero's FCMP++ lays some groundwork for payment channels, but it still lacks the nessisary timelocks. Also there was DLSAG which could have enabled payment channels I think, but it's no longer relevant. I also insist that you would need to change the tokenomics to favor greater inflation (maybe you could make coinbase scale linearly with hashrate?), otherwise the miner reward would be economicially insufficient.
Lightning - a fast, instant, bitcoin layer 2 network - is perfectly sufficient for micropayments. Volatility is a no-issue in this case, as you can freely trade the 5 cents in realtime into other assets and minimize holding time of BTC. You will loose the spread, ofc.
1) it requires you to open a lightning channel. So you need some amount of initial investment (which usually means you need a credit card and an account on a bitcoin exchange), not just any computer which is the issue that using mining shares solves. The initial investment is my main grudge, as it is way too much friction for 402 Payment Required applications. Also as bitcoiners like sztorc note, this isn't feasible for most of the world's population due to bitcoin's small block size.
2) It's not private. People are going to be linking these things to their identities on crypto exchanges. So now feds can basicially track everything you do on the internet that way. LN is only private in the sense that not every transaction is broadcast to everyone else on the network, which is a very low bar. Chainalysis is possible with the right connections.
To a lesser extent, 3) Centralization in the lightning routing protocol which contributes to the effect of #2.
Friction-free is a big ask. Brave tried something a while back with their Brave Payments, but of course no one trusts them and it’s not going anywhere.
Frictionless micropayments is THE business opportunity dangling here. I'm so disappointed that the Bitcoin bros never made it easy and fast to use Bitcoin for peer to peer micropayments.
Re audience, the audience for Gentoo bugzilla went just to zero. If even 1% of the current bugzilla users would use the micropayments, that's literally infinitely more organic human visitors than today.
How many pages does the average software developer visit everyday? 1.000? Price at 0.001 per load and it’ll be completely impractical for crawlers but super cheap for the average connected human on earth.
I don't really understand why there is resistance to in-browser Anubis-style gating using crypto mining for public sites like this.
You mine crypto and my free service remains cost-neutral. Yes, that transfers some of the cost to you, but it is marginal.
You want to bang on my servers with the fury of a thousand madmen, then I'll scale up the servers and you can pay the marginal cost of your access.
Or micropayments, of course, but hard to get normal users to sign up for micropayments. Micropayments could of course be the way to bypass the crypto mining gate.
>I don't really understand why there is resistance to in-browser Anubis-style gating using crypto mining for public sites like this.
But in the case of anubis it's not even used for crypto. It's just wasted.
>You mine crypto and my free service remains cost-neutral. Yes, that transfers some of the cost to you, but it is marginal.
No, the problem is time wasted. I don't care about the electricity cost. Spending 10s to solve a challenge on a 5W SoC translates to 0.0005 cents (yes, cents, not dollars). Meanwhile if I click on a link and it doesn't load in 5s, I'm seriously questioning the value of your blog or whatever, and will probably just close the tab.
Instead of punishing a criminal, you make everybody pay.
This is not only unfair to legitimate users having to pay.
It also needs to explain what actually happens when the Ddos succeeds (I know, you are talking about scrapers, but what’s the difference really?). Does that mean that the attacker just gets to shrug it off.
Don’t take my statements as facts, I just want to outline a few reason I could come up with that show a purely technical solution might not be enough.
Akin to “just use Cloudflare, it’s free”.
Yes it works but why conceding defeat and say “oh from now on you depend on a business to publish a server”?
There are definitely patterns you can use against the scrapers. This maintainer just didn't have time for it, which is understandable.
We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time.
Most scrapers are relatively honest in some way shape or form.
>We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time.
What's the point of this compared to letting cloudflare handle everything automagically? Presumably whatever heuristics they come up with are going to be better than you can, given limited time and budget?
One, their functionality is bad, two, we don't want to block robot traffic. We're redicting it to a separate sever so that overly-large waves don't bring down our main servers. We definitely want to be scraped.
To your point: I think you're simply mistaken that they offer a good scraper-detection solution, but I'm all ears.
What kind of uptime/expectations/etc. are you working with?
Scraper "attacks" don't take down our robot-specific server very often; it's safe for us to take heavy-handed approaches that sometimes redirect users there. 99% (made-up high number) of the time, the misdirected users don't realize anything is amiss.
Start by analyzing your traffic, specifically user agents. Look for "robot" or even "bot" in the user agent and load balance those to a robot-specific server. This can all be done within Cloudflare. The only code is the user agent condition. Note: I'm very open to input here if anyone reading notices that we're shooting ourselves in the feet. Based on our analysis, the remaining traffic is a good picture of our human users.
We have loads of other conditions, mostly balancing specific IP ranges for entities when we know exactly who they are, but this is a good start.
Hedgewars had to lock down its bugzilla and repo too, for the exact same reason. So far the really simple method used (basic auth with the creds posted on the main site) seems to have resisted the scrapers. Presumably if they are too stupid to scrape correctly they are also too stupid to incorporate a trivial auth.
The question is, even if you identify who is the source of the traffic, are they even in a place where you can realistically sue them? The problematic traffic generally is not the bots that identify themselves, but the ones that are using residential proxies and try to be a non-fingerprintable as possible.
Then enforcement will target companies providing residential proxy services.
In whatever country they are.
I am not saying this is the magic bullet that will make it all go away. Some will, some will not.
Enforcement’s aim is not to completely eradicate crime: make it anti-economical for the majority and it’s a good deal enough.
Go to Berlin and torrent a movie: you will be fined, for a few hundred euros. Can you VPN and get away with it? I suppose so, but your average Joe is just not going to do it: why pay 10 euro per month on a VPN and risk? Just rent it.
DDOS is also, in general, a criminal charge. What would happen to prices once you get a few inducements? If it’s a crime and enforced, what are the roles of App Stores that let users download residential proxy? Are they also committing a crime? Residential proxy can be used for legitimate purposes? I don’t think that line of defence worked well for Pirate Bay, no?
And yes pirate bay is still around, some scraping will always be there.
I agree in principle. The tech community naturally has a tendency to reach for tech solutions to societal problems. But without jurisdiction, laws are meaningless.
If you create a law that says you have to honour robots.txt files, what do you do if an IP from another country fails to do so?
You check the laws of your country. Likely they say that DDOS are criminal offences, and you can get 10 or 20 years of jail time, already.
Then you report a crime and send the logs. And maybe report to some politician that is tough on crime. And you are not alone.
I am sure you follow through, Pls check also my other comments.
Laws without enforcement are meaningless, without jurisdiction they are less effective.
Maybe yes, some poor guy will find a letter that says he needs to pay 50 USD or EUR to some faraway country because the puzzle app in his tv has weird ToS. Or whatever, a cease and desist letter? A “you are part of a hacking circle please uninstall this app” letter, signed by a judge?
I don’t advocate for people getting thrown in jail, I just think that the whole “residential proxies” issue might be a charade that would not last longer if citizens and businesses would claim rights instead of play victim.
I also really dislike grasping for a tech solution because I don’t want to start to mine crypto to read an article or to depend on some “free” cloud service to host an open bug tracker.
It just feels so weird: why is it that no-one is doing it? I feel like I an one “comment away” from someone explaining to me how naive I am, and yet it’s never coming.
I like lesswrong's approach to this. You get 4xx if you scrape that site but greaterwrong.com serves some kind of static site that is scrape friendly. Seems fine to me.
Though, checking, I see that it requires a login but is still up. I misunderstood what 'closed' meant. This seems fine.
Are we assuming that this is the companies themselves scraping data from training or is this "agents" acting on behalf of users? Nowadays every major chat UI (ChatGPT, Claude etc) has a "tool" that allows LLM to load web pages, so it must generate some traffic.
No, most of the load comes from armies of residential IP addresses that look like Google Chrome on the wire. The major chat UIs properly identify themselves. These waves of attacks do not.
What are they scraping the gentoo bugzilla for? I'm confused. Unless you're actively using Gentoo why would this be a resource? Very confusing. Also you'd think we'd have LLM BitTorrent by now, where if they want to scrape something we get a DHT hash for the content and share it with one another, rather than melt servers with the millionth request of the day.
At this point they've mostly run out of material, so ANY type of content is valuable. Your small personal website, why would they scrape that? It's 10.000 additional words, wouldn't want to miss that. My Github repos.... got to get buggy code from somewhere I guess.
I get what you're asking, and I'm wondering the same. Not all sources are created equally and we see the results all the time. LLMs outputs nonsense all the time, like Flock cameras containing 5 grams of gold and ounces of copper, because they are completely on critical of their sources. Perhaps there's some weights that says: Kernel mailing list, MariaDB documentation and Microsofts Learning sites are 100% trust, Reddit 50%, 4Chan 10%, but I doubt it.
Anthropic might care a little bit, seeing as they scan books, but again, is it just all books? Because other than some flowery language I don't really see the point in scanning a 1970s paperback only spy novel.
They're mining the written output of technical talent discussing troubleshooting, bug finding, patches, inter-dependencies, etc. It's an AI trainer's wet dream.
Also: where exactly are AI companies incentivized to be anything but shitty 'neighbors'?
Spam, DDoS attacks and other network abuse used to cause your hosting company or provider to call you and tell you to knock it off or you'll get disconnected, if your provider was reputable. If your provider wasn't reputable, it was likely a matter of time before they would get a nasty call from their upstream provider.
Now it just gets you a thank-you from the sales team for all the bandwidth you bought.
Meanwhile, do any of the cloud providers have any incentive to do anything about this? Hell no. They're making money off you having to ramp up extra or bigger instances. They're making money off the bandwidth. They're making money off the people doing the crawling, too. They're incentivized to do the exact opposite of effectively help you with your AI bot problem.
There's a reddit thread from 9 years ago with people complaining the site blocking crawlers and thus not being indexed by search engines. So I'm not sure what they consider "overload." At this point they should dump the db on thepiratebay.
Dumping the database won't stop the scrappers because humans running the scrappers are not even aware Gentoo exists. They're just scrapping every website their bots find.
going on a bit of a tangent here - some discussion there seems to be imply that AI bots mostly use IPv4s, which makes sense to me given that they are probably bots hosted by some cloud. Whereas IPv6 may be more organic traffic from e.g. mobile users (not in this case probably)... which made me think if at some point IPv6 may at some point win against IPv4 just because its more organic traffic (i.e. not from big tech cloud), leading to pages blocking IPv4? Just speculation on my side
Gentoo has pretty much always had a Bugzilla, even back in the early 00s... there never was an alternative plan.
The biggest thing about what's happened today though, is how a "janitor" on the infrastructure team can shut down the entire distros ability to work on bugs. The decision to shut down should have been left to someone else like robbat.
Mgorny has consistently caused issues for Gentoo, and many people have left the community after having to deal with him, but it seems comrel/devrel can't/won't remove him for whatever reason.
Edit: it looks like bgo is up now. Hopefully it stays that way.
Just put the website behind an x402 gate [0] (yes it uses stablecoins) requiring payment if a bot wants to access the content which acts like a Captcha and detect them if they are on direct data center IP or are using residential proxies.
Stablecoins have a great use-case and you get paid for bots to access the site with the humans living in peace without the site getting botted.
Corrected title: "Gentoo bugzilla closed because the guy running it, who says it is 'unusable anyway', saw a lot of traffic from different IP addresses with no clear pattern and accuses AI"
The year is 2026, somehow peoples basic web apps are not able to keep up with scrapers. Scrapers are not new. My side load is like .01 even with 10x the traffic of last year. It's called "serving static content", "caching" and many other things that are not new concepts.
Then you've got good old cloudflare which is free to use
1. You shouldn't need to be a professional sysadmin to be able to deploy a basic semi-dynamic website
2. Considering the spike of complaints over the past two years and the fact that they range from small time site operators to larger orgs (Linux Kernel, GNOME, Duke University, etc), do you think that maybe you're missing part of the problem space?
While we're dealing with the same issue at work, I sometimes still wonder exactly who these scrapers are. OpenAI, Google, Anthropic and others are normally fairly well behaved (Minus Anthropic attempting to hide behind a browser-for-hire company). Mostly you can get IP range and user-agents for the large players, while it problems mostly stem from bots pretending to be Chrome.
Our largest offenders seems to be mostly limited to South-East Asia, so probably mostly Chinese AI projects, but that's speculation. I also don't recall ever seeing Grok IP ranges or a specific Grok UA, but that doesn't mean that they're hiding, perhaps they're just not interested.
You need to account for the fact that a lot of scraping is delegated by the big players to smaller players who can take loss of reputation and can use questionable methods (residential IPs). Some of the scrapers from these AI companies have been written very poorly from performance perspective.
From what I understand the pressure has created even Google to be a lot more aggressive than what it was before. Not entirely sure but I believe google has two categories of scrapers, the regular one and a new one for AI
if they have agents write the scrapers you better believe they're playing every evil trick in the book
So it could just be 1 guy with scrapy and infinite vc money
Still not sure about the "who" question, but here's the scale of money changing hands:
> Residential proxies are everywhere, so why did proxy DDoS attacks mostly come from the U.S.? The answer is money. If you are committing fraud or circumventing content restrictions, a Russian IP address gets geo-blocked instantly. A fresh U.S. residential IP address (especially one behind carrier-grade NAT and harder to block individually) is “gold.” Customers pay up to $95 to lease a single U.S. residential IP address for 2 weeks (versus $0.30 for an Eastern European IP). Compare that to your own ARPU per subscriber and sit with it for a second. When an individual IP is worth more than the customer relationship behind it, you don’t have a technical problem. You have a market problem.[1]
1: https://www.nokia.com/blog/one-year-later-the-residential-pr...
Here is the fun part, the same companies invests millions of dollars to protect its infrastructure and services from scrapers/bots AND they also invest millions of dollars to circumvent the bots, captchas and rate limits.
Unfortunately you cant have it both ways which makes it hard problem to solve for everyone
I sometimes still wonder exactly who these scrapers are
This has zero to do with AI (it's just a convenient scapegoat that fits the narrative). Absolutely everything to do with those pushing for centralised control of the Internet.
One of the defining characteristics of AI turned out to be utter and profound facelessness. The scraping, the data centers, the slop content, it's like it all comes from thin air. Nobody is talking about who is really doing all this or why, because it's always just coming from... somewhere else outside of our place.
And then it's pushed on the end users with the same amount of explanation and reason. You must adopt this now. The FOMO is insane.
> The FOMO is insane.
do we have the same understanding of what fomo means?
You aren't stuck in internet traffic, you are internet traffic.
Most of the nonsense comes through botnets/residential proxies, so it's very hard to say.
Well, some websites claimed China is behind it, which could make sense (I would not know either way). At the same time, though, I kind of doubt your carte blanche here for all those companies. Why would you think none of them are responsible for the AI slop spam?
> Our largest offenders seems to be mostly limited to South-East Asia, so probably mostly Chinese AI projects, but that's speculation
Ok. So you also don't know. Well, I don't know either, but I don't make a speculation by claiming x, y, and z companies to be exempt. In my book they are all responsible.
We see constant abuse from the Tencent ASN/associated ACE ASN, and I've memorized the china169 backbone asn as AS4837 because of thr nonstop crawlers splattered across their network ranges. It's not possible to ID the operator running the crawlers running from these networks but there's a clear signal of the origin of some of these entities.
Well, those companies have bots that identify themselves, and you can see what they're doing. Google especially have decades of experience of designing scrapers and seem to be able to the job of scraping the whole internet without causing problems in that time.
So, it's still possible they are also running a bunch of poorly-coded scraper bots through residential proxies, but it seems a little non-obvious why they would be doing this.
(It is worth pointing out that most of this traffic seems to be dumb: it's stuff like getting lost in generated link forests of some web apps or repeatly re-querying the same endpoint on a super-high frequency. This isn't exactly going to give a good return on investment for AI training data, especially since AIUI the main race for LLM performance now is in good quality training data)
>So, it's still possible they are also running a bunch of poorly-coded scraper bots through residential proxies, but it seems a little non-obvious why they would be doing this.
why "non obvious"? The easiest explanation is gathering training data sets, and is mostly caused by AI companies guarding their pile of essentially stolen IP (given how little they care about copyright) from eachother, without sharing any competition need to get their own and re-crawl to update it too
Well, why run both a well-behaved, easily identifiable bot that probably already gets them all the data they need (they seem to be spending most of their time and money on getting higher quality data than your average internet scrape), and this crap? Like, it's possible the obvious bots are a smokescreen, don't actually work well enough, and they are actually also reliant on data sources that contain 2 million copies of gentoo's bugs database. It's even possible that they are indirectly responsible for it, by buying datasets from shady sources, but again this requires a few jumps that I would like to see justified by evidence.
Manufactured consent. This is not AI, this is just DDoS'ing for extortion and driving towards the closed proprietary "can only be accessed by a sanctioned UA" Internet that they're after. Why else do you think identity verification and attempting to turn the Internet into a walled garden became such a big thing in the recent years?
You don’t think it has anything to do with hostile states using fake profiles on social media to drum up discontent in the democratic world?
After a decade and half we had to restrict our public side and reorganize our old TED content because scrappers were really hungry for assorted captioned videos. If you have anything of value for training you get eaten alive if you stick out, it's like wearing short pants in the summer tundra.
It's going to change a lot the internet we knew, unless somehow we managed to agree on a common high quality dataset repository.
It is time for micropayments integrated in the browser. Pay 5 cents to access each bug report. Not fun, it shouldn't be like this, but better than not having a bugtracker at all.
0.05 cents even. The beautiful thing is, minuscule amounts of micropayments are already enough to fix the incentives
Not sure it works, isn't it already like that if you have the cursed javascript-only pages? It's basically proof of work to access their content (and if you are scrapping en masse you likely also need some LLM getting involved)
Problem is I'd rather have the 0.05cents go to however runs the website (the bugtracker in this case) than go to the electricity company.
I suppose the cost of hardware, network usage, and electricity is still under 0.05? I mean, we know the unit economics for LLMs don't make a ton of sense, except if enterprise customers really are what keeps the lights on.
With micropayments, the server owner makes 1 million requests times 0.05 cents = ~$500
The status quo with those js PoW pages doesn't really benefit the server owner at all, it's wasted energy.
I mean proof of work is always wasted energy, but I figure it's better to kill two birds with one stone.
CoinHive was one example of this. (I think this is a correct link? https://github.com/cazala/coin-hive). Although I think ideally you would want to have some sort of browser plugin or app that runs on bare metal instead of a proof-of-work in the browser, because RandomX is designed such that it's slow when implemented in JS (https://github.com/tevador/RandomX/blob/master/doc/design.md)
$500 would amount to just 10k requests. One million requests is $50k
10,000 requests times 0.05 cents (not dollars) = 500 cents = 5 dollars.
and do you happen to know a zero friction payment system that works internationally like the Internet itself does? because I promise you that unless you already have a captive audience, even a minuscule amount of friction to access your service will cost you an overwhelming percentage of organic human visitors.
Something like Chaum's blind-signature based ecash, or GNU Taler would be ideal in terms of efficiency, but it's still centralized in distribution. Freenet currently uses something like this.
Another option I was thinking of would be a pretty inflationary (or demurrage) cryptocurrency in which you have some sort of RandomX or other CPU-bound PoW. A web server could act as a mining pool and use mining shares interchangibly with micropayments.
You could do this mining-share method with Monero right now, it's just that you have higher transaction size in Monero and no real analogue to Bitcoin's LN-based microtransactions. Also you would want the cryptocurrency to be more inflationary (or demurrage-based) to promote usage.
Monero's FCMP++ lays some groundwork for payment channels, but it still lacks the nessisary timelocks. Also there was DLSAG which could have enabled payment channels I think, but it's no longer relevant. I also insist that you would need to change the tokenomics to favor greater inflation (maybe you could make coinbase scale linearly with hashrate?), otherwise the miner reward would be economicially insufficient.
Lightning - a fast, instant, bitcoin layer 2 network - is perfectly sufficient for micropayments. Volatility is a no-issue in this case, as you can freely trade the 5 cents in realtime into other assets and minimize holding time of BTC. You will loose the spread, ofc.
the problem with lightning is:
1) it requires you to open a lightning channel. So you need some amount of initial investment (which usually means you need a credit card and an account on a bitcoin exchange), not just any computer which is the issue that using mining shares solves. The initial investment is my main grudge, as it is way too much friction for 402 Payment Required applications. Also as bitcoiners like sztorc note, this isn't feasible for most of the world's population due to bitcoin's small block size.
2) It's not private. People are going to be linking these things to their identities on crypto exchanges. So now feds can basicially track everything you do on the internet that way. LN is only private in the sense that not every transaction is broadcast to everyone else on the network, which is a very low bar. Chainalysis is possible with the right connections.
To a lesser extent, 3) Centralization in the lightning routing protocol which contributes to the effect of #2.
Friction-free is a big ask. Brave tried something a while back with their Brave Payments, but of course no one trusts them and it’s not going anywhere.
Frictionless micropayments is THE business opportunity dangling here. I'm so disappointed that the Bitcoin bros never made it easy and fast to use Bitcoin for peer to peer micropayments.
Re audience, the audience for Gentoo bugzilla went just to zero. If even 1% of the current bugzilla users would use the micropayments, that's literally infinitely more organic human visitors than today.
The question is what exactly are you trying to achieve. Do you want to prevent them from gathering data or do you want to save resources?
If it is the latter, then, perhaps, some p2p (torrent-like) content delivery system could help.
I think it's gonna kill off the internet not just for bots but to whole class of lower income countries, especially for younger learners
Perhaps the biggest problem is 5 cents or less isn't enough to cover minimum transaction fees.
Visa fees.
Cloudflare is working exactly on enabling that, like it or not.
https://blog.cloudflare.com/monetization-gateway/
How many pages does the average software developer visit everyday? 1.000? Price at 0.001 per load and it’ll be completely impractical for crawlers but super cheap for the average connected human on earth.
That would gate the internet to whole lower income countries
Aren't many of those countries basically being de-facto blocked by Cloudflare filtering out spam already anyways?
Genuine question, I'm not up to date on how Cloudflare operates right now
The genuine answer to your question is no, they are not de-facto blocked, unless the website operator chooses to do so.
I don't really understand why there is resistance to in-browser Anubis-style gating using crypto mining for public sites like this.
You mine crypto and my free service remains cost-neutral. Yes, that transfers some of the cost to you, but it is marginal.
You want to bang on my servers with the fury of a thousand madmen, then I'll scale up the servers and you can pay the marginal cost of your access.
Or micropayments, of course, but hard to get normal users to sign up for micropayments. Micropayments could of course be the way to bypass the crypto mining gate.
>I don't really understand why there is resistance to in-browser Anubis-style gating using crypto mining for public sites like this.
But in the case of anubis it's not even used for crypto. It's just wasted.
>You mine crypto and my free service remains cost-neutral. Yes, that transfers some of the cost to you, but it is marginal.
No, the problem is time wasted. I don't care about the electricity cost. Spending 10s to solve a challenge on a 5W SoC translates to 0.0005 cents (yes, cents, not dollars). Meanwhile if I click on a link and it doesn't load in 5s, I'm seriously questioning the value of your blog or whatever, and will probably just close the tab.
Instead of punishing a criminal, you make everybody pay.
This is not only unfair to legitimate users having to pay.
It also needs to explain what actually happens when the Ddos succeeds (I know, you are talking about scrapers, but what’s the difference really?). Does that mean that the attacker just gets to shrug it off.
Don’t take my statements as facts, I just want to outline a few reason I could come up with that show a purely technical solution might not be enough.
Akin to “just use Cloudflare, it’s free”.
Yes it works but why conceding defeat and say “oh from now on you depend on a business to publish a server”?
If we could identify who is doing this, I bet that those people would be publicly burned at the stake by now.
There are definitely patterns you can use against the scrapers. This maintainer just didn't have time for it, which is understandable.
We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time.
Most scrapers are relatively honest in some way shape or form.
One thing that surpised me about gentoo is just how low budget it is as an operation. They are doing everything with a $12k budget. [1]
[1] https://www.gentoo.org/news/2026/01/05/new-year.html
Basically volunteer-run. Which makes sense in this context; what gets worked on is mostly about what people want to work on.
>We direct scraper traffic to a bot-specific server using Cloudflare's load balancer, slowly analyzing traffic and adding conditions one at a time. No accidental scraper DDoS in a long time.
What's the point of this compared to letting cloudflare handle everything automagically? Presumably whatever heuristics they come up with are going to be better than you can, given limited time and budget?
One, their functionality is bad, two, we don't want to block robot traffic. We're redicting it to a separate sever so that overly-large waves don't bring down our main servers. We definitely want to be scraped.
To your point: I think you're simply mistaken that they offer a good scraper-detection solution, but I'm all ears.
How did you implement this? My site's getting hammered, any tips would be appreciated
What kind of uptime/expectations/etc. are you working with?
Scraper "attacks" don't take down our robot-specific server very often; it's safe for us to take heavy-handed approaches that sometimes redirect users there. 99% (made-up high number) of the time, the misdirected users don't realize anything is amiss.
Start by analyzing your traffic, specifically user agents. Look for "robot" or even "bot" in the user agent and load balance those to a robot-specific server. This can all be done within Cloudflare. The only code is the user agent condition. Note: I'm very open to input here if anyone reading notices that we're shooting ourselves in the feet. Based on our analysis, the remaining traffic is a good picture of our human users.
We have loads of other conditions, mostly balancing specific IP ranges for entities when we know exactly who they are, but this is a good start.
> Most scrapers are relatively honest in some way shape or form.
Did you miss a "dis" in there?
Hedgewars had to lock down its bugzilla and repo too, for the exact same reason. So far the really simple method used (basic auth with the creds posted on the main site) seems to have resisted the scrapers. Presumably if they are too stupid to scrape correctly they are also too stupid to incorporate a trivial auth.
Enough with playing around the issue. The way out of this mess is not to protect with tech that works but with principles and laws.
This is not a tech problem. This is about what should or should not be legal.
Nor it’s a question of having time to implement solution X or Y.
If someone attacks you yes, you should have better security but you also need to have legal recourse, or it will never stop.
Ddos is already illegal.
I am not a lawyer so don’t ask me for exact resources, which vary by country anyway, but stop treating scrapers as an inescapable force of nature.
The question is, even if you identify who is the source of the traffic, are they even in a place where you can realistically sue them? The problematic traffic generally is not the bots that identify themselves, but the ones that are using residential proxies and try to be a non-fingerprintable as possible.
Then enforcement will target companies providing residential proxy services.
In whatever country they are.
I am not saying this is the magic bullet that will make it all go away. Some will, some will not.
Enforcement’s aim is not to completely eradicate crime: make it anti-economical for the majority and it’s a good deal enough.
Go to Berlin and torrent a movie: you will be fined, for a few hundred euros. Can you VPN and get away with it? I suppose so, but your average Joe is just not going to do it: why pay 10 euro per month on a VPN and risk? Just rent it.
DDOS is also, in general, a criminal charge. What would happen to prices once you get a few inducements? If it’s a crime and enforced, what are the roles of App Stores that let users download residential proxy? Are they also committing a crime? Residential proxy can be used for legitimate purposes? I don’t think that line of defence worked well for Pirate Bay, no?
And yes pirate bay is still around, some scraping will always be there.
I agree in principle. The tech community naturally has a tendency to reach for tech solutions to societal problems. But without jurisdiction, laws are meaningless.
If you create a law that says you have to honour robots.txt files, what do you do if an IP from another country fails to do so?
You check the laws of your country. Likely they say that DDOS are criminal offences, and you can get 10 or 20 years of jail time, already.
Then you report a crime and send the logs. And maybe report to some politician that is tough on crime. And you are not alone.
I am sure you follow through, Pls check also my other comments.
Laws without enforcement are meaningless, without jurisdiction they are less effective.
Maybe yes, some poor guy will find a letter that says he needs to pay 50 USD or EUR to some faraway country because the puzzle app in his tv has weird ToS. Or whatever, a cease and desist letter? A “you are part of a hacking circle please uninstall this app” letter, signed by a judge?
I don’t advocate for people getting thrown in jail, I just think that the whole “residential proxies” issue might be a charade that would not last longer if citizens and businesses would claim rights instead of play victim.
I also really dislike grasping for a tech solution because I don’t want to start to mine crypto to read an article or to depend on some “free” cloud service to host an open bug tracker.
It just feels so weird: why is it that no-one is doing it? I feel like I an one “comment away” from someone explaining to me how naive I am, and yet it’s never coming.
I like lesswrong's approach to this. You get 4xx if you scrape that site but greaterwrong.com serves some kind of static site that is scrape friendly. Seems fine to me.
Though, checking, I see that it requires a login but is still up. I misunderstood what 'closed' meant. This seems fine.
Are we assuming that this is the companies themselves scraping data from training or is this "agents" acting on behalf of users? Nowadays every major chat UI (ChatGPT, Claude etc) has a "tool" that allows LLM to load web pages, so it must generate some traffic.
No, most of the load comes from armies of residential IP addresses that look like Google Chrome on the wire. The major chat UIs properly identify themselves. These waves of attacks do not.
What are they scraping the gentoo bugzilla for? I'm confused. Unless you're actively using Gentoo why would this be a resource? Very confusing. Also you'd think we'd have LLM BitTorrent by now, where if they want to scrape something we get a DHT hash for the content and share it with one another, rather than melt servers with the millionth request of the day.
At this point they've mostly run out of material, so ANY type of content is valuable. Your small personal website, why would they scrape that? It's 10.000 additional words, wouldn't want to miss that. My Github repos.... got to get buggy code from somewhere I guess.
I get what you're asking, and I'm wondering the same. Not all sources are created equally and we see the results all the time. LLMs outputs nonsense all the time, like Flock cameras containing 5 grams of gold and ounces of copper, because they are completely on critical of their sources. Perhaps there's some weights that says: Kernel mailing list, MariaDB documentation and Microsofts Learning sites are 100% trust, Reddit 50%, 4Chan 10%, but I doubt it.
Anthropic might care a little bit, seeing as they scan books, but again, is it just all books? Because other than some flowery language I don't really see the point in scanning a 1970s paperback only spy novel.
They're scraping everything. It doesn't matter what. It doesn't matter if it makes sense. They just scrape it all.
"There Will Be Blood" but it's 2026 and Plainview's child is AI.
They're mining the written output of technical talent discussing troubleshooting, bug finding, patches, inter-dependencies, etc. It's an AI trainer's wet dream.
Also: where exactly are AI companies incentivized to be anything but shitty 'neighbors'?
Spam, DDoS attacks and other network abuse used to cause your hosting company or provider to call you and tell you to knock it off or you'll get disconnected, if your provider was reputable. If your provider wasn't reputable, it was likely a matter of time before they would get a nasty call from their upstream provider.
Now it just gets you a thank-you from the sales team for all the bandwidth you bought.
Meanwhile, do any of the cloud providers have any incentive to do anything about this? Hell no. They're making money off you having to ramp up extra or bigger instances. They're making money off the bandwidth. They're making money off the people doing the crawling, too. They're incentivized to do the exact opposite of effectively help you with your AI bot problem.
Lots of build failure detailed investigations and gcc/kernel expertise in debugging misbehaving or outright ICEs.
That's it, I guess?
There's a reddit thread from 9 years ago with people complaining the site blocking crawlers and thus not being indexed by search engines. So I'm not sure what they consider "overload." At this point they should dump the db on thepiratebay.
Dumping the database won't stop the scrappers because humans running the scrappers are not even aware Gentoo exists. They're just scrapping every website their bots find.
(Not saying that it is the only solution / the right solution)
Would fronting it with Anubis have helped at all here?
(please do not ramble about how requiring javascript is a violation of your human rights)
Indexers should pay for each request.
Maybe Taler could help ?
And people complain about cloudflare/anubis/etc. Unfortunately, it's looking like this is the alternative.
going on a bit of a tangent here - some discussion there seems to be imply that AI bots mostly use IPv4s, which makes sense to me given that they are probably bots hosted by some cloud. Whereas IPv6 may be more organic traffic from e.g. mobile users (not in this case probably)... which made me think if at some point IPv6 may at some point win against IPv4 just because its more organic traffic (i.e. not from big tech cloud), leading to pages blocking IPv4? Just speculation on my side
The hardest-to-mitigate bot traffic tends to come from residential proxies, and most residential connections are still v4-only.
However, it is also easy to get large numbers of v6 addresses cheaply.
My residential ISP doesn't support IPv6 at all. If a site is ipv6 only, I can't reach it unless I use a VPN.
This is weird, every VPS I've rented in the last 8 years had IPv6, and some didn't even had IPv4.
If a problematic bot is on any mainstream VPS it will just get shut down.
How would I shut down a bot on Digital Ocean making over a million requests a day?
Send an email to the Digitalocean abuse email with the time ranges and origin IPs.
I did this with OVH, never got any response.
abuse@ contacts never send responses (though some send acknowledgements) because they don't want to give any details about their abuse process.
So where are bugs reported now?
That's the thing, they aren't.
Gentoo still has an active IRC and mailing-list. Back to basics? Then scrapers just see the static archives from, MARC I guess.
Gentoo has pretty much always had a Bugzilla, even back in the early 00s... there never was an alternative plan.
The biggest thing about what's happened today though, is how a "janitor" on the infrastructure team can shut down the entire distros ability to work on bugs. The decision to shut down should have been left to someone else like robbat.
Mgorny has consistently caused issues for Gentoo, and many people have left the community after having to deal with him, but it seems comrel/devrel can't/won't remove him for whatever reason.
Edit: it looks like bgo is up now. Hopefully it stays that way.
I want AI to progress to a point where it can fix these static sites who fall over due to how inefficient they are.
Just put the website behind an x402 gate [0] (yes it uses stablecoins) requiring payment if a bot wants to access the content which acts like a Captcha and detect them if they are on direct data center IP or are using residential proxies.
Stablecoins have a great use-case and you get paid for bots to access the site with the humans living in peace without the site getting botted.
Job done.
[0] https://stevekrouse.com/x402
Please write a function that returns true when the user is using a residential proxy and false when it is a normal user on a residential IP address.
Every website that wants to, can just charge for dumps. Anubis or even Cloudflare can handle the rest I think.
Scrapers will ignore the dumps.
You can link to them in 10 million HTTP 429 responses, they will still ignore them.
Those scrapers can be blocked. Some scrapers won't ignore the responses, and maybe it'll lead to a meaningful reduction in scraping traffic.
AI skynet is winning. It is stealing time from humans, thus forcing down their activity, as can be seen here.
A headline would be great if those AI companies would close down. I hold them all responsible for this.
Plot twist: the AI wins not through terminators but a kafka nightmare
Corrected title: "Gentoo bugzilla closed because the guy running it, who says it is 'unusable anyway', saw a lot of traffic from different IP addresses with no clear pattern and accuses AI"
The year is 2026, somehow peoples basic web apps are not able to keep up with scrapers. Scrapers are not new. My side load is like .01 even with 10x the traffic of last year. It's called "serving static content", "caching" and many other things that are not new concepts.
Then you've got good old cloudflare which is free to use
1. You shouldn't need to be a professional sysadmin to be able to deploy a basic semi-dynamic website
2. Considering the spike of complaints over the past two years and the fact that they range from small time site operators to larger orgs (Linux Kernel, GNOME, Duke University, etc), do you think that maybe you're missing part of the problem space?
All these experts here in advanced computing but people don't know how to cache with webpages?
Are you volunteering to setup and host that extra complexity for the Gentoo project?
A bugtracker is inherently dynamic.