How exactly do they obtain this data? Residential proxy providers don't publish their IP list (you connect to one of their servers which then tunnels your traffic to the residential exit point). Plus there are tons of such providers.
In any case, residential proxies are a godsend. Because most of the everyday services like web shops, govt information portals, even personal blogs sometimes are blocking access usings captchas.
EDIT: they write this on their marketing copy [0]:
> The Spur platform identifies traffic originating from residential proxy networks by analyzing service fingerprints, ASN ownership, and behavioral indicators.
Sounds like guesswork that results in a ton of false positives. And the more innocent IPs are blocked by platforms on the basis of this data, the more the demand increases for residential proxies, from users who need access to essential services. Talk of creating the problem and then selling the "solution".
I work for IPinfo. We offer a residential proxy detection service, which you can check at ipinfo.io/my.
We had a previous discussion about surfacing visitor IP address resproxy status explicitly. Should we have some sort of badge or a more explicit alert to show if a site visitor's IP address is part of a residential proxy network?
Even though it is great for demonstrating the product's value, it is kind of a low-tier value. What can a user actually do when they realize their IP address is part of a residential proxy pool?
The first issue is that residential proxy SDK infiltration is massive. If you start connecting to different IP addresses and constantly check your IP address on our website, you will often see that many of those IP addresses were, at some point, part of a residential proxy pool. We provide frequency information showing how many times an IP address was observed in a residential proxy pool, with a default observation period of 7 days.
Then there is the question of what a user can actually do about it. If it is a controlled IT environment with paranoid IT admins, sure, they can actively monitor traffic and identify why their IPs are showing up in residential proxy pools. They can attempt to do something about it. But it is not easy even then.
Residential proxy SDKs can simply be baked into almost any smartphone or smartphone-derived OS that allows app installation through marketplaces. So, many residential networks are already cooked (because of android TVs). Moderate-scale NAT connections almost always see residential proxy flags, as do public Wi-Fi hotspot IPs, which we also detect.
Identifying the apps that are generating background network traffic is quite hard. You need some level of DNS monitoring or a network sniffer. Alternatively, you need router-level firewall software.
These SDKs are not always sending high-volume, constant traffic that makes them easy to detect. If you see a 100% residential proxy flag for your IP address, then they probably are. But in many cases, the traffic is intermittent and much harder to identify.
Nobody has an answer to what I should do when I see my IP address in a residential proxy pool. It has been accepted in spirit as a "consented malware" for the last few years. It is undetectable and extremely hard to remove because the SDK has been baked into apps themselves.
there was a blog post linked on this thread explaining how proxy IP lists (spur, synthient, ipinfo et al) have little actionable value and introducing an alternative real-time approach to detection.
But it got flagged/downvoted into removal (twice!). Someone here has lots of HN accounts and doesn't tolerate free competition.
Disclaimer: I'm the founder and main researcher of the "flagged" company.
I have been part of this community for over a decade, and in my experience the mods do take flagging and voting irregularities seriously when they are reported. If you believe there is manipulation happening on your posts, that is worth raising directly with them, since they have visibility we do not.
---
On the broader point, we process 3 trillion requests last year, have more than 80 employees, and run a dedicated privacy engineering team led by an ex-cybersecurity company founder. We invest in research on new detection methods and stay closely engaged with the developer community. If there are specific gaps you see in our product, I would be glad to hear them and discuss.
---
Whether any dataset, including residential proxy IP data, is valuable depends heavily on the application. Treating a dataset as invalid because it does not fit one particular model can lead to decisions on shaky ground.
We are regarded as one of the more if not the most accurate IP geolocation providers, and we spend considerable effort on education, solutions architecture, and documentation so customers understand what our data can and cannot support. For example, IP geolocation, even at highest level of accuracy, is not a person identifier. It will never be a 1:1 replacement of GPS geolocation.
Many of the largest companies in AI, anti-bot, fingerprinting, KYC, and CDN spaces use our data. If anti-bot systems and CAPTCHAs were fully reliable on their own, there would be less need for additional signals like residential proxy data. We do not assign a score or label an IP as good or bad. That judgment sits with the customer's own threat or analytics model.
Residential proxy IPs are, by and large, mostly used in web scraping operations of many different forms. If a company sees a moderate to high amount of traffic mimicking human behavior, it can struggle to tell bot traffic apart from real users. Anti-bot mechanisms can help, but they add friction to the user experience, and they are not cheap to run at scale.
Residential proxy detection data is one of the easiest zero-knowledge ways to gather intelligence. There is no need for users to solve a puzzle or for multi-page traversal to collect fingerprint data. All that is needed is the IP address.
Our residential proxy data customers tend to be on the more sophisticated side of cybersecurity. Suggesting that this data is a silver bullet for all their security needs would not reflect well on their expertise or ours. We present the data as is, and from there we work with customers on the right solution for their case.
In my testing, Synthient's tool[0] and offerings have performed very well for this kind of service. Synthient have also achieved impressive proven success against malicious botnets[1].
Would be great if it told me how recently it was detected. I have a dynamic IP from my ISP and it could very well be someone else's device 3 days or 3 months ago.
That's cool, but I get a new IP every 12 hours so it's still not enough to determine if I'm part of a botnet or I just got stuck with someone else's poisoned IP
this company Spur recently got millions in funding and their pricing seems to be directed at large companies. Who's buying these absolutely non-actionable IP databases? Do corporate buyers not understand you can't just block someone because they share the IP with someone else who downloaded a dodgy app?
I know that SpamHaus is using this dataset as a large (at times sole) contributor to their blocklists. So corporate buyers are doing exactly what you describe, albeit indirectly.
Lack of any IoCs also makes it hard to refute or remedy. Plus I think it paints even Tor relay nodes with the same brush as malicious proxies.
Truly kafkaesque if you start getting restricted and nobody tells you why or even knows what to tell you because the sources have all been mixed and obscured.
Spur's residential proxy data is not intended to be used as a blacklist. We recommend using it as enrichment alongside other signals, not blocking an IP just because it was associated with proxy activity.
>If people who have a smart TV have their smart TV participate in a residential proxy, then it's billions of IP getting "flagged".
There's a non-trivial quantity error here that makes the argument of a majority qualitatively incorrect.
It's not billions of IPs that are being used in residential proxies, it's not all smart TVs.
There needs to be a vulnerability and hacked devices OR there needs to be a very low quality and shady vendor that is offering products at too cheap prices and needs to make ends meet in order to compete at that price. Probably chinese.
This would be in the range of 1 to 100 million smart TVs. sorry for the wide range, but definitely not 1Billion or every TV.
So to the extent that the ratio of infected to non infected IPs is low, then providers can block the infected ones to a great effect.
I assume that eventually the flag will just become meaningless and there will be other methods of verification in use by then... because cutting off a huge chunk of your customers just isn't good business.
But for now I'm already cut off from half the internet due to endless crimeflare captcha loops... I just don't visit those sites anymore because I literally can't.
But whether your IP address is being used as a residential proxy is already important information. It answers the question (is this IP address low quality?)
Although I'll grant that it would be interesting to know if your devices are running the proxy, but you'll need an exeuctable tool for that, not a per-ip network tool.
Keep in mind these databases can be wildly inaccurate and basically impossible to prove them wrong (you can't prove a negative).
I've seen this (and verified with others) with other sites like iknowwhatyoudownload.com where they allege your connection downloaded something very illegal (like CSAM) even though you know for certain it never happened and you haven't been hacked.
There really isn't any proof that they can show, since iknowwhatyoudownload legally aren't allowed to download anything that you're seeding (since that would be considered piracy).
IIRC, they only show the filename, last-seen timestamp and user agent. I can't verify it though since their website seems to be down?
It says I was observed on a couple areas, but not sure what to do with that information. It would be great if thi tool provided links to guides to discover more.
Some info on methodology would be necessary for a paid plan for two purposes, one to audit that the quality of the methodology and the signal is good (it's not hallucinated or checking few sources).
But also to make sure it doesn't clash with other signals if used in conjunction with other sources of tool (if I have another signal, I want to know whether they are redundant or complementary, to avoid interpreting two positives as independently verified.
wow my comment with a blog post about a better approach to detect residential proxies got... removed! Not sure about HN internals, do moderators do that or is this OP using several accounts to downvote contrarian comments?
How exactly do they obtain this data? Residential proxy providers don't publish their IP list (you connect to one of their servers which then tunnels your traffic to the residential exit point). Plus there are tons of such providers.
In any case, residential proxies are a godsend. Because most of the everyday services like web shops, govt information portals, even personal blogs sometimes are blocking access usings captchas.
EDIT: they write this on their marketing copy [0]:
> The Spur platform identifies traffic originating from residential proxy networks by analyzing service fingerprints, ASN ownership, and behavioral indicators.
Sounds like guesswork that results in a ton of false positives. And the more innocent IPs are blocked by platforms on the basis of this data, the more the demand increases for residential proxies, from users who need access to essential services. Talk of creating the problem and then selling the "solution".
[0] https://spur.us/platform/residential-proxy-detection
I don't think they have false positives to the extent you're saying. It's not as much guesswork as you think.
I work for IPinfo. We offer a residential proxy detection service, which you can check at ipinfo.io/my.
We had a previous discussion about surfacing visitor IP address resproxy status explicitly. Should we have some sort of badge or a more explicit alert to show if a site visitor's IP address is part of a residential proxy network?
Even though it is great for demonstrating the product's value, it is kind of a low-tier value. What can a user actually do when they realize their IP address is part of a residential proxy pool?
The first issue is that residential proxy SDK infiltration is massive. If you start connecting to different IP addresses and constantly check your IP address on our website, you will often see that many of those IP addresses were, at some point, part of a residential proxy pool. We provide frequency information showing how many times an IP address was observed in a residential proxy pool, with a default observation period of 7 days.
Then there is the question of what a user can actually do about it. If it is a controlled IT environment with paranoid IT admins, sure, they can actively monitor traffic and identify why their IPs are showing up in residential proxy pools. They can attempt to do something about it. But it is not easy even then.
Residential proxy SDKs can simply be baked into almost any smartphone or smartphone-derived OS that allows app installation through marketplaces. So, many residential networks are already cooked (because of android TVs). Moderate-scale NAT connections almost always see residential proxy flags, as do public Wi-Fi hotspot IPs, which we also detect.
Identifying the apps that are generating background network traffic is quite hard. You need some level of DNS monitoring or a network sniffer. Alternatively, you need router-level firewall software.
These SDKs are not always sending high-volume, constant traffic that makes them easy to detect. If you see a 100% residential proxy flag for your IP address, then they probably are. But in many cases, the traffic is intermittent and much harder to identify.
Nobody has an answer to what I should do when I see my IP address in a residential proxy pool. It has been accepted in spirit as a "consented malware" for the last few years. It is undetectable and extremely hard to remove because the SDK has been baked into apps themselves.
there was a blog post linked on this thread explaining how proxy IP lists (spur, synthient, ipinfo et al) have little actionable value and introducing an alternative real-time approach to detection.
But it got flagged/downvoted into removal (twice!). Someone here has lots of HN accounts and doesn't tolerate free competition.
Disclaimer: I'm the founder and main researcher of the "flagged" company.
I have been part of this community for over a decade, and in my experience the mods do take flagging and voting irregularities seriously when they are reported. If you believe there is manipulation happening on your posts, that is worth raising directly with them, since they have visibility we do not.
---
On the broader point, we process 3 trillion requests last year, have more than 80 employees, and run a dedicated privacy engineering team led by an ex-cybersecurity company founder. We invest in research on new detection methods and stay closely engaged with the developer community. If there are specific gaps you see in our product, I would be glad to hear them and discuss.
---
Whether any dataset, including residential proxy IP data, is valuable depends heavily on the application. Treating a dataset as invalid because it does not fit one particular model can lead to decisions on shaky ground.
We are regarded as one of the more if not the most accurate IP geolocation providers, and we spend considerable effort on education, solutions architecture, and documentation so customers understand what our data can and cannot support. For example, IP geolocation, even at highest level of accuracy, is not a person identifier. It will never be a 1:1 replacement of GPS geolocation.
Many of the largest companies in AI, anti-bot, fingerprinting, KYC, and CDN spaces use our data. If anti-bot systems and CAPTCHAs were fully reliable on their own, there would be less need for additional signals like residential proxy data. We do not assign a score or label an IP as good or bad. That judgment sits with the customer's own threat or analytics model.
Residential proxy IPs are, by and large, mostly used in web scraping operations of many different forms. If a company sees a moderate to high amount of traffic mimicking human behavior, it can struggle to tell bot traffic apart from real users. Anti-bot mechanisms can help, but they add friction to the user experience, and they are not cheap to run at scale.
Residential proxy detection data is one of the easiest zero-knowledge ways to gather intelligence. There is no need for users to solve a puzzle or for multi-page traversal to collect fingerprint data. All that is needed is the IP address.
Our residential proxy data customers tend to be on the more sophisticated side of cybersecurity. Suggesting that this data is a silver bullet for all their security needs would not reflect well on their expertise or ours. We present the data as is, and from there we work with customers on the right solution for their case.
In my testing, Synthient's tool[0] and offerings have performed very well for this kind of service. Synthient have also achieved impressive proven success against malicious botnets[1].
0: https://synthient.com/context/ip/
1: https://www.wsj.com/tech/kimwolf-hack-residential-proxy-netw... / https://archive.ph/SpKVn
Would be great if it told me how recently it was detected. I have a dynamic IP from my ISP and it could very well be someone else's device 3 days or 3 months ago.
The TTL for our data is 2-3 days so you can be fairly confident that it is up to date.
That's cool, but I get a new IP every 12 hours so it's still not enough to determine if I'm part of a botnet or I just got stuck with someone else's poisoned IP
This would probably false positive every CGNAT IP, or am I misunderstanding something?
I think "residential proxy" in this context is "AI scraping bot/pay Netflix at Indian price VPN exit node", not just an IP with NAT on outside.
If there were people under the same CGNAT with someone running one of those exit nodes, then that's not a false positive.
I think CGNAT is a type of residential proxy.
Hidden from the user and provided by the ISP
CGNAT isn't a residential proxy
Yes, it did and it does.
this company Spur recently got millions in funding and their pricing seems to be directed at large companies. Who's buying these absolutely non-actionable IP databases? Do corporate buyers not understand you can't just block someone because they share the IP with someone else who downloaded a dodgy app?
I know that SpamHaus is using this dataset as a large (at times sole) contributor to their blocklists. So corporate buyers are doing exactly what you describe, albeit indirectly.
Lack of any IoCs also makes it hard to refute or remedy. Plus I think it paints even Tor relay nodes with the same brush as malicious proxies.
Truly kafkaesque if you start getting restricted and nobody tells you why or even knows what to tell you because the sources have all been mixed and obscured.
Spur's residential proxy data is not intended to be used as a blacklist. We recommend using it as enrichment alongside other signals, not blocking an IP just because it was associated with proxy activity.
More on why here: https://spur.us/blog/i-dont-like-big-gateways-and-i-cannot-l...
SpamHaus is using it as a large contributor to their blocklists, at times as the sole signal, just so you know.
thanks, that was exactly my point. Residential proxy signal's value is almost 0 but this product moves in the "I solve it all for you" price range.
My public IP is shared with some 150 people in a student dorm, and the result is negative which I find very unlikely.
Depends quite a bit on your university.
Some are quite good at limiting connections, monitoring devices, and contacting students for suspicious traffic.
what if i have an always rotating ip?
I am on mobile network and it fails to consider this as a factor that other people who might receive this IP could be having a proxy.
If the IP address you're using has been detected as a residential proxy, it doesn't matter. It's going to be flagged on lists for a long time.
Being able to check is helpful.
> If the IP address you're using has been detected as a residential proxy, it doesn't matter. It's going to be flagged on lists for a long time.
Flagged and then... What exactly?
If people who have a smart TV have their smart TV participate in a residential proxy, then it's billions of IP getting "flagged".
What's the use of flagging those?
When every IP is flagged, none is.
Mine isn't flagged. Not gonna set up a smart TV.
>If people who have a smart TV have their smart TV participate in a residential proxy, then it's billions of IP getting "flagged".
There's a non-trivial quantity error here that makes the argument of a majority qualitatively incorrect.
It's not billions of IPs that are being used in residential proxies, it's not all smart TVs.
There needs to be a vulnerability and hacked devices OR there needs to be a very low quality and shady vendor that is offering products at too cheap prices and needs to make ends meet in order to compete at that price. Probably chinese.
This would be in the range of 1 to 100 million smart TVs. sorry for the wide range, but definitely not 1Billion or every TV.
So to the extent that the ratio of infected to non infected IPs is low, then providers can block the infected ones to a great effect.
Bright Data claim to have 400 million IPs.
https://brightdata.com/proxy-types/residential-proxies
I assume that eventually the flag will just become meaningless and there will be other methods of verification in use by then... because cutting off a huge chunk of your customers just isn't good business.
But for now I'm already cut off from half the internet due to endless crimeflare captcha loops... I just don't visit those sites anymore because I literally can't.
Yeah, clicked from my mobile network without thinking and nearly jumped out my skin.
I added a warning that should help with this now.
But whether your IP address is being used as a residential proxy is already important information. It answers the question (is this IP address low quality?)
Although I'll grant that it would be interesting to know if your devices are running the proxy, but you'll need an exeuctable tool for that, not a per-ip network tool.
Keep in mind these databases can be wildly inaccurate and basically impossible to prove them wrong (you can't prove a negative).
I've seen this (and verified with others) with other sites like iknowwhatyoudownload.com where they allege your connection downloaded something very illegal (like CSAM) even though you know for certain it never happened and you haven't been hacked.
Do they show proof of the positive if you show up in there?
There really isn't any proof that they can show, since iknowwhatyoudownload legally aren't allowed to download anything that you're seeding (since that would be considered piracy).
IIRC, they only show the filename, last-seen timestamp and user agent. I can't verify it though since their website seems to be down?
Seems it only detects ipv4. Any plan to support scanning ipv6 /56 range?
It says I was observed on a couple areas, but not sure what to do with that information. It would be great if thi tool provided links to guides to discover more.
Which proxy network(s) did you get tagged in?
Needs an API to query other IPs beyond one's own.
Try https://spur.us/context/<ip> where <ip> is the IP you want to query :)
Thanks! Do you plan on a paid plan? Would you be able to provide methodology under NDA if needed?
(cyber consultant, have people who might use this for enrichment in security stacks)
+1
Some info on methodology would be necessary for a paid plan for two purposes, one to audit that the quality of the methodology and the signal is good (it's not hallucinated or checking few sources).
But also to make sure it doesn't clash with other signals if used in conjunction with other sources of tool (if I have another signal, I want to know whether they are redundant or complementary, to avoid interpreting two positives as independently verified.
wow my comment with a blog post about a better approach to detect residential proxies got... removed! Not sure about HN internals, do moderators do that or is this OP using several accounts to downvote contrarian comments?
I think it was down voted as it looks like self promotion
all links currently on this thread are self promotion, OP included. They just own more HN accounts to vote unwanted comments away ¯\_(ツ)_/¯
very nice.
Can we use it in other IPs? (without having to issue the request from that IP)
Yes. You can use https://spur.us/context/<ip> where the IP is the one you want to lookup.