At first, I thought that the fraction of people who want this kind of network of peers would be rather small. I thought that almost all people want extensive restrictions on user agents and controlling how the data is revealed to the user. But Bluesky actually does not restrict AI scraping and consumption of data. You're just yet another consumer, subject to rate-limiting etc. whether you are browser or headless agent. I have to say that I am somewhat (pleasantly) surprised that this approach is accepted by Bluesky considering the community is more anti-AI than most. The fact that you're allowed to consume the feed is amazing. Good stuff!
For blog posts, I'd look to RSS instead. That's where that content is traditionally published. Instead of SELECTing from bluesky's index, you can use OPML subscription lists. There are a bunch of places that curate feed lists, so it's significantly less likely to face API death like twitter did.
I run another variant which collects OPML blogrolls via crawling, so you can find out who else likes your favorite blog and what else they recommend. Here's the page for Simon Willison's blog (https://blogroll-network.alexsci.com/discover/feed-a34ee2a88...). Thinking of RSS and blogrolls as a network feels much more resilient than blueskys Jetstream api endpoint.
> Thinking of RSS and blogrolls as a network feels much more resilient than blueskys Jetstream api endpoint.
Nothing to do with Bluesky services. There are many independent firehoses and relays. Here's a stream of standard.site blog posts coming in over a firehose hosted in Chennai. Every single one. No curator between me and the posts, and no work by me to crawl the whole network for them. https://pdsls.dev/jetstream?instance=wss%3A%2F%2Fchennai.fir...
Edit: ok there is an aggregator, the relay is scraping all of the PDSs out there to build the event stream. Notably this is not possible with RSS, where you need to build a large index with knowledge. PDSs request relays to crawl and that's that.
Oh the opml blogroll map is pretty exciting! I occasionally try to browse around for these manually but they can be hard to explore for so having a big list is a big deal. I usually just do searches for them + use a scour.ing interest to find more. I'm excited to look at it more this weekend when I'm not so tired.
One thing I'd love to see with rss / ompl sharing sites is more ease of exploration. It feels a bit clunky browsing many of these feed sharing sites because you need to evaluate each feed by manually clicking through each one etc or worse you can only import the whole opml at once flooding your feeds. For example I've seen some sites with feeds directly showing posts as they come in for each persons feed list(s) so you can see what that batch of rss feeds looks like in action without clicking around much. There are some sites doing great work getting people to share their lists but I think there is room for qol improvements.
One site I've been liking that has feeds for each list and is trying to reduce friction on sharing is blogflock.com. You can also follow other peoples lists directly on the site too so they show up in your main feed. I believe I've also seen some people self host their blogroll on their personal sites with similar feeds but I'm having trouble finding that software atm.
Absolutely. One good thing going for this approach is that anyone can grab the OPML export (the URL is stable) and build their own frontend, I'd love to see more.
Could be a fun weekend project for frontend folks.
> The web is a few dozen companies, each holding a filing cabinet, each with a receptionist posted out front. He'll read you one file at a time, but only files you can name, as fast as he cares to read, and as long as his boss allows.
Fortunately this isn't true. Social media consists of a few big walled gardens, but the web itself is still open. Go forth and create a website!
There have been so, so many attempts to decentralize in my lifetime. I used to be a huge believer and worked on these problems, but have to admit I'm somewhat exhausted. The technology is not the problem. The problem is always quality, effort, and cost.
If decentralization is ever going to win, it needs to be turnkey, explained without showing a network diagram, be basically free to run on your laptop, and actually have the content people want to see and not just be a bunch of people who speak lojban. [1]
While I'm not sure how much of a legal leg X has to stand on, I understand why they'd rather Nitter not exist. Twitter tried (valiantly, imo) to stay open. What ultimately began the API lockdown was the need to stop bleeding financially. Ads were inevitable, and there being no reliable way to do that via API access.
Twitter solved those three problems, and no one spends nearly as much time talking or caring about the technology used to do it than people who talk about decentralization. Myself included in my younger days. Now they're just defending their moat.
I can't decide if ATProto is actually amazing or if its proponents are just dazzled by their own tech. Is it truly decentralized? What makes it better than Mastodon or Matrix or Scuttlebutt? Of course, at the end of the day, it's the network effects that matter. Attention is all you need indeed.
The team came from the IPFS and Scuttlebutt worlds. "Better" is relative to your design goals. Iroh is better for point-to-point connections & localfirst. Matrix is better for chat rooms. Etc. The general logic of atproto was to serve large scale publishing with an open model that doesn't seriously degrade the UX. The scale orientation means - at least until the non-public spaces land - you're syncing a lot of records to provide an alternative microblogger to bluesky. The account hosting system, however - the pds - remains exceedingly cheap and easy to run. Pros and cons, matching with the design goals.
I will say that I have no relation to bluesky so it's not my tech and I do find it amazing. This post from Dan Abramov might help in clarifying how 'decentralized' it is compared to the threadiverse https://overreacted.io/there-are-no-instances-in-atproto/
Feels like a solution in search of a problem - just like all the others. Also feels like only half a solution - just like anything else developers invent. Do you remember what Joel Spolsky said about P2P architectures and Napster? It doesn't matter whether your system is P2P or not, it matters whether or not the user can type in the name of a song and then listen to it. That's how YouTube is the new Napster. Because you can just type in a song and listen to it, and for the sake of determining popularity, it's completely irrelevant whether this is done by stealing the music or by legally licensing it. However, the ads that pay for that licensing are relevant.
ATProto is attempting the "if we make it P2P it'll be popular!" strategy except not with P2P.
The Twitter-competitive social site is moribund, so I think maybe what you're seeing more and more of is the pivot towards infrastructure --- always planned, but not before such a big part of the messaging.
To put it another way: Google search works by making their own copy of the Internet. It's much smaller scale, but similarly, atproto makes downloading the entire social network relatively easy. Many ordinary techies have the resources to replicate the whole thing if they wanted to. To query your replica, you use a database, designed however you like.
So the question is, are you okay with giving random people a mirror of your public posts? After all, they're public. It's like putting them in a repo on Github for anyone to clone.
Google's Go Module Mirror is a similar but more specialized service that has a copy of all the Go modules that anyone has published on the Internet.
There are invariably 3 solutions to all “federated” web problems:
- Email
- Semantic HTML
- Web indexing services
In this case, it's mostly indexing and a little of the other two if you want to add social features. I despite ATProto with all my soul for taking a problem with such a simple and standard solution and totally obscuring it behind hundreds of layers of JSON, faux federated services, and technical jargon, all in service of creating an inferior version of Twitter. I guess it wouldn't be as sexy to offer a web indexing service instead.
Genuinely curious, how would you build it differently? Would you stick with atproto's indexing model and try to simplify it, or would you use another approach?
You send a request to a web indexer with a page you want added to the network, it scans your page for outgoing links and sends an email to the owners of any domains who are also on the service.
One can imagine making it a little more featureful, for instance indexing OPML blogrolls which would allow you to see whom a person follows and https://microformats.org/wiki/h-entry which would allow outgoing links to be categorised (like, reply, etc) so notifications filtered. With that, it would be possible to add a front-end site imitating one of the popular social media paradigms (Reddit-style or Twitter-style being the most obvious).
If there's one genuine design decision I would give for it beyond what is basically a cobbling together of existing interfaces, it would be to charge users per page uploaded. Likely very detrimental to growing the service, but I think one of the simplest ways to weed out spam and junk. A real problem with many web services is that the receiver of the message pays for it in terms of attention, where in other mediums the sender has to pay. Given that uploading crap is basically free, that's all you get. Increasing the cost of upload would weed out those endless AI summaries and lists of affiliate links.
The issue with ATProto is that it operates on too many layers. You can see lacking in what I have described here the concept of durable authorship, but this is a property of content not how the content is distributed. Perhaps someone will invent a standard way to sign HTML documents, in which case you could base user accounts on that instead of DNS. AT enforces this centrally but it does not need to. It is walling itself off from the common and decentralised software ecosystem of the web for no good reason.
Isn't that how it already works but with different protocols that you don't like? You (a PDS) send a request to an indexer (relay) with a page (activity) you want added to the network.
Yeah that'd be interesting to try. If you extend the endpoints and the vocabularies around RSS/OPML enough, you could likely create a fairly robust dataset replication protocol. Microformats are a decent schema basis, but you might want to build upon them as well. If you want to make the users' sites reusable as a datastore across applications, you could introduce an oauth flow to enable 3rd party writes. You might also want to add content signatures so you can verify the data authenticity from third parties (e.g. to support the equivalent of reposts).
You would repost a page by linking to it from your own. As for uploading content, that is outside of the scope of an indexer. The user hosts their data with some hosting service, or independently. I believe signatures are already handled by XHTML which allows you to add XML signatures to documents, but regular HTML is sadly lacking here thought it would not be too hard to extend (or to just use the XML serialisation of HTML5).
I think it is generally bad to implement new features like this on the part of the indexer. It should just keep track of an existing web of documents rather than creating it's own format and walled garden.
> You would repost a page by linking to it from your own.
Ah right, of course. Fair enough. I think Dave Winer is trying some similar ideas, though I haven't looked at it beyond knowing he's trying to extend RSS to support these capabilities.
I'm sorry, but this blog post is written as if the described functionality should obviously be available and supported by the Internet and companies who expose their data.
I agree it would be neat.
But, when did we conclude this "should" be possible and thus warrant such a whiny post?
At first, I thought that the fraction of people who want this kind of network of peers would be rather small. I thought that almost all people want extensive restrictions on user agents and controlling how the data is revealed to the user. But Bluesky actually does not restrict AI scraping and consumption of data. You're just yet another consumer, subject to rate-limiting etc. whether you are browser or headless agent. I have to say that I am somewhat (pleasantly) surprised that this approach is accepted by Bluesky considering the community is more anti-AI than most. The fact that you're allowed to consume the feed is amazing. Good stuff!
For blog posts, I'd look to RSS instead. That's where that content is traditionally published. Instead of SELECTing from bluesky's index, you can use OPML subscription lists. There are a bunch of places that curate feed lists, so it's significantly less likely to face API death like twitter did.
There are multiple sites that support follower semantics over RSS. Feedland tracks subscriptions publicly, so you can see the blogs I read (https://feedland.com/?username=robalexdev), and who reads my blog (https://feedland.com/?feedurl=https%3A%2F%2Falexsci.com%2Fbl...).
I run another variant which collects OPML blogrolls via crawling, so you can find out who else likes your favorite blog and what else they recommend. Here's the page for Simon Willison's blog (https://blogroll-network.alexsci.com/discover/feed-a34ee2a88...). Thinking of RSS and blogrolls as a network feels much more resilient than blueskys Jetstream api endpoint.
> Thinking of RSS and blogrolls as a network feels much more resilient than blueskys Jetstream api endpoint.
Nothing to do with Bluesky services. There are many independent firehoses and relays. Here's a stream of standard.site blog posts coming in over a firehose hosted in Chennai. Every single one. No curator between me and the posts, and no work by me to crawl the whole network for them. https://pdsls.dev/jetstream?instance=wss%3A%2F%2Fchennai.fir...
Edit: ok there is an aggregator, the relay is scraping all of the PDSs out there to build the event stream. Notably this is not possible with RSS, where you need to build a large index with knowledge. PDSs request relays to crawl and that's that.
Oh the opml blogroll map is pretty exciting! I occasionally try to browse around for these manually but they can be hard to explore for so having a big list is a big deal. I usually just do searches for them + use a scour.ing interest to find more. I'm excited to look at it more this weekend when I'm not so tired.
One thing I'd love to see with rss / ompl sharing sites is more ease of exploration. It feels a bit clunky browsing many of these feed sharing sites because you need to evaluate each feed by manually clicking through each one etc or worse you can only import the whole opml at once flooding your feeds. For example I've seen some sites with feeds directly showing posts as they come in for each persons feed list(s) so you can see what that batch of rss feeds looks like in action without clicking around much. There are some sites doing great work getting people to share their lists but I think there is room for qol improvements.
One site I've been liking that has feeds for each list and is trying to reduce friction on sharing is blogflock.com. You can also follow other peoples lists directly on the site too so they show up in your main feed. I believe I've also seen some people self host their blogroll on their personal sites with similar feeds but I'm having trouble finding that software atm.
> more ease of exploration
Absolutely. One good thing going for this approach is that anyone can grab the OPML export (the URL is stable) and build their own frontend, I'd love to see more.
Could be a fun weekend project for frontend folks.
> The web is a few dozen companies, each holding a filing cabinet, each with a receptionist posted out front. He'll read you one file at a time, but only files you can name, as fast as he cares to read, and as long as his boss allows.
Fortunately this isn't true. Social media consists of a few big walled gardens, but the web itself is still open. Go forth and create a website!
>Simple: the network won. The developers stopped being an asset, and the API progressively closed.
You can add Reddit to that list, IPO is coming so better price out the third party apps we encouraged developers to build.
Reddit went public in 2024. What are you talking about ?
> you'll even get downvoted for "living in the past" if you attempt to bring it up later.
https://news.ycombinator.com/item?id=49390953
That comment was posted 6 days ago. What are you talking about?
There have been so, so many attempts to decentralize in my lifetime. I used to be a huge believer and worked on these problems, but have to admit I'm somewhat exhausted. The technology is not the problem. The problem is always quality, effort, and cost.
If decentralization is ever going to win, it needs to be turnkey, explained without showing a network diagram, be basically free to run on your laptop, and actually have the content people want to see and not just be a bunch of people who speak lojban. [1]
While I'm not sure how much of a legal leg X has to stand on, I understand why they'd rather Nitter not exist. Twitter tried (valiantly, imo) to stay open. What ultimately began the API lockdown was the need to stop bleeding financially. Ads were inevitable, and there being no reliable way to do that via API access.
Twitter solved those three problems, and no one spends nearly as much time talking or caring about the technology used to do it than people who talk about decentralization. Myself included in my younger days. Now they're just defending their moat.
1: https://xkcd.com/191/
We don't have to have any respect for X's profitability btw. We're allowed to despise it even while it's losing money.
ATProto has the hardest working developer relations department in the game right now.
I can't decide if ATProto is actually amazing or if its proponents are just dazzled by their own tech. Is it truly decentralized? What makes it better than Mastodon or Matrix or Scuttlebutt? Of course, at the end of the day, it's the network effects that matter. Attention is all you need indeed.
The team came from the IPFS and Scuttlebutt worlds. "Better" is relative to your design goals. Iroh is better for point-to-point connections & localfirst. Matrix is better for chat rooms. Etc. The general logic of atproto was to serve large scale publishing with an open model that doesn't seriously degrade the UX. The scale orientation means - at least until the non-public spaces land - you're syncing a lot of records to provide an alternative microblogger to bluesky. The account hosting system, however - the pds - remains exceedingly cheap and easy to run. Pros and cons, matching with the design goals.
I will say that I have no relation to bluesky so it's not my tech and I do find it amazing. This post from Dan Abramov might help in clarifying how 'decentralized' it is compared to the threadiverse https://overreacted.io/there-are-no-instances-in-atproto/
Feels like a solution in search of a problem - just like all the others. Also feels like only half a solution - just like anything else developers invent. Do you remember what Joel Spolsky said about P2P architectures and Napster? It doesn't matter whether your system is P2P or not, it matters whether or not the user can type in the name of a song and then listen to it. That's how YouTube is the new Napster. Because you can just type in a song and listen to it, and for the sake of determining popularity, it's completely irrelevant whether this is done by stealing the music or by legally licensing it. However, the ads that pay for that licensing are relevant.
ATProto is attempting the "if we make it P2P it'll be popular!" strategy except not with P2P.
The Twitter-competitive social site is moribund, so I think maybe what you're seeing more and more of is the pivot towards infrastructure --- always planned, but not before such a big part of the messaging.
To put it another way: Google search works by making their own copy of the Internet. It's much smaller scale, but similarly, atproto makes downloading the entire social network relatively easy. Many ordinary techies have the resources to replicate the whole thing if they wanted to. To query your replica, you use a database, designed however you like.
So the question is, are you okay with giving random people a mirror of your public posts? After all, they're public. It's like putting them in a repo on Github for anyone to clone.
Google's Go Module Mirror is a similar but more specialized service that has a copy of all the Go modules that anyone has published on the Internet.
There are invariably 3 solutions to all “federated” web problems:
- Email
- Semantic HTML
- Web indexing services
In this case, it's mostly indexing and a little of the other two if you want to add social features. I despite ATProto with all my soul for taking a problem with such a simple and standard solution and totally obscuring it behind hundreds of layers of JSON, faux federated services, and technical jargon, all in service of creating an inferior version of Twitter. I guess it wouldn't be as sexy to offer a web indexing service instead.
Genuinely curious, how would you build it differently? Would you stick with atproto's indexing model and try to simplify it, or would you use another approach?
You send a request to a web indexer with a page you want added to the network, it scans your page for outgoing links and sends an email to the owners of any domains who are also on the service.
One can imagine making it a little more featureful, for instance indexing OPML blogrolls which would allow you to see whom a person follows and https://microformats.org/wiki/h-entry which would allow outgoing links to be categorised (like, reply, etc) so notifications filtered. With that, it would be possible to add a front-end site imitating one of the popular social media paradigms (Reddit-style or Twitter-style being the most obvious).
If there's one genuine design decision I would give for it beyond what is basically a cobbling together of existing interfaces, it would be to charge users per page uploaded. Likely very detrimental to growing the service, but I think one of the simplest ways to weed out spam and junk. A real problem with many web services is that the receiver of the message pays for it in terms of attention, where in other mediums the sender has to pay. Given that uploading crap is basically free, that's all you get. Increasing the cost of upload would weed out those endless AI summaries and lists of affiliate links.
The issue with ATProto is that it operates on too many layers. You can see lacking in what I have described here the concept of durable authorship, but this is a property of content not how the content is distributed. Perhaps someone will invent a standard way to sign HTML documents, in which case you could base user accounts on that instead of DNS. AT enforces this centrally but it does not need to. It is walling itself off from the common and decentralised software ecosystem of the web for no good reason.
Isn't that how it already works but with different protocols that you don't like? You (a PDS) send a request to an indexer (relay) with a page (activity) you want added to the network.
Yeah that'd be interesting to try. If you extend the endpoints and the vocabularies around RSS/OPML enough, you could likely create a fairly robust dataset replication protocol. Microformats are a decent schema basis, but you might want to build upon them as well. If you want to make the users' sites reusable as a datastore across applications, you could introduce an oauth flow to enable 3rd party writes. You might also want to add content signatures so you can verify the data authenticity from third parties (e.g. to support the equivalent of reposts).
You would repost a page by linking to it from your own. As for uploading content, that is outside of the scope of an indexer. The user hosts their data with some hosting service, or independently. I believe signatures are already handled by XHTML which allows you to add XML signatures to documents, but regular HTML is sadly lacking here thought it would not be too hard to extend (or to just use the XML serialisation of HTML5).
I think it is generally bad to implement new features like this on the part of the indexer. It should just keep track of an existing web of documents rather than creating it's own format and walled garden.
> You would repost a page by linking to it from your own.
Ah right, of course. Fair enough. I think Dave Winer is trying some similar ideas, though I haven't looked at it beyond knowing he's trying to extend RSS to support these capabilities.
I'm sorry, but this blog post is written as if the described functionality should obviously be available and supported by the Internet and companies who expose their data.
I agree it would be neat.
But, when did we conclude this "should" be possible and thus warrant such a whiny post?
Not a would be cool, is cool. At least now that the atproto network has proved it is in fact cool.