I run a few VMs in a “homelab”. Was interested in redundancy so could work on system without taking down home networking… Familiar with GlusterFS, corosync/pacemaker, etc…
So to go from one system to 2, need a third system to at least ensure quorum.
Of course, storage has the same problem.
Active-active has its complexities. Maybe active-standby good enough with DRND. Still a lot to configure…
Filesystem replication now means my NVMe drives will be limited to 1Gbit/s networking link. 10GB somewhat difficult on SFF PC and expensive.
Physical location also matters. Standby probably should be somewhere else… But what about 3rd node? Wired networking not everywhere…
This ends up being a lot of moving parts, expense, and headache…
Guess what is fast? Single node. Move the storage drive to similar spare SFF PC. Two screws, takes about 3 minutes max. Local RAID-1 to guard against sudden failure… Backup a couple times a year for peace of mind.
corosync and pacemaker were my personal dark ages of distributed systems. With a well setup kubernetes clusters my faith is restored. Still not perfect but not a butt-crack sweating endeavor
“My formative memory of Python was when the Quake Live team used it for the back end work, and we wound up having serious performance problems with a few million users. My bias is that a lot (not all!) of complex “scalable” systems can be done with a simple, single C++ server.”
Games like quake are transient and have a small number of users sharing data and if it fails it doesn’t matter enough to need failover, although that could be done relatively easily with a write-ahead log. You need session stickyness at the load balancing layer but otherwise latency is more important than scale, so you don’t need a distributed system per se. Writing a small C/C++ server would make a lot of sense if you didn’t have Go, Rust, etc.
A distributed system is slower than a laptop or phone by definition of “distributed” which entails a network which has a higher latency than your local system. It’s not about raw speed/latency as much as it is about volume.
I used to compare distributed system with freight trains vs F1 cars. But there are a ton of low latency distributed systems too like your LTE network.
The key factor is how much data is shared by how many endpoints. Having these two very high makes it for very hard problems. Like streaming high definition events to millions of people.
Other than batch jobs, I can't think of a problem that can be solved these days that doesn't also require high availability - at the very least they require a warm standby.
I live in the ERP-adjacent world, and the vast majority don't have any of that, never have, and I pray most don't.
P.D: the business world is the one that needs AND tries everything, and most attempt to implement something is much worse than the simpler previous thing.
For most that I know and most I bet (even without knowing) add "high availability" or "warm standby" WILL CERTAINLY lead to worse availability.
In fact, the best setup for most people, and consider things from today (that you will see, not that different from mainframe days!):
- Single server on a *nix LTS/immutable distro with just the RDBMS + App backend, on decent hardware. Fast, lots of ram or whatever optional
- A true reliable backup system
- A decent network setup, ideally a VPN one for connect everybody, with good firewall, SSL and that is
More complex than this and is 90% certain is much worse, other 8% until the dude that do it and care leaves.
I dunno, it often matters a lot less than you think when something goes down. And distributed systems have a knack for going down in a much less predictable way, it's not going to automatically make your system more reliable.
(modern server hardware and operating systems are also surprisingly reliable nowadays, which makes it harder to reach breakeven with a distributed design)
So take 16 independent computers and have them each solve the problem separately. You'd still be saving 80% compared to the paper's benchmarks. It wasn't close.
McSherry does a lot of interesting work on making monotonic/incremental distributed systems efficient (e.g. Differential and Timely Dataflow). Those kinds of systems scale much more linearly.
> No CFO approves a $1.4 million annual commitment on the strength of a chart showing that the commitment scales.
Except the company probably approved a budget for AWS or another cloud provider, and basically gave a blank check to developers to deploy whatever is needed. So developers are going to just deploy MSK or whatever is trendy, instead of trying to get the most throughput from the servers they got from IT.
In my experience, this is the path many developers take. I’ve consulted to hundreds of companies over the years, and it’s alarming how often I’ve encountered decisions driven by what someone wanted to learn or work with, rather than what was actually right for the problem.
That happens at every level, from individual developers through to project leads and CTOs.
Consultancies are often no better. Choosing technologies that require substantial or highly specialised skill sets seems almost routine. I’m looking at you, Kubernetes.
I’m not entirely innocent here either. I owe a decent portion of my mortgage to MuleSoft consulting. That said, I don’t think I ever pretended it was always the best solution. Even while working directly for MuleSoft, my recommendation in probably half of the engagements was some variation of: ‘You’re using the wrong technology for this.’
But by then, an executive had usually tied their reputation to the project and the platform, commitments had been made, and changing course had become politically harder than continuing.
And so we persist.
In my experience, the best technology choices are boring ones. There’s still a large area of immature technology you can get creative with (like Backstage or Port for software catalogs and setting up a nice golden path”), but the meat and potatoes of development work should be a boring choice, that follows a well-tread path within a large ecosystem of developers.
There are exceptions, but they’re not for the majority of organisations.
Even for reliability, distributed gets hairy.
I run a few VMs in a “homelab”. Was interested in redundancy so could work on system without taking down home networking… Familiar with GlusterFS, corosync/pacemaker, etc…
So to go from one system to 2, need a third system to at least ensure quorum.
Of course, storage has the same problem.
Active-active has its complexities. Maybe active-standby good enough with DRND. Still a lot to configure…
Filesystem replication now means my NVMe drives will be limited to 1Gbit/s networking link. 10GB somewhat difficult on SFF PC and expensive.
Physical location also matters. Standby probably should be somewhere else… But what about 3rd node? Wired networking not everywhere…
This ends up being a lot of moving parts, expense, and headache…
Guess what is fast? Single node. Move the storage drive to similar spare SFF PC. Two screws, takes about 3 minutes max. Local RAID-1 to guard against sudden failure… Backup a couple times a year for peace of mind.
It’s way too easy to overthink stuff…
corosync and pacemaker were my personal dark ages of distributed systems. With a well setup kubernetes clusters my faith is restored. Still not perfect but not a butt-crack sweating endeavor
“My formative memory of Python was when the Quake Live team used it for the back end work, and we wound up having serious performance problems with a few million users. My bias is that a lot (not all!) of complex “scalable” systems can be done with a simple, single C++ server.”
https://x.com/ID_AA_Carmack/status/1210997702152069120
Games like quake are transient and have a small number of users sharing data and if it fails it doesn’t matter enough to need failover, although that could be done relatively easily with a write-ahead log. You need session stickyness at the load balancing layer but otherwise latency is more important than scale, so you don’t need a distributed system per se. Writing a small C/C++ server would make a lot of sense if you didn’t have Go, Rust, etc.
A distributed system is slower than a laptop or phone by definition of “distributed” which entails a network which has a higher latency than your local system. It’s not about raw speed/latency as much as it is about volume.
I used to compare distributed system with freight trains vs F1 cars. But there are a ton of low latency distributed systems too like your LTE network.
The key factor is how much data is shared by how many endpoints. Having these two very high makes it for very hard problems. Like streaming high definition events to millions of people.
Other than batch jobs, I can't think of a problem that can be solved these days that doesn't also require high availability - at the very least they require a warm standby.
I live in the ERP-adjacent world, and the vast majority don't have any of that, never have, and I pray most don't.
P.D: the business world is the one that needs AND tries everything, and most attempt to implement something is much worse than the simpler previous thing.
For most that I know and most I bet (even without knowing) add "high availability" or "warm standby" WILL CERTAINLY lead to worse availability.
In fact, the best setup for most people, and consider things from today (that you will see, not that different from mainframe days!):
- Single server on a *nix LTS/immutable distro with just the RDBMS + App backend, on decent hardware. Fast, lots of ram or whatever optional
- A true reliable backup system
- A decent network setup, ideally a VPN one for connect everybody, with good firewall, SSL and that is
More complex than this and is 90% certain is much worse, other 8% until the dude that do it and care leaves.
I dunno, it often matters a lot less than you think when something goes down. And distributed systems have a knack for going down in a much less predictable way, it's not going to automatically make your system more reliable.
(modern server hardware and operating systems are also surprisingly reliable nowadays, which makes it harder to reach breakeven with a distributed design)
So take 16 independent computers and have them each solve the problem separately. You'd still be saving 80% compared to the paper's benchmarks. It wasn't close.
McSherry does a lot of interesting work on making monotonic/incremental distributed systems efficient (e.g. Differential and Timely Dataflow). Those kinds of systems scale much more linearly.
Sometimes your entire business is just a laptop with a python dictionary and a backup power supply
> No CFO approves a $1.4 million annual commitment on the strength of a chart showing that the commitment scales.
Except the company probably approved a budget for AWS or another cloud provider, and basically gave a blank check to developers to deploy whatever is needed. So developers are going to just deploy MSK or whatever is trendy, instead of trying to get the most throughput from the servers they got from IT.
In my experience, this is the path many developers take. I’ve consulted to hundreds of companies over the years, and it’s alarming how often I’ve encountered decisions driven by what someone wanted to learn or work with, rather than what was actually right for the problem.
That happens at every level, from individual developers through to project leads and CTOs.
Consultancies are often no better. Choosing technologies that require substantial or highly specialised skill sets seems almost routine. I’m looking at you, Kubernetes.
I’m not entirely innocent here either. I owe a decent portion of my mortgage to MuleSoft consulting. That said, I don’t think I ever pretended it was always the best solution. Even while working directly for MuleSoft, my recommendation in probably half of the engagements was some variation of: ‘You’re using the wrong technology for this.’
But by then, an executive had usually tied their reputation to the project and the platform, commitments had been made, and changing course had become politically harder than continuing.
And so we persist.
In my experience, the best technology choices are boring ones. There’s still a large area of immature technology you can get creative with (like Backstage or Port for software catalogs and setting up a nice golden path”), but the meat and potatoes of development work should be a boring choice, that follows a well-tread path within a large ecosystem of developers.
There are exceptions, but they’re not for the majority of organisations.
[dead]
the point the article makes is good (albeit not new). the style sounds very LLM to me.