Cloud is insanely expensive and inefficient. Too many engineers don't understand how slow I/O back and forth is on network storage, or VMs with chatty neighbors are compared to bare metal...
RAM is insanely expensive and inefficient. Try to keep important info in L3 cache. Every additional hop from the CPU transistors adds exponential overhead.
Everywhere we look we see people making the best choice for them. Big apps using Electron vs native. Ports of 250MB games like Starcraft now taking 5GB of "compatibility" libraries to run on modern PCs.
I completely agree that people need to be aware of Latency Numbers Every Programmer Should Know (https://gist.github.com/jboner/2841832), but in reality we don't all have the budget, time, or personal to do things the right way when it's the company ordering us around vs our own projects where we can do things "the right way".
Honestly, I'm glad companies are wasteful. If large companies were super conservative and agile then our economy would be in really, really bad shape. It's thanks to these inefficiencies that small startups can jump in and capitalize in ways the many bureaucracies of enterprise can't.
Then again, the dev in me would LOVE it if everyone rewrote everything in Rust or Zig and we put wasteful scripting languages to death for good.
The math has drastically changed over the last few years. For example, the on-prem database server we bought 4 years ago for $14K is now pricing out at $96K.
> Cloud providers have to buy new hardware too to grow and to replace failing/obsolete units.
They can negotiate better prices based on their scale, and even on their ability to internally source key components if needed.
E.g. if AWS/Microsoft/Google/Apple buy 80% of Micron's entire output it's probably at a much better price than the rest of us competing over the other 20% are going to get.
> They can negotiate better prices based on their scale, and even on their ability to internally source key components if needed.
A global supply chain storage is a global supply chain shortage.
Most famous recent example being Apple who were absorbing the rapidly increasing cost of supply until even they had to put up the prices to reflect procurement and manufacturing reality.
And this is Apple. The company that even buys its manufacturers plant equipment in return for it being dedicated to Apple.
Yes the big boys get volume discount. But that doesn't magically entitle them to last decades's pricing.
> Yes the big boys get volume discount. But that doesn't magically entitle them to last decades's pricing.
You’re trying to refute an argument I did not make.
They do not have to be cheaper than 2016. They have to be cheaper than what your business could otherwise string together paying current market prices. And they will be.
It’s probably cheaper per unit when they buy at massive scale, but that was also true before the hardware inflation of the past couple of years. So I’m sure they are still paying much higher prices than in previous years. Many cloud services have increased prices and I suspect that will continue for at least a few more years.
Even just going to second tier cloud providers will tend to let you rent similar capacity much cheaper.
AWS and the other big cloud providers only gets close to cost efficient when you're big enough to negotiate prices that are nowhere near their public list prices.
Four years ago, I needed 256gb of ram, and it was costing about $1-2k a month in the cloud, which I replaced with a server on ebay with 1,5tb of ram for about €2k- which I was pricing against equivalent to what I was renting for about $6k or so on newegg. And I think the dumbest IT procurement manager in the planet could've gotten it from Lenovo or HPE or something for $10k providing the easiest 20% savings if they move on-prem.
Do you want to spend thousands of dollar upfront for that DB server with sweet sweet NVME SSDs, of which your application might not even use 1% before the app get scraped?
I would have argued against you a couple of years ago, because I've also built stuff this way and it required a developer or two to care at least a little bit about database administration, and few of the one's I've met do.
However, these days, I notice my DBA skills atrophying because LLMs have encoded most of the answers and diagnostics I need in their weights. Coupled with something like Nix, spinning up new servers including backups and everything, it's easier easier than ever to run your own production database instance.
It's easy doing a lot of stuff you don't understand quickly these days. But when the shit hits the fan and you have customers you need to know what you're doing.
Hiding behind the LLM, or even a cloud service like RDS makes you think you can get out of a corner, but you are nowhere near that. Practice, drills, experience are the only solution here. The illusion of that is dangerous.
I spent a couple of days recently digging one of our associated businesses out of the shit because they killed a postgres on RDS instance (AWS bug!) and couldn't google or LLM their way out of it. It hit the trade press and did a lot of damage to them.
If you're not going to do the work, don't take on customers.
> when the shit hits the fan and you have customers you need to know what you're doing. [...] don't take on customers.
I think you overestimate the level of DBA skills smaller startups have in house.
If you have found a way for a company to hold off on taking on customers until they can afford a proper DBA, I'd love to hear more about it. But it sounds backwards to me. In my experience, it typically goes the other way around: only after taking on enough customers to be decently profitable can a company afford a real DBA.
Until then, the choice used to be between managed database services or striking luck with a developer that had some database interest. These days, it's between managed database services, striking luck, or giving the developers access to an LLM. The LLM doesn't replace DBA skills that used to be there. There weren't any DBA skills there to begin with.
I haven't seen a small company hire for a dedicated DBA in over 15 years. It was unnecessary then, it's unnecessary now. As "agile" and "devops" took over, many of the more specialized roles disappeared.
No it's not. If "bad software delivered faster" is a problem, it's a problem long before LLMs. But the history of software-based products has shown that "delivered faster" is consistently more important, as bad software that attains significant usage will be made into good software one way or another.
I'm not skeptical of the idea that LLMs will be useful tools to cost-effetively develop high-quality software. I am, however, skeptical that the input to these LLMs will remain mostly informal and conversational.
Sure, so long as you have processes in place to verify the result.
It would really suck to learn that your LLM-generated disaster recovery plan isn't viable when attempting to use it to recover from an actual disaster.
This isn't a whole lot different than what you should be doing to ensure systems not managed by LLMs are designed and operating properly, of course, which I guess is sort of my point.
You need that RAID card which is not available anywhere around you.
You need to visit your Data Center, but you suddenly have issues with security and you need to arrange all that stuff.
You need to buy DDR stick but you need to wait a month for delivery.
The guy who knows how to insert a stick took a vacation week ago, and everyone else is too afraid to touch big iron box.
With cloud, you either don't notice that, because they have scale and they have spare parts and they have personnel 24/7. Or worst case you order identical server and deploy your app, hopefully in a few hours restoring from backup.
And for some people it might never broke. But it's all the uncertainty and fear that leads people to cloud solutions. You don't plan for average outcome, you plan for worst outcome.
That exact line of thinking is how our costs increased 8x over 2 years.
And you're not immune from those issues in AWS. What happens when you run out of EBS IOPS or have to change instance type and it won't boot? What happens when your instance fails? What happens if you hit a quota limit in the middle of a failover?
AWS support is fucking awful unless you pay $5k a month or 10% of your total spend (whatever is greater) and then it's only mediocre.
> You need that RAID card which is not available anywhere around you.
If you're small enough to not be able to justify spares, then rent servers instead. It's still cheaper than the big cloud providers.
> You need to visit your Data Center, but you suddenly have issues with security and you need to arrange all that stuff.
You rarely do. Anywhere beyond the cheapest third-rate colo provider provides "remote hands" services.
Even when managing several racks worth of hardware, I'd go into the colo facilities a couple of times a year at most.
> and they have spare parts and they have personnel 24/7
Even tiny colo-providers still have personnel 24/7, and you can buy fractional devops services for small retainers from any number of providers (like me).
Yes, you will need to keep a spare running. And have written (and tested!) procedures around how to switch to the spare when members of your team are away.
All additional concerns and requirements that you need money, time, and people for. I'm not claiming a server-as-a-service / cloud has zero skill requirements, but it's a lot less than managing your own hardware.
I can't comment on total costs / costs over time though.
In my experience cloud deployments also need documentation and processes. It is of course a different set of documentation and processes, but cloud's not magically free of these requirements.
Single boxes now are also insanely powerful compared to what we had back in the day too. Unless you are genuinely running the next Google, you might not even need to scale beyond one. As you said you might get a second box purely as a hot spare.
I remember we ran a pretty large UK website on 5 x Pentium II boxes, using a Perl stack which is not exactly memory efficient. Later we ran a chat server for hundreds of thousands of school kids on a single server which was (if I remember this right) a Pentium III with 128 MB (yes, megabytes) of memory and a spinning hard disk.
I used to run a webmail service that we had ~2 million user accounts on. It took several racks of equipment. The combined capacity (RAM, IO capacity, disk storage, CPU) was all below the 250 GBP mini PC under my desk...
I built my own server for home last year, all consumer hardware (AMD Ryzen, 96GiB RAM, NVME SSD, HDDs etc). Total cost was ~£3,000 at the time - £6000 up front cost for a business seems reasonable; and you could get away with less.
However, the main cost would be colocation - it's risen quite a bit.
Nothing wrong with Ryzen for a server, you can get AM5 server boards from Asrock and they support ECC. If you need more they make AM5 Epyc CPUs that are just Ryzen parts with Epyc written on the box and IHS
AMD will sell you an Epyc AM4/AM5 CPU with the same specs at the same MSRP for most of their Zen4/Zen5 chips (not the APUs and I think there may be a couple missing skus on the Epyc side).
I doubt there's much of a meaningful difference other than a jumper for the brand. Retail pricing and availability might differ.
if you do the same type of governance, cloud is still cheap too (everything on one server)
Where the money gets wasted, is doing a different server per project, putting everything in different resource groups, and then overprovisioning those servers many times over. So much wasted capacity because of simple project structure preferences.
And then there is also the fact some features are locked behind more expensive models, which is also rarely cost efficient.
All I am saying is, if you simply create 1 database server on the cloud, with a database per app, and different schema's, isolated users, ... it can stay super cheap. But good look getting through an enterprise cybersecurity audit or ISO27000+SOC2 certification.
You can rent bare metal server for that. The real advantage of cloud solution is a cohesive offer, ease of scalability, and many other things which boils down to "we take care of things for you". It can be worth the $, i can also be a huge waste of $.
Far lower than 0.01x. Batching can get like a 30x boost, and horizontal scaling makes that impossible, so your upper limit is basically 0.03x. Then you throw it on a volume with 1/1000 the IOPS of a basic consumer drive. So more like 0.00001x, and you can pay through the nose to bring it up to 0.01x maybe.
People have zero concept of how incredibly absurdly fast even a slow computer is. Storage is literally 1000x faster than it was 15 years ago. My first job was firmware for enterprise storage arrays and IIRC we were aiming for 3M IOPS. That's not much more than what kids can put in their PS5 now, and these were arrays running companies like Walmart and JPMC.
You also fail to realize that even managing a physical DB server is much easier now with AI, enabling people to more easily understand what their infrastructure needs are, what maintenance is needed, how things can be automated.
True. We do have to remember that this is only possible in the last few months. The costs to manage more and more infrastructure is steadily going down.
Renting (either just the hardware or a dedicated box at a hosting provider), leasing, lease to own, or just plain regular payment plans are all options that tend to work out at a fraction of the big cloud providers.
In 30 years of hosting on-prem or in colo facilities, I've never once bought hardware up front for corporate accounts.
And yet we do multi year purchase commitment contracts with cloud providers that break that "flexibility" premise.
With the multi million dollars A MONTH purchase commitment contracts I've seen in my previous employers, you could use a single month budget to buy hardware that could serve our applications for years.
Why are you paying for hardware that might not even be used ? If you get an unexpected burst of demand that's not the worst problem to have. You can even jump on a cloud for a few months.
Being set up to be able to quickly "jump on a cloud" is the best way of ensuring it's even less cost efficient for you to actually use the cloud most of the time.
Which is to say, it's good advice.
It lets you load your dedicated servers far higher than what you can safely do if you don't have the option of using a cloud provider for burst traffic, but in practice most people will very rarely need it.
The issue is that on prem suffers from too many edge failure cases. It's cheaper to pay S3 to deal with the problem than to have a full team dealing with bit rot from RAID issues. It's one thing to have an home office/home lab server, it's another thing entirely when you have terabytes of content that can't tolerate any data loss and you don't have engineers on call at 3am.
I am no fan of the idea of reinventing everything to run on cloud primitives but you can't deny reliability of online software is a lot higher now than the pre HTML5 era. Back then, having more than two nines of uptime for random backend heavy web apps was hardly common while now it's table stakes.
No to mention, the great thing about the cloud is that you can get away with availability problems because it usually means everyone else is down, too.
The very thought of a single storage request hitting the network unnecessarily boils my blood.
Some latency increase must be traded for scalability. But at least we get the required utility in return.
But without that utility, one is just hiking up the cost. Any competitor who can deliver at reduced cost holds a potent weapon that you simply cannot survive.
A company that has efficient cost structure across its operational domains holds an advantage which is near impossible to out-compete. A sense of inevitability and resilience is what reduction in cost buys you.
Cost is a weapon. One of the most fundamental weapons in almost all domains.
Latency can also be traded for availability. EBS storage provides replication and can trivially be removed/attached to new instances if the database server for some reason needs to be taken down. Any database with built-in replication/consensus (Kafka, Spanner, Dynamo, Cockroach, etc.) is making this same tradeoff and is effectively hitting the network on every storage request.
> Any database with built-in replication/consensus (Kafka, Spanner, Dynamo, Cockroach, etc.) is making this same tradeoff and is effectively hitting the network on every storage request.
I don't think this is true. I am not sure about those you described, but say you have PG replicas, you can run query on replica node without hitting some network attached storage. This is obviously true for reads but not for writes.
Yeah, you are correct. I was thinking about writes but not reads. Even the databases I mentioned typically have optimizations such that when there is a stable leader, reads don't require any network.
Considering that the topic is an article is comparing raw NVMe vs EBS performance for postgres storage and GP saying that cloud is unnecessarily inefficient, I misinterpreted your comment as implying that networked storage is always an unnecessary cost. In many cases the cost of replication and availability is lower than the cost of data loss and downtime.
Depends on your metrics. Say you're an early startup - short on man-hours and bare-metal skills, with backers who think that "cloud" sounds fast, cool, and scalable.
Or you're a VP at some big company - with C-suite mandate to deliver Cool Solutions ASAP, whatever the cost...but a byzantine mess of middle managers between you and the actual engineers. Deciding "we'll use CloudCo" can rob your steaming heap of PHB's of a lot of opportunities to make bad decisions and drag things out.
SQLite + NVMe feels like cheating sometimes. If my #1 requirement is lowest possible latency, I can't imagine picking anything else.
Kestrel (AspNetCore) + SQLite + NVMe all on the same machine is where things start to get really spicy in my experience. Imagine handling an entire form submission that mutates database state in tens to low-hundreds of microseconds. Total wall clock time required to service any HTTP request can be well under a single millisecond. If your average cpu time per request is 30uS, you could approach half a million per second on a 16-32 core machine.
Intel P5800X user, checking in. The latency improvement when I switched from a Samsung 990 to the P5800X wasn't quite as mind-blowing or revelatory as the HDD to (SATA) SSD switch was ~16 years ago (OCZ Vertex LE -- $300 for 100 GB!) but compared to standard nvme it still feels like a dramatic improvement. It's startling to realize how much performance we're leaving on the table even with nvme ssds. Build times for what I work on dropped by much more than 50%, if I remember correctly, when I switched and everything on my system feels more responsive.
Edit: optane also shows what the SSD/nvme sequential-throughput marketing wars have cost us. Latency and random 4k iops always should have been the focus (in research, development, reviews, and benchmarks), not sequential throughput.
There's rumors floating around that Chinese firms will soon be bringing new PCM storage solutions to the datacenter market. Not nearly as good as 3dXPoint/Optane, but should nonetheless be lower latency than Flash and much better endurance (DWPD).
If the pricing is competitive it will easily be replacing DRAM for many lower-level uses, thanks to its support for better PCIe standards (5.0 at least, while aftermarket Optane is stuck on 4.0).
How good is your Mandarin Chinese? You can easily find recent press release from Demingli and more general commentary about the state of the market inckuding mentions of other, lesser known firms.
Same experience here, from a multiplayer game server. Go, one SQLite file per game world, one process owns each file, all on a local SSD.
In a load test with 512 players all active on one world, command acks were 4-6 ms at the median and under 50 ms at p95, on a single Ryzen 9 3900X box. And most of that time is game logic and fanning out updates over websockets, not the database.
The part people worry about is durability, because SSDs have limited write cycles. So I took care to have backups of the state, recovery in short time etc.
I stream the WAL to object storage with Litestream, with one gotcha: I had to turn off SQLite’s own auto-checkpoint and let only Litestream checkpoint, or the two step on each other.
We tested this out extensively. DuckDB + Nvme kicks ass, but if you ingesting and analytzing near real-time quotes/trades, DuckDB doesn't handle well because DuckDB can only support one reader and one writer in read-write mode.
Ok so you have multiple writers in write-only mode and then you switch to read-mode if you need to read? How would that work e.g in an IoT ingest system with lots of telemetry coming in.
It doesn't. That's the issue. DuckDB for us is a pure read-only service pipe. We need other tools to ingest and write to parquet, which has its own limitation.
SQLite does a great job to support single writer and multiple readers. That's what we use for aggregation right now. DuckDB is purely post analytics.
CloudNativePG supports Kubernetes Local Storage as one of its main features [0]
Persistent volume management, including:
Support for Local Persistent Volumes with PVC templates.
Reuse of Persistent Volumes storage in Pods.
Separate volumes for WAL files and tablespaces.
Chapter Temporary PostgreSQL Cluster Degradation [1] describes how to set it up and starts with this
> While the standard approach ensures service reliability and leverages Kubernetes' self-healing capabilities, there are scenarios where operating with a temporarily degraded cluster may be acceptable. This is particularly relevant for PostgreSQL clusters relying on node-local storage, where the storage is local to the Kubernetes worker node running the PostgreSQL database. Node-local storage, or simply local storage, is employed to enhance performance.
Recently moved our RDS instances from r8g. to r8gd., which adds local NVMe, and I think the average I/O latency went from 1.0ms to 0.1ms. Adds up in big queries!
"Back in the day", we used to have to carefully plan the underlying storage array to eek out every ounce of performance from the database server.
I think there's a generation of devs that just takes big memory pools and SSDs for granted; there's a mental disconnect to the fact databases map the underlying storage.
It becomes obvious once I bring it up with them in discussion and I can almost see the bulb go off (happened while discussing partitioning and sharding), but they never make the connection that a query is mapping to some artifacts sitting on storage.
Had a recent conversation with someone about something similar…
In our case the primary index on a table was a something like a sequential ID but the table stored data from multiple tenants so each page had data from multiple tenants
Took a while to convince people that the DB was always going to read 16KB chunks so we could reduce IO by changing the primary index so it was based around an ID that was unique to the tenant
> Took a while to convince people that the DB was always going to read 16KB chunks so we could reduce IO by changing the primary index so it was based around an ID that was unique to the tenant
Yikes. I'm mostly frontend and even to me this seems pretty clear.
Honestly the network-attached storage only ever made sense with HDDs, it's ridiculous to have to pay 10x penalty for both iops and latency just for some "operational flexibility". You get way simpler and much more robust architecture if you just stick to local NVMe and replication -- not that you can forgo one in the (also replicated) network block storage solution anyway...
SSD NAS still a lot faster than HDD NAS. And having network-attached storage allows for a lot of cool things. Like you can bind it to kubernetes, so your database instance can easily migrate between nodes while data is kept intact just remounting between nodes. In fact it makes things a lot easier whenever you need to manage persistence. But you're paying big price for it, yeah. Sometimes simplicity worth it. Sometimes you can't afford to lose performance.
NAS/SAN still make sense when you need to share storage between multiple DBs for cost efficiency: most application DBs are tiny compared to the size of an SSD.
Mostly agree, but then you have to take care of replication, failover, and backup. By using a replicated block storage, you can mostly externalize that problem.
I can't be the only one that always has to set up a separate backup system in addition to the RDS automatic backups, because it uses a weird format and is slow.
Agreed. This is probably the trickiest part of building managed Postgres on local NVMe. Postgres already gives you a lot of the building blocks: physical streaming replication, synchronous replication, WAL archiving/PITR with backup toold, etc. The hard part is making failovers, recovery, and node replacements seamless.
We’ve spent a fair bit of time on this at ClickHouse through our operational expertise huilding Postgres managed services multiple times. We’re fortunate to have deep internal and partner Postgres expertise, folks who originally built WAL-G back in the Citus Data days and are active contributors now and solid partnership with Ubicloud.
It’s definitely more engineering work, but the performance/scalability gains make it worth it. Relaying this based on 100s of customers who migrated to ClickHouse Managed Postgres - new, powerful world surpassing previous expectations.
This is the fundamental issue with networked storage: all apps on top assume small io is fast an local, so network latency doesn't amortize.
With application-level replication, you can amortize it to a few steps (e.g. on commit, do 2PC).
Obviously there's nuance, esp if you have to read from another node, but it's why moving replication as far up the stack as possible generally performs better with DBs -- the DB knows when to touch the network better than the disk.
I was using postgres for embarrassingly long time before I've learned the difference between "index scans" and "index only scans".
Including relevant columns in the index can massively improve the performance, because postgres does not need to read the randomly sorted tuple in the original table.
Just be wary that if your app is a high-write-throughput or high-update-frequency system, this is not a free lunch!
The indices will bloat in size (postgres does not vacuum them in the way a table is vacuumed), and begin to bottleneck primary writer latency.
You should repack the indices every once in a while (you can set up monitors on index bloat) with something like pg_repack, and make sure that your indices are minimally-sized (i.e. omit rows that are never scanned, use pg_stat to verify which indices are actually used by the primary or read-replicas).
io2 has lower latency than gp3, and better durability. We switched workloads off gp3, didn’t even need to provision crazy iops. It had a cascading effect of calming down.
> But no database that ensures ACID can avoid that one.
fsync itself doesn't guarantee your data will survive, so it is not essential.
Other ideas to try: you have N replicas, and you ask them to do in-memory commit. M server simultanious death is probably more unlikely compared to single server storage system death.
...but now you have a distributed systems problem to deal with instead!
Yes, there are various levers you can pull and it's rare to find a solution that will be optimal for every use case. But, in general, sticking with the most mainstream solution is usually better than going off-piste unless you are very certain that it doesn't cover your niche.
Is that entirely true? If you have a replication protocol like Viewstamped Replication you have the option to make a trade off for how many machines you the will fail concurrently vs forcing an fsync.
I think I'm a bit of an outlier here on that k never really understood why durability requires fsync. It doesn't help you if the storage dies. If durable storage is important to you, then isn't the only solution to synchronously replicate to a quorum of machines at which point an fsync isn't that relevant? What am I missing?
You don't want other parts of the system thinking that the data is committed to durable storage when it isn't.
"Storage dies" is a broad set of causes. If there is power loss or a server crash due to a multitude of reasons, you want committed transactions to actually be committed.
Having replicas doesn't change the problem. If the orchestrator requires at least N replicas to be done committing the data, you'll have to wait for fsync in N of them.
Without fsync, the data anywhere could be sitting on a write back cache in memory instead of being flushed to the disk.
But then again, only the transaction log has to be physically on the drive to guarantee durability, and modern drives are very good at sequential writes. Writing tables/indexes can be delayed (to a "checkpoint", as it is known in most database systems).
I recently interviewed for a company (for the purpose of not making this an advertisement, I will not name it) that uses FPGAs to speed up SQL queries. They claimed at least two orders of magnitude speedups for joins!
They also told me that they've started as a h/w shop and that they have competitors who are building ASICs rather than programming FPGAs to do the same (with, potentially, even more performance gains).
This completely changed the perspective for me. Before, I had this idea that if I need to optimize database performance, I'd first look at indices. Then, at how much memory is available to the database for various tasks. Only after that, I'd look at I/O performance, but, mostly, because of fsync configuration.
It wouldn't even appear to me as a possibility that the compute can become the culprit of optimization. But, apparently, it's a new hot thing.
* * *
From my experience with storage testing, which also included quite a bit of pg-bench exposure... My impression was this: the sort of query that is used for performance benchmark matters the most. It will define the optimization direction. Blindly trying to add or speed-up one resource or another is not going to be an efficient strategy. In the database world, there's this idea of an "ideal" benchmark. Without too much detail, it imagines an "ideal" business accounting back-end, the sort of queries it would have to execute. It defines an "ideal" unit that roughly encompasses one "sale", and then it measures how many such "sales" a business can perform per unit of time. This will translate into some ratio of selects / inserts / deletes / updates with some index hits and some linear searches etc. But it's all extremely vague and it's very hard to adapt this benchmark to the actual business needs of the database being tested.
There's also a question what to optimize for: the resource usage? the response speed? the concurrency? In some scenarios, I can see how storage would factor in, but there are also certainly scenarios where using a faster, in some sense, storage will not result in any meaningful gains.
There was a whole bunch of work on GPUs for databases before GPUs became hard to get, which slowed it down a bit. A lot of the columnar database performance gain is from vectorization. Now which direct pcie DMA to GPU from NVMe this is even more attractive. FPGA seems much more work than GPU.
My understanding is that FPGAs are targeted not because they are ideal for the task, but because they are commercially available. As in: the customers may rent a VM from a major cloud vendor with FPGAs attached.
Trying to grow a business that relies on bespoke hardware in the age of cloud monopoly is an uphill battle.
Yeah, I remember seeing those. It felt really promising. I have no idea why they didn't continue in this direction.
The company I talked to explained the use of FPGA by saying that FPGAs are offered by major cloud vendor, and so are a commercially viable direction. While making their own dedicated chips is not only a lot more expensive, but also doesn't have a lot of customers (people don't want to maintain their own h/w, especially of a bespoke variety). The bespoke part might be why Samsung didn't pursue the idea. Essentially, if eg. AWS doesn't buy this from you and makes it an offering that customers may reasonably rely on, it's too hard to sell.
Was my question too. Now, I have not tested the technology, so, I can't vouch for the veracity of the claim, but the answer I was given made sense. This is what it was: in the same way how a dedicated switch is a lot more efficient when dealing with IP packets than a general-purpose CPU is. In particular, comparison takes a lot less time because it is able to dramatically reduce the number of load operations. I found this paper: https://www.sfu.ca/~zhenman/files/C31-FCCM2023-SQL2FPGA.pdf (there's also a Github repo with some experimental code for Spark: https://github.com/SFU-HiAccel/SQL2FPGA) that does something similar.
But why was EBS chosen instead of NVMe, which is faster? Since I only know how to use cloud programming and don’t understand the internal implementation, I don’t really get why it works out this way.
Is it because of ease of backup? Or data movement?
When the host dies with local storage, the VM is dead. With EBS it's not, since it simply can be booted up on another host. EBS is also replicated accross multiple hosts.
Thank you for your kind answer. I understand EBS as, in other words, virtualized storage, so then somewhere the logical block addresses have to be connected to actual physical storage devices, right? Then where and how is that mapping implemented? If you just give me the keywords, I’ll search for them myself.
Since I’m a freelance programmer, I mainly work on small-to-medium-sized projects, so I don’t know much about this kind of infrastructure programming.
Look up “SAN” or “storage area network.” EBS looks like a regular, local block device, another disk in /dev, but is actually served over the network. The implementation is transparent.
> Look up “SAN” or “storage area network.” EBS looks like a regular, local block device, another disk in /dev, but is actually served over the network.
When I think of "SAN" I think of Fibre Channel, which can be quite fast. EBS sounds more like iSCSI.
The ability to scale up the DB disk size and select the right performance level... Especially when starting out, you may have no idea how much space or performance you need. EBS volumes help make that decision easier and more cost efficient.
I'm an incorrigible contrarian so this seems like a good opportunity to shill for Oracle, where I currently have a part time gig doing research in the database group and am thus highly financially conflicted; read what follows with that in mind.
Well written databases are frequently I/O bottlenecked and getting a good storage/network setup can thus grant massive performance gains. But this is hard, operationally. The difficulty of developing the right opinions about hardware design, acquiring it, properly wiring it up, tuning it and ensuring really excellent performance across the stack was recognized by Oracle decades ago, which led to the creation of ExaData.
ExaData is a bit like Oxide, if you've encountered them. It's basically a cloud database in a pre-manufactured rack. You buy an ExaData rack and it turns up at your on-prem colo facility, you plug it in, and now you have a fast database. Internally it has a dedicated high speed InfiniBand or RoCE RDMA capable backplane network and handles replicated between storage devices itself. There's an alerting system and repl-like tool that guides you through swapping out failed storage devices, you can expand it by adding more racks, and there are various cloud hosted versions of all of this if your budget doesn't stretch to your own full system.
The software stack on top of that provides fast multi-master synchronous writes by using RAM as a shared disk cache. If a node needs to read a block off disk it will prefer to read it directly out of the RAM of another node using RDMA across the dedicated backplane than go to storage, so if you buy the high capacity variant then you have a three tier storage system: HDDs for cold data, flash as a cache and DRAM as another cache on top of that, all kept in sync using cache coherency protocols.
There are lots of optimizations on top of that which a Postgres-on-EBS architecture can't match, for example, the storage nodes that handle the disks and serve blocks can have some of the query computations pushed down to them to reduce IO across the backplane and accelerate query execution. A final 'optimization' is that Oracle uses a different MVCC algorithm that doesn't require vacuuming and can be more IO efficient in general (doesn't copy whole rows); there are patch sets that switch Postgres to an Oracle-style MVCC engine but none of them were ever merged.
The whole thing acts as a single coherent database, so e.g. rolling upgrades across the rack's cluster don't cause downtime because transactions can be handed off across nodes. Clients can discover the IP addresses of the individual servers automatically and shift their traffic across nodes in response to load balancing requests from the system, etc.
I'm not sure it's a good idea to run a Postgres cluster like the one described in the article unless you have a lot of appetite for manual operations. If you can tolerate downtime or if you're renting bare metal where you can configure a proper RAID setup, then OK, but if you care a lot about query performance to begin with then a commercial database will win in a lot of ways (doesn't have to be Oracle, commercial RDBMS in general is a much more competitive space than open source).
Obligatory disclaimer: opinions above are not corporate approved.
I think that cloud made people forget how fast computers actually are. Hardware specs got reduced to "vCPUs" count, which isn't a useful abstraction if you are doing anything beyond basic stuff. Similarly, network attached storage became the default, and we started treating its latency as Postgres’s latency. These abstractions made choices seem simpler, and in an era of cheap hardware that was fine, since you could just throw more resources at the problem.
Too many times I see people making the same avoidable mistakes time and time again.
They get stuck in the dream of "portable" schemas and don't use their brain to design a database-specific schema.
They have no clue how to write SQL queries (or again, have stupid dreams of "portable" queries). As a result performance suffers.
Then they come to your desk complaining the databse is slow, and insisting that they absolutely MUST have a Redis caching layer.
I'm sick and tired of people using databases as blackbox dumping grounds. If slow storage was always the problem, my life would be far less stressful. Since the prescription would be easier for the doctor to write. ;)
You can spend a week crafting the most optimal query possible, making sure the plans are perfect and all the best index types are used
And then you push it to production and get completely different results
Statistics change everything. Queries which produced beautiful plans in local and staging environments breakdown when given production statistics, some users get the same plans you did, others get nasty nested loops, indexes don't get used, and debugging and fixing it requires poking around in prod statistics which is never a good place to be. Don't even get me started on how RLS impacts query optimisation
Postgres is great but the authors refusal to add plan hints is a decision that comes to bite me at least a few times a year
Mostly agree although you should probably spend a bit more time testing on a more realistic setup.
At least run some plans on the real database with real sized data and real statistics.
Similar to how mocks/stubs are great until they aren't.
That won't magic away statistics problems, but it's still good practice to acknowledge the lower app environments tend to not reflect performance of the production one.
Yeah, a lot of people can't figure out how to structure databases or what indices to add to them, but no, that is not the only problem just because it's the one you deal with.
> that is not the only problem just because it's the one you deal with
Sure. But to be clear: I am not denying storage is a genuine problem for some people.
I am just saying in the majority of cases it's probably not "the storage". For most people "the storage" will be quite far down the list of likely causes of a "slow" database.
Throw all the Enterprise SSDs or other storage solutions you like at a shitty schema and it will still struggle under load (or sometimes just struggle full stop).
Optimizing column layout has been enough for me to leave OLAP workloads in Postgres a few times when I otherwise would've had to spin up something dedicated for them.
Sit down, think about the queries you are using, and think about how the tables are updated. Now take that and rethink your indexes. Use the right index type for the right situation, don’t just slap a basic index on a serial id field and call it a day. And while you’re at it, think about the ID fields you use, sometimes better options exist. UUIDv7 is now built in, use it.
Lots of things in PGSQL benefit from thinking about how your use case is manipulating the underlying B-tree, if you work in performance sensitive stuff the devil is really in the details. And having to rebuild indexes that have decayed over time is painful.
> Which postgres specific schema/query features do you use to significantly improve performance?
Erm. Isn't this kind of a schema specific question ?
I mean, sometimes I see schemas that are so goddam awful that the answer to your question would be "anything, literally anything".
Other times its the old story of failing to use or misconfiguring indexes.
And then, yes you end up in the realms of Postgres specific stuff. What you do there clearly depends on what you're looking at.
But the point is, I could probably count on one hand the number of times the problem has been "the storage" or "the database". Most of the time its "the developer". :D
The parent post emphasizes how portable schemas or queries cost performance. Now I understand important parts of the schema are postgres specific (e.g. the configuration of non-trivial indexes), but I think most of my queries happen to be portable. So I wondered where people sacrifice performance in the name of portability, and why that's so common that it's worth calling out.
Postgres (and similar DBs) are a byproduct of a pre-2023 era. Give me one reason why I shouldn't get my agents to write a use-case specific DB that is almost certainly going to be faster than a general-purpose DB?
Imagine not reinventing the wheel for no reason.
But don't worry, I imagine that modern prompting techniques will be used such as "do not make mistakes" so everything will be fine (/s)
Opportunity cost. If getting rid of Postgres is going to make you more money, go right ahead. More likely though, you have other problems to be dealing with that are more directly in line with your work.
There's a balance to be had between a highly specialised custom tool that does exactly what you need now, versus a generalist tool that does what you do today and more but is less optimised for your specific use-case today. That, and standing on the shoulders of giants, knowing why those fences are there, and so on.
try it and see :) check out that Goose language. you’ll quickly realize your agents are idiots and need reams of PDFs that YOU don’t know how to select and filter down to even have good ideas, and then will drift forever failing to “seam” the contracts between the storage, query planning, and compilation layers.
1. You don’t have those speed requirements, and if you do, you already know the answer to your question.
2. Data tends to be somewhat important to a lot of companies. A vibe-coded database increases your risk of losing that data.
3. Your database might have clients using different languages. Now, your vibe-coded drivers become another layer of potential problems.
4. People has been doing this since way before 2023 btw, most probably agree Postgres would be the sane choice.
You just don't write some class of software unless that is the product itself. DBs are one of them. Another one is operating systems. It takes a huge level of domain knowledge, real life use and time.
I feel its hard to prove that either way, but even the content of the article itself is full of LLM generated text, for e.g. here is one line from the conclusion "The bottom line is deliberately narrower than the title".
Yes. My experience: "Oh interesting title". "Oh god... slop galore. Disappointment, sadness, dejection, a feeling of being dismissed and not taken seriously, alienation from the world", "OK close the link".
Happy to still be here among my fellow humans. Hackernews the last bastion it feels like.
Can we somehow filter out slop from the front page? That would be great :).
some kinda system running the linked page through pangram and then putting the result in a small font next to the submission time would be perfect I think. A small font to keep the idea that such things may potentially hold value while satisfying those who aren't interested. Nobody has to argue about it themselves then because it's unambiguous, and everyone can filter them out if you choose.
I'm generally very pro-AI, including for writing, as long as the "producer" (I wouldn't want to call the human responsible the author) has vetted it and will judge based on whether something reads well rather than on whether it feels AI-written.
But using detectors to label so people can choose seems entirely reasonable to me even with a risk of false positives, and I think a lot of people who are generally pro-AI would feel that's fine.
Cloud is insanely expensive and inefficient. Too many engineers don't understand how slow I/O back and forth is on network storage, or VMs with chatty neighbors are compared to bare metal...
RAM is insanely expensive and inefficient. Try to keep important info in L3 cache. Every additional hop from the CPU transistors adds exponential overhead.
Everywhere we look we see people making the best choice for them. Big apps using Electron vs native. Ports of 250MB games like Starcraft now taking 5GB of "compatibility" libraries to run on modern PCs.
I completely agree that people need to be aware of Latency Numbers Every Programmer Should Know (https://gist.github.com/jboner/2841832), but in reality we don't all have the budget, time, or personal to do things the right way when it's the company ordering us around vs our own projects where we can do things "the right way".
Honestly, I'm glad companies are wasteful. If large companies were super conservative and agile then our economy would be in really, really bad shape. It's thanks to these inefficiencies that small startups can jump in and capitalize in ways the many bureaucracies of enterprise can't.
Then again, the dev in me would LOVE it if everyone rewrote everything in Rust or Zig and we put wasteful scripting languages to death for good.
>Cloud is insanely expensive
The math has drastically changed over the last few years. For example, the on-prem database server we bought 4 years ago for $14K is now pricing out at $96K.
Cloud providers have to buy new hardware too to grow and to replace failing/obsolete units. They will have to raise their prices sooner or later too.
> Cloud providers have to buy new hardware too to grow and to replace failing/obsolete units.
They can negotiate better prices based on their scale, and even on their ability to internally source key components if needed.
E.g. if AWS/Microsoft/Google/Apple buy 80% of Micron's entire output it's probably at a much better price than the rest of us competing over the other 20% are going to get.
> They can negotiate better prices based on their scale, and even on their ability to internally source key components if needed.
A global supply chain storage is a global supply chain shortage.
Most famous recent example being Apple who were absorbing the rapidly increasing cost of supply until even they had to put up the prices to reflect procurement and manufacturing reality.
And this is Apple. The company that even buys its manufacturers plant equipment in return for it being dedicated to Apple.
Yes the big boys get volume discount. But that doesn't magically entitle them to last decades's pricing.
> Yes the big boys get volume discount. But that doesn't magically entitle them to last decades's pricing.
You’re trying to refute an argument I did not make.
They do not have to be cheaper than 2016. They have to be cheaper than what your business could otherwise string together paying current market prices. And they will be.
It’s probably cheaper per unit when they buy at massive scale, but that was also true before the hardware inflation of the past couple of years. So I’m sure they are still paying much higher prices than in previous years. Many cloud services have increased prices and I suspect that will continue for at least a few more years.
And yet they remain an expensive option.
Even just going to second tier cloud providers will tend to let you rent similar capacity much cheaper.
AWS and the other big cloud providers only gets close to cost efficient when you're big enough to negotiate prices that are nowhere near their public list prices.
> only gets close to cost efficient when you're big enough to negotiate prices that are nowhere near their public list prices
And by which time you are spending an ungodly amount of money per month !
Which is why second tier providers are the way to go.
You get better prices. Much clearer price models. Better customer service....
... And all without having to commit to spending more than the GDP of a small country every month on your cloud bill.
AWS will happily sell you instances based on 12 year old Haswell.
O_o
Four years ago, I needed 256gb of ram, and it was costing about $1-2k a month in the cloud, which I replaced with a server on ebay with 1,5tb of ram for about €2k- which I was pricing against equivalent to what I was renting for about $6k or so on newegg. And I think the dumbest IT procurement manager in the planet could've gotten it from Lenovo or HPE or something for $10k providing the easiest 20% savings if they move on-prem.
How much is your $96k server on AWS?
It's all about trade off.
Do you want to spend thousands of dollar upfront for that DB server with sweet sweet NVME SSDs, of which your application might not even use 1% before the app get scraped?
Back in the old days we chucked it all on one box. Then scaled up when we needed to.
If you can’t afford to throw down enough money on a single box to bring a business up then you’re fucked anyway.
I know whole businesses that make a lot of money and never needed more than that box. (And a hot spare)
I would have argued against you a couple of years ago, because I've also built stuff this way and it required a developer or two to care at least a little bit about database administration, and few of the one's I've met do.
However, these days, I notice my DBA skills atrophying because LLMs have encoded most of the answers and diagnostics I need in their weights. Coupled with something like Nix, spinning up new servers including backups and everything, it's easier easier than ever to run your own production database instance.
It's easy doing a lot of stuff you don't understand quickly these days. But when the shit hits the fan and you have customers you need to know what you're doing.
Hiding behind the LLM, or even a cloud service like RDS makes you think you can get out of a corner, but you are nowhere near that. Practice, drills, experience are the only solution here. The illusion of that is dangerous.
I spent a couple of days recently digging one of our associated businesses out of the shit because they killed a postgres on RDS instance (AWS bug!) and couldn't google or LLM their way out of it. It hit the trade press and did a lot of damage to them.
If you're not going to do the work, don't take on customers.
> when the shit hits the fan and you have customers you need to know what you're doing. [...] don't take on customers.
I think you overestimate the level of DBA skills smaller startups have in house.
If you have found a way for a company to hold off on taking on customers until they can afford a proper DBA, I'd love to hear more about it. But it sounds backwards to me. In my experience, it typically goes the other way around: only after taking on enough customers to be decently profitable can a company afford a real DBA.
Until then, the choice used to be between managed database services or striking luck with a developer that had some database interest. These days, it's between managed database services, striking luck, or giving the developers access to an LLM. The LLM doesn't replace DBA skills that used to be there. There weren't any DBA skills there to begin with.
I haven't seen a small company hire for a dedicated DBA in over 15 years. It was unnecessary then, it's unnecessary now. As "agile" and "devops" took over, many of the more specialized roles disappeared.
This is incidentally exactly why I never buy any products unless they've been around 10 years. It's a risk.
An LLM can absolutely do the harder stuff too. I wonder what model you have used where you think it's only for the easy stuff.
Of course you need understanding for when shit hits the fan, but that applies regardless of whether it's a human or an LLM writing the code.
We've had the best people using the best and most expensive models. They confidently produce shit they thought they understood but didn't.
LLMs remove personal exploration from tasks and that's where you really get to know things.
We've all seen best people confidently produce shit throughout the history, before the LLMs.
Now they can do it faster than ever and to the detriment of everyone. Which is the problem.
When you optimise for fast and not correct, things are fucked.
Then they've been fucked since C beat Lisp, and that hasn't been enough to cause Silicon Valley to implode.
That is a completely different argument.
No it's not. If "bad software delivered faster" is a problem, it's a problem long before LLMs. But the history of software-based products has shown that "delivered faster" is consistently more important, as bad software that attains significant usage will be made into good software one way or another.
I'm not skeptical of the idea that LLMs will be useful tools to cost-effetively develop high-quality software. I am, however, skeptical that the input to these LLMs will remain mostly informal and conversational.
Don't LLMs make it easier to write all the low level/boring stuff that makes managing a database deployment directly viable without the middleman?
Sure, so long as you have processes in place to verify the result.
It would really suck to learn that your LLM-generated disaster recovery plan isn't viable when attempting to use it to recover from an actual disaster.
This isn't a whole lot different than what you should be doing to ensure systems not managed by LLMs are designed and operating properly, of course, which I guess is sort of my point.
[dead]
It's all good while it works.
The day it broke, it's a PITA.
You need that RAID card which is not available anywhere around you.
You need to visit your Data Center, but you suddenly have issues with security and you need to arrange all that stuff.
You need to buy DDR stick but you need to wait a month for delivery.
The guy who knows how to insert a stick took a vacation week ago, and everyone else is too afraid to touch big iron box.
With cloud, you either don't notice that, because they have scale and they have spare parts and they have personnel 24/7. Or worst case you order identical server and deploy your app, hopefully in a few hours restoring from backup.
And for some people it might never broke. But it's all the uncertainty and fear that leads people to cloud solutions. You don't plan for average outcome, you plan for worst outcome.
That exact line of thinking is how our costs increased 8x over 2 years.
And you're not immune from those issues in AWS. What happens when you run out of EBS IOPS or have to change instance type and it won't boot? What happens when your instance fails? What happens if you hit a quota limit in the middle of a failover?
AWS support is fucking awful unless you pay $5k a month or 10% of your total spend (whatever is greater) and then it's only mediocre.
> You need that RAID card which is not available anywhere around you.
If you're small enough to not be able to justify spares, then rent servers instead. It's still cheaper than the big cloud providers.
> You need to visit your Data Center, but you suddenly have issues with security and you need to arrange all that stuff.
You rarely do. Anywhere beyond the cheapest third-rate colo provider provides "remote hands" services.
Even when managing several racks worth of hardware, I'd go into the colo facilities a couple of times a year at most.
> and they have spare parts and they have personnel 24/7
Even tiny colo-providers still have personnel 24/7, and you can buy fractional devops services for small retainers from any number of providers (like me).
Yes, you will need to keep a spare running. And have written (and tested!) procedures around how to switch to the spare when members of your team are away.
All additional concerns and requirements that you need money, time, and people for. I'm not claiming a server-as-a-service / cloud has zero skill requirements, but it's a lot less than managing your own hardware.
I can't comment on total costs / costs over time though.
In my experience cloud deployments also need documentation and processes. It is of course a different set of documentation and processes, but cloud's not magically free of these requirements.
I do devops work for people, and clients using cloud services consistently have more devops issues than those on dedicated hardware.
Single boxes now are also insanely powerful compared to what we had back in the day too. Unless you are genuinely running the next Google, you might not even need to scale beyond one. As you said you might get a second box purely as a hot spare.
I remember we ran a pretty large UK website on 5 x Pentium II boxes, using a Perl stack which is not exactly memory efficient. Later we ran a chat server for hundreds of thousands of school kids on a single server which was (if I remember this right) a Pentium III with 128 MB (yes, megabytes) of memory and a spinning hard disk.
I used to run a webmail service that we had ~2 million user accounts on. It took several racks of equipment. The combined capacity (RAM, IO capacity, disk storage, CPU) was all below the 250 GBP mini PC under my desk...
[flagged]
I built my own server for home last year, all consumer hardware (AMD Ryzen, 96GiB RAM, NVME SSD, HDDs etc). Total cost was ~£3,000 at the time - £6000 up front cost for a business seems reasonable; and you could get away with less.
However, the main cost would be colocation - it's risen quite a bit.
Just buy one off Hetzner or something.
I wouldn't use Ryzen on a production server myself.
Nothing wrong with Ryzen for a server, you can get AM5 server boards from Asrock and they support ECC. If you need more they make AM5 Epyc CPUs that are just Ryzen parts with Epyc written on the box and IHS
AMD will sell you an Epyc AM4/AM5 CPU with the same specs at the same MSRP for most of their Zen4/Zen5 chips (not the APUs and I think there may be a couple missing skus on the Epyc side).
I doubt there's much of a meaningful difference other than a jumper for the brand. Retail pricing and availability might differ.
Just out of curiosity why not?
No ECC support on current models, dubious ECC support on previous models, thermal performance issues, crap chipsets, not enough PCIe lanes.
Uh, its only the models with builtin GPU (the G series) that lack ECC. The rest seems to support it as long as the motherboard does.
if you do the same type of governance, cloud is still cheap too (everything on one server)
Where the money gets wasted, is doing a different server per project, putting everything in different resource groups, and then overprovisioning those servers many times over. So much wasted capacity because of simple project structure preferences.
And then there is also the fact some features are locked behind more expensive models, which is also rarely cost efficient.
All I am saying is, if you simply create 1 database server on the cloud, with a database per app, and different schema's, isolated users, ... it can stay super cheap. But good look getting through an enterprise cybersecurity audit or ISO27000+SOC2 certification.
I've led and completed SOC2 and 27001 cert on own hardware.
It's absolutely fine. Everyone has just gone stupid recently.
You can rent bare metal server for that. The real advantage of cloud solution is a cohesive offer, ease of scalability, and many other things which boils down to "we take care of things for you". It can be worth the $, i can also be a huge waste of $.
Ease of scalability - "it starts at 0.01x, but adding extra increments of 0.01x is easy, provided your app is designed for horizontal scalability"
Far lower than 0.01x. Batching can get like a 30x boost, and horizontal scaling makes that impossible, so your upper limit is basically 0.03x. Then you throw it on a volume with 1/1000 the IOPS of a basic consumer drive. So more like 0.00001x, and you can pay through the nose to bring it up to 0.01x maybe.
People have zero concept of how incredibly absurdly fast even a slow computer is. Storage is literally 1000x faster than it was 15 years ago. My first job was firmware for enterprise storage arrays and IIRC we were aiming for 3M IOPS. That's not much more than what kids can put in their PS5 now, and these were arrays running companies like Walmart and JPMC.
You also fail to realize that even managing a physical DB server is much easier now with AI, enabling people to more easily understand what their infrastructure needs are, what maintenance is needed, how things can be automated.
True. We do have to remember that this is only possible in the last few months. The costs to manage more and more infrastructure is steadily going down.
No, you can just rent a VPS with local NVMe storage if even renting bare metal is too much.
Renting (either just the hardware or a dedicated box at a hosting provider), leasing, lease to own, or just plain regular payment plans are all options that tend to work out at a fraction of the big cloud providers.
In 30 years of hosting on-prem or in colo facilities, I've never once bought hardware up front for corporate accounts.
So like, 0.5 extra salary months to make sure the whole company's app is fast?
Sounds cheaper than AWS :)
And yet we do multi year purchase commitment contracts with cloud providers that break that "flexibility" premise.
With the multi million dollars A MONTH purchase commitment contracts I've seen in my previous employers, you could use a single month budget to buy hardware that could serve our applications for years.
Why are you paying for hardware that might not even be used ? If you get an unexpected burst of demand that's not the worst problem to have. You can even jump on a cloud for a few months.
Being set up to be able to quickly "jump on a cloud" is the best way of ensuring it's even less cost efficient for you to actually use the cloud most of the time.
Which is to say, it's good advice.
It lets you load your dedicated servers far higher than what you can safely do if you don't have the option of using a cloud provider for burst traffic, but in practice most people will very rarely need it.
scraped -> scrapped
totally different words
And then you get people having to spend more to work around its limitations e.g. what Discord did here: https://discord.com/blog/how-discord-supercharges-network-di...
The issue is that on prem suffers from too many edge failure cases. It's cheaper to pay S3 to deal with the problem than to have a full team dealing with bit rot from RAID issues. It's one thing to have an home office/home lab server, it's another thing entirely when you have terabytes of content that can't tolerate any data loss and you don't have engineers on call at 3am.
I am no fan of the idea of reinventing everything to run on cloud primitives but you can't deny reliability of online software is a lot higher now than the pre HTML5 era. Back then, having more than two nines of uptime for random backend heavy web apps was hardly common while now it's table stakes.
No to mention, the great thing about the cloud is that you can get away with availability problems because it usually means everyone else is down, too.
The very thought of a single storage request hitting the network unnecessarily boils my blood.
Some latency increase must be traded for scalability. But at least we get the required utility in return.
But without that utility, one is just hiking up the cost. Any competitor who can deliver at reduced cost holds a potent weapon that you simply cannot survive.
A company that has efficient cost structure across its operational domains holds an advantage which is near impossible to out-compete. A sense of inevitability and resilience is what reduction in cost buys you.
Cost is a weapon. One of the most fundamental weapons in almost all domains.
Latency can also be traded for availability. EBS storage provides replication and can trivially be removed/attached to new instances if the database server for some reason needs to be taken down. Any database with built-in replication/consensus (Kafka, Spanner, Dynamo, Cockroach, etc.) is making this same tradeoff and is effectively hitting the network on every storage request.
> Any database with built-in replication/consensus (Kafka, Spanner, Dynamo, Cockroach, etc.) is making this same tradeoff and is effectively hitting the network on every storage request.
I don't think this is true. I am not sure about those you described, but say you have PG replicas, you can run query on replica node without hitting some network attached storage. This is obviously true for reads but not for writes.
Yeah, you are correct. I was thinking about writes but not reads. Even the databases I mentioned typically have optimizations such that when there is a stable leader, reads don't require any network.
"Unnecessary" is the keyword you missed. Refer to sentences 1, 2 and 3 in my response.
The argument of cost is fundamental and no amount of Kafka, etc. can override it.
Gotcha, seems like we agree then.
Considering that the topic is an article is comparing raw NVMe vs EBS performance for postgres storage and GP saying that cloud is unnecessarily inefficient, I misinterpreted your comment as implying that networked storage is always an unnecessary cost. In many cases the cost of replication and availability is lower than the cost of data loss and downtime.
Even during replication, any unnecessary network payload beyond the absolute minimum required is still a cost that boils my blood :-).
> Cloud is insanely expensive and inefficient.
Depends on your metrics. Say you're an early startup - short on man-hours and bare-metal skills, with backers who think that "cloud" sounds fast, cool, and scalable.
Or you're a VP at some big company - with C-suite mandate to deliver Cool Solutions ASAP, whatever the cost...but a byzantine mess of middle managers between you and the actual engineers. Deciding "we'll use CloudCo" can rob your steaming heap of PHB's of a lot of opportunities to make bad decisions and drag things out.
> Too many engineers don't understand how (...) VMs with chatty neighbors are compared to bare metal
sell a lie, blame the victim?
I can have an app which scales to zero and costs me nothing, but could instantly handle everyone using it. That's not possible on-prem.
Could instantly handle everyone using it, at the sweet sweet price of the money you saved when it scaled to zero and a little bit more.
For the low low price of unlimited liability.
No thanks.
SQLite + NVMe feels like cheating sometimes. If my #1 requirement is lowest possible latency, I can't imagine picking anything else.
Kestrel (AspNetCore) + SQLite + NVMe all on the same machine is where things start to get really spicy in my experience. Imagine handling an entire form submission that mutates database state in tens to low-hundreds of microseconds. Total wall clock time required to service any HTTP request can be well under a single millisecond. If your average cpu time per request is 30uS, you could approach half a million per second on a 16-32 core machine.
>If my #1 requirement is lowest possible latency, I can't imagine picking anything else.
optane
Intel P5800X user, checking in. The latency improvement when I switched from a Samsung 990 to the P5800X wasn't quite as mind-blowing or revelatory as the HDD to (SATA) SSD switch was ~16 years ago (OCZ Vertex LE -- $300 for 100 GB!) but compared to standard nvme it still feels like a dramatic improvement. It's startling to realize how much performance we're leaving on the table even with nvme ssds. Build times for what I work on dropped by much more than 50%, if I remember correctly, when I switched and everything on my system feels more responsive.
Edit: optane also shows what the SSD/nvme sequential-throughput marketing wars have cost us. Latency and random 4k iops always should have been the focus (in research, development, reviews, and benchmarks), not sequential throughput.
There's rumors floating around that Chinese firms will soon be bringing new PCM storage solutions to the datacenter market. Not nearly as good as 3dXPoint/Optane, but should nonetheless be lower latency than Flash and much better endurance (DWPD).
If the pricing is competitive it will easily be replacing DRAM for many lower-level uses, thanks to its support for better PCIe standards (5.0 at least, while aftermarket Optane is stuck on 4.0).
Oh that sounds interesting!
I love toying with optane stuff but yeah it's starting to show it's age and the gen4 gear is both old-ish and still expensive.
So a new toy would be cool
Where can I read more about these rumors?
How good is your Mandarin Chinese? You can easily find recent press release from Demingli and more general commentary about the state of the market inckuding mentions of other, lesser known firms.
Same experience here, from a multiplayer game server. Go, one SQLite file per game world, one process owns each file, all on a local SSD.
In a load test with 512 players all active on one world, command acks were 4-6 ms at the median and under 50 ms at p95, on a single Ryzen 9 3900X box. And most of that time is game logic and fanning out updates over websockets, not the database.
The part people worry about is durability, because SSDs have limited write cycles. So I took care to have backups of the state, recovery in short time etc.
I stream the WAL to object storage with Litestream, with one gotcha: I had to turn off SQLite’s own auto-checkpoint and let only Litestream checkpoint, or the two step on each other.
What approach do you take for backups?
Windows server storage replica in async mode.
What happens when a new version is deployed?
Have you checked out DuckDB for time series data?
We tested this out extensively. DuckDB + Nvme kicks ass, but if you ingesting and analytzing near real-time quotes/trades, DuckDB doesn't handle well because DuckDB can only support one reader and one writer in read-write mode.
Ok so you have multiple writers in write-only mode and then you switch to read-mode if you need to read? How would that work e.g in an IoT ingest system with lots of telemetry coming in.
It doesn't. That's the issue. DuckDB for us is a pure read-only service pipe. We need other tools to ingest and write to parquet, which has its own limitation.
SQLite does a great job to support single writer and multiple readers. That's what we use for aggregation right now. DuckDB is purely post analytics.
Gotcha. Makes sense
NVMe + LMDB. :-)
CloudNativePG supports Kubernetes Local Storage as one of its main features [0]
Chapter Temporary PostgreSQL Cluster Degradation [1] describes how to set it up and starts with this> While the standard approach ensures service reliability and leverages Kubernetes' self-healing capabilities, there are scenarios where operating with a temporarily degraded cluster may be acceptable. This is particularly relevant for PostgreSQL clusters relying on node-local storage, where the storage is local to the Kubernetes worker node running the PostgreSQL database. Node-local storage, or simply local storage, is employed to enhance performance.
[0] https://cloudnative-pg.io/docs/devel/#main-features [1] https://cloudnative-pg.io/docs/devel/kubernetes_upgrade/#tem...
Recently moved our RDS instances from r8g. to r8gd., which adds local NVMe, and I think the average I/O latency went from 1.0ms to 0.1ms. Adds up in big queries!
"Back in the day", we used to have to carefully plan the underlying storage array to eek out every ounce of performance from the database server.
I think there's a generation of devs that just takes big memory pools and SSDs for granted; there's a mental disconnect to the fact databases map the underlying storage.
It becomes obvious once I bring it up with them in discussion and I can almost see the bulb go off (happened while discussing partitioning and sharding), but they never make the connection that a query is mapping to some artifacts sitting on storage.
Part of the problem here is that large parts of the cloud seems to be stuck in 2017 when it comes to IO speeds and latency.
Had a recent conversation with someone about something similar…
In our case the primary index on a table was a something like a sequential ID but the table stored data from multiple tenants so each page had data from multiple tenants
Took a while to convince people that the DB was always going to read 16KB chunks so we could reduce IO by changing the primary index so it was based around an ID that was unique to the tenant
> Took a while to convince people that the DB was always going to read 16KB chunks so we could reduce IO by changing the primary index so it was based around an ID that was unique to the tenant
Yikes. I'm mostly frontend and even to me this seems pretty clear.
IME lots of people don't ever consider how data is stored on disk
They might be used to Postgres, which doesn't have primary index.
Honestly the network-attached storage only ever made sense with HDDs, it's ridiculous to have to pay 10x penalty for both iops and latency just for some "operational flexibility". You get way simpler and much more robust architecture if you just stick to local NVMe and replication -- not that you can forgo one in the (also replicated) network block storage solution anyway...
SSD NAS still a lot faster than HDD NAS. And having network-attached storage allows for a lot of cool things. Like you can bind it to kubernetes, so your database instance can easily migrate between nodes while data is kept intact just remounting between nodes. In fact it makes things a lot easier whenever you need to manage persistence. But you're paying big price for it, yeah. Sometimes simplicity worth it. Sometimes you can't afford to lose performance.
NAS/SAN still make sense when you need to share storage between multiple DBs for cost efficiency: most application DBs are tiny compared to the size of an SSD.
Mostly agree, but then you have to take care of replication, failover, and backup. By using a replicated block storage, you can mostly externalize that problem.
I can't be the only one that always has to set up a separate backup system in addition to the RDS automatic backups, because it uses a weird format and is slow.
And because those backups lifecycle is usually tied to the instance itself. Someone deletes your instance and your backups are gone as well.
It’s kinda crazy that this is the case, and then they make it sooo convoluted to copy those backups elsewhere. Typical cloud.
That's scary! And wrong...
Agreed. This is probably the trickiest part of building managed Postgres on local NVMe. Postgres already gives you a lot of the building blocks: physical streaming replication, synchronous replication, WAL archiving/PITR with backup toold, etc. The hard part is making failovers, recovery, and node replacements seamless.
We’ve spent a fair bit of time on this at ClickHouse through our operational expertise huilding Postgres managed services multiple times. We’re fortunate to have deep internal and partner Postgres expertise, folks who originally built WAL-G back in the Citus Data days and are active contributors now and solid partnership with Ubicloud.
It’s definitely more engineering work, but the performance/scalability gains make it worth it. Relaying this based on 100s of customers who migrated to ClickHouse Managed Postgres - new, powerful world surpassing previous expectations.
This is the fundamental issue with networked storage: all apps on top assume small io is fast an local, so network latency doesn't amortize.
With application-level replication, you can amortize it to a few steps (e.g. on commit, do 2PC).
Obviously there's nuance, esp if you have to read from another node, but it's why moving replication as far up the stack as possible generally performs better with DBs -- the DB knows when to touch the network better than the disk.
Been working with postgres, never had issues regarding performance. Postgres is the only db i wannna work with for non-web apps.
I was using postgres for embarrassingly long time before I've learned the difference between "index scans" and "index only scans".
Including relevant columns in the index can massively improve the performance, because postgres does not need to read the randomly sorted tuple in the original table.
Just be wary that if your app is a high-write-throughput or high-update-frequency system, this is not a free lunch!
The indices will bloat in size (postgres does not vacuum them in the way a table is vacuumed), and begin to bottleneck primary writer latency.
You should repack the indices every once in a while (you can set up monitors on index bloat) with something like pg_repack, and make sure that your indices are minimally-sized (i.e. omit rows that are never scanned, use pg_stat to verify which indices are actually used by the primary or read-replicas).
Yep, this can be huge. I learned this in the MySQL days with "covering indexes."
I'd like clouds to offer NVMe with value-add features like atomic snapshots and asynchronous replication on top.
Stay tuned, you’ll hear something from us soon!
io2 has lower latency than gp3, and better durability. We switched workloads off gp3, didn’t even need to provision crazy iops. It had a cascading effect of calming down.
Postgres isn't slow, your data just isn't in the cache.
When writing, it needs to fsync to ensure ACID
But no database that ensures ACID can avoid that one.
If that's the problem, we should say "Postgres isn't slow, you're just on ACID".
> But no database that ensures ACID can avoid that one.
fsync itself doesn't guarantee your data will survive, so it is not essential.
Other ideas to try: you have N replicas, and you ask them to do in-memory commit. M server simultanious death is probably more unlikely compared to single server storage system death.
...but now you have a distributed systems problem to deal with instead!
Yes, there are various levers you can pull and it's rare to find a solution that will be optimal for every use case. But, in general, sticking with the most mainstream solution is usually better than going off-piste unless you are very certain that it doesn't cover your niche.
distributed sync IS a mainstream solution, all fault tolerant dbs have it
Is that entirely true? If you have a replication protocol like Viewstamped Replication you have the option to make a trade off for how many machines you the will fail concurrently vs forcing an fsync.
I think I'm a bit of an outlier here on that k never really understood why durability requires fsync. It doesn't help you if the storage dies. If durable storage is important to you, then isn't the only solution to synchronously replicate to a quorum of machines at which point an fsync isn't that relevant? What am I missing?
You don't want other parts of the system thinking that the data is committed to durable storage when it isn't.
"Storage dies" is a broad set of causes. If there is power loss or a server crash due to a multitude of reasons, you want committed transactions to actually be committed.
Having replicas doesn't change the problem. If the orchestrator requires at least N replicas to be done committing the data, you'll have to wait for fsync in N of them.
Without fsync, the data anywhere could be sitting on a write back cache in memory instead of being flushed to the disk.
Synchronous replication to a quorum is a lot more expensive than an fsync.
But then again, only the transaction log has to be physically on the drive to guarantee durability, and modern drives are very good at sequential writes. Writing tables/indexes can be delayed (to a "checkpoint", as it is known in most database systems).
For what its worth, some databases just run like trash. I've put a database on nvme raid and still had some queries take 5 seconds.
Either you aren’t using indexes or your queries must be doing a lot of JOINS
I recently interviewed for a company (for the purpose of not making this an advertisement, I will not name it) that uses FPGAs to speed up SQL queries. They claimed at least two orders of magnitude speedups for joins!
They also told me that they've started as a h/w shop and that they have competitors who are building ASICs rather than programming FPGAs to do the same (with, potentially, even more performance gains).
This completely changed the perspective for me. Before, I had this idea that if I need to optimize database performance, I'd first look at indices. Then, at how much memory is available to the database for various tasks. Only after that, I'd look at I/O performance, but, mostly, because of fsync configuration.
It wouldn't even appear to me as a possibility that the compute can become the culprit of optimization. But, apparently, it's a new hot thing.
* * *
From my experience with storage testing, which also included quite a bit of pg-bench exposure... My impression was this: the sort of query that is used for performance benchmark matters the most. It will define the optimization direction. Blindly trying to add or speed-up one resource or another is not going to be an efficient strategy. In the database world, there's this idea of an "ideal" benchmark. Without too much detail, it imagines an "ideal" business accounting back-end, the sort of queries it would have to execute. It defines an "ideal" unit that roughly encompasses one "sale", and then it measures how many such "sales" a business can perform per unit of time. This will translate into some ratio of selects / inserts / deletes / updates with some index hits and some linear searches etc. But it's all extremely vague and it's very hard to adapt this benchmark to the actual business needs of the database being tested.
There's also a question what to optimize for: the resource usage? the response speed? the concurrency? In some scenarios, I can see how storage would factor in, but there are also certainly scenarios where using a faster, in some sense, storage will not result in any meaningful gains.
There was a whole bunch of work on GPUs for databases before GPUs became hard to get, which slowed it down a bit. A lot of the columnar database performance gain is from vectorization. Now which direct pcie DMA to GPU from NVMe this is even more attractive. FPGA seems much more work than GPU.
My understanding is that FPGAs are targeted not because they are ideal for the task, but because they are commercially available. As in: the customers may rent a VM from a major cloud vendor with FPGAs attached.
Trying to grow a business that relies on bespoke hardware in the age of cloud monopoly is an uphill battle.
Fun fact: Samsung made some SSDs ("SmartSSD") with an embedded programmable FPGA and one of the demos was using it along a modified version of postgresql to offload filtering and scans to the storage itself: https://pages.cs.wisc.edu/~jignesh/publ/SmartSSD.pdf https://www.xilinx.com/publications/product-briefs/xilinx-sm...
Though it looks like they didn't follow up on these and can't find them for sale
Yeah, I remember seeing those. It felt really promising. I have no idea why they didn't continue in this direction.
The company I talked to explained the use of FPGA by saying that FPGAs are offered by major cloud vendor, and so are a commercially viable direction. While making their own dedicated chips is not only a lot more expensive, but also doesn't have a lot of customers (people don't want to maintain their own h/w, especially of a bespoke variety). The bespoke part might be why Samsung didn't pursue the idea. Essentially, if eg. AWS doesn't buy this from you and makes it an offering that customers may reasonably rely on, it's too hard to sell.
Why would an FPGA speed up joins by two orders of magnitude?
Was my question too. Now, I have not tested the technology, so, I can't vouch for the veracity of the claim, but the answer I was given made sense. This is what it was: in the same way how a dedicated switch is a lot more efficient when dealing with IP packets than a general-purpose CPU is. In particular, comparison takes a lot less time because it is able to dramatically reduce the number of load operations. I found this paper: https://www.sfu.ca/~zhenman/files/C31-FCCM2023-SQL2FPGA.pdf (there's also a Github repo with some experimental code for Spark: https://github.com/SFU-HiAccel/SQL2FPGA) that does something similar.
But why was EBS chosen instead of NVMe, which is faster? Since I only know how to use cloud programming and don’t understand the internal implementation, I don’t really get why it works out this way. Is it because of ease of backup? Or data movement?
When the host dies with local storage, the VM is dead. With EBS it's not, since it simply can be booted up on another host. EBS is also replicated accross multiple hosts.
Thank you for your kind answer. I understand EBS as, in other words, virtualized storage, so then somewhere the logical block addresses have to be connected to actual physical storage devices, right? Then where and how is that mapping implemented? If you just give me the keywords, I’ll search for them myself.
Since I’m a freelance programmer, I mainly work on small-to-medium-sized projects, so I don’t know much about this kind of infrastructure programming.
Look up “SAN” or “storage area network.” EBS looks like a regular, local block device, another disk in /dev, but is actually served over the network. The implementation is transparent.
> Look up “SAN” or “storage area network.” EBS looks like a regular, local block device, another disk in /dev, but is actually served over the network.
When I think of "SAN" I think of Fibre Channel, which can be quite fast. EBS sounds more like iSCSI.
I just think of anything that isn't a NAS?
Thank you for taking the time to let me know the keywords. I’m grateful for your kindness, and have a good day.
The ability to scale up the DB disk size and select the right performance level... Especially when starting out, you may have no idea how much space or performance you need. EBS volumes help make that decision easier and more cost efficient.
I'm an incorrigible contrarian so this seems like a good opportunity to shill for Oracle, where I currently have a part time gig doing research in the database group and am thus highly financially conflicted; read what follows with that in mind.
Well written databases are frequently I/O bottlenecked and getting a good storage/network setup can thus grant massive performance gains. But this is hard, operationally. The difficulty of developing the right opinions about hardware design, acquiring it, properly wiring it up, tuning it and ensuring really excellent performance across the stack was recognized by Oracle decades ago, which led to the creation of ExaData.
ExaData is a bit like Oxide, if you've encountered them. It's basically a cloud database in a pre-manufactured rack. You buy an ExaData rack and it turns up at your on-prem colo facility, you plug it in, and now you have a fast database. Internally it has a dedicated high speed InfiniBand or RoCE RDMA capable backplane network and handles replicated between storage devices itself. There's an alerting system and repl-like tool that guides you through swapping out failed storage devices, you can expand it by adding more racks, and there are various cloud hosted versions of all of this if your budget doesn't stretch to your own full system.
The software stack on top of that provides fast multi-master synchronous writes by using RAM as a shared disk cache. If a node needs to read a block off disk it will prefer to read it directly out of the RAM of another node using RDMA across the dedicated backplane than go to storage, so if you buy the high capacity variant then you have a three tier storage system: HDDs for cold data, flash as a cache and DRAM as another cache on top of that, all kept in sync using cache coherency protocols.
There are lots of optimizations on top of that which a Postgres-on-EBS architecture can't match, for example, the storage nodes that handle the disks and serve blocks can have some of the query computations pushed down to them to reduce IO across the backplane and accelerate query execution. A final 'optimization' is that Oracle uses a different MVCC algorithm that doesn't require vacuuming and can be more IO efficient in general (doesn't copy whole rows); there are patch sets that switch Postgres to an Oracle-style MVCC engine but none of them were ever merged.
The whole thing acts as a single coherent database, so e.g. rolling upgrades across the rack's cluster don't cause downtime because transactions can be handed off across nodes. Clients can discover the IP addresses of the individual servers automatically and shift their traffic across nodes in response to load balancing requests from the system, etc.
I'm not sure it's a good idea to run a Postgres cluster like the one described in the article unless you have a lot of appetite for manual operations. If you can tolerate downtime or if you're renting bare metal where you can configure a proper RAID setup, then OK, but if you care a lot about query performance to begin with then a commercial database will win in a lot of ways (doesn't have to be Oracle, commercial RDBMS in general is a much more competitive space than open source).
Obligatory disclaimer: opinions above are not corporate approved.
I think that cloud made people forget how fast computers actually are. Hardware specs got reduced to "vCPUs" count, which isn't a useful abstraction if you are doing anything beyond basic stuff. Similarly, network attached storage became the default, and we started treating its latency as Postgres’s latency. These abstractions made choices seem simpler, and in an era of cheap hardware that was fine, since you could just throw more resources at the problem.
[flagged]
[flagged]
[flagged]
[dead]
Sounds like blame shifting to me.
Postgres isn't slow, your schema and queries are.
There. Fixed it for you.
Too many times I see people making the same avoidable mistakes time and time again.
They get stuck in the dream of "portable" schemas and don't use their brain to design a database-specific schema.
They have no clue how to write SQL queries (or again, have stupid dreams of "portable" queries). As a result performance suffers.
Then they come to your desk complaining the databse is slow, and insisting that they absolutely MUST have a Redis caching layer.
I'm sick and tired of people using databases as blackbox dumping grounds. If slow storage was always the problem, my life would be far less stressful. Since the prescription would be easier for the doctor to write. ;)
You can spend a week crafting the most optimal query possible, making sure the plans are perfect and all the best index types are used
And then you push it to production and get completely different results
Statistics change everything. Queries which produced beautiful plans in local and staging environments breakdown when given production statistics, some users get the same plans you did, others get nasty nested loops, indexes don't get used, and debugging and fixing it requires poking around in prod statistics which is never a good place to be. Don't even get me started on how RLS impacts query optimisation
Postgres is great but the authors refusal to add plan hints is a decision that comes to bite me at least a few times a year
Mostly agree although you should probably spend a bit more time testing on a more realistic setup.
At least run some plans on the real database with real sized data and real statistics.
Similar to how mocks/stubs are great until they aren't.
That won't magic away statistics problems, but it's still good practice to acknowledge the lower app environments tend to not reflect performance of the production one.
Yeah, a lot of people can't figure out how to structure databases or what indices to add to them, but no, that is not the only problem just because it's the one you deal with.
> that is not the only problem just because it's the one you deal with
Sure. But to be clear: I am not denying storage is a genuine problem for some people.
I am just saying in the majority of cases it's probably not "the storage". For most people "the storage" will be quite far down the list of likely causes of a "slow" database.
Throw all the Enterprise SSDs or other storage solutions you like at a shitty schema and it will still struggle under load (or sometimes just struggle full stop).
Which postgres specific schema/query features do you use to significantly improve performance?
Column Tetris is one example: https://stackoverflow.com/questions/2966524/calculating-and-....
Optimizing column layout has been enough for me to leave OLAP workloads in Postgres a few times when I otherwise would've had to spin up something dedicated for them.
Sit down, think about the queries you are using, and think about how the tables are updated. Now take that and rethink your indexes. Use the right index type for the right situation, don’t just slap a basic index on a serial id field and call it a day. And while you’re at it, think about the ID fields you use, sometimes better options exist. UUIDv7 is now built in, use it.
Lots of things in PGSQL benefit from thinking about how your use case is manipulating the underlying B-tree, if you work in performance sensitive stuff the devil is really in the details. And having to rebuild indexes that have decayed over time is painful.
Having indexes fitting the query is of course important, but it's not postgres specific.
> Which postgres specific schema/query features do you use to significantly improve performance?
Erm. Isn't this kind of a schema specific question ?
I mean, sometimes I see schemas that are so goddam awful that the answer to your question would be "anything, literally anything".
Other times its the old story of failing to use or misconfiguring indexes.
And then, yes you end up in the realms of Postgres specific stuff. What you do there clearly depends on what you're looking at.
But the point is, I could probably count on one hand the number of times the problem has been "the storage" or "the database". Most of the time its "the developer". :D
The parent post emphasizes how portable schemas or queries cost performance. Now I understand important parts of the schema are postgres specific (e.g. the configuration of non-trivial indexes), but I think most of my queries happen to be portable. So I wondered where people sacrifice performance in the name of portability, and why that's so common that it's worth calling out.
PGHero an pgtuner are two of our favorite tools to use when optimizing PGSQL
A good schema and good queries won't save you if your storage is physically far enough away :P
Good storage won't save you if your schema and queries are bad enough, either.
Postgres (and similar DBs) are a byproduct of a pre-2023 era. Give me one reason why I shouldn't get my agents to write a use-case specific DB that is almost certainly going to be faster than a general-purpose DB?
Reliability. A general-purpose DBMS probably has multiple decades of battle-testing, your vibecoded DBMS doesn't.
Imagine not reinventing the wheel for no reason. But don't worry, I imagine that modern prompting techniques will be used such as "do not make mistakes" so everything will be fine (/s)
Opportunity cost. If getting rid of Postgres is going to make you more money, go right ahead. More likely though, you have other problems to be dealing with that are more directly in line with your work.
There's a balance to be had between a highly specialised custom tool that does exactly what you need now, versus a generalist tool that does what you do today and more but is less optimised for your specific use-case today. That, and standing on the shoulders of giants, knowing why those fences are there, and so on.
For the same reason I want my car to have FSD instead of my FSD to know how to assemble a car to drive by itself.
try it and see :) check out that Goose language. you’ll quickly realize your agents are idiots and need reams of PDFs that YOU don’t know how to select and filter down to even have good ideas, and then will drift forever failing to “seam” the contracts between the storage, query planning, and compilation layers.
Who is stopping you?
Why stop at DB? May as well just write a purpose built OS from scratch.
1. You don’t have those speed requirements, and if you do, you already know the answer to your question. 2. Data tends to be somewhat important to a lot of companies. A vibe-coded database increases your risk of losing that data. 3. Your database might have clients using different languages. Now, your vibe-coded drivers become another layer of potential problems. 4. People has been doing this since way before 2023 btw, most probably agree Postgres would be the sane choice.
DB extensions, standardized backup/recovery solutions, etc
You just don't write some class of software unless that is the product itself. DBs are one of them. Another one is operating systems. It takes a huge level of domain knowledge, real life use and time.
Good luck!
[dead]
It feels jarring to see these LLM generated blog titles everywhere, cant bring myself to read the content of it even if I am interested.
That’s a pretty common title format for talks or blog posts long before LLMs existed.
I support this claim. Also the title already includes everything relevant compared to „we made Postgres 20x faster“.
On the other side if dev(ops) does not know that local storage is faster than network/cluster filesystems…
I feel its hard to prove that either way, but even the content of the article itself is full of LLM generated text, for e.g. here is one line from the conclusion "The bottom line is deliberately narrower than the title".
I'm starting to find complaints about AI generated content more annoying than the content being complained about.
Agreed. Almost every submission has someone claiming it's AI now, it's pretty exhausting.
That's a load-bearing statement.
Yes. My experience: "Oh interesting title". "Oh god... slop galore. Disappointment, sadness, dejection, a feeling of being dismissed and not taken seriously, alienation from the world", "OK close the link".
Happy to still be here among my fellow humans. Hackernews the last bastion it feels like.
Can we somehow filter out slop from the front page? That would be great :).
some kinda system running the linked page through pangram and then putting the result in a small font next to the submission time would be perfect I think. A small font to keep the idea that such things may potentially hold value while satisfying those who aren't interested. Nobody has to argue about it themselves then because it's unambiguous, and everyone can filter them out if you choose.
I'm generally very pro-AI, including for writing, as long as the "producer" (I wouldn't want to call the human responsible the author) has vetted it and will judge based on whether something reads well rather than on whether it feels AI-written.
But using detectors to label so people can choose seems entirely reasonable to me even with a risk of false positives, and I think a lot of people who are generally pro-AI would feel that's fine.
Perhaps LLM users who are open and happy to enable reader choice could label AI-assisted writing as such.