We self-host sentry in Hetzner, but with a high-end server. 96c, 512gb. It ends up only costing around $300 a month, however with the scale of events that it processes, the managed version would be in the 10's of thousands.
The overhead at low volume is pretty high, but in the higher volumes (25M transactions/24h) it's a massive cost saving for us.
Edit:
There were just some initial headaches with needing to increase kafka partitions and add replications to the transaction processors, otherwise we didn't quite leverage the available compute and the backpressure would fill Redis up until OOM.
Same here with the community maintained Helm chart. Not the easiest thing but quite reasonable for almost two years now. This is for 50M transactions per month and we're seeing massive cost savings compared to SaaS at this volume as well.
For those interested in only errors, the self-hosted version recently introduced errors-only mode which should cut down on the containers.
I do have one major complaint though, in dotnet, the tracing/errors are always captured regardless of the sampling rate. So you end up with a lot more memory usage on high throughput/low memory services with no way to lower it.
There's a ticket now open to stop this, but it's still in progress.
Avoiding allocations when a transaction isn’t sampled should be pretty trivial. Only gotcha is that you’d still trace id propagation to tie errors together regardless of transactions. But tracing and transactions got decoupled a while ago so shouldn’t be a big problem.
I’ll leave a comment on ticket pointing to this post
> It ends up only costing around $300 a month, however with the scale of events that it processes, the managed version would be in the 10's of thousands.
I think this is a repeated question but... are you considering the cost of the people managing the deployment, security oversight, dealing with downtime etc?
If you can keep the people doing all the things, they become cheaper over time. Because as your system settles and people become more competent, both downtime and effort required to mend these problems reduce dramatically, and you can give more responsibilities to the same people without overloading them.
You are incentivised to argue that it is good to keep employing sysadmins for self hosting, because that will keep you employed. You have a monetary incentive, thus you are a bit biased, in my opinion.
I think I didn't elaborate my point enough, so there's a misunderstanding.
What I said is true for places where they already have sysadmins for various tasks. For the job I do (it's easy to find), you have to employ system administrations to begin with.
So, at least for my job, working the way I described in my original comment is the modus operandi for the job itself.
If the company you're working in doesn't prefer self-hosting things, and doesn't need system administrators for anything, you might be true, but having a couple of capable sysadmins on board both enables self-hosting and allows this initiative to grow without much extra cost, because it gets cheaper as the sysadmins learn and understand what they're doing, so they can handle more things with the same/less effort.
See, system administrators are lazy people. They'd rather solve problems for once and for all and play PacMan in their spare time.
I am the person, it's occasionally I log in to delete a log file that I just haven't setup to rotate. About once a month, apart from that, no intervention needed (so far).
> Transactions like full user flows start to finish, or 1 transaction = 1 post/get and 1 response?
For most applications we are talking closer to 1 transportation 1 web request. Distributed tracing across microservices is possible, the level of extra effort required depends on your stack. But that's also the out of the box, plug and play stuff. With lower level APIs you define your own transactions, when they start and end, which is needed for tracing applications where there isn't a built in framework integration (e.x not a web application).
Just be forewarned it doesn't seem to offer one iota of IAM, so whether or not one is "overpaying" for a cloud provider depends on what you're getting from them. If you mean "rent a machine," then likely. If you mean "have the machines heal themselves instead of Pagerduty waking me up" then reasonable people can differ about where that money is going
This absolutely mirrors my experience. Sentry was straightforward to deploy years ago, but now seems like one of the more egregious offenders in the, 'self host-able but increasingly prohibitively complex by design' category.
As others have said, we've [0] found the only practical way to deploy this for our clients is Kuberentes + Helm chart, and that's on bare-metal servers (mostly Hetzner). It runs well if you can throw hardware and engineering time at it, which thankfully we can. But given the option we would love a simpler solution.
In our case we have a collection of Ansible roles we use for the purpose. We run databases using the Stackgres operator either using logical replication on local fast NVMe dives, on top of OpenEBS/Mayastor replicated block-storage.
But we specialise in this so that our clients don't have to. As much as I do actually love Kubernetes, the fact that the _easiest_ way to self-host Sentry is via Kubernetes is not a good sign. And choosing to spin up a Kubernetes cluster just to run Sentry would feel a lot like the lady who swallowed a fly[0].
When I posted this myself on Reddit, I said the following:
I've long held off on actually posting this article to a platform like this one (don't bash your competition and all that), but "isn't Sentry self-hosted?" _is_ one of the most asked questions I get, and multiple people have told me this blog-post actually explains the rationale for Bugsink better than the rest of the site, so there you have it.
Well done! I came to the same conclusion (with the exact same bewilderment steps) as I do love Sentry myself. I will definitely try Bugsink, it’s something i’ve been looking for ages.
Feedback on competition bashing: sometimes they deserve it, they should really just come out and say it: “open sourcing our stuff isn’t working for us, we want to keep making money on the hosting”, and that would be ok
We enable self-hosting because not everyone can use a cloud service (e.g. government regulation), otherwise we probably wouldn't even spend energy on it. We dont commercialize it at all, and likely never will. I strongly believe people should not run many systems themselves, and something that monitors your reliability is one such system. The lesson you learn building a venture backed company, and one that most folks miss: focus on growth, not cost-cutting. Self-hosting for many is a form of cost-cutting.
We do invest in making it easier, and its 100% a valid complaint that the entire thing is awful today to self-host, and most people dont need a lot of the functionality we ship. Its not intentional by any means, its just really hard to enable a tiny-scale use-case while also enabling someone like Disney Plus.
Despite my earlier comment, the way Sentry approaches open source is at least a little better than the majority, so its good that you at least provide an escape hatch that means heavy integration with Sentry isn't a complete vendor lock.
On your second point, you're right that self-hosting is cost cutting, but I found that when a business is growing, the features that sentry offers are more of a nice to have along the way, not quite critical to its success (No point capturing errors if no-one's using your bloody thing). Its easy to rack up that monthly bill with nothing but nice to have hosted services.
What I'd love to have is dumb versions of tools to self host initially when the requirements (and traffic) is very low. Kibana for example is a pig to self host, at one point it was taking up 25% of our production capacity. I found Loki is much better for simple cases in the beginning.
Good example that strikes a balance is Grafana / Prometheus. Its practically impossible to run a software shop without them, and everywhere I worked went through the same phases: Chuck a set of random containers in prod -> Deploy helm chart -> Migrate to Thanos -> Move to hosted Grafana when user/teams management gets out of hand.
Benefit is that absolutely everyone is familiar (and even likes) your tooling, I hope that's enough of a reason to give away a simpler offering.
We also found the same problem as OP with self hosting sentry. Each release would unleash more containers and consume more memory until we couldn't run anything on the 32gb server except Sentry.
We looked at both GlitchTip and BugSink and only settled on GlitchTip as it was maintained by a bigger team. Feature wise they were quite similar and both good alternatives.
So far so good with GlitchTip.
And thanks Op for making BugSink, the more alternatives the better.
GlitchTip had replaced our Sentry (9.x, pre-Clickhouse madness). It was just a matter of updating DSN in a few Configmaps/Secrets, good to go from day one. The UI is a bit buggy and "resolve" doesn't always work, but it does 99% of what Sentry did with 10% of the effort to maintain a modern Sentry setup.
Although with Bugsink (which is what came out of this origin story of annoyance) I'm aiming for _even more_ simple (1 Docker container at minimum, rather than 4), 30 seconds up and running etc.
Hello, ping me (GlitchTip lead) if you want to collaborate. Your stack is also Django. I'd be open to simplifying - we could probably make Redis optional. I have an experimental script that runs celery and Django in one process. But I think Postgres is a must have. So that gets down to two. My hope is that GlitchTip works for super small use cases and scales with minimal thought required.
I saw the headline and wanted to make sure someone mentioned GlitchTip. It doesn't have all of the functionality of Sentry but has all of the functionality I need. We have been running it in production for a year with no problems. Given our small user base (<1000 users), Sentry did not make sense.
Also a fan of GlitchTip so far. I only recently (last month or so) started using it. I made a railway template for it and have been using it to monitor a handful sites. I used valkey and minio for storage. Makes it super easy to spool up an instance.
We run them both as we are evaluating glitchtip, but, at least for us, it has so many bugs vs sentry. But it's so much lighter so we try to stick with it.
The biggest issue I have with these solutions is indeed local debugging.
I use Sentry with most of my clients, and for effective debugging I need to spin my own Sentry in a Docker container which ends up being quite heavy on my machine especially when combined with Grafana and Prometheus.
I'm really unhappy with virtually all monitoring/telemetry/tracking solutions.
It really feels they are all designed to vendor lock you in their expensive cloud solutions and I really don't feel I'm getting my $s back at all. Worst of all those cloud vendors would rather add new features non-stop rather than honing what they currently have.
Their sales are everywhere, I've seen two different clients getting Datadog sales people join private Slacks to evangelize their products.
Both times I escalated to the CTO, both times I ended up suspecting someone in management had something to gain from pushing teams to adopt those solutions.
Sentry's sales team is incredibly aggressive. I've seen multiple colleagues hijacked for sales presentations over a few months. It would not surprise me at all if they just asked random employees to be added to the company slack, and even less if those people then just did it.
I can only commend the hustle on their part, but it does feel a little like a high pressure time share situation.
If you - or anyone reading this - ever ends up in a situation where we came off as aggressive send me a direct email and I will take care of it. This is not something we believe in at Sentry, and while you cant manage everything, its important to us that we never become "of those companies" like so many other successful companies become.
I still love Sentry, but it’s so enormous now that it isn’t practical to self-host for smaller businesses. A “small” alternative is always great to see!
I’m not sure how I feel about the license though (Polyform Shield, basically use-but-don’t-compete). It’s a totally valid choice – I just wish it would convert to FOSS at some point. (I understand the concern as I’ve had to make a similar decision when releasing https://lunni.dev/. I went with AGPL, but I’m still overthinking it :-)
So tired of hearing this trope. Electron is alright. Memory is cheap. Tell me a single better way to write cross-platform UI other than a worse version of Electron.
If you're running on commodity hardware, sure; if you happen to be a $1T company that solders 3x marked-up RAM then that's definitely not true https://www.apple.com/shop/buy-mac/macbook-pro/16-inch-space... is the entry model, and clicking the 48GB option dials the price up to $3k
Great ending, honestly. I hope that dev got fired and truly understood what he had done, and felt the appropriate amount of shame. Not for his error, because we all make mistakes, but for his hubris that allowed him to keep making that same mistake over and over, while insisting he was doing the right thing.
Ok so your dev can't be told off for bringing the stuff out (and for being a moron doing sync calls to a logging service) and this brought the company down
But it was a good call sending it to the cloud. Better than "my problem" it is something being "somebody else's problem"
The amount of shell script that needs to be executed to install is a bit of a no no for me. It also doesn't make sense to spin up a 16GB machine (minimum!) to track the errors on those 4-8GB VPS which are running my production services.
> we don’t necessarily recommend self-hosting to everyone. In addition to existing hidden costs, as Sentry evolves, our self-hosted version will become more complex, demanding additional types of infrastructure.
Any insights on why Sentry is so complex and needs so much resources? Is collecting, storing, and organizing errors messages and stack traces at scale difficult? Or it's the other features on top of this?
- they had enough money that they never needed to think seriously about maintenence cost, and the sales process was strokg enough to keep customers arriving anyway (look to Oracle for another example of hopelessly complicated installation process but people keep using it anyway)
- at some point someone realized this was actually a feature: the more complicated it got, the harder it became to self host. And from that perspective it is a win-win for the company: they can claim it is open source without being afraid that most people will choose to self host.
I'd say that the architecture that is desirable from the point of view of a large scale SaaS is very different than the one that's desirable from the point of view of a tool that just needs to work for a single organization. And since the SaaS is bringing in all the money, that's where the architecture follows.
> actually a feature
I would guess that for a few people people (e.g. the ones who made the scary visual of rising costs) this is explicitly so, but for most people it's more implied. i.e. I don't think anyone advanced their career with Sentry by making self-hosting easier.
Anything gets complex at scale. This is the same software they us to host their SaaS system. Presumably has to scale to many thousands of customer and a huge number of events per second.
They have all sorts of caching, autoscaling, distributed systems and other stuff thats complete overkill for all except that largest installation. Plus all sorts of software features only needed by a few customers and extra layers to be multi-customer.
It's the difference between a hoop in your back yard and a NBA Stadium
Is this a common architectural issue for self-hosted options from SaaS companies?
As in, a huge SaaS company offers their product for self-hosting to individual companies, but it's not practical to self-host because the code is highly specialized for supporting hundreds of companies instead of just one? And it's hard to have an architecture that works well for just one and for hundreds?
It's more about scale than tenancy. Not many SaaS companies offer such an option in the first place but it is typical that the in-house product is the priority and the architectural decisions are made with that in mind firstly, and self-hosting second if at all.
For example Sentry requires ClickHouse, Postgres, Kafka, and Redis presumably because they were the right tools for their needs and either they have the resources to operate them all or the money to buy the managed options from vendors.
Also, the main concern people have with hosting Sentry is the sheer number of containers required but most of them are just consumers for different Kafka queues which again is presumably this way because Sentry ops prefers it this way, whether it be for fine tuning the scaling of each one or whatever the reason.
What makes sense for a SaaS company rarely translates to sensible for self-hosting.
I use both hosted and self hosted sentry. I prefer hosted it's less to manage but self hosted isn't too awful as long as you can budget the resources. If you just want the team sentry plan it's going to be cheaper to use hosted. I would only self host if you had to for legal/compliance reasons. In the many years I've been managing sentry we've only had it truly crap out once and that was an upgrade from 9 to their new date versioning basically the whole hosted method changed and it was just easier to start over from scratch.
This is also my opinion of Backstage. It's positioned as this easy-to-use open source tool for IDPs - but in reality, it's a product that needs a team to maintain it constantly.
So many services are going this route. I remembered Reddit had an install script and almost everything worked out of the box 10-15 years ago on a Ubuntu VPS.
I'm a big fan of self-hosting, but it requires the right infrastructure. You need a robust virtualization solution like Proxmox, equipped with sufficient CPU cores and RAM. For instance, allocating 16GB of RAM shouldn't be an issue if your server has 128GB to share.
In my experience, solutions like Mailcow, which involve multiple services and containers (such as SMTP, IMAP, Redis, SSO, webmail, Rspamd, etc.), work very well. I have extensive experience running these systems, performing backups, restoring data, and updating their ecosystems.
Additionally, I've had a positive experience setting up and running a self-hosted Sentry instance with Docker for a project that spanned several years. However, this experience might be somewhat outdated, as it was a few years ago.
> First, there’s a signal being given about the complexity and fragility of the software. A 16GB RAM requirement just screams “this is a big, complex piece of software that will break in mysterious ways”. Not very scientific, I know, but based on experience.
This lines up with my experience self hosting a headless BI service. In "developer mode" it takes maybe 1GB RAM. But as soon as you want to go prod you need multiple servers with 4+ cores and 16GB+ RAM that need a strongly consistent shared file store. Add confusing docs to the mix, mysterious breakages, incomplete logging and admin APIs, and a reliance on community templates for stuff like k8s deployment... it was very painful. I too gave up on self hosted.
In most corporate projects that I worked with, the day it went to production started the clock to disband the core team and optimizations were not on the table. When the product works, they move on. The only area where performance optimization is a mandatory step is video games, but that is where your clients are external. Most business software is in the "finally, but barely works" state and new features beat performance improvements.
This is caused by short sighted management that need to deliver and move on. "Long term" is a contradiction with their business model. In this case "long term" means "after product launch".
I worked in a startup around 2019-2020 where we started using Sentry for distributed tracing. It was a company in the IOT space and we had a lot of events going through. The bill started going in the $3-5000 range just from the tracing, so we decided to host it on our own. When looking at the sheer complexity of the project, I was flabbergasted. We need a name for managing the infrastructure of company based open source, self-hosted solutions. Often times, the right choice would be to choose a different, simpler, open source solution. In this case there are some neat distributed tracing solutions, that are easier to manage.
It might not do everything Sentry does but it definitely has helped with tracking down some issues, even production ones and runs in a fairly manageable setup (in comparison to how long even the Sentry self-hosted Docker Compose file is).
What’s more, if you want, you can even use regular PostgreSQL as the backing data store (might not be quite as efficient as ElasticSearch for a metrics use case, but also doesn’t eat your RAM like crazy).
At first glance it's not immediately obvious to me, why would you pick Sentry or Bugsink over something included in the Grafana stack? What's the use use
I haven't used Sentry's obserability portions, but the error capture has very strong deduplication built into it, and can optionally integrate with your SCM, CI, and issue tracker, if one wishes. Thus, the flow can go like:
- error happens, can be attributed to release 1.2.3
- every subsequent time that error happens to a different user, it can track who was affected by it, without opening a new error report
- your project can opt-in to accepting end-user feedback on error: "please tell us what you were doing when this exploded, or feel free to rant and rave, we read them all"
- it knows from the stack trace that the error is in src/kaboom/onoz.py line 55
- onoz.py:55 was last changed by claude@example.com last week, in PR #666
- sentry can comment upon said PR to advise the reviewers of the bad outcome
- sentry can create a Jira with the relevant details
- claude.manager@example.com can mark the bug as "fixed in the next release", which will cause sentry to suppress chirping about it until it sees a release 1.2.4
- if it happens again it will re-open the prior error report, marking it as a regression
Unless you know something I don't, Grafana does *ABSOLUTELY NONE* of that
We've just applied a helm chart a while back. It just works. We maybe had like a few incidents over the years, requiring stuff like Kafka queues to be wiped.
The argument that you have to read a sh script doesn't make sense to me. Are you gonna read source code of any software is referenced in this script or any you download too? No? What's the difference between that and a bash script, at the end of the day both can do damage.
We used the helm chart but things didn't get updated often enough to keep our container security stuff happy.
Helm is a huge pain in the butt if you have mitigation obligations because the overall supply chain for a 1-command install can involve several different parties, who all update things at different frequencies :/
So chart A includes subchart B, which consumes an image from party C, who haven't updated to foobar X yet. You either need to wait for 3 different people to update stuff to get mainline fixed, or you roll up your sleeves and start rebuilding things, hosting your own images and forking charts. At first you build 1 image and set a value but the problem grows over time.
If you update independently you end up running version combinations of software that the OG vendor has never tested.
This is not helm's fault of course; it's just the reality of deploying software with a lot of moving parts.
Rereading that section, I'd agree it's probably not the best-argued point because it implies security concerns... I guess what I'm saying is: for something I'm setting up to keep around for a while, I'd like to know a bit what's in the package before I deploy it. In that sense, the shell script serves as a table of contents... and if the table of contents is 800 lines, that makes me wonder how many moving parts there are and how many things might break at inconvenient times because of that.
For me I would just run it on a clean cluster/VM somewhere (to be destroyed after that) just to see what happens. If you have no local resources to spare, an hour of even very high end (to save time) VMs/cluster at a provider e.g. AWS costs next to nothing
That solution didn't apply for me at the time, since I was in an environment that combined security-consciousness with thick layers of bureaucracy, meaning that hardware came at a premium (and had to be on premise).
Sure, but I'm not suggesting running there, just testing there. We also have to run in specific providers in specific location, but nothing stops us from renting a clean large VM in AWS for an hour or two, for testing stuff without using any customer data. Hell, that costs pretty much nothing so if my employer didn't allowed it, I would just pay with my own money - it's much better for your work efficiency to work out the kinks of this without having to do 10 cleanups after failed deployment, it's much easier than to just delete a VM.
Oh and the most difficult part when setting up, from what I remember, was setting up GitHub SSO and GitHub and Slack integration as it wasn't well documented.
Software that's only available as x86 binaries and a whole system that needs 16 gigs of memory? It's not surprising that it comes from the corporate world and not the open source world.
I respect the fact that the Sentry people are honest about their teetering tower.
That's the inevitable evolution of any popular open-source product/company. Light and enterprise-ready is hardly compatible, and you can expect this from any project that has a cloud offering.
I don't have servers like this but the author makes it easy to understand, and it applies to a lot of other things.
The Comparative Costs picture does tell it all.
The purpose of so much software is to reduce human and machine costs through time, and this apparently turned out to do just the opposite, apparently after long-term testing under real-world conditions.
Could be an unsurmountable fundamental structure of technical debt or something like that which metastasizes.
Then this, anyone could say about anything:
>I’m not going to run a piece of software that requires 16GB of RAM, has a complex installation script, and is known to be a pain to maintain. I’m not going to recommend it to anyone else either.
Easier said than done, sometimes it's the only option.
It's a "script". Maybe that's why you have to "rehearse" it more than once before you barely get it right, and then you might have to really go the extra mile (and have a bit of good fortune) before you can achieve a "command performance".
How do you think it feels these days to have to settle for this kind of thing in expensive proprietary software too?
This isn't okay - the author is selling their own alternative to Sentry, 'reusing' Sentry's open-source client SDK's, while spreading FUD about self-hosting Sentry.
I've been self-hosting Sentry for over 10 years: Sentry is installed by running `git clone`, editing one or two configuration files, and running ./install.sh. It requires 16GB RAM if you enable the full feature set. That includes automatically recording user sessions on web or mobile for replay, and support for end-to-end tracing (so seeing which database queries your api asks for in response to a button tap in your mobile app).
Sentry has a wonderful self-hosting team that's working hard to make Sentry's commercial feature set available, for free, to everyone that needs a mature error tracking solution. You can talk to them on discord and they listen and help.
Just for transparency, are you by any chance a member of this self-hosting team or a sentry employee? Is it a coincidence that your keybase name is georghendrik according to your profile, and the painter _Georg Hendrik_ Breitner painted a picture called "Sentry"? https://www.demesdagcollectie.nl/en/collection/hwm0055
This is some Olympics level of badly coping with valid and welcome criticism.
Georg Hendrik = "George Henry", pretty common name. The fact that Google returned a result when you searched "Georg Hendrik Sentry" should not be considered weird.
I was born 400 years ago in the Highlands of Scotland. I am Immortal, and I am not alone. Now is the time of the Gathering, when the stroke of the sword will release the power of the Quickening. In the end, there can be only one.
It seems like all the FUD in the article is copy-pasted from Sentry's own docs, though, no? And assuming Sentry's SDKs are open source and licensed appropriately (which seems to be the case), there's no issue (legal or moral) with directing users to use the Sentry SDK to communicate with a different, compatible product.
OP built a product because they were frustrated by Sentry's seeming hostility toward self-hosting. It doesn't feel like OP decided to build a competing product and then thought it would be a good marketing strategy to falsely push the idea that Sentry is difficult to self-host.
FWIW I've never self-hosted Sentry, but poking around at their docs around it turns me off to the idea. (I do need a Sentry-like thing for a new project I'm building right now, but I doubt I'll be using Sentry itself for it.) Even if it's possible to run with significantly less than 16GB (like 1GB or so), just listing the requirements that way suggests to me that running in a bare-bones, low-memory configuration isn't well tested. Maybe it's all fine! But I don't feel confident about it, and that's all that matters, really.
TBH most of the FUD in the OP is straight from Sentry's own website.
Regarding using the SDKs, I'm telling my users to take Sentry at their word when they wrote "Permission is hereby granted, free of charge [..] to deal in the Software without restriction, including without limitation the rights to use"
I am not sure why there isn't a even lightweight sentry endpoint that does
- save post body to folders (use uuid as folder name to avoid spam)
- dir listing, and count number of entries
- render posted json to html, highlight stacktrace with js
- download raw json
- rotate, compress old entries.
I give those requirements to LLM, and I get a pretty much working rust implementation after few tweaks. It uses <5M ram idle.
That covers the most basic feature that Sentry has (one-off error reporting, with rudimentary statistics). Sure, if that's enough for you then go ahead, but for most people using Sentry that's only the tip of the iceberg.
I was looking into moving away from managed Sentry for GDPR reasons. I was only using Sentry for notifications about unhandled exceptions in my app. All the fancy APM features I did not use. Grouping of similar exceptions, pretty display of tracebacks were nice but not essential.
In my Django app I wrote a logging handler that stores the log records (including traceback) in a database table. I can inspect the log records through Django admin, and a cron job sends me daily emails saying "X new log records in the last 24 hours" so I know to check them out. And that's it :-)
Of course, this does a lot less than Sentry, and has various limitations (e.g. what if the error is about the database being down...), but it fits my needs.
TL;DR: Not so much "gave up" as "never tried" because I was scared away by 16GB RAM requirement.
FWIW, we've been self-hosting 2 instances (and purchase a third hosted instance), for, it looks like 8 years now, and it's had a few small bumps but hasn't been too bad. Our instances are running with 20GB of RAM, so pretty small. ~90% of updates have gone smoothly, which leaves 10% of updates that have had some sort of problem. Solutions have been found in the issue tracker in all our cases. We are a little afraid of doing updates, but do them every couple of months.
Sentry is an amazing piece of software and it is fantastic that they offer a self-hosing version, and all things considered it is a fairly easy self-host.
I "gave up" from the perspective of returning to Sentry after a couple of years, and finding an entirely different beast from the tool I loved before. At that point I indeed didn't make it past Sentry's own FUD.
It's wild that Sentry themselves are spreading literal FUD about about bad it is to self-host their own product. So confounding that they're so ready to shit on their own product.
> I’m not going to run a piece of software that requires 16GB of RAM, has a complex installation script, and is known to be a pain to maintain.
This has to be self-hosted eventually either by you or Sentry themselves so the full cost of this is coming down somewhere. The planet is round and all that and there's no getting away from these inefficiencies, but you can kick the proverbial can down the road.
Also, they are incentivized to make the open product as hard to use as possible so color me surprised the self-hosted edition is hard to operate.
Sure, but "when using Sentry" is an artificial constraint. There's no reason to constrain yourself that way when there are alternatives that might be sufficient for your needs, that don't require a beefy machine and a lot of maintenance effort when self-hosting.
I no longer work at Sentry but the complexity of operating Sentry is something that we still worked on when I was there. There is no secret incentive to make Sentry hard to operate. You have to consider that engineers run a version of Sentry on their machines too and any complexity ends up there too.
One of the biggest pain points of running Sentry is the number of containers and a lot of this comes from how Sentry works with Kafka and Rabbit. That pain point is actively being addressed by moving to a virtualized system [1] though I am not sure how long it will take to actually adopt this.
I don't think Sentry is being evil or is trying to make it harder to self host, but I do think that Sentry doesn't have a big incentive to spend a lot of effort making it easier to self host. Again, I don't think that's evil. I think it's entirely reasonable and ok, even if it does make life harder for some people.
Maybe this virtualized system will make things easier. If so, that's great. But if it ends up not working out, or if it does, but over the longer term things get more difficult again, I think that's still just kinda how things happen sometimes, and that's ok.
One of Sentry's goals is for Sentry themselves to operate it as a hosted cloud service. Architecture decisions made to further that goal can naturally and reasonably be at odds with another goal to make it simpler to self host. Sometimes things can't be one-size-fits-all.
I already mentioned this elsewhere here but Sentry does have an incentive to improve this. That’s because the complexity creates challenges in the operation of smaller instances and during development.
> There is no secret incentive to make Sentry hard to operate
It's frustrating when half the comments on a company that dares to open their product is always about how they are obviously intentionally very evil to not do it perfectly/for totally free/with 0 friction/etc.
but in this case the order of events was not "awesome product that was open-sourced as-is later" but "OS product that became harder and harder to install over time"
Products get bigger and more complex over time, so "harder and harder to install, configure and manage" is somewhat normal. The money spent on free or "community" edition is very difficult to justify, so it slowly degrades with time until it becomes unusable and people move on to something else. It is part of the normal lifecycle of software.
A fairly easy answer is "one of the purposes of Sentry is running Sentry-as-a-service, at large scale". Which comes with compromises that make smaller deployments more "bloated" too, but is a very different thing than intentionally choosing to bloat it to make it hard to run.
It's definitely true that what Sentry is today, is a function of operating it at scale. Thankfully one of the intended purposes of Sentry is also that it scales down much better than today [1]. That it does not is also something that Sentry as a company is not happy with which is why there are desires to change that, which will also benefit self hosted.
[1]: this is because a) developing on sentry has become harder and harder on local machines and b) operating single-tenant installations for some customers brings many of the same challenges that people on self-hosted run into. c) we want more regions of Sentry and smaller ones (such as Europe today) have higher fixed costs than ideal.
Yes.. we're getting there. "one of the purposes of Sentry is running Sentry-as-a-service". Why? To make the dwarves happy? It's making money. And with it comes a whole truckload of very obvious incentives. Notice I don't say anyone is intentionally being evil. That's quite unnecessary.
No, its complicated and scales to servicing every customer in the world. Thats not the same thing.
Doesnt mean the complaints about self-hosted arent valid, but "literally has to scale to the most insane volumes of data" and "is not good software" are two different things.
We're building a cloud service at the end of the day - its a lot easier to optimize a multi-tenant install than it is a single-tenant install, and that complexity shows up in the self-hosted repo. Can't really avoid it.
We self-host sentry in Hetzner, but with a high-end server. 96c, 512gb. It ends up only costing around $300 a month, however with the scale of events that it processes, the managed version would be in the 10's of thousands.
The overhead at low volume is pretty high, but in the higher volumes (25M transactions/24h) it's a massive cost saving for us.
Edit:
There were just some initial headaches with needing to increase kafka partitions and add replications to the transaction processors, otherwise we didn't quite leverage the available compute and the backpressure would fill Redis up until OOM.
Same here with the community maintained Helm chart. Not the easiest thing but quite reasonable for almost two years now. This is for 50M transactions per month and we're seeing massive cost savings compared to SaaS at this volume as well.
For those interested in only errors, the self-hosted version recently introduced errors-only mode which should cut down on the containers.
Yeah I fully get how that's a volume where going self-hosted Sentry makes perfect sense at the bottom line and including any upkeep you might have.
Bugsink's also quite scalable[0], but I wouldn't recommend it a 25M/day.
[0] https://www.bugsink.com/scalable-and-reliable/
> Bugsink's also quite scalable[0], but I wouldn't recommend it a 25M/day.
Well, your homepage disagrees with this statement:
> Bugsink can deal with millions of events per day on dirt cheap hardware
To me, "millions" usually means less than 10M. 25M falls into "tens of millions" to me.
But it's a very fuzzy way of quantifying something, and open to various interpretations.
Fair point, but if it can cope with millions on "dirt cheap hardware" it logically follows that it can do 25M on more expensive hardware.
I do have one major complaint though, in dotnet, the tracing/errors are always captured regardless of the sampling rate. So you end up with a lot more memory usage on high throughput/low memory services with no way to lower it.
There's a ticket now open to stop this, but it's still in progress.
Avoiding allocations when a transaction isn’t sampled should be pretty trivial. Only gotcha is that you’d still trace id propagation to tie errors together regardless of transactions. But tracing and transactions got decoupled a while ago so shouldn’t be a big problem. I’ll leave a comment on ticket pointing to this post
It's open source, you guys could always create a PR to fix it. That's the power of open source!
there's no guarantee it will get merged though, even if a PR is created.
Forking has down sides that can't be hand waved away too, especially for a service like this.
It's just a client library, what's the alternative, put a proxy in front that drops 90% of the events
Any chance you can link me to the ticket?
Feel free to email - david at sentry
It looks like there was some motion a week ago.
https://github.com/getsentry/sentry-dotnet/issues/3636#event...
> It ends up only costing around $300 a month, however with the scale of events that it processes, the managed version would be in the 10's of thousands.
I think this is a repeated question but... are you considering the cost of the people managing the deployment, security oversight, dealing with downtime etc?
If you can keep the people doing all the things, they become cheaper over time. Because as your system settles and people become more competent, both downtime and effort required to mend these problems reduce dramatically, and you can give more responsibilities to the same people without overloading them.
Disclosure: I'm a sysadmin.
I wonder what is your managers take on this, given your incentives here.
Honestly asking, what my incentives are looking like from there?
You are incentivised to argue that it is good to keep employing sysadmins for self hosting, because that will keep you employed. You have a monetary incentive, thus you are a bit biased, in my opinion.
I think I didn't elaborate my point enough, so there's a misunderstanding.
What I said is true for places where they already have sysadmins for various tasks. For the job I do (it's easy to find), you have to employ system administrations to begin with.
So, at least for my job, working the way I described in my original comment is the modus operandi for the job itself.
If the company you're working in doesn't prefer self-hosting things, and doesn't need system administrators for anything, you might be true, but having a couple of capable sysadmins on board both enables self-hosting and allows this initiative to grow without much extra cost, because it gets cheaper as the sysadmins learn and understand what they're doing, so they can handle more things with the same/less effort.
See, system administrators are lazy people. They'd rather solve problems for once and for all and play PacMan in their spare time.
I have to imagine at that size they have an ops team already for all the other services so those are pretty amortized.
I am the person, it's occasionally I log in to delete a log file that I just haven't setup to rotate. About once a month, apart from that, no intervention needed (so far).
Am I reading correctly that your software generates 25 million error messages per day?
Sentry does a lot more than tracking errors - presumably most of those transactions are 'breadcrumb'-style events.
Nope, 25M transactions. In Sentry a transaction is more like an OTEL-trace. Errors are much lower ;)
sorry if this is a silly question, the quick google search didn’t give me clues.
Transactions like full user flows start to finish, or 1 transaction = 1 post/get and 1 response?
> Transactions like full user flows start to finish, or 1 transaction = 1 post/get and 1 response?
For most applications we are talking closer to 1 transportation 1 web request. Distributed tracing across microservices is possible, the level of extra effort required depends on your stack. But that's also the out of the box, plug and play stuff. With lower level APIs you define your own transactions, when they start and end, which is needed for tracing applications where there isn't a built in framework integration (e.x not a web application).
That is more likely performance traces or session replays.
> a high-end server. 96c, 512gb. It ends up only costing around $300 a month
Wow, that's really cheap. I'm seriously overpaying for my cloud provider and need to try Hetzner. I always assumed Hetzner was only European based.
Just be forewarned it doesn't seem to offer one iota of IAM, so whether or not one is "overpaying" for a cloud provider depends on what you're getting from them. If you mean "rent a machine," then likely. If you mean "have the machines heal themselves instead of Pagerduty waking me up" then reasonable people can differ about where that money is going
FWIW, https://lowendbox.com/ is good fun for the former set of things, too
This absolutely mirrors my experience. Sentry was straightforward to deploy years ago, but now seems like one of the more egregious offenders in the, 'self host-able but increasingly prohibitively complex by design' category.
As others have said, we've [0] found the only practical way to deploy this for our clients is Kuberentes + Helm chart, and that's on bare-metal servers (mostly Hetzner). It runs well if you can throw hardware and engineering time at it, which thankfully we can. But given the option we would love a simpler solution.
[0]: https://lithus.eu
And how do you install and maintain/upgrade kubernetes? Are you running databases also on kubernetes?
In our case we have a collection of Ansible roles we use for the purpose. We run databases using the Stackgres operator either using logical replication on local fast NVMe dives, on top of OpenEBS/Mayastor replicated block-storage.
But we specialise in this so that our clients don't have to. As much as I do actually love Kubernetes, the fact that the _easiest_ way to self-host Sentry is via Kubernetes is not a good sign. And choosing to spin up a Kubernetes cluster just to run Sentry would feel a lot like the lady who swallowed a fly[0].
[0]: https://en.wikipedia.org/wiki/There_Was_an_Old_Lady_Who_Swal...
Thanks for the poem, it seems pretty apt for IT in 2025.
That said I would honestly prefer if the industry would just settle on K8s as our OS.
I really do not see any benefit that sentry could bring on its own compared to a solid set of Helm charts for k8s.
Does it really have to be a proper cluster, though? Can it be e.g. a single node k3s?
Yes, you could definitely do that. Although if going that route I'd consider taking a run at their docker compose self-hosting instructions first.
Hey, that's me!
When I posted this myself on Reddit, I said the following:
I've long held off on actually posting this article to a platform like this one (don't bash your competition and all that), but "isn't Sentry self-hosted?" _is_ one of the most asked questions I get, and multiple people have told me this blog-post actually explains the rationale for Bugsink better than the rest of the site, so there you have it.
Well done! I came to the same conclusion (with the exact same bewilderment steps) as I do love Sentry myself. I will definitely try Bugsink, it’s something i’ve been looking for ages.
Feedback on competition bashing: sometimes they deserve it, they should really just come out and say it: “open sourcing our stuff isn’t working for us, we want to keep making money on the hosting”, and that would be ok
fwiw I was always pretty transparent about our priorities:
https://blog.sentry.io/building-an-open-source-service/
We enable self-hosting because not everyone can use a cloud service (e.g. government regulation), otherwise we probably wouldn't even spend energy on it. We dont commercialize it at all, and likely never will. I strongly believe people should not run many systems themselves, and something that monitors your reliability is one such system. The lesson you learn building a venture backed company, and one that most folks miss: focus on growth, not cost-cutting. Self-hosting for many is a form of cost-cutting.
We do invest in making it easier, and its 100% a valid complaint that the entire thing is awful today to self-host, and most people dont need a lot of the functionality we ship. Its not intentional by any means, its just really hard to enable a tiny-scale use-case while also enabling someone like Disney Plus.
Despite my earlier comment, the way Sentry approaches open source is at least a little better than the majority, so its good that you at least provide an escape hatch that means heavy integration with Sentry isn't a complete vendor lock.
On your second point, you're right that self-hosting is cost cutting, but I found that when a business is growing, the features that sentry offers are more of a nice to have along the way, not quite critical to its success (No point capturing errors if no-one's using your bloody thing). Its easy to rack up that monthly bill with nothing but nice to have hosted services.
What I'd love to have is dumb versions of tools to self host initially when the requirements (and traffic) is very low. Kibana for example is a pig to self host, at one point it was taking up 25% of our production capacity. I found Loki is much better for simple cases in the beginning.
Good example that strikes a balance is Grafana / Prometheus. Its practically impossible to run a software shop without them, and everywhere I worked went through the same phases: Chuck a set of random containers in prod -> Deploy helm chart -> Migrate to Thanos -> Move to hosted Grafana when user/teams management gets out of hand.
Benefit is that absolutely everyone is familiar (and even likes) your tooling, I hope that's enough of a reason to give away a simpler offering.
Your project is awesome, you should do a show HN later.
Thanks for the kind words!
In fact I did one last week, but it got only a fraction of today's article's traction... I'll try again in whatever the prescribed interval is :-)
You can submit 2-3 times over a couple days.
FOSS Sentry fork GlitchTip keeps things more simple and self-hosting friendly.
https://glitchtip.com/
+1 for Glitchtip.
We also found the same problem as OP with self hosting sentry. Each release would unleash more containers and consume more memory until we couldn't run anything on the 32gb server except Sentry.
We looked at both GlitchTip and BugSink and only settled on GlitchTip as it was maintained by a bigger team. Feature wise they were quite similar and both good alternatives.
So far so good with GlitchTip.
And thanks Op for making BugSink, the more alternatives the better.
GlitchTip had replaced our Sentry (9.x, pre-Clickhouse madness). It was just a matter of updating DSN in a few Configmaps/Secrets, good to go from day one. The UI is a bit buggy and "resolve" doesn't always work, but it does 99% of what Sentry did with 10% of the effort to maintain a modern Sentry setup.
And actually open source, which matters to some folks
Indeed it does!
Although with Bugsink (which is what came out of this origin story of annoyance) I'm aiming for _even more_ simple (1 Docker container at minimum, rather than 4), 30 seconds up and running etc.
Hello, ping me (GlitchTip lead) if you want to collaborate. Your stack is also Django. I'd be open to simplifying - we could probably make Redis optional. I have an experimental script that runs celery and Django in one process. But I think Postgres is a must have. So that gets down to two. My hope is that GlitchTip works for super small use cases and scales with minimal thought required.
I saw the headline and wanted to make sure someone mentioned GlitchTip. It doesn't have all of the functionality of Sentry but has all of the functionality I need. We have been running it in production for a year with no problems. Given our small user base (<1000 users), Sentry did not make sense.
Also a fan of GlitchTip so far. I only recently (last month or so) started using it. I made a railway template for it and have been using it to monitor a handful sites. I used valkey and minio for storage. Makes it super easy to spool up an instance.
We run them both as we are evaluating glitchtip, but, at least for us, it has so many bugs vs sentry. But it's so much lighter so we try to stick with it.
Can you say what kind of bugs? This application looks interesting; I want to try it instead of https://healthchecks.io/
The biggest issue I have with these solutions is indeed local debugging.
I use Sentry with most of my clients, and for effective debugging I need to spin my own Sentry in a Docker container which ends up being quite heavy on my machine especially when combined with Grafana and Prometheus.
I'm really unhappy with virtually all monitoring/telemetry/tracking solutions.
It really feels they are all designed to vendor lock you in their expensive cloud solutions and I really don't feel I'm getting my $s back at all. Worst of all those cloud vendors would rather add new features non-stop rather than honing what they currently have.
Their sales are everywhere, I've seen two different clients getting Datadog sales people join private Slacks to evangelize their products.
Both times I escalated to the CTO, both times I ended up suspecting someone in management had something to gain from pushing teams to adopt those solutions.
I actually wrote about that scenario!
Killing flies with hammers and all, but since I really like my hammer I actually do all my local development with my full-blown error tracker too:
https://www.bugsink.com/blog/local-development/
Sentry's sales team is incredibly aggressive. I've seen multiple colleagues hijacked for sales presentations over a few months. It would not surprise me at all if they just asked random employees to be added to the company slack, and even less if those people then just did it.
I can only commend the hustle on their part, but it does feel a little like a high pressure time share situation.
If you - or anyone reading this - ever ends up in a situation where we came off as aggressive send me a direct email and I will take care of it. This is not something we believe in at Sentry, and while you cant manage everything, its important to us that we never become "of those companies" like so many other successful companies become.
david at sentry.io
I still love Sentry, but it’s so enormous now that it isn’t practical to self-host for smaller businesses. A “small” alternative is always great to see!
I’m not sure how I feel about the license though (Polyform Shield, basically use-but-don’t-compete). It’s a totally valid choice – I just wish it would convert to FOSS at some point. (I understand the concern as I’ve had to make a similar decision when releasing https://lunni.dev/. I went with AGPL, but I’m still overthinking it :-)
also in recent news: https://it-notes.dragas.net/2024/12/28/i-almost-died-for-a-f...
> Code needs to be written properly; you can’t just waste money and resources endlessly to cover up inefficiencies.
Quite rare to hear this wise line these days. An I guess with AI coding assistant, this is only the beginning of this kind of horror story
For me, the horror story started when people ditched optimal desktop apps for Electron because they knew js and css.
Perhaps we can make an AI translate electron apps into native apps…
So tired of hearing this trope. Electron is alright. Memory is cheap. Tell me a single better way to write cross-platform UI other than a worse version of Electron.
> Memory is cheap
If you're running on commodity hardware, sure; if you happen to be a $1T company that solders 3x marked-up RAM then that's definitely not true https://www.apple.com/shop/buy-mac/macbook-pro/16-inch-space... is the entry model, and clicking the 48GB option dials the price up to $3k
The cheapest Apple laptops start with 16GB now. That's enough for quite a few desktop apps regardless of Electron.
Sure, but what do you think would count as better?
Great ending, honestly. I hope that dev got fired and truly understood what he had done, and felt the appropriate amount of shame. Not for his error, because we all make mistakes, but for his hubris that allowed him to keep making that same mistake over and over, while insisting he was doing the right thing.
Ok so your dev can't be told off for bringing the stuff out (and for being a moron doing sync calls to a logging service) and this brought the company down
But it was a good call sending it to the cloud. Better than "my problem" it is something being "somebody else's problem"
enjoyable read. thanks for sharing
The amount of shell script that needs to be executed to install is a bit of a no no for me. It also doesn't make sense to spin up a 16GB machine (minimum!) to track the errors on those 4-8GB VPS which are running my production services.
> we don’t necessarily recommend self-hosting to everyone. In addition to existing hidden costs, as Sentry evolves, our self-hosted version will become more complex, demanding additional types of infrastructure.
Any insights on why Sentry is so complex and needs so much resources? Is collecting, storing, and organizing errors messages and stack traces at scale difficult? Or it's the other features on top of this?
Some ideas:
- they had enough money that they never needed to think seriously about maintenence cost, and the sales process was strokg enough to keep customers arriving anyway (look to Oracle for another example of hopelessly complicated installation process but people keep using it anyway)
- at some point someone realized this was actually a feature: the more complicated it got, the harder it became to self host. And from that perspective it is a win-win for the company: they can claim it is open source without being afraid that most people will choose to self host.
I'd say that the architecture that is desirable from the point of view of a large scale SaaS is very different than the one that's desirable from the point of view of a tool that just needs to work for a single organization. And since the SaaS is bringing in all the money, that's where the architecture follows.
> actually a feature
I would guess that for a few people people (e.g. the ones who made the scary visual of rising costs) this is explicitly so, but for most people it's more implied. i.e. I don't think anyone advanced their career with Sentry by making self-hosting easier.
Well I actually did as I was literally hired for that and worked on it for almost 3 years :)
Anything gets complex at scale. This is the same software they us to host their SaaS system. Presumably has to scale to many thousands of customer and a huge number of events per second.
They have all sorts of caching, autoscaling, distributed systems and other stuff thats complete overkill for all except that largest installation. Plus all sorts of software features only needed by a few customers and extra layers to be multi-customer.
It's the difference between a hoop in your back yard and a NBA Stadium
Is this a common architectural issue for self-hosted options from SaaS companies?
As in, a huge SaaS company offers their product for self-hosting to individual companies, but it's not practical to self-host because the code is highly specialized for supporting hundreds of companies instead of just one? And it's hard to have an architecture that works well for just one and for hundreds?
It's more about scale than tenancy. Not many SaaS companies offer such an option in the first place but it is typical that the in-house product is the priority and the architectural decisions are made with that in mind firstly, and self-hosting second if at all.
For example Sentry requires ClickHouse, Postgres, Kafka, and Redis presumably because they were the right tools for their needs and either they have the resources to operate them all or the money to buy the managed options from vendors.
Also, the main concern people have with hosting Sentry is the sheer number of containers required but most of them are just consumers for different Kafka queues which again is presumably this way because Sentry ops prefers it this way, whether it be for fine tuning the scaling of each one or whatever the reason.
What makes sense for a SaaS company rarely translates to sensible for self-hosting.
I use both hosted and self hosted sentry. I prefer hosted it's less to manage but self hosted isn't too awful as long as you can budget the resources. If you just want the team sentry plan it's going to be cheaper to use hosted. I would only self host if you had to for legal/compliance reasons. In the many years I've been managing sentry we've only had it truly crap out once and that was an upgrade from 9 to their new date versioning basically the whole hosted method changed and it was just easier to start over from scratch.
This is also my opinion of Backstage. It's positioned as this easy-to-use open source tool for IDPs - but in reality, it's a product that needs a team to maintain it constantly.
It has its own conference now! Definitely not a signal for something that is reliable set-it-and-forget-it tech.
So many services are going this route. I remembered Reddit had an install script and almost everything worked out of the box 10-15 years ago on a Ubuntu VPS.
HyperDX is a great alternative to Sentry, much easier to self-host, and also open source.
It's relatively new and did take some tinkering to make it work properly, so I wrote a short article about it: https://weberdominik.com/blog/self-host-hyperdx
But the feature set and user experience is great!
I'm a big fan of self-hosting, but it requires the right infrastructure. You need a robust virtualization solution like Proxmox, equipped with sufficient CPU cores and RAM. For instance, allocating 16GB of RAM shouldn't be an issue if your server has 128GB to share.
In my experience, solutions like Mailcow, which involve multiple services and containers (such as SMTP, IMAP, Redis, SSO, webmail, Rspamd, etc.), work very well. I have extensive experience running these systems, performing backups, restoring data, and updating their ecosystems.
Additionally, I've had a positive experience setting up and running a self-hosted Sentry instance with Docker for a project that spanned several years. However, this experience might be somewhat outdated, as it was a few years ago.
I am running Mailcow on just some VM with docker. No Proxmox needed. So far it works.
> First, there’s a signal being given about the complexity and fragility of the software. A 16GB RAM requirement just screams “this is a big, complex piece of software that will break in mysterious ways”. Not very scientific, I know, but based on experience.
This lines up with my experience self hosting a headless BI service. In "developer mode" it takes maybe 1GB RAM. But as soon as you want to go prod you need multiple servers with 4+ cores and 16GB+ RAM that need a strongly consistent shared file store. Add confusing docs to the mix, mysterious breakages, incomplete logging and admin APIs, and a reliance on community templates for stuff like k8s deployment... it was very painful. I too gave up on self hosted.
In most corporate projects that I worked with, the day it went to production started the clock to disband the core team and optimizations were not on the table. When the product works, they move on. The only area where performance optimization is a mandatory step is video games, but that is where your clients are external. Most business software is in the "finally, but barely works" state and new features beat performance improvements.
This is caused by short sighted management that need to deliver and move on. "Long term" is a contradiction with their business model. In this case "long term" means "after product launch".
I worked in a startup around 2019-2020 where we started using Sentry for distributed tracing. It was a company in the IOT space and we had a lot of events going through. The bill started going in the $3-5000 range just from the tracing, so we decided to host it on our own. When looking at the sheer complexity of the project, I was flabbergasted. We need a name for managing the infrastructure of company based open source, self-hosted solutions. Often times, the right choice would be to choose a different, simpler, open source solution. In this case there are some neat distributed tracing solutions, that are easier to manage.
A good APM tool I’ve been using for a few years is Apache Skywalking: https://skywalking.apache.org/
It might not do everything Sentry does but it definitely has helped with tracking down some issues, even production ones and runs in a fairly manageable setup (in comparison to how long even the Sentry self-hosted Docker Compose file is).
What’s more, if you want, you can even use regular PostgreSQL as the backing data store (might not be quite as efficient as ElasticSearch for a metrics use case, but also doesn’t eat your RAM like crazy).
Any product that uses etcd is immediately off my radar[1]; it is nice of them to have a whole page dedicated to how to babysit the PoS https://skywalking.apache.org/docs/skywalking-banyandb/next/...
1: yes, I'm a huge k8s fanboi and yes I long every day for them to allow me to swap out etcd for something sane
That’s kind of the beauty of it: you can opt for a different storage solution too https://skywalking.apache.org/docs/main/v10.2.0/en/setup/bac...
Personally, no hate towards their BanyanDB but after getting burnt by OrientDB in Sonatype Nexus, I very much prefer more widespread options.
At first glance it's not immediately obvious to me, why would you pick Sentry or Bugsink over something included in the Grafana stack? What's the use use
I haven't used Sentry's obserability portions, but the error capture has very strong deduplication built into it, and can optionally integrate with your SCM, CI, and issue tracker, if one wishes. Thus, the flow can go like:
- error happens, can be attributed to release 1.2.3
- every subsequent time that error happens to a different user, it can track who was affected by it, without opening a new error report
- your project can opt-in to accepting end-user feedback on error: "please tell us what you were doing when this exploded, or feel free to rant and rave, we read them all"
- it knows from the stack trace that the error is in src/kaboom/onoz.py line 55
- onoz.py:55 was last changed by claude@example.com last week, in PR #666
- sentry can comment upon said PR to advise the reviewers of the bad outcome
- sentry can create a Jira with the relevant details
- claude.manager@example.com can mark the bug as "fixed in the next release", which will cause sentry to suppress chirping about it until it sees a release 1.2.4
- if it happens again it will re-open the prior error report, marking it as a regression
Unless you know something I don't, Grafana does *ABSOLUTELY NONE* of that
We've just applied a helm chart a while back. It just works. We maybe had like a few incidents over the years, requiring stuff like Kafka queues to be wiped.
The argument that you have to read a sh script doesn't make sense to me. Are you gonna read source code of any software is referenced in this script or any you download too? No? What's the difference between that and a bash script, at the end of the day both can do damage.
We used the helm chart but things didn't get updated often enough to keep our container security stuff happy.
Helm is a huge pain in the butt if you have mitigation obligations because the overall supply chain for a 1-command install can involve several different parties, who all update things at different frequencies :/
So chart A includes subchart B, which consumes an image from party C, who haven't updated to foobar X yet. You either need to wait for 3 different people to update stuff to get mainline fixed, or you roll up your sleeves and start rebuilding things, hosting your own images and forking charts. At first you build 1 image and set a value but the problem grows over time.
If you update independently you end up running version combinations of software that the OG vendor has never tested.
This is not helm's fault of course; it's just the reality of deploying software with a lot of moving parts.
Rereading that section, I'd agree it's probably not the best-argued point because it implies security concerns... I guess what I'm saying is: for something I'm setting up to keep around for a while, I'd like to know a bit what's in the package before I deploy it. In that sense, the shell script serves as a table of contents... and if the table of contents is 800 lines, that makes me wonder how many moving parts there are and how many things might break at inconvenient times because of that.
For me I would just run it on a clean cluster/VM somewhere (to be destroyed after that) just to see what happens. If you have no local resources to spare, an hour of even very high end (to save time) VMs/cluster at a provider e.g. AWS costs next to nothing
That solution didn't apply for me at the time, since I was in an environment that combined security-consciousness with thick layers of bureaucracy, meaning that hardware came at a premium (and had to be on premise).
Sure, but I'm not suggesting running there, just testing there. We also have to run in specific providers in specific location, but nothing stops us from renting a clean large VM in AWS for an hour or two, for testing stuff without using any customer data. Hell, that costs pretty much nothing so if my employer didn't allowed it, I would just pay with my own money - it's much better for your work efficiency to work out the kinks of this without having to do 10 cleanups after failed deployment, it's much easier than to just delete a VM.
Oh and the most difficult part when setting up, from what I remember, was setting up GitHub SSO and GitHub and Slack integration as it wasn't well documented.
> It’s a ton of docker containers. Things will fail randomly or maybe with a lot of traffic, don’t remember well.
maybe they should put in a system to monitor the docker containers.
Software that's only available as x86 binaries and a whole system that needs 16 gigs of memory? It's not surprising that it comes from the corporate world and not the open source world.
I respect the fact that the Sentry people are honest about their teetering tower.
As a computer user I do not allow connections to sentry.io
For example, Firefox tries to connect
Heh, saying that in a thread about self hosting Sentry is pretty funny :-)
---
IMHO, the RUM tooling[1] is the worst offender, tracking mouse movement and keystrokes. At least Sentry is usually just trying to report JS kabooms
1: https://docs.datadoghq.com/real_user_monitoring/session_repl... and https://docs.newrelic.com/docs/browser/browser-monitoring/br... et al
That's the inevitable evolution of any popular open-source product/company. Light and enterprise-ready is hardly compatible, and you can expect this from any project that has a cloud offering.
Not only open source.
I don't have servers like this but the author makes it easy to understand, and it applies to a lot of other things.
The Comparative Costs picture does tell it all.
The purpose of so much software is to reduce human and machine costs through time, and this apparently turned out to do just the opposite, apparently after long-term testing under real-world conditions.
Could be an unsurmountable fundamental structure of technical debt or something like that which metastasizes.
Then this, anyone could say about anything:
>I’m not going to run a piece of software that requires 16GB of RAM, has a complex installation script, and is known to be a pain to maintain. I’m not going to recommend it to anyone else either.
Easier said than done, sometimes it's the only option.
It's a "script". Maybe that's why you have to "rehearse" it more than once before you barely get it right, and then you might have to really go the extra mile (and have a bit of good fortune) before you can achieve a "command performance".
How do you think it feels these days to have to settle for this kind of thing in expensive proprietary software too?
Maybe Sentry don't want you to have self-hosted version?
I actually learn from this post that there is a self-hosted version...
What gave you that impression?
I didn't downvote you but as one metric their docker-compose is 500 lines long https://github.com/getsentry/self-hosted/blob/25.4.0/docker-... as compared to back in the good-old days when it was basically python, redis, and postgres https://github.com/getsentry/self-hosted/blob/9.1.2/docker-c...
This isn't okay - the author is selling their own alternative to Sentry, 'reusing' Sentry's open-source client SDK's, while spreading FUD about self-hosting Sentry.
I've been self-hosting Sentry for over 10 years: Sentry is installed by running `git clone`, editing one or two configuration files, and running ./install.sh. It requires 16GB RAM if you enable the full feature set. That includes automatically recording user sessions on web or mobile for replay, and support for end-to-end tracing (so seeing which database queries your api asks for in response to a button tap in your mobile app).
Sentry has a wonderful self-hosting team that's working hard to make Sentry's commercial feature set available, for free, to everyone that needs a mature error tracking solution. You can talk to them on discord and they listen and help.
Just for transparency, are you by any chance a member of this self-hosting team or a sentry employee? Is it a coincidence that your keybase name is georghendrik according to your profile, and the painter _Georg Hendrik_ Breitner painted a picture called "Sentry"? https://www.demesdagcollectie.nl/en/collection/hwm0055
This is some Olympics level of badly coping with valid and welcome criticism.
Georg Hendrik = "George Henry", pretty common name. The fact that Google returned a result when you searched "Georg Hendrik Sentry" should not be considered weird.
I was born 400 years ago in the Highlands of Scotland. I am Immortal, and I am not alone. Now is the time of the Gathering, when the stroke of the sword will release the power of the Quickening. In the end, there can be only one.
It seems like all the FUD in the article is copy-pasted from Sentry's own docs, though, no? And assuming Sentry's SDKs are open source and licensed appropriately (which seems to be the case), there's no issue (legal or moral) with directing users to use the Sentry SDK to communicate with a different, compatible product.
OP built a product because they were frustrated by Sentry's seeming hostility toward self-hosting. It doesn't feel like OP decided to build a competing product and then thought it would be a good marketing strategy to falsely push the idea that Sentry is difficult to self-host.
FWIW I've never self-hosted Sentry, but poking around at their docs around it turns me off to the idea. (I do need a Sentry-like thing for a new project I'm building right now, but I doubt I'll be using Sentry itself for it.) Even if it's possible to run with significantly less than 16GB (like 1GB or so), just listing the requirements that way suggests to me that running in a bare-bones, low-memory configuration isn't well tested. Maybe it's all fine! But I don't feel confident about it, and that's all that matters, really.
> OP built a product because they were frustrated by Sentry's seeming hostility toward self-hosting.
this is indeed the timeline.
TBH most of the FUD in the OP is straight from Sentry's own website.
Regarding using the SDKs, I'm telling my users to take Sentry at their word when they wrote "Permission is hereby granted, free of charge [..] to deal in the Software without restriction, including without limitation the rights to use"
I am not sure why there isn't a even lightweight sentry endpoint that does
I give those requirements to LLM, and I get a pretty much working rust implementation after few tweaks. It uses <5M ram idle.That covers the most basic feature that Sentry has (one-off error reporting, with rudimentary statistics). Sure, if that's enough for you then go ahead, but for most people using Sentry that's only the tip of the iceberg.
I was looking into moving away from managed Sentry for GDPR reasons. I was only using Sentry for notifications about unhandled exceptions in my app. All the fancy APM features I did not use. Grouping of similar exceptions, pretty display of tracebacks were nice but not essential.
In my Django app I wrote a logging handler that stores the log records (including traceback) in a database table. I can inspect the log records through Django admin, and a cron job sends me daily emails saying "X new log records in the last 24 hours" so I know to check them out. And that's it :-)
Of course, this does a lot less than Sentry, and has various limitations (e.g. what if the error is about the database being down...), but it fits my needs.
BTW, IIUC, Sentry in its early beginnings was also doing just that – logging to database: https://github.com/dcramer/django-db-log
TL;DR: Not so much "gave up" as "never tried" because I was scared away by 16GB RAM requirement.
FWIW, we've been self-hosting 2 instances (and purchase a third hosted instance), for, it looks like 8 years now, and it's had a few small bumps but hasn't been too bad. Our instances are running with 20GB of RAM, so pretty small. ~90% of updates have gone smoothly, which leaves 10% of updates that have had some sort of problem. Solutions have been found in the issue tracker in all our cases. We are a little afraid of doing updates, but do them every couple of months.
Sentry is an amazing piece of software and it is fantastic that they offer a self-hosing version, and all things considered it is a fairly easy self-host.
FYI: Just did the upgrade from 25.2.0 to 25.4.0 and it went smoothly.
You're right, but so am I... :-)
I "gave up" from the perspective of returning to Sentry after a couple of years, and finding an entirely different beast from the tool I loved before. At that point I indeed didn't make it past Sentry's own FUD.
I find it very funny that the post is basically complaining about fearmongering by the maker of Sentry to scare people into not hosting themselves.
And then he does the exact same thing, on behalf of Sentry.
I hope he got paid for this. Otherwise it would just be sad.
I doubt he would have been paid to promote a competing service ..
If that were to happen I think the blog post (guaranteed virality) would just write itself...
It's wild that Sentry themselves are spreading literal FUD about about bad it is to self-host their own product. So confounding that they're so ready to shit on their own product.
> I’m not going to run a piece of software that requires 16GB of RAM, has a complex installation script, and is known to be a pain to maintain.
This has to be self-hosted eventually either by you or Sentry themselves so the full cost of this is coming down somewhere. The planet is round and all that and there's no getting away from these inefficiencies, but you can kick the proverbial can down the road.
Also, they are incentivized to make the open product as hard to use as possible so color me surprised the self-hosted edition is hard to operate.
> This has to be self-hosted eventually either by you or Sentry themselves
Not really. There are alternatives. Which seems to be the point of the article.
You mean like having a different product internally, super optimized etc? Or what do you mean?
The post itself is advocating for an alternative, so nobody really has to rely on or run Sentry.
Ah, I see. I meant that when using Sentry someone will have to host it. Not that we can't live our lives without it.
Sure, but "when using Sentry" is an artificial constraint. There's no reason to constrain yourself that way when there are alternatives that might be sufficient for your needs, that don't require a beefy machine and a lot of maintenance effort when self-hosting.
> color me surprised
last time a similar thing was pointed out on HN Armin Ronacher (former Sentry) came out and pointed at the following issue:
https://github.com/getsentry/team-ospo/issues/232
but that is now closed with
> We are no longer working on this effort for now
I no longer work at Sentry but the complexity of operating Sentry is something that we still worked on when I was there. There is no secret incentive to make Sentry hard to operate. You have to consider that engineers run a version of Sentry on their machines too and any complexity ends up there too.
One of the biggest pain points of running Sentry is the number of containers and a lot of this comes from how Sentry works with Kafka and Rabbit. That pain point is actively being addressed by moving to a virtualized system [1] though I am not sure how long it will take to actually adopt this.
[1]: https://github.com/getsentry/taskbroker/
I don't think Sentry is being evil or is trying to make it harder to self host, but I do think that Sentry doesn't have a big incentive to spend a lot of effort making it easier to self host. Again, I don't think that's evil. I think it's entirely reasonable and ok, even if it does make life harder for some people.
Maybe this virtualized system will make things easier. If so, that's great. But if it ends up not working out, or if it does, but over the longer term things get more difficult again, I think that's still just kinda how things happen sometimes, and that's ok.
One of Sentry's goals is for Sentry themselves to operate it as a hosted cloud service. Architecture decisions made to further that goal can naturally and reasonably be at odds with another goal to make it simpler to self host. Sometimes things can't be one-size-fits-all.
I already mentioned this elsewhere here but Sentry does have an incentive to improve this. That’s because the complexity creates challenges in the operation of smaller instances and during development.
> There is no secret incentive to make Sentry hard to operate
It's frustrating when half the comments on a company that dares to open their product is always about how they are obviously intentionally very evil to not do it perfectly/for totally free/with 0 friction/etc.
How entitled have we become lol?
but in this case the order of events was not "awesome product that was open-sourced as-is later" but "OS product that became harder and harder to install over time"
Products get bigger and more complex over time, so "harder and harder to install, configure and manage" is somewhat normal. The money spent on free or "community" edition is very difficult to justify, so it slowly degrades with time until it becomes unusable and people move on to something else. It is part of the normal lifecycle of software.
[dead]
Hi Armin :-)
I've seen you make this point on similar posts in the past, and I believe that you believe it.
The counterpoint would be that "the purpose of a system is what it does"...
A fairly easy answer is "one of the purposes of Sentry is running Sentry-as-a-service, at large scale". Which comes with compromises that make smaller deployments more "bloated" too, but is a very different thing than intentionally choosing to bloat it to make it hard to run.
It's definitely true that what Sentry is today, is a function of operating it at scale. Thankfully one of the intended purposes of Sentry is also that it scales down much better than today [1]. That it does not is also something that Sentry as a company is not happy with which is why there are desires to change that, which will also benefit self hosted.
[1]: this is because a) developing on sentry has become harder and harder on local machines and b) operating single-tenant installations for some customers brings many of the same challenges that people on self-hosted run into. c) we want more regions of Sentry and smaller ones (such as Europe today) have higher fixed costs than ideal.
Yes.. we're getting there. "one of the purposes of Sentry is running Sentry-as-a-service". Why? To make the dwarves happy? It's making money. And with it comes a whole truckload of very obvious incentives. Notice I don't say anyone is intentionally being evil. That's quite unnecessary.
From the Sentry homepage
> Application monitoring software considered "not bad" by 4 million developers.
Sounds pretty bad to me
As a software developer I can say that it is not bad. It's pretty great, actually.
But the infrastructure behind sounds complicated and fragile.
No, its complicated and scales to servicing every customer in the world. Thats not the same thing.
Doesnt mean the complaints about self-hosted arent valid, but "literally has to scale to the most insane volumes of data" and "is not good software" are two different things.
We're building a cloud service at the end of the day - its a lot easier to optimize a multi-tenant install than it is a single-tenant install, and that complexity shows up in the self-hosted repo. Can't really avoid it.