What the author is saying in practice is: it's nice to have Meta level funding for effectively infinite distributed storage and compute. In other areas of the industry where you want to stay profitable, metrics stay metrics, because you're submitting 2 integers for an aggregate, rather than tens of megabytes of JSON logs per minute. Then you want to retrieve this in seconds on demand... so that's either preprocessing (ongoing compute) or some serious parallel IO.
I'm glad the author likes wide events for everything. I do too. But once you try to justify the cost, you understand why it's metrics, logs and (often sampled) traces.
All true and fair. But I do like having the instrumentation API in your code speak wide events. Then have your telemetry config decide what you want to transmit as a verbatim big event, what you want to sample/transmit as a metric, or a trace (or whether you, at the moment of transmission, need to go grab a few things out of the local tracing context stack and use them for preprocessing), and so on.
If you do all that on the client side (not in a sidecar, ideally: actually in your client code when deciding what to transmit) you have a fairly nice way to temporarily turn on e.g. rich trace recording to hunt a bug, or start spraying event bodies to a local file when debugging a troublesome service instance, and so on. The rest of the time, things can transmit a few big payloads and a lot of small metrics, and things stay cheap.
In other words, it's easy to go from "recorder.record_rich_typed_event(...)" to capturing any or all of events, traces, metrics, or logs. It's not easy to go from e.g. "metrics.record_integer_metric("somemetric", 15)" to recording a richer, more complex event in a pinch.
> I’m going to blame Open Telemetry. Yes, it’s powering the modern observability stack and yet I blame it for the mass confusion
Of course. It’s the classic Internet dev thing of “make a blog post about how using tool X well was a boon to what you were trying to do” and a mass of people walk away with “tool X magically solves things in its default configuration without me learning anything”.
The nice bit is this post will then lead to a bunch of companies’ log storage prices spiking and then a post about how you don’t need wide events trending.
Is it really that cost prohibitive? Actual numbers would be useful to make sense of this discussion.
The question comes down to value. If you’re never actually using the data, fine. But once you need it, and you don’t have it, might the cost sometimes be greater than it would have been to preserve it?
The choice of how much telemetry to keep seems similar to how much you want to spend on an insurance policy. The same kinds of risk evaluation apply.
Nope, as long as you didn’t also read the blog posts about how your tiny site needs multi-region support and failover and went with a cloud provider that is happy to ingest whatever you got even if you didn’t explicitly sign up for that fee.
Ludicrous take. I'm tired of people who aren't experts on observability having such strong opinions.
Having used scuba at meta, I can authoritatively say it's bottom-tier quality. Especially in the era of having competent LLMS able to write sophisticated queries.
Genuine expert level observability is being able to query things like "Of all the users who updated versions of the APP in the last week, what % of them had fewer errors after the update (per hour of usage) than before"
I don't see how you can possibly consider scuba to be bottom-tier. I have used nothing even remotely close to scuba at any other company. For example, Amazon Athena is a joke compared to it.
Your example seems like it would be a great system to have access to, but it seems like that question would be formed after using a more general system like Scuba where you can iterate on the data rapidly.
A metric is a measure that can be aggregated over time. You can derive a metric from logs (count) but that doesn't make them the same thing. Spans have start and end times and parent pointer. Logs are arbitrary data, but can be correlated, can have application/org defined structure that's beyond the scope of the observability stack.
You could draw a class diagram with "Wide" - or whatever - as the base class, metric, span, log subclasses. Metric class requires an additional metric field, span requires start, end, parent, log is just open.
As someone who has been on that side of things for 10 years, can we stop building really complicated stuff that requires this pile of shit and dedicated staff to run it.
The whole otel, logging and metrics thing is based around "record everything" and dig through the pile of shit looking for things that might be helpful. Fine until you have 50TB of logs/traces online which is our problem. Oh that and the 50 gig prometheus instance. 99.99999% of all information we record is entirely and utterly useless and costs us money
Make simpler things, log only exceptional things, use the rule of silence.
The latter is the problem. When your observability stack takes 20% of your cluster operation costs, 40% of your license costs and 25% of your salary costs, you’ve been licking toads.
Objectively we did the same job without it 15 years ago under the same traffic conditions.
Very ignorant take. If all of us had infinite infrastructure there would indeed be no reason to separate the three at generation, but for those that have to actually count the costs, paying for servers to store that amount of duplication is real issue, and even more (for some reason, by order of magnitude or more) if you're using one of big observability providers.
It is divided mostly because database to store metrics, and database to store essentially freeform documents are very different products and need different features from query language, and having "just metrics" allows to store them with compression reaching 1byte per metric data point or less.
Logs and traces I admit, split here is unnecessary, and it should be just presentation difference. But metrics is often just unrelated to any single event summary of the state of the system. Sure you can just log every single log entry with request metadata and get request rate, percentiles etc. by calculating the events, but that is just too wasteful for most.
Wide events are great if you don't face an error that can only be exposed by a more detailed metric, log or trace. And that in my experience is a big if.
Meh. This is just structured logging with some post processing. You can definitely derive metrics and traces based on them, but doing so is more expensive in many ways.
It's very obviously LLM-generated if you've been reading some of the many other LLM-generated comments you'll find at the bottom of the comments on almost all other submissions to this website. Pangram says 100% LLM, too, but you don't need Pangram to see this, just compare the way it's written to all the other ones.
The Scuba example is convincing for debugging. What the post doesn’t cover is cost. If you keep raw wide events instead of pre-aggregating, how do you handle storage and query cost at scale? Is sampling the main lever, or do teams still keep cheap metrics for alerting and use events only for investigation?
The whole wide events thing is Honeycomb’s marketing pitch. It’s essentially offering developer velocity, ie you don’t have to think about what should be a metric in advance. They have an open source project called Refinery which is a smart sampling proxy that samples successful requests but always forwards errors, slow queries, etc.
You can do anything you like with any approach but if they are mostly about ad hoc queries, dumping everything into BigQuery as a wide event can be the most economical. However if you want to run alerting, SLOs, etc with frequent queries over aggregated metrics, the best bet is using OTel Collector and metrics store of your choice like Prometheus or Mimir.
Most companies could save a bundle on observability if they were to move to a hybrid that includes both of those approaches instead of cloud provider logging, log-based metrics, and alerting.
What the author is saying in practice is: it's nice to have Meta level funding for effectively infinite distributed storage and compute. In other areas of the industry where you want to stay profitable, metrics stay metrics, because you're submitting 2 integers for an aggregate, rather than tens of megabytes of JSON logs per minute. Then you want to retrieve this in seconds on demand... so that's either preprocessing (ongoing compute) or some serious parallel IO.
I'm glad the author likes wide events for everything. I do too. But once you try to justify the cost, you understand why it's metrics, logs and (often sampled) traces.
All true and fair. But I do like having the instrumentation API in your code speak wide events. Then have your telemetry config decide what you want to transmit as a verbatim big event, what you want to sample/transmit as a metric, or a trace (or whether you, at the moment of transmission, need to go grab a few things out of the local tracing context stack and use them for preprocessing), and so on.
If you do all that on the client side (not in a sidecar, ideally: actually in your client code when deciding what to transmit) you have a fairly nice way to temporarily turn on e.g. rich trace recording to hunt a bug, or start spraying event bodies to a local file when debugging a troublesome service instance, and so on. The rest of the time, things can transmit a few big payloads and a lot of small metrics, and things stay cheap.
In other words, it's easy to go from "recorder.record_rich_typed_event(...)" to capturing any or all of events, traces, metrics, or logs. It's not easy to go from e.g. "metrics.record_integer_metric("somemetric", 15)" to recording a richer, more complex event in a pinch.
The nice bit is this post will then lead to a bunch of companies’ log storage prices spiking and then a post about how you don’t need wide events trending.
Is it really that cost prohibitive? Actual numbers would be useful to make sense of this discussion.
The question comes down to value. If you’re never actually using the data, fine. But once you need it, and you don’t have it, might the cost sometimes be greater than it would have been to preserve it?
The choice of how much telemetry to keep seems similar to how much you want to spend on an insurance policy. The same kinds of risk evaluation apply.
Nope, as long as you didn’t also read the blog posts about how your tiny site needs multi-region support and failover and went with a cloud provider that is happy to ingest whatever you got even if you didn’t explicitly sign up for that fee.
Ludicrous take. I'm tired of people who aren't experts on observability having such strong opinions.
Having used scuba at meta, I can authoritatively say it's bottom-tier quality. Especially in the era of having competent LLMS able to write sophisticated queries.
Genuine expert level observability is being able to query things like "Of all the users who updated versions of the APP in the last week, what % of them had fewer errors after the update (per hour of usage) than before"
That will never happen in a "wide event."
I don't see how you can possibly consider scuba to be bottom-tier. I have used nothing even remotely close to scuba at any other company. For example, Amazon Athena is a joke compared to it.
Your example seems like it would be a great system to have access to, but it seems like that question would be formed after using a more general system like Scuba where you can iterate on the data rapidly.
A metric is a measure that can be aggregated over time. You can derive a metric from logs (count) but that doesn't make them the same thing. Spans have start and end times and parent pointer. Logs are arbitrary data, but can be correlated, can have application/org defined structure that's beyond the scope of the observability stack.
You could draw a class diagram with "Wide" - or whatever - as the base class, metric, span, log subclasses. Metric class requires an additional metric field, span requires start, end, parent, log is just open.
(2024)
prev. submission - https://news.ycombinator.com/item?id=39529775
As someone who has been on that side of things for 10 years, can we stop building really complicated stuff that requires this pile of shit and dedicated staff to run it.
The whole otel, logging and metrics thing is based around "record everything" and dig through the pile of shit looking for things that might be helpful. Fine until you have 50TB of logs/traces online which is our problem. Oh that and the 50 gig prometheus instance. 99.99999% of all information we record is entirely and utterly useless and costs us money
Make simpler things, log only exceptional things, use the rule of silence.
I’m having trouble understanding your take here. Why is recording logs with embedded metrics at full resolution a problem, assuming you can afford to?
The latter is the problem. When your observability stack takes 20% of your cluster operation costs, 40% of your license costs and 25% of your salary costs, you’ve been licking toads.
Objectively we did the same job without it 15 years ago under the same traffic conditions.
Those are completely made-up numbers. We don’t pay anywhere near that. About 10% of our cloud spend is on observability, and I targeted 15%.
That or you’re getting suckered.
Do you have a steaming pile of 1000 microservices?
Unfortunately yes, but we’re working to improve that :-)
How do you implement that? Would you like to explain, I am very interested!
How do you decide what is exceptional when you have only silence to compare to?
When something shits the bed in your software, write a simple log out. Otherwise STFU.
Resume driven development baby!
Vendor PR driven development.
wide events feel like structured logs, great for high cardinality data but not so much for time series
The time series are derived from the events.
Very ignorant take. If all of us had infinite infrastructure there would indeed be no reason to separate the three at generation, but for those that have to actually count the costs, paying for servers to store that amount of duplication is real issue, and even more (for some reason, by order of magnitude or more) if you're using one of big observability providers.
It is divided mostly because database to store metrics, and database to store essentially freeform documents are very different products and need different features from query language, and having "just metrics" allows to store them with compression reaching 1byte per metric data point or less.
Logs and traces I admit, split here is unnecessary, and it should be just presentation difference. But metrics is often just unrelated to any single event summary of the state of the system. Sure you can just log every single log entry with request metadata and get request rate, percentiles etc. by calculating the events, but that is just too wasteful for most.
With AWS this compares to EMF with (additional) properties?
You get logs with full content + metrics you can query.
Or am I missing something?
Potential PII nightmare. You're now potentially circumventing access controls from sensitive data.
As of a few years ago when I left, this was completely solved by the data catalog. The columns are typed and require identification.
Tbh, it doesn't seem much different than having to watch out for not stuffing PII in logs.
Besides, with wide events, it seems possible to apply data lineage ideas and mark certain fields as sensitive and restrict access to them.
To me this seems easier than the alternative model (logs, metrics, traces), in that you only have one abstraction to contemplate handling PII in?
In what way is this worse than the alternative?
Wide events are great if you don't face an error that can only be exposed by a more detailed metric, log or trace. And that in my experience is a big if.
Meh. This is just structured logging with some post processing. You can definitely derive metrics and traces based on them, but doing so is more expensive in many ways.
Why is 77rushi77's comment flagged + dead? Seemed like a good question.
Edit: Looks like it's been restored.
Looks like it's because of:
> Don't post generated text or AI-edited text. HN is for conversation between humans.
But who can tell, you could ask the mods (email found in the guidelines)
Doesn't seem particularly robot-generated to me, nor to the classifier I made for a related purpose: https://xkqr.org/aicomment/#lang=none&prior=50&c=Pc_BbcMwEAT...
It's very obviously LLM-generated if you've been reading some of the many other LLM-generated comments you'll find at the bottom of the comments on almost all other submissions to this website. Pangram says 100% LLM, too, but you don't need Pangram to see this, just compare the way it's written to all the other ones.
Then that check accounts history and tell me if you see a pattern or not, this one is really obvious.
Not this bullshit again
The Scuba example is convincing for debugging. What the post doesn’t cover is cost. If you keep raw wide events instead of pre-aggregating, how do you handle storage and query cost at scale? Is sampling the main lever, or do teams still keep cheap metrics for alerting and use events only for investigation?
The whole wide events thing is Honeycomb’s marketing pitch. It’s essentially offering developer velocity, ie you don’t have to think about what should be a metric in advance. They have an open source project called Refinery which is a smart sampling proxy that samples successful requests but always forwards errors, slow queries, etc.
You can do anything you like with any approach but if they are mostly about ad hoc queries, dumping everything into BigQuery as a wide event can be the most economical. However if you want to run alerting, SLOs, etc with frequent queries over aggregated metrics, the best bet is using OTel Collector and metrics store of your choice like Prometheus or Mimir.
Most companies could save a bundle on observability if they were to move to a hybrid that includes both of those approaches instead of cloud provider logging, log-based metrics, and alerting.