This fails to acknowledge that synthesized noise can lack the detail and information in the original noise.
When you watch a high-quality encode that includes the actual noise, there is a startling increase in resolution from seeing a still to seeing the video. The noise is effectively dancing over a signal, and at 24 fps the signal is still perfectly clear behind it.
Whereas if you lossily encode a still that discards the noise and then adds back artificial noise to match the original "aesthetically", the original detail is non-recoverable if this is done frame-by-frame. Watching at 24 fps produces a fundamentally blurrier viewing experience. And it's not subtle -- on old noisy movies the difference in detail can be 2x.
Now, if h.265 or AV1 is actually building its "noise-removed" frames by always taking into account several preceding and following frames while accounting for movement, it could in theory discover the signal of the full detail across time and encode that, and there wouldn't be any loss in detail. But I don't think it does? I'd love to know if I'm mistaken.
But basically, the point is: comparing noise removal and synthesis can't be done using still images. You have to see an actual video comparison side-by-side to determine if detail is being thrown away or preserved. Noise isn't just noise -- noise is detail too.
Grain is independent frame-to-frame. It doesn't move with the objects in the scene (unless the video's already been encoded strangely). So long as the synthesized noise doesn't have an obvious temporal pattern, comparing stills should be fine.
Regarding aesthetics, I don't think AV1 synthesized grain takes into account the size of the grains in the source video, so chunky grain from an old film source, with its big silver halide crystals, will appear as fine grain in the synthesis, which looks wrong (this might be mitigated by a good film denoiser). It also doesn't model film's separate color components properly, but supposedly that doesn't matter because Netflix's video sources are often chroma subsampled to begin with: https://norkin.org/pdf/DCC_2018_AV1_film_grain.pdf
Disclaimer: I just read about this stuff casually so I could be wrong.
> Grain is independent frame-to-frame. It doesn't move with the objects in the scene (unless the video's already been encoded strangely)
That might seem like a reasonable assumption, but in practice it’s not really the case. Due to nonlinear response curves, adding noise to a bright part of an image has far less effect than a darker part. If the image is completely blown out the grain may not be discernible at all. So practically speaking, grain does travel with objects in a scene.
This means detail is indeed encoded in grain to an extent. If you algorithmically denoise an image and then subtract the result from
the original to get only the grain, you can easily see “ghost” patterns in the grain that reflect the original image. This represents lost image data that cannot be recovered by adding synthetic grain.
> If you algorithmically denoise an image and then subtract the result from the original to get only the grain, you can easily see “ghost” patterns in the grain that reflect the original image. This represents lost image data that cannot be recovered by adding synthetic grain.
The synthesized grain is dependent on the brightness. If you were to just replace the frames with the synthesized grain described in the OP post instead of adding it, you would see something very similar.
> So long as the synthesized noise doesn't have an obvious temporal pattern, comparing stills should be fine.
The problem is that the initial noise-removal and compression passes still removed detail (that is more visible in motion than in stills) that you aren't adding back.
If you do noise-removal well you don't have to lose detail over time.
But it's much harder to do streaming-level video compression on a noisy source without losing that detail.
The grain they're adding somewhat distracts from the compression blurriness but doesn't bring back the detail.
>The grain they're adding somewhat distracts from the compression blurriness but doesn't bring back the detail.
Instead of wasting bits trying to compress noise, they can remove noise first, then compress, then add noise back. So now there aren't wasted bits compressing noise, and those bits can be used to compress detail instead of noise. So if you compare FGS compression vs non-FGS compression at the same bitrate, the FGS compression did add some detail back.
I imagined that at some point someone would come up with the idea “let’s remove more noise to compress things better and then add it back on the client”. Turns out, it is Netflix (I mean, who else wins so much from saving bandwidth).
Personally I rejected the idea after thinking about it for a couple of minutes, and I’m not yet sure I was wrong.
The challenge with noise is that it is actually cannot be perfectly automatically distinguished and removed from what could be finer details and textures even in a still photo, not to mention high-resolution footage. If removing noise was as simple as that, digital photography would be completely different. If you have removed noise, you can’t just add back missing detail later—if you could, you would not have removed it in the first place (alas, no algorithm is good enough, and even human eye can be faulty).
That might be true; however, if this takes hold I would be surprised if they choose to keep producing and shipping the tasty grain high fidelity footage.
Considering that NR is generally among the very first steps in development pipeline (as that’s where it is the most effective), and the rest of dynamic range wrangling and colour grading comes on top of it, they might consider it a “waste” to 1) process two times (once with this new extreme NR, once with minimal NR that leaves the original grain), 2) keep around both copies, and especially (the costliest step) to 3) ship that delicious analog noise over Internet to people who want quality.
I mean, how far do we go? It’ll take even less bandwidth to just ship prompts to a client that generates the entire thing on the fly. Imagine the compression ratios…
> Grain is independent frame-to-frame. It doesn't move with the objects in the scene (unless the video's already been encoded strangely). So long as the synthesized noise doesn't have an obvious temporal pattern, comparing stills should be fine.
Sorry if I wasn't clear -- I was referring to the underlying objects moving. The codec is trying to capture those details, the same way our eye does.
But regardless of that, you absolutely cannot compare stills. Stills do not allow you to compare against the detail that is only visible over a number of frames.
People often assume noise is normal and IID but it usually isn't. It's s fine approximation but isn't the same thing, which is what the parent is discussing.
Here's an example that might help you intuit why this is true.
Let's suppose you have a digital camera and walk towards a radiation source and then away. Each radioactive particle that hits the CCD causes it to over saturate, creating visible noise in the image. The noise it introduces is random (Poisson) but your movement isn't.
Now think about how noise is introduced. There's a lot of ways actually, but I'm sure this thought exercise will reveal to you how some cause noise across frames to be dependent. Maybe as a first thought, think about from sitting on a shelf degrading.
It's the same thing. Yes, not related to the movement of the camera, but I thought that would be easier to build your intuition about silver particles being deposited onto film. You make in batches, right?
The point is that just because things are random doesn't mean there aren't biases.
To get much more accurate, it helps to understand what randomness actually is. It is a measurement of uncertainty. A measurement of the unknown. This is even true for quantum processes that are truly random. That means we can't know. But just because we can't know doesn't mean it's completely unknown, right? We have different types of distributions and different parameters in those distributions. That's what we're trying to build intuition about
I think you've missed the point here: the noise in the originals acts as dithering, and increases the resolution of the original video. This is similar to the noise introduced intentionally in astronomy[1] and in signal processing[2].
Smoothing the noise out doesn't make use of that additional resolution, unless the smoothing happens over the time axis as well.
Perfectly replicating the noise doesn't help in this situation.
Your first link doesn't seem to be about introducing noise, but removing it by averaging the value of multiple captures. The second is to mask quantizer-correlated noise in audio, which I'd compare to spatial masking of banding artifacts in video.
Noise is reduced to make the frame more compressible. This reduces the resolution of the original only because it inevitably removes some of the signal that can't be differentiated from noise. But even after noise reduction, successive frames of a still scene retain some frame-to-frame variance, unless the noise removal is too aggressive. When you play back that sequence of noise-reduced frames you still get a temporal dithering effect.
Here's[1] a more concrete source, which summarizes dithering in analog to digital converters as follows:
With no dither, each analog input voltage is assigned one and only one code. Thus, there is no difference in the output for voltages located on the same ‘‘step’’ of the ADC’s ‘‘staircase’’ transfer curve. With dither, each analog input voltage is assigned a probability distribution for being in one of several digital codes. Now, different voltages with-in the same ‘‘step’’ of the original ADC transfer function are assigned different probability distributions. Thus, one can see how the resolution of an ADC can be improved to below an LSB.
In actual film, I presume the random inconsistencies of the individual silver halide grains is the noise source, and when watching such a film, I presume the eyes are doing the averaging through persistence of vision[2].
In either case, a key point is that you can't bring back any details by adding noise after the fact.
One thing worth noting is that this extra detail from dithering can be recovered when denoising by storing the image to higher precision. This is a lot of the reason 10 bit AV1 is so popular. It turns out that by adding extra bits of image, you end up with an image that is easier to compress accurately since the encoder has lower error from quantization.
The AR coefficients described in the paper are what allow basic modeling of the scale of the noise.
> In this case, L = 0 corresponds to the case of modeling Gaussian noise whereas higher values of L may correspond to film grain with larger size of grains.
To illustrate the temporal aspect: consider a traditional film projector. Between every frame, we actually see complete darkness for a short time. We could call that darkness "noise", and if we were to linger on that moment, we'd see nothing of the original signal. But since our visual systems tend to temporally average things out to a degree, we barely even notice that flicker (https://en.wikipedia.org/wiki/Flicker_fusion_threshold). I suspect noise and grain are perceived in a similar way, where they become less pronounced compared to the stable parts of the signal/image.
Astrophotographers stack noisy images to obtain images with higher SNR. I think our brains do a bit of that too, and it doesn't mean we're hallucinating detail that isn't there; the recorded noise - over time - returns to the mean, and that mean represents a clearer representation of the actual signal (though not entirely, due to systematic/non-random noise, but that's often less significant).
Denoising algorithms that operate on individual frames don't have that context, so they will lose detail (or will try to compensate by guessing). AV1 doesn't specify a specific algorithm to use, so I suppose in theory, a smart algorithm could use the temporal context to preserve some additional detail.
The noise does not contain a signal, does not dance over it, and is not detail. It is purely random fluctuations that are added to a signal.
If you have a few static frames and average them, you improve SNR by retaining the unchanged signal and having the purely random noise cancel itself out. Retaining noise itself is not useful.
I suspect the effect you might be seeing is either just an aesthetic preference for the original grain behavior, or that you are comparing low bandwidth content with heavy compression artifacts like smoothing/low pass filtering (not storing fine detail saves significant bandwidth) to high bandwidth versions that maintain full detail, entirely unrelated to the grain overlaid on top.
> If you have a few static frames and average them, you improve SNR by retaining the unchanged signal and having the purely random noise cancel itself out.
That's exactly the point of GP though. Even though each individual frame might be almost indistinguishable from random noise you can still extract patterns over time. This is also the case if you don't average the frames in software but let the viewer's brain do it. If you just remove all "noise" from each frame and then add random noise back those patterns will be lost.
In practice you won't have static frames but also movement so recovering the signal from the noise becomes a lot more complicated.
Anything with a pattern is by definition not noise, and the comment was that noise had signal. If you remove all noise, no signal or pattern is lost by definition.
However, the issue is that lossy compression removes various types of minute detail, smoothing surfaces to reduce the amount of data that has to be stored, be it noise "grain" or skin pores, according to compression settings. Storing the original noise as it was would basically make any compression impossible.
Denoising generally removes signal too. Removing noise and reconstituting similar noise to maintain the apparent qualities of an input can help compression, but you are also cutting out true details (typically fine detail).
The effect GP is pointing out is how denoisers damage detail, which is true. This detail can persist over multiple frames, which is why many denoisers include a temporal comparison component to mitigate the damage, but you still lose detail.
Definitely, but that's a result of compressing (with any algorithm) past a bitrate that would allow detail to be retained with high frequency details including noise being the first to go, not the result of noise having signal as was stated and more importantly unrelated to film grain synthesis which is about adding something new on the output side.
I love this concept/principle, one similar example I often bring up when I talk about machine learning, is comparing how a human would analyse night footage from a camera, and how a ML algorithm can pick up things no human would think about, even artifacts from the sensors which can be used as features. Noise is rarely ever just noise.
Some of the new 4K discs use DRR and the denoising process seems to remove the pores on people's faces occasionally, leaving actors looking like their face is made of wax.
Don't want to sound too snarky, but aren't you just saying that good denoisers must be temporal in addition to spatial? Something like an improved V-BM3D (cf https://arxiv.org/abs/2001.01802) and even traditional V-BM3D works fine even on non-Gaussian noise.
Together with external noise generation (something like https://old.reddit.com/r/AV1/comments/r86nsb/custom_photonno...) that'd support external denoising (pass the reference and denoised video to get the grain spec), the whole FGS thing would be much more interesting.
There are definite philosophical questions over the merits of adding noise, but the problem with their example here is their denoising process appears to excessively blur everything, so both it and the synthesized grain image look noticeably less sharp than the source. The grain itself also looks too much like basic noise, and not really grain like.
> both it and the synthesized grain image look noticeably less sharp than the source
That's true, but at a given bitrate (until you get to very high bitrates), the compressed original will usually look worse and less sharp because so many bits are spent trying to encode the original grain. As a result, that original grain tends to get "smeared" over larger areas, making it look muddy. You lose sharpness in areas of the actual scene because it's trying (and often failing) to encode sharp grains.
Film Grain Synthesis makes sense for streaming where bandwidth is limited, but I'll agree that in the examples, the synthesized grain doesn't look very grain-like. And, depending on the amount and method of denoising, it can definitely blur details from the scene.
It seems like a shame that they didn’t include a screenshot of the original (with natural grain), after suffering from low-bitrate streaming. Aka the actual baseline.
I can see why they want to compare against the actual local copy of the video with the natural grain. But that’s the perfect copy that they can’t actually hope to match.
But why do they need to emphasize it even more than the examples they gave? The "AV1 with FGS @ 2804 kbps" already looks as good or better than the "AV1 (without FGS) @ 8274 kbps", so it'll definitely look better than AV1 without FGS at an even lower bandwidth.
I think the distinction here is, they provide the "regular" stream, and the FGS stream noting that it's much smaller yet looks similar. What they don't have is a lower-bandwidth "regular" one, what 2000-or-so kbps looks like without FGS.
Since the beginning of film editors have added tricks in post.
I would love for them to provide an option to view it with film simulation vs without.
One of my favorite movies of all time, The Holdovers, did film simulation extremely well. It's set in the '70s so it attempts to look like a movie of that era.
It looked great to me, but if you're an actual film nerd you're going to notice a lot of things aren't exactly accurate.
Maybe in the near future we'll see Netflix being able to process some post effects on the client. So if you're color blind, you get a mode for that. If you don't want fake grain you can turn it off.
mpv can already do this: `--vf=format:film-grain=no` turns off grain synthesis. There are also people making custom shaders for things like emulating CRT monitors (originally for retro gaming, but I see there are also mpv versions).
Holdovers was pretty great. Sorta like an 80’s school comedy but with the perspective of the adults included, making it a totally different type of movie.
AV1 has tunable FGS levels, and to my eye they went very slightly higher than they should have (though there are tradeoffs; at some bitrates the blurring+renoising is so much better than the other visual artifacts you will otherwise get, that you do want it that high).
A few things to note:
- still-frames are also a mediocre way to evaluate video quality.
- a theoretically perfect[1] noise-removal filter will always look less detailed than the original source, since your brain/eye system will invent more detail for a noisy image than for a blurry image.
1: By which I mean a filter that preserves 100% of the non-grain detail present, not one that magically recovers detail lost due to noise.
These days, when we see noise/grain in an end product it has likely been added in post-production. So, ideally, studios would provide distributors with a noiseless source plus grain synthesis parameters. Bonus: many viewers would welcome an option to turn it off.
> provide distributors with a noiseless source plus grain synthesis parameters.
What parameters would that be? Make it look like Eastman Ektachrome High-Speed Daylight Film 7251 400D? For years, people have taken film negative onto telecines and created content of grain to be used as overlays. For years, colorists have come up with ways of simulating the color of specific film stocks by using reference film with test patterns that's been made available.
If a director/producer wants film grain added to their digital content, that's where it should be done in post. Not by some devs working for a streaming platform. The use of grain or not is a creative decision made by the creators of the work. That's where it should remain
> If a director/producer wants film grain added to their digital content, that's where it should be done in post. Not by some devs working for a streaming platform. The use of grain or not is a creative decision made by the creators of the work. That's where it should remain
Why? If you're spending a significant chunk of your bits just transmitting data that could be effectively recreated on the client for free, isn't that wasteful? Sure, maybe the grains wouldn't be at the exact same coordinates, but it's not like the director purposefully placed each grain in the first place.
I recognize that the locally-produced grain doesn't look quite right at the moment, but travel down the hypothetical with me for a moment. If you could make this work, why wouldn't you?
--------
...and yes, I acknowledge that once the grain is being added client side, the next logical step would be "well, we might as well let viewers turn it off." But, once we've established that client-side grain makes sense, what are you going to do about people having preferences? Should we outlaw de-noising video filters too?
I agree that the default setting should always match what the film maker intended—let's not end up with a TV motion smoothing situation, please for the love of god—but if someone actively decides "I want to watch this without the grain for my own viewing experience"... okay? You do you.
...and I will further acknowledge that I would in fact be that person! I hate grain. I modded Cuphead to remove the grain and I can't buy the Switch version because I know it will have grain. I respect the artistic decision but I don't like it and I'm not hurting anyone.
> Why? If you're spending a significant chunk of your bits just transmitting data that could be effectively recreated on the client for free, isn't that wasteful? Sure, maybe the grains wouldn't be at the exact same coordinates, but it's not like the director purposefully placed each grain in the first place.
I'm sorry your tech isn't good enough to recreate the original. That does not mean you get to change the original because your tech isn't up to the task. Update your task to better handle the original. That's like saying an image of the Starry Night doesn't retain the details, so we're going to smear the original to fit the tech better. No. Go fix the tech. And no, this is not fixing the tech. It is a band-aid to cover the flaws in the tech.
> I'm sorry your tech isn't good enough to recreate the original. That does not mean you get to change the original because your tech isn't up to the task.
The market has spoken and it says that people want to watch movies even when they don't have access to a 35mm projector or a projector than can handle digital cinema packages, so nobody is seeing the original outside a theater.
Many viewers are bandwidth limited, so there's tradeoffs ... if this film grain stuff improves available picture quality at a given bandwidth, that's a win. IMHO, Netflix blogs about codec things seem to focus on bandwidth reduction, so I'm never sure if users with ample bandwidth end up getting less quality or not; that's a valid question to ask.
The differences are actual film grain vs some atrocious RGB noise artificially added by the streamer. How is that unclear? What else could we be talking about?
In theory though, I don't see any reason why client-side grain that looks identical to the real thing shouldn't be achievable, with massive bandwidth savings in the process.
It won't be, like, pixel-for-pixel identical, but that was why I said no director is placing individual grain specks anyway.
Let's be clear. The alternative isn't "higher bandwidth" it's "aggressive denoising during stream encode". If the studio is adding grain in post then describing that as a set of parameters will result in a higher quality experience for the vast majority of those viewing it in this day and age.
If the original is an actual production shot on film, the film grain is naturally part of it. Removing it never looks good. If it is something shot on a digital camera and had grain added in post, then you can go back to before the grain was added and then do it client side without degradation. But you can never have identical when it originated on film. That's like saying you can take someone's freckles away and put them back in post just rearranged and call it the same.
> once we've established that client-side grain makes sense, what are you going to do about people having preferences?
I already normalize audio (modern US produced media is often atrocious), modify gamma and brightness, and probably some other stuff. Oh and it's not as though I'm viewing on a color calibrated monitor in the first place.
The purists can purchase lossless physical copies. The rest of us would benefit from such improvements.
Netflix has their own in-house studio, right? The encoding and lossy compression is going to happen anyway. It seems like an easy win, for their directors to provide a description of the grain they want, so it can be replicated on the user side.
what does having an in-house studio have to do with it? they stream more content than just their own, and so they would not have creative license to alter content. they would only have some type of distribution license to stream the content as provided
Because they previously did not do commercials their original TV shows were not written with pauses every few minutes. They have approved cameras. They spend heavily on movie star salaries and skimp on set production.
And yet here we are: DNR -> fancy grain -> DNR -> basic, approximated grain. Because noise doesn’t compress. And you get compression artifacts even in Blu-ray releases. What’s the point of applying fancy grain when what a lot viewers end up seeing is an ugly smudge?
Because it looks amazing in the editing studio. Just like the sound mix is incredible on the Atmos monitors in the sound mixing room, even though the home viewers have a soundbar at best and tiny stereo speakers in a flat panel typically. The dynamics and dialog channel will be fucked. But that’s user error.
What, cramped into a room with tons of other people who can't help themselves from making noise or moving in front of you while you only have limited and overpriced snack options. As nice as a big screen and professional sound equipment is the downsides make the experience strictly worse than a big OLED + half decent soundbar.
You must live alone. Even at home, you have people called family or friends that do not necessarily sit idle while watching TV. Watching with kids at home is worse than at the movies because they are at home and don't have the same social rules to follow. People at home are also not under the social rules of not using their devices, and second screening has become the norm.
Yes, a theater probably holds more people than your typical viewing experience at home. Unless you go to the movies during the week and avoid crowds. The last movie I saw at the theater was on a Tuesday after opening weekend as 4pm. There might have been 2 other people in the entire theater. It was amazing.
Yeah I always roll my eyes when people get so mad about compressed (as in reduced dynamic range) audio. I just want to watch / listen to stuff without annoying my neighbors and don't particularly care whether or not the volume of gunshots is realistic.
Theatrical release qualifies for certain awards and shiny statues. That's their concern. If a streaming platform wants to give them enough cash to beat out projected box office earnings, then they'll take it if they don't have any grandiose visions of golden statues.
At some point, it would not surprise me for Netflix to require this to be provided. While not negating what you concluded, I just think the impetus for the result is important to distinguish.
Currently, in order to deal with noisy masters, Netflix has to either:
1. Denoise the master, then add AV1 FGS metadata to tell players how to reconstruct the noise in the master (which is what the blog post is about) to get back the original image the director saw and approved
2. Do nothing (which is what they were doing), and let some of the noise get blurred or erased by the quantization step of the encoding process, or worse, burn shittons of coding bits trying to describe the exact noise in the frame, which hurts visual quality of the things people actually look at
All of these imply changes to the image that the director decided on to get around the underlying fact that deliberately adding noise to an image is, from a signal processing perspective, really stupid. But if we are going to do it, we can at least ensure it happens as far down the chain as possible so that Netflix's encoding doesn't destroy the noise. That's the idea you responded to: have the production company deliver a master with FGS metadata instead of baked-in film grain.
Agree, as someone who has spent way too much time studying the way motion picture film looks up close, this isn’t very realistic looking. It’s really just a form of dithering.
that's an understatement. it just looks like RGB noise effect was added. film grain does not look like RGB noise. to me, film grain is only one part of what gave film the film look. the way the highlights bloom rather than clip. it also was more natural/organic/some descriptive other than the ultrasharp of modern digital acquisition. using some SoftFX or Black Mist type filters help, but it's just not the same as it is a digital vs analog type of acquisition. all of these attempts at making something look like it's not just keep falling down in the same ways. but hey, there's a cool tech blog about it this time. film grain filters have been around for a long time, yet people just don't care for them. even in Blu-ray time frame, there was attempts at removing the grain in the encode and applying it in playback. Netflix isn't coming up with anything new, and apparently nothing exciting either based on the results.
Agree, I really prefer luminance noise when simulating film grain rather than treating RGB so independently - the RGB here looks wrong to me. If you are thinking about film, you wouldn't model it as pure photon/detector noise, because film has some crystalline structure and usually there is some correlation between the channels. (Some of the best grain is on B&W films though, which have wonderful structure and statistics.)
I only have their single sample in the blog post to go by, but the original source frame also looks just like RGB noise? And the recreation looks really close if I swap between them. Looks fine to me.
So as long they're analyzing the style of grain in the source properly, which the technical part of the post mentions they do...
The "at scale" part is the real story here. Film Grain Synthesis has been available in the usual AV1 encoders for a while, but required some amount of manual tweaking to avoid creating problems, meaning it was only used in production when you had a very limited catalog, or for particularly important titles. They do not provide a lot of details here about how they overcame those problems, but it is nice to see it being deployed more broadly.
To the comments hating on grain: everything naturally has some amount of noise or grain - even the best digital sensors. Heck, even your eyes do. It's useful beyond just aesthetics. It tends to increase perceived sharpness and hides flaws like color banding and compression artifacts.
That's not to say that all noise and grain is good. It can be unavoidable, due to inferior technology, or a result of poor creative choices. It can even be distracting. But the alternative where everything undergoes denoising (which many of our cameras do by default now) is much worse in my opinion. To my eyes, the smoothing that happens with denoising often looks unrealistic and far more distracting.
My issue is that grain is good based on the creative decisions of the creators of the content. It is not something that a group of nerds compressing 1s and 0s should be making
I agree. However, let's look at it practically. Let's assume someone is watching content streamed on a low bandwidth connection. As a content creator, what version of the compressed content would you rather your audience experience:
a) Compressed original with significant artifacts from the codec trying to represent original grain
b) A denoised version with fewer compression artifacts, but looks "smoothed" by the denoising
c) A denoised version with synthesized grain that looks almost as good as the original, though the grain doesn't exactly match
I personally think the FGS needs better grain simulation (to look more realistic), but even in its current state, I think I'd probably go with choice C. I'm all for showing the closest thing to the author's intent. We just need to remember that compression artifacts are not the author's intent.
In an ideal world where we can deliver full, uncompressed video to everyone, then obviously - don't mess with it at all!
For content that we're concerning ourselves with this level of detail, I'd prefer the old iTunes method of prefetching the file and not stream it. For typical YT content, streaming is fine. For typical sitcom or other content, streaming is fine. For something like a feature that I'm so concerned about the details of grain, I have no problem downloading to play a local version. No, not a torrent.
We live in a world of resource constraints. The nerds compressing 1s and 0s are very much concerned about creative intent. They noticed that compression wrecks that creative intent and wanted to better communicate it in a world with bandwidth limitations.
Case in point: the HBO into animation. It uses static from the old analog days. It looks like shit even at 4k because random noise is impossible to compress without the exact strategy outlined here (stripping it out and rendering it later).
> This grain, formed from tiny particles during the film’s development, is more than just a visual effect. It plays a key role in storytelling by enhancing the film’s depth and contributing to its realism.
I never understood the “grain = realism” thing. my real eyes don’t have grain. I do appreciate the role of grain as an artistic tool though, so this is still cool tech
The article points out the masking effect of grain, which hides the fake-looking compression artifacts, and also the familiarity/nostalgia aspect. But I will offer an additional explanation.
Look around you: nearly all surfaces have some kind of fine texture and are not visually uniform. When this is recorded as video, the fine texture is diminished due to things like camera optics, limited resolution, and compression smoothing. Film grain supplies some of the high frequency visual stimulus that was lost.
Our eyes and brains like that high frequency stimulation and aren't choosy about whether the exact noise pattern from the original scene is reproduced. That's why the x265 video encoder (which doesn't have grain synthesis since it produces H.265 video) has a psy-rd parameter that basically says, "try to keep the compressed video as 'energetic' as the original, even if the energy isn't in the exact same spot", and even a psy-rdoq parameter that says, "prefer higher 'energy' in general". These parameters can be adjusted to make a compressed video look better without needing to store more data.
They definitely do at night when it's dark out. There's a kind of "sparkling" or "static" that comes in faint light.
Fortunately, our eyes have way better sensitivity than cameras. But the "realism" just comes from how it was captured using the technology of the day. It's no different from phonograph hiss or the way a CRT signal blurs. The idea is to be "real" to the technology that the filmmaker used, and the way they knew their movie would be seen.
It's the same way Van Gogh's brush strokes were real to his paintings. You wouldn't want his oil paintings sanded down to become flat. It's the reality of the original medium. And so even when we have a digital print of the film, we want to retain as much of the reality of the original as we can.
Your Van Gogh analogy makes sense for old movies. It doesn't quite explain why we're still adding grain to new movies, except for those few which are purposefully evoking older movies.
Even modern cameras have grain. If you need to integrate your scene with motion graphics, background replacement, or vfx, you'll need to remove grain on part of the image, edit it, add the original grain back where possible and synthesize new grain elsewhere.
Often it can also make sense to modify the grain for aesthetics. Denoising usually produces a less detailed result, but what you can do is denoise only the color channels, not the brightness channel. Brightness noise looks normal to us, while color noise tends to look very artificial. But by keeping the brightness noise, you avoid losing detail to the denoiser.
We don't use obvious grain, usually. That's generally precisely to evoke something about the past -- for flashbacks, a period look, etc. A sense of grittiness like 70s movies, etc.
On the other hand, a small amount of constant grain or noise is intentionally often introduced because otherwise images feel too static and end up looking almost fake. Similarly, dithering is intentionally added to audio, like mastering CD's or tracks. It helps prevent artifacts in video and in audio.
...what makes the frame rate thing especially weird is how many TVs ship with motion smoothing enabled by default, which looks so much worse than actual high frame rate content would.
I don't like it at all. But someone must, or the TV manufacturers wouldn't do it.
Part of me thinks that it's only the people who know what "frame rate" means who prefer low frame rates, and a majority of the general public actually prefers high frame rates but lacks the terminology or knowledge to express this desire.
People are always trying to rationalize and justify aesthetic preferences. The depth and nuance of your understanding of a thing will change how you perceive variations of that thing, whether it's guitar tonewoods, style of music, types of paint, flavor of beer, or the grain in film. If you know a lot about a subject, you can tell a lot about the history of a thing, and that's going to change how you feel about a thing.
A child watching a Buster Keaton skit and gasping and giggling and enjoying it is going to have a different subjective aesthetic experience of the media than a film critic who knows exactly what type of film and camera were used, and what the meaning of all the different abstractions imply about the scene, and the fabric of Keaton's costume, and so on, and so forth.
Subjective aesthetic preferences are in the realm of cognition - we need a formal theory of intelligence mapped to the human brain, and all of these subjective phenomena collapse into individualized data processing and initial conditions.
There's something about film grain contrasted against clean cel animation which might make it easier for people to suspend disbelief. They are conditioned to think that absence of grain is associated with unreal animation, particular types of media, and CGI. Home video and news and so forth had grain and low quality, so grain gets correlated with "real". In my view, there's nothing deeper than that - we're the product of our times. In 40 years, media will have changed, and it may be that film grain is associated with surrealism, or edited out completely, as it's fundamentally noise.
The way I see it is that grain makes the film look more detailed than it really is, it can also hide compression artefacts and blurriness.
I don't know the psychovisuals behind that. Maybe it adds some high frequencies that compression often washes out, or maybe acts like some kind of dithering.
As for your eyes, I am pretty sure that they have grain, that's how quantum physics work, you just don't perceive it because your brain filters it out. But again, I don't know how it interacts with film grain.
Video signal without the noise or grain is annoying to watch as it makes everything in the ”out of focus” zone look smooth blurry. Your eyes want to focus yet it is an illusion of depth without an actual depth. Noise texture emphasizes that this is just a 2D plane after all so your eyes can rest and the viewer doesn’t feel like they need glasses. This is just my theory of it based on observation. No research behind it.
This reminds me of modern windows having fake panes. They’re just strips that are applied to give the impressions that there are multiple smaller panes because people are used to that and it feels “correct”.
I have to imagine past glassmakers would have been absolutely enthralled by the ability we now have to make uniform, large sheets of glass, but here we are emulating the compromises they had to make because we are used to how it looks.
> They’re just strips that are applied to give the impressions that there are multiple smaller panes because people are used to that and it feels “correct”.
It is more than just 'feeling correct': windows and their various (sub-)elements that make them up (can) change the architectural proportions and how the building is perceived as a whole:
It is similar with columns: they're not just 'tall-and-narrow', but rather have certain proportions and shapes depending on the style and aesthetic/feeling one wishes to convey:
I strongly doubt that multiple smaller panes would have ever become a common style if we could have always made large glass panes. This is a perfect example of people becoming very used to a style forced by a technological limitation that is emulated even after the limitation doesn't exist.
It used to be a bigger deal (when digital cameras started being used) since people felt like digital video didn't look real/as good - movies shot on film were generally better looking (as crews were used shooting with it and digital video wasn't as sophisticated as today) and HAD grain.
It might be that there is a large part of the population that still has that association.
Cinephiles are also more likely to watch older (i.e. with grain) movies that ARE well shot and beautiful (which is why they are classics and watched by cinephiles) and not see bad film movies, only the cream of the crop, while being exposed to the whole gamut of quality when watching todays movies shot digitally. Would reinforce that grain = good while not being necessarily the case - and their opinion might be heard more than gen pop.
At any rate, it can be a neat tool to lower sharpness!
23.976 fps has been put on a pedestal as the "correct" look. Just look at the reaction to The Hobbit. However it does provide some objective advantages. 60 fps requires more lighting. Adding more lights means more electric setup and heat for actors in heavy makeup and costume. In post production that's more frames to edit.
23.976fps is only "correct" when telecining to a 59.94Hz format. 24fps round is the "correct" format in terms of actual filmmaking. I don't know of many films specifically shot at 23.976fps.
It's funny, because I've always had a subtle visual snow that looks like a film grain on top of everything I see, and for a long time I thought everybody had it and that that's why artificial grain is added, to make pictures appear more realistic.
I've since learned that not everybody sees the world like I do, but I still do love to see grain and noise in pictures.
Only RGB noise I often find dreadfully ugly when looked at up close, which is a shame, since that is exactly what most color cameras include.
film grain adds realism in the same way that high frame rate films look wrong or vinyl sounds "warmer" or tube guitar amps sound "better" - It is what we are used to.
grain and 24fps and widescreen trigger certain contextual emotions around the movie-watching experience. remove them and your brain contextualizes the video very differently.
this is likely the result of ~100 years of film-based filmmaking and projection. hell, we still call it filmmaking.
Yes, it is only the result of familiarity. We could gradually increase the frame rate of movies made in a year by 1 fps per year and then no one would even notice after 24 years every new movie would be 48fps.
no i will be watching 24fps films for the remainder of my life, which may be 40 more years. if all new films went up 1fps per year i would still go out to the movies in 20 years and be like “wtf is this crap?”
it would take a generation or more to eradicate this cultural context. casablanca is never going to be in 48fps.
Where did I refuse to watch anything? No amount of this attitude is going to make Casablanca or The Godfather or Pulp Fiction or Infinity War or Star Wars or Fight Club or 2001 or Top Gun be in any framerate other than 24fps.
If they start making films in higher frame rates today, that won’t change that fact. Do you think people are going to stop watching films from the 50s, 60s, 70s, 80s, 90s, and 2000s?
It would take a concerted effort, beginning now, at least an entire generation (30-50 years) to remove or reduce the cultural impact of the 24 frames per second widescreen tradition. No such effort is presently underway, because higher framerates don’t help movies’ visual storytelling, and they may hurt it.
The Hobbit was Peter Jackson’s attempt. It went nowhere.
"no i will be watching 24fps films for the remainder of my life, which may be 40 more years. if all new films went up 1fps per year i would still go out to the movies in 20 years and be like “wtf is this crap?”"
The Hobbit looked great at 48fps. Higher FPS failed only because reactionary people as yourself irrationally rejected it because "it looks weird!".
You need to understand that 24fps is a compromise chosen to save film. It is the bare minimum frame rate to have smooth motion under most but not all circumstances. It really hurts action films because anything moving too fast has excessive motion blur.
Permanently rejecting faster frame rates for movies is like rejecting printed text in favor of handwritten manuscripts. You are rejecting a technological advance for extremely arbitrary reasons. If 23 or 25 or 30fps had become the standard you would be insisting that it was just as special.
Grain = realism because real captured grain isn't total random noise. It's authentic noisy data. It's part of captured scene. It adds subtle tiny but real detail to the scene. Unless I am corrected here and that real grain is also total random noise.
Film grain can create stochastic resonance with the underlying ground truth. In practice, this can improve the perceived image quality over having none.
Honestly to me it reads like a solution looking for a problem. I’ve never considered non deterministic imperfections that happen during recording of the movie to be essential to storytelling.
> Picture this: you’re watching a classic film, and the subtle dance of film grain adds a layer of authenticity and nostalgia to every scene
It just adds visual noise that obscures details of the authentic scene, and nothing prevents nostalgia from being tied to many of the more prominent visual cues like old actors or your own old memories from when you watched it first...
> contributing to [film's] realism
But there is no grain in reality, so it does the opposite
Otherwise I'm glad AV1 marches along and instead of wasting bitrate encoding visual garbage has an algorithmic replacement mechanism- which also means you could turn it off easier.
Documentaries might care about accurately representing reality. For every other cinematic genre, "authenticity" is not an inherent goal. If film grain is part of the director's vision, then that's just as valid as a choice to have dramatic non-diagetic music playing in the background of a scene (which is highly inauthentic, but also highly effective at evoking emotion, which is the point of art).
I don't see how that is supposed to be relevant to the article's claim that grain adds authenticity and realism. Someone claiming that non-diegetic music adds to authenticity would also be weird.
I think what the author meant is that film grain is part of the authentic visual aesthetic of cinema produced using celluloid materials. A de-noised image is therefore inauthentic as that is contrary to the appearance of the original projected image or source/master elements.
> But there is no grain in reality, so it does the opposite
Well ackchually -- illumination is inherently random, so all time-bounded captures of a scene (including what your eyes do) are subject to shot noise: https://en.wikipedia.org/wiki/Shot_noise
You misackchuallied grain with any noise, for example, the bottom right square of the image at the wiki page is not grainy despite being technically shot-noisy
> But there is no grain in reality, so it does the opposite
The author did not do a particularly good job of selling the merits of film grain. I am not sure what he meant by that but I will say that it does increase the perceived sharpness of the image as well as perceived detail, even if it is just an illusion. The cinematographer Steve Yedlin describes it as giving the viewer’s eyes something to “latch” onto.
When you talk on a cellphone the codec AMR-WB nominally captures 50 Hz - 7000 Hz. However that's only on the optional highest bitrate 23.85 Kbps. The most common bitrate 12.65 Kbps only goes up to 6400 Hz and synthesizes 6400 - 7000 Hz from lower frequencies and noise as it sounds better than not having the noise!
We got rid of interlaced TV. We got rid of the low-bandwidth color changes of NTSC. (NTSC video can only go through the color gamut about ten times per scan line, which is why colored text on NTSC is truly awful.) We got rid of projector film transport jank. (Except for really bad Youtube videos which use a filter to produce fake film transport jank.) Nobody wants any of those back.
It's a bit frustrating that the footage is first shot, then denoised in post, then renoised in post, then denoised in encoding and then renoised at decoding.
Only if you are aware of it, which 99.9% of people consuming video content are not. It’s simply an unimportant implementation detail (from a viewer’s perspective who doesn’t really care about bitrate-as-cost).
As a mirrorless photographer I use a lot of processing to create a unique look. If I at all can, I set the aperture really small, use denoising, sharpening and color grading. If people think my photo came out of an old book that’s great.
I have looked at various “film look” filters and so far haven’t found one I like. When I think “film grains” I think of irregular shaped crystals, but most of the fake film grain I’ve seen makes a square pixel grain which doesn’t convince me at all. My understanding is the grain already exists before the image is shot in the sense that the films has irregular “pixels” which respond more or less because of shot noise and variation of the grain and I haven’t found anything that really simulates this.
Everything is fake now. I want a technology which works with a raw film scans, not even compressing them to JPEG, which is a 1st step in loosing the details BTW. Motion detection, key frames, delta frames - fine. But with a lossless video. On a Blu Ray off course, i don't care much about streaming.
A 4K/24p film encoded with Apple ProRes 4444 XQ (not even ProRes RAW) is 716 GB per hour, so you would need to swap a total of 30 Blu-ray discs once every 4 minutes in order to watch a 2 hour movie.
I wonder how much you'd get with such a technology... truly uncompressed 4K video you're talking about something on the order of a few terabytes for a 90-minute movie, so way way bigger than the biggest 4K Blu-ray discs. Lossless compression would get you under that number, but far enough to matter?
The key point some people are missing here is that removing noise from a video before compression makes it more compressible. So if you compare a video compressed without denoising/degrading at bitrate X to a video that was filtered and then compressed at bitrate X with "fake noise" added later, then the latter will have edges and details that the former doesn't.
The principle of post-processing videos to add noise or sharpening is something that desktop media players have supported for years, some are even reasonably good at it.
Exactly. Everyone is going on about grain quality but are in fact comparing wheat and spelt (if I may).
What matters to Netflix is the bandwith/quality ratio, that’s it. Removing noise before and adding it in software at playback time will produces a significantly better result, at similar bit rate. Because compressing noise produces horrible results.
Between 24fps and film grain people are way too attached to fundamentally inferior technology. With how strongly people resist frame rates faster than 24 I'm surprised people accepted color and sound, which were much bigger changes.
>This grain, formed from tiny particles during the film’s development, is more than just a visual effect. It plays a key role in storytelling by enhancing the film’s depth and contributing to its realism
That's one hell of an exaggeration. But even if that was true, I don't see how removing it, then adding fake on in would help.
That entire sentence really has a "written by ChatGPT" smell. >It's not just X. It's Y".
I think there is a bit more to it. For example when developing a game when CRTs were popular they were using CRTs to view their game and making decisions based on what they saw on the CRT. If you display the same game with perfect square pixels it looks different. If the developers were viewing square pixels when developing the game they would make different decisions.
I don't think this conflicts with what I'm saying; I've seen what you talk about -- and yet in modern days people will emulate the "square pixel bad style" regardless.
Agree. Purely opining, but I assume that it's because of the emotional connection that artistic media has on people, despite the flaws.
People remember the emotions the artwork engendered, and thus the whole work is associated with the feelings, flaws and all. If the work is particularly widely known, the flaws can become a stand-in for the work itself.
I see this in video games - I'm fond of the NES-era "flaws" and limitations (palette limits, sprite limits, sound channel limits), but less connected to the Atari 2600 or SNES/PS1/NDS/etc flaws. Shovel Knight is charming; A Short Hike, while great, doesn't resonate on a style level.
Even if you ignore all of the emotional attachment aspects, I think you would expect this to happen simply because people who are good at their craft will learn how to work with the shortcomings of their tools. If your cameras unavoidably add grain to everything you shoot you'll find ways to make that look good, and some of the things you come up with might not work without grain.
I've noticed this in a few things the last few years. The top and bottom of the shot are out of focus and it is super distracting to me. Maybe it is meant to draw the eye to the middle of the frame.
Yup, notice that as well. A blurry vignette. Needle sharp in the center, but objects that are the same distance are blurry at the edges of the frame. At first I thought they use vintage lenses, but it's probably done in post.
I sometimes do something similar on photos, where I darken the edges a bit.
Not necessarily, there are still lenses being produced today with vignetting, especially wide open.
Avoiding vignetting is a trade off between complexity, cost, weight, and other optical flaws. That said modern optical design and manufacturing has made it a lot easier to have nearly flawless (clinical) designs.
Or Snyder's terrible zombie movie Army of the Dead where he uses lenses with very shallow depth of field that makes almost everything look out of focus. It is very annoying.
I don't think it's just that here. Our brains simply prefer high entropy signals over low entropy ones. It's why mobile phones (and digital cameras before them) use sharpen filters to the point where the result looks unnatural.
I remember years ago when digital projection was just becoming the norm. I saw a movie (I don't remember which one) and during the opening scene I remember thinking "why are the credits jittering around? Oh. This is actually being projected on film!"
I'm in my early 50s so I remember film quite well. Just like vinyl or cassettes, I ain't going back and unless it is an artistic choice I don't want films to emulate what I consider to be an inferior technology.
There's an influx of high-profile directors/films right now and in pipeline filmed for IMAX (F1: The Movie I think, Mission Impossible, etc) and Christopher Nolan's Odyssey coming next year shot entirely on IMAX film with newly developed smaller/quieter cameras made to accomplish it.
I've read that a 15-perf 65mm IMAX negative shot with slower film stocks is "virtually grainless", even when viewed on a 70ft screen. Grain is apparently noticeable in IMAX films when large/fast stocks are used and pushed toward their limits, and (of course) when smaller-format film stocks have been blown up.
I like this. Not really because I feel modern media should have added grain, but because for older media this is a method to get closer to the original but at much lower bitrates without excessive smoothing. What’s not to like?
Also, the author had me at God of Gamblers 2. So good. I will take him up on his recommendation to rewatch.
The original grain that is captured is actually a detail and not total random noise. I believe you can make up the vague sense of original scene if you could somehow extract that grain/noise alone.
It's like reducing an image to tiny dots with dithering (reminds of Atinkson dithering). Those grains are not a noise, they are a detail, actual data. That's why real grain looks good IMO.
This is just a fact of lossy compression: you want to throw away information that contributes less to the perception of the video so that you can describe it with fewer bits of information.
There are two possible advantages for this kind of grain synthesis. For Netflix, they could produce the same perceived quality at lower bitrates, which reduces costs per view and allows customers with marginally slow connections to get a higher quality version. For a consumer, the advantage would be getting more non-grain detail for a fixed bitrate.
You are right that if you subtract the dentists frame from the raw one, showing only the estimated noise, you would get some impression of the scene. I think there’s two reasons for this. Firstly, the places where the denoiser produced a blurry line that should be sharp may show up as faint lines. I don’t think this is ‘hidden information’ so much as it is information lost to lossy compression. In the same way, if you look at the difference between a raw image and one with compression, you may see some emphasized edges due to compression artefacts. Secondly, the less exposed regions of the film will have more noise so noisiness becomes a proxy for darkness, allowing some reproduction of the scene. I would expect this detail to be lost after adjusting for the piecewise linear function for grain intensity at different brightness levels.
Perhaps a third thing is the level of noise in the blacks and the ‘grain size’ or other statistical properties tell you about the kind of film being used, but I think those things are captured in the film grain simulation model.
Possibly there are some other artefacts like evidence of special effects, post processing, etc.
If you extract the grain (e.g. by subtracting a blurred version of the image), the result contains mostly noise, not meaningful scene information outside of some variation according to image brightness. Film grain is random, so the extracted "grain layer" doesn’t encode original image detail but film grain itself encodes relative lightness in its _density_.
Mostly noise but not just noise. There is a reason why quality de-noising is a hard problem and the lost information becomes more apparent for videos where our brain is better able to extract it from the original noise.
Ok, very cool. But I want Netflix to explain it related to Seinfeld, which at 10-12 feet looks fine, but up close looks insane. Blocky MJPEG + grain filter??
It's not like we're on Pentium II processors anymore -- I can filter just about anything with ShaderGlass [0] on a shitty computer (and some of the CRT shaders like crt-hyllian-curvature are brilliant, especially on old shows like NewsRadio that only exist on DVD) .. and I'm shocked that Netflix doesn't just have this built into their Apple TV app or whatever. I'm shocked PLEX doesn't have it! (that I know of)
I made a comment on a different post about imagining a world where local AI/LLM/whatever does some favorable processing for you, by you, on your device, of web content, to enhance your experience. I really believe media (streamers all the way down to open source devs) need to begin to incorporate whatever's out there that reduces friction and increases joy. It's all out there already! The heavy lifting has been done! Just make Family Matters look like how it looked when I was locking in on a Friday night for TGIF LOL
Film grain simulation can be an effect, but that's not the primary application being described here. As @ricardobeat succinctly put it elsewhere in this post, this process usually supports "a reproduction of the original grain pattern, compressed separately from the underlying image content. The result is closer to the original picture than the denoised/compressed version".
> If kids grew up without seeing film grain, how would they feel about it?
Film isn't quite dead yet, with ~7% of features in 2025 still being shot on film and maybe 1% more taking the digital→film→digital (a.k.a. "analog intermediate") route. Unless their media diet is all super-recent and low-quality, even today's kids will have most likely seen many movies shot on film with visible grain. Popular candidates would include the Harry Potter franchise, most episodes of the Star Wars franchise, early Marvel movies, many animated Disney and Studio Ghibli features, etc.
It was an analogy. What I meant is that when I'm watching something in super high quality like a 4K Blu-ray, the presence of heavy film grain feels like a step backward visually, kind of like how VHS looked compared to today's standards. It takes away from the clean, sharp image I expect from that format.
happier and happier about leaving behind digital media to return to physical. to me this is literally slop. i want the uncompressed file stop selling me stepped on product
Lossless is lossless. But I shudder to imagine the file size of a 2 hour 13 channel 192/24 flac soundtrack.
However I notice a critical oversight on your part. You have assumed a channel depth of merely 8 bits when the minimum for a decent workflow is 12. Thus rather than 3.9 TiB (ie 4.3 TB) we arrive at 5.9 TiB.
Of course a modern feature length film is likely closer to 2.5 hours (7.3 TiB). Lossless HEVC should get that down closer to 2.5 TiB. At approximately 290 MiB per second that's going to demand dropping an SSD in the mail as the only practical method of distribution and playback.
If either lossless or uncompressed was reasonable in this instance, I would interpret it as such. Uncompressed video is completely bonkers of course. So is lossless. Shrug.
There's no need for low-quality comments cutting and pasting from your favourite LLM. You can literally just type it into your browser's search bar and learn what it is, or spend a minute reading the wiki page (https://en.wikipedia.org/wiki/AV1).
I disagree. I wish that when obscure terms were used, there would be an explanation of that specific term either in the article itself or, failing that, in an HN comment. This is especially helpful because sometimes a term will have multiple meanings in different verticals, which then requires each reader to not only search for the term but work out which context it's all about. Rather than each reader do that work, why not attach a good & appropriate explanation closer to the article itself? That's what I was attempting to do.
This is not fake grain - it's a reproduction of the original grain pattern, compressed separately from the underlying image content. The result is closer to the original picture than the denoised / compressed version.
If I understood it correctly they take one sample of grain and do random smaller samples of it to fill the picture with. That’s not even the original grain from each frame, but one sample for the whole video.
The technique is awesome, saves huge amount of bandwidth.
Personally I just like the original grain from each frame much much more, from an artistic standpoint.
This fails to acknowledge that synthesized noise can lack the detail and information in the original noise.
When you watch a high-quality encode that includes the actual noise, there is a startling increase in resolution from seeing a still to seeing the video. The noise is effectively dancing over a signal, and at 24 fps the signal is still perfectly clear behind it.
Whereas if you lossily encode a still that discards the noise and then adds back artificial noise to match the original "aesthetically", the original detail is non-recoverable if this is done frame-by-frame. Watching at 24 fps produces a fundamentally blurrier viewing experience. And it's not subtle -- on old noisy movies the difference in detail can be 2x.
Now, if h.265 or AV1 is actually building its "noise-removed" frames by always taking into account several preceding and following frames while accounting for movement, it could in theory discover the signal of the full detail across time and encode that, and there wouldn't be any loss in detail. But I don't think it does? I'd love to know if I'm mistaken.
But basically, the point is: comparing noise removal and synthesis can't be done using still images. You have to see an actual video comparison side-by-side to determine if detail is being thrown away or preserved. Noise isn't just noise -- noise is detail too.
Grain is independent frame-to-frame. It doesn't move with the objects in the scene (unless the video's already been encoded strangely). So long as the synthesized noise doesn't have an obvious temporal pattern, comparing stills should be fine.
Regarding aesthetics, I don't think AV1 synthesized grain takes into account the size of the grains in the source video, so chunky grain from an old film source, with its big silver halide crystals, will appear as fine grain in the synthesis, which looks wrong (this might be mitigated by a good film denoiser). It also doesn't model film's separate color components properly, but supposedly that doesn't matter because Netflix's video sources are often chroma subsampled to begin with: https://norkin.org/pdf/DCC_2018_AV1_film_grain.pdf
Disclaimer: I just read about this stuff casually so I could be wrong.
> Grain is independent frame-to-frame. It doesn't move with the objects in the scene (unless the video's already been encoded strangely)
That might seem like a reasonable assumption, but in practice it’s not really the case. Due to nonlinear response curves, adding noise to a bright part of an image has far less effect than a darker part. If the image is completely blown out the grain may not be discernible at all. So practically speaking, grain does travel with objects in a scene.
This means detail is indeed encoded in grain to an extent. If you algorithmically denoise an image and then subtract the result from the original to get only the grain, you can easily see “ghost” patterns in the grain that reflect the original image. This represents lost image data that cannot be recovered by adding synthetic grain.
It sounds like the "scaling function" mentioned in the article may be intended to account for the nonlinear interaction of the noise.
> If you algorithmically denoise an image and then subtract the result from the original to get only the grain, you can easily see “ghost” patterns in the grain that reflect the original image. This represents lost image data that cannot be recovered by adding synthetic grain.
The synthesized grain is dependent on the brightness. If you were to just replace the frames with the synthesized grain described in the OP post instead of adding it, you would see something very similar.
> So long as the synthesized noise doesn't have an obvious temporal pattern, comparing stills should be fine.
The problem is that the initial noise-removal and compression passes still removed detail (that is more visible in motion than in stills) that you aren't adding back.
If you do noise-removal well you don't have to lose detail over time.
But it's much harder to do streaming-level video compression on a noisy source without losing that detail.
The grain they're adding somewhat distracts from the compression blurriness but doesn't bring back the detail.
>The grain they're adding somewhat distracts from the compression blurriness but doesn't bring back the detail.
Instead of wasting bits trying to compress noise, they can remove noise first, then compress, then add noise back. So now there aren't wasted bits compressing noise, and those bits can be used to compress detail instead of noise. So if you compare FGS compression vs non-FGS compression at the same bitrate, the FGS compression did add some detail back.
I imagined that at some point someone would come up with the idea “let’s remove more noise to compress things better and then add it back on the client”. Turns out, it is Netflix (I mean, who else wins so much from saving bandwidth).
Personally I rejected the idea after thinking about it for a couple of minutes, and I’m not yet sure I was wrong.
The challenge with noise is that it is actually cannot be perfectly automatically distinguished and removed from what could be finer details and textures even in a still photo, not to mention high-resolution footage. If removing noise was as simple as that, digital photography would be completely different. If you have removed noise, you can’t just add back missing detail later—if you could, you would not have removed it in the first place (alas, no algorithm is good enough, and even human eye can be faulty).
I'm not saying that the final result is as good as the original.
I'm saying that the final result is better than standard compression at the same bitrate.
That might be true; however, if this takes hold I would be surprised if they choose to keep producing and shipping the tasty grain high fidelity footage.
Considering that NR is generally among the very first steps in development pipeline (as that’s where it is the most effective), and the rest of dynamic range wrangling and colour grading comes on top of it, they might consider it a “waste” to 1) process two times (once with this new extreme NR, once with minimal NR that leaves the original grain), 2) keep around both copies, and especially (the costliest step) to 3) ship that delicious analog noise over Internet to people who want quality.
I mean, how far do we go? It’ll take even less bandwidth to just ship prompts to a client that generates the entire thing on the fly. Imagine the compression ratios…
> Grain is independent frame-to-frame. It doesn't move with the objects in the scene (unless the video's already been encoded strangely). So long as the synthesized noise doesn't have an obvious temporal pattern, comparing stills should be fine.
Sorry if I wasn't clear -- I was referring to the underlying objects moving. The codec is trying to capture those details, the same way our eye does.
But regardless of that, you absolutely cannot compare stills. Stills do not allow you to compare against the detail that is only visible over a number of frames.
People often assume noise is normal and IID but it usually isn't. It's s fine approximation but isn't the same thing, which is what the parent is discussing.
Here's an example that might help you intuit why this is true.
Let's suppose you have a digital camera and walk towards a radiation source and then away. Each radioactive particle that hits the CCD causes it to over saturate, creating visible noise in the image. The noise it introduces is random (Poisson) but your movement isn't.
Now think about how noise is introduced. There's a lot of ways actually, but I'm sure this thought exercise will reveal to you how some cause noise across frames to be dependent. Maybe as a first thought, think about from sitting on a shelf degrading.
I think this is geared towards film grain noise, which is independent from movement?
It's the same thing. Yes, not related to the movement of the camera, but I thought that would be easier to build your intuition about silver particles being deposited onto film. You make in batches, right?
The point is that just because things are random doesn't mean there aren't biases.
To get much more accurate, it helps to understand what randomness actually is. It is a measurement of uncertainty. A measurement of the unknown. This is even true for quantum processes that are truly random. That means we can't know. But just because we can't know doesn't mean it's completely unknown, right? We have different types of distributions and different parameters in those distributions. That's what we're trying to build intuition about
I think you've missed the point here: the noise in the originals acts as dithering, and increases the resolution of the original video. This is similar to the noise introduced intentionally in astronomy[1] and in signal processing[2].
Smoothing the noise out doesn't make use of that additional resolution, unless the smoothing happens over the time axis as well.
Perfectly replicating the noise doesn't help in this situation.
[1]: https://telescope.live/blog/improve-image-quality-dithering [2] https://electronics.stackexchange.com/questions/69748/using-...
Your first link doesn't seem to be about introducing noise, but removing it by averaging the value of multiple captures. The second is to mask quantizer-correlated noise in audio, which I'd compare to spatial masking of banding artifacts in video.
Noise is reduced to make the frame more compressible. This reduces the resolution of the original only because it inevitably removes some of the signal that can't be differentiated from noise. But even after noise reduction, successive frames of a still scene retain some frame-to-frame variance, unless the noise removal is too aggressive. When you play back that sequence of noise-reduced frames you still get a temporal dithering effect.
Here's[1] a more concrete source, which summarizes dithering in analog to digital converters as follows:
With no dither, each analog input voltage is assigned one and only one code. Thus, there is no difference in the output for voltages located on the same ‘‘step’’ of the ADC’s ‘‘staircase’’ transfer curve. With dither, each analog input voltage is assigned a probability distribution for being in one of several digital codes. Now, different voltages with-in the same ‘‘step’’ of the original ADC transfer function are assigned different probability distributions. Thus, one can see how the resolution of an ADC can be improved to below an LSB.
In actual film, I presume the random inconsistencies of the individual silver halide grains is the noise source, and when watching such a film, I presume the eyes are doing the averaging through persistence of vision[2].
In either case, a key point is that you can't bring back any details by adding noise after the fact.
[1]: https://www.ti.com/lit/an/snoa232/snoa232.pdf section 3.0 - Dither
[2]: https://en.wikipedia.org/wiki/Persistence_of_vision
One thing worth noting is that this extra detail from dithering can be recovered when denoising by storing the image to higher precision. This is a lot of the reason 10 bit AV1 is so popular. It turns out that by adding extra bits of image, you end up with an image that is easier to compress accurately since the encoder has lower error from quantization.
The AR coefficients described in the paper are what allow basic modeling of the scale of the noise.
> In this case, L = 0 corresponds to the case of modeling Gaussian noise whereas higher values of L may correspond to film grain with larger size of grains.
This is a really good point.
To illustrate the temporal aspect: consider a traditional film projector. Between every frame, we actually see complete darkness for a short time. We could call that darkness "noise", and if we were to linger on that moment, we'd see nothing of the original signal. But since our visual systems tend to temporally average things out to a degree, we barely even notice that flicker (https://en.wikipedia.org/wiki/Flicker_fusion_threshold). I suspect noise and grain are perceived in a similar way, where they become less pronounced compared to the stable parts of the signal/image.
Astrophotographers stack noisy images to obtain images with higher SNR. I think our brains do a bit of that too, and it doesn't mean we're hallucinating detail that isn't there; the recorded noise - over time - returns to the mean, and that mean represents a clearer representation of the actual signal (though not entirely, due to systematic/non-random noise, but that's often less significant).
Denoising algorithms that operate on individual frames don't have that context, so they will lose detail (or will try to compensate by guessing). AV1 doesn't specify a specific algorithm to use, so I suppose in theory, a smart algorithm could use the temporal context to preserve some additional detail.
The noise does not contain a signal, does not dance over it, and is not detail. It is purely random fluctuations that are added to a signal.
If you have a few static frames and average them, you improve SNR by retaining the unchanged signal and having the purely random noise cancel itself out. Retaining noise itself is not useful.
I suspect the effect you might be seeing is either just an aesthetic preference for the original grain behavior, or that you are comparing low bandwidth content with heavy compression artifacts like smoothing/low pass filtering (not storing fine detail saves significant bandwidth) to high bandwidth versions that maintain full detail, entirely unrelated to the grain overlaid on top.
> If you have a few static frames and average them, you improve SNR by retaining the unchanged signal and having the purely random noise cancel itself out.
That's exactly the point of GP though. Even though each individual frame might be almost indistinguishable from random noise you can still extract patterns over time. This is also the case if you don't average the frames in software but let the viewer's brain do it. If you just remove all "noise" from each frame and then add random noise back those patterns will be lost.
In practice you won't have static frames but also movement so recovering the signal from the noise becomes a lot more complicated.
Anything with a pattern is by definition not noise, and the comment was that noise had signal. If you remove all noise, no signal or pattern is lost by definition.
However, the issue is that lossy compression removes various types of minute detail, smoothing surfaces to reduce the amount of data that has to be stored, be it noise "grain" or skin pores, according to compression settings. Storing the original noise as it was would basically make any compression impossible.
Denoising generally removes signal too. Removing noise and reconstituting similar noise to maintain the apparent qualities of an input can help compression, but you are also cutting out true details (typically fine detail).
The effect GP is pointing out is how denoisers damage detail, which is true. This detail can persist over multiple frames, which is why many denoisers include a temporal comparison component to mitigate the damage, but you still lose detail.
Definitely, but that's a result of compressing (with any algorithm) past a bitrate that would allow detail to be retained with high frequency details including noise being the first to go, not the result of noise having signal as was stated and more importantly unrelated to film grain synthesis which is about adding something new on the output side.
I love this concept/principle, one similar example I often bring up when I talk about machine learning, is comparing how a human would analyse night footage from a camera, and how a ML algorithm can pick up things no human would think about, even artifacts from the sensors which can be used as features. Noise is rarely ever just noise.
Some of the new 4K discs use DRR and the denoising process seems to remove the pores on people's faces occasionally, leaving actors looking like their face is made of wax.
Don't want to sound too snarky, but aren't you just saying that good denoisers must be temporal in addition to spatial? Something like an improved V-BM3D (cf https://arxiv.org/abs/2001.01802) and even traditional V-BM3D works fine even on non-Gaussian noise.
Together with external noise generation (something like https://old.reddit.com/r/AV1/comments/r86nsb/custom_photonno...) that'd support external denoising (pass the reference and denoised video to get the grain spec), the whole FGS thing would be much more interesting.
You could do an ai or compressed sensing upscale first, with multiple frames helping, have that be the base video sent, and then regranularize that.
There are definite philosophical questions over the merits of adding noise, but the problem with their example here is their denoising process appears to excessively blur everything, so both it and the synthesized grain image look noticeably less sharp than the source. The grain itself also looks too much like basic noise, and not really grain like.
> both it and the synthesized grain image look noticeably less sharp than the source
That's true, but at a given bitrate (until you get to very high bitrates), the compressed original will usually look worse and less sharp because so many bits are spent trying to encode the original grain. As a result, that original grain tends to get "smeared" over larger areas, making it look muddy. You lose sharpness in areas of the actual scene because it's trying (and often failing) to encode sharp grains.
Film Grain Synthesis makes sense for streaming where bandwidth is limited, but I'll agree that in the examples, the synthesized grain doesn't look very grain-like. And, depending on the amount and method of denoising, it can definitely blur details from the scene.
It seems like a shame that they didn’t include a screenshot of the original (with natural grain), after suffering from low-bitrate streaming. Aka the actual baseline.
I can see why they want to compare against the actual local copy of the video with the natural grain. But that’s the perfect copy that they can’t actually hope to match.
> It seems like a shame that they didn’t include a screenshot of the original (with natural grain), after suffering from low-bitrate streaming.
Isn't that the image captioned "Regular AV1 (without FGS) @ 8274 kbps"?
I think I misread the figures.
But still, they have:
> A source video frame from They Cloned Tyrone
> Regular AV1 (without FGS) @ 8274 kbps
> AV1 with FGS @ 2804 kbps
Just to emphasize the problem, would it be nice to see:
Regular AV1 (without FGS) @ 2804 kbps
It should look really bad, right? Which would emphasize their results.
But why do they need to emphasize it even more than the examples they gave? The "AV1 with FGS @ 2804 kbps" already looks as good or better than the "AV1 (without FGS) @ 8274 kbps", so it'll definitely look better than AV1 without FGS at an even lower bandwidth.
I think the distinction here is, they provide the "regular" stream, and the FGS stream noting that it's much smaller yet looks similar. What they don't have is a lower-bandwidth "regular" one, what 2000-or-so kbps looks like without FGS.
They did.
Since the beginning of film editors have added tricks in post.
I would love for them to provide an option to view it with film simulation vs without.
One of my favorite movies of all time, The Holdovers, did film simulation extremely well. It's set in the '70s so it attempts to look like a movie of that era.
It looked great to me, but if you're an actual film nerd you're going to notice a lot of things aren't exactly accurate.
Maybe in the near future we'll see Netflix being able to process some post effects on the client. So if you're color blind, you get a mode for that. If you don't want fake grain you can turn it off.
mpv can already do this: `--vf=format:film-grain=no` turns off grain synthesis. There are also people making custom shaders for things like emulating CRT monitors (originally for retro gaming, but I see there are also mpv versions).
Holdovers was pretty great. Sorta like an 80’s school comedy but with the perspective of the adults included, making it a totally different type of movie.
AV1 has tunable FGS levels, and to my eye they went very slightly higher than they should have (though there are tradeoffs; at some bitrates the blurring+renoising is so much better than the other visual artifacts you will otherwise get, that you do want it that high).
A few things to note:
- still-frames are also a mediocre way to evaluate video quality.
- a theoretically perfect[1] noise-removal filter will always look less detailed than the original source, since your brain/eye system will invent more detail for a noisy image than for a blurry image.
1: By which I mean a filter that preserves 100% of the non-grain detail present, not one that magically recovers detail lost due to noise.
A film that explores this is Antonioni’s Blowup: https://en.wikipedia.org/wiki/Blowup
These days, when we see noise/grain in an end product it has likely been added in post-production. So, ideally, studios would provide distributors with a noiseless source plus grain synthesis parameters. Bonus: many viewers would welcome an option to turn it off.
> provide distributors with a noiseless source plus grain synthesis parameters.
What parameters would that be? Make it look like Eastman Ektachrome High-Speed Daylight Film 7251 400D? For years, people have taken film negative onto telecines and created content of grain to be used as overlays. For years, colorists have come up with ways of simulating the color of specific film stocks by using reference film with test patterns that's been made available.
If a director/producer wants film grain added to their digital content, that's where it should be done in post. Not by some devs working for a streaming platform. The use of grain or not is a creative decision made by the creators of the work. That's where it should remain
> If a director/producer wants film grain added to their digital content, that's where it should be done in post. Not by some devs working for a streaming platform. The use of grain or not is a creative decision made by the creators of the work. That's where it should remain
Why? If you're spending a significant chunk of your bits just transmitting data that could be effectively recreated on the client for free, isn't that wasteful? Sure, maybe the grains wouldn't be at the exact same coordinates, but it's not like the director purposefully placed each grain in the first place.
I recognize that the locally-produced grain doesn't look quite right at the moment, but travel down the hypothetical with me for a moment. If you could make this work, why wouldn't you?
--------
...and yes, I acknowledge that once the grain is being added client side, the next logical step would be "well, we might as well let viewers turn it off." But, once we've established that client-side grain makes sense, what are you going to do about people having preferences? Should we outlaw de-noising video filters too?
I agree that the default setting should always match what the film maker intended—let's not end up with a TV motion smoothing situation, please for the love of god—but if someone actively decides "I want to watch this without the grain for my own viewing experience"... okay? You do you.
...and I will further acknowledge that I would in fact be that person! I hate grain. I modded Cuphead to remove the grain and I can't buy the Switch version because I know it will have grain. I respect the artistic decision but I don't like it and I'm not hurting anyone.
> Why? If you're spending a significant chunk of your bits just transmitting data that could be effectively recreated on the client for free, isn't that wasteful? Sure, maybe the grains wouldn't be at the exact same coordinates, but it's not like the director purposefully placed each grain in the first place.
I'm sorry your tech isn't good enough to recreate the original. That does not mean you get to change the original because your tech isn't up to the task. Update your task to better handle the original. That's like saying an image of the Starry Night doesn't retain the details, so we're going to smear the original to fit the tech better. No. Go fix the tech. And no, this is not fixing the tech. It is a band-aid to cover the flaws in the tech.
> I'm sorry your tech isn't good enough to recreate the original. That does not mean you get to change the original because your tech isn't up to the task.
The market has spoken and it says that people want to watch movies even when they don't have access to a 35mm projector or a projector than can handle digital cinema packages, so nobody is seeing the original outside a theater.
Many viewers are bandwidth limited, so there's tradeoffs ... if this film grain stuff improves available picture quality at a given bandwidth, that's a win. IMHO, Netflix blogs about codec things seem to focus on bandwidth reduction, so I'm never sure if users with ample bandwidth end up getting less quality or not; that's a valid question to ask.
Because the specks of grain aren't at the exact same coordinates? What differences are we talking about here exactly?
The differences are actual film grain vs some atrocious RGB noise artificially added by the streamer. How is that unclear? What else could we be talking about?
Right, the current implementation is bad.
In theory though, I don't see any reason why client-side grain that looks identical to the real thing shouldn't be achievable, with massive bandwidth savings in the process.
It won't be, like, pixel-for-pixel identical, but that was why I said no director is placing individual grain specks anyway.
> with massive bandwidth savings in the process
Let's be clear. The alternative isn't "higher bandwidth" it's "aggressive denoising during stream encode". If the studio is adding grain in post then describing that as a set of parameters will result in a higher quality experience for the vast majority of those viewing it in this day and age.
If the original is an actual production shot on film, the film grain is naturally part of it. Removing it never looks good. If it is something shot on a digital camera and had grain added in post, then you can go back to before the grain was added and then do it client side without degradation. But you can never have identical when it originated on film. That's like saying you can take someone's freckles away and put them back in post just rearranged and call it the same.
Sorry, I am talking about the case where grain was added in post. You originally said, and I quoted:
> If a director/producer wants film grain added to their digital content, that's where it should be done in post.
To me, this philosophy seems like a patent waste of bandwidth.
> once we've established that client-side grain makes sense, what are you going to do about people having preferences?
I already normalize audio (modern US produced media is often atrocious), modify gamma and brightness, and probably some other stuff. Oh and it's not as though I'm viewing on a color calibrated monitor in the first place.
The purists can purchase lossless physical copies. The rest of us would benefit from such improvements.
Netflix has their own in-house studio, right? The encoding and lossy compression is going to happen anyway. It seems like an easy win, for their directors to provide a description of the grain they want, so it can be replicated on the user side.
what does having an in-house studio have to do with it? they stream more content than just their own, and so they would not have creative license to alter content. they would only have some type of distribution license to stream the content as provided
Because they previously did not do commercials their original TV shows were not written with pauses every few minutes. They have approved cameras. They spend heavily on movie star salaries and skimp on set production.
Again, I ask, what does this have to do with anything regarding TFA? Anything in-house studio produces will not be shot on film.
And yet here we are: DNR -> fancy grain -> DNR -> basic, approximated grain. Because noise doesn’t compress. And you get compression artifacts even in Blu-ray releases. What’s the point of applying fancy grain when what a lot viewers end up seeing is an ugly smudge?
Because it looks amazing in the editing studio. Just like the sound mix is incredible on the Atmos monitors in the sound mixing room, even though the home viewers have a soundbar at best and tiny stereo speakers in a flat panel typically. The dynamics and dialog channel will be fucked. But that’s user error.
Movies are best enjoyed in the theater.
What, cramped into a room with tons of other people who can't help themselves from making noise or moving in front of you while you only have limited and overpriced snack options. As nice as a big screen and professional sound equipment is the downsides make the experience strictly worse than a big OLED + half decent soundbar.
You must live alone. Even at home, you have people called family or friends that do not necessarily sit idle while watching TV. Watching with kids at home is worse than at the movies because they are at home and don't have the same social rules to follow. People at home are also not under the social rules of not using their devices, and second screening has become the norm.
Yes, a theater probably holds more people than your typical viewing experience at home. Unless you go to the movies during the week and avoid crowds. The last movie I saw at the theater was on a Tuesday after opening weekend as 4pm. There might have been 2 other people in the entire theater. It was amazing.
If your kids are badly behaved that's on you. If others are badly behaved in a semi-public setting there is not as much you can do about it.
Yeah I always roll my eyes when people get so mad about compressed (as in reduced dynamic range) audio. I just want to watch / listen to stuff without annoying my neighbors and don't particularly care whether or not the volume of gunshots is realistic.
This is exactly why theatrical releases are so important to movie producers, isn't it?
Theatrical release qualifies for certain awards and shiny statues. That's their concern. If a streaming platform wants to give them enough cash to beat out projected box office earnings, then they'll take it if they don't have any grandiose visions of golden statues.
The grain is there to hide the ugly smudge. that's the question they rather you didn't ask
I think at some point studios will give de-grained versions to Netflix directly.
At some point, it would not surprise me for Netflix to require this to be provided. While not negating what you concluded, I just think the impetus for the result is important to distinguish.
GPs point is that those streaming platforms will remove the grain anyway and that process loses information compared to working with the original.
Currently, in order to deal with noisy masters, Netflix has to either:
1. Denoise the master, then add AV1 FGS metadata to tell players how to reconstruct the noise in the master (which is what the blog post is about) to get back the original image the director saw and approved
2. Do nothing (which is what they were doing), and let some of the noise get blurred or erased by the quantization step of the encoding process, or worse, burn shittons of coding bits trying to describe the exact noise in the frame, which hurts visual quality of the things people actually look at
All of these imply changes to the image that the director decided on to get around the underlying fact that deliberately adding noise to an image is, from a signal processing perspective, really stupid. But if we are going to do it, we can at least ensure it happens as far down the chain as possible so that Netflix's encoding doesn't destroy the noise. That's the idea you responded to: have the production company deliver a master with FGS metadata instead of baked-in film grain.
I'll keep the film grain, I just want to be able to turn off laugh tracks.
Why not just watch things where the creators don't feel they need to add laugh tracks in order to keep the audience engaged in the first place.
Agree, as someone who has spent way too much time studying the way motion picture film looks up close, this isn’t very realistic looking. It’s really just a form of dithering.
Video codecs use a lot of tricks based on human perception, perhaps it's much closer to the real thing when in motion vs a still image?
Forget the film grain, give us film lint and film hair!
> and not really grain like
that's an understatement. it just looks like RGB noise effect was added. film grain does not look like RGB noise. to me, film grain is only one part of what gave film the film look. the way the highlights bloom rather than clip. it also was more natural/organic/some descriptive other than the ultrasharp of modern digital acquisition. using some SoftFX or Black Mist type filters help, but it's just not the same as it is a digital vs analog type of acquisition. all of these attempts at making something look like it's not just keep falling down in the same ways. but hey, there's a cool tech blog about it this time. film grain filters have been around for a long time, yet people just don't care for them. even in Blu-ray time frame, there was attempts at removing the grain in the encode and applying it in playback. Netflix isn't coming up with anything new, and apparently nothing exciting either based on the results.
Agree, I really prefer luminance noise when simulating film grain rather than treating RGB so independently - the RGB here looks wrong to me. If you are thinking about film, you wouldn't model it as pure photon/detector noise, because film has some crystalline structure and usually there is some correlation between the channels. (Some of the best grain is on B&W films though, which have wonderful structure and statistics.)
I only have their single sample in the blog post to go by, but the original source frame also looks just like RGB noise? And the recreation looks really close if I swap between them. Looks fine to me.
So as long they're analyzing the style of grain in the source properly, which the technical part of the post mentions they do...
Get the retro gaming nerds on this, they'll make a film grain shader that's indistinguishable from reality in no time flat.
The "at scale" part is the real story here. Film Grain Synthesis has been available in the usual AV1 encoders for a while, but required some amount of manual tweaking to avoid creating problems, meaning it was only used in production when you had a very limited catalog, or for particularly important titles. They do not provide a lot of details here about how they overcame those problems, but it is nice to see it being deployed more broadly.
There's adaptive variant nowadays which makes things much easier to automate.
To the comments hating on grain: everything naturally has some amount of noise or grain - even the best digital sensors. Heck, even your eyes do. It's useful beyond just aesthetics. It tends to increase perceived sharpness and hides flaws like color banding and compression artifacts.
That's not to say that all noise and grain is good. It can be unavoidable, due to inferior technology, or a result of poor creative choices. It can even be distracting. But the alternative where everything undergoes denoising (which many of our cameras do by default now) is much worse in my opinion. To my eyes, the smoothing that happens with denoising often looks unrealistic and far more distracting.
Grain in state of the art digital sensors is nothing compared to what is added to the average movie.
My issue is that grain is good based on the creative decisions of the creators of the content. It is not something that a group of nerds compressing 1s and 0s should be making
I agree. However, let's look at it practically. Let's assume someone is watching content streamed on a low bandwidth connection. As a content creator, what version of the compressed content would you rather your audience experience:
a) Compressed original with significant artifacts from the codec trying to represent original grain
b) A denoised version with fewer compression artifacts, but looks "smoothed" by the denoising
c) A denoised version with synthesized grain that looks almost as good as the original, though the grain doesn't exactly match
I personally think the FGS needs better grain simulation (to look more realistic), but even in its current state, I think I'd probably go with choice C. I'm all for showing the closest thing to the author's intent. We just need to remember that compression artifacts are not the author's intent.
In an ideal world where we can deliver full, uncompressed video to everyone, then obviously - don't mess with it at all!
For content that we're concerning ourselves with this level of detail, I'd prefer the old iTunes method of prefetching the file and not stream it. For typical YT content, streaming is fine. For typical sitcom or other content, streaming is fine. For something like a feature that I'm so concerned about the details of grain, I have no problem downloading to play a local version. No, not a torrent.
We live in a world of resource constraints. The nerds compressing 1s and 0s are very much concerned about creative intent. They noticed that compression wrecks that creative intent and wanted to better communicate it in a world with bandwidth limitations.
Case in point: the HBO into animation. It uses static from the old analog days. It looks like shit even at 4k because random noise is impossible to compress without the exact strategy outlined here (stripping it out and rendering it later).
> This grain, formed from tiny particles during the film’s development, is more than just a visual effect. It plays a key role in storytelling by enhancing the film’s depth and contributing to its realism.
I never understood the “grain = realism” thing. my real eyes don’t have grain. I do appreciate the role of grain as an artistic tool though, so this is still cool tech
The article points out the masking effect of grain, which hides the fake-looking compression artifacts, and also the familiarity/nostalgia aspect. But I will offer an additional explanation.
Look around you: nearly all surfaces have some kind of fine texture and are not visually uniform. When this is recorded as video, the fine texture is diminished due to things like camera optics, limited resolution, and compression smoothing. Film grain supplies some of the high frequency visual stimulus that was lost.
Our eyes and brains like that high frequency stimulation and aren't choosy about whether the exact noise pattern from the original scene is reproduced. That's why the x265 video encoder (which doesn't have grain synthesis since it produces H.265 video) has a psy-rd parameter that basically says, "try to keep the compressed video as 'energetic' as the original, even if the energy isn't in the exact same spot", and even a psy-rdoq parameter that says, "prefer higher 'energy' in general". These parameters can be adjusted to make a compressed video look better without needing to store more data.
> my real eyes don’t have grain.
They definitely do at night when it's dark out. There's a kind of "sparkling" or "static" that comes in faint light.
Fortunately, our eyes have way better sensitivity than cameras. But the "realism" just comes from how it was captured using the technology of the day. It's no different from phonograph hiss or the way a CRT signal blurs. The idea is to be "real" to the technology that the filmmaker used, and the way they knew their movie would be seen.
It's the same way Van Gogh's brush strokes were real to his paintings. You wouldn't want his oil paintings sanded down to become flat. It's the reality of the original medium. And so even when we have a digital print of the film, we want to retain as much of the reality of the original as we can.
Your Van Gogh analogy makes sense for old movies. It doesn't quite explain why we're still adding grain to new movies, except for those few which are purposefully evoking older movies.
Even modern cameras have grain. If you need to integrate your scene with motion graphics, background replacement, or vfx, you'll need to remove grain on part of the image, edit it, add the original grain back where possible and synthesize new grain elsewhere.
Often it can also make sense to modify the grain for aesthetics. Denoising usually produces a less detailed result, but what you can do is denoise only the color channels, not the brightness channel. Brightness noise looks normal to us, while color noise tends to look very artificial. But by keeping the brightness noise, you avoid losing detail to the denoiser.
We don't use obvious grain, usually. That's generally precisely to evoke something about the past -- for flashbacks, a period look, etc. A sense of grittiness like 70s movies, etc.
On the other hand, a small amount of constant grain or noise is intentionally often introduced because otherwise images feel too static and end up looking almost fake. Similarly, dithering is intentionally added to audio, like mastering CD's or tracks. It helps prevent artifacts in video and in audio.
Same reason why movies still use extremely low frame rates - because people have gotten used to that effect.
It doesn't help that a lot of great movies are older thus those limitations of older technology become subconsciously associated with quality.
...what makes the frame rate thing especially weird is how many TVs ship with motion smoothing enabled by default, which looks so much worse than actual high frame rate content would.
I don't like it at all. But someone must, or the TV manufacturers wouldn't do it.
Part of me thinks that it's only the people who know what "frame rate" means who prefer low frame rates, and a majority of the general public actually prefers high frame rates but lacks the terminology or knowledge to express this desire.
People are always trying to rationalize and justify aesthetic preferences. The depth and nuance of your understanding of a thing will change how you perceive variations of that thing, whether it's guitar tonewoods, style of music, types of paint, flavor of beer, or the grain in film. If you know a lot about a subject, you can tell a lot about the history of a thing, and that's going to change how you feel about a thing.
A child watching a Buster Keaton skit and gasping and giggling and enjoying it is going to have a different subjective aesthetic experience of the media than a film critic who knows exactly what type of film and camera were used, and what the meaning of all the different abstractions imply about the scene, and the fabric of Keaton's costume, and so on, and so forth.
Subjective aesthetic preferences are in the realm of cognition - we need a formal theory of intelligence mapped to the human brain, and all of these subjective phenomena collapse into individualized data processing and initial conditions.
There's something about film grain contrasted against clean cel animation which might make it easier for people to suspend disbelief. They are conditioned to think that absence of grain is associated with unreal animation, particular types of media, and CGI. Home video and news and so forth had grain and low quality, so grain gets correlated with "real". In my view, there's nothing deeper than that - we're the product of our times. In 40 years, media will have changed, and it may be that film grain is associated with surrealism, or edited out completely, as it's fundamentally noise.
The way I see it is that grain makes the film look more detailed than it really is, it can also hide compression artefacts and blurriness.
I don't know the psychovisuals behind that. Maybe it adds some high frequencies that compression often washes out, or maybe acts like some kind of dithering.
As for your eyes, I am pretty sure that they have grain, that's how quantum physics work, you just don't perceive it because your brain filters it out. But again, I don't know how it interacts with film grain.
> As for your eyes, I am pretty sure that they have grain
And lots of it, actually. Just close your eyes or look at any non-textured surface. Tons of noise.
The decreasing signal-to-noise ratio is also highly noticeable when it gets darker.
Video signal without the noise or grain is annoying to watch as it makes everything in the ”out of focus” zone look smooth blurry. Your eyes want to focus yet it is an illusion of depth without an actual depth. Noise texture emphasizes that this is just a 2D plane after all so your eyes can rest and the viewer doesn’t feel like they need glasses. This is just my theory of it based on observation. No research behind it.
This reminds me of modern windows having fake panes. They’re just strips that are applied to give the impressions that there are multiple smaller panes because people are used to that and it feels “correct”.
I have to imagine past glassmakers would have been absolutely enthralled by the ability we now have to make uniform, large sheets of glass, but here we are emulating the compromises they had to make because we are used to how it looks.
> They’re just strips that are applied to give the impressions that there are multiple smaller panes because people are used to that and it feels “correct”.
It is more than just 'feeling correct': windows and their various (sub-)elements that make them up (can) change the architectural proportions and how the building is perceived as a whole:
* https://www.youtube.com/watch?v=uAMyUoDz4Og
* https://www.youtube.com/watch?v=_c8Ahs9Tcnc&t=49
It is similar with columns: they're not just 'tall-and-narrow', but rather have certain proportions and shapes depending on the style and aesthetic/feeling one wishes to convey:
* https://en.wikipedia.org/wiki/Classical_order
And these proportions can even be 'fractal': the window panes related to windows as a whole, related to the building as a whole:
* https://www.youtube.com/watch?v=J-0XJpPnlrA&t=3m13s
* https://en.wikipedia.org/wiki/Golden_rectangle
* https://en.wikipedia.org/wiki/List_of_works_designed_with_th...
* https://www.nngroup.com/articles/golden-ratio-ui-design/
I strongly doubt that multiple smaller panes would have ever become a common style if we could have always made large glass panes. This is a perfect example of people becoming very used to a style forced by a technological limitation that is emulated even after the limitation doesn't exist.
> I strongly doubt that multiple smaller panes would have ever become a common style if we could have always made large glass panes.
Perhaps, but if you're going to have them anyways you might as well make a conscious choice as to how they add to the overall design of the structure.
It used to be a bigger deal (when digital cameras started being used) since people felt like digital video didn't look real/as good - movies shot on film were generally better looking (as crews were used shooting with it and digital video wasn't as sophisticated as today) and HAD grain.
It might be that there is a large part of the population that still has that association.
Cinephiles are also more likely to watch older (i.e. with grain) movies that ARE well shot and beautiful (which is why they are classics and watched by cinephiles) and not see bad film movies, only the cream of the crop, while being exposed to the whole gamut of quality when watching todays movies shot digitally. Would reinforce that grain = good while not being necessarily the case - and their opinion might be heard more than gen pop.
At any rate, it can be a neat tool to lower sharpness!
Mine do, at least when it's very dark
Yeah I've had visual snow, I think only when I'm tired, stood up too fast, dehydrated, etc. https://en.wikipedia.org/wiki/Visual_snow_syndrome
It's not even that, just under low light conditions (after you let your eyes adjust to see anything at all) everyone will see quite a bit of noise.
It's doubling back on itself. The film grain makes the footage look "cinematic" because it's how old movies looked.
Film grain and 24fps are both examples of people being far too attached to the technical limitations of film.
23.976 fps has been put on a pedestal as the "correct" look. Just look at the reaction to The Hobbit. However it does provide some objective advantages. 60 fps requires more lighting. Adding more lights means more electric setup and heat for actors in heavy makeup and costume. In post production that's more frames to edit.
23.976fps is only "correct" when telecining to a 59.94Hz format. 24fps round is the "correct" format in terms of actual filmmaking. I don't know of many films specifically shot at 23.976fps.
Isn't it strange how people accepted much bigger changes like sound and color but are now balking at higher frame rates?
Those came much earlier in the history of film and have a much more significant qualitative impact.
Higher frame rate has a huge impact on the smoothness of motion and really benefits actions movies.
> 60 fps requires more lighting.
Assuming the comparison uses the same image sensor for each.
If that were an argument then frame rates would have increased with more efficient lighting.
It's funny, because I've always had a subtle visual snow that looks like a film grain on top of everything I see, and for a long time I thought everybody had it and that that's why artificial grain is added, to make pictures appear more realistic.
I've since learned that not everybody sees the world like I do, but I still do love to see grain and noise in pictures. Only RGB noise I often find dreadfully ugly when looked at up close, which is a shame, since that is exactly what most color cameras include.
film grain adds realism in the same way that high frame rate films look wrong or vinyl sounds "warmer" or tube guitar amps sound "better" - It is what we are used to.
My vision is grainy because of visual snow. Which is why I turn off film grain in games, it stacks on my vision and makes a visual mess.
grain and 24fps and widescreen trigger certain contextual emotions around the movie-watching experience. remove them and your brain contextualizes the video very differently.
this is likely the result of ~100 years of film-based filmmaking and projection. hell, we still call it filmmaking.
Yes, it is only the result of familiarity. We could gradually increase the frame rate of movies made in a year by 1 fps per year and then no one would even notice after 24 years every new movie would be 48fps.
No, it would take a lot longer than that. You’re fighting 100 years of tradition.
You would notice 1 fps more per year?
no i will be watching 24fps films for the remainder of my life, which may be 40 more years. if all new films went up 1fps per year i would still go out to the movies in 20 years and be like “wtf is this crap?”
it would take a generation or more to eradicate this cultural context. casablanca is never going to be in 48fps.
That is a very very silly attitude.
It’s not an attitude, it’s the reality of the world we live in. Cinema and film has shaped our society for a century.
No. You are no different than someone refusing to watch color movies because they were in black and white initially.
Where did I refuse to watch anything? No amount of this attitude is going to make Casablanca or The Godfather or Pulp Fiction or Infinity War or Star Wars or Fight Club or 2001 or Top Gun be in any framerate other than 24fps.
If they start making films in higher frame rates today, that won’t change that fact. Do you think people are going to stop watching films from the 50s, 60s, 70s, 80s, 90s, and 2000s?
It would take a concerted effort, beginning now, at least an entire generation (30-50 years) to remove or reduce the cultural impact of the 24 frames per second widescreen tradition. No such effort is presently underway, because higher framerates don’t help movies’ visual storytelling, and they may hurt it.
The Hobbit was Peter Jackson’s attempt. It went nowhere.
https://gizmodo.com/the-hobbit-an-unexpected-masterclass-in-...
"no i will be watching 24fps films for the remainder of my life, which may be 40 more years. if all new films went up 1fps per year i would still go out to the movies in 20 years and be like “wtf is this crap?”"
The Hobbit looked great at 48fps. Higher FPS failed only because reactionary people as yourself irrationally rejected it because "it looks weird!".
You need to understand that 24fps is a compromise chosen to save film. It is the bare minimum frame rate to have smooth motion under most but not all circumstances. It really hurts action films because anything moving too fast has excessive motion blur.
If you think the purpose of consuming film is to trigger rational responses in people, I don’t know what to tell you.
Do you do the same thing for poetry or food, too?
Permanently rejecting faster frame rates for movies is like rejecting printed text in favor of handwritten manuscripts. You are rejecting a technological advance for extremely arbitrary reasons. If 23 or 25 or 30fps had become the standard you would be insisting that it was just as special.
Grain = realism because real captured grain isn't total random noise. It's authentic noisy data. It's part of captured scene. It adds subtle tiny but real detail to the scene. Unless I am corrected here and that real grain is also total random noise.
Grain in movies has a similar role to grain in leather products, it is an indicator of authenticity
If your eyes did have grain, then it would still be applied to watching an "ungrained" film, as you're still using the same eyes.
Your eyes have lots of grain in low light conditions.
Film grain can create stochastic resonance with the underlying ground truth. In practice, this can improve the perceived image quality over having none.
Honestly to me it reads like a solution looking for a problem. I’ve never considered non deterministic imperfections that happen during recording of the movie to be essential to storytelling.
> Picture this: you’re watching a classic film, and the subtle dance of film grain adds a layer of authenticity and nostalgia to every scene
It just adds visual noise that obscures details of the authentic scene, and nothing prevents nostalgia from being tied to many of the more prominent visual cues like old actors or your own old memories from when you watched it first...
> contributing to [film's] realism
But there is no grain in reality, so it does the opposite
Otherwise I'm glad AV1 marches along and instead of wasting bitrate encoding visual garbage has an algorithmic replacement mechanism- which also means you could turn it off easier.
Documentaries might care about accurately representing reality. For every other cinematic genre, "authenticity" is not an inherent goal. If film grain is part of the director's vision, then that's just as valid as a choice to have dramatic non-diagetic music playing in the background of a scene (which is highly inauthentic, but also highly effective at evoking emotion, which is the point of art).
I don't see how that is supposed to be relevant to the article's claim that grain adds authenticity and realism. Someone claiming that non-diegetic music adds to authenticity would also be weird.
I think what the author meant is that film grain is part of the authentic visual aesthetic of cinema produced using celluloid materials. A de-noised image is therefore inauthentic as that is contrary to the appearance of the original projected image or source/master elements.
> But there is no grain in reality, so it does the opposite
Well ackchually -- illumination is inherently random, so all time-bounded captures of a scene (including what your eyes do) are subject to shot noise: https://en.wikipedia.org/wiki/Shot_noise
You misackchuallied grain with any noise, for example, the bottom right square of the image at the wiki page is not grainy despite being technically shot-noisy
> But there is no grain in reality, so it does the opposite
The author did not do a particularly good job of selling the merits of film grain. I am not sure what he meant by that but I will say that it does increase the perceived sharpness of the image as well as perceived detail, even if it is just an illusion. The cinematographer Steve Yedlin describes it as giving the viewer’s eyes something to “latch” onto.
> It just adds visual noise that obscures details of the authentic scene
Does it add any more than modern video compression techniques? What constitutes noise in cinema, is somewhat subjective.
Which modern compression artifacts that are still visible at high bitrates do you have in mind that would similarly detail-obscuring?
When you talk on a cellphone the codec AMR-WB nominally captures 50 Hz - 7000 Hz. However that's only on the optional highest bitrate 23.85 Kbps. The most common bitrate 12.65 Kbps only goes up to 6400 Hz and synthesizes 6400 - 7000 Hz from lower frequencies and noise as it sounds better than not having the noise!
This sent me down a very interesting rabbit hole, thanks!
Film grain needs to die. Its time is past. Sepia photographs and running 16 FPS silent film at 24 FPS are already dead. Next, film grain.
Eastman Business Park in Rochester has been demolished.
Also, please stop putting dust and scratches on YouTube videos. Thank you.
Fake film grain maybe, but saying all film grain needs to die is like saying oil painting brush strokes need to die.
But why does filmgrain need to die?
Because it's noise.
We got rid of interlaced TV. We got rid of the low-bandwidth color changes of NTSC. (NTSC video can only go through the color gamut about ten times per scan line, which is why colored text on NTSC is truly awful.) We got rid of projector film transport jank. (Except for really bad Youtube videos which use a filter to produce fake film transport jank.) Nobody wants any of those back.
It's a bit frustrating that the footage is first shot, then denoised in post, then renoised in post, then denoised in encoding and then renoised at decoding.
You worry too much, it's all fake. What you think is the "footage" will be many layers from various sources composited together.
Fake lights, fake shadows, fake sky, ...
Only if you are aware of it, which 99.9% of people consuming video content are not. It’s simply an unimportant implementation detail (from a viewer’s perspective who doesn’t really care about bitrate-as-cost).
As a mirrorless photographer I use a lot of processing to create a unique look. If I at all can, I set the aperture really small, use denoising, sharpening and color grading. If people think my photo came out of an old book that’s great.
I have looked at various “film look” filters and so far haven’t found one I like. When I think “film grains” I think of irregular shaped crystals, but most of the fake film grain I’ve seen makes a square pixel grain which doesn’t convince me at all. My understanding is the grain already exists before the image is shot in the sense that the films has irregular “pixels” which respond more or less because of shot noise and variation of the grain and I haven’t found anything that really simulates this.
Everything is fake now. I want a technology which works with a raw film scans, not even compressing them to JPEG, which is a 1st step in loosing the details BTW. Motion detection, key frames, delta frames - fine. But with a lossless video. On a Blu Ray off course, i don't care much about streaming.
A 4K/24p film encoded with Apple ProRes 4444 XQ (not even ProRes RAW) is 716 GB per hour, so you would need to swap a total of 30 Blu-ray discs once every 4 minutes in order to watch a 2 hour movie.
UHD discs used for 4K movies today already store much more than 24 GB per disc.
I wonder how much you'd get with such a technology... truly uncompressed 4K video you're talking about something on the order of a few terabytes for a 90-minute movie, so way way bigger than the biggest 4K Blu-ray discs. Lossless compression would get you under that number, but far enough to matter?
The key point some people are missing here is that removing noise from a video before compression makes it more compressible. So if you compare a video compressed without denoising/degrading at bitrate X to a video that was filtered and then compressed at bitrate X with "fake noise" added later, then the latter will have edges and details that the former doesn't.
The principle of post-processing videos to add noise or sharpening is something that desktop media players have supported for years, some are even reasonably good at it.
Exactly. Everyone is going on about grain quality but are in fact comparing wheat and spelt (if I may).
What matters to Netflix is the bandwith/quality ratio, that’s it. Removing noise before and adding it in software at playback time will produces a significantly better result, at similar bit rate. Because compressing noise produces horrible results.
Between 24fps and film grain people are way too attached to fundamentally inferior technology. With how strongly people resist frame rates faster than 24 I'm surprised people accepted color and sound, which were much bigger changes.
>This grain, formed from tiny particles during the film’s development, is more than just a visual effect. It plays a key role in storytelling by enhancing the film’s depth and contributing to its realism
That's one hell of an exaggeration. But even if that was true, I don't see how removing it, then adding fake on in would help. That entire sentence really has a "written by ChatGPT" smell. >It's not just X. It's Y".
Yup, yet another example of the thing I'll never stop finding fascinating:
ANY noticeable percieved "flaw" in any creative media will eventually become an aesthetic choice.
I think there is a bit more to it. For example when developing a game when CRTs were popular they were using CRTs to view their game and making decisions based on what they saw on the CRT. If you display the same game with perfect square pixels it looks different. If the developers were viewing square pixels when developing the game they would make different decisions.
I don't think this conflicts with what I'm saying; I've seen what you talk about -- and yet in modern days people will emulate the "square pixel bad style" regardless.
Agree. Purely opining, but I assume that it's because of the emotional connection that artistic media has on people, despite the flaws.
People remember the emotions the artwork engendered, and thus the whole work is associated with the feelings, flaws and all. If the work is particularly widely known, the flaws can become a stand-in for the work itself.
I see this in video games - I'm fond of the NES-era "flaws" and limitations (palette limits, sprite limits, sound channel limits), but less connected to the Atari 2600 or SNES/PS1/NDS/etc flaws. Shovel Knight is charming; A Short Hike, while great, doesn't resonate on a style level.
Even if you ignore all of the emotional attachment aspects, I think you would expect this to happen simply because people who are good at their craft will learn how to work with the shortcomings of their tools. If your cameras unavoidably add grain to everything you shoot you'll find ways to make that look good, and some of the things you come up with might not work without grain.
For an example, watch Shogun. Director apparently thought that most of the screen being out of focus was a positive. Quite distracting.
I've noticed this in a few things the last few years. The top and bottom of the shot are out of focus and it is super distracting to me. Maybe it is meant to draw the eye to the middle of the frame.
Yup, notice that as well. A blurry vignette. Needle sharp in the center, but objects that are the same distance are blurry at the edges of the frame. At first I thought they use vintage lenses, but it's probably done in post.
I sometimes do something similar on photos, where I darken the edges a bit.
> I sometimes do something similar on photos, where I darken the edges a bit.
Which is also an emulation of the shortcomings of older/worse technology.
Not necessarily, there are still lenses being produced today with vignetting, especially wide open.
Avoiding vignetting is a trade off between complexity, cost, weight, and other optical flaws. That said modern optical design and manufacturing has made it a lot easier to have nearly flawless (clinical) designs.
Or Snyder's terrible zombie movie Army of the Dead where he uses lenses with very shallow depth of field that makes almost everything look out of focus. It is very annoying.
I don't think it's just that here. Our brains simply prefer high entropy signals over low entropy ones. It's why mobile phones (and digital cameras before them) use sharpen filters to the point where the result looks unnatural.
I remember years ago when digital projection was just becoming the norm. I saw a movie (I don't remember which one) and during the opening scene I remember thinking "why are the credits jittering around? Oh. This is actually being projected on film!"
I'm in my early 50s so I remember film quite well. Just like vinyl or cassettes, I ain't going back and unless it is an artistic choice I don't want films to emulate what I consider to be an inferior technology.
How much grain is there in IMAX films?
There's an influx of high-profile directors/films right now and in pipeline filmed for IMAX (F1: The Movie I think, Mission Impossible, etc) and Christopher Nolan's Odyssey coming next year shot entirely on IMAX film with newly developed smaller/quieter cameras made to accomplish it.
> How much grain is there in IMAX films?
I've read that a 15-perf 65mm IMAX negative shot with slower film stocks is "virtually grainless", even when viewed on a 70ft screen. Grain is apparently noticeable in IMAX films when large/fast stocks are used and pushed toward their limits, and (of course) when smaller-format film stocks have been blown up.
I like this. Not really because I feel modern media should have added grain, but because for older media this is a method to get closer to the original but at much lower bitrates without excessive smoothing. What’s not to like?
Also, the author had me at God of Gamblers 2. So good. I will take him up on his recommendation to rewatch.
The original grain that is captured is actually a detail and not total random noise. I believe you can make up the vague sense of original scene if you could somehow extract that grain/noise alone.
It's like reducing an image to tiny dots with dithering (reminds of Atinkson dithering). Those grains are not a noise, they are a detail, actual data. That's why real grain looks good IMO.
This is just a fact of lossy compression: you want to throw away information that contributes less to the perception of the video so that you can describe it with fewer bits of information.
There are two possible advantages for this kind of grain synthesis. For Netflix, they could produce the same perceived quality at lower bitrates, which reduces costs per view and allows customers with marginally slow connections to get a higher quality version. For a consumer, the advantage would be getting more non-grain detail for a fixed bitrate.
You are right that if you subtract the dentists frame from the raw one, showing only the estimated noise, you would get some impression of the scene. I think there’s two reasons for this. Firstly, the places where the denoiser produced a blurry line that should be sharp may show up as faint lines. I don’t think this is ‘hidden information’ so much as it is information lost to lossy compression. In the same way, if you look at the difference between a raw image and one with compression, you may see some emphasized edges due to compression artefacts. Secondly, the less exposed regions of the film will have more noise so noisiness becomes a proxy for darkness, allowing some reproduction of the scene. I would expect this detail to be lost after adjusting for the piecewise linear function for grain intensity at different brightness levels.
Perhaps a third thing is the level of noise in the blacks and the ‘grain size’ or other statistical properties tell you about the kind of film being used, but I think those things are captured in the film grain simulation model.
Possibly there are some other artefacts like evidence of special effects, post processing, etc.
If you extract the grain (e.g. by subtracting a blurred version of the image), the result contains mostly noise, not meaningful scene information outside of some variation according to image brightness. Film grain is random, so the extracted "grain layer" doesn’t encode original image detail but film grain itself encodes relative lightness in its _density_.
Mostly noise but not just noise. There is a reason why quality de-noising is a hard problem and the lost information becomes more apparent for videos where our brain is better able to extract it from the original noise.
Ok, very cool. But I want Netflix to explain it related to Seinfeld, which at 10-12 feet looks fine, but up close looks insane. Blocky MJPEG + grain filter??
It's not like we're on Pentium II processors anymore -- I can filter just about anything with ShaderGlass [0] on a shitty computer (and some of the CRT shaders like crt-hyllian-curvature are brilliant, especially on old shows like NewsRadio that only exist on DVD) .. and I'm shocked that Netflix doesn't just have this built into their Apple TV app or whatever. I'm shocked PLEX doesn't have it! (that I know of)
I made a comment on a different post about imagining a world where local AI/LLM/whatever does some favorable processing for you, by you, on your device, of web content, to enhance your experience. I really believe media (streamers all the way down to open source devs) need to begin to incorporate whatever's out there that reduces friction and increases joy. It's all out there already! The heavy lifting has been done! Just make Family Matters look like how it looked when I was locking in on a Friday night for TGIF LOL
[0] https://github.com/mausimus/ShaderGlass
Is film grain a nostalgia effect? If kids grew up without seeing film grain, how would they feel about it?
> Is film grain a nostalgia effect?
Film grain simulation can be an effect, but that's not the primary application being described here. As @ricardobeat succinctly put it elsewhere in this post, this process usually supports "a reproduction of the original grain pattern, compressed separately from the underlying image content. The result is closer to the original picture than the denoised/compressed version".
> If kids grew up without seeing film grain, how would they feel about it?
Film isn't quite dead yet, with ~7% of features in 2025 still being shot on film and maybe 1% more taking the digital→film→digital (a.k.a. "analog intermediate") route. Unless their media diet is all super-recent and low-quality, even today's kids will have most likely seen many movies shot on film with visible grain. Popular candidates would include the Harry Potter franchise, most episodes of the Star Wars franchise, early Marvel movies, many animated Disney and Studio Ghibli features, etc.
When I watch an 80 GB 4K movie from Blu-ray, the last thing I want is film grain, which makes it look like a VHS recording from the 90s.
Film grain (real or artificial) is nothing like VHS artifacts.
It was an analogy. What I meant is that when I'm watching something in super high quality like a 4K Blu-ray, the presence of heavy film grain feels like a step backward visually, kind of like how VHS looked compared to today's standards. It takes away from the clean, sharp image I expect from that format.
Please also add more noise to the audio track.
happier and happier about leaving behind digital media to return to physical. to me this is literally slop. i want the uncompressed file stop selling me stepped on product
Your statement makes me think that one of the following is true:
1. You prefer Betamax or VHS to digital media (highly unlikely)
2. You own laserdiscs (limited to 480i)
3. You own 35mm prints of film.
Since all other formats film has been made available on are both digital media and compressed.
> 2. You own laserdiscs (limited to 480i)
Down-scaling to 480i is also a form of compression so you can discount that one.
Uncompressed 4k video is ~5Gbps (3840 * 2160 * 3 * 24 * 8). A 2-hour movie clocks in at about 4.3TB. (3840 * 2160 * 3 * 24 * 60 * 60 * 2)
All that is 24fps.
That's without audio, which I assume you also want to be uncompressed.
Not that it changes the argument but most movies are not 16:9 aspect ratio so don't need to make use of 2160 pixels height.
TBF even as bloated as 48/24 FLAC is it's a rounding error at that point.
But FLAC is compressed. Don't forget 13 channels. Only the best! Probably 192 khz.
Lossless is lossless. But I shudder to imagine the file size of a 2 hour 13 channel 192/24 flac soundtrack.
However I notice a critical oversight on your part. You have assumed a channel depth of merely 8 bits when the minimum for a decent workflow is 12. Thus rather than 3.9 TiB (ie 4.3 TB) we arrive at 5.9 TiB.
Of course a modern feature length film is likely closer to 2.5 hours (7.3 TiB). Lossless HEVC should get that down closer to 2.5 TiB. At approximately 290 MiB per second that's going to demand dropping an SSD in the mail as the only practical method of distribution and playback.
Even though op wrote "uncompressed" you should be able to deduce from context that they meant not lossy compressed.
If either lossless or uncompressed was reasonable in this instance, I would interpret it as such. Uncompressed video is completely bonkers of course. So is lossless. Shrug.
Physical media is what pioneered "stepping on" the product with things like Pan & Scan
agree
I love grain! 16mm vibes
They should use FPGA acceleration for this.
"Your scientists were so preoccupied with whether or not they could, they didn't stop to think if they should."
[dead]
[flagged]
[flagged]
What is the motivation for writing HN comments with AI?
Low quality AI comment
[flagged]
There's no need for low-quality comments cutting and pasting from your favourite LLM. You can literally just type it into your browser's search bar and learn what it is, or spend a minute reading the wiki page (https://en.wikipedia.org/wiki/AV1).
I disagree. I wish that when obscure terms were used, there would be an explanation of that specific term either in the article itself or, failing that, in an HN comment. This is especially helpful because sometimes a term will have multiple meanings in different verticals, which then requires each reader to not only search for the term but work out which context it's all about. Rather than each reader do that work, why not attach a good & appropriate explanation closer to the article itself? That's what I was attempting to do.
Fake grain really disgusts me.
This is not fake grain - it's a reproduction of the original grain pattern, compressed separately from the underlying image content. The result is closer to the original picture than the denoised / compressed version.
If I understood it correctly they take one sample of grain and do random smaller samples of it to fill the picture with. That’s not even the original grain from each frame, but one sample for the whole video. The technique is awesome, saves huge amount of bandwidth. Personally I just like the original grain from each frame much much more, from an artistic standpoint.
Depends on your definition of fake. I consider this to have crossed the line from lossy compression to a generative product.
When you ran out of things to innovate on.