This reminds me of how a lot of older TV shows and movies had an 'enhance' button that could somehow magically take a tiny, blurred image and blow it up to a level of detail where even the most minor things could be seen again. Like a suspect's eyes in the reflection of a window that looked like it was maybe 4 pixels in size in the original image.
The AI examples and reasoning in the article are exactly why this can't work, and is so unrealistic. The info isn't just hidden away in the info in a way some tool can figure out what it is with another effort, it's simply not present entirely. Any tool that tries to enhance a low quality image or video is just going to be making up the missing data.
> Any tool that tries to enhance a low quality image or video is just going to be making up the missing data.
So there is some things that instinctively don't sound correct here.
A frame may not contain the info to enhance, maybe it's too blurry... but a series of frames can (I'm pretty sure) between them have enough info to deblur an object in the background, it really helps to know the focal length, and distance maps...
My only point is information in a video is more than the sum of the information of it's frames, it also captures delta, not just from frame to frame, but relevant frame to relevant frame. That's a fair bit to work off of.
I do agree with our general point when information is 'absent'. I disagree how much absent information is in a video which can't be reconstructed by this delta. It certainly isn't zero, or 100%.
Inventing colors from a grayscale is a good example of absent information and of where this goes awry. Deblurring/blurring operations, less so.
> but a series of frames can (I'm pretty sure) between them have enough info to deblur an object in the background
You're probably thinking of geometrical super-resolution, such as gigapixel photography. Blurring is the discarding of high-frequency information, and only a very very small part of this information can be recovered using techniques like this (as in, so little that you wouldn't be able to notice it).
> My only point is information in a video is more than the sum of the information of it's frames,
It's actually less than the sum of the information of it's frames. An off-the-shelf lossless compression algorithm can give you an upper bound for the amount of information present in a video file.
I'm genuinely curious what you thought "AI restoration" was doing if the fact that it added details that it thought were reasonable was surprising to you
Yeah, this is fundamentally expected behavior but the results can vary wildly based on how much of a gap there is between available information and the 'resolution' of the desired output.
I feel like managing the level of information available and judging if its sufficient to make an extension of details lead to a reasonable outcome is the responsibility of the human/user more than the AI. If half the image is destroyed, it is less of a restoration and more of an imagination but the process is the same as if it just had some scuff marks to buff out. Its on the person using the tool to recognize the extent of impact that using the tool is having and to communicate that extent when discussing the results.
otherwise i could give the "restoration process" a torn corner of a photo from my great grandmas house and after it completely makes up a photo i call it restored
But this is exactly what you expect when you ask intelligence (machine or human) to fill in missing information. It picks some plausible, middle-of-the-road filler. If you were a restoration artist with an airbrush (or photoshop) with just the instruction to make the picture nicer, that's what you'd do. If you got told the right colour, you use it, and if not, you guess.
> But this is exactly what you expect when you ask intelligence (machine or human) to fill in missing information.
I haven’t read the blog post since it’s currently hugged to death (so I’m admittedly filling in missing information), but one could alternatively expect the intelligence to research the correct missing information, and to report back if it is unable to.
Your last sentence is the important one, yeah? On a spectrum of intelligence or expertise, the higher end would be defined (ish) by knowing the right color or having better guesses anyway.
Like someone restoring the Sistine Chapel would ideally have some of: knows what it looked like before because they've been working on it for thirty years, because they've been researching and/or have access to photos or some other historical documents of it's priors, because they're educated on the types of materials and pigments from the time, maybe they know something about the daily use/tourism of the place and how/whether it should change some parts of the restoration. In the absence of those things, then infer that something should be purple-ish because it's blue to the left and red to the right. This is good, but perhaps below great. Maybe the gap should have a hard line between the two colors or maybe there was an object in there.
And of course plenty of people maybe just don't care, aren't getting paid enough, don't have enough discipline to get the right purple, or are playing to their supervisor's eye rather than their trained intuition. The last one having a lot to do with the notion of "plausible". This effort aspect is I guess different, but still related/adjacent to what comes out of or how that "intelligence" is deployed.
This reminds me of an old problem I used to run into with duplicate data.
Duplicates are not always what they seem - duplicates have a relative quantity of how duplicate they are, for example - 100% duplicate identical is what it sounds like, but then what is a 50% duplicate? Well.. that could be a duplicate where semantic meaning that only a human is aware of (missing information) could be used to determine that these two datum are in fact duplicates in different forms, where 50% is identical, but the other 50% is semantically identical. Or 50% truly identical could, in some case, be "good enough" to be considered identical. These are just examples.
Why this example is relevant is because, when deduplicating data, we often have to consider which value(s) we keep as the true singular representation.
The problem is, long after the data is created, this may be impossible to do.
The solution? Often times it is no better than flipping a coin and guessing, i.e. randomly keeping one datum and discarding what we consider to be the duplicate. Or you can try to be fancy and merge the data or choose, based on some rule, what you assume is the data with the highest valid signal and discard or ignore the noise.
This happens every day in practical information systems with messy data.
I always come to the conclusion that in the end, one cannot truly replace missing signal precisely - one can only infer the correct signal. If the information was not recorded and or compressed or corrupted beyond recoverability, there is no way to get it back - only to infer or guess with some degree of confidence, and accept or reject the outcome.
This to me actually clarifies something deeply about the conversion between the real world to the digital one as data, and the conversion of data into information- it is a real physical process.
In my mind it makes clear that the recording of data (or generation of data) is a real physical process of transfer, from an instrument making observations to physical bits in a physical system, which become the raw data.
Without the raw, precise, complete data, your observation is lossy, and therefore everything above it is as well- e.g. information/interpretation, etc.
This isn't a negative. If stated correctly, the above just helps to explain how we imperfectly navigate through the real world with our tools.
All the same, we can do some pretty amazing things in the absence of data, even if some of it is "false" or "wrong".
I spent some time working in the data industry and was fascinated with this topic as well
dont forget the data itself is already a compression of reality. text data has lost tone, situational context, emphasis, etc.
then youve also got different data that yields the roughly same information that isnt exact but serves the same purpose. for example, birthday vs age; one can be used to infer the other although it is more accurate moving in one direction than the other.
youve got resolution of data. for example, A specific hexcode value vs "deep navy blue" vs "blue". sometimes the signal is low resolution but the data type is high resolution and you end up with some tells like when a lot of date data is the 1st of the month or 1st day of the year; implying the signal was at the month level or year level, no day signal available. which to call back an earlier example, can be caused by getting a signal for age but storing it in a birthday data type field
to your point, often the "data" signal is not a single field but set, and then youve got your source updating every month or year, and its not always the same format or exact set, maybe it was collected differently or two or more different sources contributed. its hard to apply any rules without assessing the type of data being considered. newer data that is missing older data, sometimes you carry the old pieces forward and sometimes you dont. sometimes you change the shape of the data by inferring events. job title changed? well the old job title isnt wrong data, its just not their current title and now can be moved to a past job titles" field. maybe past job titles arent something that add value to your use case and it can be dropped, etc.
This reminds me of how a lot of older TV shows and movies had an 'enhance' button that could somehow magically take a tiny, blurred image and blow it up to a level of detail where even the most minor things could be seen again. Like a suspect's eyes in the reflection of a window that looked like it was maybe 4 pixels in size in the original image.
The AI examples and reasoning in the article are exactly why this can't work, and is so unrealistic. The info isn't just hidden away in the info in a way some tool can figure out what it is with another effort, it's simply not present entirely. Any tool that tries to enhance a low quality image or video is just going to be making up the missing data.
> Any tool that tries to enhance a low quality image or video is just going to be making up the missing data.
So there is some things that instinctively don't sound correct here.
A frame may not contain the info to enhance, maybe it's too blurry... but a series of frames can (I'm pretty sure) between them have enough info to deblur an object in the background, it really helps to know the focal length, and distance maps...
My only point is information in a video is more than the sum of the information of it's frames, it also captures delta, not just from frame to frame, but relevant frame to relevant frame. That's a fair bit to work off of.
I do agree with our general point when information is 'absent'. I disagree how much absent information is in a video which can't be reconstructed by this delta. It certainly isn't zero, or 100%.
Inventing colors from a grayscale is a good example of absent information and of where this goes awry. Deblurring/blurring operations, less so.
> but a series of frames can (I'm pretty sure) between them have enough info to deblur an object in the background
You're probably thinking of geometrical super-resolution, such as gigapixel photography. Blurring is the discarding of high-frequency information, and only a very very small part of this information can be recovered using techniques like this (as in, so little that you wouldn't be able to notice it).
> My only point is information in a video is more than the sum of the information of it's frames,
It's actually less than the sum of the information of it's frames. An off-the-shelf lossless compression algorithm can give you an upper bound for the amount of information present in a video file.
I'm genuinely curious what you thought "AI restoration" was doing if the fact that it added details that it thought were reasonable was surprising to you
Yeah, this is fundamentally expected behavior but the results can vary wildly based on how much of a gap there is between available information and the 'resolution' of the desired output.
I feel like managing the level of information available and judging if its sufficient to make an extension of details lead to a reasonable outcome is the responsibility of the human/user more than the AI. If half the image is destroyed, it is less of a restoration and more of an imagination but the process is the same as if it just had some scuff marks to buff out. Its on the person using the tool to recognize the extent of impact that using the tool is having and to communicate that extent when discussing the results.
otherwise i could give the "restoration process" a torn corner of a photo from my great grandmas house and after it completely makes up a photo i call it restored
But this is exactly what you expect when you ask intelligence (machine or human) to fill in missing information. It picks some plausible, middle-of-the-road filler. If you were a restoration artist with an airbrush (or photoshop) with just the instruction to make the picture nicer, that's what you'd do. If you got told the right colour, you use it, and if not, you guess.
> But this is exactly what you expect when you ask intelligence (machine or human) to fill in missing information.
I haven’t read the blog post since it’s currently hugged to death (so I’m admittedly filling in missing information), but one could alternatively expect the intelligence to research the correct missing information, and to report back if it is unable to.
Your last sentence is the important one, yeah? On a spectrum of intelligence or expertise, the higher end would be defined (ish) by knowing the right color or having better guesses anyway.
Like someone restoring the Sistine Chapel would ideally have some of: knows what it looked like before because they've been working on it for thirty years, because they've been researching and/or have access to photos or some other historical documents of it's priors, because they're educated on the types of materials and pigments from the time, maybe they know something about the daily use/tourism of the place and how/whether it should change some parts of the restoration. In the absence of those things, then infer that something should be purple-ish because it's blue to the left and red to the right. This is good, but perhaps below great. Maybe the gap should have a hard line between the two colors or maybe there was an object in there.
And of course plenty of people maybe just don't care, aren't getting paid enough, don't have enough discipline to get the right purple, or are playing to their supervisor's eye rather than their trained intuition. The last one having a lot to do with the notion of "plausible". This effort aspect is I guess different, but still related/adjacent to what comes out of or how that "intelligence" is deployed.
This reminds me of an old problem I used to run into with duplicate data.
Duplicates are not always what they seem - duplicates have a relative quantity of how duplicate they are, for example - 100% duplicate identical is what it sounds like, but then what is a 50% duplicate? Well.. that could be a duplicate where semantic meaning that only a human is aware of (missing information) could be used to determine that these two datum are in fact duplicates in different forms, where 50% is identical, but the other 50% is semantically identical. Or 50% truly identical could, in some case, be "good enough" to be considered identical. These are just examples.
Why this example is relevant is because, when deduplicating data, we often have to consider which value(s) we keep as the true singular representation.
The problem is, long after the data is created, this may be impossible to do.
The solution? Often times it is no better than flipping a coin and guessing, i.e. randomly keeping one datum and discarding what we consider to be the duplicate. Or you can try to be fancy and merge the data or choose, based on some rule, what you assume is the data with the highest valid signal and discard or ignore the noise.
This happens every day in practical information systems with messy data.
I always come to the conclusion that in the end, one cannot truly replace missing signal precisely - one can only infer the correct signal. If the information was not recorded and or compressed or corrupted beyond recoverability, there is no way to get it back - only to infer or guess with some degree of confidence, and accept or reject the outcome.
This to me actually clarifies something deeply about the conversion between the real world to the digital one as data, and the conversion of data into information- it is a real physical process.
In my mind it makes clear that the recording of data (or generation of data) is a real physical process of transfer, from an instrument making observations to physical bits in a physical system, which become the raw data.
Without the raw, precise, complete data, your observation is lossy, and therefore everything above it is as well- e.g. information/interpretation, etc.
This isn't a negative. If stated correctly, the above just helps to explain how we imperfectly navigate through the real world with our tools.
All the same, we can do some pretty amazing things in the absence of data, even if some of it is "false" or "wrong".
I spent some time working in the data industry and was fascinated with this topic as well
dont forget the data itself is already a compression of reality. text data has lost tone, situational context, emphasis, etc.
then youve also got different data that yields the roughly same information that isnt exact but serves the same purpose. for example, birthday vs age; one can be used to infer the other although it is more accurate moving in one direction than the other.
youve got resolution of data. for example, A specific hexcode value vs "deep navy blue" vs "blue". sometimes the signal is low resolution but the data type is high resolution and you end up with some tells like when a lot of date data is the 1st of the month or 1st day of the year; implying the signal was at the month level or year level, no day signal available. which to call back an earlier example, can be caused by getting a signal for age but storing it in a birthday data type field
to your point, often the "data" signal is not a single field but set, and then youve got your source updating every month or year, and its not always the same format or exact set, maybe it was collected differently or two or more different sources contributed. its hard to apply any rules without assessing the type of data being considered. newer data that is missing older data, sometimes you carry the old pieces forward and sometimes you dont. sometimes you change the shape of the data by inferring events. job title changed? well the old job title isnt wrong data, its just not their current title and now can be moved to a past job titles" field. maybe past job titles arent something that add value to your use case and it can be dropped, etc.
I say "f*ck you; pay me" when the app begs me for my data; that's what.
(sorry for the censored swear)
Maybe you forgot to put "Enhance" in the prompt?
/s