Can someone ELI5? I understand that a person can now generate an image with the same hash as an illegal image (such as child porn), but I don't understand how they can get it on someone's phone and I don't understand why someone would get in trouble for an image, when finally examined, that is clearly not child pornography.
A vulnerability by itself is not that dangerous, but in combination with a sophisticated attack, or another vulnerability can be disastrous. State actors have the resources to exploit a number of unknown bugs in combination with this collision to have Apple's systems flag persons of interest.
This, combined with human error during the manual review process might result in someone getting reported. Seeing as twitter (and other social media sites) jump on the bandwagon whenever someone gets accused of being a pedophile, this might destroy someones life.
The entire story might seem a bit to far fetched, but based on past events, you never know how bad something 'simple' as a hash collision can be.
No one is going to send gray blobs, they will be finding legal porn (like pussy close ups, tongue pics, whatever) and then disturbing it to trigger a CSAM hit.
The low res derivative will match, perhaps even closely, because pussy closeups look similar to an apple employee when its grayscale 64 by 64 pixels (remember: it's illegal for Apple to transmit CSAM, so it must be so visually degraded to the point where it's arguably not visual).
The victim will get raided, be considered a paedophile by their workplace, media, and family, and perhaps even go into jail.
The attacker in this case can be users of Pegasus unhappy with a journalist.
Ok, so you posit an attacker could find/generate 30+ pictures that are
1. accepted by innocent user,
2. flagged as known CSAM by NeuralHash,
2b. also flagged by the second algorithm Apple will run over flagged images server side as known CSAM,
3. apparently CSAM in the "visual derivative".
That strikes me as a rather remote scenario, but worth investigating. Having said that, if it's a 3-letter adversary using Pegasus unhappy with a journalist, couldn't they just put actual CSAM onto the journalist's phone? And couldn't they have done that for many years?
Incorrect. A Chinese state actor can't just go around imprisoning journalists they don't like in America, but they can now do this through planting child porngoraphy via remote malware (Pegasus) and watch their enemies get arrested by the US Feds.
Disagree. It isn't just jurisdiction. It is resource access. If the Chinese gov't were coming after little old me right now, I'd be properly worried, even though I'm safely within the boundaries of the US.
It's not true at all. You're assuming state actors are in the same jurisdiction. This isn't always the case - think an oppressive authoritian regime wanting to get an American journalist arrested for child pornography.
It's always possible before, but client-side CSAM detection and alerting has weaponised this.
Previously, you always had to somehow alert an unfriendly jurisdiction. Now, you just use malware like Pegasus to drop CSAM, whether real or disturbed from legal porn, and watch as Apple tips off the Feds on your enemies.
State actors will just Gitmo you, without all this wasteful effort on hashes. This system offers no benefit sufficient to make it worth their time if they want to cull you from the population somehow.
No, China can't just Gitmo an American journalist on American soil.
But now China can send some legal pornography (eg closeup pussy pictures), disturbed to match a CSAM hit, to a journalist they don't like and get them in jail.
Why can't China do this before? Because previously, they'd still need to tip off authorities, which has an attribution trail and credibility barrier. Now, they can just use Pegasus to plant these images and then watch as Apple turns them into the Feds. Zero links to the attacker.
The scenario you describe has already been extant for the past ten years. Unreported zerodays could have been used at any time to inject a CSAM hit into someone's camera roll, way back in time where they wouldn't see it, in order to get them investigated. Their phone would have uploaded it to iCloud or Google Photos or Dropbox Whatever and the CSAM detections at each place would have fired off. No need for any of this fancy AI static nonsense.
I know of zero instances of this attack being executed on anyone, so apparently even though it's been possible for years, it isn't a material threat to any Apple customers today. If you have information to the contrary, please present it.
What new attacks are possible upon device owners when the CSAM scanning of iCloud uploads is shifted to the device, that were not already a viable attack at any time in the past decade?
This conversely means that all illegal content can be freely texted and this system won't even catch the distribution of CP unless those pictures are imported into the photos library and icloud updates enabled.
There's a pretty good chance that it was inevitably going to get expanded to handle pictures arriving at the phone through other means.
Whatsapp has a feature where all images are automatically saved to device as they are received. If automatic iCloud upload is on, then all the conditions are there for a person to innocently click on a spammy Whatsapp message, see a bunch of nonsense grayscale images, and continue on with their day--not realizing they are now being monitored for CSAM.
For one, you can't know if that's true as the image could have been manipulated to appear as such. For example you wouldn't know if a kind of steganography has been used to hide image in an image and that neuralhash picked on a hidden image.
> but I don't understand how they can get it on someone's phone
There is many vectors. For example you can leave phone unattended and someone can snap a picture of an image or since a collision may look innocent to you, you would overlook it in an email etc...
> For example you wouldn't know if a kind of steganography has been used to hide image in an image and that neuralhash picked on a hidden image.
How would NeuralHash pick a "hidden image"? It only uses the pixels of the image to get the hash. Any hidden image in the metadata would not even be picked up and no amount of steganography can fool NeuralHash.
> There is many vectors. For example you can leave phone unattended and someone can snap a picture of an image or since a collision may look innocent to you, you would overlook it in an email etc...
As iterated elsewhere in this thread, random gibberish pixels colliding with CSAM would definitely not be useful in incriminating anyone. The manual process would catch that. Also, if the manual process is overloaded, I'm pretty sure basic object recognition can filter out most of the colliding gibberish .
The steganography is not to fool the NeuralHash, but to fool the viewer.
The NeuralHash would "see" the planted image, but for the viewer it would appear innocent.
I am trying to say that a person reviewing image manually, without special tools will not be able to tell if the image is a false positive and would have to report everything.
