What If AI Image Generation Is Used for Harm…
Translated from the 中文 original.
It started when I scrolled past a gossip thread using photos to take down a celebrity (with receipts). I don’t know whether those photos were real, and I don’t care. But photos like that — you can generate them with AI now, right?
AI-generated images being used as refund evidence or legal evidence is already happening. Maybe in the future courts can decline to accept image data, and merchants can refuse to trust it or adopt stricter processes, but malicious AI-generated images can still be posted to public platforms to steer opinion, spread rumors, smear people…
There are a few potential solutions to this kind of problem.
One is to make AI image generation models not generate celebrities. With the progress of open-source image generation models and the spread of fine-tuning techniques, this approach has already failed. No matter how strong anti-jailbreak measures get, they have nothing to do with open-source models. And this only protects celebrities… what about an ordinary person being smeared? Being lied about?
Another is to make images generated by AI models carry a watermark, like how images generated by Google’s nano banana carry an invisible SynthID watermark. But if open-source models can achieve a similar effect, a lot of AI-generated images will have no watermark. You can’t really force an invisible watermark onto an open-source model — someone will always be able to remove the watermarking part of the model. (If people can strip out even the anti-nsfw tendency baked into LLM weights, it's hard to imagine the watermarking part being un-removable.)
What I think is more reliable is not to label the fake images, but to label the real ones. Give photos taken by phones/cameras a way to verify authenticity and integrity. It can only come down to phone manufacturers giving photos a way to prove their own integrity and authenticity.
Don’t detect the fake images, just verify the real ones? If you can’t prove you’re real, we just assume you’re fake by default.
I did a bit of research, and it turns out there’s already a solution?
There’s this thing called the Content Authenticity Initiative (CAI), which has developed an open-source toolchain that lets an image prove its own edit history, and lets anyone verify that entire image edit history?
Cryptography, my friend.
But the core difficulty for an initiative like this is support from capture devices and the adoption of the relevant tools. Most ordinary people won’t proactively go use tools like these, so whether it succeeds depends entirely on whether the photos taken by phones and cameras come with a similar feature built in.
This tech has been applied to the Leica edition of the Xiaomi 17 Ultra… so the regular version doesn’t have it?
Even though we can already download CAI’s official camera app and try it directly, and use the website they provide to verify, at this stage the ecosystem support looks pretty bad…
They list a lot of partners, but Google isn’t at the table, Apple isn’t at the table, Qualcomm is at the table — but they’ve been on board since the 888 generation, so why is there no splash now? If the phone manufacturers aren’t at the table, all we can say is this initiative still has a long road ahead… https://verify.contentauthenticity.org/
As audio editing models mature, the same problem will reach the audio and video domains too. This CAI seems to support audio and video as well? But do we have to run through this whole process once for every domain?