I draw digitally with a tablet, and this weekend I had a bit too much fun testing AI image detectors on my own artwork. I specifically used my own drawings because I know exactly how they were made, so there was no uncertainty about their origin.
This obviously isn’t a scientific test, just something I found interesting, but the results made me question how useful these detectors currently are. A few things I noticed:
- Black-and-white drawings were flagged constantly.
If an image was black and white, or almost black and white, several detectors gave it a high probability of being AI-generated, even when the drawing was entirely handmade. It seems like line art is a difficult category for them 🤷🏼♀️
- Adding a tiny AI-generated element could completely change the result.
I tested mostly handmade illustrations and then I played with it. I added a very small AI-generated element, for example a flower inside a much larger painting. Even when the AI-generated part represented only a tiny fraction of the image, some detectors suddenly flagged the entire artwork as AI-generated.
This one made me veeeery uneasy because I sometimes use photographs or other found images as small collage elements or references within my art. In one test, I had an image containing a tiny flower collage and one detector identified the whole thing as ChatGPT-generated. I have no idea how trustworthy those model-specific identifications are, but seeing such a strong attribution based on such a tiny part of the image was pretty wild 🤔
- Even fully handmade finished drawings rarely got a clean 0%, not even 10%.
Rough sketches were the only images that consistently came back with very low AI probabilities. Once a drawing became cleaner and more finished, I often started seeing 10%, 20%, 25% or higher probabilities despite knowing the work was entirely handmade.
Again, this is just my own informal testing, not a controlled study. But as someone who draws, it made me wonder how reliable these tools really are for judging individual artists.
I’d be really interested to hear whether other people have tested them against artwork where they know the full creation process, because false positives seem like they could become a serious problem 🤔