The 3 Fs of Spotting Photo Fraud
Written by Kevin Connor & Hany Farid   

Recent advances in digital imaging allow for the creation of visually compelling photographic fakes. The undermining of the public’s trust in photographs has impacted law enforcement, national security, the media, advertising, e-commerce, and more.


Advertisement

Fortunately, the field of photo forensics has emerged to help restore some trust in digital photographs.

It would be terrific to have a single technique that could reliably tell you whether a photograph can be trusted. The reality, however, is that a large toolbox of techniques is required for sniffing out various clues to an image’s history. We characterize such forensic techniques into one of three main categories: Files, Footprints, and Flaws. By applying a variety of forensic techniques in each of these three categories, you can objectively assess the veracity of an image and sniff out photo fraud.

Files

Three Fs of Spotting Photo Fraud FourMatch evidence forensic science Kevin Connor Hany Farid Fourandsix TechnologiesThe first type of clue has very little to do with the content of the image itself, but instead concerns itself with the way an image has been packaged into a file. More specifically, these clues relate to the file format used to store the image, as well as the additional information stored in the file along with the image.

Nearly all cameras save images in the JPEG format. Though this is a universal format, it permits a nearly endless variety of ways for a JPEG file to be constructed and stored. This variety results in a highly distinctive set of signatures from each hardware and software product. Because modern-day devices store the camera’s make and model in an image’s metadata, an image’s signature can be compared against a database of known camera-specific signatures. When a signature correctly matches a known signature from the device that captured the photo, it is highly likely that the image is an unedited original.

There is no single definition of what composes an image signature. In our product FourMatch, we marry four distinctive aspects of the JPEG file to define a signature:

Three Fs of Spotting Photo Fraud FourMatch evidence forensic science Kevin Connor Hany Farid Fourandsix TechnologiesThree Fs of Spotting Photo Fraud FourMatch evidence forensic science Kevin Connor Hany Farid Fourandsix Technologies
Figure 1—Here are the results from a FourMatch analysis used to determine if a JPEG image is an original.

1) The first, and simplest, component is simply the image dimensions. The dimensions distinguish between devices with varying sensor size and resolution settings. If you are presented with an image that was recorded from a particular camera, but the size of the image is different than what you know the camera can capture, then it is a safe bet that the image is not an original.

2) JPEG is known as a lossy compression scheme, which means that it reduces the quality of the recorded image in return for a reduced file size. There are hundreds of parameters that specify precisely how this compression is achieved. Every camera and software manufacturer customizes these parameters to suit their needs. Due to the large variety of possible parameters, the JPEG compression settings are the most distinctive aspect of the image signature.

3) Many devices create and store a thumbnail sized version of the full-resolution image. This thumbnail is used for quick previewing of the image. The thumbnail is typically stored as a second JPEG with its own compression values within the main JPEG file. Thus, the third component of the signature is the thumbnail size and compression parameters.

4) Modern-day devices store a variety of image and device data in the image’s EXIF metadata. The EXIF standard specifies five main directories into where this data is stored. The standard does not, however, specify precisely what information should be stored there. As such, each camera manufacturer is free to store as much or as little information as desired. The fourth component of the signature is the number of data fields stored in each of these five directories.

Of course, to make meaningful use of the image signature, you must be able to compare it against verified original signatures from the camera that first captured the image. Note also that, because cameras generally have multiple options for image dimensions, compression, and metadata, there can be dozens of possible original signatures for a single camera. FourMatch simplifies this work by including a database of signatures from thousands of devices and performing the comparison automatically. In Figure 1, you can see the results of two of these comparisons. On the left, the image signature exactly matches a known signature and is therefore most likely an original, while on the right the image signature is not a match. Moreover, parts of the signature match the characteristics of Adobe editing software, suggesting that this file has likely been opened and recompressed by an Adobe product sometime after it was captured.

Footprints

Just as a criminal in a crime scene may leave behind traces, so image-editing tools may leave tell-tale clues to their usage. Every digital image is a collection of pixels defined by numerical values that represent individual colors. Each tool in Photoshop works by combining and manipulating those numerical values in different ways. Inevitably, each type of editing operation can leave behind different statistical traces of what was done.

Three Fs of Spotting Photo Fraud FourMatch evidence forensic science Kevin Connor Hany Farid Fourandsix Technologies
Figure 2—Here you can see the results of clone detection applied to a photo depicting an Iranian missile launch. When the third missile from the left failed to launch, it was later digitally added to the image. (Photo: Agence France-Presse)

For example, cloning or content-aware fill can result in patterns of pixels that are repeated in multiple locations in the same image. When performed carefully, it can be very difficult to visually detect such a manipulation. In theory, detecting a cloned region should be easy: simply find any two regions that are suspiciously similar in appearance. In practice, however, this is a hard problem because of the computational cost of analyzing all pairs of image regions of unknown shape and size. In addition, some regions in an image naturally appear similar, but are not necessarily the result of cloning (for example, a blue sky), so it is critical to distinguish between these regions and those that are a result of cloning. Recent advances have led to efficient and accurate forensic techniques for detecting image cloning. Shown in Figure 2, for example, is the result of a forensic clone detection on the infamous Iranian missile photo. In addition to finding parts of the cloned missile and smoke, there were no false matches on the highly self-similar sky and ground.

Flaws

Three Fs of Spotting Photo Fraud FourMatch evidence forensic science Kevin Connor Hany Farid Fourandsix TechnologiesThree Fs of Spotting Photo Fraud FourMatch evidence forensic science Kevin Connor Hany Farid Fourandsix Technologies
Figure 3—The shadows in the photo on the left are authentic because the shadow constraints converge to a single point, while the shadows in the right-hand photo are physically impossible.

Though the increasing sophistication of editing tools has made image manipulation much easier, truly emulating reality remains difficult. More importantly, we are not as adept as we may think in spotting many errors in realism. Studies have shown that people are particularly insensitive to errors in lighting, shadows, and reflections. Thus, when someone is editing a photo and simply eyeballs things to get their shadows right, there’s a good chance they’ll get them wrong. Armed with the proper techniques, you can spot these and other flaws in realism in order to identify fraudulent photos.

For example, the geometry of cast shadows is dictated by the 3D shape of an object, its surrounding surfaces, and the source (or sources) of light. It is surprising, therefore, that there is a simple and intuitive 2D image-based geometric analysis that can verify the authenticity of shadows. Locate any point on a shadow and its corresponding point on the object, and draw a line through them. Repeat for as many clearly defined shadow-and-object points as possible. As you do this, you will find that all of the lines should intersect at one point—the location of the illuminating light (Figure 3).

Why does this work? Since light travels in a straight line, a point on a shadow, its corresponding point on the object, and the light source must all lie on a single line. Under the rules of perspective projection, straight lines project to straight lines; this basic geometry is preserved in the 2-D image of a scene. Notice that this constraint holds regardless of the shape or orientation of the surface onto which a shadow is cast.

Three Fs of Spotting Photo Fraud FourMatch evidence forensic science Kevin Connor Hany Farid Fourandsix Technologies
Figure 4—When standing in front of a mirror, remember that light travels in straight lines.

Somewhat surprisingly, the analysis of shadows can also be applied to the analysis of reflections in glass, water, or any shiny surface. To see why, consider standing in front of a mirror and looking at your reflection. As shown in Figure 4, an imaginary straight line connects each point on your body with its reflection. These lines are perpendicular to the mirror’s surface and are parallel to one another. When photographed at an oblique angle, however, these imaginary lines will converge to a single point.
This suggests a simple forensic technique for verifying the integrity of reflections: Locate any point on an object and its corresponding point on the reflection and draw a line through them. Repeat for as many clearly defined object-and-reflection points as possible. As you do this, you should find that all of the lines should intersect at one point (Figure 5).

 

Three Fs of Spotting Photo Fraud FourMatch evidence forensic science Kevin Connor Hany Farid Fourandsix Technologies
Three Fs of Spotting Photo Fraud FourMatch evidence forensic science Kevin Connor Hany Farid Fourandsix Technologies
Figure 5—The reflections in the top photo are authentic because the reflection constraints converge to a single point, while the reflections in the bottom photo are physically impossible.

Conclusions

The three Fs provide a convenient scaffolding onto which you can classify most forensic image-analysis techniques. Each technique has its strengths and weaknesses. A collection of many such tools, however, provides a forensic image analyst with a powerful set of tools to determine if an image is real or fake. You can read more about various forensic techniques on our blog.

About the Authors

Kevin Connor has more than 20 years experience in the computer software industry. He spent 15 years guiding the development of the Photoshop product line at Adobe. His experience led him to found Fourandsix Technologies with Hany Farid, where he is now CEO, with the goal of providing tools and services that restore trust in digital media.

Hany Farid received his undergraduate degree in Computer Science and Applied Mathematics from the University of Rochester in 1989. He received his PhD in Computer Science from the University of Pennsylvania in 1997. Following a two-year post-doctoral position in Brain and Cognitive Sciences at MIT, he joined the faculty at Dartmouth in 1999, where he is a Professor of Computer Science. Hany is also the Chief Technology Officer and co-founder of Fourandsix Technologies, Inc.

View this article in its original format in our March-April 2013 Digital Edition

 
< Prev   Next >






Editorial

ONE OF THE CHALLENGES of writing and editing a magazine is telling a story in a relatively small amount of space. Sometimes it seems like there is never enough room to say everything that needs to be said. I find myself making tough decisions about what parts stay and what parts go.

Read more...