Exposing Digital Forgeries by Detecting Inconsistent Geometry

On April 25, 2014, there was an conflict between the legislator Cheng-Yuan Tsai and several anti-nuclear protesters in front of the Legislative Yuan in Taiwan. You could find the background story of the event in Mr. Tsai's wikipage.

What causes my interest is the following image pair. The first one, posted by Cheng-Yuan Tsai, supports Tsai's claim that there was a man (presumably an anti-nuclear protester) who broke the windshield when he was about to leave the Legislative Yuan around 4:15 PM. 

Image source: a facebook post from the legislator Cheng-Yuan Tsai on April 25 

In the evening (8:12 PM), the Chinatimes released several photos captured during the conflicts. Here I show one of photos below. You could see the visible highlights on the windshield. While the report did not mention the broken windshield, this photo seems to implicitly verify the claim by Mr. Cheng-Yuan Tsai. 

Image source: Chinatimes released at 8:12 PM on April 25

However, from the video below, it shows the contrary. This makes me wonder if the photo had been manipulated with the intention to cover the truth.



So, how to verify whether the image from Chinatimes is manipulated/photoshoped or not? My first attempt is to try Hany Farid's software izitru, which uses automated forensic analysis techniques (e.g., sensor pattern, JPEG ghost, double JPEG detection, and JPEG structure analysis) to certify unmodified digital camera images and give credibility to the captured photos. Yet, both of the images are flagged as "Potential file modification" because they have been re-saved since initial capture.
Well, let's try to address this problem from another perspective then. The main idea is that, if both photos are untouched, then there should be clear geometric relations between the two images (since they both captured the same car).

Thus, I check whether the spatial points between the two images satisfied the Epipolar constraint.


"Epipolar geometry is the geometry of stereo vision. When two cameras view a 3D scene from two distinct positions, there are a number of geometric relations between the 3D points and their projections onto the 2D images that lead to constraints between the image points.These relations are derived based on the assumption that the cameras can be approximated by the pinhole camera model." - Wikipedia

Image source: Epipolar geometry from Wikipedia

The figure below best illustrates the Epipolar geometry. Basically, a point in the left image must have its corresponding point somewhere on the line in the right image. Similarly, a point in the right image could find its correspondence point along a line in the left image. Such relationship could be captured compactly by   for all pairs of corresponding points (x' and x), where F is the fundamental matrix.

Image source: Epipolar geometry from Wikipedia

You may be interested in learning more about it by listening to the Fundamental Matrix Song. XD

To estimate the fundamental matrix that relates points in this wide baseline stereo images, I first manually labeled a set of spatial correspondence pairs. I then solve the fundamental matrix using the off-the-shelf Normalized Eight-Point algorithm with robust estimation using the RANSAC algorithm. The figure below shows the inlier point matching between two frames.

Once we got the fundamental matrix, we could now verify whether the position of "highlight (presumably where the shield was hit)" satisfy the epipolar geometry constraint.

I first clicked on the position of the highlight in the left image (the red circle). Through the fundamental matrix, we get a red line in the right image. The red line suggests that where should we look for the corresponding point of the hole in the left image. If the right image was not manipulated, the red line should pass through the highlight position. Yet, you could see that there is slight inconsistency between the highlight positions in two images.

Similarly, we could click on the highlight position in the right image (the yellow circle). We show the yellow line in the left image to check if the corresponding point lies on the line. Again, there is a slight geometric inconsistency between the two images. 

So, from the analysis, could we say that the image from the Chinatimes is manipulated/photoshoped? Well, because the noisy manual labeling and the limited image resolution, nothing is absolutely certain. As for now, we could only say that it is very likely that the image had been manipulated. But, how likely? We will need to do more work on quantifying the input errors and and determine its the uncertainty as described in this paper. 



Anyway, it could have been much easier if Mr. Cheng-Yuan Tsai could provide the recordings from his vehicle video recorder. Orz