Image Completion using Planar Structure Guidance

First row: input image (region to fill shown in red). Second row: our results. Input images (from left to right) are from ©Flickr users micromegas, Nicu Buculei, Theen Moy, used under the Creative Common license.

Filling or replacing regions in images with plausibly synthesized content is a common image editing technique. This task, known as image completion, is used in applications ranging from the removal of unwanted objects in personal photos to movie post-production. It is also an important step in many graphics algorithms, e.g., for generating a clean background plate or reshuffling image contents.

While much progress has been made, image completion remains a challenging problem. This is because some amount of higher level understanding of the scene is often required. The state-of-the-art automatic algorithms typically rely on low-level cues; they synthesize an image that locally appears like the known input everywhere.

In this paper (to be presented in SIGGRAPH 2014), we show how image completion can be substantially improved by automatically guiding the low-level synthesis algorithm using mid-level structural analysis of the known region. Our algorithm significantly improves performance for challenging man-made scenes such as those of architecture and indoors. In the absence of any detected structural cues, e.g., for most natural landscape images, our algorithm falls back to standard unconstrained completion, e.g., Photoshop content-aware fill.

Limitations of current state-of-the-art methods. 

Here we show a few examples to visualize the completion process in a coarse-to-fine fashion.

 

Image credit: ©Flickr user Nicu Buculei

 

Image credit: ©Flickr users aigle_dore

Image credit: ©Flickr user cedrennes 

Image credit: ©Flickr user the_dugghouse

Image credit: ©Flickr user sunshinepictures

However, the method still has difficulty on generating satisfactory results on quite a few challenging scenes. Among them, this building in London being demolished to make way for the construction of "The Cheesegrater" is particularly interesting to me. Here, I tried to recovery the deconstruction process. Yet, our algorithm is not able to find the correct demarcation lines between different perspective planes when the unknown region is large.

Image credit: ©Flickr user addictive_picasso

You can find more completion result comparisons below. Please also visit our project page for more information! :D

Here we show sample comparisons with prior works.

   

  

  

  

  

  

  

  

    

          

Failure cases

  

   

References:

• Jia-Bin Huang, Sing Bing Kang, Narendra Ahuja, and Johannes Kopf, Image Completion using Planar Structure Guidance ACM Transactions on Graphics (Proceedings of SIGGRAPH 2014), 33(4), To Appear
• Jia-Bin Huang, Johannes Kopf, Narendra Ahuja, and Sing Bing Kang, Transformation Guided Image Completion, Proceedings of IEEE Conference on International Conference on Computational Photography (ICCP), 2013