OpenCV Inpaint

Parallelizing OpenCV’s Inpaint algorithm, with integration into Android demo app

Vector Fabrics’ Pareon is a tool to analyze and optimize source code for multi-core processors. As a demo, we have optimized an image processing algorithm written in C++ and made the results available as an Android App in Google Play. The optimization resulted in a speedup of up to 4x on quad-core ARM architectures, and 2x on dual-core architectures.

The algorithm

OpenCV

OpenCV is a widely known open source library for real-time Computer Vision. It is written in C and C++ and contains a lot of image processing code, including conversions between color models, calculating cross-correlations between images, stitching maps or finding edges or other image features that can help with enabling vision projects for computers.

Inpaint

OpenCV comes with an algorithm for Inpainting – a mechanism to remove scratches or other artifacts from a photo. In the OpenCV case, this is implemented by cloning pixels outside the scratch area into the damaged area, which is identified by a 1-bit mask. Inpaint is a CPU-intensive algorithm, where the number of cycles spent is related to the number of damaged pixels – its compute time does not depend on the size of the image.

Android Demo

The OpenCV library itself already comes with support for the Android platform, through the Android NDK. It is a so-called 'native library', as it is written in C++ and compiled for a specific processor architecture. This is in contrast to regular Android applications, which are written in Java and compiled to a cross-platform byte code.

Vector Fabrics wrote an Android Java application that uses the OpenCV library. The application takes a picture using the camera, then allows you to mimic image artifacts by drawing them on the screen, and then filters these artifacts away by using the Inpaint algorithm.

From single threaded to multicore

Parallelization of the algorithm was done using Vector Fabrics’ Pareon tool. A whitepaper explains the optimization process in more detail and also presents more background on OpenCV and the Inpaint algorithm.

Speedup

The below screenshot comes from a Sony XPeria S phone, running Android 2.3.7. The processor in this phone is a Qualcomm MSM8260, a dual core processor.

In general, a dual core can achieve up to 2.0 speedup, a quad core can achieve up to a 4.0x speedup.