openCV2, Qt and Histogram back projection--00

  This post will talk about how to implement histogram back projection with openCV2.First of all,
the algorithm was proposed by Michael J. Swainiand Dana H. Ballard in their paper.
Besides, I gain many helps from abid rahman, the author of the blog OpenCV Python Tutorials,
he helps me understand what is the algorithm doing about, the core of the codes in this post are translated by the codes of his tutorial--Histograms - 4 : Backprojection.His blog is very good, If you are interesting about computer vision, please give it a visit.

   I prefer to re-implement the algorithms I don't quite understand for two reasons.

1. This is a good method to study how do the algorithms work
2. Many algorithms are anti-intuitive of humans, if I don't write them down, when I forget how do they work in the future, I need to spend more times to refresh what do they do.
3. Re-implement the algorithm can help me learn how to use the algorithm, like when should I use them, their bottleneck, use conditions and so on.

   In this tutorial, I would not talk about how to combine openCV2 and Qt5 together but leave it to the next post.

   This post will focus on the algorithm only(no GUI).The images use in this post could download from target, region of intereset. Source codes could download from github.

  

Custom histogram back projection

   The principle of histogram back projection is measure the probability between the color histogram of the target and the region of interest(ROI, the object you want to know where is it in the target).
   The blog OpenCV Python Tutorials use two dimensions histogram(hue, saturation) to measure the ratio, I will compare about the result between one dimension histogram(hue) and two dimensions histogram(hue, saturation) in this post.Full source codes are uploading to github, this post would talk about the concepts only.

Step 1--load the region of interest and target

cv::Mat model = cv::imread("/Users/Qt/program/"
                           "experiment_apps_and_libs/"
                           "blogsCodes/pic/rose_region.png");
cv::Mat input = cv::imread("/Users/Qt/program/"
                           "experiment_apps_and_libs/"
                           "blogsCodes/pic/rose_small.png");

About the model(region of interest--ROI), we should make sure the selected ROI as big as possible, because this could increase the quality of the probability map.

Step 2--get histograms

Now we need to get the histograms of the model_ and input_.Why hsv but not bgr?
Because hsv could separate the color from intensities, the intensity of b, g, r may be the same,
but hue could separate which one is red, green or blue.

template
inline
cv::MatND calc_histogram(std::initializer_list<cv::Mat> images, 
                         std::initializer_list<int> channels,
                         std::initializer_list<int> hist_sizes, 
                         std::array<float[2], N> const &ranges,
                         cv::InputArray const &mask = cv::Mat(), 
                         bool uniform = true,
                         bool accumulate = false)
{//...implementation could be found on github}

cv::Mat convert_to_hsv(cv::Mat &input)
{
  if(input.channels() == 4){
        cv::cvtColor(input, input, CV_BGRA2BGR);
    }

  cv::Mat hsv;
  cv::cvtColor(input, hsv, CV_BGR2HSV);

  return hsv;
}

inline cv::MatND get_hist(cv::Mat &input, int histogram_dimension)
{        
    if(histogram_dimension == 2){              
        return OCV::calc_histogram<2>({input}, {0, 1}, {180, 256}, 
               {{ {0, 180}, {0, 256} }});
    }else{
        return OCV::calc_histogram<1>({input}, {0}, 
               {180}, {{ {0, 180} }});
    }
}

cv::Mat input_hsv = convert_to_hsv(input);
cv::Mat model_hsv = convert_to_hsv(model);
//generate 2 dimensions histogram(hue, saturation)
cv::Mat input_hist = get_hist(2);
cv::Mat model_hist = get_hist(2);

Step 3--find ratio

cv::Mat ratio = model_hist / (input_hist + 1);

Step 4--generate probability map

This part is a little verbose, to ease the pain of writing hand made loops, I develop some generic algorithms  base on the principle on this post Generic algorithm and openCV.

//a generic for loop which iterate all of the pixels of the image
template<typename T, typename BinaryFunc, typename Mat>
BinaryFunc for_each_channels(Mat &&input_1, Mat &&input_2, 
                             BinaryFunc func)
{
    //...implementation could be found on github
}

cv::Mat generate_probability_map()
{                   
    std::vector<cv::Mat&gt hsv;
    cv::split(input_hsv, hsv);
    //the cv::Mat create by create() is 100% continuous
    probability_map_.create(hsv[0].size(), CV_32F); 
    auto map_ptr = probability_map_.ptr<float>(0);
    OCV::for_each_channels<uchar>(hsv[0], hsv[1], 
    [&](int hue, int saturation)
    {
        //I use ratio_.ptr to access the pixels of ratio_, because 
        //I can't make sure they are continous
        *map_ptr = std::min<float>(*(ratio_.ptr<float>(hue) + 
                   saturation), 1.0);
        ++map_ptr;
    });
   
    return probability_map_.clone();
}

Step 5--blurring, have the effect to "close" the hole

cv::Mat disc = cv::getStructuringElement(cv::MORPH_ELLIPSE, 
               cv::Size(17, 17));
cv::filter2D(probability_map, probability_map, -1, disc);
cv::normalize(probability_map, probability_map, 0, 255, CV_MINMAX);
probability_map.convertTo(probability_map, CV_8U);

Step 6--separate target and background

cv::Mat mask;
cv::threshold(probability_map, mask, 50, 255, cv::THRESH_BINARY);
cv::Mat result;
input_.copyTo(result, mask);

last step--save the result

cv::imwrite("result.png", result);
cv::waitKey();

Results of custom histogram back project

Result of two dimension histogram

Result of one dimension histogram

calcBackProject of openCV

openCV2 provide us a built in function cv::calcBackProject to find the probability map by histogram back projection. Step 1 and step 2 are same as previous, I will start from step 3

Step 3--find probability map by calcBackProject

//encapsulate calcBackProject of openCV2
template
inline
cv::Mat calc_back_project(std::initializer_list<cv::mat> images, 
                          std::initializer_list<int> channels, 
                          cv::InputArray hist,
                          std::array<float n=""&gt ranges, 
                          double scale = 1, bool uniform = true)
{//...implementation could be found on github}                          

cv::Mat probability_map = OCV::calc_back_project<2>(
                          {input_hsv_}, 
                          {0, 1}, 
                          model_hist, 
                          {{ {0, 180}, {0, 256} }});
                          

This is the only different when compare to the previous solution.

Step 4--blurring, have the effect to "close" the hole

cv::Mat disc = cv::getStructuringElement(cv::MORPH_ELLIPSE, 
               cv::Size(5, 5));
cv::filter2D(probability_map, probability_map, -1, disc);
cv::normalize(probability_map, probability_map, 0, 255, CV_MINMAX);
probability_map.convertTo(probability_map, CV_8U);

Step 5--separate target and background

cv::Mat mask;
cv::threshold(probability_map, mask, 50, 255, cv::THRESH_BINARY);
cv::Mat result;
input_.copyTo(result, mask);

last step--save the result

cv::imwrite("result.png", result);
cv::waitKey();

Results of calcBackProject

Result of two dimension histogram

Result of one dimension histogram