perspective correction for quadrilateral markers

    When I am studying the marker base augmented reality, I come to realize that the markers in the images always has perspective distortion.To read the information from the markers correctly, we need to correct the distortion.

     The link show us how to detect the points of the quadrilateral by cv::HoughLinesP, it works well, but what if there are more than one quadrilateral in the image or there are more than four segments we could detected(graph_00)?

graph_00

       If we apply cv::HoughLinesP on the graph_00, there will more than 4 lines(graph_01), so the solution suggested by the link can't work on this case.Apparently, we need another approach to reach our goal.

cv::Mat src = cv::imread("/Users/Qt/program/blogsCodes/"
                         "pic/perspective08.jpg");
if (src.empty()){
   std::cerr<<"can't open image"<<std::endl;
   return;
}
cv::Mat bw;
cv::blur(src, bw, cv::Size(3, 3));
cv::Canny(bw, bw, 30, 90);

std::vector<cv::Vec4i> lines;
cv::HoughLinesP(bw, lines, 1, CV_PI / 180, 70, 30, 10);

for(auto const &data : lines){
    cv::line(src, cv::Point(data[0], data[1]), 
             cv::Point(data[2], data[3]), 
             cv::Scalar(0, 0, 255));
}

graph_01

step 1 : convert the graph_00 to binary image by otsu-threshold

cv::Mat binary;
cv::cvtColor(src, binary, CV_BGR2GRAY);
cv::threshold(binary, binary, 0, 255, cv::THRESH_BINARY + cv::THRESH_OTSU);

graph_02

step 2 : find the contours of binary image

ContourType find_contours(cv::Mat const &binary_input)
{
    ContourType contours;
    cv::findContours(binary_input, contours, 
                     CV_RETR_EXTERNAL, CV_CHAIN_APPROX_NONE);

    return contours;
}
//.....
auto contours = find_contours(binary);

step 3 : find the corners of quadrilaterals(pokers on the table)

CornersType get_corners(ContourType &contours)
{
    //remove the contours size < 100 and size > 100
    OCV::remove_contours(contours, 100, 50000);

    std::vector approx;
    CornersType corners;
    for(auto const &data : contours){
        cv::approxPolyDP(data, approx,
                 cv::arcLength(data, true) * 0.02, true);
        if(approx.size() == 4 && cv::isContourConvex(approx)){            
            OCV::sort_rect_corners(approx);
            corners.emplace_back(std::move(approx));
        }
    }

    return corners;
}
//.....
auto const corners = get_corners(contours);

graph_03

To get the corners, we need to

1 : remove small contours
2 : approximate the contours to a polygon.
3 : the polygon should have 4 points.
4 : it should be a convex
5 : sort the corners by the order of top left, top right, bottom right, bottom left

check github to get the implementation of OCV::sort_rect_corners.The implementation of OCV::sort_rect_corners is base on the link but with some refinement.

step 4 : correct the perspective distortion by cv::warpPerspective

cv::Mat target(150, 150, src.type());
std::vector<cv::point2f> target_points
{
 {0, 0}, {target.cols - 1, 0},
 {target.cols - 1, target.rows - 1}, 
 {0, target.rows - 1}
};
std::vector points;
for(size_t i = 0; i != corners.size(); ++i){
   for(auto const &point : src){
     points.emplace_back(point.x, point.y);
   }
   cv::Mat const trans_mat = cv::getPerspectiveTransform(points, 
                                                       target_points);
   cv::warpPerspective(src, target, trans_mat, target.size());
}

  To project the original poker, we need to declare a new image and two group of points.The first group is the corners point of the original image, the second group is the region we want to project to.In most of the times we would set the points of the second group as large as the new image.Do remember to keep the order of the two group of points same, else the direction of the poker after projection will vary.

graph_04

graph_05

graph_06

graph_07

graph_08

  The codes could download from github. The lib used by this small project could be found at here. Do not need to build the whole library, what you need to do is include the header files and compile the utility.cpp with your project.