Use person re-id model to identify person do not exist in the data set by c++

    Person re-id comparing two images of person captured under different conditions, recently this field achieve big improvement with the helps of deep learning, but is it good enough to identify person do not exist in the data set? This is the question I want to figure out in this post.

    Let me show you an example before we start.

    The results are not perfect yet, let us hope that better techniques and larger data sets would release in the future. The algorithm itself is very easy, main flows are drawn in pic00

pic00

    For those who wants to read the source codes directly, please go to github, in order to compile it, you will need opencv3.4.2 and mxnet. You can pick every build tools you like, I use qmake in this example.  If you want to know how to reproduce the results, please read on.

1. Download pretrained model of person re-id

    Download pretrained model from here. Precision and mAP of this model perform on market1501 are

top1:0.923100
top5:0.972090
top10:0.984264
mAP:0.797564

     If you want to train it by yourself, please follow the guide of gluoncv, it is quite easy.

2. Download pretrained model of yolo v3

    Download pretrained model from here. This is the model converted from the pretrained model of gluoncv.

3. Detect person from video by yolo v3

    Before we perform person re-id, we need to detect person from the video, yolo v3 works well for this task, you could find more details in this blog. It show you how to load the models trained by gluoncv(or mxnet) too, you will need that skills to load the model of person re-id too.

4. Extract features of person

    After we find out bounding boxes of the persons, we need to extract the features of the persons, this could be done by mxnet without much issues.
   

cv::Mat_<float> person_feautres_extractor::get_features(const cv::Mat &input)
{
    //convert cv::Mat to ndarray
    auto data = to_ndarray_->convert(input);
    data.CopyTo(&executor_->arg_dict()["data"]);
    executor_->Forward(false);

    cv::Mat_<float> result(1, 2048);
    if(!executor_->outputs.empty()){
        //copy data to cpu by synchronize api since 

        //Forward api of mxnet is async
        executor_->outputs[0].SyncCopyToCPU(result.ptr<float>(), 2048);
    }

    return result;
}

5. Find out most similar persons from the features pool

    I use cosine similarity to compare two features in this experiment.

float cosine_similarity:: 

compare_feature(const cv::Mat_<float> &lhs, const cv::Mat_<float> &rhs)
{
    cv::multiply(lhs, rhs, numerator_);
    cv::pow(lhs, 2, lhs_pow_);
    cv::pow(rhs, 2, rhs_pow_);

    auto const numerator = cv::sum(numerator_)[0];
    auto const denom_lhs = cv::sum(lhs_pow_)[0];
    auto const denom_rhs = cv::sum(rhs_pow_)[0];

    auto const denominator = std::sqrt(denom_lhs * denom_rhs);

    return static_cast<float>(denominator != 0.0 ? numerator / 
                              denominator : 0.0);
}

  Then find out the most similar features in the db, return the id in the db if similarity value greater

than threshold,  else create a new id and return it.

std::vector<visitor_identify::visitor_info> visitor_identify:: 
detect_and_identify_visitors(const cv::Mat &input)
{
    //detect persons in the input
    obj_det_->forward(input);
    auto const input_size = cv::Size(input.cols, input.rows);
    auto const detect_results = obj_filter_->filter(obj_det_->get_outputs(), input_size );
    
    std::vector<visitor_info> result;
    for(auto const &det : detect_results){
       //extract features from the person
       auto const feature = 
              feature_extract_->get_features(input(det.roi_).clone());
       //find most similar features in the database
       auto const id_info = db_->find_most_similar_id(feature);
       visitor_info vinfo;
       vinfo.roi_ = det.roi_;
       //if the confident(similarity) of the most similar features 

       //were greather than the threshold
       //return the id found in the db, else add a new id and return it
       if(id_info.confident_ > re_id_threshold_){
           vinfo.id_ = id_info.id_;
           vinfo.confidence_ = id_info.confident_;
       }else{
           auto const new_id = db_->add_new_id(feature);
           vinfo.id_ = new_id;
           vinfo.confidence_ = 1.0f;
       }
       result.emplace_back(std::move(vinfo));
    }

    return result;
}