Saturday, 9 March 2019

Build mxnet 1.3.1 on windows

    If you were like me, tried to build mxnet 1.3.1 on windows, you may suffer a lot of pains since mxnet do not have decent support on windows, apparently the developers of mxnet do not perform enough tests(maybe none) on windows before they release the stable version. Despite of all of the troubles mxnet brought, it is still a nice tool of deep learning, that is why I am still prefer to work with it.

    I believe one of the best way to make the open source project become better is contribute something back to it, that is why I would like to write down how to build mxnet 1.3.1 on windows step by step.

1. Do not build mxnet on windows with intel mkl

    Do not do this unless you are asking for trouble, please check the details on stackoverflow and issue 14343.

2. Build openBLAS with native msvc ABI


    The openBLAS post at here do not work with vc2015 anymore(if you updated your vc2015), the abi are not compatible with msvc. The easiest solution to solve this issue is build the openBLAS by yourself.The steps are

a. Clone openBLAS of xianyi from github
b. Compile openBLAS as the instruction shown here. Do not install Anaconda and miniconda together, just pick one of them. If you do not know where is your vcvars64.bat on your pc, I suggest you use Everything to find the path.
c. Copy the files(cblas.h, f77blas.h) from the generated  folder into the build folder.

3. Clone mxnet fork by me


git clone --recursive https://github.com/stereomatchingkiss/incubator-mxnet
cd mxnet 
git checkout 1.3.1_win_compile_fix
 
    This branch fix some type mismatch errors.
 

4. Comment out codes in shuffel_op.cu

     This file is under the folder "mxnet\src\operator\random", there is a function ShuffleForwardGPU, 
comment out the implementation else there will have a lot of compile times errors(no suitable 
user-defined conversion from "mshadow::Tensor<mxnet::gpu, 1, mxnet::index_t>" to 
"const mshadow::Tensor<mshadow::gpu, 1, unsigned int>" exists).
 
     I guess this function would not be called when doing inference task, after all who would like to 
make their inference results become unpredictable? If you were like me, only want to use 
cpp_package to do the inference task, you should be safe to comment out the codes. 

5. Open cmake


  Open your cmake and select msvc with 64bits.

6. Configuration




   The most important note are

1. Do not use anything related to intel MKL.

2. Do not build cpp_package at the first time

    Without mkl mxnet cannot exploit full power of the cpu, but with it your app cannot run at all, 
depending on how you build it, your app may throw the error 
"Intel MKL FATAL ERROR: Cannot load mkl_intel_thread.dll." or 
"Check failed: MXNDArrayWaitToRead(blob_ptr_->handle_) == 0 (-1 vs. 0)"

    If you do not need cuda, uncheck following options, USE_CUDA and USE_CUDNN. After that, click Configure->uncheck BUILD_TESTING->click Configure->click generate.

7. Build mxnet without cpp_package


    a. Open your ALL_BUILD.vcxproj
    b. Navigate to the project "mxnet"
    c. Right click your mouse, select "Properties"
    d. Select Linker->Input
    e. Link to flangmain.lib, flangrti.lib, flang.lib, ompstub.lib. For example, my paths are

    C:\Users\yyyy\Anaconda3\pkgs\flang-5.0.0-he025d50_20180525\Library\lib\flangmain.lib  
    C:\Users\yyyy\Anaconda3\pkgs\flang-5.0.0-he025d50_20180525\Library\lib\flangrti.lib
    C:\Users\yyyy\Anaconda3\pkgs\flang-5.0.0-he025d50_20180525\Library\lib\flang.lib
    C:\Users\yyyy\Anaconda3\pkgs\flang-5.0.0-he025d50_20180525\Library\lib\ompstub.lib

    If you do not know where are they, use Everything to find the path.

    f. Navigate to the project "ALL_Build"
    g.Right click your mouse, click build.

8. Configure cmake to build with cpp_package


    Now we can build mxnet with cpp_package, let us go to cmake again and change some settings.

a. (Optional)Change your install path, else you may not be able to install(ex : change to C:/Users/yyyy/programs/Qt/3rdLibs/mxnet/build_gpu_1_3_1_temp/install).

b. Make sure you have set the PATH of python, if you are building 32/64bits version of mxnet, you need python of 32/64bits, else you wouldn't be able to generate op.h. I suggest you use Rapid environment to manage your path on windows.


If your vc complain it cannot find the python exe, reopen your vc.

c. Check USE_CPP_PACKAGE->uncheck BUILD_TESTING->configure->generate

d. Remove the example projects since they will hinder the build process, those projects are alexnet, charRNN, googleNet, inception_bn, lenet, lenet_with_mxdataiter, mlp, mlp_cpu, mlp_gpu, resnset.

e. Go to your build/Release folder, copy the libmxnet.dll into any folder which could be found by
the windows(the path in the Path), let us assume that path call global_path.

f. Open your Developer command prompt(mine is developer command prompt for vs2015), let us call it DCP

g. Navigate your DCP to global_path

h. Enter "dumpbin /dependents libmxnet.dll", this command will show you the dependencies of this
dll. In my case, it show

flangrti.dll
flang.dll
ompstub.dll
cudnn64_7.dll
cublas64_92.dll
cufft64_92.dll
cusolver64_92.dll
curand64_92.dll
nvrtc64_92.dll
nvcuda.dll
KERNEL32.dll
VCOMP140.DLL

We only need to copy flangrti.dll, flang.dll, ompstub.dll into global_dll in order to generate op.h, because another dll already exist in the PATH. Again, please use Everything to find the path.

i. Your mxnet need to link to Link to flangmain.lib, flangrti.lib, flang.lib, ompstub.lib again since generate clear them.

j. Navigate to the project "ALL_Build"   

k.  Right click your mouse, click build.

l. Navigate to the project "INSTALL"

m.  Right click your mouse, click build.

n. Copy cpp-package\include\mxnet-cpp into build/install/include

o. Copy mxnet\3rdparty\tvm\nnvm\include\nnvm into build/install/include

9. Add mx_float for scale, int for num_filter

    The op.h generated by this solution, there are two parameters lack type declaration, you need to add them by yourself.

Conclusion

    Congratulation, now you have build the mxnet successfully, to tell you the truth, this is not a pleasant journey, there are too many bugs/issues when I try to build mxnet1.3.1on windows(1.4.0 got more bugs on windows when you try to build it) , there are many bugs should be found before they release the major version if them have tried to build mxnet on windows. I believe windows and cpp_package are not their main concern yet, let us hope that 

a. Someday they can put more love into windows and cpp_package. Windows still dominate market of desktop/laptop and cpp_package is a much better choice than python if you want to do edge deployment.

b. Adopt a commit system like opencv(whenever you commit your codes, opencv build it on every single platforms they support), this could prevent a log of bugs, the later you adopt, the more cost you need to pay for cross-platform.

c. Let us cross our finger, hope them can fix all of these bugs before next version release


Sunday, 24 February 2019

Age and gender classification by opencv, dlib and mxnet

    In this post, I will show you how to build an age gender classification application with the infrastructures I created in the last post. Almost everything are same as before, except the part of parsing the NDArray in the forward function.

    Before we dive into the source codes, let us have some examples. The images are predicted by two networks and concatenate, left side predicted by light model, right side predicted by heavy model based on resnet50.

   







    The results do not looks bad for both of the models if we don't know their ages :), let us use the model to predict the ages of famous person, like trumps(with resnet50).


 

    Unfortunately, results of age classification are not that good under different angles and expressions, this is because age classification from an image is very difficult, even human cannot accurately predict ages of persons by looking at a single image.    

1. Difference of face recognition and age gender classification

    The codes of parsing face recognition features are

std::vector<insight_face_key> result;
size_t constexpr feature_size = 512;
Shape const shape(1, feature_size);
for(size_t i = 0; i != batch_size; ++i){
    NDArray feature(features.GetData() + i * feature_size, shape, Context(kCPU, 0));
    result.emplace_back(std::move(feature));
}

    The codes of parsing age and gender classification are

std::vector<insight_age_gender_info> result;
int constexpr features_size = 202;
for(size_t i = 0; i != batch_size; ++i){
    auto const *ptr = features.GetData() + i * features_size;
    insight_age_gender_info info;
    info.gender_ = ptr[0] > ptr[1] ? gender_info::female_ : gender_info::male_;
    for(int i = 2; i < features_size; i += 2){
        if(ptr[i + 1] > ptr[i]){
                info.age_ += 1;
        }
    }
    result.emplace_back(info);
}


    Except of these part, everything are the same as before.

2. Make codes easier to reuse

    It is a pain to maintain similar codes with minor difference, in order to alleviate the prices of maintenance, I create a generic predictor as a template class with three policies, implement the face recognition and age/gender classification with this generic predictor.

template<typename Return, typename ProcessFeature, typename ImageConvert = dlib_mat_to_separate_rgb>
class generic_predictor
{
/*please check the details on github*/
}

    We could use it like to create age gender predictor as following


struct predict_age_gender_functor
{
    std::vector<insight_age_gender_info>
    operator()(const mxnet::cpp::NDArray &features, size_t batch_size) const
    {
        std::vector<insight_age_gender_info> result;
        int constexpr features_size = 202;
        for(size_t i = 0; i != batch_size; ++i){
            auto const *ptr = features.GetData() + i * features_size;
            insight_age_gender_info info;
            info.gender_ = ptr[0] > ptr[1] ? gender_info::female_ : gender_info::male_;
            for(int i = 2; i < features_size; i += 2){
                if(ptr[i + 1] > ptr[i]){
                    info.age_ += 1;
                }
            }
            result.emplace_back(info);
        }
        return result;
    }
 };
 
using insight_age_gender_predict = mxnet_aux::generic_predictor<insight_age_gender_info, predict_age_gender_functor>;

    Please check github if you want to know the implementation details.

3. Summary

    Gender prediction works very well, unfortunately age predictions is far from ideal. If we could obtain huge data set, which contain the face of the same person with different range of ages, expression, angles and the number of races are not super imbalance, accuracy of age accuracy may improve very much, but a huge data set like this is very hard to collect. 

    The source codes could find on github.

Contact me

    If you need someone to help you develop computer vision/Qt apps, please contact me on upwork.

Monday, 18 February 2019

Face recognition with mxnet, dlib and opencv

   In this post I will show you how to implement an industrial level, portable face recognition application with a small, reuseable example, without relying on any commercial library(except of Qt5, unless the module I use in this example support LGPL license).

    Before deep learning become main stream technology in computer vision fields, 2D face recognition only works well under strict environments, this make it an impractical technology.

    Thanks to the contributions of open source communities like dlib, opencv and mxnet, today, high accuracy 2D face recognition is not a difficult problem anymore.

    Before we start, let us see an interesting example(video_00).


video_00

     Although different angles and expressions affect the confidence value a lot, but in most of the time the algorithm still able to find out the most similar faces from 25 faces.

     The flow of face recognition on github are composed by 4 critical steps.


pic_00

Detect face by dlib   

std::vector<mmod_rect> face_detector::forward_lazy(const cv::Mat &input)
{
    //make sure input image got 3 channels
    CV_Assert(input.channels() == 3);

    //Resize the input image to certain width, 
    //The bigger the face_detect_width_, more 
    //faces could be detected, but will consume
    //more memory, and slower
    if(input.cols != face_detect_width_){
        //resize_cache_ is a simple trick to reduce the
        //number of memory allocation
        double const ratio = face_detect_width_ / 
                             static_cast<double>(input.cols);
        cv::resize(input, resize_cache_, {}, ratio, ratio);
    }else{
        resize_cache_ = input;
    }

    //1. convert cv::Mat to dlib::matrix
    //2. Swap bgr channel to rgb
    img_.set_size(resize_cache_.rows, resize_cache_.cols);
    dlib::assign_image(img_, dlib::cv_image<bgr_pixel>(resize_cache_));

    return net_(img_);
}
 
    Face detector of dlib perform very well, you can check the results on their post.

    If you want to know the details, please study the example provided by dlib, if you want to know more options, please study the excellent post of Learn Opencv.

Perform face alignment by dlib

    We can treat face alignment as a data normalization skills develop for face recognition, usually you would align the faces before training your model, and align the faces when predict, this could help you obtain higher accuracy.

    With dlib, face alignment become very simple. Just a few lines of codes.

//rect contain the roi of the face
dlib::matrix<rgb_pixel> face_detector::
get_aligned_face(const mmod_rect &rect)
{
    //Type of pose_model_ is dlib::shape_predictor
    //It return the landmarks of the face
    auto shape = pose_model_(img_, rect);
    matrix<rgb_pixel> face_chip;
    auto const details = 
          get_face_chip_details(shape, face_aligned_size_, 0.25);
    //extract face after aligned from the image
    extract_image_chip(img_, details, face_chip);
    return face_chip;
}

Extract features of face by mxnet

    This section will need to load the model from mxnet, unlike dlib or opencv, the c++ api of mxnet is more complicated, if you do not know how to load the model of mxnet yet, I recommend you study this post.

    This section is the most complicated part, because it contains three main points

1.  Extract the features of faces.
2.  Perform batch processing.
3.  Convert aligned face of dlib(store as matrix<rgb_pixel>) to a memory continuous float array with
the format expected by the mxnet model.


A.Load the model with variable batch size


    In order to load the model which support variable batch size, all we need to do is add one more argument to the argument list.


std::unique_ptr<Executor> create_executor(const std::string &model_params,
                                          const std::string &model_symbols,
                                          const Context &context,
                                          const Shape &input_shape)
{    
    Symbol net;
    std::map<std::string, NDArray> args, auxs;
    load_check_point(model_params, model_symbols, &net, 
                     &args, &auxs, context);

    //if "data" throw exception, try another key, like "data0"
    args["data"] = NDArray(input_shape, context, false);
    //we only need to add the new key if batch size larger than 1
    if(input_shape[0] > 1){
        //all we need is the new key "data1"
        args["data1"] = NDArray(Shape(1), context, false);
    }

    std::unique_ptr<Executor> executor;
    executor.reset(net.SimpleBind(context, 
                                  args, 
                                  std::map<std::string, NDArray>(),
                                  std::map<std::string, OpReqType>(), 
                                  auxs));

    return executor;
}

B.Convert aligned face to array

    Unlike the example of yolo v3, the input data of deepsight need more preprocess steps before you can feed the aligned face into the model. Instead of arranged the pixels as rgb order, you need to split each channels of the face into separate "page". Simply put, instead of arrange the pixels as

R1G1B1R2G2B2......RnGnBn

We should arrange the pixels as

R1R2....RNG1G2......GNB1B2.....BN


//using dlib_const_images_ptr = std::vector<matrix<rgb_pixel> const*>;
void face_key_extractor::
dlib_matrix_to_float_array(dlib_const_images_ptr const &rgb_image)
{
    size_t index = 0;
    for(size_t i = 0; i != rgb_image.size(); ++i){
        for(size_t ch = 0; ch != 3; ++ch){
            for(long row = 0; row != rgb_image[i]->nr(); ++row){
                for(long col = 0; col != rgb_image[i]->nc(); ++col){
                    auto const &pix = (*rgb_image[i])(row, col);
                    switch(ch){
                    case 0:
                        //image_vector_ is a std::vector<float>, resized in 
                        //constructor.

                        //image_vector_.resize(params_->shape_.Size())
                        //params_->shape_.Size() return total number 
                        //of elements in the tenso
                        image_vector_[index++] = pix.red;
                        break;
                    case 1:
                        image_vector_[index++] = pix.green;
                        break;
                    case 2:
                        image_vector_[index++] = pix.blue;
                        break;
                    default:
                        break;
                    }
                }
            }
        }
    }
}


C.Forward aligned faces with variable batch size

    There are two things you must know before we dive into the source codes.

1. To avoid memory reallocation, we must allocate memory for the largest possible batch size and reuse that same memory when batch size is smaller.
2.  The batch size of the float array input to the model must be the same as the largest possible batch size


//input contains all of the aligned faces detected from the image
std::vector<face_key> face_key_extractor::
forward(const std::vector<dlib::matrix<dlib::rgb_pixel> > &input)
{
    if(input.empty()){
        return {};
    }

    //Size of the input may not divisible by batch size
    //That is why we need some preprocess job to make sure
    //features of every faces are extracted
    auto const forward_count = static_cast<size_t>(
         std::ceil(input.size() / static_cast<float>(params_->shape_[0])));
    std::vector<face_key> result;
    for(size_t i = 0, index = 0; i != forward_count; ++i){
        dlib_const_images_ptr faces;
        for(size_t j = 0; 
            j != params_->shape_[0] && index < input.size(); ++j){
            faces.emplace_back(&input[index++]);
        }
        dlib_matrix_to_float_array(faces);
        auto features = 
             forward(image_vector_, static_cast<size_t>(faces.size()));
        std::move(std::begin(features), std::end(features), 
                  std::back_inserter(result));
    }

    return result;
} 

D.Extract features of faces

std::vector<face_key> face_key_extractor::
forward(const std::vector<float> &input, size_t batch_size)
{
    executor_->arg_dict()["data"].SyncCopyFromCPU(input.data(), 
                                                  input.size());
    //data1 tell the executor, how many face(s) need to process
    executor_->arg_dict()["data1"] = batch_size;
    executor_->Forward(false);
    std::vector<face_key> result;
    if(!executor_->outputs.empty()){
        //shape of features is [batch_size, 512]
        auto features = executor_->outputs[0].Copy(Context(kCPU, 0));
        Shape const shape(1, step_per_feature);
        features.WaitToRead();
        //split features into and array
        for(size_t i = 0; i != batch_size; ++i){
            //step_per_feature is 512, memory 
            //of NDArray is continuous make things easier
            NDArray feature(features.GetData() + i * step_per_feature, 
                            shape, Context(kCPU, 0));
            result.emplace_back(std::move(feature));
        }
        return result;
    }

    return result;
} 
 
 

Find most similar faces from database

    I use cosine similarity to compare similarity in this small example, it is quite easy with the help of 
opencv. 

A.Similarity compare


double face_key::similarity(const face_key &input) const
{
    CV_Assert(key_.GetData() != nullptr && 
              input.key_.GetData() != nullptr);

    cv::Mat_<float> const key1(1, 512, 
                               const_cast<float*>(input.key_.GetData()), 0);
    cv::Mat_<float> const key2(1, 512, 
                               const_cast<float*>(key_.GetData()), 0);
    auto const denominator = std::sqrt(key1.dot(key1) * key2.dot(key2));
    if(denominator != 0.0){
        return key2.dot(key1) / denominator;
    }

    return 0;
}

B.Find most similar face

    Find the most similar face is really easy, all we need to do is compare the features stored in the array one by one and return the one with the highest confidence.


//for simplicity, I put struct at here in this blog
struct id_info
{
   double confident_ = -1.0;
   std::string id_;
};

struct face_info
{
   face_key key_;
   std::string id_;
};

face_reg_db::id_info face_reg_db::
find_most_similar_face(const face_key &input) const
{
    id_info result;
    //type of face_keys_ is std::vector<face_info>
    for(size_t i = 0; i != face_keys_.size(); ++i){
        auto const confident = 
             face_keys_[i].key_.similarity(input);
        if(confident > result.confident_){
            result.confident_ = confident;
            result.id_ = face_keys_[i].id_;
        }
    }

    return result;
}

Summary

    In today's post, I show you the most critical parts of face recognize with opencv, dlib and mxnet. I believe this is a great starting point if you want to build a high quality face recognition app by c++.

    Real world applications are much more complicated than this small example since they always need to support more features and required to be efficient, but no matter how complex they are, the main flow of the 2D face recognition are almost the same as this post show you.

Contact me

    If you need someone to help you develop computer vision/Qt apps, please contact me on upwork.

Tuesday, 5 February 2019

Use person re-id model to identify person do not exist in the data set by c++

    Person re-id comparing two images of person captured under different conditions, recently this field achieve big improvement with the helps of deep learning, but is it good enough to identify person do not exist in the data set? This is the question I want to figure out in this post.

    Let me show you an example before we start.



    The results are not perfect yet, let us hope that better techniques and larger data sets would release in the future. The algorithm itself is very easy, main flows are drawn in pic00

pic00
    For those who wants to read the source codes directly, please go to github, in order to compile it, you will need opencv3.4.2 and mxnet. You can pick every build tools you like, I use qmake in this example.  If you want to know how to reproduce the results, please read on.

1. Download pretrained model of person re-id


    Download pretrained model from here. Precision and mAP of this model perform on market1501 are

top1:0.923100
top5:0.972090
top10:0.984264
mAP:0.797564

     If you want to train it by yourself, please follow the guide of gluoncv, it is quite easy.

2. Download pretrained model of yolo v3


    Download pretrained model from here. This is the model converted from the pretrained model of gluoncv.

3. Detect person from video by yolo v3


    Before we perform person re-id, we need to detect person from the video, yolo v3 works well for this task, you could find more details in this blog. It show you how to load the models trained by gluoncv(or mxnet) too, you will need that skills to load the model of person re-id too.

4. Extract features of person


    After we find out bounding boxes of the persons, we need to extract the features of the persons, this could be done by mxnet without much issues.
   
cv::Mat_<float> person_feautres_extractor::get_features(const cv::Mat &input)
{
    //convert cv::Mat to ndarray
    auto data = to_ndarray_->convert(input);
    data.CopyTo(&executor_->arg_dict()["data"]);
    executor_->Forward(false);

    cv::Mat_<float> result(1, 2048);
    if(!executor_->outputs.empty()){
        //copy data to cpu by synchronize api since 
        //Forward api of mxnet is async
        executor_->outputs[0].SyncCopyToCPU(result.ptr<float>(), 2048);
    }

    return result;
}
 


5. Find out most similar persons from the features pool

    I use cosine similarity to compare two features in this experiment.
float cosine_similarity:: 

compare_feature(const cv::Mat_<float> &lhs, const cv::Mat_<float> &rhs)
{
    cv::multiply(lhs, rhs, numerator_);
    cv::pow(lhs, 2, lhs_pow_);
    cv::pow(rhs, 2, rhs_pow_);

    auto const numerator = cv::sum(numerator_)[0];
    auto const denom_lhs = cv::sum(lhs_pow_)[0];
    auto const denom_rhs = cv::sum(rhs_pow_)[0];

    auto const denominator = std::sqrt(denom_lhs * denom_rhs);

    return static_cast<float>(denominator != 0.0 ? numerator / 
                              denominator : 0.0);
}

 
  Then find out the most similar features in the db, return the id in the db if similarity value greater
than threshold,  else create a new id and return it.
 
std::vector<visitor_identify::visitor_info> visitor_identify:: 
detect_and_identify_visitors(const cv::Mat &input)
{
    //detect persons in the input
    obj_det_->forward(input);
    auto const input_size = cv::Size(input.cols, input.rows);
    auto const detect_results = obj_filter_->filter(obj_det_->get_outputs(), input_size );
    
    std::vector<visitor_info> result;
    for(auto const &det : detect_results){
       //extract features from the person
       auto const feature = 
              feature_extract_->get_features(input(det.roi_).clone());
       //find most similar features in the database
       auto const id_info = db_->find_most_similar_id(feature);
       visitor_info vinfo;
       vinfo.roi_ = det.roi_;
       //if the confident(similarity) of the most similar features 

       //were greather than the threshold
       //return the id found in the db, else add a new id and return it
       if(id_info.confident_ > re_id_threshold_){
           vinfo.id_ = id_info.id_;
           vinfo.confidence_ = id_info.confident_;
       }else{
           auto const new_id = db_->add_new_id(feature);
           vinfo.id_ = new_id;
           vinfo.confidence_ = 1.0f;
       }
       result.emplace_back(std::move(vinfo));
    }

    return result;
} 
 

Contact me

    If you need someone to help you develop computer vision/Qt apps, please contact me on upwork