IntraFace

The IntraFace (IF) is a library with state of the art algorithms for feature tracking, head pose estimation, facial attribute recognition, multi-person face detection and analysis, facial expression recognition, and facial synchrony detection.

IF implements facial feature detection and tracking using the Supervised Descent Method (SDM). The facial attributes are classified using a linear SVM with HoG features. The Selective Transfer Machine (STM) is adopted to compensate face variations, such as pose, scale, illumination, occlusion, and individual differences in facial morphology and behavior.

The library provides algorithms to perform basic emotion recognition and facial Action Units (AU) detection. To consolidate the results, IF algorithms were compared against generic and state of the art ones, such as linear SVM, Kernel Mean Matching (KMM), Transductive SVM ( T-SVM), and Domain Adaptation SVM (DA-SVM).

It was used the CK+ and the FERA datasets. Both datasets contain videos where subjects perform one emotional state, however FERA is a more challenging dataset, i.e. it is harder to segment an emotional state from the neutral expression.

In the average IF performed better in both datasets in the tasks of emotion recognition and AU detection. In the CK+ dataset, SVM performed close to IF due to the separability of the emotions from the neutral face. In the FERA, the DA-SVM performed closest to IF, as the SVM adapt to the emotion.

Code

In the author’s website it is provided the IntraFace C++ library, however its usage is free of charge only for academics purposes. The source code is closed and their algorithms are patent pending.

Reference

“IntraFace”, F. De La Torre, W. Chu, X. Xiong, F. Vicente, J. F. Cohn – IEEE International Conference on Face and Gesture Recognition , 2015.

Advertisements

STASM library on MacOS or *nix

In a previous post I wrote about the STASM library for detecting facial landmarks. The source code is available in the project’s website, but it has instructions only for Windows. If you want to compile in OS X or another *nix variant, you can use the content of the MingW directory.

First of all, install OpenCV; if you are in OS X, you can install it via Brew. Download and uncompress the STASM-4.0 package. Inside the STASM directory, create a directory named build and copy to it the contents of the mingw directory. Edit the Makefile inside build to adjust the OpenCV includes, changing the following lines in the header:

# Makefile: osx makefile for stasm

EXE=.bin

# Note: high_gui is needed for the apps but not for the Stasm library per se.

LIBRARIES=`pkg-config –libs opencv`

# -Wno-long-long disables the “long long warnings” for the OpenCV headers
# -Wno-unused-parameter allows virtual func defs that don’t use all params
# -Wno-unknown-pragmas allows OpenMP pragmas without complaint

CFLAGS=-std=c++11 -O2 -Wall -Wextra -pedantic\
-Wno-long-long -Wno-unused-parameter -Wno-unknown-pragmas -Wstrict-aliasing

INCLUDES=`pkg-config –cflags opencv` -I../stasm -I../apps

CXX=g++

Now you may be able to perform the make clean all. It will generate a couple of binaries that you can test if the library works. I recommend the reading of the STASM manual before using it.

If you want to code using the library, it may be useful to isolate what is important in the package. Create a new directory and copy to it the stasm and the data directories, and also the the apps/minimal.cpp file. Now download my custom Makefile, which has only the lines that are necessary to compile the library and minimal.cpp example. Compile it by make minimal.bin and test using the ./minimal.bin binary.

Note the use of C++11 in CFLAGS=-std=c++11. Remove this if you can not use C++11 with your compiler. +

Active Shape Models with SIFT Descriptors and MARS

The paper presents an algorithm to detect facial landmarks using active shape models (ASM), SIFT and multivariate adaptive regression splines (MARS). The base idea is to iteratively adjust a set of facial landmarks. Each landmark is compared with image patches being at a neighbour position, moving the landmark to the patch with the best model descriptor. The initial set of landmarks is pre-computed by a training procedure.

Active shape models is a method to align a given set of points, each one with a descriptor, with a new shape. The properties of the original set are kept during the alignment, such as relative distances. This method is used in the alignment of the trained set of landmarks and the new face; and also at each iteration.

SIFT is a image descriptor that uses histograms of gradient information. For each image patch it computes a histogram of gradient intensities, where each bin represent an angle range. In this article, to improve performance, the descriptors uses fixed scale and orientation.

MARS is regression method used to measure the distance among descriptors. Euclidian distance is inadequate, for it lacks of histogram weighting; Mahalanobis, nearest-neighbors (standard for the SIFT framework) and SVM are adequate, however they are costly. MARS performed as good as SVM, but 80% faster.

Code

Source code available in the author’s website. It is an C++ code distributed under BSD license. It requires OpenCV with the non-free SIFT module.

Reference

Milborrow, Stephen, and Fred Nicolls. “Active shape models with SIFT descriptors and MARS.” VISAPP 1.2 (2014): 5.

OpenBR with Brew

OpenBR is a new framework based on OpenCV to develop, compare and deploy biometrics algorithms. The description of algorithms uses a specific syntax when in the command line; and an C++ alternative is also available, as the library is entirely implemented using this language. More about OpenBR in this post, here I will describe how to install in the MacOS using Brew, and how to use with C++.

Install

If you just want to test OpenBR in the command line, you should install in the easy way:

brew install openbr

However, if you want to use the C++ Plugin and perform more advanced integration with OpenCV and C++ projects, it may be better to install in the hard way; i.e. when you carefully install each component.

First install OpenCV using the instructions of this post.

Then install Qt in developer mode. This can take a long time, e.g. in my i5 quad core machine it took about 100 minutes to compile.

brew install –with-developer qt5

If you don’t care about the last release of the OpenBR, you may use:

brew install openbr

However, if you want the last release use the instructions of the OpenBR website. The important part is to download and compile OpenBR:

$ git clone https://github.com/biometrics/openbr.git
$ cd openbr
$ git checkout 0.5
$ git submodule init
$ git submodule update
$ mkdir build # from the OpenBR root directory
$ cd build
$ cmake -DCMAKE_PREFIX_PATH=/usr/local/opt/qt5/ -DCMAKE_BUILD_TYPE=Release ..
$ make -j4
$ sudo make install

Usage

To test if everything is right, use the following command to perform face detection on every image of the directory of images. For each image it will be presented a window with the face ROI and the center of the eyes.

br -algorithm ShowFaceDetection -enrollAll -enroll

This C++ code provides the same behaviour of the above command. To compile, use the following Makefile, which manually sets where to find Qt libraries.

PKG_CONFIG_PATH := /usr/local/opt/qt5/lib/pkgconfig/:/usr/local/lib/pkgconfig/:/usr/lib/pkgconfig/
export PKG_CONFIG_PATH
SHELL := /bin/bash
CC = g++
CFLAGS=-std=c++11 -Wall -Wno-overloaded-virtual -O2 `pkg-config –cflags opencv Qt5Core Qt5Concurrent`
CXXFLAGS=$(CFLAGS)
LDFLAGS=-lm -lopenbr `pkg-config –libs opencv Qt5Core Qt5Concurrent`
+

Kullback Leibler Divergence

Concept

The Kullback-Leibler (KL) divergence is a statistical concept used to measure how similar are two probabilities distributions. Considering two probability density functions (PDF), f and g, the KL divergence satisfies these properties:

  1. Self similarity: D(f || f) = 0
  2. Self identification: D(f || g) = 0 only if f = g
  3. Positivity: D(f || g) >= 0 for all f, g.

Divergence using GMM

The KL divergence does not have a closed form expression for all cases. In this post I will focus on gaussians PDF, more specifically on Mixture of Gaussians (GMM). In the literature, one may find methodologies to estimate the KL divergence for this kind of PDF, but there is no closed form to achieve an exact value. The equations bellow describe f and g as mixture of gaussians:

description

The most common technique to estimate the KL divergence between two GMM is using Monte Carlo (MC). This a technique is based on sampling points randomly, and applying some computation based on these samples.

Consider that we can sample n points to cover the GMM f, the KL divergence by MC is computed by the equation bellow. The intuition of this equation is that we sample points in one gaussian f, then we compute the probability of this point for the gaussian f and the gaussian g. So, basically we are analysing how the function f overlaps function g by an approximation.

kl_mc

One can imagine that the MC approach is time consuming, as the quality of the estimation relies on the amount of samples. Some more recent methodologies to estimate the divergence are available. I will present one of them: the variational approach, that has this definition:

kl_var

The fist equation is the variational divergence itself, and the second equation is the definition of the KL divergence between two simple gaussians (not a GMM, i.e. not a mixture of gaussians, just a single gaussian distribution function). The intuition of this approach is a little trick. There is no necessity to sample points, because this is a direct formulation.

For each gaussian component of f, it is computed the divergence of this component with all other components of f and also with all components of g, using the entropy concept as a logarithmic function. This will provide a lower bound divergence.

Coding

I am working in a Mathematica CDF file to provide an interactive way to explain the KL concept. For now you can get on my github the Mathematica notebook files for the Monte Carlo and Variational approaches; and also the C++ code for the variational KL. Remember that these codes are experimental, use at your own risks.

Final remarks

If you need to code something with Kullback Leibler divergence, I strongly recommend you to read the article I used as reference. However, sometimes it is hard to understand all the statistical concepts, so I wrote this article to provide an overview of the problem. I am not a pro in statistics, but if you need some assistance to understand the equations or the code, don’t hesitate to contact me. Also, this is my first draw about this subject, but I plan to improve this post +

References

Hershey, J. and Olsen, P. (2007). Approximating the kullback leibler divergence between gaussian mixture models. In Acoustics, Speech and Signal Processing, 2007. ICASSP 2007. IEEE International Conference on, volume 4, pages IV–317–IV–320.

OpenCV with brew

Installation

To install OpenCV v2.4.9 with OpenNI support, first you need to install Apple XCode Command Line Tools, Java and Java SDK. Then you can use the following brew commands:

brew tap homebrew/science
brew tap totakke/openni
brew install openni
brew install sensor-kinect
brew install nite
brew install –with-cuda –with-ffmpeg –with-openni –with-tbb opencv

Update: In the last version of OpenCV (2.4.11_1) I could use libc++ and c++11 capabilities. Install OpenCV with c++11 flag: brew install –c++11 –with-cuda –with-ffmpeg –with-openni –with-tbb opencv

Choose the desired flags, in this case the Nvidia CUDA, FFMpeg, OpenNI and Intel TBB were active. When using TBB, sometimes the pkg-config needs to be fixed by changing the -llibtbb.dylib to /usr/local/lib/libtbb.dylib in the file _/usr/local/lib/pkgconfig/opencv.pc

Install also libfreenect, so you can use commands like freenect-glview

brew install libfreenect

Obs. Apple XCode Command Line Tools can be installed using:

xcode-select –install

Usage

To compile some OpenCV code, probably you need to use libstdc++ (GNU C++) instead of libc++ (LLVM). See an example of the compilation command line:

g++ -stdlib=libstdc++ `pkg-config –cflags opencv` simpleflow.cpp `pkg-config –libs opencv` -o simpleflow

It is more elegant to use a Makefile, as the one bellow:

SHELL := /bin/bash
CFLAGS=-Wall -Wno-overloaded-virtual -stdlib=libstdc++ -O2 `pkg-config –cflags opencv`
CXXFLAGS=$(CFLAGS)
LDFLAGS=`pkg-config –libs opencv`

Notice that the first line just tells the Makefile the correct shell environment to use with make. This fix compatibility issues when you use an alternative shell – I use and recommend the fishshell.

Update: In the last version of OpenCV I could use libc++ and c++11 capabilities. See my updated Makefile:

SHELL := /bin/bash

CC = g++

CFLAGS=-std=c++11 -Wall -Wno-overloaded-virtual -O2 `pkg-config –cflags opencv`

CXXFLAGS=$(CFLAGS)

LDFLAGS=`pkg-config –libs opencv`

+

Open Source Biometric Recognition

OpenBR is an open source framework for testing and deploying biometrics systems. It uses OpenCV for basic computer vision and implements some state-of-the-art advanced features such as face recognition and eyes detection.

The objective of the project is to provide a baseline, being easy to modify and test new biometrics algorithms. The focus of the project is face recognition. For this task OpenBR has a good performance, being the best open source solution, however proprietary face matchers have better recognition rates.

In OpenBR is available also age and genre estimation, face detection and recognition, eyes detection, key points, landmarks, normalisation, representation by binary patterns, key points description, oriented histograms, and wavelets, feature extraction and matching. Finally, OpenBR is able to handle the image gallery for persistence and parallelisation, and evaluation of recognition rates.

Code

You may install OpenBR in MacOS using the Brew

brew install openbr

The entire source code is available in the project’s Github:

https://github.com/biometrics/openbr

More info in the project’s website:

http://openbiometrics.org

Reference

Klontz, Joshua C., et al. “Open source biometric recognition.” Biometrics: Theory, Applications and Systems (BTAS), 2013 IEEE Sixth International Conference on. IEEE, 2013.