Active Appearance Models in C++ (Paamela)

2010 August 12

My implementation of the Active Appearance Models (AAMs) in C++ is almost done, it is called Paamela. I am currently fixing a couple design issues and finishing up the documentation. Even though I am still not sure whether or not I will make the code open source, I thought it would be nice to share what I have developed so far, in order to help other developers working on similar problems.

What is Paamela?

Paamela, my implementation of the AAMs, is based on Ian Matthews and Simon Baker’s Inverse Compositional approach. I have developed it in C++ with OpenCV 2.1, and I am really proud to say that it is very accurate and really fast! As of today, the AAMs open source implementations available on the Internet are only partly working, and their respective source codes look pretty messy to me (I can say that because I actually took the time to take a look at the source codes). I am not going to cite any project name here, because I do not want to be disrespectful to the developers who spent time implementing them. A rapid lookup in your favorite search engine will orient you, from there read the code, and make your own opinion. I am not criticizing their design choices, but the coding style, the naming conventions, and the documentation. Indeed, they are inexistent! Therefore, except for the developers themselves, no one is going to understand what the code is doing, and as a result, no one will really be able to reuse the code (which was the whole point of making it open source in the first place). The only clean open source AAMs available now is Mikkel B. Stegmann’s AAM-API, but unfortunately this is not implementing the inverse compositional approach. Since I wanted to have a clean C++ Inverse Compositional AAMs implementation, and as it was inexistent, I decided to do it myself.

I have tried to make the code as clean as possible to enable other developers to understand what the different equations are doing. I have also written an API with a detailed documentation, in order to allow users to understand how to adapt it to their specific problems. The API allows users to create models based on collections of appearances, and to perform AAM fitting of input images from these models. I have also made the implementation independent from the type of data being treated, which enables Paamela to handle any kind of input data: faces, hands, bone structures, etc.

Implemented features

I have implemented many features for Paamela, in order to make the fitting more accurate. These features include:

  1. Different illumination normalization techniques: HE (Histogram Equalization) and CLAHE (Contrast Limited Adaptive Histogram Equalization).
  2. Object detection to initiate the fitting process: for instance, Haar Feature-based Cascade Classifiers can be specified to roughly detect the position of the objects to fit, and enhance the accuracy of AAMs fitting tasks.
  3. Bilinear interpolation during appearance warping.
  4. Thresholded variances for the shape and the appearance statistical models.
  5. Damping for the gradient-descent update step.
  6. Convergence detection.

And I will add soon:

  1. Support for fitting using live video through OpenCV.
  2. Multi-resolution model based on a Gaussian pyramid.

Example of a fitting process

In order to show what Paamela can do, I am using here the IMM Face Database, which is freely available. Be careful if you use this database: people have been nice enough to put this together, so please respect the terms of use given in the description.

For this example, I first build a model based on four different persons. Then I am using a Haar cascade classifier to estimate the initial configuration of the face. Finally, the fitting process goes on and clips the facial features in the input image. Figure 1 shows this fitting process. After only seven iterations, the fit is done, and convergence is declared after 10 iterations.

Figure 1: AAM Fitting

The development environment

I am also including below a view of what my development environment looks like. While I am performing experiments, I constantly keep an eye on every single matrix in the model, as shown below in Figure 2. Oh, and you can see vim in the background 😉

Figure 2: AAM development environment (click to enlarge)

Conclusion

So here it is, Paamela is almost ready! I am probably not going to make it open source right now, as I am still deciding if I want to implement something else based on it, and then release this thing and Paamela all at once. Nevertheless, if you have any questions or if you want to know more, feel free to drop a comment 🙂

24 Responses leave one →
  1. shine permalink
    October 22, 2010

    If you don’t mention that….
    Could I ask for a sample code on how to developed that system?

    How to build facial patches(mesh structures).?
    and so on…

    If you permit that, then I will very pleased .
    I will use only for studying . and also, I will let you know firstly, when I writing papers .

  2. October 24, 2010

    Hi,

    I am sorry, but I am not going to make my AAM code open source any time soon. However, I recommend using Mikkel B. Stegmann’s implementation, which is open source (link given in the article).

    Regarding the facial mesh, just apply Delaunay triangulation on the set of points used for your model, and you’ll get the mesh! That is all 🙂

    Manu

  3. November 12, 2010

    First of all well done, it looks like a great project.
    I am currently trying to do something very similar. I understand that you are not yet willing to distribute your code at this time, but I was hoping that you could help me get my head around AAMs.

    I agree with what you said above about there being very little on the web explaining this properly. I have come across a million papers, all of which are very equation heavy and hard to understand. And I have tried looking at several APIs and AAM tools, but none of which I can make any sense of.

    What I am really looking for is a simple break down or step by step guide on how to go about building AAMs. (I am a decent programmer so that part shouldn’t be a problem).

    I have annotated pictures of faces, and understand that I need to apply PCA on them to get the average face shape. but from there I’m not sure how to plot that onto a picture of a face (or ultimately live input from a webcam to track head movement)

    Anyway, I hope you can help me without just referring me to Tim Coots’ website.
    Please email me soon,
    Tim.

  4. November 12, 2010

    Hi Tim,

    As I said in the email I sent you, a step by step guide would be a serious time commitment, and I do not have this time right now.

    However, I hope that what I answered to you by email will help you.

    Manu

  5. Loai Ghoraba permalink
    July 3, 2011

    Hi

    Well, I was looking forward for some open source implementation of AAM using openCV, feeling sorry that you won’t make yours public for now, though I completely understand. But do you know some other open source implementation of it using openCV ? because the one by Mikkel B. Stegmann’s is using MS, which I cannot use with Ubuntu..

  6. chris permalink
    July 10, 2011

    Same here,

    I’ve been searching a lot on the web for some open source AAM code but unfortunately it was all in vain. I cannot use Stegmann’s code since I’m running OS X.

    Was hoping that you would accept to share your code, but just as Loai said, I also understand if you don’t.

    What did you use to build Paamela? OpenCV? Can you provide some hints if you don’t want to share the whole code?

    Thanks in advance!

    chris

  7. July 31, 2011

    Hi guys,

    Sorry for the delay. I am still not inclined to make the code public at the moment. As for other implementations, I would recommend Luca
    Vezzaro’s IC AAM implementation for Matlab, available at https://sourceforge.net/projects/icaam/. Sure, it’s not C++ code, but you could easily port it to C++.

    Regarding Paamela, I only used OpenCV and developed everything from scratch just reading Matthews and Baker’s paper. I learned a lot about implementing algorithms from scientific publications while coding the IC AAM. I am preparing a blog post that sums up what I learned, which I think this could be useful to you, so I’ll email you when it’s ready 🙂

    Manu

  8. marco permalink
    April 12, 2012

    Bonjour,
    Je suis curieux de savoir où en est le projet Paamela, et surtout s’il serait possible d’avoir accès à quelques algorithmes AAM (en particulier le fitting).
    J’en aurais certainement utilité pour ma thèse de Master.
    Cordialement
    Marco (Paris)

    • April 13, 2012

      Bonjour Marco,

      Le projet Paamela est clôt. Je l’utilise actuellement dans des applications commerciales, je ne peux donc pas partager le code pour le moment.

      Bonne continuation,

      Emmanuel

  9. marco permalink
    April 26, 2012

    Bonjour,
    C’est dommage !
    Quelles applications sont développées avec ?

    Btw. Y aurait-il d’autres implémentations en C/C++ sur le web (j’ai pas trouvé) 🙁

    En tout cas bonne chance pour les projets commerciaux.
    Amicalement
    Marco

  10. saku permalink
    May 29, 2013

    Do you have the paper of this project that I can study from the paper?

    Thank you

  11. Abhishek permalink
    August 31, 2013

    Hi Emmanauel,

    I am also planning to implement AAM on my own but unlike you I am an average programmer 🙂 Could you please share your learnings while building Pamela or point me to some resources that would help me in building AAM.

    • September 2, 2013

      Most of what I have learned regarding implementing papers can be found in another article I wrote [1], and this is the process I used to implement AAMs. The IC-AAM as it is described by Matthews and Baker is not that stable, so I would strongly recommend that you start with ASMs first, and see if they fit your needs. Cootes’ papers are your best friends, as well as his booklet “Statistical Models of Appearance for Computer Vision” [2]. The booklet version I have linked is the one from 2004 and is the latest I’ve read, but you should look around as there might be a newer version by now. I hope this helped, good luck! 🙂

      [1] http://codecapsule.com/2012/01/18/how-to-implement-a-paper/
      [2] http://www.isbe.man.ac.uk/~bim/Models/app_models.pdf

  12. KongCDY permalink
    October 14, 2013

    Hello~I’m very interesting in your project. I have read two paper about the IC AAM. One is Active Appearance Models Revisited, and another is Active Appearance Model (JIA Pei). I have some question about the Jacobian of W(x;p), two paper deduced two different result. How do you understand it? Thank You vern much~

    • October 15, 2013

      I wish I could help you, however I haven’t worked with AAMs for quite a long time now, and my memories of the exact use of Jacobians are quite fuzzy. Hopefully someone else reading your comment will be able to answer you.

      • KongCDY permalink
        October 24, 2013

        Thank you very much. I have read some other code and know the Jacobians. The whole algorithm could running, but I noticed something. When the shape converge( error become smallest ), if I continue iterating the loop, the shape will become bad and the error will become larger. According to the paper, when the algorithm converge, the shape will almost not move ( move around the local minimum ). So what’s your situation ?

        • October 28, 2013

          Yes, the algorithm is supposed to converge. And it is also true that convergence is quite unstable in AAMs, and that the algorithm can diverge quite easily. I’ve also had cases of shapes moving around a local minimum, and a rule of thumb is that with good initial conditions, the algorithm will generally converge in at most 10 to 20 iterations.

          • KongCDY permalink
            November 18, 2013

            I have another question for helping. My model can fit my trainning data, but can’t fit my face ( I used the MUCT data, it has more than 200 faces). My PCA appearance model can reconstruct my face. So I don’t know why my AAM can’t fit my face.

          • November 22, 2013

            The behavior you’re seeing is expected. The choice between ASMs and AAMs is a tradeoff between precision and generalization. If the goal is to fit a small set of predefined faces, by small I mean 1 to 4 different faces, then training an AAM over a large variation of poses and illuminations of those faces will give better results than with an ASM. This is what you would use for the special effects in a movie for example, building one very precise and separate model for each actor. If the goal is to generalize and detect facial features in new faces in the wild, “new” meaning not in the training set, then an ASM will perform better than an AAM — though it will never be perfect. This is what you would use for building a surveillance system for instance, to be able to fit new faces.

  13. Kiran permalink
    December 28, 2013

    Thanks Emmanuel. This post was really helpful. Especially the last comment on when to use ASM or AAM. I am trying to do head pose estimation for “new” faces using ASM with VOSM [1] library. In ASM, will increasing the number of feature points result in better fit ? Will the increase in features points, take more cpu time/iterations to do the fitting ? Hope using ASM with POSIT for pose estimation is not a bad idea.

    [1] http://www.visionopen.com/downloads/open-source-software/VOSM/

    • January 14, 2014

      What really makes a good fit is how well the features have been selected. For a human face, the face contours, the mouth, the nose, the eyes and the eyebrows are a logical choice, because they are segmenting the image naturally. Increasing the number of feature points would increase the accuracy of the fit, but only up to a certain point. Reducing the number of feature points on the contrary would for sure decrease the accuracy of the fit. Most ASM facial models I have seen are using around 80 feature points at key locations around facial features, and I would argue that this is enough for a good fit. However, I have never experimented myself or seen any study of the influence of the number of feature points on the quality of the fit, so that could be an interesting thing to try. As for the CPU requirements, I would expect a model built on more feature points to require more CPU, although how much more CPU would depend greatly on how optimized is the exact implementation.

      Another way to improve the quality of the fit is to normalize the input images in terms of illumination and contrast so that they match the parameters of the images used to train the model. This is what has worked best for me, unfortunately this is not always possible with ASM fitting faces taken “in the wild”.

  14. Mustafa KARA permalink
    May 9, 2014

    Hi, thanks for great work.

    Is it possible you to send me the download link of Paamela?

    Thank you

  15. AAlan permalink
    July 7, 2014

    Hello Emmanuel,
    your introduction about AAM is very interesting: I’m interested in your project, is it possible to take a look to the capabilities of your implementation? I’m planning to develop on my own an AAM lib exploiting my proprietary computer vision library (No OpenCV is desired in my -future- commercial products). For the sake of true the AAM step is necessary even if my project is not related to “faces”.

    Can you share with me some data (better a binary) to analyze performance of your implementation (I’m worried about cpu-load / frame rate). My target is Windows for a demo, ARM for the market.

    Regards and thanks,
    AAlan

Trackbacks and Pingbacks

  1. Facial transformation framework on the iPhone | Code Capsule

Leave a Reply

Note: You can use basic XHTML in your comments. Your email address will never be published.

Subscribe to this comment feed via RSS