Vincent's SFM Toolbox for MATLAB

Over the years, I had to use several common SFM algorithm. I collected all my work into a toolbox so that younger researchers do not waste their time in implementing the following:

  • data generation (random rigid/non-rigid scene, popular non-rigid examples, different camera models, random or smooth random camera positions, noise addition ...)
  • 2D geometry (homography computations ...)
  • 3D geometry (quaternion, globally optimal pose estimation, absolute orientation, triangulation ...)
  • rigid 3D reconstruction: orthographic/projective calibrated/uncalibrated cameras (Tomasi Kanade with or without orthonormality constraints, eight point, Sturm Triggs 96, Oliensis Hartley 07, global optimum from Chandraker 09), bundle adjustment ...
  • non-rigid 3D reconstruction: orthographic (Torresani 08, Xiao Kanade 04, Rabaud 09 )
  • visualization for rigid/non-rigid (sequences , multiple-views at once, comparisons ...)

Please check out the Vincent's SFM toolbox page and send me any comment ! I put a lot of effort into having efficient and well-documented code. I would therefore appreciate an acknowledgement in a paper using my toolbox :)

Linear Embeddings in Non-Rigid Structure from Motion


How about computing NRSFM by comparing pairs/triplets of frames ?

Vincent Rabaud and Serge Belongie
Linear Embeddings in Non-Rigid Structure from Motion [pdf] [code]
CVPR 2009, Miami, Florida.

I joined the team at VideoSurf !!

VideoSurf is a video search engine that uses computer vision. It's really fun and we're using a lot of CV stuff, not sure how much I can tell though :)

So just go to and you can check out videos with funny stuff, music, news, sport ... I personally am more involved with the celebrity videos. We're located in San Mateo so if you want any info contact me !

Oh, if you want to hack with it, check out the video search API or the VideoSurf Firefox extension (pretty useful when you want to jump to the right place when you watch tv online or watch free movies). They also have a video blog that gives updates on all that.

Re-Thinking Non-Rigid Structure From Motion


A new way of considering non-rigid structure from motion by considering the possible shapes of a deforming objects as belonging to a non-linear manifold.

Vincent Rabaud and Serge Belongie
Re-Thinking Non-Rigid Structure From Motion [pdf]
CVPR 2008, Anchorage, Alaska.

Soylent Grid

painting panorama

CAPTCHA become more and more easy to solve by machines. Why not switching common Turing tests to harder vision problems than OCR ? We present an infrastructure to bridge the gap between computer vision image labeling and stronger Turing tests.

Stephan Steinbach, Vincent Rabaud and Serge Belongie
Soylent Grid: It's Made of People ! Link [pdf]
ICV 2007, in conjunction with ICCV 2007, Rio de Janeiro, Brazil.

Painting Panorama

painting panorama

Just a painting stitching project I did for the summer. You can find more info and results here as well as a very cool demo ! (Sorry, it is password protected for Copyright reasons, I hope I can open it soon !)

It is mostly a tweaked version of Brown's excellent IJCV paper

Here is the code and the help (that you can also get when compiling the Doxygen doc of the code).

Non-Isometric Tangent Manifold Learning

eye manifold

The following poster provides a good introduction to our work. LSML is a method for determining a warping from a point on a manifold to its neighbors on the manifold. A direct application of this method is performed on video sequences where the ways of moving on the underlying manifold are learned and then used to move within and out of the training set. The warping is also applied to an unseen frame in order to transfer the tranformations.

Piotr Dollár, Vincent Rabaud and Serge Belongie
Learning to traverse image manifolds [pdf] [version with technical report]
NIPS 2006, Vancouver, B.C., Canada.

Piotr Dollár, Vincent Rabaud and Serge Belongie
Non-Isometric Manifold Learning: Analysis and an Algorithm [pdf]
ICML 2007, Corvallis, Oregon.

Code: please check the LSML project page.

Counting Crowded Moving Objects

crowd image

When dealing with a crowd of similar objects, severe occlusions happen, leading to a very hard tracking problem. Nonetheless, what if we simply want to count the number of moving objects ? To achieve this goal, we rely on the fact that all the objects in the scene are of the same kind and hence have the same behavior. The main characteristics of our approach are:

  • a massively parallel and enhanced KLT tracker
  • a new regularization technique for feature trajectories
  • simple training data (some video footage with the corresponding number of objects through time)
  • a simple learned descriptor, proper to the observed object kind

Vincent Rabaud and Serge Belongie
Counting Crowded Moving Objects [pdf] [poster]
CVPR 2006, New York, NY.

Library Movie: [Archive 1] [Archive 2]

Surveillance Video Entertainment System

sven snapshot

An art project in which I am doing the computer vision side: SVEN

Briefly, imagine the surveillance cameras are bored with their usual footage and task. What if they wanted to find rockstars in videos for a change ? Well, that is where computer vision (and I) help to create a rockstar detector/tracker. This is basically a BraMBLe tracker with a simple person descriptors (colors, direction, mask, facial expression ...).

The source code for the tracker is here: version 1.0 and the binary here. My coding abilities have improved A LOT so don't judge me on that :). Please email me if you find any bug or way of improving it.

Amy Alexander and Vincent Rabaud
SVEN: Surveillance Video Entertainment Network
UCDARNet, Los Angeles, CA.

Whitney video, at the Whitney museum of American Art, New York City, USA

Computer Vision, Fact & Fiction

A DVD I participated to. What about asking the leaders in computer vision what they think of the use of their field in movies ?

The official website here.

Icon Vectorization

blown-up icon original icon

Computer icons are tiny images conceived to be perceived in a certain way. Therefore a lot less ambiguity exists on the family of higher resolution images that could create the icon (by shrinkage). The trick is to figure out which properties a higher resolution version of an icon should verify.

This implies constraints on region type (gradient, unform...) and perceived colors. Once these are figured out, a cute snake smoothes everything.

Vincent Rabaud and Serge Belongie
Big Little Icons [pdf]
CVAVI 2005, San Diego, CA.

Smart Vivarium


The Smart Vivarium is an ongoing project aiming at drastically improving the life of laboratory animals by:

  • constantly monitoring them
  • keeping statistics
  • detecting normal/abnormal behaviors
  • minimizing external effects by only relying on video footage

Kristin Branson, Vincent Rabaud and Serge Belongie
Three Brown Mice: See How They Run [pdf]
VS-PETS 2003, in conjunction with ICCV 2003, Nice, France (what a beautiful country)

Serge Belongie, Kristin Branson, Piotr Dollár, and Vincent Rabaud
Monitoring Animal Behavior in the Smart Vivarium [pdf]
Measuring Behavior 2005, Wageningen, The Netherlands.

Piotr Dollár, Vincent Rabaud, Garrison Cottrell and Serge Belongie
Behavior Recognition via Sparse Spatio-Temporal Features [pdf]
VS-PETS 2005, in conjunction with ICCV 2005, Beijing, China.

Space Mechanics

If you thought hyperboloids and paraboloids were only a mathematical curiosity, ask Béatrice to give you this paper and you will discover the wonders of AVOIDANCE, the little software I conceived to intersect ellipses (orbiting debris trajectories) with hyperboloid (generated by linear pieces of launching object trajectories).

Vincent Rabaud and Béatrice Deguine
A Geometrical Approach To Determine Blackout Windows At Launch
2003 AAS/AIAA Space Flight Mechanics Meeting, Ponce, Puerto Rico, AAS 03-187