Feature descriptor comparison report特征评测集锦 zz

source http://computer-vision-talks.com/2011/08/feature-descriptor-comparison-report/


For this test i have written special test framework, which allows me to easily add the new kind of descriptors and test cases and generate report data in CSV-like format. Than i upload it in Google docs and create this awesome charts. Five quality and one performance test was done for each kind of descriptor.

 Test case

  • Rotation test –  this test shows how the feature descriptor depends on feature orientation.
  • Scaling test –  this test shows how the feature descriptor depends on feature size.
  • Blur test –  this test shows how the feature descriptor is robust against blur.
  • Lighting test –  this test shows how the feature descriptor is robust against lighting.
  • Pattern detection test – this test performs detection of planar object (image) on the real video. In contrast to the synthetic tests, this test gives a real picture of the overall stability of the particular descriptor.
  • Performance test is a measurement of description extraction time.

All quality tests works in similar way. Using a given source image we generate a synthetic test data: transformed images corresponding feature points. The transformation algorithm depends on the particular test. For the rotation test case, it’s the rotation of the source image around it’s center for 360 degrees, for scaling – it’s resizing of image from 0.25X to 2x size of original. Blur test uses gaussian blur with several steps and the lighting test changes the overall picture brightness.

The pattern detection test deserves a special attention. This test is done on very complex and noisy video sequence. So it’s challenging task for any feature descriptor algorithm to demonstrate a good results in this test.

The metric for all quality tests is the percent of correct matches between the source image and the transformed one. Since we use planar object, we can easily select the inliers from all matches using the homography estimation. I use OpenCV’s function cvFindHomography for this. This metric gives very good and stable results. I do no outlier detection of matches before homography estimation because this will affect the results in unexpected way. The matching of descriptors is done via brute-force matching from the OpenCV.

 Rotation test

Descriptor's invariance to rotation summary report

In this test i obtain pretty expectable results, because all descriptors are rotation invariant expect the BRIEF. Slight changes in stability can be explained by the feature orientation calculation algorithm and descriptor nature. A detailed study of why the descriptor behaves exactly as it is, takes time and effort. It’s a topic for another article. Maybe later on….

 Scaling test

Descriptor's invariance to scaling summary report

SURF and SIFT descriptors demonstrate us very good stability in this test because they do expensive keypoint size calculation. Other descriptors uses fixed-size descriptor and you can see what it leads to. Currently for LAZY descriptor i do not have separate LAZY feature detector (i use ORB detector for tests) but I’m thinking on lightweight feature detector with feature size calculation, because it’s a must-have feature. Actually, scale invariance is much more important rather than precise orientation calculation.

 Blur test

Descriptor's invariance to blur summary report

In this test i tried to simulate the motion blur which can occurs if camera moves suddenly. All descriptors demonstrate good results in this test. By “good” I mean that the more blur size is applied the less percent of correct matches is obtained. Which is expected behavior.

 Lighting test

Descriptor's invariance to lighting summary report

In lighting test the transformed images differs only in overall image brightness. All kinds of descriptors works well in this case. The major reason is that all descriptors extracted normalized, e.g the norm_2 of the descriptor vector equals 1. This normalization makes descriptor invariant to brightness changes.

Pattern detection on real video

Pattern detection test

Detection of the object on real video is the most complex task since ground truth contains rotation, scaling and motion blur. Also other objects are also present. And finally, it’s not HD quality. These conditions are dictated by the actual conditions of application of computer vision.

As you can see on diagram, the SIFT and SURF descriptors gives the best results, nevertheless they are far away from ideal, it’s quite enough for such challenging video. Unfortunately, scale-covariant descriptors show very bad results in this test because pattern image appears in 1:1 scale only at the beginning of the video (The “spike” near frame 20). On the rest of the video sequence target object moves from the camera back and scale-covariant descriptors can’t handle this situation.

 Performance summary

Descriptor extraction time summaryThis chart shows the extraction time for N features. I made Y-axis as logarithm scale to make it more readable. For all descriptor extraction algorithm the extraction time depends on number of features linearly. Local spikes is probably caused by some vector resizing or L2 cache misses. This performance test was done on Mac Book Pro 2.2 with Core 2 Duo 2.13 Ghz.

Future works

Add new quality test cases. One additional test i know for sure – affine transformations. Your ideas for other tests are welcome!

  • Add new kind of descriptors. Definitely will add an A-SIFT implementation.
  • Create an LAZY detector with feature size and orientation estimation.
  • Improve the LAZY descriptor extraction procedure. Expect at least 20% performance gain.
  • Generate matching video for each test to demonstrate the behavior of each descriptor algorithm.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




Just another WordPress.com site

Jing's Blog

Just another WordPress.com site

Start from here......







Just another WordPress.com site

Where On Earth Is Waldo?

A Project By Melanie Coles

the Serious Computer Vision Blog

A blog about computer vision and serious stuff

Cauthy's Blog

paper review...

Cornell Computer Vision Seminar Blog

Blog for CS 7670 - Special Topics in Computer Vision


Life through nerd-colored glasses

Luciana Haill

Brainwaves Augmenting Consciousness



Dr Paul Tennent

and the university of nottingham

turn off the lights, please

A bunch of random, thinned and stateless thoughts around the Web

%d bloggers like this: