You can download here the code for the paper Structure from Motion with Objects presented at CVPR 2016. The code (SfMwO.zip) and data (dataset.zip) provide the MATLAB functions and evaluation to replicate the results of the paper (SfMwO.pdf).

VGM CODE Structure from Motion with Objects CVPR 2016 image 1VGM CODE Structure from Motion with Objects CVPR 2016 image 2

 

 

Abstract

This work shows for the first time that is possible to reconstruct the position of rigid objects and to jointly recover affine camera calibration solely from a set of object detections in a video sequence. In practice, this work can be considered as the extension of Tomasi and Kanade factorization method using objects. Instead of using points to form a rank constrained measurement matrix, we can form a matrix with similar rank properties using 2D object detection proposals. In detail, we first fit an ellipse onto the image plane at each bounding box as given by the object detector. The collection of all the ellipses in the dual space is used to create a measurement matrix that gives a specific rank constraint. This matrix can be factorised and metrically upgraded in order to provide the affine camera matrices and the 3D position of the objects as an ellipsoid. Moreover, we recover the full 3D quadric thus giving additional information about object occupancy and 3D pose. Finally, we also show that 2D points measurements can be seamlessly included in the framework to reduce the number of objects required. This last aspect unifies the classical point-based Tomasi and Kanade approach with objects in a unique framework. Experiments with synthetic and real data show the feasibility of our approach for the affine camera case.

Please remember to cite our paper if you use code and dataset:

@inproceedings{Crocco:etal:2015,

  title={Structure from Motion with Objects}, 
  author={M. Crocco and C. Rubino and A. {Del Bue}}, 
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, 
  month = {June}, 
  year={2016}
}