Coupling active depth estimation and visual servoing via a large projection operator

The goal of this paper is to propose a coupling between the execution of an image-based visual servoing task and an active structure from motion strategy. The core idea is to modify online the camera trajectory in the null-space of the (main) servoing task for rendering the camera motion ‘more informative’ with respect to the estimation of the 3-D structure. Consequently, the structure from motion convergence rate and accuracy is maximized during the servoing transient. The improved structure from motion performance also benefits the servoing execution, since a higher accuracy in the 3-D parameters involved in the interaction matrix improves the image-based visual servoing convergence by significantly mitigating the negative effects (instability, loss of feature visibility) of a poor knowledge of the scene structure. Active maximization of the structure from motion performance results, in general, in a deformed camera trajectory with respect to what would be obtained with a classical image-based visual servoing: therefore, we also propose an adaptive strategy able to automatically activate/deactivate the structure from motion optimization as a function of the current level of accuracy in the estimated 3-D structure. We finally report a thorough experimental validation of the overall approach under different conditions and case studies. The reported experiments support well the theoretical analysis and clearly show the benefits of the proposed coupling between visual control and active perception.