Three-dimensional (3D) kinematic models are widely-used in video-based figure tracking. We show that these models can suffer from singularities when motion is directed along the viewing axis of a single camera. The single camera case is important because it arises in many interesting applications, such as motion capture from movie footage, video surveillance, and vision-based user-interfaces.
We describe a novel two-dimensional scaled prismatic model (SPM) for figure registration. In contrast to 3D kinematic models, the SPM has fewer singularity problems and does not require detailed knowledge of the 3D kinematics. We fully characterize the singularities in the SPM and demonstrate tracking through singularities using synthetic and real examples.
We demonstrate the application of our model to motion capture from movies. Fred Astaire is tracked in a clip from the film “Shall We Dance”. We also present the use of monocular hand tracking in a 3D user-interface. These results demonstrate the benefits of the SPM in tracking with a single source of video.
Get full access to this article
View all access options for this article.
References
1.
Bergen, J., et al. 1992. Hierarchical model-based motion estimation . In Second European Conference on Computer Vision, pp. 237-252 , Santa Margherita Liguere, Italy, Springer-Verlag, Berlin.
2.
Brand, M. 1999. Shadow puppetry . In Proceedings of International Conference on Computer Vision, Vol. II, pp. 1237-1244 , Kerkyra, Greece.
3.
Bregler, C., and Malik, J. 1998. Tracking people with twists and exponential maps. In Computer Vision and Pattern Recognition, pp. 8-15, Santa Barbara, CA .
4.
Cham, T.-J., and Rehg, J. M. 1999. A multiple hypothesis approach to figure tracking . In Computer Vision and Pattern Recognition, Vol. II, pp. 239-245 , Fort Collins, Colorado.
5.
Cox, I. J., and Hingorani, S. L. 1996. An efficient implementation of reid's multiple hypothesis tracking algorithm and its evaluation for the purpose of visual tracking . IEEE Transactions on Pattern Analysis and Machine Intelligence18(2): 138-150 .
6.
Delamarre, Q., and Faugeras, O. 2001. 3D articulated models and multi-viewtracking with physical forces . Computer Vision and Image Understanding (CVIU), 81: 328-357 .
7.
Dennis, J., and Schnabel, R. 1983. Numerical Methods for Unconstrained Optimization and Nonlinear Equations, Prentice-Hall, Englewood Cliffs, NJ .
8.
Deutscher, J., Blake, A., and Reid, I. 2000. Articulated body motion capture by annealed particle filtering . In Computer Vision and Pattern Recognition, Vol. 2, pp. 126-133 , Hilton Head, SC .
9.
DiFranco, D. E., Cham, T.-J., and Rehg, J. M. November 2001. Reconstruction of 3D figure motion from 2D correspondences. In Computer Vision and Pattern Recognition, Kauai, Hawaii .
10.
Espiau, B., Chaumette, F., and Rives, P. 1992. A new approach to visual servoing in robotics . IEEE Transactions on Robotics and Automation, 8: 313-326 .
11.
Fablet, R., and Black, M. J. 2002. Automatic detection and tracking of human motion with a view-based representation . In European Conference on Computer Vision (ECCV), pp. 476-491 .
12.
Gavrila, D. M., and Davis, L. S. 1996. 3D model-based tracking of humans in action: A multi-view approach. In Computer Vision and Pattern Recognition, pp. 73-80, San Francisco, CA .
13.
Goncalves, L., Bernardo, E. D., Ursella, E., and Perona, P. 20-23 June 1995. Monocular tracking of the human arm in 3D . In Proceedings of International Conference on Computer Vision, pp. 764-770 , Cambridge, MA.
14.
Heap, A. J., and Hogg, D. C. 1998. Wormholes in shape space: Tracking through discontinuous changes in shape . In Proceedings of International Conference on Computer Vision, Bombay, India.
15.
Hogg, D. 1983. Model-based vision: a program to see a walking person . Image Vision Computing1(1): 5-20 .
16.
Howe, N., Leventon, M., and Freeman, W. November 1999. Bayesian reconstruction of 3D human motion from single-camera video. In Neural Information Processing Systems (NIPS), Denver, CO .
17.
Irani, M., Rousso, B., and Peleg, S. 1994. Computing occluding and transparent motions . International Journal of Computer Vision12(1): 5-16 .
18.
Isard, M., and Blake, A. April 1996. Contour tracking by stochastic propagation of conditional density . In European Conference on Computer Vision (ECCV), Vol. I, pp. 343-356 , Cambridge, UK.
19.
Ju, S. X., Black, M. J., and Yacoob, Y. 1996. Cardboard people: A parameterized model of articulated image motion . In International Conference of Automatic Face and Gesture Recognition, pp. 38-44 , Killington, VT.
20.
Kakadiaris, I., and Metaxas, D. June 1995. 3D human body model acquisition from multiple views . In Proceedings of International Conference on Computer Vision, pp. 618-623 , Cambridge, MA.
21.
Kakadiaris, I., and Metaxas, D. 18-20 June 1996. Modelbased estimation of 3D human motion with occlusion based on active multi-viewpoint selection. In Computer Vision and Pattern Recognition, pp. 81-87, San Francisco, CA .
22.
Kanatani, K., and Morris, D. D. 2001. Gauge and gauge transformations for uncertainty description of geometric structure with indeterminacy . IEEE Transactions on Information Theory47(5): 2017-2028 .
23.
Khosla, P. 1986. Real-time Control and Identification of Direct-Drive Manipulators. PhD Thesis, Carnegie Mellon University, Department of ECE.
24.
Koenderink, J. J., and van Doorn, A. J. 1991. Affine structure from motion . Journal of the Optical Society of America8(2): 377-385 .
25.
MacCormick, J., and Blake, A. 1999. A probabilistic exclusion principle for tracking multiple objects . In Proceedings of International Conference on Computer Vision, pp. 572-578 .
26.
MacCormick, J., and Isard, M. 2000. Partitioned sampling, articulated objects, and interface-quality hand tracking . In European Conference on Computer Vision (ECCV), Vol. 2, pp. 3-19 .
27.
McCarthy, J. 1990. An Introduction to Theoretical Kinematics, MIT Press, Cambridge, MA .
28.
Morris, D. D., and Rehg, J. M. 23-25 June 1998. Singularity analysis for articulated object tracking. In Computer Vision and Pattern Recognition, pp. 289-296, Santa Barbara, CA .
29.
Nakamura, Y. 1991. Advanced Robotics: Redundancy and Optimization, Addison-Wesley, Reading, MA .
30.
Nelson, B. J., and Khosla, P. K. 1994. The resolvability ellipsoid for visual servoing. In Computer Vision and Pattern Recognition, pp. 829-832.
31.
O'Rourke, J., and Badler, N. 1980. Model-based image analysis of human motion using constraint propagation . IEEE Transactions on Pattern Analysis and Machine Intelligence2(6): 522-536 .
32.
Pai, D. K. May 1988. Singularity, Uncertainty, and Compliance of Robot Manipulators. PhD Thesis, Cornell University, Ithaca, NY.
33.
Papanikolopoulos, N., Khosla, P., and Kanade, T. 1993. Visual tracking of a moving target by a camera mounted on a robot: A combination of control and vision . IEEE Transactions on Robotics and Automation9(1): 14-35 .
34.
Pavlović, V., Rehg, J., Cham, T., and Murphy, K. 1999. A dynamic Bayesian network approach to figure tracking using learned dynamic models . In Proceedings of International Conference on Computer Vision, Vol. I, pp. 94-101 , Corfu, Greece.
35.
Pentland, A., and Horowitz, B. 1991. Recovery of nonrigid motion and structure . IEEE Transactions on Pattern Analysis and Machine Intelligence13(7): 730-742 .
36.
Rehg, J. M. April 1995. Visual Analysis of High DOF Articulated Objects with Application to Hand Tracking. PhD Thesis, Carnegie Mellon University. Available as School of Computer Science Technical Report CMU-CS-95-138.
37.
Rehg, J. M. January 2000. Motion capture from movies . In Proceedings of Asian Conference on Computer Vision, Vol. II, pp. 1125-1131 , Taipei, Taiwan. Invited Paper.
38.
Rehg, J. M., and Kanade, T. 1993. Digiteyes: Vision-based human hand tracking. Technical Report CMU-CS-TR-93-220, Carnegie Mellon University, School of Computer Science.
39.
Rehg, J. M., and Kanade, T. 1994a. Digiteyes: Vision-based hand tracking for human-computer interaction . In J. K. Aggarwal and T. S. Huang, editors, Proceedings of Workshop on Motion of Non-Rigid and Articulated Objects, pp. 16-22 , Austin, TX.
40.
Rehg, J. M., and Kanade, T. 1994b. Visual tracking of high DOF articulated structures: an application to human hand tracking . In European Conference on Computer Vision (ECCV), Vol. 2, pp. 35-46 , Stockholm, Sweden.
41.
Rehg, J. M., and Kanade, T. 1995. Model-based tracking of self-occluding articulated objects . In Proceedings of International Conference on Computer Vision, pp. 612-617 , Cambridge, MA.
42.
Rehg, J. M., Kang, S. B., and Cham, T.-J. September 2000. Video editing using figure tracking and image-based rendering . In International Conference on Image Processing, Vancouver, B.C. Invited Paper.
43.
Rehg, J. M., and Morris, D. D. 1997. Singularities in articulated object tracking with 2D and 3D models. Technical Report CRL 97/8, DEC Cambridge Research Lab.
44.
Rijpkema, H., and Girard, M. 1991. Computer animation of knowledge-based human grasping . Computer Graphics25(4): 339-348 .
45.
Robert, L. 1995. Camera calibration without feature extraction . Computer Vision, Graphics, and Image Processing63(2): 314-325 .
46.
Rohr, K. 1994. Towards model-based recognition of human movements in image sequences . Computer Vision, Graphics, and Image Processing59(1): 94-115 .
47.
Rowley, H. A., and Rehg, J. M. 17-19 June 1997. Analyzing articulated motion using expectation-maximization. In Computer Vision and Pattern Recognition, pp. 935-941, San Juan, Puerto Rico .
48.
Sharma, R., and Hutchinson, S. August 1997. Motion perceptibility and its application to active vision-based servo control . IEEE Transactions on Robotics and Automation13(4): 607-617 .
49.
Shimada, N., Shirai, Y., Kuno, Y., and Miura, J. 1998. Hand gesture estimation and model refinement using monocular camera—ambiguity limitation by inequality constraints . In Proceeding of the Third International Conference on Automatic Face and Gesture Recognition, pp. 268-273 , Nara, Japan.
50.
Sidenbladh, H., Black, M. J., and Fleet, D. J. 2000. Stochastic tracking of 3D human figures using 2D image motion . In Proceedings of the European Conference on Computer Vision, Dublin, Ireland.
51.
Sidenbladh, H., De la Torre, F., and Black, M. J. March 2000. A framework for modeling the appearance of 3D articulated figures . In Proceedings of the International Conference on Automatic Face and Gesture Recognition, pp. 368-375 , Grenoble, France.
52.
Sminchisescu, C., and Triggs, B. 2001. Covariance scaled sampling for monocular 3D body tracking . In Computer Vision and Pattern Recognition, Vol. 1, pp. 447-454 .
53.
Spong, M. 1989. Robot Dynamics and Control, Wiley, New York .
54.
Tomasi, C., and Kanade, T. November 1992. Shape and motion from image streams under orthography: a factorization method . International Journal of Computer Vision9(2): 137-154 .
55.
Toyama, K., and Blake, A. 2001. Probabilistic tracking in a metric space . In Proceedings of International Conference on Computer Vision, Vancouver, Canada.
56.
Wachter, S., and Nagel, H.-H. June 1999. Tracking persons in monocular image sequences . Computer Vision and Image Understanding (CVIU)74(3): 174-192 .
57.
Weiss, L., Sanderson, A., and Neuman, C. October 1987. Dynamic sensor-based control of robots with visual feedback . IEEE Transactions on Robotics and Automation, 3(5): 404-417 .
58.
Yamamoto, M., and Koshikawa, K. 1991. Human motion analysis based on a robot arm model. In Computer Vision and Pattern Recognition, pp. 664-665. Also see ETL TR 90-46.
