A tutorial on visual servo control
IEEE Transactions on Robotics and AutomationPublished 1 January 1996
Seth Hutchinson, Gregory D. Hager, Peter Corke
Citations3,487
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This article provides a tutorial introduction to visual servo control of robotic manipulators by reviewing the prerequisite topics from robotics and computer vision, including a brief review of coordinate transformations, velocity representation, and a description of the geometric aspects of the image formation process.
Abstract
This article provides a tutorial introduction to visual servo control of robotic manipulators. Since the topic spans many disciplines our goal is limited to providing a basic conceptual framework. We begin by reviewing the prerequisite topics from robotics and computer vision, including a brief review of coordinate transformations, velocity representation, and a description of the geometric aspects of the image formation process. We then present a taxonomy of visual servo control systems. The two major classes of systems, position-based and image-based systems, are then discussed in detail. Since any visual servo system must be capable of tracking image features in a sequence of images, we also include an overview of feature-based and correlation-based methods for tracking. We conclude the tutorial with a number of observations on the current directions of the research field of visual servo control.
Keywords
Computer ScienceEngineering
Communications of the ACMRandom sample consensus
25,463 Citations1981Martin A. Fischler, Robert C. Bolles
New results are derived on the minimum number of landmarks needed to obtain a solution, and algorithms are presented for computing these minimum-landmark solutions in closed form that provide the basis for an automatic system that can solve the Location Determination Problem under difficult viewing.
International Journal of Computer VisionSnakes: Active contour models
16,951 Citations1988Michael Kass, Andrew Witkin +1 more
This work uses snakes for interactive interpretation, in which user-imposed constraint forces guide the snake near features of interest, and uses scale-space continuation to enlarge the capture region surrounding a feature.
HAL (Le Centre pour la Communication Scientifique Directe)An Iterative Image Registration Technique with an Application to Stereo Vision
11,614 Citations1981Bruce D. Lucas, Takeo Kanade
This work presents a new image registration technique that makes use of the spatial intensity gradient of the images to find a good match using a type of Newton-Raphson iteration, and can be generalized to handle rotation, scaling and shearing.
Good features to track
6,858 Citations1994Jianbo Shi, Tomasi
A feature selection criterion that is optimal by construction because it is based on how the tracker works, and a feature monitoring method that can detect occlusions, disocclusions, and features that do not correspond to points in the world are proposed.
CERN Document Server (European Organization for Nuclear Research)Applied Optimal Estimation
6,388 Citations1974A. Gelb
This is the first book on the optimal estimation that places its major emphasis on practical applications, treating the subject more from an engineering than a mathematical orientation, and the theory and practice of optimal estimation is presented.
IEEE Journal on Robotics and AutomationA versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses
5,816 Citations1987R. Tsai
A new technique for three-dimensional camera calibration for machine vision metrology using off-the-shelf TV cameras and lenses using two-stage technique has advantage in terms of accuracy, speed, and versatility over existing state of the art.
Robot Motion Planning
5,423 Citations1991Jean‐Claude Latombe
Journal of the Optical Society of America AClosed-form solution of absolute orientation using unit quaternions
4,075 Citations1987Berthold K. P. Horn
Addison-Wesley Longman Publishing Co., Inc. eBooksComputer and Robot Vision
3,948 Citations1991Robert M. Haralock, Linda G. Shapiro
This two-volume set is an authoritative, comprehensive, modern work on computer vision that covers all of the different areas of vision with a balanced and unified approach.
IEEE Transactions on Pattern Analysis and Machine IntelligenceLeast-Squares Fitting of Two 3-D Point Sets
3,930 Citations1987K.S. Arun, Thomas S. Huang +1 more
An algorithm for finding the least-squares solution of R and T, which is based on the singular value decomposition (SVD) of a 3 × 3 matrix, is presented.
Robot Vision
3,632 Citations1986Berthold K. P. Horn
This book presents a coherent approach to the fast-moving field of computer vision, using a consistent notation based on a detailed understanding of the image formation process, and will provide a useful and current reference for professionals working in the fields of machine vision, image processing, and pattern recognition.
Feedback Control of Dynamic Systems
2,309 Citations1986Gene F. Franklin, David J. Powell +1 more
This introductory book provides an in-depth, comprehensive treatment of a collection of classical and state-space approaches to control system design and ties the methods together so that a designer is able to pick the method that best fits the problem at hand.
Graduate texts in mathematicsFoundations of Differentiable Manifolds and Lie Groups
1,832 Citations1983F. W. Warner
Lecture notes in computer scienceA new approach to visual servoing in robotics
1,356 Citations1993Bernard Espiau, François Chaumette +1 more
Vision-based control in robotics based on considering a vision system as a specific sensor dedicated to a task and included in a control servo loop is described, and stability and robustness questions arise.
IEEE Transactions on Robotics and AutomationA new technique for fully autonomous and efficient 3D robotics hand/eye calibration
1,329 Citations1989R. Tsai, R. Lenz
A novel technique for computing position and orientation of a camera relative to the last joint of a robot manipulator in an eye-on-hand configuration aimed at simplicity, efficiency, and accuracy while giving ample geometric and algebraic insights is described.
IEEE Transactions on Robotics and AutomationA new approach to visual servoing in robotics
1,327 Citations1992Bernard Espiau, François Chaumette +1 more
Journal of the Optical Society of America AClosed-form solution of absolute orientation using orthonormal matrices
1,133 Citations1988Berthold K. P. Horn, Hugh M. Hilden +1 more
International Journal of Computer VisionA computational framework and an algorithm for the measurement of visual motion
1,059 Citations1989P. Anandan
This paper describes a hierarchical computational framework for the determination of dense displacement fields from a pair of images, and an algorithm consistent with that framework, based on a scale-based separation of the image intensity information and the process of measuring motion.
IEEE Transactions on Pattern Analysis and Machine IntelligenceFitting parameterized three-dimensional models to images
829 Citations1991David Lowe
Current methods of parameter solving are extended to handle objects with arbitrary curved surfaces and with any number of internal parameters representing articulation, variable dimensions, or surface deformations to allow model-based vision to be used for a much wider class of problems than was possible with previous methods.
IEEE Journal on Robotics and AutomationDynamic sensor-based control of robots with visual feedback
822 Citations1987L.E. Weiss, Arthur C. Sanderson +1 more
Simulation studies of two- and three-degree-of-freedom systems show the application of an adaptive control algorithm to overcome unknown and nonlinear relations in the feature to world space mapping.
IEEE Transactions on Robotics and AutomationVisual tracking of a moving target by a camera mounted on a robot: a combination of control and vision
557 Citations1993Nikos Papanikolopoulos, P.K. Khosla +1 more
The authors present algorithms for robotic (eye-in-hand configuration) real-time visual tracking of arbitrary 3D objects traveling at unknown velocities in a 2D space (depth is given as known).
Journal of Dynamic Systems Measurement and ControlThe Mathematics of Coordinated Control of Prosthetic Arms and Manipulators
550 Citations1972Daniel E. Whitney
Proceedings of the IEEEMotion and structure from feature correspondences: a review
464 Citations1994Thomas S. Huang, Arun N. Netravali
Some of the mathematical techniques borrowed from algebraic geometry, projective geometry, and homotopy theory that are required to solve three-dimensional (3D) motion and structure of rigid objects when their corresponding features are known at different times or are viewed by different cameras are mentioned.
IEEE Transactions on Pattern Analysis and Machine IntelligenceDetermination of the attitude of 3D objects from a single perspective view
453 Citations1989Michel Dhome, M. Richetin +2 more
It is shown on examples related to polyhedra that this approach leads to results useful for both location and recognition of 3D objects because few admissible hypotheses are retained from the interpolation of the three line segments.
IEEE Transactions on Robotics and AutomationAutomated tracking and grasping of a moving object with a robotic hand-eye system
443 Citations1993Peter K. Allen, A. Timčenko +2 more
An attempt to achieve a high level of interaction between a real-time vision system capable of tracking moving objects in 3-D and a robot arm with gripper that can be used to pick up a moving object is described.
Medical Entomology and ZoologyRobot Control: The Task Function Approach
441 Citations1991Claude Samson, Bernard Espiau +1 more
Computer Vision Graphics and Image ProcessingAn analytic solution for the perspective 4-point problem
377 Citations1989Radu Horaud, Bernard Conio +2 more
Machine Vision and ApplicationsDynamic monocular machine vision
346 Citations1988Ernst D. Dickmanns, Volker Graefe
A new approach to real-time machine vision in dynamic scenes is presented based on special hardware and methods for feature extraction and information processing using integral spatio-temporal models that by-passes the nonunique inversion of the perspective projection by applying recursive least squares filtering.
Versatile visual servoing without knowledge of true Jacobian
345 Citations2002Koh Hosoda, Minoru Asada
The proposed visual servoing control scheme ensures the convergence of the image-features to desired trajectories, by using the estimated Jacobian matrix, which is proved by the Lyapunov stability theory.
Analysis and solutions of the three point perspective pose estimation problem
305 Citations2002R.M. Haralick, Donghoon Lee +2 more
It is shown that even in cases where the solution is not near the geometric unstable region considerable care must be exercised in the calculation, and an analytical method is presented which produces a numerically stable calculation.
IEEE Transactions on Robotics and AutomationVision-guided servoing with feature-based trajectory generation (for robots)
305 Citations1989John T. Feddema, O.R. Mitchell
Manipulator control with image-based visual servo
299 Citations2002Koichi Hashimoto, T. Kimoto +2 more
A visual feedback control scheme, called image-based visual servo, is proposed for manipulators with cameras on their hands that is stable even under noisy conditions, while the conventional position-based servo tends to be unstable.
IEEE Transactions on Pattern Analysis and Machine IntelligenceDetermination of camera location from 2-D to 3-D line and point correspondences
295 Citations1990Yan Liu, Thomas S. Huang +1 more
A method for the determination of camera location from two-dimensional to three-dimensional (3-D) straight line or point correspondences is presented and results can be obtained in the presence of noise if more than the minimum required number of correspondences are used.
IEEE Transactions on Automatic ControlAdaptive robotic visual tracking: theory and experiments
262 Citations1993Nikos Papanikolopoulos, P.K. Khosla
Algorithms are proposed for the solution of the robotic (eye-in-hand configuration) visual tracking and servoing problem and the computational complexity and the experimental results demonstrate that the proposed algorithms can be implemented in real time.
Decomposition of transformation matrices for robot vision
248 Citations2005Sriram Ganapathy
IEEE Transactions on Robotics and AutomationDynamic effects in visual closed-loop systems
222 Citations1996Peter Corke, M C Good
The concept of visual dynamic control is introduced, which is concerned with dynamic effects due to the manipulator and machine vision sensor which limit the performance, and combined with axis velocity control, is shown to result in robust and high performance tracking.
Pattern RecognitionGuiding a robot by visual feedback in assembling tasks
212 Citations1973Y. Shirai, Haruka INOUE
In this experiment, the difference between the desired position and the actual position of the object is recognized visually and corrected by moving the manipulator.
IEEE Transactions on Robotics and AutomationWeighted selection of image features for resolved rate visual feedback control
198 Citations1991John T. Feddema, C.S.G. Lee +1 more
The authors develop methodologies for the automatic selection of image features to be used to visually control the relative position and orientation (pose) between the end-effector of an eye-in-hand robot and a workpiece.
CVGIP Image UnderstandingRobust Methods for Estimating Pose and a Sensitivity Analysis
194 Citations1994Rajesh Kumar, Allen R. Hanson
It is shown that for small field of view systems, offsets in the image center do not significantly affect the location of the camera in a world coordinate system, and errors in the focal length significantly affect only the component of translation along the optical axis in the pose computation.
IEEE Control SystemsRobot hand-eye coordination based on stereo vision
174 Citations1995Gregory D. Hager, Wen‐Chung Chang +1 more
The theory and implementation of a system that positions a robot manipulator using visual information from two cameras is described, and a control law that moves the robot to drive this error to zero is derived.
Medical Entomology and ZoologyA Robot Ping-Pong Player: Experiments in Real-Time Intelligent Control
162 Citations1988Russell L. Anderson
A Robot Ping-Pong Player is described, the first robot able to play, and even beat, human ping-pong players and presages future robot systems with many joints and sensors that must do the same, no matter what the task.
Active, uncalibrated visual servoing
151 Citations2002Billibon H. Yoshimi, Peter K. Allen
The authors propose a method for visual control of a robotic system which does not require the formulation of an explicit calibration between image space and the world coordinate system and uses an approximation to the image Jacobian to provide smooth, near-continuous control.
IEEE Transactions on Pattern Analysis and Machine IntelligenceExact and approximate solutions of the perspective-three-point problem
142 Citations1992Daniel DeMenthon, L.S. Davis
The authors show that by using approximations to perspective, 2D lookup tables can be built for each of the triangles of the models by using paraperspective and orthoperspective.
IEEE Transactions on Robotics and AutomationVisual compliance: task-directed visual servo control
139 Citations1994A. Castaño, Seth Hutchinson
This paper introduces visual compliance, a new vision-based control scheme that lends itself to task-level specification of manipulation goals and derives the hybrid Jacobian matrix that is used to effect visual compliance.
Pattern Recognition LettersDecomposition of transformation matrices for robot vision
109 Citations1984Sundaram Ganapathy
A new and simple analytical technique is presented that accomplishes the inverse problem of computing the camera location and orientation from the transformation matrix rather easily and has applications to a wide variety of problems in Computer Vision.
Image and Vision ComputingUncalibrated stereo hand-eye coordination
98 Citations1994Nicholas J. Hollinghurst, Roberto Cipolla
A system that combines stereo vision with a 5-DOF robotic manipulator to enable it to locate and reach for objects in an unstructured environment using an affine stereo algorithm, a simple but robust approximation to the geometry of stereo vision.
The International Journal of Robotics ResearchCamera-Space Manipulation
90 Citations1987Steven B. Skaar, W.H. Brockman +1 more
IEEE Transactions on Robotics and AutomationAn active visual estimator for dexterous manipulation
88 Citations1996A.A. Rizzi, Daniel E. Koditschek
This work presents a working implementation of a dynamics based architecture for visual sensing that provides field rate estimates of the positions and velocities of two independent falling balls in the face of repeated visual occlusions and departures from the field of view.
Feature-based visual servoing of an eye-in-hand robot with improved tracking performance
81 Citations2002Woojin Jang, Zeung nam Bien
By means of various examples, the method of feature-based servoing of a robot proposed is proved to be very effective for conducting object-oriented robotic tasks.
The International Journal of Robotics ResearchEye-to-Hand Coordination for Vision-Guided Robot Control Applications
80 Citations1993Sardha Wijesoma, Derek Wolfe +1 more
A real-time vision-control strategy for "hand-to-eye coordination" that gives good performance even in the presence of significant vision system miscalibrations and kinematic model parametric errors is described.
The International Journal of Robotics ResearchThree-Dimensional Camera Space Manipulation
75 Citations1990Steven B. Skaar, W.H. Brockman +1 more
This article extends to three-dimensional tasks the method of camera space manipulation to rigid body placement tasks and is illustrated with experiments involving the positioning of one rigid body on a second.
IEEE Transactions on Robotics and AutomationDynamic sensing in a ping-pong playing robot
74 Citations1989Russell L. Andersson
The author examines issues in the context of a functioning robot system that plays ping-pong that must be able to integrate the stream of sensor data into its task plan, compensating for changed sensor data and continuously optimizing the plan.
Calibration-free visual control using projective invariance
60 Citations2002Gregory D. Hager
First, it is shown how results from projective geometry can be used to perform online calibration and second, results on projective invariance are used to define setpoints for visual control that are independent of viewing location.
Minerva Access (University of Melbourne)High-Performance Visual Closed-Loop Robot Control
57 Citations1994Peter Corke
A distinction between visual kinematic and visual dynamic control is introduced, concerned with how the manipulator should move in response to perceived visual features, which is well addressed in the literature.
Lecture notes in control and information sciencesIntegrating sensor placement and visual tracking strategies
53 Citations2005Bradley J. Nelson, Pradeep K. Khosla
Image and Vision ComputingOn the visual mathematics of tracking
44 Citations1991John Aloimonos, Dimitris P. Tsakiris
The attempt is to develop correspondence-free tracking schemes and get rid of the limitations inherent in the optical flow formalism, and it is shown how a monocular observer can track an initially foveated object and keep it stationary in the centre of the visual field assuming that the target is always visible during the tracking phase.
Integrating sensor placement and visual tracking strategies
40 Citations2002Bradley J. Nelson, P.K. Khosla
A technique is introduced for integrating a visual tracking strategy with dynamically determined sensor placement criteria that allows the system to automatically determine, in real-time, proper camera motion for tracking objects successfully while accounting for the undesirable, but often unavoidable, characteristics of camera-lens and manipulator systems.
Autonomous RobotsVisual grasping with long delay time of a free floating object in orbit
34 Citations1994Ch. Fagerer, Dirk Dickmanns +1 more
In the US-German Spacelab mission D2 (April/May 1993), a new level of automation capabilities has been achieved with the ROTEX-‘freeflyer’-experiment of DLR and the contributions of UniBwM in the fields of monocular motion-stereo vision, state prediction and fully automatic grasping under long delay times are discussed in detail.
Lecture notes in control and information sciencesOn a unified concept for a new generation of light-weight robots
31 Citations1990J. Dietrich, G. Hirzinger +2 more
A light multisensory gripper as the critical end-mass of any robot is presented that contains different-type redundant force-torque sensors, tactile arrays, 9 laser-range-finders including a scanner, and a tiny stereo-camera.
Calibration and use of a light stripe range sensor mounted on the hand of a robot
30 Citations2005Gerald J. Agin
This report describes a sensor that uses a Puma robot to carry a cage in which are mounted a light-stripe projector and a lightweight television camera in fixed relationship to each other.
Feature-based visual servoing and its application to telerobotics
24 Citations2002Gregory D. Hager, Gerhard Grunwald +1 more
The basic theory behind feature-based visual Servoing is described, the issues involved in integrating visual servoing into the ROTEX space teleoperation system are discussed, and the system is described and discussed.
Lecture notes in control and information sciencesControl of robotic systems through singularities
22 Citations2006S. Chiaverini, L. Sciavicco +1 more
The goal of this work is to provide an overview of major control techniques that manage the occurrence of singularities for robotic systems and the following solutions are discussed; namely, the Jacobian transpose, theJacobian pseudoinverse, and the damped least-squares Jacobian inverse.
A "robust" convergent visual servoing system
22 Citations2002D. Kim, A.A. Rizzi +2 more
A simple visual servoing control algorithm capable of robustly positioning a three degree of freedom end effector based only on information from a stereo vision system is described.
Mathematical and Computer ModellingOnline computation of exterior orientation with application to hand-eye calibration
21 Citations1996Chia-Hui Lu, Eric Mjolsness +1 more
A new least squares objective function is formulated that leads to a good initialization scheme based on weak-perspective projection, as well as a robust and efficient descent algorithm using absolute orientation which minimizes 3-D object space error rather than 2-D image error.
Realtime tracking in five degrees of freedom using two writs-mounted laser range finders
17 Citations2002S. Venkatesan, C. Archibald
The results of experiments in real-time tracking of moving objects using wrist-mounted sensors are presented and a performance study of the tracking experiment is presented.
RoboticaConcepts of Augmented Image Space and Transformed Feature Space for Efficient Visual Servoing of an “Eye-in-Hand Robot”
17 Citations1991Won Jang, Kyungjin Kim +2 more
Simulations and real experiments demonstrate that the proposed concepts are useful and versatile tools for the industrial robot vision tasks, and thus the visual servoing problem can be dealt with more systematically.
IFAC Proceedings VolumesVisual grasping with long delay time of a free floating object in orbit
6 Citations1994Ernst D. Dickmanns, Ch. Fagerer +1 more
International Journal of Computer VisionConstrained pose refinement of parametric objects
3 Citations1994Robert R. Goldberg
It is demonstrated in this paper that this constrained pose refinement formulation is no more difficult than the original problem based on numerical analysis techniques, including active set methods and lagrange multiplier analysis.
…