www.DownloadPaper.ir - CiteSeerX

4 downloads 0 Views 258KB Size Report
315–320. [11] K. L. Koay, D. S. Syrdal, M. L. Walters, and K. Dautenhahn, “Living with ...... Brandon K. Chen, Student Member, IEEE,. Xinyu Liu, Student Member, ...
www.DownloadPaper.ir

IEEE TRANSACTIONS ON ROBOTICS, VOL. 26, NO. 1, FEBRUARY 2010

[5] T. Tasaki, K. Komatani, T. Ogata, and H. Okuno, “Spatially mapping of friendliness for human–robot interaction,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., 2005, pp. 52–526. [6] E. A. Sisbot et al., “A human aware mobile robot motion planner,” IEEE Trans. Robot., vol. 23, no. 5, pp. 874–883, Oct. 2007. [7] M. L. Walters, K. Dautenhahn, K. L. Koay, C. Kaouri, R. te Boekhorst, C. L. Nehaniv, I. Werry, and D. Lee, “Close encounters: Spatial distances between people and a robot of mechanistic appearance,” in Proc. IEEERAS Int. Conf. Humanoid Robots, 2005, pp. 450–455. [8] K. Dautenhahn et al., “How may I serve you?: A robot companion approaching a seated person in a helping context,” in Proc. ACM SIGCHI/SIGART Conf. Human-Robot Interact., 2006, pp. 172–179. [9] R. Gockley, J. Forlizzi, and R. G. Simmons, “Natural person-following behavior for social robots,” in Proc. ACM/IEEE Int. Conf. Human-Robot Interact., 2007, pp. 17–24. [10] E. Pacchierotti, H. I. Christensen, and P. Jensfelt, “Evaluation of passing distance for social robots,” in Proc. IEEE Int. Workshop Robot Human Interact. Commun., 2006, pp. 315–320. [11] K. L. Koay, D. S. Syrdal, M. L. Walters, and K. Dautenhahn, “Living with robots: Investigating the habituation effect in participants’ preferences during a longitudinal human–robot interaction study,” in Proc. IEEE Int. Conf. Robot Human Interact. Commun., 2007, pp. 564–569. [12] H. Kuzuoka, K. Yamazaki, A. Yamazaki, J. Kosaka, Y. Suga, and C. Heath, “Dual ecologies of robot as communication media: thoughts on coordinating orientations and project ability,” in Proc. SIGCHI Conf. Human Factors Comput. Syst., 2004, pp. 183–190. [13] H. Huettenrauch, K. S. Eklundh, A. Green, and E. A. Topp, “Investigating spatial relationships in human–robot interaction,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., 2006, pp. 5052–5059. [14] T. Kanda, H. Ishiguro, M. Imai, and T. Ono, “Development and evaluation of interactive humanoid robots,” Proc. IEEE, vol. 92, no. 11, pp. 1839– 1850, Nov. 2004. [15] R. Siegwart et al., “Robox at expo. 02: A large scale installation of personal robots,” Robot. Auton. Syst., vol. 42, pp. 203–222, 2003. [16] K. Hayashi, D. Sakamoto, T. Kanda, M. Shiomi, S. Koizumi, H. Ishiguro, T. Ogasawara, and N. Hagita, “Humanoid robots as a passive-social medium—A field experiment at a train station,” in Proc. ACM Annu. Conf. Human-Robot Interact., 2007, pp. 137–144. [17] K. Nohara, T. Tajika, M. Shiomi, T. Kanda, H. Ishiguro, and N. Hagita, “Integrating passive RFID tag and person tracking for social interaction in daily life,” in Proc. Int. Symp. Robot Human Interact. Commun., 2008, pp. 545–552. [18] O. Sugiyama, T. Kanda, M. Imai, H. Ishiguro, and N. Hagita, “Humanlike conversation with gestures and verbal cues based on a three-layer attentiondrawing model,” Connect. Sci., vol. 18, no. 4, pp. 379–402, 2006. [19] B. Mutlu, J. K. Hodgins, and J. Forlizzi, “A storytelling robot: Modeling and evaluation of human-like gaze behavior,” in Proc. IEEE-RAS Int. Conf. Humanoid Robots, 2006, pp. 518–523. [20] Y. Kuno, K. Sadazuka, M. Kawashima, K. Yamazaki, A. Yamazaki, and H. Kuzuoka, “Museum guide robot based on sociological interaction analysis,” in Proc. SIGCHI Conf. Human Factors Comput. Syst., 2007, pp. 1191–1194. [21] K. Hirai, M. Hirose, Y. Haikawa, and T. Takenaka, “The development of the Honda humanoid robot,” in Proc. IEEE Int. Conf. Robot. Autom, 1998, pp. 1321–1326. [22] T. Kanda, T. Miyashita, T. Osada, Y. Haikawa, and H. Ishiguro, “Analysis of humanoid appearances in human–robot interaction,” in Proc. IEEE/RSJ Int. Conf. Intell. Robot. Syst., 2005, pp. 62–69.

195

Path Planning for Improved Visibility Using a Probabilistic Road Map Matthew Baumann, Simon L´eonard, Elizabeth A. Croft, and James J. Little Abstract—This paper focuses on the challenges of vision-based motion planning for industrial manipulators. Our approach is aimed at planning paths that are within the sensing and actuation limits of industrial hardware and software. Building on recent advances in path planning, our planner augments probabilistic road maps with vision-based constraints. The resulting planner finds collision-free paths that simultaneously avoid occlusions of an image target and keep the target within the field of view of the camera. The planner can be applied to eye-in-hand visual-targettracking tasks for manipulators that use point-to-point commands with interpolated joint motion. Index Terms—Computer vision, path planning for industrial manipulators, sensor positioning, visual servoing.

I. INTRODUCTION Integrating vision guidance into industrial-robot systems that are currently deployed on assembly lines and factories poses a significant challenge, especially where the nature of the task is not entirely structured (e.g., random bin picking and general assembly, as opposed to predefined pick and place, and line-following operations). Existing methods, such as visual servoing, aim to replace the position controllers that are common to most commercial systems with controllers that close the loop with visual feedback. Typical industrial robots, however, are accessed through proprietary interfaces that only accept point-topoint commands. Upgrading these interfaces to accept the velocity commands used in visual servoing requires substantial investments in programming the communication interface and the accompanying safety-monitoring systems. Thus, despite significant interest in deploying vision-guided manipulators on existing assembly lines, robotics companies are hesitant to replace current controllers with visual-servoing systems. Given the many existing robotics systems that are already deployed, manufacturers are interested in approaches that allow the current systems to be upgraded with a vision-guidance module. In this study, we present a modular approach that addresses the problem of integrating vision guidance into semi-structured industrial tasks, where several practical challenges, such as visual occlusions and whole-arm collision avoidance are, for the most part, ignored by current visual-servoing methods. This paper introduces a vision-based probabilistic road map (VBPRM) that harmonizes the constraints that are inherent to visionguided robotics, with the reality of the manipulators found on assembly lines. The VBPRM finds paths that avoid collisions in the workspace and maintain the visibility of a target during the motion. To achieve this, a VBPRM uses the hard constraints, which are imposed by a Manuscript received July 9, 2009; revised October 22, 2009. Current version published February 9, 2010. This paper was recommended for publication by Associate Editor F. Lamiraux and Editor L. Parker upon evaluation of the reviewers’ comments. This work was supported by Precarn, Inc., by the Natural Sciences and Engineering Research Council of Canada, and by Braintech, Inc. M. Baumann and J. J. Little are with the Department of Computer Science, The University of British Columbia, Vancouver, BC V6T 1Z4, Canada (e-mail: [email protected]; [email protected]). S. L´eonard and E. A. Croft are with the Department of Mechanical Engineering, The University of British Columbia, Vancouver BC V6T 1Z4, Canada (e-mail: [email protected]; [email protected]). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TRO.2009.2035745

1552-3098/$26.00 © 2009 IEEE Authorized licensd use limted to: IE Xplore. Downlade on May 13,20 at 1:53 UTC from IE Xplore. Restricon aply.

196

www.DownloadPaper.ir

IEEE TRANSACTIONS ON ROBOTICS, VOL. 26, NO. 1, FEBRUARY 2010

PRM [1] to avoid collisions and a series of weights on road-map edges to provide soft visibility constraints. These visibility constraints form the main contribution of this paper. This paper introduces the following soft vision-based constraints: 1) An image target must remain within the field of view (FOV) of a camera as much as possible, and 2) the target must remain unoccluded by obstacles in the workspace as much as possible. These constraints are addressed by two algorithms: dynamic visibility checking (DVC) and dynamic occlusion checking (DOC). II. PREVIOUS WORK Motion control that is based on visual feedback, which is also known as visual servoing [2], aims to bridge the gap between vision, control, and motion planning. One important consideration in visual servoing is that the vision system must “see” the objects or image features to close the control loop. The objects must lie within the region of space visible to the camera, which is called the FOV. Additionally, no other objects may occlude the line of sight between the visual sensor and the target object. While the FOV constraint is implicitly addressed in image-based visual servoing (IBVS), it presents a challenge for tasks that are expressed in Cartesian space, as in the case of position-based visual servoing (PBVS). Several attempts were made to address the problem of FOV with PBVS. A switching-control approach is proposed in [3]. The controller switches between a PBVS mode and a backward motion. PBVS is used when the target is within a region of interest in the center of the image. When a part of the target leaves this region, a backward motion is used to move the target back into the center. Other switching methods between IBVS and PBVS are described in [4] and [5]. As an alternative to switching control, hybrid controllers combine both IBVS and PBVS elements simultaneously, e.g., see, [6] and [7]. An increasingly popular research area is online-trajectory generation [8]–[10]. Trajectories are generated according to guidelines that prevent joint-limit violations, avoid kinematic singularities, and prevent targets from leaving the FOV. Despite addressing several important challenges, neither IBVS nor PBVS can be transferred to industrial manipulators without replacing the controllers and the programming schemes supplied by most robot manufacturers. This conflict between architectures stems from the resolved-rate nature of visual servoing that outputs velocity commands to the motor system, whereas programming languages used with industrial robots accept point-to-point commands. Also, the trajectories provided with typical industrial systems cannot be customized to include the arbitrarily complex trajectories generated in visual servoing. Typical industrial controllers provide linear interpolation in Cartesian, spherical, cylindrical, and joint spaces. Furthermore, many industrial systems require specification of all trajectory waypoints prior to executing the motion. Finally, the important problem of planning camera positions within the restrictions of industrial systems has received little attention. The problem of constrained sensor positioning with regard to occlusions has been considered in work, such as [11], in which a boundary-based algorithm is proposed that computes the boundary of the region of space from which a target is visible. This space partitioning is used as a binary constraint on an optimization of sensor positions. III. VISION-BASED PROBABILISTIC ROAD MAP A PRM defines a sparse representation of the configuration space Q of a robot by a graph G(V, E) [1]. The vertices of the graph V ⊂ Q represent the set of configurations that are used by the planner as waypoints. The edges are undirected and represent the trajectories that are available to the planner, such that ej, k ∈ E if vertex qk is reachable

Authorized licensd use limted to: IE Xplore. Downlade on May 13,20 at 1:53 UTC from IE Xplore. Restricon aply.

from vertex qj by following a known collision-free trajectory. This section presents the algorithm to construct a VBPRM. First, Section III-A reviews the algorithm to determine whether an edge ej, k is collisionfree. Then, Section III-B and C introduces the motion-segmentation (MS) algorithm that divides each motion into segments that satisfy one, both, or neither of the FOV and occlusion constraints. The VBPRM assigns a penalty to each edge based on the length of segments that violate the constraints. The path planner will seek the path with the lowest cumulative penalty, thus avoiding visibility loss as a soft constraint. In a typical PRM, edge ej, k implies that the robot can move from configuration qj to qk without colliding with the environment. Each edge can be weighted by a cost value wj, k that is inversely proportional to the motion’s desirability. In the case of the VBPRM, a weight wj, k that is associated with the edge ej, k represents the distance that the camera must travel while unable to see the target due to occlusions or the FOV limits. Given a graph G, with an initial configuration qi and a final configuration qf , the planner finds a path between qi and qf by following a sequence of edges ei , j , . . . , ek , f with the least cost. Thus, the PRM can implement both hard and soft constraints simultaneously. VBPRM construction proceeds in a simple series of steps as follows. 1) Randomly sample configurations in Q, and store collision-free configurations as road-map vertices V . 2) Connect neighboring vertices with edges E if the edges are collision free (see Section III-A). 3) Determine whether each edge (i.e., motion) contains segments where visibility is lost due to the FOV or occlusion constraints (see Section III-B and C) 4) Assign each edge a penalty wj, k proportional to its visibility loss. Once the VBPRM is complete, Dijkstra’s algorithm provides an efficient means of computing low-visibility-penalty paths between any two vertices [12]. The following sections elaborate each step of the VBPRM’s construction. First, Section III-A reviews dynamic collision checking (DCC) [13]. Section III-B presents a variant of DCC, which is called DVC, to address the FOV constraint. Finally, Section III-C introduces the occlusion-checking algorithm to address the occlusion constraint. A. Dynamic Collision Checking The DCC algorithm tests all the edges in the PRM for collisions between the manipulator and obstacles in the workspace. The algorithm proposed by Schwarzer et al. [13] defines a test to determine collisions between bodies. Within the context of path planning, DCC is able to determine whether a robot collides with the environment during a motion between two configurations. Specifically, given the initial configuration qi and the final configuration qf , DCC determines whether the motion, which is defined by q(t) = qi + (qf − qi )t for the parameter 0 ≤ t ≤ 1, results in a collision. DCC uses the relationship between the shortest distance between the manipulator and any obstacle and the longest distance traveled by any point on the manipulator’s hull during a motion. Specifically, let dD C C (qi ) and dD C C (qf ) denote the initial and final distances between the manipulator and an obstacle, respectively. Also, given a motion q(t), let ℓD C C (q(t)) be the longest distance traveled by any point on manipulator during the motion. Then, given dD C C (qi ), dD C C (qf ), and ℓD C C (q(t)), DCC determines that the manipulator does not collide with the obstacle if ℓD C C (q(t)) < dD C C (qi ) + dD C C (qf ).

(1)

Equation (1) states that if every point on the robot’s hull travels a short distance during q(t), and the robot is far enough from an obstacle, then it is impossible for the robot to collide with the obstacle. Equation (1)

www.DownloadPaper.ir

IEEE TRANSACTIONS ON ROBOTICS, VOL. 26, NO. 1, FEBRUARY 2010

Fig. 1.

Frustum of a camera and its four associated planes.

is a sufficient condition, however, it is not a necessary condition for collision-free motion. If (1) is not satisfied, the algorithm proceeds by dividing the motion q(t) in two halves, i.e., q(u) for 0 ≤ u < t/2, and q(v) for t/2 ≤ v ≤ 1, and (1) is evaluated for both motions. This procedure is applied recursively until every motion satisfies (1), or a collision is detected in the process. B. Dynamic Visibility Checking In a PRM, DCC is used to test whether a motion q(t) results in a collision. In a VBPRM, DVC [14] is used to test whether an image target leaves the FOV of the camera during a motion q(t). The FOV constraint requires that the target point falls within the rectangular boundaries of the camera image. It is distinct from the occlusion constraint, which requires that no object lies between the camera and the target. The parameters involved in the solution of the FOV problem are limited to the intrinsic and extrinsic parameters of the camera. Although images are two-dimensional, it is convenient to formulate DVC in three dimensions. Let C P be a three-dimensional (3-D) point expressed in the coordinate frame of a camera mounted on the end-effector, and let q(t) define a motion in joint space. Then, the problem of finding if C P leaves the FOV of the camera during q(t) is equivalent to find if C P collides with the 3-D frustum of the camera (see Fig. 1). Based on this formulation, the DCC algorithm is modified to test for a collision between the frustum and C P. The formulation of DVC begins by outlining the pinhole model that is used to represent the camera. Let C P = [C X C Y C Z]T denote the coordinates of a 3-D point in the camera coordinate frame C (see Fig. 1). The 3-D frustum of the camera is defined by four planes, i.e., π1 , π2 , π3 , and π4 (see Fig. 1). The inward normals of these planes are determined by the normals of the four triangles that connect the origin of the camera’s coordinate frame to the four points representing the corners of the image plane. The coordinates of these four points are determined from the intrinsic parameters of the camera [15]. For C P to remain within the FOV during the motion q(t), the point must not collide with π1 , π2 , π3 , or π4 . Using this formulation, we adapt the DCC algorithm, which is presented in Section III-A, to solve the dynamic FOV-checking problem. In order to evaluate the inequality of (1) for DVC, the shortest distance between C P, and the four planes, i.e., π1 , π2 , π3 , and π4 , must be computed, and the length of the trajectory, which is described by C P in the camera frame, must be evaluated for a motion q(t). 1) Shortest Distance to Collision: The inequality of (1) requires us to compute the shortest distance between the C P, and the four planes, i.e., π1 , π2 , π3 , and π4 . The distance is computed directly by evaluating   nπ k C P . (2) dD V C = min 1 ≤k ≤4 nπ k 

Authorized licensd use limted to: IE Xplore. Downlade on May 13,20 at 1:53 UTC from IE Xplore. Restricon aply.

197

Finally, we note that if C P is outside the FOV of the camera, then dD V C (q) ≤ 0. 2) Length of Trajectory: The other element of (1) is ℓD C C (q(t)). For DVC, ℓD V C (q(t)) corresponds to evaluating the length of trajectory, which is described by C P as the robot moves from qi to qf according to a motion that is represented by q(t) = qi + (qf − qi )t, with 0 ≤ t ≤ 1. Although C P represents an inertial target, the motion of the camera with respect to an inertial coordinate frame results in a trajectory C P(q(t)) in the coordinate frame C. Thus, given q(t), the kinematics of the robot, and the position and the orientation of the camera with respect to the end-effector ℓD V C (q(t)) is computed by the integral, which is given by  1 ˙ (3) |C P(q(t))| ℓD V C (q(t)) = 2 dt. 0

To avoid evaluating the previous integral, the inequality in (1) is used approximate ℓD V C (q(t)) by an upper bound O(ℓD V C (q(t))). The bound, which is presented in this section, applies to manipulators with revolute joints and reverses the demonstration of the bound presented in [13]. The evaluation of O(ℓD V C (q(t))) proceeds by bounding the contribution of each joint for the evaluation of ℓD V C (q(t)). First, let ∆q = |qi − qf | be the magnitude of rotation of each joint. Then, ˙ in (3) for each joint j, we find the configuration that maximizes C P and use the bound  1    ˙  (4) max C P(q(t)) ℓD V C (qj (t)) ≤  dt 0

t

2

   ˙  = max C P(q(t))  ∆qj . t

(5)

Finally, the upper bound on ℓD V C (q(t)) is computed by ℓD V C (q(t)) ≤

ℓD V C (qj (t)).

(6)

j

˙ For Given (6), the task is to find the configurations that maximize C P. a manipulator with revolute joints, these configurations are the ones that maximize the radius between the coordinate frame of the jth link C ℓD V C (q(t)) ≤ and ! P. Let rj denote this radius; then, (6) becomes 1 j rj ∆qj . For the first link, j = 1, and r1 = | P|2 , i.e., the length of C P transformed in the coordinate frame of the first link. The second radius r2 is found by adding the length of the second link to r1 , etc. For the jth link, this ensures that rj is maximum because the arm remains fully stretched. We note that r1 depends on the coordinates of C P and, therefore, it is impossible to precompute all the ri offline. Radii rj are computed from the length of each link, which are derived from the kinematics of the robot. In contrast, the bound presented in [13] always involves the tool control point, and as such, it is possible to precompute all the rj offline. The computation of each rj , however, is more complex because it involves bounding the volume swept by the robot. 3) Motion Segmentation: The formulations for dD V C and ℓD V C could be used directly in an adaptive sampling framework similar to the DCC algorithm [13]. In this case, the algorithm would use binary subdivision of the motion until a collision is detected, at which point, it would terminate. Termination on the first-detected collision would give a binary evaluation of each road-map edge, thereby disqualifying edges that cause a loss of visibility due to the FOV. However, the goal is to provide a numerical penalty based on the degree of visibility loss instead of rejecting the edge. Therefore, a modified version of the adaptive sampling algorithm is necessary: one that can detect the number and position of all changes in visibility along a given motion. We refer to this algorithm as MS.

198

www.DownloadPaper.ir

IEEE TRANSACTIONS ON ROBOTICS, VOL. 26, NO. 1, FEBRUARY 2010

Fig. 2. In DOC, the MS algorithm computes the intersection points between the camera motion q j, k and the occlusion boundary L v .

MS is detailed in Algorithm 1. MS performs binary subdivision on a motion between configurations qj and qk . The subdivision samples the motion, looking for configurations at which dD V C is less than a sampling threshold ǫ. These are configurations at which the target point is within a very small distance of the frustum boundary. As before, subdivision only continues when the distance to the boundary dD V C is less than the upper bound on displacement ℓD V C effected by the current-motion segment. Subdivision (and, thus, additional sampling) occurs adaptively: only in segments that are near the boundary and have some chance of containing a crossing. Thus, the sampling can occur at high densities near potential crossings, however, at low densities elsewhere. Unlike in DCC, the overall algorithm does not terminate when a collision is detected but, rather, records the colliding configuration in a list I and continues examining the rest of the motion until all segments are examined. At the limit, the displacement bound ℓD V C decreases in proportion to the shrinking size of the subdivisions. To prevent oversampling, the subdivision also terminates when ℓD V C (q1 , q2 ) < ǫ. The list of transitions I divides the motion between qj and qk into a set of segments. Since the value of dD V C is signed, it is simple to determine which segments place the target in or out of the FOV by testing the value of dD V C (q) at the midpoint of each segment. Thus, the weight wj, k that is assigned to edge ej, k is the sum of the lengths of all segments of the motion between qj and qk that do not afford visibility. The length can be computed in the configuration space or in the task space, depending on whether it is suitable to the application. The VBPRM edges are, thus, weighted with respect to their ability to satisfy the FOV constraint. C. Occlusion Constraint The occlusion constraint dictates that the planner should avoid motions that allow an object to interpose itself between the camera and the target. When the target position is known, we can compute the occluded region, i.e., a region of the task space from which the view of target is occluded by obstacles. In DVC, the MS algorithm detects configurations where the target moves outside the camera’s FOV. In the case of DOC, the MS algorithm detects configurations where the camera’s optical center moves in or out of the occluded region (see Fig. 2). 1) Occluded Space: The problem of computing the occluded space can be addressed using the techniques proposed by Tarabanis et al. [11]. The proposed boundary-based algorithm computes the boundary LV of the occluded region, which is created by objects described by a triangle

Authorized licensd use limted to: IE Xplore. Downlade on May 13,20 at 1:53 UTC from IE Xplore. Restricon aply.

mesh. We detect points at which the camera’s trajectory crosses this boundary in order to divide the camera’s motion into segments that do or do not afford target visibility. Lv partitions the task space P into the visible region Pv is and the occluded region Pocc . Any point in Pv is has an unobstructed line of sight to the target (see Fig. 2). 2) Distance to the Occlusion Boundary: In the case of occlusion avoidance, we define the distance function for DOC as dD O C (q) for a configuration q. dD O C differs from dD C C and dD V C in that dD O C determines the distance from the camera’s optical center to the boundary Lv in the task space. Thus, we have in dD O C (q) = FK(q) − P Pm Lv 

(7)

where FK(q) is a forward kinematics computation that determines the in camera’s optical center in configuration q, and P Pm L v is the closest point on Lv to FK(q). The adaptively sampled distance field [16] provides an efficient technique to perform multiple point-to-surface distance queries. Since dD O C (q) will be used in a sampling technique, such efficiency is highly desirable. 3) Length of Trajectory: The second component required for adaptive sampling is the displacement bound ℓD O C (q1 , q2 ). In the DCC collision detector, ℓD C C (qt ) represented the maximum displacement of any point on the robot due to the parameterized motion qt . In the case of the occlusion constraint, ℓD O C (qt ) represents the maximum possible displacement of the camera’s optical-center point due to the motion qt . We denote the displacement due to a segment of a motion as ℓD O C (q1 , q2 ), where q1 and q2 are the endpoints of that segment. 4) Motion Segmentation: The MS algorithm for DOC is identical to the one for FOV checking, i.e., Algorithm 1. However, in this case, we replace dD V C with dD O C and ℓD V C with ℓD O C . Thus, the MS algorithm now finds a set I of configurations along the motion between qj and qk , where dD O C < ǫ, thereby indicating that the camera is in contact with the boundary of Lv . Two such crossings are shown in Fig. 2, which are marked as “intersection.” Each segment of the motion must again be labeled as visible or invisible, which depends on whether it affords visibility of the target points or not. It is sufficient to perform a valence test between the midpoint of each segment and the target point to resolve this. A valence test determines how many times a line segment between two points intersects with a surface, namely, Lv . If the number of intersections is even, then the two points lie in the same partition of space. The target point, by definition, lies in the visible region and, thus, serves as a point of comparison. As in DVC, the weight wj, k that is assigned to edge ej, k is proportional to the length of the occluded segments of the motion from qj to

www.DownloadPaper.ir

IEEE TRANSACTIONS ON ROBOTICS, VOL. 26, NO. 1, FEBRUARY 2010

199

Fig. 4. Sequence of frames from the arm-mounted camera. The occluder has been marked in red for clarity. TABLE I EXPERIMENTAL VISIBILITY PENALTY t-TEST

Fig. 3. Experimental setup. The robot resides underneath an occluding panel with a square “S” cutout. The robot must move such that the sensor has a line of sight of the target above the panel. The green curve shows the intersection of the robot’s line of sight during the motion and the occluding plane. Note how the line of sight approximates the shape of the opening, thereby keeping the target visible through most of the path.

qk . The VBPRM edges are, thus, weighted with respect to their ability to satisfy the occlusion constraint. D. Constraint Integration A traditional PRM uses weights that are proportional to edge length as inputs for Dijkstra’s algorithm. Dijkstra’s algorithm will thus compute the shortest interpolations between a specified initial vertex qi and all reachable vertices. Replacing or mixing the length-based PRM weights with the visibility-based VBPRM weights will cause the paths to avoid edges with high visibility penalties, thus providing improved target visibility along the path, as permitted by the available edges in the road map. Ideally, the PRM would contain a very large number of vertices and edges to provide the maximum number of options for the planner. In practice, a PRM with a high number of vertices of low to moderate degree is sufficient, if the edges are chosen wisely. When creating the PRM, each vertex is connected to its k-nearest neighbors, where “nearest” is determined based on the difference in sensor position. A PRM consisting of these short-sensor-displacement motions will provide the path planner with the best array of options for paths whose constraints are based on the sensor position, i.e., occlusion avoidance. IV. EXPERIMENTS Our experiments test all the constraints involved in a VBPRM, namely, the planner must compute paths from a start configuration to a goal configuration, such that the manipulator avoids collisions with obstacles, and the motion maintains the visibility of the target for as much of the path as possible. A. Methods We tested the planner using a CRS A465 6-degree-of-freedom (DOF) robot manipulator with a Sony XC-HR70 camera mounted on the end-effector.The experimental setup is shown in Fig. 3. The robot must observe an overhead target through a square “S”-shaped open-

Authorized licensd use limted to: IE Xplore. Downlade on May 13,20 at 1:53 UTC from IE Xplore. Restricon aply.

ing in an occluding panel. The environment was modeled as a series of cuboids, for the purposes of collision avoidance, and as a triangle mesh, for the occlusion computations. We used a 5000-vertex, 104 078-edge road map with an average vertex degree of 41.6. Only vertices that oriented the camera with its axis above the horizontal plane were permitted. The initial PRM generation using DCC required less than 1 h to complete on a 2-GHz Intel Core 2 Duo. The visibility penalty computations required 26.1 h. The performance bottleneck is the point–mesh distance computation, which could be improved greatly by using an distance-field-based boundary representation [16]. B. Results Fig. 3 shows the resulting path from the initial to the final configurations in an example task. The intersection of the line of sight with the occluder’s plane is shown (green curve). When the curve overlaps the occluder, the line of sight is intersecting with the occluder, and thus, the target is invisible to the camera, such as at point (4), or after point (8). When the line of sight passes through the openings in the occluder, the target becomes visible. Fig. 4 shows eight images captured by the arm-mounted camera at the marked points on the path. The planned path moves the camera so that the line of sight largely follows the shape of the opening. Notably, the hard constraint of the collision-avoidance PRM means that the robot’s motion is restricted to collision-free motions, and therefore, it cannot approach too close to obstacles, such as a vertical pillar (see Fig. 3). Thus, at point (4), the robot is unable to extend far enough to avoid an occlusion, since collision avoidance takes precedence. On the opposite side [see points (6) and (7)], there is no obstacle to interfere, and thus, the planner can extend the path far enough to avoid occlusions. To validate the planner’s ability to improve the visibility of a target, we performed a t-test on 100 randomly selected queries in simulation on the same experimental setup. The planner computed both the shortest path in joint space (i.e., PRM) and the occlusion-avoiding path (i.e., VBPRM) between a pair of randomly selected vertices. The shortest paths without vision-based constraints had a penalty with a mean of 4.100 m. Using the VBPRM for the same queries produced a mean penalty of 0.713 m. The results of the t-test on the penalty difference between the PRM and VBPRM are summarized in Table I. In practical terms, the 95% confidence interval represents a reduction in penalized distance between 2.916 and 3.859 m, i.e., the

200

www.DownloadPaper.ir

IEEE TRANSACTIONS ON ROBOTICS, VOL. 26, NO. 1, FEBRUARY 2010

VBPRM reduced the distance that the camera traveled, while it was unable to see, by 71%–94%. Thus, the occlusion-aware planner made significant reductions to the distance that the camera must travel while it was unable to see the target. Simulations of different tasks showed similar reductions in penalized distance. V. CONCLUSION This paper introduced the VBPRM planner to compute paths that involve constraints that are inherent to industrial manipulators with the constraints of placing and orienting a camera in space. This research aims to provide manipulators, such as those found on assembly lines, with the capability to perform vision-based tasks that require motions with linear interpolation of joint positions, collision avoidance, and observation an image target with an onboard camera. Our strategy is based on extending the PRM by weighting the edges of the graph according to the visibility of a fixed target. Results presented in this paper demonstrate the value of the VBPRM for tasks that require a robot equipped with an eye-in-hand camera to move while avoiding collisions with obstacles and keeping a target within its sight. Future work will include performance improvements, particularly in the visibility penalty computation, which would benefit greatly from acceleration structures, such as an adaptively sampled distance field [16] for the geometric tests. Our experiments were conducted using a uniformly sampled road map; however, a road map whose vertices were sampled using a visibility or collision property as a bias function could potentially see improved results. The VBPRM provides a practical path planner to compute motions that satisfy vision constraints of an arm-mounted camera in a cluttered environment. Its compatibility with point-to-point interfaces, which is commonly found on industrial robots, makes it suitable to provide visibility awareness to existing industrial systems. REFERENCES [1] L. E. Kavraki, P. Svestka, J.-C. Latombe, and M. H. Overmars, “Probabilistic roadmaps for path planning in high-dimensional configuration space,” IEEE Trans. Robot. Autom., vol. 12, no. 4, pp. 566–580, Aug. 1996. [2] F. Chaumette and S. Hutchinson, “Visual servo control Part I: Basic approaches,” IEEE Robot. Autom. Mag., vol. 13, no. 4, pp. 82–90, Dec. 2006. [3] G. Chesi, K. Hashimoto, D. Prattichizzo, and A. Vicino, “Keeping features in the field of view in eye-in-hand visual servoing: A switching approach,” IEEE Trans. Robot., vol. 20, no. 5, pp. 908–913, Oct. 2004. [4] K. Hashimoto and T. Noritsugu, “Potential switching control in visual servo,” in Proc. IEEE Int. Conf. Robot. Autom., Apr. 2000, pp. 2765– 2770. [5] N. Mansard and F. Chaumette, “A new redundancy formalism for avoidance in visual servoing,” in Proc. IEEE/RSJ Int. Conf. Intell. Robots Syst., Edmonton, AB, Canada, Aug. 2005, pp. 468–474. [6] P. I. Corke and S. A. Hutchinson, “A new partitioned approach to imagebased visual servo control,” IEEE Trans. Robot. Autom., vol. 17, no. 4, pp. 507–515, Aug. 2001. [7] E. Malis, F. Chaumette, and S. Boudet, “2-1/2-d visual servoing,” IEEE Trans. Robot. Autom., vol. 15, no. 2, pp. 238–250, Apr. 1999. [8] Y. Mezouar and F. Chaumette, “Path planning for robust image-based control,” IEEE Trans. Robot. Autom., vol. 18, no. 4, pp. 534–549, Aug. 2002. [9] F. Schramm, F. Geffard, G. Morel, and A. Micaelli, “Calibration free image point path planning simultaneously ensuring visibility and controlling camera path,” in Proc. IEEE Int. Conf. Robot. Autom., Roma, Italy, Apr. 2007, pp. 2074–2079. [10] B. Thuilot, P. Martinet, L. Cordesses, and J. Gallice, “Position based visual servoing: Keeping the object in the field of vision,” in Proc. IEEE Int. Conf. Robot. Autom., Washington, DC, May 2002, pp. 1624–1629. [11] K. Tarabanis, R. Y. Tsai, and A. Kaul, “Computing occlusion-free viewpoints,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 18, no. 3, pp. 279– 292, Mar. 1996.

[12] T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein, Introduction to Algorithms, 1st ed. Cambridge, MA: MIT Press, 1997. [13] F. Schwarzer, M. Saha, and J.-C. Latombe, “Adaptive dynamic collision checking for single and multiple articulated robots in complex environments,” IEEE Trans. Robot., vol. 21, no. 3, pp. 338–353, Jun. 2005. [14] S. Leonard, E. A. Croft, and J. J. Little, “Dynamic visibility checking for vision-based motion planning,” in Proc. IEEE Int. Conf. Robot. Autom., Los Angeles, CA, May 2008, pp. 2283–2288. [15] E. Trucco and A. Verri, Introductory Techniques for 3-D Computer Vision. Englewood Cliffs, NJ: Prentice-Hall, 1998. [16] S. Frisken, R. Perry, A. Rockwood, and T. Jones, “Adaptively sampled distance fields: A general representation of shape for computer graphics,” in Proc. 27th Annu. Conf. Comput. Graph. Interactive Techn., 2000, pp. 249–254.

Autonomous Robotic Pick-and-Place of Microobjects Yong Zhang, Student Member, IEEE, Brandon K. Chen, Student Member, IEEE, Xinyu Liu, Student Member, IEEE, and Yu Sun, Senior Member, IEEE Abstract—This paper presents a robotic system that is capable of both picking up and releasing microobjects with high accuracy, reliability, and speed. Due to force-scaling laws, large adhesion forces at the microscale make rapid, accurate release of microobjects a long-standing challenge in micromanipulation, thus representing a hurdle toward automated robotic pick-and-place of micrometer-sized objects. The system employs a novel microelectromechanical systems (MEMS) microgripper with a controllable plunging structure to impact a microobject that gains sufficient momentum to overcome adhesion forces. The performance was experimentally quantified through the manipulation of 7.5–10.9 µm borosilicate glass spheres in an ambient environment. Experimental results demonstrate that the system, for the first time, achieves a 100% success rate in release (which is based on 700 trials) and a release accuracy of 0.45±0.24 µm. High-speed, automated microrobotic pick-and-place was realized by visually recognizing the microgripper and microspheres, by visually detecting the contact of the microgripper with the substrate, and by vision-based control. Example patterns were constructed through automated microrobotic pick-and-place of microspheres, achieving a speed of 6 s/sphere, which is an order of magnitude faster than the highest speed that has been reported in the literature. Index Terms—Adhesion forces, automated operation, microelectromechanical systems (MEMS) microgrippers, micromanipulation, robotic pick-and-place.

I. INTRODUCTION The past decade has witnessed significant efforts in the pursuit of automated robotic operation at the micrometer scale [1]. Among many types of microrobotic operation, pick-and-place of microobjects promises specificity, precision, and programmed motion, which are Manuscript received March 28, 2009; revised July 23, 2009. First published December 1, 2009; current version published February 9, 2010. This paper was recommended for publication by Associate Editor M. Sitti and Editor K. Lynch upon evaluation of the reviewers’ comments. This work was supported by the Natural Sciences and Engineering Research Council of Canada under a Discovery Grant, by the Ontario Ministry of Research and Innovation under an Early Researcher Award Program and a Proof of Principle (POP) Grant, by the Canada Research Chairs Program, and by Canadian Microelectronics Corporation under a Financial Assistance Program for Microfabrication. This paper was presented in part at the IEEE International Conference on Robotics and Automation, Kobe, Japan, May 2009. The authors are with the Advanced Micro and Nanosystems Laboratory, University of Toronto, Toronto, ON M5S 3G8, Canada (e-mail: sun@mie. utoronto.ca). Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org. Digital Object Identifier 10.1109/TRO.2009.2034831

1552-3098/$26.00 © 2009 IEEE Authorized licensd use limted to: IE Xplore. Downlade on May 13,20 at 1:53 UTC from IE Xplore. Restricon aply.